OPTIM: lb-chash: limit the scan to few consecutive nodes

Strongly imbalanced configs such as one server of weight 256 and another
one of weight 1 can result in high CPU usage with consistent hashing when
the first attempted server is the one to be avoided, or when affinity is
set to maxconn or maxqueue, because in this case, the frequent server may
appear a lot of consecutive times, on average 256 after a node before
meeting the other server, and if terribly unlucky, 4096 times.

Also in properly balanced farms, all servers are more or less well
distributed, so that after visiting a number of them we've seen them
all.

This patch takes this into account to avoid too long scans of the whole
tree caused by such imbalances. What it does is:
  - limit the search of another server to 3 consecutive nodes of the
    same server. If this happens, the tree is severely imbalanced so
    there's no point looking for another server: not only will this
    take a lot of time, but it may even result in a lot of traffic
    being transferred to it at ratios well above the weights ratios ;

  - limit the search of an eligible server to twice the number of servers
    in the farm, considering that once we've witnessed 2N server changes,
    it's fairly unlikely that a remaining server is worth being considered
    beyond that point (as a reminder, each server is present 16 times).

For the last point we could also have decided to stop based on the
distance, e.g. 1/8 of the whole spectrum, but it would be more
probabilistic and there it remains understandable.

diff --git a/src/lb_chash.c b/src/lb_chash.c
index cc63a9ae79bf2ded32126efdac6753cb08938dbb..6d54e9f35c6a9d3b76b346087c0a3392fe4ae876 100644 (file)
--- a/src/lb_chash.c
+++ b/src/lb_chash.c
@@ -471,6 +471,11 @@ struct server *chash_get_server_hash(struct proxy *p, unsigned int hash, const s
                nsrv = psrv;
        }
 
+       /* arbitrary search limitation to twice the number of server, offering
+        * enough chances to spot a non-saturated server when using maxconn
+        * with many servers.
+        */
+       tot *= 2;
        loop = 0;
        hashafnty = p->options3 & PR_O3_HASHAFNTY_MASK;
 
@@ -483,13 +488,29 @@ struct server *chash_get_server_hash(struct proxy *p, unsigned int hash, const s
                                nsrv->maxconn &&
                                nsrv->maxqueue &&
                                nsrv->served + nsrv->queueslength >= srv_dynamic_maxconn(nsrv) + nsrv->maxqueue)) {
-               next = eb32_next(next);
-               if (!next) {
-                       next = eb32_first(root);
-                       if (++loop > 1) // protection against accidental loop
-                               break;
-               }
-               nsrv = eb32_entry(next, struct tree_occ, node)->server;
+               /* try to find another server in consecutive nodes */
+               int tries = 3;
+               struct server *osrv = nsrv;
+
+               do {
+                       next = eb32_next(next);
+                       if (!next) {
+                               next = eb32_first(root);
+                               if (++loop > 1) // protection against accidental loop
+                                       goto out;
+                       }
+                       nsrv = eb32_entry(next, struct tree_occ, node)->server;
+
+                       /* too many times the same server => strongly imbalanced
+                        * weights, don't insist, we can have up to 4k of these!
+                        */
+                       if (--tries <= 0)
+                               goto out;
+               } while (unlikely(nsrv == osrv));
+
+               /* tried too many servers ? */
+               if (--tot <= 0)
+                       goto out;
        }
 
  out: