BUG/MINOR: haload: -ee could burst past its floor when many users fail at once

On a large, near-simultaneous outage (e.g. -u 10000 on several
threads), -ee's 1 req/s floor could be blown past. A burst of several
thousand requests could fire right when the ramp starts. This defeats
the whole point of probing gently.

Two separate gaps caused this:

  - hld_rate_task() creates replacement users after a kill. It only
    counted a new user once it later connected, not when it was
    created. Many kills in a row wake this task up again and again,
    before any replacement has had time to run. Each time, it still
    saw an empty counter, and created one more than it should. It now
    counts a new user right when it decides to create it.

  - hld_rate_task() has a "no rate limit, fill directly" path. It is
    used to refill users after a kill when -R is not set. It only
    checked arg_rate. On a mass failure, another thread can already be
    inside hld_err_trigger(), with its new arg_rate not visible here
    yet. This path would then still fire, and fill the whole thread's
    quota at once. It now also checks err_ramp_pending. This flag is
    set first by hld_err_trigger(). Seeing it at 0 means no trigger is
    running.

diff --git a/src/haload.c b/src/haload.c
index 0dbae94c657103873f977304fdb4b5a37f2efd1a..4d0850edac4a69fcaf7a11b9c5c60ca772774f47 100644 (file)
--- a/src/haload.c
+++ b/src/haload.c
@@ -2217,13 +2217,16 @@ static struct task *hld_rate_task(struct task *t, void *context, unsigned int st
                        nb_usr = nb_usr ? nb_usr : 1;
                }
 
-               if (!HA_ATOMIC_LOAD(&arg_rate) && HA_ATOMIC_LOAD(&usr_cnt) >= arg_usr) {
-                       /* 0 means no rate limit, same as everywhere else in
-                        * this file. Only take over once the initial ramp-up
-                        * (done directly by the main task, see usr_cnt) is
-                        * over: fill up to nb_usr directly, no per-thread
-                        * budget, so a kill under -ee can still be refilled
-                        * even without -R.
+               if (!HA_ATOMIC_LOAD(&arg_rate) && !HA_ATOMIC_LOAD(&err_ramp_pending) &&
+                   HA_ATOMIC_LOAD(&usr_cnt) >= arg_usr) {
+                       /* 0 means no rate limit here too. Only do this after
+                        * the main task's initial ramp-up (usr_cnt >= arg_usr):
+                        * fill up to nb_usr directly, no per-thread budget, so
+                        * a kill under -ee can still be refilled without -R.
+                        *
+                        * err_ramp_pending is checked too. It is set first by
+                        * hld_err_trigger(). Seeing it at 0 means no other
+                        * thread just started a trigger.
                         */
                        budget = nb_usr;
                }
@@ -2255,6 +2258,12 @@ static struct task *hld_rate_task(struct task *t, void *context, unsigned int st
                                break;
                        }
 
+                       /* count this new user right now, not later when it
+                        * really connects. Without this, a fast re-wakeup of
+                        * this task (many kills in a row) would still see an
+                        * empty counter and create too many replacements.
+                        */
+                       hld_update_freq_ctr(&thrs_info[tid].conn_rate, 1, date);
                        HA_ATOMIC_INC(&running_tasks);
                }