BUG/MEDIUM: mux-h2: serve all pending buffer allocations in the buffer callback
The demux buffer, the mux buffer and the streams' rxbufs all share the
same buffer_wait entry in the h2c, and the blocking causes are indicated
by the H2_CF_DEM_DALLOC, H2_CF_MUX_MALLOC and H2_CF_DEM_SALLOC flags.
However h2_buf_available() only serves the first cause it manages to
allocate for and returns 1, which dequeues the connection. If another
cause was pending, its flag remains set while the connection is not in
the queue anymore, so nothing will ever retry it:
- a stale H2_CF_DEM_DALLOC prevents h2_recv() from receiving anymore;
- a stale H2_CF_DEM_SALLOC blocks the demux forever;
- a stale H2_CF_MUX_MALLOC prevents h2_send() from calling
h2_process_mux(), so that window updates, pings and GOAWAY frames are
not sent anymore, and streams blocked on the connection's flow
control are never resumed.
This was reproduced with a debug patch faking both a demux buffer and a
mux buffer allocation failure on an H2 connection, then calling
offer_buffers() once 200ms later: a GET for a 200kB object forwarded to
a server stops after exactly 65535 bytes of DATA despite the client's
WINDOW_UPDATE frames, and stays there until the client gives up after
5s, while the whole object is received instantly once both causes are
served.
Let's serve all pending causes in a single call, wake the connection up
if any progress was made, and only accept to leave the queue once none
remain. This needs two simultaneous buffer allocation failures on the
same connection, or a single one combined with the issue fixed by commit
"BUG/MEDIUM: mux-h2: always return already allocated buffers in
h2_get_buf()", so it's rare. The problem exists since the buffer
allocators were merged in 1.9 by commit
44e973f50 ("MEDIUM: h2: use a
single buffer allocator"). The fix may be backported to all supported
versions after the aforementioned one.