MINOR: http-client: Add an option to not stop the reception of the response

Since commit b24260ec9 ("BUG/MEDIUM: http-client: Only consume input buffer
when hc one is empty"), the http-client applet stops to consume data as soon
as its response buffer is not empty. The applet is only unblocked when the
caller consumes these data by calling httpclient_res_xfer().

However, some callers are not consuming the response payload on the
fly. They are waiting for the end of the response to retrieve the whole
payload from the httpclient response buffer. For these callers,
httpclient_res_xfer() is never called. So the applet remains blocked and the
response is never fully received. The transfer is stuck till the server
timeout expiration.

To fix this, the HTTPCLIENT_O_RES_ACCUM option is added. It must be set by
the callers accumulating the whole response payload in the httpclient
response buffer. In that case, the applet must not stop the reception. Of
course, it means the whole payload must fit in the response buffer for these
callers. But it was already the case.

This patch depends on:

  * "BUG/MEDIUM: http-client: Don't use the httpclient context if it was released".

It is required to fix a bug and must be backported as far as 3.3 with the
next commits.

diff --git a/include/haproxy/http_client-t.h b/include/haproxy/http_client-t.h
index 1040c1f8507d1c7ada1870e00f6f0666e7b19f65..d862e330598d8e4609ff1b74fadb5ea7ad627ea6 100644 (file)
--- a/include/haproxy/http_client-t.h
+++ b/include/haproxy/http_client-t.h
@@ -52,6 +52,8 @@ struct httpclient {
 /* options */
 #define    HTTPCLIENT_O_HTTPPROXY     0x00000001 /* the request must use an absolute URI */
 #define    HTTPCLIENT_O_RES_HTX      0x00000002 /* response is stored in HTX */
+#define    HTTPCLIENT_O_RES_ACCUM    0x00000004 /* The whole response payload is accumulated in <res.buf>. The caller
+                                                 * only retrieves it when the response is finished. */
 
 /* States of the HTTP Client Appctx */
 enum {

diff --git a/src/http_client.c b/src/http_client.c
index d2b75309c1bbbdad6dd4b6d941370f745028946e..d97ec0e30f2bf08d62bcad25cb5e18c748a8e44d 100644 (file)
--- a/src/http_client.c
+++ b/src/http_client.c
@@ -879,8 +879,17 @@ void httpclient_applet_io_handler(struct appctx *appctx)
        }
 
 out:
-       if (hc && appctx->st0 != HTTPCLIENT_S_RES_END && !b_is_null(&hc->res.buf)) {
-               /* Don't accept more data while the httpclient response buffer is not empty */
+       if (hc && appctx->st0 != HTTPCLIENT_S_RES_END &&
+           !(hc->options & HTTPCLIENT_O_RES_ACCUM) && !b_is_null(&hc->res.buf)) {
+               /* Don't accept more data while the httpclient response buffer is
+                * not empty. It will be re-enabled by the caller, when it will
+                * consume these data.
+                *
+                * This is only performed for callers consuming the response on
+                * the fly. The others expect to retrieve the whole payload at
+                * the end of the response. For these ones, we must continue to
+                * fill the response buffer.
+                */
                applet_wont_consume(appctx);
        }
        return;