From: Willy Tarreau Date: Mon, 28 Sep 2026 03:21:40 +0000 (+0200) Subject: MINOR: log: add the +utf8 encoding option to let valid UTF-8 pass X-Git-Tag: v3.5-dev8~107 X-Git-Url: https://git.kaiwu.me/?a=commitdiff_plain;h=82a9570b3d8e02935450cf8f6adb3c32400322ac;p=haproxy.git MINOR: log: add the +utf8 encoding option to let valid UTF-8 pass Since commit db9492bcf ("BUG/MEDIUM: log: encode the whole non-printable range in the +json output"), the +json encoder escapes every byte outside the 32..126 range. This guarantees a pure ASCII output that cannot corrupt a terminal, and it also purposely affects valid UTF-8 text: U+00E9, encoded as 0xC3 0xA9, is logged as "\u00C3\u00A9", which a JSON decoder turns into U+00C3 U+00A9. Some users relying on +json to log non-ASCII text now get unreadable records. User @Wahnes reported in GH issue #3512 that it breaks their use case where UTF-8 was expected to land as-is in the output, as permitted by RFC 8259. However, the patch above purposely avoided this because a number of valid UTF-8 sequences can have effects on the terminal, such as direction overrides (RTL/LTR), zero-width characters and a few others that could affect how the text is rendered, and be used to hide actvities if retrieved verbatim from header field values for example. This patch adds a new "+utf8" log option that explicitly lets valid UTF-8 pass unencoded in JSON, and only that, so that non-utf-8 bytes above 126 remain encoded. The C1 class of control chars (0x80..0x9f) which may appear encoded in UTF-8 (e.g. 0xc2 0x9b for 0x9b or \u009b) remain encoded to protect terminals. This flag is stored in a new flag LOG_OPT_UTF8 that is only used by LOG_OPT_ENCODE_JSON for now. The doc and the regtest were updated accordingly. This may be backported wherever db9492bcf is backported, along with the previous commit, so that users of versions where it gets backported can restore the ability to log UTF-8 text. --- diff --git a/doc/configuration.txt b/doc/configuration.txt index 91e235f77..7c5299963 100644 --- a/doc/configuration.txt +++ b/doc/configuration.txt @@ -29755,6 +29755,24 @@ Supported item flags are (may be enabled/disabled from item's arguments): Incomplete numerical values (e.g.: '%B' when logasap is used), which are normally prefixed with '+' without encoding, will be encoded as-is. Also, '+E' option will be ignored. + As described in section 8.6, characters outside the 32..126 range + are escaped one byte at a time, using the JSON escape sequences, + so that the output remains pure printable ASCII whatever the + input. This includes valid UTF-8 sequences, meaning that non-ASCII + text will appear as its individual bytes once decoded (e.g. + U+00E9, encoded as 0xC3 0xA9, is logged as "\u00C3\u00A9"). See + "utf8" below to preserve such text. + * utf8: combined with "json", permits to let valid multi-byte UTF-8 sequences + pass unescaped. The C1 control characters (U+0080 to U+009F) are + still escaped, represented by their code point. Invalid sequences are + escaped one byte at a time so that the output always remains valid + UTF-8. Note that since valid UTF-8 is emitted as-is, a client may use + some special characters to alter the rendering of a log line on a + terminal or in a log viewer, such as direction changes, zero-width + characters, or line and paragraph separators. Thus it is recommended + to only use it for contents coming from trusted sources, or when logs + are not meant to be displayed on a terminal. It has no effect without + "json". * cbor: automatically encode value in CBOR format (when set globally, only named logformat items are considered) By default, cbor encoded data is represented in HEX form so @@ -29775,6 +29793,7 @@ Supported item flags are (may be enabled/disabled from item's arguments): log-format-sd %{+Q,+E}o\ [exampleSDID@1234\ header=%[capture.req.hdr(0)]] log-format "%{+json}o %(request)r %(custom_expr)[str(custom)]" + log-format "%{+json,+utf8}o %(request)r %(custom_expr)[str(custom)]" log-format "%{+cbor}o %(request)r %(custom_expr)[str(custom)]" Please refer to the table below for currently defined aliases : @@ -30753,6 +30772,10 @@ This does not apply to the cbor encoding, which already uses either safe hex digits, or properly delimited binary blocks that will not be sent to a terminal but will be consumed by a CBOR parser. +The json encoding uses the JSON escape sequences instead, and the +utf8 option +also lets valid multi-byte UTF-8 sequences pass as-is (see the "utf8" flag in +section 8.2.6). + Note that the space character (' ') is not encoded in headers, which can cause issues for tools relying on space count to locate fields. A typical header containing spaces is "User-Agent". diff --git a/include/haproxy/log-t.h b/include/haproxy/log-t.h index 7bca3b7a6..db84cbc99 100644 --- a/include/haproxy/log-t.h +++ b/include/haproxy/log-t.h @@ -48,7 +48,8 @@ #define LOG_OPT_ESC 0x00000040 #define LOG_OPT_MERGE_SPACES 0x00000080 #define LOG_OPT_BIN 0x00000100 -/* unused: 0x00000200 ... 0x00000800 */ +#define LOG_OPT_UTF8 0x00000200 // only used with LOG_OPT_ENCODE_JSON (utf-8 pass-through) +/* unused: 0x00000400 ... 0x00000800 */ #define LOG_OPT_ENCODE_JSON 0x00001000 #define LOG_OPT_ENCODE_CBOR 0x00002000 #define LOG_OPT_ENCODE 0x00003000 diff --git a/reg-tests/log/log_json.vtc b/reg-tests/log/log_json.vtc index f2fe4dc87..673f75be7 100644 --- a/reg-tests/log/log_json.vtc +++ b/reg-tests/log/log_json.vtc @@ -6,7 +6,9 @@ feature ignore_unknown_macro # JSON encoding has no reason to be more permissive than the plain one. # Escaping them all also makes the output valid UTF-8, which RFC 8259 #8.1 # requires, without having to guess what the logged bytes represent: a header -# value is arbitrary bytes, not text. +# value is arbitrary bytes, not text. The +utf8 option bypasses this, letting +# valid multi-byte UTF-8 sequences pass as-is, and still escapes all other +# bytes above 126 one at a time so that the output remains valid UTF-8. haproxy h1 -conf { global @@ -18,6 +20,10 @@ haproxy h1 -conf { size 16384 format raw + ring rawring + size 16384 + format raw + defaults mode http timeout connect "${HAPROXY_TEST_TIMEOUT-5s}" @@ -30,6 +36,13 @@ haproxy h1 -conf { log ring@myring local0 log-format "{\"hdr\":%{+json}[capture.req.hdr(0)],\"bin\":%{+json}[str(Gw==),b64dec]}" http-request return status 200 + + frontend fe_raw + bind "fd@${fe_raw}" + capture request header x len 64 + log ring@rawring local0 + log-format "%{+json,+utf8}o %(hdr)[capture.req.hdr(0)] %(bin)[str(w6nCm+KC),b64dec]" + http-request return status 200 } -start # an ESC, which has no two-character form, a tab, which has one, a double quote @@ -48,6 +61,15 @@ client c2 -connect ${h1_fe_sock} { expect resp.status == 200 } -run +# in raw mode: a valid UTF-8 sequence, a lone continuation byte, an overlong +# form, an encoded surrogate, a C1 control character, a 4-byte sequence, one +# beyond U+10FFFF and a sequence truncated by the end of the value +client c3 -connect ${h1_fe_raw_sock} { + txreq -url "/raw" -hdr "x: A\xc3\xa9B\xaaC\xc0\x81D\xed\xa0\x80E\xc2\x9bF\xf0\x9f\x98\x80G\xf4\x90\x80\x80H\xe2\x82" + rxresp + expect resp.status == 200 +} -run + delay 0.2 # the ESC comes out as the six-character form and the tab as the short one, in @@ -75,3 +97,18 @@ haproxy h1 -cli { send "show events myring -s" expect !~ "\\?" } + +# in raw mode, the valid UTF-8 sequences are emitted as-is, the C1 control +# character as its code point's escape sequence, and all other bytes above the +# printable range get one escape per byte, so that the output remains valid +# UTF-8 +haproxy h1 -cli { + send "show events rawring" + expect ~ "\"hdr\": \"A\xc3\xa9B\\\\u00AAC\\\\u00C0\\\\u0081D\\\\u00ED\\\\u00A0\\\\u0080E\\\\u009BF\xf0\x9f\x98\x80G\\\\u00F4\\\\u0090\\\\u0080\\\\u0080H\\\\u00E2\\\\u0082\"" +} + +# same in the binary sample path, where the truncated sequence ends the sample +haproxy h1 -cli { + send "show events rawring" + expect ~ "\"bin\": \"\xc3\xa9\\\\u009B\\\\u00E2\\\\u0082\"" +} diff --git a/src/log.c b/src/log.c index 8c1beb304..18c93dacb 100644 --- a/src/log.c +++ b/src/log.c @@ -428,6 +428,7 @@ struct logformat_node_args node_args_list[] = { { "X", LOG_OPT_HEXA }, { "E", LOG_OPT_ESC }, { "bin", LOG_OPT_BIN }, + { "utf8", LOG_OPT_UTF8 }, { "json", LOG_OPT_ENCODE_JSON }, { "cbor", LOG_OPT_ENCODE_CBOR }, { 0, 0 } @@ -2021,6 +2022,7 @@ struct lf_buildctx { int options; /* LOG_OPT_* options */ int typecast; /* same as logformat_node->typecast */ int in_text; /* inside variable-length text */ + int utf8_skip; /* JSON: continuation bytes of an already emitted UTF-8 sequence */ }; static THREAD_LOCAL char lf_buildbuf[256]; /* fixed size buffer for building small strings */ @@ -2267,9 +2269,57 @@ static inline char *_lf_json_escape_seq(char *start, char *stop, unsigned char b return start; } +/* Checks whether starts with a multi-byte UTF-8 sequence which is valid as + * per RFC 3629 and does not span more than bytes: no overlong form, no + * surrogate (U+D800..U+Dfff), nothing beyond U+10FFFF (U+FFFE and U+FFFF, + * which utf8_next() rejects as well, are simply escaped). The sequence may be + * terminated by a NUL byte before bytes, since it is not a continuation + * byte. On success the code point is stored into and the sequence length + * (2 to 4) is returned, otherwise zero is returned. + */ +static inline int _lf_json_utf8_len(const char *s, size_t len, unsigned int *c) +{ + unsigned char ret; + + ret = utf8_next(s, MIN(len, 4), c); + if (utf8_return_code(ret) != UTF8_CODE_OK) + return 0; + + ret = utf8_return_length(ret); + if (ret < 2 || ret > 4 || *c > 0x10FFFF) + return 0; + + return ret; +} + +/* Writes at the valid UTF-8 sequence of bytes found at and + * representing code point , as reported by _lf_json_utf8_len(). It is + * copied as-is, except for the C1 control characters (U+0080 .. U+009F) which, + * just like the C0 ones (U+0000 .. U+001F), could be interpreted by a + * terminal, and which are written as their \u00XX escape sequence instead. + * + * The function returns the address of the byte immediately after the last + * written byte, or NULL if the whole sequence does not fit before , so + * that a sequence is never truncated. It will not append a terminating NULL + * byte. + */ +static inline char *_lf_json_utf8_seq(char *start, char *stop, + const char *seq, int len, unsigned int c) +{ + if (c < 0xa0) + return _lf_json_escape_seq(start, stop, c); + + if (start + len > stop) + return NULL; + memcpy(start, seq, len); + return start + len; +} + /* helper function for _lf_encode_bytes() to encode a single byte * and escape it with if found in or escape it with - * '\' if found in json_escape_map + * '\' if found in json_escape_map. With LOG_OPT_UTF8, valid multi-byte + * UTF-8 sequences are emitted at once when their first byte is met + * instead, the following ones being skipped. * * The function assumes that at least 1 byte is available for writing * @@ -2282,9 +2332,24 @@ static inline char *_lf_json_escape_byte(char *start, char *stop, const char **pending, uint8_t cbor_string_prefix, struct lf_buildctx *ctx) { + unsigned int c; + int len; + + if (ctx->utf8_skip) { + /* already emitted with the sequence's first byte */ + ctx->utf8_skip--; + return start; + } + if (!ha_bit_test((unsigned char)(*byte), map)) { if (!ha_bit_test((unsigned char)(*byte), json_escape_map)) *start++ = *byte; + else if ((ctx->options & LOG_OPT_UTF8) && + (len = _lf_json_utf8_len(byte, end ? end - byte : 4, &c))) { + start = _lf_json_utf8_seq(start, stop, byte, len, c); + if (start) + ctx->utf8_skip = len - 1; + } else start = _lf_json_escape_seq(start, stop, *byte); } @@ -2326,8 +2391,10 @@ static char *_lf_encode_bytes(char *start, char *stop, const char **pending, uint8_t cbor_string_prefix, struct lf_buildctx *ctx); - if (ctx->options & LOG_OPT_ENCODE_JSON) + if (ctx->options & LOG_OPT_ENCODE_JSON) { encode_byte = _lf_json_escape_byte; + ctx->utf8_skip = 0; + } else if (ctx->options & LOG_OPT_ENCODE_CBOR) encode_byte = _lf_cbor_map_escape_byte; else if (ctx->options & LOG_OPT_ESC) @@ -2443,26 +2510,40 @@ static char *lf_encode_chunk(char *start, char *stop, /* Same as escape_string() but for a JSON string content: the bytes tagged in * json_escape_map cannot all be expressed with the two-character form that - * escape_string() emits, the others need the \u00XX one. + * escape_string() emits, the others need the \u00XX one. When is set, + * valid multi-byte UTF-8 sequences are emitted as-is instead (see + * _lf_json_utf8_seq()). * * Return the address of the \0 character, or NULL on error */ static char *_lf_json_escape_string(char *start, char *stop, - const char *string, const char *string_stop) + const char *string, const char *string_stop, + int raw) { if (start < stop) { stop--; /* reserve one byte for the final '\0' */ while (start < stop && string < string_stop && *string != '\0') { - if (!ha_bit_test((unsigned char)(*string), json_escape_map)) - *start++ = *string; - else { - char *next = _lf_json_escape_seq(start, stop, *string); + unsigned int c; + char *next; + int len; + + if (!ha_bit_test((unsigned char)(*string), json_escape_map)) { + *start++ = *string++; + continue; + } - if (!next) - break; /* does not fit, truncate here */ - start = next; + len = raw ? _lf_json_utf8_len(string, string_stop - string, &c) : 0; + if (len) + next = _lf_json_utf8_seq(start, stop, string, len, c); + else { + len = 1; + next = _lf_json_escape_seq(start, stop, *string); } - string++; + + if (!next) + break; /* does not fit, truncate here */ + start = next; + string += len; } *start = '\0'; return start; @@ -2517,7 +2598,8 @@ static inline char *_lf_text_len(char *dst, const char *src, else if (ctx->options & LOG_OPT_ENCODE_JSON) { char *ret; - ret = _lf_json_escape_string(dst, dst + size, src, src + len); + ret = _lf_json_escape_string(dst, dst + size, src, src + len, + !!(ctx->options & LOG_OPT_UTF8)); if (ret == NULL) return NULL; len = ret - dst; @@ -3790,7 +3872,9 @@ static void init_log() * spares us from having to care about the UTF-8 validity that RFC 8259 * #8.1 demands, since a header value is made of arbitrary bytes and not * of text. On top of that, RFC 8259 #7 mandates escaping the double - * quote, the backslash and everything below 0x20. + * quote, the backslash and everything below 0x20. The raw JSON encoding + * is the only exception: its encoders detect the valid multi-byte UTF-8 + * sequences by themselves and emit them as-is. */ memset(json_escape_map, 0, sizeof(json_escape_map));