From: Manu Nicolas Date: Wed, 2 Sep 2026 10:09:14 +0000 (+0000) Subject: MEDIUM: resolvers: retry truncated UDP responses over TCP X-Git-Tag: v3.5-dev6~19 X-Git-Url: https://git.kaiwu.me/?a=commitdiff_plain;h=bc20f8c0b;p=haproxy.git MEDIUM: resolvers: retry truncated UDP responses over TCP RFC 1123 and RFC 2181 recommend retrying a truncated DNS response over a transport capable of carrying a larger message. HAProxy instead treats TC as a resolver error unless the nameserver is configured to use TCP for every query. Give unprefixed IPv4, IPv6 and hostname nameservers both a datagram and a stream server. Queries use UDP first and the existing stream code opens TCP connections lazily. Explicit datagram prefixes remain UDP-only, explicit stream prefixes remain TCP-only, and nameservers loaded from resolv.conf gain the default fallback behavior. When a valid UDP response is truncated, rebuild its query over TCP with the same ID, question type and nameserver. Taking the type from the response preserves late A or AAAA responses received after a query-type switch. Queueing the fallback refreshes the retry deadline without consuming a retry or initiating another A/AAAA switch. A truncated SRV response can still be partially processed when no TCP fallback can be initiated for that response. This preserves existing behavior for explicit UDP and TCP nameservers and processes a partial SRV response received during TCP fallback. Other truncated TCP responses follow the normal error path, and no TCP response triggers another fallback. Failed queue operations use the existing send-error path, while connection failures are handled by the resolver timeout and retry path. Add a tcp_fallback counter to the CLI, resolver statistics and Prometheus output. It counts fallback queries successfully queued to the stream server. --- diff --git a/addons/promex/README b/addons/promex/README index 9ed37debe..978052ec8 100644 --- a/addons/promex/README +++ b/addons/promex/README @@ -430,5 +430,6 @@ listed below. Metrics from extra counters are not listed. | haproxy_resolver_other | | haproxy_resolver_invalid | | haproxy_resolver_too_big | +| haproxy_resolver_tcp_fallback | | haproxy_resolver_outdated | +----------------------------------------------------+ diff --git a/doc/configuration.txt b/doc/configuration.txt index 26e930f4b..bb80e83be 100644 --- a/doc/configuration.txt +++ b/doc/configuration.txt @@ -20391,10 +20391,24 @@ accepted_payload_size nameserver
[:port] [param*] Used to configure a nameserver. of the nameserver should ne unique. - By default the
is considered of type datagram. This means if an - IPv4 or IPv6 is configured without special address prefixes (paragraph 11.) - the UDP protocol will be used. If an stream protocol address prefix is used, - the nameserver will be considered as a stream server (TCP for instance) and + By default an IPv4, IPv6 or hostname
is queried over UDP. When a + valid UDP response has its truncated (TC) flag set, the same query is retried + over TCP against the same address and port. The TCP connection is established + lazily, only after a truncated response is received, and uses the regular DNS + stream connection pooling, pipelining, timeouts and idle connection cleanup. + + An explicit datagram address prefix such as "udp@", "udp4@", "udp6@" or + "dgram+" forces UDP-only operation and disables TCP fallback. An explicit + stream address prefix such as "tcp@" forces TCP-only operation. + + A failure to connect or send over TCP is handled by the regular resolver + timeout and retry mechanism. Starting TCP fallback does not consume a retry + or switch the query between A and AAAA. A truncated response received over + TCP follows the normal resolver error path and does not start another + fallback. + + If a stream protocol address prefix is used, the nameserver will be + considered as a stream server (TCP for instance) and "server" parameters found in 5.2 paragraph which are relevant for DNS resolving will be considered. Note: currently, in TCP mode, 4 queries are pipelined on the same connections. A batch of idle connections are removed @@ -20404,7 +20418,8 @@ nameserver
[:port] [param*] parse-resolv-conf Adds all nameservers found in /etc/resolv.conf to this resolvers nameservers list. Ordered as if each nameserver in /etc/resolv.conf was individually - placed in the resolvers section in place of this directive. + placed in the resolvers section in place of this directive. These nameservers + use UDP normally and retry truncated responses over TCP. hold Upon receiving the DNS response , determines whether a server's state diff --git a/include/haproxy/dns-t.h b/include/haproxy/dns-t.h index cf3c21043..39dac4b85 100644 --- a/include/haproxy/dns-t.h +++ b/include/haproxy/dns-t.h @@ -154,7 +154,7 @@ struct dns_nameserver { int (*process_responses)(struct dns_nameserver *ns, enum dns_server_type type); /* callback used to process responses */ struct dns_dgram_server *dgram; /* used for dgram dns */ - struct dns_stream_server *stream; /* used for tcp dns */ + struct dns_stream_server *stream; /* used for stream DNS and TCP fallback */ EXTRA_COUNTERS(extra_counters); char *extra_counters_storage; /* storage used for extra_counters above */ @@ -184,7 +184,8 @@ struct dns_counters { long long invalid; /* - malformed DNS response */ long long too_big; /* - too big response */ long long outdated; /* - outdated response (server slower than the other ones) */ - long long truncated; /* - truncated response */; + long long truncated; /* - truncated response */ + long long tcp_fallback; /* - truncated response retried over TCP */ } resolver; } app; /* application specific counteurs */ }; diff --git a/include/haproxy/dns.h b/include/haproxy/dns.h index c929415d0..13b38b424 100644 --- a/include/haproxy/dns.h +++ b/include/haproxy/dns.h @@ -43,4 +43,10 @@ static inline int dns_server_type_is_stream(enum dns_server_type type) return type == DNS_SERVER_STREAM; } +/* Returns nonzero if owns both servers needed for TCP fallback. */ +static inline int dns_nameserver_has_fallback(const struct dns_nameserver *ns) +{ + return ns->dgram && ns->stream; +} + #endif // _HAPROXY_DNS_H diff --git a/include/haproxy/resolvers-t.h b/include/haproxy/resolvers-t.h index cc923c851..6b32c7bc8 100644 --- a/include/haproxy/resolvers-t.h +++ b/include/haproxy/resolvers-t.h @@ -218,6 +218,7 @@ struct resolv_resolution { int try; /* current resolution try */ int nb_queries; /* count number of queries sent */ int nb_responses; /* count number of responses received */ + int nb_tcp_fallback; /* count TCP fallback responses still outstanding */ struct resolv_response response; /* structure hosting the DNS response */ struct resolv_query_item response_query_records[DNS_MAX_QUERY_RECORDS]; /* query records */ diff --git a/src/resolvers.c b/src/resolvers.c index e9d97a67a..1bb98ec3a 100644 --- a/src/resolvers.c +++ b/src/resolvers.c @@ -96,28 +96,30 @@ enum { RSLV_STAT_INVALID, RSLV_STAT_TOO_BIG, RSLV_STAT_TRUNCATED, + RSLV_STAT_TCP_FALLBACK, RSLV_STAT_OUTDATED, RSLV_STAT_END, }; static struct stat_col resolv_stats[] = { - [RSLV_STAT_ID] = { .name = "id", .desc = "ID" }, - [RSLV_STAT_PID] = { .name = "pid", .desc = "Parent ID" }, - [RSLV_STAT_SENT] = { .name = "sent", .desc = "Sent" }, - [RSLV_STAT_SND_ERROR] = { .name = "send_error", .desc = "Send error" }, - [RSLV_STAT_VALID] = { .name = "valid", .desc = "Valid" }, - [RSLV_STAT_UPDATE] = { .name = "update", .desc = "Update" }, - [RSLV_STAT_CNAME] = { .name = "cname", .desc = "CNAME" }, - [RSLV_STAT_CNAME_ERROR] = { .name = "cname_error", .desc = "CNAME error" }, - [RSLV_STAT_ANY_ERR] = { .name = "any_err", .desc = "Any errors" }, - [RSLV_STAT_NX] = { .name = "nx", .desc = "NX" }, - [RSLV_STAT_TIMEOUT] = { .name = "timeout", .desc = "Timeout" }, - [RSLV_STAT_REFUSED] = { .name = "refused", .desc = "Refused" }, - [RSLV_STAT_OTHER] = { .name = "other", .desc = "Other" }, - [RSLV_STAT_INVALID] = { .name = "invalid", .desc = "Invalid" }, - [RSLV_STAT_TOO_BIG] = { .name = "too_big", .desc = "Too big" }, - [RSLV_STAT_TRUNCATED] = { .name = "truncated", .desc = "Truncated" }, - [RSLV_STAT_OUTDATED] = { .name = "outdated", .desc = "Outdated" }, + [RSLV_STAT_ID] = { .name = "id", .desc = "ID" }, + [RSLV_STAT_PID] = { .name = "pid", .desc = "Parent ID" }, + [RSLV_STAT_SENT] = { .name = "sent", .desc = "Sent" }, + [RSLV_STAT_SND_ERROR] = { .name = "send_error", .desc = "Send error" }, + [RSLV_STAT_VALID] = { .name = "valid", .desc = "Valid" }, + [RSLV_STAT_UPDATE] = { .name = "update", .desc = "Update" }, + [RSLV_STAT_CNAME] = { .name = "cname", .desc = "CNAME" }, + [RSLV_STAT_CNAME_ERROR] = { .name = "cname_error", .desc = "CNAME error" }, + [RSLV_STAT_ANY_ERR] = { .name = "any_err", .desc = "Any errors" }, + [RSLV_STAT_NX] = { .name = "nx", .desc = "NX" }, + [RSLV_STAT_TIMEOUT] = { .name = "timeout", .desc = "Timeout" }, + [RSLV_STAT_REFUSED] = { .name = "refused", .desc = "Refused" }, + [RSLV_STAT_OTHER] = { .name = "other", .desc = "Other" }, + [RSLV_STAT_INVALID] = { .name = "invalid", .desc = "Invalid" }, + [RSLV_STAT_TOO_BIG] = { .name = "too_big", .desc = "Too big" }, + [RSLV_STAT_TRUNCATED] = { .name = "truncated", .desc = "Truncated" }, + [RSLV_STAT_TCP_FALLBACK] = { .name = "tcp_fallback", .desc = "TCP fallback" }, + [RSLV_STAT_OUTDATED] = { .name = "outdated", .desc = "Outdated" }, }; static struct dns_counters dns_counters; @@ -185,6 +187,9 @@ static int resolv_fill_stats(struct stats_module *mod, struct extra_counters *ct case RSLV_STAT_TRUNCATED: metric = mkf_u64(FN_GAUGE, counters->app.resolver.truncated); break; + case RSLV_STAT_TCP_FALLBACK: + metric = mkf_u64(FN_GAUGE, counters->app.resolver.tcp_fallback); + break; case RSLV_STAT_OUTDATED: metric = mkf_u64(FN_GAUGE, counters->app.resolver.outdated); break; @@ -454,9 +459,10 @@ static int resolv_send_query(struct resolv_resolution *resolution) int len; /* Update resolution */ - resolution->nb_queries = 0; - resolution->nb_responses = 0; - resolution->last_query = now_ms; + resolution->nb_queries = 0; + resolution->nb_responses = 0; + resolution->nb_tcp_fallback = 0; + resolution->last_query = now_ms; len = resolv_build_query(resolution->query_id, resolution->query_type, resolvers->accepted_payload_size, @@ -483,6 +489,34 @@ static int resolv_send_query(struct resolv_resolution *resolution) return 0; } +/* Retries query type over the stream server of nameserver . + * Returns 1 if the query was queued and 0 otherwise. + */ +static int resolv_send_tcp_fallback(struct resolv_resolution *resolution, + struct dns_nameserver *ns, int query_type) +{ + struct resolvers *resolvers = resolution->resolvers; + int len; + + len = resolv_build_query(resolution->query_id, query_type, + resolvers->accepted_payload_size, + resolution->hostname_dn, resolution->hostname_dn_len, + trash.area, trash.size); + if (len < 0) + return 0; + + if (dns_send_nameserver(ns, DNS_SERVER_STREAM, trash.area, len) < 0) + return 0; + + resolution->nb_queries++; + resolution->nb_tcp_fallback++; + resolution->last_query = now_ms; + LIST_DEL_INIT(&resolution->list); + LIST_APPEND(&resolvers->resolutions.curr, &resolution->list); + ns->counters->app.resolver.tcp_fallback++; + return 1; +} + /* Prepares and sends a DNS resolution. It returns 1 if the query was sent, 0 if * skipped and -1 if an error occurred. */ @@ -577,6 +611,7 @@ static void resolv_reset_resolution(struct resolv_resolution *resolution) resolution->last_resolution = now_ms; resolution->nb_queries = 0; resolution->nb_responses = 0; + resolution->nb_tcp_fallback = 0; resolution->query_type = resolution->prefered_query_type; /* clean up query id */ @@ -1009,12 +1044,14 @@ srv_found: * * The result is stored in ' response, buf_response, * response_query_records and response_answer_records members. + * A truncated SRV response may be parsed when is set. * * This function returns one of the RSLV_RESP_* code to indicate the type of * error found. */ static int resolv_validate_dns_response(unsigned char *resp, unsigned char *bufend, - struct resolv_resolution *resolution, int max_answer_records) + struct resolv_resolution *resolution, int max_answer_records, + int allow_truncated_srv) { unsigned char *reader; char *previous_dname, tmpname[DNS_MAX_NAME_SIZE]; @@ -1148,10 +1185,20 @@ static int resolv_validate_dns_response(unsigned char *resp, unsigned char *bufe query->class = reader[0] * 256 + reader[1]; reader += 2; + /* A and AAAA responses may arrive after the query type switched. */ + if (query->class != DNS_RCLASS_IN || + (query->type != resolution->query_type && + !((query->type == DNS_RTYPE_A || query->type == DNS_RTYPE_AAAA) && + (resolution->query_type == DNS_RTYPE_A || + resolution->query_type == DNS_RTYPE_AAAA)))) + goto invalid_resp; + /* TRUNCATED flag must be checked after we could read the query type * because a TRUNCATED SRV query type response can still be exploited + * when no TCP fallback can be performed. */ - if (query->type != DNS_RTYPE_SRV && flags & DNS_FLAG_TRUNCATED) { + if ((flags & DNS_FLAG_TRUNCATED) && + (query->type != DNS_RTYPE_SRV || !allow_truncated_srv)) { cause = RSLV_RESP_TRUNCATED; goto return_error; } @@ -2377,11 +2424,16 @@ static int resolv_process_responses(struct dns_nameserver *ns, enum dns_server_t /* known query id means a resolution in progress */ res = eb32_entry(eb, struct resolv_resolution, qid); + if (dns_server_type_is_stream(type) && + dns_nameserver_has_fallback(ns) && res->nb_tcp_fallback) + res->nb_tcp_fallback--; /* number of responses received */ res->nb_responses++; max_answer_records = (max_payload_size - DNS_HEADER_SIZE) / DNS_MIN_RECORD_SIZE; - dns_resp = resolv_validate_dns_response(buf, bufend, res, max_answer_records); + dns_resp = resolv_validate_dns_response(buf, bufend, res, max_answer_records, + !(dns_server_type_is_dgram(type) && + dns_nameserver_has_fallback(ns))); switch (dns_resp) { case RSLV_RESP_VALID: @@ -2417,6 +2469,11 @@ static int resolv_process_responses(struct dns_nameserver *ns, enum dns_server_t case RSLV_RESP_TRUNCATED: res->status = RSLV_STATUS_OTHER; ns->counters->app.resolver.truncated++; + if (dns_server_type_is_dgram(type) && + dns_nameserver_has_fallback(ns) && + resolv_send_tcp_fallback(res, ns, + res->response_query_records[0].type)) + continue; break; case RSLV_RESP_NO_EXPECTED_RECORD: @@ -2560,8 +2617,12 @@ struct task *process_resolvers(struct task *t, void *context, unsigned int state } else { /* Otherwise resend the DNS query and requeue the resolution */ - if (!res->nb_responses || res->prefered_query_type != res->query_type) { - /* No response received (a real timeout) or fallback already done */ + if (!res->nb_responses || res->nb_tcp_fallback || + res->prefered_query_type != res->query_type) { + /* No response was received, a TCP fallback response is + * still outstanding, or the query-type fallback was + * already performed. + */ res->query_type = res->prefered_query_type; res->try--; } @@ -3109,21 +3170,22 @@ static int cli_io_handler_dump_resolvers_to_buffer(struct appctx *appctx) list_for_each_entry_from(ns, &resolvers->nameservers, list) { chunk_reset(&trash); chunk_appendf(&trash, " nameserver %s:\n", ns->id); - chunk_appendf(&trash, " sent: %lld\n", ns->counters->sent); - chunk_appendf(&trash, " snd_error: %lld\n", ns->counters->snd_error); - chunk_appendf(&trash, " valid: %lld\n", ns->counters->app.resolver.valid); - chunk_appendf(&trash, " update: %lld\n", ns->counters->app.resolver.update); - chunk_appendf(&trash, " cname: %lld\n", ns->counters->app.resolver.cname); - chunk_appendf(&trash, " cname_error: %lld\n", ns->counters->app.resolver.cname_error); - chunk_appendf(&trash, " any_err: %lld\n", ns->counters->app.resolver.any_err); - chunk_appendf(&trash, " nx: %lld\n", ns->counters->app.resolver.nx); - chunk_appendf(&trash, " timeout: %lld\n", ns->counters->app.resolver.timeout); - chunk_appendf(&trash, " refused: %lld\n", ns->counters->app.resolver.refused); - chunk_appendf(&trash, " other: %lld\n", ns->counters->app.resolver.other); - chunk_appendf(&trash, " invalid: %lld\n", ns->counters->app.resolver.invalid); - chunk_appendf(&trash, " too_big: %lld\n", ns->counters->app.resolver.too_big); - chunk_appendf(&trash, " truncated: %lld\n", ns->counters->app.resolver.truncated); - chunk_appendf(&trash, " outdated: %lld\n", ns->counters->app.resolver.outdated); + chunk_appendf(&trash, " sent: %lld\n", ns->counters->sent); + chunk_appendf(&trash, " snd_error: %lld\n", ns->counters->snd_error); + chunk_appendf(&trash, " valid: %lld\n", ns->counters->app.resolver.valid); + chunk_appendf(&trash, " update: %lld\n", ns->counters->app.resolver.update); + chunk_appendf(&trash, " cname: %lld\n", ns->counters->app.resolver.cname); + chunk_appendf(&trash, " cname_error: %lld\n", ns->counters->app.resolver.cname_error); + chunk_appendf(&trash, " any_err: %lld\n", ns->counters->app.resolver.any_err); + chunk_appendf(&trash, " nx: %lld\n", ns->counters->app.resolver.nx); + chunk_appendf(&trash, " timeout: %lld\n", ns->counters->app.resolver.timeout); + chunk_appendf(&trash, " refused: %lld\n", ns->counters->app.resolver.refused); + chunk_appendf(&trash, " other: %lld\n", ns->counters->app.resolver.other); + chunk_appendf(&trash, " invalid: %lld\n", ns->counters->app.resolver.invalid); + chunk_appendf(&trash, " too_big: %lld\n", ns->counters->app.resolver.too_big); + chunk_appendf(&trash, " truncated: %lld\n", ns->counters->app.resolver.truncated); + chunk_appendf(&trash, " tcp_fallback: %lld\n", ns->counters->app.resolver.tcp_fallback); + chunk_appendf(&trash, " outdated: %lld\n", ns->counters->app.resolver.outdated); if (applet_putchk(appctx, &trash) == -1) goto full; ctx->ns = ns; @@ -3491,6 +3553,53 @@ void resolvers_setup_proxy(struct proxy *px) px->options2 |= PR_O2_INDEPSTR | PR_O2_SMARTCON; } +/* Adds the stream transport used for TCP fallback, named and targeting + * , to . Errors are stored in when it is non-NULL. Returns a + * combination of ERR_* flags. + */ +static int resolv_add_tcp_fallback(struct dns_nameserver *ns, const char *file, + int linenum, const char *name, + const struct sockaddr_storage *sk, + char **errmsg) +{ + char *address = NULL; + char *tcp_addr = NULL; + char *tcp_args[5] = { "nameserver", NULL, NULL, "", "" }; + int err_code = 0; + + address = sa2str(sk, get_host_port(sk), 0); + if (!address || !memprintf(&tcp_addr, "tcp@%s", address)) { + if (errmsg) + memprintf(errmsg, "parsing [%s:%d] : out of memory.", file, linenum); + err_code |= ERR_ALERT | ERR_ABORT; + goto out; + } + + tcp_args[1] = (char *)name; + tcp_args[2] = tcp_addr; + err_code |= parse_server(file, linenum, tcp_args, curr_resolvers->px, NULL, + SRV_PARSE_PARSE_ADDR|SRV_PARSE_INITIAL_RESOLVE); + if (err_code & (ERR_FATAL|ERR_ABORT)) { + if (errmsg && !*errmsg) + memprintf(errmsg, "parsing [%s:%d] : failed to initialize TCP fallback for nameserver '%s'.", + file, linenum, name); + err_code |= ERR_ABORT; + goto out; + } + + if (dns_stream_init(ns, proxy_last_server(curr_resolvers->px)) < 0) { + if (errmsg) + memprintf(errmsg, "parsing [%s:%d] : out of memory.", file, linenum); + err_code |= ERR_ALERT | ERR_ABORT; + goto out; + } + + out: + ha_free(&address); + ha_free(&tcp_addr); + return err_code; +} + static int parse_resolve_conf(char **errmsg, char **warnmsg) { struct dns_nameserver *newnameserver = NULL; @@ -3591,6 +3700,14 @@ static int parse_resolve_conf(char **errmsg, char **warnmsg) goto resolv_out; } + err_code |= resolv_add_tcp_fallback(newnameserver, "/etc/resolv.conf", + resolv_linenum, address, sk, errmsg); + if (err_code & (ERR_FATAL|ERR_ABORT)) { + dns_nameserver_deinit(newnameserver); + free(newnameserver); + goto resolv_out; + } + newnameserver->conf.file = strdup("/etc/resolv.conf"); if (newnameserver->conf.file == NULL) { if (errmsg) @@ -3757,7 +3874,9 @@ int cfg_parse_resolvers(const char *file, int linenum, char **args, int kwm) else if (strcmp(args[0], "nameserver") == 0) { /* nameserver definition */ struct dns_nameserver *newnameserver = NULL; struct sockaddr_storage *sk; + struct net_addr_type addr_type; int port1, port2; + int add_fallback = 0; struct protocol *proto; if (!*args[2]) { @@ -3785,7 +3904,7 @@ int cfg_parse_resolvers(const char *file, int linenum, char **args, int kwm) } } - sk = str2sa_range(args[2], NULL, &port1, &port2, NULL, &proto, NULL, + sk = str2sa_range(args[2], NULL, &port1, &port2, NULL, &proto, &addr_type, &errmsg, NULL, NULL, NULL, PA_O_RESOLVE | PA_O_PORT_OK | PA_O_PORT_MAND | PA_O_DGRAM | PA_O_STREAM | PA_O_DEFAULT_DGRAM); if (!sk) { @@ -3794,6 +3913,10 @@ int cfg_parse_resolvers(const char *file, int linenum, char **args, int kwm) goto out; } + add_fallback = (addr_type.xprt_type == PROTO_TYPE_DGRAM && + !addr_type.explicit_type && + (sk->ss_family == AF_INET || sk->ss_family == AF_INET6)); + if ((newnameserver = calloc(1, sizeof(*newnameserver))) == NULL) { ha_alert("parsing [%s:%d] : out of memory.\n", file, linenum); err_code |= ERR_ALERT | ERR_ABORT; @@ -3816,11 +3939,26 @@ int cfg_parse_resolvers(const char *file, int linenum, char **args, int kwm) goto out; } } - else if (dns_dgram_init(newnameserver, sk) < 0) { - ha_alert("parsing [%s:%d] : out of memory.\n", file, linenum); - err_code |= ERR_ALERT | ERR_ABORT; - free(newnameserver); - goto out; + else { + if (dns_dgram_init(newnameserver, sk) < 0) { + ha_alert("parsing [%s:%d] : out of memory.\n", file, linenum); + err_code |= ERR_ALERT | ERR_ABORT; + free(newnameserver); + goto out; + } + + if (add_fallback) { + err_code |= resolv_add_tcp_fallback(newnameserver, file, + linenum, args[1], sk, &errmsg); + if (err_code & (ERR_FATAL|ERR_ABORT)) { + if (errmsg) + ha_alert("%s\n", errmsg); + err_code |= ERR_ABORT; + dns_nameserver_deinit(newnameserver); + free(newnameserver); + goto out; + } + } } if ((newnameserver->conf.file = strdup(file)) == NULL) {