Fix recently-exposed portability issue in regex optimization.

fixempties() counts the number of in-arcs in the regex NFA and then
allocates an array of that many arc pointers.  If the NFA contains no
arcs, this amounts to malloc(0) for which some platforms return NULL.
The code mistakenly treats that as indicating out-of-memory.  Thus,
we can get a bogus "out of memory" failure for some unsatisfiable
regexes.

This happens only in v15 and earlier, since bea3d7e38 switched to
using palloc() rather than bare malloc().  And at least of the
platforms in the buildfarm, only AIX seems to return NULL.  So the
impact is pretty narrow.  But I don't especially want to ship code
that is failing its own regression tests, so let's fix this for
this week's releases.

A quick code survey says that there is only the one place in
src/backend/regex/ that is at risk of doing malloc(0), so we'll just
band-aid that place.  A more future-proof fix could be to install a
malloc() wrapper similar to pg_malloc().  But this code seems unlikely
to change much more in the affected branches, so that's probably
overkill.

The only known test case for this involves a complemented character
class in a bracket expression, for example [^\s\S], and we didn't
support that in v13.  So it may be that the problem is unreachable
in v13.  But I'm not 100% sure of that, so patch v13 too.

Discussion: https://postgr.es/m/661fd81b-f069-8f30-1a41-e195c57449b4@gmail.com

1 files changed, 4 insertions, 1 deletions

diff --git a/src/backend/regex/regc_nfa.c b/src/backend/regex/regc_nfa.c
index 06bbcae0ee3..4b19740788e 100644
--- a/src/backend/regex/regc_nfa.c
+++ b/src/backend/regex/regc_nfa.c
@@ -2229,9 +2229,12 @@ fixempties(struct nfa *nfa,
 	 * current target state.  totalinarcs is probably a considerable
 	 * overestimate of the space needed, but the NFA is unlikely to be large
 	 * enough at this point to make it worth being smarter.
+	 *
+	 * Note: totalinarcs could be zero, and some machines return NULL for
+	 * malloc(0).  Don't throw an error if so.
 	 */
 	arcarray = (struct arc **) MALLOC(totalinarcs * sizeof(struct arc *));
-	if (arcarray == NULL)
+	if (arcarray == NULL && totalinarcs != 0)
 	{
 		NERR(REG_ESPACE);
 		FREE(inarcsorig);