diff options
author | Tom Lane <tgl@sss.pgh.pa.us> | 2020-06-13 13:43:24 -0400 |
---|---|---|
committer | Tom Lane <tgl@sss.pgh.pa.us> | 2020-06-13 13:43:24 -0400 |
commit | 874372a941a6ee31ba4034db0367e0929e5f49bc (patch) | |
tree | edc8d95f6455518da6ae061821968b0696fb85d0 /src/backend/access/gist/gistbuildbuffers.c | |
parent | 81cd796b825cf8d09bb6acea75482f9583e6a951 (diff) | |
download | postgresql-874372a941a6ee31ba4034db0367e0929e5f49bc.tar.gz postgresql-874372a941a6ee31ba4034db0367e0929e5f49bc.zip |
Fix behavior of float aggregates for single Inf or NaN inputs.
When there is just one non-null input value, and it is infinity or NaN,
aggregates such as stddev_pop and covar_pop should produce a NaN
result, because the calculation is not well-defined. They used to do
so, but since we adopted Youngs-Cramer aggregation in commit e954a727f,
they produced zero instead. That's an oversight, so fix it. Add tests
exercising these edge cases.
Affected aggregates are
var_pop(double precision)
stddev_pop(double precision)
var_pop(real)
stddev_pop(real)
regr_sxx(double precision,double precision)
regr_syy(double precision,double precision)
regr_sxy(double precision,double precision)
regr_r2(double precision,double precision)
regr_slope(double precision,double precision)
regr_intercept(double precision,double precision)
covar_pop(double precision,double precision)
corr(double precision,double precision)
Back-patch to v12 where the behavior change was accidentally introduced.
Report and patch by me; thanks to Dean Rasheed for review.
Discussion: https://postgr.es/m/353062.1591898766@sss.pgh.pa.us
Diffstat (limited to 'src/backend/access/gist/gistbuildbuffers.c')
0 files changed, 0 insertions, 0 deletions