Fix lossy KNN GiST when ordering operator returns non-float8 value.

KNN GiST with recheck flag should return to executor the same type as ordering
operator, GiST detects this type by looking to return type of function which
implements ordering operator. But occasionally detecting code works after
replacing ordering operator function to distance support function.
Distance support function always returns float8, so, detecting code get float8
instead of actual return type of ordering operator.

Built-in opclasses don't have ordering operator which doesn't return
non-float8 value, so, tests are impossible here, at least now.

Backpatch to 9.5 where lozzy KNN was introduced.

Author: Alexander Korotkov
Report by: Artur Zakirov

1 files changed, 10 insertions, 2 deletions

diff --git a/src/backend/access/gist/gistscan.c b/src/backend/access/gist/gistscan.c
index ad392948756..ee3289ae8ea 100644
--- a/src/backend/access/gist/gistscan.c
+++ b/src/backend/access/gist/gistscan.c
@@ -230,6 +230,10 @@ gistrescan(PG_FUNCTION_ARGS)
 		{
 			ScanKey		skey = scan->keyData + i;
 
+			/*
+			 * Copy consistent support function to ScanKey structure
+			 * instead of function implementing filtering operator.
+			 */
 			fmgr_info_copy(&(skey->sk_func),
 						   &(so->giststate->consistentFn[skey->sk_attno - 1]),
 						   so->giststate->scanCxt);
@@ -285,8 +289,6 @@ gistrescan(PG_FUNCTION_ARGS)
 					 GIST_DISTANCE_PROC, skey->sk_attno,
 					 RelationGetRelationName(scan->indexRelation));
 
-			fmgr_info_copy(&(skey->sk_func), finfo, so->giststate->scanCxt);
-
 			/*
 			 * Look up the datatype returned by the original ordering
 			 * operator. GiST always uses a float8 for the distance function,
@@ -301,6 +303,12 @@ gistrescan(PG_FUNCTION_ARGS)
 			 */
 			so->orderByTypes[i] = get_func_rettype(skey->sk_func.fn_oid);
 
+			/*
+			 * Copy distance support function to ScanKey structure
+			 * instead of function implementing ordering operator.
+			 */
+			fmgr_info_copy(&(skey->sk_func), finfo, so->giststate->scanCxt);
+
 			/* Restore prior fn_extra pointers, if not first time */
 			if (!first_time)
 				skey->sk_func.fn_extra = fn_extras[i];