index BRIN BRIN stands for Block Range Index. BRIN is designed for handling very large tables in which certain columns have some natural correlation with their physical location within the table. BRIN works in terms of block ranges (or page ranges). A block range is a group of pages that are physically adjacent in the table; for each block range, some summary info is stored by the index. For example, a table storing a store's sale orders might have a date column on which each order was placed, and most of the time the entries for earlier orders will appear earlier in the table as well; a table storing a ZIP code column might have all codes for a city grouped together naturally. BRIN indexes can satisfy queries via regular bitmap index scans, and will return all tuples in all pages within each range if the summary info stored by the index is consistent with the query conditions. The query executor is in charge of rechecking these tuples and discarding those that do not match the query conditions — in other words, these indexes are lossy. Because a BRIN index is very small, scanning the index adds little overhead compared to a sequential scan, but may avoid scanning large parts of the table that are known not to contain matching tuples. The specific data that a BRIN index will store, as well as the specific queries that the index will be able to satisfy, depend on the operator class selected for each column of the index. Data types having a linear sort order can have operator classes that store the minimum and maximum value within each block range, for instance; geometrical types might store the bounding box for all the objects in the block range. The size of the block range is determined at index creation time by the pages_per_range storage parameter. The number of index entries will be equal to the size of the relation in pages divided by the selected value for pages_per_range. Therefore, the smaller the number, the larger the index becomes (because of the need to store more index entries), but at the same time the summary data stored can be more precise and more data blocks can be skipped during an index scan. At the time of creation, all existing heap pages are scanned and a summary index tuple is created for each range, including the possibly-incomplete range at the end. As new pages are filled with data, page ranges that are already summarized will cause the summary information to be updated with data from the new tuples. When a new page is created that does not fall within the last summarized range, the range that the new page belongs into does not automatically acquire a summary tuple; those tuples remain unsummarized until a summarization run is invoked later, creating the initial summary for that range. There are several ways to trigger the initial summarization of a page range. If the table is vacuumed, either manually or by autovacuum, all existing unsummarized page ranges are summarized. Also, if the index's parameter is enabled, which it isn't by default, whenever autovacuum runs in that database, summarization will occur for all unsummarized page ranges that have been filled, regardless of whether the table itself is processed by autovacuum; see below. Lastly, the following functions can be used: brin_summarize_new_values(regclass) which summarizes all unsummarized ranges; brin_summarize_range(regclass, bigint) which summarizes only the range containing the given page, if it is unsummarized. When autosummarization is enabled, a request is sent to autovacuum to execute a targeted summarization for a block range when an insertion is detected for the first item of the first page of the next block range, to be fulfilled the next time an autovacuum worker finishes running in the same database. If the request queue is full, the request is not recorded and a message is sent to the server log: LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was not recorded When this happens, the range will remain unsummarized until the next regular vacuum run on the table, or one of the functions mentioned above are invoked. Conversely, a range can be de-summarized using the brin_desummarize_range(regclass, bigint) function, which is useful when the index tuple is no longer a very good representation because the existing values have changed. See for details. The core PostgreSQL distribution includes the BRIN operator classes shown in . The minmax operator classes store the minimum and the maximum values appearing in the indexed column within the range. The inclusion operator classes store a value which includes the values in the indexed column within the range. Name Indexed Data Type Indexable Operators int8_minmax_ops bigint < <= = >= > bit_minmax_ops bit < <= = >= > varbit_minmax_ops bit varying < <= = >= > box_inclusion_ops box << &< && &> >> ~= @> <@ &<| <<| |>> |&> bytea_minmax_ops bytea < <= = >= > bpchar_minmax_ops character < <= = >= > char_minmax_ops "char" < <= = >= > date_minmax_ops date < <= = >= > float8_minmax_ops double precision < <= = >= > inet_minmax_ops inet < <= = >= > network_inclusion_ops inet && >>= <<= = >> << int4_minmax_ops integer < <= = >= > interval_minmax_ops interval < <= = >= > macaddr_minmax_ops macaddr < <= = >= > macaddr8_minmax_ops macaddr8 < <= = >= > name_minmax_ops name < <= = >= > numeric_minmax_ops numeric < <= = >= > pg_lsn_minmax_ops pg_lsn < <= = >= > oid_minmax_ops oid < <= = >= > range_inclusion_ops any range type << &< && &> >> @> <@ -|- = < <= = > >= float4_minmax_ops real < <= = >= > int2_minmax_ops smallint < <= = >= > text_minmax_ops text < <= = >= > tid_minmax_ops tid < <= = >= > timestamp_minmax_ops timestamp without time zone < <= = >= > timestamptz_minmax_ops timestamp with time zone < <= = >= > time_minmax_ops time without time zone < <= = >= > timetz_minmax_ops time with time zone < <= = >= > uuid_minmax_ops uuid < <= = >= > The BRIN interface has a high level of abstraction, requiring the access method implementer only to implement the semantics of the data type being accessed. The BRIN layer itself takes care of concurrency, logging and searching the index structure. All it takes to get a BRIN access method working is to implement a few user-defined methods, which define the behavior of summary values stored in the index and the way they interact with scan keys. In short, BRIN combines extensibility with generality, code reuse, and a clean interface. There are four methods that an operator class for BRIN must provide: BrinOpcInfo *opcInfo(Oid type_oid) Returns internal information about the indexed columns' summary data. The return value must point to a palloc'd BrinOpcInfo, which has this definition: typedef struct BrinOpcInfo { /* Number of columns stored in an index column of this opclass */ uint16 oi_nstored; /* Opaque pointer for the opclass' private use */ void *oi_opaque; /* Type cache entries of the stored columns */ TypeCacheEntry *oi_typcache[FLEXIBLE_ARRAY_MEMBER]; } BrinOpcInfo; BrinOpcInfo.oi_opaque can be used by the operator class routines to pass information between support functions during an index scan. bool consistent(BrinDesc *bdesc, BrinValues *column, ScanKey key) Returns whether the ScanKey is consistent with the given indexed values for a range. The attribute number to use is passed as part of the scan key. bool addValue(BrinDesc *bdesc, BrinValues *column, Datum newval, bool isnull) Given an index tuple and an indexed value, modifies the indicated attribute of the tuple so that it additionally represents the new value. If any modification was done to the tuple, true is returned. bool unionTuples(BrinDesc *bdesc, BrinValues *a, BrinValues *b) Consolidates two index tuples. Given two index tuples, modifies the indicated attribute of the first of them so that it represents both tuples. The second tuple is not modified. The core distribution includes support for two types of operator classes: minmax and inclusion. Operator class definitions using them are shipped for in-core data types as appropriate. Additional operator classes can be defined by the user for other data types using equivalent definitions, without having to write any source code; appropriate catalog entries being declared is enough. Note that assumptions about the semantics of operator strategies are embedded in the support functions' source code. Operator classes that implement completely different semantics are also possible, provided implementations of the four main support functions described above are written. Note that backwards compatibility across major releases is not guaranteed: for example, additional support functions might be required in later releases. To write an operator class for a data type that implements a totally ordered set, it is possible to use the minmax support functions alongside the corresponding operators, as shown in . All operator class members (functions and operators) are mandatory. Operator class member Object Support Function 1 internal function brin_minmax_opcinfo() Support Function 2 internal function brin_minmax_add_value() Support Function 3 internal function brin_minmax_consistent() Support Function 4 internal function brin_minmax_union() Operator Strategy 1 operator less-than Operator Strategy 2 operator less-than-or-equal-to Operator Strategy 3 operator equal-to Operator Strategy 4 operator greater-than-or-equal-to Operator Strategy 5 operator greater-than To write an operator class for a complex data type which has values included within another type, it's possible to use the inclusion support functions alongside the corresponding operators, as shown in . It requires only a single additional function, which can be written in any language. More functions can be defined for additional functionality. All operators are optional. Some operators require other operators, as shown as dependencies on the table. Operator class member Object Dependency Support Function 1 internal function brin_inclusion_opcinfo() Support Function 2 internal function brin_inclusion_add_value() Support Function 3 internal function brin_inclusion_consistent() Support Function 4 internal function brin_inclusion_union() Support Function 11 function to merge two elements Support Function 12 optional function to check whether two elements are mergeable Support Function 13 optional function to check if an element is contained within another Support Function 14 optional function to check whether an element is empty Operator Strategy 1 operator left-of Operator Strategy 4 Operator Strategy 2 operator does-not-extend-to-the-right-of Operator Strategy 5 Operator Strategy 3 operator overlaps Operator Strategy 4 operator does-not-extend-to-the-left-of Operator Strategy 1 Operator Strategy 5 operator right-of Operator Strategy 2 Operator Strategy 6, 18 operator same-as-or-equal-to Operator Strategy 7 Operator Strategy 7, 13, 16, 24, 25 operator contains-or-equal-to Operator Strategy 8, 14, 26, 27 operator is-contained-by-or-equal-to Operator Strategy 3 Operator Strategy 9 operator does-not-extend-above Operator Strategy 11 Operator Strategy 10 operator is-below Operator Strategy 12 Operator Strategy 11 operator is-above Operator Strategy 9 Operator Strategy 12 operator does-not-extend-below Operator Strategy 10 Operator Strategy 20 operator less-than Operator Strategy 5 Operator Strategy 21 operator less-than-or-equal-to Operator Strategy 5 Operator Strategy 22 operator greater-than Operator Strategy 1 Operator Strategy 23 operator greater-than-or-equal-to Operator Strategy 1 Support function numbers 1-10 are reserved for the BRIN internal functions, so the SQL level functions start with number 11. Support function number 11 is the main function required to build the index. It should accept two arguments with the same data type as the operator class, and return the union of them. The inclusion operator class can store union values with different data types if it is defined with the STORAGE parameter. The return value of the union function should match the STORAGE data type. Support function numbers 12 and 14 are provided to support irregularities of built-in data types. Function number 12 is used to support network addresses from different families which are not mergeable. Function number 14 is used to support empty ranges. Function number 13 is an optional but recommended one, which allows the new value to be checked before it is passed to the union function. As the BRIN framework can shortcut some operations when the union is not changed, using this function can improve index performance. Both minmax and inclusion operator classes support cross-data-type operators, though with these the dependencies become more complicated. The minmax operator class requires a full set of operators to be defined with both arguments having the same data type. It allows additional data types to be supported by defining extra sets of operators. Inclusion operator class operator strategies are dependent on another operator strategy as shown in , or the same operator strategy as themselves. They require the dependency operator to be defined with the STORAGE data type as the left-hand-side argument and the other supported data type to be the right-hand-side argument of the supported operator. See float4_minmax_ops as an example of minmax, and box_inclusion_ops as an example of inclusion.