aboutsummaryrefslogtreecommitdiff
path: root/src/backend
Commit message (Collapse)AuthorAge
* Improve memory handling across SQL-callable backup functionsMichael Paquier2022-10-22
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Since pg_backup_start() and pg_backup_stop() exist, the tablespace map data and the backup state data (backup_label string until 7d70809) have been allocated in the TopMemoryContext. This approach would cause memory leaks in the session calling these functions if failures happen before pg_backup_stop() ends, leaking more memory on repeated failures. Both things need little memory so that would not be really noticeable for most users, except perhaps connection poolers with long-lived connections able to trigger backup failures with these functions. This commit improves the logic in this area by not allocating anymore the backup-related data that needs to travel across the SQL-callable backup functions in TopMemoryContext, by using instead a dedicated memory context child of TopMemoryContext. The memory context is created in pg_backup_start() and deleted when finishing pg_backup_stop(). In the event of an in-flight failure, this memory context gets reset in the follow-up pg_backup_start() call, so as we are sure that only one run worth of data is leaked at any time. Some cleanup was already done for the backup data on a follow-up call of pg_backup_start(), but using a memory context makes the whole simpler. BASE_BACKUP commands are executed in isolation, relying on the memory context created for replication commands, hence these do not need such an extra logic. Author: Bharath Rupireddy Reviewed-by: Robert Haas, Alvaro Herrera, Cary Huang, Michael Paquier Discussion: https://postgr.es/m/CALj2ACXqvfKF2B0beQ=aJMdWnpNohmBPsRg=EDQj_6y1t2O8mQ@mail.gmail.com
* Add CHECK_FOR_INTERRUPTS while restoring changes during decoding.Amit Kapila2022-10-21
| | | | | | | | | | | Previously in commit 42681dffaf, we added CFI during decoding changes but missed another similar case that can happen while restoring changes spilled to disk back into memory in a loop. Reported-by: Robert Haas Author: Amit Kapila Backpatch-through: 10 Discussion: https://postgr.es/m/CA+TgmoaLObg0QbstbC8ykDwOdD1bDkr4AbPpB=0DPgA2JW0mFg@mail.gmail.com
* Refactor more logic for compilation of regular expressions in hba.cMichael Paquier2022-10-21
| | | | | | | | | | | | | | | | | | | | It happens that the parts of hba.conf that are planned to be extended to support regular expressions would finish by using the same error message as the one used currently for pg_ident.conf when a regular expression cannot be compiled, as long as the routine centralizing the logic, regcomp_auth_token(), knows from which file the regexp comes from and its line location in the so-said file. This change makes the follow-up patches slightly simpler, and the logic remains the same. I suspect that this makes the proposal to add support for file inclusions in pg_ident.conf and pg_hba.conf slightly simpler, as well. Extracted from a larger patch by the same author. This is similar to the refactoring done in fc579e1. Author: Bertrand Drouvot Discussion: https://postgr.es/m/fff0d7c1-8ad4-76a1-9db3-0ab6ec338bf7@amazon.com
* Improve the accuracy of numeric power() for integer exponents.Dean Rasheed2022-10-20
| | | | | | | | | | | | | | | | | | | | | | | | | | This makes the choice of result scale of numeric power() for integer exponents consistent with the choice for non-integer exponents, and with the result scale of other numeric functions. Specifically, the result scale will be at least as large as the scale of either input, and sufficient to ensure that the result has at least 16 significant digits. Formerly, the result scale was based only on the scale of the first input, without taking into account the weight of the result. For results with negative weight, that could lead to results with very few or even no non-zero significant digits (e.g., 10.0 ^ (-18) produced 0.0000000000000000). Fix this by moving responsibility for the choice of result scale into power_var_int(), which already has code to estimate the result weight. Per report by Adrian Klaver and suggested fix by Tom Lane. No back-patch -- arguably this is a bug fix, but one which is easy to work around, so it doesn't seem worth the risk of changing query results in stable branches. Discussion: https://postgr.es/m/12a40226-70ac-3a3b-3d3a-fdaf9e32d312%40aklaver.com
* Use proper macro to access TransactionIdAlvaro Herrera2022-10-20
| | | | | | | | | | In commit f10a025cfe97 I mistakenly used list_member_oid in a place where list_member_xid is called for. (Currently innocuous as both typedefs are pretty much identical, but if we change either, it'll become broken.) Repair. Author: Hou Zhijie <houzj.fnst@fujitsu.com> Discussion: https://postgr.es/m/OS0PR01MB5716E2399494D4CB1A28A091942A9@OS0PR01MB5716.jpnprd01.prod.outlook.com
* Fix assertion failures while processing NEW_CID record in logical decoding.Amit Kapila2022-10-20
| | | | | | | | | | | | | | | | | | | | | | | | | | | When the logical decoding restarts from NEW_CID, since there is no association between the top transaction and its subtransaction, both are created as top transactions and have the same LSN. This caused the assertion failure in AssertTXNLsnOrder(). This patch skips the assertion check until we reach the LSN at which we start decoding the contents of the transaction, specifically start_decoding_at LSN in SnapBuild. This is okay because we don't guarantee to make the association between top transaction and subtransaction until we try to decode the actual contents of transaction. The ordering of the records prior to the start_decoding_at LSN should have been checked before the restart. The other assertion failure is due to the reason that we forgot to track that we have considered top-level transaction id in the list of catalog changing transactions that were committed when one of its subtransactions is marked as containing catalog change. Reported-by: Tomas Vondra, Osumi Takamichi Author: Masahiko Sawada, Kuroda Hayato Reviewed-by: Amit Kapila, Dilip Kumar, Kuroda Hayato, Kyotaro Horiguchi, Masahiko Sawada Backpatch-through: 10 Discussion: https://postgr.es/m/a89b46b6-0239-2fd5-71a9-b19b1f7a7145%40enterprisedb.com Discussion: https://postgr.es/m/TYCPR01MB83733C6CEAE47D0280814D5AED7A9%40TYCPR01MB8373.jpnprd01.prod.outlook.com
* Get rid of XLogCtlInsert->forcePageWritesAlvaro Herrera2022-10-19
| | | | | | | | | After commit 39969e2a1e4d, ->forcePageWrites is no longer very interesting: we can just test whether runningBackups is different from 0. This simplifies some code, so do away with it. Reviewed-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> Discussion: https://postgr.es/m/39969e2a1e4d7f5a37f3ef37d53bbfe171e7d77a
* Track LLVM 15 changes.Thomas Munro2022-10-19
| | | | | | | | | | | | Per https://llvm.org/docs/OpaquePointers.html, support for non-opaque pointers still exists and we can request that on our context. We have until LLVM 16 to move to opaque pointers, a much larger change. Back-patch to 11, where LLVM support arrived. Author: Thomas Munro <thomas.munro@gmail.com> Author: Andres Freund <andres@anarazel.de> Discussion: https://postgr.es/m/CAMHz58Sf_xncdyqsekoVsNeKcruKootLtVH6cYXVhhUR1oKPCg%40mail.gmail.com
* Remove pg_backup_start_callback and reuse similar codeAlvaro Herrera2022-10-19
| | | | | | | | | | | | | | | | | | | | | We had two copies of almost identical logic to revert shared memory state when a running backup aborts; we can remove pg_backup_start_callback if we adapt do_pg_abort_backup so that it can be used for this purpose too. However, in order for this to work, we have to repurpose the flag passed to do_pg_abort_backup. It used to indicate whether to throw a warning (and the only caller always passed true). It now indicates whether the callback is being called at start time (in which case the session backup state is known not to have been set to RUNNING yet, so action is always taken) or shmem time (in which case action is only taken if the session backup state is RUNNING). Thus the meaning of the flag is no longer superfluous, but it's actually quite critical to get right. I (Álvaro) chose to change the polarity and the code flow re. the flag from what Bharath submitted, for coding clarity. Co-authored-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> Discussion: https://www.postgresql.org/message-id/20221013111330.564fk5tkwe3ha77l%40alvherre.pgsql
* Rework shutdown callback of archiver modulesMichael Paquier2022-10-19
| | | | | | | | | | | | | | | | | | | | | | | As currently designed, with a callback registered in a ERROR_CLEANUP block, the shutdown callback would get called twice when updating archive_library on SIGHUP, which is something that we want to avoid to ease the life of extension writers. Anyway, an ERROR in the archiver process is treated as a FATAL, stopping it immediately, hence there is no need for a ERROR_CLEANUP block. Instead of that, the shutdown callback is not called upon before_shmem_exit(), giving to the modules the opportunity to do any cleanup actions before the server shuts down its subsystems. While on it, this commit adds some testing coverage for the shutdown callback. Neither shell_archive nor basic_archive have been using it, and one is added to shell_archive, whose trigger is checked in a TAP test through a shutdown sequence. Author: Nathan Bossart, Bharath Rupireddy Reviewed-by: Kyotaro Horiguchi, Michael Paquier Discussion: https://postgr.es/m/20221015221328.GB1821022@nathanxps13 Backpatch-through: 15
* Fix typos in logical/launcher.cMichael Paquier2022-10-19
| | | | | Author: Peter Smith Discussion: https://postgr.es/m/CAHut+Pvbma5HCc7==-B1ycyLQVyu7Fqq-qV=jhC5Zx4pWqk3uw@mail.gmail.com
* Refactor regular expression handling in hba.cMichael Paquier2022-10-19
| | | | | | | | | | | | | | | | | | | | | | | | | | | AuthToken gains a regular expression, and IdentLine is changed so as it uses an AuthToken rather than tracking separately the ident user string used for the regex compilation and its generated regex_t. In the case of pg_ident.conf, a set of AuthTokens is built in the pre-parsing phase of the file, and an extra regular expression is compiled when building the list of IdentLines, after checking the sanity of the fields in a pre-parsed entry. The logic in charge of computing and executing regular expressions is now done in a new set of routines called respectively regcomp_auth_token() and regexec_auth_token() that are wrappers around pg_regcomp() and pg_regexec(), working on AuthTokens. While on it, this patch adds a routine able to free an AuthToken, free_auth_token(), to simplify a bit the logic around the requirement of using a specific free routine for computed regular expressions. Note that there are no functional or behavior changes introduced by this commit. The goal of this patch is to ease the use of regular expressions with more items of pg_hba.conf (user list, database list, potentially hostnames) where AuthTokens are used extensively. This will be tackled later in a separate patch. Author: Bertrand Drouvot, Michael Paquier Discussion: https://postgr.es/m/fff0d7c1-8ad4-76a1-9db3-0ab6ec338bf7@amazon.com
* Fix confusion about havingQual vs hasHavingQual in planner.Tom Lane2022-10-18
| | | | | | | | | | | | | | | | | | | | | | | Preprocessing of the HAVING clause will reduce havingQual to NIL if the clause is constant-TRUE. This is one case where that convention is rather unfortunate, because "HAVING TRUE" is not at all the same as not having any HAVING clause at all. (Per the SQL spec, it still forces the query to be grouped.) The planner deals with this by having a boolean hasHavingQual that records whether havingQual was originally nonempty; places that just want to check whether HAVING was specified are supposed to consult that. I found three places that got that wrong. Fortunately, these could only affect cost estimates not correctness. It'd be hard even to demonstrate the errors; for example, the one in allpaths.c would only matter in a query that has HAVING TRUE but no GROUP BY and no aggregates, which would require a completely variable-free SELECT list, making the case probably of only academic interest. Hence, while these are worth fixing before someone copies the incorrect coding somewhere more critical, they don't seem worth back-patching. I didn't bother trying to devise regression tests, either. Discussion: https://postgr.es/m/2503888.1666042643@sss.pgh.pa.us
* Improve errhint for ALTER SUBSCRIPTION ADD/DROP PUBLICATIONAlvaro Herrera2022-10-18
| | | | | | | | | | | | | | | The original hint says to use SET PUBLICATION when really ADD/DROP PUBLICATION is called for, so this is arguably a bug fix. Also, a very similar message elsewhere was using an inconsistent SQLSTATE. While at it, unwrap some strings. Backpatch to 15. Author: Hou zj <houzj.fnst@fujitsu.com> Discussion: https://postgr.es/m/OS0PR01MB57160AD0E7386547BA978EB394299@OS0PR01MB5716.jpnprd01.prod.outlook.com
* Remove compatibility declarations for InitMaterializedSRF()Michael Paquier2022-10-18
| | | | | | | | | | These routines have been renamed in a19e5ce. There is no need to keep the compatibility declarations on HEAD, as once an extension moves to the new routine name when compiling with v16~ the code would work the same way when recompiled on v15. No backpatch to v15 for this one, because ABI compatibility has to be maintained there. Discussion: https://postgr.es/m/20221013194820.ciktb2sbbpw7cljm@awork3.anarazel.de
* Rename SetSingleFuncCall() to InitMaterializedSRF()Michael Paquier2022-10-18
| | | | | | | | | | | | | | | | | | Per discussion, the existing routine name able to initialize a SRF function with materialize mode is unpopular, so rename it. Equally, the flags of this function are renamed, as of: - SRF_SINGLE_USE_EXPECTED -> MAT_SRF_USE_EXPECTED_DESC - SRF_SINGLE_BLESS -> MAT_SRF_BLESS The previous function and flags introduced in 9e98583 are kept around for compatibility purposes, so as any extension code already compiled with v15 continues to work as-is. The declarations introduced here for compatibility will be removed from HEAD in a follow-up commit. The new names have been suggested by Andres Freund and Melanie Plageman. Discussion: https://postgr.es/m/20221013194820.ciktb2sbbpw7cljm@awork3.anarazel.de Backpatch-through: 15
* Record dependencies of a cast on other casts that it requires.Tom Lane2022-10-17
| | | | | | | | | | | | | | | | | | | | | | | | | | | When creating a cast that uses a conversion function, we've historically allowed the input and result types to be binary-compatible with the function's input and result types, rather than necessarily being identical. This means that the new cast is logically dependent on the binary-compatible cast or casts that it references: if those are defined by pg_cast entries, and you try to restore the new cast without having defined them, it'll fail. Hence, we should make pg_depend entries to record these dependencies so that pg_dump knows that there is an ordering requirement. This is not the only place where we allow such shortcuts; aggregate functions for example are similarly lax, and in principle should gain similar dependencies. However, for now it seems sufficient to fix the cast-versus-cast case, as pg_dump's other ordering heuristics should keep it out of trouble for other object types. Per report from David Turoň; thanks also to Robert Haas for preliminary investigation. I considered back-patching, but seeing that this issue has existed for many years without previous reports, it's not clear it's worth the trouble. Moreover, back-patching wouldn't be enough to ensure that the new pg_depend entries exist in existing databases anyway. Discussion: https://postgr.es/m/OF0A160F3E.578B15D1-ONC12588DA.003E4857-C12588DA.0045A428@notes.linuxbox.cz
* Reject non-ON-SELECT rules that are named "_RETURN".Tom Lane2022-10-17
| | | | | | | | | | | | DefineQueryRewrite() has long required that ON SELECT rules be named "_RETURN". But we overlooked the converse case: we should forbid non-ON-SELECT rules that are named "_RETURN". In particular this prevents using CREATE OR REPLACE RULE to overwrite a view's _RETURN rule with some other kind of rule, thereby breaking the view. Per bug #17646 from Kui Liu. Back-patch to all supported branches. Discussion: https://postgr.es/m/17646-70c93cfa40365776@postgresql.org
* Guard against table-AM-less relations in planner.Tom Lane2022-10-17
| | | | | | | | | | | | | | | The executor will dump core if it's asked to execute a seqscan on a relation having no table AM, such as a view. While that shouldn't really happen, it's possible to get there via catalog corruption, such as a missing ON SELECT rule. It seems worth installing a defense against that. There are multiple plausible places for such a defense, but I picked the planner's get_relation_info(). Per discussion of bug #17646 from Kui Liu. Back-patch to v12 where the tableam APIs were introduced; in older versions you won't get a SIGSEGV, so it seems less pressing. Discussion: https://postgr.es/m/17646-70c93cfa40365776@postgresql.org
* Fix EXPLAIN of SEARCH BREADTH FIRST with a constant initial value.Tom Lane2022-10-16
| | | | | | | | | | | | | | | | | | | | | | If the non-recursive term of a SEARCH BREADTH FIRST recursive query has only constants in its target list, the planner will fold the starting RowExpr added by rewrite into a simple Const of type RECORD. The executor doesn't have any problem with that --- but EXPLAIN VERBOSE will encounter the Const as the ultimate source of truth about what the field names of the SET column are, and it didn't know what to do with that. Fortunately, we can pull the identifying typmod out of the Const, in much the same way that record_out would. For reasons that remain a bit obscure to me, this only fails with SEARCH BREADTH FIRST, not SEARCH DEPTH FIRST or CYCLE. But I added regression test cases for both of those options too, just to make sure we don't break it in future. Per bug #17644 from Matthijs van der Vleuten. Back-patch to v14 where these constructs were added. Discussion: https://postgr.es/m/17644-3bd1f3036d6d7a16@postgresql.org
* Rename parser token REF to REF_P to avoid a symbol conflict.Tom Lane2022-10-16
| | | | | | | | | | | | | | | | | | | | | | | In the latest version of Apple's macOS SDK, <sys/socket.h> fails to compile if "REF" is #define'd as something. Apple may or may not agree that this is a bug, and even if they do accept the bug report I filed, they probably won't fix it very quickly. In the meantime, our back branches will all fail to compile gram.y. v15 and HEAD currently escape the problem thanks to the refactoring done in 98e93a1fc, but that's purely accidental. Moreover, since that patch removed a widely-visible inclusion of <netdb.h>, back-patching it seems too likely to break third-party code. Instead, change the token's code name to REF_P, following our usual convention for naming parser tokens that are likely to have symbol conflicts. The effects of that should be localized to the grammar and immediately surrounding files, so it seems like a safer answer. Per project policy that we want to keep recently-out-of-support branches buildable on modern systems, back-patch all the way to 9.2. Discussion: https://postgr.es/m/1803927.1665938411@sss.pgh.pa.us
* Disallow MERGE cleanly for foreign partitionsAlvaro Herrera2022-10-15
| | | | | | | | | | | | While directly targetting a foreign table with MERGE was already expressly forbidden, we failed to catch the case of a partitioned table that has a foreign table as a partition; and the result if you try is an incomprehensible error. Fix that by adding a specific check. Backpatch to 15. Reported-by: Tatsuhiro Nakamori <bt22nakamorit@oss.nttdata.com> Discussion: https://postgr.es/m/bt22nakamorit@oss.nttdata.com
* pgstat: Track time of the last scan of a relationAndres Freund2022-10-14
| | | | | | | | | | | | | | | | | | | | | It can be useful to know when a relation has last been used, e.g., when evaluating whether an index is still required. It was already possible to infer the time of the last usage by tracking, e.g., pg_stat_all_indexes.idx_scan over time. But far from everybody does so. To make it easier to detect the last time a relation has been scanned, track that time in each relation's pgstat entry. To minimize overhead a) the timestamp is updated only when the backend pending stats entry is flushed to shared stats b) the last transaction's stop timestamp is used as the timestamp. Bumps catalog and stats format versions. Author: Dave Page <dpage@pgadmin.org> Reviewed-by: Andres Freund <andres@anarazel.de> Reviewed-by: Bruce Momjian <bruce@momjian.us> Reviewed-by: Vik Fearing <vik@postgresfriends.org> Discussion: https://postgr.es/m/CA+OCxozrVHNFVEPkweUHMZje+t1tfY816d9MZYc6eZwOOusOaQ@mail.gmail.com
* Have GetCurrentTransactionStopTimestamp() set xactStopTimestamp if unsetAndres Freund2022-10-14
| | | | | | | | | | | | | | | | Previously GetCurrentTransactionStopTimestamp() computed a new timestamp whenever xactStopTimestamp was unset and xactStopTimestamp was only set when a commit or abort record was written. An upcoming patch will add additional calls to GetCurrentTransactionStopTimestamp() from pgstats. To avoid computing timestamps multiple times, set xactStopTimestamp in GetCurrentTransactionStopTimestamp() if not already set. Author: Dave Page <dpage@pgadmin.org> Reviewed-by: Andres Freund <andres@anarazel.de> Reviewed-by: Vik Fearing <vik@postgresfriends.org> Discussion: https://postgr.es/m/20220906155325.an3xesq5o3fq36gt%40awork3.anarazel.de
* Add auxiliary lists to GUC data structures for better performance.Tom Lane2022-10-14
| | | | | | | | | | | | | | | | | | | | | | | | | | | The previous patch made addition of new GUCs cheap, but other GUC operations aren't improved and indeed get a bit slower, because hash_seq_search() is slower than just scanning a pointer array. However, most performance-critical GUC operations only need to touch a relatively small fraction of the GUCs; especially so for AtEOXact_GUC(). We can improve matters at the cost of a bit more space by adding dlist or slist links to the GUC data structures. This patch invents lists that track (1) all GUCs with non-default "source"; (2) all GUCs with nonempty state stack (implying they've been changed in the current transaction); (3) all GUCs due for reporting to the client. All of guc.c's performance-critical cases can make use of one or another of these lists to avoid searching the whole hash table. In particular, the stack list means that transaction end doesn't take time proportional to the number of GUCs, but only to the number changed in the current transaction. Discussion: https://postgr.es/m/2982579.1662416866@sss.pgh.pa.us
* Replace the sorted array of GUC variables with a hash table.Tom Lane2022-10-14
| | | | | | | | | | | | | | | | This gets rid of bsearch() in favor of hashed lookup. The main advantage is that it becomes far cheaper to add new GUCs, since we needn't re-sort the pointer array. Adding N new GUCs had been O(N^2 log N), but now it's closer to O(N). We need to sort only in SHOW ALL and equivalent functions, which are hopefully not performance-critical to anybody. Also, merge GetNumConfigOptions() into get_guc_variables(), because in a world where the set of GUCs isn't fairly static you really want to consider those two results as tied together not independent. Discussion: https://postgr.es/m/2982579.1662416866@sss.pgh.pa.us
* Store GUC data in a memory context, instead of using malloc().Tom Lane2022-10-14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | The only real argument for using malloc directly was that we needed the ability to not throw error on OOM; but mcxt.c grew that feature awhile ago. Keeping the data in a memory context improves accountability and debuggability --- for example, without this it's almost impossible to detect memory leaks in the GUC code with anything less costly than valgrind. Moreover, the next patch in this series will add a hash table for GUC lookup, and it'd be pretty silly to be using palloc-dependent hash facilities alongside malloc'd storage of the underlying data. This is a bit invasive though, in particular causing an API break for GUC check hooks that want to modify the GUC's value or use an "extra" data structure. They must now use guc_malloc() and guc_free() instead of malloc() and free(). Failure to change affected code will result in assertion failures or worse; but thanks to recent effort in the mcxt infrastructure, it shouldn't be too hard to diagnose such oversights (at least in assert-enabled builds). One note is that this changes ParseLongOption() to return short-lived palloc'd not malloc'd data. There wasn't any caller for which the previous definition was better. Discussion: https://postgr.es/m/2982579.1662416866@sss.pgh.pa.us
* Make some minor improvements in memory-context infrastructure.Tom Lane2022-10-14
| | | | | | | | | | | | | | | We lack a version of repalloc() that supports MCXT_ALLOC_NO_OOM semantics, so invent repalloc_extended() with the usual set of flags. repalloc_huge() becomes a legacy wrapper for that. Also, fix dynahash.c so that it can support HASH_ENTER_NULL requests when using the default palloc-based allocator. The only reason it didn't do that already was the lack of the MCXT_ALLOC_NO_OOM option when that code was written, ages ago. While here, simplify a few overcomplicated tests in mcxt.c. Discussion: https://postgr.es/m/2982579.1662416866@sss.pgh.pa.us
* Standardize format for printing PIDsPeter Eisentraut2022-10-14
| | | | | | | | | | | | | | Most code prints PIDs as %d, but some code tried to print them as long or unsigned long. While this is in theory allowed, the fact that PIDs fit into int is deeply baked into all PostgreSQL code, so these random deviations don't accomplish anything except confusion. Note that we still need casts from pid_t to int, because on 64-bit MinGW, pid_t is long long int. (But per above, actually supporting that range in PostgreSQL code would be major surgery and probably not useful.) Discussion: https://www.postgresql.org/message-id/289c2e45-c7d9-5ce4-7eff-a9e2a33e1580@enterprisedb.com
* Fix incorrect comment regarding command completion tagsDavid Rowley2022-10-14
| | | | | | | | | | | | The comment talked about some Asserts which did not exist and also a variable name which seems to have long since disappeared. Rewrite the comment in a way that will hopefully stand the test of time and inform people why we always write "INSERT 0 <nrows>" instead of "INSERT <nrows>" in the command completion tag for INSERT. Reviewed-by: Mark Dilger Discussion: https://postgr.es/m/CAApHDvpiUg09AvvGAVopNAKemA9z-kCmt7Fi6HKauc32bKzx4w@mail.gmail.com
* Allow batch insertion during COPY into a foreign table.Etsuro Fujita2022-10-13
| | | | | | | | | | | | | | | | | | | | | | | | | | | Commit 3d956d956 allowed the COPY, but it's done by inserting individual rows to the foreign table, so it can be inefficient due to the overhead caused by each round-trip to the foreign server. To improve performance of the COPY in such a case, this patch allows batch insertion, by extending the multi-insert machinery in CopyFrom() to the foreign-table case so that we insert multiple rows to the foreign table at once using the FDW callback routine added by commit b663a4136. This patch also allows this for postgres_fdw. It is enabled by the "batch_size" option added by commit b663a4136, which is disabled by default. When doing batch insertion, we update progress of the COPY command after performing the FDW callback routine, to count rows not suppressed by the FDW as well as a BEFORE ROW INSERT trigger. For consistency, this patch changes the timing of updating it for plain tables: previously, we updated it immediately after adding each row to the multi-insert buffer, but we do so only after writing the rows stored in the buffer out to the table using table_multi_insert(), which I think would be consistent even with non-batching mode, because in that mode we update it after writing each row out to the table using table_tuple_insert(). Andrey Lepikhov, heavily revised by me, with review from Ian Barwick, Andrey Lepikhov, and Zhihong Yu. Discussion: https://postgr.es/m/bc489202-9855-7550-d64c-ad2d83c24867%40postgrespro.ru
* Improve the WARNING message for CREATE SUBSCRIPTION.Amit Kapila2022-10-13
| | | | | | Author: Peter Smith Reviewed-By: Alvaro Herrera, Tom Lane, Amit Kapila Discussion: https://postgr.es/m/CAHut+PvqdqOanheWSHDyhQiF+Z-7w=-+k4U+bwbT=b6YQ_hrXQ@mail.gmail.com
* Fix ordering issue with WAL operations in GIN fast insert pathMichael Paquier2022-10-13
| | | | | | | | | | | | | | | | | | Contrary to what is documented in src/backend/access/transam/README, ginHeapTupleFastInsert() had a few ordering issues with the way it does its WAL operations when inserting items in its fast path. First, when using a separate list, XLogBeginInsert() was being always called before START_CRIT_SECTION(), and in this case a second thing was wrong when merging lists, as an exclusive lock was taken on the tail page *before* calling XLogBeginInsert(). Finally, when inserting items into a tail page, the order of XLogBeginInsert() and START_CRIT_SECTION() was reversed. This commit addresses all these issues by moving the calls of XLogBeginInsert() after all the pages logged are locked and pinned, within a critical section. Author: Matthias van de Meent, Zhang Mingli Discussion: https://postgr.es/m/CAEze2WhL8uLMqynnnCu1LAPwxD5RKEo0nHV+eXGg_N6ELU88HQ@mail.gmail.com
* doc: Fix description of replication command CREATE_REPLICATION_SLOTMichael Paquier2022-10-13
| | | | | | | | | | | The output plugin name is a mandatory option when creating a logical slot, but the grammar documented was not described as such. While on it, fix two comments in repl_gram.y to show that TEMPORARY is an optional grammar choice. Author: Ayaki Tachikake Discussion: https://postgr.es/m/OSAPR01MB2852607B2329FFA27834105AF1229@OSAPR01MB2852.jpnprd01.prod.outlook.com Backpatch-through: 15
* Fix shadow variable in postgres.cMichael Paquier2022-10-12
| | | | | | | | -Wshadow=compatible-local is added by default since 0fe954c, and this warning was detected under -DWRITE_READ_PARSE_PLAN_TREES. Reviewed-by: David Rowley Discussion: https://postgr.es/m/Y0Ya5SH0QiaO9kKG@paquier.xyz
* Simplify some maths in xlogreader.cMichael Paquier2022-10-12
| | | | | | | | | | | | An LSN was calculated from a segment number, a segment size and a position offset, matching exactly the LSN given by the caller of XLogReaderValidatePageHeader(). This change removes the extra LSN calculation, relying only on the LSN given by the function caller instead. Author: Bharath Rupireddy Reviewed-by: Richard Guo, Álvaro Herrera, Kyotaro Horiguchi Discussion: https://postgr.es/m/CALj2ACXuh4Ms9j9sxMYdtHEe=5sFcyrs-GAHyADu_A_G71kZTg@mail.gmail.com
* Harden pmsignal.c against clobbered shared memory.Tom Lane2022-10-11
| | | | | | | | | | | | | | | | | | | | | | | | | | | The postmaster is not supposed to do anything that depends fundamentally on shared memory contents, because that creates the risk that a backend crash that trashes shared memory will take the postmaster down with it, preventing automatic recovery. In commit 969d7cd43 I lost sight of this principle and coded AssignPostmasterChildSlot() in such a way that it could fail or even crash if the shared PMSignalState structure became corrupted. Remarkably, we've not seen field reports of such crashes; but I managed to induce one while testing the recent changes around palloc chunk headers. To fix, make a semi-duplicative state array inside the postmaster so that we need consult only local state while choosing a "child slot" for a new backend. Ensure that other postmaster-executed routines in pmsignal.c don't have critical dependencies on the shared state, either. Corruption of PMSignalState might now lead ReleasePostmasterChildSlot() to conclude that backend X failed, when actually backend Y was the one that trashed things. But that doesn't matter, because we'll force a cluster-wide reset regardless. Back-patch to all supported branches, since this is an old bug. Discussion: https://postgr.es/m/3436789.1665187055@sss.pgh.pa.us
* Yet further fixes for multi-row VALUES lists for updatable views.Tom Lane2022-10-11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | DEFAULT markers appearing in an INSERT on an updatable view could be mis-processed if they were in a multi-row VALUES clause. This would lead to strange errors such as "cache lookup failed for type NNNN", or in older branches even to crashes. The cause is that commit 41531e42d tried to re-use rewriteValuesRTE() to remove any SetToDefault nodes (that hadn't previously been replaced by the view's own default values) appearing in "product" queries, that is DO ALSO queries. That's fundamentally wrong because the DO ALSO queries might not even be INSERTs; and even if they are, their targetlists don't necessarily match the view's column list, so that almost all the logic in rewriteValuesRTE() is inapplicable. What we want is a narrow focus on replacing any such nodes with NULL constants. (That is, in this context we are interpreting the defaults as being strictly those of the view itself; and we already replaced any that aren't NULL.) We could add still more !force_nulls tests to further lobotomize rewriteValuesRTE(); but it seems cleaner to split out this case to a new function, restoring rewriteValuesRTE() to the charter it had before. Per bug #17633 from jiye_sw. Patch by me, but thanks to Richard Guo and Japin Li for initial investigation. Back-patch to all supported branches, as the previous fix was. Discussion: https://postgr.es/m/17633-98cc85e1fa91e905@postgresql.org
* Add a common function to generate the origin name.Amit Kapila2022-10-11
| | | | | | | | | | | | | Make a common replication origin name formatting function to replace multiple snprintf() expressions. This also includes logic previously done by ReplicationOriginNameForTablesync(). This makes the code to generate the origin name consistent among apply worker and tablesync worker. Author: Peter Smith Reviewed-By: Aleksander Alekseev Discussion: https://postgr.es/m/CAHut%2BPsa8hhfSE6ozUK-ih7GkQziAVAf4f3bqiXEj2nQiu-43g%40mail.gmail.com
* Add support for COPY TO callback functionsMichael Paquier2022-10-11
| | | | | | | | | | | | | | | | | This is useful as a way for extensions to process COPY TO rows in the way they see fit (say auditing, analytics, backend, etc.) without the need to invoke an external process running as the OS user running the backend through PROGRAM that requires superuser rights. COPY FROM already provides a similar callback for logical replication. For COPY TO, the callback is triggered when we are ready to send a row in CopySendEndOfRow(), which is the same code path as when sending a row to a frontend or a pipe/file. A small test module, test_copy_callbacks, is added to provide some coverage for this facility. Author: Bilva Sanaba, Nathan Bossart Discussion: https://postgr.es/m/253C21D1-FCEB-41D9-A2AF-E6517015B7D7@amazon.com
* Harden memory context allocators against bogus chunk pointers.Tom Lane2022-10-10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Before commit c6e0fe1f2, functions such as AllocSetFree could pretty safely presume that they were given a valid chunk pointer for their own type of context, because the indirect call through a memory context object and method struct would be very unlikely to work otherwise. But now, if pfree() is mistakenly invoked on a pointer to garbage, we have three chances in eight of ending up at one of these functions. That means we need to take extra measures to verify that we are looking at what we're supposed to be looking at, especially in debug builds. Hence, add code to verify that the chunk's back-link to a block header leads to a memory context object that satisfies the right sort of IsA() check. This is still a bit weaker than what we did before, but for the moment assume that an IsA() check is sufficient. As a compromise between speed and safety, implement these checks as Asserts when dealing with small chunks but plain test-and-elogs when dealing with large (external) chunks. The latter case should not be too performance-critical, but the former case probably is. In slab.c, all chunks are small; but nonetheless use a plain test in SlabRealloc, because that is certainly not performance-critical, indeed we should be suspicious that it's being called in error. In aset.c, additionally add some assertions that the "value" field of the chunk header is within the small range allowed for freelist indexes. Without that, we might find ourselves trying to wipe most of memory when CLOBBER_FREED_MEMORY is enabled, or scribbling on a "freelist header" that's far away from the context object. Eventually, field experience might show us that it's smarter for these tests to be active always, but for now we'll try to get away with just having them as assertions. While at it, also be more uniform about asserting that context objects passed as parameters are of the type we expect. Some places missed that altogether, and slab.c was for no very good reason doing it differently from the other allocators. Discussion: https://postgr.es/m/3578387.1665244345@sss.pgh.pa.us
* Simplify our Assert infrastructure a little.Tom Lane2022-10-10
| | | | | | | | | | | | | Remove the Trap and TrapMacro macros, which were nearly unused and confusingly had the opposite condition polarity from the otherwise-functionally-equivalent Assert macros. Having done that, it's very hard to justify carrying the errorType argument of ExceptionalCondition, so drop that too, and just let it assume everything's an Assert. This saves about 64K of code space as of current HEAD. Discussion: https://postgr.es/m/3928703.1665345117@sss.pgh.pa.us
* Remove unnecessary semicolons after goto labelsJohn Naylor2022-10-10
| | | | | | | | | | | According to the C standard, a label must followed by a statement. If there was ever a time we needed an empty statement here, it was a long time ago. Japin Li Reviewed by Julien Rouhaud Discussion: https://www.postgresql.org/message-id/MEYP282MB16690F40189A4F060B41D56DB65E9%40MEYP282MB1669.AUSP282.PROD.OUTLOOK.COM
* Use C library functions instead of Abs() for int64Peter Eisentraut2022-10-10
| | | | | | | | | | Instead of Abs() for int64, use the C standard functions labs() or llabs() as appropriate. Define a small wrapper around them that matches our definition of int64. (labs() is C90, llabs() is C99.) Reviewed-by: Zhang Mingli <zmlpostgres@gmail.com> Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us> Discussion: https://www.postgresql.org/message-id/flat/4beb42b5-216b-bce8-d452-d924d5794c63%40enterprisedb.com
* pgstat: Prevent stats reset from corrupting slotname by removing slotnameAndres Freund2022-10-08
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously PgStat_StatReplSlotEntry contained the slotname, which was mainly used when writing out the stats during shutdown, to identify the slot in the serialized data (at runtime the index in ReplicationSlotCtl->replication_slots is used, but that can change during a restart). Unfortunately the slotname was overwritten when the slot's stats were reset. That turned out to only cause "real" problems if the slot was active during the reset, triggering an assertion failure at the next pgstat_report_replslot(). In other paths the stats were re-initialized during pgstat_acquire_replslot(). Fix this by removing slotname from PgStat_StatReplSlotEntry. Instead we can get the slot's name from the slot itself. Besides fixing a bug, this also is architecturally cleaner (a name is not really statistics). This is safe because stats, for a slot removed while shut down, will not be restored at startup. In 15 the slotname is not removed, but renamed, to avoid changing the stats format. In master, bump PGSTAT_FILE_FORMAT_ID. This commit does not contain a test for the fix. I think this can only be tested by a tap test starting pg_recvlogical in the background and checking pg_recvlogical's output. That type of test is notoriously hard to be reliable, so committing it shortly before the release is wrapped seems like a bad idea. Reported-by: Jaime Casanova <jcasanov@systemguards.com.ec> Author: Andres Freund <andres@anarazel.de> Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com> Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com> Discussion: https://postgr.es/m/YxfagaTXUNa9ggLb@ahch-to Backpatch: 15-, where the bug was introduced in 5891c7a8ed8f
* Use fabsf() instead of Abs() or fabs() where appropriatePeter Eisentraut2022-10-08
| | | | | | | | This function is new in C99. Reviewed-by: Zhang Mingli <zmlpostgres@gmail.com> Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us> Discussion: https://www.postgresql.org/message-id/flat/4beb42b5-216b-bce8-d452-d924d5794c63%40enterprisedb.com
* Fix self-referencing foreign keys with partitioned tablesAlvaro Herrera2022-10-07
| | | | | | | | | | | | | | | | | | | | | | | | | | | There are a number of bugs in this area. Two of them are fixed here, namely: 1. get_relation_idx_constraint_oid does not restrict the type of constraint that's returned, so with sufficient bad luck it can return the OID of a foreign key constraint. This has the effect that a primary key in a partition can end up as a child of a foreign key, which makes no sense (it needs to be the child of the equivalent primary key.) Change the API contract so that only index-backed constraints are returned, mimicking get_constraint_index(). 2. Both CloneFkReferenced and CloneFkReferencing clone a self-referencing foreign key, so the partition ends up with a duplicate foreign key. Change the former function to ignore such constraints. Add some tests to verify that things are better now. (However, these new tests show some additional misbehavior that will be fixed later -- namely that there's a constraint marked NOT VALID.) Backpatch to 12, where these constraints are possible at all. Author: Jehan-Guillaume de Rorthais <jgdr@dalibo.com> Discussion: https://postgr.es/m/20220603154232.1715b14c@karst
* Remove unnecessary uses of Abs()Peter Eisentraut2022-10-07
| | | | | | | | Use C standard abs() or fabs() instead. Reviewed-by: Zhang Mingli <zmlpostgres@gmail.com> Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us> Discussion: https://www.postgresql.org/message-id/flat/4beb42b5-216b-bce8-d452-d924d5794c63%40enterprisedb.com
* Improve our ability to detect bogus pointers passed to pfree et al.Tom Lane2022-10-06
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Commit c6e0fe1f2 was a shade too trusting that any pointer passed to pfree, repalloc, etc will point at a valid chunk. Notably, passing a pointer that was actually obtained from malloc tended to result in obscure assertion failures, if not worse. (On FreeBSD I've seen such mistakes take down the entire cluster, seemingly as a result of clobbering shared memory.) To improve matters, extend the mcxt_methods[] array so that it has entries for every possible MemoryContextMethodID bit-pattern, with the currently unassigned ID codes pointing to error-reporting functions. Then, fiddle with the ID assignments so that patterns likely to be associated with bad pointers aren't valid ID codes. In particular, we should avoid assigning bit patterns 000 (zeroed memory) and 111 (wipe_mem'd memory). It turns out that on glibc (Linux), malloc uses chunk headers that have flag bits in the same place we keep MemoryContextMethodID, and that the bit patterns 000, 001, 010 are the only ones we'll see as long as the backend isn't threaded. So we can have very robust detection of pfree'ing a malloc-assigned block on that platform, at least so long as we can refrain from using up those ID codes. On other platforms, we don't have such a good guarantee, but keeping 000 reserved will be enough to catch many such cases. While here, make GetMemoryChunkMethodID() local to mcxt.c, as there seems no need for it to be exposed even in memutils_internal.h. Patch by me, with suggestions from Andres Freund and David Rowley. Discussion: https://postgr.es/m/2910981.1665080361@sss.pgh.pa.us
* meson: Add support for building with precompiled headersAndres Freund2022-10-06
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This substantially speeds up building for windows, due to the vast amount of headers included via windows.h. A cross build from linux targetting mingw goes from 994.11user 136.43system 0:31.58elapsed 3579%CPU to 422.41user 89.05system 0:14.35elapsed 3562%CPU The wins on windows are similar-ish (but I don't have a system at hand just now for actual numbers). Targetting other operating systems the wins are far smaller (tested linux, macOS, FreeBSD). For now precompiled headers are disabled by default, it's not clear how well they work on all platforms. E.g. on FreeBSD gcc doesn't seem to have working support, but clang does. When doing a full build precompiled headers are only beneficial for targets with multiple .c files, as meson builds a separate precompiled header for each target (so that different compilation options take effect). This commit therefore only changes target with at least two .c files to use precompiled headers. Because this commit adds b_pch=false to the default_options new build directories will have precompiled headers disabled by default, however existing build directories will continue use the default value of b_pch, which is true. Note that using precompiled headers with ccache requires setting CCACHE_SLOPPINESS=pch_defines,time_macros to get hits. Reviewed-by: Peter Eisentraut <peter.eisentraut@enterprisedb.com> Reviewed-by: Justin Pryzby <pryzby@telsasoft.com> Discussion: https://postgr.es/m/CA+hUKG+50eOUbN++ocDc0Qnp9Pvmou23DSXu=ZA6fepOcftKqA@mail.gmail.com Discussion: https://postgr.es/m/c5736f70-bb6d-8d25-e35c-e3d886e4e905@enterprisedb.com Discussion: https://postgr.es/m/20190826054000.GE7005%40paquier.xyz
generated by cgit v1.2.3 (git 2.25.1) at 2025-08-05 02:50:46 +0000