On 16th of January 2024, Peter Eisentraut committed patch:
Support identity columns in partitioned tables Previously, identity columns were disallowed on partitioned tables. (The reason was mainly that no one had gotten around to working through all the details to make it work.) This makes it work now. Some details on the behavior: * A newly created partition inherits identity property The partitions of a partitioned table are integral part of the partitioned table. A partition inherits identity columns from the partitioned table. An identity column of a partition shares the identity space with the corresponding column of the partitioned table. In other words, the same identity column across all partitions of a partitioned table share the same identity space. This is effected by sharing the same underlying sequence. When INSERTing directly into a partition, the sequence associated with the topmost partitioned table is used to calculate the value of the corresponding identity column. In regular inheritance, identity columns and their properties in a child table are independent of those in its parent tables. A child table does not inherit identity columns or their properties automatically from the parent. (This is unchanged.) * Attached partition inherits identity column A table being attached as a partition inherits the identity property from the partitioned table. This should be fine since we expect that the partition table's column has the same type as the partitioned table's corresponding column. If the table being attached is a partitioned table, the identity properties are propagated down its partition hierarchy. An identity column in the partitioned table is also marked as NOT NULL. The corresponding column in the partition needs to be marked as NOT NULL for the attach to succeed. * Drop identity property when detaching partition A partition's identity column shares the identity space (i.e. underlying sequence) as the corresponding column of the partitioned table. If a partition is detached it can longer share the identity space as before. Hence the identity columns of the partition being detached loose their identity property. When identity of a column of a regular table is dropped it retains the NOT NULL constraint that came with the identity property. Similarly the columns of the partition being detached retain the NOT NULL constraints that came with identity property, even though the identity property itself is lost. The sequence associated with the identity property is linked to the partitioned table (and not the partition being detached). That sequence is not dropped as part of detach operation. * Partitions with their own identity columns are not allowed. * The usual ALTER operations (add identity column, add identity property to existing column, alter properties of an indentity column, drop identity property) are supported for partitioned tables. Changing a column only in a partitioned table or a partition is not allowed; the change needs to be applied to the whole partition hierarchy. Author: Ashutosh Bapat <ashutosh.bapat.oss@gmail.com> Reviewed-by: Peter Eisentraut <peter@eisentraut.org> Discussion: https://www.postgresql.org/message-id/flat/CAExHW5uOykuTC+C6R1yDSp=o8Q83jr8xJdZxgPkxfZ1Ue5RRGg@mail.gmail.com
Commit message is pretty long and descriptive, so let's just see it in action.
=$ CREATE TABLE some_data ( id int8 generated always AS IDENTITY NOT NULL, created_at timestamptz NOT NULL, payload TEXT ) PARTITION BY RANGE ( created_at ) ; =$ CREATE TABLE some_data_old PARTITION OF some_data FOR VALUES FROM (MINVALUE) TO ('2022-01-01'); =$ CREATE TABLE some_data_y2022 PARTITION OF some_data FOR VALUES FROM ('2022-01-01') TO ('2023-01-01'); =$ CREATE TABLE some_data_y2023 PARTITION OF some_data FOR VALUES FROM ('2023-01-01') TO ('2024-01-01'); =$ CREATE TABLE some_data_y2024 PARTITION OF some_data FOR VALUES FROM ('2024-01-01') TO ('2025-01-01'); =$ CREATE TABLE some_data_y2025 PARTITION OF some_data FOR VALUES FROM ('2025-01-01') TO ('2026-01-01');
Now, I can insert to all of these tables:
=$ INSERT INTO some_data (created_at, payload) SELECT now() - '5 years'::INTERVAL * random(), 'Row #' || i FROM generate_series(1,10) i; INSERT 0 10 =$ SELECT tableoid::regclass, * FROM some_data ORDER BY created_at; tableoid │ id │ created_at │ payload ─────────────────┼────┼───────────────────────────────┼───────── some_data_old │ 8 │ 2020-12-14 08:40:15.272099+00 │ ROW #8 some_data_old │ 7 │ 2021-01-27 12:28:57.147299+00 │ ROW #7 some_data_old │ 4 │ 2021-03-27 15:24:49.524899+00 │ ROW #4 some_data_old │ 2 │ 2021-10-14 01:33:39.531299+00 │ ROW #2 some_data_y2022 │ 10 │ 2022-10-04 06:04:15.905699+00 │ ROW #10 some_data_y2023 │ 6 │ 2023-01-02 05:19:54.267299+00 │ ROW #6 some_data_y2023 │ 9 │ 2023-03-14 09:37:02.196899+00 │ ROW #9 some_data_y2023 │ 1 │ 2023-04-09 14:36:47.566499+00 │ ROW #1 some_data_y2023 │ 3 │ 2023-10-28 22:47:40.980899+00 │ ROW #3 some_data_y2023 │ 5 │ 2023-11-14 14:18:36.593699+00 │ ROW #5 (10 ROWS)
As you can see ids are being assigned in order of insertion, regardless of which partition the data ends up in.
This also works with direct inserts to the partitions:
=$ INSERT INTO some_data_y2024 (created_at, payload) VALUES (now(), 'manual #1') returning *; id │ created_at │ payload ────┼──────────────────────────────┼─────────── 11 │ 2024-02-07 15:54:46.95793+00 │ manual #1 (1 ROW) =$ INSERT INTO some_data_y2023 (created_at, payload) VALUES (now() - '1 year'::INTERVAL, 'manual #2') returning *; id │ created_at │ payload ────┼───────────────────────────────┼─────────── 12 │ 2023-02-07 15:54:55.463043+00 │ manual #2 (1 ROW)
One issue it that we can't have primary key or unique on the id column:
=$ ALTER TABLE some_data ADD PRIMARY KEY (id); ERROR: UNIQUE CONSTRAINT ON partitioned TABLE must include ALL partitioning COLUMNS DETAIL: PRIMARY KEY CONSTRAINT ON TABLE "some_data" lacks COLUMN "created_at" which IS part OF the partition KEY. =$ ALTER TABLE some_data ADD UNIQUE (id); ERROR: UNIQUE CONSTRAINT ON partitioned TABLE must include ALL partitioning COLUMNS DETAIL: UNIQUE CONSTRAINT ON TABLE "some_data" lacks COLUMN "created_at" which IS part OF the partition KEY.
Which isn't perfect, but we can always add unique on the partitions directly:
=$ ALTER TABLE some_data_old ADD UNIQUE (id); =$ ALTER TABLE some_data_y2022 ADD UNIQUE (id); =$ ALTER TABLE some_data_y2023 ADD UNIQUE (id); =$ ALTER TABLE some_data_y2024 ADD UNIQUE (id); =$ ALTER TABLE some_data_y2025 ADD UNIQUE (id);
Of course, Pg can't know which partition has row with specific id, so it will have to scan them all:
=$ EXPLAIN SELECT * FROM some_data WHERE id = 7; QUERY PLAN ────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Append (cost=0.00..12.35 ROWS=5 width=48) -> Seq Scan ON some_data_old some_data_1 (cost=0.00..1.05 ROWS=1 width=48) FILTER: (id = 7) -> Seq Scan ON some_data_y2022 some_data_2 (cost=0.00..1.01 ROWS=1 width=48) FILTER: (id = 7) -> Seq Scan ON some_data_y2023 some_data_3 (cost=0.00..1.07 ROWS=1 width=48) FILTER: (id = 7) -> Seq Scan ON some_data_y2024 some_data_4 (cost=0.00..1.01 ROWS=1 width=48) FILTER: (id = 7) -> INDEX Scan USING some_data_y2025_id_key1 ON some_data_y2025 some_data_5 (cost=0.15..8.17 ROWS=1 width=48) INDEX Cond: (id = 7) (11 ROWS)
But if will use index on id (once there will be enough data), so it should be fast.
On the other hand, if your partitioning schema is using identity column as base for partitioning, it will work well, and can have primary key on it:
=$ CREATE TABLE some_data ( id int8 generated always AS IDENTITY NOT NULL, payload TEXT, PRIMARY KEY (id) ) PARTITION BY RANGE ( id ) ; =$ CREATE TABLE some_data_old PARTITION OF some_data FOR VALUES FROM (MINVALUE) TO (100); =$ CREATE TABLE some_data_1xx PARTITION OF some_data FOR VALUES FROM (100) TO (200); =$ CREATE TABLE some_data_2xx PARTITION OF some_data FOR VALUES FROM (200) TO (300); =$ CREATE TABLE some_data_3xx PARTITION OF some_data FOR VALUES FROM (300) TO (400); =$ CREATE TABLE some_data_4xx PARTITION OF some_data FOR VALUES FROM (400) TO (500);
and then:
=$ \d some_data_2xx TABLE "public.some_data_2xx" COLUMN │ TYPE │ Collation │ NULLABLE │ DEFAULT ─────────┼────────┼───────────┼──────────┼────────────────────────────── id │ BIGINT │ │ NOT NULL │ generated always AS IDENTITY payload │ text │ │ │ Partition OF: some_data FOR VALUES FROM ('200') TO ('300') Indexes: "some_data_2xx_pkey" PRIMARY KEY, btree (id)
As a side note, such syntax will break direct inserts to partitions:
$ INSERT INTO some_data_2xx (payload) VALUES ('a'); ERROR: NEW ROW FOR relation "some_data_2xx" violates partition CONSTRAINT DETAIL: Failing ROW contains (1, a).
This is, of course, because identity is shared, and newly given ids will be “too old" for some_data_2xx, and should go to some_data_old.
I think it's great thing, and I really hope we will be able to get primary keys over partitions, even if they don't align with partition key.
Thanks to all involved 🙂
thank you so much