(How) can I convert a table to a foreign table in Postgres?


Martin K

We have a large table in a Postgres production database and we want to start "sharding" using external tables and inheritance.

The required architecture would be to have 1 (empty) table (defining the schema) and several external tables that inherit from the empty "parent" table. (for Postgres 9.5)

I found this well written article https://www.depesz.com/2015/04/02/waiting-for-9-5-allow-foreign-tables-to-participate-in-inheritance/ that explains on how to start from scratch. My question is how to minimize the required data migration.

Now that we have this 100+ GB table, it should be our first "shard". We will regularly add new "shards" in the future. At some point the older shards will be moved to another tablespace (on cheaper hardware since they become less critical).

Now my question is: is there a way to "change" an existing table to be a foreign table?

Chris Travers

There is no way to use alter table to do this.

You actually have to do this manually. It's (practically) no different than doing table partitioning. Create partitions and load data. You read and write directly to the partition.

Now, in your case, as far as sharding is concerned, I would consider a number of tools to ease this pain. First, if you make sure to split the table the way you like first, you can use a logical replication solution like Bucardo to replicate the writes when you move everything.

There are a few other approaches (parallel readers and writers) that save some time, at the cost of db loading, but these are niche tools.

There is no native solution for standard PostgreSQL's shard management (I don't know enough about Postgres-XL in this regard to know how well it manages changing shard conditions). But with a little work and knowledge, almost anything is possible.

Related


(How) can I convert a table to a foreign table in Postgres?

Martin K We have a large table in a Postgres production database and we want to start "sharding" using external tables and inheritance. The required architecture would be to have 1 (empty) table (defining the schema) and several external tables that inherit fr

How to emulate a Postgres foreign key into a partitioned table

Jeff G I have a partitioned table (call it A) with a serial primary key that is referenced by another table (call it B). I know that I can't actually create foreign keys from one to the other (since I don't know on which partition the data is actually stored),

How to emulate a Postgres foreign key into a partitioned table

Jeff G I have a partitioned table (call it A) with a serial primary key that is referenced by another table (call it B). I know that I can't actually create foreign keys from one to the other (since I don't know on which partition the data is actually stored),

How to emulate a Postgres foreign key into a partitioned table

Jeff G I have a partitioned table (call it A) with a serial primary key that is referenced by another table (call it B). I know that I can't actually create foreign keys from one to the other (since I don't know on which partition the data is actually stored),

How to emulate a Postgres foreign key into a partitioned table

Jeff G I have a partitioned table (call it A) with a serial primary key that is referenced by another table (call it B). I know that I can't actually create foreign keys from one to the other (since I don't know on which partition the data is actually stored),

Postgres - How to bulk insert into table with foreign keys

Dopma I want to do bulk inserts in my postgreSQL database. Database not yet online PostgreSQL 13 I have a temporary temp table where I can bulk insert data TABLE public.temp_inverter_location ( id integer , inverter_num_in_sld integer, lift_require

How to emulate a Postgres foreign key into a partitioned table

Jeff G I have a partitioned table (call it A) with a serial primary key that is referenced by another table (call it B). I know that I can't actually create foreign keys from one to the other (since I don't know on which partition the data is actually stored),

How can I convert this table to the correct DIV?

Anthony Horne I'm having trouble getting the correct set of DIVs for the content in the table. Apparently the table version cannot be "scaled" or resized. I'm a developer and old school, so I use Tables (the working part of the brain that knows how to format t

How can I convert this table to the correct DIV?

Anthony Horne I'm having trouble getting the correct set of DIVs for the content in the table. Apparently the table version cannot be "scaled" or resized. I'm a developer and old school, so I use Tables (the working part of the brain that knows how to format t

How can I refer to the foreign key of the query table?

Sean Wilson Other fields of my Access database MainTableinclude Event Description. I get event description from another table EventTablewith two fields , EventIDandEvent Description My form has a combo box that lets the user select an "event description" and i

How can I add two fields to a table as foreign keys?

amir_70 I have two tables "teams" and "matches". Each team can play in a different game, and each game contains two or more teams, so it's a many-to-many relationship. Here is my contest model: class Match(models.Model): result = models.CharField(max_lengt

How can I refer to the foreign key of the query table?

Sean Wilson Other fields of my Access database MainTableinclude Event Description. I get event description from another table EventTablewith two fields , EventIDandEvent Description My form has a combo box that lets the user select an "event description" and i

How can I refer to the foreign key of the query table?

Sean Wilson Other fields of my Access database MainTableinclude Event Description. I get event description from another table EventTablewith two fields , EventIDandEvent Description My form has a combo box that lets the user select an "event description" and i

How can I refer to the foreign key of the query table?

Sean Wilson Other fields of my Access database MainTableinclude Event Description. I get event description from another table EventTablewith two fields , EventIDandEvent Description My form has a combo box that lets the user select an "event description" and i

How can I refer to the foreign key of the query table?

Sean Wilson Other fields of my Access database MainTableinclude Event Description. I get event description from another table EventTablewith two fields , EventIDandEvent Description My form has a combo box that lets the user select an "event description" and i

How can I refer to the foreign key of the query table?

Sean Wilson Other fields of my Access database MainTableinclude Event Description. I get event description from another table EventTablewith two fields , EventIDandEvent Description My form has a combo box that lets the user select an "event description" and i