pg_vector index support

[34code]

I'm unable to create an HNSW index using the following command:

CREATE INDEX ON com_ebay_items USING hnsw (title_vec vector_l2_ops);

I have citus installed alongside pg_vector (create extension if not exists vector)

Here is the error message I'm getting:

ERROR:  could not resize shared memory segment "/PostgreSQL.1153325950" to 1048576 bytes: No space left on device
CONTEXT:  while executing command on citus_worker.2.p2lw3ox7d6907doqtzz3crk12:5432

[onurctirtir]

To speed-up the index build, hnsw tries to allocate maintenance_work_mem (seems set to 1G in your setup) amount of memory from shared memory if it decides that it can use multiple processes to build the index, so the error is actually caused by hnsw not Citus.

(Also note that for serial index builds, this is not the case because then hnsw won't try allocating memory from shared-memory but from the same backend's memory pool.)

This especially becomes a problem when your database is setup on a docker container.

https://www.timescale.com/learn/how-to-address-error-could-not-resize-shared-memory-segment
https://stackoverflow.com/a/56754077

If increasing shm-size doesn't help, please let us know again.

[34code]

Hi I've successfully managed to increase shm_size to 8g using docker's daemon.json but this error comes back with a larger number instead of 1048576 bytes.

[onurctirtir]

Let me ask a few more questions to better understand what's happening:

How does your Citus cluster setup look like?

• How many Citus (worker) nodes do you have?
• Your worker node instances are served from the same docker container or from separate ones?

And how many shards does com_ebay_items have?

[34code]

I've given up on multi node citus and am moving back to vanilla postgres on one single big node now due to this issue.

Answers to questions:

• one coordinator node and 2 worker nodes
• worker nodes are on separate hosts with separate containers on each host using docker swarm
• 32 shards