S3: ListObjectsV2, https response error StatusCode: 429 Operation qps limit is exceeded

[Huahubo]

ListObjectsV2 Used to list objects in a bucket, which may trigger QPS limits due to excessive requests. Can it support GetObject?

2024/10/23 10:36:20.304773 error one of Download go-routine return error: one of downloadTableData go-routine return error: operation error S3: ListObjectsV2, https response error StatusCode: 429, RequestID: 7785298319281714615, HostID: 4707fc0955cdce9857cee3b9173dffd1, api error OperationQpsLimitExceeded: Operation qps limit is exceeded. Slow down qps, or contact the administrator. too many ListObjects requests: max qps 20

[Slach]

Actually, we don't use ListObjectV2 so often during download

Which is your compression_format?
clickhouse-backup print-config | grep compression

[Huahubo]

Because we are using a self-built S3, there will be some limitations. Using ListObjects may put too much pressure on S3,So they recommended that I use GetObject . In the config.yml file of clickhouse-backup, S3 is not configured to use compression

[root@XXXXXXXXXXXX ~]# clickhouse-backup print-config | grep compression
compression_level: 1
compression_format: none
compression_level: 1
compression_format: tar
compression_format: tar
compression_level: 1
compression_format: tar
compression_level: 1
compression_format: tar
compression_level: 1
compression_level: 1
compression_format: tar

[Slach]

try
replace

s3:
 compression_format: none

to

s3:
 compression_format: tar

[Slach]

you also could tune following settings

general:
 download_concurrency: 2
s3:
 concurrency: 1