quota-backend-bytes parameter does not work

[ivashutin]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via etcd maintainers.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

the quota-backend-bytes parameter does not work. We pass this parameter via the environment variable ETCD_QUOTA_BACKEND_BYTES.
The value of this variable is 6442450944, but the NOSPACE alarm is triggered by the default value of 2 GiB.
At the same time, the etcd_server_quota_backend_bytes metric returns the value 6442450944.

What did you expect to happen?

when using the environment variable ETCD_QUOTA_BACKEND_BYTES, the NOSPACE alarm is triggered by the value specified in this variable

How can we reproduce it (as minimally and precisely as possible)?

NA

Anything else we need to know?

No response

Etcd version (please run commands below)

ETCD version 3.5.6

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

{"level":"warn","ts":"2025-01-12T03:54:03.676+0700","caller":"etcdserver/apply.go:752","msg":"alarm raised","alarm":"NOSPACE","from":"98ca18942631f0a4"}

[jmhbnz]

Hi @ivashutin - Thanks for your question. Can you please confirm you are exporting the environment variable?

When I run the following I observe the environment variable working as expected:

export ETCD_QUOTA_BACKEND_BYTES=6442450944                                                                                                    

./etcd
{"level":"info","ts":"2025-01-16T17:26:35.400258+1300","caller":"flags/flag.go:114","msg":"recognized and used environment variable","variable-name":"ETCD_QUOTA_BACKEND_BYTES","variable-value":"6
442450944"}                                                                                                                                                                                        

It would be best if you can provide your full etcd configuration and startup logs so we can attempt to reproduce the issue using the same configuration.

[ivashutin]

etcd is started via systemd.
systemd unit looks like this

[Unit]
Description=etcd service
Documentation=https://github.com/etcd-io/etcd
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
User=etcd
Group=etcd
Type=notify
ExecStart=/usr/local/bin/etcd
EnvironmentFile=/etc/sysconfig/etcd
Restart=always
RestartSec=10s
LimitNOFILE=40000

[Install]
WantedBy=multi-user.target

EnvironmentFile looks like this

...
ETCD_QUOTA_BACKEND_BYTES=6442450944

I see it in the startup logs

{"level":"info","ts":"2025-01-12T05:07:18.418+0700","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_QUOTA_BACKEND_BYTES","variable-value":"6442450944"}

but at the same time NOSPACE alarm is triggered when the database size is 2 GiB

[jmhbnz]

Interesting ok thanks for the additional info. Was the quota always set at 6442450944 for this cluster or was it changed post initial creation?

Also - Does the issue occur with a more recent etcd version? v3.5.6 was released over two years ago and we've fixed lots of bugs and security vulnerabilities since then.

[ivashutin]

At first there was a default value, then we set the value to 6442450944
we have one etcd cluster, so I haven't tested it on others

[jmhbnz]

Ok I've just run a test with v3.5.16 at my end as described below and I cannot recreate the issue.

Please review and let me know if we should try anything differently:

1 - Start etcd server with default backend quota

We can see the default quota of 2147483648 appear in the startup logs below.

./etcd --version                                                                                                                       
etcd Version: 3.5.16                                                                                                                                                                               
Git SHA: 17ab1fa6d                                                                                                                                                                                 
Go Version: go1.22.8                                                                                                                                                                               
Go OS/Arch: linux/amd64 

./etcd                                                                                                                                 
{"level":"warn","ts":"2025-01-16T20:37:01.354809+1300","caller":"embed/config.go:689","msg":"Running http and grpc server on single port. This is not recommended for production."}                
{"level":"info","ts":"2025-01-16T20:37:01.354871+1300","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["./etcd"]}                                                                         
{"level":"warn","ts":"2025-01-16T20:37:01.354901+1300","caller":"etcdmain/etcd.go:105","msg":"'data-dir' was empty; using default","data-dir":"default.etcd"}                                      
{"level":"warn","ts":"2025-01-16T20:37:01.354915+1300","caller":"embed/config.go:689","msg":"Running http and grpc server on single port. This is not recommended for production."}                
{"level":"info","ts":"2025-01-16T20:37:01.354920+1300","caller":"embed/etcd.go:128","msg":"configuring peer listeners","listen-peer-urls":["http://localhost:2380"]}                               
{"level":"info","ts":"2025-01-16T20:37:01.355785+1300","caller":"embed/etcd.go:136","msg":"configuring client listeners","listen-client-urls":["http://localhost:2379"]}                           
{"level":"info","ts":"2025-01-16T20:37:01.356000+1300","caller":"embed/etcd.go:311","msg":"starting an etcd server","etcd-version":"3.5.16","git-sha":"17ab1fa6d","go-version":"go1.22.8","go-os":"
linux","go-arch":"amd64","max-cpu-set":16,"max-cpu-available":16,"member-initialized":false,"name":"default","data-dir":"default.etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"default.et
cd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"max-wals":5,"max-snapshots":5,"snapshot-cat
chup-entries":5000,"initial-advertise-peer-urls":["http://localhost:2380"],"listen-peer-urls":["http://localhost:2380"],"advertise-client-urls":["http://localhost:2379"],"listen-client-urls":["ht
tp://localhost:2379"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"default=http://localhost:2380","initial-cluster-state":"new","initial-cluster-token":"etcd-cl
uster","quota-backend-bytes":2147483648,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","compact-c
heck-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-prox
y":"","downgrade-check-interval":"5s"}

2 - Stop etcd server, and restart with increase quota

We can see the new quota reflected in the second startup logs

export ETCD_QUOTA_BACKEND_BYTES=3242450944

./etcd
{"level":"info","ts":"2025-01-16T20:38:49.573188+1300","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_QUOTA_BACKEND_BYTES","variable-value":"3
242450944"}

3 - Use benchmark tool to quickly fill the database

We can see nospace is not reached until > 3GB of data is stored

./benchmark put --key-size 1024 --val-size 202022 --total 90000

# Observe database size and alarm status
./etcdctl endpoint status -w table --cluster                                                                                        
+-----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------------------------------+
|       ENDPOINT        |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX |             ERRORS             |
+-----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------------------------------+
| http://localhost:2379 | 8e9e05c52164694d |  3.5.16 |  3.2 GB |      true |      false |         4 |      14619 |              14619 |  memberID:10276657743932975437 |
|                       |                  |         |         |           |            |           |            |                    |                 alarm:NOSPACE  |
+-----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------------------------------+

# Logs from etcd server for nospace
{"level":"warn","ts":"2025-01-16T20:40:38.110830+1300","caller":"etcdserver/apply.go:752","msg":"alarm raised","alarm":"NOSPACE","from":"8e9e05c52164694d"}

Given your scenario appears to work fine in v3.5.16 I would suggest updating to a newer etcd version. @ahrtr do you have any ideas on what might be happening here?

[ahrtr]

• You need to set the same value (either --quota-backend-bytes or env ETCD_QUOTA_BACKEND_BYTES) for all members;
• Please check the real db file size
• If the NOSPACE alarm has already been triggered, then it won't disappear automatically when you set a bigger storage quota.