Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

limit maxBytes with compression calculate uncompressed size #5905

Open
paragor opened this issue Sep 19, 2024 · 1 comment
Open

limit maxBytes with compression calculate uncompressed size #5905

paragor opened this issue Sep 19, 2024 · 1 comment
Labels
defect Suspected defect such as a bug or regression

Comments

@paragor
Copy link

paragor commented Sep 19, 2024

Observed behavior

I have stream with compression and limit maxBytes 6Gi

apiVersion: jetstream.nats.io/v1beta2
kind: Stream
metadata:
  name: logs
spec:
  name: logs
  subjects: ["logs.*"]
  storage: file
  maxAge: 12h
  maxBytes: 6442450944
  replicas: 3
  account: logs
  compression: s2

but actualy disk space occupied 536.8M
5.5 gigabytes underutilized

/data/jetstream/$G/streams/logs # df -h /data
Filesystem                Size      Used Available Use% Mounted on
/dev/vdb                  9.7G    536.8M      9.2G   5% /data
/data/jetstream/$G/streams/logs # curl localhost:7777/metrics -s | grep nats_stream_total_bytes
# HELP nats_stream_total_bytes Total stored bytes from a stream
# TYPE nats_stream_total_bytes gauge
nats_stream_total_bytes{account="$G",account_id="$G",cluster="nats",domain="",is_meta_leader="true",is_stream_leader="true",meta_leader="nats-0",server_id="NC6ARVXMQOBMIQNT3ULUK2MVNLX43DPZD43YKA3OMKHGTXQD55MIP357",server_name="nats-0",stream_leader="nats-0",stream_name="logs"} 6.442450821e+09
> nats stream info logs
Information for Stream logs created 2024-09-17 22:50:57

              Subjects: logs.*
              Replicas: 3
               Storage: File
           Compression: S2 Compression

Options:

             Retention: Limits
       Acknowledgments: true
        Discard Policy: Old
      Duplicate Window: 2m0s
     Allows Msg Delete: true
          Allows Purge: true
        Allows Rollups: false

Limits:

      Maximum Messages: unlimited
   Maximum Per Subject: unlimited
         Maximum Bytes: 6.0 GiB
           Maximum Age: 12h0m0s
  Maximum Message Size: unlimited
     Maximum Consumers: unlimited

Cluster Information:

                  Name: nats
                Leader: nats-0
               Replica: nats-1, current, seen 21ms ago
               Replica: nats-2, current, seen 21ms ago

State:

              Messages: 6,157,140
                 Bytes: 6.0 GiB
        First Sequence: 27,979,737 @ 2024-09-19 04:05:43
         Last Sequence: 34,136,876 @ 2024-09-19 13:56:38
      Active Consumers: 0
    Number of Subjects: 11

Expected behavior

nats calculate maxBytes for limits as actual size on disk

Server and client version

nats-server: v2.10.19

Host environment

No response

Steps to reproduce

No response

@paragor paragor added the defect Suspected defect such as a bug or regression label Sep 19, 2024
@ripienaar
Copy link
Contributor

Limits are always calculated on the message size rather than how the storage would store it. We have had a few issues filed around this in the past but this is working as designed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
defect Suspected defect such as a bug or regression
Projects
None yet
Development

No branches or pull requests

2 participants