Disk buffer related queries - Version 0.22.0

[hemanthofficial3009]

hello,
We are using the vector instance as both agent and collector on different platforms. version v 0.22.0 (the new disk buffer implementation)

We would like to understand the behavior of the buffer.max_size. Please correct me if my understanding is wrong.
Note: The size of each event used in the test is around 616 bytes. (refer to line 1819 of the internal metrics log file attached below)
Here is config file that i have used for the test:

1. What is the significance of buffer.max_size? We expected that at any given point of time the maximum number of bytes that could be held in the buffer = buffer.max_size.
2. what would happen to the incoming events when the buffer.max_size is met? According to the configuration I have set the buffer.when_full to "block", is there any way to test if this is happening?
3. When the vector agent is down what would happen to the events that are already present in the buffer, would the time out secs be still active to send out the events(present in the buffer) to the downstream vector instance or would it remain in the buffer until the vector agent comes back live?
4. When would the files "buffer.lock", "buffer.db" , "buffer-data-0.dat" get deleted from the disk? Is there a ceiling point for the size of .dat file where it would not allow the further growth of the file. We have certain limitations related to the availability of disk space on few systems where we would like to run the vector instances.
5. Is there any relation between buffer_byte_size and buffer_max_byte_size, (refer to line 1554 and 1559 in the internal metrics log file)

We have enabled internal metrics and filtered out the metrics related to buffer to observe the flow of bytes/events in and out of the vector agent instance.

internal_metrics-2022-06-06.log

[jszwedko]

Hi @hemanthofficial3009 !

• What is the significance of buffer.max_size? We expected that at any given point of time the maximum number of bytes that could be held in the buffer = buffer.max_size.

That's correct, it is the maximum size of the disk buffer, in bytes. I believe it does round to the nearest "block" size though so in practice it might be a bit larger. cc/ @tobz to confirm that behavior.

• what would happen to the incoming events when the buffer.max_size is met? According to the configuration I have set the buffer.when_full to "block", is there any way to test if this is happening?

That's correct, if when_full = "block" the disk buffer will stop accepting new events once full and propagate back pressure up to input components. You can verify this by tracking the buffer_received_events_total metric. It will flatline when the buffer is full.

• When the vector agent is down what would happen to the events that are already present in the buffer, would the time out secs be still active to send out the events(present in the buffer) to the downstream vector instance or would it remain in the buffer until the vector agent comes back live?

The timeout_secs refers to the batch rather than the buffer. If Vector goes offline, any events in the disk buffer will be sent when Vector comes back online and reads pending events from the buffer.

• When would the files "buffer.lock", "buffer.db" , "buffer-data-0.dat" get deleted from the disk? Is there a ceiling point for the size of .dat file where it would not allow the further growth of the file. We have certain limitations related to the availability of disk space on few systems where we would like to run the vector instances.

@tobz can probably better answer this since it pertains to the internals of the buffer, but my understanding is that it will compact and delete files. I'm not sure what the theoretical maximum disk usage for a given buffer.max_size would be though.

• Is there any relation between buffer_byte_size and buffer_max_byte_size, (refer to line 1554 and 1559 in the internal metrics log file)

buffer_byte_size refers to the number of bytes currently in the buffer. buffer_max_byte_size refers to the maximum bytes allowed in the buffer. You can divide these two metrics (buffer_byte_size / buffer_max_byte_size) to get an idea of how full a buffer is. Note that the buffer_byte_size metric is refreshed on an interval and represents a point-in-time measure.

[tobz]

Just to follow up:

What is the significance of buffer.max_size? We expected that at any given point of time the maximum number of bytes that could be held in the buffer = buffer.max_size.

buffer.max_size attempts to be the upper limit on the total disk usage for the buffer, not just the size of the encoded events that end up in the buffer. So, ideally, the total size of the files on disk should not exceed buffer.max_size, with one caveat: buffer.max_size has a minimum value of 256MB.

Each data file is limited to ~128MB, and we always need to allow a minimum of two data files to co-exist to ensure buffer writes can continue when a full data file is still being read, hence 2x 128MB, and thus 256MB. This is enforced automatically even if you set the value lower in your configuration.

When would the files "buffer.lock", "buffer.db" , "buffer-data-0.dat" get deleted from the disk? Is there a ceiling point for the size of .dat file where it would not allow the further growth of the file. We have certain limitations related to the availability of disk space on few systems where we would like to run the vector instances.

So as mentioned above, we use 128MB as the maximum size for an individual data file.

In terms of when buffer files are deleted:

• buffer.db never gets deleted unless you delete it manually
• buffer.lock is created when Vector starts, and deleted when it stops, and helps prevent two Vector instances from trying to open the same buffer data
• buffer-data-###.dat is deleted once all records (events) from the data file have been read and acknowledged

Putting it all together:

• you always need at least 256MB of space
• the size of a buffer will go down in 128MB chunks (as each data file is eventually deleted)

[hemanthofficial3009]

Thanks @jszwedko , @tobz for the quick response. The information that you have provided is very helpful. Just a suggestion from our end, please do mention in the documentation regarding the min default size of disk buffer so that it might help other users as well.
Just few follow up question regarding the buffer.max_size

1. Is there any way to reduce the minimum value(~256MB) of the buffer.max_size in the configuration or during the compilation of vector itself.
2. We have certain use cases where we would like to run vector instances on few low level systems which might not accommodate 256MB for the buffering the event, what would be the out come when the available diskspace is less than the default buffer.max_size?

We would like to understand few details related to the buffer.db file.

1. What is the significance of the buffer.db file?
2. When would the size of this file increase?

Thanks again for making out time to help us understand better.

[tobz]

1. Is there any way to reduce the minimum value(~256MB) of the buffer.max_size in the configuration or during the compilation of vector itself.

You can indeed change the constant that defines the default maximum file size if you wanted to compile Vector yourself. The constant DEFAULT_MAX_DATA_FILE_SIZE controls that. As mentioned previously, though, we'll always enforce that the max buffer size is at least the size of the max data file size, which translates to needing to provision enough storage for 2x the value of DEFAULT_MAX_DATA_FILE_SIZE.

2. We have certain use cases where we would like to run vector instances on few low level systems which might not accommodate 256MB for the buffering the event, what would be the out come when the available diskspace is less than the default buffer.max_size?

Vector doesn't currently check to assert there's enough free space to hold the buffer at it's maximum size, or even that the volume it's on is big enough.

Thus, Vector would start up and run normally so long as there was enough space, but if there was no more space, an I/O error would occur on writes. I forget if the events would simply drop and fail to be written or if the sink would stop processing altogether... I need to double check that and report back.

We would like to understand few details related to the buffer.db file.

1. What is the significance of the buffer.db file?

It controls the state of the buffer i.e. the last acknowledged event, last written event, etc. Do not delete this file, or bad things (including data loss) may occur.

2. When would the size of this file increase?

It would only possibly change size between releases of Vector if we added new fields to the file, which is very unlikely to happen.

Thanks again for making out time to help us understand better.

Of course! We're excited for people to use it so I'm very happy to talk through the design decisions with interested users. :)

[hemanthofficial3009]

@tobz

in our network we would like to use vector off the shelf binary, could you please consider if the BUFFER_SIZE (currently 256 MB) can be made configurable in the future release.

[tobz]

It's something we have an issue open for (#13120) but may take a while to get to.

[tobz]

@hemanthofficial3009 Alright, so I double checked: if an out of space I/O error is encountered when writing to the buffer, or flushing it to disk (not every write flushes it to disk), we issue a panic!(...), which kills the component that sent the event which encountered the error.

Since our components are all loosely linked via channels, the component that tries to send into the buffer, and fails due to insufficient space, would stop, and then any other component trying to send to that one would also die, in a cascading fashion back up to the source.

I opened #13043 as a high-level issue for tracking working related to improving this, both in terms of better errors on what is happening when a write fails and all of the components that don't have the buffer seem to die, as well as in terms of trying to gracefully handle out-of-space errors if possible.

[tobz]

Testing again, I was able to replicate what you're seeing.

This error shouldn't be getting turned into a panic, but simply emitted as an error while the reader continues. I've opened #13178 to track this bug, and the subsequent work to fix it.

In this particular instance, with the current version of Vector, it will not be able to move past the error and you would need to delete the buffer data files to allow the buffer to be recreated.

[bmallya]

@tobz @jszwedko , thanks for creating a ticket for this and my apologies if I am re-opening this discussion again. I was wondering if the default max_buffer_size that you mentioned as 256 MB can be made configurable in the coming releases.
We are looking at buffering events to a max of some KBs, which seems to not be possible now.

[tobz]

@bmallya No worries.

The minimum buffer size is dictated by some other constants, which are in scope to potentially be made configurable in the future (#13120). I can't speak to the timeline, however, as making these types of changes involves a significant time investment due to correctness testing.

The best thing to do would be to put a thumbs up on the issue, as we informally use the interest shown in a particular issue/feature request as a gauge for what the community is looking for.

[hemanthofficial3009]

@tobz @jszwedko , thanks for handling the panic issue. I have tried out the latest version 0.22.3 (after 0.22.0) and observed that the minimum value for buffer.max_size is set to 134217728 bytes.
case 1: when the buffer.max_size id set to something lower to the above mentioned value, the vector instance doesn't start and throws an error stating the same.

case 2: value set to 140000000 bytes, I observe a block happening after it hits 135MB size on buffer.

case 3: value set to 180000000 bytes, I observe a similar behaviour i.e., block happening after it hits 135MB size on buffer.

Q1. Can you please look into this to help us understand the behavior of latest buffer changes.

Q2. Are there any particular intervals set major/minor releases otherwise can you let us know the frequency of major/minor releases so that we could plan a particular pattern to pull the latest stable version of the binary.

Thanks for all the support.

[jszwedko]

Hi @hemanthofficial3009 !

I'll let @tobz respond to Q1 there since he's more familiar with the buffer implementation, but for Q2 you can see https://github.com/vectordotdev/vector/blob/master/RELEASES.md for our release cadence (every 6 weeks for major releases).

[tobz]

Q1. Can you please look into this to help us understand the behavior of latest buffer changes.

In your example case, what you're observing is how the disk buffer deals with adjusting the buffer limits to ensure it has enough space to provide certain invariants so the buffer can operate correctly.

Prior to 0.23.0

Prior to 0.23.0, the logic for the buffer looked like this:

1. round the user-specified maximum buffer size (buffer.max_size) down to the nearest multiple of the maximum data file size (128MB)
2. calculate the logical maximum buffer size as the greater of the value from step one, or 128MB

After that, we limited the buffer to storing as many outstanding events (not read/processed yet) as would fit in N bytes, where N is the value from step 2. In reality, we may have used up to 2*N bytes on disk due to the way data file deletion is handled. Crucially, however, due to the way we were rounding to a multiple of the maximum data file size, this would lead to the maximum buffer size essentially being limited to multiples of 128MB.

As you observed, even after going past ~128MB, up to your second value of ~170MB, you still were not able to buffer additional events, because the value was being rounded down to 128MB.

After 0.23.0

In 0.23.0, specifically in #13356, we changed that logic to be stricter about guarantees of maximum data file size, as well as the error message when the specified buffer.max_size is too small, which will now correctly tell you that it needs to be greater than or equal to ~256MB.

We also avoid rounding to a multiple of the maximum data file size now, so as long as the buffer maximum size is greater than 256MB, you can expect to be able to buffer up to the configured maximum buffer size -- in terms of observed data usage on disk -- while you'll observing a soft limit in terms of outstanding buffered items of the configured maximum size, minus the maximum data file size: 128MB at the absolute minimum, or N - 128MB where N is the configured maximum buffer size.

[hemanthofficial3009]

@tobz, thanks for the details. Also can you look at the case 2 and case3 mentioned above the Q1?

[tobz]

My answer is specifically about the second and third case.

[Ilmarii]

@tobz Sorry for the necroposting, but I have a very small question. Could you please explain what does "soft limit" mean. Is the useful buffer size for events now always buffer.max_size-128MB or will the full buffer.max_size be used in some cases? If the buffer is busy at buffer.max_size-128MB, can I assume that the sink has overflowed?

[tobz]

@Ilmarii No worries. I'll be working on revamping the official disk buffer docs in the next few weeks, too, which should hopefully explain this all a lot better. Definitely don't hesitate to open a new discussion, though, if you have more questions.

Overall, you can expect to have a "useful" buffer capacity between buffer.max_size - 128MB and buffer.max_size. If the reader (sink) is processing slower than writer, you could consume up to buffer.max_size initially. Let's say that max size is split across two buffer data files. Until the reader processes all events in the first file, we can't delete it from disk. This means the full file (128MB) will be counting against the used buffer capacity until it is entirely processed and then deleted.

It will still be better to look at the buffer metrics (see the bottom of the page for the docs of any sinks, which list out these metrics) to determine how many events are actually outstanding in the buffer (received - sent).