Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FBE use case for storing and replaying financial tickdata. using Varint for reducing size? #101

Open
alokjadhav opened this issue Sep 2, 2024 · 0 comments

Comments

@alokjadhav
Copy link

I am currently storing tick data in binary encoding format of my own (primarily using varint). I am looking into FBE as an alternative to store and fast replaying of data in my application. After playing a bit with FBE, i found that the storage requirement of FBE encoded data is must larger than my own version in order of 5x to 7x on an average. This is not useful for me as small storage is also one of the main requirements of the tick data. I am exploring my options which takes less storage space and is faster to decode and replay.
FBE is optimized for fast decoding but it is still storing native data as it is (double, float, decimal, long etc) and hence the message size is still relatively large.
By introducing Varint in encoding and decoding of the data, the message size can be reduced substantially by giving away few cycles of computation. I do feel that there could be a gain in performance due to much reduced message size as well.
I understand that in FBE, the message size is fixed and writes the length in advance and varint will defeat this purpose, but it will be much easier to just change the length after encoding the message. (Varint wil make the message size variable instead of fixed as it is now).
I am wondering if there is any plan to introduce Varint encoding and decoding to FBE, at least as an option so some users may use it and others not. I would like to do a performance analysis when I get time, wanted some input from others if anyone has explored this option or do see any benefits of this? Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant