Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support more perfomant serialization format for the log #129

Open
danthegoodman1 opened this issue Sep 18, 2024 · 0 comments
Open

Support more perfomant serialization format for the log #129

danthegoodman1 opened this issue Sep 18, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@danthegoodman1
Copy link
Owner

Instead of JSONL (which can be the default), we can also support other format(s) like protobuf and flatbuffer.

Protobuf will end up being smaller files, which might actually allow it to pull ahead of flatbuffers for larger log files if more time is spent on downloading the extra flatbuffer data than deserializing protobuf.

Either way both formats should be more performant, and no reason we can't support both. Easy to dynamically support them too just by looking at the file extension (.jsonl, .proto/.pb, .flat/fp)

Will need to update https://github.com/danthegoodman1/IceDBS3Proxy/ as well to add support for the extra formats

Question: for the pb and fp formats, do we have some delimiter? Or have the delimitation inside the structs themselves? Probably easier to do the latter, since we need to read it all anyway to get the log snapshot

@danthegoodman1 danthegoodman1 added the enhancement New feature or request label Sep 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant