You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Instead of JSONL (which can be the default), we can also support other format(s) like protobuf and flatbuffer.
Protobuf will end up being smaller files, which might actually allow it to pull ahead of flatbuffers for larger log files if more time is spent on downloading the extra flatbuffer data than deserializing protobuf.
Either way both formats should be more performant, and no reason we can't support both. Easy to dynamically support them too just by looking at the file extension (.jsonl, .proto/.pb, .flat/fp)
Question: for the pb and fp formats, do we have some delimiter? Or have the delimitation inside the structs themselves? Probably easier to do the latter, since we need to read it all anyway to get the log snapshot
The text was updated successfully, but these errors were encountered:
Instead of JSONL (which can be the default), we can also support other format(s) like protobuf and flatbuffer.
Protobuf will end up being smaller files, which might actually allow it to pull ahead of flatbuffers for larger log files if more time is spent on downloading the extra flatbuffer data than deserializing protobuf.
Either way both formats should be more performant, and no reason we can't support both. Easy to dynamically support them too just by looking at the file extension (.jsonl, .proto/.pb, .flat/fp)
Will need to update https://github.com/danthegoodman1/IceDBS3Proxy/ as well to add support for the extra formats
Question: for the pb and fp formats, do we have some delimiter? Or have the delimitation inside the structs themselves? Probably easier to do the latter, since we need to read it all anyway to get the log snapshot
The text was updated successfully, but these errors were encountered: