Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[idea] compression with schema #2

Open
cometkim opened this issue Aug 4, 2021 · 5 comments
Open

[idea] compression with schema #2

cometkim opened this issue Aug 4, 2021 · 5 comments
Labels
enhancement New feature or request

Comments

@cometkim
Copy link
Member

cometkim commented Aug 4, 2021

If the schema of the message is fixed, more efficient compression is possible.

Keep the msgpack and build a treemap for shorter key names. It will need the same schema to restore original message.

@cometkim cometkim added the enhancement New feature or request label Aug 4, 2021
@cometkim
Copy link
Member Author

cometkim commented Aug 4, 2021

This is for compression, not validation. So it will probably need to be able to integrate with external validators.

The name json (with) schema is not appropriate because it is likely to be confused with the JSON Schema spec, which contains advanced features such as references or support for dynamic messages.

Maybe a name like fixjson? or other ideas

@kyoungduck
Copy link
Collaborator

kyoungduck commented Aug 4, 2021

@cometkim
Do you think it uses extension type which in messagepack spec?

or is this package's compression logic executed and then encoded like this according to the message pack specification?

encode(json, schema);

@cometkim
Copy link
Member Author

cometkim commented Aug 4, 2021

The schema can be bound in constructors, so there is no need to pass it as an argument.

And I don't think support for extensions will be needed as the schema user can make more assumptions about the message before using. A subset would be enough

One common rule is that efficiency and flexibility are usually trade-offs.

@cometkim
Copy link
Member Author

cometkim commented Aug 4, 2021

An interesting idea is that this also can flatten nested structures beyond simply mapping key names.

Given that we currently have poor performance for complex structures, this could be a good breakthrough.

@cometkim cometkim changed the title [idea] pack with json schema [idea] compression with schema Oct 3, 2021
@cometkim
Copy link
Member Author

cometkim commented Oct 3, 2021

Another idea: LZ-based compression for large strings (str8, str16, str32...)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants