Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optional rounding of XYShape #47

Open
OAGr opened this issue Jun 22, 2020 · 4 comments
Open

Optional rounding of XYShape #47

OAGr opened this issue Jun 22, 2020 · 4 comments

Comments

@OAGr
Copy link
Contributor

OAGr commented Jun 22, 2020

Right now distributions in the dist builder result in a lot of unnecessary precision. For instance,

"xs":[12.919944318760631,12.968993194817127,13.018042070873623,13.067090946930119,13.116139822986614,13.16518869904311,13.214237575099606,13.263286451156102,13.312335327212597,13.361384203269093,13.410433079325589,13.459481955382085,13.50853083143858,13.557579707495076,13.606628583551572,13.655677459608068,13.704726335664564,13.75377521172106,13.802824087777555,13.851872963834051,13.900921839890547,13.949970715947043,13.999019592003538,14.048068468060034,14.09711734411653,14.146166220173026,14.195215096229521,14.244263972286017,14.293312848342513,14.342361724399009,14.391410600455504,14.440459476512,14.489508352568496,14.538557228624992,14.587606104681488,14.636654980737983,14.685703856794479,14.734752732850975,14.78380160890747,14.832850484963966,14.88189936102

This takes up a lot of extra space and doesn't add anything really. There is a way on the server to request rounding, but it probably doesn't currently work because it's not supported by the DistPlus ReasonML library.

A crude method would be something like, "use n digits of precision for each number". One downside to this that it's possible that the entire distribution may be between very precise numbers, like 3.48382 to 3.48398.

Ideally there would be some way of approximating how many digits of precision is reasonable to use, or having different floats use different levels of precision.

Note that these values are sent via the GraphQL API and saved to the database, so shortness would likely help a fair bit. It also takes time to render them; I imagine improvements here could speed things up a fair bit.

@skosch
Copy link
Collaborator

skosch commented Jun 30, 2020

Honestly, I wouldn't worry about this. AFAIK JS engines use 32 bits for floats across the board, and so does PostgreSQL's real type, so there's really nothing we can do. Also, these numbers only look big when written out in decimal. In exponent/mantissa format, 4 bytes is 4 bytes regardless of whether you chop off the decimals or not.

@OAGr
Copy link
Contributor Author

OAGr commented Jun 30, 2020

I think it would make a difference for graphql/api. Sending distributions over client-server now does take a fair amount of time. I imagine that reducing this could reduce that by 1/2 or less. My impression is that when it's sent as a GraphQL param, it encodes it in JSON, and sent as basically a string, though I'm not sure.

It's also something we could test experimentally, maybe with a simpler system (like, something that just trunctates after 2 decimal places)

@skosch
Copy link
Collaborator

skosch commented Jun 30, 2020

Okay, in that case I think an easier solution is to pack the numbers into a Float32Array, and send that over the wire as UTF-8 (via TextEncoder/TextDecoder browser API). That should be quite fast.

@OAGr
Copy link
Contributor Author

OAGr commented Jun 30, 2020

Huh, that seems neat, it could be a really good fit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants