Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: parallelizing VID polynomial evaluation #496

Merged
merged 6 commits into from
Feb 23, 2024
Merged

feat: parallelizing VID polynomial evaluation #496

merged 6 commits into from
Feb 23, 2024

Conversation

mrain
Copy link
Contributor

@mrain mrain commented Feb 23, 2024

Description

closes: #494

On my local laptop (M1 pro), with payload size 1 << 25, payload_chunk_size = 256 and num_storage_nodes = 512.
Running with RAYON_NUM_THREADS=8.

Previous result for polynomial evaluation:

··Start:   encode payload bytes into polynomials
··End:     encode payload bytes into polynomials ...................................899.311ms
··Start:   compute all storage node evals for 4229 polynomials with 256 coefficients
··End:     compute all storage node evals for 4229 polynomials with 256 coefficients 1.026s

Now we have:

··Start:   encode payload bytes into polynomials
··End:     encode payload bytes into polynomials ...................................168.235ms
··Start:   compute all storage node evals for 4229 polynomials with 256 coefficients
··End:     compute all storage node evals for 4229 polynomials with 256 coefficients 80.797ms

Before we can merge this PR, please make sure that all the following items have been
checked off. If any of the checklist items are not applicable, please leave them but
write a little note why.

  • Targeted PR against correct branch (main)
  • Linked to GitHub issue with discussion and accepted design OR have an explanation in the PR that describes this work.
  • Wrote unit tests
  • Updated relevant documentation in the code
  • Added a relevant changelog entry to the Pending section in CHANGELOG.md
  • Re-reviewed Files changed in the GitHub PR explorer

Copy link
Contributor

@ggutoski ggutoski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks great! Just a nit about trait bounds

primitives/src/vid/advz.rs Outdated Show resolved Hide resolved
primitives/src/vid/advz.rs Outdated Show resolved Hide resolved
primitives/src/vid/advz.rs Outdated Show resolved Hide resolved
primitives/src/vid/advz/payload_prover.rs Outdated Show resolved Hide resolved
primitives/src/vid/advz/precomputable.rs Outdated Show resolved Hide resolved
@mrain
Copy link
Contributor Author

mrain commented Feb 23, 2024

Further improves bytes_to_polys, right now

··Start:   encode payload bytes into polynomials
··End:     encode payload bytes into polynomials ...................................44.674ms
··Start:   compute all storage node evals for 4229 polynomials with 256 coefficients
··End:     compute all storage node evals for 4229 polynomials with 256 coefficients 82.175ms

Copy link
Contributor

@ggutoski ggutoski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LFG! 🚀

@mrain mrain merged commit 41fe9e5 into main Feb 23, 2024
5 checks passed
@mrain mrain deleted the cl/vid-parallel branch February 23, 2024 19:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add parallelism for VID polynomial evaluations
2 participants