Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use multiple threads to compute checksum #66

Open
ahadas opened this issue May 10, 2022 · 0 comments
Open

Use multiple threads to compute checksum #66

ahadas opened this issue May 10, 2022 · 0 comments
Labels
enhancement Enhancing the system by adding new feature or improving performance or reliability performance

Comments

@ahadas
Copy link
Member

ahadas commented May 10, 2022

Currently we compute a checksum using single thread, which is not efficient
for reading data and for computing a checksum.

Using multiple NBD connections, we can read data up to 200% faster. We see
this speed up for upload and download disks.

Using N threads, we can compute checksum up to N times faster. This is easy
to scale, but the checksum must be changed to compute checksum per segment
so we can compute segment checksum in parallel. The final checksum will be
the checksum of the segment checksums.

Based on POC[1] we can have 320-350% speedup with 6 workers for real images.

Choosing number of workers:

  • add configuration for maximum number of workers.
  • daemon should limit maximum number based on number of cores and
    configuration.
  • client should consider number of cores

[1] https://gerrit.ovirt.org/c/ovirt-imageio/+/113379

Since this was opened we have a better way to create checksums:
https://gitlab.com/nirs/blkhash

The best way is to run the blksum tool with a path to the image,
or with a NBD URL (if qemu-nbd is already exposing the image).
But imageio is not the place to run external tools, vdsm is.

Original bug: https://bugzilla.redhat.com/1974117

@nirs nirs added enhancement Enhancing the system by adding new feature or improving performance or reliability performance labels May 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Enhancing the system by adding new feature or improving performance or reliability performance
Projects
None yet
Development

No branches or pull requests

2 participants