You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently we compute a checksum using single thread, which is not efficient
for reading data and for computing a checksum.
Using multiple NBD connections, we can read data up to 200% faster. We see
this speed up for upload and download disks.
Using N threads, we can compute checksum up to N times faster. This is easy
to scale, but the checksum must be changed to compute checksum per segment
so we can compute segment checksum in parallel. The final checksum will be
the checksum of the segment checksums.
Based on POC[1] we can have 320-350% speedup with 6 workers for real images.
Choosing number of workers:
add configuration for maximum number of workers.
daemon should limit maximum number based on number of cores and
configuration.
The best way is to run the blksum tool with a path to the image,
or with a NBD URL (if qemu-nbd is already exposing the image).
But imageio is not the place to run external tools, vdsm is.
Currently we compute a checksum using single thread, which is not efficient
for reading data and for computing a checksum.
Using multiple NBD connections, we can read data up to 200% faster. We see
this speed up for upload and download disks.
Using N threads, we can compute checksum up to N times faster. This is easy
to scale, but the checksum must be changed to compute checksum per segment
so we can compute segment checksum in parallel. The final checksum will be
the checksum of the segment checksums.
Based on POC[1] we can have 320-350% speedup with 6 workers for real images.
Choosing number of workers:
configuration.
[1] https://gerrit.ovirt.org/c/ovirt-imageio/+/113379
Since this was opened we have a better way to create checksums:
https://gitlab.com/nirs/blkhash
The best way is to run the blksum tool with a path to the image,
or with a NBD URL (if qemu-nbd is already exposing the image).
But imageio is not the place to run external tools, vdsm is.
Original bug: https://bugzilla.redhat.com/1974117
The text was updated successfully, but these errors were encountered: