Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Great piece of software! but... 128 bit and beyond? #281

Open
gitmko0 opened this issue Oct 19, 2020 · 2 comments
Open

Great piece of software! but... 128 bit and beyond? #281

gitmko0 opened this issue Oct 19, 2020 · 2 comments

Comments

@gitmko0
Copy link

gitmko0 commented Oct 19, 2020

I would like to have 128 bit bitmap and i understand that only 32bit is compatible across all programming platforms so possible to give example / suggestions on the best practises to break down 128 bit into 32bit segment?

i'm using it as a detection of bit added / "filter" and not as a bitset /mapping function. it's great to use this as "unique filter" for the compression results it gives. 5 bil items at 1meg ram total. fantastic memory savings for 64bit.

does anyone have prior example / experience in using this as a filter as i've mentioned?
thx for this software in advance.

@natalie-o-perret
Copy link

natalie-o-perret commented Dec 16, 2020

I'm thinking about a cheap workaround / solution, for example for the 128 bits (ie. 16 bytes) using two uint64 which might just do, just dunno how bad is the bottleneck of splitting a 128bits into two uint64 👩‍🚀, it's still adding a few extra cycles... I'd rather have a single data structure to achieve the same thing which would help to further compress while using less cycles.

@guymolinari
Copy link
Contributor

The roaring libraries support Bit Slice Indices (BSI). This is included as as subpackage in the GoLang version of RoaringBitmap. It could and should be ported to other languages implementing Roaring Bitmap. A BSI can now support arbitrarily large values. 128 bit values such as hashes or UUID can be stored in a BSI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants