Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement xxhash algorithms as part of the expression API #14044

Open
HectorPascual opened this issue Jan 8, 2025 · 2 comments
Open

Implement xxhash algorithms as part of the expression API #14044

HectorPascual opened this issue Jan 8, 2025 · 2 comments
Labels
enhancement New feature or request

Comments

@HectorPascual
Copy link

HectorPascual commented Jan 8, 2025

Is your feature request related to a problem or challenge?

I am currently in need of using datafusion SQL query engine (through delta-rs merge operation) to hash with a specific hashing algorithm.

https://xxhash.com/
https://github.com/Cyan4973/xxHash
Rust implementation : https://crates.io/crates/twox-hash

Describe the solution you'd like

I'd like the hashing functions xxhash32 and xxhash64 to be part of datafusion expression API so the functions can be used in SQL context for hashing data.

@HectorPascual HectorPascual added the enhancement New feature or request label Jan 8, 2025
@alamb
Copy link
Contributor

alamb commented Jan 8, 2025

You could also implement this as a user defined function perhaps

@HectorPascual
Copy link
Author

Hi, thanks for the reply.

That is true, although, my concern comes from this other issue raised in delta-rs project (link below), since I need to use this hash operation in a Delta merge operation.

Unless I change the codebase for the delta-rs project, I'd be unable to create an UDF (not sure if there's a way to create and register it in the delta-rs context in python, sounds complicated)

delta-io/delta-rs#3105 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants