FR: Support for LZ4?

[gcflymoto]

First of all thank you for such an excellent library which provides our engineering community with simple, transparent, performant, and pythonic library to handle all kinds of compressed and uncompressed files. It reduces the complexity by 10x of handling all these formats.

We've recently been deploying LZ4, especially since the latest release 1.10.0 supports multi-threaded compression and decompression

Unfortunately https://github.com/python-lz4/python-lz4 development seems to have ceased.

Is there any possibility for adding python-zstandard style library? or support for invoking "lz4" CLI for now?

[marcelm]

Thanks for the kind words!

My first thought is that adding LZ4 sounds reasonable. Are you interested in submitting a PR? A couple of thoughts in random order:

• Spawning lz4 as a process would be easiest for now.
• Using python-lz4 as a library should also be needed, but the question is whether it should be a required or optional dependency. I don’t like that the lz4 wheel has 1.3 MB. This seems to be due to debugging info being included and could be reduced to around 100 kB. The Conda packages for lz4 itself plus the Python bindings (lz4-c and lz4) total <200 kB.
• python-lz4 apparently not being maintained is a bummer.
• Maybe the tests need some refactoring to avoid running too many of them?

@rhpvorderman What do you think?