feather file has issue with compression

[behrica]

Reading

https://github.com/scicloj/scicloj.ml-tutorials/blob/main/data/tweets_sentiment.feather?raw=true

fails with

Execution error at net.jpountz.lz4.LZ4FrameOutputStream$FLG/validate (LZ4FrameOutputStream.java:362).
Dependent block stream is unsupported (BLOCK_INDEPENDENCE must be set)

I followed the setup instructions for arrow support in TMD.

[cnuernber]

Thanks, will take a look. What compressed this file?

[cnuernber]

Tracking this upstream - lz4/lz4-java#190

[cnuernber]

A temporary (hopefully) solution I am going to try is to use FFI bindings to call into the C library directly. This one is going to be a tough one as the only example of dependent frame compression I can find is the go library.

@behrica - How did you produce this file?

[cnuernber]

The point of the question is is this pathway going to be the standard pathway everyone is using or did you produce this file with some magic set of options that very few other people are going to use?

[cnuernber]

You will have to now also include jna and ensure that liblz4 is on your system which is system-dependent. My recommendation is to avoid dependent block compression on lz4 so if that was a parameter set it to false.

[behrica]

The point of the question is is this pathway going to be the standard pathway everyone is using or did you produce this file with some magic set of options that very few other people are going to use?

Not that I remember.

I think I created it in the simplest possible way from R:

x=readr::read_csv( ...)
arrow::write_feather(x , ...)

It came out while I was working on the file from #292 ,
and the above was my attempt to get the data into clojure (via feather ...)

[cnuernber]

That was my fear - then these things will be all over the place.