MultiZarrToZarr does not pass target_options to the reference filesystems

[reweeden]

Hi there! I was trying to get MultiZarrToZarr to work on a set of files that have been gzip compressed. Fsspec's fsspec.open function has a compression parameter that can be passed to tell fsspec to decompress the files on the fly. I have gotten this to work using zarr files by doing something like this:

fsspec.filesystem(
    "reference",
    fo="s3://bucket/my-compressed-zarr-file.json.gz",
    target_options={'compression': 'gzip'},
    remote_protocol='s3',
    remote_options={
        'anon': False,
    },
)

I was hoping that this would also work with MultiZarrToZarr and be equally as simple as passing the target_options, however, it seems that while MultiZarrToZarr opens the files using the target_options here to get the file name:

kerchunk/kerchunk/combine.py

Line 265 in dc66b2c

for of in fsspec.open_files(self.path, **self.target_options):

It then does not pass the target options to the fsspec.filesystem call here:

kerchunk/kerchunk/combine.py

Lines 277 to 283 in dc66b2c

	fsspec.filesystem(
	"reference",
	fo=fo,
	remote_protocol=self.remote_protocol,
	remote_options=self.remote_options,
	)
	for fo in fo_list

Is this a bug or is there some reason why the target_options can't be passed to fsspec.filesystem there?

Analysis of Compressed files

From what I can tell by running this with a debugger, this is what's happening:

1. The call to open_files here succeeds:
   

   kerchunk/kerchunk/combine.py

   Line 265 in dc66b2c

   for of in fsspec.open_files(self.path, **self.target_options):

2. The call to fs.cat does not include the target_options so the file is not decompressed when read:
   

   kerchunk/kerchunk/combine.py

   Line 270 in dc66b2c

   fo_list = fs.cat(self.path)

3. The json decoding fails because it's trying to decode the compressed data and the fo_list is set to the original list of filenames.
   

   kerchunk/kerchunk/combine.py

   Line 274 in dc66b2c

   fo_list = self.path

4. The fsspec.filesystem call opens the files again, but since the target_options are not passed in, the data is not decompressed and the json decoding fails:
   

   kerchunk/kerchunk/combine.py

   Line 277 in dc66b2c

   fsspec.filesystem(

Solution

I think there are 2 additional places that the target options need to be passed in

1. In the fs.cat call via the **kwargs parameter:
   

   kerchunk/kerchunk/combine.py

   Line 270 in dc66b2c

   fo_list = fs.cat(self.path)

API Docs for fs.cat: https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem
Source code for fs.cat: https://github.com/fsspec/filesystem_spec/blob/4517882f67d635d50b54cd53fd04ee3a37b6943c/fsspec/spec.py#L844

EDIT: After trying this out, it seems that the s3fs implementation for cat_file doesn't work the way the syncronous abstract class does where the kwargs are passed to a call to fs.open. The s3fs _cat_file doesn't support kwargs at all: https://github.com/fsspec/s3fs/blob/f3f63cbfbfe71a4355abd63cafd8c678c4a5a0af/s3fs/core.py#L1113

2. In the fs.filesystem call:
   

   kerchunk/kerchunk/combine.py

   Line 277 in dc66b2c

   fsspec.filesystem(

Workaround

I believe I can work around this by opening the files myself and passing in the zarr dictionaries directly. It's just more code for me to write :)