Private cloud storage

[jim-bo]

Hello,
I was wondering if there is support for private google cloud storage buckets containing ome-ngff (zarr)?

Thanks,
James

[ilan-gold]

What do you mean by support? I think you'd be looking for signed URLs. @manzt do you know if zarr.js supports url args being re-appended per-request? @jim-bo the worst case scenario would be creating your own ZarrStore to handleu URL args which you might be able to commit to zarr.js if the maintainers were interested in adding this functionality.

[manzt]

Viv supports data access via the http:// and https:// URL protocols. It doesn't not support private s3:// or gc:// out of the box, and will require exposing authenticated HTTP access to these object stores. One method of doing this authentication is with signed URLs, as @ilan-gold mentioned. You should be able to learn more in the GCS docs. Provided a signed URL, Viv should be able to load your zarr data (as Zarr.js forwards query parameters for all requests).

[jim-bo]

Thanks for the response! When I paste a signed url for a single ome-tiff into the viewer at say viv I get some CORS errors which maybe I can work through on my google cloud side. I think another issue is that the app is always looking for ".offsets.json" which doesn't exist?

For zarr formatted files, I don't think you can pre-sign an object storage prefix such as gs://bucket/pathtozarrfolder, each object needs to be signed (at least for google). I guess perhaps a proxy can be inserted between the cloud storage and the user but you loose some of the power of cloud storage. Maybe there is a way to leverage the google storage APIs and create a custom ZarrStore.

Are you aware of examples of using viv stack behind authentication and how folks have done that?

[ilan-gold]

I know KPMP has s3 pre-signed set up for OME-TIFF files: https://atlas.kpmp.org/spatial-viewer/view but I am not aware of anyone doing this for zarr stores. At HuBMAP we do something similar to signed URLs but have control over the auth service so can generate the tokens properly (and we also have a backend for creating view configurations for vitessce, which allows us to generate the URLs in the first place). But it was not built for that purpose, so I'm not aware of any redirect/proxy situations in production. The 10x folks don't seem to sign their zarr URLs and use s3.

[manzt]

For a local, simple (proxy-like) workaround, maybe have a look at https://github.com/manzt/simple-zarr-server. You can wrap an authenticated fsspec filesystem and expose it over http locally.