on file upload, checksum is not stored in iCAT

[carsten-jahn]

I'm using cloud browser 1.0.1 to upload to iRODS 4.1.8. The core.re rules, e.g. acPostProcForPut, were not modified (i.e. they do not include a call to msiSysChksumDataObj).

I thought with a default jargon.properties being present in the iRODS cloud browser class path (from WEB-INF/lib/jargon-core-4.0.3.1-SNAPSHOT.jar), the existing setting transfer.computeandvalidate.checksum=true would cause a client (=Jargon) checksum to be transferred to the iRODS server during upload. I also assume that iRODS will put it in the catalog then, visible with ils -L. But I don't see a checksum for files uploaded with iRODS cloud browser. I was expecting a behavior similar to iput -K, which does the end-to-end check and also stores the checksum in iCAT.

Is this an issue in iRODS cloud browser, Jargon, or in my understanding? 😃

[michael-conway]

Hi Jon, that setting is used and put/get transfers, and uploads use the
distinct streaming i/o protocol. That's why it didn't compute. However,
your observation makes me think it would be a nice feature to add to
streaming i/o and I can add that capability.

[carsten-jahn]

Hi Mike, sorry I don't really understand this... is this setting just applicable in certain situations? Thanks!

[michael-conway]

Yes, it was applicable to the put/get transfers (analogous to iput and
iget). In iRODS the streaming i/o (open stream, write to stream, close
stream) is a distinct code path and doesn't have checksum 'knobs' as part
of that protocol.

However, I can observe the setting by wrapping a streaming i/o operation
from cloud browser with checksum logic, and I think that's a nice feature.
We'll also be adding a compute checksum action in the browser itself in an
upcoming feature release.

[carsten-jahn]

Thanks, now I get it. The cloud browser uses streaming API because it already has a stream, from the web upload.

Of course it would be nice to improve this for the cloud browser as you suggest - but isn't it possible to checksum a stream just as well directly in the Jargon API? And the iRODS server feature that receives a stream could checksum what it received as well. I don't know if it does though. So if its possible, it would be great to have this directly in Jargon. If not, please add a small "best practice" documentation snippet about the wrapping approach for cloud browser, so that other developers can use it as well.

[michael-conway]

Yes, checksumming of a stream as it 'passes' is in Java already, so we
just need to add a hook to use that. We can turn this issue into that
feature and do it down in Jargon, that was exactly my thought. Your
original point is correct that the behavior of streams versus put/get
violates the principle of least surprise.

MC

[carsten-jahn]

Sounds great! Can we move the issue into the Jargon project then?

[michael-conway]

yes I will do that this morning

[carsten-jahn]

created issue: DICE-UNC/jargon#194