Gracefully handle IO errors from the underlying file system

[cccapon]

Cryptomator 1.4.2
OS: Windows 10 64-bit build 17134.623
Vault access: Dokany
Vault location: Google Drive File Stream v29.1.78.2103

When running Cryptomator using Dokany and when decrypting a large number of files, cryptomator fails then closes the Dokany session. All further access fails until the vault is manually closed and reopened. This failure occurs on different files at different offsets seemingly at random. It occurs more frequently when more than one vault is open. The vaults are being accessed through Google Drive File Stream and the problem seems to correlate with the load on the internet connection.

17:46:48.937 [Thread-68108] TRACE o.c.frontend.dokany.OpenFile - Reading 471040-475136 (0-1048576)
17:46:48.937 [Thread-68108] TRACE o.c.frontend.dokany.OpenFile - Reading 475136-479232 (0-1048576)
17:46:48.937 [Thread-68108] TRACE o.c.frontend.dokany.OpenFile - Reading 479232-483328 (0-1048576)
17:46:48.937 [Thread-68108] TRACE o.c.frontend.dokany.OpenFile - Reading 483328-487424 (0-1048576)
17:46:48.937 [Thread-68108] TRACE o.c.frontend.dokany.OpenFile - Reading 487424-491520 (0-1048576)
17:47:25.418 [Thread-68108] TRACE o.c.f.dokany.locks.DataRLockImpl - Released read data lock for '/Case/0024/Source7/POSITIONS - Tous les Comptes - Part 2.txt'
17:47:25.418 [Thread-68108] TRACE o.c.f.dokany.locks.PathRLockImpl - Released read path lock for '/Case/0024/Source7/POSITIONS - Tous les Comptes - Part 2.txt'
17:47:25.418 [Thread-68108] TRACE o.c.f.dokany.locks.PathRLockImpl - Released read path lock for '/Case/0024/Source7'
17:47:25.418 [Thread-68108] TRACE o.c.f.dokany.locks.PathRLockImpl - Released read path lock for '/Case/0024'
17:47:25.418 [Thread-68108] TRACE o.c.f.dokany.locks.PathRLockImpl - Released read path lock for '/Case'
17:47:25.418 [Thread-68108] TRACE o.c.f.dokany.locks.PathRLockImpl - Released read path lock for ''
17:47:25.418 [Thread-68108] DEBUG o.c.frontend.dokany.ReadWriteAdapter - (13513) readFile(): IO error while reading file /Case/0024/Source7/POSITIONS - Tous les Comptes - Part 2.txt.
17:47:25.424 [Thread-68108] DEBUG o.c.frontend.dokany.ReadWriteAdapter - Error is:
java.io.IOException: Incorrect function
	at java.base/sun.nio.ch.FileDispatcherImpl.pread0(Native Method)
	at java.base/sun.nio.ch.FileDispatcherImpl.pread(Unknown Source)
	at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
	at java.base/sun.nio.ch.IOUtil.read(Unknown Source)
	at java.base/sun.nio.ch.FileChannelImpl.readInternal(Unknown Source)
	at java.base/sun.nio.ch.FileChannelImpl.read(Unknown Source)
	at org.cryptomator.cryptofs.ChunkLoader.load(ChunkLoader.java:34)
	at org.cryptomator.cryptofs.ChunkCache$1.load(ChunkCache.java:32)
	at org.cryptomator.cryptofs.ChunkCache$1.load(ChunkCache.java:29)
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528)
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277)
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044)
	at com.google.common.cache.LocalCache.get(LocalCache.java:3952)
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
	at org.cryptomator.cryptofs.ChunkCache.get(ChunkCache.java:40)
	at org.cryptomator.cryptofs.OpenCryptoFile.read(OpenCryptoFile.java:95)
	at org.cryptomator.cryptofs.CryptoFileChannel.internalRead(CryptoFileChannel.java:184)
	at org.cryptomator.cryptofs.CryptoFileChannel.lambda$read$0(CryptoFileChannel.java:67)
	at org.cryptomator.cryptofs.CryptoFileChannel.blockingIo(CryptoFileChannel.java:283)
	at org.cryptomator.cryptofs.CryptoFileChannel.read(CryptoFileChannel.java:67)
	at org.cryptomator.frontend.dokany.OpenFile.readNext(OpenFile.java:96)
	at org.cryptomator.frontend.dokany.OpenFile.read(OpenFile.java:54)
	at org.cryptomator.frontend.dokany.ReadWriteAdapter.readFile(ReadWriteAdapter.java:324)
	at com.dokany.java.DokanyOperationsProxy$ReadFileProxy.callback(DokanyOperationsProxy.java:72)
	at jdk.internal.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at com.sun.jna.CallbackReference$DefaultCallbackProxy.invokeCallback(CallbackReference.java:520)
	at com.sun.jna.CallbackReference$DefaultCallbackProxy.callback(CallbackReference.java:551)

17:47:25.504 [Background Thread 3] TRACE o.c.frontend.dokany.ReadWriteAdapter - unmounted() is called.

[overheadhunter]

The cause is within the underlying file system implementation:

java.io.IOException: Incorrect function
	at java.base/sun.nio.ch.FileDispatcherImpl.pread0(Native Method)
	at java.base/sun.nio.ch.FileDispatcherImpl.pread(Unknown Source)
	at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
	at java.base/sun.nio.ch.IOUtil.read(Unknown Source)
	at java.base/sun.nio.ch.FileChannelImpl.readInternal(Unknown Source)
	at java.base/sun.nio.ch.FileChannelImpl.read(Unknown Source)

I.e. Google File Stream doesn't seem to behave as defined. Seems to be an upstream bug, as there is no Cryptomator code involved here.

[overheadhunter]

When you google for "incorrect function" in conjuction with IO errors, you find plenty of reports such as this post in the evernote forum or this comment in another GitHub project.

Since the root cause is within the basic I/O operations (FileChannel.read), this issue is not related to Dokany in any way. You should get the same issue with WebDAV-based drives.

At the moment we can only wait for a fix from either Google (if it is indeed an I/O bug in file stream) or the JDK (if it can be solved by a more resilient implementation on unreliable file systems).

[cccapon]

Thank you, @overheadhunter .
But, I wonder if Cryptomator's handling of a read error situation on Dokany could be improved.

Right now, the entire vault is placed in a failed state and no further file access is successful. The only solution currently known is to close then reopen the vault. But Cryptomator was designed specifically for use with these types of cloud access interfaces, and perhaps should handle failures more gracefully.

Though I have experienced occasional read errors with WebDAV, they do not seem to shut down the vault altogether as they do with Dokany.
Thanks,
Chris.

[overheadhunter]

Good point, I'll migrate this issue to the dokany-nio-adapter then.