Storing pictures and other files (feat. encryption)

[tschbc]

Hello!

I'm currently working on a mobile app that uses an encrypted Realm DB and we're now looking at what our options are for storing files (<= 5MB), like pictures or PDFs.

The Realm DB mainly handles transitory data—anything the user wants to actually save will be uploaded to a server, so we aren't concerned with long-term storage.

Right now, I see two options:

1. Store file in Realm

• Pros:
  • Kept as close as possible to other relevant information
  • Takes advantage of encryption/decryption that is already set up
• Cons:
  • Goes against basic database storage paradigms
  • May (probably will) affect performance negatively

2. Store file in file system and store its filepath in Realm

• Pros:
  • (Probably) much better performance by just holding filepaths instead of files themselves
• Cons:
  • Files could be moved/deleted externally, "breaking its logical link" to a saved filepath
  • Need to set up a new encryption/decryption mechanism to support files

---

Questions:

1. What is the actual performance impact of storing files in an encrypted Realm database?
   • If terrible, at what file size does it become terrible? 200kb? 1mb?
   • Does its "terribleness" instead scale with size?
   • Does it scale with the number of files saved as well?
2. What is the recommended way to handle this from the Realm team?
   • If there is none, does the Realm team know what other projects are doing to handle this?
3. Are there any suggestions for manually encrypting/decrypting files in .NET?

[nirinchev]

Are you using Atlas Device Sync or are you just using the local database with a custom server? Asking mainly because we have a project coming that will simplify storing binary data by uploading the structured data to Atlas while at the same time uploading the binary files to S3/whatever other storage server you want to use.

[tschbc]

We're using the local database with a custom server, no Atlas integration.

[nirinchev]

Gotcha, we're still in the design phase of this project, so not sure how exactly that'll look like for the local database, but we'll use the input here in our considerations.

In terms of the original questions, we generally recommend storing outside of the file as having large blobs will slow things down. Realm files are memory-mapped and we'll need to load more data into memory as there's no way to distinguish between binary and structured data since they are interwoven with one another. There's no cutoff for when performance becomes terrible and instead, the larger the file becomes, the worse performance is. It gets really bad if the files become larger than what can fit into memory, because then we'll need to do pagination and hit the disk much more often.

In terms of the cons of storing them outside the file, I'd say Files could be moved/deleted externally shouldn't be a major concern as technically, the Realm file itself can be moved/manipulated from the outside, so while individual files are indeed easier to mess around with, if someone wanted to break their app, they could achieve that, even if the files were stored in the database.,

[tschbc]

Thank you for the insight, and fair points on the external factors.

My main goal / ideal state is to re-use the existing Realm encryption that I have set up to protect blob storage, rather than rolling my own mechanism.

For now, I could probably refer to how Realm's encryption is implemented, then just re-use the key for encrypting the files. What are your thoughts?

[nirinchev]

Realm uses standard AES256 encryption, where the first 32 bytes of the supplied encryption key are used for the AES encryption, while the second 32 bytes are used for HMAC integrity validation.

Here's a straightforward tutorial for implementing the encryption part in C#: https://www.jasonpenniman.com/aes256-encryption-csharp - you'll need to change it a bit so that you work with byte[]/Stream rather than strings, but the core concepts should be fairly easy to translate.

[tschbc]

Appreciate the info and link.