base: Add media retention policy to EventCacheStore

[zecakeh]

Allows to prevent the media cache from growing indefinitely by setting different limitations like size or last access time. I tried to select sensible defaults for those values.

Currently cleanups must be triggered manually, a future PR will add the possibility to run a regular cleanup task so they happen automatically.

[stefanceriu]

Sounds awesome but it's a biggie so I think it's safer to pass it over to somebody more familiar with the SDK, congrats Benji! 😁

[codecov]

Codecov Report

Attention: Patch coverage is 95.87629% with 20 lines in your changes missing coverage. Please review.

Project coverage is 84.33%. Comparing base (7f4e79e) to head (02a3630).
Report is 9 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/matrix-sdk-sqlite/src/event_cache_store.rs	90.32%	6 Missing ⚠️
...rix-sdk-base/src/event_cache_store/memory_store.rs	91.37%	5 Missing ⚠️
...s/matrix-sdk-base/src/event_cache_store/wrapper.rs	75.00%	5 Missing ⚠️
...rates/matrix-sdk-base/src/event_cache_store/mod.rs	88.57%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3918      +/-   ##
==========================================
+ Coverage   84.18%   84.33%   +0.15%     
==========================================
  Files         267      268       +1     
  Lines       28036    28554     +518     
==========================================
+ Hits        23601    24082     +481     
- Misses       4435     4472      +37     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[zecakeh]

Rebased on main to solve a conflict.

[bnjbvr]

Hi, thanks for the PR, since this is a large commit it might take a bit of time to get to review it. I'll try my best.

[bnjbvr]

My main feedback is that I wouldn't want to have external users have to deal with a EventCacheStoreWrapper, but rather with the EventCacheStore; so EventCacheStore (the trait) would likely need to be renamed into something else, maybe EventCacheStoreBackend or EventCacheStoreImpl, or something else that makes this clear enough?

I haven't looked at the code in details, so forwarding review to somebody else.

deferring to Ivan or Damir

[zecakeh]

I know this is a big commit, so for reviewers I am going to reiterate what I said in the Matrix room to help reviewing part by part:

I would recommend starting in matrix-sdk-base/event_cache_store with mod.rs and traits.rs. This is where the new API is actually defined with docs to explain the available settings.

Then in any order:

• The (currently named) EventCacheStoreWrapper exposes a simpler public API to users of the SDK.
• The implementations of the memory store and the SQLite store can be reviewed separately.
• Check the integration tests (this is actually the biggest part by far!).

[Hywan]

Before reviewing this PR, I'm wondering: don't we have an MSC about media retention policy? I see there is MSC1763. I don't know the state of this MSC, but I think we should link both (a media is an event after all). I don't know if the /sync can tell a client that a message is too old and should be dropped (that's possibly not the case, I didn't read the MSC). Anyway, I think we should think more globally about data retention. Why? Because with this PR, this is client-specific. A user with 2 devices would likely want to see this information “synced” between its devices. Moreover, the media can be removed from the client, but would continue to live on the server: it's pretty easy to view this as a lie from the client. Worst: if a user has 2 devices with 2 different configurations, the user can be pretty confused to see the media removed on one device, but not on another one.

What do you think about this?

[zecakeh]

This is on purpose a device-specific setting. The sole goal here is for the client or the user (if the client offers a setting for it) to be allowed to limit the size of the cache on the device.

This kind of setting can usually change from one device to the other, e.g. the limitations would be more strict on an phone where disk space is limited than on a computer with a lot of free space.