[sda-download] Implement caching from the database

[pontus]

Things in the database aren't supposed to change very much (at least for published datasets) and when using a cache with a sensible ttl, as far as I can see it should roughly be problem free caching all kinds of requests from the database.

[pontus]

Most implementations seem to offer fairly similar interfaces. I've used ristretto in !NBISweden/sdafs and think it works well but there are of course others (one possible consideration could be that ristretto doesn't manage key collisions so some other solution that guarantees you can't get a response for another key than the requested one could be more suitable).

[jbygdell]

Caching is already implemented,

sensitive-data-archive/sda-download/api/sda/sda.go

Line 117 in f735b10

cache := middleware.GetCacheFromContext(c)

It's not used properly, or to be exact, the downstream file handling does a whole bunch of unnecessary stuff.

For example GetObject calls GetDatasetFileInfo which is an overly complex DB call that shouldn't be used.

[pontus]

Ok, we can rephrase it as redo :)

The caching as implemented only seems to keep track of datasets (filtered through what should be allowed for the user), which while something doesn't get us very far.

It also seems to me the current design makes life unnecessary hard.

By a quick look, I don't see anything in database that couldn't do caching on the "function" level, which seems easier than trying to implement higher up. That doesn't mean we couldn't implement other caching higher up, but it means we can add effective caching very easily.

Similarly, for the current use (making sure we don't need to call out to LS Login all the time), I believe that could also well be managed with a cache doing lookups on the token provided rather than involving the session token (if we're looking at supporting generic s3 clients, we can probably not expect cookies to be supported).

(Those two are mostly orthogonal, and again, having a higher level cache doesn't mean we can't do caching on lower levels, we e.g. don't mind if postgres caches the response as long as it's right.)