Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regular, thread-safe exchange rate updates #60

Open
jpmckinney opened this issue Dec 1, 2021 · 0 comments
Open

Regular, thread-safe exchange rate updates #60

jpmckinney opened this issue Dec 1, 2021 · 0 comments
Milestone

Comments

@jpmckinney
Copy link
Member

jpmckinney commented Dec 1, 2021

Problems:

  • The rates are loaded from the DB at worker startup (bootstrap()). However, once loaded, they are never refreshed. So, if the worker runs for many days/weeks/months, then checks that contain recent data won’t be able to perform currency conversion.
  • import_data loads the rates into global variables, which are then accessed by all threads. As such, a solution that involves running import_data again would not be thread-safe.
  • workers/extract/kingfisher_process runs update_from_fixer_io, which adds rates to the DB. However, the other workers will have already bootstrapped the data, and so updating the rates here has no effect on active workers.

Things done so far:

  • Add an update-exchange-rates command, currently scheduled to run twice a day (can change to once), to update the rates in the DB.

Process-level options:

  1. Do nothing. Currency extrapolation already works. Workers will be restarted occasionally through normal operations.
  2. Restart workers on a regular basis (either daily or weekly) with a cronjob (docker-compose down and up -d)

Code-level options (can do one or more):

This is mainly about making the existing code thread-safe.

  1. Instead of running clear() in import_data, just assign the new values to the globals once computed.
  2. Add a way to incrementally update the in-memory cache.
  3. Auto-expire the in-memory cache (e.g. using a @cachetools decorator or a ttl on a memcache key).
  4. Adopt memcached as part of this.
@jpmckinney jpmckinney added this to the Cleanup milestone Jan 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant