Set up wireguard network for task worker from manager container

[elfkuzco]

Rationale

Set up wireguard network for task woker from manager container

Changes

• Make scheduler interval to be in seconds for finer control
• Seperate manager and task code
• Create top command mirrors-qa-backend which takes update-mirrors, create-worker, scheduler as subcommands. E.g, mirrors-qa-backend update-mirrors updates the mirrors list.
• Add curl in backend Dockerfile. Needed to enable healthcheck as worker_manager often starts when mirrors are being updated. To be removed as things progress.
• Set up worker manager which sets up network for task worker and cleans up resources on exit. This is more like a proof of concept as the task worker just queries a mullvad endpoint and writes results to a mount point set up the manager.
• Add documentation on how to set up services in dev folder.
• Add seperate CI/CD jobs for the task_worker

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 8.33333% with 77 lines in your changes missing coverage. Please review.

Project coverage is 69.62%. Comparing base (9074157) to head (2abcb0d).

Files	Patch %	Lines
backend/src/mirrors_qa_backend/entrypoint.py	0.00%	29 Missing ⚠️
backend/src/mirrors_qa_backend/cli/worker.py	0.00%	26 Missing ⚠️
backend/src/mirrors_qa_backend/cli/mirrors.py	0.00%	15 Missing ⚠️
...ckend/src/mirrors_qa_backend/settings/scheduler.py	0.00%	4 Missing ⚠️
backend/src/mirrors_qa_backend/cli/scheduler.py	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #21      +/-   ##
==========================================
- Coverage   76.52%   69.62%   -6.91%     
==========================================
  Files          25       27       +2     
  Lines         656      711      +55     
  Branches       57       64       +7     
==========================================
- Hits          502      495       -7     
- Misses        140      202      +62     
  Partials       14       14              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[rgaudin]

Thank you ; that's quite the PR!

I did not actually run the code ; will do in a later review.

---

When refactoring, it's better to use git mv first, commit and then do your edits (I see you did it for unedited ones). Here we have new files (that are not new) and deleted files (that have not been deleted).
It's OK for this one, project is still young and history is not that important on those files.

For the backend CLI scripts, you have coupled the implementation with the argument parsing. update_mirrors and others now take a Namespace as args.
I think we should have a tiny entrypoint that takes the namespace and calls a business-only function with appropriate, clearly defined/types/exposed params (if any).

manager must be hardened. At no point should anything raise an unexpected exception that would have it halt ; because it's unsupervised. This is verbose and annoying and that's one of the reason we want it as dumb as possible.

docker commands that queries the API should update values first. For instance, if you query status, attrs, etc, those are data that change overtime but which are cached in dockerpy. You want to make sure you reload() before reading this

[elfkuzco]

Thanks for your feedback on the initial changes. A lot of the changes have made the code better, I think. I have marked some of the changes as resolved and the ones I left unresolved are cases where I did things a bit differently.
You can review and let me know what you think.

In summary, the things that are a bit new are:

• use Docker profiles to conditionally start the worker-manager
• make most of the time env variables to be parsed using humanfriendly
• catch all exceptions in the run method of the WorkerManager
• improve dev README

[rgaudin]

That's very good ; thank you