Connectors, Sync, Async, and (maybe) dropping support for Psycopg2 & Aiopg

[ewjoachim]

I'm working on untangling the sync/async situation (happening in #753 and still very messy), using asgiref's sync_to_async and async_to_sync decorators to make it easier to call synchronous stuff in the worker (which today easily results in hard to fix errors).

I realized that psycopg3 has a much better support for async than aiopg (I get crashes trying to launch aiopg coroutines in a thread, while psycopg3 seems to handle this much better).

It's clear that finishing the work will involve making Psycopg3 the main supported connector. But I'm wondering what the best path should be for psycopg2 and aiopg connectors: we can leave them around (maybe in the contrib package, maybe with a disclaimer in the doc or a (deprecation) warning in the code), or we can remove them.

In #881, @turicas expressed that it might be complex for projects using Django at a version that doesn't support psycopg3 to use both procrastinate with psycopg3 and django with psycopg2 in the same project. I'd love to discuss this, and gather some feedback from users so as to make the best choices going forward.

So @turicas would you mind giving a little more details on why that might be complicated ? And other people, would you mind expressing your thoughts on the matter?

Pinging people who interacted with the repo recently:
@ashleyheath @turicas @paulzakin @JacobCoffee @aleksandr-shtaub @jerluc

[paulzakin]

Our two cents - it does not seem that Django supports setting two different versions of psycopg, so it probably is not very straightforward. Might require mucking around in the internals.

However, Django seems pretty intent on deprecating psycopg2 (below) and this release feels like a modernization/breaking release anyway - so we fully support the “ripping off the bandaid”. This would entail taking them out and moving everyone on to v3. Frankly, I worry about the maintenance burden of keeping all those connectors around - legacy interfaces are never fun(!) and feel the reduction in complexity would make it easier for future users.

TLDR: We back a switch to v3 as a breaking change.

https://docs.djangoproject.com/en/5.0/ref/databases/

[JacobCoffee]

I'm interested in whatever gets us to psycopg3 and/or asyncpg (the later of which is what I use exclusively)

[ewjoachim]

Note

psycopg3 support has already been added recently :)

[turicas]

If we can implement a procrastinate connector that reuses the Django postgres connection, then I think we won't have problems. But it's not something easy, since we would have to support both psycopg2 and psycopg3, depending on Django version (and how the database is configured).

[ewjoachim]

It's probably just a lack of me using Django and Procrastinate together in a real life project, but can anyone expand on why that would cause issues having psycopg2 and 3 together ? I believe they can be installed and used alongside and don't use the same namespace ?

[turicas]

Yes, we can install both and import them separately. I'm just not sure if we're going to have trouble when declaring the tasks because of the sync/async things. And since Django already manages the database connections, I think it would be good for the worker to reuse it instead of creating another one (the number of connections to the server matters).

[ewjoachim]

Ok, I've released 0.34.0b1:

• https://github.com/procrastinate-org/procrastinate/releases/tag/0.34.0b1
• https://pypi.org/project/procrastinate/0.34.0b1/

It's the code from #753 .

In the end, I kept aiopg and psycopg2, but in the contrib module. Both the PR and the release contain a description of the breaking changes, feel free to report anything unclear or broken.
Also, the code hasn't been reviewed yet (but all the tests pass with 98% coverage, which does not necessarily means it works :D ).

Any time you may be able to spend on testing will be appreciated :)

Cheers

[paulzakin]

Yup, testing now!

[ewjoachim]

https://pypi.org/project/procrastinate/0.34.0b2/ is out with a fix :)

[jerluc]

Does anyone know anything about how well asgiref.sync_to_async(...) plays with other event loops or alternative executors? In particular, I can see how this simplifies managing the differences between sync/async database connectors by offloading sync I/O to a separate thread, but this might be a different story for running jobs in workers which can be more CPU intensive (at least is the case for some of my tasks).

Should running jobs allow for a pooled process executor? Or would the suggested pattern be to launch a process executor from the task code itself?

[ewjoachim]

Other even loops

Just to make sure: we're talking about other loop implementations under asyncio, such as uvloop, and not other asyncio frameworks such as trio?

As far as I know (and I'm constantly learning) the sync_to_async code operates in a higher abstraction so it should be compatible with using a different event loop such as uvloop. Much of the progress of Django becoming compatible with async are based on sync_to_async, and the Django Asgi doc mentions using Uvicorn, so I think we're safe here.

It could be worth adding tests for that.

Passing a different executor, though, is not supported by sync_to_async.

Maybe at some point we might consider this for Procrastinate, but no efforts have been made in the lib so far to make it multi-process friendly. I'd tend to say the easiest way to parallelize CPU-intensive work with Procrastinate is to launch multiple workers per machine using your process manager (be it systemd, k8s, etc). Of course, you could also make a top-level python module that would create n subprocesses and call run_worker_async in each worker.

Of course, this may have implications in terms of RAM and number of PG connections.

[jerluc]

As far as I know (and I'm constantly learning) the sync_to_async code operates in a higher abstraction so it should be compatible with using a different event loop such as uvloop. Much of the progress of Django becoming compatible with async are based on sync_to_async, and the Django Asgi doc mentions using Uvicorn, so I think we're safe here.

Ah okay great, thanks @ewjoachim. Yes our current production deployments of Procrastinate all use uvloop, and looking a bit deeper into asgirefs implementation of sync_to_async shows that indeed it's not making any assumptions about implementation.

I'd tend to say the easiest way to parallelize CPU-intensive work with Procrastinate is to launch multiple workers per machine using your process manager (be it systemd, k8s, etc).

👍 yeah this is current approach we're taking at the moment, I just wasn't sure if the change to leverage a threaded executor would impact (it sounds like it doesn't).

[ewjoachim]

👍 yeah this is current approach we're taking at the moment, I just wasn't sure if the change to leverage a threaded executor would impact (it sounds like it doesn't).

Haha, honestly, as long as I can avoid introducing multiprocessing issues to Procrastinate, I'll leave it this way. I'm not smart enough for this. I thought I understood async and I'm paying the price, I'll think twice before opening this kind of can of worms again :)

[ewjoachim]

Release is planned for sometime in the next 24h :)

[ewjoachim]

Procrastinate 1.0.0 is out :)
https://pypi.org/project/procrastinate/1.0.0/