How external references behave in 0.19.2 and in 0.22.3?

[letalboy]

I'm using PyO3 in a protocol lib interface to extend it used to Python, since it was developed in Rust. However, my main class in python currently uses multiprocess since it needs to start a server in one process, which makes one process keep locked in that thing. The other ones perform other kind of operations using shared references using an internal system done in rust.

From that, I have two questions:

1. I need to be able to share the same object that the PyO3 is currently holding for the initialization of the socket. Does it have any internal lock state that might cause conflicts?
2. Multiprocess isolates it in a process making variables not reached in the other side which was the cause that makes we need to develop a way to share those states, if we switch to use object references in threads instead of processes in python side, we will have access to the states initialized in that references, or it will still be a new reference of that structure that is not initialized yet?

I can provide a representative model of what we have right now. It looks something like that:

But our goal here is to make something like this:

Remove the IPCs, and use a single ref. But since we are dealing with multi threading in the python side and references in the backend, I need to know if I would have some problems with GIL or other internal state inside PyO3 crate. Thanks in advance!

[davidhewitt]

Sorry for the slow reply. To be honest, this question is so high-level that it's really hard to give any concrete response. I don't see any reason why your proposed architecture wouldn't work, so I suggest you have a go and see how it works.

Throughput might be limited by the GIL unless you can run on freethreaded Python 3.13, which we will release support for soon in 0.23.

[letalboy]

That's okay, I was very busy here too. Yeah, that was my concern! The GIL blocking, I tried to do that last year and wasn't able to find a good solution with some low complexity. I also was trying to get rid of this GIL limitation together with some people here however this involves lots of different aspects, I only was able to bypass this using the RustPyNet on top of PyO3 that works fine for my case here, but It's an extra layer, and we want to remove this extra steps to make it simpler which will result in better performance in the end, I'm glad that in v0.23 it will integrate with Python 3.13, I'm very excited with that because it will solve lots of things in my projects here, and I will be able to integrate two sessions together without the constraints impose by the GIL, thanks for the feedback!

[davidhewitt]

I would strongly suggest using 0.22 and getting away from the "GIL Refs" of 0.19. The older API is less performant and does have internal state which is more likely to cause complexities.

[letalboy]

Yeah, I'll have to prioritize that update. The limitations in 0.19 are becoming an issue. I integrated it with PyO3 using a single GIL reference for multithreading, which works for now, but it’s far from ideal since I only was able to have one link to python per process, and when I initialize a new one from python side then I have to initialize every state again. I hope the updates make these scenarios easier to manage and make it easier to do parallel calls from python to rust without the GIL constraints.