[Usage]: Distributed Inference Without Docker.

[Abdulhanan535]

Your current environment

The output of `python env.py`

Not needed

How would you like to use Aphrodite?

I want to run a single model on multiple machines in a network but the issue is i can't install docker on them any way i can use it without the docker?
the script of distributed inference on multi node (uses docker).

https://aphrodite.pygmalion.chat/pages/usage/distributed.html

[AlpinDale]

Yes you can, it's just much more difficult without docker. Are the machines physically connected to each other? If they're not, I'd recommend not doing it - it would be slow to the point of being unusable. It's likely everything you're running will timeout before a successful all-reduce/gather.

[Abdulhanan535]

Yes you can, it's just much more difficult without docker. Are the machines physically connected to each other? If they're not, I'd recommend not doing it - it would be slow to the point of being unusable. It's likely everything you're running will timeout before a successful all-reduce/gather.

they're on a LAN network.

[Abdulhanan535]

can you guide? kinda confused. (tried llamacpp it's good like getting 5t/s on 70b but i need more speed.)