Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

document for huggingface(vllm) servingruntime for multi-node #402

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

document for huggingface(vllm) servingruntime for multi-node #402

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
4e7cca4
document for huggingface(vllm) servingruntime for multi-node
Jooho Oct 8, 2024
1860f33
update doc based on the change of logic
Jooho Oct 17, 2024
9b4d994
add consideration
Jooho Oct 22, 2024
a5a8661
follow up for comments
Jooho Oct 31, 2024
6e4a702
follow up comments
Jooho Nov 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions docs/modelserving/v1beta1/llm/huggingface/multi-node/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,21 @@ This guide provides step-by-step instructions on setting up multi-node and multi

You must have **exactly one head pod** in your setup. The replica count for this head pod can be adjusted using the `min_replicas` or `max_replicas` settings in the `InferenceService (ISVC)`. However, creating additional head pods will cause them to be excluded from the Ray cluster, resulting in improper functioning. Ensure this limitation is clearly documented.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a way to limit this in the code?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is kind of generic features so adding checking logic would not be simple.


### Consideration

Using the multi-node feature likely indicates that you are trying to deploy a very large model. In such cases, you should consider increasing the `initialDelaySeconds` for the `livenessProbe`, `readinessProbe`, and `startupProbe`. The default values may not be suitable for your specific needs.

~~~
..
livenessProbe:
failureThreshold: 2
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
initialDelaySeconds: 10
..
~~~
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe add a note on how and where to do it?
If using custom runtime or isvc.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this needs to be set in servingruntime. like huggingfaceserver-multinode
I add this simply to the doc.


## WorkerSpec and ServingRuntime

To enable multi-node/multi-GPU inference, `workerSpec` must be configured in both ServingRuntime and InferenceService. The `huggingface-server-multinode` `ServingRuntime` already includes this field and is built on **vLLM**, which supports multi-node/multi-GPU feature. Note that this setup is **not compatible with Triton**.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
To enable multi-node/multi-GPU inference, `workerSpec` must be configured in both ServingRuntime and InferenceService. The `huggingface-server-multinode` `ServingRuntime` already includes this field and is built on **vLLM**, which supports multi-node/multi-GPU feature. Note that this setup is **not compatible with Triton**.
To enable multi-node/multi-GPU inference, the `workerSpec` field must be configured in both ServingRuntime and InferenceService. The `huggingface-server-multinode` `ServingRuntime` already includes this field and is based on **vLLM**, which supports multi-node/multi-GPU feature. Note that this setup is **not compatible with Triton**.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe original comments are simpler and make sense. So I will keep it.

Expand Down