You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
1. I have searched related issues but cannot get the expected help.
2. The bug has not been fixed in the latest version.
3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
5. Please use English, otherwise it will be closed.
Describe the bug
When I use v0.3.3.post1-cu121-srt docker image to deploy qwen2-72B model, it crashes after a few requests served. I think the stability is still a serious problem. This crash also occurs on v0.3.0 version, also both on A100 and A800 hardware.
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
Similar symptoms were mentioned in #1270 as well. There are several tips that help to alleivate the stability issue for now.
Try to disable custom all reduce by --disable-custom-all-reduce. Custom allreduce is mostly for small batch sizes so it may not affect the throughput a lot.
Try --disable-cuda-graph-padding. Again, this won't hurt peak throughput.
Similar symptoms were mentioned in #1270 as well. There are several tips that help to alleivate the stability issue for now.
Try to disable custom all reduce by --disable-custom-all-reduce. Custom allreduce is mostly for small batch sizes so it may not affect the throughput a lot.
Try --disable-cuda-graph-padding. Again, this won't hurt peak throughput.
Try NCCL_ALGO=Tree
Try to match cuda driver versions.
I've tested your method on the latest docker image v0.3.3.post1-cu121-srt, the following is the results:
server side adds --disable-custom-all-reduce, also hangs, very quickly
server side adds --disable-cuda-graph-padding, server errors: "sglang | [23:56:42 TP4] Detected errors during sampling! NaN in the probability."
server add NCCL_ALGO: Tree, server errors: "sglang | [23:56:42 TP4] Detected errors during sampling! NaN in the probability."
It seems the second method may disrupt the machine cuda environment, so the third test also failed. I have to hardly reboot the machine to recovery it.
Checklist
Describe the bug
When I use v0.3.3.post1-cu121-srt docker image to deploy qwen2-72B model, it crashes after a few requests served. I think the stability is still a serious problem. This crash also occurs on v0.3.0 version, also both on A100 and A800 hardware.
Reproduction
server side:
After I do some load testing with heavy requests, it crashes.
The screen logs:
Environment
python3 -m sglang.check_env
Python: 3.10.14 (main, Apr 6 2024, 18:45:05) [GCC 9.4.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA A800-SXM4-40GB
GPU 0,1,2,3,4,5,6,7 Compute Capability: 8.0
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.4, V12.4.131
CUDA Driver Version: 535.86.10
PyTorch: 2.4.0+cu121
sglang: 0.3.0
flashinfer: 0.1.6+cu124torch2.4
triton: 3.0.0
transformers: 4.44.2
requests: 2.32.3
tqdm: 4.66.5
numpy: 1.26.4
aiohttp: 3.10.5
fastapi: 0.112.2
hf_transfer: 0.1.8
huggingface_hub: 0.24.6
interegular: 0.3.3
packaging: 24.1
PIL: 10.4.0
psutil: 6.0.0
pydantic: 2.8.2
uvicorn: 0.30.6
uvloop: 0.20.0
zmq: 26.2.0
vllm: 0.5.5
multipart: 0.0.9
openai: 1.43.0
anthropic: 0.34.1
NVIDIA Topology:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV8 NV8 NV8 NV8 NV8 NV8 NV8 PXB NODE SYS SYS NODE 0-27,56-83 0 N/A
GPU1 NV8 X NV8 NV8 NV8 NV8 NV8 NV8 PXB NODE SYS SYS NODE 0-27,56-83 0 N/A
GPU2 NV8 NV8 X NV8 NV8 NV8 NV8 NV8 NODE PXB SYS SYS NODE 0-27,56-83 0 N/A
GPU3 NV8 NV8 NV8 X NV8 NV8 NV8 NV8 NODE PXB SYS SYS NODE 0-27,56-83 0 N/A
GPU4 NV8 NV8 NV8 NV8 X NV8 NV8 NV8 SYS SYS PXB NODE SYS 28-55,84-111 1 N/A
GPU5 NV8 NV8 NV8 NV8 NV8 X NV8 NV8 SYS SYS PXB NODE SYS 28-55,84-111 1 N/A
GPU6 NV8 NV8 NV8 NV8 NV8 NV8 X NV8 SYS SYS NODE PXB SYS 28-55,84-111 1 N/A
GPU7 NV8 NV8 NV8 NV8 NV8 NV8 NV8 X SYS SYS NODE PXB SYS 28-55,84-111 1 N/A
NIC0 PXB PXB NODE NODE SYS SYS SYS SYS X NODE SYS SYS NODE
NIC1 NODE NODE PXB PXB SYS SYS SYS SYS NODE X SYS SYS NODE
NIC2 SYS SYS SYS SYS PXB PXB NODE NODE SYS SYS X NODE SYS
NIC3 SYS SYS SYS SYS NODE NODE PXB PXB SYS SYS NODE X SYS
NIC4 NODE NODE NODE NODE SYS SYS SYS SYS NODE NODE SYS SYS X
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
NIC Legend:
NIC0: mlx5_2
NIC1: mlx5_3
NIC2: mlx5_4
NIC3: mlx5_5
NIC4: mlx5_bond_0
ulimit soft: 1048576
The text was updated successfully, but these errors were encountered: