[Router] Implement router backbone

[ByronHsu]

Motivation

The first step of #1732. See full context in the design doc
This PR implements:

1. Two-level HTTP architecture
2. Periodic Garbage Collection (GC)
3. Dynamic Scaling
4. Basic request routing (round robin and random)

Modifications

1. Create router/ folder under srt
2. Implement router and http server
3. Handle GC and dynamic scaling

Usage

1. Launch workers

export CUDA_VISIBLE_DEVICES=0; python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --host 127.0.0.1 --port 9000
export CUDA_VISIBLE_DEVICES=1; python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --host 127.0.0.1 --port 9002

2. Launch a router

Note: for multi-node, replace worker-urls with accessible http endpoint

python -m sglang.srt.router.launch_router --host 127.0.0.1 --port 8080 --policy round_robin --worker-urls http://127.0.0.1:9000 http://127.0.0.1:9002

3. Send a curl request to the router

curl -X POST http://127.0.0.1:8080/generate  -H "Content-Type: application/json" -d '{
    "text": "Once upon a time,",
    "sampling_params": {
      "max_new_tokens": 16,
      "temperature": 0
    }
  }'

Tests

1. Add a ci test to test router with DP = 2, ensuring gsm8k score is reasonable.
2. Do a manual test to ensure the failure of a worker does not crash the router

TODO

1. Implement cache-aware routing (collab w @Ying1123 @yichuan520030910320 )
2. Profile and improve performance
3. Add tests for fault tolerance (like killing a worker in the middle), but make sure the test is not flaky on CI
4. Add complete support of SGLang compatible apis (like /health_generate etc)
5. (stretched) Implement router in another language to reduce overhead (e.g. rust)

Checklist

• Format your code according to the Contributor Guide.
• Add unit tests as outlined in the Contributor Guide.
• Update documentation as needed, including docstrings or example tutorials.

[ByronHsu]

will revert once review is done. this is for fast ci iteration

[merrymercy]

this has been disabled

[ByronHsu]

ditto

[ByronHsu]

ditto

[merrymercy]

Per our discussion offline, we would like to go with rust. Should we still merge this?

[ByronHsu]

Due to the low performance of 2-level HTTP with python server, i will try to re-implement in rust. Closing the issue for now!