[Feature] Make vLLM optional in model code

[ByronHsu]

Motivation

This is a tracker of removing vLLM dependencies in general model code (not considering quantization). This is our current import from vLLM, and we want to remove all them.

from vllm.config import CacheConfig
from vllm.distributed import get_tensor_model_parallel_world_size
from vllm.model_executor.layers.rotary_embedding import get_rope
from vllm.model_executor.layers.vocab_parallel_embedding import (
   ParallelLMHead,
   VocabParallelEmbedding,
)

Tracker

• Remove CacheConfig: [1/N] Remove CacheConfig import in all model files #1658
• Remove RoPE: Support vLLM-style rope flashinfer-ai/flashinfer#530
• Remove get_tensor_model_parallel_world_size
• Remove ParallelLMHead:
• Remove VocabParallelEmbedding: