why max_num_batched_tokens must be <= 65528 when LoRA is enabled? can it be extended longer

[junior-zsy]

When I used LoRA, I found that Due to limitations of the custom LoRA CUDA kernel, max_num_batched_tokens must be <= 65528 when LoRA is enabled. ，I would like to know the specific reason for this. Can it be extended for a longer period of time? I have a need in this area. Do you have any specific ideas for me? Thank you @Yard1 @simon-mo

[jeejeelee]

Version 0.5.5 has already removed this restriction, see: #7288.

[brando90]

@jeejeelee what version of pytorch and flash-attn do you have?

[brando90]

this worked for me

- Recommended vllm (it works with lora adapters)

pip install torch==2.4.0
pip install vllm==0.5.5

make sure the local libs are installed

pip install -e ~/ultimate-utils --no-deps

make sure you really have the right torch and vllm version

pip install torch==2.4.0
pip install vllm==0.5.5

test vllm lora

save env now

pip freeze > ~/ultimate-utils/requirements.txt