Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] MLCEngine model parameter gets converted to a tuple internally but not handled correctly #3106

Open
hank opened this issue Jan 23, 2025 · 0 comments
Labels
bug Confirmed bugs

Comments

@hank
Copy link

hank commented Jan 23, 2025

🐛 Bug

MLCEngine model parameter gets converted to a tuple internally but not handled correctly.

To Reproduce

Steps to reproduce the behavior:

  1. Create a script as shown in the code sample
  2. Run the script
from mlc_llm import MLCEngine

# Create engine
model = "./dist/prebuilt/Llama-3-8B-Instruct-q4f16_1-MLC",
model_lib="./dist/prebuilt/lib/Llama-3-8b-Instruct/Llama-3-8B-Instruct-q4f16_1-mali.so",
device="opencl"
engine = MLCEngine(model=model, model_lib=model_lib, device=device)

# Run chat completion in OpenAI API.
for response in engine.chat.completions.create(
    messages=[{"role": "user", "content": "What is the meaning of life?"}],
    stream=True,
):
    for choice in response.choices:
        print(choice.delta.content, end="", flush=True)
print("\n")

engine.terminate()

Output:

Traceback (most recent call last):
  File "/home/hank/repos/mlc-llm/test.py", line 7, in <module>
    engine = MLCEngine(model=model, model_lib=model_lib, device=device)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hank/repos/mlc-llm/python/mlc_llm/serve/engine.py", line 1466, in __init__
    super().__init__(
  File "/home/hank/repos/mlc-llm/python/mlc_llm/serve/engine_base.py", line 590, in __init__
    ) = _process_model_args(models, device, engine_config)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hank/repos/mlc-llm/python/mlc_llm/serve/engine_base.py", line 171, in _process_model_args
    model_args: List[Tuple[str, str]] = [_convert_model_info(model) for model in models]
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hank/repos/mlc-llm/python/mlc_llm/serve/engine_base.py", line 171, in <listcomp>
    model_args: List[Tuple[str, str]] = [_convert_model_info(model) for model in models]
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hank/repos/mlc-llm/python/mlc_llm/serve/engine_base.py", line 124, in _convert_model_info
    model_path = download_cache.get_or_download_model(model.model)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hank/repos/mlc-llm/python/mlc_llm/support/download_cache.py", line 226, in get_or_download_model
    if model.startswith("HF://"):
       ^^^^^^^^^^^^^^^^
AttributeError: 'tuple' object has no attribute 'startswith'

PDB run:

> /home/hank/repos/mlc-llm/python/mlc_llm/support/download_cache.py(227)get_or_download_model()
-> if model.startswith("HF://"):
(Pdb) model
('./dist/prebuilt/Llama-3-8B-Instruct-q4f16_1-MLC',)

Expected behavior

Expected the local model to be loaded correctly as this use of MLCEngine directly matches the documentation.

Environment

  • Platform (e.g. WebGPU/Vulkan/IOS/Android/CUDA): OpenCL
  • Operating system (e.g. Ubuntu/Windows/MacOS/...): Armbian
  • Device (e.g. iPhone 12 Pro, PC+RTX 3090, ...): OrangePi 5 Max 16GB
  • How you installed MLC-LLM (conda, source): source
  • How you installed TVM-Unity (pip, source): source
  • Python version (e.g. 3.10): 3.11.11
  • GPU driver version (if applicable): N/A
  • CUDA/cuDNN version (if applicable): N/A
  • TVM Unity Hash Tag: d6c1489b5d030c03fa9132accdb2d72f237f3e3e
  • Any other relevant information: N/A

Thanks! The model does work when using HF:// URLs.

@hank hank added the bug Confirmed bugs label Jan 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Confirmed bugs
Projects
None yet
Development

No branches or pull requests

1 participant