Unsetting litellm_params from the client per request?

[Manouchehri]

On the client side, how would I "unset" seed here?

model_list:
  - model_name: gpt-4-0125-preview
    litellm_params:
      model: azure/gpt-4-0125-preview
      api_key: os.environ/AZURE_API_KEY_EASTUS
      api_version: "2024-03-01-preview"
      api_base: "https://aimoda-is-epic.openai.azure.com"
      seed: 1337

i.e. instead of 4242, for this specific one example, I don't want there to be any seed sent to the upstream LLM.

curl -v "${OPENAI_API_BASE}/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "gpt-4-0125-preview",
    "seed": 4242,
    "messages": [
      {
        "role": "user",
        "content": "what is your name"
      }
    ]
  }'

[krrishdholakia]

hey @Manouchehri missed this

• if the upstream llm doesn't already support this param, it'll be dropped via litellm_settings::drop_params: true

--
We could also let you enforce only the param values you set in config are used and not the ones dynamically passed in

Would that work?

[zazer0]

if the upstream llm doesn't already support this param

Could you clarify how this is defined? Especially for custom endpoints, there's no way of clarifying what param(s) are/aren't appropriate.

(Though, custom/ endpoints currently give {"error":{"message":"litellm.APIConnectionError: litellm.APIConnectionError: Expecting value: line 1 column 1 (char 0)","type":null,"param":null,"code":500}}% anyways).

Is it possible to make the model field optional for a custom provider (e.g, OpenRouter)? They support a custom models array feature, and I'm able to get LiteLLM to parse a JSON array, but then OpenRouter errors due to having been passed both model and models.

[krrishdholakia]

hey @zazer0

i think you could just use openrouter/ for this as openrouter is openai compatible

if you pass us any params which aren't openai params - e.g. models in this case, we assume it's provider specific and send it as a kwarg in the request body

[zazer0]

Hi @krrishdholakia, thanks for replying - unfortunately, Openrouter rejects requests with both a model (single) and models (plural) param - but litellm enforces the model param to be sent.

I wish I was more familiar with the litellm codebase and knew which spot to edit to remove the model param check!

[krrishdholakia]

how does openrouter expect you to make the request? @zazer0

i thought they were openai-compatible - if so, they should also accept the model param

[zazer0]

They expect the exact same format; just with a models array instead of the single model string parameter. They are openai-compatible if you're not using the model array functionality!
@krrishdholakia

[krrishdholakia]

@Manouchehri
re: i.e. instead of 4242, for this specific one example, I don't want there to be any seed sent to the upstream LLM.

you can now do this by specifying which params you want to drop
https://docs.litellm.ai/docs/completion/drop_params#specify-params-to-drop

- litellm_params:
    api_base: my-base
    model: openai/my-model
    additional_drop_params: ["response_format"] # 👈 KEY CHANGE
  model_name: my-model

[Manouchehri]

This works to drop the parameter from the client completely for all requests, but in my case, I wanted to override it for only some clients (and without needing the LiteLLM server to force it).

[Manouchehri]

To control this per request, use null.

curl -v "${OPENAI_API_BASE}/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "gpt-3.5-turbo",
    "seed": null,
    "messages": [
      {
        "role": "user",
        "content": "what is your name"
      }
    ]
  }'