[Bug]: bedrock fallback not working

[zegerius]

What happened?

While using Bedrock with Claude Sonnet 3.5 v2, I see a lot of the following in my application:

openai.APIError: litellm.APIConnectionError: Bad response code, expected 200: {'status_code': 400, 'headers': {':exception-type': 'serviceUnavailableException', ':content-type': 'application/json', ':message-type': 'exception'}, 'body': b'{"message":"Bedrock is unable to process your request."}'}

The combination of a 400 and ServiceUnavailable seems to be impossible and the error message is quite useless. https://docs.aws.amazon.com/bedrock/latest/userguide/troubleshooting-api-error-codes.html

These errors occur seemingly randomly, and moments later the same requests works fine. I have configured fallbacks in my configuration:

model_list:
    litellm_params:
      model: bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0
      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
      aws_region_name: os.environ/AWS_DEFAULT_REGION
  - model_name: bedrock-claude-haiku-35-v1
    litellm_params:
      model: bedrock/anthropic.claude-3-5-haiku-20241022-v1:0
      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
      aws_region_name: os.environ/AWS_DEFAULT_REGION
router_settings:
  fallbacks: [{"bedrock-claude-sonnet-35-v2": ["bedrock-claude-haiku-35-v1"]}]

I also tested

        litellm_client = openai.AsyncOpenAI(
            api_key=settings.litellm_api_key, base_url=settings.litellm_url
        )
        ...
        response = await litellm_client.chat.completions.create(
            model=model_slug,
            user=str(user_id),
            temperature=settings.litellm_model_temp,
            top_p=settings.litellm_model_top_p,
            messages=messages,
            stream=True,
            metadata={"tags": ["thread"]},
            extra_body={"fallbacks": ["bedrock-claude-haiku-35-v1", "gemini-1-5-pro"]},
        )

The documentation on fallbacks this way is quite sparse.

I ran LiteLLM in debug mode with the most verbose logging I have seen in a while, but I don't see a mention of retries.

Having debugged this for days now, I am not sure what is left.

Relevant log output

No response

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

v1.57.3

Twitter / LinkedIn details

No response