Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Proxied Model Support #379

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

JGalego
Copy link

@JGalego JGalego commented May 16, 2024

This PR adds support for models served behind OpenAI-compatible proxies e.g. LiteLLM or Bedrock Access Gateway (missing image generation support). The constraint is to change as little as possible of the original implementation.

Example (LiteLLM >> Amazon Bedrock)

  1. Set up AWS credentials

    export AWS_ACCESS_KEY_ID=...
    export AWS_SECRET_ACCESS_KEY=...
    export AWS_SESSION_TOKEN=...
  2. Start LiteLLM proxy

    litellm --config config.yaml
    # config.yaml
    # Adapted from https://litellm.vercel.app/docs/proxy/configs
    
    model_list:
      - model_name: claude3-sonnet
        litellm_params: 
          model: bedrock/anthropic.claude-3-sonnet-20240229-v1:0
          aws_region_name: us-east-1
          temperature: 0.0
      - model_name: sdxl
        litellm_params: 
          model: bedrock/stability.stable-diffusion-xl-v1
          aws_region_name: us-east-1
    
    litellm_settings:
      drop_params: True
      modify_params: True  # hack to get around Claude's Message API restrictions
    
    general_settings: 
      master_key: correct-horse-battery-staple
  3. Set up OpenAI and ChatDev env vars

    # OpenAI
    export OPENAI_BASE_URL=http://0.0.0.0:4000
    export OPENAI_API_KEY=correct-horse-battery-staple
    
    # ChatDev
    export CHATDEV_CUSTOM_MODEL=claude3-sonnet
    export CHATDEV_NUM_MAX_TOKEN=200000
    export CHATDEV_CUSTOM_IMAGE_MODEL=sdxl
  4. Run ChatDev

    python3 run.py --task "Design a simple game of tic-tac-toe" --name "TicTacToe"  --org "THUNLP" --config "Default"
chatdev_bedrock.mp4

@JGalego JGalego marked this pull request as ready for review May 16, 2024 11:17
@JGalego
Copy link
Author

JGalego commented May 29, 2024

Hey @thinkwee, reaching out b/c you're currently listed as the top contributor.
Can we have someone review this PR?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant