README.md

AI Gateway CE

A powerful, AI Gateway designed from scratch for AI

Documentation | Quick Start | Community

✨ Features

• 🚀 High Performance: Built in Go for maximum efficiency and minimal latency
• 🌍 Multi-Provider: Multiple LLM provider support
• 🤖 AI-Ready: Optimized for AI model providers (OpenAI, Anthropic, etc.)
• 🔄 Fallback Ready: Built-in model fallback capabilities
• 🔄 Load Balancing: Advanced algorithms including round-robin, weighted round-robin, and IP hash
• 🔒 Security: Built-in authentication, rate limiting, and token management
• 🔌 Plugin System: Extensible architecture for custom functionality
• ⚡ Real-time Config: Dynamic configuration without restarts
• ☁️ Cloud Agnostic: Deploy anywhere with cloud-agnostic architecture

🚀 Quick Start

Using Docker Compose

# Clone the repository
git clone https://github.com/NeuralTrust/ai-gateway-ce.git
cd ai-gateway-ce

# Start the services
docker compose -f docker-compose.prod.yaml up -d

Using Kubernetes

# Apply configurations
kubectl apply -f k8s/storage.yaml
kubectl apply -f k8s/deployment.yaml

Local Development

# Start dependencies
docker compose up -d redis postgres

# Run the servers
./scripts/run_local.sh

🏗️ Architecture

AI Gateway CE consists of two main components:

1. Admin API (Port 8080)

   • Gateway management
   • Configuration management
   • API key management
   • Plugin configuration

2. Proxy API (Port 8081)

   • Request routing
   • Load balancing
   • Plugin execution

🔌 Plugins

Extend functionality with plugins:

type Plugin interface {
    Name() string
    Execute(ctx *Context) error
    Configure(config map[string]interface{}) error
}

Current Plugins:

• Rate Limiter
• Token Rate Limiter
• External API Call

🔜 Coming Soon Plugins

Security

• Jailbreak Protection

  • Azure Prompt Guard
  • AWS Prompt Guard
  • GCP Prompt Guard

• Toxicity Detection

  • Open AI Moderation API integration
  • Legacy toxicity detection methods

• Prompt Moderation

  • Keywords & REGEX filtering
  • Topic detection (accepted/denied)

• Data Masking

  • Pre-defined entity masking
  • Custom data masking (keywords, regex patterns)

• Network Security

  • CORS protection
  • SQL Injection prevention
  • Cross-site injection protection

• Load Balancing

  • Weight-based routing
  • Round-robin distribution
  • Prompt templates support

🤝 Contributing

We love contributions! Please see our Contributing Guide for details.

1. Fork the repository
2. Create your feature branch
3. Commit your changes
4. Push to the branch
5. Open a Pull Request

📜 License

AI Gateway CE is licensed under the Apache License 2.0 - see the LICENSE file for details.

🌟 Star History

📫 Community & Support

• Documentation
• Slack Community
• GitHub Issues
• Twitter
• Blog

Made with ❤️ by NeuralTrust