Cache/Store prompts and their completions.

[BillSchumacher]

🚀 Feature Request

Feature description

Integrate prompt completions tracking with the SQLAlchemy model and cache similar prompts in Redis to avoid making additional API calls for similar requests.

Rationale and use cases

Reducing the number of API calls can save costs and improve response times for users. By storing prompt completions in the SQLAlchemy model and caching similar prompts in Redis, we can optimize the system to deliver faster responses and minimize external API calls.

Proposed implementation (if applicable)

• Implement a caching mechanism using Redis to store similar input prompts and their responses.
• When a user submits a prompt, first check Redis for cached responses with similar prompts. If found, return the cached response.
• If the prompt is not found in the Redis cache, query the database for similar prompts and their responses. If found, store the response in Redis and return it to the user.
• If the prompt is not found in the cache or database, make an API call to get the completion, store it in the database and Redis cache, and return the response to the user.

Potential drawbacks and alternatives

• Managing cache consistency and expiration could become complex as the number of prompts and completions grows.
• Depending on the similarity threshold, the system may return responses that are not an exact match for the input prompt, which may affect the user experience.
• Alternative solutions include using more advanced caching mechanisms, such as machine learning-based models to predict and prefetch prompt completions, or implementing a content-based recommendation system to identify and return similar prompts.