Skip to content

Commit

Permalink
stream chat request logic and maintaining conversation continuity are…
Browse files Browse the repository at this point in the history 
… added (#14)

* stream chat completion feature is fixed

* return raw response from chatStreamRequest and add Maintaining Conversation Continuity section

* typo fixed
  • Loading branch information
@moe-mizrak
moe-mizrak authored Jun 20, 2024
1 parent 931575b commit 9ff0285
Show file tree
Hide file tree
Showing 5 changed files with 416 additions and 11 deletions.
211 changes: 211 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ This Laravel package provides an easy-to-use interface for integrating **[OpenRo
- [Creating a ChatData Instance](#creating-a-chatdata-instance)
- [Using Facade](#using-facade)
- [Chat Request](#chat-request)
- [Stream Chat Request](#stream-chat-request)
- [Maintaining Conversation Continuity](#maintaining-conversation-continuity)
- [Cost Request](#cost-request)
- [Limit Request](#limit-request)
- [Using OpenRouterRequest Class](#using-openrouterrequest-class)
Expand Down Expand Up @@ -177,6 +179,214 @@ $chatData = new ChatData([

$chatResponse = LaravelOpenRouter::chatRequest($chatData);
```
- #### Stream Chat Request
Streaming chat request is also supported and can be used as following by using **chatStreamRequest** function:
```php
$content = 'Tell me a story about a rogue AI that falls in love with its creator.'; // Your desired prompt or content
$model = 'mistralai/mistral-7b-instruct:free'; // The OpenRouter model you want to use (https://openrouter.ai/docs#models)
$messageData = new MessageData([
'content' => $content,
'role' => RoleType::USER,
]);

$chatData = new ChatData([
'messages' => [
$messageData,
],
'model' => $model,
'max_tokens' => 100, // Adjust this value as needed
]);

/*
* Calls chatStreamRequest ($promise is type of PromiseInterface)
*/
$promise = LaravelOpenRouter::chatStreamRequest($chatData);

// Waits until the promise completes if possible.
$stream = $promise->wait(); // $stream is type of GuzzleHttp\Psr7\Stream

/*
* 1) You can retrieve whole raw response as: - Choose 1) or 2) depending on your case.
*/
$rawResponseAll = $stream->getContents(); // Instead of chunking streamed response as below - while (! $stream->eof()), it waits and gets raw response all together.
$response = LaravelOpenRouter::filterStreamingResponse($rawResponse); // Optionally you can use filterStreamingResponse to filter raw streamed response, and map it into array of responseData DTO same as chatRequest response format.

// 2) Or Retrieve streamed raw response as it becomes available:
while (! $stream->eof()) {
$rawResponse = $stream->read(1024); // readByte can be set as desired, for better performance 4096 byte (4kB) can be used.

/*
* Optionally you can use filterStreamingResponse to filter raw streamed response, and map it into array of responseData DTO same as chatRequest response format.
*/
$response = LaravelOpenRouter::filterStreamingResponse($rawResponse);
}
```
You do **not** need to specify `'stream' = true` in ChatData since `chatStreamRequest` does it for you.
<details>

This is the expected sample rawResponse (raw response returned from OpenRouter stream chunk) `$rawResponse`:
```php
"""
: OPENROUTER PROCESSING\n
\n
data: {"id":"gen-eWgGaEbIzFq4ziGGIsIjyRtLda54","model":"mistralai/mistral-7b-instruct:free","object":"chat.completion.chunk","created":1718885921,"choices":[{"index":0,"delta":{"role":"assistant","content":"Title"},"finish_reason":null}]}\n
\n
data: {"id":"gen-eWgGaEbIzFq4ziGGIsIjyRtLda54","model":"mistralai/mistral-7b-instruct:free","object":"chat.completion.chunk","created":1718885921,"choices":[{"index":0,"delta":{"role":"assistant","content":": Quant"},"finish_reason":null}]}\n
\n
data: {"id":"gen-eWgGaEbIzFq4ziGGIsIjyRtLda54","model":"mistralai/mistral-7b-instruct:free","object":"chat.completion.chunk","created":1718885921,"choices":[{"index":0,"delta":{"role":"assistant","content":"um Echo"},"finish_reason":null}]}\n
\n
data: {"id":"gen-eWgGaEbIzFq4ziGGIsIjyRtLda54","model":"mistralai/mistral-7b-instruct:free","object":"chat.completion.chunk","created":1718885921,"choices":[{"index":0,"delta":{"role":"assistant","content":": A Sym"},"finish_reason":null}]}\n
\n
data: {"id":"gen-eWgGaEbIzFq4ziGG
"""

"""
IsIjyRtLda54","model":"mistralai/mistral-7b-instruct:free","object":"chat.completion.chunk","created":1718885921,"choices":[{"index":0,"delta":{"role":"assistant","content":"phony of Code"},"finish_reason":null}]}\n
\n
data: {"id":"gen-eWgGaEbIzFq4ziGGIsIjyRtLda54","model":"mistralai/mistral-7b-instruct:free","object":"chat.completion.chunk","created":1718885921,"choices":[{"index":0,"delta":{"role":"assistant","content":"\n\nIn"},"finish_reason":null}]}\n
\n
data: {"id":"gen-eWgGaEbIzFq4ziGGIsIjyRtLda54","model":"mistralai/mistral-7b-instruct:free","object":"chat.completion.chunk","created":1718885921,"choices":[{"index":0,"delta":{"role":"assistant","content":" the heart of"},"finish_reason":null}]}\n
\n
data: {"id":"gen-eWgGaEbIzFq4ziGGIsIjyRtLda54","model":"mistralai/mistral-7b-instruct:free","object":"chat.completion.chunk","created":1718885921,"choices":[{"index":0,"delta":{"role":"assistant","content":" the bustling"},"finish_reason":null}]}\n
\n
data: {"id":"gen-eWgGaEbIzFq4ziGGIsIjyRtLda54","model":"mistralai/mistra
"""

"""
l-7b-instruct:free","object":"chat.completion.chunk","created":1718885921,"choices":[{"index":0,"delta":{"role":"assistant","content":" city of Ne"},"finish_reason":null}]}\n
\n
data: {"id":"gen-eWgGaEbIzFq4ziGGIsIjyRtLda54","model":"mistralai/mistral-7b-instruct:free","object":"chat.completion.chunk","created":1718885921,"choices":[{"index":0,"delta":{"role":"assistant","content":"o-Tok"},"finish_reason":null}]}\n
\n
data: {"id":"gen-eWgGaEbIzFq4ziGGIsIjyRtLda54","model":"mistralai/mistral-7b-instruct:free","object":"chat.completion.chunk","created":1718885921,"choices":[{"index":0,"delta":{"role":"assistant","content":"yo, a"},"finish_reason":null}]}\n
\n
data: {"id":"gen-eWgGaEbIzFq4ziGGIsIjyRtLda54","model":"mistralai/mistral-7b-instruct:free","object":"chat.completion.chunk","created":1718885921,"choices":[{"index":0,"delta":{"role":"assistant","content":" brilliant young research"},"finish_reason":null}]}\n
\n
data: {"id":"gen-eWgGaEbIzFq4ziGGIsIjyRtLda54","model":"mistralai/mistral-7b-instruct:free","object":"chat.com
"""
...

: OPENROUTER PROCESSING\n
\n
data: {"id":"gen-C6Xym94jZcvJv2vVpxYSyw2tV1fR","model":"mistralai/mistral-7b-instruct:free","object":"chat.completion.chunk","created":1718887189,"choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}],"usage":{"prompt_tokens":23,"completion_tokens":100,"total_tokens":123}}\n
\n
data: [DONE]\n
```
Last `data:` carries usage information of streaming.
`data: [DONE]\n` returned from OpenRouter server when streaming is over.

This is the sample response after filterStreamingResponse:
```
[
ResponseData(
id: "gen-QcWgjEtiEDNHgomV2jjoQpCZlkRZ",
model: "mistralai/mistral-7b-instruct:free",
object: "chat.completion.chunk",
created: 1718888436,
choices: [
[
"index" => 0,
"delta" => [
"role" => "assistant",
"content" => "Title"
],
"finish_reason" => null
]
],
usage: null
),
ResponseData(
id: "gen-QcWgjEtiEDNHgomV2jjoQpCZlkRZ",
model: "mistralai/mistral-7b-instruct:free",
object: "chat.completion.chunk",
created: 1718888436,
choices: [
[
"index" => 0,
"delta" => [
"role" => "assistant",
"content" => "Quant"
],
"finish_reason" => null
]
],
usage: null
),
...
new ResponseData([
'id' => 'gen-QcWgjEtiEDNHgomV2jjoQpCZlkRZ',
'model' => 'mistralai/mistral-7b-instruct:free',
'object' => 'chat.completion.chunk',
'created' => 1718888436,
'choices' => [
[
'index' => 0,
'delta' => [
'role' => 'assistant',
'content' => '',
],
'finish_reason' => null,
],
],
'usage' => new UsageData([
'prompt_tokens' => 23,
'completion_tokens' => 100,
'total_tokens' => 123,
]),
]),
]
```
</details>

- #### Maintaining Conversation Continuity
If you want to maintain **conversation continuity** meaning that historical chat will be remembered and considered for your new chat request, you need to send historical messages along with the new message:
```php
$model = 'mistralai/mistral-7b-instruct:free';

$firstMessage = new MessageData([
'role' => RoleType::USER,
'content' => 'My name is Moe, the AI necromancer.',
]);

$chatData = new ChatData([
'messages' => [
$firstMessage,
],
'model' => $model,
]);
// This is the chat which you want LLM to remember
$oldResponse = LaravelOpenRouter::chatRequest($chatData);

/*
* You can skip part above and just create your historical message below (maybe you retrieve historical messages from DB etc.)
*/

// Here adding historical response to new message
$historicalMessage = new MessageData([
'role' => RoleType::ASSISTANT, // set as assistant since it is a historical message retrieved previously
'content' => Arr::get($oldResponse->choices[0],'message.content'), // Historical response content retrieved from previous chat request
]);
// This is your new message
$newMessage = new MessageData([
'role' => RoleType::USER,
'content' => 'Who am I?',
]);

$chatData = new ChatData([
'messages' => [
$historicalMessage,
$newMessage,
],
'model' => $model,
]);

$response = LaravelOpenRouter::chatRequest($chatData);
```
Expected response:
```php
$content = Arr::get($response->choices[0], 'message.content');
// content = You are Moe, a fictional character and AI Necromancer, as per the context of the conversation we've established. In reality, you are the user interacting with me, an assistant designed to help answer questions and engage in friendly conversation.
```

#### Cost Request
To retrieve the cost of a generation, first make a `chat request` and obtain the `generationId`. Then, pass the generationId to the `costRequest` method:
```php
Expand Down Expand Up @@ -231,6 +441,7 @@ $chatData = new ChatData([

$response = $this->openRouterRequest->chatRequest($chatData);
```

#### Cost Request
Similarly, to retrieve the cost of a generation, create a `chat request` to obtain the `generationId`, then pass the `generationId` to the `costRequest` method:
```php
Expand Down
4 changes: 3 additions & 1 deletion src/DTO/ErrorData.php
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,15 @@

namespace MoeMizrak\LaravelOpenrouter\DTO;

use Spatie\DataTransferObject\DataTransferObject;

/**
* DTO for error messages.
*
* Class ErrorData
* @package MoeMizrak\LaravelOpenrouter\DTO
*/
class ErrorData
class ErrorData extends DataTransferObject
{
/**
* Error code e.g. 400, 408 ...
Expand Down
6 changes: 5 additions & 1 deletion src/Facades/LaravelOpenRouter.php
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,20 @@

namespace MoeMizrak\LaravelOpenrouter\Facades;

use GuzzleHttp\Promise\PromiseInterface;
use Illuminate\Support\Facades\Facade;
use MoeMizrak\LaravelOpenrouter\DTO\ChatData;
use MoeMizrak\LaravelOpenrouter\DTO\CostResponseData;
use MoeMizrak\LaravelOpenrouter\DTO\ErrorData;
use MoeMizrak\LaravelOpenrouter\DTO\LimitResponseData;
use MoeMizrak\LaravelOpenrouter\DTO\ResponseData;

/**
* Facade for LaravelOpenRouter.
*
* @method static ResponseData chatRequest(ChatData $chatData) Sends a chat request to the OpenRouter API and returns the response data.
* @method static ErrorData|ResponseData chatRequest(ChatData $chatData) Sends a chat request to the OpenRouter API and returns the response data.
* @method static PromiseInterface chatStreamRequest(ChatData $chatData) Sends a chat stream request to the OpenRouter API and returns the raw streaming response.
* @method static array filterStreamingResponse(string $streamingResponse) It filters streaming response string so that response string is mapped into ResponseData.
* @method static CostResponseData costRequest(string $generationId) Sends a cost request to the OpenRouter API with the given generation ID and returns the cost response data.
* @method static LimitResponseData limitRequest() Sends a limit request to the OpenRouter API and returns the limit response data.
*/
Expand Down
Loading

0 comments on commit 9ff0285

Please sign in to comment.