Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry-After headers when 'CreateChatCompletionStream' fails due rate limit hits #867

Open
kanutron opened this issue Oct 5, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@kanutron
Copy link

kanutron commented Oct 5, 2024

I believe this could be a duplicate of #452, but the discussion there diverged from the actual point.

Problem

When using the CreateChatCompletionStream, and hitting rate limits errors (Code: 429), the returned error nor the stream doesn't contain the http headers. I need to retrieve 'Retry-After' headers to pause the job for the specified time.

Those headers are standard and provide the amount of time (seconds/time-stamp on 'Retry-After', milliseconds on 'Retry-After-Ms').

If a 429 is obtained, what I do with CreateChatCompletion is to parse those headers and pause to go routine for the specified amount of time.

	resp, err := client.client.CreateChatCompletion(ctx, *openaiReq)
	if err != nil {
		... check status code ...
		parseAndPause(resp.Header()) // even with error, we still get `resp`
	}

But for CreateChatCompletionStream the above code could not work as:

	stream, err := client.client.CreateChatCompletionStream(ctx, *openaiReq)
	if err != nil {
		... check status code ...
		parseAndPause(stream.Header()) // stream.header could be nil, thus this panics
	}

Proposed solution

I believe here:

https://github.com/sashabaranov/go-openai/blob/7c145ebb4be68610bc3bb5377b754944307d44fd/client.go#L166C1-L172C3

	resp, err := client.config.HTTPClient.Do(req) //nolint:bodyclose // body is closed in stream.Close()
	if err != nil {
		return new(streamReader[T]), err
	}
	if isFailureStatusCode(resp) {
		return new(streamReader[T]), client.handleErrorResp(resp)
	}

Those returns should add the response into the streamReader struct.

@kanutron kanutron added the enhancement New feature or request label Oct 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant