User's audio is not received in Pipecat [finalize failed]

[Atharv24]

Sometimes, the user keeps speaking, and the bot behaves as if user is on mute. Did some digging around with trace logs, and can see the finalize call fails just after the UserStoppedSpeakingFrame. Therefore, LLM does not receive the Final transcription frame, and no response is generated from the LLM.

[Atharv24]

I believe the error arises from here, since the deepgram docs specify that the return value is not guaranteed
Deepgram Finalize Docs

async_client finalize method from deepgram python SDK

[markbackman]

Was subsequent audio transcribed by Deepgram after the finalize failed message? That is, was only that one audio sample not transcribed and subsequent chunks were transcribed?

[Atharv24]

@markbackman The issue once arises, remains for a few seconds (sometimes minutes too). Although it fixes itself after that.

[chadbailey59]

From @Atharv24 in the Pipecat Discord:

However, there were InterimTranscriptionFrames being received before the final flush failed. Maybe we can add some sort of error boundary and stitch the interim frames together even if the final flush fails?

This is a good idea, but it would be difficult to test, since we can't reliably reproduce a 'finalize failed' from Deepgram. (Maybe we can hack the client to test it?)