Support Gemini Structured outputs

[fernando-deka]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

Gemini models support structured output JSON schemas:

https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/control-generated-output#generativeaionvertexai_gemini_controlled_generation_response_schema-drest

Currently langchain only supports setting the responseMimeType to json but does not set the responseSchema to ensure that the JSON follows the given schema.

Motivation

Currently when using the built in .withStructuredOutput the model usually fails with complex schemas. With this feature the reliability would increase a lot.

Proposal (If applicable)

No response

[glorat]

I've just run into this limitation too. I can sort of see what needs to be done to support...

• GoogleAIModelParams+GeminiGenerationConfig extended to add responseSchema
• formatGenerationConfig to also accept the new param
• withStructuredOutput to support method=="jsonMode" with this methodology (rather than function calling)

It's not trivial to support but shouldn't be too hard for the author either. Hope this module is still supported and someone will pick up

[glorat]

Here is a holy hack to make it work!

// Custom ChatVertexAI with schema support
import { ChatVertexAI } from '@langchain/google-vertexai'
import logger from '../logger'

export class CustomChatVertexAI extends ChatVertexAI {
  private responseSchema?: any
  private originalFormatData: any

  constructor(
    fields: ConstructorParameters<typeof ChatVertexAI>[0] & {
      responseSchema?: any
    }
  ) {
    const { responseSchema, ...rest } = fields
    super(rest)
    this.responseSchema = responseSchema

    // Store the original formatData method
    this.originalFormatData = this.connection.formatData

    // Replace formatData with our interceptor
    this.connection.formatData = async (...args: any[]) => {
      const result = await this.originalFormatData.apply(this.connection, args)
      logger.info('Connection formatData result:', result)
      if (result.generationConfig.responseMimeType === 'application/json') {
        result.generationConfig.responseSchema = this.responseSchema
      }
      return result
    }
  }
}