ValidationError: messages.1.data.content: Cast to string failed for value (type Array)

[PatrickkZhao]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain.js documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain.js rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

  callbacks: [
    {
      handleLLMStart: (llm, prompts, runId, parentRunId, extraParams) => {
        // Callback logic
      },
    },
  ],
});

const image_url = 'https://example.com/image.jpg'; // Example image URL
new HumanMessage({
  content: [
    {
      type: "image_url",
      image_url: {
        url: image_url,
      },
    },
  ],
});```

### Error Message and Stack Trace (if applicable)

ValidationError: messages.1.data.content: Cast to string failed for value "[
  {
    type: 'image_url',
    image_url: {
      url: 'xxx'
    }
  }
]" (type Array) at path "content"

### Description

I'm encountering a ValidationError while using the ChatOpenAI class in Langchain. The error message indicates that the content of a HumanMessage is expected to be a string but is being provided as an array.

### System Info

node -v
v20.17.0

└─ langchain@npm:0.3.2
   ├─ Instances: 1
   ├─ Version: 0.3.2
   │
   └─ Dependencies
      ├─ @langchain/openai@npm:>=0.1.0 <0.4.0 → npm:0.3.7
      ├─ @langchain/textsplitters@npm:>=0.0.0 <0.2.0 → npm:0.1.0
      ├─ js-tiktoken@npm:^1.0.12 → npm:1.0.12
      ├─ js-yaml@npm:^4.1.0 → npm:4.1.0
      ├─ jsonpointer@npm:^5.0.1 → npm:5.0.1
      ├─ langsmith@npm:^0.1.56-rc.1 → npm:0.1.66
      ├─ openapi-types@npm:^12.1.3 → npm:12.1.3
      ├─ p-retry@npm:4 → npm:4.6.2
      ├─ uuid@npm:^10.0.0 → npm:10.0.0
      ├─ yaml@npm:^2.2.1 → npm:2.4.1
      ├─ zod-to-json-schema@npm:^3.22.3 → npm:3.22.5
      └─ zod@npm:^3.22.4 → npm:3.22.4

[dosubot]

@PatrickkZhao Hello there! I'm here to help you with any bugs, questions, or contributions. Let's tackle this together!

To resolve the ValidationError you're encountering in LangChain.js, where the content of a HumanMessage is expected to be a string but is provided as an array, you need to ensure that the content property of the HumanMessage is a string. The HumanMessage class expects the content to be a string or a list of objects, but in your case, it seems to be misinterpreted due to the array structure.

Here's how you can modify your code to fix the error:

1. Convert the array content to a string if possible, or handle it appropriately before creating a HumanMessage.

2. If you need to include structured data like an image URL, consider encoding it as a string or using a different approach to pass this information.

Here's an example of how you might adjust your code:

new HumanMessage({
  content: JSON.stringify([
    {
      type: "image_url",
      image_url: {
        url: image_url,
      },
    },
  ]),
});

By converting the array to a JSON string, you ensure that the content is a string, which should resolve the ValidationError ^[1].

To continue talking to Dosu, mention @dosu.