Specmatic rejects external examples with invalid query parameters

[jhomswk]

Description:

Hi! 👋

Specmatic seems to reject external examples containing query parameters
that violate the spec even though the status code indicates an error.

Please, let me know whether I am missing something.
Thank you in advance.

Minimal Example:

🧾 OpenAPI specification:

openapi: 3.0.0

info:
  title: Users Service
  version: 1.0.0

servers:
  - url: http://localhost:9000

paths:
  /find-user:
    get:
      parameters:
        - in: query
          name: id
          required: true
          schema:
            type: string
            format: uuid
      responses:
        200:
          description: user found
        400:
          description: validation error

✔️ Example that works:

{
  "http-request": {
    "path": "/find-user",
    "method": "GET",
    "query": {
      "id": "4cee792f-6a93-43ec-80c7-a4f2e4e9ed49"
    }
  },
  "http-response": {
    "status": 200
  }
}

❌ Example that fails:

{
  "http-request": {
    "path": "/find-user",
    "method": "GET",
    "query": {
      "id": "invalid"
    }
  },
  "http-response": {
    "status": 400
  }
}

💻 Error:

Steps To Reproduce:

I have prepared a small repository that can be used to reproduce the issue.

• git clone https://github.com/jhomswk/specmatic-invalid-query-parameters
• execute docker compose up
• check the logs of the specmatic container

System Information:

• image: znsio/specmatic:2.0.48

[harikrishnan83]

hello @jhomswk , thanks for the detailed issue report. I will have a look shortly.

[StarKhan6368]

Hello again @jhomswk, thank you for the comprehensive report. Utilizing the provided repository, we were able to replicate the issue and have successfully identified the root cause. We are currently working on a solution and will keep you informed of our progress.

[jhomswk]

Thank you @StarKhan6368.
There's no hurry at all, I am just exploring specmatic for contract testing.
I think it's great and would like to thank your contribution to the testing community! 👍

[harikrishnan83]

@jhomswk Thanks a lot for your kind words of appreciation.

While we wait for the fix, just want to highlight that Specmatic Stub server will anyway return 400 response if the query parameter that you send is not as per OpenAPI spec. If the HTTP 400 response is what matters for your use case (and not he exact schema of the response) you can put this to use even without explicitly setting an example with a schema invalid query parameter. Hope this helps.