Support jq filters directly?

[baodrate]

Currently, hurl supports capturing responses by transforming them with jsonpath expressions. It would be nice if hurl could additionally support jq filters. jq provides much more powerful transformations and its syntax is well known among developers.

jaq is a rust implementation of the jq language, and could be integrated into hurl relatively easily.

I would like to work on this, if this feature would be desired by the project

[jcamiel]

Hi @baod-rate

Thanks for the idea. After discussion, whereas it can be useful, we're not going to include it:

• we have a lot of issues with JSONPath: as there is discrepancy between the "reference" document (https://goessner.net/articles/JsonPath/) and various implementations of JSONPath, we have issues about why queries are not working in Hurl, whereas they're working with https://jsonpath.com. Having multiple implementations, between what we think is the spec and other implementations raises issues about what's "the correct behaviour". If the crate jaq decides to have slightly different behaviour compared to jq, we'll have the same type of issues
• speaking of jaq, the crates has a lot of other dependencies and we're always reluctant to increment the number of crates dependencies,
• we have now filters that works on all Hurl queries (jsonpath, xpath etc...) so we prefer to favor the usage of filter instead of adding another way of querying JSON.

That said, I confess don't know all the powers of jq. Do you have examples of what is difficult to do with our actual jsonpath / filters combination and would be easy with a jq filter ?

[baodrate]

If the crate jaq decides to have slightly different behaviour compared to jq, we'll have the same type of issues

That's fair. And though jaq has been good at maintaining consistency with jq where possible, there are certain irreconcilable (albeit minor) differences that arise because of jaq's more strict language design

speaking of jaq, the crates has a lot of other dependencies and we're always reluctant to increment the number of crates dependencies

totally fair

we have now filters that works on all Hurl queries (jsonpath, xpath etc...) so we prefer to favor the usage of filter instead of adding another way of querying JSON

The issue I was having was it wasn't possible (AFAIK) to produce the filter I needed with the expressions that jsonpath allows.

Do you have examples of what is difficult to do with our actual jsonpath / filters combination and would be easy with a jq filter?

Admittedly, my use-case might be uncommon so I'm totally open to it simply being out-of-scope. But I find myself reaching for jq when I need anything more sophisticated than extracting a simple path. To use an example from MDN

GET https://heroes.org/api/squads
[QueryStringParams]
name: "Super hero squad"

HTTP 200
[Captures]
# get the name of the oldest member that has either 'Radiation resistance' or 'Immortality'
member-name: jq ".members | map(select(.powers[]|.=='Radiation resistance' or .=='Immortality')) | sort_by(.age)[-1] | .name"
# 

Although jq is many orders of magnitude more expressive than this example, this is the simplest one I can think of that (AFAICT) is not possible with only Hurl filters. I sometimes make involved hurl request chains and the lack of expressiveness makes crafting them a bit of a headache.

I do recognize that jq's filters overlap a great deal with hurl's. Design wise I have to admit that it's probably better to work on extending hurl's language-agnostic filters. Still, a lot of the time I wish I could just fall back to jq. I wish it were possible to simply pipe the result into a command and capture the output, allowing unbounded expressiveness in the query syntax.

[Captures]
foobar: sh "jq -ecr '.authors | map(select(.books|map(select(.published_year<1990))|length>3)) | sort_by(.last_name)[0]'"
"

although I assume the maintainers have already considered that design and dismissed it