Support array types

[plcplc]

What

We now recognize array types in several places:

• Database introspection columns and comparison operators of array type
• Columns of array type can be queried
• Arguments of array type can be input
• Arrays work in tables and native queries alike

Notable limitations, due in part to the modest expressiveness of the ndc-spec in its current state:

• There is no way to filter array fields
• There is no way to project fields from the elements of an array field
• We only support monomorphic comparison operators

How

The introspection query has had its types sub-query split into scalar_types and array_types subqueries. We benefit from the fact that postgres only supports one-dimensional array types (i.e., array dimensionality is a dynamic aspect of an array value), so we don't have to recurse to construct the json-representation of array types.

Most of the rust-language changes of this PR follow mechanically from changing metadata::ScalarType to the new metadata::Type, which also models array types.

However, since our field types now include arrays we need to observe that certain parts (e.g. ComparisonOperators) only accept base scalar types (e.g. in version1.rs, filtering.rs).

In defining the new type for column types we also take care to maintain backwards compatibility, which means deserialization gets a bit more verbose than just the automatically derived implementations we used to get by with (see metadata/database.rs).

Since we also support array-typed arguments we have to be able to output SQL array constructors (see values.rs, sql/convert.rs).

We introduce two new tests:

• One test which just verifies that we can output an array column from a native query
• Another which verifies that we can take an array-valued argument to a native query.

Update

Altering the representation of column types in the configuration constitues a breaking change. Even if we are still able to ingest the historic configuration formats these are not described by the published json schema, and we would only ever output in the newest format.

In order to address the issue of versioning the original contents of this PR is being split in two:

• The first part (this PR) contains the logic for dealing with array types, and includes a new level of indirection between the api transport types and those used internally.
• A second follow-up PR will introduce a version 2 of the configuration format, update introspection to include array types, and add tests of the same.

An interesting consequence of introducing the distinction between api transport types and internal business types is that the various tables.json files of the query-engine translation tests contain json-serialized data of business types.

Worth highlighting is this means we cannot simply copy a subsection of a deployment configuration into a tables.json file like we used to, as the business types do not necessarily correspond to any api-version.

While this may seem confusing it also has some benefits:

• Test files have no version drift. They are always representative of the current internal business data model
• The test remains coupled with the query-engine crate rather than the connector crate.

Updating test files can become a chore too. There is no tool that can upgrade them automatically like we do for deployments, and it would probably not be worth the effort to try and make one.

What I ended up doing in the particular case of this PR was a small shell script:

for f in $(fd -t f 'tables\.json')
do
  jq 'walk(if type == "object" and has("type") then .type |= {scalarType: .} else . end)' < "$f" \
  | sponge "$f"
done

The jq script matches any object that has a type key and wraps its value with a {"scalarType": ..} object. Happily this covered the full breadth of my change.

sponge is a tool I just discovered which buffers all the stdin it receives and writes everything into the file it's given. Essentially this endows our command with the ability to make in-place edits with shell pipe filters.

[danieljharvey]

We do this mapping on every request don't we? I wonder if in future we do it whenever RawConfiguration becomes Configuration (one of Connector trait functions run by the config server IIRC), and keep RuntimeConfiguration inside Configuration.

Not in this PR, mind. Probably worth benchmarking it too to check a speculative speed up becomes a real one.

[plcplc]

I'm not sure this makes sense.

IIUC, what the user will consider as being "their configuration metadata" is the json-serialized Configuration, not just the RawConfiguration, meaning that the whole thing is what needs to be able to exist in different versions.

It's still a bit nebulous to me though, so I could be wrong.

[danieljharvey]

All I mean is, currently we take the whole Configuration and derive RuntimeConfiguration from it on each request, surely we could just do this once?

It's not really relevant to this PR, mind, just something I noticed because you made a change near the code that does this.

[danieljharvey]

This all seems sensible. Left a couple of questions, but they could probably both be answered in a couple of follow-up tidy ups or optimisation passes at some later point.