Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Composite ("cube") sources MVP #2012

Merged
merged 31 commits into from
Nov 18, 2024
Merged

Composite ("cube") sources MVP #2012

merged 31 commits into from
Nov 18, 2024

Conversation

christopherswenson
Copy link
Contributor

@christopherswenson christopherswenson commented Nov 15, 2024

This PR adds the notion of a "composite" source to Malloy. This feature is experimental, and can be enabled with the ##! experimental.composite_sources tag. Composite sources are used for handling cubed data.

Syntax:
Note, this syntax is subject to change

  • source: x is composite(source_a, source_b, ...)

Semantics:

  • Queries against composite sources will use the first input source which contains all the fields used in that query

Known Limitations:

  • Composite fields used in on clauses of always joins which are not referenced elsewhere in the query will not be counted
  • Joins of composite sources appearing in extend blocks in subsequent stages will not be resolved, resulting in bad SQL generation (SQL will contain placeholder fields, and will not compile)
  • Composite sources use SourceDefs instead of SourceRefs currently—this means it is really easy to blow up the compiler with source: a is ...; b is compose(a, a); c is compose(b, b); ...
  • Indexing a composite source will not work, and will generate SQL with placeholder values; no validation of cube fields will happen

Other Future Work:

  • In some use cases, multiple composite source inputs will be derived from the same source, where different fields are "turned on or off." Ergonomics for this use case are not great, as it requires using accept or except in order to narrow down the fields. Ergonomically it might be better to have a syntax for manually specifying the fields and types.

…e backtracking case. Get filters and definitions on top of cube to pass through.
…he cube usage was determined (and thus when the query field def is generated)
@christopherswenson christopherswenson changed the title Cube sources MVP Composite ("cube") sources MVP Nov 18, 2024
@christopherswenson christopherswenson merged commit 8a304ab into main Nov 18, 2024
11 checks passed
@christopherswenson christopherswenson deleted the crs-cube branch November 18, 2024 20:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant