Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dagster + SQLMesh Metrics: Use DuckDB as a pre-warmed cache for rolling metrics #2445

Open
ravenac95 opened this issue Nov 4, 2024 · 0 comments
Assignees

Comments

@ravenac95
Copy link
Member

What is it?

Based on some previous testing I've done (seen as part of #2430) we can actually get the metrics to run in a semi-performant way with a very large duckdb instance. Due to the way that the sqlmesh rolling windows ran upon our initial version, deletes + writes into trino were exceedingly slow. Using duckdb as a pre-warmed cache, we can distribute the calculation of metrics to a cluster of pre-warmed duckdbs and then write that back to the trino warehouse.

@ravenac95 ravenac95 self-assigned this Nov 4, 2024
@github-project-automation github-project-automation bot moved this to Backlog in OSO Nov 4, 2024
@ravenac95 ravenac95 moved this from Backlog to In Progress in OSO Nov 4, 2024
@ravenac95 ravenac95 moved this from In Progress to Up Next in OSO Nov 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Up Next
Development

No branches or pull requests

1 participant