Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

describe on tables can be slower than equivalent query on information_schema.columns #443

Open
1 task done
posulliv opened this issue Nov 1, 2024 · 0 comments · May be fixed by #444
Open
1 task done

describe on tables can be slower than equivalent query on information_schema.columns #443

posulliv opened this issue Nov 1, 2024 · 0 comments · May be fixed by #444
Labels
enhancement New feature or request

Comments

@posulliv
Copy link
Member

posulliv commented Nov 1, 2024

Describe the feature

At least with Iceberg and Hive tables, we have observed that describe <table> is consistently slower than the equivalent query against information_schema.columns to retrieve column information:

select column_name, data_type
from information_schema.columns
where
    table_catalog = '{{ relation.database | lower }}'
    and table_schema = '{{ relation.schema | lower }}'
    and table_name = '{{ relation.identifier }}'

Given how often describe statements are run, we would like to switch to using the faster query against information_schema.columns.

Describe alternatives you've considered

We considered enabling metastore caching in trino. This is disabled in Iceberg and cannot be enabled so we don't consider it a valid option.

Why the describe statement is consistently slower in trino is an unknown. We feel updating the dbt-trino adapter is a quicker solution than tracking the down the issue in trino.

Who will benefit?

This will speed up dbt pipelines without requiring any trino configuration changes to enable any caching.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!
@posulliv posulliv added the enhancement New feature or request label Nov 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant