-
Notifications
You must be signed in to change notification settings - Fork 167
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: list_tables method in glue catalog now only return tables. #1258
Conversation
Hi Team , @kevinjqliu @sungwy Error Details:When views are included in the output of list_tables and subsequently processed, the absence of the table_type parameter often raises an error similar to: |
Hey @omkenge thanks for pointing this out. I was unaware of this issue. Are you interested in providing a patch to fix this? |
Hi @Fokko |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@omkenge Thanks for reporting the issue and providing the fix!
NoSuchPropertyException: Property 'table_type' missing, could not determine type
I was able to reproduce this issue and verify that this PR can fix it.
Let's get this into 0.8.0!
Co-authored-by: Honah J. <[email protected]>
HI @HonahX |
Thanks for the contribution @omkenge ! |
Description:
This pull request addresses an issue where the list_tables function was returning views alongside Iceberg tables. Since views lack the table_type property or have it set differently than "iceberg", they were incorrectly included in the list of tables. This change modifies the filtering logic to ensure that only actual Iceberg tables are returned.
Problem:
In the existing list_tables function, both Iceberg tables and Glue views were being returned. Glue views generally do not contain the table_type parameter or may have it unset, which causes them to be mistakenly included when listing Iceberg tables in a namespace. This behavior makes it challenging for users to retrieve only Iceberg tables as intended.
Solution:
The __is_iceberg_table helper method has been modified to:
Check if the table_type parameter is present in each table’s properties.
Ensure that the table_type parameter is explicitly set to "iceberg".
Testing:
Tests were added to verify that:
Only Iceberg tables are returned by list_tables.
Glue views, which lack the table_type parameter or have it set to a non-Iceberg value, are excluded from the list.