Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add support for abfs:// #75

Merged
merged 4 commits into from
Sep 3, 2024
Merged

Conversation

gdubya
Copy link

@gdubya gdubya commented Sep 2, 2024

Add support for abfs, as requested by @sugibuchi in #72

@samansmink
Copy link
Collaborator

Thanks for the PR @gdubya! Our CI should run against a test container on azure, see https://github.com/duckdb/duckdb_azure/blob/main/test/sql/cloud/hierarchical_namespace.test

I think you should just be able to copy over a few of those statements and make them use the abfs:// prefix to ensure both formats work.

@gdubya
Copy link
Author

gdubya commented Sep 2, 2024

@samansmink Ok, we'll see if it's that simple. 😅

But part of my concern is that the client behaviour changes depending on the authentication mode used. From the documentation:

Scheme identifier: The abfs protocol is used as the scheme identifier. If you add an s at the end (abfss) then the ABFS Hadoop client driver will always use Transport Layer Security (TLS) irrespective of the authentication method chosen. If you choose OAuth as your authentication, then the client driver will always use TLS even if you specify abfs instead of abfss because OAuth solely relies on the TLS layer. Finally, if you choose to use the older method of storage account key, then the client driver interprets abfs to mean that you don't want to use TLS.

But I guess that's the client driver's responsibility, not this extension.

@gdubya gdubya marked this pull request as ready for review September 2, 2024 10:13
@gdubya gdubya changed the title draft: feat: add support for abfs:// feat: add support for abfs:// Sep 2, 2024
@gdubya
Copy link
Author

gdubya commented Sep 2, 2024

@samansmink I copied some "abfss" tests and it looks like the checks still pass, but I can't find the tests in the logs. Can you see them?

Copy link
Collaborator

@samansmink samansmink left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea I now realize the CI that requires secrets does not run on PRs for obvious security reasons 😅

I have tested this locally and can confirm it works now after a some minor fixes. CI should properly test this on merging so I think this is good to go

@samansmink samansmink merged commit b0ffe7a into duckdb:main Sep 3, 2024
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants