From 835a1dcab6e7fc795bb66abec8bb4e543af8827e Mon Sep 17 00:00:00 2001 From: Oz Katz Date: Tue, 2 Jul 2024 15:31:41 +0300 Subject: [PATCH] add lakefs-spec duckdb example (#7876) --- docs/integrations/duckdb.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/docs/integrations/duckdb.md b/docs/integrations/duckdb.md index aa2ea453bc8..6c942dcacb9 100644 --- a/docs/integrations/duckdb.md +++ b/docs/integrations/duckdb.md @@ -70,6 +70,26 @@ USING SAMPLE reservoir(50000 ROWS) REPEATABLE (100); COPY sampled_population TO 's3://example-repo/main/data/population/sample.parquet'; -- actual write happens here ``` +## Using DuckDB in Python with lakefs-spec + +Python users can use DuckDB by leveraging the [lakefs-spec](https://lakefs-spec.org/latest/) package. + +**Note** This library is a third-party package and not maintained by the lakeFS developers; please file issues and bug reports directly +in the [lakefs-spec](https://github.com/aai-institute/lakefs-spec) repository. +{: .note} + +Using lakefs-spec, querying lakeFS could be done using [pre-signed URLs](../reference/security/presigned-url.html), allowing for efficient and secure I/O, where the data files are read directly from the underlying object store. + +```python + +import duckdb +from fsspec import filesystem + +duckdb.register_filesystem(filesystem('lakefs')) + +duckdb.sql("SELECT * FROM 'lakefs://example-repo/main/data/population/sample.parquet'") +``` + ## Using DuckDB in the lakeFS web UI The lakeFS web UI includes DuckDB in the Object viewer page.