feat(backends): support creation from a DB-API con #9603

deepyaman · 2024-07-16T06:00:06Z

Description of changes

Enable backend creation from an existing DB-API connection

Issues closed

Resolves #8877

github-actions · 2024-07-16T06:00:29Z

ACTION NEEDED

Ibis follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message.

Please update your PR title and description to match the specification.

ibis/backends/sql/__init__.py

cpcloud · 2024-07-16T15:20:53Z

ibis/backends/tests/test_client.py

@@ -1764,3 +1764,8 @@ def test_insert_using_col_name_not_position(con, first_row, second_row, monkeypa
    # Ideally we'd use a temp table for this test, but several backends don't
    # support them and it's nice to know that data are being inserted correctly.
    con.drop_table(table_name)
+
+
+def test_from_dbapi_connection(con):


Ideally we can run the entire test suite using a backend constructed this way, which will give us more confidence that the API works.

That can be done in a follow up though.

How do you envision doing this without running the whole test suite again? Or it's fine to run it again?

You can (somewhat arbitrarily) pick a backend and modify its conftest.py to use this way of initialization.

But then we wouldn't have the other way of initialization, right? Or, because there's ddl_con and con fixtures both, it would be OK?

Will look to explore this in a followup then; will just get the rest of the backends added and tests passing here first.

ibis/backends/__init__.py

jcrist

It'd be nice to make this available as ibis.<backend>.from_connection as well. The code that does this is gross, but lives here. Could add it with a hasattr check here, or with adding from_connection to _top_level_methods on base SQL backend (and the snowflake backend, since it appends). No strong thoughts.

deepyaman · 2024-07-16T15:44:26Z

ibis/backends/clickhouse/__init__.py

@@ -163,6 +163,20 @@ def do_connect(
            **kwargs,
        )

+    @classmethod
+    def from_dbapi_connection(cls, con: cc.driver.Client) -> Backend:


The latest request from this functionality comes from @sh-rp and the dltHub team: #8877 (comment)

However, for a case like ClickHouse, I'm not 100% sure we can bridge the gap, because dltHub uses clickhouse-driver (community package, longer history, potentially better at scale?) and Ibis uses clickhouse-connect (official ClickHouse package, better HTTP support). From ClickHouse/clickhouse-connect#91, it looks like there was a lot of consideration around unification, but, in the end, doesn't seem likely.

Scale seems like a non-issue (e.g., ClickHouse/clickhouse-connect#91 (comment)).

We switched to clickhouse-connect due to a bunch of data type issues and choices that didn't align well with Ibis (ClickHouse enums were completely broken due to the refusal to play nice with Python's built in Enum types), and at the time the maintainer was unresponsive and hadn't made a release in many months.

We're definitely not prepared to support multiple ways to connect to ClickHouse right now and it's unlikely we will do so in the future.

Makes sense! I just wanted to note this, and appreciate the explanation of why Ibis moved to clickhouse-connect.

deepyaman · 2024-07-20T15:05:42Z

~~Oh, I forgot about BigQuery and Snowflake...~~

Done.

cpcloud

Excellent work!

cpcloud · 2024-07-22T18:24:48Z

Gonna run BQ and Snowflake locally real quick...

cpcloud · 2024-07-22T18:34:52Z

Found a small bug in BQ, fixing it now.

cpcloud · 2024-07-22T20:18:14Z

Tested the clouds locally, but running here for completeness. Will merge on green and then cut 9.2!

ibis/backends/bigquery/__init__.py

deepyaman force-pushed the feat/from-dbapi-con branch from 6fa7d34 to b359aaf Compare July 16, 2024 15:03

cpcloud reviewed Jul 16, 2024

View reviewed changes

ibis/backends/sql/__init__.py Outdated Show resolved Hide resolved

cpcloud reviewed Jul 16, 2024

View reviewed changes

jcrist reviewed Jul 16, 2024

View reviewed changes

ibis/backends/__init__.py Outdated Show resolved Hide resolved

jcrist reviewed Jul 16, 2024

View reviewed changes

deepyaman commented Jul 16, 2024

View reviewed changes

deepyaman changed the title ~~[WIP] feat(backends): support creation from a DB-API con~~ feat(backends): support creation from a DB-API con Jul 16, 2024

deepyaman force-pushed the feat/from-dbapi-con branch 3 times, most recently from 1bd470a to 9f380c9 Compare July 17, 2024 06:20

sh-rp mentioned this pull request Jul 17, 2024

[experiment] ibis integration with connection sharing dlt-hub/dlt#1491

Draft

deepyaman force-pushed the feat/from-dbapi-con branch 3 times, most recently from 18fdb69 to 79db1fe Compare July 18, 2024 21:16

deepyaman mentioned this pull request Jul 18, 2024

ci(exasol): add the astronauts dataset for testing #9635

Merged

deepyaman force-pushed the feat/from-dbapi-con branch 10 times, most recently from e892fb8 to fa7de69 Compare July 20, 2024 04:55

deepyaman marked this pull request as ready for review July 20, 2024 05:48

deepyaman force-pushed the feat/from-dbapi-con branch from fa7de69 to dfabfb3 Compare July 20, 2024 05:48

deepyaman force-pushed the feat/from-dbapi-con branch from dfabfb3 to a21de5b Compare July 20, 2024 15:40

deepyaman added 6 commits July 22, 2024 11:59

feat(trino): support backend create from dbapi con

251c81f

feat(bigquery): create a backend from_connection

0a9cc32

feat(snowflake): create from SnowflakeConnection

88d7214

docs(datafusion): capitalize the 'F' in DataFusion

4f3c4b3

feat(backends): add from_connection to top level

442c9f6

chore(backends): mark from_connection experimental

c8267e6

deepyaman force-pushed the feat/from-dbapi-con branch from 0633643 to c8267e6 Compare July 22, 2024 17:59

cpcloud approved these changes Jul 22, 2024

View reviewed changes

cpcloud added the backends Issues related to all backends label Jul 22, 2024

jcrist mentioned this pull request Jul 22, 2024

feat: to_duckdb #9629

Closed

1 task

cpcloud force-pushed the feat/from-dbapi-con branch from 4bd25c5 to ce48b22 Compare July 22, 2024 20:01

test: change test to not require schema information

ff776cb

cpcloud force-pushed the feat/from-dbapi-con branch from ce48b22 to 8556c0c Compare July 22, 2024 20:14

cpcloud added the ci-run-cloud Add this label to trigger a run of BigQuery, Snowflake, and Databricks backends in CI label Jul 22, 2024

ibis-docs-bot bot removed the ci-run-cloud Add this label to trigger a run of BigQuery, Snowflake, and Databricks backends in CI label Jul 22, 2024

deepyaman commented Jul 22, 2024

View reviewed changes

ibis/backends/bigquery/__init__.py Outdated Show resolved Hide resolved

deepyaman commented Jul 22, 2024

View reviewed changes

ibis/backends/bigquery/__init__.py Outdated Show resolved Hide resolved

feat(bigquery): implement from_connection

2184b77

cpcloud force-pushed the feat/from-dbapi-con branch from 8556c0c to 2184b77 Compare July 22, 2024 21:04

cpcloud added the ci-run-cloud Add this label to trigger a run of BigQuery, Snowflake, and Databricks backends in CI label Jul 22, 2024

ibis-docs-bot bot removed the ci-run-cloud Add this label to trigger a run of BigQuery, Snowflake, and Databricks backends in CI label Jul 22, 2024

cpcloud enabled auto-merge (squash) July 22, 2024 21:37

cpcloud merged commit fc4d1e3 into ibis-project:main Jul 22, 2024
88 checks passed

deepyaman deleted the feat/from-dbapi-con branch July 22, 2024 22:33

jcrist mentioned this pull request Jul 23, 2024

Add Option to Pass psychopg2 Connection #9051

Closed

1 task

This was referenced Jul 23, 2024

docs: replace all double graves with single graves #9679

Merged

depr(snowflake): deprecate from_snowpark method #9680

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(backends): support creation from a DB-API con #9603

feat(backends): support creation from a DB-API con #9603

deepyaman commented Jul 16, 2024

github-actions bot commented Jul 16, 2024

cpcloud Jul 16, 2024

deepyaman Jul 16, 2024

cpcloud Jul 16, 2024

deepyaman Jul 16, 2024

jcrist left a comment

deepyaman Jul 16, 2024

cpcloud Jul 16, 2024

deepyaman Jul 16, 2024

deepyaman commented Jul 20, 2024 •

edited

Loading

cpcloud left a comment

cpcloud commented Jul 22, 2024

cpcloud commented Jul 22, 2024 •

edited

Loading

cpcloud commented Jul 22, 2024

feat(backends): support creation from a DB-API con #9603

feat(backends): support creation from a DB-API con #9603

Conversation

deepyaman commented Jul 16, 2024

Description of changes

Issues closed

github-actions bot commented Jul 16, 2024

cpcloud Jul 16, 2024

Choose a reason for hiding this comment

deepyaman Jul 16, 2024

Choose a reason for hiding this comment

cpcloud Jul 16, 2024

Choose a reason for hiding this comment

deepyaman Jul 16, 2024

Choose a reason for hiding this comment

jcrist left a comment

Choose a reason for hiding this comment

deepyaman Jul 16, 2024

Choose a reason for hiding this comment

cpcloud Jul 16, 2024

Choose a reason for hiding this comment

deepyaman Jul 16, 2024

Choose a reason for hiding this comment

deepyaman commented Jul 20, 2024 • edited Loading

cpcloud left a comment

Choose a reason for hiding this comment

cpcloud commented Jul 22, 2024

cpcloud commented Jul 22, 2024 • edited Loading

cpcloud commented Jul 22, 2024

deepyaman commented Jul 20, 2024 •

edited

Loading

cpcloud commented Jul 22, 2024 •

edited

Loading