Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Got GCP credential error when using delta table_format and GCS filesystem #1965

Open
toanlekafi opened this issue Oct 18, 2024 · 0 comments · May be fixed by #2041
Open

Got GCP credential error when using delta table_format and GCS filesystem #1965

toanlekafi opened this issue Oct 18, 2024 · 0 comments · May be fixed by #2041
Assignees
Labels
bug Something isn't working

Comments

@toanlekafi
Copy link

dlt version

dlt 1.2.0

Describe the problem

Seem like my Google Cloud ADC isn't used when table_format is 'delta'.

Got "OSError: Generic GCS error: GCP credential error: Unable to decode service account file: invalid type: null"

Destination: dlt.destinations.filesystem with no setting for credentials
Code run just fine if I don't set table_format argument.
Code using the write_deltalake function of deltalake package also run fine (same data, same gcs bucket)

Expected behavior

Expect my data to write to GCS in delta format

Steps to reproduce

My sample code:

import dlt
from dlt.destinations import filesystem

@dlt.resource(
    table_name="test_delta_table",
    write_disposition={"disposition": "merge", "strategy": "upsert"},
    primary_key="id",
    table_format="delta"
)
def get_list():
    data = [
        {'id': 1, 'name': 'DAT1'},
        {'id': 2, 'name': 'DAT2'},
        {'id': 3, 'name': 'DAT3'},
    ]

    for d in data:
        yield d

dlt_destination = filesystem(
    bucket_url='gs://sample_bucket'
)

pipeline = dlt.pipeline(
    destination=dlt_destination,
)

load_info = pipeline.run(get_list())

Stacktrace:

Traceback (most recent call last):
  File "/Users/toanlt/miniconda3/envs/fiin_scrapper_env/lib/python3.10/site-packages/dlt/common/destination/reference.py", line 394, in run_managed
    self.run()
  File "/Users/toanlt/miniconda3/envs/fiin_scrapper_env/lib/python3.10/site-packages/dlt/destinations/impl/filesystem/filesystem.py", line 125, in run
    delta_table = self._delta_table()
  File "/Users/toanlt/miniconda3/envs/fiin_scrapper_env/lib/python3.10/site-packages/dlt/destinations/impl/filesystem/filesystem.py", line 169, in _delta_table
    if DeltaTable.is_deltatable(self.make_remote_url(), storage_options=self._storage_options):
  File "/Users/toanlt/miniconda3/envs/fiin_scrapper_env/lib/python3.10/site-packages/deltalake/table.py", line 462, in is_deltatable
    return RawDeltaTable.is_deltatable(table_uri, storage_options)
OSError: Generic GCS error: GCP credential error: Unable to decode service account file: invalid type: null, expected a string at line 1 column 152

Operating system

macOS

Runtime environment

Local

Python version

3.10

dlt data source

Just a list of dict

dlt destination

Filesystem & buckets

Other deployment details

No response

Additional information

No response

@rudolfix rudolfix added the bug Something isn't working label Oct 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: In Progress
Development

Successfully merging a pull request may close this issue.

3 participants