You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Seem like my Google Cloud ADC isn't used when table_format is 'delta'.
Got "OSError: Generic GCS error: GCP credential error: Unable to decode service account file: invalid type: null"
Destination: dlt.destinations.filesystem with no setting for credentials
Code run just fine if I don't set table_format argument.
Code using the write_deltalake function of deltalake package also run fine (same data, same gcs bucket)
Expected behavior
Expect my data to write to GCS in delta format
Steps to reproduce
My sample code:
import dlt
from dlt.destinations import filesystem
@dlt.resource(
table_name="test_delta_table",
write_disposition={"disposition": "merge", "strategy": "upsert"},
primary_key="id",
table_format="delta"
)
def get_list():
data = [
{'id': 1, 'name': 'DAT1'},
{'id': 2, 'name': 'DAT2'},
{'id': 3, 'name': 'DAT3'},
]
for d in data:
yield d
dlt_destination = filesystem(
bucket_url='gs://sample_bucket'
)
pipeline = dlt.pipeline(
destination=dlt_destination,
)
load_info = pipeline.run(get_list())
Stacktrace:
Traceback (most recent call last):
File "/Users/toanlt/miniconda3/envs/fiin_scrapper_env/lib/python3.10/site-packages/dlt/common/destination/reference.py", line 394, in run_managed
self.run()
File "/Users/toanlt/miniconda3/envs/fiin_scrapper_env/lib/python3.10/site-packages/dlt/destinations/impl/filesystem/filesystem.py", line 125, in run
delta_table = self._delta_table()
File "/Users/toanlt/miniconda3/envs/fiin_scrapper_env/lib/python3.10/site-packages/dlt/destinations/impl/filesystem/filesystem.py", line 169, in _delta_table
if DeltaTable.is_deltatable(self.make_remote_url(), storage_options=self._storage_options):
File "/Users/toanlt/miniconda3/envs/fiin_scrapper_env/lib/python3.10/site-packages/deltalake/table.py", line 462, in is_deltatable
return RawDeltaTable.is_deltatable(table_uri, storage_options)
OSError: Generic GCS error: GCP credential error: Unable to decode service account file: invalid type: null, expected a string at line 1 column 152
Operating system
macOS
Runtime environment
Local
Python version
3.10
dlt data source
Just a list of dict
dlt destination
Filesystem & buckets
Other deployment details
No response
Additional information
No response
The text was updated successfully, but these errors were encountered:
dlt version
dlt 1.2.0
Describe the problem
Seem like my Google Cloud ADC isn't used when table_format is 'delta'.
Got "OSError: Generic GCS error: GCP credential error: Unable to decode service account file: invalid type: null"
Destination: dlt.destinations.filesystem with no setting for credentials
Code run just fine if I don't set table_format argument.
Code using the write_deltalake function of deltalake package also run fine (same data, same gcs bucket)
Expected behavior
Expect my data to write to GCS in delta format
Steps to reproduce
My sample code:
Stacktrace:
Operating system
macOS
Runtime environment
Local
Python version
3.10
dlt data source
Just a list of dict
dlt destination
Filesystem & buckets
Other deployment details
No response
Additional information
No response
The text was updated successfully, but these errors were encountered: