Skip to content

Commit

Permalink
add warning for large delta memory footprint on filesystem docs page (#…
Browse files Browse the repository at this point in the history
…2036)

* add warning for large delta memory footprint

* fix wording
  • Loading branch information
sh-rp authored Nov 7, 2024
1 parent 25550bb commit 95ca6e6
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions docs/website/docs/dlt-ecosystem/destinations/filesystem.md
Original file line number Diff line number Diff line change
Expand Up @@ -645,6 +645,10 @@ def my_delta_resource():

> `dlt` always uses Parquet as `loader_file_format` when using the `delta` table format. Any setting of `loader_file_format` is disregarded.
:::caution
Beware that when loading a large amount of data for one table, the underlying rust implementation will consume a lot of memory. This is a known issue and the maintainers are actively working on a solution. You can track the progress [here](https://github.com/delta-io/delta-rs/pull/2289). Until the issue is resolved, you can mitigate the memory consumption by doing multiple smaller incremental pipeline runs.
:::

#### Delta table partitioning
A Delta table can be partitioned ([Hive-style partitioning](https://delta.io/blog/pros-cons-hive-style-partionining/)) by specifying one or more `partition` column hints. This example partitions the Delta table by the `foo` column:

Expand Down

0 comments on commit 95ca6e6

Please sign in to comment.