Python: Migrate from iceberg to iceberg-python
#3
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Bumps [duckdb](https://github.com/duckdb/duckdb) from 0.7.0 to 0.7.1. - [Release notes](https://github.com/duckdb/duckdb/releases) - [Changelog](https://github.com/duckdb/duckdb/blob/master/tools/release-pip.py) - [Commits](duckdb/duckdb@v0.7.0...v0.7.1) --- updated-dependencies: - dependency-name: duckdb dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [coverage](https://github.com/nedbat/coveragepy) from 7.2.0 to 7.2.1. - [Release notes](https://github.com/nedbat/coveragepy/releases) - [Changelog](https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst) - [Commits](nedbat/coveragepy@7.2.0...7.2.1) --- updated-dependencies: - dependency-name: coverage dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
When we pass in a warehouse identifier, we don't want to emit a warning. Also, the help message won't be helpful: ``` No preferred file implementation for scheme: ``` Since there is no scheme.
Bumps [moto](https://github.com/getmoto/moto) from 4.1.3 to 4.1.4. - [Release notes](https://github.com/getmoto/moto/releases) - [Changelog](https://github.com/getmoto/moto/blob/master/CHANGELOG.md) - [Commits](getmoto/moto@4.1.3...4.1.4) --- updated-dependencies: - dependency-name: moto dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [pydantic](https://github.com/pydantic/pydantic) from 1.10.5 to 1.10.6. - [Release notes](https://github.com/pydantic/pydantic/releases) - [Changelog](https://github.com/pydantic/pydantic/blob/v1.10.6/HISTORY.md) - [Commits](pydantic/pydantic@v1.10.5...v1.10.6) --- updated-dependencies: - dependency-name: pydantic dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [fastavro](https://github.com/fastavro/fastavro) from 1.7.2 to 1.7.3. - [Release notes](https://github.com/fastavro/fastavro/releases) - [Changelog](https://github.com/fastavro/fastavro/blob/master/ChangeLog) - [Commits](fastavro/fastavro@1.7.2...1.7.3) --- updated-dependencies: - dependency-name: fastavro dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Integration tests * First version * Add caching * Add caching * Restore pyproject * WIP * NaN seems to be broken * WIP * Coming along * Cleanup * Install duckdb * Cleanup * Revert changes to poetry * Make it even nicer * Revert unneeded change * Update Spark version * Make test passing * comments
Bumps [coverage](https://github.com/nedbat/coveragepy) from 7.2.1 to 7.2.2. - [Release notes](https://github.com/nedbat/coveragepy/releases) - [Changelog](https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst) - [Commits](nedbat/coveragepy@7.2.1...7.2.2) --- updated-dependencies: - dependency-name: coverage dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
… conflict (#7136)
* Python: Add support for ORC Creates fragments based on the FileFormat. Blocked by: apache/iceberg#6997 * Revert * TableScan add limit * pyarrow limit number of rows fetched from files if limit is set * add tests for scan limit * python ci rebuild container if changes on python/dev/ * remove support for ORC * remove unused imports * increase sleep before running tests * update python docs to include limit in table query * docs fix format --------- Co-authored-by: Fokko Driesprong <[email protected]> Co-authored-by: Daniel Rückert García <[email protected]>
* Python: Use Prettier to format TOML and more files * Python: Make Prettier skip the mkdocs admonoitions
Bumps [rich](https://github.com/Textualize/rich) from 13.3.3 to 13.3.4. - [Release notes](https://github.com/Textualize/rich/releases) - [Changelog](https://github.com/Textualize/rich/blob/master/CHANGELOG.md) - [Commits](Textualize/rich@v13.3.3...v13.3.4) --- updated-dependencies: - dependency-name: rich dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.3.0 to 7.3.1. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pytest@7.3.0...7.3.1) --- updated-dependencies: - dependency-name: pytest dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [moto](https://github.com/getmoto/moto) from 4.1.6 to 4.1.7. - [Release notes](https://github.com/getmoto/moto/releases) - [Changelog](https://github.com/getmoto/moto/blob/master/CHANGELOG.md) - [Commits](getmoto/moto@4.1.6...4.1.7) --- updated-dependencies: - dependency-name: moto dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [zstandard](https://github.com/indygreg/python-zstandard) from 0.20.0 to 0.21.0. - [Release notes](https://github.com/indygreg/python-zstandard/releases) - [Changelog](https://github.com/indygreg/python-zstandard/blob/main/docs/news.rst) - [Commits](indygreg/python-zstandard@0.20.0...0.21.0) --- updated-dependencies: - dependency-name: zstandard dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Add register table implementation for sql catalog * Fix lint
* Python: Add __repr__ method to Table class * update test for __repr__ of Table class * add sort_order to table repr --------- Co-authored-by: Thi Cam Tu Phan <[email protected]>
* Python: improve Glue catalog using Boto3 types * fix: address PR feedback --------- Co-authored-by: Fokko Driesprong <[email protected]>
* Python: Add `__repr__` for `Table` and `Catalog` * remove properties in `__repr__` output * remove table `__repr__` * clean up catalog __repr__ and add test * actually use repr instead of str * Update python/tests/catalog/test_base.py Co-authored-by: Fokko Driesprong <[email protected]> * Update python/pyiceberg/catalog/__init__.py * Update python/tests/catalog/test_base.py --------- Co-authored-by: Fokko Driesprong <[email protected]>
… with `ruff` (#8408)
Bumps [mkdocs](https://github.com/mkdocs/mkdocs) from 1.5.2 to 1.5.3. - [Release notes](https://github.com/mkdocs/mkdocs/releases) - [Commits](mkdocs/mkdocs@1.5.2...1.5.3) --- updated-dependencies: - dependency-name: mkdocs dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…#8632) Bumps [mkdocs-material-extensions](https://github.com/facelessuser/mkdocs-material-extensions) from 1.1.1 to 1.2. - [Release notes](https://github.com/facelessuser/mkdocs-material-extensions/releases) - [Changelog](https://github.com/facelessuser/mkdocs-material-extensions/blob/master/changelog.md) - [Commits](facelessuser/mkdocs-material-extensions@1.1.1...1.2) --- updated-dependencies: - dependency-name: mkdocs-material-extensions dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [moto](https://github.com/getmoto/moto) from 4.2.3 to 4.2.4. - [Release notes](https://github.com/getmoto/moto/releases) - [Changelog](https://github.com/getmoto/moto/blob/master/CHANGELOG.md) - [Commits](getmoto/moto@4.2.3...4.2.4) --- updated-dependencies: - dependency-name: moto dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.4.1 to 7.4.2. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pytest@7.4.1...7.4.2) --- updated-dependencies: - dependency-name: pytest dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.3.1 to 9.4.1. - [Release notes](https://github.com/squidfunk/mkdocs-material/releases) - [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG) - [Commits](squidfunk/mkdocs-material@9.3.1...9.4.1) --- updated-dependencies: - dependency-name: mkdocs-material dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [mkdocs-literate-nav](https://github.com/oprypin/mkdocs-literate-nav) from 0.6.0 to 0.6.1. - [Release notes](https://github.com/oprypin/mkdocs-literate-nav/releases) - [Commits](oprypin/mkdocs-literate-nav@v0.6.0...v0.6.1) --- updated-dependencies: - dependency-name: mkdocs-literate-nav dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* add ManifestWriter and ManifestListWriter * fix lint issue * remove assert, fix format issue * add prepare to ManifestWriter, remove TODO, fix format issue * fix some nit issue, add prepare... to ensure the correctness of data written * fix format issue * fix lint issue * avoid creating too much objects, handling v1, v2 data_file_type and DataFile class properly. * modify tests * refactor the way of handling two version of DataFile record * add integration tests, fix bugs, change PartitionSummary to a function * fix format issue * make data_type_v2 constants
Fokko
force-pushed
the
fd-with-history
branch
3 times, most recently
from
382ec5a
to
010bf26
Compare
Fokko
force-pushed
the
fd-with-history
branch
from
010bf26
to
0a9f113
Compare
ajantha-bhat
approved these changes
Sep 29, 2023
nastra
approved these changes
Sep 29, 2023
rdblue
approved these changes
Sep 29, 2023
hussein-awala
approved these changes
Sep 30, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This time with history