Datacube v1.8.11

Administrative release to address two technical issues in release 1.8.10:

1. 1.8.10 attempted to improve compatibility with recent versions of xarray, however further work was needed to fully achieve that compatibility.
2. 1.8.10 was released with dependency discrepancies between setup.py (pip), conda-environment.yml (conda) and constrains.in (Docker).

Full list of changes:

• Simplify Github actions (#1393)
• Update conda create environment README (#1394)
• Update conda environment file and add notes to release process to ensure pip and conda
  dependencies are in sync and up-to-date. (#1395)
• Update docker constraints (#1396)
• Compatible with the changes w.r.t. MultiIndex and coord/dims introduced since xarray>2022.3.0 (#1397)
• Final sync of conda/pip dependencies and update release notes. (#1398)

Includes contributions from @SpacemanPaul, @pindge, @Ariana-B, @emmaai and @omad.

Datacube v1.8.10

Bug fixes, enhancements, and updates to work with newer versions of upstream libraries.

Notes

1. The new APIs for bulk-reads, bulk-writes and index cloning should be considered unstable and may change in subsequent releases.
2. Recent refactoring in the XArray library has lead to changes in behaviour that affect some ODC operations and are unlikely to be addressed by the XArray team. This release includes changes in the way the ODC works with XArray to circumvent these issues. If you experience Xarray issues with this ODC release, please raise an issue on Github and we will try to address them before the next release.

Full list of changes:

• Add grid_spec to list_products (#1357)
• Add database relationship diagram to doc (#1350)
• Maintain search field index tables, and use them for dataset queries (#1360)
• Change Github lint action to use conda and remove flake8 from action (#1361)
• Fix database relationship diagram instruction for docker (#1362)
• Document group_by for dataset.load (#1364)
• Add search_by_metadata facility for products (#1366)
• Postgis driver cleanup - remove faux support for lineage (#1368)
• Add support for nested database transactions (#1369)
• Fix Github doc lint action (#1370)
• Tighten EO3 enforcement in postgis driver, refactor tests, and rename Dataset.type to Dataset.product
  (with type alias for compatibility) (#1372)
• Fix deprecation message due to distutils Version classes (#1375)
• Postgresql drivers cleanup - consolidate split_uri into utils and removed unused constants (#1378)
• Postgresql drivers cleanup - Handle NaNs in search fields and allow caching in sanitise_extent (#1379)
• Fix example product yaml documentation (#1384)
• Bulk read/write API methods and fast whole-index cloning. Cloning does NOT include lineage information yet,
  and new API methods may be subject to change. (#1381)
• Documentation update. (#1385)
• Clean up datetime functions (#1387)
• Dependency updates (#1388, #1391)
• Upgrades for compatibility with newer versions of Shapely and Xarray. (#1389)
• Finalise release notes for 1.8.10 release (#1392)

Includes contributions from: @pindge, @omad, @Ariana-B, @sanjeev0210 and @SpacemanPaul

Datacube v1.8.9

Most notable changes:

1. patch_url argument to dc.load() and dc.load_data() (introduced in v1.8.8) is now also supported for Dask loading.
2. Fixed a day-zero bug affecting search over range-type search fields where the target and search field value is of zero-width.
3. Performance improvements to CRS geometry class.
4. Numerous improvements to documentation and github actions.

Full list of changes:

• Performance improvements to CRS geometry class (#1322)
• Extend patch_url argument to dc.load() and dc.load_data() to Dask loading. (#1323)
• Add sphinx.ext.autoselectionlabel extension to readthedoc conf to support :ref: command (#1325)
• Add pyspellcheck for .rst documentation files and fix typos (#1327)
• Add rst documentation lint github action and apply best practices (#1328)
• Follow PEP561_ to make type hints available to other packages (#1331)
• Updated GitHub actions config to remove deprecated set-output (#1333)
• Add what's new page link to menu and general doc fixes (#1335)
• Add search_fields to required for metadata type schema and update doc (#1339)
• Fix typo and update metadata documentation (#1340)
• Add readthedoc preview github action (#1344)
• Update nodata in readthedoc for products page (#1347)
• Add eo-datasets to extensions & related software doc page (#1349)
• Fix bug affecting searches against range types of zero width (#1352)
• Add 1.8.9 release date and missing PR to whats_news.rst (#1353)

Includes contributions from @SpacemanPaul, @omad, @pindge, @snowman2.

With thanks and appreciation to all contributors, users and supporting organisations, expecially Geoscience Australia.

Datacube v1.8.8

Ofiicial release (Same as 1.8.8-rc1)

Most notable new features are:

1. the new database transaction API, as discussed in ODC-EP07 Database Transaction API. Simple example:

with dc.index.transaction() as trans:
   # Archive old datasets and add new ones in single transaction
   dc.index.datasets.archive([old_ds1.id, old_ds2.id])
   dc.index.datasets.add(ds1)
   dc.index.datasets.add(ds2)

   # If execution gets to here, the transaction is committed.
   # If an exception was raised by any of the above methods, the transaction is rolled back.

2. Add patch_url argument to dc.load and dc.load_data allowing signing of URIs as required by some commercial data providers (e.g. Microsoft Planetary Computer).

Also includes an update of the main test docker build to Ubuntu 22.04 and Python 3.10, and significant progress on the new postgis index driver.

Please note that the postgis index driver is still flagged as "experimental" and is missing several key features.

The release is almost identical to 1.8.8rc1. Full list of changes since 1.8.7:

• Migrate main test docker build to Ubuntu 22.04 and Python 3.10. (#1283)
• Dynamically create tables to serve as spatial indexes in postgis driver. (#1312)
• Populate spatial index tables, automatically and manually. (#1314)
• Perform spatial queries against spatial index tables in postgis driver. (#1316)
• EO3 data fixtures and tests. Fix SQLAlchemy bugs in postgis driver. (#1309)
• Dependency updates. (#1308, #1313)
• Remove several features that had been deprecated in previous releases. (#1275)
• Fix broken paths in api docs. (#1277)
• Fix readthedocs build. (#1269)
• Add support for Jupyter Notebooks pages in documentation (#1279)
• Add doc change comparison for tuple and list types with identical values (#1281)
• Add flake8 to Github action workflow and correct code base per flake8 rules (#1285)
• Add dataset id check to dataset doc resolve to prevent uuid returning error when id used in None (#1287)
• Add how to run targeted single test case in docker guide to README (#1288)
• Add help message for all dataset, product and metadata subcommands when required arg is not passed in (#1292)
• Add error code 1 to all incomplete dataset, product and metadata subcommands (#1293)
• Add exit_on_empty_file message to product and dataset subcommands instead of returning no output when file is empty (#1294)
• Add flags to index drivers advertising what format datasets they support (eo/eo3/non-geo (e.g. telemetry only))
  and validate in the high-level API. General refactor and cleanup of eo3.py and hl.py. (#1296)
• Replace references to 'agdc' and 'dataset_type' in postgis driver with 'odc' and 'product'. (#1298)
• Add warning message for product and metadata add when product and metadata is already in the database. (#1299)
• Ensure SimpleDocNav.id is of type UUID, to improve lineage resolution (#1304)
• Replace SQLAlchemy schema and query definitions in experimental postgis driver with newer "declarative" style ORM.
  Portions of API dealing with lineage handling, locations, and dynamic indexes are currently broken in the postgis
  driver. As per the warning message, the postgis driver is currently flagged as "experimental" and is not considered
  stable. (#1305)
• Implement patch_url argument to dc.load() and dc.load_data() to provide a way to sign dataset URIs, as
  is required to access some commercial archives (e.g. Microsoft Planetary Computer). API is based on the odc-stac
  implementation. Only works for direct loading. More work required for deferred (i.e. Dask) loading. (#1317)
• Implement public-facing index-driver-independent API for managing database transactions, as per Enhancement Proposal
  EP07 (#1318)
• Update Conda environment to match dependencies in setup.py (#1319)
• Final updates to whats_new.rst for release (#1320)

Includes contributions from @SpacemanPaul @tijmenr @pindge and @omad

Thanks to the ODC Steering Council and Geoscience Australia for their ongoing support of ODC development.

1.8.8rc1

RC release to facilitate development in downstream packages using the new transaction API.

Most notable new feature is the new database transaction API, as discussed in ODC-EP07 Database Transaction API. API Example:

with dc.index.transaction() as trans:
   # Archive old datasets and add new ones in single transaction
   dc.index.datasets.archive([old_ds1.id, old_ds2.id])
   dc.index.datasets.add(ds1)
   dc.index.datasets.add(ds2)

   # If execution gets to here, the transaction is committed.
   # If an exception was raised by any of the above methods, the transaction is rolled back.

Also includes an update of the main test docker build to Ubuntu 22.04 and Python 3.10, and significant progress on the new postgis index driver.

Please note that the postgis index driver is still flagged as "experimental" and is missing several key features.

Full list of changes since 1.8.7:

• Migrate main test docker build to Ubuntu 22.04 and Python 3.10. (#1283)
• Dynamically create tables to serve as spatial indexes in postgis driver. (#1312)
• Populate spatial index tables, automatically and manually. (#1314)
• Perform spatial queries against spatial index tables in postgis driver. (#1316)
• EO3 data fixtures and tests. Fix SQLAlchemy bugs in postgis driver. (#1309)
• Dependency updates. (#1308, #1313)
• Remove several features that had been deprecated in previous releases. (#1275)
• Fix broken paths in api docs. (#1277)
• Fix readthedocs build. (#1269)
• Add doc change comparison for tuple and list types with identical values (#1281)
• Add flake8 to Github action workflow and correct code base per flake8 rules (#1285)
• Add dataset id check to dataset doc resolve to prevent uuid returning error when id used in None (#1287)
• Add how to run targeted single test case in docker guide to README (#1288)
• Add help message for all dataset, product and metadata subcommands when required arg is not passed in (#1292)
• Add error code 1 to all incomplete dataset, product and metadata subcommands (#1293)
• Add exit_on_empty_file message to product and dataset subcommands instead of returning no output when file is empty (#1294)
• Add flags to index drivers advertising what format datasets they support (eo/eo3/non-geo (e.g. telemetry only))
  and validate in the high-level API. General refactor and cleanup of eo3.py and hl.py. (#1296)
• Replace references to 'agdc' and 'dataset_type' in postgis driver with 'odc' and 'product'. (#1298)
• Add warning message for product and metadata add when product and metadata is already in the database. (#1299)
• Ensure SimpleDocNav.id is of type UUID, to improve lineage resolution (#1304)
• Replace SQLAlchemy schema and query definitions in experimental postgis driver with newer "declarative" style ORM.
  Portions of API dealing with lineage handling, locations, and dynamic indexes are currently broken in the postgis
  driver. As per the warning message, the postgis driver is currently flagged as "experimental" and is not considered
  stable. (#1305)
• Implement patch_url argument to dc.load() and dc.load_data() to provide a way to sign dataset URIs, as
  is required to access some commercial archives (e.g. Microsoft Planetary Computer). API is based on the odc-stac
  implementation. Only works for direct loading. More work required for deferred (i.e. Dask) loading. (#1317)
• Implement public-facing index-driver-independent API for managing database transactions, as per Enhancement Proposal
  EP07 (#1318)
• Update Conda environment to match dependencies in setup.py (#1319)

1.8.7

• Cleanup mypy typechecking compliance. (#1266)
• When dataset add operations fail due to lineage issues, the produced error message now clearly indicates that
  the problem was due to lineage issues. (#1260)
• Added support for group-by financial years to virtual products. (#1257, #1261)
• Remove reference to rasterio.path. (#1255)
• Cleaner separation of (experimental) postgis and (stable) postgres drivers, and suppress SQLAlchemy cache warnings. (#1254)
• Prevent Shapely deprecation warning. (#1253)
• Fix DATACUBE_DB_URL parsing to understand syntax like: postgresql:///datacube?host=/var/run/postgresql (#1256)
• Clearer error message when local metadata file does not exist. (#1252)
• Address upstream security alerts and update upstream library versions. (#1250)
• Clone postgres index driver as postgis, and flag as experimental. (#1248)
• Implement a local non-persistent in-memory index driver, with maximal backwards-compatibility
  with default postgres index driver. Doesn't work with CLI interface, as every invocation
  will receive a new, empty index, but useful for testing and small scale proof-of-concept
  work. (#1247)
• Performance and correctness fixes backported from odc-geo. (#1242)
• Deprecate use of the celery executor. Update numpy pin in rtd-requirements.txt to suppress
  Dependabot warnings. (#1239)
• Implement a minimal "null" index driver that provides an always-empty index. Mainly intended
  to validate the recent abstraction work around the index driver layer, but may be useful
  for some testing scenarios, and ODC use cases that do not require an index. (#1236)
• Regularise some minor API inconsistencies and restore redis-server to Docker image. (#1234)
• Move (default) postgres driver-specific files from datacube.index to datacube.index.postgres.
  datacube.index.Index is now an alias for the abstract base class index interface definition
  rather than postgres driver-specific implementation of that interface. (#1231)
• Update numpy and netcdf4 version in docker build (#1229)
  rather than postgres driver-specific implementation of that interface. (#1227)
• Migrate test docker image from datacube/geobase to osgeo/gdal. (#1233)
• Separate index driver interface definition from default index driver implementation. (#1226)
• Prefer WKT over EPSG when guessing CRS strings. (#1223, #1262)
• Updates to documentation. (#1208, #1212, #1215, #1218, #1240, #1244)
• Tweak to segmented in geometry to suppress Shapely warning. (#1207)
• Fix to ensure skip_broken_datasets is correctly propagated in virtual products (#1259)
• Deprecate Rename, Select and ToFloat built-in transforms in virtual products (#1263)

Includes contributions from @whatnick, @alexgleith, @maawoo, @jeremyh, @iamtekson, @alfredoahds, @SpacemanPaul, @Kirill888, @robbitbt, @tebadi, @uchchwhash, and @mpaget.

Acknowledgements to the Open Datacube Steering Council and all supporting organisations, including Geoscience Australia, Digital Earth Africa, CSIRO, Frontier SI and Aerometrex.

1.8.6

• Added dataset purge command for hard deletion of archived datasets #1199
• Trivial fixes to CLI help output #1197
• Fix to enable searching for multiple products #1201

1.8.5

• Fix unguarded dependencies on boto libraries #1174 #1172
• Various documentation fixes #1175
• Address import problems on Windows due to use of Unix only functions #1176
• Address numpy.bool deprecation warnings #1184

Datacube v1.8.4

v1.8.4 (6 August 2021)

• Removed example and contributed notebooks from the repository. Better notebook examples exist
• Removed datacube_apps, as these are not used and not maintained
• Add cloud_cover to EO3 metadata
• Add erosion functionality to Virtual products' ApplyMask to supplement existing dilation functionality #1049
• Fix numeric precision issues in compute_reproject_roi when pixel size is small #1047
• Follow up fix to #1047 to round scale to nearest integer if very close
• Add support for 3D Datasets #1099
• New feature: search by URI from the command line datacube dataset uri-search
• Added new "license" and "description" properties to DatasetType to enable easier access to product information #1143 #1144
• Revised the Datacube.list_products function to produce a simpler and more useful product list table #1145
• Refactor docs, making them more up to date and simpler #1137 #1128
• Add new dataset_predicate param to dc.load and dc.find_datasets for more flexible temporal filtering (e.g. loading data for non-contiguous time ranges such as specific months or seasons over multiple years) #1148 #1156
• Fix to GroupBy to ensure output output axes are correctly labelled when sorting observations using sort_key #1157
• GroupBy is now its own class to allow easier custom grouping and sorting of data #1157
• add support for IAM authentication for RDS databases in AWS #1168

1.8.3

• More efficient band alias handling
• More documentation cleanups
• Bug fixes in datacube.utils.aws, credentials handling when AWS_UNSIGNED is set
• Product definition can now optionally include per-band scaling factors
• Fix issue where new updated columns aren't created on a fresh database
• Fix bug around adding updated columns locking up active databases