Merge pull request #83 from Deltares/v1/final_checks

V1/final checks

.zenodo.json

@@ -1,17 +1,17 @@
 {
-    "title": "hydroMT-sfincs",
-    "description": "SFINCS plugin for hydroMT.",
+    "title": "HydroMT-SFINCS",
+    "description": "SFINCS plugin for HydroMT.",
     "upload_type": "software",
     "creators": [
+        {
+            "affiliation":"Deltares",
+            "name": "de Goede, Roel"
+        },
         {
             "affiliation": "Deltares",
             "name": "Eilander, Dirk",
             "orcid": "0000-0002-0951-8418"
         },
-        {
-            "affiliation":"Deltares",
-            "name": "de Goede, Roel"
-        }
         {
             "affiliation": "Deltares",
             "name": "Leijnse, Tim",
@@ -21,8 +21,12 @@
             "affiliation": "Deltares",
             "name": "Winsemius, Hessel C.",
             "orcid": "0000-0001-5471-172X"
+        },
+        {
+            "affiliation": "Deltares-USA",
+            "name": "van Ormondt, Maarten"
         }
     ],
     "access_right": "open",
-    "license": "GPL-3.0",
+    "license": "GPLv3"
 }

README.rst

@@ -41,7 +41,7 @@ from conda-forge in a clean environment, see `installation guide. <docs_install>
 
 How to cite?
 ------------
-To reference the software please use the the DOI provided in the Zenodo badge |doi| that points to the latest release.
+To reference the software please use the the DOI provided in the Zenodo badge that points to the latest release |doi|.
 
 The following paper presents a real-world application of HydroMT-SFINCS:
 

docs/changelog.rst

@@ -5,41 +5,90 @@ All notable changes to this project will be documented in this page.
 The format is based on `Keep a Changelog`_, and this project adheres to
 `Semantic Versioning`_.
 
-Unreleased
-----------
+v1.0 (17 April 2023)
+====================
+
+This release is a major update of the SfincsModel interface. It contains many new features, 
+such as support for *rotated grids*, *subgrid* and improved support for *building models from Python* scripts.
+The documentation and exmaples have been updated to reflect these changes.
+
+The release however also contains several breaking changes as we have tried to improve the 
+consistency of the interface and match it more closely to the SFINCS model itself.
+Please carefully check the API reference for the new methods and arguments.
+
+Main differences
+----------------
+- `setup_region` has been replaced by `setup_grid_from_region` and  `setup_grid`. 
+  This method actually creates an empty regular grid based on a region of interest or user-defined coordinates, shape, rotation, etc..
+- `setup_dep` has replaced `setup_topobathy` and `setup_merge_topobathy`. 
+  This method can now also be used to setup a bathymetry map from multiple sources at once.
+- `setup_mask_active` has replaced `setup_mask`. 
+- `setup_mask_bounds` has replaced `setup_bounds`	
+- `setup_waterlevel_forcing` has replaced `setup_h_forcing` and now supports merging fording from several data sources 
+- `setup_discharge_forcing` has replaced `setup_q_forcing` and now supports merging fording from several data sources
+- `setup_discharge_forcing_from_grid` has replaces `setup_q_forcing_from_grid` 
+- `setup_precip_forcing` has replaced `setup_p_forcing` 
+- `setup_precip_forcing_from_grid` has replaced `setup_p_forcing_from_grid`
+- `setup_observation_points` has replace `setup_gauges`
+
+New methods
+-----------
+- `setup_grid` to setup a user-defined regular grid based coordinates, shape, rotation, etc.
+- `setup_subgrid` to setup subgrid tables (sbgfile) based on one ore more elevation and Manning roughness datasets
+- `setup_constant_infiltration` to setup a constant infiltration rate maps (qinffile)
+- `setup_waterlevel_bnd_from_mask` to setup water level boundary points (bndfile) based on the SFINCS model mask (mskfile)
+- `setup_tiles` to create tiles of the model for fast visualization
+
+Changed methods
+---------------
+- `setup_river_inflow` and `setup_river_outflow` are now based river centerline data (which can be derivded from hydrography data).
+  This is more robust compared to the previous method which was based on reprojected flow direction data.
+
+Removed methods (not replaced)
+------------------------------
+- `setup_basemaps` This method was already deprecated in v0.2.1 and has now been removed.
+- `setup_river_hydrography` This method was removed as reprojection of the hydrography data is no longer required for river inflow/outflow.
+- `setup_river_bathymetry` This method was removed as river bathymetry should ideally be burned in the subgrid data of the model rather 
+  than the dep file itself to be able to include rivers with widths smaller than the model grid cell size. A new option to burn rivers 
+  in the subgrid data will be added in to `setup_subgrid` a future release.
+
+
+New low-level classes
+---------------------
+These classes are not intended to be used directly by the user, but are used internally by the SfincsModel class.
+
+- The `SfincsInput` class contains methods to generate, read and write SFINCS input files
+- The `RegularGrid` class contains methods to create and manipulate regular grids
+- The `SubgridTableRegular` class contains methods to create and manipulate subgrid tables for regular grids
 
-Changed
-^^^^^^^
-- **setup_q_forcing_from_grid** also saves the snapped gauge locations.
-- **snap_discharge** method checks for valid (nonnull) discharge cells
 
 v0.2.1 (23 February 2022)
--------------------------
+=========================
 
 Deprecated
-^^^^^^^^^^
+----------
 - **setup_basemaps** has been replaced by **setup_topobathy**
 - In **setup_mask**, the "active_mask_fn" argument has been renamed to "mask_fn" for consistency
 - In **setup_river_inflow** and **setup_river_outflow** the "basemaps_fn" argument has been renamed to "hydrography_fn" for consistency
 - In **setup_river_outflow** the "outflow_width" argument has been renamed to "river_width" for consistency with setup_river_inflow
 - **setup_q_forcing_from_grid** and **workflows.snap_discharge** have a "rel_error" and "abs_error" argument instead of a single "max_error" argument.
 
 Bugfix
-^^^^^^
+------
 - bugfix **setup_p_forcing** to ensure the data is 1D when passed to set_forcing_1d method
 - bugfix **setup_p_forcing_from_grid** when aggregating with a multi polygon region.
 - bugfix **read_results** with new corner_x/y instead of edge_x/y dimensions in sfincs_map.nc
 
 New
-^^^
+---
 - **setup_region** method to set the (hydrological) model region of interest (before part of **setup_basemaps**).
 - **setup_river_hydrography** allows to derive hydrography data ['flwdir', 'uparea'] from the model elevation or reproject it from a global dataset.
   Derived 'uparea' and 'flwdir' maps are saved in the GIS folder and can be reused later (if kept together with the model)
 - **setup_river_bathymetry** to estimate a river depth based on bankfull discharge and river width. A mask of river cells 'rivmsk' is kept in the GIS folder.
 - Added parameter mapping file for ESA Worldcover dataset
 
 Changed
-^^^^^^^
+-------
 - **setup_mask** and **setup_bounds** both have a "mask_fn", "include_mask_fn" and "exclude_mask_fn" polygon and "min_elv" and "max_elv" elevation arguments to determine valid / boundary cells. 
 - **setup_mask** and **setup_bounds** have a "reset_mask" and "reset_bounds" option respectively to start with a clean mask or remove previously set boundary cells.
 - **setup_mask** takes a new "drop_area" argument to drop regions of contiguous cells smaller than this maximum area threshold, useful to remove (spurious) small islands.
@@ -62,15 +111,15 @@ Changed
 
 
 v0.2.0 (2 August 2021)
-----------------------
+======================
 
 Bugfix
-^^^^^^
+------
 - scsfile variable changed to maximum soil moisture retention [inch]; was curve number [-]
 - fix setting delimited text based geodatasets for h and q forcing.
 
 Changed
-^^^^^^^
+-------
 - Bumped minimal hydromt version to 0.4.2
 - splitted ``setup_topobathy`` into multiple smaller methods: ``setup_merge_topobathy``, ``setup_mask`` and ``setup_bounds``
 - separated many low-level methods into utils.py and plots.py
@@ -80,7 +129,7 @@ Changed
 
 
 Added
-^^^^^
+-----
 support for SFINCS files:
 
 - structures: sfincs.thd & sfincs.weir
@@ -106,31 +155,31 @@ new workflows:
 - ``river_inflow_points`` & ``river_outflow_points`` 
 
 Documentation
-^^^^^^^^^^^^^
+-------------
 - build from python example
 - overviews with SfincsModel setup components & SfincsModel data
 
 Deprecated
-^^^^^^^^^^^
+-----------
 - ``setup_p_gridded``
 
 v0.1.0 (18 May 2021)
---------------------
+====================
 Noticeable changes are a new ``setup_river_inflow`` and ``setup_river_outflow`` methods
 
 Added
-^^^^^
+-----
 
 - setup_river_outflow method to set ouflow (msk=3) boundary at river outflow points
 
 Changed
-^^^^^^^
+-------
 
 - Updated to hydromt v0.4.1
 
 
 Documentation
-^^^^^^^^^^^^^
+-------------
 
 - Now **latest** and **stable** versions.
 - Updated build instructions

docs/dev_guide/roadmap.rst

@@ -9,8 +9,10 @@ Short-term plans
 ----------------
 - Add the possibility to setup, read and write the pump and culvert to the :py:class:`~hydromt_sfincs.SfincsModel` class. Note that SFINCS already supports this functionality.
 - Next to the `inifile`, also support reading and writing of the `rstfile` option to restart a simulation from a previous state.
-- Add more eleborate options to account for spatially varying infiltration
+- Add more eleborate options to account for spatially varying infiltration.
+- Add functionality to burn rivers in subgrid tabels.
 - Add functionality to locally adjust subgrid tables, e.g. to account for green infrastructure.
+- Add support for other forcings such as wind and waves.
 
 Long-term plans
 ---------------

docs/getting_started/installation.rst

@@ -9,37 +9,34 @@ Prerequisites
 For more information about the prerequisites for an installation of the HydroMT package and related dependencies, please visit the
 documentation of `HydroMT core <https://deltares.github.io/hydromt/latest/getting_started/installation.html#installation-guide>`_
 
+.. Note::
+
+    We recommend using the `mamba` package manager. However, in the commands below you can exchange `mamba` for `conda`, see
+    `here <https://deltares.github.io/hydromt/latest/getting_started/installation.html#installation-guide>`_ for the difference between both.
+
 If you already have a python & conda installation but do not yet have mamba installed, 
-we recommend installing it into your *base* environment using:
+you can install it into your *base* environment using:
 
 .. code-block:: console
 
   $ conda install mamba -n base -c conda-forge
 
-
 Installation
 ============
 
 HydroMT-SFINCS is available from pypi and conda-forge. We recommend installing from conda-forge in a new conda environment.
 
-.. Note::
-
-    In the commands below you can exchange `mamba` for `conda`, see
-    `here <https://deltares.github.io/hydromt/latest/getting_started/installation.html#installation-guide>`_ for the difference between both.
 
 Install HydroMT-SFINCS in a new environment (recommended!)
 ----------------------------------------------------------
-You can install HydroMT-SFINCS in a new environment called `hydromt-sfincs` together with a few additional dependencies with:
+You can install HydroMT-SFINCS in a new environment called `hydromt-sfincs` together with a few additional dependencies 
+which are required for the HydroMT-SFINCS examples. To do so, create a new environment from the `environment.yml` file 
+in the repository root with the following command:
 
 .. code-block:: console
 
   $ mamba env create -f https://raw.githubusercontent.com/Deltares/hydromt_sfincs/main/environment.yml
 
-.. Note::
-
-    The environment yaml-files in https://raw.githubusercontent.com/Deltares/hydromt_sfincs/envs are meant for testing and development, 
-    so please use the one mentioned above. In case you want to develop, see the :ref:`Developer installation guide <dev_env>`.
-
 Then, activate the environment (as stated by mamba/conda) to start making use of HydroMT-Wflow:
 
 .. code-block:: console
@@ -52,12 +49,19 @@ Then, activate the environment (as stated by mamba/conda) to start making use of
     `conda env remove -n hydromt-sfincs` **or** set a new name for the environment 
     by adding `-n <name>` to the line above. 
 
+.. Note::
+
+    The environment yaml-files in `envs` folder are meant for testing and development, 
+    so please use the `environment.yml` file in the repository root for normal usage. 
+    In case you want to develop, see the :ref:`Developer installation guide <dev_env>`.
+
+
 Install HydroMT-SFINCS in an existing environment
 -------------------------------------------------
 
 To install HydroMT-SFINCS in an existing environment execute the command below 
 where you replace `<environment_name>` with the name of the existing environment. 
-Note that if some dependencies are not installed from conda-forge but from other 
+Note that if some dependencies are not installed from conda-forge but from pip or other 
 channels the installation may fail.
 
 .. code-block:: console

docs/user_guide/sfincs.rst

@@ -1,3 +1,5 @@
+.. _working_with_sfincs:
+
 =============================
 Working with the SFINCS model
 =============================
@@ -6,11 +8,14 @@ There are 2 main ways to use HydroMT to build your SFINCS model:
 
 **1. Command Line Interface (basic user)**: 
       Provide some information about the model configuration in a .yml-file and quickly build the model using the Command Line Interface (CLI).
+      The .yml-file provides a way to create a reproducible model setup recipe, which can be easily be shared with others.
+      Addtionally, no Python knowledge is required to use the CLI.
 
 **2. Python scripting (advanced user)**: 
       Dive into the underlying Python functions and use those to build your model from scratch in a Python script. 
-      This option is recommended when the user wants to (locally) adjust the model input data before building the model, 
-      e.g. in the case of modifications of the bed levels or variations on the boundary conditions.
+      This option is recommended for the expert user who wants to (locally) adjust the model input data as part of the model building process, 
+      e.g. in the case of in-memory modifications of the bed levels or variations on the boundary conditions.
+      The Python interface provides a lot of flexibility and access to the full HydroMT-SFINCS API, but requires some knowledge of Python.
 
 .. toctree::
     :maxdepth: 2

docs/user_guide/sfincs_build_update.rst

@@ -4,11 +4,11 @@
 Building or updating a model
 =============================
 
-This plugin allows users to build or update a SFINCS model from available data. 
-For beginning users, we recommend to use the :ref:`command line interface <sfincs_cli>` to build or update a model. 
-In case you want to (locally) modify the model input data before generating a model, we recommend to use the :ref:`Python scripting <sfincs_python>`. 
+This plugin allows users to build or update a SFINCS model from available data using 
+the :ref:`command line interface <sfincs_cli>` or :ref:`Python scripting <sfincs_python>`. 
+For a brief overview of the differences, see :ref:`Working with the SFINCS model <working_with_sfincs>`.
 
-In the following sections, examples are provided in iPython notebooks how to build your SFINCS model with HydroMT using either the CLI or python scripting.
+In the following sections, examples are provided how to build your SFINCS model with HydroMT using either the CLI or Python scripting.
 
 .. _sfincs_cli:
 
@@ -67,26 +67,28 @@ in its corresponding section. See the HydroMT core documentation for more info a
 Note that the order in which the components are listed in the yml-file is important (methods are executed from top to bottom): 
 
 - :py:func:`~hydromt_sfincs.SfincsModel.setup_grid` or :py:func:`~hydromt_sfincs.SfincsModel.setup_grid_from_region` should always be run first to define the model grid.
-- a lot of methods (e.g. :py:func:`~hydromt_sfincs.SfincsModel.setup_mask_active`) need elevation data to work properly, so :py:func:`~hydromt_sfincs.SfincsModel.setup_dep` should be run before most other methods.
-- if discharge locations are inferred from hydrography, :py:func:`~hydromt_sfincs.SfincsModel.setup_river_inflow` should be run before :py:func:`~hydromt_sfincs.SfincsModel.setup_discharge_forcing` or :py:func:`~hydromt_sfincs.SfincsModel.setup_discharge_forcing_from_grid`.
-
-Data libraries
-----------------
-
-Data sources in HydroMT are provided in one of several yaml libraries. These libraries contain required
-information on the different data sources so that HydroMT can process them for the different models. There
-are three ways for the user to select which data libraries to use:
-
-- If no yaml file is selected, HydroMT will use the data stored in the
-  `hydromt-artifacts <https://github.com/DirkEilander/hydromt-artifacts>`_
-  which contains an extract of global data for a small region around the Piave river in Northern Italy.
-- Another options for Deltares users is to select the deltares-data library (requires access to the Deltares
-  P-drive). In the command line interface, this is done by adding either **-dd** or **--deltares-data**
-  to the build / update command line.
-- Finally, the user can prepare its own yaml libary (or libraries) (see
+- Many methods (e.g., :py:func:`~hydromt_sfincs.SfincsModel.setup_mask_active`) need elevation data to work properly, hence :py:func:`~hydromt_sfincs.SfincsModel.setup_dep` should be run before most other methods.
+- If discharge locations are inferred from hydrography, :py:func:`~hydromt_sfincs.SfincsModel.setup_river_inflow` should be run before :py:func:`~hydromt_sfincs.SfincsModel.setup_discharge_forcing` or :py:func:`~hydromt_sfincs.SfincsModel.setup_discharge_forcing_from_grid`.
+- If water level bounary points are inferred from the water level mask cells,  :py:func:`~hydromt_sfincs.SfincsModel.setup_waterlevel_bnd_from_mask` should be run before :py:func:`~hydromt_sfincs.SfincsModel.setup_waterlevel_forcing`.
+
+Data Catalogs
+-------------
+
+Data sources are provided to HydroMT in one or more user-definfed data catalog (yaml) files 
+or from pre-defined data catalogs. These data catalogs contain required information on the 
+different data sources so that HydroMT can process them for the different models. 
+There are three ways for the user to select which data catalog to use:
+
+- There are several `pre-defined data catalog <https://deltares.github.io/hydromt/latest/user_guide/data_existing_cat.html>`_ 
+  Amongst other, these include the `deltares_data` data catalog for Deltares users which requires access to the Deltares P-drive. 
+  More pre-defined data catalogs will be added in the future.
+- Furthermore, the user can prepare its own yaml libary (or libraries) (see
   `HydroMT documentation <https://deltares.github.io/hydromt/latest/index>`_ to check the guidelines).
   These user libraries can be added either in the command line using the **-d** option and path/to/yaml or in the **yml file**
   with the **data_libs** option in the  `global` section (see example above).
+- Finally, if no catalog is provided, HydroMT will use the data stored in the
+  `hydromt-artifacts <https://github.com/DirkEilander/hydromt-artifacts>`_
+  which contains an extract of global data for a small region around the Piave river in Northern Italy.
 
 Example
 --------