Updating the env_fix branch with changes from main (#182)

* Update README.rst

Removed outdated license text

* Update pyproject.toml

Update license

* Modifications made after Frederique's review

Modifications made after Frederique's review. Hydromt sfincs functions are out of hydromt fiat now. Every dataset required for hydromt fiat can be done in advace with hydromt sfincs.

* update directory path

* Making hazard functions explicits after Dirk review

Makin hazard function explicit after Dirk review

* Change of .rio for .raster hyndromt core function

Change of .rio for .raster hyndromt core function

* Adding parameter geom when reading raster data

Adding parameter geom when reading raster data. It requires that self.region is always available.

* Removing test using sfincs_map.nc

Removing test using sfincs_map.nc. They are not processed anymore through hydromt fiat.

* Updated building footprints code and test, added assert statements

* Added to gitignore

* Cleaned folder structure

* inner merge to not discard values

* add check that original and new exposure have same length

* Update, function does not yet work

* Added example to update the ground floor height and finalized code and test

* Added polygons to overlay

* Deleted output files

* split the get_table_and_metadata function into two functions

* Update the geoms and table properties of the FiatModel now only when writing the model, not before.

* Updated settings.toml configuration file output

* Update docs.yml

* update by copying from hydromt

* test docs

* changed hydromt to hydromt-fiat-docs

* test

* added aggregation zones example

* test docs workflow

* update

* another try

* update

* try again

* Update paths

* update paths

* Fixed small bugs in example

* Start of integration into the code base

* Added Sarah as author

* Bug fixes and update in fiat.py to read from the correct config variable

* implemented empty functions for the roads and changed name of setup_exposure_vector function to setup_exposure_buildings

* spelling mistake

* removed docs branch in workflow

* setting up functions

* function for hazared and fraction values, create test for function

* Update

* update test

* add unit test road

* update function and test

* update, roads test passed succesfully

* update test roads

* Tested and complete for now

* Update LICENSE

* add roadnotebook

* update road notebook

* Small update

* update notebook roads

* finalize road notebook

* update file paths

* update dictionary aggregation

* Updated logo

* update configuration.yaml and road notebook

* update aggregation notebook and config

* Updating the Z-score method (small change)

* Update on view model

* update

* update notebook, config and gitignore

* update notebook, add 2nd config

* create 2 config files. Clean up notebook

* added to gitignore

* add check that polygons dont overlap

* remove unused package

* add notebook output files to gitignore

* update absolute to relative path

* writing a model is now possible from the GUI

* added ground floor height (found_ht)

* getting ground floor height from NSI data and setting geometry in a different way

* Enable taking the GFH from the NSI immediately

* update examples data

* Update settings file and git ignore file

Update settings file and git ignore file

* Update gitignore

Update gitignore

* Changing .gpkg files

Changing .gpkg files

* Fixing integration hazard test, removing use on data catalog

Fixing integration hazard test, removing use on data catalog

* Remove path to area of interest and define it as geojson in the test files

Remove path to area of interest and define it as geojson in the test files

* fix issue of the vulnerability curves with the same name but different damage types

* small update

* Formatted cell

* Implementation of setup ground elevation function including test

Implementation of setup ground elevation function including test

* Data produced in the runs

Data produced in the runs, this data must be included in the .gitignore file

* Change gitignore to avoid conflicts

Change gitignore to avoid conflicts

* Update gitignore

Update gitignore

* update gitignore

update gitignore

* Update git ignore

Update git ignore

* update test

* Changes in names and more setup functions in the view model

* visual damage curve, distinguish data input/output

* Changed default road types to all road types

* Update index.rst

* Fixes examples for documentation

* added aggregation areas to view model

* fixing documentation errors

* make aggregation areas and roads optional to build the config and run hydromt_fiat

* small bug

* remove roads from yaml standard

* Changed types

* Create hazus_iwr_occupancy_classes.csv

* update datatypes aggregation Union(List, strings)

* assert no polygons overlap

* remove "aggregation Label": Label name...

* update docstring

* correct error message of overlapping polygons

* Update hydromt_fiat_catalog_USA.yml

* Adding changes to update maximum potential damage and ground elevation in new developed areas

Adding changes to update maximum potential damage and ground elevation in new developed areas

* formatting of the data catalog

* Changes to merges branch with #145

Changes to merges branch with #145

* Update gitignore to merge branch with #145

Update gitignore to merge branch with #145

* update default road types

* Added missing class descriptions

* Fixed issues #190 , #165 , #204 & #163

- Finished the correct standardization procedure for the SVI #190
- Created extra column in exposure data with the highest scoring domain #165
- Automated the process of downloading block group shapefiles on the county scale and linking them to the block group data and removing having to download the shapefiles manually #204
- Created the set up to grab equity data, save that data with geometry data + created a test

* Update hydromt_fiat/fiat.py

Co-authored-by: frederique-hub <[email protected]>

* Update hydromt_fiat/workflows/equity_data.py

Co-authored-by: frederique-hub <[email protected]>

* Update hydromt_fiat/workflows/equity_data.py

Co-authored-by: frederique-hub <[email protected]>

* Update hydromt_fiat/workflows/equity_data.py

Co-authored-by: frederique-hub <[email protected]>

* Changed function name to match the equity data

* Added the SVI View Model

* no changes

* no changes

* small changes

* Change ground elevation test data

Change ground elevation test data

* Data required for tests

Data required for tests

* Update test data

Update test data

* Made it possible to just make a FIAT model with roads and some minor bug fixes in vulnerability.py

* Small bug fix

* Integrate ground elevation from a digital elevation model into the writing function

Integrate ground elevation from a digital elevation model into the writing function

* Added list of states and counties in the US

* Update gitignore

* Fix bug with step function threshold = 1

* Update vulnerability curves replacing

* Update xarray-spatial in environment and project .toml

Update xarray-spatial in environment and project .toml

* Improvement of tests for assigning ground elevation and maximun potential damage from users data

Improvement of tests for assigning ground elevation and maximun potential damage from users data

* Updating ground elevation test, and updating maximum potential damage with user data test

Updating ground elevation test, and updating maximum potential damage with user data test

* Update test vulnerability exposure with dem data

Update test vulnerability exposure with dem data

* remove comented lines that were remplazed with ground_elevation_from_dem() function

remove comented lines that were remplazed with ground_elevation_from_dem() function

* take the right function

* Update tests/test_setup_new_composite_areas_ground_elevation.py

* Update hydromt_fiat/workflows/exposure_vector.py

* Adding again the missing options

* fix bugs, all tests running green now

* Added geopy as dependency

* added the us states and counties csv

* Implemented automatic geolocating, for the state and county of the exposure data

* Updating tests, cleaning data, adding different folders in the output

* Changed names of columns

* Delete the index_right column

* updated according to deprication warning

* bugfix

* avoiding double downloading

* Added ground floor height and part of damages connection in API

* Adding separate function for the ground floor height and max potential damage updates

* Bug fix for finding counties and parishes (are there other names for counties?)

* api update, separated the update functions

* create output folder with aggregation areas/additional attributes

* create Building_footprints output folder

* update single BF input to multiple. Add BF folder to exposure folder.

* create add. attributes in root folder

* update path (fix) add. attributes folder in root

* deleted redundant code

* Removed unnecessary lines

* Implemented road damage to be set to 1 for road outages

* Added ground elevation function to fiat.py and updated, still to be added to the api

* small update fixing hazard map input

* Updates according to bugfixing after failing tests

* Added output from testdata to the gitignore

* Updated path to equity data

* Updating examples

* Update hydromt_fiat/fiat.py

Co-authored-by: frederique-hub <[email protected]>

* Update hydromt_fiat/fiat.py

Co-authored-by: frederique-hub <[email protected]>

* Implemented that the SVI and equity shapefiles are saved in the right folder

* Change of folder

* The data was saved in the fiat folder where the model was read and not in the output. I moved the script to the write function to write it to the correct folder

* Added additional assert for correctly saving the files

* aligning with floodadapt needs

* allow to directly provide a DataArray for the hazard raster

* Added census block aggregation area in the exposure data when using setup_equity

* Adding aggregation areas to new composite area

* add self.svi variable

* All good for the aggregation areas and ground elevation in new development areas

* create variable of updated exposure df from run_hydromt()

* reverse svi changes

* update self.buildings with model output

* #235 #237 #239

* update in hazard band naming

* corrected unit conversion

---------

Co-authored-by: Mares2022 <[email protected]>
Co-authored-by: sarah.rautenbach <[email protected]>
Co-authored-by: Santonia27 <[email protected]>
Co-authored-by: LiekeMeijer <[email protected]>
Co-authored-by: Mares2022 <[email protected]>
Co-authored-by: luisrodriguezgalvez <[email protected]>
Co-authored-by: LiekeMeijer <[email protected]>
Co-authored-by: Panos Athanasiou <[email protected]>
Co-authored-by: LuukBlom <[email protected]>

.github/workflows/docs.yml

@@ -1,11 +1,23 @@
-name: Build Docs
+name: Build Documentation
 
-# We don't want pushes (or PRs) to gh-pages to kick anything off
 on:
+  workflow_dispatch:
   push:
     branches: [ main ]
+    paths:
+      - tests/*
+      - hydromt_fiat/*
+      - docs/*
+      - examples/*
+      - pyproject.toml
   pull_request:
     branches: [ main ]
+    paths:
+      - tests/*
+      - hydromt_fiat/*
+      - docs/*
+      - examples/*
+      - pyproject.toml
 
 jobs:
   # Build docs on Linux
@@ -18,45 +30,38 @@ jobs:
       run:
         shell: bash -l {0}
     steps:
-      - name: Checkout source
-        uses: actions/checkout@v2
-
-      # - name: Get tags
-      #   run: git fetch --depth=1 origin +refs/tags/*:refs/tags/*
+      - name: Checkout code
+        uses: actions/checkout@v3
 
-      - name: Setup environment
+      - name: Setup Mambaforge
         uses: conda-incubator/setup-miniconda@v2
         with:
-          python-version: "3.9"
-          mamba-version: "*"
-          channels: conda-forge,defaults
-          channel-priority: true
-          environment-file: envs/hydromt-fiat-dev.yml
-          activate-environment: hydromt-fiat-dev
+          python-version: "3.10"
+          miniforge-variant: Mambaforge
+          miniforge-version: latest
+          channels: conda-forge
+          use-mamba: true
 
-      - name: Install hydromt-fiat
-        run: flit install
+      - name: Install and activate environment
+        run: |
+          mamba env create -f envs/hydromt-fiat-docs.yml
+          conda activate hydromt-fiat-docs
+          pip install .
 
-      # Move examples folder before make html for nbsphinx
-      - name: Build docs
+      - name: Generate docs
+        if: ${{ github.event_name != 'pull_request' && !github.event.act }}
         run: |
-          mv examples docs/_examples
-          pushd docs
-          make html
-          popd
-      # This overrides the version "dev" with the proper version if we're building off a
-      # branch that's not main (which is confined to n.nn.x above) or on a tag.
-      - name: Set doc version
-        if: ${{ github.event_name != 'push' || !contains(github.ref, 'main') }}
-        run: echo "DOC_VERSION=$(python -c 'from hydromt_fiat import __version__ as v; print("dev" if "dev" in v else "v"+v.replace(".dev",""))')" >> $GITHUB_ENV
+          export PATH=/usr/share/miniconda3/bin:$PATH
+          PYTHONPYCACHEPREFIX=~/pycache mamba run -n hydromt-fiat-docs sphinx-build -M html ./docs ./docs/_build
+          echo "DOC_VERSION=$(mamba run -n hydromt-fiat-docs python -c 'from hydromt_fiat import __version__ as v; print("dev" if "dev" in v else "v"+v.replace(".dev",""))')" >> $GITHUB_ENV
 
       - name: Upload to GitHub Pages
-        if: ${{ github.event_name != 'pull_request'}}
-        uses: peaceiris/actions-gh-pages@v3.8.0
+        if: ${{ github.event_name != 'pull_request' && !github.event.act }}
+        uses: peaceiris/actions-gh-pages@v3.9.3
         with:
           github_token: ${{ secrets.GITHUB_TOKEN }}
           publish_dir: ./docs/_build/html
-          exclude_assets: '.buildinfo,_sources/*'
+          exclude_assets: .buildinfo,_sources/*,_examples/*.ipynb
           destination_dir: ./${{ env.DOC_VERSION }}
           keep_files: false
-          full_commit_message: Deploy ${{ env.DOC_VERSION }} to GitHub Pages
+          full_commit_message: Deploy ${{ env.DOC_VERSION }} to GitHub Pages

.gitignore

@@ -119,4 +119,26 @@ venv.bak/
 
 /examples/data/miami_dade_landuse.gpkg
 cache
-/examples/FIAT_database
+/examples/FIAT_database
+hydromt_fiat/data/damage_functions/flooding/AllDDF_HAZUS_fractions.xlsx
+hydromt_fiat/data/damage_functions/flooding/Hazus_IWR_curves.csv
+hydromt_fiat/data/damage_functions/flooding/Hazus_IWR_curves.xlsx
+
+# notebooks
+/examples/aggregation_zones_example_files/
+/examples/data/aggregation_zones/aggregation_zones_test1/
+/examples/data/aggregation_zones/aggregation_zones_test2/
+examples/aggregation_zones_example.html
+examples/aggregation_zones_example_files/
+examples/data/road_network/FIAT model/
+examples/data/global_OSM_JRC/fiat_model/
+examples/data/update_ground_floor_height/test_update_ground_floor_height_points/
+examples/data/update_ground_floor_height/test_update_ground_floor_height_polygons/
+examples/data/building_footprints/fiat_model_bfs/
+examples/data/aggregation_zones/output/
+examples/data/setup_exposure_buildings_with_dem_data/test_vulnerability_and_exposure_NSI_with_dem_data/
+examples/data/setup_new_composite_area/test_setup_new_composite_area_datum/
+examples/data/setup_new_composite_area/test_setup_new_composite_area_elevation/
+examples/data/setup_new_composite_area/test_setup_new_composite_area_geom/
+examples/data/update_max_potential_damage/test_update_max_potential_damage_points/
+examples/data/update_max_potential_damage/test_update_max_potential_damage_polygons/

LICENSE

@@ -671,4 +671,4 @@ into proprietary programs.  If your program is a subroutine library, you
 may consider it more useful to permit linking proprietary applications with
 the library.  If this is what you want to do, use the GNU Lesser General
 Public License instead of this License.  But first, please read
-<https://www.gnu.org/licenses/why-not-lgpl.html>.
+<https://www.gnu.org/licenses/why-not-lgpl.html>.

README.rst

@@ -1,5 +1,5 @@
-HydroMT-FIAT: Delft-FIAT plugin for HydroMT
-#########################################
+HydroMT-FIAT: HydroMT plugin for Delft-FIAT
+############################################
 
 .. image:: https://codecov.io/gh/Deltares/hydromt_fiat/branch/main/graph/badge.svg
     :target: https://codecov.io/gh/Deltares/hydromt_fiat
@@ -20,7 +20,7 @@ HydroMT-FIAT: Delft-FIAT plugin for HydroMT
     :target: https://mybinder.org/v2/gh/Deltares/hydromt_fiat/main?urlpath=lab/tree/examples
 
 
-hydroMT_ is a python package, developed by Deltares, to build and analyze hydro models.
+HydroMT_ is a python package, developed by Deltares, to build and analyze hydro models.
 It provides a generic model api with attributes to access the model schematization,
 (dynamic) forcing data, results and states. This plugin provides an implementation 
 for the Delft-FIAT_ model.
@@ -52,25 +52,9 @@ For developing on HydroMT-FIAT, do:
 Documentation
 -------------
 
-Learn more about the hydroMT_fiat plugin in its `online documentation <https://deltares.github.io/hydromt_fiat/>`_
+Learn more about the HydroMT-FIAT plugin in its `online documentation <https://deltares.github.io/hydromt_fiat/>`_
 
 Contributing
 ------------
 
-You can find information about contributing to hydroMT at our `Contributing page <https://deltares.github.io/hydromt_fiat/latest/contributing.html>`_.
-
-License
--------
-
-Copyright (c) 2021, Deltares
-
-This program is free software: you can redistribute it and/or modify it under the terms of the GNU General 
-Public License as published by the Free Software Foundation, either version 3 of the License, or (at your 
-option) any later version.
-
-This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the 
-implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License 
-for more details.
-
-You should have received a copy of the GNU General Public License along with this program. If not, 
-see <https://www.gnu.org/licenses/>.
+You can find information about contributing to HydroMT at our `Contributing page <https://deltares.github.io/hydromt/latest/dev/contributing>`_.