Process inlets outlets (#24)

Contains two notebooks with structures at RWS boundaries:
- `nl-kunstwerken.ipynb` to create a GeoPackage from the data in
https://samenwerkenaankunstwerken.nl/#. Writes to:
https://deltares.thegood.cloud/f/54241
- `uitlaten_inlaten.ipynb` to create a HyDAMO GeoPackage from our own
collection of inlets_outlets. Writes to:
https://deltares.thegood.cloud/f/52625

`environment.yml` is moved and considered to be a project-yml. For now
we only need openpyxl as extra so we can read Excel-files in Pandas.
Notebooks can run with this environment, after including hydamo from the
src/hydamo dir added by:

```
pip install -e .
`` 

In the activated enviroment

.github/workflows/python_tests.yml

@@ -18,8 +18,6 @@ jobs:
       matrix:
         os:
           - ubuntu-latest
-          - macOS-latest
-          - windows-latest
         arch:
           - x86
     steps:

.typos.toml

@@ -1,2 +1,2 @@
 [files]
-extend-exclude = ["*.csv", "*.json"]
+extend-exclude = ["*.csv", "*.json", "*.ipynb"]

environment.yml

@@ -0,0 +1,36 @@
+# ribasim environment with some extra goodies to work in NL-context
+name: ribasim
+
+channels:
+  - conda-forge
+
+dependencies:
+  - black
+  - black-jupyter
+  - build
+  - datamodel-code-generator
+  - geopandas
+  - jupyterlab
+  - matplotlib
+  - mypy
+  - openpyxl
+  - pandas!=2.1.0
+  - pandas-stubs
+  - pandera
+  - pip
+  - pip:
+    - quartodoc
+  - pre-commit
+  - pyarrow
+  - pydantic=1
+  - pyogrio
+  - pytest
+  - pytest-cov
+  - python>=3.9
+  - ruff
+  - shapely>=2.0
+  - tomli
+  - tomli-w
+  - twine
+  - xarray
+  - xmipy

notebooks/nl-kunstwerken.ipynb

@@ -0,0 +1,199 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import geopandas as gpd\n",
+    "from shapely.geometry import Point\n",
+    "from pathlib import Path\n",
+    "import os\n",
+    "import requests"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Voorbereiding\n",
+    "\n",
+    "Globale variabelen\n",
+    "`DATA_DIR`: De locale directory waar project-data staat opgeslagen\n",
+    "`EXCEL_FILE`: Het Excel-bestand dat moet worden ingelezen\n",
+    "`CRS`: De projectile waarin de ruimtelijke data moet worden opgeslagen (28992 = Rijksdriehoekstelsel) "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# environmnt variables\n",
+    "DATA_DIR = os.getenv(\"RIBASIM_NL_DATA_DIR\")\n",
+    "RIBASIM_NL_CLOUD_PASS = os.getenv(\"RIBASIM_NL_CLOUD_PASS\")\n",
+    "assert DATA_DIR is not None\n",
+    "assert RIBASIM_NL_CLOUD_PASS is not None\n",
+    "\n",
+    "EXCEL_FILE = r\"# Overzicht kunstwerken primaire keringen waterschappen_ET.xlsx\"\n",
+    "CRS = 28992\n",
+    "RIBASIM_NL_CLOUD_USER = \"nhi_api\"\n",
+    "WEBDAV_URL = \"https://deltares.thegood.cloud/remote.php/dav\"\n",
+    "BASE_URL = f\"{WEBDAV_URL}/files/{RIBASIM_NL_CLOUD_USER}/D-HYDRO modeldata\"\n",
+    "\n",
+    "# file-paths\n",
+    "kunstwerken_xlsx = Path(DATA_DIR) / EXCEL_FILE\n",
+    "kunstwerken_gpkg = kunstwerken_xlsx.parent / \"nl_kunstwerken.gpkg\"\n",
+    "\n",
+    "\n",
+    "def upload_file(url, path):\n",
+    "    with open(path, \"rb\") as f:\n",
+    "        r = requests.put(\n",
+    "            url, data=f, auth=(RIBASIM_NL_CLOUD_USER, RIBASIM_NL_CLOUD_PASS)\n",
+    "        )\n",
+    "    r.raise_for_status()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Inlezen NL kunstwerken vanuit data dir\n",
+    "We lezen de geleverde excel in en:\n",
+    "- skippen de eerste 6 regels boven de header\n",
+    "- gooien, voor dit project, irrelevante kolommen weg\n",
+    "- hernoemen de kolom met organisatie-naam, x en y coordinaat\n",
+    "- transformeren de x en y coordinaat; wordt NaN wanneer data dat niet toelaat (text of missend)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# skip first rows\n",
+    "kunstwerken_df = pd.read_excel(kunstwerken_xlsx, skiprows=6)\n",
+    "\n",
+    "# drop irrelevant columns\n",
+    "columns = kunstwerken_df.columns[1:13]\n",
+    "kunstwerken_df = kunstwerken_df.loc[:, columns]\n",
+    "\n",
+    "# rename columns into our liking\n",
+    "kunstwerken_df.rename(\n",
+    "    columns={\n",
+    "        \"Unnamed: 1\": \"organisatie\",\n",
+    "        \"Y coördinaat RD\": \"y\",\n",
+    "        \"X coördinaat RD\": \"x\",\n",
+    "    },\n",
+    "    inplace=True,\n",
+    ")\n",
+    "\n",
+    "# drop no-data rows\n",
+    "kunstwerken_df = kunstwerken_df[~kunstwerken_df[\"organisatie\"].isna()]\n",
+    "\n",
+    "# convert x/y to numeric\n",
+    "kunstwerken_df[\"x\"] = pd.to_numeric(kunstwerken_df[\"x\"], errors=\"coerce\")\n",
+    "kunstwerken_df[\"y\"] = pd.to_numeric(kunstwerken_df[\"y\"], errors=\"coerce\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Aanmaken niet-ruimtelijke table\n",
+    "Waar x/y coordinaten mizzen maken we een niet-ruimtelijke table die we wegschrijven in geen GeoPackage"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# make a non-spatial GeoDataFrame where x/y are missing\n",
+    "kunstwerken_non_spatial_df = kunstwerken_df[\n",
+    "    kunstwerken_df[\"x\"].isna() | kunstwerken_df[\"y\"].isna()\n",
+    "]\n",
+    "kunstwerken_non_spatial_gdf = gpd.GeoDataFrame(\n",
+    "    kunstwerken_non_spatial_df, geometry=gpd.GeoSeries(), crs=28992\n",
+    ")\n",
+    "\n",
+    "# writ to GeoPackage\n",
+    "kunstwerken_non_spatial_gdf.to_file(\n",
+    "    kunstwerken_gpkg, layer=\"kunstwerken (geen coordinaten)\"\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Aanmaken ruimtelijke table\n",
+    "Waar x/y coordinaten beschikbaar zijn maken we een ruimtelijke table die we wegschrijven in geen GeoPackage"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# make a spatial GeoDataFrame where x/y exist\n",
+    "kunstwerken_spatial_df = kunstwerken_df[\n",
+    "    ~kunstwerken_df[\"x\"].isna() & ~kunstwerken_df[\"y\"].isna()\n",
+    "]\n",
+    "geometry_series = gpd.GeoSeries(\n",
+    "    kunstwerken_spatial_df.apply((lambda x: Point(x.x, x.y)), axis=1)\n",
+    ")\n",
+    "kunstwerken_spatial_gdf = gpd.GeoDataFrame(\n",
+    "    kunstwerken_spatial_df, geometry=geometry_series, crs=CRS\n",
+    ")\n",
+    "\n",
+    "# write to GeoPackage\n",
+    "kunstwerken_spatial_gdf.to_file(kunstwerken_gpkg, layer=\"kunstwerken\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Upload geopackage"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "to_url = f\"{BASE_URL}/Rijkswaterstaat/{kunstwerken_gpkg.name}\"\n",
+    "upload_file(to_url, kunstwerken_gpkg)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}