Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OSM tiling #440

Merged
merged 48 commits into from
Sep 22, 2023
Merged
Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
9eaa699
Working on osm...
dalmijn Feb 15, 2023
5c348d2
Merge branch 'main' into osm
dalmijn Feb 15, 2023
297212d
Merge branch 'main' into osm
dalmijn Feb 17, 2023
f36b3dd
Merge branch 'main' into osm
dalmijn Feb 22, 2023
2278dd3
Merge branch 'main' into osm
dalmijn May 15, 2023
3e860b6
Merge branch 'main' into osm
dalmijn Jun 29, 2023
b377106
Merge branch 'main' into osm
dalmijn Jul 10, 2023
955e859
Added 'to_osm_tiles'
dalmijn Jul 12, 2023
6bea417
Merge branch 'main' into osm
dalmijn Jul 12, 2023
9bc42d0
Cleared output, deleted unused variable
dalmijn Jul 12, 2023
951d3e1
Restored showing of DataCatalog
dalmijn Jul 12, 2023
c1ab0e3
Linting...
dalmijn Jul 12, 2023
ed12d3d
Removed whitespace
dalmijn Jul 12, 2023
149d420
Readded tmpdir
dalmijn Jul 12, 2023
da11a4e
Added OSM tiling test
dalmijn Jul 12, 2023
bb08273
No concats in paths..
dalmijn Jul 12, 2023
f884bbf
Chunking like this blows up memory, removed it
dalmijn Jul 13, 2023
7836a40
Properly induce the reprojected pixel size of the source data
dalmijn Jul 14, 2023
08db166
Merge branch 'main' into osm
dalmijn Jul 26, 2023
1e8fa3f
Updated changelog post v0.8.0 release
dalmijn Jul 26, 2023
a85d458
snake_case to_osm
dalmijn Aug 11, 2023
206b587
Merge branch 'main' into osm
dalmijn Aug 11, 2023
dc01c39
moved 'create_folder'
dalmijn Aug 11, 2023
c15ee9b
small osm fixes
dalmijn Aug 11, 2023
d21e47a
Added small docstring to 'create_folder'
dalmijn Aug 11, 2023
3ed491a
Added a period (apparently...)
dalmijn Aug 11, 2023
6707107
Improved readability; added a tiling function that caters to webviewers
dalmijn Sep 14, 2023
371b71c
Merge branch 'main' into osm
dalmijn Sep 14, 2023
47ff64d
Fixed dependency; linting stuff
dalmijn Sep 14, 2023
cbd1ff0
linting.....
dalmijn Sep 14, 2023
5f9c17b
remove whitespace
dalmijn Sep 14, 2023
45bf02e
remove unnecessary create_folder method
DirkEilander Sep 20, 2023
f948149
simplify slippy tiles
DirkEilander Sep 21, 2023
e865cc0
add optional dependencies
DirkEilander Sep 21, 2023
5579d40
Merge branch 'main' into osm
DirkEilander Sep 22, 2023
f603d8e
test and small bugfixes; update changelog
DirkEilander Sep 22, 2023
b9abc00
move mercantile back to deps for docs; improve test coverage
DirkEilander Sep 22, 2023
eef06e5
fix test after including extra zoom lvl
DirkEilander Sep 22, 2023
6d353fd
Merge branch 'main' into osm
DirkEilander Sep 22, 2023
0c65f5d
remove unecessary flipud
DirkEilander Sep 22, 2023
1f1a0ed
fix stack level
savente93 Sep 22, 2023
d1f432e
Updated version
dalmijn Sep 22, 2023
56c4b4e
Revamped zoom level determination
dalmijn Sep 22, 2023
e6e1162
Added extra example for the oil painting
dalmijn Sep 22, 2023
a0ba124
some docs for the colormap
dalmijn Sep 22, 2023
180f24f
Merge branch 'osm' of https://github.com/Deltares/hydromt into osm
dalmijn Sep 22, 2023
8135548
Restored yaml filename to folder; pinned zl in example
dalmijn Sep 22, 2023
2155a2f
Some last simplifications
dalmijn Sep 22, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -459,6 +459,7 @@ Raster writing methods
:toctree: _generated

DataArray.raster.to_xyz_tiles
DataArray.raster.to_osm_tiles
DataArray.raster.to_raster
Dataset.raster.to_mapstack

Expand Down
2 changes: 1 addition & 1 deletion docs/changelog.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Unreleased

Added
-----
-
- New raster method ``to_osm_tiles``: tiling of a raster dataset according to the osm structure.

Changed
-------
Expand Down
2 changes: 1 addition & 1 deletion examples/prep_data_catalog.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -773,7 +773,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.6"
"version": "3.11.4"
},
"vscode": {
"interpreter": {
Expand Down
76 changes: 71 additions & 5 deletions examples/working_with_tiled_raster_data.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### tiling raster with XYZ stucture\n",
"### Tiling raster with XYZ stucture\n",
"\n",
"First let's have a look at the XYZ structure.\n",
"an xarray.DataArray is simple written to a tile database in XYZ structure via .raster.to_xyz_tiles\n",
Expand Down Expand Up @@ -92,12 +92,55 @@
"At last, a .yml file is produced which can be read by the [DataCatalog](https://deltares.github.io/hydromt/latest/_generated/hydromt.data_catalog.DataCatalog.html) of HydroMT."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Tiling raster with OSM stucture\n",
"\n",
"Now let's have a look at tiling according to the OSM structure\n",
"an xarray.DataArray is simple written to a tile database in OSM structure via .raster.to_osm_tiles"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Write the database in XYZ stucture\n",
"name_osm = f\"{source}_osm\"\n",
"root_osm = join(\"tmpdir\", name_osm)\n",
"zoom_levels = [0, 1, 2, 5] # note that you have to present all levels\n",
"da0.raster.to_osm_tiles(\n",
" root=root_osm,\n",
" zoom_levels=zoom_levels,\n",
" min_lvl=9, # minimal level from where to upscale from (prevent memory issues)\n",
" driver=\"GTiff\", # try also 'netcdf4'\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The tiles in the 'merit_hydro_OSM' folder now contains all the zoom_levels from the minimal starting level (9) until the highest level (0).\n",
"\n",
"Every tile, regardless of the zoomlevel, has a resolution of 256 by 256 pixels.\n",
"\n",
"Zoomlevel 0 is at the scale of the entire world (one tile) and is the most downscaled. Zoomlevel 9 contains the highest resolution (most tiles) in regards to this tile database.\n",
"\n",
"A mosaic is created per zoomlevel of these tiles in a .vrt file.\n",
"\n",
"At last, a .yml file is produced which can be read by the [DataCatalog](https://deltares.github.io/hydromt/latest/_generated/hydromt.data_catalog.DataCatalog.html) of HydroMT."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### reading tiled raster data with zoom levels\n",
"### Reading tiled raster data with zoom levels\n",
"\n",
"With DataCatalog.[get_rasterdataset](https://deltares.github.io/hydromt/latest/_generated/hydromt.data_catalog.DataCatalog.get_rasterdataset.html) a raster (.vrt) can be retrieved. In case of a tile database it can be done for a certain zoomlevel. E.g."
]
Expand All @@ -112,12 +155,22 @@
"from hydromt import DataCatalog\n",
"\n",
"# Load the yml into a DataCatalog\n",
"data_catalog = DataCatalog(join(root, f\"{name}.yml\"), logger=logger)\n",
"data_catalog = DataCatalog([join(root, f\"{name}.yml\"), join(root_osm, f\"{name_osm}.yml\")], logger=logger)\n",
"\n",
"# View the structure of the DataCatalog\n",
"data_catalog[name]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# For the osm dataset\n",
"data_catalog[name_osm]"
]
},
{
"cell_type": "code",
"execution_count": null,
Expand All @@ -129,6 +182,17 @@
"da0.raster.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# And for OSM\n",
"da1 = data_catalog.get_rasterdataset(name_osm)\n",
"da1.raster.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
Expand All @@ -137,8 +201,10 @@
"source": [
"# Request a raster from the Datacatolog based on zoom resolution & unit\n",
"da = data_catalog.get_rasterdataset(name, zoom_level=(1/600, 'degree'))\n",
"da = data_catalog.get_rasterdataset(name_osm, zoom_level=(1/600, 'degree'))\n",
"\n",
"da = data_catalog.get_rasterdataset(name, zoom_level=(1e3, 'meter'))\n"
"da = data_catalog.get_rasterdataset(name, zoom_level=(1e3, 'degree'))\n",
"da = data_catalog.get_rasterdataset(name_osm, zoom_level=(1e4, 'meter'))\n"
]
},
{
Expand Down Expand Up @@ -220,7 +286,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.9"
"version": "3.11.4"
},
"orig_nbformat": 4,
"vscode": {
Expand Down
Loading
Loading