notebooks/ICESat-2_Cloud_Access/ICESat2-CryoSat2-Coincident.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "de81ea4e-1dad-4f1d-af4b-416dd256af6f",
   "metadata": {},
   "source": [
    "# **Plotting ICESat-2 and CryoSat-2 Freeboards**\n",
    "\n",
    "\n",
    "<div align=\"center\"><img width=\"40%\"  src='https://raw.githubusercontent.com/nsidc/NSIDC-Data-Tutorials/is2-cryo2/notebooks/ICESat-2_Cloud_Access/img/icesat2-cryosat2.png'/></div>\n",
    "\n",
    "### **Credits**\n",
    "This notebook was created by Mikala Beig and Andy Barrett, NSIDC\n",
    "\n",
    "### **Learning Objectives** \n",
    "\n",
    "1. use `earthaccess` to search for ICESat-2 ATL10 data using a spatial filter\n",
    "2. open cloud-hosted files using direct access to the ICESat-2 S3 bucket; \n",
    "3. use cs2eo script to download files into your hub instance\n",
    "3. load an HDF5 group into an `xarray.Dataset`;  \n",
    "4. visualize freeboards using `hvplot`.\n",
    "5. map the locations of ICESat-2 and CryoSat-2 freeboards using `cartopy`\n",
    "\n",
    "### **Prerequisites**\n",
    "\n",
    "1. An EC2 instance in the us-west-2 region. **NASA cloud-hosted data are in Amazon Region us-west2. So you also need an EC2 instance in the us-west-2 region.** .\n",
    "2. An Earthdata Login is required for data access. If you don't have one, you can register for one [here](https://urs.earthdata.nasa.gov/).\n",
    "3. Experience using cs2eo to query for coincident data.\n",
    "4. A cs2eo download script for CryoSat-2 data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2db2b3c-97bf-42aa-8fd1-0eb588afa80e",
   "metadata": {},
   "source": [
    "### **Tutorial Steps**\n",
    "\n",
    "#### Query for coincident ICESat-2 and CryoSat-2 data\n",
    "\n",
    "Using the cs2eo coincident data explorer, query for ATL10 and CryoSat-2, L2, SAR, POCA, Baseline E data products using a spatial and temporal filter.\n",
    "\n",
    "**Download the basic result metadata and the raw access scripts.**  Upload the ESA download script (SIR_SAR_L2_E_download_script.py) into the folder from which you are running this notebook.\n",
    "\n",
    "![cs2eo query](./img/ATL10_CS2_L2_SAR_query_med.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94cbed67-e58a-4579-b153-c9fcbe1e2ab4",
   "metadata": {},
   "source": [
    "#### Import Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e911afb1-b247-4412-9342-1e8865b3084e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import platform\n",
    "from ftplib import FTP\n",
    "import sys\n",
    "\n",
    "\n",
    "# For searching and accessing NASA data\n",
    "import earthaccess\n",
    "\n",
    "# For reading data, analysis and plotting\n",
    "import xarray as xr\n",
    "import hvplot.xarray\n",
    "\n",
    "# For nice printing of python objects\n",
    "import pprint  \n",
    "\n",
    "# For plotting\n",
    "import matplotlib.pyplot as plt\n",
    "import cartopy.crs as ccrs\n",
    "import cartopy.feature as cfeature\n",
    "\n",
    "#downloading files using cs2eo script\n",
    "from SIR_SAR_L2_E_download_script import download_files\n",
    "\n",
    "## use your own email here\n",
    "user_email = 'your email here'\n",
    "path = './data/'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58c3ef53-7223-42da-afd4-e14e87caa85d",
   "metadata": {},
   "source": [
    "#### Download CryoSat-2 data to your hub instance\n",
    "\n",
    "Copy the list of ESA files from within SIR_SAR_L2_E_download_script.py "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac8e451e-b687-4c13-a5b8-9b700a519a46",
   "metadata": {},
   "outputs": [],
   "source": [
    "esa_files = ['SIR_SAR_L2/2019/12/CS_LTA__SIR_SAR_2__20191227T110305_20191227T111751_E001.nc', \n",
    "             'SIR_SAR_L2/2020/03/CS_LTA__SIR_SAR_2__20200329T163208_20200329T164044_E001.nc', \n",
    "             'SIR_SAR_L2/2020/01/CS_LTA__SIR_SAR_2__20200114T203033_20200114T204440_E001.nc', \n",
    "             'SIR_SAR_L2/2019/11/CS_LTA__SIR_SAR_2__20191103T134759_20191103T135125_E001.nc', \n",
    "             'SIR_SAR_L2/2020/02/CS_LTA__SIR_SAR_2__20200204T191657_20200204T192558_E001.nc', \n",
    "             'SIR_SAR_L2/2019/12/CS_LTA__SIR_SAR_2__20191216T215645_20191216T220909_E001.nc', \n",
    "             'SIR_SAR_L2/2020/03/CS_LTA__SIR_SAR_2__20200315T065755_20200315T071241_E001.nc', \n",
    "             'SIR_SAR_L2/2019/10/CS_LTA__SIR_SAR_2__20191030T135252_20191030T135600_E001.nc', \n",
    "             'SIR_SAR_L2/2020/02/CS_LTA__SIR_SAR_2__20200219T081800_20200219T083303_E001.nc', \n",
    "             'SIR_SAR_L2/2020/01/CS_LTA__SIR_SAR_2__20200110T203717_20200110T204612_E001.nc', \n",
    "             'SIR_SAR_L2/2020/04/CS_LTA__SIR_SAR_2__20200409T053748_20200409T054151_E001.nc', \n",
    "             'SIR_SAR_L2/2020/04/CS_LTA__SIR_SAR_2__20200413T053254_20200413T053659_E001.nc', \n",
    "             'SIR_SAR_L2/2020/02/CS_LTA__SIR_SAR_2__20200208T191154_20200208T192117_E001.nc', \n",
    "             'SIR_SAR_L2/2020/03/CS_LTA__SIR_SAR_2__20200319T065300_20200319T070802_E001.nc', \n",
    "             'SIR_SAR_L2/2020/03/CS_LTA__SIR_SAR_2__20200304T175209_20200304T180102_E001.nc', \n",
    "             'SIR_SAR_L2/2019/11/CS_LTA__SIR_SAR_2__20191128T122800_20191128T123212_E001.nc', \n",
    "             'SIR_SAR_L2/2019/10/CS_LTA__SIR_SAR_2__20191009T150801_20191009T151142_E001.nc', \n",
    "             'SIR_SAR_L2/2019/11/CS_LTA__SIR_SAR_2__20191121T231659_20191121T232817_E001.nc', \n",
    "             'SIR_SAR_L2/2020/02/CS_LTA__SIR_SAR_2__20200215T082253_20200215T083741_E001.nc', \n",
    "             'SIR_SAR_L2/2020/01/CS_LTA__SIR_SAR_2__20200121T094259_20200121T095800_E001.nc', \n",
    "             'SIR_SAR_L2/2019/10/CS_LTA__SIR_SAR_2__20191005T151255_20191005T151621_E001.nc', \n",
    "             'SIR_SAR_L2/2020/04/CS_LTA__SIR_SAR_2__20200427T150701_20200427T151544_E001.nc', \n",
    "             'SIR_SAR_L2/2019/10/CS_LTA__SIR_SAR_2__20191024T004201_20191024T005059_E001.nc', \n",
    "             'SIR_SAR_L2/2020/03/CS_LTA__SIR_SAR_2__20200308T174708_20200308T175621_E001.nc', \n",
    "             'SIR_SAR_L2/2020/04/CS_LTA__SIR_SAR_2__20200402T162707_20200402T163602_E001.nc']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eaa73f90-84f0-4429-854a-784c95afde66",
   "metadata": {},
   "source": [
    "Download the CryoSat-2 files into your hub instance by calling the download_files function you imported from the script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ecef7122-b3d5-4ca0-b207-7ad3c35ca34d",
   "metadata": {},
   "outputs": [],
   "source": [
    "download_files(user_email, esa_files)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e9d27c7-d9d9-40ae-aab6-34888aedfe8f",
   "metadata": {},
   "source": [
    "Stashing the files in a data folder to keep our notebook directory less cluttered."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f6c0779-77cc-4d60-a7a6-cb4a0ce1ea25",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mv CS_LTA__SIR*.nc data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aeb468e5-7a5a-456c-96cb-6808af3f5eaa",
   "metadata": {},
   "source": [
    "#### Use `earthaccess` for direct S3 access of ATL10\n",
    "\n",
    "First we authenticate using `earthaccess`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2165876a-c5fa-4df9-b7f5-c82f01e7bdba",
   "metadata": {},
   "outputs": [],
   "source": [
    "auth = earthaccess.login()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb7adbf2-1148-4075-8578-91e01fbeacfc",
   "metadata": {},
   "source": [
    "Then we use a spatial filter to search for ATL10 granules that intersect our area of interest.  This is the same area we used in our cs2eo query above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "faa2297b-2699-466d-a872-1527114dfd46",
   "metadata": {},
   "outputs": [],
   "source": [
    "# url_list from ATL10 download script provided by cs2eo\n",
    "on_prem_urls = ['https://n5eil01u.ecs.nsidc.org/DP5/ATLAS/ATL10.006/2020.03.19/ATL10-01_20200319030948_12740601_006_01.h5', \n",
    "                'https://n5eil01u.ecs.nsidc.org/DP5/ATLAS/ATL10.006/2019.11.21/ATL10-01_20191121192822_08540501_006_01.h5', \n",
    "                'https://n5eil01u.ecs.nsidc.org/DP5/ATLAS/ATL10.006/2019.10.09/ATL10-01_20191009110835_01920501_006_02.h5', \n",
    "                'https://n5eil01u.ecs.nsidc.org/DP5/ATLAS/ATL10.006/2019.12.27/ATL10-01_20191227071321_00090601_006_01.h5', \n",
    "                'https://n5eil01u.ecs.nsidc.org/DP5/ATLAS/ATL10.006/2020.03.04/ATL10-01_20200304141724_10520601_006_01.h5', \n",
    "                'https://n5eil01u.ecs.nsidc.org/DP5/ATLAS/ATL10.006/2020.03.29/ATL10-01_20200329130147_00460701_006_01.h5', \n",
    "                'https://n5eil01u.ecs.nsidc.org/DP5/ATLAS/ATL10.006/2019.11.28/ATL10-01_20191128083720_09540501_006_01.h5', \n",
    "                'https://n5eil01u.ecs.nsidc.org/DP5/ATLAS/ATL10.006/2020.04.02/ATL10-01_20200402125328_01070701_006_02.h5', \n",
    "                'https://n5eil01u.ecs.nsidc.org/DP5/ATLAS/ATL10.006/2019.10.30/ATL10-01_20191030100118_05120501_006_01.h5', \n",
    "                'https://n5eil01u.ecs.nsidc.org/DP5/ATLAS/ATL10.006/2020.01.14/ATL10-01_20200114164843_02900601_006_01.h5', \n",
    "                'https://n5eil01u.ecs.nsidc.org/DP5/ATLAS/ATL10.006/2020.04.27/ATL10-01_20200427113753_04880701_006_02.h5', \n",
    "                'https://n5eil01u.ecs.nsidc.org/DP5/ATLAS/ATL10.006/2019.10.23/ATL10-01_20191023205221_04120501_006_02.h5', \n",
    "                'https://n5eil01u.ecs.nsidc.org/DP5/ATLAS/ATL10.006/2019.10.05/ATL10-01_20191005111655_01310501_006_02.h5', \n",
    "                'https://n5eil01u.ecs.nsidc.org/DP5/ATLAS/ATL10.006/2020.04.09/ATL10-01_20200409020227_02070701_006_02.h5', \n",
    "                'https://n5eil01u.ecs.nsidc.org/DP5/ATLAS/ATL10.006/2020.02.19/ATL10-01_20200219043343_08320601_006_01.h5', \n",
    "                'https://n5eil01u.ecs.nsidc.org/DP5/ATLAS/ATL10.006/2020.04.13/ATL10-01_20200413015408_02680701_006_02.h5', \n",
    "                'https://n5eil01u.ecs.nsidc.org/DP5/ATLAS/ATL10.006/2020.03.08/ATL10-01_20200308140907_11130601_006_01.h5', \n",
    "                'https://n5eil01u.ecs.nsidc.org/DP5/ATLAS/ATL10.006/2020.03.15/ATL10-01_20200315031809_12130601_006_01.h5', \n",
    "                'https://n5eil01u.ecs.nsidc.org/DP5/ATLAS/ATL10.006/2019.12.16/ATL10-01_20191216181242_12350501_006_01.h5', \n",
    "                'https://n5eil01u.ecs.nsidc.org/DP5/ATLAS/ATL10.006/2020.01.21/ATL10-01_20200121055741_03900601_006_01.h5', \n",
    "                'https://n5eil01u.ecs.nsidc.org/DP5/ATLAS/ATL10.006/2020.02.15/ATL10-01_20200215044202_07710601_006_01.h5', \n",
    "                'https://n5eil01u.ecs.nsidc.org/DP5/ATLAS/ATL10.006/2020.02.04/ATL10-01_20200204154125_06100601_006_01.h5', \n",
    "                'https://n5eil01u.ecs.nsidc.org/DP5/ATLAS/ATL10.006/2020.02.08/ATL10-01_20200208153304_06710601_006_01.h5', \n",
    "                'https://n5eil01u.ecs.nsidc.org/DP5/ATLAS/ATL10.006/2020.01.10/ATL10-01_20200110165702_02290601_006_01.h5', \n",
    "                'https://n5eil01u.ecs.nsidc.org/DP5/ATLAS/ATL10.006/2019.11.03/ATL10-01_20191103095257_05730501_006_01.h5']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24a41b73-43e4-4fad-9d71-bde840b0da8a",
   "metadata": {},
   "source": [
    "The following function replaces the on prem urls in the ATL10 download script with the equivalent cloud urls for earthaccess to utilize.  This won't be necessary once cs2eo is accessing the cloud copy of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "902b81e5-6719-48bb-a66d-03daf8fd7374",
   "metadata": {},
   "outputs": [],
   "source": [
    "def cloud_atl10_urls(on_prem_urls):\n",
    "    import re\n",
    "    urls = []\n",
    "    for url in on_prem_urls:\n",
    "        match = re.search(r'/ATL10\\.006/(\\d{4})\\.(\\d{2})\\.(\\d{2})/(ATL10-.*\\.h5)', url)\n",
    "        if match:\n",
    "            year, month, day, filename = match.groups()\n",
    "            cloud_url = f'https://data.nsidc.earthdatacloud.nasa.gov/nsidc-cumulus-prod-protected/ATLAS/ATL10/006/{year}/{month}/{day}/{filename}'\n",
    "            urls.append(cloud_url)\n",
    "    return urls\n",
    "\n",
    "new_urls = cloud_atl10_urls(on_prem_urls)\n",
    "\n",
    "# This is due a bug in the most recent version.\n",
    "earthaccess.__store__.in_region = False\n",
    "icesat2_files = earthaccess.open(new_urls)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e58e1111-1b07-47ec-952b-036290346f4c",
   "metadata": {},
   "source": [
    "We use earthaccess.open() to directly access the ATL10 files within their S3 bucket.  earthaccess.open() creates a file-like object, which is required because AWS S3 uses object storage, and we need to create a virtual file system to work with the HDF5 library."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d24f9d50-3f95-4cdf-b877-1ac06be4471b",
   "metadata": {},
   "source": [
    "We can use xarray to examine the contents of our files (one group at a time)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d1102e5-561e-41e8-81df-e4ff543d19c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_is2 = xr.open_dataset(icesat2_files[1], group='gt2r/freeboard_segment')\n",
    "ds_is2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c603f6a6-7dcb-4a6f-9130-0bab933ff3c6",
   "metadata": {},
   "source": [
    "And we can use hvplot to plot one of the variables within that group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0503dfee-7d66-49fa-8482-ad2dc330ff52",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_is2['beam_fb_height'].hvplot(kind='scatter', s=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a89c207-028f-47df-b7d9-5723bd5e836f",
   "metadata": {},
   "source": [
    "#### Open and plot downloaded CryoSat-2 data \n",
    "\n",
    "We need a list of the downloaded CryoSat-2 files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60c1704a-0b17-4d4f-a7a8-dac8fe5b9701",
   "metadata": {},
   "outputs": [],
   "source": [
    "downloaded_files = os.listdir(path)\n",
    "downloaded_files"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d280de0-5de7-4a0c-9986-c94ab6b6d05a",
   "metadata": {},
   "source": [
    "We use xarray to access the contents of our netcdf file.  In this case, we are not \"streaming\" data from an S3 bucket, but are accessing the data locally."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8657ed36-d583-46b9-8ed2-9b1b57cb5786",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_cs2 = xr.open_dataset(path + downloaded_files[0])\n",
    "ds_cs2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "795b0bdd-0895-4de4-8008-b6f15fcf2b3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_cs2['radar_freeboard_20_ku'].hvplot(kind='scatter', s=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40a8164d-f5c1-4049-aefe-aeb08295092b",
   "metadata": {},
   "source": [
    "#### Plot ICESat-2 and CryoSat-2 Freeboards on same map"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad5f7f2c-2ff8-47cc-8ea8-337f9183c20e",
   "metadata": {},
   "source": [
    "Here we're plotting one file from each data set to save time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11476d13-0f1f-44f7-a364-4a949554d8c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "projection = ccrs.Stereographic(central_latitude=90.,\n",
    "                                central_longitude=-45.,\n",
    "                                true_scale_latitude=70.)\n",
    "extent = [-2500000.000, 2500000., -2500000., 2500000.000]\n",
    "\n",
    "\n",
    "fig = plt.figure(figsize=(10,10))\n",
    "ax = fig.add_subplot(projection=projection)\n",
    "ax.set_extent(extent, projection)\n",
    "ax.add_feature(cfeature.OCEAN)\n",
    "ax.add_feature(cfeature.LAND)\n",
    "ax.gridlines(draw_labels=True)\n",
    "\n",
    "vmin = 0.\n",
    "vmax = 1.\n",
    "\n",
    "# Plot Cryosat freeboard\n",
    "cs2_img = ax.scatter(ds_cs2.lon_poca_20_ku, ds_cs2.lat_poca_20_ku, 5,\n",
    "                        c=ds_cs2.radar_freeboard_20_ku, \n",
    "                        vmin=vmin, vmax=vmax,  # Set max and min values for plotting\n",
    "                        cmap='Reds',   # shading='auto' to avoid warning\n",
    "                        transform=ccrs.PlateCarree())  # coords are lat,lon but map if NPS \n",
    "\n",
    "# Plot IS2 freeboard \n",
    "is2_img = ax.scatter(ds_is2.longitude, ds_is2.latitude, 5,\n",
    "                     c=ds_is2.beam_fb_height, \n",
    "                     vmin=vmin, vmax=vmax,  \n",
    "                     cmap='Purples', \n",
    "                     transform=ccrs.PlateCarree())\n",
    "\n",
    "# Add colorbars\n",
    "fig.colorbar(cs2_img, label='Cryosat-2 Radar Freeboard (m)')\n",
    "fig.colorbar(is2_img, label='ICESat-2 Lidar Freeboard (m)')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}