L1_L_RSLC-Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"1\">Copyright 2021, by the California Institute of Technology. ALL RIGHTS RESERVED. United States Government sponsorship acknowledged. Any commercial use must be negotiated with the Office of Technology Transfer at the California Institute of Technology.</font>\n",
    "    \n",
    "<font size=\"1\">This software may be subject to U.S. export control laws and regulations. By accepting this document, the user agrees to comply with all applicable U.S. export laws and regulations. User has the responsibility to obtain export licenses, or other export authority as may be required, before exporting such information to foreign countries or providing access to foreign persons.<font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Datasets\n",
    "This notebook will use the Pele API to query available datasets and download one from S3 based on the metadata returned by the query.\n",
    "\n",
    "## Setup\n",
    "\n",
    "This notebook assumes you've already gone through the first notebook and registered a user and password, as well sa populated your .netrc file. Let's go ahead and set things up so that we can utilize the Pele client library to query our datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import requests, json, getpass\n",
    "from requests.auth import HTTPBasicAuth\n",
    "import urllib3\n",
    "from urllib.parse import urlparse\n",
    "\n",
    "urllib3.disable_warnings()\n",
    "\n",
    "# this block makes sure the directory set-up/change is only done once and relative to the notebook's directory\n",
    "try:\n",
    "    start_dir\n",
    "except NameError:\n",
    "    start_dir = os.getcwd()\n",
    "    !mkdir -p ./notebook_output/L1_L_RSLC-Analysis\n",
    "    os.chdir('notebook_output/L1_L_RSLC-Analysis')\n",
    "    \n",
    "# set the base url to interact with the goddess, Pele\n",
    "#base_url = input(\"Enter Pele REST API base url (e.g. https://<mozart_ip>/pele/api/v0.1) then press <Enter>: \")\n",
    "base_url = \"https://172.31.29.154/pele/api/v0.1\"\n",
    "print(\"Using base url {}.\".format(base_url))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's validate that we can interact with Pele:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pele_client.client import PeleRequests\n",
    "\n",
    "# instantiate PeleRequests object\n",
    "pr = PeleRequests(base_url, verify=False, auth=False)\n",
    "\n",
    "# now use like requests module (`request()`, `get()`, `head()`, `post()`, `put()`, `delete()`, `patch()`)\n",
    "r = pr.get(base_url + '/test/echo', params={'echo_str': 'hello world'})\n",
    "\n",
    "# expect 200\n",
    "print(\"status code: {}\".format(r.status_code))\n",
    "print(json.dumps(r.json(), indent=2))\n",
    "assert r.status_code == 200"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Querying\n",
    "\n",
    "Let's see what datasets we have:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get datasets\n",
    "r = pr.get(base_url + '/pele/datasets')\n",
    "\n",
    "# expect 200\n",
    "print(\"status code: {}\".format(r.status_code))\n",
    "res = r.json()\n",
    "print(json.dumps(res, indent=2))\n",
    "assert r.status_code == 200\n",
    "assert \"L1_L_RSLC\" in res['datasets']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the `L1_L_RSLC` dataset type and query for dataset ids:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# query for all dataset IDs of the `L1_L_RSLC` dataset\n",
    "r = pr.get(base_url + '/pele/dataset/L1_L_RSLC/dataset_ids')\n",
    "\n",
    "# expect 200\n",
    "print(\"status code: {}\".format(r.status_code))\n",
    "res = r.json()\n",
    "print(json.dumps(res, indent=2))\n",
    "assert r.status_code == 200"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have a number of `L1_L_RSLC` granules but the API returns only 10. Let's collect all of them by iterating over the paged results 10 at a time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rslc_ids = res['dataset_ids']\n",
    "while len(rslc_ids) != res['total']:\n",
    "    r = pr.get(base_url + '/pele/dataset/L1_L_RSLC/dataset_ids', params={'offset': res['offset']+res['page_size']})\n",
    "    #print(f\"Response:\\n{r}\")\n",
    "    res = r.json()\n",
    "    #print(f\"res: {res}\")\n",
    "    if 'dataset_ids' in res:\n",
    "        rslc_ids.extend(res['dataset_ids'])\n",
    "    else:\n",
    "        break\n",
    "    \n",
    "# expect 200\n",
    "print(\"All L1_L_RSLC ids: {}\".format(rslc_ids))\n",
    "print(len(rslc_ids))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the metadata for one of those granules:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# query for metadata of a specific `L1_L_RSLC` dataset\n",
    "r = pr.get(base_url + '/pele/dataset/NISAR_L1_PR_RSLC_007_147_D_144_2800_HH_20070101T053038_20070101T053045_D00200_P_F_001')\n",
    "\n",
    "# expect 200\n",
    "print(\"status code: {}\".format(r.status_code))\n",
    "res = r.json()\n",
    "print(json.dumps(res, indent=2))\n",
    "\n",
    "# get hdf5 file name\n",
    "h5_file = res['result']['metadata']['FileName']\n",
    "print(h5_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You essentially get the granule's entire JSON metadata. Let's pull the URLs so that we can download the granule:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pull the urls\n",
    "urls = res['result']['urls']\n",
    "print(\"urls: {}\".format(urls))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to utilize the S3 URL so that we can utilize the S3 API for faster downloads:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_url = None\n",
    "for i in urls:\n",
    "    if i.startswith('s3://'): s3_url = i\n",
    "assert s3_url is not None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now let's download that dataset from S3, but before we can we need to populate the .aws/credentials file with the access key information. \n",
    "\n",
    "Use a terminal to execute aws-login:\n",
    "\n",
    "    aws-login -pub -p default -r us-west-2\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# get the S3 url format that awscli requires\n",
    "url = 's3://{}'.format(urlparse(s3_url).path[1:])\n",
    "print(url)\n",
    "local_dir = os.path.basename(url)\n",
    "print (local_dir)\n",
    "!aws s3 sync {url} {local_dir}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pwd\n",
    "!ls -al {local_dir}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!h5ls -r $local_dir/$h5_file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gdalinfo HDF5:\"$local_dir/$h5_file\"://science/LSAR/RSLC/swaths/frequencyA/HH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Next let's translate the SLC to the ENVI format to be read in by GDAL and visualized by \n",
    "\n",
    "Use a terminal to execute aws-login:\n",
    "\n",
    "    aws-login -pub -p default -r us-west-2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gdal_translate -of ENVI HDF5:\"$local_dir/$h5_file\"://science/LSAR/RSLC/swaths/frequencyA/HH HH.slc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gdalinfo HH.slc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from osgeo import gdal\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "ds = gdal.Open(\"HH.slc\", gdal.GA_ReadOnly)\n",
    "\n",
    "# extract a subset of the SLC to display\n",
    "x0 = 0\n",
    "y0 = 10\n",
    "x_offset = 1000\n",
    "y_offset = 1000\n",
    "#x_offset = 500\n",
    "#y_offset = 500\n",
    "\n",
    "#slc = ds.GetRasterBand(1).ReadAsArray()           \n",
    "slc = ds.GetRasterBand(1).ReadAsArray(x0, y0, x_offset, y_offset)\n",
    "#print(slc)\n",
    "ds = None\n",
    "\n",
    "fig = plt.figure(figsize=(20, 30))\n",
    "#fig = plt.figure(figsize=(14, 12))\n",
    "\n",
    "# display amplitude of the slc\n",
    "ax = fig.add_subplot(2,1,1)\n",
    "ax.imshow(np.abs(slc), vmin = -2, vmax=2, cmap='gray')\n",
    "ax.set_title(\"amplitude\")\n",
    "\n",
    "#display phase of the slc\n",
    "ax = fig.add_subplot(2,1,2)\n",
    "ax.imshow(np.angle(slc))\n",
    "ax.set_title(\"phase\")\n",
    "\n",
    "plt.show()\n",
    "\n",
    "slc = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"1\">This notebook is compatible with NISAR Jupyter Server Stack v1.4 and above</font>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "isce",
   "language": "python",
   "name": "isce"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}