2i2c JupyterHub Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "59806ecf-ec22-4fcc-8de8-a2bec6f40178",
   "metadata": {},
   "source": [
    "# Cloud Computing with JupyterHub and M²LInES\n",
    "\n",
    "James Munroe, [2i2c](https://2i2c.org) Product and Community Lead"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c0f74ef-636a-49a6-b2e2-b4b8056b894a",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Welcome and Introductions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fac8c251-3d23-48eb-b677-57327214bdb3",
   "metadata": {},
   "source": [
    "Goals:\n",
    "- Guided Tour of 2i2c's Managed JupyterHub\n",
    "- Git / GitHub Workflow for Collaborating in Research\n",
    "- Data on the Hub and in the Cloud\n",
    "- Machine Learning Possibilities"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7e82bf6-e51e-4954-85e8-97c975433753",
   "metadata": {},
   "source": [
    "Audience assumptions:\n",
    "- Previous experience with Python and Jupyter Notebooks\n",
    "- Working with Machine Learning (e.g. scikit-learn or PyTorch)\n",
    "- Innovating the new advances in Earth Systems Modelling"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5b45cba-21ce-4779-a218-cc520b8f1c58",
   "metadata": {},
   "source": [
    "## Jupyter Notebooks vs JupyterHub"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2407593c-d1ad-41ed-af8c-74c044a8b5cf",
   "metadata": {},
   "source": [
    "Computational notebooks, such as Jupyter Notebooks (`.ipynb`), are ushering in a new wave of interactive, collaborative science. \n",
    "\n",
    "**Erdmann, C., S. Stall, B. Hanson, L. Lyon, B. Sedora, M. Giampoala, and M. Ricci (2022), [Notebooks Now! elevating computational notebooks](https://doi.org/10.1029/2022EO225024), Eos, 103, (18 Aug 2022):**\n",
    "\n",
    "> - Researchers are increasingly using computational notebooks to share workflows and data analyses with others.\n",
    "> - Research computing services often highlight their support of notebooks as a method to interact with their services and facilitate collaboration and sharing. \n",
    "> - As a result, notebooks are fast becoming ubiquitous in research workflows.\n",
    "> - Notebooks fuse research narratives with data, visualization, and executable code to create an interactive experience.\n",
    "\n",
    "Jupyter Notebook = Text + Code + Output + Plots + More\n",
    "\n",
    "<img src=\"https://jupyter.org/assets/homepage/main-logo.svg\" style=\"display:block; margin-left: auto; margin-right: auto;width:10%\"/>\n",
    "\n",
    "[Project Jupyter](https://jupyter.org/) supports an ecosystem of related products: Jupyter Notebooks, JupyterLab, JupyterHub\n",
    "\n",
    "\n",
    "<img src=\"https://jupyter.org/assets/homepage/hublogo.svg\" style=\"display:block; margin-left: auto; margin-right: auto;width:20%\"/>\n",
    "\n",
    "JupyterHub is a multi-user version of the notebook designed for companies, classrooms and research labs.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5886a9cb-e659-4496-b299-b54489ae6042",
   "metadata": {},
   "source": [
    "## Where is the Hub?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "309e12d5-eff7-46fa-888b-62b25092e393",
   "metadata": {},
   "source": [
    "The JupyterHub is available at\n",
    "\n",
    "https://m2lines.2i2c.cloud\n",
    "\n",
    "Open infrastructure: All of the code to manage this cloud-based JupyterHub is available on GitHub (minus security keys/tokens). 2i2c supports *Right to Replicate* meaning, if at some point in the future, you wanted to spin up your own hub you are completely welcome to do so.  See our infrastructure repo on [GitHub](https://github.com/2i2c-org/infrastructure/tree/master/config/clusters/m2lines). \n",
    "\n",
    "Some technical details:\n",
    "- Google Cloud Platform\n",
    "- Region: us-central1 (Council Bluffs, Iowa), 115,000 sq.ft.,97% Carbon Free Energy\n",
    "- N1 machine series: General-purpose machine series available on Skylake, Broadwell, Haswell, Sandy Bridge, and Ivy Bridge CPU platforms.\n",
    " - Support up to 96 vCPUs and 624 GB of memory\n",
    " - Optional: NVIDIA Telsa K80 GPUs (4992 CUDA cores)\n",
    " \n",
    "<img src=\"https://lh3.googleusercontent.com/_T5y2eKUusOWBn44MkgTDc1EQVsiGkvWXDDgbNZxeKOp1aHKYpIMS56JhU3esg6F_V6sbmGmxmThuk5ugETygfPdv2ssbVRjHD3fcw=w1200-l100-sg-rj-c0xffffff\" style=\"display:block; margin-left: auto; margin-right: auto;width:50%\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8565d12b-a3af-4ae4-b588-ba5aea81569d",
   "metadata": {},
   "source": [
    "**Activity: Start your own Jupyter server**\n",
    "- Open a web browser: https://m2lines.2i2c.cloud\n",
    "- Click on \"Log in to continue\"\n",
    "- Authentication is handled by membership in the m2lines GitHub organization\n",
    "- Select the **Huge** instance to start your own, private, Jupyter server instance\n",
    "- While this is getting started, let's chat about what is occuring behind the scenes:\n",
    "  - Kubernetes: container orchestration\n",
    "  - Containers vs Virtual Machines\n",
    "  - 1. Auto-scaling cluster: if a node of the right size isn't available, it will be provisioned (takes a moment)\n",
    "  - 2. Download container image (https://github.com/pangeo-data/pangeo-docker-images pangeo/pangeo-notebook+2022.06.02)\n",
    "  - 3. Start Jupyter Server"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac7bfa66-1626-4963-ae5a-2bd964f3fb53",
   "metadata": {},
   "source": [
    "<img src=\"JupyterHubEventLog.png\" width=80%/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64e0b150-dd8a-4697-8335-b76a6bd3c78a",
   "metadata": {},
   "source": [
    "### [2i2c's Shared Responsibility Model](https://docs.2i2c.org/en/latest/about/service/shared-responsibility.html)\n",
    "\n",
    "2i2c shares responsibility for each hub with the communities we serve. We do this by defining the responsibilities that are a good fit for the skills and goals of each organization. This “Shared Responsibility Model” is a useful way to understand what actions communities are still expected to perform under a service agreement with 2i2c.\n",
    "\n",
    "<img src=\"https://drive.google.com/uc?export=download&id=1SIhHrzPXSFBZ0yyVpxHm0WYs63k0SBRQ\"/>\n",
    "\n",
    "An overview of some categories of shared responsibility between the [Cloud Engineering Team](https://docs.2i2c.org/en/latest/about/service/team.html#term-Cloud-Engineering-Team) and the [Community Leadership Team](https://docs.2i2c.org/en/latest/about/service/team.html#term-Community-Leadership-Team).\n",
    "\n",
    "<img src=\"https://drive.google.com/uc?export=download&id=1S6Y9TQcXXLkrGrhgXQc7kLzq7dxcuw9a\"/>\n",
    "\n",
    "An overview of some categories of shared responsibility between the [Community Support Team](https://docs.2i2c.org/en/latest/about/service/team.html#term-Community-Support-Team) and the [Community Leadership Team](https://docs.2i2c.org/en/latest/about/service/team.html#term-Community-Leadership-Team)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c81db820-91a1-4f0d-9387-4177958c76f5",
   "metadata": {},
   "source": [
    "Think of **2i2c** as your data engineering and cloud infrastructure focused team member so you can concentrate on doing the science!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35560c60-141e-43f0-b6be-22fe20f2e29a",
   "metadata": {},
   "source": [
    "## Introduction to Jupyter and Git"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c677103-2138-429a-8b2b-933782114f94",
   "metadata": {},
   "source": [
    "Some suggestions made to me from Ryan Abernathy:\n",
    "\n",
    "- Explain how the 2i2c service works, not teach them everything they might need to know to do their science. Spend time on things that seem obvious to you, like\n",
    "  - starting and stopping your server\n",
    "  - selecting a machine type\n",
    "  - opening and closing notebooks\n",
    "- Spend A LOT of time on the git / github workflow. A good goal would be to make sure that all participants are able to push to github via `github-scoped-creds`\n",
    "- Make them understand how the environment works, how it evolves, and how to customize it via pip / conda installs\n",
    "\n",
    "Reference: https://github.com/fperez/demo-jupyter-git\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ebecc09-2e16-4582-b4cb-e4ad7ceb878b",
   "metadata": {},
   "source": [
    "### JupyterHub Activity: Starting and Stopping the Server\n",
    "\n",
    "- You can stop your server using the Hub Control Panel https://m2lines.2i2c.cloud/hub/home\n",
    "- Jupyter server stays running while you are not active\n",
    "  - Independent of web browser. You can shift from a workstation to a laptop and keep on working.\n",
    "- Python kernels are 'culled' after 1 hour of no activity\n",
    "- Jupyter servers are not culled. Please shutdown to conserve resources.\n",
    "  - But this can be changed as needed."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1eaaf70f-e3bd-42f2-bbb6-9244f435fcd0",
   "metadata": {},
   "source": [
    "### Git Activity: Fork and clone a repo\n",
    "\n",
    "This tutorial is available on GitHub at https://github.com/jmunroe/2i2c-m2lines\n",
    "\n",
    "1. Fork this repo into your own GitHub account\n",
    "2. Clone your forked repo in the JupyterHub (address will be something like https://github.com/GITHUB_USERID/2i2c-m2lines )\n",
    "3. Add a new file and push back to GitHub (use `github-scoped-creds`)\n",
    "\n",
    "Discuss Git/GitHub workflows."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0edf02d-d979-4cae-9ba6-a3de30382bfc",
   "metadata": {},
   "source": [
    "### Conda Activity: Explore L96 Model Notebooks\n",
    "\n",
    "JupyterBook: [Learning Machine Learning with Lorenz-96](https://m2lines.github.io/L96_demo)\n",
    "\n",
    "GitHub repo: https://github.com/m2lines/L96_demo\n",
    "\n",
    "Example: `Neural_network_for_Lorenz96.ipynb` \n",
    "\n",
    "How to get `torch` (or any other library) installed if it is not there?\n",
    "- Option 1: Install the exact same environment that was used before\n",
    "    - `conda create --prefix L96M2lines --file conda-linux-64.lock`\n",
    "- Option 2: Use a container image that has PyTorch already installed\n",
    "- Option 3: Manage your own environment\n",
    "  - `conda env create --prefix myenv`\n",
    "  - Also works with `pip install`\n",
    "\n",
    "Managing your own environment is also great for testing packages at the \"bleeding edge.\"\n",
    "\n",
    "*Careful* `/home/jovyan` is limited to 10 GB.  It's designed for notebooks, analysis scripts, and small datasets (< 1GB).  Trying to manage multiple large conda environment could lead to storage issues.\n",
    "- Solution: Create a community image with the packages, software, and libraries your community uses."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad4dfc23-cdfe-491d-8dd0-9f79f445a4d6",
   "metadata": {},
   "source": [
    "# [Files and Data in the Cloud](https://docs.2i2c.org/en/latest/user/storage.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1def2ff-9ad4-454f-af90-f723aca1aabe",
   "metadata": {},
   "source": [
    "- Use your home directory to store code, notebooks, and small data files (<1 GB) for personal use\n",
    "- Use cloud object storage to store larger datasets and to share data across your team\n",
    "- Consider whether your project would benefit from other cloud-native data storage solutions such as a database, data warehouse, or data lake"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7721a125-b2d2-407c-a943-a48ab4ca0bd0",
   "metadata": {},
   "source": [
    "### The JupyterHub Filesystem"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0885233e-6731-48c1-80d7-fda9aa611bc5",
   "metadata": {},
   "source": [
    "When you start a Jupyter server on the Hub, it is effectively a private Linux 'virtual machine'\n",
    "\n",
    "To move files to and from JupyterHub:\n",
    "- Drag and Drop a file to file browser to upload\n",
    "- Right-click to download back out\n",
    "\n",
    "Terminal\n",
    "- You can ssh/scp/ftp to a remote system\n",
    "- However, you can't ssh in!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "587f432f-114f-4292-bcb6-80f405c4f355",
   "metadata": {},
   "source": [
    "#### Your Home Directory\n",
    "\n",
    "Your username is `jovyan`, and your home directory is `/home/jovyan`. This is the same for all users, but no one else can see or access the files in your home directory.\n",
    "\n",
    "`/home/jovyan` is a persistant network-attached drive. Any files you put there will be there when you log out and log back into the JupyterHub.\n",
    "\n",
    "The `/home/jovyan` space is typically limited to 10 GB. Consequently, your home directory is intended only for notebooks, analysis scripts, and small datasets (< 1 GB). It is not an appropriate place to store large datasets.\n",
    "\n",
    "#### The `shared` Directory\n",
    "All users have a directory called shared in their home directory. This is a readonly directory - anybody on the hub can access and read from the shared directory. The hub administrator may choose to distribute shared materials via this directory. The shared directory is not intended as a way for hub users to share data with each other.\n",
    "\n",
    "#### The `/tmp` Directory\n",
    "Any directory outside of `/home/jovyan` is emphemeral on Cloud-hosted JupyterHubs. This means if you add data or scripts under a writeable directory like `/tmp/myfile.txt` it will not be there when you log out and log back in.\n",
    "\n",
    "Nevertheless, `/tmp` is a convenient location for storing data temporarily because it is a fast SSD drive. The space available depends on your server but will generally be much larger than `/home/jovyan` (50-100s of GB).\n",
    "\n",
    "You can use the full path in your code or add a symlink from your home directory: `ln -s /tmp ~/tmp`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11d17cdd-e85b-4bc1-a23b-2c9f8980294e",
   "metadata": {},
   "source": [
    "### Using Git / GitHub\n",
    "\n",
    "The recommended way to move code in and out of the hub is via git / GitHub. You should clone your project repo from the terminal and use git pull / git push to update and push changes. In order to push data to GitHub from the hub, you will need to set up GitHub authentication. ``gh-scoped-creds` should be already setup on your 2i2c managed JupyterHub, and we shall use that to authenticate to GitHub for push / pull access.\n",
    "\n",
    "Open a terminal in JupyterHub, run gh-scoped-creds and follow the prompts.\n",
    "\n",
    "Alternatively, in a notebook, run the following code and follow the prompts:\n",
    "\n",
    "```\n",
    "import gh_scoped_creds\n",
    "%ghscopedcreds\n",
    "```\n",
    "\n",
    "You should now be able to push to GitHub from the hub! These credentials will expire after 8 hours (or whenever your JupyterHub server stops), and you’ll have to repeat these steps to fetch a fresh set of credentials. Once you authenticate, you’ll be provided with a link to a GitHub App that you have to install on the repositories you want to be able to push to from this particular JupyterHub. You only need to do this once per JupyterHub, and can revoke access any time. You can always provide access to your own personal repositories, but might need approval from admins of GitHub organizations if you want to push to repos in that organization."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f92bb20-70fc-44af-8821-234c78e92ebb",
   "metadata": {},
   "source": [
    "### Cloud Object Storage\n",
    "\n",
    "Your hub lives in the cloud. The preferred way to store data in the cloud is using cloud object storage, such as Google Cloud Storage (GCS). Cloud object storage is essentially a key/value storage system. They keys are strings, and the values are bytes of data. Data is read and written using HTTP calls.\n",
    "\n",
    "The performance of object storage is very different from file storage. On one hand, each individual read / write to object storage has a high overhead (10-100 ms), since it has to go over the network. On the other hand, object storage “scales out” nearly infinitely, meaning that we can make hundreds, thousands, or millions of concurrent reads / writes. This makes object storage well suited for distributed data analytics. However, data analysis software must be adapted to take advantage of these properties."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55e5444b-fb8c-4214-af59-7fe74ff8a641",
   "metadata": {},
   "source": [
    "#### Scratch Bucket\n",
    "The M2LInES 2i2c environments is configured with a “scratch bucket,” which allows you to temporarily store data (for example, when you need to store intermediate files during data transformations). Credentials to write to the scratch bucket are pre-loaded into your Hub’s user environment.\n",
    "\n",
    "*Warning*: Any data in scratch buckets will be deleted once it is 7 days old. Do not use scratch buckets to store data permanently.\n",
    "\n",
    "The location of your scratch bucket is contained in the environment variable `SCRATCH_BUCKET`.\n",
    "\n",
    "A common set of credentials is currently used for accessing scratch buckets. This means users can read, and potentially remove / overwrite, each others’ data. You can avoid this problem by always using `SCRATCH_BUCKET` as a prefix. Still, you should not store any sensitive or mission-critical data in the scratch bucket."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95f279cc-30e1-4688-ac70-b7dcdf800e88",
   "metadata": {},
   "source": [
    "#### Working with Object Storage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20553ca1-b188-4617-94ef-9fd48fd016a2",
   "metadata": {},
   "source": [
    "**Case Study**: Janni Yuval and Paul O’Gorman's work on \"Parameterizations in Atmospheric Models\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11ee1a41-b69b-4d6b-8805-9287e35bd252",
   "metadata": {},
   "source": [
    "Why did I choose this example? \n",
    "- Code available (GitHub)\n",
    "- Data available (Google Drive)\n",
    "- Talk available (YouTube)\n",
    "- Nice example of Open Science! "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "951b2139-b20e-4cdc-9fe0-0819871c383f",
   "metadata": {},
   "source": [
    "Observations:\n",
    "- Fortran model generating the \"high-resolution models\" on HPC\n",
    "- Matlab codes for model post-processing\n",
    "- Python scripts for machine learning\n",
    "- Jupyter notebooks"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "951403dc-fff0-49eb-8d82-8e45a4091bd2",
   "metadata": {},
   "source": [
    "### Abstract data access with `fsspec`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c958fef-b88e-433a-a1a0-c171bc34890f",
   "metadata": {},
   "source": [
    "`fsspec` is a Python library that abstracts the idea of a File System for *many* different cloud storage models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "216a8d90-5b22-4d22-99b2-5074845b0759",
   "metadata": {},
   "outputs": [],
   "source": [
    "import fsspec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "891fae12-6665-45ef-b23a-637fdb3a73b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "fsspec.available_protocols()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53656b74-1c47-434c-83cc-c192b8fc560b",
   "metadata": {},
   "source": [
    "Google Cloud Storage - GCS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9d628e9-79c4-4d50-8b02-1bb942459712",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "SCRATCH_BUCKET = os.environ['SCRATCH_BUCKET']\n",
    "SCRATCH_BUCKET"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "419565a5-901e-42a3-b08b-f989c1f41431",
   "metadata": {},
   "outputs": [],
   "source": [
    "gcs = fsspec.filesystem('gcs')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43ac7c09-f25e-48e1-993c-01cd32bda7ef",
   "metadata": {},
   "source": [
    "With an object, you can put a file (data) in the cloud and associate it with a key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "294370da-5a92-4ba5-b575-0dc94efbdc53",
   "metadata": {},
   "outputs": [],
   "source": [
    "gcs.put('README.md', f'{SCRATCH_BUCKET}/README.md')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c75db152-6424-4682-b101-3a8c52efa90a",
   "metadata": {},
   "source": [
    "But since object storage is really just a key-value mapping, this works too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b323c76d-8412-49a6-b427-d46d77ea6b68",
   "metadata": {},
   "outputs": [],
   "source": [
    "scratch = gcs.get_mapper(SCRATCH_BUCKET)\n",
    "scratch['MyNewFile'] = b'This is a Byte String'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4252a5f9-c445-4c44-8223-e7244e27df20",
   "metadata": {},
   "outputs": [],
   "source": [
    "gcs.ls(SCRATCH_BUCKET)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b314d73-f9d6-419a-bec4-43533e613fac",
   "metadata": {},
   "outputs": [],
   "source": [
    "scratch['README.md']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5223670-bec1-4852-8410-884ddb062e13",
   "metadata": {},
   "source": [
    "I've already taken one of Janni's high resolution atmospheric model runs (found at [this google drive](https://drive.google.com/drive/folders/1TRPDL6JkcLjgTHJL9Ib_Z4XuPyvNVIyY)) and pushed it into Google Cloud Storage.\n",
    "\n",
    "(Briefly, my approach was to use fsspec to connect to both Google Drive and Google Cloud Storage and copy the key-value pairs in a parallelized loop.  The Google Drive support in `fsspec` is experimental -- there are likely much better ways of pushing data directly into Google Cloud Storage with specialized tools.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4291f022-bbe9-4f04-a409-e50b3fcdb17c",
   "metadata": {},
   "outputs": [],
   "source": [
    "uri = 'gcs://m2lines-scratch/jmunroe/filesqobskm12x576'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1761c0a7-8b77-4e72-b106-3e36682d60cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "nc_files = gcs.ls(uri)\n",
    "nc_files[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3714971e-a498-4f92-8f87-9c5cb3874f83",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'Total number of NetCDF4 files: {len(nc_files)}')\n",
    "print(f'Size of dataset: {gcs.du(uri) / 2**30:.1f} GB')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de463920-e04f-4230-a5e4-550c7be2e989",
   "metadata": {},
   "source": [
    "Remember this scratch bucket will only keep files for 7 days -- not for permanent storage."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "091895df-94a7-4d15-bbb8-de8013a424c0",
   "metadata": {},
   "source": [
    "#### Transfer one file from the \"cloud\" to the local filesystem on the hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9c0b19b-b120-459d-97fc-3e09a8fbdfcb",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "gcs.get(nc_files[0], '~/') # this will copy the file into /home/jovyan with the same basename"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4dc49c4d-68ce-4ed0-a4d9-43e3fa82abbd",
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = os.path.basename(nc_files[0])\n",
    "filename"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c745686-3583-4136-bf15-d6a064a4c77a",
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls -lh ~/{filename}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d1f428f-354c-4f44-b414-8ad7e81d90b5",
   "metadata": {},
   "source": [
    "These are pretty decently sized files (880 MB) and it took ~10 seconds to download the file. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0778725f-add5-440e-b821-3052890ab6de",
   "metadata": {},
   "source": [
    "#### Using Xarray for data access"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d2af756-a7a5-4d47-a524-15abf4bb00cb",
   "metadata": {},
   "source": [
    "To read a NetCDF4 files, use the `xarray` library and `hvplot` from the Holoviews [ecosystem](https://hvplot.holoviz.org/) for visualization.\n",
    "\n",
    "See [xarray Tutorial](https://tutorial.xarray.dev) for additional guidance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "699979b8-5d32-4778-bbfb-56cff769763b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import hvplot.xarray\n",
    "import xarray as xr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2380f9de-9a47-4e19-acdf-e5f513739b70",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds = xr.open_dataset(f'~/{filename}')\n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8cbb1da-1acc-49ba-afa1-cb29d2f41515",
   "metadata": {},
   "source": [
    "Let's explore what we find in this *Water World*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a63b473-ecaf-4a2c-b34f-f6896a74b23b",
   "metadata": {},
   "source": [
    "#### Water Vapour"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d8ddfd0-8707-4d1c-801f-9575fcc15c26",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ds.Q.hvplot(x='x', y='y', clim=(0, 15), width=800, height=400)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "297233d5-b6cb-45dc-ac3d-c4e1b128bb59",
   "metadata": {},
   "source": [
    "Humidity highest along the warm equator?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1779f5d5-aa52-4941-b444-a1dc8476e3af",
   "metadata": {},
   "source": [
    "#### Non-precipitating Condensate (Water+Ice)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d99023d-fa29-4af0-91f6-cc47ee270027",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds.QN.hvplot(x='x', y='y',  clim=(0,0.5), width=800, height=400, cmap='blues_r')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a7d258b-e3ae-4c03-b8c7-b74952b9fd08",
   "metadata": {},
   "source": [
    "Clouds?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed302c02-1437-4a44-8d81-f659bd62120c",
   "metadata": {},
   "source": [
    "#### Precipitating Water (Rain + Snow)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0732551-6d1c-4375-8e34-dd6f88461337",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds.QP.hvplot(x='x', y='y', clim=(0,0.2), width=800, height=400, cmap='viridis')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adeb0e6c-def3-46ef-9503-25802d028292",
   "metadata": {},
   "source": [
    "Weather on Water World!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9d4a559-5800-487e-9e8a-da4a61a55bc1",
   "metadata": {},
   "source": [
    "#### Temperature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe527ee7-93fb-4c02-b4a2-2bf3a1e6e9ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds.TABS.hvplot(x='x', y='y', clim=(270, 310), width=800, height=600, cmap='coolwarm')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95cfe2e2-aecc-4500-a5a1-255d60e64cce",
   "metadata": {},
   "source": [
    "Preciptitation correlated with temperature fronts?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1223a209-0fc8-43cf-923b-4e2d88e0bab3",
   "metadata": {},
   "source": [
    "But that's only one time-step of this 4800 timestep dataset. At even 10s/timestep, it will take"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ec2b945-1e3f-43af-bfdc-5d81403e3ef7",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'{4800 * 10 / 3600:.1f} hours')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9115fe3b-3ff7-43a6-bfda-766be5f33524",
   "metadata": {},
   "source": [
    "To download the dataset it will take more than half a day (and 4 TB of local storage) onto this Jupyter Server. Not the right approach when working with cloud infrastructure."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcba7429-ba30-44da-93ce-ddbb9326cabf",
   "metadata": {
    "tags": []
   },
   "source": [
    "## [Pangeo-Forge](https://pangeo-forge.org/)\n",
    "\n",
    "A **Big Idea** of cloud computing is to avoid downloading data to be able to analyze it. Instead, bring your analysis to where the data is located.\n",
    "\n",
    "Framework for creating *Analysis Ready, Cloud Optimized* datasets in the cloud. This is non-trivial task and efforts to make those conversion could be shared across several communities.\n",
    "\n",
    "<img src=\"https://pangeo-forge.org/pangeo-forge-diagram.png\" width=800/>\n",
    "\n",
    "\n",
    "Covered in a seminar two weeks ago? How did that go?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b67dbaa5-2298-47c5-a18d-cf2595977ea1",
   "metadata": {},
   "source": [
    "- (Ryan A) For existing ARCO datasets, use stuff from https://pangeo-forge.org/catalog and https://catalog.pangeo.io/. Explain that Pangeo Forge is the main pathway to getting data into the cloud."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d696f96-2f0b-4093-86fa-cd99420c6d26",
   "metadata": {},
   "source": [
    "#### [Global Precipitation Climatology Project](https://pangeo-forge.org/dashboard/feedstock/42)\n",
    "\n",
    "daily, global 1x1-deg gridded fields of precipitation totals for 1996-2021 based on merged data sources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "62197a3d-661b-47cc-b073-224ff338db53",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "store = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/gpcp-feedstock/gpcp.zarr'\n",
    "ds = xr.open_dataset(store, engine='zarr', chunks={})\n",
    "ds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbf5f27f-8a0b-456f-829c-05f9d1e8691a",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds.precip.hvplot(x='longitude', y='latitude', clim=(0, 80), width=800, height=400, cmap='viridis', widget_type=\"scrubber\",\n",
    "    widget_location=\"bottom\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8f4e9fa-a426-4619-8f55-9817a4ab5a19",
   "metadata": {},
   "source": [
    "##### Compute a climatology"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0fa63c0-b7fb-4a6c-bb3f-17a11158b9d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds.precip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a884a25-93a3-42af-b599-90239f1a2764",
   "metadata": {},
   "outputs": [],
   "source": [
    "# filter out flagged data\n",
    "precip_climatology = ds.precip.where((ds.precip >= 0) & (ds.precip < 1000)).mean('time')\n",
    "precip_climatology"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6082ab24-75d9-4237-ad51-81800e318380",
   "metadata": {},
   "source": [
    "The calculation is *lazy*.  It will only be completed when needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d826dfc-6c1b-4b90-8a69-743353fe83f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from dask.diagnostics import ProgressBar\n",
    "with ProgressBar():\n",
    "    precip_climatology.load() # force the calculation to occur"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "052c7e37-b506-4c1c-a32e-73494711d6ba",
   "metadata": {},
   "source": [
    "Reducing 2.2 GB of remote data in ~5 seconds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c076497-f8f0-4b9d-a174-98ef84334f48",
   "metadata": {},
   "outputs": [],
   "source": [
    "precip_climatology.hvplot(x='longitude', y='latitude', width=800, height=400, cmap='viridis')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6071d6b8-f3b4-4286-b312-6ef6b3993542",
   "metadata": {},
   "source": [
    "But our *Water World* dataset isn't (yet?) converted to Zarr or on Pangeo-Forge. Can we still make progress?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa9922bb-b3ae-42d0-b0f1-a29a91f5e4af",
   "metadata": {},
   "source": [
    "## Kerchunk"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd06e1a1-e8ba-4cc6-b771-0b8bb54cc268",
   "metadata": {},
   "source": [
    "[Kerchunk](https://fsspec.github.io/kerchunk/) is library for abstracting out chunked, compressed data and virtually aggregating in to a Analysis Ready, Cloud-Optimized dataset.\n",
    "\n",
    "- Not quite good performance as Zarr\n",
    "- But there is **lots** of data on data providers using archival formats (like NetCDF4)\n",
    "- Avoids having to have multiple copies of the same data in different formats\n",
    "- Written by the same developers as `fsspec`\n",
    "\n",
    "Brand new Medium article (Sept 11, 2022): https://medium.com/pangeo/accessing-netcdf-and-grib-file-collections-as-cloud-native-virtual-datasets-using-kerchunk-625a2d0a9191"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78abdd8c-a841-4313-9f42-c51aa3219860",
   "metadata": {},
   "outputs": [],
   "source": [
    "from kerchunk.hdf import SingleHdf5ToZarr\n",
    "from kerchunk.combine import MultiZarrToZarr\n",
    "import ujson"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d34b74a5-51f7-4844-8bab-ebc789752788",
   "metadata": {},
   "outputs": [],
   "source": [
    "so = dict(mode='rb', default_fill_cache=False, default_cache_type='first')\n",
    "\n",
    "# compute the \"offsets\" into a NetCDF file\n",
    "def gen_json(u):\n",
    "    with gcs.open(u, **so) as infile:\n",
    "        \n",
    "        p = u.split('/')\n",
    "        fname = os.path.splitext(os.path.basename(u))[0]\n",
    "        outf = f'{fname}.json'\n",
    "        \n",
    "        if not os.path.exists(outf):\n",
    "            h5chunks = SingleHdf5ToZarr(infile, u, inline_threshold=300)\n",
    "            with open(outf, 'wb') as f:\n",
    "                f.write(ujson.dumps(h5chunks.translate()).encode());\n",
    "                \n",
    "    return outf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27f478a1-00ff-4ed0-ada7-b2d1453e550b",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "path = nc_files[1]  # we have not downloaded this file locally\n",
    "ref_json = gen_json(path) # one-time cost"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "062a7912-9ced-4ccf-bcf1-c5c4f1d564c6",
   "metadata": {},
   "source": [
    "Discuss what is really in this 'reference' JSON file.\n",
    "\n",
    "But with this file, we can lazily open the NetCDF file stored on Google Cloud Storage *as if* it was zarr file by using `fsspec`'s `reference` file system:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f28d1c99-9078-414c-aa70-a378bcf7c5b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "ds = xr.open_dataset(\"reference://\", engine=\"zarr\", chunks={},\n",
    "                     backend_kwargs={\n",
    "                        \"consolidated\": False,\n",
    "                        \"storage_options\": {\"fo\": ref_json, \n",
    "                                            \"remote_protocol\": \"gcs\"}\n",
    "                    })\n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a0c94b7-712f-4827-bc79-a0187ba4e276",
   "metadata": {},
   "source": [
    "Let's plot the zonal wind profile:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4f82a05-dfb0-4186-93a2-395d2ee11e2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "ds.U.mean('x').hvplot.line(y='y', width=400, height=600,)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc099245-f821-4e13-8d35-869618461cc1",
   "metadata": {},
   "source": [
    "#### Combine multiple kerchunk'd datasets into a single logical aggregate dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15a65b87-734e-4da9-aaa9-2dd986db8445",
   "metadata": {},
   "source": [
    "Preprocess all of the reference files for the JSON files. Since each NetCDF is completely independent, we can parallelize this one-time operation with Dask:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0aebc628-9eb3-4638-9419-5481d9f6b9f2",
   "metadata": {},
   "source": [
    "```\n",
    "import dask\n",
    "\n",
    "results = dask.compute(*[dask.delayed(gen_json)(u) for u in nc_files], retries=10)\n",
    "\n",
    "json_list = sorted(glob.glob('*.json'))\n",
    "\n",
    "mzz = MultiZarrToZarr(json_list, remote_protocol='gcs',\n",
    "        concat_dims=['time'], identical_dims = ['x', 'y', 'z'],\n",
    "    )\n",
    "\n",
    "mzz.translate('qobskm12x576.json')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "515743b9-a9ee-4be3-84ed-3c98d72a59ae",
   "metadata": {},
   "source": [
    "It took about 35 minutes to preprocess the entire 4TB archive and produce this reference file. I put this file in the `shared` folder at `/home/jovyan/shared/jmunroe/qobskm12x576.json`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01da21ed-4c67-45fb-bb93-812f8b39dcd2",
   "metadata": {},
   "outputs": [],
   "source": [
    "ref_json = '/home/jovyan/shared/jmunroe/qobskm12x576.json'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6aad10d-6431-4334-a4c4-d9ed4e6fdb42",
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls -lh {ref_json}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "628c1b5e-6eb6-4ee9-bead-08e376d9f3f4",
   "metadata": {},
   "source": [
    "This is 442 MB index file to the entire 4TB dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79831e03-d951-49b2-9fba-e4e486ed2e04",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "backend_args = { \"consolidated\": False,\n",
    "                 \"storage_options\": { \"fo\": ref_json,\n",
    "                 \"remote_protocol\": \"gcs\"  }}\n",
    "ds = xr.open_dataset(\"reference://\", engine=\"zarr\",\n",
    "                     chunks={},\n",
    "                     backend_kwargs=backend_args)\n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6e1a716-dafb-4b52-a27c-0ecad07023f7",
   "metadata": {},
   "source": [
    "So now we can open up the entire dataset at once (still lazy access):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9989de2-ee87-4eee-a63b-673509a1c307",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds.QP.hvplot(x='x', y='y', clim=(0,0.2), width=800, height=400, cmap='viridis')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44b0bb54-b076-401f-8726-df28bc5ea0fa",
   "metadata": {},
   "source": [
    "Back to the **case study**, a usual thing to do was to consider the vertically integrated precipitation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cf4562b-e55e-43b3-bfbb-5fbba3a09417",
   "metadata": {},
   "outputs": [],
   "source": [
    "QP = ds.QP.sum('z')\n",
    "QP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "779a8a9c-47ef-4179-9151-a65df3aa72e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "QP.hvplot(x='x', y='y', clim=(0, 1), width=800, height=400, cmap='viridis', widget_type=\"scrubber\",\n",
    "    widget_location=\"bottom\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d2da95c-e677-4f1c-8996-4a0ee8d471af",
   "metadata": {},
   "source": [
    "#### Compute a \"10 day\" climatology"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37e62000-fb3a-4cb5-8c85-d76112b35e1d",
   "metadata": {},
   "outputs": [],
   "source": [
    "QP_climatology = QP.isel(time=slice(0, 80)).mean('time')\n",
    "QP_climatology"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52b2d4de-cb13-4c7e-8ee0-7d1788352eab",
   "metadata": {},
   "outputs": [],
   "source": [
    "with ProgressBar():\n",
    "    QP_climatology.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6892c5d7-7fe9-458c-af6b-521f8505c086",
   "metadata": {},
   "outputs": [],
   "source": [
    "QP_climatology.hvplot(x='x', y='y', clim=(0, 1), width=800, height=400, cmap='viridis')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ff1c611-10c0-40b4-ab78-11684607b940",
   "metadata": {},
   "source": [
    "Functional, but we could also make the effort to convert to Zarr if the vertically integrated precip was going to be needed again:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aae439a6-15e0-481b-a2a1-06e4143c7e21",
   "metadata": {},
   "source": [
    "```\n",
    "# Takes about 35 mins of processing\n",
    "QP = QP.chunk({'time':20, 'y':1440, 'x':576}) # make chunk sizes larger. Typically, ~100MB is recommended for Zarr\n",
    "QP.to_dataset().to_zarr(f'gcs://m2lines-scratch/jmunroe/qobskm12x576_QP.zarr', consolidated=True)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "771ae100-303f-4605-be46-5ca3efde27d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "ds = xr.open_zarr(f'gcs://m2lines-scratch/jmunroe/qobskm12x576_QP.zarr', consolidated=True)\n",
    "ds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18ff5a10-2ed1-4980-8072-b7e29514f494",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds.QP.hvplot(x='x', y='y', clim=(0,1), width=800, height=400, cmap='viridis',  widget_type=\"scrubber\",\n",
    "    widget_location=\"bottom\",)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fad2cb37-217f-4cce-b441-222cf7f5b3d4",
   "metadata": {
    "tags": []
   },
   "source": [
    "We can now interactively compute the climatology the entire 4800 time step long sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c1ae54b-b06a-469b-b4b8-ba3e13baddfd",
   "metadata": {},
   "outputs": [],
   "source": [
    "with ProgressBar():\n",
    "    QP_mean = ds.QP.mean('time').compute()   # above we used .load() to replace a lazy array with its calculated version\n",
    "                                             # we can also use .compute() to force computation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30204c12-1634-48c5-8a96-0d46bd7553a5",
   "metadata": {},
   "source": [
    "And we can plot both the mean and the zonal average"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c26befa0-4fc1-4beb-ae2e-293075be2c11",
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    QP_mean.hvplot(x='x', y='y', clim=(0,1), width=800, height=400, cmap='viridis') + \n",
    "     QP_mean.mean('x').hvplot()\n",
    ").cols(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36194ac0-fc7c-4023-b945-b0e6768f9ee4",
   "metadata": {},
   "source": [
    "Continuing, we may want to compare this 'high-res' model (*truth*) to some coarsened representation (*observation*)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95826f90-7a22-45ff-af43-1bd79a3ff6b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "QP_coarse = ds.QP.coarsen(x=16, y=16).mean()\n",
    "\n",
    "with ProgressBar():\n",
    "    QP_coarse.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2344b370-db16-4551-a106-759dd8541e6e",
   "metadata": {},
   "outputs": [],
   "source": [
    "options = dict(x='x', y='y', clim=(0,1), width=800, height=400, cmap='viridis')\n",
    "(\n",
    "    ds.QP.hvplot(**options) + \n",
    "    QP_coarse.hvplot(**options)\n",
    ").cols(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c84a0377-8cc9-4198-bbab-aee58bd3a9d6",
   "metadata": {},
   "source": [
    "# Next Steps"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bd15799-0407-47e7-9360-8edd6f249849",
   "metadata": {},
   "source": [
    "Where to get more information:\n",
    "- Join the Pangeo Discourse [https://discourse.pangeo.io/](https://discourse.pangeo.io/) \n",
    "- GitHub issues are good for specific package topics but Discourse(s) can cover these science domain, multi-package problems"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e31754a4-ece3-47af-a1cf-94b6f5f436c4",
   "metadata": {},
   "source": [
    "What are your pain points? (computational or community related)\n",
    "\n",
    "How might infrastructure for interactive cloud computing improve to make your research more **impactful**, **accessible**, and **delightful**?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}