Assignment_week_04a.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Week 04a Assignment weather data\n",
    "\n",
    "Welcome to week four of this course programming 1. Analyzing time related data such as estimating seasonal effect, or year effect might be a challenge. How to filter the essential information from the noise? How to apply signal analysis with noisy data. How to make compact useful visualizations? Python has several constructs to handle date time related data. The relevant classes for making plots are Locators and Formatters. Locators determine where the ticks are, and formatters control the formatting of tick labels. The relevant class for date time data is the pandas datetime data type, which has methods like resample and several possibilities to display data (frequencies). As a study case we will work with weather data. If you have data that fits the learning goals, you can bring your own data.\n",
    "\n",
    "Keywords: signal processing, smoothing, resample, formatters and locators, datetime object\n",
    "\n",
    "More to read: \n",
    "\n",
    "- https://fennaf.gitbook.io/bfvm22prog1/\n",
    "- https://machinelearningmastery.com/time-series-data-visualization-with-python/\n",
    "- https://towardsdatascience.com/how-to-plot-time-series-86b5358197d6\n",
    "- In the https://pandas.pydata.org/docs/reference/offset_frequency.html you can find more about frequencies and in the documentation\n",
    "- https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html you can read all the methods of this datetime object.\n",
    "- https://en.wikipedia.org/wiki/Smoothing\n",
    "\n",
    "\n",
    "Learning objectives\n",
    "\n",
    "- load, inspect and clean a dataset\n",
    "- reshape dataframes to group data in a certain frequency\n",
    "- apply smoothing technologies\n",
    "- Create useful visualisation with timeseries data\n",
    "- Maintain development environment \n",
    "- Apply coding standards and FAIR principles\n",
    "\n",
    "Please add topics you want to learn here: https://padlet.com/ffeenstra1/z9duo25d39dcgezz\n",
    "\n",
    "\n",
    "## Assignment\n",
    "\n",
    "You will to organise your data into the required format and apply smoothing. In this assignment we will work with weatherdata from the KNMI. A subset of weatherdata is for you available in the file: `KNMI_20181231`. The data consist of several stations with daily weather data of several years. Your task is to make a plot similar to the plot below. \n",
    "\n",
    "<img src=\"../images/weather.png\" alt=\"drawing\" width=\"400\"/>\n",
    "\n",
    "\n",
    "Furthermore the plot needs the following enhancements\n",
    "\n",
    "1. proper titles and ticks\n",
    "2. widgets selecting a particular year or all years\n",
    "3. lines need to be smoothed\n",
    "3. legends needs to be added\n",
    "\n",
    "Use your creativity. Consider colors, alpha settings, sizes etc. \n",
    "\n",
    "Learning outcomes\n",
    "\n",
    "- load, inspect and clean a dataset \n",
    "- reformat dataframes\n",
    "- apply smoothing technologies\n",
    "- visualize timeseries data\n",
    "\n",
    "The assignment consists of 6 parts:\n",
    "\n",
    "- [part 1: load the data](#0)\n",
    "- [part 2: clean the data](#1)\n",
    "- [part 3: reformat data](#2)\n",
    "- [part 4: smooth the data](#3)\n",
    "- [part 5: visualize the data](#4)\n",
    "- [part 6: Challenge](#5)\n",
    "\n",
    "Part 1 and 5 are mandatory, part 6 is optional (bonus)\n",
    "Mind you that you cannot copy code without referencing the code. If you copy code you need to be able to explain your code verbally and you will not get the full score. \n",
    "\n",
    "\n",
    "NB if you want to make a plot with more actual data you can download data from https://openweathermap.org/api \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='0'></a>\n",
    "## Part 1: Load the data\n",
    "\n",
    "Either load the dataset `KNMI_20181231.csv` or `KNMI_20181231.txt.tsv`. \n",
    "Preferably we read the data not with a hard coded data path but using a config file. See https://fennaf.gitbook.io/bfvm22prog1/data-processing/configuration-files/yaml. The dataheaders contain spaces and are not very self explainable. Change this into more readable ones. Select data from a station. Station 270 is in the neighborhood of Groningen. For our plot we only need the the mean, minimum and maximum temperature. Of course you are welcome to select other data if you think it might be useful for your visualization. The data should look something like this:\n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "       station       Date  Tmean  Tmin  Tmax\n",
    "97641      270 2000-01-01   42.0  -4.0  79.0\n",
    "97642      270 2000-01-02   55.0  33.0  74.0\n",
    "97643      270 2000-01-03   74.0  49.0  89.0\n",
    "97644      270 2000-01-04   46.0  22.0  75.0\n",
    "97645      270 2000-01-05   41.0  14.0  56.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='1'></a>\n",
    "## Part 2: Clean the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data ia not clean. There are empty cells in the dataframe which needs to be replaced with NaN's and the temperature is in centidegrees which needs to be transformed into degrees. The date field needs a datetime format. For visualization convience we would like to remove the leap year. Conduct the cleaning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#replace cells with spaces to NaN\n",
    "#change data formats\n",
    "#change temperatures to celcius degrees\n",
    "#remove leap year"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>    \n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
    "</summary>\n",
    "<ul><li>pd.to_datetime(df['Date'].astype(str), format='%Y%m%d')</li>\n",
    "    <li>regex for empty cells = `^\\s*$` </li>\n",
    "    <li>remove month == 2 & day == 29</li> \n",
    "</ul>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Test your outcome\n",
    "#write code to check if you have done the above"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Expected outcome"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "DatetimeIndex: 24820 entries, 1951-01-01 to 2018-12-31\n",
    "Data columns (total 5 columns):\n",
    " #   Column   Non-Null Count  Dtype         \n",
    "---  ------   --------------  -----         \n",
    " 0   station  24820 non-null  int64         \n",
    " 1   Date     24820 non-null  datetime64[ns]\n",
    " 2   Tmean    24819 non-null  float64       \n",
    " 3   Tmin     24819 non-null  float64       \n",
    " 4   Tmax     24819 non-null  float64       \n",
    "dtypes: datetime64[ns](1), float64(3), int64(1)\n",
    "memory usage: 1.1 MB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='2'></a>\n",
    "## Part 3: Reform your data\n",
    "\n",
    "First we will split the data in data from 2018 and data before 2018. Best is to split this in two dataframes. \n",
    "Next we need for the non 2018 data the minimum values for each day and the maximum values for each day. So we look for the minimum value out of all january-01 minimum values (regardless the year). Create a dataframe with 365 days containing the ultimate minimum and the ultimate maximum per day. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def month_day(df_multipleyears):\n",
    "    #your code to reform data here\n",
    "    \n",
    "    print(df_groupedbymonthday)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Test your code\n",
    "def test_reformed(df)\n",
    "    #\n",
    "    df = df[(df.index.year > 2007) & (df.index.year < 2018)]\n",
    "    month_day(df)\n",
    "\n",
    "test_reformed(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>    \n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
    "</summary>\n",
    "<ul><li>use the dt.month and dt.day to groupby</li>\n",
    "</ul>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Expected outcome\n",
    "Note, the layout or names my differ, but the length should be 365 and the minimum values should be the same"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "month  day\n",
    "1      1      -5.8\n",
    "       2      -7.5\n",
    "       3     -12.6\n",
    "       4      -4.1\n",
    "       5      -6.0\n",
    "              ... \n",
    "12     27     -4.8\n",
    "       28     -4.2\n",
    "       29     -6.7\n",
    "       30    -10.2\n",
    "       31    -10.6\n",
    "Name: Tmin, Length: 365, dtype: float64"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='3'></a>\n",
    "## Part 4: Smooth the data\n",
    "\n",
    "Make a function that takes an array or a dataframe column and returns an array of smoothed data. Explain in words why you choose a certain smoothing algoritm. Ask the signal analysis teacher if you want some advice.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here\n",
    "#your motivation here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='4'></a>\n",
    "## Part 5: Visualize the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the mean temperature of the year 2018. Create a shaded band with the ultimate minimum values and the ultimate maximum values from the multi-year dataset. Add labels, titles and legends. Use proper ranges. Be creative to make the plot attractive. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>    \n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
    "</summary>\n",
    "<ul><li>use from bokeh.models import Band</li>\n",
    "    <li>use ColumnDataSource to parse data arrays</li>\n",
    "    <li>look for xaxis tick formatters</li>\n",
    "</ul>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='5'></a>\n",
    "## Part 6: Challenge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make a widget in which you can select the year range for the multiyear set. Or maybe a widget were you choose a different station. Add this to your layout to make the plot interactive. Add another widget to select or deselect the smoother. Inspiration: https://demo.bokeh.org/weather"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.4 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  },
  "vscode": {
   "interpreter": {
    "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}