{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Calculate Number of Frost Days"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook computes the annual number of frost days for a given year, by using the daily minimum temperature (`tasmin`). The number of frost days index is the annual count of days where `tasmin` < 0 °C."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This Jupyter notebook is meant to run in the Jupyterhub server of the German Climate Computing Center [DKRZ](https://www.dkrz.de/) which is an [ESGF](https://esgf.llnl.gov/) repository that hosts 4 petabytes of CMIP6 data. Please, choose the Python 3 unstable kernel on the Kernel tab above, it contains all the common geoscience packages. See more information on how to run Jupyter notebooks at DKRZ [here](https://www.dkrz.de/up/systems/mistral/programming/jupyter-notebook). Find there how to run this Jupyter notebook in the DKRZ server out of the Jupyterhub, which will entail that you create the environment accounting for the required package dependencies. Running this Jupyter notebook in your premise, which is also known as [client-side](https://en.wikipedia.org/wiki/Client-side) computing, will also require that you install the necessary packages on you own but it will anyway fail because you will not have direct access to the data pool. Direct access to the data pool is one of the main benefits of the [server-side](https://en.wikipedia.org/wiki/Server-side) data-near computing we demonstrate in this use case. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this use case you will learn the following:\n",
    "- How to access a dataset from the DKRZ CMIP6 model data archive\n",
    "- How to count the annual number of frost days globally for a specified year\n",
    "- How to visualize the results\n",
    "\n",
    "\n",
    "You will use:\n",
    "- [Intake](https://github.com/intake/intake) for finding the data in the catalog of the DKRZ archive\n",
    "- [Xarray](http://xarray.pydata.org/en/stable/) for loading and processing the data\n",
    "- [Cartopy](https://pypi.org/project/Cartopy/) for visualizing the data in the Jupyter notebook and save the plots in your local computer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Import Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import intake           # to find data in a catalog, this notebook explains how it works\n",
    "import xarray as xr     # handling labelled multi-dimensional arrays\n",
    "\n",
    "# plotting libraries\n",
    "import matplotlib.pyplot as plt\n",
    "import cartopy.crs as ccrs\n",
    "from cartopy.util import add_cyclic_point"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Select the Year for Frost Days Calculation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "year = \"2100\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Intake Catalog\n",
    "Similar to the shopping catalog at your favorite online bookstore, the intake catalog contains information (e.g. model, variables, and time range) about each dataset (the title, author, and number of pages of the book, for instance) that you can access before loading the data. It means that thanks to the catalog, you can find where is the book just by using some keywords and you do not need to hold it in your hand to know the number of pages, for instance.\n",
    "\n",
    "### 2.1. Load the Intake Catalog and Orientate\n",
    "We load the catalog descriptor with the intake package. The catalog is updated daily. The catalog descriptor is created by the DKRZ developers that manage the catalog, you do not need to care so much about it, knowing where it is and loading it is enough:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Path to master catalog on the DKRZ server\n",
    "#dkrz_catalog=intake.open_catalog([\"https://dkrz.de/s/intake\"])\n",
    "#\n",
    "#only for the web page we need to take the original link:\n",
    "parent_col=intake.open_catalog([\"https://gitlab.dkrz.de/data-infrastructure-services/intake-esm/-/raw/master/esm-collections/cloud-access/dkrz_catalog.yaml\"])\n",
    "list(parent_col)\n",
    "\n",
    "# Open the catalog with the intake package and name it \"col\" as short for \"collection\"\n",
    "col=parent_col[\"dkrz_cmip6_disk\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what is inside the intake catalog. The underlying data base is given as a pandas dataframe which we can access with `col.df`. Hence, `col.df.head()` shows us the first rows of the table of the catalog."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "col.df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This catalog contains all datasets of the CMIP6 archive at DKRZ. Before searching for the needed data file we will have a closer look at the cataloge."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "100 climate models `source_id` are part of the catalog."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "col.unique([\"source_id\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The catalog offers 1162 variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#col.unique([\"variable_id\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can create a subset of the catalog with pandas operations. Here we select all CMIP activities (`col.df[\"activity_id\"] == \"CMIP\"`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "col.df[col.df[\"activity_id\"] == \"CMIP\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2. Browse the Intake Catalog\n",
    "\n",
    "The most elegant way for creating subsets is a query with a dictionary. In our case we design the search dictionary as follows: We chose the Max-Planck Earth System Model in Low Resolution Mode (\"MPI-ESM1-2-LR\") and the minimm temperature near surface (`tasmin`) as variable. We also choose an experiment. CMIP6 comprises several kind of experiments. Each experiment has various simulation members. you can find more information in the CMIP6 Model and Experiment Documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is how we tell intake what data we want\n",
    "\n",
    "query = dict(\n",
    "    source_id      = \"MPI-ESM1-2-LR\", # here we choose Max-Plack Institute's Earth Sytem Model in high resolution\n",
    "    variable_id    = \"tasmin\",        # temperature at surface, minimum\n",
    "    table_id       = \"day\",           # daily frequency\n",
    "    experiment_id  = \"ssp585\",        # what we selected in the drop down menu,e.g. SSP2.4-5 2015-2100\n",
    "    member_id      = \"r10i1p1f1\",     # \"r\" realization, \"i\" initialization, \"p\" physics, \"f\" forcing\n",
    ")\n",
    "\n",
    "# Intake looks for the query we just defined in the catalog of the CMIP6 data pool at DKRZ\n",
    "col.df[\"uri\"]=col.df[\"uri\"].str.replace(\"lustre/\",\"lustre02/\") \n",
    "cat = col.search(**query)\n",
    "\n",
    "del col # Make space for other python objects\n",
    "\n",
    "# Show query results\n",
    "cat.df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result of the query are like the list of results you get when you search for articles in the internet by writing  keywords in your search engine (Duck duck go, Ecosia, Google,...). Thanks to the intake package, we did not need to know the path of each dataset, just selecting some keywords (the model name, the variable,...) was enough to obtain the results. If advance users are still interested in the location of the data inside the DKRZ archive, intake also provides the path and the OpenDAP URL (see the last columns above). \n",
    "\n",
    "\n",
    "Now we will find which file in the dataset contains our selected year so in the next section we can just load that specific file and not the whole dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3. Find the Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a copy of cat.df, thus further modifications do not affect it \n",
    "query_result_df = cat.df.copy() # new dataframe to play with\n",
    "\n",
    "# Each dataset contains many files, extract the initial and final year of each file \n",
    "query_result_df[\"start_year\"] = query_result_df[\"time_range\"].str[0:4].astype(int) # add column with start year\n",
    "query_result_df[\"end_year\"] = query_result_df[\"time_range\"].str[9:13].astype(int) # add column with end year\n",
    "\n",
    "# Delete the time range column\n",
    "query_result_df.drop(columns=[\"time_range\"], inplace = True) # \"inplace = False\" will drop the column in the view but not in the actual dataframe\n",
    "query_result_df.iloc[0:3]\n",
    "\n",
    "# Select the file that contains the year we selected in the drop down menu above, e.g. 2015\n",
    "selected_file = query_result_df[(int(year) >= query_result_df[\"start_year\"]) & (\n",
    "                   int(year) <= query_result_df[\"end_year\"])]\n",
    "\n",
    "# Path of the file that contains the selected year    \n",
    "selected_path = selected_file[\"uri\"].values[0] \n",
    "\n",
    "# Show the path of the file that contains the selected year\n",
    "selected_path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Load the Model Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds = xr.open_dataset(selected_path)\n",
    "\n",
    "# Open variable \"tasmin\" over the whole time range\n",
    "ds_tasmin = ds[\"tasmin\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_tasmin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_tasmin_year = ds_tasmin.sel(time=year)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Count the number of frost days"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_tasmin_year_count = ds_tasmin_year.where(ds_tasmin_year < 273.15).count(dim='time')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before plotting, a cyclic point has to be added, otherwise there will be a gap at the prime meridian."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lon = ds_tasmin_year_count.lon\n",
    "lat = ds_tasmin_year_count.lat\n",
    "#ds_tasmin_year_count, lon = add_cyclic_point(ds_tasmin_year_count, lon)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Plot data with `cartopy`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize=(10, 5))\n",
    "ax = fig.add_subplot(1, 1, 1, projection=ccrs.Mollweide())\n",
    "\n",
    "plt.contourf(lon, lat, ds_tasmin_year_count, 60,\n",
    "            transform=ccrs.PlateCarree(),\n",
    "            cmap='Blues')\n",
    "ax.coastlines()\n",
    "ax.set_global()\n",
    "\n",
    "# Add a color bar\n",
    "plt.colorbar(ax=ax)\n",
    "plt.title('Number of Frost Days in Year ' +year)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "python3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}