{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Data Standardization with CMOR via Python CDO\n", "\n", "This series introduces you to the functions of **cdo cmor**.\n", "\n", "**cdo cmor** calls the CMOR library to standardize climate model output for different projects." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Why refining data?\n", "\n", "Part of a sustainable scientifc workflow is the refinement of the produced data to make it **FAIR**. **FAIR** research data is **I**nteroperable and **R**eusable. Adopting a data standard helps to make data *interoperable* and *reusable* in many ways.\n", "\n", "**Interoperability**:\n", "\n", "- Use of a widely accepted common **data format** across all generated data such as [NetCDF]()\n", "- The inherited **compliance** to domain specific **conventions** like [CF]() simplifies processing.\n", "\n", "**Reusability**\n", "\n", "- Data that is stored with sufficient meta data according to the data standard is analyzable without external information and therefore **self-descriptive**. \n", " - E.g. statistical operations and interpolations over space and time are only processable if all temporal and spatial information including cells and interval bounds and a full description of the vertical axis is available in the input file.\n", "- Continoulsy developped applications aim at being **compatible** to accepted data standards ensuring a long-term usability\n", "- Definition of a **Data Reference Syntax** including templates for storage pathes and file names allows to identify a single file by the set of specified project attributes " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### The scope\n", "\n", "For (very) large climate community projects like the Coupled Model Intercomparison Project (CMIP), *systematic* analysis across models only easy to do if model output is provided as **FAIR** data. In CMIP6, 2000 unique variables are defined and which can be submitted for 100 different experiments." ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "## Approaches\n", "\n", "Two different approaches are possible:\n", "\n", "- Model output adaptation\n", "- Post-processing with specialized tools" ] }, { "cell_type": "markdown", "metadata": { "jp-MarkdownHeadingCollapsed": true, "tags": [] }, "source": [ "Reasons **against** model output adaptation\n", "\n", "- Data standards are *evolving* which require continous updates on output writing\n", "- Once adapted to a specific standard, the output is *inflexible* for new and other standards\n", "- Conservative scientists using stone-age but proofed software will be hard to convince to switch to a new output format\n", "- Since rarely data standard experts work on the adaptation, the task is time consuming and error-prone" ] }, { "cell_type": "markdown", "metadata": { "jp-MarkdownHeadingCollapsed": true, "tags": [] }, "source": [ "Reasons **for** Post-processing with specialized tools\n", "\n", "+ **Developer specialization**: experts on model development (post-processing) can *focus* on model development (post-processing) goals\n", "+ **Guarantee**: standardizing software *ensure*s data standard generation i.e. no flaws.\n", "+ **Compatibility**: Other older tools remain *compatible* with the original model raw output\n", "+ **Flexibility**: Enabling of quick adaptation of other data standards" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Definitions\n", "\n", "- [CMIP6](https://www.wcrp-climate.org/wgcm-cmip)\n", " - The recent phase 6 of the Coupled Model Intercomparison Project\n", "- [CMIP Data Standard](https://goo.gl/neswPr)\n", " - Convetion on climate data accepted in CMIP\n", "- [CMOR]()\n", " - the Climate Model Output Rewriter] can generate data compliant to the CMIP Data Standard.\n", "- [CDO](https://code.mpimet.mpg.de/)\n", " - Collection of operators to process climate data.\n", " - The python binding is a wrapper to call a specific binary correctly" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## CMOR\n", "\n", "The [Climate Model Output Rewriter]() tool can generate data compliant to the CMIP Data Standard.\n", "\n", "### Features\n", "\n", "- **Different** (CMIP-like) data standards can be produced\n", " - No user side preparation of data standard description\n", "- CMOR **ensures** that output is conform to the data standard. Building upon CMOR means using synergies which\n", " - avoids repeating work \n", " - helps to concentrated on the actual goal instead of debugging own cmor-lite developments" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```{note}\n", "**CMIP-like data standard** means:\n", "\n", "Each file must \n", "- contain only a single output data variable \n", "- cover only a single simulation \n", "- include coordinates and additional meta data\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Why integrating CMOR into CDOs?\n", "\n", "CDO\n", "\n", "- is widely used and accepted\n", "- has an active support by both users and developers\n", "- has an interface that allows\n", " - different infile formats\n", " - to access to all infile information no matter how structured\n", "- is fast because it is written in C++" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Installation of CDO with CMOR\n", "\n", "If you work on DKRZ HPC system, we recommend to work versions installed here:\n", "\n", "```bash\n", "ls -1 /work/bm0021/cdo_incl_cmor/\n", "```\n", "\n", "The older CMOR Version 2 is used to generate CMIP5 and CORDEX-CMIP5 data standard. Due to a design change in CMOR functions, there is **up- and downward incompatibility** of CMOR input data.\n", "\n", "The interface of cdo cmor and the format of user input does not change with the installed CMOR versions. Scripts and files used for one project can be the starting point for the next project." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Installation with conda\n", "\n", "Only the recent CMOR3 version can be installed and linked to CDO via conda:\n", "\n", "```bash\n", "conda update conda\n", "conda create --name cdocmorenv conda-forge/label/dev::cdo -c conda-forge\n", "source activate ${cdoenv}\n", "```\n", "\n", "The environment for CDO with cmor shall be **cdocmorenv** set by `--name cdocmorenv`. CDO will be installed from the develop-channel which contains CMOR by specifying `conda-forge/label/dev::cdo`. All other packages come from the **conda-forge channel** with `-c conda-forge`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Updating a conda installation of cdo with cmor:\n", "\n", "Depending on which additional packages you have installed, you may have to lower the *channel_priority* first.\n", "\n", "```bash\n", "conda config --set channel_priority flexible\n", "conda install --name ${cdoenv} conda-forge/label/dev::cdo -c conda-forge\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```{note}\n", "Debian CDO (`sudo apt-get install cdo`) is installed **without CMOR**\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1. Preparation. \n", "\n", "#### Option A: On Levante\n", "\n", "Define vars for CDO and working directories" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#- Recent path:\n", "import os\n", "pwd=os.getcwd()\n", "#\n", "workdir=\"/work/bm0021/cdo_incl_cmor/examples/\"\n", "cdodir=\"/work/bm0021/cdo_incl_cmor/\"\n", "#\n", "cdocmorinfo=workdir+\".cdocmorinfo\"" ] }, { "cell_type": "markdown", "metadata": { "tags": [] }, "source": [ "#### Option B: Local PC\n", "\n", "Clone repo for material:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!git clone https://gitlab.dkrz.de/dicad-pp/cdo-incl-cmor.git" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#- Recent path:\n", "import os\n", "pwd=os.getcwd()\n", "#\n", "basedir=pwd+\"/cdo-incl-cmor\"\n", "workdir=basedir+\"/application/handson/\"\n", "cdocmorinfo=pwd+\"/.cdocmorinfo\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2. Set-up cdo in python" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#set cdo binary to the one installed in the environment of the kernel\n", "import sys\n", "import os\n", "cdobin=\"/\".join(sys.executable.split(os.path.sep)[:-1])+\"/cdo\"\n", "#" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#import python cdo \n", "from cdo import *\n", "cdo = Cdo(cdobin)\n", "cdo.debug=True\n", "#This prohibits that existing files are created a second time\n", "cdo.forceOutput = False" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "help(cdo)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3. Interface" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "%%capture --no-stdout\n", "cdo.cmor(options=\"-h\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The operator requires one parameter and one argument. The first parameter is always the MIP-table. The argument is the input file." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Project data standard\n", "\n", "The project data standard is build up by 4 different type of documents:\n", "\n", "- **The Data Request (Dreq)**: A data standard will only be defined for variables that are *requested* for and by the project, e.g. [CMIP6](https://cmip6dr.github.io/Data_Request_Home/)\n", "- **Output requirements (OR)**: Technical specifications for the structure, content and format of files, e.g. [CMIP6](https://goo.gl/neswPr)\n", "- **Global attributes (GA)**: Specifications for required and optional global attributes, e.g. [CMIP6](https://goo.gl/v1drZl )\n", "- **A registry**: Only names of institutions and ESMs that are registred are valid values of global attributes like *institution* or *source*. E.g. [CMIP6](https://github.com/WCRP-CMIP/CMIP6_CVs)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Controlled Vocabularies (CVs) and MIP-Tables\n", "\n", "**DReq, GAs** and the **registry** are translated into *controlled vocabularies*, CVs. This set is also called **MIP-Tables**, e.g. [CIMP6](https://github.com/PCMDI/cmip6-cmor-tables).\n", "\n", "- One *CV*-MIP-Table contains a condensed form of all CVs which are version controlled in the registry. It contains\n", " - required and optional CMIP attributes\n", " - allowed values for attributes\n", " - restrictions resulting from a setting of attributes (e.g. min. simulation years of an experiment)\n", " - whether additional attributes must be specified (e.g. parent attributes)\n", "- All other MIP-Tables contain variable information" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```{tip}\n", "The MIP-Tables are *input* for CMOR. Therefore, it is guaranted that **CMOR output is CMIP compliant** as it also implements the **OR** specifications.\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A variable can be requested for different *frequencies, dimensions or cell_methods*. E.g., it can be reasonable to provide data on model level for reuse in ESMs while having another version of the data on pressure levels for easy analysis.\n", "\n", "MIP-tables are divided by their variables'\n", "\n", "- realm\n", "- frequencies\n", "- grid and vertical axis types\n", "- time cell method.\n", "\n", "so that a variable only occur **once** in the MIP-table. Also, this division is made to keep them short.\n", "\n", "In CMIP6, the MIP-table name is constructed by a *Prefix, Frequency, Suffix and a Qualifier*. However, neither all of these four parts need to be included in MIP-table name nor all of the possible combinations exists as a MIP-Table." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For this notebook, we working with the example on CMIP6. You can clone the MIP-Tables repository yourself or use the submodule inside the workshop material." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%bash\n", "#!git clone https://github.com/PCMDI/cmip6-cmor-tables.git {mip_tables_dir}\n", "rel_mip_tables_dir=configuration/cmip6/cmip6-cmor-tables/\n", "cd $(pwd)/cdo-incl-cmor \n", "git submodule init ${rel_mip_tables_dir} \n", "git submodule update ${rel_mip_tables_dir} \n", "cd ${rel_mip_tables_dir} \n", "git checkout --track origin/01.00.31" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can **parse** the tables with the `json` package. The *Amon* MIP-Table contains a **Header**, and **variable_entries**." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "mip_tables_dir=basedir+\"/configuration/cmip6/cmip6-cmor-tables/\"\n", "import json\n", "with open(mip_tables_dir+\"/Tables/CMIP6_Amon.json\") as f:\n", " amon=json.load(f)\n", "print(amon.keys())\n", "print(amon[\"Header\"].keys())\n", "print(amon[\"variable_entry\"].keys())" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#%%capture --no-stdout\n", "#Standardize all variables of example_interface.nc for CMIP6_Amon.json:\n", "infotabledir=\"cdo-incl-cmor/configuration/cmip6/cmip6-cdocmorinfo/\"\n", "cdocmorinfos=[infotabledir+k \n", " for k in [\"dkrz_atts\",\n", " \"historical_atts\",\n", " \"mpi-esm1-2-lr_atts\",\n", " \"cdocmorcontrol_atts\",\n", " \"member_atts\",\n", " \"nominalresolution_atts\"]\n", " ]\n", "cdocmorinfostring=','.join(cdocmorinfos)\n", "cdo.cmor(mip_tables_dir+'/Tables/CMIP6_Amon.json,'\n", " 'i='+cdocmorinfostring,\n", " input=workdir+'/example_interface.nc',\n", " options=\"-v\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### CMOR variable\n", "\n", "The entry of a variable inside a MIP-Table is called **cmor name** of the variable.\n", "\n", "A **CMOR-variable** is the unique combination of the **cmor name** and the corresponding **MIP-table** which includes the **cmor name**. \n", "\n", "The data standard of the same variable is different from one MIP-Table to another. That can include all variable information from cell methods to its grid.\n", "\n", "- In CMIP6, the request for monthly air temperature (CMIP6_Amon.json) is different compared to daily air temperature (CMIP6_day.json)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4. cdocmorinfo " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Global attributes and operator control keywords are specified in a cdocmorinfo file." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Global attributes and the CV:\n", "\n", "Project dependence of attribute nomenclature:\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Attribute\\projectCMIP6CMIP5
MIP:activity_idproject_id
Model:\n", "source_idmodel_id
Institute:institution_idinstitute_id
Ensemble member:variant_labelmember
Grid resolution:nominal_resolution
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Experiments are registered in the CV with attached predefined attributes:\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
Attributes\\experiment_id1pctCO2amipssp585
activity_idCMIPCMIPScenarioMIP
experiment1 percent per year increase in CO2AMIPupdate of RCP8.5 based on SSP5
sub_experiment_idnonenonenone
parent_activity_idCMIPno parentCMIP
parent_experiment_idpiControlno parenthistorical
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%capture --no-stdout\n", "#Since there is \n", "#1. a default for i which is '.cdocmorinfo'\n", "#2. the attribute MIP_table_dir specified in cdocmorinfo,\n", "#We only need to copy cdocmorinfo to our pwd:\n", "#!rm {pwd}/cdocmorinfo\n", "for c in cdocmorinfos:\n", " !cat {c} >>{pwd}/.cdocmorinfo\n", "#so that it is sufficient to call:\n", "cdo.cmor(mip_tables_dir+'/Tables/CMIP6_Amon.json',input=workdir+'example_interface.nc')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 5. Select subset of variables" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%capture --no-stdout\n", "#Only process variable with cmor_name=tas\n", "cdo.cmor('Amon,cmor_name=tas',input=workdir+'example_interface.nc')\n", "#Same process, but with short keyword cn:\n", "#cdo.cmor('Amon,cn=tas', input=workdir+'examples/example_interface.nc')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Variable mapping" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "How to map variables?\n", "\n", "1. Know the CMOR variable you aim to produce\n", "1. Link to the matching infile variable(s)\n", " - specify a recipe\n", "1. Provide attributes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "\n", "
KeywordShort nameValue formatDefault
cmor_namecnVariable name included in MIP-table
namenInput variable name
codecThree digits integer. GRIB code.
unitsuString. Must be readable by udunits.
cell_methodscmCharacter (see below)m
positivepu=upward, d=downward
variable_commentvcString
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%capture --no-stdout\n", "#Map Variable witch code=167 to CMOR Variable tas.\n", "# All Mapping information are infile variable descriptions.\n", "cdo.cmor('Amon,cn=tas,'\n", " 'code=167,units=K,cell_methods=m',\n", " input=workdir+'example_mapping.grb')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%capture --no-stdout\n", "#Write mapping information to mapping table:\n", "with open(workdir+'mapping_table.txt', 'a') as mapping_table:\n", " mapping_table.write('¶meter cmor_name=tas code=167 units=K cell_methods=m /\\n')\n", "mapping_table.close()\n", "#Select a specific variable in the command line to be mapped with mapping_table.txt:\n", "cdo.cmor('Amon,cn=tas',\n", " 'mapping_table='+workdir+'mapping_table.txt',\n", " input=workdir+'example_mapping.grb')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%capture --no-stdout\n", "#Process and map all variables which are in example_collect.grb with mtPERFECT.txt:\n", "cdo.cmor('Amon',\n", " 'mt='+workdir+'mtPERFECT.txt',\n", " input=workdir+'example_collect.grb')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 7. Coordinates" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%capture --no-stdout\n", "#Define value for z_axis height2m as 1.5m:\n", "with open(workdir+'.cdocmorinfo.txt', 'a') as info:\n", " info.write('height2m=1.5\\n')\n", "#\n", "cdo.cmor('Amon,cn=tas,\n", " 'z_axis=height2m',\n", " 'mapping_table=mapping_table.txt',\n", " input=workdir+'example_T_3M.nc')" ] } ], "metadata": { "kernelspec": { "display_name": "python3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.13" } }, "nbformat": 4, "nbformat_minor": 4 }