- https://climateintelligence.eu/
- https://clint.dkrz.de/
- https://climateintelligence.github.io/smartduck-docs/sections/intro.html
Gridded observational climate datasets such as HadCRUT5 often contain many missing values. To tackle this problem, it is common to fill in the gaps by using the state-of-the-art kriging interpolation or the principal component analysis-based infilling. Unfortunately, these techniques are limited by the statistical information contained in the original dataset and might not be able to efficiently reconstruct large regions of missing values and complex climatic patterns. Artificial Intelligence and more specifically deep-learning based technologies give the possibility to go beyond these techniques by learning the underlying physical patterns in the data.
The method presented here is a deep-learning inpainting technique based on a U-Net architecture. It is making use of partial convolutional layers and a loss function designed to produce semantically meaningful predictions. Models are trained using climate data from reanalysis and/or historical simulations and can be used to reconstruct missing information in observational datasets with minimal computational resources.
CRAI is a state-of-the-art deep learning based inpainting technology to infill missing values in climate datasets.
The infilling algorithm is trained with complete climate datasets and masks defining realistic regions of missing observational data.
As shown in the figure, the neural network has a U-net architecture and makes use of partial convolutional layers instead of standard convolution layers. A partial convolutional layer is a combination of two successive operations:
-
a partial convolution: $x'=\left{\begin{array}{ll}\boldsymbol{W}^T\left(\boldsymbol{X}\odot\boldsymbol{M}\right) \frac{\text{sum}(\boldsymbol{1})}{\text{sum}(\boldsymbol{M})} +b, & \text{if sum}(\boldsymbol{M})>0 \ 0, &\text{otherwise}\end{array}\right.$
-
a mask update: $m'=\left{\begin{array}{ll}1, & \text{if sum}(\boldsymbol{M})>0 \ 0, &\text{otherwise}\end{array}\right.$
where
Compared to other inpainting methods 1, the partial convolution technique offers several advantages:
- it operates robustly on irregular holes
- it can infill large holes
- it produces semantically meaningful predictions
CRAI is written in Python and based on the PyTorch framework and the NVIDIA CUDA® Deep Neural Network library (cuDNN). It is licensed under the terms of the BSD 3-Clause.
Step 1: clone the branch of the repository:
git clone -b clint --single-branch https://github.com/FREVA-CLINT/climatereconstructionAI.gitStep 2: create en environment with all the required dependencies using Anaconda
conda env create -f environment.ymlStep3: activate the environment
conda activate craiStep4: install the Python package using pip
pip install .The software can be used to:
- train a model (training)
- infill climate datasets using a trained model (evaluation)
It can be used as:
- a command line interface (CLI):
- training:
crai-train
- evaluation:
crai-evaluate
- a Python library:
- training:
from climatereconstructionai import train train()
- evaluation:
from climatereconstructionai import evaluate evaluate()
For more details about the execution and the input arguments, please have a look to the documentation and demo examples hosted in the repository