Home

Awesome

IceNet: Seasonal Arctic sea ice forecasting with probabilistic deep learning

DOI

This codebase accompanies the Nature Communications paper Seasonal Arctic sea ice forecasting with probabilistic deep learning. It includes code to fully reproduce all the results of the study from scratch. It also includes code to download the data generated by the study, published on the Polar Data Centre, and reproduce all the paper's figures.

The flexibility of the code simplifies possible extensions of the study. The data processing pipeline and custom IceNetDataLoader class lets you dictate which variables are input to the networks, which climate simulations are used for pre-training, and how far ahead to forecast. The architecture of the IceNet model can be adapted in icenet/models.py. The output variable to forecast could even be changed by refactoring the IceNetDataLoader class.

A demonstrator of this codebase (downloading pre-trained IceNet networks, then generating and analysing forecasts) produced by @acocac can be found in The Environmental Data Science Book.

The guidelines below assume you're working in the command line of a Unix-like machine with a GPU. If aiming to reproduce all the results of the study, 1 TB of space should safely cover the storage requirements from the data downloaded and generated.

If you run into issues or have suggestions for improvement, please raise an issue or email me (tomand@bas.ac.uk).

Steps to plot paper figures using the paper's results & forecasts

To reproduce the paper figures directly from the paper's results and forecasts, run the following after setting up the conda environment (see Step 1 below):

Steps to reproduce the paper's results from scratch

0) Preliminary setup

1) Set up conda environment

After cloning the repo, run the commands below in the root of the repository to set up the conda environment:

2) Download data

The CMIP6 variable naming convention is used throughout this project - e.g. tas for surface air temperature, siconca for sea ice concentration, etc.

Warning: some downloads are slow and the net download time can take 1-2 days. It may be advisable to write a bash script to automatically execute all these commands in sequence and run it over a weekend.

3) Process data

3.1) Set up IceNet's custom data loader

3.2) Preprocess the raw data

4) Train IceNet

4.1) OPTIONAL: Run the hyperparameter search (skip if using default values from paper)

4.2) Run training

5) Produce forecasts

6) Analyse forecasts

7) Run the permute-and-predict method to explore IceNet's most important input variables

8) Generate the paper figures and tables

wget -O data/sea_ice_outlook_errors.csv 'https://ramadda.data.bas.ac.uk/repository/entry/get/sea_ice_outlook_errors.csv?entryid=synth%3A71820e7d-c628-4e32-969f-464b7efb187c%3AL3Jlc3VsdHMvb3V0bG9va19lcnJvcnMvc2VhX2ljZV9vdXRsb29rX2Vycm9ycy5jc3Y%3D'

Misc

Project structure: simplified output from tree

.
├── data
│   ├── obs
│   ├── cmip6
│   │   ├── EC-Earth3
│   │   │   ├── r10i1p1f1
│   │   │   ├── r12i1p1f1
│   │   │   ├── r14i1p1f1
│   │   │   ├── r2i1p1f1
│   │   │   └── r7i1p1f1
│   │   └── MRI-ESM2-0
│   │       ├── r1i1p1f1
│   │       ├── r2i1p1f1
│   │       ├── r3i1p1f1
│   │       ├── r4i1p1f1
│   │       └── r5i1p1f1
│   ├── forecasts
│   │   ├── icenet
│   │   │   ├── 2021_06_15_1854_icenet_nature_communications
│   │   │   │   └── unet_tempscale
│   │   │   └── 2021_06_30_0954_icenet_pretrain_ablation
│   │   │       └── unet_tempscale
│   │   ├── linear_trend
│   │   └── seas5
│   │       ├── EASE
│   │       └── latlon
│   ├── masks
│   └── network_datasets
│       └── dataset1
│           ├── meta
│           ├── obs
│           ├── transfer
│           └── norm_params.json
├── dataloader_configs
│   ├── 2021_06_15_1854_icenet_nature_communications.json
│   └── 2021_06_30_0954_icenet_pretrain_ablation.json
├── figures
├── icenet
├── logs
│   ├── cmip6_download_logs
│   ├── era5_download_logs
│   ├── seas5_download_logs
│   └── wind_rotation_logs
├── results
│   ├── forecast_results
│   │   └── 2021_07_01_183913_forecast_results.csv
│   ├── permute_and_predict_results
│   │   └── permute_and_predict_results.csv
│   └── uncertainty_results
│       ├── ice_edge_region_results.csv
│       ├── sip_bounding_results.csv
│       └── uncertainty_results.csv
└── trained_networks
    └── 2021_06_15_1854_icenet_nature_communications
        ├── obs_train_val_data
        │   ├── numpy
        │   └── tfrecords
        │       ├── train
        │       └── val
        └── unet_tempscale
            └── networks
                ├── network_tempscaled_36.h5
                ├── network_tempscaled_37.h5
                :

Acknowledgements

Thanks to James Byrne (BAS) and Tony Phillips (BAS) for direct contributions to this codebase.