In [1]:
from IPython.display import Image
Image(url='https://vesg.ipsl.upmc.fr/thredds/fileServer/IPSLFS/jservon/CliMAF_Notebooks_html/CliMAF-logo-small.png') 
Out[1]:

A science-oriented framework to ease the analysis of climate model simulations

WP5 ANR Convergence
Development team: Stéphane Sénési (CNRM-GAME), Gaëlle Rigoudy (CNRM-GAME), Jérôme Servonnat (LSCE-IPSL), Ludivine Vignon (CNRM-GAME), Laurent Franchisteguy (CNRM-GAME), Patrick Brockmann (LSCE-IPSL)
Beta-testing: Olivier Marti (LSCE-IPSL), Marie-Pierre Moine (CERFACS), Emilia Sanchez-Gomez (CERFACS)

contact: climaf@meteo.fr
users list: climaf-users@meteo.fr

Gathering my results in an html page

Gathering your figures in an html page is very convenient to both:

  • store your results and visualize them
  • share them with collaborators

We will see in this notebook how we can:

    1. make a portable atlas = html file + figures together in a single directory => can be read on your local machine or from Jupyter
    1. use /prodigfs/ipslfs/dods to store your atlas and share only the url with your collaborators
    1. And a little bonus to see how to execute the plots in parallel

First, import climaf

In [2]:
from climaf.api import *
CliMAF version = 1.2.13
CliMAF install => /ciclad-home/jservon/Evaluation/CliMAF/climaf_installs/climaf_V1.2.13_post
python => /modfs/modtools-phw/miniconda2/envs/analyse_2.7/bin/python
---
Required softwares to run CliMAF => you are using the following versions/installations:
ncl 6.6.2 => /modfs/modtools-phw/miniconda2/envs/analyse_2.7/bin/ncl
cdo 1.9.6 => /opt/nco/1.9/bin/cdo
nco (ncks) 4.5.2 => /opt/nco-4.5.2/bin/ncks
ncdump fichier => /modfs/modtools-phw/miniconda2/envs/analyse_2.7/bin/ncdump
Check stamping requirements
nco (ncatted) found -> /opt/nco-4.5.2/bin/ncatted
convert found -> /usr/bin/convert
pdftk found -> /usr/bin/pdftk
exiv2 found -> /ciclad-home/jservon/Evaluation/CliMAF/climaf_installs/climaf_V1.2.13_post/bin/exiv2
---
Cache directory set to : /data/jservon/climafcache (use $CLIMAF_CACHE if set) 
Cache directory for remote data set to : /data/jservon/climafcache/remote_data (use $CLIMAF_REMOTE_CACHE if set) 
warning  : When defining temp_penalty : duplicate declaration for input #0
warning  : When defining cquantile : duplicate declaration for input #0
warning  : When defining cquantile : duplicate declaration for input #0
Available macros read from ~/.climaf.macros are : []

And set verbosity ('critical' -> minimum ; 'debug' -> maximum)

In [3]:
clog('critical') # min verbosity = critical < warning < info < debug = max verbosity

... and dont' forget to open the documentation in case you have questions.

http://climaf.readthedocs.org/

-> Use the "Quick search" space to search for what you are interested in, it is really powerfull!

Preliminary: we get a CMIP5 multi-model ensemble as an illustration

In [4]:
# -- We use ds() to get the dataset
req_cmip5 = ds(project='CMIP5',
               model='*',
               variable='tos',
               experiment='historical',
               period='1980',
               frequency='monthly',
               version='latest'
               )
req_cmip5.explore('choices')
Out[4]:
{'model': ['ACCESS1-0',
  'ACCESS1-3',
  'CCSM4',
  'CESM1-BGC',
  'CESM1-CAM5',
  'CESM1-CAM5-1-FV2',
  'CESM1-FASTCHEM',
  'CESM1-WACCM',
  'CMCC-CESM',
  'CMCC-CM',
  'CMCC-CMS',
  'CNRM-CM5',
  'CNRM-CM5-2',
  'CSIRO-Mk3-6-0',
  'CanCM4',
  'CanESM2',
  'EC-EARTH',
  'FGOALS-g2',
  'GFDL-CM2p1',
  'GFDL-CM3',
  'GFDL-ESM2G',
  'GFDL-ESM2M',
  'GISS-E2-H',
  'GISS-E2-H-CC',
  'GISS-E2-R',
  'GISS-E2-R-CC',
  'HadCM3',
  'HadGEM2-AO',
  'HadGEM2-CC',
  'HadGEM2-ES',
  'IPSL-CM5A-LR',
  'IPSL-CM5A-MR',
  'IPSL-CM5B-LR',
  'MIROC-ESM',
  'MIROC-ESM-CHEM',
  'MIROC5',
  'MRI-CGCM3',
  'MRI-ESM1',
  'NorESM1-M',
  'NorESM1-ME',
  'bcc-csm1-1',
  'bcc-csm1-1-m',
  'inmcm4'],
 'realm': 'ocean',
 'table': 'Omon'}
In [5]:
ens_cmip5 = req_cmip5.explore('ensemble')

summary(ens_cmip5)
Keys - values:
{'domain': 'global', 'realm': 'ocean', 'version': 'latest', 'period': 1980, 'simulation': '', 'project': 'CMIP5', 'experiment': 'historical', 'frequency': 'monthly', 'realization': 'r1i1p1', 'variable': 'tos', 'table': 'Omon', 'model': 'ACCESS1-0', 'root': '/bdd'}
-- Ensemble members:
ACCESS1-0
/bdd/CMIP5/output/CSIRO-BOM/ACCESS1-0/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_ACCESS1-0_historical_r1i1p1_185001-200512.nc
--
ACCESS1-3
/bdd/CMIP5/output/CSIRO-BOM/ACCESS1-3/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_ACCESS1-3_historical_r1i1p1_185001-200512.nc
--
CCSM4
/bdd/CMIP5/output/NCAR/CCSM4/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_CCSM4_historical_r1i1p1_185001-200512.nc
--
CESM1-BGC
/bdd/CMIP5/output/NSF-DOE-NCAR/CESM1-BGC/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_CESM1-BGC_historical_r1i1p1_185001-200512.nc
--
CESM1-CAM5
/bdd/CMIP5/output/NSF-DOE-NCAR/CESM1-CAM5/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_CESM1-CAM5_historical_r1i1p1_185001-200512.nc
--
CESM1-CAM5-1-FV2
/bdd/CMIP5/output/NSF-DOE-NCAR/CESM1-CAM5-1-FV2/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_CESM1-CAM5-1-FV2_historical_r1i1p1_185001-200512.nc
--
CESM1-FASTCHEM
/bdd/CMIP5/output/NSF-DOE-NCAR/CESM1-FASTCHEM/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_CESM1-FASTCHEM_historical_r1i1p1_185001-200512.nc
--
CESM1-WACCM
/bdd/CMIP5/output/NSF-DOE-NCAR/CESM1-WACCM/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_CESM1-WACCM_historical_r1i1p1_185001-200512.nc
--
CMCC-CESM
/bdd/CMIP5/output/CMCC/CMCC-CESM/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_CMCC-CESM_historical_r1i1p1_197601-198112.nc
--
CMCC-CM
/bdd/CMIP5/output/CMCC/CMCC-CM/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_CMCC-CM_historical_r1i1p1_198001-198912.nc
--
CMCC-CMS
/bdd/CMIP5/output/CMCC/CMCC-CMS/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_CMCC-CMS_historical_r1i1p1_198001-198912.nc
--
CNRM-CM5
/bdd/CMIP5/output/CNRM-CERFACS/CNRM-CM5/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_CNRM-CM5_historical_r1i1p1_198001-198912.nc
--
CNRM-CM5-2
/bdd/CMIP5/output/CNRM-CERFACS/CNRM-CM5-2/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_CNRM-CM5-2_historical_r1i1p1_198001-198912.nc
--
CSIRO-Mk3-6-0
/bdd/CMIP5/output/CSIRO-QCCCE/CSIRO-Mk3-6-0/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_CSIRO-Mk3-6-0_historical_r1i1p1_185001-200512.nc
--
CanCM4
/bdd/CMIP5/output/CCCma/CanCM4/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_CanCM4_historical_r1i1p1_196101-200512.nc
--
CanESM2
/bdd/CMIP5/output/CCCma/CanESM2/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_CanESM2_historical_r1i1p1_185001-200512.nc
--
EC-EARTH
/bdd/CMIP5/output/ICHEC/EC-EARTH/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_EC-EARTH_historical_r1i1p1_198001-198912.nc
--
FGOALS-g2
/bdd/CMIP5/output/LASG-CESS/FGOALS-g2/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_FGOALS-g2_historical_r1i1p1_195001-199912.nc
--
GFDL-CM2p1
/bdd/CMIP5/output/NOAA-GFDL/GFDL-CM2p1/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_GFDL-CM2p1_historical_r1i1p1_197601-198012.nc
--
GFDL-CM3
/bdd/CMIP5/output/NOAA-GFDL/GFDL-CM3/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_GFDL-CM3_historical_r1i1p1_198001-198412.nc
--
GFDL-ESM2G
/bdd/CMIP5/output/NOAA-GFDL/GFDL-ESM2G/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_GFDL-ESM2G_historical_r1i1p1_197601-198012.nc
--
GFDL-ESM2M
/bdd/CMIP5/output/NOAA-GFDL/GFDL-ESM2M/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_GFDL-ESM2M_historical_r1i1p1_197601-198012.nc
--
GISS-E2-H
/bdd/CMIP5/output/NASA-GISS/GISS-E2-H/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_GISS-E2-H_historical_r1i1p1_195101-200512.nc
--
GISS-E2-H-CC
/bdd/CMIP5/output/NASA-GISS/GISS-E2-H-CC/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_GISS-E2-H-CC_historical_r1i1p1_195101-201012.nc
--
GISS-E2-R
/bdd/CMIP5/output/NASA-GISS/GISS-E2-R/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_GISS-E2-R_historical_r1i1p1_197601-200012.nc
--
GISS-E2-R-CC
/bdd/CMIP5/output/NASA-GISS/GISS-E2-R-CC/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_GISS-E2-R-CC_historical_r1i1p1_197601-200012.nc
--
HadCM3
/bdd/CMIP5/output/MOHC/HadCM3/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_HadCM3_historical_r1i1p1_195912-198411.nc
--
HadGEM2-AO
/bdd/CMIP5/output/NIMR-KMA/HadGEM2-AO/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_HadGEM2-AO_historical_r1i1p1_186001-200512.nc
--
HadGEM2-CC
/bdd/CMIP5/output/MOHC/HadGEM2-CC/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_HadGEM2-CC_historical_r1i1p1_195912-200511.nc
--
HadGEM2-ES
/bdd/CMIP5/output/MOHC/HadGEM2-ES/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_HadGEM2-ES_historical_r1i1p1_195912-200512.nc
--
IPSL-CM5A-LR
/bdd/CMIP5/output/IPSL/IPSL-CM5A-LR/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_IPSL-CM5A-LR_historical_r1i1p1_185001-200512.nc
--
IPSL-CM5A-MR
/bdd/CMIP5/output/IPSL/IPSL-CM5A-MR/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_IPSL-CM5A-MR_historical_r1i1p1_185001-200512.nc
--
IPSL-CM5B-LR
/bdd/CMIP5/output/IPSL/IPSL-CM5B-LR/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_IPSL-CM5B-LR_historical_r1i1p1_185001-200512.nc
--
MIROC-ESM
/bdd/CMIP5/output/MIROC/MIROC-ESM/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_MIROC-ESM_historical_r1i1p1_185001-200512.nc
--
MIROC-ESM-CHEM
/bdd/CMIP5/output/MIROC/MIROC-ESM-CHEM/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_MIROC-ESM-CHEM_historical_r1i1p1_185001-200512.nc
--
MIROC5
/bdd/CMIP5/output/MIROC/MIROC5/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_MIROC5_historical_r1i1p1_185001-201212.nc
--
MRI-CGCM3
/bdd/CMIP5/output/MRI/MRI-CGCM3/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_MRI-CGCM3_historical_r1i1p1_185001-200512.nc
--
MRI-ESM1
/bdd/CMIP5/output/MRI/MRI-ESM1/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_MRI-ESM1_historical_r1i1p1_185101-200512.nc
--
NorESM1-M
/bdd/CMIP5/output/NCC/NorESM1-M/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_NorESM1-M_historical_r1i1p1_185001-200512.nc
--
NorESM1-ME
/bdd/CMIP5/output/NCC/NorESM1-ME/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_NorESM1-ME_historical_r1i1p1_185001-200512.nc
--
bcc-csm1-1
/bdd/CMIP5/output/BCC/bcc-csm1-1/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_bcc-csm1-1_historical_r1i1p1_185001-201212.nc
--
bcc-csm1-1-m
/bdd/CMIP5/output/BCC/bcc-csm1-1-m/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_bcc-csm1-1-m_historical_r1i1p1_185001-201212.nc
--
inmcm4
/bdd/CMIP5/output/INM/inmcm4/historical/mon/ocean/Omon/r1i1p1/latest/tos/tos_Omon_inmcm4_historical_r1i1p1_185001-200512.nc
--

Compute a climatology, regrid on a regular grid and select the North Atlantic region

In [6]:
# -- Climatology
clim_ens_cmip5 = clim_average(ens_cmip5, 'ANM')

# -- Regridding
rgrd_clim_ens_cmip5 = regridn(clim_ens_cmip5, cdogrid='r360x180')

# -- Select the region
NATL_cmip5 = llbox(rgrd_clim_ens_cmip5, lonmin=-90, lonmax=20, latmin=0, latmax=90)

Quick check:

In [7]:
#implot(NATL_cmip5['IPSL-CM5A-LR'], focus='ocean', color='WhViBlGrYeOrRe', offset=-273.15,
#       min=0, max=25, delta=1
#        )
pp = dict(focus='ocean', color='WhViBlGrYeOrRe', offset=-273.15,
          min=0, max=25, delta=1)
implot(NATL_cmip5['IPSL-CM5A-LR'], **pp)
Out[7]:

Ok, we have our ensemble ready.

Let's start the html page.

In [8]:
# -- Pour construire une page html:
from climaf.html import *
from climaf.cache import cachedirs
import os

# -- Directory where we create the atlas => need to be on same files
subdir = 'demo_atlas'
outdir = cachedirs[0]+'/'+subdir
index_name = 'atlas.html'

# -- Addresses a utiliser pour voir le repertoire depuis la page web
#alt_dir_name = "/thredds/fileServer/IPSLFS"+str.split(outdir,'dods')[1]
#root_url = "https://vesg.ipsl.upmc.fr"

atlas_head_title = 'Demo html atlas in CliMAF'
style_file = cpath[0]+'/cesmep_style_css'
thumbnail_size = '400*250'

if not os.path.isdir(outdir):
    os.makedirs(outdir)
else:
    os.system('rm -f '+outdir+'/*.png')
In [9]:
# -----------------------------------------------------------------------------------
# - Init html index
# -----------------------------------------------------------------------------------
index = header(atlas_head_title, style_file=style_file)

# -- Start a section
index += section('Title of the section', level=4)
index += start_line('And a title for your line of plots')

# -- On va faire les plots de tous les modeles
# -> pp = dictionaire de parametres de plot a prendre depuis la page de doc CliMAF sur plot: https://climaf.readthedocs.io/en/latest/scripts/plot.html
pp = dict(focus='ocean', color='WhViBlGrYeOrRe', offset=-273.15,
          min=0, max=25, delta=1)

safe_mode=True
for model in NATL_cmip5.order:
    print model
    my_plot = plot(NATL_cmip5[model], title=model, **pp)
    index += cell("", safe_mode_cfile_plot(my_plot, safe_mode=safe_mode),
                  thumbnail=thumbnail_size, hover=False, dirname=outdir)
    # safe_mode_cfile_plot execute le plot; si il rencontre une erreur, il ne s'arrete pas et renvoie
    # simplement une image blanche

# -- Close the section
index += close_line() + close_table()


# -- Add compare companion -> image selector (P. Brockmann)
# ----------------------------------------------------------------------------------------------
index += compareCompanion()

# -- End the index
index += trailer()
ACCESS1-0
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%ACCESS1-0%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='ACCESS1-0')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
ACCESS1-3
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%ACCESS1-3%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='ACCESS1-3')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
CCSM4
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%CCSM4%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='CCSM4')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
CESM1-BGC
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%CESM1-BGC%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='CESM1-BGC')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
CESM1-CAM5
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%CESM1-CAM5%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='CESM1-CAM5')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
CESM1-CAM5-1-FV2
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%CESM1-CAM5-1-FV2%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='CESM1-CAM5-1-FV2')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
CESM1-FASTCHEM
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%CESM1-FASTCHEM%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='CESM1-FASTCHEM')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
CESM1-WACCM
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%CESM1-WACCM%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='CESM1-WACCM')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
CMCC-CESM
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%CMCC-CESM%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='CMCC-CESM')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
CMCC-CM
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%CMCC-CM%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='CMCC-CM')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
CMCC-CMS
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%CMCC-CMS%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='CMCC-CMS')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
CNRM-CM5
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%CNRM-CM5%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='CNRM-CM5')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
CNRM-CM5-2
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%CNRM-CM5-2%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='CNRM-CM5-2')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
CSIRO-Mk3-6-0
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%CSIRO-Mk3-6-0%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='CSIRO-Mk3-6-0')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
CanCM4
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%CanCM4%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='CanCM4')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
CanESM2
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%CanESM2%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='CanESM2')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
EC-EARTH
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%EC-EARTH%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='EC-EARTH')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
FGOALS-g2
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%FGOALS-g2%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='FGOALS-g2')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
GFDL-CM2p1
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%GFDL-CM2p1%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='GFDL-CM2p1')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
GFDL-CM3
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%GFDL-CM3%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='GFDL-CM3')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
GFDL-ESM2G
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%GFDL-ESM2G%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='GFDL-ESM2G')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
GFDL-ESM2M
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%GFDL-ESM2M%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='GFDL-ESM2M')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
GISS-E2-H
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%GISS-E2-H%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='GISS-E2-H')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
GISS-E2-H-CC
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%GISS-E2-H-CC%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='GISS-E2-H-CC')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
GISS-E2-R
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%GISS-E2-R%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='GISS-E2-R')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
GISS-E2-R-CC
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%GISS-E2-R-CC%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='GISS-E2-R-CC')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
HadCM3
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%HadCM3%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='HadCM3')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
HadGEM2-AO
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%HadGEM2-AO%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='HadGEM2-AO')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
HadGEM2-CC
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%HadGEM2-CC%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='HadGEM2-CC')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
HadGEM2-ES
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%HadGEM2-ES%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='HadGEM2-ES')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
IPSL-CM5A-LR
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%IPSL-CM5A-LR%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='IPSL-CM5A-LR')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
IPSL-CM5A-MR
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%IPSL-CM5A-MR%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='IPSL-CM5A-MR')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
IPSL-CM5B-LR
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%IPSL-CM5B-LR%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='IPSL-CM5B-LR')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
MIROC-ESM
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%MIROC-ESM%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='MIROC-ESM')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
MIROC-ESM-CHEM
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%MIROC-ESM-CHEM%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='MIROC-ESM-CHEM')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
MIROC5
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%MIROC5%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='MIROC5')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
MRI-CGCM3
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%MRI-CGCM3%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='MRI-CGCM3')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
MRI-ESM1
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%MRI-ESM1%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='MRI-ESM1')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
NorESM1-M
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%NorESM1-M%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='NorESM1-M')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
NorESM1-ME
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%NorESM1-ME%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='NorESM1-ME')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
bcc-csm1-1
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%bcc-csm1-1%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='bcc-csm1-1')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
bcc-csm1-1-m
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%bcc-csm1-1-m%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='bcc-csm1-1-m')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
inmcm4
--> Successfully plotted  plot(llbox(regridn(time_average(ds('CMIP5%%tos%1980%global%/bdd%inmcm4%Omon%historical%r1i1p1%monthly%ocean%latest')),cdogrid='r360x180'),latmax=90,latmin=0,lonmax=20,lonmin=-90),color='WhViBlGrYeOrRe',delta=1,focus='ocean',max=25,min=0,offset=-273.15,title='inmcm4')
index_atlas in html.py =  /data/jservon/climafcache/demo_atlas/index_atlas
In [10]:
# -- Finalize and save atlas
# ----------------------------------------------------------------------------------------------
outfile = outdir+'/'+index_name
with open(outfile,"w") as filout : filout.write(index)

os.system('cp -r '+outdir+' '+os.getcwd())
print("index written as : "+outfile)
tmp = str.split(outdir,'/')
print "Available in current directory in: ", tmp[-1]
index written as : /data/jservon/climafcache/demo_atlas/atlas.html
Available in current directory in:  demo_atlas

2. Use /thredds/ipsl to store your atlas and share only the url with your collaborators

We build on the example above.

The second block is exactly the same.

We just point to differences with the code above.

In [11]:
# -- Dods directory (web server)
dodsdir = '/thredds/ipsl/jservon'

# -- Url to use to access the page from the web
alt_dir_name = "/thredds/fileServer/IPSLFS"+str.split(dodsdir,'thredds/ipsl')[1]+'/'+subdir
root_url = "https://vesg.ipsl.upmc.fr"
In [12]:
# -- Copy on dods... 
# ----------------------------------------------------------------------------------------------
os.system('cp -r '+outdir+' '+dodsdir)
print("index copied in : "+dodsdir+'/'+subdir)

# -- and return the url of the atlas
# ----------------------------------------------------------------------------------------------
print("Available at this address "+root_url+outfile.replace(outdir,alt_dir_name))
index copied in : /thredds/ipsl/jservon/demo_atlas
Available at this address https://vesg.ipsl.upmc.fr/thredds/fileServer/IPSLFS/jservon/demo_atlas/atlas.html

3. Execute a list of CliMAF objects in parallel

Really interesting for heavy treatments, not so much for light ones (more effective in sequential execution)

In this example, we extracted from the atlas code above only the lines that lead to the plots (not the build of the html page).

  • The principle is that we gather the CRS of all the CliMAF objects (here plots) in a list, and we provide this list to joblib.Parallel that will execute the function provided to joblib.delayed on the elements of the list in parallel. Find the documentation on joblib here: https://joblib.readthedocs.io/en/latest/ and an easy start here: https://blog.dominodatalab.com/simple-parallelization/

  • Once all the plots have been executed and stored in the cache, we need to synchronize the cache.

  • You can then run the code of the atlas (with the building of the html page) and benefit from the cache to just pick the existing results from it.

!! Need to specify a number of cpus when submitting your interactive session at CLIMERI before running jupyter

In [16]:
from joblib import Parallel, delayed
import multiprocessing

# -- We store all the CRS of all the plots in the list plots_crs
plots_crs = []
for model in NATL_cmip5.order:
    plots_crs.append( plot(NATL_cmip5[model], title=model, **pp) )

# -- Get the number of available cpus
#num_cores = multiprocessing.cpu_count()
num_cores = 31 # -- the interactive session had 32 proc,
               #    you can set the number manually to avoid poor usage of the resource

# -- Execute the plots in parallel
r = Parallel(n_jobs=num_cores)(delayed(safe_mode_cfile_plot)(climaf_crs) for climaf_crs in plots_crs)

# -- Synchronize the cache
csync(True)

# --> Now the plots are stored in the cache; simply re-run the code of your atlas
#     to assemble the html page with the pre-computed plots

This was Gathering_my_results_in_an_html_page!

(with a little bonus on a quick solution to execute your CliMAF objects in parallel)

-> go back to the other examples to see more on data access, how to work with ensembles, build an html page...