In [1]:
from IPython.display import Image
Image(url='https://vesg.ipsl.upmc.fr/thredds/fileServer/IPSLFS/jservon/CliMAF_Notebooks_html/CliMAF-logo-small.png')
Out[1]:
A science-oriented framework to ease the analysis of climate model simulations¶
WP5 ANR Convergence¶
Development team: Stéphane Sénési (CNRM-GAME), Gaëlle Rigoudy (CNRM-GAME), Jérôme Servonnat (LSCE-IPSL), Ludivine Vignon (CNRM-GAME), Laurent Franchisteguy (CNRM-GAME), Patrick Brockmann (LSCE-IPSL)¶
Beta-testing: Olivier Marti (LSCE-IPSL), Marie-Pierre Moine (CERFACS), Emilia Sanchez-Gomez (CERFACS)¶
¶
contact: climaf@meteo.fr¶
users list: climaf-users@meteo.f¶
Understand how the CliMAF cache works and manage the content¶
In this notebook we will see how the CliMAF cache works and how to:
- totally erase the content of the cache: craz
- CliMAF stores its results and knows that those results exist (and are linked with sequences of operations)
- remove one specific result: cdrop
- remove results using criteria: pattern, age... => crm
- VERY IMPORTANT: remove everything but a list of results you want to keep: cprotect
- csync: synchronize the index with the content of the cache
The goal of this notebook is to give you the keys to manage the content of your cache by yourself (and potentially setup automatic cleaning at the end of your scripts)¶
See the documentation on the tools to manage the cache for more details:¶
https://climaf.readthedocs.io/en/master/functions_results_cache.html
First, import climaf¶
In [2]:
from climaf.api import *
And set verbosity ('critical' -> minimum ; 'debug' -> maximum)¶
In [2]:
clog('critical') # min verbosity = critical < warning < info < debug = max verbosity
... and dont' forget to open the documentation in case you have questions.¶
http://climaf.readthedocs.org/¶
-> Use the "Quick search" space to search for what you are interested in, it is really powerfull!¶
¶
1. Start with a brand new cache: totally erase the content of the cache¶
In [3]:
# -- Check which cache your are using
from climaf.cache import cachedirs
cachedirs
Out[3]:
If you already have a cache, you can see an index file at its root path:¶
In [4]:
cachedir = cachedirs[0]
!cat $cachedir/index
clist will retrieve the list of the CRS of the existing results from the index file $CLIMAF_CACHE/index¶
In [5]:
clist()
Out[5]:
For demo purposes, we will start with a clean cache.¶
Use craz() to fully erase your cache and restart with a clean one¶
https://climaf.readthedocs.io/en/master/functions_results_cache.html?highlight=craz#craz-reset-cache
In [6]:
craz()
In [7]:
clist()
Out[7]:
As an example, we will extract a dataset from the CMIP5 archive and do some pretreatments to add results to the cache¶
In [9]:
# -- Let's get some data as a start
dat_cmip5 = ds(project='CMIP5',
model='CNRM-CM5',
variable='tos',
experiment='historical',
period='1980-2000',
frequency='monthly',
simulation='r1i1p1',
)
summary(dat_cmip5)
Out[9]:
At this stage, dat_cmip5 is only a python object. CliMAF does not load data in memory.¶
In [10]:
dat_cmip5.crs
Out[10]:
In [11]:
cfile(dat_cmip5)
Out[11]:
When using cfile on your CliMAF object, CliMAF stores the CRS of the new results in the cache in the memory of the python session¶
In [12]:
clist()
Out[12]:
If you reinvoke cfile on the same object, it will:¶
- scan this list
- see that this sequence of operations has already been executed
- and directly return the file associated with it
In [13]:
cfile(dat_cmip5)
Out[13]:
Let's add some more stuff in the cache¶
In [18]:
ref = ds(project='ref_climatos', variable='tos', product='WOA13-v2',clim_period='195501-201212')
summary(ref)
Out[18]:
In [19]:
# -- Compute a DJF climatology
djf_dat = clim_average(dat_cmip5, 'DJF')
djf_ref = clim_average(ref, 'DJF')
# -- Regrid ref and simulation on the same regular grid
rgrd_dat = regridn(djf_dat, cdogrid='r360x180')
rgrd_ref = regridn(djf_ref, cdogrid='r360x180')
# -- And compute the difference
bias = minus(rgrd_dat, rgrd_ref)
# -- Make a simple plot
myplot = plot(bias)
# -- And execute the final object
cfile(myplot)
Out[19]:
In [20]:
clist()
Out[20]:
3. Remove one specific result: cdrop¶
Let's say that we want to remove the regridded climatologies only. We use cdrop to do this:¶
In [21]:
cdrop(rgrd_dat)
cdrop(rgrd_ref)
Out[21]:
You can confirm with clist() that the files are removed:¶
In [22]:
clist()
Out[22]:
4. Remove results using conditions (pattern, age, size): crm¶
In [23]:
crm(pattern='CNRM-CM5')
Out[23]:
Now you should only have the reference and its climatology¶
In [24]:
clist()
Out[24]:
We can also use crm() to remove files on their age. For instance, let's remove all the results that haven't been touched for more than 20 days:¶
In [25]:
crm(age='+20')
(We've started with a new cache... so the demo might not be very impressive... it won't actually return anything)¶
Or remove files on their sizes; remove all the files bigger than 2M¶
In [26]:
crm(size='2M')
Out[26]:
Check the sizes of the elements in the cache with cdu()¶
In [27]:
cdu()
=> you can combine those conditions:¶
In [23]:
crm(pattern='CNRM-CM5', age='+20', size='3M')
Let's start by putting some results back in the cache:¶
In [28]:
# -- And execute the final object
cfile(myplot)
Out[28]:
In [29]:
clist()
Out[29]:
In [30]:
cprotect(djf_dat)
cprotect(djf_ref)
cprotect(myplot)
Now I just clean the cache with craz(), and it will erase everything but my protected files¶
In [31]:
craz()
In [32]:
clist()
Out[32]:
This way you only keep the results you really need, if your intermediate results take too much space or inodes!¶
If you want to delete a protected file, use cdrop()¶
In [36]:
cdrop(myplot, force=True)
Out[36]:
In [37]:
clist()
Out[37]:
6. Synchronize the file $CLIMAF_CACHE/index with the actual content of the cache: csync(True)¶
CliMAF reads the content of the cache at the beginning of the session, and then stores the CRS of the new results in memory (and not in the index file during the session)¶
This can have drawbacks when:¶
- you have multiple processes working on the same cache => typically when you use parallel processes
- some results of the cache have been manually removed ### You then need to synchronize your index file with the actual content of the cache. To do this, we use csync():
In [33]:
# -- Check the content of the index file:
!cat $cachedir/index
In [34]:
# -- Synchronize
csync(True)
In [35]:
# -- And check the index file after
!cat $cachedir/index
In [ ]: