What’s new¶
Changes, newest first:
V3.0:
- Compatibility break:
- Python2 compatibility is dropped in this version.
- climaf module html has been renamed chtml to avoid conflicts with python standard package html
- because CliMAF now uses xarray as sole netcdf package, users on shared machine must export OPENBLAS_NUM_THREADS=1 or another sensible value
- CliMAF now runs on new IPSL clusters ‘spirit1’ and “spirit2’ and at TGCC on machine ‘irene’ and ‘irene-amd’ (including tests suite), all CliMAF examples have been adapted; also, the default format set for CDO outputs is now NetDCF4.
- function
fds()
now sets the dataset’s frequency using file’s global attribute ‘frequency’ - dataset’s method
check()
has been reworked, accepts flags for setting what to check, and provides a consistent return value (which can be None if a decision cannot be made) - function
climaf.class.ds()
can resolve period=* if it is the only wildcard; triggered by setting classes.auto_resolve to True - default is now to deactivate optimization for CMIP6 data indexing, which seems buggy
- msftyz data for IPSLCM6 and CMIP6 are automatically fixed for their degenerated dimension, provided environment variable CLIMAF_FIX_IPSL_CMIP6_MSFTYZ is set (to any value)
- when dealing with netcdf files without time bounds, it must be specified when the project is defined using option ‘use_frequency=True’. ‘frequency’ must also be a facet of the project and it must be initialized using cdef. ‘frequency’ currently support the following values: ‘mon’ (month), ‘MS’ (month), ‘D’ (day), ‘H’ (hour), ‘M’ (minute).
- A project can now be derived from an existing one (facets, dataloc).
- Bugs :
- operators regrid and regridn take care of discarding any ancillary variable in input data (such as area with sos), in order that CDO succeeds in regriding
- fix on processing period=last_XX for a CMIP5 case
- fix bugs in find_files (for case period = fx), in clist
- Technical :
- clean-up of netcdf libraries use : xarray becomes the sole package used; module anynetcdf is discarded and only functions provided by module nectdfbasics are used throughout CliMAF code, except for a very few calls to xarray (in driver.cread and classes.cdatset.check)
- remove dependency to module ‘cesemp_env’ in bin/climaf
- ease installation by embarking epstopdf
- Engine mcdo.py now applies first the selection on date, in order to take advantage of much improved CDO/NetCDF performance on this operation
- Compatibility break:
V2.0.2:
CliMAF can call ESMValTool diagnostic scripts, feeding them with data accessed and pre-processed by CliMAF calls. See Calling ESMValTools diagnostic scripts
The computation cost for all objects is now handled, and can be queried, using function
ccost()
. Cost is expressed as elapsed time. For getting the cost of a scalar, callcvalue()
with argument``cost=True``log:
- CliMAF check for external softwares version at init stage can be
silent, using
clog()
, with a log level of ‘critical’- Other messages at init stage can be discarded by redirecting stderr
cache:
- Two bugs related to cache were fixed : cache content was not updated on exit, and with Python 3, it was not read at startup. Errors related to reading / writing cache are no more hidden.
- The following cache variables are now handled in env.environment module: currentCache, cachedirs, indexFileName
Fix issue 193<https://github.com/rigoudyg/climaf/issues/193> regarding behaviour of listfiles for ambiguous cases
Finding CMIP6 datasets is optimized when facets/attributes have shell-like wildcards ( * or ?):
- This relies on building tables that allows to derive wildcard facet’s list of values by using other facet’s value. Building the table may take a while, but the table is saved for further sessions (in the cache).
- Also, an incremental testing of all possible data paths (built with possible facet values) allows to avoid costly glob.glob operations.
- As an example, searching CMIP6 ‘tos’ data for expermient historical and all models, realizations, and grids (and not for peridos) takes 77 seconds on Ciclad; searching also for all periods takes 20 minutes, instead of 37 minutes.
- This is yet limited to CMIP6 data, and active unless you set env.optimize_cmip6_wildcards to False
- See module documentation for
optimize
New dataset’s method glob is proposed as a replacement for method explore. It may be of simpler use, as it can return a list of all facets value combinations that match data. See
glob()
.
V2.0.1:
- Scripts can now process multiple ensembles, and ensembles which are not the first argument:
- Each ensemble argument will be replaced by the list of files corresponding to the ensemble,
each list being ordered by the ensemble order (see
cens()
); unless intentionally, the order is the same among all ensembles (if they are indexed by the same list of member labels)
- Each ensemble argument will be replaced by the list of files corresponding to the ensemble,
each list being ordered by the ensemble order (see
- Scripts can now process multiple ensembles, and ensembles which are not the first argument:
V2.0.0:
Python 3 compatibility:
- This new version is fully compatible with both python2 and python3. It has been tested with python 2.7 and python 3.7. Tests with other versions will come soon.
Scalar values computed using
cvalue()
are now cached; users can reset the corresponding in-memory and disk caches (both together) usingraz_cvalues()
. Variablecache.handle_cvalues
allows (when set to False) to de-activate this cache.Note
function
cvalue()
now returns a Python float (instead of a numpy.float64)- Internals:
- this uses an in-memory dict (
cvalues
) and a Json file (cvalues.json); - by default, the dict key is the hashed CRS of cvalue’s object argument (with cvalue’s index argument as suffix)
- variable
handle_cvalues
can be set to - False, for de-activating the cache - ‘by_crs’, for using the objects CRS has dict key value (but some CRS are very long)
- this uses an in-memory dict (
- Internals:
Project definition:
- Data location can now depends on table, realm in addition to experiment, project… via
dataloc()
- Data location can now depends on table, realm in addition to experiment, project… via
Operator:
- Standard operator
slice
has been renamed intocslice_average
andcslice_select
has been created. - New operators scripts/ccdo2_flip and ccdo3_flip allow CliMAF to keep track of the variable available as output of those CDO operators which use an ancilary field as first argument (as e.g. ‘ifthen’ and ‘ifthenelse’).
- Standard operator
Functions:
- Function
merge_periods()
will merge consecutive periods even if they miss a 31st december (such as in 360-days calendars)
- Function
Structure:
- To avoid cycling import, the module
climaf.operator
has been split into three modules:climaf.operator
(still here),climaf.operator_derive
andclimaf.operator_scripts
. ceval()
has been modified to call functions specific to CliMAF types:ceval_for_cdataset()
,ceval_for_ctree()
,ceval_for_ScriptChild()
,ceval_for_cpage()
,ceval_for_cpage_pdf()
,ceval_for_cens()
,ceval_for_string()
.- Variables used everywhere in CliMAF and available for everyone have been moved to module
env.environment
:cprojects
,aliases
,frequencies
,realms
,cscripts
,operators
,derived_variables
,cmacros
,known_formats
,graphic_formats
,none_formats
,locs
.
- To avoid cycling import, the module
V1.2.13:
Structure:
- Change in CliMAf structure, creation of directory env which contains site_settings.py (former climaf/site_settings.py) and clogging.py (former climaf.clogging.py). This allows the use of the logger and the environement variables which are used to determine on which server CliMAF runs in scripts.
- Rewrite mcdo.sh into mcdo.py to allow more flexibility and improve ccdo calls.
Re-work the handling of data selection for better performance:
allow to save heavy dataset selection : selection on time, variable, domain, … can be done once for all and will now be duly reused if CliMAF is forced to cache its value, simply by using :
>>> cfile(ds(...some dataset...))
A new feature was introduced to automatically avoid redundant selection on variable/period/domain … for datasets which already underwent such a selection. This implies that the operators which actually need to know the name of the variable(s) for the dataset(s) provided by CliMAF must now use keyword ‘Var’ instead of ‘var’ in the script calling sequence declaration (see
cscript
)allow to systematically avoid automatic CliMAF selection on variable, time, domain, aliasing for a given operator; when applicable, this may save disk space; see argument select in doc of
cscript
and the exampleHow to handle CliMAF operators that concatenate data over time
Cache:
- cache structure is changed for saving i-nodes (files) (divide by 10 …); please run script CLIMAF/scripts/reshape_cache.sh for reshaping your cache, after you definitely moved to this CliMAF version; this is not mandatory, but will actually lower your inodes/files use
- Cache speed improvement: when computing a CliMAF object, default behaviour is to search cache for ‘including’ or ‘begin’ objects (i.e. similar objects with different period) but this could be expensive. Toggle driver.dig_hard_into_cache can be set to False to avoid it.
- Variable climaf.cache.stamping can be set to None, which means : put a stamp if possible, but don’t bother if impossible. Reminder : the stamp is a NetCDF (or PNG, or PDF) metadata which includes the Climaf Reference Synatx description of the data
Graphics:
- operator curves also uses dash patterns when showing more than 10 curves
cpage
has two additional arguments : insert for the filename of an image to insert in the page, centered at the bottom, and insert_width for tuning its size- operator plot can superimpose a second overlay field, as for e.g. stippling + hatching for AR6 figures. See shade2 in plot: map, cross-section and profile plot of one or two fields, and vectors plot over a map.
- fixes for operator plot : it actually uses user-provided max and min for scaling field s order of magnitude. If using argument colors , min and max will be derived from extreme values. This is useful when field has very large values, largely beyond limits of explicit levels
- operator plot can superimpose a second overlay field, as for e.g. stippling + hatching for AR6 figures. See shade2 in plot: map, cross-section and profile plot of one or two fields, and vectors plot over a map.
Operators :
- handle operators which concatenate data over time, through new
cscript
argument doCatTime. See its doc and the detailed exampleHow to handle CliMAF operators that concatenate data over time
- standard operator ccdo3 allows to use e.g. CDO ternary operators such as ifthenelse
- for climaf operators needing multiple optional input objects, providing a void object is possible using an empty string (useful when wanting to provide another, which comes after in the argument objects list)
- handle operators which concatenate data over time, through new
Datasets and projects:
CliMAF startup can be quicker if you don’t need that it checks all external tools it uses; this is activated by setting enviornment variable CLIMAF_CHECK_DEPENDENCIES to ‘no’ or ‘0’
Variable climaf.cache.stamping can be set to None, which means : put a stamp if possible, but don’t bother if impossible. Reminder : the stamp is a NetCDF (or PNG, or PDF) metadata which includes the CliMAF Reference Syntax description of the data
calias()
has new argconditions
which allows to restrict its effect, based on the value of some facets, through a dictionary of criteria. Example, for a given model which CMIP6 data has an error for variableevspsbl
on some data versions :>>> calias('CMIP6,'evspsbl,scale=-1,conditions={ "model":"CanESM5" , "version": ["20180103", "20190112"] })
Add function
cvalid()
for declaring a list of allowed values for project facets/keywords. This allows to better constrain the identification of files for a dataset, as e.g. for CMIP6 when using wildcard such as grid=”g*”. It avoids mismatch between patterns for fixed fields and pattenrs for variable fields. Example :>>> cvalid('grid', ["gr", "gn", "gr1", "gr2"], project="CMIP6")
Projects CMIP5 and CMIP6 are defined even on systems where there is no known root location for that data; so, user can define facet ‘root’ later on, to match their data architecture, without hacking the distributed code
Handling attribute ‘version’ for derived variables : if a derived variable (say ‘P-E’) is defined with e.g. version ‘v20190801’, this value will apply to selecting the dataset for variable ‘P’, but value ‘latest’ will be used for ‘E’; this because there are some occasions where the value of attribute ‘version’ is not the same among the variables
Remove a constraining, buggy check on ensemble members order
Change log level of message about how DJF clim works
Fixes:
- Bugfix for cache searching of a subperiod of a derived variable already in the cache.
V1.2.12:
- the version of the tools used is now displayed when CliMAF is loaded.
- add several features to run CliMAF with C-ESM-EP at IPSL, Cerfacs and CNRM
- fixes:
- fix order in
fline
function - fix
netcdftime
import - fix fixed fields issue
- fix documentation and change sphinx version
- fix order in
V1.2.11:
- This is the last release which is truly handled on the ‘senesis/climaf’ Github account repository; next releases
will be handled on the ‘rigoudyg/climaf’ account; anyway Github will redirect your
git clone
commands to the new location - add an example for declaring
a script which has multiple output files
- dataset’s method
explore()
is improved:- option
resolve
handle variable aliasing correctly - option
resolve
will instantiate periodslike last_YYY
andfirst_YYY
- option
ensemble
allow for single-member ensembles
- option
- function
dataloc()
can use keyword${period}
in filename patterns - fixes:
- for project CMIP5, add a pattern matching fixed fields
- fix a bug and clean code in
init_period()
- improve doc (typos, english), and testing/test_1.py
- This is the last release which is truly handled on the ‘senesis/climaf’ Github account repository; next releases
will be handled on the ‘rigoudyg/climaf’ account; anyway Github will redirect your
V1.2.10:
- Ensembles are sorted on their label using module natsort (when it is available)
- Init variable site_settings.atCerfacs based on existence of /scratch/globc
- Fix for data files without a date in filename, which were sometimes disregarded
V1.2.9:
- ensembles can be built on multiple attributes (e.g model+realization);
eds()
has new argument ‘first’ for driving which attribute appears first in member labels - period=’first_10y’ and variants are allowed.
- when declaring a project, allowed pattern for dates in filenames is now ${PERIOD}
(see
dataloc
); but anyway, upward compatibility is ensured. - cache results can be protected against deletion by normal calls to
craz()
andcdrop()
; seecprotect()
; erase functions have new argument ‘force’ to overcome this protection - dataset now have a (prototype) method ‘errata()’, which opens a Firefox tab for CMIP6 datasets, querying the ES-Doc errata service. However, this service doesn’t yet support value ‘latest’ for attribute ‘version’
- fix issue with filenames which includes a plus (+) sign
- ensembles can be built on multiple attributes (e.g model+realization);
V1.2.7:
- period=’last_50y’ (and variants, e.g. LAST_3Y) are allowed
- dataset methods listfiles and baseFiles will trigger an error if the dataset is ambiguous on some attribute; but calling it with ‘ensure_dataset=False’ allows to bypass this error
- this point breaks upward comaptibility: project CMIP5 now uses attribute ‘realization’ instead of ‘member’; and now, it has organization ‘generic’
- the cache index, when synchronized, will know about dropped objects
- add module easyCMIP, yet un-documented
V1.2.6:
- a versatile datafiles and periods exploration method is associated to datasets: see
explore()
- a versatile datafiles and periods exploration method is associated to datasets: see
V1.2.2:
- Introduce doc for project CMIP6:
cmip6
and exampledata_cmip6drs.py
- Introduce doc for project CMIP6:
V1.2:
- New operators:
ensemble_ts_plot
(ensemble_ts_plot : plot time series), with python-user-friendly shortcutts_plot
(ts_plot()
): an alternative tocurves
(curves : plot multiple profiles (along p, lat, lon, time, …)) for time series, with more possibilities for customizationcLinearRegression
(cLinearRegression : performs a linear regression between X and Y): computes linear regressions between two lon/lat/time datasets (same size) or between a time series and a lon/lat/timecurl_tau_atm
(curl_tau_atm : computes wind stress curl on regular grid (usually atmosphere)): to compute the wind stress curl on regular grids (typically atmospheric grids).
lonlatvert_interpolation
(lonlatvert_interpolation()
) replaceslonlatpres_interpolation
to make it more generic and suitable for CliMAF- Added
crealms()
to have a mechanism of alias on the realms names (as for the frequencies) - when creating a portable atlas (directory containing the html index with the figures, using the ‘dirname’ option of cell()), we now have an index file in the directory listing all the figures of the atlas with their CRS. This allows any tool to parse this index and filter the figures with keywords (in the CRS).
- Added
cscalar()
: this function returns a float in python when applied on a CliMAF object that is supposed to have one single value. It uses cMA to import a masked array and returns only the float. - Allow to choose log directory (for files climaf.log and last.out), using environment variable CLIMAF_LOG_DIR
- Bug fix on ds() for the access to daily datasets with the CMIP5 project
- New operators:
2017/05/02:
- Change ncl default for operator
curves
(see curves : plot multiple profiles (along p, lat, lon, time, …)): plot axis range is now between minimum and maximum of all fields (instead of first member of ensemble)
- Change ncl default for operator
2017/04/18:
- Transparent ftp access to remote data is now possible.
See toward the end of entry
dataloc
to know how to describe a project for remote data. A local directory holds a copy of remote data. This directory is set using environment variable ‘CLIMAF_REMOTE_CACHE’ (defaults to $CLIMAF_CACHE/remote_data), see Using CliMAF at CNRM, on Ciclad or Climserv: fast track.
- Transparent ftp access to remote data is now possible.
See toward the end of entry
2017/02/21:
- Fixes a bug about a rarely used case (operator secondary outputs)
2017/01/25:
- New arguments for standard operator
plot
(see plot: map, cross-section and profile plot of one or two fields, and vectors plot over a map):xpolyline
,ypolyline
andpolyline_options
for adding a polyline to the plot - The scripts output files, which have temporary unique names until they are fully tagged with their CRS and moved to the cache, are now located in cache (instead of /tmp)
- New arguments for standard operator
2016/12/14:
- Update paths for CNRM (due to restructuring Lustre file system)
- Significant improvement of cache performance (70 times faster for a cache containing more than 8000 objects).
You must reset your cache for getting the improvement. You may use ‘rm -fR $CLIMAF_CACHE’ or function
craz()
- Fix issue occurring in parallel runs (especially for PDF outputs):
- the scripts output files now have temporary unique names until they are fully tagged with their CRS and moved to the cache
- a new shell script is available to clean corrupted PDF files in cache - i.e. without CRS (see
clean_pdf.sh
)
- New standard operator
ccdfmean_profile_box
to compute the vertical profile of horizontal means for 3D fields on a given geographical domain (see ccdfmean_profile_box : computes the vertical profile of horizontal means for 3D fields on a given geographical domain) - New method of
cdataset
class:check()
checks time consistency of first variable of a dataset or ensemble members - Bug fixes for operator
curves
(see curves : plot multiple profiles (along p, lat, lon, time, …)) when time data conversion is necessary - New standard operator
hovm
for plotting Hovmöller diagrams (time/lat or lon, or time/level) for any SST/climate boxes and provides many customization parameters; see hovm: plot Hovmöller diagrams on a given or global domain (SST/climate boxes, … ) - Function
hovm_params()
provides domain for some SST/climate boxes - Changes for default argument
title
: if no title value is provided when invoking graphic operators, no title will be displayed (formerly, the CRS expression for an object was provided as a default value fortitle
) - Bug fixes in test install
- Bug fixes for
plot
(see plot: map, cross-section and profile plot of one or two fields, and vectors plot over a map) when using argument ‘proj’ with an empty string
2016/06/30:
- Input for function
fixed_fields()
, which allows to provide fixed fields to operators: path of fixed fields may depend now also on grid of operator’s first operand - automatic fix of Aladin outputs attribute ‘coordinates’ issues, provided you set environment variable
CLIMAF_FIX_ALADIN_COORD to anything but ‘no’. This adresses the wrong variable attribute ‘coordinates’ with
‘lat lon’ instead of ‘latitude longitude’ (
../scripts/mcdo.sh
, see function aladin_coordfix()) - exiv2 (Image metadata manipulation tool) is embarked in CliMAF distribution:
<your_climaf_installation_dir>/bin/exiv2
- Input for function
2016/05/24:
- Change default for arguments
scale_aux
andoffset_aux
for standard operatorsplot
(see plot: map, cross-section and profile plot of one or two fields, and vectors plot over a map) andcurves
(see curves : plot multiple profiles (along p, lat, lon, time, …)): no scaling instead of main field scaling - Changes for standard operator
plot
(see plot: map, cross-section and profile plot of one or two fields, and vectors plot over a map):- add argument
date
for selecting date in the format ‘YYYY’, ‘YYYYMM’, ‘YYYYMMDD’ or ‘YYYYMMDDHH’ time
,date
andlevel
extractions apply on all fields now from 2D to 4D, instead of only 3D and 4D- log messages, when a time or level extraction is made, are also performed
- Bug fixes when using
mpCenterLonF
argument
- add argument
- Change default for arguments
2016/05/04 - Version 1.0.1:
- html package:
- Change interface for function
line()
: now use a list of pairs (label,figure_filename) as first arg - add function
link_on_its_own_line()
- Change interface for function
- html package:
2016/04/22 - Version 1.0:
Ensembles are now handled as dictionnaries. This breaks upward compatibility. This allows to add and pop members easily. The members can be ordered. See
cens()
Function
plot_params()
provides plot parameters (colormap, values range, …) for a number of known geophysical variables, and allows its customization. The expectation is that everybody will contribute values that can be shared, for improving easy common intepretation of evaluation plotsNew standard operators:
ml2pl
to interpolate a 3D variable on a model levels to pressure levels; works only if binary ml2pl is in your PATHccdo2
andccdo_ens
coming in addition to the very usefulccdo
swiss knife;ccdo2
takes two datasets as input, andccdo_ens
takes an ensemble of CliMAF datasets (built witheds
orcens
).Warning
ccdo_ens
is not yet optimized for large datasets which data for a single member are spread over numerous filesregridll
for regridding to a lat-lon box (see regridll : regrid an object to a lat-lon box of a CDO regular grid)
A whole new set of functions, that are mainly ‘science oriented’ shortcuts for specific use of CliMAF operators:
fadd
,fsub
,fmul
andfdiv
(now providing the 4 arithmetic operations). Work between two CliMAF objects of same size, or between a CliMAF object and a constant (provided as string, float or integer)fadd()
fsub()
fmul()
fdiv()
apply_scale_offset
to apply a scale and offset to a CliMAF objectapply_scale_offset()
diff_regrid
anddiff_regridn
-> returns the difference between two CliMAF datasets after regridding ( based on regrid : regrid an object to the grid of another one and regridn : regrid an object to a named grid )diff_regrid()
diff_regridn()
clim_average
provides a simple way to compute climatological averages (annual mean, seasonal averages, one-month climatology…)clim_average()
annual_cycle
returns the 12-month climatological annual cycle of a CliMAF objectannual_cycle()
zonmean
,diff_zonmean
andzonmean_interpolation
to work on zonal mean fieldszonmean()
diff_zonmean()
zonmean_interpolation()
,
Two functions to display a plot in an IPython Notebook:
iplot
andimplot
iplot()
implot()
Functions for an interactive use of ds() and projects:
summary
lists the files linked with a ds() request, and the pairs facet/values actually used by ds()summary()
projects
returns the listing of the available projects with the associated facets (fancy version of cprojects)projects()
New Drakkar CDFTools operators interfaced (see example
cdftools.py
):Modification for example
atlasoce.py
because CDFTools were modifiedNew function
cerr()
displays file ‘last.out’ (stdout and stderr of script call)New arguments for standard operators
plot
(see plot: map, cross-section and profile plot of one or two fields, and vectors plot over a map) andcurves
(see curves : plot multiple profiles (along p, lat, lon, time, …)):scale_aux
andoffset_aux
to scale the input auxiliary field forplot
and to scale of the second to the nth input auxiliary field forcurves
.Changes for standard operator
plot
(see plot: map, cross-section and profile plot of one or two fields, and vectors plot over a map):- Tick marks are smartly adapted to the time period duration for (t,z) profiles
- new arg
fmt
to change time axis labels format - new arg
color
to define your own color map using named colors - you can now use argument
invXY
for cross-section - Add possibility to turn OFF the data re-projection when model is already on a known native grid (currently
- Lambert only) (see relevant § of the doc)
- Bug fixes:
- for argument
reverse
- when reading latitude and longitude in file ‘coordinates.nc’ for curvilinear grid;
- for y axis style when
invXY
is used for (t,z) profiles
- for argument
Change for standard operator
slice
: extract a slice on specified dimension now at a given range instead of a given value before (see slice : extract a slice at a given dimension’s range)Technical:
- it is possible to discard stamping of files in cache (see cache.stamping)
- disambiguating filenames in cache relies only on their length (60)
- scripts execution duration is now only logged, at level ‘info’
- critical errors now exit
- fix in mcdo.sh:nemo_timefix
- project ‘em’ is based on generic organization
- re-design code of gplot.ncl
2016/03/25:
- Changes for standard operator
plot
(see plot: map, cross-section and profile plot of one or two fields, and vectors plot over a map):- new argument
reverse
to reverse colormap; - a change breaking backward compatibility: optional argument
linp
was renamedy
and its default was modified (now default is a vertical axis with data-linear spacing, so you have to specify y=”log” to obtain the same plot make without argument linp before); min
andmax
was extended to define the range of main field axis for profiles;- this operator can now plot (t,z) profiles;
- bug fixes if data file only contains latitude or longitude;
- bug fixes to custom color of auxiliary field for profiles via argument
aux_options
- new argument
- Changes for standard operator
curves
(see curves : plot multiple profiles (along p, lat, lon, time, …)):- new arguments:
aux_options
for setting NCL graphic resources directly for auxiliary field (it is recommended to use this argument only if you plot exactly two fields);min
andmax
to define min and max values for main field axis
- a change breaking backward compatibility: optional argument
linp
was renamedy
, a new axis style is proposed (data-linear spacing) and its default was modified (now default is a vertical axis with data-linear spacing, so you have to specify y=”log” to obtain the same plot make without argument linp before); - add field unit after ‘long_name’ attribute of field in title of field axis
- new arguments:
- New standard operators
slice
,mask
andncpdq
: see slice : extract a slice at a given dimension’s range, mask : set a given value as missing value and ncpdq: netCDF Permute Dimensions Quickly - A new example in the distribution: see
atlasoce.py
- File ‘angle_EM.nc’ in ‘tools’ directory was renamed
angle_data_CNRM.nc
to be compatible with the new project ‘data_CNRM’ - Adapt to Ciclad new location for CMIP5 data, and improve install doc for Ciclad
- Changes for standard operator
2016/02/25:
- Changes for standard operator
plot
(see plot: map, cross-section and profile plot of one or two fields, and vectors plot over a map):- new arguments:
shade_below
andshade_above
to shade contour regions for auxiliary field;options
,aux_options
andshading_options
for setting NCL graphic resources directly
- color filling is smoothed to contours
- new arguments:
- Standard operator ‘curves’ now handle multiple profile cases: time series, profile along lat or lon, and profile in pressure/z_index. It also allows to set NCL graphic ressources directly: see curves : plot multiple profiles (along p, lat, lon, time, …).
- Standard operators ‘lines’ and ‘timeplot’ were removed, and replaced by ‘curves’: see curves : plot multiple profiles (along p, lat, lon, time, …)
- New function
cpage_pdf()
allows to create a PDF page of figures array using ‘pdfjam’. See examplefigarray
. - A new output format allowed for graphic operators : eps; see
cscript()
. This needs an install of ‘exiv2’ - see Requirements and acknowledgements - A new standard operator, to crop eps figures to their minimal size:
cepscrop
; see cepscrop : crop eps figures to their minimal size - Changes for several functions of package
climaf.chtml
(which easily creates an html index which includes tables of links -or thumbnails- to image files). Seelink()
,cell()
,line()
,fline()
,flines()
:- new arguments:
dirname
to create a directory wich contains hard links to the figure files; allows to create an autonomous, portable atlashover
for displaying a larger image when you mouse over the thumbnail image
- change for
thumbnail
argument: it can also provide the geometry of thumbnails as ‘witdh*height’
- new arguments:
- Technical changes:
- For function
cpage_pdf()
(which creates a PDF page of figures array using ‘pdfjam’): you can set or not a backslash before optional argument ‘pt’ (for title font size) as LaTeX commands. See examplefigarray
. - Data access was modified for several examples:
- For
cdftools
,cdftools_multivar
andcdftransport
: a new project ‘data_CNRM’ was declared instead of ‘NEMO’ old project; this new project uses data available at CNRM in a dedicated directory “/cnrm/est/COMMON/climaf/test_data”, which contains both Nemo raw outputs, monitoring outputs (with VT-files) and fixed fields. - Example
gplot
: now works with project ‘example’ (instead of ‘EM’ project) and also with the new project ‘data_CNRM’ at CNRM for rotating vectors from model grid on geographic grid.
- For
- Two examples
gplot
andcdftools_multivar
were added to the script which tests all examplestest_examples
- cpdfcrop, which is used by operators ‘cpdfcrop’ and ‘cepscrop’ tools, is embarked in CliMAF distribution:
<your_climaf_installation_dir>/bin/pdfcrop
- Python 2.7 is required and tested in
test_install
- Bug fixes in
anynetcdf
to import a module from ‘scipy.io.netcdf’ library (for reading and writing NetCDF files). - Change format for log messages. For restoring former, verbose format see Experts’ corner.
cshow()
, when it displays pdf or eps figures, does use a multi-page capable viewer (xdg-open) if it is available. Otherwise, it uses ‘display’
- For function
- Changes for standard operator
2015/12/08:
- Allow operator plot to use a local coordinates file, for dealing with Nemo data files having un-complete ‘nav_lat’ and ‘nav_lon’. See navlat issues with plot. Such files are available e.g. at CNRM in /cnrm/ioga/Users/chevallier/chevalli/Partage/NEMO/
- Change for
cpage()
:- argument
orientation
is now deprecated and preferably replaced by new argumentspage_width
andpage_height
for better control on image resolution - better adjustment of figures in height (if
fig_trim
is True).
- argument
- Fix function cfile() for case hard=True
- 2015/11/27 - Version 0.12:
- Changes for standard operator
plot
(see plot: map, cross-section and profile plot of one or two fields, and vectors plot over a map):- new arguments:
level
andtime
for selecting time or level;resolution
for controling image resolutionformat
: graphical format: either png (default) or pdf- 17 new optional arguments to adjust title, sub-title, color bar, label font, label font height, … (see More plot optional arguments )
trim
to turn on triming for PNG figures- optional argument
levels
was renamedcolors
- code re-design
- if running on Ciclad, you must load NCL Version 6.3.0; see Configuring CliMAF
- new arguments:
- New arguments for
cpage()
:title
. See examplefigarray
format
: graphical output format : either png (default) or pdf
- Two new output formats allowed for operators: ‘graph’ and ‘text’; see
cscript()
- ‘graph’ allows the user to choose between two graphic output formats: ‘png’ and ‘pdf’ (new graphic ouput format), if the corresponding operator supports it (this is the case for plot());
- ‘txt’ allows to use any operator that just ouputs text (e.g. ‘ncdump -h’). The text output is not managed by CliMAF (but only displayed).
- Two new standard operators:
ncdump
: show only the header information of a netCDF file; see ncdump : show only the header information of a netCDF filecpdfcrop
: crop pdf figures to their minimal size, preserving metadata; see cpdfcrop : crop pdf figures to their minimal size
- An operator for temporary use:
curves
(see curves : plot multiple profiles (along p, lat, lon, time, …)):
- Changes for standard operator
- 2015/10/19 - Version 0.11:
- For
cpage()
(which creates an array of figures), default keywords changed: fig_trim=False -> fig_trim=True, page_trim=False -> page_trim=True. See examplefigarray
. - New function
efile()
allows to applycfile()
to an ensemble object. It writes a single file with variable names suffixed by member label. - The general purpose plot operator (for plotting 1D and 2D datasets: maps, cross-sections and profiles), named
plot
, was significantly enriched. It now allows for plotting an additional scalar field displayed as contours and for plotting an optional vector field, for setting the reference longitude, the contours levels for main or auxiliary field, the reference length used for the vector field plot, the rotation of vectors from model grid to geographic grid, … See plot: map, cross-section and profile plot of one or two fields, and vectors plot over a map
- For
- 2015/09/23 - Version 0.10:
- Interface to Drakkar CDFTools: a number of operators now come in two versions: one accepting multi-variable inputs, and one accepting only mono-variable inputs (with an ‘m’ suffix)
- Multi-variable datasets are managed. This is handy for cases where variables are grouped in a file. See an example
in:
cdftransport.py
, where variable ‘products’ is assigned - Package
climaf.chtml
has been re-designed: simpler function names (fline()
,flines()
, addition of basic functionline()
for creating a simple links line; improve doc - New function
fds()
allows to define simply a dataset from a single data file. See example indata_file.py
- 2015/09/08 - Version 0.9:
- Operator ‘lines’ is smarter re.time axis: (see curves : plot multiple profiles (along p, lat, lon, time, …)):
- Tick marks are smartly adapted to the time period duration.
- When datasets does not cover the same time period, the user can choose wether time axis will be aligned to the same origin or just be the union of all time periods
- Interface to Drakkar CDFTools: cdfmean, cdftransport, cdfheatc, cdfmxlheatc, cdfsections, cdfstd, cdfvT; you need to
have a patched version of Cdftools3.0; see CDFTools operators and examples:
cdftransport.py
andcdftools.py
- CliMAF can provide fixed fields to operators, which path may depend on project and simulation of operator’s first
operand (see
fixed_fields()
) - Fixes:
- datasets of type ‘short’ are correctly read
- operator’s secondary output variables are duly renamed, according to the name given to operator’s the secondary
output when declaring it using
script()
- Operator ‘lines’ is smarter re.time axis: (see curves : plot multiple profiles (along p, lat, lon, time, …)):
- 2015/08/27 - Version 0.8:
- Basics
- A CHANGE BREAKING BACKWARD COMPATIBILITY: default facet/attribute ‘experiment’ was renamed ‘simulation’. It is used for hosting either CMIP5’s facet/attribute ‘rip’, or for ‘EXPID’ at CNRM, or for JobName at IPSL. All ‘projects’ and examples, and this documentation too, have been changed accordingly. Please upgrade to this version if you want a consistent documentation. A facet named ‘experiment’ was added to project CMIP5 (for hosting the ‘CMIP5-controlled-vocabulary’ experiment name, as e.g. ‘historical’).
- default values for facets are now handled on a per-project basis. See
cdef()
andcdataset()
. - Binary
climaf
can be used as a back end in your scripts, feeding it with a string argument. See Using CliMAF as a back end in your scripts
- Outputs and rendering
- Package climaf.chtml allows to easily create an html index, which includes tables of links (or thumbnails) to
image files; iterating on e.g. seasons and variables is handled by CliMAF. See:
- a screen dump for such an index:
- the corresponding rendering code in
index_html.py
- the package documentation:
climaf.chtml
- Function
cfile()
can create hard links: the same datafile (actually: the samer inode) will exists with two filenames (one in CliMAF cache, one which is yours), while disk usage is counted only for one datafile; you may remove any of the two file(name)s as you want, without disturbing accessing the data with the other filename. - When creating a symlink between a CliMAF cache file and another filename with function
cfile()
: the symlink source file is now ‘your’ filename; hence, no risk that some CliMAF command does erase it ‘in your back’; and CliMAf will nicely handle broken symlinks, when you erase ‘your’ files
- Package climaf.chtml allows to easily create an html index, which includes tables of links (or thumbnails) to
image files; iterating on e.g. seasons and variables is handled by CliMAF. See:
- Inputs
- climatology files, which have a somewhat intricated time axis (e.g. monthly averages over a 10 year period) can now
be handled with CliMAF regular time axis management, on the fly, by modifying the basic data selection script: it
can enforce a reference time axis by intepreting the data filename. This works e.g. for IPSL’s averaged annual-cycle
datafiles. If needed, you may change function timefix() near line 30 in
mcdo.sh
- automatic fix of CNRM’s Nemo old data time_axis issues, provided you set environment variable CLIMAF_FIX_NEMO_TIME to anything but ‘no’. This will add processing cost. This adresses the wrong time coordinate variable t_ave_01month and t_ave_00086400
- speed-up datafiles scanning, incl. for transitory data organization during simulation run with libIGCM
- climatology files, which have a somewhat intricated time axis (e.g. monthly averages over a 10 year period) can now
be handled with CliMAF regular time axis management, on the fly, by modifying the basic data selection script: it
can enforce a reference time axis by intepreting the data filename. This works e.g. for IPSL’s averaged annual-cycle
datafiles. If needed, you may change function timefix() near line 30 in
- fixes and minor changes:
- check that no dataset attribute include the separator defined for corresponding project
- fix issues at startup when reading cache index
- rename an argument for operator ‘plot’: domain -> focus
- scripts argument ‘labels’ now uses ‘$’ as a separator
- Basics
- 2015/05/20 - Version 0.7:
- Handle explicitly defined objects ensembles (see
cens
) and explicit dataset ensembles (seeeds()
. Operators which are not ensemble-capable will be automagically looped over members. See examples inensemble.py
. - New standard operator
lines
for plotting profiles or other xy curves for ensembles; see curves : plot multiple profiles (along p, lat, lon, time, …) - Standard operator
plot
has new arguments:contours
for adding contour lines,domain
for greying out land or ocean; see plot: map, cross-section and profile plot of one or two fields, and vectors plot over a map - Extended access to observation data as managed by VDR at CNRM: GPCC, GPCP, ERAI, ERAI-LAND, CRUTS3, CERES (in
addition to OBS4MIPS, and CAMI); see Pre-defined projects and datasets and examples in
data_obs.py
. - Special keyword
crs
is replaced by keywordtitle
: the value of CRS expression for an object is provided to script-based operators under keywordtitle
, if no title value is provided when invoking the operator. Scripts can also independanlty use keywordcrs
for getting the CRS value - cpage keywords changed: widths_list -> widths, heights_list -> heights
- Handle explicitly defined objects ensembles (see
- 2015/05/11 - Version 0.6.1:
- Add a macro feature: easy definition of a macro from a compound object; you can save, edit, load… and macros
are used for interpreting cache content. See
cmacro()
and an example inmacro
. - A general purpose plot operator, named
plot
, is fine for plotting 1D and 2D datasets (maps, cross-sections, profiles, but not Hoevmoeller…) and replaces plotxesc and plotmap. It allows for setting explicit levels in palette, stereopolar projection, vertical coordinate… See plot: map, cross-section and profile plot of one or two fields, and vectors plot over a map - Can list or erase cache content using various filters (on age, size, modif date…); disk usage can be
displayed. See
clist()
,cls()
,crm()
,cdu()
,cwc()
- Can create an array of figures using
cpage()
. See examplefigarray
. - Can cope with un-declared missing values in data files, as e.g. Gelato outputs with value=1.e+20 over land,
which is not the declared missing value; See
calias()
andem
- When declaring data re-scaling, can declare units of the result (see
calias()
) - Can declare correspondance between project-specific frequency names and normalized names (see
cfreqs()
). - Add: howto ..record a CliMAF session
- Cache content index is saved on exit
- Add an example of seaice data handling and plotting. See
seaice.py
- Add a macro feature: easy definition of a macro from a compound object; you can save, edit, load… and macros
are used for interpreting cache content. See
- 2015/04/22 - Version 0.6.0:
- Add operator
plotxsec
(removed in 0.6.1, see replacement at plot: map, cross-section and profile plot of one or two fields, and vectors plot over a map ) - A number of ‘projects’ are built-in, which describe data organization and data location for a number of analyses and simulations datasets available at one of our data centers, as e.g. CMIP5, OBS4MIPS, OCMPI5, EM, …; see Pre-defined projects and datasets
- Variable alias and variable scaling are now managed, on a per-project basis. See function
calias()
- Derived variables can now be defined on a per-project basis. See function
derive()
- CliMAF was proved to work under a CDAT install which uses Python 2.6
- Better explain how to install CliMAf (or not), to run it or to use it as a library; see Using CliMAF at CNRM, on Ciclad or Climserv: fast track and Using CliMAF as a library
- Add operator
2015/04/14 - Version 0.5.0:
A versionning scheme is now used, which is based on recommendations found at http://semver.org.
Starting CliMAF:
- Binary
climaf
allows to launch Python and import Climaf at once. See Running CliMAF interactively - File
~/.climaf
is read as configuration file, at the end of climaf.api import
- Binary
Input data:
New projects can be defined, with project-specific facets/attributes. See
cproject
A number of projects are ‘standard’: CMIP5, OCMPIP5, OBS4MIPS, EM, CAMIOBS, and example
Data location is automatically declared for CMIP5 data at CNRM and on Ciclad (in module site_settings)
Discard pre-defined organizations ‘OCMPI5_Ciclad’, ‘example’, etc, and replace it by smart use of organization ‘generic’.
Note
this leads to some upward incompatibility regarding how data locations are declared for these datasets; please refer to the examples in
data_generic.py
).Access to fixed fields is now possible, and fixed fields may be specific to a given simulation. See examples in
data_generic.py
anddata_cmip5drs.py
Operators:
- Explanation is available on how to know how a given operator is declared to CliMAF, i.e. what is the calling sequence for the external script or binary; see ..know which CliMAF operators are available and what they do
- Simplify declaration of scripts with no output (just omit ${out})
- plotmap: this operator now zoom on the data domain, and plot data across Greenwich meridian correctly
Running CliMAF - messages, cache, errors:
- Verbosity, and cache directory, can be set using environment variables. See Configuring CliMAF
- Simplify use of function
clog()
- Log messages are indented to show recursive calls of ceval()
- Quite extended use of Python exceptions for error handling
2015/04/06:
- time period in CRS and as an argument to ‘ds’ is shortened unambiguously and may show only one date
- function cfile has new arguments: target and link
- CMIP5 facets ‘realm’ and ‘table’ are handled by ‘ds’, ‘dataloc’ and ‘cdef’
- organization called ‘generic’ allow to describe any data file hierarchy and naming
- organization called ‘EM’ introduced, and allows to handle CNRM-CM outputs as managed by EM
- default option for operator regrid is now ‘remapbil’ rather than ‘remapcon2’
- log messages are tabulated
- a log file is added, with own severity level, set by clog_file
- operators with format=None are also evaluated as soon as applied - i.e. cshow no more needednon ncview(…)
Note
Issues with CliMAF and future work are documented at https://github.com/rigoudyg/climaf/issues