Functions for managing/viewing results¶
There are a few functions for managing CliMAF results
cfile : get the file value of a CliMAF object¶
-
climaf.driver.
cfile
(object, target=None, ln=None, hard=None, deep=None)[source]¶ Provide the filename for a CliMAF object, or copy this file to target. Launch computation if needed.
Parameters: - object (CliMAF object) – either a dataset or a ‘compound’ object (e.g. the result of a CliMAF operator)
- target (str, optional) – name of the destination file in case you really need another filename; CliMAF will then anyway also store the result in its cache, either as a copy (default), or a sym- or a hard-link (see below);
- ln (logical, optional) – if True, CliMAF cache file is created as a symlink to the target; this allows to cross filesystem boundaries, while still saving disk space (wrt to a copy); CliMAF will manage the broken link cases (at the expense of a new computation)
- hard (logical, optional) – if True, CliMAF cache file is created as a hard link to the target; this allows to save disk space, but does not allow to cross filesystem boundaries
- deep (logical, optional) –
governs the use of cached values when computing the object:
- if missing, or None : use cache as much as possible (speed up the computation)
- False : make a shallow computation, i.e. do not use cached values for the top level operation
- True : make a deep computation, i.e. do not use any cached value
Returns: - if target is provided, returns this filename (or linkname) if computation is successful (‘target’ contains the result), and None otherwise;
- if target is not provided, returns the filename in CliMAF cache, which contains the result (and None if failure)
efile : create the file for an ensemble of CliMAF objects¶
-
climaf.driver.
efile
(obj, filename, force=False)[source]¶ Create a single file for an ensemble of CliMAF objects (launch computation if needed).
This is a convenience function. Such files are not handled in CliMAF cache
Parameters: - obj (CliMAF object) – an ensemble of CliMAF objects (‘cens’ objet)
- filename (str) – output filename. It will include a field for each ensemble’s member, with a variable name suffixed by the member label (e.g. : tas_CNRM-CM, tas_IPSL-CM… ) (more formally : ‘var(obj.order[n])’_’obj.ens[order[n]]’)
- force (logical, optional) – if True, CliMAF will overwrite the file ‘filename’ if it already exists
cMA : get the Masked Array value of a CliMAF object¶
-
climaf.driver.
cMA
(obj, deep=None)[source]¶ Provide the Masked Array value for a CliMAF object. Launch computation if needed.
Parameters: - obj (CliMAF object) – either a datset or a ‘compound’ object (like the result of a CliMAF standard operator)
- deep (logical, optional) –
governs the use of cached values when computing the object
- if missing, or None : use cache as much as possible
- False : make a shallow computation, i.e. do not use cached values for top level operation
- True : make a deep computation, i.e. do not use any cached value
Returns: a Masked Array containing the object’s value
cvalue : get the scalar value of a CliMAF object (at a given index)¶
-
climaf.driver.
cvalue
(obj, index=0, deep=None, cost=None)[source]¶ Return the value of the array for an object, at a given index, after MV flattening
Arg DEEP is used as for
climaf.driver.cfile()
If arg COST is True, a tuple is returned, which second element is the total compute cost for the value
Example, where the target object is a scalar (0d) field:
>>> data=ds(project='mine',variable='tas', ...) >>> data1=time_average(data) >>> data2=space_average(data1) >>> v=cvalue(data2)
Does use the file representation of the object
cshow : dump an object or display a ‘figure’¶
cpage : create an array of figures (output: ‘png’ or ‘pdf’ figure)¶
-
class
climaf.classes.
cpage
(fig_lines=None, widths=None, heights=None, fig_trim=True, page_trim=True, format='png', orientation=None, page_width=1000.0, page_height=1500.0, title='', x=0, y=26, ybox=50, pt=24, font='Times-New-Roman', gravity='North', background='white', insert='', insert_width=200)[source]¶ Builds a CliMAF cpage object, which represents an array of figures (output: ‘png’ or ‘pdf’ figure)
Parameters: - fig_lines (a list of lists of figure objects or an ensemble of figure objects) – each sublist of ‘fig_lines’ represents a line of figures
- widths (list, optional) –
the list of figure widths, i.e. the width of each column. By default, if fig_lines is:
- a list of lists: spacing is even
- an ensemble: one column is used
- heights (list, optional) – the list of figure heights, i.e. the height of each line. By default spacing is even
- fig_trim (logical, optional) – to turn on/off triming for all figures. It removes all the surrounding extra space of figures in the page, either True (default) or False
- page_trim (logical, optional) – to turn on/off triming for the page. It removes all the surrounding extra space of the page, either True (default) or False
- format (str, optional) – graphic output format, either ‘png’ (default) or ‘pdf’(not recommended)
- page_width (float, optional) – width resolution of resultant image; CLiMAF default: 1000.
- page_height (float, optional) – height resolution of resultant image; CLiMAF default: 1500.
- orientation (str,optional) – if set, it supersedes page_width and page_height with values 1000*1500 (for portrait) or 1500*1000 (for landscape)
- title (str, optional) – append a label below or above (depending on optional argument ‘gravity’) figures in the page.
- insert (str,optional) – the filename for an image to insert (centered at the bottom)
- insert_width (int,optional) – the width at which the inserted image will be scaled (in pixels)
If title is activated:
- x, y (int, optional): annotate the page with text.
x is the offset towards the right from the upper left corner
of the page, while y is the offset upward or the bottom
according to the optional argument ‘gravity’ (i.e. ‘South’ or ‘North’
respectively); CLiMAF default: x=0, y=26. For more details, see:
http://www.imagemagick.org/script/command-line-options.php?#annotate ;
where x and y correspond respectively to tx and ty
in
-annotate {+-}tx{+-}ty text
- ybox (int, optional): width of the assigned box for title; CLiMAF default: 50. For more details, see: http://www.imagemagick.org/script/command-line-options.php?#splice
- pt (int, optional): font size of the title; CLiMAF default: 24
- font (str, optional): set the font to use when creating title; CLiMAF default: ‘Times-New-Roman’. To print a complete list of fonts, use: ‘convert -list font’
- gravity (str, optional): the choosen direction specifies where to position title; CLiMAF default: ‘North’. For more details, see: http://www.imagemagick.org/script/command-line-options.php?#gravity
- background (str, optional): background color of the assigned box for title; default: ‘white’. To print a complete list of color names, use: ‘convert -list color’
Example
Using no default value, to create a page with 2 columns and 3 lines:
>>> tas_ds=ds(project='example',simulation='AMIPV6ALB2G', variable='tas', period='1980-1981') >>> tas_avg=time_average(tas_ds) >>> fig=plot(tas_avg,title='title') >>> my_page=cpage([[None, fig],[fig, fig],[fig,fig]], widths=[0.2,0.8], ... heights=[0.33,0.33,0.33], fig_trim=False, page_trim=False, ... format='pdf', title='Page title', x=10, y=20, ybox=45, ... pt=20, font='Utopia', gravity='South', background='grey90', ... page_width=1600., page_height=2400.)
cpage_pdf : create an array of figures (output: ‘pdf’ figure)¶
-
class
climaf.classes.
cpage_pdf
(fig_lines=None, widths=None, heights=None, orientation=None, page_width=1000.0, page_height=1500.0, scale=1.0, openright=False, title='', x=0, y=2, titlebox=False, pt='Huge', font='\familydefault', background='white')[source]¶ Builds a CliMAF cpage_pdf object, which represents an array of figures (output: ‘pdf’ figure). Figures are automatically centered in the page using ‘pdfjam’ tool; see http://www2.warwick.ac.uk/fac/sci/statistics/staff/academic-research/firth/software/pdfjam
Parameters: - fig_lines (a list of lists of figure objects or an ensemble of figure objects) – each sublist of ‘fig_lines’ represents a line of figures
- widths (list, optional) –
the list of figure widths, i.e. the width of each column. By default, if fig_lines is:
- a list of lists: spacing is even
- an ensemble: one column is used
- heights (list, optional) – the list of figure heights, i.e. the height of each line. By default spacing is even
- page_width (float, optional) – width resolution of resultant image; CLiMAF default: 1000.
- page_height (float, optional) – height resolution of resultant image; CLiMAF default: 1500.
- orientation (str,optional) – if set, it supersedes page_width and page_height with values 1000*1500 (for portrait) or 1500*1000 (for landscape)
- scale (float, optional) – to scale all input pages; default:1.
- openright (logical, optional) – this option puts an empty figure before the first figure; default: False. For more details, see: http://ftp.oleane.net/pub/CTAN/macros/latex/contrib/pdfpages/pdfpages.pdf
- title (str, optional) – append a label in the page.
If title is activated, it is by default horizontally centered:
- x (int, optional): title horizontal shift (in cm).
- y (int, optional): vertical shift from the top of the page (in cm); only positive (down) values have an effect, default=2 cm
- titlebox (logical, optional): set it to True to frame the text in a box, frame color is ‘black’
- pt (str, optional): title font size; CLiMAF default: ‘Huge’ (corresponding to 24 pt). You can set or not a backslash before this argument.
- font (str, optional): font abbreviation among available LaTex fonts; default: ‘\familydefault’.
- background (str, optional): frame fill background color; among LaTex ‘fcolorbox’ colors; default: ‘white’.
Left and right margins are set to 2cm.
Example
Using no default value, to create a PDF page with 2 columns and 3 lines:
>>> tas_ds=ds(project='example',simulation='AMIPV6ALB2G', variable='tas', period='1980-1981') >>> tas_avg=time_average(tas_ds) >>> fig=plot(tas_avg,title='title',format='pdf') >>> crop_fig=cpdfcrop(fig) >>> my_pdfpage=cpage_pdf([[crop_fig,crop_fig],[crop_fig, crop_fig],[crop_fig,crop_fig]], ... widths=[0.2,0.8], heights=[0.33,0.33,0.33], page_width=800., page_height=1200., ... scale=0.95, openright=True, title='Page title', x=-5, y=10, titlebox=True, ... pt='huge', font='ptm', background='yellow') # Font name is 'Times'
html : create an html index, with tables of links to figures¶
CliMAF module html
defines functions for building some html index
giving acces to figure files, through links bearing a label or through
thumbnails. It eases iterating over lines and columns in tables.
See a code example in index_html.py
or a screen dump for a similar code
here
-
climaf.chtml.
header
(title, style_file=None)[source] Returns text for an html document header, with provided title. If a style filename is not provided, a default style sheet will apply
-
climaf.chtml.
trailer
()[source] Returns the text for closing an html document
-
climaf.chtml.
section
(title, level=1, key='None')[source] Returns text for a section header in an html document with given title. Style depends on level value. Arg key is not yet used
-
climaf.chtml.
open_table
(title='', columns=[], spacing=5)[source] Returns header text for an html table. title will go as title for first column. columns should be a list of column titles
-
climaf.chtml.
close_table
()[source] Returns text for closing an html table
-
climaf.chtml.
open_line
(title='')[source]
-
climaf.chtml.
close_line
()[source]
-
climaf.chtml.
line
(list_of_pairs, title='', thumbnail=None, hover=True, dirname=None, altdir=None)[source] Create an html line with labels and links from first args list_of_pairs (and when this is not a pair, only put the label). Put a line title if provided. Replace labels with thumbnail figures if arg thumbnail is set to a size (in pixels) and display figures when you mouse over it if arg hover is set to True or to a size (in pixels); in that case, dic can also be a list of filenames. If ‘dirname’ is not None, creates hardlinks to the filenames, in directory dirname, and named as ‘climaf_atlas’([0-9]+).ext (where ‘ext’ is ‘png’, ‘pdf’ or ‘eps’). This allows to generate a portable atlas in dirname
-
climaf.chtml.
link
(label, filename, thumbnail=None, hover=True)[source] Creates the provided label, with a link to the provided image filename (if not None) and possibly showing a thumbnail for the image (with the provided thumbnail size) and possibly displaying this image when mouse is over it (with the provided hover size).
- ‘thumbnail’ can be an integer or a string with width or height (in these cases, width=height), or a string with width and height separated by character ‘x’ or ‘*’. The size is in pixels; default : None (no thumbnail)
- ‘hover’ can be a logical, or a string with width or height (in this
case, width=height), or a string with width and height separated by
character ‘x’ or ‘*’. The size is in pixels; default is True, which
tanslates according to value of thumbnail :
- if thumbnail is not None, hover width and height are respectively set as 3 times that of thumbnail width and height
- if thumbnail is None, size is ‘200*200’
-
climaf.chtml.
link_on_its_own_line
(label, filename, thumbnail=None, hover=True)[source] Does the same as
link()
,but for a link which is sole on its own line
-
climaf.chtml.
cell
(label, filename=None, thumbnail=None, hover=True, dirname=None, altdir=None)[source] Create a table cell with the provided label, which bears a link to the provided filename and possibly shows a thumbnail for the link with the provided thumbnail size (in pixels) and possibly display it when you mouse over it (with the provided hover size in pixels).
If ‘dirname’ is not None, creates a hard link in directory dirname to file filename. This allow to generate a portable atlas in this directory. Hard links are named after pattern climaf_atlas<digit>.<extension>
‘dirname’ can be a relative or absolute path, as long as filename and dirname paths are coherent
If ‘altdir’ is not None (and ‘dirname is None), the HREF links images in index have the prefix of their absolute path changed from $CLIMAF_CACHE to ‘altdir’ (use case : when the Http server only knows another filesystem). Example:
- CLIMAF_CACHE=/prodigfs/ipslfs/dods/fabric/coding_sprint_NEMO/stephane
- URL https://vesg.ipsl.upmc.fr /thredds/fileServer/IPSLFS/fabric/coding_sprint_NEMO/stephane/ …/fig.png
-
climaf.chtml.
fline
(func, farg, sargs, title=None, common_args=[], other_args=[], thumbnail=None, hover=True, dirname=None, **kwargs)[source] Create the html text for a line of table cells, by iterating calling a function, once per column, with at least two arguments. Cells have a label and possibly a link
- ‘func’ is a python function which computes and labels and/or figures;
- ‘farg’ is any object, used a 1st arg for ‘func’;
- ‘sargs’ is a list or a dict, used for providing the 2nd arg to each call to ‘func’.
- ‘title’ is a line title (for first column); if missing, ‘farg’ is used
- see further below fo remaining arguments
So, there will be one column/cell per item in ‘sargs’; each cell shows a label which can be an active link. Both the label value and the link value can be the result of calling ‘func’ arg with the pair of arguments (‘farg’ and the running element of ‘sargs’); the function can return a single value (either a label or a figure filename) or both
Use cases :
a line showing just numeric values; we assume that function average(var,mask) returns such a numeric value, which is a gloabl average of a variable over a mask:
>>> rep=fline(average, 'tas', ... [ 'global', 'sea', 'land', 'tropics'], 'tas averages')
a line showing the same average values, but each value is a link to e.g. a figure of the time series of global average values : same call, but just let function ‘average’ compute the average and the figure, and return a couple : average, figure filename
a line showing pre-defined labels, which here are shortcuts for mask names, and which carry links to same figures as above : let function ‘average’ only return the figure filename, and call :
>>> rep=fline(average, 'tas', ... {'global':'GLB','sea':'SEA','land':'LND'}, 'tas averages')
Advanced arguments :
- common_args : a list of additionnal arguments to pass to ‘func’, whatever the value of its second argument
- other_args : a dictionnary of lists of additionnal arguments to pass to ‘func’; only the entry which key equals running value of second argument is passed to ‘func’ (after common_args)
- thumbnail : if ‘func’ returns a filename, generate a thumbnail image of that size (in pixels)
- hover : if ‘func’ returns a filename, display image of that size
(in pixels) when you mouse over it. If hover is True:
- hover width and height are respectively set as 3 times that of thumbnail width and height if thumbnail is not None
- hover is set to ‘200*200’ if thumbnail is None
- dirname : if ‘func’ returns a filename, creates a directory (if doesn’t exist) wich contains filename as a hard link to the target dirname/’climaf_atlas’([0-9]+).ext (‘ext’ is ‘png’, ‘pdf’ or ‘eps’)
-
climaf.chtml.
flines
(func, fargs, sargs, common_args=[], other_fargs=[], other_sargs=[], thumbnail=None, hover=True, dirname=None, **kwargs)[source] See doc for
fline()
firstCreates a table by iterating calling fline over ‘fargs’ (which can be a list or a dict) with :
- ‘farg’ being the running element (or key) of ‘fargs’
- ‘title’ being the corresponding ‘fargs’ value (or ‘farg’ if not a dict)
- ‘common_args’ being forwarded to
fline()
- ‘other_args’ being the merge of other_fargs[farg] and other_sargs
It forwards remaining keyword arguments (kwargs) to
fline()
Example : assuming that function avg returns the filename for a figure showing the average value of a variable over a mask, create a table of links for average values of two variables over two masks, with thumbnail of images and displaying images when you mouse over it with ‘hover’ argument:
>>> t=table_lines(avg,['tas','tos'],['land','sea'],thumbnail=40,hover='60x80')
-
climaf.chtml.
vspace
(nb=1)[source]
clist : tell what’s in cache, and much more¶
-
climaf.cache.
clist
(size='', age='', access=0, pattern='', not_pattern='', usage=False, count=False, remove=False, CRS=False, special=False)[source]¶ Internal function used by its front-ends :
cls()
,crm()
,cdu()
,cwc()
List the content of CliMAF cache according to some search criteria and operate possibly an action (usage, count or remove) on this list.
Please consider the cost and benefit of first updating CliMAF cache index (by scanning files on disk) using
csync()
Parameters: - size (string, optional) –
n[ckMG] Search files using more than n units of disk space, rounding up. The following suffixes can be used:
- ”c” for bytes (default)
- ”k” for Kilobytes (units of 1,024 bytes)
- ”M” for Megabytes (units of 1,048,576 bytes)
- ”G” for Gigabytes (units of 1,073,741,824 bytes)
- age (string, optional) –
Number of 24h periods. Search files which status was last changed n*24 hours ago. Any fractional part is ignored, so to match age=’+1’, a file has to have been changed at least two days ago. Numeric arguments can be specified as:
- +n for greater than n
- -n for less than n,
- n for exactly n.
- access (int, optional) – n Search files which were last accessed more than n*24 hours ago. Any fractional part is ignored, so to match access=’1’, a file has to have been accessed at least two days ago.
- pattern (string, optional) – Scan through crs and filenames looking for the first location where the regular expression pattern produces a match.
- not_pattern (string, optional) – Scan through crs and filenames looking for the location where the regular expression not_pattern does not produce a match.
- usage (bool, optional) – Estimate found files space usage, for each found file and total size. If count is True, estimate only found files total space usage.
- count (bool, optional) – Return the number of found files. If CRS is True, also return crs of found files.
- remove (bool, optional) – Remove the found files. This argument is exclusive.
- CRS (bool, optional) – if True, print also CRS expression. Useful only if count is True.
Returns: - The dictionary corresponding to the request and associated action ( or dictionary
of CliMAF cache index if no argument is provided)
- Example to search files using more than 3M of disk space, which status
was last changed more than 15 days ago and containing the pattern ‘1980-1981’ either in crs or filename. For found files, we want to estimate only found files total space usage:
>>> clist(size='3M', age='+15', pattern= '1980-1981', usage=True, count=True)
- size (string, optional) –
cls : tell what’s in cache¶
crm : remove some files from cache¶
-
climaf.cache.
crm
(**kwargs)[source]¶ Remove the cache files found by ‘clist()’ when using same arguments. See
clist()
Example to remove files using more than 3M of disk space, which status was last changed more than 15 days ago and containing the pattern ‘1980-1981’ either in crs or filename:
>>> crm(size='3M', age='+15', pattern='1980-1981')
cdu : disk cache usage¶
-
climaf.cache.
cdu
(**kwargs)[source]¶ Report disk usage, for files matching some criteria, as specified for
clist()
. With count=True, report only total disk usage.Example to search files using more than 3M of disk space, which status was last changed more than 15 days ago and containing the pattern ‘1980-1981’ either in crs or filename. For found files, we want to estimate only found files total space usage:
>>> cdu(size='3M', age='+15', pattern= '1980-1981', count=True)
cwc : count some files in cache¶
-
climaf.cache.
cwc
(**kwargs)[source]¶ Report number of cache files matching some criteria, as specified for
clist()
. If CRS is True, also return CRS expression of found files.Example to return the number and crs associated of files using more than 3M of disk space, which status was last changed more than 15 days ago and containing the pattern ‘1980-1981’ either in crs or filename:
>>> cwc(size='3M', age='+15', pattern= '1980-1981', CRS=True)
craz : reset cache¶
cdrop : erase a result’s file¶
-
climaf.cache.
cdrop
(obj, rm=True, force=False)[source]¶ Deletes the cached file for a CliMAF object, if it exists
Parameters: - obj (cobject or string) – object to delete, or its string representation (CRS)
- force (bool) – should we delete the object even if it is ‘protected’
- rm (bool) – for advanced use only; should we actually delete (rm) the file, or just forget it in CliMAF cache index
Returns: None if object does not exists, False if failing to delete, True if OK
Example
>>> dg=ds(project='example', simulation='AMIPV6ALB2G', variable='tas', period='1980-1981') >>> f=cfile(dg) >>> os.system('ls -al '+f) >>> cdrop(dg)
cprotect : protect a result’s file against cdrop and craz¶
ccost : provide compute cost for an object¶
-
climaf.cache.
ccost
(cobject)[source]¶ Provide the compute costs (elapsed time) for an object
Returned value is of class cost, which has fields tc (for total_cost) and lc (for last_op_cost):
- tc is the sum of all compute operations that lead to the object
- lc is the cost for the last operation (top level)
When a single operation provides multiple outputs, all outputs are charged with the cost of that operation (cost is not shared). This may lead to an overestimated cost when such outputs are re-used together in further operations