Functions for managing/viewing results

There are a few functions for managing CliMAF results

cfile : get the file value of a CliMAF object

climaf.driver.cfile(object, target=None, ln=None, hard=None, deep=None)

Provide the filename for a CliMAF object, or copy this file to target. Launch computation if needed.

Parameters:

• object (CliMAF object) – either a dataset or a ‘compound’ object (e.g. the result of a CliMAF operator)
• target (str, optional) – name of the destination file in case you really need another filename; CliMAF will then anyway also store the result in its cache, either as a copy (default), or a sym- or a hard-link (see below);
• ln (logical, optional) – if True, CliMAF cache file is created as a symlink to the target; this allows to cross filesystem boundaries, while still saving disk space (wrt to a copy); CliMAF will manage the broken link cases (at the expense of a new computation)
• hard (logical, optional) – if True, CliMAF cache file is created as a hard link to the target; this allows to save disk space, but does not allow to cross filesystem boundaries
• deep (logical, optional) –

  governs the use of cached values when computing the object:

  • if missing, or None : use cache as much as possible (speed up the computation)
  • False : make a shallow computation, i.e. do not use cached values for the top level operation
  • True : make a deep computation, i.e. do not use any cached value

Returns:

• if target is provided, returns this filename (or linkname) if computation is successful (‘target’ contains the result), and None otherwise;
• if target is not provided, returns the filename in CliMAF cache, which contains the result (and None if failure)

efile : create the file for an ensemble of CliMAF objects

climaf.driver.efile(obj, filename, force=False)

Create a single file for an ensemble of CliMAF objects (launch computation if needed).

This is a convenience function. Such files are not handled in CliMAF cache

Parameters:

• obj (CliMAF object) – an ensemble of CliMAF objects (‘cens’ objet)
• filename (str) – output filename. It will include a field for each ensemble’s member, with a variable name suffixed by the member label (e.g. : tas_CNRM-CM, tas_IPSL-CM… ) (more formally : ‘var(obj.order[n])’_’obj.ens[order[n]]’)
• force (logical, optional) – if True, CliMAF will overwrite the file ‘filename’ if it already exists

cMA : get the Masked Array value of a CliMAF object

climaf.driver.cMA(obj, deep=None)

Provide the Masked Array value for a CliMAF object. Launch computation if needed.

Parameters:

• obj (CliMAF object) – either a datset or a ‘compound’ object (like the result of a CliMAF standard operator)
• deep (logical, optional) –

  governs the use of cached values when computing the object

  • if missing, or None : use cache as much as possible
  • False : make a shallow computation, i.e. do not use cached values for top level operation
  • True : make a deep computation, i.e. do not use any cached value

Returns:

a Masked Array containing the object’s value

cvalue : get the scalar value of a CliMAF object (at a given index)

climaf.driver.cvalue(obj, index=0, deep=None, cost=None)

Return the value of the array for an object, at a given index, after MV flattening

Arg DEEP is used as for climaf.driver.cfile()

If arg COST is True, a tuple is returned, which second element is the total compute cost for the value

Example, where the target object is a scalar (0d) field:

>>> data=ds(project='mine',variable='tas', ...)
>>> data1=time_average(data)
>>> data2=space_average(data1)
>>> v=cvalue(data2)

Does use the file representation of the object

cshow : dump an object or display a ‘figure’

climaf.driver.cshow(obj)

Provide the in-memory value of a CliMAF object. For a figure object, this will lead to display it ( launch computation if needed. )

cpage : create an array of figures (output: ‘png’ or ‘pdf’ figure)

class climaf.classes.cpage(fig_lines=None, widths=None, heights=None, fig_trim=True, page_trim=True, format='png', orientation=None, page_width=1000.0, page_height=1500.0, title='', x=0, y=26, ybox=50, pt=24, font='Times-New-Roman', gravity='North', background='white', insert='', insert_width=200)

Builds a CliMAF cpage object, which represents an array of figures (output: ‘png’ or ‘pdf’ figure)

Parameters:

• fig_lines (a list of lists of figure objects or an ensemble of figure objects) – each sublist of ‘fig_lines’ represents a line of figures
• widths (list, optional) –

  the list of figure widths, i.e. the width of each column. By default, if fig_lines is:

  • a list of lists: spacing is even
  • an ensemble: one column is used

• heights (list, optional) – the list of figure heights, i.e. the height of each line. By default spacing is even
• fig_trim (logical, optional) – to turn on/off triming for all figures. It removes all the surrounding extra space of figures in the page, either True (default) or False
• page_trim (logical, optional) – to turn on/off triming for the page. It removes all the surrounding extra space of the page, either True (default) or False
• format (str, optional) – graphic output format, either ‘png’ (default) or ‘pdf’(not recommended)
• page_width (float, optional) – width resolution of resultant image; CLiMAF default: 1000.
• page_height (float, optional) – height resolution of resultant image; CLiMAF default: 1500.
• orientation (str,optional) – if set, it supersedes page_width and page_height with values 1000*1500 (for portrait) or 1500*1000 (for landscape)
• title (str, optional) – append a label below or above (depending on optional argument ‘gravity’) figures in the page.
• insert (str,optional) – the filename for an image to insert (centered at the bottom)
• insert_width (int,optional) – the width at which the inserted image will be scaled (in pixels)

If title is activated:

• x, y (int, optional): annotate the page with text. x is the offset towards the right from the upper left corner of the page, while y is the offset upward or the bottom according to the optional argument ‘gravity’ (i.e. ‘South’ or ‘North’ respectively); CLiMAF default: x=0, y=26. For more details, see: http://www.imagemagick.org/script/command-line-options.php?#annotate ; where x and y correspond respectively to tx and ty in -annotate {+-}tx{+-}ty text
• ybox (int, optional): width of the assigned box for title; CLiMAF default: 50. For more details, see: http://www.imagemagick.org/script/command-line-options.php?#splice
• pt (int, optional): font size of the title; CLiMAF default: 24
• font (str, optional): set the font to use when creating title; CLiMAF default: ‘Times-New-Roman’. To print a complete list of fonts, use: ‘convert -list font’
• gravity (str, optional): the choosen direction specifies where to position title; CLiMAF default: ‘North’. For more details, see: http://www.imagemagick.org/script/command-line-options.php?#gravity
• background (str, optional): background color of the assigned box for title; default: ‘white’. To print a complete list of color names, use: ‘convert -list color’

Example

Using no default value, to create a page with 2 columns and 3 lines:

>>> tas_ds=ds(project='example',simulation='AMIPV6ALB2G', variable='tas', period='1980-1981')
>>> tas_avg=time_average(tas_ds)
>>> fig=plot(tas_avg,title='title')
>>> my_page=cpage([[None, fig],[fig, fig],[fig,fig]], widths=[0.2,0.8],
... heights=[0.33,0.33,0.33], fig_trim=False, page_trim=False,
... format='pdf', title='Page title', x=10, y=20, ybox=45,
... pt=20, font='Utopia', gravity='South', background='grey90',
... page_width=1600., page_height=2400.)

cpage_pdf : create an array of figures (output: ‘pdf’ figure)

class climaf.classes.cpage_pdf(fig_lines=None, widths=None, heights=None, orientation=None, page_width=1000.0, page_height=1500.0, scale=1.0, openright=False, title='', x=0, y=2, titlebox=False, pt='Huge', font='\familydefault', background='white')

Builds a CliMAF cpage_pdf object, which represents an array of figures (output: ‘pdf’ figure). Figures are automatically centered in the page using ‘pdfjam’ tool; see http://www2.warwick.ac.uk/fac/sci/statistics/staff/academic-research/firth/software/pdfjam

Parameters:

• fig_lines (a list of lists of figure objects or an ensemble of figure objects) – each sublist of ‘fig_lines’ represents a line of figures
• widths (list, optional) –

  the list of figure widths, i.e. the width of each column. By default, if fig_lines is:

  • a list of lists: spacing is even
  • an ensemble: one column is used

• heights (list, optional) – the list of figure heights, i.e. the height of each line. By default spacing is even
• page_width (float, optional) – width resolution of resultant image; CLiMAF default: 1000.
• page_height (float, optional) – height resolution of resultant image; CLiMAF default: 1500.
• orientation (str,optional) – if set, it supersedes page_width and page_height with values 1000*1500 (for portrait) or 1500*1000 (for landscape)
• scale (float, optional) – to scale all input pages; default:1.
• openright (logical, optional) – this option puts an empty figure before the first figure; default: False. For more details, see: http://ftp.oleane.net/pub/CTAN/macros/latex/contrib/pdfpages/pdfpages.pdf
• title (str, optional) – append a label in the page.

If title is activated, it is by default horizontally centered:

• x (int, optional): title horizontal shift (in cm).
• y (int, optional): vertical shift from the top of the page (in cm); only positive (down) values have an effect, default=2 cm
• titlebox (logical, optional): set it to True to frame the text in a box, frame color is ‘black’
• pt (str, optional): title font size; CLiMAF default: ‘Huge’ (corresponding to 24 pt). You can set or not a backslash before this argument.
• font (str, optional): font abbreviation among available LaTex fonts; default: ‘\familydefault’.
• background (str, optional): frame fill background color; among LaTex ‘fcolorbox’ colors; default: ‘white’.

Left and right margins are set to 2cm.

Example

Using no default value, to create a PDF page with 2 columns and 3 lines:

>>> tas_ds=ds(project='example',simulation='AMIPV6ALB2G', variable='tas', period='1980-1981')
>>> tas_avg=time_average(tas_ds)
>>> fig=plot(tas_avg,title='title',format='pdf')
>>> crop_fig=cpdfcrop(fig)
>>> my_pdfpage=cpage_pdf([[crop_fig,crop_fig],[crop_fig, crop_fig],[crop_fig,crop_fig]],
... widths=[0.2,0.8], heights=[0.33,0.33,0.33], page_width=800., page_height=1200.,
... scale=0.95, openright=True, title='Page title', x=-5, y=10, titlebox=True,
... pt='huge', font='ptm', background='yellow') # Font name is 'Times'

html : create an html index, with tables of links to figures

CliMAF module html defines functions for building some html index giving acces to figure files, through links bearing a label or through thumbnails. It eases iterating over lines and columns in tables.

See a code example in index_html.py or a screen dump for a similar code here

climaf.chtml.header(title, style_file=None)

Returns text for an html document header, with provided title. If a style filename is not provided, a default style sheet will apply

climaf.chtml.trailer()

Returns the text for closing an html document

climaf.chtml.section(title, level=1, key='None')

Returns text for a section header in an html document with given title. Style depends on level value. Arg key is not yet used

climaf.chtml.open_table(title='', columns=[], spacing=5)

Returns header text for an html table. title will go as title for first column. columns should be a list of column titles

climaf.chtml.close_table()

Returns text for closing an html table

climaf.chtml.open_line(title='')

climaf.chtml.close_line()

climaf.chtml.line(list_of_pairs, title='', thumbnail=None, hover=True, dirname=None, altdir=None)

Create an html line with labels and links from first args list_of_pairs (and when this is not a pair, only put the label). Put a line title if provided. Replace labels with thumbnail figures if arg thumbnail is set to a size (in pixels) and display figures when you mouse over it if arg hover is set to True or to a size (in pixels); in that case, dic can also be a list of filenames. If ‘dirname’ is not None, creates hardlinks to the filenames, in directory dirname, and named as ‘climaf_atlas’([0-9]+).ext (where ‘ext’ is ‘png’, ‘pdf’ or ‘eps’). This allows to generate a portable atlas in dirname

climaf.chtml.link(label, filename, thumbnail=None, hover=True)

Creates the provided label, with a link to the provided image filename (if not None) and possibly showing a thumbnail for the image (with the provided thumbnail size) and possibly displaying this image when mouse is over it (with the provided hover size).

• ‘thumbnail’ can be an integer or a string with width or height (in these cases, width=height), or a string with width and height separated by character ‘x’ or ‘*’. The size is in pixels; default : None (no thumbnail)
• ‘hover’ can be a logical, or a string with width or height (in this case, width=height), or a string with width and height separated by character ‘x’ or ‘*’. The size is in pixels; default is True, which tanslates according to value of thumbnail :
  • if thumbnail is not None, hover width and height are respectively set as 3 times that of thumbnail width and height
  • if thumbnail is None, size is ‘200*200’

climaf.chtml.link_on_its_own_line(label, filename, thumbnail=None, hover=True)

Does the same as link() ,but for a link which is sole on its own line

climaf.chtml.cell(label, filename=None, thumbnail=None, hover=True, dirname=None, altdir=None)

Create a table cell with the provided label, which bears a link to the provided filename and possibly shows a thumbnail for the link with the provided thumbnail size (in pixels) and possibly display it when you mouse over it (with the provided hover size in pixels).

If ‘dirname’ is not None, creates a hard link in directory dirname to file filename. This allow to generate a portable atlas in this directory. Hard links are named after pattern climaf_atlas<digit>.<extension>

‘dirname’ can be a relative or absolute path, as long as filename and dirname paths are coherent

If ‘altdir’ is not None (and ‘dirname is None), the HREF links images in index have the prefix of their absolute path changed from $CLIMAF_CACHE to ‘altdir’ (use case : when the Http server only knows another filesystem). Example:

• CLIMAF_CACHE=/prodigfs/ipslfs/dods/fabric/coding_sprint_NEMO/stephane
• URL https://vesg.ipsl.upmc.fr /thredds/fileServer/IPSLFS/fabric/coding_sprint_NEMO/stephane/ …/fig.png

climaf.chtml.fline(func, farg, sargs, title=None, common_args=[], other_args=[], thumbnail=None, hover=True, dirname=None, **kwargs)

Create the html text for a line of table cells, by iterating calling a function, once per column, with at least two arguments. Cells have a label and possibly a link

• ‘func’ is a python function which computes and labels and/or figures;
• ‘farg’ is any object, used a 1st arg for ‘func’;
• ‘sargs’ is a list or a dict, used for providing the 2nd arg to each call to ‘func’.
• ‘title’ is a line title (for first column); if missing, ‘farg’ is used
• see further below fo remaining arguments

So, there will be one column/cell per item in ‘sargs’; each cell shows a label which can be an active link. Both the label value and the link value can be the result of calling ‘func’ arg with the pair of arguments (‘farg’ and the running element of ‘sargs’); the function can return a single value (either a label or a figure filename) or both

Use cases :

• a line showing just numeric values; we assume that function average(var,mask) returns such a numeric value, which is a gloabl average of a variable over a mask:

  >>> rep=fline(average, 'tas',        ...   [ 'global', 'sea', 'land', 'tropics'], 'tas averages')
  

• a line showing the same average values, but each value is a link to e.g. a figure of the time series of global average values : same call, but just let function ‘average’ compute the average and the figure, and return a couple : average, figure filename

• a line showing pre-defined labels, which here are shortcuts for mask names, and which carry links to same figures as above : let function ‘average’ only return the figure filename, and call :

  >>> rep=fline(average, 'tas',
  ...  {'global':'GLB','sea':'SEA','land':'LND'}, 'tas averages')
  

Advanced arguments :

• common_args : a list of additionnal arguments to pass to ‘func’, whatever the value of its second argument
• other_args : a dictionnary of lists of additionnal arguments to pass to ‘func’; only the entry which key equals running value of second argument is passed to ‘func’ (after common_args)
• thumbnail : if ‘func’ returns a filename, generate a thumbnail image of that size (in pixels)
• hover : if ‘func’ returns a filename, display image of that size (in pixels) when you mouse over it. If hover is True:
  • hover width and height are respectively set as 3 times that of thumbnail width and height if thumbnail is not None
  • hover is set to ‘200*200’ if thumbnail is None
• dirname : if ‘func’ returns a filename, creates a directory (if doesn’t exist) wich contains filename as a hard link to the target dirname/’climaf_atlas’([0-9]+).ext (‘ext’ is ‘png’, ‘pdf’ or ‘eps’)

climaf.chtml.flines(func, fargs, sargs, common_args=[], other_fargs=[], other_sargs=[], thumbnail=None, hover=True, dirname=None, **kwargs)

See doc for fline() first

Creates a table by iterating calling fline over ‘fargs’ (which can be a list or a dict) with :

• ‘farg’ being the running element (or key) of ‘fargs’
• ‘title’ being the corresponding ‘fargs’ value (or ‘farg’ if not a dict)
• ‘common_args’ being forwarded to fline()
• ‘other_args’ being the merge of other_fargs[farg] and other_sargs

It forwards remaining keyword arguments (kwargs) to fline()

Example : assuming that function avg returns the filename for a figure showing the average value of a variable over a mask, create a table of links for average values of two variables over two masks, with thumbnail of images and displaying images when you mouse over it with ‘hover’ argument:

>>> t=table_lines(avg,['tas','tos'],['land','sea'],thumbnail=40,hover='60x80')

climaf.chtml.vspace(nb=1)

clist : tell what’s in cache, and much more

climaf.cache.clist(size='', age='', access=0, pattern='', not_pattern='', usage=False, count=False, remove=False, CRS=False, special=False)

Internal function used by its front-ends : cls(), crm(), cdu(), cwc()

List the content of CliMAF cache according to some search criteria and operate possibly an action (usage, count or remove) on this list.

Please consider the cost and benefit of first updating CliMAF cache index (by scanning files on disk) using csync()

Parameters:

• size (string, optional) –

  n[ckMG] Search files using more than n units of disk space, rounding up. The following suffixes can be used:

  • ”c” for bytes (default)
  • ”k” for Kilobytes (units of 1,024 bytes)
  • ”M” for Megabytes (units of 1,048,576 bytes)
  • ”G” for Gigabytes (units of 1,073,741,824 bytes)

• age (string, optional) –

  Number of 24h periods. Search files which status was last changed n*24 hours ago. Any fractional part is ignored, so to match age=’+1’, a file has to have been changed at least two days ago. Numeric arguments can be specified as:

  • +n for greater than n
  • -n for less than n,
  • n for exactly n.

• access (int, optional) – n Search files which were last accessed more than n*24 hours ago. Any fractional part is ignored, so to match access=’1’, a file has to have been accessed at least two days ago.
• pattern (string, optional) – Scan through crs and filenames looking for the first location where the regular expression pattern produces a match.
• not_pattern (string, optional) – Scan through crs and filenames looking for the location where the regular expression not_pattern does not produce a match.
• usage (bool, optional) – Estimate found files space usage, for each found file and total size. If count is True, estimate only found files total space usage.
• count (bool, optional) – Return the number of found files. If CRS is True, also return crs of found files.
• remove (bool, optional) – Remove the found files. This argument is exclusive.
• CRS (bool, optional) – if True, print also CRS expression. Useful only if count is True.

Returns:

The dictionary corresponding to the request and associated action ( or dictionary

of CliMAF cache index if no argument is provided)

Example to search files using more than 3M of disk space, which status

was last changed more than 15 days ago and containing the pattern ‘1980-1981’ either in crs or filename. For found files, we want to estimate only found files total space usage:

>>> clist(size='3M', age='+15', pattern= '1980-1981', usage=True, count=True)

cls : tell what’s in cache

climaf.cache.cls(**kwargs)

List CliMAF cache objects. Synonym to clist(). See clist()

crm : remove some files from cache

climaf.cache.crm(**kwargs)

Remove the cache files found by ‘clist()’ when using same arguments. See clist()

Example to remove files using more than 3M of disk space, which status was last changed more than 15 days ago and containing the pattern ‘1980-1981’ either in crs or filename:

>>> crm(size='3M', age='+15', pattern='1980-1981')

cdu : disk cache usage

climaf.cache.cdu(**kwargs)

Report disk usage, for files matching some criteria, as specified for clist(). With count=True, report only total disk usage.

Example to search files using more than 3M of disk space, which status was last changed more than 15 days ago and containing the pattern ‘1980-1981’ either in crs or filename. For found files, we want to estimate only found files total space usage:

>>> cdu(size='3M', age='+15', pattern= '1980-1981', count=True)

cwc : count some files in cache

climaf.cache.cwc(**kwargs)

Report number of cache files matching some criteria, as specified for clist(). If CRS is True, also return CRS expression of found files.

Example to return the number and crs associated of files using more than 3M of disk space, which status was last changed more than 15 days ago and containing the pattern ‘1980-1981’ either in crs or filename:

>>> cwc(size='3M', age='+15', pattern= '1980-1981', CRS=True)

craz : reset cache

climaf.cache.craz(force=False, hideError=False)

Clear CliMAF cache : erase existing files content, reset in-memory index

Parameters:

• force (bool) – should we erase also all ‘protected’ files
• hideError (bool) – if True, will not warn for non existing cache

cdrop : erase a result’s file

climaf.cache.cdrop(obj, rm=True, force=False)

Deletes the cached file for a CliMAF object, if it exists

Parameters:

• obj (cobject or string) – object to delete, or its string representation (CRS)
• force (bool) – should we delete the object even if it is ‘protected’
• rm (bool) – for advanced use only; should we actually delete (rm) the file, or just forget it in CliMAF cache index

Returns:

None if object does not exists, False if failing to delete, True if OK

Example

>>> dg=ds(project='example', simulation='AMIPV6ALB2G', variable='tas', period='1980-1981')
>>> f=cfile(dg)
>>> os.system('ls -al '+f)
>>> cdrop(dg)

cprotect : protect a result’s file against cdrop and craz

climaf.cache.cprotect(obj, stop=False)

Protects the cache file for a given object (or stops protection with arg ‘stop=True’).

In order to erase it, argument ‘force=True’ must then be used with function craz() or cdrop()

ccost : provide compute cost for an object

climaf.cache.ccost(cobject)

Provide the compute costs (elapsed time) for an object

Returned value is of class cost, which has fields tc (for total_cost) and lc (for last_op_cost):

• tc is the sum of all compute operations that lead to the object
• lc is the cost for the last operation (top level)

When a single operation provides multiple outputs, all outputs are charged with the cost of that operation (cost is not shared). This may lead to an overestimated cost when such outputs are re-used together in further operations