Functions for handling the cache

clist : tell what’s in cache, and much more

climaf.cache.clist(size='', age='', access=0, pattern='', not_pattern='', usage=False, count=False, remove=False, CRS=False, special=False)[source]

Internal function used by its front-ends : cls(), crm(), cdu(), cwc()

List the content of CliMAF cache according to some search criteria and operate possibly an action (usage, count or remove) on this list.

Please consider the cost and benefit of first updating CliMAF cache index (by scanning files on disk) using csync()

Parameters:
  • size (string, optional) –

    n[ckMG] Search files using more than n units of disk space, rounding up. The following suffixes can be used:

    • ”c” for bytes (default)
    • ”k” for Kilobytes (units of 1,024 bytes)
    • ”M” for Megabytes (units of 1,048,576 bytes)
    • ”G” for Gigabytes (units of 1,073,741,824 bytes)
  • age (string, optional) –

    Number of 24h periods. Search files which status was last changed n*24 hours ago. Any fractional part is ignored, so to match age=’+1’, a file has to have been changed at least two days ago. Numeric arguments can be specified as:

    • +n for greater than n
    • -n for less than n,
    • n for exactly n.
  • access (int, optional) – n Search files which were last accessed more than n*24 hours ago. Any fractional part is ignored, so to match access=’1’, a file has to have been accessed at least two days ago.
  • pattern (string, optional) – Scan through crs and filenames looking for the first location where the regular expression pattern produces a match.
  • not_pattern (string, optional) – Scan through crs and filenames looking for the location where the regular expression not_pattern does not produce a match.
  • usage (bool, optional) – Estimate found files space usage, for each found file and total size. If count is True, estimate only found files total space usage.
  • count (bool, optional) – Return the number of found files. If CRS is True, also return crs of found files.
  • remove (bool, optional) – Remove the found files. This argument is exclusive.
  • CRS (bool, optional) – if True, print also CRS expression. Useful only if count is True.
Returns:

The dictionary corresponding to the request and associated action ( or dictionary

of CliMAF cache index if no argument is provided)

Example to search files using more than 3M of disk space, which status

was last changed more than 15 days ago and containing the pattern ‘1980-1981’ either in crs or filename. For found files, we want to estimate only found files total space usage:

>>> clist(size='3M', age='+15', pattern= '1980-1981', usage=True, count=True)

cls : tell what’s in cache

climaf.cache.cls(**kwargs)[source]

List CliMAF cache objects. Synonym to clist(). See clist()

crm : remove some files from cache

climaf.cache.crm(**kwargs)[source]

Remove the cache files found by ‘clist()’ when using same arguments. See clist()

Example to remove files using more than 3M of disk space, which status was last changed more than 15 days ago and containing the pattern ‘1980-1981’ either in crs or filename:

>>> crm(size='3M', age='+15', pattern='1980-1981')

cdu : disk cache usage

climaf.cache.cdu(**kwargs)[source]

Report disk usage, for files matching some criteria, as specified for clist(). With count=True, report only total disk usage.

Example to search files using more than 3M of disk space, which status was last changed more than 15 days ago and containing the pattern ‘1980-1981’ either in crs or filename. For found files, we want to estimate only found files total space usage:

>>> cdu(size='3M', age='+15', pattern= '1980-1981', count=True)

cwc : count some files in cache

climaf.cache.cwc(**kwargs)[source]

Report number of cache files matching some criteria, as specified for clist(). If CRS is True, also return CRS expression of found files.

Example to return the number and crs associated of files using more than 3M of disk space, which status was last changed more than 15 days ago and containing the pattern ‘1980-1981’ either in crs or filename:

>>> cwc(size='3M', age='+15', pattern= '1980-1981', CRS=True)

craz : reset cache

climaf.cache.craz(force=False, hideError=False)[source]

Clear CliMAF cache : erase existing files content, reset in-memory index

Parameters:
  • force (bool) – should we erase also all ‘protected’ files
  • hideError (bool) – if True, will not warn for non existing cache

cdrop : erase a result’s file

climaf.cache.cdrop(obj, rm=True, force=False)[source]

Deletes the cached file for a CliMAF object, if it exists

Parameters:
  • obj (cobject or string) – object to delete, or its string representation (CRS)
  • force (bool) – should we delete the object even if it is ‘protected’
  • rm (bool) – for advanced use only; should we actually delete (rm) the file, or just forget it in CliMAF cache index
Returns:

None if object does not exists, False if failing to delete, True if OK

Example

>>> dg=ds(project='example', simulation='AMIPV6ALB2G', variable='tas', period='1980-1981')
>>> f=cfile(dg)
>>> os.system('ls -al '+f)
>>> cdrop(dg)

cprotect : protect a result’s file against cdrop and craz

climaf.cache.cprotect(obj, stop=False)[source]

Protects the cache file for a given object (or stops protection with arg ‘stop=True’).

In order to erase it, argument ‘force=True’ must then be used with function craz() or cdrop()

ccost : provide compute cost for an object

climaf.cache.ccost(cobject)[source]

Provide the compute costs (elapsed time) for an object

Returned value is of class cost, which has fields tc (for total_cost) and lc (for last_op_cost):

  • tc is the sum of all compute operations that lead to the object
  • lc is the cost for the last operation (top level)

When a single operation provides multiple outputs, all outputs are charged with the cost of that operation (cost is not shared). This may lead to an overestimated cost when such outputs are re-used together in further operations