Functions for handling the cache¶
clist : tell what’s in cache, and much more¶
-
climaf.cache.
clist
(size='', age='', access=0, pattern='', not_pattern='', usage=False, count=False, remove=False, CRS=False, special=False)[source] Internal function used by its front-ends :
cls()
,crm()
,cdu()
,cwc()
List the content of CliMAF cache according to some search criteria and operate possibly an action (usage, count or remove) on this list.
Please consider the cost and benefit of first updating CliMAF cache index (by scanning files on disk) using
csync()
Parameters: - size (string, optional) –
n[ckMG] Search files using more than n units of disk space, rounding up. The following suffixes can be used:
- ”c” for bytes (default)
- ”k” for Kilobytes (units of 1,024 bytes)
- ”M” for Megabytes (units of 1,048,576 bytes)
- ”G” for Gigabytes (units of 1,073,741,824 bytes)
- age (string, optional) –
Number of 24h periods. Search files which status was last changed n*24 hours ago. Any fractional part is ignored, so to match age=’+1’, a file has to have been changed at least two days ago. Numeric arguments can be specified as:
- +n for greater than n
- -n for less than n,
- n for exactly n.
- access (int, optional) – n Search files which were last accessed more than n*24 hours ago. Any fractional part is ignored, so to match access=’1’, a file has to have been accessed at least two days ago.
- pattern (string, optional) – Scan through crs and filenames looking for the first location where the regular expression pattern produces a match.
- not_pattern (string, optional) – Scan through crs and filenames looking for the location where the regular expression not_pattern does not produce a match.
- usage (bool, optional) – Estimate found files space usage, for each found file and total size. If count is True, estimate only found files total space usage.
- count (bool, optional) – Return the number of found files. If CRS is True, also return crs of found files.
- remove (bool, optional) – Remove the found files. This argument is exclusive.
- CRS (bool, optional) – if True, print also CRS expression. Useful only if count is True.
Returns: - The dictionary corresponding to the request and associated action ( or dictionary
of CliMAF cache index if no argument is provided)
- Example to search files using more than 3M of disk space, which status
was last changed more than 15 days ago and containing the pattern ‘1980-1981’ either in crs or filename. For found files, we want to estimate only found files total space usage:
>>> clist(size='3M', age='+15', pattern= '1980-1981', usage=True, count=True)
- size (string, optional) –
cls : tell what’s in cache¶
crm : remove some files from cache¶
-
climaf.cache.
crm
(**kwargs)[source] Remove the cache files found by ‘clist()’ when using same arguments. See
clist()
Example to remove files using more than 3M of disk space, which status was last changed more than 15 days ago and containing the pattern ‘1980-1981’ either in crs or filename:
>>> crm(size='3M', age='+15', pattern='1980-1981')
cdu : disk cache usage¶
-
climaf.cache.
cdu
(**kwargs)[source] Report disk usage, for files matching some criteria, as specified for
clist()
. With count=True, report only total disk usage.Example to search files using more than 3M of disk space, which status was last changed more than 15 days ago and containing the pattern ‘1980-1981’ either in crs or filename. For found files, we want to estimate only found files total space usage:
>>> cdu(size='3M', age='+15', pattern= '1980-1981', count=True)
cwc : count some files in cache¶
-
climaf.cache.
cwc
(**kwargs)[source] Report number of cache files matching some criteria, as specified for
clist()
. If CRS is True, also return CRS expression of found files.Example to return the number and crs associated of files using more than 3M of disk space, which status was last changed more than 15 days ago and containing the pattern ‘1980-1981’ either in crs or filename:
>>> cwc(size='3M', age='+15', pattern= '1980-1981', CRS=True)
craz : reset cache¶
-
climaf.cache.
craz
(force=False, hideError=False)[source] Clear CliMAF cache : erase existing files content, reset in-memory index
Parameters: - force (bool) – should we erase also all ‘protected’ files
- hideError (bool) – if True, will not warn for non existing cache
cdrop : erase a result’s file¶
-
climaf.cache.
cdrop
(obj, rm=True, force=False)[source] Deletes the cached file for a CliMAF object, if it exists
Parameters: - obj (cobject or string) – object to delete, or its string representation (CRS)
- force (bool) – should we delete the object even if it is ‘protected’
- rm (bool) – for advanced use only; should we actually delete (rm) the file, or just forget it in CliMAF cache index
Returns: None if object does not exists, False if failing to delete, True if OK
Example
>>> dg=ds(project='example', simulation='AMIPV6ALB2G', variable='tas', period='1980-1981') >>> f=cfile(dg) >>> os.system('ls -al '+f) >>> cdrop(dg)
cprotect : protect a result’s file against cdrop and craz¶
ccost : provide compute cost for an object¶
-
climaf.cache.
ccost
(cobject)[source] Provide the compute costs (elapsed time) for an object
Returned value is of class cost, which has fields tc (for total_cost) and lc (for last_op_cost):
- tc is the sum of all compute operations that lead to the object
- lc is the cost for the last operation (top level)
When a single operation provides multiple outputs, all outputs are charged with the cost of that operation (cost is not shared). This may lead to an overestimated cost when such outputs are re-used together in further operations