CachedPath

MacroModule

genre	File
author	Jan-Martin Kuhnigk
package	FMEstable/ReleaseMeVis
definition	CachedPath.def
see also	ImageCache, HTTPFileDownload, DirectoryDownloader
keywords	cache, buffer, remote, store, directory, folder, path, file, image, large

Purpose

Caches slow or unreliable (typically) remote folders or files in the local temp folder for faster and in doubt offline use.

One main use case is as a temporary network drop-in (e.g. for debugging), but it can also be used in applications.

Usage

Place the module beside your file loading module in the network. Then copy/set the source data path to Source Path and connect Result Path to the loader module’s input file (or folder) name field.

If the source folder is configured for caching (indicated by <i>Cache Support:</i>, you can edit your system’s configuration by adding/editing the environment variable DirCache_RootPaths), it will be cached on Update.

If the source is not located in a pre-configured source path for caching, but you still want to cache it, use the Force caching field.

Press Update to update Result Path and possibly synchronize the cache.

Check Result Path (or Is Cache Path) to check if the result path comes from the cache or the original source. Also, check Has Valid Output and Sync. Successful for possible errors.

Details

Caching Strategy

• If Source Path points to a folder, or Required Resources is set to ParentDirectory, the folder in question will be cached fully recursively.

• If Source Path it located below an existing cache root path, but that root path cannot be accessed or is empty (e.g. because it is currently not mounted), and Sync. Successful will be set to False and the following happens:

  • If a version of Source Path is found in the cache, it will be used, Has Valid Output will be set to True

  • If a version of Source Path is not found in the cache, neither the source nor the cache can provide the data, so the output will be invalid (and Has Valid Output will be set to False).

• If Force caching is enabled but Source Path also matches a pre-configured cache, the latter will take precedence (no new ad-hoc cache is created)

Cache Clearing

• As the purpose of the module is to prevent remote copying repeatedly from a slow remote source even/especially across multiple MeVisLab startups, the module currently does no automatic cache clearing

• However, when the module is up-to-date with Is Cache Path == True, it is possible to explicitly remove the cached resources from the cache via the Clear Cached Resources trigger.

• Also, you can always remove the cached resources manually from your hard disk while the module is not up-to-date

Windows

Default Panel

Parameter Fields

Field Index

[]: Trigger	Is Cache Path: Bool	Sync. Successful: Bool
<i>Cache Support:</i>: Bool	On Input Change Behavior: Enum	Update: Trigger
Clear: Trigger	Prune target: Bool
Clear Cached Resources: Trigger	Required Resources: Enum
doNotClearOnFailedUpdate: Bool	Result Path: String
Force caching: Bool	Source Path: String
Has Valid Output: Bool	Status Code: Enum
inVerboseCaching: Bool	Status Message: String

Visible Fields

Update

name: update, type: Trigger

Initiates update of all output field values.

Clear

name: clear, type: Trigger

Clears all output field values to a clean initial state.

On Input Change Behavior

name: onInputChangeBehavior, type: Enum, default: Clear, deprecated name: shouldAutoUpdate,shouldUpdateAutomatically

Declares how the module should react if a value of an input field changes.

Values:

Title	Name	Deprecated Name
​Update	​Update	​TRUE
​Clear	​Clear	​FALSE

[]

name: updateDone, type: Trigger, persistent: no

Notifies that an update was performed (Check status interface fields to identify success or failure).

Has Valid Output

name: hasValidOutput, type: Bool, persistent: no

Indicates validity of output field values (success of computation).

Status Code

name: statusCode, type: Enum, persistent: no

Reflects module’s status (successful or failed computations) as one of some predefined enumeration values.

Values:

Title	Name
​Ok	​Ok
​Invalid input object	​Invalid input object
​Invalid input parameter	​Invalid input parameter
​Internal error	​Internal error

Status Message

name: statusMessage, type: String, persistent: no

Gives additional, detailed information about status code as human-readable message.

Required Resources

name: inRequiredResources, type: Enum, default: OnlyPath

Defines the resources to be cached, in relation to Source Path.

Values:

Title	Name	Description
​Only Path	​OnlyPath	​Only the path (folder or file) referenced by Source Path will be cached.
​Parent Directory	​ParentDirectory	​Also place the directory containing Source Path into the cache. This can be useful when the loader just needs a single file (e.g. an image), but other parts of the network need other files in the same folder (e.g. a mask for the image, or a CSO), and you don’t want to add another CachedPath module for each of those.

Source Path

name: inSourcePath, type: String

File system path to cache

Force caching

name: inForceCaching, type: Bool, default: FALSE

Cache the resources locally even if the source is not configured for caching in the system cache.

If the field is set to TRUE but there is a pre-configured cache matching the source, the pre-configured cache will take precedence.

Prune target

name: inPruneTargetFolder, type: Bool, default: TRUE

When caching a directory, any extra files in the target folder will be removed (if the source folder can be accessed and checked for its contents!)

Clear Cached Resources

name: clearCachedResources, type: Trigger

Clear the cached path associated with Source Path (if any).

Result Path

name: outPath, type: String, persistent: no

Result path, i.e. the location of the cached version of the source path for cacheable paths, or the value of Source Path otherwise.

<i>Cache Support:</i>

name: outIsSourceSupportedBySystemCache, type: Bool, persistent: no

Indicates if the source folder is configured for caching.

You can edit your system’s configuration by setting/editing the environment variable DirCache_RootPaths. Alternatively you can force caching even for non-preconfigured paths via the Force caching field.

Is Cache Path

name: outResultPathIsCachePath, type: Bool, persistent: no

Indicates if Result Path is a cached path or not (in which case it is just Source Path).

Sync. Successful

name: outCacheSyncSuccessful, type: Bool, persistent: no

Indicates if a synchronization was successfully performed. The main reason for failures is that the data source is not available (e.g. not mounted). If such cases, the cached copy will be used (if available).

Hidden Fields

doNotClearOnFailedUpdate

name: doNotClearOnFailedUpdate, type: Bool, persistent: no

Prevents automated clear after update failed. This does not affect status fields. It enables the developer to analyze module’s state after failure.

inVerboseCaching

name: inVerboseCaching, type: Bool, default: TRUE

If caching is attempted for the current source, log caching info to the console.