Purpose

MLImageFormatFileCache stores the content of a connected input image in a cache file and passes it to the output. Or, alternatively, it overwrites a specific region in an already existing image with new data with the Update Region button.

This module is essential for the composition, modification, and caching of very large volumes (see Processing Large Volumes in MeVisLab) which do not fit into memory or exceed the 2GB limit.

Accessing a cached image content and passing it to the output image connector is possible even if the input connection is disconnected or invalidated. Thus MLImageFormatFileCache is comparable to the MemoryCache module which caches the input image in memory.

MLImageFormatFileCache is especially useful to cache images which are too large to be stored in memory and therefore cannot be handled by the module MemoryCache.

See also the BitImageArithmetic module which can be configured to cache a BitImage and the general documentation about the MLImageFormat and MLDataCompressors.

Tips

There are some useful and different modes for which the MLImageFormatFileCache modules can be used.

  1. The first mode is to store a large image temporarily in a file instead of storing it in a MemoryCache module to reduce the memory usage of an application. Since output pages and the file system also cache output information, access times often as fast as if a MemoryCache module is used. Note that in many cases the application even becomes faster and more robust against out-of-memory problems with file caching, because more memory is available for other purposes.
  2. The second mode is useful to handle and compose very large images. It is recommended to do this with a specifically defined file name in the Temp File Prefix and enabled Preserve Cache File flag. This approach first creates a file written from the input image which already must have the final result extent. Often TestPattern is connected and Update stores it in the cache. Since pages with unique values are packed, those files are created and stored fast while the file remains very small if unique voxel values are defined as input image data. In the next step with the Update Region functionality subregions or slices can be written step by step into that file. Since everything works page based there is nearly no limitation about the file size. Even file sizes of tera byte sizes can be composed on 32 bit systems.
  3. This mode is similar to the previous one. It allows opening an existing .mlimage file with the Preserve Cache File flag enabled. This file then can be updated, partially overwritten or changed with the Update Region functionality. Especially if you want to compose super large files then consider creating the .mlimage file with the Create button in MLImageFormatSave before opening it in MLImageFormatFileCache. Although this only allows the creation of file contents with unique voxel values, it will be significantly faster for extreme sized images, because it does not need to calculate any data for the created pages. Afterwards regions in such an empty file can be overwritten step by step as described in the previous paragraph.
  4. Note that the default compressor - if available - is LZ4. This typically reduces the cached files while normally not reducing storage times. This, however, can lead to file fragmentation in cases when non-empty pages are overwritten and with less compression. This is not necessarily a problem, however, if this use case it heavily used it might be safer to disable compression.

Windows

Default Panel

../../../Modules/ML/MLImageFormat/mhelp/Images/Screenshots/MLImageFormatFileCache._default.png

Input Fields

input0

name: input0, type: Image

The module has one input which is optional.

Therefore an already cached file remains up to date when Update Mode is set to Off or Auto Update, even if the connection to the input is removed or the input is invalidated.

Output Fields

output0

name: output0, type: Image

The module has one image output which is invalid if no data is cached; otherwise it provides the internally cached image.

Parameter Fields

Field Index

boolHintVal6: Bool dblHintVisible8: Bool progress: Float
boolHintVal7: Bool dblHintVisible9: Bool Region Offset: IntVector6
boolHintVal8: Bool Full cache file name: String Remove: Trigger
boolHintVal9: Bool hintName0: String Status: String
boolHintVisible0: Bool hintName1: String Stop: Trigger
boolHintVisible1: Bool hintName2: String strHintVal6: String
boolHintVisible2: Bool hintName3: String strHintVal7: String
boolHintVisible3: Bool hintName4: String strHintVal8: String
boolHintVisible4: Bool hintName5: String strHintVal9: String
boolHintVisible5: Bool hintName6: String strHintVisible0: Bool
boolHintVisible6: Bool hintName7: String strHintVisible1: Bool
boolHintVisible7: Bool hintName8: String strHintVisible2: Bool
boolHintVisible8: Bool hintName9: String strHintVisible3: Bool
boolHintVisible9: Bool intHintVal6: Integer strHintVisible4: Bool
Cache Valid: Bool intHintVal7: Integer strHintVisible5: Bool
Clear: Trigger intHintVal8: Integer strHintVisible6: Bool
Compression: Enum intHintVal9: Integer strHintVisible7: Bool
dblHintVal6: Double intHintVisible0: Bool strHintVisible8: Bool
dblHintVal7: Double intHintVisible1: Bool strHintVisible9: Bool
dblHintVal8: Double intHintVisible2: Bool Suppress world position: Bool
dblHintVal9: Double intHintVisible3: Bool Tag List: String
dblHintVisible0: Bool intHintVisible4: Bool Temp File Prefix: String
dblHintVisible1: Bool intHintVisible5: Bool trueFileName: String
dblHintVisible2: Bool intHintVisible6: Bool Update: Trigger
dblHintVisible3: Bool intHintVisible7: Bool Update Mode: Enum
dblHintVisible4: Bool intHintVisible8: Bool Update Region: Trigger
dblHintVisible5: Bool intHintVisible9: Bool updateHintControls: Trigger
dblHintVisible6: Bool isSaving: Bool  
dblHintVisible7: Bool Preserve Cache File: Bool  

Visible Fields

Compression

name: compression, type: Enum, default: LZ4

Defines the compression algorithm. If available it usually defaults to a fast compressor such as LZ4. For possible side effects see Update Region and Tips section.

See MLDataCompressors for general information about data compressors.

Values:

Title Name
None None
MLCTComp2 MLCTComp2
Lzf LZF
Zlib ZLIB
Lzma LZMA
BZip2 BZip2
Lz4 LZ4

Update Mode

name: updateMode, type: Enum, default: AutoUpdate

Determines how changes of parameters of the the input image influence the currently cached image data and the output image.

Values:

Title Name Description
Off Off Changes of parameters or input image do not influence the cached image nor the output image.
Auto Update AutoUpdate Changes of parameters or of the input image update the cache and the input image.
Auto Clear AutoClear Changes of parameters or the input image clear the cached image and invalidate the output image.

Remove

name: remove, type: Trigger

If pressed, the cached image file is deleted (even in preserve file mode) and the output image is disabled.

Stop

name: stop, type: Trigger

If pressed, the file storing is terminated and the cached image invalidated.

Status

name: status, type: String, persistent: no

Shows information about the current cache state and/or the recent caching operation.

Tag List

name: tagList, type: String, persistent: no

Shows a list of tags in the file.

Region Offset

name: regionOffset, type: IntVector6, default: 0 0 0 0 0 0

Sets a 6D offset vector that is added to the position where an input region is written on Update Region.

If Suppress world position is off then the destination of the updated input region is calculated as follows: The origin of the input region in world coordinates is taken and the corresponding voxel coordinate in the cached image is determined. Then Region Offset is added and the input region is written at that position.

Enable Suppress world position if position calculation shall be performed in voxel coordinates.

Suppress world position

name: suppressWorldPos, type: Bool, default: FALSE

Determines whether world coordinates of an input region are used to determine the destination position in target image if Update Region is used. If this field is off then the world coordinates of the origin of the input image + Region Offset are used as target position for subimages written into already written files. If this field is on then only the position given by Region Offset will be used as target position for written subimages.

Cache Valid

name: valid, type: Bool, persistent: no

Indicates whether a file is cached

Preserve Cache File

name: preserveCacheFile, type: Bool, default: FALSE

If enabled then an existing cache file is used and modified, but never cleared. If false then a cache file is automatically cleared/removed.

Clear

name: clear, type: Trigger

Clears the cached image file or - in preserve file mode - only disables the output image.

Update

name: update, type: Trigger

In preserve file mode with a valid cache file this field loads the data from the cache file and passes it to the output. If the cache file does not exist or if preserved file mode is off then it stores the input image in the cache file. Note: To update the content of an existing preserved cache file use Remove to delete the file and then Update to save new data.

Update Region

name: updateRegion, type: Trigger

If pressed, the cached file is overwritten with data from the input image at its world or voxel location, dependent on Suppress world position.

Note:

If non empty and compressed pages are overwritten with new data which could not be compressed with the same or better ration this can lead to fragmentation of the cache file. If this is used heavily and file sizes must be limited then it might be safer to disable compression. This, however, is rarely a serious problem. In nearly all cases using a fast compressor such as LZ4 is more beneficial than using no compression.

Temp File Prefix

name: tmpFileNamePrefix, type: String

Sets a preserved cache file name or prefix for automatic file name while using file buffering.

Full cache file name

name: outFullCacheFilePath, type: String, persistent: no

This field is read-only and shows the expanded path to the currently used cache file if is any. If not then it is set to an empty string. This path is usually set before enabling the Cache Valid flag and after disabling it.

Hidden Fields

updateHintControls

name: updateHintControls, type: Trigger

hintName0

name: hintName0, type: String, persistent: no

dblHintVisible0

name: dblHintVisible0, type: Bool, persistent: no

intHintVisible0

name: intHintVisible0, type: Bool, persistent: no

strHintVisible0

name: strHintVisible0, type: Bool, persistent: no

boolHintVisible0

name: boolHintVisible0, type: Bool, persistent: no

hintName1

name: hintName1, type: String, persistent: no

dblHintVisible1

name: dblHintVisible1, type: Bool, persistent: no

intHintVisible1

name: intHintVisible1, type: Bool, persistent: no

strHintVisible1

name: strHintVisible1, type: Bool, persistent: no

boolHintVisible1

name: boolHintVisible1, type: Bool, persistent: no

hintName2

name: hintName2, type: String, persistent: no

dblHintVisible2

name: dblHintVisible2, type: Bool, persistent: no

intHintVisible2

name: intHintVisible2, type: Bool, persistent: no

strHintVisible2

name: strHintVisible2, type: Bool, persistent: no

boolHintVisible2

name: boolHintVisible2, type: Bool, persistent: no

hintName3

name: hintName3, type: String, persistent: no

dblHintVisible3

name: dblHintVisible3, type: Bool, persistent: no

intHintVisible3

name: intHintVisible3, type: Bool, persistent: no

strHintVisible3

name: strHintVisible3, type: Bool, persistent: no

boolHintVisible3

name: boolHintVisible3, type: Bool, persistent: no

hintName4

name: hintName4, type: String, persistent: no

dblHintVisible4

name: dblHintVisible4, type: Bool, persistent: no

intHintVisible4

name: intHintVisible4, type: Bool, persistent: no

strHintVisible4

name: strHintVisible4, type: Bool, persistent: no

boolHintVisible4

name: boolHintVisible4, type: Bool, persistent: no

hintName5

name: hintName5, type: String, persistent: no

dblHintVisible5

name: dblHintVisible5, type: Bool, persistent: no

intHintVisible5

name: intHintVisible5, type: Bool, persistent: no

strHintVisible5

name: strHintVisible5, type: Bool, persistent: no

boolHintVisible5

name: boolHintVisible5, type: Bool, persistent: no

hintName6

name: hintName6, type: String, persistent: no

dblHintVal6

name: dblHintVal6, type: Double, default: 0

intHintVal6

name: intHintVal6, type: Integer, default: 0

strHintVal6

name: strHintVal6, type: String

boolHintVal6

name: boolHintVal6, type: Bool, default: FALSE

dblHintVisible6

name: dblHintVisible6, type: Bool, persistent: no

intHintVisible6

name: intHintVisible6, type: Bool, persistent: no

strHintVisible6

name: strHintVisible6, type: Bool, persistent: no

boolHintVisible6

name: boolHintVisible6, type: Bool, persistent: no

hintName7

name: hintName7, type: String, persistent: no

dblHintVal7

name: dblHintVal7, type: Double, default: 0

intHintVal7

name: intHintVal7, type: Integer, default: 0

strHintVal7

name: strHintVal7, type: String

boolHintVal7

name: boolHintVal7, type: Bool, default: FALSE

dblHintVisible7

name: dblHintVisible7, type: Bool, persistent: no

intHintVisible7

name: intHintVisible7, type: Bool, persistent: no

strHintVisible7

name: strHintVisible7, type: Bool, persistent: no

boolHintVisible7

name: boolHintVisible7, type: Bool, persistent: no

hintName8

name: hintName8, type: String, persistent: no

dblHintVal8

name: dblHintVal8, type: Double, default: 0

intHintVal8

name: intHintVal8, type: Integer, default: 0

strHintVal8

name: strHintVal8, type: String

boolHintVal8

name: boolHintVal8, type: Bool, default: FALSE

dblHintVisible8

name: dblHintVisible8, type: Bool, persistent: no

intHintVisible8

name: intHintVisible8, type: Bool, persistent: no

strHintVisible8

name: strHintVisible8, type: Bool, persistent: no

boolHintVisible8

name: boolHintVisible8, type: Bool, persistent: no

hintName9

name: hintName9, type: String, persistent: no

dblHintVal9

name: dblHintVal9, type: Double, default: 0

intHintVal9

name: intHintVal9, type: Integer, default: 0

strHintVal9

name: strHintVal9, type: String

boolHintVal9

name: boolHintVal9, type: Bool, default: FALSE

dblHintVisible9

name: dblHintVisible9, type: Bool, persistent: no

intHintVisible9

name: intHintVisible9, type: Bool, persistent: no

strHintVisible9

name: strHintVisible9, type: Bool, persistent: no

boolHintVisible9

name: boolHintVisible9, type: Bool, persistent: no

trueFileName

name: trueFileName, type: String, persistent: no, deprecated name: autoFileName

progress

name: progress, type: Float, persistent: no

Shows the amount of stored image data in percent.

isSaving

name: isSaving, type: Bool, persistent: no