ApplyTileProcessorPageWise

MLModule

genre	ML_Inference_PatchBased
authors	Jan-Martin Kuhnigk, Jan Hendrik Moltz
package	FMEstable/ReleaseMeVis
dll	MLApplyTileProcessorPageWise
definition	ApplyTileProcessorPageWise.def
see also	ProcessTiles, TensorFlowServingTileProcessor, SetTileProcessorProperties, AutoProposeTileProcessorProperties, ApplyTileProcessor
keywords	classific, prediction, inference, machine, deep, learning, deeplearning, patch, batch, RemoteVoxelClassification, TensorFlowVoxelClassification, ApplyTileClassifier

Purpose

Like ProcessTiles, it applies a C++ TileProcessor demand driven and a page-by-page to the input image. In addition, the user can modify/add (more) proposed tile properties (so it’s basically a combination of SetTileProcessorProperties and ProcessTiles.

It can also be used for “whole image” processing (see Tips section for more info), but usually, you may want to use ApplyTileProcessor instead for such cases.

If you need more than a single input or output, you also need to use the module ApplyTileProcessor instead.

Usage

Connect an image and a processor, make sure patch size, padding and the other tile properties are set correctly for the connected processor, and press update.

Details

Tile Processor Properties

Detailed Property Information

For detailed information on the meaning of the individual properties, please check out the TileProcessorProperties Schema Documentation.

Proposed vs. Custom vs. Used Properties

Usually, the input comes with proposed tile properties (output tile size, padding, fill mode, …). However, each of the proposals can be overwritten (at your own risk) here via the “inUseCustom…” and “inCustom…” fields, e.g. Use, Padding.

The final choices used are documented in the outUsed… fields (e.g. Used Padding).

Note that if the input is well parameterized, the only property you are likely to adapt will be Custom Output Tile Size in order to tune performance.

Tile Sizes

All outputTileSize fields refer to output tile sizes without padding, i. e. when processTile( outputTile, inputTile ) is called for the connected processor, the following applies if no input channel dimension (multi-channel input data) is used:

• outputTile has size Used Output Tile Size.

• inputTile has size Used Output Tile Size * Used Stride + 2 * Used Padding (padded symmetrically around the outputTile region)

Note that:

• strides may also be fractional, e.g. a stride of 0.5 would correspond to a factor 2 upsampling. However, any parameterization will be rejected that would result in non-integer (w.r.t. a small epsilon) input tile sizes.

• not all output tile sizes may be supported by all processors. Thus, the used output tile size shown in Used Output Tile Size may differ from the input as the processor may have “snapped” it to the closest supported tile size greater or equal to the specified one.

Channel Dimensions

Up to two channel dimensions can be used for multi-dimensional input values (e.g. RGB) and output values (e.g. multiple labels, additional result data such as dropout results). The user can specify which dimension(s) in input and output images shall be mapped to/from the processor’s channel dimensions.

Regarding the tile size computation above: The number of output channels must be specified in the output tile sizes channel (“Ch”/”Ch1”/”Ch2”) dimensions. If an input channel dimension (multi-channel input data) is used, the input tile size is increased to full extent in the “external” channel dimension (so you cannot/don’t have to specify the expected number of input channels).

Batch Dimension

Using a batch size > 1 (i. e. using tiles with more than one patches) can be helpful if the processor is more efficient when provided with multiple patches per tile, e.g. if the GPU is used for a processor supporting a batch dimension. To change the batch size, adapt the output tile size’s extent in the batch (“Bt”) dimension.

Filling

Any input tile regions located outside of the valid (reference) input image region are filled according to Used Fill Mode (and Used Fill Value, if FillValue was chosen as fill mode).

NOTE: In Reflect fill mode, only the values still within padding range from the valid (reference) input image region are guaranteed to have the correct value (input tiles may be larger)! However, that should not be a true limitation, since padding should always be chosen large enough to comprise everything needed for output tile computation.

Processing ROI

A processing ROI (cf. Restrict output to ROI, [VC]) can be set to restrict processing to a sub-region of the input image, which is mostly useful for speeding up computation. See ProcessTiles documentation (Details section) for more info.

Sharing a TileProcessor between Multiple Modules

You can use two ProcessTiles or ApplyTileProcessor modules on the same tile processor if you take care to Clear the first one before calling Update on the second instance. Having two or more user modules doing interleaving processBatch calls on the same TileProcessor is currently not supported, the first one ‘updating’ locks the TileProcessor until it is cleared.

This limitation is due to the (allowed) side effects of prepareProcessBatch() calls and could be avoided if necessary, however with some implementation effort.

Debugging/Profiling of Individual Batch Processing Requests

Set the environment variable TILEPROCESSING_ENABLE_REQUEST_DEBUGGING to a non “0” value to enable sending a unique request ID string with each individual tile request. This simplifies tracking for debugging and profiling in the inference providers (i.e. tile processors).

Interaction

Unlike other “AlgorithmModules”, the module updates most of its output parameters also if On Input Change Behavior is set to ‘clear’ (assuming a valid processor is connected) to make its usage easier. Most importantly, outParameterInfo and the “outUsed…” fields are always adapted to the current settings.

However, the processing is of course only started on update.

Tips

Using ApplyTileProcessor for Image Processing

The stride can actually be used to implement an image processing instead of a “classic” localized tile processing. The trick is to ensure, that the stride is always identical to the input image extent. As the connected processor naturally does not know the image extent, you have to override it’s proposed stride by enabling Use and setting Stride to the input image extent. Thus, the input tile will comprise the entire input image, and the output tile only a single voxel (spanning the entire image region).

NOTE: As the stride is to be specified in processor dimension order which may differ from the input image’s one, make sure to sort the input image extents accordingly before applying them to Stride!

It would be best to use a dedicated wrapper macro named e.g. “ApplyImageProcessor” for this purpose, which could take care of such nasty details. Please check in your MeVisLab copy if it is already available!

Windows

Default Panel

Input Fields

inTileProcessor

name: inTileProcessor, type: TileProcessor/TileProcessorContainer(MLBase), deprecated name: inTileClassifier

Used to provide an input processor (implementing the ml::TileProcessor interface) and optionally proposed tile properties. If the tile processor comes with tile property proposals, the input must be BaseList (which serves as TileProcessorContainer) containing both the TileProcessor and a TileProperiesSource.

See SetTileProcessorProperties, SetTileProcessorPropertiesFromJson and AutoProposeTileProcessorProperties for more info on how to generate proposed tile properties.

Output Fields

outParameterInfo

name: outParameterInfo, type: MLBase

ParameterInfo for the connected processor as a python object.

outParameterInfoCpp (hidden)

name: outParameterInfoCpp, type: ParameterInfo(MLBase)

ParameterInfo for the connected processor as a C++ object. Hidden since C++ parameter info support is limited (mostly, there is a python wrapper that allows conversion to python, but you can use outParameterInfo instead).

For accessing this object via scripting, see the Scripting Reference: MLParameterInfoWrapper.

Parameter Fields

Field Index

[]: Trigger	inCustomDimensions: String	Update: Trigger
[VC]: String	inCustomOutput: String	Use (inUseCustomOutputTileSize): Bool
Available Outputs: String	inOutputSelectionMode: Enum	Use (inUseCustomPadding): Bool
BATCH (inCustomExternalDimensionForBatch): Enum	inUseCustomDimensions: Bool	Use (inUseCustomStride): Bool
BATCH (outUsedExternalDimensionForBatch): Enum	On Input Change Behavior: Enum	Use (inUseCustomFillMode): Bool
CHANNEL2 (inCustomExternalDimensionForChannel2): Enum	outMaxNumModuleInputs: Integer	Use (inUseCustomFillValue): Bool
CHANNEL2 (outUsedExternalDimensionForChannel2): Enum	outMaxNumModuleOutputs: Integer	Use (inUseCustomExternalDimensionForChannel1): Bool
Clear: Trigger	outUsedDimensions: String	Use (inUseCustomExternalDimensionForChannel2): Bool
Custom Dimension for CHANNEL1: Enum	Padding: IntVector6	Use (inUseCustomExternalDimensionForBatch): Bool
Custom Output Tile Size: IntVector6	Restrict output to ROI: Bool	Used Dimensions - CHANNEL1: Enum
doNotClearOnFailedUpdate: Bool	Selected Output: String	Used Fill Mode: Enum
Expected Inputs: String	Snap to valid: Bool	Used Fill Value: Double
Fill Mode: Enum	Status Code: Enum	Used Output Tile Size: IntVector6
Fill Value: Double	Status Message: String	Used Padding: IntVector6
Has Valid Output: Bool	Stride: Vector6	Used Stride: Vector6

Visible Fields

Update

name: update, type: Trigger

Initiates update of all output field values.

Clear

name: clear, type: Trigger

Clears all output field values to a clean initial state.

On Input Change Behavior

name: onInputChangeBehavior, type: Enum, default: Clear, deprecated name: shouldAutoUpdate,shouldUpdateAutomatically

Declares how the module should react if a value of an input field changes.

Values:

Title	Name	Deprecated Name
​Update	​Update	​TRUE
​Clear	​Clear	​FALSE

Status Code

name: statusCode, type: Enum, persistent: no

Reflects module’s status (successful or failed computations) as one of some predefined enumeration values.

Values:

Title	Name
​Ok	​Ok
​Invalid input object	​Invalid input object
​Invalid input parameter	​Invalid input parameter
​Internal error	​Internal error

Status Message

name: statusMessage, type: String, persistent: no

Gives additional, detailed information about status code as human-readable message.

Has Valid Output

name: hasValidOutput, type: Bool, persistent: no

Indicates validity of output field values (success of computation).

[]

name: updateDone, type: Trigger, persistent: no

Notifies that an update was performed (Check status interface fields to identify success or failure).

Expected Inputs

name: availableInputs, type: String, persistent: no

Comma-separated list of all inputs declared by the current TileProcessor

Available Outputs

name: availableOutputs, type: String, persistent: no

Comma-separated list of all outputs declared by the current TileProcessor

Restrict output to ROI

name: inUseProcessingRoi, type: Bool, default: FALSE, deprecated name: inUseClassificationRoi

If enabled, you can restrict processing (and hence the output image) to a rectangular Region of Interest defined by [VC].

[VC]

name: inProcessingRoi_voxel, type: String, default: 0 0 0 0 0 0 -1 -1 -1 -1 -1 -1, deprecated name: inClassificationRoi\_voxel

If Restrict output to ROI is enabled, this box defines the processing (and hence output image) region in voxel coordinates.

Snap to valid

name: inEnableOutputTileSizeSnapping, type: Bool, default: TRUE

If enabled (default) the module snaps the output tile size to a valid value greater or equal to the proposed one, if this is supported by the processor. Disabling the flag is only for those rare cases in which the processor errs on what he supports. Of course, it is better to fix the proposed properties than to disable this flag.

See modules SetTileProcessorProperties and SetTileProcessorPropertiesFromJson on how to adapt the proposed tile properties that snapping will always work correctly for you processor.

Use (inUseCustomOutputTileSize)

name: inUseCustomOutputTileSize, type: Bool, default: FALSE, deprecated name: inUseCustomTileSize

If enabled, Custom Output Tile Size is used instead of the proposition from the connected processor.

Custom Output Tile Size

name: inCustomOutputTileSize, type: IntVector6, default: 128 128 1 1 1 1, deprecated name: inCustomTileSize

If Use is enabled, this overrides the processor’s proposal for the output tile size used for processing. If you have a multi-channel result, this must include the output channel extent in the CHANNEL dimension specified in outUsedDimensions. Any ‘0’ entry will be automatically mapped to the corresponding input image extent.

The “0 to full extent mapping” makes it also easier to use the module for whole image classification, but it is still recommended to use ApplyTileProcessor for that purpose.

See Details section for details.

Used Output Tile Size

name: outUsedOutputTileSize, type: IntVector6, persistent: no, deprecated name: outUsedTileSize

Tile size (without padding) actually used. See ‘Details’ section for details.

Selected Output

name: outUsedOutput, type: String, persistent: no, deprecated name: outUsedModelOutput

Name of the actually used model output.

Use (inUseCustomPadding)

name: inUseCustomPadding, type: Bool, default: FALSE

If enabled, Padding is used instead of the proposition from the connected processor.

Padding

name: inCustomPadding, type: IntVector6, default: 0 0 0 0 0 0

If Use is enabled, this overrides the processor’s proposal for the (symmetric) padding applied to the tile size. Unit is ‘number of voxels’. See Details section for details.

Used Padding

name: outUsedPadding, type: IntVector6, persistent: no

Padding (in number of voxels) actually used to pad input tile symmetrically. See ‘Details’ section for details.

Use (inUseCustomStride)

name: inUseCustomStride, type: Bool, default: FALSE

If enabled, Stride is used instead of the proposition from the connected processor.

Stride

name: inCustomStride, type: Vector6, default: 1 1 1 1 1 1

If Use is enabled, this overrides the processor’s proposal for the stride representing the ratio between input and output image size (before padding). Unit is ‘number of voxels’. See Details section for details.

Used Stride

name: outUsedStride, type: Vector6, persistent: no

Stride (in number of voxels) actually used to define ratio between input and output tile sizes (before padding). See ‘Details’ section for details.

Use (inUseCustomFillMode)

name: inUseCustomFillMode, type: Bool, default: FALSE

If enabled, Fill Mode is used instead of the proposition from the connected processor.

Fill Mode

name: inCustomFillMode, type: Enum, default: FillValue

If Use is enabled, this overrides the processor’s proposal for the fill mode, which defines how undefined input regions are to be filled.

Values:

Title	Name	Description
​Fill Value	​FillValue	​Fill with a fixed value
​Reflect	​Reflect	​Reflect valid region (but only within padding range). See ‘Details’ for details.

Used Fill Mode

name: outUsedFillMode, type: Enum, persistent: no

Actually used mode for filling undefined areas in the input tiles.

Values:

Title	Name	Description
​Fill Value	​FillValue	​Fill with Used Fill Value
​Reflect	​Reflect	​Reflect valid region (but only within padding range). See ‘Details’ for details.

Use (inUseCustomFillValue)

name: inUseCustomFillValue, type: Bool, default: FALSE

If enabled, Fill Value is used instead of the proposition from the connected processor.

Fill Value

name: inCustomFillValue, type: Double, default: 0

If Use is enabled, this overrides the processor’s proposal for the value to be used for filling if Used Fill Mode is ‘FillValue’.

Used Fill Value

name: outUsedFillValue, type: Double, persistent: no

Actually used fill value in ‘FillValue’ mode.

Use (inUseCustomExternalDimensionForChannel1)

name: inUseCustomExternalDimensionForChannel1, type: Bool, default: FALSE

If enabled, Custom Dimension for CHANNEL1 is used instead of the proposition from the connected processor.

Custom Dimension for CHANNEL1

name: inCustomExternalDimensionForChannel1, type: Enum, default: C, deprecated name: inExternalChannelDimension,inExternalDimensionForChannel1

If Use is enabled, this overrides the processor’s proposal for the image dimension to map to the CHANNEL1 processor dimension.

Values:

Title	Name
​X	​X
​Y	​Y
​Z	​Z
​C	​C
​T	​T
​U	​U

Used Dimensions - CHANNEL1

name: outUsedExternalDimensionForChannel1, type: Enum, persistent: no

Defines which (external, i.e. in input and output image) dimension is used by the processor as primary CHANNEL (=CHANNEL1) dimension (if it needs one).

Values:

Title	Name
​None	​NONE
​X	​X
​Y	​Y
​Z	​Z
​C	​C
​T	​T
​U	​U

Use (inUseCustomExternalDimensionForChannel2)

name: inUseCustomExternalDimensionForChannel2, type: Bool, default: FALSE

If enabled, CHANNEL2 is used instead of the proposition from the connected processor.

CHANNEL2 (inCustomExternalDimensionForChannel2)

name: inCustomExternalDimensionForChannel2, type: Enum, default: U, deprecated name: inExternalDimensionForChannel2

If Use is enabled, this overrides the processor’s proposal for the image dimension to map to the CHANNEL2 processor dimension.

Values:

Title	Name
​X	​X
​Y	​Y
​Z	​Z
​C	​C
​T	​T
​U	​U

CHANNEL2 (outUsedExternalDimensionForChannel2)

name: outUsedExternalDimensionForChannel2, type: Enum, persistent: no

Defines which (external, i.e. in input and output image) dimension is used by the processor as CHANNEL2 dimension (if it needs one).

Values:

Title	Name
​None	​NONE
​X	​X
​Y	​Y
​Z	​Z
​C	​C
​T	​T
​U	​U

Use (inUseCustomExternalDimensionForBatch)

name: inUseCustomExternalDimensionForBatch, type: Bool, default: FALSE

If enabled, BATCH is used as BATCH dimension. Otherwise, the module tries to auto-guess a suitable dimension. In that case, it uses the dimension with the largest extent among those that are otherwise unused.

BATCH (inCustomExternalDimensionForBatch)

name: inCustomExternalDimensionForBatch, type: Enum, default: T

If Use is enabled, this overrides the module’s proposal for the batch dimension to map to the BATCH processor dimension.

Values:

Title	Name
​X	​X
​Y	​Y
​Z	​Z
​C	​C
​T	​T
​U	​U

BATCH (outUsedExternalDimensionForBatch)

name: outUsedExternalDimensionForBatch, type: Enum, persistent: no, deprecated name: outUsedExternalBatchDimension

If the processor requires a BATCH dimension, the module will auto-select a “free” dimension, which is documented in this field. The selected BATCH dimension is the lowest (in terms of low=x -> y -> … -> u=high) dimension not explicitly specified in outUsedDimensions for which the input image has extent > 1.

Using a batch size > 1 (i. e. using tiles with more than one patches) can be helpful if the processor is more efficient when provided with mupltiple patches per tile, e.g. if the GPU is used for a processor supporting a batch dimension.

Values:

Title	Name
​X	​X
​Y	​Y
​Z	​Z
​C	​C
​T	​T
​U	​U
​None	​NONE

Hidden Fields

doNotClearOnFailedUpdate

name: doNotClearOnFailedUpdate, type: Bool, persistent: no

Prevents automated clear after update failed. This does not affect status fields. It enables the developer to analyze module’s state after failure.

outMaxNumModuleInputs

name: outMaxNumModuleInputs, type: Integer, persistent: no

Maximum number of supported module inputs. The module will not be able to process more inputs than this at the same time.

Contact the author/maintainer if you need more.

outMaxNumModuleOutputs

name: outMaxNumModuleOutputs, type: Integer, persistent: no

Maximum number of supported module outputs. The module will not be able to process more outputs than this at the same time.

Contact the author/maintainer if you need more.

outUsedDimensions

name: outUsedDimensions, type: String, persistent: no, deprecated name: outUsedExternalToClassifierDimensionMapping,outUsedExternalToProcessorDimensionMapping,outUsedInputDimensions

Documents the actually used dimension mapping.

inOutputSelectionMode

name: inOutputSelectionMode, type: Enum, default: Single, deprecated name: inModelOutputSelectionMode

Choose how the module decides which of the processor outputs to pick. If it has only one output, just leave “Single”.

Values:

Title	Name	Description
​Single	​Single	​If the processor defines only a single output, it will be selected. Otherwise, an error will be reported.
​Custom	​Custom	​You have to specify the exact name of the output to select in inCustomOutput

inCustomOutput

name: inCustomOutput, type: String, deprecated name: inCustomModelOutput,inRequestedOutputs

Name of the model output to pick in inOutputSelectionMode ‘Custom’

inCustomDimensions

name: inCustomDimensions, type: String, default: X, Y, CHANNEL, BATCH, deprecated name: inCustomExternalToClassifierDimensionMapping,inCustomExternalToProcessorDimensionMapping,inCustomInputDimensions

Comma-separated list describing the order in which the input data dimensions are expected by the processor, in ML order, i.e. smallest stride (X) first. On top of the data (typically spatial) dimensions provided to the processor, you can specify where the input channel and batch dimensions are located. Valid elemental values are X, Y, Z, C, T, U, as well as CHANNEL (or CHANNEL1), CHANNEL2, BATCH and UNUSED. UNUSED can be used for all dimensions not required by the processor (but it can also be omitted).

Typical examples are

X, Y, CHANNEL, BATCH

for 2D processors with a channel and batch dimension, or

X, Y, Z, CHANNEL, BATCH

for 3D processors with a channel and batch dimension.

** Normally, you should not have to edit this here, as the processor should already propose the configuration it needs, so that inUseCustomDimensions can be left disabled. **

inUseCustomDimensions

name: inUseCustomDimensions, type: Bool, default: FALSE, deprecated name: inUseCustomExternalToClassifierDimensionMapping,inUseCustomExternalToProcessorDimensionMapping,inUseCustomInputDimensions

If enabled, inCustomDimensions is used instead of the one provided by the processor. Should usually not be necessary, try to reconfigure your processor instead!