2.3. Image Classes

2.3.1.  ImageProperties

The ml::ImageProperties class describes the basic image properties

Images are rectangular grids without gaps, and all voxels are of identical extent and types. The six image dimensions in the ML are interpreted as the three spatial dimensions (x, y and z), a color extent (c dimension), a time extent (t dimension) and a user (u) dimension. For example, a dynamic sequence of three dimensional color images that exist in different image acquisitions or reconstructions can be handled by the ML as a single image object.

See mlImageProperties.h in project ML for more information.

2.3.2.  MedicalImageProperties

The ml::MedicalImageProperties class is derived from ImageProperties (Section 2.3.1, “ ImageProperties). It contains additional information specialized for medical data sets:

  • a voxel size,

  • a 4x4 transformation matrix (world matrix) to specify 3D transformations (e.g., for registration purposes),

  • an anonymous reference to a list that stores DICOM tag information if the input file(s) have been in DICOM file format,

  • color channel information (as strings) for the 4th dimension. The string list std::vector<std::string> &getCDimensionInfos() describes the significance for the channels e.g., "RED", "GREEN", and "BLUE" for channels 0, 1 and 2 when the RGB color model is used,

  • time point information for the t extent of the image. The list std::vector<DateTime> &getTDimensionInfos() contains this information for each time point,

  • u dimension information given as a list accessible with std::vector<std::string> &getUDimensionInfos(). The stored strings describe the subimages with different u components. Often, strings such as "CT", "MR", etc. are stored.

[Note]Note

For the c (color) and u dimension, there is a set of constants available describing the image contents, such as ML_RED, ML_BLUE, ML_SATURATION, ML_HUE for the c dimension, or ML_CT, ML_MR, ML_PET for the u dimension (see mlDataTypes.h ). The components of the list for the t dimension are given by the class DateTime (see mlDateTime.h ).

See mlImageProperties.h in project ML for more information.

2.3.3.  ImagePropertyExtension

ml::ImagePropertyExtension is used to append additional and user-defined property information to an ML image. This class is independent of the classes ImageProperties and MedicalImageProperties (see Section 2.3.1, “ ImageProperties and Section 2.3.2, “ MedicalImageProperties). It is an abstract class that serves as a base class from which an application or programmer can derive new properties. These properties are added to the ImagePropertyExtensionContainer that is a member of the class MedicalProperties.

A derived ImagePropertyExtension must meet some requirements:

  • It must implement the copy constructor and assignment operator correctly, because objects of its type are copied from one image to another.

  • It requires a virtual createClone() method that returns a copy or a new instance of the class so that a copy of the correct derived class is returned.

  • It must implement ML runtime typing and must be registered in the runtime type system of the ML in such a way that the ML can create new instances of the user-defined class only from its name and compare class types.

  • It must implement set and get methods to set/get the property value as a string, because ML modules must be able to store/load property settings in/from a file.

  • It must implement equality and inequality operators to compare instances.

Most methods to be implemented are pure virtual in the base class ImagePropertyExtension, hence compilation will not work without implementing them

The following programming example demonstrates how to implement a newly derived ImagePropertyExtension:

#include "mlModuleIncludes.h"

ML_START_NAMESPACE

//! Implement a ImagePropertyExtension object which can be passed to the ML.
class MODULE_TESTS_EXPORT OwnImagePropertyExtension : public ImagePropertyExtension
{

public:

  //! Constructor.
  OwnImagePropertyExtension() : ImagePropertyExtension()
  {
    _extInfoString = "NewImageInfosString";
  }

  //! Destructor.
  virtual ~OwnImagePropertyExtension() { }

  //! Implement correct copy construction.
  OwnImagePropertyExtension(const OwnImagePropertyExtension &origObj) :
    ImagePropertyExtension(origObj)
  {
    _extInfoString = origObj._extInfoString;
  }

  //! Implement correct assignment.
  OwnImagePropertyExtension &operator=(const OwnImagePropertyExtension &origObj)
  {
    if (&origObj != this) { _extInfoString = origObj._extInfoString; }
    return *this;
  }

  //! Implement pure virtual equality operation to work even on base class pointers.
  virtual bool equals(const ImagePropertyExtension &extImageProps) const
  {
    if (extImageProps.getTypeId() == getClassTypeId()) {

      // Types are equal, compare contents.
      return _extInfoString == ((OwnImagePropertyExtension&)(extImageProps))._extInfoString;

    } else {
      return false; // Types differ, thus objects also differ.
    }
  }

  //! Creates a copy of the correct derived object (for comparisons / runtime type determination).
  virtual ImagePropertyExtension *createClone() const
  {
    return new OwnImagePropertyExtension(*this);
  }

  //! Returns value of property as string.
  virtual std::string getValueAsString() const
  {
    return _extInfoString;
  }

  //! Set value of property from string value.
  virtual MLErrorCode setValueFromString(const std::string &str)
  {
    _extInfoString = str;
    return ML_RESULT_OK;
  }


private:

  //! The string values used as additional image property.
  std::string _extInfoString;

  //! Implements interface for the runtime type system of the ML.
  ML_CLASS_HEADER(OwnImagePropertyExtension)
};

ML_END_NAMESPACE

Implement the C++ part of the class interface to the runtime type system:

ML_START_NAMESPACE

  //! Implements code for the runtime type system of the ML.
  ML_CLASS_SOURCE(OwnImagePropertyExtension, ImagePropertyExtension);

ML_END_NAMESPACE

Register the class to the runtime type system of the ML when the .dll/.so file is loaded. This is typically done in the InitDll file of the project:

ML_START_NAMESPACE

  int MyProjectInit()
  {
    OwnImagePropertyExtension::initClass();
  }

ML_END_NAMESPACE

In the method calculateOutputImageProperties of your ML module, you can add a copy of your own image property to the output image:

ML_START_NAMESPACE

  MyModule::calculateOutputImageProperties(int outIndex, PagedImage* outImage)
  {
    OwnImagePropertyExtension myNewImgProp;
    outImage->getImagePropertyContainer().appendEntry(&myNewImgProp, true);
  }

ML_END_NAMESPACE

See mlImagePropertyExtension.h and mlImagePropertyExtensionContainer.h in project ML for more information.

2.3.4.  PagedImage

The class ml::PagedImage is dedicated to managing paged images in the ML and to representing image outputs of ML modules. See mlPagedImage.h in project ML.

The ML mainly works with pages and tiles. Since ML does usually not process entire images, it is necessary to break them down into smaller fractions of identical extent, the so-called pages. Pages can easily be buffered, cached and processed in parallel without spending too much memory or time. Caching in the ML works exclusively with pages. Moreover, only the pages that overlap with the actually requested image region must be processed. All other pages are not processed. Common page extents are, for example, 128x128x1x1x1x1 voxels. However, they may also have a real six-dimensional extent as do all images in the ML. Often, other image fractions (also called tiles) which do not have standard page extents are needed. Tiles are usually composed from pages and are used by the application or as input for image processing algorithms. In the ML, tiles are usually only temporary, i.e., the ML does not cache tiles.

For algorithms where a page-based implementation is difficult, classes such as VirtualVolume, BitImage or MemoryImage provide special interfaces to simplify efficient implementations. See Section 2.3.7, “ VirtualVolume, Section 2.3.6, “ BitImage and Section 2.3.8, “ MemoryImage for more information.

2.3.5. SubImage/TSubImage

ml::SubImage is an important class representing image and subimage buffers. It is used to manage, copy, etc. chunks of voxel data. It contains fast data access methods. See mlSubImage.h .

ml::TSubImage is the typed version of SubImage which permits typed data accesses. See mlTSubImage.h in project ML.

ml::TSubImageCursor and ml::ConstTSubImageCursor are cursor classes that allow access to a given TSubImage using cursor positioning and movement.

The SubImage class represents a rectangular 6D image region with linearly organized memory. It offers:

  • Methods for setting/accessing the datatype, the box defining the subimage region, the source image extent and the valid region (which is the intersection of the source image extent and the box).

  • A pointer to the memory data containing the image data as a void pointer. Alternatively the data can be stored as a MLMemoryBlockHandle to manage data via the MLMemoryManager.

  • With this class, the Host manages and encapsulates rectangular image regions (e.g., for pages, tiles, cached image results) and passes them to the image processing algorithms. The Host usually does not need information about the actual data.

  • The typical image processing methods in the ML are located in overloaded methods of Module::calculateOutputSubImage(). In these methods, the untyped memory chunks given as SubImage are usually wrapped again to typed subimages. See TSubImage for more information.

  • The type of the image data in memory is handled via a void pointer; the type, however, is managed as an enum type to support typed access in derived classes (see TSubImage). Consequently, SubImage does not support typed access to image voxels.

  • The typed access to voxels is implemented on top of this class as the template class TSubImage.

2.3.5.1. Example

The following paragraphs show some typical use cases of the class SubImage.

This creates a SubImage instance that provides access to a chunk of double data of 16 x 32 x 8 x 1 x 1 x 1 voxels given by the pointer dataPtr:

SubImage subImgBuf(SubImageBox(ImageVector(0,0,0,0,0,0),
                           ImageVector(15,31,7,0,0,0)),
                 MLdoubleType,
                 dataPtr);

The caller is responsible for the data chunk to be sufficiently large.

This first fills the entire subimage with the value 7.7. Then the rectangular region outside the area given by (3,3,3,0,0,0) and (5,5,5,0,0,0) is filled with the value 19.3:

subImgBuf.fill(7.7);
subImgBuf.fillBordersWithLDoubleValue(SubImageBox(ImageVector(3,3,3,0,0,0),
                                                ImageVector(5,5,5,0,0,0)),
                                      19.3);
Assuming another SubImage object srcSubImg, the overlapping areas can simply be copied (and if necessary, cast to the target type) into subImgBuf, and optionally rescaled with value 0.5:

subImgBuf.copySubImage(srcSubImg, ScaleShiftData(0.5, 0) );

Untyped data access to the voxel data is available for example at position (1,2,3,0,0,0) with

void *voxPtr = subImgBuf.getImagePointer( ImageVector(1,2,3,0,0,0,0) );

For typed data management, the class TSubImage can be used almost in the same way:

TSubImage<MLdouble> subImgBufT(SubImageBox(ImageVector(0,0,0,0,0,0),
                                       ImageVector(15,31,7,0,0,0)),
                             MLdoubleType,
                             dataPtr);

The class TSubImage, however, provides a number of typed access functions, such as

MLdouble *voxPtrT = subImgBufT.getImagePointer( ImageVector(1,2,3,0,0,0,0) );
*voxPtrT = 29.2;

The untyped SubImage and the templated TSubImage classes also provide a variety of other methods to manipulate, copy, fill, allocate and delete data, or to check for a certain value, or to retrieve statistical information such as minimum or maximum. They are powerful classes that can be used in many contexts when memory or voxel buffers have to be managed.

See files mlSubImage.h and mlTSubImage.h for more information.

2.3.6.  BitImage

In the page-based image processing concept of the ML, Boolean data types are not available (nor are they planned).

The BitImage class can be used as an alternative.

The following set of operations is available for this class type:

  • full 6D support in all methods,

  • set, get, clear and toggle bits at coordinates,

  • filling (=clearing or setting) and inverting subimage boxes,

  • copying from/to subimages (with thresholding),

  • saving/loading to/from file,

  • position checking,

  • creating downscaled BitImages,

  • creating BitImages from image data where first the mask area is determined and then the smallest possible BitImage is returned,

  • cursor movement in all dimensions,

  • exception handling support for safe operations on images.

2.3.7.  VirtualVolume

The ml::VirtualVolume and the ml::TVirtualVolume classes manage efficient voxel access to the output image of an input module or to a 'standalone' image.

So it is possible to implement random access to a paged input image or to a pure virtual image without mapping more than a limited number of bytes. Pages of the input volume are mapped temporarily into memory when needed. If no input volume is specified, the pages are created and filled with a fill value. When the permitted memory size is exceeded, older mapped pages are removed. When pages are written, they are mapped until the virtual volume instance is removed or until they are explicitly cleared by the application. Virtual volumes can easily be accessed by using setValue and getValue. These kinds of access are well-optimized code that might need 9 (1D), 18 (3D) and 36 (6D) instructions per voxel if the page at the position is already mapped.

A cursor manager for moving the cursor with moveCursor* (forward) and reverseMoveCursor* (backward) is also available. setCursorValue and getCursorValue provide voxel access. Good compilers and already mapped pages might require about 5-7 instructions. So the cursor approach will probably be faster for data volumes with more than 2 dimensions.

All the virtual volume access calls can be executed with or without error handling (see last and default constructor parameters). If areExceptionsOn is true, every access to the virtual volume is tested and if necessary, exceptions are thrown that can be caught by the code calling the virtual volume methods. Otherwise, most functions do not perform error handling.

[Note]Note

Exception handling versions are slower than versions with disabled exceptions. However, this is the only way to handle accesses safely.

[Tip]Tip

This class is the recommended alternative to global image processing algorithms.

2.3.7.1. Code Examples

The following code gives an example of how to use the VirtualVolume class:

Example 2.6. How to Use the VirtualVolume Class

Header:

VirtualVolume *_virtVol;

Constructor:

_virtVol = NULL;

Create/Update the virtual volume in calculateOutputImageProperties() and invalidate the output image on errors, so that calculateOutputSubImage() is not called on bad virtual volume later.

if (_virtVolume != NULL) { delete _virtVolume; }

_virtVolume = new VirtualVolume(this, 0, getInputImage(0)->getDataType());

if (!_virtVolume || (_virtVolume && !_virtVolume->isValid())){
  outImage->setInvalid(); return;

When you do not want to use a 'standalone' virtual volume:

_virtVolume = new VirtualVolume(ImageVector(1024,1024,1,1,1,1), 0, MLuint8Type));

if ((_virtVolume == NULL) || (_virtVolume && !_virtVolume->isValid())){
  outImage->setInvalid(); return;
}

Example of how to access image data directly: calculateOutputSubImage()

// Create wrapper for typed voxel access.
TVirtualVolume<DATATYPE> vVol(*_virtVolume);

ImageVector pos(7,3,0,0,0,0);
DATATYPE value;

vVol.setValue(pos, value);             // Simple setting of an arbitrary voxel.
value = vVol.getValue(pos);            // Reading of an arbitrary voxel.
vVol.fill(outSubImg->getBox(), value); // Fill region with value.

// Now copy valid region of virtVolume to outSubimg.
vVol.copySubImage(*outSubImg);

Example of how to access image data via a cursor: calculateOutputSubImage():

// Create wrapper for typed voxel access.
TVirtualVolume<DATATYPE> vVol(*_virtVolume);

ImageVector pos(7,3,0,0,0,0);
DATATYPE value;

vVol.setCursorPosition(pos);                // Set cursor to any position in volume.
vVol.moveCursorX();                         // Move cursor >F<orward
vVol.moveCursorY();                         // in (positive) X, C and U direction.
vVol.moveCursorZ();
vVol.reverseMoveCursorT();                         // Move cursor >B<ackwards in (negative) T
vVol.reverseMoveCursorZ();                         // and Z direction.

val = vVol.getCursorValue();           // Reading voxel below cursor.
vVol.setCursorValue(10);               // Set voxel value below cursor to 10.

Additionally, the following helper routines are available:

// Fill region of virtual volume with a certain value.

void fill(const SubImageBox &box, DATATYPE value);


// Copy region from the virtual volume into a typed subimg.

void copyToSubImage(TSubImage<DATATYPE> &outSubImg);


// Copy a region from a typed subimg into the virtual volume.

void copyFromSubImage(TSubImage<DATATYPE> &inImg,
                    const SubImageBox &box,
                    const ImageVector &pos); 

There are also some routines to get the boxes of the currently written pages. It is also possible to read/write the data of the written pages directly.

[Note]Note

The class VirtualVolume contains data structures for data management and table caching; its creation is expensive in comparison to the TVirtualVolume class which is only a "lightweight" access interface that can rapidly be created and destroyed on top of a VirtualVolume object. In the case of algorithms which implement template support for arbitrary data types it is recommended to create an untyped VirtualVolume as class member and a TVirtualVolume class in calculateOutputSubImage for maximum performance.

However, there are also convenience constructors of the TVirtualVolume class which internally create the VirtualVolume instance automatically ; these constructors are more expensive and should not be used on each calculateOutputSubImage call. Nevertheless they can be useful when a class works only with a fixed data type or without templates.

Using virtual Volume instances that create untyped virtual volume instances automatically.

Creating a TVirtualVolume with a convenience constructor. It creates a VirtualVolume internally. It provides float data access to the input image 0, even if the input image is of another type. Note that the connected input image must be valid:

// Create a typed VirtualVolume from input connector 0 of
// this Module with voxels of type float directly without
// creating the untyped VirtualVolume manually.

TVirtualVolume<float> vVol(this, 0);

2.3.7.2.  Using Exceptions for Safe VirtualVolume Usage

The standard usage of the VirtualVolume and the TVirtualVolume classes does not include error handling. For safe usage, areExceptionsOn == true is passed as a parameter to the constructor, and errors will throw the following exceptions:

Note that areExceptionsOn == true degrades voxel access performance.

  • ML_OUT_OF_RANGE

    ML Error Code

    MLErrorCode is thrown if cursor positioning or voxel addressing tries to access invalid image regions. The exception leaves the virtual volume, the cursor position, the voxel content, etc. unchanged and the invalid flag of the virtual volume is not set. The call is just terminated and ignored, i.e., The call can continue and accesses to other voxels are attempted.

  • ML_NO_MEMORY

    MLErrorCode is thrown if an allocation fails because of insufficient memory. The valid virtual volume is invalidated, i.e., Its valid flag is cleared.

  • ML_BAD_DIMENSION

    MLErrorCode is thrown if the image data extent is invalid. This could indicate a programming error or invalid input image data. The valid virtual volume is invalidated, i.e., Its valid flag is cleared.

  • ML_BAD_DATA_TYPE

    MLErrorCode is thrown if an invalid image data type is encountered. This could indicate a programming error or invalid input image data. The valid virtual volume is invalidated, i.e., Its valid flag is cleared.

  • Other exceptions that result from page request failures could also be thrown. They are usually returned, when a getTile command that attempts to request data from an input image fails.

If the areExceptionsOn == false, no exception is thrown and many errors are handled by calling the ML_PRINT*() error macros and terminating the function/method. The virtual volume instance will be invalidated. Invalid voxel access or memory failures will destroy the program state or cause unknown exceptions.

2.3.7.3. Performance Issues on VirtualVolume Usage

  • Voxel access performance is best when the page extents of input pages are powers of 2.

  • Working locally on virtual volumes is generally faster than jumping randomly through the image, because less pages must be swapped.

  • Coordinate-specific voxel access performance is better for images of a lower dimension, because less calculations have to be performed.

  • If the virtual volume wraps a paged input image, voxel access is not permitted when the input connection or the module has become invalid.

  • The virtual volume must not be used in parallel in calculateOutputSubImage() calls, because getValue and setValue methods potentially call getTile*() which would start recursive multithreading. Therefore be sure that multithreading remains disabled in areas where VirtualVolume or TVirtualVolume use calculateOutputSubImage() or you must make accesses to them thread-safe by using critical sections, semaphores or similar concepts. Even if no paged image is used as an input, write access is not capable of multithreading due to performance reasons.

  • If an image has n dimensions (e.g., 3), components >= n in cursor positioning and voxel access are simply ignored for performance reasons and do not cause errors if they are set even if this means that the cursor was outside the image.

  • In some cases, the virtual volume approach is slower than a global approach.

    Consider the following reasons:

    • The virtual volume approach is completely page-based, i.e., it fits perfectly in the optimized page-based concept of the ML.

    • The virtual volume approach only requests image areas of the input image that are really needed (processing on demand) so that less input image regions are calculated. Global approaches always request the entire input image which is often expensive to calculate.

    • The virtual volume approach usually locks less memory than the global approach, so the operating system must swap less memory, and other modules can work faster.

    • Next versions will not duplicate memory as their own tiles (as a global approach needs to), but will directly try to use ML cache pages.

2.3.8.  MemoryImage

MemoryImage can be used for algorithms that need fast random access to entire images, especially if they work “against“ paging e.g., OrthoReformat, MPR, MemCache.

[Important]Important
  • A MemoryImage object is always buffered at the output of the connected input module.

  • Try to avoid this approach! It only supports limited image sizes that depend on the available memory! See Section 4.3.3, “VirtualVolume Concept” for information on how to avoid this concept.

Properties:

  • The MemoryImage object is part of each PagedImage, i.e., there is one (usually empty and unused) MemoryImage object per output.

  • If possible, all connected ML modules copy or reference data directly from the MemoryImage object.

There are two ways of how to use the memory image at a module output:

  • The module completely controls the MemoryImage object at the image output (reset, clear, set, resize, update...). Thus connected modules benefit (see Version 1).

  • The entire input image is requested as one page (with the note to buffer it as a memory image). Further requests (also from other modules) will be answered immediately by passing the pointer to the memory image or by copying page data from it (see Version 2).

  • Version 1: The module controls the memory image at the output:

    Example 2.7. Controlling the MemoryImage by the Module

    // Constructor: Enables the operator control of the memory output at output 0.
    
    getOutputImage(0)->getMemoryImage().setUserControlled(true);
    
    
    // Resize and copy input image into the memory image output:
    
    MLErrorCode result = getOutputImage(0)->getMemoryImage().update(
            getInputImage(0),
            getInputImage(0)->getImageExtent(),
            getInputImage(0)->getDataType());
    if (ML_RESULT_OK != result) { handleErrorHere(result); }
    
    
    // Get data pointer and draw into memory image at output:
    
    drawSomethingsIntoImg(getOutputImage(0)->getMemoryImage().getImage()); 

  • Version 2: The memory image is cached at the output of preceding module:

    Example 2.8. Using/Requesting a MemoryImage of the Predecessor Module

    // Request input tile caching in output of input module
    void MemoryInTest::calculateOutputImageProperties(int outIndex, PagedImage* outImage)
    {
      ...
    
      outImage->setInputSubImageUseMemoryImage(0, true);
    
    }
    
    // Request input tile of size of input volume (other sizes cause warnings!)
    
    SubImageBox MemoryInTest::calculateInputSubImageBox (int /*inIndex*/,
                                               const SubImageBox & /*outSubImgBox*/,
                                               int /*outIndex*/)
    {
      return getInputImage(0)->getBoxFromImageExtent();
    }

Advantages:

  • All connected modules can benefit from the memory image, because it is part of the image output.

  • It is easy to implement and fast; it does not break the paging concept.

Disadvantages:

  • The image size is limited by the size of the largest free memory chunk.

  • It cannot/should not be used in bigger networks or applications.

  • It must map the entire image and blocks large memory areas for a long time.