MERACrawler

MLModule

genre	MERA
author	Wolf Spindler
package	FMEstable/ReleaseMeVis
dll	MLMERA
definition	MLMERA.def
see also	MERAQuery, MERAVolumeInspector, DirectDicomImport, MultiFileVolumeListBaseOutput, MultiFileVolumeListImageOutput
keywords	crawler, parser, directoy, search, index, volume, storage, data, base, query, search, find

Purpose

MERACrawler traverses one or multiple directories, looks for configuration files in them and extracts (image) volumes from the files found in them. The volume information is stored as MultiFileVolume caches in a Volume Storage. Volumes in the volume storage can be inspected MERAVolumeInspector module and fed into MeVisLab network. Also pairs of key-values can be extracted to be used by MERAQuery module to run queries for volumes which can easily be displayed and inspected also with MERAVolumeInspector.

Windows

Default Panel

Parameter Fields

Field Index

Absolute Default Configuration File Name: String	Delete Entries of Volume Storage: Trigger	Ids to build from: String
Absolute Validation Config File Name: String	Delete Query DB: Trigger	Name Of Configuration Files: String
Absolute Volume Storage Directory: String	Delete Volume Storage: Trigger	New Directory To Crawl: String
Append: Trigger	Directories to crawl: String	Num Level Entries: Integer
Crawl: Trigger	Directory of Volume Storage to Append: String	Replace: String
Create Compressed Query DB Files: Trigger	Do Extra Std Cout File Logging: Bool	Replacement: String
Create Query DB: Trigger	Estimated Volume Storage Size: Integer	Report: String
Delete Entries of Query DB: Trigger	Ids of Entries to Delete: String	Start Id in Source: Integer

Visible Fields

Name Of Configuration Files

name: nameOfConfigurationFiles, type: String, default: MERAConfig.cfg

File name (without directory path) of configuration files possibly found in crawled directories.

Absolute Default Configuration File Name

name: absoluteDefaultConfigurationFileName, type: String, default: MERADefaultConfig.cfg

Absolute file path of the default configuration file to be used if no other configuration file is available in crawled directories.

Absolute Validation Config File Name

name: absoluteValidationConfigFileName, type: String, default: MERAValidationConfig.cfg

Absolute path to the validation file for configurations.

Absolute Volume Storage Directory

name: absoluteVolumeStorageDirectory, type: String

The destination directory into which the created volumes shall be stored.

New Directory To Crawl

name: newDirectoryToCrawl, type: String

Selector for a directory to crawl which is automatically appended to Directories to crawl.

Directories to crawl

name: directoriesToCrawl, type: String

Newline-separated list of directories to crawl.

Crawl

name: crawl, type: Trigger

Starts crawling when notified.

Create Query DB

name: createQueryDB, type: Trigger

Creates the query index from a generated volume storage.

Delete Volume Storage

name: deleteVolumeStorage, type: Trigger

Deletes the volume storage (which was created with Crawl) from disk without further notice.

Delete Query DB

name: deleteQueryDB, type: Trigger

Deletes the query data base (which was created with Create Query DB) from disk without further notice.

Delete Entries of Volume Storage

name: deleteIdsFromVolumeStorage, type: Trigger

Deletes volumes with ids from Ids of Entries to Delete from the volume storage from disk without further notice.

Delete Entries of Query DB

name: deleteIdsFromQueryDB, type: Trigger

Deletes entries with ids from Ids of Entries to Delete from the query data base* from disk without further notice.

Ids of Entries to Delete

name: idsToDelete, type: String

The is a field specifying IDs of entries to delete either from the volume storage with Delete Entries of Volume Storage or from the query data base with Delete Entries of Query DB. It has the same format as in MERAQuery.volumeIds in MERAQuery, which means that it allows single ids as well as range specifications.

Report

name: report, type: String, persistent: no

The output (and read-only) field for the crawl report.

Estimated Volume Storage Size

name: estimatedVolumeStorageSize, type: Integer, default: 250000

For cases when a volume store is created, for example if no one exists and crawling is started, then Estimated Volume Storage Size determines the number of volumes which probably shall be stored in it. Too small or too large volume storages may be inefficient for a number of volumes otherwise. In case of doubts better use a larger number than a too small one. See also Num Level Entries.

Num Level Entries

name: numLevelEntries, type: Integer, default: 1000

For cases when a volume store is created, for example if no one exists and crawling is started, then Num Level Entries determines the number of files or directories to be used for each level in the directory/file tree of the volume storage. Too small or too large volume storages may be inefficient. Very small values may lead to the creation of a very deep directory tree in the volume storage, large ones may slow down displaying the directories of the volume storage in file explorers or accessing files. Recommended values are usually between 10 and 500. See also Estimated Volume Storage Size.

Ids to build from

name: volumeIdsToBuildKeyValuePairsFrom, type: String, default: 0-

A set of ids of the current volume storage from which query DB information is extracted when Create Query DB is notified. This is an advanced option which especially can be useful after extending the volume storage with new volumes with Append. It has the same format as in MERAQuery.volumeIds in MERAQuery, which means that it allows single ids as well as range specifications.

Append

name: appendVolumeStorage, type: Trigger

If Append is notified then all volumes starting with volume with id Start Id in Source of the volume storage at Directory of Volume Storage to Append are appended to the current volume storage given with Absolute Volume Storage Directory. See Replace and Replacement if the appended volume storage is relative to another base directory and shall be rebased to the current one.

Directory of Volume Storage to Append

name: volumeStorageToAppendDirectory, type: String

The volume storage to be appended to the current one when Append is notified.

Start Id in Source

name: volumeStorageToAppendStartId, type: Integer, default: 0

The id of the first volume to append from Directory of Volume Storage to Append if Append is notified.

Replace

name: volumeStorageToAppendFromBase, type: String

When Append is running and the volume storage Directory of Volume Storage to Append has volumes under another base directory than the current one, then Replace at the begin of paths in volumes of Directory of Volume Storage to Append is replaced with Replacement.

Replacement

name: volumeStorageToAppendToBase, type: String

This is the string to be inserted at the begin of path if a rebase of volumes shall be performed while running Append; see Replacement and Append for details.

Do Extra Std Cout File Logging

name: doExtraStdCoutFileLogging, type: Bool, default: FALSE

Additional information about the current crawl or query DB extraction is logged to std::cout. This is especially meant to be written into the MeVisLab.log file in cases of long running batch processes.

Create Compressed Query DB Files

name: compressQueryDB, type: Trigger

After creating a final Query DB is can optionally be compressed. Although this costs time, especially MERAQuery users over slow network connections may experience significant speed ups when using cache directories where the compressed files can be stored and decompressed.