CurveFilter

MLModule

genre	Diagram
authors	Caroline Kuehnel, Tobias Block, Volker Dicken
package	FMEstable/ReleaseMeVis
dll	MLCurveUtils
definition	MLCurveUtils.def
see also	CurveGenerator, SoDiagram2D, BaseBypassOp, CurveProperties, CurveListConvert, CurveInfo
keywords	curve, diagram, filter, base, parser, differentiate, integrate, smooth, median, average, min, max, quantile, histogram, DVH, dose, volume, histogram, resample

Purpose

The module CurveFilter filters an input Curve by some criteria and generates a filtered copy as output.

Usage

Connect a plain Curve or CurveList to the input and select a filter mode. Use CurveListConvert to get alternative Curve data in a suitable format.

Windows

Default Panel

Input Fields

inCurves

name: inCurves, type: MLBase

The input defines the input curve(s).

Use CurveGenerator and affiliated modules like ConcatenateCurve, CurveInfo, etc. for input.

Output Fields

outCurves

name: outCurves, type: MLBase

Filtered result curve(s).

Parameter Fields

Field Index

a =: Double	Debug Level: Integer	Percentage[%]: Double
Average Of Parameter: Double	Error: String	Process only limited interval: Bool
b =: Double	Expression: String	Range End: Double
Bin Size: Double	Filter Mode: Enum	Range Start: Double
c =: Double	Interpolation Mode: Enum	StdDev Of Parameter: Double
Constant Value: Double	Model Fitting Rel Center Of Drop Points: Bool	Step: Double
correctInvalidInterval: Bool	Num. Points: Integer	Strength: Double
d =: Double	Number Of Curve Points: Integer	Use index coordinates for range: Bool
Data To Process: Enum	Out Of Range Treatment: Enum	Use Step Function: Bool

Visible Fields

Debug Level

name: debugLevel, type: Integer, default: 0

Controls the verbosity of the module. Values > 0 will result in possibly many console output lines. Intended for debugging only.

Filter Mode

name: filterMode, type: Enum, default: Off

Defines the filter operation.

Values:

Title	Name	Description
​Off	​Off	​Forwards a copy of the curve
​Math. expression	​Math. expression	​Enter your own expression for filtering
​Smooth	​Smooth	​Smoothing by using a moving average in a window of size 2 * Strength. (CAVE this uses index number not world coordinates to determine the window width)
​Differentiate	​Differentiate	​Numerical differentiation (compute quotient of differences)
​Integrate	​Integrate	​Numerical integration (simple numerical integarion)
​Average	​Average	​Compute flat curve at average. The average value for the first curve is written to field constCurveParam. Hint: Similarly other constant values for a curve will be written to that field for some modes resulting in one value per curve.
​Std Dev	​StdDev	​Compute flat curve at average
​Median	​Median	​Compute local median (for a flat curve at median value use quantile with parameter 50%)
​Resample	​Resample	​Does a crude resampling of the x range (CAVE: X Range may change due to resampling !)
​Max	​Max	​Compute flat curve at max
​Min	​Min	​Compute flat curve at min
​Quantile	​Quantile	​Compute stated quantile of curve values. Constant curve at quantile parameter (stated in %)
​Cumulated Histogram	​CumulatedHistogram	​Compute cumulated histogram curve (= integral normalized to 100%)
​Inverse Cum Histogram	​InverseCumHistogram	​Compute inverse cumulated histogram curve (= 100% - integral normalized to 100%) (mode for things like dose-volume-histograms (DVH))
​Max Pos	​MaxPos	​Curve values: (max-min) x delta( . - maxPos) Peak at leftmost maximum position. Position of max written to constCurveParam
​Mean Pos	​MeanPos	​max - (max- min) x heavyside(. - meanPos) Change near mean position (X-grid points left of mean at max, above mean+1 at min, @ meanInx & meanIdx+1 at average (min,max)) Mean written to constCurveParam
​Lower Drop Point	​LowerDropPoint	​Position below peak were value first falls below percentage of max value
​Upper Drop Point	​UpperDropPoint	​Position above peak were value first falls below percentage of max value
​Center Of Drop Points	​CenterOfDropPoints	​Middle position of the two drop points
​Width At Drop Points	​WidthAtDropPoints	​Distance of the two drop points
​Normalized Integral Curve	​NormalizedIntegralCurve	​Computes normalized curve whose Integral is 1
​Normalized Height Curve	​NormalizedHeightCurve	​Computes normalized curve whose maximum height is 1
​Hist Average	​HistAverage	​Assumes curve is a histogram and computes expected value / mean
​Hist Std Dev	​HistStdDev	​Assumes curve is a histogram and computes std dev ( 1/N version)
​Curvature	​Curvature	​Curvature = 1/ radius of circle fitted through each point and its neighbours (at distance selected by _fldSmoothStrength)
​Histogram Bins	​HistogramBins	​Assumes curve is a histogram and re-samples it according to a given bin size
​Zero Of Lower Tangent	​ZeroOfLowerTangent	​Find zero of approximate tangent defined as secant between lower drop point at percentage p and drop point at level p + (100%-p)/2
​Zero Of Upper Tangent	​ZeroOfUpperTangent	​Find zero of approximate tangent defined as secant between upper drop point at percentage p and drop point at level p + (100%-p)/2
​Zero Of Lower Secant	​ZeroOfLowerSecant	​Find zero of secant between lower drop point at percentage p and peak or center from drop points at level p
​Zero Of Upper Secant	​ZeroOfUpperSecant	​Find zero of secant between upper drop point at percentage p and peak or center from drop points at level p
​Lower Next Local Min	​LowerNextLocalMin	​Next local min left of peak with value below stated percentage of peak
​Upper Next Local Min	​UpperNextLocalMin	​Next local min right of peak with value below stated percentage of peak
​Zero Of Lower Parabola Fit	​ZeroOfLowerParabolaFit	​Find zero of parabola fit between lower drop point at percentage p and peak or center from drop points at level p
​Zero Of Upper Parabola Fit	​ZeroOfUpperParabolaFit	​Find zero of parabola fit between upper drop point at percentage p and peak or center from drop points at level p
​Count Non Zero Points	​CountNonZeroPoints	​Used e.g. to counts spikes in a histogram of upscaled integer images to revert the scaling
​Max Non Zero Point	​MaxNonZeroPoint	​Used e.g. find range in a histogram
​Min Non Zero Point	​MinNonZeroPoint	​Used e.g. find range in a histogram
​Center Of Non Zero Range	​CenterOfNonZeroRange	​Used e.g. find center of range in a histogram
​Width Of Non Zero Range	​WidthOfNonZeroRange	​Used e.g. find width of range in a histogram
​Subtract Baseline	​SubtractBaseline

Interpolation Mode

name: interpolationMode, type: Enum, default: CubicSpline

Defines the interpolation mode if the Filter Mode is Resample.

Values:

Title	Name
​Constant	​Constant
​Linear	​Linear
​Quadratic	​Quadratic
​Cubic Spline	​CubicSpline
​Pw Cubic Hemite	​PwCubicHemite
​Centered Cubic	​CenteredCubic

Process only limited interval

name: onlyProcessSubRange, type: Bool, default: FALSE, deprecated name: onlyProcessMarkedInterval

If checked, only a specific interval of the selected curves are filtered.

Out Of Range Treatment

name: outOfRangeTreatment, type: Enum, default: Copy

Describe what values will be filled outside of startIndex and endIndex when processing a limited range of the curve data.

Values:

Title	Name	Description
​Copy	​Copy	​Default behaviour: just copy original curve values (for legacy reasons)
​Fill Zeros	​FillZeros	​Might become future standard behavior
​Fill Curve Min	​FillCurveMin	​Alternative std behavior ?
​Fill Curve Max	​FillCurveMax	​Special purpose mode
​Fill Const Curve Value	​FillConstCurveValue	​Could be reasonable choice for certain modes
​Fill Range End Values	​FillRangeEndValues	​Most resonable choice for certain modes, e.g. integral

Range Start

name: rangeStart, type: Double, default: 0, deprecated name: markerStart

Sets the start index of the interval of curves to be filtered.

Range End

name: rangeEnd, type: Double, default: 0, deprecated name: markerEnd

Sets the end index of the interval of curves to be filtered.

Use index coordinates for range

name: rangeInIndexCoordinates, type: Bool, default: TRUE

If checked the range will be specified in curve index values instead of X axis coordinates (legacy mode, this is the way these parameters were interpreted in the first implementation).

Number Of Curve Points

name: numberOfCurvePoints, type: Integer, persistent: no

Provides number of grid positions as basic curve info.

Expression

name: parserExpression, type: String, default: x^2+2*x

Sets a mathematical expression for the Filter Mode Math. expression.

Data To Process

name: parserData, type: Enum, default: Y-Data

Defines in which dimension the data should be processed.

Values:

Title	Name	Description
​X-Data	​X-Data	​Apply formula to parameter x
​Y-Data	​Y-Data	​Apply formula to parameter y
​XY-Data	​XY-Data	​Apply formula to parameter x and y

Error

name: parserErrorText, type: String

Shows an error message of the function parser.

a =

name: parserExtValueA, type: Double, default: 0

Sets one of four variables for use in the expression.

b =

name: parserExtValueB, type: Double, default: 0

Sets one of four variables for use in the expression.

c =

name: parserExtValueC, type: Double, default: 0

Sets one of four variables for use in the expression.

d =

name: parserExtValueD, type: Double, default: 0

Sets one of four variables for use in the expression.

Strength

name: smoothingStrength, type: Double, default: 10

Smoothing over that many curve points / distance (in # points). Also distance to take secants in curvature mode. (CAVE: This parameter is not then smoothing window width in X range, if x step != 1)

Num. Points

name: numResamplePoints, type: Integer, default: 10

Sets the number of target re-sample points.

Step

name: resampleInterval, type: Double, default: 1

Sets the step width for re-sampling.

Percentage[%]

name: percentage, type: Double, default: 50, minimum: 0, maximum: 100, deprecated name: quantilePercent

Percentage values used e.g. for drop points and quantile Some values possibly of interest related to normal distribution histograms:

For a Gaussian distribution the following theoretical values might be of relevance:

At 1.0 sigma drop to 60.65% with 68.27% inside and 15.87% each above resp. below (about 1 in 3 outside, 1 in 6 above/below) At 1.5 sigma drop to 32.45% with 86.64% inside and 6.68% each above resp. below (about 1 in 7 outside , 1 in 15 above/below) At 1.96 sigma drop to 14.65% with ~95.00% inside and 2.5% each above resp. below(about 1 in 20 outside , 1 in 40 above/below) At 2.0 sigma drop to 13.53% with 95.45% inside and 2.275% each above resp. below(about 1 in 22 outside , 1 in 44 above/below) At 2.5 sigma drop to 4.393% with 98.75807% inside and 0.62% each above resp. below (about 1 in 81 outside , 1 in 161 above/below) At 3.0 sigma drop to 1.111% with 99.733002% inside and 0.135% each above resp. below (about 1 in 370 outside , 1 in 741 above/below) At 3.5 sigma drop to 0.219% with 99.953474% inside and 0.0233% each above resp. below (about 1 in 2149 outside , 1 in 4298 above/below)

(source https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule and value look up in gnuplot renderings of exp(-x*x/2) or erf(x/sqrt(2) )

20% above / below @ ~0.841262 sigma with drop to 70.197% (above/below ~70% drop level ~ 20%) 10% above / below @ ~1.28155 sigma with drop to 43.991% (above/below ~44% drop level ~ 10%) 5.0% above / below @ ~1.64485 sigma with drop to 25.85% (above/below ~26% drop level ~ 5%) 2.5% above / below @ ~1.959964 sigma with drop to 14.65% (above/below ~15% drop level ~ 2.5%) 2.0% above / below @ ~2.053749 sigma with drop to 12.137% (above/below ~12% drop level ~ 2.0%) 1.5% above / below @ ~2.170090 sigma with drop to 9.4927% (above/below ~9.5% drop level ~ 1.5%) 1.0% above / below @ ~2.326348 sigma with drop to 6.680% (above/below ~6.7% drop level ~ 1.0%) 0.5% above / below @ ~2.57583 sigma with drop to 3.624% (above/below ~3.6% drop level ~ 0.5%) 0.25% above / below @ ~2.80703 sigma with drop to 1.945% (above/below ~1.95% drop level ~ 0.25%) 0.1% above / below @ ~3.09023 sigma with drop to 0.844% (above/below ~0.8% drop level ~ 0.1%) 0.05% above / below @ ~3.29052 sigma with drop to 0.445% (above/below ~0.4% drop level ~ 0.05%)

Correctness of these drop values values were approximately verified on a 200 cube float volume featuring Gaussian Noise with sigma = 1000 after strongly smoothing (with window 50) the histogram (taken with bin size 1).

See …FMEstable/ReleaseMeVis/Modules/ML/MLCurveUtils/networks/CurveFilter_ManualDropLevelTest.mlab

Constant Value

name: constCurveParam, type: Double, default: 0

Value for first curve in list

Average Of Parameter

name: averageOfParameter, type: Double, default: 0

Average over curve parameters over all curves in list

StdDev Of Parameter

name: stdDevOfParameter, type: Double, default: 0

StdDev over curve parameters over all curves in list

Bin Size

name: histogramBinSize, type: Double, default: 1, minimum: 0.001

Resample curve so it collects values into bins of this size

Use Step Function

name: useStepFunction, type: Bool, default: TRUE

Draw histogram curve as a bar plot

Model Fitting Rel Center Of Drop Points

name: ModelFittingRelCenterOfDropPoints, type: Bool, default: FALSE

Select if true max or center of peak value (from drop points) is used for parabola or secant model fitting

Hidden Fields

correctInvalidInterval

name: correctInvalidInterval, type: Bool, default: TRUE