CurveFilter

MLModule

genre	Diagram
authors	Caroline Kuehnel, Tobias Block, Volker Dicken
package	FMEstable/ReleaseMeVis
dll	MLCurveUtils
definition	MLCurveUtils.def
see also	CurveGenerator, SoDiagram2D, BaseBypassOp, CurveProperties, CurveListConvert, CurveInfo
keywords	curve, diagram, filter, base, parser, differentiate, integrate, smooth, median, average, min, max, quantile, histogram, DVH, dose, volume, histogram, resample

Purpose

The module CurveFilter filters an input Curve by some criteria and generates a filtered copy as output.

Usage

Connect a plain Curve or CurveList to the input and select a filter mode. Use CurveListConvert to get alternative Curve data in a suitable format.

Windows

Default Panel

Input Fields

inCurves

name: inCurves, type: MLBase

The input defines the input curve(s).

Use CurveGenerator and affiliated modules like ConcatenateCurve, CurveInfo, etc. for input.

Output Fields

outCurves

name: outCurves, type: MLBase

Filtered result curve(s).

Parameter Fields

Field Index

a =: Double	Debug Level: Integer	Percentage[%]: Double
Average Of Parameter: Double	Error: String	Process only limited interval: Bool
b =: Double	Expression: String	Range End: Double
Bin Size: Double	Filter Mode: Enum	Range Start: Double
c =: Double	Interpolation Mode: Enum	StdDev Of Parameter: Double
Constant Value: Double	Model Fitting Rel Center Of Drop Points: Bool	Step: Double
correctInvalidInterval: Bool	Num. Points: Integer	Strength: Double
d =: Double	Number Of Curve Points: Integer	Use index coordinates for range: Bool
Data To Process: Enum	Out Of Range Treatment: Enum	Use Step Function: Bool

Visible Fields

Debug Level

name: debugLevel, type: Integer, default: 0

Controls the verbosity of the module. Values > 0 will result in possibly many console output lines. Intended for debugging only.

Filter Mode

name: filterMode, type: Enum, default: Off

Defines the filter operation.

Values:

Title	Name	Description
Off	Off	Forwards a copy of the curve
Math. expression	Math. expression	Enter your own expression for filtering
Smooth	Smooth	Smoothing by using a moving average in a window of size 2 * Strength. (CAVE this uses index number not world coordinates to determine the window width)
Differentiate	Differentiate	Numerical differentiation (compute quotient of differences)
Integrate	Integrate	Numerical integration (simple numerical integarion)
Average	Average	Compute flat curve at average. The average value for the first curve is written to field constCurveParam. Hint: Similarly other constant values for a curve will be written to that field for some modes resulting in one value per curve.
Std Dev	StdDev	Compute flat curve at average
Median	Median	Compute local median (for a flat curve at median value use quantile with parameter 50%)
Resample	Resample	Does a crude resampling of the x range (CAVE: X Range may change due to resampling !)
Max	Max	Compute flat curve at max
Min	Min	Compute flat curve at min
Quantile	Quantile	Compute stated quantile of curve values. Constant curve at quantile parameter (stated in %)
Cumulated Histogram	CumulatedHistogram	Compute cumulated histogram curve (= integral normalized to 100%)
Inverse Cum Histogram	InverseCumHistogram	Compute inverse cumulated histogram curve (= 100% - integral normalized to 100%) (mode for things like dose-volume-histograms (DVH))
Max Pos	MaxPos	Curve values: (max-min) x delta( . - maxPos) Peak at leftmost maximum position. Position of max written to constCurveParam
Mean Pos	MeanPos	max - (max- min) x heavyside(. - meanPos) Change near mean position (X-grid points left of mean at max, above mean+1 at min, @ meanInx & meanIdx+1 at average (min,max)) Mean written to constCurveParam
Lower Drop Point	LowerDropPoint	Position below peak were value first falls below percentage of max value
Upper Drop Point	UpperDropPoint	Position above peak were value first falls below percentage of max value
Center Of Drop Points	CenterOfDropPoints	Middle position of the two drop points
Width At Drop Points	WidthAtDropPoints	Distance of the two drop points
Normalized Integral Curve	NormalizedIntegralCurve	Computes normalized curve whose Integral is 1
Normalized Height Curve	NormalizedHeightCurve	Computes normalized curve whose maximum height is 1
Hist Average	HistAverage	Assumes curve is a histogram and computes expected value / mean
Hist Std Dev	HistStdDev	Assumes curve is a histogram and computes std dev ( 1/N version)
Curvature	Curvature	Curvature = 1/ radius of circle fitted through each point and its neighbours (at distance selected by _fldSmoothStrength)
Histogram Bins	HistogramBins	Assumes curve is a histogram and re-samples it according to a given bin size
Zero Of Lower Tangent	ZeroOfLowerTangent	Find zero of approximate tangent defined as secant between lower drop point at percentage p and drop point at level p + (100%-p)/2
Zero Of Upper Tangent	ZeroOfUpperTangent	Find zero of approximate tangent defined as secant between upper drop point at percentage p and drop point at level p + (100%-p)/2
Zero Of Lower Secant	ZeroOfLowerSecant	Find zero of secant between lower drop point at percentage p and peak or center from drop points at level p
Zero Of Upper Secant	ZeroOfUpperSecant	Find zero of secant between upper drop point at percentage p and peak or center from drop points at level p
Lower Next Local Min	LowerNextLocalMin	Next local min left of peak with value below stated percentage of peak
Upper Next Local Min	UpperNextLocalMin	Next local min right of peak with value below stated percentage of peak
Zero Of Lower Parabola Fit	ZeroOfLowerParabolaFit	Find zero of parabola fit between lower drop point at percentage p and peak or center from drop points at level p
Zero Of Upper Parabola Fit	ZeroOfUpperParabolaFit	Find zero of parabola fit between upper drop point at percentage p and peak or center from drop points at level p
Count Non Zero Points	CountNonZeroPoints	Used e.g. to counts spikes in a histogram of upscaled integer images to revert the scaling
Max Non Zero Point	MaxNonZeroPoint	Used e.g. find range in a histogram
Min Non Zero Point	MinNonZeroPoint	Used e.g. find range in a histogram
Center Of Non Zero Range	CenterOfNonZeroRange	Used e.g. find center of range in a histogram
Width Of Non Zero Range	WidthOfNonZeroRange	Used e.g. find width of range in a histogram
Subtract Baseline	SubtractBaseline

Interpolation Mode

name: interpolationMode, type: Enum, default: CubicSpline

Defines the interpolation mode if the Filter Mode is Resample.

Values:

Title	Name
Constant	Constant
Linear	Linear
Quadratic	Quadratic
Cubic Spline	CubicSpline
Pw Cubic Hemite	PwCubicHemite
Centered Cubic	CenteredCubic

Process only limited interval

name: onlyProcessSubRange, type: Bool, default: FALSE, deprecated name: onlyProcessMarkedInterval

If checked, only a specific interval of the selected curves are filtered.

Out Of Range Treatment

name: outOfRangeTreatment, type: Enum, default: Copy

Describe what values will be filled outside of startIndex and endIndex when processing a limited range of the curve data.

Values:

Title	Name	Description
Copy	Copy	Default behaviour: just copy original curve values (for legacy reasons)
Fill Zeros	FillZeros	Might become future standard behavior
Fill Curve Min	FillCurveMin	Alternative std behavior ?
Fill Curve Max	FillCurveMax	Special purpose mode
Fill Const Curve Value	FillConstCurveValue	Could be reasonable choice for certain modes
Fill Range End Values	FillRangeEndValues	Most resonable choice for certain modes, e.g. integral

Range Start

name: rangeStart, type: Double, default: 0, deprecated name: markerStart

Sets the start index of the interval of curves to be filtered.

Range End

name: rangeEnd, type: Double, default: 0, deprecated name: markerEnd

Sets the end index of the interval of curves to be filtered.

Use index coordinates for range

name: rangeInIndexCoordinates, type: Bool, default: TRUE

If checked the range will be specified in curve index values instead of X axis coordinates (legacy mode, this is the way these parameters were interpreted in the first implementation).

Number Of Curve Points

name: numberOfCurvePoints, type: Integer, persistent: no

Provides number of grid positions as basic curve info.

Expression

name: parserExpression, type: String, default: x^2+2*x

Sets a mathematical expression for the Filter Mode Math. expression.

Data To Process

name: parserData, type: Enum, default: Y-Data

Defines in which dimension the data should be processed.

Values:

Title	Name	Description
X-Data	X-Data	Apply formula to parameter x
Y-Data	Y-Data	Apply formula to parameter y
XY-Data	XY-Data	Apply formula to parameter x and y

Error

name: parserErrorText, type: String

Shows an error message of the function parser.

a =

name: parserExtValueA, type: Double, default: 0

Sets one of four variables for use in the expression.

b =

name: parserExtValueB, type: Double, default: 0

Sets one of four variables for use in the expression.

c =

name: parserExtValueC, type: Double, default: 0

Sets one of four variables for use in the expression.

d =

name: parserExtValueD, type: Double, default: 0

Sets one of four variables for use in the expression.

Strength

name: smoothingStrength, type: Double, default: 10

Smoothing over that many curve points / distance (in # points). Also distance to take secants in curvature mode. (CAVE: This parameter is not then smoothing window width in X range, if x step != 1)

Num. Points

name: numResamplePoints, type: Integer, default: 10

Sets the number of target re-sample points.

Step

name: resampleInterval, type: Double, default: 1

Sets the step width for re-sampling.

Percentage[%]

name: percentage, type: Double, default: 50, minimum: 0, maximum: 100, deprecated name: quantilePercent

Percentage values used e.g. for drop points and quantile Some values possibly of interest related to normal distribution histograms:

For a Gaussian distribution the following theoretical values might be of relevance:

At 1.0 sigma drop to 60.65% with 68.27% inside and 15.87% each above resp. below (about 1 in 3 outside, 1 in 6 above/below) At 1.5 sigma drop to 32.45% with 86.64% inside and 6.68% each above resp. below (about 1 in 7 outside , 1 in 15 above/below) At 1.96 sigma drop to 14.65% with ~95.00% inside and 2.5% each above resp. below(about 1 in 20 outside , 1 in 40 above/below) At 2.0 sigma drop to 13.53% with 95.45% inside and 2.275% each above resp. below(about 1 in 22 outside , 1 in 44 above/below) At 2.5 sigma drop to 4.393% with 98.75807% inside and 0.62% each above resp. below (about 1 in 81 outside , 1 in 161 above/below) At 3.0 sigma drop to 1.111% with 99.733002% inside and 0.135% each above resp. below (about 1 in 370 outside , 1 in 741 above/below) At 3.5 sigma drop to 0.219% with 99.953474% inside and 0.0233% each above resp. below (about 1 in 2149 outside , 1 in 4298 above/below)

(source https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule and value look up in gnuplot renderings of exp(-x*x/2) or erf(x/sqrt(2) )

20% above / below @ ~0.841262 sigma with drop to 70.197% (above/below ~70% drop level ~ 20%) 10% above / below @ ~1.28155 sigma with drop to 43.991% (above/below ~44% drop level ~ 10%) 5.0% above / below @ ~1.64485 sigma with drop to 25.85% (above/below ~26% drop level ~ 5%) 2.5% above / below @ ~1.959964 sigma with drop to 14.65% (above/below ~15% drop level ~ 2.5%) 2.0% above / below @ ~2.053749 sigma with drop to 12.137% (above/below ~12% drop level ~ 2.0%) 1.5% above / below @ ~2.170090 sigma with drop to 9.4927% (above/below ~9.5% drop level ~ 1.5%) 1.0% above / below @ ~2.326348 sigma with drop to 6.680% (above/below ~6.7% drop level ~ 1.0%) 0.5% above / below @ ~2.57583 sigma with drop to 3.624% (above/below ~3.6% drop level ~ 0.5%) 0.25% above / below @ ~2.80703 sigma with drop to 1.945% (above/below ~1.95% drop level ~ 0.25%) 0.1% above / below @ ~3.09023 sigma with drop to 0.844% (above/below ~0.8% drop level ~ 0.1%) 0.05% above / below @ ~3.29052 sigma with drop to 0.445% (above/below ~0.4% drop level ~ 0.05%)

Correctness of these drop values values were approximately verified on a 200 cube float volume featuring Gaussian Noise with sigma = 1000 after strongly smoothing (with window 50) the histogram (taken with bin size 1).

See …FMEstable/ReleaseMeVis/Modules/ML/MLCurveUtils/networks/CurveFilter_ManualDropLevelTest.mlab

Constant Value

name: constCurveParam, type: Double, default: 0

Value for first curve in list

Average Of Parameter

name: averageOfParameter, type: Double, default: 0

Average over curve parameters over all curves in list

StdDev Of Parameter

name: stdDevOfParameter, type: Double, default: 0

StdDev over curve parameters over all curves in list

Bin Size

name: histogramBinSize, type: Double, default: 1, minimum: 0.001

Resample curve so it collects values into bins of this size

Use Step Function

name: useStepFunction, type: Bool, default: TRUE

Draw histogram curve as a bar plot

Model Fitting Rel Center Of Drop Points

name: ModelFittingRelCenterOfDropPoints, type: Bool, default: FALSE

Select if true max or center of peak value (from drop points) is used for parabola or secant model fitting

Hidden Fields

correctInvalidInterval

name: correctInvalidInterval, type: Bool, default: TRUE