TestCaseManager

MacroModule

genre	Test
author	MeVis Medical Solutions AG
package	MeVisLab/Standard
definition	TestCaseManager.def
keywords	testing, functional

Purpose

Detailed documentation about the TestCaseManager and the TestCenter can be found on the MeVisLab Help Overview page. See Developing a Test Case and TestCenter Manual.

Windows

Default Panel

Parameter Fields

Field Index

Always use current MeVisLab: Bool	Max Error Messages Per Test Function: Integer	Report name would look like: String
anythingSelected: Bool	Max Info Messages Per Test Function: Integer	Secure testing: Bool
Append TestCase name: Bool	MeVisLab Path: String	Should generate HTML report: Bool
Append timestamp: Bool	Name: String	Should generate JUnit report: Bool
Associated Module: String	Network: String	Show only failed test functions: Bool
associatedModuleExists: Bool	New Test Case Name: String	Show reports in TestCaseManager panel: Bool
Bullseye coverage: Bool	New Test Case Type: Enum	Stop and inspect network on error: Bool
bullseyeAvailable: Bool	New Test Comment: String	Suite File: String
codeTestExecutableExists: Bool	newTestCaseProject: String	Test Network: Enum
codeTestNodeSelected: Bool	newTestCaseProjectPath: String	testCaseHasNetworkFile: Bool
codeTestSelected: Bool	Options: String	testCaseInfo: String
Collapse all test functions: Bool	Package Identifier: String	testCaseLoaded: Bool
Compile mode: Enum	packageSelected: Bool	testCaseLoadedAndVerified: Bool
dialogText: String	progress: Float	testCaseSelected: Bool
directoryStructure: Enum	progressDialogCurrentFunction: String	testCaseSuiteSelected: Bool
ensureReportViewerInPanelIsVisible: Trigger	progressDialogStatus: String	testCaseSuiteTestNames: String
Format: Enum	progressDialogText: String	testCaseWasExecutedNotSecure: Bool
Hide info messages: Bool	Python Coverage: Enum	Timestamp format: String
Hide system messages: Bool	Reload Python modules when test cases reload: Bool	Use MESA for secure testing: Bool
isDisplayingReport: Bool	Report directory: String
Log info messages: Bool	Report name: String

Visible Fields

Python Coverage

name: pythonCoverage, type: Enum, default: Disabled

If checked, for each test a coverage report will be generated for all utilized python modules loaded so far.

Note

The Python coverage report might list modules from the Python cache that were used earlier but are not relevant to the current test. Only the initialization code of these unrelated modules will appear as executed in the report. Technically, this is correct because all modules from the cache are reloaded before each test run. At this point, it is not possible to distinguish between the relevant modules for the test and other modules.

To avoid this issue, you can use ‘Secure testing’, which starts a new MeVisLab instance for each test, ensuring a clean Python cache.

Values:

Title	Name	Deprecated Name
​Disabled	​Disabled	​FALSE
​Per Test Case	​PerTestCase	​TRUE
​Global	​Global

Bullseye coverage

name: bullseyeCoverage, type: Bool, default: FALSE

If checked, the currently instrumented codes in the COVFILE are used to measure coverage values on a per-test basis for each test.

For this to work, Bullseye Coverage must be installed, and the path to its binaries must be included in your PATH environment variables.

Additionally, this only works correctly with SecureTesting enabled.

Secure testing

name: secureTesting, type: Bool, default: FALSE

If checked, a separate MeVisLab process for running the test(s) is started.

This uses the settings Always use current MeVisLab, MeVisLab Path, Options, Compile mode, and Use MESA for secure testing from the Configuration tab.

Use MESA for secure testing

name: useMesaForSecureTesting, type: Bool, default: FALSE

If checked, MeVisLab will try to use the MESA software OpenGL driver when running tests in Secure testing (instead of the system-supplied driver).

This only works on Windows.

Stop and inspect network on error

name: stopOnError, type: Bool, default: FALSE

If checked, the test execution stops on the first error and the active test network is opened in MeVisLab.

Log info messages

name: logInfoMessages, type: Bool, default: TRUE

If checked, all informational messages are logged.

Reload Python modules when test cases reload

name: shouldReloadPythonModulesWhenReloadingTestCases, type: Bool, default: TRUE

Report directory

name: reportDirectory, type: String

Report name

name: reportName, type: String, default: TestCenterReport

Append timestamp

name: reportNameAddTimestamp, type: Bool, default: FALSE

Append TestCase name

name: reportNameAddTestCaseName, type: Bool, default: FALSE

Timestamp format

name: reportTimestampConfig, type: String, default: _%y_%m_%d__%H_%M

See Python’s strftime function for help.

Collapse all test functions

name: reportCollapseAll, type: Bool, default: FALSE

Show only failed test functions

name: reportShowFailedOnly, type: Bool, default: FALSE

Hide info messages

name: reportHideInfoMessages, type: Bool, default: FALSE

Hide system messages

name: reportHideSystemMessages, type: Bool, default: FALSE

Report name would look like

name: reportNameExample, type: String, persistent: no

Should generate HTML report

name: shouldGenerateHTMLReport, type: Bool, default: TRUE

Should generate JUnit report

name: shouldGenerateJUnitReport, type: Bool, default: FALSE

Format

name: jUnitFormat, type: Enum, default: proprietary

Defines the number of the resulting report files and the used format.

Values:

Title	Name	Description
​Proprietary	​proprietary	​One report file for each test case is created. The test case name is used as file name. The report format is JUnit like: <testsuites errors="0" failures="0" name="NameOfTheTestCase" skipped="0" tests="4" time="0.5"> <testsuite errors="0" failures="0" name="TEST001_Test_Function" skipped="0" tests="2" time="0.5"> <testcase name="Some logging written by the test" /> ... </testsuite> ... </testsuites>
​JUnit	​junit	​All test case results are written to one JUnit file named junitOutput.xml. The report format is JUnit, which can be used in Jenkins: <testsuites tests="4" errors="0" failures="0" time="0.5"> <testsuite tests="4" failures="0" errors="0" skipped="0" time="0.5" name="NameOfTheTestCase"> <testcase name="TEST001_Test_Function" time="0.5"> <system-out>All logging written to std-out by the test</system-out> <system-err>All logging written to err-out by the test</system-err> <failure>All logging that indicates a test failure (e.g., from an EXPECT_EQ())</failure> <error>All logging that indicates an unexpected test error (e.g., a crash)</error> </testcase> ... </testsuite> ... </testsuites>

Max Info Messages Per Test Function

name: maxInfoMessagesPerTestFunction, type: Integer, default: 0

Max Error Messages Per Test Function

name: maxErrorMessagesPerTestFunction, type: Integer, default: 0

Always use current MeVisLab

name: cfgUseCurrent, type: Bool, default: TRUE

If checked, the path from :field:cfgExecutable is ignored.

Instead, the MeVisLab instance from which the test center is called is used.

This option should generally remain enabled. It should only be disabled if you explicitly want to run tests in/from a different MeVisLab installation or if path detection fails for some reason.

MeVisLab Path

name: cfgExecutable, type: String, persistent: no

Sets the path to the installed MeVisLab. This is ignored if Always use current MeVisLab is set.

Options

name: cfgOptions, type: String

Sets command line options to use for secure testing (in addition to required options).

Compile mode

name: cfgCompileMode, type: Enum, default: Automatic, deprecated name: cfgUseDebug

Defines the compile mode of the MeVisLab executable used in secure testing.

Values:

Title	Name	Deprecated Name	Description
​Release	​Release	​TRUE	​Use the fast, optimized Release mode version of MeVisLab.
​Debug	​Debug	​FALSE	​Use the slower, unoptimized Debug version of MeVisLab, which is suitable for C++ debugging.
​Automatic	​Automatic		​Automatically select the same compile mode as the currently running MeVisLab (Release mode if not started from MeVisLab).

New Test Case Name

name: newTestCaseName, type: String

New Test Comment

name: newTestComment, type: String

Network

name: newTestNetworkPath, type: String

New Test Case Type

name: newTestCaseType, type: Enum, default: Inventor

Values:

Title	Name
​Inventor	​Inventor
​ML	​ML
​Macros	​Macros
​Other	​Other

Test Network

name: newTestNetworkMode, type: Enum, default: NewEmptyTestNetwork

Values:

Title	Name
​Empty Network	​NewEmptyTestNetwork
​Import Network	​ImportExistingTestNetwork
​No Network	​NoTestNetwork

Name

name: testCaseSuiteName, type: String, persistent: no

Suite File

name: testCaseSuiteFilename, type: String, persistent: no

Associated Module

name: associatedModule, type: String

Package Identifier

name: packageIdentifier, type: String

Show reports in TestCaseManager panel

name: showReportViewerInPanel, type: Bool, default: FALSE

Hidden Fields

testCaseLoaded

name: testCaseLoaded, type: Bool, persistent: no

testCaseHasNetworkFile

name: testCaseHasNetworkFile, type: Bool, persistent: no, deprecated name: testCaseHasInternal

Shows whether the selected test has an internal network file.

testCaseLoadedAndVerified

name: testCaseLoadedAndVerified, type: Bool, persistent: no

anythingSelected

name: anythingSelected, type: Bool, persistent: no

testCaseSuiteSelected

name: testCaseSuiteSelected, type: Bool, persistent: no

testCaseSelected

name: testCaseSelected, type: Bool, persistent: no

packageSelected

name: packageSelected, type: Bool, persistent: no

codeTestSelected

name: codeTestSelected, type: Bool, persistent: no

codeTestNodeSelected

name: codeTestNodeSelected, type: Bool, persistent: no

codeTestExecutableExists

name: codeTestExecutableExists, type: Bool, persistent: no

testCaseWasExecutedNotSecure

name: testCaseWasExecutedNotSecure, type: Bool, persistent: no, deprecated name: testCaseExecuted

Shows whether the selected test was executed with disabled “Secure Testing”.

bullseyeAvailable

name: bullseyeAvailable, type: Bool, persistent: no

testCaseInfo

name: testCaseInfo, type: String

newTestCaseProject

name: newTestCaseProject, type: String, persistent: no

newTestCaseProjectPath

name: newTestCaseProjectPath, type: String, persistent: no

directoryStructure

name: directoryStructure, type: Enum, persistent: no

Values:

Title	Name
​Classic	​Classic
​Self-contained	​Self-contained

testCaseSuiteTestNames

name: testCaseSuiteTestNames, type: String, persistent: no

associatedModuleExists

name: associatedModuleExists, type: Bool, persistent: no

dialogText

name: dialogText, type: String, persistent: no

progressDialogText

name: progressDialogText, type: String, persistent: no

progressDialogCurrentFunction

name: progressDialogCurrentFunction, type: String, persistent: no

progressDialogStatus

name: progressDialogStatus, type: String, persistent: no

progress

name: progress, type: Float, minimum: 0, maximum: 1, persistent: no

isDisplayingReport

name: isDisplayingReport, type: Bool, persistent: no

ensureReportViewerInPanelIsVisible

name: ensureReportViewerInPanelIsVisible, type: Trigger