TestCaseManager

MacroModule
genre Test
author MeVis Medical Solutions AG
package MeVisLab/Standard
definition TestCaseManager.def
keywords testing, functional

Purpose

Detailed documentation about the TestCaseManager and the TestCenter can be found on the MeVisLab Help Overview page. See Developing a Test Case and TestCenter Manual.

Windows

Default Panel

../../../Modules/Macros/Tests/TestCenterAdvanced/mhelp/Images/Screenshots/TestCaseManager._default.png

Parameter Fields

Field Index

Always use current MeVisLab: Bool Max Error Messages Per Test Function: Integer Report name would look like: String
anythingSelected: Bool Max Info Messages Per Test Function: Integer Secure testing: Bool
Append TestCase name: Bool MeVisLab Path: String Should generate HTML report: Bool
Append timestamp: Bool Name: String Should generate JUnit report: Bool
Associated Module: String Network: String Show only failed test functions: Bool
associatedModuleExists: Bool New Test Case Name: String Show reports in TestCaseManager panel: Bool
Bullseye coverage: Bool New Test Case Type: Enum Stop and inspect network on error: Bool
bullseyeAvailable: Bool New Test Comment: String Suite File: String
codeTestExecutableExists: Bool newTestCaseProject: String Test Network: Enum
codeTestNodeSelected: Bool newTestCaseProjectPath: String testCaseHasNetworkFile: Bool
codeTestSelected: Bool Options: String testCaseInfo: String
Collapse all test functions: Bool Package Identifier: String testCaseLoaded: Bool
Compile mode: Enum packageSelected: Bool testCaseLoadedAndVerified: Bool
dialogText: String progress: Float testCaseSelected: Bool
directoryStructure: Enum progressDialogCurrentFunction: String testCaseSuiteSelected: Bool
ensureReportViewerInPanelIsVisible: Trigger progressDialogStatus: String testCaseSuiteTestNames: String
Format: Enum progressDialogText: String testCaseWasExecutedNotSecure: Bool
Hide info messages: Bool Python Coverage: Enum Timestamp format: String
Hide system messages: Bool Reload Python modules when test cases reload: Bool Use MESA for secure testing: Bool
isDisplayingReport: Bool Report directory: String  
Log info messages: Bool Report name: String  

Visible Fields

Python Coverage

name: pythonCoverage, type: Enum, default: Disabled

If checked, for each test a coverage report will be generated for all utilized python modules loaded so far.

Note

The Python coverage report might list modules from the Python cache that were used earlier but are not relevant to the current test. Only the initialization code of these unrelated modules will appear as executed in the report. Technically, this is correct because all modules from the cache are reloaded before each test run. At this point, it is not possible to distinguish between the relevant modules for the test and other modules.

To avoid this issue, you can use ‘Secure testing’, which starts a new MeVisLab instance for each test, ensuring a clean Python cache.

Values:

Title Name Deprecated Name
Disabled Disabled FALSE
Per Test Case PerTestCase TRUE
Global Global  

Bullseye coverage

name: bullseyeCoverage, type: Bool, default: FALSE

If checked, the currently instrumented codes in the COVFILE are used to measure coverage values on a per-test basis for each test.

For this to work, Bullseye Coverage must be installed, and the path to its binaries must be included in your PATH environment variables.

Additionally, this only works correctly with SecureTesting enabled.

Secure testing

name: secureTesting, type: Bool, default: FALSE

If checked, a separate MeVisLab process for running the test(s) is started.

This uses the settings Always use current MeVisLab, MeVisLab Path, Options, Compile mode, and Use MESA for secure testing from the Configuration tab.

Use MESA for secure testing

name: useMesaForSecureTesting, type: Bool, default: FALSE

If checked, MeVisLab will try to use the MESA software OpenGL driver when running tests in Secure testing (instead of the system-supplied driver).

This only works on Windows.

Stop and inspect network on error

name: stopOnError, type: Bool, default: FALSE

If checked, the test execution stops on the first error and the active test network is opened in MeVisLab.

Log info messages

name: logInfoMessages, type: Bool, default: TRUE

If checked, all informational messages are logged.

Reload Python modules when test cases reload

name: shouldReloadPythonModulesWhenReloadingTestCases, type: Bool, default: TRUE

Report directory

name: reportDirectory, type: String

Report name

name: reportName, type: String, default: TestCenterReport

Append timestamp

name: reportNameAddTimestamp, type: Bool, default: FALSE

Append TestCase name

name: reportNameAddTestCaseName, type: Bool, default: FALSE

Timestamp format

name: reportTimestampConfig, type: String, default: _%y_%m_%d__%H_%M

See Python’s strftime function for help.

Collapse all test functions

name: reportCollapseAll, type: Bool, default: FALSE

Show only failed test functions

name: reportShowFailedOnly, type: Bool, default: FALSE

Hide info messages

name: reportHideInfoMessages, type: Bool, default: FALSE

Hide system messages

name: reportHideSystemMessages, type: Bool, default: FALSE

Report name would look like

name: reportNameExample, type: String, persistent: no

Should generate HTML report

name: shouldGenerateHTMLReport, type: Bool, default: TRUE

Should generate JUnit report

name: shouldGenerateJUnitReport, type: Bool, default: FALSE

Format

name: jUnitFormat, type: Enum, default: proprietary

Defines the number of the resulting report files and the used format.

Values:

Title Name Description
Proprietary proprietary

One report file for each test case is created. The test case name is used as file name. The report format is JUnit like:

<testsuites errors="0" failures="0" name="NameOfTheTestCase" skipped="0" tests="4" time="0.5">
     <testsuite errors="0" failures="0" name="TEST001_Test_Function" skipped="0" tests="2" time="0.5">
          <testcase name="Some logging written by the test" />
          ...
     </testsuite>
...
</testsuites>
JUnit junit

All test case results are written to one JUnit file named junitOutput.xml. The report format is JUnit, which can be used in Jenkins:

<testsuites tests="4" errors="0" failures="0" time="0.5">
    <testsuite tests="4" failures="0" errors="0" skipped="0" time="0.5" name="NameOfTheTestCase">
        <testcase name="TEST001_Test_Function" time="0.5">
            <system-out>All logging written to std-out by the test</system-out>
            <system-err>All logging written to err-out by the test</system-err>
            <failure>All logging that indicates a test failure (e.g., from an EXPECT_EQ())</failure>
            <error>All logging that indicates an unexpected test error (e.g., a crash)</error>
        </testcase>
        ...
    </testsuite>
    ...
</testsuites>

Max Info Messages Per Test Function

name: maxInfoMessagesPerTestFunction, type: Integer, default: 0

Max Error Messages Per Test Function

name: maxErrorMessagesPerTestFunction, type: Integer, default: 0

Always use current MeVisLab

name: cfgUseCurrent, type: Bool, default: TRUE

If checked, the path from :field:cfgExecutable is ignored.

Instead, the MeVisLab instance from which the test center is called is used.

This option should generally remain enabled. It should only be disabled if you explicitly want to run tests in/from a different MeVisLab installation or if path detection fails for some reason.

MeVisLab Path

name: cfgExecutable, type: String, persistent: no

Sets the path to the installed MeVisLab. This is ignored if Always use current MeVisLab is set.

Options

name: cfgOptions, type: String

Sets command line options to use for secure testing (in addition to required options).

Compile mode

name: cfgCompileMode, type: Enum, default: Automatic, deprecated name: cfgUseDebug

Defines the compile mode of the MeVisLab executable used in secure testing.

Values:

Title Name Deprecated Name Description
Release Release TRUE Use the fast, optimized Release mode version of MeVisLab.
Debug Debug FALSE Use the slower, unoptimized Debug version of MeVisLab, which is suitable for C++ debugging.
Automatic Automatic   Automatically select the same compile mode as the currently running MeVisLab (Release mode if not started from MeVisLab).

New Test Case Name

name: newTestCaseName, type: String

New Test Comment

name: newTestComment, type: String

Network

name: newTestNetworkPath, type: String

New Test Case Type

name: newTestCaseType, type: Enum, default: Inventor

Values:

Title Name
Inventor Inventor
ML ML
Macros Macros
Other Other

Test Network

name: newTestNetworkMode, type: Enum, default: NewEmptyTestNetwork

Values:

Title Name
Empty Network NewEmptyTestNetwork
Import Network ImportExistingTestNetwork
No Network NoTestNetwork

Name

name: testCaseSuiteName, type: String, persistent: no

Suite File

name: testCaseSuiteFilename, type: String, persistent: no

Associated Module

name: associatedModule, type: String

Package Identifier

name: packageIdentifier, type: String

Show reports in TestCaseManager panel

name: showReportViewerInPanel, type: Bool, default: FALSE

Hidden Fields

testCaseLoaded

name: testCaseLoaded, type: Bool, persistent: no

testCaseHasNetworkFile

name: testCaseHasNetworkFile, type: Bool, persistent: no, deprecated name: testCaseHasInternal

Shows whether the selected test has an internal network file.

testCaseLoadedAndVerified

name: testCaseLoadedAndVerified, type: Bool, persistent: no

anythingSelected

name: anythingSelected, type: Bool, persistent: no

testCaseSuiteSelected

name: testCaseSuiteSelected, type: Bool, persistent: no

testCaseSelected

name: testCaseSelected, type: Bool, persistent: no

packageSelected

name: packageSelected, type: Bool, persistent: no

codeTestSelected

name: codeTestSelected, type: Bool, persistent: no

codeTestNodeSelected

name: codeTestNodeSelected, type: Bool, persistent: no

codeTestExecutableExists

name: codeTestExecutableExists, type: Bool, persistent: no

testCaseWasExecutedNotSecure

name: testCaseWasExecutedNotSecure, type: Bool, persistent: no, deprecated name: testCaseExecuted

Shows whether the selected test was executed with disabled “Secure Testing”.

bullseyeAvailable

name: bullseyeAvailable, type: Bool, persistent: no

testCaseInfo

name: testCaseInfo, type: String

newTestCaseProject

name: newTestCaseProject, type: String, persistent: no

newTestCaseProjectPath

name: newTestCaseProjectPath, type: String, persistent: no

directoryStructure

name: directoryStructure, type: Enum, persistent: no

Values:

Title Name
Classic Classic
Self-contained Self-contained

testCaseSuiteTestNames

name: testCaseSuiteTestNames, type: String, persistent: no

associatedModuleExists

name: associatedModuleExists, type: Bool, persistent: no

dialogText

name: dialogText, type: String, persistent: no

progressDialogText

name: progressDialogText, type: String, persistent: no

progressDialogCurrentFunction

name: progressDialogCurrentFunction, type: String, persistent: no

progressDialogStatus

name: progressDialogStatus, type: String, persistent: no

progress

name: progress, type: Float, minimum: 0, maximum: 1, persistent: no

isDisplayingReport

name: isDisplayingReport, type: Bool, persistent: no

ensureReportViewerInPanelIsVisible

name: ensureReportViewerInPanelIsVisible, type: Trigger