TestCenter Manual

Test cases use the MeVisLab infrastructure. Similar to the database for modules, there is a database for test cases. This database is filled with entries by searching for *.def files in the TestCases subdirectories of all packages. Each *.def file may contain the definition of multiple test cases.

The two test types are defined by blocks of source like the following:

FunctionalTestCase FunctionalExample {
  scriptFile = $(LOCAL)/FunctionalExample.py
}

GenericTestCase GenericExample {
  scriptFile = $(LOCAL)/GenericExample.py
}

The given example creates a functional and a generic test case. The information provided is collected in an entry that can be accessed using the testCaseInfo method of the TestCaseDatabase.

The scriptFile key specifies which file contains the actual test functions (see Section 2.1.2, “Implementing the Test Functions”). The script files are used to load the actual test cases, which behave like macro modules in the MeVisLab context. This has the benefit that a network named after the test case and located in the same folder (for example $(LOCAL)/GenericExample.mlab) is loaded as part of the context, too.

All test cases must reside in a package's TestCases directory. It is recommended to use a substructure similar to the modules directory (for example TestCases/FunctionalTests/[module_type]/[module_name]).

The location of generic tests is important as the set of modules being tested by them depends on the package they are in.

Test cases are scripted in Python. Each test case consists of an arbitrary number of test functions. These functions all have their own status. The worst status of a test case's functions will determine the overall status, see Section 2.1.4, “Logging and Status Detection”.

Similar to the Google test framework, test functions are given a special prefix like TEST.

There are three major types of test functions that can be used: simple, iterative and field-value test functions. In addition to this, multiple simple test functions can be combined to a group. The following sections describe the available test function prefixes and types.

Two special functions are available that can be used to maintain global information for a test case. They are called setUpTestCase and tearDownTestCase and are called prior to the execution of the first test function and after the last, respectively. The outputs of setUpTestCase or tearDownTestCase are appended to the first / last test function.

	Note
	Please note that this is different from what is common in other unit-testing frameworks; there, the setup function would usually be called prior to every test function and teardown after every function.

A testing framework must somehow determine whether a test has failed or succeeded.

The MeVisLab debug console shows messages with a type. All sources of output are mapped to this console: standard output, standard error and other special methods provided inside MeVisLab for logging. Based on this information, the TestCenter determines the current test function status.

Aside of these status types, there are the TestCenter-specific types “Timeout” and “Crash”. They are used for situations where the testing fails. For more information on failure handing, see Section 2.2, “Master-Slave Approach”.

The test case status is determined by the worst status of one of its test functions. In addition to this, the TestCenter collects all messages printed in the debug console to display them in the reports for detailed introspection on what happened.

The following example shows a simple test function that verifies a condition and prints an error message if the condition was not met. This results in the test function having the status “Error”.

import mevis
def TEST_Simple_Test ():
  if not condition:
    mevis.MLAB.logError("The condition wasn't met!")
  else:
    mevis.MLAB.log("Everything is fine.")

As the mechanism leads to messages being collected from resources besides the test functions, it allows detecting all possible problems occurring while testing. For example, if an ML module fails and prints messages of type “ML Error”, this will result in the test function being marked as failed.

Messages coming from the TestCenter itself may look different to messages from modules. This way, irrelevant parts of the messages can be filtered out more easily. The distinction is done using special logging methods inside the TestCenter which are part of the TestSupport Python package (see Section 2.3, “TestSupport Python Package”). Example:

from TestSupport import Logging
def TEST_Simple_Test ():
  if not condition:
    Logging.error("The condition wasn't met!")
  else:
    Logging.info("Everything is fine.")

	Tip
	Other examples for info output methods would be print "The variable has changed." and MLAB.log("The variable has changed."), but using the Logging Python module makes it easy to distinguish the test as message source.

During a test, the environment is usually changed. For example, a test function would set field values and touch triggers to start a computation based on the set values. This is a problem when the next test function is influenced by these changes.

In the TestCenter, the problem is solved by the TestSupport.Fields Python module. It provides methods to set a field value in a way that makes resetting the original values possible.

The foundation is the ChangeSet concept (see the TestCenterReference, ChangeSet.py). It relies on the idea that a subset of input field values are set to parameterize the following computation. Setting the input values to their original values should be sufficient to (re)set the environment to an expected state. This is only heuristic!

	Note
	The ChangeSets only work for field value changes made using the according methods. They have no effect on indirect changes, for example changes due to the computation started by touching trigger buttons.

The TestCenter itself maintains a stack of such ChangeSets to split the TestCase execution into logical blocks. A first ChangeSet is pushed while loading a TestCase and popped while destructing it. When running a test function, a second ChangeSet is pushed before and popped afterwards.

	Tip
	You can also use the stack to push your own ChangeSets (see the pushChangeSet and popChangeSet methods of the TestSupport.Base Python module, see Section 2.3, “TestSupport Python Package”). Make sure to clean up the stack afterwards, otherwise error messages will be printed.

The output values could also be input values of the next computation. However, this requires a special handling like a manual reset to circumvent the ChangeSet concept.

Test cases can be grouped into test case suites. This allows to run several related tests via the TestCaseManager. Test case suites can be declared in any *.def file below the TestCases subdirectory of any packages. Each *.def file may contain the definition of multiple test case suites and test cases.

The following code shows how to define a TestCaseSuite:

TestCaseSuite ExampleTestCaseSuite {
  testCases = "ExampleTest1, ExampleTest2, ..."
}

The available test case suites are listed in the TestCaseManager and can be selected and executed. The result report will show an overview of all executed tests and links to the individual tests.

In almost all cases, testing requires verifying computed results against some sort of ground truth data. This data could be anything from simple numbers to images. While simple data types are part of the test case scripting, files must be made available for the testing process.

2.4.1.2. Using an external data storage

	Note
	Never try to connect to a network share directly via a fixed URL like \\fileserver\testcenter-data as this is not supported on all platforms and will cause problems if the resource is not available.

To connect external data storage, use the ctx.expandFilename method. It expands variables either predefined or as set in the MeVisLab preferences file. This way, the path to the share can be given in the *.prefs file and the share can be loaded on all platforms. Having the share defined in the *.prefs file instead of a dozen test files also makes changing the path much easier, for example if the share has to move.

The following *.prefs file would define two shares for TestCenter data that for usage on windows:

Settings {
  TESTCENTER_EXTERNAL_DATA_MMS = \\fs.mms.example\tc
  TESTCENTER_EXTERNAL_DATA_FME = \\fs.fme.example\tc
}

With these two variables set, the filename ${TESTCENTER_EXTERNAL_DATA_MMS}/testcasename/filename can be used to get access to the given filename using the expandFilename method to get the actual path.

	Note
	expandFilename will only expand known variables. If a test case is instanciated on a site that does not have access to the resource, the variable is not specified and the result will be an invalid path and an error. To prevent problems, use the Base.getExternalDataDirectory method that returns NULL if the variable is not defined, and the path otherwise.

Test cases must be able to create results. The result files must be saved and added to the report. The location for saving files should be retrieved using the getResultDirectory method. Using other directories is possible, but not recommended. Only the retrieved result directories will be cleaned up when starting the next test run.

The files created in test functions will not appear in the report automatically. They must be added explicitly using one of the following two methods:

Both methods will add special messages to the logfile. The result handlers will detect these messages, copy the linked files, and add them to the report.

The master will collect all the results the slave created while testing and store them for later usage. As local and automatic testing have different requirements for presenting the results, the master uses an XML file as an intermediate representation of the results. Results are saved to a file after all tests have been finished.

The structure of the result file is mostly taken from the responses sent by the slave. It contains information on all executed test cases, all modules being tested, and the results of the actual tests. This information is used to create a detailed report.

	Note
	The further processing of the XML file is not up to the TestCenter but the calling mechanism. For local testing there are methods that convert to static HTML pages. For automatic testing, those results are written to a database. The presentation of the results is therefore up to another entity. For more information see Chapter 4, TestCenter Reports.

	Tip
	For a hands-on introduction to working with the TestCenter, see the Getting Started, “Using the TestCenter”.

The testcase-centric TestCaseManager is started via File → Run TestCaseManager, or by adding the TestCenterManager macro module to the network. It is used to create, edit and run functional test cases and set some basic TestCenter configuration. For the configuration file, see Chapter 5, TestCenter Configuration File.

The TestCenterManager offers an option to use the same instance of MeVisLab (leave Secure testing unchecked). This is much faster than starting a separate instance but has the drawback that crashes will take down the testing MeVisLab instance.

The TestCaseManager incorporates its own result viewer that uses the Webkit integration in MeVisLab to show the resulting static HTML files, see Section 4.1, “Static HTML Pages”. Its advantage is that it can handle special schemes (part of the links), for example to open files at the according line.

For standard browsers, these schemes will cause display problems. To avoid this, go to TestCaseManager → Test Reports, select a report and click the button Create External to create HTML output suitable for standard browsers.

The Module Tests functionality is module-centric. It runs the test cases available for a set of modules. This includes the generic test cases that apply to the given modules and the functional test cases that are associated with them. The connection between a module and a test case is done using the associatedTests keyword in the *.def file of a module.

The Module Tests functionality can be used on three levels inside MeVisLab:

The results generated with the Module Tests functionality are converted to static HTML pages and displayed with the result viewer integrated into the TestCaseManager module.