TestSuite

Portland Test Suite

In xdg-utils a test suite is included.

Running the testsuite is easy:

View the submitted test reports for the latest version, view all reports or submit your own test reports.

More about WritingXdgTests.

Running Tests

You can help improving xdg-utils by simply running the tests on your favourite distribution and submitting the resulting test report.

Tests must be run as a normal (non-root) user for maximum coverage. Preferably they should be run in a terminal window opened by the logged in desktop user. Tests will then try to su (or sudo) to root to verify system options are correct.

Some tests are interactive, since it is otherwise difficult to verify the correct user experience for the end user.

See also the README.

The behavior of the testrunner can be modified. Do './testrun -h' to see the options. Apart from running the entire test suite it is also possible to run tests individually, or in groups by specifying the tests or test directories of the tests you wish to run. Additionally, individual tests can be run by themselves on the command line, provided the xdg-utils can be found in $PATH and $XDG_TEST_DIR is set to the directory containing the test runner and test directories.

Test Results

Tests return one of the following:

PASS

Test returned the correct result

FAIL

The behavior being tested did not return the correct result.

NORESULT

Indicates that some prerequisite did not work as expected. For example, if we are testing the uninstall feature and the install feature failed, the uninstall feature cannot be tested. A NORESULT result needs investigation so that the "real" test can be run.

UNTESTED

Indicates that the test was not run. This is OK. For example, it is only appropriate to run some tests as root, so if it is not running as root, this will be the result.

WARN

Extra unexpected output was found on STDERR or STDOUT. WARN may also occur if test failed, but the behavior being tested is not well defined. This may indicate a problem with the specification and should be examined and discussed.

Test Cleanup

Tests should clean up after themselves. However, if the test is interrupted or dies unexpectedly, it can leave test files all over the filesystem. Running 'make tests-clean' as root will clean up most files that the test installed on the system. (If no tests ran as root, 'make tests-userclean' is a quicker version.)

Note that tests may result in the creation of certain directories or cache-files which can have an impact on future system behavior. In particular, tests can fail on a "fresh system" but then succeed from that point on, or after restarting the desktop session. Such failures should be reported as they represent a problem with the operating system and/or desktop environment and cause a serious degradation of the user experience when the end user attempts to install applications on a freshly installed system.