This document describes how the autopkgtest tester core (the program adt-run) interprets and executes tests found in Debian source packages.
The source package provides a test metadata file debian/tests/control. This is a file containing zero or more RFC822-style stanzas, along these lines:
Tests: fred bill bongo Restrictions: needs-root breaks-testbed
This example defines three tests, called fred, bill and bongo. The tests will be performed by executing debian/tests/fred, debian/tests/bill, etc. Each test program should, on success, exit with status 0 and print nothing to stderr; if a test exits nonzero, or prints to stderr, it is considered to have failed.
The cwd of each test is guaranteed to be the root of the source package, which will have been unpacked but not built. However note that the tests must test the installed version of the program. Tests may not modify the source tree (and may not have write access to it).
If the file to be executed has no execute bits set, chmod a+x is applied to it (this means that tests can be added in patches without the need for additional chmod; contrast this with debian/rules).
During execution of the test, the environment variable $ADTTMP will point to a directory for the execution of this particular test, which starts empty and will be deleted afterwards (so there is no need for the test to clean up files left there).
If tests want to create artifacts which are useful to attach to test results, such as additional log files or screenshots, they can put them into the directory specified by the $ADT_ARTIFACTS environment variable. When using the --output-dir option, they will be copied into outputdir/artifacts/.
Tests must declare all applicable Restrictions - see below.
The fields which may appear in debian/tests/control stanzas are:
This field is mandatory. It names the tests which are defined by this stanza. All of the other fields in the same stanza apply to all of the named tests.
Test names are separated by whitespace and should contain only characters which are legal in package names. It is permitted, but not encouraged, to use upper-case characters as well.
Declares that the specified packages must be installed for the test to go ahead. This supports all features of dpkg dependencies (see https://www.debian.org/doc/debian-policy/ch-relationships.html), plus the following extensions:
@ stands for the package(s) generated by the source package containing the tests; each dependency (strictly, or-clause, which may contain |s but not commas) containing @ is replicated once for each such binary package, with the binary package name substituted for each @ (but normally @ should occur only once and without a version restriction).
@builddeps@ will be replaced by the package's Build-Depends:, Build-Depends-Indep:, and build-essential. This is useful if you have many build dependencies which are only necessary for running the test suite and you don't want to replicate them in the test Depends:. However, please use this sparingly, as this can easily lead to missing binary package dependencies being overlooked if they get pulled in via build dependencies.
If no Depends field is present, Depends: @ is assumed. Note that the source tree's Build-Dependencies are not necessarily installed, and if you specify any Depends, no binary packages from the source are installed unless explicitly requested.
Replaces the path segment debian/tests in the filenames of the test programs with path. I. e., the tests are run by executing built/source/tree/path/testname. path must be a relative path and is interpreted starting from the root of the built source tree.
This allows tests to live outside the debian/ metadata area, so that they can more palatably be shared with non-Debian distributions.
Any unknown fields will cause the whole stanza to be skipped.
The test, when run, is liable to break the testbed system. This includes causing data loss, causing services that the machine is running to malfunction, or permanently disabling services; it does not include causing services on the machine to temporarily fail.
When this restriction is present the test will usually be skipped unless the testbed's virtualisation arrangements are sufficiently powerful, or alternatively if the user explicitly requests.
The tests need to be run from a built source tree. The test runner will build the source tree (honouring the source package's build dependencies), before running the tests. However, the tests are not entitled to assume that the source package's build dependencies will be installed when the test is run.
Please use this considerately, as for large builds it unnecessarily builds the entire project when you only need a tiny subset (like the tests/ subdirectory). It is often possible to run make -C tests instead, or copy the test code to $ADTTMP and build it there with some custom commands. This cuts down the load on the Continuous Integration servers and also makes tests more robust as it prevents accidentally running them against the built source tree instead of the installed packages.
There are no currently defined Features.
To allow test execution environments to discover packages which provide tests, their source packages should have a Testsuite: header containing autopkgtest (which is currently the only defined value). Multiple values get comma separated, as usual in control files.
This tag can be set manually in debian/control by adding
XS-Testsuite: autopkgtest
in the Source: paragraph. Future versions of dpkg-source might add this automatically when a debian/tests/control file is present.