Good Integration Practises

Work with virtual environments

We recommend to use virtualenv environments and use easy_install (or pip) for installing your application dependencies as well as the pytest package itself. This way you will get a much more reproducible environment. A good tool to help you automate test runs against multiple dependency configurations or Python interpreters is tox.

Use tox and Continuous Integration servers

If you frequently release code to the public you may want to look into tox, the virtualenv test automation tool and its pytest support. The basic idea is to generate a JUnitXML file through the --junitxml=PATH option and have a continuous integration server like Jenkins pick it up and generate reports.

Create a py.test standalone script

If you are a maintainer or application developer and want others to easily run tests you can generate a completely standalone “py.test” script:

py.test --genscript=runtests.py

generates a runtests.py script which is a fully functional basic py.test script, running unchanged under Python2 and Python3. You can tell people to download the script and then e.g. run it like this:

python runtests.py

Integrating with distutils / python setup.py test

You can integrate test runs into your distutils or setuptools based project. Use the genscript method to generate a standalone py.test script:

py.test --genscript=runtests.py

and make this script part of your distribution and then add this to your setup.py file:

from distutils.core import setup, Command
# you can also import from setuptools

class PyTest(Command):
    user_options = []
    def initialize_options(self):
        pass
    def finalize_options(self):
        pass
    def run(self):
        import sys,subprocess
        errno = subprocess.call([sys.executable, 'runtests.py'])
        raise SystemExit(errno)
setup(
    #...,
    cmdclass = {'test': PyTest},
    #...,
)

If you now type:

python setup.py test

this will execute your tests using runtests.py. As this is a standalone version of py.test no prior installation whatsoever is required for calling the test command. You can also pass additional arguments to the subprocess-calls such as your test directory or other options.

Integration with setuptools/distribute test commands

Distribute/Setuptools support test requirements, which means its really easy to extend its test command to support running a pytest from test requirements:

from setuptools.command.test import test as TestCommand
import sys

class PyTest(TestCommand):
    def finalize_options(self):
        TestCommand.finalize_options(self)
        self.test_args = []
        self.test_suite = True
    def run_tests(self):
        #import here, cause outside the eggs aren't loaded
        import pytest
        errno = pytest.main(self.test_args)
        sys.exit(errno)

setup(
    #...,
    tests_require=['pytest'],
    cmdclass = {'test': PyTest},
    )

Now if you run:

python setup.py test

this will download py.test if needed and then run py.test as you would expect it to.

Conventions for Python test discovery

py.test implements the following standard test discovery:

• collection starts from the initial command line arguments which may be directories, filenames or test ids.
• recurse into directories, unless they match norecursedirs
• test_*.py or *_test.py files, imported by their package name.
• Test prefixed test classes (without an __init__ method)
• test_ prefixed test functions or methods are test items

For examples of how to customize your test discovery Changing standard (Python) test discovery.

Within Python modules, py.test also discovers tests using the standard unittest.TestCase subclassing technique.

Choosing a test layout / import rules

py.test supports common test layouts:

• inlining test directories into your application package, useful if you want to keep (unit) tests and actually tested code close together:

  mypkg/
      __init__.py
      appmodule.py
      ...
      test/
          test_app.py
          ...

• putting tests into an extra directory outside your actual application code, useful if you have many functional tests or want to keep tests separate from actual application code:

  mypkg/
      __init__.py
      appmodule.py
  tests/
      test_app.py
      ...

In both cases you usually need to make sure that mypkg is importable, for example by using the setuptools python setup.py develop method.

You can run your tests by pointing to it:

py.test tests/test_app.py       # for external test dirs
py.test mypkg/test/test_app.py  # for inlined test dirs
py.test mypkg                   # run tests in all below test directories
py.test                         # run all tests below current dir
...

Note

If py.test finds a “a/b/test_module.py” test file while recursing into the filesystem it determines the import name as follows:

• find basedir – this is the first “upward” (towards the root) directory not containing an __init__.py. If both the a and b directories contain an __init__.py the basedir will be the parent dir of a.
• perform sys.path.insert(0, basedir) to make the test module importable under the fully qualified import name.
• import a.b.test_module where the path is determined by converting path separators / into ”.” characters. This means you must follow the convention of having directory and file names map directly to the import names.

The reason for this somewhat evolved importing technique is that in larger projects multiple test modules might import from each other and thus deriving a canonical import name helps to avoid surprises such as a test modules getting imported twice.