CMDTEST(1)

NAME
cmdtest - blackbox testing of Unix command line tools

SYNOPSIS

DESCRIPTION
cmdtest black box tests Unix command line tools. Given some test scripts, their inputs, and expected outputs, it verifies that the command line produces the expected output. If not, it reports problems, and shows the differences.

Each test case foo consists of the following files:

foo.script
    a script to run the test (this is required)

foo.stdin
    the file fed to standard input

foo.stdout
    the expected output to the standard output

foo.stderr
    the expected output to the standard error

foo.exit
    the expected exit code

foo.setup
    a shell script to run before the test

foo.teardown
    a shell script to run after test

Usually, a single test is not enough. All tests are put into the same directory, and they may share some setup and teardown code:

setup-once
    a shell script to run once, before any tests

setup
    a shell script to run before each test

teardown
    a shell script to run after each test

teardown-once
    a shell script to run once, after all tests

cmdtest is given the name of the directory with all the tests, or several such directories, and it does the following:

• execute setup-once

• for each test case (unique prefix foo):

    — execute setup

    — execute foo.setup

    — execute the command, by running foo.script, and redirecting standard input to come from foo.stdin, and capturing standard output and error and exit codes

    — execute foo.teardown

    — execute teardown

    — report result of test: does exit code match foo.exit, standard output match foo.stdout, and standard error match foo.stderr?

• execute teardown-once

Except for foo.script, all of these files are optional. If a setup or teardown script is missing, it is simply not executed. If one of the standard input, output, or error files is missing, it is treated as if it were empty. If the exit code file is missing, it is treated as if it specified an exit code of zero.

The shell scripts may use the following environment variables:

DATADIR
    a temporary directory where files may be created by the test

TESTNAME
    name of the current test (will be empty for setup-once and teardown-once)

SRCDIR
    directory from which cmdtest was launched

OPTIONS

EXAMPLE
To test that the echo(1) command outputs the expected string, create a file called echo-tests/hello.script containing the following content:

    #!/bin/sh
    echo hello, world

Also create the file echo-tests/hello.stdout containing:

    hello, world

Then you can run the tests:

    $ cmdtest echo-tests
    test 1/1
    1/1 tests OK, 0 failures

If you change the stdout file to be something else, cmdtest will report the differences:

    $ cmdtest echo-tests
    FAIL: hello: stdout diff:
    --- echo-tests/hello.stdout	2011-09-11 19:14:47 +0100
    +++ echo-tests/hello.stdout-actual	2011-09-11 19:14:49 +0100
    @@ -1 +1 @@
    -something else
    +hello, world
    test 1/1
    0/1 tests OK, 1 failures

Furthermore, the echo-tests directory will contain the actual output files, and diffs from the expected files. If one of the actual output files is actually correct, you can actualy rename it to be the expected file. Actually, that's a very convenient way of creating the expected output files: you run the test, fixing things, until you've manually checked the actual output is correct, then you rename the file.

SEE ALSO
cliapp(5).