=encoding utf-8

=head1 NAME

Lintian::Tutorial::TestSuite -- Quick intro to running the Lintian testsuite

=head1 SYNOPSIS

This guide will quickly introduce you to running the Lintian test
suite and some tricks.  The Lintian test suite is fairly large and
accordingly it can take a substantial amount of time to run.  To speed
up development, there are various options to limit the tests to run.

=head1 DESCRIPTION

The Lintian test suite is an extensive collection of various test
cases.  The test suite is divided into 4 "sub-suites".  The majority
of all tests are currently located in the "tests" sub-suite.

To run the full suite in all of its glory, simply invoke:

 $ debian/rules runtests

 OR

 $ mkdir -p debian/test-out
 $ t/runtests -k --dump-logs t debian/test-out

But while writing a new tag (or check) you probably only want to run a
particular (subset of the) test(s).  See L</Running a subset of the
tests> for the available options.

When run via I<debian/rules>, the test suite respects the
"DEB_BUILD_OPTIONS=parallel=N". When using I<t/runtests> directly, use
I<-jN> to choose the number of threads.  Note that "N" denotes the
amount of "worker" threads.  The test runner will generally have 2
threads more than that.  Also each "worker" will run lintian, which
runs multiple unpacking jobs in parallel as well.

In case you are wondering about the 2 extra threads in the test
runner, the first of them is the "coordinator"-thread (which will
generally be waiting when the workers are active) and the second one
is the "output"-thread (which handles the fancy output).

=head2 Running a subset of the tests

The following options are available:

=over 4

=item Running a single test

To run a single test by its name, use:

 $ debian/rules runtests onlyrun=$name

 OR

 $ t/runtests --dump-logs t debian/test-out $name

=item Running all tests for a check

To run all tests for a given check, use:

 $ debian/rules runtests onlyrun=$check

 OR

 $ t/runtests --dump-logs -k t debian/test-out $check

$check must be the name of a check (it will test for
checks/$check.desc) or "legacy".  This will run all tests that starts
with "$check-".

Note: The "changes" sub-suite in the new test suite does not support
this.

=item Running all tests in a sub-suite

To run all tests in a given sub-suite, use:

 $ debian/rules runtests onlyrun=suite:$suites

 OR

 $ t/runtests --dump-logs -k t debian/test-out suite:$suites

$suites is a comma-separated list of names of sub-suites to run.

Note: this cannot be used to influence the order, in which the sub-suites
are run.

=item Running all tests designed for a specific tag

To run all tests that has a "Test-For" or a "Test-Against" for a given
tag, use:

 $ debian/rules runtests onlyrun=tag:$tag

 OR

 $ t/runtests --dump-logs -k t debian/test-out tag:$tag

=back

=head2 Running tests under coverage

It is possible to run most of the tests under L<Devel::Cover>.  This is
done by passing I<--coverage> to I<t/runtests>.  Example:

  $ t/runtests --coverage --dump-logs -j1 -k t debian/test-out

Please note that L<Devel::Cover> does not seem to handle multiple
threads too well.  You may see spurious warnings/errors if you run the
tests with 2 or more active worker threads.

B<Caveat> 1: Coverage for collections (i.e. programs in I<collection/>)
does not seem to work at the moment. Therefore, they often end up with
(next to) zero coverage in the generated reports.

B<Caveat> 2: L<Devel::Cover> sometimes changes the output of Lintian
or tools called by Lintian. Obviously, this leads to test
failures. Therefore, you may see weird test failures (or warnings)
when running under coverage.

=head3 Collecting the coverage you want in a reasonable time

Collecting coverage is excruciatingly slow.  This is not helped by the
fact that it becomes unreliable when run under 2 or more threads.

Fortunately, L<Devel::Cover> "appends" to its cover database.  This
allows you to "slowly" build up the coverage database over multiple
runs. Example:

  $ t/runtests --coverage --dump-logs -j1 -k t debian/test-out suite:scripts
  $ t/runtests --coverage --dump-logs -j1 -k t debian/test-out suite:debs
  $ t/runtests --coverage --dump-logs -j1 -k t debian/test-out suite:source
  ...

Or:

  $ t/runtests --coverage --dump-logs -j1 -k t debian/test-out $check
  $ t/runtests --coverage --dump-logs -j1 -k t debian/test-out legacy


=cut

