sqlite/doc/testrunner.md

The testrunner.tcl Script

• 1. Overview
• 2. Binary Tests
  • 2.1. Organization of Tcl Tests
  • 2.2. Commands to Run Tests
  • 2.3. Investigating Binary Test Failures
  • 3. Source Tests
    • 3.1. Commands to Run SQLite Tests
    • 3.2. Running ZipVFS Tests
    • 3.3. Investigating Source Code Test Failures
    • 4. Extra testrunner.tcl Options
    • 5. Controlling CPU Core Utilization

    1. Overview

    testrunner.tcl is a Tcl script used to run multiple SQLite tests using multiple jobs. It supports the following types of tests:

    • Tcl test scripts.

    • Tests run with make commands. Examples:

      • make mdevtest
      • make releasetest
      • make sdevtest
      • make testrunner

    testrunner.tcl pipes the output of all tests and builds run into log file testrunner.log, created in the current working directory. Search this file to find details of errors. Suggested search commands:

    • grep "^!" testrunner.log
    • grep failed testrunner.log

    testrunner.tcl also populates SQLite database testrunner.db. This database contains details of all tests run, running and to be run. A useful query might be:

      SELECT * FROM script WHERE state='failed'
    

    Running the command:

      ./testfixture $(TESTDIR)/testrunner.tcl status
    

    in the directory containing the testrunner.db database runs various queries to produce a succinct report on the state of a running testrunner.tcl script. Running:

      watch ./testfixture $(TESTDIR)/testrunner.tcl status
    

    in another terminal is a good way to keep an eye on a long running test.

    Sometimes testrunner.tcl uses the testfixture binary that it is run with to run tests (see "Binary Tests" below). Sometimes it builds testfixture and other binaries in specific configurations to test (see "Source Tests").

    2. Binary Tests

    The commands described in this section all run various combinations of the Tcl test scripts using the testfixture binary used to run the testrunner.tcl script (i.e. they do not invoke the compiler to build new binaries, or the make command to run tests that are not Tcl scripts). The procedure to run these tests is therefore:

    1. Build the "testfixture" (or "testfixture.exe" for windows) binary using whatever method seems convenient.

    2. Test the binary built in step 1 by running testrunner.tcl with it, perhaps with various options.

    The following sub-sections describe the various options that can be passed to testrunner.tcl to test binary testfixture builds.

    2.1. Organization of Tcl Tests

    Tcl tests are stored in files that match the pattern *.test. They are found in both the $TOP/test/ directory, and in the various sub-directories of the $TOP/ext/ directory of the source tree. Not all *.test files contain Tcl tests - a handful are Tcl scripts designed to invoke other *.test files.

    The veryquick set of tests is a subset of all Tcl test scripts in the source tree. In includes most tests, but excludes some that are very slow. Almost all fault-injection tests (those that test the response of the library to OOM or IO errors) are excluded. It is defined in source file test/permutations.test.

    The full set of tests includes all Tcl test scripts in the source tree. To run a "full" test is to run all Tcl test scripts that can be found in the source tree.

    File permutations.test defines various test "permutations". A permutation consists of:

    • A subset of Tcl test scripts, and

    • Runtime configuration to apply before running each test script (e.g. enabling auto-vacuum, or disable lookaside).

    Running all tests is to run all tests in the full test set, plus a dozen or so permutations. The specific permutations that are run as part of "all" are defined in file testrunner_data.tcl.

    2.2. Commands to Run Tests

    To run the "veryquick" test set, use either of the following:

      ./testfixture $TESTDIR/testrunner.tcl
      ./testfixture $TESTDIR/testrunner.tcl veryquick
    

    To run the "full" test suite:

      ./testfixture $TESTDIR/testrunner.tcl full
    

    To run the subset of the "full" test suite for which the test file name matches a specified pattern (e.g. all tests that start with "fts5"), either of:

      ./testfixture $TESTDIR/testrunner.tcl fts5%
      ./testfixture $TESTDIR/testrunner.tcl 'fts5*'
    

    Strictly speaking, for a test to be run the pattern must match the script filename, not including the directory, using the rules of Tcl's [string match] command. Except that before the matching is done, any "%" characters specified as part of the pattern are transformed to "*".

    To run "all" tests (full + permutations):

      ./testfixture $TESTDIR/testrunner.tcl all
    

    2.3. Investigating Binary Test Failures

    If a test fails, testrunner.tcl reports name of the Tcl test script and, if applicable, the name of the permutation, to stdout. This information can also be retrieved from either testrunner.log or testrunner.db.

    If there is no permutation, the individual test script may be run with:

      ./testfixture $PATH_TO_SCRIPT
    

    Or, if the failure occured as part of a permutation:

      ./testfixture $TESTDIR/testrunner.tcl $PERMUTATION $PATH_TO_SCRIPT
    

    TODO: An example instead of "$PERMUTATION" and $PATH_TO_SCRIPT?

    3. Source Code Tests

    The commands described in this section invoke the C compiler to build binaries from the source tree, then use those binaries to run Tcl and other tests. The advantages of this are that:

    • it is possible to test multiple build configurations with a single command, and

    • it ensures that tests are always run using binaries created with the same set of compiler options.

    The testrunner.tcl commands described in this section may be run using either a testfixture (or testfixture.exe) build, or with any other Tcl shell that supports SQLite 3.31.1 or newer via "package require sqlite3".

    TODO: ./configure + Makefile.msc build systems.

    3.1. Commands to Run SQLite Tests

    The mdevtest command is equivalent to running the veryquick tests and the make fuzztest target once for each of two --enable-all builds - one with debugging enabled and one without:

      tclsh $TESTDIR/testrunner.tcl mdevtest
    

    In other words, it is equivalent to running:

      $TOP/configure --enable-all --enable-debug
      make fuzztest
      make testfixture
      ./testfixture $TOP/test/testrunner.tcl veryquick
    
      # Then, after removing files created by the tests above:
      $TOP/configure --enable-all OPTS="-O0"
      make fuzztest
      make testfixture
      ./testfixture $TOP/test/testrunner.tcl veryquick
    

    The sdevtest command is identical to the mdevtest command, except that the second of the two builds is a sanitizer build. Specifically, this means that OPTS="-fsanitize=address,undefined" is specified instead of OPTS="-O0":

      tclsh $TESTDIR/testrunner.tcl sdevtest
    

    The release command runs lots of tests under lots of builds. It runs different combinations of builds and tests depending on whether it is run on Linux, Windows or OSX. Refer to testrunner_data.tcl for the details of the specific tests run.

      tclsh $TESTDIR/testrunner.tcl release
    

    As with source code tests, one or more patterns may be appended to any of the above commands (mdevtest, sdevtest or release). In that case only Tcl tests (no fuzz or other tests) that match the specified pattern are run. For example, to run the just the Tcl rtree tests in all builds and configurations supported by "release":

      tclsh $TESTDIR/testrunner.tcl release rtree%
    

    3.2. Running ZipVFS Tests

    testrunner.tcl can build a zipvfs-enabled testfixture and use it to run tests from the Zipvfs project with the following command:

      tclsh $TESTDIR/testrunner.tcl --zipvfs $PATH_TO_ZIPVFS
    

    This can be combined with any of "mdevtest", "sdevtest" or "release" to test both SQLite and Zipvfs with a single command:

      tclsh $TESTDIR/testrunner.tcl --zipvfs $PATH_TO_ZIPVFS mdevtest
    

    3.3. Investigating Source Code Test Failures

    Investigating a test failure that occurs during source code testing is a two step process:

    1. Recreating the build configuration in which the test failed, and

    2. Re-running the actual test.

    To recreate a build configuration, use the testrunner.tcl script command to create a build script. A build script is a bash script on Linux or OSX, or a dos *.bat file on windows. For example:

      # Create a script that recreates build configuration "Device-One" on 
      # Linux or OSX:
      tclsh $TESTDIR/testrunner.tcl script Device-One > make.sh 
    
      # Create a script that recreates build configuration "Have-Not" on Windows:
      tclsh $TESTDIR/testrunner.tcl script Have-Not > make.bat 
    

    The generated bash or *.bat file script accepts a single argument - a makefile target to build. This may be used either to run a make command test directly, or else to build a testfixture (or testfixture.exe) binary with which to run a Tcl test script, as described above.

    4. Extra testrunner.tcl Options

    The testrunner.tcl script options in this section may be used with both source code and binary tests.

    The --buildonly option instructs testrunner.tcl just to build the binaries required by a test, not to run any actual tests. For example:

      # Build binaries required by release test.
      tclsh $TESTDIR/testrunner.tcl --buildonly release"
    

    The --dryrun option prevents testrunner.tcl from building any binaries or running any tests. Instead, it just writes the shell commands that it would normally execute into the testrunner.log file. Example:

      # Log the shell commmands that make up the mdevtest test.
      tclsh $TESTDIR/testrunner.tcl --dryrun mdevtest"
    

    The --explain option is similar to --dryrun in that it prevents testrunner.tcl from building any binaries or running any tests. The difference is that --explain prints on standard output a human-readable summary of all the builds and tests that would have been run.

      # Show what builds and tests would have been run
      tclsh $TESTDIR/testrunner.tcl --explain mdevtest
    

    5. Controlling CPU Core Utilization

    When running either binary or source code tests, testrunner.tcl reports the number of jobs it intends to use to stdout. e.g.

      $ ./testfixture $TESTDIR/testrunner.tcl
      splitting work across 16 jobs
      ... more output ...
    

    By default, testfixture.tcl attempts to set the number of jobs to the number of real cores on the machine. This can be overridden using the "--jobs" (or -j) switch:

      $ ./testfixture $TESTDIR/testrunner.tcl --jobs 8
      splitting work across 8 jobs
      ... more output ...
    

    The number of jobs may also be changed while an instance of testrunner.tcl is running by exucuting the following command from the directory containing the testrunner.log and testrunner.db files:

      $ ./testfixture $TESTDIR/testrunner.tcl njob $NEW_NUMBER_OF_JOBS