31. Pike Test Suite

The test suite is used to check the compiler, interpreter and standard modules. It detects regressions, exercises error handling and boundary cases, and helps find problems that depend on the operating system, available libraries or build configuration. Tests should be added when introducing a feature or fixing a bug, and should typically be kept near the code they exercise.

Most tests are written in testsuite.in files in the src/ and lib/ trees. These files are processed with an m4 script bin/mktestsuite that expands their macros into files named testsuite, which are then executed one test at a time by Tools.Standalone.test_pike. Larger tests may instead be standalone Pike scripts with a .test suffix. Such files are useful for tests involving callbacks, processes or an event loop.

31.1. Running Tests

From the source tree's top directory, the usual command is:

make verify

This generates the test suites, builds a test installation and runs it. Use BUILDDIR to select a particular build configuration when needed. The principal test-related make targets are:

testsuites

Generate tests from testsuite.in files, including the compiler and interpreter suite, native modules and Pike modules.

tinstall

Build a test installation in the build directory, replacing the previous test installation.

verify

Generate suites, make the test installation and run its tests. This also exercises the installation layout.

just_verify

Generate suites and run the Pike in the build tree without making a new test installation.

verify_installed

Generate suites and test the Pike installed in the configured installation directory instead of the build-tree interpreter.

check, sure

Aliases for verify.

verbose_verify

Run the build-tree tests with additional verbosity.

gdb_verify

Run the build-tree test runner under gdb.

valgrind_verify, valgrind_just_verify

Run the test installation or build-tree interpreter under Valgrind. Use VALGRINDARGS for Valgrind options.

TESTARGS supplies arguments to the Tools.Standalone.test_pike test runner, not to the installation program. It replaces the default arguments rather than adding to them. The defaults normally run each suite in a separate process and discover suites in the build tree and script tests in the source trees. Include the discovery options or explicit suite paths you need when overriding them. For example, to run only the core suite with detailed diagnostics:

make just_verify TESTARGS="-v3 -f testsuite"

The make recipes run the test suite from the build directory, so relative paths in TESTARGS are relative to that directory. Generated native-module suites are under modules/ and post_modules/; suites from lib/ are under tlib/.

31.1.1. Running an individual suite

pike -x test_pike runs tests using the interpreter that executes the command. To test an uninstalled build, use the bin/pike script, which selects its master and module paths. Replace arch below with the actual build-directory name:

make testsuites
bin/pike -x test_pike -v3 build/arch/testsuite
pike -x test_pike -v3 lib/modules/Protocols.pmod/WebSocket.test

The last command tests the installed Pike, including its modules. Generated macro suites must have the basename testsuite; the runner does not accept arbitrary names for that format. bin/test_pike.pike is a compatibility entry point for the standalone tool. Use pike -x test_pike --help to inspect the options provided by the interpreter being tested.

31.1.2. Runner options

-a, --auto[=directory]

Discover files named testsuite and files ending in .test recursively. The default directory to search is the current directory. Repeat the option to search several trees. Presence of the file no_testsuites excludes its containing directory and its descendants.

-v[level], --verbose[=level]

Increase verbosity. Level 1 shows progress, level 2 identifies cases and skipped conditions, level 3 prints test source, level 4 also reports timings, and level 10 prints the final source including the runner's wrappers. Repeated verbosity options add their levels.

-sN, --start-test=N

Start at the one-based test number N in each suite. Earlier setup tests are not executed.

-eN, --end-after=N

Stop after the completed-test counter exceeds N. The current runner checks after execution, so -e0 runs one case and -e1 can run two.

-f, --fail

Stop after the first failed test.

-F, --forked

Run each suite in a separate subprocess, isolating its interpreter state from other suites.

-lN, --loop=N

Repeat the suites N times; the default is one pass.

-p, --prompt

Ask before executing each test.

-tN, --trace=N

Enable Pike tracing at the requested level.

-c[level], --check[=level]

Enable interpreter consistency checks when supported by the build. Positive levels add checks before tests; higher levels add checks around compilation and execution. Level 4 also forces garbage collection before tests. A negative level checks periodically instead.

-m, --mem, --memory

Report memory allocations after the tests.

-S, --omit-slow-tests

This causes the symbol OMIT_SLOW_TESTS to be #defined, allowing tests that take time to detect slow machines and reduce work-load.

--failed-cond, --failed-conditionals

Print conditions that caused tests to be skipped.

-T, --notty

Use output suitable for a log rather than an interactive terminal.

--subproc-start=command

Prefix subprocess commands, for example when running child interpreters under Valgrind.

--no-watchdog

Disable timeout monitoring. Normally the watchdog interrupts an unresponsive test after ten minutes, or forty minutes in a dmalloc build. Disable it only when needed for investigation, such as when running in a debugger.

--watchdog=pid

Run the watchdog process for a specified process. This option is normally used internally by the runner.

-d, --debug

Open a debug port when supported by the build.

-h, --help

List the runner's available options.

bin/pike -x test_pike -v3 -f -s12 -e0 build/arch/testsuite
bin/pike -x test_pike -F --auto=lib/modules

A nonzero exit status indicates failure. Failed tests typically output the failed test's testsuite filename and line number and the actual and expected values. The summary also reports skipped tests; these are tests for which the preconditionals failed (like eg missing a module or the wrong OS).

31.2. Writing New Tests

Put a small, self-contained regression test in the suite nearest the affected code or in a testsuite that already covers that subsystem. A test suite should aim to cover common inputs, defaults, empty values, limits, invalid arguments and cleanup after errors. Use deterministic inputs and local resources; avoid depending on an external server or exact wall-clock timing.

31.2.1. A complete testsuite.in example

A macro suite is delimited by START_MARKER and END_MARKER. Use m4 quoting, [[ and ]], around code containing commas or other macro syntax. For example, save the following as /tmp/pike-example/testsuite.in:

START_MARKER

test_eq(sizeof("Pike"), 4)
test_equal([[reverse(({1, 2, 3}))]], [[({3, 2, 1})]])
test_any([[
  array(int) values = ({1, 2, 3});
  return sizeof(values);
]], 3)
test_eval_error([[error("Expected failure.\n");]])

END_MARKER

From the source tree's top directory, generate and run it with:

bin/mktestsuite /tmp/pike-example/testsuite.in > /tmp/pike-example/testsuite
pike -x test_pike -v3 /tmp/pike-example/testsuite

Normal in-tree tests are generated by make testsuites. A generated case contains wrapper functions, usually a() for the operation and b() for its expected result. Each case is compiled separately. Local variables and declarations in one case therefore do not establish local state for another. If state needs to be passed to later tests add_constant() may be used. Note that such symbols must be removed by the time the test suite completes. The expected expression is evaluated too, so keep it simple and independent of the operation being tested.

31.2.2. test_eq and test_any

test_eq(expression, expected) compares the two results with ==. test_any(code, expected) performs the same comparison, but its first argument is a function body that must return the result. Use it when setup or several statements are needed. For separate arrays, mappings or multisets, use the deep-comparison variants instead.

test_eq(1e1, 10.0)
test_any([[
  string text = "hello";
  return upper_case(text);
]], "HELLO")

31.2.3. test_equal and test_any_equal

These variants compare results with equal(), which compares container contents recursively. test_equal takes expressions; test_any_equal takes a function body followed by an expected expression. Equal contents do not imply that two containers have the same identity.

test_equal([[({1, 2}) + ({3})]], [[({1, 2, 3})]])
test_any_equal([[
  array(int) values = ({1, 2, 3});
  return reverse(values);
]], [[({3, 2, 1})]])

31.2.4. test_approx and test_any_approx

These macros compare floating-point results using a tolerance derived from the interpreter's floating-point precision. test_approx takes expressions and test_any_approx takes a function body and an expected expression. Prefer them to exact equality when rounding differences are expected. They do not accept a caller-specified tolerance; use an explicit bounds check when the algorithm requires one.

test_approx(0.1 + 0.2, 0.3)
test_any_approx([[return 0.1 + 0.2;]], 0.3)

31.2.5. test_true, test_false and test_do

test_true(expression) requires a true result and test_false(expression) requires a false result, using Pike's normal truth rules. test_do(code) compiles and executes statements; it fails on compilation errors, unexpected warnings or an uncaught exception. It does not compare a return value. Use an assertion inside the body when necessary.

test_true(has_prefix("Pike", "Pi"))
test_false(has_prefix("Pike", "Py"))
test_do([[
  object file = Stdio.File();
  destruct(file);
]])

31.2.6. test_compile and test_compile_any

test_compile(code) wraps statements in a function and verifies that they compile without errors or warnings. It does not execute them. test_compile_any(program) compiles a complete program without adding a function wrapper; use it for top-level declarations, inheritance and classes.

test_compile([[Stdio.File file = Stdio.File();]])
test_compile_any([[
  class Counter {
    int value;
    int increment() { return ++value; }
  }
]])

31.2.7. test_compile_error and test_compile_error_any

These verify that compilation fails, rather than that execution throws an exception. test_compile_error wraps statements in a function. test_compile_error_any takes a complete program. Use an unambiguous error so an unrelated diagnostic does not accidentally satisfy the test.

test_compile_error([[int value = "not an integer";]])
test_compile_error_any([[
  int value = "not an integer";
]])

31.2.8. test_compile_warning

This wraps statements in a function and succeeds if compilation produces a warning without a compilation error. It does not require a particular warning text or run the resulting program. For example, a constant that is hidden by an inherited variable produces a warning:

test_compile_warning([[
  class A { int value = 6; }
  class B {
    constant value = 5;
    inherit A;
  }
]])

Ordinary execution tests also reject unexpected warnings. If a known warning is incidental to another test, the affected cases should be wrapped with ignore_warning(message, tests). Avoid broad suppression that could hide new warnings. That helper suppresses matching warnings; it does not assert their presence.

31.2.9. test_eval_error

test_eval_error(code) expects an exception while compiling or executing the supplied function body. For historical reasons, a compilation error also satisfies this macro. To test a runtime failure specifically, keep the body compilable and use dynamic arguments where a static type check would otherwise reject the call. The macro does not inspect the exception's contents; use catch and a comparison test when those matter.

test_eval_error([[error("Expected failure.\n");]])
test_any([[
  mixed failure = catch { error("Expected failure.\n"); };
  return failure[0];
]], "Expected failure.\n")

31.2.10. test_define_program

test_define_program(name, body) creates a class from the supplied body and installs it as a global constant for later tests. It is useful when several cases need the same helper program. Remove the constant after use so it does not affect subsequent tests. A selected range must include the definition before cases that depend on it.

test_define_program(GuideCounter, [[
  int value;
  int increment() { return ++value; }
]])
test_any([[
  object counter = GuideCounter();
  counter->increment();
  return counter->increment();
]], 2)
test_do(add_constant("GuideCounter"))

31.2.11. test_program

test_program(program) compiles a complete program, instantiates it and calls its a() method. That method must return a true value for the test to succeed. This permits helper methods, variables, inheritance and lfuns in the same test program. test_program_eq(program, expected) and test_program_equal(program, expected) compare its result with == and equal(), respectively.

test_program([[
  int value;
  int increment() { return ++value; }
  int a() { return increment() == 1 && increment() == 2; }
]])

31.2.12. cond, cond_resolv and ifefun

cond(expression, tests) runs the enclosed cases only if its Pike expression is true. A false condition skips cases rather than failing them; an error in a condition is a test failure. Conditions are cached by the runner, so they should check stable capabilities rather than changing per-test state.

cond_resolv(name, tests) checks whether a module or other named value can be resolved. ifefun(name, tests) checks for a global built-in function or constant. These are runtime availability checks, not C preprocessor tests. Use cond_begin(expression) and cond_end when a condition spans a longer group of cases; always balance them.

cond([[sizeof("Pike") == 4]], [[
  test_eq(lower_case("PIKE"), "pike")
]])
cond_resolv(Stdio, [[
  test_true(programp(Stdio.File))
]])
ifefun(sizeof, [[
  test_eq(sizeof(""), 0)
]])

Guard optional features with their actual availability conditions. Do not make a broken feature look optional by skipping on the result being tested. Use --failed-cond to check why cases were skipped.

31.2.13. Regression tests

A regression test uses the ordinary test macros; no special nonregression macro is provided by the current suite generator. Reduce the original failure to a small example and assert the intended result or error. Add a short comment recording the bug or issue and why the case is significant. Test the smallest relevant boundary on either side of the failure, and ensure the case does not pass merely because unrelated setup failed.

31.3. Standalone script tests

A script test has a .test suffix and begins a comment containing the marker TEST: RUN-AS-PIKE-SCRIPT. The runner starts it in a child interpreter using the interpreter under test. A script may use its own assertion helpers and event loop. It should report its result with Tools.Testsuite.report_result(); and report skipped cases separately from successful and failed assertions.

// -*- pike -*- TEST: RUN-AS-PIKE-SCRIPT

int main()
{
  int succeeded;
  int failed;

  if (equal(reverse(({1, 2, 3})), ({3, 2, 1})))
    succeeded++;
  else {
    failed++;
    werror("reverse() returned an unexpected result.\n");
  }

  Tools.Testsuite.report_result(succeeded, failed);
  return failed ? 1 : 0;
}

Save the above as example.test and run pike -x test_pike -v3 example.test. An uncaught exception or abnormal child-process termination is a failure. A script containing several assertions should report their individual counts rather than just whether main() completed.

A macro suite can invoke a script with Tools.Testsuite.run_script() and use test_tests with a complete program whose a() returns the resulting array of success, failure and skip counts. Prefer a directly discovered .test file when no macro-suite integration is needed. See lib/modules/Protocols.pmod/WebSocket.test for a larger script test.

31.4. Investigating and reporting failures

Re-run the smallest failing suite using the same Pike binary and module paths. Increase verbosity to inspect the generated code and select the failing test with -s and -e. If that test depends on earlier setup, include those cases too. Use -l to investigate failures that require repeated execution, and an interpreter with runtime debugging enabled when investigating consistency-check failures.

Distinguish an unavailable optional feature from a failed assertion, an unexpected warning, a compiler error, a crash or a watchdog timeout. Keep the complete diagnostics and record the Pike version, platform, build options and command used to reproduce the problem. Where relevant, compare debug and optimized builds or use gdb and Valgrind. After fixing the focused case, run the surrounding suite and make verify to check for interactions.