Rules for the Berkeley DB and Berkeley DB-XML test suites

1.  Test Naming

The primary script for running Berkeley DB scripts is named
'test.tcl'.  The primary script for running DB-XML is named
'xmltest.tcl'.

Tests are named with a (prefix, test number) combination.  The
prefix indicates the type of test (lock, log, xml, etc.).  The
prefix 'test' is used for plain vanilla DB testing.  Test numbers
are 3 digits long, starting with 001.

Procedures common to a group of tests, or to all tests, are placed
in files named 'xxxutils.tcl'.  At the moment, we have the following
utilities files:

testutils.tcl	Utilities common to all DB tests
reputils.tcl	Utilities for replication testing.
siutils.tcl	Utilities for secondary index testing.
xmlutils.tcl 	Utilities for XML testing.

2.  Internal test structure

Each line within a test should be no more than 80 characters long.

Each test starts with a section like the following:

# See the file LICENSE for redistribution information.
#
# Copyright (c) 1996,2007 Oracle.  All rights reserved.
#
# $Id: README,v 12.5 2007/05/17 15:15:55 bostic Exp $
#
# TEST	test001
# TEST	Small keys/data
# TEST		Put/get per key
# TEST		Dump file
# TEST		Close, reopen
# TEST		Dump file
# TEST
# TEST	Use the first 10,000 entries from the dictionary.
# TEST	Insert each with self as key and data; retrieve each.
# TEST	After all are entered, retrieve all; compare output to original.
# TEST	Close file, reopen, do retrieve and re-verify.

First we refer to the license and assert copyright, then comes the CVS
header string.   The section of lines beginning # TEST is used to
automatically maintain the TESTS file, a listing of all tests and
what they do.   Use this section to briefly describe the test's purpose
and structure.

Next comes the main procedure of the test, which has the same name
as the tcl file.  The test should be liberally commented, and also
should use 'puts' to send messages to the output file.

Sections of a test are identified with letters: test001.a, test001.b,
test001.c.

Here's some typical output:

	puts "Test$tnum: $method ($args) $nentries equal key/data pairs"
 	puts "\tTest$tnum.a: put/get loop"
	puts "\tTest$tnum.b: dump file"
	puts "\tTest$tnum.c: close, open, and dump file"
	puts "\tTest$tnum.d: close, open, and dump file in reverse direction"

The reporting of the current value of the args is particularly
useful, allowing us to say at a glance that "testxxx is failing in
btree" or whatever.  Each line of output must begin with the test name.
We use this to separate expected informational output from errors.

Ancillary procedures follow the main procedure.   Procedures used
by more than one test should go into the appropriate XXXutils.tcl
file.

3.  Reporting failures

Failures in tests are reported with a message starting with the
prefix "FAIL:".  Failures in tests are usually caught with the
error_check_good and error_check_bad routines to compare an
actual return value to an expected return value.  These routines
take care of putting the "FAIL:" prefix on the message.

4.  Running tests

Any single test can be run from the tclsh prompt by typing the
name of the test.  If it's a test from the 'testxxx' group, you
should also specify the method you'd like to test:

	log001
	test001 btree

To run one of the 'testxxx' tests for all methods, use the
run_test procedure:

	run_test test001

Any group of tests (the subsystems lock, log, test, etc.) can be
run by typing

	r $sub

where sub is the name of the subsystem.

For any of the following methods

run_method
run_secmethod
run_secenv
run_reptest
run_repmethod
run_envmethod
run_recd

you can type

run (suffix method start stop).

For example, to run test010 through test020 in btree using
run_method:

	run method btree 10 20

Or the same tests in repmethod:

	run repmethod btree 10 20

Notice the missing underbar.

If you omit the start and stop numbers, you'll get all the tests:

	run method btree

run_recd is a special case, in that it runs the recdxxx tests;
all the others run the testxxx tests.

To run the standard test suite, type run_std at the tclsh prompt.
To run all the tests, type run_all.

If you are running run_std or run_all, you may use the run_parallel
interface to speed things up or to test under conditions of high
system load.  Run_parallel creates a list of all tests in the run,
reorders the tests randomly, then runs the tests in a number of
parallel processes.  To run run_std in five processes type

	run_parallel 5 run_std