# Test suite for clitest
This is the test file for the `clitest` program. Yes, the program can test itself!
This file runs all the files inside the `test` folder and checks the results. The command line options are also tested.
Usage: ./clitest test.md
## Preparing
Make sure we're on the same folder as `clitest`, since all the file paths here are relative, not absolute.
```
$ test -f ./clitest; echo $?
0
$ test -d ./test/; echo $?
0
$
```
Set a default terminal width of 80 columns. It's used by separator lines.
```
$ shopt -u checkwinsize 2> /dev/null # bash: disable automatic check
$ unset COLUMNS # mksh: first unset, then one can manually set it
$ COLUMNS=80
$ export COLUMNS
$
```
Ok. Now the real tests begins.
## Variables are persistent between tests?
```
$ echo $COLUMNS
80
$ not_exported=1
$ echo $not_exported
1
$ echo $not_exported #=> 1
$ echo $not_exported #=> --regex ^1$
```
## Check the temporary dir creation
```
$ TMPDIR___SAVE="$TMPDIR"
$ TMPDIR=/notfound
$ export TMPDIR
$ ./clitest test/ok-1.sh 2>&1 | grep ^clitest | sed 's/clitest\..*$/clitest.XXXXXX/'
clitest: Error: cannot create temporary dir: /notfound/clitest.XXXXXX
$ TMPDIR="$TMPDIR___SAVE"
$
```
## I/O, file reading (message and exit code)
Missing input file
```
$ ./clitest; echo $?
clitest: Error: no test file informed (try --help)
2
$ ./clitest --
clitest: Error: no test file informed (try --help)
$ ./clitest --list
clitest: Error: no test file informed (try --help)
$
```
File not found
```
$ ./clitest notfound; echo $?
clitest: Error: cannot read input file: notfound
2
$
```
File is a directory
```
$ ./clitest .
clitest: Error: input file is a directory: .
$ ./clitest ./
clitest: Error: input file is a directory: ./
$ ./clitest /etc
clitest: Error: input file is a directory: /etc
$
```
## No test found (message and exit code)
```
$ ./clitest test/no-test-found.sh; echo $?
clitest: Error: no test found in input file: test/no-test-found.sh
2
$ ./clitest test/empty-file.sh
clitest: Error: no test found in input file: test/empty-file.sh
$ ./clitest test/empty-prompt-file.sh
clitest: Error: no test found in input file: test/empty-prompt-file.sh
$ ./clitest test/empty-prompts-file.sh
clitest: Error: no test found in input file: test/empty-prompts-file.sh
$
```
## Option --version
The exit code must always be zero.
```
$ ./clitest --version > /dev/null; echo $?
0
$
```
Test the output text and the short option `-V`.
```
$ ./clitest --version
clitest 0.4.0
$ ./clitest -V
clitest 0.4.0
$
```
## Option --help
Test the full help text contents and the exit code (zero).
```
$ ./clitest --help; echo $?
Usage: clitest [options] <file ...>
Options:
-1, --first Stop execution upon first failed test
-l, --list List all the tests (no execution)
-L, --list-run List all the tests with OK/FAIL status
-t, --test RANGE Run specific tests, by number (1,2,4-7)
-s, --skip RANGE Skip specific tests, by number (1,2,4-7)
--pre-flight COMMAND Execute command before running the first test
--post-flight COMMAND Execute command after running the last test
-q, --quiet Quiet operation, no output shown
-V, --version Show program version and exit
Customization options:
-P, --progress TYPE Set progress indicator: test, number, dot, none
--color WHEN Set when to use colors: auto, always, never
--diff-options OPTIONS Set diff command options (default: '-u')
--inline-prefix PREFIX Set inline output prefix (default: '#=> ')
--prefix PREFIX Set command line prefix (default: '')
--prompt STRING Set prompt string (default: '$ ')
See also: https://github.com/aureliojargas/clitest
0
$
```
The short option `-h` is working? Testing just the first and last lines for brevity.
```
$ ./clitest -h | sed -n '1p; $p'
Usage: clitest [options] <file ...>
See also: https://github.com/aureliojargas/clitest
$
```
## Option --quiet and exit code
```
$ ./clitest -q test/ok-2.sh; echo $?
0
$ ./clitest --quiet test/ok-2.sh; echo $?
0
$ ./clitest --quiet test/ok-2.sh test/ok-2.sh; echo $?
0
$ ./clitest --quiet test/fail-2.sh; echo $?
1
$ ./clitest --quiet test/fail-2.sh test/fail-2.sh; echo $?
1
$ ./clitest --quiet test/ok-2.sh test/fail-2.sh; echo $?
1
$
```
## Option --quiet has no effect in error messages
```
$ ./clitest --quiet notfound
clitest: Error: cannot read input file: notfound
$
```
## Option --quiet has no effect in --debug (disabled)
```
# $ ./clitest --quiet --debug test/ok-2.sh
# [INPUT_LINE: $ echo ok]
# [ LINE_CMD: $ echo ok]
# [ NEW_CMD: echo ok]
# [INPUT_LINE: ok]
# [ LINE_*: ok]
# [ OK_TEXT: ok]
# [INPUT_LINE: $ echo ok #=> ok]
# [ LINE_CMD: $ echo ok #=> ok]
# [ EVAL: echo ok]
# [ OUTPUT: ok]
# [ NEW_CMD: echo ok ]
# [ OK_INLINE: ok]
# [ OK_TEXT: ok]
# [ EVAL: echo ok ]
# [ OUTPUT: ok]
# [ LOOP_OUT: $test_command=]
# $
```
## Option --color
Invalid value
```
$ ./clitest --color foo test/ok-1.sh; echo $?
clitest: Error: invalid value 'foo' for --color. Use: auto, always or never.
2
$
```
Color ON
```
$ ./clitest --color always test/ok-1.sh
#1 echo ok
[32mOK:[m 1 of 1 test passed
$ ./clitest --color yes test/ok-1.sh
#1 echo ok
[32mOK:[m 1 of 1 test passed
$
```
Color OFF
```
$ ./clitest --color never test/ok-1.sh
#1 echo ok
OK: 1 of 1 test passed
$ ./clitest --color no test/ok-1.sh
#1 echo ok
OK: 1 of 1 test passed
$
```
Color AUTO
Inside this file, the output is not a terminal,
so the default is no colored output.
```
$ ./clitest test/ok-1.sh
#1 echo ok
OK: 1 of 1 test passed
$
```
The real default `--color auto` cannot be tested here.
Test it by hand at the command line.
```
## $ ./clitest test/ok-1.sh
## [32mOK![m The single test has passed.
## $ ./clitest --color auto test/ok-1.sh
## [32mOK![m The single test has passed.
## $
```
## Option --list
Listing a file with no tests
```
$ ./clitest --list test/empty-file.sh; echo $?
clitest: Error: no test found in input file: test/empty-file.sh
2
$
```
Normal results and exit code
```
$ ./clitest --list test/no-nl-command.sh; echo $?
#1 printf 'ok\n'
#2 printf 'fail'
#3 printf 'ok\nok\nfail'
#4 printf 'ok\n'
#5 printf 'fail'
#6 printf 'ok'; echo
#7 printf 'ok'
0
$
```
Short option `-l`
```
$ ./clitest -l test/no-nl-command.sh
#1 printf 'ok\n'
#2 printf 'fail'
#3 printf 'ok\nok\nfail'
#4 printf 'ok\n'
#5 printf 'fail'
#6 printf 'ok'; echo
#7 printf 'ok'
$
```
Multifile and exit code
```
$ ./clitest --list test/no-nl-command.sh test/ok-1.sh; echo $?
---------------------------------------- test/no-nl-command.sh
#1 printf 'ok\n'
#2 printf 'fail'
#3 printf 'ok\nok\nfail'
#4 printf 'ok\n'
#5 printf 'fail'
#6 printf 'ok'; echo
#7 printf 'ok'
---------------------------------------- test/ok-1.sh
#8 echo ok
0
$
```
## Option --list-run
Listing a file with no tests
```
$ ./clitest --list-run test/empty-file.sh; echo $?
clitest: Error: no test found in input file: test/empty-file.sh
2
$
```
Normal results (using colors) and exit code
```
$ ./clitest --list-run --color yes test/no-nl-command.sh; echo $?
[32m#1 printf 'ok\n'[m
[31m#2 printf 'fail'[m
[31m#3 printf 'ok\nok\nfail'[m
[32m#4 printf 'ok\n' [m
[31m#5 printf 'fail' [m
[32m#6 printf 'ok'; echo [m
[32m#7 printf 'ok' [m
1
$
```
Normal results (no colors, use OK/FAIL column) and exit code
```
$ ./clitest --list-run test/no-nl-command.sh; echo $?
#1 OK printf 'ok\n'
#2 FAIL printf 'fail'
#3 FAIL printf 'ok\nok\nfail'
#4 OK printf 'ok\n'
#5 FAIL printf 'fail'
#6 OK printf 'ok'; echo
#7 OK printf 'ok'
1
$
```
Short option `-L`
```
$ ./clitest -L test/no-nl-command.sh
#1 OK printf 'ok\n'
#2 FAIL printf 'fail'
#3 FAIL printf 'ok\nok\nfail'
#4 OK printf 'ok\n'
#5 FAIL printf 'fail'
#6 OK printf 'ok'; echo
#7 OK printf 'ok'
$
```
Multifile and exit code
```
$ ./clitest -L test/no-nl-command.sh test/ok-1.sh; echo $?
---------------------------------------- test/no-nl-command.sh
#1 OK printf 'ok\n'
#2 FAIL printf 'fail'
#3 FAIL printf 'ok\nok\nfail'
#4 OK printf 'ok\n'
#5 FAIL printf 'fail'
#6 OK printf 'ok'; echo
#7 OK printf 'ok'
---------------------------------------- test/ok-1.sh
#8 OK echo ok
1
$ ./clitest -L test/ok-1.sh; echo $?
#1 OK echo ok
0
$
```
## Option --progress
First, some invalid values:
```
$ ./clitest --progress test/ok-1.sh
clitest: Error: no test file informed (try --help)
$ ./clitest --progress '' test/ok-1.sh
clitest: Error: invalid value '' for --progress. Use: test, number, dot or none.
$ ./clitest --progress foo test/ok-1.sh
clitest: Error: invalid value 'foo' for --progress. Use: test, number, dot or none.
$ ./clitest --progress DOT test/ok-1.sh
clitest: Error: invalid value 'DOT' for --progress. Use: test, number, dot or none.
$ ./clitest --progress @@ test/ok-1.sh
clitest: Error: invalid value '@@' for --progress. Use: test, number, dot or none.
$ ./clitest --progress -1 test/ok-1.sh; echo $?
clitest: Error: invalid value '-1' for --progress. Use: test, number, dot or none.
2
$
```
If no `--progress` option, defaults to `--progress test`:
```
$ ./clitest test/ok-1.sh
#1 echo ok
OK: 1 of 1 test passed
$ ./clitest --progress test test/ok-1.sh
#1 echo ok
OK: 1 of 1 test passed
$
```
Numbers:
```
$ ./clitest --progress number test/ok-10.sh
1 2 3 4 5 6 7 8 9 10
OK: 10 of 10 tests passed
$ ./clitest --progress n test/ok-10.sh
1 2 3 4 5 6 7 8 9 10
OK: 10 of 10 tests passed
$ ./clitest --progress 0 test/ok-10.sh
1 2 3 4 5 6 7 8 9 10
OK: 10 of 10 tests passed
$ ./clitest --progress 5 test/ok-10.sh
1 2 3 4 5 6 7 8 9 10
OK: 10 of 10 tests passed
$ ./clitest --progress 9 test/ok-10.sh
1 2 3 4 5 6 7 8 9 10
OK: 10 of 10 tests passed
$
```
Chars:
```
$ ./clitest --progress dot test/ok-10.sh
..........
OK: 10 of 10 tests passed
$ ./clitest --progress . test/ok-10.sh
..........
OK: 10 of 10 tests passed
$ ./clitest --progress @ test/ok-10.sh
@@@@@@@@@@
OK: 10 of 10 tests passed
$ ./clitest --progress x test/ok-10.sh
xxxxxxxxxx
OK: 10 of 10 tests passed
$
```
No progress:
```
$ ./clitest --progress none test/ok-1.sh
OK: 1 of 1 test passed
$ ./clitest --progress no test/ok-1.sh
OK: 1 of 1 test passed
$
```
Short option `-P`:
```
$ ./clitest -P dot test/ok-1.sh
.
OK: 1 of 1 test passed
$ ./clitest -P no test/ok-1.sh
OK: 1 of 1 test passed
$
```
Ok & fail functionality with dot:
```
$ ./clitest --progress . test/ok-1.sh
.
OK: 1 of 1 test passed
$ ./clitest --progress . test/ok-2.sh
..
OK: 2 of 2 tests passed
$ ./clitest --progress . test/ok-50.sh
..................................................
OK: 50 of 50 tests passed
$ ./clitest --progress . test/fail-1.sh
.
--------------------------------------------------------------------------------
[FAILED #1, line 1] echo ok
@@ -1 +1 @@
-fail
+ok
--------------------------------------------------------------------------------
FAIL: 1 of 1 test failed
$
```
Multifile with dot:
```
$ ./clitest --progress . test/ok-1.sh test/ok-2.sh test/ok-10.sh
Testing file test/ok-1.sh .
Testing file test/ok-2.sh ..
Testing file test/ok-10.sh ..........
ok fail skip
1 - - test/ok-1.sh
2 - - test/ok-2.sh
10 - - test/ok-10.sh
OK: 13 of 13 tests passed
$ ./clitest --progress . test/ok-1.sh test/fail-1.sh
Testing file test/ok-1.sh .
Testing file test/fail-1.sh .
--------------------------------------------------------------------------------
[FAILED #2, line 1] echo ok
@@ -1 +1 @@
-fail
+ok
--------------------------------------------------------------------------------
ok fail skip
1 - - test/ok-1.sh
- 1 - test/fail-1.sh
FAIL: 1 of 2 tests failed
$ ./clitest --progress . test/fail-1.sh test/ok-1.sh
Testing file test/fail-1.sh .
--------------------------------------------------------------------------------
[FAILED #1, line 1] echo ok
@@ -1 +1 @@
-fail
+ok
--------------------------------------------------------------------------------
Testing file test/ok-1.sh .
ok fail skip
- 1 - test/fail-1.sh
1 - - test/ok-1.sh
FAIL: 1 of 2 tests failed
$
```
Multifile with no progress:
```
$ ./clitest --progress none test/ok-1.sh test/ok-2.sh test/ok-10.sh
Testing file test/ok-1.sh
Testing file test/ok-2.sh
Testing file test/ok-10.sh
ok fail skip
1 - - test/ok-1.sh
2 - - test/ok-2.sh
10 - - test/ok-10.sh
OK: 13 of 13 tests passed
$ ./clitest --progress none test/ok-1.sh test/fail-1.sh
Testing file test/ok-1.sh
Testing file test/fail-1.sh
--------------------------------------------------------------------------------
[FAILED #2, line 1] echo ok
@@ -1 +1 @@
-fail
+ok
--------------------------------------------------------------------------------
ok fail skip
1 - - test/ok-1.sh
- 1 - test/fail-1.sh
FAIL: 1 of 2 tests failed
$ ./clitest --progress none test/fail-1.sh test/ok-1.sh
Testing file test/fail-1.sh
--------------------------------------------------------------------------------
[FAILED #1, line 1] echo ok
@@ -1 +1 @@
-fail
+ok
--------------------------------------------------------------------------------
Testing file test/ok-1.sh
ok fail skip
- 1 - test/fail-1.sh
1 - - test/ok-1.sh
FAIL: 1 of 2 tests failed
$
```
### Option --progress and skipped tests
Since skipped tests affect the output (show nothing), it's worth
testing if the line break issues won't appear.
```
$ ./clitest --progress . --skip 1 test/ok-2.sh
.
OK: 1 of 2 tests passed (1 skipped)
$ ./clitest --progress . --skip 2 test/ok-2.sh
.
OK: 1 of 2 tests passed (1 skipped)
$ ./clitest --progress . --skip 1 test/fail-2.sh
.
--------------------------------------------------------------------------------
[FAILED #2, line 3] echo ok
@@ -1 +1 @@
-fail
+ok
--------------------------------------------------------------------------------
FAIL: 1 of 2 tests failed (1 skipped)
$ ./clitest --progress . --skip 2 test/fail-2.sh
.
--------------------------------------------------------------------------------
[FAILED #1, line 1] echo ok
@@ -1 +1 @@
-fail
+ok
--------------------------------------------------------------------------------
FAIL: 1 of 2 tests failed (1 skipped)
$
```
Error messages appear with no leading blank line?
```
$ ./clitest --progress . --skip 1,2 test/ok-2.sh
clitest: Error: no test found. Maybe '--skip 1,2' was too much?
$
```
## Options --quiet, --progress, --list and --list-run are mutually exclusive
* Only one can be active, the others must be off.
* The last informed will be the one used.
```
$ ./clitest --list --quiet test/ok-1.sh
$ ./clitest --list-run --quiet test/ok-1.sh
$ ./clitest --progress . --quiet test/ok-1.sh
$ ./clitest --list --list-run --progress . --quiet test/ok-1.sh
$ ./clitest --quiet --progress . --list-run --list test/ok-1.sh
#1 echo ok
$ ./clitest --quiet --progress . --list --list-run test/ok-1.sh
#1 OK echo ok
$ ./clitest --quiet --list --list-run --progress . test/ok-1.sh
.
OK: 1 of 1 test passed
$
```
## Option --test and --skip combined with --list and --list-run
Error: Out of range
```
$ ./clitest --list -t 99 test/ok-10.sh
clitest: Error: no test found for the specified number or range '99'
$ ./clitest --list-run -t 99 test/ok-10.sh; echo $?
clitest: Error: no test found for the specified number or range '99'
2
$
```
Error: Skipped all tests
```
$ ./clitest --list -s 1-10 test/ok-10.sh
clitest: Error: no test found. Maybe '--skip 1-10' was too much?
$ ./clitest --list-run -s 1-10 test/ok-10.sh; echo $?
clitest: Error: no test found. Maybe '--skip 1-10' was too much?
2
$
```
Error: The combination of `-t` and `-s` resulted in no tests
```
$ ./clitest --list -t 9 -s 9 test/ok-10.sh
clitest: Error: no test found. The combination of -t and -s resulted in no tests.
$ ./clitest --list-run -t 9 -s 9 test/ok-10.sh; echo $?
clitest: Error: no test found. The combination of -t and -s resulted in no tests.
2
$
```
Using `-t` alone
```
$ ./clitest --list -t 3,5-7 test/ok-10.sh
#3 echo 3
#5 echo 5
#6 echo 6
#7 echo 7
$ ./clitest --list-run -t 3,5-7 test/ok-10.sh
#3 OK echo 3
#5 OK echo 5
#6 OK echo 6
#7 OK echo 7
$
```
Reverse ranges and repeated numbers are supported
```
$ ./clitest --list -t 3,7-5,3,6,5 test/ok-10.sh
#3 echo 3
#5 echo 5
#6 echo 6
#7 echo 7
$
```
Using `-t` to limit to a range and the `-s` exclude some more
```
$ ./clitest --list -t 3,5-7 -s 6 test/ok-10.sh
#3 echo 3
#5 echo 5
#7 echo 7
$ ./clitest --list-run -t 3,5-7 -s 6 test/ok-10.sh
#3 OK echo 3
#5 OK echo 5
#7 OK echo 7
$
```
Multifile, using `-t` alone
```
$ ./clitest --list -t 1,3,5-7 test/ok-1.sh test/fail-2.sh test/ok-10.sh
---------------------------------------- test/ok-1.sh
#1 echo ok
---------------------------------------- test/fail-2.sh
#3 echo ok
---------------------------------------- test/ok-10.sh
#5 echo 2
#6 echo 3
#7 echo 4
$ ./clitest --list-run -t 1,3,5-7 test/ok-1.sh test/fail-2.sh test/ok-10.sh
---------------------------------------- test/ok-1.sh
#1 OK echo ok
---------------------------------------- test/fail-2.sh
#3 FAIL echo ok
---------------------------------------- test/ok-10.sh
#5 OK echo 2
#6 OK echo 3
#7 OK echo 4
$
```
Multifile, using `-t` and `-s`
```
$ ./clitest --list -t 1,3,5-7 -s 3,6 test/ok-1.sh test/fail-2.sh test/ok-10.sh
---------------------------------------- test/ok-1.sh
#1 echo ok
---------------------------------------- test/fail-2.sh
---------------------------------------- test/ok-10.sh
#5 echo 2
#7 echo 4
$ ./clitest --list-run -t 1,3,5-7 -s 3,6 test/ok-1.sh test/fail-2.sh test/ok-10.sh
---------------------------------------- test/ok-1.sh
#1 OK echo ok
---------------------------------------- test/fail-2.sh
---------------------------------------- test/ok-10.sh
#5 OK echo 2
#7 OK echo 4
$
```
## Single file, OK
```
$ ./clitest test/ok-1.sh
#1 echo ok
OK: 1 of 1 test passed
$ ./clitest test/ok-2.sh
#1 echo ok
#2 echo ok
OK: 2 of 2 tests passed
$ ./clitest test/ok-50.sh | tail -1
OK: 50 of 50 tests passed
$ ./clitest test/ok-100.sh | tail -1
OK: 100 of 100 tests passed
$ ./clitest test/ok-2.sh
#1 echo ok
#2 echo ok
OK: 2 of 2 tests passed
$
```
## Multifile, all OK
```
$ ./clitest test/ok-2.sh test/ok-2.sh
Testing file test/ok-2.sh
#1 echo ok
#2 echo ok
Testing file test/ok-2.sh
#3 echo ok
#4 echo ok
ok fail skip
2 - - test/ok-2.sh
2 - - test/ok-2.sh
OK: 4 of 4 tests passed
$ ./clitest test/ok-1.sh test/ok-10.sh test/ok-100.sh test/ok-50.sh | grep -v ^#
Testing file test/ok-1.sh
Testing file test/ok-10.sh
Testing file test/ok-100.sh
Testing file test/ok-50.sh
ok fail skip
1 - - test/ok-1.sh
10 - - test/ok-10.sh
100 - - test/ok-100.sh
50 - - test/ok-50.sh
OK: 161 of 161 tests passed
$ ./clitest test/ok-?.sh test/ok-10.sh
Testing file test/ok-1.sh
#1 echo ok
Testing file test/ok-2.sh
#2 echo ok
#3 echo ok
Testing file test/ok-10.sh
#4 echo 1
#5 echo 2
#6 echo 3
#7 echo 4
#8 echo 5
#9 echo 6
#10 echo 7
#11 echo 8
#12 echo 9
#13 echo 10
ok fail skip
1 - - test/ok-1.sh
2 - - test/ok-2.sh
10 - - test/ok-10.sh
OK: 13 of 13 tests passed
$
```
## Multifile, OK and fail
```
$ ./clitest test/ok-1.sh test/fail-1.sh test/ok-2.sh test/fail-2.sh
Testing file test/ok-1.sh
#1 echo ok
Testing file test/fail-1.sh
#2 echo ok
--------------------------------------------------------------------------------
[FAILED #2, line 1] echo ok
@@ -1 +1 @@
-fail
+ok
--------------------------------------------------------------------------------
Testing file test/ok-2.sh
#3 echo ok
#4 echo ok
Testing file test/fail-2.sh
#5 echo ok
--------------------------------------------------------------------------------
[FAILED #5, line 1] echo ok
@@ -1 +1 @@
-fail
+ok
--------------------------------------------------------------------------------
#6 echo ok
--------------------------------------------------------------------------------
[FAILED #6, line 3] echo ok
@@ -1 +1 @@
-fail
+ok
--------------------------------------------------------------------------------
ok fail skip
1 - - test/ok-1.sh
- 1 - test/fail-1.sh
2 - - test/ok-2.sh
- 2 - test/fail-2.sh
FAIL: 3 of 6 tests failed
$ ./clitest test/ok-1.sh test/fail-1.sh test/ok-2.sh test/fail-2.sh
Testing file test/ok-1.sh
#1 echo ok
Testing file test/fail-1.sh
#2 echo ok
--------------------------------------------------------------------------------
[FAILED #2, line 1] echo ok
@@ -1 +1 @@
-fail
+ok
--------------------------------------------------------------------------------
Testing file test/ok-2.sh
#3 echo ok
#4 echo ok
Testing file test/fail-2.sh
#5 echo ok
--------------------------------------------------------------------------------
[FAILED #5, line 1] echo ok
@@ -1 +1 @@
-fail
+ok
--------------------------------------------------------------------------------
#6 echo ok
--------------------------------------------------------------------------------
[FAILED #6, line 3] echo ok
@@ -1 +1 @@
-fail
+ok
--------------------------------------------------------------------------------
ok fail skip
1 - - test/ok-1.sh
- 1 - test/fail-1.sh
2 - - test/ok-2.sh
- 2 - test/fail-2.sh
FAIL: 3 of 6 tests failed
$
```
## Fail messages
```
$ ./clitest --prefix tab -P none test/fail-messages.md #=> --file test/fail-messages.out.txt
$
```
## Fails
```
$ ./clitest test/fail-1.sh
#1 echo ok
--------------------------------------------------------------------------------
[FAILED #1, line 1] echo ok
@@ -1 +1 @@
-fail
+ok
--------------------------------------------------------------------------------
FAIL: 1 of 1 test failed
$ ./clitest test/fail-2.sh
#1 echo ok
--------------------------------------------------------------------------------
[FAILED #1, line 1] echo ok
@@ -1 +1 @@
-fail
+ok
--------------------------------------------------------------------------------
#2 echo ok
--------------------------------------------------------------------------------
[FAILED #2, line 3] echo ok
@@ -1 +1 @@
-fail
+ok
--------------------------------------------------------------------------------
FAIL: 2 of 2 tests failed
$ ./clitest test/fail-50.sh | tail -1
FAIL: 50 of 50 tests failed
$ ./clitest -1 test/fail-2.sh
#1 echo ok
--------------------------------------------------------------------------------
[FAILED #1, line 1] echo ok
@@ -1 +1 @@
-fail
+ok
--------------------------------------------------------------------------------
$ ./clitest --first test/fail-2.sh
#1 echo ok
--------------------------------------------------------------------------------
[FAILED #1, line 1] echo ok
@@ -1 +1 @@
-fail
+ok
--------------------------------------------------------------------------------
$ ./clitest --first test/fail-2.sh
#1 echo ok
--------------------------------------------------------------------------------
[FAILED #1, line 1] echo ok
@@ -1 +1 @@
-fail
+ok
--------------------------------------------------------------------------------
$ ./clitest test/fail-2.sh
#1 echo ok
--------------------------------------------------------------------------------
[FAILED #1, line 1] echo ok
@@ -1 +1 @@
-fail
+ok
--------------------------------------------------------------------------------
#2 echo ok
--------------------------------------------------------------------------------
[FAILED #2, line 3] echo ok
@@ -1 +1 @@
-fail
+ok
--------------------------------------------------------------------------------
FAIL: 2 of 2 tests failed
$
```
## Inline output with #=>
```
$ ./clitest test/inline.sh
#1 echo 'one space'
#2 echo 'one tab'
#3 echo 'multi spaces'
#4 echo 'multi tabs'
#5 echo 'mixed'
#6 echo ' leading space'
#7 echo ' leading spaces'
#8 printf '\tleading tab\n'
#9 printf '\t\tleading tabs\n'
#10 echo 'trailing space '
#11 echo 'trailing spaces '
#12 printf 'trailing tab\t\n'
#13 printf 'trailing tabs\t\t\n'
#14 echo ' '
#15 echo ' '
#16 printf '\t\n'
#17 printf '\t\t\t\n'
#18 printf ' \t \t\t \n'
#19 echo "both inline and normal output"
OK: 19 of 19 tests passed
$
```
## Inline match modes
Mode #=> --text
* This is the default mode.
* The --text part can be omitted.
```
$ ./clitest --list-run test/inline-match-text.sh
#1 OK echo 'abc'
#2 OK echo 'abc'
#3 OK printf '%s\n' '\t'
#4 OK printf '%s\n' '\n'
#5 OK echo '$PWD'
#6 OK echo '$(date)'
#7 OK echo '$'
#8 OK echo '>'
#9 OK echo '?'
#10 OK echo '!'
#11 OK echo '*'
#12 OK echo '['
#13 OK echo '('
#14 OK echo
#15 OK echo "not inline output" #=>
#16 OK echo '123456789'
#17 OK echo '1 3 7 9'
#18 OK echo ' 5 '
#19 OK echo ' leading space'
#20 OK echo ' leading spaces'
#21 OK printf '\tleading tab\n'
#22 OK printf '\t\tleading tabs\n'
#23 OK echo 'trailing space '
#24 OK echo 'trailing spaces '
#25 OK printf 'trailing tab\t\n'
#26 OK printf 'trailing tabs\t\t\n'
#27 OK echo ' '
#28 FAIL echo ' '
#29 OK printf '\t\n'
#30 OK printf '\t\t\t\n'
#31 OK printf ' \t \t\t \n'
#32 OK printf 'ok\n'
#33 FAIL printf 'fail'
#34 OK printf 'ok'; echo
#35 FAIL echo 'fail'
#36 FAIL echo 'fail'
#37 OK echo ' ok'
#38 OK echo '--text'
#39 OK echo '--textual'
#40 OK echo '--text is cool'
$
```
Mode #=> --eval
```
$ ./clitest --list-run test/inline-match-eval.sh
#1 OK folder=$(pwd)
#2 OK echo $folder
#3 OK var='abc'
#4 OK echo abc
#5 OK echo 4
#6 OK today=$(date +%D)
#7 OK echo "Today is $today"
#8 OK printf 'ok'
#9 OK echo ' leading space'
#10 OK echo ' leading spaces'
#11 OK printf '\tleading tab\n'
#12 OK printf '\t\tleading tabs\n'
#13 OK echo 'trailing space '
#14 OK echo 'trailing spaces '
#15 OK printf 'trailing tab\t\n'
#16 OK printf 'trailing tabs\t\t\n'
#17 OK echo ' '
#18 OK echo ' '
#19 OK printf '\t\n'
#20 OK printf '\t\t\t\n'
#21 OK printf ' \t \t\t \n'
#22 FAIL echo 'fail'
#23 FAIL echo 'fail'
#24 OK echo '--eval'
#25 OK echo '--evaluate'
#26 OK echo '--eval is evil'
$
```
Mode #=> --egrep
```
$ ./clitest --list-run test/inline-match-egrep.sh | sed 's/^\(#1[56].\)[A-Z]*/\1?/'
#1 OK echo 'abc123'
#2 OK echo 'abc123'
#3 OK echo 'abc123'
#4 OK echo 'abc123'
#5 OK echo 'abc123'
#6 OK echo 'abc123'
#7 OK echo 'abc123'
#8 OK echo 'abc123'
#9 OK echo 'abc 123'
#10 OK echo ' '
#11 OK echo ' '
#12 OK printf '\t\n'
#13 OK printf '\t\t\t\n'
#14 OK printf ' \t \t\t \n'
#15 ? printf 'may\tfail'
#16 ? printf 'may\tfail'
#17 OK printf 'will\tmatch'
#18 FAIL printf 'will\nfail'
#19 FAIL printf 'will\nfail'
#20 OK printf '1\n2\n3\n4\nok\n'
#21 OK printf 'ok'
#22 OK printf 'ok\n'
#23 FAIL echo 'fail'
#24 FAIL echo 'fail'
#25 OK echo ' ok'
#26 OK echo '--egrep'
#27 OK echo '--egreppal'
#28 OK echo '--egrep is cool'
$
```
Mode #=> --perl
* --regex is an alias to --perl
```
$ ./clitest --list-run test/inline-match-perl.sh
#1 OK echo 'abc123'
#2 OK echo 'abc123'
#3 OK echo 'abc123'
#4 OK echo 'abc123'
#5 OK echo 'abc123'
#6 OK echo 'abc123'
#7 OK echo 'abc123'
#8 OK echo 'abc123'
#9 OK echo 'abc 123'
#10 OK echo ' '
#11 OK echo ' '
#12 OK printf '\t\n'
#13 OK printf '\t\t\t\n'
#14 OK printf ' \t \t\t \n'
#15 OK echo '01/01/2013'
#16 OK echo "won't fail"
#17 OK printf 'will\tmatch'
#18 OK printf 'will\tmatch'
#19 OK printf 'will\tmatch'
#20 FAIL printf 'will\nfail'
#21 OK printf 'will\nmatch'
#22 FAIL printf 'will\nfail'
#23 OK printf 'will\nmatch'
#24 FAIL printf 'will\nfail'
#25 OK printf 'will\nmatch'
#26 OK printf 'ok'
#27 OK printf 'ok\n'
#28 OK printf '1\n2\n3\n'
#29 OK printf '1\n2\n3\n'
#30 FAIL echo 'fail'
#31 FAIL echo 'fail'
#32 OK echo ' ok'
#33 OK echo '--perl'
#34 OK echo '--perlism'
#35 OK echo '--perl is cool'
$
```
Mode #=> --file
```
$ ./clitest --list-run test/inline-match-file.sh
#1 OK printf '$ echo ok\nok\n'
#2 OK echo 'ok' > /tmp/foo.txt
#3 OK echo 'ok'
#4 FAIL echo 'fail'
#5 FAIL echo 'fail'
#6 OK echo '--file'
#7 OK echo '--filer'
#8 OK echo '--file is cool'
$
```
Mode #=> --lines
```
$ ./clitest --list-run test/inline-match-lines.sh
#1 OK a=1
#2 OK echo 'ok'
#3 OK printf '1\n2\n3\n'
#4 OK printf 'no-nl'
#5 OK printf '1\n2\nno-nl'
#6 FAIL echo 'fail'
#7 FAIL echo 'fail'
#8 FAIL echo 'fail'
#9 FAIL echo 'fail'
#10 OK echo '--lines'
#11 OK echo '--linesout'
#12 OK echo '--lines is cool'
$ ./clitest --first test/inline-match-lines.sh
#1 a=1
#2 echo 'ok'
#3 printf '1\n2\n3\n'
#4 printf 'no-nl'
#5 printf '1\n2\nno-nl'
#6 echo 'fail'
--------------------------------------------------------------------------------
[FAILED #6, line 16] echo 'fail'
Expected 99 lines, got 1.
--------------------------------------------------------------------------------
$
```
Mode #=> --exit
```
$ ./clitest --list-run test/inline-match-exit.sh
#1 OK true
#2 OK false
#3 OK sh -c 'exit 3'
#4 OK command-not-found
#5 OK echo "STDOUT ignored"
#6 OK cut
#7 OK echo "STDOUT ignored"
#8 OK cut
#9 FAIL echo 'fail'
#10 FAIL echo 'fail'
#11 FAIL echo 'fail'
#12 OK echo '--exit'
#13 OK echo '--exitout'
#14 OK echo '--exit is cool'
$ ./clitest --first test/inline-match-exit.sh
#1 true
#2 false
#3 sh -c 'exit 3'
#4 command-not-found
#5 echo "STDOUT ignored"
#6 cut
#7 echo "STDOUT ignored"
#8 cut
#9 echo 'fail'
--------------------------------------------------------------------------------
[FAILED #9, line 25] echo 'fail'
Expected exit code 99, got 0
--------------------------------------------------------------------------------
$
```
Errors for #=> --egrep
```
$ ./clitest test/inline-match-egrep-error-1.sh; echo $?
clitest: Error: empty --egrep at line 1 of test/inline-match-egrep-error-1.sh
2
$ ./clitest test/inline-match-egrep-error-2.sh 2>&1 | sed 's/^e*grep: .*/egrep: ERROR_MSG/'
#1 echo "error: malformed regex"
egrep: ERROR_MSG
clitest: Error: check your inline egrep regex at line 1 of test/inline-match-egrep-error-2.sh
$
```
Errors for #=> --perl (and --regex)
```
$ ./clitest test/inline-match-perl-error-1.sh; echo $?
clitest: Error: empty --perl at line 1 of test/inline-match-perl-error-1.sh
2
$ ./clitest test/inline-match-perl-error-2.sh
#1 echo "error: malformed regex"
Unmatched ( in regex; marked by <-- HERE in m/( <-- HERE / at -e line 1.
clitest: Error: check your inline Perl regex at line 1 of test/inline-match-perl-error-2.sh
$
```
Errors for #=> --file
```
$ ./clitest test/inline-match-file-error-1.sh; echo $?
clitest: Error: empty --file at line 1 of test/inline-match-file-error-1.sh
2
$ ./clitest test/inline-match-file-error-2.sh; echo $?
#1 echo "error: file not found"
clitest: Error: cannot read inline output file 'test/notfound', from line 1 of test/inline-match-file-error-2.sh
2
$ ./clitest test/inline-match-file-error-3.sh; echo $?
#1 echo "error: directory"
clitest: Error: cannot read inline output file '/etc/', from line 1 of test/inline-match-file-error-3.sh
2
$
```
Errors for #=> --lines
```
$ ./clitest test/inline-match-lines-error-1.sh
clitest: Error: --lines requires a number. See line 1 of test/inline-match-lines-error-1.sh
$ ./clitest test/inline-match-lines-error-2.sh
clitest: Error: --lines requires a number. See line 1 of test/inline-match-lines-error-2.sh
$ ./clitest test/inline-match-lines-error-3.sh
clitest: Error: --lines requires a number. See line 1 of test/inline-match-lines-error-3.sh
$ ./clitest test/inline-match-lines-error-4.sh; echo $?
clitest: Error: --lines requires a number. See line 1 of test/inline-match-lines-error-4.sh
2
$
```
Errors for #=> --exit
```
$ ./clitest test/inline-match-exit-error-1.sh
clitest: Error: --exit requires a number. See line 1 of test/inline-match-exit-error-1.sh
$ ./clitest test/inline-match-exit-error-2.sh
clitest: Error: --exit requires a number. See line 1 of test/inline-match-exit-error-2.sh
$ ./clitest test/inline-match-exit-error-3.sh
clitest: Error: --exit requires a number. See line 1 of test/inline-match-exit-error-3.sh
$ ./clitest test/inline-match-exit-error-4.sh; echo $?
clitest: Error: --exit requires a number. See line 1 of test/inline-match-exit-error-4.sh
2
$
```
Errors for #=> --eval
```
$ ./clitest test/inline-match-eval-error-1.sh; echo $?
clitest: Error: empty --eval at line 1 of test/inline-match-eval-error-1.sh
2
$
```
## Option -t, --test
Error: Invalid argument
```
$ ./clitest -t - test/ok-2.sh
clitest: Error: invalid argument for -t or --test: -
$ ./clitest -t -1 test/ok-2.sh
clitest: Error: invalid argument for -t or --test: -1
$ ./clitest -t 1- test/ok-2.sh
clitest: Error: invalid argument for -t or --test: 1-
$ ./clitest -t 1--2 test/ok-2.sh
clitest: Error: invalid argument for -t or --test: 1--2
$ ./clitest -t 1-2-3 test/ok-2.sh; echo $?
clitest: Error: invalid argument for -t or --test: 1-2-3
2
$
```
Error: Out of range
```
$ ./clitest -t 99 test/ok-2.sh; echo $?
clitest: Error: no test found for the specified number or range '99'
2
$
```
If range = zero or empty, run all tests
```
$ ./clitest -t '' test/ok-2.sh
#1 echo ok
#2 echo ok
OK: 2 of 2 tests passed
$ ./clitest -t 0 test/ok-2.sh
#1 echo ok
#2 echo ok
OK: 2 of 2 tests passed
$
```
* Empty values inside range are ignored
* The bogus `0-0` range is ignored
* The resulting range is zero
```
$ ./clitest -t ,,,0,0-0,,, test/ok-2.sh
#1 echo ok
#2 echo ok
OK: 2 of 2 tests passed
$
```
Normal operation, using `--test` and `-t`
```
$ ./clitest -t 1 test/ok-10.sh
#1 echo 1
OK: 1 of 10 tests passed (9 skipped)
$ ./clitest --test 1 test/ok-10.sh
#1 echo 1
OK: 1 of 10 tests passed (9 skipped)
$
```
Ranges `0-1` and `1-0` expand to `1`
```
$ ./clitest -t 0-1,1-0 test/ok-10.sh
#1 echo 1
OK: 1 of 10 tests passed (9 skipped)
$
```
Range `1-1` expand to `1`
```
$ ./clitest -t 1-1 test/ok-10.sh
#1 echo 1
OK: 1 of 10 tests passed (9 skipped)
$
```
Repeated values are OK
```
$ ./clitest -t 1,1,1,0,1 test/ok-10.sh
#1 echo 1
OK: 1 of 10 tests passed (9 skipped)
$
```
Range terminator is out of bounds
```
$ ./clitest -t 10-20 test/ok-10.sh
#10 echo 10
OK: 1 of 10 tests passed (9 skipped)
$
```
Inverted ranges
```
$ ./clitest -t 3,2,1 test/ok-10.sh
#1 echo 1
#2 echo 2
#3 echo 3
OK: 3 of 10 tests passed (7 skipped)
$ ./clitest -t 3-1 test/ok-10.sh
#1 echo 1
#2 echo 2
#3 echo 3
OK: 3 of 10 tests passed (7 skipped)
$
```
Multifile. The test numbers always increase sequentially, regardless of the file changes.
```
$ ./clitest -t 1,5,13 test/ok-?.sh test/ok-10.sh
Testing file test/ok-1.sh
#1 echo ok
Testing file test/ok-2.sh
Testing file test/ok-10.sh
#5 echo 2
#13 echo 10
ok fail skip
1 - - test/ok-1.sh
- - 2 test/ok-2.sh
2 - 8 test/ok-10.sh
OK: 3 of 13 tests passed (10 skipped)
$ ./clitest -t 1,5 test/ok-[12].sh test/fail-2.sh
Testing file test/ok-1.sh
#1 echo ok
Testing file test/ok-2.sh
Testing file test/fail-2.sh
#5 echo ok
--------------------------------------------------------------------------------
[FAILED #5, line 3] echo ok
@@ -1 +1 @@
-fail
+ok
--------------------------------------------------------------------------------
ok fail skip
1 - - test/ok-1.sh
- - 2 test/ok-2.sh
- 1 1 test/fail-2.sh
FAIL: 1 of 5 tests failed (3 skipped)
$ ./clitest -t 1 test/ok-[12].sh test/fail-2.sh
Testing file test/ok-1.sh
#1 echo ok
Testing file test/ok-2.sh
Testing file test/fail-2.sh
ok fail skip
1 - - test/ok-1.sh
- - 2 test/ok-2.sh
- - 2 test/fail-2.sh
OK: 1 of 5 tests passed (4 skipped)
$
```
## Option -s, --skip
Error: Invalid argument
```
$ ./clitest -s - test/ok-2.sh
clitest: Error: invalid argument for -s or --skip: -
$ ./clitest -s -1 test/ok-2.sh
clitest: Error: invalid argument for -s or --skip: -1
$ ./clitest -s 1- test/ok-2.sh
clitest: Error: invalid argument for -s or --skip: 1-
$ ./clitest -s 1--2 test/ok-2.sh
clitest: Error: invalid argument for -s or --skip: 1--2
$ ./clitest -s 1-2-3 test/ok-2.sh; echo $?
clitest: Error: invalid argument for -s or --skip: 1-2-3
2
$
```
Error: Skipped all tests
```
$ ./clitest -s 1 test/ok-1.sh; echo $?
clitest: Error: no test found. Maybe '--skip 1' was too much?
2
$
```
Out of range: no problem, you just skipped a non-existent test. All tests will be run.
```
$ ./clitest -s 99 test/ok-2.sh
#1 echo ok
#2 echo ok
OK: 2 of 2 tests passed
$
```
If range = zero or empty, run all tests
```
$ ./clitest -s '' test/ok-2.sh
#1 echo ok
#2 echo ok
OK: 2 of 2 tests passed
$ ./clitest -s 0 test/ok-2.sh
#1 echo ok
#2 echo ok
OK: 2 of 2 tests passed
$
```
* Empty values inside range are ignored
* The bogus `0-0` range is ignored
* The resulting range is zero
```
$ ./clitest -s ,,,0,0-0,,, test/ok-2.sh
#1 echo ok
#2 echo ok
OK: 2 of 2 tests passed
$
```
Normal operation, using `--skip` and `-s`
```
$ ./clitest -s 1 test/ok-2.sh
#2 echo ok
OK: 1 of 2 tests passed (1 skipped)
$ ./clitest --skip 1 test/ok-2.sh
#2 echo ok
OK: 1 of 2 tests passed (1 skipped)
$
```
Ranges `0-1` and `1-0` expand to `1`
```
$ ./clitest -s 0-1,1-0 test/ok-2.sh
#2 echo ok
OK: 1 of 2 tests passed (1 skipped)
$
```
Range `1-1` expand to `1`
```
$ ./clitest -s 1-1 test/ok-2.sh
#2 echo ok
OK: 1 of 2 tests passed (1 skipped)
$
```
Repeated values are OK
```
$ ./clitest -s 1,1,1,0,1 test/ok-2.sh
#2 echo ok
OK: 1 of 2 tests passed (1 skipped)
$
```
Range terminator is out of bounds
```
$ ./clitest -s 2-10 test/ok-2.sh
#1 echo ok
OK: 1 of 2 tests passed (1 skipped)
$
```
Inverted ranges
```
$ ./clitest -s 10,9,8,7,6,5,4 test/ok-10.sh
#1 echo 1
#2 echo 2
#3 echo 3
OK: 3 of 10 tests passed (7 skipped)
$ ./clitest -s 10-4 test/ok-10.sh
#1 echo 1
#2 echo 2
#3 echo 3
OK: 3 of 10 tests passed (7 skipped)
$
```
Multifile. The test numbers always increase sequentially, regardless of the file changes.
```
$ ./clitest -s 2,3,13 test/ok-?.sh test/ok-10.sh
Testing file test/ok-1.sh
#1 echo ok
Testing file test/ok-2.sh
Testing file test/ok-10.sh
#4 echo 1
#5 echo 2
#6 echo 3
#7 echo 4
#8 echo 5
#9 echo 6
#10 echo 7
#11 echo 8
#12 echo 9
ok fail skip
1 - - test/ok-1.sh
- - 2 test/ok-2.sh
9 - 1 test/ok-10.sh
OK: 10 of 13 tests passed (3 skipped)
$ ./clitest -s 2,3,4 test/ok-[12].sh test/fail-2.sh
Testing file test/ok-1.sh
#1 echo ok
Testing file test/ok-2.sh
Testing file test/fail-2.sh
#5 echo ok
--------------------------------------------------------------------------------
[FAILED #5, line 3] echo ok
@@ -1 +1 @@
-fail
+ok
--------------------------------------------------------------------------------
ok fail skip
1 - - test/ok-1.sh
- - 2 test/ok-2.sh
- 1 1 test/fail-2.sh
FAIL: 1 of 5 tests failed (3 skipped)
$ ./clitest -s 2-10 test/ok-[12].sh test/fail-2.sh
Testing file test/ok-1.sh
#1 echo ok
Testing file test/ok-2.sh
Testing file test/fail-2.sh
ok fail skip
1 - - test/ok-1.sh
- - 2 test/ok-2.sh
- - 2 test/fail-2.sh
OK: 1 of 5 tests passed (4 skipped)
$
```
## Option --test combined with --skip
Error: The combination of `-t` and `-s` resulted in no tests
```
$ ./clitest -t 9 -s 9 test/ok-10.sh; echo $?
clitest: Error: no test found. The combination of -t and -s resulted in no tests.
2
$
```
The order does not matter, `-s` always wins
```
$ ./clitest -s 9 -t 9 test/ok-10.sh
clitest: Error: no test found. The combination of -t and -s resulted in no tests.
$
```
Using `-t` to limit to a range and the `-s` exclude some more
```
$ ./clitest -t 3,5-7 -s 6 test/ok-10.sh
#3 echo 3
#5 echo 5
#7 echo 7
OK: 3 of 10 tests passed (7 skipped)
$
```
Same as previous, but now multifile
```
$ ./clitest -t 1,3,5-7 -s 3,6 test/ok-1.sh test/fail-2.sh test/ok-10.sh
Testing file test/ok-1.sh
#1 echo ok
Testing file test/fail-2.sh
Testing file test/ok-10.sh
#5 echo 2
#7 echo 4
ok fail skip
1 - - test/ok-1.sh
- - 2 test/fail-2.sh
2 - 8 test/ok-10.sh
OK: 3 of 13 tests passed (10 skipped)
$
```
## Option --diff-options
```
$ ./clitest test/option-diff-options.sh
#1 echo " diff -w to ignore spaces "
--------------------------------------------------------------------------------
[FAILED #1, line 3] echo " diff -w to ignore spaces "
@@ -1 +1 @@
-diff -w to ignore spaces
+ diff -w to ignore spaces
--------------------------------------------------------------------------------
#2 echo " diff -w now inline "
--------------------------------------------------------------------------------
[FAILED #2, line 5] echo " diff -w now inline "
@@ -1 +1 @@
-diff -w now inline
+ diff -w now inline
--------------------------------------------------------------------------------
FAIL: 2 of 2 tests failed
$ ./clitest --diff-options '-u -w' test/option-diff-options.sh
#1 echo " diff -w to ignore spaces "
#2 echo " diff -w now inline "
OK: 2 of 2 tests passed
$
```
## Option --prompt
```
$ ./clitest test/option-prompt.sh; echo $?
clitest: Error: no test found in input file: test/option-prompt.sh
2
$ ./clitest --prompt 'prompt$ ' test/option-prompt.sh
#1 echo "1"
#2 echo "2"
#3 echo "3"
OK: 3 of 3 tests passed
$ ./clitest --prompt '♥ ' test/option-prompt-unicode.sh
#1 echo "1"
#2 echo "2"
#3 echo "3"
OK: 3 of 3 tests passed
$
```
## Option --inline-prefix
```
$ ./clitest test/option-inline-prefix.sh
#1 echo "1 space" #==> 1 space
--------------------------------------------------------------------------------
[FAILED #1, line 3] echo "1 space" #==> 1 space
@@ -0,0 +1 @@
+1 space
--------------------------------------------------------------------------------
#2 echo "8 spaces" #==> 8 spaces
--------------------------------------------------------------------------------
[FAILED #2, line 4] echo "8 spaces" #==> 8 spaces
@@ -0,0 +1 @@
+8 spaces
--------------------------------------------------------------------------------
#3 echo "2 tabs" #==> 2 tabs
--------------------------------------------------------------------------------
[FAILED #3, line 5] echo "2 tabs" #==> 2 tabs
@@ -0,0 +1 @@
+2 tabs
--------------------------------------------------------------------------------
FAIL: 3 of 3 tests failed
$ ./clitest --inline-prefix '#==>' test/option-inline-prefix.sh
#1 echo "1 space"
--------------------------------------------------------------------------------
[FAILED #1, line 3] echo "1 space"
@@ -1 +1 @@
- 1 space
+1 space
--------------------------------------------------------------------------------
#2 echo "8 spaces"
--------------------------------------------------------------------------------
[FAILED #2, line 4] echo "8 spaces"
@@ -1 +1 @@
- 8 spaces
+8 spaces
--------------------------------------------------------------------------------
#3 echo "2 tabs"
--------------------------------------------------------------------------------
[FAILED #3, line 5] echo "2 tabs"
@@ -1 +1 @@
- 2 tabs
+2 tabs
--------------------------------------------------------------------------------
FAIL: 3 of 3 tests failed
$ ./clitest --inline-prefix '#==> ' test/option-inline-prefix.sh
#1 echo "1 space"
#2 echo "8 spaces"
#3 echo "2 tabs"
OK: 3 of 3 tests passed
$
```
## Option --prefix
```
$ ./clitest --prefix ' ' test/option-prefix.sh
#1 echo "1"
#2 echo "2"
#3 echo "3"
#4 echo "4"
#5 echo "5"
#6 echo; echo "6"; echo; echo "7"
OK: 6 of 6 tests passed
$ ./clitest --prefix 4 test/option-prefix.sh
#1 echo "1"
#2 echo "2"
#3 echo "3"
#4 echo "4"
#5 echo "5"
#6 echo; echo "6"; echo; echo "7"
OK: 6 of 6 tests passed
$ ./clitest --prefix '\t' test/option-prefix-tab.sh
#1 echo "1"
#2 echo "2"
#3 echo "3"
#4 echo "4"
#5 echo "5"
#6 echo; echo "6"; echo; echo "7"
OK: 6 of 6 tests passed
$ ./clitest --prefix tab test/option-prefix-tab.sh
#1 echo "1"
#2 echo "2"
#3 echo "3"
#4 echo "4"
#5 echo "5"
#6 echo; echo "6"; echo; echo "7"
OK: 6 of 6 tests passed
$
```
## Option --prefix: glob gotchas
```
$ ./clitest --prefix '?' test/option-prefix-glob.sh
#1 echo 'prefix ?'
#2 echo 'prefix ?'
OK: 2 of 2 tests passed
$ ./clitest --prefix '*' test/option-prefix-glob.sh
#1 echo 'prefix *'
#2 echo 'prefix *'
OK: 2 of 2 tests passed
$ ./clitest --prefix '#' test/option-prefix-glob.sh
#1 echo 'prefix #'
#2 echo 'prefix #'
OK: 2 of 2 tests passed
$ ./clitest --prefix '%' test/option-prefix-glob.sh
#1 echo 'prefix %'
#2 echo 'prefix %'
OK: 2 of 2 tests passed
$ ./clitest --prefix '##' test/option-prefix-glob.sh
#1 echo 'prefix ##'
#2 echo 'prefix ##'
OK: 2 of 2 tests passed
$ ./clitest --prefix '%%' test/option-prefix-glob.sh
#1 echo 'prefix %%'
#2 echo 'prefix %%'
OK: 2 of 2 tests passed
$ ./clitest --prefix '#*' test/option-prefix-glob.sh
#1 echo 'prefix #*'
#2 echo 'prefix #*'
OK: 2 of 2 tests passed
$ ./clitest --prefix '*#' test/option-prefix-glob.sh
#1 echo 'prefix *#'
#2 echo 'prefix *#'
OK: 2 of 2 tests passed
$ ./clitest --prefix '%*' test/option-prefix-glob.sh
#1 echo 'prefix %*'
#2 echo 'prefix %*'
OK: 2 of 2 tests passed
$ ./clitest --prefix '*%' test/option-prefix-glob.sh
#1 echo 'prefix *%'
#2 echo 'prefix *%'
OK: 2 of 2 tests passed
$
```
## Option --prompt: glob gotchas (char + space)
```
$ ./clitest --prompt '? ' test/option-prompt-glob-space.sh
#1 echo 'prompt ? '
#2 echo 'prompt ? '
OK: 2 of 2 tests passed
$ ./clitest --prompt '* ' test/option-prompt-glob-space.sh
#1 echo 'prompt * '
#2 echo 'prompt * '
OK: 2 of 2 tests passed
$ ./clitest --prompt '# ' test/option-prompt-glob-space.sh
#1 echo 'prompt # '
#2 echo 'prompt # '
OK: 2 of 2 tests passed
$ ./clitest --prompt '% ' test/option-prompt-glob-space.sh
#1 echo 'prompt % '
#2 echo 'prompt % '
OK: 2 of 2 tests passed
$ ./clitest --prompt '## ' test/option-prompt-glob-space.sh
#1 echo 'prompt ## '
#2 echo 'prompt ## '
OK: 2 of 2 tests passed
$ ./clitest --prompt '%% ' test/option-prompt-glob-space.sh
#1 echo 'prompt %% '
#2 echo 'prompt %% '
OK: 2 of 2 tests passed
$ ./clitest --prompt '#* ' test/option-prompt-glob-space.sh
#1 echo 'prompt #* '
#2 echo 'prompt #* '
OK: 2 of 2 tests passed
$ ./clitest --prompt '*# ' test/option-prompt-glob-space.sh
#1 echo 'prompt *# '
#2 echo 'prompt *# '
OK: 2 of 2 tests passed
$ ./clitest --prompt '%* ' test/option-prompt-glob-space.sh
#1 echo 'prompt %* '
#2 echo 'prompt %* '
OK: 2 of 2 tests passed
$ ./clitest --prompt '*% ' test/option-prompt-glob-space.sh
#1 echo 'prompt *% '
#2 echo 'prompt *% '
OK: 2 of 2 tests passed
$
```
## Option --prompt: glob gotchas (chars only)
```
$ ./clitest --prompt '?' test/option-prompt-glob-1.sh
#1 echo 'prompt ?'
#2 echo 'prompt ?'
OK: 2 of 2 tests passed
$ ./clitest --prompt '*' test/option-prompt-glob-1.sh
#1 echo 'prompt *'
#2 echo 'prompt *'
OK: 2 of 2 tests passed
$ ./clitest --prompt '#' test/option-prompt-glob-1.sh
#1 echo 'prompt #'
#2 echo 'prompt #'
OK: 2 of 2 tests passed
$ ./clitest --prompt '%' test/option-prompt-glob-1.sh
#1 echo 'prompt %'
#2 echo 'prompt %'
OK: 2 of 2 tests passed
$ ./clitest --prompt '##' test/option-prompt-glob-2.sh
#1 echo 'prompt ##'
#2 echo 'prompt ##'
OK: 2 of 2 tests passed
$ ./clitest --prompt '%%' test/option-prompt-glob-2.sh
#1 echo 'prompt %%'
#2 echo 'prompt %%'
OK: 2 of 2 tests passed
$ ./clitest --prompt '#*' test/option-prompt-glob-2.sh
#1 echo 'prompt #*'
#2 echo 'prompt #*'
OK: 2 of 2 tests passed
$ ./clitest --prompt '*#' test/option-prompt-glob-2.sh
#1 echo 'prompt *#'
#2 echo 'prompt *#'
OK: 2 of 2 tests passed
$ ./clitest --prompt '%*' test/option-prompt-glob-2.sh
#1 echo 'prompt %*'
#2 echo 'prompt %*'
OK: 2 of 2 tests passed
$ ./clitest --prompt '*%' test/option-prompt-glob-2.sh
#1 echo 'prompt *%'
#2 echo 'prompt *%'
OK: 2 of 2 tests passed
$
```
## Options --pre-flight and --post-flight
```
$ ./clitest --pre-flight 'tt_test_number=99; tt_nr_total_tests=99' test/ok-1.sh
#100 echo ok
OK: 100 of 100 tests passed
$ ./clitest --post-flight 'tt_nr_total_fails=50' test/ok-50.sh | tail -1
FAIL: 50 of 50 tests failed
$ ./clitest --pre-flight 'false' test/ok-1.sh; echo $?
clitest: Error: pre-flight command failed with status=1: false
2
$ ./clitest --post-flight 'false' test/ok-1.sh; echo $?
#1 echo ok
clitest: Error: post-flight command failed with status=1: false
2
$
```
## Invalid option
```
$ ./clitest --quiet --foo test/ok-1.sh
clitest: Error: invalid option --foo
$ ./clitest --first --foo test/ok-1.sh
clitest: Error: invalid option --foo
$ ./clitest --foo test/ok-1.sh
clitest: Error: invalid option --foo
$ ./clitest --foo
clitest: Error: invalid option --foo
$ ./clitest -Z; echo $?
clitest: Error: invalid option -Z
2
$
```
## Options terminator --
```
$ ./clitest -t 99 -- --quiet
clitest: Error: cannot read input file: --quiet
$
```
## File - meaning STDIN
```
$ cat test/ok-1.sh | ./clitest -
#1 echo ok
OK: 1 of 1 test passed
$ cat test/ok-1.sh | ./clitest -- -; echo $?
#1 echo ok
OK: 1 of 1 test passed
0
$
```
## Read test file from /dev/stdin
```
$ cat test/ok-1.sh | ./clitest /dev/stdin
#1 echo ok
OK: 1 of 1 test passed
$
```
## Test file is a symlink
```
$ ln -s test/ok-1.sh testsymlink
$ ./clitest testsymlink
#1 echo ok
OK: 1 of 1 test passed
$ rm testsymlink
$
```
## Gotchas
Test exit code and STDOUT/STDERR at the same time
```
$ ./clitest notfound; echo $?
clitest: Error: cannot read input file: notfound
2
$ ./clitest test/exit-code-and-stdout.sh
#1 echo "zero"; echo $?
#2 echo "two"; sh -c "exit 2"; echo $?
OK: 2 of 2 tests passed
$
```
STDIN and STDOUT
```
$ ./clitest test/stdout-stderr.sh
#1 echo "stdout"
#2 echo "stdout" 2> /dev/null
#3 echo "stderr" 1>&2
#4 echo "stdout" > /dev/null
#5 echo "stdout" 2> /dev/null 1>&2
#6 ./clitest notfound
#7 ./clitest notfound > /dev/null
#8 ./clitest notfound 2>&1
#9 ./clitest notfound 2> /dev/null
#10 ./clitest notfound > /dev/null 2>&1
OK: 10 of 10 tests passed
$
```
Multiple commands in one line
```
$ ./clitest test/multi-commands.sh
#1 echo 1; echo 2; echo 3; echo 4; echo 5
#2 (echo 1; echo 2; echo 3; echo 4; echo 5) | sed -n 3p
#3 (echo 1; echo 2; echo 3; echo 4; echo 5) | sed -n 3p
OK: 3 of 3 tests passed
$
```
A `cd` command in one test should not affect the next
```
$ ./clitest test/cd.sh test/ok-2.sh
Testing file test/cd.sh
#1 cd
Testing file test/ok-2.sh
#2 echo ok
#3 echo ok
ok fail skip
1 - - test/cd.sh
2 - - test/ok-2.sh
OK: 3 of 3 tests passed
$
```
Syntax: End-of-file or empty prompt closes the previous command
```
$ ./clitest test/close-command.sh
#1 echo 1
#2 echo 2
#3 echo 3
OK: 3 of 3 tests passed
$
```
Windows files (CR+LF)
```
$ ./clitest test/windows.sh
#1 echo "a file with CRLF line ending"
#2 echo "inline output"
#3 echo "inline regex"
OK: 3 of 3 tests passed
$
```
Unicode chars
```
$ ./clitest test/special-chars.sh | tail -1
OK: 206 of 206 tests passed
$
```
Blanks (space, tab, newline) in the output
```
$ ./clitest test/blank-output.sh
#1 echo ' '
#2 echo ' '
#3 printf '\t\n'
#4 printf '\t\t\t\n'
#5 printf ' \t \t\t \n'
#6 printf '\n \n \n \n \n\n'
#7 printf '\n\t\n\t\t\n\t\t\t\n\t\t\t\t\n\n'
#8 printf '\n'
#9 printf '\n\n'
#10 printf '\n\n\n\n'
OK: 10 of 10 tests passed
$
```
Files with no newline (`\n`) at the end
1. No empty prompt at the last line
2. Empty prompt at the last line
3. Inline output
```
$ ./clitest test/no-nl-file-1.sh
#1 printf '%s\n' 'a file with no \n at the last line'
OK: 1 of 1 test passed
$ ./clitest test/no-nl-file-2.sh
#1 printf '%s\n' 'another file with no \n at the last line'
OK: 1 of 1 test passed
$ ./clitest test/no-nl-file-3.sh
#1 printf '%s\n' 'oneliner, no \n'
OK: 1 of 1 test passed
$
```
Commands whose output has no `\n`
```
$ ./clitest test/no-nl-command.sh
#1 printf 'ok\n'
#2 printf 'fail'
--------------------------------------------------------------------------------
[FAILED #2, line 6] printf 'fail'
@@ -1 +1 @@
-fail
+fail
\ No newline at end of file
--------------------------------------------------------------------------------
#3 printf 'ok\nok\nfail'
--------------------------------------------------------------------------------
[FAILED #3, line 8] printf 'ok\nok\nfail'
@@ -1,3 +1,3 @@
ok
ok
-fail
+fail
\ No newline at end of file
--------------------------------------------------------------------------------
#4 printf 'ok\n'
#5 printf 'fail'
--------------------------------------------------------------------------------
[FAILED #5, line 17] printf 'fail'
@@ -1 +1 @@
-fail
+fail
\ No newline at end of file
--------------------------------------------------------------------------------
#6 printf 'ok'; echo
#7 printf 'ok'
FAIL: 3 of 7 tests failed
$
```
## And now, the colored output tests
```
$ ./clitest --color yes --first test/fail-2.sh
#1 echo ok
[31m--------------------------------------------------------------------------------[m
[31m[FAILED #1, line 1] echo ok[m
@@ -1 +1 @@
-fail
+ok
[31m--------------------------------------------------------------------------------[m
$
```