Codebase list clitest / 9c24930
Imported Upstream version 0.0+git20150312.339b52 Giovani Augusto Ferreira 8 years ago
70 changed file(s) with 5343 addition(s) and 0 deletion(s). Raw diff Collapse all Expand all
0 /dev/other/
1 /docs/
0 The MIT License (MIT)
1
2 Copyright (c) 2013 Aurelio Jargas
3
4 Permission is hereby granted, free of charge, to any person obtaining a copy of
5 this software and associated documentation files (the "Software"), to deal in
6 the Software without restriction, including without limitation the rights to
7 use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
8 the Software, and to permit persons to whom the Software is furnished to do so,
9 subject to the following conditions:
10
11 The above copyright notice and this permission notice shall be included in all
12 copies or substantial portions of the Software.
13
14 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
15 IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
16 FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
17 COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
18 IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
19 CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
0 # clitest – Command Line Tester
1
2 clitest is a [portable][1] POSIX shell script that performs automatic
3 testing in Unix command lines.
4
5 It's the same concept as in Python's [doctest][2] module: you document
6 both the commands and their expected output, using the familiar
7 interactive prompt format, and a specialized tool tests them.
8
9 In fact, the doctest [official][3] description can also be used for
10 clitest:
11
12 * The **doctest** module searches for pieces of text that look like
13 interactive **Python sessions**, and then executes those **sessions**
14 to verify that they work exactly as shown.
15
16 * The **clitest** command searches for pieces of text that look like
17 interactive **Unix command lines**, and then executes those
18 **command lines** to verify that they work exactly as shown.
19
20
21 ## Download
22
23 The full program is just [a single shell script file][4].
24
25 Save it and make it executable: `chmod +x clitest`
26
27
28 ## Quick Intro
29
30 Save the commands and their expected output in a text file:
31
32 ♦ [examples/intro.txt][5]
33
34 ```
35 $ echo "Hello World"
36 Hello World
37 $ cd /tmp
38 $ pwd
39 /tmp
40 $ cd "$OLDPWD"
41 $
42 ```
43
44 Use clitest to run these commands and check their output:
45
46 ```
47 $ clitest examples/intro.txt
48 #1 echo "Hello World"
49 #2 cd /tmp
50 #3 pwd
51 #4 cd "$OLDPWD"
52 OK: 4 of 4 tests passed
53 $
54 ```
55
56
57 ## CLI Syntax
58
59 There's no syntax to learn.
60
61 The test files are identical to the good old command line interface
62 (CLI) you're so familiar:
63
64 ♦ [examples/cut.txt][6]
65
66 ```
67 $ echo "one:two:three:four:five:six" | cut -d : -f 1
68 one
69 $ echo "one:two:three:four:five:six" | cut -d : -f 4
70 four
71 $ echo "one:two:three:four:five:six" | cut -d : -f 1,4
72 one:four
73 $ echo "one:two:three:four:five:six" | cut -d : -f 4,1
74 one:four
75 $ echo "one:two:three:four:five:six" | cut -d : -f 1-4
76 one:two:three:four
77 $ echo "one:two:three:four:five:six" | cut -d : -f 4-
78 four:five:six
79 $
80 ```
81
82 That's it.
83
84 Just paste your shell session inside a text file and you have a
85 ready-to-use test suite.
86
87 ```
88 $ clitest examples/cut.txt
89 #1 echo "one:two:three:four:five:six" | cut -d : -f 1
90 #2 echo "one:two:three:four:five:six" | cut -d : -f 4
91 #3 echo "one:two:three:four:five:six" | cut -d : -f 1,4
92 #4 echo "one:two:three:four:five:six" | cut -d : -f 4,1
93 #5 echo "one:two:three:four:five:six" | cut -d : -f 1-4
94 #6 echo "one:two:three:four:five:six" | cut -d : -f 4-
95 OK: 6 of 6 tests passed
96 $
97 ```
98
99
100 ## Test Documents
101
102 Ever wanted to test the command line instructions you give in the
103 `INSTALL.txt` or `README.md` files for your projects? Now you can!
104
105 clitest can also extract and run command lines from technical documents.
106
107 Given the following Markdown sample document, which uses tabs to mark
108 code blocks:
109
110 ♦ [examples/cut.md][7]
111
112 ```
113 The numeric ranges of the Unix command "cut"
114 ============================================
115
116 Use single numbers to extract one specific field:
117
118 $ echo "one:two:three:four:five:six" | cut -d : -f 1
119 one
120 $ echo "one:two:three:four:five:six" | cut -d : -f 4
121 four
122
123 Use commas to inform more than one field:
124
125 $ echo "one:two:three:four:five:six" | cut -d : -f 1,4
126 one:four
127
128 Note that inverting the order will *not* invert the output:
129
130 $ echo "one:two:three:four:five:six" | cut -d : -f 4,1
131 one:four
132
133 Use an hyphen to inform a range of fields, from one to four:
134
135 $ echo "one:two:three:four:five:six" | cut -d : -f 1-4
136 one:two:three:four
137
138 If you omit the second range number, it matches until the last:
139
140 $ echo "one:two:three:four:five:six" | cut -d : -f 4-
141 four:five:six
142
143 cut is cool, isn't it?
144 ```
145
146 It's easy to convert it to a readable HTML document with your favorite
147 Markdown program. It's also easy to test this file directly with
148 clitest, just inform that the command lines are prefixed by a tab:
149
150 ```
151 $ clitest --prefix tab examples/cut.md
152 #1 echo "one:two:three:four:five:six" | cut -d : -f 1
153 #2 echo "one:two:three:four:five:six" | cut -d : -f 4
154 #3 echo "one:two:three:four:five:six" | cut -d : -f 1,4
155 #4 echo "one:two:three:four:five:six" | cut -d : -f 4,1
156 #5 echo "one:two:three:four:five:six" | cut -d : -f 1-4
157 #6 echo "one:two:three:four:five:six" | cut -d : -f 4-
158 OK: 6 of 6 tests passed
159 $
160 ```
161
162 For Markdown files with 4-spaces indented code blocks, use `--prefix 4`.
163
164 Of course, this [README.md][8] file you are now reading is also
165 testable. Since it uses non-indented fenced code blocks (\`\`\`),
166 no prefix option is needed: `clitest README.md`.
167
168
169 ## Alternative Syntax: Inline Output
170
171 Now a nice extension to the original idea. Using the special marker
172 `#→` you can embed the expected command output at the end of the
173 command line.
174
175 ```
176 $ echo "foo" #→ foo
177 $ echo $((10 + 2)) #→ 12
178 ```
179
180 This is the same as doing:
181
182 ```
183 $ echo "foo"
184 foo
185 $ echo $((10 + 2))
186 12
187 $
188 ```
189
190 Inline outputs are very readable when testing series of commands that
191 result in short texts.
192
193 ```
194 $ echo "abcdef" | cut -c 1 #→ a
195 $ echo "abcdef" | cut -c 4 #→ d
196 $ echo "abcdef" | cut -c 1,4 #→ ad
197 $ echo "abcdef" | cut -c 1-4 #→ abcd
198 ```
199
200 > Note: The Unicode character `→` (U+2192) was chosen because it's
201 > meaningful and less likely to appear on a real command. If needed,
202 > you can change this marker (i.e., to `#->`) at the top of the script
203 > or using the `--inline-prefix` option.
204
205
206 ## Advanced Tests
207
208 When using the `#→` marker, you can take advantage of special options
209 to change the default output matching method.
210
211 ```
212 $ head /etc/passwd #→ --lines 10
213 $ tac /etc/passwd | tac #→ --file /etc/passwd
214 $ cat /etc/passwd #→ --egrep ^root:
215 $ echo $((2 + 10)) #→ --regex ^\d+$
216 $ pwd #→ --eval echo $PWD
217 ```
218
219 * Using `#→ --lines` the test will pass if the command output has
220 exactly `N` lines. Handy when the output text is variable
221 (unpredictable), but the number of resulting lines is constant.
222
223 * Using `#→ --file` the test will pass if the command output matches
224 the contents of an external file. Useful to organize long/complex
225 outputs into files.
226
227 * Using `#→ --egrep` the test will pass if `egrep` matches at least
228 one line of the command output.
229
230 * Using `#→ --regex` the test will pass if the command output is
231 matched by a [Perl regular expression][9]. A multiline output is
232 matched as a single string, with inner `\n`'s. Use the `(?ims)`
233 modifiers when needed.
234
235 * Using `#→ --eval` the test will pass if both commands result in the
236 same output. Useful to expand variables which store the full or
237 partial output.
238
239
240 ## Options
241
242 ```
243 $ clitest --help
244 Usage: clitest [options] <file ...>
245
246 Options:
247 -1, --first Stop execution upon first failed test
248 -l, --list List all the tests (no execution)
249 -L, --list-run List all the tests with OK/FAIL status
250 -t, --test RANGE Run specific tests, by number (1,2,4-7)
251 -s, --skip RANGE Skip specific tests, by number (1,2,4-7)
252 --pre-flight COMMAND Execute command before running the first test
253 --post-flight COMMAND Execute command after running the last test
254 -q, --quiet Quiet operation, no output shown
255 -V, --version Show program version and exit
256
257 Customization options:
258 -P, --progress TYPE Set progress indicator: test, number, dot, none
259 --color WHEN Set when to use colors: auto, always, never
260 --diff-options OPTIONS Set diff command options (default: '-u')
261 --inline-prefix PREFIX Set inline output prefix (default: '#→ ')
262 --prefix PREFIX Set command line prefix (default: '')
263 --prompt STRING Set prompt string (default: '$ ')
264 $
265 ```
266
267 When running sequential tests, where the next test depends on the
268 correct result of the previous test, use the `--first` option to abort
269 the execution if any test fails.
270
271 To rerun a specific problematic test, or to limit the execution to a
272 set of tests, use `--test`. To ignore one or more tests, use `--skip`.
273 If needed, you can combine both options to inform a very specific test
274 range. Examples:
275
276 clitest --test 1-10 tests.txt # Run the first 10 tests
277 clitest --test 1,2,6-8 tests.txt # Run tests #1, #2, #6, #7 and #8
278 clitest --skip 11,15 tests.txt # Run all tests, except #11 and #15
279 clitest -t 1-10 -s 5 tests.txt # Run first 10 tests, but skip #5
280
281 You can run a preparing script or command before the first test with
282 `--pre-flight`, for setting env variables and create auxiliary files.
283 At the end of all tests, run a final cleanup script/command with
284 `--post-flight` to remove temporary files or other transient data.
285 Example:
286
287
288 clitest --pre-flight ./test-init.sh --post-flight 'rm *.tmp' tests.txt
289
290 Use the customization options to extract and test command lines from
291 documents or wiki pages. For example, to test all the command line
292 examples listed inside a Markdown file using the 4-spaces syntax for
293 code blocks:
294
295 clitest --prefix 4 README.md
296
297 Or maybe you use a different prompt (`$PS1`) in your documentation?
298
299 clitest --prefix 4 --prompt '[john@localhost ~]$ ' README.md
300
301 When automating the tests execution, use `--quiet` to show no output
302 and just check the exit code to make sure all tests have passed.
303 Example:
304
305 if clitest --quiet tests.txt
306 then
307 # all tests passed
308 else
309 # one or more tests failed :(
310 fi
311
312
313 ## Nerdiness
314
315 * Use any file format for the tests, it doesn't matter. The command
316 lines just need to be grepable and have a fixed prefix (or none).
317 Even Windows text files (CR+LF) will work fine.
318
319 * The cmdline power is available in your test files: use variables,
320 pipes, redirection, create files, folders, move around…
321
322 * All the commands are tested in the same shell. Defined variables,
323 aliases and functions will persist between tests.
324
325 * Both STDIN and STDOUT are catch, you can also test error messages.
326
327 * To test the exit code, just add a `;echo $?` after the command.
328
329 * Use an empty `$` prompt to close the last command output.
330
331 * In the output, every single char (blank or not) counts. Any
332 difference will cause a test to fail. To ignore the difference in
333 blanks, use `--diff-options '-u -w'`.
334
335 * Unlike doctest's `<BLANKLINE>`, in clitest blank lines in the
336 command output aren't a problem. Just insert them normally.
337
338 * To test outputs with no final `\n`, such as `printf foo`, use `#→
339 --regex ^foo$`.
340
341 * In multifile mode, the current folder (`$PWD`) is reset when
342 starting to test a new file. This avoids that a `cd` command in a
343 previous file will affect the next.
344
345 * Multiline prompts (`$PS2`) are not yet supported.
346
347 * Ellipsis (as in doctest) are not supported. Use `#→ --regex`
348 instead.
349
350 * Simple examples in [examples/][10]. Hardcore examples in
351 [dev/test.md][11] and [dev/test/][12], the clitest own test-suite.
352
353
354 ## Portability
355
356 This script was carefully coded to be portable between [POSIX][13]
357 shells.
358
359 It was tested in:
360
361 * Bash 3.2
362 * dash 0.5.5.1
363 * ksh 93u 2011-02-08
364
365 Portability issues are considered serious bugs, please
366 [report them][14]!
367
368 Developers: Learn more about portability in POSIX shells:
369
370 * [How to make bash scripts work in dash][15]
371 * [Ubuntu — Dash as /bin/sh][16]
372 * [Rich’s sh (POSIX shell) tricks][17]
373 * [lintsh][18]
374 * [Official POSIX specification: Shell & Utilities][19]
375
376
377 ## [KISS][20]
378
379 A shell script to test shell commands.
380 No other language or environment involved.
381
382
383 ## Meta
384
385 * Author: [Aurelio Jargas][21]
386 * Created: 2013-07-24
387 * Language: Shell Script
388 * License: [MIT][22]
389
390
391 [1]: #portability
392 [2]: http://en.wikipedia.org/wiki/Doctest
393 [3]: http://docs.python.org/3/library/doctest.html
394 [4]: https://raw.github.com/aureliojargas/clitest/master/clitest
395 [5]: https://github.com/aureliojargas/clitest/blob/master/examples/intro.txt
396 [6]: https://github.com/aureliojargas/clitest/blob/master/examples/cut.txt
397 [7]: https://github.com/aureliojargas/clitest/blob/master/examples/cut.md
398 [8]: https://github.com/aureliojargas/clitest/blob/master/README.md
399 [9]: http://perldoc.perl.org/perlre.html
400 [10]: https://github.com/aureliojargas/clitest/blob/master/examples/
401 [11]: https://github.com/aureliojargas/clitest/blob/master/dev/test.md
402 [12]: https://github.com/aureliojargas/clitest/blob/master/dev/test/
403 [13]: http://en.wikipedia.org/wiki/POSIX
404 [14]: https://github.com/aureliojargas/clitest/issues
405 [15]: http://mywiki.wooledge.org/Bashism
406 [16]: https://wiki.ubuntu.com/DashAsBinSh
407 [17]: http://www.etalabs.net/sh_tricks.html
408 [18]: http://code.dogmap.org/lintsh/
409 [19]: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/contents.html
410 [20]: http://en.wikipedia.org/wiki/KISS_principle
411 [21]: http://aurelio.net/about.html
412 [22]: https://github.com/aureliojargas/clitest/blob/master/LICENSE.txt
0 #!/bin/sh
1 # clitest - Tester for Unix command lines
2 #
3 # Author: Aurelio Jargas (http://aurelio.net)
4 # Created: 2013-07-24
5 # License: MIT
6 # GitHub: https://github.com/aureliojargas/clitest
7 #
8 # POSIX shell script:
9 # This script was coded to be compatible with POSIX shells.
10 # Tested in Bash 3.2, dash 0.5.5.1, ksh 93u 2011-02-08.
11 # Note: Can't set -o posix nor POSIXLY_CORRECT: test env must be intact.
12 #
13 # Exit codes:
14 # 0 All tests passed, or normal operation (--help, --list, ...)
15 # 1 One or more tests have failed
16 # 2 An error occurred (file not found, invalid range, ...)
17 #
18 # Test environment:
19 # By default, the tests will run in the current working directory ($PWD).
20 # You can change to another dir normally using 'cd' inside the test file.
21 # All the tests are executed in the same shell, using eval. Test data
22 # such as variables and working directory will persist between tests.
23 #
24 # Namespace:
25 # All variables and functions in this script are prefixed by 'tt_' to
26 # avoid clashing with test's variables, functions, aliases and commands.
27
28 tt_my_name="$(basename "$0")"
29 tt_my_version='HEAD'
30 tt_my_version_url="https://github.com/aureliojargas/clitest/tree/$tt_my_version"
31
32 # Customization (if needed, edit here or use the command line options)
33 tt_prefix=''
34 tt_prompt='$ '
35 tt_inline_prefix='#→ ' # Problem with Unicode? Use '#=> ' or '### '
36 tt_diff_options='-u'
37 tt_color_mode='auto' # auto, always, never
38 tt_progress='test' # test, number, dot, none
39 # End of customization
40
41 # --help message, keep it simple, short and informative
42 tt_my_help="\
43 Usage: $tt_my_name [options] <file ...>
44
45 Options:
46 -1, --first Stop execution upon first failed test
47 -l, --list List all the tests (no execution)
48 -L, --list-run List all the tests with OK/FAIL status
49 -t, --test RANGE Run specific tests, by number (1,2,4-7)
50 -s, --skip RANGE Skip specific tests, by number (1,2,4-7)
51 --pre-flight COMMAND Execute command before running the first test
52 --post-flight COMMAND Execute command after running the last test
53 -q, --quiet Quiet operation, no output shown
54 -V, --version Show program version and exit
55
56 Customization options:
57 -P, --progress TYPE Set progress indicator: test, number, dot, none
58 --color WHEN Set when to use colors: auto, always, never
59 --diff-options OPTIONS Set diff command options (default: '$tt_diff_options')
60 --inline-prefix PREFIX Set inline output prefix (default: '$tt_inline_prefix')
61 --prefix PREFIX Set command line prefix (default: '$tt_prefix')
62 --prompt STRING Set prompt string (default: '$tt_prompt')"
63
64 # Temporary files (using files because <(...) is not portable)
65 tt_temp_dir="${TMPDIR:-/tmp}/clitest.$$"
66 tt_temp_file="$tt_temp_dir/temp.txt"
67 tt_test_ok_file="$tt_temp_dir/ok.txt"
68 tt_test_output_file="$tt_temp_dir/output.txt"
69
70 # Flags (0=off, 1=on), most can be altered by command line options
71 tt_debug=0
72 tt_use_colors=0
73 tt_stop_on_first_fail=0
74 tt_separator_line_shown=0
75
76 # The output mode values are mutually exclusive
77 tt_output_mode='normal' # normal, quiet, list, list-run
78
79 # Globals (all variables are globals, for better portability)
80 tt_nr_files=0
81 tt_nr_total_tests=0
82 tt_nr_total_fails=0
83 tt_nr_total_skips=0
84 tt_nr_file_tests=0
85 tt_nr_file_fails=0
86 tt_nr_file_skips=0
87 tt_nr_file_ok=0
88 tt_files_stats=
89 tt_original_dir=$(pwd)
90 tt_pre_command=
91 tt_post_command=
92 tt_run_range=
93 tt_run_range_data=
94 tt_skip_range=
95 tt_skip_range_data=
96 tt_failed_range=
97 tt_test_file=
98 tt_input_line=
99 tt_line_number=0
100 tt_test_number=0
101 tt_test_line_number=0
102 tt_test_command=
103 tt_test_inline=
104 tt_test_mode=
105 tt_test_status=2
106 tt_test_output=
107 tt_test_diff=
108 tt_test_ok_text=
109 tt_missing_nl=0
110
111 # Special handy chars
112 tt_tab=' '
113 tt_nl='
114 '
115
116 # Handle command line options
117 while test "${1#-}" != "$1"
118 do
119 case "$1" in
120 -1|--first ) shift; tt_stop_on_first_fail=1 ;;
121 -l|--list ) shift; tt_output_mode='list' ;;
122 -L|--list-run ) shift; tt_output_mode='list-run' ;;
123 -q|--quiet ) shift; tt_output_mode='quiet' ;;
124 -t|--test ) shift; tt_run_range="$1"; shift ;;
125 -s|--skip ) shift; tt_skip_range="$1"; shift ;;
126 --pre-flight ) shift; tt_pre_command="$1"; shift ;;
127 --post-flight ) shift; tt_post_command="$1"; shift ;;
128 --debug ) shift; tt_debug=1 ;;
129 -P|--progress ) shift; tt_progress="$1"; tt_output_mode='normal'; shift ;;
130 --color|--colour) shift; tt_color_mode="$1"; shift ;;
131 --diff-options ) shift; tt_diff_options="$1"; shift ;;
132 --inline-prefix ) shift; tt_inline_prefix="$1"; shift ;;
133 --prefix ) shift; tt_prefix="$1"; shift ;;
134 --prompt ) shift; tt_prompt="$1"; shift ;;
135 -h|--help)
136 printf '%s\n' "$tt_my_help"
137 exit 0
138 ;;
139 -V|--version)
140 printf '%s %s\n%s\n' $tt_my_name $tt_my_version $tt_my_version_url
141 exit 0
142 ;;
143 --) shift; break ;;
144 *) break ;;
145 esac
146 done
147
148 # Command line options consumed, now it's just the files
149 tt_nr_files=$#
150
151
152 ### Utilities
153
154 tt_clean_up ()
155 {
156 rm -rf "$tt_temp_dir"
157 }
158 tt_message ()
159 {
160 test "$tt_output_mode" = 'quiet' && return 0
161 test $tt_missing_nl -eq 1 && echo
162 printf '%s\n' "$*"
163 tt_separator_line_shown=0
164 tt_missing_nl=0
165 }
166 tt_message_part () # no line break
167 {
168 test "$tt_output_mode" = 'quiet' && return 0
169 printf '%s' "$*"
170 tt_separator_line_shown=0
171 tt_missing_nl=1
172 }
173 tt_error ()
174 {
175 test $tt_missing_nl -eq 1 && echo
176 printf '%s\n' "$tt_my_name: Error: $1" >&2
177 tt_clean_up
178 exit 2
179 }
180 tt_debug () # $1=id, $2=contents
181 {
182 test $tt_debug -ne 1 && return 0
183 if test INPUT_LINE = "$1"
184 then
185 # Original input line is all blue
186 printf "${tt_color_blue}[%10s: %s]${tt_color_off}\n" "$1" "$2"
187 else
188 # Highlight tabs and inline prefix
189 printf "${tt_color_blue}[%10s:${tt_color_off} %s${tt_color_blue}]${tt_color_off}\n" "$1" "$2" |
190 sed "/LINE_CMD:/ s/$tt_inline_prefix/${tt_color_red}&${tt_color_off}/g" |
191 sed "s/$tt_tab/${tt_color_green}<tab>${tt_color_off}/g"
192 fi
193 }
194 tt_separator_line ()
195 {
196 printf "%${COLUMNS}s" ' ' | tr ' ' -
197 }
198 tt_list_test () # $1=normal|list|ok|fail
199 {
200 # Show the test command in normal mode, --list and --list-run
201 case "$1" in
202 normal | list)
203 # Normal line, no color, no stamp (--list)
204 tt_message "#${tt_test_number}${tt_tab}${tt_test_command}"
205 ;;
206 ok)
207 # Green line or OK stamp (--list-run)
208 if test $tt_use_colors -eq 1
209 then
210 tt_message "${tt_color_green}#${tt_test_number}${tt_tab}${tt_test_command}${tt_color_off}"
211 else
212 tt_message "#${tt_test_number}${tt_tab}OK${tt_tab}${tt_test_command}"
213 fi
214 ;;
215 fail)
216 # Red line or FAIL stamp (--list-run)
217 if test $tt_use_colors -eq 1
218 then
219 tt_message "${tt_color_red}#${tt_test_number}${tt_tab}${tt_test_command}${tt_color_off}"
220 else
221 tt_message "#${tt_test_number}${tt_tab}FAIL${tt_tab}${tt_test_command}"
222 fi
223 ;;
224 esac
225 }
226 tt_parse_range () # $1=range
227 {
228 # Parse numeric ranges and output them in an expanded format
229 #
230 # Supported formats Expanded
231 # ------------------------------------------------------
232 # Single: 1 :1:
233 # List: 1,3,4,7 :1:3:4:7:
234 # Range: 1-4 :1:2:3:4:
235 # Mixed: 1,3,4-7,11,13-15 :1:3:4:5:6:7:11:13:14:15:
236 #
237 # Reverse ranges and repeated/unordered numbers are ok.
238 # Later we will just grep for :number: in each test.
239
240 case "$1" in
241 # No range, nothing to do
242 0 | '')
243 return 0
244 ;;
245 # Error: strange chars, not 0123456789,-
246 *[!0-9,-]*)
247 return 1
248 ;;
249 esac
250
251 # OK, all valid chars in range, let's parse them
252
253 tt_part=
254 tt_n1=
255 tt_n2=
256 tt_operation=
257 tt_range_data=':' # :1:2:4:7:
258
259 # Loop each component: a number or a range
260 for tt_part in $(echo "$1" | tr , ' ')
261 do
262 # If there's an hyphen, it's a range
263 case "$tt_part" in
264 *-*)
265 # Error: Invalid range format, must be: number-number
266 echo "$tt_part" | grep '^[0-9][0-9]*-[0-9][0-9]*$' > /dev/null || return 1
267
268 tt_n1=${tt_part%-*}
269 tt_n2=${tt_part#*-}
270
271 tt_operation='+'
272 test $tt_n1 -gt $tt_n2 && tt_operation='-'
273
274 # Expand the range (1-4 => 1:2:3:4)
275 tt_part=$tt_n1:
276 while test $tt_n1 -ne $tt_n2
277 do
278 tt_n1=$(($tt_n1 $tt_operation 1))
279 tt_part=$tt_part$tt_n1:
280 done
281 tt_part=${tt_part%:}
282 ;;
283 esac
284
285 # Append the number or expanded range to the holder
286 test $tt_part != 0 && tt_range_data=$tt_range_data$tt_part:
287 done
288
289 test $tt_range_data != ':' && echo $tt_range_data
290 return 0
291 }
292 tt_reset_test_data ()
293 {
294 tt_test_command=
295 tt_test_inline=
296 tt_test_mode=
297 tt_test_status=2
298 tt_test_output=
299 tt_test_diff=
300 tt_test_ok_text=
301 }
302 tt_run_test ()
303 {
304 tt_test_number=$(($tt_test_number + 1))
305 tt_nr_total_tests=$(($tt_nr_total_tests + 1))
306 tt_nr_file_tests=$(($tt_nr_file_tests + 1))
307
308 # Run range on: skip this test if it's not listed in $tt_run_range_data
309 if test -n "$tt_run_range_data" && test "$tt_run_range_data" = "${tt_run_range_data#*:$tt_test_number:}"
310 then
311 tt_nr_total_skips=$(($tt_nr_total_skips + 1))
312 tt_nr_file_skips=$(($tt_nr_file_skips + 1))
313 tt_reset_test_data
314 return 0
315 fi
316
317 # Skip range on: skip this test if it's listed in $tt_skip_range_data
318 # Note: --skip always wins over --test, regardless of order
319 if test -n "$tt_skip_range_data" && test "$tt_skip_range_data" != "${tt_skip_range_data#*:$tt_test_number:}"
320 then
321 tt_nr_total_skips=$(($tt_nr_total_skips + 1))
322 tt_nr_file_skips=$(($tt_nr_file_skips + 1))
323 tt_reset_test_data
324 return 0
325 fi
326
327 case "$tt_output_mode" in
328 normal)
329 # Normal mode: show progress indicator
330 case "$tt_progress" in
331 test)
332 tt_list_test normal
333 ;;
334 number)
335 tt_message_part "$tt_test_number "
336 ;;
337 none)
338 :
339 ;;
340 *)
341 tt_message_part "$tt_progress"
342 ;;
343 esac
344 ;;
345 list)
346 # List mode: just show the command and return (no execution)
347 tt_list_test list
348 tt_reset_test_data
349 return 0
350 ;;
351 esac
352
353 #tt_debug EVAL "$tt_test_command"
354
355 # Execute the test command, saving output (STDOUT and STDERR)
356 eval "$tt_test_command" > "$tt_test_output_file" 2>&1
357
358 #tt_debug OUTPUT "$(cat "$tt_test_output_file")"
359
360 # The command output matches the expected output?
361 case $tt_test_mode in
362 output)
363 printf %s "$tt_test_ok_text" > "$tt_test_ok_file"
364 tt_test_diff=$(diff $tt_diff_options "$tt_test_ok_file" "$tt_test_output_file")
365 tt_test_status=$?
366 ;;
367 text)
368 # Inline OK text represents a full line, with \n
369 printf '%s\n' "$tt_test_inline" > "$tt_test_ok_file"
370 tt_test_diff=$(diff $tt_diff_options "$tt_test_ok_file" "$tt_test_output_file")
371 tt_test_status=$?
372 ;;
373 eval)
374 eval "$tt_test_inline" > "$tt_test_ok_file"
375 tt_test_diff=$(diff $tt_diff_options "$tt_test_ok_file" "$tt_test_output_file")
376 tt_test_status=$?
377 ;;
378 lines)
379 tt_test_output=$(sed -n '$=' "$tt_test_output_file")
380 test -z "$tt_test_output" && tt_test_output=0
381 test "$tt_test_output" -eq "$tt_test_inline"
382 tt_test_status=$?
383 tt_test_diff="Expected $tt_test_inline lines, got $tt_test_output."
384 ;;
385 file)
386 # If path is relative, make it relative to the test file path, not $PWD
387 if test $tt_test_inline = ${tt_test_inline#/}
388 then
389 tt_test_inline="$(dirname "$tt_test_file")/$tt_test_inline"
390 fi
391 # Abort when ok file not found/readable
392 if test ! -f "$tt_test_inline" || test ! -r "$tt_test_inline"
393 then
394 tt_error "cannot read inline output file '$tt_test_inline', from line $tt_line_number of $tt_test_file"
395 fi
396
397 tt_test_diff=$(diff $tt_diff_options "$tt_test_inline" "$tt_test_output_file")
398 tt_test_status=$?
399 ;;
400 egrep)
401 egrep "$tt_test_inline" "$tt_test_output_file" > /dev/null
402 tt_test_status=$?
403
404 # Test failed: the regex not matched
405 if test $tt_test_status -eq 1
406 then
407 tt_test_diff="egrep '$tt_test_inline' failed in:$tt_nl$(cat "$tt_test_output_file")"
408
409 # Regex errors are common and user must take action to fix them
410 elif test $tt_test_status -eq 2
411 then
412 tt_error "check your inline egrep regex at line $tt_line_number of $tt_test_file"
413 fi
414 ;;
415 perl | regex)
416 # Escape regex delimiter (if any) inside the regex: ' => \'
417 if test "$tt_test_inline" != "${tt_test_inline#*\'}"
418 then
419 tt_test_inline=$(printf %s "$tt_test_inline" | sed "s/'/\\\\'/g")
420 fi
421
422 # Note: -0777 to get the full file contents as a single string
423 perl -0777 -ne "exit(!m'$tt_test_inline')" "$tt_test_output_file"
424 tt_test_status=$?
425
426 case $tt_test_status in
427 0) # Test matched, nothing to do
428 :
429 ;;
430 1) # Test failed: the regex not matched
431 tt_test_diff="Perl regex '$tt_test_inline' not matched in:$tt_nl$(cat "$tt_test_output_file")"
432 ;;
433 127) # Perl not found :(
434 tt_error "Perl not found. It's needed by --$tt_test_mode at line $tt_line_number of $tt_test_file"
435 ;;
436 255) # Regex syntax errors are common and user must take action to fix them
437 tt_error "check your inline Perl regex at line $tt_line_number of $tt_test_file"
438 ;;
439 *)
440 tt_error "unknown error when running Perl for --$tt_test_mode at line $tt_line_number of $tt_test_file"
441 ;;
442 esac
443 ;;
444 *)
445 tt_error "unknown test mode '$tt_test_mode'"
446 ;;
447 esac
448
449 # Test failed :(
450 if test $tt_test_status -ne 0
451 then
452 tt_nr_file_fails=$(($tt_nr_file_fails + 1))
453 tt_nr_total_fails=$(($tt_nr_total_fails + 1))
454 tt_failed_range="$tt_failed_range$tt_test_number,"
455
456 # Decide the message format
457 if test "$tt_output_mode" = 'list-run'
458 then
459 # List mode
460 tt_list_test fail
461 else
462 # Normal mode: show FAILED message and the diff
463 if test $tt_separator_line_shown -eq 0 # avoid dups
464 then
465 tt_message "${tt_color_red}$(tt_separator_line)${tt_color_off}"
466 fi
467 tt_message "${tt_color_red}[FAILED #$tt_test_number, line $tt_test_line_number] $tt_test_command${tt_color_off}"
468 tt_message "$tt_test_diff" | sed '1 { /^--- / { N; /\n+++ /d; }; }' # no ---/+++ headers
469 tt_message "${tt_color_red}$(tt_separator_line)${tt_color_off}"
470 tt_separator_line_shown=1
471 fi
472
473 # Should I abort now?
474 if test $tt_stop_on_first_fail -eq 1
475 then
476 tt_clean_up
477 exit 1
478 fi
479
480 # Test OK
481 else
482 test "$tt_output_mode" = 'list-run' && tt_list_test ok
483 fi
484
485 tt_reset_test_data
486 }
487 tt_process_test_file ()
488 {
489 # Reset counters
490 tt_nr_file_tests=0
491 tt_nr_file_fails=0
492 tt_nr_file_skips=0
493 tt_line_number=0
494 tt_test_line_number=0
495
496 # Loop for each line of input file
497 # Note: changing IFS to avoid right-trimming of spaces/tabs
498 # Note: read -r to preserve the backslashes
499 while IFS='' read -r tt_input_line || test -n "$tt_input_line"
500 do
501 tt_line_number=$(($tt_line_number + 1))
502 #tt_debug INPUT_LINE "$tt_input_line"
503
504 case "$tt_input_line" in
505
506 # Prompt alone: closes previous command line (if any)
507 "$tt_prefix$tt_prompt" | "$tt_prefix${tt_prompt% }" | "$tt_prefix$tt_prompt ")
508 #tt_debug 'LINE_$' "$tt_input_line"
509
510 # Run pending tests
511 test -n "$tt_test_command" && tt_run_test
512 ;;
513
514 # This line is a command line to be tested
515 "$tt_prefix$tt_prompt"*)
516 #tt_debug LINE_CMD "$tt_input_line"
517
518 # Run pending tests
519 test -n "$tt_test_command" && tt_run_test
520
521 # Remove the prompt
522 tt_test_command="${tt_input_line#"$tt_prefix$tt_prompt"}"
523
524 # Save the test's line number for future messages
525 tt_test_line_number=$tt_line_number
526
527 # This is a special test with inline output?
528 if printf '%s\n' "$tt_test_command" | grep "$tt_inline_prefix" > /dev/null
529 then
530 # Separate command from inline output
531 tt_test_command="${tt_test_command%"$tt_inline_prefix"*}"
532 tt_test_inline="${tt_input_line##*"$tt_inline_prefix"}"
533
534 #tt_debug NEW_CMD "$tt_test_command"
535 #tt_debug OK_INLINE "$tt_test_inline"
536
537 # Maybe the OK text has options?
538 case "$tt_test_inline" in
539 '--egrep '*)
540 tt_test_inline=${tt_test_inline#--egrep }
541 tt_test_mode='egrep'
542 ;;
543 '--regex '*) # alias to --perl
544 tt_test_inline=${tt_test_inline#--regex }
545 tt_test_mode='regex'
546 ;;
547 '--perl '*)
548 tt_test_inline=${tt_test_inline#--perl }
549 tt_test_mode='perl'
550 ;;
551 '--file '*)
552 tt_test_inline=${tt_test_inline#--file }
553 tt_test_mode='file'
554 ;;
555 '--lines '*)
556 tt_test_inline=${tt_test_inline#--lines }
557 tt_test_mode='lines'
558 ;;
559 '--eval '*)
560 tt_test_inline=${tt_test_inline#--eval }
561 tt_test_mode='eval'
562 ;;
563 '--text '*)
564 tt_test_inline=${tt_test_inline#--text }
565 tt_test_mode='text'
566 ;;
567 *)
568 tt_test_mode='text'
569 ;;
570 esac
571
572 #tt_debug OK_TEXT "$tt_test_inline"
573
574 # There must be a number in --lines
575 if test "$tt_test_mode" = 'lines'
576 then
577 case "$tt_test_inline" in
578 '' | *[!0-9]*)
579 tt_error "--lines requires a number. See line $tt_line_number of $tt_test_file"
580 ;;
581 esac
582 fi
583
584 # An empty inline parameter is an error user must see
585 if test -z "$tt_test_inline" && test "$tt_test_mode" != 'text'
586 then
587 tt_error "empty --$tt_test_mode at line $tt_line_number of $tt_test_file"
588 fi
589
590 # Since we already have the command and the output, run test
591 tt_run_test
592 else
593 # It's a normal command line, output begins in next line
594 tt_test_mode='output'
595
596 #tt_debug NEW_CMD "$tt_test_command"
597 fi
598 ;;
599
600 # Test output, blank line or comment
601 *)
602 #tt_debug 'LINE_*' "$tt_input_line"
603
604 # Ignore this line if there's no pending test
605 test -n "$tt_test_command" || continue
606
607 # Required prefix is missing: we just left a command block
608 if test -n "$tt_prefix" && test "${tt_input_line#"$tt_prefix"}" = "$tt_input_line"
609 then
610 #tt_debug BLOCK_OUT "$tt_input_line"
611
612 # Run the pending test and we're done in this line
613 tt_run_test
614 continue
615 fi
616
617 # This line is a test output, save it (without prefix)
618 tt_test_ok_text="$tt_test_ok_text${tt_input_line#"$tt_prefix"}$tt_nl"
619
620 #tt_debug OK_TEXT "${tt_input_line#"$tt_prefix"}"
621 ;;
622 esac
623 done < "$tt_temp_file"
624
625 #tt_debug LOOP_OUT "\$tt_test_command=$tt_test_command"
626
627 # Run pending tests
628 test -n "$tt_test_command" && tt_run_test
629 }
630
631
632 ### Init process
633
634
635 # No files?
636 if test $tt_nr_files -eq 0
637 then
638 tt_error 'no test file informed (try --help)'
639 fi
640
641 # Handy shortcuts for prefixes
642 case "$tt_prefix" in
643 tab)
644 tt_prefix="$tt_tab"
645 ;;
646 0)
647 tt_prefix=''
648 ;;
649 [1-9] | [1-9][0-9]) # 1-99
650 # convert number to spaces: 2 => ' '
651 tt_prefix=$(printf "%${tt_prefix}s" ' ')
652 ;;
653 *\\*)
654 tt_prefix="$(printf %b "$tt_prefix")" # expand \t and others
655 ;;
656 esac
657
658 # Validate and normalize progress value
659 if test "$tt_output_mode" = 'normal'
660 then
661 case "$tt_progress" in
662 test)
663 :
664 ;;
665 number | n | [0-9])
666 tt_progress='number'
667 ;;
668 dot | .)
669 tt_progress='.'
670 ;;
671 none | no)
672 tt_progress='none'
673 ;;
674 ?) # Single char, use it as the progress
675 :
676 ;;
677 *)
678 tt_error "invalid value '$tt_progress' for --progress. Use: test, number, dot or none."
679 ;;
680 esac
681 fi
682
683 # Will we use colors in the output?
684 case "$tt_color_mode" in
685 always | yes | y)
686 tt_use_colors=1
687 ;;
688 never | no | n)
689 tt_use_colors=0
690 ;;
691 auto | a)
692 # The auto mode will use colors if the output is a terminal
693 # Note: test -t is in POSIX
694 if test -t 1
695 then
696 tt_use_colors=1
697 else
698 tt_use_colors=0
699 fi
700 ;;
701 *)
702 tt_error "invalid value '$tt_color_mode' for --color. Use: auto, always or never."
703 ;;
704 esac
705
706 # Set colors
707 # Remember: colors must be readable in dark and light backgrounds
708 # Customization: tweak the numbers after [ to adjust the colors
709 if test $tt_use_colors -eq 1
710 then
711 tt_color_red=$( printf '\033[31m') # fail
712 tt_color_green=$(printf '\033[32m') # ok
713 tt_color_blue=$( printf '\033[34m') # debug
714 #tt_color_cyan=$( printf '\033[36m') # not used
715 tt_color_off=$( printf '\033[m')
716 fi
717
718 # Find the terminal width
719 # The COLUMNS env var is set by Bash (must be exported in ~/.bashrc).
720 # In other shells, try to use 'tput cols' (not POSIX).
721 # If not, defaults to 50 columns, a conservative amount.
722 : ${COLUMNS:=$(tput cols 2> /dev/null)}
723 : ${COLUMNS:=50}
724
725 # Parse and validate --test option value, if informed
726 tt_run_range_data=$(tt_parse_range "$tt_run_range")
727 if test $? -ne 0
728 then
729 tt_error "invalid argument for -t or --test: $tt_run_range"
730 fi
731
732 # Parse and validate --skip option value, if informed
733 tt_skip_range_data=$(tt_parse_range "$tt_skip_range")
734 if test $? -ne 0
735 then
736 tt_error "invalid argument for -s or --skip: $tt_skip_range"
737 fi
738
739 # Create temp dir, protected from others
740 umask 077 && mkdir "$tt_temp_dir" || tt_error "cannot create temporary dir: $tt_temp_dir"
741
742
743 ### Real execution begins here
744
745 # Some preparing command to run before all the tests?
746 if test -n "$tt_pre_command"
747 then
748 eval "$tt_pre_command" ||
749 tt_error "pre-flight command failed with status=$?: $tt_pre_command"
750 fi
751
752 # For each input file in $@
753 for tt_test_file
754 do
755 # Some tests may 'cd' to another dir, we need to get back
756 # to preserve the relative paths of the input files
757 cd "$tt_original_dir"
758
759 # Abort when test file not found/readable
760 if test ! -f "$tt_test_file" || test ! -r "$tt_test_file"
761 then
762 tt_error "cannot read input file: $tt_test_file"
763 fi
764
765 # In multifile mode, identify the current file
766 if test $tt_nr_files -gt 1
767 then
768 case "$tt_output_mode" in
769 normal)
770 # Normal mode, show message with filename
771 case "$tt_progress" in
772 test | none)
773 tt_message "Testing file $tt_test_file"
774 ;;
775 *)
776 test $tt_missing_nl -eq 1 && echo
777 tt_message_part "Testing file $tt_test_file "
778 ;;
779 esac
780 ;;
781 list | list-run)
782 # List mode, show ------ and the filename
783 tt_message $(tt_separator_line | cut -c 1-40) $tt_test_file
784 ;;
785 esac
786 fi
787
788 # Convert Windows files (CRLF) to the Unix format (LF)
789 # Note: the temporary file is required, because doing "sed | while" opens
790 # a subshell and global vars won't be updated outside the loop.
791 sed "s/$(printf '\r')$//" "$tt_test_file" > "$tt_temp_file"
792
793 # The magic happens here
794 tt_process_test_file
795
796 # Abort when no test found (and no active range with --test or --skip)
797 if test $tt_nr_file_tests -eq 0 && test -z "$tt_run_range_data" && test -z "$tt_skip_range_data"
798 then
799 tt_error "no test found in input file: $tt_test_file"
800 fi
801
802 # Save file stats
803 tt_nr_file_ok=$(($tt_nr_file_tests - $tt_nr_file_fails - $tt_nr_file_skips))
804 tt_files_stats="$tt_files_stats$tt_nr_file_ok $tt_nr_file_fails $tt_nr_file_skips$tt_nl"
805
806 # Dots mode: any missing new line?
807 # Note: had to force tt_missing_nl=0, even when it's done in tt_message :/
808 test $tt_missing_nl -eq 1 && tt_missing_nl=0 && tt_message
809 done
810
811 tt_clean_up
812
813 # Some clean up command to run after all the tests?
814 if test -n "$tt_post_command"
815 then
816 eval "$tt_post_command"
817 fi
818
819 #-----------------------------------------------------------------------
820 # From this point on, it's safe to use non-prefixed global vars
821 #-----------------------------------------------------------------------
822
823 # Range active, but no test matched :(
824 if test $tt_nr_total_tests -eq $tt_nr_total_skips
825 then
826 if test -n "$tt_run_range_data" && test -n "$tt_skip_range_data"
827 then
828 tt_error "no test found. The combination of -t and -s resulted in no tests."
829 elif test -n "$tt_run_range_data"
830 then
831 tt_error "no test found for the specified number or range '$tt_run_range'"
832 elif test -n "$tt_skip_range_data"
833 then
834 tt_error "no test found. Maybe '--skip $tt_skip_range' was too much?"
835 fi
836 fi
837
838 # List mode has no stats
839 if test "$tt_output_mode" = 'list' || test "$tt_output_mode" = 'list-run'
840 then
841 if test $tt_nr_total_fails -eq 0
842 then
843 exit 0
844 else
845 exit 1
846 fi
847 fi
848
849 # Show stats
850 # Data:
851 # $tt_files_stats -> "100 0 23 \n 12 34 0"
852 # $@ -> foo.sh bar.sh
853 # Output:
854 # ok fail skip
855 # 100 0 23 foo.sh
856 # 12 34 0 bar.sh
857 if test $tt_nr_files -gt 1 && test "$tt_output_mode" != 'quiet'
858 then
859 echo
860 printf ' %5s %5s %5s\n' ok fail skip
861 printf %s "$tt_files_stats" | while read ok fail skip
862 do
863 printf ' %5s %5s %5s %s\n' $ok $fail $skip "$1"
864 shift
865 done | sed 's/ 0/ -/g' # hide zeros
866 echo
867 fi
868
869 # The final message: OK or FAIL?
870 # OK: 123 of 123 tests passed
871 # OK: 100 of 123 tests passed (23 skipped)
872 # FAIL: 123 of 123 tests failed
873 # FAIL: 100 of 123 tests failed (23 skipped)
874 skips=
875 if test $tt_nr_total_skips -gt 0
876 then
877 skips=" ($tt_nr_total_skips skipped)"
878 fi
879 if test $tt_nr_total_fails -eq 0
880 then
881 stamp="${tt_color_green}OK:${tt_color_off}"
882 stats="$(($tt_nr_total_tests - $tt_nr_total_skips)) of $tt_nr_total_tests tests passed"
883 test $tt_nr_total_tests -eq 1 && stats=$(echo "$stats" | sed 's/tests /test /')
884 tt_message "$stamp $stats$skips"
885 exit 0
886 else
887 test $tt_nr_files -eq 1 && tt_message # separate from previous FAILED message
888
889 stamp="${tt_color_red}FAIL:${tt_color_off}"
890 stats="$tt_nr_total_fails of $tt_nr_total_tests tests failed"
891 test $tt_nr_total_tests -eq 1 && stats=$(echo "$stats" | sed 's/tests /test /')
892 tt_message "$stamp $stats$skips"
893 test $tt_test_file = 'dev/test.md' && tt_message "-t ${tt_failed_range%,}" # XXX dev helper, remove before release
894 exit 1
895 fi
0 $ echo ' '
1
2 $ echo ' '
3
4 $ printf '\t\n'
5
6 $ printf '\t\t\t\n'
7
8 $ printf ' \t \t\t \n'
9
10 $ printf '\n \n \n \n \n\n'
11
12
13
14
15
16
17 $ printf '\n\t\n\t\t\n\t\t\t\n\t\t\t\t\n\n'
18
19
20
21
22
23
24 $ printf '\n'
25
26 $ printf '\n\n'
27
28
29 $ printf '\n\n\n\n'
30
31
32
33
34 $
0 # Some tests may change the current dir.
1 # This could affect the next tests.
2 # This could affect the relative path for the next input file.
3 # To avoid problems, the current dir is always reset when a new test file is read.
4
5 $ cd
0 # Syntax: Empty prompt (with space) closes the previous command
1
2 $ echo 1
3 1
4 $
5
6 # Syntax: Empty prompt (no space) closes the previous command
7
8 $ echo 2
9 2
10 $
11
12 # Syntax: Repeated empty prompts are OK
13
14 $
15 $
16 $
17
18 # Syntax: End-of-file closes the last command
19
20 $ echo 3
21 3
(New empty file)
0 $
1 $
2 $
3
4 # Comment
5
6 $
7 $
8 $
9
10 # Comment
11
12 $
13
14 $
15
16 $
0 $ echo "ok" > /dev/null; echo $?
1 0
2 $ cp XXnotfoundXX foo 2> /dev/null; echo $?
3 1
0 $ echo ok
1 fail
0 $ echo ok
1 fail
2 $ echo ok #→ fail
0 $ echo 1 #→ fail
1 $ echo 2 #→ fail
2 $ echo 3 #→ fail
3 $ echo 4 #→ fail
4 $ echo 5 #→ fail
5 $ echo 6 #→ fail
6 $ echo 7 #→ fail
7 $ echo 8 #→ fail
8 $ echo 9 #→ fail
9 $ echo 10 #→ fail
10 $ echo 11 #→ fail
11 $ echo 12 #→ fail
12 $ echo 13 #→ fail
13 $ echo 14 #→ fail
14 $ echo 15 #→ fail
15 $ echo 16 #→ fail
16 $ echo 17 #→ fail
17 $ echo 18 #→ fail
18 $ echo 19 #→ fail
19 $ echo 20 #→ fail
20 $ echo 21 #→ fail
21 $ echo 22 #→ fail
22 $ echo 23 #→ fail
23 $ echo 24 #→ fail
24 $ echo 25 #→ fail
25 $ echo 26 #→ fail
26 $ echo 27 #→ fail
27 $ echo 28 #→ fail
28 $ echo 29 #→ fail
29 $ echo 30 #→ fail
30 $ echo 31 #→ fail
31 $ echo 32 #→ fail
32 $ echo 33 #→ fail
33 $ echo 34 #→ fail
34 $ echo 35 #→ fail
35 $ echo 36 #→ fail
36 $ echo 37 #→ fail
37 $ echo 38 #→ fail
38 $ echo 39 #→ fail
39 $ echo 40 #→ fail
40 $ echo 41 #→ fail
41 $ echo 42 #→ fail
42 $ echo 43 #→ fail
43 $ echo 44 #→ fail
44 $ echo 45 #→ fail
45 $ echo 46 #→ fail
46 $ echo 47 #→ fail
47 $ echo 48 #→ fail
48 $ echo 49 #→ fail
49 $ echo 50 #→ fail
0 # Fail Messages
1
2 ## inline text
3
4 $ echo fail #→ ok
5 $ echo fail #→ --eval echo ok
6
7 ## normal command
8
9 $ echo fail
10 ok
11 $ echo fail
12 ok 1
13 ok 2
14 ok 3
15 $
16
17 ## inline --file
18
19 $ echo fail #→ --file lorem-ipsum.txt
20
21 ## inline --lines
22
23 $ echo fail #→ --lines 9
24
25 ## inline --egrep
26
27 $ echo fail #→ --egrep ^[0-9]+$
28
29 ## inline --perl
30
31 $ echo fail #→ --perl ^[0-9]+$
32
33 ## inline --regex
34
35 $ echo fail #→ --regex ^[0-9]+$
0 --------------------------------------------------------------------------------
1 [FAILED #1, line 5] echo fail
2 @@ -1 +1 @@
3 -ok
4 +fail
5 --------------------------------------------------------------------------------
6 [FAILED #2, line 6] echo fail
7 @@ -1 +1 @@
8 -ok
9 +fail
10 --------------------------------------------------------------------------------
11 [FAILED #3, line 10] echo fail
12 @@ -1 +1 @@
13 -ok
14 +fail
15 --------------------------------------------------------------------------------
16 [FAILED #4, line 12] echo fail
17 @@ -1,3 +1 @@
18 -ok 1
19 -ok 2
20 -ok 3
21 +fail
22 --------------------------------------------------------------------------------
23 [FAILED #5, line 20] echo fail
24 @@ -1,5 +1 @@
25 -Lorem ipsum dolor sit amet, consectetur adipiscing elit.
26 -Proin euismod blandit pharetra.
27 -Vestibulum eu neque eget lorem gravida commodo a cursus massa.
28 -Fusce sit amet lorem sem.
29 -Donec eu quam leo.
30 +fail
31 --------------------------------------------------------------------------------
32 [FAILED #6, line 24] echo fail
33 Expected 9 lines, got 1.
34 --------------------------------------------------------------------------------
35 [FAILED #7, line 28] echo fail
36 egrep '^[0-9]+$' failed in:
37 fail
38 --------------------------------------------------------------------------------
39 [FAILED #8, line 32] echo fail
40 Perl regex '^[0-9]+$' not matched in:
41 fail
42 --------------------------------------------------------------------------------
43 [FAILED #9, line 36] echo fail
44 Perl regex '^[0-9]+$' not matched in:
45 fail
46 --------------------------------------------------------------------------------
47
48 FAIL: 9 of 9 tests failed
0 $ echo 'error: no contents' #→ --egrep
0 $ echo "error: malformed regex" #→ --egrep (
1
2
3 # Some grep errors:
4
5 # $ echo | egrep '(' ; echo $?
6 # egrep: parentheses not balanced
7 # 2
8 # $ echo | egrep '[' ; echo $?
9 # egrep: brackets ([ ]) not balanced
10 # 2
11 # $ echo | egrep '**' ; echo $?
12 # egrep: repetition-operator operand invalid
13 # 2
14 # $ echo | egrep '{' ; echo $?
15 # 1
16 # $
0 # Inline matching method: --egrep
1 # Matches a egrep-style regular expression in the command output
2 #
3 # In fact, it's a real egrep match: eval $command | egrep 'regex'
4 # If egrep matched, we have a successful test. That means that in
5 # a multiline result, even if just a single line matches the regex,
6 # the test is considered OK.
7 #
8 # Test your regexes with egrep at the command line before adding
9 # tests using them.
10
11 # See man re_format in your system
12 # http://www.freebsd.org/cgi/man.cgi?query=re_format&sektion=7
13
14 # Use anchors ^ and $ to match the full output text
15
16 $ echo 'abc123' #→ --egrep ^abc123$
17 $ echo 'abc123' #→ --egrep ^abc.*3$
18 $ echo 'abc123' #→ --egrep ^abc[0-9]+$
19
20 # Omit one or both anchors to make a parcial match
21
22 $ echo 'abc123' #→ --egrep ^abc
23 $ echo 'abc123' #→ --egrep 123$
24 $ echo 'abc123' #→ --egrep [0-9]+$
25 $ echo 'abc123' #→ --egrep bc
26 $ echo 'abc123' #→ --egrep .
27
28 # Blanks are preserved, no escaping or quoting needed
29
30 $ echo 'abc 123' #→ --egrep ^abc [0-9]+$
31
32 # Blank output can also be matched
33
34 $ echo ' ' #→ --egrep ^ $
35 $ echo ' ' #→ --egrep ^ $
36 $ printf '\t\n' #→ --egrep ^ $
37 $ printf '\t\t\t\n' #→ --egrep ^ $
38 $ printf ' \t \t\t \n' #→ --egrep ^ $
39
40 # In some systems, the special sequence \t is expanded to a tab in
41 # egrep regexes. You'll need to test in your system if that's the
42 # case. I recommend using a literal tab to avoid problems.
43
44 $ printf 'may\tfail' #→ --egrep ^may\tfail$
45 $ printf 'may\tfail' #→ --egrep ^may[\t]fail$
46 $ printf 'will\tmatch' #→ --egrep ^will match$
47
48 # Since it's an egrep test, regexes are not multiline.
49 # You can only match a single line.
50 # These tests will fail:
51
52 $ printf 'will\nfail' #→ --egrep will.*fail
53 $ printf 'will\nfail' #→ --egrep will\nfail
54
55 # If one line of a multiline results matches, the test is OK
56
57 $ printf '1\n2\n3\n4\nok\n' #→ --egrep ok
58
59 # As egrep is used for the test and it ignores the line break,
60 # you can match both full (with \n) and partial (without \n).
61
62 $ printf 'ok' #→ --egrep ok
63 $ printf 'ok\n' #→ --egrep ok
64
65 # Syntax: Must be exactly one space before and after --egrep
66
67 $ echo 'fail' #→ --egrep fail with 2 spaces
68 $ echo 'fail' #→ --egrep fail with tab
69
70 # Syntax: The extra space after '--egrep ' is already part of the regex
71
72 $ echo ' ok' #→ --egrep ok
73
74 # Syntax: The space after --egrep is required.
75 # When missing, the '--egrep' is considered a normal text.
76
77 $ echo '--egrep' #→ --egrep
78
79 # Syntax: Make sure we won't catch partial matches.
80
81 $ echo '--egreppal' #→ --egreppal
82
83 # Syntax: To insert a literal text that begins with '--egrep '
84 # just prefix it with --text.
85
86 $ echo '--egrep is cool' #→ --text --egrep is cool
87
88 # Syntax: Empty inline output contents are considered an error
89 # Note: Tested in a separate file: inline-match-egrep-error-1.sh
90 #
91 # $ echo 'no contents' #→ --egrep
0 $ echo 'error: no contents' #→ --eval
0 $ echo 'error: syntax error' #→ --eval $(
0 # Inline matching method: --eval
1 # Matches the text output from an arbitrary shell command
2
3 # Run a simple command
4
5 $ folder=$(pwd)
6 $ echo $folder #→ --eval pwd
7
8 # Read the contents of a variable
9
10 $ var='abc'
11 $ echo abc #→ --eval echo $var
12
13 # Use arithmetic expansion
14
15 $ echo 4 #→ --eval echo $((2+2))
16
17 # Run a subshell
18
19 $ today=$(date +%D)
20 $ echo "Today is $today" #→ --eval echo "Today is $(date +%D)"
21
22 # You can also match lines without the final \n
23
24 $ printf 'ok' #→ --eval printf 'ok'
25
26 # Blanks are preserved
27
28 $ echo ' leading space' #→ --eval echo ' leading space'
29 $ echo ' leading spaces' #→ --eval echo ' leading spaces'
30 $ printf '\tleading tab\n' #→ --eval printf '\tleading tab\n'
31 $ printf '\t\tleading tabs\n' #→ --eval printf '\t\tleading tabs\n'
32 $ echo 'trailing space ' #→ --eval echo 'trailing space '
33 $ echo 'trailing spaces ' #→ --eval echo 'trailing spaces '
34 $ printf 'trailing tab\t\n' #→ --eval printf 'trailing tab\t\n'
35 $ printf 'trailing tabs\t\t\n' #→ --eval printf 'trailing tabs\t\t\n'
36 $ echo ' ' #→ --eval echo ' '
37 $ echo ' ' #→ --eval echo ' '
38 $ printf '\t\n' #→ --eval printf '\t\n'
39 $ printf '\t\t\t\n' #→ --eval printf '\t\t\t\n'
40 $ printf ' \t \t\t \n' #→ --eval printf ' \t \t\t \n'
41
42 # Syntax: Must be exactly one space before and after --eval
43
44 $ echo 'fail' #→ --eval fail with 2 spaces
45 $ echo 'fail' #→ --eval fail with tab
46
47 # Syntax: The space after --eval is required.
48 # When missing, the '--eval' is considered a normal text.
49
50 $ echo '--eval' #→ --eval
51
52 # Syntax: Make sure we won't catch partial matches.
53
54 $ echo '--evaluate' #→ --evaluate
55
56 # Syntax: To insert a literal text that begins with '--eval '
57 # just prefix it with --text.
58
59 $ echo '--eval is evil' #→ --text --eval is evil
60
61 # Syntax: Empty inline output contents are considered an error
62 # Note: Tested in separate files: inline-match-eval-error-?.sh
63 #
64 # $ echo 'no contents' #→ --eval
0 $ echo 'error: no contents' #→ --file
0 $ echo "error: file not found" #→ --file XXnotfoundXX
0 $ echo "error: directory" #→ --file /etc/
0 # Inline matching method: --file
1 # Matches the contents of the informed file
2
3 # Just inform the file path (no quotes, no escapes)
4
5 $ printf '$ echo ok\nok\n' #→ --file ok-1.sh
6
7 # Absolute paths are also supported
8
9 $ echo 'ok' > /tmp/foo.txt
10 $ echo 'ok' #→ --file /tmp/foo.txt
11
12 # Syntax: Must be exactly one space before and after --file
13
14 $ echo 'fail' #→ --file fail-with-2-spaces.txt
15 $ echo 'fail' #→ --file fail-with-tab.txt
16
17 # Syntax: The extra space after '--file ' is already part of the filename
18
19 #$ echo 'fail' #→ --file file-with-leading-space-in-name.txt
20
21 # Syntax: The space after --file is required.
22 # When missing, the '--file' is considered a normal text.
23
24 $ echo '--file' #→ --file
25
26 # Syntax: Make sure we won't catch partial matches.
27
28 $ echo '--filer' #→ --filer
29
30 # Syntax: To insert a literal text that begins with '--file '
31 # just prefix it with --text.
32
33 $ echo '--file is cool' #→ --text --file is cool
34
35 # Syntax: Empty inline output contents are considered an error
36 # Note: Tested in a separate file: inline-match-file-error-1.sh
37 #
38 # $ echo 'no contents' #→ --file
0 $ echo 'error: no contents' #→ --lines
0 $ echo 'error: negative number' #→ --lines -1
0 $ echo 'error: float number' #→ --lines 1.0
0 $ echo 'error: not a number' #→ --lines foo
0 # Inline matching method: --lines
1 # Count the number of lines in the output
2
3 $ a=1 #→ --lines 0
4 $ echo 'ok' #→ --lines 1
5 $ printf '1\n2\n3\n' #→ --lines 3
6
7 # Lines without the final \n count as one full line
8
9 $ printf 'no-nl' #→ --lines 1
10 $ printf '1\n2\nno-nl' #→ --lines 3
11
12 # The error message is a short sentence, not a diff
13 # Example: Expected 99 lines, got 1.
14
15 $ echo 'fail' #→ --lines 99
16 $ echo 'fail' #→ --lines 0
17
18 # Syntax: Must be exactly one space before and after --lines
19
20 $ echo 'fail' #→ --lines fail with 2 spaces
21 $ echo 'fail' #→ --lines fail with tab
22
23 # Syntax: The space after --lines is required.
24 # When missing, the '--lines' is considered a normal text.
25
26 $ echo '--lines' #→ --lines
27
28 # Syntax: Make sure we won't catch partial matches.
29
30 $ echo '--linesout' #→ --linesout
31
32 # Syntax: To insert a literal text that begins with '--lines '
33 # just prefix it with --text.
34
35 $ echo '--lines is cool' #→ --text --lines is cool
36
37
38 # Note: The following are tested in separate files:
39 # inline-match-lines-error-?.sh
40 #
41 # Syntax: Empty inline output contents are considered an error
42 #
43 # $ echo 'no contents' #→ --lines
44 #
45 # Syntax: Must be an integer number
46 #
47 # $ echo 'fail' #→ --lines -1
48 # $ echo 'fail' #→ --lines 1.0
49 # $ echo 'fail' #→ --lines foo
50
0 $ echo 'error: no contents' #→ --perl
0 $ echo "error: malformed regex" #→ --perl (
1
2
3 # Some perl errors:
4
5 # $ echo | perl -0777 -ne 'exit(!/(/)' ; echo $?
6 # Unmatched ( in regex; marked by <-- HERE in m/( <-- HERE / at -e line 1.
7 # 255
8 # $ echo | perl -0777 -ne 'exit(!/[/)' ; echo $?
9 # Unmatched [ in regex; marked by <-- HERE in m/[ <-- HERE / at -e line 1.
10 # 255
11 # $ echo | perl -0777 -ne 'exit(!/**/)' ; echo $?
12 # Quantifier follows nothing in regex; marked by <-- HERE in m/* <-- HERE */ at -e line 1.
13 # 255
14 # $
0 # Inline matching method: --perl
1 # Matches a Perl-style regular expression in the command output
2 # You can also use the friendlier alias: --regex
3 #
4 # In fact, it's a real Perl match: perl -0777 -ne "exit(!m'regex')"
5 # If Perl matched, we have a successful test.
6 # All the test output lines are matched as a single string.
7 # No modifiers are used by default, inform yours if needed: (?ims)
8 # You don't need to escape the ' delimiter, the script will do it for you.
9 # Just write your regex not worrying about delimiters.
10
11
12 # Use anchors ^ and $ to match the full output text
13
14 $ echo 'abc123' #→ --perl ^abc123$
15 $ echo 'abc123' #→ --perl ^abc.*3$
16 $ echo 'abc123' #→ --perl ^abc[0-9]+$
17
18 # Omit one or both anchors to make a parcial match
19
20 $ echo 'abc123' #→ --perl ^abc
21 $ echo 'abc123' #→ --perl 123$
22 $ echo 'abc123' #→ --perl [0-9]+$
23 $ echo 'abc123' #→ --perl bc
24 $ echo 'abc123' #→ --perl .
25
26 # Blanks are preserved, no escaping or quoting needed
27
28 $ echo 'abc 123' #→ --perl ^abc [0-9]+$
29
30 # Blank output can also be matched
31
32 $ echo ' ' #→ --perl ^ $
33 $ echo ' ' #→ --perl ^ $
34 $ printf '\t\n' #→ --perl ^ $
35 $ printf '\t\t\t\n' #→ --perl ^ $
36 $ printf ' \t \t\t \n' #→ --perl ^ $
37
38 # You don't need to escape any delimiters, escapes are handled by the script
39
40 $ echo '01/01/2013' #→ --perl ^../../....$
41 $ echo "won't fail" #→ --perl ^won't \w+$
42
43 # To match a tab, you can use \t or a literal tab
44
45 $ printf 'will\tmatch' #→ --perl ^will\tmatch$
46 $ printf 'will\tmatch' #→ --perl ^will[\t]match$
47 $ printf 'will\tmatch' #→ --perl ^will match$
48
49 # You need to inform the (?i) modifier to match ignoring case
50
51 $ printf 'will\nfail' #→ --perl ^WILL
52 $ printf 'will\nmatch' #→ --perl (?i)^WILL
53
54 # You need to inform the (?s) modifier for the dot to match \n
55
56 $ printf 'will\nfail' #→ --perl ^will.fail$
57 $ printf 'will\nmatch' #→ --perl (?s)^will.match$
58
59 # You need to inform the (?m) modifier for ^ and $ to match inner lines
60
61 $ printf 'will\nfail' #→ --perl ^fail
62 $ printf 'will\nmatch' #→ --perl (?m)^match
63
64 # Perl ignores the last \n, in both the text and the regex
65
66 $ printf 'ok' #→ --perl ^ok$
67 $ printf 'ok\n' #→ --perl ^ok$
68 $ printf '1\n2\n3\n' #→ --perl ^1\n2\n3\n$
69 $ printf '1\n2\n3\n' #→ --perl ^1\n2\n3$
70
71 # Syntax: Must be exactly one space before and after --perl
72
73 $ echo 'fail' #→ --perl fail with 2 spaces
74 $ echo 'fail' #→ --perl fail with tab
75
76 # Syntax: The extra space after '--perl ' is already part of the regex
77
78 $ echo ' ok' #→ --perl ok
79
80 # Syntax: The space after --perl is required.
81 # When missing, the '--perl' is considered a normal text.
82
83 $ echo '--perl' #→ --perl
84
85 # Syntax: Make sure we won't catch partial matches.
86
87 $ echo '--perlism' #→ --perlism
88
89 # Syntax: To insert a literal text that begins with '--perl '
90 # just prefix it with --text.
91
92 $ echo '--perl is cool' #→ --text --perl is cool
93
94 # Syntax: Empty inline output contents are considered an error
95 # Note: Tested in a separate file: inline-match-perl-error-1.sh
96 #
97 # $ echo 'no contents' #→ --perl
0 # Inline matching method: --text
1 # Matches a literal text
2
3 # This is the default method, the --text part can be omitted.
4
5 $ echo 'abc' #→ --text abc
6 $ echo 'abc' #→ abc
7
8 # Special characters as \t and \n are not expanded.
9
10 $ printf '%s\n' '\t' #→ \t
11 $ printf '%s\n' '\n' #→ \n
12
13 # Variables and commands are not parsed (see #→ --eval for that).
14
15 $ echo '$PWD' #→ $PWD
16 $ echo '$(date)' #→ $(date)
17
18 # It's a literal text, with no special characters.
19
20 $ echo '$' #→ $
21 $ echo '>' #→ >
22 $ echo '?' #→ ?
23 $ echo '!' #→ !
24 $ echo '*' #→ *
25 $ echo '[' #→ [
26 $ echo '(' #→ (
27
28 # For commands that return an empty line, just leave it empty
29
30 $ echo #→
31
32 # But don't forget the blank space after the →, because in this
33 # case the #→ marker will be considered a plain comment and ignored
34
35 $ echo "not inline output" #→
36 not inline output
37 $
38
39 # Blanks are preserved
40
41 $ echo '123456789' #→ 123456789
42 $ echo '1 3 7 9' #→ 1 3 7 9
43 $ echo ' 5 ' #→ 5
44 $ echo ' leading space' #→ leading space
45 $ echo ' leading spaces' #→ leading spaces
46 $ printf '\tleading tab\n' #→ leading tab
47 $ printf '\t\tleading tabs\n' #→ leading tabs
48 $ echo 'trailing space ' #→ trailing space
49 $ echo 'trailing spaces ' #→ trailing spaces
50 $ printf 'trailing tab\t\n' #→ trailing tab
51 $ printf 'trailing tabs\t\t\n' #→ trailing tabs
52 $ echo ' ' #→
53 $ echo ' ' #→
54 $ printf '\t\n' #→
55 $ printf '\t\t\t\n' #→
56 $ printf ' \t \t\t \n' #→
57
58 # As seen in all these examples, the final \n is implied.
59 # You can't match lines with no \n.
60
61 $ printf 'ok\n' #→ ok
62 $ printf 'fail' #→ fail
63
64 # An easy workaround is to add an empty 'echo' at the end:
65
66 $ printf 'ok'; echo #→ ok
67
68 # Syntax: Must be exactly one space before and after --text
69
70 $ echo 'fail' #→ --text fail with 2 spaces
71 $ echo 'fail' #→ --text fail with tab
72
73 # Syntax: The extra space after '--text ' is already part of the output
74
75 $ echo ' ok' #→ --text ok
76
77 # Syntax: The space after --text is required.
78 # When missing, the '--text' is considered a normal text.
79
80 $ echo '--text' #→ --text
81
82 # Syntax: Make sure we won't catch partial matches.
83
84 $ echo '--textual' #→ --textual
85
86 # Syntax: To insert a literal text that begins with '--text '
87 # just prefix it with another --text.
88
89 $ echo '--text is cool' #→ --text --text is cool
0 # The blank space before the #→ marker matters?
1
2 $ echo 'one space' #→ one space
3 $ echo 'one tab' #→ one tab
4 $ echo 'multi spaces' #→ multi spaces
5 $ echo 'multi tabs' #→ multi tabs
6 $ echo 'mixed' #→ mixed
7
8 # Blank lines and comments in the middle.
9 # No need to 'close' previous command.
10
11 # Leading and trailing blank space are preserved?
12
13 $ echo ' leading space' #→ leading space
14 $ echo ' leading spaces' #→ leading spaces
15 $ printf '\tleading tab\n' #→ leading tab
16 $ printf '\t\tleading tabs\n' #→ leading tabs
17 $ echo 'trailing space ' #→ trailing space
18 $ echo 'trailing spaces ' #→ trailing spaces
19 $ printf 'trailing tab\t\n' #→ trailing tab
20 $ printf 'trailing tabs\t\t\n' #→ trailing tabs
21
22 # Blank output
23
24 $ echo ' ' #→
25 $ echo ' ' #→
26 $ printf '\t\n' #→
27 $ printf '\t\t\t\n' #→
28 $ printf ' \t \t\t \n' #→
29
30 # Inline results have precedence over normal results
31 $ echo "both inline and normal output" #→ both inline and normal output
32 Inline wins.
33 The normal output is just ignored.
34 $
0 Lorem ipsum dolor sit amet, consectetur adipiscing elit.
1 Proin euismod blandit pharetra.
2 Vestibulum eu neque eget lorem gravida commodo a cursus massa.
3 Fusce sit amet lorem sem.
4 Donec eu quam leo.
0 $ echo 1; echo 2; echo 3; echo 4; echo 5
1 1
2 2
3 3
4 4
5 5
6 $ (echo 1; echo 2; echo 3; echo 4; echo 5) | sed -n 3p
7 3
8 $ (echo 1; echo 2; echo 3; echo 4; echo 5) | sed -n 3p #→ 3
0 # All results assume a trailing newline (\n) at the last line.
1 # Outputs with no \n at the end cannot be tested directly.
2
3 $ printf 'ok\n'
4 ok
5 $ printf 'fail'
6 fail
7 $ printf 'ok\nok\nfail'
8 ok
9 ok
10 fail
11 $
12
13 # The same applies for inline output.
14
15 $ printf 'ok\n' #→ ok
16 $ printf 'fail' #→ fail
17
18 # An easy workaround is to add an empty 'echo' at the end:
19
20 $ printf 'ok'; echo #→ ok
21
22 # Another workaround is to use --regex
23
24 $ printf 'ok' #→ --regex ^ok$
0 $ printf '%s\n' 'a file with no \n at the last line'
1 a file with no \n at the last line
0 $ printf '%s\n' 'another file with no \n at the last line'
1 another file with no \n at the last line
2 $
0 $ printf '%s\n' 'oneliner, no \n' #→ oneliner, no \n
0 # A file with comments, but no test :(
1
2 # It raises an error when processed, because the user must
3 # take action in this situation:
4 #
5 # 1) fix the file, adding the tests
6 # or
7 # 2) stop sending this file to the tester
0 $ echo ok
1 ok
0 $ echo 1 #→ 1
1 $ echo 2 #→ 2
2 $ echo 3 #→ 3
3 $ echo 4 #→ 4
4 $ echo 5 #→ 5
5 $ echo 6 #→ 6
6 $ echo 7 #→ 7
7 $ echo 8 #→ 8
8 $ echo 9 #→ 9
9 $ echo 10 #→ 10
0 $ echo 1 #→ 1
1 $ echo 2 #→ 2
2 $ echo 3 #→ 3
3 $ echo 4 #→ 4
4 $ echo 5 #→ 5
5 $ echo 6 #→ 6
6 $ echo 7 #→ 7
7 $ echo 8 #→ 8
8 $ echo 9 #→ 9
9 $ echo 10 #→ 10
10 $ echo 11 #→ 11
11 $ echo 12 #→ 12
12 $ echo 13 #→ 13
13 $ echo 14 #→ 14
14 $ echo 15 #→ 15
15 $ echo 16 #→ 16
16 $ echo 17 #→ 17
17 $ echo 18 #→ 18
18 $ echo 19 #→ 19
19 $ echo 20 #→ 20
20 $ echo 21 #→ 21
21 $ echo 22 #→ 22
22 $ echo 23 #→ 23
23 $ echo 24 #→ 24
24 $ echo 25 #→ 25
25 $ echo 26 #→ 26
26 $ echo 27 #→ 27
27 $ echo 28 #→ 28
28 $ echo 29 #→ 29
29 $ echo 30 #→ 30
30 $ echo 31 #→ 31
31 $ echo 32 #→ 32
32 $ echo 33 #→ 33
33 $ echo 34 #→ 34
34 $ echo 35 #→ 35
35 $ echo 36 #→ 36
36 $ echo 37 #→ 37
37 $ echo 38 #→ 38
38 $ echo 39 #→ 39
39 $ echo 40 #→ 40
40 $ echo 41 #→ 41
41 $ echo 42 #→ 42
42 $ echo 43 #→ 43
43 $ echo 44 #→ 44
44 $ echo 45 #→ 45
45 $ echo 46 #→ 46
46 $ echo 47 #→ 47
47 $ echo 48 #→ 48
48 $ echo 49 #→ 49
49 $ echo 50 #→ 50
50 $ echo 51 #→ 51
51 $ echo 52 #→ 52
52 $ echo 53 #→ 53
53 $ echo 54 #→ 54
54 $ echo 55 #→ 55
55 $ echo 56 #→ 56
56 $ echo 57 #→ 57
57 $ echo 58 #→ 58
58 $ echo 59 #→ 59
59 $ echo 60 #→ 60
60 $ echo 61 #→ 61
61 $ echo 62 #→ 62
62 $ echo 63 #→ 63
63 $ echo 64 #→ 64
64 $ echo 65 #→ 65
65 $ echo 66 #→ 66
66 $ echo 67 #→ 67
67 $ echo 68 #→ 68
68 $ echo 69 #→ 69
69 $ echo 70 #→ 70
70 $ echo 71 #→ 71
71 $ echo 72 #→ 72
72 $ echo 73 #→ 73
73 $ echo 74 #→ 74
74 $ echo 75 #→ 75
75 $ echo 76 #→ 76
76 $ echo 77 #→ 77
77 $ echo 78 #→ 78
78 $ echo 79 #→ 79
79 $ echo 80 #→ 80
80 $ echo 81 #→ 81
81 $ echo 82 #→ 82
82 $ echo 83 #→ 83
83 $ echo 84 #→ 84
84 $ echo 85 #→ 85
85 $ echo 86 #→ 86
86 $ echo 87 #→ 87
87 $ echo 88 #→ 88
88 $ echo 89 #→ 89
89 $ echo 90 #→ 90
90 $ echo 91 #→ 91
91 $ echo 92 #→ 92
92 $ echo 93 #→ 93
93 $ echo 94 #→ 94
94 $ echo 95 #→ 95
95 $ echo 96 #→ 96
96 $ echo 97 #→ 97
97 $ echo 98 #→ 98
98 $ echo 99 #→ 99
99 $ echo 100 #→ 100
0 $ echo ok
1 ok
2 $ echo ok #→ ok
0 $ echo 1 #→ 1
1 $ echo 2 #→ 2
2 $ echo 3 #→ 3
3 $ echo 4 #→ 4
4 $ echo 5 #→ 5
5 $ echo 6 #→ 6
6 $ echo 7 #→ 7
7 $ echo 8 #→ 8
8 $ echo 9 #→ 9
9 $ echo 10 #→ 10
10 $ echo 11 #→ 11
11 $ echo 12 #→ 12
12 $ echo 13 #→ 13
13 $ echo 14 #→ 14
14 $ echo 15 #→ 15
15 $ echo 16 #→ 16
16 $ echo 17 #→ 17
17 $ echo 18 #→ 18
18 $ echo 19 #→ 19
19 $ echo 20 #→ 20
20 $ echo 21 #→ 21
21 $ echo 22 #→ 22
22 $ echo 23 #→ 23
23 $ echo 24 #→ 24
24 $ echo 25 #→ 25
25 $ echo 26 #→ 26
26 $ echo 27 #→ 27
27 $ echo 28 #→ 28
28 $ echo 29 #→ 29
29 $ echo 30 #→ 30
30 $ echo 31 #→ 31
31 $ echo 32 #→ 32
32 $ echo 33 #→ 33
33 $ echo 34 #→ 34
34 $ echo 35 #→ 35
35 $ echo 36 #→ 36
36 $ echo 37 #→ 37
37 $ echo 38 #→ 38
38 $ echo 39 #→ 39
39 $ echo 40 #→ 40
40 $ echo 41 #→ 41
41 $ echo 42 #→ 42
42 $ echo 43 #→ 43
43 $ echo 44 #→ 44
44 $ echo 45 #→ 45
45 $ echo 46 #→ 46
46 $ echo 47 #→ 47
47 $ echo 48 #→ 48
48 $ echo 49 #→ 49
49 $ echo 50 #→ 50
0 # Configurable diff options with --diff-options
1
2 $ echo " diff -w to ignore spaces "
3 diff -w to ignore spaces
4 $ echo " diff -w now inline " #→ diff -w now inline
0 # Configurable prefix with --inline-prefix
1
2 $ echo "1 space" #==> 1 space
3 $ echo "8 spaces" #==> 8 spaces
4 $ echo "2 tabs" #==> 2 tabs
0 /// Gotcha: glob chars as --prefix: ? * # ## #* *# #*# % %% %* *% %*%
1
2 /// Inline output
3
4 ?$ echo 'prefix ?' #→ prefix ?
5 *$ echo 'prefix *' #→ prefix *
6 #$ echo 'prefix #' #→ prefix #
7 %$ echo 'prefix %' #→ prefix %
8 ##$ echo 'prefix ##' #→ prefix ##
9 %%$ echo 'prefix %%' #→ prefix %%
10 #*$ echo 'prefix #*' #→ prefix #*
11 *#$ echo 'prefix *#' #→ prefix *#
12 %*$ echo 'prefix %*' #→ prefix %*
13 *%$ echo 'prefix *%' #→ prefix *%
14
15 /// Normal output
16
17 ?$ echo 'prefix ?'
18 ?prefix ?
19 ?$
20
21 *$ echo 'prefix *'
22 *prefix *
23 *$
24
25 #$ echo 'prefix #'
26 #prefix #
27 #$
28
29 %$ echo 'prefix %'
30 %prefix %
31 %$
32
33 ##$ echo 'prefix ##'
34 ##prefix ##
35 ##$
36
37 %%$ echo 'prefix %%'
38 %%prefix %%
39 %%$
40
41 #*$ echo 'prefix #*'
42 #*prefix #*
43 #*$
44
45 *#$ echo 'prefix *#'
46 *#prefix *#
47 *#$
48
49 %*$ echo 'prefix %*'
50 %*prefix %*
51 %*$
52
53 *%$ echo 'prefix *%'
54 *%prefix *%
55 *%$
0 # Test file for the --prefix option
1 # Command blocks here are prefixed by a tab
2 # Run with --prefix '\t' or --prefix tab
3
4 $ echo "1" #→ 1
5 $ echo "2"
6 2
7
8 # Any non-prefixed line closes the previous command block.
9 # The empty $ line is not needed.
10
11 # All other non-indented text is just ignored:
12
13 $ echo "ignored" # not indented
14 $ echo "ignored" #→ not indented
15
16 # Lines with the wrong indentation are also ignored
17
18 $ echo "ignored" # 2 tabs
19 $ echo "ignored" # 8 spaces
20 $ echo "ignored" # 4 spaces
21
22 # Multiple blocks supported in a single file
23
24 $ echo "3"
25 3
26
27 # What about prefixed blocks with no commands?
28
29 Prefixed line with no prompt: ignored.
30 But wait, here comes a command:
31 $ echo "4"
32 4
33 $
34 Last command closed by the empty prompt.
35 $ echo "5" #→ 5
36 Last command is auto-closed (inline output).
37
38 # Blank lines in the output are supported
39
40 $ echo; echo "6"; echo; echo "7"
41
42 6
43
44 7
45
46 # Nice.
0 # Test file for the --prefix option
1 # Command blocks here are prefixed by 4 spaces (Markdown-style)
2 # Run with --prefix ' ' or --prefix 4
3
4 $ echo "1" #→ 1
5 $ echo "2"
6 2
7
8 # Any non-prefixed line closes the previous command block.
9 # The empty $ line is not needed.
10
11 # All other non-indented text is just ignored:
12
13 $ echo "ignored" # not indented
14 $ echo "ignored" #→ not indented
15
16 # Lines with the wrong indentation are also ignored
17
18 $ echo "ignored" # 3 spaces
19 $ echo "ignored" # 5 spaces
20
21 # Multiple blocks supported in a single file
22
23 $ echo "3"
24 3
25
26 # What about prefixed blocks with no commands?
27
28 Prefixed line with no prompt: ignored.
29 But wait, here comes a command:
30 $ echo "4"
31 4
32 $
33 Last command closed by the empty prompt.
34 $ echo "5" #→ 5
35 Last command is auto-closed (inline output).
36
37 # Blank lines in the output are supported
38
39 $ echo; echo "6"; echo; echo "7"
40
41 6
42
43 7
44
45 # Nice.
0 /// Gotcha: glob chars as --prompt: ? * # %
1
2 /// Note: These tests are separated from the two chars globs
3 /// to avoid partial matches with wrong output.
4
5 /// Inline output (one char)
6
7 ?echo 'prompt ?' #→ prompt ?
8 *echo 'prompt *' #→ prompt *
9 #echo 'prompt #' #→ prompt #
10 %echo 'prompt %' #→ prompt %
11
12 /// Normal output (one char)
13
14 ?echo 'prompt ?'
15 prompt ?
16 ?
17
18 *echo 'prompt *'
19 prompt *
20 *
21
22 #echo 'prompt #'
23 prompt #
24 #
25
26 %echo 'prompt %'
27 prompt %
28 %
0 /// Gotcha: glob chars as --prompt: ? * # % ## %% #* *# %* *%
1
2 /// Inline output
3
4 ?echo 'prompt ?' #→ prompt ?
5 *echo 'prompt *' #→ prompt *
6 #echo 'prompt #' #→ prompt #
7 %echo 'prompt %' #→ prompt %
8 ##echo 'prompt ##' #→ prompt ##
9 %%echo 'prompt %%' #→ prompt %%
10 #*echo 'prompt #*' #→ prompt #*
11 *#echo 'prompt *#' #→ prompt *#
12 %*echo 'prompt %*' #→ prompt %*
13 *%echo 'prompt *%' #→ prompt *%
14
15
16 /// Normal output
17
18 ?echo 'prompt ?'
19 prompt ?
20 ?
21
22 *echo 'prompt *'
23 prompt *
24 *
25
26 #echo 'prompt #'
27 prompt #
28 #
29
30 %echo 'prompt %'
31 prompt %
32 %
33
34 ##echo 'prompt ##'
35 prompt ##
36 ##
37
38 %%echo 'prompt %%'
39 prompt %%
40 %%
41
42 #*echo 'prompt #*'
43 prompt #*
44 #*
45
46 *#echo 'prompt *#'
47 prompt *#
48 *#
49
50 %*echo 'prompt %*'
51 prompt %*
52 %*
53
54 *%echo 'prompt *%'
55 prompt *%
56 *%
0 /// Gotcha: glob chars (plus space) as --prompt: ? * # ## #* *# % %% %* *%
1
2 /// Inline output
3
4 ? echo 'prompt ? ' #→ prompt ?
5 * echo 'prompt * ' #→ prompt *
6 # echo 'prompt # ' #→ prompt #
7 % echo 'prompt % ' #→ prompt %
8 ## echo 'prompt ## ' #→ prompt ##
9 %% echo 'prompt %% ' #→ prompt %%
10 #* echo 'prompt #* ' #→ prompt #*
11 *# echo 'prompt *# ' #→ prompt *#
12 %* echo 'prompt %* ' #→ prompt %*
13 *% echo 'prompt *% ' #→ prompt *%
14
15
16 /// Normal output
17
18 ? echo 'prompt ? '
19 prompt ?
20 ?
21
22 * echo 'prompt * '
23 prompt *
24 *
25
26 # echo 'prompt # '
27 prompt #
28 #
29
30 % echo 'prompt % '
31 prompt %
32 %
33
34 ## echo 'prompt ## '
35 prompt ##
36 ##
37
38 %% echo 'prompt %% '
39 prompt %%
40 %%
41
42 #* echo 'prompt #* '
43 prompt #*
44 #*
45
46 *# echo 'prompt *# '
47 prompt *#
48 *#
49
50 %* echo 'prompt %* '
51 prompt %*
52 %*
53
54 *% echo 'prompt *% '
55 prompt *%
56 *%
0 # Configurable prompt with --prompt
1
2 ♥ echo "1" #→ 1
3 ♥ echo "2"
4 2
5
6
7 # Series closed with empty prompt
8
9 ♥ echo "3"
10 3
11
12
13 # Series closed with empty prompt (with no trailing space)
0 # Configurable prompt with --prompt
1
2 prompt$ echo "1" #→ 1
3 prompt$ echo "2"
4 2
5 prompt$
6
7 # Series closed with empty prompt
8
9 prompt$ echo "3"
10 3
11 prompt$
12
13 # Series closed with empty prompt (with no trailing space)
0 # ASCII
1
2 $ echo '0123456789' #→ 0123456789
3 $ echo 'ABCDEFGHIJKLMNOPQRSTUVWXYZ' #→ ABCDEFGHIJKLMNOPQRSTUVWXYZ
4 $ echo 'abcdefghijklmnopqrstuvwxyz' #→ abcdefghijklmnopqrstuvwxyz
5 $ echo ' ' #→
6 $ echo '!' #→ !
7 $ echo '"' #→ "
8 $ echo '#' #→ #
9 $ echo '$' #→ $
10 $ echo '%' #→ %
11 $ echo '&' #→ &
12 $ echo "'" #→ '
13 $ echo '(' #→ (
14 $ echo ')' #→ )
15 $ echo '*' #→ *
16 $ echo '+' #→ +
17 $ echo ',' #→ ,
18 $ echo '-' #→ -
19 $ echo '.' #→ .
20 $ echo '/' #→ /
21 $ echo ':' #→ :
22 $ echo ';' #→ ;
23 $ echo '<' #→ <
24 $ echo '=' #→ =
25 $ echo '>' #→ >
26 $ echo '?' #→ ?
27 $ echo '@' #→ @
28 $ echo '[' #→ [
29 $ echo '\' #→ \
30 $ echo ']' #→ ]
31 $ echo '^' #→ ^
32 $ echo '_' #→ _
33 $ echo '`' #→ `
34 $ echo '{' #→ {
35 $ echo '|' #→ |
36 $ echo '}' #→ }
37 $ echo '~' #→ ~
38
39 # !ASCII
40
41 $ echo '¡' #→ ¡
42 $ echo '¢' #→ ¢
43 $ echo '£' #→ £
44 $ echo '¤' #→ ¤
45 $ echo '¥' #→ ¥
46 $ echo '¦' #→ ¦
47 $ echo '§' #→ §
48 $ echo '¨' #→ ¨
49 $ echo '©' #→ ©
50 $ echo 'ª' #→ ª
51 $ echo '«' #→ «
52 $ echo '¬' #→ ¬
53 $ echo '­' #→ ­
54 $ echo '®' #→ ®
55 $ echo '°' #→ °
56 $ echo '±' #→ ±
57 $ echo '²' #→ ²
58 $ echo '³' #→ ³
59 $ echo '´' #→ ´
60 $ echo 'µ' #→ µ
61 $ echo '¶' #→ ¶
62 $ echo '·' #→ ·
63 $ echo '¹' #→ ¹
64 $ echo 'º' #→ º
65 $ echo '»' #→ »
66 $ echo '¼' #→ ¼
67 $ echo '½' #→ ½
68 $ echo '¾' #→ ¾
69 $ echo '¿' #→ ¿
70 $ echo 'À' #→ À
71 $ echo 'Á' #→ Á
72 $ echo 'Â' #→ Â
73 $ echo 'Ã' #→ Ã
74 $ echo 'Ä' #→ Ä
75 $ echo 'Å' #→ Å
76 $ echo 'Æ' #→ Æ
77 $ echo 'Ç' #→ Ç
78 $ echo 'È' #→ È
79 $ echo 'É' #→ É
80 $ echo 'Ê' #→ Ê
81 $ echo 'Ë' #→ Ë
82 $ echo 'Ì' #→ Ì
83 $ echo 'Í' #→ Í
84 $ echo 'Î' #→ Î
85 $ echo 'Ï' #→ Ï
86 $ echo 'Ð' #→ Ð
87 $ echo 'Ñ' #→ Ñ
88 $ echo 'Ò' #→ Ò
89 $ echo 'Ó' #→ Ó
90 $ echo 'Ô' #→ Ô
91 $ echo 'Õ' #→ Õ
92 $ echo 'Ö' #→ Ö
93 $ echo '×' #→ ×
94 $ echo 'Ø' #→ Ø
95 $ echo 'Ù' #→ Ù
96 $ echo 'Ú' #→ Ú
97 $ echo 'Û' #→ Û
98 $ echo 'Ü' #→ Ü
99 $ echo 'Ý' #→ Ý
100 $ echo 'ß' #→ ß
101 $ echo 'à' #→ à
102 $ echo 'á' #→ á
103 $ echo 'â' #→ â
104 $ echo 'ã' #→ ã
105 $ echo 'ä' #→ ä
106 $ echo 'å' #→ å
107 $ echo 'æ' #→ æ
108 $ echo 'ç' #→ ç
109 $ echo 'è' #→ è
110 $ echo 'é' #→ é
111 $ echo 'ê' #→ ê
112 $ echo 'ë' #→ ë
113 $ echo 'ì' #→ ì
114 $ echo 'í' #→ í
115 $ echo 'î' #→ î
116 $ echo 'ï' #→ ï
117 $ echo 'ð' #→ ð
118 $ echo 'ñ' #→ ñ
119 $ echo 'ò' #→ ò
120 $ echo 'ó' #→ ó
121 $ echo 'ô' #→ ô
122 $ echo 'õ' #→ õ
123 $ echo 'ö' #→ ö
124 $ echo '÷' #→ ÷
125 $ echo 'ø' #→ ø
126 $ echo 'ù' #→ ù
127 $ echo 'ú' #→ ú
128 $ echo 'û' #→ û
129 $ echo 'ü' #→ ü
130 $ echo 'ý' #→ ý
131 $ echo 'ÿ' #→ ÿ
132 $ echo 'Œ' #→ Œ
133 $ echo 'œ' #→ œ
134 $ echo 'Ÿ' #→ Ÿ
135 $ echo 'ƒ' #→ ƒ
136 $ echo 'Α' #→ Α
137 $ echo 'Β' #→ Β
138 $ echo 'Π' #→ Π
139 $ echo 'Σ' #→ Σ
140 $ echo 'Φ' #→ Φ
141 $ echo 'Ω' #→ Ω
142 $ echo 'α' #→ α
143 $ echo 'β' #→ β
144 $ echo 'η' #→ η
145 $ echo 'μ' #→ μ
146 $ echo 'π' #→ π
147 $ echo 'ω' #→ ω
148 $ echo 'ϖ' #→ ϖ
149 $ echo '–' #→ –
150 $ echo '—' #→ —
151 $ echo '‘' #→ ‘
152 $ echo '’' #→ ’
153 $ echo '‚' #→ ‚
154 $ echo '“' #→ “
155 $ echo '”' #→ ”
156 $ echo '„' #→ „
157 $ echo '†' #→ †
158 $ echo '‡' #→ ‡
159 $ echo '•' #→ •
160 $ echo '…' #→ …
161 $ echo '‰' #→ ‰
162 $ echo '′' #→ ′
163 $ echo '″' #→ ″
164 $ echo '‹' #→ ‹
165 $ echo '›' #→ ›
166 $ echo '‾' #→ ‾
167 $ echo '⁄' #→ ⁄
168 $ echo '€' #→ €
169 $ echo '™' #→ ™
170 $ echo '←' #→ ←
171 $ echo '↑' #→ ↑
172 $ echo '→' #→ →
173 $ echo '↓' #→ ↓
174 $ echo '↔' #→ ↔
175 $ echo '↵' #→ ↵
176 $ echo '⇐' #→ ⇐
177 $ echo '⇑' #→ ⇑
178 $ echo '⇒' #→ ⇒
179 $ echo '⇓' #→ ⇓
180 $ echo '⇔' #→ ⇔
181 $ echo '∈' #→ ∈
182 $ echo '∉' #→ ∉
183 $ echo '∋' #→ ∋
184 $ echo '∏' #→ ∏
185 $ echo '∑' #→ ∑
186 $ echo '−' #→ −
187 $ echo '∗' #→ ∗
188 $ echo '√' #→ √
189 $ echo '∝' #→ ∝
190 $ echo '∞' #→ ∞
191 $ echo '∫' #→ ∫
192 $ echo '∴' #→ ∴
193 $ echo '∼' #→ ∼
194 $ echo '≅' #→ ≅
195 $ echo '≈' #→ ≈
196 $ echo '≠' #→ ≠
197 $ echo '≡' #→ ≡
198 $ echo '≤' #→ ≤
199 $ echo '≥' #→ ≥
200 $ echo '⊂' #→ ⊂
201 $ echo '⊃' #→ ⊃
202 $ echo '⊄' #→ ⊄
203 $ echo '⊆' #→ ⊆
204 $ echo '⊇' #→ ⊇
205 $ echo '◊' #→ ◊
206 $ echo '♠' #→ ♠
207 $ echo '♣' #→ ♣
208 $ echo '♥' #→ ♥
209 $ echo '♦' #→ ♦
210 $ echo '★' #→ ★
0 # Output from both STDOUT and STDERR are catched by the tester.
1
2 ### STDOUT
3
4 # Showing STOUT
5
6 $ echo "stdout"
7 stdout
8 $ echo "stdout" 2> /dev/null
9 stdout
10 $
11
12 # Redirecting STDOUT to STDERR
13
14 $ echo "stderr" 1>&2
15 stderr
16 $
17
18 # Closing STDOUT
19
20 $ echo "stdout" > /dev/null
21 $ echo "stdout" 2> /dev/null 1>&2
22 $
23
24 ### STDERR
25
26 # Showing STDERR
27
28 $ cp XXnotfoundXX foo
29 cp: XXnotfoundXX: No such file or directory
30 $ cp XXnotfoundXX foo > /dev/null
31 cp: XXnotfoundXX: No such file or directory
32 $
33
34 # Redirecting STDERR to STDOUT
35
36 $ cp XXnotfoundXX foo 2>&1
37 cp: XXnotfoundXX: No such file or directory
38 $
39
40 # Closing STDERR
41
42 $ cp XXnotfoundXX foo 2> /dev/null
43 $ cp XXnotfoundXX foo > /dev/null 2>&1
44 $
0 $ echo "a file with CRLF line ending"
1 a file with CRLF line ending
2 $
3 $ echo "inline output" #→ inline output
4 $ echo "inline regex" #→ --regex ^inline regex$
0 # Test suite for clitest
1
2 This is the test file for the `clitest` program. Yes, the program can test itself!
3
4 This file runs all the files inside the `dev/test` folder and checks the results. The command line options are also tested.
5
6 Usage: ./clitest dev/test.md
7
8
9 ## Preparing
10
11 Make sure we're on the same folder as `clitest`, since all the file paths here are relative, not absolute.
12
13 ```
14 $ test -f ./clitest; echo $?
15 0
16 $ test -d ./dev/test/; echo $?
17 0
18 $
19 ```
20
21 Set a default terminal width of 80 columns. It's used by separator lines.
22
23 ```
24 $ COLUMNS=80
25 $ export COLUMNS
26 $
27 ```
28
29 Ok. Now the real tests begins.
30
31 ## Variables are persistent between tests?
32
33 ```
34 $ echo $COLUMNS
35 80
36 $ not_exported=1
37 $ echo $not_exported
38 1
39 $ echo $not_exported #→ 1
40 $ echo $not_exported #→ --regex ^1$
41 ```
42
43 ## Check the temporary dir creation
44
45 ```
46 $ TMPDIR___SAVE="$TMPDIR"
47 $ TMPDIR=/XXnotfoundXX
48 $ export TMPDIR
49 $ ./clitest dev/test/ok-1.sh 2>&1 | sed 's/clitest\.[0-9]*$/clitest.NNN/'
50 mkdir: /XXnotfoundXX: No such file or directory
51 clitest: Error: cannot create temporary dir: /XXnotfoundXX/clitest.NNN
52 $ TMPDIR="$TMPDIR___SAVE"
53 $
54 ```
55
56 ## I/O, file reading (message and exit code)
57
58 Missing input file
59
60 ```
61 $ ./clitest; echo $?
62 clitest: Error: no test file informed (try --help)
63 2
64 $ ./clitest --
65 clitest: Error: no test file informed (try --help)
66 $ ./clitest --list
67 clitest: Error: no test file informed (try --help)
68 $
69 ```
70
71 File not found
72
73 ```
74 $ ./clitest XXnotfoundXX.sh; echo $?
75 clitest: Error: cannot read input file: XXnotfoundXX.sh
76 2
77 $ ./clitest .
78 clitest: Error: cannot read input file: .
79 $ ./clitest ./
80 clitest: Error: cannot read input file: ./
81 $ ./clitest /etc
82 clitest: Error: cannot read input file: /etc
83 $
84 ```
85
86 ## No test found (message and exit code)
87
88 ```
89 $ ./clitest dev/test/no-test-found.sh; echo $?
90 clitest: Error: no test found in input file: dev/test/no-test-found.sh
91 2
92 $ ./clitest dev/test/empty-file.sh
93 clitest: Error: no test found in input file: dev/test/empty-file.sh
94 $ ./clitest dev/test/empty-prompt-file.sh
95 clitest: Error: no test found in input file: dev/test/empty-prompt-file.sh
96 $ ./clitest dev/test/empty-prompts-file.sh
97 clitest: Error: no test found in input file: dev/test/empty-prompts-file.sh
98 $
99 ```
100
101 ## Option --version
102
103 The exit code must always be zero.
104
105 ```
106 $ ./clitest --version > /dev/null; echo $?
107 0
108 $
109 ```
110
111 Test the output text and the short option `-V`.
112
113 ```
114 $ ./clitest --version
115 clitest HEAD
116 https://github.com/aureliojargas/clitest/tree/HEAD
117 $ ./clitest -V
118 clitest HEAD
119 https://github.com/aureliojargas/clitest/tree/HEAD
120 $
121 ```
122
123 ## Option --help
124
125 Test the full help text contents and the exit code (zero).
126
127 ```
128 $ ./clitest --help; echo $?
129 Usage: clitest [options] <file ...>
130
131 Options:
132 -1, --first Stop execution upon first failed test
133 -l, --list List all the tests (no execution)
134 -L, --list-run List all the tests with OK/FAIL status
135 -t, --test RANGE Run specific tests, by number (1,2,4-7)
136 -s, --skip RANGE Skip specific tests, by number (1,2,4-7)
137 --pre-flight COMMAND Execute command before running the first test
138 --post-flight COMMAND Execute command after running the last test
139 -q, --quiet Quiet operation, no output shown
140 -V, --version Show program version and exit
141
142 Customization options:
143 -P, --progress TYPE Set progress indicator: test, number, dot, none
144 --color WHEN Set when to use colors: auto, always, never
145 --diff-options OPTIONS Set diff command options (default: '-u')
146 --inline-prefix PREFIX Set inline output prefix (default: '#→ ')
147 --prefix PREFIX Set command line prefix (default: '')
148 --prompt STRING Set prompt string (default: '$ ')
149 0
150 $
151 ```
152
153 The short option `-h` is working? Testing just the first and last lines for brevity.
154
155 ```
156 $ ./clitest -h | sed -n '1p; $p'
157 Usage: clitest [options] <file ...>
158 --prompt STRING Set prompt string (default: '$ ')
159 $
160 ```
161
162
163 ## Option --quiet and exit code
164
165 ```
166 $ ./clitest -q dev/test/ok-2.sh; echo $?
167 0
168 $ ./clitest --quiet dev/test/ok-2.sh; echo $?
169 0
170 $ ./clitest --quiet dev/test/ok-2.sh dev/test/ok-2.sh; echo $?
171 0
172 $ ./clitest --quiet dev/test/fail-2.sh; echo $?
173 1
174 $ ./clitest --quiet dev/test/fail-2.sh dev/test/fail-2.sh; echo $?
175 1
176 $ ./clitest --quiet dev/test/ok-2.sh dev/test/fail-2.sh; echo $?
177 1
178 $
179 ```
180
181 ## Option --quiet has no effect in error messages
182
183 ```
184 $ ./clitest --quiet /etc
185 clitest: Error: cannot read input file: /etc
186 $
187 ```
188
189 ## Option --quiet has no effect in --debug (disabled)
190
191 ```
192 # $ ./clitest --quiet --debug dev/test/ok-2.sh
193 # [INPUT_LINE: $ echo ok]
194 # [ LINE_CMD: $ echo ok]
195 # [ NEW_CMD: echo ok]
196 # [INPUT_LINE: ok]
197 # [ LINE_*: ok]
198 # [ OK_TEXT: ok]
199 # [INPUT_LINE: $ echo ok #→ ok]
200 # [ LINE_CMD: $ echo ok #→ ok]
201 # [ EVAL: echo ok]
202 # [ OUTPUT: ok]
203 # [ NEW_CMD: echo ok ]
204 # [ OK_INLINE: ok]
205 # [ OK_TEXT: ok]
206 # [ EVAL: echo ok ]
207 # [ OUTPUT: ok]
208 # [ LOOP_OUT: $test_command=]
209 # $
210 ```
211
212 ## Option --color
213
214 Invalid value
215
216 ```
217 $ ./clitest --color foo dev/test/ok-1.sh
218 clitest: Error: invalid value 'foo' for --color. Use: auto, always or never.
219 $
220 ```
221
222 Color ON
223
224 ```
225 $ ./clitest --color always dev/test/ok-1.sh
226 #1 echo ok
227 OK: 1 of 1 test passed
228 $ ./clitest --color yes dev/test/ok-1.sh
229 #1 echo ok
230 OK: 1 of 1 test passed
231 $
232 ```
233
234 Color OFF
235
236 ```
237 $ ./clitest --color never dev/test/ok-1.sh
238 #1 echo ok
239 OK: 1 of 1 test passed
240 $ ./clitest --color no dev/test/ok-1.sh
241 #1 echo ok
242 OK: 1 of 1 test passed
243 $
244 ```
245
246 Color AUTO
247
248 Inside this file, the output is not a terminal,
249 so the default is no colored output.
250
251 ```
252 $ ./clitest dev/test/ok-1.sh
253 #1 echo ok
254 OK: 1 of 1 test passed
255 $
256 ```
257
258 The real default `--color auto` cannot be tested here.
259 Test it by hand at the command line.
260
261 ```
262 ## $ ./clitest dev/test/ok-1.sh
263 ## OK! The single test has passed.
264 ## $ ./clitest --color auto dev/test/ok-1.sh
265 ## OK! The single test has passed.
266 ## $
267 ```
268
269 ## Option --list
270
271 Listing a file with no tests
272
273 ```
274 $ ./clitest --list dev/test/empty-file.sh
275 clitest: Error: no test found in input file: dev/test/empty-file.sh
276 $
277 ```
278
279 Normal results and exit code
280
281 ```
282 $ ./clitest --list dev/test/no-nl-command.sh; echo $?
283 #1 printf 'ok\n'
284 #2 printf 'fail'
285 #3 printf 'ok\nok\nfail'
286 #4 printf 'ok\n'
287 #5 printf 'fail'
288 #6 printf 'ok'; echo
289 #7 printf 'ok'
290 0
291 $
292 ```
293
294 Short option `-l`
295
296 ```
297 $ ./clitest -l dev/test/no-nl-command.sh
298 #1 printf 'ok\n'
299 #2 printf 'fail'
300 #3 printf 'ok\nok\nfail'
301 #4 printf 'ok\n'
302 #5 printf 'fail'
303 #6 printf 'ok'; echo
304 #7 printf 'ok'
305 $
306 ```
307
308 Multifile and exit code
309
310 ```
311 $ ./clitest --list dev/test/no-nl-command.sh dev/test/ok-1.sh; echo $?
312 ---------------------------------------- dev/test/no-nl-command.sh
313 #1 printf 'ok\n'
314 #2 printf 'fail'
315 #3 printf 'ok\nok\nfail'
316 #4 printf 'ok\n'
317 #5 printf 'fail'
318 #6 printf 'ok'; echo
319 #7 printf 'ok'
320 ---------------------------------------- dev/test/ok-1.sh
321 #8 echo ok
322 0
323 $
324 ```
325
326 ## Option --list-run
327
328 Listing a file with no tests
329
330 ```
331 $ ./clitest --list-run dev/test/empty-file.sh
332 clitest: Error: no test found in input file: dev/test/empty-file.sh
333 $
334 ```
335
336 Normal results (using colors) and exit code
337
338 ```
339 $ ./clitest --list-run --color yes dev/test/no-nl-command.sh; echo $?
340 #1 printf 'ok\n'
341 #2 printf 'fail'
342 #3 printf 'ok\nok\nfail'
343 #4 printf 'ok\n' 
344 #5 printf 'fail' 
345 #6 printf 'ok'; echo 
346 #7 printf 'ok' 
347 1
348 $
349 ```
350
351 Normal results (no colors, use OK/FAIL column) and exit code
352
353 ```
354 $ ./clitest --list-run dev/test/no-nl-command.sh; echo $?
355 #1 OK printf 'ok\n'
356 #2 FAIL printf 'fail'
357 #3 FAIL printf 'ok\nok\nfail'
358 #4 OK printf 'ok\n'
359 #5 FAIL printf 'fail'
360 #6 OK printf 'ok'; echo
361 #7 OK printf 'ok'
362 1
363 $
364 ```
365
366 Short option `-L`
367
368 ```
369 $ ./clitest -L dev/test/no-nl-command.sh
370 #1 OK printf 'ok\n'
371 #2 FAIL printf 'fail'
372 #3 FAIL printf 'ok\nok\nfail'
373 #4 OK printf 'ok\n'
374 #5 FAIL printf 'fail'
375 #6 OK printf 'ok'; echo
376 #7 OK printf 'ok'
377 $
378 ```
379
380 Multifile and exit code
381
382 ```
383 $ ./clitest -L dev/test/no-nl-command.sh dev/test/ok-1.sh; echo $?
384 ---------------------------------------- dev/test/no-nl-command.sh
385 #1 OK printf 'ok\n'
386 #2 FAIL printf 'fail'
387 #3 FAIL printf 'ok\nok\nfail'
388 #4 OK printf 'ok\n'
389 #5 FAIL printf 'fail'
390 #6 OK printf 'ok'; echo
391 #7 OK printf 'ok'
392 ---------------------------------------- dev/test/ok-1.sh
393 #8 OK echo ok
394 1
395 $ ./clitest -L dev/test/ok-1.sh; echo $?
396 #1 OK echo ok
397 0
398 $
399 ```
400
401 ## Option --progress
402
403 First, some invalid values:
404
405 ```
406 $ ./clitest --progress dev/test/ok-1.sh
407 clitest: Error: no test file informed (try --help)
408 $ ./clitest --progress '' dev/test/ok-1.sh
409 clitest: Error: invalid value '' for --progress. Use: test, number, dot or none.
410 $ ./clitest --progress foo dev/test/ok-1.sh
411 clitest: Error: invalid value 'foo' for --progress. Use: test, number, dot or none.
412 $ ./clitest --progress DOT dev/test/ok-1.sh
413 clitest: Error: invalid value 'DOT' for --progress. Use: test, number, dot or none.
414 $ ./clitest --progress @@ dev/test/ok-1.sh
415 clitest: Error: invalid value '@@' for --progress. Use: test, number, dot or none.
416 $ ./clitest --progress -1 dev/test/ok-1.sh
417 clitest: Error: invalid value '-1' for --progress. Use: test, number, dot or none.
418 $
419 ```
420
421 If no `--progress` option, defaults to `--progress test`:
422
423 ```
424 $ ./clitest dev/test/ok-1.sh
425 #1 echo ok
426 OK: 1 of 1 test passed
427 $ ./clitest --progress test dev/test/ok-1.sh
428 #1 echo ok
429 OK: 1 of 1 test passed
430 $
431 ```
432
433 Numbers:
434
435 ```
436 $ ./clitest --progress number dev/test/ok-10.sh
437 1 2 3 4 5 6 7 8 9 10
438 OK: 10 of 10 tests passed
439 $ ./clitest --progress n dev/test/ok-10.sh
440 1 2 3 4 5 6 7 8 9 10
441 OK: 10 of 10 tests passed
442 $ ./clitest --progress 0 dev/test/ok-10.sh
443 1 2 3 4 5 6 7 8 9 10
444 OK: 10 of 10 tests passed
445 $ ./clitest --progress 5 dev/test/ok-10.sh
446 1 2 3 4 5 6 7 8 9 10
447 OK: 10 of 10 tests passed
448 $ ./clitest --progress 9 dev/test/ok-10.sh
449 1 2 3 4 5 6 7 8 9 10
450 OK: 10 of 10 tests passed
451 $
452 ```
453
454 Chars:
455
456 ```
457 $ ./clitest --progress dot dev/test/ok-10.sh
458 ..........
459 OK: 10 of 10 tests passed
460 $ ./clitest --progress . dev/test/ok-10.sh
461 ..........
462 OK: 10 of 10 tests passed
463 $ ./clitest --progress @ dev/test/ok-10.sh
464 @@@@@@@@@@
465 OK: 10 of 10 tests passed
466 $ ./clitest --progress x dev/test/ok-10.sh
467 xxxxxxxxxx
468 OK: 10 of 10 tests passed
469 $
470 ```
471
472 No progress:
473
474 ```
475 $ ./clitest --progress none dev/test/ok-1.sh
476 OK: 1 of 1 test passed
477 $ ./clitest --progress no dev/test/ok-1.sh
478 OK: 1 of 1 test passed
479 $
480 ```
481
482 Short option `-P`:
483
484 ```
485 $ ./clitest -P dot dev/test/ok-1.sh
486 .
487 OK: 1 of 1 test passed
488 $ ./clitest -P no dev/test/ok-1.sh
489 OK: 1 of 1 test passed
490 $
491 ```
492
493 Ok & fail functionality with dot:
494
495 ```
496 $ ./clitest --progress . dev/test/ok-1.sh
497 .
498 OK: 1 of 1 test passed
499 $ ./clitest --progress . dev/test/ok-2.sh
500 ..
501 OK: 2 of 2 tests passed
502 $ ./clitest --progress . dev/test/ok-50.sh
503 ..................................................
504 OK: 50 of 50 tests passed
505 $ ./clitest --progress . dev/test/fail-1.sh
506 .
507 --------------------------------------------------------------------------------
508 [FAILED #1, line 1] echo ok
509 @@ -1 +1 @@
510 -fail
511 +ok
512 --------------------------------------------------------------------------------
513
514 FAIL: 1 of 1 test failed
515 $
516 ```
517
518 Multifile with dot:
519
520 ```
521 $ ./clitest --progress . dev/test/ok-1.sh dev/test/ok-2.sh dev/test/ok-10.sh
522 Testing file dev/test/ok-1.sh .
523 Testing file dev/test/ok-2.sh ..
524 Testing file dev/test/ok-10.sh ..........
525
526 ok fail skip
527 1 - - dev/test/ok-1.sh
528 2 - - dev/test/ok-2.sh
529 10 - - dev/test/ok-10.sh
530
531 OK: 13 of 13 tests passed
532 $ ./clitest --progress . dev/test/ok-1.sh dev/test/fail-1.sh
533 Testing file dev/test/ok-1.sh .
534 Testing file dev/test/fail-1.sh .
535 --------------------------------------------------------------------------------
536 [FAILED #2, line 1] echo ok
537 @@ -1 +1 @@
538 -fail
539 +ok
540 --------------------------------------------------------------------------------
541
542 ok fail skip
543 1 - - dev/test/ok-1.sh
544 - 1 - dev/test/fail-1.sh
545
546 FAIL: 1 of 2 tests failed
547 $ ./clitest --progress . dev/test/fail-1.sh dev/test/ok-1.sh
548 Testing file dev/test/fail-1.sh .
549 --------------------------------------------------------------------------------
550 [FAILED #1, line 1] echo ok
551 @@ -1 +1 @@
552 -fail
553 +ok
554 --------------------------------------------------------------------------------
555 Testing file dev/test/ok-1.sh .
556
557 ok fail skip
558 - 1 - dev/test/fail-1.sh
559 1 - - dev/test/ok-1.sh
560
561 FAIL: 1 of 2 tests failed
562 $
563 ```
564
565 Multifile with no progress:
566
567 ```
568 $ ./clitest --progress none dev/test/ok-1.sh dev/test/ok-2.sh dev/test/ok-10.sh
569 Testing file dev/test/ok-1.sh
570 Testing file dev/test/ok-2.sh
571 Testing file dev/test/ok-10.sh
572
573 ok fail skip
574 1 - - dev/test/ok-1.sh
575 2 - - dev/test/ok-2.sh
576 10 - - dev/test/ok-10.sh
577
578 OK: 13 of 13 tests passed
579 $ ./clitest --progress none dev/test/ok-1.sh dev/test/fail-1.sh
580 Testing file dev/test/ok-1.sh
581 Testing file dev/test/fail-1.sh
582 --------------------------------------------------------------------------------
583 [FAILED #2, line 1] echo ok
584 @@ -1 +1 @@
585 -fail
586 +ok
587 --------------------------------------------------------------------------------
588
589 ok fail skip
590 1 - - dev/test/ok-1.sh
591 - 1 - dev/test/fail-1.sh
592
593 FAIL: 1 of 2 tests failed
594 $ ./clitest --progress none dev/test/fail-1.sh dev/test/ok-1.sh
595 Testing file dev/test/fail-1.sh
596 --------------------------------------------------------------------------------
597 [FAILED #1, line 1] echo ok
598 @@ -1 +1 @@
599 -fail
600 +ok
601 --------------------------------------------------------------------------------
602 Testing file dev/test/ok-1.sh
603
604 ok fail skip
605 - 1 - dev/test/fail-1.sh
606 1 - - dev/test/ok-1.sh
607
608 FAIL: 1 of 2 tests failed
609 $
610 ```
611
612 ### Option --progress and skipped tests
613
614 Since skipped tests affect the output (show nothing), it's worth
615 testing if the line break issues won't appear.
616
617 ```
618 $ ./clitest --progress . --skip 1 dev/test/ok-2.sh
619 .
620 OK: 1 of 2 tests passed (1 skipped)
621 $ ./clitest --progress . --skip 2 dev/test/ok-2.sh
622 .
623 OK: 1 of 2 tests passed (1 skipped)
624 $ ./clitest --progress . --skip 1 dev/test/fail-2.sh
625 .
626 --------------------------------------------------------------------------------
627 [FAILED #2, line 3] echo ok
628 @@ -1 +1 @@
629 -fail
630 +ok
631 --------------------------------------------------------------------------------
632
633 FAIL: 1 of 2 tests failed (1 skipped)
634 $ ./clitest --progress . --skip 2 dev/test/fail-2.sh
635 .
636 --------------------------------------------------------------------------------
637 [FAILED #1, line 1] echo ok
638 @@ -1 +1 @@
639 -fail
640 +ok
641 --------------------------------------------------------------------------------
642
643 FAIL: 1 of 2 tests failed (1 skipped)
644 $
645 ```
646
647 Error messages appear with no leading blank line?
648
649 ```
650 $ ./clitest --progress . --skip 1,2 dev/test/ok-2.sh
651 clitest: Error: no test found. Maybe '--skip 1,2' was too much?
652 $
653 ```
654
655
656 ## Options --quiet, --progress, --list and --list-run are mutually exclusive
657
658 * Only one can be active, the others must be off.
659 * The last informed will be the one used.
660
661 ```
662 $ ./clitest --list --quiet dev/test/ok-1.sh
663 $ ./clitest --list-run --quiet dev/test/ok-1.sh
664 $ ./clitest --progress . --quiet dev/test/ok-1.sh
665 $ ./clitest --list --list-run --progress . --quiet dev/test/ok-1.sh
666 $ ./clitest --quiet --progress . --list-run --list dev/test/ok-1.sh
667 #1 echo ok
668 $ ./clitest --quiet --progress . --list --list-run dev/test/ok-1.sh
669 #1 OK echo ok
670 $ ./clitest --quiet --list --list-run --progress . dev/test/ok-1.sh
671 .
672 OK: 1 of 1 test passed
673 $
674 ```
675
676 ## Option --test and --skip combined with --list and --list-run
677
678 Error: Out of range
679
680 ```
681 $ ./clitest --list -t 99 dev/test/ok-10.sh
682 clitest: Error: no test found for the specified number or range '99'
683 $ ./clitest --list-run -t 99 dev/test/ok-10.sh
684 clitest: Error: no test found for the specified number or range '99'
685 $
686 ```
687
688 Error: Skipped all tests
689
690 ```
691 $ ./clitest --list -s 1-10 dev/test/ok-10.sh
692 clitest: Error: no test found. Maybe '--skip 1-10' was too much?
693 $ ./clitest --list-run -s 1-10 dev/test/ok-10.sh
694 clitest: Error: no test found. Maybe '--skip 1-10' was too much?
695 $
696 ```
697
698 Error: The combination of `-t` and `-s` resulted in no tests
699
700 ```
701 $ ./clitest --list -t 9 -s 9 dev/test/ok-10.sh
702 clitest: Error: no test found. The combination of -t and -s resulted in no tests.
703 $ ./clitest --list-run -t 9 -s 9 dev/test/ok-10.sh
704 clitest: Error: no test found. The combination of -t and -s resulted in no tests.
705 $
706 ```
707
708 Using `-t` alone
709
710 ```
711 $ ./clitest --list -t 3,5-7 dev/test/ok-10.sh
712 #3 echo 3
713 #5 echo 5
714 #6 echo 6
715 #7 echo 7
716 $ ./clitest --list-run -t 3,5-7 dev/test/ok-10.sh
717 #3 OK echo 3
718 #5 OK echo 5
719 #6 OK echo 6
720 #7 OK echo 7
721 $
722 ```
723
724 Using `-t` to limit to a range and the `-s` exclude some more
725
726 ```
727 $ ./clitest --list -t 3,5-7 -s 6 dev/test/ok-10.sh
728 #3 echo 3
729 #5 echo 5
730 #7 echo 7
731 $ ./clitest --list-run -t 3,5-7 -s 6 dev/test/ok-10.sh
732 #3 OK echo 3
733 #5 OK echo 5
734 #7 OK echo 7
735 $
736 ```
737
738 Multifile, using `-t` alone
739
740
741 ```
742 $ ./clitest --list -t 1,3,5-7 dev/test/ok-1.sh dev/test/fail-2.sh dev/test/ok-10.sh
743 ---------------------------------------- dev/test/ok-1.sh
744 #1 echo ok
745 ---------------------------------------- dev/test/fail-2.sh
746 #3 echo ok
747 ---------------------------------------- dev/test/ok-10.sh
748 #5 echo 2
749 #6 echo 3
750 #7 echo 4
751 $ ./clitest --list-run -t 1,3,5-7 dev/test/ok-1.sh dev/test/fail-2.sh dev/test/ok-10.sh
752 ---------------------------------------- dev/test/ok-1.sh
753 #1 OK echo ok
754 ---------------------------------------- dev/test/fail-2.sh
755 #3 FAIL echo ok
756 ---------------------------------------- dev/test/ok-10.sh
757 #5 OK echo 2
758 #6 OK echo 3
759 #7 OK echo 4
760 $
761 ```
762
763 Multifile, using `-t` and `-s`
764
765 ```
766 $ ./clitest --list -t 1,3,5-7 -s 3,6 dev/test/ok-1.sh dev/test/fail-2.sh dev/test/ok-10.sh
767 ---------------------------------------- dev/test/ok-1.sh
768 #1 echo ok
769 ---------------------------------------- dev/test/fail-2.sh
770 ---------------------------------------- dev/test/ok-10.sh
771 #5 echo 2
772 #7 echo 4
773 $ ./clitest --list-run -t 1,3,5-7 -s 3,6 dev/test/ok-1.sh dev/test/fail-2.sh dev/test/ok-10.sh
774 ---------------------------------------- dev/test/ok-1.sh
775 #1 OK echo ok
776 ---------------------------------------- dev/test/fail-2.sh
777 ---------------------------------------- dev/test/ok-10.sh
778 #5 OK echo 2
779 #7 OK echo 4
780 $
781 ```
782
783 ## Single file, OK
784
785 ```
786 $ ./clitest dev/test/ok-1.sh
787 #1 echo ok
788 OK: 1 of 1 test passed
789 $ ./clitest dev/test/ok-2.sh
790 #1 echo ok
791 #2 echo ok
792 OK: 2 of 2 tests passed
793 $ ./clitest dev/test/ok-50.sh | tail -1
794 OK: 50 of 50 tests passed
795 $ ./clitest dev/test/ok-100.sh | tail -1
796 OK: 100 of 100 tests passed
797 $ ./clitest dev/test/ok-2.sh
798 #1 echo ok
799 #2 echo ok
800 OK: 2 of 2 tests passed
801 $
802 ```
803
804 ## Multifile, all OK
805
806 ```
807 $ ./clitest dev/test/ok-2.sh dev/test/ok-2.sh
808 Testing file dev/test/ok-2.sh
809 #1 echo ok
810 #2 echo ok
811 Testing file dev/test/ok-2.sh
812 #3 echo ok
813 #4 echo ok
814
815 ok fail skip
816 2 - - dev/test/ok-2.sh
817 2 - - dev/test/ok-2.sh
818
819 OK: 4 of 4 tests passed
820 $ ./clitest dev/test/ok-[0-9]*.sh | grep -v ^#
821 Testing file dev/test/ok-1.sh
822 Testing file dev/test/ok-10.sh
823 Testing file dev/test/ok-100.sh
824 Testing file dev/test/ok-2.sh
825 Testing file dev/test/ok-50.sh
826
827 ok fail skip
828 1 - - dev/test/ok-1.sh
829 10 - - dev/test/ok-10.sh
830 100 - - dev/test/ok-100.sh
831 2 - - dev/test/ok-2.sh
832 50 - - dev/test/ok-50.sh
833
834 OK: 163 of 163 tests passed
835 $ ./clitest dev/test/ok-?.sh dev/test/ok-10.sh
836 Testing file dev/test/ok-1.sh
837 #1 echo ok
838 Testing file dev/test/ok-2.sh
839 #2 echo ok
840 #3 echo ok
841 Testing file dev/test/ok-10.sh
842 #4 echo 1
843 #5 echo 2
844 #6 echo 3
845 #7 echo 4
846 #8 echo 5
847 #9 echo 6
848 #10 echo 7
849 #11 echo 8
850 #12 echo 9
851 #13 echo 10
852
853 ok fail skip
854 1 - - dev/test/ok-1.sh
855 2 - - dev/test/ok-2.sh
856 10 - - dev/test/ok-10.sh
857
858 OK: 13 of 13 tests passed
859 $
860 ```
861
862 ## Multifile, OK and fail
863
864 ```
865 $ ./clitest dev/test/ok-1.sh dev/test/fail-1.sh dev/test/ok-2.sh dev/test/fail-2.sh
866 Testing file dev/test/ok-1.sh
867 #1 echo ok
868 Testing file dev/test/fail-1.sh
869 #2 echo ok
870 --------------------------------------------------------------------------------
871 [FAILED #2, line 1] echo ok
872 @@ -1 +1 @@
873 -fail
874 +ok
875 --------------------------------------------------------------------------------
876 Testing file dev/test/ok-2.sh
877 #3 echo ok
878 #4 echo ok
879 Testing file dev/test/fail-2.sh
880 #5 echo ok
881 --------------------------------------------------------------------------------
882 [FAILED #5, line 1] echo ok
883 @@ -1 +1 @@
884 -fail
885 +ok
886 --------------------------------------------------------------------------------
887 #6 echo ok
888 --------------------------------------------------------------------------------
889 [FAILED #6, line 3] echo ok
890 @@ -1 +1 @@
891 -fail
892 +ok
893 --------------------------------------------------------------------------------
894
895 ok fail skip
896 1 - - dev/test/ok-1.sh
897 - 1 - dev/test/fail-1.sh
898 2 - - dev/test/ok-2.sh
899 - 2 - dev/test/fail-2.sh
900
901 FAIL: 3 of 6 tests failed
902 $ ./clitest dev/test/ok-1.sh dev/test/fail-1.sh dev/test/ok-2.sh dev/test/fail-2.sh
903 Testing file dev/test/ok-1.sh
904 #1 echo ok
905 Testing file dev/test/fail-1.sh
906 #2 echo ok
907 --------------------------------------------------------------------------------
908 [FAILED #2, line 1] echo ok
909 @@ -1 +1 @@
910 -fail
911 +ok
912 --------------------------------------------------------------------------------
913 Testing file dev/test/ok-2.sh
914 #3 echo ok
915 #4 echo ok
916 Testing file dev/test/fail-2.sh
917 #5 echo ok
918 --------------------------------------------------------------------------------
919 [FAILED #5, line 1] echo ok
920 @@ -1 +1 @@
921 -fail
922 +ok
923 --------------------------------------------------------------------------------
924 #6 echo ok
925 --------------------------------------------------------------------------------
926 [FAILED #6, line 3] echo ok
927 @@ -1 +1 @@
928 -fail
929 +ok
930 --------------------------------------------------------------------------------
931
932 ok fail skip
933 1 - - dev/test/ok-1.sh
934 - 1 - dev/test/fail-1.sh
935 2 - - dev/test/ok-2.sh
936 - 2 - dev/test/fail-2.sh
937
938 FAIL: 3 of 6 tests failed
939 $
940 ```
941
942 ## Fail messages
943
944 ```
945 $ ./clitest --prefix tab -P none dev/test/fail-messages.md #→ --file test/fail-messages.out.txt
946 $
947 ```
948
949 ## Fails
950
951 ```
952 $ ./clitest dev/test/fail-1.sh
953 #1 echo ok
954 --------------------------------------------------------------------------------
955 [FAILED #1, line 1] echo ok
956 @@ -1 +1 @@
957 -fail
958 +ok
959 --------------------------------------------------------------------------------
960
961 FAIL: 1 of 1 test failed
962 $ ./clitest dev/test/fail-2.sh
963 #1 echo ok
964 --------------------------------------------------------------------------------
965 [FAILED #1, line 1] echo ok
966 @@ -1 +1 @@
967 -fail
968 +ok
969 --------------------------------------------------------------------------------
970 #2 echo ok
971 --------------------------------------------------------------------------------
972 [FAILED #2, line 3] echo ok
973 @@ -1 +1 @@
974 -fail
975 +ok
976 --------------------------------------------------------------------------------
977
978 FAIL: 2 of 2 tests failed
979 $ ./clitest dev/test/fail-50.sh | tail -1
980 FAIL: 50 of 50 tests failed
981 $ ./clitest -1 dev/test/fail-2.sh
982 #1 echo ok
983 --------------------------------------------------------------------------------
984 [FAILED #1, line 1] echo ok
985 @@ -1 +1 @@
986 -fail
987 +ok
988 --------------------------------------------------------------------------------
989 $ ./clitest --first dev/test/fail-2.sh
990 #1 echo ok
991 --------------------------------------------------------------------------------
992 [FAILED #1, line 1] echo ok
993 @@ -1 +1 @@
994 -fail
995 +ok
996 --------------------------------------------------------------------------------
997 $ ./clitest --first dev/test/fail-2.sh
998 #1 echo ok
999 --------------------------------------------------------------------------------
1000 [FAILED #1, line 1] echo ok
1001 @@ -1 +1 @@
1002 -fail
1003 +ok
1004 --------------------------------------------------------------------------------
1005 $ ./clitest dev/test/fail-2.sh
1006 #1 echo ok
1007 --------------------------------------------------------------------------------
1008 [FAILED #1, line 1] echo ok
1009 @@ -1 +1 @@
1010 -fail
1011 +ok
1012 --------------------------------------------------------------------------------
1013 #2 echo ok
1014 --------------------------------------------------------------------------------
1015 [FAILED #2, line 3] echo ok
1016 @@ -1 +1 @@
1017 -fail
1018 +ok
1019 --------------------------------------------------------------------------------
1020
1021 FAIL: 2 of 2 tests failed
1022 $
1023 ```
1024
1025 ## Inline output with #→
1026
1027 ```
1028 $ ./clitest dev/test/inline.sh
1029 #1 echo 'one space'
1030 #2 echo 'one tab'
1031 #3 echo 'multi spaces'
1032 #4 echo 'multi tabs'
1033 #5 echo 'mixed'
1034 #6 echo ' leading space'
1035 #7 echo ' leading spaces'
1036 #8 printf '\tleading tab\n'
1037 #9 printf '\t\tleading tabs\n'
1038 #10 echo 'trailing space '
1039 #11 echo 'trailing spaces '
1040 #12 printf 'trailing tab\t\n'
1041 #13 printf 'trailing tabs\t\t\n'
1042 #14 echo ' '
1043 #15 echo ' '
1044 #16 printf '\t\n'
1045 #17 printf '\t\t\t\n'
1046 #18 printf ' \t \t\t \n'
1047 #19 echo "both inline and normal output"
1048 OK: 19 of 19 tests passed
1049 $
1050 ```
1051
1052 ## Inline match modes
1053
1054 Mode #→ --text
1055
1056 * This is the default mode.
1057 * The --text part can be omitted.
1058
1059 ```
1060 $ ./clitest --list-run dev/test/inline-match-text.sh
1061 #1 OK echo 'abc'
1062 #2 OK echo 'abc'
1063 #3 OK printf '%s\n' '\t'
1064 #4 OK printf '%s\n' '\n'
1065 #5 OK echo '$PWD'
1066 #6 OK echo '$(date)'
1067 #7 OK echo '$'
1068 #8 OK echo '>'
1069 #9 OK echo '?'
1070 #10 OK echo '!'
1071 #11 OK echo '*'
1072 #12 OK echo '['
1073 #13 OK echo '('
1074 #14 OK echo
1075 #15 OK echo "not inline output" #→
1076 #16 OK echo '123456789'
1077 #17 OK echo '1 3 7 9'
1078 #18 OK echo ' 5 '
1079 #19 OK echo ' leading space'
1080 #20 OK echo ' leading spaces'
1081 #21 OK printf '\tleading tab\n'
1082 #22 OK printf '\t\tleading tabs\n'
1083 #23 OK echo 'trailing space '
1084 #24 OK echo 'trailing spaces '
1085 #25 OK printf 'trailing tab\t\n'
1086 #26 OK printf 'trailing tabs\t\t\n'
1087 #27 OK echo ' '
1088 #28 FAIL echo ' '
1089 #29 OK printf '\t\n'
1090 #30 OK printf '\t\t\t\n'
1091 #31 OK printf ' \t \t\t \n'
1092 #32 OK printf 'ok\n'
1093 #33 FAIL printf 'fail'
1094 #34 OK printf 'ok'; echo
1095 #35 FAIL echo 'fail'
1096 #36 FAIL echo 'fail'
1097 #37 OK echo ' ok'
1098 #38 OK echo '--text'
1099 #39 OK echo '--textual'
1100 #40 OK echo '--text is cool'
1101 $
1102 ```
1103
1104 Mode #→ --eval
1105
1106 ```
1107 $ ./clitest --list-run dev/test/inline-match-eval.sh
1108 #1 OK folder=$(pwd)
1109 #2 OK echo $folder
1110 #3 OK var='abc'
1111 #4 OK echo abc
1112 #5 OK echo 4
1113 #6 OK today=$(date +%D)
1114 #7 OK echo "Today is $today"
1115 #8 OK printf 'ok'
1116 #9 OK echo ' leading space'
1117 #10 OK echo ' leading spaces'
1118 #11 OK printf '\tleading tab\n'
1119 #12 OK printf '\t\tleading tabs\n'
1120 #13 OK echo 'trailing space '
1121 #14 OK echo 'trailing spaces '
1122 #15 OK printf 'trailing tab\t\n'
1123 #16 OK printf 'trailing tabs\t\t\n'
1124 #17 OK echo ' '
1125 #18 OK echo ' '
1126 #19 OK printf '\t\n'
1127 #20 OK printf '\t\t\t\n'
1128 #21 OK printf ' \t \t\t \n'
1129 #22 FAIL echo 'fail'
1130 #23 FAIL echo 'fail'
1131 #24 OK echo '--eval'
1132 #25 OK echo '--evaluate'
1133 #26 OK echo '--eval is evil'
1134 $
1135 ```
1136
1137 Mode #→ --egrep
1138
1139 ```
1140 $ ./clitest --list-run dev/test/inline-match-egrep.sh
1141 #1 OK echo 'abc123'
1142 #2 OK echo 'abc123'
1143 #3 OK echo 'abc123'
1144 #4 OK echo 'abc123'
1145 #5 OK echo 'abc123'
1146 #6 OK echo 'abc123'
1147 #7 OK echo 'abc123'
1148 #8 OK echo 'abc123'
1149 #9 OK echo 'abc 123'
1150 #10 OK echo ' '
1151 #11 OK echo ' '
1152 #12 OK printf '\t\n'
1153 #13 OK printf '\t\t\t\n'
1154 #14 OK printf ' \t \t\t \n'
1155 #15 OK printf 'may\tfail'
1156 #16 FAIL printf 'may\tfail'
1157 #17 OK printf 'will\tmatch'
1158 #18 FAIL printf 'will\nfail'
1159 #19 FAIL printf 'will\nfail'
1160 #20 OK printf '1\n2\n3\n4\nok\n'
1161 #21 OK printf 'ok'
1162 #22 OK printf 'ok\n'
1163 #23 FAIL echo 'fail'
1164 #24 FAIL echo 'fail'
1165 #25 OK echo ' ok'
1166 #26 OK echo '--egrep'
1167 #27 OK echo '--egreppal'
1168 #28 OK echo '--egrep is cool'
1169 $
1170 ```
1171
1172 Mode #→ --perl
1173
1174 * --regex is an alias to --perl
1175
1176 ```
1177 $ ./clitest --list-run dev/test/inline-match-perl.sh
1178 #1 OK echo 'abc123'
1179 #2 OK echo 'abc123'
1180 #3 OK echo 'abc123'
1181 #4 OK echo 'abc123'
1182 #5 OK echo 'abc123'
1183 #6 OK echo 'abc123'
1184 #7 OK echo 'abc123'
1185 #8 OK echo 'abc123'
1186 #9 OK echo 'abc 123'
1187 #10 OK echo ' '
1188 #11 OK echo ' '
1189 #12 OK printf '\t\n'
1190 #13 OK printf '\t\t\t\n'
1191 #14 OK printf ' \t \t\t \n'
1192 #15 OK echo '01/01/2013'
1193 #16 OK echo "won't fail"
1194 #17 OK printf 'will\tmatch'
1195 #18 OK printf 'will\tmatch'
1196 #19 OK printf 'will\tmatch'
1197 #20 FAIL printf 'will\nfail'
1198 #21 OK printf 'will\nmatch'
1199 #22 FAIL printf 'will\nfail'
1200 #23 OK printf 'will\nmatch'
1201 #24 FAIL printf 'will\nfail'
1202 #25 OK printf 'will\nmatch'
1203 #26 OK printf 'ok'
1204 #27 OK printf 'ok\n'
1205 #28 OK printf '1\n2\n3\n'
1206 #29 OK printf '1\n2\n3\n'
1207 #30 FAIL echo 'fail'
1208 #31 FAIL echo 'fail'
1209 #32 OK echo ' ok'
1210 #33 OK echo '--perl'
1211 #34 OK echo '--perlism'
1212 #35 OK echo '--perl is cool'
1213 $
1214 ```
1215
1216 Mode #→ --file
1217
1218 ```
1219 $ ./clitest --list-run dev/test/inline-match-file.sh
1220 #1 OK printf '$ echo ok\nok\n'
1221 #2 OK echo 'ok' > /tmp/foo.txt
1222 #3 OK echo 'ok'
1223 #4 FAIL echo 'fail'
1224 #5 FAIL echo 'fail'
1225 #6 OK echo '--file'
1226 #7 OK echo '--filer'
1227 #8 OK echo '--file is cool'
1228 $
1229 ```
1230
1231 Mode #→ --lines
1232
1233 ```
1234 $ ./clitest --list-run dev/test/inline-match-lines.sh
1235 #1 OK a=1
1236 #2 OK echo 'ok'
1237 #3 OK printf '1\n2\n3\n'
1238 #4 OK printf 'no-nl'
1239 #5 OK printf '1\n2\nno-nl'
1240 #6 FAIL echo 'fail'
1241 #7 FAIL echo 'fail'
1242 #8 FAIL echo 'fail'
1243 #9 FAIL echo 'fail'
1244 #10 OK echo '--lines'
1245 #11 OK echo '--linesout'
1246 #12 OK echo '--lines is cool'
1247 $ ./clitest --first dev/test/inline-match-lines.sh
1248 #1 a=1
1249 #2 echo 'ok'
1250 #3 printf '1\n2\n3\n'
1251 #4 printf 'no-nl'
1252 #5 printf '1\n2\nno-nl'
1253 #6 echo 'fail'
1254 --------------------------------------------------------------------------------
1255 [FAILED #6, line 16] echo 'fail'
1256 Expected 99 lines, got 1.
1257 --------------------------------------------------------------------------------
1258 $
1259 ```
1260
1261 Errors for #→ --egrep
1262
1263 ```
1264 $ ./clitest dev/test/inline-match-egrep-error-1.sh
1265 clitest: Error: empty --egrep at line 1 of dev/test/inline-match-egrep-error-1.sh
1266 $ ./clitest dev/test/inline-match-egrep-error-2.sh 2>&1 | sed 's/^egrep: .*/egrep: ERROR_MSG/'
1267 #1 echo "error: malformed regex"
1268 egrep: ERROR_MSG
1269 clitest: Error: check your inline egrep regex at line 1 of dev/test/inline-match-egrep-error-2.sh
1270 $
1271 ```
1272
1273 Errors for #→ --perl (and --regex)
1274
1275 ```
1276 $ ./clitest dev/test/inline-match-perl-error-1.sh
1277 clitest: Error: empty --perl at line 1 of dev/test/inline-match-perl-error-1.sh
1278 $ ./clitest dev/test/inline-match-perl-error-2.sh
1279 #1 echo "error: malformed regex"
1280 Unmatched ( in regex; marked by <-- HERE in m/( <-- HERE / at -e line 1.
1281 clitest: Error: check your inline Perl regex at line 1 of dev/test/inline-match-perl-error-2.sh
1282 $
1283 ```
1284
1285 Errors for #→ --file
1286
1287 ```
1288 $ ./clitest dev/test/inline-match-file-error-1.sh
1289 clitest: Error: empty --file at line 1 of dev/test/inline-match-file-error-1.sh
1290 $ ./clitest dev/test/inline-match-file-error-2.sh
1291 #1 echo "error: file not found"
1292 clitest: Error: cannot read inline output file 'dev/test/XXnotfoundXX', from line 1 of dev/test/inline-match-file-error-2.sh
1293 $ ./clitest dev/test/inline-match-file-error-3.sh
1294 #1 echo "error: directory"
1295 clitest: Error: cannot read inline output file '/etc/', from line 1 of dev/test/inline-match-file-error-3.sh
1296 $
1297 ```
1298
1299 Errors for #→ --lines
1300
1301 ```
1302 $ ./clitest dev/test/inline-match-lines-error-1.sh
1303 clitest: Error: --lines requires a number. See line 1 of dev/test/inline-match-lines-error-1.sh
1304 $ ./clitest dev/test/inline-match-lines-error-2.sh
1305 clitest: Error: --lines requires a number. See line 1 of dev/test/inline-match-lines-error-2.sh
1306 $ ./clitest dev/test/inline-match-lines-error-3.sh
1307 clitest: Error: --lines requires a number. See line 1 of dev/test/inline-match-lines-error-3.sh
1308 $ ./clitest dev/test/inline-match-lines-error-4.sh
1309 clitest: Error: --lines requires a number. See line 1 of dev/test/inline-match-lines-error-4.sh
1310 $
1311 ```
1312
1313 Errors for #→ --eval
1314
1315 ```
1316 $ ./clitest dev/test/inline-match-eval-error-1.sh
1317 clitest: Error: empty --eval at line 1 of dev/test/inline-match-eval-error-1.sh
1318 $ ./clitest dev/test/inline-match-eval-error-2.sh 2>&1 | sed 's/line [0-9][0-9]*/line N/'
1319 #1 echo 'error: syntax error'
1320 ./clitest: eval: line N: unexpected EOF while looking for matching `)'
1321 ./clitest: eval: line N: syntax error: unexpected end of file
1322 --------------------------------------------------------------------------------
1323 [FAILED #1, line N] echo 'error: syntax error'
1324 @@ -0,0 +1 @@
1325 +error: syntax error
1326 --------------------------------------------------------------------------------
1327
1328 FAIL: 1 of 1 test failed
1329 $
1330 ```
1331
1332 ## Option -t, --test
1333
1334 Error: Invalid argument
1335
1336 ```
1337 $ ./clitest -t - dev/test/ok-2.sh
1338 clitest: Error: invalid argument for -t or --test: -
1339 $ ./clitest -t -1 dev/test/ok-2.sh
1340 clitest: Error: invalid argument for -t or --test: -1
1341 $ ./clitest -t 1- dev/test/ok-2.sh
1342 clitest: Error: invalid argument for -t or --test: 1-
1343 $ ./clitest -t 1--2 dev/test/ok-2.sh
1344 clitest: Error: invalid argument for -t or --test: 1--2
1345 $ ./clitest -t 1-2-3 dev/test/ok-2.sh
1346 clitest: Error: invalid argument for -t or --test: 1-2-3
1347 $
1348 ```
1349
1350 Error: Out of range
1351
1352 ```
1353 $ ./clitest -t 99 dev/test/ok-2.sh
1354 clitest: Error: no test found for the specified number or range '99'
1355 $
1356 ```
1357
1358 If range = zero or empty, run all tests
1359
1360 ```
1361 $ ./clitest -t '' dev/test/ok-2.sh
1362 #1 echo ok
1363 #2 echo ok
1364 OK: 2 of 2 tests passed
1365 $ ./clitest -t 0 dev/test/ok-2.sh
1366 #1 echo ok
1367 #2 echo ok
1368 OK: 2 of 2 tests passed
1369 $
1370 ```
1371
1372 * Empty values inside range are ignored
1373 * The bogus `0-0` range is ignored
1374 * The resulting range is zero
1375
1376 ```
1377 $ ./clitest -t ,,,0,0-0,,, dev/test/ok-2.sh
1378 #1 echo ok
1379 #2 echo ok
1380 OK: 2 of 2 tests passed
1381 $
1382 ```
1383
1384 Normal operation, using `--test` and `-t`
1385
1386 ```
1387 $ ./clitest -t 1 dev/test/ok-10.sh
1388 #1 echo 1
1389 OK: 1 of 10 tests passed (9 skipped)
1390 $ ./clitest --test 1 dev/test/ok-10.sh
1391 #1 echo 1
1392 OK: 1 of 10 tests passed (9 skipped)
1393 $
1394 ```
1395
1396 Ranges `0-1` and `1-0` expand to `1`
1397
1398 ```
1399 $ ./clitest -t 0-1,1-0 dev/test/ok-10.sh
1400 #1 echo 1
1401 OK: 1 of 10 tests passed (9 skipped)
1402 $
1403 ```
1404
1405 Range `1-1` expand to `1`
1406
1407 ```
1408 $ ./clitest -t 1-1 dev/test/ok-10.sh
1409 #1 echo 1
1410 OK: 1 of 10 tests passed (9 skipped)
1411 $
1412 ```
1413
1414 Repeated values are OK
1415
1416 ```
1417 $ ./clitest -t 1,1,1,0,1 dev/test/ok-10.sh
1418 #1 echo 1
1419 OK: 1 of 10 tests passed (9 skipped)
1420 $
1421 ```
1422
1423 Range terminator is out of bounds
1424
1425 ```
1426 $ ./clitest -t 10-20 dev/test/ok-10.sh
1427 #10 echo 10
1428 OK: 1 of 10 tests passed (9 skipped)
1429 $
1430 ```
1431
1432 Inverted ranges
1433
1434 ```
1435 $ ./clitest -t 3,2,1 dev/test/ok-10.sh
1436 #1 echo 1
1437 #2 echo 2
1438 #3 echo 3
1439 OK: 3 of 10 tests passed (7 skipped)
1440 $ ./clitest -t 3-1 dev/test/ok-10.sh
1441 #1 echo 1
1442 #2 echo 2
1443 #3 echo 3
1444 OK: 3 of 10 tests passed (7 skipped)
1445 $
1446 ```
1447
1448 Multifile. The test numbers always increase sequentially, regardless of the file changes.
1449
1450 ```
1451 $ ./clitest -t 1,5,13 dev/test/ok-?.sh dev/test/ok-10.sh
1452 Testing file dev/test/ok-1.sh
1453 #1 echo ok
1454 Testing file dev/test/ok-2.sh
1455 Testing file dev/test/ok-10.sh
1456 #5 echo 2
1457 #13 echo 10
1458
1459 ok fail skip
1460 1 - - dev/test/ok-1.sh
1461 - - 2 dev/test/ok-2.sh
1462 2 - 8 dev/test/ok-10.sh
1463
1464 OK: 3 of 13 tests passed (10 skipped)
1465 $ ./clitest -t 1,5 dev/test/ok-[12].sh dev/test/fail-2.sh
1466 Testing file dev/test/ok-1.sh
1467 #1 echo ok
1468 Testing file dev/test/ok-2.sh
1469 Testing file dev/test/fail-2.sh
1470 #5 echo ok
1471 --------------------------------------------------------------------------------
1472 [FAILED #5, line 3] echo ok
1473 @@ -1 +1 @@
1474 -fail
1475 +ok
1476 --------------------------------------------------------------------------------
1477
1478 ok fail skip
1479 1 - - dev/test/ok-1.sh
1480 - - 2 dev/test/ok-2.sh
1481 - 1 1 dev/test/fail-2.sh
1482
1483 FAIL: 1 of 5 tests failed (3 skipped)
1484 $ ./clitest -t 1 dev/test/ok-[12].sh dev/test/fail-2.sh
1485 Testing file dev/test/ok-1.sh
1486 #1 echo ok
1487 Testing file dev/test/ok-2.sh
1488 Testing file dev/test/fail-2.sh
1489
1490 ok fail skip
1491 1 - - dev/test/ok-1.sh
1492 - - 2 dev/test/ok-2.sh
1493 - - 2 dev/test/fail-2.sh
1494
1495 OK: 1 of 5 tests passed (4 skipped)
1496 $
1497 ```
1498
1499 ## Option -s, --skip
1500
1501 Error: Invalid argument
1502
1503 ```
1504 $ ./clitest -s - dev/test/ok-2.sh
1505 clitest: Error: invalid argument for -s or --skip: -
1506 $ ./clitest -s -1 dev/test/ok-2.sh
1507 clitest: Error: invalid argument for -s or --skip: -1
1508 $ ./clitest -s 1- dev/test/ok-2.sh
1509 clitest: Error: invalid argument for -s or --skip: 1-
1510 $ ./clitest -s 1--2 dev/test/ok-2.sh
1511 clitest: Error: invalid argument for -s or --skip: 1--2
1512 $ ./clitest -s 1-2-3 dev/test/ok-2.sh
1513 clitest: Error: invalid argument for -s or --skip: 1-2-3
1514 $
1515 ```
1516
1517 Error: Skipped all tests
1518
1519 ```
1520 $ ./clitest -s 1 dev/test/ok-1.sh
1521 clitest: Error: no test found. Maybe '--skip 1' was too much?
1522 $
1523 ```
1524
1525 Out of range: no problem, you just skipped a non-existent test. All tests will be run.
1526
1527 ```
1528 $ ./clitest -s 99 dev/test/ok-2.sh
1529 #1 echo ok
1530 #2 echo ok
1531 OK: 2 of 2 tests passed
1532 $
1533 ```
1534
1535 If range = zero or empty, run all tests
1536
1537 ```
1538 $ ./clitest -s '' dev/test/ok-2.sh
1539 #1 echo ok
1540 #2 echo ok
1541 OK: 2 of 2 tests passed
1542 $ ./clitest -s 0 dev/test/ok-2.sh
1543 #1 echo ok
1544 #2 echo ok
1545 OK: 2 of 2 tests passed
1546 $
1547 ```
1548
1549 * Empty values inside range are ignored
1550 * The bogus `0-0` range is ignored
1551 * The resulting range is zero
1552
1553 ```
1554 $ ./clitest -s ,,,0,0-0,,, dev/test/ok-2.sh
1555 #1 echo ok
1556 #2 echo ok
1557 OK: 2 of 2 tests passed
1558 $
1559 ```
1560
1561 Normal operation, using `--skip` and `-s`
1562
1563 ```
1564 $ ./clitest -s 1 dev/test/ok-2.sh
1565 #2 echo ok
1566 OK: 1 of 2 tests passed (1 skipped)
1567 $ ./clitest --skip 1 dev/test/ok-2.sh
1568 #2 echo ok
1569 OK: 1 of 2 tests passed (1 skipped)
1570 $
1571 ```
1572
1573 Ranges `0-1` and `1-0` expand to `1`
1574
1575 ```
1576 $ ./clitest -s 0-1,1-0 dev/test/ok-2.sh
1577 #2 echo ok
1578 OK: 1 of 2 tests passed (1 skipped)
1579 $
1580 ```
1581
1582 Range `1-1` expand to `1`
1583
1584 ```
1585 $ ./clitest -s 1-1 dev/test/ok-2.sh
1586 #2 echo ok
1587 OK: 1 of 2 tests passed (1 skipped)
1588 $
1589 ```
1590
1591 Repeated values are OK
1592
1593 ```
1594 $ ./clitest -s 1,1,1,0,1 dev/test/ok-2.sh
1595 #2 echo ok
1596 OK: 1 of 2 tests passed (1 skipped)
1597 $
1598 ```
1599
1600 Range terminator is out of bounds
1601
1602 ```
1603 $ ./clitest -s 2-10 dev/test/ok-2.sh
1604 #1 echo ok
1605 OK: 1 of 2 tests passed (1 skipped)
1606 $
1607 ```
1608
1609 Inverted ranges
1610
1611 ```
1612 $ ./clitest -s 10,9,8,7,6,5,4 dev/test/ok-10.sh
1613 #1 echo 1
1614 #2 echo 2
1615 #3 echo 3
1616 OK: 3 of 10 tests passed (7 skipped)
1617 $ ./clitest -s 10-4 dev/test/ok-10.sh
1618 #1 echo 1
1619 #2 echo 2
1620 #3 echo 3
1621 OK: 3 of 10 tests passed (7 skipped)
1622 $
1623 ```
1624
1625 Multifile. The test numbers always increase sequentially, regardless of the file changes.
1626
1627 ```
1628 $ ./clitest -s 2,3,13 dev/test/ok-?.sh dev/test/ok-10.sh
1629 Testing file dev/test/ok-1.sh
1630 #1 echo ok
1631 Testing file dev/test/ok-2.sh
1632 Testing file dev/test/ok-10.sh
1633 #4 echo 1
1634 #5 echo 2
1635 #6 echo 3
1636 #7 echo 4
1637 #8 echo 5
1638 #9 echo 6
1639 #10 echo 7
1640 #11 echo 8
1641 #12 echo 9
1642
1643 ok fail skip
1644 1 - - dev/test/ok-1.sh
1645 - - 2 dev/test/ok-2.sh
1646 9 - 1 dev/test/ok-10.sh
1647
1648 OK: 10 of 13 tests passed (3 skipped)
1649 $ ./clitest -s 2,3,4 dev/test/ok-[12].sh dev/test/fail-2.sh
1650 Testing file dev/test/ok-1.sh
1651 #1 echo ok
1652 Testing file dev/test/ok-2.sh
1653 Testing file dev/test/fail-2.sh
1654 #5 echo ok
1655 --------------------------------------------------------------------------------
1656 [FAILED #5, line 3] echo ok
1657 @@ -1 +1 @@
1658 -fail
1659 +ok
1660 --------------------------------------------------------------------------------
1661
1662 ok fail skip
1663 1 - - dev/test/ok-1.sh
1664 - - 2 dev/test/ok-2.sh
1665 - 1 1 dev/test/fail-2.sh
1666
1667 FAIL: 1 of 5 tests failed (3 skipped)
1668 $ ./clitest -s 2-10 dev/test/ok-[12].sh dev/test/fail-2.sh
1669 Testing file dev/test/ok-1.sh
1670 #1 echo ok
1671 Testing file dev/test/ok-2.sh
1672 Testing file dev/test/fail-2.sh
1673
1674 ok fail skip
1675 1 - - dev/test/ok-1.sh
1676 - - 2 dev/test/ok-2.sh
1677 - - 2 dev/test/fail-2.sh
1678
1679 OK: 1 of 5 tests passed (4 skipped)
1680 $
1681 ```
1682
1683 ## Option --test combined with --skip
1684
1685 Error: The combination of `-t` and `-s` resulted in no tests
1686
1687 ```
1688 $ ./clitest -t 9 -s 9 dev/test/ok-10.sh
1689 clitest: Error: no test found. The combination of -t and -s resulted in no tests.
1690 $
1691 ```
1692
1693 The order does not matter, `-s` always wins
1694
1695 ```
1696 $ ./clitest -s 9 -t 9 dev/test/ok-10.sh
1697 clitest: Error: no test found. The combination of -t and -s resulted in no tests.
1698 $
1699 ```
1700
1701 Using `-t` to limit to a range and the `-s` exclude some more
1702
1703 ```
1704 $ ./clitest -t 3,5-7 -s 6 dev/test/ok-10.sh
1705 #3 echo 3
1706 #5 echo 5
1707 #7 echo 7
1708 OK: 3 of 10 tests passed (7 skipped)
1709 $
1710 ```
1711
1712 Same as previous, but now multifile
1713
1714 ```
1715 $ ./clitest -t 1,3,5-7 -s 3,6 dev/test/ok-1.sh dev/test/fail-2.sh dev/test/ok-10.sh
1716 Testing file dev/test/ok-1.sh
1717 #1 echo ok
1718 Testing file dev/test/fail-2.sh
1719 Testing file dev/test/ok-10.sh
1720 #5 echo 2
1721 #7 echo 4
1722
1723 ok fail skip
1724 1 - - dev/test/ok-1.sh
1725 - - 2 dev/test/fail-2.sh
1726 2 - 8 dev/test/ok-10.sh
1727
1728 OK: 3 of 13 tests passed (10 skipped)
1729 $
1730 ```
1731
1732 ## Option --diff-options
1733
1734 ```
1735 $ ./clitest dev/test/option-diff-options.sh
1736 #1 echo " diff -w to ignore spaces "
1737 --------------------------------------------------------------------------------
1738 [FAILED #1, line 3] echo " diff -w to ignore spaces "
1739 @@ -1 +1 @@
1740 -diff -w to ignore spaces
1741 + diff -w to ignore spaces
1742 --------------------------------------------------------------------------------
1743 #2 echo " diff -w now inline "
1744 --------------------------------------------------------------------------------
1745 [FAILED #2, line 5] echo " diff -w now inline "
1746 @@ -1 +1 @@
1747 -diff -w now inline
1748 + diff -w now inline
1749 --------------------------------------------------------------------------------
1750
1751 FAIL: 2 of 2 tests failed
1752 $ ./clitest --diff-options '-u -w' dev/test/option-diff-options.sh
1753 #1 echo " diff -w to ignore spaces "
1754 #2 echo " diff -w now inline "
1755 OK: 2 of 2 tests passed
1756 $
1757 ```
1758
1759 ## Option --prompt
1760
1761 ```
1762 $ ./clitest dev/test/option-prompt.sh
1763 clitest: Error: no test found in input file: dev/test/option-prompt.sh
1764 $ ./clitest --prompt 'prompt$ ' dev/test/option-prompt.sh
1765 #1 echo "1"
1766 #2 echo "2"
1767 #3 echo "3"
1768 OK: 3 of 3 tests passed
1769 $ ./clitest --prompt '♥ ' dev/test/option-prompt-unicode.sh
1770 #1 echo "1"
1771 #2 echo "2"
1772 #3 echo "3"
1773 OK: 3 of 3 tests passed
1774 $
1775 ```
1776
1777 ## Option --inline-prefix
1778
1779 ```
1780 $ ./clitest dev/test/option-inline-prefix.sh
1781 #1 echo "1 space" #==> 1 space
1782 --------------------------------------------------------------------------------
1783 [FAILED #1, line 3] echo "1 space" #==> 1 space
1784 @@ -0,0 +1 @@
1785 +1 space
1786 --------------------------------------------------------------------------------
1787 #2 echo "8 spaces" #==> 8 spaces
1788 --------------------------------------------------------------------------------
1789 [FAILED #2, line 4] echo "8 spaces" #==> 8 spaces
1790 @@ -0,0 +1 @@
1791 +8 spaces
1792 --------------------------------------------------------------------------------
1793 #3 echo "2 tabs" #==> 2 tabs
1794 --------------------------------------------------------------------------------
1795 [FAILED #3, line 5] echo "2 tabs" #==> 2 tabs
1796 @@ -0,0 +1 @@
1797 +2 tabs
1798 --------------------------------------------------------------------------------
1799
1800 FAIL: 3 of 3 tests failed
1801 $ ./clitest --inline-prefix '#==>' dev/test/option-inline-prefix.sh
1802 #1 echo "1 space"
1803 --------------------------------------------------------------------------------
1804 [FAILED #1, line 3] echo "1 space"
1805 @@ -1 +1 @@
1806 - 1 space
1807 +1 space
1808 --------------------------------------------------------------------------------
1809 #2 echo "8 spaces"
1810 --------------------------------------------------------------------------------
1811 [FAILED #2, line 4] echo "8 spaces"
1812 @@ -1 +1 @@
1813 - 8 spaces
1814 +8 spaces
1815 --------------------------------------------------------------------------------
1816 #3 echo "2 tabs"
1817 --------------------------------------------------------------------------------
1818 [FAILED #3, line 5] echo "2 tabs"
1819 @@ -1 +1 @@
1820 - 2 tabs
1821 +2 tabs
1822 --------------------------------------------------------------------------------
1823
1824 FAIL: 3 of 3 tests failed
1825 $ ./clitest --inline-prefix '#==> ' dev/test/option-inline-prefix.sh
1826 #1 echo "1 space"
1827 #2 echo "8 spaces"
1828 #3 echo "2 tabs"
1829 OK: 3 of 3 tests passed
1830 $
1831 ```
1832
1833 ## Option --prefix
1834
1835 ```
1836 $ ./clitest --prefix ' ' dev/test/option-prefix.sh
1837 #1 echo "1"
1838 #2 echo "2"
1839 #3 echo "3"
1840 #4 echo "4"
1841 #5 echo "5"
1842 #6 echo; echo "6"; echo; echo "7"
1843 OK: 6 of 6 tests passed
1844 $ ./clitest --prefix 4 dev/test/option-prefix.sh
1845 #1 echo "1"
1846 #2 echo "2"
1847 #3 echo "3"
1848 #4 echo "4"
1849 #5 echo "5"
1850 #6 echo; echo "6"; echo; echo "7"
1851 OK: 6 of 6 tests passed
1852 $ ./clitest --prefix '\t' dev/test/option-prefix-tab.sh
1853 #1 echo "1"
1854 #2 echo "2"
1855 #3 echo "3"
1856 #4 echo "4"
1857 #5 echo "5"
1858 #6 echo; echo "6"; echo; echo "7"
1859 OK: 6 of 6 tests passed
1860 $ ./clitest --prefix tab dev/test/option-prefix-tab.sh
1861 #1 echo "1"
1862 #2 echo "2"
1863 #3 echo "3"
1864 #4 echo "4"
1865 #5 echo "5"
1866 #6 echo; echo "6"; echo; echo "7"
1867 OK: 6 of 6 tests passed
1868 $
1869 ```
1870
1871 ## Option --prefix: glob gotchas
1872
1873 ```
1874 $ ./clitest --prefix '?' dev/test/option-prefix-glob.sh
1875 #1 echo 'prefix ?'
1876 #2 echo 'prefix ?'
1877 OK: 2 of 2 tests passed
1878 $ ./clitest --prefix '*' dev/test/option-prefix-glob.sh
1879 #1 echo 'prefix *'
1880 #2 echo 'prefix *'
1881 OK: 2 of 2 tests passed
1882 $ ./clitest --prefix '#' dev/test/option-prefix-glob.sh
1883 #1 echo 'prefix #'
1884 #2 echo 'prefix #'
1885 OK: 2 of 2 tests passed
1886 $ ./clitest --prefix '%' dev/test/option-prefix-glob.sh
1887 #1 echo 'prefix %'
1888 #2 echo 'prefix %'
1889 OK: 2 of 2 tests passed
1890 $ ./clitest --prefix '##' dev/test/option-prefix-glob.sh
1891 #1 echo 'prefix ##'
1892 #2 echo 'prefix ##'
1893 OK: 2 of 2 tests passed
1894 $ ./clitest --prefix '%%' dev/test/option-prefix-glob.sh
1895 #1 echo 'prefix %%'
1896 #2 echo 'prefix %%'
1897 OK: 2 of 2 tests passed
1898 $ ./clitest --prefix '#*' dev/test/option-prefix-glob.sh
1899 #1 echo 'prefix #*'
1900 #2 echo 'prefix #*'
1901 OK: 2 of 2 tests passed
1902 $ ./clitest --prefix '*#' dev/test/option-prefix-glob.sh
1903 #1 echo 'prefix *#'
1904 #2 echo 'prefix *#'
1905 OK: 2 of 2 tests passed
1906 $ ./clitest --prefix '%*' dev/test/option-prefix-glob.sh
1907 #1 echo 'prefix %*'
1908 #2 echo 'prefix %*'
1909 OK: 2 of 2 tests passed
1910 $ ./clitest --prefix '*%' dev/test/option-prefix-glob.sh
1911 #1 echo 'prefix *%'
1912 #2 echo 'prefix *%'
1913 OK: 2 of 2 tests passed
1914 $
1915 ```
1916
1917 ## Option --prompt: glob gotchas (char + space)
1918
1919 ```
1920 $ ./clitest --prompt '? ' dev/test/option-prompt-glob-space.sh
1921 #1 echo 'prompt ? '
1922 #2 echo 'prompt ? '
1923 OK: 2 of 2 tests passed
1924 $ ./clitest --prompt '* ' dev/test/option-prompt-glob-space.sh
1925 #1 echo 'prompt * '
1926 #2 echo 'prompt * '
1927 OK: 2 of 2 tests passed
1928 $ ./clitest --prompt '# ' dev/test/option-prompt-glob-space.sh
1929 #1 echo 'prompt # '
1930 #2 echo 'prompt # '
1931 OK: 2 of 2 tests passed
1932 $ ./clitest --prompt '% ' dev/test/option-prompt-glob-space.sh
1933 #1 echo 'prompt % '
1934 #2 echo 'prompt % '
1935 OK: 2 of 2 tests passed
1936 $ ./clitest --prompt '## ' dev/test/option-prompt-glob-space.sh
1937 #1 echo 'prompt ## '
1938 #2 echo 'prompt ## '
1939 OK: 2 of 2 tests passed
1940 $ ./clitest --prompt '%% ' dev/test/option-prompt-glob-space.sh
1941 #1 echo 'prompt %% '
1942 #2 echo 'prompt %% '
1943 OK: 2 of 2 tests passed
1944 $ ./clitest --prompt '#* ' dev/test/option-prompt-glob-space.sh
1945 #1 echo 'prompt #* '
1946 #2 echo 'prompt #* '
1947 OK: 2 of 2 tests passed
1948 $ ./clitest --prompt '*# ' dev/test/option-prompt-glob-space.sh
1949 #1 echo 'prompt *# '
1950 #2 echo 'prompt *# '
1951 OK: 2 of 2 tests passed
1952 $ ./clitest --prompt '%* ' dev/test/option-prompt-glob-space.sh
1953 #1 echo 'prompt %* '
1954 #2 echo 'prompt %* '
1955 OK: 2 of 2 tests passed
1956 $ ./clitest --prompt '*% ' dev/test/option-prompt-glob-space.sh
1957 #1 echo 'prompt *% '
1958 #2 echo 'prompt *% '
1959 OK: 2 of 2 tests passed
1960 $
1961 ```
1962
1963 ## Option --prompt: glob gotchas (chars only)
1964
1965 ```
1966 $ ./clitest --prompt '?' dev/test/option-prompt-glob-1.sh
1967 #1 echo 'prompt ?'
1968 #2 echo 'prompt ?'
1969 OK: 2 of 2 tests passed
1970 $ ./clitest --prompt '*' dev/test/option-prompt-glob-1.sh
1971 #1 echo 'prompt *'
1972 #2 echo 'prompt *'
1973 OK: 2 of 2 tests passed
1974 $ ./clitest --prompt '#' dev/test/option-prompt-glob-1.sh
1975 #1 echo 'prompt #'
1976 #2 echo 'prompt #'
1977 OK: 2 of 2 tests passed
1978 $ ./clitest --prompt '%' dev/test/option-prompt-glob-1.sh
1979 #1 echo 'prompt %'
1980 #2 echo 'prompt %'
1981 OK: 2 of 2 tests passed
1982 $ ./clitest --prompt '##' dev/test/option-prompt-glob-2.sh
1983 #1 echo 'prompt ##'
1984 #2 echo 'prompt ##'
1985 OK: 2 of 2 tests passed
1986 $ ./clitest --prompt '%%' dev/test/option-prompt-glob-2.sh
1987 #1 echo 'prompt %%'
1988 #2 echo 'prompt %%'
1989 OK: 2 of 2 tests passed
1990 $ ./clitest --prompt '#*' dev/test/option-prompt-glob-2.sh
1991 #1 echo 'prompt #*'
1992 #2 echo 'prompt #*'
1993 OK: 2 of 2 tests passed
1994 $ ./clitest --prompt '*#' dev/test/option-prompt-glob-2.sh
1995 #1 echo 'prompt *#'
1996 #2 echo 'prompt *#'
1997 OK: 2 of 2 tests passed
1998 $ ./clitest --prompt '%*' dev/test/option-prompt-glob-2.sh
1999 #1 echo 'prompt %*'
2000 #2 echo 'prompt %*'
2001 OK: 2 of 2 tests passed
2002 $ ./clitest --prompt '*%' dev/test/option-prompt-glob-2.sh
2003 #1 echo 'prompt *%'
2004 #2 echo 'prompt *%'
2005 OK: 2 of 2 tests passed
2006 $
2007 ```
2008
2009 ## Options --pre-flight and --post-flight
2010
2011 ```
2012 $ ./clitest --pre-flight 'tt_test_number=99; tt_nr_total_tests=99' dev/test/ok-1.sh
2013 #100 echo ok
2014 OK: 100 of 100 tests passed
2015 $ ./clitest --post-flight 'tt_nr_total_fails=50' dev/test/ok-50.sh | tail -1
2016 FAIL: 50 of 50 tests failed
2017 $ ./clitest --pre-flight 'false' dev/test/ok-1.sh
2018 clitest: Error: pre-flight command failed with status=1: false
2019 $
2020 ```
2021
2022 ## Options terminator --
2023
2024 ```
2025 $ ./clitest -t 99 -- --quiet
2026 clitest: Error: cannot read input file: --quiet
2027 $
2028 ```
2029
2030 ## File - meaning STDIN (no support for now)
2031
2032 ```
2033 $ cat dev/test/ok-1.sh | ./clitest -
2034 clitest: Error: cannot read input file: -
2035 $
2036 ```
2037
2038 ## Gotchas
2039
2040 Exit code: normal and error
2041
2042 ```
2043 $ ./clitest dev/test/exit-code.sh
2044 #1 echo "ok" > /dev/null; echo $?
2045 #2 cp XXnotfoundXX foo 2> /dev/null; echo $?
2046 OK: 2 of 2 tests passed
2047 $
2048 ```
2049
2050 STDIN and STDOUT
2051
2052 ```
2053 $ ./clitest dev/test/stdout-stderr.sh
2054 #1 echo "stdout"
2055 #2 echo "stdout" 2> /dev/null
2056 #3 echo "stderr" 1>&2
2057 #4 echo "stdout" > /dev/null
2058 #5 echo "stdout" 2> /dev/null 1>&2
2059 #6 cp XXnotfoundXX foo
2060 #7 cp XXnotfoundXX foo > /dev/null
2061 #8 cp XXnotfoundXX foo 2>&1
2062 #9 cp XXnotfoundXX foo 2> /dev/null
2063 #10 cp XXnotfoundXX foo > /dev/null 2>&1
2064 OK: 10 of 10 tests passed
2065 $
2066 ```
2067
2068 Multiple commands in one line
2069
2070 ```
2071 $ ./clitest dev/test/multi-commands.sh
2072 #1 echo 1; echo 2; echo 3; echo 4; echo 5
2073 #2 (echo 1; echo 2; echo 3; echo 4; echo 5) | sed -n 3p
2074 #3 (echo 1; echo 2; echo 3; echo 4; echo 5) | sed -n 3p
2075 OK: 3 of 3 tests passed
2076 $
2077 ```
2078
2079 A `cd` command in one test should not affect the next
2080
2081 ```
2082 $ ./clitest dev/test/cd.sh dev/test/ok-2.sh
2083 Testing file dev/test/cd.sh
2084 #1 cd
2085 Testing file dev/test/ok-2.sh
2086 #2 echo ok
2087 #3 echo ok
2088
2089 ok fail skip
2090 1 - - dev/test/cd.sh
2091 2 - - dev/test/ok-2.sh
2092
2093 OK: 3 of 3 tests passed
2094 $
2095 ```
2096
2097 Syntax: End-of-file or empty prompt closes the previous command
2098
2099 ```
2100 $ ./clitest dev/test/close-command.sh
2101 #1 echo 1
2102 #2 echo 2
2103 #3 echo 3
2104 OK: 3 of 3 tests passed
2105 $
2106 ```
2107
2108 Windows files (CR+LF)
2109
2110 ```
2111 $ ./clitest dev/test/windows.sh
2112 #1 echo "a file with CRLF line ending"
2113 #2 echo "inline output"
2114 #3 echo "inline regex"
2115 OK: 3 of 3 tests passed
2116 $
2117 ```
2118
2119 Unicode chars
2120
2121 ```
2122 $ ./clitest dev/test/special-chars.sh | tail -1
2123 OK: 206 of 206 tests passed
2124 $
2125 ```
2126
2127 Blanks (space, tab, newline) in the output
2128
2129 ```
2130 $ ./clitest dev/test/blank-output.sh
2131 #1 echo ' '
2132 #2 echo ' '
2133 #3 printf '\t\n'
2134 #4 printf '\t\t\t\n'
2135 #5 printf ' \t \t\t \n'
2136 #6 printf '\n \n \n \n \n\n'
2137 #7 printf '\n\t\n\t\t\n\t\t\t\n\t\t\t\t\n\n'
2138 #8 printf '\n'
2139 #9 printf '\n\n'
2140 #10 printf '\n\n\n\n'
2141 OK: 10 of 10 tests passed
2142 $
2143 ```
2144
2145 Files with no newline (`\n`) at the end
2146
2147 1. No empty prompt at the last line
2148 2. Empty prompt at the last line
2149 3. Inline output
2150
2151 ```
2152 $ ./clitest dev/test/no-nl-file-1.sh
2153 #1 printf '%s\n' 'a file with no \n at the last line'
2154 OK: 1 of 1 test passed
2155 $ ./clitest dev/test/no-nl-file-2.sh
2156 #1 printf '%s\n' 'another file with no \n at the last line'
2157 OK: 1 of 1 test passed
2158 $ ./clitest dev/test/no-nl-file-3.sh
2159 #1 printf '%s\n' 'oneliner, no \n'
2160 OK: 1 of 1 test passed
2161 $
2162 ```
2163
2164 Commands whose output has no `\n`
2165
2166 ```
2167 $ ./clitest dev/test/no-nl-command.sh
2168 #1 printf 'ok\n'
2169 #2 printf 'fail'
2170 --------------------------------------------------------------------------------
2171 [FAILED #2, line 6] printf 'fail'
2172 @@ -1 +1 @@
2173 -fail
2174 +fail
2175 \ No newline at end of file
2176 --------------------------------------------------------------------------------
2177 #3 printf 'ok\nok\nfail'
2178 --------------------------------------------------------------------------------
2179 [FAILED #3, line 8] printf 'ok\nok\nfail'
2180 @@ -1,3 +1,3 @@
2181 ok
2182 ok
2183 -fail
2184 +fail
2185 \ No newline at end of file
2186 --------------------------------------------------------------------------------
2187 #4 printf 'ok\n'
2188 #5 printf 'fail'
2189 --------------------------------------------------------------------------------
2190 [FAILED #5, line 17] printf 'fail'
2191 @@ -1 +1 @@
2192 -fail
2193 +fail
2194 \ No newline at end of file
2195 --------------------------------------------------------------------------------
2196 #6 printf 'ok'; echo
2197 #7 printf 'ok'
2198
2199 FAIL: 3 of 7 tests failed
2200 $
2201 ```
2202
2203 ## And now, the colored output tests
2204
2205 ```
2206 $ ./clitest --color yes --first dev/test/fail-2.sh
2207 #1 echo ok
2208 --------------------------------------------------------------------------------
2209 [FAILED #1, line 1] echo ok
2210 @@ -1 +1 @@
2211 -fail
2212 +ok
2213 --------------------------------------------------------------------------------
2214 $
2215 ```
0 # clitest Examples
1
2 Here's some simple examples to show you how a test file looks like.
3
4
5 ## Pure CLI Tests
6
7 Take a look at the `.txt` files. They're just like a shell session
8 snapshot. You have the `$ ` prompt, the command to be executed, and
9 the results.
10
11 ```
12 $ echo "Hello World"
13 Hello World
14 $
15 ```
16
17 To test these files, just call `clitest` with no options.
18
19 ```
20 $ clitest intro.txt
21 #1 echo "Hello World"
22 #2 cd /tmp
23 #3 pwd
24 #4 cd "$OLDPWD"
25 OK: 4 of 4 tests passed
26 $
27 ```
28
29 ### Easily create your own test files:
30
31 1. Go to your terminal
32 2. Set your prompt accordingly: `PS1='$ '`
33 3. Type and run the desired commands
34 4. Copy & paste it all into a text file
35 5. Done
36
37
38
39 ## Documentation Tests
40
41 Now take a look at the `.md` files. They're normal Markdown documents
42 (with titles, paragraphs, code blocks), created to be read by humans
43 (after HTML conversion).
44
45 Inside the code blocks there are examples of command lines and their
46 results. `clitest` can extract and run these commands for you! Now you
47 can guarantee that all your examples are correct.
48
49 ```
50 $ clitest --prefix tab cut.md
51 #1 echo "one:two:three:four:five:six" | cut -d : -f 1
52 #2 echo "one:two:three:four:five:six" | cut -d : -f 4
53 #3 echo "one:two:three:four:five:six" | cut -d : -f 1,4
54 #4 echo "one:two:three:four:five:six" | cut -d : -f 4,1
55 #5 echo "one:two:three:four:five:six" | cut -d : -f 1-4
56 #6 echo "one:two:three:four:five:six" | cut -d : -f 4-
57 OK: 6 of 6 tests passed
58 $
59 ```
60
61 Note that since the code blocks in these Markdown documents are
62 prefixed by a tab, you must use the `--prefix` option.
63
64 Even this `README.md` file you're reading is testable. No options
65 needed, since the code blocks here do not use prefixes.
66
67
68 ## Play Around
69
70 Run the tests, change the expected output to force a test fail, use
71 the `--list-run` option, ...
0 The numeric ranges of the Unix command "cut"
1 ============================================
2
3 Use single numbers to extract one specific field:
4
5 $ echo "one:two:three:four:five:six" | cut -d : -f 1
6 one
7 $ echo "one:two:three:four:five:six" | cut -d : -f 4
8 four
9
10 Use commas to inform more than one field:
11
12 $ echo "one:two:three:four:five:six" | cut -d : -f 1,4
13 one:four
14
15 Note that inverting the order will *not* invert the output:
16
17 $ echo "one:two:three:four:five:six" | cut -d : -f 4,1
18 one:four
19
20 Use an hyphen to inform a range of fields, from one to four:
21
22 $ echo "one:two:three:four:five:six" | cut -d : -f 1-4
23 one:two:three:four
24
25 If you omit the second range number, it matches until the last:
26
27 $ echo "one:two:three:four:five:six" | cut -d : -f 4-
28 four:five:six
29
30 cut is cool, isn't it?
31
32 > Note: To automatically test all the shell commands in this article,
33 > just run: `clitest --prefix tab cut.md`
0 $ echo "one:two:three:four:five:six" | cut -d : -f 1
1 one
2 $ echo "one:two:three:four:five:six" | cut -d : -f 4
3 four
4 $ echo "one:two:three:four:five:six" | cut -d : -f 1,4
5 one:four
6 $ echo "one:two:three:four:five:six" | cut -d : -f 4,1
7 one:four
8 $ echo "one:two:three:four:five:six" | cut -d : -f 1-4
9 one:two:three:four
10 $ echo "one:two:three:four:five:six" | cut -d : -f 4-
11 four:five:six
12 $
0 $ echo "Hello World"
1 Hello World
2 $
0 # How to install txt2tags v2.6
1
2 > This file is an example of a technical “how to” document
3 > that can also be automatically tested:
4 > `clitest --prefix tab install-software.md`
5
6
7 ## 1. Prepare
8
9 First, move to the temporary directory, where we will download, extract
10 and test the txt2tags package before installing it to the system.
11
12 $ cd /tmp
13
14
15 ## 2. Download
16
17 Download the .tgz file for the version 2.6, directly from Google Code.
18
19 $ url="http://txt2tags.googlecode.com/files/txt2tags-2.6.tgz"
20 $ curl -O -s -S "$url"
21
22
23 ## 3. Verify
24
25 Let's verify if the downloaded package is not corrupted, by checking
26 the file size and the total number of files inside the tgz.
27
28 $ du -h txt2tags-2.6.tgz
29 532K txt2tags-2.6.tgz
30 $ tar tzf txt2tags-2.6.tgz | sed -n '$='
31 545
32
33 > Note: Using `sed` to count lines because the output format of `wc -l`
34 > differs between implementations, regarding leading blank spaces.
35
36
37 ## 4. Extract
38
39 Since the download is ok, now we can extract the package's files. If
40 `tar` shows no message, it's because everything went fine and all the
41 files were extracted.
42
43 $ tar xzf txt2tags-2.6.tgz
44
45 A new `txt2tags-2.6` directory was created. Let's enter into it and
46 list the main files, just to be sure.
47
48 $ cd txt2tags-2.6
49 $ ls -1F
50 COPYING
51 ChangeLog
52 README
53 doc/
54 extras/
55 po/
56 samples/
57 test/
58 txt2tags*
59
60
61 ## 5. Test
62
63 The main `txt2tags` file is executable? Python is installed? Python
64 version is compatible with the program? So many questions... But a
65 simple command answers them all.
66
67 $ ./txt2tags -V
68 txt2tags version 2.6 <http://txt2tags.org>
69
70 If the version was shown, it's a proof that the program was run
71 successfully: Python is installed and it's compatible.
72
73 ## 6. Install
74
75 By default, the program is installed in the `~/bin` user directory.
76 Usually this directory is already there, but let's play safe and create
77 it if necessary.
78
79 $ test -d ~/bin || mkdir ~/bin
80
81 The install process itself is just a simple file copy.
82
83 $ cp txt2tags ~/bin/
84
85 Now just a final test, executing the program directly from `~/bin`.
86
87 $ ~/bin/txt2tags -V
88 txt2tags version 2.6 <http://txt2tags.org>
89
90 Ok, we're done.
0 $ cd /tmp
1 $ url="http://txt2tags.googlecode.com/files/txt2tags-2.6.tgz"
2 $ curl -O -s -S "$url" # download
3 $ du -h txt2tags-2.6.tgz # verify size
4 532K txt2tags-2.6.tgz
5 $ tar tzf txt2tags-2.6.tgz | sed -n '$=' # verify number of files
6 545
7 $ tar xzf txt2tags-2.6.tgz # extract
8 $ cd txt2tags-2.6
9 $ ls -1F # list contents
10 COPYING
11 ChangeLog
12 README
13 doc/
14 extras/
15 po/
16 samples/
17 test/
18 txt2tags*
19 $ ./txt2tags -V # test execution
20 txt2tags version 2.6 <http://txt2tags.org>
21 $ test -d ~/bin || mkdir ~/bin # create folder if needed
22 $ cp txt2tags ~/bin/ # install to user folder
23 $ ~/bin/txt2tags -V # final test
24 txt2tags version 2.6 <http://txt2tags.org>
25 $
0 $ echo "Hello World"
1 Hello World
2 $ cd /tmp
3 $ pwd
4 /tmp
5 $ cd "$OLDPWD"
6 $
0 $ echo "Hello World"
1 Hola Mundo
2 $