-
Notifications
You must be signed in to change notification settings - Fork 3
Expand file tree
/
Copy pathREADME
More file actions
1208 lines (880 loc) · 42.7 KB
/
Copy pathREADME
File metadata and controls
1208 lines (880 loc) · 42.7 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
NAME
Test::Inter - framework for more readable interactive test scripts
DESCRIPTION
This is another framework for writing test scripts. Much of the syntax
is loosely inspired by Test::More, and Test::Inter has most of it's
functionality, but it is not a drop-in replacement.
Test::More (and other existing test frameworks) suffer from two
weaknesses, both of which have prevented me from ever using them:
None offer the ability to access specific tests in
a reasonably interactive fashion, primarily for
debugging purposes
None offer the ability to write the tests in
whatever format would make the tests the most
readable
The way I write and use test scripts, existing Test::* modules are not
nearly as useful as they could be.
Test scripts written using Test::More work fine when running as part of
the test suite, but debugging an individual test requires extra steps,
and the tests themselves are not as readable as they should be.
INTERACTIVE EXECUTION
One requirement that I have of a test framework is the ability to
interact with it.
I do most of my debugging using test scripts. When I find a bug, I write
a test case for it (typically by adding it to an existing test script)
and then debug it using the test script. Then I leave the test there to
ensure that the bug won't come back (hopefully).
Since I use test scripts in a very interactive way (often in the
debugger), I want to be able to do the following trivially:
Easy access to a specific test or tests
I'd like to be able to run only a single test, or a subset of tests.
Easily set breakpoints in the debugger
Setting a breakpoint in the debugger to run up to the start of the
Nth test is one of the most common tasks I want to do when I'm
debugging a failed test.
To illustrate the first point, in Test::More, a series of tests might be
specified in a test script as shown in the following example (line
numbers added for convenience):
...
100: # test 1
101: $result = func("apples","bushels");
102: is($result, "enough");
103:
104: # test 2
105: $result = func("grapefruit","tons");
106: is($result, "enough");
107:
108: # test 3
109: $result = func("oranges","boatloads");
110: is($result, "insufficient");
111:
112: # tests 4-6
113: foreach my $arg (qw(pears plums pineapple)) {
114: $result = func($arg,"boxes");
115: is($result, "enough");
116: }
...
Say you ran the test suite, and test 3 failed. To debug it you have to
open up the test script, find the 3rd test, and set the appropriate
breakpoint. In this case, you'll want to break at line 109.
None of these steps are impossible of course, but it will take some time
to get it right. It becomes harder when there are lots of tests (imagine
that you want to test the 117th test instead of the 3rd test) or when
tests are wrapped up in loops, embedded in subroutines, or other similar
situations.
As an example, what if it's the 5th test that fails in the example
above. Now the break point will be a conditional one, so you have to
figure out not only the line, but the condition the appropriate state
during that test. In this case, you need to stop at line 114 when $arg
is 'plums'.
Wouldn't it be far better to set a break point in func when the Nth test
is reached? With Test::Inter, you can.
So for the above script, the debugger commands that you would use to
debug the 3rd test are:
Test::More : b 109
Test::Inter: b func ($::TI_NUM==3)
and the 5th test are:
Test::More : b 114 ($arg eq 'plums')
Test::Inter: b func ($::TI_NUM==5)
It would also be nice to be able to skip the first two tests... perhaps
they take a long time to run, and I want to get right to work on test 3.
You can do this easily too by setting the $::TI_START variable.
There are some other variables that can be used to specify which test or
tests to run described in the "TEST::INTER VARIABLES" section below.
The other thing I want to do when I run the test scripts interactively
is to see more information which will assist in debugging a failed test.
This can be controlled with variables such as TI_QUIET, TI_MODE, and
TI_WIDTH described below in the "TEST::INTER VARIABLES" section.
READABLE TESTS
The other feature that I wanted in a test suite is the ability to define
the tests in a format that is natural and readable FOR THE TESTS. In
almost every case, it is best to think of a test script as consisting of
two separate parts: a script part, and a test part.
The script part of a test script is the least important part! It's
usually fairly trivial, rarely needs to be changed, and is not the focus
of the test script.
The tests part of the script IS the important part, and these should be
expressed in a form that is natural to them, easy to maintain, easy to
read, and easy to modify, and none of these should involve modifying the
script portion of the test script in general. Because the content of the
tests is the important part of the script, the emphasis should be in
making them more readable, even at the expense of the script portion. As
a general rule, if the script portion of the test script obscures the
tests in any way, it's not written correctly!
The solution to this is well understood, and is common to many other
systems where you are mixing two "languages". The task of correctly
specifying both the tests and the test script is virtually identical to
the task of creating a PHP script which consists of a mixture of PHP and
HTML, or the task of creating a template file using some templating
system where the file consists of a mixture of text to be displayed and
templating commands. It is well understood in each of these cases that
the more the two "languages" are interwoven, the less readable both are,
and the harder it is to maintain. The more you are able to separate the
two, the easier both are to read and maintain.
As often as possible, I want the tests to be written in some sort of
text format which can be easily viewed and modified (usually as a simple
table) with no perl commands interspersed. I want to the freedom to
define the tests in one section (a long string, the DATA section, or
even in a separate file) which is easily readable. This may introduce
the necessity of parsing it, but it makes it significantly easier to
maintain the tests.
This flexibility makes it much easier to read the tests (as opposed to
the script) which is the fundamental content of a test script.
Looking again at the example test script, you can see that there is far
too much perl interspersed with the tests.
It's difficult to read the tests individually in this script because
there is too much perl code among them, and virtually impossible to look
at them as a whole.
It is true that looking at this particular example, it is very simple...
but the script ISN'T the content you're interested in (and bear in mind
that many test scripts are nowhere near this simple). The REAL content
of this script are the tests, which consist of the function arguments
and the expected result. Although it's not impossible to see each of
these in the script above, it's not in a format that is conducive to
studying the tests, and especially not for examining the list of tests
as a whole.
Now, look at an alternate way of specifying the tests using this module:
$tests = "
apples bushels => enough
grapefruit tons => enough
oranges boatloads => insufficient
pears boxes => enough
plums boxes => enough
pineapple boxes => enough
";
$o->tests(tests => $tests,
func => \&func);
Here, it's easy to see the list of tests, and adding additional tests is
a breeze.
CREATING A TEST
This module supports a number of methods for defining tests, so you can
use whichever one is most convenient (including methods that are
identical to Test::More if that really is the best method).
Every test may have several pieces of information:
A name
Every test is automatically assigned a number, but it may be useful
to specify a name of a test (which is actually a short description
of the test). Whenever a test result is reported, the name will be
given (if one was specified).
The name may not have a '#' in it.
The name is completely optional, but can serve to make the results
more readable.
An expected result
In order to test something, you need to know what result was
expected (or in some cases, what result was NOT expected).
A function and arguments OR a result
You also need to know the results that you're comparing to the
expected results.
This can be obtained by simply working with a set of results, or a
function name and a set of arguments to pass to it.
Conditions
It is useful to be able to specify state information at the start of
the test suite (for example, to see if certain features are
available), and some tests may only run if those conditions are met.
If no conditions are set for a test, it will always run.
Todo tests
Some tests may be marked as 'todo' tests. These are test which are
allowed to fail (meaning that they have been put in place for an
as-yet unimplemented feature). Since it is expected that the test
will fail, the test suite will still pass, even if these tests fail.
The tests will still run and if they pass, a message is issued
saying that the feature is now implemented, and the tests should be
graduated to non-todo state.
BASE METHODS
new
$o = new Test::Inter [$name] [%options];
This creates a new test framework. There are several options which
may be used to specify which tests are run, how they are run, and
what output is given.
The entire test script can be named by passing in $name.
All options can be set in four different ways.
First, you can pass in a hash of OPT = VAL> pairs in the new method.
So, to set the start option, the %options) hash would contain:
start => VALUE
Second, you can set an environment variable. This will override any
value passed in the first way. The environment variable is named
TI_XXX where XXX is the fully capitalized option. So:
$ENV{TI_START} = VALUE
The third method, which overrides the previous two, is to set a
global variable. It is also named TI_XXX in the main namespace, so
to set it this way, set:
$::TI_START = VALUE
The final method is to call one of the methods below and these
override all other methods.
Each of the allowed options are described below in the following
base methods:
start
end
testnum
plan
abort
quiet
mode
skip_all
width
use_lib
version
$o->version();
Returns the version of the module.
encoding
$o->encoding($encoding);
$encoding is any value that can be passed as an encoding to perl's
Encode::decode function.
Use this if your test strings contain characters in other encodings.
start
$o = new Test::Inter 'start' => $N;
$o->start($N)
To define which test you want to start with, set the start option as
described in the new method above.
When the start test is defined, most tests numbered less than N are
completely ignored. If the tests are being run quietly (see the
quiet method below), nothing is printed out for these tests.
Otherwise, a skip message is printed out.
One class of tests IS still executed. Tests run using the require_ok
or use_ok methods (to test the loading of modules) are still run.
If no value (or a value of 0) is used, tests run from the first
test.
end
$o = new Test::Inter 'end' => $M;
$o->end($M);
To define which test you want to end with, set the end option as
described in the new method above.
When the end test is defined, all tests numbered more than M are
completely ignored. If the tests are being run quietly (see the
quiet method below), nothing is printed out for these tests.
Otherwise, a skip message is printed out.
If no value is given, it defaults to 0 (which means that all
remaining tests are run).
testnum
$o = new Test::Inter 'testnum' => $N;
$o->testnum($N);
To run only a single test, set the testnum option as described in
the new method above.
It is equivalent to setting both the start and end tests to $N.
plan
done_testing
$o = new Test::Inter 'plan' => $N;
$o->plan($n);
$o->done_testing();
$o->done_testing($n);
The TAP API (the 'language' used to run a sequence of tests and see
which ones failed and which ones passed) requires a statement of the
number of tests that are expected to run.
This statement can appear at the start of the test suite, or at the
end.
If you know in advance how many tests should run in the test script,
you can set the plan option as described in the new method above to
the number of tests.
If you know how many tests should run at the end of the test script,
you can pass in a non-zero integer to the done_testing method.
Frequently, you don't really care how many tests are in the script
(especially if new tests are added on a regular basis). In this
case, you still need to include a statement that says that the
number of tests expected is however many were run. To do this, call
the done_testing method with no argument.
NOTE: if the plan method is used, it MUST be used before any tests
are run (including those that test the loading of modules). If the
done_testing method is used, it MUST be called after all tests are
run. You must specify a plan or use a done_testing statement, but
you cannot do both.
It is NOT strictly required to set a plan if the script is only run
interactively, so if for some reason this module is used for test
scripts which are not part of a standard perl test suite, the plan
and done_testing statements are optional. As a matter of fact, the
script will run just fine without them... but a perl installer will
report a failure in the test suite.
abort
$o = new Test::Inter 'abort' => 0/1/2;
$o->abort(0/1/2);
To define how you want a failure to be treated, set the abort option
as described in the new method above. The abort option can take a
value of 0, 1, or 2.
If this is set to 1, the test script will run unmodified until a
test fails. At that point, all remaining tests will be skipped. If
it is set to 2, the test script will run until a test fails at which
point it will exit with an error code of 1. With a value of 0,
failed tests will be reported, but the script will continue.
In both cases, todo tests will NOT trigger the abort behavior.
quiet
$o = new Test::Inter 'quiet' => 0/1/2;
$o->quiet(0/1/2);
To define how you want failures to be reported, set the quiet option
as described in the new method above. The quiet option can take a
value of 0, 1, or 2.
If this is set to 0 (the default), all information will be printed
out. If it is set to 1, some optional information will not be
printed. If it is set to 2, all optional information will not be
printed.
mode
$o = new Test::Inter 'mode' => MODE;
$o->mode(MODE);
Test::Inter scripts can be run in either an interactive mode, or as
part of a test suite with different behaviors. To select the mode,
set the mode option as described in the new method above. The mode
option can take a value of 'inter' or 'test'.
When run in test mode, it prints out the results using the TAP
grammar (i.e. 'ok 1', 'not ok 3', etc.).
When run in interactive mode, it prints out results in a more human
readable format.
width
$o = new Test::Inter 'width' => WIDTH;
$o->width(WIDTH);
The width option can be set as described in the new method above.
WIDTH is the width of the terminal (for printing out failed test
information). It defaults to 80, but it can be set to any width (and
lines longer then this are truncated). If WIDTH is set to 0, no
truncation is done.
use_lib
$o = new Test::Inter 'use_lib' => VALUE;
$o->use_lib(VALUE);
$o->use_lib();
By default, the library included in the module distribution will be
added to the search path for modules, so a 'use MODULE' line should
find the version stored in this module distribution.
If VALUE is set to 'off', the search path will not be modified
automatically.
You may add the library path at a later time by calling:
$o->use_lib('on');
$o->use_lib();
Note: both calls must be used. The first sets the option, the second
actually modifies the search path.
skip_all
$o = new Test::Inter 'skip_all' => REASON;
$o->skip_all(REASON);
The skip_all option can be set as described in the new method above.
If this is set, the entire test script will be skipped for the
reason given. This must be done before any test is run, and before
any plan number is set.
The skip_all can also be called at any point during the script (i.e.
after tests have been run). In this case, all remaining scripts will
be skipped.
$o->skip_all(REASON,FEATURE,FEATURE,...);
$o->skip_all('',FEATURE,FEATURE,...);
This will skip all tests (or all remaining tests) unless all
<FEATURE>s are available. REASON can be entered as an empty string
and the reason the tests are skipped will be a message about the
missing feature.
feature
$o->feature($feature,$val);
This defines a feature. If $val is non-zero, the feature is
available. Otherwise it is not.
diag
note
$o->diag($message);
$o->note($message);
Both of these print an optional message. Messages printed with the
"note" method are always optional and will be omitted if the quiet
option is set to 1 or 2. Messages printed with the "diag" method are
optional and will not be printed if the quiet option is set to 2,
but they will be printed if the quiet method is set to 1.
testdir
$o->testdir();
$o->testdir('mod');
$o->testdir('lib');
Occasionally, it may be necessary to know the directory where
Test::Inter gets some of it's information. By default, the directory
containing the tests will be returned, but if the optional argument
'mod' is included, it will return the path to the module
distribution (which should include both a lib and t subdirerctory).
If the argument 'lib' is included, it will return the directory
where the libraries are stored.
METHODS FOR LOADING MODULES
Test scripts can load other modules (using either the perl "use" or
"require" commands). There are three different modes for doing this
which determine how this is done.
required
By default, this is used to test for a module that is required for
all tests in the test script.
Loading the module is treated as an actual test in the test suite.
The test is to determine whether the module is available and can be
loaded. If it can be loaded, it is, and it is reported as a
successful test. If it cannot be loaded, it is reported as a failed
test.
In the result of a failed test, all remaining tests will be skipped
automatically (except for other tests which load modules).
feature
In feature mode, loading the module is not treated as a test (i.e.
it will not print out an 'ok' or 'not ok' line. Instead, it will set
a feature (named the same as the module) which can be used to
determine whether other tests should run or not.
forbid
In a few very rare cases, we may want to test for a module but
expect that it not be present. This is the exact opposite of the
required mode.
Successfully loading the module is treated as a test failure. In the
event of a failure, all remaining tests will be skipped.
The methods available are:
require_ok
$o->require_ok($module [,$mode]);
This is used to load a module using the perl "require" function. If
$mode is not passed in, the default mode (required) is used to test
the existence of the module.
If $mode is passed in, it must be either the string 'forbid' or
'feature'.
If $mode is 'feature', a feature named $module is set if the module
was able to be loaded.
use_ok
$o->use_ok(@args [,$mode]);
This is used to load a module with "use", or check a perl version.
BEGIN { $o->use_ok('5.010'); }
BEGIN { $o->use_ok('Some::Module'); }
BEGIN { $o->use_ok('Some::Module',2.05); }
BEGIN { $o->use_ok('Some::Module','foo','bar'); }
BEGIN { $o->use_ok('Some::Module',2.05,'foo','bar'); }
are the same as:
use 5.010;
use Some::Module;
use Some::Module 2.05;
use Some::Module qw(foo bar);
use Some::Module 2.05 qw(foo bar);
Putting the use_ok call in a BEGIN block allows the functions to be
imported at compile-time and prototypes are properly honored. You'll
also need to load the Test::Inter module, and create the object in a
BEGIN block.
$mode acts the same as in the require_ok method.
METHODS FOR RUNNING TEST
There are several methods for running tests. The ok, is, and isnt
methods are included for those already comfortable with Test::More and
wishing to stick with the same format of test script. The tests method
is the suggested method though since it makes use of the full power of
this module.
ok
$o->ok(TESTS);
A test run with ok looks at a result, and if it evaluates to 0 (or
false), it fails. If it evaluates to non-zero (or true), it passes.
These tests do not require you to specify the expected results. If
expected results are given, they will be compared against the result
received, and if they differ, a diagnostic message will be printed,
but the test will still succeed or fail based only on the actual
result produced.
These tests require a single result and either zero or one expected
results.
To run a single test, use any of the following:
$o->ok(); # always succeeds
$o->ok($result);
$o->ok($result,$name);
$o->ok($result,$expected,$name);
$o->ok(\&func);
$o->ok(\&func,$name);
$o->ok(\&func,$expected,$name);
$o->ok(\&func,\@args);
$o->ok(\&func,\@args,$name);
$o->ok(\&func,\@args,$expected,$name);
If $result is a scalar, the test passes if $result is true. If
$result is a list reference, the test succeeds if the list contains
any defined values. If $result is a hash reference, the test
succeeds if the hash contains any key with a value that is not
"undef".
If "\&func" and "\@args" are passed in, then $result is generated by
passing @args to &func and behaves identically to the calls where
$result is passed in. If "\&func" is passed in but no arguments, the
function takes no arguments, but still produces a result.
If an expected value is passed in and the result does not match it,
a diagnostic warning will be printed, even if the test passes.
is
isnt
$o->is(TESTS);
$o->isnt(TESTS);
A test run with is looks at a result and tests to see if it is
identical to an expected result. If it is, the test passes.
Otherwise it fails. In the case of a failure, a diagnostic message
will show what result was actually obtained and what was expected.
A test run with isnt looks at a result and tests to see if the
result obtained is different than an expected result. If it is
different, the test passes. Otherwise it fails.
The is method can be called in any of the following ways:
$o->is($result,$expected);
$o->is($result,$expected,$name);
$o->is(\&func,$expected);
$o->is(\&func,$expected,$name);
$o->is(\&func,\@args,$expected);
$o->is(\&func,\@args,$expected,$name);
The isnt method can be called in exactly the same way.
As with the ok method, the result can be a scalar, hashref, or
listref. If it is a hashref or listref, the entire structure must
match the expected value.
tests
$o->tests($opt=>$val, $opt=>$val, ...)
The options available are described in the following section.
file
$o->file($func,$input,$outputdir,$expected,$name [,@args]);
Sometimes it may be easiest to store the input, output, and expected
output from a test in a text file. In this case, each line of output
will be treated as a single test, so the output and expected output
must match up exactly.
$func is a reference to a function which will produce a temporary
output file.
If $input is specified, it is the name of the input file. If it is
empty, no input file will be used. The input file can be fully
specified, or it can be relative to the test directory.
If $outputdir is passed in, it is the directory where the output
file will be written. It can be fully specified, or relative to the
test directory. If $outputdir is left blank, the temporary file will
be written to the test directory.
$expected is the name of a file which contains the expected output.
It can be fully specified, or it will be checked for in the test
directory.
$name is the name of this series of tests.
@args are extra arguments to pass to the test function.
The function will be called with the arguments:
&$func( [$input,] $output,@args);
$input is only passed in if it was passed in to this method. If no
input file is specified, nothing will be passed to the function.
$output is the name of a temporary file where the output will be
written to.
USING THE TESTS METHOD
It is expected that most tests (except for those that load a module)
will be run using the tests method called as:
$o->tests(%options);
The following options are available:
name
name => NAME
This sets the name of this set of tests. All tests will be given the
same name.
tests
func
expected
In order to specify a series of tests, you have to specify either a
function and a list of arguments, or a list of results.
Specifying the function and list of arguments can be done using the
pair:
func => \&FUNCTION
tests => TESTS
If the func option is not set, tests contains a list of results.
A list of expected results may also be given. They can be included
in the
tests => TESTS
option or included separately as:
expected => RESULTS
The way to specify these are covered in the next section SPECIFYING
THE TESTS.
feature
disable
feature => [FEATURE1, FEATURE2, ...]
disable => [FEATURE1, FEATURE2, ...]
The default set of tests to run is determined using the start, end,
and skip_all methods discussed above. Using those methods, a list of
tests is obtained, and it is expected that these will run.
The feature and disable options modify the list.
If the feature option is included, the tests given in this call will
only run if ALL of the features listed are available.
If the disable option is included, the tests will be run unless ANY
of the features listed are available.
skip
skip => REASON
Skip these tests for the reason given.
todo
todo => 0/1
Setting this to 1 says that these tests are allowed to fail. They
represent a feature that is not yet implemented.
If the tests succeed, a message will be printed notifying the
developer that the tests are now ready to promote to actual use.
SPECIFYING THE TESTS
A series of tests can be specified in two different ways. The tests can
be written in a very simple string format, or stored as a list.
Demonstrating how this can be done is best done by example, so let's say
that there is a function (func) which takes two arguments, and returns a
single value. Let's say that the expected output (and the actual output)
from 3 different sets of arguments is:
Input Expected Output Actual Output
----- --------------- -------------
1,2 a a
3,4 b x
5,6 c c
(so in this case, the first and third tests pass, but the 2nd one will
fail).
Specifying these tests as lists could be done as:
$o->tests(
func => &func,
tests => [ [1,2], [3,4], [5,6] ],
expected => [ [a], [b], [c] ],
);
Here, the tests are stored as a list, and each element in the list is a
listref containing the set of arguments.
If the func option is not passed in, the tests option is set to a list
of results to compare with the expected results, so the following is
equivalent to the above:
$o->tests(
tests => [ [a], [x], [c] ],
expected => [ [a], [b], [c] ],
);
If an argument (or actual result) or an expected result is only a single
value, it can be entered as a scalar instead of a list ref, so the
following is also equivalent:
$o->tests(
func => &func,
tests => [ [1,2], [3,4], [5,6] ],
expected => [ a, b, [c] ],
);
The only exception to this is if the single value is itself a list
reference. In this case it MUST be included as a reference. In other
words, if you have a single test, and the expected value for this test
is a list reference, it must be passed in as:
expected => [ [ \@r ] ]
NOT as:
expected => [ \@r ]
Passing in a set of expected results is optional. If none are passed in,
the tests are treated as if they had been passed to the ok method (i.e.
if they return something true, they pass, otherwise they fail).
The second way to specify tests is as a string. The string is a
multi-line string with each tests being separate from the next test by a
blank line. Comments (lines which begin with '#') are allowed, and are
ignored. Whitespace at the start and end of the line is ignored.
The string may contain the results directly, or results may be passed in
separately. For example, the following all give the same sets of tests
as the example above:
$o->tests(
func => &func,
tests => "
# Test 1
1 2 => a
# Test 2
3 4 => b
5 6 => c
",
);
$o->tests(
func => &func,
tests => "
1 2
3 4
5 6
",
expected => [ [a], [b], [c] ]
);
$o->tests(
func => &func,
tests => [ [1,2], [3,4], [5,6] ],
expected => "
a
b
c
",
);
$o->tests(
func => &func,
tests => "
1 2
3 4
5 6
",
expected => "
a
b
c
",
);
The expected results may also consist of only a single set of results
(in this case, it must be passed in as a listref). In this case, all of
the tests are expected to have the same results.
So, the following are equivalent:
$o->tests(
func => &func,
tests => "
1 2 => a b
3 4 => a b
5 6 => a b
",
);
$o->tests(
func => &func,
tests => "
1 2
3 4
5 6
",
expected => [ [a, b] ],
);
$o->tests(
func => &func,
tests => "
1 2
3 4
5 6
",
expected => "a b",
);
The number of expected values must either be 1 (i.e. all of the tests
are expected to produce the same value) or exactly the same number as
the number of tests.
The parser is actually quite powerful, and can handle multi-line tests,
quoted strings, and nested data structures.
The test may be split across any number of lines, provided there is not
a completely blank line (which signals the end of the test), so the
following are equivalent:
tests => "a b c",
tests => "a b
c",
Arguments (or expected results) may include data structures. For
example, the following are equivalent:
tests => "[ a b ] { a 1 b 2 }"
tests => [ [ [a,b], { a=>1, b=>2 } ] ]
Whitespace is mostly optional, but there is one exception. An item must
end with some kind of delimiter, so the following will fail:
tests => "[a b][c d]"
The first element (the list ref [a b]) must be separated from the second
element by the delimiter (which is whitespace in this case), so it must
be written as:
tests => "[a b] [c d]"
As already demonstrated, hashrefs and listrefs may be included and
nested. Elements may also be included inside parens, but this is
optional since all arguments and expected results are already treated as
lists, so the following are equivalent:
tests => "a b c"
tests => "(a b) c"
Although parens are optional, they may make things more readable, and
allow you to use something other than whitespace as the delimiter. Since
parens are actually ignored, a string '()' is also ignored, so do not
use empty parentheses.
If the character immediately following the opening paren, brace, or
bracket is a punctuation mark, then it is used as the delimiter instead
of whitespace. For example, the following are all equivalent:
[ a b c ]