GoldilocksShell¶
What Is GoldilocksShell?¶
What good are tests without a way to run them? While you can certainly write your own class for interacting with Golidlocks, we wanted to provide a quick and easy way to run your tests and suites interactively.
Assuming you have one or more TestSuites configured (see Setting Up Suites), you can set up a GoldilocksShell relatively quickly!
Setting Up¶
We import GoldilocksShell with…
#include "goldilocks/shell.hpp"
We start by defining a new GoldilocksShell
object instaces somewhere in our
code, preferably in our main()
function.
GoldilocksShell* shell = new GoldilocksShell(">> ");
The part in quotes is the prompt string, which will appear at the start of every line where the user can type in the interactive terminal.
Now we need to register all of our Goldilocks TestSuite
classes with the
shell. Note that we don’t need to create instances of these ourselves, but
merely pass the class as a type. We also need to specify the name of the
suite in quotes: this name will be what is used to identify it in the shell.
shell->register_suite<TestSuite_Brakes>("s-brakes");
shell->register_suite<TestSuite_Hologram>("s-hologram");
shell->register_suite<TestSuite_TimeRotor>("s-timerotor");
shell->register_suite<TestSuite_CloisterBell>("s-cloisterbell");
That is it! We are now ready to use GolidlocksShell.
Interactive Mode¶
As long as we’re running in the command line, we can hand over control to the GoldilocksShell via a single line of code…
shell->interactive();
The GoldilocksShell will immediately launch and take over the terminal, using IOSqueak.
Commands¶
In Interactive Mode, you are given a complete shell for executing Golilocks tests and suites.
Note
This initial version of GoldilocksTester does not offer support for the conventional shell commands, including history or arrow-key navigation. We’ll be adding these in a later version.
To get help at any point, run help
. To quit, type exit
.
In this guide, we’ll be using the default GoldilocksShell prompt symbol,
:
. This may be different for your project, depending on how you
configured GoldilocksShell.
Listing and Loading Suites and Tests¶
When you first start GoldilocksShell, no tests have been loaded into memory. However, all the suites you specified are ready to be loaded.
To see all the available suites, run listsuites
:
: listsuites
s-brakes: TARDIS Brakes Tests
s-hologram: TARDIS Holographic Interface Tests
s-timerotor: TARDIS Time Rotor Tests
s-cloisterbell: TARDIS Cloister Bell Tests
Thus, before you can run a test or suite, you must first load the suite containing it:
: load s-brakes
TARDIS Brakes Tests
Suite loaded.
Now you can run the list
command to see all the loaded tests:
: list
t-brakes-engage: TARDIS Brakes: Engage Brakes
t-brakes-warn: TARDIS Brakes: No Brakes Warning
t-brakes-disengage: TARDIS Brakes: Disengage Brakes
t-brakes-fail: TARDIS Brakes: Brake Failure Protocol
t-brakes-pressure: TARDIS Brakes: Brake Pressure Test
Note
If list
does not show any tests, be sure you’ve loaded at least
one suite first.
If you just want to load all suites, simply run the load
command without
any arguments. It will ask you to confirm your choice:
: load
Load ALL test suites? (y/N) y
TARDIS Brakes Tests loaded.
TARDIS Holographic Interface Tests loaded.
TARDIS Time Rotor Tests loaded.
TARDIS Cloister Bell Tests loaded.
You can find out more information about any test using the about
command:
: about t-brakes-engage
TARDIS Brakes: Engage Brakes
Ensures the controls are capable of engaging the brakes.
Running Tests and Suites¶
It is possible to run any test using the run
command. This command always
asks you to confirm before continuing:
: run t-brakes-engage
Run test TARDIS Brakes: Engage Brakes [t-brakes-engage]? (y/N) y
===== [TARDIS Brakes: Engage Brakes] =====
Pass 1 of 1
TEST COMPLETE
Optionally, you can repeat a test multiple times by specifying the number of times to repeat it.
: run t-brakes-engage 5 Run test TARDIS Brakes: Engage Brakes [t-brakes-engage]? (y/N) y ===== [TARDIS Brakes: Engage Brakes] ===== Pass 1 of 5 Pass 2 of 5 Pass 3 of 5 Pass 4 of 5 Pass 5 of 5 TEST COMPLETE
You can also run an entire suite in one step:
: run s-brakes
Run test suite TARDIS Brakes Tests [s-brakes]? (y/N) y
===== [TARDIS Brakes Tests] =====
===== [TARDIS Brakes: Engage Brakes] =====
Pass 1 of 1
TEST COMPLETE
===== [TARDIS Brakes: No Brakes Warning] =====
Pass 1 of 1
TEST COMPLETE
===== [TARDIS Brakes: Disengage Brakes] =====
Pass 1 of 1
TEST COMPLETE
===== [TARDIS Brakes: Brake Failure Protocol] =====
Pass 1 of 1
TEST COMPLETE
===== [TARDIS Brakes: Brake Pressure Test] =====
Pass 1 of 1
TEST COMPLETE
SUITE COMPLETE
Note
If you specify a repeat number for running a suite, it will be ignored.
Benchmarking¶
Golidlocks supports comparative benchmarking. There are two ways to run such a benchmark.
The first method requires a comparative test to be specified within a suite (see load_tests()). If you’ve done this, you can benchmark the test and its comparative, and output the complete benchmark stats:
: benchmark t-brakes-engage
Run comparative benchmark between TARDIS Brakes: Engage Brakes [t-brakes-engage] and TARDIS Brakes: Handbrake? (y/N) at 100 repetitions? (y/N) y
======================
| BENCHMARKER |
======================
Upon completion it will display the complete benchmarker stats (see Benchmarker Output).
You can also specify the number of times to run the benchmarker (the default is 100):
: benchmark t-brakes-engage 1000
Run comparative benchmark between TARDIS Brakes: Engage Brakes [t-brakes-engage] and TARDIS Brakes: Handbrake? (y/N) at 1000 repetitions? (y/N) y
======================
| BENCHMARKER |
======================
You can also run a comparative benchmark on any two tests using the compare
function. It functions in much the same way, except that you specify two
tests instead of one, and then the optional repetition count:
: compare t-brakes-engage t-brakes disengage 500
Run comparative benchmark between TARDIS Brakes: Engage Brakes [t-brakes-engage] and TARDIS Brakes: Disengage [t-brakes-disengage]? (y/N) at 1000 repetitions? (y/N) y
======================
| BENCHMARKER |
======================
Command Line Argument Mode¶
Invocation¶
GoldilocksShell is also designed to handle the same input arguments as your
typical int main()
, which allows you to invoke the shell using
command-line arguments.
This is especially useful for integrating Goldilocks into a Continuous
Integration [CI] system, such as Jenkins. If the specified tests and suites
are successful, the program will exit with code 0
; failures will cause the
program to exit with code 1
.
To use this feature, you must simply pass the argument count and argument
array to the GoldilocksShell’s command()
function. It handles its own
argument parsing.
int main(int argc, char* argv[])
{
// ...setup code here...
// If we got command-line arguments...
if(argc > 1)
{
return shell->command(argc, argv);
}
return 0;
}
Skipping Arguments¶
If you accept other arguments via command-line, you may ask GoldilocksShell to skip those. Just specify the number of arguments to skip in the third argument.
Important
GoldilocksShell already knows to skip the first argument, which is the program invocation. You only need to tell it how many extra arguments to skip.
For example…
// myprogram --goldilocks --run sometest
int main(int argc, char* argv[])
{
// ...setup code here...
// If we're supposed to invoke Goldilocks.
if(argc > 1 && strcmp(argv[1], "--goldilocks") == 0)
{
// Asking GoldilocksShell to skip one argument...
return shell->command(argc, argv, 1);
// Now it will only process arguments starting from ``--run``...
}
return 0;
}
Usage¶
GoldilocksShell’s command line interface accepts multiple arguments, which are used to load and run tests, suites, and benchmarks. Commands are always run from left to right, in order.
The basic commands are as follows:
--help
displays help.--listsuites
lists all available suites.--load suite
loads the suitesuite
.--list
lists all loaded tests.--run item
runs the test or suiteitem
.--benchmark item
benchmarks the testitem
.
Important
The command line does not include the compare
function,
nor the ability to specify the number of test repetitions.
Ordinarily, to run a test, you must first load the suite containing it. However, for the sake of convenience, if you don’t explicitly load any tests in the command, it will just load all suites. Thus…
$ tester --run t-brakes-engage
…will just load all the suites before attempting to run the test
t-brakes-engage
.
If you want to only load a single suite, perhaps to see what tests it contains,
just include the --load
argument. (Remember, if you don’t explicitly load
any suites, all the suites will be loaded.)
$ tester --load s-brakes --list
Warning
Each command only accepts one argument! If you want to load
multiple suites, you must precede each suite ID with the --load
argument.
Arguments are run in order, from left to right, and the program doesn’t exit
until all of them are finished. This means you can run multiple tests in one
command; success will only be reported (exit code 0
) if all the tests
pass.
$ tester --load s-brakes --run t-brakes-engage --run t-brakes-disengage
The above command, after loading only the specified suite, will run the
requested tests. If they both succeed, the program will exit reporting
success (exit code 0
).
Warning
Each command only accepts one argument! If you want to load
multiple suites, you must precede each suite ID with the --load
argument.
We can also run benchmarks from the command line. --benchmark
bases its
success/fail condition on the Baby Bear comparison; success means either
(a) the main test is faster than its comparative, or (b) the two tests are
roughly identical in performance (“dead heat”).
$ tester --load s-brakes --benchmark t-brakes-engage
A Complete Example¶
Let’s tie all this together. Here’s an example of a complete int main()
function set up to use GoldilocksShell, as outlined in the previous sections.
int main(int argc, char* argv[])
{
GoldilocksShell* shell = new GoldilocksShell(">> ");
shell->register_suite<TestSuite_Brakes>("s-brakes");
shell->register_suite<TestSuite_Hologram>("s-hologram");
shell->register_suite<TestSuite_TimeRotor>("s-timerotor");
shell->register_suite<TestSuite_CloisterBell>("s-cloisterbell");
// If we got command-line arguments...
if(argc > 1)
{
return shell->command(argc, argv);
}
else
{
// Shift control to the interactive console.
shell->interactive();
}
// Delete our GoldilocksShell.
delete shell;
shell = 0;
return 0;
}