Parrot4Newbies: Test Suite

The second post in my Parrot4Newbies series, today I am going to talk about Parrot's test suite. This is another great way for the average new user to get a deeper understanding about Parrot's capabilities because the test suite is, in theory, a comprehensive exercise of Parrot.

Parrot's test suite has historically been written in Perl 5, with a series of modules derived from Test::More and Test::Builder and others with some custom methods to handle creating, compiling, and executing PIR and PASM code. Now don't get me wrong, we do like Perl 5. However, we want to get rid of as much Perl 5 from the Parrot repository as we possibly can. For one thing we want to reduce the number of dependencies. For another, we have to look forward to a future where Perl 5 is actually running on Parrot, which would create a very fun bootstrapping problem. Maybe that's just wishful thinking on my part, but it's a fun goal to have in mind (and a fun project for an adventurous team of new developers to attempt!)

Run Tests, Report Problems

The fastest way to get started with tests is to get the current version of Parrot from SVN, and run this little script:

perl Configure.pl [args]
make
make test

The "test" target runs the core functionality tests, the documentation tests, and the coding standards tests, plus a few others. This can take quite a long time to execute, so running the "make coretest" is probably more attractive for most coders. The "coretest" target only runs tests of the core functionality.

If you're looking to really optimize and run tests quickly, you will want this incantation:

make -j9 [testname] TEST_JOBS=5

Where [testname] can be "test", "coretest", "fulltest", "distro_tests", or any of the other test targets. This particular command executes testing in 5 threads in parallel, which can go very very quickly on some systems.

When you run tests and see failures, you can either submit a bug report to trac, or you can be ambitious by fixing it and submitting a patch. Patches are always very nice to have, but even reports of failures (including information about your system and build options) are very good too.

Upload Smolder Reports

The smolder service maintains an online log of test progress, so we can see what tests are doing on each platform. If you're running tests regularly, please consider "make smolder_tests". This will run through the test suite and upload the results to the smolder server so our developers can keep track of them.

Even better would be to set up a smolder slave bot. Set up a script on your system to run this sequence at intervals:

cd $HOME &&
svn co https://svn.parrot.org/parrot/trunk parrot-smoke &&
cd parrot-smoke &&
perl Configure.pl &&
make &&
make smoke

This is especially important if you're on a rare system (something that isn't x86+Windows, x86+Linux, or x86+Darwin). We can never have enough reports from rare systems.

Convert Tests to PIR

Many of our tests are written in old-style PASM, and many of them are still managed by Perl 5 scripts. We need to rewrite all these test files to use pure PIR instead. Here's an example of a Perl 5 test:

use strict;
use warnings;
use Test::More
use Parrot::Test tests > 1;
pasm_output_is( <<'CODE', <<OUTPUT, "gt_ic_i_ic" );
     set I0, 10
     gt 11, I0, ok1
     print "nok gt\n"
ok1:
     print "ok 1\n"
     gt 9, I0, nok1
     print "ok 2\n"
     branch ok2
nok1:
     print "nok gt 2\n"
ok2:
     end
CODE
ok 1
ok 2
OUTPUT

And here is that same test rewritten in PIR:

.sub main :main
plan(2)
gt_ic_i_ic_1()
gt_ic_i_ic_2()
.end

.sub gt_ic_i_ic_1
I0 = 10
if 11 > $I0 goto ok_1
ok(0, "gt_ic_i_ic1")
goto end_1
ok_1:
ok(1, "gt_ic_i_ic1")
.return()
.end

.sub gt_ic_i_ic_2
$I0 = 10
if 9 > $I0 goto ok_2
ok(1, "gt_ic_i_ic1")
goto end_2
ok_2:
ok(0, "gt_ic_i_ic2")
.return()
.end

So the transformation isn't entirely straight forward but the end result is pure PIR, not a hard-to-read mixture of PASM and Perl 5. Notice that I am not line-for-line translating the PASM into PIR, I rearrange it a lot to make it more readable. The end result, the features being tested, are absolutely the same.

PMC Testing

As the policy stands right now, every file in src/pmc/ should have an associated file in t/pmc/. There is actually a test to verify that we have all these necessary tests! Just having these test files isn't enough, we need to make sure the tests are actually giving the PMCs a good workout. Look through the various tests and make sure each PMC is well tested. Some PMCs such as Integer implement many VTABLEs, and we want at least one test to exercise each of them. Writing tests like this helps not only to figure out what all the core PMC types do, but you're also forced to trace through some code to figure out where the various VTABLEs are called from so you can figure out how to test them.

Conclusion

The test suite is a key part of the Parrot development process. It helps us to find small regressions and bug earlier so they don't grow and fester into larger bugs later. The more comprehensive our test suite is, the more comfortable we can be making Parrot releases because we know that HLLs and other users of Parrot that use the features we test for will work as expected. It's also a great way for new developers to get involved and start getting an idea about the capabilities, current state, and limitations of Parrot. Once you see for yourself the areas where Parrot needs some work, you'll be more able to start making the necessary fixes yourself.