Parrot's test suite has historically been written in Perl 5, with a series of modules derived from Test::More and Test::Builder and others with some custom methods to handle creating, compiling, and executing PIR and PASM code. Now don't get me wrong, we do like Perl 5. However, we want to get rid of as much Perl 5 from the Parrot repository as we possibly can. For one thing we want to reduce the number of dependencies. For another, we have to look forward to a future where Perl 5 is actually running on Parrot, which would create a very fun bootstrapping problem. Maybe that's just wishful thinking on my part, but it's a fun goal to have in mind (and a fun project for an adventurous team of new developers to attempt!)
Run Tests, Report Problems
The fastest way to get started with tests is to get the current version of Parrot from SVN, and run this little script:
perl Configure.pl [args]
The "test" target runs the core functionality tests, the documentation tests, and the coding standards tests, plus a few others. This can take quite a long time to execute, so running the "make coretest" is probably more attractive for most coders. The "coretest" target only runs tests of the core functionality.
If you're looking to really optimize and run tests quickly, you will want this incantation:
make -j9 [testname] TEST_JOBS=5
Where [testname] can be "test", "coretest", "fulltest", "distro_tests", or any of the other test targets. This particular command executes testing in 5 threads in parallel, which can go very very quickly on some systems.
When you run tests and see failures, you can either submit a bug report to trac, or you can be ambitious by fixing it and submitting a patch. Patches are always very nice to have, but even reports of failures (including information about your system and build options) are very good too.
Upload Smolder Reports
The smolder service maintains an online log of test progress, so we can see what tests are doing on each platform. If you're running tests regularly, please consider "make smolder_tests". This will run through the test suite and upload the results to the smolder server so our developers can keep track of them.
Even better would be to set up a smolder slave bot. Set up a script on your system to run this sequence at intervals:
cd $HOME &&
svn co https://svn.parrot.org/parrot/trunk parrot-smoke &&
cd parrot-smoke &&
perl Configure.pl &&
This is especially important if you're on a rare system (something that isn't x86+Windows, x86+Linux, or x86+Darwin). We can never have enough reports from rare systems.
Convert Tests to PIR
Many of our tests are written in old-style PASM, and many of them are still managed by Perl 5 scripts. We need to rewrite all these test files to use pure PIR instead. Here's an example of a Perl 5 test:
use Parrot::Test tests > 1;
pasm_output_is( <<'CODE', <<OUTPUT, "gt_ic_i_ic" );
set I0, 10
gt 11, I0, ok1
print "nok gt\n"
print "ok 1\n"
gt 9, I0, nok1
print "ok 2\n"
print "nok gt 2\n"
And here is that same test rewritten in PIR:
.sub main :main
I0 = 10
if 11 > $I0 goto ok_1
$I0 = 10
if 9 > $I0 goto ok_2
So the transformation isn't entirely straight forward but the end result is pure PIR, not a hard-to-read mixture of PASM and Perl 5. Notice that I am not line-for-line translating the PASM into PIR, I rearrange it a lot to make it more readable. The end result, the features being tested, are absolutely the same.
As the policy stands right now, every file in src/pmc/ should have an associated file in t/pmc/. There is actually a test to verify that we have all these necessary tests! Just having these test files isn't enough, we need to make sure the tests are actually giving the PMCs a good workout. Look through the various tests and make sure each PMC is well tested. Some PMCs such as Integer implement many VTABLEs, and we want at least one test to exercise each of them. Writing tests like this helps not only to figure out what all the core PMC types do, but you're also forced to trace through some code to figure out where the various VTABLEs are called from so you can figure out how to test them.
The test suite is a key part of the Parrot development process. It helps us to find small regressions and bug earlier so they don't grow and fester into larger bugs later. The more comprehensive our test suite is, the more comfortable we can be making Parrot releases because we know that HLLs and other users of Parrot that use the features we test for will work as expected. It's also a great way for new developers to get involved and start getting an idea about the capabilities, current state, and limitations of Parrot. Once you see for yourself the areas where Parrot needs some work, you'll be more able to start making the necessary fixes yourself.