Let’s assume for the moment that you’re writ­ing a Perl mod­ule or appli­ca­tion. You’d like to main­tain some lev­el of soft­ware qual­i­ty (or kwali­tee), so you’re writ­ing a suite of test scripts. Whether you’re writ­ing them first (good for you for prac­tic­ing test-​driven devel­op­ment!) or the appli­ca­tion code is already there, you’ll prob­a­bly be reach­ing for Test::Simple, Test::More, or one of the Test2::Suite bun­dles. With the lat­ter two you’re imme­di­ate­ly con­front­ed with a choice: do you count up the num­ber of tests into a plan, or do you for­sake that in favor of leav­ing a done_testing() call at the end of your test script(s)?

There are good argu­ments for both approach­es. When you first start, you prob­a­bly have no idea how many tests your scripts will con­tain. After all, a test script can be a use­ful tool for design­ing a mod­ule’s inter­face by writ­ing exam­ple code that will use it. Your explorato­ry code would be writ­ten as if the mod­ule or appli­ca­tion was already done, test­ing it in the way you’d like it to work. Not declar­ing a plan makes per­fect sense in this case; just put done_testing() at the end and get back to defin­ing your tests.

You don’t have that option when using Test::Simple, of course — it’s so basic it only has one func­tion (ok()), and you have to pre-​declare how many tests you plan to run when useing the mod­ule, like so:

use Test::Simple tests => 23;

Test::More also sup­ports this form of plan, or you can opt to use its plan func­tion to state the num­ber of tests in your script or subtest. With Test2 you have to use plan. Either way, the plan acts as a sort of meta-​test, mak­ing sure that you exe­cut­ed exact­ly what you intend­ed: no more, no less. While there are sit­u­a­tions where it’s not pos­si­ble to pre­dict how many times a giv­en set of tests should run, I would high­ly sug­gest that in all oth­er cas­es you should clean up” your tests and declare a plan. Later on, if you add or remove tests you’ll imme­di­ate­ly be aware that some­thing has changed and it’s time to tal­ly up a new plan.

What about oth­er Perl test­ing frame­works? They can use plans, too. Here are two examples:

Thoughts? Does declar­ing a test plan make writ­ing tests too inflex­i­ble? Does not hav­ing a plan encour­age bad behav­ior? Tell me what you think in the com­ments below.

Yesterday’s pair pro­gram­ming ses­sion had Gábor Szabó and I thrash­ing around for a bit try­ing to fig­ure out how to get test cov­er­age sta­tis­tics for the appli­ca­tion. The Devel::Cover doc­u­men­ta­tion lists how to run the mod­ule sev­er­al ways, but it does­n’t exact­ly describe how to run prove by itself rather than run­ning a Makefiles tests. I worked out how to do it today, and with the Baughs’ help on Twitter I worked out a few more methods.

All exam­ples below use the bash or zsh com­mand shells and were test­ed on macOS Catalina 10.15.7 run­ning zsh 5.7.1 and Perl 5.32.1. If you’re using some­thing very dif­fer­ent (e.g., Microsoft Windows’ CMD or PowerShell), you may have to set envi­ron­ment vari­ables differently.

Ad-​hoc test coverage

If all you want to do is run one shell com­mand, here it is:

$ prove -vlre 'perl -MDevel::Cover -Ilib' t

This takes advan­tage of proves --exec option (abbre­vi­at­ed as -e) to run a dif­fer­ent exe­cutable for tests. It recur­sive­ly (-r) runs all your tests ver­bose­ly (-v) from the t direc­to­ry while load­ing your appli­ca­tion’s libraries (-l), while the perl exe­cutable uses (-M) Devel::Cover and the lib sub­di­rec­to­ry. I use a sim­i­lar tech­nique when debug­ging tests.

$ HARNESS_PERL_SWITCHES=-MDevel::Cover prove -vlr t

This does almost the same thing as above with­out run­ning a dif­fer­ent exe­cutable. It sets Test::HarnessHARNESS_PERL_SWITCHES envi­ron­ment vari­able for the dura­tion of the prove com­mand. You won’t get the text out­put of your test cov­er­age at the end, though, and will still have to run Devel::Covers cover com­mand to both see the cov­er­age and gen­er­ate web pages.

In a dedicated test session, window, or tab

If you have a ter­mi­nal ses­sion, win­dow, or tab ded­i­cat­ed sole­ly to run­ning your tests, set one of the envi­ron­ment vari­ables above for that session:

$ export HARNESS_PERL_SWITCHES=-MDevel::Cover

Now all of your test scripts will pick up that option. You can add more options by enclos­ing the envi­ron­ment vari­able’s val­ue in 'quotes'. For exam­ple, you might also want to load Devel::NYTProf for code profiling:

$ export HARNESS_PERL_SWITCHES='-MDevel::Cover -MDevel::NYTProf'

Why not PERL5OPT?

Setting the PERL5OPT envi­ron­ment vari­able also sets options for the perl run­ning prove, which means that your test cov­er­age, pro­fil­ing, etc. will pick up proves exe­cu­tion as well as your test scripts.

What about yath?

I don’t know for sure; I don’t use the Test2 suite. But it looks like it has a --cover option for load­ing and pass­ing option to Devel::Cover.

In February I wrote an arti­cle sur­vey­ing excep­tion han­dling in Perl, rec­om­mend­ing that devel­op­ers use Test::Exception to make sure their code behaves as expect­ed. A com­menter on Reddit sug­gest­ed I check out Test::Fatal as an alter­na­tive. What advan­tages does it hold over Test::Exception?

  • It only exports one func­tion com­pared to Test::Exception’s four: exception, which you can then use with the full suite of reg­u­lar Test::More func­tions as well as oth­er test­ing libraries such as Test::Deep.
  • It does­n’t over­ride the caller func­tion or use Sub::Uplevel to hide your test blocks from the call stack, so if your excep­tion returns a stack trace you’ll get out­put from the test as well as the thing throw­ing the excep­tion. The author con­sid­ers this a fea­ture since Sub::Uplevel is twitchy.”

To ease port­ing, Test::Fatal also includes two func­tions, dies_ok and lives_ok, replac­ing Test::Exception’s func­tions of the same names. dies_ok does not pro­vide the excep­tion thrown, though, so if you’re test­ing that you’ll need to use exception along with a TAP-emit­ting func­tion like is() or like().

And that’s it! Either is a valid choice; it comes down to whether you pre­fer one approach over anoth­er. Test::Exception is also includ­ed as part of Test::Mosts require­ments, so if you’re using the lat­ter to reduce boil­er­plate you’ll be get­ting the former.

Postscript:

I’d be remiss if I did­n’t also men­tion Test2::Tools::Exception, which is the pre­ferred way to test excep­tions using the Test2 frame­work. If you’re using Test2, ignore all the above and go straight to Test2::Tools::Exception.

Failure is a uni­ver­sal truth of com­put­ers. Files fail to open, web pages fail to load, pro­grams fail to install, mes­sages fail to arrive. As a devel­op­er you have no choice but to work in a seem­ing­ly hos­tile envi­ron­ment in which bugs and errors lurk around every corner.

Hopefully you find and fix the bugs dur­ing devel­op­ment and test­ing, but even with all bugs squashed excep­tion­al con­di­tions can occur. It’s your job as a Perl devel­op­er to use the tools avail­able to you to han­dle these excep­tions. Here are a few of them.

eval, die and $EVAL_ERROR ([email protected]) (updated)

Perl has a prim­i­tive but effec­tive mech­a­nism for run­ning code that may fail called eval. It runs either a string or block of Perl code, trap­ping any errors so that the enclos­ing pro­gram does­n’t crash. It’s your job then to ignore or han­dle the error; eval will return undef (or an emp­ty list in list con­text) and set the mag­ic vari­able [email protected] to the error string. (You can spell that $EVAL_ERROR if you use the English mod­ule, which you prob­a­bly should to allow for more read­able code.) Here’s a con­trived example:

use English;

eval { $foo / 0; 1 }
  or warn "tried to divide by zero: $EVAL_ERROR";

(Why the 1 at the end of the block? It forces the eval to return true if it suc­ceeds; the or con­di­tion is exe­cut­ed if it returns false.)

What if you want to pur­pose­ful­ly cause an excep­tion, so that an enclos­ing eval (pos­si­bly sev­er­al lay­ers up) can han­dle it? You use die:

use English;

eval { process_file('foo.txt'); 1 }
  or warn "couldn't process file: $EVAL_ERROR";

sub process_file {
    my $file = shift;
    open my $fh, '<', $file
      or die "couldn't read $file: $OS_ERROR";

    ... # do something with $fh
}

It’s worth repeat­ing that as a state­ment: You use excep­tions so that enclos­ing code can decide how to han­dle the error. Contrast this with sim­ply han­dling a func­tion’s return val­ue at the time it’s exe­cut­ed: except in the sim­plest of scripts, that part of the code like­ly has no idea what the error means to the rest of the appli­ca­tion or how to best han­dle the problem.

autodie

Since many of Perl’s built-​in func­tions (like open) return false or oth­er val­ues on fail­ure, it can be tedious and error-​prone to make sure that all of them report prob­lems as excep­tions. Enter autodie, which will help­ful­ly replace the func­tions you choose with equiv­a­lents that throw excep­tions. Introduced in Perl 5.10.1, it only affects the enclos­ing code block, and even goes so far as to set $EVAL_ERROR to an object that can be queried for more detail. Here’s an example:

use English;
use autodie; # defaults to everything but system and exec

eval { open my $fh, '<', 'foo.txt'; 1 } or do {
    if ($EVAL_ERROR
      and $EVAL_ERROR->isa('autodie::exception') {
        warn 'Error from open'
          if $EVAL_ERROR->matches('open');
        warn 'I/O error'
          if $EVAL_ERROR->matches(':io');
    }
    elsif ($EVAL_ERROR) {
        warn "Something else went wrong: $EVAL_ERROR";
    }
};

try and catch

If you’re famil­iar with oth­er pro­gram­ming lan­guages, you’re prob­a­bly look­ing for syn­tax like try and catch for your excep­tion needs. The good news is that it’s com­ing in Perl 5.34 thanks to the ever-​productive Paul LeoNerd” Evans; the bet­ter news is that you can use it today with his Feature::Compat::Try mod­ule, itself a dis­til­la­tion of his pop­u­lar Syntax::Keyword::Try. Here’s an example:

use English;
use autodie;
use Feature::Compat::Try;

sub foo {
    try {
        attempt_a_thing();
        return 'success!';
    }
    catch ($exception) {
        return "failure: $exception"
          if not $exception->isa('autodie::exception');

        return 'failed in ' . $exception->function
          . ' line '        . $exception->line
          . ' called with '
          . join ', ', @{$exception->args};
    }
}

Note that autodie and Feature::Compat::Try are com­ple­men­tary and can be used togeth­er; also note that unlike an eval block, you can return from the enclos­ing func­tion in a try block.

The under­ly­ing Syntax::Keyword::Try mod­ule has even more options like a finally block and a cou­ple exper­i­men­tal fea­tures. I now pre­fer it to oth­er mod­ules that imple­ment try/​catch syn­tax like Try::Tiny and TryCatch (even though we use Try::Tiny at work). If all you need is the basic syn­tax above, using Feature::Compat::Try will get you used to the seman­tics that are com­ing in the next ver­sion of Perl.

Other exception modules (updated)

autodie is nice, and some oth­er mod­ules and frame­works imple­ment their own excep­tion class­es, but what if you want some help defin­ing your own? After all, an error string can only con­vey so much infor­ma­tion, may be dif­fi­cult to parse, and may need to change as busi­ness require­ments change.

Although CPAN has the pop­u­lar Exception::Class mod­ule, its author Dave Rolsky rec­om­mends that you use Throwable if you’re using Moose or Moo. If you’re rolling your own objects, use Throwable::Error.

Using Throwable could­n’t be simpler:

package Foo;

use Moo;
with 'Throwable';

has message => (is => 'ro');

... # later...

package main; 
Foo->throw( {message => 'something went wrong'} );

And it comes with Throwable::Error, which you can sub­class to get sev­er­al use­ful methods:

package Local::My::Error;
use parent 'Throwable::Error';

... # later...

package main;
use Feature::Compat::Try;

try {
    Local::My::Error->throw('something bad');
}
catch ($exception) {
    warn $exception->stack_trace->as_string;
}

(That stack_trace attribute comes cour­tesy of the StackTrace::Auto role com­posed into Throwable::Error. Moo and Moose users should sim­ply com­pose it into their class­es to get it.)

Testing exceptions with Test::Exception

Inevitably bugs will creep in to your code, and auto­mat­ed tests are one of the main weapons in a devel­op­er’s arse­nal against them. Use Test::Exception when writ­ing tests against code that emits excep­tions to see whether it behaves as expected:

use English;
use Test::More;
use Test::Exception;

...

throws_ok(sub { $foo->method(42) }, qr/error 42/,
  'method throws an error when it gets 42');
throws_ok(sub { $foo->method(57) }, 'My::Exception::Class',
  'method throws the right exception class');

dies_ok(sub { $bar->method() }, 'method died, no params');

lives_and(sub { is($baz->method(17), 17) },
  'method ran without exception, returned right value'); 

throws_ok(sub { $qux->process('nonexistent_file.txt') },
  'autodie::exception', # hey look, it's autodie again
  'got an autodie exception',
);
my $exception = $EVAL_ERROR;
SKIP: {
    skip 'no autodie exception thrown', 1
      unless $exception
      and $exception->isa('autodie::exception');
    ok($exception->match(':socket'),
      'was a socket error:' . $exception->errno);
}

done_testing();

Note that Test::Exceptions func­tions don’t mess with $EVAL_ERROR, so you’re free to check its val­ue right after you call it.

Documenting errors and exceptions

If I can leave you with one mes­sage, it’s this: Please doc­u­ment every error and excep­tion your code pro­duces, prefer­ably in a place and lan­guage that the end-​user can under­stand. The DIAGNOSTICS sec­tion of your doc­u­men­ta­tion (you are writ­ing doc­u­men­ta­tion, right, not just code com­ments?) is a great can­di­date. You can mod­el this sec­tion after the perldiag man­u­al page, which goes into great detail about many of the error mes­sages gen­er­at­ed by Perl itself.

(A pre­vi­ous ver­sion of this arti­cle did not note that one should make sure a suc­cess­ful eval returns true, and incor­rect­ly stat­ed that Class::Exception and Throwable were dep­re­cat­ed due to a bug in the MetaCPAN web site. Thanks to Dan Book for the corrections.)