Tag: testing

  • Perl debugger superpowers, part 2

    Perl debugger superpowers, part 2

    In March I wrote The Perl debug­ger can be your super­pow­er, intro­duc­ing the step debug­ger as a bet­ter way to debug your Perl code rather than lit­ter­ing your source with tem­po­rary print state­ments or log­ging. I use the debug­ger all the time, and I’ve real­ized that some more tech­niques are worth covering.

    Although I men­tioned a caveat when debug­ging web appli­ca­tions, our apps at work all adhere to the Perl Web Server Gateway Interface (PSGI) spec­i­fi­ca­tion and thus we can use tools like Test::WWW::Mechanize::PSGI or Plack::Test to run tests and debug­ging ses­sions in the same Perl process. (Mojolicious users can use some­thing like Test::Mojo for the same effect.)

    To demon­strate, let’s get start­ed with some­thing like this which tests that a giv­en route (/say-hello) returns a cer­tain JSON struc­ture ({"message": "Hello world!"}):

    #!/usr/bin/env perl
    
    use Test::Most;
    use Test::WWW::Mechanize::PSGI;
    use JSON::MaybeXS;
    use Local::MyApp; # name of app's main module
    
    my $mech = Test::WWW::Mechanize::PSGI->new(
        # a Dancer2 app, so to_app returns a PSGI coderef
        app => Local::MyApp->to_app(),
    );
    $mech->get_ok('/say-hello');
    lives_and {
        my $json = decode_json($mech->content);
        cmp_deeply( $json, {message => 'Hello world!'} );
    } 'message is Hello world!';
    
    done_testing;

    All very fine and well, but what hap­pens if that route starts return­ing a dif­fer­ent mes­sage or worse, invalid out­put that caus­es decode_json to fail? Eventually, you’ll rewrite the test in the script to out­put the offend­ing con­tent when some­thing goes wrong, but right now you want to suss out the root cause.

    Debuggers have the con­cept of break­points, which are flags that tell the debug­ger to stop at a cer­tain line of code and wait for instruc­tions. We can set them while run­ning the debug­ger with the b com­mand or con­tin­ue to a one-​time break­point with the c com­mand, or we can insert them into the code our­selves before run­ning it through the debug­ger in the first place.

    Add this line right after the lives_and { line:

    $DB::single = 1;

    This sim­u­lates hav­ing typed the s com­mand in the debug­ger at that line, stop­ping exe­cu­tion at that point. Run our test with per­l’s -d option, and then type c to con­tin­ue to that breakpoint:

    $ perl -d -Ilib t/test_psgi.t
    
    Loading DB routines from perl5db.pl version 1.60
    Editor support available.
    
    Enter h or 'h h' for help, or 'man perldebug' for more help.
    
    [Local::MyApp:7170] core @2021-07-06 07:33:22> Built config from files: /Users/mgardner/Projects/blog/myapp/config.yml /Users/mgardner/Projects/blog/myapp/environments/development.yml in (eval 310)[/Users/mgardner/.plenv/versions/5.34.0/lib/perl5/site_perl/5.34.0/Sub/Quote.pm:3] l. 910
    Test2::API::CODE(0x7ffabea39ee8)(/Users/mgardner/.plenv/versions/5.34.0/lib/perl5/site_perl/5.34.0/Test2/API.pm:71):
    71:	    INIT { eval 'END { test2_set_is_end() }; 1' or die $@ }
    
      DB<1> c
    
    [...]
    ok 1 - GET /say-hello
    main::CODE(0x7f8069caf2c8)(t/test_psgi.t:14):
    15:	    my $json = decode_json($mech->content);
    
      DB<1> 

    From here we can exam­ine vari­ables, set oth­er break­points, or even exe­cute arbi­trary lines of code. Let’s see what became of that HTTP GET request:

      DB<1> x $mech->content
    
    0  '{"error":"Undefined subroutine &Local::MyApp::build_frog called at lib/Local/MyApp.pm line 11.\\n"}'
    
      DB<2> 

    Aha, some­thing has returned some dif­fer­ent JSON indi­cat­ing an error. Let’s look at the lines around (1020) the offend­ing line (11):

      DB<2> f lib/Local/MyApp.pm
    
      DB<3> l 10-20
    
    10:	        my $method = 'build_frog';
    11:	        $method->();
    12 	    }
    13:	    catch ($e) {
    14:	        send_as JSON => {error => $e};
    15 	    }
    16:	    send_as JSON => {message => 'Hello world!'};
    17:	};
    18
    19 	sub build_frob {
    20:	    return;
    
      DB<4>

    Yep, a typo on line 11, and one that was­n’t caught at com­pile time since it’s gen­er­at­ed at runtime.

    Just to be sure (and to demon­strate some oth­er cool debug­ger fea­tures), let’s set anoth­er break­point while in the debug­ger and then exer­cise that route again. Then we’ll check that $method vari­able against the list of avail­able meth­ods in the Local::MyApp package.

      DB<4> b 11
    
      DB<5> $mech->get('/say-hello')
    
    [...]
    Local::MyApp::CODE(0x7f8066f2db60)(lib/Local/MyApp.pm:11):
    11:	        $method->();
    
      DB<<6>> x $method
    
    0  'build_frog'
    
      DB<<7>> m Local::MyApp
    any
    app
    body_parameters
    build_frob
    captures
    config
    content
    [...]
      DB<<8>>

    No doubt about it, that vari­able is being set incorrectly.

    Quit out of the debug­ger with the q com­mand, make the fix (we prob­a­bly want errors to give some­thing oth­er than an HTTP 200 OK while we’re at it), and re-​run the test:

    $ perl -Ilib t/test_psgi.t
    
    [Local::MyApp:8277] core @2021-07-06 07:48:36> Built config from files: /Users/mgardner/Projects/blog/myapp/config.yml /Users/mgardner/Projects/blog/myapp/environments/development.yml in (eval 309) l. 910
    Name "DB::single" used only once: possible typo at t/test_psgi.t line 13.
    [...]
    ok 1 - GET /say-hello
    ok 2 - message is Hello world!
    1..2

    Note that warn­ing about leav­ing $DB::single in there. While harm­less, it’s a good reminder to remove such lines from your code so that they don’t sur­prise you or your team­mates dur­ing future debug­ging sessions.

    And that’s it. Note that because we’re using PSGI, we were able to set break­points in our web app code itself and the debug­ger stopped there and enabled us to have a look around. And as you’ve seen, once you’re at a break­point you can switch to dif­fer­ent files, add/​remove more break­points, run arbi­trary code, and more. The perlde­bug doc­u­men­ta­tion page has all the details.

    Happy debug­ging! For your ref­er­ence, here’s the full app mod­ule and test script used in this article:

    MyApp.pm

    package Local::MyApp;
    use Dancer2;
    use Feature::Compat::Try;
    
    our $VERSION = '0.1';
    
    get '/say-hello' => sub {
        try {
            no strict 'refs';
            my $method = 'build_frob';
            $method->();
        }
        catch ($e) {
            status 'error';
            send_as JSON => {error => $e};
        }
        send_as JSON => {message => 'Hello world!'};
    };
    
    sub build_frob {
        return;
    }
    
    true;

    test_psgi.t

    #!/usr/bin/env perl
    
    use Test::Most;
    use Test::WWW::Mechanize::PSGI;
    use JSON::MaybeXS;
    use Local::MyApp; # name of your app's main module goes here
    
    my $mech = Test::WWW::Mechanize::PSGI->new(
        # a Dancer2 app, so to_app returns a PSGI coderef
        app => Local::MyApp->to_app(),
    );
    $mech->get_ok('/say-hello');
    lives_and {
        my $json = decode_json($mech->content);
        cmp_deeply( $json, {message => 'Hello world!'} );
    } 'message is Hello world!';
    
    done_testing;
  • Testing Perl: To plan or not to plan

    Testing Perl: To plan or not to plan

    Let’s assume for the moment that you’re writ­ing a Perl mod­ule or appli­ca­tion. You’d like to main­tain some lev­el of soft­ware qual­i­ty (or kwali­tee), so you’re writ­ing a suite of test scripts. Whether you’re writ­ing them first (good for you for prac­tic­ing test-​driven devel­op­ment!) or the appli­ca­tion code is already there, you’ll prob­a­bly be reach­ing for Test::Simple, Test::More, or one of the Test2::Suite bun­dles. With the lat­ter two you’re imme­di­ate­ly con­front­ed with a choice: do you count up the num­ber of tests into a plan, or do you for­sake that in favor of leav­ing a done_testing() call at the end of your test script(s)?

    There are good argu­ments for both approach­es. When you first start, you prob­a­bly have no idea how many tests your scripts will con­tain. After all, a test script can be a use­ful tool for design­ing a mod­ule’s inter­face by writ­ing exam­ple code that will use it. Your explorato­ry code would be writ­ten as if the mod­ule or appli­ca­tion was already done, test­ing it in the way you’d like it to work. Not declar­ing a plan makes per­fect sense in this case; just put done_testing() at the end and get back to defin­ing your tests.

    You don’t have that option when using Test::Simple, of course—it’s so basic it only has one func­tion (ok()), and you have to pre-​declare how many tests you plan to run when useing the mod­ule, like so:

    use Test::Simple tests => 23;

    Test::More also sup­ports this form of plan, or you can opt to use its plan func­tion to state the num­ber of tests in your script or subtest. With Test2 you have to use plan. Either way, the plan acts as a sort of meta-​test, mak­ing sure that you exe­cut­ed exact­ly what you intend­ed: no more, no less. While there are sit­u­a­tions where it’s not pos­si­ble to pre­dict how many times a giv­en set of tests should run, I would high­ly sug­gest that in all oth­er cas­es you should clean up” your tests and declare a plan. Later on, if you add or remove tests you’ll imme­di­ate­ly be aware that some­thing has changed and it’s time to tal­ly up a new plan.

    What about oth­er Perl test­ing frame­works? They can use plans, too. Here are two examples:

    Thoughts? Does declar­ing a test plan make writ­ing tests too inflex­i­ble? Does not hav­ing a plan encour­age bad behav­ior? Tell me what you think in the com­ments below.

  • Perl test coverage when you don’t have a Makefile

    Yesterday’s pair pro­gram­ming ses­sion had Gábor Szabó and I thrash­ing around for a bit try­ing to fig­ure out how to get test cov­er­age sta­tis­tics for the appli­ca­tion. The Devel::Cover doc­u­men­ta­tion lists how to run the mod­ule sev­er­al ways, but it does­n’t exact­ly describe how to run prove by itself rather than run­ning a Makefiles tests. I worked out how to do it today, and with the Baughs’ help on Twitter I worked out a few more methods.

    All exam­ples below use the bash or zsh com­mand shells and were test­ed on macOS Catalina 10.15.7 run­ning zsh 5.7.1 and Perl 5.32.1. If you’re using some­thing very dif­fer­ent (e.g., Microsoft Windows’ CMD or PowerShell), you may have to set envi­ron­ment vari­ables differently.

    Ad-​hoc test coverage

    If all you want to do is run one shell com­mand, here it is:

    $ prove -vlre 'perl -MDevel::Cover -Ilib' t

    This takes advan­tage of proves --exec option (abbre­vi­at­ed as -e) to run a dif­fer­ent exe­cutable for tests. It recur­sive­ly (-r) runs all your tests ver­bose­ly (-v) from the t direc­to­ry while load­ing your appli­ca­tion’s libraries (-l), while the perl exe­cutable uses (-M) Devel::Cover and the lib sub­di­rec­to­ry. I use a sim­i­lar tech­nique when debug­ging tests.

    $ HARNESS_PERL_SWITCHES=-MDevel::Cover prove -vlr t

    This does almost the same thing as above with­out run­ning a dif­fer­ent exe­cutable. It sets Test::HarnessHARNESS_PERL_SWITCHES envi­ron­ment vari­able for the dura­tion of the prove com­mand. You won’t get the text out­put of your test cov­er­age at the end, though, and will still have to run Devel::Covers cover com­mand to both see the cov­er­age and gen­er­ate web pages.

    In a dedicated test session, window, or tab

    If you have a ter­mi­nal ses­sion, win­dow, or tab ded­i­cat­ed sole­ly to run­ning your tests, set one of the envi­ron­ment vari­ables above for that session:

    $ export HARNESS_PERL_SWITCHES=-MDevel::Cover

    Now all of your test scripts will pick up that option. You can add more options by enclos­ing the envi­ron­ment vari­able’s val­ue in 'quotes'. For exam­ple, you might also want to load Devel::NYTProf for code profiling:

    $ export HARNESS_PERL_SWITCHES='-MDevel::Cover -MDevel::NYTProf'

    Why not PERL5OPT?

    Setting the PERL5OPT envi­ron­ment vari­able also sets options for the perl run­ning prove, which means that your test cov­er­age, pro­fil­ing, etc. will pick up proves exe­cu­tion as well as your test scripts.

    What about yath?

    I don’t know for sure; I don’t use the Test2 suite. But it looks like it has a --cover option for load­ing and pass­ing option to Devel::Cover.

  • The attraction of Test::Fatal in Perl

    In February I wrote an arti­cle sur­vey­ing excep­tion han­dling in Perl, rec­om­mend­ing that devel­op­ers use Test::Exception to make sure their code behaves as expect­ed. A com­menter on Reddit sug­gest­ed I check out Test::Fatal as an alter­na­tive. What advan­tages does it hold over Test::Exception?

    • It only exports one func­tion com­pared to Test::Exception’s four: exception, which you can then use with the full suite of reg­u­lar Test::More func­tions as well as oth­er test­ing libraries such as Test::Deep.
    • It does­n’t over­ride the caller func­tion or use Sub::Uplevel to hide your test blocks from the call stack, so if your excep­tion returns a stack trace you’ll get out­put from the test as well as the thing throw­ing the excep­tion. The author con­sid­ers this a fea­ture since Sub::Uplevel is twitchy.”

    To ease port­ing, Test::Fatal also includes two func­tions, dies_ok and lives_ok, replac­ing Test::Exception’s func­tions of the same names. dies_ok does not pro­vide the excep­tion thrown, though, so if you’re test­ing that you’ll need to use exception along with a TAP-emit­ting func­tion like is() or like().

    And that’s it! Either is a valid choice; it comes down to whether you pre­fer one approach over anoth­er. Test::Exception is also includ­ed as part of Test::Mosts require­ments, so if you’re using the lat­ter to reduce boil­er­plate you’ll be get­ting the former.

    Postscript:

    I’d be remiss if I did­n’t also men­tion Test2::Tools::Exception, which is the pre­ferred way to test excep­tions using the Test2 frame­work. If you’re using Test2, ignore all the above and go straight to Test2::Tools::Exception.

  • Exceptional Perl: Failure is an option

    Failure is a uni­ver­sal truth of com­put­ers. Files fail to open, web pages fail to load, pro­grams fail to install, mes­sages fail to arrive. As a devel­op­er you have no choice but to work in a seem­ing­ly hos­tile envi­ron­ment in which bugs and errors lurk around every corner.

    Hopefully you find and fix the bugs dur­ing devel­op­ment and test­ing, but even with all bugs squashed excep­tion­al con­di­tions can occur. It’s your job as a Perl devel­op­er to use the tools avail­able to you to han­dle these excep­tions. Here are a few of them.

    eval, die and $EVAL_ERROR ($@) (updated)

    Perl has a prim­i­tive but effec­tive mech­a­nism for run­ning code that may fail called eval. It runs either a string or block of Perl code, trap­ping any errors so that the enclos­ing pro­gram does­n’t crash. It’s your job then to ignore or han­dle the error; eval will return undef (or an emp­ty list in list con­text) and set the mag­ic vari­able $@ to the error string. (You can spell that $EVAL_ERROR if you use the English mod­ule, which you prob­a­bly should to allow for more read­able code.) Here’s a con­trived example:

    use English;
    
    eval { $foo / 0; 1 }
      or warn "tried to divide by zero: $EVAL_ERROR";

    (Why the 1 at the end of the block? It forces the eval to return true if it suc­ceeds; the or con­di­tion is exe­cut­ed if it returns false.)

    What if you want to pur­pose­ful­ly cause an excep­tion, so that an enclos­ing eval (pos­si­bly sev­er­al lay­ers up) can han­dle it? You use die:

    use English;
    
    eval { process_file('foo.txt'); 1 }
      or warn "couldn't process file: $EVAL_ERROR";
    
    sub process_file {
        my $file = shift;
        open my $fh, '<', $file
          or die "couldn't read $file: $OS_ERROR";
    
        ... # do something with $fh
    }

    It’s worth repeat­ing that as a state­ment: You use excep­tions so that enclos­ing code can decide how to han­dle the error. Contrast this with sim­ply han­dling a func­tion’s return val­ue at the time it’s exe­cut­ed: except in the sim­plest of scripts, that part of the code like­ly has no idea what the error means to the rest of the appli­ca­tion or how to best han­dle the problem.

    autodie

    Since many of Perl’s built-​in func­tions (like open) return false or oth­er val­ues on fail­ure, it can be tedious and error-​prone to make sure that all of them report prob­lems as excep­tions. Enter autodie, which will help­ful­ly replace the func­tions you choose with equiv­a­lents that throw excep­tions. Introduced in Perl 5.10.1, it only affects the enclos­ing code block, and even goes so far as to set $EVAL_ERROR to an object that can be queried for more detail. Here’s an example:

    use English;
    use autodie; # defaults to everything but system and exec
    
    eval { open my $fh, '<', 'foo.txt'; 1 } or do {
        if ($EVAL_ERROR
          and $EVAL_ERROR->isa('autodie::exception') {
            warn 'Error from open'
              if $EVAL_ERROR->matches('open');
            warn 'I/O error'
              if $EVAL_ERROR->matches(':io');
        }
        elsif ($EVAL_ERROR) {
            warn "Something else went wrong: $EVAL_ERROR";
        }
    };

    try and catch

    If you’re famil­iar with oth­er pro­gram­ming lan­guages, you’re prob­a­bly look­ing for syn­tax like try and catch for your excep­tion needs. The good news is that it’s com­ing in Perl 5.34 thanks to the ever-​productive Paul LeoNerd” Evans; the bet­ter news is that you can use it today with his Feature::Compat::Try mod­ule, itself a dis­til­la­tion of his pop­u­lar Syntax::Keyword::Try. Here’s an example:

    use English;
    use autodie;
    use Feature::Compat::Try;
    
    sub foo {
        try {
            attempt_a_thing();
            return 'success!';
        }
        catch ($exception) {
            return "failure: $exception"
              if not $exception->isa('autodie::exception');
    
            return 'failed in ' . $exception->function
              . ' line '        . $exception->line
              . ' called with '
              . join ', ', @{$exception->args};
        }
    }

    Note that autodie and Feature::Compat::Try are com­ple­men­tary and can be used togeth­er; also note that unlike an eval block, you can return from the enclos­ing func­tion in a try block.

    The under­ly­ing Syntax::Keyword::Try mod­ule has even more options like a finally block and a cou­ple exper­i­men­tal fea­tures. I now pre­fer it to oth­er mod­ules that imple­ment try/​catch syn­tax like Try::Tiny and TryCatch (even though we use Try::Tiny at work). If all you need is the basic syn­tax above, using Feature::Compat::Try will get you used to the seman­tics that are com­ing in the next ver­sion of Perl.

    Other exception modules (updated)

    autodie is nice, and some oth­er mod­ules and frame­works imple­ment their own excep­tion class­es, but what if you want some help defin­ing your own? After all, an error string can only con­vey so much infor­ma­tion, may be dif­fi­cult to parse, and may need to change as busi­ness require­ments change.

    Although CPAN has the pop­u­lar Exception::Class mod­ule, its author Dave Rolsky rec­om­mends that you use Throwable if you’re using Moose or Moo. If you’re rolling your own objects, use Throwable::Error.

    Using Throwable could­n’t be simpler:

    package Foo;
    
    use Moo;
    with 'Throwable';
    
    has message => (is => 'ro');
    
    ... # later...
    
    package main; 
    Foo->throw( {message => 'something went wrong'} );

    And it comes with Throwable::Error, which you can sub­class to get sev­er­al use­ful methods:

    package Local::My::Error;
    use parent 'Throwable::Error';
    
    ... # later...
    
    package main;
    use Feature::Compat::Try;
    
    try {
        Local::My::Error->throw('something bad');
    }
    catch ($exception) {
        warn $exception->stack_trace->as_string;
    }

    (That stack_trace attribute comes cour­tesy of the StackTrace::Auto role com­posed into Throwable::Error. Moo and Moose users should sim­ply com­pose it into their class­es to get it.)

    Testing exceptions with Test::Exception

    Inevitably bugs will creep in to your code, and auto­mat­ed tests are one of the main weapons in a devel­op­er’s arse­nal against them. Use Test::Exception when writ­ing tests against code that emits excep­tions to see whether it behaves as expected:

    use English;
    use Test::More;
    use Test::Exception;
    
    ...
    
    throws_ok(sub { $foo->method(42) }, qr/error 42/,
      'method throws an error when it gets 42');
    throws_ok(sub { $foo->method(57) }, 'My::Exception::Class',
      'method throws the right exception class');
    
    dies_ok(sub { $bar->method() }, 'method died, no params');
    
    lives_and(sub { is($baz->method(17), 17) },
      'method ran without exception, returned right value'); 
    
    throws_ok(sub { $qux->process('nonexistent_file.txt') },
      'autodie::exception', # hey look, it's autodie again
      'got an autodie exception',
    );
    my $exception = $EVAL_ERROR;
    SKIP: {
        skip 'no autodie exception thrown', 1
          unless $exception
          and $exception->isa('autodie::exception');
        ok($exception->match(':socket'),
          'was a socket error:' . $exception->errno);
    }
    
    done_testing();

    Note that Test::Exceptions func­tions don’t mess with $EVAL_ERROR, so you’re free to check its val­ue right after you call it.

    Documenting errors and exceptions

    If I can leave you with one mes­sage, it’s this: Please doc­u­ment every error and excep­tion your code pro­duces, prefer­ably in a place and lan­guage that the end-​user can under­stand. The DIAGNOSTICS sec­tion of your doc­u­men­ta­tion (you are writ­ing doc­u­men­ta­tion, right, not just code com­ments?) is a great can­di­date. You can mod­el this sec­tion after the perldiag man­u­al page, which goes into great detail about many of the error mes­sages gen­er­at­ed by Perl itself.

    (A pre­vi­ous ver­sion of this arti­cle did not note that one should make sure a suc­cess­ful eval returns true, and incor­rect­ly stat­ed that Class::Exception and Throwable were dep­re­cat­ed due to a bug in the MetaCPAN web site. Thanks to Dan Book for the corrections.)