woman looking at the map

Six months ago I gave an overview of Perl’s list processing fundamentals, briefly describing what lists are and then introducing the built-​in map and grep functions for transforming and filtering them. Later on, I compiled a list (how appropriate) of list processing modules available via CPAN, noting there’s some confusing duplication of effort. But you’re a busy developer, and you just want to know the Right Thing To Do™ when faced with a list processing challenge.

First, some credit is due: these are all restatements of several Perl::Critic policies which in turn codify standards described in Damian Conway’s Perl Best Practices (2005). I’ve repeatedly recommended the latter as a starting point for higher-​quality Perl development. Over the years these practices continue to be re-​evaluated (including by the author himself) and various authors release new policy modules, but perlcritic remains a great tool for ensuring you (and your team or other contributors) maintain a consistent high standard in your code.

With that said, on to the recommendations!

Don’t use grep to check if any list elements match

It might sound weird to lead off by recommending not to use grep, but sometimes it’s not the right tool for the job. If you’ve got a list and want to determine if a condition matches any item in it, you might try:

if (grep { some_condition($_) } @my_list) {
    ... # don't do this!
}

Yes, this works because (in scalar context) grep returns the number of matches found, but it’s wasteful, checking every element of @my_list (which could be lengthy) before finally providing a result. Use the standard List::Util module’s any function, which immediately returns (“short-​circuits”) on the first match:

use List::Util 1.33 qw(any);

if (any { some_condition($_) } @my_list) {
... # do something
}

Perl has included the requisite version of this module since version 5.20 in 2014; for earlier releases, you’ll need to update from CPAN. List::Util has many other great list-​reduction, key/​value pair, and other related functions you can import into your code, so check it out before you attempt to re-​invent any wheels.

As a side note for web developers, the Perl Dancer framework also includes an any keyword for declaring multiple HTTP routes, so if you’re mixing List::Util in there don’t import it. Instead, call it explicitly like this or you’ll get an error about a redefined function:

use List::Util 1.33;

if (List::Util::any { some_condition($_) } @my_list) {
... # do something
}

This recommendation is codified in the BuiltinFunctions::ProhibitBooleanGrep Perl::Critic policy, comes directly from Perl Best Practices, and is recommended by the Software Engineering Institute Computer Emergency Response Team (SEI CERT)’s Perl Coding Standard.

Don’t change $_ in map or grep

I mentioned this back in March, but it bears repeating: map and grep are intended as pure functions, not mutators with side effects. This means that the original list should remain unchanged. Yes, each element aliases in turn to the $_ special variable, but that’s for speed and can have surprising results if changed even if it’s technically allowed. If you need to modify an array in-​place use something like:

for (@my_array) {
$_ = ...; # make your changes here
}

If you want something that looks like map but won’t change the original list (and don’t mind a few CPAN dependencies), consider List::SomeUtilsapply function:

use List::SomeUtils qw(apply);

my @doubled_array = apply {$_ *= 2} @old_array;

Lastly, side effects also include things like manipulating other variables or doing input and output. Don’t use map or grep in a void context (i.e., without a resulting array or list); do something with the results or use a for or foreach loop:

map { print foo($_) } @my_array; # don't do this
print map { foo($_) } @my_array; # do this instead

map { push @new_array, foo($_) } @my_array; # don't do this
@new_array = map { foo($_) } @my_array; # do this instead

This recommendation is codified by the BuiltinFunctions::ProhibitVoidGrep, BuiltinFunctions::ProhibitVoidMap, and ControlStructures::ProhibitMutatingListFunctions Perl::Critic policies. The latter comes from Perl Best Practices and is an SEI CERT Perl Coding Standard rule.

Use blocks with map and grep, not expressions

You can call map or grep like this (parentheses are optional around built-​in functions):

my @new_array  = map foo($_), @old_array; # don't do this
my @new_array2 = grep !/^#/, @old_array; # don't do this

Or like this:

my @new_array  = map { foo($_) } @old_array;
my @new_array2 = grep {!/^#/} @old_array;

Do it the second way. It’s easier to read, especially if you’re passing in a literal list or multiple arrays, and the expression forms can conceal bugs. This recommendation is codified by the BuiltinFunctions::RequireBlockGrep and BuiltinFunctions::RequireBlockMap Perl::Critic policies and comes from Perl Best Practices.

Refactor multi-​statement maps, greps, and other list functions

map, grep, and friends should follow the Unix philosophy of Do One Thing and Do It Well.” Your readability and maintainability drop with every statement you place inside one of their blocks. Consider junior developers and future maintainers (this includes you!) and refactor anything with more than one statement into a separate subroutine or at least a for loop. This goes for list processing functions (like the aforementioned any) imported from other modules, too.

This recommendation is codified by the Perl Best Practices-inspired BuiltinFunctions::ProhibitComplexMappings and BuiltinFunctions::RequireSimpleSortBlock Perl::Critic policies, although those only cover map and sort functions, respectively.


Do you have any other suggestions for list processing best practices? Feel free to leave them in the comments or better yet, consider creating new Perl::Critic policies for them or contacting the Perl::Critic team to develop them for your organization.

circus theme party

Last week’s article got a great response on Hacker News, and this particular comment caught my eye:

I think this is the real point about Perl code readability: it gives you enough flexibility to do things however you like, and as a result many programmers are faced with a mirror that reflects their own bad practices back at them.

orev, Hacker News

This is why Damian Conway’s Perl Best Practices (2005) is one of my favorite books and perlcritic, the code analyzer is one of my favorite tools. (Though the former could do with an update and the latter includes policies that contradict Conway.) Point perlcritic at your code, maybe add some other policies that agree with your house style, and gradually ratchet up the severity level from gentle” to brutal.” All kinds of bad juju will come to light, from wastefully using grep to having too many subroutine arguments to catching private variable use from other packages. perlcritic offers a useful baseline of conduct and you can always customize its configuration to your own tastes.

The other conformance tool in a Perl developer’s belt is perltidy, and it too has a Conway-​compatible configuration as well as its default Perl Style Guide settings. I’ve found that more than anything else, perltidy helps settle arguments both between developers and between their code in helping to avoid excessive merge conflicts.

But apart from extra tools, Perl the language itself can be bent and even broken to suit just about anyone’s agenda. Those used to more bondage-​and-​discipline languages (hi, Java!) might feel revulsion at the lengths to which this has sometimes been taken, but per the quote above this is less an indictment of the language and more of its less methodical programmers.

Some of this behavior can be rehabilitated with perlcritic and perltidy, but what about other sins attributed to Perl? Here are a few perennial favorites”:

Objects and Object-​Oriented Programming

Perl has a minimalist object system based on earlier-​available language concepts like data structures (often hashes, which it has in common with JavaScript), packages, and subroutines. Since Perl 5’s release in 1994 much verbose OO code has been written using these tools.

The good news is that since 2007 we’ve had a sophisticated metaobject-​protocol-​based layer on top of them called Moose, since 2010 a lightweight but forward-​compatible system called Moo, and a couple of even tinier options as described in the Perl OO Tutorial. Waiting in the wings is Corinna, an effort to bring next-​generation object capabilities into the Perl core itself, and Object::Pad, a testbed for some of the ideas in Corinna that you can use today in current code. (Really, please try it—the author needs feedback!)

All this is to say that 99% of the time you never need trouble yourself with bless, constructors, or writing accessors for class or object attributes. Smarter people than me have done the work for you, and you might even find a concept or three that you wish other languages had.

Contexts

There are two major ones: list and scalar. Another way to think of it is plural” vs. singular” in English, which is hopefully a thing you’re familiar with as you’re reading this blog.

Some functions in Perl act differently depending on whether the expected return value is a list or a scalar, and a function will provide a list or scalar context to its arguments. Mostly these act just as you would expect or would like them to, and you can find out how a function behaves by reading its documentation. Your own functions can behave like this too, but there’s usually no need as both scalars and lists are automatically interpreted into lists.” Again, Perl’s DWIMmery at work.

Subroutine and Method Arguments

I’ve already written about this. Twice. And presented about it. Twice. The short version: Perl has signatures, but they’ve been considered experimental for a while. In the meantime, there are alternatives on CPAN. You can even have type constraints if you want.


I’ll leave you with this: Over the past month, Neil Bowers of the Perl Steering Council has been collecting quirks like these from Perl developers. The PSC is reviewing this collection for potential documentation fixes, bug fixes, further discussion, etc. I wouldn’t expect to see any fundamental changes to the language out of this effort, but it’s a good sign that potentially confusing features are being addressed. 

person doing card trick

Perl is said (sometimes frustratingly) to be a do-​what-​I-​mean programming language. Many of its statements and constructions are designed to be forgiving or have analogies to natural languages. Still others are said to be magic,” behaving differently depending on how they’re used. Adept use of Perl asks you to not only understand this magic, but to embrace it and the expressiveness it enables. Here, then, are five ways you can bring some magic to your code.

$_

Perl has many special variables, and first among them (literally, it’s the first documented) is $_. Also spelled $ARG if you use the English module, the documentation describes it as the default input and pattern-​matching space.” Many, many functions and statements will assume it as the default or implicit argument; you can find the full list in the documentation. Here’s an example that uses it implicitly to output the numbers from 1 to 5:

say for 1 .. 5;

Output:

1
2
3
4
5

Where some languages require an iterator variable in a for or foreach loop, in the absence of one Perl assigns it to $_.

Statement modifiers

We then use our second trick; where some other languages require a block to enclose every loop or conditional (whether denoted by braces { } or indentation), Perl allows you to put said looping or conditional statement after a single other statement, in this case the say which prints its argument(s) followed by a newline.

However, above we have no arguments passed to say and so once again the default $_ is used, now containing a number from 1 to 5 which is then printed out. It’s a very powerful and expressive idiom, enabling both the writer and reader of code to concentrate on the important thing that’s happening. It’s also entirely optional. You can just as easily type:

for my $foo (1..5) {
    say $foo;
}

But where’s the magic in that?

Magic variables and use English

We mentioned the $_ variable above, and that it could also be spelled $ARG if you add use English to your code. It can be hard to read code with large amounts of punctuation, though, and even harder to remember what each variable does. Thankfully the English module provides aliases, and the perlvar man page lists them in order. It’s much easier to read and write things like $LIST_SEPARATOR, $PROCESS_ID, or $MATCH rather than $", $$, and $&, and goes a long way towards reducing Perl’s reputation as a write-​only language.

List and scalar contexts

Like natural languages, Perl has a concept of context” in which words mean different things depending on their surroundings. In Perl’s case, expressions may behave differently depending on whether they expect to produce a list of values or a single value, called a scalar. Here’s a trivial example:

my @foo = (1, 2, 3); # list context, @foo contains the list
my $bar = (1, 2, 3); # scalar context, $bar contains 3

In the first line, we assign the list of numbers (1, 2, 3) to the array @foo. But in the second line, we’re assigning to the scalar variable $bar, which now contains the last item in the list.

Here’s another example, using the reverse function:

my @foo = ('one', 'two', 'three');
my @bar = reverse @foo; # @bar contains ('three', 'two', 'one')
my $baz = reverse @foo; # $baz contains 'eerhtowteno'

In list context, reverse takes its arguments and returns them in the opposite order. But in scalar context, it concatenates all of the arguments together and returns a string with the characters in opposite order.

In general, there is no general rule for deducing a function’s behavior in scalar context from its behavior in list context.” (Dominus 1998) You’ll just have to look up the function to determine what it does, though in general, it does what you want, but if you want to force scalar context use the scalar operator:

my @foo = ('aa', 'aab', 'bbc');
my @bar = scalar grep /aa/, @foo; # returns a list (2), counting the number of matches

Hash slices

One of Perl’s three built-​in data types is the hash, also known as an associative array. It’s an unordered collection of scalars indexed by string, rather than the numbers used by normal arrays. It’s a useful construct, and you can develop complicated data structures using just scalars, arrays, and hashes. What’s not widely known is that you can access several elements of of a hash using a hash slice, using syntax that’s similar to array slices. Here’s an example:

my ($who, $home) = @ENV{'USER', 'HOME'};

It works the other way, too: you can assign to a slice.

@colors{'red', 'green', 'blue'} = (0xff0000, 0x00ff00, 0x0000ff);

I use this a lot when assigning arguments received from functions or methods (see my previous article on subroutine signatures):

use v5.24; # for postfix dereferencing
use Types::Standard qw(Str Int);
use Type::Params 'compile_named';

foo('hello', 42);

sub foo {
    state $check = compile_named(
        param1 => Str,
        param2 => Int, {optional => 1},
    );
    my ($param1, $param2) =
        $check->(@_)->@{'param1', 'param2'};

    say $param1, $param2;
}

In the example above, $check->(@_) returns the type-​checked arguments to the foo() function courtesy of Type::Paramscompile_named() function. It’s returned as a hash reference, and since hashes are unordered, we specify the order in which we want the values by dereferencing and then slicing the resulting hash. The postfix dereferencing syntax was added in Perl 5.20 and made a default feature in 5.24, and reduces the number of nested brackets and braces we have to deal with.

Conclusion

I hope this article has given you a taste of some of the magic available in the Perl language. It’s these sort of features that make programming in it a bit more joyful. As always, check the documentation for complete information on these and other topics, or look for answers and ask questions on PerlMonks or Stack Overflow.