This week’s Perl and Raku Conference 2022 in Houston was packed with great pre­sen­ta­tions, and I humbly added to them with a five-​ish minute light­ning talk on two of Perl’s more mis­un­der­stood func­tions: map and grep.

Sorry about the um”s and ah”s…

I’ve writ­ten much about list pro­cess­ing in Perl, and this talk was based on the fol­low­ing blog posts:

Overall I loved attend­ing the con­fer­ence, and it real­ly invig­o­rat­ed my par­tic­i­pa­tion in the Perl com­mu­ni­ty. Stay tuned as I resume reg­u­lar posting!

Update for Raku

On Twitter I nudged promi­nent Raku hack­er (and recov­ered Perl hack­er) Elizabeth Mattijsen to write about the Raku pro­gram­ming language’s map and grep func­tion­al­i­ty. Check out her five-​part series on

plate of eggs and hash browns

This month I start­ed a new job at Alert Logic, a cyber­se­cu­ri­ty provider with Perl (among many oth­er things) at its beat­ing heart. I’ve been learn­ing a lot, and part of the process has been under­stand­ing the APIs in the code base. To that end, I’ve been writ­ing small test scripts to tease apart data struc­tures, using Perl array-​processing, list-​processing, and hash- (i.e., asso­cia­tive array)-processing func­tions.

I’ve cov­ered map, grep, and friends a cou­ple times before. Most recent­ly, I described using List::Util’s any func­tion to check if a con­di­tion is true for any item in a list. In the sim­plest case, you can use it to check to see if a giv­en val­ue is in the list at all:

use feature 'say';
use List::Util 'any';
my @colors =
  qw(red orange yellow green blue indigo violet);
say 'matched' if any { /^red$/ } @colors;

However, if you’re going to be doing this a lot with arbi­trary strings, Perl FAQ sec­tion 4 advis­es turn­ing the array into the keys of a hash and then check­ing for mem­ber­ship there. For exam­ple, here’s a sim­ple script to check if the col­ors input (either from the key­board or from files passed as argu­ments) are in the rainbow:

#!/usr/bin/env perl

use v5.22; # introduced <<>> for safe opening of arguments
use warnings;
my %in_colors = map {$_ => 1}
  qw(red orange yellow green blue indigo violet);

while (<<>>) {
  say "$_ is in the rainbow" if $in_colors{$_};

List::Util has a bunch of func­tions for pro­cess­ing lists of pairs that I’ve found use­ful when paw­ing through hash­es. pairgrep, for exam­ple, acts just like grep but instead assigns $a and $b to each key and val­ue passed in and returns the result­ing pairs that match. I’ve used it as a quick way to search for hash entries match­ing cer­tain val­ue conditions:

use List::Util 'pairgrep';
my %numbers = (zero => 0, one => 1, two => 2, three => 3);
my %odds = pairgrep {$b % 2} %numbers;

Sure, you could do this by invok­ing a mix of plain grep, keys, and a hash slice, but it’s nois­i­er and more repetitive:

use v5.20; # for key/value hash slice 
my %odds = %numbers{grep {$numbers{$_} % 2} keys %numbers};

pairgreps com­piled C‑based XS code can also be faster, as evi­denced by this Benchmark script that works through a hash made of the Unix words file (479,828 entries on my machine):

#!/usr/bin/env perl

use v5.20;
use warnings;
use List::Util 'pairgrep';
use Benchmark 'cmpthese';

my (%words, $count);
open my $fh, '<', '/usr/share/dict/words'
  or die "can't open words: $!";
while (<$fh>) {
  $words{$_} = $count++;
close $fh;

cmpthese(100, {
  grep => sub {
    my %odds = %words{grep {$words{$_} % 2} keys %words};
  pairgrep => sub {
    my %odds = pairgrep {$b % 2} %words;
} );

Benchmark out­put:

           Rate     grep pairgrep
grep     1.47/s       --     -20%
pairgrep 1.84/s      25%       --

In gen­er­al, I urge you to work through the Perl doc­u­men­ta­tions tuto­ri­als on ref­er­ences, lists of lists, the data struc­tures cook­book, and the FAQs on array and hash manip­u­la­tion. Then dip into the var­i­ous list-​processing mod­ules (espe­cial­ly the includ­ed List::Util and CPAN’s List::SomeUtils) for ready-​made func­tions for com­mon oper­a­tions. You’ll find a wealth of tech­niques for cre­at­ing, man­ag­ing, and pro­cess­ing the data struc­tures that your pro­grams need.

woman looking at the map

Six months ago I gave an overview of Perl’s list pro­cess­ing fun­da­men­tals, briefly describ­ing what lists are and then intro­duc­ing the built-​in map and grep func­tions for trans­form­ing and fil­ter­ing them. Later on, I com­piled a list (how appro­pri­ate) of list pro­cess­ing mod­ules avail­able via CPAN, not­ing there’s some con­fus­ing dupli­ca­tion of effort. But you’re a busy devel­op­er, and you just want to know the Right Thing To Do™ when faced with a list pro­cess­ing challenge.

First, some cred­it is due: these are all restate­ments of sev­er­al Perl::Critic poli­cies which in turn cod­i­fy stan­dards described in Damian Conway’s Perl Best Practices (2005). I’ve repeat­ed­ly rec­om­mend­ed the lat­ter as a start­ing point for higher-​quality Perl devel­op­ment. Over the years these prac­tices con­tin­ue to be re-​evaluated (includ­ing by the author him­self) and var­i­ous authors release new pol­i­cy mod­ules, but perlcritic remains a great tool for ensur­ing you (and your team or oth­er con­trib­u­tors) main­tain a con­sis­tent high stan­dard in your code.

With that said, on to the recommendations!

Don’t use grep to check if any list elements match

It might sound weird to lead off by rec­om­mend­ing not to use grep, but some­times it’s not the right tool for the job. If you’ve got a list and want to deter­mine if a con­di­tion match­es any item in it, you might try:

if (grep { some_condition($_) } @my_list) {
    ... # don't do this!

Yes, this works because (in scalar con­text) grep returns the num­ber of match­es found, but it’s waste­ful, check­ing every ele­ment of @my_list (which could be lengthy) before final­ly pro­vid­ing a result. Use the stan­dard List::Util module’s any func­tion, which imme­di­ate­ly returns (“short-​circuits”) on the first match:

use List::Util 1.33 qw(any);

if (any { some_condition($_) } @my_list) {
... # do something

Perl has includ­ed the req­ui­site ver­sion of this mod­ule since ver­sion 5.20 in 2014; for ear­li­er releas­es, you’ll need to update from CPAN. List::Util has many oth­er great list-​reduction, key/​value pair, and oth­er relat­ed func­tions you can import into your code, so check it out before you attempt to re-​invent any wheels.

As a side note for web devel­op­ers, the Perl Dancer frame­work also includes an any key­word for declar­ing mul­ti­ple HTTP routes, so if you’re mix­ing List::Util in there don’t import it. Instead, call it explic­it­ly like this or you’ll get an error about a rede­fined function:

use List::Util 1.33;

if (List::Util::any { some_condition($_) } @my_list) {
... # do something

This rec­om­men­da­tion is cod­i­fied in the BuiltinFunctions::ProhibitBooleanGrep Perl::Critic pol­i­cy, comes direct­ly from Perl Best Practices, and is rec­om­mend­ed by the Software Engineering Institute Computer Emergency Response Team (SEI CERT)’s Perl Coding Standard.

Don’t change $_ in map or grep

I men­tioned this back in March, but it bears repeat­ing: map and grep are intend­ed as pure func­tions, not muta­tors with side effects. This means that the orig­i­nal list should remain unchanged. Yes, each ele­ment alias­es in turn to the $_ spe­cial vari­able, but that’s for speed and can have sur­pris­ing results if changed even if it’s tech­ni­cal­ly allowed. If you need to mod­i­fy an array in-​place use some­thing like:

for (@my_array) {
$_ = ...; # make your changes here

If you want some­thing that looks like map but won’t change the orig­i­nal list (and don’t mind a few CPAN depen­den­cies), con­sid­er List::SomeUtilsapply function:

use List::SomeUtils qw(apply);

my @doubled_array = apply {$_ *= 2} @old_array;

Lastly, side effects also include things like manip­u­lat­ing oth­er vari­ables or doing input and out­put. Don’t use map or grep in a void con­text (i.e., with­out a result­ing array or list); do some­thing with the results or use a for or foreach loop:

map { print foo($_) } @my_array; # don't do this
print map { foo($_) } @my_array; # do this instead

map { push @new_array, foo($_) } @my_array; # don't do this
@new_array = map { foo($_) } @my_array; # do this instead

This rec­om­men­da­tion is cod­i­fied by the BuiltinFunctions::ProhibitVoidGrep, BuiltinFunctions::ProhibitVoidMap, and ControlStructures::ProhibitMutatingListFunctions Perl::Critic poli­cies. The lat­ter comes from Perl Best Practices and is an SEI CERT Perl Coding Standard rule.

Use blocks with map and grep, not expressions

You can call map or grep like this (paren­the­ses are option­al around built-​in functions):

my @new_array  = map foo($_), @old_array; # don't do this
my @new_array2 = grep !/^#/, @old_array; # don't do this

Or like this:

my @new_array  = map { foo($_) } @old_array;
my @new_array2 = grep {!/^#/} @old_array;

Do it the sec­ond way. It’s eas­i­er to read, espe­cial­ly if you’re pass­ing in a lit­er­al list or mul­ti­ple arrays, and the expres­sion forms can con­ceal bugs. This rec­om­men­da­tion is cod­i­fied by the BuiltinFunctions::RequireBlockGrep and BuiltinFunctions::RequireBlockMap Perl::Critic poli­cies and comes from Perl Best Practices.

Refactor multi-​statement maps, greps, and other list functions

map, grep, and friends should fol­low the Unix phi­los­o­phy of Do One Thing and Do It Well.” Your read­abil­i­ty and main­tain­abil­i­ty drop with every state­ment you place inside one of their blocks. Consider junior devel­op­ers and future main­tain­ers (this includes you!) and refac­tor any­thing with more than one state­ment into a sep­a­rate sub­rou­tine or at least a for loop. This goes for list pro­cess­ing func­tions (like the afore­men­tioned any) import­ed from oth­er mod­ules, too.

This rec­om­men­da­tion is cod­i­fied by the Perl Best Practices-inspired BuiltinFunctions::ProhibitComplexMappings and BuiltinFunctions::RequireSimpleSortBlock Perl::Critic poli­cies, although those only cov­er map and sort func­tions, respectively.

Do you have any oth­er sug­ges­tions for list pro­cess­ing best prac­tices? Feel free to leave them in the com­ments or bet­ter yet, con­sid­er cre­at­ing new Perl::Critic poli­cies for them or con­tact­ing the Perl::Critic team to devel­op them for your organization.

Lister Platz

As pre­vi­ous­ly writ­ten, I like list pro­cess­ing. Many com­put­ing prob­lems can be bro­ken down into trans­form­ing and fil­ter­ing lists, and Perl has got the fun­da­men­tals cov­ered with func­tions like map, grep, and sort. There is so much more you might want to do, though, and CPAN has a pletho­ra of list and array pro­cess­ing modules.

However, due to the vicis­si­tudes of Perl mod­ule main­te­nance, we have a sit­u­a­tion where it’s not clear at a glance where to turn when you’ve got a list that needs pro­cess­ing. So here’s anoth­er list: the list mod­ules of CPAN. Click through to dis­cov­er what func­tions they provide.

  • We’ve got List::Util which has been released as part of Perl since ver­sion 5.7.3.
  • We’ve got List::MoreUtils which has some func­tions which are named the same as Util but behave differently.
  • We’ve got List::SomeUtils which dupli­cates MoreUtils but with few­er dependencies.
  • We’ve got List::UtilsBy which MoreUtils has also cribbed some func­tions from.
  • We’ve got List::AllUtils which attempts to con­sol­i­date Util, SomeUtils, and ListBy but has some excep­tions to called mod­ules because of the afore­men­tioned dupli­ca­tion between Util and SomeUtils.
  • We’ve got List::Util::MaybeXS which helps with pure Perl fall­backs in case your ver­sion of Util is too old to have a cer­tain function.
  • We’ve got List::MoreUtils::XS which pro­vides (some?) faster ver­sions of MoreUtils’ func­tions (but you still have to use MoreUtils).
  • And last­ly, we have Util::Any which lets you import func­tions from Util, MoreUtils, and just for good mea­sure Scalar::Util, Hash::Util, String::Util, String::CamelCase, List::Pairwise, and Data::Dumper. But it has­n’t been updat­ed since 2016, so it does­n’t nec­es­sar­i­ly export the func­tions added to those mod­ules since then.

Am I miss­ing any­thing? Probably! But these are the ones most asso­ci­at­ed with being upstream on the CPAN River, so they (or the mod­ules they con­sol­i­date) have more projects depend­ing on them.


As a Perl devel­op­er, you’re prob­a­bly aware of the lan­guage’s strengths as a text-​processing lan­guage and how many com­put­ing tasks can be bro­ken down into those types of tasks. You might not real­ize, though, that Perl is also a world-​class list pro­cess­ing lan­guage and that many prob­lems can be expressed in terms of lists and their transformations.

Chief among Perl’s tools for list pro­cess­ing are the func­tions map and grep. I can’t count how many times in my twenty-​five years as a devel­op­er I’ve run into code that could’ve been sim­pli­fied if only the author was famil­iar with these two func­tions. Once you under­stand map and grep, you’ll start see­ing lists every­where and the oppor­tu­ni­ty to make your code more suc­cinct and expres­sive at the same time.

What are lists?

Before we get into func­tions that manip­u­late lists, we need to under­stand what they are. A list is an ordered group of ele­ments, and those ele­ments can be any kind of data you can rep­re­sent in the lan­guage: num­bers, strings, objects, reg­u­lar expres­sions, ref­er­ences, etc., as long as they’re stored as scalars. You might think of a list as the thing that an array stores, and in fact Perl is fine with using an array where a list can go.

my @foo = (1, 2, 3);

Here we’re assign­ing the list of num­bers from 1 to 3 to the array @foo. The dif­fer­ence between the array and the list is that the list is a fixed col­lec­tion, while arrays and their ele­ments can be mod­i­fied by var­i­ous oper­a­tions. perlfaq4 has a great dis­cus­sion on the dif­fer­ences between the two.

Lists are everywhere, man!

Ever want­ed to sort some data? You were using a list.

join a bunch of things togeth­er into a string? List again.

split a string into pieces? You got a list back (in list con­text; in scalar con­text, you got the size of the list.)

Heck, even the hum­ble print func­tion and its cousin say take a list (and an option­al file­han­dle) as argu­ments; it’s why you can treat Perl as an upscale AWK and feed it scalars to out­put with a field sep­a­ra­tor.

You’re using lists all the time and may not even know it.

map: The list transformer

The map func­tion is devi­ous in its sim­plic­i­ty: It takes two inputs, an expres­sion or block of code, and a list to run it on. For every item in the list, it will alias $_ to it, and then return none, one, or many items in a list based on what hap­pens in the expres­sion or code block. You can call it like this:

my @foo = map bar($_), @list;

Or like this:

my @foo = map { bar($_) } @list;

We’re going to ignore the first way, though because Conway (Perl Best Practices, 2005) tells us that when you spec­i­fy the first argu­ment as an expres­sion, it’s hard­er to tell it apart from the remain­ing argu­ments, espe­cial­ly if that expres­sion uses a built-​in func­tion where the paren­the­ses are option­al. So always use a code block!

You should always turn to map (and not, say, a for or foreach loop) when gen­er­at­ing a new list from an old list. For example:

my @lowercased = map { lc } @mixed_case;

When paired with a lookup table, map is also the most effi­cient way to tell if a mem­ber of a list equals a string, espe­cial­ly if that list is static:

use Const::Fast;

const my %IS_EXIT_WORD => map { ($_ => 1) }
  qw(q quit bye exit stop done last finish aurevoir);


die if $IS_EXIT_WORD{$command};

Here we’re using maps abil­i­ty to return mul­ti­ple items per source ele­ment to gen­er­ate a con­stant hash, and then test­ing mem­ber­ship in that hash.

grep: The list filter

You may rec­og­nize the word grep” from the Unix com­mand of the same name. It’s a tool for find­ing lines of text inside of oth­er text using a reg­u­lar expres­sion describ­ing the desired result.

Perl, of course, is real­ly good at reg­u­lar expres­sions, but its grep func­tion goes beyond and enables you to match using any expres­sion or code block. Think of it as a part­ner to map; where map uses a code block to trans­form a list, grep uses one to fil­ter it down. In fact, oth­er lan­guages typ­i­cal­ly call this func­tion filter.

You can, of course, use reg­u­lar expres­sions with grep, espe­cial­ly because a reg­exp match in Perl defaults to match­ing on the $_ vari­able and grep hap­pens to pro­vide that to its code block argu­ment. So:

my @months_with_a = grep { /[Aa]/ } qw(
  January February March
  April   May      June
  July    August   September
  October November December

But grep real­ly comes into its own when used for its gen­er­al fil­ter­ing capa­bil­i­ties; for instance, mak­ing sure that you don’t acci­den­tal­ly try to com­pare an unde­fined value:

say $_ > 5
  ? "$_ is bigger"
  : "$_ is equal or smaller"
  for grep { defined } @numbers;

Or when exe­cut­ing a com­pli­cat­ed func­tion that returns true or false depend­ing on its arguments:

my @results = grep { really_large_database_query($_) }

You might even con­sid­er chain­ing map and grep togeth­er. Here’s an exam­ple for get­ting the JPEG images out of a file list and then low­er­cas­ing the results:

my @jpeg_files = map  { lc }
                grep { /\.jpe?g$/i } @files;

Side effects may include…” (updated)

When intro­duc­ing map above I not­ed that it aliased $_ for every ele­ment in the list. I used that term delib­er­ate­ly because mod­i­fi­ca­tions to $_ will mod­i­fy the orig­i­nal ele­ment itself, and that is usu­al­ly an error. Programmers call that a side effect,” and they can lead to unex­pect­ed behav­ior or at least difficult-​to-​maintain code. Consider:

my @needs_docs = grep { s/\.pm$/.pod/ && !-e }

The intent may have been to find files end­ing in .pm that don’t have a cor­re­spond­ing .pod file, but the actu­al behav­ior is replac­ing the .pm suf­fix with .pod, then check­ing whether that file­name exists. If it does­n’t, it’s passed through to @needs_docs; regard­less, @pm_files has had its con­tents modified.

If you real­ly do need to mod­i­fy a copy of each ele­ment, assign a vari­able with­in your code block like this:

my @needs_docs = grep {
                   my $file = $_;
                   $file =~ s/\.pm$/.pod/;
                   !-e $file
                 } @pm_files;

But at that point you should prob­a­bly refac­tor your multi-​line block as a sep­a­rate function:

my @needs_docs = grep { file_without_docs($_) }

sub file_without_docs {
    my $file = shift;
    $file =~ s/\.pm$/.pod/;
    return !-e $file;

In this case of using the sub­sti­tu­tion oper­a­tor s///, you could also do this when using Perl 5.14 or above to get non-​destructive sub­sti­tu­tion:

use v5.14;

my @needs_docs = grep { !-e s/\.pm$/.pod/r }

And if you do need side effects, just use a for or foreach loop; future code main­tain­ers (i.e., you in six months) will thank you.

Taking you higher

map and grep are exam­ples of higher-​order func­tions, since they take a func­tion (in the form of a code block) as an argu­ment. So con­grat­u­la­tions, you just sig­nif­i­cant­ly lev­eled up your knowl­edge of Perl and com­put­er sci­ence. If you’re inter­est­ed in more such pro­gram­ming tech­niques, I rec­om­mend Mark Jason Dominus’ Higher Order Perl (2005), avail­able for free online.