Lister Platz

As pre­vi­ous­ly writ­ten, I like list pro­cess­ing. Many com­put­ing prob­lems can be bro­ken down into trans­form­ing and fil­ter­ing lists, and Perl has got the fun­da­men­tals cov­ered with func­tions like map, grep, and sort. There is so much more you might want to do, though, and CPAN has a pletho­ra of list and array pro­cess­ing modules.

However, due to the vicis­si­tudes of Perl mod­ule main­te­nance, we have a sit­u­a­tion where it’s not clear at a glance where to turn when you’ve got a list that needs pro­cess­ing. So here’s anoth­er list: the list mod­ules of CPAN. Click through to dis­cov­er what func­tions they provide.

  • We’ve got List::Util which has been released as part of Perl since ver­sion 5.7.3.
  • We’ve got List::MoreUtils which has some func­tions which are named the same as Util but behave differently.
  • We’ve got List::SomeUtils which dupli­cates MoreUtils but with few­er dependencies.
  • We’ve got List::UtilsBy which MoreUtils has also cribbed some func­tions from.
  • We’ve got List::AllUtils which attempts to con­sol­i­date Util, SomeUtils, and ListBy but has some excep­tions to called mod­ules because of the afore­men­tioned dupli­ca­tion between Util and SomeUtils.
  • We’ve got List::Util::MaybeXS which helps with pure Perl fall­backs in case your ver­sion of Util is too old to have a cer­tain function.
  • We’ve got List::MoreUtils::XS which pro­vides (some?) faster ver­sions of MoreUtils’ func­tions (but you still have to use MoreUtils).
  • And last­ly, we have Util::Any which lets you import func­tions from Util, MoreUtils, and just for good mea­sure Scalar::Util, Hash::Util, String::Util, String::CamelCase, List::Pairwise, and Data::Dumper. But it has­n’t been updat­ed since 2016, so it does­n’t nec­es­sar­i­ly export the func­tions added to those mod­ules since then.

Am I miss­ing any­thing? Probably! But these are the ones most asso­ci­at­ed with being upstream on the CPAN River, so they (or the mod­ules they con­sol­i­date) have more projects depend­ing on them.

As a Perl devel­op­er, you’re prob­a­bly aware of the lan­guage’s strengths as a text-​processing lan­guage and how many com­put­ing tasks can be bro­ken down into those types of tasks. You might not real­ize, though, that Perl is also a world-​class list pro­cess­ing lan­guage and that many prob­lems can be expressed in terms of lists and their transformations.

Chief among Perl’s tools for list pro­cess­ing are the func­tions map and grep. I can’t count how many times in my twenty-​five years as a devel­op­er I’ve run into code that could’ve been sim­pli­fied if only the author was famil­iar with these two func­tions. Once you under­stand map and grep, you’ll start see­ing lists every­where and the oppor­tu­ni­ty to make your code more suc­cinct and expres­sive at the same time.

What are lists?

Before we get into func­tions that manip­u­late lists, we need to under­stand what they are. A list is an ordered group of ele­ments, and those ele­ments can be any kind of data you can rep­re­sent in the lan­guage: num­bers, strings, objects, reg­u­lar expres­sions, ref­er­ences, etc., as long as they’re stored as scalars. You might think of a list as the thing that an array stores, and in fact Perl is fine with using an array where a list can go.

my @foo = (1, 2, 3);

Here we’re assign­ing the list of num­bers from 1 to 3 to the array @foo. The dif­fer­ence between the array and the list is that the list is a fixed col­lec­tion, while arrays and their ele­ments can be mod­i­fied by var­i­ous oper­a­tions. perlfaq4 has a great dis­cus­sion on the dif­fer­ences between the two.

Lists are everywhere, man!

Ever want­ed to sort some data? You were using a list.

join a bunch of things togeth­er into a string? List again.

split a string into pieces? You got a list back (in list con­text; in scalar con­text, you got the size of the list.)

Heck, even the hum­ble print func­tion and its cousin say take a list (and an option­al file­han­dle) as argu­ments; it’s why you can treat Perl as an upscale AWK and feed it scalars to out­put with a field sep­a­ra­tor.

You’re using lists all the time and may not even know it.

map: The list transformer

The map func­tion is devi­ous in its sim­plic­i­ty: It takes two inputs, an expres­sion or block of code, and a list to run it on. For every item in the list, it will alias $_ to it, and then return none, one, or many items in a list based on what hap­pens in the expres­sion or code block. You can call it like this:

my @foo = map bar($_), @list;

Or like this:

my @foo = map { bar($_) } @list;

We’re going to ignore the first way, though because Conway (Perl Best Practices, 2005) tells us that when you spec­i­fy the first argu­ment as an expres­sion, it’s hard­er to tell it apart from the remain­ing argu­ments, espe­cial­ly if that expres­sion uses a built-​in func­tion where the paren­the­ses are option­al. So always use a code block!

You should always turn to map (and not, say, a for or foreach loop) when gen­er­at­ing a new list from an old list. For example:

my @lowercased = map { lc } @mixed_case;

When paired with a lookup table, map is also the most effi­cient way to tell if a mem­ber of a list equals a string, espe­cial­ly if that list is static:

use Const::Fast;

const my %IS_EXIT_WORD => map { ($_ => 1) }
  qw(q quit bye exit stop done last finish aurevoir);

...

die if $IS_EXIT_WORD{$command};

Here we’re using maps abil­i­ty to return mul­ti­ple items per source ele­ment to gen­er­ate a con­stant hash, and then test­ing mem­ber­ship in that hash.

grep: The list filter

You may rec­og­nize the word grep” from the Unix com­mand of the same name. It’s a tool for find­ing lines of text inside of oth­er text using a reg­u­lar expres­sion describ­ing the desired result.

Perl, of course, is real­ly good at reg­u­lar expres­sions, but its grep func­tion goes beyond and enables you to match using any expres­sion or code block. Think of it as a part­ner to map; where map uses a code block to trans­form a list, grep uses one to fil­ter it down. In fact, oth­er lan­guages typ­i­cal­ly call this func­tion filter.

You can, of course, use reg­u­lar expres­sions with grep, espe­cial­ly because a reg­exp match in Perl defaults to match­ing on the $_ vari­able and grep hap­pens to pro­vide that to its code block argu­ment. So:

my @months_with_a = grep { /[Aa]/ } qw(
  January February March
  April   May      June
  July    August   September
  October November December
);

But grep real­ly comes into its own when used for its gen­er­al fil­ter­ing capa­bil­i­ties; for instance, mak­ing sure that you don’t acci­den­tal­ly try to com­pare an unde­fined value:

say $_ > 5
  ? "$_ is bigger"
  : "$_ is equal or smaller"
  for grep { defined } @numbers;

Or when exe­cut­ing a com­pli­cat­ed func­tion that returns true or false depend­ing on its arguments:

my @results = grep { really_large_database_query($_) }
              @foo;

You might even con­sid­er chain­ing map and grep togeth­er. Here’s an exam­ple for get­ting the JPEG images out of a file list and then low­er­cas­ing the results:

my @jpeg_files = map  { lc }
                grep { /\.jpe?g$/i } @files;

Side effects may include…” (updated)

When intro­duc­ing map above I not­ed that it aliased $_ for every ele­ment in the list. I used that term delib­er­ate­ly because mod­i­fi­ca­tions to $_ will mod­i­fy the orig­i­nal ele­ment itself, and that is usu­al­ly an error. Programmers call that a side effect,” and they can lead to unex­pect­ed behav­ior or at least difficult-​to-​maintain code. Consider:

my @needs_docs = grep { s/\.pm$/.pod/ && !-e }
                 @pm_files;

The intent may have been to find files end­ing in .pm that don’t have a cor­re­spond­ing .pod file, but the actu­al behav­ior is replac­ing the .pm suf­fix with .pod, then check­ing whether that file­name exists. If it does­n’t, it’s passed through to @needs_docs; regard­less, @pm_files has had its con­tents modified.

If you real­ly do need to mod­i­fy a copy of each ele­ment, assign a vari­able with­in your code block like this:

my @needs_docs = grep {
                   my $file = $_;
                   $file =~ s/\.pm$/.pod/;
                   !-e $file
                 } @pm_files;

But at that point you should prob­a­bly refac­tor your multi-​line block as a sep­a­rate function:

my @needs_docs = grep { file_without_docs($_) }
                 @pm_files;

sub file_without_docs {
    my $file = shift;
    $file =~ s/\.pm$/.pod/;
    return !-e $file;
}

In this case of using the sub­sti­tu­tion oper­a­tor s///, you could also do this when using Perl 5.14 or above to get non-​destructive sub­sti­tu­tion:

use v5.14;

my @needs_docs = grep { !-e s/\.pm$/.pod/r }
                 @pm_files;

And if you do need side effects, just use a for or foreach loop; future code main­tain­ers (i.e., you in six months) will thank you.

Taking you higher

map and grep are exam­ples of higher-​order func­tions, since they take a func­tion (in the form of a code block) as an argu­ment. So con­grat­u­la­tions, you just sig­nif­i­cant­ly lev­eled up your knowl­edge of Perl and com­put­er sci­ence. If you’re inter­est­ed in more such pro­gram­ming tech­niques, I rec­om­mend Mark Jason Dominus’ Higher Order Perl (2005), avail­able for free online.

person doing card trick

Perl is said (some­times frus­trat­ing­ly) to be a do-​what-​I-​mean pro­gram­ming lan­guage. Many of its state­ments and con­struc­tions are designed to be for­giv­ing or have analo­gies to nat­ur­al lan­guages. Still oth­ers are said to be mag­ic,” behav­ing dif­fer­ent­ly depend­ing on how they’re used. Adept use of Perl asks you to not only under­stand this mag­ic, but to embrace it and the expres­sive­ness it enables. Here, then, are five ways you can bring some mag­ic to your code.

$_

Perl has many spe­cial vari­ables, and first among them (lit­er­al­ly, it’s the first doc­u­ment­ed) is $_. Also spelled $ARG if you use the English mod­ule, the doc­u­men­ta­tion describes it as the default input and pattern-​matching space.” Many, many func­tions and state­ments will assume it as the default or implic­it argu­ment; you can find the full list in the doc­u­men­ta­tion. Here’s an exam­ple that uses it implic­it­ly to out­put the num­bers from 1 to 5:

say for 1 .. 5;

Output:

1
2
3
4
5

Where some lan­guages require an iter­a­tor vari­able in a for or foreach loop, in the absence of one Perl assigns it to $_.

Statement modifiers

We then use our sec­ond trick; where some oth­er lan­guages require a block to enclose every loop or con­di­tion­al (whether denot­ed by braces { } or inden­ta­tion), Perl allows you to put said loop­ing or con­di­tion­al state­ment after a sin­gle oth­er state­ment, in this case the say which prints its argument(s) fol­lowed by a newline.

However, above we have no argu­ments passed to say and so once again the default $_ is used, now con­tain­ing a num­ber from 1 to 5 which is then print­ed out. It’s a very pow­er­ful and expres­sive idiom, enabling both the writer and read­er of code to con­cen­trate on the impor­tant thing that’s hap­pen­ing. It’s also entire­ly option­al. You can just as eas­i­ly type:

for my $foo (1..5) {
    say $foo;
}

But where’s the mag­ic in that?

Magic variables and use English

We men­tioned the $_ vari­able above, and that it could also be spelled $ARG if you add use English to your code. It can be hard to read code with large amounts of punc­tu­a­tion, though, and even hard­er to remem­ber what each vari­able does. Thankfully the English mod­ule pro­vides alias­es, and the per­l­var man page lists them in order. It’s much eas­i­er to read and write things like $LIST_SEPARATOR, $PROCESS_ID, or $MATCH rather than $", $$, and $&, and goes a long way towards reduc­ing Perl’s rep­u­ta­tion as a write-​only language.

List and scalar contexts

Like nat­ur­al lan­guages, Perl has a con­cept of con­text” in which words mean dif­fer­ent things depend­ing on their sur­round­ings. In Perl’s case, expres­sions may behave dif­fer­ent­ly depend­ing on whether they expect to pro­duce a list of val­ues or a sin­gle val­ue, called a scalar. Here’s a triv­ial example:

my @foo = (1, 2, 3); # list context, @foo contains the list
my $bar = (1, 2, 3); # scalar context, $bar contains 3

In the first line, we assign the list of num­bers (1, 2, 3) to the array @foo. But in the sec­ond line, we’re assign­ing to the scalar vari­able $bar, which now con­tains the last item in the list.

Here’s anoth­er exam­ple, using the reverse function:

my @foo = ('one', 'two', 'three');
my @bar = reverse @foo; # @bar contains ('three', 'two', 'one')
my $baz = reverse @foo; # $baz contains 'eerhtowteno'

In list con­text, reverse takes its argu­ments and returns them in the oppo­site order. But in scalar con­text, it con­cate­nates all of the argu­ments togeth­er and returns a string with the char­ac­ters in oppo­site order.

In gen­er­al, there is no gen­er­al rule for deduc­ing a func­tion’s behav­ior in scalar con­text from its behav­ior in list con­text.” (Dominus 1998) You’ll just have to look up the func­tion to deter­mine what it does, though in gen­er­al, it does what you want, but if you want to force scalar con­text use the scalar operator:

my @foo = ('aa', 'aab', 'bbc');
my @bar = scalar grep /aa/, @foo; # returns a list (2), counting the number of matches

Hash slices

One of Perl’s three built-​in data types is the hash, also known as an asso­cia­tive array. It’s an unordered col­lec­tion of scalars indexed by string, rather than the num­bers used by nor­mal arrays. It’s a use­ful con­struct, and you can devel­op com­pli­cat­ed data struc­tures using just scalars, arrays, and hash­es. What’s not wide­ly known is that you can access sev­er­al ele­ments of of a hash using a hash slice, using syn­tax that’s sim­i­lar to array slices. Here’s an example:

my ($who, $home) = @ENV{'USER', 'HOME'};

It works the oth­er way, too: you can assign to a slice.

@colors{'red', 'green', 'blue'} = (0xff0000, 0x00ff00, 0x0000ff);

I use this a lot when assign­ing argu­ments received from func­tions or meth­ods (see my pre­vi­ous arti­cle on sub­rou­tine sig­na­tures):

use v5.24; # for postfix dereferencing
use Types::Standard qw(Str Int);
use Type::Params 'compile_named';

foo('hello', 42);

sub foo {
    state $check = compile_named(
        param1 => Str,
        param2 => Int, {optional => 1},
    );
    my ($param1, $param2) =
        $check->(@_)->@{'param1', 'param2'};

    say $param1, $param2;
}

In the exam­ple above, $check->(@_) returns the type-​checked argu­ments to the foo() func­tion cour­tesy of Type::Paramscompile_named() func­tion. It’s returned as a hash ref­er­ence, and since hash­es are unordered, we spec­i­fy the order in which we want the val­ues by deref­er­enc­ing and then slic­ing the result­ing hash. The post­fix deref­er­enc­ing syn­tax was added in Perl 5.20 and made a default fea­ture in 5.24, and reduces the num­ber of nest­ed brack­ets and braces we have to deal with.

Conclusion

I hope this arti­cle has giv­en you a taste of some of the mag­ic avail­able in the Perl lan­guage. It’s these sort of fea­tures that make pro­gram­ming in it a bit more joy­ful. As always, check the doc­u­men­ta­tion for com­plete infor­ma­tion on these and oth­er top­ics, or look for answers and ask ques­tions on PerlMonks or Stack Overflow.