Tag: lists

Posted on
by

This week’s Perl and Raku Conference 2022 in Houston was packed with great presentations, and I humbly added to them with a five-​ish minute lightning talk on two of Perl’s more misunderstood functions: map and grep.

Sorry about the ​”um”s and ​”ah”s…
PDF slides of the the presentationDownload

I’ve written much about list processing in Perl, and this talk was based on the following blog posts:

Overall I loved attending the conference, and it really invigorated my participation in the Perl community. Stay tuned as I resume regular posting!

Update for Raku

On Twitter I nudged prominent Raku hacker (and recovered Perl hacker) Elizabeth Mattijsen to write about the Raku programming language’s map and grep functionality. Check out her five-​part series on DEV.to.

Posted on
by
plate of eggs and hash browns

This month I started a new job at Alert Logic, a cybersecurity provider with Perl (among many other things) at its beating heart. I’ve been learning a lot, and part of the process has been understanding the APIs in the code base. To that end, I’ve been writing small test scripts to tease apart data structures, using Perl array-​processing, list-​processing, and hash- (i.e., associative array)-processing functions.

I’ve covered map, grep, and friends a couple times before. Most recently, I described using List::Util’s any function to check if a condition is true for any item in a list. In the simplest case, you can use it to check to see if a given value is in the list at all:

use feature 'say';
use List::Util 'any';
my @colors =
  qw(red orange yellow green blue indigo violet);
say 'matched' if any { /^red$/ } @colors;

However, if you’re going to be doing this a lot with arbitrary strings, Perl FAQ section 4 advises turning the array into the keys of a hash and then checking for membership there. For example, here’s a simple script to check if the colors input (either from the keyboard or from files passed as arguments) are in the rainbow:

#!/usr/bin/env perl

use v5.22; # introduced <<>> for safe opening of arguments
use warnings;
 
my %in_colors = map {$_ => 1}
  qw(red orange yellow green blue indigo violet);

while (<<>>) {
  chomp;
  say "$_ is in the rainbow" if $in_colors{$_};
}

List::Util has a bunch of functions for processing lists of pairs that I’ve found useful when pawing through hashes. pairgrep, for example, acts just like grep but instead assigns $a and $b to each key and value passed in and returns the resulting pairs that match. I’ve used it as a quick way to search for hash entries matching certain value conditions:

use List::Util 'pairgrep';
my %numbers = (zero => 0, one => 1, two => 2, three => 3);
my %odds = pairgrep {$b % 2} %numbers;

Sure, you could do this by invoking a mix of plain grep, keys, and a hash slice, but it’s noisier and more repetitive:

use v5.20; # for key/value hash slice 
my %odds = %numbers{grep {$numbers{$_} % 2} keys %numbers};

pairgrep​’s compiled C‑based XS code can also be faster, as evidenced by this Benchmark script that works through a hash made of the Unix words file (479,828 entries on my machine):

#!/usr/bin/env perl

use v5.20;
use warnings;
use List::Util 'pairgrep';
use Benchmark 'cmpthese';

my (%words, $count);
open my $fh, '<', '/usr/share/dict/words'
  or die "can't open words: $!";
while (<$fh>) {
  chomp;
  $words{$_} = $count++;
}
close $fh;

cmpthese(100, {
  grep => sub {
    my %odds = %words{grep {$words{$_} % 2} keys %words};
  },
  pairgrep => sub {
    my %odds = pairgrep {$b % 2} %words;
  },
} );

Benchmark output:

           Rate     grep pairgrep
grep     1.47/s       --     -20%
pairgrep 1.84/s      25%       --

In general, I urge you to work through the Perl documentation​’s tutorials on references, lists of lists, the data structures cookbook, and the FAQs on array and hash manipulation. Then dip into the various list-​processing modules (especially the included List::Util and CPAN’s List::SomeUtils) for ready-​made functions for common operations. You’ll find a wealth of techniques for creating, managing, and processing the data structures that your programs need.

Posted on
by
woman looking at the map

Six months ago I gave an overview of Perl’s list processing fundamentals, briefly describing what lists are and then introducing the built-​in map and grep functions for transforming and filtering them. Later on, I compiled a list (how appropriate) of list processing modules available via CPAN, noting there’s some confusing duplication of effort. But you’re a busy developer, and you just want to know the Right Thing To Do™ when faced with a list processing challenge.

First, some credit is due: these are all restatements of several Perl::Critic policies which in turn codify standards described in Damian Conway’s Perl Best Practices (2005). I’ve repeatedly recommended the latter as a starting point for higher-​quality Perl development. Over the years these practices continue to be re-​evaluated (including by the author himself) and various authors release new policy modules, but perlcritic remains a great tool for ensuring you (and your team or other contributors) maintain a consistent high standard in your code.

With that said, on to the recommendations!

Don’t use grep to check if any list elements match

It might sound weird to lead off by recommending not to use grep, but sometimes it’s not the right tool for the job. If you’ve got a list and want to determine if a condition matches any item in it, you might try:

if (grep { some_condition($_) } @my_list) {
    ... # don't do this!
}

Yes, this works because (in scalar context) grep returns the number of matches found, but it’s wasteful, checking every element of @my_list (which could be lengthy) before finally providing a result. Use the standard List::Util module’s any function, which immediately returns (“short-​circuits”) on the first match:

use List::Util 1.33 qw(any);

if (any { some_condition($_) } @my_list) {
    ... # do something
}

Perl has included the requisite version of this module since version 5.20 in 2014; for earlier releases, you’ll need to update from CPAN. List::Util has many other great list-​reduction, key/​value pair, and other related functions you can import into your code, so check it out before you attempt to re-​invent any wheels.

As a side note for web developers, the Perl Dancer framework also includes an any keyword for declaring multiple HTTP routes, so if you’re mixing List::Util in there don’t import it. Instead, call it explicitly like this or you’ll get an error about a redefined function:

use List::Util 1.33;

if (List::Util::any { some_condition($_) } @my_list) {
    ... # do something
}

This recommendation is codified in the BuiltinFunctions::ProhibitBooleanGrep Perl::Critic policy, comes directly from Perl Best Practices, and is recommended by the Software Engineering Institute Computer Emergency Response Team (SEI CERT)’s Perl Coding Standard.

Don’t change $_ in map or grep

I mentioned this back in March, but it bears repeating: map and grep are intended as pure functions, not mutators with side effects. This means that the original list should remain unchanged. Yes, each element aliases in turn to the $_ special variable, but that’s for speed and can have surprising results if changed even if it’s technically allowed. If you need to modify an array in-​place use something like:

for (@my_array) {
    $_ = ...; # make your changes here
}

If you want something that looks like map but won’t change the original list (and don’t mind a few CPAN dependencies), consider List::SomeUtils’ apply function:

use List::SomeUtils qw(apply);

my @doubled_array = apply {$_ *= 2} @old_array;

Lastly, side effects also include things like manipulating other variables or doing input and output. Don’t use map or grep in a void context (i.e., without a resulting array or list); do something with the results or use a for or foreach loop:

map { print foo($_) } @my_array; # don't do this
print map { foo($_) } @my_array; # do this instead

map { push @new_array, foo($_) } @my_array; # don't do this
@new_array = map { foo($_) } @my_array;     # do this instead

This recommendation is codified by the BuiltinFunctions::ProhibitVoidGrep, BuiltinFunctions::ProhibitVoidMap, and ControlStructures::ProhibitMutatingListFunctions Perl::Critic policies. The latter comes from Perl Best Practices and is an SEI CERT Perl Coding Standard rule.

Use blocks with map and grep, not expressions

You can call map or grep like this (parentheses are optional around built-​in functions):

my @new_array  = map foo($_), @old_array; # don't do this
my @new_array2 = grep !/^#/, @old_array;  # don't do this

Or like this:

my @new_array  = map { foo($_) } @old_array;
my @new_array2 = grep {!/^#/} @old_array;

Do it the second way. It’s easier to read, especially if you’re passing in a literal list or multiple arrays, and the expression forms can conceal bugs. This recommendation is codified by the BuiltinFunctions::RequireBlockGrep and BuiltinFunctions::RequireBlockMap Perl::Critic policies and comes from Perl Best Practices.

Refactor multi-​statement maps, greps, and other list functions

map, grep, and friends should follow the Unix philosophy of ​“Do One Thing and Do It Well.” Your readability and maintainability drop with every statement you place inside one of their blocks. Consider junior developers and future maintainers (this includes you!) and refactor anything with more than one statement into a separate subroutine or at least a for loop. This goes for list processing functions (like the aforementioned any) imported from other modules, too.

This recommendation is codified by the Perl Best Practices-inspired BuiltinFunctions::ProhibitComplexMappings and BuiltinFunctions::RequireSimpleSortBlock Perl::Critic policies, although those only cover map and sort functions, respectively.

Do you have any other suggestions for list processing best practices? Feel free to leave them in the comments or better yet, consider creating new Perl::Critic policies for them or contacting the Perl::Critic team to develop them for your organization.

Posted on
by
Lister Platz

As previously written, I like list processing. Many computing problems can be broken down into transforming and filtering lists, and Perl has got the fundamentals covered with functions like map, grep, and sort. There is so much more you might want to do, though, and CPAN has a plethora of list and array processing modules.

However, due to the vicissitudes of Perl module maintenance, we have a situation where it’s not clear at a glance where to turn when you’ve got a list that needs processing. So here’s another list: the list modules of CPAN. Click through to discover what functions they provide.

  • We’ve got List::Util which has been released as part of Perl since version 5.7.3.
  • We’ve got List::MoreUtils which has some functions which are named the same as Util but behave differently.
  • We’ve got List::SomeUtils which duplicates MoreUtils but with fewer dependencies.
  • We’ve got List::UtilsBy which MoreUtils has also cribbed some functions from.
  • We’ve got List::AllUtils which attempts to consolidate Util, SomeUtils, and ListBy but has some exceptions to called modules because of the aforementioned duplication between Util and SomeUtils.
  • We’ve got List::Util::MaybeXS which helps with pure Perl fallbacks in case your version of Util is too old to have a certain function.
  • We’ve got List::MoreUtils::XS which provides (some?) faster versions of MoreUtils’ functions (but you still have to use MoreUtils).
  • And lastly, we have Util::Any which lets you import functions from Util, MoreUtils, and just for good measure Scalar::Util, Hash::Util, String::Util, String::CamelCase, List::Pairwise, and Data::Dumper. But it hasn’t been updated since 2016, so it doesn’t necessarily export the functions added to those modules since then.

Am I missing anything? Probably! But these are the ones most associated with being upstream on the CPAN River, so they (or the modules they consolidate) have more projects depending on them.

Posted on
by
notebook

As a Perl developer, you’re probably aware of the language’s strengths as a text-​processing language and how many computing tasks can be broken down into those types of tasks. You might not realize, though, that Perl is also a world-​class list processing language and that many problems can be expressed in terms of lists and their transformations.

Chief among Perl’s tools for list processing are the functions map and grep. I can’t count how many times in my twenty-​five years as a developer I’ve run into code that could’ve been simplified if only the author was familiar with these two functions. Once you understand map and grep, you’ll start seeing lists everywhere and the opportunity to make your code more succinct and expressive at the same time.

What are lists?

Before we get into functions that manipulate lists, we need to understand what they are. A list is an ordered group of elements, and those elements can be any kind of data you can represent in the language: numbers, strings, objects, regular expressions, references, etc., as long as they’re stored as scalars. You might think of a list as the thing that an array stores, and in fact Perl is fine with using an array where a list can go.

my @foo = (1, 2, 3);

Here we’re assigning the list of numbers from 1 to 3 to the array @foo. The difference between the array and the list is that the list is a fixed collection, while arrays and their elements can be modified by various operations. perlfaq4 has a great discussion on the differences between the two.

Lists are everywhere, man!

Ever wanted to sort some data? You were using a list.

join a bunch of things together into a string? List again.

split a string into pieces? You got a list back (in list context; in scalar context, you got the size of the list.)

Heck, even the humble print function and its cousin say take a list (and an optional filehandle) as arguments; it’s why you can treat Perl as an upscale AWK and feed it scalars to output with a field separator.

You’re using lists all the time and may not even know it.

map: The list transformer

The map function is devious in its simplicity: It takes two inputs, an expression or block of code, and a list to run it on. For every item in the list, it will alias $_ to it, and then return none, one, or many items in a list based on what happens in the expression or code block. You can call it like this:

my @foo = map bar($_), @list;

Or like this:

my @foo = map { bar($_) } @list;

We’re going to ignore the first way, though because Conway (Perl Best Practices, 2005) tells us that when you specify the first argument as an expression, it’s harder to tell it apart from the remaining arguments, especially if that expression uses a built-​in function where the parentheses are optional. So always use a code block!

You should always turn to map (and not, say, a for or foreach loop) when generating a new list from an old list. For example:

my @lowercased = map { lc } @mixed_case;

When paired with a lookup table, map is also the most efficient way to tell if a member of a list equals a string, especially if that list is static:

use Const::Fast;

const my %IS_EXIT_WORD => map { ($_ => 1) }
  qw(q quit bye exit stop done last finish aurevoir);

...

die if $IS_EXIT_WORD{$command};

Here we’re using map​’s ability to return multiple items per source element to generate a constant hash, and then testing membership in that hash.

grep: The list filter

You may recognize the word ​“grep” from the Unix command of the same name. It’s a tool for finding lines of text inside of other text using a regular expression describing the desired result.

Perl, of course, is really good at regular expressions, but its grep function goes beyond and enables you to match using any expression or code block. Think of it as a partner to map; where map uses a code block to transform a list, grep uses one to filter it down. In fact, other languages typically call this function filter.

You can, of course, use regular expressions with grep, especially because a regexp match in Perl defaults to matching on the $_ variable and grep happens to provide that to its code block argument. So:

my @months_with_a = grep { /[Aa]/ } qw(
  January February March
  April   May      June
  July    August   September
  October November December
);

But grep really comes into its own when used for its general filtering capabilities; for instance, making sure that you don’t accidentally try to compare an undefined value:

say $_ > 5
  ? "$_ is bigger"
  : "$_ is equal or smaller"
  for grep { defined } @numbers;

Or when executing a complicated function that returns true or false depending on its arguments:

my @results = grep { really_large_database_query($_) }
              @foo;

You might even consider chaining map and grep together. Here’s an example for getting the JPEG images out of a file list and then lowercasing the results:

my @jpeg_files = map  { lc }
                grep { /\.jpe?g$/i } @files;

“Side effects may include…” (updated)

When introducing map above I noted that it aliased $_ for every element in the list. I used that term deliberately because modifications to $_ will modify the original element itself, and that is usually an error. Programmers call that a ​“side effect,” and they can lead to unexpected behavior or at least difficult-​to-​maintain code. Consider:

my @needs_docs = grep { s/\.pm$/.pod/ && !-e }
                 @pm_files;

The intent may have been to find files ending in .pm that don’t have a corresponding .pod file, but the actual behavior is replacing the .pm suffix with .pod, then checking whether that filename exists. If it doesn’t, it’s passed through to @needs_docs; regardless, @pm_files has had its contents modified.

If you really do need to modify a copy of each element, assign a variable within your code block like this:

my @needs_docs = grep {
                   my $file = $_;
                   $file =~ s/\.pm$/.pod/;
                   !-e $file
                 } @pm_files;

But at that point you should probably refactor your multi-​line block as a separate function:

my @needs_docs = grep { file_without_docs($_) }
                 @pm_files;

sub file_without_docs {
    my $file = shift;
    $file =~ s/\.pm$/.pod/;
    return !-e $file;
}

In this case of using the substitution operator s///, you could also do this when using Perl 5.14 or above to get non-​destructive substitution:

use v5.14;

my @needs_docs = grep { !-e s/\.pm$/.pod/r }
                 @pm_files;

And if you do need side effects, just use a for or foreach loop; future code maintainers (i.e., you in six months) will thank you.

Taking you higher

map and grep are examples of higher-​order functions, since they take a function (in the form of a code block) as an argument. So congratulations, you just significantly leveled up your knowledge of Perl and computer science. If you’re interested in more such programming techniques, I recommend Mark Jason Dominus’ Higher Order Perl (2005), available for free online.