Tag: List::SomeUtils

Perl list processing is for hashes, too
This month I started a new job at Alert Logic, a cybersecurity provider with Perl (among many other things) at its beating heart. I’ve been learning a lot, and part of the process has been understanding the APIs in the code base. To that end, I’ve been writing small test scripts to tease apart data structures, using Perl array-processing, list-processing, and hash- (i.e., associative array)-processing functions.

I’ve covered map, grep, and friends a couple timesbefore. Most recently, I described using List::Util’s anyfunctionto check if a condition is true for any item in a list. In the simplest case, you can use it to check to see if a given value is in the list at all:
```
use feature 'say';
use List::Util 'any';
my @colors =
  qw(red orange yellow green blue indigo violet);
say 'matched' if any { /^red$/ } @colors;
```
However, if you’re going to be doing this a lot with arbitrary strings, Perl FAQ section 4 advisesturning the array into the keys of a hash and then checking for membership there. For example, here’s a simple script to check if the colors input (either from the keyboard or from files passed as arguments) are in the rainbow:
```
#!/usr/bin/env perl

use v5.22; # introduced <<>> for safe opening of arguments
use warnings;
 
my %in_colors = map {$_ => 1}
  qw(red orange yellow green blue indigo violet);

while (<<>>) {
  chomp;
  say "$_ is in the rainbow" if $in_colors{$_};
}
```
List::Util has a bunch of functions for processing lists of pairsthat I’ve found useful when pawing through hashes. pairgrep, for example, acts just like grepbut instead assigns $aand $bto each key and value passed in and returns the resulting pairs that match. I’ve used it as a quick way to search for hash entries matching certain value conditions:
```
use List::Util 'pairgrep';
my %numbers = (zero => 0, one => 1, two => 2, three => 3);
my %odds = pairgrep {$b % 2} %numbers;
```
Sure, you could do this by invoking a mix of plain grep, keys, and a hash slice, but it’s noisier and more repetitive:
```
use v5.20; # for key/value hash slice 
my %odds = %numbers{grep {$numbers{$_} % 2} keys %numbers};
```
pairgrep’s compiled C‑based XScode can also be faster, as evidenced by this Benchmarkscript that works through a hash made of the Unix wordsfile(479,828 entries on my machine):
```
#!/usr/bin/env perl

use v5.20;
use warnings;
use List::Util 'pairgrep';
use Benchmark 'cmpthese';

my (%words, $count);
open my $fh, '<', '/usr/share/dict/words'
  or die "can't open words: $!";
while (<$fh>) {
  chomp;
  $words{$_} = $count++;
}
close $fh;

cmpthese(100, {
  grep => sub {
    my %odds = %words{grep {$words{$_} % 2} keys %words};
  },
  pairgrep => sub {
    my %odds = pairgrep {$b % 2} %words;
  },
} );
```
Benchmark output:
```
           Rate     grep pairgrep
grep     1.47/s       --     -20%
pairgrep 1.84/s      25%       --
```
In general, I urge you to work through the Perl documentation’s tutorials on references, lists of lists, the data structures cookbook, and the FAQs on arrayand hash manipulation. Then dip into the various list-processing modules(especially the included List::Utiland CPAN’s List::SomeUtils) for ready-made functions for common operations. You’ll find a wealth of techniques for creating, managing, and processing the data structures that your programs need.
February 10, 2022
Better Perl: Four list processing best practices with map, grep, and more
Six months ago I gave an overview of Perl’s list processing fundamentals, briefly describing what lists are and then introducing the built-in mapand grepfunctions for transforming and filtering them. Later on, I compiled a list (how appropriate) of list processing modules available via CPAN, noting there’s some confusing duplication of effort. But you’re a busy developer, and you just want to know the Right Thing To Do™ when faced with a list processing challenge.

First, some credit is due: these are all restatements of several Perl::Critic policieswhich in turn codify standards described in Damian Conway’s Perl Best Practices(2005). I’ve repeatedly recommendedthe latter as a starting point for higher-quality Perl development. Over the years these practices continue to be re-evaluated (including by the author himself) and various authors release new policy modules, but perlcriticremains a great tool for ensuring you (and your team or other contributors) maintain a consistent high standard in your code.

With that said, on to the recommendations!

Don’t use grepto check if anylist elements match

It might sound weird to lead off by recommending notto use grep, but sometimes it’s not the right tool for the job. If you’ve got a list and want to determine if a condition matches any item in it, you might try:
```
if (grep { some_condition($_) } @my_list) {
    ... # don't do this!
}
```
Yes, this works because (in scalar context) grepreturns the number of matches found, but it’s wasteful, checking every element of @my_list(which could be lengthy) before finally providing a result. Use the standard List::Utilmodule’s anyfunction, which immediately returns (“short-circuits”) on the first match:
```
use List::Util 1.33 qw(any);

if (any { some_condition($_) } @my_list) {
    ... # do something
}
```
Perl has included the requisite version of this module since version 5.20in 2014; for earlier releases, you’ll need to update from CPAN. List::Util has many other great list-reduction, key/value pair, and other related functionsyou can import into your code, so check it out before you attempt to re-invent any wheels.

As a side note for web developers, the Perl Dancer frameworkalso includes an anykeywordfor declaring multiple HTTP routes, so if you’re mixing List::Util in there don’t import it. Instead, call it explicitly like this or you’ll get an error about a redefined function:
```
use List::Util 1.33;

if (List::Util::any { some_condition($_) } @my_list) {
    ... # do something
}
```
This recommendation is codified in the BuiltinFunctions::ProhibitBooleanGrepPerl::Critic policy, comes directly from Perl Best Practices, and is recommendedby the Software Engineering Institute Computer Emergency Response Team (SEI CERT)’s Perl Coding Standard.

Don’t change $_in mapor grep

I mentioned this back in March, but it bears repeating: mapand grepare intended as pure functions, not mutatorswith side effects. This means that the original list should remain unchanged. Yes, each element aliases in turn to the $_special variable, but that’s for speed and can have surprising results if changed even if it’s technically allowed. If you need to modify an array in-place use something like:
```
for (@my_array) {
    $_ = ...; # make your changes here
}
```
If you want something that lookslike mapbut won’t change the original list (and don’t mind a few CPAN dependencies), consider List::SomeUtils’ applyfunction:
```
use List::SomeUtils qw(apply);

my @doubled_array = apply {$_ *= 2} @old_array;
```
Lastly, side effects also include things like manipulating other variables or doing input and output. Don’t use mapor grepin a void context (i.e., without a resulting array or list); do something with the results or use a foror foreachloop:
```
map { print foo($_) } @my_array; # don't do this
print map { foo($_) } @my_array; # do this instead

map { push @new_array, foo($_) } @my_array; # don't do this
@new_array = map { foo($_) } @my_array;     # do this instead
```
This recommendation is codified by the BuiltinFunctions::ProhibitVoidGrep, BuiltinFunctions::ProhibitVoidMap, and ControlStructures::ProhibitMutatingListFunctionsPerl::Critic policies. The latter comes from Perl Best Practicesand is an SEI CERT Perl Coding Standard rule.

Use blocks with mapand grep, not expressions

You can call mapor greplike this (parentheses are optionalaround built-in functions):
```
my @new_array  = map foo($_), @old_array; # don't do this
my @new_array2 = grep !/^#/, @old_array;  # don't do this
```
Or like this:
```
my @new_array  = map { foo($_) } @old_array;
my @new_array2 = grep {!/^#/} @old_array;
```
Do it the second way. It’s easier to read, especially if you’re passing in a literal list or multiple arrays, and the expression forms can conceal bugs. This recommendation is codified by the BuiltinFunctions::RequireBlockGrepand BuiltinFunctions::RequireBlockMapPerl::Critic policies and comes from Perl Best Practices.

Refactor multi-statement maps, greps, and other list functions

map, grep, and friends should follow the Unix philosophy of “Do One Thing and Do It Well.”Your readability and maintainability drop with every statement you place inside one of their blocks. Consider junior developers and future maintainers (this includes you!) and refactor anything with more than one statement into a separate subroutine or at least a forloop. This goes for list processing functions (like the aforementioned any) imported from other modules, too.

This recommendation is codified by the Perl Best Practices-inspired BuiltinFunctions::ProhibitComplexMappingsand BuiltinFunctions::RequireSimpleSortBlockPerl::Critic policies, although those only cover mapand sortfunctions, respectively.

Do you have any other suggestions for list processing best practices? Feel free to leave them in the comments or better yet, consider creating new Perl::Critic policiesfor them or contacting the Perl::Critic teamto develop them for your organization.
October 26, 2021

Tag: List::SomeUtils

Perl list processing is for hashes, too

Better Perl: Four list processing best practices with map, grep, and more

Don’t use grepto check if anylist elements match

Don’t change $_in mapor grep

Use blocks with mapand grep, not expressions

Refactor multi-​statement maps, greps, and other list functions

Don’t use `grep`to check if anylist elements match

Don’t change `$_`in `map`or `grep`

Use blocks with `map`and `grep`, not expressions

Refactor multi-statement `map`s, `grep`s, and other list functions