The Phoenix Trap

Code, music, philosophy, etc.

Tag: List::Util

Perl list processing is for hashes, too
This month I started a new job at Alert Logic, a cybersecurity provider with Perl (among many other things) at its beating heart. I’ve been learning a lot, and part of the process has been understanding the APIs in the code base. To that end, I’ve been writing small test scripts to tease apart data structures, using Perl array-processing, list-processing, and hash- (i.e., associative array)-processing functions.

I’ve covered map, grep, and friends a couple timesbefore. Most recently, I described using List::Util’s anyfunctionto check if a condition is true for any item in a list. In the simplest case, you can use it to check to see if a given value is in the list at all:
```
use feature 'say';
use List::Util 'any';
my @colors =
  qw(red orange yellow green blue indigo violet);
say 'matched' if any { /^red$/ } @colors;
```
However, if you’re going to be doing this a lot with arbitrary strings, Perl FAQ section 4 advisesturning the array into the keys of a hash and then checking for membership there. For example, here’s a simple script to check if the colors input (either from the keyboard or from files passed as arguments) are in the rainbow:
```
#!/usr/bin/env perl

use v5.22; # introduced <<>> for safe opening of arguments
use warnings;
 
my %in_colors = map {$_ => 1}
  qw(red orange yellow green blue indigo violet);

while (<<>>) {
  chomp;
  say "$_ is in the rainbow" if $in_colors{$_};
}
```
List::Util has a bunch of functions for processing lists of pairsthat I’ve found useful when pawing through hashes. pairgrep, for example, acts just like grepbut instead assigns $aand $bto each key and value passed in and returns the resulting pairs that match. I’ve used it as a quick way to search for hash entries matching certain value conditions:
```
use List::Util 'pairgrep';
my %numbers = (zero => 0, one => 1, two => 2, three => 3);
my %odds = pairgrep {$b % 2} %numbers;
```
Sure, you could do this by invoking a mix of plain grep, keys, and a hash slice, but it’s noisier and more repetitive:
```
use v5.20; # for key/value hash slice 
my %odds = %numbers{grep {$numbers{$_} % 2} keys %numbers};
```
pairgrep’s compiled C‑based XScode can also be faster, as evidenced by this Benchmarkscript that works through a hash made of the Unix wordsfile(479,828 entries on my machine):
```
#!/usr/bin/env perl

use v5.20;
use warnings;
use List::Util 'pairgrep';
use Benchmark 'cmpthese';

my (%words, $count);
open my $fh, '<', '/usr/share/dict/words'
  or die "can't open words: $!";
while (<$fh>) {
  chomp;
  $words{$_} = $count++;
}
close $fh;

cmpthese(100, {
  grep => sub {
    my %odds = %words{grep {$words{$_} % 2} keys %words};
  },
  pairgrep => sub {
    my %odds = pairgrep {$b % 2} %words;
  },
} );
```
Benchmark output:
```
           Rate     grep pairgrep
grep     1.47/s       --     -20%
pairgrep 1.84/s      25%       --
```
In general, I urge you to work through the Perl documentation’s tutorials on references, lists of lists, the data structures cookbook, and the FAQs on arrayand hash manipulation. Then dip into the various list-processing modules(especially the included List::Utiland CPAN’s List::SomeUtils) for ready-made functions for common operations. You’ll find a wealth of techniques for creating, managing, and processing the data structures that your programs need.
February 10, 2022
Better Perl: Four list processing best practices with map, grep, and more
Six months ago I gave an overview of Perl’s list processing fundamentals, briefly describing what lists are and then introducing the built-in mapand grepfunctions for transforming and filtering them. Later on, I compiled a list (how appropriate) of list processing modules available via CPAN, noting there’s some confusing duplication of effort. But you’re a busy developer, and you just want to know the Right Thing To Do™ when faced with a list processing challenge.

First, some credit is due: these are all restatements of several Perl::Critic policieswhich in turn codify standards described in Damian Conway’s Perl Best Practices(2005). I’ve repeatedly recommendedthe latter as a starting point for higher-quality Perl development. Over the years these practices continue to be re-evaluated (including by the author himself) and various authors release new policy modules, but perlcriticremains a great tool for ensuring you (and your team or other contributors) maintain a consistent high standard in your code.

With that said, on to the recommendations!

Don’t use grepto check if anylist elements match

It might sound weird to lead off by recommending notto use grep, but sometimes it’s not the right tool for the job. If you’ve got a list and want to determine if a condition matches any item in it, you might try:
```
if (grep { some_condition($_) } @my_list) {
    ... # don't do this!
}
```
Yes, this works because (in scalar context) grepreturns the number of matches found, but it’s wasteful, checking every element of @my_list(which could be lengthy) before finally providing a result. Use the standard List::Utilmodule’s anyfunction, which immediately returns (“short-circuits”) on the first match:
```
use List::Util 1.33 qw(any);

if (any { some_condition($_) } @my_list) {
    ... # do something
}
```
Perl has included the requisite version of this module since version 5.20in 2014; for earlier releases, you’ll need to update from CPAN. List::Util has many other great list-reduction, key/value pair, and other related functionsyou can import into your code, so check it out before you attempt to re-invent any wheels.

As a side note for web developers, the Perl Dancer frameworkalso includes an anykeywordfor declaring multiple HTTP routes, so if you’re mixing List::Util in there don’t import it. Instead, call it explicitly like this or you’ll get an error about a redefined function:
```
use List::Util 1.33;

if (List::Util::any { some_condition($_) } @my_list) {
    ... # do something
}
```
This recommendation is codified in the BuiltinFunctions::ProhibitBooleanGrepPerl::Critic policy, comes directly from Perl Best Practices, and is recommendedby the Software Engineering Institute Computer Emergency Response Team (SEI CERT)’s Perl Coding Standard.

Don’t change $_in mapor grep

I mentioned this back in March, but it bears repeating: mapand grepare intended as pure functions, not mutatorswith side effects. This means that the original list should remain unchanged. Yes, each element aliases in turn to the $_special variable, but that’s for speed and can have surprising results if changed even if it’s technically allowed. If you need to modify an array in-place use something like:
```
for (@my_array) {
    $_ = ...; # make your changes here
}
```
If you want something that lookslike mapbut won’t change the original list (and don’t mind a few CPAN dependencies), consider List::SomeUtils’ applyfunction:
```
use List::SomeUtils qw(apply);

my @doubled_array = apply {$_ *= 2} @old_array;
```
Lastly, side effects also include things like manipulating other variables or doing input and output. Don’t use mapor grepin a void context (i.e., without a resulting array or list); do something with the results or use a foror foreachloop:
```
map { print foo($_) } @my_array; # don't do this
print map { foo($_) } @my_array; # do this instead

map { push @new_array, foo($_) } @my_array; # don't do this
@new_array = map { foo($_) } @my_array;     # do this instead
```
This recommendation is codified by the BuiltinFunctions::ProhibitVoidGrep, BuiltinFunctions::ProhibitVoidMap, and ControlStructures::ProhibitMutatingListFunctionsPerl::Critic policies. The latter comes from Perl Best Practicesand is an SEI CERT Perl Coding Standard rule.

Use blocks with mapand grep, not expressions

You can call mapor greplike this (parentheses are optionalaround built-in functions):
```
my @new_array  = map foo($_), @old_array; # don't do this
my @new_array2 = grep !/^#/, @old_array;  # don't do this
```
Or like this:
```
my @new_array  = map { foo($_) } @old_array;
my @new_array2 = grep {!/^#/} @old_array;
```
Do it the second way. It’s easier to read, especially if you’re passing in a literal list or multiple arrays, and the expression forms can conceal bugs. This recommendation is codified by the BuiltinFunctions::RequireBlockGrepand BuiltinFunctions::RequireBlockMapPerl::Critic policies and comes from Perl Best Practices.

Refactor multi-statement maps, greps, and other list functions

map, grep, and friends should follow the Unix philosophy of “Do One Thing and Do It Well.”Your readability and maintainability drop with every statement you place inside one of their blocks. Consider junior developers and future maintainers (this includes you!) and refactor anything with more than one statement into a separate subroutine or at least a forloop. This goes for list processing functions (like the aforementioned any) imported from other modules, too.

This recommendation is codified by the Perl Best Practices-inspired BuiltinFunctions::ProhibitComplexMappingsand BuiltinFunctions::RequireSimpleSortBlockPerl::Critic policies, although those only cover mapand sortfunctions, respectively.

Do you have any other suggestions for list processing best practices? Feel free to leave them in the comments or better yet, consider creating new Perl::Critic policiesfor them or contacting the Perl::Critic teamto develop them for your organization.
October 26, 2021
A good old-fashioned Perl log analyzer
A recent Lobsters postlauding the virtues of AWKreminded me that although the language is powerful and lightning-fast, I usually find myself exceeding its capabilities and reaching for Perl instead. One such application is analyzing voluminous log files such as the ones generated by this blog. Yes, WordPress has stats, but I’ve never let reinvention of the wheel get in the way of a good programming exercise.

So I whipped this script up on Sunday night while watching RuPaul’s Drag Racereruns. It parses my Apache web server log filesand reports on hits from week to week.
```
#!/usr/bin/env perl

use strict;
use warnings;
use Syntax::Construct 'operator-double-diamond';
use Regexp::Log::Common;
use DateTime::Format::HTTP;
use List::Util 1.33 'any';
use Number::Format 'format_number';

my $parser = Regexp::Log::Common->new(
    format  => ':extended',
    capture => [qw<req ts status>],
);
my @fields      = $parser->capture;
my $compiled_re = $parser->regexp;

my @skip_uri_patterns = qw<
  ^/+robots.txt
  [-\w]*sitemap[-\w]*.xml
  ^/+wp-
  /feed/?$
  ^/+?rest_route=
>;

my ( %count, %week_of );
while ( <<>> ) {
    my %log;
    @log{@fields} = /$compiled_re/;

    # only interested in successful or cached requests
    next unless $log{status} =~ /^2/ or $log{status} == 304;

    my ( $method, $uri, $protocol ) = split ' ', $log{req};
    next unless $method eq 'GET';
    next if any { $uri =~ $_ } @skip_uri_patterns;

    my $dt  = DateTime::Format::HTTP->parse_datetime( $log{ts} );
    my $key = sprintf '%u-%02u', $dt->week;

    # get first date of each week
    $week_of{$key} ||= $dt->date;
    $count{$key}++;
}

printf "Week of %s: % 10s\n", $week_of{$_}, format_number( $count{$_} )
  for sort keys %count;
```
Here’s some sample output:
```
Week of 2021-07-31:      2,672
Week of 2021-08-02:     16,222
Week of 2021-08-09:     12,609
Week of 2021-08-16:     17,714
Week of 2021-08-23:     14,462
Week of 2021-08-30:     11,758
Week of 2021-09-06:     14,811
Week of 2021-09-13:        407
```
I first started prototyping this on the command line as if it were an awkone-liner by using the perl -nand -aflags. The former wraps code in a whileloop over the <>“diamond operator”, processing each line from standard input or files passed as arguments. The latter splits the fields of the line into an array named @F. It looked something like this while I was listing URIs (locations on the website):
```
gunzip -c ~/logs/phoenixtrap.com-ssl_log-*.gz | \
perl -anE 'say $F[6]'
```
But once I realized I’d need to filter out a bunch of URI patterns and do some aggregation by date, I turned it into a script and turned to CPAN.

There I found Regexp::Log::Commonand DateTime::Format::HTTP, which let me pull apart the Apache log format and its timestamp strings without having to write even more complicated regular expressions myself. (As noted above, this was already a wheel-reinvention exercise; no need to compound that further.)

Regexp::Log::Common builds a compiled regular expression based on the log format and fields you’re interested in, so that’s the constructor on lines 11 through 14. The expression then returns those fields as a list, which I’m assigning to a hash slice with those field names as keys in line 29. I then skip over requests that aren’t successful or browser cache hits, skip over requests that don’t GETweb pages or other assets (e.g., POSTs to forms or updating other resources), and skip over the URI patterns mentioned earlier.

(Those patterns are worth a mention: they include the robots.txtand sitemap XMLfiles used by search engine indexers, WordPress administration pages, files used by RSSnewsreaders subscribed to my blog, and routes used by the JetpackWordPress add-on. If you’re adapting this for your site you might need to customize this list based on what software you use to run it.)

Lines 38 and 39 parse the timestamp from the log into a DateTimeobject using DateTime::Format::HTTPand then build the key used to store the per-week hit count. The last lines of the loop then grab the first date of each new week (assuming the log is in chronological order) and increment the count. Once finished, lines 46 and 47 provide a report sorted by week, displaying it as a friendly “Week of date” and the hit counts aligned to the right with sprintf. Number::Format’s format_numberfunction displays the totals with thousands separators.

Update:After this was initially published. astute reader Chris McGowan noted that I had a bug where $log{status}was assigned the value 304 with the =operator rather than compared with ==. He also suggested I use the double-diamond <<>>operatorintroduced in Perl v5.22.0 to avoid maliciously-named files. Thanks, Chris!

Room for improvement

DateTime is a very powerful module but this comes at a price of speed and memory. Something simpler like Date::WeekNumbershould yield performance improvements, especially as my logs grow (here’s hoping). It requires a bit more manual massaging of the log dates to convert them into something the module can use, though:
```
#!/usr/bin/env perl

use strict;
use warnings;
use Syntax::Construct qw<
  operator-double-diamond
  regex-named-capture-group
>;
use Regexp::Log::Common;
use Date::WeekNumber 'iso_week_number';
use List::Util 1.33 'any';
use Number::Format 'format_number';

my $parser = Regexp::Log::Common->new(
    format  => ':extended',
    capture => [qw<req ts status>],
);
my @fields      = $parser->capture;
my $compiled_re = $parser->regexp;

my @skip_uri_patterns = qw<
  ^/+robots.txt
  [-\w]*sitemap[-\w]*.xml
  ^/+wp-
  /feed/?$
  ^/+?rest_route=
>;

my %month = (
    Jan => '01',
    Feb => '02',
    Mar => '03',
    Apr => '04',
    May => '05',
    Jun => '06',
    Jul => '07',
    Aug => '08',
    Sep => '09',
    Oct => '10',
    Nov => '11',
    Dec => '12',
);

my ( %count, %week_of );
while ( <<>> ) {
    my %log;
    @log{@fields} = /$compiled_re/;

    # only interested in successful or cached requests
    next unless $log{status} =~ /^2/ or $log{status} == 304;

    my ( $method, $uri, $protocol ) = split ' ', $log{req};
    next unless $method eq 'GET';
    next if any { $uri =~ $_ } @skip_uri_patterns;

    # convert log timestamp to YYYY-MM-DD
    # for Date::WeekNumber
    $log{ts} =~ m!^
      (?<day>\d\d) /
      (?<month>...) /
      (?<year>\d{4}) : !x;
    my $date = "$+{year}-$month{ $+{month} }-$+{day}";

    my $week = iso_week_number($date);
    $week_of{$week} ||= $date;
    $count{$week}++;
}

printf "Week of %s: % 10s\n", $week_of{$_}, format_number( $count{$_} )
  for sort keys %count;
```
It looks almost the same as the first version, with the addition of a hash to convert month names to numbers and the actual conversion (using named regular expression capture groupsfor readability, using Syntax::Construct to check for that feature). On my server, this results in a ten- to eleven-second savings when processing two months of compressed logs.

What’s next? Pretty graphs? Drilling down to specific blog posts? Database storage for further queries and analysis? Perl and CPAN make it possible to go far beyond what you can do with AWK. What would you add or change? Let me know in the comments.
September 14, 2021