After a lot of procrastination, I’ve decided my talk for this week’s ePhEmeRaL miniconf will be “Cunningham’s Law: A Year of Being Wrong on the Internet, or «prêcher le faux pour savoir le vrai.»”
The event starts at 8:00 AM CST on Thursday, November 18; you can find out more about it including the full schedule and a time zone converter here.
At my work, we extensively use the Moose object system to take care of what would ordinarily be very tedious boilerplate object-oriented Perl code. In one part of the codebase, we have a family of classes that, among other things, map Perl methods to the names of various calls in a third-party API within our larger organization. Those private Perl methods are in turn called from public methods provided by roles consumed by these classes so that other areas aren’t concerned with said API’s details.
Without going into too many specifics, I had a bunch of classes all with sections that looked like this:
sub _create_method { return 'api_add' }
sub _retrieve_method { return 'api_info' }
sub _search_method { return 'api_list' }
sub _update_method { return 'api_update' }
sub _cancel_method { return 'api_remove' }
sub _suspend_method { return 'api_disable' }
sub _unsuspend_method { return 'api_restore' }
... # etc.The values returned by these very simple methods might differ from class to class depending on the API call needed, and different classes might have a different mix of these methods depending on what roles they consume.
These methods had built up over time as developers had expanded the classes’ functionality, and this week it was my turn. I decided to apply the DRY (don’t repeat yourself) principle and create them from a simple hash table like so:
my %METHOD_MAP = (
_create_method => 'api_add',
_retrieve_method => 'api_info',
_search_method => 'api_list',
_update_method => 'api_update',
_cancel_method => 'api_remove',
_suspend_method => 'api_disable',
_unsuspend_method => 'api_restore',
);At first, I thought to myself, “These look like private read-only attributes!” So I wrote:
use Moose;
...
has $_ => (
is => 'ro',
init_arg => undef,
default => $METHOD_MAP{$_},
) for keys %METHOD_MAP;Of course, I’d have to move the classes’ with statements after these definitions so the roles they consume could “see” these runtime-defined attributes. But some of the methods used to read these are class methods (e.g., called as ClassName->foo() rather than $object->foo()), and Moose attributes are only set after the construction of a class instance.
Then I thought, “Hey, Moose has a MOP (meta-object protocol)! I’ll use that to generate these methods at runtime!”
my $meta = __PACKAGE__->meta;
while (my ($method, $api_call) = each %METHOD_MAP) {
$meta->add_method( $method => sub {$api_call} );
}The add_method documentation “strongly encourage[s]” you to pass a metamethod object rather than a code reference, though, so that would look like:
use Moose::Meta::Method;
my $meta = __PACKAGE__->meta;
while (my ($method, $api_call) = each %METHOD_MAP) {
$meta->add_method( $method = Moose::Meta::Method->wrap(
sub {$api_call}, __PACKAGE__, $meta,
);
}This was getting ugly. There had to be a better way, and fortunately there was in the form of Dave Rolsky’s MooseX::ClassAttribute module. It simplifies the above to:
use MooseX::ClassAttribute;
class_has $_ => (
is => 'ro',
default => $METHOD_MAP{$_},
) for keys %METHOD_MAP;Note there’s no need for init_arg => undef to prevent setting the attribute in the constructor. Although they’re still Moose attributes, they act like class methods so long as the class consumes the roles that require them after the attribute definitions.
Lastly, if we were using Moo as a lightweight alternative to Moose, I could have instead selected Toby Inkster’s MooX::ClassAttribute. Although it has some caveats, it’s pretty much the only alternative to our initial class method definitions as Moo lacks a meta-object protocol.
The lesson as always is to check CPAN (or the appropriate mix of your language’s software repository, forums like Stack Overflow, etc.) for anything that could conceivably have application outside of your particular circumstances. Twenty-five years into my career and I’m still leaping into code without first considering that someone smarter than me has already done the work.
Template processors and engines are one of those pieces of software where it seems every developer wants to reinvent the wheel. Goodness knows I’ve done it earlier in my career. Tell me if this sounds familiar:
"double quotes" or sprintf formats. (Or maybe you investigate formats, but the less said about them the better.)Stop. Just stop. Decades of others who have walked this same path have already done this for you. Especially if you’re using a web framework like Dancer, Mojolicious, or Catalyst, where the template processor is either built-in or pluggable from CPAN. Even if you’re not developing a web application there are several general-purpose options of various capabilities like Template Toolkit and Template::Mustache. Investigate the alternatives and determine if they have the features, performance, and support you need. If you’re sure none of them truly meet your unique requirements, then maybe, maybe consider rolling your own.
Whatever you decide, realize that as your application or website grows your investment in that selection will only deepen. Porting to a new template processor can be as challenging as porting any source code to a new programming language.
Unfortunately, there are about as many opinions on how to choose a template processor as there are template processors. For example, in 2013 Roland Koehler wrote a good Python-oriented article on several considerations and the different approaches available. Although he ended up developing his own (quelle surprise), he makes a good case that a template processor ought to at least provide various logic constructs as well as embedded expressions, if not a full programming language. Koehler specifically warns against the latter, though, as a template developer might change an application’s data model, to say nothing of the possibility of executing arbitrary destructive code.
I can appreciate this reasoning. I’ve successfully used Perl template processors like the aforementioned Template::Toolkit (which has both logic directives and an optional facility for evaluating Perl code) and Text::Xslate (which supports several template syntaxes including a subset of Template::Toolkit’s, but without the ability to embed Perl code). We use the latter at work combined with Text::Xslate::Bridge::TT2Like’s emulation of various Template::Toolkit virtual methods and it’s served us well.
But using those modules’ DSLs means more sophisticated tasks need extra time and effort finding the correct logic and expressions. This also assumes that their designer(s) have anticipated my needs either through built-in features or extensions. I’m already writing Perl; why should I switch to another, more limited language and environment provided I can remain disciplined enough to avoid issues like those described above by Koehler?
So for my personal projects, I favor template processors that use the full power of the Perl language like Mojolicious’ embedded Perl renderer or the venerable Text::Template for non-web applications. It saves me time and I’ll likely want more than any DSL can provide. This may not apply to your situation, though, and I’m open to counter-arguments.
What’s your favorite template processor and why? Let me know in the comments.
Six months ago I gave an overview of Perl’s list processing fundamentals, briefly describing what lists are and then introducing the built-in map and grep functions for transforming and filtering them. Later on, I compiled a list (how appropriate) of list processing modules available via CPAN, noting there’s some confusing duplication of effort. But you’re a busy developer, and you just want to know the Right Thing To Do™ when faced with a list processing challenge.
First, some credit is due: these are all restatements of several Perl::Critic policies which in turn codify standards described in Damian Conway’s Perl Best Practices (2005). I’ve repeatedly recommended the latter as a starting point for higher-quality Perl development. Over the years these practices continue to be re-evaluated (including by the author himself) and various authors release new policy modules, but perlcritic remains a great tool for ensuring you (and your team or other contributors) maintain a consistent high standard in your code.
With that said, on to the recommendations!
grep to check if any list elements matchIt might sound weird to lead off by recommending not to use grep, but sometimes it’s not the right tool for the job. If you’ve got a list and want to determine if a condition matches any item in it, you might try:
if (grep { some_condition($_) } @my_list) {
... # don't do this!
}
Yes, this works because (in scalar context) grep returns the number of matches found, but it’s wasteful, checking every element of @my_list (which could be lengthy) before finally providing a result. Use the standard List::Util module’s any function, which immediately returns (“short-circuits”) on the first match:
use List::Util 1.33 qw(any);
if (any { some_condition($_) } @my_list) {
... # do something
}
Perl has included the requisite version of this module since version 5.20 in 2014; for earlier releases, you’ll need to update from CPAN. List::Util has many other great list-reduction, key/value pair, and other related functions you can import into your code, so check it out before you attempt to re-invent any wheels.
As a side note for web developers, the Perl Dancer framework also includes an any keyword for declaring multiple HTTP routes, so if you’re mixing List::Util in there don’t import it. Instead, call it explicitly like this or you’ll get an error about a redefined function:
use List::Util 1.33;
if (List::Util::any { some_condition($_) } @my_list) {
... # do something
}
This recommendation is codified in the BuiltinFunctions::ProhibitBooleanGrep Perl::Critic policy, comes directly from Perl Best Practices, and is recommended by the Software Engineering Institute Computer Emergency Response Team (SEI CERT)’s Perl Coding Standard.
$_ in map or grepI mentioned this back in March, but it bears repeating: map and grep are intended as pure functions, not mutators with side effects. This means that the original list should remain unchanged. Yes, each element aliases in turn to the $_ special variable, but that’s for speed and can have surprising results if changed even if it’s technically allowed. If you need to modify an array in-place use something like:
for (@my_array) {
$_ = ...; # make your changes here
}
If you want something that looks like map but won’t change the original list (and don’t mind a few CPAN dependencies), consider List::SomeUtils’ apply function:
use List::SomeUtils qw(apply);
my @doubled_array = apply {$_ *= 2} @old_array;
Lastly, side effects also include things like manipulating other variables or doing input and output. Don’t use map or grep in a void context (i.e., without a resulting array or list); do something with the results or use a for or foreach loop:
map { print foo($_) } @my_array; # don't do this
print map { foo($_) } @my_array; # do this instead
map { push @new_array, foo($_) } @my_array; # don't do this
@new_array = map { foo($_) } @my_array; # do this instead
This recommendation is codified by the BuiltinFunctions::ProhibitVoidGrep, BuiltinFunctions::ProhibitVoidMap, and ControlStructures::ProhibitMutatingListFunctions Perl::Critic policies. The latter comes from Perl Best Practices and is an SEI CERT Perl Coding Standard rule.
map and grep, not expressionsYou can call map or grep like this (parentheses are optional around built-in functions):
my @new_array = map foo($_), @old_array; # don't do this
my @new_array2 = grep !/^#/, @old_array; # don't do this
Or like this:
my @new_array = map { foo($_) } @old_array;
my @new_array2 = grep {!/^#/} @old_array;
Do it the second way. It’s easier to read, especially if you’re passing in a literal list or multiple arrays, and the expression forms can conceal bugs. This recommendation is codified by the BuiltinFunctions::RequireBlockGrep and BuiltinFunctions::RequireBlockMap Perl::Critic policies and comes from Perl Best Practices.
maps, greps, and other list functionsmap, grep, and friends should follow the Unix philosophy of “Do One Thing and Do It Well.” Your readability and maintainability drop with every statement you place inside one of their blocks. Consider junior developers and future maintainers (this includes you!) and refactor anything with more than one statement into a separate subroutine or at least a for loop. This goes for list processing functions (like the aforementioned any) imported from other modules, too.
This recommendation is codified by the Perl Best Practices-inspired BuiltinFunctions::ProhibitComplexMappings and BuiltinFunctions::RequireSimpleSortBlock Perl::Critic policies, although those only cover map and sort functions, respectively.
Do you have any other suggestions for list processing best practices? Feel free to leave them in the comments or better yet, consider creating new Perl::Critic policies for them or contacting the Perl::Critic team to develop them for your organization.
Look, I get it. You don’t like the Perl programming language or have otherwise disregarded it as “dead.” (Or perhaps you haven’t, in which case please check out my other blog posts!) It has weird noisy syntax, mixing regular expressions, sigils on variable names, various braces and brackets for data structures, and a menagerie of cryptic special variables. It’s old: 34 years in December, with a history of (sometimes amateur) developers that have used and abused that syntax to ship code of questionable quality. Maybe you grudgingly accept its utility but think it should die gracefully, maintained only to run legacy applications.
But you know what? Perl’s still going. It’s had a steady cadence of yearly releases for the past decade, introducing new features and fencing in bad behavior while maintaining an admirable level of backward compatibility. Yes, there was a too-long adventure developing what started as Perl 6, but that language now has its own identity as Raku and even has facilities for mixing Perl with its native code or vice versa.
And then there’s CPAN, the Comprehensive Perl Archive Network: a continually-updated collection of over 200,000 open-source modules written by over 14,000 authors, the best of which are well-tested and ‑documented (applying peer pressure to those that fall short), presented through a search engine and front-end built by scores of contributors. Through CPAN you can find distributions for things like:
All of this is available through a mature installation toolchain that doesn’t break from month to month.
Finally and most importantly, there’s the global Perl community. The COVID-19 pandemic has put a damper on the hundreds of global Perl Mongers groups’ meetups, but that hasn’t stopped the yearly Perl and Raku Conference from meeting virtually. (In the past there have also been yearly European and Asian conferences, occasional forays into South America and Russia, as well as hackathons and workshops worldwide.) There are IRC servers and channels for chat, mailing lists galore, blogs (yes, apart from this one), and a quirky social network that predates Facebook and Twitter.
So no, Perl isn’t dead or even dying, but if you don’t like it and favor something newer, that’s OK! Technologies can coexist on their own merits and advocates of one don’t have to beat down their contemporaries to be successful. Perl happens to be battle-tested (to borrow a term from my friend Curtis “Ovid” Poe), it runs large parts of the Web (speaking from direct and ongoing experience in the hosting business here), and it’s still evolving to meet the needs of its users.
Whether learning a programming language, working through a problem, or trying to understand a new library, it may be tempting to flail around crafting just the right search engine query or cry for help on a forum like Stack Overflow. But look at any guide to asking good questions and you’ll find this commandment at the top: do your research. And one of the primary sources of that research is the official documentation for the language or library in question.
perldoc commandFortunately, Perl and its modules come installed with extensive documentation, also available on the Web. Locally, you can use the perldoc command-line program, or if you have the documentation installed as Linux/Unix man pages (short for manual pages) you can use the standard man command. Because that isn’t always the case and because it has more Perl-specific options, I recommend perldoc.
Note that if you’re using the Perl provided by your operating system distribution you may not have any documentation installed at all without a separate package. For example, Ubuntu Linux calls this package perl-doc; you can install it with this command:
sudo apt-get install perl-doc(Splitting Perl across several packages is a common operating system distribution tactic; it’s one of the reasons I recommend installing your own Perl.)
Once you’re sure you have the command available, run the following command to get an introduction to the Perl programming language:
perldoc perlintroAll of the built-in documentation begins with “perl” as above, so you can say perldoc perlrun to find out how to execute Perl, perldoc perlfaq to get a directory of frequently asked questions, or perldoc perltoc for a table of contents listing everything available.
As I mentioned above, the perldoc command has several useful options. You can run perldoc perldoc for a full description, but here are some highlights:
perldoc -h: a brief help messageperldoc -f function: describe a specific function from perlfuncperldoc -q regular-expression: search the questions in the various perlfaq documentsperldoc -v variable-name: describe a specific predefined variable from perlvarThe last argument to perldoc can be a documentation page as described earlier, an installed module, a Perl program name in your path, or the URL to a Perl module or Plain Old Documentation (POD) file.
You may have noticed a number of links above to the perldoc.perl.org website. This is a great friendly resource if you prefer reading in a web browser, and you can use features like bookmarks/favorites and sending hyperlinks to others. It has a search engine and you can even switch between different Perl versions (including development).
Perl is more than the language and its standard modules. The Comprehensive Perl Archive Network (CPAN) has over 200,000 open-source Perl modules contributed by authors all over the world, solving common and not-so-common software development problems from asynchronous programming to XML development.
In addition to using the perldoc command described earlier to understand locally-installed modules, you can use the MetaCPAN web site’s search engine (in both regular and regular expression flavors) to discover and learn how to save time piecing together programs instead of reinventing the wheel. I always check MetaCPAN first when I need to write some Perl.
Perl has its own markup language called Plain Old Documentation (POD), and you can either interleave it with your own program source code or save it in separate files. The perlpod documentation page has all the details and the perlpodstyle page will guide you in writing in a style consistent with other documentation. And if you need to publish elsewhere, there are translators for HTML, LaTeX, PDF, and other formats on CPAN. You can also provide your documentation to others via a dedicated web application; this is an excellent tool for teams with internal-only (“DarkPAN”) code.
Documentation is like a present you give to both your later self and anybody else that needs to use your work. I highly recommend you spend time writing it in addition to the usual developer-level source code comments. This is especially important if you plan to release open source on CPAN; otherwise, no one will understand what it does or why they should care. I’m incredibly grateful to every developer who has put in the effort.
Twitter recently recommended a tweet to me (all hail the algorithm) touting what the author viewed as the “top 5 web development stacks.”
JavaScript/Node.js options dominated the four-letter acronyms as expected, but the fifth one surprised me: LAMP, the combination of the Linux operating system, Apache web server, MySQL relational database, and Perl, PHP, or Python programming languages. A quick web search for similar lists yielded similar results. Clearly, this meme (in the Dawkins sense) has outlasted its popularization by tech publisher O’Reilly in the 2000s.
Originally coined in 1998 during the “dot-com” bubble, I had thought that the term “LAMP” had faded with developers in the intervening decades with the rise of language-specific web frameworks for:
Certainly on the Perl side (with which I’m most familiar), the community has long since recommended the use of a framework built on the PSGI specification, deprecating 1990s-era CGI scripts and the mod_perl Apache extension. Although general-purpose web servers like Apache or Nginx may be part of an overall system, they’re typically used as proxies or load balancers for Perl-specific servers either provided by the framework or a third-party module.
Granted, PHP still relies on web server-specific modules, APIs, or variations of the FastCGI protocol for interfacing with a web server. And Python web applications typically make use of its WSGI protocol either as a web server extension or, like the Perl examples above, as a proxied standalone server. But all of these are deployment details and do little to describe how developers implement and extend a web application’s structure.
Note how the various four-letter JavaScript stacks (e.g., MERN, MEVN, MEAN, PERN) differentiate themselves mostly by frontend framework (e.g., Angular, React, Vue.js) and maybe by the (relational or NoSQL) database (e.g., MongoDB, MySQL, PostgreSQL). All however seem standardized on the Node.js runtime and Express backend web framework, which could, in theory, be replaced with non-JavaScript options like the more mature LAMP-associated languages and frameworks. (Or if you prefer languages that don’t start with “P”, there’s C#, Go, Java, Ruby, etc.)
My point is that “LAMP” as the name of a web development stack has outlived its usefulness. It’s at once too specific (about operating system and web server details that are often abstracted away for developers) and too broad (covering three separate programming languages and not the frameworks they favor). It also leaves out other non-JavaScript back-end languages and their associated frameworks.
The question is: what can replace it? I’d propose “NoJS” as reminiscent of “NoSQL,” but that inaccurately excludes JavaScript from its necessary role in the front-end. “NJSB” doesn’t exactly roll off the tongue, either, and still has the same ambiguity problem as “LAMP.”
How about pithy sort-of-acronyms patterned like database-frontend-backend? Here are some Perl examples:
Ultimately it comes down to community and industry adoption. If you’re involved with back-end web development, please let me know in the comments if you agree or disagree that “LAMP” is still a useful term, and if not, what should replace it.
One of Perl’s cardinal strengths is the depth and variety of add-on modules to extend its capabilities, collected for the past 26 years and counting on CPAN, the Comprehensive Perl Archive Network. Starting with version 5.004 in 1997, Perl has come packaged with a module (also called CPAN) and associated command-line client for downloading and installing from this service. Some developers favor alternative tools such as CPANPLUS and its cpanp command or cpanminus and its cpanm, or tools built on the latter such as Carton and Carmel.
My favorite of these over the past several years has been Shoichi Kaji’s cpm, mainly because it’s blazingly fast. As an example, the documentation cites an installation of Plack, the Perl web application toolkit, as taking three times as long using cpanm versus cpm. Both use the same Menlo core code but cpm achieves its speed by breaking down dependencies into individual streams, installing modules in parallel, and synchronizing the necessary worker processes.
Shoichi’s presentation from The Perl Conference 2016 provides a great summary:
cpm’ is fast” (2016)It’s very important to note that cpm is not a drop-in replacement for the cpan or cpanm command-line tools. Firstly, it uses the subcommand install, e.g., cpm install Module::Name. Also, by default, it installs modules into a subdirectory named local/ as if you specified cpanm --local-lib-contained local. You might want this if you’re setting up a Perl project with its non-core dependencies in a separate location addressed by the local::lib module; otherwise, you should use cpm install --global to install into a directory in Perl’s @INC array. I tend to do the latter when developing, declaring my project’s dependencies in a cpanfile.
Speaking of cpanfiles, like cpanm --installdeps cpm will use a cpanfile to drive project dependency installation. In fact, it defaults to looking for one if you don’t specify individual modules on the command line and supports the version-controlled cpanfile.snapshot file introduced by Carton for tracking exact dependencies used by your project. This is great for repeatedly building Docker containers and cpm makes that process even faster.
Although speed is its most important feature, cpm has a couple more tricks up its sleeve like installing from a Git repository or self-hosted “DarkPAN.” Check out its included tutorial.
The perlcritic tool is often your first defense against “awkward, hard to read, error-prone, or unconventional constructs in your code,” per its description. It’s part of a class of programs historically known as linters, so-called because like a clothes dryer machine’s lint trap, they “detect small errors with big effects.” (Another such linter is perltidy, which I’ve referenced in the past.)
You can use perlcritic at the command line, integrated with your editor, as a git pre-commit hook, or (my preference) as part of your author tests. It’s driven by policies, individual modules that check your code against a particular recommendation, many of them from Damian Conway’s Perl Best Practices (2005). Those policies, in turn, are enabled by PPI, a library that transforms Perl code into documents that can be programmatically examined and manipulated much like the Document Object Model (DOM) is used to programmatically access web pages.
perlcritic enables the following policies by default unless you customize its configuration or install more. These are just the “gentle” (severity level 5) policies, so consider them the bare minimum in detecting bad practices. The full set of included policies goes much deeper, ratcheting up the severity to “stern,” “harsh,” “cruel,” and “brutal.” They’re further organized according to themes so that you might selectively review your code against issues like security, maintenance, complexity, and bug prevention.
My favorite above is probably ProhibitEvilModules. Aside from the colorful name, a development team can use it to steer people towards an organization’s favored solutions rather than “deprecated, buggy, unsupported, or insecure” ones. By default, it prohibits Class::ISA, Pod::Plainer, Shell, and Switch, but you should curate and configure a list within your team.
Speaking of working within a team, although perlcritic is meant to be a vital tool to ensure good practices, it’s no substitute for manual peer code review. Those reviews can lead to the creation or adoption of new automated policies to save time and settle arguments, but such work should be done collaboratively after achieving some kind of consensus. This is true whether you’re a team of employees working on proprietary software or a group of volunteers developing open source.
Of course, reasonable people can and do disagree over any of the included policies, but as a reasonable person, you should have good reasons to disagree before you either configure perlcritic appropriately or selectively and knowingly bend the rules where required. Other CPAN authors have even provided their own additions to perlcritic, so it’s worth searching CPAN under “Perl::Critic::Policy::” for more examples. In particular, these community-inspired policies group a number of recommendations from Perl developers on Internet Relay Chat (IRC).
Personally, although I adhere to my employer’s standardized configuration when testing and reviewing code, I like to run perlcritic on the “brutal” setting before committing my own. What do you prefer? Let me know in the comments below.
A recent Lobsters post lauding the virtues of AWK reminded me that although the language is powerful and lightning-fast, I usually find myself exceeding its capabilities and reaching for Perl instead. One such application is analyzing voluminous log files such as the ones generated by this blog. Yes, WordPress has stats, but I’ve never let reinvention of the wheel get in the way of a good programming exercise.
So I whipped this script up on Sunday night while watching RuPaul’s Drag Race reruns. It parses my Apache web server log files and reports on hits from week to week.
#!/usr/bin/env perl
use strict;
use warnings;
use Syntax::Construct 'operator-double-diamond';
use Regexp::Log::Common;
use DateTime::Format::HTTP;
use List::Util 1.33 'any';
use Number::Format 'format_number';
my $parser = Regexp::Log::Common->new(
format => ':extended',
capture => [qw<req ts status>],
);
my @fields = $parser->capture;
my $compiled_re = $parser->regexp;
my @skip_uri_patterns = qw<
^/+robots.txt
[-\w]*sitemap[-\w]*.xml
^/+wp-
/feed/?$
^/+?rest_route=
>;
my ( %count, %week_of );
while ( <<>> ) {
my %log;
@log{@fields} = /$compiled_re/;
# only interested in successful or cached requests
next unless $log{status} =~ /^2/ or $log{status} == 304;
my ( $method, $uri, $protocol ) = split ' ', $log{req};
next unless $method eq 'GET';
next if any { $uri =~ $_ } @skip_uri_patterns;
my $dt = DateTime::Format::HTTP->parse_datetime( $log{ts} );
my $key = sprintf '%u-%02u', $dt->week;
# get first date of each week
$week_of{$key} ||= $dt->date;
$count{$key}++;
}
printf "Week of %s: % 10s\n", $week_of{$_}, format_number( $count{$_} )
for sort keys %count;Here’s some sample output:
Week of 2021-07-31: 2,672
Week of 2021-08-02: 16,222
Week of 2021-08-09: 12,609
Week of 2021-08-16: 17,714
Week of 2021-08-23: 14,462
Week of 2021-08-30: 11,758
Week of 2021-09-06: 14,811
Week of 2021-09-13: 407I first started prototyping this on the command line as if it were an awk one-liner by using the perl -n and -a flags. The former wraps code in a while loop over the <> “diamond operator”, processing each line from standard input or files passed as arguments. The latter splits the fields of the line into an array named @F. It looked something like this while I was listing URIs (locations on the website):
gunzip -c ~/logs/phoenixtrap.com-ssl_log-*.gz | \
perl -anE 'say $F[6]'But once I realized I’d need to filter out a bunch of URI patterns and do some aggregation by date, I turned it into a script and turned to CPAN.
There I found Regexp::Log::Common and DateTime::Format::HTTP, which let me pull apart the Apache log format and its timestamp strings without having to write even more complicated regular expressions myself. (As noted above, this was already a wheel-reinvention exercise; no need to compound that further.)
Regexp::Log::Common builds a compiled regular expression based on the log format and fields you’re interested in, so that’s the constructor on lines 11 through 14. The expression then returns those fields as a list, which I’m assigning to a hash slice with those field names as keys in line 29. I then skip over requests that aren’t successful or browser cache hits, skip over requests that don’t GET web pages or other assets (e.g., POSTs to forms or updating other resources), and skip over the URI patterns mentioned earlier.
(Those patterns are worth a mention: they include the robots.txt and sitemap XML files used by search engine indexers, WordPress administration pages, files used by RSS newsreaders subscribed to my blog, and routes used by the Jetpack WordPress add-on. If you’re adapting this for your site you might need to customize this list based on what software you use to run it.)
Lines 38 and 39 parse the timestamp from the log into a DateTime object using DateTime::Format::HTTP and then build the key used to store the per-week hit count. The last lines of the loop then grab the first date of each new week (assuming the log is in chronological order) and increment the count. Once finished, lines 46 and 47 provide a report sorted by week, displaying it as a friendly “Week of date” and the hit counts aligned to the right with sprintf. Number::Format’s format_number function displays the totals with thousands separators.
Update: After this was initially published. astute reader Chris McGowan noted that I had a bug where $log{status} was assigned the value 304 with the = operator rather than compared with ==. He also suggested I use the double-diamond <<>> operator introduced in Perl v5.22.0 to avoid maliciously-named files. Thanks, Chris!
DateTime is a very powerful module but this comes at a price of speed and memory. Something simpler like Date::WeekNumber should yield performance improvements, especially as my logs grow (here’s hoping). It requires a bit more manual massaging of the log dates to convert them into something the module can use, though:
#!/usr/bin/env perl
use strict;
use warnings;
use Syntax::Construct qw<
operator-double-diamond
regex-named-capture-group
>;
use Regexp::Log::Common;
use Date::WeekNumber 'iso_week_number';
use List::Util 1.33 'any';
use Number::Format 'format_number';
my $parser = Regexp::Log::Common->new(
format => ':extended',
capture => [qw<req ts status>],
);
my @fields = $parser->capture;
my $compiled_re = $parser->regexp;
my @skip_uri_patterns = qw<
^/+robots.txt
[-\w]*sitemap[-\w]*.xml
^/+wp-
/feed/?$
^/+?rest_route=
>;
my %month = (
Jan => '01',
Feb => '02',
Mar => '03',
Apr => '04',
May => '05',
Jun => '06',
Jul => '07',
Aug => '08',
Sep => '09',
Oct => '10',
Nov => '11',
Dec => '12',
);
my ( %count, %week_of );
while ( <<>> ) {
my %log;
@log{@fields} = /$compiled_re/;
# only interested in successful or cached requests
next unless $log{status} =~ /^2/ or $log{status} == 304;
my ( $method, $uri, $protocol ) = split ' ', $log{req};
next unless $method eq 'GET';
next if any { $uri =~ $_ } @skip_uri_patterns;
# convert log timestamp to YYYY-MM-DD
# for Date::WeekNumber
$log{ts} =~ m!^
(?<day>\d\d) /
(?<month>...) /
(?<year>\d{4}) : !x;
my $date = "$+{year}-$month{ $+{month} }-$+{day}";
my $week = iso_week_number($date);
$week_of{$week} ||= $date;
$count{$week}++;
}
printf "Week of %s: % 10s\n", $week_of{$_}, format_number( $count{$_} )
for sort keys %count;It looks almost the same as the first version, with the addition of a hash to convert month names to numbers and the actual conversion (using named regular expression capture groups for readability, using Syntax::Construct to check for that feature). On my server, this results in a ten- to eleven-second savings when processing two months of compressed logs.
What’s next? Pretty graphs? Drilling down to specific blog posts? Database storage for further queries and analysis? Perl and CPAN make it possible to go far beyond what you can do with AWK. What would you add or change? Let me know in the comments.