Scraping the Dragon with Perl and Mojolicious

Every extend­ed Labor Day week­end, 80,000 fans of pop cul­ture descend on Atlanta for Dragon Con. It’s a sprawl­ing choose-​your-​own adven­ture of a con­ven­tion with 38 pro­gram­ming tracks and over 5,000 hours of events. It spans five down­town host hotels, and there is no way to see it all.

Sadly, this year’s con is almost over. Still, I thought I’d share a lit­tle script I wrote to help me make sense of it all.

The offi­cial mobile appis fine for search­ing and book­mark­ing events, speak­ers, and exhibitors. Nonetheless, it’s not suit­able for scan­ning the whole land­scape at once. I want­ed a sin­gle, scrol­lable view of every event, before I even packed my cosplay.

Even in the app’s tablet ver­sion, the Dragon Con Events area is a scroll-fest.

The web ver­sion of the appgave me exact­ly what I need­ed: pre­dictable per-​day URLs and seman­ti­cal­ly marked-​up HTML. That meant I can skip the API hunt, skip the man­u­al scrolling, and go straight to scraping.

Inspecting the HTML reveals per-​day event URLs and per-​event <div>blocks.

From Chaos to Clarity in 40 lines

We’re about to turn a messy, multi-​day, multi-​hotel sched­ule into one clean, scroll-​once list. This is the forty-​five-​line Perl map that gets us there, aid­ed by the Mojoliciousweb toolkit.

Laying the Groundwork: Tools for the Job

#!/usr/bin/env perl

use v5.40;

use Carp;
use English;
use Mojo::UserAgent;
use Mojo::URL;
use Mojo::DOM;
use Mojo::Collection q(c);
use Time::Piece;
use HTML::HTML5::Entities;
use Memoize;

binmode STDOUT, ':encoding(UTF-8)'
  or croak "Couldn't encode STDOUT: $OS_ERROR";

my $ua   = Mojo::UserAgent->new();
my $site = Mojo::URL->new('https://app.core-apps.com');
my $path = '/dragoncon25/events/view_by_day';

What’s hap­pen­ing:Load the mod­ules that will do the heavy lifting–HTTP fetch­es, DOM pars­ing, date han­dling, Unicode cleanup. Lock STDOUT to UTF‑8 so char­ac­ters like curly quotes and em-​dashes don’t break the out­put. Point the script at the base sched­ule URL.

Remembering the Days Without Re-Parsing

my $date_from_dom = memoize( sub ($dom) {
  return content_at( $dom, 'div.section_header[class~="alt"]' );
} );

What’s hap­pen­ing:Create a mem­o­ized helper that plucks the date from a day’s HTML and caches it. That way, if we need it again, we skip the DOM re-​parse and keep the pipeline fast.

content_atis a helper func­tion I define later.

Starting Where the App Starts

my $today_dom = Mojo::DOM->new( $ua->get("$site$path")->result->text );

What’s hap­pen­ing:Fetch the ​“today” view–the same default the app shows. This is so we have a known start­ing point for build­ing the full timeline.

Collecting the Whole Timeline

my $day_doms = c(
  $today_dom,
  $today_dom->find(qq(div.filter_box-days > a[href^="$path?day="]))
    ->map( \&dom_from_anchor )
    ->to_array->@*,
)->sort( sub { day_epoch($a) <=> day_epoch($b) } );

What’s hap­pen­ing:Grab every day link from the fil­ter bar, fetch each day’s HTML, and sort them chrono­log­i­cal­ly. Now we’ve got the entire con’s sched­ule in mem­o­ry, ready to process.

dom_from_anchorand day_epochare two more helper func­tions explained fur­ther down.

Turning HTML into a Human-​Readable Schedule

$day_doms->each( sub {    # process each day's events
  my $date = $date_from_dom->($_);

  $_->find('a.bookmark[data-type="events"] + a.object_link')
    ->each( sub {         # output start time + title

      my $time    = content_at( $_, 'div.line[class~="two"]' );
      my $title   = content_at( $_, 'div.line[class~="one"]' );
      my ($start) = split /\s*\p{Dash_Punctuation}/, $time;

      say "$date $start: ", decode_entities($title);
    } );
} );

What’s hap­pen­ing:For each day, find every event link and pull out the start time and title. Split the time clean­ly on any dash and decode HTML enti­ties so the out­put reads like a real schedule.

The Little Routines That Make It All Work

sub dom_from_anchor ($dom) {    # fetch DOM for a day link
  return Mojo::DOM->new(
    $ua->get( Mojo::URL->new( $dom->attr('href') )->to_abs($site) )
      ->result->text );
}

sub day_epoch ($dom) {    # parse date into epoch
  return Time::Piece->strptime( $date_from_dom->($dom), '%A, %b %e' )
    ->epoch;
}

# extract and trim text from selector
sub content_at ( $dom, @args ) { return trim $dom->at(@args)->content }

What’s hap­pen­ing:

  1. dom_from_anchor: fetch and pars­es a linked days’ HTML.
  2. day_epoch: turn a date string into a sort-​able epoch.
  3. content_at: extract and trim text from a DOM frag­ment, giv­en a CSS selector.

These helpers keep the main flow read­able and re-usable.

The Schedule, Unlocked

Run the script and you get a clean, UTF-​8-​safe list of every event, in chrono­log­i­cal order, across all days. No swip­ing around, no tap­ping, no ​“what did I miss?” anx­i­ety. (Ha, who am I kid­ding? There’s too much going on at Dragon Con to notend up miss­ing something.)

An exam­ple run of the script in my ter­mi­nal. Each line is ​“Day, Date Time: Event Title”, sort­ed chrono­log­i­cal­ly across the whole con.

And here’s just a small slice of the 2,500+ lines it produces:

Sunday, Aug 31 11:30 AM: Unmasking Sherlock: Beyond the Many Faces
Sunday, Aug 31 11:30 AM: Weaponization of the FCC and Other Agencies to Chill Speech
Sunday, Aug 31 11:30 AM: Where Physics Gets Weird
. . .
Sunday, Aug 31 11:50 AM: Photo Session: Amelia Tyler
Sunday, Aug 31 11:50 AM: Photo Session: Cissy Jones
Sunday, Aug 31 11:50 AM: Photo Session: Emma Gregory
. . .
Sunday, Aug 31 12:00 PM: Dragon Con Mashups
Sunday, Aug 31 12:00 PM: James J. Butcher and R.R. Virdi signing at The Missing Volume booth# 1300
Sunday, Aug 31 12:00 PM: JoeDan Worley and Eric Dontigney signing at the Shadow Alley Press Booth# 2
. . .
Sunday, Aug 31 12:00 PM: Photo Session: Robert Duncan McNeill
Sunday, Aug 31 12:00 PM: Photo Session: Robert Picardo
Sunday, Aug 31 12:00 PM: Photo Session: Tamara Taylor

Key Techniques

Here’s the fun part–the tech­niques that make this tidy, scroll-​once list possible.

CSS selectors for precision

I used a.bookmark[data-type="events" + a.object_link]to grab only the event title links, and div.line[class~="two"/​div.line[class~="one"]for time and title, respec­tive­ly. This avoids scrap­ing unre­lat­ed elements.

Memoization for efficiency

memoizecaches the date string for each day’s DOM so I did­n’t end up re-​parsing the HTML frag­ment mul­ti­ple times.

Unicode-​safe splitting

\p{Dash_Punctuation}match­es any dash type (em, en, hyphen-​minus, etc.), so I could split times reli­ably with­out wor­ry­ing about which dash the site uses.

Functional chaining

Mojo::Collection​’s map, sort, and eachmeth­ods let me express the scrape→transform→output pipeline in a lin­ear, read­able way.

Entity decoding at output

HTML::HTML5::Entities’ decode_entitiesis applied right before print­ing, so HTML enti­ties like &amp;or &quot;are human-​readable in the final output.

A Pattern You Can Take Anywhere

The same approach that tamed Dragon Con’s chaos works any­where you’ve got:

  • Predictable URLs–so you can iter­ate with­out guesswork
  • Consistent HTML structure–so your selec­tors stay stable
  • A need to see every­thingat once–so you can make deci­sions with­out pag­ing or filtering

From fan con­ven­tions to con­fer­ence sched­ules, from local sports fix­tures to film fes­ti­val line‑ups–the same pat­tern applies. Sometimes the right tool isn’t a sprawl­ing frame­work or heavy­weight API client. It’s a forty‑odd‑line Perl script that does one thing with ruth­less clarity.

Because once you’ve tamed a sched­ule like this, the only lines you’ll stand in are the ones that feel like part of the show.


Discover more from The Phoenix Trap

Subscribe to get the latest posts sent to your email.

Mark Gardner Avatar

Hi, I’m Mark.

Hi, I’m Mark Gard­ner, and this is my personal blog. I show software developers how to level up by building production-ready things that work. Clear code, real projects, lessons learned.

Your post must include a link to this post.

How does this work?
Write a response on your own website and include a link to this post. Then submit your response’s URL here. Your response may appear here after moderation. To update or remove it, edit or delete your original post and submit the same URL again. Learn more about Webmentions.