Scraping the Dragon with Perl and Mojolicious

Every extend­ed Labor Day week­end, 80,000 fans of pop cul­ture descend on Atlanta for Dragon Con. It’s a sprawl­ing choose-​your-​own adven­ture of a con­ven­tion with 38 pro­gram­ming tracks and over 5,000 hours of events. It spans five down­town host hotels, and there is no way to see it all.

Sadly, this year’s con is almost over. Still, I thought I’d share a lit­tle script I wrote to help me make sense of it all.

The offi­cial mobile app is fine for search­ing and book­mark­ing events, speak­ers, and exhibitors. Nonetheless, it’s not suit­able for scan­ning the whole land­scape at once. I want­ed a sin­gle, scrol­lable view of every event, before I even packed my cosplay.

Even in the app’s tablet ver­sion, the Dragon Con Events area is a scroll-fest.

The web ver­sion of the app gave me exact­ly what I need­ed: pre­dictable per-​day URLs and seman­ti­cal­ly marked-​up HTML. That meant I can skip the API hunt, skip the man­u­al scrolling, and go straight to scraping.

Inspecting the HTML reveals per-​day event URLs and per-​event <div> blocks.

From Chaos to Clarity in 40 lines

We’re about to turn a messy, multi-​day, multi-​hotel sched­ule into one clean, scroll-​once list. This is the forty-​five-​line Perl map that gets us there, aid­ed by the Mojolicious web toolkit.

Laying the Groundwork: Tools for the Job

#!/usr/bin/env perl

use v5.40;

use Carp;
use English;
use Mojo::UserAgent;
use Mojo::URL;
use Mojo::DOM;
use Mojo::Collection q(c);
use Time::Piece;
use HTML::HTML5::Entities;
use Memoize;

binmode STDOUT, ':encoding(UTF-8)'
  or croak "Couldn't encode STDOUT: $OS_ERROR";

my $ua   = Mojo::UserAgent->new();
my $site = Mojo::URL->new('https://app.core-apps.com');
my $path = '/dragoncon25/events/view_by_day';

What’s hap­pen­ing: Load the mod­ules that will do the heavy lifting–HTTP fetch­es, DOM pars­ing, date han­dling, Unicode cleanup. Lock STDOUT to UTF8 so char­ac­ters like curly quotes and em-​dashes don’t break the out­put. Point the script at the base sched­ule URL.

Remembering the Days Without Re-Parsing

my $date_from_dom = memoize( sub ($dom) {
  return content_at( $dom, 'div.section_header[class~="alt"]' );
} );

What’s hap­pen­ing: Create a mem­o­ized helper that plucks the date from a day’s HTML and caches it. That way, if we need it again, we skip the DOM re-​parse and keep the pipeline fast.

content_at is a helper func­tion I define later.

Starting Where the App Starts

my $today_dom = Mojo::DOM->new( $ua->get("$site$path")->result->text );

What’s hap­pen­ing: Fetch the today” view–the same default the app shows. This is so we have a known start­ing point for build­ing the full timeline.

Collecting the Whole Timeline

my $day_doms = c(
  $today_dom,
  $today_dom->find(qq(div.filter_box-days > a[href^="$path?day="]))
    ->map( \&dom_from_anchor )
    ->to_array->@*,
)->sort( sub { day_epoch($a) <=> day_epoch($b) } );

What’s hap­pen­ing: Grab every day link from the fil­ter bar, fetch each day’s HTML, and sort them chrono­log­i­cal­ly. Now we’ve got the entire con’s sched­ule in mem­o­ry, ready to process.

dom_from_anchor and day_epoch are two more helper func­tions explained fur­ther down.

Turning HTML into a Human-​Readable Schedule

$day_doms->each( sub {    # process each day's events
  my $date = $date_from_dom->($_);

  $_->find('a.bookmark[data-type="events"] + a.object_link')
    ->each( sub {         # output start time + title

      my $time    = content_at( $_, 'div.line[class~="two"]' );
      my $title   = content_at( $_, 'div.line[class~="one"]' );
      my ($start) = split /\s*\p{Dash_Punctuation}/, $time;

      say "$date $start: ", decode_entities($title);
    } );
} );

What’s hap­pen­ing: For each day, find every event link and pull out the start time and title. Split the time clean­ly on any dash and decode HTML enti­ties so the out­put reads like a real schedule.

The Little Routines That Make It All Work

sub dom_from_anchor ($dom) {    # fetch DOM for a day link
  return Mojo::DOM->new(
    $ua->get( Mojo::URL->new( $dom->attr('href') )->to_abs($site) )
      ->result->text );
}

sub day_epoch ($dom) {    # parse date into epoch
  return Time::Piece->strptime( $date_from_dom->($dom), '%A, %b %e' )
    ->epoch;
}

# extract and trim text from selector
sub content_at ( $dom, @args ) { return trim $dom->at(@args)->content }

What’s hap­pen­ing:

  1. dom_from_anchor: fetch and pars­es a linked days’ HTML.
  2. day_epoch: turn a date string into a sort-​able epoch.
  3. content_at: extract and trim text from a DOM frag­ment, giv­en a CSS selector.

These helpers keep the main flow read­able and re-usable.

The Schedule, Unlocked

Run the script and you get a clean, UTF-​8-​safe list of every event, in chrono­log­i­cal order, across all days. No swip­ing around, no tap­ping, no what did I miss?” anx­i­ety. (Ha, who am I kid­ding? There’s too much going on at Dragon Con to not end up miss­ing something.)

An exam­ple run of the script in my ter­mi­nal. Each line is Day, Date Time: Event Title”, sort­ed chrono­log­i­cal­ly across the whole con.

And here’s just a small slice of the 2,500+ lines it produces:

Sunday, Aug 31 11:30 AM: Unmasking Sherlock: Beyond the Many Faces
Sunday, Aug 31 11:30 AM: Weaponization of the FCC and Other Agencies to Chill Speech
Sunday, Aug 31 11:30 AM: Where Physics Gets Weird
. . .
Sunday, Aug 31 11:50 AM: Photo Session: Amelia Tyler
Sunday, Aug 31 11:50 AM: Photo Session: Cissy Jones
Sunday, Aug 31 11:50 AM: Photo Session: Emma Gregory
. . .
Sunday, Aug 31 12:00 PM: Dragon Con Mashups
Sunday, Aug 31 12:00 PM: James J. Butcher and R.R. Virdi signing at The Missing Volume booth# 1300
Sunday, Aug 31 12:00 PM: JoeDan Worley and Eric Dontigney signing at the Shadow Alley Press Booth# 2
. . .
Sunday, Aug 31 12:00 PM: Photo Session: Robert Duncan McNeill
Sunday, Aug 31 12:00 PM: Photo Session: Robert Picardo
Sunday, Aug 31 12:00 PM: Photo Session: Tamara Taylor

Key Techniques

Here’s the fun part–the tech­niques that make this tidy, scroll-​once list possible.

CSS selectors for precision

I used a.bookmark[data-type="events" + a.object_link] to grab only the event title links, and div.line[class~="two" /​div.line[class~="one"] for time and title, respec­tive­ly. This avoids scrap­ing unre­lat­ed elements.

Memoization for efficiency

memoize caches the date string for each day’s DOM so I did­n’t end up re-​parsing the HTML frag­ment mul­ti­ple times.

Unicode-​safe splitting

\p{Dash_Punctuation} match­es any dash type (em, en, hyphen-​minus, etc.), so I could split times reli­ably with­out wor­ry­ing about which dash the site uses.

Functional chaining

Mojo::Collections map, sort, and each meth­ods let me express the scrape→transform→output pipeline in a lin­ear, read­able way.

Entity decoding at output

HTML::HTML5::Entitiesdecode_entities is applied right before print­ing, so HTML enti­ties like &amp; or &quot; are human-​readable in the final output.

A Pattern You Can Take Anywhere

The same approach that tamed Dragon Con’s chaos works any­where you’ve got:

  • Predictable URLs–so you can iter­ate with­out guesswork
  • Consistent HTML structure–so your selec­tors stay stable
  • A need to see every­thing at once–so you can make deci­sions with­out pag­ing or filtering

From fan con­ven­tions to con­fer­ence sched­ules, from local sports fix­tures to film fes­ti­val line‑ups–the same pat­tern applies. Sometimes the right tool isn’t a sprawl­ing frame­work or heavy­weight API client. It’s a forty‑odd‑line Perl script that does one thing with ruth­less clarity.

Because once you’ve tamed a sched­ule like this, the only lines you’ll stand in are the ones that feel like part of the show.


Discover more from The Phoenix Trap

Subscribe to get the latest posts sent to your email.

Mark Gardner Avatar

Hi, I’m Mark.

Hi, I’m Mark Gard­ner, and this is my personal blog. I show software developers how to level up by building production-ready things that work. Clear code, real projects, lessons learned.