Tag: meetup

  • Scraping the Dragon with Perl and Mojolicious

    Scraping the Dragon with Perl and Mojolicious

    Every extend­ed Labor Day week­end, 80,000 fans of pop cul­ture descend on Atlanta for Dragon Con. It’s a sprawl­ing choose-​your-​own adven­ture of a con­ven­tion with 38 pro­gram­ming tracks and over 5,000 hours of events. It spans five down­town host hotels, and there is no way to see it all.

    Sadly, this year’s con is almost over. Still, I thought I’d share a lit­tle script I wrote to help me make sense of it all.

    The offi­cial mobile app is fine for search­ing and book­mark­ing events, speak­ers, and exhibitors. Nonetheless, it’s not suit­able for scan­ning the whole land­scape at once. I want­ed a sin­gle, scrol­lable view of every event, before I even packed my cosplay.

    Even in the app’s tablet ver­sion, the Dragon Con Events area is a scroll-fest.

    The web ver­sion of the app gave me exact­ly what I need­ed: pre­dictable per-​day URLs and seman­ti­cal­ly marked-​up HTML. That meant I can skip the API hunt, skip the man­u­al scrolling, and go straight to scraping.

    Inspecting the HTML reveals per-​day event URLs and per-​event <div> blocks.

    From Chaos to Clarity in 40 lines

    We’re about to turn a messy, multi-​day, multi-​hotel sched­ule into one clean, scroll-​once list. This is the forty-​five-​line Perl map that gets us there, aid­ed by the Mojolicious web toolkit.

    Laying the Groundwork: Tools for the Job

    #!/usr/bin/env perl
    
    use v5.40;
    
    use Carp;
    use English;
    use Mojo::UserAgent;
    use Mojo::URL;
    use Mojo::DOM;
    use Mojo::Collection q(c);
    use Time::Piece;
    use HTML::HTML5::Entities;
    use Memoize;
    
    binmode STDOUT, ':encoding(UTF-8)'
      or croak "Couldn't encode STDOUT: $OS_ERROR";
    
    my $ua   = Mojo::UserAgent->new();
    my $site = Mojo::URL->new('https://app.core-apps.com');
    my $path = '/dragoncon25/events/view_by_day';

    What’s hap­pen­ing: Load the mod­ules that will do the heavy lifting–HTTP fetch­es, DOM pars­ing, date han­dling, Unicode cleanup. Lock STDOUT to UTF8 so char­ac­ters like curly quotes and em-​dashes don’t break the out­put. Point the script at the base sched­ule URL.

    Remembering the Days Without Re-Parsing

    my $date_from_dom = memoize( sub ($dom) {
      return content_at( $dom, 'div.section_header[class~="alt"]' );
    } );

    What’s hap­pen­ing: Create a mem­o­ized helper that plucks the date from a day’s HTML and caches it. That way, if we need it again, we skip the DOM re-​parse and keep the pipeline fast.

    content_at is a helper func­tion I define later.

    Starting Where the App Starts

    my $today_dom = Mojo::DOM->new( $ua->get("$site$path")->result->text );

    What’s hap­pen­ing: Fetch the today” view–the same default the app shows. This is so we have a known start­ing point for build­ing the full timeline.

    Collecting the Whole Timeline

    my $day_doms = c(
      $today_dom,
      $today_dom->find(qq(div.filter_box-days > a[href^="$path?day="]))
        ->map( \&dom_from_anchor )
        ->to_array->@*,
    )->sort( sub { day_epoch($a) <=> day_epoch($b) } );

    What’s hap­pen­ing: Grab every day link from the fil­ter bar, fetch each day’s HTML, and sort them chrono­log­i­cal­ly. Now we’ve got the entire con’s sched­ule in mem­o­ry, ready to process.

    dom_from_anchor and day_epoch are two more helper func­tions explained fur­ther down.

    Turning HTML into a Human-​Readable Schedule

    $day_doms->each( sub {    # process each day's events
      my $date = $date_from_dom->($_);
    
      $_->find('a.bookmark[data-type="events"] + a.object_link')
        ->each( sub {         # output start time + title
    
          my $time    = content_at( $_, 'div.line[class~="two"]' );
          my $title   = content_at( $_, 'div.line[class~="one"]' );
          my ($start) = split /\s*\p{Dash_Punctuation}/, $time;
    
          say "$date $start: ", decode_entities($title);
        } );
    } );

    What’s hap­pen­ing: For each day, find every event link and pull out the start time and title. Split the time clean­ly on any dash and decode HTML enti­ties so the out­put reads like a real schedule.

    The Little Routines That Make It All Work

    sub dom_from_anchor ($dom) {    # fetch DOM for a day link
      return Mojo::DOM->new(
        $ua->get( Mojo::URL->new( $dom->attr('href') )->to_abs($site) )
          ->result->text );
    }
    
    sub day_epoch ($dom) {    # parse date into epoch
      return Time::Piece->strptime( $date_from_dom->($dom), '%A, %b %e' )
        ->epoch;
    }
    
    # extract and trim text from selector
    sub content_at ( $dom, @args ) { return trim $dom->at(@args)->content }

    What’s hap­pen­ing:

    1. dom_from_anchor: fetch and pars­es a linked days’ HTML.
    2. day_epoch: turn a date string into a sort-​able epoch.
    3. content_at: extract and trim text from a DOM frag­ment, giv­en a CSS selector.

    These helpers keep the main flow read­able and re-usable.

    The Schedule, Unlocked

    Run the script and you get a clean, UTF-​8-​safe list of every event, in chrono­log­i­cal order, across all days. No swip­ing around, no tap­ping, no what did I miss?” anx­i­ety. (Ha, who am I kid­ding? There’s too much going on at Dragon Con to not end up miss­ing something.)

    An exam­ple run of the script in my ter­mi­nal. Each line is Day, Date Time: Event Title”, sort­ed chrono­log­i­cal­ly across the whole con.

    And here’s just a small slice of the 2,500+ lines it produces:

    Sunday, Aug 31 11:30 AM: Unmasking Sherlock: Beyond the Many Faces
    Sunday, Aug 31 11:30 AM: Weaponization of the FCC and Other Agencies to Chill Speech
    Sunday, Aug 31 11:30 AM: Where Physics Gets Weird
    . . .
    Sunday, Aug 31 11:50 AM: Photo Session: Amelia Tyler
    Sunday, Aug 31 11:50 AM: Photo Session: Cissy Jones
    Sunday, Aug 31 11:50 AM: Photo Session: Emma Gregory
    . . .
    Sunday, Aug 31 12:00 PM: Dragon Con Mashups
    Sunday, Aug 31 12:00 PM: James J. Butcher and R.R. Virdi signing at The Missing Volume booth# 1300
    Sunday, Aug 31 12:00 PM: JoeDan Worley and Eric Dontigney signing at the Shadow Alley Press Booth# 2
    . . .
    Sunday, Aug 31 12:00 PM: Photo Session: Robert Duncan McNeill
    Sunday, Aug 31 12:00 PM: Photo Session: Robert Picardo
    Sunday, Aug 31 12:00 PM: Photo Session: Tamara Taylor

    Key Techniques

    Here’s the fun part–the tech­niques that make this tidy, scroll-​once list possible.

    CSS selectors for precision

    I used a.bookmark[data-type="events" + a.object_link] to grab only the event title links, and div.line[class~="two" /​div.line[class~="one"] for time and title, respec­tive­ly. This avoids scrap­ing unre­lat­ed elements.

    Memoization for efficiency

    memoize caches the date string for each day’s DOM so I did­n’t end up re-​parsing the HTML frag­ment mul­ti­ple times.

    Unicode-​safe splitting

    \p{Dash_Punctuation} match­es any dash type (em, en, hyphen-​minus, etc.), so I could split times reli­ably with­out wor­ry­ing about which dash the site uses.

    Functional chaining

    Mojo::Collections map, sort, and each meth­ods let me express the scrape→transform→output pipeline in a lin­ear, read­able way.

    Entity decoding at output

    HTML::HTML5::Entitiesdecode_entities is applied right before print­ing, so HTML enti­ties like &amp; or &quot; are human-​readable in the final output.

    A Pattern You Can Take Anywhere

    The same approach that tamed Dragon Con’s chaos works any­where you’ve got:

    • Predictable URLs–so you can iter­ate with­out guesswork
    • Consistent HTML structure–so your selec­tors stay stable
    • A need to see every­thing at once–so you can make deci­sions with­out pag­ing or filtering

    From fan con­ven­tions to con­fer­ence sched­ules, from local sports fix­tures to film fes­ti­val line‑ups–the same pat­tern applies. Sometimes the right tool isn’t a sprawl­ing frame­work or heavy­weight API client. It’s a forty‑odd‑line Perl script that does one thing with ruth­less clarity.

    Because once you’ve tamed a sched­ule like this, the only lines you’ll stand in are the ones that feel like part of the show.

  • Perl lightning talk: Don’t Fear map and grep”

    This week’s Perl and Raku Conference 2022 in Houston was packed with great pre­sen­ta­tions, and I humbly added to them with a five-​ish minute light­ning talk on two of Perl’s more mis­un­der­stood func­tions: map and grep.

    Sorry about the um”s and ah”s…

    I’ve writ­ten much about list pro­cess­ing in Perl, and this talk was based on the fol­low­ing blog posts:

    Overall I loved attend­ing the con­fer­ence, and it real­ly invig­o­rat­ed my par­tic­i­pa­tion in the Perl com­mu­ni­ty. Stay tuned as I resume reg­u­lar posting!

    Update for Raku

    On Twitter I nudged promi­nent Raku hack­er (and recov­ered Perl hack­er) Elizabeth Mattijsen to write about the Raku pro­gram­ming language’s map and grep func­tion­al­i­ty. Check out her five-​part series on DEV.to.

  • Video: A Year of Being Wrong on the Internet”

    I’m busy this week host­ing my par­ents’ first vis­it to Houston, but I didn’t want to let this Tuesday go by with­out link­ing to my talk from last week’s Ephemeral Miniconf. Thanks so much to Thibault Duponchelle for orga­niz­ing such a ter­rif­ic event, to all the oth­er speak­ers for com­ing togeth­er to present, and to every­one who attend­ed for wel­com­ing me.

  • This week: The ePhEmeRaL miniconf

    This week: The ePhEmeRaL miniconf

    After a lot of pro­cras­ti­na­tion, I’ve decid­ed my talk for this week’s ePhEmeRaL mini­conf will be Cunningham’s Law: A Year of Being Wrong on the Internet, or «prêch­er le faux pour savoir le vrai.»

    The event starts at 8:00 AM CST on Thursday, November 18; you can find out more about it includ­ing the full sched­ule and a time zone con­vert­er here.

  • Speaking at Developer Career Day 2021

    I’ll be run­ning a work­shop based on my recent Blogging Outside the Bubble webi­nar dur­ing Developer Career Day 2021, which is run­ning from August 6 through 7. It’s free to reg­is­ter, with top speak­ers and prac­ti­cal career discussions.

    Find out more and reg­is­ter here!