Tag: Dist::Zilla

  • Patch-​Perfect: Smarter Homebrew Upgrades on macOS

    Patch-​Perfect: Smarter Homebrew Upgrades on macOS

    This is a sto­ry about patches.

    At first, I just want­ed Homebrew to behave a lit­tle more polite­ly. Formulae should upgrade only on patch changes, with­out being dragged through minor and major bumps. That itch became a small(ish) Perl script, brew-patch-upgrade.pl.

    Along the way, I dis­cov­ered anoth­er patch was need­ed. My own log­ging adapter, Log::Any::Adapter::MacOS::OSLog, was­n’t build­ing and installing its bun­dle cor­rect­ly. Before the script can shine in the Mac Console app, I had to fix the adapter itself.

    What fol­lows is how those two threads came togeth­er. One is a tool that keeps Homebrew upgrades patch-​perfect. The oth­er is a log­ging adapter that final­ly behaves as a first-​class Perl module.

    What is Homebrew?

    Not to be con­fused with perlbrew, Homebrew is a free and open-​source pack­age man­ag­er for macOS (and Linux). It makes it easy to install and update soft­ware from the com­mand line. You don’t have to hunt down installers on web­sites. Instead, you can type com­mands like brew install wget. Homebrew will then fetch, build, and link the tool into place.

    Homebrew ≠ Perlbrew

    Everything is orga­nized under /opt/homebrew (on Apple Silicon-​based Macs) or /usr/local (on Intel). Homebrew can even man­age both command‑line util­i­ties (“for­mu­lae”) and desk­top apps (“casks”).

    In short: it’s the miss­ing pack­age man­ag­er Apple nev­er shipped, and it’s become an essen­tial part of many devel­op­ers’ workflows.

    Homebrew makes it easy to stay up to date — some­times too easy. By default, brew upgrade jumps to the lat­est ver­sion of every­thing, even across minor and major releases.

    That’s fine when you want the newest fea­tures. Yet, it can be dis­rup­tive if all you real­ly need are the qui­et, patch-​level fixes.

    Scripted patch-​only upgrades

    My brew‑patch‑upgrade.pl takes a nar­row­er view: it pars­es brew outdated com­mand, com­pares seman­tic ver­sions, and upgrades only when the patch num­ber changes.

    That means:

    • 3.2.13.2.4 will be upgraded.
    • 3.2.13.3.0 will be skipped.
    • 3.2.14.0.0 will be skipped.

    Normally, the scrip­t’s out­put is com­pa­ra­ble to brew upgrade, stream­ing the famil­iar Homebrew out­put to your ter­mi­nal for any patch-​level updates while not­ing any skipped versions:

    % brew-patch-upgrade.pl
    Skipping harfbuzz, needs manual review before upgrade at /Users/mjg/.local/bin/brew-patch-upgrade.pl line 178.
    	main::process_formula(HASH(0x7c075dc90)) called at /Users/mjg/.local/bin/brew-patch-upgrade.pl line 121
    Skipping woodpecker-cli, needs manual review before upgrade at /Users/mjg/.local/bin/brew-patch-upgrade.pl line 178.
    	main::process_formula(HASH(0x7c0c1a180)) called at /Users/mjg/.local/bin/brew-patch-upgrade.pl line 121
    ==> Upgrading 1 outdated package:
    zstd 1.5.6 -> 1.5.7
    ==> Fetching downloads for: zstd
    ==> Fetching zstd
    ==> Downloading https://ghcr.io/v2/homebrew/core/zstd/blobs/sha256:ddb0c145060bc2366ce5d58d95aa205bb15cb4c66948f20bb85e23fdb5eba7e9
    Already downloaded: /Users/mjg/Library/Caches/Homebrew/downloads/eb865576547e163ef6908f1e5762f4dc4d7fb548940f7b896ae12c0d5b202362--zstd--1.5.7.arm64_tahoe.bottle.1.tar.gz
    ==> Upgrading zstd
      1.5.6 -> 1.5.7
    ==> Pouring zstd--1.5.7.arm64_tahoe.bottle.1.tar.gz
    🍺  /opt/homebrew/Cellar/zstd/1.5.7: 32 files, 2.2MB
    ==> Running `brew cleanup zstd`...
    Disable this behaviour by setting `HOMEBREW_NO_INSTALL_CLEANUP=1`.
    Hide these hints with `HOMEBREW_NO_ENV_HINTS=1` (see `man brew`).
    Removing: /opt/homebrew/Cellar/zstd/1.5.6... (32 files, 2.1MB)
    Removing: /Users/mjg/Library/Caches/Homebrew/zstd_bottle_manifest--1.5.6... (11.9KB)
    Removing: /Users/mjg/Library/Caches/Homebrew/zstd--1.5.6... (753.9KB)
    ==> No outdated dependents to upgrade!

    But since I intend­ed this to run unat­tend­ed on a sched­ule, you can also tell it to log to a dif­fer­ent des­ti­na­tion using the envi­ron­ment vari­able LOG_ANY_DEFAULT_ADAPTER:

    % LOG_ANY_DEFAULT_ADAPTER=Stderr brew-patch-upgrade.pl
    using log adapter Log::Any::Adapter::Stderr
    outdated formulae: {casks => [],formulae => [{current_version => "12.0.0",installed_versions => ["11.5.1"],name => "harfbuzz",pinned => bless( do{\(my $o = 0)}, 'JSON::PP::Boolean' ),pinned_version => undef},{current_version => "3.10.0",installed_versions => ["3.9.0"],name => "woodpecker-cli",pinned => $VAR1->{formulae}[0]{pinned},pinned_version => undef},{current_version => "1.5.7",installed_versions => ["1.5.6","1.5.7"],name => "zstd",pinned => $VAR1->{formulae}[0]{pinned},pinned_version => undef}]}
    Skipping harfbuzz, needs manual review before upgrade at /Users/mjg/.local/bin/brew-patch-upgrade.pl line 178.
    	main::process_formula(HASH(0xa1a01f510)) called at /Users/mjg/.local/bin/brew-patch-upgrade.pl line 121
    Skipping woodpecker-cli, needs manual review before upgrade at /Users/mjg/.local/bin/brew-patch-upgrade.pl line 178.
    	main::process_formula(HASH(0xa1acbcf30)) called at /Users/mjg/.local/bin/brew-patch-upgrade.pl line 121
    Starting brew upgrade for zstd...
    ==> Upgrading 1 outdated package:
    zstd 1.5.6 -> 1.5.7
    Finished brew upgrade for zstd
    Summary: upgraded 1, skipped 2, failed 0

    That’s use­ful for debug­ging. But, the real pay­off comes when you send logs to a more sophis­ti­cat­ed adapter. A good exam­ple is my Log::Any::Adapter::MacOS::OSLog pack­age. It tags dif­fer­ent lev­els of log mes­sages in macOS’ uni­fied log­ging sys­tem. These mes­sages can then be fil­tered and searched in the Console util­i­ty app.

    The com­mand is like the Stderr exam­ple above:

    % LOG_ANY_DEFAULT_ADAPTER=MacOS::OSLog brew-patch-upgrade.pl

    And pro­duces results that can be viewed in Console:

    Screenshot of macOS' Console app showing messages from the com.phoenixtrap.brew-patch-upgrade subsystem

    Or using the macOS log show command:

    Screenshot of macOS' iTerm2 app showing messages from the com.phoenixtrap.brew-patch-upgrade subsystem of the unified log

    At the end, the script logs a sum­ma­ry of upgrad­ed, skipped, and failed for­mu­lae. On fail­ure, it exits non-​zero to make it easy to use in automation.

    Highlights from the script source code

    For those curi­ous about the inter­nals, here are a few high­lights from the source.

    Skipping major and minor version bumps

    use version;
    ...
    sub process_formula ($formula_ref) {
        my $name = $formula_ref->{name};
        my ( $current, $installed ) = map { defined and qv($_) } (
            $formula_ref->{current_version},
            $formula_ref->{installed_versions}->[0] );
        unless ( $current and $installed ) {
            $logger->debug(
                'no current and/or installed version detected for',
                $name );
            return 'skipped';
        }
        my $result = 'failed';    # default
        my @current   = $current->{version}->@*;
        my @installed = $installed->{version}->@*;
        #<<<
        unless (@current   >= 3
            and @installed >= 3
            and $installed[0] == $current[0]
            and $installed[1] == $current[1]
            and $installed[2] <  $current[2] )
        #>>>
        {
            carp "Skipping $name, needs manual review before upgrade";
            return 'skipped';
        }
        ...

    At the start, process_formula checks whether a for­mu­la from brew outdated has a valid seman­tic ver­sion. Only then does it com­pare patch numbers.

    To that end, it uses the core Perl version mod­ule’s qv func­tion. This pars­es the cur­rent and lat­est installed ver­sion, avoid­ing a com­pli­cat­ed reg­u­lar expres­sion. The result­ing ver­sion objects can be treat­ed as hash ref­er­ences with a version key. This key is itself a ref­er­ence to an array con­tain­ing the major, minor, and patch ver­sion num­bers. If the major or minor ver­sion num­bers dif­fer between installed and cur­rent, the script calls for a man­u­al review. Then it returns a skipped” status.

    Setting up the loggers

    use Log::Any qw($logger);
    use Log::Any::Adapter;
    ...
    use constant {
        MAC_LOG_ADAPTER_CLASS => 'Log::Any::Adapter::MacOS::OSLog',
        MAC_LOG_SUBSYSTEM     => 'com.phoenixtrap.brew-patch-upgrade',
    };
    my %brew_log;
    if ( $ENV{LOG_ANY_DEFAULT_ADAPTER} ) {
        my $adapter = Log::Any::Adapter->get(__PACKAGE__);
        Log::Any::Adapter->set( q(+) . ref $adapter,
            subsystem => MAC_LOG_SUBSYSTEM )
            if $adapter->isa(MAC_LOG_ADAPTER_CLASS);
        $logger->debug( 'using log adapter', ref $adapter );
        $SIG{__WARN__} = sub ($message) { $logger->alert($message) };
        $SIG{__DIE__}  = sub ($message) { $logger->fatal($message) };
        foreach my $stdio (qw(stdout stderr)) {
            Log::Any::Adapter->set(
                { category => "brew.$stdio" },
                q(+) . ref $adapter,
                (   subsystem   => MAC_LOG_SUBSYSTEM,
                    os_category => "brew-$stdio",
                )x!!$adapter->isa(MAC_LOG_ADAPTER_CLASS),
            );
            $brew_log{$stdio}
                = Log::Any->get_logger( category => "brew.$stdio" );
        }
    }

    If LOG_ANY_DEFAULT_ADAPTER is set, the script cre­ates three log­gers: one for the pro­gram itself, one for brew.stdout, and one for brew.stderr. All three have log­ic to check if the default log­ger class is Log::Any::Adapter::MacOS::OSLog. If it is, an appro­pri­ate OS-​level log­ging cat­e­go­ry is set. A unique sub­sys­tem name for the script, com.phoenixtrap.brew-patch-upgrade”, is also assigned.

    Unfortunately, while work­ing on this macOS-​specific log­ging code, I made a dis­cov­ery. I real­ized that I need­ed to fix my log adapter class with a patch of its own.

    OSLog revisited

    I was proud of my ear­li­er work on the MacOS::OSLog log adapter — a clever wrap­per around macOS’ log­ging func­tions using Perl for­eign func­tion inter­face (FFI). I even took a vic­to­ry lap, con­vert­ing the CPAN dis­tri­b­u­tion to use Dist::Zilla for plug­gable installer cre­ation and documentation.

    But out­side the pack­aged maclog script, the mod­ule did­n’t work. Oops.

    Once I tried using the MacOS::OSLog adapter in brew-patch-upgrade.pl, the cracks became obvi­ous. The bun­dle was­n’t built or installed cor­rect­ly. My _find_my_bundle hack was brit­tle — not a foun­da­tion to build on.

    The solu­tion was to stop fak­ing it and let the FFI::Platypus tool­chain do its job. Using Dist::Zilla::Plugin::MakeMaker::Awesome in my dist.ini file, I rewrote the build process to use FFI::Build::MM in the gen­er­at­ed Makefile.PL installer. This meant that the bun­dle would com­pile and install in the right place instead of rely­ing on my _find_my_bundle hack:

    [MakeMaker::Awesome]
    delimiter = |
    ...
    header = |use FFI::Build::MM;
    header = |my $fbmm = FFI::Build::MM->new();
    header = |my %fbmm_args = $fbmm->mm_args(
    header = |  DISTNAME     => 'Log-Any-Adapter-MacOS-OSLog',
    header = |  NAME         => 'Log::Any::Adapter::MacOS::OSLog',
    header = |  VERSION_FROM => 'lib/Log/Any/Adapter/MacOS/OSLog.pm',
    header = |);
    header = |$fbmm_args{CCFLAGS} .= ' -mmacosx-version-min=10.12';
    footer = |sub MY::postamble { $fbmm->mm_postamble }
    WriteMakefile_arg = %fbmm_args

    The loca­tion of the C source code as well as com­pi­la­tion flags moved to a sep­a­rate ffi/OSLog.fbx file, which FFI::Build::MM would infer the name of based on the argu­ments passed to its mm_args method.

    {
      source => [ 'ffi/OSLog.c' ],
      cflags => [ '-mmacosx-version-min=10.12' ],
      libs   => [ '-framework', 'OSLog' ],
    }

    I also updat­ed the C source code with an #include <ffi_platypus_bundle.h> line, giv­ing the result­ing code bun­dle a prop­er entry point that FFI::Platypus could recognize.

    Finally, I set a default sub­sys­tem name (“com.example.perl”). Now users can make it their default Log::Any adapter and start log­ging immediately.

    The fix is in

    With those fix­es, the adapter now works as a native-​grade com­po­nent. No hacks. No guess­ing. Just clean, struc­tured logs flow­ing into the uni­fied log­ging system.

    Each run of brew-patch-upgrade.pl now shows a clear sub­sys­tem, with cat­e­gories for STDOUT, STDERR, and the script itself. This makes it easy to fil­ter and review. And because the script exits with a sum­ma­ry line and prop­er exit code, it slots neat­ly into automa­tion as well.

    In oth­er words, the log­ging side of brew-patch-upgrade.pl is now as patch-​perfect as the upgrade log­ic it supports.

    Looking back, I real­ized that this project was­n’t just about tam­ing Homebrew’s upgrades or fix­ing a stub­born Perl mod­ule. It was about cre­at­ing a stronger con­nec­tion between the tools I rely on and their expect­ed behav­ior. I want them to be pre­dictable, leg­i­ble, and resilient. Both pack­ages work togeth­er, keep­ing my upgrades qui­et and inten­tion­al while sur­fac­ing them in the Console app.

    Both the script and log adapter are on Codeberg, with the adapter also on CPAN. If you’d like to try them out, file issues, or sug­gest improve­ments, the repos are waiting.

  • Kaiju Boss Battle: A Dist::Zilla Journey from Chaos to Co-Op

    Kaiju Boss Battle: A Dist::Zilla Journey from Chaos to Co-Op

    Last week I wrote about devel­op­ing a Perl mod­ule enabling me to out­put log entries to macOS’ uni­fied log­ging sys­tem. This week­end’s adven­ture involved port­ing that mod­ule’s man­u­al process­es. These process­es includ­ed depen­den­cy man­age­ment, doc­u­men­ta­tion sync­ing, ver­sion bump­ing, and release. The goal was to make every­thing more auto­mat­ed and repeatable.

    And maybe even… monstrous.

    I was already famil­iar with the Dist::Zilla (DZil to its friends) suite of tools and plu­g­ins. I also knew that some Perl devel­op­ers view it as a huge bar­ri­er to entry. This per­cep­tion affects their will­ing­ness to con­tribute to oth­ers’ projects.

    So even though Log-​Any-​Adapter-​MacOS-​OSLog was a small mod­ule of inter­est to a lim­it­ed cod­ing audi­ence (macOS Perl users), I thought it wise to have both a main branch and sep­a­rate build/main branch for those pro­gram­mers who want­ed to work as though things had bare­ly changed:

    • Entire source code with full POD-​formatted documentation;
    • A Makefile.PL script to gen­er­ate a portable build­ing, test­ing, and installing Makefile;
    • Plus, every Perl dis­tri­b­u­tion should sup­ply the typ­i­cal README, MANIFEST, LICENSE, and CONTRIBUTING doc­u­men­ta­tion. This is essen­tial if it’s meant for pub­lic consumption.

    A small dis­tri­b­u­tion would also give me a mod­el I can scale up for larg­er projects. At the very least, it was anoth­er learn­ing oppor­tu­ni­ty for me.

    Boy, was it.

    Why Dist::Zilla?

    Because I know it can auto­mate away the boil­er­plate code and rep­e­ti­tion of infor­ma­tion that’s unfor­tu­nate­ly nec­es­sary in a mod­ern Perl mod­ule distribution:

    • the ver­sion num­bers in every mod­ule and script
    • the README that often includes the same intro­duc­to­ry text as the main mod­ule’s documentation
    • the nam­ing, order, and con­tent of POD sec­tions (via DZil’s sis­ter suite, Pod::Weaver), some of which repeat dis­tri­b­u­tion meta­da­ta like author, ver­sion, sup­port, copy­right, license, and so on

    If you need fur­ther detail, Dan Book’s Dist::Zilla::Starter is, as its name sug­gests. an excel­lent and remark­ably thor­ough guide to the how and why of Dist::Zilla. It even cov­ers the basic struc­ture of CPAN dis­tri­b­u­tions and the his­to­ry of Perl mod­ule author­ing tools.

    Why not something else?

    Because just as in the sto­ry of Goldilocks and the three bears, every­thing else I looked at seemed want­i­ng in some way:

    • Module::Build/​Module::Build::Tiny: Although min­i­mal, pure-​Perl, and easy to install, Module::Build prop­er still requires ship­ping extra boil­er­plate and dupli­cate meta­da­ta. ::Tiny shaves that down fur­ther but drops whole swaths of func­tion­al­i­ty. As an exam­ple, you can’t spec­i­fy at set­up time that users need a spe­cif­ic oper­at­ing sys­tem. It’s a deal-​breaker for a mod­ule that requires macOS 10.12 Sierra or newer.
    • Minilla: Opinionated con­ven­tion over con­fig­u­ra­tion, but I don’t share its opin­ions. Overriding direc­to­ry lay­out, test struc­ture, and README/​license han­dling would mean fight­ing Minilla’s defaults, DZil-​config style, with­out DZil’s plu­g­in ecosystem.
    • ShipIt: Simple, one-​file con­fig­u­ra­tion. But too sim­ple: it’s most­ly release automa­tion and not author­ing automa­tion. Everything I want­ed to tool away is still manual.
    • No or min­i­mal tool­chain: That was how I start­ed this mod­ule, with ExtUtils::MakeMaker. Totally man­u­al, with every con­trib­u­tor, includ­ing yours tru­ly, need­ing to remem­ber all the mov­ing parts by hand.

    Off to see the lizard!

    My DZil dist.ini con­fig­u­ra­tion file start­ed off sim­ply enough:

    name    = Log-Any-Adapter-MacOS-OSLog
    author  = Mark Gardner <[email protected]>
    license = Perl_5
    copyright_holder = Mark Gardner
    version = 0.0.5 ; bump as appropriate
    
    [MetaResources]
    repository.type = git
    bugtracker.web  = https://codeberg.org/mjgardner/perl-Log-Any-Adapter-MacOS-OSLog/issues
    
    [Repository]
    web = https://codeberg.org/mjgardner/perl-Log-Any-Adapter-MacOS-OSLog

    That [MetaResources] is the first evi­dence of using plu­g­ins. It pro­vides a nice tidy way to spec­i­fy the meta­da­ta. Automated tools can use this infor­ma­tion to index, exam­ine, pack­age, or install Perl distributions.

    So far so good. But then the mon­ster attacked.

    [@Filter]
    -bundle = @Basic
    -remove = GatherDir
    -remove = MakeMaker
    -remove = Readme
    ; License plugin still active here
    
    [GatherDir]
    include_dotfiles = 1
    exclude_filename = .gitignore
    exclude_filename = .build
    
    [ReadmeAnyFromPod]
    type = markdown
    filename = README.md
    location = build
    
    [CopyFilesFromBuild]
    copy = README.md
    
    ; various MakeMaker::*, Git::*, Pod::Weaver, etc. plugins

    I was get­ting errors from the dzil --build com­mand as it attempt­ed to add the same file mul­ti­ple times. These were gen­er­at­ed files com­mit­ted to the main ver­sion con­trol branch and a build/main branch for non-​DZil contributors.

    This brought me to a dead stop. I went down a rabbit-​hole exam­in­ing built files from rsync(1) with [Run::AfterBuild]., com­par­ing them to the branch I had set up.

    Plus my .perlcriticrc file was dis­ap­pear­ing from the build arti­facts despite that instruc­tion to [GatherDir] to include dot­files.

    Let them fight.”

    LICENSE to kill (files)

    Eventually I traced the LICENSE file dupli­ca­tion to duel­ing DZil plu­g­ins. The afore­men­tioned [GatherDir] was duti­ful­ly copy­ing the file from my work­ing root even as the [License] plu­g­in was gen­er­at­ing it. I did­n’t want to lose the lat­ter source of what­ev­er license I hap­pened to be using to dis­trib­ute this code.

    And .perlcriticrc? It turns out the [PruneCruft] plu­g­in does­n’t lis­ten to its friend [GatherDir] and was hoover­ing it away.

    In the end, I had to expand my manually-​configured plu­g­in ros­ter a lit­tle to keep them from fight­ing over what files came from where:

    [@Filter]
    -bundle = @Basic
    -remove = GatherDir
    -remove = PruneCruft
    -remove = MakeMaker
    -remove = Readme
    
    [PruneCruft]
    except = .perlcriticrc
    
    [GatherDir]
    include_dotfiles = 1
    exclude_filename = .gitignore
    exclude_filename = .build
    exclude_filename = LICENSE
    
    [ReadmeAnyFromPod]
    type            = markdown
    filename        = README.md
    
    [CopyFilesFromBuild]
    copy = README.md
    copy = LICENSE

    With these tweaks to Log-​Any-​Adapter-​MacOS-​OSLog’s dist.ini:

    • [License] gen­er­ates its file,
    • [CopyFilesFromBuild] copies it along with the README into the root for my commit,
    • And [GatherDir] does­n’t stomp on it like so many Tokyo city blocks.

    The build/main branch for non-​DZil con­trib­u­tors would con­tain exact­ly what they and I expect­ed. It would be a match of the DZil work­ing copy plus arti­facts, git cloneable with no surprises.

    Lessons learned, and what’s next?

    I want­ed to serve two styles of devel­op­ment and had signed myself up for a com­pli­cat­ed author­ing process. I now have more knowl­edge of which plu­g­in runs in each phase of the build. This allows me to decide between gen­er­at­ed and version-​controlled files.

    CPAN has a rich vari­ety of per­son­al Dist::Zilla::PluginBundle::s fac­tor­ing indi­vid­ual authors’ pref­er­ences into a sin­gle place. Those authors don’t have to copy-​and-​paste DZil con­fig­u­ra­tions around. It’s past time for me to mint one of my own bun­dles for more con­sis­tent and well-​understood Perl dis­tri­b­u­tions. Then I can start spin­ning up new projects with­out revis­it­ing the same pain.


    This week­end has brought on a men­tal shift. I moved from fight­ing DZil’s defaults to mak­ing it work my way. I also found sat­is­fac­tion in a pre­dictable, minimal-​effort Perl release pipeline.

    The mon­ster was just a guy in a rub­ber suit all along.