Category: Personal

  • My mini Mastodon server

    My mini Mastodon server

    Social media is how many peo­ple expe­ri­ence the Internet today. There’s a good chance you found this post through one of them. But the way we con­nect online does­n’t have to be dic­tat­ed by a hand­ful of plat­forms. Beyond the single-​app, single-​site giants, there’s the Fediverse: a con­stel­la­tion of inde­pen­dent social net­works that talk to each other.

    Think of it like email: no sin­gle com­pa­ny in charge, no cen­tral ser­vice to rule them all. It’s pow­ered by a mix of soft­ware for text, pho­tos, pod­casts, events, and more.

    Mastodon is one of the most pop­u­lar of these–a microblog­ging” plat­form like X (née Twitter), but open-​source and fed­er­at­ed. You can join one of thou­sands of servers run by oth­ers… or, if you’re me, you host your own on a tiny Mac mini sit­ting on top of a file cabinet.

    But why?

    I’d been hap­pi­ly toot­ing away on a cou­ple of well-​run Mastodon servers for years. But I kept run­ning into lit­tle things I want­ed to tweak: themes, mod­er­a­tion set­tings, even the domain of my han­dle. None of them were deal-​breakers, but they added up. Eventually, I real­ized that the only way to get exact­ly what I want­ed was to set it up myself.

    The hardware

    When I say tiny, I mean tiny: a low-​spec 2023 M2 Mac mini, 8 GB mem­o­ry, 512 GB solid-​state dri­ve. My wife and I had set it up in our home office to dri­ve a TV dis­play­ing our shared sched­ule. I had also installed a Calibre serv­er for our e‑book library.

    So the mini isn’t even close to break­ing a sweat yet. And I was encour­aged to read that oth­ers were suc­cess­ful­ly run­ning single-​user Mastodon servers on a Raspberry Pi.

    The mini was more than capa­ble. The next step was fig­ur­ing out how to run Mastodon with­out becom­ing a main­te­nance headache.

    The containers

    Mastodon’s offi­cial instal­la­tion instruc­tions involve set­ting up a vari­ety of ser­vices on a Linux VPS (vir­tu­al pri­vate serv­er). But there’s an eas­i­er, lit­er­al­ly more self-​contained way: Docker con­tain­ers, orches­trat­ed through Docker Compose and run­ning via Docker Desktop.

    Mastodon’s source code repos­i­to­ry even includes a starter docker-​compose file describing:

    • a PostgreSQL database
    • a Redis cache
    • the Mastodon web application
    • its ancil­lary stream­ing ser­vice and Sidekiq back­ground event queue

    Everything is con­tainer­ized. Still, there must be a way to keep the host safe from the open Internet.

    Avoiding overexposure

    Directly expos­ing the Mac mini to the full mal­ice of the Internet filled me with dread. And besides, my home con­nec­tion lacks a guar­an­teed fixed address to which I’d attach a domain name. My solu­tion? Cloudflare Tunnel, a ser­vice run by my domain reg­is­trar and name service.

    All I need to do is add anoth­er Docker con­tain­er ser­vice. Cloudflare man­ages the web traf­fic to and from Mastodon. The oth­er ser­vices and the host Mac then stay safe from harm.

    With the tun­nel in place, I can focus on keep­ing the set­up lean and easy to update.

    Aiming for maintainability

    Despite all of these mov­ing parts, I still aim to keep the Mac phi­los­o­phy of sim­plic­i­ty. To change only what I need, I use git to clone Mastodon’s GitHub repos­i­to­ry. Then, I check out the lat­est release tag. Finally, I’ve got this docker-compose.override.yml file with just my mod­i­fi­ca­tions for the var­i­ous con­tain­er services:

    docker-compose.override.yml
    x-mastodon-local: &mastodon-local
      secrets:
        - postgres_password
        - smtp_password
        - redis_password
        - secret_key_base
        - vapid_private
        - ar_enc_primary
        - ar_enc_deterministic
        - ar_enc_salt
      volumes:
        - ./local/scripts/with-secrets.sh:/with-secrets.sh:ro
    
    services:
      # not in upstream
      cloudflared:
        image: cloudflare/cloudflared:latest
        restart: unless-stopped
        secrets:
          - cloudflared_tunnel_token
        environment:
          TUNNEL_TOKEN_FILE: /run/secrets/cloudflared_tunnel_token
        command: tunnel --no-autoupdate run
        depends_on:
          - web
        networks:
          - external_network
    
      # overriding upstream
      db:
        restart: unless-stopped
        secrets:
          - postgres_password
        environment:
          POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
        env_file: .env.db.local
        healthcheck:
          test:
            - CMD-SHELL
            - >
              pg_isready --dbname=$$POSTGRES_DB --username=$$POSTGRES_USER
              && psql -U $$POSTGRES_USER -d $$POSTGRES_DB -c 'SELECT 1' >/dev/null
          interval: 30s
          timeout: 5s
          retries: 5
    
      # overriding upstream
      redis:
        restart: unless-stopped
    
      # overriding upstream
      web:
        <<: *mastodon-local
        command: ["/with-secrets.sh", "bundle", "exec", "puma", "-C", "config/puma.rb"]
        restart: unless-stopped
        ports: [] # doesn't actually override, just merges :-(
        expose:
          - 3000
        depends_on:
          db:
            condition: service_healthy
          redis:
            condition: service_healthy
          es:
            condition: service_healthy
    
      # overriding upstream
      streaming:
        <<: *mastodon-local
        command: ["/with-secrets.sh", "node", "./streaming/index.js"]
        restart: unless-stopped
        networks:
          - internal_network
        ports: [] # doesn't actually override, just merges :-(
        depends_on:
          db:
            condition: service_healthy
          redis:
            condition: service_healthy
    
      # overriding upstream
      sidekiq:
        <<: *mastodon-local
        command: /with-secrets.sh bundle exec sidekiq \
          -q default \
          -q ingress \
          -q mailers \
          -q pull \
          -q push \
          -q scheduler \
          -q search \
          -q indexing_scheduler
        restart: unless-stopped
        healthcheck:
          test: ["CMD", "pgrep", "-f", "sidekiq"]
          interval: 30s
          timeout: 10s
          retries: 3
        depends_on:
          db:
            condition: service_healthy
          redis:
            condition: service_healthy
        networks:
          - internal_network
          - external_network
    
      # commented out upstream
      es:
        image: docker.elastic.co/elasticsearch/elasticsearch:7.17.4
        restart: unless-stopped
        env_file: .env.es.local
        networks:
          - internal_network
        healthcheck:
          test:
            - CMD-SHELL
            - curl --silent --fail localhost:9200/_cluster/health || exit 1
        volumes:
          - ./elasticsearch:/usr/share/elasticsearch/data
        ulimits:
          memlock:
            soft: -1
            hard: -1
          nofile:
            soft: 65536
            hard: 65536
    
    secrets:
      cloudflared_tunnel_token:
        file: ./local/secrets/cloudflared_tunnel_token.txt
      postgres_password:
        file: ./local/secrets/postgres_password.txt
      smtp_password:
        file: ./local/secrets/smtp_password.txt
      redis_password:
        file: ./local/secrets/redis_password.txt
      secret_key_base:
        file: ./local/secrets/mastodon/secret_key_base.txt
      vapid_private:
        file: ./local/secrets/mastodon/vapid_private.txt
      ar_enc_primary:
        file: ./local/secrets/mastodon/activerecord/encryption_primary.txt
      ar_enc_deterministic:
        file: ./local/secrets/mastodon/activerecord/encryption_deterministic.txt
      ar_enc_salt:
        file: ./local/secrets/mastodon/activerecord/encryption_salt.txt

    Full code list­ings are expand­able on the web­site; email read­ers may need to click through to view them.

    Let’s start from the bot­tom secrets section.

    Secrets management

    Rather than Mastodon’s typ­i­cal approach of expos­ing pass­words, tokens, and oth­er sen­si­tive infor­ma­tion as envi­ron­ment vari­ables, I use a more secure approach. I mount each one as a sep­a­rate text file. Then rel­e­vant ser­vices can read these files direct­ly or pass them as vari­ables on startup.

    To make sure I don’t acci­den­tal­ly com­mit pri­vate infor­ma­tion to git, /local/secrets/ is in my repos­i­to­ry’s .git/info/exclude file.

    Shared configuration with a YAML extension

    Back to the top, and I have a x-mastodon-local YAML exten­sion for shared con­fig­u­ra­tion among sev­er­al of Mastodon’s services:

    x-mastodon-local: &mastodon-local
      secrets:
        - postgres_password
        - smtp_password
        - redis_password
        - secret_key_base
        - vapid_private
        - ar_enc_primary
        - ar_enc_deterministic
        - ar_enc_salt
      volumes:
        - ./local/scripts/with-secrets.sh:/with-secrets.sh:ro

    This lists most of the afore­men­tioned secrets, and mounts a wrap­per script to set them as envi­ron­ment variables.

    The wrapper script

    Speaking of which:

    with-secrets.sh
    #!/bin/sh
    
    set -eu
    
    # Map of secret file name to environment variable name
    for secret in \
      "postgres_password:DB_PASS" \
      "smtp_password:SMTP_PASSWORD" \
      "redis_password:REDIS_PASSWORD" \
      "secret_key_base:SECRET_KEY_BASE" \
      "vapid_private:VAPID_PRIVATE_KEY" \
      "ar_enc_primary:ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY" \
      "ar_enc_deterministic:ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY" \
      "ar_enc_salt:ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT"
    do
      # Split the pair into file name and env var name
      # ${var%%:*} -- strip everything from the first ":" onward
      # (keeps left side)
      name="${secret%%:*}"
      # ${var##*:} -- strip everything up to and including the last ":"
      # (keeps right side)
      var="${secret##*:}"
    
      file="/run/secrets/$name"
    
      # If the secret file exists, read its contents into the env var
      if [ -f "$file" ]
      then
        eval "export $var=\"$(cat "$file")\""
      fi
    done
    
    # Hand off to the original command
    exec "$@"

    Full code list­ings are expand­able on the web­site; email read­ers may need to click through to view them.

    Since Mastodon wants every­thing, secrets includ­ed, as envi­ron­ment vari­ables, this script:

    • loops through a map of their cor­re­spond­ing secrets files
    • exports each of their con­tents to the environment
    • and then runs what­ev­er com­mand was passed along for the ride.

    Customized containerized services

    Remember that docker-compose.override.yml gets merged with the upstream docker-compose.yml file, so I only need to write nec­es­sary changes and addi­tions. Let’s take them one at a time.

    Cloudflare Tunnel

      cloudflared:
        image: cloudflare/cloudflared:latest
        restart: unless-stopped
        secrets:
          - cloudflared_tunnel_token
        environment:
          TUNNEL_TOKEN_FILE: /run/secrets/cloudflared_tunnel_token
        command: tunnel --no-autoupdate run
        depends_on:
          - web
        networks:
          - external_network

    This ser­vice is com­plete­ly new, with noth­ing to over­ride. I spec­i­fy every­thing from the Docker Hub-​based image to the start­up com­mand. I also spec­i­fy the exter­nal net­work used to talk to Cloudflare’s servers.

    Of spe­cial note is the TUNNEL_TOKEN_FILE envi­ron­ment vari­able. This fea­ture was added a mere six months ago. It enables load­ing the Cloudflare-​provided authen­ti­ca­tion token direct­ly from my mount­ed file in /run/secrets. This avoids stuff­ing its con­tents into an envi­ron­ment vari­able itself. I also add a depends_on item for the Mastodon web ser­vice. This ensures that Mastodon is run­ning before I open the tunnel.

    PostgreSQL database

      db:
        restart: unless-stopped
        secrets:
          - postgres_password
        environment:
          POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
        env_file: .env.db.local
        healthcheck:
          test:
            - CMD-SHELL
            - >
              pg_isready --dbname=$$POSTGRES_DB --username=$$POSTGRES_USER
              && psql -U $$POSTGRES_USER -d $$POSTGRES_DB -c 'SELECT 1' >/dev/null
          interval: 30s
          timeout: 5s
          retries: 5

    This over­rides sev­er­al set­tings in the upstream docker-compose.yml file, most sig­nif­i­cant­ly in the secrets depart­ment, which fol­lows the same pat­tern as the above cloudflared service.

    And since the upstream git repos­i­to­ry ignores files match­ing .env*.local, I set POSTGRES_DB and POSTGRES_USER envi­ron­ment vari­ables in an .env.db.local file and hard­en the healthcheck.test to both check that PostgreSQL is accept­ing con­nec­tions and that a sim­ple query is suc­cess­ful. That check is run every thir­ty sec­onds and retried up to five times if unsuc­cess­ful. It times out after a mere five sec­onds, more than enough time because every­thing’s run­ning on the same host.

    A brief stop with Redis

      redis:
        restart: unless-stopped

    The only change to upstream’s Redis con­fig­u­ra­tion is restart­ing the ser­vice if it was­n’t man­u­al­ly stopped. (Every oth­er ser­vice in my over­ride file also uses this.) Then I can bring any of them down for main­te­nance or trou­bleshoot­ing, and not wor­ry about docker compose over-​enthusiastically start­ing them up again.

    The Mastodon services

    The Mastodon con­tain­er image itself runs two dif­fer­ent ser­vices, with a third ser­vice next door:

    • web: UI and API
    • sidekiq: back­ground jobs like fed­er­a­tion and media processing
    • streaming: real-​time updates–timelines, notifications–so they arrive instant­ly with­out page reloads
    docker-compose.override.yml (Mastodon ser­vices excerpt)
      web:
        <<: *mastodon-local
        command: ["/with-secrets.sh", "bundle", "exec", "puma", "-C", "config/puma.rb"]
        restart: unless-stopped
        expose:
          - 3000
        depends_on:
          db:
            condition: service_healthy
          redis:
            condition: service_healthy
    
      streaming:
        <<: *mastodon-local
        command: ["/with-secrets.sh", "node", "./streaming/index.js"]
        restart: unless-stopped
        networks:
          - internal_network
        depends_on:
          db:
            condition: service_healthy
          redis:
            condition: service_healthy
    
      sidekiq:
        <<: *mastodon-local
        command: /with-secrets.sh bundle exec sidekiq
        restart: unless-stopped
        healthcheck:
          test: ["CMD", "pgrep", "-f", "sidekiq"]
          interval: 30s
          timeout: 10s
          retries: 3
        depends_on:
          db:
            condition: service_healthy
          redis:
            condition: service_healthy
        networks:
          - internal_network
          - external_network

    Full code list­ings are expand­able on the web­site; email read­ers may need to click through to view them.

    Each of these ser­vices brings in the mastodon-local YAML exten­sion defined ear­li­er. This approach avoids repeat­ing the same secrets and volumes each time. They also repeat upstream’s command entries, but wrapped in the with-secrets.sh script described above.

    Each ser­vice also has a depends_on stan­za that relies on the PostgreSQL and Redis ser­vice report­ing good health. The rest is just net­work­ing tweaks. lim­it­ing exter­nal expo­sure to the web ser­vice only.

    Again, the point here is main­tain­abil­i­ty and only over­rid­ing what’s not already cov­ered by upstream’s docker-compose.yml file. 

    Bringing it all together

    The con­tain­ers are hum­ming along. Secrets are tucked safe­ly away. Cloudflare qui­et­ly han­dles the out­side world. My lit­tle Mac mini now runs a fully-​fledged Mastodon instance, still with­out break­ing a sweat. It’s not just a proof-of-concept–it’s my dai­ly dri­ver for post­ing, fol­low­ing, and explor­ing the Fediverse.

    Performance has been pleas­ant­ly unevent­ful: CPU and mem­o­ry usage stay low, even dur­ing busy fed­er­at­ed time­lines. The tun­nel has been rock-​solid, and the override-​only approach means I can pull upstream updates with­out dread­ing a merge marathon.

    Lessons learned

    • Secrets-​as-​files keep sen­si­tive data out of the envi­ron­ment and out of ver­sion control–worth the extra setup.
    • Docker over­ride files are a sanity-​saver; upstream changes flow in with­out tram­pling my tweaks.
    • Health checks aren’t just for show–they’ve already caught a mis­be­hav­ing Sidekiq ser­vice before it caused downtime.
    • Cloudflare Tunnel removes the need for a sta­t­ic IP and keeps the host off the pub­lic Internet entirely.

    If you’re thinking of trying this

    Start small. You don’t need a rack of servers–a mod­est machine and a bit of con­tain­er dis­ci­pline can get you sur­pris­ing­ly far. Keep your changes min­i­mal, doc­u­ment them as you go, and let upstream do the heavy lifting.


    Running Mastodon this way has been a reminder that self-​hosting does­n’t have to mean end­less tin­ker­ing. With the right boundaries–both in net­work expo­sure and in con­fig­u­ra­tion scope–it can be calm and pre­dictable. It offers the sat­is­fac­tion of know­ing it’s entire­ly yours. 

    I’ve since enabled ElasticSearch for full-​text search by copy­ing over upstream’s commented-​out es exam­ple ser­vice to my docker-compose.override.yml and light­ly con­fig­ur­ing it. It was a fair­ly sim­ple addi­tion that has­n’t affect­ed per­for­mance on the Mac mini.

    My next steps will involve light­weight mon­i­tor­ing and back­up strate­gies. These steps will guar­an­tee this lit­tle serv­er can keep qui­et­ly doing its job for years to come.

  • Tony Levin and Stick Men

    Tony Levin is far and away my favorite musi­cian. Even before I picked up the bass gui­tar, I kept find­ing his name in the lin­er notes of my most-​liked albums. I’ve seen him play with Peter Gabriel, King Crimson, Stick Men, and with his broth­er Pete play­ing in their Levin Brothers jazz combo.

    And of course, once I did start study­ing his bass (and Chapman Stick) lines, they were a rev­e­la­tion. Endlessly cre­ative, both dri­ving and being dri­ven by the song, only showy when the moment called for it, flu­id, some­times fierce, always the per­fect mix­ture of tech­nique and emotion.

    I’m due to see Stick Men when they swing down to Houston in two months. Until then, here’s their lat­est EP:

  • WordPress, ActivityPub, and Friends

    I’ve also been mess­ing with the Friends and ActivityPub plu­g­ins for WordPress on my blog, and I share Shelley’s con­cerns about the for­mer bloat­ing the data­base with feed items. You can con­trol this some­what by set­ting reten­tion val­ues in days or a num­ber of posts, but you have to go into each friend’s Feeds tab and do it manually–there’s no default setting.

    After read­ing that post, I’m also con­sid­er­ing dis­abling Friends in favor of a feed read­er, espe­cial­ly because (as Shelley also not­ed) there are gaps when with favorites and com­ment con­ver­sa­tions bridg­ing between WordPress and Mastodon servers. Like her, I’m not keen on installing a single-​user Mastodon instance or oth­er fedi­verse serv­er that requires man­ag­ing an unfa­mil­iar pro­gram­ming language.

    I’m also try­ing to do this in tan­dem with a suite of IndieWeb plu­g­ins, and I’m run­ning into an issue with my friends feed page not show­ing any posts when the Post Kinds plu­g­in is acti­vat­ed. I real­ly want to keep this plu­g­in because it lets me inter­act bet­ter with oth­er IndieWeb sites as well as the Bridgy POSSE/​back­feed ser­vice con­nect­ing me to oth­er social networks.

    My ide­al is a per­son­al web­site where I write every­thing, includ­ing long-​form arti­cles, short sta­tus­es, and replies like these. Folks can then find me via a sin­gle iden­ti­fi­able address and then subscribe/​follow the entire fire­hose of con­tent or choose sub­sets accord­ing to post types, top­ics, or tags. They’d then be able to reply or react on my site or their favored plat­form, which my site would col­lect regard­less of ori­gin, with sub­se­quent replies and reac­tions get­ting pushed out to them. Oh, and it should work with both ActivityPub clients and servers, IndieWeb sites, and syndicate/​backfeed to oth­er social net­works either with or akin to the Bridgy ser­vice I men­tioned above.

    So far I haven’t seen any­thing that ticks all these box­es, and I’m get­ting itchy to write my own. Perl is my favorite pro­gram­ming lan­guage, so I’m look­ing at the Yancy CMS as a base. But I know that it would still be a hell of a project, and one of the rea­sons I chose WordPress for blog­ging was that it was well-​established and ‑sup­port­ed but still eas­i­ly exten­si­ble so that I could con­cen­trate on writ­ing instead of end­less­ly tweak­ing the engine. Unfortunately, I’m start­ing to fall into that trap anyway.

  • Video: A Year of Being Wrong on the Internet”

    I’m busy this week host­ing my par­ents’ first vis­it to Houston, but I didn’t want to let this Tuesday go by with­out link­ing to my talk from last week’s Ephemeral Miniconf. Thanks so much to Thibault Duponchelle for orga­niz­ing such a ter­rif­ic event, to all the oth­er speak­ers for com­ing togeth­er to present, and to every­one who attend­ed for wel­com­ing me.

  • This week: The ePhEmeRaL miniconf

    This week: The ePhEmeRaL miniconf

    After a lot of pro­cras­ti­na­tion, I’ve decid­ed my talk for this week’s ePhEmeRaL mini­conf will be Cunningham’s Law: A Year of Being Wrong on the Internet, or «prêch­er le faux pour savoir le vrai.»

    The event starts at 8:00 AM CST on Thursday, November 18; you can find out more about it includ­ing the full sched­ule and a time zone con­vert­er here.