The Deploy Key That Could Only Do One Thing

Part of the same build as The Day My AI Agent Refused to Touch Production.

The workflow file was fine. actions/checkout, webfactory/ssh-agent, a plain rsync, done in twenty lines. What took the rest of the afternoon was the one thing the YAML doesn't show: a Linux account on a shared server that a GitHub Actions job can use, unattended, to touch exactly one directory and nothing else.

Three bugs later, it worked. None of them were where I expected.

The goal, and the one true rule

A CI deploy key should be able to do its job and nothing more. If GitHub Secrets ever leaked that key, the blast radius should be one app directory on one server, not the box. The standard tool for this is rrsync, a wrapper script that ships with rsync itself: put command="rrsync /opt/onduty/app" in front of the key's line in authorized_keys, and no matter what the client asks to run, only rsync into that one folder is possible.

That one rule, "no matter what the client asks to run," is also exactly what broke steps two and three later. Forced commands are absolute. They don't negotiate.

Bug one: the shell that eats every command, including the safe one

First attempt, testing that the key was properly restricted:

ssh -i ./onduty-ci-deploy onduty@your-server-ip "cat /etc/onduty.env"

This is supposed to fail, since the forced command should run rrsync instead of cat. It failed, but with the wrong message:

This account is currently not available.

That's not rrsync talking. That's /usr/sbin/nologin, the shell I'd deliberately set on the service account so nobody could log in as it interactively. Sensible instinct, wrong mechanism: when a client connects with a forced command, sshd doesn't skip the shell, it execs the account's shell with -c '<forced command>'. A shell whose entire job is to refuse everything refuses that too, forced command included.

The fix undoes the thing that looked like the security measure:

usermod -s /bin/bash onduty

This does not reopen the account. The forced command in authorized_keys still overrides every single thing the client asks for, on every connection, regardless of the shell. nologin was never adding protection here; it was silently breaking the protection that was already in place.

Bug two: a path that doubles itself

With a real shell in place, the same test now failed correctly:

/usr/bin/rrsync error: SSH_ORIGINAL_COMMAND does not run rsync

Good. Now the actual transfer:

rsync -az --delete ./ onduty@your-server-ip:/opt/onduty/app/

rsync: [Receiver] mkdir "/opt/onduty/app/opt/onduty/app" failed: No such file or directory (2)

rrsync had already chdir'd into /opt/onduty/app, and that's the entire point of naming it in the forced command. Handing it the full absolute path again on the client side means it gets treated as relative to where rrsync already stands, and the two copies concatenate. The fix is to stop repeating information the server side already has:

rsync -az --delete ./ onduty@your-server-ip:/

The / here isn't the server's real root. It's the root of whatever rrsync was told to sandbox into. A --dry-run test would not have caught this, either, since dry-run never calls mkdir, so the bug is invisible right up until the first real transfer.

Bug three: one key, one command, forever

Transfer working, next step: after the files land, SSH back in to build and restart.

ssh onduty@your-server-ip "cd /opt/onduty/app && npm ci && npm run build"

Same error as bug one's fixed version: does not run rsync. Not a leftover bug, but a design flaw. command="rrsync ..." doesn't inspect what the client sent before deciding whether to honor it. It runs rrsync, always, for every session that key opens. There is no way to make one forced-command key do two different fixed jobs.

The fix is a dispatcher, a few lines of shell in place of the raw rrsync call:

#!/bin/bash
set -e
case "$SSH_ORIGINAL_COMMAND" in
  rsync\ --server*)
    exec /usr/bin/rrsync /opt/onduty/app
    ;;
  build-and-restart)
    cd /opt/onduty/app
    rm -rf .next
    npm ci
    npm run build
    sudo systemctl restart onduty.service
    ;;
  *)
    echo "onduty-ci-deploy key: command not permitted" >&2
    exit 1
    ;;
esac

authorized_keys now points at the script instead of rrsync directly. Real rsync traffic still gets routed straight into the sandbox; one specific keyword triggers the fixed build-and-restart sequence; everything else is refused by default, same as before. One key, still fully restricted, now capable of exactly the two jobs it needs and nothing else.

The bonus bug, if you're testing from a Mac

While chasing bug two, I ran the exact same rsync command from my own machine and got a completely different, more confusing error: invalid rsync-command syntax or options. macOS hasn't shipped real rsync since Apple stopped bundling GPLv3 software; the binary at /usr/bin/rsync is openrsync, a BSD-licensed reimplementation stuck on protocol 29. It talks a different enough dialect that the server's modern rrsync script can't parse what it sends. brew install rsync and pointing at that binary instead fixed it in one line. If a restricted-key setup fails only when tested from a Mac and works from CI, this is almost certainly why.

The second product's plan already knew, and it still wasn't enough

A second product on the same server needed the identical setup. Its plan got rewritten first with all three fixes above baked in from day one: dispatcher script instead of raw rrsync, / instead of the doubled path, the shell fix called out before anyone reaches for nologin on instinct. None of those three came back on the real build. Two new ones did, specific to that product's own shape, plus one real outage this pipeline had no way to teach me about. That part is its own story: Reusing a Working CI/CD Pipeline Found Three New Bugs.

What actually mattered

Every one of these bugs was invisible in the YAML, invisible in the plan, and invisible in a dry run. They only showed up when the exact commands actually ran against the exact account, in the exact order a real deploy would use. Restricting an SSH key is not a five-minute command= line; it's that line plus a shell that can execute it, a path the client stops repeating, and a dispatcher if the job is more than one shape. Test the thing you're actually restricting, for real, before trusting it with a production box.

Nguyễn Hải Nam

Nguyễn Hải Nam

Project Management Lead. 16+ years from code to delivery. PMP®. Writing here about project management and engineering.

About me