Reusing a Working CI/CD Pipeline Found Three New Bugs
Part of the same build as The Day My AI Agent Refused to Touch Production and The Deploy Key That Could Only Do One Thing.
Two days after the first CI/CD pipeline went live on OnDuty, I asked for the same thing on Nanakun, a second product on the same shared server. Same shape: push to main, tests gate the deploy, a scoped key does the SSH work, no human types a deploy command by hand again. I expected an afternoon of copy-paste. I got a production outage instead, on the very first fully automatic run, from a bug the first pipeline never had a reason to teach me.
That's the actual lesson of doing this twice: a proven pattern removes the risk you already paid for and says nothing about the risk that's specific to the second thing.
What copied cleanly
The dispatcher-script trick from OnDuty (one forced SSH key, a case statement routing real rsync traffic to rrsync and one fixed keyword to a build-and-restart sequence) worked on the first try. So did the --old-args flag rsync needs when it's new enough to talk a protocol the server's rrsync script wasn't written to parse, and so did remembering that macOS ships a fake rsync (openrsync, stuck on protocol 29) so testing from a Mac needs the Homebrew one, at its full path, or the exact same "invalid rsync-command syntax" error shows up again for a completely different reason than it did the first time. Three real bugs, fully paid off, zero repeats.
What didn't
The home directory was half a fix. Setting up the second service account, I created its .ssh folder and chowned that. I didn't chown the home directory it lived in. npm ci tried to create its cache under that home dir on the first real deploy and got EACCES, then cascaded into forty lines of confusing TAR_ENTRY_ERROR and ENOTEMPTY warnings that looked like a corrupted install. It wasn't. It was one directory, one level up, that I'd never touched.
A build needs the database the app needs. Nanakun's root layout reads shop settings from Postgres, including on the 404 page, so next build fails outright without DATABASE_URL — not at runtime, at build time, during static generation. OnDuty's test suite never touches a database directly, so this exact failure mode had never existed to learn from. The fix was one line in the CI job pointing the build step at the same test Postgres service the test step already migrates. Obvious once you see the stack trace; invisible until then, because "the build doesn't need a database" was true of the last project and false of this one.
A brand-new gate finds old debt instantly. The moment npm run format:check existed as a CI step, it failed on fifty-two files that had drifted out of Prettier's formatting over months of nobody enforcing it. Not a regression, just the first time anything was checking. One prettier --write . and a clean-baseline commit fixed it permanently, but it's worth naming as its own category of "new bug": a gate that has never run yet has debt hiding behind it, and the debt surfaces as a red pipeline on day one, not on the change that actually caused any of it.
The one that took the site down
The real incident: rsync --delete on the automated deploy synced the built app to the server and removed everything on the server side that wasn't in the git source, which is exactly what --delete is for. Except one directory on the server, .media, holds uploaded files that only ever exist on the server. It was never in git, so --delete removed it as "extra," and it took the site down for real.
Not gracefully, either. The app's systemd unit declares ReadWritePaths=/opt/nanakun/app/.media as part of its sandboxing, and systemd sets that up as a bind mount before the process even starts. Delete the target of a bind mount and the service doesn't error, it fails to launch, in a loop, with 226/NAMESPACE and a message that names the missing path exactly but reads like a kernel-level problem to anyone not already looking for it. From the outside it was just nanakun.vn returning 503.
Recovery was one mkdir, one chown, one restart, and the site was back inside two minutes. Then the actual fix: add .media to the rsync exclude list, same as .env and node_modules already were, on both the automated workflow and the manual fallback script, so the next deploy can't repeat it. The uncomfortable part isn't the outage, it's that the same --delete flag had been sitting in the manual deploy script from day one, unnoticed, because every manual run before this had gotten lucky on timing or scope.
What this is actually worth
Before this week, deploying Nanakun meant SSHing in by hand, running rsync, npm ci, a build, and a restart in the right order, from memory, hoping nothing had drifted since the last time. Now a push to main runs the full test gate, and only a green gate reaches the server, without anyone typing an SSH command. That's the headline change, and it's real: the failure class of "someone forgot a step" is gone.
But the honest accounting includes the outage this same pipeline caused on its first real run, found and fixed inside a few minutes because the whole thing is now observable in one place — GitHub Actions logs, systemctl status, journalctl, one curl — instead of scattered across a terminal history nobody kept. The pipeline didn't prevent the bug. It made the bug loud, fast, and fixable in minutes instead of silent until a customer noticed. For a product with no live customers yet, that's the actual trade worth naming: this is the cheapest possible moment to find out --delete doesn't know which directories are sacred. The second pipeline, built from a working first one, still found three bugs and one outage that the first pipeline had no way to teach it — because they weren't bugs in the pattern. They were facts about this particular app that no amount of reuse was going to surface until it ran for real.
Nguyễn Hải Nam
Nguyễn Hải Nam
Project Management Lead. 16+ years from code to delivery. PMP®. Writing here about project management and engineering.