The Bug That Couldn't Exist Until Production Existed

The site had been built on staging for months. Real theme, real pages, a real product catalog, all of it working, all of it reviewed. The actual production domain had never run any of it. Tonight it finally did, and three separate things broke that staging had no way to warn me about, because none of them were bugs in the content at all.

Bug one: a database copy that was correct, byte for byte, until it wasn't

Bringing production's design settings up to date with staging's meant copying one WordPress option, a large serialized PHP array holding the theme's header layout, icon styles, product grid config. The pattern that had worked all night for small values was the same pipe:

ssh staging "wp db query \"SELECT option_value FROM wp_options WHERE option_name='theme_mods_x'\" --skip-column-names" | ssh production "wp option update theme_mods_x"

It ran clean. No error, no warning. The site's header immediately reverted to the theme's stock demo layout, generic logo, placeholder text, nothing of the real design left.

The value itself hadn't been dropped, it had been altered. mysql's CLI escapes special characters in its default tab-separated output. A short string survives that untouched. A large, deeply nested serialized array is much more likely to contain a byte sequence the escaping rewrites, and PHP's serialization format is unforgiving about it: every string inside is prefixed with its exact byte length, s:42:"...". Change even one byte inside that string and the length no longer matches, unserialize() gives up on the whole structure, and WordPress silently treats the option as if it had never been set.

Small values had made it through the same pipe all night by luck, not because the technique was sound. The fix was to stop moving serialized data as text at all: restore the last known-good value from backup, then apply only the actual field-level differences with a same-process PHP script, get_option(), change the one key that's actually different, update_option(). No text ever crosses between two hosts through a shell pipe.

$mods = get_option('theme_mods_x');
$mods['header_elements_right'] = ['cart', 'account', 'search'];
update_option('theme_mods_x', $mods);

A full JSON diff of both sides, taken once, found eighteen real differences in that array. Every one of them got patched this way in a single pass, instead of chasing one visual complaint at a time.

Bug two: production forgot which page it was

Separately, WordPress stores which specific page ID is "the homepage" as one number in an option, and which post ID each menu location points to as another. Both of those numbers meant something on production's own database. They stopped meaning anything the moment production's content tables were replaced with staging's, because staging's homepage lived at a completely different ID.

The result was immediate and total: the homepage 404'd, and the navigation menu pointed at nothing.

$WP option get page_on_front
378

Page 378 no longer existed in the new content. Staging's actual homepage did, just under a different number:

$WP option update page_on_front 2092

One value, repointed, and the front page came back. The menu needed the same treatment, one command reassigning the site's nav locations to the menu that had actually come across with the new content. Neither of these was a content bug or a code bug. They were pointers into a world that had just been replaced out from under them, and the fix list for a deploy like this has to include repointing them as a known step, not a surprise discovered from a 404.

Bug three: a bug that could not exist anywhere except production

The theme carried a small, deliberate patch, written months earlier, for a real recurring problem: a specific image URL kept getting rewritten with the wrong domain by the media-offload plugin, and a filter caught and corrected it on the way out.

$broken = '#https?://(?:dev\.example\.com|assets\.example\.com)/wp-content/uploads/2026/08/(?!path-segment/)logo\.webp(?:\?[^"\'>\s]*)?#i';

Two domains, both real, both seen breaking during months of staging work. The footer logo on production came up broken, a small image icon where the brand mark should be. The URL it was trying to load:

https://example.com/wp-content/uploads/2026/08/logo.webp

Not dev.example.com. Not assets.example.com. The site's own bare production domain, wrapped around the same malformed path. The regex had two branches, and neither matched, because this exact string, the plain production hostname combined with that broken path, is not a string that can occur anywhere except on production. Staging's domain is never production's domain. There is no test on staging, no matter how thorough, that generates the one input this code needed to handle.

$broken = '#https?://(?:dev\.example\.com|assets\.example\.com|example\.com)/wp-content/uploads/2026/08/(?!path-segment/)logo\.webp(?:\?[^"\'>\s]*)?#i';

One alternative added to the pattern, on both environments, and the class of bug is closed for good, not just today's instance of it.

What actually mattered

None of these three had anything to do with whether the site's content or design was correct, and staging had already proven both of those, thoroughly, for months. What staging cannot prove is what happens the first time code runs against production's own identity: its old page IDs, its own domain string, the exact byte-level path a value takes moving between two real hosts instead of one. Some categories of bug only exist at the seam where a system meets the specific place it's about to run, and the only way to find them is to actually make the move, once, for real, and watch closely for the first few minutes.

Nguyễn Hải Nam

Nguyễn Hải Nam

Project Management Lead. 16+ years from code to delivery. PMP®. Writing here about project management and engineering.

About me