A Broken Image, and What Was Under It
At about eleven in the morning a client messaged me: the photos on one page of her website were not loading. That is a twenty minute job. Copy the missing files, check them in a browser, reply.
I finished at eleven at night, and by then the same server had produced a document root that had been switched two months earlier and abandoned half done, an admin panel dead for seven weeks that nobody had reported, a public database login form sitting in the web root, 2.5 GB of RAM fenced off and unusable, four client sites serving browser security warnings, thirteen accounts with no backup of any kind, a firewall quietly eating every incoming email for three domains, and one account running a 500-thread SSH brute force against the rest of the internet.
Nobody was hiding any of it. Nobody had looked.
The fixes are the boring part. The shape of the day is not. Every finding was one question away from a finding I already had, and the only reason the chain kept going was that each answer got checked instead of accepted.
The chain
The images returned 404, which normally means somebody deleted them. But the CMS still had database records for every one. Files a CMS wrote and then lost are rare. Files written to one directory tree while a different tree is being served are not. There were two copies of the site on disk, and the web server had been pointed at the other one.
When did that happen? The obvious answer is the config file's modification time, and the obvious answer is wrong. A config file's mtime tells you when somebody last edited a file. It says nothing about when the change took effect, and after any routine template regeneration it lies outright. The honest clock is something the running software writes only as a consequence of serving real traffic. This site had an image cache. I built a histogram of file times per tree, and the histogram for one of them simply stops on 24 July. That dated the switch to the hour, from evidence nobody could have backdated.
While diffing an unrelated template I found the site's chat button pointing at a placeholder phone number. Seven weeks, nobody reported it, because a chat button that opens the wrong number still looks like it works.
That was the morning: 174 images back, the admin panel alive again after seven weeks, a public database login form removed, 533 MB of debug logs deleted. Then I asked to clean up five long-dead accounts, and said I would back them up first.
The number that did not add up
One account was 3.9 GB on disk and 405 MB in its backup.
Compressed archives are smaller than their contents, and eight files of plausible size looks like a finished job. But 3.5 GB is too much to wave at, so I went looking for it.
It was in the account's system mailbox: 876,215 undelivered cron failure emails. A crontab line, planted years earlier, pointing at a file that never existed, mailing an error every minute into a mailbox nobody has ever opened. Two years of it.
The crontab had a second line. That one pointed at a directory under /var/tmp and ran every minute. Inside it: a launcher that masked its own process name as crond, a binary called iptables that was not iptables, and a working directory named .ICE-unix, which is a real directory name the X11 display system uses, chosen precisely so it does not look out of place in a listing.
The payload was an SSH brute forcer. A shuffled 174 MB list of target addresses, a password list, and a launch line running it at 500 threads. This was not a quiet crypto miner. This was a server I had just been handed, attacking other people's servers, with its own IP address on every packet.
Two file timestamps matched two successful SSH logins to the exact minute, which is what turned a suspicion into a timeline. And the 876,215 emails, it turned out, were not the attacker's. They came from the older, separate compromise. The newer attacker had replaced that broken line with a working one, which is why the noise stopped in April and the silence looked like an improvement.
The firewall nobody mentioned
Later that evening I went to enable the firewall, and checked what was running first out of habit. A firewall was already active. Nobody had mentioned it, it was not the one the control panel manages, and it was allowing exactly four ports.
Three client domains had mail records pointing at that box. All three had been receiving nothing.
Testing this from my laptop proved nothing: my own connection to port 25 failed, and so did the control test to Google, because home internet providers block outbound mail. The answer was in the firewall's own log on the server: 164 dropped packets on port 25, the most recent six minutes earlier, from a real mail server trying to deliver something real.
I replaced the firewall at 21:38. At 21:45:35, fifteen minutes later, a genuine message arrived from a foreign university's mail system for one of the clients, confirming a meeting. Three more came in the first half hour. Before that night, all of it was refused at the door with no bounce, so the senders thought they had delivered and the recipient thought nobody had written.
The thing I nearly shipped
I am not going to pretend the day was clean.
The backup routine I wrote came with a companion command to delete old backups. It used find with -maxdepth 1 and no -mindepth 1. That pattern matches the parent directory itself, so the delete would have removed every backup on the box, and because of how the age filter works it would only have fired once the backup job had already stopped producing new ones. I caught it because the dry run printed nothing, and "nothing" is not an answer I trust any more. That empty output looked like proof the command was safe. It was the opposite.
That trap showed up four times in one day with four different tools. grep silent on a file containing binary chunks until I added -a. nc reporting a port closed on a server my terminal was connected to at that moment. An archive listing as empty because the panel had switched to a compression format my command did not speak. And the find above. Four confident nothings, four wrong answers.
The rule that catches all of them: before reporting that you found nothing, run the same query against a case you know is there.
What I deliberately did not do
The plan had a step to raise the database buffer pool. I measured first: 109 million reads served from memory against 3,032 that went to disk, a 99.997% hit rate, with more than half the pool still unused. Raising it would have taken memory away from seven clients to buy nothing. Step cancelled.
The same restraint applied to the code cache. Instead of the flat allocation I had written down, I counted which PHP versions actually serve traffic on that box: 21 sites on one, one site on another, zero on the remaining two. That turned 512 MB of reservation into 256.
And the honest ending on speed. After all of it, the client's site takes 0.83 seconds of server time per page. The same server returns a static file in 13 milliseconds and another client's PHP page in 50. The remaining 800 milliseconds are that site's own code, which runs 43 database queries to draw one page. There is nothing left to fix on the machine. I told her that in those words, because the alternative is selling server work that cannot help.
Where it had been talking
The finding I keep thinking about is the smallest one on the list.
The server's administrator notification address was set to a hostname that does not resolve anywhere, with no mailbox behind it. Every alert that machine had ever raised, about certificates, about backups, about anything, went into a hole.
That is the whole mechanism. The cron daemon dutifully mailed 876,215 warnings. The panel dutifully sent certificate failures to the configured address. The firewall dutifully logged every dropped delivery. Not one of those systems was broken. They were all working exactly as designed, reporting to a destination nobody read.
Neglect is rarely invisible. It is usually written down somewhere, by a system doing its job, in a place with no reader. And the fix is one field in one form, which is the cheapest thing on the entire list.
So before optimising a server, find out where it has been talking, and go read it. It has been trying to tell you for two years.
Same finding as the scorecard I built for shop owners and as running AI like a team: the work is never the hard part. Knowing what done means, and who is watching, is.
Nguyễn Hải Nam
Nguyễn Hải Nam
Project Management Lead. 16+ years from code to delivery. PMP®. Writing here about project management and engineering.