Handling Problems on a Live Project Without Making Them Worse
There is a specific silence that falls on a chat channel when production breaks in a way nobody recognizes. I have lived that silence with orders stuck between systems during business hours, real customers affected, and three teams all looking at their own dashboards showing green. What I learned from those hours is that problem handling has a correct order of operations, and the instinct to skip straight to "find the fix" is the most common way to make a bad hour worse.
First, contain; understanding can wait
The first question is never "why is this happening." It is "how do we stop the bleeding while we find out." Route orders through the manual fallback. Disable the misbehaving feature. Roll back the release. Containment buys the thing debugging needs most, which is time without new damage accumulating. Teams skip this because containment feels like defeat; it is actually what lets the real fix be done carefully instead of at panic speed. Every rollback plan I have ever written and never used was still worth writing, because the one time in ten it runs, it converts a crisis into an inconvenience.
Define the problem in one falsifiable sentence
Panic produces vague problems: "sync is broken." Vague problems cannot be divided, assigned, or verified fixed. One sentence with specifics changes everything: "orders placed after 10:40 on channel X reach the aggregator but not the CMS." Now three teams can each check their segment of that sentence, and you will know, provably, when it stops being true. Half of firefighting chaos is people solving different guesses at what the problem is.
Analyze at the boundary, and name causes, not people
Cross-system problems live at the boundaries, in queues, retries, timeouts, format changes, the territory that belongs to everyone and therefore to no one. The blame instinct actively fights the analysis: I wrote in the conflict post about two teams sliding log files at each other, each proving their side was fine, while the defect sat in a retry queue between them. The moment the review was declared blameless, the same people found it in forty minutes. Root cause analysis is a technical exercise that only works under social conditions.
Choose the fix by reversibility, not elegance
Under pressure, prefer the fix you can undo. The elegant redesign can wait for daylight; the immediate fix should be small, testable, and reversible, because your understanding of the problem is at its worst exactly when the urgency is highest. I have approved ugly patches at midnight and scheduled their proper replacement for the next sprint, and I consider both decisions correct. What I no longer approve at midnight is anything irreversible built on a two-hour-old theory.
Communicate on a clock, not on progress
During a live problem, stakeholders without information invent it, and escalation fills any silence. The discipline is updates on a fixed cadence, every thirty or sixty minutes, even when the update is "still investigating, next update at 3:00." A steady drumbeat of honest nothing keeps the space calm in a way that one triumphant update after four silent hours never does. Who gets told what, and who speaks to the customer side, is decided once, in the first ten minutes, not renegotiated per update.
Then fix the class, not just the instance
When it is over, the cheap ending is relief. The valuable ending is corrective and preventive action: this defect becomes a test that runs nightly, this blind spot becomes an alert, this confusion becomes a runbook line. My standing rule is that every incident must leave behind at least one artifact that makes its recurrence impossible or loudly visible. Problems handled this way get more expensive to have but progressively harder to repeat, and after enough cycles the silence on the chat channel stops meaning fear and starts meaning nobody has needed the channel in a while.
Nguyễn Hải Nam
Project Management Lead. 16+ years from code to delivery. PMP®. Writing here about project management and engineering.