Late in July, the 7th month of the year, an internal business dashboard I built with help from Claude Code suddenly became unreachable.
The system pulls business data on a schedule, turns it into charts, and lets colleagues view them in a browser through an encrypted connection. I should be clear that I cannot code independently. Claude Code handled most of the code and tests. I defined the requirements, decided which numbers were correct, and judged whether each change was ready to go live.
After the network recovers, queued work and new requests may return to the system at the same time.
The system went into use in mid-July and had been fairly stable. Small problems occasionally appeared, but it usually recovered quickly. So when the network went down this time, I assumed I only had to wait for the connection to return.
At first, the page would not open. When the network came back, I thought the incident was over. But several browser tabs refreshed at once. Before the system could prepare the data again, it received a batch of duplicate calculation requests. Each page also retried every 15 seconds. The service became slower and slower until it was unreachable again.
The network had recovered, but the system had not.
That was when I learned that an obvious code error is sometimes easier to judge. The more dangerous moment is when Claude Code replies, “Fixed,” and the page briefly opens again. I am tempted to believe the work is finished.
When I asked it to add retries, it did. When I asked it to save results temporarily, it did that too. What I failed to ask was what would happen when all of those mechanisms started at the same time.
The full investigation was long, so I will not reproduce the entire timeline. These are the five questions I added after the incident.
1. Test Recovery, Not Just Failure
I used to think a system had only two states: working or broken.
I would test whether a page returned the right result, whether a network failure produced a warning, and whether the program could restart after stopping. Once those checks passed, I felt much safer.
Then I realized there is another state between normal operation and failure: recovery.
When the network returns, queued jobs, page refreshes, and data recalculation may all start together.
As soon as the network returned, unfinished jobs resumed, old pages refreshed, and scheduled jobs kept running. The temporary results prepared earlier had also expired. Several waves of work collided at once.
Afterward, I cleared all prepared results, opened several pages at the same time, disconnected and restored the network, and watched specifically for slowdown during recovery. I will keep this step whenever I test a similar service.
2. Automatic Retries Can Make a Busy System Busier
Previously, a failed page request retried the entire dataset every 15 seconds.
At the time, automatic recovery felt reassuring. Letting the page try again seemed better than forcing users to keep pressing refresh.
But every retry sent another batch of requests. Once the system was already overloaded, asking again every 15 seconds only kept it overloaded.
When the system cannot keep up, fixed-interval retries bring new requests back in circles.
We lengthened the interval between retries and slightly staggered different pages. A tab can now run only one refresh cycle at a time, and it pauses when moved to the background. When the system is busy, it also stops accepting every expensive job.
Now, whenever I add automatic retries, I ask one more question: if the system is already overloaded, will this keep sending it more work? If the answer is yes, the retry count gets a limit and eventually stops.
3. Split Jobs May Still Share the Same Weak Point
The system had already divided its data work into smaller jobs. Historical data was processed day by day, and large date ranges were split into smaller ones. Each individual job was genuinely lighter.
I naturally assumed that because the jobs were separated, one failed job would not affect the others.
But all of those smaller jobs still shared the same network, login, storage location, and access route. A failure in any shared layer could still affect everything else.
The jobs were smaller, but they still shared the same network and access route.
We changed several things. If one data source fails, later sources continue processing. Tailscale became the primary access route, while a different route remained available as a backup. The program is also no longer allowed to report success merely because a job ended. It must confirm that the new data was actually stored before the page shows success.
Now, when someone says a job has been split up, I also check whether the network, login, and storage are still shared. If those layers remain tied together, one failure can still affect the other jobs.
4. Fast During Normal Use Does Not Mean Fast During Recovery
When the data had already been prepared, the page returned in a few milliseconds. I used to see that result and conclude that the speed problem was solved.
What I had not tested was what happened when no prepared result existed and several people opened the page together. During the incident, more than ten tabs entered the most expensive trend page at once. The same data was queried and processed more than ten times.
We changed one rule. For the same result, only one request performs the calculation; the others wait for that answer. When 16 pages request the same result at once, the calculation now runs only once.
Several pages can wait for the same calculation instead of repeating it from the beginning.
Now, when I test speed, I first remove the prepared results, open many pages together, and restart the program once. If I only test the easiest condition, where every result is already available, the system looks much faster than it is in real use.
5. Tests Can Only Check the Questions We Put Into Them
The dashboard did have tests before the incident.
Key numbers were compared against a known correct result, and every change to the calculation rules was checked item by item. The page code and configuration files were also checked automatically. All of those tests were green.
The original tests mainly asked whether the numbers were correct and whether the code could run. We had never written tests for several people opening the same page at once, or for work that continued in the background after a browser had already stopped waiting.
Those scenarios were never added, and I never asked about them. The green tests were not wrong. Their scope was simply too narrow.
We added those situations to the release checks: several people opening pages together, one data source failing, simple pages remaining available, and the backup route taking over. The interface also distinguishes stale data, connection failure, timeout, and system overload instead of showing the same generic “data refresh failed” message for everything.
After the changes, the trend page returned in about 1.5 seconds. When 16 pages requested the same result, it was calculated only once. Simple status pages remained available while expensive calculations were running. We also deliberately disconnected the primary route and confirmed that the backup route could continue serving the system.
Recovery and concurrent use were not written into the tests, so the tests could not discover them.
The system still has clear limits.
It still runs on one machine, and this lightweight service cannot absorb endlessly growing traffic. If usage keeps increasing, the single-machine deployment will need further changes.
The incident did not convince me that Claude Code was unreliable. Without it, I probably could not have built this system at all. But I cannot wait for it to volunteer every scenario that needs to be tested.
The next time Claude Code replies, “Fixed,” I will not immediately close the task. I will disconnect the network once more, open several pages at the same time, and check whether the data actually updated.