The Scene of the Incident
It was a leads sync pipeline I built for a business: on a schedule, pull leads from an ad platform, decrypt them, join them with ad spend data, and write everything into a Feishu (Lark) multi-dimensional table. The day it went live, I watched it run through three cycles, and the data showed up in the table, neat and orderly. I called it delivered, thinking to myself, "what could possibly go wrong with a cron job."
Three weeks later, ops asked me: were we missing leads on a few specific days?
I went back and checked the logs. The scheduled task had failed 22 times. Nobody knew — including me. The server hadn't crashed, the process was still alive, cron was firing on schedule as always — it's just that inside the task, a token had expired, and every single run was erroring out and exiting at the very first step. It failed quietly, exactly the way it had succeeded quietly.
A Silent Failure and a Silent Success Look Identical From the Outside
This is the part that stings the most. From the outside, everything checked out:
- Server running ✓
- Process alive ✓
- cron log shows it fired ✓
- No alerts ✓
Four green lights, and yet the task had been dead for three weeks.
I've since pinned this lesson to my wall: "looks like it's running" does not mean "is actually working." If an automated system doesn't proactively report to you, its silence carries zero information — you can't treat "no news" as "good news," because failure produces no news either.
The Fix: The Observability Trio
The technical fix took ten minutes (auto-refresh the token). What was actually worth something was the three things I bolted on afterward, which I now build into every automation project as standard:
- Alert on failure. The moment a task dies, someone needs to be notified immediately — Feishu (Lark), WeCom, whatever — but it has to be pushed in front of a person's eyes, not left for someone to go pull from a log.
- Heartbeat on success. A failure alert alone isn't enough — what if the alerting system itself goes down? So every successful run should also leave behind a heartbeat record, paired with a sentinel that fires if there's been "no heartbeat in over N hours." A failure alert reports "I hit an error." A heartbeat sentinel reports "I might no longer even be capable of reporting an error."
- Read your data back. I learned this one the hard way in a separate incident later: the API returned "write successful," but the data never made it into the table (an encoding issue was being silently dropped by the platform). So after any critical write, you need to read the data back and verify it. An API's return code only tells you "it received the request" — not "it did what you assumed it did."
Why Beginners (Like Me at the Time) Fall Into This Every Time
Because tutorials only teach you up to the point where things "run successfully." At the moment it runs, the demo works, you feel great, the boss is happy — there's no signal anywhere telling you the system is still only half-built.
That missing half is called operational awareness: assume everything will eventually break, and make sure you're the first to know when it does. The difference between a prototype and a production system isn't lines of code — it's this assumption.
Now, whenever I evaluate any automated system (including ones other people built), the first question is no longer "does it run?" It's:
"When it goes down, how would you know?"
If you can't answer that, it's not delivered yet.