How do you detect a silently failing AI agent?

THE SHORT ANSWER

Check freshness, not just reachability. My own source monitor reported 43 of 45 feeds healthy while four of them had published nothing for 72, 356, roughly 550, and 612 days; every one returned HTTP 200, parsed as valid XML, and carried well-formed items. The health check asked whether the source was reachable and parseable, which is the easy question. Record the newest item date from every source on each run, compare it to the run date, and threshold that gap against how often the source normally publishes. The deeper fix is to stop collapsing 'alive but quiet' and 'stopped existing' into the same empty result, because one is information and the other is a defect.

Check freshness, not reachability. They are different questions and only the first one is easy, which is why most health checks answer the second one and stop.

What that looked like for me

I built a monitor that reads 45 sources. Its health check fetched every one, confirmed the response parsed, counted items, and printed a line per source. Forty-three came back ok.

Then I looked at the dates.

SourceItems in feedNewest itemSilent for
Chip Huyen102025-01-16612 days
Zendesk1002024-03-20~550 days
Airtable152025-09-29356 days
Dan Shipper, Chain of Thought502026-07-1072 days

Every one returned HTTP 200. Every one parsed as valid XML. Every one held a healthy number of well-formed items with titles, links, and dates.

All four were dead, and the monitor called them healthy, because I had written the check that was easy to write.

The fix

Record the newest item date from each source on every run. Compare it to the run date. Threshold the gap against how often that source normally publishes.

Two thresholds in my case: 35 days for a source that normally publishes weekly, 120 days for one that publishes rarely. Flagged sources come back with a recommendation to demote or drop.

The numbers are not the point and I would not defend them. The point is that the default threshold is infinity and anything finite beats it.

The state the fix really adds

What that check does underneath is split one state into two.

Most pipelines collapse "this source is alive and had nothing to say this week" and "this source stopped existing" into the same empty result set. The first is information. The second is a defect. A system that cannot tell them apart will report the defect as information on every run, confidently, in a well-formatted table, indefinitely.

There is a second-order version worth building in. When the monitor flagged a newsletter at 72 days, the obvious read was that the author had gone quiet. The likelier read, given that the publication ships several times a week, was that I had the wrong feed URL. Those need opposite fixes, so the flag has to say which one it suspects or half the corrections will be wrong.

The one thing to do this week: take one agent you trust and find the date on the last real thing that came through its freshest input. Not whether the pipe is open. The date. If that takes longer than a minute, you have a green light rather than monitoring.

THE LONG VERSION

RELATED ANSWERS

Last reviewed 2026-09-20 · 3 min read