I Automated My Reading. The Filter Was Wrong Three Times.

I spent today automating something I have been doing by hand every week: reading a pile of sources and turning the week into post ideas.

Fetching 45 feeds took about twenty minutes. The filter took the rest of the day, and it was wrong three times, each time in a different and useful way.

I am writing the bugs down because the finished version looks obvious and the path to it was not, and because the general shape of the mistake is one I keep making in other places.

The short version

A relevance filter over 45 sources cut a 102-post week to 34. Version one weighted every keyword equally and dropped an enterprise-agents case study while keeping a demand-generation post, because it could not tell a title from a summary. Version two scored a piece about Codex at zero, because the vocabulary was built from concepts and contained no proper nouns. A separate audit meant to find companies I write about but do not follow flagged GitHub, which appears in 45 of my posts as a data source in agent blueprints rather than as a subject; the fix was to measure mentions per post instead of post count. All three bugs were proxies that looked like the real quantity, all three were invisible in the output, and all three were caught by reading the rejects rather than the keeps.

Bug one: it could not tell a title from a sentence

Version one was the obvious thing. Build a word list from my five content clusters, score each post by how many terms it matches, drop anything under the threshold.

Here is what it did:

The first is a case study of enterprise agents in production at a healthcare company, which is approximately the center of what I write about. The second is a perfectly good post about demand generation that I would not have opened.

The filter had them exactly backwards, and it was not a close call in its own terms. One point apart, in the wrong direction.

The cause is dull. "Agents," "healthcare," and "built" each scored one, the same as any word matching anywhere in the text, including the summary blurb. A word in the headline and a word in the third line of a description were worth the same amount.

That is wrong about how writing works. A title is a claim about what the piece is. A summary is whatever the CMS truncated.

Two fixes. Title matches count double. And the vocabulary splits in two: core terms like "agent," "eval," "pricing," and "margin," which are enough on their own, and supporting terms like "production," "cost," and "enterprise," which only corroborate.

Bug two: no proper nouns

Version two was better. Then I ran it over a wider window and found this:

"How to Use Codex for Work: My Complete System at OpenAI" scored zero.

Not low. Zero. No vocabulary match at all, from a post that ran in Peter Yang's newsletter about someone's actual working system with a coding agent at OpenAI.

Because "Codex" was not in my word list. Neither was Claude Code. Neither was Cursor, or LangGraph, or subagents.

I had built a vocabulary entirely out of concepts, which is what I think I write about, and omitted every proper noun, which is what I actually name. I have a post called setup-guide-claude. I have one called claude-skills-for-pms. The tools are in the titles of my own work and they were not in the list.

This is the one I would generalize hardest. If you build a keyword system from a taxonomy, you will get the abstractions and miss the names, and the names are what people put in headlines.

Bug three: post count is not interest

Different system, same species of error.

I wrote an audit whose job is to compare the companies I write about against the companies my monitor follows, and flag anything I cover repeatedly but do not watch. It scans all 555 content files.

Its first output flagged GitHub as a coverage gap. GitHub appears in 45 of my posts.

It also flagged Loom, Mixpanel, and Weaviate.

All four are wrong, and wrong in the same way. They appear constantly because my agent blueprints list them as data sources. "Pulls from six systems: GitHub, Jira, analytics, Zendesk, Salesforce, Slack." That is not coverage of GitHub. That is GitHub being furniture.

The fix is one column: mentions per post.

Density separates a subject from a tool. Miro at 16.7 is three posts that are about Miro. GitHub at 3.1 is a name on a list, forty-five times.

I also pulled "Segment" out of the candidate list entirely, because in my writing it is almost always the common noun. Customer segment. Segment health. The company by that name had nothing to do with any of it.

The thing all three have in common

None of these were logic errors. Every version ran correctly and produced a clean list.

They were proxy errors. I substituted a measurable thing for the thing I cared about, the substitute was close enough to survive a glance, and it failed in a direction the output could not reveal.

Word count stood in for relevance. A concept taxonomy stood in for vocabulary. Post count stood in for interest.

And here is why they lasted as long as they did: the output always looked fine. It is a list of plausible titles. A filter's keep list is selected to look right. That is its entire job. You cannot audit a filter by reading what it kept, because what it kept is the argument it is making for itself.

I only found all three by looking at the rejects.

So the filter now prints a drop count per source, and has a verbose mode that lists every rejected title with its score. The daily output stays clean, because clean was the point. But the discarded pile is one flag away at all times.

That is the same argument I made about enforcement last week, turned inward. You can only govern what is observable at a chokepoint. A filter's discard pile is the chokepoint, and I had built the thing without a window onto it.

What I would keep

Three things from this that are not about filters.

The threshold is not principled. It is 4, tuned against one real week, and I do not have a defense of it beyond "it produced 34 out of 102 and I read all 34 and they were right." It will need revisiting. Writing that down now, before it calcifies into a number nobody questions, is most of the value of writing it down at all.

Fix the vocabulary, not the code. When the filter is wrong the temptation is to special-case the source. I made the rule that corrections go into the cluster keyword list instead, so the filter stays tied to what the site is about and cannot drift into a private set of exceptions that only I understand.

The audit outlives the fix. The company audit exists so that next quarter, when I have written about something new for three months, something tells me rather than me remembering. The bug I fixed today is worth less than the thing that catches the next one.

Pick one thing this week. Take any filter, ranking, or triage rule you run, in code or in your head, and look at what it threw away rather than what it kept.

The keep list is designed to look right. It will.

Sources: All figures measured against this site's own content and monitoring script on 2026-09-20: 555 content files, 45 sources, 102 posts in the 2026-09-13 to 2026-09-20 window, 34 retained. Post titles quoted are from the LangChain, Tomasz Tunguz, and Peter Yang feeds in that window.

This one belongs to the running argument on Building a Company in Public: the decisions are worth more before the outcome is known, and a post about three bugs I shipped today is more useful than a post about a system that was right the first time, because the second one would not be true.

Related answer: How do you test a relevance filter you just built?