How do you test a relevance filter you just built?

Read what it threw away, not what it kept. A filter's keep list is selected to look correct and always does, because it is a list of plausible items; the errors are in the discard pile. My own filter over 45 sources was wrong three times before it was right: version one dropped 'How Included Health Built Federated Healthcare Agents with LangGraph' at a score of 2 while keeping 'When Inbound Sells Itself' at 3, because it weighted a title the same as a summary. Version two scored a post about Codex at zero, because the vocabulary was built from concepts and held no proper nouns. A related audit flagged GitHub as a coverage gap because it counted posts rather than mentions per post. All three were proxy errors, and none were visible in the output.

Why is the keep list useless for testing a filter?

Because it is the argument the filter makes for itself. A keep list is a set of plausible items selected specifically to look correct, and it will look correct whether the filter is working or not. The errors live entirely in the discard pile, so a filter that cannot show you what it rejected cannot be checked at all.

What is a proxy error?

Substituting a measurable thing for the thing you care about, where the substitute is close enough to pass a glance and fails in a direction the output cannot reveal. Word count standing in for relevance, a concept taxonomy standing in for vocabulary, post count standing in for interest. None of mine were logic errors; every version ran correctly and produced a clean list.

Why do keyword filters miss proper nouns?

Because vocabularies get built from a taxonomy, and a taxonomy holds abstractions. Mine had 'agent' and 'eval' and 'pricing' and not Codex, Claude Code, Cursor, LangGraph, or subagents, even though those names are in the titles of my own posts. The abstractions are what you think you write about; the proper nouns are what you actually name, and what other people put in headlines.

How do you tell a subject from a passing mention in a corpus?

Mentions per post, not post count. GitHub appears in 45 of my posts at 3.1 mentions each, because agent blueprints list it as a data source. Airtable appears in 8 at 12.5 each, because those 8 posts are about Airtable. Post count alone flagged GitHub, Loom, Mixpanel, and Weaviate as gaps, and all four were wrong in the same way.

Should a filter drop things silently?

The output can be clean as long as the rejections stay inspectable. Mine prints a drop count per source and has a verbose mode listing every rejected title with its score. Clean by default and auditable on demand are compatible, and skipping the second is how a tool quietly stops deserving trust.

Where should corrections to a filter go?

Into the shared vocabulary, not into per-source special cases. When mine is wrong the fix is a keyword in the cluster definitions that the rest of the site already uses, so the filter stays tied to what the site is about. Special-casing a source produces a private set of exceptions that only the author understands and that nothing else validates.

How to Test a Relevance Filter

Read the drop list. The keep list is designed to look right, and it will.

That is the whole method, and I arrived at it by being wrong three times in one day on a filter over 45 sources.

Bug one: a title is not a sentence

Version one weighted every keyword equally, wherever it appeared.

Title	Score	Verdict
How Included Health Built Federated Healthcare Agents with LangGraph	2	dropped
When Inbound Sells Itself	3	kept

The first is enterprise agents in production, which is the center of what I write about. The second is a good post about demand generation I would not have opened. One point apart, in the wrong direction.

Fix: title matches count double, and the vocabulary splits into core terms that carry weight alone and supporting terms that only corroborate.

Bug two: no proper nouns

Version two scored "How to Use Codex for Work: My Complete System at OpenAI" at zero. No vocabulary match at all.

Because Codex was not in the list. Nor Claude Code, Cursor, LangGraph, or subagents. I had built the vocabulary out of concepts, which is what I think I write about, and left out every proper noun, which is what I actually name. My own posts have those tools in their titles.

Bug three: post count is not interest

A separate audit, meant to find companies I write about but do not follow, flagged GitHub as a gap. GitHub appears in 45 of my posts. It also flagged Loom, Mixpanel, and Weaviate.

All four wrong, all in the same way: they appear constantly because agent blueprints list them as data sources.

Company	Posts	Mentions	Per post
Miro	3	50	16.7
Airtable	8	100	12.5
GitHub	45	141	3.1
Loom	7	13	1.9

Density separates a subject from a tool.

What they have in common

None were logic errors. Every version ran correctly and produced a clean list of plausible titles.

They were proxy errors: a measurable stand-in for the real quantity, close enough to survive a glance, failing in a direction the output could not show. And they lasted because the output always looked fine, which is what a keep list does.

So the filter now prints a drop count per source and has a verbose mode listing every rejection with its score. Clean by default, inspectable on demand.

Two rules I would carry to any filter. Corrections go into the shared vocabulary rather than per-source special cases, so the thing stays tied to a definition something else validates. And the threshold gets written down as unprincipled while that is still obvious, before it calcifies into a number nobody questions. Mine is 4, tuned against one real week, and it will need revisiting.

The one thing to do this week: take any filter, ranking, or triage rule you run, in code or in your head, and look at what it threw away.

How do you test a relevance filter you just built?

Bug one: a title is not a sentence

Bug two: no proper nouns

Bug three: post count is not interest

What they have in common

THE LONG VERSION

RELATED ANSWERS