Should a change to an agent's harness be treated as a redesign?

THE SHORT ANSWER

Yes, and more so than the model swap. Jakob Nielsen's expectation economy piece ends on the guideline to treat model changes as redesigns, with the August 2025 case as evidence: GPT-4o replaced by GPT-5, users revolted, the old model back within three days. His ledger is right: familiarity capital is banked by consistency and spent by redesigns, each surprise is a tax, and redesigns get a two-to-four-week A/B test and a classic path. The layer is wrong. The user of an agent product never sees the model; what they feel is the harness, the prompt, the skills, the retrieval budget, the tools. Lesson 08 of Becoming Forward Deployed Engineer has a prompt shortened on a Thursday, four lines removed, seven fewer answers out of 42, noticed two weeks later. Vivek Trivedy moved one model about thirteen points by changing only the harness while the two models in Lesson 06's review were three points apart. My own harness library has 51 commits since March 30, 30 with no name, none tested for two to four weeks, none with a classic path. By Nielsen's ledger each was a redesign. Put six lines under every harness commit: score, rows, name, undo, the previous version selectable for the test window, and a slice before the fleet. The model swap gets the same six; it is just the smallest redesign you ship.

Yes, and the harness commit is the bigger redesign. Nielsen's ledger is right and his guideline names the smallest change an agent product ships. The prompt shortened on a Thursday is the one the users felt, and my own harness has 51 of those since March with 30 unnamed.

The argument, the sizes, and the six lines are in Nielsen Says Treat a Model Change as a Redesign. The Redesign Your Users Feel Is the Harness Commit.. The four original lines are from The Prompt Commit: Score, Rows, Name, Undo Under Every Prompt Change, and the harness-versus-model sizes from Which Model Is the Smallest Knob in the Architecture Review.

SOURCES

THE LONG VERSION

RELATED ANSWERS

Last reviewed 2026-11-04 · 1 min read