
Tony Alicea's research-in-the-loop article for Nielsen Norman Group came out on November 13. It has a template in it that I would paste into a spec today, and three lines I would add first. This is an Agent Drop: the section, with the additions, ready to copy.
The short version
- Tony Alicea's November 13, 2026 NN/g article argues that coding agents made a release cheaper than a study, so research should define what each release captures and let the agent build that capture with the feature.
- His template is a Research section in the spec with five parts: what we expect, signs it worked, signs it caused problems, what to track, and when to talk to people. His example is a help desk that measures time to resolution. I billed on resolution once.
- Resolved meant no escalation within 48 hours, which held for 98% of cases, and the other 2% became more than 200 contested cases a week until the window changed to 7 days and disputes fell 60%.
- My dispute forecast missed by 12x with no line saying when to act, and 12 accounts sat on plan on the tracker while barely using the product.
- So add three lines before the agent builds: the unit and its window, the tripwire with a number, an owner, and a date, and what the dashboard will call fine.
What Alicea proposes
His starting point is a photography analogy. Film made every shot expensive, so photographers set up carefully. Digital made shots nearly free, so the learning moved to after the shot. Coding agents did that to releases. Many teams now ship, watch, and change.
His worry is the obvious one. A bad photo stays on the card. A bad release reaches people. And a dashboard shows that users abandoned a flow without showing why.
His answer is that the same drop in cost applies to research tooling. An intercept survey, event logging for a new feature, a feedback button tied to one release: each used to need engineering time. Now an agent can build them alongside the feature from a few lines in the requirements.
So he puts those lines in the spec. For a help-desk product adding direct ticket handoffs between departments, the section reads roughly like this:
- What we expect. Direct handoffs get tickets to the right team faster.
- Signs suggesting it worked. Less time from creation to resolution. Fewer tickets through three or more departments. Lower cost per ticket.
- Signs suggesting it caused problems. More tickets sent back, or reopened by customers.
- What to track. Every handoff: sender, receiver, wait, and whether it came back, on a small dashboard grouped by department.
- When to talk to people. When a department sends back an unusually high share of tickets, or tickets stall there.
He is careful about the limits. This is not yet an evaluation design. It has no comparison group or observation period, and he notes that faster resolution may only mean the ticket waited less, not that it cost less. He ends with nine open questions.
I think the template is right and I would use it. The additions come from the one time I had real money riding on his example's key word.
Line six: the unit and its window
Every sign in the section rests on an event, and every event has a definition somebody chose.
In Field Report: What Broke When We Killed Our Per-Seat Tier I describe billing per resolved ticket. Resolved meant the customer did not escalate within 48 hours. That worked for 98% of cases.
98%
The other 2% were customers who came back after 60 to 72 hours with a follow-up they considered the same issue. Our system had already counted the first ticket as resolved. At low volume we handled those one at a time. At full volume they were more than 200 contested cases a week.
We rewrote the definition to no follow-up within 7 days. It cost some revenue and cut disputes by 60%.
Alicea's sign is "less time between a ticket's creation and its resolution." An agent will build that. It has to pick a definition of resolution to do it, and if nobody wrote one, the status field is the nearest thing to hand. That was my 48-hour rule in a different coat.
So the line is: for each sign, the event, its definition, and how long you wait before counting it.
Line seven: the tripwire
"When a department sends back an unusually high share" is a sentence people nod at, and it is why nothing happens.
Unusually high compared with what, decided by whom, by when? On my migration we had projected dispute volume from the pilot. The real number came in at 12 times the forecast. Nothing in the plan said what we would do at twice the forecast, so there was no moment at which anyone was obliged to stop and look. We found out by being overwhelmed.
A tripwire is three things: a number, the person who acts when it is crossed, and a date by which they will have looked. An agent can wire an alert to it. An agent cannot wire an alert to "unusually."
Line eight: what the dashboard will call fine
Alicea already makes the point that instrumentation only sees what it was built to see. His example is support agents who skip the official handoff and message a friend in another department.
Mine cost two customers. A tracker had been watching the migration since month four. By month 18 it showed 12 accounts as on plan. They had migrated on paper and were paying their committed minimums. They were barely using the product. Nobody on the account teams escalated, because the tracker was green. When the old tier was switched off, those accounts found out the hard way, and two left.
The fix was a second dimension. Migrated was one fact and actively using was another, and only one of them was on the dashboard.
So the last line asks, before launch: which group could go quiet without moving anything we track? Name it, and track last active use for it. A green status is a claim with a shelf life like any other, and I made the case for putting a re-check date on claims in Nielsen Calls Unused Research Expensive Filing. Ship the Three Fixes, Then File the Three Claims.
The section, with eight lines
## Research
**What we expect:**
**Signs suggesting it worked:**
**Signs suggesting it caused problems:**
**What to track:**
**When to talk to people:**
**Units and windows:** for each sign, the event, its definition, and the wait before counting.
**Tripwires:** number, owner, date. One per sign of harm.
**What the dashboard will call fine:** the cohort that can go quiet, and its last-active-use metric.
The first five are Alicea's. The handbook chapter Research at Generation Speed(coming Dec 16) covers how the research side of a team changes when this becomes routine.
This week
Open the spec for the next feature a coding agent will build. Paste the section. Fill the last three lines first, because they are the ones that take an argument, and have that argument before the build, when it costs a conversation. This is the kind of spec work that AI Product Management now turns on: the agent builds whatever the page says, including the measurement.
Related answer: What should the research section of a product spec include?
Sources: Tony Alicea, "Research-In-The-Loop: How UXR Adapts to Development at AI Speed," Nielsen Norman Group, November 13, 2026, read in full. The five-part section, the help-desk example, and the limits he lists are his; the bulleted version above is my condensed paraphrase of his example, not a quotation. The 48 hours, 98%, 2%, 60 to 72 hours, 200, 7 days, 60%, 12 times, month 4, month 18, 12 accounts, and the two lost are my own, from the field report linked above.
Frequently asked
What is research-in-the-loop?+
Tony Alicea's term, in a November 13, 2026 Nielsen Norman Group article, for fitting UX research into AI-assisted development. Because a coding agent can build event logging, dashboards, and intercept surveys alongside a feature, research can specify what each release must capture before the build, study what happened after, and feed the next set of requirements. He is clear that instrumentation supports evaluation and investigation and does not replace open-ended discovery.
What goes in the Research section of a spec, per NN/g?+
Five parts, written in plain language next to the product requirements: what we expect, signs suggesting it worked, signs suggesting it caused problems, what to track, and when to talk to people. Alicea's example is a help-desk product letting departments pass tickets directly, with time to resolution, handoff counts, and reopens as the signals.
What three lines would you add?+
The unit and its window for every sign, so that a word like resolved has a definition and a waiting period. The tripwire, a number with an owner and a date at which someone acts, in place of a phrase like unusually high. And what the dashboard will call fine, meaning the cohort that can go quiet without moving any tracked metric, with last active use tracked for it.
Where do those lines come from?+
From a pricing migration I ran. Resolved meant no escalation within 48 hours and held for 98% of cases; the other 2% reopened at 60 to 72 hours and became more than 200 contested cases a week until the definition changed to no follow-up in 7 days, which cut disputes 60%. The dispute forecast missed by 12x with no stated threshold for acting. And 12 accounts were classified as on plan while barely using the product, and two were lost.
Does Alicea say the section is complete?+
No. He writes that it identifies what to measure and does not establish whether a change was caused by the feature, that a valid evaluation may need a comparison group, a sample size, and an observation period, and that faster resolution may not mean lower cost. He lists nine open questions, including who owns the research requirements.


Sign in with LinkedIn to leave a comment.
Sign in with LinkedIn