Copilot Adopters Added 1.4 Hours of Collaboration a Month. That's Critique. Plan It.

Copilot adopters added 1.4 hours of collaboration a month, much of it vetting AI output. That's critique with no format. Plan it, split it, and log it.

Falk Gottlob · · 10 min read

Green Falkster cover: seven paper kites lie on a dark workbench with their strings cut and a blank tag tied to each, orange scissors beside them, while one hand holds three kites with orange tails still flying on taut strings.
7/7
SeriesThe Design Transition

On Friday, Jakob Nielsen reported a study that should change how design leaders plan next quarter.

Giuseppe Soda and colleagues at Bocconi tracked more than 12,000 employees at a consumer-products multinational for 13 months while Microsoft Copilot rolled out. The bet most people made was that AI would make work more solitary. It went the other way. As Nielsen reports the figures, in their first three months adopters started about 1.4 more hours of collaboration a month, mostly meetings and calls, and worked with about two more colleagues.

Why? The authors' reading is that AI output has to be interpreted, checked, and endorsed by someone, and adopters took it to coworkers they already trusted. Nielsen has a word for that circle, the "vetwork." His summary is that generating got close to free while validating still costs other people's time.

Designers already have a name for colleagues checking a draft. It's critique. What Soda measured is critique with no format, no calendar, and no record, spreading through a company one "can you look at this?" at a time. The logs can't even show whether the colleague looked hard or waved it through, which Nielsen flags as a limit of the study. I'd call it the finding.

This is part 7 of The Design Transition. Part 6 put real users in front of the AI-only version. This part is about the step before that, when the first draft on the table is the AI's and the team has to judge it.

The short version

  • When AI makes the first draft, the scarce thing is judging, and judging is what a critique is for. Soda and colleagues found that Copilot adopters among 12,000 workers added about 1.4 hours of collaboration a month, much of it colleagues vetting AI output.
  • Nielsen's report of the Kombai benchmark says the draft itself is nearly solved: models met 99% of 160 design briefs, ten alternatives cost about $53, and the review time is the real cost.
  • Michelle Vaccaro's meta-analysis of 420 studies says a person stacked on the same task as the AI loses to the AI alone, while a split of subtasks holds its own. So split the critique.
  • The AI generates the set and runs the first pass against the rubric, people rank what matters and kill options with a written reason, and the reasons go back into what the agents read. Plan those hours as capacity.
  • And since the PM's spec, the marketer's story, and the field's fixes start as AI drafts too, run one critique practice across the product org, under the CPO, with design teaching it.

The draft got cheap. The judging didn't.

Nielsen's November 16 roundup covers a benchmark from Aman Rusia and colleagues at Kombai: 15 frontier models and several human designers, the same 160 briefs for screens like dashboards and forms. Every model met more than 90% of the criteria. Two met 99%, a point ahead of the humans at 98%. Nielsen's advice is to ask for ten designs and merge the best, because a model's first design is its most average one. He prices that at about $53 for ten from the most expensive model tested, and then says the real cost is the time it takes to review them.

I wrote yesterday about what that benchmark does to the brief. Here I care about his last point. Ten options for $53 means a design team is no longer limited by how many directions it can produce. The limit is how many it can judge well.

Designers were already saying this in August. In Ioana Teleanu's roundup of more than 30 designers' AI workflows, Cem Kurtaraner put it in one line: "We've lowered the cost of proposing solutions, but not the cost of judging them." He also admitted he catches himself exploring less and selecting more. That's a critique problem, and nothing in the tool stack is pointed at it.

So here's the decision for this quarter. Plan critique as capacity, the way engineering leaders plan code review. Hours per week, a queue, and a name on it.

1.4 hours

more collaboration a month for Copilot adopters, as Jakob Nielsen reports Soda and colleagues. The checking moved to coworkers. Design calls that critique.

Don't stack a critic on the draft. Split the work.

The easy move is to keep the old crit and aim it at the AI's draft. A room of people improving one generated screen, line by line. The same roundup says that's the arrangement that loses.

Michelle Vaccaro at Harvard Business School and co-authors from MIT pooled 1,955 effect sizes from 420 papers on people working with generative AI. As Nielsen reports it, when the person and the AI worked the same task, the team lost to the AI alone (a Hedges' g of -0.35). When they took different subtasks, the team matched it (+0.14, not significant). I argued in Score the Fourth Arm that a person's value shows up in what they write down for the next run. Critique is the meeting where that writing gets done.

So what's the split? A second study from Friday's roundup gives a hint. Sanchaita Hazra and co-authors had an AI critic review survey drafts, with and without simulated respondents, and compared it with six paid human pilot testers. It's a study of survey pilots and the authors rated the critiques themselves, so take it as a pointer. As Nielsen reports it, on a 0 to 3 quality scale the AI critiques scored 2.44 and 2.31, and the human pilots scored 1.53. But when researchers went through the critic's severity ratings, they demoted far more than they promoted. The critic called 46% of structural errors high severity. Only 21% kept that rating.

Machines find. People rank.

That's the split I'd build a critique on. The AI generates the set, ten directions that differ on purpose. An AI critic runs the first pass against the rubric and lists what each option breaks. Then people do the two jobs that are left. They re-rank severity, because the critic cries wolf. And they kill, with one written line of why for every option that dies.

Nielsen predicts that by 2029 AI critique will beat most human critique. I'd plan as if he's right about finding problems. The ranking and the kill will stay with people longer, I think, because they depend on what the company is trying to do this quarter, and that's rarely written anywhere a critic can read.

Smaller, earlier, and written down

What does the meeting look like? The best public description I know is Designer Fund's October piece on Anthropic's design team. Critiques there moved from two large sessions a week to pod sessions as often as four times a week, because, as Joel Lewenstein tells it, work doesn't hold still for a weekly meeting at their pace. On Kyle Turman's team, standups grew from 30 minutes to 90 so the group could argue requirements and architecture before building with Claude. And a designer built an internal tool, go/krit, that logs critique sessions and surfaces the themes that keep coming back.

The critique got smaller and more frequent, so it keeps up with generation. It moved earlier, onto the requirements, where an hour of argument saves a pile of generated dead ends. And it got a log.

The log is the part I'd copy first. In Polish Stopped Being Evidence I said the question that separates choosing from picking is "what did you kill and why." When the first draft is the AI's, those kill reasons are the most valuable thing the meeting makes, because each one is a rule the next generation should start from. Put them where agents read them, which part 4 argued is the design system's job now. A critique that leaves no written reason behind gets held again next week, about the same mistake.

Everyone's first draft is the AI's

Here's design's piece of the one-function case this week.

Soda's adopters weren't designers. They were knowledge workers across a company, each building a private circle of people to check their AI drafts. And the paper's abstract adds a detail I keep coming back to: the extra collaboration leaned on long-standing ties. People took their drafts to whoever they already trusted.

Now look at a product org. The PM's first draft is an AI's, and the spec is an eval someone has to read row by row. Product marketing's messaging file gets drafted and tested the same way. The field's fixes start as agent output too. That's a lot of generated work, judged along whoever trusts whom.

Design is the one seat with a trained critique practice: showing unfinished work, and arguing about the choice without arguing about the person. If design sits in its own org, that practice stays inside the design crit while everyone else improvises a vetwork. Put the seats in one function and it becomes the product org's judging capacity. One cadence where the spec, the screen, and the story get critiqued against one outcome, and one log of reasons. That's the arrangement The CPO Transition keeps arriving at, most recently when the story became a product artifact. I don't want design gatekeeping any of it. Design should teach it and keep the log.

What I expect

These are my predictions, not facts.

By the end of 2027, design leaders at AI-native companies will plan critique hours per week as a capacity line, and they'll watch how long generated work waits for a verdict the way engineering watches review queues.

By 2028, the first pass of most critiques will be an AI critic run against the team's rubric before anyone meets. The meeting gets shorter and covers severity and kills only.

And the critique log becomes the main source of new lines in design-system context files, ahead of anything written in a planning cycle.

One thing to try this week

Take the next critique on your calendar and rebuild it.

Ask for ten versions of the work, not one. Before the meeting, have a model critique all ten against your rubric and send the list around. In the room, do only two jobs: re-rank what the critic flagged, and kill seven options with one written line each. Then paste those seven lines into the file your design agent reads.

One more, if you have five minutes. Ask your team to count the "can you look at this?" messages they sent last week. That's your vetwork. It's already costing you hours, and now you'll know how many.

Sources: Jakob Nielsen, "UX Roundup: No More Web, Social Knowledge Work, Synthetic Users, and more," UX Tigers, November 13, 2026, for the Soda figures and caveats, the "vetwork," the Hazra study as he reports it, and his 2029 prediction. Giuseppe Soda, Alessandro Iorio, and Leonardo Rizzo, "The Effects of Generative AI on Collaboration and Social Networks Among Knowledge Workers," arXiv, October 23, 2026. Jakob Nielsen, "UX Roundup: Death of AI-Human Synergy, AI Answers Design Briefs, and more," UX Tigers, November 16, 2026, for the Kombai benchmark by Aman Rusia and colleagues and the AutoAnalysis meta-analysis by Michelle Vaccaro and co-authors, both as he reports them. Sanchaita Hazra and co-authors, arXiv 2610.25343, October 2026, cited as Nielsen reports it. Designer Fund, "When code is no longer the constraint: Inside Anthropic's design team," October 6, 2026, for the critique cadence, Joel Lewenstein, Kyle Turman's standups, and go/krit. Ioana Teleanu, "how designers actually work in 2026," AI Goodies, August 6, 2026, for Cem Kurtaraner's quote.

Part of the running argument on AI Product Management: AI collapsed the cost of the parts product teams were trained to be good at, and for design that means the draft is nearly free while judging still costs people's hours, so the critique is the work to plan, split, and write down.

Related answer: How should design critique change when AI makes the first draft?

Share this post

Frequently asked

How should design critique change when AI makes the first draft?+

Split it. The AI generates a set of options that differ on purpose and an AI critic runs the first pass against the rubric. People do the two jobs that are left: they re-rank the severity of what the critic flagged, and they kill options with one written line of why. The kill reasons go into the file the design agents read, so the next generation starts from them.

What did the Copilot collaboration study find?+

Giuseppe Soda, Alessandro Iorio, and Leonardo Rizzo tracked more than 12,000 knowledge workers for 13 months during a phased rollout of Microsoft Copilot. AI use increased collaboration time, users worked with more colleagues, and the extra collaboration leaned on long-standing ties. As Jakob Nielsen reports the figures, adopters started about 1.4 more hours of collaboration a month in their first three months and worked with about two more colleagues. The logs can't show whether those colleagues checked the output closely.

What is a vetwork?+

Jakob Nielsen's word for the circle of trusted coworkers a person takes AI output to for checking. In design terms it's a critique with no format, no calendar, and no record.

Why not keep the old critique and point it at the AI's draft?+

Because a person stacked on the same task as the AI tends to lose to the AI alone. Michelle Vaccaro's meta-analysis of 420 studies, as Jakob Nielsen reports it, found teams lost to AI alone when both worked the same task, with a Hedges' g of -0.35, and matched it when they took different subtasks. A room improving one generated screen line by line is the stacked arrangement.

Can an AI run the critique?+

It can run the first pass. In a study of survey pilots by Sanchaita Hazra and co-authors, as Jakob Nielsen reports it, AI critiques scored 2.44 and 2.31 on a 0 to 3 quality scale against 1.53 for six paid human pilot testers. But the critic rated 46% of structural errors high severity and only 21% kept that rating after researchers reviewed them. Machines find, people rank. Nielsen predicts AI critique will beat most human critique by 2029.

Why does critique belong in one function under the CPO?+

Because every seat's first draft is the AI's now: the PM's spec, product marketing's messaging, and the field's fixes as well as the screen. If each seat improvises its own circle of checkers, the work gets judged along whoever trusts whom. Design is the seat with a trained critique practice, and in one function under the CPO it becomes a shared cadence with one log of reasons, read against one outcome.

AI designdesign leadershipCPODesign TransitionProduct OSdesign critiquedesign reviewvetwork
Helpful?

Keep reading

The short answer

FDE Economics: Margins, Outcome Pricing, and When to StopForward deployment costs like services and has to earn like software. The math works only if the loop closes and the pricing puts revenue behind the result.Read the chapter

Part of

Product Leadership

About the author

Falk Gottlob

Falk Gottlob

Product Executive · Founder, Falkster.AI

Thirty years shipping product, from Microsoft Research and Adobe to Salesforce, where he grew Quip into what became Slack Canvas. Four startup exits, including a $6.5B healthcare merger and a company Microsoft bought. Four-time Chief Product Officer. Now founder of Falkster.AI, building Heidi, the company brain for AI-native businesses. This notebook is written from inside the build, not above it.

Sign in with LinkedIn to leave a comment.

Sign in with LinkedIn