Should a single benchmark score decide whether an AI agent ships?

Answered by Falk Gottlob · Reviewed · 1 min read

THE SHORT ANSWER

No. Salesforce's EGI-Bench, as Shelby Heinecke described it on November 12, 2026, scores six capabilities across more than 1,500 tasks and weights them into one global score with default weights based on experience, and its own first finding is that across 14 models no single model was best across all six dimensions. When no model leads everywhere, the weights choose the winner, and they are someone else's experience, not your customer's failure log. Keep the six subscores as a reading list ordered by what broke in the pilot, then gate on 200 rows of the customer's own tasks sliced by the kind of work that fails, with a floor per slice and a date to re-run on production rows. A classifier of mine scored 89% held out and 61% in production in six weeks; a general number does that.

No. When no model leads on every dimension, a single weighted score is a decision made by the weights, and the default weights were not set by your customer. Use the subscores to decide where to look, and ship on 200 rows of the customer's own work, sliced, with a floor and a date.

The EGI-Bench findings and the gate are in Salesforce's EGI-Bench Rolls 1,500 Tasks Into One Score. Kill the Global Score. Ship on the Slice.. Why the model is the smallest knob is in Which Model Is the Smallest Knob in the Architecture Review, and the 89-to-61 story is in Rules 97.5%, Trained 100%, Model 87.5%. The Bake-Off Table Needs a Slice and a Date..

SOURCES

THE LONG VERSION

RELATED ANSWERS