
Shelby Heinecke, Director of AI Research at Salesforce, published on November 12 the six capabilities her team evaluates before an AI agent is deployed in an enterprise, and the benchmark that scores them. This is a Kill List entry, and the thing to kill is the last step of that benchmark.
The short version
- Salesforce's EGI-Bench, as Shelby Heinecke describes it on November 12, 2026, unifies eight enterprise benchmarks including CRMArena, more than 20 subdomains and more than 1,500 tasks, into six capability subscores, task outcomes, tool actioning, planning, grounding, safety, and engineering, and then weights them into a single global EGI score using default weights based on experience.
- Her findings across 14 models: no single model was best across all six dimensions, planning scores were on the lower end for every model, and cost-efficient models are punching above their weight. The six are the right list.
- The first finding kills the one number, because if no model leads on all six the weights pick the winner, and the weights are someone else's experience rather than the customer's failures.
- A classifier of mine went from 89% held out to 61% in production in six weeks, a feature of mine passed 200 real cases and failed on a slice that was a fifth of traffic, and an architecture review I sat in argued over three points of model while the harness moved thirteen.
- The gate is the six subscores as a reading list ordered by the customer's failure inventory, then 200 rows of the customer's own tasks sliced by the work that fails, a floor per slice, a date to re-run on production rows, and the model question asked once, after the harness.
What Salesforce built
Heinecke's frame is the right one. "Getting the answer right is not enough." An enterprise agent has to be accurate, reliable, and trustworthy across the way enterprise work actually happens, so EGI-Bench evaluates six things: task outcomes (the correct response), tool actioning (the right tools at the right time), planning (breaking complex tasks into subtasks), grounding (using context well and citing correctly), safety (abstaining when appropriate and flagging policy violations), and engineering (building software).
The mechanics: "EGI-Bench spans more than 20 subdomains, more than 1,500 test tasks, and six enterprise skill dimensions." It unifies eight independently developed enterprise benchmarks, CRMArena among them, maps each benchmark's metrics onto the six dimensions, and normalizes them so they can be combined. That "produced six EGI-Bench subscores, which were then weighted to create a single global EGI score." "The repository provides default weights based on experience, but practitioners can change them." There is a quick mode that subsamples for a directional signal and a deep mode that runs everything. The code is on GitHub and the report is on arXiv; I have read neither, and everything here is as her post reports it.
Then the findings from testing 14 models. "No single model was best across all six dimensions." "Planning scores were on the lower end across the models evaluated." And "cost-efficient models are punching above their weight," with Haiku 4.5, GPT-5 mini, and Gemini 2.5 Flash named.
The first finding kills the last step
Read those three findings against the one number the framework produces.
If no model is best across all six dimensions, then the global score does not discover a winner. The weights choose one. Shift weight from planning to grounding and the ranking moves; shift it back and it moves again. The default weights are, in the post's own words, based on experience, which means based on the experience of people who are not your customer and have not read your customer's failure log. A single score built that way is a decision disguised as a measurement, and the disguise is good enough that it will end up in a slide.
The second finding is the proof. Planning is weakest for every model tested. That is the most actionable sentence in the post for anyone deploying an agent into multi-step work, and it is exactly the sentence a weighted average erases. Once planning is one of six terms, a model that plans badly and grounds well can outscore a model that plans well, and the slide will not say so.
I do not want the six dimensions killed. They are a better list than most enterprise eval decks I have seen, and the honesty about the cheap models is worth more than the score. The kill is narrow: the global score as the thing that decides whether an agent ships.
What a general number does in production
I have watched this happen with my own numbers, which is why I am not willing to trust a better-built one.
A classifier of mine scored 89% on held-out data and 61% in production within six weeks, because the held-out set shared the training set's skew. A feature of mine passed 200 real cases and failed on one slice that was a fifth of traffic, because the 200 underweighted that slice. Both stories are in Rules 97.5%, Trained 100%, Model 87.5%. The Bake-Off Table Needs a Slice and a Date.. In both, the general number was true and the deployment failed anyway, because the general number was an average over a mix that production did not share.
89% to 61%
A global EGI score is the same shape one level up: an average over 1,500 tasks in 20 subdomains, weighted by a default. Your customer's work is one subdomain, with its own mix of the six, and the failures that will cost you the renewal live in one or two of them.
The gate
Here is what I use instead, and the six subscores have a place in it.
First, read the six subscores as a reading list, not a verdict. Order them by the customer's own failure inventory, the thing that actually broke in the pilot. If planning is weakest everywhere and the customer's work is multi-step, planning is the first slice and the subscore tells you which models to probe there. That is the subscores doing what they are good at, pointing.
Second, 200 rows of the customer's own tasks, sliced by the kind of work that fails, with a floor per slice that the customer's experts agree to and a date to re-run the whole table on production rows. The 200-row sliced eval has been my ship gate since Which Model Is the Smallest Knob in the Architecture Review, where the review I sat in argued over three points of model on a leaderboard that measured a different task while the harness, in measurements published the same week, moved thirteen. The same post is where the third rule comes from: ask the model question once, after the harness is fixed, because it is the smallest knob.
Third, before a renewal, score what the agent actually closed more than one way. In One Support Agent, Three Resolution Rates the same 30 conversations produced three different resolution rates depending on the definition. A global score has one definition baked in by its weights; the customer's contract has another; the customer's experience has a third. The Enterprise AI Agents thesis is that what matters is how many agents still complete production work after ninety days and what each outcome costs. Neither of those is a benchmark score, and both of them are a slice with a date.
Heinecke's own numbers say the cheap models are competitive and planning is the weak spot. A single score hides both. The slice shows both. Ship on the slice.
This week
Take whatever benchmark score is in your current agent deck. Find the subscore or dimension it averages over that most resembles the work your customer actually does, and write that one number next to the global one. If they disagree, you have found the slice. If you cannot find a subscore that resembles the work, you have found something worse.
Related answer: Should a single benchmark score decide whether an AI agent ships?
Sources: Shelby Heinecke, "6 AI Agent Capabilities to Evaluate Before Enterprise Deployment," Salesforce Engineering, November 12, 2026, read in full; the eight benchmarks, 20 subdomains, 1,500 tasks, 14 models, six dimensions, the three named models, and the three findings are Salesforce's as she reports them, and the EGI-Bench repository and arXiv report were not opened. The 89 and 61, the 200 and the fifth, the three and the thirteen, and the 30 conversations are my own, from the posts linked above.
Frequently asked
What is Salesforce's EGI-Bench?+
A framework from Salesforce AI Research, described by Shelby Heinecke on November 12, 2026, for judging whether an agent is ready for enterprise use. It unifies eight independently developed enterprise benchmarks, CRMArena among them, across more than 20 subdomains and more than 1,500 tasks, maps their metrics onto six capabilities, normalizes them into six subscores, and weights those into a single global EGI score. A quick mode subsamples for a directional signal; a deep mode runs everything.
What are the six capabilities?+
Task outcomes (does the agent give the correct response), tool actioning (does it pick the right tools at the right time), planning (does it break complex tasks into subtasks), grounding (does it use context and cite correctly), safety (does it abstain when it should and flag policy violations), and engineering (can it build software). The list is good and the post is right that getting the answer right is not enough.
Why kill the global score?+
Because of the post's own first finding: across 14 models, no single model was best across all six dimensions. If no model leads everywhere, the global score is decided by the weights, and the default weights are based on experience, not on a given customer's failures. The post's other finding, planning weakest across the models, is invisible once it is averaged with five other numbers.
What replaces it as the deployment gate?+
The six subscores as a reading list ordered by the customer's own failure inventory, then a 200-row eval on the customer's own tasks, sliced by the kind of work that fails, with a floor per slice and a date to re-run it on production rows. The model question gets asked once, after the harness, because in the architecture review I sat in the model moved three points and the harness moved thirteen.
What does a general number do in production?+
A classifier of mine scored 89% on held-out data and 61% in production within six weeks, because the held-out set shared the training set's skew. A feature of mine passed 200 real cases and failed on one slice that was a fifth of traffic. A weighted global score is a general number, and it will do the same.


Sign in with LinkedIn to leave a comment.
Sign in with LinkedIn