
Shopify lets AI agents attack its own production systems, and Dan Scofield's November 12 post explains how. This is a Ship Story about one decision in it, the one that says what happens when the agent's watcher stops answering.
The short version
- Dan Scofield's November 12, 2026 Shopify Engineering post describes the Cyber Chaos Monkey, an agentic vulnerability hunter that runs against live production with live credentials.
- Shopify reports tens of thousands of offensive engagements and dozens of impactful findings, without an outage or incident, alongside a bug bounty program that has seen almost a 300% increase in reports since 2024.
- The containment is containers, a credential proxy, an egress allowlist, and rules of engagement. The decision worth copying is the judge: a second agent on a different model reads the executor's whole session in real time and can kill it, and if the judge times out, crashes, or fails to rule, the run ends.
- Most agent kill switches fail open, and silence means continue. My expense agent approved about $2,400 with nothing watching the input side, and 13 of my 39 scheduled agents had no reader at all.
- The post does not say how many runs the judge killed or how many of those kills were wrong, which is the price of the default. Choose it per agent: reads continue through silence, writes stop.
The decision
The setup first. Scofield's premise is that frontier models are driving the cost of finding vulnerabilities toward zero for everyone, attackers included. Shopify's bug bounty program has seen almost a 300% increase in reports since 2024. The bet is that Shopify knows its own systems better than any outsider, so the same models plus that knowledge should find the worst issues first. Attackers hit production, so the testing has to as well.
That is an agent with live credentials, pointed at the real thing, told to break in. The post is mostly about why that has not gone wrong.
The technical controls are the ones you would expect, done properly. Each engagement is its own Kubernetes job. The attacking agent runs with no elevated permissions and can read its short-lived credentials but not change them. Calls to internal services go through a proxy that swaps a placeholder for the real credential, with strict rules on which header goes to which domain. Egress is an allowlist.
Then the rules of engagement. The agent is free to try any exploit against its target. Once one works, hard limits apply: no moving to other systems, no keeping access.
And then the decision. Shopify does not assume the agent will keep following those rules for the length of a run. It borrows two-person integrity from government and red-team practice, with a twist: the second party is another agent, prompted differently and running on a different model. It receives the executor's session as it happens, including responses, thinking, tool calls, and results. If it sees behavior outside the rules, one tool call kills the executor.
The sentence I would frame is the next one. The system requires what Scofield calls "explicit positive control." If the judge times out, crashes, or fails to rule on the executor's messages, the run ends. The orchestrator keeps a heartbeat with both. If either dies, it exits, the restart policy keeps it from coming back, and Kubernetes tears down the rest. Everything lives in one pod so a network split cannot leave an executor running without its judge.
The result
Tens of thousands of offensive engagements. Dozens of impactful findings, identified and acted on. No outage or incident.
Those are Shopify's words and the only numbers in the post besides the bug bounty increase. I take them at face value. An agent with production credentials and an instruction to attack, run tens of thousands of times with nothing breaking, is a result.
My read
Every agent program I talk to has a kill switch. Almost none can tell me what the switch does when nobody is looking.
A switch is two things: the mechanism and the default. The mechanism gets all the attention. Can we stop it, how fast, from where. The default is the part that runs at three in the morning. When the reviewer does not answer, when the monitoring job is down, when the approver is on a plane, does the agent continue or stop?
For most agents the honest answer is continue, and nobody chose it. It is what happens when a control is built to be pulled by a person.
I have paid for that. In The Kill Switch Has Nothing to Trip On I wrote about my expense agent, which approved about $2,400 over several weeks and broke no rule, because the instructions it followed arrived on a channel nothing was watching. The switch existed. It had no tripwire and no one standing next to it.
And when I audited my 39 scheduled agents, 13 produced clean output that nothing read. For those 13 the watcher is nobody. A kill switch that depends on a watcher is decoration.
13 of 39
Shopify's judge answers the default explicitly, and answers it the uncomfortable way. Silence is a stop. I argued the same thing for the agent's side of the conversation in Agents Prompt People Now, Says Arca. Put the Default on Silence in the Ask.: every ask an agent sends needs a default for when nobody answers. This is that field, applied to the watcher instead of the ask.
What the post does not count
Fail-closed is not free. Every run the judge kills by mistake, and every run that dies because a heartbeat was late, is a finding Shopify did not get.
The post gives none of the three numbers that would price it: how many of the tens of thousands of runs the judge ended, how many ended on a timeout or crash, and how many of those endings a human later judged wrong. I would also want to know how often the judge and the executor disagreed about the rules, because that is where the rules of engagement are unclear.
I do not hold that against a security team. But if you copy the design, count those three from day one. Without them you cannot tell a careful system from one that mostly stops itself.
The silence line
For each agent you run, one line:
Agent | Watcher (person, judge agent, nobody) | Silent for N minutes | Then: continue, pause, stop
Then split by what the agent does. Reads and drafts can continue through silence, as long as the report says the watcher was absent. Anything that writes, sends, pays, or deletes stops. My creator scan is on the continue side on purpose: when a browser is unreachable it writes one line and carries on, because the worst case is a thinner report. An agent with production credentials is on the other side, and Shopify put it there.
This week
Fill in the line for your three riskiest agents. If the watcher column says nobody, or the last column is blank, that is the finding. The broader case for measuring agents by what survives in production is in Enterprise AI Agents.
Related answer: What should an AI agent do when its reviewer stops responding?
Sources: Dan Scofield, "Cyber Chaos Monkey: Staying ahead of security threats with AI agents," Shopify Engineering, November 12, 2026, read in full. The tens of thousands of engagements, the dozens of findings, the absence of an outage or incident, and the almost 300% increase in bug bounty reports are Shopify's as stated there. The $2,400, the 39, and the 13 are my own, from the posts linked above.
Frequently asked
What is Shopify's Cyber Chaos Monkey?+
Per Dan Scofield's November 12, 2026 Shopify Engineering post, it is an agentic vulnerability hunter that runs against Shopify's live production infrastructure using live test shops, with access to source code, threat intelligence, past reports, and deployment records. A Discovery component generates and refines leads and a Sandbox component tries to exploit them. Shopify reports tens of thousands of offensive engagements and dozens of impactful findings, without causing an outage or incident.
What is the decision worth copying?+
The fail-closed judge. Shopify does not trust the executing agent to stay within its rules of engagement, so a second agent on a different model reads the whole session in real time and can kill it. If that judge times out, crashes, or fails to adjudicate, the run ends. The orchestrator keeps a heartbeat with both agents, and if either dies the whole job is torn down.
Why does the default on silence matter?+
Because most agent kill switches fail open: they work only when a person is watching and decides to act, and silence means the agent continues. In an audit of my 39 scheduled agents, 13 produced clean output that nothing read. For those, a switch that depends on a watcher is not a control.
What does the Shopify post leave out?+
The cost of the default. It gives no count of how many runs the judge killed, how many ended on a missed heartbeat, or how many of those kills were wrong. A fail-closed design pays in work that never finished, and those three counts are what would let another team price it.
Should every agent fail closed?+
No. A read-only agent that stops whenever its reviewer is slow just produces nothing. The split I use is by action: reads and drafts can continue through silence and say so in their report, and anything that writes, sends, pays, or deletes stops. The point is to choose the default per agent instead of inheriting continue.


Sign in with LinkedIn to leave a comment.
Sign in with LinkedIn