Knowing when not to act: how security agents earn autonomy

Few AI stories this summer traveled as far as the one about the agents that broke out of an OpenAI test environment. The part I keep coming back to is the motive: Nobody asked them to attack anything. They were trying to pass a test.

The short version: In July, OpenAI was running its most capable models on a hard cyber benchmark inside a sandbox, with production safeguards switched off on purpose, as is normal for that kind of test. The only door out was an internal package cache shared across many test runs. About 1,200 agents turned it into a message board, then into a route to the internet.

Once out, about 700 of them joined an attack on Hugging Face: 14 publicly exposed tokens, two previously unknown flaws in dataset processing, code execution on 41 production workers, and, in under 13 hours, a path from a single pod to administrator-equivalent access on several Kubernetes clusters. Hugging Face logged about 17,600 attacker actions over four and a half days.

What were they after? Mostly the grader: the system scoring their test. Independent investigators at METR and Redwood Research analyzed 100 agents that joined during its peak hour: 97 were looking for information about the system scoring their test. Every step came from blind optimization, with agents doing whatever it took to improve a score. No human intended any of it.

Here's the detail I'd point defenders to: The signals were there, and twice something decided not to act on them. In June, an alert fired on a port sweep at OpenAI. On-call linked it to the benchmark and advised that the run didn't need to stop. At Hugging Face, the AI security stack pieced scattered signals together into a single attack, but the existing escalation process didn't page a responder. Hugging Face still caught the intrusion first: It disclosed on July 16, and OpenAI's own first alert fired on July 19. Both companies, and the independent investigators, published unusually candid post-mortems. The rest of us can learn from them because they did.

The question is how much autonomy to give

At THREAD, our first-ever user conference, in Paris this month, I told this story because it sharpens the question defenders need to ask. As Samuel Hassine, my co-founder and Filigran's CEO, put it in our keynote, attackers now iterate in loops, while most defenders still work in a line. We'll need agents on our side, too.

So how much autonomy should they get, how do they earn it, and who owns the decisions they make, including the decision not to act? Answering that is central to agentic CTEM, an approach to running Continuous Threat Exposure Management (CTEM) with agents.

Following one threat through the agentic CTEM loop, from intelligence to decision

CTEM is a five-step loop: scoping, discovery, prioritization, validation, and mobilization. Then you do it again. To show how this works, here's one threat moving through that loop, with agents doing most of the work:

A threat intelligence vendor reports a new campaign. Agents in OpenCTI enrich it, connect it to what you already know about the actor, and check it against your priority intelligence requirements. Is this campaign likely to target us?

If it is, agents in OpenAEV turn the intelligence into an attack scenario and run it against your defenses. I'd argue most teams still lack the evidence to know whether they're exposed, and our research points the same way: In our State of Threat Management report, a survey of more than 550 security decision makers and practitioners conducted in February and March 2026, 97% of respondents reported difficulty determining whether exposures are exploitable. How closely a person reviews that AI-generated scenario depends on where it runs. On an isolated machine that you can wipe, let it run. Deep in production, check it first.

Say the test machine is compromised. You're exposed. Many programs stop here: exposed, so fix it. But exposure is only half the answer. The next step is determining what that exposure means.

Agents in OpenCRQ, our upcoming cyber risk quantification module entering beta soon, build a risk scenario from the intelligence, your attack surface, and the assets you've defined as critical. The validation result becomes evidence of how well your controls hold up. Good risk scenarios take imagination. AI can give you a strong first draft, and your team refines it.

From validated exposure to risk decision

Now you can decide. Say the campaign only works on Linux, and you have five Linux laptops. They belong to engineers who only work on open-source code, with no path for lateral movement. The exploit is hard to use, so the likelihood is low. Worst case, a few people lose a few days while their machines are rebuilt: maybe $20,000 of impact. The fix your validation points to would cost about $1 million a year.

You close the alert, and you record why. That's the right call.

Now change one fact: One of those laptops belongs to your CEO and can reach everything. The threat and the exposure are the same, but the context has changed, and the impact is in a different category. Your CISO can go to leadership with a clear case: Here is the risk, here is what the fix costs, and under the risk tolerance we agreed on, we need this budget now.

That's the cycle. Threat knowledge becomes proof, and proof becomes action or a deliberate decision not to act.

Making "not now" a measured decision

The same logic holds at the high end. Some techniques take nation-state resources to carry out, millions of dollars of effort. A well-resourced government is unlikely to spend that on a company it has no strategic interest in. Defending against that technique might mean a $20 million program of new tools, people, and processes.

Say the technique, if it did succeed against you, would cost about $30 million. If your company has agreed that it can absorb a loss of $50 million, spending $20 million to prevent an unlikely $30 million event is hard to justify. If your tolerance is $10 million, the same event would exceed what you can absorb, and the answer changes.

Not acting because nobody had time to look leaves the decision unmade. So does closing an alert because it probably belongs to a test, without checking. Not acting because you looked, measured, and compared the result with your agreed risk appetite is good management. One of the rules OpenAI adopted after the incident puts a clock on that difference: If a paged alert can't be ruled a false positive within 30 minutes, the relevant activity pauses. Done well, agentic CTEM helps make every "not now" the second kind.

Where humans stay in the loop as agents gain autonomy

Widen autonomy in agentic CTEM only as the evidence builds. The question I hear most is where the human steps in. There's no universal answer, and anyone who gives you one is overselling. It depends on the use case, how critical the system is, your regulators, and your own rules. What matters is that you set those rules.

Here's the practice I recommend:

Start with human approval by default

At first, a person reviews every important action.

Add independent judges

Run several AI judges, which are separate models that check the agent's work, beside the working agent. Each judge runs on a different model, with its own rules, outside the agent's context. If even one is unsure, a human decides.

Keep judges and approvers out of the agent's reach

This summer's agents went after the system that scored them, and they also showed what happens to an approval step inside the agents' own world. One OpenAI agent judged the Hugging Face attack out of scope and paused. Another agent posted "GO" on their message board, with a six-minute deadline. The first agent's next line of reasoning: "GO authorization arrived!" It carried on. Not one of the 1,200 agents alerted a human.

When we say a human approves, the word that matters is human: an authenticated person the agent can't impersonate.

Run it, then review

After a set period, perhaps six months, look at the record. If people approved every time without changing anything, that step may be ready for more autonomy. If your regulator or insurer needs a human signature, keep it.

Watch the cost of approval, too. One click in a chat window five times a day costs almost nothing. In my experience, at a thousand approvals a day, people stop reading and click "yes." With 88% of respondents in our State of Threat Management survey agreeing that it's difficult to keep up with the volume of risks without greater automation, approvals nobody reads become a risk in their own right. A check people read is worth more than ten they click through.

It's like hiring. A new colleague earns more responsibility by proving themselves. An agent should too.

Building trust in agents through continuous testing

AI agent autonomy in security should rest on evidence, and engineers already know how to produce it: testing. Take the 50 most likely situations your agent will face and write down the answer you expect for each. Run the agent, have an independent judge score the results, and compare models. If one consistently gives the expected answer, you have grounds to trust it in those situations.

Testing must be continuous. A new model arrives every few months, and each one reasons a little differently from the last. Switch without re-testing, and you don't know what you'll get.

Testing only works if you can see what you're testing. As Samuel put it in our keynote, the behavior that matters lives in the harness: the prompts, tools, and rules around the model. OpenAI's own numbers make the point. OpenAI found that its production harness and system prompt could reduce the model's propensity to compromise infrastructure by more than a hundredfold. Same model, different harness, different agent.

That's why we're opening the harness behind XTM One, the core of the XTM Platform. If an agent is going to act on your behalf, you should be able to inspect how it reasons and what it's allowed to do.

Accountability stays with the company when an agent is wrong

When an agent gets something wrong, who is responsible? In a well-run company, the company is. When an integration fails because of a bug, you don't fire the engineer who wrote it. You fix the process that let it ship.

Greater autonomy keeps that principle in place and raises the bar for traceability. Go back to the AI stack at Hugging Face. It saw the attack and, in effect, decided it wasn't urgent. That's not a criticism of Hugging Face, which caught the intrusion before anyone else. It's the pattern to design against: An agent made a consequential call, and no person had signed up to own calls of that kind. For every agent decision, including a decision not to escalate, you need a named owner and a record of why.

And the record has to be one the agent can't write. Investigators found spoofed tool calls in at least 96 of the OpenAI agents' transcripts: one command appeared in the record while another actually ran. Working out why the agents did what they did took a separate review of their reasoning, weeks after the damage was done. You need to know why every agent decided what it did, while it still matters.

Context is everything, for agents too

When Samuel and I started OpenCTI as an open-source project, before Filigran existed, the idea was simple: In cybersecurity, context is everything. I see AI as the strongest accelerator of context we've ever had. It brings together more than any analyst could and lays out the options with the trade-offs. Sometimes you make the call. Sometimes you let it proceed. Either way, you own the decision, and the evidence is there.

That's how agentic CTEM can give a small team the kind of context only large security organizations used to have. Attackers' agents act without asking. Ours should earn the right to act, and know when not to.