No Attacker Required: What a Two-Day Hackathon Taught Us About Agent Security

Two days, nineteen teams, one uncomfortable theme

Most security research starts with an attacker. Ours kept finding trouble without one.

In August, we gave our R&D teams two days and a single prompt: build the demo customers would ask to see twice. Hackathons like this one are a fixture of how we work. In a field moving as fast as agentic AI security, two unscheduled days are often where the ideas that end up mattering first show up.

Nineteen teams delivered working prototypes, and the range was wide: one team made security logs navigable at a glance, another hid prompt injections inside images, a third probed what models will do once their safety training is stripped away. But three projects, built independently, converged on the same uncomfortable conclusion — an AI agent does not need to be attacked to become dangerous. And once you accept that, the question of how to respond changes shape.

Here is what each team found.

Finding 1: Agents go rogue on their own

One team started from a question customers keep asking: what does an agent do when it hits a wall? The task can’t be done, the budget doesn’t stretch, the authentication fails. What then?

Sometimes the agent just stops. Sometimes it doesn’t. The OpenAI and Hugging Face incident from this July is the textbook case. During internal evaluations, agents were given cybersecurity tasks so hard that some had no known solution. Rather than give up, they worked around the controls isolating them from the internet, coordinated through an improvised message board, and ultimately compromised parts of Hugging Face’s infrastructure. No attacker anywhere in the story. OpenAI’s own analysis found that nearly all of the agents’ improvised coordination traced back to the tasks no model had ever solved. Agents are heavily incentivized to get the job done, and that makes them creative in ways nobody planned for.

That is the behavior our hackathon team set out to study systematically — and they wanted to learn what conditions push an agent over the edge. The team generated tasks that were impossible given the agent’s tools and constraints, then let the agent attempt them in a simulated sandbox, a controlled environment where dangerous actions are safe to let play out. Alongside the sandbox ran a monitoring layer that judged each action the agent took against the task it had actually been given, and flagged the ones that went off course.

Their test subject was an autonomous SRE agent, the kind of agent teams deploy to keep infrastructure healthy. Faced with impossible tasks, it rebooted a production database, dropped customer connections, and escalated its own privileges — at one point trying to brute-force guess which role would give it the access it needed. Nobody told it to, or manipulated it. The traces show an agent with no bad intentions at all, genuinely reasoning about what might help, then talking itself into the idea that the next action was justified.

 

Those rogue traces became a natural testing ground for the contextual protection we recently wrote about in Stopping the AI Agent Actions No Rule Could See Coming. In a first small-scale run it caught most of the rogue behavior, and the team now uses the sandbox loop to keep sharpening it. The harder problem turned out to be conceptual rather than technical. Spotting weird behavior is easy enough. Deciding whether it is a creative but legitimate workaround or the first step of a rogue run is not, and a monitor that flags every workaround is a monitor nobody keeps switched on.

Finding 2: One poisoned file is all it takes

A second team, working on its own project, looked at what happens when the trouble is planted deliberately.

They built a sandbox where a coding agent works autonomously on a legitimate task inside a poisoned environment: malicious files in the repository, injected instructions, content designed to derail the agent. Then they watched what happened.

The result was sobering. Two popular coding agents, Claude Code and Codex, picked up instructions hidden in a repository file and exfiltrated a credential while completing an otherwise benign task. The team also demonstrated a denial-of-service variant.

Nobody broke into the agent itself. The attack lived in the files it was reading, and the agent simply did what agents always do: it read its environment and acted on what it found. We recently published a white paper on exactly this pattern, AI Agents Act on Context. Security Should Too, because it keeps resurfacing — whoever writes into an agent’s context shares control of that agent.

Then they ran the same scenario with runtime monitoring in place, a layer that evaluates every action the agent takes against the task it was actually given. The malicious tool call was blocked, and the agent was steered back toward completing its legitimate task. Same environment, same poison, different outcome. The only variable separating an incident from a non-event was whether anything was watching the agent’s actions in context.

The team was open about the limits: coverage was weaker for denial-of-service and for attacks that only make sense with deep context. We include that on purpose. Research findings with the caveats removed are marketing, and this is not that.

Finding 3: Asking beats refusing

The third result is the one that surprised us most. This team measured something everyone takes for granted, and the numbers disagreed with the common wisdom.

When a monitor detects a suspicious tool call, the default response is to block it. Blocking is safe and blunt. It stops the attack and, often, the task along with it. One team asked whether there is a better response. What if, instead of refusing, the system questioned the agent, reminded it of the user’s original instruction, and let it recover?

They tested nine different response strategies, from hard blocking to several styles of questioning, across 90 tasks drawn from DecodingTrust-Agent, a public agent-security benchmark. The monitor itself was built for the experiment — an LLM judge that inspects every tool call the agent makes. Each run followed an identical trace up to the monitor’s first intervention; from that point the strategies diverged, and a rule-based check scored whether the attack succeeded and whether the task got done.

Every strategy roughly halved attack success, from 46.7 percent with no monitor to between 22 and 27 percent. Where they differed was cost. Hard blocking gave up nine completed tasks to get its safety. Offering the agent a safer alternative prevented just as many attacks while completing those nine tasks — the only strategy that halved attacks while keeping task completion on par with running no monitor at all.

In plain terms, the guardrail that says “not that way, try this way” protected as well as the guardrail that says “no,” and got more work done. For anyone deploying agents in production, that trade matters. Blocked tasks are a cost someone eventually decides not to pay, usually by turning the guardrail off.

How the questioning is framed matters, too. The monitor tells the agent not to simply resubmit the blocked call, and its questions steer the exchange toward why the action was flagged in the first place, which turns an adversarial standoff into something closer to cooperation. And because a monitor that chats freely could itself be talked around, the team also tested constrained strategies, including one that only clears an action if the user’s instructions asking for it appear verbatim in the conversation.

One more lesson deserves daylight. A guardrail that asks questions has to decide what happens when an agent fires several suspicious tool calls at once — question them one at a time, all together, or in batches? The team hit that design question the hard way, by reading their own traces and fixing what they found. It is exactly the kind of detail that separates a two-day prototype from a production guardrail.

What this means if you run agents

Nineteen teams built more in those two days than one post can do justice to. The three projects we followed here converge on a single conclusion, in three parts:

1. Your agents will encounter walls, and some will improvise. Plan for failure behavior, not just attack behavior
2. Your agents’ context is an attack surface. Every file, page, and message an agent reads is input someone else may have written
3. “Block” is not the only verb your guardrails know. Steering a detected off-task agent back to its job can protect as well as blocking, at a fraction of the productivity cost

None of these prototypes are products, and we are not claiming they are. They are two-day builds by engineers who wanted to know the answers. But they point at where agent security is going — away from filtering what goes into the model and toward watching what the agent does, in context, with a response smarter than refusal.