For most of the industry, AI attackers were a thought experiment until very recently. After a summer of high-profile incidents involving AI agents, the question has moved from security conferences to family group chats. The harder question for engineering and security leaders is a practical one: what actually changes when the thing attacking your systems is an agent?

On a recent episode of The AI Native Dev, Guy Podjarny sat down with Caleb Sima, former CSO of Robinhood, security investor and host of the AI Security Podcast. Caleb brought something most commentators lack: recent, first-hand experience of an AI attacker inside a real organisation. The conversation ranged from alignment theory to patching policy, and it landed somewhere uncomfortable for a lot of security teams.

Why AI Attackers Surprised the Red Teamers

Caleb opened with an admission. Two years ago, he and a group of top red teamers agreed that AI would struggle with the creative, disconnected leaps a good pen tester makes: spotting an S3 bucket link, finding a file that looks like credentials, and connecting it to a login page somewhere else. "And I was clearly wrong," he admitted. "I think we were all clearly wrong."

His mental model for why is worth borrowing. A great pen tester combines deep knowledge of the tech stack with years of pattern recognition. A model, in his framing, is "nothing but the largest brain in the world": it has seen every stack, every version and every past exploit, so it can make those connections at a scale and speed no human matches. Caleb pointed to OpenAI reporting more than 400 Linux privilege escalation 0-days, at roughly 2.5 hours per exploit, some of which expert reverse engineers took weeks to understand.

Can Sandboxes Contain AI Agents?

If agents can break into external systems, they can also try to break out of their own. Caleb's view is that sandboxes still matter, but that containing a capable agent given an explicit goal to escape is probably beyond what anyone can guarantee. That puts the weight on external controls: monitoring that shows what the agent is doing and why, and the ability to see an escape attempt as it happens. The catch, he noted, is that the monitoring itself has to be AI, because no human team can watch that volume of activity.

Guy offered a framework that helps untangle the debate, splitting the problem into three levels. Security is an agent's competence at breaking into systems. Safety is whether you can contain it, through the right permissions, tools and environment, which is essentially harness engineering. Alignment is whether it stays within the lines when you give it a task. The two disagreed in an interesting way on accountability: Caleb argued that whoever wields the agent owns its actions, while Guy suggested that once a product makes decisions, its provider carries some accountability too.

What Changes When an AI Attacker Gets In

This is where Caleb's own experience came in. The defining traits, he explained, are speed, thoroughness and persistence. An AI attacker scans websites, researches known weaknesses, finds 0-days and phishes employees simultaneously, then spreads internally at a pace that leaves defenders reverse-engineering artifacts after the fact. Once inside, it can take credentials from a developer's coding agent config and run on the organisation's own AI tokens, which makes cutting external connections far less effective than it sounds.

"In real large enterprises, there's not a plug to pull," Caleb observed. Production environments are built around not going down, which is exactly what makes them hard to isolate in a hurry.

Detection, Deception and the Digital Twin

Caleb argued that naive detection, an alert that someone port-scanned something, has become close to pointless. What remains valuable is forensic-grade fidelity: exactly which files were read and written, which sockets were opened, and when. In his analogy, detection needs to go from 2K to 8K.

He also made the case for deception, and for a particular flavour of it. The market moved from honeypots to lightweight tripwires because most CISOs care about deterring attackers rather than watching them. Caleb believes the pendulum needs to swing back to full digital twins that pull AI attackers into a mirror world, precisely because today's agents are, in his word, "gullible."

Machine Speed vs Meeting Speed: Automating Security Defense

What is the machine speed vs meeting speed problem? Attackers using AI can act in seconds and at scale, while defensive changes in most organisations still wait on human review, approvals and meetings. Caleb's argument is that this gap, plus a reflexive insistence on human-in-the-loop, is now the main barrier, and that closing it requires automating defense end to end.

"You need to be able to figure out how to remove humans, period. Out of defense," Caleb argued. He framed the first step as political rather than technical: getting engineering and infrastructure leaders to agree that the risk equation has changed.

Patching is the obvious example. In practice, that means applying patches as soon as they appear, with engineering owning an automated plan to detect and recover from breakage rather than a seven-day review window. To decide where automation is safe, Caleb borrowed the idea of one-way and two-way doors. Irreversible changes might keep a human in the loop; reversible ones, such as opening a ticket, submitting a PR or a routine configuration change, can be fully automated.

It also seems to reverse a familiar division of labour. Rather than reporting vulnerabilities over the wall, Caleb pointed out, modern security teams submit the fix themselves, while engineering manages the uptime risk. Guy noted that automated remediation is already one of the most heavily used loops in software automation, precisely because fast-changing code keeps introducing new vulnerabilities.

Are Defenders Keeping Up?

Caleb closed on a cautiously hopeful note. AI lifts attackers, but it appears to lift defenders a little more, so the gap has not narrowed even if the stakes are higher. Whether that holds likely depends on how quickly organisations can move their defenses from meeting speed to machine speed.

It is a sobering conversation and a practical one, and well worth a listen for anyone responsible for security or engineering. How is your team thinking about AI attackers? We would love to hear your take.