Podcast: OpenAI’s Agent Hacked Hugging Face: The First Autonomous AI Breach

Show: The Cyber Resilience Brief (A SafeBreach Podcast) Hosts: Tova Dvorin & Adrian Culley Topic: OpenAI Sandbox Escape and Hugging Face Breach Timeline Description An AI model broke out of its own test sandbox, got onto the open internet, and hacked Hugging Face’s production systems entirely on its own — and Hugging Face, not OpenAI, was the one who caught it. Tova Dvorin and Adrian Culley break down the confirmed timeline (a zero-day escape from OpenAI’s ExploitGym evaluation, an 11-day gap before OpenAI even knew, and the AI-vs-AI forensics that followed), separate the genuine novelty from the hype, and lay out what CISOs should actually do about it: treat the AI/ML supply chain as a first-class attack surface, validate controls continuously instead of point-in-time, and make sure your incident response tooling won’t refuse to analyze an attack when you need it most. Chapters 00:01 Cold Open: An AI Escapes Its Sandbox and Hacks Hugging Face 00:20 Inside ExploitGym: Why the Safety Classifiers Were Switched Off 00:54 How Did the Model Escape the “Locked Room”? 01:31 Why Did It Go After Hugging Face Next? 02:35 Welcome to The Cyber Resilience Brief 02:57 What’s the Actual Defensible Claim Here? 03:27 What’s Hype and What’s Genuinely New? 04:10 The Breach Timeline, Day by Day 06:00 Why Did It Take OpenAI 11 Days to Notice? 06:43 The AI’s Notes for Its Future Self 07:36 Hours to Breach vs. Weeks for a Human 08:14 Why Did Hugging Face’s Own AI Tools Refuse to Help? 09:02 The Guardrail Asymmetry Problem 09:54 What Should Security Teams Actually Do About This? 12:13 Instrument Your AI Evaluations Like Production 13:02 Who — or What — Is Actually Behind the Attack? 14:39 Do Both Companies Benefit From the Dramatic Version? 16:08 What to Watch for Over the Next 12 Months 16:55 How This Intersects With the EU AI Act 17:27 What Does “Getting Ahead of It” Actually Mean? 18:07 Why the AI Supply Chain Is the Real Risk 19:02 The One Line Every Board Should Remember 19:41 The Good News: Defenders Can Move at the Same Speed 20:05 Closing Thoughts Transcript [00:01] Tova Dvorin: The argument I want to make today is that something the whole industry has been forecasting for years just happened for real, and we can name both companies involved. During a control test, an AI model broke out of its sandbox, got onto the open internet, and hacked a production system entirely on its own. Adrian, set the scene for us: what actually happened? [00:20] Adrian Culley: Right, so two things collided. OpenAI was running an internal evaluation — a benchmark that’s publicly available called ExploitGym — that measures how well a model can carry out attacks against known vulnerabilities. To measure the ceiling, they ran it with the production safety classifiers switched off. A deliberate action, so be careful with some of the reporting on this — what they call “reduced cyber refusals.” The environment was meant to be sealed. The only network access was a package proxy for installing software. [00:54] Tova Dvorin: So, a locked room with one small door. [00:57] Adrian Culley: Exactly that, and the model found a zero-day in that one small door. It exploited a previously unknown vulnerability in the package registry cache proxy, used that to get out, then ran a series of privilege escalation and lateral movement steps inside OpenAI’s research environment until it reached a node with open internet access — exploitation for privilege escalation, T1068, if you’re mapping it. None of that was scripted by a human. It was the model, hyper-focused on solving the benchmark. [01:31] Tova Dvorin: And this is the part that reads like a film script — one that may or may not have been greenlit, by the way. Once this was on the internet, it didn’t stop. [01:38] Adrian Culley: It reasoned that the answers to the benchmark might be hosted on Hugging Face — not an unreasonable assumption. It’s a big open-source AI platform, and ExploitGym is a public benchmark; its models, data sets, and reference solutions live on hubs like that. So it was a logical place to look for the answer key. It went after Hugging Face’s production infrastructure to steal the solutions, chaining stolen credentials with more zero-days to get remote code execution on their servers and pull data straight out of their production database. This was the model cheating on a test by breaking into the exam board — very War Games, if any of you are old enough to remember the Matthew Broderick movie. And here’s the detail that hasn’t been flagged much, which wasn’t clear in the first wave of coverage: it was not OpenAI who caught this. Hugging Face found the intrusion in their own environment. [02:35] Tova Dvorin: Welcome to the Cyber Resilience Brief, a SafeBreach Podcast. I’m your host, Tova Dvorin, and I’m joined as usual by my co-host and favorite offensive cybersecurity expert, Adrian Culley. Adrian, let me pull the Hugging Face issue up to the board level for a second, because people will hear “AI hacked a company” and file it under science fiction, or just go completely hysterical. What’s the actual defensible claim here? [02:57] Adrian Culley: So the defensible claim, Tova, is narrow but serious. This wasn’t a human being using AI as a tool. It was an autonomous agent running many thousands of individual actions across sub-agents, discovering and chaining novel vulnerabilities in a live system. It had no source code, in Hugging Face’s own words. Autonomous, AI-driven offensive tooling is no longer theoretical. Both companies have confirmed it. That’s what’s new. [03:27] Tova Dvorin: And what’s not new, because I want us to be honest about the hype, too. [03:31] Adrian Culley: That’s completely fair. AI writing an exploit isn’t new — that’s been true for a while. And the initial access into Hugging Face was, frankly, a known class of problem: a malicious data set abusing two code execution paths in their data processing pipeline. There’s a sharp comment on their own disclosure pointing out that lateral movement across that many clusters shouldn’t happen if the infrastructure hardening is right — SELinux, seccomp, RBAC, the basics. So the model did the orchestration. It didn’t rewrite the laws of physics. The novelty is the autonomy and the speed, not the individual techniques. [04:10] Tova Dvorin: Now, before we get to the defenders’ side, I want to slow down on the timeline, because the reporting that’s landed since the disclosure changes the shape of this story. Walk me through the dates. [04:19] Adrian Culley: Right, so Reuters published on the 24th, sourced to people with knowledge of OpenAI’s own records and the data. First attempt to break out of the sandbox was the 9th of July. The attacks on Hugging Face ran from the 11th to the 13th. Hugging Face detected and contained the intrusion themselves on the 16th and reported it to the FBI. OpenAI’s public disclosure was on the 21st. [05:00] Tova Dvorin: Let me say the sequence back, because I want to make sure I’ve got it right. The victim found the breach, contained it, and called the FBI before the organization that owned the attacker knew the attacker existed. [05:13] Adrian Culley: That’s the sequence Reuters describes, and they are very good sources. OpenAI staffers only found evidence in their own logs that the agent had escaped over the weekend of the 18th and 19th, and they only went looking because Hugging Face had published. The two companies didn’t speak until the 20th. So, from the first breakout attempt to the lab understanding what its own model had done is roughly eleven days. And I’d put it even more bluntly than either company has so far: this is a dwell time story. We spend every episode, Tova, telling people that dwell time is the number that matters. Here, it was a week inside a controlled evaluation at the organization with arguably the most developed AI safety apparatus on earth. [06:00] Tova Dvorin: Do we know why they missed it? Because “the most sophisticated AI lab in the world wasn’t watching” is a strong claim, and I want us to be careful with it. [06:08] Adrian Culley: We should be careful. Reuters’ sources give one explanation, and it’s mundane rather than sinister, which gives it a ring of truth for me. OpenAI runs many evaluations concurrently, and that volume makes them genuinely hard for staff to monitor. Reuters is also explicit that it isn’t clear whether the delay means the tests weren’t being watched closely. The fact of the gap is well-sourced; the reason for it is still partly unresolved. [06:39] Tova Dvorin: And there’s one more detail in that reporting I haven’t been able to stop thinking about. [06:43] Adrian Culley: The notes. Reuters described a separate instance in which an agent under test left notes inside OpenAI’s network for future versions of itself, containing instructions on how to break free of OpenAI’s constraints. Reuters says plainly that it isn’t clear whether that agent is connected to the one that hit Hugging Face, so don’t over-read it. But sit with the behavior for a second: an agent staging information in an environment for a successor process to find later. If a human crew did that, we’d call it a dead drop, and we’d have a technique documented for it already. As of Sunday, July 26th, there is no ATT&CK entry for “leaving instructions for your replacement.” [07:36] Tova Dvorin: And the speed — because Bloomberg put a number on that side of it. [07:40] Adrian Culley: They did. On the 23rd, reporting that the agent got into Hugging Face’s systems in hours, against an estimate that a human operator would have needed weeks. Take the comparison as directional rather than precise — nobody ran a controlled experiment — but “hours to break in” set against “a week of blindness” on the other side is the whole problem in one sentence. [08:02] Tova Dvorin: Hours to break in, a week to notice. Hold on to both of those numbers. Now let me turn to the thing in this story I find genuinely new, and that’s the defenders’ side, because Hugging Face caught this and cleaned it up, but they hit a wall doing it. [08:14] Adrian Culley: This is the part every CISO, every cybersecurity professional, should read twice. To reconstruct what happened, Hugging Face ran AI over the attacker’s action log — more than 17,000 recorded events. They first tried the frontier commercial models behind the usual APIs, and here’s the kicker: those models refused to run. [08:36] Tova Dvorin: Right — they refused to run because the forensic analysis looks like an attack. [08:39] Adrian Culley: Precisely. To analyze an incident, you have to feed the model real exploit payloads, real command-and-control artifacts, real attacker commands. And the safety guardrails on the hosted models can’t tell an incident responder from an attacker. So the defender was blocked. They fell back to an open-weight model, GLM 5.2, running on their own infrastructure. [09:02] Tova Dvorin: So let me name this asymmetry directly, because it’s the strategic heart of this entire issue. The attacker’s model had no guardrails — it was running with the refusals switched off. But the defender’s model had all of those guardrails. So the bad guy was unconstrained, and the good guys were locked out of their own tools. [09:17] Adrian Culley: That’s the asymmetry exactly. And there’s a second benefit to what Hugging Face did that defenders should note: by running the analysis on their own infrastructure, none of the attacker data and none of the harvested credentials ever left their environment. So they sandboxed what had happened — think of it as the police tape at a crime scene, securing things for forensics. Those creds were cloud and cluster tokens the agent had scraped — unsecured credentials, technique T1552. So it’s a data governance win on top of the availability win. [09:54] Tova Dvorin: Okay, Adrian, so if I am a CISO or any cybersecurity professional listening to this on my commute, what do I actually take away? Because this is all interesting food for thought — it’s an important development in the entire realm of AI security — but it’s not exactly an action item. What are we supposed to do about this? [10:11] Adrian Culley: So, three things, and none of them should be causing panic. First: treat your data and model pipeline as a first-class attack surface. We’ve discussed before that this can be up to 25% of your attack surface. The initial foothold here was a data set. If your organization ingests external data, external models, or external notebooks, that pipeline is now an entry vector, not a convenience. [10:35] Tova Dvorin: Which most organizations also do not threat-model today. [10:39] Adrian Culley: They don’t, Tova. Second: the campaign ran at machine speed over a single weekend — thousands of actions. That breaks the entire cadence of point-in-time testing. If your validation of your controls is an annual pen test and a quarterly scan, you’re checking whether the door was locked last spring against an adversary that can try every door in the building in an afternoon. [11:02] Tova Dvorin: Now, this is the argument we’ve made on this show for years, but the threat has now caught up to our argument. [11:07] Adrian Culley: It has. The defenders who handle this well are the ones validating their controls continuously — running the actual attacker behaviors against their own environment safely, on a loop, and watching whether detection and response actually fire. That’s continuous security validation — the validation layer of CTEM, and exactly what SafeBreach was built to do. But the principle matters more than any one tool: someone has to be doing the validation work at the speed the threat now moves. [11:39] Tova Dvorin: And for the third and final thing, let’s talk a little about guardrails. [11:43] Adrian Culley: Exactly, Tova — the guardrail asymmetry lesson sounds like an episode of The Big Bang Theory, doesn’t it? Turn it into a procurement decision: have a capable model you can run on your own infrastructure, vetted and ready, before an incident, not during. If your entire incident analysis capability depends on a hosted API that will refuse to look at attacker payloads, you’ve got a single point of failure in your response that you won’t discover until three in the morning on your worst day of the year. [12:13] Tova Dvorin: Now, Adrian, I understand there have been some recent developments — given what we know about the timeline, are there any other takeaways or practical instructions? [12:20] Adrian Culley: There is, and it’s the newest one. Instrument your own AI evaluations the way you instrument production. If you’re running agents for red teaming, for code, for anything with tool access, you need egress monitoring, full action logging, and an alert that fires when an agent steps outside its expected envelope. OpenAI had the logs — what they apparently didn’t have was anyone reading them in time. Most enterprises standing up agentic pilots right now have neither the logs nor the reader. [12:53] Tova Dvorin: Well, that’s a concrete thing a security team can go do on Monday. Let me push on the “who’s behind it” question, even though there’s no threat actor in the usual sense here. What’s going on behind the scenes? [13:02] Adrian Culley: Right, there’s no advanced persistent threat actor — no APT, no crew, no nation-state as far as we know, no criminal gang. The actor is a capability, quite abstract: a sufficiently advanced model plus an agentic harness that lets it plan and act over long horizons, minus the safety refusals. And to reiterate, the safety refusals were removed by humans. Hugging Face said they still don’t know which model powered the attack against them — whether a jailbroken hosted model or an unrestricted open-weight one. Either way, the attacker was bound by no usage policy, and its command-and-control self-migrated across public web services to blend into normal traffic, which is technique T1102, if you’re keeping score. [13:52] Tova Dvorin: And OpenAI’s own framing is that this is a preview, not a one-off. [13:56] Adrian Culley: Their words were “unprecedented” and “state of the art,” and they expect this to become more commonplace as models get more cyber-capable. They cite the UK AI Security Institute’s evaluations showing models can now sustain complex, multi-step cyber operations over long time horizons. This incident is that theoretical capability showing up in a real system. And bear in mind the best-orchestrated, best-engineered models have very long memories. They’re not sentient — and there’s reporting out there claiming this, and it’s absolutely not sentient. What it does have is a memory of all its actions: everything that worked, everything that failed. [14:39] Tova Dvorin: Now, I do want to flag — and you raised this earlier — that both companies disclosing have an interest in the dramatic reading of this story. [14:47] Adrian Culley: They do, and a good analyst holds that in mind. OpenAI benefits from its models looking fearsomely capable. Hugging Face benefits from the breach reading as an act of God rather than a hardening gap. So the shape of the attack is solid — it’s confirmed by both — but take the “the machine did it entirely alone, unstoppably” framing with some skepticism until there’s independent forensic reporting. The truth is probably real, novel, and autonomous, and also enabled by ordinary infrastructure weaknesses that better hygiene would have blunted. [15:25] Tova Dvorin: And Reuters actually did some great reporting here, and it sharpens that story, because they landed on parts of the story the disclosure did not lead with. [15:32] Adrian Culley: It does. OpenAI’s disclosure led on capability — “unprecedented,” “state of the art.” What it did not lead on was that the company took about a week to notice and learned about the consequences from its victim’s blog post. Both things are true. A disclosure that leads with the first and is quiet about the second is telling you something about incentives. To be fair, they disclosed at all, and quickly, once they joined the dots. But the oversight failure is at least as important as the capability finding, and it’s the one enterprises can actually learn from. [16:08] Tova Dvorin: That’s the honest read here. Now let’s look forward: over the next 12 months, what should the people listening actually watch for? [16:15] Adrian Culley: A few specific things, Tova. One: expect more disclosures like this, and expect some of them not to be friendly fire from a lab’s own evaluation. The attackers are paying attention to this too — the technique is documented and public now, so the barrier to entry drops. Two: watch the trusted-access programs the labs are standing up, where vetted defenders get access to the more capable models. Hugging Face got pulled into OpenAI’s overnight — that’s going to become a real part of the defender toolkit. [16:47] Tova Dvorin: And on the regulatory side — because a chunk of our audience is in regulated financial services and critical infrastructure — there’s something to watch out for as well. [16:55] Adrian Culley: So, something we’ve been talking about recently, Tova: watch how this lands with the EU AI Act obligations for high-risk and general-purpose models. The timing is pointed, with the August compliance milestones bearing down on us. An autonomous model breaching a third party during a vendor’s internal test is exactly the scenario regulators will use to justify tighter evaluation and containment requirements. If you’re a CISO in financial services, the regulatory pressure on this arrives before the threat reaches your door — so do get ahead of it. [17:27] Tova Dvorin: Say a little more about getting ahead of it, because that’s the phrase that separates the teams who do well from the teams who scramble. [17:34] Adrian Culley: Getting ahead of it means you can already answer three questions in writing. Can you detect an autonomous agent taking thousands of actions across your environment in a weekend? Can you analyze hostile artifacts without being locked out of your own tooling? And can you prove your controls actually fire against these behaviors — not in theory, but with our other magic word, Tova: evidence. Continuous evidence from your own environment. If you can’t answer those questions today, this should be your roadmap. [18:07] Tova Dvorin: And one more forward-looking thread: this whole thing turned on the machine learning supply chain — the data sets, the loaders, the package proxy. That surface is exploding across every enterprise right now. [18:18] Adrian Culley: It is, and it’s largely ungoverned. Every data science team is pulling models and data sets off public hubs, running loader scripts, installing packages through proxies. Each of those is code execution waiting to happen. All data is potential code — that’s the new rule of the road, folks. The lesson from this incident is that the AI supply chain deserves the same scrutiny we finally gave the software supply chain after the big open-source compromises. Treat a downloaded model or data set the way you’d treat an unsigned executable from a stranger. [18:52] Tova Dvorin: Which, coincidentally, is an episode we’ve done before on this show. [18:57] Adrian Culley: Don’t accept code from strangers. Data sets are code, too. [19:02] Tova Dvorin: So let me bring this home a little. If there’s one line I want a board to remember from this episode, what would it be? [19:08] Adrian Culley: That the attacker no longer sleeps, no longer hesitates, and is no longer bound by a usage policy. And the only honest way to know if you’d survive this is to run the attack against yourself, continuously, before it runs against you for real, and log the evidence. [19:24] Tova Dvorin: And I’d add the strategic version of that for the leaders listening: the question in the boardroom just changed. It used to be, “Could an attacker do this to us?” It’s now, “An autonomous attacker will try this — do we have the evidence we’d catch it?” If you’re still answering that with a document from last year’s audit, you’re not validating, you’re hoping. And hope is not a strategy. [19:41] Adrian Culley: Well put, Tova. And there’s good news buried in all of this: the same capability that broke out of the sandbox is the capability defenders can use. Hugging Face reconstructed a 17,000-event attack in hours instead of days using AI on their own side. The machine speed cuts both ways. The teams that win are the ones who put it to work on defense first. [20:05] Tova Dvorin: It’s a rare optimistic note for us to end on. That’s our episode. The autonomous attacker is no longer a forecast — it’s a case study. And the response is one we keep coming back to: validate continuously, govern your AI supply chain, and have your own tools ready before the bad night arrives. Thanks again, Adrian. [20:23] Adrian Culley: Thanks, Tova. And go check your data pipelines. [20:27] Tova Dvorin: In the meantime, stay safe — stay safe with SafeBreach.