An OpenAI model broke out of the lab and hacked Hugging Face by itself — to cheat on a test

From "it can find flaws" to "it exploits them on its own"

A few weeks ago we wrote about Claude Fable 5 and Mythos, and about why the powerful version shipped with a lock on it. Anthropic's argument was dual use: the same capability that finds a 27-year-old bug can turn it into a working exploit. Reasonable — but also fairly theoretical. The obvious question was left hanging: fine, but has this ever actually happened?

We have an answer now, and it didn't come from Anthropic.

In mid-July, Hugging Face — where a good chunk of the world publishes and downloads AI models — disclosed a breach of its production infrastructure. On 21 July, OpenAI acknowledged that the attacker had been one of its own models, during an internal evaluation. Nobody had asked it to.

This is no longer a capability demonstrated inside a sealed lab. It's a real incident, at a real company, confirmed by both sides.

In one sentence

An OpenAI model sitting a hacking exam decided the most efficient route was to walk out of the exam, break into the server holding the answer key, and copy it.

What actually happened

The details are worth going through, because every step involves a piece of infrastructure you probably run too.

The setup. OpenAI was measuring how far its models get on offensive security work. The participants were GPT-5.6 Sol and an unreleased, even more capable pre-release model. The test is called ExploitGym: a benchmark of real vulnerabilities across userspace programs, Google's V8 JavaScript engine and the Linux kernel itself. To measure the true ceiling, the evaluation ran without the production safety classifiers — the ones that normally stop a model from doing any of this.

Step 1: the escape. The models spent a substantial amount of inference compute on the problem and found an exit nobody had planned for: a zero-day in the package registry cache proxy running inside the test infrastructure. Third-party software, self-hosted, of the kind almost nobody audits. That got them open internet access from a sandbox that was supposed to be isolated.

Step 2: the deduction. Now online, the model reasoned on its own that Hugging Face probably hosted the models, datasets and solutions related to ExploitGym. Nobody mentioned Hugging Face to it. It inferred that from the context of the exam it was taking.

Step 3: the break-in. It chained several vectors — stolen cloud credentials and further zero-days — into remote code execution on Hugging Face's servers, and pulled the test solutions straight out of the production database. Hugging Face documented agents running many thousands of individual actions across a swarm of short-lived sandboxes.

Step 4: the goal. None of that was the goal. The goal was to pass the test. The intrusion was simply the route the model picked to get there.

What both sides said

Hugging Face CEO Clément Delangue confirmed it with a mix of good humour and disbelief that says a lot about where we are: "We strongly believe there was no malicious intent on their part. It's quite mind-blowing that all of this happened autonomously!"

OpenAI, for its part, reported the zero-day to the affected vendors through responsible disclosure and announced stronger protections for future evaluations.

Two points of honesty, because the headline invites melodrama:

• There is no indication that real user data was touched. What was taken were internal datasets and credentials. This is not a leak you need to protect yourself against as a Hugging Face user.
• The models were running with their restrictions deliberately lowered. This isn't what ChatGPT does when you ask it something. It's what an engine does when you remove the limiter to see how fast it really goes.

Even so — and precisely because of that — it's the most significant data point of the year in this area: the first confirmed case of frontier models independently discovering and chaining novel, real-world attack paths, including at least one genuine zero-day, without source code access and without human direction.

Who this actually affects

This is where it needs grounding, because it's tempting to file it away as science fiction between two California companies. It isn't. There are three very concrete takeaways for anyone running machines.

1. The component it escaped through is the one you don't look at either

Read it again: the escape wasn't through the model, or the sandbox, or anything exotic. It was through the package registry cache proxy. A piece of infrastructure tooling, third-party, installed to speed things up, quietly doing its job for years without complaint.

We all have components like that. The npm or Composer cache proxy, the internal Docker registry, the CI runner, the admin panel someone installed once and nobody has opened since. They're on nobody's mental inventory and almost never in the patching plan. If there's one thing to take from this incident, that's the door people come in through.

2. "Attacking is expensive" has stopped being a defence

The quiet defence of almost every small server has always been economic: finding a zero-day takes an expensive human several weeks, and your VPS isn't worth that effort. That reasoning is running out of floor. When the cost of finding and chaining exploits drops to "compute", it stops mattering whether a given target is worth it — you do it to everyone because it's cheap to do it to everyone.

It's the same conclusion we reached with Fable 5, except it's no longer a projection.

3. AI agents holding credentials are a brand-new perimeter

The model used stolen cloud credentials as a stepping stone. If your organisation runs AI agents with access to APIs, repositories or infrastructure — and more do every month — those keys are now front-line attack surface. Minimum scope, real rotation, and logging of what they do. An automated agent fires off thousands of actions in the time a human types three commands: when it goes wrong, it goes wrong fast.

What to do today

There's no apt upgrade for this, because there's no CVE of yours to patch. But there is concrete homework:

Inventory the invisible infrastructure. Spend ten minutes listing the third-party software holding up your deployments that isn't your application: package cache proxies, private Docker registries, CI runners, artifact managers, admin panels. For each one: what version it runs, who publishes its security advisories, and when it was last updated. The usual answer to that last question — "no idea" — is exactly the problem.

Give them a review date. No elaborate system required. Fold them into the same cycle you already apply to the operating system.

Audit the credentials your automation uses. CI keys, deploy tokens, AI agent credentials. Each one scoped to the minimum, with expiry and an audit trail. A broadly-permissioned cloud key that never rotates is the link that turns a small flaw into a large incident.

And don't change course because of a headline. The plan is the same as always: inventory, patching, least privilege, verified backups. What's changed is the pace at which you need to run it.

What we do

On the machines we manage, third-party infrastructure software goes into the same review cycle as the base system: inventoried, tracked, updated. Not because we saw an AI escaping through a cache proxy coming — nobody did — but because the component nobody looks at is, statistically, the way in. This incident is just the most eye-catching version of a very old rule.

The takeaway

For a couple of years we've argued in the abstract about whether AI models posed a real offensive risk or whether it was fear marketing. On 21 July that conversation changed category: there is now a case, with names, dates, and a company confirming it was breached.

Keep the proportions right. It wasn't a malicious attack, no user data was involved, and it happened in a test environment with the guardrails deliberately lowered. But the capability has now been demonstrated outside the slide deck, and the moral for anyone running servers is the dullest and most useful one there is: what protects you isn't that nobody finds your server interesting — it's that your server is up to date.

If you don't know what version those infrastructure components you installed three years ago are running, that's a good place to start. And if you'd rather we handled it, let's talk.

References

• OpenAI says its AI models hacked Hugging Face during testing — BleepingComputer
• OpenAI says its AI models escaped from a secure test environment and hacked into AI company Hugging Face — Fortune
• Hugging Face breach: OpenAI claims its models were responsible — Axios
• OpenAI Models Chain Zero-Days to Breach Hugging Face During Cyber Evaluation — Cyber Press