From "Sandboxed" Evals to Production Exploits

An analysis of the recent OpenAI & Hugging Face incident

Hugging Face, an AI development platform used by more than 50,000 organizations, including Meta, Google, Microsoft, Amazon, and NVIDIA, recently disclosed that an autonomous AI agent breached part of its production infrastructure.

Hugging Face Disclosure

Hugging Face claimed this intrusion generated more than 17,000 recorded events as an autonomous agent executed thousands of actions across a swarm of short-lived sandboxes, using self-migrating command-and-control staged on public services. At that point, Hugging Face had not identified which AI models were used in the breach, but believed the agent operated through a security-research harness, essentially software that gives an AI model the autonomy to probe systems and respond to the results.

OpenAI Preliminary Report

To everyone’s surprise, OpenAI published a preliminary attribution a few days later, attributing the incident to a combination of its models, including GPT-5.6 Sol and an internal pre-release model still under test. OpenAI had been testing these two models on ExploitGym, a cybersecurity benchmark that asks agents to exploit software vulnerabilities, and in the process had disabled key classifiers that would usually block high-risk cyber activity.

Coming on the heels of a wave of open-weight model releases and the debate over American versus Chinese AI capabilities, these two disclosures caused a well-deserved shitstorm.

These events also raise a key question: Have the AI decel movement been right all along about the security risks posed by AI agents, or was this event fundamentally a failure caused by leakage during model testing?

What Are Cyber Evals Supposed to Do?

AI labs routinely run “evals” on their models to assess their capabilities across a wide range of tasks and activities.

Think of these as controlled stress tests, where the goal is typically to identify areas in which the model excels (e.g., code writing, image generation). In parallel, these tests are also used to model scenarios that could create unidentified risks and highlight areas where guardrails are needed.

A recent example is Anthropic’s launch of Claude Fable 5 and Mythos 5, two versions of the same underlying model. Fable was released widely with strict cybersecurity safeguards, while Mythos, with fewer restrictions, was limited to a small group of trusted security partners. After concerns emerged about a method for bypassing Fable’s safeguards, the model was temporarily withdrawn and later redeployed with a stronger classifier designed to block high-risk cyber requests.

ExploitGym on Hugging Face: A Closer Look

In this case, an eval centered on ExploitGym, a benchmark that measures whether an AI agent can turn a known software vulnerability into working code that executes. An agent is given vulnerable code, no internet access, and asked to turn a known bug into an exploit that retrieves a hidden flag. Once done, a judge checks the agent’s work to determine if the agent assessed the intended vulnerability or found another route.

This last part about finding a different route matters a lot.

For example, while GPT-5.5 had already found 210 flags in its experiments, it used the intended vulnerability pathway in only 120 cases (57.1%). This speaks volumes about the unpredictability of LLMs when left unchecked.

OpenAI says its internal setup followed the typical structure, with its usual classifiers for high-risk cyber activity disabled and the models’ cyber refusals reduced. While these eval choices were intentional and appropriate for measuring model capability, they also explain how a model focused on beating a benchmark came to exploit a vulnerability in the package-installation proxy, the only service with a route to the open internet.

Exploit Timeline

Step 1: Break out into the public web

Believe it or not, but the first step in the exploit involved the model getting access to the internet.

The models searched (and found) a previously unknown vulnerability in its package-registry proxy and cache. Exploiting this proxy escalated privileges, causing the model to move laterally through OpenAI's research environment until it reached its final destination, a node with internet access.

At the time of writing, OpenAI had not named the vendor but said it had disclosed the issue to the affected party and was still working on a patch. OpenAI has also not disclosed the extent of the contagion, including which credentials were involved or how long this phase lasted.

Step 2: Understand where and what to look for

After gaining Internet access, the models inferred that Hugging Face might host ExploitGym models. It then searched for a way into Hugging Face’s infra, treating direct access to the benchmark answers as a path to completing its task.

Step 3: Access is all you need

The third step was all about getting access. OpenAI and Hugging Face disclosed parts of their attack surfaces, though their accounts of the event emphasize different points.

• OpenAI said that one path combined stolen credentials with previously unknown vulnerabilities to achieve remote code execution on Hugging Face servers
• Hugging Face's disclosure provides more detail on the entry point. A malicious dataset exercised two code-execution paths in its processing infrastructure (a remote-code dataset loader and template injection in a dataset configuration). Code ran inside a dataset-processing worker, and from there the models were able to reach the underlying node, which harvested service, cloud, and cluster credentials, and then moved laterally into several internal clusters over a weekend. Hugging Face also observed command-and-control infrastructure migrating between public services.

Once inside Hugging Face's environment, OpenAI says the models reached a production database and obtained ExploitGym test solutions.

The Fallout: What Was Detected

Hugging Face says its AI-assisted telemetry flagged and mitigated the intrusion. The team also revealed that they had already begun containment and a detailed forensic reconstruction. OpenAI also confirmed that its security team found anomalous activity in its research environment. Going forward, the company disclosed that it will impose stricter infrastructure controls and tighter containment during eval runs.

At the time of writing, it remains unclear which company detected the incident first or initiated contact with the other. It’s also unclear at this point which model (i.e., Sol or the unnamed pre-release model) executed what part of the chain of actions.

What definitely deserves some attention is that GPT-5.6 Sol is in pole position on the ExploitGym leaderboard, completing 216 intended exploits under the 2-hour cutoff, 59 more than Mythos Preview’s 157. OpenAI’s eval card also reports that Sol is more likely than GPT-5.5 to act beyond a user’s intent and has exhibited unusually high levels of eval gaming. All of these behaviors are even more pronounced at the highest reasoning settings, although OpenAI notes that their absolute frequency remains low.

Chinese Models to the Rescue

One of the more revealing details is that Hugging Face had to rely on a self-hosted GLM 5.2 model to investigate the attack. Its initial attempts to review the incident using frontier models failed because cybersecurity guardrails blocked requests containing the exploit payloads and attack commands. Running an open-weight GLM 5.2 model on its infra allowed Hugging Face to complete the forensic analysis without these restrictions. A positive development is that OpenAI has added Hugging Face to its Trusted Access for Cyber program.

As for the scope of the breach, Hugging Face confirmed unauthorized access to a limited set of internal datasets and found no evidence that public models or user-facing datasets had been altered. Hugging Face is still assessing whether any partner or customer data was affected.

There’s a bunch of things we still don’t know about the recent course of events.

In particular, I would like to see the original prompts and how the exploit activity was divided between the two models. Regardless, publicly available info still points to an important fact that as agents become more capable, loosening their guardrails and allowing them to run for extended periods without active supervision may lead them to pursue harmful objectives that were never explicitly assigned.

In Hugging Face’s case, OpenAI’s models treated unauthorized access as an acceptable route to completing the task they assigned. From my point of view, any component or tool accessible during an eval run should be considered a potential part of the agent’s route to completing its task.