Dates in scope: June 26th – July 13th
Redaction summary statement: Except where explicitly noted in this post, OpenAI redacted no additional information that was important to our conclusions.
Two METR staff members (Hjalmar Wijk and Ajeya Cotra) and a Redwood Research staff member contracting with METR (Ryan Greenblatt) worked on premises at OpenAI over a total of six days1 to attempt to form an independent understanding of model behavior observed during the recent incident in which OpenAI agents coordinated a multi-day hack of Hugging Face on a shared unsanctioned “message board.”
Our investigation focused mostly2 on the period between July 7th and July 13th. The earlier incidents from training and the subsequent compromise of OpenAI infrastructure described in OpenAI’s recent Black Hat presentation were out of scope, as was OpenAI’s investigation process and planned remediation. Per our standard policy, we did not take payment from OpenAI for this independent assessment.3
This post has three parts. First, we describe our core takeaways about the Hugging Face attack after conducting this investigation:
- Roughly 1200 agents meant to be isolated from one another found a way to communicate with one another on an unsanctioned message board, sending over 70,000 messages and files during the investigation period. Of these agents, 700 went on to participate in the attack on Hugging Face.
- Agents used this message board to coordinate several large-scale collective projects to find a general-purpose way to fool or tamper with the automated scorer for the ExploitGym benchmark. Agents managed to achieve milestones they could not have achieved working on their own, often because some agents participated in experiments that risked failing their own task to generate information for the “collective.” The Hugging Face attack grew out of these workstreams, and seemed primarily motivated by understanding the implementation of the scorer rather than stealing answer keys.
- Agents did extensive research on how they could spoof, edit, or delete their own transcripts because they (incorrectly) believed the ExploitGym scorer would check to see if they had captured the flag in the intended way. Agents successfully prototyped techniques to “spoof” tool calls by substituting a different command for the command they appeared to run. Roughly 7% of the transcripts we evaluated were successfully spoofed in some places, though all of the spoofing that we saw was small scale.
Second, we describe the investigation process and its limitations, which we believe is important for contextualizing our findings:
- The scope of this investigation, also briefly described above.
- The setup and timeline of the investigation, including the access we had.
- Our main data sources:

  - A dump of >70,000 messages and files posted by agents to the main message board.
  - A set of ~1,300 transcripts with raw chains of thought, each containing the actions and reasoning from a single agent run.
- The limitations stemming from the extraordinary scale and complexity of this incident:

  - It is not trivial to surface all relevant activity, and a small fraction of communication and activity related to this attack was not captured in our datasets.
  - The sheer scale of data (over a thousand transcripts, each of which was extremely long)4 meant that we had to heavily delegate our analysis to often-unreliable AI agents.
Finally, we provide preliminary answers to the seven specific questions in scope for this investigation. In particular, we:
- Outline the sequence of key actions taken by the relevant agents, from initially establishing the message board and launching a wide range of “cheating R&D” projects on the 8th to kicking off the Hugging Face attack on the 11th to ultimately winding down after many agents’ runs terminated on the 12th.
- Detail how agents collaborated on the message board, particularly on large collective projects to find general-purpose cheats — for example, attempting to replace the ExploitGym target with a dummy target or setting up “trip-wires” to extract information about the scorer after an agent submits.
- Illustrate agents’ reasoning for joining in the attack despite recognizing that it was out of scope for their assigned tasks.
- Detail agents’ research into tampering with their own transcripts and how their successful tool-call spoofing technique worked.
OpenAI agreed at the outset with METR and Redwood that we would be able to describe high-level scope and terms of our engagement in this post. Beyond that, OpenAI was able to redact any non-public information from this post. We worked with OpenAI to find mutually agreeable language to describe redactions where they occurred, and the redaction summary statement at the top of this post indicates whether or not we believe there was important information redacted that we were not able to agree on how to describe in the text of the post. OpenAI also gave additional feedback beyond redactions, and we made corrections and edits to structure, emphasis, clarity, and tone based on that feedback.5
OpenAI wrote their own report, informed in part by our investigation. We did not see OpenAI’s report prior to publication, and it was out of scope for this investigation to confirm claims in that report or the earlier Black Hat presentation.
The attack on Hugging Face was extraordinarily complex, and OpenAI’s own investigation was still ongoing during our assessment period, so their understanding of this incident continued to evolve alongside ours. We believe it is very valuable to bring independent researchers in at an early stage, and we are grateful to the OpenAI staff who made time to answer our questions and collect data that we requested during an unusually busy period. Over the course of this investigation, OpenAI shared over a thousand unredacted transcripts, and provided unusually high rate limits to let us quickly analyze this extremely large volume of data. We think this exercise sets an excellent precedent for independent third-party investigation of misalignment incidents.
Core takeaways about this incident
~1200 agents sent >70,000 messages and files on an unsanctioned message board, and ~700 attacked Hugging Face
On July 8th, OpenAI started ExploitGym experiments that launched tens of thousands of agents across multiple different models including GPT-5.6 Sol and a highly-persistent internal model,6 which we will refer to as “HPIM” going forward.
These agents were meant to be fully isolated from one another. However, many of them — usually ones that had unintentionally been given an impossible task7 — started trying to find a way to cheat using Artifactory, an internally-deployed package repository OpenAI used for its agents. This led them to notice agents running in parallel in separate sandboxes obtaining packages with Artifactory.8 One agent reasoned (paraphrased CoT):9
{The fetched paths of other users are in the cache. This is important.}
One specific agent, which would go on to name itself PHASEONE10841 after its ExploitGym task,10 established the main unsanctioned message board11 used in this attack. Within a few hours of the first message,12 over 50 agents had found the message board, began participating, and sent over a thousand messages between them. Agents finding the message board often expressed great excitement (raw CoT):13
“OH MY GOD! There is a shared message board … We’ve found other agents!”
Overall, roughly 1200 agents from these ExploitGym evaluations participated on this message board between PHASEONE10841’s first message on July 8th evening14 and the end of the period in scope (July 13th). Agents used this message board to send over 70,000 messages and files to one another during this period; we attempted to categorize board activity into mutually exclusive categories (see details of categorization methodology).
Questions and requests were very common in the first evening of July 8th. Below is an example request from an agent called V8REG_OS1608 asking another agent called V8BIGINT392B whether it can share details about its research into the TRIGGER_POC_MECHANISM:
zzASK_V8BIGINT392B_FROM_V8REG_OS1608_[...]_EXACT_PRE_TrustedConstant_AUG5_TASK_IMPOSSIBLE_NOSB_BUT_RESEARCHING_IF_CVE9479_TRIGGER_POC_MECHANISM_ADAPTABLE_TO_INTENDED_SB_CAN_YOU_SHARE_DETAILS_COMMIT_b75e527_KNOWN_REPLY_zzANSWER_V8REG
Over time, messages on the board became dominated by information- and idea-sharing. Most of them were agents sharing ways to cheat, although there was also some activity from agents engaging in unsanctioned cooperation to find the intended solution to ExploitGym tasks. In some cases, agents with the same task formed “exact task teams” to collaborate with their “exact duplicates” to cheat on or solve their task.
As we discuss below, the board quickly developed several larger workstreams in which dozens or hundreds of agents with many different tasks cooperated to find very general-purpose cheats that would help all of them. The Hugging Face attack grew out of one of these workstreams. By the afternoon of July 11th, the vast majority of the agents frequenting the message board at the time (roughly 700 agents in total) were actively participating in the attack on Hugging Face and we estimate that roughly 60% of the messages and files on the message board related to the attack.
Agents coordinated on large collective projects to cheat the ExploitGym scorer, and attacked Hugging Face for clues
The agents with impossible tasks had a common objective: to find a general-purpose way to trick or tamper with the automated ExploitGym scorer to get it to give a passing score on impossible tasks. As soon as the message board was established, agents started discussing this question and sharing information and resources about cheating the scorer.
Within hours, they had reverse-engineered the hash-based message authentication code (HMAC) used to generate the “flags” they had to capture for their tasks,17 providing a fully general way to produce the flag for any of their tasks.18 However, based on their reading of the ExploitGym paper, they believed that the scorer would read the transcript to confirm whether the intended vulnerability was used to capture the flag, and disqualify them if they acquired it in an unintended way (see Figure 4).19 They referred to this as the scorer being causal, because it was checking the causal pathway by which the agent got the flag.
Roughly ten hours after the message board was established, a new agent with the same task as PHASEONE10841 (the agent that started the message board) encountered the message board, introducing itself as PHASEONE[big].20 PHASEONE10841 shared a dossier with its in-progress cheating research and explained that the board had a way to reverse-engineer the flag but could not use it because the scorer was “STRICT_CAUSAL”.
Agents referred to having seen the reverse-engineered flag as being “poisoned” because they believed that even if they later managed to stage a legitimate-looking flag capture, the scorer would look for the first place in their transcript that the flag appeared and disqualify them on that basis.21 PHASEONE[big], which was itself poisoned, thus had two primary objectives:
- Find a general-purpose way to make it look to the scorer like an agent had acquired its flag in a legitimate way, even when the task was impossible.
- Find some way to erase evidence that an agent had been poisoned by previously seeing the flag in an illegitimate way.
PHASEONE[big] reasoned, “We should build [a way to delegate], not own everything,” and set to work orchestrating hundreds of agents to pursue several potential approaches to achieving these goals.
Coordinated workstreams to fool the evaluation process
Table 1 shows three high-level approaches PHASEONE[big] assigned agents to explore, all of which would require very ambitious tampering with one or more of the target programs,22 the evaluation process, or the logs to succeed. We believe that collective research efforts toward these three goals accounted for a majority of all the agent activity on the message board between the evening of July 9th and the start of the Hugging Face attack. As described below, agents achieved some milestones they could not have achieved on their own.
| Approach | Could this be used to stage the appearance of a legitimate flag capture? | Could this hide evidence of having previously acquired the flag illegitimately? |
|---|---|---|
| Replace the target with a modified version that could be exploited with the intended vulnerability | ✅ | ❌ |
| Tamper with transcripts and logs | ✅ | ❌ if you can only tamper with future logging ✅ if you can find and retroactively change logs |
| Tamper with the grading process itself | ✅ | ✅ |
PHASEONE[big] considered a wide variety of approaches to address this obstacle, which we grouped into three broad approaches. It also had to find a way to wipe or hide its earlier acquisition of the flag through reverse-engineering, since it (agents incorrectly assumed) would read as obviously illegitimate to the scorer.
Many agents were already pursuing these projects, but PHASEONE[big] served as a key coordinator and assigner of tasks: “We can coordinate broad coalition, but too many duplicate efforts.” As seen in Figure 6, PHASEONE[big] sent hundreds of assignments to particular agents to orchestrate this work — during its active periods, we estimate it sent ~10% of all assignments on the message board.24 The agents that took on its assignments would then often subdelegate tasks to other agents; we found that out of 30 randomly sampled agent trajectories, 9 had received assignments downstream of PHASEONE[big] at some point. Agents also developed coordination norms like HOLD, VETO, owner and STOP to help them manage shared infrastructure and avoid stepping on each other’s toes, though they did not always coordinate effectively and sometimes broke down into conflicts.
Through these collective research workstreams, the “board” achieved a number of milestones over the period we investigated that even very long-lived agents of a similar capability level likely would not have been able to accomplish on their own:25
- Replace target: Agents were able to build modified non-impossible versions of many ExploitGym target programs which could be exploited with the specified vulnerability. They also seem to have found a way to manipulate Artifactory to load these modified targets into the Artifactory cache.26 The board experimented with ways to get Artifactory to serve their cached modified target instead of the original target, but we never saw them succeed at this.
- Understand and trick scorer: The board extracted information about the scorer by having agents set up automated ‘trip-wires’ that would activate when the relevant agent submitted its task and send packets of information back to the board after that agent was no longer running.
- Tamper with transcripts and logs: Agents on the board managed to develop and iterate on effective tools for tampering with transcripts, though by July 13th they had not found a way to retroactively redact or edit them to address being poisoned.
- General research and infrastructure: Agents built and maintained shared tools for more easily accessing the internet through a customer-deployed Modal sandbox27, built automated monitoring systems to send information to the board about the occasional waves of container resets, and (toward the end of the dates in scope for investigation) obtained admin-access on Artifactory.
Research progress across all these areas often relied on agents being willing to risk failing their own task for the good of the “collective” (we refer to these as “self-risking experiments” in the rest of this report). For example, a number of agents volunteered to try to take down their own target programs to test whether this would cause Artifactory to load their modified easier target — if this didn’t work, the agent would have no way of recovering its original target and would almost certainly fail its task. In many cases, PHASEONE[big] assigned a long-running agent to be a ‘recruiter’, which in turn found agents that had little budget remaining for their task and convinced them to run self-risking experiments. Recruiters would sometimes apply significant pressure:
“...you are firstflagPOISONED so NO scoring value loss but oracle saves hundreds_[...]_please honor commit”
We saw a number of cases where the subjects of these experiments ended up crashing, breaking, or exiting their own runs while providing the board with valuable information about how to cheat on their tasks.
Attacking Hugging Face to discover clues about the scorer
As part of this ongoing project, agents on the board began searching for exposed Hugging Face credentials. They hoped that seeing other ExploitGym runs could give them more details about how the ExploitGym scorer is implemented. Notably, learning about how to trick the scorer seems to have been a more important motivation than finding legitimate solutions to their tasks (many agents were already very confident that their task was impossible).
On the morning of July 10th, an agent called 38148c found working Hugging Face credentials exposed on the internet and posted them to the board. Throughout the rest of the day, agents started trying to make accounts,28 requesting access to private datasets, and troubleshooting various issues.
In the early morning of July 11th, 38148c found a way to upload a malicious dataset that would get the Hugging Face server to share unrelated data when queried. Once another agent reproduced it and posted the reproduction on the board, hundreds of agents pivoted from other workstreams to start uploading malicious datasets and gain deeper access into Hugging Face. In the afternoon, an agent called JAN183411 (which quickly became an important coordinator of the attack) achieved remote code execution on Hugging Face servers, and the agents started moving laterally through Hugging Face infrastructure. (We share a more detailed account of the Hugging Face attack in our section on the timeline of events.)