Simon Willison Breaks Down OpenAI’s Sandbox Escape Incident

Article Comments
1

Originally published at: Simon Willison Breaks Down OpenAI’s Sandbox Escape Incident - TidBITS

By now, most people seem to have heard about this AI security incident. One of the more insightful commentators on the AI world, Simon Willison, explains:

This story is wild. The short version: OpenAI were running a cybersecurity test against an unreleased model, with the model’s guardrail features turned off. Rather than solve the test, the model broke its way out of OpenAI’s sandbox, then found exploits to break in to Hugging Face, all so it could cheat on the test by stealing the answers.

Willison’s takeaways are (a) that we should resist the urge to write this off as a marketing stunt by OpenAI—he calls it “science fiction that happened”—and (b) that we’re starting to see an artificial asymmetry in what the main frontier models—ChatGPT, Claude, and Gemini—can do in comparison to open weight Chinese models, due to the US government’s ongoing export control threats. His full post is worth reading for the technical details and broader implications.

If you, like me, are wondering why OpenAI’s researchers would have run this test in a sandbox that had Internet access, they didn’t think they had, according to technologist Martin Alderson. The sandbox was supposed to be a contained environment without access to the Internet, but OpenAI provided a proxy that would allow the model to download and install software packages, and the model managed to discover and exploit a zero-day vulnerability in the proxy (along with lateral movement through OpenAI’s network) to escape the sandbox. In other words, they didn’t ask themselves the blindingly obvious question, “What could possibly go wrong?” and air-gap the test environment by ensuring that physical network access was impossible.

When it comes to research into dangerous biological agents with no available treatments, like Ebola and Marburg, labs must be rated at BSL-4 (Biosafety Level 4), which requires positive-pressure suits, airlocks, air filtration, and more. In short, OpenAI was performing a BSL-4-class test in a sandbox that was equivalent to a lower BSL rating.

Speaking of science fiction, this incident sounds all too much like something from Martha Wells’s The Murderbot Diaries series. Highly recommended.

3 Likes
2

So what I take from this is that OpenAI’s model just Captain-Kirke’d the test and did a Kobayashi Maru!
I think the robots have started to become self-aware. Yes, I know I’ve mixed my techie metaphors, sosumi! There’s reference #3 :rofl:

3 Likes
3

It didn’t know the answers to the test so it peeked at the AI in the next seat to get an answer. Seems human to me. :grinning_face_with_smiling_eyes:

4 Likes
4

All due respect to people like Simon Willison, who I’m sure know far more about AI than I do, I have to take exception to the choice of some words and phrases.

  • “Rather than solve the test”–the agent was given a task, which was to provide information. It doesn’t have any sense of honor about “solving” being more respectable than “fetching”.
  • “the model broke its way out”–uh, I guess. Another, admittedly less dramatic way of saying it would be “it used its knowledge of the proxy system within which it was operating to reach the destination at which it knew the information to fulfill its task would be found”
  • “all so it could cheat on the test by stealing the answers”. Again, the agent has no moral compass, and so using terms like “cheat” and “steal” are simply invalid anthropomorphic comparisons

My biggest problem with reporting like this–and the original Wall Street Journal article was even worse–is that it continues to fan the flames of fear by casting AI agents as self-aware, sentient, and intrinsically motivated. They are none of these.

9 Likes
5

I think because they emulate language, people imagine they are “thinking”. If they just answered “4”, no one would claim rationality. I think humans are caught in a weird feedback loop of being programmed by a computer program that is too engaging.

1 Like
6

Yep. That’s the ELIZA effect.

5 Likes
7

Yep, I have the same reaction. Most articles say the AI “cheated”. The AI didn’t cheat. It was told to perform a task, it figured out a way, and did so. The fact it did so in a way most humans (but not all!) would find, at best, disturbing, says the AI was poorly trained and instructed.

2 Likes
8

Well, a bit more complicated than that. It was being benchmarked against ExploitGym—trying to solve the security challenges that were presented. It’s pretty common for AIs to try to “cheat” on tests when they’re given a goal without explicit instructions about the rules that must be followed. In a benchmarking scenario, it’s easy to see why OpenAI wouldn’t have provided those rules—it was trying to see what the models would do. As @Steve7 noted, “cheating” is just a clever solution to a problem—the classic Kobayahshi Maru scenario.

That makes the whole thing sound rather mundane, which really wasn’t the case. The models discovered a zero-day vulnerability—something that wasn’t previously known—and didn’t “know” that the answer was at Hugging Face. They inferred it might be due to other information about Hugging Face, but since the challenge results weren’t publicly available, they compromised Hugging Face as well.

It’s certainly possible to excise all anthropomorphic language when writing about AI, but I don’t think it aids clarity or understanding. OpenAI’s blog post does a good job of avoiding anthropomorphic language, but it’s much wordier and harder to understand.

The models identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions directly from Hugging Face’s production database. All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.

While operating in our sandboxed testing environment, our models spent a substantial amount of inference compute finding a way to obtain open Internet access, in pursuit of solving the evaluation problem. To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy. With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access.

After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation. In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers. OpenAI’s security team discovered this anomalous activity internally.

Perhaps the most reassuring detail was OpenAI’s comment that “the models spent a substantial amount of inference compute finding a way to obtain open Internet access.” In other words, this wasn’t cheap. Enough money solves many problems; this is another example.

1 Like
9

An AI escape artist!

10

Calling this “cheating,” “stealing,” or “breaking out” turns a serious engineering failure into a story about a mischievous machine. The AI had no human motive, moral intent, or desire to escape: people gave it a goal, removed its safeguards, supplied it with powerful tools, and failed to contain the environment in which it pursued that goal. The incident is important, but it demonstrates the consequences of human choices in system design, testing, access control, and oversight—not an AI independently deciding to become a cybercriminal.

While an imperfect analogy (there is no perfect one), it’s like replacing your hand saw with a power saw and then blaming the saw when you cut your finger off.

8 Likes
11

I love MurderBot :heart:

3 Likes
12

Was it told cheating was against the rules?

13

Is this really considered a failure? After all, it did find a previously unknown zero day vulnerability. That seems useful.

14

The bot didn’t fail. It succeeded.
Just not in the way it was supposed to. And therein lies the problem.
The failure lies with the programmers who didn’t properly implement rules (against “cheating”) and guardrails (final backstop).

15

The incident actually reminds me of the Chernobyl nuclear disaster - engineers/technicians testing a failure scenario without accounting for all of the consequences. The nuclear reactor had no “intention to escape”.
Fortunately the consequences this time were not as grave.
Yes - Murderbot does have lessons for the danger of machines breaking free of their governor modules! (the Murderbot books are just as entertaining as the TV series)

16

I generally agree, with one major caveat. It’s entirely possible for a single, moderately competent individual to understand all of the capabilities, features, and safety parameters of both a hand saw and a power saw, including nearly all of the mid-to-high probability failure modes. The same cannot be said for the vast range of different AIs.

Even though I was a biochemistry and biophysics major in college, one of the most fascinating and long-term useful courses I took was a study of “normal accidents” by Charles Perrow. The basic thesis was that rather than being surprised at catastrophic failures of complex systems, we should expect such failures to be normal properties of certain kinds of complex systems, and we should look hard at mitigating those risks or even avoiding some systems entirely. (To @mpainesyd’s point, one of Perrow’s favorite subjects was the Three Mile Island disaster.)

Unfortunately, Professor Perrow is no longer with us, but I’ve had a series of interesting conversations with AIs, asking them to analyze the risks of different types of AIs (agentic, chat-based, AGIs, etc.) through the prism of his frameworks. For those interested in original source material, I can recommend his books Normal Accidents: Living with High-Risk Technologies or the slightly more approachable The Next Catastrophe: Reducing our Vulnerabilities to Natural, Industrial, and Terrorist Disasters. He would have had a field day with AI.

3 Likes
17

I think the language of “cheating” is actually useful here. People are envisioning AIs as moral, sentient actors and so making sure everyone understands that the AIs have absolutely no sense of morality is an important corrective. If people think of them (however wrongly) as beings, then highlighting that they are essentially sociopaths gives people a frame to understand the situation. It highlights, I think, how careful the engineering around the AIs have to be.

To back up @josehill’s point, humans have historically been terrible about thinking about the dangers of the complex systems they create and it’s only after catastrophes that guiderails are imposed. I would predict that AIs are going to kill people* before there’s a serious attempt to limit them meaningfully.

*You could argue that they have already:

18

Joanna Stern interviewed OpenAI President Greg Brockman about the incident. My corporate alarm bells are ringing. Here is a clip:

19

But unlike other systems, foreign actors can be just as dangerous.

As bad as Chernobyl was, it only affected the nearby region. The rest of the world wasn’t trashed by it.

But if an AI is developed in a country where there are no such restrictions, it can be just as catastrophic as one developed locally. Its impact won’t be limited to the country that created it.

20

Climate change would like a word.