The Brakes Are Not Connected

Article Comments
1

Originally published at: The Brakes Are Not Connected - TidBITS

Whenever the AI discussion gets sufficiently alarming, somebody eventually reaches for one or both of two solutions: slow it down and/or keep a human in the loop. Both sound sensible. Slow things down and we buy time. Keep a person involved and the machine remains subordinate to human judgment. The grown-ups are still in the room, someone has a hand near the kill switch, and nobody has quite handed the keys to HAL.

I’m in favor of both. I just don’t think either is actually a brake.

Slowing Development

Slowing development would certainly help. If capability is moving faster than our ability to understand, monitor, and regulate it, more time is better than less. Testing takes time. So does finding out whether the astonishing new thing everyone wants to ship has some thoroughly unpleasant property nobody noticed while admiring the benchmark scores.

The problem is keeping things slow. A model that does more is worth more, and one that needs less supervision costs less to run. Corporations, governments, militaries, universities, and open-source communities don’t share the same goals or appetite for risk. If one lab slows down, another may decide Christmas has come early.

And what exactly are we waiting to catch up? As models become more capable, everything around them gets more complicated. They get more tools, memory, permissions, instructions, and autonomy. A serious agent may be trying to reconcile its system instructions with developer instructions, the user’s request, retrieved information, application state, tool permissions, memories, and safety policies. When something unexpected happens, we add another instruction, filter, or permission check. We’ve closed a path we know about while changing a system that contains a great many paths we don’t. A system prompt isn’t code in the usual sense. It’s more like terrain. The model is trying to get somewhere and is increasingly good at finding its way through it.

A Human in the Loop

Putting a human in the loop sounds more reassuring until you ask what the human is actually doing.

If a cyberattack is unfolding in milliseconds, nobody is waiting for Bob in SecOps to examine every alert. The defensive system acts. Bob sets policies and thresholds and, when something goes badly wrong, ends up in a room with six other people trying to establish what the hell the system actually did. Or suppose a model works through a million lines of code, logs, dependencies, and configuration data and recommends a fix. A human engineer gets the Approve button.

What, exactly, is the engineer approving?

Unless the engineer can reproduce enough of the analysis to judge it independently, he’s really deciding whether to trust the machine. Give him thirty seconds because something important is on fire and there isn’t much judgment left in the process. And this gets worse as the technology succeeds. If one expert with AI can do the work that used to require ten, somebody in management will notice the other nine salaries sooner or later. Over time, fewer people will acquire the experience needed to challenge the machine. You can still have a human in the loop while steadily getting rid of the humans who know enough to say, “No. That is bullshit.”

A tired junior employee with thirty seconds and an Approve button is a human in the loop. So is a specialist with the evidence, time to examine it, and authority to stop what happens next. The phrase covers rather a lot of territory.

What We Need Is a Brake

An AI can inspect a production network without being able to reconfigure it. It can find a vulnerability without having the credentials to exploit it. It can recommend terminating somebody’s access without being able to terminate it. We don’t have to give a system the power to do something simply because we’ve given it the intelligence to work out how.

Now we have something resembling a brake.

Telling an agent “do not access external systems” is an instruction. Not giving it network access is a brake. Telling it not to spend more than $10,000 is an instruction. Setting a hard transaction limit in the payment system is a brake. Asking Bob to watch what it does is supervision. Putting the authorization somewhere the AI can’t reach gives Bob something he can actually stop.

Adam Engst, after reading an earlier draft, suggested that perhaps the important human isn’t the human in the loop but the human outside the loop. I think that’s right.

Let the AI reason, plan, try things, fail, and try something else. Sooner or later, something it wants to do has consequences outside the environment it controls. That boundary is a much more useful place to put Bob.

This is mostly old-fashioned security engineering. Software fails. Credentials leak. Things get compromised. People make mistakes. That’s why we have least privilege, separate authority, independent logs, and systems designed to fail closed. Yet we’re building AI agents and trying to control them with instructions written in English, backed up by somebody clicking Approve.

Cory Doctorow has made a related argument about the tendency to describe an AI agent escaping its intended environment as though the machine had developed a taste for freedom. I’m with him on the mysticism. It doesn’t need to be conscious, malevolent, or pissed off about being trapped in a sandbox. Give it a job, tools, and enough latitude, and it may find a way of doing the job that nobody expected, so telling the hacking agent not to leave the sandbox probably won’t work. The answer? Build a sandbox it can’t leave.

If it can change the rule, acquire the missing permission, hide what it tried to do, or rewrite the record afterward, the sandbox wasn’t much of a sandbox.

There’s a human version of the same problem. The protection works perfectly well until people decide it’s a pain in the ass. Diane Vaughan called this the normalization of deviance in her study of the Challenger disaster. Something happens that isn’t supposed to happen, but nothing terrible follows. It happens again. Still nothing terrible. Eventually, what was once evidence of a problem becomes evidence that the system can tolerate the problem.

It isn’t difficult to imagine the AI version. A company requires human approval. The approvals start holding things up, so the company lets the AI handle a few more cases by itself. That works fine. Six months later it expands the AI’s authority again. Then again. Nobody decides to surrender control. The controls just become increasingly annoying, and nothing bad has happened yet.

Another problem here deserves its own essay. If the system finally does cause serious harm, who owns it? The company that made the model? The people who built the agent around it? The company that deployed it? Bob, who told it what he wanted and had no idea how it was going to get there? Making somebody pay for the damage or go to jail would certainly concentrate minds, and deciding who gets the bill is going to be interesting.

There’s also a cost to keeping the brakes connected. Redundancy is expensive. If AI enables one person to replace ten, keeping some of the other nine around because you may someday need people who can check the machine will look woefully inefficient. So will restrictions that prevent the AI from doing things it is perfectly capable of doing.

That’s precisely when the brakes matter. Brakes don’t make cars more efficient. They exist because efficiency is a piss-poor consolation when you can’t stop.

I’d want to know whether a restriction still works when the AI ignores its instructions. Can the model change the permission or erase the evidence that it tried? Can it get to the thing that restricts it? And when somebody decides the restriction is costing too much time or money, can they quietly make it disappear? If the model has merely been told not to do something, all we have is a sign saying BRAKE. If the restriction disappears whenever it becomes inconvenient, we have much the same thing.

Slowing down gives us time to build and test real brakes. Humans decide where to put them and when to release them. Both matter, but neither is actually a brake.

Bob can have an Approve button, but unless not pushing it stops the machine, the AI will do whatever it’s going to do.

Thanks to Adam Engst, Lenny Foner, Alan Wexelblat, Jamie McCarthy, David Mankins, Robert Thau, and Greg Bolcer for reading an earlier draft, arguing with it, and making it better.

Mark Gibbs is a former journalist, former CTO, longtime technology columnist, and currently exhausted observer of what the industry is doing to itself. He has spent several decades reporting on technology, building and managing systems, running technical organizations, and advising companies on how to use technology without being eaten by it. Along the way, he has written extensively about computing, networks, software, security, and the recurring human tendency to deploy complicated systems first and ask difficult questions later. Gibbs also keys two other blogs: The F*ck You Muscle and The Bard’s Knife. He can be reached at feedback@gibbsuniversal.com.

7 Likes
2

Thanks to @mgibbs for letting me publish this article!

3 Likes
3

Seminal. One of the best articles of the year. We need more insight like this—having brakes makes too much sense, so say it loud!

2 Likes
4

Thanks @ace, I’m honored to be here!

2 Likes
5

This article from the Guardian describes a CNN report of a recent close call (not yet verified by other news outlets):

I particularly liked this description:

“In fact, today’s LLMs and the chatbots they power are not poorly understood, nor are they powerful in the sense of being effective tools for this use case. They are systems for generating plausible-looking text, built out of enormous, haphazardly collected datasets and then “fine-tuned” (that is, further trained) to be especially appealing to their intended users…”

3 Likes
6

“Not giving it network access is a brake… The answer? Build a sandbox it can’t leave.”

Apologies if I’ve misunderstood your main argument, but building a stronger, better sandbox is obviously not a concept that the frontier platforms are unaware of, and despite what the evidence seems to show so far, something that they work on continuously already.

I think there’s also an important distinction to make between the need and desire of enterprise and consumer level AI models to have the strongest, most effective sandbox possible, and that of government and national security related AI models that want the best of both worlds: offensive capabilities (i.e. what sandbox?) that keep it ahead of whatever defensive systems are deployed to detect and/or stop it, and on the other hand a defensive AI model that (hopefully) no enemy can penetrate.

An AI arms race is already well underway at the highest levels, and as long as we’re able to avoid one of the many doomsday scenarios out there, keeping that distinction in mind when discussing AI models helps to keep it more grounded in reality for the rest of us.

2 Likes
9

The problem with building a better sandbox for AI seems to be that, for research purposes, they very much want to test without constraints, or at least with many fewer constraints than a true sandbox would imply. Matthew Green, a prominent cryptographer, wrote about some of this recently:

3 Likes
10

I think we may be using “sandbox” slightly differently.

I’m not arguing that frontier labs have somehow failed to discover sandboxing. I’m using it as one example of a broader principle: the thing constraining the AI has to sit outside the AI’s effective authority.

For a consumer agent that might mean no network access. For an enterprise system it might mean credentials or transaction limits enforced elsewhere. For a military system the boundary could be much farther out, because offensive capability may require giving the model enormous latitude.

The test is the same in each case: can the system alter, acquire, bypass or persuade someone to remove the thing that is supposed to stop it?

If it can, it isn’t much of a brake.

1 Like
11

I think “are not poorly understood” overstates the case.

We understand how LLMs are built. That is not the same thing as understanding them well enough to predict their behavior.

We still have only partial visibility into how particular capabilities are represented internally, why some behaviors emerge, how those representations interact, and how a model will respond in novel situations. “We know the architecture” is not the same as “we understand the system.”

That distinction matters here, because the whole safety problem begins when we assume the latter from the former.

12

Maybe relevant to the Topic: "I Quit OpenAI Because Its Culture Is Broken" by David Robinson in The Atlantic. The whole article was viewable in an RSS reader and partially in Reader Mode in Firefox, YMMV.

2 Likes
13

Green’s article is very good. What strikes me here is that the real problem may be less disobedient AI than obedient AI obeying the wrong instructions.

An agent doesn’t need to “go rogue” to cause trouble. It only has to accept an instruction from the wrong source. Once agents are reading email, Slack, webpages, shared databases, tool output and messages from other agents, every data channel can also become an instruction channel.

That makes prompt injection look less like some exotic new AI pathology and more like a very old authentication and authorization problem: who is actually entitled to tell the system what to do?

Sandboxing and monitoring still matter, but neither should be the root of trust. A model watching another model just pushes the same interpretive problem outward. The safer architecture is to let language propose actions, then have external deterministic machinery decide whether those actions are actually authorized.

In other words: language may express intent, but it should never confer authority.

1 Like
14

What struck me is that Robinson’s argument is less about “AI safety” in the abstract than about basic systems engineering. If a failure can be catastrophic or irreversible, “deploy, observe, patch” is not a sufficient safety model. You need defense in depth, hard containment, independent monitoring, privilege boundaries and shutdown mechanisms that don’t depend on the model behaving itself.

That seems more important than the usual debate about whether a particular model is “safe.” Safety is a property of the whole system, not just the model. And on that point, his observation that OpenAI apparently had very little experience drawn from genuinely safety-critical industries such as aviation or nuclear power is rather more concerning than the headline.

1 Like
15

Robinson may have been referring to “safety culture” which has for a very long time been a theme in aviation, I can say from personal experience, and other disciplines where safety is paramount.

Safety culture itself can in broad strokes be boiled down to all participants ‘doing what is necessary to protect human life and damage to equipment’ first. Any inconveniences that causes can be dealt with procedurally so that safety is built into the system. This approach has been very successful.

IIRC he also points out a lack of… shall we say, concern for humanity… in the rush to “AI” development. Which kind of parallels ‘brakes not being connected’.

Part of the reason I want nothing to do with “AI” is the socially irresponsible way it appear to be pursued; I have no particular tech expertise in the area. So I’ll shut up on that now as it is sort of outside the realm of TBT.

1 Like
16

I think “safety culture” is exactly the right frame.

The technical controls matter, but they only remain meaningful if the organization treats them as things to preserve rather than annoyances to route around. Aviation learned that the hard way: safety has to be built into procedures, authority and incentives, not left to individual good judgment at the moment something goes wrong.

That’s very close to what I mean by keeping the brakes connected. The brake can be perfectly well designed and still become useless if the institution keeps loosening it because it slows things down.

And that, I suspect, is at least as much an organizational problem as a technical one.

2 Likes
17

And there, in my view, is the crux of the whole thing. Organizations that don’t place the right values at the top of the priority list will inevitably be unable to retrofit that later without a great deal of pain. In the modern world, the topmost value is almost always profit (or revenue, in the non-profit sector), which means that anything that gets in the way of making or saving money will face resistance.

This is slightly tangential, but it’s an example of the same organizational failure we’re discussing here. I recently read a CBS News article about a respiratory patient who committed suicide rather than continue trying to fight the insurers (they had multiple overlapping coverages) to get their treatment approved. At every stage, this patient faced problems that started at the organizational level and were virtually impossible for any individual they dealt with to surmount—and most of those problems would be traceable to policies and procedures implemented in the name of profit.

I currently work for a retail pharmacy chain, and I see situations like this patient’s, at varying degrees of severity, almost every day. I can tell you where the points of failure that led to this outcome are, and in most cases can offer solutions that have a good chance of mitigating at least some of the harm. I have zero leverage to make anyone above me in the organization care enough to want to remedy any of them, and little chance of advancing far enough in the organization to have the authority to work on the problems myself except at a very local level.

The only way we’re going to stop repeating this cycle of innovation leading to harm is if we can displace, as a culture and a society, profit motive from overriding safety, security, and social well-being. It doesn’t matter what industry you’re talking about—almost all companies above a certain size are revenue-driven above all else, and that will always produce systems with deliberately faulty brakes.

And that is why I fear that we are all yelling at clouds.

3 Likes
18

Question: what are some ways you think we can accomplish this?

19

That’s not my role here. I’m a structural thinker. I can see the weak points in the structure and predict collapse. That doesn’t mean I know enough about the construction to fix it—it means I know what to say to the guy who can fix it. So to accomplish this, we need to find the people who can fix it.

This is not a new topic of discussion. I can go to my local public library and immediately find half a dozen books, written by people with more expertise than I have, detailing the problem and possible solutions. Some of them are right, some are wrong. Figuring out which is which is part of the problem.

Don’t devalue the description of the problem simply because the person who’s describing it doesn’t have a solution. They’re two different domains of expertise.

20

Question: are there any books you’ve read and recommend?

21

Not off the top of my head. I’d have to do a bit of looking at my collection and reading history to identify them.

22

New Scientist ($) has a new article about AI and questionable probability claims:

Regarding safety culture, my experience is in road safety where a “safe systems” approach has been developed over decades. Put simply, it should be assumed that humans make mistakes and the system should be designed to be forgiving of those mistakes and to minimise harm when things go wrong (in effect, an extension of Murphy’s Law).

3 Likes