There is a phrase that has been doing an enormous amount of quiet work in AI governance for the last few years, and it is starting to buckle under the weight of it. The phrase is “human-in-the-loop.” It is supposed to mean that however fast or strange the system gets, there is still a person somewhere who can catch the mistake before it matters. It is meant to be the seatbelt. Increasingly, it looks more like a seatbelt drawn on with markers.

I want to make a fairly direct argument here. Human-in-the-loop was never really a safeguard against bad decisions. It was a safeguard against the appearance of nobody being responsible. Those are different things, and the gap between them is exactly where the risk lives.

The loop is real. The oversight is theatre

Bloomberg ran a piece recently on the state of Silicon Valley under agentic AI, and it is worth sitting with because it is not a thought experiment, it is a description of how people are actually living right now. Founders run half a dozen or more coding agents at once, each one checking in every ten minutes or so to ask what to do next. One founder keeps his laptop open at his kids’ soccer practice so he doesn’t miss a prompt. Another has been sleeping at the office for weeks because, in his words, the company is “toast” if it doesn’t hit a revenue target in time.

That is human in the loop, technically. There is, quite literally, a human, in a loop. He is being asked for input every few minutes, all day, sometimes through the night. But nobody watching that founder would call what’s happening “oversight.” It’s the opposite of oversight. It’s a person too depleted and too rushed to meaningfully evaluate any single one of the hundred decisions they’re waving through, because the system is generating requests faster than a tired brain can genuinely interrogate them. The loop hasn’t made humans safer. It’s made the human the bottleneck everyone is quietly trying to route around, including, eventually, themselves.

This is the part that tends to get lost in the governance conversation: being present is not the same as being informed. You can be looped in on every action and still have no real view of the thing that actually matters, which is what each action was based on.

Even the regulators have stopped pretending

Sarah Breeden’s Bank of England speech at the end of June made this official in a way that’s hard to argue with. She described the shift from AI that generates content, to AI that reasons, to agentic AI that can chain sequences of actions together on its own. And then she said the quiet part: relying on a human in the loop for every agent action “is unlikely to be realistic.”

That’s a central banker telling you the comforting mental model doesn’t scale. Which tracks, because it was never really designed to scale. It was designed for a world where AI produced one output at a time and a person reviewed it before anything happened. It was not designed for a world where the AI is the one doing the acting, repeatedly, across systems, faster than a calendar invite for a review meeting could even be sent.

So here’s the sharper problem underneath Breeden’s warning. Even where a human genuinely is reviewing something, what are they usually being shown? A recommendation, a confidence score, an approve button. Clean. Plausible. Nothing that reveals what assumption the whole thing is quietly resting on. If the only thing the human ever sees is the finished, polished output, they’re not really in the loop on the decision. They’re the final click on a decision that was assembled somewhere they never looked. That’s not an oversight. It’s a signature ceremony.

The loop was checking the wrong layer

Here is the reframe I think actually matters. “Human in the loop” is a question about process. Is a person present at some point in the sequence. That is a low bar, and agentic AI is about to make it lower still, because the sequences are getting longer and the checkpoints are getting sparser.

The question that actually protects you is a different one, and it sits underneath the process question rather than alongside it: what is this decision based on? Not the workflow. Not the approval step. Not whether the model produced a tidy explanation for its own output — a model can explain itself perfectly well while resting on a completely wrong assumption about the customer, the risk, or the market. The basis is the data used, the data quietly missing, the definition of success somebody baked in months ago, the time pressure distorting the call right now. You could staff a human-in-the-loop process with the most conscientious reviewers on the planet, and if nobody has stress-tested the basis before the loop even starts, the loop is reviewing a decision that was already compromised on the way in.

This is the space I’ve been building the Needle Framework around. Not as a replacement for human review, and not as another governance layer stacked on top of the ones you already have. As the check that happens before commitment — before the system acts, before it’s automated, before anyone signs anything. What is the decision actually standing on. What assumption is carrying the most weight? What would have to be true for this to be safe. Answer that properly, upstream, and the human-in-the-loop step downstream stops being theatre, because there’s finally something real for the human to be looking at.

And this is where Nadella’s point sharpens the argument

Satya Nadella made an observation recently about the future of the firm that I think extends this well past a compliance conversation. His line, roughly, is that the real opportunity isn’t in picking the best model. Everyone has access to more or less the same frontier models. The durable advantage comes from the learning loop a company builds around its own workflows, judgement, and accumulated institutional knowledge — what he calls token capital, sitting alongside human capital.

Put that next to the decision-basis argument and something clicks into place that’s easy to miss if you only think about this as risk management. The record of what a decision was based on isn’t just a defensive artifact you produce if a regulator ever asks. It’s the raw material of the learning loop Nadella is describing. Every time you write down what assumption a decision rested on, what evidence would have changed it, what context the AI didn’t have — you’re not just protecting yourself. You’re building the thing that makes your organisation’s AI usage genuinely yours, rather than a slightly customised wrapper around whichever model everyone else is also renting this quarter.

So the model is not the moat. Fine. But neither, on its own, is the workflow, or the audit trail, or the human dutifully clicking approve every ten minutes at his kid’s soccer practice. The moat, if there is one, is whatever your organisation actually knows about the basis of its own decisions — captured early enough to be useful, specific enough to be defensible, and honest enough to survive someone actually asking about it.

Human-in-the-loop was never going to give you that. It was only ever going to give you someone to ask, after the fact, what happened. The better question is upstream of the loop entirely, and it’s the one nobody’s put a name to yet. What was the decision based on?