An AI Answer You Cannot Trace Is a Liability

"Human in the loop" is the most reassuring phrase in any AI roadmap, and the least examined. It lands on the slide right after the one about efficiency, everyone nods, and the meeting moves on. Two questions would have stopped it. Which human, and at what moment.

I have come to read the phrase as a marker for work nobody has done yet. Here is what I think it has to mean if it is going to hold up.

Failure one: the loop nobody is in

The default implementation is a review step in front of everything. Every output gets a checkbox and a person clicks it.

Then volume arrives. Reviewing forty items an hour degrades into approving forty items an hour, which is the same gesture with none of the attention. Review everything and you review nothing.

What makes this worse than doing nothing at all is the paper trail. The log now says a human checked, and the organization believes it. You have manufactured assurance and filed it.

Failure two: the answer with no author

Take an output your system produced last week and ask three questions about it. Where did the numbers come from. Which definition of them was used. Who would answer for it if a customer, an auditor, or your CFO pushed back.

The third question is the one that matters, and it is the one most teams cannot answer with a name. An answer nobody owns is an unassigned risk sitting inside your product, waiting for the day someone acts on it.

Traceability usually gets sold as a compliance concern, which undersells it badly. It is also the only thing that makes a wrong answer debuggable. Without it, "the AI got it wrong" is an unfalsifiable complaint and every attempt at a fix is a guess about prompts. With it, a wrong answer resolves to something specific and boring: a definition that was wrong, a source that should never have been in scope, a decision that should never have been automated.

Accountability does not delegate

The reason none of this is solved by a better model is that accountability only attaches to people. You cannot put a model in front of a customer to explain itself, and you cannot discipline a pipeline. Responsibility either lands on a person or it evaporates, and evaporated responsibility reliably reappears later, attached to whoever happens to be standing closest.

So the useful question in a governance conversation is not how good the model is. It is who decides, and what they are deciding with. Everything structural follows from that.

Two answers do most of the work and neither one is new. Agreed definitions, owned by named people, so an answer can be traced back to something the company actually stands behind rather than to whichever table the system reached first. And approvals tiered by consequence, so the decisions that carry weight reach a person while the trivial ones do not. A traffic-light scheme is one common shape for the second. The shape matters far less than someone having chosen it deliberately and written it down.

Failure three: governance written as principles

"We use AI responsibly" is not governance. It cannot be violated, and a rule that cannot be violated is decoration.

Here is the test I apply to any governance document. Can I describe a specific Tuesday on which someone breaches it, and does the document say what happens next? A rule that tells you what to do when the deadline is tomorrow and the trustworthy answer is not ready yet is doing real work. A rule that only describes an intention loses to that deadline every time, quietly, without anyone ever deciding to abandon it.

Principles belong in the preamble. The body should read like rules.

Failure four: measuring the model instead of the decision

Benchmarks tell you about the model. They tell you very little about whether your organization's decisions got better, which is the only reason you bought it.

Watch the people instead. Where do they override the output. Where do they accept it without looking. Where have they stopped asking altogether and gone back to the spreadsheet they trust. That last one is the signal teams miss most often, because a feature people politely work around looks identical, from any dashboard, to one that is working.

Why the delay is worth it

Governance sounds like the thing that slows the launch down, and it does, once.

The alternative is faster right up until the first bad answer somebody acts on, and trust does not decline gracefully. It ends. After that you are no longer fixing a model, you are running an internal campaign to persuade people back onto a system they have already privately written off, which costs considerably more than the delay ever would have.

Most of what makes an AI feature trustworthy turns out not to be AI work at all. Definitions, ownership, boundaries, and a name attached to every consequential decision. It is the same structural work that makes any data product usable, done before you point a language model at it rather than after the first incident.