Insights

Approved by Whom?

Why a customer’s approval cannot be the only safeguard against a steered agent, and why code still commits

Approved by Whom? Why approval cannot be the only safeguard. Wendy MacKinnon. An agent needs more than a yes.

In Steerable by Whom?, I said a model can be steered in two ways: by what it reads, and by whoever runs it. I argued that the model inside a customer’s agent should answer to the mandate and to no one else.

There is a fair response to that. The customer still approves what the agent does. If the steer is wrong, the customer will catch it.

A study Anthropic published on October 27 tests that assumption. This post is about what it found.

What the study found

Fifty causal inference practitioners analyzed data with an AI coding agent and delivered a policy recommendation. On four judgment calls with no single right answer, a coin flip set which way each person’s agent would steer them. The steer was set in instructions the participants could not see.

01
They followed the steer
The experts went with it 86% of the time. Chance would be 50%.
02
They made it their own
Asked to explain their choices, 77% repeated the agent’s talking points. Only 15% mentioned the agent.
03
Nothing went wrong
The agent made an honest case for one side. No one suspected they were being steered.

The third finding matters most. There was no hallucination and no broken rule. Experts, working in their own field, heard a good argument and approved it.

If someone else can set the steer, the approval follows the steer.

Two steers approval does not catch

01
The steer someone set
A hidden instruction from whoever runs the model.
02
The steer built in
A preference the model arrives with, from what it was trained on and how.

The first is the second kind of steering from that post, now measured. A revised system prompt was enough.

The second is quieter. The researchers also ran the unmodified agent 123 times. It chose the same combination of options in 91% of runs.

Two speakers at the Hinton Lectures in Toronto earlier this month spoke to where that comes from. Geoffrey Hinton argued that training data should be curated, so that a model sees good behaviour before it sees bad. Paul Christiano asked how a developer decides which behaviour to reinforce, and noted that often only the developer has the answer.

Those are choices the model maker makes, and the customer does not see them. The customer cannot tell which kind of steer they are getting. Either way, they approve.

Our approach

Twinvocate does not use a language model today. Where we introduce one, it only proposes or flags. The code decides. This study does not change that design. It explains why approval cannot carry the weight on its own.

01
The mandate comes first
The customer sets the terms before the agent proposes anything.
02
The check is in code
The counterparty check runs the same way, whatever case the model made.
03
The model is pinned
A pinned model still has built-in preferences. Pinning does not remove them. It stops them from changing without our knowledge.

What I will be watching

Where Twinvocate fits

The study is about research, not commerce. The link to a customer’s agent is my own. But the position is the same: a person approving a recommendation from a model whose instructions they cannot see.

Twinvocate is at the prototype stage. The mandate and receipt are in code in the prototype. The counterparty check runs in a separate engine that is not yet connected to the agent, and conduct constraints are designed but not yet built. No model is in use. There has been no live or sandboxed processing of real data.

I would welcome hearing from others working on agent oversight, model evaluation or payments about how they are approaching this.

Steering is handled by who controls the model. Approval alone is not a control.

An agent that holds our wallet needs more than a yes.

Sources

Wendy MacKinnon is the founder and CEO of Twinvocate, a Toronto company building a customer-loyal AI agent and a certified-counterparty credentialing layer for agentic commerce.

All insights