We Said Trust Is a Computation. Ours Was the Wrong One.
A person signs in with a work account that their organisation manages. Everything about them checks out: the identity provider confirms the account belongs to the organisation, the software they are using is the build we shipped, nothing about their behaviour is unusual. Every model call they make comes back from our policy engine as Challenge, with a score of 0.29.
The engine runs in observe mode, so nothing was blocked. It recorded “would have refused”, over and over, for someone it had no reason to doubt.
The obvious suspect
The identity reading looked wrong. The provider was vouching for the account at assurance level 2, and the engine was recording level 1. We built the correction. On paper it moves the identity score from its low reading to 0.5, and the overall score to about 0.48.
Still Challenge.
When fixing the input does not change the answer, the input was not the problem. The problem was the arithmetic.
The arithmetic
# Compute effective score (product of all factors)
effective_score = 1.0
for factor in factors:
effective_score *= factor.score
Then the verdict: at or above 0.7, allow. At or above 0.4, audit or challenge. At or above 0.2, challenge. Below that, deny.
Multiplying confidences has a property that is easy to miss. Take four checks that each come back 90% good: 0.9 × 0.9 × 0.9 × 0.9 is about 0.66, below the line for allow. Four good answers multiply into a doubtful one. The more you check, the worse a good request looks, and a single middling score drags everything else down with it.
Our own doctrine says not to do this, in the document the same team wrote to define how trust is computed:
The × in
Identity × Supply Chain × Behavioral × Associationnames the composition — a floor with compounding — not a literal product of scores. Each dimension is an inspector; the decision is their join.
Think of a building inspection. The electrician signs off or does not. So does the plumber, and so does the structural engineer. The building passes when every inspector passes it, and a failed foundation is not softened by excellent wiring. Nobody multiplies the three reports together and checks the result against a threshold. In the doctrine, each dimension gives its own verdict from its own evidence and the stakes of the act, and the decision is the strictest of them. Several weaknesses at once can push it stricter still:
The decision may be stricter than every inspector’s verdict; it is never more lenient than any of them.
Read that way, the run above is simple. At the stakes of a model call, an identity at level 2 is Allow, a checked supply chain is Allow, and the decision is Allow. Even at level 1 it would have been Audit, which proceeds.
In July I published the claim that trust is never a property; it is always a computation. The claim held: our system does compute trust, per action, from evidence. It computed it with the arithmetic its own doctrine names as the thing not to do. We found it by checking the engine against the doctrine line by line before reusing it elsewhere, not through an incident.
The second mistake: a ladder for two different questions
The same review found the engine’s verdicts in the wrong order: it ranked Challenge below Step-Up. We fixed the code to match the doctrine’s list, defined “block” once (it had been defined in four places), and pinned the order with a test. The test is still green. It pins the wrong shape.
Step-Up and Challenge are different questions. A bank teller who asks to see your ID is asking are you allowed? A teller who asks whether you really mean to send ten thousand to an account you have never paid is asking are you sure? Showing your ID does not answer the second question, and saying “yes, I’m sure” does not answer the first. Neither question is a stricter version of the other, so they cannot sit on one ladder. The doctrine now draws them side by side:
┌── Step-Up ("are you allowed?") ──┐
Allow < Audit ───┤ ├──> Step-Up + Challenge < Quarantine < Deny
└── Challenge ("are you sure?") ──┘
Graded, not a ladder: Step-Up asks “are you allowed?”, Challenge asks “are you sure?”; neither implies the other; where both apply, both are required.
So we were wrong twice: first the code was out of order, then the order we fixed it to was itself a simplification. The doctrine’s own list and its decision table had disagreed, and we fixed the code to the list without weighing the table. Nothing raises Step-Up yet, so neither mistake has produced a wrong decision, but the test has to be rewritten to pin the diamond.
Did the doctrine just move to meet the code?
Ask that whenever a document changes after the code embarrassed it. Here the principles held: no dimension makes up for another, a zero anywhere is a zero everywhere, trust is not a single score. What was tightened was the wording. One axiom called the composition a “product”, and the code took the word literally; it now says what the principle beside it always meant. Every edit moved the text away from the code, not toward it. The doctrine’s wording is tightened only by its own reasoning against its axioms, on the record; the code never rewrites it.
Still open
The new engine is built and tested; it is not yet in the decision path. Each dimension gives its own verdict, scaled to the stakes. The decision point takes the strictest and lets several weaknesses together push it higher, never lower. A disqualifying finding (an anonymous caller where identity is required, a tampered artifact) goes straight to Deny. Its tests check the properties written down for it: never more lenient than any single verdict, higher stakes never make the answer looser and better evidence never makes it stricter, and the same answer whatever order the evidence arrives in. Until it is wired in, the multiplication is the policy engine.
The line that stays
Having a computation is not the same as having the right one. A system can decide per action, from evidence, record every input, and still compute the wrong thing at every request.
Trust is a computation. So is a wrong answer.
