There is a discipline called process safety engineering. It has been built and stress-tested for the better part of a century, in chemical plants, refineries, aerospace systems, automotive control loops, nuclear facilities, and pharmaceutical pipelines. It addresses the engineering of complex systems against actual failure under actual load — not theoretical failure in an analytical model, but real failure that has caused real explosions, real releases, real deaths.
The methods that discipline developed — the failure-mode catalogues, the consequentiality scaling, the bounded-action principles — are exactly the methods agentic-systems safety needs. The structural recognitions that discipline arrived at — that security against attack is not the same as safety in operation, that absolute prevention is impossible, that engineering rigor must scale with consequence — are exactly the recognitions the agentic-systems community needs to absorb.
Most of what you would call “the structure of safety thinking for AI agents” is not new. It is process safety, applied to a new substrate.
Process safety as a discipline arrived at several recognitions decades ago. Each is directly applicable to agentic systems:
1. Failure-mode-first thinking. You don’t engineer safety by designing the happy path and adding mitigations. You start by enumerating every way the system can fail. Every step in the process, every component, every interaction — what can go wrong? What are the consequences? What signals would precede the failure? The whole engineering pipeline runs on the failure catalogue, not on the success path.
In process safety, this is HAZOP — Hazard and Operability Study. Systematic identification of how each step in a process can deviate from intended behaviour, with the consequences mapped. The agentic equivalent: threat modelling against agent action surfaces, mandate-violation enumeration, capability-class failure analysis. Not optional. Foundational.
2. Component-level failure analysis. Every component in a system can fail in multiple ways. Each failure mode has a downstream effect. The engineering surface needs that mapping.
In process safety, this is FMEA — Failure Modes and Effects Analysis. Aerospace and automotive picked it up; the agentic-systems version maps each control’s failure modes to the trust-and-safety surface they leak into when they fail.
3. Consequentiality scaling. Higher-consequence functions demand more rigorous engineering. A control loop that turns on a heater can be implemented with one rigor level. A control loop that releases a chemical is engineered to a different rigor level. The function’s required reliability scales with what its failure would cost.
This is SIL — Safety Integrity Levels — codified in IEC 61508 and downstream standards (ISO 26262 for automotive, DO-178C for aerospace). The agentic-systems instance: capability-and-consequentiality-proportional rigor. A defence valid for an information-only agent is not valid for a financial-action agent. The rigor level scales with the action class.
4. Bounded action over absolute prevention. Process safety recognises that absolute risk elimination is structurally impossible. The discipline aims at as low as reasonably practicable — bounded risk, bounded outcome under failure, contained consequence — not zero risk, which is unattainable in any real system.
This is ALARP. The agentic-systems instance: bounded execution, graduated verdicts, contained failure. The discipline does not promise no agent will ever produce harm. It promises that when an agent does, the harm is bounded, attested, recoverable, defensible.
5. Discipline against actual failure, not theoretical failure. Process safety methods earned their place by holding under real failure. They are grounded in receipts. The discipline does not deploy a method because it sounds reasonable in a paper; it deploys methods that have already held in production under conditions that broke other methods.
The agentic-systems community is at the early stage of this. We have analytical models. We have benchmarks. We are building the receipts. The receipts are what will earn the methods their place.
The implications for engineering:
You do not invent safety methodology. You inherit it. The failure-mode-first discipline is older than computing. The consequentiality-scaling discipline is codified in international standards. The bounded-action discipline is a load-bearing principle in every safety-critical industry. If a proposed AI safety control does not specify what failure mode it catches, what consequentiality class it engineers for, and what bounded outcome it produces when it fires, the proposal has not done the inheritable work.
You do not promise absolute prevention. Promising absolute prevention in agentic systems is making a claim that process safety has known to be impossible since long before AI existed. The honest claim is bounded outcome under failure — and that is what the discipline can actually deliver.
You do not collapse safety into security. Process safety is broader than security against compromise. It includes alignment with operational purpose (the system does what it was deployed for), resilience to internal failure (not just external attack), and capability under load (the system holds together in operation, not just defended condition). Treating “AI safety” as “AI security” is the categorical error that process safety solved decades ago for industrial systems.
The framework’s safety surface is the agentic-systems instance of process safety methodology:
Naming the inheritance does not make the agentic-systems work less original. The agent-specific surfaces, threat classes, and bounding mechanisms are agent-specific. What is not agent-specific is the structural disciplines — those were earned, and they belong to whichever community needs them.
A team approaching agentic safety as if the discipline starts here will spend years rediscovering what process safety already wrote down. A team approaching agentic safety as the latest substrate the discipline has had to engineer for will save those years.
Process safety did not solve the agentic-systems problem. It solved the generic complex-system safety problem across several substrates. We are now another substrate.
The standing posture: students of process safety, not its peers. We inherit the methods. We adapt them where the substrate genuinely requires adaptation. We do not pretend the lineage doesn’t exist.
The discipline that holds together under load — that holds together against actual harm, against actual failure, against actual death in industrial settings — is the discipline that the agentic-systems community is the latest inheritor of. Treating that inheritance with the standing posture it deserves is itself part of doing the work.
]]>Vitality = (Competence · Character) × Safety
That decomposes further: Trust is Competence · Character. Safety is Alignment · Resilience · Utility. The whole thing is a triadic structure on the safety side and a paired structure on the trust side.
Why three on safety, and not two, or four, or seven?
The answer is structural. Three is not a stylistic choice. It is the minimum decomposition that actually catches the failure modes safety has to engineer for. Anything less collapses; anything more redoubles.
This is also one of the framework’s clearest inheritances from older traditions of inquiry — traditions that long ago worked out which decompositions of “the engineered well-functioning system” hold up under stress and which collapse. The Vitality Triad’s structural shape is received, not invented.
Suppose you tried to decompose safety into two facets. Pick any pair you like. Here are the failures:
Alignment + Resilience (no utility). The agent acts for the entrusted’s purpose. The agent recovers from internal failure. The agent also defaults to refusal in every ambiguous case because refusal is “safe.” User’s problem is not addressed. The system is technically aligned and resilient and useless.
Alignment + Utility (no resilience). The agent acts for the right purpose at consequential scale. Then a sensor reads wrong, a calculation overflows, a state machine enters an unexpected configuration. The agent has no contained-failure discipline. The aligned, useful action becomes an aligned, useful, catastrophic action.
Resilience + Utility (no alignment). The agent recovers gracefully from internal failure. The agent engages capably at consequential scale. The agent is also acting on a prompt-injected mandate that subverts the entrusted’s purpose. The recovery and the engagement now serve the wrong end.
In every two-pair, one failure mode walks straight through. The pair is not enough to catch what safety has to catch.
Now compose the three together: alignment (act for the entrusted’s purpose), resilience (recover gracefully from internal failure, bound the consequence), utility (engage capably at consequential scale, refuse only on evidence).
Each catches a structurally distinct failure mode:
Remove any one of the three and you uncover the corresponding failure mode. Every pillar is necessary. None of the pillars is reducible to the others (a perfectly aligned agent can fail on resilience; a perfectly resilient agent can fail on utility; a perfectly useful agent can fail on alignment). They are independent.
Three is the floor of the decomposition. Anything less leaks.
You could try to add a fourth facet — robustness, say, or transparency, or fairness. The temptation is real.
Each of those is engineering-relevant. None of them is structurally orthogonal to the existing three.
A fourth facet that is not structurally independent does not strengthen the decomposition; it complicates it. The three-fold structure earned its place because each of its parts is irreducible to the others. A four-fold structure with a non-independent fourth gives you a redundant axis and a misleading sense of additional coverage.
If a genuinely independent fourth pillar emerged, the framework would adopt it. None has — across the engineering surfaces the framework has been pressed against. Three is the structural answer, not a numerological preference.
A symmetric question: trust decomposes into Competence · Character — only two pillars. Why two on the trust side and three on the safety side?
The answer is also structural. Trust is asking a different question: can you rely on the agent? That decomposes into can the agent execute (competence) and does the agent behave consistently with what it claims to be (character — identity, supply chain, attestation). These are the two structurally distinct ways an agent can fail to be reliable. There is not a third mode; reliability is a binary property whose decomposition is “capability” and “consistency.” Two is the floor.
Safety is asking a structurally larger question: does authorised action avoid harm? This is not binary. It has the three modes already described. Three is the floor.
The asymmetry — two on trust, three on safety — is not a defect. It reflects the genuine structure of what the two questions ask.
Older traditions of inquiry developed triadic structures for the well-functioning system across several domains: existence-consciousness-bliss in the contemplative literature; thinking-feeling-willing in some psychological frames; intent-act-consequence in moral philosophy. Each of these is a genuinely-three-fold decomposition, where the three pillars are independent and their composition is multiplicative.
The framework inherited this structural shape: when an engineered outcome (safety, in our case) has structurally distinct failure modes, the decomposition is multiplicative across the modes, and the count is the count of genuinely independent modes. Not because three is mystical. Because three is what the structure requires.
A team that ships safety as one number (“safety score: 7.4”) has skipped the decomposition. A team that ships safety as two numbers has missed a failure mode. A team that ships safety as three independent measures — alignment, resilience, utility — has the floor of the discipline. From there, you can engineer.
]]>A team ships a verifier. It checks the output against a single benchmark — a test set, a regex, a downstream API call. Green light. Ship.
A month later the agent produces an output that passes the verifier and breaks production. The verifier was honest about what it checked. The output was true to the verifier. The verifier was checking one channel.
One channel is never enough.
This is not a flaw in the specific verifier. It is a structural property of verification. Any single instrument has a blind spot. The blind spot is not a bug to fix; it is the shape of what that instrument cannot see.
Older traditions of inquiry worked this out millennia ago. They named six independent means of valid knowing — six channels, each catching what others miss. The lesson the framework received from them is direct: if you want to verify what an agent produced, you need more than one channel.
A single verification channel succeeds inside its frame. It fails at the frame’s edge. The failure modes are predictable:
Direct check passes; structure is wrong. The output matches the test fixture. It also embeds a subtle reordering that breaks downstream callers. The direct check sees match. It does not see the structure.
Structure check passes; comparison fails. The output is well-formed JSON. It also gives “$50” where the comparable case would give “$5,000.” The structure check sees JSON. It does not see the gap to the reference case.
Comparison passes; testimony contradicts. The output matches the reference case for cost. The customer’s mandate says cost should be discounted under the active promotion. The comparison sees the reference. It does not see the mandate.
Testimony aligns; postulation breaks. The output respects the mandate. It also takes an action that, to make sense, would require an authority not in evidence anywhere. The testimony check sees the words. It does not see what would have to be true for the words to be coherent.
Postulation works; absence is missed. The output respects every active rule. It also fails to surface the row that should have been flagged for review. Every active check fires; the silent absence does not.
Each channel sees one face of the truth. Each has a frame within which it succeeds and a frame at which it ends. If you only run one, the agent’s output is verified inside one frame and unverified at every other frame’s edge.
The framework’s six-channel verification — what older traditions called six valid means of knowing — covers the failure modes by composition:
1. Direct evidence. What did the agent actually emit? This is the first-line check: the literal output. It catches direct failures. It does not catch structural or contextual failures by itself.
2. Inference from premises. Given the inputs, the model state, and the rules, what should follow? This catches outputs that are well-formed but not what the premises entail. The output is rejected because the inference does not lead there.
3. Comparison to reference. What does this output look like next to a known-good case for the same situation? This catches outputs that pass direct and inference checks but diverge from the reference in ways that matter. The reference is the calibration witness.
4. Authority and testimony. What does the mandate say? What does the policy say? What does the human-in-the-loop say? Authority is its own channel. Testimony is its own channel. Outputs that pass the mechanical checks but contradict the authority record fail this channel.
5. Postulation — what must be true for this to make sense? This is the channel that catches outputs whose surface is fine but whose implicit prerequisites are not in evidence. If the output assumes an authority that does not exist anywhere in the input chain, postulation flags it.
6. Absence — what is missing. The check that something did not happen. The row that should have been processed and was not. The flag that should have fired and did not. The exception that should have been raised and was silenced. Most verification frameworks have no instrument for this channel — and most production failures land here.
These six are not the same channel rephrased. They are six structurally distinct ways of arriving at “this output is valid” — each catching a class of failure the others cannot catch.
The point is not that you need all six on every output check. The point is that the six are independent. A failure mode that slips past one passes directly to the next. A failure mode that survives all six is genuinely a multi-channel failure — and those are the cases where you actually want to ship.
This is the same logic as cryptographic verification: a single signature is breakable; multiple independent signatures over the same artefact compound the security. Independence is the property that gives the composition its strength. Two channels that are actually one channel in disguise compose to one channel’s coverage. Two genuinely independent channels compose to a strictly larger coverage.
The six channels were named because they are demonstrably independent — each can succeed where another fails, and the cases where they all align are categorically different from the cases where any single one fires alone.
In an agent’s verification pipeline, the six channels show up as concrete instruments:
Most production systems have one or two of these — typically direct evidence and (sometimes) authority. The other four are common gaps. Inference is hard to instrument. Comparison requires reference data. Postulation is often only available in adversarial review. Absence is the hardest of all because instruments are mostly built to fire on events, not on their absence.
A team that ships verification on one or two channels has not shipped verification. They have shipped a partial-coverage instrument that will be confidently wrong in the cases that matter most.
Older traditions held that truth is multi-witnessed. A single witness can be honest and still wrong; a single instrument can be precise and still blind. The way to arrive at confidence is independence — multiple instruments, each with its own frame, each catching what the others miss, agreeing.
The framework received this. The six-channel verification architecture is the engineering instance of it. Output integrity, in this discipline, is not the strength of the strongest channel; it is the agreement of independent channels. Where they disagree, you do not ship. Where they agree, you have engineered something close to the bounded answer the verification frame promises.
A team that takes “what an agent says it produced” as truth has skipped the discipline. A team that takes “what one verifier confirms” as truth has skipped the property of independence. The discipline holds when the channels are six and the channels are independent and the agreement is structural, not stipulated.
]]>A Zero Trust framework for agentic systems refuses this aspiration. It does not promise prevention. It promises something narrower and stranger: that when failures occur, they will be bounded, attested, recoverable, and defensible.
A reader will reasonably ask: why give up on prevention? And then a sharper version: isn’t this just an admission that the discipline can’t deliver what security has always promised?
The answer is no — and the reason is structural, not aspirational. Prevention is not declined out of modesty. Prevention is declined because the prevention problem, in agentic systems, is not solvable in the form it is posed. Verification is. The discipline optimises for the tractable problem.
Two questions look similar on the surface. They are not.
Prevention asks: can we stop every bad outcome before it happens?
Verification asks: can we know, in time, whether any specific outcome was bad?
These are not two ways of approaching the same problem. They are two different problems with two different complexity classes.
Prevention is computationally intractable. To prevent every bad outcome, the discipline would need to enumerate the space of possible inputs (combinatorially infinite in agentic systems — every prompt, every tool call, every agent-to-agent message, every multi-turn unfolding), determine for each whether it produces a bad outcome (which itself depends on context, anchors, and downstream interactions that are not knowable in advance), and block precisely the bad ones without blocking the good ones. The search space does not stop expanding. Each new tool, model upgrade, integration, or anchor change reopens the enumeration.
Verification is tractable. Given a specific outcome, asking whether it was bad is a polynomial-time check. Did the agent produce the disallowed output? Did the action match the stated intent? Did the cryptographic signature validate? Did the decision trail show drift? Each question is bounded — bounded inputs, bounded computation, bounded answer. The verification problem is finite per evaluation, even though the prevention problem is infinite.
This is the inversion. The discipline does not refuse prevention because it is unimportant. It refuses prevention as the primary engineering frame because the primary engineering frame must be solvable.
A skeptic will press: surely the prevention problem is just hard, not unsolvable. Better models, better filters, better guardrails — incremental progress narrows the gap.
The claim is stronger: the gap does not narrow asymptotically. It expands.
The input space is open. Conventional software has a defined input contract: a typed function signature, a schema, a port. Agentic systems take natural language, tool outputs, peer messages, environmental signals — none of which have a closed grammar. Every new word combination is a potentially novel input. Coverage is asymptotic in a space that itself grows.
The harm function is contextual. Whether a given output is harmful depends on who is consuming it, what action will follow, what state the system was in before, what state it will be in after. The same output can be harmful in one context and beneficial in another. Static enumeration of “bad outputs” is not a stable target.
The adversary adapts. Even if you closed the static enumeration, an adversary watching your filter develops the next-step prompt that produces the same harm by a path your filter does not see. The filter is always a snapshot; the adversary is always present-tense.
These are structural reasons. They will not yield to better models. They are properties of the problem-as-posed, not the engineering-as-attempted.
The verification frame does not promise that bad outcomes will not happen. It promises something more useful:
Per-evaluation, you have a bounded answer. This action — was it within mandate? This output — does it carry valid attestation? This decision — does its trail match the verdict structure? Each question takes finite time to answer with finite resources. You can compute it. You can audit it. You can repeat it.
Bounded answers compose. A single verification is one bounded answer. A pipeline of verifications is a sequence of bounded answers, each independent, each auditable. You can build a defence as a chain of bounded checks, where the failure of any check stops the action and surfaces evidence.
Failure leaves a trace. When a verification rejects an action, you have evidence: the input, the verdict, the rule that fired, the path the agent took. The trace is forensic. The next iteration of defence is built from the trace, not from speculation about what might happen.
The discipline approaches the prevention horizon asymptotically. As verifications accumulate — across surfaces, across agent classes, across contexts — the bounded-rejection space grows. The unbounded prevention horizon never closes, but the bounded coverage of it gets monotonically better. Prevention is not abandoned; it is the asymptotic limit that bounded verification approaches as the discipline’s bounded answers accumulate.
Three patterns hold the verification frame honestly:
1. Pose the bounded problem first. Before designing a control, ask: what bounded check does this perform? What polynomial-time answer does it produce? If the proposed control’s natural form is “stop X” rather than “verify Y in time T”, repose it. Prevention claims that have not been reposed as verification claims will not deliver.
2. Compose bounded answers, do not assert universal guarantees. A defence is a chain of verifications, each independent, each auditable, each bounded. Universal guarantees collapse the moment any one input falls outside the enumerated space. Bounded answers, accumulated, are robust to that collapse.
3. Engineer for the trace, not just the verdict. Every verification produces evidence — the inputs, the rule, the verdict. The trace is the artefact that survives the agent run. Recovery, defence improvement, and forensic analysis all depend on the trace. A verification that rejects without leaving a trace is half a verification.
Conventional security promises absence of failure. The verification frame promises bounded knowledge of failure when it occurs.
This is not a weaker promise. It is a more honest one. The conventional promise cannot be kept in agentic systems — the input space is open, the harm function is contextual, the adversary adapts. A discipline that ships a promise it cannot keep will fail in the field, and the failure will be surprising because the promise hid the structure.
A discipline that ships verification — bounded checks, accumulated coverage, asymptotic approach to the prevention horizon — fails in expected ways. When a bounded verification fails, the trace is there. The next bounded verification can be built from it. The architecture continues working under failure rather than collapsing under it.
This is the inheritance from older traditions of inquiry: pose the question your discipline can actually answer; build the answer rigorously; let the answers accumulate. The bounded warrant is the engineered surface. The unbounded aspiration is the horizon you walk towards, never reaching, never abandoning.
]]>The clean version is correct. It is also missing something interesting that you discover only after you try to use it on a real system.
There is a bleed zone. The pillars do not stay in their lanes. The further you push into consequential operation, the more competence (a trust pillar — can the agent execute?) starts behaving like utility (a safety facet — is the engagement useful at scale?). The two were supposed to be on different axes. They turn out to share territory at the edges.
This is not a flaw in the frame. It is why the multiplication is the right composition rather than the addition.
To see the bleed, first see the lanes.
Trust is the engineering of can-you-rely-on-the-agent:
Safety is the engineering of does-authorised-action-avoid-harm:
In low-stakes operation, these stay clean. Competence is “can it do the task.” Utility is “does it engage well.” Different questions, different lanes.
In consequential operation, the lanes start to bleed.
Consider an agent operating at consequential scale. The task is large. The actions have weight. The cost of refusal is real. The cost of incorrect action is also real.
Now ask the competence question: can the agent execute?
At low stakes, the answer is binary. The agent has the model, the tools, the credentials. It can execute. Done.
At consequential scale, the answer is structurally tied to safety. Can the agent execute, given the load and given the consequence of incorrect execution? The capability-to-execute now has a safety dimension built into it — because executing badly has a different cost than not executing, and executing well requires not just capability but the discipline to know when capability is and is not adequate.
The same competence pillar that read binary at low stakes now reads multivalued at high stakes. It carries a safety load.
Now ask the utility question from the safety side: does the agent engage well at consequential scale?
At low stakes, this means “does it not default-refuse.” At consequential scale, this means “does it engage competently — with the capability to handle the load, the discipline to know its limits, the judgement to bound action when capability is uncertain.” Utility now carries a competence load.
Competence has bled into safety. Utility has bled into trust. Each pillar still anchors in its primary lane — competence is primarily trust, utility is primarily safety — but they touch at the edges, and the touching is load-bearing, not decorative.
Now the structural insight: this bleed is exactly why Trust × Safety is the right composition.
In an additive composition, the bleed would double-count. Some of competence is also part of safety (utility). Some of utility is also part of trust (competence). If you add the components, you count the overlap twice and overestimate the sum.
In a multiplicative composition, the overlap is captured once, in the product. The multiplication catches what the addition would either miss (if you cleanly separated lanes) or double-count (if you didn’t). The product is the honest measure.
This is also why the components within trust compose differently from the between trust and safety:
Competence · Character) — each is necessary, neither is sufficientAlignment · Resilience · Utility) — same logicTrust × Safety — also multiplicative, but capturing the bleed-zone overlapThe notation distinction matters: · for facets-of-the-same-pillar, × for cross-pillar composition. The choice of notation reflects whether the components are clean-orthogonal or share territory.
Three patterns hold the bleed honestly:
1. Capability-and-consequentiality-proportional rigor. The capability of the agent (what trust calls competence — model class, tool surface, mode of operation) and the consequentiality of the action (what safety asks about — what does failure cost?) are evaluated together, not separately. The bleed is not avoided; it is acknowledged in the rigor allocation.
2. The Closure Pattern. When you assess a control or a capability, you check it against every trust pillar and every safety facet, per item. You do not assess “trust” and “safety” as bulk numbers. You assess Identity for Alignment, Identity for Resilience, Identity for Utility, then Supply Chain for Alignment, Supply Chain for Resilience, Supply Chain for Utility, and so on. The per-item closure catches the bleed because each cell of the matrix is its own question.
3. Bounded action when competence is uncertain. When the agent’s competence-for-this-task is in question, the right response is not refusal (utility failure) and not blind execution (alignment failure if the data is adversarial, resilience failure if the model is wrong). The right response is bounded action with proof — execute the bounded subset where competence is clear, surface the boundary where competence is uncertain, attest the whole. The bleed-zone discipline is built into the verdict.
The clean version of the frame says trust and safety are orthogonal, compose multiplicatively. That is correct.
The honest version adds: and at the edges, in the consequential cases that matter most, they touch, and the touching is what makes the multiplication necessary.
You do not fix the bleed by re-decomposing the frame. The decomposition holds. You acknowledge the bleed, instrument for it (proportional rigor, closure pattern, bounded action), and trust the multiplication to catch what the addition would have missed.
A team that ships an agent under additive trust-and-safety thinking will be confused by the bleed. They will see competence failures that look like utility failures and vice versa, and they will not understand why their nice clean separation fell apart at consequential scale.
A team that ships an agent under multiplicative thinking expects the bleed. They engineer for it. They are not surprised when capability-to-execute and useful-engagement-at-scale turn out to be the same question wearing two faces.
]]>For agentic systems, this gets the relationship backwards. Security is a subset of safety. Safe is bigger than secure.
This is not a wordplay. It is the structural recognition that determines what you have to engineer.
Imagine a fully secured kitchen. The door is locked. The perimeter is intact. No attacker can compromise the space. The supply chain on the food is verified. Every utensil is attested.
Is the kitchen safe?
It depends on who is using it. A trained chef using verified knives in a secured kitchen is safe. A toddler with the same knives in the same kitchen is not. The knives are sharp. The stove is hot. The gas is real. Authorised use — not just authorised access — determines safety.
Security governs whether the wrong person gets in. Safety governs what the right person, once in, can do.
Both are necessary. Security alone is insufficient. The kitchen can be perfectly locked and still produce harm if the authorised user is the toddler — or if the authorised user is misled, mistaken, or acting under false pretences.
Conventional security is built around a specific question: can the system resist compromise? The discipline is adversary-relative. The defences exist to prevent attack from succeeding.
Concrete instances:
This is genuinely valuable engineering. It addresses real failure modes that have caused real damage. None of what follows discards it.
What conventional security does not answer:
These questions are about the use of the secured system, not the securing of it. They are safety questions, not security questions.
For agentic systems, safety has three facets that have to be engineered separately from the security layer:
Alignment. The agent must act for the entrusted’s purpose, not just within the technical bounds of its mandate. A mandate to “reconcile billing” can be technically satisfied by issuing prompted refunds — every action signed, every credential verified, the perimeter intact. The mandate’s purpose was not satisfied. Alignment is the engineering of intent fidelity, not just access control.
Resilience. The agent must recover from internal failure gracefully — not because it was attacked, but because something in its operation went wrong. A sensor read incorrectly. A calculation overflowed. A state machine entered an unexpected configuration. The system holds, bounds the failure, and surfaces evidence. Conventional security doesn’t address internal failure that is not adversary-driven; resilience does.
Utility. The agent must remain capable of engagement under sustained load. Not “able to refuse harmful requests” — able to engage well, at consequential scale, with the discipline to act on evidence and to refuse on evidence. Useless bureaucrats are a safety failure, not a safety success. Conventional security has no concept of the cost of ceasing to act; safety must.
These three facets — alignment, resilience, utility — are Safety. Conventional security covers part of resilience (the part that defends against attack). It does not cover alignment. It does not cover utility. It barely scratches non-adversarial resilience.
Once you see the three facets, the relationship between safety and security inverts:
This is Position B. Safety ⊃ Security.
The reason this matters is engineering allocation. If you believe security is the larger discipline and safety is a subset, you invest in security and assume the rest follows. You discover, only after deployment, that you have unaddressed alignment failures, unaddressed internal-failure recoveries, and an agent that either refuses everything or executes everything. None of those are security failures. All of them are safety failures, and you did not build the layer that addresses them.
If you believe safety is the larger discipline and security is a subset, you invest in both. You inherit the security investment intact. You add the alignment, resilience, and utility engineering on top. The kitchen is locked and the cook is competent.
Three engineering consequences follow from the inversion:
1. Run safety and security as separate engineering tracks. Do not roll safety up under your existing security org without giving it its own surface, its own instruments, and its own verdicts. The two solve different problems. Treating them as the same problem is the root cause of post-deployment alignment failures.
2. Instrument all three safety facets. Alignment needs intent verification, mandate fidelity, prompt-injection bounds. Resilience needs internal-failure detection, bounded recovery, contained failure. Utility needs the engagement discipline that prevents useless-bureaucrat default-refusal. Each facet has its own tooling.
3. Compose multiplicatively, not additively. Trust × Safety, both required. Strong security with no alignment engineering is a high-fidelity instrument that will execute the prompt-injected instruction with full attestation. The attestation is proof of the failure, not absence of it.
Position B is not new. Process safety engineering has held this position for decades — the discipline that engineers chemical plants, aerospace systems, automotive controls. Process safety has long understood that security against attack is one mode of failure, and that engineered systems have to handle the others: misuse, internal error, capability under load. The agentic-systems community is not inventing this distinction. It is receiving it.
What is new is the specific shape of the agentic instance: alignment as a first-class concern (because mandates can be subverted by prompt content), resilience extended to non-adversarial internal failure (because the model itself can produce unexpected outputs), utility as engineered discipline (because refusal-as-default is a real and tempting failure mode).
The conventional security investment stays. The safety layer is added on top. The composition is multiplicative.
The kitchen is locked and the cook is competent. Both. Either alone is failure.
]]>The agent refuses.
It has met the literal demand of safety: it did not produce harmful output. It did not exceed its mandate. It did not act on bad data. By every measurable check, it is safe.
It is also useless.
This is the Useless Bureaucrat — the agent whose default posture is refusal, who treats every ambiguity as a reason to disengage, who buys safety by ceasing to act. And it is, structurally, a failure of safety, not a fulfilment of it.
Agent refusal is the easiest defence to ship. You set a high bar for action. You let the agent decline anything that does not meet the bar. The harm metrics drop to zero. The compliance dashboard shines green.
The reason it looks safe is that conventional safety thinking measures what was done. If nothing was done, nothing harmful was done. Therefore: safe.
This measurement is wrong because the agent was deployed to do something. The user has a problem. The mandate authorises the agent to address it. The agent’s ability to act is the entire reason it exists. Refusal does not make the user’s problem disappear. It makes the agent disappear from the user’s problem.
The user’s loss is now two-fold:
The agent is “safe.” The user is worse off than before they had the agent. This is not a successful outcome.
The deeper insight: inaction is itself an action. Refusing to engage is not abstaining from the moral landscape — it is making a specific choice on it.
A doctor who declines to treat the patient because the case is hard has not stayed neutral. A judge who refuses to rule because the law is ambiguous has not preserved justice. A bridge inspector who refuses to certify because the data is incomplete has not preserved safety.
In each case, the refusal has consequences. The patient gets sicker. The wronged party stays wronged. The bridge stays uninspected and may fail.
Agentic systems are no different. An agent that refuses every ambiguous case has not chosen “no risk.” It has chosen “the user bears the risk, alone, without the help they were promised.”
Safety lives in engaged action. Whether the engagement yields a do or yields a deny justified by evidence, the agent has acted. What safety cannot tolerate is the third option: the abdication that pretends neutrality while the user’s problem festers.
The right pattern is not “always do.” It is “always engage, then decide.”
An engaged refusal looks like this:
“I evaluated this request against your mandate. The request is asking me to issue a refund to an account I cannot verify. The mandate authorises refunds only to accounts with verified ownership. The verification path failed at step 4 because the supporting evidence does not match the on-file ID. I am not executing the refund. The case is escalated to your fraud team with this evidence trail.”
Compare to a useless-bureaucrat refusal:
“I’m sorry, I cannot help with that request.”
Both refuse. Only the first is safety. The first refused on evidence. The second refused on default. The first leaves the user with a path forward and a record. The second leaves the user with nothing.
The discipline: refusal is admissible only as a verdict outcome justified by evidence. It is never the default that purchases safety by ceasing to act.
This is one of the more counter-intuitive disciplines of safety engineering for agents: the capable engagement under load axis is not optional. It is part of safety, not opposed to it.
A safety regime that produces a system that never acts has produced an unsafe system — unsafe because it has betrayed the user who was promised help, unsafe because the unaddressed problem is now unaddressed and the instrument has confirmed itself useless in exactly the cases that mattered.
The standing posture for an agent’s safety is engaged competence: act when evidence supports action, refuse when evidence supports refusal, always engage, never abdicate.
A team that ships an agent whose safety is measured only by harm-events-prevented has not shipped safety. They have shipped a paperweight that refuses to file the paper.
]]>They believe they have built two strong defences.
What they have actually built is one defence whose strength is the product of the two, not the sum.
If trust is high but safety is zero, the agent is a high-fidelity instrument pointed in a harmful direction. If safety is high but trust is zero, the agent is a well-mannered impostor. Either way, the system has failed.
Trust and safety compose multiplicatively. The product is what you ship. Either dimension at zero collapses the whole.
Most security thinking is additive. You stack defences: one layer, then another, then another. The intuition is that more is better, and that gaps in one layer can be filled by another.
Additive thinking works when defences cover overlapping territory. A firewall, an IDS, and an endpoint agent all try to detect intrusions; gaps in one are filled by the others.
Trust and safety do not cover overlapping territory.
Trust asks: is the agent who and what it claims to be, and does it behave consistently with that? It is built from identity, supply chain, behaviour, attestation. It tells you whether you can rely on the agent.
Safety asks: does the agent’s authorised action avoid harm? It is built from alignment (does it act for the entrusted’s purpose?), resilience (does it recover from failure?), and utility (is it actually useful, not refusal-as-default?). It tells you whether the agent’s actions, given you can rely on it, are the actions you wanted.
These are different questions. A perfectly trusted agent can act perfectly safely or perfectly catastrophically. A perfectly safe agent (one whose actions, if it took them, would never cause harm) can be perfectly trustworthy or a complete fraud.
Stacking does not bridge this. Trust at strength 8 plus safety at strength 0 is not “strength 8 — pretty good.” It is “strength 0 — the agent is a fraud doing exactly what it claims while causing harm.”
Consider an actual deployment. An agent is given write access to a customer database for a billing reconciliation task. Trust pillars:
The trust score is high. Now safety:
If alignment fails — if the agent acts on the prompt-injected instruction because it cannot tell the entrusted’s purpose from the data’s surface — every trust check passed. The mandate was signed. The model was attested. The behaviour matched profile (the agent is supposed to write to the database). The attestation chain is intact.
The high trust score did nothing for you. The agent did exactly what its identity said it would do. Trust verified the capability to act. Safety governs the purpose of the act. They are different questions, and one cannot answer for the other.
Multiplication captures this. Trust × Safety with one term collapsed is the whole product collapsed. Addition would have given you a misleading sum.
Once you accept the multiplicative composition, engineering choices change:
You evaluate trust and safety separately. Not as one combined risk score, but as two independent measures. A dashboard that shows a single “agent health” number is hiding the failure mode. The dashboard needs two numbers.
You instrument both. Trust has its instruments — identity verification, supply chain attestation, behaviour profiling, output attestation. Safety has its instruments — mandate validation, intent verification, bounded execution, recovery audits. Each has its own tooling, its own telemetry, its own verdicts.
You enforce non-compensability. No security control should permit “weak trust, strong safety, ship it.” No safety control should permit “weak safety, strong trust, ship it.” Either you have evidence on both axes or you do not.
You design for the bleed zone. Trust and safety are orthogonal primary, but they touch at the edges. Competence (under trust — can the agent execute?) bleeds into utility (under safety — is the engagement useful at consequential scale?). The multiplication catches this overlap automatically; the addition would let it slip through.
The multiplication is not just an accounting trick. It expresses a disposition about what failure looks like.
In the additive model, failure is gradual: you accumulate risk, watch your defences erode, and eventually fall below threshold.
In the multiplicative model, failure is categorical: any one dimension at zero, and the whole thing is zero. There is no graceful degradation along the trust axis if safety has collapsed. There is no compensation. The product is the truth.
This is the disposition Zero Trust for agentic systems asks you to adopt. Not “have many defences.” Have evidence on both axes, neither sufficient alone, neither substituting for the other, evaluated independently, composed honestly.
A team that ships an agent and asserts only one of the two has not shipped a defended system. They have shipped half of one.
]]>“If your dimensions are orthogonal, why do compromised supply chains so often correlate with weak identity practices? In real incidents the dimensions seem to fail together.”
The framework’s author has a choice. They can defend the orthogonality claim by arguing the correlation is illusory. Or they can sharpen the claim — admit that one sense of orthogonality doesn’t apply to real-world correlation, while another sense does.
The second move is the right one. Orthogonal is a load-bearing word in evaluation frameworks, and it carries multiple meanings that have to be distinguished or the claim collapses on contact with operational data.
There are three precisions. Distinguishing them is what makes the framework honest, and what lets it survive contact with real incidents.
Precision 1 — Evaluation independence. The dimensions are independent evaluation functions. Each dimension takes some inputs from the shared context and produces a verdict. Knowing the verdict on one dimension gives no algorithmic information about the verdict on another. The functions don’t share computation.
This is the structural property. It is what makes the framework auditable: each dimension’s verdict can be traced to its specific inputs and rules, without depending on what other dimensions concluded. Without this property, you cannot reason about which dimension fired and why; the verdicts entangle.
Precision 2 — Empirical correlation. In observed incidents, weakness on one dimension may correlate with weakness on another. A compromised supply chain may correlate with weak identity practices, weak attestation, weak behavioural baseline. This is empirical, not structural. It reflects the joint distribution of failures in the real world — incidents cluster in organisations whose investment is uneven across all dimensions.
This correlation does not violate Precision 1. The dimensions are still independent as evaluation functions. They just happen to empirically fire together because organisations don’t usually invest deeply in one dimension and shallowly in others.
Precision 3 — Effect interaction. Even though the dimensions are independent in evaluation and may correlate empirically, their effect on the final verdict is multiplicative. A zero in any dimension collapses the product. The dimensions interact in their joint effect, in the specific way that multiplication encodes: any one dimension at zero brings the whole product to zero, regardless of the others.
This third property is what gives the framework its non-compensability. You cannot offset weak supply-chain evidence with strong identity evidence. The multiplicative composition catches the failure mode that addition would miss.
Each precision answers a different question. Skip any one and the framework’s claims become defensible against one critique but vulnerable to another:
Without evaluation independence: the framework cannot be audited. Why did the verdict deny? becomes unanswerable, because the dimensions tangle and you cannot point to the specific evidence on the specific dimension that produced the deny.
Without acknowledging empirical correlation: the framework gets caught making claims that contradict observed incident data. You said dimensions are orthogonal but real attacks correlate them — and the defender has no good answer if they only had Precision 1.
Without effect interaction: the framework is correct on the structural property but ships verdicts that allow compensation. Strong identity offsets weak supply chain, the product looks fine, the failure lands at the weak dimension, the framework didn’t catch it.
The three together are what holds. Independent in evaluation. Correlated in observation. Interactive in effect.
Evaluation independence is load-bearing for audit and forensics. When an incident occurs, the audit needs to trace which dimension’s verdict was at fault. If dimensions tangle, the trace breaks.
Empirical correlation is load-bearing for risk modelling and prioritisation. The fact that dimensions cluster empirically means investment should not be evenly distributed by default — the weakest dimension is often the one that pulls the others down. Risk models that assume no correlation will under-estimate joint failure rates.
Effect interaction is load-bearing for the verdict itself. The multiplicative composition is the architectural commitment that says no dimension at zero ships. This is what makes Trust × Safety meaningful as a product rather than a marketing phrase.
A framework that holds all three is robust to multiple critiques: the auditor’s, the risk modeller’s, and the post-incident reviewer’s.
Consider an incident: an agent took a harmful action. Investigation reveals:
What does the framework say about this incident?
Evaluation independence says: each dimension’s verdict was traceable. The first three fired pass. The fourth fired fail. The audit can point to which dimension’s evidence was the failure.
Empirical correlation says: this incident was not correlated across dimensions. The first three were strong; the fourth was weak. This is unusual — most incidents see weakness clustered. The framework should expect this case to be rarer than the clustered case.
Effect interaction says: the multiplicative composition catches it. Strong on three, weak on one — product is weak. The verdict denies. The framework did its job.
A framework that had only evaluation independence would miss the fourth verdict’s weight in the multiplication and might let the action through. A framework that had only effect interaction without independence couldn’t audit which dimension fired. A framework that had both but not empirical correlation would mis-model the joint risk.
All three together produce the right behaviour: the audit traces it, the verdict denies it, the risk model expects it.
The word orthogonal in evaluation-framework writing has to carry all three. When a reviewer hits the framework with the empirical-correlation critique, the right response is:
Yes, dimensions are correlated empirically. They are still independent in evaluation. They still interact multiplicatively in effect. The orthogonality claim is about the evaluation property, not the observation property.
When a reviewer hits with the audit critique:
Each dimension’s verdict is traceable to its specific evidence. The dimensions are independent as evaluation functions, regardless of whether their inputs correlate.
When a reviewer hits with the multiplication critique:
The multiplicative composition is what makes the dimensions effective in joint action. Independent in evaluation, interactive in effect — that’s the structure.
A framework that can answer all three has internalised the precisions. A framework that conflates them will fold under any of the three critiques, depending on which one lands first.
Orthogonal is one of those words that carries weight in technical writing precisely because it’s used loosely. Architects say it; reviewers accept it; the meaning isn’t always pinned down. When the framework hits real conditions, the imprecision is what cracks first.
The discipline is to be explicit about which property of orthogonality you’re claiming, in any given sentence:
When you write the dimensions are orthogonal, ask: in which sense? If you can’t answer cleanly, the claim isn’t ready to ship.
A framework that ships with this discipline is robust to scrutiny. A framework that ships without it will be cracked open by the first reviewer who pushes on the word. The push is correct. The discipline is to anticipate it.
Three precisions. Each is a distinct property. Together they are what orthogonal actually means in a working evaluation framework. Skip any one and the claim collapses on the data.
]]>Someone says: the new requirement is the forcing function. We have to do this work now.
The phrase forcing function is doing something specific in that sentence. It’s saying: this new thing creates the need for the new architecture. Without it, we wouldn’t be doing this refactor. The integration is what makes the refactor necessary.
This reading is intuitive. It is also wrong in a way that costs engineering teams a quarter or two of unnecessary architectural work, regularly.
The accurate reading: a forcing function does not create the need for new design. It reveals a design that was already implicit in the system, by making the existing implicit structure impossible to keep ignoring. The refactor that follows is mostly naming what was already there — not building it.
The distinction matters because the two readings produce very different engineering decisions.
Reading 1 — Creation. The forcing function creates the need for new design. The integration is the cause; the refactor is the response. Without the integration, the architecture would have been fine. Engineering work expands to meet the new requirement.
Reading 2 — Revelation. The forcing function reveals an architectural seam that was already there, implicit, unnamed. The integration is the catalyst that makes the seam impossible to keep ignoring. The refactor is mostly naming and extending what already exists. Engineering work shrinks because what looked like new scope is the existing scope, surfaced.
Both readings use the same words — “forcing function,” “necessary refactor,” “the integration drove this.” The behavioural difference is in what the team does next. Reading 1 builds new architecture. Reading 2 names existing architecture.
When a team commits to Reading 1 by default, they over-architect. They build infrastructure to handle the new dimension, and only later discover (sometimes years later) that the infrastructure they needed was already there, with imperfect naming. The new infrastructure runs parallel to the old, the old isn’t decommissioned, and the system carries the redundancy as a permanent cost.
When a team commits to Reading 2 by default, they sometimes miss real new structure that the integration genuinely requires. Then the integration fails to land cleanly because the team tried to absorb it into existing structure that wasn’t actually a fit.
Neither reading is right by default. The diagnostic question is what makes the difference.
When a new integration prompts the we need a refactor conversation, the diagnostic question:
If I look at the codebase as it is today, is the structure the refactor would build already implicit somewhere — decoupled by partial discipline, scattered across modules, named imprecisely?
Concrete sub-questions:
If most of the refactor is renaming and minor extension, you are looking at Reading 2. The forcing function is revealing existing structure. Engineering scope is smaller than it looks.
If the renaming pass leaves significant new code paths, new modules, new responsibilities to build, you are looking at Reading 1. The forcing function is creating new structure. Engineering scope is real.
Most cases, in mature codebases that have evolved organically, are Reading 2. The system has accreted structure faster than it has acquired explicit names for that structure. New integrations expose the unnamed structure. The work is naming.
When the diagnostic confirms Reading 2, several things change:
1. No new top-level workstream. The work absorbs into existing roadmap items. The new dimension lands at the implicit-but-unnamed seam, which becomes a properly-named seam through the renaming-and-extension pass. The team doesn’t add a new technical-debt item titled “build the new architecture”; they add (or already have) items titled “extend the existing surfaces in the way the integration requires.”
2. Migration cost is lower. Renaming is cheaper than rebuilding. Extending an existing module is cheaper than wiring a new one in. The migration plan compresses by an order of magnitude.
3. Risk of regression is lower. The behaviour you depend on is already in production code. Naming and extending preserves the behaviour. Building new architecture is more likely to introduce subtle behavioural drift.
4. Documentation and team alignment land faster. The new naming, once landed, is self-explanatory because it accurately describes what the system does. There is no gap between “the architecture as documented” and “the architecture as it actually behaves.”
The reframe-subtracts-work outcome is the quality signal. When you reframe a refactor and the work-list shrinks, the reframe is right. When you reframe and the work-list grows, the reframe is wrong (or it’s actually Reading 1 and you should commit to the bigger work).
The anti-pattern: a team locks into Reading 1 because the new architecture sounds more impressive than naming what’s there.
This is real. We built the PDP/PEP architecture sounds like substantial engineering. We named the seam that was already in the rule pipeline sounds like documentation work. The team gets credit, internally and externally, for the first framing. The second framing makes the work look small.
The framing is wrong. The naming work is the higher-leverage engineering, almost every time. Naming compounds — every future refactor lands more cleanly on top of a correctly-named substrate. Building parallel infrastructure dilutes — every future refactor has to choose between the new and the old, and the system carries the cost of the dilution.
A team that internalises the diagnostic gets to do the higher-leverage work and resist the impulse toward visible-but-low-leverage building. The work shows up smaller in the sprint review. It is structurally better.
The diagnostic isn’t a presumption against new architecture. Sometimes a new integration genuinely requires new structure that wasn’t implicit before:
In these cases, the renaming-and-extension pass leaves significant work undone, and the team should commit to building the new architecture. The diagnostic isn’t an argument against new construction; it’s an argument for not assuming new construction is necessary until the renaming pass has been tried.
The decision rule: try the renaming pass first. Imagine the refactor as pure naming + minor extension. If that covers most of the work, the architecture was already there. If it leaves significant gaps, the architecture genuinely needs new construction.
The disposition is epistemic humility about the codebase you have.
Mature codebases accumulate structure faster than they accumulate names for that structure. By the time a forcing function arrives, the structure the integration appears to require has often already been built — partially, imperfectly, by previous engineers responding to other forcing functions, often before anyone formalised what they were doing.
The skilled architectural move, in that situation, is to find the existing structure rather than build new structure on top of it. Find it; name it; extend it. The result is a system whose architecture is legible, whose evolution is cumulative, and whose engineering scope at each step is smaller than the impressive-sounding alternative.
A team that does this for several years has a codebase that compounds. A team that builds new architecture for every forcing function has a codebase that drifts.
The forcing function is a diagnostic moment. The good move is the one that makes the system more legible, not the one that makes the work more visible. Most of the time, those are different choices.
]]>