Process Safety Has Always Known This

When the agentic-systems community talks about “AI safety,” we talk like we are inventing the discipline. We are not. We are receiving it.

There is a discipline called process safety engineering. It has been built and stress-tested for the better part of a century, in chemical plants, refineries, aerospace systems, automotive control loops, nuclear facilities, and pharmaceutical pipelines. It addresses the engineering of complex systems against actual failure under actual load — not theoretical failure in an analytical model, but real failure that has caused real explosions, real releases, real deaths.

The methods that discipline developed — the failure-mode catalogues, the consequentiality scaling, the bounded-action principles — are exactly the methods agentic-systems safety needs. The structural recognitions that discipline arrived at — that security against attack is not the same as safety in operation, that absolute prevention is impossible, that engineering rigor must scale with consequence — are exactly the recognitions the agentic-systems community needs to absorb.

Most of what you would call “the structure of safety thinking for AI agents” is not new. It is process safety, applied to a new substrate.

What Process Safety Already Knew

Process safety as a discipline arrived at several recognitions decades ago. Each is directly applicable to agentic systems:

1. Failure-mode-first thinking. You don’t engineer safety by designing the happy path and adding mitigations. You start by enumerating every way the system can fail. Every step in the process, every component, every interaction — what can go wrong? What are the consequences? What signals would precede the failure? The whole engineering pipeline runs on the failure catalogue, not on the success path.

In process safety, this is HAZOP — Hazard and Operability Study. Systematic identification of how each step in a process can deviate from intended behaviour, with the consequences mapped. The agentic equivalent: threat modelling against agent action surfaces, mandate-violation enumeration, capability-class failure analysis. Not optional. Foundational.

2. Component-level failure analysis. Every component in a system can fail in multiple ways. Each failure mode has a downstream effect. The engineering surface needs that mapping.

In process safety, this is FMEA — Failure Modes and Effects Analysis. Aerospace and automotive picked it up; the agentic-systems version maps each control’s failure modes to the trust-and-safety surface they leak into when they fail.

3. Consequentiality scaling. Higher-consequence functions demand more rigorous engineering. A control loop that turns on a heater can be implemented with one rigor level. A control loop that releases a chemical is engineered to a different rigor level. The function’s required reliability scales with what its failure would cost.

This is SIL — Safety Integrity Levels — codified in IEC 61508 and downstream standards (ISO 26262 for automotive, DO-178C for aerospace). The agentic-systems instance: capability-and-consequentiality-proportional rigor. A defence valid for an information-only agent is not valid for a financial-action agent. The rigor level scales with the action class.

4. Bounded action over absolute prevention. Process safety recognises that absolute risk elimination is structurally impossible. The discipline aims at as low as reasonably practicable — bounded risk, bounded outcome under failure, contained consequence — not zero risk, which is unattainable in any real system.

This is ALARP. The agentic-systems instance: bounded execution, graduated verdicts, contained failure. The discipline does not promise no agent will ever produce harm. It promises that when an agent does, the harm is bounded, attested, recoverable, defensible.

5. Discipline against actual failure, not theoretical failure. Process safety methods earned their place by holding under real failure. They are grounded in receipts. The discipline does not deploy a method because it sounds reasonable in a paper; it deploys methods that have already held in production under conditions that broke other methods.

The agentic-systems community is at the early stage of this. We have analytical models. We have benchmarks. We are building the receipts. The receipts are what will earn the methods their place.

What This Means for AI Safety

The implications for engineering:

You do not invent safety methodology. You inherit it. The failure-mode-first discipline is older than computing. The consequentiality-scaling discipline is codified in international standards. The bounded-action discipline is a load-bearing principle in every safety-critical industry. If a proposed AI safety control does not specify what failure mode it catches, what consequentiality class it engineers for, and what bounded outcome it produces when it fires, the proposal has not done the inheritable work.

You do not promise absolute prevention. Promising absolute prevention in agentic systems is making a claim that process safety has known to be impossible since long before AI existed. The honest claim is bounded outcome under failure — and that is what the discipline can actually deliver.

You do not collapse safety into security. Process safety is broader than security against compromise. It includes alignment with operational purpose (the system does what it was deployed for), resilience to internal failure (not just external attack), and capability under load (the system holds together in operation, not just defended condition). Treating “AI safety” as “AI security” is the categorical error that process safety solved decades ago for industrial systems.

What the Framework Does With This

The framework’s safety surface is the agentic-systems instance of process safety methodology:

• The threat-modelling catalogue is HAZOP for agent action surfaces
• The control failure-mode analysis is FMEA per control
• The capability-and-consequentiality-proportional rigor is SIL applied to agent capability class
• The bounded execution + graduated verdicts is ALARP for agent action
• The forensic ledger and observatory architecture is the trace discipline that lets methods earn their place by holding under actual failure

Naming the inheritance does not make the agentic-systems work less original. The agent-specific surfaces, threat classes, and bounding mechanisms are agent-specific. What is not agent-specific is the structural disciplines — those were earned, and they belong to whichever community needs them.

A team approaching agentic safety as if the discipline starts here will spend years rediscovering what process safety already wrote down. A team approaching agentic safety as the latest substrate the discipline has had to engineer for will save those years.

The Standing Posture

Process safety did not solve the agentic-systems problem. It solved the generic complex-system safety problem across several substrates. We are now another substrate.

The standing posture: students of process safety, not its peers. We inherit the methods. We adapt them where the substrate genuinely requires adaptation. We do not pretend the lineage doesn’t exist.

The discipline that holds together under load — that holds together against actual harm, against actual failure, against actual death in industrial settings — is the discipline that the agentic-systems community is the latest inheritor of. Treating that inheritance with the standing posture it deserves is itself part of doing the work.