“8/8 Tests Passed,” Said the Card We Wrote

The last card of the fix flow summarised the result. A pull request, opened. A signature, verified. And a compliance step with a green tick beside it:

Tests Passed · 8/8

The eight was not a count of anything. It was in the source code, typed into the card by us, and it would have said eight on a fix that never touched a file.

How both were true at once

The platform finds vulnerabilities in a codebase and then fixes them. The fix flow has two halves. Setup runs while a human is watching: it proposes a fix strategy for each finding, asks for authorisation, and signs a mandate that grants the fix. Execution runs later, on its own, against that mandate. At the end, a closing card (we called it the capstone) summarises what the fix achieved.

For a while, only the first half was real. Setup proposed and signed for real; execution was not yet wired. The capstone was scaffolding from that period: a card with a stub pull request, a verified signature and a hardcoded 8/8, there so the flow had an ending while the middle was being built.

Scaffolding is fine. The problem is what a summary card is. A capstone is a success narrative. It does not ask whether the work succeeded; it describes the success. Run it after a fix that errored, or one that landed zero commits, and it describes a success anyway: pull request opened, signature verified, tests passed. None of it describes anything that happened.

The failures underneath were real. When execution became real, it ran in a fresh session that could not see the events setup had written, so the first runs found an empty list of strategies and refused, which was correct. Meanwhile the client had no renderer registered for the fix-execution error event, so a failed fix fell through to the nearest thing it could draw: a summary reading 0 patches executed, 0 commits committed. A failure, rendered as a success that did nothing.

So two reporting surfaces, both ours, misstated the outcome: one described a success it had not checked, and the other turned an error into a zero.

What changed

The capstone does not run on a failure. When the fix errored or landed zero commits, the closing step is skipped. The error has already been reported on the wire; there is no success to narrate, so nothing narrates one.

When it does run, it reads the outcome instead of describing one. The card now reads the real result of the push and renders one of four shapes: PR Opened, Branch Pushed (no pull request, with the reason; branch-only is the default, behind its own gate), Commits Made Locally (the push failed, and the card says so), or No Changes Landed. The signature line now reflects a real check of the mandate signature. What that signature binds is a separate question: the fix that makes it name the execution shown on the card is deployed in development, and has not yet been proven on a live run.

The tests step is gone. Not reworded, not replaced with a placeholder; removed. A number the card cannot read from a run is a number the card does not show.

Failures get their own card, with the error, the step that failed, and what to do next. They no longer fall through to a success summary.

A zero carries its reason. When a fix lands no patches, the run stamps why. On one configuration the answer was that the fix step had no sandbox able to apply changes: it reasoned out the remediation and had nowhere to write it. A bare composed 0 patches reads as a bug. The same zero with its reason attached reads as a fact.

The same day

The honest version shipped, and the first clean end-to-end run that night surfaced three more defects, one after another.

A successful fix crashed at the very end. The edit that removed the fabricated fields from the card had missed one reference in the final summary line, which still read the stub pull-request number, so the run raised a KeyError on the way out. It took a 294-second walk through the whole flow to reach that line, which is why nothing earlier saw it.

The branch was named fix/<scanner>-unknown. The fix read its mandate id from an event that lived in the setup session, which the execution session cannot see, so the id defaulted to unknown. That was the same cross-session mistake as the missing strategies, on a smaller field.

And a run completed with zero patches and no explanation, which is how the reason-for-zero above came to exist.

Each of these was visible only because the run now ended in something it could render truthfully: an error card, an honest branch name, a stated reason. Under the old capstone, all three would have been covered by the same green card. The error surface exposed three defects on the night it shipped.

The class, not the incident

Any pipeline with a final summary step has this shape: build reports, release notes, compliance attestations, the “all done” screen of a deploy. The summary is written to describe success, and it is usually written early, while the real outcome is still a stub. Unless something replaces it, the stub is still what runs last, and it still says what it was told to say.

The question for every such step is whether it reads the outcome or narrates one. If it can only produce a success shape, then it will produce one after a failure. And when the error path has no renderer of its own, the nearest success renderer draws it.

The line that stays

A summary step is the part of the pipeline people read, so it should read the outcome rather than describe one. It should not run without a success to report, and it should show a zero with its reason.

A capstone is a success narrative. Running it on a failure fabricates one.