A Green Suite. Zero Patches.
The suite was green. Every test passing, through the whole change and after it.
In the cloud, every automated fix the platform attempted produced zero patches. The trace of one live run showed why, in a single line, before the fixing tool had even started:
TypeError: execute() got an unexpected keyword argument 'scanner_roots'
Not an edge case. Not a flaky timeout. A hard error at the entry point of every fix, on every run, and a test suite that could not see it.
How both were true at once
The platform finds vulnerabilities in a codebase and then hands each finding to a fixing tool, which edits a copy of the repository and commits the patch. We had been chasing a zero-patch bug for about five days. The change that finally addressed one of its real causes needed a new piece of data, each finding’s root directory, to reach the tool. So we threaded a new argument through roughly ten function signatures on the way down.
There was a second, parallel entry point into the same machinery, the one production actually uses. The sweep never touched it. It didn’t accept the new argument, so the first call into it raised.
The end-to-end test for the fix path didn’t go through that entry point. It built the findings itself and handed them straight to the fixing tool, one hop below the place that broke. It was a real test, of real code, and it passed for real. It just started where the bug ended.
That is not bad luck. It is the shape of every hand-fed end-to-end test:
A test that injects a value can never catch the code that produces it.
If the test supplies the input, it proves the component that consumes the input. It cannot prove the wiring that supplies it in production, because it replaced that wiring with itself.
What the evidence said, and what the reasoning said
The record of that run is worth reading as a pair. The reasoning, from the code, was that findings must be getting lost on the way to the fixer. The ledger said otherwise: the grant handed to the fix step carried 74 fix strategies. The data arrived intact. Execution read zero because execution never started.
The session turned on two questions, both about evidence rather than argument. Did the git log actually show the patches? And, when a theory arrived: is this a probe, or reasoning? A green suite is an answer to neither.
What changed
The argument is gone. The per-finding root now rides inside the strategy object that already travels from the proposer to the fixer. No generic signature widened, so there is no parallel implementation for a widening to miss. The generic layer stays generic; the domain data travels inside the container built to carry it.
There is now a test that fails if that argument reappears. A signature-consistency test asserts that no executor on that surface takes the parameter. The regression is not “caught if someone remembers”; it cannot be reintroduced without a red test saying why.
And a rule for end-to-end tests, written down as a rule:
- No stub at the seam under test. An end-to-end test enters at the top of the flow, where production enters, not one component below where it might break.
- Assert on what crossed the seam, not on what a function returned. The path the fixing tool actually received. The workspace root actually persisted. The real git log of the repository copy. A stub can fabricate a return value. It cannot fabricate a commit.
- One test drives the whole flow, from the entry point to the artifact.
The evidence that it was over was not the suite. It was the first clean end-to-end run in the cloud: a scan that found the issues, and a fix step that landed real commits on the repository copy, one per fix (an insecure direct object reference, a security misconfiguration, an XML external entity). The patches were in the git log, which was the question all along.
The part worth keeping
Nothing about the suite was weak. It had good coverage of the components it tested, and the component that consumed findings was, in fact, correct. The suite was green about everything it was shown. It was never shown the door production walks through.
That generalises well past this platform. A green suite is evidence about the code it runs, entered the way it enters it. Where a test hands a component its input, the suite has quietly stopped testing the thing that produces that input in the real system, and every passing run reinforces a confidence it hasn’t earned.
The line that stays
Draw the end-to-end test’s entry point at the seam you ship, not at the component you can conveniently feed. Then assert on what physically crossed the wire.
A test that injects a value can never catch the code that produces it.
