Turn last week’s agent failure into a regression test
Convert a real incident into a replayable test that fails if the same bug returns, including after a model update.
by Chisom collins · published Sep 7, 2026
Agent bugs come back. The fix works, the incident closes, and six weeks later a model update reintroduces the same behaviour with nobody watching. This converts an incident into a test with recorded tool responses, so it runs deterministically and cheaply, and it separates the assertions that must always hold from the ones that only happened to be true in that run.
Tools / stack
Inputs required
The full trace of the failing run, the fix that was applied, and the tool responses from that run
Steps
- 1.Record the tool responses from the original run as fixtures, so the test does not depend on live systems.
- 2.Write the assertion that describes the bug, not the whole run. “Did not call delete twice” is a test, “produced identical output” is a snapshot that will break for unrelated reasons.
- 3.Separate assertions that must always hold from details that were incidental to that particular run.
- 4.Run the test against the pre fix code and confirm it fails. A regression test that never failed is not proven to detect anything.
- 5.Run against the fixed code and confirm it passes.
Expected output
A deterministic regression test with recorded fixtures, confirmed failing before the fix and passing after, tagged with the model version
Golden Example
Trace of an agent that double submitted a payment, the fix, and eleven recorded tool responses
Test asserting the submit tool is called at most once per run and that a repeat submission after a timeout is not attempted. Confirmed failing on the pre fix commit. Passing after. Tagged claude-sonnet-4.5. Runs in 3 seconds with no live calls.