Method
Each repository in the test set had migrated across a breaking Stripe change in its own history. The commit that did the migration is the ground truth; the agent runs on the commit just before it, with the pipeline frozen before the run and one target change per repository. The repository is installed and tested in an isolated sandbox with no network and no secrets. A fix counts as correct when it reads the replacement the maintainers read, never deletes or nulls a read, and is valid at the API version the repository is pinned to; the labels "verified by tests", "typecheck only" and "unverified" come from what the repository's own checks actually ran. The 10 repositories were chosen before any run.
Outcome
Of 10 held-out repositories, 4 were rejected as invalid comparisons: in three the code already used the replacement before the chosen commit, and in one the chosen commit was not a migration at all. Of the 6 valid ones:
- 3 fixes made, all correct and safe at the repository's pinned API version. In two of them one usage inside a webhook handler was left alone and listed in the pull request.
- 3 held back by the version gate, no diff generated and no model called: two because the Stripe client is pinned to 2024-12-18.acacia, where the replacement field does not exist yet ("this read cannot change until the version moves to 2025-03-31.basil"), and one because every usage sits in a webhook handler, whose payload version is set on the Stripe endpoint rather than in code.
Across the 6 valid repositories, 8 usage sites were allowed and 13 were blocked by the version gate.
Fix made
Held back by the version gate
- Oak-and-Sprout/sprout-track: client pinned to 2024-12-18.acacia
- Snouzy/workout-cool: client pinned to 2024-12-18.acacia
- subhamBharadwaz/scribbly: usages in a webhook handler
Rejected as invalid comparisons
- Ctrost17/Sliptail: the chosen commit only re-formatted the lines
- usesend/useSend: the replacement was already in use before the chosen commit
- LuahZhengSheng/workshop_management_system: the replacement was already in use before the chosen commit
- christiansallen/sbg-nbc: the replacement was already in use before the chosen commit
A figure that was withdrawn
The first run of this test set was reported as "8 of 10 repositories correct". An independent review of four of the generated diffs found that one of the ground-truth commits only re-indented code, so there was nothing valid to compare against, and that one fix only worked together with an API-version change the maintainers had made in the same commit. Every ground-truth commit was then re-checked with whitespace ignored, which rejected four entries, and the API version each repository is pinned to was recorded, which showed that two of the remaining fixes were not valid at that version. The version gate was built in response, the six valid repositories were rerun, and the numbers above are from that rerun. The earlier figure is withdrawn.
Run on a real Stripe release
One run on the real spec pair 2025-02-24.acacia to 2025-03-31.basil, against a small billing module on Stripe SDK 18.5.0 whose code still read the shapes Basil removed and whose own checks were already failing for that reason. The diff reported 134 changes. The pipeline found no usages for 116, judged 10 not needed, left 5 to a human because their only usages were in the test file, and made commits for 3, including the two real migrations. No commit passed CI, and the pull request was closed with nothing on it: pull request #11.
Two causes. First, a change that did not apply to this code (a removed parameter on an endpoint the code calls) was pulled in by the endpoint match, and the fix generated for it rewrote a field that still exists, so the push with all commits failed the typecheck. Second, verifying one commit at a time assumes the base branch is green; here it was red by design, so every single commit was blamed for the errors it did not cause, and the regenerations ran into the model provider's rate limit until each commit was dropped.
Limits of this test
Six repositories is a small set. None of them has a green test suite at the commit the agent ran on, so none of the fixes is "verified by tests"; one is "typecheck only" and the rest are "unverified" by the agent's own labels. Two migrations of one provider were tested, on Node and TypeScript code only. The full tables, per-repository notes and the method description live in the application repository.
The demo pull request linked from the overview (#10 on the test repository) is not part of this evaluation: its changes come from a pair of synthetic spec fixtures with one change of each type, built to exercise the pipeline, and several of them never happened in Stripe's API.