What the agent does
It watches an API provider's releases, finds the lines in a repository that a breaking change affects, and opens one draft pull request with one commit per change. It never merges. Today it covers Stripe, the OpenAI SDK and Twilio, in JavaScript/TypeScript, Python and PHP.
Every number below comes from runs against public repositories whose maintainers had already done the same migration by hand, so the agent's fix could be compared with theirs. The pipeline was frozen before each run and not tuned during it.
The numbers
| Test | Result | What it means |
|---|---|---|
| First run, Stripe, 12 repositories | 1 correct fix | The model kept deleting or nulling fields. Nothing told it where a removed field had moved. |
| Second run, Stripe, 10 new repositories | Reported as "8 of 10". Withdrawn. | See the next section. |
| Stripe, the 6 valid repositories | 3 fixed and correct, 3 held back | The 3 held back are pinned to an older API version where the fix would break at runtime. The agent says so and leaves the code alone. These verdicts stayed the same on every later rerun, including on a different model. |
| A real Stripe release on my own test repository | First attempt failed: 18 changes attached. Later: 4 attached, 2 commits, both verified by tests | The first attempt attached changes the code did not use, and could not tell the repository's old failures from new ones. Both causes were fixed. |
| OpenAI Node SDK v3 to v4, 13 repositories | 10 reached the model: 5 pass, 2 partial, 3 fail | A later rerun of the same 10 gave 4 / 2 / 4 on first attempts; the two that dropped passed when repeated unchanged. The model's answers vary from run to run. |
| Same 10, stronger paid model | 6 pass, 1 partial, 3 fail | One more correct fix, and the same three hard cases failed. The model is not the main limit. |
| Python demo (OpenAI 0.28 to 1.0) | Draft, 3 of 4 tests pass | The app code is migrated. One test needs a person, and the pull request says so. |
| PHP demo (Stripe, one removed field) | 8 runs, 3 ended green, 1 of those was wrong | See "Green is not proof". |
Three of the 13 OpenAI repositories never produced a result because they could not be installed in the test sandbox. They are counted as no result, not as failures of the agent and not as successes.
The number I withdrew
After the second Stripe run, the agent's own report said 8 of 10 fixes matched the maintainers. Before publishing it, four of the generated diffs were compared line by line with the maintainers' real commits on GitHub. That review found:
- 4 of the 10 comparisons were invalid. In those repositories the "migration commit" only re-formatted code, or the code had already migrated. There was nothing to compare against.
- 2 fixes read the right field but would have broken the code. The repositories were pinned to an older Stripe API version in which the new field does not exist. One of them would have marked every paying user as expired.
The automatic check had marked all of these as correct, because it looked for the new field name and nothing else. The agent now refuses to change a read while the pinned version is older than the change, and the comparison set is checked for validity before any run.
One caveat on that review: it was separate from the agent that wrote the fixes, but it was not done by an outside party. The diffs and the maintainers' commits are public for anyone who wants to check.
Green is not proof
The most useful finding is a list of cases where every check passed and the code was still wrong.
- A cast that hid the error. One early fix read a field that does not exist, through an
as anycast. The typecheck passed because of the cast. Generated fixes may no longer add casts or suppressions. - Tests that did not cover the line. Two early fixes dropped a value instead of reading it from its new place. The repositories' test suites passed, because no test looked at that value.
- A string nobody read. In the PHP demo the code was fixed, but the request still asked Stripe to expand the removed field:
'expand' => ['invoice.payment_intent']. The usage search did not look inside strings, the agent's self-review passed it, and the tests built their objects by hand. CI was green and the pull request was marked ready. It was found by reading the branch. - Webhook payloads. This one came from a maintainer, not from my runs (below). The types follow the SDK version; the payload follows the version set on the provider's endpoint. When they drift apart, the compiler is satisfied and the handler is wrong.
The agent's own review step is the weakest link. In eight PHP runs it passed four things that were wrong. It now reads the provider's spec on its own instead of being shown the hint the fix was written from, which stopped the worst case, but it is one model's reading and I do not treat its pass as verification. A pull request is never marked ready while CI is red, and passing tests that the agent itself adjusted are labelled and never counted.
What a maintainer told me
I asked the maintainer of an open-source SaaS starter kit how they handle Stripe upgrades. Two things from the reply changed how I see the problem.
"The Stripe SDK types describe the API version the SDK is pinned to, but the payload a webhook actually delivers follows the API version set on the endpoint. When those two drift apart, TypeScript is happy and the code is still wrong. Nothing fails at build time; you find out when a handler reads a field that moved or changed shape. So the slow part isn't the fix, it's noticing."
The second was that their other surprise was not Stripe at all. An email provider's SDK returned an error object where their code expected an exception, so rejected emails disappeared silently. That kind of change never appears in an API spec, and this agent, which reads specs and changelogs, cannot see it.
They also said that because the version is pinned, a Stripe upgrade for them is a few planned hours, not an outage. On version-pinned APIs the pain is smaller than I assumed when I started.
What I think now
- Finding the affected lines is the reliable part. Across the Stripe and OpenAI sets the agent rarely touched a wrong file, and when it held back it gave the reason.
- Writing the fix is workable for moved and renamed fields when the provider's changelog says where the field went. It is weak on wrappers, on changes that span many files, and on anything that needs a test rewritten.
- Verifying the fix is the hard part. Most small repositories have no tests that exercise the changed line, so "CI passed" often means very little.
- The honest product is an impact report with a draft fix for a person to review, not an automatic upgrade.
What is not proven
- No outside developer has installed it on their own repository yet.
- Nothing was ever sent to a real provider API. That the fixed requests are accepted rests on the published specs and documentation.
- The comparison sets are small: 6 valid Stripe repositories and 10 OpenAI ones. These are early results, not an accuracy rate.
- The Python and PHP results are one demo repository each, built by me.
- I sent 32 emails to developers asking how they handle upgrades. Four bounced and one person replied. Whether enough people want this is still an open question.
If you maintain an API integration
I would like to hear one thing: when an API you depend on changed, how did you find out, and what took the time? One line is enough: hi@tariqbaloch.com.