API Update Agent · write-up · November 2026

The slow part isn't the fix, it's noticing.

Five weeks of testing an agent that drafts API-upgrade pull requests, on real open-source repositories. What worked, what did not, and one number I had to withdraw.

Back to the project page Earlier results page

What the agent does

It watches an API provider's releases, finds the lines in a repository that a breaking change affects, and opens one draft pull request with one commit per change. It never merges. Today it covers Stripe, the OpenAI SDK and Twilio, in JavaScript/TypeScript, Python and PHP.

Every number below comes from runs against public repositories whose maintainers had already done the same migration by hand, so the agent's fix could be compared with theirs. The pipeline was frozen before each run and not tuned during it.

The numbers

TestResultWhat it means
First run, Stripe, 12 repositories 1 correct fix The model kept deleting or nulling fields. Nothing told it where a removed field had moved.
Second run, Stripe, 10 new repositories Reported as "8 of 10". Withdrawn. See the next section.
Stripe, the 6 valid repositories 3 fixed and correct, 3 held back The 3 held back are pinned to an older API version where the fix would break at runtime. The agent says so and leaves the code alone. These verdicts stayed the same on every later rerun, including on a different model.
A real Stripe release on my own test repository First attempt failed: 18 changes attached. Later: 4 attached, 2 commits, both verified by tests The first attempt attached changes the code did not use, and could not tell the repository's old failures from new ones. Both causes were fixed.
OpenAI Node SDK v3 to v4, 13 repositories 10 reached the model: 5 pass, 2 partial, 3 fail A later rerun of the same 10 gave 4 / 2 / 4 on first attempts; the two that dropped passed when repeated unchanged. The model's answers vary from run to run.
Same 10, stronger paid model 6 pass, 1 partial, 3 fail One more correct fix, and the same three hard cases failed. The model is not the main limit.
Python demo (OpenAI 0.28 to 1.0) Draft, 3 of 4 tests pass The app code is migrated. One test needs a person, and the pull request says so.
PHP demo (Stripe, one removed field) 8 runs, 3 ended green, 1 of those was wrong See "Green is not proof".

Three of the 13 OpenAI repositories never produced a result because they could not be installed in the test sandbox. They are counted as no result, not as failures of the agent and not as successes.

The number I withdrew

After the second Stripe run, the agent's own report said 8 of 10 fixes matched the maintainers. Before publishing it, four of the generated diffs were compared line by line with the maintainers' real commits on GitHub. That review found:

The automatic check had marked all of these as correct, because it looked for the new field name and nothing else. The agent now refuses to change a read while the pinned version is older than the change, and the comparison set is checked for validity before any run.

One caveat on that review: it was separate from the agent that wrote the fixes, but it was not done by an outside party. The diffs and the maintainers' commits are public for anyone who wants to check.

Green is not proof

The most useful finding is a list of cases where every check passed and the code was still wrong.

  1. A cast that hid the error. One early fix read a field that does not exist, through an as any cast. The typecheck passed because of the cast. Generated fixes may no longer add casts or suppressions.
  2. Tests that did not cover the line. Two early fixes dropped a value instead of reading it from its new place. The repositories' test suites passed, because no test looked at that value.
  3. A string nobody read. In the PHP demo the code was fixed, but the request still asked Stripe to expand the removed field: 'expand' => ['invoice.payment_intent']. The usage search did not look inside strings, the agent's self-review passed it, and the tests built their objects by hand. CI was green and the pull request was marked ready. It was found by reading the branch.
  4. Webhook payloads. This one came from a maintainer, not from my runs (below). The types follow the SDK version; the payload follows the version set on the provider's endpoint. When they drift apart, the compiler is satisfied and the handler is wrong.

The agent's own review step is the weakest link. In eight PHP runs it passed four things that were wrong. It now reads the provider's spec on its own instead of being shown the hint the fix was written from, which stopped the worst case, but it is one model's reading and I do not treat its pass as verification. A pull request is never marked ready while CI is red, and passing tests that the agent itself adjusted are labelled and never counted.

What a maintainer told me

I asked the maintainer of an open-source SaaS starter kit how they handle Stripe upgrades. Two things from the reply changed how I see the problem.

"The Stripe SDK types describe the API version the SDK is pinned to, but the payload a webhook actually delivers follows the API version set on the endpoint. When those two drift apart, TypeScript is happy and the code is still wrong. Nothing fails at build time; you find out when a handler reads a field that moved or changed shape. So the slow part isn't the fix, it's noticing."

The second was that their other surprise was not Stripe at all. An email provider's SDK returned an error object where their code expected an exception, so rejected emails disappeared silently. That kind of change never appears in an API spec, and this agent, which reads specs and changelogs, cannot see it.

They also said that because the version is pinned, a Stripe upgrade for them is a few planned hours, not an outage. On version-pinned APIs the pain is smaller than I assumed when I started.

What I think now

What is not proven

If you maintain an API integration

I would like to hear one thing: when an API you depend on changed, how did you find out, and what took the time? One line is enough: hi@tariqbaloch.com.