“Back off and retry” is one of the common rules we learned building distributed systems. When your API call fails for one reason or another, the simplest thing to do is often to just try it again. “It’s nondeterministic, so maybe it will work the second (or third, or fourth) time,” you might think. Retries work well enough for HTTP requests or calls returning traditional error codes, but when it comes to agentic task failures it’s rarely enough.
Trajectories from software engineering
As an example, we can look to the trajectories from the Nebius SWE-agent data set. In this data set, the creators ran a software engineering agent against real GitHub issues with matching PRs, and graded each run’s result on whether the agent’s changes passed the tests from the actual fix. They produced many runs for each issue (dozens in some cases), so we can see what happens when an agent keeps trying the same task.
For example, in pydantic issue 2496 it was reported that a model field with the empty tuple type always generated a broken JSON schema. The agent tried to resolve this issue 16 times. On one attempt it gave up, but the other fifteen times it submitted a change. All fifteen were unique, and all failed their tests. Each of them introduced some sort of special-case for the empty tuple, and the real fix was more general.
Contrast this with issue 2290 from the same repo. Here, the reporter noted that the copy method didn’t properly flag changed fields. This was also attempted sixteen times, and passed seven of them. Each run changed the same line of code. Seven got the change right, and the others failed. This one really was a coin flip.
Retrying only helps if the task failed due to bad luck: a transient failure, a system error, or some mangled output. We just saw examples of both, so the question is: what’s typical?
In the Nebius data set, the coin flips were the minority. In the plot below, we can see the distribution of pass rates by issue.1 If the failures were due to bad luck, we’d expect to see a distribution like the dashed line.2 But the blue distribution reveals a different pattern: most of the issues that failed never succeeded.
In the data I analyzed, once a task failed twice the next run’s pass rate was about 2.5%. If you know this number for your domain, it’s the number to build a retry policy on: what are the chances the next run will succeed?
What about better models?
This research was conducted with 2024-era models, a generation or two behind today’s. Better models can solve more tasks, meaning their pass rates increase. But when they fail, retrying doesn’t help much. Two recent papers3 reported pass rates using current models, both for single runs and for the best of several (“pass@k”). In every case, we saw the same pattern with the extra runs. Tasks that failed once mostly failed again.
Try something else?
Retrying is cheap, and sometimes it works! It’s a reasonable first-step for those “bad luck” situations. The second failure isn’t bad luck, though. There is something about the task as presented that is not going to pass. Running again with the same prompt, context, and tools is the “cut-and-paste” retry. This usually fails.
The fix is to change something. Try giving the agent different context, different examples, or a different sort of prompt. You might have more luck splitting the task into pieces. This is where you consider handing it to a human with the failing trajectories attached. This is how you learn about your agent’s failure modes.
If you’re dealing with this problem, you don’t need fancy analysis right away. Start with a table containing task ids and pass/fail for each run. Sort your tasks into those that always failed, vs. those that sometimes failed. If the first bucket is large, you know where to start to break out of the retry loop.
Only issues with at least five runs are shown. Issues that were run more times also passed more. So the “always failed” share is actually higher than shown here.
The pass rate across all runs was 16.8%. If task failures were generally independent, we’d model this as a Bernoulli trial with p=0.168 and expect per-issue pass to cluster around that. The dashed line shows that expectation.