Fan out a few dozen Claude Code agents across a repo. One hits a rate limit, or the proxy restarts, or the connection drops for a second. That agent returns nothing, the script filters it out, and the run reports completed.
We pulled the runtime apart to find out why, then counted what it has already cost on one machine.
Status Everything measured here is measured. The fix is a plan, not a release.
The workflow runtime has three retry mechanisms, and they’re decent. A stalled agent gets five restarts. A throttled-looking response gets one retry after a 45 second sleep. A schema validation failure gets five nudges.
Not the API error. This is the entire path:
Dossier
Margin
// after the stall-retry loop, in the local agent runner
if (It.apiError) {
let yr = `[${te}] failed: ${It.apiError}`;
return A.push(yr),
r({type:"progress", …}),
null
}A rate limit comes back fast, so it doesn’t even qualify for the throttle retry; that one needs the request to have run for over 90 seconds first. It goes straight to null. Then parallel() and pipeline() turn failures into null as well, and the .filter(Boolean) at the end of a generated script drops them on the floor.
| Failure | Handling | Retries |
|---|---|---|
| No progress for 180 seconds | Watchdog aborts, agent restarts from its original prompt | 5 |
| Throttled-looking response | Sleep 45 s, try once more | 1 |
| Schema validation failure | Nudge in conversation | 5 |
| Rate limit, usage limit, dropped connection, 5xx | Return null | 0 |
The tool result does carry a failures block and an error count. The status sitting next to it says completed.
This is the part worth knowing even if you never change a thing. The resume journal is written to a path built from the session id, and the session id isn’t stable.
function Ste(e) { // e = the runId
let t = dW() ?? tS(gn()); // project dir
return join(t, kt(), "subagents", "workflows", e); // kt() = CURRENT session id
}Auto-compaction mints a new session id. So the resume looks under a directory that has never contained that run, finds nothing, and cold-starts from phase one, while a complete journal sits intact one folder over. Long runs are the ones most likely to compact, which means the failure correlates neatly with the runs you most want back.
It’s filed and open, labelled bug and has repro. The reporter had a 61 KB journal with twelve entries sitting recoverable on disk while the resume started again from nothing.
A clean run shows exact parity: every agent that started returned a result. An interrupted one doesn’t. Each dot below is one agent attempt on a single machine.
The five largest workflow journals on one machine. Three of them show a large gap between agents that started and agents that returned a result: 96 started and 61 returned, 128 started and 78 returned, 107 started and 55 returned. The remaining two show exact parity at 164 and 269.
One honest caveat: a started entry is appended per attempt rather than per agent, so a gap isn’t a one-to-one count of lost work. The contrast between the parity rows and the 107/55 row is real all the same.
Cached results are keyed by a sha256 chain over the previous key, the prompt bytes and the options. Not a positional index, which had been ambiguous until the binary settled it.
function zSd(e, t, r) { // (prompt, opts, prevKey)
let n = createHash("sha256")
.update(r).update("\x00") // previous key
.update(e).update("\x00") // prompt bytes
.update(Jq_(t)).digest("hex"); // normalised opts
return `v2:${n}`;
}And the miss is harder than a prefix boundary. A sticky flag is set the first time a lookup misses, and after that no lookup is consulted again for the rest of the run, including ones whose key would still have matched. An orphaned journal isn’t a degraded resume. It’s a fully cold one.
One useful thing falls out of the same function: only schema, model, effort, isolation and agentType are normalised into the key. Labels and phases aren’t, so you can relabel or regroup agents freely without invalidating anything.
Picking one is a category error. A proxy can’t know whether an agent already committed something, and a journal can’t keep a wave alive through a five hour usage window.
A 429 with somewhere to rotate becomes a silent account swap. A 529, a 504 or a dropped socket comes back as x-should-retry with an honest delay, so the client’s own loop absorbs it. Real 400s pass straight through.
A project-scoped map of run id to journal path, so a resume finds its history whichever session wrote it. Before spending anything it reports the predicted cache-hit rate, and 0% fails loudly instead of quietly starting over.
A task isn’t done because a process exited zero. Record the worktree, the base commit, the output commit and the validation receipt, then reconcile what’s uncertain by reading git.
Three of the five research backends we put this to recommended adopting a durable execution engine instead. We’re not, and the reason is narrow: what we measured is a lookup problem, not a durability problem. The journal is intact on disk every single time. Temporal wants every model call and clock read inside an Activity and terminates a history at 51,200 events; DBOS wants an authoritative Postgres. Both are a second distributed system to fix a path that resolves wrong.
Worth noting which backends said what. The two that could read the actual repository recommended the light path; the three that could only search the web recommended the substrates, in a market where those four vendors publish heavily on exactly this question.
Phase 0 is a file copy. If it works, the 516 journals already on disk become recoverable and the rest of the plan is de-risked before anyone writes code. If it doesn’t, that’s worth knowing cheaply.
wf_<runId> directory into the current session’s tree and resume. Validates the whole premise.uncertain, leases, attempts, worktree and base commit, reconciliation by git.running. Getting from a dead run back to a live one is a missing state machine, not a keybinding.