Durability & crash safety

Pick up after a crash where the agent left off, without silently repeating a charge, an email or any other side effect.

Every step a run takes is written to the thread's log before the next one starts. If your process dies, nothing is lost: run the same thread again and it continues from the log.

What threads promises is narrower than "exactly once", and more useful: a side effect that may already have happened is never silently repeated. Either threads can prove what happened, or it stops and asks a person.

What survives a crash

  • Every input, model response, tool call and tool result already recorded.
  • Approvals, cancellations and budget spend.
  • Sandboxes and snapshots the run created, so they can be found again or cleaned up.

Streaming text deltas are not recorded. They are shown live and then replaced by the recorded model response.

Resume a thread

There is no separate "resume" call. Run the same thread again with the same agent. Before your new input is handled, threads settles anything the crash left in doubt.

// After a crash or restart, run the same thread again. Recovery runs first.
const again = await bot.run("Are you still there?", { store, thread: threadId });

A thread is pinned to the agent config it started with. Running it with a different model, tools or instructions raises a ConfigError ("a config change starts a new thread").

During recovery threads:

  • asks the model provider whether an interrupted request went through, when the provider can answer;
  • turns a tool call that finished but wasn't written up yet into its result, without running it again;
  • checks approvals, cancellation and permissions again before running a call that never started;
  • settles each interrupted side effect by its effect class (below), or parks it.

Effect classes

Declare how a tool's side effect behaves with effect. The class decides what happens when a crash lands between "started" and "finished".

effectUse it forAfter a crash mid-call
read_onlyLookups that change nothingRuns again. There is nothing to protect
sandbox_localChanges that stay inside the sandbox (the built-in file tools, and bash when the sandbox has no internet)Once the sandbox process is confirmed gone, the model is told the call was interrupted and may have partly run. Otherwise parks
idempotentAPIs that dedupe on a key you send, inside a known windowSafe to send again while still inside the window, otherwise parks
reconcilableAPIs you can ask "did this happen?"Your lookup decides: found, proven not found, or park
unguarded (default)Anything elseAlways parks for a person

Pass ctx.effectKey (TypeScript) or ctx.effect_key (Python) to your provider as the idempotency key. It is stable across retries of the same call.

const charge = tool({
  name: "charge_card",
  description: "Charge the customer's card.",
  input: z.object({ amount: z.number() }),
  runs: "host",
  effect: "idempotent",
  dedupWindowMs: 24 * 60 * 60 * 1000, // the provider dedups a key for 24 hours
  execute: async ({ amount }, ctx) => payments.charge(amount, ctx.effectKey ?? ""),
});

const refund = tool({
  name: "refund",
  description: "Refund a charge.",
  input: z.object({ amount: z.number() }),
  runs: "host",
  effect: "reconcilable",
  reconcile: {
    finality: "final",
    lookup: async (effectKey) => {
      const found = await payments.findCharge(effectKey);
      return found === undefined ? { status: "not_found" } : { status: "found", value: found };
    },
  },
  execute: async ({ amount }, ctx) => payments.charge(-amount, ctx.effectKey ?? ""),
});

const sendEmail = tool({
  name: "send_email",
  description: "Email the customer.",
  input: z.object({ to: z.string(), body: z.string() }),
  runs: "host",
  // no effect declared: unguarded, so a crash mid-send parks for a human
  execute: async ({ to, body }) => mailer.send(to, body),
});

idempotent needs dedupWindowMs / dedup_window_ms, and reconcilable needs reconcile. Set finality: "final" only when a "not found" from your lookup really proves the call never happened. With "nonfinal", a "not found" parks instead.

Parked effects

When threads can't prove what happened, the run returns status: "parked" with the effects it is waiting on. Nothing runs on that thread until a person decides.

Settle a parked effect with resolveParked / resolve_parked:

  • "assume_done": the effect happened. The run continues without sending it again.
  • "assume_not_done": it didn't. The call may be sent again, and the log records that a person accepted the risk of a duplicate.
const operator = { issuer: "api", tenant: "local", subject: "operator" };
const opened = await openThread(store, threadId);
if (!opened.ok) throw new Error(opened.error.message);

if (result.status === "parked") {
  for (const p of result.pending) console.log(p.kind, p.id);
}
const resolved = await opened.value.resolveParked("<effect_key>", "assume_done", operator);
if (!resolved.ok) console.log(resolved.error.code, resolved.error.message);

Then run the thread again to continue. A host exposes the same action over HTTP, see HTTP API & SSE. Approvals park the same way; see Human-in-the-loop.

One process per thread

Only one process can drive a thread at a time. A run holds the thread while it works, and a second run on the same thread fails with branch_busy instead of racing it. If the holder dies, another process can take over once its hold expires, and anything the old holder tries to send after that is refused.

Providers threads can't guard are refused

This holds for adapters too: every model, sandbox and memory call goes through a check that the run still owns the thread. When a provider's SDK sends requests threads can't intercept, the adapter is refused at setup with ConfigError("transport_fence_unsupported") rather than shipped with a weaker guarantee. Today that applies to:

  • mem0() memory, in both languages;
  • modal() sandboxes in TypeScript. Modal works in Python, see Modal.

Logs work across languages

TypeScript and Python use the same store format. A thread written by a TypeScript agent can be opened, inspected and forked from Python against the same store directory, and the other way round.

Inspect a thread

Read every step of a thread with timeline().

Edit on GitHub

On this page