How to build agents that survive crashes with Pi Durable

An agent can spend twenty minutes investigating a problem, then disappear when its Node process crashes. A saved chat helps you explain what happened. Recovering the unfinished work needs more information: which tool was running, what already finished, and which requests are still waiting.

Pi Durable stores that information as the agent works. Reopen its storage after a process crash, and it can continue the unfinished run.

It’s a TypeScript library from Earendil, the team behind Pi. You use it to build your own agent application, including the tools and interface you want. The regular Pi terminal app remains available for interactive coding.

The Syntax video “My Code Fixes Itself Now” shows a useful application. A Sentry error starts an agent, which investigates the bug and prepares a pull request. The demonstration also stops and resumes the work. Sentry, GitHub, and the screens around that agent are application code built around Pi Durable.

Let’s build something smaller so we can see how recovery works. We’ll give an agent a read-only service-status tool, store its work in SQLite, kill the process while the tool runs, then restart it with the same request.

There are eight interactive drawings throughout this tutorial. Click a drawing, or focus it and press Enter or Space, to move through its states.

Pi Durable is experimental. These examples follow the package API checked in October 2026. Keep your package lockfile, because the API can change between releases.

What Pi Durable stores

An agent application has a model, tools the model can call, and code that runs their conversation. That surrounding code is often called a harness.

Pi Durable provides the harness. It commits conversation entries and execution state to storage as work progresses. A model request and its tool calls run as durable tasks.

You provide the model credentials, tool implementations, and somewhere to store the data.

The model still runs through its provider. SQLite stores the agent’s state locally. Putting a harness on your laptop doesn’t move an OpenAI model onto your laptop.

This also explains why “durable” doesn’t mean the process stays alive forever. When the process is down, work stops. A supervisor, container platform, or you must start another process. Pi Durable gives that process the saved work to resume.

Create the project

Use a current Node.js LTS release with the built-in node:sqlite module. You’ll also need an OpenAI API key with access to the model used below. Model calls are billed through that API account.

Create an empty directory:

mkdir pi-durable-demo
cd pi-durable-demo
npm init -y
npm pkg set type=module
npm install --ignore-scripts @earendil-works/pi-durable @earendil-works/pi-ai @earendil-works/chord
npm install --save-dev --ignore-scripts tsx typescript @types/node

pi-ai handles model access. chord provides the state and cancellation primitives Pi Durable uses. tsx runs our TypeScript files.

Our project will have two source files:

pi-durable-demo/
  agent.ts
  tools.ts

Running it will also create agent.sqlite and possibly SQLite journal files. Keep those out of Git, along with credentials:

node_modules/
.env
agent.sqlite*

Save that as .gitignore.

Give the agent a tool

We need a tool whose behavior is easy to inspect. Our service_status tool looks up a service in a small fixture. The website is healthy; checkout is degraded after a deployment.

It deliberately waits ten seconds before returning. That gives us time to kill the process during the check.

Save this complete file as tools.ts:

import { setTimeout as delay } from 'node:timers/promises'
import { Type } from '@earendil-works/pi-ai'
import { defineExtension, defineTool } from '@earendil-works/pi-durable'

const services = {
  website: { status: 'healthy', lastDeploy: 'deploy-104' },
  checkout: { status: 'degraded', lastDeploy: 'deploy-105' },
}

const serviceStatus = defineTool({
  name: 'service_status',
  description: 'Read the demo status for website or checkout',
  parameters: Type.Object({
    service: Type.Union([
      Type.Literal('website'),
      Type.Literal('checkout'),
    ]),
  }),
  replay: 'safe',
  execute: async ({ service }, api, context) => {
    api.output(`Checking ${service}...\n`)
    console.log(`Tool started: ${service}`)
    await delay(10000, undefined, { signal: context.abortSignal })
    return {
      output: [{ type: 'text', text: JSON.stringify(services[service]) }],
    }
  },
})

export const StatusTools = defineExtension({
  name: 'status-tools',
  tools: [serviceStatus],
})

This is demo data. We aren’t checking a real website or contacting a monitoring service.

parameters describes the allowed arguments and validates them before execution. The model can request website or checkout.

api.output() publishes progress while the tool runs. The returned output contains the result the model receives. context.abortSignal lets the delay stop when the harness cancels this call.

The important setting is replay: 'safe'. It tells Pi Durable that this tool can run again after an interrupted execution. Reading this fixture has no external side effect, so repeating it is safe.

The extension gives the tool a stable name in the registry. Reinstall the same extension whenever you open this agent’s storage.

Open SQLite and submit a request

Now we can connect the model, tool registry, and database.

Save this complete file as agent.ts:

import { BACKGROUND_CONTEXT } from '@earendil-works/chord/context'
import { createModels } from '@earendil-works/pi-ai/models'
import { openaiProvider } from '@earendil-works/pi-ai/providers/openai'
import {
  AssistantEntry,
  createRegistry,
  Harness,
} from '@earendil-works/pi-durable'
import { openNodeSqliteStorage } from '@earendil-works/pi-durable/storage/sqlite/node'
import { StatusTools } from './tools.js'

const requestId = process.argv[2]
const prompt = process.argv.slice(3).join(' ')
if (!requestId || !prompt) {
  throw new Error('Pass a request ID followed by a prompt')
}
if (!process.env.OPENAI_API_KEY) {
  throw new Error('Set OPENAI_API_KEY before running the agent')
}

const context = BACKGROUND_CONTEXT
const models = createModels()
models.setProvider(openaiProvider())
const registry = createRegistry()
registry.install(StatusTools)

const harness = await Harness.open(
  await openNodeSqliteStorage('./agent.sqlite'),
  { models, registry },
  context,
)

try {
  const root = await harness.root(context, {
    agent: {
      model: { provider: 'openai', modelId: 'gpt-6-sol' },
      instructions: 'Use service_status for service questions. Explain the demo results briefly. Never claim to repair or deploy anything.',
    },
  })
  harness.resume()
  const submission = await root.submit(
    { type: 'input', content: prompt, requestId },
    context,
  )
  console.log(`Submission: ${submission.id}`)
  const settled = await submission.wait(context)
  if (settled.status !== 'done' || settled.type !== 'input') {
    throw new Error(`The input was not answered: ${JSON.stringify(settled)}`)
  }
  const answer = await root.commit(
    (tx) => tx.entry(AssistantEntry, settled.answer),
    context,
  )
  for (const message of answer?.model ?? []) {
    if (message.role !== 'assistant') continue
    for (const part of message.content) {
      if (part.type === 'text') console.log(part.text)
    }
  }
} finally {
  await harness.close(context)
}

Notice the ./tools.js import. That’s the output path TypeScript resolves to tools.ts when using Node’s module conventions; tsx handles it while running the source.

harness.root() creates the root conversation on first use. Opening the same database returns that conversation again. Its initial agent settings apply when it is created. To change an existing conversation later, use root.configure().

harness.resume() starts recovery of pending tasks. Submitting or waiting also starts the scheduler; the explicit call makes the restart behavior visible here.

A submission is the request we admitted to the conversation. Its ID lets us identify that same request after a restart. wait() waits for an answer, and the final transaction reads the assistant entry containing it.

The finally block closes the harness on an ordinary exit. A hard process kill skips that block, which is exactly what we want to test later.

Run the agent

Put your API key in the shell environment. For example, in zsh you can enter it without echoing it or putting the key in your command history:

read -rs 'OPENAI_API_KEY?OpenAI API key: '
export OPENAI_API_KEY

Run a status check:

npx tsx agent.ts incident-105 'Use service_status to check checkout. Explain the result.'

incident-105 is our application’s request ID. It is separate from the submission ID printed by Pi Durable.

The tool prints Tool started: checkout, waits, then returns the fixture. The wording of the model’s answer varies, but it should explain that checkout is degraded after deploy-105.

The agent has only the tool we installed. It has no shell, file-editing tool, deployment credentials, or ability to repair checkout.

You can type-check both files with:

npx tsc --noEmit --strict --skipLibCheck --module nodenext --target esnext agent.ts tools.ts

Kill it halfway through

Now let’s interrupt a new request while its tool is running.

In your first terminal, run:

npx tsx agent.ts incident-106 'Use service_status to check checkout. Explain the result.'

Wait for Tool started: checkout.

In a second terminal, find the Node process running this example:

pgrep -fl 'agent.ts'

Identify the Node process executing agent.ts, then send kill -9 followed by that process’s numeric PID. If tsx has a launcher and a child, target the child executing the TypeScript file. Check the command before killing it.

Wait until that process has exited. From the original project directory, restart with the exact same command:

npx tsx agent.ts incident-106 'Use service_status to check checkout. Explain the result.'

The submission ID should match the one printed before the crash. Our replay-safe tool starts again, waits ten seconds, and returns its result.

Pi Durable resumes from durable checkpoints. It doesn’t restore a suspended JavaScript stack at the exact instruction that was interrupted. Our tool function starts over because it was interrupted before its result was committed.

An interrupted model request can also be sent again. A stored partial answer is marked as aborted; it isn’t a completed answer. Another model request can mean another API charge.

The recovery path here was tested with SQLite and a scripted provider from Pi’s testing utilities. The test killed a subprocess inside the real tool, reopened its database, and checked the submission ID. That verifies persistence and replay behavior without depending on a live model’s wording. It doesn’t measure model quality or exercise a real monitoring integration.

Retry a request without creating a second job

Run the completed incident-106 command again.

It should print the same submission ID and saved answer. It shouldn’t start the tool again.

The requestId deduplicates admission of a submission. Retrying delivery of the same logical job should reuse its ID. A genuinely new status check needs a new ID, even if its prompt is identical.

Don’t reuse an old ID for a different prompt. You’re identifying the original job, rather than asking for an updated answer.

For a webhook application, derive this ID from a stable event identifier. If your policy is one investigation per incident, keep an incident-to-investigation mapping too. Monitoring systems can send several different events about the same incident.

Decide which tools may run again

Imagine replacing our fixture with an email sender.

The email provider accepts the message. Then the process dies before Pi Durable commits the tool result. When it restarts, the local database can’t tell whether the email went out.

Automatically running that tool again could send a duplicate.

A tool’s default replay policy is unsafe. Leave it that way for an action you cannot safely repeat. After an interruption, the harness reports an interrupted result to the model instead of automatically rerunning that call.

That still leaves an application decision. The model could request a fresh tool call. Your application must prevent that from repeating an action whose outcome is uncertain.

For actions such as sending email or creating a pull request, use an operation ID and the external service’s idempotency mechanism when available. Otherwise, reconcile against the external system before allowing another attempt. If you cannot establish the outcome, ask a person to resolve it.

Only declare an effectful tool replay-safe once its implementation makes repeated execution safe.

Submission deduplication and tool idempotency solve different parts of this problem. A stable requestId alone doesn’t make an external action happen exactly once.

Separate the transcript from the work

The transcript records what happened in the conversation. Tasks record the execution needed to get to an answer.

A model can request a tool. That tool runs as a task owned by the generation task. Once the tool finishes, another model turn can interpret its result and answer the user.

Saving only chat messages is insufficient for recovery. The application also needs to know about work that has started but hasn’t produced a final message yet.

Pi Durable can store your own typed application documents beside the transcript. A commit can update those documents, append entries, and create tasks atomically. That is useful for an incident’s investigation status, for example.

That transaction covers Pi Durable’s storage. It doesn’t include a remote GitHub or email API call in the same atomic operation.

Connect more than one screen

The terminal script is one way to submit work. A web interface or Slack integration can submit to the same conversation through your application server.

Keep one process responsible for opening and owning a particular store. Other clients connect to that process. Pi Durable doesn’t provide cross-process locking for several independent harnesses opening the same database.

For a UI, root.viewState(context) exposes a subscribable conversation view. It includes committed entries, running tool output, and queued inputs. A reconnecting client can show the current state without reconstructing it from a private stream it might have missed.

Your server still provides transport, authentication, and authorization. Installing the package doesn’t create a hosted multiplayer app.

A message submitted while the conversation is busy becomes a follow-up by default. A steering input can join the current work at a tool-round boundary:

await root.submit({
  type: 'input',
  content: 'Check website as well before answering.',
  whenBusy: 'steer',
}, context)

This snippet belongs inside the open harness’s try block, where root and context are available. To send it during the current run, the application must accept another input while the first submission is still waiting.

Steering doesn’t instantly interrupt an executing tool. Also, cancelling a wait only stops that caller from waiting. Explicitly aborting the conversation is a separate action.

Keep the storage when you deploy

SQLite is convenient for a single Node process. It only helps across a deployment if the database remains available afterwards.

A container’s temporary filesystem can disappear when the container is replaced. Mount persistent storage, keep the same database path, and stop the old owner before starting its replacement.

Treat conversations and tool output as application data. They can contain private prompts and service information, so restrict access and back them up.

The package also ships JSONL and in-memory storage. In-memory storage is useful for tests, but it disappears with the process. Other runtimes can use adapters. A Cloudflare Durable Object is one possible host; Pi Durable doesn’t require Cloudflare.

Process-crash recovery also has limits. A deleted database cannot be resumed. SQLite’s shipped Node adapter uses WAL with synchronous = NORMAL, so a machine power failure can lose the newest commits even though a process crash is recoverable. Choose your storage and backup policy around the failures your application needs to survive.

Long conversations still have context limits

Persistent storage can keep a conversation’s history longer than a model can accept in a request.

Pi Durable supports compaction to summarize older messages and make room for new work. The summary becomes part of what the model sees next.

A summary can omit details. Store facts the application must preserve in structured documents or another appropriate database. For an incident investigation, that might include its incident ID, repository, and associated pull request.

Likewise, subagents need an implementation. Pi Durable provides conversations, ownership, tasks, and forks that you can build them from. Its foreground and background subagent examples show the patterns.

From this demo to an agent that investigates errors

The Syntax demonstration starts with an incoming monitoring event. Our command-line script starts with a request ID and a prompt. The next step is to have a webhook handler produce those values.

The application around it would:

1. Verify the webhook and check that the event belongs to a project it handles.
2. Find the incident’s existing investigation, or create one.
3. Submit the investigation with its stable request ID.
4. Give the agent narrow tools for reading the error and working on an isolated repository copy.
5. Run checks and prepare a pull request for review.
6. Record the PR and reconcile later events with that investigation.

Pi Durable supplies the persisted agent runtime inside that flow. You supply the event handling and the rules for changing code or contacting GitHub.

I would start with a tool that reads one real service’s status. Keep the crash test, add a test for an interrupted unsafe action, and check that a duplicate event creates no second investigation. Add the code-changing tools once those paths behave the way you expect.

The upstream README covers the APIs used here. Its runnable examples go further into recovery, approvals, UI events, and subagents. Earendil’s introduction to Pi Durable explains the design behind them.

Want me to talk about your product? You can sponsor this site.