Operations

Running this for real: topologies, the store contract, the background sweep, and what it reports about itself.

Ownership and fencing

Plane instances are stateless. Each run has at most one owner, held as a lease with an epoch.

Every append carries the writer’s epoch, and the store compares it inside the same transaction that writes. There is no window between “am I still the owner?” and the write for a paused instance to slip through, because there is no gap to slip into.

Two failure modes, deliberately distinct because they need opposite responses:

Failover is not a special code path — it is the crash-recovery path. Lease expires, another instance claims at epoch + 1, resumes via replay. That is the payoff of building on replay: HA costs one lease table and an epoch column.

The claim is initiated by the sweep. A run crashed mid-step with no message, timer or operator pending has no other driver and appears in no backlog. An expired lease that still names an owner means “an instance died holding this run”, because every clean exit — sealed, failed, suspended — releases. See the sweeper.

A live run renews; only a dead owner’s lease expires

Expiry answers is this owner dead? Without renewal it also answers a question nobody asked — a healthy run that outlives its TTL looks exactly like a crashed one, and agent runs routinely outlive a lease because one model call can. The run would be taken over and the original fenced mid-flight, having already done real work.

So the runtime renews while a run executes, at a third of the TTL, and stops the moment execution returns. A renewal the store fails to answer is logged and retried at the next period; renewal stops only when the lease is gone. Set the TTL with RuntimeBuilder::lease_ttl: it bounds how long a crashed owner strands its runs, not how long a run may take.

Anything under two seconds is refused at build. Both stores keep expiry in whole seconds and lapse on expires_at <= now, so a one-second lease is expired for part of every second it exists and no renewal frequency saves it — a plane configured that way would lose runs under load and nowhere else.

The owner string identifies a process, not an agent

Fencing rests on the epoch, not the owner: acquire refuses every live lease, the caller’s own included, and renew names the exact epoch it holds. What the owner answers is which process holds this run — the question an operator asks of a stuck one, and what LeaseHeld reports. Two processes sharing a string still cannot both write; an operator just cannot tell which one holds the run.

Runtime::owner_id() therefore defaults to a value that is unique per process and per runtime instance. Override it with a real instance identity — a pod name — never with the agent’s name:

Runtime::builder(store).owner(std::env::var("POD_NAME")?).build()

Several instances of one agent are the normal way to run one, so the agent’s name is exactly the wrong choice. The agent is named by its manifest; the process is named here, and the two are different questions.

The owner lives in the lease table and never in the chain, so changing it has no bearing on replay.

Releasing frees the lease without forgetting the epoch

Every clean exit — sealed, failed, suspended — hands its lease back, so the next instance need not wait out the TTL. What a release must not do is delete the row: the epoch lives there, and without it append has nothing to fence against while the next acquire starts again at 1 — so a writer already fenced at 2 outranks the new owner and the mechanism inverts. Releasing marks the row expired instead, and the next takeover advances the epoch as any takeover does.

A crash releases nothing, so the embedded store lapses every live lease when it opens: its file admits one opener, so such a lease belongs to a process that is gone. A restarted process resumes its runs at once. The shared store cannot know that, and waits out the TTL.

Releasing is something a run does, never something a shutdown does. A release says takeover is safe now; for a run still inside a tool call it is not, because the next owner would perform the call a second time while the first is in flight — the epoch stops the second append, not the second send. An instance on its way out finishes what it can and leaves the rest to expire, which the recovery sweep reads as an instance died holding this run.

Stopping an instance

A process killed mid-step leaves an announced effect with no terminal record, and nothing in the journal can say whether that call reached the world. The effect’s declared recovery then decides — perform it again, or wait for a person — which is the right answer to an accident and an expensive one to schedule on every deploy. Draining is the difference between a stop somebody chose and a crash. Moving a plane to a new build starts with one → upgrading.

agentplane serve drains on SIGTERM and SIGINT. The signal does two things at once and then waits for four:

Closing admission with the listeners rather than after them is the part worth stating. A request that arrives on an already-open connection after the signal is answered DRAINING and retries elsewhere, which is what that refusal is for — and it is what lets an open subscription end. A stream is a long poll over the journal, so a subscription to a run that sleeps for five working days would otherwise hold the graceful shutdown open for five working days.

--drain-secs (default 25) has to fit inside the supervisor’s own grace period, which is what sends SIGKILL afterwards — 30 seconds on Kubernetes and Docker unless raised. Runs still executing when it ends are named on stderr and at warn, and left to the recovery sweep on the path a crash takes.

In the published image the binary is the entrypoint, so it runs as PID 1. That is the right shape — the signal reaches the process that has to act on it, with no shell in between — and it is also why the handler is not optional: PID 1 has no default signal dispositions, so a SIGTERM with no handler registered is discarded and docker stop falls through to SIGKILL.

Two things a drain deliberately does not do. It does not gate resumes: a resume continues work this plane already owns, and refusing one would turn a shutting-down instance into a source of failed wakes for work the next instance is about to take. And it does not gate an agent commissioning another one — that is a step of a run the drain is itself waiting for.

An embedder driving its own server runs Runtime::drain(grace) beside their own graceful shutdown rather than after it — tokio::join! of the two — and reads DrainReport::unfinished. Serialising them either way deadlocks one on the other: a run awaited inside a request handler is finished by the server’s shutdown, and a subscription is ended by the drain.

What a caller is told while an instance drains

Admission answers RuntimeError::Draining — before the lease, the quota slot and the first append, so nothing is half-written. On the A2A surface that is -32031 with the ErrorInfo reason DRAINING, which is its own code beside QUOTA_EXHAUSTED and HALTED because it asks something neither of them does: a ceiling clears when a run finishes on this plane, a halt clears when a person lifts it, and this one clears the moment the caller reaches a different process. There is no window to wait out, so a compliant peer retries immediately.

The endpoint race is the deployment’s to close

Endpoint removal and SIGTERM are not sequenced against each other: a pod can receive the signal while requests are still being routed to it. No shutdown handler can serve a request that arrives after its listener is closed, so the fix is a preStop sleep covering the propagation delay, and a terminationGracePeriodSeconds large enough to hold the sleep and --drain-secs:

terminationGracePeriodSeconds: 60
lifecycle:
  preStop:
    exec: { command: ["sleep", "15"] }

There is no readiness route to flip, for the reason there is no /health at all — see the operator surface. A draining instance closes its listeners, so any probe that connects already fails.

Deploying a plane

Two artifacts run the same plane; both start from the files agentplane init --serve writes (getting started).

Compose (compose.yaml, written beside the others) runs the :full image of the version that wrote it against Postgres. The plane starts once Postgres answers its health check, as the user that owns the 0600 token file — never root, which init --serve refuses — with a read-only root filesystem; the manifest and policy are mounted read-only and the token file as a compose secret, never an environment variable. Every port is published on the host’s loopback only: A2A on 127.0.0.1:8080, MCP on 127.0.0.1:8081, the operator API on 127.0.0.1:9090. To serve other machines, publish 8080 and 8081 on an address they reach, set --url to the public A2A endpoint, and add a --mcp-allowed-host line for every name callers use to reach the MCP port. Postgres requires a password init --serve generated (postgres.password, 0600, read through POSTGRES_PASSWORD_FILE); the plane reads its connection string from store.env (0600) as AGENTPLANE_STORE, so no secret is on a command line. The network Postgres sits on is internal, but on Linux the host has an address on every bridge network, so the password — not the network — is what keeps a local process out.

The Helm chart (deploy/helm/agentplane/, not published to a chart repository) takes the manifest and policy as --set-file values, the token file only as an existing Secret, and a Postgres connection string as an existing Secret too (storeSecret.existingSecret), read into the pod’s environment rather than its args; a store value holding a password is refused. The pod runs non-root and read-only with every capability dropped; A2A and MCP share one Service and the operator listener has its own ClusterIP Service. Rendering fails for more than one replica without a Postgres store, since a redb file admits one writer, and the grace period is the drain plus ten seconds. With more than one replica the shared Service sets sessionAffinity: ClientIP, because a 2025-11-25 MCP session lives in the memory of the pod that opened it and any other pod answers it 404; behind an ingress that hides client addresses, route on the Mcp-Session-Id header instead. A changed manifest or policy rolls the Deployment: the chart reconciles nothing, because an upgrade is a rollout. serve reads the token file and the store once, at start, and never reloads them: a rotated Secret rolls the pod on the next helm upgrade (the chart hashes both Secrets as the cluster holds them), and otherwise needs kubectl rollout restart.

Two backends, one contract

JournalStore states three guarantees and requires them atomically: fencing, exactly-once, chaining. They are storage invariants deliberately — application logic can be bypassed by the next caller, a constraint cannot.

The contract is written once, in testkit::conformance, and every backend is run against the same battery. It ships rather than living in tests/ because an embedder bringing their own store needs it too.

Two design choices in the battery:

• It reports every violation, not the first. Bringing up a backend is iterative, and stopping at the first failure hides whether the second is a separate bug or a consequence.
• It fails if it checked nothing. A battery that silently runs zero checks reports success, which is the worst outcome available to it.

At-most-once admission is part of the contract

A RunAdmitted record carrying an idempotency key claims (tenant, key) in the same transaction as the append, and a second one under an issued key is refused with DuplicateAdmission naming the holder. The battery checks the refusal, that the named holder is the run that actually won, that the key is free before the append and taken after it, that a rejected batch spends no key, that retirement frees a key without touching the run it named, and that an unkeyed admission claims nothing — the direction that would otherwise break every plane never using the feature.

It also checks that a live lease is not stealable: LeaseHeld, distinct from Fenced, because a fenced writer must drop the run and a writer refused a live lease should wait.

The case layer

Five more stores, each settling one race:

Each has a battery in testkit::conformance_case, run against both backends. The container tag is pinned in the test rather than inherited: the testcontainers-modules default is postgres:11-alpine, which has been end of life since November 2023, so the default would certify this backend against a release nobody should be running.

Postgres settles several more cleanly than the embedded store: UPDATE … RETURNING collapses a read-then-write into one statement, so there is no window to reason about because there is no second statement.

A paged read is part of the contract

JournalStore::recent_runs(after, limit) is the discovery index behind A2A task listing, and the battery checks three things a paged read gets silently wrong: the order is (updated_at, run) descending and total, the limit is honoured, and paging through it in twos reassembles the whole index exactly — no run served twice, none skipped.

The tie-break on run id is contract rather than detail. Both backends keep whole-second timestamps, so runs written back to back share one; without a tie-break a cursor landing inside the tie drops or duplicates whichever moved.

A sequential test cannot detect a race

So correlation also has a racing check — eight racers across four keys, because two concurrent callers serialise often enough that dropping the arbitrating constraint goes undetected — and mutation testing proves the check can fail. A race check corroborates; the constraint in the store is what makes absence real. redb admits one writer at a time and passes trivially.

The guards suite races every store-side concurrency claim against a real PostgreSQL: quota admission, a tool’s rate ceiling across two planes, authority draws, task claims, timer sweeps, case correlation and case-state writes.

Postgres

Three traps, each a plausible way to be nearly right:

• Exactly-once is a partial unique index. A SELECT then INSERT has a window with two writers, and closing that window is the entire point.
• Fencing reads the lease FOR UPDATE inside the appending transaction. Checking the epoch first and appending second re-opens the gap a paused instance wakes up into.
• seq comes from the run’s own chain, never a sequence. Postgres sequences are non-transactional and leave gaps; a gap is indistinguishable from a deleted record during verification.

Each is held by a mutation that weakens the store and requires the battery to name the right invariant.

What the gate costs

The first question a runtime built on “the journal is the plan of record” invites. just perf answers it; these figures are from an Apple-silicon laptop and are quoted with the command precisely because they are hardware-specific.

Each control is measured on its own store and reported by its best of three runs, as the delta it adds to a bare effect. spread is what the method resolves — the gap between identical baseline runs — and a delta smaller than it is the machine rather than the control:

2000 effects × 3 runs, redb in memory, 3 policy rules
  journal     0.094 ms/effect      10586 effects/sec   canonicalize, chain, commit
  policy      0.120 ms/effect     +0.026 ms vs journal  one authorize per effect
  sink        0.098 ms/effect       under the spread  label gate, one protected field
  replay      0.003 ms/effect     291838 effects/sec   the read path, nothing performed
  spread      0.011 ms/effect                     across 3 baseline runs, what this resolves

400 effects × 3 runs, redb on disk, 3 policy rules
  journal     8.471 ms/effect        118 effects/sec   canonicalize, chain, commit
  policy      8.152 ms/effect       under the spread  one authorize per effect
  sink        8.137 ms/effect       under the spread  label gate, one protected field
  replay      0.004 ms/effect     250646 effects/sec   the read path, nothing performed
  spread      0.758 ms/effect                     across 3 baseline runs, what this resolves

One axis dominates, and it is not the one people ask about

The durability point is the whole figure. Authorization costs about 26 µs per effect at three rules, and the label gate over a protected field does not rise above what three identical runs disagree by. Against 8.5 ms of fsync neither is resolvable at all — the gate’s own checks are two orders of magnitude below the commit that records them. An adopter tuning this tunes the store, not the policy set.

Two consequences worth stating. Policy evaluation scales with the rule set, so a plane running forty rules pays proportionally more than the three measured here — still nowhere near the commit, but it is the axis that grows. And a control that reads as free here is not free everywhere: these are the costs of checking, and what a refusal costs the work it stops is a different measurement, taken against a deployment’s own bundle rather than against this loop.

Read the gap, not the numbers

Two fsyncs per effect. An effect crosses the protocol twice — EffectStarted before dispatch, its terminal record after — and both are durable commits at Durability::Immediate, because intent precedes action: the announcement must survive the process before anything reaches the world. The ~8.5 ms is that guarantee’s price, not an inefficiency waiting to be tuned away. Batching the two would break the only thing standing between a crash and an unrecorded payment.

Against what an effect normally is, this is noise. A model call is seconds; a tool call is tens of milliseconds. At 8.5 ms, journaling is well under 1 % of a real agent’s wall clock, and the whole design is priced for effects that reach the world.

It stops being noise when effects are cheap and many. cx.now(), a case read, a memory recall — a plan doing hundreds of those pays 8.5 ms each. If a step is looping over cheap effects, that loop is the cost.

On redb it is a plane-wide ceiling, not a per-run one. redb has a single writer, which is exactly what makes its exactly-once key and its fencing free of races — and it means ~118 effects/sec is the whole plane’s durable write budget on this hardware, not one run’s. A single-node plane running agents that call models will never notice. A plane running many concurrent runs of cheap effects will, and that is the signal to move to PostgreSQL, which is also the answer for more than one instance. It is a flag, not a rewrite: every verb takes --store postgres://… beside --tenant, including serve, in a build with postgres → which verb needs which feature.

Replay is not the same cost, by a wide margin. It performs nothing and reads history back — 2000× faster on disk. That matters more than it looks: a divergence check, a crash recovery and an offline audit all pay the read price, so the expensive half is the half that only happens once.

PostgreSQL is deliberately not measured here. It is a different machine, a different fsync, and usually a network hop; quoting a number produced on a laptop’s container would be worse than quoting none.

The sweeper

One tick, several findings — not all of them alarms, and the first two routine healing that still means something died. Recovery runs first, because an abandoned run may be one step from meeting a deadline the passes below would otherwise breach; then deadlines, task windows, event redelivery and dead-lettering, and finally timers:

now is passed in rather than read, so the caller controls the clock. That keeps the sweeper testable at all, and lets a simulation drive a year of obligations through in milliseconds.

Not every field of SweepReport is an alarm, and not all of them are numbers. timers_fired is the system working — its own documentation says not to alert on it — and runs_recovered is routine healing whose real message is that an instance died. saturated is a set of flags saying which passes came back full and so may not have seen everything that was waiting; record names the sealed run holding the tick’s own evidence, when the tick did anything. Two fields sit apart from the counters. evidence_lost is the most serious thing the report can carry: the tick decided something — obligations breached, cases escalated — and the durable, tamper-evident account of who decided that and when could not be written. census_unavailable means the gauges could not be read this tick, so the census in the report is a default rather than a reading — a blind spot wearing a zero.

The witness pass is the one phase whose counterparty is somebody else’s server, which is why it runs last: a slow witness must not delay a breach or a recovery. cosignatures is evidence accumulating — the system working, like timers_fired. witness_shortfall is a plane running with fewer independent parties vouching for its history than the deployment declared it required — counted by distinct witness key, so one witness listed twice counts once — and nothing else will clear it. witness_integrity is a witness refusing on integrity grounds — the log shrank, forked, or claimed growth the witness could not verify — and it is reported even when the quorum was met, because two honest cosigners do not answer a third that remembers a different history. It is also the one sweep finding that goes to tracing as well as to the report: the audience for a witness says this history moved is not only the operator running the plane.

needs_attention() is the alerting predicate, and it enumerates exactly what a human should see: breached, tasks_expired or dead_lettered above zero, any saturated pass, recovery_failures or wake_failures above zero, evidence_lost, census_unavailable, and either witness count above zero. Recoveries that succeeded stay off that list — they are the plane healing, findable in the report and in the sweep’s own run. is_quiet() is the broader predicate: a healthy plane sweeps silently, so a non-silent sweep means something happened, even when it was only the system working.

The recovery pass: who resumes a crashed run

The candidate set is exact, not heuristic. Every clean exit hands its lease back — sealed, failed, suspended — so JournalStore::abandoned_runs answers one precise question: which leases expired while still naming an owner. That is the set of runs somebody was executing when their process stopped, and the sweep resumes each one: takeover bumps the epoch, the store fences the dead owner’s next append, and replay reads completed effects back rather than redoing them. A run whose last record is already its conclusion died between concluding and handing the lease back; the sweep finishes that — seals it or releases the lease — and never retries the work, whatever the outcome was. cargo run --example recovered_run walks the whole thing on two in-process instances: one dies mid-run, the other’s sweep finds it by its lapsed lease and finishes it, no stage repeated.

Three details are worth knowing at 3 a.m.:

• runs_recovered above zero means an instance died, even though the runs themselves are fine. The healing is routine; the dying is not. A steady recovery rate with allegedly healthy instances is a contradiction — go look at why leases are lapsing (GC stalls and CPU starvation produce exactly this).
• recovery_failures is one stuck run, not many. The run stays listed and is retried every tick, so a persistent count is the same run failing repeatedly — and nothing else will unstick it. The reason is in the log line of the failing tick. This is the one recovery number needs_attention() fires on. A recovery refused by the journal itself — a record this build cannot decode, a chain that does not verify, a canonicalization it does not implement, a recorded chain it cannot rehydrate — would be refused the same way on every tick, so it is not retried: the run is quarantined with the refusal named, and lands on the quarantine backlog for a person to reopen or abandon. Everything else, a plane-dependent refusal included, is retried.
• The batch is small (32) on purpose. Recovering a run replays its journal and then executes live from the frontier — which may dispatch a model call — so a mass failure drains over several ticks with saturated.recovery up rather than holding one tick hostage. The flag means at least a batch was waiting, never that the batch was all there was.

Each takeover is written into the sweep’s own sealed run as run_recovered, because a takeover fences the previous owner and who fenced whom, and why must be answerable from the journal rather than inferred from an epoch gap. The note lands before the resume: a concluded resume releases the lease and leaves the recovery queue, so an account written afterwards would be the one write a crash could lose with no retry ever selecting the run again. A note that cannot be written skips the takeover for a tick that can write it, and the outcome stays out of the note — the run’s own journal answers it.

One state recovers to nothing, deliberately: a lease over an empty journal means admission acquired and died before its first append landed. No run exists — the atomic admission batch never committed, so nothing was declared, authorized or performed — and clearing the lease is the whole recovery.

The claimed-event window is covered by its own pass. A delivery can die between an event’s claim and the resume that consumes it — a crash in that window, or an owner that outlives the delivery’s bounded retry — and the message then belongs to nobody: claimed, so no longer waiting; undelivered, so no run holds it. The delivery parks the pair, and the sweep walks the parked pairs — never every registered wait, whose long legitimate members would fill its page first — finishes those deliveries and reports each under events_redelivered. A parked wait already holds its message, so no second message is matched to it while it waits for redelivery; a parked wait whose run has since been sealed is retired rather than tried, as is every wait a run held when a resume repairs the seal a crash left missing. The redelivery is the system healing; a persistent count means deliveries keep dying, which is worth asking why.

A finding has to be findable

Every conclusion this runtime reaches is queryable by whoever must clear it, without them already knowing which run, case or matter to open. Each backlog is a question, not an id:

curl -H "$AUTH" 'https://plane/runs?outcome=quarantined'   # history you cannot trust
curl -H "$AUTH" 'https://plane/cases?status=escalated'     # matters somebody must pick up
curl -H "$AUTH" 'https://plane/obligations'                # windows we missed
curl -H "$AUTH" 'https://plane/tasks'                      # decisions waiting on me
curl -H "$AUTH" 'https://plane/dead-letters'               # messages that reached nobody
curl -H "$AUTH" 'https://plane/push'                       # receivers that stopped accepting

All six page the same way: truncated says whether there is more, and the order puts the item you are most likely to want first — newest for runs and cases, longest-overdue for obligations. Oldest-first is safe for obligations only because the listing drains: POST /obligations/acknowledge removes a breach from it.

/runs reads an index derived from the RunConcluded record inside append, in the same transaction, so it rebuilds from the chain and is never an authority. The last conclusion wins — a failed run moves to succeeded when a resume concludes it, so the failed backlog drains. Failure does not seal: a failed, exhausted or quarantined run stays open, and only succeeded, cancelled and abandoned freeze the journal. A quarantine is a pause rather than an ending — the runtime is saying it does not know — and freezing its chain would lock out the one record that answers it.

The single-run view’s sealed field is backed by Merkle inclusion, not by the presence of RunConcluded. That distinction is observable: a failed run has a conclusion and reports sealed: false, while a succeeded run reports true. Exhaustion remains structured in the journal and in operator push events, so automation can inspect the exact ceiling without parsing reason.

/cases defaults to escalated. An unrecognised status is a 400 rather than a quiet fallback.

/obligations takes no status: it lists breaches. It reads the obligation’s own row rather than the case’s status, so it still answers after the matter is closed.

It lists breaches nobody has accounted for. POST /obligations/acknowledge names the case and the obligation and records who looked, taken from the authenticated caller rather than the body. The obligation stays Breached — what ends is the question, not the fact — and the account is readable afterwards on GET /cases/{case}. The call is idempotent and says which happened: recorded is false when somebody had already answered, because the first account stands and a retry must not rewrite who looked or when.

Alert on agentplane.obligations.breached, which is the same figure as a level rather than a page.

/dead-letters and /push are the two backlogs that are diagnoses rather than work items, and both carry less than the store holds. A dead letter means a correlation key does not match what a run subscribed to, so the listing gives the identity and the keys and not the message body — that is the counterparty’s content, and on a sealed plane it is not even readable without a key. A parked registration gives its config redacted, because the token a receiver registered is its own correlation secret and a listing is not where it is handed back. POST /push/rearm answers rearmed: false when nothing was parked under that name — already live, or never there — rather than reporting success to somebody who would then wait for a sweep with nothing to do.

Grant the read verbs explicitly. api:run.list, api:run.live, api:case.list, api:obligation.list, api:deadletter.list, api:push.list and api:halt.list are what an on-call person needs, plus api:obligation.acknowledge for whoever answers a breach — separate from reading the list, because the party allowed to see what a deployment missed is not automatically the party allowed to declare it answered. An allowlist built from route names alone will miss them; under a default-deny engine an ungranted verb means the backlog is refused to everybody. Enumerate action::ALL when writing rules rather than reading the route table. api:obligation.list is separate from api:case.list so a compliance function can be given what did we miss without the contents of every matter, and api:task.takeover is separate from api:task.claim so displacing an absent colleague can go to a queue lead without going to every reviewer. api:run.live is separate from api:run.list on the same principle: one enumerates what has finished, the other is a live map of what a tenant is doing right now.

Answering a quarantine

A quarantine is the runtime saying it does not know. An effect was announced, the process died or the provider went quiet, and whether the call reached the world is unanswerable from the journal — so nothing unwinds, because compensating around an unknown outcome is a refund for money nobody took.

A run is also quarantined when a resume is refused because the declaration or policy bundle this plane holds is not the one the run was admitted under; its reason names both digests, and it is answered by bringing back the recorded revision and reopening, or by abandoning. Either way the run stays listed by attention until somebody answers it.

That is the one conclusion a resume cannot clear, and the only backlog whose level moves because a person moved it. Three verbs, and they are deliberately three: supplying a fact is not deciding a run, and deciding a run is not declaring its outcome.

1 — Find out what is in doubt. The run view names the call, not just the situation:

curl -H "$AUTH" https://plane/runs/$RUN

{
  "run": "run_01K…", "status": "quarantined",
  "reason": "the provider could not establish whether the call landed",
  "undecided": [
    { "effect": "9f2c…", "step": 1, "phase": "forward",
      "kind": "tool://payments/charge", "doubt": "inconclusive" }
  ]
}

doubt says where to start looking. announced means the runtime never heard back — the crash shape, nothing known beyond the fact that the request left. inconclusive means something did report and could not tell, so there is a message to read and a provider that has already been asked once.

2 — Say what actually happened. Look the call up in the system that would know, and record the answer:

curl -H "$AUTH" -X POST https://plane/runs/$RUN/reconcile -d '{
  "effect": "9f2c…",
  "disposition": "landed",
  "output": { "charge_id": "ch_9RtQ", "captured": true },
  "note": "charge ch_9RtQ exists in the provider console, created 12:41Z"
}'

did_not_happen needs no output and leaves the effect safe to perform again. landed carries the result the run reads back. Three rules are worth knowing before you use it:

• It supplies a missing fact and never replaces a recorded one. Only an effect listed under undecided may be answered. Anything else is a 409 — otherwise an operator could talk a run out of compensating work that is standing in the world, and the journal would show an orderly reconciliation while it happened.
• Your name is on it. The record is the same EffectReconciled a probe writes, plus asserted_by. “The provider told us” and “somebody asserted it” are different evidence and the chain keeps them apart.
• The value you supply is untrusted. No effect produced it, so it takes the conservative point of the lattice, and a run that needed that output to be trusted is refused at its next gate and unwinds.

3 — Hand it back, or write it off.

curl -H "$AUTH" -X POST https://plane/runs/$RUN/reopen  -d '{"reason":"charge confirmed in provider ledger"}'
curl -H "$AUTH" -X POST https://plane/runs/$RUN/abandon -d '{"reason":"two weeks of provider tickets; nobody can say"}'

reopen does not declare an outcome. It hands the run back to the runtime, which re-derives its verdict from a history that now holds what you established — and quarantines it again, on the record, if the answer was wrong or incomplete. The response is what the run actually reached, so you learn immediately rather than polling. One decision authorizes one pass: a run that quarantines again needs a fresh one.

abandon closes it where it stands. Nothing is unwound — the world keeps whatever the run left in it, because unwinding around an unknown outcome is exactly what quarantine exists to forbid and impatience is not evidence. The run seals as abandoned and leaves the quarantine backlog.

The doubt does not leave with it. Abandoning takes the run off the only listing that carried it, so what it left standing becomes an agentplane audit finding derived from the journal, where no later action can clear it:

run run_01K… is sealed but effect 9f2c… (step 1, inconclusive) never reached a
known outcome — nothing may resume a sealed run, so whether the call changed the
outside world is permanently undecided

Cancelling a quarantined run is refused, not recorded. cancel promises to unwind and put the world back, which is the one thing a run holding an unknown outcome may not do; the refusal names these two verbs instead.

Three verbs, three policy actions: api:effect.reconcile, api:run.reopen, api:run.abandon. Keep them apart in your rules. The person who can look a charge up in a provider console is often not the person who decides what the run does next, and neither of them is necessarily the person who may write off an unexplained mutation.

Retention for the admission index

Admission keys are kept until you retire them:

agentplane forget-admissions --store ./journal.redb --older-than-days 30

The window has no default. Retiring a key reopens the door it closed, so a window shorter than your emitter’s retry horizon admits a second run.

The index is a row per admitted message and grows with inbound volume — a size your database monitoring already reports. The verb prints what it retired, so a pass that found nothing is distinguishable from one that said nothing.

Taking the record away

Two verbs read a journal and nothing else — no manifest, no source tree, no Rust toolchain — because that is what an auditor or a departing tenant holds:

agentplane export --store ./journal.redb > history.jsonl
agentplane audit  --store ./journal.redb > report.json
agentplane verify history.jsonl --checkpoint cp.note   # check a copy, offline
agentplane restore history.jsonl --store ./rebuilt.redb
agentplane policy check --bundle ./policy --from history.jsonl   # re-derive the verdicts

--checkpoint is the deletion check, and without it there is none. The Merkle root rebuilt from the file can otherwise only be compared with the file’s own header — which whoever dropped a run rewrites too — so the report lists deletion under not_checked rather than calling the file sound. Pass the checkpoint an earlier audit printed, or the tlog-checkpoint note a witness cosigned: the point is that it comes from somewhere other than the file being checked.

export writes JSON Lines: a header naming the log, its checkpoint and the canonicalization rule the digests were computed under; one line per record carrying prev_hash and hash, so the chain re-walks from the file alone; one line per case — the case layer is beside the journal, not derivable from it, so a file without it rebuilds a journal whose records name matters that no longer exist; and a trailer. The trailer’s absence is the signal — a file cut short by a full disk or a killed pipe ends without one, and every line in it is still valid JSON, so counting is no help to a reader who does not have the source. Runs that could not be read are named in the trailer rather than quietly missing, and the trailer’s case count is what catches the case layer stripped whole.

Case state travels as stored: on a sealed plane that is ciphertext, because an export of plaintext would quietly undo erasure — the key destroyed tomorrow would no longer reach the copy taken today. Two questions stay live rather than offline, and verify reports them as unchecked instead of passed: whether the blob bytes behind the exported digests are present, and whether sealed state’s keys still unwrap.

Those two are Runtime::drill’s job, run with the stores the plane actually runs with: every case’s blob digests read through that case’s own handle — the unit-scoped address the plane wrote to, opened through the sealing envelope when a ring is wired — then re-hashed (get, never has — presence without integrity passes over altered bytes), and sealed state proven to open with the plaintext dropped on the spot. The report’s verdict has four answers: intact; erased by design — a tombstone, or a destroyed key on a case whose erasure record accounts for it, counted and never a finding; lost, which pages — a key gone with no erasure record is lost, since a key service cannot tell a destroyed key from one it never held; and the bytes are gone and their tombstone does not read, which pages too — the erasure may well have run, and nothing left in the store can say so.

The same rehearsal has a CLI verb for deployments that never write Rust: agentplane drill opens the store the flags name, prints the report as JSON, and exits non-zero only on loss — erased-by-design counts stay informative. A store file holds no blob backend and no key ring, so those halves land in the report’s unchecked list rather than being silently passed; the library call on the running plane remains the complete form.

The report also carries releases and warrants — every point at which a label was raised, and what authorized each run. The gate on both is integrity, not innocence: a run whose chain did not verify shows neither, because nothing drawn from those records can be trusted, and a run that verified and then failed a check shows both, because that is the run an investigator opened the report for.

Read warrants with unadmitted, its complement: every run that verified is in exactly one of the two. A run with no admission record is listed under unadmitted with the outcome it concluded under — the sweeper’s own runs, for one.

audit prints the report as JSON and exits non-zero on findings — but not on not_checked, which is a separate list and the one worth reading. An audit given no public key and no earlier checkpoint still walks every chain, and says in that list that it could establish neither authorship nor deletion. Supply them to narrow it:

agentplane audit --store ./journal.redb \
  --key plane-1=<64 hex chars> \        # the signer's Ed25519 public key
  --prior last-report.json              # the `current` field of an earlier report

--key is the authorship check, repeatable per trusted signer; add --require-signatures to make an unsigned record a failure rather than a note — off by default, because history written before signing was configured is legitimately unsigned. --prior is the deletion check: the report’s own current field, saved from an earlier pass, and a log that shrank or forked since then is a finding — each audit prints the checkpoint the next one checks against.

Both verbs default to every sealed outcome. --outcome narrows and --limit bounds, and reaching the limit is never a pass. An audit records it in the report’s truncated field — {"limit": …, "reached": [outcomes]}, so it survives > report.json — and exits 5 when nothing it read was wrong. An export that reached it is refused before a byte is written, because a truncated file is framed exactly like a complete one: raise --limit, narrow --outcome, or pass --allow-partial to write it anyway, which still exits 5. See exit statuses.

Disclosing one matter

A request for one matter is answered with a package, not the whole export:

agentplane export --store ./journal.redb --case case_<ulid> \
  --to "Supervisory authority, ref 2026-114" --actor dpo@example > matter.jsonl
agentplane disclosures --store ./journal.redb --case case_<ulid>

The package carries the runs the case holds now (--run adds single runs), each sealed one with its inclusion path, and only the cases those runs belong to; the format page has its shape and how both readers verify it. The recipient checks it against a checkpoint obtained from you some other way: at the package’s size it is compared by root, at another size it is reported not compared. restore, replay --strict --from, policy check and grants refuse a package by name.

Before a byte reaches the destination the disclosure is recorded — recipient, runs, sealed or not, checkpoint, package digest, actor — so a later erasure names the copy. If the record cannot be written nothing is delivered. An unknown case or run, an --output that is a directory, or one whose directory does not exist exits 2 with nothing recorded. The package is staged as .<name>.dsc_<ulid>.partial beside --output, synced, and renamed into place once recorded; a process killed in between leaves that file, which nothing reads and you may delete. Once recorded the act stays recorded: a package written to a standard output that closes early is reported as recorded and not delivered.

Each case travels as its whole block — state, deadlines, blob digests, hold reason, and the ids of every run it holds, including runs the package does not carry. A package proves the inclusion of what it carries, never that it carries everything the matter holds. The register is a row in the store, like a hold: disclosures lists it and says so.

Re-deriving the policy verdicts

policy check reads an export and a policy bundle — the same --bundle loader serve --policy uses: one .cedar file, or a directory holding policy.cedar and optionally schema.json and entities.json, and nothing else — and evaluates every gated request the records support, offline. It opens no store and writes nothing. It needs the cedar feature (the :full image has it).

agentplane policy check --bundle ./policy --from history.jsonl --tenant acme
agentplane policy check --bundle ./policy --candidate ./policy-next --from history.jsonl --json

Each run is recorded (the bundle’s identity is the one its admission names, so it was evaluated), a mismatch (another bundle governed it; not evaluated), or ungoverned. A finding is a recorded permit the bundle refuses. --candidate adds, per run, the recorded permits the candidate would newly deny or cannot evaluate — measured against what happened, so a mismatched run is measured too. The report lists what it could not rebuild and why (sealed, erased, a refusal’s request, a gate the record cannot show was passed); security has the list. --tenant is needed on a multi-tenant plane: every request carries the tenant and no record does, so the report says whether it was supplied.

It exits 1 on a finding, a mismatch or a non-empty candidate diff, 5 when it evaluated nothing, and 0 otherwise. It checks agreement between a bundle and the record, not the record’s integrity: verify the file first.

Replaying an edited declaration

replay --strict re-executes a recorded run under the manifest it is handed and serves every model completion, tool result and peer reply from the journal. Handed an edited manifest, it answers the question an author has in review: would this change have made last week’s runs do anything differently, and where?

agentplane replay run_01M3… --manifest edited.yaml --strict --store runs.redb
agentplane replay --manifest edited.yaml --strict --from monday.jsonl --from tuesday.jsonl

run run_01M3… — diverged
  recorded: summariser 1.0.0 420d054e…
  candidate: summariser 1.0.0 9623cd59…
  (different digest)
  first divergence: step s0, forward phase, `model.complete` — history ek:024c…, this build ek:3dff…

Each run gets one verdict, and every verdict names both revisions — the declaration digest the run was admitted under and the one in hand:

It writes nothing — no lease, no conclusion, no index moves — and it reaches nothing: every provider, tool server and peer the manifest names is answered by a stand-in that refuses any call, so no provider credential is read and --mcp and --peer are refused rather than ignored. A model call’s identity includes its driver’s request profile (endpoint, schema mode, streaming), which is deployment wiring, so the stand-in takes it from the record: the verdict is about the declaration, not about this machine’s provider setup.

--from reads an export instead of a store: each file is rebuilt in memory, checked against its own checkpoint, and every run in it is replayed unless a run id names one. A corpus exits with its worst verdict — 4 if a run could not be read, then 1 if any diverged, then 5 if any could not be replayed, else 0 — so a CI job holding exports and a pull request’s manifest reads one status.

What it cannot see: a policy bundle edit (a strict replay replays recorded decisions and never asks the engine, which is what keeps a relaxed rule from re-judging old runs), and anything the declaration contributes outside the effects a run made — a raised ceiling diverges only on runs the old one stopped. A replay that serves some boundaries from the record and runs others live is refused, not offered: a live answer grafted onto old history is an observation the journal never recorded, and a model call made to explore an edit is a fresh run. A resume under a different declaration is still quarantined before anything replays.

One run’s timeline

agentplane history <run> --store <file|postgres://…> prints a run’s journal in sequence order, one line per record: sequence, kind, step, and the record’s payload. --from <seq> starts later. Every line passes through the escaping a reviewer’s rendering uses, so a bidirectional override or an escape sequence in a recorded string is printed as \u{…} rather than reaching the terminal. --json prints each record as GET /runs/{run}/history serves it; it is machine output, and JSON escapes only U+0000–U+001F, so DEL, the C1 controls and the bidirectional and invisible characters print as they are — read the text form on a terminal. A run the store does not hold exits 1, whatever --from names.

The command line’s exit statuses

One table for every verb, printed at the foot of agentplane --help, because a scheduler reads the status and nothing else:

A finding and an outage are different pages, which is the reason for the split.

Output. The verbs whose answer is a report — audit, verify, export, attention, tasks, waiting, drill and the other operator verbs — print JSON on stdout, always. The verbs whose answer is a sentence — halt, init, validate, digest — print text for a person, and one JSON document on stdout with --json; so does policy check, whose report is long enough that a person reads the summary and a pipeline reads the JSON. Everything else goes to stderr. serve --log-format json (AGENTPLANE_LOG_FORMAT) writes one JSON object per log line for a collector, and agentplane --version names the features the binary was built with.

Which verb needs which feature

--features cli builds every verb against a redb file, with every model provider and the fake one. What reaches further is one feature away, and a build without it refuses with exit 2 and names the flag to reinstall with. The :full image carries all of these; :slim carries only cli.

What no audit can answer: the runs that never started

A run refused before it exists — a policy denial on run:admit, a tenant ceiling, a standing halt — has no run id and so no chain to append to. Nothing about it is in the journal or in the report, so how often did policy stop a run from starting is a question a clean report answers with silence rather than with zero.

Recording it would give a caller a way to grow the store by being denied. Every refusal after admission is journaled, which is why an effect-level denial has a PolicyDenied record and this one does not. The number lives in the agentplane.policy.denials metric and agentplane.policy.denied telemetry, both carrying the action, with the durability of whatever collects them.

Auditing open runs (--outcome failed, say) is not an alarm: an open run has no Merkle leaf, so it is checked on chain and signatures and the report says in not_checked that nothing pins its tail until it seals. The finding is the opposite case — a run whose own records carry a sealing conclusion, in a log that holds no leaf for it. That is history the log no longer commits to.

A sealed run is also held to its own transactional brackets: a GroupOpened with no GroupSettled under a sealing conclusion is a finding. In an open run that shape is the ordinary crash the resume repairs; under a seal nothing may resume, so whether the group’s members were taken or taken back is permanently undecided — a state no honest writer produces.

For a sealed run, the log’s leaf is held to the verified chain’s own head, before the tree math and independent of any checkpoint race, so a truncated but internally consistent prefix does not pass. A mismatch is a finding: records were removed or replaced after sealing.

verify is the drill, and it takes the file alone. It re-seals every record through the same function the store sealed with — so agreement is evidence about the bytes, not the file agreeing with itself — checks sequences are contiguous, holds every record to the run block it sits under (a relabelled block passes chain, leaf and Merkle checks, because those verify the bytes and only the label lied), then rebuilds the Merkle log from the positions each run block carries and compares the root against the checkpoint in the header. With --key it also verifies signatures, strictly: inside a signed history, an unsigned record is the one an attacker who cannot sign would add.

That last check is what catches a whole run deleted from an export: every remaining chain still verifies, and only the rebuilt tree notices the missing leaf.

Exit codes: findings fail, not_checked does not. A pass with no public key has established less rather than failed, and the report says which.

Both verbs print one document, so a redirect cannot separate the verdict from the grounds. Beside the report is an anchor object:

held_to on the report names the checkpoint itself; the anchor says how it was obtained, and never repeats the value.

Name a second --witness. Two witnesses holding one tree size with two different roots is a split view, which no check against the store alone can find. Witnesses at different sizes are not a split view.

How fresh the anchor is. A witness signs a timestamp with every cosignature. audit --max-checkpoint-age SECS judges each witness key’s latest signed time against the auditor’s own clock: older than the bound — or ahead of it by more — is a finding; without the flag, freshness is listed as not checked, and verify never judges it. A gap bounds when records could have been cut without a witness noticing, not whether they were. A plane keeps the times fresh by declaring serve --witness-submit URL --witness-key NAME=KEY --log-key NAME=PATH --witness-interval SECS: the sweeper re-submits an unchanged checkpoint once the interval passes, an interval shorter than --sweep-every is refused, and one with no witness is refused at build. One --witness-submit URL given twice is refused, since a quorum counts distinct witnesses. The second reader checks the same cosignatures on a note anchor under its own --witness-key NAME=BASE64, and judges freshness under --max-checkpoint-age SECS (with --now RFC3339 for a fixed clock) by the same rule; a time whose cosignature did not verify is an unauthenticated number, and neither reader judges it.

A grader’s verdict, bound to what it judged. agentplane bind <export> --run R [--last-seq N] --content FILE --out SIDECAR writes an unsigned sidecar binding opaque verdict bytes to records 1..=N of a run; the grader signs it with its own key. verify --grader-verdict SIDECAR --grader-key ID=HEX reports each one bound, refused or not checked, and a refused one fails the command. It proves which records the verdict names and who signed it — not what the grader saw.

restore rebuilds a journal from an export and proves it by one comparison: equal Merkle roots at equal size. That is a far stronger statement than “the rows loaded” — it means every record, in every sealed run, in the order the log recorded them, rebuilt to the same commitment. A run restored this way strict-replays on a plane that never executed it.

Sealed is the load-bearing word, and it decides what to export. The log commits to runs that ended, so a file carrying no in-flight run rebuilds to exactly the same root at exactly the same size as one carrying all of them — a clean verify says nothing either way. And the runs still in flight are the ones no outcome index names: runs_by_outcome indexes conclusions, so a sleeping run, a run awaiting a message and a run waiting on a person are in none of them. agentplane export asks for them separately and says on stderr how many it carried; export::runs_in_flight is the same selection for an embedder. Narrowing with --outcome turns it off, because naming outcomes is a request for exactly those conclusions.

It rebuilds the case layer beside the journal: every matter is queryable again — case, correlation, the status worklist, the deadline sweep, the blob links erasure walks, the legal holds that stop a retention pass from erasing a matter, and the erasure record that keeps an erased case erased — and the conformance battery holds the import to every one of those read paths on both backends. A store that already holds a case refuses it: a restore rebuilds a case layer, it does not merge one.

Before anything is written, each record line must be replayable as written: its hash must cover its raw bytes and the chain before it, and its v must be the version this build writes. A file failing either is refused whole, with the store untouched — append re-derives every hash from what it is handed and stamps this build’s version, so replaying such a line would rebuild a history the export never committed to. A target that seals payloads as it writes (a journal wrapped by the keyring) is refused the same way: sealed payloads are restored as the ciphertext they are, so restore into the unwrapped store and open it with the ring. A failure after the first write — the store refusing a write, or a rebuilt hash missing the file’s — leaves a partial store that a retry refuses as already holding the run: discard it and restore into a fresh one.

It replays the ordinary append path rather than writing rows, so every derived index append maintains is rebuilt. Runs are sealed in log-index order, because that order is the Merkle log, and epoch is carried, not re-derived, because it is inside the hashed body.

The fencing token picks up where the history left off. A first lease on a restored run is issued one past the highest epoch the run’s own journal records, not at 1, so a fencing token cannot go backwards across a restore. An epoch issued and never written under leaves no record and is not seen, so the old deployment must not reach the restored store.

What does not survive is named in the report rather than left to be discovered.

A restored wait is armed by nothing. RestoreReport::awaiting lists the runs whose history ends in a wait, as ids. The wait is journaled, but its timer, subscription and worklist row live in stores the export does not carry, and the run released its lease when it suspended, so the recovery pass does not see it either.

Resuming it repairs it. A resume replays to the wait, finds no terminal record, and re-arms the timer, re-subscribes and re-opens the task row from the journal, at the instant the original recorded. restore cannot do it — a resume runs the agent’s own code — so the runbook step is resume every run in awaiting.

Four layers are not in the export at all: the worklist and the decisions recorded against it, inbound events nobody claimed, webhook registrations with their delivery cursors, and governed memory beside the batch, quota and standing-authority ledgers. A decision a run already consumed survives, because that run journaled it; one nobody had consumed does not. Re-establish webhook registrations by hand — nothing journals a delivery cursor, so nothing can rebuild one.

Signatures: append attests as the restoring store’s signer, so a history signed by a key this store does not hold comes back unsigned — hashes and the root are unaffected, since a signature is taken over the chain hash and stored beside it, but authorship is gone. Configure the same signer if you are restoring your own log. Activity timestamps: the discovery index is rebuilt at restore time, so recent_runs orders by when history was restored. It is documented as ordering and cursor stability only, and nothing derives a decision from it.

What is exported is what the chain committed to. With a key ring configured that is ciphertext, deliberately: an export of plaintext would put a copy beyond the reach of key destruction and undo the erasure the key ring exists for.

Break-glass

Reaching another tenant’s data is the one exception to isolation, so it is recorded in that tenant’s journal — actor, roles, and a reason that cannot be blank — before anything is served:

let plane = planes.cross(&caller, &target, "INC-42: stuck settlement").await?;

Planes::cross hands back the plane only once the crossing is on that tenant’s record; if the record cannot be written, nothing is served. The whole Caller goes in, so the record names who authenticated.

The ordinary lookup, Planes::get, takes the same Caller and serves its tenant, so no path reaches another tenant’s plane by accident. An embedder writing its own Authenticator decides what a credential means and can mint a Caller for any tenant; that seam is where a deployment defines identity.

cross is what an admin surface should use. record_break_glass is for an embedder recording a crossing they made some other way.

Same-tenant crossings are refused, because recording routine access as break-glass buries the real ones. A tenant this process does not serve is refused rather than defaulted, exactly as the ordinary gate refuses one.

The crossing seals like any other run, so it enters the Merkle log, verifies offline, and lists with the rest:

curl -H "$AUTH" 'https://plane/runs?outcome=broke-glass'

GET /runs/{run} on one answers status: "broke-glass", decided_by with the operator, and reason with the words they gave.

Who may pull it is your policy engine’s decision, not this crate’s.

One matter, one scan

Listing a case’s runs and reading each misses every record written by a run the case does not own — a sweep that escalated it, for one. So the journal carries the case on the record and indexes it:

let history = store.case_history(case, 200).await?;

One range scan, tenant-first like every other key here. The conformance battery holds both backends to finding this matter’s records and returning no other’s.

GET /cases/{id} includes it, with history_truncated when the bound bit, because a shortened list is shaped exactly like a complete one.

The sweep writes its own history

The sweeper breaches obligations, escalates cases and expires tasks with no run asking it to. A tick that decides anything writes its decisions into a sealed run of its own: Swept { subject, action, detail }, one per action, with a typed action rather than a message. It inherits the chain, the per-record signature and the Merkle inclusion every other run has, so the external audit tool checks it without being taught what a sweep is.

let report = plane.sweep(now, grace).await?;
if let Some(run) = report.record {
    // Exactly what this tick decided, verifiable like any other run.
    let entries = store.read(run, 1).await?;
}

A quiet tick writes nothing.

A breach is applied first and noted after; a warning or an expiry is noted first and corrected if it lost. A breach is conditional: a run that met the obligation after the sweep read it wins, and the store refuses the stale breach — so a refused breach writes nothing. An applied one is marked in the same write as owing its account, and the deadline_breached and case_escalated notes retire the mark. A crash between the breach and its notes leaves the mark, and the next tick notes every owed breach before it reads due — once. A warning, and a task expiry that meets an answer somebody already gave, are noted before they act; when the act does not apply the sweep appends not_applied for the same subject, so the last word about each subject is what happened to it. Read a subject’s deadline_warned or task_expired as withdrawn when a not_applied follows it. An expiry that meets a person’s answer also settles the task to that answer, so the next tick does not meet it again.

Dead-lettered events are not in it: the event store reports how many aged out and not which, so they are counted in the report and the emitted event. A sweep run seals as swept, never as succeeded; GET /runs/{run} answers swept with no reason — what it decided is on its records.

A capped tick says it was capped

Each sweep takes a bounded batch — 128 timers, 512 obligations, 512 expired tasks, 128 stalled deliveries, and 32 abandoned runs, the smallest because recovering one executes live from the frontier — so one tick is bounded. SweepReport::saturated names which sweeps came back full, and needs_attention() is true when any did. A saturated tick means at least the cap was waiting — never that the cap was all there was.

let report = plane.sweep(now, grace).await?;
if report.saturated.deadlines {
    // More obligations were outstanding than one tick will take.
    // Sweep more often, or find out why they are accumulating.
}

Outbound delivery: two ceilings, and what each one means

A push sweep is bounded twice, and the two answer different questions:

Raising limit without raising max_in_flight buys a longer tick rather than a faster one; raising max_in_flight without limit leaves work behind that one tick was never going to reach.

Registrations are served concurrently; the records within one are not. A receiver still gets its records in journal order, and the cursor still advances only on a 2xx, because that is the only thing a cursor means.

Metrics

The runtime does not measure durations

Ambient clocks are lint-denied outside named escapes for values that get journaled or are store metadata, and a replayed run would re-measure durations belonging to calls it never made. Durations are derived from spans, by the collector. The spans carry the mode and the replay flag, so a collector can compute latency and exclude replays, which an in-crate histogram could not.

Counters are emitted; gauges are observed

Gauges come from a census query against the store, never from increment/decrement counters a crash can unbalance, and the sweeper emits them on each tick. A gauge is never read from a limit-bounded query, which would flatten at the page size.

A counter cannot report a backlog. agentplane.quarantines counts the moment a run is set aside, in the process, so after a restart it reads zero over a backlog of forty. The number to alert on is agentplane.runs.quarantined — the level, counted from the store’s outcome index on every sweep, which is the same index GET /runs?outcome=quarantined pages.

It falls when somebody answers one — see Answering a quarantine. It is not risk retired: an abandoned run leaves this count while what it left in the world stays, and the doubt becomes an agentplane audit finding.

Two rules, both guarded

A dimension is a variant, never a rendered message. Display on a budget error embeds the allowed and used figures; a label carrying those is one time series per distinct budget, which is how a metrics backend falls over. Every dimension comes from an as_str() accessor.

The catalogue is not a wish list. A declared-but-unemitted counter reads as a hard zero; tests/guards/layering.rs fails the build if a catalogue entry has no emitter.

Observability

tracing spans and events, so the runtime is usable by any subscriber — OTel, JSON logs, a test recorder — without the crate choosing an exporter.

agentplane.run                     gen_ai.operation.name = invoke_agent
                                   gen_ai.agent.name, gen_ai.conversation.id
                                   agentplane.run.id, .mode, .semconv
└── agentplane.step                agentplane.step.id, .capability; agentplane.phase
    └── agentplane.effect          .kind, .key, .attempt, .mutates, .replayed
                                   agentplane.mode; error.type when it failed
                                   gen_ai.operation.name = execute_tool | chat
                                   chat:          gen_ai.provider.name,
                                                  gen_ai.request.model,
                                                  gen_ai.response.model,
                                                  gen_ai.response.finish_reasons,
                                                  gen_ai.usage.{input,output}_tokens,
                                                  gen_ai.usage.cache_{read,write}.input_tokens
                                   execute_tool:  gen_ai.tool.name

Spans follow the OpenTelemetry GenAI semantic conventions where they apply: a tool call is execute_tool, a completion is chat. Each effect declares its own operation and its own attributes rather than having either inferred from its name, so a new effect type cannot pick up a label by accident — and the keys do not cross, because a completion naming a tool would make every model call look like a tool invocation.

Exported through tracing-opentelemetry, spans take the convention’s names and kinds from otel.name and otel.kind: invoke_agent {agent} (internal), chat {model} (client), execute_tool {server/tool} (internal). The tracing names above stay this crate’s own. gen_ai.provider.name is the driver’s well-known value (openai, anthropic, aws.bedrock, gcp.gemini); a chat-completions endpoint names its service with ChatCompletions::gen_ai_provider, or reports the name its model is routed by.

An embedder’s own attributes go on its own span. A run’s span is a child of the span current where the run starts, awaited or spawned, so a span the embedder opens around it — its task, its verdict, its policy decision — sits above the run in one trace.

Where the convention names a fact, the convention’s name is what is emitted. An agent’s name and a conversation’s identity are gen_ai.agent.name and gen_ai.conversation.id, not a second spelling in this crate’s namespace that no generic tooling reads. The conversation is this plane’s case, which is also what the A2A surface answers contextId with.

gen_ai.response.model is the provider’s own word about which weights served the call. It is absent where the wire does not say — Bedrock’s Converse names no model on the way back — and never filled in from the request.

error.type carries the class of fault on any attempt that failed — timeout, refused, rate_limited, metered — beside agentplane.outcome, which says whether it succeeded.

agentplane.effect.key is the join between a trace and the evidence: the journal’s own identity for that attempt, under which GET /runs/{run}/history serves records. It names the attempt without carrying what it sent.

Effects that are not GenAI operations — reading the clock, arming a timer, writing case state — carry no gen_ai.operation.name.

What is deliberately never emitted

The conventions define Opt-In attributes that carry the content of a call: gen_ai.input.messages, gen_ai.output.messages, gen_ai.system_instructions, gen_ai.tool.definitions and gen_ai.tool.call.arguments. This plane emits none of them, and it is not a default to flip. A prompt is exactly where governed values arrive; the sink gates exist to keep those values inside a declared ceiling; and a trace exporter is an egress those gates do not cover. Sensitivity is a property of a value, and a span is not a sink that can carry one.

What a deployment gets instead is the shape of every call — who was asked, what it cost, which tool ran, how it ended — and the journal for the content, where the same values are sealed, labelled and erasable.

The convention defines more that this plane does not emit, each for its own reason. gen_ai.response.id, because no shared part of a completion carries one. gen_ai.tool.call.id on a tool called with call_tool, which answers no model call; the tool-calling loop and StepCtx::call_tool_for carry it. gen_ai.provider.name on the run span, because the agent is executed here rather than served by anyone — the completion spans beneath it each name the provider that answered them. And server.address, because a driver’s endpoint is deployment configuration, identical for every call it makes: that belongs on the OTel resource your exporter sets, not repeated on every span.

The same rule reaches the loud events. A run’s failure reason, a quarantine’s, an undecidable effect’s detail and a compensation’s error are free text a provider, a peer or a skill wrote, and it quotes the request it refused — the caller’s data. The journal seals those fields under a key ring and an erasure destroys them; a log line carrying the same words would outlive both. So each of those events carries the run, the class of fault as error_type (undecidable, timeout, failed, …) and reason_digest — sha256: and the first sixteen hex digits of SHA-256 over the text — and never the text. Read the words through the operator API (GET /runs/{run}), which honours erasure; the digest is what joins the two.

Transport failures follow the same line one layer down: a webhook’s, a peer’s or a media source’s URL routinely carries a bearer secret, so an outbound error is rendered with the host it failed at and never the URL — in a parked push registration, a log line and an effect’s failure alike.

The conventions are still pre-1.0 and published without releases, so the revision targeted is pinned in telemetry::SEMCONV_VERSION by the date of the semantic-conventions-genai commit it was read at, rather than tracked. An upstream change becomes a deliberate migration instead of a silent shift in what your dashboards mean.

Further decisions worth knowing:

• One span per effect attempt. A retried call shows as several spans rather than one long one, which is what makes “how often does this driver need a second try” answerable.
• Replay is marked on every span. A replayed run re-executes its skills and emits spans again. An effect served from the journal is reported as an event with replayed = true, never as an effect span — otherwise “effect latency by driver” averages real calls with journal reads.
• Spans attach to futures, never to threads. Span::enter returns a guard bound to the current thread; held across an .await it stays entered while the future is suspended, so whatever runs next is attributed to it. With concurrent siblings that silently reparents their work. Instrument is the only form that survives a suspension, and tests/guards/layering.rs bans the guard in async code.
• The vocabulary lives in runtime::telemetry, never typed inline at a call site.

Every failure that must not pass silently has its own event target:

That list is telemetry::LOUD_EVENTS, and this table is checked against it: tests/guards/docs.rs fails the build if the runtime promises an event this page does not name, tests/guards/layering.rs if any of them has no emitter, and tests/process/telemetry.rs asserts on what a subscriber actually received.

The last mile: instrumented is not monitored

No exporter ships, so a deployment closes three gaps by hand; cargo run --example observability is a runnable bridge that does all three and asserts on the result:

1. Latency must exclude replay. A replayed effect opens no span at all; it emits a debug event on the agentplane.effect target with replayed = true, plus the agentplane.effects.replayed counter. A histogram built from spans is clean; one keyed on the target sees both.
2. Scheduling the sweep is scheduling your gauges. Gauges come from a census the sweep takes, so a plane with no sweep loop has counters and no gauges, and nothing says so.
3. SweepReport::needs_attention() is the alert predicate. It already folds in breaches, expiries, dead letters, saturation, failed recoveries, lost evidence and an unreadable census; do not re-derive it from counters.

The example’s module docs carry the OTLP wiring verbatim — pinned crate versions, the tracing-opentelemetry layer, and where each of the three rules goes — so the difference between it and a production subscriber is the exporter and nothing else.

Then schedule the two loops, or let the binary do it:

agentplane serve agent.yaml \
  --sweep-every 30      `# gauges, deadlines, task expiry, dead letters` \
  --drill-every 86400   `# the recovery rehearsal` \
  --drain-secs 25       `# how long a SIGTERM may keep working`

The operator surface

Feature http, off by default.

There are two surfaces. The HTTP API acts on a running plane; each CLI verb opens a store. redb admits one writer process, so on the embedded backend a CLI verb runs while the plane is down, and the API while it is up.

So every remedy agentplane attention names is a verb the CLI has, and every remedy GET /attention names is a route the API has — or, where the API has none (resuming a run, re-running a drill), the agentplane verb that does it. A remedy with nothing to run says no verb: and why. Each condition also lists its subjects — the run, task, <case>/<obligation>, message or <run>/<registration> the remedy’s verb takes, at most ten, newest first for run conclusions and most overdue first for waits and tasks, with unlisted counting the rest — so the next command can be typed from the answer alone:

agentplane attention --store ./journal.redb
# { "condition": "run.quarantined", "found": 1, "subjects": ["run_01JD…"],
#   "unlisted": 0, "remedy": "establish an undecided effect with `reconcile`, …" }

A failed run is listed only when it stands on landed work nothing undid — run.failed_with_landed_work, whose remedy is replay to finish the work or cancel to unwind it. Any other failed run is one a resume may clear, and is not a condition.

reconcile establishes an effect the runtime could not decide; quarantine reopens or abandons the run afterwards; tasks lists the worklist and decide <task> approve|reject|answer answers one; acknowledge accounts for a breached obligation; cancel stops a run the ceiling or the halt will not release; rearm revives a parked push registration. Each but rearm requires --actor, recorded as asserted (the same act through the API records authenticated), and --reason.

Two of them cannot finish the job from a terminal, and say so. quarantine records the decision and reports "applied": false: a terminal holds the journal and not the agent, so the next resume applies it. decide records the decision durably, completes the task and reports what the delivery did — Buffered, from a terminal that holds no agent — and prints the agentplane replay <run> --manifest … --store … that continues the run. cancel from a terminal records the stop and leaves it to the plane that holds the agent, which observes it on its next resume.

Only a plane that can drive a run judges it. A resume first asks whether this plane provides every capability the run’s plans name, and a plane that does not — a terminal holds no agent — stops there: NoProvider, reported as Buffered by a delivery and as a recorded, undriven request by a cancel. A plane that does hold the agent under a different policy bundle or an edited declaration quarantines the run; a rollout across a mixed fleet has to plan for that.

Serving it

Embedders wire Api into their own process; the agentplane binary’s serve verb does the same wiring from flags, hosting one manifest or one room file per process.

A listener per audience. The peer surface binds to --addr; the operator surface this section documents is opt-in beside it on --operator-addr, and the MCP surface agent frameworks call on --mcp-addr, so a network policy can treat each differently.

The repeatable flags take no environment variable; everything else in the table does. The witness flags are under the audit.

Identity comes from the request, never from its body

Four-eyes is enforced in TaskStore::claim, which takes an actor and a set of roles. Over HTTP a reviewer who could name themselves could name the person who proposed the action, so the wire type has no field to hold it:

pub struct DecisionRequest {   // no `actor`. no `roles`.
    pub approved: bool,
    pub reason: String,
    pub amendment: Value,
    pub digest: Option<Digest>,   // the task version decided on
}

The handler builds the Decision from the authenticated Caller. What lands on the record is an Operator whose basis is authenticated; the same approval given at a terminal records asserted.

The body is deny_unknown_fields, so one carrying "actor": "alice" is a 422 rather than silently ignored.

amendment is not a note. On an approved call task it is the call: the runtime dispatches the reviewer’s arguments in place of the model’s, labelled as the reviewer’s — trusted, provenance task:agent.approve_call, the original arguments’ sensitivity — and every sink gate judges them like any other value. An amendment that does not fit the tool’s declared argument schema dispatches nothing. On a rejection it stays what it reads as: recorded advice (“no, and here is what would pass”).

Two gates, and the surface will not start without the second

Authentication says who; it does not say what they may do. An operator surface that stops there hands every authenticated caller the whole plane. So every route runs gate(), which authenticates, resolves the caller’s tenant to a plane, and then authorizes through that plane’s PolicyEngine under an api: action — and Api::new returns an error if any registered plane has none, or if none was registered at all. DenyAll exists for wiring the surface up before the rules are written.

Checking a driver against the real thing

just test-live exercises the OpenAI, Gemini and compatible-wire drivers plus the embedding wire against the actual APIs, loading keys from .env. Each battery skips on its own credential, so one key runs one battery and the rest say so. It is gated twice — AGENTPLANE_LIVE=1 and the key — and is never part of just ci, so an exported OPENAI_API_KEY bills nobody for the test suite.

The compatible-wire battery runs on HF_TOKEN against Hugging Face’s router, or on CHAT_COMPLETIONS_BASE_URL pointed at a local engine.

Putting a tenant on your telemetry

Off by default: a tenant name is frequently a customer name, and an observability backend is usually the least protected system in a deployment.

.tenant_label(TenantLabel::Name)

It is the tenant field of every metric event and agentplane.tenant on the run span (every other span is inside that run’s trace). Under the default the attribute is absent rather than blank. The label is this plane’s tenant, so cardinality is the number of planes you wired, never something a request can grow.

There is no pseudonymous option: the tenant also appears in store keys, blob paths, the policy request, the checkpoint origin published to witnesses, and the tenant field on an Agent Card served unauthenticated. If customer names must not leak, do not put them in the tenant id — TenantId::new("t-9f3a") covers every one of those places at once. A tenant id is at most 64 characters and holds no /, :, control character, whitespace or +, because it becomes part of composite keys and of the checkpoint origin.

Per-tenant ceilings

Budgets bound one run. They do not bound a tenant: a caller that can start runs can start a thousand, each perfectly within its own ceiling, and the compute and the model bill are somebody else’s problem. RuntimeBuilder::quota sets a tenant’s limits and points at the store that accounts them. A plane built with Runtime::builder_on or builder_with already has the backend’s quota store with no ceilings, so halts and rate ceilings hold without this call; .quota states the limits.

.quota(store.clone() as Arc<dyn QuotaStore>, TenantQuota {
    max_concurrent_runs: Some(50),
    max_tokens_per_period: Some(20_000_000),
    period: Period::Monthly,
    ..Default::default()
})

The accounting is durable, so a ceiling holds across instances. The reservation is one transaction that counts and inserts, so two instances racing for the last slot serialise and one loses.

Concurrency bounds runs executing. A slot is taken at admission and given back when the instance finishes with the run — including when it suspends. A resume is not gated: that work was admitted already.

Spend bounds a period by reserving each run’s worst case at admission. A run can spend its own ceiling and end one call past it per step it has in flight — a call’s cost is known only when it returns — so admission holds ceiling + width × one call against the period, in the same store transaction that checks the ceiling and takes the slot, and refuses when settled spend plus every outstanding hold plus this one would pass the ceiling. The width is max_parallel_steps, or the admitted plan’s node count, and the run is dispatched no wider. What one call can cost comes from the declaration: each model role’s max_input_tokens plus its output ceiling, at its price (see the manifest reference); a Rust deployment states it on its Budget as max_call_tokens / max_call_minor_units. agentplane validate prints the figure per agent.

Under a spend ceiling, a run whose budget leaves that unit unbounded — no run ceiling, or no per-call bound — is refused with QuotaError::Unbounded naming the field. Admitted, it would hold nothing against a period it can spend without limit.

Every pass settlement moves that pass’s spend out of the run’s hold and into the settled total, and the pass that concludes the run releases the rest. A suspended run keeps what it holds, and a resume is never refused for spend. One live execution pass belongs to the period in which it starts: settlement charges it there even when midnight or month-end passes before it finishes. A resume in a later period moves what the run still holds into that period — released from the old one, held in the new one, unconditionally.

What this bounds. A period’s settled total stays within its ceiling through the work admitted in it, however many runs suspend, run concurrently, or admit from several instances at once. What still reaches past it: holds carried in by resumes from an earlier period (new admissions are refused until they settle); a model call that reports more input than its role declares, which fails but is billed as reported; an embedder’s own effects, bounded only as exactly as the per-call figure it declares for them; and compensating calls, which no ceiling gates. A commissioned run reserves and settles its own spend; the commissioning run’s own ceiling still counts it, and the period is charged once.

A QuotaError::SpentOut refusal names settled and reserved spend apart: settled spend resets with the period, reserved spend is held by open runs. Runtime::reservations(limit) lists the holders, and agentplane attention names the ones stopped waiting on a person — quarantined, exhausted, withheld or failed — since what they hold comes back only when somebody concludes them. A halt does not release a hold: the workload scopes stop admission, and only a subject: halt reaches work already running.

Wiring a quota store records every active run even when all ceilings are None, so running() is truthful and adding a limit starts from the work actually in flight.

Settlement is crash-safe rather than best-effort. Before a pass can dispatch an effect, QuotaPassStarted records its period and whether it owns the admission slot. Admission writes it with the run’s first records; a resume writes it with the first record the resume itself appends, so a resume that appends nothing — one reaching the conclusion the run already holds — leaves no marker, and a marker is never a run’s last record. At the end, QuotaStore::settle writes an exact (run, epoch) receipt, accrues that pass’s spend, reduces the run’s hold by it (releasing the rest when the pass concludes the run), and releases its slot in one store transaction. A lost acknowledgement repeats the same receipt and charges nothing twice; a changed retry is corruption. The run is physically sealed and its lease is released only after settlement succeeds. If settlement is unavailable, the lease expires still owned and the ordinary abandonment sweep derives the spend from the journal and retries it.

The journal and quota store may be separate systems; this is not one distributed transaction. It converges after a transient outage on either side, and cannot make progress through a permanent partition or survive the loss of one backend.

A plane with no quota store resumes a run only when none of its passes accrued spend to a period, and refuses one that did, since that spend would go unbilled. It cannot give back the concurrency slot of an admission whose process died mid-pass; the sweep of a plane with the quota store releases it once the run is sealed, and counts it in SweepReport::slots_released.

RuntimeBuilder starts at Budget::unlimited(), so a Rust deployment that sets a tenant spend ceiling has to give its runs a ceiling and a per-call bound too — otherwise every run is refused as unbounded rather than admitted past the ceiling. A manifest without budgets already fails validation.

A refusal is RuntimeError::QuotaExceeded, not a policy denial: a ceiling means not right now. Over A2A it comes back as -32029 with the ErrorInfo reason QUOTA_EXHAUSTED, so a peer backs off.

An unreachable quota store refuses rather than admits. Concurrency is tracked as a set of runs, not a counter, so a process that dies mid-run strands a slot an operator can name: QuotaStore::running_runs(limit).

let held = quotas.running_runs(100).await?;
let stranded = journal.abandoned_runs(100).await?;
// A slot whose run has no live lease is the one to look at. The recovery sweep
// resumes it and settlement gives the slot back.

The listing empties as runs settle.

A tool’s rate ceiling

A grant’s rate_limit is counted in the same quota store, per tenant per tool, so it needs one wired: a plane built from the journal alone (Runtime::builder) whose declarations state a ceiling and that was given no quota store is refused at build. Each dispatch of a ceilinged tool costs one store transaction; an unceilinged tool pays nothing. An unreachable store refuses the call, as it refuses admission.

A run stopped by one concludes exhausted, and agentplane attention lists it with the other exhausted runs. The remedy is time, not a raise: wait for the window to pass, then agentplane replay the run. A replay inside a window that is still full stops it again on the refusal already recorded; after the window it records a re-admission beside the refusal and carries on. Changing a ceiling means publishing a declaration; a new revision does not reset the count.

The reservations are operational state, like delivery cursors: they are not in the export, and they age out with the widest window stated for their tool.

The emergency stop

Runtime::set_halt(&scope, &by, at, reason) stops new work from starting, across every instance, because the flag lives in the quota store rather than in the process. The refusal is its own error carrying the operator’s reason, never a ceiling, which would invite a retry.

It names what it stops

The first three ask what is running; the last asks who it runs for. That is read from the chain the run was admitted under — on a served plane, the caller’s, not the operator’s — and matches any link on it: the person at the root, each workload it was delegated through, and the one acting — so withdrawing alice stops what a service is doing on her behalf too, and leaves everyone else’s alone.

It is the only scope that reaches work already running, and it pauses. A run under a withdrawn credential stops at its next step boundary as withheld — or sooner, at its next call to a peer that is told who asked, which reads the halts before presenting a credential. Throwing or lifting a subject halt drops the cached credentials held for that subject. The run’s completed work stands, and after the lift a replay continues it; it is not unwound. To reverse it, cancel it. A step paused inside an open effect group leaves the group open; a cancel re-runs that step to abandon the group, starts no new work, then unwinds.

Both halves are journaled: AuthorityWithheld at the pause and AuthorityRestored at the lift, the second beside the first. A reader takes the last word.

agentplane halt  --store ./journal.redb --scope 'agent:payments-clerk' \
                 --reason "incident 42: looping" --actor ops-carol
agentplane halt list --store ./journal.redb      # what is stopped right now
agentplane halt  --store ./journal.redb --scope 'agent:payments-clerk' --lift \
                 --actor ops-dave
agentplane halt list --store ./journal.redb --lifted   # who lifted what, newest first

--scope takes tenant, agent:<metadata.name>, revision:<manifest digest> or subject:<principal on the chain> — which stops every run with that principal anywhere on its delegation chain: the person at the root, a workload they delegated through, or the one acting. Those are the forms HaltScope::parse accepts, and the refusal you get for a typo lists them.

--actor is required to throw one, and the row says it was asserted. Through the operator API the name comes from the verified credential and is recorded as authenticated; at a terminal it proves only that whoever ran the command could open the store.

A lift is recorded, and --actor is required for it too. Before the row goes, the lift is written to the journal as the one HaltLifted record of a sealed run of its own, outcome halt-lifted: who lifted it and on what basis, when, and the whole stop it ended — scope, reason, who threw it and when. A lift that cannot be recorded does not happen. The record attests the instruction and the row as read, not that the removal succeeded; halt list is the answer to what is in force. Only the halt the record names is removed: one re-thrown between the read and the removal stays standing, and the lift fails naming the record’s run, as it does when the removal itself fails (a 500 over the API). The answer’s removed is false when another lift removed the row first. halt list --lifted (a page of --limit, default 100) and GET /halts?state=lifted read the lifts back, newest first, and the runs travel in an export like any other sealed run. Where the stop reached a run — the subject: scope, the one that does — the run’s own journal also holds both halves, with the operator from the halt on AuthorityWithheld.

Those commands open the store, and redb admits one writer process — so against the file an agentplane serve is holding they fail, saying so. Two ways round it:

# 1. The operator API reaches a running plane, whichever backend is under it.
curl -sX POST "$PLANE/halts" -H "authorization: Bearer $TOKEN" \
     -d '{"scope":"agent:payments-clerk","reason":"incident 42: looping"}'
curl -s "$PLANE/halts"  -H "authorization: Bearer $TOKEN"
curl -sX POST "$PLANE/halts/lift" -H "authorization: Bearer $TOKEN" \
     -d '{"scope":"agent:payments-clerk"}'

# 2. On the shared store, the CLI and a serving plane coexist.
agentplane halt --store "$DATABASE_URL" --tenant acme \
                --scope 'agent:payments-clerk' --reason "incident 42" --actor ops-carol

Throwing the stop and lifting it are separate capabilities — api:halt.place and api:halt.lift. A lift over the API is recorded under the authenticated caller, and its answer names the lifter and the run holding the record; a lift at the terminal is recorded under --actor, as asserted.

A halt closes the door; it does not empty the room. Under a workload scope runs already executing carry on. To reach work in flight, cancel it — GET /runs/live is the listing that tells you which, with the agent, the revision and the delegation subject beside each id. A slot marked stranded belongs to the recovery sweep, which resumes it; cancelling one unwinds work that was about to finish.

Withdrawing a credential:

# Nothing new starts for alice, and what is running for her pauses.
curl -sX POST "$PLANE/halts" -H "authorization: Bearer $TOKEN" \
     -d '{"scope":"subject:alice","reason":"credential withdrawn: laptop lost"}'

# What is paused or still finishing under it — cancel what you want unwound.
curl -s "$PLANE/runs/live?subject=alice" -H "authorization: Bearer $TOKEN"

Scopes are independent rows. Halting the whole tenant while an agent is halted, and then lifting the agent’s, leaves the tenant’s standing. Where several halts cover one run the narrowest match is the reason reported.

An ungoverned run — a skill registered directly on the plane, with no manifest — is stopped only by a tenant halt.

A peer over A2A receives -32030 with the ErrorInfo reason HALTED and a fixed message that carries none of the operator’s words. A batch pass returns the halt as its error and leaves the items it stopped pending; the next pass admits them again.

--reason is required to halt and refused to lift. Who is asked of both.

What a workload-scoped halt does not stop

Runs already executing, and suspended runs resuming. Work in flight is stopped by cancelling it (request_cancel), which unwinds what it did and records who asked. cargo run --example operator_stop runs both brakes side by side.

The subject: scope is the exception: it does reach runs already executing, which pause as withheld. The response to POST /halts says which of the two you got, derived from the scope — a credential withdrawal preserves work that a cancel would unwind.

One surface, many tenants

Api::new takes Planes, a registry keyed by tenant, so one process can serve several — a single-tenant deployment passes its one runtime. Which plane answers comes from the caller’s tenant, which the Authenticator derives from the credential exactly as it derives actor and roles.

The gate hands each route its resolved plane, and Api holds no runtime of its own, so a handler cannot read a store without having established whose it is. A caller whose tenant has no plane is refused rather than served by a default.

The gate runs before the path is parsed, so a denied caller cannot learn whether a run id exists by comparing a 400 against a 404.

What the endpoints are for

Each route’s shape — its parameters, body, every status it can answer and the body of each, and the api: action it asks the policy — is in the OpenAPI 3.1 document, which agentplane openapi also prints (a build with the http feature). It describes the operator API only: the A2A surface is described by its Agent Card and protocol, the MCP server by its tool listing. No operation takes a tenant; it comes from the credential. Every refusal is {"error": "<sentence>"}, including a body the API could not read. Every build serves every operation the document lists; a plane built without the store or feature an operation needs answers it 501 after the usual gate. The document is outside the compatibility promise: generate a client per version.

A standard-library Python client generated from it is kept in the repository at clients/python/agentplane_operator.py: one method per operation, the parsed answer returned, OperatorError (status and sentence) raised for every refusal and for every redirect, which it never follows: a followed redirect turns a POST into a GET and carries the token to whatever host it names. Copy the file; it calls operator routes and runs no agent code.

POST /events speaks CloudEvents, and its own shape. A bus posts a CloudEvents 1.0 message in either content mode — structured (Content-Type: application/cloudevents+json) or binary (ce-specversion, ce-id, ce-source, ce-type headers with the data as the body) — and anything else is read as this plane’s own {id, kind, correlation, payload} — with non-empty id and kind, for the same reasons the CloudEvents shape requires its three. The type is the event kind the policy gate is asked about, either way. An envelope the plane has not understood is a 400, never a guess: an unknown specversion, a missing id/source/type, a control character in any of the three, data_base64, a repeated ce- header, or an extension name that is not lowercase alphanumeric or that shadows a core attribute. A store outage is a 503, so a bus retries what a 4xx would make it drop.

The source a run is woken under is the authenticated caller, never the one in the body, so a caller cannot deduplicate against another party’s messages. The producer’s own source rides inside the buffered event’s id, so a gateway relaying two counterparties that both number their messages from one still delivers two events.

The agentplane. kind namespace is the plane’s own. A human task’s answer reaches its run as an event of such a kind, so any kind in it is a 403 here, and an A2A message addressed to a run waiting on a task is refused the same way. A task is decided with POST /tasks/{task}/decide, where the claim, eligibility and four-eyes run.

One CloudEvents attribute correlates: subject, the standard’s own “what this event is about”, becomes the key ("subject", value) — a run that expects to be woken by a bus correlates on the subject its counterparty will name. Extensions deliberately do not; a producer that must correlate on richer keys posts the native shape, where they are stated.

A suspended run says what it is waiting for. GET /runs/{run} reports the SuspendReason from the run’s last record — whether to approve something, chase a counterparty, or page somebody.

The worklist says when it was cut off. The response is an object carrying truncated, taken by asking the store for one more than the page.

Each worklist item says whether this caller may decide it. A reviewer barred by four-eyes still sees the task and is told on the item. The flag calls Task::may_decide, the same predicate the store enforces.

Each item carries the text a person should read, beside the value a program reads. justification is the structured value; rendering is the same task with every invisible or direction-changing code point escaped in place (\u{202E}), escaped saying whether any was, and every word that mixes scripts listed in mixed_script with where it occurs. agentplane tasks prints the same rendering. A proposal the plane cannot open carries withheld — sealed (no key ring here), erased or undecodable — and then neither justification nor rendering carries a proposal or evidence at all, never the envelope. Approving it is refused with 422 and the reason, before anything is claimed or recorded; rejecting it still records → what a reviewer is shown.

Deciding on a version

Every task view — GET /tasks, GET /tasks/{task}, the claim answers and agentplane tasks — carries digest, the version of the stored row (not of the served justification, which withholds what the plane cannot open). A decision may name it: "digest" in the decide body, --digest on agentplane decide. A row that changed since is refused before anything is claimed or recorded — 412, or exit 1 naming the tasks --show that re-reads it — and if it changes between the read and the claim, the claim taken is released. Naming no digest compares nothing.

No authenticator is shipped

Authenticator is handed the whole header map, so a deployment may authenticate by bearer token, mutual TLS, or a signed header from a gateway. Yours is held to testkit::conformance_auth, where the rule easiest to get wrong is that Missing and Rejected are different answers → testing.

Claiming is what stops duplicated work

claim reserves a task so two reviewers do not read the same case in parallel; release gives it back. Claiming is not advisory. TaskStore::claim runs four-eyes and role eligibility in the same transaction that reserves, so an ineligible reviewer is refused before they read the case rather than after they have made up their mind.

Eligibility outranks availability

A refused claim is a 403 (this will never be yours) or a 409 (try again, or ask the holder), and the permanent refusal is checked first. The order is part of the TaskStore contract, and the conformance battery holds both backends to it:

NotFound → Excluded → WrongRole → NotPending → AlreadyClaimed

Every verb that can refuse follows the same classification: an id that names nothing is a 404 wherever it appears — claim, decide, cancel — and a store outage is a 500, never a 409. A release by somebody who does not hold the task reports ClaimError::NotHeld, not success and not NotFound; the battery holds both backends to it.

Escalation widens the audience — and then leaves the expiry scan

When a task’s window closes unanswered, the sweep applies the policy the task was opened with. deny and proceed record a decision and resume the run. escalate does three things in one store transaction: the declared escalate_to roles join the audience (a union — the original reviewers remain eligible), the stale claim is cleared so the widened audience can actually take the row, and the state moves to escalated. Four-eyes survives it: whoever proposed the action stays barred, however wide the audience gets. The escalated row stays in the ordinary queue, ranked and claimable — that queue, filtered by the caller’s roles, is where the wider audience meets it.

An escalated task does not reappear in the overdue scan, although it is still pending and past due; otherwise escalated rows would fill the scan’s bounded batch and the expiry policies of every task behind them would stop firing.

A task id mixes in the run it belongs to

An EffectKey is unique within a run, while the worklist is a table every run shares and TaskStore::open is idempotent by id. So TaskId::derive hashes the run in as well, and the field is private. The ("task", …) correlation key inherits that, being derived from the id.

🗄️ Retention and erasure

A full-fidelity journal is simultaneously an asset and a GDPR liability. The window is yours to choose — a retention period is a legal decision and a crate that picked one would be choosing somebody else’s — but running it is not: Runtime::retain walks it, agentplane retention plan lists what a walk would erase, and erasure is the full account.

What exists:

• The journal keeps everything, indefinitely. It is append-only and hash-chained: nothing in it can be edited or removed without invalidating every record after it. That is the property Article 12 wants and precisely the property an erasure request does not.

• A record over 1 MiB is refused rather than written, so bulk content is pushed out of the chain by construction. Bytes that big belong in the content-addressed blob store, with only the digest journaled — see the cookbook. Deleting a blob is then a filesystem or bucket operation, and the chain still verifies afterwards because it only ever committed to the digest.

• Blob bytes can be erased, and the erasure is recorded. BlobStore::expire drops the content and leaves a tombstone. A reader afterwards gets Expired with the date and the reason — not NotFound. Expiring twice keeps the first tombstone.

There is no TTL here. Object-store lifecycle rules (S3, GCS, Azure) expire objects without your process — but a lifecycle rule deletes, it does not tombstone, so a blob removed that way reads as NotFound. Use lifecycle rules for bulk age-based expiry, and the erasure verbs for erasure requests, where when and why must stay answerable.

The erasure unit is the case. A run bound to a case writes bytes through cx.store_blob, which records the link to the case; cx.blobs() is read-only for such a run. Answer a request with one call:

let erased = agentplane::blob::erase_case(
    Some(blobs.as_ref()), cases.as_ref(), Some(journal.as_ref()),
    Some(keys.as_ref()), Some(disclosures.as_ref()),
    &tenant, case, now, "art-17 request",
).await?;
// erased.blobs: references tombstoned; erased.copies: disclosed copies it did not reach

The key-ring argument exists with the keyring feature. On a sealed plane, passing the ring destroys the case’s wrapping key after every tombstone is written, so each sealed copy becomes unreadable at once; pass None on a plane that stores blobs unsealed, and the call is plain tombstoning.

Nothing is destroyed for a case under a legal hold, a case still open, or a case with a run that has not concluded under an outcome nothing resumes from — EraseError::UnderLegalHold, CaseStillOpen, RunNotConcluded. Without a journal, a case with runs is refused. The erasure is recorded on the case first, and an erased case refuses reopening, writes and holds (StoreError::CaseErased, 409 on the API). A retry of an erasure that did not complete runs again under the recorded instant and reason.

Every blob that case produced is tombstoned with the same reason. Other cases are untouched — including ones that stored identical bytes: the link scopes the erasure, not the content. blob::erase_run erases a case-less run’s sealed payloads and refuses a case-bound run or one that may still resume.

On a window, rather than one case at a time. The same act on a clock is Runtime::retain(older_than, at, reason) — or retention::retain(stores, journal, older_than, at, reason) outside a runtime — and its plan:

agentplane retention plan --store ./journal.redb --older-than-days 2555

The CLI form lists; the pass itself is the library call. It erases closed cases only, measures the window from opened_at, skips a case already erased, and returns not_erasable beside its count — read that list, it is the coverage statement.

What still cannot be erased is anything written into a journal record. The chain is append-only by design, which is the point — keep personal data out of records rather than expecting erasure to reach it. Two ways to do that, and both are enforcement points rather than advice: write bytes through cx.store_blob so the chain commits to a digest, and declare a ceiling on what may be written down at all — spec.security.max_sensitivity_journaled in a manifest, or RuntimeBuilder::max_sensitivity_journaled on a plane of hand-written skills. The 1 MiB record refusal pushes bulk content out by construction, but a name, an address and an IBAN are a few hundred bytes and fit comfortably. Erasure has the full table of what lands where; regulation says the same in the obligations’ own terms.

🧯 Disaster recovery

Two numbers and a rehearsal. The numbers are properties of your backup schedule; what this page can state is where each one comes from and what the runtime contributes to it.

RPO — how much history a failure can cost. An effect’s intent is durable before it is dispatched, so the plane itself loses nothing it acknowledged. The exposure is entirely the gap between your last durable copy and the failure, which makes RPO a property of whichever of these you rely on:

An export is not a substitute for the first two. It carries the journal and the case layer and nothing else, by design — see what an operator re-establishes, below.

The export row names its backend because the condition is real: redb admits one writer process, so agentplane export cannot open a file agentplane serve is holding. On the embedded store the scheduled copy is a file snapshot and the export is what you take when the plane is stopped. On postgres the export runs beside the serving plane, so the interval is one you can actually schedule — agentplane export --store "$DATABASE_URL" --tenant acme.

RTO — how long until the plane serves again. Four terms, and only the last grows with how much work was in flight:

1. Stand the store back up (restore, replica promotion, or agentplane restore).

2. Verify it. agentplane verify checks an export offline against its own checkpoint; agentplane restore reports whether the rebuilt store commits to the same root at the same size, and exits non-zero when it does not.

3. Start the plane. Nothing is replayed at startup; a run is replayed when it is resumed.

4. Re-arm the suspended runs. Resuming each one is what repairs its waits: replay reaches the announced wait, finds no terminal record, and re-arms the timer, re-subscribes and re-opens the task row. Which runs those are is a query rather than something to have kept:

   agentplane waiting --store ./restored.redb      # soonest due first

   Or GET /runs/waiting on a serving plane, under api:run.waiting. Both answer from the journal — an index the write path maintains, one row per currently waiting run — not from the timer and subscription tables a restore is missing. A run leaves the listing by being resumed.

The drill

agentplane export  --store ./journal.redb > plane.jsonl
agentplane verify  plane.jsonl --checkpoint ./checkpoint.json
agentplane restore plane.jsonl --store ./restored.redb
agentplane drill   --store ./restored.redb      # every case's references, live

verify given no anchor reports deletion — not checked, and means it: a rebuild against the file’s own header proves self-consistency, which is exactly what an editor who dropped a run and rewrote the header achieves. Keep a checkpoint somewhere the plane cannot reach — an earlier audit’s output, or a witness.

restore writes into the store it is pointed at, which is normally a different tenant of a database that survived. That relabels the log, and the report says so in not_carried — the verdict is the commitment, not the name.

What an operator re-establishes

An export carries the journal and the case layer. Everything below lives in stores it does not travel with, and is listed so that none of it is discovered during an incident:

agentplane drill checks the last two rows: it walks every case, holds each reference against the live stores, and reports unchecked for a store it was not given.

The verdict stays on the plane. Every drill writes one row, which the next drill replaces:

agentplane drill --last --store "$DATABASE_URL" --tenant acme
# {"drilled":true,"at":"2026-09-20T04:11:02Z","sound":true,
#  "cases":812,"findings":0,"not_checked":1,
#  "origin":"acme-prod","log_size":91_204}

GET /drill serves the same answer under api:drill.read — its own capability, so a read-only auditor credential need not carry the operational roll-up as well. Three things to read off it:

• drilled: false is a finding, not a clean answer; both verbs exit non-zero on it.
• origin says which store it ran against, from the plane’s own checkpoint — a drill over a restored copy says nothing about production.
• not_checked sits beside sound, not inside it — the example above was given no key ring.

agentplane attention reports a failed rehearsal as drill.failed, and keeps reporting it until a later drill passes.

A message delivered to a restored plane before its run is resumed buffers, and a buffered message nobody claims dead-letters. So the order is: restore, resume everything awaiting names, then open the gates.

Evidence

postgres_restores_a_plane_that_then_serves runs the whole sequence against a real PostgreSQL server, restoring into a tenant of a database another tenant is already using, and asserts each claim above: equal roots at equal size, records hash-for-hash, the matter and its obligation and its artifact, isolation in both directions, a lease past the journal’s highest epoch, a wait repaired by a resume, new work whose seal extends the restored log, and a drill of the restored case layer without a blob store reporting it unchecked.

🚑 Runbook