A handle is the evidence of detachment
Amended by leases-and-budgets: the death-clock’s creation-age ceiling becomes an idle-observation lease over a universal worker registry, and “detached workers are unmanaged by design” is retired in favour of unmanaged by default. The root/foreground split, the handle model,
await/raceunification,forget’s deletion, and the shared reaper all stand. session-ledger further restates the&/job-table separation at the listing layer: REPLjobsfolds over stopped groups and detached handles alike, whilefg/bgstay pgid-typed and the mechanisms remain distinct.
Given a turn evaluator with a durable root cancel scope distinct from the
swappable foreground scope, a worker’s parent follows the primitive that
creates the worker, and a handle is the public evidence that the worker is
detached. spawn and REPL-only watch reify a Value::Handle and hang their
workers at the root. Pipelines create no handle and remain foreground work.
par is the compromise: it returns no handle and joins within the expression on
the success path, but it is prelude code over spawn, so its fan-out is
root-parented rather than nurseried; an early exit can leave the unjoined tail at
the root.
Context
This rests on unify-turn-evaluation:
the turn evaluator has a durable root CancelScope on Shell
(shell.session.root) that no turn swaps, distinct from the swappable
foreground scope (shell.turn.cancel). The 30 s wall and host-routed user
interrupt cancel the foreground scope; the root survives. Plain
process-global SIGTERM/SIGHUP and batch/Windows SIGINT still trip
SIGNAL_COUNT in every polling thread; those are outside the foreground-scope
path this ADR relies on. The root split fixes the collateral kill — a
background worker dying because the foreground turn that launched it timed out —
by parenting detached workers at the root. The open question this ADR answers:
given the split, how are the primitives recast?
The source framing still needs this distinction. The module header of
core/src/builtins/concurrency.rs:1 labels the whole set “structured
concurrency primitives” — yet spawn and watch produce handles that outlive
the turn and are observed on later turns. Shell::spawn_thread
(core/src/types/shell/inherit.rs, ~line 195) takes a child of
shell.session.root, and its sole live caller is spawn_child
(concurrency.rs:88) — the path spawn and watch share. That makes the
scope edge explicit for handle-producing thread workers: the public handle marks
detachment, and the worker’s cancel scope hangs at the root.
A grounding correction the code forced: the
pipeline does not flow through spawn_thread. Its stages are OS processes
in a PipelineGroup, and current launch code passes the active foreground
scope straight through (core/src/runtime/pipeline/launch.rs:141) rather than
minting a pipeline child scope. Making that edge explicit is cutover work from
the sibling ADR, not a current fact. par is prelude code over spawn
(core/src/prelude.ral:148), so its workers are spawn workers — detached by
parenting, joined by sequential await on the success path. A foreground
deadline, a scope-routed interrupt, or a failed await cuts that join short and
leaves the unjoined tail root-parented. The detached/structured line is
therefore drawn at the thread-spawn primitive (spawn/REPL watch, the
only spawn_thread users) and at join discipline (does the launching
expression await every handle before returning?), not at a single uniform
scope choice.
Decision
The organising principle and the per-primitive recast:
- The handle is the evidence of detachment, not the whole story. A primitive
that returns a
Value::Handleparents its worker at the root. A primitive that creates no handle and owns its children directly stays in the foreground.paris explicitly the exception left as library code overspawn: structured result, detached workers.
| primitive | reifies a handle? | parent scope | lifetime |
|---|---|---|---|
spawn | yes | root | completion · cancel · exarch one hour ceiling if armed · root abort / session exit |
watch | yes, REPL frame only | root | completion · cancel · root abort / session exit |
pipeline | | no (OS process group) | foreground; pipeline child scope is cutover work | joined before the turn returns |
par | no (joins its spawn handles) | root (its workers are spawn workers) | awaited in input order on success; foreground cancel or first failed await can orphan the unjoined tail |
await | — | runs in the foreground | wall-bounded wait |
poll | — | — | total, non-blocking |
race | — | runs in the foreground | wall-bounded wait; loser cancellation is explicit handle cancellation |
cancel | — | — | the explicit handle reaper |
forget | — | — | recommend deletion |
-
spawn— parent at the root (detached). It acquires the host’s detached worker policy: exarch arms a one hour lifetime ceiling; the REPL does not. Cross-turn persistence becomes defined behaviour, not the happy-path accident the sibling ADR documents for the REPL. -
watch— root-detached, but REPL-only. A watched worker writes to a live host output sink as it runs. In the REPL that sink is durable (rustyline’s external printer) and can outlive the turn. In exarch the active stdout/stderr sinks are per-call capture buffers; a root-surviving watcher would keep writing into a stale buffer afterrun_shellhad already taken it. The exarch frame therefore does not admitwatch. Detachment may hold only root-owned or handle-owned resources, never a foreground frame’s capture state. -
pipeline
|— bounded by the foreground. Semantics unchanged, but the implementation cutover must make the scope edge explicit: stages are processes in one group under a foreground child scope owned by the pipeline, reaped by group-terminate when the foreground is cancelled (deadline / foreground interrupt). This is the cancel→drain→collect path the sibling ADR names;RunningChild::observe’s typestate already enforces wait-before-drain. -
par— keep the prelude compromise.parspawns each item withspawn { … }andawaits the handles in input order before returning (prelude.ral:148). The workers therefore hang at the root, not the foreground. On the all-success path the expression is structured even though the workers are detached. Under a foreground deadline or scope-routed interrupt, aparin flight unwinds like any blockedawait, and its root-parented fan-out survives. If an awaited worker fails,awaitre-raises and the same early-exit rule leaves later handles unjoined. In exarch the one hour lifetime ceiling reaps those orphans. In the REPL there is no ceiling, so a REPL foreground Ctrl-C can leave them until explicitcancel, root abort (Ctrl-\), or session exit. That orphan window is accepted to keepparordinary prelude code rather than adding a foreground-parented nursery spawn path. -
await— becomes cancellation-aware, sharing one wait loop withrace. Currentawait_handle(concurrency.rs:284) blocks on a barerx.recv(). That receive does not pollprocess::check, so a root-scoped worker that outlives the foreground can keepawaitasleep past the wall.race(concurrency.rs:354) already does the right thing:loop { try_settle; process::check?; sleep(1ms) }. Unify them.awaitblocks in the foreground (wall-bounded); if the wait is cut short by the wall or a foreground-scope interrupt it unwinds viaprocess::check, but the root-scoped worker survives to be observed on a later turn. This dissolves both the bare-recv()hang and the collateral kill. -
poll— unchanged, promoted to the primary cross-turn observation. It is already total, non-blocking, and wall-friendly (handle-settle). The canonical idiom for a long-running agent:spawn→pollacross turns →awaitonce settled (which then returns instantly off the cachedCompletedHandle). -
race— mechanism unchanged. It is the cancel-aware loopawaitadopts; it cancels the losers with explicit handle cancellation and projects the winner. -
cancel— unchanged surface, explicit cause internally. With detachment the default,cancelis how a detached worker is stopped before its ceiling: it fires the worker’s scope with an explicit handle-cancel cause and detaches the handle. -
forget— RECOMMEND DELETION. Its current behaviour (builtin_forgetconcurrency.rs:217→detach_handleconcurrency.rs:458): it sets stateForgotten, drops the result receiver, and clears the cache — but it does not touchhandle.cancel, so it never stops the worker. Withspawnand admittedwatchalready rooted at the durable root, this is no longer detachment; it is “discard the observation channel while letting the worker run.” That is not identical to merely delayingawait: retaining a handle preserves laterpoll/await/cancel, while a forgotten handle is intentionally unobservable (ensure_live,concurrency.rs:228, makes observing one an error). The only residual surface is an explicit discard-result operation, and the ordinary way to express that is to stop retaining the handle. The extra verb earns no behaviour worth preserving; record the recommendation as deletion.
New behavior: detached worker lifetime is a frame policy
The one genuinely new mechanism for exarch, and the management story for
abandoned detached workers there: a one hour lifetime ceiling
(“death-clock”) the exarch frame arms on every detached spawn worker. A worker
that has neither completed nor been cancelled within one hour is
force-reaped. There is no listing, inspecting, or enumerate-and-kill primitive
for detached workers — a worker runs, a live handle may cancel it, and if exarch
abandons it the ceiling eventually kills it. Without the ceiling an abandoned
spawn { loop { … } } is an immortal zombie hanging at the root, unreachable
once the agent loses the handle binding across context compaction; the ceiling
is what makes detachment safe for a long-running agent.
- A frame policy, like the foreground wall. The ceiling is supplied by the
frame (the sibling ADR’s missing type), not baked into core: the exarch
frame arms a one hour ceiling on each detached
spawnworker so an agent cannot accumulate zombies; the REPL frame arms none, leaving an interactive worker tocancel, root abort, or session exit.watchis REPL-only and therefore has no exarch ceiling. A worker’s lifetime is set by its host, not the model. - A capped default, not a per-spawn knob. One hour is deliberately longer
than exarch’s 30 s foreground wall (
CALL_TIMEOUT_SECS,exarch/src/tools/ral.rs:31) but still bounded enough for abandoned fan-out and lost handles. There is no per-call dial. - The reaper is the worker’s own scope, fired by the shared timer service.
A worker stores its
CancelScope; reaping it is a(scope, now + 1 hour)entry in the one timer/reaper service the sibling ADR introduces for the foreground wall — not a watchdog thread per worker, which thread-per-spawnwould require and not scale. At expiry the service flips the scope withDeadline; the worker unwinds at its nextcheck, and its child-wait loop SIGTERM→SIGKILLs any subprocess group, exactly as the wall reaps the foreground.
Consequences
- The detached/foreground line makes “does this worker survive the foreground
turn?” answerable by reading the construct:
spawn, REPLwatch, and&survive; pipelines do not;parjoins on success, but its implementation fan-out is detached and can orphan on foreground early exit or worker failure. awaitno longer hangs past the wall and no longer collaterally kills the worker it awaits — the two failure modes the barerecv()produced both go away onceawaitsharesrace’s cancel-aware loop and the worker lives at the root.- Deleting
forgetremoves a primitive whose name lies. The lost surface is only explicit “discard this result while the worker keeps running”; retaining a handle already preserves observation and cancellation, and dropping the binding already expresses fire-and-forget. - Exarch detached workers can no longer leak forever: the one hour frame
lifetime ceiling reaps abandoned
spawnworkers. REPL workers remain interactive session state and die bycancel, root abort, or session exit. parkeeps its semantics, but its workers are now documented as root-parented — a fact a reader would otherwise have to derive from the prelude.- Detached workers are unmanaged by design — there is no introspection
primitive. No
jobs-for-handles, no enumerate-and-kill: an abandoned exarch worker is reaped by the lifetime ceiling, not found and stopped by hand. The REPLJobTablecovers stopped foreground process groups, notspawnhandles.jobs/fg/bg/disowntherefore gain no spawn-handle analogue.
What shipped
The recast landed on turn-local-state.
forgetdeleted — the builtin, theHandleState::Forgottenstate, and every arm that handled it.detach_handlenow only ever transitions toCancelled; dropping a handle binding is the fire-and-forget gesture.awaitis cancel-aware, sharing onewait_first_settledloop withrace: it blocks in the foreground pollingprocess::check, so a deadline or interrupt unwinds the wait, but the root-scoped worker survives. The bare-recvhang and the collateral kill are both gone, pinned byawait_unwinds_on_foreground_cancel_sparing_the_worker.- The detached-worker policy is a frame axis. It rides
TurnFrameintoTurnState, flowing down throughinherit_fromand into workers throughspawn_thread. The REPL arms no ceiling; exarch arms a one-hour ceiling. (The axis first shipped asDetachedPolicy { lifetime, watch }; watch-repl-builtin then collapsed it to the baredetached_ceiling: Option<Duration>oncewatchadmission left the frame.) watch’s host-specificity — once a frame gate refusing under capture buffers — is now registration-by-absence:watchis a host-installed builtin (the ral binary installs it, exarch does not), per watch-repl-builtin.- The death-clock rides a shared reaper.
process::reaper(“deadlines as data”) is one process-global daemon holding(when, scope)entries;spawnarms(worker_scope, now + 1 h)under the exarch frame as a kept, fire-and-forget entry. This is the single timer/reaper service unify-turn-evaluation named but left unbuilt — built here. - exarch’s foreground wall moved onto the same reaper.
run_shellarms its 30 s deadline as a disarmable reaper entry rather than a watchdog thread per tool call, and the guard disarms it when the turn finishes early. The wall and the death-clock are now one service, differing only in scope,when, and whether the entry is kept (the death-clock) or disarmed on completion (the wall).
Not adopted. The explicit pipeline-child scope is still cutover work —
stages share the foreground scope. par is unchanged prelude code over
spawn, but its fan-out now inherits the frame’s detached policy through
spawn_thread, so a par under exarch is ceiling-armed like any other spawn.
Alternatives considered
- Keep one foreground parent for every worker (the rejected collateral-kill
shape). Rejected: this is exactly the bug. Every worker a child of the
swappable foreground (
shell.turn.cancel) means a foreground timeout reaches a worker meant to outlive the turn. A boolean flag onspawn_thread(“detached: bool”) was the obvious patch and is rejected for the same reason the sibling ADR rejects flag-soup — the handle already carries the distinction, so the scope choice should read off it, not off a parallel parameter. - Keep
forget. Two ways to keep it were weighed. (a) Redefine it as an explicit discard-result marker — but after that call the same binding cannotpollorawait, while dropping the binding already expresses “do not observe this result” without poisoning the handle state. (b) Reserve the name for a future reachability-GC (a worker whose handle is no longer reachable is reaped) — a real idea, but it is a different mechanism with no implementation here, and squatting the name now is premature. Rejected: delete it; revisit the name if reachability-GC is ever built. - A foreground-nursery
par. Rejected for now. It would require a foreground-parented internal spawn path solely for preludepar; keepingparordinary library code overspawnis simpler, and the orphan window (foreground deadline, scope-routed interrupt, or worker failure) is bounded in exarch by the lifetime ceiling. In the REPL the tradeoff is visible: a foreground Ctrl-C duringparcan leave root workers until explicitcancel, root abort, or session exit. watchin every host. Rejected: a detached watcher must write only to root-owned or handle-owned resources. exarch’s stdout/stderr buffers are foreground-frame resources collected at turn end, so a root-surviving watcher would write into stale capture state.watchis admitted only by a frame that supplies a durable watch sink; today that is the REPL.- A per-spawn deadline knob (
spawn { … } timeout: 5m). Rejected in favour of the capped default: ordinaryspawnstays one boring primitive with host-owned lifetime policy. A legitimately long-lived exarch job should be a second, explicit mechanism — for example a promotion/disown operation into a host-managed durable job — rather than a timeout dial hidden on every spawn. That mechanism is future work and must not be confused with the REPL’s POSIXdisown.
Open questions
- Long-running exarch work. One hour is the default detached-worker
ceiling. If exarch becomes a genuinely long-running agent, some work may need
to outlive that bound. Do not solve that by adding a timeout knob to ordinary
spawn; design a second mechanism that intentionally promotes a handle into a host-managed durable job, with its own ownership, listing, cancellation, and lifetime policy.disownis a possible spelling, but the REPL already uses that word for POSIX job control, so the eventual surface needs care. - The kill-all gesture — what cancels the durable root, given foreground-scope interrupts cancel only the foreground — lives in unify-turn-evaluation. Referenced, not duplicated.
& is spawn; the JobTable stays separate
cmd & is not a second parallel backgrounding mechanism — the elaborator
desugars it to spawn { cmd } (core/src/elaborator.rs:578, “pure sugar”). So
& inherits this ADR’s handle model wholesale: root-parented, detached,
surviving the command, reaped by completion / cancel / the host lifetime policy
/ root abort or session exit. There is no second “task” vocabulary to reconcile.
The Unix JobTable (jobs / fg / bg / disown, see
jobs) is not a management layer over & or spawn.
It tracks foreground process groups that the kernel stopped and the evaluator
reported as Escape::Stopped; ral/src/jobs.rs explicitly says an
&-backgrounded pipeline is an in-process spawn handle and is not tracked
there. This keeps the language/exarch rule intact: there is no introspection
primitive for detached handles. If ral ever grows POSIX-style job-control
backgrounding distinct from spawn, that is a separate feature, not this ADR.
See also unify-turn-evaluation (the
premise: the root/foreground split), handle-settle
(the settle poll/await/race project),
hot-path-cancellation (where workers
poll the scope), scoping, map: builtins,
output-capture-and-detachment (how
this plays out for a never-closing pipe — the long-running-server flow), and
docs/SPEC.md §13.3.