Cancellation
Stopping in-flight work is one delivery mechanism — a cooperative,
cause-bearing scope tree that asks the evaluator to unwind at its next poll
point — backed by an escalation ladder that forces an exit when a user
insists. A signal handler or a TUI input thread holds neither a Shell nor a
scope, so process-global slots bridge the async edge to the live tree with
async-signal-safe atomics; the platform handlers translate each delivered
signal into a cause on those slots
(signals-are-causes). Both live in
core/src/process/signal.rs (see io-process); the
gestures that drive them differ per host.
The two pieces answer different questions:
- The scope tree (
CancelScope) answers “which subtree should unwind, and why?” — a structured-concurrency primitive that names a cause and reaches exactly the workers that inherited the cancelled scope. It is the only thingprocess::check(shell)polls. - The ladder (
ESCALATION: AtomicU8) answers “is the user escalating toward kill?” — the third delivery forces_exit(128 + sig). It is a blunt, host-agnostic floor for a process whose cooperative delivery is wedged, never a delivery mechanism itself.
The escalation ladder
The platform handler fetch_adds the ladder on every delivered termination
signal; the third hit calls libc::_exit(128 + sig) — bypassing atexit so a
wedged process always dies. Nothing else reads it for control flow: clear()
resets it at acknowledgment boundaries (a fresh prompt, a turn compile, a
session reboot), and escalation_pending() exposes it for observability only.
The force-exit floor is reachable only in non-interactive paths. The ral
batch launcher binds SIGINT to handler (main.rs, install_handlers); the
interactive REPL rebinds SIGINT to the relay (below), which never touches the
ladder. So repeated Ctrl-C at an interactive prompt is cooperative, never a
hard kill — the escalation belongs to batch scripts, to external SIGTERM/SIGHUP,
and to exarch’s async signal forward.
This is the esc-non-escalating-interrupt discipline taken to its end state: the user-facing interrupt writes a cause, and only a real delivered signal walks the ladder.
The scope tree and its cause lattice
A CancelScope is a node in a tree of Arc-linked AtomicU8 flags. A scope is
cancelled iff its own flag or any ancestor’s flag is set.
-
cancel(cause)is afetch_max— cancellation is one-way and monotone: a later, weaker cause can never mask a stronger one already in force. -
is_cancelledwalks the parent chain reading flags;causewalks it taking the maximum. No mutex, no allocation — oneAtomicU8::loadper ancestor. -
The cause is an escalation order,
CancelCause:cause value meaning who raises it Interrupt1 user asked the foreground to stop Ctrl-C / Esc / batch SIGINT Explicit2 a targeted worker teardown cancel <handle>,raceloserDeadline3 a wall-clock / lifetime ceiling expired process::reaperTerminate4 the process was asked to shut down SIGTERM / SIGHUP RootAbort5 the session root is being reaped Ctrl- \
check maps the strongest cause to CancelCause::message and
CancelCause::exit_code — the one vocabulary every poll point shares:
"interrupted", "cancelled", "timed out", "aborted" at status 130,
"terminated" at status 143 (128 + SIGTERM, what a supervisor that
SIGTERMed the process expects to read back).
Two typed scopes name the one invariant
The tree’s load-bearing rule — a turn’s foreground scope is always a descendant of the session’s durable root — is spelled in the type system, not left to discipline (unify-turn-evaluation).
DurableRoot(shell.session.root) is minted once perShell. Detached workers —spawn,&,watch— parent under it, so a foreground cancel never reaches them (concurrency-detached-vs-structured).ForegroundScope(shell.turn.cancel) is the turn’s work scope. It can be minted only from aDurableRoot(or by nesting another foreground), so an unrelated root can never be installed as a foreground by accident.- Children are minted at exactly four sites: shell init (
root.child()→ foreground),build_turn(foreground.child()per nested turn), the concurrency worker (root.child()— root-parented, not foreground), and the IPC inherit path. Pipelines mint no scope of their own — they are bounded by the foreground scope they run under and byPipelineGroup::Drop, which group- SIGKILLs on teardown (see pipeline-execution).
The signal-reachable slots
A signal handler must not lock and cannot hold a CancelScope by value. Two
process-global AtomicPtr<AtomicU8> slots publish a borrowed pointer into the
live scope’s flag for the async edge to set.
FOREGROUND_SCOPEandDURABLE_ROOT_SCOPEare published together, for the turn’s whole extent, byTurnGuard::install(core/src/turn.rs) wheneval_turnswaps aTurnStatein. Publication is a swap, not a store, so a re-entrant turn nests its scope above the outer one and reveals it again on drop (turn-local-state).- At most one session per process publishes — the signal-facing one, marked
by
SessionState::publishes_signal_slots. The swap/restore discipline is LIFO on a single thread, which concurrent sessions would violate; and a signal must target the primary session’s turn, never whichever session dispatched last. A forked session (Shell::fork_session— exarch’s sub-agents) never publishes; its host cancels it through a clonable handle on its durable root (Shell::cancel_handle) (per-agent-eval-cancel). request_foreground_cancel(cause)/request_root_cancel(cause)load the slot andfetch_maxthe cause onto the borrowed flag — the exact storescope.cancel(cause)performs, and itself async-signal-safe. A null slot (between turns) makes the request a no-op, so an idle Ctrl-C or Ctrl-\touches nothing.- Drop order is the safety argument:
TurnGuarddeclares the slot guards before the displaced frame, so they un-publish before the scopeArcthey borrow can free — a slot never points at a freed flag.
The slots are the seam the unify-turn-evaluation ADR calls the “cancel-translation slots”: the semantic collapse is onto the scope tree, never into the signal handler.
Where cancellation is observed: poll points
A cancel is a request; nothing stops until the evaluator next polls. The poll is
process::check(shell), called at:
- the trampoline (
evaluator/comp.rs,evaluator/trampoline.rs) — every tail step, so any loop ofralcalls is preemptible (the original hot-path-cancellation insight); - the iterating builtins (
builtins/collections.rs,builtins/concurrency.rs) —map/filter/eachand the worker-join loops poll between elements; - pipeline launch (
runtime/pipeline.rs,runtime/pipeline/launch.rs) — before and between stage spawns.
A computation that never reaches a poll point (a tight Rust loop inside one builtin) is not interruptible by the scope path — the contract is cooperative.
External children: teardown by cause
A blocked waitpid does not consult the scope, so RunningChild::wait
(runtime/command/child.rs) wraps it in a cancel-aware poll loop with
exponential backoff (5 ms → 100 ms cap). On each iteration it try_waits
(WUNTRACED, so a SIGSTOP’d child is seen, not spun on) and reads
self.cancel.cause(). The teardown is cause-directed:
Interrupt→ SIGINT-first, a 500 ms grace, then a group SIGKILL — a child that traps SIGINT still dies, and its grandchildren with it.Explicit/Deadline/Terminate→ SIGTERM-first with the same grace then group SIGKILL — decisive, without pretending to be a user keystroke; aTerminatehands the tree the very signal the supervisor sent ral.RootAbort→ an immediate group SIGKILL, no grace.
Every external wait goes through this one loop — the interactive REPL
foreground included (park_on_stop = true there makes a SIGSTOP classify as
a parked job instead of a kill-and-reap; it no longer selects a different,
blocking wait). A foreground external still gets its Ctrl-C from the kernel
directly — it owns the terminal (see jobs) — but a SIGTERM
delivered to ral now preempts even that wait through the root cause.
The gestures, per host
The same two mechanisms are driven by different keys on different surfaces.
| gesture | surface | what fires | effect |
|---|---|---|---|
| Ctrl-C | ral REPL, mid-eval | SIGINT → sigint_relay | request_foreground_cancel(Interrupt) + relay SIGINT to external pgids; counter untouched |
| Ctrl-C | ral REPL, idle prompt | line editor reads it as a byte | abandons the partial buffer, process::clear(); no signal |
Ctrl-\ | ral REPL | SIGQUIT → sigquit_handler | request_root_cancel(RootAbort) — reaps foreground and every detached worker; the REPL loop observes the sticky root and exits |
| Ctrl-C | ral batch / -c | SIGINT → handler | request_foreground_cancel(Interrupt) + ladder +1; third press _exits |
| SIGTERM / SIGHUP | any ral host | handler (term disposition) | request_root_cancel(Terminate) — foreground and detached workers unwind, externals torn down SIGTERM-first, exit 143; ladder +1, third delivery _exits |
| Ctrl-C / Esc | exarch TUI, active turn | cancel::raise_interrupt | cancels the per-turn Token, interrupt_foreground_child, request_foreground_cancel(Interrupt) |
| Ctrl-C / Ctrl-D | exarch TUI, idle prompt | key table → quit | drops the TUI guard; no cancellation |
| Ctrl-C / Ctrl-D / Esc | exarch TUI overlay | key table → close overlay | returns to the underlying prompt / turn; no root cancel |
| async SIGINT | exarch | chained handler | cancels the Token, then forwards into ral’s non-escalating sigint_relay |
| async SIGTERM / SIGHUP | exarch | chained handler | cancels the Token, then forwards into ral’s handler → root Terminate + ladder |
ral interactive signal dispositions
jobs::setup_signals then boot::setup_signals (ral/src/repl/session/boot.rs)
fix the interactive dispositions:
- SIGINT → relay (
sigint_relay). The relay keeps the controlling tty with the shell while a mixed pipeline (internal threads + external processes) runs, fanning SIGINT out to up to eight active external pgids via theRELAY_PGIDSslot array (PipelineRelayRAII). It alsorequest_foreground_cancels so an in-process foreground computation unwinds. It is a no-op when idle. - SIGQUIT →
sigquit_handler, the louder “reap everything” gesture (agent-binding-reaping keeps it as cancellation, never deletion). It is a cooperativerequest_root_cancel, not the default core-dump — so it satisfies “Ctrl-\must not core-dump the shell” by reaping, not by ignoring. (A priorbootline re-bound SIGQUIT toSIG_IGNimmediately afterjobs::setup_signalsinstalled the handler, leaving Ctrl-\dead in the REPL against the ADR’s shipped intent; the override is removed.) - SIGTERM/SIGHUP →
handler— translates to a rootTerminate(the whole session unwinds, the REPL loop exits 143) and walks the escalation ladder; SIGTSTP/SIGTTOU/SIGTTIN/SIGPIPE →SIG_IGN(the shell drives job control bywaitpidand rewrites terminal state without being stopped).
exarch: the chained handler and the per-turn token
exarch layers a per-root-turn cancellation Token over ral’s machinery
(per-root-turn-cancel).
Tokenis anArc<AtomicBool>, one sticky token per agent for its whole life; the drive loop threads clones throughapply/dispatch/tools, so cancelling any share halts that agent’s turn (provider streaming, staged tools). The trunk’s token flag is published into exarch’s ownCURRENTslot — the same lock-free pattern as ral’s — and a genuine turn boundaryToken::resets the flag, so a prior turn’s Esc never bleeds into the next.- The registry cascade is two-layer.
AgentRegistry::cancel(behindagent-cancel, the per-agent idle lease, and the/clear/replyreaps) cancels each descendant’sTokenand its own session’sDurableRoot(Shell::cancel_handle, held per registry entry). The token stops the drive loop between steps; the root cancel unwinds araleval already in flight at the evaluator’s poll points — without it, a cancelled agent would grind to its tool’stimeout_secswall before noticing. The trunk carries no eval-root handle: its session outlives any cancel, and Esc reaches its turn through the published foreground slot instead (per-agent-eval-cancel). installchains ral’sterm_handler: the exarch handlerraises the token then forwards into ral’s disposition, so the root-Terminatetranslation and the escalation ladder survive. Install order matters — ral’s handler first, then exarch’s chain — andbootstrap::boot_shellre-establishes it after every/clearrebuild.- Raw mode disables
ISIG, so a TUI keystroke is not a kernel signal. The TUI’s key table (exarch/src/tui.rs) separates UI shape from cancellation: idle Ctrl-C/Ctrl-D quit, overlays close, and only active-turn Ctrl-C/Esc route toraise_interrupt.deliver_interruptre-creates the SIGINT the kernel would have sent a foreground external child viainterrupt_foreground_child(Windows re-injectsCTRL_C_EVENT).
Why interactive Ctrl-C cannot force-exit
A deliberate asymmetry worth stating plainly: the third-signal _exit floor is
unreachable from an interactive prompt. Interactive SIGINT goes to the relay,
which never ticks the ladder; the TUI’s active-turn Ctrl-C goes to
raise_interrupt, which writes a flag and a cause but never the ladder.
Repeated presses re-write the same cause (fetch_max), never escalate. The hard
floor exists for batch scripts (handler), for external SIGTERM/SIGHUP, and for
an async signal exarch forwards into ral. Interactive cancellation is cooperative
by construction; the root-reap gesture is REPL Ctrl-\, not a TUI key.
See also
- signals-are-causes — the collapse of
signal delivery onto the scope tree:
Terminate, the scope-onlycheck, the one wait loop. - unify-turn-evaluation — the
root/foreground split, the
CancelCauseorder, and the per-turn cancel slots. - esc-non-escalating-interrupt — why the user interrupt writes a flag, not a counter.
- per-root-turn-cancel — exarch’s shared root-turn token and its mint-as-reset.
- hot-path-cancellation — the original cooperative-poll insight.
- output-capture-and-detachment and pipeline-execution — the foreground-deadline and group-teardown paths that read the scope.
- io-process (signals, process groups), jobs
(relay, fg/bg), agent (the turn loop the token wraps),
and
core/src/process/signal.rsitself.