AI agent orchestration driver: reconciliation loops versus events

Which execution model suits a long-running AI agent orchestration driver: one that spans days, parks at human-approval gates, is sometimes worked by hand outside the driver, and whose worker processes will die. It compares four models (a resident in-memory loop, a reconciler that re-derives state from durable storage, event or hook-driven invocation, and durable-execution engines) against primary sources such as the Kubernetes API conventions, Kubernetes and Temporal documentation, and recognised practitioners. AR Labs carried out the research in August 2026 for the driver that moves our unattended agent runs through the coding pipeline.

Method: deep research over two passes, about 30 sources, using our research protocol (CRAAP source scoring and triangulation).

Summary

The evidence is strongly consistent and one-directional. For a LONG-RUNNING, PARKING, OUT-OF-BAND-EDITED, CRASH-PRONE orchestration driver, authoritative practice endorses re-deriving desired-vs-current state from durable storage on each iteration and treating events/hooks as a latency optimisation, not a correctness mechanism. This is the Kubernetes controller model, stated verbatim in the Kubernetes API conventions ("the system's behavior is level-based rather than edge-based... This enables robust behavior in the presence of missed intermediate state changes") and by Kubernetes co-maintainer Tim Hockin ("State is more useful than events... Clients can check and re-check state at any time. This is the heart of Kubernetes' controller model").

A resident in-memory loop that holds the job's authoritative position in memory is an anti-pattern for this workload: multiple independent sources describe crash-loss ("if your agent state lives in memory, it doesn't exist") and the model has no way to observe out-of-band human edits. The specific proposed model, a supervised resident process that re-derives from durable state each tick, with events as hints, is not a novel invention; it is precisely how Kubernetes controllers, the kubelet, and database-backed job runners (Solid Queue, outbox pollers) are built. Verdict: SOUND, established practice.

The four models are not mutually exclusive. The correct architecture is a composition: a supervised long-lived process (Model 1's ergonomics) whose loop body is a reconciler (Model 2's correctness), optionally woken by hooks (Model 3's latency win), and, only if you need cross-machine durability guarantees and are willing to pay the determinism tax, backed by a durable-execution engine (Model 4). The reconciler is load-bearing; everything else is optional optimisation.

The four models: verified trade-offs

Model	Survives a crash	Sees work done by hand outside the driver	Long human-approval waits	Main cost
1. Resident in-memory loop	No: in-memory position is lost	No: it trusts its own memory	Holds a live process	Only suits short-lived, low-latency work with no run history
2. Reconciler re-deriving from durable state	Yes: state lives in the store	Yes: it re-reads the artefacts each tick	A park is just "nothing to do"; it notices the approval whenever it lands	Idempotent actions, status computable from observation, reconcile latency
3. Event or hook-driven invocation	Missed events stay missed without a fallback resync	No: a hand-done step emits no event	Waits for the next event	Unsafe as the sole correctness mechanism; best as a latency hint over a reconciler
4. Durable-execution engine	Strongest guarantees	Diverges unless every human touchpoint is routed through signals or updates	Zero-compute waits via signals, durable timers or task tokens	Determinism and replay constraints, workflow versioning

Model 1: Resident in-memory loop (while true: step(); sleep)

The authoritative position holder lives in process memory.

Tradeoffs (verified):

• Crash-loss is fatal here. "When holding execution state in process memory, a timeout or crash erases the in-memory state and forces a restart from step one. If your agent state lives in memory, it doesn't exist." (Isuru Chathuranga, Your Agent Has Amnesia, Medium, link). Corroborated: "In regular async code, memory is volatile. If your app restarts, all local variables vanish." (Vercel, Platform engineer's guide to workflow orchestration, link). Corroborated: DBOS, Why Durable Execution Should Be Lightweight (link). [Triangulated, high confidence]
• Blind to out-of-band changes. Because the loop trusts its own memory, a human doing a step by hand between runs is invisible to it. This is the same class of failure the level-triggered literature warns about (see Model 2).
• Where it IS appropriate: short-lived, single-turn, low-latency work where run history is not required. "Stateless executions run in memory and need to be short-lived. Use stateless execution when you have low-latency requirements where Run History isn't required." (Azure Logic Apps docs via ServerlessNotes, link); "A loop can answer a turn... A durable workflow can survive time." (The Agent Stack Pt.4, link). [Triangulated, high confidence]

Net: As the authority for job position, unsound for this workload. As a host for a reconciler (see synthesis), fine.

Model 2: Reconciler / control loop re-deriving from durable state

Read persisted state, compute gap to desired state, act, repeat. Holds no authoritative state in memory.

Tradeoffs (verified):

• Self-healing / tolerant of missed events and out-of-band change. "In Kubernetes, controllers are control loops that watch the state of your cluster, then make or request changes where needed. Each controller tries to move the current cluster state closer to the desired state." (Kubernetes docs, Controllers, link). "This reconciliation model is inherently self-healing—missing a transient event doesn't break the system because the controller keeps verifying alignment." (James Bowes, mirror). [Triangulated, high confidence]
• Idempotency is mandatory. "reconcile must be idempotent and must always look at the current state. If it does not match the desired state, the next reconcile fixes it." (controller-runtime cache docs, kubernetes.io blog, link). [High confidence]
• Cost: eventual consistency and reconcile latency; requires careful "computable-from-observation" status design. "status must be computable from observation, because the controller may restart and lose every cached fact about what it last did." (reconciliation-loop analysis, link). [Moderate-high]

Net: Correct core for interruptible, out-of-band-edited, crash-prone work.

Model 3: Event / hook-driven invocation

Driver runs only when a hook/event fires (e.g. post-step hook).

Tradeoffs (verified):

• Edges get missed; not safe as the sole correctness mechanism. "what happens if the CPU doesn't detect a pulse? The interrupt doesn't get serviced!" (Tim Hockin, Edge vs. Level triggered logic, link). Kubernetes API conventions: level-based design "enables robust behavior in the presence of missed intermediate state changes" (link). [Triangulated, high confidence]
• Blind to out-of-band change specifically. "out-of-band changes produce no watch event, and a periodic resync is the only way to notice them." (controller-runtime cache docs, link). This is the decisive point for a human working the task tree by hand: a hook only fires on driver actions, so a hand-done step emits no event. [High confidence]
• Where it IS appropriate: as a latency optimisation layered on top of a reconciler. Kubernetes controllers are triggered by watch events but their logic is level-based; "The trigger is event-driven; the behavior is not." (GoLinuxCloud, link). [High confidence]

Net: Good accelerator, unsound as sole driver.

Model 4: Durable-execution engines (Temporal, Azure Durable Functions, AWS Step Functions, Restate, DBOS)

Engine persists an event history/journal; on crash it replays to reconstruct state and resumes.

Tradeoffs (verified), headlines:

• Strongest crash/wait guarantees. Waiting workflows consume zero compute; durable timers "survive Worker restarts, deployments, and infrastructure migrations." (Temporal docs, Reliable document approvals, link). [Triangulated]
• Determinism/replay tax. "Determinism is required in the control flow because durable execution re-executes code for recovery." (Jack Vanlightly, 2025-11-24, link). Versioning pain: "Every time workflow code changed, durable execution required versioning, and if not done correctly, replay would fail." (Long Quanzheng, ex-Indeed, iWF creator, link). [Triangulated]
• Tension with out-of-band human work. Replay-based engines assume they are the source of truth for execution history. A human editing the underlying artefacts by hand does not update the engine's journal, so the workflow's model can diverge from reality unless every human touchpoint is funnelled through signals/updates (Temporal's Update handler dedupes by update ID; docs, Reliable document approvals). A reconciler that re-reads the artefacts each tick observes the hand-edit for free.

Net: Best-in-class durability if you can pay the determinism tax and route ALL state changes through the engine. Weaker fit than a reconciler when humans mutate the substrate out-of-band.

Load-bearing questions: direct answers

Q: Does authoritative practice endorse "re-derive from state each iteration, treat events as latency optimization not correctness"? Yes, explicitly and from primary sources. Kubernetes API conventions: level-based "enables robust behavior in the presence of missed intermediate state changes." Tim Hockin (Kubernetes co-maintainer, Google): "State is more useful than events... This is the heart of Kubernetes' controller model." controller-runtime docs: reconcile receives only a name, must re-fetch current state, must be idempotent; "out-of-band changes produce no watch event, and a periodic resync is the only way to notice them." Three independent authoritative sources. [Triangulated, high confidence]

Q: Is holding long-running orchestration state in memory an anti-pattern? Yes for this workload. Crash-loss (Chathuranga, Vercel, DBOS) plus blindness to out-of-band change (level-triggered literature). In-memory loops are appropriate only for short-lived, low-latency, no-history work. [Triangulated, high confidence]

Q: Is "supervised resident loop re-deriving from durable state each tick" a recognised, sound pattern? Yes. It is the kubelet (supervised long-lived process; on restart "reconciles the running containers against the desired state" from the API server, not memory; kubernetes.io What happens on restart link); Kubernetes controllers/operators (non-terminating control loop plus periodic resync); and database-backed job runners (Solid Queue's fork-supervisor plus DB-backed state; outbox pollers). [Triangulated, high confidence]

Q: Human-in-the-loop / long pauses: engines vs reconciler polling. Engines model waiting via awaitables that consume no resources: Temporal signals plus wait_condition plus durable timers; Step Functions waitForTaskToken (pause up to 1 year, resume via SendTaskSuccess); Azure Durable Functions WaitForExternalEvent. A reconciler instead polls durable state each tick and notices the approval whenever it lands, including if a human set it out-of-band. For arbitrary-length parks with out-of-band completion, the reconciler tolerates the human doing the step by hand for free; the engine requires that completion to be routed through a signal/token/event or the engine's model diverges. [Triangulated]

Q: Reliability primitives each model needs.

• Reconciler: idempotent actions plus status computable from observation.
• In-memory loop: nothing saves it from crash; that is the point.
• Hook-driven: idempotency plus a fallback resync (else missed edges are permanent).
• Durable engine: determinism in control flow, idempotent side effects (at-least-once), versioning discipline, exactly-once effects via idempotency keys. "generate a deterministic key per intended action... Pass it to the downstream API's idempotency-key header" (Chathuranga, link); at-least-once plus duplicate-as-given (Sidekiq analysis, link). [Triangulated]

Q: 12-Factor Agents / current agent-orchestration practice (2024-2026). HumanLayer's 12-Factor Agents endorses owning your control flow (Factor 8) and launch/pause/resume with simple APIs (Factor 6): "certain types of tool calls may be reason to break out of the loop and wait for a response from a human or another long-running task"; agents "resume exactly where they left off by reloading the Thread from a ThreadStore." (factor-08, link; 12 Factor Agents overview, link). Restate frames agent loops as distributed systems: "every tool call is a remote hop, every user interaction is a pause, every retry risks doing the same work twice... a single network hickup can mean that the agent loses all context and progress." (Ewen/van Dongen/Shilman, 2025-06-19, link). This literature aligns with the reconciler-over-durable-store position: own the loop, keep state in a store, resume from the store. [Triangulated]

Verdict on the proposed model

"Supervised resident loop re-deriving from durable state each iteration, with events as a latency optimization": SOUND, established practice. Not novel; it is the Kubernetes controller model applied outside Kubernetes. It correctly separates the host (a supervised long-lived process for ergonomics/liveness) from the authority (durable state re-read each tick). Its defining property, re-deriving desired-vs-current from durable state, is exactly the property that makes it tolerate the three hard requirements: (1) crashes (state is in the store, not memory), (2) arbitrary parks (a park is just "current == desired, nothing to do"), and (3) out-of-band human steps (the reconciler observes the new state on the next tick; no event needed). Treating hooks as latency optimisation, not correctness, is explicitly the Kubernetes stance.

One caveat the proposed model must honour to be sound: the reconciler must be idempotent and its notion of "done" must be computable from observed state, not from remembered actions. If any step's completion cannot be re-derived by looking at the artefacts (i.e., you must remember "I already did X"), you need an idempotency key or a durable step-ledger. This is where a durable-execution engine or an outbox/step-record table earns its place inside the reconciler.

Strongest dissenting view

The sharpest dissent is not against reconcile-from-state; it is against reaching for heavyweight durable-execution engines (Model 4). Long Quanzheng (ex-Principal Engineer, Indeed; Cadence/Temporal contributor; creator of iWF) argues "Durable Execution is NOT the ONLY way": replay-based determinism creates "Non-Deterministic errors" and versioning complexity, and he built iWF specifically to avoid replay: "there is no versioning API at all in iWF because there is no replay at all." (link). Jack Vanlightly independently documents the determinism tax and warns "The durable function is not an atomic transaction" (link). A durable-execution vendor (Restate) argues the opposite: that even simple agent loops need durable execution or a single network blip loses all progress. This is a documented disagreement: the anti-engine camp says a DB-backed reconciler with idempotency keys is simpler and avoids replay pitfalls; the pro-engine camp says hand-rolled recovery is where teams get subtle at-least-once bugs. Both agree state must be durable; they disagree on who owns the recovery machinery. [Disputed, documented with equal prominence]

Claims that could not be triangulated (two sources or fewer)

1. Michael Hausenblas as a named authority on level-vs-edge. The research question referenced Hausenblas (Programming Kubernetes). No Hausenblas primary source explicitly framing level-vs-edge could be retrieved (his site timed out; searches surfaced Bowes and Hockin instead). The level-vs-edge argument is nonetheless strongly triangulated via Hockin, the API conventions and Bowes. [Hausenblas attribution: unverified]
2. Exact slide wording of Tim Hockin's deck came via WebFetch summarisation of Speaker Deck, not a transcript. The quoted phrases ("State is more useful than events"; "the heart of Kubernetes' controller model") should be treated as high-confidence-but-verify-against-original. [Single-fetch, moderate confidence on exact wording; high on substance]
3. James Bowes verbatim quotes. The canonical HackerNoon/Medium URLs returned HTTP 403, so a third-party mirror (wiki.malloc.dog) was used. Substance triangulates with Hockin and the API conventions; exact Bowes wording is [mirror-sourced, verify].
4. "Solid Queue supervisor + DB state" as an intentional reconcile-from-store design is sourced from the project README/search snippet, a single source with promotional purpose. The broader "database-backed job runner re-derives from durable state" pattern triangulates via outbox-pattern sources, but Solid Queue's specific internal design is [single-source].
5. DBOS/Restate feature specifics beyond the durability headline are vendor-authored (purpose score low on CRAAP); they are treated as vendor claims, not neutral fact.

Methodology

Search strategy: Pass 1 covered breadth across 6 themes (level-vs-edge; in-memory anti-pattern; 12-factor; durable engines; kubelet supervision; HITL). Pass 2 was a deep-dive on the level-vs-edge crux and durable-engine tradeoffs via primary sources (K8s API conventions, K8s docs, Hockin deck, controller-runtime docs, Temporal/AWS/Azure docs). Sources analysed: about 30 across primary docs, recognised practitioners, and vendors. CRAAP: primary K8s docs and Hockin (Google/K8s maintainer) scored 4.5 or higher on authority; vendor blogs were downranked on purpose but used for feature facts; SEO recap blogs were used only as corroboration, never as sole source. Confidence distribution: high on the central verdict and level-vs-edge; moderate on exact verbatim quotes behind 403 walls (mitigated by triangulation); disputed on engine-vs-hand-rolled.