Controllers

A Controller is a reconciliation loop that owns one or more object kinds in Xema. Controllers are the platform's answer to "how does state stay consistent under concurrent writes, partial failures, and out-of-order events?". They are the same pattern Kubernetes uses: read the desired state, observe the current state, drive the gap to zero, emit follow-up events. The platform does not fire-and-forget — it reconciles.

A controller is not a daemon you write ad-hoc. It is a typed, registered, observable component of the kernel control plane.


The reconciliation loop

Every controller in Xema follows the same shape:

text
1            ┌─────────────────────────────────────┐
2            │   Watch (desired + observed state)  │
3            └──────────────┬──────────────────────┘
4                           │
5                           ▼
6            ┌─────────────────────────────────────┐
7            │   Diff (desired − observed)         │
8            └──────────────┬──────────────────────┘
9                           │
10                           ▼
11            ┌─────────────────────────────────────┐
12            │   Act (apply one step toward gap=0) │
13            └──────────────┬──────────────────────┘
14                           │
15                           ▼
16            ┌─────────────────────────────────────┐
17            │   Emit (CloudEvent on transition)   │
18            └─────────────────────────────────────┘
19                           │
20                           └───── back to Watch ──────┐
21                                                      │
22                                                      ▼
23                                              (next observation)

The loop is level-triggered, not edge-triggered: a controller reads the current state on every pass, not "the event that woke me up". An event that is missed, duplicated, or arrives out of order is not a correctness problem — the next reconciliation pass sees the same observed state and converges to the same decision.


Desired state vs observed state

Desired state and observed state are typed, distinct surfaces. They are never read from the same row.

SurfaceWhat it holdsWhere it lives
DesiredWhat the user / admin / upstream object asked forThe owning object's spec fields
ObservedWhat is actually true right now in the worldThe owning object's status fields + external system probes

A controller writes only to status. A human, an admin tool, or an upstream controller writes only to spec. This separation is a design rule of the controller model. It is not enforced by a registry-level guard today — the object registry has no spec/status write split, so a controller that writes to the desired surface is a bug the reviewer catches, not a request the platform refuses. Treat the rule as binding on the code you write.


Where controllers live

A controller may be shipped two ways. The decision is uniform with every other extension surface:

SourceExampleWhen to use
Platform service (apps/)workload-runtime-api reconciles workload pods; connector-gateway-api reconciles binding tokensThe reconciliation is generic and every sub-app needs it
Biome contributionA biome's manifest declares a controller contribution kindThe reconciliation is domain-specific (e.g. a biome that mirrors a Linear board into Xema issues)

Biome-shipped controllers register through the manifest the same way other contributions register. The kernel's controller registry tracks both kinds uniformly; an admin can list every controller running against the org's data with one call.


What a controller declares

A controller declaration is a closed, typed record:

ts
1interface ControllerDescriptor {
2  name:           string;            // 'workload-pod-reconciler', etc.
3  watches:        XemaObjectKind[];  // the object kinds it observes
4  emits:          string[];          // CloudEvent types it publishes
5  environment:    EnvironmentRef;    // the execution environment it runs under
6  reconcileEvery: number;            // seconds between full passes; default 30
7  maxConcurrent:  number;            // per-instance concurrent reconciliations
8}

watches, emits, and environment are surfaced to the Permission Digest at install time: the admin sees exactly which object kinds a biome's controller will read and what events it will emit before approving the install. No controller runs unobserved.


Error handling — fail-fast, retry-safe

Reconciliation errors are categorised; the dispatcher acts on the category, not on a free-form exception message:

OutcomeMeaningNext pass
okDesired = observed; nothing to doSleep until reconcileEvery
progressOne step applied; not yet convergedRe-queue immediately
transientExternal dependency unavailableRe-queue with exponential backoff
permanentThe desired state is impossibleWrite status.condition = Failed, stop retrying, emit <kind>.reconcile.failed.v1
conflictOptimistic-concurrency clashRe-read, retry once; second clash → transient

There is no silent retry-forever, no swallow-and-log, no "best-effort" path. A permanent failure surfaces on the object's status and in the audit log — the human gets the typed condition, not a 3 AM alert about a stuck loop.


Concurrency and safety

Controllers are deployed as ordinary replicas of the owning service. The kernel guarantees:

  1. Single-writer per object. A PG advisory lock keyed on (kind, id) serializes concurrent reconciliations of the same object across replicas.
  2. At-most-one action per pass. A pass that diffs five missing sub-resources creates one, then returns progress. The next pass creates the second. This keeps every step idempotent and rollback-friendly.
  3. Reconciliation budgets. A controller that exceeds its declared maxConcurrent queues subsequent objects; it does not fan out unbounded.

These are platform guarantees, not biome responsibilities — a biome-shipped controller that simply implements the Controller interface gets them for free.


Controllers you can already name

The kernel ships several first-party controllers; they are the working examples of the pattern:

ControllerWatchesDrives
Workload reconciler (workload-runtime-api)WorkloadPod desired-state → observed pod state
Binding reconciler (connector-gateway-api)ConnectorBindingCredential ready / degraded / revoked transitions
Biome lifecycle reconciler (biome-host-api)BiomeInstallationThe seven-state install lifecycle
Workflow run reconciler (workflow-engine-api)WorkflowRunRun status from job-run observations

All four expose their reconcile-failure events through the same CloudEvent registry the Service Registry advertises. A biome that wants to react to "the workload pod for my session died" subscribes to the workload reconciler's workload.reconcile.failed.v1 and gets the typed envelope — no polling, no scraping.


When a controller is the wrong tool

Controllers are for state convergence. They are the wrong fit for:

  • Stream processing. A controller is level-triggered; it does not see every event. Use a CloudEvent subscription instead.
  • Interactive request/response. A controller does not return a value to a caller. Use a capability call.
  • Cron-style scheduled work. A controller fires on state change, not on the clock. Use a scheduled workflow.

If a proposed feature reads as "every N minutes, do X regardless of state", it is not a controller — it is a scheduled workflow. If a feature reads as "keep Y matching Z", it is a controller.


  • Runners — controllers dispatch their action steps through the same runner abstraction every capability uses.
  • Service Registry — controllers register their descriptor at boot and renew a lease the same way services do.
  • Objects — every controller watches the object kinds it declares and writes only status.
  • Capabilities — the action a controller takes is itself a capability invocation, mediated by policy.

Previous: ← Runners Next: Service Registry →