Deployment failure checks and queue monitoring

Deployment checks protect the serving generation and detect work that remains queued after a rollout. Every activation must pass the exact-SHA readiness fence. The control-plane wait has a 900-second budget for cold host builds; host-local checks retain a 120-second budget after the build.

When an Actions deployment sets DEPLOY_PEER_JOB, deploy/watch_peer.py supervises only the read-only readiness wait. It reads the peer host job from the current run attempt with an Actions read token. A failed or cancelled job ends the wait and removes its named one-off container. If the Actions API is unavailable, the bounded readiness fence continues; activation still requires success. The watcher never interrupts migrations or service replacement. Deployments outside Actions use the bounded check directly.

Drain through both activations

A single-host rollout that must not interrupt work uses one explicit host as a database-backed admission fence. The orchestrator runs on the control-plane machine, outside the pool it is draining. It first reads the host’s current owner token and canonical drain_fence_epoch, then asks the API to replace that exact pair with a fresh opaque owner token. The database accepts the change only if both observations are still current and rotates the epoch as it commits.

The orchestrator then waits until that exact host still reports draining and slots_used=0. A claim that committed just before Drain is allowed to finish; no later claim can commit while the status remains draining. The host is activated while the fence stays in place. Control-plane activation sets AGENT_DEPLOY_DRAINING_HOST to the exact enrollment name, which authorizes the deployment readiness check to accept only that named draining host. When new-chat placement is pinned to a different enrolled host, that host must be online at the same target commit; the name difference is not itself a failure. The check requires zero occupied slots on the draining host, the target deployment capability, a fresh heartbeat and a successful reverse sandbox manager probe, then rereads the same conditions after the network await. Ordinary readiness remains online-only.

Only a normal fenced release holding the exact active owner token may put the host online. An identical acquire or release retry is idempotent, including after a lost success response. A successor may transfer ownership only from the exact token and epoch it observed; an operator action, lifecycle or credential change, placement change, or competing rollout rotates the epoch and makes the stale request fail. The database also refuses deletion of a host while its fence is active. Deployment remains serialized for the rest of the activation protocol, but drain safety no longer depends on that serialization or on operators leaving the row untouched.

On cancellation or an uncertain result, fail closed: leave the host draining and retry with the same verified owner. Queued work waits; an in-flight turn is never killed or replayed. Migration 0051_host_drain_fence_epoch and the AGENT_DEPLOY_DRAIN_FENCE_V2 recipe marker are the rollback floor after V2 activation. The V1 marker remains only for an older orchestrator performing the bridge upgrade.

Before stopping a serving runner

Source and artifact deployments resolve the candidate runner image through Compose, then run cloud_agent_host.candidate_check before stopping the old runners. The candidate container has no network, enrollment state, sandbox manager token or live stack mounts other than the Docker socket. It creates and removes its own temporary guest on network none, verifies kernel isolation and the guest observer, and compares the built image’s runtime/helper labels with the candidate’s pins.

The stop signal drains the host service itself: it stops claiming and waits for every tracked turn child to exit without forwarding SIGTERM to those children. Each child owns exactly one turn and exits naturally when that turn settles. Turn children also install their drain handler before preflight as protection against an out-of-band signal, but deployment does not rely on that fallback.

If that check fails, the old readiness marker and runners remain in place. The replacement runners, relays and sandbox manager must still pass their live checks before mark-ready; a later failure uses the recorded-generation recovery path.

Host runtime qualification

The host service performs the full Kata probe at startup, whenever the sandbox image or root-owned runtime epoch changes, and at least every 30 minutes. This last interval bounds detection of changes made outside the supported installer. A Docker restart also restarts the runner, which probes again.

The installer writes an updating epoch before changing runtime files and a new ready generation only after verification. Updating or malformed evidence blocks acceptance. Deployment initializes a missing epoch on existing hosts after the normal runtime check. A failed qualification stops claims, logs the failure once, and retries after five seconds with exponential backoff capped at five minutes. Probe containers carry org.freeinference.chat-probe-created, so the sandbox manager excludes them from orphan-chat adoption. Each probe removes abandoned probe containers in its namespace that are over an hour old.

The host adds its qualification verdict to the child’s stdin payload after stripping claim-supplied underscore keys. The child rereads the epoch, declines during an update, and probes for itself if the epoch changed. It verifies the host network before accepting the turn. A missing epoch requires the full per-turn probe; deployment canaries without a payload always run it. The probe reads the guest kernel from the same temporary container used for member isolation and observer delivery.

See Kata runtime qualification generation for the marker format and installer requirements.

Network verification and startup work

The sandbox manager verifies a group’s bridge when acquiring a turn activation and returns network_verified. The runner may rely on that proof for up to one minute, matching the network guard’s existing freshness window. Commands after that minute verify again, including after a long setup script or at turn end. Terminals and sandbox manager workspace operations always check, as does every command when the sandbox manager did not supply that proof. The turn child still installs and reads back host rules before acceptance.

Network-rule reads share a disposable helper; route reads use a separate helper without capabilities. Reconciliation caches inspections only within one command, rereads after mutations, and verifies the final bridge identity, membership and relay addresses. It skips unchanged guest hosts/resolver writes while retaining ownership, mode and symlink checks. These optimizations preserve the network policy and freshness requirements.

A new chat’s repository is cloned, scrubbed and archived on the host while the container starts. Its checkout measurement includes the remaining clone wait (source_wait) and import (source_import). Clone failures still fail at that stage; an earlier stop discards prepared source. Startup measurements attribute qualification to preflight and cover claim request through prompt dispatch, excluding queue wait and model response time. Declined children do not publish a successful startup. A host at capacity checks completed children every second.

Empty-queue waits and CI evidence

After an empty claim, the host calls /v1/agent/worker/wait. The control plane wakes waiting hosts when Postgres reports queued, requeued or ended work (migration 0029), or after five seconds. The host then claims again; the wait endpoint never assigns work. Its dedicated database listener reconnects independently and wakes all waiters on connection or disconnection. Changes the triggers cannot observe, such as a moved placement pin or a finished Fork, are therefore still found within the normal five-second interval.

If the endpoint returns 404, the host resumes polling and retries the endpoint after five minutes. Hosts can deploy before the control plane.

CI runs the candidate check on the real Kata host for reviewed dev commits before CI Gate permits release. It uses uniquely tagged images and never runs Compose against the deployed stack. This job does not send PR code to that machine; static checks across artifacts also run on PRs. A local Docker test does not establish that the real Kata-host check passed: use the named CI job’s result as evidence.

Old process, candidate schema

The Postgres CI job finds the latest agent-* release tag reachable from the change’s base and runs scripts/check_rolling_upgrade.py. It creates a random rolling_test_* database, migrates it with the previous release’s Alembic graph, and warms that release’s storage pool with conversation reads and writes and with the writes a runner’s turn makes through the worker routes: a host heartbeat, the claim, a lease heartbeat, an event, the sandbox reservation, residency, admission and settlement, and the terminal transition, for a new chat and for a follow-up in a retained one. While the old process remains alive, it applies the candidate migration and repeats those operations on the same database backend. It also tests an idempotent migration retry, then removes the database and archived source.

Run it against disposable local Postgres. Replace the quoted placeholder with an existing prior agent-* release tag:

ROLLING_BASE_TAG='agent-<previous-sha>'
TEST_DB_HOST=127.0.0.1 TEST_DB_PORT=5432 \
  uv run python scripts/check_rolling_upgrade.py --base "$ROLLING_BASE_TAG"

This checks common storage paths against the previous release. Authenticated browser flows and compatibility with every historical release need separate evidence.

Detection after deployment

The main API runs one queue monitor; the MCP listener runs none. After two consecutive observations one minute apart, it reports queue heads at least ten minutes old:

Event	Meaning
queued_without_host	Waiting work has no enrolled, fresh, compatible host, accounting for retained ownership and the new-chat placement pin.
queue_not_starting	Old queue heads remain and no job is running, even if host heartbeats look healthy.
queue_health_unavailable	Repeated database observations failed.
queue_recovered	A reported problem cleared.

Follow-ups waiting behind an earlier turn are excluded. Occupied slots with running work are normal backpressure. The monitor reports state; it never claims jobs, restarts hosts, changes scheduling or relaxes isolation. Compute-managed hosts follow a new control-plane commit through the compute worker’s upgrade operation, which uses the host recipe’s drain. External process/HTTP monitoring remains necessary because a dead API cannot monitor itself.

Events are logged even without external delivery. To enable notifications, set AGENT_ALERT_WEBHOOK_URL in the control-plane .env to an operator-owned receiver. Queue alerts send JSON with text, event, deployment_sha and aggregate queue counts. The receiver must accept that contract; chat providers may need an adapter. Deployment-workflow alerts belong to the downstream distribution.

Queue notifications contain no prompts, repository names, job IDs, host credentials or log dumps. Redirects are refused. Delivery failures are logged without the webhook URL and do not affect requests or the reaper. Notifications follow state changes, repeat at most once every 30 minutes while unchanged, and retry failed delivery the next minute. External alerts are active only after the deployment owner configures the receiver and its credentials.

Recovery boundaries

The backup and recovery runbook defines initial retention and recovery targets, coordinated database/host recovery sets, encryption-key preservation, restore-drill acceptance, and previous artifact-release recovery.

Inspect both machines’ deployment markers and the failure reason, then retry the same verified target. A failed host rollout remains a failed rollout even if it restored the recorded generation. Control-plane failure leaves compatible migrations in place; there is no automatic database downgrade.

A hard-cancelled host activation may leave the readiness marker pending while Docker Engine continues the unbounded graceful runner stop. Running turns still finish naturally without a deployment signal reaching their child processes; after the runner exits, the host remains unavailable until activation is rerun for the same verified target. Do not publish readiness by hand.

If the host succeeds before the control plane fails, the exact-SHA gate can keep new work queued until the retry succeeds. The queue monitor detects this state. These checks protect and expose interrupted deployments; activation still spans two machines and PostgreSQL without a single transaction.

Snapshot publication failures

Snapshot publication currently has no owner retry action or automatic recovery after a failure. A rejected patch, unavailable GitHub credential, or failed Git/PR request marks the snapshot failed and its sandbox group dirty; the included Agent turns remain successful. Capture and publication require every earlier snapshot in that group to be published, and later turn settlement preserves dirty. A transient publication failure can therefore stop all subsequent snapshots in the same retained workspace. Capture retries and published-commit import retries do not recover this state. See publish_worker.py, snapshot.py and sandbox.py.

Inspect the snapshot’s last_error, the anchor job’s shared_snapshot_publish_rejected event, and GitHub’s branch/PR state while preserving the retained workspace and snapshot. Restoring GitHub access or network availability does not requeue an already failed snapshot. There is no supported database reset: marking it published or clearing the group’s dirty state would bypass the ordered publication lineage. Resetting it to captured also cannot safely recover an ambiguous push: if the push succeeded before the PR request failed, the publisher treats that remote commit as branch movement and refuses it.

A future recovery path should persist the exact prepared commit SHA before pushing, then offer an owner-authorized retry of the earliest failed snapshot. Retries must repeat entitlement and patch checks, recognize only the persisted commit as a prior push, preserve snapshot order, and continue to reject other branch movement without force-pushing. Commit-message text and the patch hash alone are insufficient evidence of the publisher’s prior commit.