Backend hard-limit recovery

When a coding agent hits a provider hard limit — Claude’s 5-hour session clock, a weekly cap, or a monthly spend cap — warden can do more than pause and wait. The reactive backend recovery coordinator picks the next eligible subscription backend from the backend registry, switches the agent over through the existing hot-swap lifecycle, and only clears the recovery once the new backend demonstrably stabilizes.

No operator action is required unless every known backend is exhausted.

The claim: recovery is reactive and automatic — the coordinator fires on a confirmed hard-limit signal, ranks all subscription backends by their minimum known remaining headroom, and advances to the next candidate if the winning backend is immediately rate-limited. Only when every candidate is exhausted does the session enter waiting_for_capacity, where it auto-retries on the earliest known quota reset without operator action.

How it works

Detection

A confirmed rate_limited status transition starts at most one recovery generation per agent. Pane-driven detections hand the scheduler a fresh RateLimitObservation (excerpt + timestamp + opaque fingerprint) so reset parsing and diagnostic capture use detection-time bytes rather than a stale LastPaneExcerpt snapshot. The poller feeds that observation when:

• The agent’s backend implements pane-based rate-limit detection (Claude, Codex, Cursor, Antigravity, Aider) and the captured tmux pane shows a provider hard-limit banner, or
• Claude’s rate-limit menu (“Stop and wait for limit to reset” / “Upgrade your plan”) is recognized and the safe wait option is confirmed selected — this fires recovery immediately, without waiting for a later banner, since the post-menu pane can render with no parseable reset clause at all (e.g. a weekly reset: Requesting rate limit reset for weekly limit). A later banner for the same incident is still captured for diagnostics but is a safe no-op for recovery, or
• The backend is pane-blind (OpenCode, Crush, Goose — Tea TUIs with no parseable limit text) and the daemon’s periodic usage snapshot reports rate_limited for that backend (~60 s poll lag via limitSessionsFromSnapshot).

Menu, banner, and usage-API observations for the same agent all converge on the same handler, so whichever evidence arrives first claims the generation and every later signal for that incident is a no-op — never a duplicate recovery.

Antigravity specifics. The banner may carry an absolute clock (Available at 15:30) or a relative duration (Individual quota reached … Resets in 10m5s); both are parsed into an absolute reset time. The banner is only honoured in the region directly above (or inside) agy’s input box, so an old banner higher in scrollback, or quoted text in the conversation, is not mistaken for a live limit. The agent’s quota bucket also follows the model agy is actually running: when the status line shows a model family (Gemini … vs Claude …/GPT-OSS …) whose bucket differs from the bound one, the binding is re-bound and a quota_rebound event is recorded, so usage reconciliation watches the bucket that is really draining.

A duplicate transition while a generation is already running is a no-op — the coordinator owns the session exclusively. Reactive recovery runs for every agent session; it is not gated on handover settings (those still control context-fill handover only).

Usage refresh

The coordinator reads authoritative backend-usage windows from internal/backendusage. Each window carries:

• A stable scope key (the provider’s constraint identity)
• Model selectors (which models the window applies to)
• An optional used_percent (nil = unknown, never fabricated as zero)
• An optional resets_at (nil when the provider supplies no reset time)

Candidate ranking

Eligible candidates are (backend, model) pairs from the backend registry where:

• The backend is installed, enabled, not local-only, and on a subscription or free tier
• The model is enabled and has auto-assign set
• Existing tier and role rules are respected (a tier-1 role gets tier-1 candidates first)

Candidates are ranked by minimum known headroom across all applicable usage windows:

• headroom = clamp(100 - used_percent, 0, 100) per window
• The tightest overlapping constraint is authoritative
• Unknown headroom (no data or refresh failure) ranks after known-positive headroom — never fabricated as zero or 100%

Switching

• If the top candidate is the original pool, the coordinator uses the same-backend Restore path (same process, same key/prompt).
• Any other candidate uses the existing HotSwap lifecycle (hot-swap handoff, new process, same session identity).

Both paths preserve the session ID, worktree, branch, pipeline job, Autopilot run/task, role, parentage, and all tags.

Stabilization

A candidate is declared stable only after the agent stays in a live non-rate-limited status (working, idle, or waiting-for-input) for at least rate_limit.recovery.stabilization_window (default 10 seconds). A process launch alone is insufficient.

If the candidate is immediately rate-limited after switching, the attempt is recorded as immediate_hard_limit, and the coordinator advances to the next candidate in the same generation.

Exhaustion and waiting

When no unattempted candidate with positive known headroom remains, the agent transitions to waiting_for_capacity. The coordinator:

1. Persists the full attempt list, generation, and known reset times
2. Schedules a retry at the earliest known resets_at across all limited pools — or falls back to rate_limit.retry_interval (default 30 m) when no reset time is available
3. Fires the retry automatically when the timer fires

If the original pool is restored and eligible, it resumes in place. If a different candidate wins the re-ranking, HotSwap is used.

Manual actions always win

A manual warden switch, stop, or delete supersedes automatic recovery immediately:

1. The recovery record is cleared
2. The active timer is cancelled
3. A backend_recovery_superseded event is emitted (plus Phase 9 recovery_superseded on the durable event stream and append-only audit trail)
4. Stale timers and late callbacks are generation-guarded and become no-ops

Recovery state in the UI

The agent’s status shows the current phase wherever status is shown (TUI, web, CLI warden status <id>, API, MCP):

Phase	Displayed as
refreshing_usage	recovering: refreshing usage
switching	recovering: trying <backend>/<model>
stabilizing	recovering: stabilizing <backend>/<model>
waiting_for_capacity	waiting for capacity; retry <time>; attempted N

The backend_recovery field on every session object is non-null while recovery is active. It contains phase, trigger_source (usage/menu/banner/manual), capacity_domain, bucket_key, freshness, reason, the current candidate, the full attempt history, known reset times per scope, and next_retry_at. Null means no recovery is active.

SSE subscribers see a notification on every phase transition so the web UI and any connected client stay current without polling.

Event log and audit trail

Recovery emits legacy backend_recovery_* events for compatibility and the Phase 9 vocabulary below. Both land on the durable agent event log (warden status <id> / TUI detail / get_agent). The Phase 9 names are also written to the append-only audit trail (~/.warden/audit.jsonl, warden inspect audit) so operators can answer why a swap did or did not occur without relying on the systemd journal alone.

Records carry safe domain/bucket identifiers, timestamps, agent IDs, trigger source, freshness, and reasons — never tokens, raw credentials, pane text, or transcripts.

Event	When
usage_snapshot_received / usage_snapshot_failed	Provider capacity poll succeeded or failed
quota_bucket_exhausted	A fresh mandatory bucket reported exhausted
quota_impact_calculated	Bucket→agent impact reconciliation finished
recovery_started	A recovery generation claimed the agent
candidate_attempted / candidate_result	A replacement was tried and its outcome
waiting_for_capacity	Exhaustion; retry scheduled
recovery_stabilized / recovery_superseded	Generation completed or operator override
backend_recovery_started	First hard-limit detection (legacy alias)
backend_usage_refreshed	After each usage read (ok or unavailable)
backend_recovery_candidate_selected	A candidate is chosen (legacy alias)
backend_recovery_switch_started	HotSwap or Restore begins
backend_recovery_stabilizing	Switch succeeded; stabilization window started
backend_recovery_attempt_failed	Attempt outcome: launch_failed or immediate_hard_limit
backend_recovery_waiting_for_capacity	Exhaustion; retry scheduled (legacy alias)
backend_recovery_stabilized	Recovery complete; candidate confirmed stable (legacy alias)
backend_recovery_superseded	Manual action cancelled the recovery

Events never contain provider credentials, account identifiers, raw terminal banners, prompts, or transcripts. Usage percentages appear only on local authenticated surfaces; notifications say “capacity known/unknown” and the reset time.

Configuration

rate_limit:
  recovery:
    enabled: true              # set false to fall back to same-backend auto-resume only
    stabilization_window: 10s  # Go duration; how long candidate must stay live before clearing
    usage_reconciliation:      # opt-in provider usage polling (default off)
      enabled: false           # true = periodic capacity snapshots + bulk reconcile
      interval: 60s            # poll cadence when enabled
      stale_after: 15m         # freshness window for authoritative exhaustion
      max_parallel_swaps: 3    # bounded concurrency for bulk recovery fan-out

The defaults are safe for most users. Lower stabilization_window to clear recovery faster in tests; raise it if false-positive stabilizations appear in a flaky environment. Keep usage_reconciliation.enabled false unless you want the daemon to poll provider usage APIs in the background — wd usage recover still works when polling is off.

Deprecated fields

handover.threshold_percent and handover.rolling_quota_threshold are deprecated as of this release. They are decoded for one compatibility window but have no effect on routing — reactive recovery replaced proactive quota prediction. context_fill_threshold (the context handoff trigger) is unchanged and still active.

Observing recovery state from an agent or MCP

# CLI
warden status <id> --json | jq .backend_recovery

# MCP
get_agent {ticket: "<id>"}
# → session.backend_recovery.phase, .current, .attempts, .next_retry_at

# list_agents includes backend_recovery on every session row
list_agents {}

Operator playbooks

Enable or disable usage polling

Background usage polling is off by default. Edit ~/.warden/config.yaml and restart (or wait for hot-reload where applicable):

rate_limit:
  recovery:
    usage_reconciliation:
      enabled: true    # or false to stop all provider usage network calls
      interval: 60s
      stale_after: 15m
      max_parallel_swaps: 3

When enabled is false, the daemon makes no provider usage network calls. Manual wd usage recover remains an explicit operator action and may still fetch fresh snapshots for its filtered scope.

Dry-run impact before recovering

warden usage recover --dry-run
warden usage recover --dry-run --aicli claude --project /path/to/repo

Dry-run always fetches fresh supported snapshots, calculates bucket→agent impact, and prints exhausted buckets, affected/skipped agents (with reasons such as unbound_legacy), candidate decisions, and would-start / would-wait outcomes — without claiming fences or starting recovery. Drop --dry-run to apply.

MCP: usage_recover with dry_run: true.

Unknown or unsupported providers

Failed, unsupported, partial, or timed-out usage fetches are diagnostic only. They never zero out prior known capacity and never force exhaustion. Agents on providers without a usable adapter stay on pane/menu evidence (when available) or wait for an operator wd usage recover after credentials/adapters are fixed. Check usage_snapshot_failed in the agent event log / warden inspect audit.

Interpreting waiting_for_capacity

Every eligible replacement currently has known-exhausted (or policy-required unknown) mandatory buckets. The session keeps its recovery generation, records attempt history, and arms a retry at the earliest known reset (or rate_limit.retry_interval). No operator action is required unless you want a different backend now — see override below. Treat this like rate_limited in fleet triage.

Permission mode across a backend swap

A cross-backend swap (warden switch, quota/hard-limit recovery, autopilot rotation) never hands the successor a mode string from the old backend’s vocabulary. Each backend maps its own modes to a neutral intent — default, plan, read-only, accept-edits, skip-all — and the stored mode is translated by intent into the successor’s own mode. For example Claude bypassPermissions → Codex danger-full-access → Antigravity dangerously-skip-permissions (all skip-all), and Claude acceptEdits → Codex workspace-write. A same-backend swap keeps the stored mode unchanged.

• No equivalent mode. When the successor has no mode for that intent (or the old mode has no known intent), warden falls back to the agent’s role default, then the configured default, then the successor’s own default posture — each validated against the modes the successor accepts. The fallback is never skip-all when the original intent is known and weaker, so a swap cannot silently widen permissions.
• Visible. A change is recorded as a hot-swap-permission-mode event and the agent record’s permission_mode is updated to the successor’s mode on success.

Failed swaps are reported and retryable

After launching the successor, warden watches it for about a second. If the CLI exits at once (bad flag, missing binary, rejected mode) the swap fails with launch_failed plus the first lines of the pane, a hot-swap-failed event is appended, and the agent record is left untouched — old backend, model, permission mode and session id intact — so nothing disappears from list_agents and you can simply retry warden switch <id> … (or restore). Manual switches, quota recovery and autopilot rotation all share this path, so the autopilot guardian sees the failure and degrades/retries instead of losing its manager.

Manually overriding recovery

warden switch <id> --aicli codex --model gpt-5-codex   # or stop / delete

Manual switch/stop/delete supersedes automatic recovery immediately: the recovery record clears, timers cancel, and recovery_superseded / backend_recovery_superseded are recorded. Stale callbacks are generation-guarded no-ops.

Legacy agents without a quota binding

Agents that predate QuotaBinding (backend/model only) show as unbound_legacy. They keep running, appear in ls/status, and are skipped by bulk usage reconciliation — warden never invents an account/bucket to mass-swap them. They acquire a binding on the next safe HotSwap (or a fresh spawn). Restore alone does not guess a binding.

Operator diagnostics: usage vs pane vs cooldown vs manual

Four signals look similar in the UI but answer different questions. Mixing them up is the usual reason an operator thinks recovery is “wrong” while wd usage still shows headroom.

Signal	What it answers	Where to look	What it is not
Provider usage (wd usage)	Aggregate subscription / window headroom for a backend	warden usage, web 🧩 panel, backend registry	Proof that a specific agent pane did or did not emit a hard-limit banner
Pane-confirmed hard limit	This agent’s tmux pane just produced a provider-specific limit signal — a hard-limit banner, or (Claude) a confirmed “wait for limit to reset” menu selection — or the usage-API fallback reported rate_limited for a pane-blind backend	Agent status rate_limited, backend_recovery non-null, events backend_recovery_started / …_switch_started	A re-read of wd usage; recovery uses a fresh pane observation (RateLimitObservation), not a stale LastPaneExcerpt, and does not wait for a banner once the menu confirms
Durable cooldown	A (backend, model) pool that just hard-limited stays ineligible until its parsed reset (or conservative fallback) expires — even after recovery stabilizes	Next recovery generation skips that pool; persists across daemon restart	Cleared by the short stabilization window; unrelated to wd usage bars flipping green
Manual intervention	Operator overrode automation	warden switch / stop / delete → backend_recovery_superseded; recovery record cleared	An automatic retry — supersede cancels timers; stale callbacks are generation-guarded no-ops

Practical triage

1. Agent flipping backends while wd usage looks healthy → check the agent’s event log for backend_recovery_* and whether the pane showed a provider banner or (Claude) a confirmed limit-menu selection. Usage headroom does not veto a confirmed pane signal.
2. Same backend reselected seconds after a successful swap → expect durable cooldown on the previous pool; if it still cycles, capture warden status <id> --json and the protected ratelimit capture under <data_dir>/ratelimit-captures/ (not ordinary audit text — raw banners stay off the event log).
3. Recovery stuck or unwanted → warden switch <id> --aicli … (or stop/delete) supersedes; attach only if you need to drive the pane by hand.
4. Cockpit layout blank/narrow after a hot-swap → quit and reopen the TUI, or let the local reattach path recover; terminal-kind cockpit panes are orthogonal to agent recovery and never start a generation.

Interaction with pipelines and Autopilot

A recovering pipeline job stays running/recovering — never falsely failed. The Autopilot guardian observes recovery state but does not intervene; the coordinator is the sole recovery owner. Run/task ownership, slot IDs, and plan parentage are preserved end to end.

Restart safety

If the daemon restarts while an agent is waiting_for_capacity, the coordinator reconstructs exactly one retry timer from persisted state on startup — no retry is lost or duplicated.