warden | Blog

My whole agent fleet showed up dead. Every process was alive.

Mon, 27 Jul 2026 00:00:00 GMT

The cockpit showed six agents. Every one of them orphaned — the status warden uses for “the session backing this agent is gone.”

So I checked. ps showed six agent processes, all alive. Their tmux sessions were there. The agents were, at that very moment, happily editing code in the dark. Nothing had crashed, nothing had logged an error, and the supervisor was certain the entire fleet was dead.

The bug wasn’t in my code. It was in which tmux server my code was talking to — and tmux never once told me I had the wrong one.

Here’s the context you need, then the three footguns that took me embarrassingly long to untangle.

The model: one tmux session per agent

warden runs a fleet of coding agents. The design decision underneath all of it is boring on paper: every agent gets its own tmux session, and one daemon supervises them all. tmux buys persistence for free (detach, reattach, survive a client crash), a stable handle to a live terminal, and a dead-simple way to capture what an agent is doing right now (capture-pane). The daemon is the single writer — it shells out to tmux to spawn, poke, and reap sessions, and it decides an agent is alive by asking tmux has-session -t <id>.

That last line is the whole story. has-session answers a yes/no question — does this session exist? — but it answers it about a specific server, and which server it means is decided by ambient state you never explicitly named. If the daemon asks the wrong server, every session “doesn’t exist,” and a perfectly healthy fleet reports as orphaned. Which is exactly what I was staring at.

There are three separate ways to end up addressing the wrong server. They bite in this order.

Footgun 1: `$TMUX` hijacks every child tmux command

tmux sets an environment variable called $TMUX in every process it spawns — it looks like /tmp/tmux-1000/default,12345,0. That variable is how a tmux client knows which server it belongs to. Handy for interactive use. A landmine for a supervisor.

Because when $TMUX is set, a bare tmux command binds to that server — regardless of any other configuration you thought you set. So retrace my dead fleet:

I’d started the daemon from inside a tmux session (I was already in one — who isn’t?).
The daemon inherited $TMUX pointing at my outer tmux server.
The agent sessions lived on a different server than the one $TMUX named.
The daemon ran tmux has-session -t agent-9d2e7b53 to check liveness. That command followed $TMUX to the outer server, which had never heard of agent-9d2e7b53.
has-session returned non-zero. The poller concluded the session was gone and marked the agent orphaned — six times over.

Nothing crashed. The agents were alive. tmux was doing exactly what it was told. I’d just told it something I didn’t mean to, via a variable I didn’t set.

In Go this is trivially easy to do by accident, because os/exec inherits the parent environment when you don’t set Cmd.Env:

func (ExecRunner) Run(ctx context.Context, dir, name string, args ...string) (string, error) {
  cmd := exec.CommandContext(ctx, name, args...)
  cmd.Dir = dir            // we set the working directory…
  out, err := cmd.CombinedOutput()  // …but Cmd.Env is nil, so the child
  return string(out), err  // inherits the daemon's whole environment — $TMUX and all
}

Cmd.Env == nil means “use os.Environ()”. So the daemon’s environment is every agent’s tmux environment, transitively. The supervisor’s launch context silently became part of the architecture.

The fix is to scrub $TMUX whenever a command must target a specific server:

env -u TMUX tmux has-session -t agent-9d2e7b53

The broader lesson is the one that generalizes past tmux: for a long-lived supervisor, the environment it launches in is not incidental — it’s config. Pin it. That’s why warden’s daemon is meant to run from a known, clean environment (a systemd unit, say) rather than “wherever I happened to type warden daemon.” A stray $TMUX in that environment is enough to make the whole fleet look dead.

warden also leans on the same variable deliberately, on the client side — the cockpit checks it to decide its layout:

// InsideTmux reports whether we're running inside a tmux session.
func InsideTmux() bool { return os.Getenv("TMUX") != "" }

If you launch the TUI from inside tmux, it lays the cockpit out as native tmux panes instead of nesting a second tmux inside the first. Same variable — a signal when you read it on purpose, a footgun when you inherit it by accident.

Footgun 2: the default socket is not where you think it is

Say you’ve learned footgun 1 and you want real isolation: a throwaway warden instance for a demo, on its own tmux server, touching nothing real. So you set TMUX_TMPDIR to point tmux at a private directory and move on.

Then you try to talk to that server explicitly with -S and guess the socket path — and you guess wrong. The default tmux socket is not $TMUX_TMPDIR/default. It’s:

$TMUX_TMPDIR/tmux-<uid>/default

That tmux-<uid> subdirectory (e.g. tmux-1000 for uid 1000) is easy to forget, because you normally never type the socket path at all. The instant you start passing -S by hand — in a script, a test harness, a capture rig — you have to reconstruct the exact path tmux would have derived, tmux-<uid> and all. Miss it and you’ve created a second empty server next to the one you meant to reach, and now nothing lines up.

The cleaner move is usually to not pass -S at all: set TMUX_TMPDIR, scrub $TMUX, and let tmux compute its own default socket underneath your private tmpdir:

env -u TMUX TMUX_TMPDIR=/tmp/demo-tmux tmux new-session -d -s agent-demo

Let the tool derive the path. The moment you hard-code it, you’ve taken on a piece of tmux’s internal layout as your own responsibility.

Footgun 3: precedence — `$TMUX` beats `TMUX_TMPDIR`

Footguns 1 and 2 combine into a nastier third one, and it’s pure precedence.

Inside an existing tmux session, $TMUX is set. If you then set TMUX_TMPDIR and run a bare tmux, you might expect the fresh TMUX_TMPDIR to win — you just set it, after all. It doesn’t. For a bare tmux client, $TMUX wins: the command follows the inherited session to the old server and blithely ignores the tmpdir you carefully pointed somewhere isolated.

So “I set TMUX_TMPDIR, therefore I’m isolated” is false while $TMUX is still in the environment. Isolation requires both: scrub $TMUX and set TMUX_TMPDIR. One without the other is a quiet no-op that looks like it worked right up until two servers’ worth of state bleed into each other.

# WRONG — $TMUX still set, so this targets the outer server despite TMUX_TMPDIR
TMUX_TMPDIR=/tmp/demo-tmux tmux new-session -d -s agent-demo

# RIGHT — scrub the inherited session first, then point at the private tmpdir
env -u TMUX TMUX_TMPDIR=/tmp/demo-tmux tmux new-session -d -s agent-demo

The pattern behind all three

Every one of these is the same shape: tmux resolves “which server” from ambient state — an inherited variable, an implicit socket path — and the resolution is invisible until it’s wrong. The commands never error in a way that says “you’re on the wrong server.” has-session just says no. new-session just quietly makes a second one. The state diverges silently and you debug it as a warden problem (why is the fleet orphaned?) when it’s really a tmux addressing problem one layer down.

For anything that supervises tmux from code, three rules pay for themselves:

Pin the environment. A supervisor’s launch environment is config. Scrub $TMUX for any command that must target a specific server, and don’t let the daemon inherit an interactive shell’s tmux context.
Don’t hand-author the socket path. Set TMUX_TMPDIR and let tmux derive the tmux-<uid>/default socket itself. Reconstructing it by hand means you now own a piece of tmux’s internal layout.
Isolation is env -u TMUX and TMUX_TMPDIR, never one alone. Precedence will punish the half-measure.

In warden these live at the boundary where the daemon shells out to tmux, and they’re the difference between “the cockpit shows six live agents” and “the cockpit shows six tombstones while the agents keep happily working in the dark.”

warden is an open-source Go binary for running a fleet of coding agents from one cockpit. If you want to see the tmux-per-agent model in action, the architecture docs start here, or grab it from GitHub.

The database said zero. The agents were still running.

Sun, 26 Jul 2026 00:00:00 GMT

I restarted the warden daemon. The cockpit came up with an empty fleet.

Except it wasn’t empty. tmux ls listed every agent session, alive, each one mid-task. The processes were fine. The work was fine. The daemon had simply loaded zero records from its session store on boot and concluded it was supervising nobody.

If you read the last post, this symptom will feel familiar — the supervisor insists the fleet is gone while every agent is demonstrably alive. Last time the lie came from tmux server addressing. This time it came from one layer deeper: the storage engine. And the twist is that the data was never lost. The index over it was.

The store: append-only segments, keyed by id

warden keeps its state in an embedded store — ScrivaDB, a small Go engine that every warden store embeds (sessions, pipelines, snapshots, schedules, shared context). The session store is the one that bit me. Its shape, straight from the code:

// FileStore persists sessions in an embedded ScrivaDB rooted at
// <dir>/sessions-db/: live sessions in an "active" collection and archived
// ones in a "closed" collection, each record keyed by the session id.
// A write appends one record instead of rewriting a whole per-session JSON
// file (the write-amplification the previous FileStore carried).

The important word is appends. Each write is a new record appended to a segment file; the engine keeps an index mapping each key to the latest record for that key. Reads consult the index, not the raw segments. This is a completely standard log-structured design, and it has a completely standard sharp edge.

Append-only earns you one guarantee for free: you never tear a record mid-write, because you never overwrite. warden’s own comment leans on exactly that —

// …the last write surviving a power-loss is not a requirement
// (append-only segments rule out torn reads regardless).

But notice what that guarantee covers: the data. It says nothing about the index — the derived structure that has to be maintained alongside the append, and rebuilt or rotated as segments grow. That’s the part that broke.

Rotation: the rare path that only fires in production

Append-only logs can’t grow forever, so the engine rotates: when the active segment hits a size threshold, it’s sealed and a new one starts, and the index is updated to span the new layout. Rotation is the classic place storage bugs hide, for a mean reason — it only triggers after the store has grown past a threshold. In dev, with a handful of records, you never rotate. You ship. The store fills up in production over days of real use, crosses the threshold, and then the rare path runs for the first time, on your real data.

That’s precisely what happened. The root cause was a name collision during segment rotation: the rotation logic could reuse a segment name in a way that left the index pointing at the wrong bytes. The append had succeeded — every session record was physically there, on disk, intact. But after the rotation, the index that was supposed to find those records was corrupt. On the next open, the engine rebuilt its view from that index, found nothing coherent, and handed warden an empty collection.

Zero records. Not an error — an answer. And that’s the crux of what made it nasty.

”Zero records” is indistinguishable from “empty”

Here’s the failure mode that turned a storage bug into a warden mystery: a store that returns zero records looks exactly like a store that is legitimately empty. There’s no exception, no missing file, no torn read to catch. The daemon asked “how many sessions?”, got 0, and did the only reasonable thing with that answer — showed an empty cockpit.

The supervisor trusted a number it had no way to distinguish from a lie. Which is the same shape as the tmux bug: the failing call didn’t say “you’re asking the wrong place” — has-session just said no, and here the store just said zero. Silent-wrong is so much worse than loud-wrong.

Why nothing was actually lost

The reason this is a war story and not a data-loss postmortem is a design decision that predates the bug: warden treats sessions-db/ as a derived cache, not the source of truth. The store’s own boot path spells it out:

// …if the sentinel is absent (never imported, or a prior attempt died
// partway) the derived sessions-db is wiped and rebuilt from the read-only
// legacy JSON, then the sentinel is written LAST — so a crash mid-import
// loses nothing.

The store keeps a sentinel file that means “the database is authoritative.” The database is populated (once) from plain JSON, and the sentinel is written last. The invariant: nothing lives in the fast store that can’t be reconstructed from something more durable and dumber.

So the recovery from a corrupt index was, bluntly, to throw the index away and rebuild — the exact same wipe-and-reimport path the store already runs when a first-boot import is interrupted. The corruption surface and the crash-recovery surface turned out to be the same surface, and the mitigation was already built. That’s not luck; it’s the payoff of never letting derived state pretend to be primary.

The real fix, of course, was upstream: the rotation name-collision was root-caused and patched in ScrivaDB, and warden pinned the fixed version. But the reason a rotation bug in the storage engine cost a restart and not a fleet is the cache/source-of-truth split.

The lessons

Two of them, and the first is the one I’d tattoo on a storage layer:

Append-only protects the data, not the index. “We never overwrite, so we can’t corrupt” is only true of the records. Every derived structure laid over them — the index, the rotation bookkeeping, the compaction state — is a fresh corruption surface with none of the append guarantee. Audit those paths as if they were mutable, because they are.
Rotation and compaction are where the bugs wait. They’re the code that only runs once the store is big and old — invisible in every test that starts from empty. If you write a log-structured store, force a rotation in your tests, then reopen and assert every key still resolves. The bug that ate my fleet would have surfaced the instant a test crossed the threshold and reopened.

And one structural rule that turned a scary bug into a boring one:

Keep a dumb, durable source of truth under your fast store. If your performant on-disk format can be wiped and rebuilt from something simpler, corruption is a rebuild, not a catastrophe. warden’s session store can always fall back to plain JSON; that single decision is why “the database said zero” ended in a restart instead of a support thread.

A store returning zero is a valid answer to a question you didn’t mean to ask. Build the layer underneath so that when it does, you can just rebuild and move on.

warden is an open-source Go binary for running a fleet of coding agents from one cockpit. The storage model — an embedded append-only store over a plain-JSON source of truth — is described in the architecture docs, or grab it from GitHub.

Running a fleet of coding agents without losing your mind

Sat, 27 Jun 2026 00:00:00 GMT

A few months ago I hit a wall that I suspect a lot of people are about to hit.

I’d gotten good at running coding agents (Claude Code, in my case). So good that one agent was no longer enough. I’d kick off a refactor in one terminal, a test-writing pass in another, a PR review in a third. Then a fourth. Then I’d cmd+tab into a window and have no idea which task it was, whether it was working or quietly waiting for me to approve a db:migrate, how much context it had burned, or what it had cost me.

I was spending more energy managing the agents than the agents were saving me. The thing that was supposed to give me leverage had turned into a second job: human tmux scheduler.

So I built warden — a single Go binary that runs a fleet of coding agents and gives you exactly one place to see all of them. It started Claude-only; it now drives multiple agent backends, Claude Code being the default.

The core idea: agents are cattle, not pets

The mental shift that made everything click was treating agent sessions as disposable, observable units of work — not precious terminal windows I had to babysit one at a time.

In warden, you spawn an agent against a task and walk away:

warden start "review the auth module for security issues"

Every write-type agent gets its own isolated git worktree by default, so I can have three agents editing the same repo in parallel and they never collide on the same tree. That one decision removed a whole category of “wait, who touched this file?” panic.

Then I watch the whole fleet from a cockpit instead of from eight terminals.

The cockpit: eight agents, one screen

warden tui is a tmux-composited cockpit — the agents list grouped by directory, an Approvals row for anything blocked on me, a shell, and the selected agent’s live session in a full-height pane. I drive the entire fleet without leaving it.

If I’m not at my terminal, the same fleet is a self-hosted web dashboard — Fleet summary, grouped agents, and a Metrics tab with per-agent and fleet-total CPU/memory:

No SaaS, no login, no telemetry. It’s a loopback REST API and an embedded file-backed store on disk. Runs on my laptop or a box I SSH into. By default nothing leaves the machine — the only outbound traffic is what you opt into (webhook/Slack alerts, remote access, or --calibrate).

The part I didn’t expect to care about: knowing what it costs

When you run one agent, cost is abstract. When you run a fleet, it’s a line item. warden spend prices each agent’s real model usage into dollars and gives me a budget gate, so a runaway agent doesn’t quietly run up a bill while I’m in a meeting. (Dollar pricing covers the Claude backend today; bring-your-own-model backends report tokens.)

The flip side is warden savings — an append-only ledger of the tokens warden keeps out of agents’ context (the lifecycle plumbing that would otherwise bloat every prompt), with a without-vs-with A/B benchmark. It turns “this feels more efficient” into a number I can actually point at.

An agent can drive the whole fleet

This is the part that surprised people I showed it to. warden mcp exposes the whole fleet as MCP tools. So an orchestrator agent session can spawn agents, query their status, and talk to a specific running agent — one agent managing a team of agents (a Claude session managing Claudes, say), with warden as the substrate. You’re not limited to driving it by hand.

Why one binary, no database

Every design decision bent toward “you should be able to try this in 60 seconds and trust it with your repos”:

One Go binary (warden, aliased wd). No runtime, no containers required.
A local daemon as the single writer to an embedded, file-backed store. No Postgres, no Redis, no cloud account — no database server at all.
Self-hosted and free, Apache-2.0. It’s your machine and your data.

go install github.com/srjn45/warden/cmd/warden@latest
warden setup --yes     # install missing deps (tmux, claude, …)
warden tutorial        # guided tour: spawn → watch → commit → tear down
warden start "your first task here"
warden tui             # open the cockpit

Where it’s going

warden is open source and I’m building it in the open. If you’re running more than one coding agent at a time — or you’re about to — I’d love for you to try it and tell me where it falls short.

Repo: https://github.com/srjn45/warden
Docs & guide: the rest of this site

If you’ve felt that same “I have too many agents and no cockpit” feeling, this was built for exactly that.