There is no daemon

The first version of our build pipeline was a service. It stayed up, owned a queue, held a state machine in memory, and had a comment in it promising a real database once the state machine got serious. We ran it for a few weeks and deleted it.

What replaced it has no process at all. Three times a day a scheduled session wakes up, reads the state of the world, does one bounded slice of work, and exits. In between, nothing of ours is running. There is no queue holding anything, and nothing anywhere keeping a heartbeat.

That sounds like a downgrade. It’s the decision that made every later failure survivable, which is why it gets a post.

What the daemon actually cost

The service worked. The problem was that it became the thing we watched.

Its state lived in its own memory and its own tables, which meant its state could disagree with reality. A ticket could be “in progress” according to the service and untouched according to git. A restart lost whatever was mid-flight. Debugging a bad run meant reconstructing what the service thought was happening, which was a different question from what had happened. We had built a second source of truth and then spent our time reconciling it with the first one.

The sentence I kept typing into postmortems was that the code was fine and the state was wrong.

Where the state lives now

Nothing in the pipeline stores its own state. It borrows.

The ticket queue is GitHub Issues. An open issue with the queued label is the queue. Lifecycle is labels: claimed, ready for QA, validated, blocked. Done is not a label; done is the issue being closed. The code and its history are git. What’s actually deployed is whatever the deploy platform says is deployed, which we ask rather than remember.

All three of those existed before the pipeline did, are inspectable by a human with a browser, and survive any crash of ours, because none of them are ours.

A fire, then, is not a step in a workflow. It is a function of the world: read everything, decide what the next bounded piece of work is, do it, exit.

Re-running is therefore free. If a fire dies halfway because of a killed process, closed laptop, or expired token, the next one reads the world and continues. There is no resume protocol because there is no position to resume from, and firing at an empty queue does nothing at all.

Failure heals the same way. When a ticket fails, its lifecycle label goes back to queued and the fire leaves a comment saying what broke. That isn’t error handling bolted on the side; it’s the same mechanism as everything else, pointed at a worse day. The failure has become part of the state of the world, so the next fire sees it, and the comment is context for whoever picks it up.

Idempotency is not free

Here’s where I have to be honest, because “each run is a pure function of the world” is a lovely sentence that took us two real incidents to earn.

The first: for a while a build fire could spawn a second agent that shared its run identifier. Both were reading the same world and both concluded, correctly, that a ticket was unclaimed and theirs to take. Two pull requests, same ticket, same code, opened minutes apart. The reads were fine. The problem was that reading and claiming weren’t one step, so the claim was decorative.

So we added a mutex, a claim a fire has to take before it touches a ticket. That produced the second incident, and this one I still think is funny.

Fires often run inside isolated checkouts of the repo, one per run, so that no agent can reach into another’s working tree. The mutex lived in a directory inside the checkout. Every fire in its own checkout got its own private copy of the lock, took it successfully, and proceeded. A lock that always grants is the same as no lock with extra steps. The watchdog had the mirror-image bug: it read an empty ledger in a fresh checkout and announced the queue had gone stale while the queue was in fact moving.

The fix is one line of intent: resolve the shared checkout, and put the ledger and the mutex there. One ledger, one lock, however many isolated workspaces. The lesson generalizes past our setup. If your coordination state can be accidentally copied, it isn’t coordination state.

Drift, and the healer that runs first

Labels drift. Two fires overlap, a human edits an issue, an external tool moves something. Left alone, drift produces label combinations that make no sense: a ticket that’s both queued and validated, or in-flight with no live run behind it.

Every fire’s first action is reconciliation, before it looks at any work. Two rules do almost all of it. Closed wins, because a closed issue is the strongest statement anyone can make about a ticket. And when an issue carries a contradictory set of labels, it resolves to the least advanced state in the set.

That second rule is deliberately pessimistic and I’d defend it hard. If the state is ambiguous about whether something was validated, the safe reading is that it wasn’t. Resolving downward means the worst case is redundant work. Resolving upward means shipping something nobody checked.

Nobody reads the log

The last piece is unglamorous. A headless session that fails at night produces a log that no human will ever open.

So the pipeline doesn’t rely on logs for anything that matters. Every fire ends by posting unresolved problems to one pinned status issue, deduplicated by fingerprint so a standing problem doesn’t generate a new comment every run. If something is broken for three days, it’s one comment that stays visible, not nine identical ones nobody scrolls to.

The morning experience is a short list on one page. That is the entire monitoring story, and six weeks in I haven’t wanted a dashboard.

What we gave up

Latency. Work sits until the next fire, so a ticket finished at noon waits for the evening validation and the night promotion. A daemon could have done all three in twenty minutes.

We pay that gladly, because the thing we bought is that no failure in our pipeline can produce a state a human can’t read in a browser and fix by editing a label. Crash-only design has been written about for two decades in the context of servers. It applies just as cleanly to agents, and more urgently. An agent’s failure modes are stranger than a server’s, and the ability to say “throw the run away, the world is still correct” matters more when the thing that failed was capable of being creative about it.

The next post in this series is about the gates: where human judgment gets encoded so that agents can be trusted to run unattended, and why the production step is deliberately the least intelligent stage in the whole system.