Shipped while we slept

Last Sunday I woke up, made coffee, and read an argument four reviewers had while I slept.

They were arguing about a pull request none of them wrote. The change fixed how bulk imports decide which team’s accounts they land in. It had been written overnight by an AI agent working alone in an isolated checkout of our repo. One reviewer signed off. Another, prompted to care about exactly one thing, refused: the change could have let an import touch accounts belonging to a different family than the one that uploaded the file. It was right. The pull request went back, got fixed, and was merged into our integration branch before I finished the coffee.

Nobody on the team saw any of this happen. FinButler is built this way on purpose, and this post is about the machinery.

The shape of the thing

Our pipeline is three scheduled “fires” a day. In the morning, a build fire picks up every ticket in the queue and implements it. In the evening, a QA fire validates the day’s work against a live environment with a real browser. At night, a production fire promotes what passed and deploys it.

Between fires, nothing is running. No orchestrator service, no daemon, no queue worker with a heartbeat. The entire state of the pipeline lives in things that already exist: GitHub Issues carry the tickets and their lifecycle, git carries the code, the deploy platform knows what’s live. Each fire is a fresh session that reads the state of the world, does one bounded slice of work, and exits. If a fire crashes halfway, the next one reads the world and continues. If a fire runs twice, the second run finds nothing to do. That one decision, no daemon, does more work than anything else in the system, and it gets its own post later in this series.

Intake is the creative act

Tickets are GitHub Issues, and a human writes them. This turns out to be the highest-leverage thing a human does all day.

Not because the writing is hard. The scoping is. Oversized tickets, not bad code, are the pipeline’s worst failure mode. “Internationalize the whole app” is not a ticket, it’s a project wearing a ticket’s clothes, and an agent will happily spend a day on it producing a review tail that never converges. The same work, cut into nine bounded tickets, flows through in two days. We learned this the annoying way, more than once.

So intake is where judgment concentrates: what to build, what not to build, where the seams are. Everything after that is machinery.

The build fire

Each ticket gets its own agent, Claude Code in our case, working in its own isolated worktree where it can’t trample its neighbors. The agent works test-first: failing test, minimal implementation, green, commit. Then it opens a pull request into our integration branch.

The part I’d defend in a bar argument is the review. We don’t use one reviewer; we use a panel, and each member is prompted to care about exactly one thing: correctness, security, test honesty, or whether any query could cross a team boundary in our multi-tenant family-finance product. A single reviewer wants to approve; it’s agreeable by temperament. A panel where each member is told to find a reason to reject behaves differently. The team-scoping reviewer from my Sunday coffee story exists because of an earlier near-miss, and it has paid rent ever since.

CI has to be green and the panel has to be satisfied before anything merges. Failures don’t stall the batch: a ticket that can’t pass gets bounced back to the queue with a structured comment explaining why, and the rest keep moving.

The QA fire

Merged is not done. In the evening, a second fire deploys the integration branch to a live QA environment and validates every ticket the way a user would: a real browser, driven by Playwright, logging in and clicking through the actual feature in English and in Spanish, because our users live in both.

This step exists because mocked tests lie. We have shipped changes that were green in unit tests, green in CI, and simply did not work when a browser hit the real backend. Once, the frontend expected a field the actual API never sent, and every mock agreed with the frontend. The only evidence we now accept for “done” is the deployed feature doing the thing in front of a browser.

When a ticket fails live validation, the fire reverts that ticket’s changes from the integration branch and bounces the issue back to the queue. The failed work leaves; the batch stays. Per-ticket failure isolation is what keeps one bad change from poisoning a release.

The production fire

The night fire is the boring one, deliberately. It runs a promotability gate, a deterministic script that checks that everything on the integration branch traces back to a validated ticket, then promotes to the production branch, deploys, and smoke-tests. If the gate finds anything it can’t account for, it stops and files an escalation instead of shipping.

This is the one stage where we removed the AI on purpose. Building benefits from a swarm; review benefits from adversarial diversity; production promotion benefits from neither. It’s a fail-closed sequence with one owner, and adding creative agents to it adds nothing except new ways to race. Deciding where not to apply AI turned out to matter as much as deciding where to apply it. That’s the third post in this series.

What goes wrong

Plenty. A concurrency bug once let two overlapping build fires claim the same ticket and open duplicate pull requests for it. The machine that runs our CI wedged mid-batch, and two finished, approved tickets sat unmergeable until it recovered. Tickets get bounced two or three times before a human admits the ticket itself was badly scoped and rewrites it.

What makes all of this survivable is that failure is cheap. Because state lives in the world, a failed fire corrupts nothing. The ticket lands back in the queue, and the next fire picks it up with the failure comment as added context. Standing problems don’t die in a log file either: every fire ends by posting anything unresolved to a pinned status issue, so we wake up to a short list instead of silence.

What’s left for us

Scope. Taste. Go and no-go. Reading the escalations. Deciding a feature shouldn’t exist at all is still a human call, and it’s the call that matters most. The honest answer to “so are you obsolete?” is that the job didn’t shrink. It moved. That’s the last post in this series.

This post is a ticket. It was scoped at intake, built in a worktree, reviewed by the panel, checked in a browser on the QA site, and promoted by the night fire, just like everything else we ship.