The gates are the product

An agent once tried to deploy a commit that didn’t exist.

Our monorepo tracks its services as submodules, so the parent repo records a pointer to a specific commit in each child repo. There’s a plumbing command that writes one of those pointers directly. It takes a hash. It does not check that the hash is a commit, or that the commit is anywhere. You can hand it forty hex characters you made up and it will succeed, quietly, and git status will look clean.

An agent, reconstructing a pointer it needed, produced a hash that had never existed. Every local check passed. The pull request was green. The failure surfaced at the very end, on the deploy host, as an error about a ref that wasn’t ours, several minutes and one production deploy attempt after the mistake was made.

The fix wasn’t a better agent. It was a check that runs before every deploy and asks the boring question: does each recorded pointer actually exist on its remote? If it can’t get a definitive yes, it fails. Twenty-odd lines of script, which have since stopped more bad deploys than any review I’ve read.

The asymmetry

Agents are extremely good at producing things that look right. That is close to a definition of what they do. Plausibility is the output.

Most of the time plausible and correct coincide. When they diverge, nothing in the artifact tells you which one you’re holding. Reviewing harder doesn’t reliably close that gap because review is also a judgment about plausibility. The fabricated hash had the right length, the right alphabet, and sat in the right field.

What closes it is a check that consults reality instead of the artifact. Every gate worth having has that shape: it can only be satisfied by something outside the thing being checked. Don’t ask whether this looks like a commit; ask the remote whether it has one.

The gates, roughly in order

Nothing merges without a pull request. The integration and production branches are protected, and no agent has a path around that. This is the least clever gate and it does the most work. It means every change has a diff, a check run, and a place for a reviewer to object, with no exceptions for urgency. “Urgent” is exactly when the exception gets used.

The review panel disagrees on purpose. The first post described this panel; what it didn’t say is how narrow each brief is. The reviewer that watches for data crossing a team boundary is not asked whether the code is good. It is asked whether this could reach a row belonging to another family, and told that finding one is the outcome we want. Reviewers shaped like that reject things. Reviewers asked to “review the pull request” approve them, because approving is what agreement looks like and agreement is the default posture of anything trained to be helpful.

Promotion requires provenance. Before anything moves to production, a script walks everything on the integration branch and asks, per change, which validated ticket it belongs to. Anything it can’t account for stops the release.

The interesting part of that gate is a wrinkle we got wrong first. We squash some merges and merge-commit others, so the same change can appear on both branches with different hashes and look like an unaccounted stranger. The gate now compares patch content, not identity, so a change already present in a different shape is recognized instead of blocking. We shipped the strict version first and it cost us a handful of stops on changes that were fine. I’ll take that trade.

Configuration is pinned rather than inspected. Preflight doesn’t merely read the deployed settings, it asserts exact expected values for the ones that matter and refuses to deploy on a mismatch. The drift incidents are the ones I remember years later, because nothing is down, every health check is green, and the behavior is quietly wrong.

Fail closed, and pay for it

The pointer check fails closed: when it can’t verify, it refuses. That is the right default and it is not free.

One afternoon the remote API it consults was slow, the check couldn’t confirm pointers that were perfectly valid, and it held a release red for the better part of an hour. Nothing was wrong. The gate was doing exactly what we asked and the answer was still a wasted afternoon.

I want to be plain that this is the deal, not a bug we’re about to fix. A gate that gives up when it’s uncertain is guaranteed to be absent on exactly the day the infrastructure is having trouble, which is the day it’s for. The cost of fail-closed is false alarms. The cost of fail-open is a class of incident you find out about from users.

There is a documented emergency override, used rarely and leaving a trail. It exists so that skipping a gate is a deliberate, accountable act rather than something someone accomplishes by quietly commenting out a line under pressure.

The stage with no intelligence in it

We took the agents out of the last stage on purpose.

Everything reaching it has already been reviewed and validated, so there is no judgment left to exercise. Only a sequence remains: check provenance, promote, deploy, smoke-test, realign. One owner, no concurrency, nothing in it that could have an idea.

Adding creative agents there would buy no quality at all and several new ways to race: two promotions interleaving, pointers bumped underneath each other, a rollback landing on top of a deploy. When the work is already correct, all the remaining risk lives in the moving.

The build stage is where a swarm helps. The gate stage is where a swarm is the threat model.

It feels like bookkeeping

The honest failure mode of all this is human, not technical.

When a gate stops a release, the first feeling is that it’s being pedantic: the check is confused, the work is obviously fine, and the right move is to satisfy the gate’s paperwork and move on. I’ve had that feeling and been wrong about it more than once. In our case, a promotion block that read like a bookkeeping complaint was a correct report that unvalidated work was sitting on the branch.

So the rule we hold ourselves to: when a gate says no, the first hypothesis is that the gate is right. Gates are not infallible; the false-alarm afternoon says otherwise. But in our case, the bookkeeping complaint correctly reported that unvalidated work was sitting on the branch, and we very nearly talked ourselves past it.

In the first post on this blog I wrote that our permission system is the product, not a wrapper on top of it. Same claim, one level down: what we’ve actually built around these agents is a set of things that can tell them no.