Green isn't green until a browser says so

We once shipped a fix that was green everywhere and did nothing.

The change read a field off an API response and displayed it. Unit tests passed. The mocked responses in those tests contained the field, with plausible values. Continuous integration was green. Review found nothing because the code was correct with respect to everything it was checked against.

The backend did not send that field. It never had.

Nobody made a careless mistake. The author believed the API returned that field, wrote code on that belief, and then wrote mocks that also expressed it. The tests weren’t checking the belief; they were made of it. Adding more of those tests would have made the wall of green taller and no more informative.

Tests are a closed loop

That’s the general shape, and it’s worth stating plainly because it’s uncomfortable: a test suite written alongside the code it tests shares the code’s assumptions. Where the assumption is wrong, the suite agrees enthusiastically.

Mocks make it sharper, since a mock is a written-down claim about a system you aren’t running. But it isn’t only mocks. A few more from our own history were all found by something outside the loop:

Our fast test path runs against SQLite instead of Postgres, and SQLite doesn’t enforce foreign keys by default. A referential bug passed the quick suite cleanly and only appeared when the same tests ran against a real database.

Our fixtures created accounts and their transactions at the same moment. Real accounts are older than the history someone imports into them. A whole class of date-boundary bug was invisible until we built fixtures with entities older than their data.

And one that still makes me wince: a refactor renamed a test and, in the process, relaxed what it asserted. The name still described the old, stronger check. The suite stayed green while the coverage got quietly worse. The way we caught it was running the old test body against the new code.

So a browser has to look

Every evening, the QA fire deploys the day’s merged work to a live environment and drives it with a real browser. It logs in as a real account, navigates to the actual feature, and looks at what’s on screen. It does that in Spanish as well as English, where a missing translation key is invisible to any test that never renders.

Nothing about that is sophisticated. That’s the point. The browser doesn’t share the author’s beliefs, isn’t reading the mocks, and hits the real API. If the backend doesn’t send the field, the browser shows an empty space, and an empty space is a finding.

The rule we’ve settled on: the only evidence we accept that something works is the deployed thing doing it. Not a green check, and above all not an agent reporting success. A confident summary is the cheapest artifact in this entire system to produce.

When a ticket fails live validation, its changes are reverted off the integration branch and the ticket goes back to the queue. The rest of the batch keeps moving. That isolation is what makes the gate usable in practice: if a failure meant holding the whole release, there’d be pressure to wave things through, and a gate you’re motivated to bypass is not a gate.

Knowing which failures are real

The hard part of validating live isn’t the browser. It’s that live environments produce failures that aren’t defects, and treating those as defects is its own way to lose a day.

The clearest example: a set of tests that failed reliably when run together and passed reliably when run alone. Everyone’s first instinct, including mine, was shared state, perhaps some fixture leaking between files. We looked for it for a while. It wasn’t there. The tests run in parallel workers, and in combination they were starving each other for CPU on a loaded machine and tripping their own timeouts. Running the same set serially passed every time, which was the proof. The fix was timeout headroom, not isolation.

The second: immediately after a rebuild, a deployed site can serve a 404 for a few seconds while the container restarts. We’ve seen it more than once, including on the release of the first post in this series. The right response is to retry for a bounded window and check whether other services are healthy. If the API and the dashboard are answering, a 404 on one freshly restarted container is convergence, not breakage. Declaring failure there would trigger a rollback of something that was working.

Both of those are the same skill: distinguishing a wrong result from a noisy one. Chasing a phantom costs you a morning. Learning to ignore red costs you the gate, and the dashboard goes on looking like you still have one.

What this costs

The evening validation is the slowest part of our pipeline and by some distance the most annoying to maintain. Browser automation breaks for reasons that have nothing to do with the product. Every so often a genuinely correct change gets bounced and re-fixed the next morning.

I would still not give it up. Every other gate we run checks an artifact against a rule. This one checks the artifact against the world, and it is the only one that could ever have caught a field that was never there.

Which leaves one question, and it’s the one I get asked at dinner.