Blog Archive

There's More Than One Way to Get Observability Right

The specialize-versus-unify argument feels like a religious war. It isn't. Both sides are right — they're answering different questions. There are several ways to get observability right. The way to get it wrong is to never ask which one you're building for.

Continuous Acceptance Tests

An acceptance test run once before deploy proves the data was correct for one instant. The data does not stay correct because the deploy was green. Stop retiring your best test the moment it passes. Run it forever.

Put Dex In Front of Google OAuth

Google OAuth has two surprises that make every internal-service auth story uglier than it should be. The standard workaround involves domain-wide delegation and a service account JSON key shipped to every application that wants group-based authorization. There is a much better answer that doesn't require any of that.

Incident Management

The role-based incident-response model Jesse Robbins brought from the fire service into web operations, written down from memory as I learned it. Incident Commander, Scribe, SMEs, severity, the CAN format, and the discipline that makes the framework actually work when the page goes off at 3 AM.

C-Style Thinking vs Go-Style Thinking

You can identify a programmer's native language by the tools they build. Go and Perl natives build tools with sane defaults that work out of the box. C and C++ natives build tools that require you to understand the entire problem space before you can do anything at all.

'Can' vs 'Does'

The difference between a system that can fail and a system that does fail is time. Murphy's Law is not a joke. It is a design constraint. Every moving part you add is another bet against the house, and the house always wins.

Control Repositories

A control repository is a git repository whose contents declare what should be running, intended to be reconciled by an automated controller. It looks like a source repo, but the discipline is different — reviews focus on what the system should be, duplication is often intentional, and the head of main is the state of the platform right now.

Trunk-Based Development

Long-running branches are an anti-pattern. The longer a branch is open, the more expensive the integration becomes, the less informative the diff, and the further the branch drifts from whatever is actually shipping. Trunk-based development is the alternative the continuous delivery community converged on two decades ago, and the evidence for it is overwhelming.