The Digital Plumber

Ask an SRE what they actually do and you’ll get a hundred answers, most of them tool names. Here’s mine, and it hasn’t changed in years: an SRE wears three hats — advisor, plumber, and firefighter — and not one of them is “product.” We are not product people. We are the people who make sure the product can be built cheaply, shipped reliably, and kept alive at three in the morning — whatever the product happens to be.

That last clause — whatever the product happens to be — is the part that takes years to fully believe. It also dissolves the single most common misconception about this job, so let me start there, because understanding why it’s wrong is understanding what the job actually is.

Systems Are Systems, Bytes Are Bytes

The myth is that a good SRE has to understand every product they support in intimate detail — every feature, every line of domain logic, every business rule — or they’ve got no business keeping it running. It would be lovely to know every product that deeply. With infinite time and infinite energy, I would. But nobody has infinite time and energy — an SRE supporting a dozen services least of all — and here’s the part that surprises people: you don’t need it.

Let me say this next part carefully, because I’ve phrased it badly for years and it lands wrong every time: this is not that an SRE doesn’t care about the product. We care enormously. The whole job exists to get the product into customers’ hands and keep those nice people who give us money happy — that is the point of all of it. Caring about a product and having the time and memory to know it the way its authors do are simply two different things, and if you’re not careful, your time and attention get spent where they don’t do a lot of good. The constraint was never a lack of care. It’s a lack of hours and a lack of headspace. And it turns out you can keep a system alive on a different, smaller set of things than knowing everything it does.

Because every deeply experienced SRE and cloud architect I know arrives, independently, at the same slightly heretical conclusion: after enough systems, the products stop looking different. They blend together. You stop seeing “a trading platform” or “a photo-sharing app” and start seeing the same handful of shapes — a thing that takes requests, moves bytes around a network, stores some, computes on some, and hands bytes back — bounded by the same short list of resources: CPU, memory, network, storage, time, energy, and people. That’s the whole periodic table; everything is built out of those. It’s bytes flowing over a network. Same shit, different day — and I mean that as a professional compliment.

Yes, there are nuances, and no, I’m not pretending a financial trading system is identical to a website that shows cute cat pictures. But the difference is mostly a reshuffling of priorities, not a difference in kind. The trading system pushes latency and correctness to the top and will spend enormous money to shave microseconds and guarantee the ledger balances. The cat-picture site pushes cost-per-view and cache-hit ratio to the top. But the cat site still wants to stay up, still doesn’t want to get hacked or defaced, still has to move bytes from a store to a browser without losing them or leaking them. Same disciplines, different dials. Turn the knobs, and one becomes the other.

I’ll give you the most deflating example I have, and I mean it as a compliment to the craft. I worked on Apple Pay. I literally rebuilt the command and control infrastructure behind it from the ground up between 2015 and 2017. 30k bare metal servers, or so they told me (I certainly did not login to each one in turn!). From a product standpoint it was genuinely special — real cryptography, real stakes, real brilliance in the parts that mattered. From an SRE standpoint? It was a JVM application running on Oracle’s RHEL-compatible Linux, managed with Puppet and later Chef. Managing it across an entire planet. That’s it. There was nothing about keeping it running that a thousand other shops weren’t also doing with a JVM app on Red-Hat-family Linux and a config-management tool (though perhaps not quite at that scale). The application was extraordinary. The plumbing was completely ordinary — and that’s exactly the point. The plumbing is supposed to be ordinary. Ordinary is what reliable looks like. You want ordinary in plumbing. In plumbing, the extraordinary and the surprising are almost always bad.

I’ve made this exact argument in a narrower place: Web3 is just infrastructure with a hoodie. Strip the rebranding off wallets and smart contracts and consensus and you’re looking at SSH keys, RPC endpoints, and replicated state machines that have been in use for ages. Products in general are the same trick at a larger scale. The mystique is a hoodie. Under it is the same body of engineering you’ve been doing all along — which is exactly why an SRE can be genuinely useful on a system whose product they couldn’t fully explain: the product is unfamiliar, but the system rarely is.

Once you internalize that bytes are bytes, you stop being intimidated by any product’s mystique, and you start being useful across all of them.

You Can Read A System Cold

There’s a corollary to this that reliably offends people, so let me say it plainly: a senior SRE or platform engineer — the titles are overloaded and largely interchangeable — can walk into a system they have never seen before and, quite quickly, tell you which patterns are healthy and which are going to hurt. Not the product logic; the shape of the company’s platform and SDLC. And the reaction, more often than you’d hope, is instant offense: “How can you possibly say that? You just got here!”

I understand the reflex, but it mistakes where the judgment comes from. The specifics absolutely differ by company, product, and system — but the patterns don’t, and there aren’t many of them. A single point of failure looks like a single point of failure whether it’s in a bank or a game studio. An unowned shared environment, a deploy that isn’t reproducible, a secret baked into an image, a retry with no backoff, a database whose backups nobody has ever restored — each has a signature, and after enough systems you read the signature the way a doctor reads a chart. You’re not claiming to know their product better than they do after an afternoon. You’re recognizing a pattern you’ve seen a hundred times, in its hundred-and-first costume.

I joined the AWS Global Accelerator team in 2023, and inside five days — before I’d been handed a single incident — I could see the shape of the whole problem. Two patterns did it. The first was an imperative tangle of overlapping control layers: so many puppets and so many puppet masters that the strings had wound around each other, everything pulling on everything else. That exact mess is what later inspired a whole piece on the why imperative orchestration always hits a ceiling. The second was what I call flash-photography observability: health wasn’t monitored continuously, it was checked — a human looked at dashboards and ran scripts by hand right before a deployment to confirm things looked healthy, except the picture drifted in the gap between the check and the moment the deploy actually began, so every release started by acting on a stale snapshot. Put those two together and the failure mode writes itself.

By the end of week one I could not only see the problem, I could predict the complaints — the specific ways it was hurting people day to day — without having received a single one of them yet. I knew the emails and the Slack threads were out there, because the patterns guarantee them: a tangled control plane and a strobe-light view of health produce exactly that catalog of pain, every time. When I said so, people were amazed and offended in roughly equal measure. Both reactions are the same misunderstanding — I wasn’t reading their situation so much as recognizing it.

It’s the martial-arts thing again. The solution set in any bounded domain is far smaller than the people inside it believe — a human has two arms, two legs, one head, joints that bend in limited ways. It feels like infinite possibility from the inside; from the outside, with enough reps, it’s a short list of constrained, predictable responses. Systems are the same. The costumes are endless; the bodies underneath are nearly all the same body. That isn’t arrogance and it isn’t magic — it’s what pattern recognition looks like once the pattern library is large enough, and it’s a great deal of what you’re actually hiring a senior engineer for.

Every Company Thinks It’s Special

Which runs headlong into a problem, because every company is convinced its product is uniquely intricate and precious, and therefore wants its senior infrastructure hire to know and care about every detail of it, and to know the exact current stack. The conviction is understandable, and often even correct about the product. It is still the wrong thing to ask of the SRE, and the reason is opportunity cost.

A senior SRE’s scarce resource is the narrow band of things only they can do — the architecture calls, the platform, the security posture, the cost levers of each cloud provider and bare metal approach, the recovery at 3 a.m. Spend that person’s finite time and energy re-deriving product knowledge the company already owns, in its developers and its testers, and you have quietly converted a senior SRE into an expensive, redundant junior product engineer. You bought the rare skill and put it to work on the common one. With infinite time and energy, sure, have everyone know everything. Nobody has infinite time and energy, and the company has already hired the people who hold the product knowledge. The SRE’s job is to bring what only they know, and do what only they can, and use it to multiply those people — the developers, the testers, the release managers — not to duplicate them.

I learned this from an unlikely place. Years ago, when I taught martial arts, one of my assistant instructors described his own role to me like this: his job, he said, was to do anything the two of us could both do — so that I was freed to do the things only I could do. It was generous, it was exactly right, and it has shaped how I think about senior work ever since. An SRE stands in that same relation to a product team, and the discipline is to run it on purpose: don’t pour your scarce, non-fungible capability into what the team can already do without you. Spend it where nothing else can cover it. Caring about the product is why you do that — you free yourself for the work only you can do precisely because you want the product to win.

None of which means product knowledge is wasted on an SRE — the opposite. Every bit of it you do have helps, often immensely: it sharpens your diagnosis, makes your advice concrete instead of generic, and turns a blind incident into an informed one. An SRE who understands the product is strictly better than one who doesn’t, all else equal. The point is only that this knowledge is bought with the same finite budget of time and energy as everything else on your plate — so you acquire it deliberately, where it pays off most, rather than being expected to hold all of it, on every product, all the time. Invest in product depth the way you invest in anything scarce: on purpose, and where the return is highest.

And notice the tell in that demand: companies want you to know the exact current stack. That gets the value exactly backwards — it’s the same error as hiring by keyword, scoring the token match on tool names instead of the mind that can pick up any tool in the category. Specific tools are the most perishable knowledge there is — they churn every couple of years, and any one of them can be learned in a week. What’s durable, and what actually takes years to build, is fluency in the domain the tools live in. Take observability. The field has a handful of commercial players — with a new one every quarter — sitting on top of a mature open-source set, and they are all solving the same small problem in slightly different wrappers. If you understand what observability is actually for — metrics, logs, and traces, and what a system has to expose to be operable at all — you can walk into any shop and judge their choice on the merits: what they’re observing, whether it’s sufficient, and whether they’re spending the money well. You don’t need to have used their particular tool. You need to know the domain it serves.

Here’s that as a concrete, slightly impolitic example. Tell me you run DataDog and I can tell you, before I see a single invoice, that you are spending a lot of money — that’s simply what it costs at scale. The question worth asking is whether the value matches the bill, because in my experience the same signal — the same metrics, logs, and traces, the same answers when the pager goes off — can usually be delivered for something like a tenth to a twelfth of the cost on a well-built stack. (Cheaper, note, not free; “well-built” is doing real work in that sentence.) So which do you actually want: the information you need at a price that makes sense, or the fact that you’re “using DataDog”? Those are not the same goal, and the distinction is enormous. Plenty of organizations, if they’re honest, are buying the brand and the comfort rather than the outcome — and it takes someone fluent in the domain, not a power user of the tool, to tell you which one you’re doing and hand you the cheaper path if you want it. (And I’m not even knocking DataDog here. They make a successful product, and they charge all that the market can bear - as they should.)

And there’s a sharper reason not to chase the person who “knows all the things,” beyond the economics: that person is a hazard. They do emerge — give any organization enough time and one individual accretes the whole picture in their head. At which point they are a load-bearing human, a single point of failure who is, by definition, irreplaceable — right up until the day they move on and take the context with them. Building your reliability around one omniscient person is not a strength you engineered; it’s a fuse you wired into your own critical path. The healthy structure is the opposite of omniscience: knowledge spread across the people who own each piece, clean boundaries between the roles — which is most of what the rest of this essay is about — and a senior SRE deployed as a force multiplier, not a substitute.

Three Hats, One Job

The three hats aren’t three jobs. They’re one job seen from three angles, and they feed each other.

The advisor. Before anything is built, SRE’s job is to help Dev design a product that delivers the maximum customer value for the minimum cost — where “cost” is money and time and effort and future maintenance and operational risk. This is where the deepest leverage is, because the cheapest incident is the one the architecture made impossible. Most of what I argue about fewer moving parts and simpler paths is advisory work: every component you talk a team out of adding is one you never have to keep alive.

The plumber. SRE builds the pipes — the deployment systems, the pipelines, the reconciling control loops that move bytes from where they’re written to where they run. A pipe has exactly one promise to keep: if you put X in one end, X comes out the other end, in the environment it was meant for, unaltered. Not almost X. Not X-if-the-weather’s-good. X. That promise is the entire value proposition of the plumbing, and everything I’ve written about reproducible, drift-free infrastructure is in service of it.

The firefighter. SRE carries the pager. We keep the systems up and secure, and when the alarm goes off we respond — ideally to a system that’s already healing itself, and never as the one irreplaceable human who has to be awake for recovery to happen. Firefighting is the most visible hat and the one you want to wear least.

And here’s the structure that ties them together: the better you are as an advisor and a plumber, the less you have to be a firefighter. Good advice removes the fires that get designed in. Good plumbing removes the fires that get deployed in. The pager going quiet is not luck — it’s the compounding dividend of the first two hats done well. An SRE org measured only on incident response is an org being paid to mop the floor while the tap runs. That’s honestly why I left Apple. There’s a world of difference between “running a fire department” and “making things fireproof”. When a management change moved the job from from the latter to the former, I knew it was no longer a place for me.

Which is exactly why you cannot build a product in isolation, throw it over the wall, and expect SRE to “monitor it” into reliability on the far side. Operability is not a coat of paint you roll on at the end — it is the same mistake as “we’ll add security later”, running in reverse, and it fails for the identical reason. Observability, reliability, security, and the ability to be run by someone who didn’t build it are properties of a system, which means consequences of how a thing is designed, not features you can staple onto a finished artifact. Monitoring a bad design doesn’t make it reliable; it just narrates, in real time and high resolution, exactly how it’s failing — often to a dark corner nobody observes, which is yet another waste of resources (like dumping everything to Splunk or Cloud Watch in the hopes it’ll “be useful to someone, someday”). You have to be in the room early — which is what the advisor hat is for.

The Pipe And The Payload

Now the distinction that resolves more turf wars than any other, if people would just understand it. SRE owns the pipe. Dev owns what goes through it.

The plumber does not need to know what the bytes mean. That isn’t indifference — I care a great deal that the right thing ships — it’s a separation of responsibility. My job is to guarantee that when Dev hands me version X to deploy, version X arrives, runs, and is reachable in the environment it was destined for. What X does once it’s running — whether the feature works, whether the math is right, whether the button is the correct shade of blue — is not the pipe’s concern. That’s the payload, and the payload is Dev’s.

Which gives you a diagnostic rule so simple it should be on a poster:

Problem with the pipe? Call the plumber. The deploy failed, the pipeline is red, an infrastructure component is down, X went in and X did not come out. That’s SRE.

Problem with what came out of the pipe? Call the author. It deployed fine, it’s running, and it crashes, or shows the wrong data, or the feature doesn’t work. X came out exactly as X, and X is broken. That’s Dev.

This is also the cleanest way to see why SRE, QA, and Dev are different jobs. QA and Dev are both entirely product-focused — Dev makes the thing, QA verifies the thing is good enough to ship to a customer. SRE is not product-focused at all. And QA sits downstream of both of us: they can’t test a product that Dev hasn’t built, in an environment that SRE hasn’t stood up and kept stable. QA depends on the payload being ready and the pipe being sound. Confusing these three roles — treating SRE as senior QA, or QA as junior Dev — is how organizations end up with nobody actually owning reliability.

Environments Have Owners

Here is where the clean model meets the messiest real-world failure, and it is almost always the same failure: an environment that nobody owns becomes everybody’s playground and nobody’s responsibility.

I have watched a QA environment get silently commandeered as a development sandbox — torn down and rebuilt on a whim to suit whatever someone was hacking on that week. It is completely untenable, and not because the developers are bad people. It’s untenable because QA cannot do their job — cannot certify that version X is good — if the ground is moving under them, if the thing they tested yesterday is not the thing running today, because someone else treated their environment as scratch space. You cannot measure anything against a moving baseline.

The fix is ownership, and it is not complicated:

• Every environment has exactly one owner. QA owns the QA environments. Full stop. They, and only they, decide what is deployed there.
• The owner deploys from a menu. There’s a list of components and their available versions; the owner picks the versions they want and deploys them. Predictable, repeatable, theirs.
• The two-call rule still holds. If the deployment breaks — the menu won’t dispense, the pipeline’s red — that’s SRE; call the plumber. If the deployed thing misbehaves — crashes, wrong output, broken feature — that’s Dev; call the author.

Notice SRE doesn’t own the QA environment in this model. SRE owns the pipe into it. QA owns what’s in it. That’s the separation working exactly as designed.

Unity Of Command

Production follows the identical rule, raised to the appropriate seriousness. There is a person / people / department — call it ‘release management’, call it whatever your org calls it — who holds the authority to deploy to production when a deploy is authorized. When that entity decides to ship version Z, they ship version Z. Their call. Their action.

The principle underneath this is old and military: unity of command. One person is responsible for an environment, and that same person holds the authority to act on it. Responsibility without authority is utter fiction at best, and a trap at worst — you cannot hold someone accountable for an outcome they were not empowered to control. So if Person X is responsible for Environment Y and wants Version Z, Person X is the one who deploys Version Z, and it is on them whether now was a good time to do it. That’s what “responsible” actually means. Not “attends the meeting.” Owns the outcome, holds the button.

SRE stands by — especially in the early phases, while the hand-off from the people who built it is still maturing — but SRE does not own the go/no-go on the payload. We own the pipe. If the deployment mechanism fails, that’s ours, and we come running. We’re on call 24x7 anyway. It’s part of the role. If you simply don’t like the results of Version Z, that’s not a pipe failure — you roll it back, which under decent GitOps is reverting a commit and letting the system reconcile, and then you either bother the author for a fix or schedule the next roll-forward. The rollback is a plumbing operation. The reason you wanted it is a payload judgment. Different hats, different owners, clean line between them.

Bytes Are Bytes

So that’s the job. Not “product.” The advice that keeps products cheap to build, the pipes that move their bytes without altering them, and the fire brigade that keeps the whole thing standing — with the roles drawn cleanly enough that when something goes wrong, everyone knows exactly whose phone should ring.

Keep the advice honest and the pipes clean, and the fires get rare. The pipes should be utterly transparent, and totally leak proof. Draw the lines between pipe and payload, and between one environment and its single owner, and the turf wars mostly evaporate. And stop being impressed or intimidated by any particular product’s mystique, because underneath every one of them — the trading engine, the payment system, the cat pictures — it’s the same job. Systems are systems. Bytes are bytes. The plumbing is supposed to be ordinary.