What Is Site Reliability Engineering (SRE)? Error Budgets, Toil and More

Every product team faces the same tug of war. The business wants new features shipped fast. The people keeping the system running want fewer changes, because changes are what break things. Without a shared way to decide, the loudest voice wins, and either customers get an unstable product or the roadmap stalls.

Site reliability engineering (SRE) offers a way out of that argument. It replaces opinions about “how reliable is reliable enough” with agreed numbers, and it gives teams clear rules for what happens when those numbers slip.

What SRE is

SRE began at Google. Benjamin Treynor Sloss, who joined Google in 2003 to run a small production team that grew into its SRE organization, famously describes it as “what happens when you ask a software engineer to design an operations team.”

In practice, that means:

• Reliability is measured from the user’s point of view, not by whether servers are switched on.
• Targets for reliability are explicit and agreed with the business.
• Repetitive manual work is treated as a problem to engineer away.
• Incidents are studied to fix systems, not to blame people.

SRE is both a job title and a set of practices. You can adopt the practices long before you hire anyone called an SRE.

SRE vs DevOps

The 2 ideas are closely related. DevOps is a broad culture of shared ownership between development and operations. SRE is a specific, opinionated way to put much of that culture into practice, with concrete tools such as service level objectives and error budgets.

A common shorthand is that SRE is one way to implement DevOps. DevOps says “developers and operators should share responsibility for reliability.” SRE says “here is how to measure reliability, how much is enough and what to do when you fall short.”

Reliability targets: SLIs and SLOs

SRE starts by defining what “working” means.

• A service level indicator (SLI) is a measurement of user experience, such as the share of requests that succeed or the share served in under 300 milliseconds.
• A service level objective (SLO) is the target for that measurement over a period, such as “99.9% of checkout requests succeed over 30 days.”

The key SRE insight is that 100% is the wrong target. Users cannot tell the difference between very high reliability and perfect reliability, because their own Wi-Fi, phones and internet providers fail more often than that. Chasing 100% costs a fortune and freezes all change. Our guide to SLA vs SLO vs SLI walks through setting these with a worked example.

Error budgets: the heart of SRE

Google’s SRE book defines the error budget simply: one minus the availability target. If your SLO is 99.9%, your error budget is the remaining 0.1%.

That 0.1% is not failure to be ashamed of. It is a budget the team is allowed to spend on things that carry risk, such as releasing features, running experiments or doing infrastructure upgrades.

Here is what different targets allow over a 30-day month:

For request-based SLOs, the same idea applies to failed requests. Google’s SRE workbook gives the example of a 99.9% SLO allowing 1,000 errors out of 1 million requests over 4 weeks.

What an error budget policy does

The budget only works if everyone agrees in advance what happens when it runs out. Google’s example error budget policy says that if a service exceeds its budget over the previous 4 weeks, the team halts changes and releases except urgent bug fixes and security fixes until the service is back within its SLO.

The policy is not a punishment. It gives teams permission to stop shipping features and focus on reliability, without needing to win an argument every time. And when the budget is healthy, it gives them permission to move fast.

Key takeaway: An error budget turns “should we ship or should we stabilize?” into a data question. Budget left: ship. Budget gone: fix reliability first. Agreeing on that rule before an incident removes most of the conflict.

Toil: the work SRE tries to eliminate

Google’s SRE book defines toil as operational work that is manual, repetitive, automatable, tactical (interrupt-driven), lacking enduring value, and that grows linearly as the service grows. Examples include:

• Manually restarting a service every time it runs out of memory
• Copying data between systems by hand each week
• Resetting user permissions through a ticket queue
• Clicking through the same deployment checklist for every release

Not all operations work is toil. Investigating a brand-new problem or designing a better architecture requires judgment. Toil is the repeatable, low-value stuff that would double if your customers doubled.

Google’s advertised goal is to keep toil below 50% of each SRE’s time, with the rest spent on engineering that reduces future toil or improves the service. The book notes that its internal surveys have averaged around 33%. The exact number matters less than the principle: if a team spends all its time keeping the lights on, nobody is making the system better.

Other core SRE practices

Monitoring and alerting tied to users

SRE teams alert on symptoms users feel, such as rising errors or slow responses, rather than on every CPU spike. Every page should be actionable. Good telemetry makes this possible; see our comparison of observability vs monitoring.

Incident response

Clear on-call rotations, defined roles during an incident (someone leads, someone communicates, others investigate), and a focus on restoring service first and finding root causes later.

Blameless postmortems

After significant incidents, the team writes up what happened, why, and what will change. Google’s SRE book describes a blame-free postmortem culture. People who feel safe share the full story, which is the only way to fix the underlying weakness.

Capacity planning and release engineering

Forecasting demand, testing how systems behave under load, and making releases gradual and reversible, for example with canary releases that go to a small share of users first. These connect closely to designing for high availability.

How SRE applies outside big tech

You do not need Google’s scale, or a dedicated SRE team, to benefit. Here is how the practices translate:

For many mid-sized companies, the most practical model is “embedded” SRE: product teams own their services and SLOs, while a small group of reliability specialists provides tools, coaching and shared platforms. That overlaps heavily with platform engineering.

Where to start

You can start SRE practices this month with these 6 steps:

1. Pick 1 important user journey, such as login or checkout.
2. Define 1 or 2 SLIs for it, such as success rate and response time, using data you already collect.
3. Set a realistic SLO based on recent performance, not on what sounds impressive.
4. Write a 1-paragraph error budget policy and get the product owner to agree to it.
5. Start a toil list and automate the most painful item first.
6. Write a blameless postmortem for the next customer-affecting incident and track its action items to completion.

Review SLO performance monthly, and adjust the target if it turns out to be too strict or too loose.

The bottom line

SRE treats reliability as an engineering problem with measurable goals. SLOs define how reliable a service needs to be, error budgets decide when to ship and when to stabilize, toil limits protect time for improvement, and blameless postmortems make every incident a lesson. None of this requires big-tech scale. Start with 1 journey, 1 SLO and 1 agreed rule, and build from there.

Sources

• Google SRE book: Introduction — origins of SRE, 50% operations cap, error budget definition, blameless postmortems
• Google SRE book: Eliminating Toil — definition and characteristics of toil, 50% goal
• Google SRE workbook: Example error budget policy — what happens when an error budget is exhausted
• OpenTelemetry: Observability primer — definitions of SLIs and SLOs