Thankfully, since Site Reliability Engineering developed internally at Google and only recently has started to make its way into the broader community, it is fairly well-defined. What isn't, though, is web operations (or "systems administration" - as an example of the lack of clarity, you use both in your question). It's difficult to discuss the differences between two things when you're not altogether sure what one of them is.
But I'm an adventurous fellow, so I'll give it a shot.
In very traditional shops, developers and sysadmins are very siloed from each other. The devs build an app, then consider their job complete as soon as their code has been committed. The sysadmins take the build artifacts (which may be just the code, if it's an interpreted language) and deploy it to production servers. It's the sysadmins' job to keep the application running smoothly, and in general manage the production environment. However, often performance problems come from architecture issues in the app; the sysadmins don't have the programming knowledge to know what the app is doing, and the developers don't know how the app acts in the production topology with production traffic, so no one is equipped by themselves to solve the problem.
Additionally, the developers are usually judged on how quickly they can produce new features, while the sysadmins are judged on how infrequent the app breaks in production. Since change is one of the leading causes of breakage, this puts the two departments at odds with each other - an old rivalry that hurts the business and the people involved.
At some point, some developer-centric companies got so annoyed at this that they began practicing "NoOps" - they eliminated their operations departments and the perceived roadblocks that came with them. In reality, this meant that developers took on operations roles, but maintained their old titles.
In a discussion surrounding NoOps, John Allspaw, then VP of Technical Operations at Etsy and an editor of the well-respected Web Operations book, defined roles at Etsy this way:
Etsy Operations is responsible for:
- Responding to outages, takes on-call
- Alerting systems thresholding, design
- Architecture design and review
- Building metrics collection
- Application configuration
- Infrastructure buildout/management
Etsy Development is responsible for:
- Responding to outages, takes on-call
- Alerting systems thresholding, design
- Architecture design and review
- Building metrics collection
- Application configuration
- Shipping public-facing code
Neither of those lists are comprehensive, I'm sure I'm missing
something there. While Etsy Ops has made production-facing application
changes, they're few but real (and sometimes quite deep). While Etsy
Dev makes Chef changes, they're few but real. If there's so much
overlap in responsibilities, why the difference, you might ask? Domain
expertise and background. Not many Devs have deep knowledge of how TCP
slow start works, but Ops does. Not many Ops have a comprehensive
knowledge of sorting or relevancy algorithms, but Dev does. Ops has
years of experience in forecasting resource usage quickly with
acceptable accuracy, Dev doesn't. Dev might not be aware of the pros
and cons of distributing workload options across all layers1-7, maybe
only just at 7, Ops does. Entity-relationship modeling may come
natural to a developer, it may not to ops. In the end, they both
discover solutions to various forms of Byzantine failure scenarios and
resilience patterns, at all tiers and layers.
In his world, developers and ops engineers had very similar high-level skill sets and responsibilities; where they differed was in their expertise. Their differing specialties encouraged them to work together to solve problems, and their common base-level skills gave them a language in which to do that.
This is generally the definition of web operations that I land on for most cases. So it's the one we're going to continue along with.
So then, what is Site Reliability Engineering?
The Google SRE book opens with a definition of SRE... and then another one... and then spends a chapter continuing to define the role and an entire book covering the specifics. Even when developed in one organization, it seems that it's difficult to condense the job down to one single agreed-upon definition.
To start with, we need to walk back to 2003, when Ben Traynor joined Google and founded what came to be the first Site Reliability Engineering team. Recall that a few paragraphs ago we were in the early 2010s; but in 2003, the industry was still pretty set on the sysadmin/developer divide as the natural way of things. So when Ben says that SRE was what would happen if a software engineer created an operations team, this was a much more radical melding of the two worlds than it appears now.
The definition given in the preface emphasizes each of the three words individually:
- Engineering - the use of computer science and engineering concepts to solve problems
- Reliability - a focus on making systems more scalable, more reliable, and more efficient
- Service - the later evolution of "site", emphasizing that SREs are responsible for networked services
The introduction chapter lists the tenets of Site Reliability Engineering as:
- Ensuring a durable focus on engineering - taking pre-emptive action to avoid frequent pages and other "toil"
- Persuing maximum change velocity without violating a service's SLO - a subject that can easily have its own several-hundred word answer, but roughly summarized as helping developers make changes, as long as they don't cause too many issues
- Monitoring - automatic alerts when things go wrong
- Emergency response - fixing things when they're broken
- Change management
- Capacity planning
- Provisioning
- Efficiency and performance - ensuring that a service performs at an expected level - bottlenecking hurts users, but excess capacity costs money
I'd categorize Site Reliability Engineering as a specialized subset of modern Web Operations. An SRE organization focuses heavily on automating everything, to a degree that is only cost-effective in fairly large companies. Ideas like error budgets can only work when your service has many, many requests, as otherwise you lose granularity (for a smaller service, a particular error could affect 0-20% of your requests, depending on the minute). Related areas like security are absent from the SRE definition because companies large enough to have true SRE teams have dedicated teams for security.
The SRE program, as defined by Google, is web ops developed for the specific needs of Google, and not necessarily applicable elsewhere.
However, Site Reliability Engineering has been expanding in broader industry use recently. My current job title is an SRE, even though I work at a much smaller company and my job description fits pretty well with John Allspaw's 2012 Etsy web ops definition. My theory is that we've been progressing through titles as a shorthand for espousing the evolution of a single field:
- We started as sysadmins.
- Then as web sites became more of a "thing", job postings started to refer to web operations engineers to distinguish sysadmins who specialized in the web from those who also handled general office IT.
- Then DevOps was supposed to separate out those who were comfortable using programming to reduce their web ops workload.
- But as DevOps got muddled by the lack of a clear definition, we adopted Site Reliability Engineering to specify that we're looking for people who are on-call supporting production services.
So what is the difference between a sysadmin and an SRE? The year in which they received their title. What is the difference between traditional operations and site reliability engineering? SRE is merely the current incarnation of ops, using new tools (hello, containers!) and, as networked programs continue to become more larger and more important, an increased focus on practices that allow one engineer to do more.
