Modern applications require modern operations, and modern

operations requires a new definition for ownership that

most classical organizations must provide.

Today I continue my discussion on modern ops with Beth Long.

Are you ready? Let's go.

This is the Modern Digital Business Podcast, the technical

leader's guide to modernizing your applications and digital business.

Whether you're a business technology leader or a small business

innovator, keeping up with the digital business revolution is a

must. Here to help make it easier with actionable insights and

recommendations, as well as thoughtful interviews with industry experts.

Lee Atchison in this episode of Modern Digital Business,

I continue my conversation on Modern Operations with my good

friend SRE engineer and operations manager Beth

Long. This conversation, which focuses on service

ownership and measurement, is a continuation of our

conversation on SLAs in Modern Applications.

In a previous episode, we talked about Stosa, and this fits very much into

that idea is the idea that how you organize

your teams so that each team has a certain

set of responsibilities. We won't go into all the details of Stosa, but bottom

line is ownership is critical to the

Stosa model. Ownership is critical towards all DevOps

models. If you own a service, you're responsible

for how that service performs, because other teams are depending on

you to perform what those performance,

what it means to perform. The definition of what it

means to perform is what an SLA is all about.

Yeah. So what does a good SLA look like?

Beth that's a great question. Let's get to the measurement.

It does get into measurement.

That is always a hard question to answer.

If you look at the textbook

discussions of Slis and SLOs and

SLAs in particular, you'll often see references

to a lot of the things that are measurable. So you'll

have your golden signals of error, rate,

latency, saturation. So you have

these things that allow you to say, okay,

we're going to tolerate this many errors,

or this many of this type of error, this much

latency. But all of that is kind of trying

to distill down the customer experience

into these things that can be measured and

put on a dashboard. The term smart goals comes

to mind, right. That I think, is a good

measure. I know the idea of smart goals really hasn't been tied to

SLAs too closely, but I think there's a lot of similarities here. So

smart goals are five specific criteria. They're specific

measurable, attainable,

relevant, and time bound. So

now I think all five of those actually apply here

as well. Too right. When you create your SLAs,

they have to be specific. You can't say, yeah, we'll meet your

needs. That's not a good experience. But

in my mind, a good measurement is something

like, we will maintain

five milliseconds latency on average

for 90% of all requests that come in.

And I also like to put in an assuming.

Assuming you meet these criteria, such

as amount of traffic, the traffic load is

less than X, number of requests permitted or whatever the

criteria is. So in my mind, it's a specific

measurement with bounds for what that

means. Under assumptions. And these are the

assumptions. So something like five

milliseconds average latency for 90% of requests,

assuming the request rate is less than

5000 requests per second,

and assuming both those things occur. And you could also have assuming the

request rate is at least 100 /second because

caching can warming caches can have an effect there too. And things

like that. So you can have both bounded numbers. There

something like that is a very specific it's specific. It's

measurable. All of those numbers I specified are all things you could

measure. They're something you could see. Specific

measurable. You want to make sure they're attainable within

the service. That's your responsibility as the owner of a

service. If another team says, I need

this level of performance, it is your responsibility as the owner. Before

you accept that is to say yes, I can do that. So they have to

be attainable to you. And this actually gets at something very

important in implementing these sorts of things, which is to make sure that

you are starting with goals that are near what you're currently

actually doing and step your way towards

improvement instead of setting impossible goals. And then

punishing teams when they don't achieve something that was so far outside of

their ability. Oh absolutely there's two things that make a

goal bad. One is when the goal is so easy that

it's irrelevant. The other one is when it's so difficult that it's never

set never hit. You should set

goals that are in the case of

SLAs, your goal needs to hit the

SLA 100% of the time, but it

can't be three times what you are ever

going to see. Because giving you plenty of room

to have all sorts of problems because then that doesn't make it relevant to

the consumer of the goal. They need something better than that. That's

where the attainable and that's where relevant comes in. And

relevant is so important because it's so tempting. This is where when

it's the engineers that set those goals those

objectives in isolation you tend to get things that are

measurable and specific and

attainable but not relevant, right? I will

guarantee my service will have a latency of less than

37 seconds for this simple request guaranteed I

can promise you that, right? And the consumer will say

well I'm sorry I need ten milliseconds 37 seconds doesn't

that sounds an absurd number but you and I have both

heard numbers like that right? Where they're so far out of bounds they're

totally irrelevant, they're not worth even discussing.

Yes and a sneakier example would be something

like setting an objective

around how your infrastructure is behaving in ways that

don't translate directly to

the benefit to the customer. If you own a web

service that is serving directly to end

users. And your primary measures of

system health are around

CPU and I

O. Well, those might tell you something about what's

happening, but they are not directly

relevant to the customer. You need to have those on your dashboards for when

you're troubleshooting, when there is a problem, but that's not indicating the health

of the system. Right. So specific measurable

attainable relevant. So relevant

means the consumer of your service has to find them

to be useful. Attainable means that you as provider

of the service, need to be able to meet them. Measurable

means need to be measurable specific.

They can't be general purpose and ambiguous. They have to

be very specific. So all those make sense. Does time bound really apply

here? I think it does, but in the sense

that when you're setting these agreements,

you tend to say, this is my commitment, and

you tend to measure over a span of time and

there is a sense of the clock getting reset.

That's true. We'll handle this much traffic

over this period of time. You're right. That's a form of time bound. I think

when you talk about smart goals, they're really talking about the time

when you'll accomplish the goal. And what we're saying

is the time you accomplish the goal is now. It's

not really a goal, it's an agreement as far

as it's a habit. Rather than a habit.

And that's actually a good point. These aren't goals.

I'm going to try to make this no, this is what you're

going to be performing to and you can change them and improve them over time.

You can have a goal that says I'm going to improve my

SLA over time and make

my SLA twice as good by the state.

That's a perfectly fine goal. But that's what a goal is

versus an SLA, which says your SLA is

something like five millisecond latency

with less than 10,000 requests. And you can say, that's

great, I have a goal to make. It a two millisecond latency

with 5000 requests, and by this time next

quarter, and at that point in time then your SLA is now two

milliseconds. But the SLA is what it is and

what you're agreeing to, committing to now, it's a

failure if you don't meet it right

now. As opposed to a goal, which is what you're striving towards.

Yeah, towards completing something. Right.

One anecdote. That a well known anecdote that I

think is interesting to talk about. Here is

the example that Google gave. This is in the SRE

book of actually

overshooting and having a service that

was too reliable. I can't remember which service it was

off the top of my head, but they actually had a service that they did

not want to guarantee 100% uptime, but they ended up

getting over delivering on quality for a while.

And when that service did fail,

users were incensed because there was sort of this

implicit SLA. Well, it's been performing so well.

And so what I love about that story is that they ended

up deliberately introducing failures into the system

so that users would not become accustomed to too high of

a performance level. And what this

underscores is how much this is about

ultimately the experience of whatever person it is

that needs to use your service. This is not a purely

technical problem. This is very much about understanding

how your system can be maximally healthy

and maximally serve

whoever it is that's using it. So I love that story. I

didn't know that story before, but it plays very well into

the Netflix Chaos Monkey approach to testing. And that is

the idea that the way you ensure

your system as a whole keeps performing is you keep causing it to fail on

a regular basis to make sure that you can handle those failures.

So what the Chaos Monkey does, and I'm sure at some point in time we're

going to do an episode on Chaos Monkey. Matter of fact, we should add it

to our list. What Chaos Monkey is all about is the idea

that you intentionally insert faults into your system

at irregular times so that you can

verify that the

response your application is supposed to have to self heal around the

problems that are occurring can be tested to make sure they

occur. Now, you don't do this in staging, you don't do this in

dev, you do it in production. But you do it in production

during times when people are around. So that if

it does cause a real problem, if you turn off the service

and that causes a real problem and customers are really affected,

everyone's on board and you can solve the problem right away as opposed

to the exact same thing happening by happen chance at

02 00:12:56

00 in the morning when everyone's drowsy and sleeping and

02 00:13:00

not knowing what's going on. You can address the problem right there

02 00:13:04

right then as opposed to later on. And the other

02 00:13:08

thing it helps with is this problem that you were addressing which

02 00:13:11

is getting too

02 00:13:15

used to things working. So if you deploy a new

02 00:13:19

change and let's say I own a service, and one of the

02 00:13:22

things I'm doing service A and I call Service B and I need to

02 00:13:26

expect a service B will fail occasionally, well, I'm going to write

02 00:13:30

code into Service A to do different things. If Service B

02 00:13:33

doesn't work well, what if I introduce an error in that

02 00:13:37

code that I'm not aware of and then I deploy my

02 00:13:41

code? Well it's going to function, it's going to work,

02 00:13:44

everything's going to be fine until Service B fails and Service A is also going

02 00:13:48

to fail. But if Service B is regularly

02 00:13:52

failing, you're going to notice that a

02 00:13:56

lot sooner, perhaps immediately after deployment,

02 00:13:59

and you're going to be able to fix that problem, roll it back if necessary,

02 00:14:03

or roll forward with a fix to it to

02 00:14:06

get the situation resolved. The more

02 00:14:10

chaotic you put code into, the more stable the

02 00:14:13

code is going to be. It's a weird thought

02 00:14:17

to think that way, but the more chaotic a system, the

02 00:14:21

more stable the code that's in that system behaves

02 00:14:25

over the long term. I'm so glad you bring this up. And what I

02 00:14:28

love about this is that we're really touching

02 00:14:32

on similar themes in different contexts

02 00:14:35

because both Chaos Engineering and the DevOps

02 00:14:39

approach are really about

02 00:14:43

understanding that we don't just have a technical system,

02 00:14:46

we have a sociotechnical system. We have this intertwined human and

02 00:14:50

technology system. And so with DevOps, one

02 00:14:54

of the advantages of DevOps is that it changes the behavior of the people

02 00:14:58

who are creating the system itself. Because

02 00:15:01

again, if you're going to deploy code

02 00:15:05

and you know that if something goes wrong, it's going to wake up that person

02 00:15:08

over there that you don't even know.

02 00:15:12

You just build your services differently.

02 00:15:16

You're not as rigorous as

02 00:15:19

when you know you're going to be the one woken up at 02:00 A.m.. And

02 00:15:23

similarly with chaos engineering, if you know that

02 00:15:26

service B is going to fail absolutely in the coming

02 00:15:30

week, you're just going to be like, well, I may as well deal with this

02 00:15:34

now. As opposed to like, well, I'm under deadline. Service b is usually

02 00:15:37

stable. I'm just going to run the risk and we'll deal with it later.

02 00:15:41

So it really drives behavior inserted into

02 00:15:44

systems. Right. And the other thing

02 00:15:48

I love about how you kind of unpacked chaos

02 00:15:52

Engineering is it does work

02 00:15:56

on this very counterintuitive idea that you should be

02 00:15:59

running towards incidents and problems

02 00:16:03

instead of running away from them, you should embrace them.

02 00:16:06

And that will actually help you, as you said,

02 00:16:10

make the system more stable because you

02 00:16:13

are proactively encountering those issues rather

02 00:16:17

than letting them come to you. Yeah, that's absolutely great.

02 00:16:21

That's great. Yeah, you're right.

02 00:16:25

We're not talking about coding. We're talking about social systems here. We're

02 00:16:29

talking about systems of people that happen to include

02 00:16:32

code as opposed to systems of code. And that the vast

02 00:16:36

majority of incidents that happen have a

02 00:16:40

socio component to. It not just a code

02 00:16:43

problem. It's someone who said this is good

02 00:16:47

enough or someone who didn't spend the time

02 00:16:50

to think about whether or not it would be good enough or not and

02 00:16:54

therefore missed something. Right. And these aren't bad

02 00:16:58

people doing bad things. These are good people that are making mistakes that

02 00:17:01

are caused by the environment of which they're

02 00:17:05

working. And that's why environment and

02 00:17:09

systems of people and how they're structured and how they're organized

02 00:17:12

is so important. I keep hearing people

02 00:17:16

say how you

02 00:17:20

organize your companies irrelevant. Right? It shouldn't matter.

02 00:17:24

Nothing could be further from the truth. It matters the

02 00:17:28

way you organize a company.

02 00:17:32

I hate saying it this way because I don't always work in this one, but

02 00:17:34

how clean your desk is is a good indication of how clean the system

02 00:17:38

is. And I don't mean that literally because I've had dirty

02 00:17:42

desks too, but it really is a good indication

02 00:17:45

here. It's how well you organize your

02 00:17:49

environment, how well you organize your team,

02 00:17:52

how well you organize your organization,

02 00:17:57

gives an indication for how well you're going to perform as a

02 00:18:00

company from the standpoint. Yes,

02 00:18:06

when we look at the realm of incidents which

02 00:18:10

are messy and frustrating and scary and expensive,

02 00:18:14

and every tech company knows that they

02 00:18:17

are probably one really

02 00:18:21

bad incident away from going out of

02 00:18:24

business, every company knows

02 00:18:28

that there's that really bad

02 00:18:32

thing that could collapse the whole

02 00:18:36

structure. And so incidents are really high

02 00:18:39

stakes, but

02 00:18:43

that drives us to look for certainty and look for clarity. And

02 00:18:46

so we look to a lot of these things that people have been talking

02 00:18:50

about for years around incident metrics. So you've

02 00:18:54

got your mean time metrics, what's your mean time to resolution

02 00:18:58

or your mean time between failure and it's? This attempt

02 00:19:01

to bring some kind of order

02 00:19:06

and sense to this very scary and chaotic world

02 00:19:10

of incidents. But so

02 00:19:13

many of those, what are now often being called shallow

02 00:19:17

incident metrics end up giving short

02 00:19:21

shrift to what we were just talking about, which is that

02 00:19:25

this is a very complex system.

02 00:19:29

The technology itself is very complex. The

02 00:19:33

sociotechnical system is complex.

02 00:19:36

We're trying to kind of get a handle on

02 00:19:40

how do you surface those complexities and make them

02 00:19:43

intelligible and make them sensible without

02 00:19:47

falling back to some of these shallow metrics. That

02 00:19:52

Niall Murphy, who was back to SRE, one of the authors of

02 00:19:55

the original SRE book, had a paper out recently where he kind

02 00:19:59

of unpacks the ways that these mean time

02 00:20:03

and other shallow metrics aren't

02 00:20:06

statistically meaningful and

02 00:20:09

aren't helping us make good decisions

02 00:20:13

in the wake of these incidents. And so much of what we're talking

02 00:20:17

about is SLAs are how do

02 00:20:20

you make decisions about what work you're going to do and how

02 00:20:24

much you invest in reliability versus new features

02 00:20:28

and incident follow up is so much about what

02 00:20:31

decisions do we make based on what we learned

02 00:20:35

in this event. Yeah, you add a whole new dimension

02 00:20:39

here to the metric discussion here, because

02 00:20:43

it's so easy to think about metrics along the line of

02 00:20:46

how we're performing and when we don't perform, it's a failure

02 00:20:50

oops, but there's a lot of data in the

02 00:20:53

Oops, and you're right. Things like meantime

02 00:20:57

to detect and meantime to resolution. And those are important,

02 00:21:01

but they're very superficial compared to the depth that you

02 00:21:05

can get. And I'm not talking about Joe's team caused five

02 00:21:08

incidents last week. That's a problem for Joe. I'm not talking about

02 00:21:12

that. I'm talking about the

02 00:21:15

undercovering,

02 00:21:18

the sophisticated connection between

02 00:21:22

things that can cause problems to occur.

02 00:21:28

Thank you for tuning in to Modern Digital Business. This

02 00:21:31

podcast exists because of the support of you, my listeners.

02 00:21:35

If you enjoy what you hear, will you please leave a review on Apple

02 00:21:38

podcasts or directly on our website at MDB

02 00:21:42

FM slash Reviews If you'd like to suggest a topic for an

02 00:21:46

episode or you are interested in becoming a guest, please contact

02 00:21:50

me directly by sending me a message at MDB FM

02 00:21:53

contact. And if you'd like to record a quick question or

02 00:21:57

comment, click the microphone icon in the lower right hand corner of our

02 00:22:00

website. Your recording might be featured on a future

02 00:22:04

episode. To make sure you get every new episode when they become

02 00:22:08

available, click subscribe in your favorite podcast player

02 00:22:11

or check out our website at MDB FM. If

02 00:22:15

you want to learn more from me, then check out one of my books, courses

02 00:22:18

or articles by going to Lee Atchison.com and

02 00:22:22

all of these links are included in the show. Notes thank you for

02 00:22:26

listening and welcome to the world of the modern digital business.