W. Curtis Preston: If your organization was hit with a disaster or cyber attack

tomorrow, would each person know exactly what their responsibilities were?

If your answer is not an unequivocal yes, then you need a Dr.

Runbook.

What is the Dr.

Runbook?

Well, you're in the right place.

It's the culmination of all the planning and designing that you did

when putting together your DR system.

When a disaster strikes, you should simply be able to pull it out and follow it.

Every person in the org should know what their role is and who

they should be communicating with.

Our goal here is to turn unappreciated backup admins into cyber recovery heroes.

So let's talk about how to make a runbook and what it should look like.

This is the backup Wrap up.

Welcome to the show.

I'm your host, w Curtis Preston.

And I have with me my celebration consultant Prasanna.

Malaiyandi.

How's it going?

Prasanna,

I'm doing well.

What are we celebrating?

W. Curtis Preston: think, you know, we're celebrating my, my big

new contract that I landed today

Woo.

W. Curtis Preston: Very excited about that.

Um, we.

Thanks.

Uh, I'm, I'm gonna be working a lot with a new company and helping them to spread

their word about what they do and, we'll talk more about that in later days,

but today is the day that I actually heard that I got the contract, so I'm

Woo hoo.

W. Curtis Preston: What do you, what do you think I should do?

Uh, I think you should go for Korean barbecue.

W. Curtis Preston: By myself without the rest of the family.

Yeah, yeah.

No, you're right.

That's okay.

Fine.

Then how about going and watching a movie?

W. Curtis Preston: Oh yeah, maybe I'll go, maybe I'll go do just that.

I'll do that.

Um, yeah.

Well 'cause, 'cause you know, you know, my wife's down in San

Diego training to be a trainer.

I.

To train poll workers.

So she's down in San Diego with her mom, uh, and my, the rest of

my family is spread to the wind.

So yeah, if I went to Korean barbecue, I would be going there all by myself

and they would never speak to me again.

Given, given

Just don't tell them.

W. Curtis Preston: thing to Do What's that?

Just don't tell them.

Do you think they listen to the podcast?

Do you think they know

W. Curtis Preston: I just don't tell.

I would feel guilty, uh, but

I wouldn't feel guilty going, uh, to go see a movie because I do have the Regal

Unlimited Pass so I can go see a movie.

Do it.

W. Curtis Preston: So, uh, we've been talking about DR for a while now,

and, you know, once you've decided, I mean, we talked so much about it.

I, I think this is our.

Sixth episode about, you know, basically putting together a DR plan.

You've got to decide what's in it.

You've got to decide what you know, uh, what your RTO and

RPO are for each application.

You need to decide whether or not you're going to do this yourself

or you're gonna use a service.

Where are you gonna do a cold site, hot site, warm site?

You can use the cloud, you can use on-prem stuff.

You gotta make all those decisions and hopefully you put a lot of

planning into this and hopefully you got a lot of input from the business

that this was not an IT driven.

I mean, it could be an IT driven, I suppose, but just really, you know,

I hope that you got input from all of the stakeholders, all of the.

People that will be impacted by this Dr.

Plan.

And you put it all together to make all of them as happy as possible, right?

you're gonna be the knight and shining armor right?

When something happens.

W. Curtis Preston: knight and shining armor.

The uh, and this will date me, but here I come to save the day.

No one's gonna get that reference.

Anyway.

Did you get that reference?

Mighty Mouse.

W. Curtis Preston: Mighty Mouse.

Yeah.

Good job.

Good job.

All right.

Um, so I, so, so when we've done all those decisions, when, when

you have an actual disaster, you should be able to pull out what.

I was just gonna say, you don't run around like

a chicken one with its head cut off scrambling and everyone just being

like, panic, panic, panic, panic.

W. Curtis Preston: Yeah, that is not what you're supposed to do.

Um, what are you gonna pull out?

Prasanna.

Well, what you're supposed to do is pull out your Dr.

Runbook, which has been created ahead of time.

Everyone has read, everyone knows about it, and you just

follow what it says in the Dr.

Runbook and you're good to go.

W. Curtis Preston: Yeah.

And, and I will say that, you know, we're talking mainly

here about disaster recovery.

A Dr.

Runbook fits within a cyber recovery runbook.

This is part of it.

This is the actual.

Putting everything back together.

Part if what we're talking about is recovering from a cyber attack,

you do have to address that first.

And, uh, we should talk about that.

We, we will talk about that in other episodes, but at this point, basically you

have decided, you know, you're going to restore your organization and everybody

should basically know just exactly.

Where they're going, what they're doing, what they're not doing, who they're

telling, who they're not telling.

Um, because you know, Dr.

You know, an actual disaster recovery does not happen in a vacuum.

What do you think about that?

Yeah, no, it, yeah, it doesn't happen in a vacuum.

There's usually other causes, and then it's also a very,

very, very high stress event.

Right, and so you wanna make sure that you're not thinking on the fly,

trying to figure out what to do, that everything has sort of been thought

ahead of time, so you just can execute.

It's like a doctor, right?

And yes, I know there will be cases that you need to worry

about, but a doctor, right?

They go to school, they do the procedures, they know what to do, and so they are

able to say, okay, I'm doing this, this, and this every single time.

W. Curtis Preston: Yeah, actually I, I may have a better

example to that and that was.

You know, I, I know I've told you the story, but I don't think I've told

the listeners, and that is the time that we lost an engine on a plane

when I was in flight on an MD 80.

And for those of you that don't know, an MD 80 only has two engines.

Right?

So we were on initial ascent and the um.

Uh, we, we lost an engine.

And for those of you that have never experienced that, which is probably

most of you, if not all of you, uh, if you've ever been in a car where

you've accidentally and you're, and you're sort of accelerating and

then you slipped out of gear, right?

Like, you, you, your leg bumped the, you know, the, uh, the stick shift or

the, uh, what, what do you call that?

The, the, the gear shift?

I don't know.

The,

you know, in an automatic.

Whatever.

Basically, suddenly you find yourself in neutral.

There's that sort of lunge that happens, right?

And, um, uh, if you've never felt that, then I, I don't

know what to, what to tell you.

But, but imagine that, but up in the air.

So suddenly you're luing, you're lunging forward, and you're diving, you're,

you're, you know, you're, you're rapidly.

Um, uh, losing altitude and, uh, flight attendants are running back and forth.

They're doing their thing.

We were on initial ascent, so everybody was still strapped in and everything.

And, um, the, and then the, the.

The plane did this massive like banking maneuver

to take us back and it was definitely one of those, you know,

Yep.

W. Curtis Preston: and they, they, you know, 'cause they got us back.

They, they basically, as far as I know, they turned that plane around as

quickly, whatever the tightest turning radius that you could possibly do.

I'm sure that there were massive alerts that went off in the, in the tower, right?

The, the air traffic controller tower.

They cleared the runways, we went back and we landed, and then, you know, and

then the firetrucks came out and uh, and that's when everybody freaked out, right?

Once we were down and then the firetrucks came out.

Prior to that, everybody just sat there and white knuckled it and, um, you know,

and just, you know, just tried not to cry.

My my point of all of this was that pilot had done that hundreds of times

already, because he goes to a simulator.

And, uh, and trains for that.

And they had trained for all of this.

The tower knew what to do.

Why?

Because they had a runbook.

They

followed to the letter because they've got lives in their hands.

They knew it.

They, they, they had a, you know, a checklist that they go down when you

have to make an emergency landing with a plane that's lost an engine.

That's just a, you know, they, everybody just went and did

everything that they needed to do.

Which I will say apparently didn't involve letting us know

what the hell was happening.

That was,

that was not their, that was not their concern.

Their concern

you had, yeah, because there was nothing

you as passengers could do.

W. Curtis Preston: There was one thing we could do.

You know what it was?

Ah,

Yeah.

W. Curtis Preston: this is basically, um, all right, so, yeah, and, and so

that's why you want to have a runbook.

You wanna have a runbook, you want to train, you want to practice, and you want

this stuff to be basically muscle memory.

And that involves who you notify, how you notify, whether or not, you

know, depending on who you are as an organization, whether or not you have

to notify the media, whether or not you have to notify stockholders, you know,

all of the, all of these things, right?

Um, uh, law enforcement depending on why you're, why you're,

you know, uh, doing this.

Yep.

Or

your insurance

W. Curtis Preston: a list or your, yeah.

Your insurance company.

Your cyber insurance

Yeah.

W. Curtis Preston: Exactly.

Um, so I have a list I made, uh, I like, uh, alliterations.

And I made an eight item alliteration for things that would go, that

sort of make a good runbook.

And the first one, uh, I'll, we'll just go back and forth with

them.

All right.

Yep.

W. Curtis Preston: The first one is that it needs to be authoritative,

and by that I mean there needs to be one runbook right there.

There can be versioning over time, but when the disaster, uh, strikes,

there's one place where everyone knows to go to get the runbook.

Now, the runbook may be.

Electronic.

It may be printed, it may be on an iPad.

It may be, it may be a, a bunch of different things, but everyone should

know exactly where and when that is.

And there, and it may take different forms.

There may be a paper version and an electronic version, but

those two versions should match.

There should be one authoritative, uh, version of the runbook.

So how in organizations, and I'm sure this

can be solved, how do you make sure everyone is going off the same runbook

W. Curtis Preston: Well, I, I, I think that's just a matter of.

process?

W. Curtis Preston: it all goes to training, right?

Everyone just has continually explained this is where the

runbook is when it's time, right?

And then whenever there are, um, the, the, the biggest.

Problem slash concern is if you are keeping a paper version of the runbook,

right?

You have to make sure that when you come up with a new runbook, that

you destroy the old paper version.

Just realize that if that's the case, this, this is going to be a

risk to the first thing, which is that it should be authoritative.

Um, the electronic version, assuming we're all accessing the same electronic version,

uh, that should be much easier 'cause it's one file and you can control it.

One file that's maybe replicated in

multiple places, but you know, one

but just make sure that you are aware, like if one

of the disasters you're worried about is lack of electricity or flooding or

other aspects like that, that you wanna make sure it's on some medium that

it, that can handle whatever disaster you're looking to support in that area.

W. Curtis Preston: Which is why some people like the paper

copy, right?

Um, I can't argue against it.

It's just, it's just hard to

keep up to date.

Right.

And, um, you know, and you're gonna kill a bunch of trees.

so the next one is accurate, which basically

means that the runbook contains all the correct information.

So I.

You know exactly what servers, what applications, all the

people, whoever is required.

So if people leave right that, that it is constantly updated and

it is accurate with what you need.

W. Curtis Preston: Yeah,

that it matches the current situation, right.

Yeah.

So here's a question for you, Curtis.

Do you think it's worse to have a no runbook or to have a outdated runbook?

W. Curtis Preston: I still think no runbooks worse.

Right?

Right.

If, if you, you pull out the runbook and it says, you know, please

turn to Curtis and tell him to

do the empty squat.

And you're like, uh, Curtis quit three weeks ago.

You still got a plan, right?

Uh, no.

A bad plan is still better than no plan, I

think.

Okay.

W. Curtis Preston: But, um, yeah.

Yeah.

Um, so the next one that I have is accessible.

It needs to be accessible.

We've talked about this a little bit already, and that is just make sure that

no matter what happens, you know, you alluded to that earlier, no matter what

happens, make sure that it's accessible.

To me, that would mean that I would have an automatic system.

Where there is the authoritative copy where we always do edits,

we always only edit here.

Right?

And then that is then replicated via an automated system to all of the other

places where that are read-only copies.

You might replicate it to an application.

Maybe your company has its own application that runs on iPads.

That application is, is automatically updated every time

you update the documentation.

Maybe you do print out a new one, maybe you figure out a way where you do loose

leaf papers and you, you, and you only reprint the pages that have changed.

And the new thing, you know, you could do that so you're not killing

an entire, you know, forest every time you, you update the DR book.

But, uh, but I, I do think that you need to, um.

Created in such a way that it, it's available no matter what happens.

And I think that last bit is key because remember, if this is a

corporate environment, say you don't have access to your two-factor authentication

or your active directory, right?

How is someone gonna get access to the runbook in order to be

able to restore those services?

I.

W. Curtis Preston: exactly.

What was the company, it was a big company a few months ago

that remember there was a big.

Was an attack or a disaster?

Some.

And they were locked out of the building.

Do you remember this?

Yeah, this was,

W. Curtis Preston: It was a big company.

yeah, because there are key cards.

Was it meta?

W. Curtis Preston: I think it was Meta, right?

They were, they had an outage and then they were locked out of the building.

They couldn't get into the building.

because the authentication system was,

Yeah.

don't, don't be that person.

Um,

So the next one is absorbed, which is basically is

your runbook understood by everyone in it, which I think a lot of people

kind of take that for granted.

They're like, oh yeah, I just wrote this.

Anyone in it should be able to read it, but can they really read

it and understand what's in there?

W. Curtis Preston: Yeah, this is, I, I think this is kind

of about marketing really.

Right?

You know, um, in the previous episode we talked about how good of a recovery

mindset does your organization have, and this is part of that, right?

So how well does your, you know, does, does the word runbook, if you said Dr.

Runbook, does everybody in your organization know

what we're talking about?

Because just like.

We want to make sure it's available in all of the different scenarios.

We wanna make sure that if only Steve is available right

now, Steve knows where the Dr.

Runbook is.

D knows he, he knows what the Dr.

Runbook is.

He knows where to get it

right, and he knows how to not get an old version of it,

right?

So this is, this is about making sure everybody's all on the same, uh,

all on the same

And it's, would you say though, Curtis, I know we

mentioned specifically around it, but is it really just limited to it or is

it pretty much all functional heads should be aware of this because it

could be touching them as well, right?

W. Curtis Preston: Well, I think anybody that needs to actually use the runbook.

Yeah.

W. Curtis Preston: Right.

I mean, there, there are people in the runbook that you're

going to want to contact so that they're aware of what's happening.

They don't necessarily need to know too much about the runbook.

I mean, would not hurt that they would know about it.

So the next one that I have is that it needs to be active and, and by

this, just that, um, it's very similar to the concept of accurate, and that

is that it needs to be part of your.

It change and control process.

Whenever you add a server you have a change control process.

This needs to be part of the change control process so that it's an

active part of your organization, that you're continually updating the

runbook every time you make a change.

Um, that shouldn't be too hard.

Again, assuming you have a solid recovery mindset.

Yeah, and I think for that one too, it's is that part

of your DNA of the company, right?

That has to be part of the culture of it that says, Hey, whenever we

are doing a change, we have to make sure that yes, this, this, and this.

Just like we always talk about backup, go update your backup systems to make sure

you can back up that new application.

Dr.

Runbook should also be updated as well.

Uh, the next one is, and I know I kind of touched on this

earlier, right, is adaptable.

And this really goes back to.

Everything in the runbook may not be perfect.

Things may change outside the environment.

The situation may change, and so you need to make sure that you have backup plans,

not backup, like data backup, but backup like alternate plans to make sure that

you are still able to execute and recover your systems, even if not a hundred

percent of either the infrastructure is available or other aspects like that.

W. Curtis Preston: Yeah, it's sort of like, I dunno if you've ever read any

contracts, but there's like a common phrase in a contract that like any, if

any part of this contract is invalid, that doesn't invalid, that only invalidates

that part, not the entire contract.

Sort of like here, like if you hit a single bump in the road

and then your entire run buck is off kilter, uh, that's not good.

You need

to this, I think this is probably more about.

The practicing it and doing it right.

Um, that, uh, you get, and you, you, you, you use the,

you know, like the chaos monkey

concept to throw wrenches into the works, uh, to, to see how well people handle,

uh, wrenches along the way.

Yeah, it's like your example you gave Curtis of your flight

experience and losing an engine, right?

Maybe the pilot hadn't trained for engine failure win on ascent, right?

But maybe he was able to adapt and figure out, okay, yeah,

this is where I'm going to do.

W. Curtis Preston: Yeah, exactly.

Uh, so the next one that I have is auditable.

And you know, this is.

Um, I think the best way, you know, you've heard me say it a bunch of times, the best

way to have your Dr, uh, system tested is to have someone who isn't the person

who put it together, do the one that's actually doing the, the thing, right?

So, um, if you can bring in an outside IT person and they follow your Dr.

Runbook and they recover your organization, ominous

dominus, man, you, you're

the best.

Do you wanna talk about your experiences with Dr.

Runbooks at the big bank you used to work at?

I know we've talked

W. Curtis Preston: Yeah.

Yeah, yeah.

I know, I know.

Yeah.

Not everybody's listen to every episode.

So, you know, we used to do a, we used to do a full DR test every six months.

And, uh, it was a really big thing.

Like we, you know, we shuttered everybody up for the weekend and, and, uh, we

spent lots of money doing these DR tests and we tried to do it like this where,

you know, I had a lot of documentation.

I had my runbook out there and they, they would, they would go to follow it and.

Um, we defined a successful recovery as a recovery where the person running,

the DR didn't have to turn around and go, so, Curtis, what did you mean by.

This.

Right.

And it by that standard, we never had a successful recovery during

the three years that I worked there.

Uh, we always recovered, but there was always some, some turnaround.

And so that goes back to that, that adaptable thing that

I, you know, that I said.

Right.

Um, but it also goes to.

Testing more often every six months isn't good enough, right?

Every six months was the only way we could do it back then.

But nowadays with the cloud, you know how big of a fan I'm with the cloud.

But nowadays with, with infrastructure as code and all these other things that

we're able to do today for many, if not most organizations, they could do this

in the cloud and they can automate it.

And that a lot of this stuff that, that we have to figure out in our heads

could actually be completely automated.

Um,

And.

And I pretty much alluded to the next,

Yeah.

The next one, which is basically Affirm, so test regularly testing is

a best way to make sure that your Dr.

Runbook is accurate and that it works.

And I think just like you gave the example Curtis, in the three years, you

didn't really get to that level that you guys set, and that's perfectly fine.

Don't expect perfection in your Dr.

Runbook.

It will require tweaks.

It will require changes over time, and that's okay.

It's a starting point.

It's better than having nothing like you had alluded to earlier.

W. Curtis Preston: Yeah.

Agreed.

Um, yeah, so that, that's sort of the elements, if you

will, of what makes a good Dr.

Runbook.

Let's talk about what sort of the ingredients of a Dr.

Runbook, right.

It, it's

just,

That's the way.

It's not just, we set up the infrastructure and we

just randomly run a Dr.

Runbook.

That's not at all.

W. Curtis Preston: Yeah.

Yeah.

Um, so the, the first thing here, and this is one of those where the, the first

thing is the last thing that you do, and that is you have, uh, an overview, you

have an executive summary, um, and, um.

And, and like a technical overview to just give, give everybody,

uh, you know, in that scenario of like we lost all our IT people.

Uh, now you've got some IT people that are not your IT people, but they're

different IT people and they're following, you know, so you give them an overview,

uh, as well as, uh, executive overview.

You, you, you need to write this at the end because you don't know what

you're, you know, what you're doing.

An overview of right.

Yeah, and I would say that a lot of technical folks

might think, Hey, isn't a runbook, just here are the procedures that I do.

And I think these first couple that we're gonna talk about of what goes

in a runbook are critical, which will help you either in marketing or just

to make it more complete for people who may not be doing this day to day.

Right.

So the next one is really.

An inventory.

So all your products and vendors and the contacts for those.

So who is your, what is your SaaS provider for email, right?

Is it Microsoft?

Is it Google?

What is your account there?

Who is the contact that you reach out to in the vendor in case, like

do you have a technical account manager, an account rep, et cetera.

So these are all critical things to document because you

might need their help, right?

It's not just, Hey, I need to recover my own systems, but you might need help with.

Recovering some things that are in their infrastructure as well,

so just things to be aware of.

W. Curtis Preston: And there can be an escalation there, right?

So you know, who is the person that you normally talk to at Microsoft or Dell

or HP or whatever it is that you know, where you're buying your infrastructure.

Uh, M Center, you know, who is your normal point of contact.

And then you should also, and by the way, you should have.

Multiple

ways to contact them, email, whatever, whatever you have.

If you have, you know, phone, right?

If you have, you know, their Twitter account, whatever.

Um, and then also like management, right?

You know, if, if, if, if, if they, if you go to reach out to Steve

and Steve, I use that name a lot.

You go to reach out to Steve and Steve isn't there, um, then you

know who then, then what do you do?

Because in the middle of a disaster, you may need some additional stuff.

You may need some additional hardware, you know, storage, whatever.

Um, and so you, you know, or you may need some additional expertise.

I push the button and the button didn't go.

Who do I, you know, where's my, where's my 9 1 1?

Um, can you think what else would be there

Yeah, and it doesn't have to necessarily be even your vendor.

It could be an MSP, right?

Your consulting partners, other folks as well who are important to keep track of

because they could be there to help you and assist you when you're recovering.

W. Curtis Preston: Right,

Or it could be your cyber insurance company information too.

W. Curtis Preston: Yeah.

Yeah.

We, we've mentioned them a couple of times in this, in this episode.

Uh, the next one sounds very similar, and that is the contacts for the

people needed, the DR Recovery.

What I'm thinking here is, this is your staff,

right?

Um, this is everybody.

And again, you know, we, we joked about it earlier.

Um, and that is that.

The, the successful one is one where we didn't have to turn

around and ask for Curtis.

Well, if it's a real disaster, it's okay to call Curtis,

right?

Well, not me, but, 'cause I didn't do your Dr.

Runbook.

But you, you've got a Curtis in your organization.

Everyone should have a Curtis in their organization.

Don't you think?

Prasanna.

How about everyone should have a

Prasanna in their organization?

W. Curtis Preston: Everybody should have a, a person who knows way too

much about many random subjects.

Yes, definitely, definitely.

Someone should have a, you should have a Curtis and a

Prasanna in every environment.

Uh, so that's step one in your Dr.

Road because identify the Curtis in the Prasannas in your environment.

Um, and, um, the, um, uh, but yeah, this is contacts and again, escalation.

Right.

I reached out to Curtis.

Curtis didn't, you know, didn't answer his phone.

Who do I call now?

Right?

Who?

You know, you call, you know Curtis's boss and Curtis's boss's boss.

Right.

Um, and, and also, um, contacts within the organization from a

here's who we need to keep, uh,

you know, abreast of everything that we're doing.

Right.

Management.

Um, and hopefully you have like.

A designated person in your senior management that you, you just go

to them and let them communicate everything to everybody else because

you're kind of busy doing the thing.

Yeah.

W. Curtis Preston: Right.

Um, and, uh, this is what back in the day nowadays, we would, um.

We would do it differently, but back in the day, this is what we used pagers for.

We had an automated paging system and we would just send a page out to everybody.

Everybody that was involved in the process got a page at every

step of the process so that everybody knew what was go, what was going on,

which means you need to maintain a list of all those pager numbers and

all that kind of stuff, which of course, no one.

In my mind as you're talking about sort of

the escalations, I was thinking about your story about the Bobs,

W. Curtis Preston: Yeah.

Toms.

Toms

oh, Toms.

Toms.

W. Curtis Preston: The two Toms.

Yeah.

Uh, the two Toms standing over the left and right shoulder.

Yeah.

That was a, that was a pretty funny moment where we were in the middle of

an actual disaster and we were, um.

You know, one of our guys was in the knock and he was on speaker phone with what?

With the guy that was actually in the data center swapping tapes.

And he said, he said, oh, are, are you in the knock?

And he is like, yeah.

And he goes, so let me guess you got Tom and Tom over your left and

right shoulder, except he didn't realize he was on speaker phone.

And, and Tom and Tom, which was their bosses.

Let's see their boss's boss's boss and their boss's boss's boss's boss.

Wow.

W. Curtis Preston: And, and, and they did indeed have, 'cause this was a big bank,

right?

So there were many layers of, so this was basically the CEO's second in command

and then the one after that, right?

And they were standing over his left and right shoulder.

And when they, when they said this, they, they both just took one step back.

Oh, good times.

Good times.

Yeah.

So the next thing is really what a lot of people think about when they

think about a runbook, which is really the procedures which are those

step-by-step instructions on how to actually recover from that disaster.

So this might spell out sort of, okay, what exactly from an

infrastructure I need to bring up, what applications and what order, right?

It's probably multiple sections because it's not like one big bang and you

bring everything up in one go, right?

So depending on what the disaster is, it might have multiple.

Different sections with different procedures for each one, but it should

be labeled numbered and easy to find.

W. Curtis Preston: Yeah, so you've defined, this is

why I said this is the ult.

This is the culmination of all of your efforts.

You've defined the RTO and RPO of everything in your environment, and

then you defined a recovery priority of everything in your environment.

And so this is gonna say, okay, you know, recovery, you know, let's say you've got.

I don't know, three recovery groups, right?

The stuff you're gonna do very, you know, the stuff that's gotta come

up before anything else comes up.

And then the next set, which is like the stuff that comes up right

after all that stuff comes up.

And then everything else.

And the first group is probably really small.

Second group is probably decent sized.

And then the third group is probably the biggest

because not everything has to be up and running, you know, immediately to

be able to turn the lights back on.

So all of that should be spelled out in the, you know, you, you may also, because

we've talked about this previously, you may also use completely different

recovery methods based on the RPO

and the RTO of different.

You know,

uh, parts of your organization, you might use a DR as a service product

that does replication based, uh, DR for your tier one applications.

And you might use backup replications for your tier two, and you might use,

you know, tape, uh, or, or maybe a replicated copy of your backup into

S3, uh, and hope you don't ever have to

use it.

For your, uh, for your tier, you know, your tier three

things right?

And all of that should be spelled out on all those procedures.

Um, what, what else do you think?

Any other thoughts on the procedures?

No, I think that's, but this is where it goes

back to that, uh, what was active that we had talked about earlier, right?

When you are making changes to your infrastructure, that's why

it's critical that you go and you update these procedures.

Because when you need to execute, you wanna make sure that it is accurate,

so you can have the best version of.

Or best, best chance of recovery

W. Curtis Preston: Yeah.

Yeah, exactly.

Uh, and then, um, our final step is escalations.

And we've already talked about this, but nothing's gonna go right.

Prasanna Malaiyandi: really, Curtis, why not?

I thought that's why we, and plan and

document, and.

W. Curtis Preston: As much as, I'm gonna say like a disaster re an actual disaster.

Recovery is much more like a rugby match.

Then it is like a baseball game, right?

Baseball game.

It's like you got clear lines, right?

I'm gonna go this way, then I'm gonna go that way, then I'm gonna go that

way, then I'm gonna go that way.

Or for the rest of the world, a cricket game, right?

Or you know, rounders, uh, you know what, you know, basically, you know, rugby.

Uh, you know, is, is is like, it just seems to me comparatively speaking like a

giant mess of, there's an ultimate goal.

We all have the same goal, but, uh, you know, it's just, it, it you, there's

this constant adjustment around, and so you you need to have those lines

of escalation so that people know.

And also there may be a point in your recovery where.

You need Tom to call the, to call the equivalent of the TOM over

in the other organization, uh, where you're not getting any love,

right?

It's, it's a weekend and your favorite IT vendor is not answering their phone.

Uh, or they're, or they're not giving you the amount of love, you know, they've,

you, you've contracted for a four hour.

Uh, delivery time of new hardware and, and they haven't even called you

back and you're at like three hours.

This is when, this is when you escalate on your side so that you can get

them to escalate on the other side.

Uh, and you've just gotta have all those, um, you know, all those numbers in there.

Um, and then I'll tell you what not to do.

Here's, here's another story.

Um, today's about the stories.

So we had, we had a, an employee that and she got handed this pile of tapes

and we were told her, she's like, Hey, you got this pile of tapes at some

point, bank of America's gonna call you.

They'll tell you what to do with these tapes.

And she got handed the escalation list that came with the tapes.

And after a few hours working, she decided she couldn't handle the

anticipation anymore, so she decided to.

Go down the escalation list to see if she could get an answer as to what she

was supposed to do with these tapes.

So, but I guess she wasn't familiar with how to use an escalation list.

She

just looked at the

oh no.

W. Curtis Preston: and so at three o'clock in the morning, she called

the CEO of Bank of America at home

and said, I got these pile of tapes, got these pile of tapes here.

I don't know what I'm supposed to do with 'em.

No.

W. Curtis Preston: That's not how you use an escalation list.

Yeah.

W. Curtis Preston: That was, that was my point of bringing that story up.

So basically that's the Dr.

Runbook, I mean.

It's, it's putting down all of the plans and procedures and the what to do when

the plans and procedures don't work into a form where you can easily index and

get through it and whatnot and follow it.

I do think it that there should be an authoritative one version

that you're constantly updating.

Um, and then just make sure it gets, you know, automatically

disseminated to the other things.

And, um.

Uh, and you should be doing this frequently.

You should be testing it frequently so that an actual disaster.

So when you're doing an actual disaster, you're not testing just the DR plan.

You're also testing your runbook, right?

You're testing your team's ability to follow that runbook.

I a hundred percent agree and I think this is

where it becomes critical though.

In technology, we typically do a poor job of documentation.

Everyone wants to go build the cool stuff, put things together, and then

they hate writing documentation.

This is one of those things that'll pay tremendous dividends when you actually

encounter a disaster by actually having a documented runbook ahead of

time and following these procedures and doing the testing, like you said.

So instead of something taking.

Say a month trying to recover as everyone's scrambling because

no one knows what to do 'cause they've never documented or tested.

It's all documented.

At least you'll reduce the stress level of everyone in the organization

as you're going through this.

W. Curtis Preston: And there you go.

This, welcome to those of you that listen to this entire series.

Welcome to the end of the Disaster Recovery.

Series six episodes.

If you didn't listen to the other five, go back, go back.

This was some good stuff.

And, uh, Prasanna, thanks for sticking with me.

Uh, and by the way, you know, for those of you that didn't know, we're basically just

reading from my book, which is Modern Data Protection, uh, which is from O'Reilly.

Um, you know, for those of you that didn't know that's who I was, I'm,

you know, I'm w Curtis Preston, the author of the book, and, um, you know.

Uh, Mr.

Backup, whatever.

So, um, thanks, Prasanna.

Great comments as always.

Uh, likewise Curtis.

W. Curtis Preston: And thanks for those of you that have, uh, stuck with

us, uh, through these six episodes.

And for the rest of you go back and listen to the other episodes of

this is the only one you listen to.

Um, that is a wrap.