ATR2500x-USB Microphone & Logitech BRIO: This week, we're talking about disasters

and I don't just mean the monster snow storm hitting the Northeast right now.

We're talking disasters to take out your entire infrastructure.

We're getting back to basics on perhaps the most important, most

often neglected aspect of backup preparedness your disaster recovery plan.

We'll walk through the essential elements.

Every business needs to address before catastrophe strikes.

A box of tapes or backups in the cloud.

Is not a Dr.

Plan.

I'll even share a crazy story from the early days of my it career.

That taught me the hard way.

Why a documented recovery plan matter.

I honestly still can't believe what my employers did that day.

I'm w Curtis Preston, AKA Mr.

Backup.

And I've sat where you're sitting.

I want to turn you and everyone like you and to a cyber recovery hero.

This is the backup wrap-up.

W. Curtis Preston: Welcome to the backup wrap up.

I'm your host, w Curtis Preston, and I have with me a guy who as awaiting the

results of my battery test as much as I am Prasanna Malaiyandi how's it going?

Prasanna

I am good, Curtis.

Yes.

I am dying to understand like, do you have any degradation in

your carb because you sort of ran into an issue last week, right?

W. Curtis Preston: maybe, maybe.

Um, yeah, I killed my battery in my Tesla and I'm currently running a, a battery

test and I will, I will just say this.

It's, it's not, it's clearly not designed for like the, for me, uh, it's, it's

actually something inside service mode of the Tesla and you have to like, go into

super secret mode or whatever to do it.

And, um, you know, it's okay that I'm, that I do it, but I will just

say that the UI is really crappy.

Yeah, it's not intended for you.

It's intended for a service technician.

W. Curtis Preston: Yeah, and so like.

It finished, you know, it discharged, uh, to 13% last night, which I

thought it was gonna go like closer to zero and then it charged overnight

and now it's doing something.

Last time I checked, like you said, it's probably measuring voltages and

whatnot, but like there's nothing on the screen that says doing something.

The only way you can do is you can say, Hey, show me the report of the battery

test, and it could go, you haven't done a battery test for 500 miles.

And, um, and so it's like, oh, I guess it's still running.

Like, it's just a really weird user interface, but, um, so we'll see.

But I'm

Crossing my fingers.

At least you'll get some data.

W. Curtis Preston: yeah, yeah, absolutely.

Data, data data's king.

Mm-Hmm.

W. Curtis Preston: So, uh, speaking of data, let's talk

about the news of the week.

I'm gonna say the big story that jumped up.

You know, whenever we do this, I, I, I, I just google backup news

and I see if anything pops up.

And the thing, that thing that popped up was kind of a big deal,

and that is that there are 50,000 WordPress sites that are affected by

a critical bug in a backup plugin.

Called backup migration.

That helps admins automate site backups to local storage

or to a Google Drive account.

And, um, it's a scary, it's a scary little, uh, uh, bug, wouldn't you say?

Oh yeah, especially because most of these backup tools,

of course, need admin level access.

So if you have a flaw in that, of course someone who exploits it

now has full access to everything.

W. Curtis Preston: Yeah, and this was scary because it says, it, it it, you

can do it without, uh, user interaction.

You can, uh, and, and once you do it, you gain.

Uh, basically full control of the site.

Um, and, uh, it says, um, basically they just have to pass the right

values to the right thing and poof, they end up controlling your site.

The fir you know what, the first thing I did was right.

I.

Check.

Check to make sure you're not running it.

W. Curtis Preston: Yeah, that's exactly, 'cause I knew I had a migration plugin

'cause I might, I migrated backup Central from, um, one, well, it's

actually within the same company, but I migrated it from, essentially from one.

Server to another server not that long ago.

And I used a migration plugin to do it, and I saw that word backup migration.

I was like, wait, was it, what is this the one I used?

Right?

And, uh, and it wasn't so, um, I, but I, but I, then I, I took the moment I

realized that the migration plugin that I used was still installed and activated.

And I'm like, you know what?

I'm just gonna go ahead and install

Yeah.

Now, uh, the one good thing is once the developer was aware of this issue, they

pa they had a patch available within a couple hours, which is good, right?

They were very responsive.

However, now they just need to make sure that all the sites get updated

who are using the plugin, and that's, I think the bigger challenge

W. Curtis Preston: Yeah.

It really is the bigger challenge, especially given that, uh, the,

the developer doesn't mention anything about this on the.

wordpress.org plugin website.

So I'm a little disappointed in that, but you know.

Um, I dunno what to say about that, but yeah, so I, I'll just say that.

I'll just put it as a general note.

If you do have a WordPress site or you know, whatever, if you

have a website, look at any, you know, any plugins that you have.

Like in this case, I really should have deactivated this migration.

Plug it a long time ago.

Make sure that any plugins that you have that are running, you actually

need them, deactivate them, uh, and remove them, uh, if you're not.

So, um, what's our next, uh, news item?

So the next news item is Veeam just came out with

their big launch and they've basically doubled down on a lot of cybersecurity

related features, and I thought maybe we could talk about some of those.

W. Curtis Preston: Yeah, there are a lot.

Um, right.

So, uh, you know, some of them you might expect, right?

The idea of like inline malware detection, right?

Detecting the malware as it's coming in, in the backup, looking for a file system

activity that looks a little weird, right?

That, that would include, or that would include things like, hey,

there, somebody's, uh, appears to be.

Changing a lot of files, changing the file names, different file types.

You never had this file type.

And where you see that is if you see an encryption attack.

They tend to create a file type as the encrypted files have their own file

type and they're also integrating with, EDR and XDR tools and SEIM/SOAR tools.

Which that sounds pretty good.

So they're also integrating with ServiceNow to, it's

sort of looking at, Hey, here's another part of your environment that we could

potentially detect bad activity, so let's make sure we're plugging into your

existing infrastructure tools, rather than you having a report somewhere else

that you rarely ever look at, right?

That then you're like, oh, I saw that my backup system noticed

this like 30 days ago, and I'm just starting to see that now.

So I'm glad they're integrating into the rest of the environment.

The reporting environment, which makes sense.

Reporting and monitoring.

one thing from that list that I did like, and I know Curtis, you talk about

this a lot, is, I don't know if you saw, there's mention of four Eyes, right?

So now in case you need to make certain modifications to backup settings and

other things like that, you basically need two admins or four eyes in

order to do those sort of operations.

So.

This helps in a lot of the ransomware cases where a ransomware actor gets in

and they basically change the backup policy, change retention down to one

day, and then all your backups are gone.

Right?

So these sort of things could potentially now require having four uh, two

people in order to be able to do that.

W. Curtis Preston: Yeah, I like that.

I also like the integration, uh, with Yara.

Uh, so basically once you've identified some malware that you have, they can

find that malware and get rid of it in your backups, which that sounds great.

Uh, they got KMS integration.

They got a lot.

I mean, it's just like there's really too much for us to cover in,

in this level of, in this level.

Uh.

Of report,

um,

link, include some links to the articles, to

the websites and places you could go look to see more details.

But I'm glad that there, there's so much focus though on cyber security

just given what we see in backup environments with ransomware and

other threat actors going purposefully after those backup infrastructure.

W. Curtis Preston: Yeah.

Uh, there's a great blog from Jorge DeLaCruz.

Uh, we'll, we'll link to that, I think probably be the, it lists all of them

and sort of explains why, you know, why the different features are important.

So that is the news of the week.

I hope you enjoyed the news, or maybe you didn't enjoy it because you, uh, maybe

there was bad news in there for you.

I don't know.

In our continued series of, uh, backup to basics.

We're talking today about DR.

Plans, and today we're gonna talk about what goes into a DR

plan and, um, you know, because.

One of the things you, you may have heard me say, uh, you know, is, is

although this, this phrase is now pretty dated, a box of tapes is not a DR plan.

Uh, a a bunch of a bunch of backups in the cloud is also not a DR plan, right?

You're like, oh, I got, I got all my stuff up in S3 with object lock, turn on.

Good,

Now what?

W. Curtis Preston: But, um, yeah, now what?

Right?

So, um, the first thing here.

That I've got, uh, you know, you need to assume that you're starting from scratch

if you're doing a disaster recovery plan.

I will say that the most likely reason today that you're going to

be actually using your DR plan is that you had a ransomware attack,

but that you cannot assume that and.

Also, you cannot assume, depending on decisions that hopefully you made well in

advance, you cannot assume the first thing that should be in a DR plan, and that is

that you know where you're competing and infrastructure is gonna come from, right?

Uh, why?

Why would that not be the case?

Right?

You got a ransomware attack.

They didn't blow up your servers, they didn't set 'em on fire.

Why might not you have servers to

Because those servers you might need to

preserve for forensic purposes to really figure out what's going on.

You don't know if you can cleanly delete everything on them, start from scratch.

It's probably gonna take you more time to do that.

Although one can argue given sort of supply chain issues right now,

that maybe it'll be less time.

To do that than to actually buy new servers, have 'em

drop ship to your data center.

But, uh, not saying that's the only option for you.

W. Curtis Preston: I'm personally in the Yes, you need to

take a forensic copy, right?

So that's the, that's one of the first things you, you, you want

do is take a forensic copy of any servers that it appears have

been affected or, or infected.

The, I'm personally a wipe and restore person.

Right.

Um, you know, everybody there, there, other people have different opinions

on that, but I, I think you, you've struck on it a pretty good one.

Supply chain, which is another reason why perhaps where your

competing infrastructure come, will come from in a DR is the cloud.

Right.

But just the make,

It.

It could.

Right?

And I think for a lot of cases that makes sense.

But I know for instance, if it's like a mission critical application

for your business that you cannot lose, I'm pretty sure even before you

get hit by a ransomware, you should already have a DR plan in place.

Maybe that implies sort of replicating to another co-location facility that you

have equipment already set up ready to go.

So you don't necessarily have to worry about the procurement process, but

W. Curtis Preston: Yeah, I think, I think that'll be a running story here, right?

Basically the whole point of the DR plan is the second word, right?

Is the plan part, right?

Hopefully you're doing this, um, you know, the, um, you wanna take the next

Yeah.

So the next one is, uh.

Thinking about once you now have your disaster recovery site up and

running, how do you protect it?

Because you've failed over, you've recovered to that environment.

Your production applications are running there.

You need to make sure that you have a way to protect it in case you need to worry

about how do I restore from a backup or something else happens while I'm running

in that disaster recovery site, because that's literally your only copy of data.

W. Curtis Preston: Yeah.

Um, and you know, and some people say, oh, gee, Chris, you know, like, you

know, why are you talking about backups?

We haven't even got to the Restore yet because make sure that you

have put this as part of your plan.

Make sure that once you have that.

You are now immediately going to start backing up.

Don't be the person that got the environment up and running and then

something else bad happened, right?

Because there could be, for example, a second part, there could be a

second wave of a ransomware attack.

Um, so yeah, make sure that you've got that, that replacement,

the, the backup, um, of the

and, and Curtis, I guess the question for you is

when you talked about failing over, like potentially using cloud as your

infrastructure, make sure if that requires using a different backup

solution, that you are trained on it, you've procured it, right.

You know how to use it

W. Curtis Preston: Mm-Hmm.

you need to start protecting if you

fail over into the cloud.

W. Curtis Preston: Yeah.

Great, great.

So if you're, if you're currently have an on-prem infrastructure and

now you're using cloud as your, uh, recovery mechanism, then make sure you

already know how to protect the cloud.

Right.

Uh, because it, the cloud is not magic, right?

So it's just somebody else's computer.

Um, so the next is about recovery requirements, right?

If we are making a plan, then you need to know what your requirements are.

What are where, what is your RTO and RPO, your recovery time

objective and your recovery point

Isn't it zero?

Zero for

everything?

W. Curtis Preston: yeah, zero.

Well, you, yeah.

Thi this is, you know, I, I think we've done an entire episode on how

to come up with the RTO and RPO.

The, um, the challenge there, you know, when, when you go to somebody, when

you, when you go to a business owner, because as we often say, this is not,

these are not numbers that should come from you, they should come from the

businesses, the business units, the, uh, or whatever it is you call in, in

a, in a governmental world, um, you know, the, the business units that are

going to be affected by the disaster.

They should be the ones who come up and say, well, uh, you know,

these are our requirements.

And yes, every single one of them, when you ask them, they will say zero and zero.

And then you come back to 'em and you say, okay, well we can do zero and zero.

It's going to cost $1 billion.

And, and then they go, oh, well we can't afford zero and zero.

So what can you afford?

Right?

And so you, you work, you know, but you, you, you have

to know what your RTO and RPO.

Are going into, uh, doing a disaster.

and you also set the expectations that way too, right?

Because now everyone signed off before the disaster happens that Yes, in order

to recover this application, because I'm willing to spend X amount of dollars, I

expect that it will take 12 hours or 24 hours or whatever it is, and that's what

we've all agreed to and signed off on.

W. Curtis Preston: Yeah, exactly.

Uh, the other thing to just remember about RTO, especially RTO, is that

that's fully functioning, restore, you know, of the environment, right?

So that means that the application was working here.

And now it's fully the application that I need and the other

applications that it needs.

Those are all running and functioning.

It doesn't just mean the restore.

A lot of people focus on just the restore and just remember that that's

literally only one part of the, of

And, and when you say restore, restore of the data, right?

W. Curtis Preston: Restore of the data, right?

Yeah.

Well even restore of the, you know, the server and getting

the server up and running.

You're like, okay, the server's up and running.

Right.

Is the application up and running?

Right.

Uh, which may, which may res require, um,

Prasanna Malaiyandi: Coordination with other,

W. Curtis Preston: other applications, right?

Yeah.

Which is a great way to lead into the next one, which is

what are your recovery priorities and prerequisites, because like you mentioned,

Curtis, in order to get this application functioning, I might need five other

applications to also be up and running.

So I need to make sure that those five are up before I can ensure that

this application is up and running.

So.

Understanding the dependencies and also the priorities.

Like, is Curtis' laptop the most important thing in the

business to bring up and running?

Or is it really our billing system, right?

Or our e-commerce site, right?

And so understanding what is critical to get the business running is important

because you're not gonna have time to bring everything up at the same time.

W. Curtis Preston: Well, if you were following Curtis's usual

suggestions, you weren't using active directory for your backup server.

And so the only thing that you need to get into the backup server is the password

manager that's running on Curtis's laptop.

I'm gonna say yes, Curtis' laptop Being restored is

What happens if Curtis' laptop is infected

with ransomware encrypted?

W. Curtis Preston: Well, then you're screwed.

Right?

That's just, that's what, that's what we, we had a, we had a good episode on that.

Right?

The, you know, restoring from, from nothing.

Right.

How do you, how do you go from nothing?

Um, the, um, the next one, uh, is about people.

This is an interesting one because, um.

You could look at this, uh, you know, basically how, how

will you get the people, right?

Um, you need to prepare for a number of eventualities.

One of them are that the people are either not alive,

Prasanna Malaiyandi: available, I think is a.

W. Curtis Preston: They're not, well, they're not, they're

not alive, or they're not.

They can't access your environment, right?

So depending on what we're talking about, especially in this remote

world that we live in now, right?

Where a lot of people are working remote, the first thing you need are,

you know, uh, people that can, you know, the, the people at the keyboard.

So, um, how will they get to that keyboard?

How will they, um, you know, well, how will they get to

Or, or a virtual

W. Curtis Preston: you're talking about.

Or virtual keyboard.

Right.

Um, the, um, how, how, how will that happen?

Right.

And you need to include all the things that could happen if you live in a, you

know, if we're talking about natural disasters that you're protecting from, you

need to say, oh, well, the roads are out.

The roads in between A and B are out, the roads are where they're snowed over.

Uh, we had a flood and you can't get from A to B.

So these are things that you need to take into account.

I think there's a great episode there where we have the, we interviewed

the, the guy that was, you know, the, the, the guy on the ground for that

disaster at the Caribbean island.

And, uh, he talked about all the, the critical infrastructure

that was not available to him.

Um, and, um.

It's like all those

W. Curtis Preston: Yeah.

Yeah.

You,

Yeah.

W. Curtis Preston: yeah.

And, and also speaking of the people, you, you do need to figure that out,

thanks for bringing that up, is, uh, you, once you get the people in,

they're probably not gonna leave.

So, so figure out where they're gonna, um, sleep and also figure out how they're

gonna eat and, you know, use the bathroom showers, uh, hopefully at some point.

Um,

one of the things, I don't know if where it falls under, maybe

it's like part of the infrastructure ones, maybe it's part of the people one, do

you remember the episode about the dire show that hit and the land hurricane?

Right.

And one of the problems, and I can't, unfortunately, I can't

remember the guest's name, but one of the things you mentioned was

they had no internet connectivity.

So he literally sent people to the next town over so they can sit in a

Starbucks, connect in and manage their infrastructure and systems because no one

in that area had access internet access.

W. Curtis Preston: Yeah.

Um, and we'll just go to a Starbucks is probably not a good,

I don't know if they actually went to a Starbucks, but

W. Curtis Preston: So, no, I'm just saying that's what he did, uh,

because that was his only choice.

Uh, just make sure you, you, you figure that aspect, that figure, that aspect out.

Um, and then we have a.

just another quick thing is also realize that

the people you're relying on, they might have other priorities as well.

For instance, if it's an earthquake in the local region, maybe they're

taking care of family, maybe they're dealing with other things.

I know we had the episode with the person talking about the female

response right to, and how to prepare yourself for disasters.

And that's something that we don't always think about, but you should

also take into consideration.

W. Curtis Preston: Yeah, absolutely.

You mean people don't prioritize company above all us?

We would like to think that, but yes, real life happens.

W. Curtis Preston: Yeah.

All right.

You get the next one.

All right, so the next, and I like this one, Curtis,

because I always have a, I always remember your story, you tell, and it's really

how good is your documentation, right?

And.

Only as, because like we talked about people, if people aren't there,

right, who can now operate and make sure and recover your environment?

Or if people are there, who created it, who's the one who's coordinating

and making sure, okay, are all the steps in order to bring up this

application, I need to go talk.

I.

And make sure that these five other applications are up and running, that

I have the networking there that I've already pre-configured the compute, right?

There's a whole bunch of steps and so making sure everything's documented.

'cause the last thing you wanna do is have a disaster and be like,

oh, I forgot steps four and six.

And those are critical for bringing up this application and sitting

there and trying to troubleshoot in the midst of a crisis.

W. Curtis Preston: Yeah.

Um, this was something that was basically pounded into me very early in my career.

The bank that I worked with, um, had, um, I.

You had to have, because it was a bank, the the SEC would, you know,

required us to, to do these things.

Right.

And, uh, we had documentation, and I remember a fight over what

the documentation was going to be created in the, the, the company's

standard was word perfect.

And we had this one guy that really liked Microsoft Word.

Uh, we all know that, how that worked out.

I mean word, word, uh, took over.

But, and that was like, he's like, we're perfect stakes.

And you know, Microsoft Word is so much better.

And we're like, it doesn't matter.

You gotta do the documentation in the format that we, that we use.

Right.

And um, and part of that was, well, a big part of that was that we were a.

Uh, like not everybody had a, like a, nobody had a laptop, but not everybody

necessarily had a, a desktop that ran Windows where you could run these things.

We actually used word perfect because it ran on Unix-based infrastructure.

Right.

Uh, and we could do shared documentation.

So.

Yeah, the important thing is to write this documentation and test it, test it, test

it, test it, and make sure that you te that you have other people that test the

documentation that you wrote because they will find the things that you left out.

Um, also make sure to think about I.

How and where you keep this documentation do not out rule the

idea of making multiple printed copies of that, of that documentation.

The downside of that is, you know, you gotta kill a couple of

trees every time you update it.

But, uh, maybe use loose leaf notebooks so that you can, uh, easily update a

section of the plan if you change it.

But, uh, and have it in multiple formats like PD, F, and, you know, have it

available in places where you can easily access it, uh, paper, online, maybe

even, you know, and, and a, a and a truly online, like not anything in your, in your

infrastructure, something in the cloud somewhere where you can possibly get it.

Just make sure all of that stuff that's, you know, authenticated and secure

and don't put passwords in that documentation please.

W. Curtis Preston: Don't put passwords on that documentation.

Exactly.

Um, another thing, uh, here that I've got is, you know how much of this Dr.

Runbook that you have, how much of it is docu is, um, is automated?

Because the more you can automate.

The more that a disaster recovery, uh, actual execution is going to work.

And, um, you know, so the question is in that DR plan, just make sure you

understand that level of automation that you have or don't have, uh, and

continually strive for more automation.

You know, because a two things.

One is your.

You're more than likely to have a successful recovery

if you have automation.

And second, it makes testing the recovery a, a whole lot easier.

Again, I'll go back to when I used to have to do this in the field and

what we did was we, we threw, you know, we blew away a weekend, right?

We'd have to come in at early Saturday morning and we would start the disaster

recovery test and basically it took the entire weekend and it was hell.

Right.

I was just like, and, and basically the worst part was that there were

two things that really stunk about it.

One was if you were the one at the keyboard, super high stress, right?

Or if you were me, it wasn't me at the keyboard, but I was back there.

I was the one that had written the documentation that there

following that hopefully works.

Uh, and then you got the boss and they're stressed out whether or

not this is gonna look, you know, whether or not this is gonna work

because then they'll look good or bad.

Um, and then meanwhile you got a whole bunch of other people that are just.

This is so boring.

They're just standing around just in case they're needed, hoping that

they're not needed, but they're

Just in case.

W. Curtis Preston: standing around.

And we didn't have, we didn't have YouTube either.

So, uh, you know, you needed to, you needed to bring a

book, uh, to have something to

So here's a question, Curtis.

So from your documentation, from this experience at the bank, and I'm guessing

that a lot of it was probably more manual than automated, just given the timing.

W. Curtis Preston: Yeah, there was no

Um, how often were you successful at really

running through your entire Dr.

Playbook?

I don't mean to put you on the spot,

W. Curtis Preston: So,

I think it's

W. Curtis Preston: yeah.

So if you define success as.

We had the documentation.

Someone other than Curtis followed the documentation, and we got to

the end of that recovery without having to consult Curtis on what

he really meant by this sentence.

The answer is never.

Yeah.

W. Curtis Preston: Right.

So the goal of every DR test was to, um, you know, was to help essentially

make the, the documentation better.

And, um, the only way really to do that is to do it, you know, to

fire it in anger, as we say, right?

Um, and, and so we just, we just set that, that was our, um, we,

we considered that a success if we found something that we could fix,

Better to do it while testing rather than in when you need it.

W. Curtis Preston: Yeah, it was, it was always successful in that we

always restored all the data that we were supposed to and the applications

that we were supposed to restore, uh, because my backups worked.

Dang it.

Uh, but, uh, it was not successful by that other metric.

I think this goes to the last point in the plan, right, which

is make sure that it has been tested.

Because only because you guys were doing this periodic testing, did you

realize, hey, there was some issues with the documentation, or in other

cases there might have been issues with.

Uh, I know when we had the episode with the disaster on the island and

the tropics, right, um, that one of the issues they had was they had satellite

connectivity back to the mainland, to the us and that's where their active

directory infrastructure was running.

And so when they got hit by this hurricane, they lost

connectivity, which means that their systems weren't functioning.

And so if they had done that sort of testing, they would've rec realized

it earlier on and been able to have a solution rather than putting a

person on a plane with some backups of active directory to fly out to

the island to then do the restore so they can get up and running.

W. Curtis Preston: Yeah.

Testing, testing, testing.

Right.

And this, and again, this is why I like automation, is you can test.

You can, the more you automate, the more often you can test and

the more you know, you can know that things are actually working.

Um, yeah, there, there, there's just no substitute for

but I think one of the things to realize is you don't need to

strive for a hundred percent automation.

It's okay in your first versions to have a bunch of things manual and

improve on it over time because you don't know if the audit like what to

automate until you've actually done the manual process and laid it all out.

So start somewhere, have something there and then figure out, okay, what

are the pieces I can start automating?

How should I automate it?

And all the rest.

W. Curtis Preston: Yeah, absolutely.

Absolutely.

Do you have a recommendation on how to test?

Because I know sometimes testing is expensive from an

infrastructure perspective.

I know you went in on a Saturday when the bank was closed, right?

Or they, it wasn't as busy, and so you were able to do the testing.

A lot of people don't necessarily have the infrastructure in order to be able

to do these periodic testing because you probably at least want to test your DR.

Environment.

At least once every couple of months just to make sure you're not

missing anything and all the rest.

So is there a better way people can do DR testing?

I.

W. Curtis Preston: Well, I think this is a great time to discuss like

my opinions about the use of the cloud, if at all possible, right?

The more you can utilize the cloud, the more it can basically set aside

some of these, uh, budget constraints.

You're still gonna need to take time and there's, there's.

Um, there's nothing I can tell you there other than that, you know, a backup in Dr.

That hasn't been tested isn't really a backup in Dr.

And, uh, you, you, you just won't know.

If it's any good until you really, really, really need it.

And, uh, please don't do what we had the one guest do where he wiped

out his entire data center and then, and then did the restore.

Oh my lord, that story just gave me like a heart attack.

Um, yeah, I would say use the cloud as much as possible.

I realize it's maybe not possible for everything, but, um, you know, um.

Yeah.

Another thing that you can do is you can use new infrastructure to test the

recovery of current infrastructure, right?

Like so before you put a particular set of like if you're buying physical servers

before you buy, before you actually put those servers into production, consider

using those servers to conduct DR tests.

That's a thought.

I don't know, do you have any other, other ideas

I think, yeah, that's pretty much it.

I know it gets expensive and I think that's one of the things that people

sort of are like, oh, it's so costly from a time and money perspective that they

don't necessarily know how to approach it.

But I do like what you talked about.

Try to leverage infrastructure.

You could spin up and spin down quickly that doesn't require you to

own it and manage it the entire time.

Makes sense.

W. Curtis Preston: Right.

Um, so we'll, we'll finish with, uh, telling the story that you alluded

to earlier, and that is, uh, go back.

Um.

About, uh, 29 years, and I was, um, I was in the, you know,

I was working at the bank.

I had all the documentation I had for restorers.

It wasn't a Dr per se, but it was a recovery of a critical app.

And, uh, my daughter, uh, had just been born and I was in the hospital.

Um, you know, with my wife, you know, you know the whole classic

thing of like, father, do you know, son, daughter, brand new baby there.

And you're, you know, you're having that wonderful moment.

And the phone in my wife's hotel room rang.

Um, again, for those of you that are, are born in this world, this was a world

where we didn't have cell phones, right?

You meant the Hoss,

W. Curtis Preston: I had left

Prasanna Malaiyandi: meant the hospital room,

W. Curtis Preston: What's that?

Hotel

W. Curtis Preston: say?

Did I say a hotel room?

Yeah.

W. Curtis Preston: Um, so we were.

Uh, yeah.

So no cell phone.

And I had even left my beeper behind, right?

That was the way that you got in touch.

I'm like, I'm going to see my daughter and my wife.

I, you know, I, I don't need the beeper.

Uh, and what had happened while I was gone is that there had been this like really

big restore that had gone down and they.

Called me in my wife's hospital room.

Right.

They called the hospital 'cause they knew the hospital.

'cause the hospital was literally within line of sight to the, to the bank where

I worked, uh, where I worked, right.

This was Christian Hospital in Newark, Delaware.

Uh, 'cause I was at, what at that time was MBNA and it was just

literally right down the road.

The, the phone rang and, and I answered it, and it was somebody from

the bank, and they wanted to know if I could help them with this restore

that, you know, they were doing.

And I said, well, did you, did you like look at the document?

Because I made, I, I made such a thing about documentation.

Right.

Did you look at the documentation and was there a problem with the documentation?

Did you, you know, is there something, a question that I, a, you know,

that I, that I need to answer?

They basically started, um, they started saying, well, no, I hadn't

really looked at the documentation.

I just called you first.

And I just, I just hung up.

I just, I just, just like, I can't believe that you just called

me in my wife's hospital room.

Just after she'd given birth to our first child because of a stupid restore.

No matter how, you know, a a, you know, without even looking at the documentation.

This is why you need to have documentation.

This is why you need to make sure it's followed.

But of course, if you've got stupid people that don't wanna read

it, there's not much you can do.

But, um, all right.

Well, uh, as always, thanks a lot persona

thanks Curtis.

It's always fun.

W. Curtis Preston: And, uh, thank you to the listeners.

We'd be nothing without you.

That is a wrap.