W. Curtis Preston: One of the first steps, designing a disaster recovery

plan is figuring out where you're going to recover once disaster strikes.

If your data center or competing environment gets attacked by ransomware.

Swallowed up in a hurricane or a sinkhole, you better have an alternate

site lined up to restore your operations.

We're going to explore the details of the three main options most companies

have for their disaster recovery site.

Including rolling your own.

Set-up third-party Dr.

Services and leveraging the public cloud.

Each approach has its own pros, cons risks, and costs.

How do you keep a Dr.

Site in sync?

What happens if a regional disaster takes down the recovery provider?

Can the cloud scale to meet your Dr.

Needs.

In this episode, we answer all of these questions and more.

If this is your first time joining us.

Hi, I'm W.

Curtis Preston.

AKA Mr.

Backup.

I was a backup admin, just like you for many years.

And then I started helping other organizations design their backup and Dr.

Plans.

Now I'm using this podcast to turn unappreciated backup admins

into cyber recovery heroes.

This is the backup wrap-up.

Welcome to the show.

I'm your host, w Curtis Preston, and with me, I have the most flexible

co-host in the podcast world.

How's it going?

Prasanna?

I am good Curtis.

Yeah.

Uh, we did have some logistics issues today or this week, so, why don't

you tell folks where you're at?

Yeah.

W. Curtis Preston: so logistics being, I didn't realize I

was going to be nowhere near.

Where I normally record anything.

And so I happen to actually be at Pod Fest right now, which is a trade

show designed specifically for.

Podcasting.

And I will say that that is why there's background noise that

you're not used to hearing.

So if, if this is a little noisy, I apologize for it.

We'll do our best to trim that out.

But I'm sitting here with a lavalier on, uh, and I can see in my immediate

view, there's like, I don't know, a hundred people around me as I

try to record a podcast episode.

And by the way, I want to do a big thanks.

The Go Box Studio.

I I, I'm gonna take a picture of this, uh, setup that I have here.

So this is a unit that you can buy to do podcasting, literally Absolutely.

Anywhere.

So, so that's what we're doing.

We're doing, uh, I

one next time when you start traveling.

W. Curtis Preston: yeah, as I was walking around, I was like, Hey.

Any chance I could use your cool setup?

And he said, sure.

And I was, and so I said, I would mention the product,

it's Go Box Studio, it's great.

And then, then I called you and then I said, Hey, do you

wanna do record a podcast?

Like in five minutes?

He said, you said Okay.

So that's why I'm saying, you know, thanks for, thanks for your flexibility today.

no, I'm glad you were able to find a

space and some gear, you know.

W. Curtis Preston: Yeah, absolutely.

We're continuing in our backup to basic series.

Today we're talking about build, building a recovery site.

We're, of course, continuing to work through our book, uh, modern Data

Protection and, uh, that you can get.

Uh, there he goes.

Each of us hold up for a copy and, um.

We're starting to talk about Dr.

Right?

We talked about, uh, before we talked about the, you know, building,

uh, coming up with what's in a DR plan, what's not in a DR plan.

We also talked about RTO and RPA, right?

And or an RPO and RPA.

And now if you're going to do a disaster recovery, if you're going to recover

from a disaster, what will you need?

Prasanna.

You need someplace.

To recover too.

W. Curtis Preston: Yeah, exactly.

You need what we call a recovery site, right?

The idea is that you, you, you must obviously like this, should,

this should be obvious, but this is a backup to basic section.

So, uh, when the disaster strikes, right, you've got a

massive flood, you've got a fire.

You've got an earthquake.

I grew up in Florida.

A sinkhole could take your entire data center.

Um, you, you cannot assume that your data center will be, will be available

for you when you go to do a disaster.

Um, can you think, can you remember, uh, a little story from

our friend that was in, uh, the

Yeah, the basically an island nation in the Caribbean that

got hit by a hurricane and then, uh, lost power, lost internet connectivity

back to the mainland, uh, tons of damage.

And they were trying to bring up their remote office that was on the island and.

It couldn't connect back up to the mainland because that's where they

had all of their active directory and other services, so nothing could come

online, even though they had the people that they flew in from the mainland

on site, but they couldn't become operational until they got a whole

bunch of other things up and running.

W. Curtis Preston: Yeah, exactly.

They, and they did what the, the first of what we're gonna talk,

the three different options that you can do for a recovery site, um,

which is to build your own, right.

So they had, as I recall, they had three different.

Physical locations on the island, right?

And so they chose one as their recovery site.

Does that sound about right?

Yep.

That was

right.

W. Curtis Preston: So the, uh, so the first option, there

are three different options.

The first option is to roll your own DR site, and, uh, I'm sure

you've seen people do this quite a

bit.

exactly.

When I used to work at other storage vendors, well known names, um, there were

a lot of companies who would say, yes, I am going to have a production environment

stood up in one data center, and I'm going to buy either the exact same configuration

or something very similar and put it in another data center that I own, or

that I'm leasing in another location.

Sometimes it is within the same.

Campus, if you will.

So it's close by.

Sometimes it's further away depending on what they are looking to recover from.

Sometimes they might have multiple disaster recovery sites,

depending on the type of location,

sometimes they might have multiple disaster recovery sites, depending

on the type of disaster that occurs.

W. Curtis Preston: Right.

If they're a multi, uh, state, multi-county, multi, you know, whatever

they call it in other countries, then they could potentially use

the other parts of the country.

As their recovery site.

Right.

They, they use, like, so in our case, let's say we have, uh,

California and Florida, right?

And, uh, two states I picked just 'cause I used to live there.

Um, the, you know, your, your recovery site for California could

be in Florida and vice versa,

You know, that would be a really bad in one scenario.

Do you know what scenario that is

W. Curtis Preston: uh, what would it, which one?

having both sites in on the coast?

W. Curtis Preston: Oh yeah.

Yeah.

Um.

Actually, you know, that you, you, that you, you know that you bring that up.

Um, I did work at a company that had their recovery sites.

They did the, they built their own and they had their recovery

sites in two different places.

They had, um, one of them was in a data center in Dallas and

the other was in the basement of the World Trade Center building.

Right.

And uh, this is pre nine 11.

And there was a freak snowstorm in Dallas that took out that data center, and this

was the week of the, uh, bombing of the, that happened in the World Trade Center.

So basically two things happened, uh, at the same time, two

completely unrelated things.

Uh, sometimes the, you know, the best, best laid plans.

Um, so, but, but that's what you can do.

And you can have, you know, you could have, the more sites you

have, you can basically each.

Site can take over for another site.

The problem with that though, is that we need extra hardware, right?

We need, we need, uh, we need compute, we need storage, we need networking,

and all of that costs money.

Um, I'd say that's probably the biggest

Yep.

And.

W. Curtis Preston: downside to this, wouldn't

the well, and just going along with the money

aspect, don't forget the people, right, that you're gonna need on the

other site in order to maintain it.

And speaking of maintaining it, when you make a change in production, say you roll

out a new application or something else, you have to make sure that you're taking

into consideration the ongoing costs on your disaster recovery site as well.

So.

W. Curtis Preston: Yeah.

What, what I have seen is I've seen people, uh, basically as they refresh

hardware in the production site.

They move these slightly older hardware to the, uh, recovery site, and that's

a perfectly valid way to do things.

It's just you, you know, you're right.

You have to keep it up to date and it's, it's a bit weird because it's

like, it's like maintaining a car.

That you never get to drive.

Right?

My car,

W. Curtis Preston: you know what, it's actually have, I have a better analogy.

It's like me with the pool, right?

That I have, right?

I, I always wanted a pool.

Now there's a pool in the back of my yard, in, in my backyard,

and, uh, no one ever goes in it.

All I have to do is I'm the one that has to clean it, right?

I have to clean it.

And, uh, and all of the things that, that, that comes with the

pool, uh, I get to do all that work, uh, for a zero amount of joy.

And so what, what tends to happen is the site tends to get behind

thing, you know, and you don't really find out until you do a, a DR test.

And, uh, so that's, um, but I, I think just in short.

The reason why most people don't do this is the cost of doing it right?

That they're, that they have to buy all the hardware, you

have to get it in advance.

Um, and all of that hardware has to be taken care of and,

you know, all of that stuff.

Can you think of anything else that's a, like a downside of this method?

Um, no, uh, that's pretty much it.

Yeah.

I think it's just, uh, always sitting there, always.

Like idle, basically, like you said, now there are things you could do

to keep it sort of more functional.

You could use it for say, doing your backups.

I know some vendors and some companies like to do their backups off of their

DR copy just because otherwise those resources are just sitting idle.

And so that is one possible use of your DR site.

And I know we've talked about sort of what can you do with copies before.

So go back, listen to one of those episodes.

W. Curtis Preston: Right, the one on copy data management, I, yeah, absolutely.

Uh, the next option that we have is this idea of, uh, just like everything else,

we have recovery site as a service.

And the idea here is that there is a place that will make sure that they have enough

servers of a certain kind, enough storage, of a certain capacity, and, and also,

uh, capability and of course all of the networking equipment that you would need.

So that the idea is, uh, well, I'm just gonna say I was, I almost said

something, but I'm gonna, the historical idea here is you show up with your box

of tapes and then you do the recovery.

So, uh, I'm gonna throw that out there as one of the challenges

with recovery site as a service.

What, why might that be a challenge?

Because you have the, because of the time it

takes to rebuild stuff, right?

Was that what you were

W. Curtis Preston: close.

Well, so, so in the old days I would show up with a box of tapes.

How do I do that now?

Oh, you replicate the data

W. Curtis Preston: Right.

Which means that I need to have, right.

So in addition to providing me hardware and so, or hardware and networking.

When I need it, they also have to provide me, I have to pay

to basically keep a copy of my replicated backups there, which will not be free.

Yep.

Yeah, and I know that there are multiple companies who look at recovery as a

service, some of them pre-provision.

I always worry about these companies like what happens if I.

Like the New York area gets wiped out and everyone's trying to fail over to their

DR site in New Jersey and they under provisioned in that scenario because

they're not gonna keep all that hardware around to guarantee no over provisioning.

Right.

So.

W. Curtis Preston: Yeah.

The, I'd say that the recovery site as a service model, that

is its biggest Achilles heel.

Right?

So the, the first is that.

I mean, I mean, the good news is it's gonna be less expensive

than the first option, right?

Because you're not, because you're not paying for, I mean, you could, I, I'm sure

that, you know, and by the way, like an example of the kind of service that we're

talking about is like sunguard, right?

Um, and that company's been around a really long time and I, I think

they've morphed actually over, over the years to also do the third

method that we're gonna talk about.

But the.

You know, they have to make money.

And so if they're going to maintain an identical, you know, amount of

infrastructure that is dedicated to you, that's the key dedicated to you, then

that's going to cost you a ton of money.

It's, it's essentially going to be more expensive than option one.

Because you have the cost of option one plus their margin.

exactly.

W. Curtis Preston: So, so they're, they're gonna do that.

Um, but what you're, but what you're describing from before is

I think the more common model.

And that is that you have paid for the ability to have a

certain amount of infrastructure available to you in a disaster.

And the risk is that.

A bunch of you have a disaster.

Right?

And I, and, and, and it's not, it's not the same as the cloud

where there is this seemingly infinite amount of infrastructure.

You have a, you have an actual data center that is of a certain size that is meant

to be, uh, you know, it was provisioned.

For a certain number of customers declaring a disaster at the same time.

And then what happens when you have a glo not a global disaster, but

a, a regional disaster takes out everything that I, I think that's,

that's probably my biggest concern

I think, you know, I know we'll talk about it when

we talk about the third option, but honestly I don't think that's any

different than like a public cloud.

It's just the scale is different.

But we'll talk about that when we get to the third one.

Uh, the one other comment I wanted to make, and I don't know if you have

experience with this or not, Curtis, is do they still allow you to do, have like.

The agility and flexibility of say, my infrastructure is changing, how quickly

can I push that out to my DR site?

Or what's the process look like and how much of a lag is there?

W. Curtis Preston: Yeah, I think that it's going to be.

The, the lag will be the same as what would happen if you did

it plus some amount of time.

Right.

Again, it's, it's gonna be the cost of what you did it, plus of, of it's going

to be the cost of what it would be for you plus their margin in terms of the

lag time between production changes to changes in your recovery environment.

It's going to be the same amount of time because it co, it takes them the same

amount of time to order servers as it does for you, uh, plus some amount of time

for them to sort of figure that all out.

Right?

Um, and the o the only thing that may not be a problem here is that they probably

over proficient their environment,

I would hope so.

W. Curtis Preston: they, they hopefully over proficient their environment

because if you need to, there have been times in the last year or two where.

A provisioning in an order of a new server is like 90 to 120 days, right?

So hopefully they've ordered stuff in advance and they always have extra, you

know, capacity, which is something that they can afford to do that you might not.

And that's where the savings comes in.

Again, that savings comes at a risk of, uh, you know, a run on the bank.

Uh, have you please tell me you have seen.

Because, you know, I always, I know it's always with you and me.

I bring up a movie and you're like, I have never seen that.

Have you seen?

It's a wonderful life.

Okay.

All right.

You remember the run on the bank that happens at the end?

Yeah.

That's basically what you and I are worried about, right?

Is everybody coming up and wanting the resources all at once

because it's a, a, a disaster.

. Prasanna Malaiyandi: Do you know if when you go with the recovery as a

service option, if all of the providers allow you to customize a hardware

to what is there in production?

So if I use a particular vendor.

Would they allow me to use that or is it sort of, here are your standard options?

Go forth.

W. Curtis Preston: I, I think both of those are.

Right.

Every, you know, uh, this is business.

Everything is negotiable, right?

Use our standard options and it costs you this much.

Start customizing it and it starts and it starts costing more.

Right?

Um, and so this is where the fact that most of us are running Wintel

servers, right, really helps out.

Most of us are doing virtualization, most of us are using, um, you know,

uh, Broadcom, I think that's the new name of the, of the company, right?

And, and so as long as you're sticking with sort of the standard things,

uh, from a hardware perspective, it's probably not a problem.

But if you decide I've got to have vendor X, then um, you know, uh,

your cost is gonna go up because they're gonna buy that hardware just

for you, again, plus margin, right?

So, um, I, I think it's, it's, this is like.

Every one of these choices.

None of these are perfect for everyone, right?

Let's talk about the third option and, and, and I'm sure everybody

knows what the third option is, and that is recover in the public cloud.

And the idea here is that they have so much infrastructure.

That you can very easily on demand with little planning.

I mean, you, you need to do planning, but it's not the same as with, um,

the recovery site as a service.

And that is you.

Um, you can at any time just literally press a button and you

can do infrastructure as code and poof, you've got as many.

Um, virtualization servers or VMs or, uh, you know, RDBMSs that

you need, uh, and that you don't really, you know, within reason.

You don't really have to do any planning ahead.

You just have to make sure that your billing system works or you wanna

talk your, you're disagreeing with me.

I just want, 'cause there is a lot of

misconceptions about the public cloud.

Just remember it's someone else's servers, you're just borrowing them, right?

And so.

W. Curtis Preston: Yeah.

If, say an example is if you took AWS and they

just didn't have the infrastructure to meet your DR needs in that region,

then you may not be able to recover.

So make sure that you have a conversation, especially if you're at that large

of a scale with your cloud provider, to make sure that if you do need

those resources, they're available because someone else may be using it.

W. Curtis Preston: Yeah.

And, and I guess that's what I meant by, I, I don't remember my exact words,

but, but you need to do planning.

You don't, what I'm saying is you don't need to do planning, like we're

gonna declare a disaster Friday.

Right.

I, I mean, even then it is not a bad thing to.

Actually, if, if you're doing testing, I would do zero amount of

notification to the cloud vendor.

That's what I would do.

And just to see how they respond.

I, I guess you, you, you do need to do that type of planning.

Um, what I'm just saying is

You don't need to pre for provision.

Yeah.

W. Curtis Preston: yeah.

You don't need to pre-provision that they should have enough infrastructure

and you can have those conversations.

You say, listen, when we declare a disaster, we're gonna need.

Thousand VMs, is that gonna be a problem?

And they're gonna either gonna go 50,000, I thought you were gonna say a million.

Or they're gonna say, uh, yeah, you should probably give us a call.

Right.

Um.

But I, I think that the response is probably gonna be more the latter.

Right?

And, um, the, the other beautiful thing about the co, about the cloud is that,

and this is not true on either of the other two options, is the cost, because.

In order for a cloud DR system to work you, you need to have the,

the data there already, right?

Um, and, and so you need to do, you need to be backing up essentially

to the cloud, but that's all you're paying for at that point.

You're paying for the storage of.

You know the amount of storage that you need from that cloud vendor,

depending on how you're doing it.

It could be block storage, it could be object storage one, you

know, the object storage is gonna be the less expensive option.

The block storage is going to be the ready to use it anytime you want it option.

But the point is that 'cause with, with object storage, you're going to have to

restore it to, to block storage, right?

So it's not gonna be.

As quick as a restore if you, if you've essentially already restored your servers.

But the beautiful thing is that you don't have to pay for the compute until the time

of, uh, testing or declaring a disaster.

One other thing to add to that, that I think is a

big advantage of the public cloud is I don't actually need the exact same

resources compute wise on in the cloud for my testing as I do for production.

I.

So for testing, I might pick smaller instances, lower powered compute

in order to do my testing because I don't need to spin up the beefy

systems that I would need to actually run the production workloads.

W. Curtis Preston: Yeah, that is a beautiful thing where,

because again, you can't really do that in the first two options.

You, you just sort of, you get what you get, right?

You don't throw a fit, but here you can say, look, I, I need, you

know, I need an M four X large for production, but, you know, an M two

medium will do fine for my test.

The performance isn't gonna be great.

I'm not testing the performance, I'm testing the functionality

of the recovery process.

Uh, but, but I think honestly the, the greatest advantage

here is going to be cost.

And I think that, and also that all of that infrastructure is available as code.

What do I mean by that?

Basically you don't need to go around configuring things.

You could code it all up, have it all available, and basically push

a button and say, okay, go build me my environment based, right?

All based on what you've already pre-configured, and it'll go

do everything you needed to do.

And you don't have to sit there and manually spin up things and provision,

connect things with networks, go log into 20 different switches and environments in

order to get everything up and running.

W. Curtis Preston: Yeah, I mean, you know, we saw this at

our previous employer, right?

That, that you can automate that and that you can, assuming you did enough, you

know, planning, uh, and you configured everything correctly, you should just

be able to literally press a button and then it configures everything that

you need to do, just like you said.

And then just as magical, you can shut it all down, right?

And as soon as you shut it all down, the billing stops.

Right?

That, um, you know, basically you just pay for the amount of

time that that infrastructure was

Yep.

Which hopefully means,

yeah, and that also hopefully means you can do testing more often because,

what do you like to say, Curtis?

W. Curtis Preston: Yeah.

Test, test, test, test, test.

Yeah.

Um, now I'm gonna say, I'm gonna, I'm gonna argue with

something and you said like.

15 minutes ago.

And that is, this is, here's another thing that I like about the public

cloud versus the recovery as a service.

While it is true, the public cloud is not unlimited, what I can do in the

public cloud that I can't really do with a DR as a service is I can pick

whatever region I want to recover to.

If I am, if I'm a, you know, a business in New York, I will

most likely pick, uh, a, um.

You know, like we talk about SunGard, I will pick a SunGard

that's in New Jersey, right?

You're gonna pick something that is close by because you are most likely physically

going to the place to make this happen.

With the cloud, you are never gonna see that data center.

And so it doesn't matter where you pick it.

Well, let me rephrase that.

It doesn't matter where you pick it, but it doesn't have

to be down the road, right?

And so you're not gonna have this.

It, it, it significantly decreases the, the, the worry that there's a flood in

the general vicinity where you live and it takes out all the businesses, and

then it also takes out the, you know, the recovery site and everybody's using

the recovery site all at the same time.

I'm just saying you can spread the load out is all

even the recovery as a service, you could technically go

from New York to Texas, assuming that your recovery as a service provider

is in Texas, replicate the data there.

Yes, it is gonna be more painful because you may not be on site, but.

W. Curtis Preston: It, it, it is true.

Yeah.

That is true.

I, I guess, I guess I'm just thinking that generally speaking,

these types of recoveries were done.

person.

W. Curtis Preston: Face to face.

Yeah.

Um, just I have this,

And in your mind, I know Veeam has a lot of DR partners

that can take Veeam backups and spin them up and use 'em for DR purposes.

Would you consider that as option two, recovery as a service, or

more in line with option three?

Even though we're using the word public cloud, there.

W. Curtis Preston: Yeah.

I, I think, you know, like, you know, you're talking about like

cloud IBR, like those guys.

I.

I, I think that that is, that is because you're, you're generally, you're using

the public cloud as the underlying infrastructure, even if you are, well,

I, I think don't they all use the public

They might have their own, they might have their own

infra, but I think it's still more in line with the public cloud rather than the

W. Curtis Preston: yeah.

I, I think the way they behave, it behaves more like the public cloud.

Um, and I like this, right, because it.

It doesn't require you to use a different piece of software for Dr.

I like this idea of using one piece of software to do, to do both backup and Dr.

It saves a ton of money for the company as long as it meets your

RTO and RPO requirements, right?

Um, and it backs up all of the applications that

you need to back up then.

Um, you know, it, it's a great way to do it.

Um, can you think of anything else that we like, we haven't covered?

no, I think that covers all the option.

Well, there is a fourth option,

W. Curtis Preston: All right.

Dang it.

There's no, I'm sorry.

I'm looking in my book and I don't see a fourth

Which is basically you don't have a

disaster recovery site and you're

host.

W. Curtis Preston: You had me worried there for a minute.

Yeah.

That is the fourths option.

No recovery site, no backup.

No plan, no joy.

Um, yeah, don't do that.

Um, so I, I think if, if you can learn anything from I.

You know, what we've been covering is that plan, plan, plan, test, test, test,

and uh, also, uh, update your software and have a good password management

system and have MFA, all of those things.

And, um, yeah, so now's, you know, the, uh, what's the thing of, uh, uh, what's,

what's that old thing of like, the best time to plan something is yesterday?

The next

W. Curtis Preston: Uh, the second best time is today.

Something like that.

Something like that.

There's some, uh, something like that.

Well, anyway, fun as always, Prasanna.

Likewise, Curtis.

W. Curtis Preston: All right, and I hope, uh, you listeners enjoyed that.

We do it for you and, uh, be sure to subscribe.

We don't want you to miss any episodes.

It's so much easier.

It just pops up on your phone there and, uh, you can listen to our cheerful voices.

That is a wrap.