W. Curtis Preston: One of the first steps, designing a disaster recovery
plan is figuring out where you're going to recover once disaster strikes.
If your data center or competing environment gets attacked by ransomware.
Swallowed up in a hurricane or a sinkhole, you better have an alternate
site lined up to restore your operations.
We're going to explore the details of the three main options most companies
have for their disaster recovery site.
Including rolling your own.
Set-up third-party Dr.
Services and leveraging the public cloud.
Each approach has its own pros, cons risks, and costs.
How do you keep a Dr.
Site in sync?
What happens if a regional disaster takes down the recovery provider?
Can the cloud scale to meet your Dr.
Needs.
In this episode, we answer all of these questions and more.
If this is your first time joining us.
Hi, I'm W.
Curtis Preston.
AKA Mr.
Backup.
I was a backup admin, just like you for many years.
And then I started helping other organizations design their backup and Dr.
Plans.
Now I'm using this podcast to turn unappreciated backup admins
into cyber recovery heroes.
This is the backup wrap-up.
Welcome to the show.
I'm your host, w Curtis Preston, and with me, I have the most flexible
co-host in the podcast world.
How's it going?
Prasanna?
Prasanna Malaiyandi: I am good Curtis.
Yeah.
Uh, we did have some logistics issues today or this week, so, why don't
you tell folks where you're at?
Yeah.
W. Curtis Preston: so logistics being, I didn't realize I
was going to be nowhere near.
Where I normally record anything.
And so I happen to actually be at Pod Fest right now, which is a trade
show designed specifically for.
Podcasting.
And I will say that that is why there's background noise that
you're not used to hearing.
So if, if this is a little noisy, I apologize for it.
We'll do our best to trim that out.
But I'm sitting here with a lavalier on, uh, and I can see in my immediate
view, there's like, I don't know, a hundred people around me as I
try to record a podcast episode.
And by the way, I want to do a big thanks.
The Go Box Studio.
I I, I'm gonna take a picture of this, uh, setup that I have here.
So this is a unit that you can buy to do podcasting, literally Absolutely.
Anywhere.
So, so that's what we're doing.
We're doing, uh, I
Prasanna Malaiyandi: one next time when you start traveling.
W. Curtis Preston: yeah, as I was walking around, I was like, Hey.
Any chance I could use your cool setup?
And he said, sure.
And I was, and so I said, I would mention the product,
it's Go Box Studio, it's great.
And then, then I called you and then I said, Hey, do you
wanna do record a podcast?
Like in five minutes?
He said, you said Okay.
So that's why I'm saying, you know, thanks for, thanks for your flexibility today.
Prasanna Malaiyandi: no, I'm glad you were able to find a
space and some gear, you know.
W. Curtis Preston: Yeah, absolutely.
We're continuing in our backup to basic series.
Today we're talking about build, building a recovery site.
We're, of course, continuing to work through our book, uh, modern Data
Protection and, uh, that you can get.
Uh, there he goes.
Each of us hold up for a copy and, um.
We're starting to talk about Dr.
Right?
We talked about, uh, before we talked about the, you know, building,
uh, coming up with what's in a DR plan, what's not in a DR plan.
We also talked about RTO and RPA, right?
And or an RPO and RPA.
And now if you're going to do a disaster recovery, if you're going to recover
from a disaster, what will you need?
Prasanna.
Prasanna Malaiyandi: You need someplace.
To recover too.
W. Curtis Preston: Yeah, exactly.
You need what we call a recovery site, right?
The idea is that you, you, you must obviously like this, should,
this should be obvious, but this is a backup to basic section.
So, uh, when the disaster strikes, right, you've got a
massive flood, you've got a fire.
You've got an earthquake.
I grew up in Florida.
A sinkhole could take your entire data center.
Um, you, you cannot assume that your data center will be, will be available
for you when you go to do a disaster.
Um, can you think, can you remember, uh, a little story from
our friend that was in, uh, the
Prasanna Malaiyandi: Yeah, the basically an island nation in the Caribbean that
got hit by a hurricane and then, uh, lost power, lost internet connectivity
back to the mainland, uh, tons of damage.
And they were trying to bring up their remote office that was on the island and.
It couldn't connect back up to the mainland because that's where they
had all of their active directory and other services, so nothing could come
online, even though they had the people that they flew in from the mainland
on site, but they couldn't become operational until they got a whole
bunch of other things up and running.
W. Curtis Preston: Yeah, exactly.
They, and they did what the, the first of what we're gonna talk,
the three different options that you can do for a recovery site, um,
which is to build your own, right.
So they had, as I recall, they had three different.
Physical locations on the island, right?
And so they chose one as their recovery site.
Does that sound about right?
Prasanna Malaiyandi: Yep.
That was
right.
W. Curtis Preston: So the, uh, so the first option, there
are three different options.
The first option is to roll your own DR site, and, uh, I'm sure
you've seen people do this quite a
bit.
Prasanna Malaiyandi: exactly.
When I used to work at other storage vendors, well known names, um, there were
a lot of companies who would say, yes, I am going to have a production environment
stood up in one data center, and I'm going to buy either the exact same configuration
or something very similar and put it in another data center that I own, or
that I'm leasing in another location.
Sometimes it is within the same.
Campus, if you will.
So it's close by.
Sometimes it's further away depending on what they are looking to recover from.
Sometimes they might have multiple disaster recovery sites,
depending on the type of location,
sometimes they might have multiple disaster recovery sites, depending
on the type of disaster that occurs.
W. Curtis Preston: Right.
If they're a multi, uh, state, multi-county, multi, you know, whatever
they call it in other countries, then they could potentially use
the other parts of the country.
As their recovery site.
Right.
They, they use, like, so in our case, let's say we have, uh,
California and Florida, right?
And, uh, two states I picked just 'cause I used to live there.
Um, the, you know, your, your recovery site for California could
be in Florida and vice versa,
Prasanna Malaiyandi: You know, that would be a really bad in one scenario.
Do you know what scenario that is
W. Curtis Preston: uh, what would it, which one?
Prasanna Malaiyandi: having both sites in on the coast?
W. Curtis Preston: Oh yeah.
Yeah.
Um.
Actually, you know, that you, you, that you, you know that you bring that up.
Um, I did work at a company that had their recovery sites.
They did the, they built their own and they had their recovery
sites in two different places.
They had, um, one of them was in a data center in Dallas and
the other was in the basement of the World Trade Center building.
Right.
And uh, this is pre nine 11.
And there was a freak snowstorm in Dallas that took out that data center, and this
was the week of the, uh, bombing of the, that happened in the World Trade Center.
So basically two things happened, uh, at the same time, two
completely unrelated things.
Uh, sometimes the, you know, the best, best laid plans.
Um, so, but, but that's what you can do.
And you can have, you know, you could have, the more sites you
have, you can basically each.
Site can take over for another site.
The problem with that though, is that we need extra hardware, right?
We need, we need, uh, we need compute, we need storage, we need networking,
and all of that costs money.
Um, I'd say that's probably the biggest
Prasanna Malaiyandi: Yep.
And.
W. Curtis Preston: downside to this, wouldn't
Prasanna Malaiyandi: the well, and just going along with the money
aspect, don't forget the people, right, that you're gonna need on the
other site in order to maintain it.
And speaking of maintaining it, when you make a change in production, say you roll
out a new application or something else, you have to make sure that you're taking
into consideration the ongoing costs on your disaster recovery site as well.
So.
W. Curtis Preston: Yeah.
What, what I have seen is I've seen people, uh, basically as they refresh
hardware in the production site.
They move these slightly older hardware to the, uh, recovery site, and that's
a perfectly valid way to do things.
It's just you, you know, you're right.
You have to keep it up to date and it's, it's a bit weird because it's
like, it's like maintaining a car.
That you never get to drive.
Right?
Prasanna Malaiyandi: My car,
W. Curtis Preston: you know what, it's actually have, I have a better analogy.
It's like me with the pool, right?
That I have, right?
I, I always wanted a pool.
Now there's a pool in the back of my yard, in, in my backyard,
and, uh, no one ever goes in it.
All I have to do is I'm the one that has to clean it, right?
I have to clean it.
And, uh, and all of the things that, that, that comes with the
pool, uh, I get to do all that work, uh, for a zero amount of joy.
And so what, what tends to happen is the site tends to get behind
thing, you know, and you don't really find out until you do a, a DR test.
And, uh, so that's, um, but I, I think just in short.
The reason why most people don't do this is the cost of doing it right?
That they're, that they have to buy all the hardware, you
have to get it in advance.
Um, and all of that hardware has to be taken care of and,
you know, all of that stuff.
Can you think of anything else that's a, like a downside of this method?
Prasanna Malaiyandi: Um, no, uh, that's pretty much it.
Yeah.
I think it's just, uh, always sitting there, always.
Like idle, basically, like you said, now there are things you could do
to keep it sort of more functional.
You could use it for say, doing your backups.
I know some vendors and some companies like to do their backups off of their
DR copy just because otherwise those resources are just sitting idle.
And so that is one possible use of your DR site.
And I know we've talked about sort of what can you do with copies before.
So go back, listen to one of those episodes.
W. Curtis Preston: Right, the one on copy data management, I, yeah, absolutely.
Uh, the next option that we have is this idea of, uh, just like everything else,
we have recovery site as a service.
And the idea here is that there is a place that will make sure that they have enough
servers of a certain kind, enough storage, of a certain capacity, and, and also,
uh, capability and of course all of the networking equipment that you would need.
So that the idea is, uh, well, I'm just gonna say I was, I almost said
something, but I'm gonna, the historical idea here is you show up with your box
of tapes and then you do the recovery.
So, uh, I'm gonna throw that out there as one of the challenges
with recovery site as a service.
What, why might that be a challenge?
Prasanna Malaiyandi: Because you have the, because of the time it
takes to rebuild stuff, right?
Was that what you were
W. Curtis Preston: close.
Well, so, so in the old days I would show up with a box of tapes.
How do I do that now?
Prasanna Malaiyandi: Oh, you replicate the data
W. Curtis Preston: Right.
Which means that I need to have, right.
So in addition to providing me hardware and so, or hardware and networking.
When I need it, they also have to provide me, I have to pay
to basically keep a copy of my replicated backups there, which will not be free.
Prasanna Malaiyandi: Yep.
Yeah, and I know that there are multiple companies who look at recovery as a
service, some of them pre-provision.
I always worry about these companies like what happens if I.
Like the New York area gets wiped out and everyone's trying to fail over to their
DR site in New Jersey and they under provisioned in that scenario because
they're not gonna keep all that hardware around to guarantee no over provisioning.
Right.
So.
W. Curtis Preston: Yeah.
The, I'd say that the recovery site as a service model, that
is its biggest Achilles heel.
Right?
So the, the first is that.
I mean, I mean, the good news is it's gonna be less expensive
than the first option, right?
Because you're not, because you're not paying for, I mean, you could, I, I'm sure
that, you know, and by the way, like an example of the kind of service that we're
talking about is like sunguard, right?
Um, and that company's been around a really long time and I, I think
they've morphed actually over, over the years to also do the third
method that we're gonna talk about.
But the.
You know, they have to make money.
And so if they're going to maintain an identical, you know, amount of
infrastructure that is dedicated to you, that's the key dedicated to you, then
that's going to cost you a ton of money.
It's, it's essentially going to be more expensive than option one.
Because you have the cost of option one plus their margin.
Prasanna Malaiyandi: exactly.
W. Curtis Preston: So, so they're, they're gonna do that.
Um, but what you're, but what you're describing from before is
I think the more common model.
And that is that you have paid for the ability to have a
certain amount of infrastructure available to you in a disaster.
And the risk is that.
A bunch of you have a disaster.
Right?
And I, and, and, and it's not, it's not the same as the cloud
where there is this seemingly infinite amount of infrastructure.
You have a, you have an actual data center that is of a certain size that is meant
to be, uh, you know, it was provisioned.
For a certain number of customers declaring a disaster at the same time.
And then what happens when you have a glo not a global disaster, but
a, a regional disaster takes out everything that I, I think that's,
that's probably my biggest concern
Prasanna Malaiyandi: I think, you know, I know we'll talk about it when
we talk about the third option, but honestly I don't think that's any
different than like a public cloud.
It's just the scale is different.
But we'll talk about that when we get to the third one.
Uh, the one other comment I wanted to make, and I don't know if you have
experience with this or not, Curtis, is do they still allow you to do, have like.
The agility and flexibility of say, my infrastructure is changing, how quickly
can I push that out to my DR site?
Or what's the process look like and how much of a lag is there?
W. Curtis Preston: Yeah, I think that it's going to be.
The, the lag will be the same as what would happen if you did
it plus some amount of time.
Right.
Again, it's, it's gonna be the cost of what you did it, plus of, of it's going
to be the cost of what it would be for you plus their margin in terms of the
lag time between production changes to changes in your recovery environment.
It's going to be the same amount of time because it co, it takes them the same
amount of time to order servers as it does for you, uh, plus some amount of time
for them to sort of figure that all out.
Right?
Um, and the o the only thing that may not be a problem here is that they probably
over proficient their environment,
Prasanna Malaiyandi: I would hope so.
W. Curtis Preston: they, they hopefully over proficient their environment
because if you need to, there have been times in the last year or two where.
A provisioning in an order of a new server is like 90 to 120 days, right?
So hopefully they've ordered stuff in advance and they always have extra, you
know, capacity, which is something that they can afford to do that you might not.
And that's where the savings comes in.
Again, that savings comes at a risk of, uh, you know, a run on the bank.
Uh, have you please tell me you have seen.
Because, you know, I always, I know it's always with you and me.
I bring up a movie and you're like, I have never seen that.
Have you seen?
It's a wonderful life.
Okay.
All right.
You remember the run on the bank that happens at the end?
Yeah.
That's basically what you and I are worried about, right?
Is everybody coming up and wanting the resources all at once
because it's a, a, a disaster.
. Prasanna Malaiyandi: Do you know if when you go with the recovery as a
service option, if all of the providers allow you to customize a hardware
to what is there in production?
So if I use a particular vendor.
Would they allow me to use that or is it sort of, here are your standard options?
Go forth.
W. Curtis Preston: I, I think both of those are.
Right.
Every, you know, uh, this is business.
Everything is negotiable, right?
Use our standard options and it costs you this much.
Start customizing it and it starts and it starts costing more.
Right?
Um, and so this is where the fact that most of us are running Wintel
servers, right, really helps out.
Most of us are doing virtualization, most of us are using, um, you know,
uh, Broadcom, I think that's the new name of the, of the company, right?
And, and so as long as you're sticking with sort of the standard things,
uh, from a hardware perspective, it's probably not a problem.
But if you decide I've got to have vendor X, then um, you know, uh,
your cost is gonna go up because they're gonna buy that hardware just
for you, again, plus margin, right?
So, um, I, I think it's, it's, this is like.
Every one of these choices.
None of these are perfect for everyone, right?
Let's talk about the third option and, and, and I'm sure everybody
knows what the third option is, and that is recover in the public cloud.
And the idea here is that they have so much infrastructure.
That you can very easily on demand with little planning.
I mean, you, you need to do planning, but it's not the same as with, um,
the recovery site as a service.
And that is you.
Um, you can at any time just literally press a button and you
can do infrastructure as code and poof, you've got as many.
Um, virtualization servers or VMs or, uh, you know, RDBMSs that
you need, uh, and that you don't really, you know, within reason.
You don't really have to do any planning ahead.
You just have to make sure that your billing system works or you wanna
talk your, you're disagreeing with me.
Prasanna Malaiyandi: I just want, 'cause there is a lot of
misconceptions about the public cloud.
Just remember it's someone else's servers, you're just borrowing them, right?
And so.
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: If, say an example is if you took AWS and they
just didn't have the infrastructure to meet your DR needs in that region,
then you may not be able to recover.
So make sure that you have a conversation, especially if you're at that large
of a scale with your cloud provider, to make sure that if you do need
those resources, they're available because someone else may be using it.
W. Curtis Preston: Yeah.
And, and I guess that's what I meant by, I, I don't remember my exact words,
but, but you need to do planning.
You don't, what I'm saying is you don't need to do planning, like we're
gonna declare a disaster Friday.
Right.
I, I mean, even then it is not a bad thing to.
Actually, if, if you're doing testing, I would do zero amount of
notification to the cloud vendor.
That's what I would do.
And just to see how they respond.
I, I guess you, you, you do need to do that type of planning.
Um, what I'm just saying is
Prasanna Malaiyandi: You don't need to pre for provision.
Yeah.
W. Curtis Preston: yeah.
You don't need to pre-provision that they should have enough infrastructure
and you can have those conversations.
You say, listen, when we declare a disaster, we're gonna need.
Thousand VMs, is that gonna be a problem?
And they're gonna either gonna go 50,000, I thought you were gonna say a million.
Or they're gonna say, uh, yeah, you should probably give us a call.
Right.
Um.
But I, I think that the response is probably gonna be more the latter.
Right?
And, um, the, the other beautiful thing about the co, about the cloud is that,
and this is not true on either of the other two options, is the cost, because.
In order for a cloud DR system to work you, you need to have the,
the data there already, right?
Um, and, and so you need to do, you need to be backing up essentially
to the cloud, but that's all you're paying for at that point.
You're paying for the storage of.
You know the amount of storage that you need from that cloud vendor,
depending on how you're doing it.
It could be block storage, it could be object storage one, you
know, the object storage is gonna be the less expensive option.
The block storage is going to be the ready to use it anytime you want it option.
But the point is that 'cause with, with object storage, you're going to have to
restore it to, to block storage, right?
So it's not gonna be.
As quick as a restore if you, if you've essentially already restored your servers.
But the beautiful thing is that you don't have to pay for the compute until the time
of, uh, testing or declaring a disaster.
Prasanna Malaiyandi: One other thing to add to that, that I think is a
big advantage of the public cloud is I don't actually need the exact same
resources compute wise on in the cloud for my testing as I do for production.
I.
So for testing, I might pick smaller instances, lower powered compute
in order to do my testing because I don't need to spin up the beefy
systems that I would need to actually run the production workloads.
W. Curtis Preston: Yeah, that is a beautiful thing where,
because again, you can't really do that in the first two options.
You, you just sort of, you get what you get, right?
You don't throw a fit, but here you can say, look, I, I need, you
know, I need an M four X large for production, but, you know, an M two
medium will do fine for my test.
The performance isn't gonna be great.
I'm not testing the performance, I'm testing the functionality
of the recovery process.
Uh, but, but I think honestly the, the greatest advantage
here is going to be cost.
And I think that, and also that all of that infrastructure is available as code.
What do I mean by that?
Prasanna Malaiyandi: Basically you don't need to go around configuring things.
You could code it all up, have it all available, and basically push
a button and say, okay, go build me my environment based, right?
All based on what you've already pre-configured, and it'll go
do everything you needed to do.
And you don't have to sit there and manually spin up things and provision,
connect things with networks, go log into 20 different switches and environments in
order to get everything up and running.
W. Curtis Preston: Yeah, I mean, you know, we saw this at
our previous employer, right?
That, that you can automate that and that you can, assuming you did enough, you
know, planning, uh, and you configured everything correctly, you should just
be able to literally press a button and then it configures everything that
you need to do, just like you said.
And then just as magical, you can shut it all down, right?
And as soon as you shut it all down, the billing stops.
Right?
That, um, you know, basically you just pay for the amount of
time that that infrastructure was
Prasanna Malaiyandi: Yep.
Which hopefully means,
yeah, and that also hopefully means you can do testing more often because,
what do you like to say, Curtis?
W. Curtis Preston: Yeah.
Test, test, test, test, test.
Yeah.
Um, now I'm gonna say, I'm gonna, I'm gonna argue with
something and you said like.
15 minutes ago.
And that is, this is, here's another thing that I like about the public
cloud versus the recovery as a service.
While it is true, the public cloud is not unlimited, what I can do in the
public cloud that I can't really do with a DR as a service is I can pick
whatever region I want to recover to.
If I am, if I'm a, you know, a business in New York, I will
most likely pick, uh, a, um.
You know, like we talk about SunGard, I will pick a SunGard
that's in New Jersey, right?
You're gonna pick something that is close by because you are most likely physically
going to the place to make this happen.
With the cloud, you are never gonna see that data center.
And so it doesn't matter where you pick it.
Well, let me rephrase that.
It doesn't matter where you pick it, but it doesn't have
to be down the road, right?
And so you're not gonna have this.
It, it, it significantly decreases the, the, the worry that there's a flood in
the general vicinity where you live and it takes out all the businesses, and
then it also takes out the, you know, the recovery site and everybody's using
the recovery site all at the same time.
I'm just saying you can spread the load out is all
Prasanna Malaiyandi: even the recovery as a service, you could technically go
from New York to Texas, assuming that your recovery as a service provider
is in Texas, replicate the data there.
Yes, it is gonna be more painful because you may not be on site, but.
W. Curtis Preston: It, it, it is true.
Yeah.
That is true.
I, I guess, I guess I'm just thinking that generally speaking,
these types of recoveries were done.
Prasanna Malaiyandi: person.
W. Curtis Preston: Face to face.
Yeah.
Um, just I have this,
Prasanna Malaiyandi: And in your mind, I know Veeam has a lot of DR partners
that can take Veeam backups and spin them up and use 'em for DR purposes.
Would you consider that as option two, recovery as a service, or
more in line with option three?
Even though we're using the word public cloud, there.
W. Curtis Preston: Yeah.
I, I think, you know, like, you know, you're talking about like
cloud IBR, like those guys.
I.
I, I think that that is, that is because you're, you're generally, you're using
the public cloud as the underlying infrastructure, even if you are, well,
I, I think don't they all use the public
Prasanna Malaiyandi: They might have their own, they might have their own
infra, but I think it's still more in line with the public cloud rather than the
W. Curtis Preston: yeah.
I, I think the way they behave, it behaves more like the public cloud.
Um, and I like this, right, because it.
It doesn't require you to use a different piece of software for Dr.
I like this idea of using one piece of software to do, to do both backup and Dr.
It saves a ton of money for the company as long as it meets your
RTO and RPO requirements, right?
Um, and it backs up all of the applications that
you need to back up then.
Um, you know, it, it's a great way to do it.
Um, can you think of anything else that we like, we haven't covered?
Prasanna Malaiyandi: no, I think that covers all the option.
Well, there is a fourth option,
W. Curtis Preston: All right.
Dang it.
There's no, I'm sorry.
I'm looking in my book and I don't see a fourth
Prasanna Malaiyandi: Which is basically you don't have a
disaster recovery site and you're
host.
W. Curtis Preston: You had me worried there for a minute.
Yeah.
That is the fourths option.
No recovery site, no backup.
No plan, no joy.
Um, yeah, don't do that.
Um, so I, I think if, if you can learn anything from I.
You know, what we've been covering is that plan, plan, plan, test, test, test,
and uh, also, uh, update your software and have a good password management
system and have MFA, all of those things.
And, um, yeah, so now's, you know, the, uh, what's the thing of, uh, uh, what's,
what's that old thing of like, the best time to plan something is yesterday?
Prasanna Malaiyandi: The next
W. Curtis Preston: Uh, the second best time is today.
Something like that.
There's some, uh, something like that.
Well, anyway, fun as always, Prasanna.
Prasanna Malaiyandi: Likewise, Curtis.
W. Curtis Preston: All right, and I hope, uh, you listeners enjoyed that.
We do it for you and, uh, be sure to subscribe.
We don't want you to miss any episodes.
It's so much easier.
It just pops up on your phone there and, uh, you can listen to our cheerful voices.
That is a wrap.