You found the backup wrap up your go-to podcast for all things
backup recovery and cyber recovery.
In this episode, we talk about some hard earned disaster recovery lessons
from major events like nine 11.
We talk about what we learned about DR that day, and we talk about
those critical human elements of DR.
That people often forget, like where your recovery team is going to
sleep when the hotels are all gone.
Disasters happen and they're never convenient.
Whether it's terrorists, hurricanes, or ransomware, you
need to think through what you do.
If you're completely isolated from the world, the time to
learn these lessons are now.
So I hope you enjoy this episode.
By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr. Backup,
and I've been passionate about backup and disaster recovery for over 30 years.
Ever since.
I had to tell my boss that there were no backups of the
big database that we just lost.
I don't want that to happen to you, and that's why I do this podcast.
On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes.
This is the backup wrap up.
Welcome to the show.
Hi, I am w Curtis Preston, AKA, Mr. Backup, and I have with me a guy who
is as lazy as I am when it comes to how to move rugs around the house.
Prasanna Malaiyandi, how's it going?
Prasanna.
I am doing well, Curtis, by the way, isn't it amazing how quiet a room gets
when you put something on the floor?
Yeah, I, I, I wonder if anybody, uh, if anybody notices the difference
in sound because it, you know, and it's been, they probably just got
so used to the other sound, right?
Because I. I went with LVP, you know, upstairs and um, and then in
my office and it just got so echoy and I put the stuff on this wall.
And this over here is a acoustic panels.
Um, you know, behind me there are acoustic panels up on the ceiling
and it still wasn't the same.
I. And then I found, you know, I, I, I, this, this rug is from actually my
living room because we bought a nicer rug and, uh, a nicer, bigger rug.
And so we put that one down there and then I moved here.
But when I moved it in, I, I, I did what I thought was like
a really weird way to do it.
I didn't actually move the furniture.
I was like lifting it up and trying to do it without having to move everything out.
And you said, well, that's exactly how I did it.
exactly.
Yeah.
And I, I, I thought it was just me, but apparently.
It wasn't.
because at least with the rug, well, it depends on how your desk is too.
But like you can at least unfurl part of it and just kind of like shove it down.
Like you just lift
up the front part and
then you lift up the back part.
So,
yeah.
I, I think different people might go, no, no, we're gonna,
we're gonna move everything out.
We're gonna put the rug down and we'll move everything back in.
And they might think of that as easier, but I was like, I
don't wanna move everything.
well, it's more than moving.
It's setting everything back up again.
Yeah, I, I looked into it.
I could have potentially moved the desk without disassembling everything.
It was just, you know, but, um, yeah, so I, it was just nice to hear
when I was talking to you for once.
You weren't like, 'cause so many times I tell you about something that I'm
doing and then you're like, that's the dumbest thing I've ever heard.
Why would you do it that way?
And you're like, oh no, that's, uh, that's how I do it.
Yeah.
Especially if you're space limited.
Like for
What's that?
especially if you are space limited
Yes, space limited is true.
I was just looking, so I have my desk right here and then
like that way, that way I
have the door, but there's no way I could fit my desk outside of
the door with it fully assembled.
Oh yeah.
I'd have to at least take the top off, which is just a giant chore.
So.
Oh yeah.
Well, in my case, I don't have that excuse 'cause I have two French doors, so I
wouldn't have had that excuse, but yeah.
But it still, it still would've been, it still would've been annoying.
And so, but we figured, we figured it out and now I have better sound.
It's a beautiful thing.
And, uh, so, uh, today we're gonna talk about DR and disaster recovery and
lessons learned, and especially, um, some lessons learned from at least one
major event, uh, one major disaster.
And, you know, I, I don't want to, um, you know, the, these, these events,
especially this event, disasters are, are always difficult on, on the people.
Uh, and, and I don't want in any way make light of those disasters
or I, I don't know, whatever, whatever I'm trying to say.
Right.
So proper respect to the people.
'cause some of the, well, especially the one, the main one that we're
gonna talk about, people died in these disasters and, um, you know,
with respect to those people.
Having said that.
We can learn from the things that happened, uh, at that time.
Right?
And so let's talk about, and and the disaster.
The, the main disaster I'm talking about at this point would be what
we all called nine 11, right?
So September 11th, 2001.
Uh, I lived through it.
You lived through it.
Uh, so how old would you have been in 2001?
I would rather not say
I was very young.
You were very young.
I, I was actually in college.
okay.
I was here, um, and um, I was in this house and my kids were young.
My oldest would've been, uh, seven, and my younger one would've been in four.
And what I remember was she might've been in, she might've been five,
she might've been in preschool or might've been in kindergarten.
What I remember, and for those that don't know, I live in North County San Diego,
which means I live just south of Camp Pendleton, which is, uh, you know, one
of the biggest, uh, Marine Corps bases.
The, the idea was that we were under attack because there were multiple events
that were happening simultaneously.
Right.
Multiple planes hit the, the trade center, there was the plane that hit the Pentagon.
There was the plane that went down in Pennsylvania.
There was this feeling that, like we were being attacked
as a country, which we were, and living near a military base.
I don't know what I thought I was going to accomplish by keeping my kids home
from school, but that's what I did.
Yeah.
I was just like, I'm gonna, I'm gonna, I'm gonna, I'm gonna, I'm
gonna keep my kid, like, it, it, like, I, I, I don't even know what
I was thinking, but like, you know, thinking back on it now I'm thinking
that probably I was thinking that, um.
I, I, I, if the world's gonna end, I want my kids near me.
I
mean, it was like, it was, it was, it was, it's kind of morbid.
But that's,
I, I just remember that I didn't, you know, that, that I kept my kids near me.
Um, and I remember that I knew multiple people that worked in
the Real World Trade Center.
None of them were hurt that day
Mm.
and for various reasons.
One, uh, and I know one person that was in the World Trade Center that day.
I knew other people that worked in the World Trade Center, but
for one reason or another, chose not to go into work that day.
Hmm.
I know another person that was supposed to be on Flight 11, which
was from Boston to New York, which was the flight that ended up going.
I don't know if that's the one that went into the Pentagon or
if that's the one that crashed in Pennsylvania, but I knew a person that
was supposed to be on that flight.
That person did not get on that flight.
And, uh, and then the other person that, that I knew was the person in
the, in the trade center, it was, uh, Michael Hingis that was, uh, the, the
blind person that made it, that made it down thanks to his seeing eye dog.
Um, and he, you know, he.
Somewhat famous.
As a result.
He actually went on to become a motivational speaker.
And, um, and, uh, just to, just to lighten things up, I, I'll talk about,
uh, how I first met Michael, and that was, and I, it seems like I've told
this story relatively recently, but I was at my very first trade show.
This would've been in the early nineties, and it was at, it was in
New York, um, at the Javit Center.
In Manhattan, and it was Unix Expo, and I saw, uh, these hot swappable
dish drives, you know, where, where you
could push a button and pull the drives out.
And I just thought that was the coolest thing I'd ever seen.
'cause at the, at the time that, that was unlike anything I'd ever
seen it, it, that was really, really new and really, really cool.
Now we just take it for granted.
But back then it was really cool and he was, he was the one that was.
The se standing in the booth demonstrating these, these, uh, disc drives and
literally his, his shtick was.
It's so easy a blind man can do it.
Right?
And um, and we were like, this is amazing.
We're like, you guys are the only ones with it.
And he goes, yes, we are the only ones with this product.
And then we walked through the trade center or the trade show and we saw
several other vendors with the product.
And one of us said to the other.
Well, in his defense, he can't see the other vendors.
So true.
Um, do you, do you remember, do you have memories of that day?
Of course You do.
Oh yeah, so I remember, so I went to school, I was in Pittsburgh actually.
Mm-hmm.
And so we had a lot of friends, or I had a lot of friends whose families were
living in the city in New York City.
Right.
And so I remember everyone just was very, very concerned.
Um, and like you had mentioned, there was sort of the plane flying towards.
Uh, over Pennsylvania.
And so everyone in Pittsburgh was keeping a close eye being like, Hey,
where is that plane actually going?
Because
it was supposed to fly over Pittsburgh,
right.
right?
So I remember everyone sort of being worried, concerned because they had family
and relatives and they weren't able, like all the phone lines were shut down, right?
So.
No one was able to figure out like what was going on.
I remember going with a bunch of other folks down to the common area
and just kind of like just being like shellshocked as we're watching the news.
Yeah, I, yeah, absolutely.
Um, there's a, there's a great line in the beginning of a movie that I like.
It's become problematic now.
Um, maybe it was always problematic, but there's a movie called Love, actually,
and in the beginning, which is, uh, voiceover from, um, Hugh, Hugh Grant.
And, and, and he was saying on nine 11.
There were a lot of phone calls that were made from the plains,
and he said, to my knowledge, none of them were messages of hate.
They were
messages of love.
Hmm.
I just got a little of a clump there.
Anyway, so, okay.
So, um, when we think about that event, there is, in my world, we
immediately started talking about.
The, we saw the things that happened to companies, and we're gonna talk
about, I'm gonna talk about sort of two, two different kinds of companies.
One of them.
So first off, let's talk about, uh, the, the difference between
a cold site and a hot site.
Do you, so when we talk about disaster recovery, there was this
idea that we, that we're gonna have another site, like ready to go
Yep.
and we talk about cold site and the hot site.
Do you want to talk about what that means?
Yeah, so a hot site basically is you have a site, right?
A disaster recovery site that is fully operational, has all the
equipment, has everything replicated to it, and basically once you push
the BRI big red button, right?
Everything sort of fails over.
It's available, operational, all ready to go, sort of ready
to serve traffic and take over, usually minutes to hours within a
failure.
right.
And then a cold site is very much the opposite of that, right?
It's a site that's sort of ready to start a restore.
Uh, like I suppose there'd be a, there'd be a no site,
Yeah.
right?
That's where, uh, a bad thing happened.
And we're gonna go find some hardware to restore
to, uh, cold site, the, the.
Implication there is that you have some hardware ready to go, but
you haven't restored anything.
A warm site is somewhere in between those two things,
I've have, have you heard of this term that some people call pilot?
A pilot light
Yeah.
talk to.
Talk to me
So it's basically not quite cold, but not quite warm or hot, right?
So
it's a little bit better than cold, but not quite to the extent.
And a lot of the trade off comes from not necessarily having to
eat all of the costs upfront.
So
as an example, if your pilot site is, say, in the cloud, you might have your
data available, but not necessarily your compute and everything else ready to go.
Yeah, I think that I, I think I would still call that a warm sight, but
I mean, there is this concept, it comes from the concept of a pilot
light.
I. Right.
Which for those of you that don't know, when you have a
gas old school that
nowadays we have electronic ignition, but there used to be this in inside,
if you had a gas water heater or a gas furnace, there would be this
little flame that would burn all the
time.
And that would, that's called your pilot light.
And if the pilot light goes out, then you're gonna have to relight it
because otherwise your, your heat won't work.
Um, modern days we use.
electronic electronic ignition,
but, um, because a pilot like just wastes a lot of,
yep.
It's always running and you're always consuming gas.
Yeah.
Um, so there, so the, the best from a DR perspective, right?
The, the, the, the Cadillac, if you will, is, um.
The, I don't know if that's the right term anymore because nobody buys Cadillac, the
Ferrari, the Rolls Royce.
Does anybody buy, do they still make Rolls
Oh yeah.
Okay.
Um, is the hot site
that it's, it's, it's ready to go.
Number one.
It's ready to go when you need it, and, and two, it's kept and
it's ready to go within a, a few.
Minutes or seconds, right?
It's kept as up to date as possible, as much as technology
would allow you to do so.
And one of the challenges with a hot site is, is latency.
And so you might want to put the hot site
as close
as you can to, uh, the, the site that you're preparing
and what happened on nine 11.
So there were many companies to, for their hot site, uh, or had
their main data center in one tower
and had their hot site in another data in the other tower.
which makes perfect sense.
As long as nothing would take out both towers
Yep.
and.
So unfortunately, as we know, you know, both towers, I mean, that,
that's what I woke up to by the way.
I, I
woke up to my wife saying both of the, 'cause I'm on the west coast, so both
of the towers had already collapsed
as you know, as I was waking up.
And so people lost or companies lost their.
their.
primary site and their hot site in, in the same moment.
And so one of the things we learned, nine 11 is to make sure that you are,
if you're doing some sort of hot site or warm site, is to put that site, I'm
gonna say nowhere near, um, that the, the site that you're being protected.
Now let's talk about that.
Um, there were actually some attempts, um, I lived through
this because I was working.
For companies at the time, and that is there were attempts at regulation to say,
If you're a bank or whatever you need to put, if you're financial training,
you need to put a copy of your data.
Uh, that's, that's hot over 200 miles away.
Yep.
was an attempt at regulation.
What's the problem with that?
Um, 200 miles away if you're, depending on the type of disaster isn't far enough.
And so that's
not the problem.
But the second is latency.
yeah, that's the problem, right?
It's just simply not feasible because the, the, the round trip time, the 200 mile
round trip time, uh, was just far too long that couldn't keep the data up to date.
depending on the
Based on the technology at the time,
Yeah.
Well, and I think it depends.
So I do so.
Many years after nine 11, I was working at a storage company and one of the
things that they did was they also like talking to financial customers, right?
Is many of 'em had what they would call dark fiber,
Yep.
right?
Where they would basically run fiber optics, two fiber network between
their two sites, and it would.
You're right.
It wouldn't be completely eliminate the latency, but it would definitely
help versus say routing it over a public network of any type.
Yeah, we definitely, you would.
I, I think that there was an assumption of dark fiber, uh, at that point, but even
200 miles on a straight piece of glass, speed of light has a speed of, speed of
light is not, it's not instantaneous.
It's whatever it is.
187,000 miles a second or whatever,
six hundred or something like
something like that.
Right.
Um, the, the round trip time is gonna be measured in milliseconds.
It's not gonna be, it's not, it's, it's, it's going to significantly increase
latency, especially if we start talking about synchronous transfer of data.
Right.
Now let's talk about synchronous versus sacred.
Synchronous versus asynchronous transfer of data.
You want to give that a shot?
Yeah, so synchronous is.
Basically a right comes into your production site.
It gets forwarded over to your DR site.
The right gets, now there are different flavors, but typically the right gets
committed on the DR site acknowledged back to the primary site, and then the
primary site acknowledges the client during which, so you have to add up
basically the latency of going over the writes on both sides, coming back before
the client acknowledges, in which case you're guaranteed that that right has hit
both sites and so the client can move on.
Which is great as long as it doesn't take too long.
Yes.
So that's synchronous and asynchronous is something, is, is uh, different than that.
And we, we've had some different, between you and I, we've had some
different understandings of different kinds of asynchronous, but go ahead.
Yeah, so for me, asynchronous is, well, what I would call semi synchronous, but
that's a different case is where, uh, you accept the right on the production,
you forward it over to the secondary or DR site, but, but while it's in
process of being committed on the other side, you can acknowledge the client.
So there is a lag.
Um, you could decide how long that lag is, depending on technology and the vendor.
Some allow you to specify at an IO level so you can say, I want
10 transactions outstanding.
Others allow you to do it in terms of seconds.
So I'm allowing up to 10 seconds or 30 seconds, um,
Before you start, before you start kicking back a performance issue to the client.
Before
you start Yeah.
Putting back pressure.
Now
interestingly, there is a mode, which I'm not sure if you're aware of,
uh, that some financial co companies requested, which is called Domino mode.
Talk to me.
So it's a form of synchronous replication where it, because in
synchronous replication, typically you write to the client, it sends it over.
If the right fails on the secondary, it'll still accept it on the primary.
And acknowledge back to the client, right?
So you're not guaranteed that it'll stop writes if it can't write to
both sides At the same time, there's a mode called domino mode where if
it can't write to both sides, it will not acknowledge the client.
So that's why it's called a Domino.
One takes out the other.
I, I would think that,
this is one of those, this is one of those things where, you know, uh, this
is reality versus the idea, right?
To me.
domino mode that you described, that's synchronous.
Anything other than that is not synchronous, right?
And anything other than the domino mode that you described,
I would call asynchronous, right?
So these either synchronous, it's sort of like immutable and not immutable, right?
It's either synchronous or it's not synchronous.
And if it's synchronous, then it shouldn't acknowledge the right to the
client until both writes have been done.
And if one of them fails, then they're not done.
yeah.
So at least from most of the vendors I've seen,
they've never implemented it that way.
Yeah.
That's interesting.
Well, they're wrong.
So, um, uh, yeah, so that was, so that was a lesson we thought we
learned at the time, but we need to make sure we put it far enough away.
But then they were like, it's gotta be synchronous and
it's gotta be 200 miles away.
They're like, eh, it's not gonna work.
Right.
Um, nowadays, you know, you hinted at it earlier, nowadays we would do this with
the cloud and we can put it actually, because, you know, you, you did say that
your first problem was that it wasn't far enough, and that's probably true, right?
Because especially when we start talking about certain areas
like Southern Florida, right?
200 miles isn't far enough.
Um, and um, so with the cloud, you can put it.
Pretty much anywhere.
Now, if we're going to do that, if, especially if we're gonna use
public networks, we're pretty much going to have to use asynchronous
of some sort, right?
Uh, so we're gonna send the data and put it another place we're going to,
you know, like you said, you can have a buffer, you can have a certain amount
of time that it's allowed to get behind before it, uh, like you said, put, what
do you mean when you say back pressure?
So this is where you start to, um, elongate the time
before acknowledging a right.
So
to the client, because typically your client will sort of throttle itself
because at some point your latencies are gonna get into the seconds
and they'll be like, no, no, no.
Something's going on.
I'll slow down.
right.
Because otherwise you're just gonna start dropping the writes.
And.
And, and this is a, a configuration choice on the part of the customer
where they can say, I don't want to ever put back pressure.
I wanna, uh, you know, that, that the data protection is less important
than actually getting the job done.
And then other clients would say, if I don't back it up, I don't
wanna write it in the first place.
Right.
Um, I, I, obviously, I tend to be more towards the latter than the former.
Yeah.
Uh, from, uh, most of the companies I've seen are vendors.
Yeah.
They're not in line with you.
Meaning Meaning that they would just go ahead and do it anyway?
Yeah, because most customers Right.
Unless you
have very, very strict regulations.
Right.
They're like, best effort
Yeah.
I, I think it's
because they will
that I care about data protection more than the average
person,
Because the thing is, at some point it will catch back up.
Hopefully,
Hopefully yes.
Depending on what the problem was.
right.
Um, and, and again, as long as we're okay with the potential,
right, um, uh, I, I would think that there should still be some number.
number might be measured in hours if we're hours behind updating
our other copy, something.
Might be drastically wrong that we need to look at.
Yep.
And the other thing to also mention is with a lot of this.
High end.
I normally refer to it as tier one storage systems
Yeah,
because these are tier one applications with very strict requirements.
Um, usually also they provide the ability to do automatic failover.
So it's kind of, think of it like high availability plus clustering.
right.
So if you take a look at a lot of the tier one storage, right, they might have two
storage systems in both locations with the drives that are all interconnected.
So in case one unit fails, the other unit can take over the diss of the other side.
Um, the clients are also connected to both sides, so they don't
have to worry about failing over.
Um, I can't remember what it was called.
It's like the optimized and non-optimized connectivity for fiber channel,
which allows it to have a preferred path and a non-preferred path.
So you still
have connectivity, so your clients will automatically fail over, so you
don't have to do anything, and so your writes can still continue to happen.
Yeah.
So that's a big thing, is like, you know, we, we, we learned that
we should have it farther away.
We learned that maybe we shouldn't have it too far away, but, but,
but now with the, with the.
With the cloud, we can potentially have it pretty much anywhere, but we
definitely have to rely on some sort of asynchronous, uh, uh, communication.
When, I think about our, our friends that came on the show.
Talk about their experiences with disaster.
I think there's some, that was a major, that was a hurricane
that took out an island
and they had multiple data centers on that island.
One was more in the high ground than the other.
So one was flooded, the other was not.
And they were gonna recover from one data center to the other.
And that's, and they had everything they needed there.
They did.
They needed personnel.
They had to fly people to the island.
But other things happened that we can also learn from.
Do you, what do you remember from those?
So from, so there were two things I remember.
One was sort of.
The people process stuff, which I think we rarely think about.
And then the other was the technology piece.
So from a people process perspective, it was, do you have the right people in
country who have the expertise and the
skills to
recover?
Where are they going to sleep?
Where are they going to get food?
Right?
All the things that you kind of take for granted, right?
They had none of that.
Like how do they communicate
Yeah.
He, as I recall, he, he turned a, a, um, a conference room into a
hotel room
Right.
And slept on a cot and ate rice and beans for two
weeks.
And he was lucky to
get rice and beans right?
Because there were a lot of people who didn't even get that.
Yeah.
Um.
Yeah, that's, that is definitely a, a lesson learned is that, you
know, make sure to take the human element into your DR design.
Right.
Um, and the one that, the one that stands out to me was
the reliance on the mainland.
Yeah.
Right.
That when you have a true disaster, whether it's on an island or just.
You know, wherever you might not be able to get connectivity to the rest
of your computing infrastructure.
And in this case, their authentication and authorization, their IAM system
relied on active directory, which
all of which was in, was in the mainland.
And um, so this is just, you know, the lesson there is to just make sure that.
To just take that into consideration, right.
Just the, the realize that in a real disaster, you may be very
isolated from the rest, rest of the world, and you need to take
that into consideration in your DR.
Design.
Yeah.
And, but I think this becomes so difficult, right, Curtis, because
like how many scenarios are you gonna play in your head and how much time
are you gonna focus on some of these?
Now granted, a hurricane on a tropical island is probably
a high likelihood event,
Yes.
Right.
Well, I,
of that, but.
well, I, I, I think it's, I think it's a totally, I think that there's two
things that you cannot count on, right.
Um, I. and and I, I would just say one thing that covers the number of
things, and that is utilities, right?
You cannot count on the internet.
You cannot count on power.
You cannot count on, um, you know, uh, water,
right?
You can't count on those three things.
And so I, I'm just saying, I don't, I don't think you, you don't need
to think of all of the reasons that one or more of those might not.
Yeah.
available.
You just need to plan for them not being available.
If you don't have internet, if you don't have whatever it is you,
however you communicate between your sites, you are going to be isolated.
If you don't have power, you're gonna need to supply your own power,
right?
Um, these are just things that you can think, you can
think through these things, and these are all things that you can say either.
I'm just saying, have that discussion.
And say, you know what, if we don't have power, we're not gonna do anything.
We're not
gonna spend $15 billion on power generators
Yeah,
in case we, you know, we might, or, or you might be subject
to regulations, or you might
have, uh, financial reasons why downtime is enough.
That, or the cost of downtime is enough that you're gonna pay for generators.
Um, do you remember how they got internet over there?
It was satellite.
Yeah.
satellite.
I like this idea of making sure that you think about what would
you do if the utilities that you're normally counting on are not
available?
If you get completely isolated, what would you do?
There are a number of reasons why that might end up being the case.
or the, or the people that you normally depend on are not available.
Yes.
This is why we had this little thing called documentation.
Yeah.
Um,
And,
and
go ahead.
and hopefully I know when we've had Mike, Dr. Mike on the podcast, right?
Um, he's talked about sort of doing tabletop exercises, right?
So have you done a tabletop exercise, which is like, Hey, what happens if the
main IT person, Susie, is unavailable?
Right, right.
And, and that's what you do.
You work through those various scenarios.
You, you hire somebody to come in as an outsider is the best way to do that.
When we start talking about ransomware, um, Dr. Mike Sailor's company would,
would, would be a great resource for that.
It's good to use a very negative person.
Think the most negative person in your environment, the most
pess, pessimistic person.
And, uh, no, no.
Pessimist thinks they're negative.
They, they think they're, they're realists,
will realize
Um, yeah.
Sort of the final thought that I'm thinking is, and, and I, I remember.
Doing a disaster recovery of my own, and that is, I did it where.
I had not yet tested the throughput of the
backup system that we were doing and the throughput of the backup system we
were doing, it turned out to be crap.
Right?
And as a result, uh, that was a really hard day.
And in Curtis land, right?
This is
very early in my career, I learned it.
What's that?
Yes, it was the compression thing.
Yeah.
So make sure that when, when you make configuration changes to your
backup system or your DR system, make sure that you test that.
Just realize that in general, restore speed is slower than backup speed.
It just is.
It's the way it, you know it, you know, back in the day with tape, it
was because we were doing multiplexing.
Nowadays it would would with dedupe.
It's because of dedupe.
Um, and so just sort of plan for that.
But don't, don't just assume how much slower it is, uh, test it and see how much
slower it is and make sure you figure that into the, to the disaster recovery plan.
But, um, so, um, with that, I think that's enough for now in terms of lessons learned
from just, you know, the pains of others.
Um.
Just think of the scenarios that might be, you know, that you might be subject to.
Um, make sure you've got at least one copy of your data
that's far away from everything.
Um, and the way to probably do that today is cloud taped can still play a role.
In fact, our previous episode we talk about why tape is
still not dead in backup.
Uh, it certainly is on life support, but there
is, you know.
There is, there is a use for tape and backup and it's disaster recovery.
So, um, um, especially when we start talking about disaster
recovery from a ransomware attack.
So, well, thanks for chatting again, my friend.
No, it was good.
Uh, one thing I'm surprised you, I thought you were gonna mention, but you did not,
The 3, 2, 1 rule.
1 rule.
You were just
You know what's funny is we were so close.
We were so close.
Yeah.
The whole 3, 2, 1 rule with, you know, three copies of your data on two
different media, one of which is offsite.
That, that, that's a core design concept for anything.
Uh, backup in dr. And, and that's, yeah, you're right.
We never, we, we didn't mention, but now we have, so you
have corrected our oversight.
Thank you very much.
And thank you to our listeners.
We'd be nothing without you.
That is a wrap.
The backup wrap up is written, recorded, and produced by me w Curtis Preston.
If you need backup or Dr. Consulting content generation or expert witness
work, check out backup central.com.
You can also find links from my O'Reilly Books on the same website.
Remember, this is an independent podcast and any opinions that
you hear are those of the speaker and not necessarily an employer.
Thanks for listening.