You found the backup wrap up your go-to podcast for all things
backup recovery and cyber recovery.
In this episode, we jump into disaster recovery testing, and trust me, you don't
wanna learn these lessons the hard way.
I've got some wild stories about DR.
Tests gone wrong.
Including one from my early days at a bank that'll make you cringe.
My co-host persona, and I break down exactly how to approach DR.
Testing the right way, starting with the basics and working your way up.
We'll tell you why non-destructive testing is absolutely critical.
Seriously, you don't wanna blow up your production environment just to test Dr.
And how to set realistic success criteria that won't make you cry Another episode
from The Lessons From the Trenches.
I hope you like it.
By the way, if you don't know who I am, you're a first time listener.
I'm w Curtis Preston, AKA, Mr.
Backup, and I've been passionate about backup and recovery for over 30 years.
Ever since.
I had to tell my boss that we had no backups of the production
database that we had just lost.
I don't want that to happen to you.
That's why I do this podcast.
On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes.
This is the backup wrap up.
Welcome to the show.
Hi, I am your host, w Curtis Preston, AKA, Mr.
Backup, and if you could take just a quick moment to subscribe
or follow so that you'll get our great content, that would be great.
I am sitting here with none other than the guy who's very concerned.
About my obsession with serial killers lately, or at least one serial killer.
How's it going?
Persona?
I am good, Curtis.
Yeah,
about me?
so I think after this you should watch Hannibal,
the TV show, not the movie.
They both are good, but
Right?
And then we could discuss about whether people taste differently
depending on what they eat.
that's always something you talk about.
Uh,
your latest obsession is Dexter.
Yeah.
Which, which, for the record, I'm rewatching, right?
I, I enjoyed Dexter when it was on and when I had to wait a week, right?
going back to those old days
oh Lord,
I, you know, I'm, I'm kind.
I think you're, aren't you?
Like, don't you watch you, you do not watch shows when they're on.
You wait until they're done and then you binge them,
right?
Yeah, for
the most part, although these days I don't actually have like broadcast tv,
Right.
it's whatever's available on Netflix or Amazon or Take your pick.
We, we, we actually have YouTube tv, so we have broadcasts and there's
some shows that we watch on there.
Um, but there shows where, like, you know, this episode, that episode,
they're, they're all the same, you know?
And
Yeah.
you don't, it's not like, uh, it's not like
Dexter where there's an ongoing
so here's the funny thing, right?
So it was, it started off first on.
tv.
Right.
And like you said, you had to wait a
Showtime.
Yeah.
And then what I used to do, right, so I didn't have Showtime,
Mm-Hmm.
so I had to wait for it to come out on Netflix DVDs,
Oh,
right.
I would request the DVDs.
And of course I was cheap, frugal, however you want to say it, right?
And so I had like the two DVD plan, so I'd request like two DVDs, right?
So you'd get like six episodes.
Yeah.
Five or six episodes, you'd binge watch those and then you'd send them back.
And then you'd have to wait a week to then get the next set of DVDs
so funny.
in
order to watch it.
And that's I think, how I ended up watching Dexter.
And I think Breaking Bad might've been the same way too.
Yeah, you're just, you're cheaper than me.
Dexter is currently on Netflix,
so, um, yeah.
Anyway, so, and you could do, you can watch all these things while you do.
Disaster recovery testing.
'cause there's a lot of time, there's a lot of time when you do DR testing,
there's a lot of downtime, right?
You sit there and you stare at the screen.
Um, and, um, I'm gonna, I'm
gonna, I'm gonna start out a story.
What's that?
but can you really?
What,
I understand like if you're watching Dexter or pick one of these
very, the shows that pull you in,
uh huh.
Would you actually be focused or would your DR testing basically balloon like
10 times the normal amount of time?
Because you're
like, oh yeah,
I forgot to get back to that.
well, it de, you know what, it's gonna depend on the type of DR test
you do because some DR tests, there's a lot of waiting, there's a lot of,
I'm gonna start the restore part, and then I sit there for many, many hours.
And if you got one of those, then.
Doesn't really matter whether you're focused or not, as long
as somebody's keeping an eye on the, uh, percentage done.
And I'm gonna start this with a story from back in the day.
The first backup, uh, the first restore test I ever did.
The first, like, well, at least the first one I can really remember.
And this was when we had, um, you know, I was at the bank.
And which MBNA, which at the time was the second largest credit card corporation,
and I was in charge of backups.
And we had, I had talked the boss into moving to what would become
the first of many commercial backup products that I had, uh, used.
And that product was a product called SMarch, which as I've mentioned
before, should have been called SM Back because it was not an archive
product, it was a backup product.
Well, they were out, they were out of, uh, Minnesota area and,
and we had converted to them, but I like a good.
Good little backup guy.
I had done like a parallel implementation and so I was still
running my old dump tapes and I was running this new, uh, fancy tool.
And one of the things that this tool had was built-in, uh, compression.
Um, I wasn't using the compression on the tape drives.
I was using compression in the
software,
uh, to compress, uh, to go to the tape drives.
So we had our first major.
Failure, uh, file server, HP FS oh one.
I still remember the name of the server
just like get burned into your
Yeah.
And, and this is related to another story that we, to a friend of mine that,
that I've referred to before where she was a consultant and she accidentally
basically re, this was a self-inflicted disaster where a consultant was trying
to clean up home directories and she just did a really, really good job
of cleaning up all the home directors.
So I, I put my, uh, SMR tapes.
Uh, you know, in my front pocket and I put my backup tapes in my back, my
back pocket, and I went down there like Mighty Mouse here comes said the day.
And um, some of you will get
Not available on iTunes, I
Not available at iTunes.
And we went in there and I put, these were DDS, uh, tapes, right?
And these, 'cause this, we were all hp, HP loved DDS.
And so I popped into DDS tape, I pulled up my SMR software
and I started the restore test.
And, well, it wasn't a restore test.
It was, it was, I was testing, I was testing in anger, uh, you know, we
were testing like this was for real.
And, and I had not done any actual testing.
I.
And so I kicked off the Restore and um, it.
I'm, I'm watching and I, I, I, you know, like a good little Unix boy.
I had a, for a while loop
running and it was doing a, a DF on the, on the, um, on the, you know, to
display the size of the file system.
And I'm watching, and like a long time was going by and there
was no change in the size of
the file system.
And I was like, this is weird.
And then I went over and I looked, I just.
Just outta curiosity, I looked at the
tape drive, right.
You know, it's kind like, it's kinda like your car dies, you know, you
open up the hood like, I have any idea what's going on in inside there.
You know, look in there
and I see the, I see the light on the tape drive and, and it
goes, B blink, B blink, blink,
one 1002, 1003, 1004,
blink, blink.
Right?
And there were these giant pauses in between.
The blinks.
And so I'm like, that's strange.
It always blinks when it's, you know, when it's reading or writing data, right?
So I called up, I called up the guys and I said, Hey,
The
vendor.
Yeah,
vendor.
Yeah.
Called it the vendor.
What's going on here?
They said, well, by any chance did you use the compression feature?
And I said yes.
Yes I did.
They go, yeah.
So let us explain to you how the compression feature works.
So when we're backing up files, um, basically, um, we do the equivalent
of compress minus CI think it was, to, to send the result to standard out
and it send it straight to the tape.
But we don't know how to do that on the way back in.
And so what we do is.
We, uh, we, we read the tape, we read the entire file that we're gonna restore
into, and we restore it to temp, and then we run uncompress in place in
temp, and then we move the res, the
uncompressed restored file.
To where it's gonna go.
And we do that one file at a
Oh,
because we're concerned that we're gonna fill up temp.
And I'm like, oh.
So like if temp is, if I have a single file that's bigger than
temp, it's just not gonna work.
They're like, yeah.
And they're like, if this is a concern, we suggest perhaps
you don't use this feature.
it would've been helpful to know before I needed
Right, right.
And so thank God I had the, I had the backup tape in my backup
pocket and I pulled 'em out.
I was like, okay, we're just using, you know,
uh, dump here.
And, um, and we restored and everything was basic, right?
Um, and luckily I had backups from the previous night.
And really the moral of this story is.
Don't do that.
Right.
Don't test your backups before
you need them.
Right.
Do a DR test and that's what we're talking about today.
Do a DR test before you actually need to do DR and, and thi this May, we
will see I the, you know, sometimes when we have these episodes, persona,
you know, you know, we have these conversations beforehand and I'm
like, I don't know if we can fill up an entire episode over this topic.
We said that I think over the last one.
Yeah,
And it ended up not being a problem at all.
This is not one of those episodes.
This is one of those episodes where I think we might go along and we'll end
up turning this into two episodes, um, because there's a lot to talk about here
because before we do any testing, right, um, what do you think we need to do?
Well, you, well, you need to understand,
there.
By the way, there's probably no wrong answer here.
Uh, but
well, well, I think before you could
un unless your answer is, unless your answer is nothing,
um,
no, no.
So I was, I was going through my head.
I was thinking even before you get to testing,
Yeah.
you need to understand what are the business requirements for how quickly
you need to bring up that site, which in which actually you don't
necessarily think about as testing, but that's actually part of your
backup system design before you even get to the testing part.
Yeah.
So we, we have to agree on what success would be, right?
And we have to agree on obviously what we're gonna test, but we have to
agree on the parameters of that test.
And so, um.
What?
What do you think are going what?
Go ahead.
You and I was just going to ask about like your parameters, right?
What are
those parameters in my mind?
Some of those are how quickly do I need to be able to bring up what the scope is,
right.
Am I failing over and
trying to recover a file, an application, a data center, right?
What am I looking to actually test?
So it depends what you're trying to test.
And what the extent is that you want to test.
And also, I think the other thing is how close do you want to get to testing
an actual disaster as part of that?
Right.
Um, and I, I would say that there are different kinds of DR tests and, um.
The, there's, and, and if you haven't done any, I would say let's start small, right?
If we've never, if, if we've never done a DR test of any kind, I would
probably start with, what are you
I was thinking about the test you don't wanna try is who is the guy in Alaska?
Oh yeah,
yeah.
Okay.
We definitely have to put a link to that episode.
That was the most amazing episode ever.
In fact, you know what?
We'll pro we'll probably replay it over, uh, over the holiday break, Uh,
by the way, we love him and it, and, and it had a happy ending, but oh my God.
What, what a
nightmare,
think he was rebuilding a raid array, if I recall.
He was swapping the disks around
it was self-inflicted in that he said, I want to move the discs around.
Because they were like.
different sizes I
Yeah.
Like, and he's like, the only way I can do this is to just, to just
wipe everything and start over.
And he is like, yeah, okay.
So that's what I'll do.
I'll just wipe everything and then I'll restore everything.
My s my backup system works.
And this is how he tested his backup system.
Please do.
He definitely learned some things along the way, but, um, yeah, so don't do
Don't do that.
Yeah,
Don't do that.
Uh, but
learn from his example.
Listen to that episode and, and,
and have
a heart attack as you're listening.
because the thing that ran through my head was someone who's never done DR
testing would've been like, yeah, I'm just gonna shoot in the head my production
site with all the applications, and I make sure that it, or just test it out
and see does it fail over properly?
Yes.
The key here is non-destructive DR testing.
Right?
Um, and so, yeah, so, so to go back to like setting the scope, I, if this
is your first time doing DR testing, I would set the scope as small as possible.
In fact, if you've never done DR testing, I would not even be doing DR testing.
I would just be doing restore testing
and
What's the difference, Curtis?
What's that?
What's the
the question is, the question is are we going to declare a disaster
and say that this site is down in some way, um, and then we're
gonna fail over to another site?
Or are we simply just going to bring another server online?
Right.
De depending on how you define, uh, some, and, you know, depending on the
day of the week and the day of the year in which year it is, I have called.
Every time you have to restore, uh, a server of any kind, a disaster.
It's just a disaster of different levels, right?
So if the server, if a server caught on fire, that's a disaster,
right?
Um, and, or, or if a server just died, right?
Or if, like the, the story in the beginning, I think that was a disaster
because it took out a production workload,
right?
So I would say perhaps start with a single.
Production workload and restore it.
First off, just restore it, you know, like in place, not, not in place, not
in place, I meant like within the data
center or within whatever computing environment you're using.
And then start talking about VPNs and
you know, that sort of stuff.
So, uh, that would be defining the scope as small as you can
for the first test that you can.
so maybe like, uh, like maybe it might be a directory within a file server.
If you've never done
any restore testing, yes.
A simple directory within a file server, does our backup system work at all?
Right.
Um, and then the next, the next is going to be some type of
recovery of an entire server.
Hopefully you have that server virtualized because doing that is going to be,
uh, obviously much easier assuming you're treating that VM as a vm.
Uh, if it's not virtualized, then we we're gonna start going down the
level of, um, bare metal recovery,
right?
BMR By the way, we should probably do an episode on that.
I
didn't think about that.
We should do an episode on that.
I know you keep mentioning server as you're talking about this.
Would you also qualify that that could be server or an application?
thanks for bringing that up.
That's why you keep me around, you know?
So it depends on what you mean by application, right?
Um.
You could, these are like the different
level.
You talk about restoring a directory.
I would also look at restoring a single database within a server, right?
If that's what you mean by application, then I would say yes.
Sometimes when we say application, actually a lot of times when we
say application, what we mean is
application
Multiple
on multiple servers and multiple things, all interconnect.
And if that's what you're talking about, I'm gonna say no
yeah.
yeah, I
was thinking like a, my,
time out.
yeah, I was thinking more like restore your MySQL database.
Exactly
right.
Um,
table within your MySQL database.
Right.
yeah.
Um, and again, non-destructively,
Yeah,
right.
Um, I.
Uh, just, just a, just an entity and you can try all of these things, right?
You can try restoring a database.
You can try restoring an entire, all of the, all of the databases on a server.
You can try restoring a server with the databases, um, and or any other
applications that might like a web server.
You can try all of these things individually and make sure that
you've got those pieces down.
That this is about defining the scope.
Try the all of the different pieces first and make sure you've got the,
the recovery path for each of those parts of your infrastructure down
before you decide, okay, we're gonna pretend we're gonna blow up the data
center.
and the other reason to bring that up, it's important to test these different
types of components because there are gonna be different nuances like how
you deal with databases and recovering.
That is definitely gonna be different than file servers, which is probably
also gonna be different than servers.
And so it's important to understand the nuances of what is possible and
the steps for each one of these.
And also the different backup systems.
So if you've got.
Physical, you know, you've got physical servers that are not virtualized.
You've got servers running in Hyper V or VMware or you know, any, any, any sort
of on-premises virtualization set.
What was the third one
Broadcom,
brought?
VMware,
It's, it will always be
VMware to me.
It will always be world, um, the, um, if you've got.
VMs running in, uh, AWS, GCP Azure.
If you've got basically all of the different places that you
have infrastructure, probably have different backup and recovery
methodologies for each of them.
And so you should also be looking at testing all of those.
And you should be looking at testing each of them individually as a,
an overall process that you're working towards developing, uh,
declaring a much bigger disaster.
Yeah.
I, I think it's important also to be familiar because sometimes you might have
a real disaster that doesn't require you
to recover every single component within that higher level application, right?
So being familiar with the individual components is also important
depending on what the disaster is.
Yeah.
Agreed.
Um, and you know, there, there, there's an application that we haven't
talked about in terms of including or not including it in your Dr.
Scope and that is, what about SaaS applications like Microsoft 365?
Um, there are a couple of different scenarios there.
One is your.
Um, account is damaged in some sort of logical way, meaning logical
corruption, meaning a ransomware attack.
You deleted it.
Right?
We, we've covered that
on,
provider damaged it.
yeah.
The provider.
Well, that's, that, that was, I'm gonna list that as like a
third.
Well, if they damaged your account.
And there is an example of that.
Uh, for example, the sales for
Yep.
That's what I was thinking.
story where they went and blew up everybody's permissions and everybody
had to restore that themselves.
Um, that man, I hate that story.
I really do, because Salesforce, in my opinion, did not own up to.
Uh, they, they, didn't step up to the plate
Yeah.
at, at the time.
I remember writing a blog post for, uh, Druva.
I was working for Druva at the time, and I remember writing for Blo, a blog
post that said something like, proof that, that Salesforce should not be
trusted with your backup infrastructure.
Um, 'cause they clearly don't know what they're doing.
The but there's, but, so there's your account being damaged in some way.
And then there's OVH Cloud.
And what happened there Where the entire infrastructure goes?
Poof.
Right.
So, uh, I think you should have that as, as you, you should come up with that
as a scenario that you need to test.
It's going to be challenging in most cases because just let's just
talk about the different scenarios.
Let's say it's AWS the way most people back up AWS If AWS goes down
and takes your backups with it.
You're screwed.
Um, the way that most people back up most cloud infrastructure.
And then there's the fact that most people trust their SaaS provider for data
protection, which they should not, right?
I talk about that all the time.
They should not, But, but but even if you're backing up your, your
data, um, to a third party and, and it's not, they should not.
Right?
I talk about that all the time.
They should not.
This has been
a day in eight.
Problem though,
This is an age old problem, but we're talking about testing today.
So by the way, this is definitely gonna be two episodes.
We haven't even gotten to the testing yet.
All we're talking about is setting up the requirements.
So,
so this is definitely gonna be a second
episode.
Um, so we talk about, we talk about setting the scope and we
talk about starting small first.
Each of these individual components in your infrastructure and um,
were you about to say something?
that this is not destructive and
not disruptive.
Yeah.
Nice, nice.
I like that.
Non-destructive, non-disruptive DR testing.
I
like it.
because you don't wanna take down your production.
You don't wanna affect, say, ongoing Dr.
Resiliency that's available on the secondary site because you're
about to do some of this testing.
There are cases where you do want to impact that, but when you're doing sort
of these individual component levels, you may not want to impact your overall DR.
Posture while you're doing this
Yeah, that that last one is probably the hardest and it
and may actually be impossible.
What are we talking about there?
We're saying that if you have a DR system, I hope you have a DR system
if you have one, you know, see if there's a way
to test your DR without messing up your dr.
Um, I'm not sure if that's possible
in, in many scenarios, but.
but one of the things I think about right is a lot of data protection
vendors, they do test and dev, right?
You can spin
up a copy off of your storage that is a writeable copy that you can then use
for testing out your recovery systems
without impacting the actual recovery instance.
right.
And so that's what I'm thinking about is just like those sort of scenarios or maybe
you're able to wheel in an extra server or beg, borrow steel, an extra server to use
for your recovery testing or DR testing,
Yeah.
You know, it's, it's funny that you say that because I, I, you know, when
you, I like this idea wheeling in a, a
server.
I mean, I, I, I think many of our listeners, you know, and, and I, and every
time I, by default, I'm always talking about data center, even though I, you
know, and then in my brain goes, Hey,
nobody has a data center anymore.
Um, I don't think many people are wheeling in a server.
I think that they're, they're, you know, they're looking at.
Cloud
infrastructure as a way.
I, I know that not everybody can do that.
And that's obviously another thing that you have to decide upfront is
how are we going to do this recovery?
Are we going to do it in the cloud?
Are we gonna do it?
The alternate infras, alternate infrastructure, um, these are all
things that you have to decide upfront.
We have to decide what it is we're gonna restore.
We have to decide, um, where we're going to restore and, and how.
Right and.
The deciding the where.
Again, this is all about planning.
This is all stuff that needs to be
decided upfront.
This is also something that's going to be part of your backup and Dr.
Design, you will have decided up upfront, you know how we're going to do Dr.
Uh, well, disaster recovery, how we're going to do it.
And that place is most likely the place that you're going
to, uh, be doing DR testing.
And, you know, there are a lot of choices here.
Cloud infrastructure, I think, is the best choice for most environments.
Another choice is aging infrastructure, right?
So
you move, you know, you move your older stuff out and that
becomes your DR environment.
Um, another choice, another very common choice is basically, um,
there, there's two ways to basically.
Rent infrastructure for the purposes of disaster recovery.
One way is to contract with, um, you know, a company that will provide
what you need if and when you need it.
Uh, and that costs, you know, let's say this much.
And then there's a company that will provide everything you need,
always available to you all the time.
Even when you don't need it.
And that will cost this much.
Yeah, exactly.
I can't do the fingers, I gotta do the hands.
Right.
It's significantly more.
Uh, and, and this is why I push everybody to the cloud as much as you can.
'cause that's the beautiful thing about the cloud, is that you can just literally
snap your fingers and, um, you know, and use, and you can also use infrastructure
as code so that you can just make all of the hardware that you need
magically appear.
What's that?
and work right.
And work.
Yeah.
Magically appear when you need it.
And then the moment you no longer need it because your test is over, you can
also snap your fingers and it all goes away and you just pay for it only when,
uh, it's, you know, up and running.
Um, so the, the next thing to talk about is, so we decided
what it's we're gonna test.
We decided where we're gonna test it, how we're gonna test it,
what about our success criteria?
Yeah.
I think this is important to note upfront what it means to be
successful, but I think it's also important to be realistic, right.
With your success criteria, especially if this is sort of your
first time doing this, because I remember the story you tell Curtis
about your, your runbooks that you
Yeah.
when you used to work at the bank.
And how a lot of times they were not able to get through and
actually do the recovery because they, a step would be skipped or
something wouldn't work properly.
And so I think it's sort of a learning process.
So don't be too hard on yourself.
If the first 10, 20, a hundred times you try doing your recovery testing, it's
not like a hundred percent successful.
Yeah.
may not be your fault.
Things change, environments change, hard
work may change, right?
There's so many other factors,
but.
this is, this is very closely related to the discussions we've had over, um,
doing, uh, um, cyber recovery testing.
Right?
And your disaster recovery very likely will be part of an
overall cyber recovery process.
And I agree with you that I.
Um, it's interesting.
I was go mentally, I was going somewhere completely different.
But you, you, once again, this is why we make such a good team.
Um, you were more like, Hey, make the re make the, um, um,
you know, the requirements.
Uh, be nice to yourself, especially if it's the first time out.
You know, requirement number one, no one dies, right?
Yep.
No fires are created.
No one quits.
Um.
And, um, the, you know, set your expectations low and nobody can
take you, take 'em away from you.
I'm borrowing heavily from, uh, Michael p Connolly, the comedian who,
who says that's the, the happiness to life is to lower your expectations.
And he's like, my goal for today is to go to the bathroom outside my pants.
Um,
so yeah.
Set the, uh, you know, the success criteria low in the beginning.
Um, where I was going was of course, REO and RPO, which we talked about before.
The, that, that is definitely going to determine and the overall success,
once you click the stopwatch and you begin the recovery process.
And then you click it, that everything is back up and running and, and you've
tested that, that, that whatever it is that you destroyed or you pretended you
destroyed, is now up and running and fully functional Again, as we've mentioned
in the RTO and RPO, that doesn't just mean the time of the restore, it's,
it's, you know, it's.
ah, here's a question.
Is it RTO and RPO or is it RTA and RPA?
the, it's, that is the objective,
right?
The RTO and RPO are, are the objective that we are shooting for.
And so the goal, um, in a recovery test of any kind is that the RTA, that's recovery
time, actual, uh, and recovery point actual, are less than, uh, or equal to
the RTO and RPO.
go back and listen to our episode.
We covered it, I think a couple episodes ago, right?
No, it was, well, well, I don't know.
It depends on when this one gets published.
We just, we just did it.
Yeah.
That's why I'm saying we just did it not.
Oh, was that really the last episode?
Well, it just published.
Um, uh, but again, I don't know, you know,
we'll see how these things, you know, but, um, yeah, if you're, if, if RTO and RPO
don't just roll off your tongue and you don't know what they are and, and, and,
you know, all of that stuff, it, they literally should be in every conversation
having anything to do with backup and Dr.
Design.
But that to me is the ultimate success criteria, right?
Another success criteria is the degree, and this, this is gonna be
a percentage, um, a per a percentage achieved, and that is the degree to which
you were able to follow your runbook
and just do what's in the runbook.
This is
what you, this is what you were alluding to before, right?
Well, and hopefully you have a runbook.
Yeah.
Like you said, that, that, that assumes you have a run book.
Yeah.
Um.
And, and the way we always did this at the bank, as we mentioned
before, is that we would have someone other than the person who runs the
backup system do the DR testing.
Right?
Here's the runbook, please follow it because we're gonna pretend
that Curtis got hit by a bus.
It was never anything nice.
It was never
that I won the lottery and then just flew the coop.
It was always Curtis got hit by a bus or got swallowed up in the, in the sink.
The great sinkhole of 2024.
Um,
long do you think it's gonna be until people don't even know what a bus is?
I think, I think we're good.
I think we're
good.
You know, I, um, it's funny, definitely an aside, when I was working the
election, uh, the last two weeks we're at a school and one of the hardest
time we're, we're, we're at a, like a multifunction building next to a school.
And one of the hardest times was during pickup and drop off because
there are all these parents that are picking up and dropping off their kids.
And I was like.
You know, if only there was like a large vehicle that we could put all the kids
in and like we could like paint it yellow so that people could see it and then like
have a sign that comes out and make sure, oh, traffic stops in both directions.
If only we could do all that.
And then all these parents wouldn't have to like, uh, take their kids
to school and waste all of that gas and all of those cars sitting there.
And I'm sure they're, the gas is running while they're sitting
there for a half hour waiting for their kid to come outta school.
Anyway, I digress.
I don't know.
Hmm.
But, um, so that, I, I think that, we'll, we'll stop there because
basically it, the number one thing, the number one success criteria that
I'm gonna say is that you know what your success criteria are before.
What are we gonna restore?
How are we gonna restore it?
Where are we going to restore it, and how long is it, you know, what, what
timeframe are we trying to fit in?
And also what other, what other, like the thing we talked about with the.
With the, uh, uh, the other criteria being that, that we can
do it without Curtis' help, right?
Or what, whatever, you know, whatever your, your Curtis is, right?
Um, decide on what all of those are upfront and you have a much
better chance of being successful when you actually do the recovery.
What do you think?
No, that makes sense.
Okay.
And with that, I will thank you once again for being a great co-host persona.
I try, I try.
This was a fun, I I like talking about disaster recovery testing.
Yeah, absolutely.
And we will, uh, hope you guys enjoyed this as well.
Uh, that is a wrap.
The backup wrap up is written, recorded, and produced by me w Curtis Preston.
If you need backup or Dr.
Consulting content generation or expert witness work,
check out backup central.com.
You can also find links from my O'Reilly Books on the same website.
Remember, this is an independent podcast and any opinions that
you hear are those of the speaker and not necessarily an employer.
Thanks for listening.