You found the backup wrap up your go-to podcast for all things
backup recovery and cyber recovery.
In this episode, we explore how to design your backups to make
them more resilient to ransomware.
I.
We'll discuss the importance of understanding dwell time, the need
for longer retention periods, and the value of frequent backups.
We'll also delve into innovative recovery solutions, including the use of snapshots,
replication, and cloud-based solutions.
We also talk about the difference between database and file system recoveries.
With regards to ransomware, we get down in the nitty gritty this week.
I hope you like the episode.
By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr.
Backup, and I've been passionate about backup and recovery for
over 30 years, ever since.
I had to tell my boss that we had no backups of this database.
That was really important.
That we just deleted.
I don't want that to happen to you, and that's why I do this show.
On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes.
This is the backup wrap up.
Welcome to show.
I'm your host, w Curtis Preston, ak.
Mr.
Backup, and before we get started, if I could please ask, please, like,
share, subscribe, so you never miss a beat when it comes to this show.
Did, did, did that give you joy?
I met, it's been a while since I've done it.
You know that.
Yeah.
Well, I get to introduce you now after you've introduced me, and I'm
going to introduce you as my fall post-Traumatic stress, counselor.
Prasanna Malaiyandi how's going?
Prasanna
I am good, Curtis, how are you feeling by the way?
I, you know, I, I, I think tho those that watch on YouTube, right?
You, you can watch this on the backup wrap up channel on YouTube.
Those that watch on YouTube can see there.
There's no, uh, this is where I would've, on the left side of my
face, I would've expected a shiner.
Uh, I don't have any broken bones and I, I just have a little bit of soreness left.
What are we talking about?
Yeah.
Well, well, before we talk about
what happened, for those people who may not know, a shiner is
not someone who shines shoes.
A shiner is sort of when someone punches you and you get like a black eye
around, right?
It's like a bruised.
I would've expected a shiner.
Yeah.
So I kind
happened, Curtis?
Yeah.
I kind of fell down an entire flight of stairs.
Um.
By the way, this, we can, we can blame the new office on this.
Um, or we could blame this on the new office because I'm
spending more time upstairs now.
Um, yeah, I was literally, it happened like 20 feet to the
left of me and I was on a ladder.
And I, the, there's a hall next to a landing and I, when I came down from
the ladder, my right foot came down in the hall, my left foot came down
in the landing and which was about, you know, it's a stair, a stair tread
less so that's like, what, six inches?
And, And, and, that was all the momentum I need.
I completely lost my balance and I gained forward momentum.
Going straight down the stairs, so I fell down face first.
I didn't ball up like you might typically do when you fall because
I, I was worried that I would then tumble down the stairs and
I just knew I
would break everything.
I think, yeah,
if you had tumbled down, it probably would've been a lot worse,
like head over heels.
yeah.
And so I stiffed up, stiffened, stiffened up, and then put my hands
out to brace my fall because I fell.
90 plus 45.
Right?
So 135 degrees, right?
I was standing straight up and then, yeah, like a tree going
down, but, but there's no crowd.
And so I fell face first on the stairs, put my hands out, managed
to stop, you know, whatever.
And then I slid face first all the way to the bottom of the stairs, my face.
This is why I said I was expecting a shiner.
My left side of my face hit the ground first.
And then there was enough momentum that I kept going, and then my daughter made me
go to the emergency room and I had a whole
bunch of X-rays and a CAT scan and uh, yeah.
You know, and you're almost 60 years old and you fall down
an entire flight of stairs.
You don't really get a choice.
You go to the
Well, and and
I think you should tell people what the doctor said to you when
he came up to you to check on you.
yeah, he walks actually ev pretty much every medical professional, like
as they turned around the corner and then they saw me and they were like.
Are you, are you William?
Because that's my first name for those of, that's what the W stands for.
Uh, are you William?
And I'm like, yes.
And they're like, you're the guy that fell down a flight of stairs.
And I'm like, yeah.
And he is like, I was kind of expecting blood and guts and you know, extruding
bone and
Yeah.
He is like, you look fine.
I'm like, I know.
And they made me come anyway.
Um, and so they were, yeah, so they did a CAT scan.
They did, you know, I don't know, probably like 15 x-rays of my arms.
'cause they were, they were all, you know, whatnot.
And, uh, and then, you know, sent me home and they, they asked me if I wanted any
pain medicine and I, I literally said no, because I remember the last time
I was in the ER when I broke my nose.
For those longtime fans of the show may remember that, um, I.
They said, they said they gave me something for the pain, they gave me
like a Tylenol and like the lowest level, like narcotic painkiller you can give.
And they charged me $800 for that.
So I was like, I'm good.
Um, I'll, I'll sug it up.
Um,
I got a fifth of tequila somewhere in the house.
or some edibles.
Maybe some edibles.
Um, but yeah, so it was, um,
But I'm glad to.
Yeah.
Amazingly, no major injuries.
I've got some sore spots, you know, but no major injuries,
So the podcast shall continue.
Our series on ransomware shall continue.
And, uh, we're taking a break with Mike this week.
Um, and we're gonna talk about.
Specifically the backup side of things.
We've already done an episode or two that I will summarize as follows.
Your backup server is, is at risk, right?
It is
at high
risk
Yeah, it is, it is at high risk.
And, and, and there are, we have numerous, um, data points to back that up.
My favorite, I think what did Dwayne say and what, first of all, what, who was
Dwayne and what, what, what did he say?
yeah, Dwayne is a red teamer, right?
So he pretends to be the bad guy in attack systems.
Yeah,
I think he basically said, I love the backup system.
That's the first system I target because I get access to that.
I get access to all your data in your
environment because everyone backs up into a single place.
It's the key to the kingdom.
Yeah.
So if a red chamber thinks that, and by the way, if you haven't heard
the Red Team episode, go back, Maybe
two months.
Yeah.
Uh, there's an episode called, uh, you Know, about Red Team.
And, he clarified that, you know, both the backup system itself in
terms of how powerful it is, how much you get access to it, and also.
In terms of how poorly it often is designed from a security standpoint.
He talked about things like service accounts, right?
He said he loves the, the backup service account.
Uh, do you remember what he
said
default With the defaults password
or no password?
Yep.
But do you remember what he said about what was unique
about it?
Yeah.
With the backup service account, nothing gets logged in the system
I
you access it using the backup service account because it assumes you're
gonna be using it all the time and reading everything in the file system.
So I bother logging anything.
Yeah, exactly.
And so, you know, that I think that was an episode where we just said,
listen, you really need to understand your backup server is at risk.
We also had an episode or two where we talked about how to design the
server itself, um, in order to.
Better insulated from that risk.
Do you remember the kinds of things we talked about there?
Yeah, I think these were ideas such as segmentation.
It also included, don't have your backup server connected to your
normal active directory instance.
Kind of keep it isolated, separate, um, make sure you're.
Just to also the normal stuff, right?
Like make sure you patch your systems, including the servers,
right?
Keep up to date on those.
Yeah.
Uh, yeah.
So all the, the, usual stuff of, um, you know, obviously patch management,
password management, and MFA obviously,
right?
Yes.
I think the big one that you mentioned that I harp on a lot is
to separate it as much as possible.
Don't have it on the active directory domain, don't have it.
Um, you know, don't use the same username and password there
that you use anywhere else.
Um, I mean, that should be true anyway, but whatever.
This is a practice you should have everywhere, but you
should definitely have it here.
And that is don't ever log in as root or administrator log in as you and become
root or administrator you use, the concept of least privilege so that you can, uh,
minimize the damage that any one person can do
One I wanted to add there, because I know we've talked about this on
episodes a while ago, is don't save your password to access your backup
management system in your web browser.
Yeah, please don't do that.
Please don't do that.
Um, the, and, and, and you we're, we're big proponents of password management
here and, and and specifically password management systems, not your browser.
Uh, again, the browser's better than nothing.
Perhaps, perhaps not
in this case.
the Thing that, that I remember when we talked about third
party password managers, it.
Floored me the first time I installed Dashlane and it said, Hey, should
we go get the passwords that you stored in your browser for you?
And I'm like, wait, you can just get them,
Yep.
they're not like encrypted or anything.
You can just ask the browser, Hey, what's the password you have for this?
And they just sucked them outta there.
Well, it's encrypted, but it probably just has API access.
So as long as you
granted API access.
yeah.
But if, if you can get access by api, any other system,
software running on your
Yeah, yeah, so, so some sort, some sort of third party password management system.
And I do believe that it should be a different password management
system than the rest of the world.
Again, segregate, segregate, segregate, right?
As much as possible.
Obviously more than any other server, shut down any services that you don't need.
Especially my favorite, the ransomware deployment protocol.
Um,
Otherwise known as Windows RDP.
For those who don't, who may not recognize that acronym.
Um, and, um, but today I wanted to talk about how do we, um, design the
backups themselves, uh, to be more useful in times of ransomware, right?
Probably the first thing you need to do is actually do backups,
right?
That should be like your step one of your
strategy.
I live in this, I live in this fantasy world Prasanna where like
everybody does backups, right?
Actually just before this recording, I was on another.
You know, uh, another thing, and, and, and I realize that I'm speaking
to laypeople in this particular recording, uh, people in the legal
profession, and I was asked like, what is a backup and what is a restore?
And it's like, and, and I know that, you know, they're, they're catering
to an audience that doesn't understand
this stuff, but I just live in this world where everybody backs up their stuff,
Yeah,
that's why you need the 88 in the room.
IE me.
Exactly.
Everybody runs a third party backup of their iPhone because they
know that iCloud is not a backup.
iCloud is not a backup.
iCloud is a synchronization product.
Uh, not a backup product.
go listen to that episode if you're interested.
Yeah.
Go listen to the how to properly back up your iPhone.
When we think about designing a backup system for the purposes of, of, uh,
responding to a ransomware attack and then being able to recover, I think
it's important to think about, um.
A, a lot of things in terms of how does ransomware typically behave?
How does a ransomware response event typically take place?
And so I, I think the first thing to talk about is this concept of dwell time.
Do you wanna talk about that?
Yeah, so I think a lot of, and if you go back and listen to some of the
previous episodes and if, if you look, listen to what Mike has talked about
before, I think everyone thinks, oh.
I got hit with ransomware, I just got infected, and then boom, just
that next instance, everything in my environment is encrypted,
right?
That's what I think a lot of people think about it.
It's almost, but it's not how that works.
It's like you get infected by a disease, right?
It might take you a day before you start to get like a cold, and
then maybe a couple days later you start to spike a fever, right?
And so similarly for ransomware, we talk about something called a
dwell time, which is how long is.
The ransomware actually in your environment, even though it may
not be actively encrypting data,
Right.
Or exfiltrating data or whatever it is, but it's already got a
foothold in your environment and it
exists somewhere in your network.
Yeah, and and I, I think it's important to understand, again, and we've talked
about this on other episodes, that.
Remember that ransomware isn't a single piece of software,
right?
Um, maybe the actual ransomware is a single piece of software, but the
entire, there there is a suite of tools that ransomware actors are
going to be using to, to get, number one, to get into your environment.
Number two, two, spread around in your environment and to figure
out what's going on in your environment and that it's that final.
Uh, tool, the one that you're, you know, the actual ransomware tool
that's doing the, the, uh, encryption and or doing extraction, right?
Doing, um,
exfiltration.
But before that happens, you're right.
There is this, this process of like going through the network
and figuring out what is, um.
You know, figuring out what they could do.
There was a great story that Mike talked about where he said that they were in
an environment and they were doing a, um, they did a tabletop, and during that
tabletop they used the incident response plan, and they obviously shared the
incident response plan around everywhere.
And what they found out was they got a, they got a ransomware
attack right after this.
And what they found out was that, um.
That they had already been attacked and that the attacker was in
their system for quite a while.
And so he got to see the, you know, the, the, uh, what do you call it, the
incident response plan and all this stuff.
And they got to see like how much insurance they, all this stuff, right?
Uh, so that, that's a really big time.
And so what's the concern when
we start talking about restoring what?
oh, I was just going to mention like, I know this has come up in
the past, but one of the banks that I bank with that's a credit union,
they were hit with ransomware.
Basically over the 4th of July that shut down everything.
Mm-Hmm
Right.
But they finally published a, a analysis of what happened.
And they say that, so July 4th or July 1st is when everything got shut
down.
Everything was encrypted, right?
They said that they were in their network starting May 23rd.
So six weeks almost.
That would be the, that would be the dwell time.
Right.
And if you look, if you do some Googling, you'll find that the average
dwell time is actually really long.
Right.
Um, like the mean dwell time, last time I looked was like close to 90 days.
Uh, which means that there's.
Ones that are way
beyond that.
Right.
Um, it's not like they're all 90 days and it works out that way.
Yeah, because I think for these actors, right, there are
two things they want to do.
One, they wanna spread everywhere, so they have access to as much as possible.
And two, they want to figure out what's valuable in your environment,
Yeah.
right?
And so they
beat in there as long as they can.
Yeah.
And so if they go in and immediately start encrypting, right, using the
ransomware, you're going to notice and now they've lost that opportunity.
So it's kind of a
balance on their side, right?
They want to be in there spreading, observing, but the longer they're
in there, then the more likely it is for them to be detected as well.
So it's kind of this balance.
Yeah.
It is a balance.
Um, but the longer they're in there, the bigger the possible reward.
The other thing that they could do, and there, there's a couple
things that they could do while they're in there a long time.
One is they could start encrypt, encrypting.
And again, this is one where I'm gonna describe what I've been told has happened.
I haven't verified this, but it seems reasonable to me, and that
is that they start encrypting stuff that nobody's looking at.
Like the
Right, like older data that
nobody's looking at.
And, um, they do that because they could, they could do it and get away
with it because nobody's looking.
Uh, number one.
The other thing, what do you, what do you think is the other thing that
they could potentially do if they're in your system for a long time?
Exfil trading data.
Exfiltrating data, right?
More than likely you will end up paying the ransom.
Um, even though I hate the idea and all that kind of
stuff, right?
But it's a very different argument of like, oh no, I
don't have to pay the ransom.
Ah, I got good backups in a DR plan and it's a response plan.
They're like, yeah, but we still all your data, we're gonna
tell everybody what you did.
Um, um, the longer they're in there, the easier exfiltration is, right?
Because they can do it slower.
They can, you know, send it out.
Which again is why I continue to say, please figure
out some way to track outgoing traffic.
Yeah, which is actually what happened at this credit union.
They ended up exfiltrating data from their database in addition
to encrypting everything.
So social security
Are they back up by the way,
that
Uh, yeah, I think they are back up.
Uh, last month they had to send out paper statements 'cause they
weren't fully up and running, but I believe now they're up and running.
But all customer data is out there on the dark web.
Yay.
Yeah, because, uh, again, fans of the podcast may remember that, uh, my medical,
uh, group got attacked with ransomware in May, and I found out last week
they're still not fully up and running.
Yeah.
maybe they don't have backups.
Curtis is what I'm guessing.
I, I don't even wanna know.
Um,
all I know is that for months, the only way I could make an appointment
was to go into, I could, I had to physically drive into the doctor's
office, which luckily for me is like, I.
15 minutes from my
office, uh, I could go, you know, drive there, make an appointment.
And it was, it was actually kind of good because it meant it was a pain to
make an appointment, which meant that it was easy to make an appointment.
Is that, you understand what I'm saying?
It was, it was logistically difficult to make an appointment, which made it
easier to make an appointment once you
were there because
No
many people, yeah.
No one wanted to deal with it.
Right.
Uh, but yeah, what I found out, again, this was just last week,
that the phone part of the.
System is still not back up and running.
Um, yeah.
Yeah.
Com completely crazy.
What's worse?
Is it pre It's pretending that it's up and running.
I
called the number, I waited on hold for like 20 minutes and
they're like, you know, press one.
You know, to, if you're a provider, press one.
If you're a patient, press two.
And if you wanna speak to you wanna make an appointment, press two.
Okay.
And, and then it's like for the.
Chula Vista office, press one for the Encinitas office, press two, you
know, and I'm like eight, like I'm
gonna sit there with, you know, and then I finally press eight.
And then it's like, please hold.
And then, and it was like, it was like 20 minutes later, click.
Ah,
I was like, what the
hello.
So I was like, well, if I'd known first off, I'd have known
it was gonna take so long.
I would've just, I would've just drove over there.
Let alone, if I'd have known, maybe, you know, maybe I would, should have
done is got on the phone and then Dr.
And then drove.
And then drove there.
Yeah.
So anyway, so I made my appointment, um, and and then, and then
proceeded to fall down the stairs.
Um, all right, so if there's a long dwell time, what do we need to be
talking about with regards to, um,
backups?
Well.
So when you are getting to the point of recovering from your ransomware
incident, you wanna make sure that the data you're restoring.
Is clean, right?
That it doesn't contain any bits and pieces of the ransomware,
right,
Or any of those other intrusions that have happened in the past.
So you
really wanna find a clean point so it doesn't sort of
restart itself and you end up
Right.
So how would that affect, how would a long dwell time affect
the design of your backup system?
You need a longer retention because you need to make
sure your backup is
a clean backup before you got infected.
I say this because it's very common, and again, I, I had a
conversation with Mike about that.
It's very common.
For people to say, oh, well I only need 90 days, right?
I only need 90 days for my, my backup retention.
And I understand the reasonings behind that.
Right.
Um, and uh, and, and Mike was saying that it's very common he's seen it in
the field where people go to restore their stuff and the retention period
of their backups is less than the dwell time of the product, and they're
unable to successfully restore.
I have a question about this
though.
So I totally get it.
Ideally, you want a clean backup, but
if you go, say you have to go back three months, right?
So you restore from something from three months, and now you
have to sort of roll forward
to get back to the current point in time.
yeah.
That's the next thing to talk about.
yeah.
My question though is, is that better than restoring a non clean backup and
then surgically going and cleaning it up?
Yeah.
So the, that's a really good question.
Uh, let's finish this
point and then let's go to that point.
So all I'm saying is you, you know, to have options,
you need a much longer restore, uh, a much longer retention period than 90 days.
Um,
my, my general thing would be like a year.
Right, like a minimum of a year.
I would actually, I would say for a business minimum of, of 13 months.
Because sometimes you want stuff from like the annual report from a
year ago.
Um, I don't see any problem with going a couple of years, any
retention, you know, I don't want you to go much longer than that.
I was gonna say, don't consider this an archive.
Don't use your backups for archiving,
but yes.
Um, but the, you know, there, there's nothing wrong with having a couple
of years retention for your backups, maybe even three years right now.
That's not really a ransomware defense at that point.
Um, but you want options.
Okay.
Um, what, so my, Mike and I spent a quite a bit of time talking
about this, this issue of.
Clean versus, you know, completely clean versus clean versus cleaning it.
Does that make sense?
Right.
So he, he explained a, a couple of things.
One, and, and it, the, the, I'll just say this is a complicated question
and it's, and it's more complicated when we start talking about file servers.
So first off, let's just, let's just do the, the, the, the.
The, you know, good, better, best, right?
the the best.
When you ask someone that's responsible for that, that has been
through this, they will tell you that the best thing you can do is a
complete wipe and restore afterwards,
a, a complete wipe and a restore, especially when we're
talking about the system,
right?
The os.
Um, that
a clean slate, right.
start from a clean slate.
Okay.
Because there, there's two different things here.
There's restoring the systems and then there's restoring the applications.
There's actually three, and then there's
restoring sort of just d like unstructured data,
right?
So regarding the systems, uh, he feels pretty strongly that this
should be a, a clean wipe install.
It is possible, it is possible to do what you're talking about where you restore
the, the system and you and you're able to find, you know, you find out what
variant of ransomware you have, you find out what tools that variant installs.
You uninstall those tools.
The concern
with
might have missed something.
restore and yeah, but not just that, but.
You.
The big thing is did it install something like in the boot block
to basically re-enable, you know,
Yeah.
if you can also fix that, right?
yeah.
Well, and I think this is some of the challenges today too.
I don't know if you saw, but there's a new malware variant that
injects itself in the UEFI boot.
Which is
literally baked into the motherboards,
right?
That's supposed to be super secure.
And so in cases like that, you're basically hosed, right?
You don't even wanna restore that data because it's always going, like
you said, it always keep coming back over and over, no matter what you do,
You.
You know what?
I'm gonna pull a Prasanna.
what,
Prasanna, you
just used an acronym.
what is.
UEFI?
I know what it is.
I just, I just realized I have absolutely no idea what that stands for.
But first off, what is A-U-F-U-E-F-I?
Boo.
What are you talking about?
So UEFI boot is how the system boots up based on, so in the past, right,
you had the master boot record and
it had sort of certain blocks where it knew where to go in order to load it,
in order to make things larger because there were certain limitations and
also more secure like windows and other things use what they call A-U-E-F-I mode.
In order to be able to boot your operating system.
It stands for Unified Extensible Firmware Interface.
So it's basically the, it's the next generation.
It's been that way for a long time.
Right.
Um, right.
Uh, you know, it's been a long time since MBR was that Master boot record.
The
NBR was the only option, but yeah, so this is just what I'm talking about.
I I guess, again, and this is why I'm, you know, I'm such a fan
of having an expert in the room.
This is why you bring in somebody like Black Swan Security, right.
Uh, to, to to come in, which is my Mike's company.
But the, the.
But to go back to the design issue, there are two things that you
want to make sure you have, right?
I'm gonna say three things, but two things that you wanna make sure you
have is, you know, the retention period, and also a high enough frequency
that you know you have, again, you have options.
What does frequency, when we talk about.
There's a couple of, couple of acronyms that we, we actually, we
actually haven't said it in a while.
You know what else we haven't said in a while?
Yeah.
RTO and RPO.
Which one?
You know what?
What else we haven't said in a while?
What.
The 3, 2, 1 rule.
The three 12 hasn't come up in a while.
We've been talking about ransomware so much.
We haven't talked enough about backup, but so when we talk about RTO and
RPO, right, recovery time objective, that's how fast you can, you, you,
you, you want to be able to restore and then recovery point objective,
how much data you're allowed to lose.
So frequency is going to impact which of those,
RPO
and RTO technically
potentially depend, depends, right?
okay.
Yeah.
I guess you're, I guess you're right.
Yeah.
Yeah, yeah, yeah.
Um, it, I, I, think maybe like levels and things
combined with frequency might affect, affect your RTO, so, um, it's just so
funny, like I've spent so much time.
In this new world where we don't do levels right, we just do one full and
incrementals forever, which is the way all backups should be done, but whatever.
I'd digress.
Um, the, um, I, I really do think the concept of like
repeated fulls is really a, a, a
concept that
needs to be done away with.
Yeah.
But, um, so you wanna make sure you have a long enough, um.
Retention period.
You wanna make sure you have a frequent enough backup.
Uh, and then what I want to talk about this is, this is my third thing that
I'm talking on the end of my two things, and that is you need options during a
restore because when you are, uh, when I asked Mike to sort of walk through
what a, what he felt was like a typical restore scenario and what he described.
Was many, many restores of the same system that then allowed you to
pick apart, to say, okay, do you know we wanna restore this version?
Nope, that one's infected.
We wanna restore
this one.
No.
And that's infected.
And keep going backwards until you get to a system that is
not infected.
Right?
That's clean.
If you didn't build that into your design, a ransomware recovery is
going to take significantly longer.
Now, I'm
yeah.
Yeah.
gonna give you, I'm gonna give you a freebie.
What
design element I.
That's backup related.
Could you be using that would allow you to have infinite recovery points
without, no, sorry.
You fail in, let me finish my sentence.
Allow you to have virtually unlimited recovery points and while in RT
with RTOs of like next to zero.
Do I have to answer this?
I don't like this answer.
I know what it is.
What is it?
It's, three letters.
Oh, no, wait.
Oh, no, no, not, not infinite.
Sorry, not infinite.
Number of recovery points ne, nearly, nearly
So, uh, so, so, okay, so let me, let me check.
So, in my mind as you're describing this, I'm thinking of CDP, which
is continuous data protection.
Yeah, that would be infinite.
Okay.
Uh, there is also snapshot based replication.
Go.
Right.
Okay.
So it only took me two tries, which isn't bad.
So I gave you, I gave you a leading question, but I guess
I didn't lead you enough.
I did.
I thought that, you know, given your background, this would
just, just jump right out.
But maybe, maybe they beat you out of it enough at your previous employer.
So, so.
This is where snapshots and what I would call near CDP, right near
continuous state of protection, this is where snapshots can be so useful.
Because how, how I, you tell me Prasanna, how else can you
restore hundreds of versions of the same server without much pain?
You can't.
Unless every copy or every backup was on a separate tape, device or tape,
and you had infinite number of tapes connected to infinite number of devices.
Right,
right.
So you could do the
restores in parallel, basically the
and by the way, that's so you can restore one server a hundred
different ways, and then you've got a hundred other servers.
Yep.
So, yeah, you would need infinite, infinite, infinite, infinite, infinite.
infinite.
I, I guess what I'm saying here is think about this, right?
Think about, um, and this is where, uh, and again, we, we get into more storage
here than backup, but you, you know that I'm a fan of this, this concept
of snapshots and replication and that if we think about like there, there's
a lot of technology that allows you to.
Store your virtualization world, and I'm a big fan of virtualization, store your
virtualization world on, on a filer,
right?
That allows you to take snapshots and replicate those snapshots and
replicate them even to an immutable, uh,
device if you want.
Right?
And then you have infinite number of recovery points.
Yep.
Um, I
I,
And I, I
I wanna add one more thing on top of
okay, sure.
I think
deduplication becomes important.
Because everything you're talking about right now is for a single
server, but now say you have a hundred servers that are all based on like
a similar image or whatever else.
I think
I I think deduplication can
add a lot of value in terms of, it can really help
reduce the
amount of storage that's gonna be used.
Yes.
Um, I don't think it's required in terms, it, it is just gonna save you money.
Right.
But the idea of, I guess I just, this is where, uh, you know, this, this
is one of the, this, this is why I am such a fan of, you know, not just
NetApp NetApp's not the only one.
There's so many companies, and
it's not just filers.
It's not just na, it's, they're also SAN devices.
There are iSCSI devices.
There are modern scale out storage arrays that have.
Infinite, or, you know, short, short or close to infinite number of snapshots That
don't impact performance, And that would offer you, um, some real choices here.
I, I think also we can throw in the concept of copy data management.
There are, there are CDM products, like, is it still
called acto after it got acquired?
I think it's still called Actifio.
Yep.
so products like that, basically, I guess what I'm just saying is
this is a real problem, right?
This is a huge problem and this is a potential, really useful
tool towards this problem.
Maybe it won't solve all known, you know, things, but when we start talking about.
Uh, hey, I wanna do a hundred copy.
You know, I wanna keep retention for this long, and I want, I want
to potentially restore my server a hundred times and I, but I don't want
to restore my server a hundred times.
Yep.
Um, it just seems like snapshots and replication would
really be your friend here.
Yeah.
Oh, definitely.
Yeah.
I don't see how else you're gonna be able to even figure
out if a copy is clean, right?
Without something like this,
And, and this is, you know, the another, so I'll throw this here.
Another possible friend is the cloud,
right?
If you are using a cloud-based recovery system, and if.
That recovery system has the ability to scale out and say, I wanna
recover, bang, bang, bang, bang, bang.
Right?
Yep.
It's just that all the ones that I've seen, the, at least the ones that
I've seen, you know, essentially with my own eyes, when we start talking
about recovering many, many copies.
They can scale it out.
Right?
so
they, so they, they do, you know, you remember earlier when you said
if you had a infinite tape drives and
all that, they have
the cloud lets you do that.
Yeah.
the cloud lets you do that.
Right.
But, uh, the restore of the actual server will still take.
A finite amount of time.
Right.
And again, build that into the design, figure that out, go to the vendor,
say, Hey, here's what we wanna do.
We wanna be able to do, restore the server a hundred times
and pick which one we want.
Can I do that?
Well, yeah, it's gonna take you three years.
You know,
figure, have that discussion now.
You know, build that into the design.
Um, I, I guess I'm.
I'm just really, and, and maybe this means you change vendors,
right?
Maybe this means you change storage systems.
Um,
to your vendor and see how do I solve this?
Here's my
Well, you, you definitely should do.
that first.
You know, I'm a
fan of, I, I'm not a fan of like, uh,
uh, steeplechase, I call it, right?
Not a fan of just going, you know, place to place just to, you know, I
always think you should, you should.
Let your current, you know, give your current vendor the problem.
Don't go to them with the design.
Right.
Go to them
with your requirements.
Right?
this is what I'm looking to do.
What's the best way to do this?
Because maybe they have a mechanism that.
Yeah, I listened to this podcast and Curtis said I need to be able to
restore my server a hundred times.
Like how, how do I do that?
Is there a way to do that with your product?
Right.
Um, or, well,
actually, technically what?
What you, you would say, I need to be able to,
Identify a clean
copy of
a clean image.
And so maybe there's a way to do that.
Um, yeah, the, the example I always give for dictating the
requirements and not the design.
I live in San Diego and we have Coronado, which is, it's not an island,
but people call it Coronado Island.
Um, it's Coronado Peninsula, but that doesn't sound as cool.
And I always give the example of, of say, listen, I need a hundred
thousand people to be able to go to and from that island every day.
That's a, that is a requirement.
And maybe it's a tunnel.
Maybe it's a ferry.
Maybe
it's a bridge.
Right.
Or in the case of San Diego, maybe it's all three.
Not a tunnel.
But we do have a bridge.
We have a ferry, and we have the long way.
The
long way.
Uh, it's funny, in San
Diego you can literally see Coronado, it's like right there.
It's like a half a mile on the other side of the water.
But the long way without the bridge, it's like four, like a 40 minute
drive because you gotta go to Mexico.
No
kidding.
You gotta go to Mexico, turn around and come back.
Um,
yeah.
Anyway, you could just swim across.
Anyway, I think that, I think that was good.
It was a
good conversation.
We did not talk about two things though
that you had brought up earlier.
The one is you had talked about restoring the server, but not the databases or
Yep.
Yep, yep.
Yeah, yeah,
yeah, yeah, yeah.
So, all right, so a couple things that we didn't talk about, right?
Uh, this idea of restoring the server and perhaps restoring the data later.
I, I, I'm thinking mainly about the idea of like using an image
perhaps to restore the server,
and then we restore the, and that that image would probably
have a clean copy of Oracle or
whatever it is that you wanna restore, and then potentially restoring
the data as a secondary thing.
I, I think this is another.
Potential way to do this.
And by the way, virtualization makes all this so much easier.
Right?
Um, but which is, you know, on the list of why I, I'm such a fan of virtualization.
But that is a potential, again, just look at these designs and
then, and then work with it.
When we look at that method of doing it, the, I think this is
a much more valid method for restoring, say, databases today.
Because I say this, you know, I'm recording this on August 21st, 2024.
Today, they don't tend to attack databases directly, meaning they
don't go into the database and, and mess up the individual contents.
If they encrypt the database, the encrypt the database file.
Right?
So you just wanna restore.
You wanna restore the database.
Yes.
You restore the file.
You restore the file from before it was encrypted.
You're good to go.
Generally speaking, so I think that's a really valid way to restore a database
server and an application server that
has something like a database on it.
the the real concern I do have
is when we start talking about unstructured data and file systems,
because what did we start this?
What did we start this podcast talking about?
Do you remember the phrase that we defined in the beginning?
The dwell time.
So, so what, why is having a long dwell time?
It's like, hey, I get, you said 90 days, right?
I got, I got six months.
I got, I got a
year.
Curtis said to do two years.
I'm doing two years.
I got two years of backups.
So what if the dwell time is 90 days?
What's the
Well, well, because you, 'cause it is gonna go through and
like we talked about, right?
It might decide, I'm gonna start with the old data and
slowly start encrypting things.
And maybe you notice, maybe you don't notice, but.
Now you have to go figure out like a needle in a haystack.
Except the haystack is a small number of files in say, a
billion or 5 billion files.
So the good news is there's gonna be one of two scenarios, and
most likely you're going to be the first of the two scenarios.
The good news is, I think most of the time.
You're gonna look at the server, you're gonna look at the even unstructured
data, and you will be able to easily identify which files were encrypted,
and you're gonna find that they were all encrypted at the same time.
They were all encrypted, all on the same day.
I think that this idea, again, August 21st, 2024, I think this
idea of slowly encrypting the files over time, one, one of two things I
think it is at, at a minimum it is.
More rare than the other method because again, the
moment they start encrypting files, they really set off
alarms, right?
So I think it's pretty rare and I even think it's possibly a boogeyman.
I,
I, I don't know for sure, but if you have this problem
though, there's no good answer.
Uh, you know that, well, the, the, the only one that I am aware of,
right there was, you know, from, from our previous employer, they
had a solution to this problem where they, they, they, they had this, they
called like image curation, right?
Where you could give them a range of time.
They go and they would go in and automatically pick the last good
version of every file prior to it being encrypted, doing that manually.
Yeah.
If you have this slow encryption, doing that manually is, is,
you know,
And then, and then you just need to think if it's all the old data, it's
encrypting, do I really care about it?
yeah.
It's the, um, take it.
Um.
Again, I'm going back to a hundred years ago when we had this, uh, this
old server that we were decommissioning and it had been around so long
that nobody knew what was on it.
And so basically we got down to like the final one or two servers that were part
of the, it was a, it was at and t's first attempt at a multi-processing computer.
Right?
And so it had multiple computers inside the computer.
And so we got down to like the last one or two, and basically the, the idea was.
Uh, we just turn it off and then see who yells
Sometimes that works the
because we couldn't figure out, you know, we couldn't figure
out who was on it and what it was doing.
And so that's the same kind of thing.
Like, you know, if a, if a file gets encrypted and nobody reads
it, that it really get encrypted.
It's like, you know, like if a tree falls in a forest, if a
file gets attacked by ransomware and nobody wants to file, who gives a crap?
Uh, that's our, that's our, uh, that's
our final piece of advice.
All right.
Well, this, this has been fun.
I think it's
good.
You, again, it's, this is a little bit more far reaching in terms of
some of the design elements and design ideas that you would put in there.
Um, but I think it's one that, that people should really be thinking about.
Yeah.
And this isn't intended to be a conclusive list,
right?
But this is just initial thoughts to get you thinking and go have your
discussions with other experts, with your vendors, right, to see what
else you should be thinking about.
Exactly.
Well, thanks Pana.
This was fun.
I know this was fun and Curtis, I'm glad you didn't die.
'cause then I would be sad.
One, one final thing on that, uh, 'cause I don't think I mentioned it earlier.
You know, we, we, we have some people that are renting are
renting a room from us here.
And he, he happened to be the only one that was home when this happened.
And he heard it happen and he was very glad to hear me yell at his name.
'cause he's like, he was scared to go out.
He, he was like, he heard it happen and it sounded awful.
And he is like, I, I hope he is not dead.
And then he heard me yell at his name and he is like, oh, thank
God.
thank God.
dead.
Well, I'm glad I'm not dead too.
Prasanna, so that we can
I'm sure our listeners are as
Yeah, I, yeah, you don't care.
All right.
If there's anybody out there that's glad I'm not dead and they're still listening,
send me a note on backup wrap up.com.
Send a message.
This, uh, I'm glad you're not dead.
Or put it as a comment on the, uh, on the YouTube video or a comment on
the, you know, on the, uh, on the,
on the backup wrap up.
Anyway, well, uh, thanks to our listeners.
Uh, you know, we kid, but we love you.
You're the only reason we do
this.
Otherwise, just a couple of guys just talking.
Uh, and we'd probably just talk about barbecue then.
So that is a wrap.