You found the backup wrap up your go-to podcast for all things

backup recovery and cyber recovery.

In this episode, we explore how to design your backups to make

them more resilient to ransomware.

I.

We'll discuss the importance of understanding dwell time, the need

for longer retention periods, and the value of frequent backups.

We'll also delve into innovative recovery solutions, including the use of snapshots,

replication, and cloud-based solutions.

We also talk about the difference between database and file system recoveries.

With regards to ransomware, we get down in the nitty gritty this week.

I hope you like the episode.

By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr.

Backup, and I've been passionate about backup and recovery for

over 30 years, ever since.

I had to tell my boss that we had no backups of this database.

That was really important.

That we just deleted.

I don't want that to happen to you, and that's why I do this show.

On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes.

This is the backup wrap up.

Welcome to show.

I'm your host, w Curtis Preston, ak.

Mr.

Backup, and before we get started, if I could please ask, please, like,

share, subscribe, so you never miss a beat when it comes to this show.

Did, did, did that give you joy?

I met, it's been a while since I've done it.

You know that.

Yeah.

Yeah.

Well, I get to introduce you now after you've introduced me, and I'm

going to introduce you as my fall post-Traumatic stress, counselor.

Prasanna Malaiyandi how's going?

Prasanna

I am good, Curtis, how are you feeling by the way?

I, you know, I, I, I think tho those that watch on YouTube, right?

You, you can watch this on the backup wrap up channel on YouTube.

Those that watch on YouTube can see there.

There's no, uh, this is where I would've, on the left side of my

face, I would've expected a shiner.

Uh, I don't have any broken bones and I, I just have a little bit of soreness left.

What are we talking about?

Yeah.

Well, well, before we talk about

what happened, for those people who may not know, a shiner is

not someone who shines shoes.

A shiner is sort of when someone punches you and you get like a black eye

around, right?

It's like a bruised.

I would've expected a shiner.

Yeah.

So I kind

happened, Curtis?

Yeah.

I kind of fell down an entire flight of stairs.

Um.

By the way, this, we can, we can blame the new office on this.

Um, or we could blame this on the new office because I'm

spending more time upstairs now.

Um, yeah, I was literally, it happened like 20 feet to the

left of me and I was on a ladder.

And I, the, there's a hall next to a landing and I, when I came down from

the ladder, my right foot came down in the hall, my left foot came down

in the landing and which was about, you know, it's a stair, a stair tread

less so that's like, what, six inches?

And, And, and, that was all the momentum I need.

I completely lost my balance and I gained forward momentum.

Going straight down the stairs, so I fell down face first.

I didn't ball up like you might typically do when you fall because

I, I was worried that I would then tumble down the stairs and

I just knew I

would break everything.

I think, yeah,

if you had tumbled down, it probably would've been a lot worse,

like head over heels.

yeah.

And so I stiffed up, stiffened, stiffened up, and then put my hands

out to brace my fall because I fell.

90 plus 45.

Right?

So 135 degrees, right?

I was standing straight up and then, yeah, like a tree going

down, but, but there's no crowd.

And so I fell face first on the stairs, put my hands out, managed

to stop, you know, whatever.

And then I slid face first all the way to the bottom of the stairs, my face.

This is why I said I was expecting a shiner.

My left side of my face hit the ground first.

And then there was enough momentum that I kept going, and then my daughter made me

go to the emergency room and I had a whole

bunch of X-rays and a CAT scan and uh, yeah.

You know, and you're almost 60 years old and you fall down

an entire flight of stairs.

You don't really get a choice.

You go to the

Well, and and

I think you should tell people what the doctor said to you when

he came up to you to check on you.

yeah, he walks actually ev pretty much every medical professional, like

as they turned around the corner and then they saw me and they were like.

Are you, are you William?

Because that's my first name for those of, that's what the W stands for.

Uh, are you William?

And I'm like, yes.

And they're like, you're the guy that fell down a flight of stairs.

And I'm like, yeah.

And he is like, I was kind of expecting blood and guts and you know, extruding

bone and

Yeah.

Yeah.

He is like, you look fine.

I'm like, I know.

And they made me come anyway.

Um, and so they were, yeah, so they did a CAT scan.

They did, you know, I don't know, probably like 15 x-rays of my arms.

'cause they were, they were all, you know, whatnot.

And, uh, and then, you know, sent me home and they, they asked me if I wanted any

pain medicine and I, I literally said no, because I remember the last time

I was in the ER when I broke my nose.

For those longtime fans of the show may remember that, um, I.

They said, they said they gave me something for the pain, they gave me

like a Tylenol and like the lowest level, like narcotic painkiller you can give.

And they charged me $800 for that.

So I was like, I'm good.

Um, I'll, I'll sug it up.

Um,

I got a fifth of tequila somewhere in the house.

or some edibles.

Maybe some edibles.

Um, but yeah, so it was, um,

But I'm glad to.

Yeah.

Yeah.

Amazingly, no major injuries.

I've got some sore spots, you know, but no major injuries,

So the podcast shall continue.

Our series on ransomware shall continue.

And, uh, we're taking a break with Mike this week.

Um, and we're gonna talk about.

Specifically the backup side of things.

We've already done an episode or two that I will summarize as follows.

Your backup server is, is at risk, right?

It is

at high

risk

Yeah, it is, it is at high risk.

And, and, and there are, we have numerous, um, data points to back that up.

My favorite, I think what did Dwayne say and what, first of all, what, who was

Dwayne and what, what, what did he say?

yeah, Dwayne is a red teamer, right?

So he pretends to be the bad guy in attack systems.

Yeah,

I think he basically said, I love the backup system.

That's the first system I target because I get access to that.

I get access to all your data in your

environment because everyone backs up into a single place.

It's the key to the kingdom.

Yeah.

So if a red chamber thinks that, and by the way, if you haven't heard

the Red Team episode, go back, Maybe

two months.

Yeah.

Uh, there's an episode called, uh, you Know, about Red Team.

And, he clarified that, you know, both the backup system itself in

terms of how powerful it is, how much you get access to it, and also.

In terms of how poorly it often is designed from a security standpoint.

He talked about things like service accounts, right?

He said he loves the, the backup service account.

Uh, do you remember what he

said

default With the defaults password

or no password?

Yep.

But do you remember what he said about what was unique

about it?

Yeah.

With the backup service account, nothing gets logged in the system

I

you access it using the backup service account because it assumes you're

gonna be using it all the time and reading everything in the file system.

So I bother logging anything.

Yeah, exactly.

And so, you know, that I think that was an episode where we just said,

listen, you really need to understand your backup server is at risk.

We also had an episode or two where we talked about how to design the

server itself, um, in order to.

Better insulated from that risk.

Do you remember the kinds of things we talked about there?

Yeah, I think these were ideas such as segmentation.

It also included, don't have your backup server connected to your

normal active directory instance.

Kind of keep it isolated, separate, um, make sure you're.

Just to also the normal stuff, right?

Like make sure you patch your systems, including the servers,

right?

Keep up to date on those.

Yeah.

Yeah.

Uh, yeah.

So all the, the, usual stuff of, um, you know, obviously patch management,

password management, and MFA obviously,

right?

Yes.

I think the big one that you mentioned that I harp on a lot is

to separate it as much as possible.

Don't have it on the active directory domain, don't have it.

Um, you know, don't use the same username and password there

that you use anywhere else.

Um, I mean, that should be true anyway, but whatever.

This is a practice you should have everywhere, but you

should definitely have it here.

And that is don't ever log in as root or administrator log in as you and become

root or administrator you use, the concept of least privilege so that you can, uh,

minimize the damage that any one person can do

One I wanted to add there, because I know we've talked about this on

episodes a while ago, is don't save your password to access your backup

management system in your web browser.

Yeah, please don't do that.

Please don't do that.

Um, the, and, and, and you we're, we're big proponents of password management

here and, and and specifically password management systems, not your browser.

Uh, again, the browser's better than nothing.

Perhaps, perhaps not

in this case.

the Thing that, that I remember when we talked about third

party password managers, it.

Floored me the first time I installed Dashlane and it said, Hey, should

we go get the passwords that you stored in your browser for you?

And I'm like, wait, you can just get them,

Yep.

they're not like encrypted or anything.

You can just ask the browser, Hey, what's the password you have for this?

And they just sucked them outta there.

Well, it's encrypted, but it probably just has API access.

So as long as you

granted API access.

yeah.

But if, if you can get access by api, any other system,

software running on your

Yeah, yeah, so, so some sort, some sort of third party password management system.

And I do believe that it should be a different password management

system than the rest of the world.

Again, segregate, segregate, segregate, right?

As much as possible.

Obviously more than any other server, shut down any services that you don't need.

Especially my favorite, the ransomware deployment protocol.

Um,

Otherwise known as Windows RDP.

For those who don't, who may not recognize that acronym.

Um, and, um, but today I wanted to talk about how do we, um, design the

backups themselves, uh, to be more useful in times of ransomware, right?

Probably the first thing you need to do is actually do backups,

right?

That should be like your step one of your

strategy.

I live in this, I live in this fantasy world Prasanna where like

everybody does backups, right?

Actually just before this recording, I was on another.

You know, uh, another thing, and, and, and I realize that I'm speaking

to laypeople in this particular recording, uh, people in the legal

profession, and I was asked like, what is a backup and what is a restore?

And it's like, and, and I know that, you know, they're, they're catering

to an audience that doesn't understand

this stuff, but I just live in this world where everybody backs up their stuff,

Yeah,

that's why you need the 88 in the room.

IE me.

Exactly.

Everybody runs a third party backup of their iPhone because they

know that iCloud is not a backup.

iCloud is not a backup.

iCloud is a synchronization product.

Uh, not a backup product.

go listen to that episode if you're interested.

Yeah.

Go listen to the how to properly back up your iPhone.

When we think about designing a backup system for the purposes of, of, uh,

responding to a ransomware attack and then being able to recover, I think

it's important to think about, um.

A, a lot of things in terms of how does ransomware typically behave?

How does a ransomware response event typically take place?

And so I, I think the first thing to talk about is this concept of dwell time.

Do you wanna talk about that?

Yeah, so I think a lot of, and if you go back and listen to some of the

previous episodes and if, if you look, listen to what Mike has talked about

before, I think everyone thinks, oh.

I got hit with ransomware, I just got infected, and then boom, just

that next instance, everything in my environment is encrypted,

right?

That's what I think a lot of people think about it.

It's almost, but it's not how that works.

It's like you get infected by a disease, right?

It might take you a day before you start to get like a cold, and

then maybe a couple days later you start to spike a fever, right?

And so similarly for ransomware, we talk about something called a

dwell time, which is how long is.

The ransomware actually in your environment, even though it may

not be actively encrypting data,

Right.

Or exfiltrating data or whatever it is, but it's already got a

foothold in your environment and it

exists somewhere in your network.

Yeah, and and I, I think it's important to understand, again, and we've talked

about this on other episodes, that.

Remember that ransomware isn't a single piece of software,

right?

Um, maybe the actual ransomware is a single piece of software, but the

entire, there there is a suite of tools that ransomware actors are

going to be using to, to get, number one, to get into your environment.

Number two, two, spread around in your environment and to figure

out what's going on in your environment and that it's that final.

Uh, tool, the one that you're, you know, the actual ransomware tool

that's doing the, the, uh, encryption and or doing extraction, right?

Doing, um,

exfiltration.

But before that happens, you're right.

There is this, this process of like going through the network

and figuring out what is, um.

You know, figuring out what they could do.

There was a great story that Mike talked about where he said that they were in

an environment and they were doing a, um, they did a tabletop, and during that

tabletop they used the incident response plan, and they obviously shared the

incident response plan around everywhere.

And what they found out was they got a, they got a ransomware

attack right after this.

And what they found out was that, um.

That they had already been attacked and that the attacker was in

their system for quite a while.

And so he got to see the, you know, the, the, uh, what do you call it, the

incident response plan and all this stuff.

And they got to see like how much insurance they, all this stuff, right?

Uh, so that, that's a really big time.

And so what's the concern when

we start talking about restoring what?

oh, I was just going to mention like, I know this has come up in

the past, but one of the banks that I bank with that's a credit union,

they were hit with ransomware.

Basically over the 4th of July that shut down everything.

Mm-Hmm

Right.

But they finally published a, a analysis of what happened.

And they say that, so July 4th or July 1st is when everything got shut

down.

Everything was encrypted, right?

They said that they were in their network starting May 23rd.

So six weeks almost.

That would be the, that would be the dwell time.

Right.

And if you look, if you do some Googling, you'll find that the average

dwell time is actually really long.

Right.

Um, like the mean dwell time, last time I looked was like close to 90 days.

Uh, which means that there's.

Ones that are way

beyond that.

Right.

Um, it's not like they're all 90 days and it works out that way.

Yeah, because I think for these actors, right, there are

two things they want to do.

One, they wanna spread everywhere, so they have access to as much as possible.

And two, they want to figure out what's valuable in your environment,

Yeah.

right?

And so they

beat in there as long as they can.

Yeah.

And so if they go in and immediately start encrypting, right, using the

ransomware, you're going to notice and now they've lost that opportunity.

So it's kind of a

balance on their side, right?

They want to be in there spreading, observing, but the longer they're

in there, then the more likely it is for them to be detected as well.

So it's kind of this balance.

Yeah.

It is a balance.

Um, but the longer they're in there, the bigger the possible reward.

The other thing that they could do, and there, there's a couple

things that they could do while they're in there a long time.

One is they could start encrypt, encrypting.

And again, this is one where I'm gonna describe what I've been told has happened.

I haven't verified this, but it seems reasonable to me, and that

is that they start encrypting stuff that nobody's looking at.

Like the

Right, like older data that

nobody's looking at.

And, um, they do that because they could, they could do it and get away

with it because nobody's looking.

Uh, number one.

The other thing, what do you, what do you think is the other thing that

they could potentially do if they're in your system for a long time?

Exfil trading data.

Exfiltrating data, right?

More than likely you will end up paying the ransom.

Um, even though I hate the idea and all that kind of

stuff, right?

But it's a very different argument of like, oh no, I

don't have to pay the ransom.

Ah, I got good backups in a DR plan and it's a response plan.

They're like, yeah, but we still all your data, we're gonna

tell everybody what you did.

Um, um, the longer they're in there, the easier exfiltration is, right?

Because they can do it slower.

They can, you know, send it out.

Which again is why I continue to say, please figure

out some way to track outgoing traffic.

Yeah, which is actually what happened at this credit union.

They ended up exfiltrating data from their database in addition

to encrypting everything.

So social security

Are they back up by the way,

that

Uh, yeah, I think they are back up.

Uh, last month they had to send out paper statements 'cause they

weren't fully up and running, but I believe now they're up and running.

But all customer data is out there on the dark web.

Yay.

Yeah, because, uh, again, fans of the podcast may remember that, uh, my medical,

uh, group got attacked with ransomware in May, and I found out last week

they're still not fully up and running.

Yeah.

maybe they don't have backups.

Curtis is what I'm guessing.

I, I don't even wanna know.

Um,

all I know is that for months, the only way I could make an appointment

was to go into, I could, I had to physically drive into the doctor's

office, which luckily for me is like, I.

15 minutes from my

office, uh, I could go, you know, drive there, make an appointment.

And it was, it was actually kind of good because it meant it was a pain to

make an appointment, which meant that it was easy to make an appointment.

Is that, you understand what I'm saying?

It was, it was logistically difficult to make an appointment, which made it

easier to make an appointment once you

were there because

No

many people, yeah.

No one wanted to deal with it.

Right.

Uh, but yeah, what I found out, again, this was just last week,

that the phone part of the.

System is still not back up and running.

Um, yeah.

Yeah.

Com completely crazy.

What's worse?

Is it pre It's pretending that it's up and running.

I

called the number, I waited on hold for like 20 minutes and

they're like, you know, press one.

You know, to, if you're a provider, press one.

If you're a patient, press two.

And if you wanna speak to you wanna make an appointment, press two.

Okay.

And, and then it's like for the.

Chula Vista office, press one for the Encinitas office, press two, you

know, and I'm like eight, like I'm

gonna sit there with, you know, and then I finally press eight.

And then it's like, please hold.

And then, and it was like, it was like 20 minutes later, click.

Ah,

I was like, what the

hello.

So I was like, well, if I'd known first off, I'd have known

it was gonna take so long.

I would've just, I would've just drove over there.

Let alone, if I'd have known, maybe, you know, maybe I would, should have

done is got on the phone and then Dr.

And then drove.

And then drove there.

Yeah.

So anyway, so I made my appointment, um, and and then, and then

proceeded to fall down the stairs.

Um, all right, so if there's a long dwell time, what do we need to be

talking about with regards to, um,

backups?

Well.

So when you are getting to the point of recovering from your ransomware

incident, you wanna make sure that the data you're restoring.

Is clean, right?

That it doesn't contain any bits and pieces of the ransomware,

right,

Or any of those other intrusions that have happened in the past.

So you

really wanna find a clean point so it doesn't sort of

restart itself and you end up

Right.

So how would that affect, how would a long dwell time affect

the design of your backup system?

You need a longer retention because you need to make

sure your backup is

a clean backup before you got infected.

I say this because it's very common, and again, I, I had a

conversation with Mike about that.

It's very common.

For people to say, oh, well I only need 90 days, right?

I only need 90 days for my, my backup retention.

And I understand the reasonings behind that.

Right.

Um, and uh, and, and Mike was saying that it's very common he's seen it in

the field where people go to restore their stuff and the retention period

of their backups is less than the dwell time of the product, and they're

unable to successfully restore.

I have a question about this

though.

So I totally get it.

Ideally, you want a clean backup, but

if you go, say you have to go back three months, right?

So you restore from something from three months, and now you

have to sort of roll forward

to get back to the current point in time.

yeah.

That's the next thing to talk about.

yeah.

My question though is, is that better than restoring a non clean backup and

then surgically going and cleaning it up?

Yeah.

So the, that's a really good question.

Uh, let's finish this

point and then let's go to that point.

So all I'm saying is you, you know, to have options,

you need a much longer restore, uh, a much longer retention period than 90 days.

Um,

my, my general thing would be like a year.

Right, like a minimum of a year.

I would actually, I would say for a business minimum of, of 13 months.

Because sometimes you want stuff from like the annual report from a

year ago.

Um, I don't see any problem with going a couple of years, any

retention, you know, I don't want you to go much longer than that.

I was gonna say, don't consider this an archive.

Don't use your backups for archiving,

but yes.

Um, but the, you know, there, there's nothing wrong with having a couple

of years retention for your backups, maybe even three years right now.

That's not really a ransomware defense at that point.

Um, but you want options.

Okay.

Um, what, so my, Mike and I spent a quite a bit of time talking

about this, this issue of.

Clean versus, you know, completely clean versus clean versus cleaning it.

Does that make sense?

Right.

So he, he explained a, a couple of things.

One, and, and it, the, the, I'll just say this is a complicated question

and it's, and it's more complicated when we start talking about file servers.

So first off, let's just, let's just do the, the, the, the.

The, you know, good, better, best, right?

the the best.

When you ask someone that's responsible for that, that has been

through this, they will tell you that the best thing you can do is a

complete wipe and restore afterwards,

a, a complete wipe and a restore, especially when we're

talking about the system,

right?

The os.

Um, that

a clean slate, right.

start from a clean slate.

Okay.

Because there, there's two different things here.

There's restoring the systems and then there's restoring the applications.

There's actually three, and then there's

restoring sort of just d like unstructured data,

right?

So regarding the systems, uh, he feels pretty strongly that this

should be a, a clean wipe install.

It is possible, it is possible to do what you're talking about where you restore

the, the system and you and you're able to find, you know, you find out what

variant of ransomware you have, you find out what tools that variant installs.

You uninstall those tools.

The concern

with

might have missed something.

restore and yeah, but not just that, but.

You.

The big thing is did it install something like in the boot block

to basically re-enable, you know,

Yeah.

if you can also fix that, right?

yeah.

Well, and I think this is some of the challenges today too.

I don't know if you saw, but there's a new malware variant that

injects itself in the UEFI boot.

Which is

literally baked into the motherboards,

right?

That's supposed to be super secure.

And so in cases like that, you're basically hosed, right?

You don't even wanna restore that data because it's always going, like

you said, it always keep coming back over and over, no matter what you do,

You.

You know what?

I'm gonna pull a Prasanna.

what,

Prasanna, you

just used an acronym.

what is.

UEFI?

I know what it is.

I just, I just realized I have absolutely no idea what that stands for.

But first off, what is A-U-F-U-E-F-I?

Boo.

What are you talking about?

So UEFI boot is how the system boots up based on, so in the past, right,

you had the master boot record and

it had sort of certain blocks where it knew where to go in order to load it,

in order to make things larger because there were certain limitations and

also more secure like windows and other things use what they call A-U-E-F-I mode.

In order to be able to boot your operating system.

It stands for Unified Extensible Firmware Interface.

So it's basically the, it's the next generation.

It's been that way for a long time.

Right.

Um, right.

Uh, you know, it's been a long time since MBR was that Master boot record.

The

NBR was the only option, but yeah, so this is just what I'm talking about.

I I guess, again, and this is why I'm, you know, I'm such a fan

of having an expert in the room.

This is why you bring in somebody like Black Swan Security, right.

Uh, to, to to come in, which is my Mike's company.

But the, the.

But to go back to the design issue, there are two things that you

want to make sure you have, right?

I'm gonna say three things, but two things that you wanna make sure you

have is, you know, the retention period, and also a high enough frequency

that you know you have, again, you have options.

What does frequency, when we talk about.

There's a couple of, couple of acronyms that we, we actually, we

actually haven't said it in a while.

You know what else we haven't said in a while?

Yeah.

RTO and RPO.

Which one?

You know what?

What else we haven't said in a while?

What.

The 3, 2, 1 rule.

The three 12 hasn't come up in a while.

We've been talking about ransomware so much.

We haven't talked enough about backup, but so when we talk about RTO and

RPO, right, recovery time objective, that's how fast you can, you, you,

you, you want to be able to restore and then recovery point objective,

how much data you're allowed to lose.

So frequency is going to impact which of those,

RPO

and RTO technically

potentially depend, depends, right?

okay.

Yeah.

I guess you're, I guess you're right.

Yeah.

Yeah, yeah, yeah.

Um, it, I, I, think maybe like levels and things

combined with frequency might affect, affect your RTO, so, um, it's just so

funny, like I've spent so much time.

In this new world where we don't do levels right, we just do one full and

incrementals forever, which is the way all backups should be done, but whatever.

I'd digress.

Um, the, um, I, I really do think the concept of like

repeated fulls is really a, a, a

concept that

needs to be done away with.

Yeah.

But, um, so you wanna make sure you have a long enough, um.

Retention period.

You wanna make sure you have a frequent enough backup.

Uh, and then what I want to talk about this is, this is my third thing that

I'm talking on the end of my two things, and that is you need options during a

restore because when you are, uh, when I asked Mike to sort of walk through

what a, what he felt was like a typical restore scenario and what he described.

Was many, many restores of the same system that then allowed you to

pick apart, to say, okay, do you know we wanna restore this version?

Nope, that one's infected.

We wanna restore

this one.

No.

And that's infected.

And keep going backwards until you get to a system that is

not infected.

Right?

That's clean.

If you didn't build that into your design, a ransomware recovery is

going to take significantly longer.

Now, I'm

yeah.

Yeah.

gonna give you, I'm gonna give you a freebie.

What

design element I.

That's backup related.

Could you be using that would allow you to have infinite recovery points

without, no, sorry.

You fail in, let me finish my sentence.

Allow you to have virtually unlimited recovery points and while in RT

with RTOs of like next to zero.

Do I have to answer this?

I don't like this answer.

I know what it is.

What is it?

It's, three letters.

Oh, no, wait.

Oh, no, no, not, not infinite.

Sorry, not infinite.

Number of recovery points ne, nearly, nearly

So, uh, so, so, okay, so let me, let me check.

So, in my mind as you're describing this, I'm thinking of CDP, which

is continuous data protection.

Yeah, that would be infinite.

Okay.

Uh, there is also snapshot based replication.

Go.

Right.

Okay.

So it only took me two tries, which isn't bad.

So I gave you, I gave you a leading question, but I guess

I didn't lead you enough.

I did.

I thought that, you know, given your background, this would

just, just jump right out.

But maybe, maybe they beat you out of it enough at your previous employer.

So, so.

This is where snapshots and what I would call near CDP, right near

continuous state of protection, this is where snapshots can be so useful.

Because how, how I, you tell me Prasanna, how else can you

restore hundreds of versions of the same server without much pain?

You can't.

Unless every copy or every backup was on a separate tape, device or tape,

and you had infinite number of tapes connected to infinite number of devices.

Right,

right.

So you could do the

restores in parallel, basically the

and by the way, that's so you can restore one server a hundred

different ways, and then you've got a hundred other servers.

Yep.

So, yeah, you would need infinite, infinite, infinite, infinite, infinite.

infinite.

I, I guess what I'm saying here is think about this, right?

Think about, um, and this is where, uh, and again, we, we get into more storage

here than backup, but you, you know that I'm a fan of this, this concept

of snapshots and replication and that if we think about like there, there's

a lot of technology that allows you to.

Store your virtualization world, and I'm a big fan of virtualization, store your

virtualization world on, on a filer,

right?

That allows you to take snapshots and replicate those snapshots and

replicate them even to an immutable, uh,

device if you want.

Right?

And then you have infinite number of recovery points.

Yep.

Um, I

I,

And I, I

I wanna add one more thing on top of

okay, sure.

I think

deduplication becomes important.

Because everything you're talking about right now is for a single

server, but now say you have a hundred servers that are all based on like

a similar image or whatever else.

I think

I I think deduplication can

add a lot of value in terms of, it can really help

reduce the

amount of storage that's gonna be used.

Yes.

Um, I don't think it's required in terms, it, it is just gonna save you money.

Right.

But the idea of, I guess I just, this is where, uh, you know, this, this

is one of the, this, this is why I am such a fan of, you know, not just

NetApp NetApp's not the only one.

There's so many companies, and

it's not just filers.

It's not just na, it's, they're also SAN devices.

There are iSCSI devices.

There are modern scale out storage arrays that have.

Infinite, or, you know, short, short or close to infinite number of snapshots That

don't impact performance, And that would offer you, um, some real choices here.

I, I think also we can throw in the concept of copy data management.

There are, there are CDM products, like, is it still

called acto after it got acquired?

I think it's still called Actifio.

Yep.

so products like that, basically, I guess what I'm just saying is

this is a real problem, right?

This is a huge problem and this is a potential, really useful

tool towards this problem.

Maybe it won't solve all known, you know, things, but when we start talking about.

Uh, hey, I wanna do a hundred copy.

You know, I wanna keep retention for this long, and I want, I want

to potentially restore my server a hundred times and I, but I don't want

to restore my server a hundred times.

Yep.

Um, it just seems like snapshots and replication would

really be your friend here.

Yeah.

Oh, definitely.

Yeah.

I don't see how else you're gonna be able to even figure

out if a copy is clean, right?

Without something like this,

And, and this is, you know, the another, so I'll throw this here.

Another possible friend is the cloud,

right?

If you are using a cloud-based recovery system, and if.

That recovery system has the ability to scale out and say, I wanna

recover, bang, bang, bang, bang, bang.

Right?

Yep.

It's just that all the ones that I've seen, the, at least the ones that

I've seen, you know, essentially with my own eyes, when we start talking

about recovering many, many copies.

They can scale it out.

Right?

so

they, so they, they do, you know, you remember earlier when you said

if you had a infinite tape drives and

all that, they have

the cloud lets you do that.

Yeah.

the cloud lets you do that.

Right.

But, uh, the restore of the actual server will still take.

A finite amount of time.

Right.

And again, build that into the design, figure that out, go to the vendor,

say, Hey, here's what we wanna do.

We wanna be able to do, restore the server a hundred times

and pick which one we want.

Can I do that?

Well, yeah, it's gonna take you three years.

You know,

figure, have that discussion now.

You know, build that into the design.

Um, I, I guess I'm.

I'm just really, and, and maybe this means you change vendors,

right?

Maybe this means you change storage systems.

Um,

to your vendor and see how do I solve this?

Here's my

Well, you, you definitely should do.

that first.

You know, I'm a

fan of, I, I'm not a fan of like, uh,

uh, steeplechase, I call it, right?

Not a fan of just going, you know, place to place just to, you know, I

always think you should, you should.

Let your current, you know, give your current vendor the problem.

Don't go to them with the design.

Right.

Go to them

with your requirements.

Right?

this is what I'm looking to do.

What's the best way to do this?

Because maybe they have a mechanism that.

Yeah, I listened to this podcast and Curtis said I need to be able to

restore my server a hundred times.

Like how, how do I do that?

Is there a way to do that with your product?

Right.

Um, or, well,

actually, technically what?

What you, you would say, I need to be able to,

Identify a clean

copy of

a clean image.

And so maybe there's a way to do that.

Um, yeah, the, the example I always give for dictating the

requirements and not the design.

I live in San Diego and we have Coronado, which is, it's not an island,

but people call it Coronado Island.

Um, it's Coronado Peninsula, but that doesn't sound as cool.

And I always give the example of, of say, listen, I need a hundred

thousand people to be able to go to and from that island every day.

That's a, that is a requirement.

And maybe it's a tunnel.

Maybe it's a ferry.

Maybe

it's a bridge.

Right.

Or in the case of San Diego, maybe it's all three.

Not a tunnel.

But we do have a bridge.

We have a ferry, and we have the long way.

The

long way.

Uh, it's funny, in San

Diego you can literally see Coronado, it's like right there.

It's like a half a mile on the other side of the water.

But the long way without the bridge, it's like four, like a 40 minute

drive because you gotta go to Mexico.

No

kidding.

You gotta go to Mexico, turn around and come back.

Um,

yeah.

Anyway, you could just swim across.

Anyway, I think that, I think that was good.

It was a

good conversation.

We did not talk about two things though

that you had brought up earlier.

The one is you had talked about restoring the server, but not the databases or

Yep.

Yep, yep.

Yeah, yeah,

yeah, yeah, yeah.

So, all right, so a couple things that we didn't talk about, right?

Uh, this idea of restoring the server and perhaps restoring the data later.

I, I, I'm thinking mainly about the idea of like using an image

perhaps to restore the server,

and then we restore the, and that that image would probably

have a clean copy of Oracle or

whatever it is that you wanna restore, and then potentially restoring

the data as a secondary thing.

I, I think this is another.

Potential way to do this.

And by the way, virtualization makes all this so much easier.

Right?

Um, but which is, you know, on the list of why I, I'm such a fan of virtualization.

But that is a potential, again, just look at these designs and

then, and then work with it.

When we look at that method of doing it, the, I think this is

a much more valid method for restoring, say, databases today.

Because I say this, you know, I'm recording this on August 21st, 2024.

Today, they don't tend to attack databases directly, meaning they

don't go into the database and, and mess up the individual contents.

If they encrypt the database, the encrypt the database file.

Right?

So you just wanna restore.

You wanna restore the database.

Yes.

You restore the file.

You restore the file from before it was encrypted.

You're good to go.

Generally speaking, so I think that's a really valid way to restore a database

server and an application server that

has something like a database on it.

the the real concern I do have

is when we start talking about unstructured data and file systems,

because what did we start this?

What did we start this podcast talking about?

Do you remember the phrase that we defined in the beginning?

The dwell time.

So, so what, why is having a long dwell time?

It's like, hey, I get, you said 90 days, right?

I got, I got six months.

I got, I got a

year.

Curtis said to do two years.

I'm doing two years.

I got two years of backups.

So what if the dwell time is 90 days?

What's the

Well, well, because you, 'cause it is gonna go through and

like we talked about, right?

It might decide, I'm gonna start with the old data and

slowly start encrypting things.

And maybe you notice, maybe you don't notice, but.

Now you have to go figure out like a needle in a haystack.

Except the haystack is a small number of files in say, a

billion or 5 billion files.

So the good news is there's gonna be one of two scenarios, and

most likely you're going to be the first of the two scenarios.

The good news is, I think most of the time.

You're gonna look at the server, you're gonna look at the even unstructured

data, and you will be able to easily identify which files were encrypted,

and you're gonna find that they were all encrypted at the same time.

They were all encrypted, all on the same day.

I think that this idea, again, August 21st, 2024, I think this

idea of slowly encrypting the files over time, one, one of two things I

think it is at, at a minimum it is.

More rare than the other method because again, the

moment they start encrypting files, they really set off

alarms, right?

So I think it's pretty rare and I even think it's possibly a boogeyman.

I,

I, I don't know for sure, but if you have this problem

though, there's no good answer.

Uh, you know that, well, the, the, the only one that I am aware of,

right there was, you know, from, from our previous employer, they

had a solution to this problem where they, they, they, they had this, they

called like image curation, right?

Where you could give them a range of time.

They go and they would go in and automatically pick the last good

version of every file prior to it being encrypted, doing that manually.

Yeah.

If you have this slow encryption, doing that manually is, is,

you know,

And then, and then you just need to think if it's all the old data, it's

encrypting, do I really care about it?

yeah.

It's the, um, take it.

Um.

Again, I'm going back to a hundred years ago when we had this, uh, this

old server that we were decommissioning and it had been around so long

that nobody knew what was on it.

And so basically we got down to like the final one or two servers that were part

of the, it was a, it was at and t's first attempt at a multi-processing computer.

Right?

And so it had multiple computers inside the computer.

And so we got down to like the last one or two, and basically the, the idea was.

Uh, we just turn it off and then see who yells

Sometimes that works the

because we couldn't figure out, you know, we couldn't figure

out who was on it and what it was doing.

And so that's the same kind of thing.

Like, you know, if a, if a file gets encrypted and nobody reads

it, that it really get encrypted.

It's like, you know, like if a tree falls in a forest, if a

file gets attacked by ransomware and nobody wants to file, who gives a crap?

Uh, that's our, that's our, uh, that's

our final piece of advice.

All right.

Well, this, this has been fun.

I think it's

good.

You, again, it's, this is a little bit more far reaching in terms of

some of the design elements and design ideas that you would put in there.

Um, but I think it's one that, that people should really be thinking about.

Yeah.

And this isn't intended to be a conclusive list,

right?

But this is just initial thoughts to get you thinking and go have your

discussions with other experts, with your vendors, right, to see what

else you should be thinking about.

Exactly.

Exactly.

Well, thanks Pana.

This was fun.

I know this was fun and Curtis, I'm glad you didn't die.

'cause then I would be sad.

One, one final thing on that, uh, 'cause I don't think I mentioned it earlier.

You know, we, we, we have some people that are renting are

renting a room from us here.

And he, he happened to be the only one that was home when this happened.

And he heard it happen and he was very glad to hear me yell at his name.

'cause he's like, he was scared to go out.

He, he was like, he heard it happen and it sounded awful.

And he is like, I, I hope he is not dead.

And then he heard me yell at his name and he is like, oh, thank

God.

thank God.

dead.

Well, I'm glad I'm not dead too.

Prasanna, so that we can

I'm sure our listeners are as

Yeah, I, yeah, you don't care.

All right.

If there's anybody out there that's glad I'm not dead and they're still listening,

send me a note on backup wrap up.com.

Send a message.

This, uh, I'm glad you're not dead.

Or put it as a comment on the, uh, on the YouTube video or a comment on

the, you know, on the, uh, on the,

on the backup wrap up.

Anyway, well, uh, thanks to our listeners.

Uh, you know, we kid, but we love you.

You're the only reason we do

this.

Otherwise, just a couple of guys just talking.

Uh, and we'd probably just talk about barbecue then.

So that is a wrap.