You found the backup wrap up your go-to podcast for all things

backup recovery and cyber recovery.

In this episode, we're declaring the death of the 3, 2, 1 rule.

Sort of the 3, 2, 1 rule has been a foundation of backup,

uh, best practices since.

The nineties, but it's time to admit that it's not quite enough.

Ransomware has changed everything.

Threat actors are going after your backups too.

So the 3, 2, 1 rule had to evolve.

It's now 3, 2, 1, 1 0.

We'll break down what those extra numbers mean and why immutable

and air gap copies are now.

Non-negotiable and why?

Zero backup failures matters more than ever.

By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr.

Backup, and I've been passionate about backup and recovery for over 30 years.

That's a long time ever since I had to tell my boss there were no backups of that

production database that we just lost.

I don't want that to happen to you, and that's why I do this.

On this podcast, we turn unappreciated admins into cyber recovery heroes.

This is the backup wrap up.

Welcome to the show.

Hi, I am w Curtis Preston, AKA, Mr. Backup, and I have with me a guy who

I used to think was smart until he told me the story he just told me.

Prasanna Malaiyandi how's it going?

Prasanna.

I am good and I, I'm glad I am bringing you to reality rather than

putting me up on a pedestal with all

So, so what did you just tell me?

Why did you have a, a cut on your forehead?

Yeah.

So for people who, by the way, we do YouTube, so the podcast, if you want to

Yeah, if you wanna watch this on YouTube or, yeah, yeah.

Yeah.

wrap up channel.

You can watch us there.

But, uh, well, yeah, when I was younger, I decided I wanted to be like,

you know, all those wrestlers, like where they're like, girl, and then

they crush the can on their forehead.

Yeah, you did that and

So I had an empty can and I was like, oh yeah, that's cool.

I'll try that too.

Yeah, let's just say that

no bueno.

up with the cut.

Yeah.

Yeah, we of course, started this conversation because of this little

gash on my forehead due to just my inability to navigate me, me

trying to prove, once again, prove.

A fundamental law of physics, you know, that no two objects can occupy

the same space at the same time.

Y you know, I think someone's just moving stuff around on you,

Curtis, I think like, like yeah.

Things jumping out and places you don't expect.

I think someone's messing with

I wish I could blame that on this.

Unfortunately, this, you know, this was a shelf.

A shelf which I mounted.

Yeah.

So I can't really blame moving around stuff, but.

It's

Uh, yeah, but speaking of moving around stuff, move, you know,

the 3, 2, 1 rule has been moved around a bit, I think, you know?

what is a 3, 2, 1 rule?

Yeah.

Let's start with that.

And, and I would say that it is like, you know, first off, um, let,

let's, let's just say I, I would say it's one of the most fundamental

sort of concepts in backup, right?

Do you remember when every episode had the 3, 2, 1 rule in it?

Yeah.

Yeah.

It, it came up a lot.

Right.

Um, and so, and, and, and, and we've, it's somewhat morphed over time in terms

of what we, what the 3, 2, 1 means.

Um, uh, but, but let's just start with it.

You know, it's three copies of your data on two different media.

One of which is somewhere else.

Um, and that we used to say, one of which is offsite.

I've changed that to somewhere else because that, that sort of

goes with the cloud concept, right?

By the way, if you wanna listen to more about where the 3, 2, 1 rule came from.

We actually did an episode with the person who created the

Yes.

Yeah.

The coin, Peter.

It, was it Kro?

I think it's pronounced Kro.

Um, it's K-R-O-G-H-I believe.

Uh, yeah, with the guy that coined the term, he, he was a, or is a

digital photographer, coined the term back in the nineties and we

actually had him on the podcast.

That was very cool.

I'll put a link to that in the, uh, in the podcast.

And, and he was just trying to help.

Um.

Uh, you know, digital photographers do the right thing and, and we're like,

yes, we like this right thing, you know?

And, and, and, and the thing is, it should be more just like what

we're gonna talk about today.

It, it's like a very basic, like if you, if you aren't at least doing

this, you aren't doing backups.

Yeah,

Um, it, it's not like this is like the way to architect a backup system.

This is if you don't have at least three copies of your data, if you don't have

at, on at least two pieces of media and at least one of them somewhere

else, you, you just, what, what?

Don't even, don't even talk to me.

Right.

Um, where, where do we, where do we end up using this rule A lot as a

sort of a proof point to say, well, you are not doing backups, do you?

Do you remember where that comes up A lot.

I think this came up when we started looking at like SaaS backups.

Yeah, or just what, or, or the non-existence thereof.

Right.

Um, right, because there, there are so many people, it's like, uh.

I, I found, I, I have found myself arguing with people online, you know?

I,

Uh,

laughing 'cause I think one of the first times, maybe like six months

after we met, or maybe a year after we

yeah,

I remember you going on this rant how frustrated you were with Microsoft.

yeah,

Claiming Microsoft 365 need backup.

Yeah.

And there was, there, there was, and is a guy that.

Actually is a Microsoft 365 expert, and he wrote a book about Microsoft 365 and I

bought it just to see what he had to say.

And literally it was like the fir, the chapter on backup basically

said, strictly speaking, you don't need to backup Microsoft 365.

And I was like, ah.

Right.

Um.

And, and again, it's not, it's nothing against Microsoft 365.

It is.

Just to go to the, to this topic, you're your, your, your, your copies as I make

quotes in the air aren't copies, number one, it it, like, it doesn't, it doesn't

conform to any of the 3, 2, 1 rule.

Right?

So you're, you're, there aren't multiple copies.

There are, there can be versioning within, uh, within.

365, but your copies aren't copies.

Why aren't they copies?

Because, well, one of two things.

One, they either rely on something else, like they're part of production,

right?

right.

Uh, so as an example, if you're

using Recycle bin to hold your copies or whatever else,

right?

It's still part of the same production database.

And

so you have that issue or the fact that like these are things that

you don't have access to as back.

Up admin.

Yeah, it's, yeah, it's, it is multiple things, right?

Like you said, it's, it's not, well, it's not a copy 'cause it's not a copy, right?

It's not, it's all, everything's all in, in one place.

It's not, you're not actually taking a copy of it and putting that copy

on some other piece of storage.

If you copy yes, if you go on your hard drive.

And, um, or your drive, I guess, you know, it's not always a hard drive these days,

but usually not a hard drive these days.

If you go on your laptop and you right click, uh, on your, your, uh, your, your

drive and you say, copy this file and then paste it, you've, you've, you've

essentially made a copy of the file

Mm-hmm.

you at least have a separate.

Um, instance of that file that is a copy, it's not yet a backup.

Yep.

What's the difference between a copy and a backup?

It's on something else.

It's still

Yeah,

next to your production.

yeah.

That's the, that's the whole 3, 2, 1.

You know, if you just make a bunch of copies.

And you don't separate those copies from the drive, right?

So this is like, you could do three copies, but all on the same drive.

You don't really have a backup at that point.

You need to put it on a, another drive,

It's

right?

about NAS systems, right?

And one of my former employers, right?

Yeah,

okay.

You can do snapshots of your production volume, which is great.

You

yeah.

versions, like you said, but typically those snapshots lived with the production.

So if you

Yeah.

volume or you lost the system, you lost your copies.

And those are not copies, right?

Those are, those are virtual copies.

But you know, you were saying it sits on the same thing.

It, it's relying on the storage.

So that's not a copy at all.

So that doesn't follow the 3, 3, 2, 1 rule.

Right?

So, uh, usually when, so my point is that usually when we use the 3, 2,

1 rule as a cudgel these days, we're using it to just basically prove, it's

like look snapshots by themselves.

Are not backups, SaaS, uh, you know, recycle bin, uh, et cetera, et cetera,

retention policies in Microsoft 365, which is like a fancier version of

the, um, of the, um, recycle bin.

You can say, Hey, uh, you know, you can't, you know, every object, which

is a term would, that would include emails, files, spreadsheets, et cetera.

Every object has to be retained for at least 30 days or 90 days, whatever.

You could create a retention policy that every.

Object once it's created, is retained for at least 90 days,

um, even after its deletion.

And you can say that, uh, that copy while being stored is immutable, right?

You can say that it cannot be deleted out of that, that thing, right?

Um, but that it, it's all, it really is, is a big fancy database.

With a, with a, with a, you know, it's, it's a special purpose database.

It hold that holds the emails, it holds the files, it holds all of that.

Uh, and so you're not copying anything anywhere.

, If you're not copying it anywhere, it's not a backup.

It's a, it's a convenience copy.

Just like the copying and pasting a file within the same hard drive.

Okay, so Microsoft would argue that with Exchange or with Microsoft 365,

Yeah.

replicate their data

Yep,

offsite location,

yep,

Secondary, just in case something happens to the primary data center,

A and,

your service.

and that replication has a delay in it, right?

So it is a delayed, replicated copy, which is great.

The only problem is you get no access to that.

Right.

So, and I've verified this as a customer of a very large company,

paying, you know, a crap ton of money every month to Microsoft.

Say, Hey, let's say somebody obliterated a, you know, we

obliterated a user within 365 or ransomware attacked a user within 365.

Could we use that delayed copy?

As a method to restore it.

And the answer was an emphatic no.

That is not what it's for,

Yep.

It's, it's there for, um, um, what do you call it?

It's there for the, essentially dr for them, right?

If the, if the, which is good to know, right?

That they do have DR for their, um, you know, for their environment.

Uh, but it, it's not for you, right?

and, and I think that's important to understand as you're going from workloads.

On systems that you own, that you operate,

Right,

versus SaaS you now have this split responsibility model,

and you might have to do additional things in the SaaS environments that

you think you don't have to do, but it's actually more important that

you do backup being one of them.

Yeah.

And, and if you, if you don't have access to that backup, you do.

If you don't have independent control over that backup, then

you don't really have a backup.

Right?

So this is what the 3, 2, 1 rule is about, right?

Is that, is that we, we, we need to make another copy.

That copy needs to be in your hot little hands, not necessarily physically.

Right in your hot little hands.

It could be in the cloud, it could be in an, it just needs to be

somewhere else again, and, and, because in the SaaS world, they

don't even conform to the two, right?

They don't conform to the three, they don't conform to the two, and they

definitely don't conform to the one, which is why we end up using the 3, 2, 1

rules of cudgel for saying that the stuff, the stuff that they do is not a backup.

Yep.

Um, and so while the title might of this podcast might have suggested

that the 3, 2, 1 rule is dead, it, it's not, it, it still has a purpose.

Um, and that is, and, and the purpose that it serves primarily is it's a

cudgel to say, Hey, that's not a, that's, those aren't backups at all.

but, but why does it matter to say that those are not backups,

Well.

does that matter

Yeah.

That's a great, yeah, because again, if something happens, specifically the

number one reason people are restoring these days, ransomware, if a ransomware

attack attacks your 365 account and you're unable to use your backups,

uh, to put everything back, uh, then why, why were you even making them?

Right.

The the other thing, and, and, and again, this isn't so much on the 3, 2,

1 rule, but the other thing is that.

The other thing about, especially when we talk about 365 and, and similar

products, the, the thing they have that, that some people think of as backup,

it also is really bad at Restore.

Right.

Just functionality wise.

Yeah.

It's good at bringing back a handful of files or a handful of emails that

you were looking for, but in terms of put my inbox back to the way it looked

before this thing happened, it just simply doesn't have that functionality.

I,

it's a bit more like an e-discovery tool or an

archive like system rather than a backup and restore tool.

Right, right.

Um, so.

Why then?

So if, if we still think that it has value, why then do

people say 3, 2, 1 rule is dead.

People like Rick Vanover over at

I, I, I was,

Veeam.

well, I remember when was on the podcast

Yeah,

he was, I think he added a few extra numbers

yeah,

2, 1, right.

yeah.

He has indeed, in fact.

When I was researching for this podcast, uh, you know, nobody ever actually

visits the, uh, the site anymore.

You just get the, the Google summary, right?

Um, and, uh.

But, but in the Google summary, this is new.

I've actually never seen this until just now.

In the Google summary, there was a video, uh, and it was, it was rickatron

doing the 3, 2, 1, 1 0 on the, on the, uh, the, the glass, uh, thing.

Nice.

and I was like, oh, look at, look at Rick.

He's, he's popular enough that he shows up in the, in the thing there.

The question that was on the table was, why do people say that

the 3, 2, 1 rule is not enough?

Because 3, 2, 1 is good for most cases, but things have changed.

I know you alluded to it earlier that most of the time when you're

recovering, it's due to a cybersecurity

incident or ransomware or something else like that,

where just having that one copy offsite is not good enough.

Well, it, it's not just that it's offsite, it, the, the big thing

is that it needs to be immutable.

Right.

That's the thing that we really, you know, in my first shoot, my first

25 years in the backup space, I don't recall ever using that term.

I

did.

You did.

I did, yes, but it was more intended from a compliance perspective.

So it was

around like Sarbanes Oxley and

those sort of requirements rather than backup and

yeah, and that shows sort of your journey versus my journey, right?

I was focused mainly on backup and recovery and, um, the, um, uh, what's

funny is I, I do remember a lot of SOCS compliance stuff that we had to do, but.

I, I think we, we still didn't use that term.

Right.

Uh, let's just, again, let's define the term immutable.

Immutable just literally means that it cannot be changed, right?

And this is true that cybersecurity incidents, AKA ransomware usually,

um, is I think the number one reason that people do restores these days.

And since that's the case, and since we know that the first thing that a

ransomware, a threat actor is going to do is attempt to disable your backup

system, um, the, this is why the first thing, the, the next thing you need to be

adding to that 3, 2, 1 is that at least one of your copies needs to be immutable.

Now, do you wanna define immutable?

So immutable means that you're not able to delete the copy.

No one can really delete the copy before a certain time period has elapsed.

So you might say, okay, keep the copies around for three months and before three

months, and admin can't go delete it.

Uh, malicious actor can't go delete it.

The system doesn't allow you to delete it unless you basically go and.

Pull the drives out and

Yeah.

on it,

Yeah.

giant drill.

'cause nothing is immutable if you have physical access.

Yep.

Right.

Um,

yeah.

Immutable just means it can't be changed.

Right.

But when we talk about this world, essentially, I think the

only thing that's truly immutable is if no one can change it.

Meaning you can't change it even if you have all super powerful

admin access to the system.

If you can still change your mind and then, and then delete it, then that's

not really immutable, in my opinion.

Or change a retention period

Or ch Yeah.

Change of retention period, which then causes it to be deleted.

Right.

If you can do that after the fact, you know, if you turn the, the, if you, if you

go to the super, you know, extra special level, then that's not really immutable.

Now there are some, uh, systems where.

You know, I know that you know where the normal customer can't delete

it, and so there are some system situations where a vendor has the

ability to go in and delete it.

That's, I have less of a problem with that, but it's still not truly immutable.

If you can still, if there's still a back door, even if the back door has

lots of humans in front of it, right?

Humans can be engineered,

Yep.

That's exactly what I was thinking about

was especially with AI out there and

People being able to do deep fakes and other things like that.

It is a possibility.

yeah, I, I, I think that really the way to do it is to have, you know,

to, to use something like object lock with, with object storage, right?

Where you basically say, I'm gonna put this thing here.

It's gonna be here 90 days, or whatever number you have.

It's gonna be here 90 days.

And no one, including me can delete it before that timeframe.

And I can't change my mind once I put it there.

I can't then go, oh, 90 days, well, did I say 90 days?

I meant, I meant three days.

Right.

Um, because that's another, that's another way that, that, um, a threat actor might

delete, delete the backups by just simply telling the backups to delete themselves.

and I think people should go research the immutable functionality of

their storage systems because most do support two modes, One that's

very strict, that won't allow anyone versus one that does allow

super users to tweak things.

So

just make sure you understand the implications of the systems you're using.

Thanks for bringing that up.

It's human nature.

To choose the less restrictive of the two options because you're

like, I can't change my mind.

Right?

What happens if we have this big part of the company and then, you know,

we have a big reduction or something, we lay off a bunch of people, we want

to, you know, we want to whatever, whatever, whatever things stuck in your

mind and you want to then go in and, and prematurely delete those backups.

And so you, so you choose the less secure option.

It's human nature to choose that.

And I'm saying if you're able to do it, that means a threat actor can

possibly do it if they're able to get in with, you know, a stolen credential

and, and MFA fatigue, or they're able to exploit a, you know, some sort of

system weakness so that boom, they are now you in the backup system,

they can do whatever it is you can do.

Um, then I, I, I, I don't like that idea.

Yeah, a hundred

Yeah.

Yeah.

So that's the 3, 2, 1, 1, and then the zero is just zero.

Um, uh, failures.

Right?

Which sounds nice.

Um, this is, you know, I.

you say failures, what are you referring to?

Because it could be backup failures, it could be storage failures, it could be

Yeah.

Well, it's mainly what, what, when, when Rick talks about it, he's talking

about zero zero backup failures, right?

The thing is backup failures happen.

Generally speaking, they happen due to process failures today.

It used to be that backups would fail on a very regular basis due to that

wonderful thing we call tape, right?

And again, there was nothing wrong with tape, it was just how we were using tape.

Uh, and there was a fundamental mismatch of technology.

But, uh, we've generally solved a lot of these issues

and so.

Backups, generally speaking work, you know and and are successful most of the time.

You do need to make, as part of your system, you need to need to make sure

that you are regularly monitoring the success of your backups and when they

don't succeed, you do something about it.

Can I add a.

Nope.

Sorry, that's the end of the podcast.

Sorry.

No, I, I think I was, as you were talking, I was thinking about sort of

ransomware and other things like that.

You know what?

3 2 1 1 0 does not really capture

What?

the fact that there is a clean backup without any ransomware

that exists in your system.

Yeah.

You know, that's a good Yeah, that, that's a good one.

I, I like that.

Um, and I, and I don't have an issue with that.

Actually, it's a good point, and that may be something that, that Rick

talks about in his, um, I guess maybe I didn't think about it, but that's

actually a really good point, right?

Is that you should be scanning your backups for ransomware.

The, the, the hard part is that, um.

A lot of times the backup itself doesn't have ransomware in it.

What it does have is encrypted files in it.

Um, 'cause the ransomware is the actual code that's making this happen.

And then the encrypted files are, are, are worthless.

Right.

Um, I will say that when we, you know, when we wrote the book that's

coming out in January, uh, learning ransomware and recco, uh, learning

ransomware response and recovery, um.

This was something we debated quite a bit about, right?

When you go to do a restore, how do you verify that the thing that

you're restoring to is clean?

How do you verify the thing that you're restoring is clean?

And the answer is, it's actually a really hard answer, right?

Um, you do it as best as you can upfront.

I like systems that check the backups when they're doing them.

I like systems that check the backups as they're doing restores.

Um, and then, uh, I think you should be checking, you hopefully.

In the initial phase, uh, which we, you know, we covered all of the

different phases in the book on, on.

On actually responding to an attack, definitely early on you

should have figured out what it was that attacked you, right?

You should know what it is you're looking for.

You should know the actual ransomware variant that you have, and so

you should be able to relatively, easily, easily look for that in

anything that you're restoring,

Mm-hmm.

Um, but yeah, so, so to go back to the topic at hand, 3, 2, 1, 1 0, uh, it's

just again, 3, 2, 1 is great, uh, but it's time for it to grow up a little bit.

Uh, if you don't have 3, 2, 1, then you don't have a backup.

But if you don't have 3, 2, 1, 1 0, then you, um.

You, you don't have a backup that's gonna be helpful in the time of a

ransomware response, which is most likely gonna be the number one reason.

Why do, why do, why did it become the number one reason besides the

fact that ransomware took off?

Why did, why did this become the number one reason people restore?

What do you think?

I have an opinion.

well, two things.

One is ransomware actors, right?

They're gonna ask for ransom,

Hmm.

and so what's one good way not to pay the ransom is to do the recovery yourself.

And so that's why you have your backups.

Yeah.

why you need that immutable copy.

So then the second part is ransomware actors got smart and then they

started targeting backup systems

Yeah.

if you can't fix stuff, then there's more likely to pay them.

And so that's why it sort of has come under attack.

Yeah, that, that was all very valid, wasn't what I was going for.

Uh,

it, it, it's just, it is just me sort of waxing philosophical,

Yeah.

Yeah.

the reason why ransomware became.

Like the number one reason is that we fixed the old number

one reason as I was growing up.

Technically, the, the, the number one reason we were restoring

was technical failure, right?

Hard drives were failing, servers were failing, and we, we

addressed that right hardware.

And storage has become so much more resilient that we are

almost never restoring because the hardware itself failed.

We are restoring you due to one of two reasons.

One is somebody did something stupid.

Well, actually one reason somebody did something stupid, right?

Either they deleted something they shouldn't have deleted, or

they clicked on something they, they shouldn't have clicked on

or did something else that then resulted in a ransomware attack.

Right?

Yeah.

And so the only reason we restore anything these days is stupid people.

Yeah.

No, I, I wonder like, as we're talking about the 3 2 1 1 0,

yeah.

and maybe Rick has stats on this, I wonder how many organizations are

actually following 3, 2, 1, 1, 0.

Not enough.

It's a great question.

You just look at all of the ransomware stories, and we have many of

them as case studies in the book.

So many of them are, and then the backups were then.

Deleted or encrypted or whatever, right?

Locked outta the backup system.

Uh, not enough is the answer to that question.

Um, hopefully that will change over time, but, uh, hope maybe the book, maybe the

book will, you know, uh, solved this.

But yeah.

All right, well, there you go.

3, 2, 1 rule is dead long live.

3, 2 1, 1 0.

Does that mean I need to update the tattoo on my arm?

Um, yeah, you're, you're just, you.

I never know what's gonna come outta your mouth, I tell you.

All right.

Well thanks.

Thanks for chatting, Prasanna.

I will be making my appointment with the tattoo artist as

soon as we're off this call.

Absolutely.

And thanks to all of you, your 3, 2, 1 tattoos.

Uh, by the way, if anybody's got any weird backup tattoos, I'd love to see that.

That is a wrap.