W. Curtis Preston: Immutable backups are a critical defense against the rising

tide of cyber threats, and at a time when ransomware attacks are targeting not just

your primary data, but your backups too.

Understanding immutability is more critical than ever.

We'll explore what makes a backup truly immutable.

The spectrum of immutability options available and why.

This matters so much in your data protection strategy.

From tape backups to cutting edge cloud solutions, we'll cover it all.

Are you immutable?

Backups actually immutable.

Let's find out.

I.

By the way, if you don't know who I am, I'm w Curtis Preston, also known as Mr.

Backup.

I've been passionate about this topic for over 30 years, ever since.

I had to tell my boss that there were no backups of the

database that we had just lost.

I don't want that to ever happen to me again.

I don't want it to happen to you, and that's why we do this.

On this podcast, we turn unappreciated backup admins into Cyber recovery Heroes.

This is the backup wrap up.

Welcome to the show.

I'm your host, w Curtis Preston, AKA, Mr.

Backup, and I have with me my SMB backup Difficulty.

Sympathizer Prasanna Malaiyandi.

How's it going?

Persona.

I, I've learned more about dis space backups than, and tape

based backups in the last, like three weeks from you than any time before.

So yeah, how are you doing?

W. Curtis Preston: oh, I just like, if I could get like a, like if I

could have a wall here, just, you know, bang my head into the wall.

That's sort of how I feel like I'm doing.

I do

the problem is that you left it for 20 years

and now you're coming back to it.

And so

W. Curtis Preston: That's not the problem that that implies that I am old

or incompetent or something like that.

That is not the problem.

me going back.

It's like me going back and coding.

I could do it, but I'm not gonna be efficient.

W. Curtis Preston: The, the, the, um.

It, it, it's been, it's been really weird, right?

Because, you know, you change something and you change this.

You're like, well, let's try this.

And then you try that and you're like, that just doesn't work.

That just, you know, like the, the, the, and the really weird thing that I.

Has been pla plaguing me over the last, like, I don't know, several days as

I've been trying to make a major change to this backup configuration is that

the backup starts out, you know, I'm running like 20 backups at the same time.

Backups start out and it, and it's running like 500 megabytes per second.

This is great.

This is what I want.

And then about a half hour later he goes,

you're like.

To a slow drip.

W. Curtis Preston: happened?

This is not, this is not tape, this is, you know, this is not

like, there's not some weird tape, you know, shoe shining thing.

This is disc two, disc via a server in the middle.

It is probably some random option somewhere

W. Curtis Preston: random,

haven't done this.

W. Curtis Preston: data buffers thing.

Oh, oh, oh.

You're, you wanna do this on a Thursday, you need to do the, you

need to do the setting and, uh.

what I said, that because you haven't done this

day in to day out, you don't know.

It's like you lose touch of these or you don't know

W. Curtis Preston: Yeah.

And I, I will say that when I was, when I was, when I was last doing this really in

earnest, this backups were relatively new.

Um, that, you know, that really dates me, but that is, that is the case, but.

So, yeah.

And I just, you know, and I, I've been thankful for you to talk to you.

It's like, you know, so here's the thing, you know, I got this, I got

this setting, I got that setting.

I've been playing with jumbo frames, I've been playing with,

with, uh, buffer settings.

I've been playing with all these different things.

And you know, and you're like, what about the thing?

You know, what about this setting?

Oh, man.

And you know what?

In the end, once you figure out what's going on, it's probably

just gonna be some simple thing,

W. Curtis Preston: Mm-hmm.

right?

That you just didn't know existed.

And you're gonna fix it and be like, oh yeah, now it's like, boom.

W. Curtis Preston: Yeah.

And then it's gonna run like a thousand megabytes per second, and then,

we'll, everything will be amazing.

Um, so, we're gonna talk about some things that are really new in terms of when, when

I first started with backup, we really didn't talk about this concept when I, you

know, was doing backup back in the day.

There's a big thing that happened in the last.

Two decades.

That is one of the biggest reasons why we didn't use to talk about

cybersecurity and backups, and now we talk about it all the time.

Do you know what that is?

Persona?

No,

W. Curtis Preston: It's

one change.

No.

One seismic change in backups in the last 20 years.

Dis space backups.

W. Curtis Preston: Yes, exactly right.

Tapes weren't hackable.

Tapes were offline by design.

Right?

Ooh, that rhymes anytime.

Um, yeah.

So tapes were, tapes were not sitting there just files ready

to be deleted or encrypted.

Tapes were inherently hard, even if they were online, even if, if they were in a

tape library, they were hard to get to.

And then.

Uh, over, you know, and, and if they were offline, if they were sitting,

you know, in a, a tape vaulting vendor, then they were really hard to get to.

But the, um, what happened when we went to dis backups, we

basically made backups hackable.

And so that, and, and I don't think, in fact, I know, I know.

We really didn't take that through back when we were.

Unintended.

W. Curtis Preston: Yeah, we were just thinking about all the problems

that we were having with tape, and we wanted to fix those problems.

And so we went with this space backups to fix those problems.

And, uh, and remember, I will, we'll, we'll always say this.

It wasn't, the tape was too slow.

It was, the tape was too fast.

And we can, you know, if you don't know what I'm talking about, then you need

to, you need to listen to more episodes.

And we've, we've come a long way, but for a really long time backwards, were

just sitting there as files either in a, in a regular, uh, you know,

direct attached file system or an NFS or SMB attached file system that were

directly deletable by a privileged user.

Uh, as long as you, you know, as long as you had access to that drive, we've.

Hopefully in most cases we've gotten away from that.

I think still many people go and buy a, you know, pick your favorite backup,

you know, disc vendor, and they just mount it as a backup drive and then

start backing up to it, and it's, it's, it's eco back slash backups.

And which just like screams, come delete me.

And you know, I, I, I know people still do that and they shouldn't do that.

They should ask their backup vendor, Hey, is there a way that I can

transfer data to my dis space backup storage without it showing up as a,

as a dry letter on my uh, system or a mounted drive on my Unix system?

But that is a huge change in security of backups.

Yeah.

No, I agree.

It made life easier, right?

You could now do all the instant access, quickly, restore your data, do all the

test in dev, but there was a fatal flaw.

W. Curtis Preston: Yeah, it made life easier for everybody.

Yeah.

W. Curtis Preston: Yeah, absolutely.

And then replicated backups, right?

We had replicated backups, and then if you, you know, if you have a, a,

you know, a screw up in your backup system, it, the replication just makes

the screw up, uh, more effective.

Right.

Yeah.

W. Curtis Preston: Yeah, so this is why we have to talk about immutability.

Um, but before we talk about that, we'll talk about some things

that haven't changed at all.

So the thing I want to talk about is this concept of immutable

backups, and that really wasn't.

Something we talked about back in the day.

Well, I think a lot of it back in the day was your backups

were kind of already immutable, right?

If you were,

W. Curtis Preston: the,

were going off tape.

W. Curtis Preston: Yeah.

Right then you didn't have to worry about the scenarios

that led you to need immutable backups.

W. Curtis Preston: Yeah.

That, that's really true.

And that, and that's one of the things that we're gonna talk about, right?

I, I would say the biggest reason was that no one was attacking the backups.

No one was trying to attack the backups.

And the, the worst things that we were worried about were things like I.

Um, we were worried about a natural disaster or a terrorist

attack, taking out the entire place where backups were stored.

So we needed to make sure to have a, an air gap, a physical separation between

the, the backups and, uh, you know, and the, the copy of the backups, right?

We weren't.

on a tropical island.

W. Curtis Preston: Exactly right.

Exactly.

And so we weren't really, we, we, we weren't at all.

I, I could just, just absolutely tell you that we were not at all

ever thinking about a cyber attack.

Trying to, I, I don't even think we had that term.

I mean, we had the internet, but we didn't really have the, the term cyber attack.

Um, at least that I remember.

We certainly never used it in backup circles,

Or it was different, right?

Because you weren't necessarily worried about people stealing

your data or things like that.

I remember you were the one who recommended me to read

the Cuckoo's test, right?

And so those were the types of attacks that you sort of

saw back in those days, right?

Stealing resources, not necessarily stealing your data

W. Curtis Preston: Yeah, cuckoo egg, by the way.

Or sorry, cuckoo.

W. Curtis Preston: which by the way, if, if, if you're out there and you don't

know what we're talking about, great book.

Um, uh, and it's a relatively short read.

It's a great book to read, and, and it's a true story that, you know, involves

a, a cyber attack against a university.

It's great.

So before we talk about immutable backups, I want to talk about some

of the things that really haven't changed that are also, um, impediments

to having a safe backup system.

And the first is the fact that backups never were, never have been, and never

will be the sexy part of the data center.

Yeah, no, I a hundred percent agree.

No one wants to do backups.

No one wants to spend the money on backups, right?

Everyone wants to be like, Ooh, I need to do, given the recent trend

these days, I wanna focus on AI and do this and that and all the rest.

But no one wants to focus on how do I protect it other than people like Curtis.

W. Curtis Preston: Other than a few, few weird people.

I would say that it's a, there were a lot of reasons.

The biggest reason is easily that it is a very difficult job.

We, we started out this, this episode talking about how

sometimes, how difficult it can be.

You are invisible or you're in trouble.

No one, no one remembers the millions of backups that you got, right?

They only remember the one restore that you got wrong.

And so nobody wants to have that pressure on themselves.

And so they, nobody wants to raise their hand in, you know, in the,

uh, in the room to say, Hey, I'll, I'll be in charge of backups.

So this is true in terms of.

Um, it's not the sexy part of the data center, which means nobody

wants to do it, number th number one.

Number two, nobody wants to give that part of the data center any money

number, and then number three I'd say is that as a result, it goes ignored.

Prasanna Malaiyandi: Oh, I agree with that.

I think there was a study, I can't remember when it was that.

What percentage of an IT budget do you think goes towards backup?

W. Curtis Preston: It's a very small portion.

I think it was less than a percent.

W. Curtis Preston: Yeah.

Yeah.

That sounds about right.

Yeah.

So it, it goes, it goes ignored.

It goes ignored.

From a budget perspective, it goes ignored from.

A cybersecurity perspective, which is really what we're talking about.

It's what we've been talking about the last several episodes is, is

cybersecurity and, and backups.

And it, it means that it doesn't get put front of the line when we start

talking about putting out patches.

It doesn't get put front of the line in terms of looking

for trends that could be scary.

Um, it's just, it literally just goes ignored.

Um, and that, and that's a really scary thing.

I.

Yeah, and it's not even thought of initially when you're,

I know we've talked on the episodes that when you are deploying a new application,

you should be thinking about backup or at least have the backup person in the room.

But sometimes things develop so quickly that.

Backup is an afterthought, right?

And sometimes it's very painful.

W. Curtis Preston: Yeah, very much so.

I can remember, I can remember one example of that when we bought, uh, the comp, the

bank that I worked at bought a, an HP.

Was it 1000 I think was the HPT 1000.

That's what it was.

It was a T 1000 and the, it came in and it had, um, it was huge.

It was, are you ready?

A hundred gigabytes of data.

It was huge.

Humongous.

W. Curtis Preston: It was humongous.

Now your phone has more than that.

W. Curtis Preston: Prior to this server or prior to this server showing

up at our data center, the biggest server we had was like four, and then

suddenly we had a single server that was 25 times bigger than the next

biggest server, and it came with.

A DDS drive, which was a, a, um, four gigabyte drive.

And, and I, I just remember asking like, who, who's gonna stand here

all night long and swap tapes 25 times to get to get it back up?

And that was, and, and nobody had knew, nobody had even thought of that.

That's the point that nobody had even thought, well, maybe we should

have enough backup infrastructure to back up this new gigantic server.

And that was, by the way, how I got the, my first justification to

buy my first, uh, tape robot, which was the first time I sent money

to Spectral Logic back in the day.

You weren't just sitting there all night swapping tapes.

Curtis.

W. Curtis Preston: definitely not gonna do that.

Definitely not.

Um, so yeah, so that, that problem's not a new problem.

Uh, the other thing, and as a result of all of those things that we just

talked about, a very common thing for.

Computing infrastructure or for computing environments to do is

to give backups to the new person,

right?

Uh, it was the first job I ever got.

I, I was just editing the, the podcast.

So with Dwayne Lalo, he said it was the first job he ever got.

And the problem with that really is you have this really, really important,

really secure, or what should be a really secure job, and you're handing

it to the least experienced member of the, of the, you know, the team.

Right.

I, I mean, I, I had been at the bank,

you know, essentially a week.

I, I had been, I had been.

In computers, like, like commercial computers.

And I had been introduced to Unix for a few months.

I, I, I had been at this company, I, I think I had worked at this

company for like two months, uh, prior to going to the bank.

And I had been, you know, I had been given some lessons in Unix

and I had been demonstrated the importance of a really good backup.

Uh, here's, this was the first time in my career when.

Prasanna Malaiyandi: Theoretical versus practical.

W. Curtis Preston: Yeah, so we were a, uh, we were a blended call management

system, which you might refer to as an auto dialer, and that was a, that was a,

a naughty word back in the day, right?

We were not an auto dialer.

We were an intelligent call management system.

And, but it was essentially an Auto Tyler and it had this very fancy

Unix-based computer behind it.

And we would go out and do this, this, it was like this long involved install

because it was a very, this was before VoIP and all that kind of stuff.

We had, you know, we had literally cards that went in the server and those, each

card connected to a POTS line, right?

Mm-Hmm.

W. Curtis Preston: And so we had this big physical install that we had

to do, and it took about two weeks.

And then, um, you know, you, you sent a couple of SCS on site and I was one

of the sc I was obviously the junior se, and this other guy had been, um,

doing all this configuration and.

The job that we were supposed to do before we left was to pop in a tape,

by the way, a tape, a nine track

Ginormous.

Yeah.

W. Curtis Preston: and pop in a nine track tape and back up the OS

configuration that we had just finished.

That was the last thing that we were supposed to do, and just

before doing that, my, my compadre.

Said he was looking at the root file system and he noticed that there were

a couple of garbage files in the root file system and one of them started

with, um, question mark and had a bunch of gar dego stuff after it.

So he didn't want to type all of the gar dego, so he typed, um,

RM minus RF question mark, star

Oh,

W. Curtis Preston: from the root file system.

Wanting to clean up the OS before we took the backup.

oh geez.

That must have been interesting.

W. Curtis Preston: Yeah, so that was the, that was the day that I

learned the value of a good backup.

The best time to do a backup is yesterday.

The second best time is today.

today.

W. Curtis Preston: Um, this was definitely the second best time,

so we lost a whole bunch of work.

Uh, and luckily it was not my fault, so I just learned a very valuable lesson.

Anyway, sorry, that was bringing me, bringing me back, back in the day.

Um, so yeah, so.

It's a horrible practice that we put the most junior person in charge of the

backups and that should absolutely stop, but I don't know how to merge those two.

How do you, how do you, how do you properly reward this job that

is really, really important such that someone who isn't junior is

going to want to take the job?

I don't know.

I don't have an answer to that question.

Do you, do you persona.

Well, no, but it's No, but if you think

about this, this really isn't any different than say A DBA, right?

A database admin who starts off initially does some Basic things, right?

And over time they sort of become like that database architect who sort

of figuring out overall layouts and things like that rather than sort

of just administering the system.

Now the difference is in the case of the DBA, you're probably starting

off in like test and dev instances or like non-critical applications versus

as a backup admin, you're probably starting off on like the critical

infrastructure that you are protecting.

W. Curtis Preston: Yeah.

Yeah.

That's why I'm, I, I'm not sure I agree with your, with your comparison

there because, well, first off, people actually wanna be DBAs.

Yeah.

Well, and yeah, maybe.

Maybe what you need to do is like a cross training program where you do

a rotation in like a different part of the organization and then you come

into the backup work, because that way at least you're also get more

familiar with tools and other things.

W. Curtis Preston: Right, right.

Yeah.

Yeah.

That way you're not the junior person.

W. Curtis Preston: exactly, exactly.

Yeah.

You, you need, I, I guess the best thing we want to do here is to make sure.

That you're, that you're making sure that your junior person, that you're giving

the, the backups to has some really good cybersecurity, some basic, at least some

basic cybersecurity training, right?

Because you're handing them in order to do the backups.

They are a very powerful person and they have the ability to

read and write every file.

Right, which means they have the ability to overwrite every

file in the organization.

Uh, even if you're using proper modern backup design that uses role-based

administration and all of that, you're, you're still most likely

handing them the keys to the kingdom.

Uh,

Is that true though?

Because normally don't, you have sort of help desk restore operators and

then the more senior people are kind of those who are doing like the backup

architecture and the design, you probably have people who are figuring out

W. Curtis Preston: In a large environment, that would be the case, but that's not

the case in most companies, I don't think.

I think it's just one person, actually, in most companies it's a collateral duty.

Right.

So, but let's, so let's get to the meat of the episode here where we

wanna talk about immutable backups.

So first off, just generally speaking, what does immutable mean?

It basically means you can't change it,

W. Curtis Preston: Yeah.

It's literally, it's, it's, it's, it's a word that's used outside

of it, but within it, yeah.

It means that it cannot be changed

which also includes deletions.

W. Curtis Preston: yeah.

Changed.

Yeah.

Deletion is a change.

Right.

Um, and the difficulty here.

Is that, that sounds like a binary condition,

Mm-Hmm.

W. Curtis Preston: Like pregnant or dead.

Right.

Not the words I would've chosen, but Sure.

W. Curtis Preston: Both pregnancy and death are binary conditions.

You're either pregnant or you're not, or you're dead or alive, right?

These are binary conditions.

Um, immutable should be a binary condition that that is what is implied.

When we say immutable, it can't be changed.

It can't be deleted.

It can't be encrypted.

It can't be modified.

It can't be, I don't know.

I think I covered 'em all.

Yeah, but there's really a spectrum though.

W. Curtis Preston: it is definitely a spectrum, and that is the

problem, is that immutable.

And that's, that's the thing I wanted to just mention.

So let me, so let me ask you this.

Why, why has immutable backups, why has that really come to the fore,

uh, in the last say, 10 years or so?

Well, like you were talking about earlier

in the episode, cyber attacks.

Ransomware attacks, they not only go after your production, but they

realize that, hey, you have all this data sitting in your backup systems

that you could use to recover your environment and not pay the ransom.

So instead, we're gonna go attack your backup systems, delete all

your backups so you can't recover.

And then also because they're doing the double extortion where

they also exfiltrate your data.

That's a treasure trove, right?

They don't have to go hit 20 different systems.

If they get to your backup system.

All your data's there and backup systems are really good at restoring data so

they can quickly pull the data out and then blackmail you in case you decide

that you don't want to pay the ransom.

W. Curtis Preston: Yeah, exactly.

Now immutability won't address that second part.

Uh, it is important to me.

Right tho those are, those are the two main concerns that we have when

we talk about the backup system.

And, uh, you should definitely listen to the episode that went live.

Um, let's see, that would've been May 26th now, um, the one where with Dwayne Lalo,

where he talks a bit from the Red Team, the Red Team Cybersecurity episode that.

Um, where he talks about what a treasure trove the backup system is and how he

loves, he's a, you know, he is a red team.

He's a hacker, uh, that, you know, an offensive cybersecurity person.

Uh, he's very offensive.

And, and he loves going after the backup system, right?

When we talk about immutability, we're primarily concerned with

making sure that the backups themselves aren't modified, right.

And which, you know, as we said, you know, and, and.

Like you said, this wasn't a problem.

I don't know, even five years ago, maybe, uh, this wasn't a problem,

but hackers weren't going directly after the backups, but now they

definitely are because they're the best defense we have against ransomware.

Right.

I.

Yeah.

And and I would say before immutable backups slash immutable archiving was

really intended as a compliance use case.

W. Curtis Preston: Yeah.

And that's what I, that's kind of where, where I wanted to go

with this is let's talk about the different things that people mean

when they say immutable, right?

And prior to cyber security attacks, when we talked about the backup system

or the archive system, which are different things when we talk about the

backup system or the archive system.

Being immutable, what we're primarily, um, wanting to be able to say in a

court of law that this file, this document that I'm presenting in court,

is exactly the same as it was I.

When it was created, right.

This, this email that is the smoking gun we can state.

It is a, it is a faithful reproduction of the original email that the plaintiff

made or the defendant, I get It's, it's a's a faithful reproduction of

the email that the defendant made.

That shows that they were doing the thing that they weren't supposed to be doing.

Right?

And, and so the, so that the, the defense can't say, oh, well

that's a fake email, right?

You're like, no, this is a faithful reproduction of the email that this,

you know, and here's the, here's the chain of custodying, the chain

of technology to get from A to B.

And so, um, you know, it, it's, fine,

It's as good as the source.

It's as good as the original.

W. Curtis Preston: It's as good.

Yeah, we are.

We are essentially looking directly into the email system from three months

ago or three years ago or whatever.

It's right.

That's what we were primarily concerned with and we are still concerned with that.

Right.

We are still concerned with immutability for that reason from

a legal perspective, but, um.

What we're now more concerned with is that someone could go in and modify the backups

in such a way that they encrypt them.

They, you know, they, they delete them.

Uh, they could actually modify them.

I, I don't, I don't, I don't know of any scenario in this case, but

they could potentially go in and selectively modify them to mess

with people for whatever reason.

Um, but that shouldn't be able to be the case either.

Right?

Yeah, and that just seems like a lot of work to go selectively

modify unless you're trying to do something sneakily without being detected.

W. Curtis Preston: Right, right.

Um, so that's the, that's one way that backup systems and archive systems use

the term immutability is they say, I.

We can say that, you know, these two things were, you know, are the

same.

So the, the next thing that I see is that.

I see vendors that say, well, we have a A, right?

Only we have an append only file system that once backups are

written into our append only file system, they cannot be modified.

Have you seen backup folks saying that I.

Yeah, though I've seen some storage vendors, right?

You have companies who do backup appliances like Rubrik who say,

yeah, once you write your backup to our system, right, you can't

go and overwrite that data again.

W. Curtis Preston: Right,

of a right only and.

Having worked at a storage company which was famous for a write

Anywhere file layout, I could see the benefits of doing it that way.

W. Curtis Preston: Right.

Yeah.

And, and what this says is that it stops direct attacks against

the backup storage itself.

That if I, if I'm able to get into the file system, if I, if I'm able

to get into the operating system of that backup system, I wouldn't

be able to modify or delete the backup files in order to, you know.

Well, well essentially delete them.

I wouldn't be able to encrypt or delete them because this is

this append only file system.

Which is interesting because I do wonder

about certain scenarios like, Hey, I'm running low on space.

I need to go delete a whole bunch of backups or backup sets

in order to free things up.

How?

What do I do?

Am I just stuck?

W. Curtis Preston: Yeah, that's gonna, that's gonna be

depending on the product, right?

So the question, my, my question is.

With the, the product in question and that this isn't necessarily Rubrik,

this is just, I'm, it's just the, this is the question that we're asking

is, I, I understand you're saying that it's an append only file system.

What about the system that's controlling the backups itself?

What protection is there in, in that system?

Meaning if I am a privileged user.

Am I able to, um.

Delete

backups, right?

Am I able, am I able to prematurely expire backups?

Because even with immutable storage, they can have, it can

have an expiration date, right?

So you can say this, this, these backups are immutable for 90

days and then they can expire.

But if I am, you know, the super high cloud administrator of this backup system,

am I able to then expire backups early?

And if you are, then.

That is, again, this is why I say immutable backups.

Are a spectrum because the, the, the question is, if I am, you know,

super high up in the echelon, if I make a phone call to this person, is

there a way to expire older backups?

I, I would

also ask, is there a privilege to count in the operating system that allows

me to essentially expire all backups?

Yeah.

W. Curtis Preston: that, that, that is the case in some backup systems, that even

though they're immutable, if I have access to this one privileged account, I can wipe

out all backups with a single command.

Or reset the system back to a factory state, right?

That's sometimes common.

W. Curtis Preston: Right.

Right.

Um, so that, that is another one that would be called immutable.

Right.

Um, and, and I'm not saying that any of these aren't immutable, I'm just

saying that there are a whole bunch of things that are called immutable.

right.

right.

Yeah.

W. Curtis Preston: Um, another one, and, uh, and this is

where, when, and, and again.

This is like good, better, best, right?

Anything would be better than backups.

Sitting on just an open Windows file system would zero security checks, right?

Anything would be better than that,

and,

I think I have a worse scenario.

W. Curtis Preston: okay.

A window system open to the internet.

W. Curtis Preston: Yeah.

Yeah.

With, uh, with, uh,

RDP

W. Curtis Preston: Um, R-D-P-R-D-P enabled.

Yeah.

Uh, the, um, the next, you know, one of the most surprising things when,

when I listened to the episode with Dwayne again was when he said that

so many times that they, they, they go in with the default password.

And that it works.

That's just, that just hurts.

The next one is really about something.

So not the best, right?

But better than the Windows system, right?

Which is good.

Which is having a server that has been hardened and locked down and secured.

Such that you are protecting from most common issues and

security vulnerabilities, but it may not still be perfect.

And in this example, right, we had Dave Russell and Rick Manover

from Veeam over talking about the hardened backup repository.

I think is the official name, which is basically a server which they

locked down, which allows you to write Veeam backups to it, and they have an

immutable flag in the Linux repository such that the backups cannot just be

normally deleted, casually deleted.

I would say.

W. Curtis Preston: Right.

Yeah.

They can't be deleted by anyone except for root, right, and they can't even

be accidentally deleted by root.

The root would have a person with root would have to unset

the immutability flag on the file and then delete the file, right?

So.

Uh, this is what, you know, I agree with you that it, it is better than,

you know, having a Windows file system.

I mean, first off, it's just better in that it's Linux and not Windows.

And I don't, I don't just mean that Linux is better than Windows.

What I mean is it's something different, right?

And then the, uh, 'cause I definitely do not think that Linux is.

Perfect.

Right.

But the, uh, I do still, like the Linux way of everything is off

by default versus the Windows way where everything is on by default.

Although I think they've gotten better at that over the years.

The um, um.

It is important to understand that if someone gains root, then they're

able to, uh, modify these files.

Now they do a really good job.

I think, uh, specifically with the Veeam repository, they do a really good job of.

Giving you a list of things that you need to do where it's like essentially.

But once you've configured this system, you can configure it, for example, to

automatically update itself with security patches and that you don't need to be

logging into it regularly as route.

So you can like super secure that, um, thing.

So that if, if anyone ever does log into it as rude, it can set off all

sorts of, uh, bells and whistles so that you can know that something.

bad is going on.

Yeah, and I know we've talked in the past about

like patch management, go do your patch management all the time.

And having an automated process and keeping it up to date I think

is a great feature that that has.

W. Curtis Preston: Right.

And then I would say, what's the, the next, and I'm, I'm gonna

put this up in terms of the best as far as I can see, is that I.

Here's an immutable system where backups cannot be modified even by a privileged

account, even by somebody with root.

Um, the, and there would be two types of systems like that.

One would be a backup, a, a storage system that could be on-prem.

Another would be something like a cloud system like S3 with the

immutable flag turned onto that.

And I would, I would actually put.

Them in terms of like the, the on-prem system isn't as secure as

the cloud system, in my opinion.

Why?

And why would that be?

because you can physically access the system and

yank the drives and all the rest.

W. Curtis Preston: Yeah.

Physical access trumps, all right.

So it's funny, a lot of people are like, oh, the cloud is insecure.

And I'm like, well, physical system's on-prem or insecure.

Right, because I can easily go up to that and I can do bad things to

it as long as I know where it's at.

Yeah.

And even with the physical storage systems, I know at NetApp and even

with AWS's Object Lock, right?

They do have those two different modes, compliance and governance.

I.

For their immutability options.

One which sort of allows an admin to reset

W. Curtis Preston: Mm-Hmm.

the flag, right?

Like we had talked about earlier.

So it's good enough, but there are cases where, yeah, I need to unset it.

So they do give you that option.

And then the other one is you can't touch it unless you basically, in the case

of AWS, blow away your account, right?

So it's really nothing you could do as an admin will help you.

W. Curtis Preston: Yeah.

And you know, just realize that a, what, what I would

call a truly immutable system.

If you change your mind, you can't change your mind, right?

If it's, if it's truly immutable, you should not be able to, to blow it away.

Now, I, I still haven't done this experiment, and I, I, I really am gonna

do this at one point where I create an AWS account, turn on immutability, put

a bunch of stuff in there, and then delete the account and see what happens.

Um, I don't, I don't know what happens there, and then stop paying my bill.

No, no immutability system.

Outlives not paying for your, your, uh, your storage

Yeah, they will probably reclaim that

data as quickly as they can.

Curtis.

W. Curtis Preston: do garbage collection.

Goodbye Curtis's data.

The, I I mean, I will say that in that case it's kind of your own fault, but, um,

so this is what we mean when we say that.

Immutable backups.

You wanna make sure that a black hat, a bad actor that gets into your environment,

cannot do damage to the backup system.

And so you want to separate.

Backups as much as you can from the, the, the bad actor.

And the best way to do that is to store them in an immutable system.

Just realize that there is a, um, a spectrum of immutability and

all of them, they all say their backups are immutable, right?

I think it's also important to note that you may not need

the best solution for your use case and your needs, so also make sure you

understand what you're protecting against.

There is gonna be cost, there is gonna be complexity potentially

with those solutions, and maybe it isn't needed to go to that extent.

And maybe like better is good enough.

W. Curtis Preston: Yeah.

Never let, never let perfect be the enemy of better.

Right?

So, you know, you, you could be better.

Um, and that's okay.

Always strive to be better, but don't.

Don't let say, well, th you know, th this immutable system

doesn't work in this one scenario.

Don't let that prevent you from doing something.

Something is always better than nothing, right?

One thing I do want to mention is that MFA is a big part of this.

Some of these systems that have it's immutable except for the super powerful

person, or it's immutable unless you make these 17 phone calls to.

Um, you know, you, you've gotta really, really authenticate yourself.

I have seen scenarios where companies have had their email system hacked

in addition to their backup system being hacked, and the result was that

they were able to defeat MFA because they were using email as their MFA.

Or the other thing is also phone because now they

can impersonate someone, call in, they have all the access to the emails

and can spoof like two factor codes.

And talk to the person on the other end, and they don't know the difference.

W. Curtis Preston: Yeah, the, the, the, um, the ability of AI systems these

days to mimic the sound of a, of another person's voice, uh, in, in fact, uh,

just as a demonstration, uh, the next.

30 seconds of audio is completely fake.

How are you doing, Prasanna?

Prasanna Malaiyandi: I'm doing fine, Curtis.

What are we talking about today?

W. Curtis Preston: I think we're talking about immutable backups, and

I hear that they are on a Spectrum.

You mean like IBM Spectrum Protect?

W. Curtis Preston: No, that's a backup product.

You mean the Irvine Spectrum?

W. Curtis Preston: No, that's a mall.

I mean that there are many things that are called immutable, and

they all have different levels of protection against attacks.

Nice talking with you, fake Prasanna.

You too, fake Curtis.

W. Curtis Preston: So that was neither me nor persona talking.

For the record, that was a little tool that we use.

That is very useful in the right scenario, but very dangerous in the wrong scenario.

Um, yeah, you, you really have to think about that, right?

When you, when you put all these layers, if you have all these layers of

security between you and the ability to delete your quote immutable data, think

about all of the modern things that people are able to do, um, before, um.

You know, you subtle with one of those systems.

And again, don't let perfect be the enemy of the good.

But, but, but just think about all the things that are possible these days.

Uh, you know, they can fake video, they can fake audio.

Uh, they can certainly fake email and phones, right?

SMS is a horrible way to do things, right.

But, um, um.

With that, that's our discussion on immutable backups.

I hope you enjoyed a persona.

No, this was a good topic.

I like immutable backups.

W. Curtis Preston: Yeah.

Immutable backups.

Good.

And, uh, hope, uh, you enjoyed that as a listener.

Hope you learned a thing or two.

And with that, that's a wrap.