On this episode of restored all we're talking about how to secure your backup

data, to make sure that it doesn't get attacked in a ransomware attack.

Hope you enjoy.

The episode.

Hi, and welcome to Backup Central's Restore it All podcast.

I'm your host, W.

Curtis Preston, a k a, Mr.

Backup, and I with me, a guy whose only major flaw is that he

won't go see scary movies with me.

Prasanna Malaiyandi.

How's it going?

Prasanna.

I am good, Curtis.

I, yeah, I'm not a big fan of scary movies, especially on a large screen.

Um, I don't like, I know some people like you love it, but it's just I don't

find joy in watching a scary movie.

I do remember in college watching the Ring.

On a big screen.

That movie messed with my head.

I remember coming home and asking someone to take my picture.

And saying, is there a thing in the picture?

You know, I was scared.

I was scared to take the picture.

Yeah.

That movie, that was, that was a fun movie.

Um, it really kind of messed with your head.

And you just saw a scary movie just recently.

Yeah.

It, it,

Well, I wouldn't know if I'd called

movie per se, it's more like a thriller.

Um, It's called the Menu and it stars.

Uh, Ralph finds, um, a k a Voldemort, and uh, Anya, the girl from the Queens gambit.

Yeah.

Um, and some other, other character actors that you may be familiar with.

Um, and, uh, it, it's, it's a movie that I, I enjoyed the movie.

I will just say this, um, trigger alert for those who are bothered by such things.

It does contain suicide.

Um, having said that, um, within the context of the show, it, it was a

significant part of the storyline.

Mm-hmm.

Um, and um, but having said that, it

was a really unique storyline.

That's a commentary on Modern Day Society.

You know, there's a class thing, there's a, there's a, yeah.

It's not often that you come out of a movie and be

like, oh my God, that was so good.

yeah, it was, it was, I really enjoyed it.

I was like, oh, that was, I, I thought the ending.

I don't know.

I think I would've done something slightly differently within the ending, but the um,

That is why you are not the director, Curtis.

We're the writer.

Yeah, that's why I'm not the director or the writer.

I, I was not the only one, by the way.

I read some reviews and stuff, and a lot of people felt the ending

was somewhat anti cli climactic.

But it was one of those movies where you're like, what?

What is over?

Like, it's one of those kind of movies, but everything all the

way up to that point I thought, I thought was really, really good.

All the actors are really good.

Um,

never seen a Bollywood movie?

yeah, what's

Where it's like anti-climactic at the very end.

It's like they literally, a lot of the movies boiled the ending down

to like the last like five minutes.

interesting.

Um, this one,

Shrek, oh, you would be proud.

Shrek.

And I watched while you were watching this movie, Shrek and

I watched a Bollywood movie.

Uhhuh,

On Netflix and it was also suspenseful and kind of a comedy

thriller ish, but I would say that they basically wrapped up all the entire

movie in the last like five minutes.

And then at the end, Shraddha and I were like, what did we just watch?

Interesting.

because we, because it left so many things open at the end and

we're like, did we just watch all of that?

Just for that last closing part?

That's funny.

it's like two and a half hours also, so,

Interesting thing about this movie is, you know, how the movie's

gonna end about halfway into the movie.

Like they tell you how the movie's gonna end.

Um, it's just how it gets there.

It, it, it's just was, was really good.

Um, so I recommend it.

Um, and, uh, you know, the, the

are you and

saga continues.

What I.

Are you and Jeff gonna talk about it on

your next, on the other podcast,

know, he keeps threatening to record another

episode and then he doesn't do it.

But I'll , I'll take

I think Curtis is calling you out here.

Jeff and I actually recorded a new episode this week, so look

for it at, The Things That Entertain Us podcast, wherever you listen to podcasts.

Today we're going, we're continuing in our backup to basic series, and we're

covering, again, we're covering the book, modern Data Protection on Hold.

For those of you watching this on backup central.com where you can watch the video

of Prasanna and I waxing philosophical.

Um, and, uh, you know, what we're doing is we're sort of going through the book and

just discussing, you know, some important topics that I, that I found in the book.

Um, and this next one is about protecting backup and archive data, which I

think we can all agree is important.

If you would like to download a free e-book copy of Modern Data Protection,

you can do so at druva.com  slash ebook.

Uh, we have to protect it from a couple of different things, right?

We have to, mainly what I'm talking about in here is protecting

it against attacks, right?

Um, both attacks from like a, um, like a disaster type type attack.

But, but mainly what I'm talking about is protecting it from cyber attacks, which.

Um, I'm gonna have these same conversations over and over, so

sometimes I forget what we say here versus what I say other places, but

it, it is a well acknowledged fact that some of the largest of the ransomware

bad actors are specifically targeting backup systems in order to do the.

The extortion style of ransomware attack, and let me specify what I'm saying there.

So, you know, a, a bunch of companies started realizing that, well, if I just

get a decent backup in recovery and disaster recovery system, I can say, you

know, pound sand to the ransomware folks.

Well, the ransomware folks then said, oh, well, we'll show you.

And so they developed this, this exfiltration style, right.

and the um, The idea there is that they find databases and file

systems and spreadsheets and whatnot that contains sensitive data.

That could be two different things.

That could be your company secrets.

It could be your company plans.

It could be your 11 herbs and spices or your, your, um, You know, your

corporate plans for the next year, you know, um, but it could also be

secrets that are like embarrassing.

Uh, I think the best example of that, and I know we've talked

about it, is the Sony attack

This is the one where someone broke in to the

movie studios and got a bunch of emails talking about scripts and

actors and all the rest of that.

and very, some, some, and not very, not very flattering terms.

Yeah.

So it could be that, right?

But the point is, there is no way to protect against that once the data's out.

Yep.

It's not like the other attack.

If you get the, the old school attack, if you had a good backup system, you're good.

It doesn't matter how good of a backup system you have, if the data is

exfiltrated, uh, you're gonna be forced to make some major, uh, decisions, right?

So, um, so what, what these, uh, ransomware groups have started

realizing is, they, uh, instead the, the people are starting to

harden their primary environments.

Right?

And so, but one, you know, you, you, you've heard me talk, did

I say one thing that has never changed in backup and recovery in

the world of backup and recovery?

Do you, do you know what it is?

I often say like the one thing that has never changed,

that the junior person is always responsible for backups

and recovery.

that is technically a cause of the thing of the thing that

Prasanna Malaiyandi:

No one who cares about

no one wants to do the backups, right?

No one wants to raise their hand and say that they want to be the backup guy.

So why does that matter?

Well, it's because it means that, that it ends up being an

an or and an ignored system.

Yep.

and as a result, it means that it gets ignored by everyone,

including the cybersecurity folks.

Right?

They're like, ah, that's just a systematic in the corner.

It's fine.

We don't care about it except it contains all of our data.

Yeah.

And depending on how, how you handle, um, things like encryption,

they may be able to unen encrypt.

Said backups and then, you know, get the keys to the kingdom.

Yeah, and like you were saying earlier, Curtis,

the fact that a lot of people focus on production systems harden

them, but they may not always be considering these backup systems.

Right, right, right.

Easy target.

Easy target.

And so you need to.

not be such an easy target, right?

It's like the old joke about the guy, two guys crossing a field and they

see, they see a bull and the guy says, you know, can you outrun that bull?

He said, Nope, I don't have to.

What are you talking about?

All I have to do is outrun you,

You

So, so you just want to be less of an interesting

target than the person next to you.

So the, the first thing that we talk about in terms of protecting

backup data is encryption,

Yeah, which everyone should be doing.

Like if no, if you aren't doing encrypting your backups, I don't

know what's wrong with you.

Like even if it's not encrypted, honestly,

Tell us what you really think.

No.

, no, because encryption isn't some heavyweight thing that it used to be.

Right.

You don't have to go physically encrypt your data as it comes out.

You could do disc level encryption, right?

There's so.

Ways you could leverage encryption to fit into your environment.

It should be a no-brainer that all your data at rest is encrypted.

And in a lot of companies, right?

If you wanna fit, be sort of like, uh, what is it?

If you wanna look at HIPAA com, uh, readiness or other things, right?

Other regulatory obligations.

A lot of 'em say All your data at at rest should be encrypt.

Mm-hmm.

. Well, I, um, I do think, I do think backup page backup data should be

encrypted for this very reason.

Right?

And I'll give you different reasons for the different things, right?

So if it's, if you got a backup server on-prem and you're

storing your backups on a.

File system, you know, an NFS mounted file system.

Then if they're unencrypted, you know

Everyone has access.

Yeah.

Everyone has access to those backups and a lot of these PR

backup products they offer like to, to, to, so some of the backup products store

the backups, not in a backup format.

They're in native format, so you could just mount a disc, right?

Other backup products, put it in a backup format.

but they offer an ability to read that backups like it,

it's, it's for DR purposes.

So, so if you're not encrypting them, then you're just, you're

just asking for trouble.

So that's the disc side.

The second side will be the tape side.

So if you are, and believe it or not, many people are still making backup

tapes and they're making 'em primarily, I think for offsite, uh, storage.

Yep.

And so, And this is the, this is the one where I'm like,

if you are ma, if you are putting your company's like crown jewels onto a little,

you know, plastic tape and then you're handing it to a man in a van and you are

not at this point encrypting that data, you know, professional malfeasance at this

Well, and I wonder, it's almost as if that should be like

a default, you know, like you can't write data out without encrypting it,

but I guess you have to deal with key management and all the rest of that,

and so, , do you have flexibility to shoot yourself in the foot?

yeah, exactly.

Um, but I will also say this encryption is only good, is only as good as the

authentication and authorization and key management system that is connected to it.

And if you can easily defeat.

That, uh, system then, you know, and, and, and I mentioned, so for example,

I know we mentioned it on the podcast, please do not make your backup server

part of your, uh, active directory.

What, what is it, what is it called?

A cloud?

Just a domain.

Yeah.

Um, please don't do that.

, so the next term is one that we can debate.

Uh, and I, I honestly, I don't even need you.

I can debate it myself, right?

It's one of those where I go back and forth, um, you

know, and that's the term air.

What is like air gap?

Curtis?

well, well, there is, what was an air gap?

Yeah.

It's, and it is where it, it is from once we get the term.

So the idea was to put a gap of error between the protection

copy and the protected thing.

Or, or since you're dealing with plumbing

right now, since you're dealing with plumbing right now, right?

You have an air gap normally, right?

When you think about your dishwasher

Now you made me think about the dishwasher that

or,

life a living hell.

Fine.

Or a toilet.

How about that?

that's actually technically a water gap, but

W. Curtis Preston:

Don't, you're killing me.

W. Curtis Preston:

Don't be, don't be bringing plumbing into this.

W. Curtis Preston:

Um, anyway, yeah.

W. Curtis Preston:

So you, you had to separate the, and it meant, again, it meant the,

W. Curtis Preston:

it meant the man in the van, right?

W. Curtis Preston:

You, you, you, you put some tapes in a box and you handed that box to a, you

W. Curtis Preston:

know, a man or a woman, and then they got into a van and they took it someplace.

W. Curtis Preston:

And in order to get to that thing, you needed to physically, Um, access it.

W. Curtis Preston:

Right?

W. Curtis Preston:

Which meant that if you were a hacker and you wanted to get to it, it,

W. Curtis Preston:

it would be virtually impossible.

W. Curtis Preston:

Not completely impossible.

W. Curtis Preston:

There's no, there's no, cuz I remember,

than cyber.

yeah, I remember for example, we would, uh, once a

year at an unannounced time, we would attempt to circumvent the security of.

Uh, storage fender and we would, uh, use really good like liars and whatnot.

And more specifically, we would use someone like me that they

knew, they knew the person.

And so they would go over there and they would, and we would just have

this really, really like elaborate.

Scenario that basically would want, would want the, it would make the

person that works there want to leave us alone in the vault with the tapes.

And at least once we were able to do that,

Hmm.

Was a person go, or did the person have to go through training, I guess.

Yeah, I don't, I, you know, honestly, I was so long ago, I don't

remember what happened to that, to that person who left us alone in the vault.

But, so that's what I'm saying.

It's not, it, it was never 100%.

Right.

Um, the, um, I mean, you've seen the movies, you know, I've

seen Oceans 11 and 12 and 13.

The um, but the problem is most of us don't use tape.

Mm-hmm.

And most of us, or many of us don't even

use a data center anymore.

So this idea of putting a, an actual air gap between A and B is quite difficult.

And so we've sort of adopted or adapted to the, you know, the virtual air gap.

Right.

Um, the, um, Uh, so this is separated in as many ways as we can.

I don't know.

You want to talk about

Yeah, and I think we talked about this on the episode about

backups, or maybe it was restores, but.

It's even though they are available online, right, that second copy,

so it's not the same as having that tape in a vault somewhere.

You wanna make that, make it as close to that as possible.

And we totally get that.

You still have to have connectivity, right?

It's still gonna be up, but can you lock it down?

Can you basically make that target as inaccessible as possible?

Uh, to prevent it from being attacked by say, someone, um, gaining access

to your backup source, right?

And now they have access to your vault.

Uh, some of the examples are if you're backing up in the cloud, right, put

it in a separate account that no one has access to other than whatever

is transferring the data, right?

Make it in a different region.

So it's not easily accessible.

And there are many, many, many other things you could do as well.

But those are just some of the example.

Yeah, I like if we're talking cloud, I like the

idea of putting it in a different region and a different account.

Right.

Um, because we're not just talking about hacking.

We're also, we also need to think about like, Disaster recovery

and things like that, right?

So again, O V H Cloud, we don't want to have that happen to us.

So we want to make sure that a copy of the backup data is, is well far away

from the thing that it's protecting.

And yeah, separate account in a much more limited, an account with much

more limited access, and you can.

Um, you can consider using immutable storage, but we're

gonna talk about that later.

But essentially, this is the, we, we separated as much as possible.

This is why, um, you know, one of your previous employers data domain, right?

And, and, and, All of those things, they would always talk about

replicating to another data domain.

And I always, I always remember thinking, well, if I'm replicating to

another data domain, it's essentially like in order to do that, I gotta

have it in the same like land.

Right?

Or, or at least it's gonna look like it's in the land.

Right.

Um,

that's why they actually introduced a

feature, specifically talking about data domain, right?

Their cyber recovery solution.

Mm-hmm.

that allowed for writing to a lockdown data domain that

didn't have access from anywhere else.

And going back to the point you were talking about, Curtis, okay, is it

on the network and I can connect to it then, doesn't that make it open?

And so what it would actually do is shut down network ports right

outside of your backup windows, such that it's not available, or

at least your attack surface is.

Could you delete the backups?

Was it.

Could you delete the backups once you send it to it via the backup interface?

Typically no.

Okay.

Because that, cuz that would be, that would be an important part.

Right.

It would also replicate like your

backup environment, right?

So you'd have your own backup server in that vault, if you will.

So there's no connectivity to anything outside, because if there

is, then you're just open to risk.

Right, right.

Um, yeah, so just like, so I've got a list here on virtual air gap here.

So I talk about disabling or impairing R D P.

Why does that matter?

Ah, so many ransomware issues have happened.

Rdp, which

what is R D P by the way?

Thank you,

right?

Which is how most people connect remotely to a Windows client.

Um, most people forget to turn it off or they leave it on because it's just easier.

They don't need to physically connect to it.

Uh, The only downside is it's a big attack surface, and there are

a lot of exploits using R D P.

And so when you enable R D P, you're letting yourself or leaving yourself

open to hackers and other exploits coming in, which could then move

laterally across your network and take out other pieces of your environment.

Yeah, there is a way to automate sort of disabling and

enabling R D P, um, like on a large scale.

Um, I also talk about putting it in a different operating system, right?

Not using the same operating system for your backups in your

primary, uh, if that's possible.

I know that there's a lot of big window shops and they use windows.

Uh, backup servers.

I don't think that's a good idea.

Um, but you know, Um, and by the way, I, I forgot to throw out our disclaimer.

I'll throw out, um, Prasanna.

Prasanna.

And I work for different companies.

I work for Druva.

He works for Zoom.

And this is not a podcast of either company.

This is an independent podcast and the opinions that you hear are ours.

And, uh, be sure to rate us@ratethispodcast.com slash restore.

And, um, if you'd like to join the conversation, you want to talk.

What's going on in your world?

Uh, you know, whether you're, uh, you know, an end user.

If you're an end user, we really want to have you, you know, real

people that actually do backups out there in the trenches.

Weed.

You're, you're it, man.

No I do, uh, you know, as soon as I think about, um, Back when I used to

be the person in the trenches, like when I was a consultant, and I, and

I, I flashback to this one time where we were completely redoing the backups

of, I'll just say a large television station, like, like a national

television station, and I remember.

I remember that time when the, when the, the, the director of it basically said,

nobody's going home until this is done.

He literally, I mean, there's a long story following up to this, but he basically,

um, held us hostage in the data center and said, no one's going home until this,

this thing that you said is gonna happen.

You said it's gonna happen.

I'm like, yeah, but it's not gonna finish until.

10 30 at night, he's like, yeah, so we're not going home until it's done.

And I'm like, okay, well this is kidnapping, but whatever.

Um, you know, uh, so I think about stuff like that, right?

I think about those moments of terror.

But yeah, so I, you know, I wanna live vicariously through those who have

been through those moments of terror.

Um,

interesting to learn like what the more recent challenges

are in environments, because otherwise

That too.

Yeah,

otherwise, how do you build great products, you know, or build

Exactly, exactly.

So what, what I was thinking about saying, which is what reminded me to do the

disclaimer, is just that, um, you know, one way to have a different operating

system is to use a SaaS provider.

Druva is not the only one, uh, but there are SaaS providers

that are cloud native or use.

Non windows, uh, tools that, um, you know, you do that.

Also, you, you gain the second thing that I list here, which

is about separating the storage.

Please don't, um, you know, we talked, we talked about that already,

the, these replication, right?

Uh, but the, um,

but it's also like, don't use nfs.

Yeah, exactly.

Yes.

Don't use NFS as a way to back up to the server use.

There are ways to back up to, for example, a data domain box in other boxes

without exposing the backups via nfs.

Uh, I also talk about using object storage.

I'm a huge fan of using object storage.

Now, some of you go, well, object storage is too slow, to which I

say, then you're using it wrong.

Right.

Um, Druva uses object storage.

All our backups are on object storage, and we are not too slow.

In fact, we get into competitions all the time with these big

on-prem companies and we win.

And here we are, we're a copy in the cloud and they're an on-prem

appliance, and we win that restore test.

Object storage is not too stor slow, but if you, if you treat it like file system

storage, I think then it is, right?

If you put.

Yeah.

all your backups in one object, right?

Tonight's, tonight's backup of server X is an image, and that

image becomes an object on, uh, s3.

Yeah.

Then it's gonna be slow, right?

Um, we talk about using aut storage.

We're gonna come back to that in a minute.

Um, that, that's an option that we talked

And then,

S3

yeah, and then your.

And then your favorite topic is using tape.

I know, Curtis, you always like to throw out the disclaimer.

Actually, I should thank you because before this podcast,

I knew very little about tape.

Now I know just a bit more about tape, given the number of experts

who've come on in talking to you.

But yeah, it's

There there's no, there's no beating the tape when it comes.

That's an actual air gap.

Yep.

right?

If, if you, if you want a real air gap, I'm, I know there's challenges with

it, you know, uh, but there is, you know, there is, there is an air gap

Yep.

Uh, and I already talked about using a backup service.

So let's talk about immutability.

Yep.

Um, this,

Is it a made up word?

It sounds a little like a made up word,

you know, this is another one of those things

where we have to go back in time.

Historically immutability wa it, it, it was like a legal term that you needed

to be able to prove that the thing you were presenting in court had not changed.

It was immutable.

That's all, you know, immutable just means cannot be changed.

So when you're, when you.

uh, you pres, you, you know, you have to preserve chain of custody and you

have to, you have to be able to say, or it's really helpful if you're able

to say, this email that I'm showing you is exactly the same email as the email

that we got five years ago from Steve.

Yep, and nothing changed and no one had

the ability to change it yet.

That's where we used to talk about immutability.

Now we talk a lot about it in terms of cyber attacks.

And I would add to that, we also talk about it in terms of things like bit rot.

We've talked about bit rot on this podcast where this is

silent data corruption, right?

That bits just flip underneath magnetic storage devices.

It's just the way they work.

and the, you know, and if, if you've got, if you've got a bunch of petabytes

of data, you got flipped bits in there.

It's just a matter of, it's just

when you're gonna run across it, right?

yeah.

Yeah.

Um, so, so when we talk about immutability, we say, well, we, if

somebody says they have immutable.

Backups or immutable storage.

What you're saying is stuff that gets put here can't be changed.

And here's the thing.

on how you look at it, it's complete nonsense.

Because everything's just software.

Anything can be cheap.

is changeable.

Give me a torch, I'll change that.

S o b, right?

Give me phy.

So physical.

All bets are off.

Everyone knows that, um, if it's something on a server that you can, that, that,

that is in your data center, that also, in my opinion, is not really immutable.

Um, because if someone has root, uh, on that server, they can wipe the server.

Uh, it's even built, like, you know, when we talk about immutable storage on

Lennox, it's not really that immutable.

It's it's immutable ish.

So, so why do I say, like, why do I talk about it if it's bs?

Well, here's the point.

Nothing was ever a hundred percent immutable,

Even that document right, that you were talking about.

Yeah, exactly right.

As long as like if, if, if you put something on a a, a, a, right once

cd, uh, optical platter, right?

A worm tape, it's immutable to a point.

If you have physical access, it's no longer immutable.

You could, you could say from a legal standpoint, you could later, if you have

the same disc and you've preserved chain of custody, and you could say, we can

show that this disc was not destroyed, and we can show via these check sums and

whatnot, that the stuff that we're giving you is the stuff that we have before.

Saying that you've got a storage system that can't ever, ever, ever be changed.

Um, I would argue it's just nonsense.

Right.

So it's, it's a, it's a, it's a, it's a degree.

It's a what, what did we call it?

Like a, like a spectrum.

Thank you.

I knew there was a word in there.

And even

prism is what was coming to mind.

and even for those storage systems that have immutability,

right, that are used for like financial records and everything else, a lot

of that is certified by the storage vendors saying they have all the checks

in place and all the rest, right?

In the end it's all just software, right?

So they've built in the, into the software, those checks to make sure

that data cannot be deleted, right.

And is present.

And the, and you know, after we're doing all this, like

we're gonna, I'm slamming it and then I'm gonna talk about how important it's

. But the other is, um, Uh, I'd say the, the one that I like the best right now

is like the, the object lock and s3.

And what I like about that is because it's, it is at least

physically separated from.

You.

Now, that doesn't mean that somebody can't go crazy in the AWS data

center and, and blow it up, but even that, it's built into that.

So it, it would have to be like, it would have to be a really concerted

attack to be able to attack multiple locations of S3 to be able to do damage.

it's, it's not only multiple

is as low as, what's that?

It's not just multiple locations of s3, but also

they have to tack your primary site as well, so everything needs to be

coordinated across multiple vendors and corporations, which will probably have

their own security practices, et cetera.

this is, this is that, this is that, uh, spectrum, right?

I, I'd put, I'd put that and write, you know, worm tape, write once,

read, write once, read many tape.

Um, and then, and, and optical plat.

I put that on one end of the spectrum.

on the other put at the end of the spectrum is I would have unencrypted

backup stored on a NFS mounted storage system behind a Windows backup server.

Yep.

that's your spectrum.

So we wanna be closer to this end than that end.

Right?

Um, nothing is ever

downsides.

Yeah.

And there are some downsides to immutability too, right?

Uh, you can't, like once the data gets written, you can't delete it

before that time period typically.

So there are some challenges as well.

So, um, so what we need to do, Is we need to, I, I think

there's multiple things there is that, um, we need to protect against access.

We need to protect against, you know, change deletion, corruption.

Um, uh, what was I, what was I talking about?

Um, bit rot, right?

So there are vendors out there.

they're like, well, we have a, append only file system, and and,

and we have data lock on the backups.

And so they say, we're immutable.

And, and I'm like, okay.

And, and they say because they're saying that like, you can't encrypt the backups

with a, with a ransomware attack.

Yep.

and I, and I go, that's good.

That's farther, that's, that's closer to the other one than the other ones.

But it's like, well, what about other things?

Right.

What about other types of attacks that attack the operating system itself?

Right.

You know, if if you gain privileged access to that server,

right?

And I wanna clarify though, what you're talking

about is someone who's written their own file system or is leveraging a file

system on top of a standard server.

Yes.

Not a storage appliance, because I think that's a

little bit more, that's like further along in the spectrum, I would say

like a purpose-built storage appliance

yeah.

So, okay.

So, so, so we got a couple different types of backup servers here, right?

So we got the purpose-built backup appliance of various

flavors.

Yep.

flavors.

Sure.

Um, and I would put them.

More immutable, right?

So immutable is supposed to be a binary condition, but I

don't see it as that, right?

So, um, I put it more immutable than I bought a box.

I installed Windows or Linux, and I install my favorite backups or software.

Why?

Why, why do you feel that way?

I agree with you.

Why do you feel that way?

I feel that way because when it comes to that appliance,

typically there's more things locked down.

There's less configuration options, right?

It's kind of purpose built for that reason.

Versus when you're rolling your own, you have to worry

about all those dependencies.

What flavor of the OS are you taking?

Are you running through all the security patches?

Is it, are there any nuances in the way it's being deployed

today that leads to security vulnerabilities and things like that?

Yeah, exactly.

And, and when you update those appliances, you update an

image which updates everything.

Versus if I have a a box, a Windows box, I gotta update Windows or Linux.

I've gotta make

the backup

following the new security, then you gotta

update the backup software.

So, yeah.

And these are, and I would say that's, so I would say that the

appliances are more immutable than.

Than the build drill in box.

Um, and the, uh, but I, but I'm just gonna say that if you, if we're

talking physical access, I still, I'm gonna put a service like S3 or a

service like Druva that's up in the cloud, farther down the immutability

spectrum than a survey, than a server that is sitting in your data center.

Yep.

And I think it's important because everyone is now doing virtualized, right?

It doesn't matter if it's a physical server or a virtual server.

Anything that's running on your infrastructure or in your

Right,

is less secure in that immutability spectrum.

Yeah.

Um, the, uh, yeah, I put in the book a lot of things are mislabeled, immutable,

uh, I just, again, it's a spectrum, but.

If, if all you're doing is writing backups to a Linux file

system with the immutability flag turned on, that is not immutable.

I mean, it's, it is more immutable than not But if I, but if someone

with root can and, and you gotta have someone with root, right?

So if a person with root can go in and unset all those immutability flag,

It don't matter.

That's, I don't know.

That's just not, it's be, it's better than nothing.

Right?

So I, I don't want to be the, the, you know, perfect is the enemy of good or

whatever, but I, I don't need perfect.

And so I don't wanna say that that's crap, but I, I think it's, it's not as immutable

as those appliances that we talked about.

And I still think that a service where nobody gets access to your

servers is more immutable than that.

Yeah.

and I still want everybody to back up everything to Tape

Oh, Curtis, you and

although I work at, I work at a tapeless backup company.

Um, so

But, but here's

Go

one thing I wanna ask, and I know we've

covered this on prior podcasts.

What are things that an admin can do to understand where on that spectrum,

when someone says, when a vendor says immutability, what are questions

that they should be asking to be able

a great question.

Who has root or admin, right?

Who has that and what controls are placed over that?

So, um, if it's an appliance, so like I, I know of like one company.

They do have root, they have a password, they have the root password

on your system, or they have the password to an account that has

a u i D of zero for those of you.

Um, which is essentially the same thing, but in order to log into that account,

they can't log into that account remotely.

You have to.

You have to do an SSH tunnel and all that stuff, right?

You have to open up a door for them to log in.

Um, you know, what kind of protections are put against that.

Uh, if the answer is it's just a local box and you're the one that has route,

or there's unprotected route access from someone, um, I just, I worry well.

you know, so if, if, if you or anyone in your ministry in your

place has root, that's not very immutable, it's better than nothing,

but it's not very immutable, right?

If you know root, like if, if, if you normally never log into

the system as root, right?

No one in your, you only at you, you go to a ui, you log it as you, right?

Um, and.

There isn't direct route access by anyone.

And the only way you can get route access is you can, um, you do the SSH

tunnel thing that's more immutable.

I like that better.

Um, you still, but again, if you don't know this already, you still have

to do physical security against that

Yep.

Exactly.

do every other box.

Right.

Um, and then, You know, you ask a lot of que just ask a lot of questions.

What happens if so, for example, I'm currently wondering because I haven't

found a good answer online, I'm currently wondering what happens with Amazon S3

object lock if I delete my account.

I don't think they allow you to.

It depends on what type of object lock you're using, because there are

know what, I know what you're saying, right.

Um, but if I'm u is it the compliance mode?

It's the more,

Yeah.

yeah.

So assuming I'm using compliance

Oh wait.

Compliance is R L C R, lg.

I think compliance is less in governance is more, or is it the

whichever it

Yeah.

Whichever way

I think it's the other way around.

But anyway, so I'm using the more strict one.

and my credit card stops working.

Right.

It's not like they're gonna keep my storage forever.

You, when you say they won't let you delete it.

Well, I don't have to delete it.

I'm just gonna have my credit card not work anymore.

Right.

What

like you said, it's not protected in all scenarios, right?

Right.

I don't know what the answer is.

I am curious.

I actually bought, I, I keep forgetting to do this, but I want to

go create an Amazon account separate.

Mm-hmm.

Put some object lock stuff in there.

Not a whole lot, just like 90 days or something, but, but like a couple of

gigabytes or something, and then go delete my account and see what happens.

See if they let me delete the account.

Um, I did read something somewhere that, that there is

this like 30, 60 day timeframe.

Hmm.

So again, that's still better because I'm assuming

that if you're using Amazon S3 and.

by.

or any of the other folks you're gonna, you're gonna know

that somebody deleted your account

Yeah.

Right.

Um, and I know what I would want you to ask is what happens if, what happens

if someone gets through someone?

So I was just seeing a thing that said that it's like more than half of hacks

are through compromised credentials.

So what happens if someone, you know, you're a backup service?

What happens if someone gets ahold of my admin credentials

and is able to circumvent mfa?

Yep.

How much damage can they

do you have against?

Yeah, how much damage they can do.

Can you undo any of that damage, et cetera, et cetera, et cetera.

I know that if we go back in time, let's say two years, I don't think

Druva had good answers to that question.

Right.

I remember being in those meetings and saying, Hey, we need to address this.

We have addressed that.

I think we now have really good answers to those questions.

Um, but does your, does you know, if you're not using Druva, does your vendor

have good answer to those questions?

I don't know.

Right.

So, yeah, so come up with worst case scenarios and that one.

Of compromised admin credentials, whether you're talking a service or

so, like for example, I know that like Druva has data lock, you know,

rubric and cohesive have data locks.

I know that with them, if you, if you compromise, uh, credentials and

you log into them and try to delete, if you, if you enable datalock,

you will not be able to do so.

Right.

So, , what is your vendor's answer to that question?

Um, and um, and then also ask 'em about bit rot.

What are they doing about bit rot?

If they're using object storage, I feel better cuz object storage

will automatically detect bit rot happening underneath.

Uh, cuz it will change the hash and somebody will be like, Hey, well not

somebody, a program will figure it out.

Um, I think Biro is rare.

Just for the record.

I think it's rare.

Um,

It's rare.

bit error.

Right.

But,

But it's still bad

not, not a problem.

Yeah.

And the problem is when it's silent, that's the worst.

Exactly.

That's the worst.

All right.

Well enough about protecting backup data.

You got any final thoughts?

Well, I think we covered it all.

I really like virtual air gaps, but that's just me

I like the old school air gaps, but it's really

hard to do those these days.

All right.

Well thanks for listening to us.

Be sure to subscribe so that you can restore it all.