On this episode of restored all we're talking about how to secure your backup
data, to make sure that it doesn't get attacked in a ransomware attack.
Hope you enjoy.
The episode.
W. Curtis Preston: Hi, and welcome to Backup Central's Restore it All podcast.
I'm your host, W.
Curtis Preston, a k a, Mr.
Backup, and I with me, a guy whose only major flaw is that he
won't go see scary movies with me.
Prasanna Malaiyandi.
How's it going?
Prasanna.
Prasanna Malaiyandi: I am good, Curtis.
I, yeah, I'm not a big fan of scary movies, especially on a large screen.
Um, I don't like, I know some people like you love it, but it's just I don't
find joy in watching a scary movie.
I do remember in college watching the Ring.
On a big screen.
W. Curtis Preston: That movie messed with my head.
I remember coming home and asking someone to take my picture.
And saying, is there a thing in the picture?
You know, I was scared.
I was scared to take the picture.
Yeah.
That movie, that was, that was a fun movie.
Um, it really kind of messed with your head.
Prasanna Malaiyandi: And you just saw a scary movie just recently.
W. Curtis Preston: Yeah.
It, it,
Prasanna Malaiyandi: Well, I wouldn't know if I'd called
W. Curtis Preston: movie per se, it's more like a thriller.
Um, It's called the Menu and it stars.
Uh, Ralph finds, um, a k a Voldemort, and uh, Anya, the girl from the Queens gambit.
Yeah.
Um, and some other, other character actors that you may be familiar with.
Um, and, uh, it, it's, it's a movie that I, I enjoyed the movie.
I will just say this, um, trigger alert for those who are bothered by such things.
It does contain suicide.
Um, having said that, um, within the context of the show, it, it was a
significant part of the storyline.
Prasanna Malaiyandi: Mm-hmm.
W. Curtis Preston: Um, and um, but having said that, it
was a really unique storyline.
That's a commentary on Modern Day Society.
You know, there's a class thing, there's a, there's a, yeah.
Prasanna Malaiyandi: It's not often that you come out of a movie and be
like, oh my God, that was so good.
W. Curtis Preston: yeah, it was, it was, I really enjoyed it.
I was like, oh, that was, I, I thought the ending.
I don't know.
I think I would've done something slightly differently within the ending, but the um,
Prasanna Malaiyandi: That is why you are not the director, Curtis.
We're the writer.
W. Curtis Preston: Yeah, that's why I'm not the director or the writer.
I, I was not the only one, by the way.
I read some reviews and stuff, and a lot of people felt the ending
was somewhat anti cli climactic.
But it was one of those movies where you're like, what?
What is over?
Like, it's one of those kind of movies, but everything all the
way up to that point I thought, I thought was really, really good.
All the actors are really good.
Um,
Prasanna Malaiyandi: never seen a Bollywood movie?
W. Curtis Preston: yeah, what's
Prasanna Malaiyandi: Where it's like anti-climactic at the very end.
It's like they literally, a lot of the movies boiled the ending down
to like the last like five minutes.
W. Curtis Preston: interesting.
Um, this one,
Prasanna Malaiyandi: Shrek, oh, you would be proud.
Shrek.
And I watched while you were watching this movie, Shrek and
I watched a Bollywood movie.
W. Curtis Preston: Uhhuh,
Prasanna Malaiyandi: On Netflix and it was also suspenseful and kind of a comedy
thriller ish, but I would say that they basically wrapped up all the entire
movie in the last like five minutes.
And then at the end, Shraddha and I were like, what did we just watch?
W. Curtis Preston: Interesting.
Prasanna Malaiyandi: because we, because it left so many things open at the end and
we're like, did we just watch all of that?
Just for that last closing part?
W. Curtis Preston: That's funny.
Prasanna Malaiyandi: it's like two and a half hours also, so,
W. Curtis Preston: Interesting thing about this movie is, you know, how the movie's
gonna end about halfway into the movie.
Like they tell you how the movie's gonna end.
Um, it's just how it gets there.
It, it, it's just was, was really good.
Um, so I recommend it.
Um, and, uh, you know, the, the
Prasanna Malaiyandi: are you and
W. Curtis Preston: saga continues.
What I.
Prasanna Malaiyandi: Are you and Jeff gonna talk about it on
your next, on the other podcast,
W. Curtis Preston: know, he keeps threatening to record another
episode and then he doesn't do it.
But I'll , I'll take
Prasanna Malaiyandi: I think Curtis is calling you out here.
W. Curtis Preston: Jeff and I actually recorded a new episode this week, so look
for it at, The Things That Entertain Us podcast, wherever you listen to podcasts.
Today we're going, we're continuing in our backup to basic series, and we're
covering, again, we're covering the book, modern Data Protection on Hold.
For those of you watching this on backup central.com where you can watch the video
of Prasanna and I waxing philosophical.
Um, and, uh, you know, what we're doing is we're sort of going through the book and
just discussing, you know, some important topics that I, that I found in the book.
Um, and this next one is about protecting backup and archive data, which I
think we can all agree is important.
If you would like to download a free e-book copy of Modern Data Protection,
you can do so at druva.com slash ebook.
Uh, we have to protect it from a couple of different things, right?
We have to, mainly what I'm talking about in here is protecting
it against attacks, right?
Um, both attacks from like a, um, like a disaster type type attack.
But, but mainly what I'm talking about is protecting it from cyber attacks, which.
Um, I'm gonna have these same conversations over and over, so
sometimes I forget what we say here versus what I say other places, but
it, it is a well acknowledged fact that some of the largest of the ransomware
bad actors are specifically targeting backup systems in order to do the.
The extortion style of ransomware attack, and let me specify what I'm saying there.
So, you know, a, a bunch of companies started realizing that, well, if I just
get a decent backup in recovery and disaster recovery system, I can say, you
know, pound sand to the ransomware folks.
Well, the ransomware folks then said, oh, well, we'll show you.
And so they developed this, this exfiltration style, right.
and the um, The idea there is that they find databases and file
systems and spreadsheets and whatnot that contains sensitive data.
That could be two different things.
That could be your company secrets.
It could be your company plans.
It could be your 11 herbs and spices or your, your, um, You know, your
corporate plans for the next year, you know, um, but it could also be
secrets that are like embarrassing.
Uh, I think the best example of that, and I know we've talked
about it, is the Sony attack
Prasanna Malaiyandi: This is the one where someone broke in to the
movie studios and got a bunch of emails talking about scripts and
actors and all the rest of that.
W. Curtis Preston: and very, some, some, and not very, not very flattering terms.
Yeah.
So it could be that, right?
But the point is, there is no way to protect against that once the data's out.
Prasanna Malaiyandi: Yep.
W. Curtis Preston: It's not like the other attack.
If you get the, the old school attack, if you had a good backup system, you're good.
It doesn't matter how good of a backup system you have, if the data is
exfiltrated, uh, you're gonna be forced to make some major, uh, decisions, right?
So, um, so what, what these, uh, ransomware groups have started
realizing is, they, uh, instead the, the people are starting to
harden their primary environments.
Right?
And so, but one, you know, you, you, you've heard me talk, did
I say one thing that has never changed in backup and recovery in
the world of backup and recovery?
Do you, do you know what it is?
I often say like the one thing that has never changed,
Prasanna Malaiyandi: that the junior person is always responsible for backups
and recovery.
W. Curtis Preston: that is technically a cause of the thing of the thing that
Prasanna Malaiyandi: No one who cares about
W. Curtis Preston: no one wants to do the backups, right?
No one wants to raise their hand and say that they want to be the backup guy.
So why does that matter?
Well, it's because it means that, that it ends up being an
an or and an ignored system.
Prasanna Malaiyandi: Yep.
W. Curtis Preston: and as a result, it means that it gets ignored by everyone,
including the cybersecurity folks.
Right?
Prasanna Malaiyandi: They're like, ah, that's just a systematic in the corner.
It's fine.
We don't care about it except it contains all of our data.
W. Curtis Preston: Yeah.
And depending on how, how you handle, um, things like encryption,
they may be able to unen encrypt.
Said backups and then, you know, get the keys to the kingdom.
Prasanna Malaiyandi: Yeah, and like you were saying earlier, Curtis,
the fact that a lot of people focus on production systems harden
them, but they may not always be considering these backup systems.
W. Curtis Preston: Right, right, right.
Prasanna Malaiyandi: Easy target.
W. Curtis Preston: Easy target.
And so you need to.
not be such an easy target, right?
It's like the old joke about the guy, two guys crossing a field and they
see, they see a bull and the guy says, you know, can you outrun that bull?
He said, Nope, I don't have to.
What are you talking about?
All I have to do is outrun you,
Prasanna Malaiyandi: You
W. Curtis Preston: So, so you just want to be less of an interesting
target than the person next to you.
So the, the first thing that we talk about in terms of protecting
backup data is encryption,
Prasanna Malaiyandi: Yeah, which everyone should be doing.
Like if no, if you aren't doing encrypting your backups, I don't
know what's wrong with you.
Like even if it's not encrypted, honestly,
W. Curtis Preston: Tell us what you really think.
Prasanna Malaiyandi: No.
, no, because encryption isn't some heavyweight thing that it used to be.
Right.
You don't have to go physically encrypt your data as it comes out.
You could do disc level encryption, right?
There's so.
Ways you could leverage encryption to fit into your environment.
It should be a no-brainer that all your data at rest is encrypted.
And in a lot of companies, right?
If you wanna fit, be sort of like, uh, what is it?
If you wanna look at HIPAA com, uh, readiness or other things, right?
Other regulatory obligations.
A lot of 'em say All your data at at rest should be encrypt.
W. Curtis Preston: Mm-hmm.
. Well, I, um, I do think, I do think backup page backup data should be
encrypted for this very reason.
Right?
And I'll give you different reasons for the different things, right?
So if it's, if you got a backup server on-prem and you're
storing your backups on a.
File system, you know, an NFS mounted file system.
Then if they're unencrypted, you know
Prasanna Malaiyandi: Everyone has access.
Yeah.
W. Curtis Preston: Everyone has access to those backups and a lot of these PR
backup products they offer like to, to, to, so some of the backup products store
the backups, not in a backup format.
They're in native format, so you could just mount a disc, right?
Other backup products, put it in a backup format.
but they offer an ability to read that backups like it,
it's, it's for DR purposes.
So, so if you're not encrypting them, then you're just, you're
just asking for trouble.
So that's the disc side.
The second side will be the tape side.
So if you are, and believe it or not, many people are still making backup
tapes and they're making 'em primarily, I think for offsite, uh, storage.
Prasanna Malaiyandi: Yep.
W. Curtis Preston: And so, And this is the, this is the one where I'm like,
if you are ma, if you are putting your company's like crown jewels onto a little,
you know, plastic tape and then you're handing it to a man in a van and you are
not at this point encrypting that data, you know, professional malfeasance at this
Prasanna Malaiyandi: Well, and I wonder, it's almost as if that should be like
a default, you know, like you can't write data out without encrypting it,
but I guess you have to deal with key management and all the rest of that,
and so, , do you have flexibility to shoot yourself in the foot?
W. Curtis Preston: yeah, exactly.
Um, but I will also say this encryption is only good, is only as good as the
authentication and authorization and key management system that is connected to it.
And if you can easily defeat.
That, uh, system then, you know, and, and, and I mentioned, so for example,
I know we mentioned it on the podcast, please do not make your backup server
part of your, uh, active directory.
What, what is it, what is it called?
A cloud?
Just a domain.
Yeah.
Um, please don't do that.
, so the next term is one that we can debate.
Uh, and I, I honestly, I don't even need you.
I can debate it myself, right?
It's one of those where I go back and forth, um, you
know, and that's the term air.
Prasanna Malaiyandi: What is like air gap?
Curtis?
W. Curtis Preston: well, well, there is, what was an air gap?
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: It's, and it is where it, it is from once we get the term.
So the idea was to put a gap of error between the protection
copy and the protected thing.
Prasanna Malaiyandi: Or, or since you're dealing with plumbing
right now, since you're dealing with plumbing right now, right?
You have an air gap normally, right?
When you think about your dishwasher
W. Curtis Preston: Now you made me think about the dishwasher that
Prasanna Malaiyandi: or,
W. Curtis Preston: life a living hell.
Prasanna Malaiyandi: Fine.
Or a toilet.
How about that?
that's actually technically a water gap, but
W. Curtis Preston: Don't, you're killing me.
Don't be, don't be bringing plumbing into this.
Um, anyway, yeah.
So you, you had to separate the, and it meant, again, it meant the,
it meant the man in the van, right?
You, you, you, you put some tapes in a box and you handed that box to a, you
know, a man or a woman, and then they got into a van and they took it someplace.
And in order to get to that thing, you needed to physically, Um, access it.
Right?
Which meant that if you were a hacker and you wanted to get to it, it,
it would be virtually impossible.
Not completely impossible.
There's no, there's no, cuz I remember,
Prasanna Malaiyandi: than cyber.
W. Curtis Preston: yeah, I remember for example, we would, uh, once a
year at an unannounced time, we would attempt to circumvent the security of.
Uh, storage fender and we would, uh, use really good like liars and whatnot.
And more specifically, we would use someone like me that they
knew, they knew the person.
And so they would go over there and they would, and we would just have
this really, really like elaborate.
Scenario that basically would want, would want the, it would make the
person that works there want to leave us alone in the vault with the tapes.
And at least once we were able to do that,
Prasanna Malaiyandi: Hmm.
Was a person go, or did the person have to go through training, I guess.
W. Curtis Preston: Yeah, I don't, I, you know, honestly, I was so long ago, I don't
remember what happened to that, to that person who left us alone in the vault.
But, so that's what I'm saying.
It's not, it, it was never 100%.
Right.
Um, the, um, I mean, you've seen the movies, you know, I've
seen Oceans 11 and 12 and 13.
The um, but the problem is most of us don't use tape.
Prasanna Malaiyandi: Mm-hmm.
W. Curtis Preston: And most of us, or many of us don't even
use a data center anymore.
So this idea of putting a, an actual air gap between A and B is quite difficult.
And so we've sort of adopted or adapted to the, you know, the virtual air gap.
Right.
Um, the, um, Uh, so this is separated in as many ways as we can.
I don't know.
You want to talk about
Prasanna Malaiyandi: Yeah, and I think we talked about this on the episode about
backups, or maybe it was restores, but.
It's even though they are available online, right, that second copy,
so it's not the same as having that tape in a vault somewhere.
You wanna make that, make it as close to that as possible.
And we totally get that.
You still have to have connectivity, right?
It's still gonna be up, but can you lock it down?
Can you basically make that target as inaccessible as possible?
Uh, to prevent it from being attacked by say, someone, um, gaining access
to your backup source, right?
And now they have access to your vault.
Uh, some of the examples are if you're backing up in the cloud, right, put
it in a separate account that no one has access to other than whatever
is transferring the data, right?
Make it in a different region.
So it's not easily accessible.
And there are many, many, many other things you could do as well.
But those are just some of the example.
W. Curtis Preston: Yeah, I like if we're talking cloud, I like the
idea of putting it in a different region and a different account.
Right.
Um, because we're not just talking about hacking.
We're also, we also need to think about like, Disaster recovery
and things like that, right?
So again, O V H Cloud, we don't want to have that happen to us.
So we want to make sure that a copy of the backup data is, is well far away
from the thing that it's protecting.
And yeah, separate account in a much more limited, an account with much
more limited access, and you can.
Um, you can consider using immutable storage, but we're
gonna talk about that later.
But essentially, this is the, we, we separated as much as possible.
This is why, um, you know, one of your previous employers data domain, right?
And, and, and, All of those things, they would always talk about
replicating to another data domain.
And I always, I always remember thinking, well, if I'm replicating to
another data domain, it's essentially like in order to do that, I gotta
have it in the same like land.
Right?
Or, or at least it's gonna look like it's in the land.
Right.
Um,
Prasanna Malaiyandi: that's why they actually introduced a
feature, specifically talking about data domain, right?
Their cyber recovery solution.
W. Curtis Preston: Mm-hmm.
Prasanna Malaiyandi: that allowed for writing to a lockdown data domain that
didn't have access from anywhere else.
And going back to the point you were talking about, Curtis, okay, is it
on the network and I can connect to it then, doesn't that make it open?
And so what it would actually do is shut down network ports right
outside of your backup windows, such that it's not available, or
at least your attack surface is.
W. Curtis Preston: Could you delete the backups?
Was it.
Could you delete the backups once you send it to it via the backup interface?
Prasanna Malaiyandi: Typically no.
W. Curtis Preston: Okay.
Because that, cuz that would be, that would be an important part.
Right.
Prasanna Malaiyandi: It would also replicate like your
backup environment, right?
So you'd have your own backup server in that vault, if you will.
So there's no connectivity to anything outside, because if there
is, then you're just open to risk.
W. Curtis Preston: Right, right.
Um, yeah, so just like, so I've got a list here on virtual air gap here.
So I talk about disabling or impairing R D P.
Why does that matter?
Prasanna Malaiyandi: Ah, so many ransomware issues have happened.
Rdp, which
W. Curtis Preston: what is R D P by the way?
Thank you,
Prasanna Malaiyandi: right?
Which is how most people connect remotely to a Windows client.
Um, most people forget to turn it off or they leave it on because it's just easier.
They don't need to physically connect to it.
Uh, The only downside is it's a big attack surface, and there are
a lot of exploits using R D P.
And so when you enable R D P, you're letting yourself or leaving yourself
open to hackers and other exploits coming in, which could then move
laterally across your network and take out other pieces of your environment.
W. Curtis Preston: Yeah, there is a way to automate sort of disabling and
enabling R D P, um, like on a large scale.
Um, I also talk about putting it in a different operating system, right?
Not using the same operating system for your backups in your
primary, uh, if that's possible.
I know that there's a lot of big window shops and they use windows.
Uh, backup servers.
I don't think that's a good idea.
Um, but you know, Um, and by the way, I, I forgot to throw out our disclaimer.
I'll throw out, um, Prasanna.
Prasanna.
And I work for different companies.
I work for Druva.
He works for Zoom.
And this is not a podcast of either company.
This is an independent podcast and the opinions that you hear are ours.
And, uh, be sure to rate us@ratethispodcast.com slash restore.
And, um, if you'd like to join the conversation, you want to talk.
What's going on in your world?
Uh, you know, whether you're, uh, you know, an end user.
If you're an end user, we really want to have you, you know, real
people that actually do backups out there in the trenches.
Weed.
You're, you're it, man.
No I do, uh, you know, as soon as I think about, um, Back when I used to
be the person in the trenches, like when I was a consultant, and I, and
I, I flashback to this one time where we were completely redoing the backups
of, I'll just say a large television station, like, like a national
television station, and I remember.
I remember that time when the, when the, the, the director of it basically said,
nobody's going home until this is done.
He literally, I mean, there's a long story following up to this, but he basically,
um, held us hostage in the data center and said, no one's going home until this,
this thing that you said is gonna happen.
You said it's gonna happen.
I'm like, yeah, but it's not gonna finish until.
10 30 at night, he's like, yeah, so we're not going home until it's done.
And I'm like, okay, well this is kidnapping, but whatever.
Um, you know, uh, so I think about stuff like that, right?
I think about those moments of terror.
But yeah, so I, you know, I wanna live vicariously through those who have
been through those moments of terror.
Um,
Prasanna Malaiyandi: interesting to learn like what the more recent challenges
are in environments, because otherwise
W. Curtis Preston: That too.
Yeah,
Prasanna Malaiyandi: otherwise, how do you build great products, you know, or build
W. Curtis Preston: Exactly, exactly.
So what, what I was thinking about saying, which is what reminded me to do the
disclaimer, is just that, um, you know, one way to have a different operating
system is to use a SaaS provider.
Druva is not the only one, uh, but there are SaaS providers
that are cloud native or use.
Non windows, uh, tools that, um, you know, you do that.
Also, you, you gain the second thing that I list here, which
is about separating the storage.
Please don't, um, you know, we talked, we talked about that already,
the, these replication, right?
Uh, but the, um,
Prasanna Malaiyandi: but it's also like, don't use nfs.
W. Curtis Preston: Yeah, exactly.
Yes.
Don't use NFS as a way to back up to the server use.
There are ways to back up to, for example, a data domain box in other boxes
without exposing the backups via nfs.
Uh, I also talk about using object storage.
I'm a huge fan of using object storage.
Now, some of you go, well, object storage is too slow, to which I
say, then you're using it wrong.
Right.
Um, Druva uses object storage.
All our backups are on object storage, and we are not too slow.
In fact, we get into competitions all the time with these big
on-prem companies and we win.
And here we are, we're a copy in the cloud and they're an on-prem
appliance, and we win that restore test.
Object storage is not too stor slow, but if you, if you treat it like file system
storage, I think then it is, right?
If you put.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: all your backups in one object, right?
Tonight's, tonight's backup of server X is an image, and that
image becomes an object on, uh, s3.
Yeah.
Then it's gonna be slow, right?
Um, we talk about using aut storage.
We're gonna come back to that in a minute.
Um, that, that's an option that we talked
Prasanna Malaiyandi: And then,
W. Curtis Preston: S3
Prasanna Malaiyandi: yeah, and then your.
And then your favorite topic is using tape.
I know, Curtis, you always like to throw out the disclaimer.
Actually, I should thank you because before this podcast,
I knew very little about tape.
Now I know just a bit more about tape, given the number of experts
who've come on in talking to you.
But yeah, it's
W. Curtis Preston: There there's no, there's no beating the tape when it comes.
That's an actual air gap.
Prasanna Malaiyandi: Yep.
W. Curtis Preston: right?
If, if you, if you want a real air gap, I'm, I know there's challenges with
it, you know, uh, but there is, you know, there is, there is an air gap
Prasanna Malaiyandi: Yep.
W. Curtis Preston: Uh, and I already talked about using a backup service.
So let's talk about immutability.
Prasanna Malaiyandi: Yep.
W. Curtis Preston: Um, this,
Prasanna Malaiyandi: Is it a made up word?
It sounds a little like a made up word,
W. Curtis Preston: you know, this is another one of those things
where we have to go back in time.
Historically immutability wa it, it, it was like a legal term that you needed
to be able to prove that the thing you were presenting in court had not changed.
It was immutable.
That's all, you know, immutable just means cannot be changed.
So when you're, when you.
uh, you pres, you, you know, you have to preserve chain of custody and you
have to, you have to be able to say, or it's really helpful if you're able
to say, this email that I'm showing you is exactly the same email as the email
that we got five years ago from Steve.
Prasanna Malaiyandi: Yep, and nothing changed and no one had
the ability to change it yet.
W. Curtis Preston: That's where we used to talk about immutability.
Now we talk a lot about it in terms of cyber attacks.
And I would add to that, we also talk about it in terms of things like bit rot.
We've talked about bit rot on this podcast where this is
silent data corruption, right?
That bits just flip underneath magnetic storage devices.
It's just the way they work.
and the, you know, and if, if you've got, if you've got a bunch of petabytes
of data, you got flipped bits in there.
It's just a matter of, it's just
Prasanna Malaiyandi: when you're gonna run across it, right?
W. Curtis Preston: yeah.
Yeah.
Um, so, so when we talk about immutability, we say, well, we, if
somebody says they have immutable.
Backups or immutable storage.
What you're saying is stuff that gets put here can't be changed.
And here's the thing.
on how you look at it, it's complete nonsense.
Prasanna Malaiyandi: Because everything's just software.
Anything can be cheap.
W. Curtis Preston: is changeable.
Give me a torch, I'll change that.
S o b, right?
Give me phy.
So physical.
All bets are off.
Everyone knows that, um, if it's something on a server that you can, that, that,
that is in your data center, that also, in my opinion, is not really immutable.
Um, because if someone has root, uh, on that server, they can wipe the server.
Uh, it's even built, like, you know, when we talk about immutable storage on
Lennox, it's not really that immutable.
It's it's immutable ish.
So, so why do I say, like, why do I talk about it if it's bs?
Well, here's the point.
Nothing was ever a hundred percent immutable,
Prasanna Malaiyandi: Even that document right, that you were talking about.
W. Curtis Preston: Yeah, exactly right.
As long as like if, if, if you put something on a a, a, a, right once
cd, uh, optical platter, right?
A worm tape, it's immutable to a point.
If you have physical access, it's no longer immutable.
You could, you could say from a legal standpoint, you could later, if you have
the same disc and you've preserved chain of custody, and you could say, we can
show that this disc was not destroyed, and we can show via these check sums and
whatnot, that the stuff that we're giving you is the stuff that we have before.
Saying that you've got a storage system that can't ever, ever, ever be changed.
Um, I would argue it's just nonsense.
Right.
So it's, it's a, it's a, it's a, it's a degree.
It's a what, what did we call it?
Like a, like a spectrum.
Thank you.
I knew there was a word in there.
Prasanna Malaiyandi: And even
W. Curtis Preston: prism is what was coming to mind.
Prasanna Malaiyandi: and even for those storage systems that have immutability,
right, that are used for like financial records and everything else, a lot
of that is certified by the storage vendors saying they have all the checks
in place and all the rest, right?
In the end it's all just software, right?
So they've built in the, into the software, those checks to make sure
that data cannot be deleted, right.
And is present.
W. Curtis Preston: And the, and you know, after we're doing all this, like
we're gonna, I'm slamming it and then I'm gonna talk about how important it's
. But the other is, um, Uh, I'd say the, the one that I like the best right now
is like the, the object lock and s3.
And what I like about that is because it's, it is at least
physically separated from.
You.
Now, that doesn't mean that somebody can't go crazy in the AWS data
center and, and blow it up, but even that, it's built into that.
So it, it would have to be like, it would have to be a really concerted
attack to be able to attack multiple locations of S3 to be able to do damage.
Prasanna Malaiyandi: it's, it's not only multiple
W. Curtis Preston: is as low as, what's that?
Prasanna Malaiyandi: It's not just multiple locations of s3, but also
they have to tack your primary site as well, so everything needs to be
coordinated across multiple vendors and corporations, which will probably have
their own security practices, et cetera.
W. Curtis Preston: this is, this is that, this is that, uh, spectrum, right?
I, I'd put, I'd put that and write, you know, worm tape, write once,
read, write once, read many tape.
Um, and then, and, and optical plat.
I put that on one end of the spectrum.
on the other put at the end of the spectrum is I would have unencrypted
backup stored on a NFS mounted storage system behind a Windows backup server.
Prasanna Malaiyandi: Yep.
W. Curtis Preston: that's your spectrum.
So we wanna be closer to this end than that end.
Right?
Um, nothing is ever
Prasanna Malaiyandi: downsides.
Yeah.
And there are some downsides to immutability too, right?
Uh, you can't, like once the data gets written, you can't delete it
before that time period typically.
So there are some challenges as well.
W. Curtis Preston: So, um, so what we need to do, Is we need to, I, I think
there's multiple things there is that, um, we need to protect against access.
We need to protect against, you know, change deletion, corruption.
Um, uh, what was I, what was I talking about?
Um, bit rot, right?
So there are vendors out there.
they're like, well, we have a, append only file system, and and,
and we have data lock on the backups.
And so they say, we're immutable.
And, and I'm like, okay.
And, and they say because they're saying that like, you can't encrypt the backups
with a, with a ransomware attack.
Prasanna Malaiyandi: Yep.
W. Curtis Preston: and I, and I go, that's good.
That's farther, that's, that's closer to the other one than the other ones.
But it's like, well, what about other things?
Right.
What about other types of attacks that attack the operating system itself?
Right.
You know, if if you gain privileged access to that server,
right?
Prasanna Malaiyandi: And I wanna clarify though, what you're talking
about is someone who's written their own file system or is leveraging a file
system on top of a standard server.
W. Curtis Preston: Yes.
Prasanna Malaiyandi: Not a storage appliance, because I think that's a
little bit more, that's like further along in the spectrum, I would say
like a purpose-built storage appliance
W. Curtis Preston: yeah.
So, okay.
So, so, so we got a couple different types of backup servers here, right?
So we got the purpose-built backup appliance of various
Prasanna Malaiyandi: flavors.
Yep.
W. Curtis Preston: flavors.
Sure.
Um, and I would put them.
More immutable, right?
So immutable is supposed to be a binary condition, but I
don't see it as that, right?
So, um, I put it more immutable than I bought a box.
I installed Windows or Linux, and I install my favorite backups or software.
Why?
Why, why do you feel that way?
I agree with you.
Why do you feel that way?
Prasanna Malaiyandi: I feel that way because when it comes to that appliance,
typically there's more things locked down.
There's less configuration options, right?
It's kind of purpose built for that reason.
Versus when you're rolling your own, you have to worry
about all those dependencies.
What flavor of the OS are you taking?
Are you running through all the security patches?
Is it, are there any nuances in the way it's being deployed
today that leads to security vulnerabilities and things like that?
W. Curtis Preston: Yeah, exactly.
And, and when you update those appliances, you update an
image which updates everything.
Versus if I have a a box, a Windows box, I gotta update Windows or Linux.
I've gotta make
Prasanna Malaiyandi: the backup
W. Curtis Preston: following the new security, then you gotta
update the backup software.
So, yeah.
And these are, and I would say that's, so I would say that the
appliances are more immutable than.
Than the build drill in box.
Um, and the, uh, but I, but I'm just gonna say that if you, if we're
talking physical access, I still, I'm gonna put a service like S3 or a
service like Druva that's up in the cloud, farther down the immutability
spectrum than a survey, than a server that is sitting in your data center.
Prasanna Malaiyandi: Yep.
And I think it's important because everyone is now doing virtualized, right?
It doesn't matter if it's a physical server or a virtual server.
Anything that's running on your infrastructure or in your
W. Curtis Preston: Right,
Prasanna Malaiyandi: is less secure in that immutability spectrum.
W. Curtis Preston: Yeah.
Um, the, uh, yeah, I put in the book a lot of things are mislabeled, immutable,
uh, I just, again, it's a spectrum, but.
If, if all you're doing is writing backups to a Linux file
system with the immutability flag turned on, that is not immutable.
I mean, it's, it is more immutable than not But if I, but if someone
with root can and, and you gotta have someone with root, right?
So if a person with root can go in and unset all those immutability flag,
Prasanna Malaiyandi: It don't matter.
W. Curtis Preston: That's, I don't know.
That's just not, it's be, it's better than nothing.
Right?
So I, I don't want to be the, the, you know, perfect is the enemy of good or
whatever, but I, I don't need perfect.
And so I don't wanna say that that's crap, but I, I think it's, it's not as immutable
as those appliances that we talked about.
And I still think that a service where nobody gets access to your
servers is more immutable than that.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: and I still want everybody to back up everything to Tape
Prasanna Malaiyandi: Oh, Curtis, you and
W. Curtis Preston: although I work at, I work at a tapeless backup company.
Um, so
Prasanna Malaiyandi: But, but here's
W. Curtis Preston: Go
Prasanna Malaiyandi: one thing I wanna ask, and I know we've
covered this on prior podcasts.
What are things that an admin can do to understand where on that spectrum,
when someone says, when a vendor says immutability, what are questions
that they should be asking to be able
W. Curtis Preston: a great question.
Who has root or admin, right?
Who has that and what controls are placed over that?
So, um, if it's an appliance, so like I, I know of like one company.
They do have root, they have a password, they have the root password
on your system, or they have the password to an account that has
a u i D of zero for those of you.
Um, which is essentially the same thing, but in order to log into that account,
they can't log into that account remotely.
You have to.
You have to do an SSH tunnel and all that stuff, right?
You have to open up a door for them to log in.
Um, you know, what kind of protections are put against that.
Uh, if the answer is it's just a local box and you're the one that has route,
or there's unprotected route access from someone, um, I just, I worry well.
you know, so if, if, if you or anyone in your ministry in your
place has root, that's not very immutable, it's better than nothing,
but it's not very immutable, right?
If you know root, like if, if, if you normally never log into
the system as root, right?
No one in your, you only at you, you go to a ui, you log it as you, right?
Um, and.
There isn't direct route access by anyone.
And the only way you can get route access is you can, um, you do the SSH
tunnel thing that's more immutable.
I like that better.
Um, you still, but again, if you don't know this already, you still have
to do physical security against that
Prasanna Malaiyandi: Yep.
Exactly.
W. Curtis Preston: do every other box.
Right.
Um, and then, You know, you ask a lot of que just ask a lot of questions.
What happens if so, for example, I'm currently wondering because I haven't
found a good answer online, I'm currently wondering what happens with Amazon S3
object lock if I delete my account.
Prasanna Malaiyandi: I don't think they allow you to.
It depends on what type of object lock you're using, because there are
W. Curtis Preston: know what, I know what you're saying, right.
Um, but if I'm u is it the compliance mode?
It's the more,
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: yeah.
So assuming I'm using compliance
Prasanna Malaiyandi: Oh wait.
Compliance is R L C R, lg.
I think compliance is less in governance is more, or is it the
W. Curtis Preston: whichever it
Prasanna Malaiyandi: Yeah.
Whichever way
W. Curtis Preston: I think it's the other way around.
But anyway, so I'm using the more strict one.
and my credit card stops working.
Right.
It's not like they're gonna keep my storage forever.
You, when you say they won't let you delete it.
Well, I don't have to delete it.
I'm just gonna have my credit card not work anymore.
Right.
What
Prasanna Malaiyandi: like you said, it's not protected in all scenarios, right?
W. Curtis Preston: Right.
I don't know what the answer is.
I am curious.
I actually bought, I, I keep forgetting to do this, but I want to
go create an Amazon account separate.
Prasanna Malaiyandi: Mm-hmm.
W. Curtis Preston: Put some object lock stuff in there.
Not a whole lot, just like 90 days or something, but, but like a couple of
gigabytes or something, and then go delete my account and see what happens.
See if they let me delete the account.
Um, I did read something somewhere that, that there is
this like 30, 60 day timeframe.
Prasanna Malaiyandi: Hmm.
W. Curtis Preston: So again, that's still better because I'm assuming
that if you're using Amazon S3 and.
Prasanna Malaiyandi: by.
W. Curtis Preston: or any of the other folks you're gonna, you're gonna know
that somebody deleted your account
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: Right.
Um, and I know what I would want you to ask is what happens if, what happens
if someone gets through someone?
So I was just seeing a thing that said that it's like more than half of hacks
are through compromised credentials.
So what happens if someone, you know, you're a backup service?
What happens if someone gets ahold of my admin credentials
and is able to circumvent mfa?
Prasanna Malaiyandi: Yep.
How much damage can they
W. Curtis Preston: do you have against?
Yeah, how much damage they can do.
Can you undo any of that damage, et cetera, et cetera, et cetera.
I know that if we go back in time, let's say two years, I don't think
Druva had good answers to that question.
Right.
I remember being in those meetings and saying, Hey, we need to address this.
We have addressed that.
I think we now have really good answers to those questions.
Um, but does your, does you know, if you're not using Druva, does your vendor
have good answer to those questions?
I don't know.
Right.
So, yeah, so come up with worst case scenarios and that one.
Of compromised admin credentials, whether you're talking a service or
so, like for example, I know that like Druva has data lock, you know,
rubric and cohesive have data locks.
I know that with them, if you, if you compromise, uh, credentials and
you log into them and try to delete, if you, if you enable datalock,
you will not be able to do so.
Right.
So, , what is your vendor's answer to that question?
Um, and um, and then also ask 'em about bit rot.
What are they doing about bit rot?
If they're using object storage, I feel better cuz object storage
will automatically detect bit rot happening underneath.
Uh, cuz it will change the hash and somebody will be like, Hey, well not
somebody, a program will figure it out.
Um, I think Biro is rare.
Just for the record.
I think it's rare.
Um,
Prasanna Malaiyandi: It's rare.
W. Curtis Preston: bit error.
Right.
But,
Prasanna Malaiyandi: But it's still bad
W. Curtis Preston: not, not a problem.
Prasanna Malaiyandi: Yeah.
And the problem is when it's silent, that's the worst.
W. Curtis Preston: Exactly.
That's the worst.
All right.
Well enough about protecting backup data.
You got any final thoughts?
Prasanna Malaiyandi: Well, I think we covered it all.
I really like virtual air gaps, but that's just me
W. Curtis Preston: I like the old school air gaps, but it's really
hard to do those these days.
All right.
Well thanks for listening to us.
Be sure to subscribe so that you can restore it all.