In case you haven't heard, ransomware is after your backups.
This encore episode is one of our most popular episodes from the last few years,
and it covers exactly how to protect your backups from ransomware, why attackers
are exfiltrating the data before they even try to encrypt it, uh, what real
immutability looks like, uh, who should have root access, and the questions that
you need to ask your backup vendor before you give them, uh, your data and trust
them with your last line of defense.
If this is your first time watching me or listening to me, I'm W.
Curtis Preston, AKA Mr. Backup.
I've been obsessing about backup and recovery for over 30 years now.
If that's your bag, then I'm your guy.
You're not gonna find anyone that cares about it more than me.
Ever since 1993, when I had to tell my boss that there were no backups of
the database that we had just lost.
Now, I've written five O'Reilly books, a blog, and a podcast.
Here, we turn unappreciated admins into cyber recovery heroes.
This is the Backup Wrap-Up
Hi, and welcome to Backup Central's podcast.
I'm your host, W. Curtis Preston,
Oh.
a, Mr. Backup, and I with me, a guy whose only major flaw is that he
won't go see scary movies with me.
Prasanna Malaiyandi.
How's it going?
Prasanna.
I am good, Curtis.
I, yeah, I'm not a big fan of scary movies, especially on a large screen.
Um, I don't like, I know some people like you love it,
ah,
but it's just I don't find joy in watching a scary movie.
I do remember in college watching the Ring.
On a big screen.
That movie messed with my head.
I remember coming home and asking someone to take my picture.
And saying, is there a thing in the picture?
You know, I was scared.
I was scared to take the picture.
Yeah.
was, that was a fun movie.
Um, it really kind of messed
Yeah.
And you just saw a scary movie just recently.
Yeah.
It, it,
it's not a
know if I'd called
Scarer.
Yeah.
per se, it's more like a thriller.
Um, It's called the Menu and it stars.
Uh, Ralph finds, um, a k a Voldemort, and uh, Anya, the
Hello.
Queens gambit.
Yeah.
Um, and some other, other character actors that you may be familiar with.
Um, uh, it, it's, it's a movie that I, I enjoyed the movie.
I will just say this, um, trigger alert for those who are bothered by such things.
It does contain suicide.
Um, having said that, within the context of the show, it, it was a
fits in.
part of the storyline.
Mm-hmm.
Um, and um, having said that, it was a really unique storyline.
That's a commentary on Modern Day Society.
You know, there's a class thing, there's a, there's
Yeah.
yeah.
It's not often that you come out of a movie and be like,
oh my God, that was so good.
yeah,
Like, I don't think,
enjoyed
yeah.
like, oh, that was, I thought the ending.
I don't know.
think I would've done something slightly differently within the ending, but the um,
That is why you are not the director, Curtis.
We're the writer.
Yeah, that's why I'm not the director or the writer.
I, I was not the only one, by the way.
I read some reviews and stuff, and a lot of people felt the ending
was somewhat anti cli climactic.
But it was one of those movies where you're like, what?
What is over?
Like, it's one of those kind of movies, but everything all the
way up to that point I thought, I thought was really, really good.
All the actors are really good.
Um,
never seen a Bollywood movie?
yeah, what's
Where it's like anti-climactic at the very end.
It's like they literally, a lot of the movies boiled the ending down
to like the last like five minutes.
interesting.
Um,
Yeah.
Shrek, oh, you would be proud.
Shrek.
And I watched while you were watching this movie, Shrek and
I watched a Bollywood movie.
Uhhuh,
On Netflix and it was also suspenseful
mm-hmm.
of a comedy thriller ish, but I would say that they basically
wrapped up all the entire movie in the last like five minutes.
And then at the end, Shraddha and I were like, what did we just watch?
Interesting.
because we, because it left so many things open at the end and we're
like, did we just watch all of that?
Just for that last closing part?
That's funny.
it's like two and a half hours also, so,
Interesting thing about this movie is, you know, how the movie's gonna
end about halfway into the movie.
Hmm.
Like they tell you how the movie's gonna end.
it's just how it gets there.
It, it, it's just was, was
Yeah.
Nice.
Um, so I recommend it.
Um, and, uh, you know, the, the
are you and
saga continues.
yeah.
Are you and Jeff gonna talk about it on your next, on the other podcast,
know, he keeps threatening to record another episode and then he doesn't do it.
But I'll , I'll take
I think Curtis is calling you out here.
Jeff and I actually recorded a new episode this week, so look for it at,
The Things That Entertain Us podcast, wherever you listen to podcasts.
Today we're going, we're continuing in our backup to basic series, and
we're covering, we're covering the book, modern Data Protection on Hold.
those of you watching this on backup central.com where you can watch the video
of Prasanna and I waxing philosophical.
Um, and, uh, you know, what we're doing is we're sort of going through the book
just discussing, you know, some important topics that I, that I found in the book.
and this next one is about protecting backup and archive data, which
think we can all agree is important.
If you would like to download a free e-book copy of Modern Data Protection,
you can do so at druva.com slash ebook.
Uh, we have to protect it from a couple of different things, right?
We have to, mainly what I'm talking about in here is protecting
it against attacks, right?
both attacks like a, um, like a disaster type type attack.
But, but mainly what I'm talking about is protecting it from cyber attacks,
Um, I'm gonna have these same conversations over and over, so
sometimes I forget what we say here versus what I say other places, but
it, it is a well acknowledged fact that some of the largest of the ransomware
bad actors are specifically targeting backup systems in order to do the.
The extortion style of ransomware attack, and let me specify what I'm saying there.
So, you know, a, a bunch of companies started realizing that, well, if I just
get a decent backup in recovery and disaster recovery system, I can say, you
know, pound sand the ransomware folks.
Mm-hmm.
the ransomware folks then said, oh, well, we'll show you.
And so they developed this, this exfiltration style, right.
and the um, The idea there is that they find databases and file
systems and spreadsheets and whatnot that contains sensitive data.
That could be two different things.
That could be your company secrets.
It could be your company plans.
It could be your 11 herbs and spices or your, your, um, You know, your
corporate plans for the next year, you know, um, but it could also be
secrets that are like embarrassing.
Uh, I think the best example of that, and I know we've talked
about it, is the Sony attack
This is the one where someone broke in to the movie studios and got a
bunch of emails talking about scripts and actors and all the rest of that.
and very, some, some, and not very
Yeah.
flattering terms.
Yeah.
So it could be that, right?
the point is, there is no way to protect against that once the data's out.
Yep.
It's not like the other attack.
If you get the, the old school attack, if you had a good backup system, you're good.
It doesn't matter how good of a backup system you have, if the
data is exfiltrated, you're gonna be forced to make some major, uh,
Yep.
right?
So, um, what, what these, uh, ransomware groups have started realizing is,
they, uh, instead the, the people are starting to harden
their primary environments.
Mm-hmm.
And so, but one, you know, you, you, you've heard me talk, did
I say one thing that has never changed in backup and recovery in
the world of backup and recovery?
Do you, do you know what it is?
I often say like
It's
that has never changed,
that the junior person is always responsible for backups
and recovery.
that is technically a cause of the thing the thing that
No one who cares about
no one wants to do the backups, right?
one wants to raise their hand and say that they want to be the backup guy.
So why does that matter?
Well, it's because it means that, that it ends up being an an or and ignored system.
Yep.
and as a result, it that it gets ignored by everyone, including
the cybersecurity folks.
Right?
They're like, ah, that's just a systematic in the corner.
It's fine.
We don't care about it except it contains all of our data.
Yeah.
And depending on how, how you handle, things like encryption,
they may be able to unen encrypt.
Said backups and then, you know, get the keys to the kingdom.
Yeah, and like you were saying earlier, Curtis, the fact that a lot
of people focus on production systems harden them, but they may not always
be considering these backup systems.
Right, right, right.
Easy target.
And so you need to.
not be such an easy target, right?
Yeah.
the old joke about the guy, two guys crossing a field and they see,
they see a bull and the guy says, you know, can you outrun that bull?
He said, Nope, I don't have to.
What are you talking about?
All I have to do is outrun you,
You
So, so you just want to be less of an interesting target
than the person next to you.
So the, the first thing that we talk about in terms of protecting
backup data is encryption,
Yeah, which
uh,
everyone should be doing.
Like if no, if you aren't doing encrypting your backups, I don't
know what's wrong with you.
Like even if it's not encrypted, honestly,
like that should be your default.
Persona.
no, because encryption isn't some heavyweight thing that it used to be.
Right.
You don't have to go physically encrypt your data as it comes out.
You could do disc level encryption, right?
There's so.
Ways you could leverage encryption to fit into your environment.
It should be a no-brainer that all your data at rest is encrypted.
And in a lot of companies, right?
If you wanna fit, be sort of like, uh, what is it?
If you wanna look at HIPAA com, uh, readiness or other things, right?
Other regulatory obligations.
A lot of 'em say All your data at at rest should be encrypt.
Mm-hmm.
. Well, I, um, do think, I do think backup page backup data should be
encrypted for this very reason.
Right?
And I'll give you different reasons for the different things, right?
So if it's, if you got a backup server on-prem and you're storing
your backups on a. File system, you know, an NFS mounted file system.
Then they're unencrypted, you know
Everyone has access.
Yeah.
Everyone has access to those backups and a lot of these PR backup products they offer
to, to, to, so some of the backup products store the backups, not in a backup format.
They're in native format, so you could just mount a disc, right?
Yeah.
backup products, put it in a backup format.
but they offer an ability to read that backups like it,
it's, it's for DR purposes.
So, so if you're not encrypting them, then you're just, you're
just asking for trouble.
Yeah.
the disc side.
The second side will be the tape side.
So if you are, and believe it or not, many people are still making backup
tapes and they're making 'em primarily, I think for offsite, uh, storage.
Yep.
And so, And this is the, this is the one where I'm like, if you are ma, if you are
putting your company's like crown jewels onto a little, you know, plastic tape and
then you're handing it to a man in a van
Yep.
you are not at this point encrypting that data,
you know, professional malfeasance at this
point.
and I wonder, it's almost as if that should be like a default, you know,
like you can't write data out without encrypting it, but I guess you have to
deal with key management and all the rest of that, and so, , do you have
flexibility to shoot yourself in the foot?
yeah, exactly.
but I will also say this encryption is only good, is only as good as the
authentication and authorization and key management system that is connected to it.
Yep.
And if you can easily defeat.
That, uh, system then, you know, and, and, and I mentioned, so for example,
I know we mentioned it on the podcast, please do not make your backup server
part of your, uh, active directory.
What,
Domain.
it,
Domain.
domain.
Yeah.
Um, Just please don't do that.
so the next term is one that we can debate.
No
I, honestly, I don't even need you.
I can debate it myself, right?
It's one of those where I go back and forth, you know, and that's the term air.
What is like air gap?
Curtis?
well, well, there
Traditionally?
Yeah.
It's, and it is where it, it is from once we get the term.
the idea was to put a gap of error between the protection copy and the protected
Yeah.
And for
you're dealing with plumbing right now, since you're dealing
with plumbing right now, right?
You have an air gap normally, right?
When you think about your dishwasher
Now you made me think about the dishwasher that
has made my
or,
fine.
Fine.
Or a toilet.
How about that?
that's actually technically a water gap, but
Don't, you're killing me.
Don't be, don't be bringing plumbing into this.
anyway, yeah.
So you, you had to separate the, it meant, again, it meant the, it
meant the man in the van, right?
You, you, you, you put some tapes in a box and you handed that box to a, you
know, a man or a woman, and then they got into a van and they took it someplace.
And in order to get to that thing, needed to physically, Um, access it.
Right?
Yep.
that if you were a hacker and you wanted to get to it, it, it
would be virtually impossible.
Not completely impossible.
There's no,
It's
cuz
penetration rather than cyber.
yeah, remember for example, we would, uh, once a year at an
unannounced time, we would attempt to circumvent the security of.
Uh, storage fender
Mm.
we would, uh, use really good like liars and whatnot.
And more specifically, we would use someone like me that they
knew, they knew the person.
And so they would go over there and they would, and we would just have
this really, really like elaborate.
Scenario
yeah.
that basically would want, would want the, it would make the person
that works there want to leave us alone in the vault with the tapes.
And at least once we were able to do that,
Hmm.
Was a person go, or did the person have to go through training, I guess.
Yeah, I don't, I, you know, honestly, I was so long ago, I don't remember
what happened to that, to that person who left us alone in the vault.
But, so that's what I'm saying.
It's not, it, it was never 100%.
Right.
Um, the, um, you've seen the movies, you
Yep.
I've seen Oceans 11 and 12 and 13.
The um, but the problem is most of us don't use tape.
Mm-hmm.
And most of us, or many of us don't even use a data center anymore.
Yep.
So this idea of putting a, an actual air gap between A and B is quite difficult.
And
Yep.
we've sort of adopted or adapted
the, you know, the virtual air gap.
Right.
Um, the, um, Uh, so this is in as many ways as we can.
I don't know.
You want to talk about
Yeah, and I think we talked about this on the episode about backups,
or maybe it was restores, but.
It's even though they are available online, right, that second copy,
so it's not the same as having that tape in a vault somewhere.
You wanna make that, make it as close to that as possible.
And we totally get that.
You still have to have connectivity, right?
It's still gonna be up, but can you lock it down?
Can you basically make that target as inaccessible as possible?
Uh, to prevent it from being attacked by say, someone, um, gaining access
to your backup source, right?
And now they have access to your vault.
Uh, some of the examples are if you're backing up in the cloud, right, put
it in a separate account that no one has access to other than whatever
is transferring the data, right?
Make it in a different region.
So it's not easily accessible.
And there are many, many, many other things you could do as well.
But those are just some of the example.
Yeah, I like if we're talking cloud, I like the idea of putting it in a
different region and a different account.
Right.
Um, because we're not just talking about hacking.
We're also, we also need to think about like, Disaster recovery and things
Yep.
right?
So again, O V H Cloud, we don't want to have that happen to us.
So we want to make sure that a copy of the backup data is, is well far away
from the thing that it's protecting.
And yeah, separate account a much more limited, an account
with much more limited access,
Yep.
and you can.
Um, you can consider using immutable storage, but we're
gonna talk about that later.
But essentially, this is the, we, we separated as much as This
Yep.
um, you know, one of your previous employers data domain, right?
And, and, and, All of those things, they would always talk about
replicating to another data domain.
And I always, I always remember thinking, well, if I'm replicating to another data
domain, it's essentially like in order to do that, I gotta have it in the same
So,
like land.
Right?
Or, or at
or Van.
like it's in the land.
Yeah.
Well,
Um,
that's why they actually introduced a feature, specifically talking
about data domain, right?
Their cyber recovery solution.
Mm-hmm.
that allowed for writing to a lockdown data domain that didn't
have access from anywhere else.
And going back to the point you were talking about, Curtis, okay, is it
on the network and I can connect to it then, doesn't that make it open?
And so what it would actually do is shut down network ports right
outside of your backup windows, such that it's not available, or
at least your attack surface is.
Could you delete the backups?
Was it.
Could you delete the backups once you send it to it via the backup interface?
Typically no.
Okay.
Because that, cuz that would be, that would be an important
Yeah.
It would also replicate like your backup environment, right?
So you'd have your own backup server in that vault, if you will.
So there's no connectivity to anything outside, because if there
is, then you're just open to risk.
Right, right.
Um, so just like, so I've got a list here on virtual air gap here.
talk about disabling or impairing R D P. Why does that matter?
Ah, so many ransomware issues have happened.
Rdp, which
is Remote Desktop
the way?
Protocol,
you,
right?
Which is how most people connect remotely to a Windows client.
Um, most people forget to turn it off or they leave it on because it's just easier.
They don't need to physically connect to it.
Uh, The only downside is it's a big attack surface, and there are
a lot of exploits using R D P.
And so when you enable R D P, you're letting yourself or leaving yourself
open to hackers and other exploits coming in, which could then move
laterally across your network and take out other pieces of your environment.
Yeah, there is a way to automate sort of disabling and enabling R
D P, um, like on a large scale.
Um, I also talk about putting it in a different operating system, right?
Not
Yep.
operating system for your backups in your primary,
Yep.
that's possible.
I know that there's a lot of big window shops and they use windows.
Uh, backup servers.
I don't think that's a good idea.
Um, but you know, Um, and by the way, I, I forgot to throw out our disclaimer.
I'll throw out, um, Prasanna.
Prasanna.
And I work for different companies.
I work for Druva.
He works for Zoom.
this is not a podcast of either company.
This is an independent podcast and the opinions that you hear are ours.
And, uh, be sure to rate us@ratethispodcast.com slash restore.
um, if you'd like to join the conversation, you want to talk.
What's going on in your world?
Uh, you know, whether you're, uh, you know, an end user.
If you're an end user, we really want to have you, you know, real
people that actually do backups out there in the trenches.
Weed.
You're,
Do.
Do you miss it, Curtis
No
I do, you know, as soon as I think about, um, Back when I used to be the
person in the trenches, like when I was a consultant, and I, and I, I flashback
to this one time where we were completely redoing the backups of, I'll just say
a large television station, like, like a television station, and remember.
I remember that time when the, when the, the, the of it basically said,
going home until this is done.
He literally, I mean, there's a long story following up to this, but he basically,
um, held us hostage in the data center said, no one's going home until this,
this thing you said is gonna happen.
You said it's gonna happen.
I'm like, yeah, but it's not gonna finish until.
10 30 at night, like, yeah, so we're not going home until it's done.
And I'm like, okay, well this is kidnapping, but whatever.
Um, you know, uh, so I think about stuff like that, right?
Yeah.
moments of terror.
But yeah, I, you know, I wanna live vicariously through those who
have been through those moments of
Yeah.
Well, and it's also interesting to learn like what the more recent challenges
are in environments, because otherwise
how do you
Yeah,
otherwise, how do you build great products, you know, or build
products to solve
exactly.
So what, what I was thinking about saying, which is what reminded me to do the
disclaimer, is just that, um, you know, one way to have a different operating
system is to use a SaaS provider.
Druva is not the only one, but there are SaaS providers
that are cloud native or use.
Non windows, uh, tools um, you know, you do that.
Also, you, you gain the second thing that I list here, which
is about separating the storage.
Yeah,
don't, um, you know, we talked, we talked about that already,
the, these replication, right?
Uh, but um,
but it's also
you're just.
use nfs.
Yeah, exactly.
Yes.
Don't use NFS as a way to back up to the server use.
There are ways to back up to, for example, a data domain box in other
boxes exposing the backups via nfs.
Uh, I also talk about using object storage.
I'm a huge fan of using object storage.
Now, some of you go, well, object storage is too slow, to which I
say, then you're using it wrong.
Yep.
Right.
Yep.
Druva uses object storage.
All our backups are on object storage, and we are not too slow.
In fact, we get into competitions all the time with these big
on-prem companies and we win.
And here we are, we're a copy in the cloud and they're an on-prem
appliance, and we win that restore test.
Object storage is not too stor slow, but if you, if you treat it like file
system storage, I think then it is,
And it's a challenge.
Yeah.
all your backups in one object, Tonight's, tonight's backup of
server X is an image, and that image becomes an object on, uh, s3.
Yeah.
Then it's gonna be slow, right?
Um, we talk about using aut storage.
We're gonna come back to that in a minute.
Um, that, that's an option that we talked
And then,
S3
yeah, and then your.
And then your favorite topic is using tape.
I know, Curtis, you always like to throw out the disclaimer.
Actually, I should thank you because before this podcast,
I knew very little about tape.
Now I know just a bit more about tape,
yeah,
the number of experts who've come on in talking to you.
But yeah, it's
not
there's no, there's no beating the tape when it comes.
That's an actual air gap.
Yep.
right?
If, if you, if you want a real air gap, I'm, I know there's challenges with
it, you know, uh, but there is, you know, there is, there is an air gap
Yep.
Uh, and I already talked about using a backup service.
So let's talk about immutability.
Yep.
What is immutability?
Is it a made up word?
It sounds a little like a made up word,
you know, this is another one of those things where we have to go back in time.
Historically immutability wa it, it, it was like a legal term that you needed
to be able to prove that the thing you were presenting in court had not changed.
It was immutable.
That's all, you know, immutable just means cannot be changed.
Yep.
when you're, when you.
uh, you pres, you, you know, you have to preserve chain of custody and you
have to, you have to be able to say, or it's really helpful if you're able
to say, this email that I'm showing you is exactly the same email as the email
that we got five years ago from Steve.
Yep, and nothing changed and no one had the ability to change it yet.
That's where we used to talk about immutability.
Now we talk a lot about it in terms of cyber attacks.
And I would add to that, we also talk about it in terms of things like bit rot.
We've talked about bit rot on this podcast where this is
silent data corruption, right?
That bits just flip underneath magnetic storage devices.
It's just the way they work.
and the, you know, and if, if you've got, if you've got a bunch of petabytes
of data, you got flipped bits in there.
It's just a matter of, it's just
when you're gonna run across it, right?
yeah.
Yeah.
Um,
so, when we talk about immutability, we say, well, we, if somebody
says they have immutable.
Backups or immutable storage.
What you're saying is stuff that gets put here can't be changed.
Mm-hmm.
here's the thing.
Depending on how you look at it, it's complete nonsense.
Because everything's just software.
Anything can be cheap.
is changeable.
Yeah.
me a torch, I'll change that.
S o b, right?
Give me phy.
So All bets are off.
Everyone knows that, um, if it's something on a server that you can, that, that,
that is in your data center, that also, in my opinion, is not really immutable.
Um, because if someone has root, uh, on that server, they can wipe the server.
it's even built, like, you know, when we talk about immutable storage on
Lennox, it's not really that immutable.
It's it's immutable ish.
So, so why do I say, like, why do I talk about it if it's bs?
Well, here's the point.
Nothing was ever a hundred percent immutable,
Even that document right, that you were talking about.
Yeah, exactly right.
As long as like if, if, if you put something on a a, a, a, right once
cd, uh, optical platter, right?
A worm tape, it's immutable to a point.
If you have physical access, it's no longer immutable.
Yeah.
You could, you could say from a legal standpoint, you could later, if you have
the same disc and you've preserved chain of custody, and you could say, we can
show that this disc was not destroyed,
Yeah.
we can show via these check sums and whatnot, that the stuff that we're giving
you is the stuff that we have before.
Saying that you've got a storage system that can't ever, ever, ever be changed.
Um, I would argue it's just nonsense.
Right.
So it's, it's a, it's a, it's a, it's a degree.
It's a what, what did we call it?
Like a,
Spectrum.
like a spectrum.
Thank you.
I
Yeah.
was a word in there.
And even
prism is what was coming to mind.
and even for those storage systems that have immutability, right,
that are used for like financial records and everything else,
Mm-hmm.
of that is certified by the storage vendors saying they have all the checks
in place and all the rest, right?
In the end it's all just software, right?
So they've built in the, into the software, those checks to make
sure that data cannot be deleted,
right.
is present.
And the, and you know, we're doing all this, like we're gonna, I'm
slamming it and then I'm gonna talk about how important it's . But the
other is, um, Uh, I'd say the, one that I like the best right now is
like the, the object lock and s3.
And what I like about that is because it's, it is at least
physically separated from.
You.
Now, that doesn't mean that somebody can't go crazy in the AWS data
center and, and blow it up, but even that, it's built into that.
So it, it would have to be like, it would have to be a really concerted
attack to be able to attack multiple locations of S3 to be able to do damage.
it's, it's not only
the
multiple
is
location.
as what's
Yeah.
It's not just multiple locations of s3, but also they have to tack your primary
site as well, so everything needs to be coordinated across multiple vendors and
corporations, which will probably have their own security practices, et cetera.
this is, this is that, this is that, uh, spectrum, right?
Yeah.
I'd put, I'd put that and write, you know, worm tape, write once,
read, write once, read many tape.
Um, and then, and, and optical plat.
I put that on one end of the spectrum.
on the other put at the end of the spectrum is I would have unencrypted
backup stored on a NFS mounted storage system behind a Windows backup server.
Yep.
that's your spectrum.
So we wanna be closer to this end than that
Yep.
Right?
Um,
But there are some
ever
downsides.
Yeah.
And there are some downsides to immutability too, right?
Uh, you can't, like once the data gets written, you can't delete it
before that time period typically.
So there are some challenges as well.
So, um,
so what we need to do, Is we need to, I, I think
I.
multiple things there is that, um, we need to protect against access.
We need to protect against, you know, change deletion, corruption.
Um, uh, was I, what was I talking about?
Um, bit rot,
Yep.
So there are vendors out there.
That, they're like, well, we have a, append only file system, and, we,
and we have data lock the backups.
And so they say, we're immutable.
and I'm like, okay.
And, and they say because they're saying that like, you can't encrypt the backups
with a, with a ransomware attack.
Yep.
and I, and I go, that's good.
That's farther, that's, that's closer to the other one than the other ones.
But it's like, well, what about other things?
Right.
What about other types of attacks that attack the operating system itself?
Right.
You gain privileged access to that server,
yeah.
And I wanna clarify though, what you're talking about is someone
who's written their own file system or is leveraging a file
system on top of a standard server.
Yes.
Not a storage appliance, because I think that's a little bit more,
that's like further along in the spectrum, I would say like a
Yeah,
storage appliance entire too.
So,
yeah.
So, So, so, we got a couple different types of backup servers here, right?
So
Yeah.
the purpose-built backup appliance of various
shades, shades flavors.
Sure.
Um, and I would put them.
More immutable, right?
So immutable is supposed to be a binary
Yeah.
I don't see it as that, right?
So, I put it more immutable than I bought a box.
I installed Windows or Linux, and I install my favorite backups or
Yeah,
Why?
Why, why do you feel that way?
I agree with you.
Why do you feel that way?
I feel that way because when it comes to that appliance, typically
there's more things locked down.
There's less configuration options, right?
It's kind of purpose built for that reason.
Versus when you're rolling your own, you have to worry
about all those dependencies.
What flavor of the OS are you taking?
Are you running through all the security patches?
Is it, are there any nuances in the way it's being deployed
today that leads to security vulnerabilities and things like that?
Yeah, exactly.
And, and when you update those appliances, you update an
image which updates everything.
Yep.
if I have a a box, a Windows box, I gotta update Windows or Linux.
And then the backup
software.
new security, then you gotta update the backup
Yep.
So, yeah.
And these are, and I would say that's, so I would say that the
appliances are more immutable than.
Than the build drill in box.
Yep.
and the, uh, but I, but I'm just gonna say that if you, if we're
talking physical access, I still, I'm gonna put a service like S3 or a
service like Druva that's up in the cloud, farther down the immutability
spectrum than a survey, than a server that is sitting in your data center.
Yep.
And I think it's important because everyone is now doing virtualized, right?
It doesn't matter if it's a physical server or a virtual server.
Anything that's running on your infrastructure or in your
infrastructure is less secure in that immutability spectrum.
Yeah.
the, uh, yeah, I put in the book a lot of things are mislabeled, immutable,
uh, I just, again, it's a spectrum, but.
If, if all you're doing is writing backups to a Linux file
system with the immutability flag turned on, that is not immutable.
I mean, it's, it is more immutable than not But if I, but if someone
with root can and, and you gotta have someone with root, right?
Yep.
if a person with root can go in and unset all those immutability flag,
It don't matter.
That's, I don't know.
That's just not, it's be, it's better than nothing.
Yep.
Right?
So I, I don't want to be the, the, you know, is the enemy of good or
whatever, but I don't need perfect.
And so I don't wanna say that that's crap, but I, I think it's, it's not as immutable
as those appliances that we talked about.
And I still think that a service nobody gets access to your servers
is more immutable than that.
Yeah.
and I still want everybody to back up everything to Tape
Oh, Curtis, you and
your tapes.
at, I work at a tapeless backup company.
Um, so
here's
Go
one thing I wanna ask, and I know we've covered this on prior podcasts.
What are things that an admin can do to understand where on that spectrum,
when someone says, when a vendor says immutability, what are questions
that they should be asking to be able
to determine?
Who has root or admin, right?
Determined.
Who has that and what controls are placed over that?
So, um, if it's an appliance, so like I, I know of like one company.
They do have root, they have a they have the root password on your
system, or they have the password to an account that has a u i D of zero
Yep.
of you.
Um, is essentially the same thing, but in order to log into that account, they
can't log into that account remotely.
You have to.
You have to do an SSH
yeah.
stuff, right?
You have to open up a door for them to log in.
Um, you know, what kind of protections are put against that.
Uh, if the answer is it's just a local box and you're the one that has route,
or there's unprotected route access from someone, um, I just, I worry well.
you know, so if, if, if you or anyone in your ministry in your place has root, not
very immutable, it's better than nothing,
Yep.
not very immutable, right?
you know root, like if, if, if you normally never log into
the system as root, right?
No one in your, you only at you, you go to a ui, you log it as you, right?
Um, and.
There isn't direct route access by anyone.
And the only way you can get route access is you can, um, you do the SSH
tunnel thing that's more immutable.
I like that better.
Um, you still, but again, if you don't know this already, you
still have to do physical security
Yeah.
Yep.
Exactly.
do every other box.
Right.
Um, and then, You know, you ask a lot of que just ask a lot of questions.
What happens if so, for example, I'm currently wondering because I haven't
found a good answer online, I'm currently wondering what happens with Amazon S3
object lock if I delete my account.
I don't think they allow you to.
It depends on what type of object lock you're using, because there are
two.
what, I know what you're saying, right.
Um, but if I'm u is it the compliance mode?
It's the more,
Yeah.
yeah.
So assuming I'm using compliance
Oh wait.
Compliance is R L C R, lg.
I think compliance is less in governance is more, or is it the
other way around?
it
Yeah.
Whichever way
it's.
think it's the other way around.
But anyway, so I'm using the more strict one.
and my credit card stops working.
Right.
It's not like they're gonna keep my storage forever.
You, when you say they won't let you delete it.
Well, I don't have to delete it.
I'm just gonna have my credit card not work anymore.
Yeah.
Right.
What
like you said, it's not protected in all scenarios, right?
Right.
I don't know what the answer is.
I am curious.
I actually bought, I, I keep forgetting to do this, but I want to
go create an Amazon account separate.
Mm-hmm.
Put some object lock stuff in there.
Not a whole lot, just like days or something,
Yeah.
like a couple of gigabytes or something, and then go delete my account and see what
And See what happens.
let me delete the account.
Yeah.
I did read something somewhere that, that there is this like 30, 60 day timeframe.
Hmm.
So again, that's still better because I'm assuming that if you're using Amazon S3
You don't want it to go by.
or any of the other folks you're gonna, you're gonna know that
somebody deleted your account
Yeah.
Right.
Um, and I know what I would want you to ask is what happens if, what happens
if someone gets through someone?
So I was just seeing a thing that said that it's like more than half of hacks
are through compromised credentials.
So what happens if someone, you know, you're a backup service?
What happens if someone ahold of my admin credentials and is able to circumvent mfa?
Yep.
What
damage can they
do
you have
Yeah.
Yeah, how much damage they can do.
Can you undo any of that damage, et cetera, et cetera, et cetera.
I know that go back in time, let's say two years, I don't think Druva
had good answers to that question.
Yeah.
Right.
I remember being in those meetings and saying, Hey, we need to address this.
We have addressed that.
I think we now have really good answers to those questions.
but does your, does you know, if you're not using Druva,
does your vendor have good
Yeah.
questions?
I don't know.
Right.
So, yeah, so come up with worst case scenarios and that one.
compromised admin credentials, whether you're talking a service or
so, like for example, I know that like Druva has data lock, you know,
rubric and cohesive have data locks.
I know that with them, if you, if you compromise, uh, credentials
and you log into them and try to delete, if you, if you enable
datalock, you will not be able to do
Mm-hmm.
So, , what is your vendor's answer to that question?
Um, and um, and then also ask 'em about bit rot.
What are they doing about bit rot?
If they're using object storage, I feel better object storage will automatically
detect bit rot happening underneath.
Yeah.
cuz it will change the hash and somebody will be like, Hey, well not
somebody, a program will figure it out.
Um, I think Biro is rare.
Just for the record.
I think it's rare.
Um,
It's rare.
bit error.
Yeah.
But,
But it's still bad
when it happens.
Yeah.
And the problem is when it's silent, that's the worst.
Exactly.
That's the worst.
All right.
Well enough about protecting backup data.
You got any final thoughts?
Well, I think we covered it all.
I really like virtual air gaps, but that's just me
I like the old school air gaps, but it's really hard to do those these days.
All right.
Well thanks for listening to us.
The Backup Wrap-Up is written, recorded, and produced by me, W. Curtis Preston.
If you need backup or DR consulting, content generation, or expert witness
work, check out backupcentral.com.
You can also find links for my O'Reilly books on the same website.
Remember, this is an independent podcast, and any opinions that
you hear are those of the speaker and not necessarily an employer.
Thanks for listening