W. Curtis Preston: Immutable backups are a critical defense against the rising
tide of cyber threats, and at a time when ransomware attacks are targeting not just
your primary data, but your backups too.
Understanding immutability is more critical than ever.
We'll explore what makes a backup truly immutable.
The spectrum of immutability options available and why.
This matters so much in your data protection strategy.
From tape backups to cutting edge cloud solutions, we'll cover it all.
Are you immutable?
Backups actually immutable.
Let's find out.
I.
By the way, if you don't know who I am, I'm w Curtis Preston, also known as Mr.
Backup.
I've been passionate about this topic for over 30 years, ever since.
I had to tell my boss that there were no backups of the
database that we had just lost.
I don't want that to ever happen to me again.
I don't want it to happen to you, and that's why we do this.
On this podcast, we turn unappreciated backup admins into Cyber recovery Heroes.
This is the backup wrap up.
Welcome to the show.
I'm your host, w Curtis Preston, AKA, Mr.
Backup, and I have with me my SMB backup Difficulty.
Sympathizer Prasanna Malaiyandi.
How's it going?
Persona.
Prasanna Malaiyandi: I, I've learned more about dis space backups than, and tape
based backups in the last, like three weeks from you than any time before.
So yeah, how are you doing?
W. Curtis Preston: oh, I just like, if I could get like a, like if I
could have a wall here, just, you know, bang my head into the wall.
That's sort of how I feel like I'm doing.
I do
Prasanna Malaiyandi: the problem is that you left it for 20 years
and now you're coming back to it.
And so
W. Curtis Preston: That's not the problem that that implies that I am old
or incompetent or something like that.
That is not the problem.
Prasanna Malaiyandi: me going back.
It's like me going back and coding.
I could do it, but I'm not gonna be efficient.
W. Curtis Preston: The, the, the, um.
It, it, it's been, it's been really weird, right?
Because, you know, you change something and you change this.
You're like, well, let's try this.
And then you try that and you're like, that just doesn't work.
That just, you know, like the, the, the, and the really weird thing that I.
Has been pla plaguing me over the last, like, I don't know, several days as
I've been trying to make a major change to this backup configuration is that
the backup starts out, you know, I'm running like 20 backups at the same time.
Backups start out and it, and it's running like 500 megabytes per second.
This is great.
This is what I want.
And then about a half hour later he goes,
you're like.
Prasanna Malaiyandi: To a slow drip.
W. Curtis Preston: happened?
This is not, this is not tape, this is, you know, this is not
like, there's not some weird tape, you know, shoe shining thing.
This is disc two, disc via a server in the middle.
Prasanna Malaiyandi: It is probably some random option somewhere
W. Curtis Preston: random,
Prasanna Malaiyandi: haven't done this.
W. Curtis Preston: data buffers thing.
Oh, oh, oh.
You're, you wanna do this on a Thursday, you need to do the, you
need to do the setting and, uh.
Prasanna Malaiyandi: what I said, that because you haven't done this
day in to day out, you don't know.
It's like you lose touch of these or you don't know
W. Curtis Preston: Yeah.
And I, I will say that when I was, when I was, when I was last doing this really in
earnest, this backups were relatively new.
Um, that, you know, that really dates me, but that is, that is the case, but.
So, yeah.
And I just, you know, and I, I've been thankful for you to talk to you.
It's like, you know, so here's the thing, you know, I got this, I got
this setting, I got that setting.
I've been playing with jumbo frames, I've been playing with,
with, uh, buffer settings.
I've been playing with all these different things.
And you know, and you're like, what about the thing?
You know, what about this setting?
Oh, man.
Prasanna Malaiyandi: And you know what?
In the end, once you figure out what's going on, it's probably
just gonna be some simple thing,
W. Curtis Preston: Mm-hmm.
Prasanna Malaiyandi: right?
That you just didn't know existed.
And you're gonna fix it and be like, oh yeah, now it's like, boom.
W. Curtis Preston: Yeah.
And then it's gonna run like a thousand megabytes per second, and then,
we'll, everything will be amazing.
Um, so, we're gonna talk about some things that are really new in terms of when, when
I first started with backup, we really didn't talk about this concept when I, you
know, was doing backup back in the day.
There's a big thing that happened in the last.
Two decades.
That is one of the biggest reasons why we didn't use to talk about
cybersecurity and backups, and now we talk about it all the time.
Do you know what that is?
Persona?
Prasanna Malaiyandi: No,
W. Curtis Preston: It's
one change.
No.
One seismic change in backups in the last 20 years.
Prasanna Malaiyandi: Dis space backups.
W. Curtis Preston: Yes, exactly right.
Tapes weren't hackable.
Tapes were offline by design.
Right?
Ooh, that rhymes anytime.
Um, yeah.
So tapes were, tapes were not sitting there just files ready
to be deleted or encrypted.
Tapes were inherently hard, even if they were online, even if, if they were in a
tape library, they were hard to get to.
And then.
Uh, over, you know, and, and if they were offline, if they were sitting,
you know, in a, a tape vaulting vendor, then they were really hard to get to.
But the, um, what happened when we went to dis backups, we
basically made backups hackable.
And so that, and, and I don't think, in fact, I know, I know.
We really didn't take that through back when we were.
Prasanna Malaiyandi: Unintended.
W. Curtis Preston: Yeah, we were just thinking about all the problems
that we were having with tape, and we wanted to fix those problems.
And so we went with this space backups to fix those problems.
And, uh, and remember, I will, we'll, we'll always say this.
It wasn't, the tape was too slow.
It was, the tape was too fast.
And we can, you know, if you don't know what I'm talking about, then you need
to, you need to listen to more episodes.
And we've, we've come a long way, but for a really long time backwards, were
just sitting there as files either in a, in a regular, uh, you know,
direct attached file system or an NFS or SMB attached file system that were
directly deletable by a privileged user.
Uh, as long as you, you know, as long as you had access to that drive, we've.
Hopefully in most cases we've gotten away from that.
I think still many people go and buy a, you know, pick your favorite backup,
you know, disc vendor, and they just mount it as a backup drive and then
start backing up to it, and it's, it's, it's eco back slash backups.
And which just like screams, come delete me.
And you know, I, I, I know people still do that and they shouldn't do that.
They should ask their backup vendor, Hey, is there a way that I can
transfer data to my dis space backup storage without it showing up as a,
as a dry letter on my uh, system or a mounted drive on my Unix system?
But that is a huge change in security of backups.
Prasanna Malaiyandi: Yeah.
No, I agree.
It made life easier, right?
You could now do all the instant access, quickly, restore your data, do all the
test in dev, but there was a fatal flaw.
W. Curtis Preston: Yeah, it made life easier for everybody.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: Yeah, absolutely.
And then replicated backups, right?
We had replicated backups, and then if you, you know, if you have a, a,
you know, a screw up in your backup system, it, the replication just makes
the screw up, uh, more effective.
Right.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: Yeah, so this is why we have to talk about immutability.
Um, but before we talk about that, we'll talk about some things
that haven't changed at all.
So the thing I want to talk about is this concept of immutable
backups, and that really wasn't.
Something we talked about back in the day.
Prasanna Malaiyandi: Well, I think a lot of it back in the day was your backups
were kind of already immutable, right?
If you were,
W. Curtis Preston: the,
Prasanna Malaiyandi: were going off tape.
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: Right then you didn't have to worry about the scenarios
that led you to need immutable backups.
W. Curtis Preston: Yeah.
That, that's really true.
And that, and that's one of the things that we're gonna talk about, right?
I, I would say the biggest reason was that no one was attacking the backups.
No one was trying to attack the backups.
And the, the worst things that we were worried about were things like I.
Um, we were worried about a natural disaster or a terrorist
attack, taking out the entire place where backups were stored.
So we needed to make sure to have a, an air gap, a physical separation between
the, the backups and, uh, you know, and the, the copy of the backups, right?
We weren't.
Prasanna Malaiyandi: on a tropical island.
W. Curtis Preston: Exactly right.
Exactly.
And so we weren't really, we, we, we weren't at all.
I, I could just, just absolutely tell you that we were not at all
ever thinking about a cyber attack.
Trying to, I, I don't even think we had that term.
I mean, we had the internet, but we didn't really have the, the term cyber attack.
Um, at least that I remember.
We certainly never used it in backup circles,
Prasanna Malaiyandi: Or it was different, right?
Because you weren't necessarily worried about people stealing
your data or things like that.
I remember you were the one who recommended me to read
the Cuckoo's test, right?
And so those were the types of attacks that you sort of
saw back in those days, right?
Stealing resources, not necessarily stealing your data
W. Curtis Preston: Yeah, cuckoo egg, by the way.
Prasanna Malaiyandi: Or sorry, cuckoo.
W. Curtis Preston: which by the way, if, if, if you're out there and you don't
know what we're talking about, great book.
Um, uh, and it's a relatively short read.
It's a great book to read, and, and it's a true story that, you know, involves
a, a cyber attack against a university.
It's great.
So before we talk about immutable backups, I want to talk about some
of the things that really haven't changed that are also, um, impediments
to having a safe backup system.
And the first is the fact that backups never were, never have been, and never
will be the sexy part of the data center.
Prasanna Malaiyandi: Yeah, no, I a hundred percent agree.
No one wants to do backups.
No one wants to spend the money on backups, right?
Everyone wants to be like, Ooh, I need to do, given the recent trend
these days, I wanna focus on AI and do this and that and all the rest.
But no one wants to focus on how do I protect it other than people like Curtis.
W. Curtis Preston: Other than a few, few weird people.
I would say that it's a, there were a lot of reasons.
The biggest reason is easily that it is a very difficult job.
We, we started out this, this episode talking about how
sometimes, how difficult it can be.
You are invisible or you're in trouble.
No one, no one remembers the millions of backups that you got, right?
They only remember the one restore that you got wrong.
And so nobody wants to have that pressure on themselves.
And so they, nobody wants to raise their hand in, you know, in the,
uh, in the room to say, Hey, I'll, I'll be in charge of backups.
So this is true in terms of.
Um, it's not the sexy part of the data center, which means nobody
wants to do it, number th number one.
Number two, nobody wants to give that part of the data center any money
number, and then number three I'd say is that as a result, it goes ignored.
Prasanna Malaiyandi: Oh, I agree with that.
I think there was a study, I can't remember when it was that.
What percentage of an IT budget do you think goes towards backup?
W. Curtis Preston: It's a very small portion.
Prasanna Malaiyandi: I think it was less than a percent.
W. Curtis Preston: Yeah.
Yeah.
That sounds about right.
Yeah.
So it, it goes, it goes ignored.
It goes ignored.
From a budget perspective, it goes ignored from.
A cybersecurity perspective, which is really what we're talking about.
It's what we've been talking about the last several episodes is, is
cybersecurity and, and backups.
And it, it means that it doesn't get put front of the line when we start
talking about putting out patches.
It doesn't get put front of the line in terms of looking
for trends that could be scary.
Um, it's just, it literally just goes ignored.
Um, and that, and that's a really scary thing.
I.
Prasanna Malaiyandi: Yeah, and it's not even thought of initially when you're,
I know we've talked on the episodes that when you are deploying a new application,
you should be thinking about backup or at least have the backup person in the room.
But sometimes things develop so quickly that.
Backup is an afterthought, right?
And sometimes it's very painful.
W. Curtis Preston: Yeah, very much so.
I can remember, I can remember one example of that when we bought, uh, the comp, the
bank that I worked at bought a, an HP.
Was it 1000 I think was the HPT 1000.
That's what it was.
It was a T 1000 and the, it came in and it had, um, it was huge.
It was, are you ready?
A hundred gigabytes of data.
It was huge.
Prasanna Malaiyandi: Humongous.
W. Curtis Preston: It was humongous.
Prasanna Malaiyandi: Now your phone has more than that.
W. Curtis Preston: Prior to this server or prior to this server showing
up at our data center, the biggest server we had was like four, and then
suddenly we had a single server that was 25 times bigger than the next
biggest server, and it came with.
A DDS drive, which was a, a, um, four gigabyte drive.
And, and I, I just remember asking like, who, who's gonna stand here
all night long and swap tapes 25 times to get to get it back up?
And that was, and, and nobody had knew, nobody had even thought of that.
That's the point that nobody had even thought, well, maybe we should
have enough backup infrastructure to back up this new gigantic server.
And that was, by the way, how I got the, my first justification to
buy my first, uh, tape robot, which was the first time I sent money
to Spectral Logic back in the day.
Prasanna Malaiyandi: You weren't just sitting there all night swapping tapes.
Curtis.
W. Curtis Preston: definitely not gonna do that.
Definitely not.
Um, so yeah, so that, that problem's not a new problem.
Uh, the other thing, and as a result of all of those things that we just
talked about, a very common thing for.
Computing infrastructure or for computing environments to do is
to give backups to the new person,
right?
Uh, it was the first job I ever got.
I, I was just editing the, the podcast.
So with Dwayne Lalo, he said it was the first job he ever got.
And the problem with that really is you have this really, really important,
really secure, or what should be a really secure job, and you're handing
it to the least experienced member of the, of the, you know, the team.
Right.
I, I mean, I, I had been at the bank,
you know, essentially a week.
I, I had been, I had been.
In computers, like, like commercial computers.
And I had been introduced to Unix for a few months.
I, I, I had been at this company, I, I think I had worked at this
company for like two months, uh, prior to going to the bank.
And I had been, you know, I had been given some lessons in Unix
and I had been demonstrated the importance of a really good backup.
Uh, here's, this was the first time in my career when.
Prasanna Malaiyandi: Theoretical versus practical.
W. Curtis Preston: Yeah, so we were a, uh, we were a blended call management
system, which you might refer to as an auto dialer, and that was a, that was a,
a naughty word back in the day, right?
We were not an auto dialer.
We were an intelligent call management system.
And, but it was essentially an Auto Tyler and it had this very fancy
Unix-based computer behind it.
And we would go out and do this, this, it was like this long involved install
because it was a very, this was before VoIP and all that kind of stuff.
We had, you know, we had literally cards that went in the server and those, each
card connected to a POTS line, right?
Prasanna Malaiyandi: Mm-Hmm.
W. Curtis Preston: And so we had this big physical install that we had
to do, and it took about two weeks.
And then, um, you know, you, you sent a couple of SCS on site and I was one
of the sc I was obviously the junior se, and this other guy had been, um,
doing all this configuration and.
The job that we were supposed to do before we left was to pop in a tape,
by the way, a tape, a nine track
Prasanna Malaiyandi: Ginormous.
Yeah.
W. Curtis Preston: and pop in a nine track tape and back up the OS
configuration that we had just finished.
That was the last thing that we were supposed to do, and just
before doing that, my, my compadre.
Said he was looking at the root file system and he noticed that there were
a couple of garbage files in the root file system and one of them started
with, um, question mark and had a bunch of gar dego stuff after it.
So he didn't want to type all of the gar dego, so he typed, um,
RM minus RF question mark, star
Prasanna Malaiyandi: Oh,
W. Curtis Preston: from the root file system.
Wanting to clean up the OS before we took the backup.
Prasanna Malaiyandi: oh geez.
That must have been interesting.
W. Curtis Preston: Yeah, so that was the, that was the day that I
learned the value of a good backup.
The best time to do a backup is yesterday.
The second best time is today.
Prasanna Malaiyandi: today.
W. Curtis Preston: Um, this was definitely the second best time,
so we lost a whole bunch of work.
Uh, and luckily it was not my fault, so I just learned a very valuable lesson.
Anyway, sorry, that was bringing me, bringing me back, back in the day.
Um, so yeah, so.
It's a horrible practice that we put the most junior person in charge of the
backups and that should absolutely stop, but I don't know how to merge those two.
How do you, how do you, how do you properly reward this job that
is really, really important such that someone who isn't junior is
going to want to take the job?
I don't know.
I don't have an answer to that question.
Do you, do you persona.
Prasanna Malaiyandi: Well, no, but it's No, but if you think
about this, this really isn't any different than say A DBA, right?
A database admin who starts off initially does some Basic things, right?
And over time they sort of become like that database architect who sort
of figuring out overall layouts and things like that rather than sort
of just administering the system.
Now the difference is in the case of the DBA, you're probably starting
off in like test and dev instances or like non-critical applications versus
as a backup admin, you're probably starting off on like the critical
infrastructure that you are protecting.
W. Curtis Preston: Yeah.
Yeah.
That's why I'm, I, I'm not sure I agree with your, with your comparison
there because, well, first off, people actually wanna be DBAs.
Prasanna Malaiyandi: Yeah.
Well, and yeah, maybe.
Maybe what you need to do is like a cross training program where you do
a rotation in like a different part of the organization and then you come
into the backup work, because that way at least you're also get more
familiar with tools and other things.
W. Curtis Preston: Right, right.
Yeah.
Prasanna Malaiyandi: That way you're not the junior person.
W. Curtis Preston: exactly, exactly.
Yeah.
You, you need, I, I guess the best thing we want to do here is to make sure.
That you're, that you're making sure that your junior person, that you're giving
the, the backups to has some really good cybersecurity, some basic, at least some
basic cybersecurity training, right?
Because you're handing them in order to do the backups.
They are a very powerful person and they have the ability to
read and write every file.
Right, which means they have the ability to overwrite every
file in the organization.
Uh, even if you're using proper modern backup design that uses role-based
administration and all of that, you're, you're still most likely
handing them the keys to the kingdom.
Uh,
Prasanna Malaiyandi: Is that true though?
Because normally don't, you have sort of help desk restore operators and
then the more senior people are kind of those who are doing like the backup
architecture and the design, you probably have people who are figuring out
W. Curtis Preston: In a large environment, that would be the case, but that's not
the case in most companies, I don't think.
I think it's just one person, actually, in most companies it's a collateral duty.
Right.
So, but let's, so let's get to the meat of the episode here where we
wanna talk about immutable backups.
So first off, just generally speaking, what does immutable mean?
Prasanna Malaiyandi: It basically means you can't change it,
W. Curtis Preston: Yeah.
It's literally, it's, it's, it's, it's a word that's used outside
of it, but within it, yeah.
It means that it cannot be changed
Prasanna Malaiyandi: which also includes deletions.
W. Curtis Preston: yeah.
Changed.
Yeah.
Deletion is a change.
Right.
Um, and the difficulty here.
Is that, that sounds like a binary condition,
Prasanna Malaiyandi: Mm-Hmm.
W. Curtis Preston: Like pregnant or dead.
Right.
Prasanna Malaiyandi: Not the words I would've chosen, but Sure.
W. Curtis Preston: Both pregnancy and death are binary conditions.
You're either pregnant or you're not, or you're dead or alive, right?
These are binary conditions.
Um, immutable should be a binary condition that that is what is implied.
When we say immutable, it can't be changed.
It can't be deleted.
It can't be encrypted.
It can't be modified.
It can't be, I don't know.
I think I covered 'em all.
Prasanna Malaiyandi: Yeah, but there's really a spectrum though.
W. Curtis Preston: it is definitely a spectrum, and that is the
problem, is that immutable.
And that's, that's the thing I wanted to just mention.
So let me, so let me ask you this.
Why, why has immutable backups, why has that really come to the fore,
uh, in the last say, 10 years or so?
Prasanna Malaiyandi: Well, like you were talking about earlier
in the episode, cyber attacks.
Ransomware attacks, they not only go after your production, but they
realize that, hey, you have all this data sitting in your backup systems
that you could use to recover your environment and not pay the ransom.
So instead, we're gonna go attack your backup systems, delete all
your backups so you can't recover.
And then also because they're doing the double extortion where
they also exfiltrate your data.
That's a treasure trove, right?
They don't have to go hit 20 different systems.
If they get to your backup system.
All your data's there and backup systems are really good at restoring data so
they can quickly pull the data out and then blackmail you in case you decide
that you don't want to pay the ransom.
W. Curtis Preston: Yeah, exactly.
Now immutability won't address that second part.
Uh, it is important to me.
Right tho those are, those are the two main concerns that we have when
we talk about the backup system.
And, uh, you should definitely listen to the episode that went live.
Um, let's see, that would've been May 26th now, um, the one where with Dwayne Lalo,
where he talks a bit from the Red Team, the Red Team Cybersecurity episode that.
Um, where he talks about what a treasure trove the backup system is and how he
loves, he's a, you know, he is a red team.
He's a hacker, uh, that, you know, an offensive cybersecurity person.
Uh, he's very offensive.
And, and he loves going after the backup system, right?
When we talk about immutability, we're primarily concerned with
making sure that the backups themselves aren't modified, right.
And which, you know, as we said, you know, and, and.
Like you said, this wasn't a problem.
I don't know, even five years ago, maybe, uh, this wasn't a problem,
but hackers weren't going directly after the backups, but now they
definitely are because they're the best defense we have against ransomware.
Right.
I.
Prasanna Malaiyandi: Yeah.
And and I would say before immutable backups slash immutable archiving was
really intended as a compliance use case.
W. Curtis Preston: Yeah.
And that's what I, that's kind of where, where I wanted to go
with this is let's talk about the different things that people mean
when they say immutable, right?
And prior to cyber security attacks, when we talked about the backup system
or the archive system, which are different things when we talk about the
backup system or the archive system.
Being immutable, what we're primarily, um, wanting to be able to say in a
court of law that this file, this document that I'm presenting in court,
is exactly the same as it was I.
When it was created, right.
This, this email that is the smoking gun we can state.
It is a, it is a faithful reproduction of the original email that the plaintiff
made or the defendant, I get It's, it's a's a faithful reproduction of
the email that the defendant made.
That shows that they were doing the thing that they weren't supposed to be doing.
Right?
And, and so the, so that the, the defense can't say, oh, well
that's a fake email, right?
You're like, no, this is a faithful reproduction of the email that this,
you know, and here's the, here's the chain of custodying, the chain
of technology to get from A to B.
And so, um, you know, it, it's, fine,
Prasanna Malaiyandi: It's as good as the source.
It's as good as the original.
W. Curtis Preston: It's as good.
Yeah, we are.
We are essentially looking directly into the email system from three months
ago or three years ago or whatever.
It's right.
That's what we were primarily concerned with and we are still concerned with that.
Right.
We are still concerned with immutability for that reason from
a legal perspective, but, um.
What we're now more concerned with is that someone could go in and modify the backups
in such a way that they encrypt them.
They, you know, they, they delete them.
Uh, they could actually modify them.
I, I don't, I don't, I don't know of any scenario in this case, but
they could potentially go in and selectively modify them to mess
with people for whatever reason.
Um, but that shouldn't be able to be the case either.
Right?
Prasanna Malaiyandi: Yeah, and that just seems like a lot of work to go selectively
modify unless you're trying to do something sneakily without being detected.
W. Curtis Preston: Right, right.
Um, so that's the, that's one way that backup systems and archive systems use
the term immutability is they say, I.
We can say that, you know, these two things were, you know, are the
same.
So the, the next thing that I see is that.
I see vendors that say, well, we have a A, right?
Only we have an append only file system that once backups are
written into our append only file system, they cannot be modified.
Have you seen backup folks saying that I.
Prasanna Malaiyandi: Yeah, though I've seen some storage vendors, right?
You have companies who do backup appliances like Rubrik who say,
yeah, once you write your backup to our system, right, you can't
go and overwrite that data again.
W. Curtis Preston: Right,
Prasanna Malaiyandi: of a right only and.
Having worked at a storage company which was famous for a write
Anywhere file layout, I could see the benefits of doing it that way.
W. Curtis Preston: Right.
Yeah.
And, and what this says is that it stops direct attacks against
the backup storage itself.
That if I, if I'm able to get into the file system, if I, if I'm able
to get into the operating system of that backup system, I wouldn't
be able to modify or delete the backup files in order to, you know.
Well, well essentially delete them.
I wouldn't be able to encrypt or delete them because this is
this append only file system.
Prasanna Malaiyandi: Which is interesting because I do wonder
about certain scenarios like, Hey, I'm running low on space.
I need to go delete a whole bunch of backups or backup sets
in order to free things up.
How?
What do I do?
Am I just stuck?
W. Curtis Preston: Yeah, that's gonna, that's gonna be
depending on the product, right?
So the question, my, my question is.
With the, the product in question and that this isn't necessarily Rubrik,
this is just, I'm, it's just the, this is the question that we're asking
is, I, I understand you're saying that it's an append only file system.
What about the system that's controlling the backups itself?
What protection is there in, in that system?
Meaning if I am a privileged user.
Am I able to, um.
Delete
backups, right?
Am I able, am I able to prematurely expire backups?
Because even with immutable storage, they can have, it can
have an expiration date, right?
So you can say this, this, these backups are immutable for 90
days and then they can expire.
But if I am, you know, the super high cloud administrator of this backup system,
am I able to then expire backups early?
And if you are, then.
That is, again, this is why I say immutable backups.
Are a spectrum because the, the, the question is, if I am, you know,
super high up in the echelon, if I make a phone call to this person, is
there a way to expire older backups?
I, I would
also ask, is there a privilege to count in the operating system that allows
me to essentially expire all backups?
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: that, that, that is the case in some backup systems, that even
though they're immutable, if I have access to this one privileged account, I can wipe
out all backups with a single command.
Prasanna Malaiyandi: Or reset the system back to a factory state, right?
That's sometimes common.
W. Curtis Preston: Right.
Right.
Um, so that, that is another one that would be called immutable.
Right.
Um, and, and I'm not saying that any of these aren't immutable, I'm just
saying that there are a whole bunch of things that are called immutable.
right.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: Um, another one, and, uh, and this is
where, when, and, and again.
This is like good, better, best, right?
Anything would be better than backups.
Sitting on just an open Windows file system would zero security checks, right?
Anything would be better than that,
and,
Prasanna Malaiyandi: I think I have a worse scenario.
W. Curtis Preston: okay.
Prasanna Malaiyandi: A window system open to the internet.
W. Curtis Preston: Yeah.
Yeah.
With, uh, with, uh,
Prasanna Malaiyandi: RDP
W. Curtis Preston: Um, R-D-P-R-D-P enabled.
Yeah.
Uh, the, um, the next, you know, one of the most surprising things when,
when I listened to the episode with Dwayne again was when he said that
so many times that they, they, they go in with the default password.
And that it works.
That's just, that just hurts.
Prasanna Malaiyandi: The next one is really about something.
So not the best, right?
But better than the Windows system, right?
Which is good.
Which is having a server that has been hardened and locked down and secured.
Such that you are protecting from most common issues and
security vulnerabilities, but it may not still be perfect.
And in this example, right, we had Dave Russell and Rick Manover
from Veeam over talking about the hardened backup repository.
I think is the official name, which is basically a server which they
locked down, which allows you to write Veeam backups to it, and they have an
immutable flag in the Linux repository such that the backups cannot just be
normally deleted, casually deleted.
I would say.
W. Curtis Preston: Right.
Yeah.
They can't be deleted by anyone except for root, right, and they can't even
be accidentally deleted by root.
The root would have a person with root would have to unset
the immutability flag on the file and then delete the file, right?
So.
Uh, this is what, you know, I agree with you that it, it is better than,
you know, having a Windows file system.
I mean, first off, it's just better in that it's Linux and not Windows.
And I don't, I don't just mean that Linux is better than Windows.
What I mean is it's something different, right?
And then the, uh, 'cause I definitely do not think that Linux is.
Perfect.
Right.
But the, uh, I do still, like the Linux way of everything is off
by default versus the Windows way where everything is on by default.
Although I think they've gotten better at that over the years.
The um, um.
It is important to understand that if someone gains root, then they're
able to, uh, modify these files.
Now they do a really good job.
I think, uh, specifically with the Veeam repository, they do a really good job of.
Giving you a list of things that you need to do where it's like essentially.
But once you've configured this system, you can configure it, for example, to
automatically update itself with security patches and that you don't need to be
logging into it regularly as route.
So you can like super secure that, um, thing.
So that if, if anyone ever does log into it as rude, it can set off all
sorts of, uh, bells and whistles so that you can know that something.
bad is going on.
Prasanna Malaiyandi: Yeah, and I know we've talked in the past about
like patch management, go do your patch management all the time.
And having an automated process and keeping it up to date I think
is a great feature that that has.
W. Curtis Preston: Right.
And then I would say, what's the, the next, and I'm, I'm gonna
put this up in terms of the best as far as I can see, is that I.
Here's an immutable system where backups cannot be modified even by a privileged
account, even by somebody with root.
Um, the, and there would be two types of systems like that.
One would be a backup, a, a storage system that could be on-prem.
Another would be something like a cloud system like S3 with the
immutable flag turned onto that.
And I would, I would actually put.
Them in terms of like the, the on-prem system isn't as secure as
the cloud system, in my opinion.
Why?
And why would that be?
Prasanna Malaiyandi: because you can physically access the system and
yank the drives and all the rest.
W. Curtis Preston: Yeah.
Physical access trumps, all right.
So it's funny, a lot of people are like, oh, the cloud is insecure.
And I'm like, well, physical system's on-prem or insecure.
Right, because I can easily go up to that and I can do bad things to
it as long as I know where it's at.
Prasanna Malaiyandi: Yeah.
And even with the physical storage systems, I know at NetApp and even
with AWS's Object Lock, right?
They do have those two different modes, compliance and governance.
I.
For their immutability options.
One which sort of allows an admin to reset
W. Curtis Preston: Mm-Hmm.
Prasanna Malaiyandi: the flag, right?
Like we had talked about earlier.
So it's good enough, but there are cases where, yeah, I need to unset it.
So they do give you that option.
And then the other one is you can't touch it unless you basically, in the case
of AWS, blow away your account, right?
So it's really nothing you could do as an admin will help you.
W. Curtis Preston: Yeah.
And you know, just realize that a, what, what I would
call a truly immutable system.
If you change your mind, you can't change your mind, right?
If it's, if it's truly immutable, you should not be able to, to blow it away.
Now, I, I still haven't done this experiment, and I, I, I really am gonna
do this at one point where I create an AWS account, turn on immutability, put
a bunch of stuff in there, and then delete the account and see what happens.
Um, I don't, I don't know what happens there, and then stop paying my bill.
No, no immutability system.
Outlives not paying for your, your, uh, your storage
Prasanna Malaiyandi: Yeah, they will probably reclaim that
data as quickly as they can.
Curtis.
W. Curtis Preston: do garbage collection.
Goodbye Curtis's data.
The, I I mean, I will say that in that case it's kind of your own fault, but, um,
so this is what we mean when we say that.
Immutable backups.
You wanna make sure that a black hat, a bad actor that gets into your environment,
cannot do damage to the backup system.
And so you want to separate.
Backups as much as you can from the, the, the bad actor.
And the best way to do that is to store them in an immutable system.
Just realize that there is a, um, a spectrum of immutability and
all of them, they all say their backups are immutable, right?
Prasanna Malaiyandi: I think it's also important to note that you may not need
the best solution for your use case and your needs, so also make sure you
understand what you're protecting against.
There is gonna be cost, there is gonna be complexity potentially
with those solutions, and maybe it isn't needed to go to that extent.
And maybe like better is good enough.
W. Curtis Preston: Yeah.
Never let, never let perfect be the enemy of better.
Right?
So, you know, you, you could be better.
Um, and that's okay.
Always strive to be better, but don't.
Don't let say, well, th you know, th this immutable system
doesn't work in this one scenario.
Don't let that prevent you from doing something.
Something is always better than nothing, right?
One thing I do want to mention is that MFA is a big part of this.
Some of these systems that have it's immutable except for the super powerful
person, or it's immutable unless you make these 17 phone calls to.
Um, you know, you, you've gotta really, really authenticate yourself.
I have seen scenarios where companies have had their email system hacked
in addition to their backup system being hacked, and the result was that
they were able to defeat MFA because they were using email as their MFA.
Prasanna Malaiyandi: Or the other thing is also phone because now they
can impersonate someone, call in, they have all the access to the emails
and can spoof like two factor codes.
And talk to the person on the other end, and they don't know the difference.
W. Curtis Preston: Yeah, the, the, the, um, the ability of AI systems these
days to mimic the sound of a, of another person's voice, uh, in, in fact, uh,
just as a demonstration, uh, the next.
30 seconds of audio is completely fake.
How are you doing, Prasanna?
Prasanna Malaiyandi: I'm doing fine, Curtis.
What are we talking about today?
W. Curtis Preston: I think we're talking about immutable backups, and
I hear that they are on a Spectrum.
Prasanna Malaiyandi: You mean like IBM Spectrum Protect?
W. Curtis Preston: No, that's a backup product.
Prasanna Malaiyandi: You mean the Irvine Spectrum?
W. Curtis Preston: No, that's a mall.
I mean that there are many things that are called immutable, and
they all have different levels of protection against attacks.
Nice talking with you, fake Prasanna.
Prasanna Malaiyandi: You too, fake Curtis.
W. Curtis Preston: So that was neither me nor persona talking.
For the record, that was a little tool that we use.
That is very useful in the right scenario, but very dangerous in the wrong scenario.
Um, yeah, you, you really have to think about that, right?
When you, when you put all these layers, if you have all these layers of
security between you and the ability to delete your quote immutable data, think
about all of the modern things that people are able to do, um, before, um.
You know, you subtle with one of those systems.
And again, don't let perfect be the enemy of the good.
But, but, but just think about all the things that are possible these days.
Uh, you know, they can fake video, they can fake audio.
Uh, they can certainly fake email and phones, right?
SMS is a horrible way to do things, right.
But, um, um.
With that, that's our discussion on immutable backups.
I hope you enjoyed a persona.
Prasanna Malaiyandi: No, this was a good topic.
I like immutable backups.
W. Curtis Preston: Yeah.
Immutable backups.
Good.
And, uh, hope, uh, you enjoyed that as a listener.
Hope you learned a thing or two.
And with that, that's a wrap.