Speaker:

You found the backup wrap up your go-to podcast for all things

Speaker:

backup recovery and cyber recovery.

Speaker:

In this episode, we look at the reality of modern data centers.

Speaker:

Human error has become the number two reason that we do restores today.

Speaker:

Number one, of course, would be cyber attacks.

Speaker:

Back in the day, you know, a hundred years ago when I first started, it was

Speaker:

really all about hardware failures, hard drives, crashing, servers dying.

Speaker:

But now it's people making mistakes.

Speaker:

Sometimes it's a user accidentally deleting a file, or maybe it's an admin

Speaker:

wiping out an entire directory structure.

Speaker:

Humans are the weakest link.

Speaker:

I've got, I don't know, four or five war stories that'll make you cringe.

Speaker:

And we talk about insider threats as well, and why your backup systems

Speaker:

need to be designed with human error and human frailty in mind.

Speaker:

Trust me, these stories will remind you why we do what we do.

Speaker:

By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr.

Speaker:

Backup, and I've been passionate about backup and recovery ever

Speaker:

since I had to explain to my boss.

Speaker:

While there were no backups of the production database that

Speaker:

we had just lost, I don't want that to ever again happen to me.

Speaker:

I don't want it to happen to you.

Speaker:

That's why I do this.

Speaker:

On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes.

Speaker:

This is the backup wrap up.

Speaker:

Welcome to the show.

Speaker:

Hi, I am w Curtis Preston, AKA, Mr. Backup, and with me, I once again have

Speaker:

my security deposit consultant persona.

Speaker:

Molly, Andi, how's it going?

Speaker:

Persona,

Speaker:

I am good, Curtis, although I thought you would've said you're a person who

Speaker:

was successful at DIYing something for probably the first time in their life.

Speaker:

a guy that, a guy that finally joined the ranks because, because you're

Speaker:

completed now with your, with your.

Speaker:

I am.

Speaker:

If people

Speaker:

Yeah,

Speaker:

episode, I, uh, had to redo some.

Speaker:

stuff for

Speaker:

yeah,

Speaker:

system, and it was a little painful.

Speaker:

yeah,

Speaker:

lie.

Speaker:

Um, and especially because I had to replace a shutoff valve, which meant I had

Speaker:

to shut off the water for the entire house

Speaker:

yeah.

Speaker:

I was repair, replacing that shutoff valve.

Speaker:

And then it turned out that I was leaking and blah, blah, blah, blah, blah.

Speaker:

So, eventually got around to it, turned out, um, make sure you wrap the

Speaker:

Teflon tape in the right direction.

Speaker:

Yes, yes, yes.

Speaker:

You figured that out, didn't you?

Speaker:

Yes.

Speaker:

And also having the right tools helps, especially a pipe wrench.

Speaker:

Yes.

Speaker:

More than one, right?

Speaker:

Yeah.

Speaker:

So once I did that, then actually doing all the PVC gluing and

Speaker:

That part was easy, right?

Speaker:

So I remember purple stuff first, then the blue stuff.

Speaker:

Yeah, yeah,

Speaker:

I should say.

Speaker:

yeah.

Speaker:

Oh, you use clear stuff?

Speaker:

Yeah.

Speaker:

Oh, that explains why I saw the blue.

Speaker:

I saw the purple.

Speaker:

Yeah.

Speaker:

Okay.

Speaker:

But it worked and it held, and it's knock on wood.

Speaker:

So far so good.

Speaker:

So

Speaker:

Well, I'm glad to hear that you can now sprinkle your yard once again.

Speaker:

I of course don't have that challenge 'cause I have a fake yard.

Speaker:

well it's not a yard, it's just for the plants and all the drip irrigation, so.

Speaker:

Oh, gotcha, gotcha.

Speaker:

It's a yard, it's just, it's not grass.

Speaker:

saw grass.

Speaker:

Yeah.

Speaker:

Right, right.

Speaker:

especially last weekend.

Speaker:

No, over the weekend, I think the temperature at 87 here in the Bay

Speaker:

Area and today it's 61, so yeah.

Speaker:

Yeah, we actually, we had a heat wave here as well.

Speaker:

Um, it actually got up to like 90 at one point,

Speaker:

Dang.

Speaker:

Today it was cold and rainy, so

Speaker:

Yeah.

Speaker:

I think may.

Speaker:

Gray back, no Uns shining outside.

Speaker:

Hmm.

Speaker:

I don't know.

Speaker:

Who knows?

Speaker:

You never know in San Diego, right?

Speaker:

Especially in May.

Speaker:

Right.

Speaker:

May Gray and June gloom.

Speaker:

For those that don't live here, it's pretty much overcast all the

Speaker:

time for two months of the year.

Speaker:

But it's why the temperature is,

Speaker:

Even Keter.

Speaker:

Yeah.

Speaker:

Yeah.

Speaker:

Um,

Speaker:

dude.

Speaker:

come here.

Speaker:

What?

Speaker:

ahead, finish.

Speaker:

Well, I just said people come here and they, um, they're like, why?

Speaker:

Where?

Speaker:

I'm like, this is like, do you do no research on the places you go to?

Speaker:

Right,

Speaker:

Well, it's like the time that, or I think that twice that I've been up to

Speaker:

Seattle and both times it was nice and sunny and so in my mind it's imprinted

Speaker:

that Seattle is always nice and sunny.

Speaker:

Yeah.

Speaker:

Oh yeah.

Speaker:

I, yeah, I spent, I spent four months in Seattle in 1998, and

Speaker:

it was the most perfect four months in the history of Seattle.

Speaker:

It rained like two days during those four months, and it was just, and

Speaker:

yeah, I could, I could have moved there, but it's not normally like that.

Speaker:

exactly.

Speaker:

Yeah.

Speaker:

But, um, well, uh, today, speaking of.

Speaker:

Humans making mistakes today.

Speaker:

I thought we'd talk a little bit about human error.

Speaker:

Um, and, uh, as, as I often, uh, you know, find myself doing, I'll talk about

Speaker:

back in the day, back in the day, um.

Speaker:

The reason when I started my IT career, the, the number one reason

Speaker:

we were doing restores was, um, actual hardware failure, right?

Speaker:

Because I am Prera, right?

Speaker:

Um, we, we had mission critical servers running on individual hard drives,

Speaker:

Which is bonkers.

Speaker:

A

Speaker:

right?

Speaker:

Right.

Speaker:

do you think, what is, what is it like this, isn't there like a satellite

Speaker:

that's been out there for like 40 years

Speaker:

Oh yeah, yeah,

Speaker:

and do you think that still also runs on a single disk?

Speaker:

I,

Speaker:

yeah.

Speaker:

Def absolutely.

Speaker:

Yeah.

Speaker:

crazy.

Speaker:

Yeah.

Speaker:

Well, it might, do they have, do they have disc drives?

Speaker:

I don't know.

Speaker:

No.

Speaker:

I mean, it's, if it's been there that long.

Speaker:

Uh, but yeah, it, it's, I mean, I'm pretty sure RAID was invented before I actually

Speaker:

saw it in the data center, but in my data center, which was a, a big, you know,

Speaker:

$35 billion company, we didn't have any.

Speaker:

Right.

Speaker:

And, um,

Speaker:

Was it

Speaker:

and go ahead.

Speaker:

was it the difference though between raid on open systems versus raid on mainframes?

Speaker:

Yeah, I don't, yeah, I don't know.

Speaker:

Um, it just, it certainly wasn't, you know, I remember the first

Speaker:

time and I talked about it a few episodes the first time I saw

Speaker:

hot swappable disc drives, right?

Speaker:

Where you, you had the raid arrays, right.

Speaker:

Um, and all of that sort of, it sort of all went down right around

Speaker:

that timeframe, but for a few years.

Speaker:

We were, we were losing entire databases or servers because an

Speaker:

individual R drive went down.

Speaker:

And so we, we were restoring that stuff all the time.

Speaker:

And in fact, um, my first, once I left the bank and I went into

Speaker:

consulting, um, one of the reasons I got so good at Bare Metal Recovery

Speaker:

was that I was doing it all the time.

Speaker:

Um, because I was at a. Uh, the headquarters of a

Speaker:

large oil and gas company.

Speaker:

And, um, they had all these servers that had not been, uh,

Speaker:

maintained in a really long time.

Speaker:

And so we were doing crazy things like installing patches and rebooting them

Speaker:

with significant portion of the time.

Speaker:

They just didn't come back up.

Speaker:

So,

Speaker:

makes perfect though, right Curtis?

Speaker:

yeah.

Speaker:

And so, yeah.

Speaker:

Um, and so my, my, my point is that back then.

Speaker:

A significant portion of the time, the reason you were doing a restore, it

Speaker:

was because the actual hardware failed, failed, and that is just pretty much.

Speaker:

Gone.

Speaker:

I, you know, uh, between the fact that we now use solid state a as hard

Speaker:

drives, and the fact that we use that, that no one, no one does anything

Speaker:

important on, on an individual.

Speaker:

I mean, your laptop, your phone, these are individual drives, but not

Speaker:

nothing in a, in a server, right?

Speaker:

Um, and.

Speaker:

and also probably just reliability of components has also gone up.

Speaker:

Absolutely.

Speaker:

The result of all of that is that at this point the, there are,

Speaker:

generally speaking only two reasons.

Speaker:

We restore things and both of them had to do with people, right?

Speaker:

Either people purposefully did something to damage the systems or they accidentally

Speaker:

did something to damage the systems.

Speaker:

Are you with me?

Speaker:

Yep.

Speaker:

Yeah, so

Speaker:

failure.

Speaker:

it is not a hardware failure, right?

Speaker:

So either bad people, you know, bad actors, you know, it's a si,

Speaker:

it's a, it's a ransomware attack, it's a cyber attack of some sort.

Speaker:

Uh, or stupid users

Speaker:

Would you consider like the OVH

Speaker:

or stupid admins.

Speaker:

So, so that is still, that is still a human error in my opinion.

Speaker:

I mean, okay, so what, we can add that as a third category, which

Speaker:

is natural disasters, right?

Speaker:

Um, so if, if a fire happens, you know, um, yeah, I, we'll, we'll put a fire in

Speaker:

a natural disaster, although that fire, I'm not sure was a natural disaster.

Speaker:

But yeah, so that, that, that is a third reason, you know, I, I'll take that.

Speaker:

But, but, but even that, the reason they end up having to do a restore,

Speaker:

I think, was human error, right?

Speaker:

Meaning that they, they just configured everything all wrong.

Speaker:

Um, and I, and that's my story, and I'm sticking to it.

Speaker:

so let's talk about the accidental humans

Speaker:

Yeah.

Speaker:

Yeah.

Speaker:

'cause we talk a lot about the, the, the bad actors.

Speaker:

Yeah.

Speaker:

Um, so let's talk about the, you know, and, and basically what this

Speaker:

kind of boils down to is, uh, either stupid users or stupid admins.

Speaker:

Right?

Speaker:

I, so there are two stories I want you to tell,

Speaker:

Okay.

Speaker:

The first one is, uh, restoring a document.

Speaker:

The, the document called resume,

Speaker:

Yeah.

Speaker:

Yeah.

Speaker:

And

Speaker:

Uh.

Speaker:

second one I want you to talk about is the file server with the user home directories

Speaker:

Lord.

Speaker:

Yeah, yeah.

Speaker:

Sorry, sorry.

Speaker:

Uh, sorry.

Speaker:

Ganda.

Speaker:

We're gonna call you stupid admin.

Speaker:

Oh, okay.

Speaker:

Um, yeah.

Speaker:

So that, that was, you know, back in the day when, when, you

Speaker:

know, I got, we, we, you could.

Speaker:

Hmm.

Speaker:

We, we had 12,000 employees and we had a help desk, and you could call in.

Speaker:

This was, you know, pre-web, pre, you know, everything.

Speaker:

You, you, you lost a file.

Speaker:

You called into the help desk, and the help desk would maybe try to

Speaker:

help you find the file, but if not, they would issue a restore request.

Speaker:

We got a lot of restore requests, very regular.

Speaker:

Another reason why I got good at doing this, because unlike a lot of places, we

Speaker:

did new backups just to do them, we did restores on a very regular basis, right.

Speaker:

There was this person who called and, um, they, they, they asked to have a file

Speaker:

restored and it was called Resume Doc.

Speaker:

And we were like, really?

Speaker:

Like that's the what happened to Resume doc?

Speaker:

Well, I was updating it.

Speaker:

Where are you?

Speaker:

Is that right?

Speaker:

Yes.

Speaker:

I was updating my re resume dot doc and um, I, I, you know,

Speaker:

I fat fingered it basically.

Speaker:

And, uh, and I, you know,

Speaker:

to resume?

Speaker:

yeah.

Speaker:

What were you resuming?

Speaker:

Yeah, trying to resume your career.

Speaker:

That might've been, you know, depending on who we told that might've

Speaker:

been a resume producing event.

Speaker:

Uh, uh, yeah, so that was, that one was kind of funny.

Speaker:

But, but I think that the, the, the, the, the story that really drives

Speaker:

home the, you know, because, you know, admins are not perfect, right?

Speaker:

And you, um.

Speaker:

You, you know, you do things, you do things to, to make things better,

Speaker:

but sometimes you do the wrong thing.

Speaker:

Right?

Speaker:

I mean, I think of like the KPMG.

Speaker:

The K-P-M-G-I think is like the, that's like the, the best one that I had nothing

Speaker:

to do with, where they're just trying to delete one guy's chats and, and they

Speaker:

were, they were being stopped by Microsoft Retention policies and so they made a new

Speaker:

retention policy that had no retention and then they thought they moved the

Speaker:

one guy into it, but instead they moved.

Speaker:

The entire company into it, and they deleted it was like 150,000 people's

Speaker:

chats, uh, like, you know, like that.

Speaker:

And, and, um, and there was no backup.

Speaker:

Right?

Speaker:

But, uh, but yeah, this other, this other story, uh, is, is a good one.

Speaker:

And you know, Gonda, we love you.

Speaker:

Um, so she, the, we had a file server.

Speaker:

I remember it was HP FS oh one, HP file server.

Speaker:

You know, one, and we had a lot of employees and we had a lot of turnover.

Speaker:

And over time, the, and, and this is, this is, you know, old school Unix.

Speaker:

You had home one, you know Curtis.

Speaker:

Right?

Speaker:

And after the, the person left the company, we did not have an

Speaker:

off-boarding process would be the, uh, or if we had one, it didn't

Speaker:

include deleting the old data.

Speaker:

And so somebody said, we think we have a lot of.

Speaker:

Of directories out there.

Speaker:

And, and we knew that we had all of this on tape and then we knew that we

Speaker:

kept all of our tapes for seven years.

Speaker:

And so we knew that if, if anybody ever actually needed

Speaker:

this data, we could get it back.

Speaker:

And so they hired, uh, my friend to, uh, to programmatically work their way

Speaker:

through the home directory and then delete any directories that were not.

Speaker:

From a valid user.

Speaker:

Okay.

Speaker:

And so the directories were named after the user.

Speaker:

And so you would see, you know, the c Preston directory.

Speaker:

And then she would look and, you know, she would just traverse the tree and she

Speaker:

would find the, the directory name, and then she would look to see if there was a

Speaker:

user by that name and the password file.

Speaker:

And then if the user wasn't there, she would delete it.

Speaker:

And, um, the um, but.

Speaker:

There was, she made one small, one small calculation or miscalculation,

Speaker:

and that is that there was a two level directory structure, so it was, it was

Speaker:

like Home one slash a slash Alfred,

Speaker:

Mm-hmm.

Speaker:

know, home one slash c slash c Preston.

Speaker:

And um, so she worked her way down the tree.

Speaker:

She got the home one A.

Speaker:

Is there a username, a. In the, in the tree, in the password file.

Speaker:

Nope.

Speaker:

All right.

Speaker:

Delete a

Speaker:

Yeah.

Speaker:

Oh, no.

Speaker:

And she, and she was like a third of the way through, uh, the directories

Speaker:

when people started, they, they, they were unable to log in 'cause

Speaker:

they had no home directory.

Speaker:

And, uh, we got a whole bunch of calls to the thing.

Speaker:

And luckily she, she stopped it out.

Speaker:

And I, I still remember she called me that day and her, her first words

Speaker:

were, she's like, Curtis, um, how were the backups of HP FSO one last night?

Speaker:

And I'm like.

Speaker:

Um, they were fine.

Speaker:

Why are you asking?

Speaker:

And she, uh, we proceed to restore all the home directories.

Speaker:

Um,

Speaker:

yeah, that's, uh, that's a scary thing because imagine if

Speaker:

you didn't have those backups,

Speaker:

yeah,

Speaker:

right?

Speaker:

That's a third of the company's data.

Speaker:

AOG or users on that

Speaker:

yeah, yeah.

Speaker:

Um, I,

Speaker:

And your data too, because

Speaker:

what's that?

Speaker:

And your data too, because you're in the cs.

Speaker:

That's right.

Speaker:

Yes.

Speaker:

I was gone.

Speaker:

Um, uh, the, uh, you, you probably would've been safe, you know, with

Speaker:

the, with the being and the peas there.

Speaker:

Um, there's another, another great, uh, sort of, um, uh, do

Speaker:

you have, do you have a story?

Speaker:

want you to tell another story,

Speaker:

Oh, okay.

Speaker:

Okay.

Speaker:

So there's another great, um.

Speaker:

Story.

Speaker:

It, it, it actually involves the deletion of backup data.

Speaker:

But it, it, it drives home this issue of basically users

Speaker:

make or admins make mistakes.

Speaker:

And this has to do, this has to do with your, your former employer, uh, legato.

Speaker:

Um, and they had, they introduced a feature.

Speaker:

Uh, two features actually in, in, it seemed like it was version five maybe.

Speaker:

And one of the problems was when you were, when, when you, when you

Speaker:

made, when you expired tapes and you were gonna reuse those tapes,

Speaker:

you would relabel those tapes.

Speaker:

And by that I mean you would put an electronic label on the front of the

Speaker:

tape, and by doing that, you would, you effectively erase the rest of the tape

Speaker:

because it puts an end of data marker at the end of the, the, you know, the, um.

Speaker:

The label and then you, you can't get past that with a tape drive.

Speaker:

And so the um.

Speaker:

But the problem was even if you had like several tape drives, um, the process,

Speaker:

it would label the tapes one at a time.

Speaker:

So if you had a whole bunch of tapes, it, it took a long time.

Speaker:

And then also when you would, uh, label it, it would, uh,

Speaker:

confirm the deletion each time.

Speaker:

Okay.

Speaker:

you sure you wanna relabel the state?

Speaker:

Yeah.

Speaker:

And so they introduced the fast and silent.

Speaker:

Uh, option where they would use every available tape drive in the

Speaker:

library and they would not ask.

Speaker:

Okay?

Speaker:

So if you check those two boxes, then, um, basically you would, um, you know, you

Speaker:

would relabel whatever tapes you selected.

Speaker:

But this was accompanied by a bug.

Speaker:

You no.

Speaker:

And the bug was that if you, if, well, if you had a list of tapes.

Speaker:

In a, in a tape library, and you double clicked on one of those tapes, you would

Speaker:

expect that what would pop up would be a dialogue that would include just

Speaker:

that tape that you just clicked on.

Speaker:

That's not how it worked.

Speaker:

If you clicked on the tape, what would pull up was slot one to slot,

Speaker:

however many slots there were.

Speaker:

Oh no.

Speaker:

Right.

Speaker:

So if you wanted to relabel just this one tape, you would double click it.

Speaker:

And if you weren't paying attention, it would pop up a dialogue box.

Speaker:

And if you click yes, fast and silent, go, what you just told it to do was

Speaker:

relabel every tape in the tape library.

Speaker:

Oh no.

Speaker:

And I was there when, uh, I. When, uh, it was actually a Legato employee, uh, that

Speaker:

was at the, it was a, it was a healthcare company up in, up in the LA area.

Speaker:

And he, uh, he managed to relabel this customer's tapes, all of them.

Speaker:

Um, again, I think he got like halfway through the tape library

Speaker:

before he realized what he had done.

Speaker:

there's no going back, right?

Speaker:

Mm-hmm.

Speaker:

Mm. Because once you put that, again, once you put the electronic label on

Speaker:

the front, it puts an end of data marker after that, and you can't get past that.

Speaker:

Um, I hope the customer had copies of those tapes, but the

Speaker:

customer was not very happy.

Speaker:

I would not be either.

Speaker:

I.

Speaker:

yeah.

Speaker:

So

Speaker:

Yeah.

Speaker:

we've been talking about this for a while, but there is a third story

Speaker:

or fourth story I want you to tell.

Speaker:

Oh, you have another story.

Speaker:

All right.

Speaker:

want you to tell the story about, uh, temp and source code.

Speaker:

That's a good one.

Speaker:

That's a good one.

Speaker:

Yeah.

Speaker:

So, so for those that, that, that don't live in the Unix world, you know,

Speaker:

temp, uh, you know, slash TMP or slash TMP depending on which os you were

Speaker:

talking about, which, you know, which distribution was typically where you,

Speaker:

you just put garbage files, right?

Speaker:

You, you, it's temp, it's temporary.

Speaker:

Right?

Speaker:

And, um, the, um, and, and you just use it to, for garbage files, right?

Speaker:

Uh, for scripting and stuff like that.

Speaker:

And, um, the, uh, and, and then on one, some of the distributions and

Speaker:

specifically on HPUX, which is what we were running, temp, was actually in Ram.

Speaker:

Mm.

Speaker:

And so when you rebooted the box, anything in temp would go, bye-bye.

Speaker:

Yeah.

Speaker:

Right?

Speaker:

It's temp, that's the point of temp. And we, uh, one day I got a, um,

Speaker:

a restore request for a directory structure in temp, and I said.

Speaker:

What, what is this?

Speaker:

And they go, oh, well, it's a, it's a source code tree that we've been

Speaker:

working on, this group of consultants, like 20 of us or something.

Speaker:

We've been working on it for a few months and it was in temp and

Speaker:

this, the box, it got rebooted or you know, it just, it got deleted.

Speaker:

They said like, they didn't know how it got deleted.

Speaker:

It got deleted.

Speaker:

I'm like, it's in temp. And they're like, yeah.

Speaker:

I'm like, we don't back up Temp. We, we never have backed

Speaker:

up temp. It's temp, right?

Speaker:

It's like, it's like backing up your garbage can, like what,

Speaker:

what, what is the point of that?

Speaker:

Right?

Speaker:

And um, and so they're like, well, you, you don't understand like, heads are gonna

Speaker:

roll if we lose this, this source country.

Speaker:

And I'm like, not my head like, yeah.

Speaker:

was the genius who decided to put it in temp?

Speaker:

Yeah, some.

Speaker:

It was a developer.

Speaker:

It was a developer, like a whole team of developers, and nobody was like,

Speaker:

should, is this where we should put this?

Speaker:

So know we're, we've been focused on like human error doing things accidentally,

Speaker:

Yeah.

Speaker:

right?

Speaker:

And it's not necessarily just an end user who could be at fault.

Speaker:

It could be an admin, it could just be company processes are broken.

Speaker:

Right?

Speaker:

But it's still like.

Speaker:

I think the key here is asking questions before things go wrong, so you know how

Speaker:

your data's being backed up and where it's

Speaker:

Right, right.

Speaker:

And, you know, and, and, and a lot of the, a lot of these problems, uh,

Speaker:

could be fixed by versioning, right.

Speaker:

Snapshots and things like that.

Speaker:

So we don't have to use the quote unquote backup system, but I, I

Speaker:

do think that this is like the number one reason that we do that.

Speaker:

We do restores right.

Speaker:

Do you think it's still the case today?

Speaker:

Absolutely.

Speaker:

Well, I mean, I think right now the number one, well again, the number one

Speaker:

reason we do restores are, are, is humans.

Speaker:

Right?

Speaker:

We probably do more restores having to do with cyber incidents than

Speaker:

we do stupid user, uh, incidents.

Speaker:

But they, but they still happen.

Speaker:

They still happen all the time,

Speaker:

Do

Speaker:

right?

Speaker:

a lot of the user.

Speaker:

Because you mentioned like the help desk, right?

Speaker:

People would call in, they would ask for things.

Speaker:

Do you think though, with a lot of the technologies out there that users

Speaker:

are now doing sort of self-service restores where they don't need to be

Speaker:

embarrassed by, Hey, I accidentally deleted this file, my resume dot

Speaker:

doc, can you please bring it back?

Speaker:

Yeah, I think, well, I think a couple of, I think a couple of things there.

Speaker:

One is, uh, there's a lot of shadow it going on where people are doing their

Speaker:

own backups or something like backups.

Speaker:

Uh, and so they, they use that instead of having to, um, you know, call a help desk.

Speaker:

And then the other is, I do think that there are a lot of technologies that

Speaker:

have been deployed in data centers.

Speaker:

Snapshots being the primary one.

Speaker:

That have allowed users to essentially do their own restore, right?

Speaker:

They know I can go to ~Snapshot and I can go in there and I can get whatever

Speaker:

I want from whatever timeframe I want, and I, and I can get it from an hour ago,

Speaker:

Yep.

Speaker:

Just before I did the stupid thing.

Speaker:

Uh, so I do think that.

Speaker:

There's probably less of a reason for people to do, to call into an actual

Speaker:

help desk or to go to a website and say, Hey, I need this file restored.

Speaker:

Um, because there's a lot of self service.

Speaker:

What's that?

Speaker:

What's that?

Speaker:

or the trash bin.

Speaker:

Right.

Speaker:

Yeah, yeah, yeah.

Speaker:

have a trash bin where you can

Speaker:

Yeah.

Speaker:

your data.

Speaker:

Well, the trash bin is usually only helpful if you delete files.

Speaker:

If you corrupt the files, trash bins aren't usually helpful.

Speaker:

Yeah.

Speaker:

Um, and, you know, thi this, this is one of the things that

Speaker:

I used to hear about Salesforce.

Speaker:

I go, well, Salesforce has a, you know, it has a, again, a trash bin.

Speaker:

And I'm like, right.

Speaker:

But when you modify files, when you modify, uh, what they would call, um.

Speaker:

Objects.

Speaker:

call it records, actually, an object.

Speaker:

They, they use the term object very, in my opinion, very weirdly to them,

Speaker:

an object is like the user table.

Speaker:

Oh, that's

Speaker:

That's an object.

Speaker:

I never understood that, but whatever, uh, I would call it a record, right?

Speaker:

When you modify a record, there is no place where the previous

Speaker:

version of that record is stored.

Speaker:

Yep.

Speaker:

so, um, and I remember one time where I, um.

Speaker:

I screwed up my, we had a Salesforce database that had a couple million records

Speaker:

Hmm

Speaker:

and I screwed up every million of 'em by accidentally.

Speaker:

Um, first off, it should be a federal crime that when you have an Excel

Speaker:

spreadsheet and you click on a column

Speaker:

That

Speaker:

and then you say, sort.

Speaker:

column and not everything.

Speaker:

That should be, that should, whoever came up with that feature

Speaker:

needs to be tarred and feathered.

Speaker:

How is that?

Speaker:

What, in what world is that what anyone wants to do?

Speaker:

Right.

Speaker:

I mean, I, I think now when you do it, it it, it'll, it'll say, did you mean to

Speaker:

sort, you know, and you just have to push.

Speaker:

You've got, yeah, no, I want to sort the whole thing, but like,

Speaker:

why would that ever be the case?

Speaker:

You know what I mean?

Speaker:

It should be like, it should be the, like, what they did in AWS with o

Speaker:

with open buckets where you have to like really, really, really try

Speaker:

really hard to do an open bucket.

Speaker:

It should be like that.

Speaker:

Um, and if you, if you actually want to do a sort by just that

Speaker:

column, you should have to really, you should have to really say so.

Speaker:

It is like provide a blood oat and

Speaker:

Yeah.

Speaker:

Yeah.

Speaker:

Because, you know, once you do that, uh, and so what I, what

Speaker:

I did was I accidentally sorted the, uh, I, I downloaded the phone

Speaker:

number column, which was, um.

Speaker:

All the phone numbers, they're formatted, like all different.

Speaker:

And so I figured out, I, I wrote a program to basically take out all

Speaker:

of the formatting and then put back the formatting the way I wanted it.

Speaker:

Right.

Speaker:

Except I sorted it.

Speaker:

Oh,

Speaker:

And so when I uploaded that back to Salesforce using the data loader, I

Speaker:

think that's what that was called.

Speaker:

It's been a while.

Speaker:

Um.

Speaker:

It.

Speaker:

I just put all the wrong phone numbers to all the wrong places,

Speaker:

but luckily I had downloaded

Speaker:

Uh

Speaker:

table prior to doing that.

Speaker:

Luckily, I had a backup there.

Speaker:

Um.

Speaker:

So do you wanna even touch on the second category?

Speaker:

Because I think that's basically what we talk about on the

Speaker:

podcast, like 97.2% of the time.

Speaker:

Oh, the, the, um, the, um, the cybersecurity stuff?

Speaker:

Yeah.

Speaker:

Um, just, just a little bit.

Speaker:

I do wanna specifically talk about the insider threat 'cause I don't

Speaker:

think we talk about that as much.

Speaker:

Right.

Speaker:

Um, and I do think that you need to.

Speaker:

Design your backup systems to be resilient against the insider threat.

Speaker:

And we don't, and I don't think that people think about that enough.

Speaker:

what's the

Speaker:

think,

Speaker:

thread?

Speaker:

Thread?

Speaker:

basically, it's someone who it's, it's a, it's a person who, um.

Speaker:

Works in on the inside, they are one of your employees or contractors

Speaker:

who have full access to whatever system we're talking about, and

Speaker:

either they have been compromised or their user ID has been compromised.

Speaker:

Right?

Speaker:

Um, it, so typically what we mean, we're referring to them

Speaker:

being compromised, right?

Speaker:

They're, they're, you know,

Speaker:

a bad actor inside the

Speaker:

they're a bad actor inside the company.

Speaker:

The, the most famous or infamous one,

Speaker:

so this is a story from 2002, and the story that I'm looking at is actually

Speaker:

a very ugly thing here on justice.gov and it talks about basically, um.

Speaker:

I'll just read the opening sentence here.

Speaker:

A disgruntled computer systems administrator for UBS was charged

Speaker:

today with using a logic bomb

Speaker:

Mm.

Speaker:

cause more than $3 million in damage to the company's computer network and

Speaker:

with securities fraud for his failed plan to drive the company's stock down

Speaker:

with the activation of the logic bomb.

Speaker:

You know the other story,

Speaker:

Yeah.

Speaker:

doing the one about unify where they claimed that they were being

Speaker:

hacked and it was an insider?

Speaker:

All right.

Speaker:

Right.

Speaker:

That,

Speaker:

years ago.

Speaker:

yeah.

Speaker:

Yeah, yeah, yeah.

Speaker:

So, you know, if you have somebody that's on the inside, they can do a lot of damage

Speaker:

to your systems without, um, without you, before you're able to stop them.

Speaker:

with great power comes great responsibility.

Speaker:

Yeah.

Speaker:

Well, and I will also say with great power comes great restriction

Speaker:

because, because the more powerful a person's role is, the more controls

Speaker:

should be placed around that power.

Speaker:

Right.

Speaker:

Um, and you should.

Speaker:

You know, everything that you do should be logged and, you know, and, and,

Speaker:

and the more destructive things you should be able to have to, you should

Speaker:

have to get, you know, multi person authentication, you know, otherwise

Speaker:

known as four eyes authentication.

Speaker:

Um, and, um.

Speaker:

Because, and, and this is just, just, you have to think about that because not

Speaker:

only are you protecting from the insider threat, you're protecting from a bad

Speaker:

actor, getting a, getting access there.

Speaker:

There's been a lot of stories right with that.

Speaker:

There was of course the LastPass incident, which is, you know, was the

Speaker:

nail in the coffin for me regarding LastPass, uh, where basically they,

Speaker:

the, the hacker was able to gain access to a backup of the vault.

Speaker:

Because they had a, they had, they had, they weren't able to decrypt the vault,

Speaker:

but they were get a, they were able to get access to the vault because there

Speaker:

was this backup script that had hard coded, um, you know, credentials in it.

Speaker:

And then once they gained there, there was a, there was an initial breach

Speaker:

and then that breach gave them access to see the script, which then gave

Speaker:

them access to the credentials, which gave them access to the backup, which

Speaker:

gave them access to, uh, the vault.

Speaker:

Which was, I think then used for targeted crypto attacks.

Speaker:

was used for targeted crypto attacks against people.

Speaker:

Yeah.

Speaker:

Um, and so really all we're, you know, we're just talking about, just

Speaker:

realize that human error is still one of the val, the biggest reasons that

Speaker:

we do anything in the backup world.

Speaker:

And, and when we talk about this just.

Speaker:

Try to implement the concept of least privilege.

Speaker:

Try to implement the concept of minimizing the blast radius, minimizing the ability

Speaker:

for somebody to do something or, you know, the, the more dangerous something is,

Speaker:

the more controls you can put around it.

Speaker:

Uh, or should put around it.

Speaker:

so going to a sentence, you just said,

Speaker:

Yeah.

Speaker:

humans are always at risk,

Speaker:

Humans.

Speaker:

Yeah.

Speaker:

Right.

Speaker:

Yeah.

Speaker:

So.

Speaker:

Humans are always de-risk, I would say.

Speaker:

Yeah.

Speaker:

So what if people are like, Hey, forget about humans.

Speaker:

Let's switch over to ai.

Speaker:

No, no, no.

Speaker:

I, I wanna play a thought experiment,

Speaker:

uh.

Speaker:

What, uh, would your views change?

Speaker:

Would your, you would recommend, because for humans, right?

Speaker:

We talk about putting controls, doing multifactor authentication, multi

Speaker:

authentication, All of these things, if people start to move towards ai.

Speaker:

How does this change?

Speaker:

I know this episode is talking about human error, right?

Speaker:

But just to leave

Speaker:

don't, I don't think I'm ready to answer that question.

Speaker:

We'll, we'll cross that bridge when we come to it.

Speaker:

How's that?

Speaker:

Okay.

Speaker:

I, well, I'll just say this in general, I'm a fan of automation and AI can

Speaker:

enable automation, but it's gotta be put into a, it's gotta beral.

Speaker:

and

Speaker:

Yeah.

Speaker:

else

Speaker:

Yeah.

Speaker:

Again,

Speaker:

may be

Speaker:

with

Speaker:

but

Speaker:

power

Speaker:

yeah.

Speaker:

great.

Speaker:

You know, responsibility.

Speaker:

Right.

Speaker:

And so you want to, you want to, um, um, sorry.

Speaker:

I just found myself thinking about, um,

Speaker:

Spider-Man.

Speaker:

Stanwell Stan Lee.

Speaker:

I wonder if Stan Lee thinks about d the fact that his Spider-Man line would be

Speaker:

quoted at a IT podcast decades later.

Speaker:

Um.

Speaker:

The, uh, yeah.

Speaker:

So the more power the AI has, the more control you gotta put around it.

Speaker:

Because it could do things right.

Speaker:

It could be used to do things.

Speaker:

Yeah.

Speaker:

All right.

Speaker:

Well, this has been fun, sad, depressing, uh, et cetera, but that's why we back up.

Speaker:

And,

Speaker:

And, um,

Speaker:

and that's why we're human.

Speaker:

yes.

Speaker:

And I, I wanna wish you, you, you have an upcoming trip.

Speaker:

Uh.

Speaker:

I

Speaker:

Yeah, wanna wish you a good trip, Bon voyage.

Speaker:

Thank you.

Speaker:

Thank you.

Speaker:

I hope our listeners get to have a trip coming up.

Speaker:

You know, maybe you have some fun.

Speaker:

At least somebody will have some fun.

Speaker:

And, uh, with that, that is a wrap.

Speaker:

The backup wrap up is written, recorded, and produced by me w Curtis Preston.

Speaker:

If you need backup or Dr. Consulting content generation or expert witness

Speaker:

work, check out backup central.com.

Speaker:

You can also find links from my O'Reilly Books on the same website.

Speaker:

Remember, this is an independent podcast and any opinions that

Speaker:

you hear are those of the speaker and not necessarily an employer.

Speaker:

Thanks for listening.