You found the backup wrap up your go-to podcast for all things
Speaker:backup recovery and cyber recovery.
Speaker:In this episode, we look at the reality of modern data centers.
Speaker:Human error has become the number two reason that we do restores today.
Speaker:Number one, of course, would be cyber attacks.
Speaker:Back in the day, you know, a hundred years ago when I first started, it was
Speaker:really all about hardware failures, hard drives, crashing, servers dying.
Speaker:But now it's people making mistakes.
Speaker:Sometimes it's a user accidentally deleting a file, or maybe it's an admin
Speaker:wiping out an entire directory structure.
Speaker:Humans are the weakest link.
Speaker:I've got, I don't know, four or five war stories that'll make you cringe.
Speaker:And we talk about insider threats as well, and why your backup systems
Speaker:need to be designed with human error and human frailty in mind.
Speaker:Trust me, these stories will remind you why we do what we do.
Speaker:By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr.
Speaker:Backup, and I've been passionate about backup and recovery ever
Speaker:since I had to explain to my boss.
Speaker:While there were no backups of the production database that
Speaker:we had just lost, I don't want that to ever again happen to me.
Speaker:I don't want it to happen to you.
Speaker:That's why I do this.
Speaker:On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes.
Speaker:This is the backup wrap up.
Speaker:Welcome to the show.
Speaker:Hi, I am w Curtis Preston, AKA, Mr. Backup, and with me, I once again have
Speaker:my security deposit consultant persona.
Speaker:Molly, Andi, how's it going?
Speaker:Persona,
Speaker:I am good, Curtis, although I thought you would've said you're a person who
Speaker:was successful at DIYing something for probably the first time in their life.
Speaker:a guy that, a guy that finally joined the ranks because, because you're
Speaker:completed now with your, with your.
Speaker:I am.
Speaker:If people
Speaker:Yeah,
Speaker:episode, I, uh, had to redo some.
Speaker:stuff for
Speaker:yeah,
Speaker:system, and it was a little painful.
Speaker:yeah,
Speaker:lie.
Speaker:Um, and especially because I had to replace a shutoff valve, which meant I had
Speaker:to shut off the water for the entire house
Speaker:yeah.
Speaker:I was repair, replacing that shutoff valve.
Speaker:And then it turned out that I was leaking and blah, blah, blah, blah, blah.
Speaker:So, eventually got around to it, turned out, um, make sure you wrap the
Speaker:Teflon tape in the right direction.
Speaker:Yes, yes, yes.
Speaker:You figured that out, didn't you?
Speaker:Yes.
Speaker:And also having the right tools helps, especially a pipe wrench.
Speaker:Yes.
Speaker:More than one, right?
Speaker:Yeah.
Speaker:So once I did that, then actually doing all the PVC gluing and
Speaker:That part was easy, right?
Speaker:So I remember purple stuff first, then the blue stuff.
Speaker:Yeah, yeah,
Speaker:I should say.
Speaker:yeah.
Speaker:Oh, you use clear stuff?
Speaker:Yeah.
Speaker:Oh, that explains why I saw the blue.
Speaker:I saw the purple.
Speaker:Yeah.
Speaker:Okay.
Speaker:But it worked and it held, and it's knock on wood.
Speaker:So far so good.
Speaker:So
Speaker:Well, I'm glad to hear that you can now sprinkle your yard once again.
Speaker:I of course don't have that challenge 'cause I have a fake yard.
Speaker:well it's not a yard, it's just for the plants and all the drip irrigation, so.
Speaker:Oh, gotcha, gotcha.
Speaker:It's a yard, it's just, it's not grass.
Speaker:saw grass.
Speaker:Yeah.
Speaker:Right, right.
Speaker:especially last weekend.
Speaker:No, over the weekend, I think the temperature at 87 here in the Bay
Speaker:Area and today it's 61, so yeah.
Speaker:Yeah, we actually, we had a heat wave here as well.
Speaker:Um, it actually got up to like 90 at one point,
Speaker:Dang.
Speaker:Today it was cold and rainy, so
Speaker:Yeah.
Speaker:I think may.
Speaker:Gray back, no Uns shining outside.
Speaker:Hmm.
Speaker:I don't know.
Speaker:Who knows?
Speaker:You never know in San Diego, right?
Speaker:Especially in May.
Speaker:Right.
Speaker:May Gray and June gloom.
Speaker:For those that don't live here, it's pretty much overcast all the
Speaker:time for two months of the year.
Speaker:But it's why the temperature is,
Speaker:Even Keter.
Speaker:Yeah.
Speaker:Yeah.
Speaker:Um,
Speaker:dude.
Speaker:come here.
Speaker:What?
Speaker:ahead, finish.
Speaker:Well, I just said people come here and they, um, they're like, why?
Speaker:Where?
Speaker:I'm like, this is like, do you do no research on the places you go to?
Speaker:Right,
Speaker:Well, it's like the time that, or I think that twice that I've been up to
Speaker:Seattle and both times it was nice and sunny and so in my mind it's imprinted
Speaker:that Seattle is always nice and sunny.
Speaker:Yeah.
Speaker:Oh yeah.
Speaker:I, yeah, I spent, I spent four months in Seattle in 1998, and
Speaker:it was the most perfect four months in the history of Seattle.
Speaker:It rained like two days during those four months, and it was just, and
Speaker:yeah, I could, I could have moved there, but it's not normally like that.
Speaker:exactly.
Speaker:Yeah.
Speaker:But, um, well, uh, today, speaking of.
Speaker:Humans making mistakes today.
Speaker:I thought we'd talk a little bit about human error.
Speaker:Um, and, uh, as, as I often, uh, you know, find myself doing, I'll talk about
Speaker:back in the day, back in the day, um.
Speaker:The reason when I started my IT career, the, the number one reason
Speaker:we were doing restores was, um, actual hardware failure, right?
Speaker:Because I am Prera, right?
Speaker:Um, we, we had mission critical servers running on individual hard drives,
Speaker:Which is bonkers.
Speaker:A
Speaker:right?
Speaker:Right.
Speaker:do you think, what is, what is it like this, isn't there like a satellite
Speaker:that's been out there for like 40 years
Speaker:Oh yeah, yeah,
Speaker:and do you think that still also runs on a single disk?
Speaker:I,
Speaker:yeah.
Speaker:Def absolutely.
Speaker:Yeah.
Speaker:crazy.
Speaker:Yeah.
Speaker:Well, it might, do they have, do they have disc drives?
Speaker:I don't know.
Speaker:No.
Speaker:I mean, it's, if it's been there that long.
Speaker:Uh, but yeah, it, it's, I mean, I'm pretty sure RAID was invented before I actually
Speaker:saw it in the data center, but in my data center, which was a, a big, you know,
Speaker:$35 billion company, we didn't have any.
Speaker:Right.
Speaker:And, um,
Speaker:Was it
Speaker:and go ahead.
Speaker:was it the difference though between raid on open systems versus raid on mainframes?
Speaker:Yeah, I don't, yeah, I don't know.
Speaker:Um, it just, it certainly wasn't, you know, I remember the first
Speaker:time and I talked about it a few episodes the first time I saw
Speaker:hot swappable disc drives, right?
Speaker:Where you, you had the raid arrays, right.
Speaker:Um, and all of that sort of, it sort of all went down right around
Speaker:that timeframe, but for a few years.
Speaker:We were, we were losing entire databases or servers because an
Speaker:individual R drive went down.
Speaker:And so we, we were restoring that stuff all the time.
Speaker:And in fact, um, my first, once I left the bank and I went into
Speaker:consulting, um, one of the reasons I got so good at Bare Metal Recovery
Speaker:was that I was doing it all the time.
Speaker:Um, because I was at a. Uh, the headquarters of a
Speaker:large oil and gas company.
Speaker:And, um, they had all these servers that had not been, uh,
Speaker:maintained in a really long time.
Speaker:And so we were doing crazy things like installing patches and rebooting them
Speaker:with significant portion of the time.
Speaker:They just didn't come back up.
Speaker:So,
Speaker:makes perfect though, right Curtis?
Speaker:yeah.
Speaker:And so, yeah.
Speaker:Um, and so my, my, my point is that back then.
Speaker:A significant portion of the time, the reason you were doing a restore, it
Speaker:was because the actual hardware failed, failed, and that is just pretty much.
Speaker:Gone.
Speaker:I, you know, uh, between the fact that we now use solid state a as hard
Speaker:drives, and the fact that we use that, that no one, no one does anything
Speaker:important on, on an individual.
Speaker:I mean, your laptop, your phone, these are individual drives, but not
Speaker:nothing in a, in a server, right?
Speaker:Um, and.
Speaker:and also probably just reliability of components has also gone up.
Speaker:Absolutely.
Speaker:The result of all of that is that at this point the, there are,
Speaker:generally speaking only two reasons.
Speaker:We restore things and both of them had to do with people, right?
Speaker:Either people purposefully did something to damage the systems or they accidentally
Speaker:did something to damage the systems.
Speaker:Are you with me?
Speaker:Yep.
Speaker:Yeah, so
Speaker:failure.
Speaker:it is not a hardware failure, right?
Speaker:So either bad people, you know, bad actors, you know, it's a si,
Speaker:it's a, it's a ransomware attack, it's a cyber attack of some sort.
Speaker:Uh, or stupid users
Speaker:Would you consider like the OVH
Speaker:or stupid admins.
Speaker:So, so that is still, that is still a human error in my opinion.
Speaker:I mean, okay, so what, we can add that as a third category, which
Speaker:is natural disasters, right?
Speaker:Um, so if, if a fire happens, you know, um, yeah, I, we'll, we'll put a fire in
Speaker:a natural disaster, although that fire, I'm not sure was a natural disaster.
Speaker:But yeah, so that, that, that is a third reason, you know, I, I'll take that.
Speaker:But, but, but even that, the reason they end up having to do a restore,
Speaker:I think, was human error, right?
Speaker:Meaning that they, they just configured everything all wrong.
Speaker:Um, and I, and that's my story, and I'm sticking to it.
Speaker:so let's talk about the accidental humans
Speaker:Yeah.
Speaker:Yeah.
Speaker:'cause we talk a lot about the, the, the bad actors.
Speaker:Yeah.
Speaker:Um, so let's talk about the, you know, and, and basically what this
Speaker:kind of boils down to is, uh, either stupid users or stupid admins.
Speaker:Right?
Speaker:I, so there are two stories I want you to tell,
Speaker:Okay.
Speaker:The first one is, uh, restoring a document.
Speaker:The, the document called resume,
Speaker:Yeah.
Speaker:Yeah.
Speaker:And
Speaker:Uh.
Speaker:second one I want you to talk about is the file server with the user home directories
Speaker:Lord.
Speaker:Yeah, yeah.
Speaker:Sorry, sorry.
Speaker:Uh, sorry.
Speaker:Ganda.
Speaker:We're gonna call you stupid admin.
Speaker:Oh, okay.
Speaker:Um, yeah.
Speaker:So that, that was, you know, back in the day when, when, you
Speaker:know, I got, we, we, you could.
Speaker:Hmm.
Speaker:We, we had 12,000 employees and we had a help desk, and you could call in.
Speaker:This was, you know, pre-web, pre, you know, everything.
Speaker:You, you, you lost a file.
Speaker:You called into the help desk, and the help desk would maybe try to
Speaker:help you find the file, but if not, they would issue a restore request.
Speaker:We got a lot of restore requests, very regular.
Speaker:Another reason why I got good at doing this, because unlike a lot of places, we
Speaker:did new backups just to do them, we did restores on a very regular basis, right.
Speaker:There was this person who called and, um, they, they, they asked to have a file
Speaker:restored and it was called Resume Doc.
Speaker:And we were like, really?
Speaker:Like that's the what happened to Resume doc?
Speaker:Well, I was updating it.
Speaker:Where are you?
Speaker:Is that right?
Speaker:Yes.
Speaker:I was updating my re resume dot doc and um, I, I, you know,
Speaker:I fat fingered it basically.
Speaker:And, uh, and I, you know,
Speaker:to resume?
Speaker:yeah.
Speaker:What were you resuming?
Speaker:Yeah, trying to resume your career.
Speaker:That might've been, you know, depending on who we told that might've
Speaker:been a resume producing event.
Speaker:Uh, uh, yeah, so that was, that one was kind of funny.
Speaker:But, but I think that the, the, the, the, the story that really drives
Speaker:home the, you know, because, you know, admins are not perfect, right?
Speaker:And you, um.
Speaker:You, you know, you do things, you do things to, to make things better,
Speaker:but sometimes you do the wrong thing.
Speaker:Right?
Speaker:I mean, I think of like the KPMG.
Speaker:The K-P-M-G-I think is like the, that's like the, the best one that I had nothing
Speaker:to do with, where they're just trying to delete one guy's chats and, and they
Speaker:were, they were being stopped by Microsoft Retention policies and so they made a new
Speaker:retention policy that had no retention and then they thought they moved the
Speaker:one guy into it, but instead they moved.
Speaker:The entire company into it, and they deleted it was like 150,000 people's
Speaker:chats, uh, like, you know, like that.
Speaker:And, and, um, and there was no backup.
Speaker:Right?
Speaker:But, uh, but yeah, this other, this other story, uh, is, is a good one.
Speaker:And you know, Gonda, we love you.
Speaker:Um, so she, the, we had a file server.
Speaker:I remember it was HP FS oh one, HP file server.
Speaker:You know, one, and we had a lot of employees and we had a lot of turnover.
Speaker:And over time, the, and, and this is, this is, you know, old school Unix.
Speaker:You had home one, you know Curtis.
Speaker:Right?
Speaker:And after the, the person left the company, we did not have an
Speaker:off-boarding process would be the, uh, or if we had one, it didn't
Speaker:include deleting the old data.
Speaker:And so somebody said, we think we have a lot of.
Speaker:Of directories out there.
Speaker:And, and we knew that we had all of this on tape and then we knew that we
Speaker:kept all of our tapes for seven years.
Speaker:And so we knew that if, if anybody ever actually needed
Speaker:this data, we could get it back.
Speaker:And so they hired, uh, my friend to, uh, to programmatically work their way
Speaker:through the home directory and then delete any directories that were not.
Speaker:From a valid user.
Speaker:Okay.
Speaker:And so the directories were named after the user.
Speaker:And so you would see, you know, the c Preston directory.
Speaker:And then she would look and, you know, she would just traverse the tree and she
Speaker:would find the, the directory name, and then she would look to see if there was a
Speaker:user by that name and the password file.
Speaker:And then if the user wasn't there, she would delete it.
Speaker:And, um, the um, but.
Speaker:There was, she made one small, one small calculation or miscalculation,
Speaker:and that is that there was a two level directory structure, so it was, it was
Speaker:like Home one slash a slash Alfred,
Speaker:Mm-hmm.
Speaker:know, home one slash c slash c Preston.
Speaker:And um, so she worked her way down the tree.
Speaker:She got the home one A.
Speaker:Is there a username, a. In the, in the tree, in the password file.
Speaker:Nope.
Speaker:All right.
Speaker:Delete a
Speaker:Yeah.
Speaker:Oh, no.
Speaker:And she, and she was like a third of the way through, uh, the directories
Speaker:when people started, they, they, they were unable to log in 'cause
Speaker:they had no home directory.
Speaker:And, uh, we got a whole bunch of calls to the thing.
Speaker:And luckily she, she stopped it out.
Speaker:And I, I still remember she called me that day and her, her first words
Speaker:were, she's like, Curtis, um, how were the backups of HP FSO one last night?
Speaker:And I'm like.
Speaker:Um, they were fine.
Speaker:Why are you asking?
Speaker:And she, uh, we proceed to restore all the home directories.
Speaker:Um,
Speaker:yeah, that's, uh, that's a scary thing because imagine if
Speaker:you didn't have those backups,
Speaker:yeah,
Speaker:right?
Speaker:That's a third of the company's data.
Speaker:AOG or users on that
Speaker:yeah, yeah.
Speaker:Um, I,
Speaker:And your data too, because
Speaker:what's that?
Speaker:And your data too, because you're in the cs.
Speaker:That's right.
Speaker:Yes.
Speaker:I was gone.
Speaker:Um, uh, the, uh, you, you probably would've been safe, you know, with
Speaker:the, with the being and the peas there.
Speaker:Um, there's another, another great, uh, sort of, um, uh, do
Speaker:you have, do you have a story?
Speaker:want you to tell another story,
Speaker:Oh, okay.
Speaker:Okay.
Speaker:So there's another great, um.
Speaker:Story.
Speaker:It, it, it actually involves the deletion of backup data.
Speaker:But it, it, it drives home this issue of basically users
Speaker:make or admins make mistakes.
Speaker:And this has to do, this has to do with your, your former employer, uh, legato.
Speaker:Um, and they had, they introduced a feature.
Speaker:Uh, two features actually in, in, it seemed like it was version five maybe.
Speaker:And one of the problems was when you were, when, when you, when you
Speaker:made, when you expired tapes and you were gonna reuse those tapes,
Speaker:you would relabel those tapes.
Speaker:And by that I mean you would put an electronic label on the front of the
Speaker:tape, and by doing that, you would, you effectively erase the rest of the tape
Speaker:because it puts an end of data marker at the end of the, the, you know, the, um.
Speaker:The label and then you, you can't get past that with a tape drive.
Speaker:And so the um.
Speaker:But the problem was even if you had like several tape drives, um, the process,
Speaker:it would label the tapes one at a time.
Speaker:So if you had a whole bunch of tapes, it, it took a long time.
Speaker:And then also when you would, uh, label it, it would, uh,
Speaker:confirm the deletion each time.
Speaker:Okay.
Speaker:you sure you wanna relabel the state?
Speaker:Yeah.
Speaker:And so they introduced the fast and silent.
Speaker:Uh, option where they would use every available tape drive in the
Speaker:library and they would not ask.
Speaker:Okay?
Speaker:So if you check those two boxes, then, um, basically you would, um, you know, you
Speaker:would relabel whatever tapes you selected.
Speaker:But this was accompanied by a bug.
Speaker:You no.
Speaker:And the bug was that if you, if, well, if you had a list of tapes.
Speaker:In a, in a tape library, and you double clicked on one of those tapes, you would
Speaker:expect that what would pop up would be a dialogue that would include just
Speaker:that tape that you just clicked on.
Speaker:That's not how it worked.
Speaker:If you clicked on the tape, what would pull up was slot one to slot,
Speaker:however many slots there were.
Speaker:Oh no.
Speaker:Right.
Speaker:So if you wanted to relabel just this one tape, you would double click it.
Speaker:And if you weren't paying attention, it would pop up a dialogue box.
Speaker:And if you click yes, fast and silent, go, what you just told it to do was
Speaker:relabel every tape in the tape library.
Speaker:Oh no.
Speaker:And I was there when, uh, I. When, uh, it was actually a Legato employee, uh, that
Speaker:was at the, it was a, it was a healthcare company up in, up in the LA area.
Speaker:And he, uh, he managed to relabel this customer's tapes, all of them.
Speaker:Um, again, I think he got like halfway through the tape library
Speaker:before he realized what he had done.
Speaker:there's no going back, right?
Speaker:Mm-hmm.
Speaker:Mm. Because once you put that, again, once you put the electronic label on
Speaker:the front, it puts an end of data marker after that, and you can't get past that.
Speaker:Um, I hope the customer had copies of those tapes, but the
Speaker:customer was not very happy.
Speaker:I would not be either.
Speaker:I.
Speaker:yeah.
Speaker:So
Speaker:Yeah.
Speaker:we've been talking about this for a while, but there is a third story
Speaker:or fourth story I want you to tell.
Speaker:Oh, you have another story.
Speaker:All right.
Speaker:want you to tell the story about, uh, temp and source code.
Speaker:That's a good one.
Speaker:That's a good one.
Speaker:Yeah.
Speaker:So, so for those that, that, that don't live in the Unix world, you know,
Speaker:temp, uh, you know, slash TMP or slash TMP depending on which os you were
Speaker:talking about, which, you know, which distribution was typically where you,
Speaker:you just put garbage files, right?
Speaker:You, you, it's temp, it's temporary.
Speaker:Right?
Speaker:And, um, the, um, and, and you just use it to, for garbage files, right?
Speaker:Uh, for scripting and stuff like that.
Speaker:And, um, the, uh, and, and then on one, some of the distributions and
Speaker:specifically on HPUX, which is what we were running, temp, was actually in Ram.
Speaker:Mm.
Speaker:And so when you rebooted the box, anything in temp would go, bye-bye.
Speaker:Yeah.
Speaker:Right?
Speaker:It's temp, that's the point of temp. And we, uh, one day I got a, um,
Speaker:a restore request for a directory structure in temp, and I said.
Speaker:What, what is this?
Speaker:And they go, oh, well, it's a, it's a source code tree that we've been
Speaker:working on, this group of consultants, like 20 of us or something.
Speaker:We've been working on it for a few months and it was in temp and
Speaker:this, the box, it got rebooted or you know, it just, it got deleted.
Speaker:They said like, they didn't know how it got deleted.
Speaker:It got deleted.
Speaker:I'm like, it's in temp. And they're like, yeah.
Speaker:I'm like, we don't back up Temp. We, we never have backed
Speaker:up temp. It's temp, right?
Speaker:It's like, it's like backing up your garbage can, like what,
Speaker:what, what is the point of that?
Speaker:Right?
Speaker:And um, and so they're like, well, you, you don't understand like, heads are gonna
Speaker:roll if we lose this, this source country.
Speaker:And I'm like, not my head like, yeah.
Speaker:was the genius who decided to put it in temp?
Speaker:Yeah, some.
Speaker:It was a developer.
Speaker:It was a developer, like a whole team of developers, and nobody was like,
Speaker:should, is this where we should put this?
Speaker:So know we're, we've been focused on like human error doing things accidentally,
Speaker:Yeah.
Speaker:right?
Speaker:And it's not necessarily just an end user who could be at fault.
Speaker:It could be an admin, it could just be company processes are broken.
Speaker:Right?
Speaker:But it's still like.
Speaker:I think the key here is asking questions before things go wrong, so you know how
Speaker:your data's being backed up and where it's
Speaker:Right, right.
Speaker:And, you know, and, and, and a lot of the, a lot of these problems, uh,
Speaker:could be fixed by versioning, right.
Speaker:Snapshots and things like that.
Speaker:So we don't have to use the quote unquote backup system, but I, I
Speaker:do think that this is like the number one reason that we do that.
Speaker:We do restores right.
Speaker:Do you think it's still the case today?
Speaker:Absolutely.
Speaker:Well, I mean, I think right now the number one, well again, the number one
Speaker:reason we do restores are, are, is humans.
Speaker:Right?
Speaker:We probably do more restores having to do with cyber incidents than
Speaker:we do stupid user, uh, incidents.
Speaker:But they, but they still happen.
Speaker:They still happen all the time,
Speaker:Do
Speaker:right?
Speaker:a lot of the user.
Speaker:Because you mentioned like the help desk, right?
Speaker:People would call in, they would ask for things.
Speaker:Do you think though, with a lot of the technologies out there that users
Speaker:are now doing sort of self-service restores where they don't need to be
Speaker:embarrassed by, Hey, I accidentally deleted this file, my resume dot
Speaker:doc, can you please bring it back?
Speaker:Yeah, I think, well, I think a couple of, I think a couple of things there.
Speaker:One is, uh, there's a lot of shadow it going on where people are doing their
Speaker:own backups or something like backups.
Speaker:Uh, and so they, they use that instead of having to, um, you know, call a help desk.
Speaker:And then the other is, I do think that there are a lot of technologies that
Speaker:have been deployed in data centers.
Speaker:Snapshots being the primary one.
Speaker:That have allowed users to essentially do their own restore, right?
Speaker:They know I can go to ~Snapshot and I can go in there and I can get whatever
Speaker:I want from whatever timeframe I want, and I, and I can get it from an hour ago,
Speaker:Yep.
Speaker:Just before I did the stupid thing.
Speaker:Uh, so I do think that.
Speaker:There's probably less of a reason for people to do, to call into an actual
Speaker:help desk or to go to a website and say, Hey, I need this file restored.
Speaker:Um, because there's a lot of self service.
Speaker:What's that?
Speaker:What's that?
Speaker:or the trash bin.
Speaker:Right.
Speaker:Yeah, yeah, yeah.
Speaker:have a trash bin where you can
Speaker:Yeah.
Speaker:your data.
Speaker:Well, the trash bin is usually only helpful if you delete files.
Speaker:If you corrupt the files, trash bins aren't usually helpful.
Speaker:Yeah.
Speaker:Um, and, you know, thi this, this is one of the things that
Speaker:I used to hear about Salesforce.
Speaker:I go, well, Salesforce has a, you know, it has a, again, a trash bin.
Speaker:And I'm like, right.
Speaker:But when you modify files, when you modify, uh, what they would call, um.
Speaker:Objects.
Speaker:call it records, actually, an object.
Speaker:They, they use the term object very, in my opinion, very weirdly to them,
Speaker:an object is like the user table.
Speaker:Oh, that's
Speaker:That's an object.
Speaker:I never understood that, but whatever, uh, I would call it a record, right?
Speaker:When you modify a record, there is no place where the previous
Speaker:version of that record is stored.
Speaker:Yep.
Speaker:so, um, and I remember one time where I, um.
Speaker:I screwed up my, we had a Salesforce database that had a couple million records
Speaker:Hmm
Speaker:and I screwed up every million of 'em by accidentally.
Speaker:Um, first off, it should be a federal crime that when you have an Excel
Speaker:spreadsheet and you click on a column
Speaker:That
Speaker:and then you say, sort.
Speaker:column and not everything.
Speaker:That should be, that should, whoever came up with that feature
Speaker:needs to be tarred and feathered.
Speaker:How is that?
Speaker:What, in what world is that what anyone wants to do?
Speaker:Right.
Speaker:I mean, I, I think now when you do it, it it, it'll, it'll say, did you mean to
Speaker:sort, you know, and you just have to push.
Speaker:You've got, yeah, no, I want to sort the whole thing, but like,
Speaker:why would that ever be the case?
Speaker:You know what I mean?
Speaker:It should be like, it should be the, like, what they did in AWS with o
Speaker:with open buckets where you have to like really, really, really try
Speaker:really hard to do an open bucket.
Speaker:It should be like that.
Speaker:Um, and if you, if you actually want to do a sort by just that
Speaker:column, you should have to really, you should have to really say so.
Speaker:It is like provide a blood oat and
Speaker:Yeah.
Speaker:Yeah.
Speaker:Because, you know, once you do that, uh, and so what I, what
Speaker:I did was I accidentally sorted the, uh, I, I downloaded the phone
Speaker:number column, which was, um.
Speaker:All the phone numbers, they're formatted, like all different.
Speaker:And so I figured out, I, I wrote a program to basically take out all
Speaker:of the formatting and then put back the formatting the way I wanted it.
Speaker:Right.
Speaker:Except I sorted it.
Speaker:Oh,
Speaker:And so when I uploaded that back to Salesforce using the data loader, I
Speaker:think that's what that was called.
Speaker:It's been a while.
Speaker:Um.
Speaker:It.
Speaker:I just put all the wrong phone numbers to all the wrong places,
Speaker:but luckily I had downloaded
Speaker:Uh
Speaker:table prior to doing that.
Speaker:Luckily, I had a backup there.
Speaker:Um.
Speaker:So do you wanna even touch on the second category?
Speaker:Because I think that's basically what we talk about on the
Speaker:podcast, like 97.2% of the time.
Speaker:Oh, the, the, um, the, um, the cybersecurity stuff?
Speaker:Yeah.
Speaker:Um, just, just a little bit.
Speaker:I do wanna specifically talk about the insider threat 'cause I don't
Speaker:think we talk about that as much.
Speaker:Right.
Speaker:Um, and I do think that you need to.
Speaker:Design your backup systems to be resilient against the insider threat.
Speaker:And we don't, and I don't think that people think about that enough.
Speaker:what's the
Speaker:think,
Speaker:thread?
Speaker:Thread?
Speaker:basically, it's someone who it's, it's a, it's a person who, um.
Speaker:Works in on the inside, they are one of your employees or contractors
Speaker:who have full access to whatever system we're talking about, and
Speaker:either they have been compromised or their user ID has been compromised.
Speaker:Right?
Speaker:Um, it, so typically what we mean, we're referring to them
Speaker:being compromised, right?
Speaker:They're, they're, you know,
Speaker:a bad actor inside the
Speaker:they're a bad actor inside the company.
Speaker:The, the most famous or infamous one,
Speaker:so this is a story from 2002, and the story that I'm looking at is actually
Speaker:a very ugly thing here on justice.gov and it talks about basically, um.
Speaker:I'll just read the opening sentence here.
Speaker:A disgruntled computer systems administrator for UBS was charged
Speaker:today with using a logic bomb
Speaker:Mm.
Speaker:cause more than $3 million in damage to the company's computer network and
Speaker:with securities fraud for his failed plan to drive the company's stock down
Speaker:with the activation of the logic bomb.
Speaker:You know the other story,
Speaker:Yeah.
Speaker:doing the one about unify where they claimed that they were being
Speaker:hacked and it was an insider?
Speaker:All right.
Speaker:Right.
Speaker:That,
Speaker:years ago.
Speaker:yeah.
Speaker:Yeah, yeah, yeah.
Speaker:So, you know, if you have somebody that's on the inside, they can do a lot of damage
Speaker:to your systems without, um, without you, before you're able to stop them.
Speaker:with great power comes great responsibility.
Speaker:Yeah.
Speaker:Well, and I will also say with great power comes great restriction
Speaker:because, because the more powerful a person's role is, the more controls
Speaker:should be placed around that power.
Speaker:Right.
Speaker:Um, and you should.
Speaker:You know, everything that you do should be logged and, you know, and, and,
Speaker:and the more destructive things you should be able to have to, you should
Speaker:have to get, you know, multi person authentication, you know, otherwise
Speaker:known as four eyes authentication.
Speaker:Um, and, um.
Speaker:Because, and, and this is just, just, you have to think about that because not
Speaker:only are you protecting from the insider threat, you're protecting from a bad
Speaker:actor, getting a, getting access there.
Speaker:There's been a lot of stories right with that.
Speaker:There was of course the LastPass incident, which is, you know, was the
Speaker:nail in the coffin for me regarding LastPass, uh, where basically they,
Speaker:the, the hacker was able to gain access to a backup of the vault.
Speaker:Because they had a, they had, they had, they weren't able to decrypt the vault,
Speaker:but they were get a, they were able to get access to the vault because there
Speaker:was this backup script that had hard coded, um, you know, credentials in it.
Speaker:And then once they gained there, there was a, there was an initial breach
Speaker:and then that breach gave them access to see the script, which then gave
Speaker:them access to the credentials, which gave them access to the backup, which
Speaker:gave them access to, uh, the vault.
Speaker:Which was, I think then used for targeted crypto attacks.
Speaker:was used for targeted crypto attacks against people.
Speaker:Yeah.
Speaker:Um, and so really all we're, you know, we're just talking about, just
Speaker:realize that human error is still one of the val, the biggest reasons that
Speaker:we do anything in the backup world.
Speaker:And, and when we talk about this just.
Speaker:Try to implement the concept of least privilege.
Speaker:Try to implement the concept of minimizing the blast radius, minimizing the ability
Speaker:for somebody to do something or, you know, the, the more dangerous something is,
Speaker:the more controls you can put around it.
Speaker:Uh, or should put around it.
Speaker:so going to a sentence, you just said,
Speaker:Yeah.
Speaker:humans are always at risk,
Speaker:Humans.
Speaker:Yeah.
Speaker:Right.
Speaker:Yeah.
Speaker:So.
Speaker:Humans are always de-risk, I would say.
Speaker:Yeah.
Speaker:So what if people are like, Hey, forget about humans.
Speaker:Let's switch over to ai.
Speaker:No, no, no.
Speaker:I, I wanna play a thought experiment,
Speaker:uh.
Speaker:What, uh, would your views change?
Speaker:Would your, you would recommend, because for humans, right?
Speaker:We talk about putting controls, doing multifactor authentication, multi
Speaker:authentication, All of these things, if people start to move towards ai.
Speaker:How does this change?
Speaker:I know this episode is talking about human error, right?
Speaker:But just to leave
Speaker:don't, I don't think I'm ready to answer that question.
Speaker:We'll, we'll cross that bridge when we come to it.
Speaker:How's that?
Speaker:Okay.
Speaker:I, well, I'll just say this in general, I'm a fan of automation and AI can
Speaker:enable automation, but it's gotta be put into a, it's gotta beral.
Speaker:and
Speaker:Yeah.
Speaker:else
Speaker:Yeah.
Speaker:Again,
Speaker:may be
Speaker:with
Speaker:but
Speaker:power
Speaker:yeah.
Speaker:great.
Speaker:You know, responsibility.
Speaker:Right.
Speaker:And so you want to, you want to, um, um, sorry.
Speaker:I just found myself thinking about, um,
Speaker:Spider-Man.
Speaker:Stanwell Stan Lee.
Speaker:I wonder if Stan Lee thinks about d the fact that his Spider-Man line would be
Speaker:quoted at a IT podcast decades later.
Speaker:Um.
Speaker:The, uh, yeah.
Speaker:So the more power the AI has, the more control you gotta put around it.
Speaker:Because it could do things right.
Speaker:It could be used to do things.
Speaker:Yeah.
Speaker:All right.
Speaker:Well, this has been fun, sad, depressing, uh, et cetera, but that's why we back up.
Speaker:And,
Speaker:And, um,
Speaker:and that's why we're human.
Speaker:yes.
Speaker:And I, I wanna wish you, you, you have an upcoming trip.
Speaker:Uh.
Speaker:I
Speaker:Yeah, wanna wish you a good trip, Bon voyage.
Speaker:Thank you.
Speaker:Thank you.
Speaker:I hope our listeners get to have a trip coming up.
Speaker:You know, maybe you have some fun.
Speaker:At least somebody will have some fun.
Speaker:And, uh, with that, that is a wrap.
Speaker:The backup wrap up is written, recorded, and produced by me w Curtis Preston.
Speaker:If you need backup or Dr. Consulting content generation or expert witness
Speaker:work, check out backup central.com.
Speaker:You can also find links from my O'Reilly Books on the same website.
Speaker:Remember, this is an independent podcast and any opinions that
Speaker:you hear are those of the speaker and not necessarily an employer.
Speaker:Thanks for listening.