1 00:00:00,049 --> 00:00:04,250 You found the backup wrap up your go-to podcast for all things 2 00:00:04,250 --> 00:00:06,949 backup recovery and cyber recovery. 3 00:00:07,579 --> 00:00:11,360 In this episode, we look at the reality of modern data centers. 4 00:00:11,854 --> 00:00:17,854 Human error has become the number two reason that we do restores today. 5 00:00:18,664 --> 00:00:21,184 Number one, of course, would be cyber attacks. 6 00:00:21,845 --> 00:00:25,474 Back in the day, you know, a hundred years ago when I first started, it was 7 00:00:25,474 --> 00:00:29,405 really all about hardware failures, hard drives, crashing, servers dying. 8 00:00:29,704 --> 00:00:32,824 But now it's people making mistakes. 9 00:00:33,155 --> 00:00:37,535 Sometimes it's a user accidentally deleting a file, or maybe it's an admin 10 00:00:37,535 --> 00:00:39,754 wiping out an entire directory structure. 11 00:00:40,369 --> 00:00:42,830 Humans are the weakest link. 12 00:00:43,489 --> 00:00:47,390 I've got, I don't know, four or five war stories that'll make you cringe. 13 00:00:47,794 --> 00:00:52,414 And we talk about insider threats as well, and why your backup systems 14 00:00:52,624 --> 00:00:58,054 need to be designed with human error and human frailty in mind. 15 00:00:58,474 --> 00:01:02,285 Trust me, these stories will remind you why we do what we do. 16 00:01:02,974 --> 00:01:07,084 By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr. 17 00:01:07,505 --> 00:01:11,315 Backup, and I've been passionate about backup and recovery ever 18 00:01:11,315 --> 00:01:13,535 since I had to explain to my boss. 19 00:01:14,120 --> 00:01:17,419 While there were no backups of the production database that 20 00:01:17,419 --> 00:01:21,049 we had just lost, I don't want that to ever again happen to me. 21 00:01:21,139 --> 00:01:22,460 I don't want it to happen to you. 22 00:01:22,639 --> 00:01:23,900 That's why I do this. 23 00:01:24,680 --> 00:01:29,660 On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes. 24 00:01:29,989 --> 00:01:32,014 This is the backup wrap up. 25 00:01:48,477 --> 00:01:49,347 Welcome to the show. 26 00:01:50,172 --> 00:01:55,632 Hi, I am w Curtis Preston, AKA, Mr. Backup, and with me, I once again have 27 00:01:55,632 --> 00:01:58,332 my security deposit consultant persona. 28 00:01:58,722 --> 00:01:59,922 Molly, Andi, how's it going? 29 00:01:59,922 --> 00:02:00,372 Persona, 30 00:02:00,537 --> 00:02:06,207 I am good, Curtis, although I thought you would've said you're a person who 31 00:02:06,207 --> 00:02:10,257 was successful at DIYing something for probably the first time in their life. 32 00:02:12,567 --> 00:02:16,542 a guy that, a guy that finally joined the ranks because, because you're 33 00:02:16,542 --> 00:02:18,522 completed now with your, with your. 34 00:02:18,627 --> 00:02:19,287 I am. 35 00:02:19,287 --> 00:02:19,767 If people 36 00:02:19,947 --> 00:02:20,367 Yeah, 37 00:02:20,547 --> 00:02:23,757 episode, I, uh, had to redo some. 38 00:02:24,882 --> 00:02:25,662 stuff for 39 00:02:25,797 --> 00:02:26,187 yeah, 40 00:02:26,472 --> 00:02:29,592 system, and it was a little painful. 41 00:02:30,027 --> 00:02:30,657 yeah, 42 00:02:30,732 --> 00:02:31,062 lie. 43 00:02:31,122 --> 00:02:35,112 Um, and especially because I had to replace a shutoff valve, which meant I had 44 00:02:35,112 --> 00:02:36,972 to shut off the water for the entire house 45 00:02:37,227 --> 00:02:37,647 yeah. 46 00:02:37,782 --> 00:02:40,062 I was repair, replacing that shutoff valve. 47 00:02:40,062 --> 00:02:43,482 And then it turned out that I was leaking and blah, blah, blah, blah, blah. 48 00:02:43,602 --> 00:02:48,012 So, eventually got around to it, turned out, um, make sure you wrap the 49 00:02:48,012 --> 00:02:50,082 Teflon tape in the right direction. 50 00:02:50,262 --> 00:02:52,602 Yes, yes, yes. 51 00:02:52,602 --> 00:02:53,922 You figured that out, didn't you? 52 00:02:54,342 --> 00:02:54,732 Yes. 53 00:02:54,732 --> 00:02:58,212 And also having the right tools helps, especially a pipe wrench. 54 00:02:58,812 --> 00:02:59,412 Yes. 55 00:02:59,532 --> 00:03:01,032 More than one, right? 56 00:03:01,152 --> 00:03:01,542 Yeah. 57 00:03:02,382 --> 00:03:05,982 So once I did that, then actually doing all the PVC gluing and 58 00:03:06,612 --> 00:03:07,752 That part was easy, right? 59 00:03:07,802 --> 00:03:10,712 So I remember purple stuff first, then the blue stuff. 60 00:03:11,117 --> 00:03:12,137 Yeah, yeah, 61 00:03:12,302 --> 00:03:12,842 I should say. 62 00:03:13,397 --> 00:03:13,577 yeah. 63 00:03:13,607 --> 00:03:14,657 Oh, you use clear stuff? 64 00:03:14,717 --> 00:03:15,017 Yeah. 65 00:03:15,377 --> 00:03:17,027 Oh, that explains why I saw the blue. 66 00:03:17,327 --> 00:03:18,077 I saw the purple. 67 00:03:18,077 --> 00:03:18,317 Yeah. 68 00:03:18,347 --> 00:03:18,377 Okay. 69 00:03:19,052 --> 00:03:22,232 But it worked and it held, and it's knock on wood. 70 00:03:22,232 --> 00:03:23,342 So far so good. 71 00:03:23,342 --> 00:03:23,642 So 72 00:03:28,772 --> 00:03:32,492 Well, I'm glad to hear that you can now sprinkle your yard once again. 73 00:03:32,492 --> 00:03:35,672 I of course don't have that challenge 'cause I have a fake yard. 74 00:03:36,032 --> 00:03:39,092 well it's not a yard, it's just for the plants and all the drip irrigation, so. 75 00:03:39,662 --> 00:03:40,892 Oh, gotcha, gotcha. 76 00:03:41,312 --> 00:03:43,532 It's a yard, it's just, it's not grass. 77 00:03:43,727 --> 00:03:44,177 saw grass. 78 00:03:44,177 --> 00:03:44,387 Yeah. 79 00:03:44,467 --> 00:03:45,152 Right, right. 80 00:03:45,167 --> 00:03:46,817 especially last weekend. 81 00:03:46,877 --> 00:03:50,897 No, over the weekend, I think the temperature at 87 here in the Bay 82 00:03:50,897 --> 00:03:53,927 Area and today it's 61, so yeah. 83 00:03:54,197 --> 00:03:56,657 Yeah, we actually, we had a heat wave here as well. 84 00:03:56,867 --> 00:03:59,657 Um, it actually got up to like 90 at one point, 85 00:03:59,747 --> 00:04:00,347 Dang. 86 00:04:01,037 --> 00:04:02,897 Today it was cold and rainy, so 87 00:04:03,857 --> 00:04:04,067 Yeah. 88 00:04:04,067 --> 00:04:04,577 I think may. 89 00:04:04,577 --> 00:04:06,797 Gray back, no Uns shining outside. 90 00:04:06,797 --> 00:04:07,247 Hmm. 91 00:04:07,457 --> 00:04:07,937 I don't know. 92 00:04:08,087 --> 00:04:08,537 Who knows? 93 00:04:09,257 --> 00:04:10,487 You never know in San Diego, right? 94 00:04:10,487 --> 00:04:11,807 Especially in May. 95 00:04:11,987 --> 00:04:12,407 Right. 96 00:04:12,497 --> 00:04:13,667 May Gray and June gloom. 97 00:04:13,667 --> 00:04:16,007 For those that don't live here, it's pretty much overcast all the 98 00:04:16,007 --> 00:04:17,627 time for two months of the year. 99 00:04:17,987 --> 00:04:19,787 But it's why the temperature is, 100 00:04:20,207 --> 00:04:21,167 Even Keter. 101 00:04:21,407 --> 00:04:21,917 Yeah. 102 00:04:22,037 --> 00:04:22,427 Yeah. 103 00:04:22,832 --> 00:04:23,522 Um, 104 00:04:23,687 --> 00:04:23,927 dude. 105 00:04:24,062 --> 00:04:24,572 come here. 106 00:04:24,932 --> 00:04:25,382 What? 107 00:04:25,487 --> 00:04:25,907 ahead, finish. 108 00:04:26,282 --> 00:04:29,942 Well, I just said people come here and they, um, they're like, why? 109 00:04:31,467 --> 00:04:31,817 Where? 110 00:04:32,402 --> 00:04:36,062 I'm like, this is like, do you do no research on the places you go to? 111 00:04:37,202 --> 00:04:37,562 Right, 112 00:04:38,327 --> 00:04:41,807 Well, it's like the time that, or I think that twice that I've been up to 113 00:04:41,807 --> 00:04:46,067 Seattle and both times it was nice and sunny and so in my mind it's imprinted 114 00:04:46,067 --> 00:04:47,897 that Seattle is always nice and sunny. 115 00:04:48,482 --> 00:04:48,842 Yeah. 116 00:04:49,112 --> 00:04:50,342 Oh yeah. 117 00:04:50,912 --> 00:04:56,102 I, yeah, I spent, I spent four months in Seattle in 1998, and 118 00:04:56,102 --> 00:04:59,642 it was the most perfect four months in the history of Seattle. 119 00:04:59,642 --> 00:05:02,912 It rained like two days during those four months, and it was just, and 120 00:05:03,512 --> 00:05:07,232 yeah, I could, I could have moved there, but it's not normally like that. 121 00:05:07,337 --> 00:05:07,827 exactly. 122 00:05:09,362 --> 00:05:09,812 Yeah. 123 00:05:09,872 --> 00:05:13,947 But, um, well, uh, today, speaking of. 124 00:05:14,522 --> 00:05:17,642 Humans making mistakes today. 125 00:05:17,642 --> 00:05:19,562 I thought we'd talk a little bit about human error. 126 00:05:19,592 --> 00:05:27,182 Um, and, uh, as, as I often, uh, you know, find myself doing, I'll talk about 127 00:05:27,182 --> 00:05:31,262 back in the day, back in the day, um. 128 00:05:32,687 --> 00:05:38,117 The reason when I started my IT career, the, the number one reason 129 00:05:38,117 --> 00:05:45,047 we were doing restores was, um, actual hardware failure, right? 130 00:05:45,437 --> 00:05:49,397 Because I am Prera, right? 131 00:05:49,637 --> 00:05:56,897 Um, we, we had mission critical servers running on individual hard drives, 132 00:05:57,407 --> 00:05:58,877 Which is bonkers. 133 00:05:58,877 --> 00:05:58,937 A 134 00:05:59,237 --> 00:05:59,687 right? 135 00:05:59,687 --> 00:06:00,227 Right. 136 00:06:00,827 --> 00:06:05,327 do you think, what is, what is it like this, isn't there like a satellite 137 00:06:05,327 --> 00:06:08,357 that's been out there for like 40 years 138 00:06:09,167 --> 00:06:10,007 Oh yeah, yeah, 139 00:06:10,007 --> 00:06:12,977 and do you think that still also runs on a single disk? 140 00:06:12,977 --> 00:06:13,007 I, 141 00:06:13,187 --> 00:06:13,487 yeah. 142 00:06:13,487 --> 00:06:14,567 Def absolutely. 143 00:06:14,567 --> 00:06:14,927 Yeah. 144 00:06:15,172 --> 00:06:15,332 crazy. 145 00:06:16,427 --> 00:06:16,847 Yeah. 146 00:06:16,847 --> 00:06:20,057 Well, it might, do they have, do they have disc drives? 147 00:06:20,417 --> 00:06:20,837 I don't know. 148 00:06:20,912 --> 00:06:21,132 No. 149 00:06:22,127 --> 00:06:23,807 I mean, it's, if it's been there that long. 150 00:06:24,557 --> 00:06:30,167 Uh, but yeah, it, it's, I mean, I'm pretty sure RAID was invented before I actually 151 00:06:30,167 --> 00:06:35,897 saw it in the data center, but in my data center, which was a, a big, you know, 152 00:06:36,017 --> 00:06:38,987 $35 billion company, we didn't have any. 153 00:06:39,077 --> 00:06:39,437 Right. 154 00:06:40,052 --> 00:06:40,832 And, um, 155 00:06:41,087 --> 00:06:41,717 Was it 156 00:06:41,812 --> 00:06:42,792 and go ahead. 157 00:06:42,977 --> 00:06:49,307 was it the difference though between raid on open systems versus raid on mainframes? 158 00:06:49,877 --> 00:06:51,647 Yeah, I don't, yeah, I don't know. 159 00:06:51,827 --> 00:06:55,457 Um, it just, it certainly wasn't, you know, I remember the first 160 00:06:55,457 --> 00:06:58,247 time and I talked about it a few episodes the first time I saw 161 00:06:58,247 --> 00:06:59,837 hot swappable disc drives, right? 162 00:06:59,837 --> 00:07:01,967 Where you, you had the raid arrays, right. 163 00:07:02,417 --> 00:07:06,767 Um, and all of that sort of, it sort of all went down right around 164 00:07:06,767 --> 00:07:08,717 that timeframe, but for a few years. 165 00:07:09,547 --> 00:07:15,217 We were, we were losing entire databases or servers because an 166 00:07:15,217 --> 00:07:16,597 individual R drive went down. 167 00:07:16,597 --> 00:07:18,697 And so we, we were restoring that stuff all the time. 168 00:07:19,357 --> 00:07:24,757 And in fact, um, my first, once I left the bank and I went into 169 00:07:24,757 --> 00:07:31,807 consulting, um, one of the reasons I got so good at Bare Metal Recovery 170 00:07:32,167 --> 00:07:33,547 was that I was doing it all the time. 171 00:07:33,817 --> 00:07:39,452 Um, because I was at a. Uh, the headquarters of a 172 00:07:39,452 --> 00:07:41,192 large oil and gas company. 173 00:07:42,272 --> 00:07:46,472 And, um, they had all these servers that had not been, uh, 174 00:07:46,502 --> 00:07:48,002 maintained in a really long time. 175 00:07:48,002 --> 00:07:50,942 And so we were doing crazy things like installing patches and rebooting them 176 00:07:51,992 --> 00:07:53,222 with significant portion of the time. 177 00:07:53,222 --> 00:07:54,332 They just didn't come back up. 178 00:07:55,082 --> 00:07:55,232 So, 179 00:07:55,502 --> 00:07:56,667 makes perfect though, right Curtis? 180 00:07:56,822 --> 00:07:57,242 yeah. 181 00:07:58,022 --> 00:07:58,922 And so, yeah. 182 00:07:58,922 --> 00:08:03,422 Um, and so my, my, my point is that back then. 183 00:08:04,052 --> 00:08:09,182 A significant portion of the time, the reason you were doing a restore, it 184 00:08:09,182 --> 00:08:14,582 was because the actual hardware failed, failed, and that is just pretty much. 185 00:08:15,182 --> 00:08:15,902 Gone. 186 00:08:16,052 --> 00:08:21,842 I, you know, uh, between the fact that we now use solid state a as hard 187 00:08:21,842 --> 00:08:26,852 drives, and the fact that we use that, that no one, no one does anything 188 00:08:26,852 --> 00:08:28,742 important on, on an individual. 189 00:08:28,742 --> 00:08:33,782 I mean, your laptop, your phone, these are individual drives, but not 190 00:08:33,872 --> 00:08:36,062 nothing in a, in a server, right? 191 00:08:36,422 --> 00:08:37,277 Um, and. 192 00:08:37,457 --> 00:08:41,570 and also probably just reliability of components has also gone up. 193 00:08:43,817 --> 00:08:44,627 Absolutely. 194 00:08:44,797 --> 00:08:51,187 The result of all of that is that at this point the, there are, 195 00:08:51,757 --> 00:08:54,247 generally speaking only two reasons. 196 00:08:54,247 --> 00:08:58,447 We restore things and both of them had to do with people, right? 197 00:08:58,777 --> 00:09:03,877 Either people purposefully did something to damage the systems or they accidentally 198 00:09:03,877 --> 00:09:05,527 did something to damage the systems. 199 00:09:06,367 --> 00:09:07,027 Are you with me? 200 00:09:07,147 --> 00:09:07,627 Yep. 201 00:09:07,972 --> 00:09:08,872 Yeah, so 202 00:09:09,157 --> 00:09:09,577 failure. 203 00:09:10,312 --> 00:09:11,842 it is not a hardware failure, right? 204 00:09:11,842 --> 00:09:17,572 So either bad people, you know, bad actors, you know, it's a si, 205 00:09:17,602 --> 00:09:20,692 it's a, it's a ransomware attack, it's a cyber attack of some sort. 206 00:09:21,142 --> 00:09:24,232 Uh, or stupid users 207 00:09:25,777 --> 00:09:28,447 Would you consider like the OVH 208 00:09:28,462 --> 00:09:29,512 or stupid admins. 209 00:09:32,482 --> 00:09:35,902 So, so that is still, that is still a human error in my opinion. 210 00:09:35,902 --> 00:09:39,592 I mean, okay, so what, we can add that as a third category, which 211 00:09:39,592 --> 00:09:42,678 is natural disasters, right? 212 00:09:42,828 --> 00:09:51,948 Um, so if, if a fire happens, you know, um, yeah, I, we'll, we'll put a fire in 213 00:09:51,948 --> 00:09:55,758 a natural disaster, although that fire, I'm not sure was a natural disaster. 214 00:09:57,663 --> 00:10:00,813 But yeah, so that, that, that is a third reason, you know, I, I'll take that. 215 00:10:00,813 --> 00:10:04,623 But, but, but even that, the reason they end up having to do a restore, 216 00:10:04,623 --> 00:10:06,213 I think, was human error, right? 217 00:10:06,243 --> 00:10:09,063 Meaning that they, they just configured everything all wrong. 218 00:10:09,753 --> 00:10:12,573 Um, and I, and that's my story, and I'm sticking to it. 219 00:10:12,573 --> 00:10:15,093 so let's talk about the accidental humans 220 00:10:15,408 --> 00:10:16,068 Yeah. 221 00:10:16,158 --> 00:10:16,458 Yeah. 222 00:10:16,488 --> 00:10:19,728 'cause we talk a lot about the, the, the bad actors. 223 00:10:19,743 --> 00:10:19,953 Yeah. 224 00:10:20,238 --> 00:10:23,928 Um, so let's talk about the, you know, and, and basically what this 225 00:10:23,928 --> 00:10:29,058 kind of boils down to is, uh, either stupid users or stupid admins. 226 00:10:29,778 --> 00:10:30,168 Right? 227 00:10:30,333 --> 00:10:32,913 I, so there are two stories I want you to tell, 228 00:10:33,438 --> 00:10:34,098 Okay. 229 00:10:34,758 --> 00:10:39,798 The first one is, uh, restoring a document. 230 00:10:42,318 --> 00:10:45,378 The, the document called resume, 231 00:10:46,198 --> 00:10:46,458 Yeah. 232 00:10:46,938 --> 00:10:47,388 Yeah. 233 00:10:47,928 --> 00:10:48,168 And 234 00:10:48,288 --> 00:10:48,708 Uh. 235 00:10:48,738 --> 00:10:54,978 second one I want you to talk about is the file server with the user home directories 236 00:10:55,308 --> 00:10:55,698 Lord. 237 00:10:55,968 --> 00:10:57,738 Yeah, yeah. 238 00:10:57,828 --> 00:10:58,788 Sorry, sorry. 239 00:10:58,788 --> 00:10:59,688 Uh, sorry. 240 00:10:59,688 --> 00:11:00,018 Ganda. 241 00:11:00,018 --> 00:11:01,428 We're gonna call you stupid admin. 242 00:11:01,518 --> 00:11:01,578 Oh, okay. 243 00:11:01,848 --> 00:11:02,988 Um, yeah. 244 00:11:02,988 --> 00:11:06,528 So that, that was, you know, back in the day when, when, you 245 00:11:06,528 --> 00:11:07,606 know, I got, we, we, you could. 246 00:11:08,043 --> 00:11:08,283 Hmm. 247 00:11:08,723 --> 00:11:12,443 We, we had 12,000 employees and we had a help desk, and you could call in. 248 00:11:13,193 --> 00:11:15,323 This was, you know, pre-web, pre, you know, everything. 249 00:11:15,508 --> 00:11:17,033 You, you, you lost a file. 250 00:11:17,033 --> 00:11:19,403 You called into the help desk, and the help desk would maybe try to 251 00:11:19,553 --> 00:11:23,063 help you find the file, but if not, they would issue a restore request. 252 00:11:23,063 --> 00:11:26,513 We got a lot of restore requests, very regular. 253 00:11:26,513 --> 00:11:30,893 Another reason why I got good at doing this, because unlike a lot of places, we 254 00:11:30,893 --> 00:11:34,613 did new backups just to do them, we did restores on a very regular basis, right. 255 00:11:36,193 --> 00:11:41,083 There was this person who called and, um, they, they, they asked to have a file 256 00:11:41,083 --> 00:11:43,483 restored and it was called Resume Doc. 257 00:11:44,053 --> 00:11:46,423 And we were like, really? 258 00:11:46,543 --> 00:11:51,013 Like that's the what happened to Resume doc? 259 00:11:51,163 --> 00:11:52,483 Well, I was updating it. 260 00:11:52,723 --> 00:11:53,293 Where are you? 261 00:11:53,473 --> 00:11:54,313 Is that right? 262 00:11:54,433 --> 00:11:54,703 Yes. 263 00:11:54,703 --> 00:11:59,953 I was updating my re resume dot doc and um, I, I, you know, 264 00:11:59,953 --> 00:12:01,393 I fat fingered it basically. 265 00:12:01,453 --> 00:12:02,983 And, uh, and I, you know, 266 00:12:03,123 --> 00:12:03,873 to resume? 267 00:12:04,033 --> 00:12:04,273 yeah. 268 00:12:04,273 --> 00:12:05,293 What were you resuming? 269 00:12:05,688 --> 00:12:07,368 Yeah, trying to resume your career. 270 00:12:10,428 --> 00:12:12,558 That might've been, you know, depending on who we told that might've 271 00:12:12,558 --> 00:12:14,418 been a resume producing event. 272 00:12:14,598 --> 00:12:18,978 Uh, uh, yeah, so that was, that one was kind of funny. 273 00:12:18,978 --> 00:12:23,958 But, but I think that the, the, the, the, the story that really drives 274 00:12:23,958 --> 00:12:29,298 home the, you know, because, you know, admins are not perfect, right? 275 00:12:29,298 --> 00:12:31,848 And you, um. 276 00:12:32,963 --> 00:12:37,188 You, you know, you do things, you do things to, to make things better, 277 00:12:37,488 --> 00:12:39,048 but sometimes you do the wrong thing. 278 00:12:39,048 --> 00:12:39,318 Right? 279 00:12:39,318 --> 00:12:41,358 I mean, I think of like the KPMG. 280 00:12:41,478 --> 00:12:45,378 The K-P-M-G-I think is like the, that's like the, the best one that I had nothing 281 00:12:45,378 --> 00:12:50,508 to do with, where they're just trying to delete one guy's chats and, and they 282 00:12:50,508 --> 00:12:55,938 were, they were being stopped by Microsoft Retention policies and so they made a new 283 00:12:55,938 --> 00:12:59,988 retention policy that had no retention and then they thought they moved the 284 00:12:59,988 --> 00:13:01,698 one guy into it, but instead they moved. 285 00:13:01,898 --> 00:13:06,848 The entire company into it, and they deleted it was like 150,000 people's 286 00:13:06,848 --> 00:13:09,608 chats, uh, like, you know, like that. 287 00:13:09,638 --> 00:13:12,968 And, and, um, and there was no backup. 288 00:13:12,998 --> 00:13:13,328 Right? 289 00:13:14,408 --> 00:13:18,308 But, uh, but yeah, this other, this other story, uh, is, is a good one. 290 00:13:18,308 --> 00:13:20,318 And you know, Gonda, we love you. 291 00:13:20,468 --> 00:13:25,268 Um, so she, the, we had a file server. 292 00:13:25,388 --> 00:13:28,778 I remember it was HP FS oh one, HP file server. 293 00:13:29,583 --> 00:13:34,563 You know, one, and we had a lot of employees and we had a lot of turnover. 294 00:13:34,563 --> 00:13:38,883 And over time, the, and, and this is, this is, you know, old school Unix. 295 00:13:39,183 --> 00:13:41,193 You had home one, you know Curtis. 296 00:13:41,193 --> 00:13:41,523 Right? 297 00:13:41,523 --> 00:13:45,303 And after the, the person left the company, we did not have an 298 00:13:45,303 --> 00:13:48,963 off-boarding process would be the, uh, or if we had one, it didn't 299 00:13:48,963 --> 00:13:50,103 include deleting the old data. 300 00:13:50,103 --> 00:13:52,413 And so somebody said, we think we have a lot of. 301 00:13:52,803 --> 00:13:54,273 Of directories out there. 302 00:13:54,273 --> 00:13:57,333 And, and we knew that we had all of this on tape and then we knew that we 303 00:13:57,333 --> 00:13:59,013 kept all of our tapes for seven years. 304 00:13:59,253 --> 00:14:02,343 And so we knew that if, if anybody ever actually needed 305 00:14:02,343 --> 00:14:03,453 this data, we could get it back. 306 00:14:03,453 --> 00:14:09,813 And so they hired, uh, my friend to, uh, to programmatically work their way 307 00:14:09,813 --> 00:14:13,923 through the home directory and then delete any directories that were not. 308 00:14:14,328 --> 00:14:15,558 From a valid user. 309 00:14:15,618 --> 00:14:15,908 Okay. 310 00:14:16,068 --> 00:14:19,458 And so the directories were named after the user. 311 00:14:19,698 --> 00:14:23,178 And so you would see, you know, the c Preston directory. 312 00:14:23,628 --> 00:14:26,928 And then she would look and, you know, she would just traverse the tree and she 313 00:14:26,928 --> 00:14:30,048 would find the, the directory name, and then she would look to see if there was a 314 00:14:30,108 --> 00:14:32,658 user by that name and the password file. 315 00:14:34,128 --> 00:14:35,958 And then if the user wasn't there, she would delete it. 316 00:14:37,308 --> 00:14:41,988 And, um, the um, but. 317 00:14:42,378 --> 00:14:48,708 There was, she made one small, one small calculation or miscalculation, 318 00:14:48,918 --> 00:14:55,398 and that is that there was a two level directory structure, so it was, it was 319 00:14:55,398 --> 00:15:00,528 like Home one slash a slash Alfred, 320 00:15:00,593 --> 00:15:01,013 Mm-hmm. 321 00:15:01,068 --> 00:15:04,128 know, home one slash c slash c Preston. 322 00:15:04,758 --> 00:15:08,328 And um, so she worked her way down the tree. 323 00:15:08,328 --> 00:15:09,708 She got the home one A. 324 00:15:09,948 --> 00:15:14,223 Is there a username, a. In the, in the tree, in the password file. 325 00:15:14,853 --> 00:15:15,333 Nope. 326 00:15:15,663 --> 00:15:16,113 All right. 327 00:15:16,113 --> 00:15:17,253 Delete a 328 00:15:17,298 --> 00:15:17,568 Yeah. 329 00:15:18,553 --> 00:15:19,133 Oh, no. 330 00:15:19,158 --> 00:15:23,373 And she, and she was like a third of the way through, uh, the directories 331 00:15:23,373 --> 00:15:26,073 when people started, they, they, they were unable to log in 'cause 332 00:15:26,073 --> 00:15:27,123 they had no home directory. 333 00:15:27,753 --> 00:15:30,573 And, uh, we got a whole bunch of calls to the thing. 334 00:15:30,573 --> 00:15:32,553 And luckily she, she stopped it out. 335 00:15:32,793 --> 00:15:37,683 And I, I still remember she called me that day and her, her first words 336 00:15:37,683 --> 00:15:43,773 were, she's like, Curtis, um, how were the backups of HP FSO one last night? 337 00:15:43,773 --> 00:15:44,223 And I'm like. 338 00:15:44,613 --> 00:15:46,533 Um, they were fine. 339 00:15:46,713 --> 00:15:47,943 Why are you asking? 340 00:15:51,123 --> 00:15:55,263 And she, uh, we proceed to restore all the home directories. 341 00:15:55,983 --> 00:15:56,673 Um, 342 00:15:56,988 --> 00:16:00,348 yeah, that's, uh, that's a scary thing because imagine if 343 00:16:00,348 --> 00:16:01,878 you didn't have those backups, 344 00:16:02,248 --> 00:16:02,538 yeah, 345 00:16:02,778 --> 00:16:03,108 right? 346 00:16:03,138 --> 00:16:05,058 That's a third of the company's data. 347 00:16:05,268 --> 00:16:07,398 AOG or users on that 348 00:16:07,718 --> 00:16:09,273 yeah, yeah. 349 00:16:09,393 --> 00:16:10,263 Um, I, 350 00:16:10,668 --> 00:16:11,958 And your data too, because 351 00:16:12,033 --> 00:16:12,513 what's that? 352 00:16:12,798 --> 00:16:14,418 And your data too, because you're in the cs. 353 00:16:14,778 --> 00:16:15,198 That's right. 354 00:16:15,648 --> 00:16:16,008 Yes. 355 00:16:16,008 --> 00:16:16,818 I was gone. 356 00:16:17,448 --> 00:16:21,978 Um, uh, the, uh, you, you probably would've been safe, you know, with 357 00:16:21,978 --> 00:16:23,898 the, with the being and the peas there. 358 00:16:24,408 --> 00:16:29,988 Um, there's another, another great, uh, sort of, um, uh, do 359 00:16:29,988 --> 00:16:31,158 you have, do you have a story? 360 00:16:31,278 --> 00:16:32,643 want you to tell another story, 361 00:16:33,078 --> 00:16:33,303 Oh, okay. 362 00:16:33,798 --> 00:16:34,218 Okay. 363 00:16:34,638 --> 00:16:36,438 So there's another great, um. 364 00:16:36,843 --> 00:16:37,353 Story. 365 00:16:37,443 --> 00:16:40,443 It, it, it actually involves the deletion of backup data. 366 00:16:40,443 --> 00:16:44,853 But it, it, it drives home this issue of basically users 367 00:16:44,853 --> 00:16:47,193 make or admins make mistakes. 368 00:16:47,583 --> 00:16:52,353 And this has to do, this has to do with your, your former employer, uh, legato. 369 00:16:53,043 --> 00:16:58,048 Um, and they had, they introduced a feature. 370 00:16:59,058 --> 00:17:03,738 Uh, two features actually in, in, it seemed like it was version five maybe. 371 00:17:04,278 --> 00:17:07,368 And one of the problems was when you were, when, when you, when you 372 00:17:07,368 --> 00:17:10,848 made, when you expired tapes and you were gonna reuse those tapes, 373 00:17:10,848 --> 00:17:12,168 you would relabel those tapes. 374 00:17:12,168 --> 00:17:15,978 And by that I mean you would put an electronic label on the front of the 375 00:17:15,978 --> 00:17:19,518 tape, and by doing that, you would, you effectively erase the rest of the tape 376 00:17:19,518 --> 00:17:24,468 because it puts an end of data marker at the end of the, the, you know, the, um. 377 00:17:25,353 --> 00:17:28,743 The label and then you, you can't get past that with a tape drive. 378 00:17:29,493 --> 00:17:31,953 And so the um. 379 00:17:34,293 --> 00:17:40,443 But the problem was even if you had like several tape drives, um, the process, 380 00:17:40,713 --> 00:17:43,293 it would label the tapes one at a time. 381 00:17:43,293 --> 00:17:46,683 So if you had a whole bunch of tapes, it, it took a long time. 382 00:17:46,683 --> 00:17:51,003 And then also when you would, uh, label it, it would, uh, 383 00:17:51,033 --> 00:17:53,043 confirm the deletion each time. 384 00:17:53,058 --> 00:17:53,348 Okay. 385 00:17:53,493 --> 00:17:54,843 you sure you wanna relabel the state? 386 00:17:54,843 --> 00:17:54,844 Yeah. 387 00:17:56,043 --> 00:17:59,613 And so they introduced the fast and silent. 388 00:18:00,048 --> 00:18:05,118 Uh, option where they would use every available tape drive in the 389 00:18:05,118 --> 00:18:08,238 library and they would not ask. 390 00:18:08,628 --> 00:18:09,018 Okay? 391 00:18:09,258 --> 00:18:17,148 So if you check those two boxes, then, um, basically you would, um, you know, you 392 00:18:17,148 --> 00:18:19,218 would relabel whatever tapes you selected. 393 00:18:20,748 --> 00:18:22,668 But this was accompanied by a bug. 394 00:18:23,238 --> 00:18:23,418 You no. 395 00:18:23,568 --> 00:18:27,558 And the bug was that if you, if, well, if you had a list of tapes. 396 00:18:29,148 --> 00:18:33,648 In a, in a tape library, and you double clicked on one of those tapes, you would 397 00:18:33,648 --> 00:18:38,778 expect that what would pop up would be a dialogue that would include just 398 00:18:38,778 --> 00:18:40,398 that tape that you just clicked on. 399 00:18:41,088 --> 00:18:42,258 That's not how it worked. 400 00:18:42,288 --> 00:18:46,548 If you clicked on the tape, what would pull up was slot one to slot, 401 00:18:46,548 --> 00:18:47,838 however many slots there were. 402 00:18:47,958 --> 00:18:48,888 Oh no. 403 00:18:48,933 --> 00:18:49,323 Right. 404 00:18:49,833 --> 00:18:53,193 So if you wanted to relabel just this one tape, you would double click it. 405 00:18:53,283 --> 00:18:56,073 And if you weren't paying attention, it would pop up a dialogue box. 406 00:18:56,073 --> 00:19:00,903 And if you click yes, fast and silent, go, what you just told it to do was 407 00:19:00,903 --> 00:19:02,583 relabel every tape in the tape library. 408 00:19:02,703 --> 00:19:03,258 Oh no. 409 00:19:03,573 --> 00:19:09,603 And I was there when, uh, I. When, uh, it was actually a Legato employee, uh, that 410 00:19:09,603 --> 00:19:13,953 was at the, it was a, it was a healthcare company up in, up in the LA area. 411 00:19:14,643 --> 00:19:19,173 And he, uh, he managed to relabel this customer's tapes, all of them. 412 00:19:19,533 --> 00:19:22,413 Um, again, I think he got like halfway through the tape library 413 00:19:22,413 --> 00:19:23,763 before he realized what he had done. 414 00:19:23,913 --> 00:19:25,473 there's no going back, right? 415 00:19:25,803 --> 00:19:25,893 Mm-hmm. 416 00:19:25,983 --> 00:19:30,303 Mm. Because once you put that, again, once you put the electronic label on 417 00:19:30,303 --> 00:19:34,233 the front, it puts an end of data marker after that, and you can't get past that. 418 00:19:35,688 --> 00:19:40,278 Um, I hope the customer had copies of those tapes, but the 419 00:19:40,278 --> 00:19:41,688 customer was not very happy. 420 00:19:41,853 --> 00:19:42,678 I would not be either. 421 00:19:43,118 --> 00:19:43,238 I. 422 00:19:43,398 --> 00:19:43,908 yeah. 423 00:19:44,823 --> 00:19:45,213 So 424 00:19:45,228 --> 00:19:45,618 Yeah. 425 00:19:45,633 --> 00:19:47,853 we've been talking about this for a while, but there is a third story 426 00:19:48,003 --> 00:19:49,533 or fourth story I want you to tell. 427 00:19:49,758 --> 00:19:50,988 Oh, you have another story. 428 00:19:51,498 --> 00:19:52,038 All right. 429 00:19:52,203 --> 00:19:57,063 want you to tell the story about, uh, temp and source code. 430 00:19:59,943 --> 00:20:00,663 That's a good one. 431 00:20:00,723 --> 00:20:01,473 That's a good one. 432 00:20:01,923 --> 00:20:02,463 Yeah. 433 00:20:02,763 --> 00:20:08,523 So, so for those that, that, that don't live in the Unix world, you know, 434 00:20:08,523 --> 00:20:13,503 temp, uh, you know, slash TMP or slash TMP depending on which os you were 435 00:20:13,503 --> 00:20:18,723 talking about, which, you know, which distribution was typically where you, 436 00:20:18,783 --> 00:20:20,073 you just put garbage files, right? 437 00:20:21,033 --> 00:20:23,253 You, you, it's temp, it's temporary. 438 00:20:23,253 --> 00:20:23,703 Right? 439 00:20:23,763 --> 00:20:31,233 And, um, the, um, and, and you just use it to, for garbage files, right? 440 00:20:31,323 --> 00:20:32,913 Uh, for scripting and stuff like that. 441 00:20:33,423 --> 00:20:40,383 And, um, the, uh, and, and then on one, some of the distributions and 442 00:20:40,383 --> 00:20:45,453 specifically on HPUX, which is what we were running, temp, was actually in Ram. 443 00:20:45,673 --> 00:20:45,893 Mm. 444 00:20:46,428 --> 00:20:50,778 And so when you rebooted the box, anything in temp would go, bye-bye. 445 00:20:50,808 --> 00:20:51,108 Yeah. 446 00:20:51,168 --> 00:20:51,528 Right? 447 00:20:51,858 --> 00:21:02,148 It's temp, that's the point of temp. And we, uh, one day I got a, um, 448 00:21:02,418 --> 00:21:09,408 a restore request for a directory structure in temp, and I said. 449 00:21:09,753 --> 00:21:10,803 What, what is this? 450 00:21:10,803 --> 00:21:13,803 And they go, oh, well, it's a, it's a source code tree that we've been 451 00:21:14,043 --> 00:21:18,513 working on, this group of consultants, like 20 of us or something. 452 00:21:18,933 --> 00:21:21,873 We've been working on it for a few months and it was in temp and 453 00:21:22,083 --> 00:21:25,713 this, the box, it got rebooted or you know, it just, it got deleted. 454 00:21:25,713 --> 00:21:27,513 They said like, they didn't know how it got deleted. 455 00:21:27,513 --> 00:21:28,113 It got deleted. 456 00:21:28,863 --> 00:21:31,413 I'm like, it's in temp. And they're like, yeah. 457 00:21:31,443 --> 00:21:35,943 I'm like, we don't back up Temp. We, we never have backed 458 00:21:35,943 --> 00:21:38,553 up temp. It's temp, right? 459 00:21:39,093 --> 00:21:42,543 It's like, it's like backing up your garbage can, like what, 460 00:21:42,543 --> 00:21:43,983 what, what is the point of that? 461 00:21:44,013 --> 00:21:44,463 Right? 462 00:21:44,883 --> 00:21:49,293 And um, and so they're like, well, you, you don't understand like, heads are gonna 463 00:21:49,293 --> 00:21:52,353 roll if we lose this, this source country. 464 00:21:52,383 --> 00:21:56,673 And I'm like, not my head like, yeah. 465 00:21:56,733 --> 00:21:58,803 was the genius who decided to put it in temp? 466 00:21:59,703 --> 00:22:00,543 Yeah, some. 467 00:22:00,543 --> 00:22:01,383 It was a developer. 468 00:22:01,383 --> 00:22:05,463 It was a developer, like a whole team of developers, and nobody was like, 469 00:22:05,553 --> 00:22:07,113 should, is this where we should put this? 470 00:22:08,118 --> 00:22:13,938 So know we're, we've been focused on like human error doing things accidentally, 471 00:22:14,163 --> 00:22:14,523 Yeah. 472 00:22:14,868 --> 00:22:15,258 right? 473 00:22:15,438 --> 00:22:20,238 And it's not necessarily just an end user who could be at fault. 474 00:22:20,238 --> 00:22:24,918 It could be an admin, it could just be company processes are broken. 475 00:22:25,308 --> 00:22:25,788 Right? 476 00:22:26,088 --> 00:22:27,468 But it's still like. 477 00:22:28,458 --> 00:22:33,528 I think the key here is asking questions before things go wrong, so you know how 478 00:22:33,528 --> 00:22:35,148 your data's being backed up and where it's 479 00:22:35,298 --> 00:22:36,408 Right, right. 480 00:22:37,158 --> 00:22:40,668 And, you know, and, and, and a lot of the, a lot of these problems, uh, 481 00:22:40,728 --> 00:22:42,468 could be fixed by versioning, right. 482 00:22:42,468 --> 00:22:43,878 Snapshots and things like that. 483 00:22:43,878 --> 00:22:48,048 So we don't have to use the quote unquote backup system, but I, I 484 00:22:48,048 --> 00:22:51,438 do think that this is like the number one reason that we do that. 485 00:22:51,438 --> 00:22:52,698 We do restores right. 486 00:22:53,448 --> 00:22:55,608 Do you think it's still the case today? 487 00:22:56,793 --> 00:22:57,513 Absolutely. 488 00:22:57,723 --> 00:23:01,743 Well, I mean, I think right now the number one, well again, the number one 489 00:23:01,743 --> 00:23:04,323 reason we do restores are, are, is humans. 490 00:23:04,863 --> 00:23:05,373 Right? 491 00:23:06,423 --> 00:23:11,253 We probably do more restores having to do with cyber incidents than 492 00:23:11,253 --> 00:23:14,133 we do stupid user, uh, incidents. 493 00:23:14,613 --> 00:23:16,083 But they, but they still happen. 494 00:23:16,083 --> 00:23:17,133 They still happen all the time, 495 00:23:17,133 --> 00:23:17,463 Do 496 00:23:17,553 --> 00:23:17,853 right? 497 00:23:18,003 --> 00:23:19,713 a lot of the user. 498 00:23:20,898 --> 00:23:22,368 Because you mentioned like the help desk, right? 499 00:23:22,368 --> 00:23:24,198 People would call in, they would ask for things. 500 00:23:24,558 --> 00:23:28,668 Do you think though, with a lot of the technologies out there that users 501 00:23:28,668 --> 00:23:32,238 are now doing sort of self-service restores where they don't need to be 502 00:23:32,238 --> 00:23:36,768 embarrassed by, Hey, I accidentally deleted this file, my resume dot 503 00:23:36,768 --> 00:23:38,298 doc, can you please bring it back? 504 00:23:39,318 --> 00:23:42,708 Yeah, I think, well, I think a couple of, I think a couple of things there. 505 00:23:42,708 --> 00:23:46,668 One is, uh, there's a lot of shadow it going on where people are doing their 506 00:23:46,668 --> 00:23:48,588 own backups or something like backups. 507 00:23:49,158 --> 00:23:54,288 Uh, and so they, they use that instead of having to, um, you know, call a help desk. 508 00:23:54,978 --> 00:23:58,968 And then the other is, I do think that there are a lot of technologies that 509 00:23:58,968 --> 00:24:00,468 have been deployed in data centers. 510 00:24:00,588 --> 00:24:02,508 Snapshots being the primary one. 511 00:24:03,648 --> 00:24:07,428 That have allowed users to essentially do their own restore, right? 512 00:24:07,658 --> 00:24:11,718 They know I can go to ~Snapshot and I can go in there and I can get whatever 513 00:24:11,718 --> 00:24:16,308 I want from whatever timeframe I want, and I, and I can get it from an hour ago, 514 00:24:16,428 --> 00:24:16,698 Yep. 515 00:24:16,998 --> 00:24:18,468 Just before I did the stupid thing. 516 00:24:18,978 --> 00:24:20,358 Uh, so I do think that. 517 00:24:20,838 --> 00:24:26,268 There's probably less of a reason for people to do, to call into an actual 518 00:24:26,268 --> 00:24:29,898 help desk or to go to a website and say, Hey, I need this file restored. 519 00:24:30,408 --> 00:24:33,138 Um, because there's a lot of self service. 520 00:24:33,228 --> 00:24:33,558 What's that? 521 00:24:33,558 --> 00:24:34,038 What's that? 522 00:24:34,098 --> 00:24:34,878 or the trash bin. 523 00:24:34,883 --> 00:24:35,173 Right. 524 00:24:35,598 --> 00:24:36,978 Yeah, yeah, yeah. 525 00:24:37,188 --> 00:24:38,538 have a trash bin where you can 526 00:24:38,538 --> 00:24:38,928 Yeah. 527 00:24:39,018 --> 00:24:39,498 your data. 528 00:24:40,218 --> 00:24:44,058 Well, the trash bin is usually only helpful if you delete files. 529 00:24:44,328 --> 00:24:47,598 If you corrupt the files, trash bins aren't usually helpful. 530 00:24:47,878 --> 00:24:48,078 Yeah. 531 00:24:48,138 --> 00:24:51,198 Um, and, you know, thi this, this is one of the things that 532 00:24:51,198 --> 00:24:53,208 I used to hear about Salesforce. 533 00:24:53,208 --> 00:24:56,508 I go, well, Salesforce has a, you know, it has a, again, a trash bin. 534 00:24:57,258 --> 00:24:58,218 And I'm like, right. 535 00:24:58,218 --> 00:25:03,978 But when you modify files, when you modify, uh, what they would call, um. 536 00:25:04,248 --> 00:25:04,938 Objects. 537 00:25:05,028 --> 00:25:06,858 call it records, actually, an object. 538 00:25:07,068 --> 00:25:10,308 They, they use the term object very, in my opinion, very weirdly to them, 539 00:25:10,308 --> 00:25:12,108 an object is like the user table. 540 00:25:12,258 --> 00:25:12,468 Oh, that's 541 00:25:12,528 --> 00:25:13,368 That's an object. 542 00:25:13,368 --> 00:25:18,138 I never understood that, but whatever, uh, I would call it a record, right? 543 00:25:18,138 --> 00:25:22,638 When you modify a record, there is no place where the previous 544 00:25:22,638 --> 00:25:23,958 version of that record is stored. 545 00:25:24,123 --> 00:25:24,343 Yep. 546 00:25:24,438 --> 00:25:29,268 so, um, and I remember one time where I, um. 547 00:25:29,883 --> 00:25:34,803 I screwed up my, we had a Salesforce database that had a couple million records 548 00:25:35,053 --> 00:25:35,133 Hmm 549 00:25:35,793 --> 00:25:39,693 and I screwed up every million of 'em by accidentally. 550 00:25:40,383 --> 00:25:46,113 Um, first off, it should be a federal crime that when you have an Excel 551 00:25:46,113 --> 00:25:50,403 spreadsheet and you click on a column 552 00:25:50,598 --> 00:25:50,748 That 553 00:25:50,793 --> 00:25:52,023 and then you say, sort. 554 00:25:52,068 --> 00:25:53,478 column and not everything. 555 00:25:53,973 --> 00:25:56,763 That should be, that should, whoever came up with that feature 556 00:25:56,973 --> 00:25:58,773 needs to be tarred and feathered. 557 00:25:59,643 --> 00:26:00,213 How is that? 558 00:26:00,213 --> 00:26:04,353 What, in what world is that what anyone wants to do? 559 00:26:05,748 --> 00:26:06,228 Right. 560 00:26:06,498 --> 00:26:11,118 I mean, I, I think now when you do it, it it, it'll, it'll say, did you mean to 561 00:26:11,268 --> 00:26:13,758 sort, you know, and you just have to push. 562 00:26:13,848 --> 00:26:16,728 You've got, yeah, no, I want to sort the whole thing, but like, 563 00:26:16,908 --> 00:26:18,438 why would that ever be the case? 564 00:26:18,438 --> 00:26:19,038 You know what I mean? 565 00:26:19,638 --> 00:26:22,968 It should be like, it should be the, like, what they did in AWS with o 566 00:26:22,968 --> 00:26:27,108 with open buckets where you have to like really, really, really try 567 00:26:27,108 --> 00:26:28,698 really hard to do an open bucket. 568 00:26:28,728 --> 00:26:29,478 It should be like that. 569 00:26:30,843 --> 00:26:35,343 Um, and if you, if you actually want to do a sort by just that 570 00:26:35,343 --> 00:26:39,963 column, you should have to really, you should have to really say so. 571 00:26:40,533 --> 00:26:42,663 It is like provide a blood oat and 572 00:26:43,293 --> 00:26:43,743 Yeah. 573 00:26:43,863 --> 00:26:44,283 Yeah. 574 00:26:44,343 --> 00:26:47,583 Because, you know, once you do that, uh, and so what I, what 575 00:26:47,583 --> 00:26:52,953 I did was I accidentally sorted the, uh, I, I downloaded the phone 576 00:26:52,953 --> 00:26:55,113 number column, which was, um. 577 00:26:56,148 --> 00:26:59,298 All the phone numbers, they're formatted, like all different. 578 00:26:59,508 --> 00:27:06,618 And so I figured out, I, I wrote a program to basically take out all 579 00:27:06,618 --> 00:27:10,458 of the formatting and then put back the formatting the way I wanted it. 580 00:27:10,668 --> 00:27:10,968 Right. 581 00:27:11,958 --> 00:27:13,608 Except I sorted it. 582 00:27:16,063 --> 00:27:16,283 Oh, 583 00:27:16,488 --> 00:27:21,528 And so when I uploaded that back to Salesforce using the data loader, I 584 00:27:21,528 --> 00:27:22,878 think that's what that was called. 585 00:27:22,878 --> 00:27:23,598 It's been a while. 586 00:27:24,228 --> 00:27:24,918 Um. 587 00:27:25,323 --> 00:27:25,743 It. 588 00:27:26,283 --> 00:27:28,983 I just put all the wrong phone numbers to all the wrong places, 589 00:27:28,983 --> 00:27:30,843 but luckily I had downloaded 590 00:27:31,083 --> 00:27:31,563 Uh 591 00:27:31,653 --> 00:27:33,003 table prior to doing that. 592 00:27:33,633 --> 00:27:34,863 Luckily, I had a backup there. 593 00:27:35,728 --> 00:27:35,948 Um. 594 00:27:37,563 --> 00:27:40,533 So do you wanna even touch on the second category? 595 00:27:40,593 --> 00:27:43,233 Because I think that's basically what we talk about on the 596 00:27:43,233 --> 00:27:47,163 podcast, like 97.2% of the time. 597 00:27:47,283 --> 00:27:51,543 Oh, the, the, um, the, um, the cybersecurity stuff? 598 00:27:51,543 --> 00:27:51,873 Yeah. 599 00:27:52,533 --> 00:27:54,783 Um, just, just a little bit. 600 00:27:56,073 --> 00:27:59,823 I do wanna specifically talk about the insider threat 'cause I don't 601 00:27:59,823 --> 00:28:01,683 think we talk about that as much. 602 00:28:01,743 --> 00:28:02,193 Right. 603 00:28:02,673 --> 00:28:06,688 Um, and I do think that you need to. 604 00:28:09,003 --> 00:28:13,983 Design your backup systems to be resilient against the insider threat. 605 00:28:13,983 --> 00:28:16,893 And we don't, and I don't think that people think about that enough. 606 00:28:17,088 --> 00:28:17,478 what's the 607 00:28:17,523 --> 00:28:17,763 think, 608 00:28:17,868 --> 00:28:18,258 thread? 609 00:28:18,483 --> 00:28:18,903 Thread? 610 00:28:19,233 --> 00:28:24,093 basically, it's someone who it's, it's a, it's a person who, um. 611 00:28:24,783 --> 00:28:28,533 Works in on the inside, they are one of your employees or contractors 612 00:28:28,533 --> 00:28:33,123 who have full access to whatever system we're talking about, and 613 00:28:33,213 --> 00:28:37,653 either they have been compromised or their user ID has been compromised. 614 00:28:37,653 --> 00:28:38,013 Right? 615 00:28:38,163 --> 00:28:43,023 Um, it, so typically what we mean, we're referring to them 616 00:28:43,023 --> 00:28:44,253 being compromised, right? 617 00:28:44,253 --> 00:28:45,903 They're, they're, you know, 618 00:28:45,978 --> 00:28:46,908 a bad actor inside the 619 00:28:46,983 --> 00:28:48,783 they're a bad actor inside the company. 620 00:28:48,783 --> 00:28:51,153 The, the most famous or infamous one, 621 00:28:51,330 --> 00:28:57,870 so this is a story from 2002, and the story that I'm looking at is actually 622 00:28:57,870 --> 00:29:06,990 a very ugly thing here on justice.gov and it talks about basically, um. 623 00:29:08,385 --> 00:29:10,035 I'll just read the opening sentence here. 624 00:29:10,035 --> 00:29:14,715 A disgruntled computer systems administrator for UBS was charged 625 00:29:14,715 --> 00:29:17,145 today with using a logic bomb 626 00:29:17,360 --> 00:29:17,580 Mm. 627 00:29:17,625 --> 00:29:21,375 cause more than $3 million in damage to the company's computer network and 628 00:29:21,375 --> 00:29:25,635 with securities fraud for his failed plan to drive the company's stock down 629 00:29:26,205 --> 00:29:27,855 with the activation of the logic bomb. 630 00:29:27,900 --> 00:29:29,580 You know the other story, 631 00:29:29,985 --> 00:29:30,585 Yeah. 632 00:29:30,870 --> 00:29:34,980 doing the one about unify where they claimed that they were being 633 00:29:34,980 --> 00:29:37,440 hacked and it was an insider? 634 00:29:38,220 --> 00:29:39,030 All right. 635 00:29:39,060 --> 00:29:39,570 Right. 636 00:29:39,630 --> 00:29:39,840 That, 637 00:29:39,900 --> 00:29:40,620 years ago. 638 00:29:40,770 --> 00:29:41,160 yeah. 639 00:29:41,190 --> 00:29:42,960 Yeah, yeah, yeah. 640 00:29:42,960 --> 00:29:49,230 So, you know, if you have somebody that's on the inside, they can do a lot of damage 641 00:29:49,230 --> 00:29:54,810 to your systems without, um, without you, before you're able to stop them. 642 00:29:54,900 --> 00:29:57,120 with great power comes great responsibility. 643 00:29:57,630 --> 00:29:57,920 Yeah. 644 00:29:59,325 --> 00:30:03,465 Well, and I will also say with great power comes great restriction 645 00:30:03,495 --> 00:30:10,485 because, because the more powerful a person's role is, the more controls 646 00:30:10,485 --> 00:30:11,925 should be placed around that power. 647 00:30:12,375 --> 00:30:12,825 Right. 648 00:30:12,915 --> 00:30:14,415 Um, and you should. 649 00:30:16,815 --> 00:30:20,775 You know, everything that you do should be logged and, you know, and, and, 650 00:30:20,775 --> 00:30:24,225 and the more destructive things you should be able to have to, you should 651 00:30:24,225 --> 00:30:28,455 have to get, you know, multi person authentication, you know, otherwise 652 00:30:28,455 --> 00:30:30,255 known as four eyes authentication. 653 00:30:31,485 --> 00:30:33,465 Um, and, um. 654 00:30:34,110 --> 00:30:37,890 Because, and, and this is just, just, you have to think about that because not 655 00:30:37,890 --> 00:30:41,790 only are you protecting from the insider threat, you're protecting from a bad 656 00:30:41,790 --> 00:30:44,190 actor, getting a, getting access there. 657 00:30:44,250 --> 00:30:46,560 There's been a lot of stories right with that. 658 00:30:46,840 --> 00:30:51,130 There was of course the LastPass incident, which is, you know, was the 659 00:30:51,130 --> 00:30:56,200 nail in the coffin for me regarding LastPass, uh, where basically they, 660 00:30:56,525 --> 00:31:01,870 the, the hacker was able to gain access to a backup of the vault. 661 00:31:02,605 --> 00:31:08,215 Because they had a, they had, they had, they weren't able to decrypt the vault, 662 00:31:08,215 --> 00:31:12,145 but they were get a, they were able to get access to the vault because there 663 00:31:12,145 --> 00:31:17,635 was this backup script that had hard coded, um, you know, credentials in it. 664 00:31:17,635 --> 00:31:20,845 And then once they gained there, there was a, there was an initial breach 665 00:31:21,235 --> 00:31:24,925 and then that breach gave them access to see the script, which then gave 666 00:31:24,925 --> 00:31:28,675 them access to the credentials, which gave them access to the backup, which 667 00:31:28,675 --> 00:31:31,465 gave them access to, uh, the vault. 668 00:31:32,380 --> 00:31:36,640 Which was, I think then used for targeted crypto attacks. 669 00:31:37,095 --> 00:31:39,765 was used for targeted crypto attacks against people. 670 00:31:39,825 --> 00:31:40,185 Yeah. 671 00:31:40,935 --> 00:31:44,115 Um, and so really all we're, you know, we're just talking about, just 672 00:31:44,175 --> 00:31:48,045 realize that human error is still one of the val, the biggest reasons that 673 00:31:48,045 --> 00:31:49,665 we do anything in the backup world. 674 00:31:49,965 --> 00:31:52,455 And, and when we talk about this just. 675 00:31:53,335 --> 00:31:56,935 Try to implement the concept of least privilege. 676 00:31:56,935 --> 00:32:02,995 Try to implement the concept of minimizing the blast radius, minimizing the ability 677 00:32:02,995 --> 00:32:07,675 for somebody to do something or, you know, the, the more dangerous something is, 678 00:32:07,675 --> 00:32:09,355 the more controls you can put around it. 679 00:32:09,745 --> 00:32:10,825 Uh, or should put around it. 680 00:32:11,020 --> 00:32:12,880 so going to a sentence, you just said, 681 00:32:13,225 --> 00:32:13,735 Yeah. 682 00:32:13,810 --> 00:32:15,310 humans are always at risk, 683 00:32:16,345 --> 00:32:17,035 Humans. 684 00:32:17,245 --> 00:32:17,635 Yeah. 685 00:32:17,770 --> 00:32:18,220 Right. 686 00:32:18,925 --> 00:32:19,495 Yeah. 687 00:32:19,540 --> 00:32:19,810 So. 688 00:32:20,005 --> 00:32:21,565 Humans are always de-risk, I would say. 689 00:32:21,565 --> 00:32:21,655 Yeah. 690 00:32:22,225 --> 00:32:24,775 So what if people are like, Hey, forget about humans. 691 00:32:24,775 --> 00:32:26,155 Let's switch over to ai. 692 00:32:28,810 --> 00:32:29,305 No, no, no. 693 00:32:29,335 --> 00:32:30,775 I, I wanna play a thought experiment, 694 00:32:31,630 --> 00:32:31,631 uh. 695 00:32:31,975 --> 00:32:34,855 What, uh, would your views change? 696 00:32:34,855 --> 00:32:38,875 Would your, you would recommend, because for humans, right? 697 00:32:38,875 --> 00:32:43,315 We talk about putting controls, doing multifactor authentication, multi 698 00:32:43,615 --> 00:32:50,005 authentication, All of these things, if people start to move towards ai. 699 00:32:51,655 --> 00:32:53,935 How does this change? 700 00:32:53,995 --> 00:32:57,145 I know this episode is talking about human error, right? 701 00:32:57,145 --> 00:32:58,195 But just to leave 702 00:32:58,255 --> 00:33:00,685 don't, I don't think I'm ready to answer that question. 703 00:33:02,755 --> 00:33:04,915 We'll, we'll cross that bridge when we come to it. 704 00:33:04,945 --> 00:33:05,545 How's that? 705 00:33:05,590 --> 00:33:05,880 Okay. 706 00:33:06,055 --> 00:33:09,715 I, well, I'll just say this in general, I'm a fan of automation and AI can 707 00:33:09,715 --> 00:33:14,335 enable automation, but it's gotta be put into a, it's gotta beral. 708 00:33:14,785 --> 00:33:14,935 and 709 00:33:14,935 --> 00:33:15,235 Yeah. 710 00:33:15,235 --> 00:33:15,445 else 711 00:33:15,505 --> 00:33:15,865 Yeah. 712 00:33:16,195 --> 00:33:16,645 Again, 713 00:33:16,735 --> 00:33:16,945 may be 714 00:33:17,095 --> 00:33:17,425 with 715 00:33:17,425 --> 00:33:17,815 but 716 00:33:18,025 --> 00:33:18,595 power 717 00:33:18,595 --> 00:33:18,895 yeah. 718 00:33:19,165 --> 00:33:19,615 great. 719 00:33:20,005 --> 00:33:21,235 You know, responsibility. 720 00:33:21,235 --> 00:33:21,445 Right. 721 00:33:21,445 --> 00:33:27,625 And so you want to, you want to, um, um, sorry. 722 00:33:27,625 --> 00:33:29,485 I just found myself thinking about, um, 723 00:33:30,085 --> 00:33:30,755 Spider-Man. 724 00:33:30,985 --> 00:33:31,855 Stanwell Stan Lee. 725 00:33:33,115 --> 00:33:37,285 I wonder if Stan Lee thinks about d the fact that his Spider-Man line would be 726 00:33:37,285 --> 00:33:40,600 quoted at a IT podcast decades later. 727 00:33:41,500 --> 00:33:41,720 Um. 728 00:33:43,270 --> 00:33:44,380 The, uh, yeah. 729 00:33:44,380 --> 00:33:48,490 So the more power the AI has, the more control you gotta put around it. 730 00:33:48,640 --> 00:33:50,080 Because it could do things right. 731 00:33:50,080 --> 00:33:51,520 It could be used to do things. 732 00:33:51,610 --> 00:33:51,850 Yeah. 733 00:33:53,560 --> 00:33:53,980 All right. 734 00:33:53,980 --> 00:33:59,230 Well, this has been fun, sad, depressing, uh, et cetera, but that's why we back up. 735 00:34:00,265 --> 00:34:00,485 And, 736 00:34:00,490 --> 00:34:01,210 And, um, 737 00:34:01,330 --> 00:34:02,245 and that's why we're human. 738 00:34:02,920 --> 00:34:03,550 yes. 739 00:34:03,820 --> 00:34:06,910 And I, I wanna wish you, you, you have an upcoming trip. 740 00:34:07,150 --> 00:34:07,660 Uh. 741 00:34:07,720 --> 00:34:07,840 I 742 00:34:08,095 --> 00:34:11,125 Yeah, wanna wish you a good trip, Bon voyage. 743 00:34:11,200 --> 00:34:11,560 Thank you. 744 00:34:11,560 --> 00:34:12,040 Thank you. 745 00:34:12,545 --> 00:34:14,765 I hope our listeners get to have a trip coming up. 746 00:34:14,765 --> 00:34:16,385 You know, maybe you have some fun. 747 00:34:17,135 --> 00:34:18,545 At least somebody will have some fun. 748 00:34:19,415 --> 00:34:21,575 And, uh, with that, that is a wrap. 749 00:34:25,367 --> 00:34:30,047 The backup wrap up is written, recorded, and produced by me w Curtis Preston. 750 00:34:30,617 --> 00:34:35,417 If you need backup or Dr. Consulting content generation or expert witness 751 00:34:35,417 --> 00:34:37,997 work, check out backup central.com. 752 00:34:38,717 --> 00:34:41,837 You can also find links from my O'Reilly Books on the same website. 753 00:34:42,497 --> 00:34:46,487 Remember, this is an independent podcast and any opinions that 754 00:34:46,487 --> 00:34:50,387 you hear are those of the speaker and not necessarily an employer. 755 00:34:51,317 --> 00:34:52,007 Thanks for listening.