1 00:00:00,000 --> 00:00:02,279 W. Curtis Preston: Welcome to the exciting final episode in 2 00:00:02,309 --> 00:00:04,499 our series on cloud disasters. 3 00:00:04,799 --> 00:00:07,529 During which we conclusively answered the question. 4 00:00:07,679 --> 00:00:10,409 Do I have to back up my data in the cloud? 5 00:00:10,829 --> 00:00:12,689 The answer is an unequivocal. 6 00:00:12,689 --> 00:00:13,319 Yes. 7 00:00:13,559 --> 00:00:17,069 Not having a good backup and disaster recovery plan for your data in 8 00:00:17,069 --> 00:00:20,729 the cloud is nothing short of an existential threat to your company. 9 00:00:21,299 --> 00:00:25,559 Two out of the 10 companies we covered in this series immediately went out 10 00:00:25,559 --> 00:00:27,029 of business after what happened. 11 00:00:27,419 --> 00:00:30,119 And another killed off an entire line of business. 12 00:00:30,479 --> 00:00:33,869 The cloud is an amazing place and I consider myself an 13 00:00:33,869 --> 00:00:36,569 advocate, but it is not magic. 14 00:00:36,689 --> 00:00:39,059 It is just someone else's computer. 15 00:00:39,839 --> 00:00:43,799 In this episode, we will summarize the lessons that we learned and the major 16 00:00:43,799 --> 00:00:48,779 arguments that we settled from our coverage of no less than 10 disasters. 17 00:00:49,199 --> 00:00:52,529 If you're just joining us, hopefully this episode will wet your appetite 18 00:00:52,529 --> 00:00:56,729 enough to have you go back and listen to, or watch those other episodes. 19 00:00:57,119 --> 00:00:58,739 We had a good time making them. 20 00:00:58,799 --> 00:01:00,209 And we also learned a lot. 21 00:01:00,779 --> 00:01:05,459 By the way, if you don't know who I am, I'm w Curtis press an AKA Mr. 22 00:01:05,459 --> 00:01:05,969 Backup. 23 00:01:06,359 --> 00:01:10,739 And I've been passionate about backup and recovery for over 30 years. 24 00:01:10,799 --> 00:01:14,759 Ever since I had to tell my boss that we had no backups of the 25 00:01:14,759 --> 00:01:16,829 production database, we just lost. 26 00:01:17,399 --> 00:01:19,049 I don't want that to happen to you. 27 00:01:19,199 --> 00:01:22,019 And that's why I do this on this podcast. 28 00:01:22,019 --> 00:01:26,519 We turn unappreciated backup admins into cyber recovery heroes. 29 00:01:26,759 --> 00:01:28,979 This is the backup wrap up. 30 00:01:42,878 --> 00:01:43,698 W. Curtis Preston: Welcome to the show. 31 00:01:43,698 --> 00:01:47,290 I'm w Curtis Preston, AKA, Mr. 32 00:01:47,290 --> 00:01:51,245 Backup, and I have with me my trauma sympathizer Prasanna 33 00:01:51,605 --> 00:01:53,805 Malaiyandi, how's it going? 34 00:01:53,955 --> 00:01:54,445 Persona. 35 00:01:55,630 --> 00:01:57,340 Prasanna Malaiyandi: I am good, Curtis. 36 00:01:57,640 --> 00:01:57,700 Yeah. 37 00:01:57,850 --> 00:02:03,100 Um, it's interesting of all the things you can complain about or that are 38 00:02:03,100 --> 00:02:09,310 wrong in this world for you, the biggest issue is different operating 39 00:02:09,310 --> 00:02:11,830 systems using back slashes differently. 40 00:02:12,855 --> 00:02:12,975 W. Curtis Preston: Y. 41 00:02:12,980 --> 00:02:14,050 Yeah, so. 42 00:02:15,370 --> 00:02:21,555 It's because I'm being forced to operate a Windows based net backup server. 43 00:02:22,140 --> 00:02:22,260 I. 44 00:02:22,260 --> 00:02:26,070 And I only know how to work in like Unix land, like when I'm doing automation 45 00:02:26,070 --> 00:02:29,160 and scripting and stuff, which is the only way I know how to do anything. 46 00:02:29,250 --> 00:02:29,520 Right. 47 00:02:29,525 --> 00:02:30,960 I, I, I don't understand. 48 00:02:31,230 --> 00:02:35,075 I, I guess people, if they grew up in Windows Land, they learn PowerShell. 49 00:02:36,275 --> 00:02:40,145 They learn, you know, batch programming and what I didn't learn all that stuff. 50 00:02:40,145 --> 00:02:41,825 I learned shell scripting. 51 00:02:42,095 --> 00:02:46,115 And so when I want to administer a Windows-based system, the first 52 00:02:46,120 --> 00:02:50,225 thing I do is install siggu so that I have a Unix-based environment. 53 00:02:51,215 --> 00:02:55,415 But when I'm interacting with, um. 54 00:02:56,390 --> 00:02:57,350 Files. 55 00:02:58,460 --> 00:02:58,880 Right. 56 00:02:59,450 --> 00:03:06,410 Um, so sig one is nice in that it translates all the file directory names 57 00:03:06,410 --> 00:03:08,660 and everything into slashes, right? 58 00:03:08,660 --> 00:03:12,170 So it's, I just have to, um, sort of figure that out. 59 00:03:12,170 --> 00:03:18,500 And basically, like Deco backslash becomes slash sig drive slash d, so 60 00:03:18,500 --> 00:03:20,330 you just sort of, that's not too bad. 61 00:03:20,690 --> 00:03:24,650 But in the case of net backup, I'm doing things like. 62 00:03:25,305 --> 00:03:32,235 I am running commands that output file names that sometimes are in Windows 63 00:03:32,235 --> 00:03:34,515 format and sometimes are in Unix format. 64 00:03:35,205 --> 00:03:42,315 And, um, the, what I've recently discovered is that, and it took me, 65 00:03:42,315 --> 00:03:46,575 it took me literally just banging my head into the wa into the wall 66 00:03:46,575 --> 00:03:48,975 a few times to figure this out. 67 00:03:49,035 --> 00:03:50,985 Is that, um. 68 00:03:51,620 --> 00:03:55,010 Back slashes, behave differently. 69 00:03:55,940 --> 00:04:00,110 Um, you know, like I wanna massage them and the first thing I wanna do 70 00:04:00,115 --> 00:04:01,970 is change 'em and four slashes, right? 71 00:04:01,970 --> 00:04:05,210 So that I can, so that they're not gonna be escaping everything in my unit. 72 00:04:05,300 --> 00:04:06,230 She script, right? 73 00:04:06,830 --> 00:04:15,260 Um, they behave differently on like in standard in then they do in a file. 74 00:04:16,100 --> 00:04:20,450 So if I take the output of a command, I. 75 00:04:21,125 --> 00:04:27,335 Such as BPPL include, which is, you know, show me the files that are in this policy. 76 00:04:28,145 --> 00:04:32,705 Uh, and I pipe that directly into a, uh, you know, into a series of 77 00:04:33,065 --> 00:04:39,365 pipes to massage the, the output, uh, that backslash behaves differently. 78 00:04:39,370 --> 00:04:42,455 I have to escape it differently if I'm just doing a, a straight. 79 00:04:44,180 --> 00:04:44,450 Prasanna Malaiyandi: Yeah, 80 00:04:45,290 --> 00:04:48,890 W. Curtis Preston: if I out, if I take the, the command and actually 81 00:04:48,890 --> 00:04:51,920 output it to a file and then 82 00:04:52,040 --> 00:04:53,630 and then look at it. 83 00:04:54,710 --> 00:05:00,830 Um, and then basically I have to es like if I'm doing it on the command 84 00:05:00,830 --> 00:05:03,620 line, I have to escape the escape twice. 85 00:05:03,620 --> 00:05:07,790 If I'm doing, if I'm doing it in a filing, I only have to escape it once. 86 00:05:09,230 --> 00:05:12,440 And that's been driving me just absolutely bonkers. 87 00:05:14,150 --> 00:05:17,660 Prasanna Malaiyandi: So I think what you do is you just add a whole 88 00:05:17,660 --> 00:05:20,360 bunch of escapes in there, you know, 89 00:05:20,375 --> 00:05:20,675 W. Curtis Preston: I 90 00:05:20,900 --> 00:05:21,530 Prasanna Malaiyandi: slashes 91 00:05:21,545 --> 00:05:22,145 W. Curtis Preston: it works. 92 00:05:22,295 --> 00:05:24,905 I'm just like, does two will two work? 93 00:05:24,930 --> 00:05:25,150 No. 94 00:05:25,210 --> 00:05:26,165 Two, no, no. 95 00:05:26,165 --> 00:05:27,185 3, 3, 3. 96 00:05:27,215 --> 00:05:27,575 No. 97 00:05:27,580 --> 00:05:31,355 And then three, like three, uh, tells me that I, that I have too 98 00:05:31,355 --> 00:05:35,135 many and I have like, uh, it, it, it, it escapes like something 99 00:05:35,135 --> 00:05:36,516 later and I'm like, no, no, no, no. 100 00:05:36,521 --> 00:05:37,895 Okay, so I won't take four. 101 00:05:37,895 --> 00:05:42,455 You know, and I just, I just do a bunch of them until. 102 00:05:42,845 --> 00:05:43,415 I get what I 103 00:05:43,595 --> 00:05:44,075 Prasanna Malaiyandi: It works. 104 00:05:44,910 --> 00:05:45,200 W. Curtis Preston: Yeah. 105 00:05:45,365 --> 00:05:47,795 Prasanna Malaiyandi: It's like, it's like, uh, it's like writing code, right? 106 00:05:48,035 --> 00:05:49,355 You compile it, it fails. 107 00:05:49,355 --> 00:05:51,515 You're like, oh, I'll fix this, and then it fails again. 108 00:05:51,515 --> 00:05:53,825 It's like, oh, I'll fix this, and then you just keep going until 109 00:05:53,825 --> 00:05:55,145 eventually you get zero errors. 110 00:05:55,400 --> 00:06:00,740 W. Curtis Preston: And, and, and then, so I, I find that as annoying as this is, I 111 00:06:00,740 --> 00:06:06,230 find that it's actually easier to take a net backup command, output it to a file, 112 00:06:06,650 --> 00:06:11,090 and then hash through that file, rather than just doing it as a series of pipes. 113 00:06:11,750 --> 00:06:18,740 And as annoying as that is, it's more, it seems more consistent, um, albeit slower. 114 00:06:19,250 --> 00:06:22,460 And it's also slower because the first thing I have to do on that 115 00:06:22,460 --> 00:06:23,630 file, you know what I have to do? 116 00:06:24,560 --> 00:06:27,260 I have to run the dos2unix command on it 117 00:06:28,100 --> 00:06:29,930 because of the, the new line. 118 00:06:29,960 --> 00:06:31,850 Because the new line is also different. 119 00:06:32,780 --> 00:06:33,170 Prasanna Malaiyandi: Yeah, 120 00:06:36,891 --> 00:06:41,001 W. Curtis Preston: It, it's been fun is what I would say. 121 00:06:41,346 --> 00:06:42,456 Prasanna Malaiyandi: Oh, Curtis 122 00:06:43,971 --> 00:06:49,881 W. Curtis Preston: Ah, so, uh, so speaking of fun persona, we 123 00:06:49,886 --> 00:06:54,591 have now terminated our cloud 124 00:06:54,726 --> 00:06:55,686 Prasanna Malaiyandi: concluded. 125 00:06:55,791 --> 00:06:56,781 W. Curtis Preston: series. 126 00:06:58,086 --> 00:06:58,746 Prasanna Malaiyandi: Concluded, 127 00:06:59,331 --> 00:06:59,931 W. Curtis Preston: What's that? 128 00:07:00,726 --> 00:07:01,566 Prasanna Malaiyandi: concluded. 129 00:07:02,061 --> 00:07:03,291 W. Curtis Preston: Why did I say terminated? 130 00:07:03,936 --> 00:07:04,266 Prasanna Malaiyandi: Yeah. 131 00:07:04,266 --> 00:07:05,556 Terminated just seems. 132 00:07:05,661 --> 00:07:06,861 W. Curtis Preston: said I'm done. 133 00:07:07,371 --> 00:07:09,441 I'm done with all these people losing data. 134 00:07:09,621 --> 00:07:09,891 Yeah. 135 00:07:09,891 --> 00:07:14,391 We have concluded our cloud disaster series. 136 00:07:14,391 --> 00:07:15,231 What did you think? 137 00:07:16,371 --> 00:07:17,721 Prasanna Malaiyandi: I liked it actually. 138 00:07:18,023 --> 00:07:20,603 It's one of those things, like a disaster happens. 139 00:07:20,603 --> 00:07:22,283 You read it on the news and then you forget about it, right? 140 00:07:22,283 --> 00:07:23,183 It says to the next day. 141 00:07:24,053 --> 00:07:27,203 And or you, there are many of these that like I don't think 142 00:07:27,203 --> 00:07:28,433 any of us had heard about. 143 00:07:28,433 --> 00:07:28,763 Right. 144 00:07:28,793 --> 00:07:34,373 And just being able to go through and just understand and kind of take a look right 145 00:07:34,378 --> 00:07:38,873 and getting your perspective right and seeing, okay, what are these disasters? 146 00:07:38,878 --> 00:07:41,963 And really sort of doing the research ahead of time. 147 00:07:42,173 --> 00:07:42,953 It was kind of fun. 148 00:07:42,983 --> 00:07:43,493 I liked them. 149 00:07:43,838 --> 00:07:45,098 W. Curtis Preston: Yeah, it was fun. 150 00:07:45,098 --> 00:07:50,858 It was, um, what I liked about it was that enough time had passed with most 151 00:07:50,858 --> 00:07:56,468 of these disasters that we often had the other half of the story, right? 152 00:07:56,468 --> 00:08:00,098 When the, when, when they happen, uh, you know, at the mo you know, 153 00:08:00,098 --> 00:08:04,178 at that moment you check it and you hear like in the case of, um. 154 00:08:04,778 --> 00:08:11,253 The, uh, the Musi outage, the first thing you hear is that, you know, musi has sued 155 00:08:11,253 --> 00:08:15,273 Google because, you know, they lost their data and, and that makes the headlines 156 00:08:15,273 --> 00:08:18,783 and everybody's, oh, you know, there's a lawsuit, you know, but you know, this 157 00:08:18,783 --> 00:08:21,453 is America, so anybody can sue anything. 158 00:08:21,453 --> 00:08:24,573 I mean, you may remember that there was a woman who sued. 159 00:08:25,083 --> 00:08:29,853 Google Maps because she drove her car off a pier. 160 00:08:30,273 --> 00:08:30,633 Right. 161 00:08:30,663 --> 00:08:32,493 Because Google Maps told her to go that way. 162 00:08:32,493 --> 00:08:33,003 Right. 163 00:08:33,003 --> 00:08:36,933 You know, you could sue for anything, but that doesn't mean 164 00:08:36,933 --> 00:08:38,133 you're gonna be successful. 165 00:08:38,523 --> 00:08:44,793 And it was nice to go back and look at the, um, you know what happened 166 00:08:44,793 --> 00:08:49,303 after, in some of the cases, we got some additional insight from people 167 00:08:49,303 --> 00:08:50,953 that were, you know, in and around. 168 00:08:50,953 --> 00:08:52,573 I, I think of the. 169 00:08:53,953 --> 00:08:55,963 OOVH incident. 170 00:08:56,083 --> 00:08:56,533 Right? 171 00:08:56,563 --> 00:08:57,493 Uh, that was nice. 172 00:08:57,493 --> 00:09:05,203 We got some insight into that and how that OVH really was known for the, you 173 00:09:05,203 --> 00:09:09,133 know, being a low cost provider and so that nobody was pro, nobody was surprised 174 00:09:09,133 --> 00:09:11,618 when they, uh, didn't have, didn't have. 175 00:09:11,863 --> 00:09:12,133 Prasanna Malaiyandi: Yeah. 176 00:09:12,703 --> 00:09:12,943 Yeah. 177 00:09:13,423 --> 00:09:17,953 Well, the other thing also that I came to realize from the series is, 178 00:09:18,133 --> 00:09:19,903 like you said, time had elapsed. 179 00:09:20,203 --> 00:09:24,433 It's also interesting to see how the company, some companies try 180 00:09:24,433 --> 00:09:25,783 to sweep things under the rug, 181 00:09:26,203 --> 00:09:26,553 W. Curtis Preston: Right, 182 00:09:27,343 --> 00:09:27,613 Prasanna Malaiyandi: right? 183 00:09:27,613 --> 00:09:33,643 And thanks to the wonders of the internet where nothing is ever gone, right? 184 00:09:33,643 --> 00:09:37,063 Being able to uncover, okay, really, what was it that was initially 185 00:09:37,063 --> 00:09:38,593 said versus what do they say today? 186 00:09:39,508 --> 00:09:40,078 W. Curtis Preston: Yeah. 187 00:09:40,108 --> 00:09:41,608 Uh, thank you very much. 188 00:09:41,698 --> 00:09:44,398 Um, internet archive, right? 189 00:09:44,428 --> 00:09:46,498 That was, that was, that was very helpful. 190 00:09:46,738 --> 00:09:50,608 In some cases we weren't able to get stuff, but, and there were a lot of cases 191 00:09:50,613 --> 00:09:54,448 where we were able to get basically the first version of the story versus the, 192 00:09:54,628 --> 00:09:56,368 the later, the, the latter version. 193 00:09:56,878 --> 00:09:57,058 Prasanna Malaiyandi: Yeah. 194 00:09:57,538 --> 00:10:02,158 Uh, but, but I would say in most of these cases, right, the initial 195 00:10:02,158 --> 00:10:07,198 information that people saw was what was allowed to be released, right? 196 00:10:07,228 --> 00:10:11,098 Whatever the company had at that time, which sometimes they're working off 197 00:10:11,103 --> 00:10:14,428 incomplete information, but they still wanted to provide some information 198 00:10:14,428 --> 00:10:19,798 to their users, such as like the OVH incident or Rackspace, right? 199 00:10:19,798 --> 00:10:21,868 Where it's just, okay, what information is available? 200 00:10:21,868 --> 00:10:23,218 Let me at least push something out there. 201 00:10:24,178 --> 00:10:24,538 W. Curtis Preston: Right. 202 00:10:24,538 --> 00:10:28,378 And that, and that's what we encourage, uh, companies to do, right, is to 203 00:10:29,008 --> 00:10:32,728 be upfront about what's happening. 204 00:10:32,733 --> 00:10:37,888 And it's okay to say you don't know yet, but acknowledge what's happening. 205 00:10:37,918 --> 00:10:40,798 You know, acknowledge what you know when you know it. 206 00:10:40,858 --> 00:10:42,538 Don't make stuff up. 207 00:10:43,318 --> 00:10:43,828 Right. 208 00:10:43,918 --> 00:10:44,788 Um. 209 00:10:45,118 --> 00:10:51,063 And the, and then, and then the, the other thing that I would say is, um. 210 00:10:51,988 --> 00:10:55,108 Is we like it when they don't try to pass blame. 211 00:10:55,108 --> 00:10:56,638 That happened a couple of times. 212 00:10:57,178 --> 00:11:00,388 Uh, when the blame is, you know, clearly on you and your design, 213 00:11:00,898 --> 00:11:02,488 uh, don't try to pass blame. 214 00:11:02,968 --> 00:11:08,098 So we've got, I came up with a few lessons that I wrote down, uh, and 215 00:11:08,098 --> 00:11:11,368 if, and if, and if a few others pop up in our head while we're recording 216 00:11:11,368 --> 00:11:12,718 this episode, just come out with 'em. 217 00:11:13,583 --> 00:11:18,148 Um, and the first one is actually, uh, you put that on there. 218 00:11:18,148 --> 00:11:20,758 So you wanna, you wanna go for it? 219 00:11:21,123 --> 00:11:21,693 Prasanna Malaiyandi: Yeah. 220 00:11:21,753 --> 00:11:22,083 Yeah. 221 00:11:22,083 --> 00:11:25,023 So I would say that the most important thing for a lot of 222 00:11:25,023 --> 00:11:30,033 these is if it's important to you, the data, of course, back it up, 223 00:11:30,853 --> 00:11:31,183 W. Curtis Preston: Yeah. 224 00:11:31,963 --> 00:11:36,073 Which is, which is basically backup lesson number one. 225 00:11:36,613 --> 00:11:36,913 Is 226 00:11:37,098 --> 00:11:38,118 Prasanna Malaiyandi: Or zero, I would say. 227 00:11:38,353 --> 00:11:38,623 W. Curtis Preston: Yeah. 228 00:11:38,623 --> 00:11:38,983 Backup. 229 00:11:40,063 --> 00:11:41,293 It's Rule zero. 230 00:11:41,773 --> 00:11:46,543 Uh, it's one that I absolutely cannot argue with, right? 231 00:11:46,543 --> 00:11:48,433 It's something that I say to you, right? 232 00:11:49,238 --> 00:11:49,398 Prasanna Malaiyandi: Yep. 233 00:11:49,828 --> 00:11:52,708 W. Curtis Preston: I, I say this all the time, that if, if this data 234 00:11:52,738 --> 00:11:58,168 really is important to you, then it should be, you know, backed up, right? 235 00:11:58,198 --> 00:12:04,708 And we can talk in a little bit as to what constitutes a backup and, 236 00:12:04,738 --> 00:12:07,588 uh, that'll be some of the lessons that we learned along the way. 237 00:12:08,188 --> 00:12:09,628 But, um. 238 00:12:11,983 --> 00:12:16,423 I, I'm gonna say in almost all instances, what constitutes a 239 00:12:16,423 --> 00:12:18,373 backup is you making a backup. 240 00:12:19,423 --> 00:12:24,583 Not, not saying, oh, I got it because I got it, because it's the cloud, 241 00:12:24,583 --> 00:12:27,403 or because it's SaaS, or because it's, you know, it's not my problem. 242 00:12:27,408 --> 00:12:28,903 No, it's always your problem. 243 00:12:29,713 --> 00:12:32,593 Prasanna Malaiyandi: Yeah, and there were companies, right, like Musi went out 244 00:12:32,593 --> 00:12:36,823 of business because they lost all their data and they couldn't recover it, and 245 00:12:37,243 --> 00:12:38,863 that was the end of the company, right? 246 00:12:38,863 --> 00:12:39,043 So, 247 00:12:39,808 --> 00:12:40,168 W. Curtis Preston: Yeah. 248 00:12:40,168 --> 00:12:43,228 And, and I'd say that that's, that's the first lesson I'm gonna say that 249 00:12:43,228 --> 00:12:47,218 wasn't actually on the list, is that if you don't get this right. 250 00:12:47,743 --> 00:12:51,523 Your entire company may just go away, right? 251 00:12:51,523 --> 00:12:55,813 That happened, uh, in at least two instances in the, the list 252 00:12:55,813 --> 00:12:57,133 of the companies that we found. 253 00:12:57,523 --> 00:13:03,163 Either your company or a significant portion of your company may go away. 254 00:13:03,163 --> 00:13:04,603 I think of Rackspace, right? 255 00:13:04,608 --> 00:13:10,453 Rackspace is now completely out of the hosted exchange business because. 256 00:13:10,753 --> 00:13:13,933 Uh, of what happened with the rec based outage, right? 257 00:13:14,293 --> 00:13:17,203 So either your entire company or a significant portion of your company 258 00:13:17,203 --> 00:13:19,033 can literally just go, poof, overnight. 259 00:13:20,893 --> 00:13:20,953 Prasanna Malaiyandi: Yeah. 260 00:13:21,928 --> 00:13:25,438 So what's the first one, Curtis, in your opinion, like what's the 261 00:13:25,438 --> 00:13:28,673 most important lesson from all of the ones that we've looked at? 262 00:13:29,983 --> 00:13:30,763 W. Curtis Preston: 3, 2, 1. 263 00:13:30,763 --> 00:13:32,413 Rule still rules. 264 00:13:34,918 --> 00:13:37,738 And, and I know that there are other versions of the 3, 2, 1 rule. 265 00:13:37,888 --> 00:13:45,178 I'm mu in this case, what I'm saying is if what you are doing for backup, as I make 266 00:13:45,178 --> 00:13:51,778 quotes in the air, doesn't at least comply to the 3, 2, 1 rule, it is not a backup. 267 00:13:52,438 --> 00:13:52,828 Right? 268 00:13:53,068 --> 00:13:57,598 There are, there are many situations when we look at the stories that we, that we 269 00:13:57,603 --> 00:14:03,298 talked about here, where people had what they might have considered a backup. 270 00:14:03,628 --> 00:14:04,108 Right. 271 00:14:04,168 --> 00:14:08,278 Um, you know, musi is probably the best example of that, right? 272 00:14:08,338 --> 00:14:12,178 Um, where they had their backups with them, but basically 273 00:14:12,178 --> 00:14:13,558 they deleted their account. 274 00:14:14,128 --> 00:14:14,578 Right. 275 00:14:14,638 --> 00:14:17,278 Um, the, the other one would be Code Spaces. 276 00:14:17,278 --> 00:14:21,628 That's probably the most infamous, I think, where Code Spaces was making 277 00:14:21,633 --> 00:14:27,058 backups within the, within AWS, but they didn't follow the 3, 2, 1 rule. 278 00:14:27,058 --> 00:14:30,688 They didn't separate those backups from production. 279 00:14:30,988 --> 00:14:31,588 And so, when. 280 00:14:32,453 --> 00:14:37,943 Um, you know, when the, the bad actor went in there, they deleted the entire account. 281 00:14:37,943 --> 00:14:42,983 And again, poof, they went there, went their primary as well as their backup. 282 00:14:43,433 --> 00:14:46,763 And the same would be true of the SaaS. 283 00:14:47,243 --> 00:14:54,293 Um, people that, you know, when I think of specifically Microsoft 365, but this is 284 00:14:54,293 --> 00:14:58,223 true in so many other cases, is that, um. 285 00:14:59,173 --> 00:15:03,673 It that if you are not separating that data, if you're not making 286 00:15:03,673 --> 00:15:08,593 another copy, they don't have, uh, multiple copies of your data, right? 287 00:15:08,593 --> 00:15:10,033 That, that are accessible to you. 288 00:15:10,038 --> 00:15:16,783 They may have, may have a DR copy, but as we saw in some of the cases, the DR. 289 00:15:16,783 --> 00:15:18,343 Copy does doesn't always work. 290 00:15:18,823 --> 00:15:19,963 Um, that Dr. 291 00:15:19,963 --> 00:15:20,263 Copy 292 00:15:20,263 --> 00:15:24,103 might not be to you, and it might not, it might not work, but. 293 00:15:24,388 --> 00:15:28,288 But the point is that if you're not, if you don't have a whatever they 294 00:15:28,288 --> 00:15:30,478 have, it's all in the same place. 295 00:15:31,138 --> 00:15:34,558 And when very bad things happen, uh, your SOL 296 00:15:35,848 --> 00:15:38,368 Prasanna Malaiyandi: And specifically around the Microsoft case, right? 297 00:15:38,908 --> 00:15:42,778 They may have a DR copy, but like you said, that's for their purposes 298 00:15:42,778 --> 00:15:45,358 not to allow you to do your restores. 299 00:15:45,598 --> 00:15:49,138 That's to make sure if something blows up somewhere from an infrastructure 300 00:15:49,138 --> 00:15:52,018 perspective, they can break things back up and they'll try their 301 00:15:52,018 --> 00:15:54,298 best to give you back all your data, but there's no guarantees. 302 00:15:55,348 --> 00:15:56,548 W. Curtis Preston: Yeah, exactly. 303 00:15:56,598 --> 00:16:00,108 Actually we're gonna get more into detail into that in, in a latter step. 304 00:16:00,648 --> 00:16:04,458 But yeah, so short version is, I, I have no problem with app 305 00:16:04,458 --> 00:16:06,258 pending to the 3, 2, 1 rule. 306 00:16:06,258 --> 00:16:10,428 If you wanna say 3, 2, 1, 1, 0, 7, 9, 5, I don't care. 307 00:16:10,878 --> 00:16:11,238 Right. 308 00:16:11,538 --> 00:16:15,288 Um, and I, and I don't disagree with any of the versions that I've heard. 309 00:16:15,528 --> 00:16:15,858 Right. 310 00:16:15,858 --> 00:16:16,548 Things like I. 311 00:16:16,568 --> 00:16:19,268 Making sure that, you know, one of the copies, the biggest one is 312 00:16:19,268 --> 00:16:23,198 making sure that one of those copies is an immutable copy, which is a 313 00:16:23,198 --> 00:16:27,428 relatively new requirement, and that has to do with, uh, cyber attacks. 314 00:16:27,578 --> 00:16:28,628 I don't have a problem with that. 315 00:16:28,633 --> 00:16:33,188 But if what you're doing doesn't comply with the 3, 2, 1 rule, uh, three 316 00:16:33,193 --> 00:16:36,938 copies of your data on two different types of media, one of which is 317 00:16:37,178 --> 00:16:42,818 physically separated from the other, um, then it, you don't have a backup. 318 00:16:44,508 --> 00:16:47,358 Prasanna Malaiyandi: I do wonder, and maybe it doesn't have to be for 319 00:16:47,358 --> 00:16:54,438 this episode, if we should really figure out does the three or have a 320 00:16:54,443 --> 00:16:59,028 discussion on an episode of does the 3, 2 1 rule really need to be changed? 321 00:17:00,753 --> 00:17:01,293 W. Curtis Preston: Um, I, 322 00:17:01,388 --> 00:17:01,593 I think, 323 00:17:02,238 --> 00:17:05,448 Prasanna Malaiyandi: I think just because I know that there's like different, 324 00:17:05,448 --> 00:17:07,788 like you said, the 3, 2, 1, 1 0. 325 00:17:08,343 --> 00:17:08,643 Zero. 326 00:17:08,703 --> 00:17:09,003 Zero. 327 00:17:09,123 --> 00:17:11,253 It may not be a zero, but you know what I mean. 328 00:17:11,253 --> 00:17:11,523 Right. 329 00:17:11,838 --> 00:17:15,678 W. Curtis Preston: Yeah, well there, there's various versions 330 00:17:15,678 --> 00:17:17,418 out there and I'm fine with those. 331 00:17:17,418 --> 00:17:22,398 I do myself, like, you know, a, you know, I add to it. 332 00:17:22,428 --> 00:17:24,738 One of these copies needs to be immutable. 333 00:17:24,738 --> 00:17:27,378 I am, I fully agree with that. 334 00:17:27,588 --> 00:17:29,538 Does it need to be part of the 3, 2, 1 rule? 335 00:17:29,538 --> 00:17:29,988 I don't know. 336 00:17:29,988 --> 00:17:30,618 I, you know. 337 00:17:31,578 --> 00:17:36,108 I like the 3, 2, 1 for, for what it does, because what it does is it proves what 338 00:17:36,108 --> 00:17:41,088 we've been talking about here is that if you have a cloud service and all of 339 00:17:41,088 --> 00:17:44,958 the copies of your data are all in the same place, you don't have a backup. 340 00:17:45,648 --> 00:17:45,828 Right? 341 00:17:46,263 --> 00:17:46,623 Prasanna Malaiyandi: Yep. 342 00:17:46,833 --> 00:17:50,823 Speaking of which, uh, I would say that the next lesson learned is really around. 343 00:17:51,588 --> 00:17:51,858 Right. 344 00:17:51,858 --> 00:17:55,338 Like you mentioned, the cloud is not backing up your data. 345 00:17:56,178 --> 00:17:57,978 You may think it is backing up your data. 346 00:17:58,008 --> 00:18:02,478 They may be doing DR copies for their own purposes, but it's not a backup 347 00:18:02,958 --> 00:18:08,748 that you as a customer, a user can go restore from, can go back to any point 348 00:18:08,748 --> 00:18:11,208 in time, recover it all the rest of that. 349 00:18:12,408 --> 00:18:14,418 Which I think a lot of people mistake, right? 350 00:18:14,418 --> 00:18:20,418 They're like, oh, I have a Salesforce, or Take your pick and it's a SaaS service. 351 00:18:20,418 --> 00:18:21,558 They host my data. 352 00:18:21,558 --> 00:18:22,908 They should be doing my backups. 353 00:18:22,913 --> 00:18:27,678 Or Microsoft 365, they should be doing my backups. 354 00:18:27,678 --> 00:18:31,488 And I know Curtis way back in the day when we were first working together, right? 355 00:18:31,488 --> 00:18:34,068 You were like, have you ever taken a look at Microsoft's 356 00:18:34,068 --> 00:18:35,508 contracts, terms of service? 357 00:18:35,838 --> 00:18:37,908 Do you ever see backup in there anywhere? 358 00:18:38,163 --> 00:18:38,613 W. Curtis Preston: Right. 359 00:18:39,813 --> 00:18:40,053 Yeah. 360 00:18:40,053 --> 00:18:41,943 It's just, it is just not there, right? 361 00:18:42,183 --> 00:18:43,263 That and that and that. 362 00:18:43,683 --> 00:18:45,783 That's the part it is. 363 00:18:46,173 --> 00:18:53,673 If you think that your SaaS service is backing up your data, please go 364 00:18:53,673 --> 00:18:56,253 look for it in your contract, right? 365 00:18:56,613 --> 00:18:57,843 Because you're not gonna find it. 366 00:18:58,503 --> 00:19:00,993 There may be, there are some exceptions. 367 00:19:01,323 --> 00:19:05,508 Interestingly enough, one of the exceptions was covered on 368 00:19:05,528 --> 00:19:08,463 our, on our stories, right? 369 00:19:08,613 --> 00:19:11,763 Where we had a vendor that was backing up, we're gonna talk, we're gonna, 370 00:19:11,793 --> 00:19:16,113 we had a vendor who was advertising that part of their service was backup, 371 00:19:16,113 --> 00:19:18,333 which almost never happens, right? 372 00:19:18,333 --> 00:19:19,143 You don't see that. 373 00:19:19,148 --> 00:19:22,803 Go look at what, what Microsoft advertises for 365. 374 00:19:22,803 --> 00:19:23,733 Look at Salesforce. 375 00:19:23,978 --> 00:19:27,068 Backup is not part of the offering either in terms of what they 376 00:19:27,068 --> 00:19:29,738 advertise or what's in your contract. 377 00:19:29,738 --> 00:19:33,038 And if it's not in your contract, it doesn't exist. 378 00:19:33,608 --> 00:19:33,998 Right. 379 00:19:34,358 --> 00:19:37,778 So the next one, and this is we, we've covered this a little bit already, and 380 00:19:37,778 --> 00:19:43,868 that is if they are backing up their data center, that backup is not for you. 381 00:19:44,603 --> 00:19:45,053 Right. 382 00:19:45,323 --> 00:19:47,513 Um, and, and I see this a lot. 383 00:19:47,633 --> 00:19:51,713 Um, there's again, this, this one guy that I interact with a lot online that, 384 00:19:52,103 --> 00:19:56,453 that really believes very strongly in the Microsoft way of doing things 385 00:19:56,723 --> 00:20:04,318 and in his idea that Microsoft has, for example, multiple delayed copies 386 00:20:04,738 --> 00:20:09,833 of your data, like replicated, uh, delayed, replicated copies of your data. 387 00:20:10,958 --> 00:20:14,258 That it's, it's gonna use that in the case of disaster to be 388 00:20:14,258 --> 00:20:16,208 able to recover your environment. 389 00:20:16,958 --> 00:20:20,288 And that appears to be true. 390 00:20:20,288 --> 00:20:23,198 I, I've never found it anywhere documented. 391 00:20:23,318 --> 00:20:23,708 Right? 392 00:20:23,738 --> 00:20:27,188 Which, and if it's not documented, if it's not in my contract that I don't care. 393 00:20:27,548 --> 00:20:31,748 But it appears to be true, however. 394 00:20:33,038 --> 00:20:34,958 The story of KPMG. 395 00:20:35,018 --> 00:20:42,158 If that story with the size of KPMG 145,000 employees, if what happened 396 00:20:42,158 --> 00:20:48,218 to KPMG doesn't prove to you that you don't get to use Microsoft's backup to 397 00:20:48,218 --> 00:20:52,478 save your butt, I don't know what will. 398 00:20:52,508 --> 00:20:59,168 This is a, this is a giant company paying tons of money to Microsoft every month. 399 00:20:59,963 --> 00:21:05,753 Even they weren't able to recover from this giant blunder because it 400 00:21:05,753 --> 00:21:11,873 was their fault, and that was not enough to have Microsoft use their 401 00:21:11,878 --> 00:21:14,903 Dr copy, uh, to, to save them. 402 00:21:15,288 --> 00:21:15,508 Yep. 403 00:21:15,758 --> 00:21:17,678 Prasanna Malaiyandi: Or, or if we go back and think about like 404 00:21:17,678 --> 00:21:19,628 the Salesforce example, right? 405 00:21:19,628 --> 00:21:23,108 Where they changed on permissions and everyone had 406 00:21:23,108 --> 00:21:24,908 access to every record, right? 407 00:21:24,938 --> 00:21:27,008 And it was like, how do we undo that? 408 00:21:27,013 --> 00:21:31,658 And so they tried their best and hopefully if you had a sandbox copy, they were 409 00:21:31,658 --> 00:21:33,098 able to help you out a little bit. 410 00:21:33,098 --> 00:21:36,848 But they were basically like, sorry, uh, I can't help you. 411 00:21:36,908 --> 00:21:37,328 Good luck. 412 00:21:37,748 --> 00:21:39,848 W. Curtis Preston: Yeah, that's interesting because the Salesforce 413 00:21:39,848 --> 00:21:42,308 story, it's, it was their blunder, 414 00:21:43,028 --> 00:21:43,208 Prasanna Malaiyandi: Yep. 415 00:21:43,838 --> 00:21:44,138 W. Curtis Preston: right? 416 00:21:44,143 --> 00:21:47,073 It was their blunder that, that, um, I. 417 00:21:47,438 --> 00:21:49,208 Where they messed up everybody's account. 418 00:21:49,748 --> 00:21:55,088 And even then, when we know that Salesforce has a, a backup, as I make 419 00:21:55,088 --> 00:22:01,538 quotes in the air, um, that you can pay for like $10,000 an instance, 420 00:22:01,538 --> 00:22:02,733 and it takes like weeks to get it. 421 00:22:03,878 --> 00:22:09,608 But they were like, no, we're not gonna crack open Pandora's box, even though it 422 00:22:09,608 --> 00:22:11,648 was us that messed up all these accounts. 423 00:22:12,158 --> 00:22:12,728 So, yeah. 424 00:22:12,728 --> 00:22:17,738 So th those two incidents, if, if those don't prove to you that these DR copies 425 00:22:17,743 --> 00:22:22,088 that these vendors are making aren't for your purposes, I, I don't know what will. 426 00:22:23,588 --> 00:22:25,328 Prasanna Malaiyandi: Yeah, and these are large companies, right? 427 00:22:25,328 --> 00:22:27,428 It's not like a mom and pop cloud vendor, right? 428 00:22:27,428 --> 00:22:29,263 Doing something or a SaaS application. 429 00:22:30,758 --> 00:22:31,328 W. Curtis Preston: Yeah. 430 00:22:33,978 --> 00:22:37,158 Prasanna Malaiyandi: Going alongside that, so we're saying don't trust 431 00:22:37,158 --> 00:22:40,008 the cloud vendor to do your backup. 432 00:22:40,488 --> 00:22:40,878 Right. 433 00:22:40,908 --> 00:22:45,198 And instead you really should be using a third party backup service. 434 00:22:45,198 --> 00:22:51,768 If we looked at some of the cases like Deduce where academics, people 435 00:22:51,768 --> 00:22:54,018 in academics and research, they lost. 436 00:22:55,323 --> 00:22:56,193 Studies, right? 437 00:22:56,193 --> 00:23:02,283 Because, uh, DDU went down or they lost some data, and if they had simply done 438 00:23:02,283 --> 00:23:06,783 a backup or even simply exported their data, right, they would've been fine. 439 00:23:06,788 --> 00:23:10,323 They wouldn't have had an issue, but they didn't because there wasn't part 440 00:23:10,323 --> 00:23:13,473 of the service, there wasn't anything that could easily back that up. 441 00:23:13,473 --> 00:23:15,003 And so people lost data. 442 00:23:16,008 --> 00:23:17,328 W. Curtis Preston: Yeah, exactly. 443 00:23:17,328 --> 00:23:22,668 I mean the, the Deduce story, uh, the Rackspace story, where it would've been 444 00:23:23,088 --> 00:23:28,158 probably much easier for companies when they, when Rackspace went down and then 445 00:23:28,158 --> 00:23:32,268 they immediately migrated everybody over to, to 365 would've been much easier 446 00:23:32,268 --> 00:23:36,528 for them to recover from that situation, uh, if they had their own backup. 447 00:23:36,888 --> 00:23:43,548 Um, and, and of course the OVH disaster where, uh, you know, the, they, so both 448 00:23:43,548 --> 00:23:49,968 in the case of OVH and um, deduce, we know that backup was something that 449 00:23:49,968 --> 00:23:52,278 they advertised as part of the service. 450 00:23:52,488 --> 00:23:55,548 By the way, specifically in Deduce, you looked, if you look at their 451 00:23:55,578 --> 00:23:59,028 website from back then, it says they're doing nightly backups. 452 00:23:59,988 --> 00:24:06,498 And, and we now know that we don't know exactly how or why, but the closest 453 00:24:06,498 --> 00:24:11,118 available backup was o was a month out and the one that was, and, and there 454 00:24:11,118 --> 00:24:13,278 was something a little funky with it. 455 00:24:13,283 --> 00:24:14,808 So they actually went two months out. 456 00:24:15,378 --> 00:24:19,788 Um, what, how, how is that a nightly backup? 457 00:24:19,818 --> 00:24:20,743 Right, right. 458 00:24:21,078 --> 00:24:27,528 So I, I guess what you're saying is, or what I'm saying is that when. 459 00:24:27,993 --> 00:24:30,153 Bad things happen and bad things happen. 460 00:24:30,183 --> 00:24:31,623 Admins do wrong things. 461 00:24:31,623 --> 00:24:33,663 Software engineers do wrong things. 462 00:24:34,503 --> 00:24:39,363 When you're trusting a single vendor to do both your IT and your backups 463 00:24:39,363 --> 00:24:44,463 of your it, if they turn out to be incompetent, they're probably also 464 00:24:44,463 --> 00:24:47,163 incompetent in your backup, right? 465 00:24:47,898 --> 00:24:54,828 And so this isn't, because both of us used to work for a cloud backup vendor, right? 466 00:24:54,858 --> 00:24:58,068 This isn't us trying to, you know, get money for our firm employer. 467 00:24:58,578 --> 00:25:02,688 Uh, this is just common sense man. 468 00:25:03,123 --> 00:25:06,963 Prasanna Malaiyandi: Yeah, and I would also say for those companies where 469 00:25:06,963 --> 00:25:12,843 they are focused on both your primary IT application as well as the backup, 470 00:25:13,473 --> 00:25:18,993 typically they're not going to put all the money, all the effort in the backup side 471 00:25:19,353 --> 00:25:24,783 because their main goal is to sell and add new features to the primary application. 472 00:25:25,368 --> 00:25:30,018 And so usually you end up with something that's either half baked, not up 473 00:25:30,018 --> 00:25:34,038 to date, doesn't support all your various scenarios, that someone who is 474 00:25:34,043 --> 00:25:36,648 specialized in backup is thinking about. 475 00:25:37,953 --> 00:25:39,843 W. Curtis Preston: Yeah, all those things there. 476 00:25:39,843 --> 00:25:40,083 Yeah. 477 00:25:40,083 --> 00:25:45,303 There are gonna be lots of important features, both from a security standpoint, 478 00:25:45,303 --> 00:25:52,923 a recovery standpoint, all of that, that are going to be useful to you if 479 00:25:52,923 --> 00:25:54,693 you ever actually need to do a restore. 480 00:25:55,248 --> 00:25:59,148 In the case of any of the events that happened to the people 481 00:25:59,148 --> 00:26:02,238 in our, in the stories that we covered in the last, what was it? 482 00:26:02,238 --> 00:26:03,708 Like eight weeks, something like that. 483 00:26:05,358 --> 00:26:07,248 Uh, what we got next. 484 00:26:08,913 --> 00:26:11,973 Prasanna Malaiyandi: So, I think you did touch on this a bit earlier, 485 00:26:11,973 --> 00:26:19,623 but be careful when someone sells you a feature and tells you it's 486 00:26:19,623 --> 00:26:21,183 backup, but it's not backup. 487 00:26:22,738 --> 00:26:23,028 W. Curtis Preston: Yeah. 488 00:26:24,243 --> 00:26:27,903 Yeah, so this one's interesting because, and I, it sounds like 489 00:26:27,903 --> 00:26:29,523 we pick on Microsoft a lot. 490 00:26:29,583 --> 00:26:31,263 I, it's not Microsoft. 491 00:26:31,293 --> 00:26:32,703 My issue is not Microsoft. 492 00:26:32,703 --> 00:26:37,533 It's the people that seem to, that seem to, like. 493 00:26:37,593 --> 00:26:43,768 In the case of Microsoft 365, you will not see the documentation referring 494 00:26:43,888 --> 00:26:45,843 to retention policies as backup. 495 00:26:46,803 --> 00:26:47,643 You won't see it. 496 00:26:48,363 --> 00:26:51,933 Retention policies are an e-discovery feature. 497 00:26:53,013 --> 00:26:57,453 E discovery and archive have nothing to do with backup, right? 498 00:26:57,573 --> 00:27:00,483 They behave completely differently in terms of the way they save 499 00:27:00,483 --> 00:27:02,463 data, the way they pull out data. 500 00:27:03,483 --> 00:27:10,383 Um, and if you, if you don't understand what I'm saying, you 501 00:27:10,383 --> 00:27:17,433 don't believe what I'm saying, go and try to restore a mailbox. 502 00:27:18,528 --> 00:27:23,598 Using 365 retention policies, please go try that. 503 00:27:24,198 --> 00:27:27,738 Um, the, um, 504 00:27:28,053 --> 00:27:32,223 Prasanna Malaiyandi: tell us if it could turn out to be a plausible point 505 00:27:32,323 --> 00:27:35,303 in time that that mailbox looked like 506 00:27:35,958 --> 00:27:36,888 W. Curtis Preston: doesn't do it. 507 00:27:37,283 --> 00:27:37,573 Prasanna Malaiyandi: Yeah. 508 00:27:37,983 --> 00:27:38,193 W. Curtis Preston: It. 509 00:27:38,193 --> 00:27:42,783 It's all about retention policies and the associated E-discovery feature 510 00:27:43,263 --> 00:27:48,153 is all about how to get all email that ever went into or came out 511 00:27:48,153 --> 00:27:50,103 of that box over a period of time. 512 00:27:50,523 --> 00:27:55,293 It is not designed to restore your mailbox to the way it looked yesterday. 513 00:27:56,063 --> 00:27:56,453 Right. 514 00:27:56,543 --> 00:27:57,983 Doesn't restore folders. 515 00:27:57,983 --> 00:28:01,703 It doesn't restore all kinds of stuff, and it doesn't know how. 516 00:28:01,703 --> 00:28:06,683 It, again, just you, you gotta, maybe you have to experience it to truly 517 00:28:06,688 --> 00:28:11,813 understand this, and that is that it doesn't know how to make your mailbox 518 00:28:11,813 --> 00:28:13,613 look the way it looked yesterday. 519 00:28:13,793 --> 00:28:15,743 It doesn't understand that concept. 520 00:28:15,743 --> 00:28:18,413 That is a backup and restore concept. 521 00:28:19,133 --> 00:28:19,613 Right? 522 00:28:20,063 --> 00:28:21,863 And so that's, that's 523 00:28:21,873 --> 00:28:23,458 Prasanna Malaiyandi: Not an archive and retrieval. 524 00:28:23,613 --> 00:28:25,383 W. Curtis Preston: Yeah, it's about arch archive and retrieval. 525 00:28:25,383 --> 00:28:25,683 Right? 526 00:28:25,683 --> 00:28:28,563 Which, if you don't know what we're talking about, there's a whole 527 00:28:28,563 --> 00:28:31,203 episode that we talk about why arch archive and backup are different. 528 00:28:31,473 --> 00:28:32,823 Go, go check that out. 529 00:28:32,823 --> 00:28:34,533 Uh, put a link in the show notes. 530 00:28:35,133 --> 00:28:42,923 But the, but the other reason with 365 is when we look at that story of KPMG if 531 00:28:42,923 --> 00:28:45,323 you, if you have a backup software, um. 532 00:28:46,208 --> 00:28:48,428 Let's say, you know, pick your favorite backup software. 533 00:28:49,088 --> 00:28:55,838 If you configure it really, really, really, really wrong, like just the 534 00:28:55,838 --> 00:28:58,808 worst backup configuration ever. 535 00:28:58,868 --> 00:29:00,938 You know what it doesn't do, 536 00:29:01,808 --> 00:29:03,053 Prasanna Malaiyandi: Blow away your production. 537 00:29:03,248 --> 00:29:04,928 W. Curtis Preston: it doesn't blow away your production. 538 00:29:08,963 --> 00:29:12,713 Which is another key difference between Microsoft retention 539 00:29:12,713 --> 00:29:18,248 policies and backup, because that's what happened in the KPMG story. 540 00:29:19,343 --> 00:29:22,763 Prasanna Malaiyandi: you almost want a separation of duties, right? 541 00:29:22,823 --> 00:29:25,043 That's the roles and responsibilities. 542 00:29:25,048 --> 00:29:28,973 It's like no different than a cloud provider and the customers who are 543 00:29:28,973 --> 00:29:30,323 on top of the cloud provider, right? 544 00:29:30,323 --> 00:29:34,733 Each person has their own responsibilities and their roles and what they do 545 00:29:34,793 --> 00:29:38,663 and what is provided, and it's all clearly articulated in contracts. 546 00:29:39,278 --> 00:29:41,528 W. Curtis Preston: Yeah, retention policies. 547 00:29:42,158 --> 00:29:46,658 When this person went to go do what they wanted to do, which was just delete one 548 00:29:46,663 --> 00:29:54,398 person's chat, it's like, if I get this right, I will delete one person's chat 549 00:29:54,398 --> 00:29:59,138 and I will have a really, uh, poor. 550 00:29:59,873 --> 00:30:04,403 Facsimile of backup that is really lousy at Restore. 551 00:30:04,553 --> 00:30:06,533 That's the best case scenario. 552 00:30:06,803 --> 00:30:10,613 If I get it wrong, I'm gonna delete the personal chat 553 00:30:10,613 --> 00:30:13,043 history of 145,000 employees. 554 00:30:13,973 --> 00:30:16,193 It could, by the way, it could have been worse. 555 00:30:16,463 --> 00:30:19,073 It could have been worse because. 556 00:30:19,658 --> 00:30:23,528 Retention policies when you're in them, when you, when you first open them. 557 00:30:23,918 --> 00:30:27,578 The, by the way, retention policy, the manual is 25 pages, right? 558 00:30:28,178 --> 00:30:32,048 But you can go in there and you can go for all services, right? 559 00:30:32,053 --> 00:30:32,558 Meaning 560 00:30:32,858 --> 00:30:34,178 Prasanna Malaiyandi: It's not just for chat. 561 00:30:34,238 --> 00:30:34,448 Yeah. 562 00:30:34,928 --> 00:30:36,488 W. Curtis Preston: for all services. 563 00:30:37,178 --> 00:30:41,003 Please set retention to one copy. 564 00:30:42,353 --> 00:30:42,833 Prasanna Malaiyandi: So, 565 00:30:43,178 --> 00:30:43,418 W. Curtis Preston: copy. 566 00:30:43,448 --> 00:30:49,388 Literally in a, in a few mouse clicks, you can blow away your entire 365 environment. 567 00:30:49,763 --> 00:30:52,438 Prasanna Malaiyandi: could you imagine if that had happened? 568 00:30:52,578 --> 00:30:57,443 And KPMG, which is a huge company, loses everything. 569 00:30:57,578 --> 00:30:58,118 W. Curtis Preston: Yeah. 570 00:30:59,678 --> 00:31:01,148 I mean, it's bad enough that what they lost. 571 00:31:01,148 --> 00:31:02,528 People are like, oh, it's just chat. 572 00:31:02,588 --> 00:31:04,448 Well, chat is often quite valuable. 573 00:31:05,138 --> 00:31:05,498 Yeah, 574 00:31:05,948 --> 00:31:08,318 Clearly you've never lived in a company that used chat. 575 00:31:11,858 --> 00:31:12,218 Uh, 576 00:31:12,218 --> 00:31:13,538 Prasanna Malaiyandi: don't use smoke signals. 577 00:31:13,748 --> 00:31:14,048 W. Curtis Preston: what's that? 578 00:31:15,188 --> 00:31:16,418 Prasanna Malaiyandi: They don't use smoke signals 579 00:31:16,778 --> 00:31:19,718 W. Curtis Preston: Yeah, it reminds me, it reminds me there was a company. 580 00:31:20,348 --> 00:31:26,438 It was, I'm pretty sure this was a Xerox commercial years ago. 581 00:31:26,438 --> 00:31:31,448 And it was like, does Xero, is your, is this, describe your company, like 582 00:31:31,688 --> 00:31:36,848 if your copier goes down, do people say it's okay, we'll use carbon paper. 583 00:31:39,368 --> 00:31:40,148 Do you remember that? 584 00:31:40,148 --> 00:31:40,358 That was 585 00:31:40,463 --> 00:31:42,938 a, that was a, that was a long time ago. 586 00:31:43,538 --> 00:31:45,368 Um, so what's our next lesson? 587 00:31:46,658 --> 00:31:50,228 Prasanna Malaiyandi: So going alongside what you talked about with Microsoft 588 00:31:50,233 --> 00:31:55,568 365 and retention policies is, that's like a specific case for Microsoft. 589 00:31:56,018 --> 00:31:58,868 But also cloud providers in general. 590 00:31:59,288 --> 00:32:01,508 How about their designs are just awful. 591 00:32:01,538 --> 00:32:04,448 Like, I think that's actually being too kind to say awful, 592 00:32:04,743 --> 00:32:05,033 W. Curtis Preston: Yeah. 593 00:32:05,978 --> 00:32:06,308 Prasanna Malaiyandi: right? 594 00:32:06,308 --> 00:32:09,638 I think it's just downright pathetic. 595 00:32:10,928 --> 00:32:13,389 Uh, one of the ones that comes to mind is OVH. 596 00:32:14,228 --> 00:32:14,578 W. Curtis Preston: Right, 597 00:32:14,618 --> 00:32:16,778 Prasanna Malaiyandi: know I hate picking on OVH 'cause we always 598 00:32:16,778 --> 00:32:19,148 pick on OVH, but I'm picking on OVH. 599 00:32:19,508 --> 00:32:19,808 Right? 600 00:32:19,808 --> 00:32:25,118 And it's really around the fact that they offered a backup service and 601 00:32:25,123 --> 00:32:31,688 they basically had the backup server in the same data center, like just a 602 00:32:31,693 --> 00:32:34,448 couple rows down in a separate rack. 603 00:32:34,928 --> 00:32:39,398 And so when they had a fire that damaged both production and their backup data. 604 00:32:41,633 --> 00:32:43,643 W. Curtis Preston: Yeah, I mean, I mean, that's bad. 605 00:32:43,703 --> 00:32:49,103 Uh, I, I don't knows some equally bad, similarly bad. 606 00:32:49,463 --> 00:32:55,073 Um, when I think about the fact, you know, if we go back in time, there was a time 607 00:32:55,073 --> 00:32:56,723 when Carbonite was everywhere, right? 608 00:32:56,723 --> 00:32:57,503 There were advertise. 609 00:32:57,653 --> 00:33:00,053 You would think that Carbonite was the single biggest 610 00:33:00,053 --> 00:33:01,553 backup vendor on the planet. 611 00:33:02,543 --> 00:33:07,373 And they were storing customer backup data on raid five arrays. 612 00:33:08,753 --> 00:33:09,203 Right. 613 00:33:09,743 --> 00:33:15,893 I'm gonna say pro prosumer quality raid five arrays with, with no 614 00:33:15,983 --> 00:33:19,493 dual parity, you know, um, nothing. 615 00:33:20,003 --> 00:33:23,783 And then they, you know, and it was really just a matter of time before 616 00:33:23,783 --> 00:33:25,073 what happened to them happened. 617 00:33:25,193 --> 00:33:25,613 Right. 618 00:33:26,243 --> 00:33:28,943 Um, you know, they had a double disc failure and Right. 619 00:33:28,973 --> 00:33:30,083 You know, there was that one. 620 00:33:30,083 --> 00:33:31,553 And then, you know, we have the other one. 621 00:33:32,183 --> 00:33:36,143 Of, um, you know, code spaces where they stored their backup in the same 622 00:33:36,143 --> 00:33:37,313 place that they stored production. 623 00:33:37,313 --> 00:33:39,173 We already talked about that with the 3, 2, 1 rule. 624 00:33:39,413 --> 00:33:42,918 Prasanna Malaiyandi: Yeah, and I think the other one there to also talk about is. 625 00:33:43,868 --> 00:33:47,768 And I don't know if it's really poor design or just not understanding 626 00:33:47,768 --> 00:33:51,488 backup and backup requirements is, I would actually say Rackspace, 627 00:33:52,298 --> 00:33:52,688 right? 628 00:33:52,688 --> 00:33:57,788 They had a hosted exchange offering, which is great, but they didn't 629 00:33:57,793 --> 00:33:59,438 really have a backup solution. 630 00:33:59,498 --> 00:34:00,998 They had never tested it out. 631 00:34:00,998 --> 00:34:05,978 They were sort of trying things on the fly after they ran into issues. 632 00:34:06,803 --> 00:34:09,683 W. Curtis Preston: Yeah, the part of that story that was the worst was 633 00:34:09,688 --> 00:34:12,623 it's like, okay, we've been working on this for a couple weeks now. 634 00:34:12,803 --> 00:34:17,063 We think we have a good plan for what, you know, for how 635 00:34:17,063 --> 00:34:18,383 we're gonna get your data back. 636 00:34:18,923 --> 00:34:23,153 Uh, you know, and then like a little bit more time passes and it's like, 637 00:34:23,213 --> 00:34:25,043 oh, we've now tested the plan. 638 00:34:25,508 --> 00:34:26,498 It works. 639 00:34:26,588 --> 00:34:27,968 Now we need to go do the plan. 640 00:34:28,088 --> 00:34:31,748 Like weeks are going by, so you're like, okay, so what you're telling us 641 00:34:31,748 --> 00:34:35,348 is you're just making this up as you go along, that you never tested this before. 642 00:34:35,708 --> 00:34:37,988 Uh, yeah, not, not good. 643 00:34:39,578 --> 00:34:50,198 Um, the, um, uh, this next one here is that if your cloud vendor is 644 00:34:50,198 --> 00:34:53,828 the low cost leader, it will show. 645 00:34:56,363 --> 00:34:58,373 This was the case of OVH. 646 00:34:58,553 --> 00:35:05,003 They were well acknowledged to be the low cost leader in that space, and it means 647 00:35:05,003 --> 00:35:08,633 that they cut corners in a lot of areas. 648 00:35:08,633 --> 00:35:10,883 They cut corners in terms of firefighting. 649 00:35:11,033 --> 00:35:14,153 They cut corners in terms of not segregating the backup 650 00:35:14,153 --> 00:35:15,323 data, all of that stuff, 651 00:35:15,703 --> 00:35:15,993 Prasanna Malaiyandi: Yeah. 652 00:35:17,723 --> 00:35:23,993 I don't think there's anything wrong with being a low cost offering, as 653 00:35:24,173 --> 00:35:28,643 long as people understand that and the expectations are there, right? 654 00:35:28,643 --> 00:35:30,863 You get what you paid for, right? 655 00:35:30,863 --> 00:35:34,673 Not everyone can afford the Ferrari they may buy, right? 656 00:35:35,123 --> 00:35:37,943 People go buy the Geo Metro, right? 657 00:35:38,663 --> 00:35:39,803 Way back in the day, so. 658 00:35:40,378 --> 00:35:43,228 There are offerings for different people, and that totally makes sense. 659 00:35:43,228 --> 00:35:47,158 But I think what becomes a challenge is when you're not transparent 660 00:35:47,158 --> 00:35:52,408 about what people are getting or you set expectations incorrectly and 661 00:35:52,408 --> 00:35:54,058 misrepresent what they are getting. 662 00:35:55,343 --> 00:35:56,333 W. Curtis Preston: Yeah, agreed. 663 00:35:56,393 --> 00:36:01,853 Um, and there are, and well, and, and again, if you're using a low cost 664 00:36:01,943 --> 00:36:05,513 provider, we go back to rule number one. 665 00:36:06,593 --> 00:36:10,193 Make sure that you're, make sure that you're backing up the data somewhere else. 666 00:36:10,198 --> 00:36:14,483 I mean, you should do it like no matter what your provider is, but definitely if 667 00:36:14,483 --> 00:36:16,313 you're doing a low cost provider, right. 668 00:36:16,763 --> 00:36:19,013 Um, make sure that you, you know, because they're gonna, they're 669 00:36:19,013 --> 00:36:20,543 gonna, they're gonna cut corners. 670 00:36:21,093 --> 00:36:24,183 Prasanna Malaiyandi: So the final one, uh, in our lessons learned is. 671 00:36:24,528 --> 00:36:26,628 SSDs aren't perfect. 672 00:36:27,078 --> 00:36:31,278 And I know Curtis, we've talked, I think we've had multiple episodes about this. 673 00:36:31,278 --> 00:36:36,888 In fact, at home when, uh, I was looking for a new sort of, not really backup 674 00:36:36,888 --> 00:36:42,018 drive, but just extra storage and debating between SSD or just getting spinning 675 00:36:42,018 --> 00:36:45,468 disc, and I know chatting with you, I was like, yeah, we should probably get SS or. 676 00:36:45,843 --> 00:36:50,433 Probably you should get a disc drive because we're not using the data often 677 00:36:50,433 --> 00:36:54,843 and just the issues that we've seen around SSDs, if you're not using it all the time, 678 00:36:55,773 --> 00:36:57,633 the data isn't guaranteed to be there. 679 00:36:58,353 --> 00:36:59,283 And, uh. 680 00:36:59,958 --> 00:37:04,188 This is one of those things where, uh, one of my favorite YouTube channels that 681 00:37:04,188 --> 00:37:09,198 I watch all the time, life Uncontained, they lost a bunch of data on SSDs 682 00:37:09,198 --> 00:37:10,518 and people think they're bulletproof. 683 00:37:10,518 --> 00:37:14,988 Yes, there's less mechanical spinning parts versus traditional hard drives, 684 00:37:14,988 --> 00:37:17,448 but they could still lose data. 685 00:37:18,258 --> 00:37:20,928 W. Curtis Preston: Yeah, I, I think we could say that they, they probably 686 00:37:20,928 --> 00:37:26,598 are more reliable than hard drives, especially if you're moving them around. 687 00:37:26,778 --> 00:37:27,258 Right. 688 00:37:27,708 --> 00:37:31,668 Um, you know, which is the case in your, obviously in your mobile 689 00:37:31,668 --> 00:37:38,538 device or any sort of camera in the field, but they are not infallible 690 00:37:38,658 --> 00:37:40,818 and there are some specific. 691 00:37:42,723 --> 00:37:46,443 Um, ways that they behave that are unique to them. 692 00:37:46,443 --> 00:37:49,563 Some u you know, some unique problems to SSDs. 693 00:37:50,193 --> 00:37:56,433 One of them, you know, you alluded to there, is that it does require, um, you 694 00:37:56,438 --> 00:38:02,373 know, a, a power to ensure that all of the voltages are there and over time. 695 00:38:03,648 --> 00:38:08,688 If you're not using a given, um, SSD drive, those voltages can drop 696 00:38:08,688 --> 00:38:10,788 and you can just lose data there. 697 00:38:10,878 --> 00:38:14,508 There, um, I, I think we're, I think based on, we got a really good 698 00:38:14,508 --> 00:38:17,298 message from one of our listeners, and I want to thank them for that. 699 00:38:18,018 --> 00:38:21,168 Uh, I, I wanna do some more research and talk to somebody 700 00:38:21,173 --> 00:38:22,878 who is a specialist in this area. 701 00:38:22,878 --> 00:38:24,888 It's not my area of expertise. 702 00:38:24,888 --> 00:38:26,028 I don't think it's yours either. 703 00:38:26,778 --> 00:38:29,058 Uh, but, but the short version is. 704 00:38:29,418 --> 00:38:34,188 That while SSDs and, and various other, you know, not just SSDs, but, 705 00:38:34,188 --> 00:38:41,898 but, but solid state devices, uh, of every kind that, while they, I 706 00:38:41,898 --> 00:38:46,938 think they are more reliable than the, uh, a standard hard drive, 707 00:38:46,938 --> 00:38:48,438 especially if you're moving it around. 708 00:38:48,438 --> 00:38:50,178 So they're great for laptops. 709 00:38:50,988 --> 00:38:54,978 Uh, they are not infallible and they still need to be backed up. 710 00:38:55,188 --> 00:38:57,108 Which brings us back to Rule zero. 711 00:38:57,168 --> 00:38:57,978 And what's that 712 00:38:58,923 --> 00:39:00,333 Prasanna Malaiyandi: if it's important. 713 00:39:00,333 --> 00:39:01,053 Back it up. 714 00:39:01,878 --> 00:39:02,748 W. Curtis Preston: exactly? 715 00:39:03,288 --> 00:39:04,818 So the cloud isn't perfect. 716 00:39:05,268 --> 00:39:06,828 SSDs aren't perfect. 717 00:39:08,208 --> 00:39:10,308 Uh, software engineers aren't perfect. 718 00:39:10,848 --> 00:39:12,168 Admins aren't perfect. 719 00:39:12,468 --> 00:39:17,568 All of these are why we back up and, um, 720 00:39:17,943 --> 00:39:20,103 Prasanna Malaiyandi: Don't forget malicious users are out there. 721 00:39:20,358 --> 00:39:22,608 W. Curtis Preston: Of course malicious users, malicious 722 00:39:22,608 --> 00:39:24,138 and, bad actors in general. 723 00:39:24,138 --> 00:39:24,438 Right. 724 00:39:24,443 --> 00:39:27,378 So, you know, like, like, uh, cyber attacks. 725 00:39:27,958 --> 00:39:32,278 Only a few of the incidents that we covered were, were cyber attacks. 726 00:39:32,278 --> 00:39:35,488 The rest were, uh, mistakes. 727 00:39:35,818 --> 00:39:44,278 Um, there was a fire, trying to think what else, but generally it was just mistakes. 728 00:39:44,398 --> 00:39:46,168 This is why we make backups. 729 00:39:46,693 --> 00:39:46,993 Prasanna Malaiyandi: Yeah. 730 00:39:47,893 --> 00:39:51,223 and and I think it's also, and I think the other important thing. 731 00:39:52,483 --> 00:39:55,453 Like these are just cases we found, right? 732 00:39:55,453 --> 00:39:59,833 There are probably hundreds out there that aren't publicized, right? 733 00:39:59,833 --> 00:40:04,693 The company's either gone out of business or no one noticed, right, that 734 00:40:04,693 --> 00:40:06,193 their data was missing or other things. 735 00:40:06,193 --> 00:40:11,203 These are just things that we have been able to find and at least some information 736 00:40:11,203 --> 00:40:16,843 to be able to report out to our listeners on, Hey, here's what we see going on. 737 00:40:17,323 --> 00:40:17,953 W. Curtis Preston: Exactly. 738 00:40:19,483 --> 00:40:20,683 Well, this has been fun. 739 00:40:20,923 --> 00:40:26,323 I hope that our, um, you know, listeners have enjoyed it as much as you and 740 00:40:26,323 --> 00:40:28,123 I have enjoyed making the series. 741 00:40:28,693 --> 00:40:28,903 Prasanna Malaiyandi: Yeah. 742 00:40:28,933 --> 00:40:29,173 Yeah. 743 00:40:29,173 --> 00:40:30,643 No, this has been great, Curtis. 744 00:40:30,643 --> 00:40:31,903 Thank you for suggesting this. 745 00:40:31,958 --> 00:40:34,568 W. Curtis Preston: All right, well, uh, thanks to our listeners 746 00:40:34,658 --> 00:40:38,318 and, uh, be sure to subscribe so that you don't miss an episode. 747 00:40:38,708 --> 00:40:40,118 That is a wrap