1 00:00:00,000 --> 00:00:04,800 This week's cloud disaster story is a shocking one from 2009. 2 00:00:05,190 --> 00:00:07,950 And one that really chapped my hide at the time. 3 00:00:08,430 --> 00:00:13,320 Today, we dive into the turbulent history of Carbonite, a once prominent cloud 4 00:00:13,320 --> 00:00:18,840 backup vendor from posting fake reviews on Amazon to a major data loss incident 5 00:00:19,140 --> 00:00:21,210 and a multimillion dollar lawsuit. 6 00:00:21,510 --> 00:00:25,440 Carbonite's story is a cautionary tale for both customers and 7 00:00:25,440 --> 00:00:27,030 vendors in the backup industry. 8 00:00:27,540 --> 00:00:33,630 Join me and my co-host as we uncovered the details of Carbonite's missteps, including 9 00:00:33,630 --> 00:00:39,000 their use of woefully, inadequate storage arrays, lack of data, redundancy, 10 00:00:39,270 --> 00:00:41,970 and failure to take responsibility. 11 00:00:42,118 --> 00:00:44,490 For a huge customer data loss. 12 00:00:45,060 --> 00:00:49,170 We'll also discuss the lessons we can take away from Carbonite's mistakes. 13 00:00:49,500 --> 00:00:54,810 One big discussion is guidance on what customers should look for when they're 14 00:00:54,810 --> 00:00:57,480 evaluating a cloud backup provider. 15 00:00:57,741 --> 00:01:02,301 You'll learn what happens when you blindly trust vendor marketing. 16 00:01:02,457 --> 00:01:06,327 And you will learn the importance of thorough due diligence when 17 00:01:06,357 --> 00:01:08,307 entrusting your data to a third party. 18 00:01:08,534 --> 00:01:10,214 If this is your first time listening. 19 00:01:10,244 --> 00:01:13,394 Hi, I'm w Curtis Preston, AKA Mr. 20 00:01:13,394 --> 00:01:18,434 Backup, this podcast is my attempt to make sure that what happened to me 30 21 00:01:18,434 --> 00:01:20,774 years ago will never happen to you. 22 00:01:21,104 --> 00:01:25,394 My employer lost their purchasing database and I couldn't restore it. 23 00:01:25,874 --> 00:01:30,404 I dedicated my career to making sure that that wouldn't ever again happen to me. 24 00:01:30,644 --> 00:01:32,294 And now I'm passing it on to you. 25 00:01:32,894 --> 00:01:37,664 We take unappreciated backup admins and turn them into cyber recovery heroes. 26 00:01:37,904 --> 00:01:40,154 This is the backup wrap up. 27 00:01:54,178 --> 00:01:55,198 W. Curtis Preston: Welcome to the show. 28 00:01:55,918 --> 00:02:01,438 Hi, I am w Curtis Preston, and I have with me a fellow possessor 29 00:02:01,498 --> 00:02:05,668 of useless knowledge Prasanna Malaiyandi, how's it going? 30 00:02:05,668 --> 00:02:06,818 Prasanna. 31 00:02:06,838 --> 00:02:08,128 Prasanna Malaiyandi: I am good, Curtis. 32 00:02:08,128 --> 00:02:12,298 Yeah, I was, uh, I, yes, I do tend to have a lot of what I 33 00:02:12,298 --> 00:02:13,978 would consider random knowledge. 34 00:02:13,978 --> 00:02:14,968 Not always useless. 35 00:02:14,968 --> 00:02:16,318 I know sometimes it's helped 36 00:02:16,603 --> 00:02:17,053 W. Curtis Preston: Okay. 37 00:02:17,173 --> 00:02:18,973 You do have useless knowledge. 38 00:02:18,973 --> 00:02:20,923 You do also have useful knowledge. 39 00:02:21,253 --> 00:02:23,083 We, you know, I've kidded you about it in time. 40 00:02:23,083 --> 00:02:26,983 I'll be doing something and I'll be like, you know, working on that timing chain on 41 00:02:26,983 --> 00:02:29,143 my Prius, a car that you've never honed. 42 00:02:29,353 --> 00:02:32,323 And you're like, well, did you do the thing with the thing with the thing? 43 00:02:32,323 --> 00:02:33,613 And I'm like, how do you know this? 44 00:02:33,613 --> 00:02:35,443 And you're like, well, YouTube, you know 45 00:02:35,683 --> 00:02:37,332 Prasanna Malaiyandi: I watch a lot of YouTube XI 46 00:02:37,332 --> 00:02:44,090 W. Curtis Preston: But recently you and this person to which I am married, have 47 00:02:44,090 --> 00:02:47,990 been kidding me a lot about my recent, 48 00:02:49,115 --> 00:02:49,775 Prasanna Malaiyandi: I don't know if it's, 49 00:02:49,880 --> 00:02:51,710 W. Curtis Preston: useless knowledge. 50 00:02:51,755 --> 00:02:53,195 Prasanna Malaiyandi: I don't think it's recent though. 51 00:02:53,195 --> 00:02:57,755 I think you've had this in your head for a while, and I've only 52 00:02:57,760 --> 00:03:01,145 discovered it recently that you have this random knowledge in your head. 53 00:03:02,060 --> 00:03:04,285 W. Curtis Preston: so, so I. 54 00:03:05,450 --> 00:03:07,340 Well, it's relatively recent, by the way. 55 00:03:07,340 --> 00:03:14,270 It, it's within the last year, uh, that I asked myself, what are the different 56 00:03:14,270 --> 00:03:16,130 kind of palm trees in California? 57 00:03:16,370 --> 00:03:21,740 And I found out that, first off, I found out that there's over 3000 58 00:03:21,740 --> 00:03:25,670 palm tree species in the world, and that there's something like 50 to 59 00:03:25,670 --> 00:03:30,830 90 palm trees depending on who you ask that grow, you know, here in 60 00:03:30,830 --> 00:03:31,850 California. 61 00:03:32,600 --> 00:03:32,900 Yeah. 62 00:03:32,930 --> 00:03:33,530 No, no, no, no, 63 00:03:33,770 --> 00:03:34,190 no. 64 00:03:34,835 --> 00:03:36,785 50 to 90 that grow, period. 65 00:03:36,875 --> 00:03:37,145 There's 66 00:03:37,145 --> 00:03:40,535 only one species of palm tree that is native to California. 67 00:03:40,535 --> 00:03:40,955 Prasanna Malaiyandi: look at this. 68 00:03:40,955 --> 00:03:42,695 You're already correcting me, the expert, 69 00:03:42,695 --> 00:03:42,815 the. 70 00:03:43,355 --> 00:03:43,865 W. Curtis Preston: yeah. 71 00:03:44,075 --> 00:03:49,355 And that, but, but with all adding all of that up, if you can correctly 72 00:03:49,355 --> 00:03:54,365 identify roughly five species of palm trees, at least in Southern California 73 00:03:54,365 --> 00:03:59,255 where I live, you can cover 99% of the palm trees that you're going to see. 74 00:03:59,585 --> 00:03:59,945 Right. 75 00:04:00,435 --> 00:04:02,025 And I point this out to people. 76 00:04:03,050 --> 00:04:05,265 I, I, I kind of, I bring it up. 77 00:04:05,265 --> 00:04:05,625 That's the 78 00:04:05,625 --> 00:04:10,515 weird part is I bring it up, I'm like, Hey, uh, look, it's, you know, and, and I 79 00:04:10,515 --> 00:04:14,865 did this with this, this lady that I was talking to, and I was like, oh man, I got 80 00:04:14,865 --> 00:04:17,235 a lot of king palms in this neighborhood. 81 00:04:17,235 --> 00:04:21,405 And she's, and she said, you know, the different kind of palms 82 00:04:21,945 --> 00:04:23,775 Prasanna Malaiyandi: Curtis is like, my life mission is 83 00:04:23,850 --> 00:04:26,460 W. Curtis Preston: like, yes, yes I do. 84 00:04:26,460 --> 00:04:27,900 Would you like to know? 85 00:04:28,170 --> 00:04:31,530 And so I start sharing with, with, with this person, uh, 86 00:04:31,530 --> 00:04:32,390 the different kind of palms. 87 00:04:32,390 --> 00:04:36,480 And then she said, can you tell me the kind of palms that are in my yard? 88 00:04:36,720 --> 00:04:37,830 And I said, yes, I can. 89 00:04:41,085 --> 00:04:44,985 And for the record, she had a combination of queen palms and king palms. 90 00:04:45,195 --> 00:04:47,145 Queen palms are from South America. 91 00:04:47,325 --> 00:04:52,575 King palms are from Australia, and uh, only the Mexican fan 92 00:04:52,575 --> 00:04:54,975 palm is native to California. 93 00:04:54,975 --> 00:04:58,935 And for those of you that have looked at California, been to California, 94 00:04:59,025 --> 00:05:03,075 the Mexican fan palms are the ones that grow like 150 feet tall. 95 00:05:03,145 --> 00:05:03,435 Prasanna Malaiyandi: Yeah. 96 00:05:04,415 --> 00:05:08,465 W. Curtis Preston: Right with the tiny little sprig of green on the top, uh, 97 00:05:08,465 --> 00:05:11,405 they're all, they're also 20 feet tall, but they, they're the ones that grow 98 00:05:11,405 --> 00:05:13,025 to, to the really, really tall ones. 99 00:05:13,445 --> 00:05:15,395 Um, so yeah. 100 00:05:15,755 --> 00:05:18,785 But you got, you and my wife have been making fun of me and 101 00:05:18,785 --> 00:05:21,395 my, my possession of this random 102 00:05:21,395 --> 00:05:21,965 knowledge. 103 00:05:22,215 --> 00:05:25,635 Prasanna Malaiyandi: so are you sure that this woman was not 104 00:05:25,665 --> 00:05:28,395 paid to ask you this question? 105 00:05:29,670 --> 00:05:30,975 W. Curtis Preston: No, she was not big. 106 00:05:31,665 --> 00:05:36,225 I, I am pretty confident that she expressed genuine interest 107 00:05:36,705 --> 00:05:39,375 in the species of palm trees that were in her neighborhood. 108 00:05:39,915 --> 00:05:40,245 Prasanna Malaiyandi: Now? 109 00:05:40,305 --> 00:05:43,995 Now, now are you sure that she expressed genuine interest or now that you're 110 00:05:43,995 --> 00:05:49,335 replaying the story in your head, you think she was expressing genuine interest? 111 00:05:50,955 --> 00:05:55,095 W. Curtis Preston: I am convinced Sir, and I stand by my story. 112 00:05:55,425 --> 00:05:57,285 Prasanna Malaiyandi: Okay, so now that you've finished 113 00:05:57,465 --> 00:05:59,205 palm trees, what comes next? 114 00:05:59,880 --> 00:06:00,840 W. Curtis Preston: I, I don't know. 115 00:06:01,020 --> 00:06:03,120 It, it, it, it, it has to come upon me. 116 00:06:03,120 --> 00:06:05,820 It has to be, I, I need a muse, you know? 117 00:06:06,180 --> 00:06:07,350 I need to be inspired. 118 00:06:07,575 --> 00:06:09,585 Prasanna Malaiyandi: You, you know what inspired, I think last time when you 119 00:06:09,585 --> 00:06:11,805 did this, we used to walk all the time. 120 00:06:12,510 --> 00:06:13,200 W. Curtis Preston: We did. 121 00:06:13,200 --> 00:06:14,250 That is what happened. 122 00:06:14,250 --> 00:06:17,400 We used to walk together separately, walk together 123 00:06:17,400 --> 00:06:20,460 separately, right, uh, on the phone. 124 00:06:20,550 --> 00:06:26,040 And I was realizing in my neighborhood the incredible diversity of palm trees. 125 00:06:26,250 --> 00:06:31,200 Now that I have this knowledge, I need to redo that walk and then see, um, see how 126 00:06:31,200 --> 00:06:33,990 many of them are actually, uh, you know. 127 00:06:34,965 --> 00:06:35,865 The Just five. 128 00:06:35,895 --> 00:06:37,005 Just of the five. 129 00:06:37,935 --> 00:06:39,135 Um, yeah. 130 00:06:39,165 --> 00:06:42,495 So anyway, uh, 131 00:06:42,525 --> 00:06:43,035 speaking 132 00:06:43,065 --> 00:06:43,815 Prasanna Malaiyandi: so, so 133 00:06:43,935 --> 00:06:44,410 W. Curtis Preston: what's that? 134 00:06:44,865 --> 00:06:48,855 Prasanna Malaiyandi: So listeners, if you are walking and you encounter 135 00:06:48,855 --> 00:06:56,175 a palm tree and you're on Twitter slash x, post it there and tag Curtis 136 00:06:56,235 --> 00:06:57,825 and he will respond. 137 00:06:58,125 --> 00:06:58,395 Yep. 138 00:07:00,105 --> 00:07:02,920 W. Curtis Preston: Yeah, I will respond if it's one of the five that. 139 00:07:06,000 --> 00:07:08,670 Um, yeah, it was kind of funny. 140 00:07:09,300 --> 00:07:13,500 Um, what is funny is when I, like if I, if I'm trying to teach somebody 141 00:07:13,740 --> 00:07:18,060 the difference, for example, between a queen palm and a king palm and, 142 00:07:18,120 --> 00:07:20,040 um, they don't get it at first. 143 00:07:20,040 --> 00:07:20,370 And I'm like, 144 00:07:20,370 --> 00:07:21,330 it's so obvious. 145 00:07:21,330 --> 00:07:25,500 Like, look, and at those two trees, the difference between a king and a queen 146 00:07:25,500 --> 00:07:27,690 is so obvious and it takes 'em a minute. 147 00:07:27,690 --> 00:07:29,460 And then, and then they have the aha moment. 148 00:07:29,520 --> 00:07:29,700 And 149 00:07:29,700 --> 00:07:31,230 I'm, and I feel. 150 00:07:32,355 --> 00:07:37,635 It's like, it's like the moment when somebody brings up the 3, 2, 1 rule out 151 00:07:37,635 --> 00:07:38,895 of, out of nowhere. 152 00:07:40,305 --> 00:07:42,165 And I say, yeah. 153 00:07:43,575 --> 00:07:45,315 So speaking of the 3, 2, 1 rule, 154 00:07:45,778 --> 00:07:47,798 so, So here's my question, Prasanna, 155 00:07:47,818 --> 00:07:48,058 we. 156 00:07:49,288 --> 00:07:55,498 Have been talking a lot about cloud outages and cloud disasters and, um, 157 00:07:55,678 --> 00:08:01,078 so far they've all been like a cloud provider that is supposed to do the 158 00:08:01,078 --> 00:08:09,688 thing and the, and in every one of the cases, if the customer had just 159 00:08:09,688 --> 00:08:13,228 had a backup, they would've been fine. 160 00:08:14,758 --> 00:08:15,598 So. 161 00:08:16,663 --> 00:08:17,293 Right. 162 00:08:17,353 --> 00:08:17,803 Right. 163 00:08:18,238 --> 00:08:20,488 Prasanna Malaiyandi: in, but the only thing I wanna, I 164 00:08:20,488 --> 00:08:22,258 was wondering though, is OVH 165 00:08:22,423 --> 00:08:23,803 W. Curtis Preston: A proper backup. 166 00:08:24,238 --> 00:08:24,658 Prasanna Malaiyandi: Yes. 167 00:08:26,543 --> 00:08:28,648 W. Curtis Preston: You know, we, we rely on these backups 168 00:08:28,648 --> 00:08:31,968 a lot and so today's story. 169 00:08:34,293 --> 00:08:40,613 You know, the story that went live yesterday really gets me upset because 170 00:08:40,643 --> 00:08:44,423 in that story, the vendor lied. 171 00:08:44,423 --> 00:08:44,453 I. 172 00:08:44,843 --> 00:08:46,133 To their customers. 173 00:08:46,643 --> 00:08:50,843 They said that, you know, it's proven in court that, that the 174 00:08:50,843 --> 00:08:52,733 vendor lied to their customers. 175 00:08:52,733 --> 00:08:57,113 They told them that the server was, was in a fit, at least one customer. 176 00:08:57,118 --> 00:09:01,343 We know that they specifically said, I want a server in another data center. 177 00:09:01,523 --> 00:09:02,933 And they said, you got it. 178 00:09:03,113 --> 00:09:04,343 And then it turns out that they didn't 179 00:09:04,348 --> 00:09:04,763 have it. 180 00:09:05,063 --> 00:09:07,523 Prasanna Malaiyandi: Because contractual contracts and 181 00:09:07,523 --> 00:09:09,143 operations don't always align 182 00:09:10,103 --> 00:09:11,843 two separate people, two separate org 183 00:09:11,843 --> 00:09:12,203 W. Curtis Preston: Yeah. 184 00:09:13,118 --> 00:09:13,418 Prasanna Malaiyandi: Yeah, 185 00:09:15,113 --> 00:09:18,833 W. Curtis Preston: One of the things that we tell people to do is to, 186 00:09:19,253 --> 00:09:22,223 is I'm a fan of Cloud-based backup. 187 00:09:23,138 --> 00:09:24,328 Prasanna Malaiyandi: I never would've guessed. 188 00:09:27,383 --> 00:09:28,883 W. Curtis Preston: I think it's the killer app for the cloud. 189 00:09:28,883 --> 00:09:29,543 I really do. 190 00:09:29,693 --> 00:09:30,083 Right. 191 00:09:30,863 --> 00:09:36,713 Which is why this story bothers me so much, and that is that we 192 00:09:36,713 --> 00:09:44,333 have a story of a, of a backup vendor that, you know, lost. 193 00:09:44,738 --> 00:09:45,758 Customer data 194 00:09:46,628 --> 00:09:53,378 and it's, it's so frustrating from the story. 195 00:09:53,438 --> 00:09:53,858 Right? 196 00:09:54,063 --> 00:09:54,283 So 197 00:09:54,413 --> 00:09:58,043 Prasanna Malaiyandi: well, well it, I would say that it's not even just lost 198 00:09:58,043 --> 00:10:04,793 customer data, but also the way that they've been operating isn't necessarily 199 00:10:04,793 --> 00:10:09,053 what you would look for in a company that you're trusting your data to. 200 00:10:10,028 --> 00:10:14,648 W. Curtis Preston: Yeah, there are multiple incidents of, um, 201 00:10:14,708 --> 00:10:20,348 I'll call it everything from, you could call it lack of candor. 202 00:10:20,798 --> 00:10:23,828 You could, you know, depending on how you want to characterize 203 00:10:23,828 --> 00:10:25,658 it, some of it is outright. 204 00:10:26,483 --> 00:10:31,853 I don't know if I'm gonna say lie to the degree that OVH did, because that 205 00:10:31,853 --> 00:10:35,993 was, you know, proven in court that they purposefully misrepresented. 206 00:10:36,623 --> 00:10:43,043 Um, in this case we do see them misrepresenting things, but, you know, 207 00:10:43,043 --> 00:10:46,613 do we know for a fact, has it been proven in the court that it was a lie? 208 00:10:46,643 --> 00:10:47,063 No. 209 00:10:47,063 --> 00:10:49,403 But the activities that we see happening. 210 00:10:50,228 --> 00:10:51,578 Weren't good. 211 00:10:52,148 --> 00:10:55,268 Um, and so what are we talking about? 212 00:10:55,268 --> 00:10:55,728 Prasanna? 213 00:10:56,918 --> 00:10:57,548 are we teasing? 214 00:10:57,548 --> 00:10:58,448 What are we teasing? 215 00:10:58,988 --> 00:11:00,668 Prasanna Malaiyandi: So the company, 216 00:11:01,163 --> 00:11:01,448 W. Curtis Preston: a big 217 00:11:01,453 --> 00:11:01,988 name, 218 00:11:02,078 --> 00:11:03,728 A big name, right? 219 00:11:03,728 --> 00:11:09,098 This is a name who, a, you know, say four or five years ago you couldn't go 220 00:11:09,098 --> 00:11:11,018 to an airport without seeing this name. 221 00:11:11,378 --> 00:11:13,118 It was everywhere, 222 00:11:13,358 --> 00:11:13,808 right? 223 00:11:14,558 --> 00:11:15,458 What are we talking about? 224 00:11:16,028 --> 00:11:18,698 Prasanna Malaiyandi: We are talking about the company called Carbonite. 225 00:11:20,108 --> 00:11:20,528 W. Curtis Preston: Yeah. 226 00:11:20,738 --> 00:11:21,008 Prasanna Malaiyandi: Right. 227 00:11:21,088 --> 00:11:25,828 It was a large company that recently was acquired or merged 228 00:11:26,128 --> 00:11:31,168 and is now part of OpenText, uh, which is a Canadian based company. 229 00:11:32,328 --> 00:11:34,273 W. Curtis Preston: Which is gonna figure into the story. 230 00:11:34,558 --> 00:11:35,158 Prasanna Malaiyandi: yes. 231 00:11:35,818 --> 00:11:38,878 So maybe we should roll back things and just go way back 232 00:11:38,878 --> 00:11:41,278 in time and talk about Yeah. 233 00:11:41,308 --> 00:11:41,398 Mm-Hmm. 234 00:11:41,488 --> 00:11:42,868 W. Curtis Preston: let's go way back in time. 235 00:11:43,168 --> 00:11:46,078 So first, um, you know, I'm gonna bring up a story. 236 00:11:46,078 --> 00:11:49,143 that It's not on topic, but it's relevant. 237 00:11:49,443 --> 00:11:49,953 Okay. 238 00:11:50,373 --> 00:11:56,103 A couple of months before the key story that we're gonna talk about happened. 239 00:11:56,628 --> 00:12:05,418 Carbonite got busted by essentially a, a part-time, like internet sleuth, right? 240 00:12:05,418 --> 00:12:06,678 It is just some blogger guy. 241 00:12:07,533 --> 00:12:15,093 Um, and what they got busted, uh, doing was posting fake reviews on Amazon. 242 00:12:15,543 --> 00:12:16,023 Right. 243 00:12:16,113 --> 00:12:21,633 And the, the, this, this sleuth uncovered it, you know, he busted. 244 00:12:21,663 --> 00:12:25,203 I, I remember, you know, I wrote a blog post about it back when this happened. 245 00:12:25,203 --> 00:12:27,543 This was in, by the way, this was 2009. 246 00:12:28,293 --> 00:12:33,633 And, um, this, this person found that, um. 247 00:12:34,248 --> 00:12:38,448 Uh, that basically they had created, uh, a combination of like fake 248 00:12:38,898 --> 00:12:43,308 accounts and, and in some cases just really poorly disguised accounts. 249 00:12:43,848 --> 00:12:50,568 The, and all of the metadata pointed that, um, that these reviews that have been po, 250 00:12:50,598 --> 00:12:56,148 these really positive reviews that have been on, on Amazon were in fact written by 251 00:12:56,178 --> 00:13:01,788 either, um, Carbonite employees or by, um. 252 00:13:03,108 --> 00:13:05,538 Uh, people related to Carbonite 253 00:13:05,538 --> 00:13:06,378 employees, right? 254 00:13:06,768 --> 00:13:12,018 And this guy, this guy uncovered it and, uh, he blogged about it. 255 00:13:12,198 --> 00:13:17,288 And then New York Times covered it, uh, a blog called Pogue' s Post. 256 00:13:17,618 --> 00:13:20,588 And by the way, this, this is one of the stories that's, it's a little 257 00:13:20,588 --> 00:13:24,848 difficult for us to cover because some of the things that we're referring to. 258 00:13:25,988 --> 00:13:29,438 ar Arnold all, they're not left like the original blog that started, 259 00:13:29,438 --> 00:13:31,118 the investigation seems to be gone, 260 00:13:31,418 --> 00:13:31,868 right? 261 00:13:31,958 --> 00:13:35,648 Um, the, I don't know if the entire blog is gone or just the blog post. 262 00:13:35,918 --> 00:13:39,548 Luckily the New York Times article is still around that refers to this. 263 00:13:39,998 --> 00:13:45,398 But yeah, so they were posting, um, and, and they, what I remember was 264 00:13:45,398 --> 00:13:48,458 that they tried to pass it off as nothing and then they, you know, 265 00:13:48,458 --> 00:13:52,688 they just sort of got busted doing this and that, that just, and that. 266 00:13:53,213 --> 00:13:57,773 Story broke just a couple of months before the big story that we're gonna talk about. 267 00:13:57,773 --> 00:13:59,003 Any, any thoughts about that? 268 00:14:00,263 --> 00:14:03,443 Prasanna Malaiyandi: Like I just think back to like 2009. 269 00:14:04,103 --> 00:14:04,613 W. Curtis Preston: Yeah. 270 00:14:04,973 --> 00:14:07,373 Prasanna Malaiyandi: in the days where like people weren't 271 00:14:07,373 --> 00:14:09,173 doing it all that much, right? 272 00:14:09,178 --> 00:14:12,173 Posting fake reviews, and you didn't have the smart 273 00:14:12,173 --> 00:14:14,153 algorithms to try to detect this. 274 00:14:14,153 --> 00:14:19,343 You didn't have all the sort of review farms or paid reviews and all those 275 00:14:19,343 --> 00:14:20,963 things happening like you do today. 276 00:14:21,713 --> 00:14:25,523 And so maybe they just thought, yeah, maybe we could just get away with this, 277 00:14:25,523 --> 00:14:29,783 and like that was probably, they did. 278 00:14:29,813 --> 00:14:32,453 They probably didn't think that there would be a risk of get being caught. 279 00:14:32,663 --> 00:14:32,993 Right. 280 00:14:34,238 --> 00:14:34,658 W. Curtis Preston: Yeah. 281 00:14:34,658 --> 00:14:38,168 One of the people that was posting a, a a review was the vice 282 00:14:38,173 --> 00:14:39,878 president of marketing at Carbonite. 283 00:14:40,778 --> 00:14:41,228 Right. 284 00:14:41,378 --> 00:14:45,893 Um, so, so that's sort of the, the, the 285 00:14:46,868 --> 00:14:48,728 timing under which, this happened. 286 00:14:48,728 --> 00:14:48,878 What's 287 00:14:49,043 --> 00:14:51,923 Prasanna Malaiyandi: by the, by the way, we're not saying it's bad to post reviews. 288 00:14:52,583 --> 00:14:54,233 It's, there's nothing wrong with that, right? 289 00:14:54,263 --> 00:14:57,263 We're just saying it's bad to post reviews, pretending to be other 290 00:14:57,263 --> 00:15:01,163 people trying to give an unbiased or, or give an unbiased opinion. 291 00:15:01,598 --> 00:15:06,368 W. Curtis Preston: Yeah, it is actually, um, you know, you can get in 292 00:15:06,368 --> 00:15:10,658 trouble, like in the us you can get in trouble with the SEC if you do that. 293 00:15:12,668 --> 00:15:12,908 Prasanna Malaiyandi: Later, 294 00:15:13,718 --> 00:15:14,588 W. Curtis Preston: Yeah, yeah. 295 00:15:14,618 --> 00:15:17,708 Like the SECC will resurface in the story. 296 00:15:18,788 --> 00:15:19,898 Okay, so, 297 00:15:20,528 --> 00:15:22,028 so here's what happened. 298 00:15:22,338 --> 00:15:27,438 2009, a couple of months after that story basically. 299 00:15:27,558 --> 00:15:33,648 Um, it's, it's interesting, so I'll tell it not in, this is sort of like Star Wars. 300 00:15:33,653 --> 00:15:36,258 I'm gonna tell it in the order that it happened, not in the order 301 00:15:36,263 --> 00:15:38,688 that we found out about it, right? 302 00:15:38,958 --> 00:15:40,728 Because how we found out about it is. 303 00:15:41,403 --> 00:15:43,023 Beyond my comprehension. 304 00:15:43,083 --> 00:15:51,783 So what happened was that Carbonite had outsourced the backup part 305 00:15:52,083 --> 00:15:58,118 of their backup software to an IT vendor that then purchased. 306 00:15:59,348 --> 00:16:06,338 Disc drives from a company called Promise and $3 million worth, 307 00:16:06,338 --> 00:16:09,728 by the way, $3 million worth of hardware from this vendor. 308 00:16:10,928 --> 00:16:15,068 They then had a, uh, was it a dual disc failure? 309 00:16:15,073 --> 00:16:16,118 Is that that what is that, what 310 00:16:16,118 --> 00:16:17,228 we, Yeah. 311 00:16:17,288 --> 00:16:24,193 They had a dual disc failure in their RAID arrays, and as a result of that, they 312 00:16:24,193 --> 00:16:32,033 lost the backup data of 7,500 customers that doesn't mean that 7,500 customers 313 00:16:32,033 --> 00:16:34,613 lost the data that they cared about. 314 00:16:34,613 --> 00:16:37,433 They lost the most recent backup. 315 00:16:38,213 --> 00:16:40,643 And then what they were able to do is they, they, they 316 00:16:40,763 --> 00:16:42,533 immediately restarted the backup. 317 00:16:43,043 --> 00:16:50,933 And they're saying that what we now know is that 54 customers out of that 7,500. 318 00:16:52,248 --> 00:16:55,218 The data that they were backing up was damaged in some way. 319 00:16:55,218 --> 00:16:56,628 Their PC crashed or they 320 00:16:56,628 --> 00:16:57,618 deleted it or something. 321 00:16:57,618 --> 00:17:02,478 And then they went to go restore data and they were unable to do so because 322 00:17:02,478 --> 00:17:07,518 the data that they wanted to restore was in that, that data that was lost, 323 00:17:07,908 --> 00:17:08,268 Prasanna Malaiyandi: Yeah. 324 00:17:08,568 --> 00:17:09,963 W. Curtis Preston: um, which is. 325 00:17:11,208 --> 00:17:15,318 Formerly working at a backup vendor, that number seemed high to me. 326 00:17:16,008 --> 00:17:19,968 The, the number of customers that would restore 54 out of 7,500. 327 00:17:20,028 --> 00:17:20,298 Right? 328 00:17:20,298 --> 00:17:20,718 That's 329 00:17:21,288 --> 00:17:22,938 0.1%. 330 00:17:23,898 --> 00:17:26,778 Prasanna Malaiyandi: Yes, but right. 331 00:17:26,778 --> 00:17:32,508 That's just customers who encountered another error while before they 332 00:17:32,508 --> 00:17:34,758 could, because who knows how long it took 'em to back up. 333 00:17:34,758 --> 00:17:35,028 Right. 334 00:17:35,058 --> 00:17:38,778 It might have taken them days depending on how much data they had, 'cause they 335 00:17:38,778 --> 00:17:41,238 had to basically back up everything again. 336 00:17:42,618 --> 00:17:45,408 W. Curtis Preston: Yeah, I, I guess I'm just saying that, you know, knowing 337 00:17:45,408 --> 00:17:52,938 what I knew about, about our former employer, people just don't restore stuff. 338 00:17:53,298 --> 00:17:55,158 They just don't do a lot of restores. 339 00:17:55,428 --> 00:17:55,818 I would be 340 00:17:55,818 --> 00:17:56,418 surprised, 341 00:17:56,508 --> 00:17:57,408 Prasanna Malaiyandi: unless they needed to. 342 00:17:57,408 --> 00:17:57,618 though. 343 00:17:57,678 --> 00:17:57,948 Right. 344 00:17:57,953 --> 00:17:58,218 And in 345 00:17:58,218 --> 00:17:59,628 these cases, yeah. 346 00:18:00,228 --> 00:18:00,408 yeah. 347 00:18:00,583 --> 00:18:01,998 So, so, and that's their hope, right? 348 00:18:01,998 --> 00:18:06,408 It's that they just hope that nothing bad will happen while everything 349 00:18:06,408 --> 00:18:07,578 is getting backed up again. 350 00:18:07,638 --> 00:18:09,828 And of course, for those 54 people. 351 00:18:10,803 --> 00:18:11,253 W. Curtis Preston: Yeah, 352 00:18:12,078 --> 00:18:12,408 Prasanna Malaiyandi: Did 353 00:18:12,483 --> 00:18:15,423 W. Curtis Preston: it's sort of like how you hope that another disc drive won't 354 00:18:15,423 --> 00:18:17,583 fail when you're only using raid five. 355 00:18:18,213 --> 00:18:20,818 Um, you hope that that won't happen while you're rebuilding. 356 00:18:21,288 --> 00:18:24,798 Prasanna Malaiyandi: So having worked for storage companies in the past, I am 357 00:18:24,828 --> 00:18:25,118 W. Curtis Preston: Yeah. 358 00:18:26,148 --> 00:18:29,113 Prasanna Malaiyandi: but not surprised that they were not able 359 00:18:29,113 --> 00:18:31,608 to handle a double disc failure. 360 00:18:32,298 --> 00:18:34,518 Um, but. 361 00:18:35,268 --> 00:18:39,048 It is a little crazy that they built a system like that and where it's 362 00:18:39,048 --> 00:18:40,428 like, yeah, this is good enough. 363 00:18:40,818 --> 00:18:41,268 Um 364 00:18:41,628 --> 00:18:42,738 W. Curtis Preston: Let's get, let's get to that. 365 00:18:42,738 --> 00:18:43,668 Let's get to that in a minute. 366 00:18:43,728 --> 00:18:46,188 I wanna ask you a really big question, Prasanna. 367 00:18:46,488 --> 00:18:46,818 Prasanna Malaiyandi: huh. 368 00:18:47,268 --> 00:18:49,308 W. Curtis Preston: How did we find out about this? 369 00:18:49,638 --> 00:18:54,408 Prasanna Malaiyandi: Oh, so it was published in some articles right after 370 00:18:54,438 --> 00:19:00,288 the fact, but the only reason that this was even picked up by the news was because 371 00:19:00,528 --> 00:19:03,108 Carbonite decided to sue the vendors. 372 00:19:03,913 --> 00:19:04,203 W. Curtis Preston: Yeah, 373 00:19:04,443 --> 00:19:08,313 Prasanna Malaiyandi: breach of contract and breach of warranty and 374 00:19:08,378 --> 00:19:10,863 W. Curtis Preston: the IT vendor and the storage vendor. 375 00:19:10,983 --> 00:19:11,433 Prasanna Malaiyandi: Yep. 376 00:19:11,613 --> 00:19:13,473 In order to recoup some of the costs. 377 00:19:13,473 --> 00:19:17,403 And that's where all this information came out as they were seeking damages. 378 00:19:18,183 --> 00:19:18,573 W. Curtis Preston: Yeah. 379 00:19:18,573 --> 00:19:25,923 And I remember being, you know, like backup guy at the time, and 380 00:19:26,553 --> 00:19:28,623 I mean, this is what, 15 years ago 381 00:19:29,523 --> 00:19:29,673 and. 382 00:19:30,693 --> 00:19:36,903 I remember like to the day, it's like 15 years ago and I re my fir. 383 00:19:37,293 --> 00:19:42,213 What's interesting was timing wise, 15 years ago I had my 384 00:19:42,218 --> 00:19:45,423 first Mac that had Thunderbolt. 385 00:19:45,678 --> 00:19:46,278 Prasanna Malaiyandi: Hmm. 386 00:19:46,443 --> 00:19:46,803 W. Curtis Preston: Okay. 387 00:19:47,163 --> 00:19:49,653 Why does, why does that figure into the story? 388 00:19:49,753 --> 00:19:50,503 Here's why. 389 00:19:50,683 --> 00:19:57,523 At the time I was running a company and I needed, I needed a decent sized 390 00:19:57,523 --> 00:20:04,573 storage array for my, um, by the way, the the computer I'm talking 391 00:20:04,573 --> 00:20:05,803 about is literally right over there. 392 00:20:05,803 --> 00:20:07,363 It's literally sitting over there. 393 00:20:07,603 --> 00:20:09,433 It's a, it's a I, um, 394 00:20:09,853 --> 00:20:10,243 what, uh, 395 00:20:10,338 --> 00:20:10,933 Prasanna Malaiyandi: Hi Mac. 396 00:20:11,803 --> 00:20:14,163 W. Curtis Preston: The, yeah, the I, the new imacs, 397 00:20:14,263 --> 00:20:14,743 right? 398 00:20:15,463 --> 00:20:19,513 And it had Thunderbolt and we were doing video editing on it, and we 399 00:20:19,513 --> 00:20:21,823 needed a nice sized array to buy it. 400 00:20:22,663 --> 00:20:25,453 And the only arrays that you could buy at the time that were 401 00:20:25,453 --> 00:20:28,033 thunderbolt were promise arrays. 402 00:20:28,963 --> 00:20:34,753 And when you assert at the time, the promise arrays, they, they 403 00:20:34,758 --> 00:20:35,863 were, I mean, they were great. 404 00:20:35,863 --> 00:20:36,823 I, you know, I had it. 405 00:20:36,823 --> 00:20:40,753 I actually still have the promise array somewhere, and. 406 00:20:41,578 --> 00:20:45,718 But the thing was that they were also known for being the cheapest 407 00:20:45,838 --> 00:20:47,338 arrays that you could buy at the 408 00:20:47,338 --> 00:20:47,878 time. 409 00:20:48,688 --> 00:20:51,718 So when this story broke, I. 410 00:20:52,258 --> 00:20:54,463 I, I I sort of realized a bunch of things. 411 00:20:54,463 --> 00:21:01,303 I'm like, wait, this big cloud backup company is using the cheapest storage 412 00:21:01,303 --> 00:21:05,173 arrays that, and they're just using like regular old storage arrays. 413 00:21:05,178 --> 00:21:06,883 Not like good ones. 414 00:21:06,888 --> 00:21:10,063 They're not, you know, they're like, they're supposed to be the cloud. 415 00:21:10,068 --> 00:21:13,543 They're supposed to be, they're supposed to be using advanced technology and, 416 00:21:13,878 --> 00:21:15,493 and, you know, so, so there was that. 417 00:21:15,493 --> 00:21:16,963 The other thought I had was. 418 00:21:17,473 --> 00:21:20,473 Wait, they just have like one comp, they just have one 419 00:21:20,473 --> 00:21:22,303 copy of each customer's data. 420 00:21:22,573 --> 00:21:28,213 And it's not even using RAID dp, it's not using mirroring, it's not using RAID dp. 421 00:21:28,213 --> 00:21:31,063 So, so all it takes is one double disc failure 422 00:21:31,603 --> 00:21:34,423 And and you're done. 423 00:21:35,263 --> 00:21:39,463 And, and then the other thing was I, I said to myself. 424 00:21:40,738 --> 00:21:44,578 Why, if this was me first off, I wouldn't have, I, I don't think 425 00:21:44,578 --> 00:21:45,598 I would've made that decision. 426 00:21:45,598 --> 00:21:45,958 I think 427 00:21:45,958 --> 00:21:47,548 I would've done something very differently. 428 00:21:48,298 --> 00:21:55,408 Second, if I had made that decision, it's the, I'm gonna publicly 429 00:21:55,408 --> 00:22:03,298 announce to the world that I made that decision and like, what, what 430 00:22:03,358 --> 00:22:05,368 was going through anybody's mind? 431 00:22:06,633 --> 00:22:06,853 Prasanna Malaiyandi: It 432 00:22:06,858 --> 00:22:08,228 seems like a big PR 433 00:22:08,228 --> 00:22:08,618 snafu. 434 00:22:09,788 --> 00:22:13,268 W. Curtis Preston: big, PR snafu, a bad business decision to buy 435 00:22:13,268 --> 00:22:14,588 the arrays in the first place. 436 00:22:14,588 --> 00:22:17,138 I'm not saying that promise arrays are bad. 437 00:22:17,618 --> 00:22:23,888 I will say that having that much data without having RAID DP bad, 438 00:22:24,578 --> 00:22:29,408 right, that much data without having mirroring or something like 439 00:22:29,408 --> 00:22:32,438 that, that where a double disc failure just takes your data out. 440 00:22:33,008 --> 00:22:35,108 Double disc failures happen all the time. 441 00:22:36,128 --> 00:22:37,178 You, you, you used 442 00:22:37,183 --> 00:22:39,068 to work at a company that did this? 443 00:22:39,263 --> 00:22:39,773 Prasanna Malaiyandi: That's why. 444 00:22:39,773 --> 00:22:44,213 So there is a quote, and I don't know if we, yeah, and I will cover it now. 445 00:22:44,603 --> 00:22:48,713 Uh, where basically, uh, the CEO David Friend 446 00:22:49,778 --> 00:22:50,378 W. Curtis Preston: at the time. 447 00:22:50,378 --> 00:22:51,728 he's no longer the CEO. 448 00:22:51,728 --> 00:22:52,238 He was the Yeah. 449 00:22:52,403 --> 00:22:52,673 Prasanna Malaiyandi: Yeah. 450 00:22:52,703 --> 00:22:57,563 So he had explained sort of, okay, this is why they had crashed because we were using 451 00:22:57,563 --> 00:23:01,163 Promise Technology and we're using RAID, and we suffered as single disc failure. 452 00:23:01,583 --> 00:23:05,303 So he said that we switched to a popular Dell server that uses RAID six. 453 00:23:05,978 --> 00:23:09,008 That allows a loss of three of the 15 drives simultaneously 454 00:23:09,008 --> 00:23:10,418 before you lose any data. 455 00:23:10,868 --> 00:23:14,438 The configuration is, in theory, 36 million times more reliable 456 00:23:14,438 --> 00:23:15,608 than a single disk drive. 457 00:23:15,998 --> 00:23:20,468 The chances of three out of 15 drives failing at the same time are almost dill. 458 00:23:21,038 --> 00:23:26,288 Now, having worked at multiple storage companies, I want to call, 459 00:23:26,348 --> 00:23:30,968 uh, BS on that, nearly nil, right? 460 00:23:31,223 --> 00:23:33,413 W. Curtis Preston: On the three drives, failing at the same time. 461 00:23:33,608 --> 00:23:36,698 Prasanna Malaiyandi: Yeah, having worked at companies, especially given 462 00:23:36,698 --> 00:23:42,698 the size of discs these days, what I've experienced is that during the 463 00:23:42,698 --> 00:23:45,638 times when you're actually doing a raid rebuild, that's actually when you're 464 00:23:45,638 --> 00:23:47,798 putting additional stress on all the 465 00:23:47,798 --> 00:23:48,578 other discs. 466 00:23:48,848 --> 00:23:52,118 As you're doing a whole bunch of reads, recomputing parody, in order to write 467 00:23:52,118 --> 00:23:55,418 out and recover the new disc, that's actually the time in which those 468 00:23:55,423 --> 00:23:58,868 other discs, if there's any weakness in them, that's when they fail. 469 00:24:00,038 --> 00:24:02,198 W. Curtis Preston: I feel a story coming on Prasanna. 470 00:24:02,558 --> 00:24:03,398 Prasanna Malaiyandi: What's the story? 471 00:24:04,658 --> 00:24:08,648 W. Curtis Preston: The story is my oracle, the, so this is the 472 00:24:08,648 --> 00:24:11,048 first time, so I, I, I, I left. 473 00:24:11,108 --> 00:24:12,518 I was no longer like an it. 474 00:24:13,243 --> 00:24:13,363 I. 475 00:24:13,363 --> 00:24:14,963 Regular IT practitioner. 476 00:24:14,963 --> 00:24:20,693 I was now a consultant and I was at this major oil and gas company and 477 00:24:20,723 --> 00:24:22,493 I was just supposed to be the admin. 478 00:24:22,523 --> 00:24:25,283 I wasn't supposed to be the backup guy, but I went around and I've seen 479 00:24:25,283 --> 00:24:26,693 that the backups were just broken. 480 00:24:26,693 --> 00:24:30,383 And, you know, and I, and, and one of the servers that really bothered 481 00:24:30,383 --> 00:24:32,213 me was, it was, it was a Solaris. 482 00:24:32,963 --> 00:24:36,083 I remember it was running Solaris two three, which at the time was a 483 00:24:36,083 --> 00:24:38,423 seriously buggy version of Solaris. 484 00:24:39,263 --> 00:24:46,433 And, um, I was trying to get the first ever backup of this Oracle database, 485 00:24:46,433 --> 00:24:50,033 which was a three 300 gigabyte, 486 00:24:50,963 --> 00:24:51,408 three oh 487 00:24:51,683 --> 00:24:53,843 gigabyte Oracle database. 488 00:24:54,233 --> 00:24:56,303 Massive, huge, right? 489 00:24:56,873 --> 00:24:58,133 And, um. 490 00:24:59,278 --> 00:25:02,213 It was what was called a DSS decision support system. 491 00:25:02,213 --> 00:25:05,513 It was all the rage back in the day and it was the kind of thing that took 492 00:25:05,513 --> 00:25:07,823 imports nightly from the mainframe. 493 00:25:08,753 --> 00:25:12,623 And, um, it had never been backed up, ever. 494 00:25:13,403 --> 00:25:17,993 And I had to argue with the DBA that um, he didn't want me to do. 495 00:25:18,083 --> 00:25:21,743 He didn't want, he didn't want me turning on archive log mode because AR archive 496 00:25:21,743 --> 00:25:24,473 log mode corrupted Oracle databases. 497 00:25:25,073 --> 00:25:26,243 That was his 498 00:25:26,903 --> 00:25:27,383 Prasanna Malaiyandi: stance. 499 00:25:28,073 --> 00:25:28,463 Yeah. 500 00:25:28,568 --> 00:25:31,268 W. Curtis Preston: I'm like, is that the stance of Oracle or is that just 501 00:25:31,268 --> 00:25:32,798 some random thing that's in your head? 502 00:25:32,798 --> 00:25:33,668 And it was the latter. 503 00:25:34,238 --> 00:25:40,028 Um, and then I, um, and he, uh, he said, well, if it doesn't corrupt it, it 504 00:25:40,028 --> 00:25:44,168 cau, it causes performance, uh, lower, you know, it lowers the performance. 505 00:25:45,413 --> 00:25:48,023 And, um, I was like, well look, we've gotta get this 506 00:25:48,023 --> 00:25:49,793 thing backed up at least once. 507 00:25:49,913 --> 00:25:53,183 And so either I shut down the database and back it up one night, 508 00:25:53,393 --> 00:25:55,913 or, which they didn't want to do 'cause that's when they did the night, 509 00:25:56,183 --> 00:26:00,323 the nightly uploads, or I need to put it in archive log mode 510 00:26:00,323 --> 00:26:02,633 at least for a couple of days. 511 00:26:03,458 --> 00:26:05,258 So I can do a hot backup of it. 512 00:26:05,888 --> 00:26:10,238 And um, and then I went to go, I, I went and that's when I looked at 513 00:26:10,238 --> 00:26:13,248 it and I saw that not only was it Solaris two three, but it was Solaris 514 00:26:13,268 --> 00:26:14,918 two three, completely unpatched, 515 00:26:15,548 --> 00:26:15,818 right. 516 00:26:15,818 --> 00:26:18,878 It, it, it had been like a couple of years 517 00:26:18,908 --> 00:26:19,538 Prasanna Malaiyandi: one wanted touch 518 00:26:19,598 --> 00:26:24,968 W. Curtis Preston: Patches and, and so I, I knew because I had already 519 00:26:24,968 --> 00:26:28,898 encountered it at this client, that if I put the backup software that I was using. 520 00:26:29,978 --> 00:26:35,318 The features of Solaris that it used during a backup would cause it to crash. 521 00:26:35,378 --> 00:26:39,608 If I kicked it off, I knew I had to put on the latest jumbo patch of 522 00:26:39,608 --> 00:26:42,068 Solaris in order to do the backup. 523 00:26:43,328 --> 00:26:44,888 And so, uh, I did. 524 00:26:45,188 --> 00:26:47,108 And uh, I got it rebooted. 525 00:26:47,168 --> 00:26:53,588 And this was like a Friday afternoon and I got my first ever backup 526 00:26:53,588 --> 00:26:55,268 of this 300 gigabyte database. 527 00:26:55,448 --> 00:26:58,778 And then over the weekend, five drives failed. 528 00:27:00,548 --> 00:27:01,028 Right. 529 00:27:02,008 --> 00:27:04,138 Prasanna Malaiyandi: Should never have rebooted the server, Curtis. 530 00:27:04,238 --> 00:27:04,418 W. Curtis Preston: yeah. 531 00:27:04,628 --> 00:27:05,018 Yeah. 532 00:27:05,288 --> 00:27:10,748 So, um, so I came in Monday morning and, and basically the, the drives failed. 533 00:27:10,748 --> 00:27:15,518 I got, and by the way, it was the only an additional wrinkle was this was a place 534 00:27:15,518 --> 00:27:17,978 who didn't buy Sun Service from Sun. 535 00:27:18,008 --> 00:27:19,023 They bought like pieces 536 00:27:19,328 --> 00:27:19,418 and 537 00:27:19,418 --> 00:27:20,018 parts 538 00:27:21,083 --> 00:27:21,233 Prasanna Malaiyandi: and. 539 00:27:21,238 --> 00:27:22,668 put it together themselves, and then, 540 00:27:23,153 --> 00:27:23,693 W. Curtis Preston: Yeah, 541 00:27:23,753 --> 00:27:24,023 yeah. 542 00:27:24,023 --> 00:27:28,163 But this server was from Sun and so I was able to call sun, get sun in 543 00:27:28,163 --> 00:27:32,573 over the weekend, replace the drives, you know, rebuild the file system, 544 00:27:32,603 --> 00:27:36,803 restore the database, and it was up and running come Monday morning, 545 00:27:37,043 --> 00:27:39,083 and I was cock of the walk, baby. 546 00:27:39,443 --> 00:27:39,803 Right? 547 00:27:40,823 --> 00:27:45,413 And then this, this, um, this, uh, this DBA comes up to me 548 00:27:45,473 --> 00:27:46,673 and he sees me and he's like. 549 00:27:47,468 --> 00:27:49,658 I bet you think you're hot shit, don't you? 550 00:27:50,678 --> 00:27:52,598 And I'm like, yeah, yeah, I kind of do. 551 00:27:52,778 --> 00:27:54,878 You know, I'm feeling pretty good about myself right 552 00:27:54,878 --> 00:27:55,178 now. 553 00:27:55,928 --> 00:27:58,808 And he said, well, I'll tell you what, you don't get any credit. 554 00:27:59,588 --> 00:28:00,338 I was like, why is that? 555 00:28:00,343 --> 00:28:05,598 He goes, well, he goes, I'm not exactly sure what killed the server or those disk 556 00:28:05,618 --> 00:28:07,418 drives, but I think it was your fault. 557 00:28:08,918 --> 00:28:10,418 He is like, I have a couple theories. 558 00:28:10,418 --> 00:28:14,558 One is you did this backup and it exercised the disc drives more 559 00:28:14,558 --> 00:28:15,998 than they were used to being used. 560 00:28:15,998 --> 00:28:18,338 And so that's what caused the thing, uh, to fail. 561 00:28:18,758 --> 00:28:22,208 Or it was, uh, the fact that Oracle was in, uh, arch archive 562 00:28:22,208 --> 00:28:26,648 log mode and that's maybe what caused it or the fact that you put in that 563 00:28:26,648 --> 00:28:30,698 big jumbo patch, um, and one of, one of those things killed the disc drives. 564 00:28:30,698 --> 00:28:35,258 And so since you killed the disc drives you get no credit for, for saving us. 565 00:28:35,618 --> 00:28:37,958 I was like, okay, whatever dude. 566 00:28:40,643 --> 00:28:41,063 So 567 00:28:41,123 --> 00:28:44,723 yes, multiple disc drives in this case, five disc 568 00:28:44,723 --> 00:28:47,453 drives can fail all at once. 569 00:28:47,903 --> 00:28:49,193 This is why we back up. 570 00:28:49,193 --> 00:28:50,663 This is why we do mirroring. 571 00:28:50,663 --> 00:28:54,263 This is why we do all kinds of, I, I guess it just really disappointed 572 00:28:54,263 --> 00:28:59,213 me at the time, and so it, it was really surprising that a company who 573 00:28:59,213 --> 00:29:00,898 this is what, this is what they do. 574 00:29:02,078 --> 00:29:06,218 They, they were just using the same RAID arrays, you know, 575 00:29:06,278 --> 00:29:07,628 that I'm putting on the back, 576 00:29:07,868 --> 00:29:09,338 on the back of my iMac. 577 00:29:10,178 --> 00:29:11,378 I was very disappointed. 578 00:29:11,588 --> 00:29:14,828 Um, I, I, I spoke a lot there, 579 00:29:14,918 --> 00:29:17,138 but it just bugged me when it happened. 580 00:29:17,138 --> 00:29:17,258 What 581 00:29:17,288 --> 00:29:17,558 Prasanna Malaiyandi: yeah. 582 00:29:17,918 --> 00:29:19,478 No, I, I agree with you. 583 00:29:19,478 --> 00:29:26,438 It's, and the challenge is their marketing messages and everything else at the time. 584 00:29:27,218 --> 00:29:27,578 Right. 585 00:29:27,668 --> 00:29:30,008 Portrayed it as, uh, high quality. 586 00:29:31,313 --> 00:29:31,433 I. 587 00:29:31,628 --> 00:29:34,058 Backup solution for customers. 588 00:29:34,058 --> 00:29:37,118 So they don't need to worry about backup and everything else that 589 00:29:38,228 --> 00:29:39,668 that comes along with it, right? 590 00:29:39,668 --> 00:29:41,618 They're like, yeah, we will deal with everything for you. 591 00:29:41,618 --> 00:29:45,488 And when customers put their trust and their faith in that, uh, the 592 00:29:45,493 --> 00:29:47,198 company didn't quite meet the bar. 593 00:29:48,218 --> 00:29:48,818 W. Curtis Preston: Yeah. 594 00:29:48,908 --> 00:29:50,588 Yeah, and, and I guess it's just. 595 00:29:50,948 --> 00:29:55,748 Even to this day in the subsequent articles that came out after the original 596 00:29:55,748 --> 00:30:01,268 article, when the CEO David friend was going to the people say, Hey, you know, 597 00:30:01,268 --> 00:30:02,978 I just wanna correct the, this, you 598 00:30:02,978 --> 00:30:03,833 Prasanna Malaiyandi: It's not 7,500. 599 00:30:04,143 --> 00:30:04,433 Yeah. 600 00:30:04,808 --> 00:30:07,418 W. Curtis Preston: it wasn't 7,500 customers that lost data. 601 00:30:07,418 --> 00:30:11,528 It was 54 customers lost data because only 54 customers couldn't restore. 602 00:30:12,158 --> 00:30:12,428 Right? 603 00:30:12,453 --> 00:30:16,868 But still in those further comments, he was still like blaming, promise. 604 00:30:17,813 --> 00:30:21,113 Like he even, you know, um, and by the way, I do wanna say I was 605 00:30:21,113 --> 00:30:26,283 able to find the, the, docket, uh, the actual filing and , the 606 00:30:26,288 --> 00:30:28,373 lawsuit was settled outta court, 607 00:30:28,493 --> 00:30:28,823 right? 608 00:30:28,823 --> 00:30:32,843 So there's no, there's no public record of what the actual thing is, 609 00:30:32,843 --> 00:30:37,973 but I highly doubt that, um, you know, they're a storage array vendor 610 00:30:37,973 --> 00:30:41,483 disk drives fail and sometimes monitoring stuff fails and. 611 00:30:42,803 --> 00:30:48,743 You know, I, I, I just, I could not, for the life of me fathom the idea of, of 612 00:30:49,403 --> 00:30:51,623 suing your, your storage array vendor. 613 00:30:51,623 --> 00:30:54,563 It's a little bit like when Musey sued 614 00:30:55,073 --> 00:30:55,823 Google. 615 00:30:56,093 --> 00:30:57,773 because they deleted their own data. 616 00:30:57,948 --> 00:30:58,298 Right. 617 00:30:59,453 --> 00:31:00,738 Um, yeah. 618 00:31:00,863 --> 00:31:02,453 I, I just don't understand that. 619 00:31:02,503 --> 00:31:07,028 So there was an article from Backblaze, uh, which for the record 620 00:31:07,388 --> 00:31:11,053 is a competitor to Carbonite, but I still agree with their point. 621 00:31:11,858 --> 00:31:15,008 And they were saying that they felt that the coverage of this. 622 00:31:15,863 --> 00:31:17,603 Incident missed. 623 00:31:17,753 --> 00:31:21,983 One of the crucial points of this story, and that is that a backup vendor 624 00:31:21,983 --> 00:31:26,213 outsourced the backup and, and they made a, you know, that they, that 625 00:31:26,213 --> 00:31:29,933 they made a point of saying that, that it's like Google Outsourcing search. 626 00:31:30,908 --> 00:31:31,388 Right. 627 00:31:31,643 --> 00:31:33,848 And, and I, and I have to agree with 'em, right? 628 00:31:33,848 --> 00:31:35,978 They, they weren't saying they were reselling backup. 629 00:31:35,978 --> 00:31:39,548 They were, they were saying they were a backup vendor, and you would 630 00:31:39,548 --> 00:31:42,218 think that they would deal with that. 631 00:31:42,248 --> 00:31:42,638 Right? 632 00:31:42,698 --> 00:31:47,348 Um, and, uh, at the time, the, the response to that was that 633 00:31:47,348 --> 00:31:51,428 the CEO said that they now, now, write their own software. 634 00:31:51,998 --> 00:31:52,418 Right. 635 00:31:53,948 --> 00:31:56,078 So, I don't know. 636 00:31:58,568 --> 00:32:03,458 The, um, yeah, in, in that blog it says Building robust online 637 00:32:03,458 --> 00:32:05,018 backup technology is difficult. 638 00:32:05,288 --> 00:32:08,288 There's certainly lots of complexities involved to ensure data is 639 00:32:08,288 --> 00:32:09,908 backed up, redundant and secure. 640 00:32:10,178 --> 00:32:13,058 It is the role of the online backup service provider to have 641 00:32:13,058 --> 00:32:17,198 the technical expertise and laser focused to work through these items. 642 00:32:17,738 --> 00:32:22,778 Pushing it off to an outside company seems a bit risky. 643 00:32:23,003 --> 00:32:23,293 Prasanna Malaiyandi: Yeah. 644 00:32:23,678 --> 00:32:27,848 Well, and the fact that we started with this right as the 3, 2, 1 rule. 645 00:32:28,478 --> 00:32:31,808 They were not doing the 3, 2, 1 rule at all, you know? 646 00:32:32,303 --> 00:32:35,003 W. Curtis Preston: yeah, yeah, I mean, you know, there are, there 647 00:32:35,003 --> 00:32:39,713 are probably a bunch of vendors out there that, for example, are relying 648 00:32:39,743 --> 00:32:45,473 on, you know, backup vendors that are storing only one copy on S3, 649 00:32:45,548 --> 00:32:45,898 right? 650 00:32:46,448 --> 00:32:46,738 Yeah. 651 00:32:47,408 --> 00:32:49,388 Now as three stores, three copies. 652 00:32:49,388 --> 00:32:54,338 But you still are, you still are relying, but in that case, you're relying on what 653 00:32:54,338 --> 00:33:00,518 arguably is like the most tested storage array service in the world, right? 654 00:33:00,563 --> 00:33:01,763 Prasanna Malaiyandi: Yeah, exactly. 655 00:33:01,928 --> 00:33:02,438 W. Curtis Preston: Yeah. 656 00:33:02,588 --> 00:33:06,578 Uh, but as opposed to promise, which was a little known 657 00:33:06,578 --> 00:33:08,858 product that, that I'm willing to bet. 658 00:33:08,858 --> 00:33:10,928 The reason they bought 'em is they were the least expensive. 659 00:33:11,168 --> 00:33:11,378 That's 660 00:33:11,378 --> 00:33:11,918 why I bought 'em. 661 00:33:12,138 --> 00:33:12,428 Prasanna Malaiyandi: Yeah. 662 00:33:12,968 --> 00:33:15,038 W. Curtis Preston: That and the fact that they did Thunderbolt at the time. 663 00:33:16,388 --> 00:33:18,788 It was really hard to get Thunderbolt based storage array 664 00:33:19,013 --> 00:33:22,433 Prasanna Malaiyandi: I think at the time it was probably OWC and Promise. 665 00:33:23,138 --> 00:33:23,588 W. Curtis Preston: Yeah. 666 00:33:23,858 --> 00:33:24,218 Yeah. 667 00:33:24,223 --> 00:33:26,078 And OWC was probably more expensive, so I 668 00:33:26,078 --> 00:33:26,588 didn't buy it. 669 00:33:27,608 --> 00:33:32,948 Um, so what happened after that story Prasanna, which I didn't even know 670 00:33:32,948 --> 00:33:36,458 about until we were researching this, this story until you brought it up. 671 00:33:36,908 --> 00:33:39,218 Prasanna Malaiyandi: So apparently I came across while looking 672 00:33:39,218 --> 00:33:40,718 for lawsuits against Carbonite. 673 00:33:41,048 --> 00:33:46,628 Uh, it looks like they just settled one, um, in January, uh, 31st of 2024. 674 00:33:47,048 --> 00:33:50,918 So a couple of months ago from this recording, uh, for 27 and a 675 00:33:50,918 --> 00:33:54,068 half million dollars apparently. 676 00:33:54,098 --> 00:33:59,678 So Carbonite was acquired in 2019 by a Canadian, Canadian 677 00:33:59,678 --> 00:34:01,268 company called OpenText, 678 00:34:02,133 --> 00:34:03,233 W. Curtis Preston: So five years ago 679 00:34:03,503 --> 00:34:04,598 they were acquired. 680 00:34:04,658 --> 00:34:05,018 Prasanna Malaiyandi: Yep. 681 00:34:05,348 --> 00:34:10,598 But before they were acquired, Carbonite apparently had made some statements that 682 00:34:10,598 --> 00:34:16,088 misled investors and they were releasing a new backup solution, uh, called 683 00:34:16,088 --> 00:34:22,808 Server VM edition that they promised was super strong and extremely competitive. 684 00:34:23,558 --> 00:34:27,938 That launched in 2018 October, and the company said the product would 685 00:34:27,938 --> 00:34:32,978 allow businesses to recover virtual machines data from a single location. 686 00:34:34,058 --> 00:34:37,658 And then they got sued by investors because they said the product, it never 687 00:34:37,658 --> 00:34:42,968 worked and that it never once successfully backed up a customer's data and that 688 00:34:42,973 --> 00:34:47,828 the executives basically knew it did not work and still made those statements. 689 00:34:48,608 --> 00:34:51,908 And so this was just settled for 27 and a half million 690 00:34:51,908 --> 00:34:54,038 dollars to resolve the claims. 691 00:34:55,568 --> 00:34:56,258 W. Curtis Preston: Yeah, it's interesting. 692 00:34:56,258 --> 00:35:00,248 So they accused the company of violating the Securities Exchange Act of 1934. 693 00:35:01,178 --> 00:35:06,128 After this news came out and the stock price fell more than 24% the next day, 694 00:35:06,578 --> 00:35:10,658 uh, the judge initially dismissed the case, but the court of appeals reversed 695 00:35:10,658 --> 00:35:17,138 'em in 2021 and, um, and certified the, the class as a class action. 696 00:35:17,768 --> 00:35:19,988 And then it looks like they finally settled it, 697 00:35:20,858 --> 00:35:24,158 um, and agreed to pay $27 million. 698 00:35:26,018 --> 00:35:27,038 That's a lot of money. 699 00:35:28,178 --> 00:35:28,598 Prasanna Malaiyandi: Yeah. 700 00:35:29,603 --> 00:35:31,388 W. Curtis Preston: I wonder how many promise arrays that'll buy. 701 00:35:32,408 --> 00:35:35,438 Sorry, that was, that was, that was just mean. 702 00:35:35,918 --> 00:35:44,048 Um, I, yeah, I don't, I, you know what I, what I used to, what I used to say 703 00:35:44,048 --> 00:35:47,888 about Carbonite was, you know, just Google, like if you're, when you're 704 00:35:47,888 --> 00:35:53,198 looking, when you're looking at vendors, just Google their name and lawsuit. 705 00:35:53,228 --> 00:35:57,548 'cause at the time, Carbonite was the only one that that would come up with. 706 00:35:58,388 --> 00:35:59,258 With results. 707 00:35:59,708 --> 00:36:04,608 What's interesting here, like, it's sort of like with with OVH, 708 00:36:05,298 --> 00:36:07,578 why in the world would anyone use. 709 00:36:08,328 --> 00:36:10,308 The, a vendor, why would they go public? 710 00:36:10,308 --> 00:36:11,628 Why would somebody invest? 711 00:36:11,958 --> 00:36:16,128 And it's because these stories, they just go under the, under the 712 00:36:16,128 --> 00:36:16,848 radar. 713 00:36:17,188 --> 00:36:23,858 I think a, I think a, a, you know, an IAS vendor like OVH, who 714 00:36:23,858 --> 00:36:27,998 showed that they don't know how to build data centers that they 715 00:36:27,998 --> 00:36:28,058 Prasanna Malaiyandi: they 716 00:36:28,058 --> 00:36:28,868 went IPO. 717 00:36:29,798 --> 00:36:32,078 W. Curtis Preston: Didn't, yeah, they didn't do fire suppression. 718 00:36:32,078 --> 00:36:36,308 They didn't do, um, you know, I Why would you, why would you ever 719 00:36:36,308 --> 00:36:37,718 want to do business with them? 720 00:36:37,718 --> 00:36:39,218 And I have a similar feeling. 721 00:36:39,218 --> 00:36:44,438 This is just my feeling, my opinion, if a company is so, so, 722 00:36:44,598 --> 00:36:47,538 two incidents of clear misjudgment. 723 00:36:47,568 --> 00:36:52,458 One in terms of the original design and outsourcing to another vendor, and 724 00:36:52,458 --> 00:36:59,868 then the second, the decision to, to make this public by, by blaming the 725 00:36:59,868 --> 00:37:00,168 vendor. 726 00:37:00,173 --> 00:37:03,108 What that, what that says to me is you take no blame yourself. 727 00:37:03,993 --> 00:37:08,733 If, if they had done a mea culpa, if they had said, Hey, uh, we really screwed up. 728 00:37:08,733 --> 00:37:10,863 We shouldn't have trusted everything to this vendor. 729 00:37:11,463 --> 00:37:12,813 Uh, maybe even trashed him. 730 00:37:12,813 --> 00:37:13,593 I don't care. 731 00:37:13,803 --> 00:37:14,283 Right. 732 00:37:14,503 --> 00:37:16,813 Now that I think back on it, that's the part that bothered 733 00:37:16,818 --> 00:37:19,393 me the most in the, in the story 734 00:37:19,633 --> 00:37:24,733 is that to this day, they take no responsibility for that. 735 00:37:25,033 --> 00:37:29,863 Uh, they also, they also take no, they know, um, there's no, um, wrongdoing. 736 00:37:29,863 --> 00:37:31,873 They admit no wrongdoing in the SEC, uh, 737 00:37:31,873 --> 00:37:32,443 the settlement. 738 00:37:33,108 --> 00:37:33,328 Prasanna Malaiyandi: Yep. 739 00:37:35,468 --> 00:37:36,888 W. Curtis Preston: Uh, at least they're consistent. 740 00:37:40,048 --> 00:37:43,498 Prasanna Malaiyandi: So, yeah, so I, I think. 741 00:37:44,383 --> 00:37:48,223 Like you said, this is useful as you're, because I don't 742 00:37:48,223 --> 00:37:49,423 think people actually do this. 743 00:37:49,423 --> 00:37:53,773 So when you're searching for a vendor, we always talk about go read reviews, 744 00:37:54,043 --> 00:37:58,873 go talk to other people, look at, uh, analyst results and or analysts 745 00:37:58,873 --> 00:38:00,043 opinions and things like that. 746 00:38:00,048 --> 00:38:05,743 Look at like the IDC surveys and, uh, the Magic Quadrant from Gartner. 747 00:38:06,343 --> 00:38:10,273 I don't think that we've told people, go Google the Vendor plus lawsuit. 748 00:38:10,808 --> 00:38:10,928 I. 749 00:38:11,053 --> 00:38:11,413 You know, 750 00:38:11,548 --> 00:38:11,968 W. Curtis Preston: Yeah. 751 00:38:12,673 --> 00:38:14,893 Prasanna Malaiyandi: go look and see what has come up. 752 00:38:14,893 --> 00:38:18,823 Like is this a company that you are comfortable doing business with 753 00:38:18,823 --> 00:38:23,173 and trusting that they will stand beside you when things go wrong? 754 00:38:23,203 --> 00:38:24,313 If things go wrong? 755 00:38:25,528 --> 00:38:25,768 W. Curtis Preston: Yeah. 756 00:38:25,768 --> 00:38:28,588 And, and I think it's important to say that I have no ax to grind with 757 00:38:28,588 --> 00:38:30,568 these, with these folks, right? 758 00:38:30,838 --> 00:38:33,028 I, I don't work for a competitor. 759 00:38:33,028 --> 00:38:35,608 I don't, they didn't personally wrong me. 760 00:38:36,028 --> 00:38:39,268 Uh, they've never discussed me publicly that I know of. 761 00:38:39,348 --> 00:38:40,848 It chaps my hide. 762 00:38:40,848 --> 00:38:42,828 It just the idea that that. 763 00:38:43,523 --> 00:38:46,853 You accepted no responsibility for that. 764 00:38:46,973 --> 00:38:53,393 And then again, apparently even up to recently, it, it appears 765 00:38:53,393 --> 00:38:57,871 that you were doing stuff, um, that seems a little shady, 766 00:38:57,976 --> 00:38:58,266 Prasanna Malaiyandi: Yeah. 767 00:38:58,591 --> 00:39:01,586 Now granted that was 2018 when they made those statements, but. 768 00:39:02,386 --> 00:39:03,961 W. Curtis Preston: That was not that long ago, man. 769 00:39:04,021 --> 00:39:05,431 That was just before Covid. 770 00:39:06,301 --> 00:39:07,741 That was like, I was like, yes. 771 00:39:07,861 --> 00:39:10,756 Covid iss like a day like Covid, COVID. 772 00:39:11,376 --> 00:39:13,711 The entire period of covid counts as like a day. 773 00:39:14,341 --> 00:39:16,411 I, yeah, that was 2019. 774 00:39:16,461 --> 00:39:16,821 Prasanna Malaiyandi: Yeah. 775 00:39:17,451 --> 00:39:21,141 Now the one thing though that we have to state is since the 776 00:39:21,146 --> 00:39:25,581 acquisition by OpenText, things could be different, right? 777 00:39:26,016 --> 00:39:26,586 W. Curtis Preston: that is true. 778 00:39:26,991 --> 00:39:29,901 Prasanna Malaiyandi: So I think we should at least give 'em that benefit 779 00:39:29,901 --> 00:39:31,191 of the doubt that things could 780 00:39:31,236 --> 00:39:32,616 W. Curtis Preston: That's actually a really good point. 781 00:39:32,646 --> 00:39:33,606 Thanks for bringing that up. 782 00:39:33,786 --> 00:39:37,536 You know, we're, we're commenting mainly about the company that was. 783 00:39:38,916 --> 00:39:41,616 Carbonite before it was acquired by OpenText. 784 00:39:41,646 --> 00:39:47,736 And perhaps this acquisition of OpenText will significantly change things, uh, or 785 00:39:47,741 --> 00:39:49,806 has already significantly changed things. 786 00:39:50,256 --> 00:39:54,306 This is really what, what we need to, um. 787 00:39:55,386 --> 00:39:59,946 When I think about like the OVH situation, what can we learn from this? 788 00:39:59,946 --> 00:40:05,466 What can people do to protect themselves from this when 789 00:40:05,471 --> 00:40:07,506 using a cloud backup vendor? 790 00:40:07,986 --> 00:40:10,446 I would, I would number one, be asking them what kind of 791 00:40:10,446 --> 00:40:11,886 storage technology they're using. 792 00:40:12,111 --> 00:40:12,461 Right. 793 00:40:13,581 --> 00:40:13,881 Prasanna Malaiyandi: I would 794 00:40:13,881 --> 00:40:17,001 ask about the availability and redundancy, 795 00:40:18,026 --> 00:40:18,456 W. Curtis Preston: Yeah. 796 00:40:18,966 --> 00:40:22,776 Um, you know, are you using storage arrays or are you using, are you using S3? 797 00:40:22,776 --> 00:40:23,016 Are you 798 00:40:23,016 --> 00:40:27,336 using, you know, what, how are you storing my data and how are you making 799 00:40:27,336 --> 00:40:29,556 sure you know that it stays available? 800 00:40:30,126 --> 00:40:33,906 Prasanna Malaiyandi: I, I, think though, as a user, maybe what 801 00:40:33,906 --> 00:40:38,016 underlying technology is less important because I could see some vendors 802 00:40:38,016 --> 00:40:40,121 maybe would not discuss that I. 803 00:40:40,396 --> 00:40:44,086 But at least ensuring the characteristics that you expect of S3. 804 00:40:44,086 --> 00:40:47,476 So for instance, is it geographically spread out? 805 00:40:47,536 --> 00:40:47,866 Right? 806 00:40:47,866 --> 00:40:49,276 Is it redundant? 807 00:40:49,666 --> 00:40:53,266 Is it a replicated copy, independent copy, right. 808 00:40:53,266 --> 00:40:54,466 Those sort of things. 809 00:40:54,901 --> 00:40:56,071 W. Curtis Preston: I understand what you're saying. 810 00:40:56,071 --> 00:40:59,221 They might not want to say, but like, you know, where we used to work, 811 00:40:59,221 --> 00:41:00,961 we just say, yeah, it's, it is on S3. 812 00:41:01,411 --> 00:41:01,831 Right. 813 00:41:01,951 --> 00:41:03,061 If you could say that, 814 00:41:03,511 --> 00:41:06,451 I think that gives you a certain amount of, right. 815 00:41:07,241 --> 00:41:11,856 Just like you just said, is this designed to survive, um, you know, 816 00:41:11,856 --> 00:41:13,766 a fire, like what happened at OVH? 817 00:41:13,926 --> 00:41:15,846 Is this designed to survive? 818 00:41:16,431 --> 00:41:18,801 Um, you know, a multiple disc failure. 819 00:41:18,861 --> 00:41:22,551 What happens in a mul, in, in, in a, in a catastrophic failure of, 820 00:41:23,061 --> 00:41:27,621 of, you know, because let's just, let's just, let's just say this. 821 00:41:27,831 --> 00:41:28,581 'cause here's the thing. 822 00:41:28,941 --> 00:41:31,611 A single fire would've done the same thing. 823 00:41:32,361 --> 00:41:32,751 Prasanna Malaiyandi: Yep. 824 00:41:32,991 --> 00:41:34,971 W. Curtis Preston: And then who are they, who are they gonna sue then 825 00:41:35,001 --> 00:41:36,681 their fire extinguisher company. 826 00:41:37,401 --> 00:41:37,911 Right? 827 00:41:38,211 --> 00:41:43,491 What the core problem here is that they had a single copy of customer data. 828 00:41:44,811 --> 00:41:45,291 Prasanna Malaiyandi: yep, 829 00:41:45,636 --> 00:41:46,176 W. Curtis Preston: That's it, 830 00:41:46,626 --> 00:41:46,956 right? 831 00:41:46,956 --> 00:41:47,916 No redundancy. 832 00:41:47,916 --> 00:41:48,216 No. 833 00:41:48,216 --> 00:41:49,446 No geo redundancy. 834 00:41:49,641 --> 00:41:51,026 Prasanna Malaiyandi: Or if they had Acho go 835 00:41:51,186 --> 00:41:51,966 W. Curtis Preston: redundancy. 836 00:41:52,716 --> 00:41:53,196 A derecho. 837 00:41:53,856 --> 00:41:54,846 You love talking about that 838 00:41:54,846 --> 00:41:55,686 derecho, don't you? 839 00:41:55,881 --> 00:41:57,381 Prasanna Malaiyandi: I love talking about the DRE show. 840 00:41:57,591 --> 00:41:57,801 It's an 841 00:41:57,996 --> 00:41:58,386 W. Curtis Preston: I never 842 00:41:58,386 --> 00:41:59,676 even heard of a derecho until we 843 00:41:59,681 --> 00:42:00,846 talked about it on the show. 844 00:42:01,071 --> 00:42:01,581 Prasanna Malaiyandi: Exactly. 845 00:42:02,616 --> 00:42:05,196 W. Curtis Preston: Yeah, it's a hurricane that starts over land. 846 00:42:05,316 --> 00:42:08,076 For those of you that don't know what a derecho is, and don't ask me why. 847 00:42:08,916 --> 00:42:12,786 It's the same as the word or really close to the word for right. 848 00:42:12,816 --> 00:42:13,596 In Spanish. 849 00:42:14,916 --> 00:42:22,176 Um, any, can you think of any other, you know, things that 850 00:42:22,176 --> 00:42:23,886 a, that a customer can do to, 851 00:42:24,491 --> 00:42:26,171 Because you know, the, you know what this reminds me of? 852 00:42:26,201 --> 00:42:28,361 You know, there's, you know, you remember the old thing of like, on the 853 00:42:28,361 --> 00:42:29,711 internet, nobody knows you're a dog. 854 00:42:29,801 --> 00:42:30,131 Right. 855 00:42:31,811 --> 00:42:37,001 It's like what can a customer do to ensure that they don't go to 856 00:42:37,481 --> 00:42:42,221 essentially a another vendor that behaves the way Carbonite did back 857 00:42:42,221 --> 00:42:43,211 in 2009? 858 00:42:44,441 --> 00:42:47,291 Prasanna Malaiyandi: I think another thing you should do, and it's not 859 00:42:47,291 --> 00:42:50,831 a guarantee, but at least gives you some protections, is take a look in 860 00:42:50,831 --> 00:42:53,951 their SLAs and in the contract, right? 861 00:42:53,951 --> 00:42:58,331 Do they talk about disaster recovery redundancy? 862 00:42:58,691 --> 00:43:00,371 Do they talk about SLAs? 863 00:43:00,371 --> 00:43:04,871 Do they talk anything about geographic separation for storage? 864 00:43:05,591 --> 00:43:05,981 W. Curtis Preston: Right. 865 00:43:06,161 --> 00:43:06,491 Prasanna Malaiyandi: Right? 866 00:43:06,761 --> 00:43:08,381 If it's there, that's good. 867 00:43:08,681 --> 00:43:10,511 If it's missing, ask the questions. 868 00:43:11,621 --> 00:43:12,191 W. Curtis Preston: Yeah. 869 00:43:12,521 --> 00:43:12,881 Yeah. 870 00:43:14,171 --> 00:43:15,761 I think ask as many questions as 871 00:43:15,821 --> 00:43:17,261 Prasanna Malaiyandi: yeah, it's not foolproof. 872 00:43:17,261 --> 00:43:19,391 as we found with, like we found with OVH. 873 00:43:19,391 --> 00:43:19,571 Right? 874 00:43:19,576 --> 00:43:21,881 It was written in the contracts, but it still didn't mean anything. 875 00:43:22,391 --> 00:43:27,521 But at least it gives you a more legal standing for if there is 876 00:43:27,526 --> 00:43:29,291 a lawsuit that comes from this. 877 00:43:30,461 --> 00:43:30,941 W. Curtis Preston: Yeah. 878 00:43:31,481 --> 00:43:36,641 Um, and, uh, I, I hope, I hope the audience has appreciated this. 879 00:43:36,646 --> 00:43:39,071 This is our best attempt to, to deduce. 880 00:43:39,071 --> 00:43:45,281 I'm looking up here at my browser and there are like 17 tabs open to the story, 881 00:43:45,426 --> 00:43:49,001 to the different story, all of which we're trying to cover the same article. 882 00:43:49,151 --> 00:43:52,421 And, and one of the reasons for that is that the, the, 883 00:43:52,481 --> 00:43:54,161 the original article is gone. 884 00:43:54,161 --> 00:43:56,081 The original article in the Boston. 885 00:43:56,621 --> 00:43:57,761 Uh, globe. 886 00:43:57,761 --> 00:44:01,391 Is it that original article is gone and I couldn't, I also couldn't 887 00:44:01,391 --> 00:44:03,011 find it in the internet archive. 888 00:44:03,011 --> 00:44:03,911 I tried doing that. 889 00:44:04,661 --> 00:44:09,071 So we had to look at a whole bunch of different blogs that were, uh, essentially 890 00:44:09,071 --> 00:44:10,751 trying to summarize the same article. 891 00:44:11,201 --> 00:44:14,621 But, um, you know, this is our best attempt at trying to, 892 00:44:14,741 --> 00:44:16,301 to see what we saw from all 893 00:44:16,301 --> 00:44:16,871 these various 894 00:44:17,186 --> 00:44:19,376 Prasanna Malaiyandi: and, and it may not be a hundred percent accurate, right? 895 00:44:19,376 --> 00:44:21,506 We're just going off the information 896 00:44:21,620 --> 00:44:23,060 W. Curtis Preston: Based on the articles that we have. 897 00:44:23,090 --> 00:44:23,390 Yeah. 898 00:44:23,690 --> 00:44:23,960 Yeah. 899 00:44:24,470 --> 00:44:26,330 Yeah, All right. 900 00:44:26,330 --> 00:44:29,030 Well, it's been not fun. 901 00:44:29,120 --> 00:44:30,080 It's not a fun one. 902 00:44:30,380 --> 00:44:30,860 I don't like it. 903 00:44:30,860 --> 00:44:31,160 This is 904 00:44:31,160 --> 00:44:34,670 my world and I This is, this is, this is people that lost data 905 00:44:34,865 --> 00:44:35,195 Prasanna Malaiyandi: it's, 906 00:44:35,210 --> 00:44:38,270 W. Curtis Preston: that did what we told 'em to do, and they still lost data. 907 00:44:38,555 --> 00:44:42,185 Prasanna Malaiyandi: so it's not fun, but I think it is hopefully 908 00:44:42,515 --> 00:44:47,465 eye-opening for folks that they need to be more inquisitive and 909 00:44:47,465 --> 00:44:49,145 not just blindly trust vendors. 910 00:44:50,405 --> 00:44:50,825 W. Curtis Preston: Yeah. 911 00:44:50,885 --> 00:44:52,415 Never blindly trust vendors. 912 00:44:52,685 --> 00:44:52,895 Prasanna Malaiyandi: Yeah. 913 00:44:53,555 --> 00:44:53,975 W. Curtis Preston: Yeah. 914 00:44:54,965 --> 00:44:55,685 All right. 915 00:44:55,805 --> 00:44:59,090 Well, uh, thank you Prasanna for your usual 916 00:44:59,610 --> 00:44:59,890 analysis. 917 00:45:00,550 --> 00:45:02,465 Prasanna Malaiyandi: I, I enjoy these. 918 00:45:02,465 --> 00:45:04,175 I actually like this series, Curtis. 919 00:45:04,175 --> 00:45:05,675 I hope our listeners do as well. 920 00:45:05,855 --> 00:45:05,885 W. Curtis Preston: I. 921 00:45:06,425 --> 00:45:08,135 Yeah, I'm, I'm enjoying the series as well. 922 00:45:08,135 --> 00:45:12,905 It's been nice to go back to these stories that I remember from 15 years ago, and 923 00:45:12,905 --> 00:45:16,910 then we're, you know, in some cases, like the OVH Fire, we're seeing some follow up. 924 00:45:17,525 --> 00:45:21,335 This one, and in this one we saw some follow up we didn't expect 925 00:45:21,335 --> 00:45:25,055 from quite a bit later, which technically unrelated, but you know. 926 00:45:25,115 --> 00:45:29,705 Anyway, um, I, you know, listeners, we appreciate you. 927 00:45:29,765 --> 00:45:34,595 Uh, you know, we'd be nothing without you and, uh, be sure to subscribe 928 00:45:34,595 --> 00:45:35,975 so that you don't miss an episode. 929 00:45:36,185 --> 00:45:37,415 That is a wrap.