1 00:00:00,000 --> 00:00:03,479 Boy, do I have a doozy of a story for you today? 2 00:00:03,930 --> 00:00:04,770 Picture this. 3 00:00:04,800 --> 00:00:08,520 You're just minding your own business, trusting your cloud provider to 4 00:00:08,520 --> 00:00:10,229 keep your data safe and sound. 5 00:00:10,740 --> 00:00:17,099 That's when a massive fire rips through and destroys their entire data center and 6 00:00:17,099 --> 00:00:19,590 suddenly your whole digital world is gone. 7 00:00:20,460 --> 00:00:24,930 That's exactly what happened in 2021 to the poor customers of OVH. 8 00:00:25,319 --> 00:00:28,560 And let me tell you the more I dug into the story, the more I 9 00:00:28,560 --> 00:00:30,960 realized just how messed up it was. 10 00:00:31,500 --> 00:00:35,639 We're talking about a company that claim to have top notch backup practices, 11 00:00:35,879 --> 00:00:40,020 but in reality, they were about as reliable as a chocolate teapot. 12 00:00:40,889 --> 00:00:44,940 This is a wake up call for anyone who's ever put their faith in the cloud. 13 00:00:45,270 --> 00:00:50,220 This episode will be a wild ride through the ups and downs of this whole fiasco. 14 00:00:50,609 --> 00:00:54,120 We'll talk about the Shadys decisions made by OVH cloud. 15 00:00:54,449 --> 00:00:57,270 Y the fire got so bad in the first place. 16 00:00:57,389 --> 00:01:01,349 Why companies lost data and the crucial lessons that every single one 17 00:01:01,349 --> 00:01:03,090 of us needs to learn from this mess. 18 00:01:03,810 --> 00:01:06,810 So whether you're an it hot shot or just someone who wants to 19 00:01:06,810 --> 00:01:08,490 keep their data out of the fire. 20 00:01:08,789 --> 00:01:09,900 You better listen up. 21 00:01:10,410 --> 00:01:14,759 By the end of this episode, you'll know exactly what questions to ask your cloud 22 00:01:14,759 --> 00:01:19,919 provider to make sure you never end up in the same boat as those poor OVH customers. 23 00:01:20,820 --> 00:01:23,100 I'm w Curtis Preston, AKA Mr. 24 00:01:23,100 --> 00:01:25,710 Backup, and no one cares about your data more than I do. 25 00:01:26,280 --> 00:01:29,490 I want to turn you the unappreciated backup admin. 26 00:01:29,789 --> 00:01:31,710 Into a cyber recovery hero. 27 00:01:32,130 --> 00:01:34,080 This is the backup wrap-up. 28 00:01:45,783 --> 00:01:46,743 W. Curtis Preston: Welcome to the show. 29 00:01:47,248 --> 00:01:55,258 I'm w Curtis Preston, and I have with me my election worker post-Traumatic Stress 30 00:01:55,438 --> 00:01:59,998 Counselor Prasanna Malaiyandi how's it 31 00:02:00,103 --> 00:02:03,703 Prasanna Malaiyandi: Ah, I'm doing well, Curtis, uh, how are 32 00:02:03,703 --> 00:02:05,923 you doing or have you recovered? 33 00:02:05,923 --> 00:02:11,833 I know that you had a pretty hectic 10 days, of which the last day was 34 00:02:11,833 --> 00:02:17,533 prob 11 days, of which the last day was probably like a lot more 35 00:02:17,533 --> 00:02:20,113 effort than the first 10 days, so, 36 00:02:21,088 --> 00:02:21,538 W. Curtis Preston: Yeah. 37 00:02:21,568 --> 00:02:27,448 Basically we got more voters on the, the actual, the actual election day. 38 00:02:27,868 --> 00:02:29,578 It was the primary, it was super Tuesday. 39 00:02:29,578 --> 00:02:33,273 For those of you, which for those of you not in the US is I. 40 00:02:33,538 --> 00:02:38,368 A day when many states have their primaries for the presidential election. 41 00:02:38,428 --> 00:02:42,298 And, uh, California is one of those, which is where I live, and 42 00:02:42,298 --> 00:02:44,158 I happen to be an election worker. 43 00:02:44,518 --> 00:02:46,528 Uh, it's something that I do volunteer. 44 00:02:46,888 --> 00:02:51,178 Well, I, I volunteer to do it, but I do get paid for the time that I spend there. 45 00:02:51,183 --> 00:02:54,668 Not a ton, but, but I do get paid for my time but you know what, 46 00:02:54,668 --> 00:02:59,678 persona, I could, I could very easily talk about this for an hour. 47 00:02:59,718 --> 00:03:01,441 Prasanna Malaiyandi: I think we've had three episodes. 48 00:03:01,604 --> 00:03:01,934 W. Curtis Preston: You know what? 49 00:03:01,934 --> 00:03:02,894 I'll put in the show notes. 50 00:03:02,894 --> 00:03:05,414 If you're curious to know my thoughts on the way elections 51 00:03:05,419 --> 00:03:08,204 work and how they actually work versus the way you think they work. 52 00:03:08,714 --> 00:03:12,104 Um, you know, you know, go, go check those. 53 00:03:12,104 --> 00:03:13,814 I'll, I'll put, I'll put 'em in the show notes and you 54 00:03:13,844 --> 00:03:16,484 Prasanna Malaiyandi: and we do have also the episode with Mark. 55 00:03:16,694 --> 00:03:17,114 Mark. 56 00:03:17,159 --> 00:03:17,459 W. Curtis Preston: Yes. 57 00:03:17,459 --> 00:03:18,689 And we have the episode with Mark. 58 00:03:18,719 --> 00:03:18,989 Yeah. 59 00:03:19,019 --> 00:03:22,349 There there's two ways that site managers run the sites. 60 00:03:22,679 --> 00:03:26,819 Mark is of the first type, which is the, I'm here to help you vote and 61 00:03:26,819 --> 00:03:29,759 if you have any questions other than that, like if you want to question 62 00:03:29,759 --> 00:03:31,889 the process, here's a phone number I. 63 00:03:32,314 --> 00:03:35,884 I'm more the, the type that will actually take the question. 64 00:03:36,124 --> 00:03:39,454 I remember one voter, I basically was grilled for 65 00:03:39,454 --> 00:03:41,764 probably 45 minutes by a voter. 66 00:03:42,214 --> 00:03:46,804 I feel that she went away with a little more confidence in the, in the system. 67 00:03:47,654 --> 00:03:51,884 And, um, what was really funny was the moment when I had two voters 68 00:03:51,884 --> 00:03:53,384 within five minutes of each other. 69 00:03:53,384 --> 00:03:57,674 One who said, and it was interesting, he's really, really old gentleman. 70 00:03:57,704 --> 00:03:58,874 Like, I'm, I'm old. 71 00:03:58,879 --> 00:04:02,024 He was old to me, so he was old, right? 72 00:04:02,324 --> 00:04:05,684 And he, he basically said, this is crazy. 73 00:04:05,684 --> 00:04:08,414 The, you know, with all this technology, we could just, we could 74 00:04:08,419 --> 00:04:12,134 just replay all this stupid paper with technology and then another person. 75 00:04:12,554 --> 00:04:16,184 Two minutes later it came up and said, there's way too much technology here. 76 00:04:16,184 --> 00:04:17,804 We should just do this with paper. 77 00:04:17,894 --> 00:04:20,894 And I was like, well, we are doing it with paper. 78 00:04:20,894 --> 00:04:23,774 But you know, I don't have time to go into that. 79 00:04:23,774 --> 00:04:29,414 But anyway, so, uh, it is time to get into our topic of the week. 80 00:04:30,054 --> 00:04:37,904 This week in our continued series on cloud disasters where people thought 81 00:04:37,904 --> 00:04:42,194 that they had backups or think that they don't need backups, and then 82 00:04:42,584 --> 00:04:48,104 they found out otherwise, this one is a pretty big one, wouldn't you say? 83 00:04:48,314 --> 00:04:48,584 Prasanna Malaiyandi: Oh yeah. 84 00:04:48,589 --> 00:04:55,064 I think it was probably one of the most impactful ones. 85 00:04:55,484 --> 00:04:55,724 Right. 86 00:04:55,724 --> 00:04:56,984 Just in terms of the customer 87 00:04:56,984 --> 00:05:00,099 W. Curtis Preston: but it both, most Im, it's most impactful. 88 00:05:00,289 --> 00:05:02,719 Customers were really messed up with this. 89 00:05:03,049 --> 00:05:04,604 And also I. 90 00:05:05,224 --> 00:05:05,734 You know it. 91 00:05:06,214 --> 00:05:09,634 So, and in another way it was not impactful at all. 92 00:05:09,664 --> 00:05:10,624 We're gonna talk about that. 93 00:05:10,684 --> 00:05:15,844 Uh, you know, in the story we're talking about the OVH Fire. 94 00:05:16,339 --> 00:05:17,959 Of 2021. 95 00:05:18,559 --> 00:05:23,149 So what's interesting is, you know, we have primarily a North American audience. 96 00:05:23,149 --> 00:05:26,389 We have, you know, plenty from other countries, but, but the bulk 97 00:05:26,394 --> 00:05:27,919 of our audiences in North America. 98 00:05:28,159 --> 00:05:31,849 And I'm willing to bet that prior to this fire, the average person in 99 00:05:31,854 --> 00:05:34,909 North America had never heard of OVH. 100 00:05:35,019 --> 00:05:35,914 Prasanna Malaiyandi: I never heard about o. 101 00:05:35,929 --> 00:05:36,829 W. Curtis Preston: I'm willing to bet. 102 00:05:37,519 --> 00:05:38,209 Me neither. 103 00:05:38,269 --> 00:05:38,659 Right. 104 00:05:39,079 --> 00:05:44,839 And I, um, I'm willing to bet that the average person still doesn't 105 00:05:44,839 --> 00:05:46,129 know anything about opiates. 106 00:05:46,219 --> 00:05:52,849 That this sort, that this story sort of came and went and it's kind of amazing 107 00:05:53,329 --> 00:05:55,969 and, um, that's a little bit sad. 108 00:05:56,449 --> 00:05:56,749 Prasanna Malaiyandi: Yep. 109 00:05:56,749 --> 00:06:01,729 W. Curtis Preston: So, um, you want to sort of tell the story of the fire? 110 00:06:01,804 --> 00:06:04,054 Prasanna Malaiyandi: Yeah, so in March. 111 00:06:04,114 --> 00:06:06,544 So I think before we get there, so a couple things. 112 00:06:06,544 --> 00:06:08,254 First people should probably know a little bit. 113 00:06:08,254 --> 00:06:09,694 I know you talked about OVH isn't 114 00:06:09,769 --> 00:06:10,789 W. Curtis Preston: Oh, that is right. 115 00:06:10,819 --> 00:06:11,089 Yep. 116 00:06:11,134 --> 00:06:11,434 Prasanna Malaiyandi: right? 117 00:06:11,434 --> 00:06:13,594 But OVH is a cloud provider. 118 00:06:13,594 --> 00:06:18,184 They are the largest European headquartered cloud provider. 119 00:06:19,189 --> 00:06:19,699 W. Curtis Preston: Right. 120 00:06:19,819 --> 00:06:21,229 And they're headquartered in France. 121 00:06:21,679 --> 00:06:25,159 Prasanna Malaiyandi: and one of the unique things of is, at least for this data 122 00:06:25,159 --> 00:06:28,429 center, the way that they built it, is they actually use shipping containers. 123 00:06:29,089 --> 00:06:31,399 Right, so they're kind of stacked on top of each other. 124 00:06:31,399 --> 00:06:35,899 They have a bunch of servers in there with power and cooling and also wooden 125 00:06:35,899 --> 00:06:39,529 walls and framing inside as well. 126 00:06:39,979 --> 00:06:43,579 Uh, because it is a shipping container, I've actually watched a bunch of YouTube 127 00:06:43,579 --> 00:06:46,099 people who do a lot of container homes. 128 00:06:46,399 --> 00:06:51,259 And so yeah, you, you need to add structure to containers in order to 129 00:06:51,259 --> 00:06:55,909 be able to do things like put floors and walls and other aspects and 130 00:06:55,909 --> 00:06:58,459 W. Curtis Preston: There's interesting, uh, relationship 131 00:06:58,459 --> 00:07:02,089 there because the videos to which you referred, they also lost data. 132 00:07:03,454 --> 00:07:03,874 Prasanna Malaiyandi: Yep. 133 00:07:04,384 --> 00:07:05,614 Yeah, the, yep. 134 00:07:05,704 --> 00:07:06,754 The one that I watched. 135 00:07:06,799 --> 00:07:08,179 W. Curtis Preston: we'll come maybe. 136 00:07:08,179 --> 00:07:08,479 Yeah. 137 00:07:08,479 --> 00:07:08,809 Maybe. 138 00:07:08,809 --> 00:07:10,639 We'll, we'll come back to that in a later date. 139 00:07:11,209 --> 00:07:11,809 Uh, yeah. 140 00:07:11,809 --> 00:07:17,869 So it was a pretty big company, uh, big enough that Well, we, we'll save the 141 00:07:17,929 --> 00:07:18,109 Prasanna Malaiyandi: Yeah. 142 00:07:18,259 --> 00:07:22,549 And, and, and they have a lot of customers and they're a lot of government 143 00:07:22,549 --> 00:07:26,809 contracts as well, so it's not like it's like a mom and pop cloud provider. 144 00:07:26,809 --> 00:07:27,019 Right. 145 00:07:27,019 --> 00:07:29,749 It is a huge cloud provider 146 00:07:29,799 --> 00:07:32,439 W. Curtis Preston: also, an important thing is that they 147 00:07:32,439 --> 00:07:34,239 offered a backup service. 148 00:07:34,794 --> 00:07:36,594 That you could pay extra for. 149 00:07:36,654 --> 00:07:40,944 Um, and, uh, it happened to be hosted Veeam, right? 150 00:07:41,274 --> 00:07:45,024 Uh, and you know, Veeam did nothing wrong here, but definitely the vendor did. 151 00:07:45,264 --> 00:07:48,024 They offered a, a hosted backup service. 152 00:07:48,624 --> 00:07:51,984 It's a little bit different than the average cloud backup. 153 00:07:52,599 --> 00:07:56,409 Problem where it's like, well, you didn't, you expected your cloud vendor 154 00:07:56,409 --> 00:07:59,679 to back up and you, they didn't back up and you should have backed up. 155 00:07:59,679 --> 00:08:02,529 And so it's nobody's fault but your own. 156 00:08:02,534 --> 00:08:06,879 But in this case, there were customers that actually paid for the backup service. 157 00:08:07,269 --> 00:08:08,229 And then what happened? 158 00:08:08,649 --> 00:08:10,419 Prasanna Malaiyandi: So what ended up happening in March 159 00:08:10,419 --> 00:08:13,374 of 2021, there was a fire and. 160 00:08:14,844 --> 00:08:18,624 So how they have it situated is there are basically four, I don't know if 161 00:08:18,624 --> 00:08:23,334 you'd call 'em data centers within that single site, but they had four 162 00:08:23,904 --> 00:08:27,474 data centers, if you will, within it, and they had a fire in one. 163 00:08:28,359 --> 00:08:30,129 And these are shipping containers. 164 00:08:30,129 --> 00:08:35,079 And apparently they did not have the right fire suppress suppression equipment. 165 00:08:35,619 --> 00:08:39,159 They did not have electrical cutoff capabilities. 166 00:08:39,159 --> 00:08:42,549 And so firefighters came and they tried their best to put out that fire, 167 00:08:42,549 --> 00:08:44,859 but the thing just spread like crazy. 168 00:08:44,979 --> 00:08:49,629 W. Curtis Preston: just, yeah, the, the, the, the craziest thing in the report that 169 00:08:49,634 --> 00:08:55,929 came out a year later, the, the bass rin report, it's, it said that it started at 170 00:08:55,929 --> 00:08:58,299 an electrical inverter on the first floor. 171 00:08:58,749 --> 00:09:00,159 But here's my favorite phrase. 172 00:09:00,164 --> 00:09:04,209 When the firefighters came, they were met with electric arcs of 173 00:09:04,209 --> 00:09:09,069 more than one meter around the exterior door of the energy room. 174 00:09:09,729 --> 00:09:12,609 Like that sounds something out of like a. 175 00:09:13,104 --> 00:09:16,104 Like a Avengers movie, you know what I mean? 176 00:09:16,104 --> 00:09:18,474 Like this is what you call in Superman. 177 00:09:18,954 --> 00:09:24,954 A a one meter long, that's about a yard for those of you who live in the US right? 178 00:09:25,404 --> 00:09:31,134 Uh, electrical arcs that long, and then no suppression. 179 00:09:31,224 --> 00:09:34,824 It said it took 'em, uh, over two hours to get the fire or to get 180 00:09:34,824 --> 00:09:38,904 the electrical cut off because they had to do it, uh, somewhere else. 181 00:09:38,904 --> 00:09:39,624 They couldn't do it in the 182 00:09:39,714 --> 00:09:43,284 Prasanna Malaiyandi: And it wasn't just the one data center that got 183 00:09:43,284 --> 00:09:45,594 affected, it also impacted the other one. 184 00:09:45,594 --> 00:09:49,374 So I think one of the other ones had some damage. 185 00:09:49,374 --> 00:09:49,644 Right? 186 00:09:49,649 --> 00:09:51,744 It wasn't major, right? 187 00:09:51,744 --> 00:09:54,324 Like the first one, but there was still quite a bit of damage. 188 00:09:54,329 --> 00:09:58,194 And then the other two, they had to bring it offline because like you mentioned, 189 00:09:58,194 --> 00:09:59,394 they had to cut the electricity. 190 00:09:59,394 --> 00:10:01,134 They weren't sure what else was broken. 191 00:10:01,554 --> 00:10:01,914 Right? 192 00:10:01,914 --> 00:10:04,104 So they basically brought everything down. 193 00:10:06,099 --> 00:10:06,579 W. Curtis Preston: Yeah. 194 00:10:06,579 --> 00:10:10,779 It, it, it, it's like you look at the pictures of this, 195 00:10:10,779 --> 00:10:14,169 this was a giant fire, right? 196 00:10:14,259 --> 00:10:23,774 And the, and initially I felt that the, um, you know, that they, I. 197 00:10:24,114 --> 00:10:27,474 They responded well in that they, they, you know, they created, they 198 00:10:27,474 --> 00:10:32,724 did, what we, we suggest people do is they created a status page, um, 199 00:10:32,729 --> 00:10:34,734 and they updated that status page. 200 00:10:34,734 --> 00:10:39,624 The, the CEO actually made a number of videos, put them out. 201 00:10:39,624 --> 00:10:43,164 I'm not sure if he put 'em on YouTube or, uh, whatever, but he, he made it. 202 00:10:43,194 --> 00:10:46,464 He was trying to reassure people that we're doing everything we can. 203 00:10:48,504 --> 00:10:52,194 Recover from this, recover with a small R there. 204 00:10:52,289 --> 00:10:54,549 Um, but um, 205 00:10:54,879 --> 00:10:56,169 Prasanna Malaiyandi: And they were transparent, right? 206 00:10:56,169 --> 00:10:59,829 The other thing is they were like, Hey, we have other data centers. 207 00:11:00,189 --> 00:11:04,719 They were quick trying to bring up new servers to move these customers 208 00:11:04,719 --> 00:11:07,269 over to that new equipment, right? 209 00:11:07,274 --> 00:11:09,039 To get them up and running as quickly as possible. 210 00:11:09,459 --> 00:11:13,419 So they were bringing up the infrastructure, but 211 00:11:14,229 --> 00:11:15,729 this is a big but, right? 212 00:11:16,209 --> 00:11:17,169 What about their data? 213 00:11:19,134 --> 00:11:24,924 W. Curtis Preston: Yeah, so apparently the backup servers were literally, you know, 214 00:11:25,404 --> 00:11:31,674 in the same data center that they were backing up, and which, you know, is not in 215 00:11:31,674 --> 00:11:34,104 keeping with, you know, the 3, 2, 1 rule. 216 00:11:34,404 --> 00:11:34,854 Right? 217 00:11:35,109 --> 00:11:39,609 So that's just, that's just a fundamental design flaw, I would say. 218 00:11:39,609 --> 00:11:42,969 If you're offering backup services, you should not be putting the 219 00:11:42,969 --> 00:11:46,869 backup servers directly next to the thing that they're backing up. 220 00:11:46,899 --> 00:11:50,829 Right now they're saying that they were physically isolated, but they were just 221 00:11:50,889 --> 00:11:55,089 literally like over in the corner, like go, you know, go sit in the corner. 222 00:11:55,089 --> 00:11:56,079 Your backups. 223 00:11:56,919 --> 00:11:57,189 Yeah. 224 00:11:57,204 --> 00:11:59,574 Prasanna Malaiyandi: which is not what you were, and, and I don't know how many 225 00:11:59,574 --> 00:12:01,974 people actually understood that, right? 226 00:12:02,004 --> 00:12:06,594 Uh, if you were managing your own backup environment in your own data 227 00:12:06,594 --> 00:12:12,174 center, you could physically validate these things when you are trusting and 228 00:12:12,174 --> 00:12:15,714 buying a service from a vendor, right? 229 00:12:15,714 --> 00:12:19,044 You just have to either ask the hard questions to really figure out 230 00:12:19,044 --> 00:12:21,354 what they're doing or just trust that they're doing the right thing. 231 00:12:21,744 --> 00:12:25,014 And that second part got them, got a lot of customers burned. 232 00:12:25,899 --> 00:12:26,799 Do you like what I did there? 233 00:12:27,999 --> 00:12:31,279 W. Curtis Preston: I, I like what you just there, so about a year later 234 00:12:31,284 --> 00:12:33,469 is when the, the fire department. 235 00:12:34,234 --> 00:12:37,894 Uh, report came out and that's when we learned that they had no fire suppression 236 00:12:37,894 --> 00:12:39,904 systems, they had no power cutoff systems. 237 00:12:40,324 --> 00:12:43,354 Uh, we learned that it damaged at least two other data centers. 238 00:12:43,359 --> 00:12:43,714 Right. 239 00:12:44,344 --> 00:12:51,184 And the, because while they initially started out being very open. 240 00:12:52,849 --> 00:12:58,399 OVH suddenly stopped talking roughly in May of 2021. 241 00:12:58,399 --> 00:13:02,029 They stopped talking and, and, and they haven't spoken publicly about the 242 00:13:02,029 --> 00:13:09,319 incident since that, and that could be because they were planning for an event 243 00:13:09,819 --> 00:13:10,989 Prasanna Malaiyandi: it was a big event. 244 00:13:12,319 --> 00:13:13,129 W. Curtis Preston: Big event. 245 00:13:13,159 --> 00:13:15,319 What event might they have been planning for 246 00:13:15,934 --> 00:13:19,324 Prasanna Malaiyandi: Yeah, so in October of 2021, which is seven months after 247 00:13:19,324 --> 00:13:24,094 the fire, they went through an IPO and went public and raised a bunch of money. 248 00:13:25,384 --> 00:13:25,654 Right. 249 00:13:25,654 --> 00:13:29,794 It was not a it, and it wasn't a failure of an IPO either. 250 00:13:29,794 --> 00:13:31,294 Like I think the stock actually went up. 251 00:13:31,294 --> 00:13:33,904 They raised a good deal of money, right? 252 00:13:33,904 --> 00:13:39,574 So it was a pretty successful IPO seven months after this incident. 253 00:13:40,999 --> 00:13:41,119 W. Curtis Preston: I. 254 00:13:42,154 --> 00:13:45,094 That just flabbergast me, right? 255 00:13:45,274 --> 00:13:49,264 Like as, as someone who worked at a pre IPO company, right? 256 00:13:50,284 --> 00:13:54,784 I can't imagine something that catastrophic happening and 257 00:13:54,784 --> 00:13:57,604 then seven months later saying, you know what we should do? 258 00:13:58,174 --> 00:13:59,254 We should have people 259 00:13:59,464 --> 00:14:02,854 Prasanna Malaiyandi: Well, well, well, one thing is the IPO was 260 00:14:02,854 --> 00:14:04,774 probably already planned in the works. 261 00:14:04,774 --> 00:14:05,164 Right? 262 00:14:05,704 --> 00:14:06,784 Started going. 263 00:14:07,714 --> 00:14:09,544 And they were hoping that, 264 00:14:09,589 --> 00:14:10,999 W. Curtis Preston: they didn't wanna press pause. 265 00:14:12,304 --> 00:14:17,674 Prasanna Malaiyandi: you gotta strike while the iron's hot because remember, 266 00:14:17,684 --> 00:14:19,309 W. Curtis Preston: The was the, was. 267 00:14:22,504 --> 00:14:22,924 Prasanna Malaiyandi: right. 268 00:14:22,924 --> 00:14:27,784 So I think that they just took the chance, rolled the dice and figured, yeah, we're a 269 00:14:27,784 --> 00:14:34,324 large public cloud provider in Europe that we have sort of a dominant position here. 270 00:14:34,324 --> 00:14:36,064 We might as well raise the money while we can. 271 00:14:36,739 --> 00:14:38,359 I don't know how things ended up there. 272 00:14:38,359 --> 00:14:43,669 And I think like, and it's not like they've gone by the wayside, right? 273 00:14:43,669 --> 00:14:45,769 OVH is still popular in Europe. 274 00:14:45,769 --> 00:14:47,839 There are still a lot of people using their services. 275 00:14:47,839 --> 00:14:48,199 Right? 276 00:14:48,199 --> 00:14:51,109 So it's not like they just crumbled up and died. 277 00:14:52,489 --> 00:14:58,699 W. Curtis Preston: Well, um, I would like to point out that they went out at 20. 278 00:14:59,569 --> 00:15:01,189 Euros, right? 279 00:15:01,529 --> 00:15:05,429 , back in, uh, October of 2022. 280 00:15:05,939 --> 00:15:08,999 Uh, their current value is, uh, half that. 281 00:15:09,669 --> 00:15:09,959 Prasanna Malaiyandi: Okay. 282 00:15:10,589 --> 00:15:14,040 W. Curtis Preston: So basically they have, they went up a little bit, 283 00:15:15,179 --> 00:15:20,069 they had a little bit of a spike for about two months, and then it has 284 00:15:20,069 --> 00:15:22,529 done nothing but go down ever since. 285 00:15:23,129 --> 00:15:24,899 Um, so. 286 00:15:26,264 --> 00:15:29,564 You know, it, it may not have been a failure in the beginning, but it 287 00:15:29,564 --> 00:15:31,004 certainly doesn't look very good now. 288 00:15:31,004 --> 00:15:32,654 It's worth less than half, 289 00:15:32,804 --> 00:15:33,044 Prasanna Malaiyandi: Yeah. 290 00:15:33,914 --> 00:15:37,424 W. Curtis Preston: which gives me a little bit of consolation, I guess. 291 00:15:37,514 --> 00:15:39,584 Um, maybe it's that 292 00:15:40,484 --> 00:15:40,694 Prasanna Malaiyandi: But, 293 00:15:40,724 --> 00:15:43,124 W. Curtis Preston: the, as the news continued to come out 294 00:15:43,784 --> 00:15:44,024 Prasanna Malaiyandi: yeah. 295 00:15:44,054 --> 00:15:45,494 W. Curtis Preston: over the, over the years. 296 00:15:45,499 --> 00:15:45,974 Right. 297 00:15:46,604 --> 00:15:49,124 And there was other news as well. 298 00:15:49,174 --> 00:15:53,074 Immediately after this happened, we started hearing 299 00:15:53,079 --> 00:15:54,604 about a class action lawsuit. 300 00:15:54,664 --> 00:15:55,024 Right. 301 00:15:55,414 --> 00:16:00,239 And, uh, there was a firm, Ziegler and Associates that filed, uh. 302 00:16:00,649 --> 00:16:04,399 A class action lawsuit in September of 2021, which for the record 303 00:16:04,399 --> 00:16:06,169 would've been a month before the IPO. 304 00:16:06,949 --> 00:16:12,109 And they, um, they fired it, you know, for clients that had lost data, 305 00:16:12,199 --> 00:16:14,749 uh, to the fire or due to the fire. 306 00:16:15,079 --> 00:16:21,589 And as of the, basically about a year ago, it said there were 307 00:16:21,589 --> 00:16:23,449 140 customers in the lawsuit. 308 00:16:23,459 --> 00:16:24,914 Prasanna Malaiyandi: And the lawsuit is still ongoing. 309 00:16:26,699 --> 00:16:29,039 W. Curtis Preston: Yeah, it appears that, it appears that it's ongoing. 310 00:16:29,039 --> 00:16:32,459 We don't have any information on, at least I don't have any 311 00:16:32,459 --> 00:16:34,679 information that it ended. 312 00:16:34,679 --> 00:16:39,359 I, I didn't see any stories that it ended, but we do have a story about two 313 00:16:39,359 --> 00:16:44,399 customers that were apparently not in that class action law class action lawsuit. 314 00:16:44,729 --> 00:16:46,499 What did you, uh, hear about 315 00:16:46,679 --> 00:16:48,569 Prasanna Malaiyandi: Yeah, so there were two customers. 316 00:16:48,629 --> 00:16:55,469 Um, the first basically had, uh, sued them because they had signed 317 00:16:55,469 --> 00:16:56,909 up for the automated service. 318 00:16:57,944 --> 00:17:03,614 And, uh, based on the contract wording, the judge sided with them 319 00:17:03,614 --> 00:17:08,534 saying that, yes, physically isolated means that you keep 'em separate. 320 00:17:08,534 --> 00:17:12,374 And having it in the same building goes against like what's expected, 321 00:17:12,374 --> 00:17:15,764 what you would do for like a backup perspective and is wrong. 322 00:17:17,144 --> 00:17:21,164 And so they basically won a nominal amount. 323 00:17:21,164 --> 00:17:22,394 I would say they did not. 324 00:17:22,394 --> 00:17:24,524 I think they were suing for like 3 million euros. 325 00:17:24,529 --> 00:17:26,294 They got like a hundred thousand euros. 326 00:17:26,774 --> 00:17:31,604 So it was a nominal win, but it still showed that, yeah. 327 00:17:33,599 --> 00:17:33,959 W. Curtis Preston: Yeah. 328 00:17:33,959 --> 00:17:35,339 And, and a couple things there. 329 00:17:35,939 --> 00:17:40,229 While I agree with the, sometimes you have the spirit of the contract 330 00:17:40,229 --> 00:17:44,519 or the letter of the contract, I agree with what the judge says. 331 00:17:44,579 --> 00:17:47,069 Obviously, who does that, right? 332 00:17:47,069 --> 00:17:53,639 Who does the backup design, where the backup systems are in the same data 333 00:17:53,639 --> 00:17:56,069 center as the, uh, primary systems. 334 00:17:56,939 --> 00:17:58,259 Uh, having said that. 335 00:17:59,069 --> 00:18:03,509 It that was stated in the contract and that that was the most shocking 336 00:18:03,509 --> 00:18:09,479 part to me, is that, you know, in my review I went and found, um, you 337 00:18:09,479 --> 00:18:17,009 know, uh, basically the both the gotta love the way back machine, right? 338 00:18:17,189 --> 00:18:18,089 The, the internet archive. 339 00:18:18,089 --> 00:18:18,989 Using the internet. 340 00:18:19,259 --> 00:18:25,559 Using the internet archive, I was able to find the contract that they were quoting. 341 00:18:26,534 --> 00:18:33,694 And, uh, what the contract says is that for the backup service that it 342 00:18:33,694 --> 00:18:37,864 says that the storage resources for the backup service will be physically, 343 00:18:37,869 --> 00:18:43,504 I isolated from the servers that it's backing up, and then it's like 344 00:18:43,504 --> 00:18:46,204 somebody stopped reading right? 345 00:18:46,714 --> 00:18:50,884 Because the very next sentence that says the storage resources will be 346 00:18:50,884 --> 00:18:55,384 located in the same data center as the resources that it's backing up. 347 00:18:55,894 --> 00:18:57,904 It's, it's literally right in the contract. 348 00:18:58,654 --> 00:19:05,314 And, and so as, as like a, like a legal person, like I, I just wanna 349 00:19:05,314 --> 00:19:08,164 say, well, how did you not read that? 350 00:19:08,229 --> 00:19:08,519 Prasanna Malaiyandi: Yeah. 351 00:19:09,274 --> 00:19:11,524 W. Curtis Preston: I'm on the side of the customer here in that they 352 00:19:11,524 --> 00:19:15,994 paid for a backup service, but I'm not on the side of the customer in 353 00:19:15,994 --> 00:19:19,234 that they should have reviewed that contract and they should have said so. 354 00:19:19,234 --> 00:19:20,374 Wait, wait. 355 00:19:20,374 --> 00:19:20,914 What? 356 00:19:21,799 --> 00:19:22,159 Prasanna Malaiyandi: Yeah. 357 00:19:22,849 --> 00:19:26,989 Well, and unlike a lot of other contracts right, or like SA services where they 358 00:19:26,989 --> 00:19:30,679 talk nothing about backups, this was clearly stated what they are doing. 359 00:19:31,249 --> 00:19:34,369 W. Curtis Preston: stated they're, and they're paying extra, right? 360 00:19:34,369 --> 00:19:37,339 They were paying something like 2 cents a gigabyte per month 361 00:19:37,609 --> 00:19:39,559 for this backup service and. 362 00:19:41,464 --> 00:19:47,074 And the thing is, I think what the judge said, I, I like, this is one 363 00:19:47,074 --> 00:19:49,294 of those things where, you know, I'm really good at arguing, and 364 00:19:49,299 --> 00:19:51,094 I can argue both sides of this. 365 00:19:51,274 --> 00:19:54,574 Like on one side they should have read the contract, the customer should have 366 00:19:54,574 --> 00:19:57,634 read the contract, and they should have asked about what that sentence mean. 367 00:19:57,904 --> 00:20:00,514 And they should have then just not paid for the backup service 368 00:20:00,514 --> 00:20:01,474 because it wouldn't be worth it. 369 00:20:02,464 --> 00:20:05,554 Um, on the other hand, I agree with the judge, like, who does that? 370 00:20:06,319 --> 00:20:06,649 Right. 371 00:20:06,649 --> 00:20:10,549 It's not like what's the point of doing a backup if it's just 372 00:20:10,549 --> 00:20:13,069 two disk drives right next door? 373 00:20:13,129 --> 00:20:13,579 Right. 374 00:20:14,179 --> 00:20:18,499 And the judge was basically saying, this isn't in keeping with, you know, the 375 00:20:18,499 --> 00:20:19,969 state of the art of the backup system. 376 00:20:19,974 --> 00:20:21,139 Agreed, agreed, agreed. 377 00:20:21,649 --> 00:20:25,189 And it wouldn't allow you to meet the requirements of the contract. 378 00:20:25,729 --> 00:20:30,079 And so while the, I think basically what the judge says, you know, even though 379 00:20:30,079 --> 00:20:31,819 the contract may have stated that. 380 00:20:32,704 --> 00:20:33,874 What are you stupid? 381 00:20:33,874 --> 00:20:34,204 Right? 382 00:20:35,014 --> 00:20:35,494 You know what I mean? 383 00:20:35,594 --> 00:20:37,604 Prasanna Malaiyandi: Which is different than the second case, 384 00:20:38,188 --> 00:20:41,518 So there were two customers who ended up suing and winning, right? 385 00:20:41,518 --> 00:20:48,418 So the second customer also sued, uh, OVH, and this time the judge also found 386 00:20:48,418 --> 00:20:53,668 in the favor of the customer, and this one was slightly different because in 387 00:20:53,668 --> 00:20:58,108 their contract they had actually written out that they had a production system 388 00:20:58,108 --> 00:21:02,368 in one data center and explicitly their backup system in a different data center. 389 00:21:03,343 --> 00:21:07,183 In the contract, it was worded that their production and backup systems 390 00:21:07,183 --> 00:21:09,673 should be separate, and OVH agreed to it. 391 00:21:10,483 --> 00:21:15,133 And what ended up happening is OVH ended up keeping both copies 392 00:21:15,133 --> 00:21:16,813 of data in the same data center. 393 00:21:17,743 --> 00:21:18,283 But, 394 00:21:18,478 --> 00:21:19,723 W. Curtis Preston: basically they just lied. 395 00:21:19,993 --> 00:21:22,693 Prasanna Malaiyandi: but here's the problem, is their management console 396 00:21:22,723 --> 00:21:25,453 kept telling the customer that they were in separate data centers. 397 00:21:27,208 --> 00:21:27,748 W. Curtis Preston: Wow. 398 00:21:28,558 --> 00:21:28,918 Yeah. 399 00:21:28,948 --> 00:21:30,208 That's, that's just bad. 400 00:21:30,358 --> 00:21:33,628 And, and that's probably the worst thing that I've read about this story. 401 00:21:34,318 --> 00:21:38,698 Because if you are a customer that's about the best you could do is say, Hey 402 00:21:38,698 --> 00:21:42,298 listen, we're demanding that you put it separately and then the, you check the 403 00:21:42,298 --> 00:21:43,948 console and the console sets it separate. 404 00:21:44,128 --> 00:21:44,818 It's the cloud. 405 00:21:44,818 --> 00:21:47,248 Like, I can't, I can't go visit it. 406 00:21:47,638 --> 00:21:48,118 Right. 407 00:21:48,208 --> 00:21:51,868 Um, so like they, so that in that case, the customer did 408 00:21:51,868 --> 00:21:53,008 everything that they could. 409 00:21:53,098 --> 00:21:58,588 Prasanna Malaiyandi: And so the judge gave, uh, fined OVH 138,000 euros, 410 00:21:58,588 --> 00:22:02,458 something like that, but that's not even the worst part of OV H'S behavior. 411 00:22:02,998 --> 00:22:07,078 So OVH eventually found the backup server, and 412 00:22:07,218 --> 00:22:07,603 W. Curtis Preston: right. 413 00:22:07,633 --> 00:22:07,963 Yeah. 414 00:22:08,038 --> 00:22:08,568 This hurts. 415 00:22:09,463 --> 00:22:11,953 Prasanna Malaiyandi: And so they were like, Hey, yeah, customer. 416 00:22:11,953 --> 00:22:12,463 Great news. 417 00:22:12,463 --> 00:22:14,653 We found your backup server so you can restore your data. 418 00:22:15,043 --> 00:22:19,123 And while they were giving the customer back the backup server, they accidentally 419 00:22:19,123 --> 00:22:21,493 deleted the data on the discs. 420 00:22:26,778 --> 00:22:28,903 W. Curtis Preston: Uh, yeah. 421 00:22:29,983 --> 00:22:33,238 So, um, yeah, the, 422 00:22:34,348 --> 00:22:37,618 Prasanna Malaiyandi: Like, I just wanna know your OVH, you claim 423 00:22:37,618 --> 00:22:42,478 to be the third or the largest EU headquartered cloud provider. 424 00:22:42,958 --> 00:22:49,468 But simple processes like this, you're not able to. 425 00:22:49,858 --> 00:22:53,008 They, I get some of it's probably maybe they don't have the experience 426 00:22:53,008 --> 00:22:55,708 or expertise or they never had to go through this process. 427 00:22:55,948 --> 00:22:58,828 Hopefully now on they have better processes in place. 428 00:22:58,978 --> 00:23:00,748 It's like a learning experience, but. 429 00:23:01,918 --> 00:23:03,148 I feel for the customers. 430 00:23:04,723 --> 00:23:07,424 W. Curtis Preston: Yeah, I feel for those customers, and I, and I, I, 431 00:23:07,513 --> 00:23:11,743 I wish the best for the, the ones in the 140 class action lawsuit. 432 00:23:12,043 --> 00:23:15,013 It doesn't sound good if the two people who specifically had it in 433 00:23:15,013 --> 00:23:17,863 their contract only got $150,000. 434 00:23:17,863 --> 00:23:19,208 By the way, I would state I. 435 00:23:19,708 --> 00:23:20,758 Well, good for them. 436 00:23:20,788 --> 00:23:22,738 Like it's sort of like. 437 00:23:24,043 --> 00:23:26,233 I, I'm sure what they'd rather have is the data back. 438 00:23:26,238 --> 00:23:29,233 It, it is sort of like, it's like a wrongful death lawsuit. 439 00:23:29,233 --> 00:23:30,313 A wrongful death lawsuit. 440 00:23:30,313 --> 00:23:33,583 Can't bring your, the, the, the deceased person back. 441 00:23:34,113 --> 00:23:39,843 It's just in some way trying to compensate you and 150,000 Euros is not 442 00:23:39,843 --> 00:23:41,583 gonna compensate anybody for anything. 443 00:23:41,793 --> 00:23:45,063 Uh, I mean, I, I wouldn't mind, just for the record, if anybody wants to send me. 444 00:23:47,088 --> 00:23:48,228 Uh, I'm good. 445 00:23:48,288 --> 00:23:52,818 Like, you know, I, I'll take that, but for a large company that lost, you 446 00:23:52,823 --> 00:23:57,118 know, their entire company and this is all they're getting, one interesting 447 00:23:57,118 --> 00:24:03,898 thing that the judge, a, a point that the judge made was that there was no 448 00:24:04,288 --> 00:24:11,188 reason, there was no incentive or, or whatever for OVH to configure backups 449 00:24:11,193 --> 00:24:13,108 in the way that they configured them. 450 00:24:13,468 --> 00:24:18,208 Meaning that they're not a brand new hosting company 451 00:24:18,213 --> 00:24:19,678 that only has one data center. 452 00:24:20,488 --> 00:24:24,478 He's saying you have data centers all over the place and you have clearly 453 00:24:24,478 --> 00:24:26,398 bandwidth between these data centers. 454 00:24:27,508 --> 00:24:32,908 So there was no reason to configure backups the way that you configured them. 455 00:24:32,958 --> 00:24:37,398 You could have just as easily made sure that, that, that the backup server is 456 00:24:37,403 --> 00:24:41,628 always in another data center, but you just couldn't be bothered doing that. 457 00:24:41,628 --> 00:24:46,818 So that, that's just the way the judge saw things and, uh, ruled in the, you know, 458 00:24:46,818 --> 00:24:48,588 in the favor of the, of the plaintiff. 459 00:24:49,218 --> 00:24:52,788 So when, when we look on this, you know. 460 00:24:53,343 --> 00:24:55,893 It's difficult to talk about lessons learned 461 00:24:56,613 --> 00:24:57,273 Prasanna Malaiyandi: Is it though? 462 00:24:57,723 --> 00:25:05,013 W. Curtis Preston: at least, well, because like, like even the customer that 463 00:25:05,373 --> 00:25:08,553 did what we would've said to do, right? 464 00:25:08,558 --> 00:25:10,623 So, you know, number one, review your contract. 465 00:25:11,343 --> 00:25:14,793 And so the customer reviewed that contract and said, Hey, what do you mean 466 00:25:14,793 --> 00:25:16,473 you're keeping in the same data center? 467 00:25:16,473 --> 00:25:17,523 That's nonsense. 468 00:25:17,793 --> 00:25:19,533 You need to put it in another data center. 469 00:25:19,623 --> 00:25:20,703 And then OVH. 470 00:25:21,453 --> 00:25:23,043 Said, sure. 471 00:25:23,223 --> 00:25:24,108 Prasanna Malaiyandi: but so. 472 00:25:24,183 --> 00:25:24,753 W. Curtis Preston: they put it at, 473 00:25:24,893 --> 00:25:28,158 Prasanna Malaiyandi: So, so here's, I think, how they could have done it, which 474 00:25:28,158 --> 00:25:31,128 is by doing DR testing or restore testing, 475 00:25:33,513 --> 00:25:33,993 W. Curtis Preston: Okay. 476 00:25:34,143 --> 00:25:34,683 All right. 477 00:25:35,013 --> 00:25:39,693 So, so trust and trust, but verify the old phrase from Mr. 478 00:25:39,693 --> 00:25:40,323 Reagan there. 479 00:25:40,398 --> 00:25:42,768 Prasanna Malaiyandi: Shoot your production in the head and try to restore your data. 480 00:25:43,893 --> 00:25:45,753 W. Curtis Preston: wait, not like the Alaska, not like. 481 00:25:45,828 --> 00:25:46,098 Prasanna Malaiyandi: Yeah. 482 00:25:46,158 --> 00:25:47,058 Not, yeah, not like 483 00:25:47,148 --> 00:25:47,808 W. Curtis Preston: like that episode. 484 00:25:48,648 --> 00:25:49,068 Okay. 485 00:25:49,488 --> 00:25:54,198 So, so simulate shooting your server in the head and then, 486 00:25:54,258 --> 00:25:56,328 uh, and then trying to restore. 487 00:25:56,808 --> 00:26:02,508 Uh, but well, but, but in, how do they simulate shutting down the data center? 488 00:26:04,863 --> 00:26:09,213 Prasanna Malaiyandi: Or you just prevent any access at a new data center to any 489 00:26:09,633 --> 00:26:11,493 IP addresses in the old data center, 490 00:26:11,508 --> 00:26:11,988 W. Curtis Preston: Like how do 491 00:26:12,088 --> 00:26:13,623 Prasanna Malaiyandi: and then try to do your restore. 492 00:26:14,763 --> 00:26:14,913 So 493 00:26:14,918 --> 00:26:15,318 basically. 494 00:26:15,693 --> 00:26:16,983 W. Curtis Preston: need to verify that. 495 00:26:17,103 --> 00:26:20,163 It's just, it is just that there is a certain degree of trust that 496 00:26:20,163 --> 00:26:24,453 you are placing in the vendor that they're going to do the things 497 00:26:24,453 --> 00:26:25,833 that they say they're doing. 498 00:26:26,463 --> 00:26:30,573 Um, you need to figure out a way to verify that. 499 00:26:30,813 --> 00:26:38,613 But I, I would say that the easier answer here is that, um. 500 00:26:39,243 --> 00:26:41,463 Is to use a third party backup service. 501 00:26:42,303 --> 00:26:49,473 And because while you could contractually, you could get like, 502 00:26:49,743 --> 00:26:51,813 like you can use Amazon backup, right? 503 00:26:51,818 --> 00:26:55,923 And you can snapshot your way to happiness and then you can replicate those 504 00:26:55,923 --> 00:26:57,783 snapshots to another region if you want. 505 00:26:58,473 --> 00:27:02,433 And you could probably figure out some way to verify that those 506 00:27:02,438 --> 00:27:04,533 snapshots are, are in another region. 507 00:27:05,553 --> 00:27:06,903 The um. 508 00:27:07,263 --> 00:27:13,323 But if you use a third party backup service, it's like you, 509 00:27:13,413 --> 00:27:17,403 it's like you're guaranteeing that the backups will be somewhere else 510 00:27:17,463 --> 00:27:18,933 other than what you're backing up. 511 00:27:19,203 --> 00:27:23,403 Because to me that's like the fundamental, you're shaking your 512 00:27:23,408 --> 00:27:24,603 head, I know what you're thinking. 513 00:27:24,603 --> 00:27:26,373 We could we, you can argue it in a minute. 514 00:27:26,553 --> 00:27:30,603 I'm just saying that to me, like the fundamental thing of backup. 515 00:27:30,993 --> 00:27:33,423 Is to make sure the backup is somewhere else. 516 00:27:34,053 --> 00:27:37,833 Uh, so you're thinking that if a customer is running in the same 517 00:27:38,523 --> 00:27:39,483 region, 518 00:27:39,528 --> 00:27:40,458 Prasanna Malaiyandi: No, no, no. 519 00:27:40,458 --> 00:27:41,718 I, I wasn't, I wasn't even, I 520 00:27:41,853 --> 00:27:43,863 W. Curtis Preston: you select what, what were you thinking? 521 00:27:43,908 --> 00:27:48,228 Prasanna Malaiyandi: I was thinking that if I'm a customer and I'm going 522 00:27:48,228 --> 00:27:55,728 to a vendor to offload my need to have infrastructure and to manage all these 523 00:27:55,733 --> 00:28:02,598 resources, am I as a customer going to go pay and find a backup service and manage 524 00:28:02,603 --> 00:28:04,758 a backup service to go deal with this? 525 00:28:05,508 --> 00:28:05,748 Right. 526 00:28:05,748 --> 00:28:10,158 I think that that bar gets pretty high depending on how complex it is. 527 00:28:10,878 --> 00:28:14,418 And because you also need to find a backup service that works with 528 00:28:14,418 --> 00:28:16,428 the cloud provider you're using. 529 00:28:17,118 --> 00:28:17,418 Right. 530 00:28:17,418 --> 00:28:22,488 Which may be harder for the non top three clouds that you have, right? 531 00:28:22,548 --> 00:28:24,828 non-Amazon, Azure, or 532 00:28:25,233 --> 00:28:25,503 W. Curtis Preston: Yeah. 533 00:28:25,508 --> 00:28:25,713 So I. 534 00:28:26,583 --> 00:28:27,273 Okay. 535 00:28:27,273 --> 00:28:29,013 I, I thought you were arguing something different. 536 00:28:29,133 --> 00:28:32,373 I, I guess I would argue that that's the job, 537 00:28:33,918 --> 00:28:34,773 Prasanna Malaiyandi: It, it is. 538 00:28:34,773 --> 00:28:35,313 I agree. 539 00:28:35,313 --> 00:28:36,423 It is a job when you move to 540 00:28:36,513 --> 00:28:39,093 W. Curtis Preston: you're developing infrastructure, if you're developing 541 00:28:39,093 --> 00:28:42,273 infrastructure backup as part of the infrastructure, if you don't 542 00:28:42,278 --> 00:28:46,773 know how to do cloud vendor A without also backing up cloud vendor 543 00:28:46,773 --> 00:28:49,113 A, then you shouldn't use Cloud vendor A, 544 00:28:50,103 --> 00:28:50,463 Prasanna Malaiyandi: Yeah. 545 00:28:50,583 --> 00:28:51,363 I agree with that. 546 00:28:51,933 --> 00:28:52,263 Yeah. 547 00:28:52,533 --> 00:28:54,393 W. Curtis Preston: okay, so what are you saying? 548 00:28:54,483 --> 00:28:56,103 Prasanna Malaiyandi: No, I, I agree. 549 00:28:56,108 --> 00:28:58,473 But people don't really think about backup. 550 00:28:58,593 --> 00:28:58,953 Right. 551 00:28:59,703 --> 00:29:00,903 W. Curtis Preston: Well, that's their problem. 552 00:29:00,963 --> 00:29:03,543 There isn't, that's why we exist, right? 553 00:29:04,203 --> 00:29:05,613 Um, I 554 00:29:05,613 --> 00:29:06,063 I can't 555 00:29:06,078 --> 00:29:06,468 Prasanna Malaiyandi: devil's. 556 00:29:06,468 --> 00:29:07,878 I'm just playing devil's advocate. 557 00:29:07,878 --> 00:29:08,988 You got a thousand things on 558 00:29:09,003 --> 00:29:10,293 W. Curtis Preston: well, you're doing a good job. 559 00:29:10,398 --> 00:29:13,038 Prasanna Malaiyandi: You have a thousand things on your mind and like Right. 560 00:29:13,043 --> 00:29:14,628 Backup is not gonna be your top priority. 561 00:29:16,083 --> 00:29:19,593 W. Curtis Preston: So that, that, that's a problem as old as time. 562 00:29:20,013 --> 00:29:22,113 No one thinks about backup until, you 563 00:29:22,313 --> 00:29:24,933 Prasanna Malaiyandi: but, but I think there is a solution though for this. 564 00:29:26,803 --> 00:29:31,803 Right, and I think the solution is as an industry, we should hold vendors 565 00:29:31,803 --> 00:29:36,273 accountable for providing the bare minimum needed, at least a bare minimum 566 00:29:36,273 --> 00:29:39,933 needed for what is safe backups, right? 567 00:29:39,933 --> 00:29:43,233 If you are saying, I'm doing backups, here's the bare minimum, right? 568 00:29:43,233 --> 00:29:44,613 Follow the 3, 2, 1 rule. 569 00:29:44,613 --> 00:29:45,273 Keep it out. 570 00:29:45,333 --> 00:29:49,683 Separate in a, uh, right, that ensures that your backups are isolated, right? 571 00:29:50,103 --> 00:29:50,733 These sort of things. 572 00:29:50,738 --> 00:29:52,563 I think there should be a bare minimum. 573 00:29:53,388 --> 00:29:55,128 We should hold vendors accountable to. 574 00:29:55,128 --> 00:29:58,578 And until we get to that point, I think customers should ask their vendors, 575 00:29:58,578 --> 00:30:02,898 Hey, here are five questions when I'm going to go use your service, and I put 576 00:30:02,898 --> 00:30:06,468 it out and ask you questions, answer these five questions related to backup, 577 00:30:08,823 --> 00:30:11,493 W. Curtis Preston: I, I think that's a perfectly valid thing to do. 578 00:30:12,063 --> 00:30:16,623 I just know that with, with most of the SaaS services, the answer is no. 579 00:30:16,833 --> 00:30:17,193 Right? 580 00:30:17,388 --> 00:30:18,888 Prasanna Malaiyandi: which is okay, but at least you have the 581 00:30:18,893 --> 00:30:19,488 answers. 582 00:30:20,373 --> 00:30:21,543 And then now you can start to think 583 00:30:21,543 --> 00:30:22,833 about, yeah. 584 00:30:23,013 --> 00:30:27,273 And now you can be like, Hey, maybe if I want to go use say ServiceNow, I should 585 00:30:27,273 --> 00:30:30,003 figure out, okay, how am I gonna back up that data if it's important to me? 586 00:30:30,933 --> 00:30:31,713 Because they don't offer 587 00:30:31,848 --> 00:30:32,358 W. Curtis Preston: Agreed. 588 00:30:33,108 --> 00:30:37,758 And maybe, maybe I'll live in a utopia that where that one day, there's two 589 00:30:37,758 --> 00:30:41,658 products that are really impressive and one of them has backup and the other 590 00:30:41,658 --> 00:30:43,428 one doesn't as part of the service. 591 00:30:43,463 --> 00:30:46,218 And the one with backup wins the 592 00:30:46,368 --> 00:30:46,698 Prasanna Malaiyandi: Yeah. 593 00:30:46,998 --> 00:30:49,638 And then the other one eventually figures out they're losing deals 594 00:30:49,638 --> 00:30:52,308 because they don't have it, and then they start offering it, right? 595 00:30:52,803 --> 00:30:54,873 W. Curtis Preston: That is, that's what we call the free market. 596 00:30:56,253 --> 00:30:59,823 Um, yeah, I, I, I think that's really important. 597 00:31:00,003 --> 00:31:06,993 Um, and I, I just think that, um, this is a really bad story because it's 598 00:31:06,993 --> 00:31:11,013 just because some people who thought they were doing the right thing weren't 599 00:31:11,013 --> 00:31:13,773 actually doing the right thing because the vendor wasn't doing the right thing 600 00:31:13,773 --> 00:31:15,543 and they were actually misrepresenting. 601 00:31:15,993 --> 00:31:17,823 That's the worst part of the story, is that they were actually 602 00:31:17,823 --> 00:31:19,308 misrepresenting what they were doing. 603 00:31:20,538 --> 00:31:22,578 And, um, 604 00:31:22,953 --> 00:31:24,093 Prasanna Malaiyandi: It's like, Hey, I bought a car. 605 00:31:24,093 --> 00:31:26,703 It's supposed to get a hundred miles per gallon, but it only gets 10. 606 00:31:29,768 --> 00:31:30,148 You know? 607 00:31:31,638 --> 00:31:32,388 W. Curtis Preston: It's more like, 608 00:31:32,403 --> 00:31:33,963 Prasanna Malaiyandi: Or, or, Or, or, it's, or 609 00:31:34,098 --> 00:31:35,063 W. Curtis Preston: like it doesn't go at all. 610 00:31:35,073 --> 00:31:35,433 Prasanna Malaiyandi: no, no, no. 611 00:31:35,433 --> 00:31:36,963 I was gonna say, or I have a car. 612 00:31:36,963 --> 00:31:37,473 I bought it. 613 00:31:37,473 --> 00:31:38,673 It has airbags. 614 00:31:38,913 --> 00:31:39,933 I get into a wreck. 615 00:31:39,933 --> 00:31:40,953 The airbags don't deploy. 616 00:31:42,738 --> 00:31:44,148 W. Curtis Preston: Well, it turns out there's no 617 00:31:44,433 --> 00:31:44,643 Prasanna Malaiyandi: Yeah. 618 00:31:44,643 --> 00:31:45,483 There are no airbags. 619 00:31:45,618 --> 00:31:46,938 W. Curtis Preston: just no airbags at all. 620 00:31:47,328 --> 00:31:50,928 And they're like, well, the airbags are only in the cars that are in the factory. 621 00:31:51,318 --> 00:31:54,438 Uh, the demos, we, we didn't put them in 622 00:31:54,573 --> 00:31:57,093 Prasanna Malaiyandi: speaking of this, so I was actually reading a Reddit article 623 00:31:57,098 --> 00:31:59,643 the other day, or a Reddit post and uh. 624 00:32:00,543 --> 00:32:04,203 Someone was complaining and the story goes that they were sleeping, they were 625 00:32:04,203 --> 00:32:07,533 renting a place, they were sleeping, and they heard a beeping and they go look. 626 00:32:07,533 --> 00:32:10,473 And it was like one of those carbon monoxide detectors that they had 627 00:32:10,473 --> 00:32:13,563 bought at a previous rental and brought it, but they never installed it. 628 00:32:14,193 --> 00:32:15,123 They're like, huh, that's weird. 629 00:32:15,128 --> 00:32:18,183 So they open all the windows and then it keeps going off and off. 630 00:32:18,813 --> 00:32:21,873 And then they called a fire department and apparently their fireplace 631 00:32:21,873 --> 00:32:23,313 was leaking carbon monoxide. 632 00:32:24,183 --> 00:32:25,953 So they're like, that's at least it ca. 633 00:32:26,658 --> 00:32:27,438 In their rental. 634 00:32:27,768 --> 00:32:28,518 So they're like, good. 635 00:32:28,518 --> 00:32:29,448 At least it was caught. 636 00:32:29,448 --> 00:32:31,098 And then the fire department goes around. 637 00:32:31,458 --> 00:32:35,388 Apparently all their smoke detectors were just shells. 638 00:32:35,988 --> 00:32:38,753 There was no actual smoke detector inside of 'em. 639 00:32:42,828 --> 00:32:46,728 W. Curtis Preston: So, yeah, well, at least that one is again, that's fixable. 640 00:32:46,758 --> 00:32:50,298 You know that you can go around and test the smoke detectors in an Airbnb you're 641 00:32:50,298 --> 00:32:53,868 saying, or staying, but that's just wrong. 642 00:32:55,083 --> 00:32:59,283 So the person had had a battery powered, 643 00:32:59,373 --> 00:33:00,693 Prasanna Malaiyandi: Carbon monoxide detector? 644 00:33:01,323 --> 00:33:03,513 W. Curtis Preston: carbon monoxide detector in their like luggage or 645 00:33:03,573 --> 00:33:03,843 Prasanna Malaiyandi: No. 646 00:33:04,143 --> 00:33:05,613 So this was a rental, right? 647 00:33:05,613 --> 00:33:08,103 So they were renting like long term, like a year rental. 648 00:33:08,553 --> 00:33:08,793 Right. 649 00:33:08,793 --> 00:33:11,493 And so they had just moved into the place and from their old place, 650 00:33:11,498 --> 00:33:12,993 they had brought whatever they had. 651 00:33:13,233 --> 00:33:16,323 And it just happened to be just sitting in like a closet somewhere. 652 00:33:18,573 --> 00:33:19,143 W. Curtis Preston: Wow. 653 00:33:19,293 --> 00:33:19,713 Saved their 654 00:33:19,893 --> 00:33:20,253 Prasanna Malaiyandi: Yep. 655 00:33:20,313 --> 00:33:21,033 Saved their life. 656 00:33:21,228 --> 00:33:21,348 W. Curtis Preston: I. 657 00:33:22,518 --> 00:33:23,298 Yeah. 658 00:33:23,388 --> 00:33:24,138 Um, 659 00:33:24,303 --> 00:33:27,308 Prasanna Malaiyandi: Go replace your check your smoke detectors, folks. 660 00:33:28,698 --> 00:33:30,798 W. Curtis Preston: your smoke detectors and your carbon monoxide 661 00:33:30,798 --> 00:33:35,838 detectors, um, at which for the record need to be two separate devices. 662 00:33:36,738 --> 00:33:41,418 For a while I was like, this is annoying that, uh, I mean, this is totally 663 00:33:41,418 --> 00:33:45,498 off the subject, but for a while I was, I was saying that CO2 detectors 664 00:33:45,678 --> 00:33:49,608 and co detector, I'm sorry, not co fire detectors or smoke detectors. 665 00:33:50,318 --> 00:33:51,333 It should be combined. 666 00:33:51,333 --> 00:33:54,063 And, and there are companies that sell combined smoke and 667 00:33:54,063 --> 00:33:56,163 fire and, and co detectors. 668 00:33:56,463 --> 00:34:00,993 Do you know why they should absolutely not be in the same place 669 00:34:02,073 --> 00:34:03,093 or the same unit? 670 00:34:03,228 --> 00:34:04,908 Prasanna Malaiyandi: Because they rise differently. 671 00:34:06,453 --> 00:34:06,903 W. Curtis Preston: Yeah. 672 00:34:07,173 --> 00:34:12,153 Smoke goes up, co goes down, so co detectors should be 673 00:34:12,153 --> 00:34:13,563 about a foot off the floor. 674 00:34:14,163 --> 00:34:16,893 Uh, and smoke detectors should be up on the ceiling and they 675 00:34:16,893 --> 00:34:19,743 should never, and they should therefore never be the same device. 676 00:34:20,733 --> 00:34:24,813 Um, and how it's actually legal to sell them in one thing, I don't know. 677 00:34:25,083 --> 00:34:25,473 Anyway. 678 00:34:25,923 --> 00:34:30,093 So, uh, maybe I just scared the crap outta somebody and they're gonna 679 00:34:30,093 --> 00:34:33,758 go Now buy, uh, separate CO two 680 00:34:33,933 --> 00:34:36,213 Prasanna Malaiyandi: you get those plugin ones, which is, which is perfect because 681 00:34:36,218 --> 00:34:37,563 the outlets are at the right height. 682 00:34:39,063 --> 00:34:39,753 W. Curtis Preston: Exactly. 683 00:34:39,813 --> 00:34:40,413 Exactly. 684 00:34:41,523 --> 00:34:42,303 Uh, all right. 685 00:34:42,303 --> 00:34:45,468 Any final thoughts on the OVH Fire? 686 00:34:46,383 --> 00:34:47,643 Prasanna Malaiyandi: Uh, no. 687 00:34:47,733 --> 00:34:51,873 I think, yeah, like you said, it's people were trying to do the right thing or 688 00:34:51,878 --> 00:34:55,203 they assumed that the right thing was being done and they got burned and yeah. 689 00:34:55,683 --> 00:34:58,713 I don't know what else we could recommend for them to do. 690 00:34:58,768 --> 00:34:58,888 W. Curtis Preston: I. 691 00:35:01,653 --> 00:35:06,933 Yeah, verify as much as you can that your backup infrastructure is separate. 692 00:35:06,933 --> 00:35:10,443 And again, I think the easiest way to do that is to use a different vendor. 693 00:35:10,953 --> 00:35:15,513 Um, but you know, maybe it's because I used to work for one of those 694 00:35:15,513 --> 00:35:19,053 companies that there was a different vendor, but I don't anymore. 695 00:35:19,058 --> 00:35:20,073 I still think that way. 696 00:35:20,613 --> 00:35:21,063 Anyway. 697 00:35:21,093 --> 00:35:23,253 Well, uh, thanks for the chat persona. 698 00:35:23,643 --> 00:35:24,133 Prasanna Malaiyandi: anytime. 699 00:35:24,863 --> 00:35:26,148 Thank you, Curtis. 700 00:35:27,558 --> 00:35:29,178 W. Curtis Preston: Thank you to our listeners. 701 00:35:29,268 --> 00:35:31,158 Uh, you know, you're why we do this. 702 00:35:31,398 --> 00:35:32,538 That is a wrap.