1 00:00:00,000 --> 00:00:04,950 This week on the backup wrap-up we dive into the disastrous data loss 2 00:00:04,950 --> 00:00:09,719 event that hit dedoose a cloud-based research data management platform. 3 00:00:09,750 --> 00:00:11,070 Back in 2014. 4 00:00:11,730 --> 00:00:17,189 Dedoose lost access to their primary data and their backups simultaneously 5 00:00:17,580 --> 00:00:21,900 resulting in the loss of over three weeks worth of customer research. 6 00:00:22,380 --> 00:00:27,090 We discussed what caused the crash as well as some questionable backup practices. 7 00:00:27,599 --> 00:00:33,599 These practices resulted in a ton of data loss for doctoral researchers who 8 00:00:33,599 --> 00:00:37,860 lost countless hours of painstaking work, they will never get back. 9 00:00:38,519 --> 00:00:41,730 We discuss what dedoose did right, and what they could have done 10 00:00:41,730 --> 00:00:43,709 better so that we can all learn. 11 00:00:43,833 --> 00:00:47,613 If you're new to the pod, I'm w Curtis Preston, AKA Mr. 12 00:00:47,613 --> 00:00:48,153 Backup. 13 00:00:48,453 --> 00:00:51,573 And there's a reason I'm so passionate about this topic. 14 00:00:51,933 --> 00:00:56,673 When I first started in backups over 30 years ago, my company lost an important 15 00:00:56,673 --> 00:00:58,713 database that I couldn't restore. 16 00:00:59,133 --> 00:01:03,423 Since then I've dedicated my career to making sure that would never again happen 17 00:01:03,423 --> 00:01:05,823 to me or anyone who will listen to me. 18 00:01:06,333 --> 00:01:10,473 We turn unappreciated backup admins into cyber recovery heroes. 19 00:01:10,773 --> 00:01:12,903 This is the backup wrap-up. 20 00:01:12,903 --> 00:01:13,473 Welcome to the show. 21 00:01:27,384 --> 00:01:28,404 W. Curtis Preston: Welcome to the show. 22 00:01:28,854 --> 00:01:31,524 I'm your host, w Curtis Preston, AKA, Mr. 23 00:01:31,524 --> 00:01:38,354 Backup, and I have with me my tax non-ad advisor, Prasanna Malaiyandi. 24 00:01:38,544 --> 00:01:39,324 How's it going? 25 00:01:39,654 --> 00:01:42,234 I, I've needed you a lot over the last few days is 26 00:01:43,269 --> 00:01:45,819 Prasanna Malaiyandi: It is that time of the year, Curtis, you know, 27 00:01:45,819 --> 00:01:47,529 I wouldn't expect anything else. 28 00:01:47,529 --> 00:01:50,794 For, for our international listeners who don't know , april 29 00:01:50,794 --> 00:01:56,114 15th IS a US' day that you have to file your yearly tax income tax 30 00:01:56,564 --> 00:01:58,064 with the government, with the IRS 31 00:01:58,064 --> 00:02:01,334 W. Curtis Preston: You can file an extension, but any money that you 32 00:02:01,334 --> 00:02:02,954 owe has to be paid by that time. 33 00:02:03,314 --> 00:02:08,504 This was, I'm gonna say one of the, if not the most complicated years that 34 00:02:08,504 --> 00:02:11,954 I've ever done tax, because halfway through the year I was laid off. 35 00:02:12,469 --> 00:02:16,069 And so then I went, you know, self-employment and I went like nine 36 00:02:16,069 --> 00:02:17,659 different ways of self-employment. 37 00:02:18,229 --> 00:02:23,124 And um, you know, and I had, I had some carryovers from previous years and I had 38 00:02:23,124 --> 00:02:27,139 some, I had some of this, some of that, and I'm like, I think I'm gonna be okay. 39 00:02:27,229 --> 00:02:28,939 I might not be okay. 40 00:02:29,089 --> 00:02:31,819 Turned out I was way okay. 41 00:02:32,359 --> 00:02:33,679 I was very okay. 42 00:02:33,679 --> 00:02:38,839 But you may recall that there was a day there or so where I was not having any. 43 00:02:38,864 --> 00:02:39,854 Prasanna Malaiyandi: I recall that. 44 00:02:40,004 --> 00:02:42,974 And, and for people who may not understand the complexities of the US 45 00:02:42,974 --> 00:02:46,424 tax system, they're probably thinking, oh, don't you just like take your 46 00:02:46,424 --> 00:02:50,114 income and they just take a check for based on a certain percentage. 47 00:02:50,114 --> 00:02:50,504 Yeah. 48 00:02:50,504 --> 00:02:52,154 That's not how the US system works. 49 00:02:52,559 --> 00:02:53,039 W. Curtis Preston: Yeah. 50 00:02:53,099 --> 00:02:57,479 And well, and then when you add in the complexities of, um, some of the 51 00:02:57,479 --> 00:03:01,169 different types of self-employment that I did, there's some very specific 52 00:03:01,169 --> 00:03:04,679 deductions that you have to take and then you have to really record stuff 53 00:03:04,679 --> 00:03:06,389 to, to, to make those deductions. 54 00:03:06,389 --> 00:03:08,429 And, and, uh, there was, um. 55 00:03:09,404 --> 00:03:10,094 It was just, 56 00:03:10,144 --> 00:03:12,754 Prasanna Malaiyandi: I remember the picture you sent of just like, Hey, here 57 00:03:12,754 --> 00:03:15,694 are the things I have to be tracking in order to be able to do this deduction. 58 00:03:15,694 --> 00:03:19,114 I was like, oh my gosh, Curtis, there must be a better way to 59 00:03:19,189 --> 00:03:19,729 W. Curtis Preston: Yeah. 60 00:03:19,729 --> 00:03:23,659 I, and, and I, I will say I did figure out a better way for that one. 61 00:03:23,659 --> 00:03:26,809 I've only done that, that particular one for like seven years. 62 00:03:27,289 --> 00:03:29,989 And, uh, I never had figured out a better way. 63 00:03:29,989 --> 00:03:32,599 And actually this year, for the first time, I figured out a better way. 64 00:03:32,734 --> 00:03:34,294 Prasanna Malaiyandi: So 2024 will be better for 65 00:03:34,339 --> 00:03:35,239 W. Curtis Preston: 2024. 66 00:03:37,834 --> 00:03:42,184 And also, uh, I, you know, uh, you helped me, uh, you know, 67 00:03:42,184 --> 00:03:43,324 come up with a spreadsheet. 68 00:03:43,384 --> 00:03:45,574 I mean, I did it on my own, but, you know, there was some, 69 00:03:45,844 --> 00:03:47,224 definitely some advice in there. 70 00:03:47,229 --> 00:03:52,324 And man, you're right, the US tax code is super complicated. 71 00:03:52,954 --> 00:03:57,034 Um, you know, because, you know, we have a progressive tax system, so like 72 00:03:57,034 --> 00:04:01,294 they tax this percentage on that amount and this percentage on that amount and, 73 00:04:01,624 --> 00:04:04,234 and, uh, so that, so it's not just. 74 00:04:05,134 --> 00:04:07,744 Figure out how much you made and take X percentage of it. 75 00:04:07,744 --> 00:04:09,604 It is so not that. 76 00:04:09,784 --> 00:04:12,754 And of course, I live in California and we have a state income tax, 77 00:04:12,754 --> 00:04:14,164 so we have that in addition. 78 00:04:14,464 --> 00:04:18,574 And just in my tax bracket from the lowest, you know, there 79 00:04:18,579 --> 00:04:19,954 are, what, what did we find it? 80 00:04:19,954 --> 00:04:22,984 Six, six different layers of that. 81 00:04:23,134 --> 00:04:29,644 It starts at 1%, then 2%, 3%, 4%, you know, all the way up to like 9%. 82 00:04:29,674 --> 00:04:32,224 I think, um, you know, once you go. 83 00:04:32,269 --> 00:04:33,024 Prasanna Malaiyandi: 12.3. 84 00:04:33,174 --> 00:04:33,464 Yeah. 85 00:04:33,884 --> 00:04:34,104 Is 86 00:04:34,189 --> 00:04:34,819 W. Curtis Preston: Oh, well, no. 87 00:04:34,819 --> 00:04:37,999 Well, I, well, I know it goes past that, but like, but like, you know, 88 00:04:37,999 --> 00:04:42,799 if you make over a certain amount, you end up paying 9% on that final, uh, 89 00:04:42,799 --> 00:04:46,459 on the, on the marginal, that's what's called the marginal tax rate, uh, 90 00:04:47,629 --> 00:04:49,279 Prasanna Malaiyandi: So it's like every dollar you make 91 00:04:49,279 --> 00:04:50,989 additional gets taxed at that rate. 92 00:04:50,989 --> 00:04:51,409 And yeah. 93 00:04:51,769 --> 00:04:54,649 And then the other thing challenge is you have kids. 94 00:04:54,654 --> 00:04:58,159 They have different deductions, credits and deductions change 95 00:04:58,159 --> 00:05:02,059 every year, depending on what gets passed, what doesn't get passed. 96 00:05:02,299 --> 00:05:05,779 Your brackets don't stay fixed because some are adjusted based on the rate 97 00:05:05,779 --> 00:05:09,559 of inflation, but some are based on one inflation rate, others is 98 00:05:09,559 --> 00:05:10,969 based on a different inflation rate. 99 00:05:10,969 --> 00:05:12,739 So it's a giant mess. 100 00:05:13,099 --> 00:05:16,159 You basically need a PhD in order to be able to do this stuff and keep up with 101 00:05:16,174 --> 00:05:16,714 W. Curtis Preston: Yeah. 102 00:05:16,924 --> 00:05:19,954 Yeah, but through, but through it all, I have contacted my 103 00:05:19,954 --> 00:05:21,184 financial non-ad advisor. 104 00:05:21,684 --> 00:05:22,624 Prasanna Malaiyandi 105 00:05:26,139 --> 00:05:27,879 Prasanna Malaiyandi: let's just say you are not the only person 106 00:05:27,879 --> 00:05:30,309 who talks to me about, uh, taxes. 107 00:05:31,224 --> 00:05:32,124 W. Curtis Preston: Is that right? 108 00:05:32,274 --> 00:05:33,924 I, that, that does not surprise me. 109 00:05:33,929 --> 00:05:36,984 'cause you're a, you're a smart, you're a smart little person. 110 00:05:37,684 --> 00:05:40,239 Prasanna Malaiyandi: It's just random bits of knowledge that I have in my head. 111 00:05:40,854 --> 00:05:41,094 W. Curtis Preston: yeah. 112 00:05:41,094 --> 00:05:44,964 I love, I love how you can just, you can just quote the marginal tax rates for 113 00:05:44,969 --> 00:05:46,914 different, for different levels of income. 114 00:05:48,939 --> 00:05:50,319 Um, yeah. 115 00:05:51,099 --> 00:05:51,519 Yeah. 116 00:05:51,579 --> 00:05:54,009 And you're like, oh, yeah, with the self-employment tax, you know, 117 00:05:54,009 --> 00:05:55,269 you could deduct half of it, right? 118 00:05:55,719 --> 00:05:56,829 I'm like, no, I didn't know that. 119 00:05:58,929 --> 00:05:59,319 Yeah. 120 00:05:59,649 --> 00:06:03,369 And of course, I, I bet a significant portion of our listeners go, yeah, 121 00:06:03,369 --> 00:06:05,679 no, that's why I have a guy, right? 122 00:06:05,739 --> 00:06:11,649 I just give the money to the guy and the guy just tells me, um, you know, uh, yeah. 123 00:06:12,339 --> 00:06:12,909 Um. 124 00:06:13,089 --> 00:06:14,769 Prasanna Malaiyandi: useful to learn, right? 125 00:06:14,769 --> 00:06:17,169 Because then you could figure out the impact of certain things that 126 00:06:17,169 --> 00:06:20,829 you're gonna be doing or like how you should be withholding. 127 00:06:20,829 --> 00:06:24,399 And honestly, like it's great to have a guy, but the guy 128 00:06:24,399 --> 00:06:26,649 charges money and it's expensive 129 00:06:26,654 --> 00:06:27,309 W. Curtis Preston: charge money. 130 00:06:27,399 --> 00:06:27,819 Yeah. 131 00:06:27,969 --> 00:06:30,279 I, I think one meeting with the guy, and by the way, it 132 00:06:30,279 --> 00:06:31,419 could also be a girl, right? 133 00:06:31,479 --> 00:06:37,119 Just, you know, one meeting with the guy and you, uh, you've spent 134 00:06:37,119 --> 00:06:40,989 like the entire thing of just what I spent with TurboTax for the year. 135 00:06:41,409 --> 00:06:43,029 And that's just one meeting, right? 136 00:06:43,329 --> 00:06:45,069 Um, yeah. 137 00:06:45,339 --> 00:06:45,699 So. 138 00:06:46,749 --> 00:06:50,829 My guy's free, except he is not allowed to be my guy. 139 00:06:51,009 --> 00:06:55,629 He's, he is my non-ad advisor, legally is not giving me financial advice. 140 00:06:56,079 --> 00:07:00,429 Um, yeah, our financial advice is get a guy. 141 00:07:02,874 --> 00:07:04,869 Um, anyway. 142 00:07:05,109 --> 00:07:06,189 Prasanna Malaiyandi: Well, at least it's done. 143 00:07:06,594 --> 00:07:07,044 W. Curtis Preston: At least. 144 00:07:07,044 --> 00:07:10,254 Yeah, it is done both last year and this year. 145 00:07:10,254 --> 00:07:15,864 I have filed my estimated taxes for Q1, like a week early. 146 00:07:16,674 --> 00:07:16,734 I 147 00:07:16,869 --> 00:07:17,889 Prasanna Malaiyandi: Look at that, Curtis. 148 00:07:17,889 --> 00:07:21,309 And I'm hoping that a lot of people who have complicated 149 00:07:21,309 --> 00:07:23,049 taxes don't wait till the end. 150 00:07:23,499 --> 00:07:23,889 W. Curtis Preston: Yeah. 151 00:07:24,249 --> 00:07:24,699 Yeah. 152 00:07:24,749 --> 00:07:28,404 I'm just so super happy to be done with my taxes, both for the last 153 00:07:28,404 --> 00:07:29,814 year and for this, this quarter. 154 00:07:29,814 --> 00:07:31,494 I'm so super happy, super excited. 155 00:07:32,244 --> 00:07:35,814 Uh, but we're gonna talk about something not so exciting. 156 00:07:35,814 --> 00:07:41,064 We have yet another episode of, you know, cloud disasters and this one. 157 00:07:41,859 --> 00:07:42,219 This 158 00:07:42,459 --> 00:07:43,149 Prasanna Malaiyandi: I don't know about you. 159 00:07:43,814 --> 00:07:44,944 W. Curtis Preston: what, go ahead. 160 00:07:45,009 --> 00:07:47,379 Prasanna Malaiyandi: I don't know about you, but I've kind of enjoyed this series. 161 00:07:47,379 --> 00:07:51,939 I know it's sort of taking joy in other people's misery, but 162 00:07:51,939 --> 00:07:54,594 I've found it to be kind of, I. 163 00:07:54,669 --> 00:07:56,019 Like interesting, right? 164 00:07:56,019 --> 00:07:58,539 To kind of look and figure out, hey, what did people do? 165 00:07:58,539 --> 00:07:59,529 What happened? 166 00:07:59,919 --> 00:08:03,639 Because these are things that could happen if you do something wrong or 167 00:08:03,669 --> 00:08:06,759 misconfigure something, or maybe it's not even your fault and you just ended up 168 00:08:06,764 --> 00:08:09,939 getting attacked by a malicious user and 169 00:08:10,989 --> 00:08:11,559 W. Curtis Preston: Yeah, I, I. 170 00:08:12,384 --> 00:08:17,334 I will say that this story, there's one particular part in the middle of it that 171 00:08:17,334 --> 00:08:24,564 is the crux of what happened that is so bad backup design that we're gonna, 172 00:08:24,564 --> 00:08:26,094 we're gonna get to right at the core. 173 00:08:26,099 --> 00:08:29,904 At the core, they had some other mistakes that, you know, the, the same mistakes 174 00:08:29,909 --> 00:08:34,104 that everybody else makes and, um, and, and sadly it appears that this was 175 00:08:34,104 --> 00:08:35,514 one where they stepped on their own. 176 00:08:35,624 --> 00:08:37,484 Toe or whatever, right? 177 00:08:37,544 --> 00:08:39,644 Um, but this was not an attack. 178 00:08:40,004 --> 00:08:43,784 This was just, uh, you know, some of the headlines, you know, use phrases 179 00:08:43,784 --> 00:08:45,404 like fat fingering and stuff like that. 180 00:08:45,824 --> 00:08:50,504 Uh, we are of course talking about a company who I'd never even heard of prior 181 00:08:50,504 --> 00:08:53,174 to this story, and it's called Dedoose. 182 00:08:53,409 --> 00:08:58,629 At least I think that's how it's pronounced because they are a, um, 183 00:08:59,019 --> 00:09:00,309 uh, you know, what do they describe? 184 00:09:00,309 --> 00:09:04,449 They're, they're a cloud-based, qualitative and mixed methods research, 185 00:09:04,689 --> 00:09:07,779 data management and analysis application. 186 00:09:07,989 --> 00:09:09,069 That's a mouthful. 187 00:09:09,069 --> 00:09:09,099 I. 188 00:09:09,579 --> 00:09:12,999 And they were in Manhattan Beach, California, which for the record is 189 00:09:12,999 --> 00:09:16,389 right up the road from me and right down the road from our friend, uh, 190 00:09:16,419 --> 00:09:19,929 Jeff Rochlin, uh, that's literally like right next door to him. 191 00:09:20,439 --> 00:09:23,679 And they, they were designed to help researchers organize, analyze, 192 00:09:23,679 --> 00:09:27,819 and collaborate on qualitative and mixed methods research data. 193 00:09:28,569 --> 00:09:33,859 And one particular group of people that used this platform a lot were people 194 00:09:33,859 --> 00:09:36,799 doing their doctoral dissertation. 195 00:09:36,799 --> 00:09:36,859 I. 196 00:09:37,649 --> 00:09:38,579 And so this is gonna 197 00:09:38,659 --> 00:09:39,919 Prasanna Malaiyandi: a bunch of research. 198 00:09:40,079 --> 00:09:44,039 W. Curtis Preston: bunch of research and that, that you're just getting 199 00:09:44,039 --> 00:09:47,789 ready to file to, to, to, you know, to what, what do you call, do you, 200 00:09:47,789 --> 00:09:49,679 what do you do with your dissertation? 201 00:09:49,684 --> 00:09:50,939 You file it, you. 202 00:09:51,709 --> 00:09:52,429 You defend. 203 00:09:52,429 --> 00:09:52,849 Yeah. 204 00:09:53,209 --> 00:09:54,829 Um, present and defend. 205 00:09:54,829 --> 00:09:55,189 Right. 206 00:09:55,249 --> 00:09:58,069 Um, and you're just at that moment where you're about to do that. 207 00:09:58,759 --> 00:10:04,729 And, um, the um, uh, and, and this, then this bad thing happened. 208 00:10:04,729 --> 00:10:08,839 So they were, they had some competitors, uh, I'll just finish this part. 209 00:10:09,049 --> 00:10:12,434 They had some competitors, Atlas Ti NVivo. 210 00:10:13,099 --> 00:10:17,599 And Max does, they all have these, these interesting names, but they were 211 00:10:17,599 --> 00:10:24,499 apparently the only SaaS based, um, you know, web-based, uh, portal, which made 212 00:10:24,499 --> 00:10:28,429 them the only game in town for this, for doing it the way that they were doing it. 213 00:10:29,074 --> 00:10:29,344 Prasanna Malaiyandi: Yeah. 214 00:10:29,464 --> 00:10:32,014 And my guess is people, like if you're an academic, you're probably 215 00:10:32,014 --> 00:10:34,564 like, yeah, I'll just throw it into spreadsheets or throw it into Microsoft 216 00:10:34,564 --> 00:10:37,384 Access and just run it locally. 217 00:10:37,714 --> 00:10:40,594 Um, but that doesn't really scale. 218 00:10:40,594 --> 00:10:42,784 And then you have to manage it, and then you have to worry about how 219 00:10:42,784 --> 00:10:47,284 do I protect it and back it up, and what happens if my laptop crashes? 220 00:10:47,289 --> 00:10:50,524 How do I sync it over to OneDrive or wherever else it has to go? 221 00:10:50,524 --> 00:10:53,824 And yeah, so I could see the appeal of this company. 222 00:10:54,649 --> 00:10:55,669 W. Curtis Preston: Yeah, absolutely. 223 00:10:55,949 --> 00:10:57,269 Prasanna Malaiyandi: You mentioned it was a cloud provider. 224 00:10:57,269 --> 00:10:59,759 They are hosted on top of Azure, 225 00:11:00,239 --> 00:11:02,119 W. Curtis Preston: Yes, they are hosted on Azure. 226 00:11:02,449 --> 00:11:06,229 And this was one of the reasons that we wanted to cover this because we were 227 00:11:06,229 --> 00:11:10,759 trying to get a story from each of the major cloud providers, and this was the 228 00:11:10,759 --> 00:11:12,799 biggest story that we could find on Azure. 229 00:11:13,129 --> 00:11:16,639 But I don't know if the fact that they were on Azure was 230 00:11:16,639 --> 00:11:18,019 really part of the story. 231 00:11:18,259 --> 00:11:20,719 It almost was as we're gonna cover it in the story. 232 00:11:20,719 --> 00:11:24,349 But, um, but in the end, I think this was all them. 233 00:11:24,349 --> 00:11:25,249 Prasanna Malaiyandi: Actually, I take it back. 234 00:11:25,249 --> 00:11:26,899 It's Azure, not Azure. 235 00:11:27,334 --> 00:11:30,034 W. Curtis Preston: Oh, it depends on who you ask. 236 00:11:30,574 --> 00:11:34,324 Um, we could go on, we could go on Microsoft copilot and 237 00:11:34,324 --> 00:11:35,404 ask them how to pronounce it. 238 00:11:36,194 --> 00:11:38,269 Do you want to describe what, what actually happened? 239 00:11:38,687 --> 00:11:41,567 Prasanna Malaiyandi: Yeah, so everything was going fine, right? 240 00:11:41,567 --> 00:11:45,977 They had the service up and running, and then sometime in 2014 241 00:11:45,982 --> 00:11:49,007 they encountered an event, right? 242 00:11:49,007 --> 00:11:51,557 And it was basically a failure of a service. 243 00:11:52,382 --> 00:11:56,042 Running on top of Azure, which initially they said was a Microsoft service that 244 00:11:56,042 --> 00:11:59,612 had an issue, but turns out it was just their service that they had been 245 00:11:59,617 --> 00:12:04,352 operating that just leverages Microsoft component infrastructure to run on. 246 00:12:04,647 --> 00:12:04,997 W. Curtis Preston: right. 247 00:12:05,312 --> 00:12:10,682 Prasanna Malaiyandi: And so, uh, what ended up happening is that failed and they 248 00:12:10,687 --> 00:12:17,132 basically lost access to their databases, which hosted the data for the end users. 249 00:12:17,582 --> 00:12:20,822 They also happened to lose access to their data itself and 250 00:12:20,822 --> 00:12:23,283 the backups at the same time. 251 00:12:25,187 --> 00:12:28,422 And so a lot of bad things happened all at once. 252 00:12:28,962 --> 00:12:29,532 W. Curtis Preston: Yeah. 253 00:12:29,892 --> 00:12:31,692 I mean, this is why we back up, right? 254 00:12:32,357 --> 00:12:32,707 Right. 255 00:12:32,772 --> 00:12:34,062 Because bad things happen. 256 00:12:34,512 --> 00:12:40,422 Um, what I, what I don't, just don't fundamentally understand is 257 00:12:40,422 --> 00:12:44,147 they go, they describe it as we were in the middle of our database 258 00:12:44,147 --> 00:12:46,157 encryption and backup process. 259 00:12:46,577 --> 00:12:48,437 What does that mean to you? 260 00:12:50,972 --> 00:12:55,292 Prasanna Malaiyandi: I read that as they're encrypting the database 261 00:12:55,432 --> 00:12:57,317 before they write out a backup. 262 00:12:58,937 --> 00:12:59,567 W. Curtis Preston: Right. 263 00:12:59,957 --> 00:13:00,197 But 264 00:13:00,227 --> 00:13:00,977 Prasanna Malaiyandi: And so my, 265 00:13:01,067 --> 00:13:01,997 W. Curtis Preston: place 266 00:13:03,047 --> 00:13:03,677 Prasanna Malaiyandi: no, I 267 00:13:03,892 --> 00:13:04,817 W. Curtis Preston: as an event. 268 00:13:07,247 --> 00:13:09,767 Prasanna Malaiyandi: my guess is, well, it's almost like streaming, right? 269 00:13:09,767 --> 00:13:12,737 So I think as they're reading data out, they're encrypting it and writing 270 00:13:12,782 --> 00:13:13,262 W. Curtis Preston: Okay. 271 00:13:13,802 --> 00:13:14,312 Okay. 272 00:13:15,002 --> 00:13:20,522 It's just that it's just that the event damaged the database. 273 00:13:20,882 --> 00:13:25,772 That's why, that's what I don't understand is why would a backup 274 00:13:25,772 --> 00:13:27,752 event damage the database? 275 00:13:28,592 --> 00:13:30,782 Prasanna Malaiyandi: Well, it's like the old days of Oracle, right? 276 00:13:31,022 --> 00:13:33,752 And I know you've been in this situation so many times, right, Curtis, where 277 00:13:33,752 --> 00:13:37,952 it's like, Hey, you did a backup, you took down my production database, right? 278 00:13:37,952 --> 00:13:39,692 They might have been an issue. 279 00:13:39,752 --> 00:13:41,852 And I know that in all your scenarios, Curtis, that's 280 00:13:41,852 --> 00:13:44,012 always been not the case, right? 281 00:13:44,222 --> 00:13:44,642 W. Curtis Preston: Right. 282 00:13:44,942 --> 00:13:46,682 Prasanna Malaiyandi: That backup doesn't take down production, right? 283 00:13:46,682 --> 00:13:49,502 Maybe they just didn't code for it properly, or maybe there 284 00:13:49,502 --> 00:13:51,632 was just some race condition, 285 00:13:52,532 --> 00:13:53,012 W. Curtis Preston: Yeah. 286 00:13:53,132 --> 00:13:55,442 Prasanna Malaiyandi: some reason while they were doing a backup. 287 00:13:55,877 --> 00:13:57,887 Our production database also had a failure. 288 00:13:58,427 --> 00:13:58,937 W. Curtis Preston: Right. 289 00:13:58,987 --> 00:14:01,777 And I think this was good that they were encrypting the backup, but 290 00:14:01,807 --> 00:14:06,997 based on what I'm hearing, what I'm inferring from this is that this was 291 00:14:06,997 --> 00:14:09,337 a homegrown backup system, right? 292 00:14:09,397 --> 00:14:10,747 Because they, they, they seem to. 293 00:14:12,232 --> 00:14:14,512 Make a big deal about the fact that they're encrypting and, and 294 00:14:14,517 --> 00:14:16,942 then, and then, you know, and then backing it up, it's like, okay, 295 00:14:16,942 --> 00:14:18,352 well that's sort of standard. 296 00:14:18,562 --> 00:14:21,322 You know, that's SOP for most people, right? 297 00:14:21,862 --> 00:14:28,912 But, um, so they had a failure of the primary database as they were backing 298 00:14:28,912 --> 00:14:32,512 it up, they were backing up and this, this was the crucial part that I 299 00:14:32,512 --> 00:14:34,162 was talking about earlier, is that. 300 00:14:34,882 --> 00:14:38,992 It's apparently they didn't have enough space or whatever that they 301 00:14:38,992 --> 00:14:45,052 ba they're overriding the last good backup of the database as they're 302 00:14:45,057 --> 00:14:52,912 backing it up and slash or they're only backing up like once a month because, 303 00:14:53,122 --> 00:14:54,562 Prasanna Malaiyandi: a bunch of incrementals 304 00:14:54,947 --> 00:14:57,472 W. Curtis Preston: well, but if they were, but if they were doing incrementals, 305 00:14:57,472 --> 00:15:01,852 they would've been able to restore to a better point in time because. 306 00:15:01,867 --> 00:15:03,787 Prasanna Malaiyandi: So maybe we should talk about that point in time that they 307 00:15:03,787 --> 00:15:05,347 actually were able to restore four. 308 00:15:05,542 --> 00:15:05,752 W. Curtis Preston: Yeah. 309 00:15:05,752 --> 00:15:07,702 So after, after, yeah. 310 00:15:07,702 --> 00:15:11,122 I guess we, we kind of jumped ahead, so we're doing the research afterwards. 311 00:15:11,302 --> 00:15:11,482 Yeah. 312 00:15:11,482 --> 00:15:13,102 So, um, yeah. 313 00:15:13,102 --> 00:15:15,412 So let's back up to, you talked about the event. 314 00:15:16,327 --> 00:15:21,517 So after that event, they, they went out and they did what we tell people to do. 315 00:15:21,517 --> 00:15:26,107 They put out a blog post and, uh, they're like, here's the situation. 316 00:15:26,437 --> 00:15:33,209 Uh, we believe we can go back to, this was May of 2014 . We know for a fact 317 00:15:33,209 --> 00:15:35,519 we can, that the March backup is good. 318 00:15:36,059 --> 00:15:39,149 We think we can also restore the April backup. 319 00:15:39,809 --> 00:15:44,639 We unfortunately believe that we are going to be unable to restore. 320 00:15:44,954 --> 00:15:46,664 The, the, the May backup. 321 00:15:47,084 --> 00:15:50,564 And so every, all of the changes and anything that you put in the system for 322 00:15:50,564 --> 00:15:53,024 the last two to three weeks will be gone. 323 00:15:53,564 --> 00:15:57,284 So I am, I am inferring from that, that they only backed up once a 324 00:15:57,284 --> 00:16:00,944 month because otherwise they would've been able to get a lot closer 325 00:16:00,949 --> 00:16:03,709 Prasanna Malaiyandi: and it wasn't just the database, like I was reading Reddit 326 00:16:03,709 --> 00:16:07,489 posts where they're like, Hey, we talked to a service person or a support person. 327 00:16:07,489 --> 00:16:11,229 They basically said if you created a new account, even in that May 328 00:16:11,229 --> 00:16:12,909 timeframe, you're basically out of luck. 329 00:16:13,659 --> 00:16:13,869 Right. 330 00:16:13,869 --> 00:16:14,169 So it 331 00:16:14,229 --> 00:16:16,239 W. Curtis Preston: Like the entire application, right? 332 00:16:16,569 --> 00:16:19,359 So it's possible that they weren't overriding the last good backup. 333 00:16:19,359 --> 00:16:22,729 It's just that the last good backup was from over a month ago. 334 00:16:23,639 --> 00:16:26,099 Prasanna Malaiyandi: Or it could be that like their more 335 00:16:26,099 --> 00:16:27,599 recent backups were corrupt. 336 00:16:27,765 --> 00:16:29,865 W. Curtis Preston: Uh, well, we don't have any information to suggest that, 337 00:16:29,865 --> 00:16:30,915 Prasanna Malaiyandi: don't have any information. 338 00:16:30,915 --> 00:16:31,245 Yeah. 339 00:16:31,245 --> 00:16:31,485 But 340 00:16:31,605 --> 00:16:36,045 W. Curtis Preston: I I also don't understand why if it, if all backups 341 00:16:36,050 --> 00:16:39,765 were equal, why didn't they immediately try restoring the one from April? 342 00:16:40,605 --> 00:16:44,445 They, they, they said, we know for a fact we can get the one from March. 343 00:16:44,535 --> 00:16:45,915 We think we're gonna get to one. 344 00:16:45,975 --> 00:16:49,845 And they go and, and, and they made a comment about, you know, we realize 345 00:16:49,845 --> 00:16:53,145 that this is, it reminds me a little bit of the Rackspace event where. 346 00:16:53,760 --> 00:16:57,330 What we can do, the quickest is we can get the March one up and running. 347 00:16:57,690 --> 00:17:01,830 Why wasn't the April one also just as good as the March one? 348 00:17:01,860 --> 00:17:02,220 Right. 349 00:17:02,850 --> 00:17:05,640 So I'm not, I'm not sure what happened there, but there was something that 350 00:17:05,645 --> 00:17:08,430 was not as good about the April backup. 351 00:17:09,000 --> 00:17:12,420 Uh, but, but it did appear that they eventually were able to get 352 00:17:12,450 --> 00:17:16,350 that up, but they lost two to three weeks worth of, of effort. 353 00:17:16,575 --> 00:17:16,845 Prasanna Malaiyandi: Yeah. 354 00:17:17,205 --> 00:17:17,415 Yeah. 355 00:17:17,415 --> 00:17:20,775 And even for the April backup, they didn't actually restore it. 356 00:17:20,775 --> 00:17:23,805 So using the way back machine right, we were able to go back and 357 00:17:23,810 --> 00:17:25,185 find the original blog post for 358 00:17:25,230 --> 00:17:25,650 W. Curtis Preston: Right. 359 00:17:26,940 --> 00:17:28,480 Why did we have do that Prasanna? 360 00:17:28,500 --> 00:17:31,530 Why did we have to use the, the, the internet archive? 361 00:17:32,505 --> 00:17:36,525 Prasanna Malaiyandi: because the company took down that post and created a 362 00:17:36,525 --> 00:17:42,345 new post at that same, for that same date, that shed different light on 363 00:17:42,350 --> 00:17:44,385 what had happened and their steps. 364 00:17:45,825 --> 00:17:46,185 W. Curtis Preston: Interesting, right? 365 00:17:47,125 --> 00:17:47,345 Um, 366 00:17:47,985 --> 00:17:52,755 Prasanna Malaiyandi: so, so from that or may, uh, blog post, they basically 367 00:17:52,755 --> 00:17:56,205 said, or soon thereafter, right, I think it was a couple weeks later, they 368 00:17:56,205 --> 00:18:00,315 were like, Hey, we're able to get you back up and running for your March and 369 00:18:00,315 --> 00:18:03,825 for April, we were able to restore the data, but they didn't wanna merge the 370 00:18:03,825 --> 00:18:05,595 data back into the original database. 371 00:18:05,805 --> 00:18:09,285 So they actually created a staging site where people had to go to manually 372 00:18:09,285 --> 00:18:12,170 in order to be able to pull down their data if they want to access it. 373 00:18:12,540 --> 00:18:15,510 W. Curtis Preston: this is so much like the Rackspace event, right? 374 00:18:15,990 --> 00:18:21,030 And so what that, what that tells me, you know, and again, we're, we're, we're not 375 00:18:21,030 --> 00:18:24,330 taking joy in this, but what that tells me is that they didn't have a regular 376 00:18:24,330 --> 00:18:28,110 tested procedure because they didn't, they didn't know that the April one was good. 377 00:18:28,380 --> 00:18:30,990 If they knew the April one was good, they would've just restored the April 378 00:18:30,990 --> 00:18:32,700 one, because by restoring the March one. 379 00:18:33,665 --> 00:18:36,245 They may, you know, you couldn't just do an incremental restore 380 00:18:36,245 --> 00:18:37,775 on top of the March database. 381 00:18:38,225 --> 00:18:41,675 So, um, they created this extra work for 382 00:18:41,675 --> 00:18:42,305 themselves and their 383 00:18:42,825 --> 00:18:45,225 Prasanna Malaiyandi: and especially their entire thing was we want to get people 384 00:18:45,225 --> 00:18:46,965 up and running as quickly as possible. 385 00:18:47,445 --> 00:18:47,655 Right. 386 00:18:47,655 --> 00:18:48,735 That's why they did the March one. 387 00:18:48,735 --> 00:18:52,695 But like you said, Curtis, if April was good, they should have just used April and 388 00:18:53,025 --> 00:18:54,435 it would've been the same amount of time. 389 00:18:55,110 --> 00:18:57,600 W. Curtis Preston: There had to be something up with the April one. 390 00:18:57,600 --> 00:19:00,060 You know, you could, you'd sort of have to read between the lines. 391 00:19:00,150 --> 00:19:03,030 Um, there was something up with the April one and then, 392 00:19:03,510 --> 00:19:04,980 we've talked about this before. 393 00:19:05,820 --> 00:19:09,840 We urge people not to do homegrown backup and when I said this last time, 394 00:19:10,500 --> 00:19:12,840 you said sometimes you have no choice. 395 00:19:13,455 --> 00:19:13,755 Prasanna Malaiyandi: Yeah, 396 00:19:14,130 --> 00:19:16,350 W. Curtis Preston: I would like to, I would like to. 397 00:19:17,145 --> 00:19:17,865 Prasanna Malaiyandi: say it again. 398 00:19:18,345 --> 00:19:21,915 W. Curtis Preston: Well, I'm, say it again, but I would like to point out the 399 00:19:21,915 --> 00:19:30,105 fact that they were running on Azure, and Azure has backup features built into it. 400 00:19:30,525 --> 00:19:31,005 Right? 401 00:19:31,395 --> 00:19:38,295 They could have taken a snapshot of their entire environment every day, every hour. 402 00:19:38,715 --> 00:19:39,735 They could have done that. 403 00:19:40,350 --> 00:19:44,100 They would've had essentially, you know, you know, a copy of 404 00:19:44,100 --> 00:19:47,010 their database from two hours ago. 405 00:19:47,940 --> 00:19:50,700 You know, they could have, and that wouldn't have been homegrown. 406 00:19:50,700 --> 00:19:52,110 I'm just saying they could have done that. 407 00:19:52,155 --> 00:19:52,545 Prasanna Malaiyandi: Well. 408 00:19:52,875 --> 00:19:55,095 The other thing they could have done is why don't they 409 00:19:55,095 --> 00:19:56,535 have a disaster recovery copy? 410 00:19:56,535 --> 00:20:00,675 Like if this is a SaaS service, wouldn't you have expected a Dr. 411 00:20:00,680 --> 00:20:05,985 Copy to be available somewhere where even if the production instance went down for 412 00:20:05,985 --> 00:20:11,025 some reason, they should have been able to come back up even if it wasn't a backup. 413 00:20:11,895 --> 00:20:15,465 W. Curtis Preston: Yeah, so this, this is where, you know, we often talk about 414 00:20:15,470 --> 00:20:18,615 that, that if your data is sitting on a SaaS service, I think it's your 415 00:20:18,620 --> 00:20:21,525 responsibility to back it up, or at least to make sure that it's backed up. 416 00:20:22,035 --> 00:20:24,615 You can, you can make an argument as to whether or not you should use 417 00:20:24,620 --> 00:20:26,325 their backup service if they have one. 418 00:20:26,505 --> 00:20:30,435 I think we've got three stories now that show that sometimes their backup 419 00:20:30,435 --> 00:20:34,785 service is as good as their it, um, and. 420 00:20:35,085 --> 00:20:38,085 The, so you, I think you should be backing up your own data, but 421 00:20:38,085 --> 00:20:42,435 generally speaking, some of these SaaS services, they will have a dr. 422 00:20:42,435 --> 00:20:46,635 Copy that isn't any good for you if you do something stupid, but, but if 423 00:20:46,635 --> 00:20:50,445 they do something stupid or there's a fire, or there's a, an attack, 424 00:20:50,475 --> 00:20:53,415 they've got a copy that they can restore the entire environment that 425 00:20:53,685 --> 00:20:58,605 that appears to be what they had here, but it was just woefully out of date. 426 00:20:59,290 --> 00:21:04,750 Um, and so the vast majority of their customers did get their data back, but 427 00:21:04,750 --> 00:21:08,020 there were a few, and I wanna, I want, I want to just point out a few that 428 00:21:08,020 --> 00:21:09,850 they were, uh, quoted in the story. 429 00:21:10,330 --> 00:21:14,230 There was, uh, Margaret Fry, a postdoctoral researcher at Harvard 430 00:21:14,230 --> 00:21:19,300 University, lost about 60 annotated texts and over a hundred hours worth of coding 431 00:21:19,300 --> 00:21:23,920 work related to her research on AIDS and sexual attraction Southern Africa. 432 00:21:24,395 --> 00:21:28,895 Jason Richardson, an associate professor at the University of Kentucky and his 433 00:21:28,895 --> 00:21:32,705 colleagues lost around a hundred hours of work on a project, analyzing job 434 00:21:32,710 --> 00:21:36,815 advertisements for school principles to assess how well Kentucky prepares 435 00:21:37,115 --> 00:21:38,705 aspiring school administrators. 436 00:21:39,185 --> 00:21:43,775 Uh, there's one comment on Dedoose Facebook page that the company had managed 437 00:21:43,775 --> 00:21:48,935 to destroy their wife's $10,000 research project, so there were real people. 438 00:21:49,710 --> 00:21:52,590 And these weren't just companies, these were just, these were just regular 439 00:21:52,590 --> 00:21:56,820 people trying to get their doctorates done and uh, they lost, you know, 440 00:21:57,270 --> 00:22:00,810 Prasanna Malaiyandi: yeah, but, but to their credit though, dedoose actually 441 00:22:00,810 --> 00:22:05,010 had no, I don't know if it was before this issue or after the event, but 442 00:22:05,010 --> 00:22:06,840 they did say they have functionality. 443 00:22:06,870 --> 00:22:09,480 Actually, no, it was even before because for staging, this is 444 00:22:09,480 --> 00:22:10,470 how you get your data out. 445 00:22:10,740 --> 00:22:12,420 There is an export functionality. 446 00:22:12,755 --> 00:22:13,105 W. Curtis Preston: Right. 447 00:22:14,445 --> 00:22:17,355 Prasanna Malaiyandi: That a user could use to pull all their data 448 00:22:17,355 --> 00:22:21,015 out of Dedoose for doing, like what you talked about, a local backup. 449 00:22:21,735 --> 00:22:26,595 So it is possible that these users could have been using that to protect 450 00:22:26,595 --> 00:22:32,535 themselves from these situations, and Dedoose is still around, and so I highly 451 00:22:32,535 --> 00:22:34,495 recommend people who are using Dedoose. 452 00:22:34,515 --> 00:22:37,545 Maybe you should periodically export your own data out. 453 00:22:38,205 --> 00:22:40,725 W. Curtis Preston: Can I, can I edit you and take the word maybe out there? 454 00:22:41,580 --> 00:22:42,030 Prasanna Malaiyandi: Yes. 455 00:22:43,020 --> 00:22:47,145 W. Curtis Preston: Yeah, so you should periodically back up your work no 456 00:22:47,145 --> 00:22:52,425 matter where it's residing, you know, a, a periodic, you know, inefficient 457 00:22:52,425 --> 00:22:55,375 backup that has a poor recovery point. 458 00:22:55,375 --> 00:22:59,155 Actual is still better than, than nothing. 459 00:22:59,305 --> 00:22:59,695 Right. 460 00:23:00,235 --> 00:23:01,645 Um, yeah. 461 00:23:01,645 --> 00:23:07,135 I remember, I, I have a, a niece that came to me with her, she had a laptop 462 00:23:07,675 --> 00:23:10,315 and she had her masters on this laptop. 463 00:23:10,495 --> 00:23:13,285 And the laptop, the drive was making those noises. 464 00:23:13,595 --> 00:23:13,895 You know 465 00:23:13,955 --> 00:23:15,065 Prasanna Malaiyandi: Oh, the click, click, click, click, 466 00:23:15,245 --> 00:23:15,485 W. Curtis Preston: Yeah. 467 00:23:15,485 --> 00:23:16,925 The hard drive should not make. 468 00:23:17,225 --> 00:23:21,755 We were able to get off her master's thesis, uh, off of that 469 00:23:21,755 --> 00:23:24,515 drive before it just went up in smoke and it was killing me. 470 00:23:24,520 --> 00:23:26,075 'cause I was like, you don't have a backup of her. 471 00:23:26,075 --> 00:23:27,335 That's really important work. 472 00:23:27,905 --> 00:23:33,335 Uh, you know, I feel for these people At the same time, you know, uh, my, 473 00:23:34,895 --> 00:23:36,905 my statement is gonna remain the same. 474 00:23:36,905 --> 00:23:39,725 That if your data is sitting in, in a SaaS service somewhere, it's 475 00:23:39,730 --> 00:23:42,935 your responsibility to, to make sure that it's being properly backed up. 476 00:23:43,700 --> 00:23:43,910 Prasanna Malaiyandi: Yeah. 477 00:23:45,365 --> 00:23:49,175 W. Curtis Preston: and and I still say even if the SaaS vendor offers 478 00:23:49,175 --> 00:23:53,645 backup, which this one apparently did, um, I still think that you 479 00:23:53,645 --> 00:23:54,665 should do it some other way. 480 00:23:55,655 --> 00:23:56,855 Um, yeah. 481 00:23:57,350 --> 00:24:00,200 Prasanna Malaiyandi: Well, do we know if they offered backup or 482 00:24:00,200 --> 00:24:01,490 was this just sort of their own 483 00:24:02,045 --> 00:24:02,375 W. Curtis Preston: right. 484 00:24:02,675 --> 00:24:04,625 It this was their, this was their internal dr. 485 00:24:04,655 --> 00:24:05,015 Right. 486 00:24:05,230 --> 00:24:09,935 Um, what I don't know is whether or not they advertise this as part of the, um. 487 00:24:10,535 --> 00:24:15,785 You know, now the CEO of Dedoose said this is a horrible moment for our company. 488 00:24:15,785 --> 00:24:17,435 We have never lost data or had a breach. 489 00:24:17,435 --> 00:24:19,925 It's impossible to say how many customers have lost data because 490 00:24:19,925 --> 00:24:23,555 of the firm doesn't monitor how customers are using their accounts. 491 00:24:23,885 --> 00:24:28,895 Only users can access, can assess losses in their projects, not Dedoose. 492 00:24:30,905 --> 00:24:33,275 Yeah, it was a, it was a 493 00:24:33,530 --> 00:24:34,010 Prasanna Malaiyandi: Yeah, if I 494 00:24:34,115 --> 00:24:34,145 W. Curtis Preston: a. 495 00:24:34,790 --> 00:24:35,630 Prasanna Malaiyandi: Oh, oh yeah. 496 00:24:35,660 --> 00:24:39,320 Like, and especially researchers, like coming back and getting 497 00:24:39,320 --> 00:24:41,210 that data is so difficult. 498 00:24:42,290 --> 00:24:44,600 And then just being like, yep, it's gone. 499 00:24:44,690 --> 00:24:45,710 Just like that puff of 500 00:24:46,010 --> 00:24:49,940 W. Curtis Preston: And, and some of that data is irreplaceable. 501 00:24:50,505 --> 00:24:50,795 Prasanna Malaiyandi: Yeah. 502 00:24:51,395 --> 00:24:53,945 W. Curtis Preston: It's interviews of subjects. 503 00:24:53,975 --> 00:24:58,085 Um, you know, I don't even want to try to, when you're in the middle of 504 00:24:58,085 --> 00:25:02,045 something that complicated, there is some data that is simply irreplaceable. 505 00:25:02,345 --> 00:25:07,535 Um, and, um, so it, you know, it's, it's a real shame what happened. 506 00:25:07,875 --> 00:25:10,270 Prasanna Malaiyandi: Did you wanna talk about what they said they're going 507 00:25:10,270 --> 00:25:11,980 to do and what they've actually done? 508 00:25:12,280 --> 00:25:12,490 W. Curtis Preston: yeah. 509 00:25:12,490 --> 00:25:15,010 That's a great, yeah, that's a great segue. 510 00:25:15,060 --> 00:25:18,840 So they said they're now going to be doing seven things and when, when I 511 00:25:18,840 --> 00:25:22,230 say now we, we mean, uh, 10 years ago. 512 00:25:22,800 --> 00:25:23,190 Right. 513 00:25:23,310 --> 00:25:27,450 Um, after the event they said they were going to, it sounds like they 514 00:25:27,450 --> 00:25:30,900 went from not much to way overboard. 515 00:25:31,020 --> 00:25:31,350 Right. 516 00:25:31,350 --> 00:25:34,290 Of course, I would never say that any backup system is overboard. 517 00:25:34,290 --> 00:25:35,610 But here's what they said they were gonna do. 518 00:25:36,090 --> 00:25:38,910 They were gonna develop a database mirror, slave, and Azure. 519 00:25:39,255 --> 00:25:44,115 A database mirror slave in Amazon S3, keeping a mirror copy of the entire 520 00:25:44,115 --> 00:25:48,795 blob storage, you know, including all the stuff in an encrypted volume 521 00:25:49,335 --> 00:25:54,195 storing nightly database backups on the V-H-D-V-H-D, virtual hard drive, I guess, 522 00:25:54,555 --> 00:25:57,015 and Azure blob storage and Amazon three. 523 00:25:57,255 --> 00:26:01,845 So they're gonna keep three nightly backups in three separate places mirroring 524 00:26:01,845 --> 00:26:04,725 all Azure file data into an S3 bucket. 525 00:26:05,360 --> 00:26:09,410 Carrying out weekly restore exercise, uh, for the database backups and 526 00:26:09,410 --> 00:26:11,510 a monthly bare bones restore. 527 00:26:11,780 --> 00:26:13,040 That all sounds great. 528 00:26:13,160 --> 00:26:20,515 Now I do, I do wanna comment, um, that anything that refers to mirroring to me. 529 00:26:21,765 --> 00:26:23,745 Isn't a backup, right? 530 00:26:23,835 --> 00:26:25,545 Um, that is a dr. 531 00:26:25,545 --> 00:26:31,155 Copy that will only be useful if what happens is like a fire, right? 532 00:26:31,155 --> 00:26:35,355 If there's any sort of attack, any of those mirrors are totally worthless 533 00:26:35,355 --> 00:26:39,285 because, or if, if you do something stupid like drop a table, all those 534 00:26:39,290 --> 00:26:42,915 mirrors are gonna immediately, uh, you know, get corrupted as 535 00:26:43,035 --> 00:26:45,135 Prasanna Malaiyandi: but it would've helped in their current, 536 00:26:45,135 --> 00:26:46,575 in the event that it happened. 537 00:26:48,255 --> 00:26:50,595 W. Curtis Preston: Uh, I'm gonna say it's unclear as to whether 538 00:26:50,595 --> 00:26:51,435 or not it would help in it, 539 00:26:51,510 --> 00:26:51,780 Prasanna Malaiyandi: Okay. 540 00:26:51,990 --> 00:26:52,230 Sorry. 541 00:26:52,230 --> 00:26:53,160 It could help. 542 00:26:53,430 --> 00:26:53,730 Yeah. 543 00:26:53,745 --> 00:26:54,735 W. Curtis Preston: It could have helped. 544 00:26:54,795 --> 00:26:55,635 Yes, yes. 545 00:26:55,965 --> 00:27:00,075 Um, mirroring is not bad just because they're putting it here and 546 00:27:00,345 --> 00:27:01,455 they're listing all these things. 547 00:27:01,785 --> 00:27:05,775 Um, you know, it's a shame that they, that they had to wait. 548 00:27:06,150 --> 00:27:07,680 Uh, after the thing. 549 00:27:07,800 --> 00:27:11,400 What I also would've liked to seen in that list is I would've liked 550 00:27:11,400 --> 00:27:16,380 to have seen that one or more of these copies, uh, was immutable. 551 00:27:17,100 --> 00:27:17,490 Right? 552 00:27:17,820 --> 00:27:22,620 Um, and, uh, and maybe they are, I don't know, you know? 553 00:27:23,170 --> 00:27:26,230 Prasanna Malaiyandi: It's a great list, but it's also one of those 554 00:27:26,230 --> 00:27:29,860 things like once a horse has left the barn, like what do you do? 555 00:27:31,420 --> 00:27:33,520 You know, it's sort of like, Hey, uh, we're just gonna throw 556 00:27:33,520 --> 00:27:37,510 everything at this and just say, this is everything we're going to do. 557 00:27:38,530 --> 00:27:39,070 W. Curtis Preston: yeah, it 558 00:27:39,250 --> 00:27:39,730 Prasanna Malaiyandi: than what 559 00:27:39,790 --> 00:27:40,060 W. Curtis Preston: yeah. 560 00:27:40,930 --> 00:27:45,340 Prasanna Malaiyandi: what is actually makes sense from a business perspective, 561 00:27:45,340 --> 00:27:49,090 because sure, you could do all these things, but just imagine the cost 562 00:27:49,090 --> 00:27:51,850 you would be spending for backup. 563 00:27:52,930 --> 00:27:53,170 Right? 564 00:27:53,170 --> 00:27:55,840 Your backup copies are probably more than your production 565 00:27:55,845 --> 00:27:57,325 copies at that point, right? 566 00:27:58,375 --> 00:27:58,945 W. Curtis Preston: Yeah. 567 00:27:59,005 --> 00:28:00,715 It depends on, depends on the way they're doing them. 568 00:28:00,715 --> 00:28:00,925 Yeah. 569 00:28:00,955 --> 00:28:01,195 Yeah. 570 00:28:01,360 --> 00:28:01,720 Prasanna Malaiyandi: Yeah. 571 00:28:02,710 --> 00:28:03,100 Right. 572 00:28:03,445 --> 00:28:03,625 W. Curtis Preston: Yeah. 573 00:28:03,625 --> 00:28:08,935 To take, to take your, to take your analogy, you know, the horse is laying 574 00:28:08,995 --> 00:28:13,585 out dead in the field, and you're like, you know what we're gonna do? 575 00:28:13,705 --> 00:28:16,885 We're gonna lock the, we're gonna lock the barn door now, and then we're 576 00:28:16,885 --> 00:28:19,435 gonna put another padlock on top of that lock, and then we're gonna put 577 00:28:19,440 --> 00:28:23,425 a fence around the area in front of the barn, just in case the lock fails. 578 00:28:23,755 --> 00:28:26,725 And then, uh, you know, and then we're gonna have a guy 579 00:28:26,725 --> 00:28:28,435 standing, standing there with a. 580 00:28:28,630 --> 00:28:31,210 With a tran gun, uh, to shoot the horse. 581 00:28:31,210 --> 00:28:35,050 If the horse comes outta the bar, it's like, yeah, that's all great, but your 582 00:28:35,050 --> 00:28:38,710 horse is dead, so you need another horse. 583 00:28:39,475 --> 00:28:39,895 Prasanna Malaiyandi: Yeah, 584 00:28:40,000 --> 00:28:41,890 W. Curtis Preston: we're some, I'm, I'm sorry that was a little 585 00:28:41,895 --> 00:28:44,861 morbid, but I apologize to the horse lovers in the, in the group. 586 00:28:45,625 --> 00:28:47,875 Prasanna Malaiyandi: but the other thing we found though, so that's what they 587 00:28:47,875 --> 00:28:52,735 had promised 10 years ago, but they do have in their, what is this called in 588 00:28:52,735 --> 00:28:59,845 their security article on Dedoose, sort of what they've actually done, right? 589 00:28:59,845 --> 00:29:00,595 So. 590 00:29:01,570 --> 00:29:07,180 What they have done is they are using redundant storage volumes on Azure. 591 00:29:07,845 --> 00:29:08,265 W. Curtis Preston: mm-Hmm. 592 00:29:08,440 --> 00:29:13,660 Prasanna Malaiyandi: All project data backed up in full on, on a nightly basis. 593 00:29:15,520 --> 00:29:20,770 A volume is onsite with two being offsite and replicated across geographic regions. 594 00:29:22,240 --> 00:29:26,920 Um, and then the storage volume is encrypted and mirrored in 595 00:29:26,920 --> 00:29:29,350 real time to Amazon S3 storage. 596 00:29:31,180 --> 00:29:36,160 They've automated everything such that on a weekly basis, it'll download the most 597 00:29:36,160 --> 00:29:38,500 recent backup from each storage volume. 598 00:29:39,130 --> 00:29:43,540 Verify they're using the correct version of the backup file, do a full 599 00:29:43,540 --> 00:29:47,590 test restoration of the database, and email reporting to all backup 600 00:29:47,595 --> 00:29:51,430 and restore process results to key members of the Dedoose team. 601 00:29:53,100 --> 00:29:55,525 W. Curtis Preston: Yeah, it's, it's still, to me, it still reads 602 00:29:55,525 --> 00:29:58,765 like a homegrown system, but, um. 603 00:29:59,125 --> 00:30:03,145 You know, because like one of the things like, you know, I, I did, I did a lot of 604 00:30:03,145 --> 00:30:04,705 homegrown systems over the years, right? 605 00:30:04,735 --> 00:30:07,195 Like one of the things like with email reporting is. 606 00:30:07,975 --> 00:30:12,775 D does your system report if it doesn't run right, is is there, 607 00:30:12,775 --> 00:30:15,565 is there a thing that, that's, you know, like what happened? 608 00:30:15,565 --> 00:30:18,535 You know, 'cause sometimes you don't notice that you're not getting an email. 609 00:30:18,985 --> 00:30:19,375 Right? 610 00:30:19,435 --> 00:30:20,545 Hey, has anybody noticed that? 611 00:30:20,545 --> 00:30:23,815 We don't get the, we stopped getting the backups, you know, 612 00:30:23,875 --> 00:30:25,375 uh, like three weeks ago. 613 00:30:25,885 --> 00:30:29,905 Um, usually when you, when you discover that is right after the, um, the 614 00:30:29,905 --> 00:30:31,555 bad thing happens, whatever, it's 615 00:30:31,615 --> 00:30:35,965 Prasanna Malaiyandi: but even though it might be sort of homegrown from a 616 00:30:36,115 --> 00:30:39,595 automation process perspective, like we don't have enough information. 617 00:30:39,895 --> 00:30:41,100 I think the fact that there are at least. 618 00:30:41,740 --> 00:30:44,230 Doing nightly backups, right? 619 00:30:44,680 --> 00:30:48,370 Verifying it on a weekly basis like these are, and also making sure that 620 00:30:48,370 --> 00:30:50,980 the data is stored offsite, right? 621 00:30:50,980 --> 00:30:55,030 I think maybe it's going a bit for like, they may not have to do like multi-cloud. 622 00:30:55,030 --> 00:30:59,170 Maybe that's going a stretch too far, but I could see that they really are 623 00:30:59,170 --> 00:31:00,760 worried about data integrity, so, 624 00:31:00,760 --> 00:31:02,590 W. Curtis Preston: There's no such thing as a stretch too 625 00:31:02,590 --> 00:31:03,910 far for backup and recovery. 626 00:31:03,910 --> 00:31:04,360 My friend. 627 00:31:05,845 --> 00:31:06,135 Prasanna Malaiyandi: well, 628 00:31:06,400 --> 00:31:09,430 W. Curtis Preston: They're doing the thing that I wished everybody did, but 629 00:31:09,435 --> 00:31:11,230 nobody does because it's too expensive, 630 00:31:11,975 --> 00:31:12,505 Prasanna Malaiyandi: Yes. 631 00:31:12,760 --> 00:31:13,210 W. Curtis Preston: right? 632 00:31:14,320 --> 00:31:16,930 Um, so I, I can't fault 'em for doing that, 633 00:31:17,665 --> 00:31:17,995 Prasanna Malaiyandi: Yeah. 634 00:31:18,280 --> 00:31:21,475 So it looks like they are doing the right things, 635 00:31:21,655 --> 00:31:22,075 W. Curtis Preston: mm-hmm. 636 00:31:22,615 --> 00:31:24,325 Prasanna Malaiyandi: um, from what we can gather. 637 00:31:24,325 --> 00:31:26,905 So it looks like they have learned and they're still around, right? 638 00:31:26,910 --> 00:31:28,495 People still continue to use 'em. 639 00:31:28,795 --> 00:31:30,415 They do have an active subreddit. 640 00:31:30,505 --> 00:31:32,545 They are continuing to release new features. 641 00:31:33,040 --> 00:31:33,160 W. Curtis Preston: Mm-Hmm. 642 00:31:34,150 --> 00:31:37,840 Yeah, and I think, I think what was in their favor is that they 643 00:31:37,840 --> 00:31:39,640 were the only SaaS game in town. 644 00:31:40,060 --> 00:31:43,300 And for a lot of people that, that, that's probably a significant. 645 00:31:44,185 --> 00:31:45,745 Reason to use their service. 646 00:31:45,835 --> 00:31:46,255 Right. 647 00:31:46,465 --> 00:31:52,375 Um, I would also hope that this is a significant reason for anyone to do a, 648 00:31:52,480 --> 00:31:54,865 a regular export of their data, right. 649 00:31:55,135 --> 00:31:59,125 When I'm using a service like that, where I tend to do is I do, I do an export. 650 00:31:59,125 --> 00:32:03,895 Like if I'm doing a manual export, I'm doing it after significant events 651 00:32:03,925 --> 00:32:06,955 like finishing my taxes, right? 652 00:32:07,285 --> 00:32:08,935 I did, you know, I did an export. 653 00:32:08,935 --> 00:32:12,895 I, I did a PD, I didn't do exports of. 654 00:32:13,495 --> 00:32:16,765 My intuit data as I was doing it right. 655 00:32:16,975 --> 00:32:20,245 But once I finished my return, I printed a PDF of that. 656 00:32:20,725 --> 00:32:22,375 Um, I have a paper copy. 657 00:32:22,375 --> 00:32:24,655 I have a PDF copy that's on my laptop. 658 00:32:24,865 --> 00:32:28,075 It's backed up using my backup software. 659 00:32:28,135 --> 00:32:32,395 And uh, I put it in, um, Google Docs, right? 660 00:32:32,395 --> 00:32:33,715 So it's all over the place, right? 661 00:32:34,075 --> 00:32:35,695 Um, but I don't do that every day. 662 00:32:35,695 --> 00:32:38,395 It's just you do it when you, when you get to that moment where you're 663 00:32:38,395 --> 00:32:40,285 like, ah, oh, this is really good. 664 00:32:40,610 --> 00:32:41,630 I hope I don't lose 665 00:32:41,785 --> 00:32:42,775 Prasanna Malaiyandi: a good stopping point. 666 00:32:42,950 --> 00:32:44,120 W. Curtis Preston: It's a good stopping point. 667 00:32:44,450 --> 00:32:44,870 Yeah. 668 00:32:45,110 --> 00:32:49,520 Um, it's like also I use voice recognition software, right. 669 00:32:50,000 --> 00:32:57,980 And sometimes it crashes and you lose whatever you said, uh, in that thing. 670 00:32:57,985 --> 00:33:00,980 And so I just got into the habit of anytime I finish, like 671 00:33:01,280 --> 00:33:03,020 a page, I just hit, you know? 672 00:33:03,200 --> 00:33:05,150 I just say, um, yeah. 673 00:33:05,210 --> 00:33:06,860 Uh, 'cause I, I can actually say. 674 00:33:07,255 --> 00:33:10,555 Um, I almost said, Hey, Siri, you know, click file save. 675 00:33:10,615 --> 00:33:11,155 That's what I say. 676 00:33:11,155 --> 00:33:12,115 Click file, save. 677 00:33:12,565 --> 00:33:16,345 And, um, and I can do that or you can just type control s right? 678 00:33:16,795 --> 00:33:21,055 Uh, just do you know, this is, I guess this is another lesson to learn 679 00:33:21,055 --> 00:33:27,715 is, is that should be a part of your mental process when you do stuff and 680 00:33:27,715 --> 00:33:31,405 it's storing that data in some place. 681 00:33:31,930 --> 00:33:37,300 At the end, at that stopping moment, do you, do you think to yourself, Hey, 682 00:33:37,300 --> 00:33:42,580 is this data backed up in a way that protects me from all stupid things? 683 00:33:43,030 --> 00:33:43,450 Prasanna Malaiyandi: You know 684 00:33:43,510 --> 00:33:43,630 W. Curtis Preston: are you 685 00:33:43,690 --> 00:33:45,460 Prasanna Malaiyandi: funny when you're, when you're talking 686 00:33:45,460 --> 00:33:47,110 about the control save, right? 687 00:33:47,140 --> 00:33:48,580 Or Control S to save it. 688 00:33:49,210 --> 00:33:52,510 So I'm so used to working on Excel spreadsheets that I would always do 689 00:33:52,510 --> 00:33:54,070 like Control S, control S, to save 690 00:33:54,135 --> 00:33:54,925 W. Curtis Preston: Right, right, 691 00:33:55,360 --> 00:33:57,760 Prasanna Malaiyandi: Even when I use Google shoots, I still do it anyway, 692 00:33:58,175 --> 00:33:58,525 W. Curtis Preston: right. 693 00:34:00,580 --> 00:34:01,925 Prasanna Malaiyandi: just out of the force of habit. 694 00:34:02,800 --> 00:34:03,250 W. Curtis Preston: Yeah. 695 00:34:03,250 --> 00:34:03,850 That's funny. 696 00:34:04,380 --> 00:34:05,220 , Prasanna Malaiyandi: So, lessons learned. 697 00:34:05,280 --> 00:34:05,640 Lessons 698 00:34:05,925 --> 00:34:06,405 W. Curtis Preston: Yeah. 699 00:34:06,435 --> 00:34:06,645 Yeah. 700 00:34:06,645 --> 00:34:10,215 So we, you know, so obviously we talk about, you know, it's still your 701 00:34:10,215 --> 00:34:11,625 responsibility to backup up SaaS. 702 00:34:11,925 --> 00:34:14,685 The other thing is a month RPO? 703 00:34:14,795 --> 00:34:16,495 A month backup frequency. 704 00:34:16,675 --> 00:34:18,475 That's, that's unacceptable. 705 00:34:18,865 --> 00:34:22,675 That's just, I can't imagine that being acceptable, uh, to 706 00:34:22,735 --> 00:34:24,535 in, in any world, by the way. 707 00:34:25,375 --> 00:34:28,075 That is determined by your business, right? 708 00:34:28,135 --> 00:34:32,275 I, I have met at least one company where a month long RPO was fine. 709 00:34:32,335 --> 00:34:34,375 They were a paper mill, right? 710 00:34:34,375 --> 00:34:38,005 They were, they were like, we don't, the, the, the, the computers, they just, they 711 00:34:38,005 --> 00:34:39,325 don't have anything that we care about. 712 00:34:39,330 --> 00:34:39,565 Right? 713 00:34:39,895 --> 00:34:40,765 They were fine with a month. 714 00:34:40,975 --> 00:34:42,655 That should be determined by your business. 715 00:34:42,925 --> 00:34:45,385 But a month seems really long to me. 716 00:34:46,255 --> 00:34:51,475 Um, and the other thing is, if your backup system. 717 00:34:53,065 --> 00:34:56,095 Is this is, we don't know if this is actually the case, 718 00:34:56,095 --> 00:34:57,325 but I'm inferring from it. 719 00:34:57,655 --> 00:35:03,025 If your backup system overwrites the last good backup as part of 720 00:35:03,025 --> 00:35:06,775 the process because you're so starved for storage, that is bad. 721 00:35:07,645 --> 00:35:08,365 That is bad. 722 00:35:08,370 --> 00:35:09,655 Bad, bad, bad, bad. 723 00:35:09,660 --> 00:35:10,795 That is basic 724 00:35:11,110 --> 00:35:12,100 Prasanna Malaiyandi: storage is cheap. 725 00:35:12,475 --> 00:35:12,865 W. Curtis Preston: Bad. 726 00:35:13,015 --> 00:35:15,505 Well, it's cheap, but I'm just saying it's cheap. 727 00:35:15,745 --> 00:35:16,075 Right? 728 00:35:16,075 --> 00:35:16,435 So much. 729 00:35:16,435 --> 00:35:19,225 So much we do or don't do is because it's, it's inexpensive. 730 00:35:19,225 --> 00:35:19,315 Right? 731 00:35:20,290 --> 00:35:23,200 Prasanna Malaiyandi: Well, given that they are keeping all these 732 00:35:23,200 --> 00:35:26,800 copies and dealing with cross site replication and everything else, 733 00:35:27,020 --> 00:35:27,440 W. Curtis Preston: Mm-Hmm. 734 00:35:28,255 --> 00:35:29,965 Well, they, yeah, they're not doing that now. 735 00:35:30,025 --> 00:35:30,415 Right. 736 00:35:30,415 --> 00:35:35,905 I'm just saying there are people, there are people that their backup system 737 00:35:35,965 --> 00:35:41,335 is overriding the last good backup, and I just, that, that is a, that is 738 00:35:41,335 --> 00:35:45,145 a really, basically, it means that if you're in the middle of the backup and 739 00:35:45,145 --> 00:35:46,645 something happens, you have no backup. 740 00:35:47,245 --> 00:35:47,605 Prasanna Malaiyandi: Yep. 741 00:35:48,355 --> 00:35:49,645 W. Curtis Preston: Unless you're backing up the backup. 742 00:35:49,705 --> 00:35:53,275 If you're backing up the backup right, then, then, then, right. 743 00:35:53,275 --> 00:35:56,605 But if this is your only backup and you're overriding it with the next 744 00:35:56,605 --> 00:35:59,305 good backup because you're too cheap to buy enough storage for multiple 745 00:35:59,310 --> 00:36:02,605 backups, um, I dunno what to tell you. 746 00:36:03,205 --> 00:36:03,505 Prasanna Malaiyandi: Yeah. 747 00:36:03,805 --> 00:36:06,385 Please do not please step away from the keyboard and have someone 748 00:36:06,385 --> 00:36:07,585 else design your backup system. 749 00:36:07,855 --> 00:36:08,065 W. Curtis Preston: step. 750 00:36:11,935 --> 00:36:12,145 That 751 00:36:12,355 --> 00:36:13,075 Prasanna Malaiyandi: is true though. 752 00:36:13,225 --> 00:36:13,705 W. Curtis Preston: perfect. 753 00:36:13,705 --> 00:36:14,155 It is true. 754 00:36:14,155 --> 00:36:14,425 Yeah. 755 00:36:14,485 --> 00:36:14,815 Yeah. 756 00:36:14,815 --> 00:36:15,175 Prasanna Malaiyandi: Yeah. 757 00:36:15,865 --> 00:36:16,345 W. Curtis Preston: Yeah. 758 00:36:16,525 --> 00:36:20,785 Uh, can you think of any other, um, lessons from this one? 759 00:36:21,395 --> 00:36:24,875 Prasanna Malaiyandi: I know that we talked about how Dedoose may not have 760 00:36:25,295 --> 00:36:28,895 had the best backup systems in place, 761 00:36:29,855 --> 00:36:32,855 but this is also one of those things like it was. 762 00:36:33,800 --> 00:36:38,750 Unfortunate that they had these multiple issues that led to such a big failure. 763 00:36:39,710 --> 00:36:43,880 But it is good that they did see, okay, here are gaps in our system. 764 00:36:43,880 --> 00:36:45,200 Here's how we're gonna fix it. 765 00:36:45,440 --> 00:36:49,100 They were transparent with the users and it looks like they've gotten 766 00:36:49,100 --> 00:36:50,450 their act together, which is good. 767 00:36:50,645 --> 00:36:50,995 Right? 768 00:36:51,740 --> 00:36:54,860 So I think it's just a matter of making sure they're continuing to stay up to 769 00:36:54,860 --> 00:36:59,210 date and reevaluating their architecture and infrastructure of their backup. 770 00:36:59,210 --> 00:37:01,700 Especially like, I don't know how they deal with ransomware. 771 00:37:02,555 --> 00:37:03,485 W. Curtis Preston: Right, right, 772 00:37:03,530 --> 00:37:05,750 Prasanna Malaiyandi: Like are they really using separate accounts, 773 00:37:05,750 --> 00:37:08,540 like when they're replicating to different buckets, different regions, 774 00:37:08,540 --> 00:37:12,680 or is it all like the same username, password, or whatever across the two? 775 00:37:13,880 --> 00:37:14,180 Right. 776 00:37:14,945 --> 00:37:15,515 W. Curtis Preston: You're killing me 777 00:37:15,560 --> 00:37:15,740 Prasanna Malaiyandi: Right. 778 00:37:15,740 --> 00:37:19,940 Because hopefully they're keeping their Azure or their AWS system isolated. 779 00:37:20,345 --> 00:37:21,125 W. Curtis Preston: Right, right. 780 00:37:21,665 --> 00:37:22,715 And hopefully again, 781 00:37:22,915 --> 00:37:23,480 Prasanna Malaiyandi: than backups. 782 00:37:23,675 --> 00:37:23,895 Yep. 783 00:37:23,915 --> 00:37:26,135 W. Curtis Preston: somewhere, along the way they're using immutable storage. 784 00:37:26,345 --> 00:37:26,735 Right. 785 00:37:30,230 --> 00:37:33,200 And we, and we do like the way that they, that they had the blog. 786 00:37:33,530 --> 00:37:37,250 They were, they were immediately, um, you know, transparent with what was going on. 787 00:37:37,250 --> 00:37:41,600 They were, they were a little bit of, uh, you know, mea culpa, right? 788 00:37:41,600 --> 00:37:43,010 They're like, Hey, this is our fault. 789 00:37:43,070 --> 00:37:46,460 You know, this is all us, you know, uh, we're so, so sorry. 790 00:37:46,850 --> 00:37:49,940 Uh, I'm a little disappointed that they deleted the blog after the fact. 791 00:37:50,405 --> 00:37:51,215 Um, 792 00:37:51,605 --> 00:37:52,145 Prasanna Malaiyandi: Put a link to the. 793 00:37:52,355 --> 00:37:53,165 W. Curtis Preston: referenced in mo. 794 00:37:53,585 --> 00:37:53,915 What's that? 795 00:37:53,915 --> 00:37:58,055 Yeah, we will put a log to a, a link to the archive version of the blog. 796 00:37:58,085 --> 00:37:59,435 'cause the internet never forgets. 797 00:37:59,915 --> 00:38:01,145 Um, yeah. 798 00:38:01,175 --> 00:38:01,535 So. 799 00:38:03,095 --> 00:38:06,395 All right, well thanks for helping me work our way through 800 00:38:06,400 --> 00:38:08,915 another cloud disaster Prasanna. 801 00:38:09,245 --> 00:38:10,325 Prasanna Malaiyandi: No, thank you Curtis. 802 00:38:10,325 --> 00:38:10,840 This was fun. 803 00:38:11,795 --> 00:38:12,305 W. Curtis Preston: Yeah. 804 00:38:12,425 --> 00:38:17,075 Fun, fun, fun and a sad way that unfortunately involved 805 00:38:17,080 --> 00:38:19,085 a, you know, a deceased 806 00:38:19,085 --> 00:38:19,565 horse. 807 00:38:19,900 --> 00:38:20,120 Prasanna Malaiyandi: but. 808 00:38:20,195 --> 00:38:20,765 W. Curtis Preston: Um, 809 00:38:26,210 --> 00:38:27,560 Prasanna Malaiyandi: Uh, poor horsey. 810 00:38:27,875 --> 00:38:28,535 W. Curtis Preston: yeah. 811 00:38:28,625 --> 00:38:33,935 Um, well subscribe folks so that you don't miss all of this amazingness. 812 00:38:34,205 --> 00:38:36,755 Uh, and, uh, that is a wrap.