1 00:00:00,000 --> 00:00:06,149 Our latest in our cloud disaster series is the 2014 StorageCraft outage 2 00:00:06,539 --> 00:00:08,879 that resulted in loss customer data. 3 00:00:09,419 --> 00:00:14,250 A server decommissioning mistake during a cloud migration resulted in metadata 4 00:00:14,250 --> 00:00:18,870 loss and StorageCraft scrambling to help customers re-seed their backups. 5 00:00:19,289 --> 00:00:22,740 This may remind you a bit of what happened to carbonate, but this 6 00:00:22,740 --> 00:00:27,329 is very different because here the CEO took full responsibility. 7 00:00:27,629 --> 00:00:30,719 And the company I think went above and beyond to help customers 8 00:00:30,719 --> 00:00:32,609 re-seed as quickly as possible. 9 00:00:33,100 --> 00:00:36,419 Prasanna and I analyze what went wrong, the lessons learned for 10 00:00:36,419 --> 00:00:38,399 backup providers and customers alike. 11 00:00:38,879 --> 00:00:41,969 And what steps you should take if you ever find yourself. 12 00:00:42,300 --> 00:00:44,339 With your backups lost in space. 13 00:00:44,789 --> 00:00:48,809 We'll discuss the importance of migration processes, backup resiliency 14 00:00:48,809 --> 00:00:52,649 measures and why trust is so critical in the backup and recovery business. 15 00:00:53,279 --> 00:00:57,029 If you're not familiar with me, I'm w Curtis Preston and there's a reason 16 00:00:57,029 --> 00:00:58,919 I'm so passionate about this subject. 17 00:00:59,399 --> 00:01:03,359 During my first job as a backup admin, my company lost a huge 18 00:01:03,359 --> 00:01:05,399 database and I couldn't restore it. 19 00:01:05,939 --> 00:01:09,539 Since that nightmare, I've dedicated my career to making sure that 20 00:01:09,539 --> 00:01:13,229 will never again happen to me or anyone who will listen to me. 21 00:01:13,709 --> 00:01:17,609 We turn unappreciated backup admins into cyber recovery heroes. 22 00:01:17,849 --> 00:01:19,889 This is the backup wrap-up. 23 00:01:33,937 --> 00:01:34,772 W. Curtis Preston: Welcome to the show. 24 00:01:36,127 --> 00:01:38,287 Hi, I'm w Curtis Preston, AKA, Mr. 25 00:01:38,287 --> 00:01:43,867 Backup, and I have with me my career reverse trauma specialist 26 00:01:43,967 --> 00:01:45,877 Prasanna Malaiyandi, how's it going? 27 00:01:45,877 --> 00:01:46,487 Prasanna? 28 00:01:47,092 --> 00:01:49,612 Prasanna Malaiyandi: I am good, Curtis, how you been? 29 00:01:49,672 --> 00:01:51,892 Are you doing a little better today? 30 00:01:55,117 --> 00:01:59,857 W. Curtis Preston: So for the first time in a long time, I 31 00:01:59,857 --> 00:02:02,742 find myself at the keyboard. 32 00:02:04,072 --> 00:02:08,092 Um, you know, in a production environment administering a 33 00:02:08,092 --> 00:02:09,862 very big net backup environment. 34 00:02:09,862 --> 00:02:15,802 I haven't been the guy on the keyboard in a while, and it, it was. 35 00:02:16,162 --> 00:02:16,882 Fun. 36 00:02:17,512 --> 00:02:21,442 Um, and doing, you know, doing performance testing and all, you know, normally 37 00:02:21,442 --> 00:02:24,802 it's been somebody else at the keyboard and I'm like advising, but this was 38 00:02:24,802 --> 00:02:27,742 me in a data center all by myself. 39 00:02:28,192 --> 00:02:29,302 Um, you 40 00:02:29,482 --> 00:02:30,277 Prasanna Malaiyandi: you back 30 years. 41 00:02:30,387 --> 00:02:31,492 W. Curtis Preston: was interesting. 42 00:02:32,332 --> 00:02:37,312 Uh, it's, it's not quite 30 years, but, but yeah, it's been, uh, it's 43 00:02:37,312 --> 00:02:41,512 been a while since I've been the guy and, you know, and standing in what 44 00:02:41,512 --> 00:02:43,372 we're going to call a data center. 45 00:02:43,912 --> 00:02:48,532 It's really just a big storage closet that happens to have a, a server rack. 46 00:02:48,812 --> 00:02:53,012 We're doing a, a backup of this, uh, customer's environment and, uh, 47 00:02:53,042 --> 00:02:55,142 let's just say it's, it's not small. 48 00:02:55,142 --> 00:03:00,152 It was, it was 400 terabytes of data sitting on a couple of little NAS boxes. 49 00:03:00,782 --> 00:03:03,902 Um, so it's been, it has been interesting. 50 00:03:03,932 --> 00:03:05,972 I will just say that, um. 51 00:03:06,737 --> 00:03:11,927 Uh, standing in a very noisy server room talking to somebody on the phone, I could 52 00:03:11,927 --> 00:03:13,727 barely hear them on my, you know, on 53 00:03:13,892 --> 00:03:14,492 Prasanna Malaiyandi: just be lucky. 54 00:03:14,492 --> 00:03:15,747 You at least have cell phones now. 55 00:03:17,357 --> 00:03:21,527 W. Curtis Preston: I, it's nice that I have cell phones, cell phones and, and 56 00:03:21,527 --> 00:03:23,987 chat and screen connect so that like. 57 00:03:24,287 --> 00:03:26,057 You know, they can see what I'm seeing. 58 00:03:26,057 --> 00:03:27,857 And, um, you know, all of that. 59 00:03:28,397 --> 00:03:31,517 And, um, yeah, it's been interesting. 60 00:03:32,147 --> 00:03:41,417 But, um, speaking of interesting backup scenarios, we're continuing our series on 61 00:03:41,417 --> 00:03:44,447 cloud disasters and we have another one. 62 00:03:45,077 --> 00:03:49,667 This is very much in the vein of what happened to Carbonite. 63 00:03:50,287 --> 00:03:53,077 It's very similar and yet different. 64 00:03:53,467 --> 00:03:54,217 So, 65 00:03:54,317 --> 00:03:54,617 Prasanna Malaiyandi: Same. 66 00:03:54,617 --> 00:03:55,127 Same, but 67 00:03:55,147 --> 00:03:57,217 W. Curtis Preston: wanna do a quick summary of what happened to Carbonite? 68 00:03:57,927 --> 00:04:01,947 Prasanna Malaiyandi: So with Carbonite, what ended up happening was they had 69 00:04:01,947 --> 00:04:03,507 their own storage that they were using. 70 00:04:03,512 --> 00:04:03,747 Right. 71 00:04:04,077 --> 00:04:05,847 Uh, and people would upload their backups to 72 00:04:05,847 --> 00:04:11,217 it, and they had a double disc failure that took 73 00:04:11,217 --> 00:04:14,007 down their storage array, and they lost all their data. 74 00:04:14,057 --> 00:04:18,127 And they basically told customers, yeah, don't worry about it. 75 00:04:18,157 --> 00:04:21,757 We're just gonna re-upload and do another backup and get all your data back up. 76 00:04:21,762 --> 00:04:28,057 And during that time, there were something like, I think it was 35 customers, 34 77 00:04:28,057 --> 00:04:33,157 customers I wanna say was the number, um, who ended up having another fault 78 00:04:33,157 --> 00:04:34,717 in their production data, right. 79 00:04:34,717 --> 00:04:36,127 On their laptop, didn't. 80 00:04:36,717 --> 00:04:40,347 Crash or whatever else happened, and they were not able to restore their 81 00:04:40,347 --> 00:04:42,112 data because there were no backups left. 82 00:04:43,882 --> 00:04:46,612 W. Curtis Preston: Yeah, the, I I, I think the part of their story that 83 00:04:46,617 --> 00:04:50,902 bothers me the most is that, you know, the company's reaction was our 84 00:04:50,907 --> 00:04:53,932 stupid storage array vendor, right? 85 00:04:53,932 --> 00:04:57,622 They blamed it on their storage array vendor and they sued their vendor. 86 00:04:57,897 --> 00:05:00,077 Um, and so while this story. 87 00:05:00,697 --> 00:05:03,547 Is gonna start out sounding kind of similar. 88 00:05:04,027 --> 00:05:08,977 Uh, it, it's very similar in terms of the impact to the customers, but the response 89 00:05:08,977 --> 00:05:11,107 from the company is completely different. 90 00:05:11,112 --> 00:05:15,037 And so I feel completely different about the company, even though 91 00:05:15,577 --> 00:05:19,117 the exact same thing happened on the, on the customer side. 92 00:05:19,607 --> 00:05:23,697 So first off, we're talking about, storage craft, which. 93 00:05:23,957 --> 00:05:30,197 Is a company that had been around for a while and then had been acquired in 94 00:05:30,197 --> 00:05:36,917 2021 by Arcserve, which is a company that has been around a long while. 95 00:05:37,127 --> 00:05:40,277 Arcserve goes all the way back to my early days. 96 00:05:40,277 --> 00:05:45,737 Arcserve was actually the first backup product I ever tested as 97 00:05:45,737 --> 00:05:47,747 a, to, to potentially buy it. 98 00:05:48,167 --> 00:05:53,497 And running it on NetWare and that boy that brings me back. 99 00:05:53,857 --> 00:06:00,007 But, uh, so Archer has been around a long time and, um, you know, it's 100 00:06:00,007 --> 00:06:04,387 been sold and, and, and I don't know what, you know, spun back out 101 00:06:04,442 --> 00:06:05,262 Prasanna Malaiyandi: equity or 102 00:06:05,502 --> 00:06:05,582 whatever. 103 00:06:05,877 --> 00:06:06,267 Yeah. 104 00:06:06,397 --> 00:06:06,637 W. Curtis Preston: Yeah. 105 00:06:06,637 --> 00:06:07,717 A private equity firm. 106 00:06:07,717 --> 00:06:09,127 They got bought by ca. 107 00:06:09,427 --> 00:06:13,537 They got spun back out with a private equity firm and then, uh, they 108 00:06:13,537 --> 00:06:19,837 acquired or they merged, Arcserve merged with StorageCraft back in 2021 109 00:06:20,827 --> 00:06:24,487 and then this story unfortunately happens not long after that. 110 00:06:24,487 --> 00:06:28,807 I think it was about five months after the acquisition what ended up happening. 111 00:06:28,807 --> 00:06:29,277 Prasanna. 112 00:06:29,602 --> 00:06:29,932 Prasanna Malaiyandi: Yeah. 113 00:06:29,932 --> 00:06:34,012 So like you had alluded to earlier, just like Carbonite, uh, 114 00:06:34,312 --> 00:06:37,582 they lost customer data, right? 115 00:06:37,612 --> 00:06:41,327 Um, but the way it happened was slightly different. 116 00:06:41,362 --> 00:06:45,442 So as part of this, they were moving from their own facilities 117 00:06:45,442 --> 00:06:48,952 and migrating to Google Cloud. 118 00:06:49,687 --> 00:06:53,827 Um, and they accidentally decommissioned a server before 119 00:06:53,827 --> 00:06:55,477 it was already up in the cloud. 120 00:06:56,317 --> 00:06:58,057 And it wasn't just any server. 121 00:06:58,062 --> 00:07:01,687 It was a server that contained metadata about the backups. 122 00:07:03,637 --> 00:07:04,627 W. Curtis Preston: Yeah, 123 00:07:04,777 --> 00:07:05,407 Prasanna Malaiyandi: Yeah, 124 00:07:05,472 --> 00:07:06,512 W. Curtis Preston: metadata is important, 125 00:07:09,382 --> 00:07:13,162 Prasanna Malaiyandi: Yeah, so they basically, um, had the same situation as 126 00:07:13,162 --> 00:07:16,582 what we had talked about with Carbonite, where the customers lost their data. 127 00:07:16,582 --> 00:07:21,922 Now, I think unlike Carbonite, I think that, like you said, they did 128 00:07:21,927 --> 00:07:25,132 a much better job of accepting blame. 129 00:07:25,882 --> 00:07:29,902 And trying to get their customers or their end users back up and 130 00:07:29,902 --> 00:07:31,762 running as quickly as possible. 131 00:07:31,762 --> 00:07:36,182 Doing things like receding and really going the extra distance to get them 132 00:07:36,182 --> 00:07:39,812 up and running as quickly as possible, including things like, Hey, if you 133 00:07:39,812 --> 00:07:43,562 have more than three terabytes, we're gonna ship you a drive so you can 134 00:07:43,562 --> 00:07:47,282 upload to the drive and then send it to help speed up the process. 135 00:07:48,932 --> 00:07:51,632 W. Curtis Preston: Yeah, so for those not familiar with this process, 136 00:07:51,632 --> 00:07:56,642 when you are a cloud backup vendor, the, you know, the hardest part 137 00:07:56,642 --> 00:07:58,862 is that initial backup, right? 138 00:07:58,862 --> 00:07:59,552 You know, it's one 139 00:07:59,702 --> 00:07:59,882 Prasanna Malaiyandi: So 140 00:07:59,887 --> 00:08:00,422 fast. 141 00:08:01,202 --> 00:08:01,472 W. Curtis Preston: Yeah. 142 00:08:01,967 --> 00:08:02,087 Yeah. 143 00:08:02,222 --> 00:08:02,612 Damn. 144 00:08:02,612 --> 00:08:03,182 Physics. 145 00:08:03,662 --> 00:08:04,472 Uh, yeah. 146 00:08:04,472 --> 00:08:07,772 So when you, you know, if, if you're back, you can back up. 147 00:08:07,922 --> 00:08:11,342 I mean, you know, I remember, uh, you know, when I worked 148 00:08:11,342 --> 00:08:14,462 at Druva, we had really, you know, multi petabyte customers. 149 00:08:14,672 --> 00:08:17,792 You can back up that amount of data over the internet, but that 150 00:08:17,792 --> 00:08:20,762 first backup is gonna be something. 151 00:08:20,942 --> 00:08:23,652 And that's why we used the snowball edge. 152 00:08:23,652 --> 00:08:27,252 which was a, an appliance that, that you would ship to them and, you know, 153 00:08:27,252 --> 00:08:28,902 it already had the technology on there. 154 00:08:28,902 --> 00:08:31,512 And then they would back up to that and then that would ship and it was 155 00:08:31,512 --> 00:08:36,642 just mu, you know, basically a very fancy sneaker net was the way to 156 00:08:36,647 --> 00:08:39,072 get the very first copy up there. 157 00:08:39,582 --> 00:08:43,872 And you needed an, you needed one of those snowball edges for I think about 158 00:08:43,872 --> 00:08:46,662 every 75 terabytes, something like that. 159 00:08:47,292 --> 00:08:50,112 And, um, then. 160 00:08:50,532 --> 00:08:52,242 And so that's what they're saying here. 161 00:08:52,482 --> 00:08:55,542 And that by the way, that is referred to as the seed for those that don't 162 00:08:55,572 --> 00:08:59,112 live in that world, that initial backup is referred to the seed. 163 00:08:59,112 --> 00:09:02,632 So when you read the story, you're right, they did accept 164 00:09:02,632 --> 00:09:05,872 responsibility and they said 165 00:09:05,872 --> 00:09:08,332 the first thing that we need to get going is that seed. 166 00:09:08,782 --> 00:09:11,422 And so, you know, you were talking about the, one of the way, just 167 00:09:11,422 --> 00:09:15,532 like I said, the, the way, uh, my former employer would do it was they 168 00:09:15,532 --> 00:09:17,242 would ship, uh, a snowball edge. 169 00:09:17,512 --> 00:09:20,602 They're shipping whatever they need to ship to these customers 170 00:09:20,932 --> 00:09:22,432 in order to make them whole. 171 00:09:22,912 --> 00:09:27,442 One additional wrinkle, I think of this was that this was 172 00:09:27,442 --> 00:09:31,102 specifically for their MSP customers. 173 00:09:31,567 --> 00:09:38,077 Um, and I don't know what percentage of their business, uh, is via MSPs, but 174 00:09:38,107 --> 00:09:43,237 that does, that does make an additional wrinkle in that not only are they 175 00:09:43,237 --> 00:09:45,757 impacting their customer, which is the Ms. 176 00:09:45,762 --> 00:09:49,537 P, they're impacting the end customer, which is the MSPs customer. 177 00:09:50,527 --> 00:09:51,127 Um. 178 00:09:52,252 --> 00:09:56,422 But again, I, you know, I want to, uh, you know, give a shout out, 179 00:09:56,482 --> 00:10:00,352 um, to the CEO, Brandon Lacey. 180 00:10:00,482 --> 00:10:04,202 I like that he, he doesn't try to pass the blame. 181 00:10:04,202 --> 00:10:07,652 He doesn't try to, you know, I, I don't know. 182 00:10:08,367 --> 00:10:08,487 I. 183 00:10:08,852 --> 00:10:11,042 Maybe it's because he didn't have a choice, right? 184 00:10:11,042 --> 00:10:14,252 You know, when you know the other vendor, they're saying, Hey, it was our, it 185 00:10:14,252 --> 00:10:16,742 was our storage array, uh, that failed. 186 00:10:16,747 --> 00:10:20,492 And that's why, you know, that's why our backups died in this case. 187 00:10:20,672 --> 00:10:24,332 They didn't really add anything to blame it on other than, dang 188 00:10:24,332 --> 00:10:26,072 it, why is my fingers here? 189 00:10:26,762 --> 00:10:30,842 They didn't really have anything other to blame it on other than human failure. 190 00:10:31,397 --> 00:10:36,827 Because it was a human that prematurely decommissioned a server before 191 00:10:36,827 --> 00:10:38,567 that server had been migrated. 192 00:10:38,792 --> 00:10:39,002 Prasanna Malaiyandi: Yeah. 193 00:10:39,752 --> 00:10:44,042 And that's just sort of a breakdown of processes, right? 194 00:10:44,042 --> 00:10:46,232 It's not a technology issue. 195 00:10:46,232 --> 00:10:46,562 Right? 196 00:10:46,562 --> 00:10:51,152 And I'm sure now they have a process in place where they're like, Hey, make 197 00:10:51,152 --> 00:10:54,692 sure that the data validate that the server is actually up in wherever it 198 00:10:54,692 --> 00:10:56,552 needs to be before you decommission. 199 00:10:58,672 --> 00:10:59,162 Because. 200 00:10:59,192 --> 00:11:02,042 W. Curtis Preston: Yeah, that is, that is probably the 201 00:11:02,042 --> 00:11:03,482 biggest disappointment, right? 202 00:11:03,482 --> 00:11:09,062 Is you would think you're doing this like major migration. 203 00:11:09,062 --> 00:11:17,327 I remember there was Datto, they did an exfiltration out of, um, Amazon and they 204 00:11:17,327 --> 00:11:21,707 built their own private data center that was a giant, and they said they believed 205 00:11:21,707 --> 00:11:25,997 at the time, and I believe this was Datto, they believed at the time it was 206 00:11:25,997 --> 00:11:34,397 the single largest exfiltration project ever to leave en mass, uh, from AWS. 207 00:11:35,447 --> 00:11:37,757 And it apparently went flawlessly. 208 00:11:38,237 --> 00:11:43,457 You, you've got to, you know, when you're doing a migration, it's like, 209 00:11:43,757 --> 00:11:45,887 you've gotta have that check, right? 210 00:11:45,947 --> 00:11:46,997 You've gotta have that check. 211 00:11:46,997 --> 00:11:48,677 We're gonna kill the thing. 212 00:11:48,917 --> 00:11:50,957 In fact, I wouldn't even kill the thing, 213 00:11:51,227 --> 00:11:52,817 Prasanna Malaiyandi: You would just sort of, yeah. 214 00:11:53,267 --> 00:11:56,117 W. Curtis Preston: I would, I would, uh, turn it off, 215 00:11:56,312 --> 00:11:56,532 Prasanna Malaiyandi: Yep. 216 00:11:57,017 --> 00:11:57,497 W. Curtis Preston: right? 217 00:11:57,527 --> 00:12:00,647 I, I would, in fact, I remember, um. 218 00:12:01,157 --> 00:12:05,117 I remember the very first migration that I had anything to do with, and 219 00:12:05,117 --> 00:12:10,877 I'm gonna, for those of you for the fellow gray hairs in the crowd, we were 220 00:12:10,877 --> 00:12:20,177 migrating off, uh, an at and T three B two 4,000, which was the at t's attempt 221 00:12:20,177 --> 00:12:23,507 at a multi-processing server, right? 222 00:12:23,507 --> 00:12:26,027 So the three B two 1000 was the. 223 00:12:26,477 --> 00:12:30,977 First computer ever built specifically for Unix. 224 00:12:31,067 --> 00:12:31,577 Okay. 225 00:12:32,102 --> 00:12:34,172 Prasanna Malaiyandi: I love how you just have like these model numbers. 226 00:12:34,177 --> 00:12:37,832 Just like right off the tongue, it just rolls right off. 227 00:12:37,832 --> 00:12:40,412 You're like, oh yeah, everyone should know about the 3D two. 228 00:12:40,742 --> 00:12:41,912 It's like R 2D two. 229 00:12:42,002 --> 00:12:42,812 you might as well call it that. 230 00:12:43,157 --> 00:12:43,607 W. Curtis Preston: Yeah. 231 00:12:44,057 --> 00:12:44,357 What's that? 232 00:12:44,732 --> 00:12:46,052 Prasanna Malaiyandi: You might as well call it R 2D two. 233 00:12:47,342 --> 00:12:50,792 W. Curtis Preston: Yeah, so the three B two 1000 was the first, you know, 'cause 234 00:12:50,792 --> 00:12:56,342 I think it was the PDP 11 was the one that, uh, that Unix initially ran on. 235 00:12:56,732 --> 00:13:00,062 But the three B two was the first computer specifically designed to run. 236 00:13:00,467 --> 00:13:06,227 Unix and it ran, uh, system five and, um, you know, directly from 237 00:13:06,227 --> 00:13:12,407 Bell Labs, right at and t and the three B two 4,000 was, you know, they 238 00:13:12,407 --> 00:13:15,497 attempted a multi-processing machine and it, it just, it wasn't very good. 239 00:13:15,677 --> 00:13:19,217 And so we were trying to migrate off of that, and I remember that. 240 00:13:19,752 --> 00:13:23,952 Basically what we would do is we would just kill things as we 241 00:13:23,952 --> 00:13:25,812 believed we had migrated it off. 242 00:13:25,812 --> 00:13:28,152 We would, we would turn off a database. 243 00:13:28,152 --> 00:13:29,652 We would turn off a server. 244 00:13:29,747 --> 00:13:30,302 Prasanna Malaiyandi: see what broke. 245 00:13:30,642 --> 00:13:34,302 W. Curtis Preston: That's the, and well, yeah, and then see if anything broke. 246 00:13:34,302 --> 00:13:34,842 Right. 247 00:13:34,992 --> 00:13:36,582 You don't, I, I don't know. 248 00:13:36,582 --> 00:13:40,722 I, I guess that's the one surprise here, is that they, they decommission 249 00:13:40,722 --> 00:13:46,302 the server to the point of actually, you know, deleting it somehow. 250 00:13:46,487 --> 00:13:49,277 Prasanna Malaiyandi: Or, or, I, I know they talk about decommissioning. 251 00:13:49,277 --> 00:13:55,397 I do wonder if it's sort of, um, because it's metadata, I wonder if it's like 252 00:13:55,402 --> 00:14:00,887 a distributed system and maybe they killed the node, uh, without, and so 253 00:14:01,337 --> 00:14:05,357 since it didn't have access to the metadata, it's sort of like maybe 254 00:14:05,357 --> 00:14:06,737 they weren't expecting that to happen. 255 00:14:08,387 --> 00:14:08,747 Right? 256 00:14:08,747 --> 00:14:09,797 So even though they say 257 00:14:09,842 --> 00:14:10,922 W. Curtis Preston: Yeah, I, yeah. 258 00:14:11,057 --> 00:14:12,227 Prasanna Malaiyandi: I don't know the details. 259 00:14:12,227 --> 00:14:12,617 So. 260 00:14:13,397 --> 00:14:14,657 W. Curtis Preston: Yeah, we don't know the details. 261 00:14:14,662 --> 00:14:17,297 We're Monday morning quarterbacking, et cetera. 262 00:14:17,777 --> 00:14:20,597 Um, but let's, but let's just talk about it. 263 00:14:20,597 --> 00:14:25,007 When you are doing a migration, you don't kill the thing. 264 00:14:25,757 --> 00:14:28,217 You don't, you don't kill the old one. 265 00:14:28,457 --> 00:14:31,307 You turn off the old thing, right? 266 00:14:31,397 --> 00:14:37,457 You turn it off and then you wait a while before you, uh, actually 267 00:14:37,487 --> 00:14:41,027 redeploy it, delete it, whatever. 268 00:14:41,267 --> 00:14:41,897 And then I would. 269 00:14:42,662 --> 00:14:46,922 Then I, you know, I, I find myself wondering, well, I, I guess there wasn't, 270 00:14:47,822 --> 00:14:49,532 there wasn't a backup of that thing. 271 00:14:49,532 --> 00:14:54,332 There wasn't a backup of the backup, which is not that uncommon 272 00:14:54,362 --> 00:14:54,582 Prasanna Malaiyandi: Yep. 273 00:14:55,452 --> 00:14:55,872 Backup 274 00:14:56,072 --> 00:14:57,272 W. Curtis Preston: uh, backup. 275 00:14:57,632 --> 00:14:58,202 Backup. 276 00:14:58,202 --> 00:14:58,232 I. 277 00:14:59,007 --> 00:15:00,087 Services. 278 00:15:00,627 --> 00:15:01,137 Right? 279 00:15:01,557 --> 00:15:09,117 Um, and as a result, when that system was deleted, it, there was no backup 280 00:15:09,117 --> 00:15:13,317 of that backup, which I think many people might be surprised by that. 281 00:15:13,317 --> 00:15:13,502 What do you think? 282 00:15:14,367 --> 00:15:14,607 Prasanna Malaiyandi: No. 283 00:15:14,697 --> 00:15:21,177 Uh, I think people will be surprised because they think that backup 284 00:15:21,182 --> 00:15:23,007 systems are highly resilient. 285 00:15:24,237 --> 00:15:27,927 And of course someone's gonna do a backup of a backup, but you have to also realize 286 00:15:27,932 --> 00:15:32,037 that there's additional costs associated with doing a backup of a backup. 287 00:15:32,817 --> 00:15:33,567 And. 288 00:15:34,467 --> 00:15:38,877 Usually backup systems, people are trying to cut costs, right? 289 00:15:39,147 --> 00:15:40,827 Because this is like the tertiary copy. 290 00:15:40,827 --> 00:15:47,067 And so having high availability or a replica copy of your backup system, unless 291 00:15:47,067 --> 00:15:50,517 it's critical, which a lot of systems are, and people do replicate their backup 292 00:15:50,522 --> 00:15:54,717 systems, but not everyone could afford to do that, and especially given that this 293 00:15:54,717 --> 00:16:00,567 was for their MSP customers who are also trying to make money and add value, right? 294 00:16:00,627 --> 00:16:02,637 They're looking at probably a. 295 00:16:03,387 --> 00:16:06,057 Reasonable offering, right, 296 00:16:06,147 --> 00:16:07,857 uh, without all the bells and whistles. 297 00:16:09,597 --> 00:16:10,047 W. Curtis Preston: Yeah. 298 00:16:10,437 --> 00:16:18,747 And in a world where services are often purchased on price. 299 00:16:19,272 --> 00:16:25,572 Not necessarily alone, but where price is the primary motivator For many 300 00:16:25,572 --> 00:16:33,492 customers, it's, this is one of those things where no one ever bought Salesforce 301 00:16:33,492 --> 00:16:34,842 because they had a really good backup. 302 00:16:35,172 --> 00:16:37,392 No one ever bought Oracle because it had a really good backup. 303 00:16:37,392 --> 00:16:41,292 They bought Oracle because it had really good features and did 304 00:16:41,292 --> 00:16:42,702 the thing they needed it to do. 305 00:16:42,762 --> 00:16:46,182 Nobody's nobody, there's, there's nowhere, no ad in the world. 306 00:16:46,662 --> 00:16:49,962 Said, buy Oracle, because our backup is really good. 307 00:16:50,292 --> 00:16:50,652 Right? 308 00:16:50,652 --> 00:16:52,452 And so the same thing is true here. 309 00:16:52,662 --> 00:16:56,712 Buy our backup service because we also have a tertiary copy of 310 00:16:56,712 --> 00:17:00,072 your data just in case we screw up the copy that you gave us. 311 00:17:00,942 --> 00:17:02,622 Nobody's gonna even know that. 312 00:17:03,297 --> 00:17:06,852 Um, and they're, and they're cer, they certainly don't want to pay for it. 313 00:17:06,852 --> 00:17:09,282 All they see is that this one is more expensive than that one. 314 00:17:09,372 --> 00:17:09,642 Prasanna Malaiyandi: Yeah. 315 00:17:09,972 --> 00:17:14,652 The one thing I am a little surprised about though is given this is a backup 316 00:17:14,652 --> 00:17:19,542 system, and it looks like they did have their metadata and data split apart just 317 00:17:19,812 --> 00:17:24,162 based on what we read in the articles, I'm surprised they didn't have a self 318 00:17:24,162 --> 00:17:27,402 describing format for the data itself. 319 00:17:28,162 --> 00:17:28,552 I know? 320 00:17:28,552 --> 00:17:33,742 there are other systems which tend to do that as sort of a fail safe, so I'm not 321 00:17:33,742 --> 00:17:36,742 necessarily going to create a copy of the data, but if I, my, something happens, 322 00:17:36,832 --> 00:17:41,122 my metadata or there's some corruption or whatever else, I should at least be 323 00:17:41,122 --> 00:17:45,352 able to build back all the metadata based on the data that I still have access to. 324 00:17:45,352 --> 00:17:45,412 I. 325 00:17:46,797 --> 00:17:48,057 W. Curtis Preston: Yeah, that's a good point. 326 00:17:48,057 --> 00:17:53,007 And, and that is a, another really good feature to have in a backup system 327 00:17:53,307 --> 00:17:57,897 that if the, basically the backup catalog, the image database, the, you 328 00:17:57,897 --> 00:18:02,697 know, the database of all the backups, if that gets messed up, it would be 329 00:18:02,702 --> 00:18:07,587 really nice if the, if the, uh, you know, if you could rebuild that. 330 00:18:07,947 --> 00:18:13,917 And I remember back when I was doing RFPs and the, one of the questions 331 00:18:13,917 --> 00:18:18,777 we asked was if the backup catalog, um, you know, gets corrupt, can, 332 00:18:18,782 --> 00:18:20,577 can we rebuild it given the backups? 333 00:18:20,847 --> 00:18:23,247 And I do remember at least one major vendor. 334 00:18:23,247 --> 00:18:25,557 I won't call 'em out by name 'cause it might not still be the 335 00:18:25,557 --> 00:18:27,207 answer, but their answer was. 336 00:18:27,707 --> 00:18:34,307 Um, you know what, they, literally, the phrase was, our backup format 337 00:18:34,517 --> 00:18:38,927 is so proprietary that even we can't read it if we don't know what's on it. 338 00:18:40,702 --> 00:18:43,817 And they, and they saw that as like a, like a feature. 339 00:18:43,942 --> 00:18:47,267 Uh, by the way, it turned out, interestingly enough, it turned out 340 00:18:47,267 --> 00:18:49,667 that, uh, it wasn't that proprietary and. 341 00:18:50,537 --> 00:18:54,047 Other vendors have figured out how to read that backup date. 342 00:18:54,107 --> 00:18:54,377 Yeah. 343 00:18:54,377 --> 00:18:57,467 As long as it's not, um, encrypted. 344 00:18:57,707 --> 00:18:58,097 Right. 345 00:18:58,307 --> 00:18:59,927 That that's a whole other, whole other thing. 346 00:18:59,977 --> 00:19:02,437 Prasanna Malaiyandi: So I don't think we've ever talked about 347 00:19:02,442 --> 00:19:04,387 this on the podcast before. 348 00:19:04,717 --> 00:19:08,677 And so is this something, I know we've talked previously about, Hey, here are 349 00:19:08,682 --> 00:19:13,507 some questions you should be asking your backup vendor when trying to figure 350 00:19:13,507 --> 00:19:14,917 out like what their solution does. 351 00:19:15,067 --> 00:19:18,547 Do you think this is worthwhile to throw into that list of questions? 352 00:19:21,102 --> 00:19:24,067 W. Curtis Preston: Of, do you have a backup of the backup? 353 00:19:24,472 --> 00:19:28,102 Prasanna Malaiyandi: Either a backup of the backup or if I lose 354 00:19:28,342 --> 00:19:32,452 part of my system, like my catalog, can I still recover my data? 355 00:19:34,537 --> 00:19:36,967 W. Curtis Preston: I, I do think that's a perfectly valid question to ask. 356 00:19:36,967 --> 00:19:37,297 Right. 357 00:19:37,597 --> 00:19:40,147 And I think the other question is perfectly valid as well of 358 00:19:40,177 --> 00:19:45,037 if, what protections do you, I would point at this story, what 359 00:19:45,037 --> 00:19:51,007 protections do you have against this happening in your environment? 360 00:19:51,007 --> 00:19:51,487 Right. 361 00:19:52,027 --> 00:19:52,687 Um, 362 00:19:53,077 --> 00:19:54,307 Prasanna Malaiyandi: because it may not be malicious. 363 00:19:54,307 --> 00:19:56,617 It's just accidental things happen. 364 00:19:56,617 --> 00:19:57,457 It's all software. 365 00:19:57,457 --> 00:19:59,707 People are writing software, right? 366 00:20:01,447 --> 00:20:05,287 W. Curtis Preston: Yeah, the, you know, so this, remind this, this story to 367 00:20:05,287 --> 00:20:10,172 me is a, is a combination of the KPMG story and the Carbonite story, right? 368 00:20:10,532 --> 00:20:14,612 Because like the KPMG story, and by the way, we have another episode on that. 369 00:20:14,617 --> 00:20:16,202 Please feel free to check that out. 370 00:20:17,132 --> 00:20:23,072 The KPMG story is one of failure of process as well, where. 371 00:20:23,717 --> 00:20:28,607 You'd like to think that at a, at a company that size and when you're 372 00:20:28,607 --> 00:20:33,167 doing something of this magnitude, I, you know, I, when I think back 373 00:20:33,172 --> 00:20:34,547 on that story, I don't think they. 374 00:20:34,907 --> 00:20:38,747 Thought this was of the, the, we're, I'm just trying to fix one 375 00:20:38,747 --> 00:20:41,447 person's, one person's account. 376 00:20:41,627 --> 00:20:46,157 They didn't think that what they were touching was a tool in this 377 00:20:46,157 --> 00:20:48,167 case, uh, retention policies. 378 00:20:48,827 --> 00:20:52,337 They were touching a tool that could wipe out everybody's data. 379 00:20:52,337 --> 00:20:52,727 They, they 380 00:20:52,787 --> 00:20:54,287 Prasanna Malaiyandi: Hundred 40,000 people. 381 00:20:54,977 --> 00:20:59,312 W. Curtis Preston: 140,000 people's data with no backup. 382 00:20:59,492 --> 00:21:01,052 Uh, that's a great, that's a great story. 383 00:21:01,052 --> 00:21:01,832 Check that one out. 384 00:21:02,612 --> 00:21:06,632 But I, the, the big thing with processes, you've, you've got to have 385 00:21:06,632 --> 00:21:10,922 a very, uh, you know, the bigger the project, the, the bigger the process. 386 00:21:11,252 --> 00:21:13,297 And I do remember, I. 387 00:21:13,592 --> 00:21:19,262 When the, the biggest one I remember being involved in it was, it, was, it, it wasn't 388 00:21:19,262 --> 00:21:22,502 a migration, it was a physical move. 389 00:21:23,102 --> 00:21:28,352 We moved, a bunch of servers from one data center to another data center, 390 00:21:28,782 --> 00:21:32,352 We had built a nice new big data center and we're gonna move the servers from the 391 00:21:32,352 --> 00:21:34,392 old data center to the new data center. 392 00:21:35,082 --> 00:21:40,422 And we had a massive process and we had tested it from one end to the other, 393 00:21:40,542 --> 00:21:43,002 and we outsourced whatever we could. 394 00:21:43,002 --> 00:21:44,112 So like we hired. 395 00:21:44,232 --> 00:21:49,632 A professional moving company to move the servers we hired, uh, and paid extra 396 00:21:49,632 --> 00:21:55,452 to have HP come in and, and, you know, unplug the servers and go ominous dominus, 397 00:21:55,692 --> 00:21:57,762 you know, you may touch this server. 398 00:21:58,032 --> 00:21:58,512 Right. 399 00:21:58,812 --> 00:22:01,572 Um, and then, and then it got handed to the movers and stuff 400 00:22:01,572 --> 00:22:03,102 and we had this big process. 401 00:22:03,492 --> 00:22:08,722 Um, but there was one major failing and that was from the old 402 00:22:08,722 --> 00:22:10,132 server to the new server room. 403 00:22:10,587 --> 00:22:12,777 The new server room had, um, Twistlock. 404 00:22:13,912 --> 00:22:17,152 You know, plugs under, underneath the floor. 405 00:22:17,422 --> 00:22:20,362 And the old server room had the old standard, uh, north 406 00:22:20,362 --> 00:22:22,312 American, uh, two-prong plugs. 407 00:22:22,942 --> 00:22:27,292 And, uh, that was the, that was one day where I truly saved the day because 408 00:22:27,292 --> 00:22:33,592 I had, uh, oh, and the electricians, that would be the ones to help us. 409 00:22:33,712 --> 00:22:37,522 Were busy at the CEO's house, putting up his Christmas lights. 410 00:22:38,342 --> 00:22:44,162 Um, and so, and so they didn't think we'd need electricians during our big move. 411 00:22:44,762 --> 00:22:46,922 And I knew how to fix this problem, right? 412 00:22:46,927 --> 00:22:49,172 I, I'm like, look, we can do this. 413 00:22:49,682 --> 00:22:52,862 I just need to go to a gray bar, uh, electrical store. 414 00:22:52,862 --> 00:22:55,622 And there's one down the road and it was closing in. 415 00:22:56,212 --> 00:23:00,382 20 minutes, uh, by the time we realized what we needed to do, and I, I got in my 416 00:23:00,382 --> 00:23:05,272 car with, with my boss and, and he looked at me and he says, I will pay the ticket. 417 00:23:05,272 --> 00:23:06,202 Just get us there. 418 00:23:06,592 --> 00:23:07,852 And we drove. 419 00:23:07,852 --> 00:23:09,292 I drove and it was raining. 420 00:23:09,652 --> 00:23:12,112 It was, I mean, there was water everywhere. 421 00:23:12,112 --> 00:23:15,172 I'm driving to this gray bar and I've got it. 422 00:23:15,177 --> 00:23:19,282 It's, it's, it's closing in 20 minutes and it's 15 minutes away. 423 00:23:19,282 --> 00:23:19,552 Right. 424 00:23:19,552 --> 00:23:21,082 I don't have time to play. 425 00:23:22,282 --> 00:23:26,032 And I remember hydroplaning going around a big semi, and I remember 426 00:23:26,032 --> 00:23:29,842 like, you know, swerving around a corner and all this stuff. 427 00:23:29,842 --> 00:23:33,862 And I remember that when we got there, there's that scene in, uh, planes, trains, 428 00:23:33,862 --> 00:23:38,812 and Automobiles where, where Steve, Steve Martin's fingers are stuck in the dash. 429 00:23:39,172 --> 00:23:42,922 Uh, that was the way, uh, Lou was when we, when we got there. 430 00:23:43,342 --> 00:23:47,782 But we got, we walked up to the door and it was five minutes prior to the, 431 00:23:47,842 --> 00:23:50,542 you know, prior to the, the closing. 432 00:23:51,397 --> 00:23:53,047 They were, the door was locked. 433 00:23:53,227 --> 00:23:57,877 And, uh, and, and they, they pointed, they pointed at her watch. 434 00:23:57,877 --> 00:24:00,397 They're like, they're pointing at their watch as a time. 435 00:24:00,397 --> 00:24:03,547 And I pointed up at the clock on their wall and I was like, 436 00:24:03,607 --> 00:24:05,467 no, it's not one o'clock yet. 437 00:24:05,467 --> 00:24:06,607 Open your damn door. 438 00:24:07,327 --> 00:24:12,397 And then we, you know, uh, and then I think, I think I like, I was like, please, 439 00:24:12,457 --> 00:24:15,427 please, you know, please let me in. 440 00:24:15,427 --> 00:24:15,967 And we got it. 441 00:24:15,972 --> 00:24:17,017 And we bought all the stuff. 442 00:24:17,257 --> 00:24:19,087 And I, I non. 443 00:24:20,032 --> 00:24:26,422 Traded non whatever electrician, but I was trained as one in the Navy, um, which at 444 00:24:26,422 --> 00:24:28,792 that time was only like five years prior. 445 00:24:29,632 --> 00:24:33,682 Um, and then we wired adapters from. 446 00:24:34,822 --> 00:24:37,072 A to B and save the day anyway. 447 00:24:37,102 --> 00:24:39,532 Yeah, you gotta test your process and you gotta be ready 448 00:24:39,537 --> 00:24:41,062 for failures in your process. 449 00:24:41,992 --> 00:24:48,652 But I, I think that, you know, so the two big this, this one I feel very different. 450 00:24:48,652 --> 00:24:51,232 Like I said in the beginning, I feel very different about 451 00:24:51,232 --> 00:24:52,702 this one versus the carbonite 452 00:24:52,912 --> 00:24:53,392 Prasanna Malaiyandi: but why 453 00:24:54,292 --> 00:24:58,162 W. Curtis Preston: because they didn't try to, they didn't try to pass the buck. 454 00:25:01,612 --> 00:25:05,452 Prasanna Malaiyandi: is that really the only difference in your mind is, 455 00:25:06,082 --> 00:25:08,962 well, I think also in the Carbonite case, the only reason we heard about 456 00:25:08,962 --> 00:25:10,492 it is when they try to sue the vendor. 457 00:25:13,942 --> 00:25:14,722 W. Curtis Preston: Yeah. 458 00:25:14,812 --> 00:25:16,432 Um, this. 459 00:25:16,792 --> 00:25:17,902 Yeah, that's a good question. 460 00:25:17,902 --> 00:25:25,162 If, would we have ever heard about the story if they, I, I don't know. 461 00:25:25,207 --> 00:25:29,362 I, I, I'd like to think that it would still have made news, but all we 462 00:25:29,362 --> 00:25:32,302 heard about was the fact that they store, they sued their storage vendor 463 00:25:32,812 --> 00:25:33,592 in this case. 464 00:25:33,632 --> 00:25:35,402 The article, right. 465 00:25:35,492 --> 00:25:36,212 Um. 466 00:25:37,202 --> 00:25:40,562 It starts out with this quote from the CEO, right? 467 00:25:40,922 --> 00:25:43,832 In terms of magnitude is the only thing I'm thinking about waking 468 00:25:43,832 --> 00:25:45,272 up and going to bed every night. 469 00:25:45,632 --> 00:25:49,172 It's orders of magnitude more important than anything else that I have going 470 00:25:49,172 --> 00:25:51,182 on in terms of operating this business. 471 00:25:51,782 --> 00:25:57,002 Um, says the CEO right, uh, of this company trying to bounce back. 472 00:25:57,332 --> 00:26:03,152 Now we haven't, I tried to find, I didn't find, this was almost two years ago now. 473 00:26:03,452 --> 00:26:04,742 I didn't see. 474 00:26:05,267 --> 00:26:07,007 An update on what happened. 475 00:26:07,007 --> 00:26:08,867 They were saying, we're trying to get people up and running. 476 00:26:08,927 --> 00:26:12,857 I didn't see any stories like we did with the Carbonite about some 477 00:26:12,857 --> 00:26:16,727 customers that actually lost data because they had to restore something 478 00:26:16,727 --> 00:26:18,437 while they were trying to be seated. 479 00:26:19,337 --> 00:26:20,597 Uh, we don't have any of those 480 00:26:20,832 --> 00:26:23,802 Prasanna Malaiyandi: Now, I know you're a big Reddit fan, and I know you 481 00:26:23,802 --> 00:26:25,872 probably were browsing threads about this. 482 00:26:26,202 --> 00:26:29,892 Did you see any, like, what was the sentiment in those threads 483 00:26:29,892 --> 00:26:34,302 from end users, customers that, if 484 00:26:34,302 --> 00:26:34,692 you can 485 00:26:34,787 --> 00:26:36,647 W. Curtis Preston: that's a, actually, that's a good one. 486 00:26:36,737 --> 00:26:38,867 Let me, uh, let me pull up that thread. 487 00:26:39,047 --> 00:26:41,687 'cause I didn't, I didn't look too much at the thread. 488 00:26:41,687 --> 00:26:42,407 This is funny. 489 00:26:42,647 --> 00:26:46,157 Seems like a backup company that loses backups is no longer a backup 490 00:26:46,157 --> 00:26:48,167 company, but a data disposal service. 491 00:26:48,377 --> 00:26:49,757 I see a pivot opportunity. 492 00:26:49,757 --> 00:26:52,997 They should certify these backups as destroyed and issued certificates. 493 00:26:52,997 --> 00:26:58,067 I'd say the Reddit response is typical of every response with a cloud outage. 494 00:26:58,067 --> 00:27:01,187 There are people that are just, you know, saying, ah, the cloud sucks, and, 495 00:27:01,187 --> 00:27:04,697 you know, and the cloud is just, it's just another computer, it's just another 496 00:27:04,702 --> 00:27:08,027 service, and you have to verify it just like you verify every other system. 497 00:27:08,477 --> 00:27:08,897 Right. 498 00:27:09,377 --> 00:27:13,907 Um, and there, there were some comments in the thread saying that. 499 00:27:14,047 --> 00:27:18,247 You know, one of the reasons that Arc Serve and StorageCraft, uh, merged was 500 00:27:18,247 --> 00:27:21,367 that they had been going down for a while in terms of quality of service. 501 00:27:21,367 --> 00:27:26,437 Again, these are pe other people's comments, um, and that they, that they 502 00:27:26,437 --> 00:27:28,417 weren't that surprised at, at this event. 503 00:27:29,017 --> 00:27:31,927 But, um, they're still around. 504 00:27:31,957 --> 00:27:33,067 They're still running. 505 00:27:33,157 --> 00:27:36,427 Um, just like Carbonite is still around and still running. 506 00:27:37,057 --> 00:27:39,727 I, I, I, you know, I wish them the best. 507 00:27:39,817 --> 00:27:42,397 Uh, this was a major migration fail. 508 00:27:43,687 --> 00:27:46,207 Prasanna Malaiyandi: Yeah, and it's also one of those things 509 00:27:46,212 --> 00:27:51,247 people have, like once you lose that trust, it get, it's difficult, 510 00:27:51,247 --> 00:27:54,367 especially in the space where you are that last line of defense. 511 00:27:55,867 --> 00:27:56,257 W. Curtis Preston: Yeah. 512 00:27:56,317 --> 00:27:58,112 I mean, people are still on LastPass. 513 00:28:00,442 --> 00:28:02,902 Prasanna Malaiyandi: Yeah, let's not go there. 514 00:28:03,397 --> 00:28:03,667 W. Curtis Preston: Yeah. 515 00:28:04,432 --> 00:28:05,122 What's that? 516 00:28:05,367 --> 00:28:05,887 Prasanna Malaiyandi: not go there. 517 00:28:07,132 --> 00:28:08,242 W. Curtis Preston: Yeah, yeah. 518 00:28:08,932 --> 00:28:09,592 Anyway. 519 00:28:09,652 --> 00:28:10,192 All right. 520 00:28:10,192 --> 00:28:14,482 Well, um, not, not a ton, not some huge lessons here, but the big one 521 00:28:14,482 --> 00:28:19,192 I think here is, you know, I, I like the Maya Culpa that the company did. 522 00:28:19,192 --> 00:28:19,222 I. 523 00:28:19,552 --> 00:28:23,092 Uh, and I like that they knew that their priority was getting the, 524 00:28:23,212 --> 00:28:26,542 the next backup, but unfortunately that's gonna take a long time. 525 00:28:26,992 --> 00:28:33,952 Um, and the, the biggest lesson here is if you're doing a migration of any 526 00:28:33,952 --> 00:28:41,452 kind, don't go shooting the other servers until you know that it's fully migrated. 527 00:28:41,932 --> 00:28:42,382 You know? 528 00:28:43,222 --> 00:28:43,672 Prasanna Malaiyandi: what, what? 529 00:28:43,672 --> 00:28:46,462 Is there any lessons learned for the end users? 530 00:28:49,147 --> 00:28:50,797 Like there's a, like what could they have done? 531 00:28:50,797 --> 00:28:54,697 I don't think there's anything that they could have done in this situation. 532 00:28:56,617 --> 00:29:00,187 W. Curtis Preston: yeah, the only, the only thing I, in terms of. 533 00:29:00,742 --> 00:29:04,432 Trying to identify, I mean, what's a vendor gonna say? 534 00:29:04,822 --> 00:29:06,982 This is the difficulty with the cloud. 535 00:29:07,072 --> 00:29:11,212 We've discussed this in previous other things, is it's hard to actually verify 536 00:29:11,212 --> 00:29:15,232 if you say, do you have processes in place for if somebody, if somebody over 537 00:29:15,232 --> 00:29:18,532 there accidentally, you know, blows up a server, do you have processes in 538 00:29:18,537 --> 00:29:22,612 place to be able to bring that server back or re of course they're gonna say, 539 00:29:22,612 --> 00:29:23,902 yeah, that's what they're gonna say. 540 00:29:25,237 --> 00:29:29,767 Um, the, the, the one thing you could think about, again, this is 541 00:29:29,767 --> 00:29:35,257 expensive, but people talk about using a service that puts stuff in more 542 00:29:35,257 --> 00:29:38,287 than one place or using more than one service, but people don't wanna pay 543 00:29:38,292 --> 00:29:41,017 for one backup service, let alone two. 544 00:29:42,187 --> 00:29:43,117 Could they do that? 545 00:29:43,117 --> 00:29:44,017 Should they do that? 546 00:29:44,017 --> 00:29:44,977 Sure. 547 00:29:45,247 --> 00:29:46,267 Will they do that? 548 00:29:46,267 --> 00:29:47,167 Nobody does that, 549 00:29:47,227 --> 00:29:47,497 Prasanna Malaiyandi: Yeah. 550 00:29:47,557 --> 00:29:49,867 No one's gonna spend the money in their budget on that. 551 00:29:51,427 --> 00:29:51,907 W. Curtis Preston: Yeah. 552 00:29:52,237 --> 00:29:53,047 Um. 553 00:29:53,377 --> 00:29:56,047 I mean, it's hard enough to get people to do backups in the first place. 554 00:29:56,047 --> 00:29:59,137 It's hard enough to get them to, to properly secure those backups. 555 00:30:00,157 --> 00:30:05,437 Um, they're not gonna pay for a completely separate, just in case they're, you know, 556 00:30:06,157 --> 00:30:08,737 uh, I will say having something like I. 557 00:30:09,217 --> 00:30:14,377 StorageCraft or carbonite, die and lose all your data is not 558 00:30:14,377 --> 00:30:15,997 the same as losing all your data. 559 00:30:16,357 --> 00:30:16,717 Right. 560 00:30:16,717 --> 00:30:18,997 It's, it's, it's all the backups of all your data. 561 00:30:19,267 --> 00:30:24,157 And there will be some, could be some ramifications to your business if you 562 00:30:24,157 --> 00:30:28,207 need to restore, but statistically speaking, you're not probably 563 00:30:28,212 --> 00:30:29,857 going to need to restore today. 564 00:30:30,247 --> 00:30:33,337 Um, just in terms of statistically Right. 565 00:30:33,847 --> 00:30:36,157 Um, it's not the same. 566 00:30:36,157 --> 00:30:37,177 So, you know, 567 00:30:37,477 --> 00:30:40,837 Prasanna Malaiyandi: So, okay, so if I was an end user, I get a 568 00:30:40,837 --> 00:30:44,287 notification from StorageCraft that says, Hey, by the way, we had an issue. 569 00:30:44,292 --> 00:30:45,367 We lost all your backups. 570 00:30:46,357 --> 00:30:49,027 They're gonna say, okay, I'm gonna try to help reseed. 571 00:30:49,057 --> 00:30:49,327 Right. 572 00:30:49,327 --> 00:30:52,027 Get all your data up there before that happens. 573 00:30:52,027 --> 00:30:54,937 Because like you said, it could take weeks, it could take months. 574 00:30:55,537 --> 00:30:55,897 Right. 575 00:30:56,572 --> 00:30:59,542 What is the first thing that the end user should do at that point? 576 00:30:59,932 --> 00:31:00,172 Right? 577 00:31:00,172 --> 00:31:03,832 Once they get that first notification to make sure that they don't end up 578 00:31:03,832 --> 00:31:07,402 losing all their data in case something happens to their production copy. 579 00:31:08,692 --> 00:31:12,382 W. Curtis Preston: Well, I, I'll say this, you got any big projects, 580 00:31:12,382 --> 00:31:15,802 uh, in process, maybe you should pause those projects, right? 581 00:31:15,982 --> 00:31:18,592 Any big changes, any big maintenance windows. 582 00:31:18,772 --> 00:31:21,082 Maybe you should put a pause on those maintenance windows. 583 00:31:21,592 --> 00:31:26,662 Uh, the other is you should go back to old school backups for the 584 00:31:26,667 --> 00:31:28,942 time being, for anything critical. 585 00:31:29,317 --> 00:31:35,017 Uh, you know, you should make a local backup of any kind that, a backup of any 586 00:31:35,017 --> 00:31:36,997 kind is better than a backup of no kind. 587 00:31:37,777 --> 00:31:39,607 Uh, that's a really good question, right. 588 00:31:39,997 --> 00:31:44,767 Um, I would put a pause on projects and I would see what I can do to get 589 00:31:44,767 --> 00:31:46,507 some local backups of stuff, right? 590 00:31:46,717 --> 00:31:49,027 There are plenty of open source tools out there. 591 00:31:49,597 --> 00:31:53,977 That can, you know, surely somewhere you've got some disc and then you've 592 00:31:53,977 --> 00:31:58,237 got some super critical servers that you really don't wanna lose. 593 00:31:58,417 --> 00:32:00,547 Get some local backups of those. 594 00:32:00,847 --> 00:32:02,042 Um, you know. 595 00:32:02,602 --> 00:32:04,552 Prasanna Malaiyandi: or I know you had mentioned in the past, right? 596 00:32:04,552 --> 00:32:07,942 You really like the iDrive service, cloud service, 597 00:32:08,442 --> 00:32:08,792 W. Curtis Preston: Right. 598 00:32:09,052 --> 00:32:09,382 Prasanna Malaiyandi: right. 599 00:32:09,382 --> 00:32:10,012 For just doing 600 00:32:10,062 --> 00:32:10,672 W. Curtis Preston: Well, the. 601 00:32:11,092 --> 00:32:11,632 Prasanna Malaiyandi: backup 602 00:32:12,082 --> 00:32:13,972 W. Curtis Preston: Yeah, the, the only problem with that is, you 603 00:32:13,972 --> 00:32:15,622 know, again, it's physics, right? 604 00:32:15,892 --> 00:32:18,622 So I'm just trying to, I'm just trying to think of something 605 00:32:18,622 --> 00:32:19,882 that you can do quickly. 606 00:32:20,482 --> 00:32:21,562 Um, you know 607 00:32:21,742 --> 00:32:23,032 Prasanna Malaiyandi: Copy it to a different machine. 608 00:32:23,032 --> 00:32:24,322 Go buy a couple external hard 609 00:32:24,562 --> 00:32:24,862 W. Curtis Preston: Yeah. 610 00:32:25,792 --> 00:32:30,472 Specifically for those super high value, super highly critical 611 00:32:30,472 --> 00:32:35,932 servers, you know, uh, things like backup servers for other customers. 612 00:32:38,347 --> 00:32:39,487 Uh uh. 613 00:32:40,147 --> 00:32:40,597 Yeah. 614 00:32:40,747 --> 00:32:41,257 Okay. 615 00:32:41,347 --> 00:32:44,917 Well, all right, well, thanks for helping me walk through that. 616 00:32:44,917 --> 00:32:46,447 That was a really good question there at the end. 617 00:32:46,447 --> 00:32:48,217 Prasanna, thanks for helping me walk through this. 618 00:32:48,502 --> 00:32:49,222 Prasanna Malaiyandi: No, no worries. 619 00:32:49,222 --> 00:32:53,812 That's why uh, I enjoy doing these podcasts 'cause it helps all of us, right? 620 00:32:53,812 --> 00:32:54,952 Just figure out what should we be 621 00:32:54,952 --> 00:32:56,452 doing in case this ever happens. 622 00:32:57,787 --> 00:32:59,917 W. Curtis Preston: That's what we're trying to do is to pass on these 623 00:32:59,917 --> 00:33:05,047 lessons to you, uh, the listener so that you can learn from other 624 00:33:05,047 --> 00:33:07,507 people's struggles, unfortunately. 625 00:33:08,017 --> 00:33:10,117 But, uh, you're why we do this. 626 00:33:10,117 --> 00:33:12,997 We want to turn you into a cyber recovery hero. 627 00:33:13,807 --> 00:33:14,527 That's a wrap 628 00:33:18,122 --> 00:33:22,802 The backup wrap up is written, recorded and produced by me w Curtis Preston. 629 00:33:23,372 --> 00:33:24,842 If you need backup or Dr. 630 00:33:24,842 --> 00:33:28,472 Consulting content generation or expert witness work, 631 00:33:28,772 --> 00:33:30,722 check out backup central.com. 632 00:33:31,472 --> 00:33:34,622 You can also find links from my O'Reilly Books on the same website. 633 00:33:35,252 --> 00:33:39,362 Remember, this is an independent podcast and any opinions that you 634 00:33:39,362 --> 00:33:41,252 hear are those of the speaker. 635 00:33:41,317 --> 00:33:43,177 And not necessarily an employer. 636 00:33:44,077 --> 00:33:44,797 Thanks for listening.