1 00:00:00,151 --> 00:00:05,551 A ransomware attack on Rackspace in 2023 left thousands of customers without 2 00:00:05,551 --> 00:00:11,191 access to their critical email data for months and led Rackspace to completely 3 00:00:11,191 --> 00:00:13,471 abandon the hosted exchange business line. 4 00:00:13,981 --> 00:00:18,481 On this week's episode of the backup wrap-up we discuss a detailed timeline 5 00:00:18,481 --> 00:00:22,381 of this event and most important, the lessons that we can learn from it. 6 00:00:22,578 --> 00:00:26,388 The incident in this episode is one of the many stories that are behind 7 00:00:26,388 --> 00:00:30,618 the recommendations that you may have heard from me throughout the years. 8 00:00:30,918 --> 00:00:33,438 I'm w your Curtis Preston, AKA Mr. 9 00:00:33,438 --> 00:00:34,008 Backup. 10 00:00:34,308 --> 00:00:37,698 And there's a reason I'm so passionate about this subject. 11 00:00:38,148 --> 00:00:42,858 It's because in my first job as a backup admin, my company lost an important 12 00:00:42,858 --> 00:00:45,408 database and I couldn't restore it. 13 00:00:45,948 --> 00:00:49,428 Since that moment, I've dedicated my career to making sure that 14 00:00:49,428 --> 00:00:51,198 would never again, happen to me. 15 00:00:51,528 --> 00:00:53,868 Or anyone who bothers to listen to me? 16 00:00:54,858 --> 00:00:59,538 We take unappreciated backup admins and turn them into cyber recovery heroes. 17 00:00:59,748 --> 00:01:01,938 This is the backup wrap up. 18 00:01:14,898 --> 00:01:15,948 W. Curtis Preston: Welcome to the show. 19 00:01:16,533 --> 00:01:19,083 I'm your host, w Curtis Preston, AKA, Mr. 20 00:01:19,083 --> 00:01:24,263 Backup, and with me, I have my consultant that will help reduce 21 00:01:24,263 --> 00:01:27,803 my level of starstruck today. 22 00:01:28,133 --> 00:01:29,063 I'm hoping 23 00:01:29,138 --> 00:01:30,338 Prasanna Malaiyandi: I don't think that's possible. 24 00:01:32,158 --> 00:01:32,603 W. Curtis Preston: I'm gonna be, 25 00:01:34,433 --> 00:01:35,213 Prasanna Malaiyandi: That sound. 26 00:01:35,453 --> 00:01:39,113 So yes, I think you should tell people who may not have caught that. 27 00:01:39,323 --> 00:01:40,403 Who are you gonna go see today? 28 00:01:41,378 --> 00:01:43,838 W. Curtis Preston: I'm gonna meet William Shatner today. 29 00:01:44,498 --> 00:01:44,918 I am 30 00:01:45,323 --> 00:01:45,653 Prasanna Malaiyandi: Are you 31 00:01:45,728 --> 00:01:47,198 W. Curtis Preston: super psyched. 32 00:01:47,228 --> 00:01:49,628 Yeah, I, I actually bought it. 33 00:01:49,628 --> 00:01:50,318 There's an event. 34 00:01:50,323 --> 00:01:54,578 There's a, there's a premiere of this new documentary that's about William Shatner. 35 00:01:54,788 --> 00:01:59,078 Um, and it's in la It's, and it's, uh, they're gonna do the screening. 36 00:01:59,078 --> 00:02:01,298 They're gonna do q and a, and then there is a. 37 00:02:01,913 --> 00:02:06,983 Uh, birthday party for him, his 93rd birthday party for him, uh, afterwards. 38 00:02:06,983 --> 00:02:12,983 And it's being held in the original, um, in the studio where 39 00:02:12,988 --> 00:02:14,723 they originally filmed the pilot. 40 00:02:15,233 --> 00:02:22,013 Um, and so it, I'll also be meeting, uh, Kevin Smith and, um, so. 41 00:02:22,633 --> 00:02:24,073 Hopefully I will. 42 00:02:24,373 --> 00:02:27,343 My dream, if I can, if I can get a selfie with William Shatner, 43 00:02:27,343 --> 00:02:29,083 that'll be, you know, um, 44 00:02:29,213 --> 00:02:30,088 Prasanna Malaiyandi: You'll be over the moon. 45 00:02:30,313 --> 00:02:31,933 W. Curtis Preston: that'll be, I'll be over the moon. 46 00:02:31,938 --> 00:02:32,263 Yeah. 47 00:02:32,263 --> 00:02:33,253 I've already met. 48 00:02:34,058 --> 00:02:37,028 I met, um, deforest Kelly. 49 00:02:37,028 --> 00:02:39,188 I met, uh, Michelle Nichols. 50 00:02:39,188 --> 00:02:44,468 I met, uh, George Decay and this will be, um, there is one 51 00:02:44,468 --> 00:02:48,068 remaining, uh, star, original Star Trek member that's still alive. 52 00:02:48,068 --> 00:02:48,818 Walter Koenig. 53 00:02:49,268 --> 00:02:53,138 Um, that would be the, the one person who's still possible 54 00:02:53,138 --> 00:02:55,118 to meet that I haven't met. 55 00:02:55,478 --> 00:02:57,878 But, uh, yeah, William, I'm super excited about that. 56 00:02:57,968 --> 00:02:58,328 So. 57 00:02:58,733 --> 00:03:00,323 Prasanna Malaiyandi: Does it count as meeting if you go visit 58 00:03:00,323 --> 00:03:01,638 the grave site of the person? 59 00:03:02,588 --> 00:03:03,848 W. Curtis Preston: Oh, that's just wrong. 60 00:03:03,923 --> 00:03:04,343 That's 61 00:03:04,458 --> 00:03:04,758 Prasanna Malaiyandi: I'm just 62 00:03:05,203 --> 00:03:06,063 W. Curtis Preston: That's just wrong. 63 00:03:09,398 --> 00:03:10,568 Uh, yeah. 64 00:03:10,568 --> 00:03:13,958 So, uh, just help me, help me keep my, my heart pitter Pat. 65 00:03:13,958 --> 00:03:18,458 I'm definitely a, definitely a fan and meeting him, uh, will be very, very cool. 66 00:03:19,328 --> 00:03:20,708 Uh, this week. 67 00:03:20,768 --> 00:03:24,428 Prasanna Malaiyandi: so wait, what is your, if you got a chance to ask 68 00:03:24,428 --> 00:03:25,563 him a question, what would it be? 69 00:03:26,483 --> 00:03:29,723 W. Curtis Preston: Oh, it's definitely not gonna be one of 70 00:03:29,723 --> 00:03:32,663 those, like an episode 57, you know? 71 00:03:33,413 --> 00:03:35,333 Um, wow. 72 00:03:35,333 --> 00:03:36,533 I'm not prepared for that question. 73 00:03:36,563 --> 00:03:37,348 I'll have to think about that. 74 00:03:37,628 --> 00:03:38,318 Prasanna Malaiyandi: Wow. 75 00:03:38,318 --> 00:03:38,618 Did I 76 00:03:38,618 --> 00:03:39,998 stump Curtis? 77 00:03:40,523 --> 00:03:41,753 W. Curtis Preston: You did, you stumped me. 78 00:03:41,903 --> 00:03:42,233 Yeah. 79 00:03:42,293 --> 00:03:45,653 I'll have to, I'll, yeah, I'll definitely, you know, I'm gonna be, I'm gonna be so 80 00:03:45,683 --> 00:03:48,113 nerded out, like I'm, I'm gonna be, yeah. 81 00:03:48,953 --> 00:03:53,123 Um, I just, I, if I get to say two words to him, I'll be, 82 00:03:53,513 --> 00:03:55,523 you know, I'll be like, hi. 83 00:03:55,943 --> 00:03:58,283 Um, you know, I, yeah. 84 00:04:00,883 --> 00:04:05,048 I, I just hope I don't do, like, I've met a lot of famous people and 85 00:04:05,048 --> 00:04:06,998 so many times I've been like, chill. 86 00:04:07,388 --> 00:04:11,438 But I remember there was this one person that I just randomly ran into in an 87 00:04:11,438 --> 00:04:16,178 airport and I literally screamed their name like a, like a 10-year-old girl. 88 00:04:16,568 --> 00:04:19,478 And, um, that was very embarrassing. 89 00:04:19,478 --> 00:04:22,808 I just hope I don't go, William, that would 90 00:04:22,878 --> 00:04:23,168 Prasanna Malaiyandi: okay. 91 00:04:23,648 --> 00:04:25,748 I'm sure he is used to it, you know? 92 00:04:26,198 --> 00:04:27,398 W. Curtis Preston: Yeah, I'm sure. 93 00:04:27,398 --> 00:04:27,788 Yeah. 94 00:04:28,268 --> 00:04:33,788 Um, so this week we're continuing our series about cloud disasters 95 00:04:33,788 --> 00:04:36,908 and this one is pretty bad. 96 00:04:37,178 --> 00:04:41,798 Um, you know, and again, this is yet another story that's gonna 97 00:04:41,803 --> 00:04:44,348 prove the point back your stuff up. 98 00:04:44,348 --> 00:04:44,438 Right. 99 00:04:45,518 --> 00:04:50,558 You know, even, even if this is actually, this is a really good. 100 00:04:51,608 --> 00:04:55,538 Story that basically proves that even if the vendor is backing it up 101 00:04:55,538 --> 00:04:59,228 for you and the backups are included as part of the package, something 102 00:04:59,228 --> 00:05:04,988 so catastrophic might happen that those backups don't come in handy. 103 00:05:05,078 --> 00:05:07,298 Does that sound about right, Pana? 104 00:05:07,808 --> 00:05:09,218 Prasanna Malaiyandi: It does, but I have. 105 00:05:10,163 --> 00:05:11,933 Two comments about that. 106 00:05:12,218 --> 00:05:12,728 W. Curtis Preston: Yeah. 107 00:05:12,788 --> 00:05:13,358 Okay. 108 00:05:13,493 --> 00:05:15,713 Prasanna Malaiyandi: the first is, this reminds me a lot about the 109 00:05:15,713 --> 00:05:17,603 OVH story that we did a while ago. 110 00:05:17,603 --> 00:05:22,463 So if you haven't let heard that episode, go back, give it a listen, because it 111 00:05:22,468 --> 00:05:26,483 was also the case with OVH that they said they were doing backups, but 112 00:05:26,483 --> 00:05:30,563 people were not able to restore their backup because they were sitting in 113 00:05:30,563 --> 00:05:36,413 the same data center as a production and there was a fire, so not so good. 114 00:05:36,728 --> 00:05:37,178 W. Curtis Preston: Yeah. 115 00:05:37,178 --> 00:05:42,158 And, and I'm gonna say, so this, this story is about Rackspace, which I'm gonna 116 00:05:42,158 --> 00:05:44,498 say I have no ill will against Rackspace. 117 00:05:44,613 --> 00:05:49,328 I, I feel for the people that had to go through this, uh, the thing I struggle 118 00:05:49,328 --> 00:05:54,248 with is the ways in which Rackspace tried to deflect, blame Rackspace. 119 00:05:54,248 --> 00:05:56,108 The company tried to deflect blame. 120 00:05:56,708 --> 00:06:00,398 Uh, and so based on that, we've got a pretty solid timeline of the 121 00:06:00,398 --> 00:06:03,818 events Now, just, just for color. 122 00:06:04,473 --> 00:06:07,083 And I didn't know this, this part I'm about to say. 123 00:06:07,083 --> 00:06:09,933 I didn't know this until, until I was researching for the story. 124 00:06:10,323 --> 00:06:16,053 Prior to this event happening, Rackspace had already suffered, uh, a sharp. 125 00:06:16,763 --> 00:06:18,593 Decline in value. 126 00:06:18,643 --> 00:06:24,553 At the height of their value, April, 2021, they were a, an $8 billion company. 127 00:06:24,943 --> 00:06:28,183 And by the time this event happened, they had dropped, 128 00:06:28,243 --> 00:06:30,133 over over 90% of their value. 129 00:06:30,838 --> 00:06:33,728 They were then an $800 million company. 130 00:06:33,778 --> 00:06:36,128 Prasanna Malaiyandi: And, as of today's recording, they 131 00:06:36,133 --> 00:06:42,578 are valued at $340 million, which is 5% of where they were with at their high. 132 00:06:43,598 --> 00:06:44,048 W. Curtis Preston: Right? 133 00:06:44,048 --> 00:06:44,498 Yeah. 134 00:06:44,558 --> 00:06:49,118 Um, so, so they were already in sort of trouble and I, I think that may 135 00:06:49,118 --> 00:06:51,698 be why they tried to deflect blame. 136 00:06:51,698 --> 00:06:58,178 So, um, let's start with sort of the, before the story, right? 137 00:06:58,183 --> 00:07:01,148 So before the story, there was something called the proxy, not she 138 00:07:01,418 --> 00:07:04,718 exploit, uh, in September of 2022. 139 00:07:05,333 --> 00:07:09,653 It was publicly announced and it basically, it allowed someone to 140 00:07:09,653 --> 00:07:11,693 gain control of an exchange server. 141 00:07:12,283 --> 00:07:17,653 It was announced September 30th, 2022, November 8th, Microsoft 142 00:07:17,658 --> 00:07:22,403 released a security update but there was a minor issue with the patch. 143 00:07:22,808 --> 00:07:27,638 And Rackspace claimed this was why they didn't install it at that time, 144 00:07:28,208 --> 00:07:32,818 but by November 17th, Microsoft had fixed that, , that issue. 145 00:07:33,308 --> 00:07:34,838 You know, we talk about three things, right? 146 00:07:34,838 --> 00:07:37,928 Password management, patch management, and MFA. 147 00:07:37,928 --> 00:07:41,558 And then if everybody just did this, then it would've stopped. 148 00:07:41,918 --> 00:07:44,078 Uh, you know, it would stop so much. 149 00:07:44,078 --> 00:07:45,998 And this story is so much I. 150 00:07:46,523 --> 00:07:51,903 Evidence of that, uh, because November 17th that minor issue 151 00:07:51,903 --> 00:07:53,343 with the patch was fixed. 152 00:07:53,703 --> 00:07:58,263 So they could have, and in my opinion, should have immediately put on this 153 00:07:58,263 --> 00:08:03,843 security patch because it was such a huge exploit there was a CVA attached to it. 154 00:08:04,543 --> 00:08:08,323 And, uh, well-known within the industry, they should have immediately patched 155 00:08:08,323 --> 00:08:10,393 all of their, uh, exchange servers. 156 00:08:10,633 --> 00:08:12,913 By the way, I should mention what we're talking about is that 157 00:08:12,913 --> 00:08:18,013 Rackspace had a hosted exchange service, not Microsoft 365. 158 00:08:18,373 --> 00:08:23,023 They ran hosted exchange on their own servers in their own, uh, data center. 159 00:08:23,693 --> 00:08:27,443 Prasanna Malaiyandi: Before you continue on, I think it's important to state that 160 00:08:27,443 --> 00:08:34,103 for that September 30th, right, there was a workaround that was deployed, right? 161 00:08:34,133 --> 00:08:34,523 That 162 00:08:34,883 --> 00:08:38,693 pretty much Microsoft was like, Hey, we haven't quite figured out the patch 163 00:08:38,693 --> 00:08:40,223 yet, which will come out November 8th. 164 00:08:40,223 --> 00:08:43,523 But in the meantime, here's a workaround to make sure you don't get impacted, 165 00:08:43,523 --> 00:08:45,443 which Rackspace did apply, apply. 166 00:08:46,163 --> 00:08:46,463 Right. 167 00:08:46,643 --> 00:08:48,113 W. Curtis Preston: they, they did or they did not. 168 00:08:49,253 --> 00:08:51,353 Prasanna Malaiyandi: They did apply the workaround. 169 00:08:51,428 --> 00:08:51,638 W. Curtis Preston: Yeah. 170 00:08:51,668 --> 00:08:51,938 Okay. 171 00:08:51,998 --> 00:08:52,388 Yeah. 172 00:08:52,703 --> 00:08:52,973 Prasanna Malaiyandi: Yeah. 173 00:08:53,303 --> 00:08:55,343 So it's not a permanent fix, but at least 174 00:08:55,553 --> 00:08:56,963 sort of protects you for now. 175 00:08:57,818 --> 00:08:59,783 So then the other thing is, um. 176 00:09:01,493 --> 00:09:05,213 around this time there were actually two exploits, Right. 177 00:09:05,213 --> 00:09:08,333 So there was a proxy, not shell exploit, and then there was 178 00:09:08,333 --> 00:09:12,023 another one, um, O-W-A-S-S-R-F. 179 00:09:12,053 --> 00:09:13,793 I don't know what that stands for, but that's what they 180 00:09:13,798 --> 00:09:15,803 called it, right? 181 00:09:16,103 --> 00:09:18,293 And these two are kind of related. 182 00:09:19,523 --> 00:09:27,233 And so the patch though, that came out in November would have fixed both. 183 00:09:28,723 --> 00:09:29,073 W. Curtis Preston: Right. 184 00:09:29,378 --> 00:09:31,988 Prasanna Malaiyandi: they were applied, but the workaround that was 185 00:09:31,988 --> 00:09:37,418 applied in the end of September only addressed the proxy nutshell issue. 186 00:09:37,568 --> 00:09:37,748 It 187 00:09:37,753 --> 00:09:39,878 did not expl address the second exploit. 188 00:09:40,823 --> 00:09:41,123 W. Curtis Preston: Yeah. 189 00:09:41,123 --> 00:09:43,943 By the way, the OWA most certainly stands for Outlook. 190 00:09:43,943 --> 00:09:45,893 Web access would be my guess. 191 00:09:45,893 --> 00:09:49,913 I don't know what SSRF but stands for, but yeah, it is kind 192 00:09:49,913 --> 00:09:51,323 of complicated that basically. 193 00:09:52,208 --> 00:09:57,038 That there was a patch that the, the PA had, they applied the patch, they would've 194 00:09:57,038 --> 00:10:00,878 fixed at a, at that time, unknown problem. 195 00:10:01,538 --> 00:10:04,328 Um, but the, but they didn't apply the patch. 196 00:10:04,868 --> 00:10:07,868 And then two weeks goes by and then what happened? 197 00:10:08,753 --> 00:10:13,703 Prasanna Malaiyandi: And then on November 29th, Rackspace says that they 198 00:10:13,703 --> 00:10:19,253 were attacked by a group called Play, which gained access to their exchange 199 00:10:19,253 --> 00:10:25,913 environment using stolen credentials, and that they had access to some of Rackspace 200 00:10:25,913 --> 00:10:28,703 exchange environments, which, if I 201 00:10:28,703 --> 00:10:31,433 was a customer on hosted exchange, I would be kind of freaked out. 202 00:10:33,623 --> 00:10:34,943 W. Curtis Preston: Yeah, exactly right. 203 00:10:34,948 --> 00:10:38,603 So they, they gain privileged access of their exchange servers. 204 00:10:38,613 --> 00:10:42,723 We're, not sure if they knew in November, but because they first 205 00:10:42,723 --> 00:10:45,063 notified people December 2nd. 206 00:10:45,648 --> 00:10:47,358 Literally at two o'clock in the morning. 207 00:10:47,358 --> 00:10:47,658 Right. 208 00:10:47,658 --> 00:10:52,758 Based on the, the, the stuff that we have, they, they may have at that point 209 00:10:52,758 --> 00:10:56,328 realized what happened and they were able to trace it back to November 29th. 210 00:10:56,788 --> 00:10:58,168 Prasanna Malaiyandi: And at 2:00 AM right? 211 00:10:58,173 --> 00:11:00,478 This is when they were like, yeah, here's what happened. 212 00:11:00,478 --> 00:11:03,538 We noticed something, and then they just brought everything down, right? 213 00:11:03,538 --> 00:11:03,540 They were 214 00:11:04,078 --> 00:11:05,818 like, yep, we're not gonna allow any more access 215 00:11:05,818 --> 00:11:06,688 W. Curtis Preston: response, right? 216 00:11:08,818 --> 00:11:13,138 The, the next part of the story is the one that really sets it apart. 217 00:11:13,138 --> 00:11:15,328 I don't know any other story like this. 218 00:11:15,778 --> 00:11:22,858 The res, their response was, and, and again, if you think about now that if 219 00:11:22,858 --> 00:11:28,828 you think about where they were as a company, this part maybe makes more sense. 220 00:11:29,638 --> 00:11:34,138 But what they decided to do was they said, you know what? 221 00:11:34,198 --> 00:11:36,058 This is gonna take us a while. 222 00:11:37,738 --> 00:11:39,358 This the, I'm making up words here. 223 00:11:39,598 --> 00:11:44,458 We've been thinking about shooting this thing in the head anyway, and so let's 224 00:11:44,458 --> 00:11:47,068 just move everybody over to Microsoft 365. 225 00:11:48,058 --> 00:11:53,128 So December 2nd at 2:00 AM is when they first started telling people that 226 00:11:53,133 --> 00:11:57,688 they had this problem, and by 8:00 PM that evening, they had made the 227 00:11:57,688 --> 00:12:00,778 decision to move everybody over to 365. 228 00:12:00,778 --> 00:12:03,688 Prasanna Malaiyandi: Yeah, that I could, I would have loved to have 229 00:12:03,688 --> 00:12:07,708 been a fly on the wall in those meetings, right when they were 230 00:12:07,708 --> 00:12:08,068 trying to 231 00:12:08,103 --> 00:12:09,453 W. Curtis Preston: not have wanted to be in the meeting. 232 00:12:10,413 --> 00:12:10,703 Prasanna Malaiyandi: Yeah. 233 00:12:10,708 --> 00:12:12,988 Yeah, I, that's why I said I wanted to be a fly on the wall. 234 00:12:13,048 --> 00:12:13,558 Right. 235 00:12:14,143 --> 00:12:14,433 W. Curtis Preston: Yeah. 236 00:12:14,818 --> 00:12:16,588 Prasanna Malaiyandi: and hearing these conversations because you 237 00:12:16,588 --> 00:12:20,818 know, it must have been a difficult decision to come to, right? 238 00:12:21,163 --> 00:12:24,763 W. Curtis Preston: Yeah, because that would've been a competing service, right? 239 00:12:24,763 --> 00:12:30,523 So if, if it's not obvious, like if you used hosted exchange, you were 240 00:12:30,528 --> 00:12:35,923 very consciously using hosted exchange, not Microsoft 365, and you had reasons 241 00:12:35,923 --> 00:12:40,003 for doing that, and they're like, guys, this is gonna take us a while. 242 00:12:40,008 --> 00:12:42,253 We're gonna move everybody over to 365. 243 00:12:42,793 --> 00:12:45,223 But what did they not move? 244 00:12:47,668 --> 00:12:49,648 Prasanna Malaiyandi: Uh, so there were two things. 245 00:12:49,648 --> 00:12:51,148 They did not move, right? 246 00:12:51,538 --> 00:12:54,988 The, probably the most important thing was their emails, 247 00:12:56,308 --> 00:12:56,638 right? 248 00:12:58,318 --> 00:13:01,198 What people cared about with the hosted exchange service, right? 249 00:13:01,588 --> 00:13:05,938 Because they basically said, we will recreate things for you, make it easy. 250 00:13:05,938 --> 00:13:10,348 So you have all your stuff up and running at Microsoft 365, but we can't 251 00:13:10,348 --> 00:13:12,418 get you back all your emails yet. 252 00:13:14,113 --> 00:13:14,713 W. Curtis Preston: Right. 253 00:13:14,938 --> 00:13:16,528 Prasanna Malaiyandi: Right, so that was one thing. 254 00:13:16,528 --> 00:13:20,518 I think the second thing, and I don't know if they were forthcoming with 255 00:13:20,523 --> 00:13:24,508 this, but the fact that with their hosted exchange implementation, they 256 00:13:24,508 --> 00:13:26,458 offered backup as part of the service, 257 00:13:27,268 --> 00:13:27,658 W. Curtis Preston: Right. 258 00:13:27,928 --> 00:13:28,228 Prasanna Malaiyandi: right? 259 00:13:28,288 --> 00:13:31,858 When they told customers, Hey, Microsoft 365 is where you should 260 00:13:31,858 --> 00:13:36,058 be looking at, I do wonder if they told people, by the way, you need to 261 00:13:36,058 --> 00:13:37,468 figure out your own backup solution. 262 00:13:38,938 --> 00:13:41,183 W. Curtis Preston: Yeah, none of the communications that we found 263 00:13:41,188 --> 00:13:44,843 between them and customers, uh, showed that they'd said anything. 264 00:13:45,323 --> 00:13:50,393 Uh, but the idea that they would, in the middle of the outage basically 265 00:13:50,393 --> 00:13:52,733 abandoned an entire business line. 266 00:13:53,558 --> 00:13:55,508 Move everybody over to 365. 267 00:13:55,508 --> 00:14:00,128 What they did do was they were able, apparently they, they did 268 00:14:00,608 --> 00:14:04,928 automate the process of creating the accounts for them over on 365. 269 00:14:05,678 --> 00:14:09,788 So you, you were able to, um, you know, essentially you, you were able 270 00:14:09,788 --> 00:14:12,008 to start sending and receiving email. 271 00:14:12,383 --> 00:14:16,853 Relatively quickly considering how long the rest of this took within 272 00:14:16,853 --> 00:14:21,983 a day, it looked like uh, or so you were able to send and receive 273 00:14:21,983 --> 00:14:24,413 email using your old email address. 274 00:14:24,653 --> 00:14:30,263 If you were an exchange hosted exchange customer, and now you're on 365. 275 00:14:31,193 --> 00:14:35,363 You just didn't have access to any of the email you had received up to that point. 276 00:14:35,363 --> 00:14:38,843 Prasanna Malaiyandi: Yeah, which I like. 277 00:14:38,843 --> 00:14:40,133 I go back and forth on that. 278 00:14:40,133 --> 00:14:41,723 It's like, great. 279 00:14:41,723 --> 00:14:44,813 I could send and I could see what people are sending me, but I have a lot 280 00:14:44,813 --> 00:14:49,313 of old stuff and I would be freaking out if I lost all of my old emails. 281 00:14:49,313 --> 00:14:53,363 Or a lot of times if these are businesses and organizations, maybe 282 00:14:53,363 --> 00:14:57,443 they have contracts which are being sent back and forth via emails. 283 00:14:58,698 --> 00:15:02,478 W. Curtis Preston: Yeah, like literally stuff that you got today, right? 284 00:15:02,478 --> 00:15:06,168 Stuff that you got yesterday, stuff you're actively working on. 285 00:15:06,228 --> 00:15:10,938 And a lot of people use their email system as sort of a somewhat, 286 00:15:10,938 --> 00:15:14,748 sometimes temporary, sometimes permanent storage system, right? 287 00:15:15,408 --> 00:15:17,328 And, and they're like, I know where that contract is. 288 00:15:17,328 --> 00:15:20,118 It's in the email that I got, you know, three days 289 00:15:20,403 --> 00:15:21,273 Prasanna Malaiyandi: and I can search for it. 290 00:15:21,273 --> 00:15:22,113 I can find it. 291 00:15:22,413 --> 00:15:22,683 Yep. 292 00:15:22,953 --> 00:15:23,853 W. Curtis Preston: You're not able. 293 00:15:23,853 --> 00:15:24,063 Yeah. 294 00:15:24,063 --> 00:15:29,583 So on one on one hand, like, like you said, I would've wanted to be a fly 295 00:15:29,583 --> 00:15:35,403 on the wall because they're saying we need to do what's, we need to get 296 00:15:35,403 --> 00:15:37,443 our customers up and running again. 297 00:15:37,443 --> 00:15:40,983 We need them, we need to get them, be able to send and receive email. 298 00:15:42,458 --> 00:15:45,303 And I think that was a good decision. 299 00:15:45,463 --> 00:15:46,323 Prasanna Malaiyandi: That's probably the most 300 00:15:46,323 --> 00:15:47,568 important thing to do first. 301 00:15:48,273 --> 00:15:49,203 W. Curtis Preston: yeah. 302 00:15:51,828 --> 00:15:55,443 The, the, the bad decisions happened way before this. 303 00:15:55,503 --> 00:16:01,143 In my opinion, this was a good decision, um, because as we're going 304 00:16:01,148 --> 00:16:06,933 to find out in the story, they didn't exactly have a good, uh, backup. 305 00:16:06,933 --> 00:16:06,993 I. 306 00:16:07,653 --> 00:16:12,453 System, um, at least not one that, that I would recognize, right? 307 00:16:12,663 --> 00:16:15,903 So they actually advertise backup as part of the service. 308 00:16:16,533 --> 00:16:21,153 And again, I I, I'm gonna put this out as this is why, when, when I say, 309 00:16:21,243 --> 00:16:25,773 even if the SaaS vendor advertise backup as part of the service, you 310 00:16:25,773 --> 00:16:28,113 might want to consider a third party. 311 00:16:28,228 --> 00:16:32,133 And I'll put an asterisk, especially if they charge extra for it. 312 00:16:33,423 --> 00:16:37,293 Um, I think in this case it was just included as part of the, the, well, 313 00:16:37,413 --> 00:16:39,723 I'm gonna put it, I'm gonna put in a, I'm gonna put it, especially, 314 00:16:39,723 --> 00:16:42,273 I'm gonna put two asterisk, especially if they charge for it. 315 00:16:42,273 --> 00:16:44,733 'cause then that gives you an incentive to pay somebody else instead. 316 00:16:45,123 --> 00:16:47,733 But then I'm gonna say, especially if they don't charge for it, which means they're 317 00:16:47,733 --> 00:16:49,143 probably not spending any money on it. 318 00:16:49,218 --> 00:16:50,298 Prasanna Malaiyandi: Enough money to do that. 319 00:16:50,303 --> 00:16:50,503 Yeah. 320 00:16:51,043 --> 00:16:52,723 I wanna focus on that for a second 321 00:16:53,953 --> 00:16:56,863 because I did remember earlier on in the episode, I said I had two 322 00:16:56,863 --> 00:16:58,123 comments and we only covered one. 323 00:16:58,273 --> 00:16:59,023 So here's a second 324 00:16:59,128 --> 00:16:59,518 W. Curtis Preston: Oh. 325 00:17:00,928 --> 00:17:08,308 Prasanna Malaiyandi: Right is, does this apply If you are using a SaaS 326 00:17:08,848 --> 00:17:14,218 or whatever backup product in and of itself, that you should have a second 327 00:17:14,218 --> 00:17:17,338 vendor because who knows what that backup vendor will do, and maybe you 328 00:17:17,338 --> 00:17:18,898 can never get your data back out. 329 00:17:19,473 --> 00:17:22,923 I understand Rackspace, they offered backup and it, the backups didn't work. 330 00:17:23,943 --> 00:17:30,153 Right now, there are a whole slew of SaaS data protection companies, or you could 331 00:17:30,158 --> 00:17:30,993 roll your own. 332 00:17:31,683 --> 00:17:32,043 Right. 333 00:17:32,173 --> 00:17:37,003 In those cases though, do you have the same recommendation that if I do decide 334 00:17:37,003 --> 00:17:41,113 to use a SaaS data protection company, I should probably use two SaaS data 335 00:17:41,113 --> 00:17:43,033 protection companies because I don't 336 00:17:43,033 --> 00:17:45,793 know if one can get me my data back? 337 00:17:46,648 --> 00:17:48,478 W. Curtis Preston: Yeah, no, that, that's not what I'm saying. 338 00:17:48,508 --> 00:17:56,068 I'm saying it because I'm talking about a SaaS service and then hiring 339 00:17:56,113 --> 00:18:00,388 a SaaS data protection company that puts your data in two different places. 340 00:18:01,468 --> 00:18:03,028 I would, I would not argue. 341 00:18:03,958 --> 00:18:07,168 Having another one, but it's gonna be really, it's a, it's already 342 00:18:07,168 --> 00:18:08,638 going to be a big enough cost. 343 00:18:08,668 --> 00:18:11,968 I think the idea of putting the data in two different, completely different 344 00:18:12,328 --> 00:18:18,508 protection zones, risk factors, you know, earthquake and flood zones, all of 345 00:18:18,628 --> 00:18:20,068 Prasanna Malaiyandi: it already, diversifies it. 346 00:18:20,458 --> 00:18:20,818 yeah. 347 00:18:21,478 --> 00:18:21,628 W. Curtis Preston: Yeah. 348 00:18:21,628 --> 00:18:23,098 I think that already diversifies it. 349 00:18:23,308 --> 00:18:24,448 Prasanna Malaiyandi: so now my second question. 350 00:18:25,918 --> 00:18:29,338 W. Curtis Preston: I'm just saying that if you have a backup, if you 351 00:18:29,338 --> 00:18:35,398 have a SaaS service, like 365, so 365 is about to start offering backup. 352 00:18:36,088 --> 00:18:40,678 Um, and and I'm just saying I still like the idea of a third party copy. 353 00:18:40,738 --> 00:18:40,978 Prasanna Malaiyandi: Yeah. 354 00:18:41,698 --> 00:18:41,998 Okay. 355 00:18:42,003 --> 00:18:47,068 So now my second question follow up to that is there's a lot of cloud public, 356 00:18:47,068 --> 00:18:52,498 cloud providers, right, that people hook into that leverage things like snapshots. 357 00:18:53,233 --> 00:18:53,523 W. Curtis Preston: Yeah. 358 00:18:53,623 --> 00:18:56,173 Prasanna Malaiyandi: So they're just an orchestration layer on top. 359 00:18:56,173 --> 00:18:57,583 They're doing all the data movement. 360 00:18:57,583 --> 00:18:59,863 They're moving your data around, they're taking the copies. 361 00:19:00,583 --> 00:19:02,713 Do you have the same concerns with those as well? 362 00:19:03,763 --> 00:19:04,393 W. Curtis Preston: I do. 363 00:19:04,483 --> 00:19:09,373 Um, basically the, that's why again, I li I like the orchestration 364 00:19:09,373 --> 00:19:12,523 companies and the ones I like best are the ones that ultimately take 365 00:19:12,523 --> 00:19:15,733 a copy of the data outside, right? 366 00:19:15,883 --> 00:19:19,963 We, we use snapshots to orchestrate and to create the backup, and then we use 367 00:19:19,963 --> 00:19:22,153 something else to get the data out of. 368 00:19:22,558 --> 00:19:25,498 You know, your favorite cloud vendor, again, getting it out, 369 00:19:25,498 --> 00:19:26,908 storing it in another place. 370 00:19:27,148 --> 00:19:30,988 Second best to that would be storing it in another region, in, in another account. 371 00:19:31,558 --> 00:19:35,698 But you know, this is just the, basically I see, like, I think 372 00:19:35,698 --> 00:19:39,058 of like, let's say AWS as NetApp. 373 00:19:39,493 --> 00:19:40,003 Right. 374 00:19:40,363 --> 00:19:46,153 So I need another copy, a final copy of the data that isn't on AWS because 375 00:19:46,153 --> 00:19:48,613 of rolling code concerns, right? 376 00:19:48,613 --> 00:19:51,823 So you get a bug and it, and it rolls, uh, you know, and takes 377 00:19:51,823 --> 00:19:53,263 out both primary and the backup. 378 00:19:54,028 --> 00:19:56,518 Prasanna Malaiyandi: then are you also concerned though, because a lot of 379 00:19:56,518 --> 00:19:59,188 these SaaS data protection companies are built on top of the big clouds. 380 00:20:01,048 --> 00:20:06,088 W. Curtis Preston: So I do, I do want to, um, then make sure that they're 381 00:20:06,093 --> 00:20:07,978 stored in different regions and whatnot. 382 00:20:08,188 --> 00:20:10,378 I, you know, I can only, I can only, 383 00:20:10,433 --> 00:20:11,443 Prasanna Malaiyandi: you can only go so far. 384 00:20:11,518 --> 00:20:14,248 W. Curtis Preston: that, yeah, you could only go so far, right? 385 00:20:14,248 --> 00:20:16,228 It'd be like the same. 386 00:20:16,438 --> 00:20:19,828 The same would be true if we weren't talking to cloud and we were saying 387 00:20:19,828 --> 00:20:26,278 they also, your backup service also uses Soliris, right At, at some point there's 388 00:20:26,278 --> 00:20:27,718 a risk that you just can't get away from. 389 00:20:28,463 --> 00:20:31,193 Prasanna Malaiyandi: Sorry, I, I know we never talked about that before, so 390 00:20:31,193 --> 00:20:32,738 I was very curious about your take on 391 00:20:32,888 --> 00:20:33,518 W. Curtis Preston: Yeah. 392 00:20:33,638 --> 00:20:34,178 Yeah. 393 00:20:34,238 --> 00:20:34,748 Okay. 394 00:20:35,348 --> 00:20:40,928 So, uh, so we were talking about December 2nd is when, um, they, you 395 00:20:40,928 --> 00:20:42,968 know, when they made this move, right? 396 00:20:43,658 --> 00:20:48,818 Um, the, the, they first mentioned that they believe it was a 397 00:20:48,818 --> 00:20:51,428 security incident on December 3rd. 398 00:20:52,178 --> 00:20:57,518 Uh, and then, uh, December 2nd or December 6th, they say that 399 00:20:57,523 --> 00:20:58,988 it was a ransomware incident. 400 00:20:59,378 --> 00:21:03,548 And then finally, 14th, they revealed the attack was from a, their, their 401 00:21:03,548 --> 00:21:06,338 words financially motivated threat actor. 402 00:21:06,938 --> 00:21:15,428 Um, so we don't know the details, uh, of, you know, the extortion, but 403 00:21:15,428 --> 00:21:17,198 there was some kind of extortion. 404 00:21:17,198 --> 00:21:20,738 We also don't know whether or not they ultimately, I. 405 00:21:21,218 --> 00:21:22,688 Paid that money. 406 00:21:23,228 --> 00:21:25,148 Um, you know, 407 00:21:25,988 --> 00:21:26,528 Prasanna Malaiyandi: and, and this. 408 00:21:26,753 --> 00:21:29,018 W. Curtis Preston: I, I, I advise as much as possible not 409 00:21:29,023 --> 00:21:30,908 to pay the money, but go ahead. 410 00:21:31,643 --> 00:21:34,163 Prasanna Malaiyandi: and this goes just the earlier thing about talking 411 00:21:34,163 --> 00:21:36,923 about a security incident and then changing it to ransomware incident. 412 00:21:37,283 --> 00:21:37,913 I get it. 413 00:21:37,913 --> 00:21:42,293 But I know you and I, we've talked in the past about vendors or companies 414 00:21:42,293 --> 00:21:48,473 should be more transparent to a certain extent about what's going on in order to. 415 00:21:49,703 --> 00:21:54,623 Build confidence in the public in terms of they have things handled, 416 00:21:54,713 --> 00:21:56,003 they're figuring things out. 417 00:21:56,003 --> 00:21:56,813 It's okay. 418 00:21:57,353 --> 00:22:03,173 And so I wanted to get your take on that messaging that came from Rackspace. 419 00:22:03,173 --> 00:22:06,263 Do you think that caused, like, do you think that would've caused a 420 00:22:06,268 --> 00:22:10,163 lot of concern for customers or the public in terms of their ability 421 00:22:10,163 --> 00:22:13,613 to handle things first, calling it a security incident and then a 422 00:22:13,613 --> 00:22:15,953 ransomware, and then sort of this back 423 00:22:15,953 --> 00:22:16,763 and forth on data? 424 00:22:16,763 --> 00:22:16,973 I. 425 00:22:18,563 --> 00:22:20,633 W. Curtis Preston: I think it's possible that they just, 426 00:22:20,783 --> 00:22:22,703 they revealed what they knew. 427 00:22:22,823 --> 00:22:26,543 Like you do have to be careful saying only what you know for sure. 428 00:22:27,173 --> 00:22:27,563 Prasanna Malaiyandi: Yeah. 429 00:22:27,623 --> 00:22:30,623 W. Curtis Preston: The thing that needs to be understood at this point as they move 430 00:22:30,623 --> 00:22:36,503 through the story is that be like, on one hand, I'm glad they did what they did. 431 00:22:36,503 --> 00:22:39,503 They moved everybody over to 365, got everybody working, but 432 00:22:39,503 --> 00:22:45,383 everybody's gonna be clamoring for their emails right from the previous. 433 00:22:45,878 --> 00:22:51,698 Uh, but because they did it the way they did it, so it, they, if they 434 00:22:51,698 --> 00:22:56,858 had migrated exchange into 365, they could have brought the data with them, 435 00:22:57,068 --> 00:23:01,478 but it would've taken longer because they still, they had dead servers. 436 00:23:01,598 --> 00:23:01,778 Right. 437 00:23:02,393 --> 00:23:02,543 Prasanna Malaiyandi: Yeah. 438 00:23:03,413 --> 00:23:04,223 They don't have the data yet. 439 00:23:04,463 --> 00:23:06,083 W. Curtis Preston: A migration takes a while too. 440 00:23:06,113 --> 00:23:06,473 Right. 441 00:23:07,043 --> 00:23:10,433 But because it did it the way they did it, the only option at this 442 00:23:10,433 --> 00:23:16,673 point is to create PSTs of individual users and then import those PSTs 443 00:23:16,678 --> 00:23:18,713 to those users on the other side. 444 00:23:19,373 --> 00:23:26,783 Um, and so they, they said on December 18th that they had, um, they 445 00:23:26,783 --> 00:23:31,583 had created in, um, that I, I, I. 446 00:23:32,243 --> 00:23:37,583 I'm a little confused where like on one hand they said that they had, I think 447 00:23:37,583 --> 00:23:42,383 they had figured out a way to start restoring the affected exchange servers, 448 00:23:42,833 --> 00:23:45,053 but they hadn't yet restored all of them. 449 00:23:45,773 --> 00:23:46,193 Right. 450 00:23:46,703 --> 00:23:49,313 And this is when they, when they first announce, they're like, so 451 00:23:49,313 --> 00:23:50,573 here's how this is gonna work. 452 00:23:50,573 --> 00:23:52,403 We're gonna restore an exchange server. 453 00:23:52,733 --> 00:23:56,903 If you're on that exchange server, you will then be able to export, 454 00:23:57,413 --> 00:24:02,273 uh, A PST file, and then you'll then be able to download a PST file for 455 00:24:02,273 --> 00:24:07,313 each user, and then you will then be able to upload that into 365. 456 00:24:07,373 --> 00:24:11,618 And they first announced this on December 18th, and then December 457 00:24:11,618 --> 00:24:13,508 Prasanna Malaiyandi: already two weeks after they shut down the service. 458 00:24:14,633 --> 00:24:15,503 W. Curtis Preston: Thank you very much. 459 00:24:15,503 --> 00:24:17,693 That is two weeks after they shut down the service. 460 00:24:17,693 --> 00:24:18,053 Right. 461 00:24:18,623 --> 00:24:25,193 Um, and uh, it also, it's important to understand that by exporting 462 00:24:25,193 --> 00:24:28,943 it as a PSD and then importing it, it's not gonna be a perfect restore. 463 00:24:28,948 --> 00:24:31,393 'cause , I'm pretty sure that you're gonna lose folders and 464 00:24:31,393 --> 00:24:33,043 all of that in this process. 465 00:24:33,673 --> 00:24:37,693 Uh, not to mention the fact that it's just you're gonna lose metadata too. 466 00:24:37,903 --> 00:24:38,293 Right. 467 00:24:39,463 --> 00:24:42,163 Prasanna Malaiyandi: So, you're so in my mind I'm thinking, oh, this is great. 468 00:24:42,163 --> 00:24:44,263 It's a way for people to do things. 469 00:24:44,743 --> 00:24:45,133 Right. 470 00:24:45,163 --> 00:24:45,793 And then 471 00:24:45,793 --> 00:24:49,033 on December 20th, they said that they had just begun testing 472 00:24:49,033 --> 00:24:50,563 the above recovery procedure. 473 00:24:52,463 --> 00:24:52,753 W. Curtis Preston: Yeah. 474 00:24:53,863 --> 00:24:54,583 Prasanna Malaiyandi: Yeah. 475 00:24:54,673 --> 00:24:59,413 So they hadn't actually tried it out or done anything. 476 00:24:59,413 --> 00:25:03,703 They're kind of shooting from the hip, and I get it. 477 00:25:03,703 --> 00:25:07,213 They're urgently trying to figure out how to get the data back for the customers. 478 00:25:08,068 --> 00:25:11,518 But at the same time, it doesn't inspire confidence. 479 00:25:13,318 --> 00:25:17,793 W. Curtis Preston: Yeah, and the thing is that if they had prepared for this 480 00:25:17,963 --> 00:25:24,238 event eventuality back in the day, they could have created a different procedure. 481 00:25:24,778 --> 00:25:29,458 Because there are backup software products that allow you to, 482 00:25:29,878 --> 00:25:33,203 um, directly extract PST data. 483 00:25:33,958 --> 00:25:36,118 From a backup, right? 484 00:25:36,118 --> 00:25:36,748 It it does. 485 00:25:36,933 --> 00:25:38,098 They, they do exist. 486 00:25:38,098 --> 00:25:38,818 It does happen. 487 00:25:39,388 --> 00:25:43,258 So if they had tested this beforehand, they wouldn't have 488 00:25:43,263 --> 00:25:44,368 done it the way they did it. 489 00:25:44,398 --> 00:25:49,138 They would've figured out a way to directly, instead of restoring 490 00:25:49,138 --> 00:25:52,438 exchange servers and then saying, Hey, customers, go get your PSDs. 491 00:25:52,438 --> 00:25:54,958 They would've been able to just directly create the PSDs. 492 00:25:55,228 --> 00:25:57,028 They would've figured that out beforehand. 493 00:25:57,508 --> 00:25:58,168 Um, 494 00:25:58,363 --> 00:25:59,383 Prasanna Malaiyandi: and then it's just a matter of 495 00:25:59,458 --> 00:26:01,408 W. Curtis Preston: think that would've been a much quicker method. 496 00:26:01,693 --> 00:26:01,963 Prasanna Malaiyandi: Yeah. 497 00:26:02,413 --> 00:26:05,173 And they would've just had to execute on it rather than trying 498 00:26:05,173 --> 00:26:09,163 to figure all this out, because I don't know if a lot of folks. 499 00:26:09,643 --> 00:26:14,533 Understand when you're trying to recover from a ransomware attack, right? 500 00:26:14,533 --> 00:26:17,323 You're not only trying to figure out everything that went wrong, 501 00:26:17,623 --> 00:26:22,813 but also all the extreme pressure you're under lack of sleep, right? 502 00:26:22,813 --> 00:26:25,603 People yelling at you possibly, right? 503 00:26:25,633 --> 00:26:28,153 Worrying about, am I gonna have a job after this? 504 00:26:29,263 --> 00:26:29,623 Yeah. 505 00:26:29,848 --> 00:26:32,068 W. Curtis Preston: I can't imagine the number of people that were, you 506 00:26:32,068 --> 00:26:34,888 know, companies that were yelling about how much money they were spending 507 00:26:34,888 --> 00:26:35,788 and blah, blah, blah, blah, blah. 508 00:26:35,788 --> 00:26:36,118 Right? 509 00:26:36,808 --> 00:26:38,278 Um, yeah. 510 00:26:38,278 --> 00:26:44,008 So December 22nd is the first day that they notified some customers that they 511 00:26:44,013 --> 00:26:45,958 could start retrieving some of their mail. 512 00:26:46,198 --> 00:26:51,148 This is three weeks since the incident, right? 513 00:26:51,898 --> 00:26:59,063 Um, and it wasn't until January 5th, which is another, what, two weeks I. 514 00:26:59,173 --> 00:27:01,333 They said they had 50% done. 515 00:27:04,033 --> 00:27:06,103 I mean, this would, this took a while. 516 00:27:06,429 --> 00:27:06,639 Prasanna Malaiyandi: Yeah. 517 00:27:06,960 --> 00:27:09,026 So here's my question, Curtis. 518 00:27:09,026 --> 00:27:12,956 I know we talked earlier about, okay, they should have had 519 00:27:12,956 --> 00:27:14,456 these procedures documented. 520 00:27:14,456 --> 00:27:17,996 They should have thought about them, right, and tested it 521 00:27:17,996 --> 00:27:18,206 out. 522 00:27:18,211 --> 00:27:23,666 So then they had a process, Do you honestly think people think about 523 00:27:23,666 --> 00:27:25,886 these scenarios and walk through? 524 00:27:25,886 --> 00:27:28,076 Because normally when you're thinking disaster recovery or 525 00:27:28,076 --> 00:27:29,366 backup and recovery, right? 526 00:27:29,366 --> 00:27:35,186 It's like, oh, I lost an email, or I lost a part of something, or someone 527 00:27:35,186 --> 00:27:38,006 accidentally deleted a user, right? 528 00:27:38,636 --> 00:27:45,116 I think that the mind needs or people's mind, people need to change. 529 00:27:45,491 --> 00:27:48,461 And start to start thinking about some of these cases. 530 00:27:49,151 --> 00:27:52,661 But when they designed the system, do you think that like they were like, Hey, 531 00:27:52,661 --> 00:27:56,651 I wonder what'll happen if the exchange servers all get hit by ransomware and 532 00:27:56,651 --> 00:28:00,761 we have to rebuild and all our customers are gonna leave us for Microsoft 365? 533 00:28:02,096 --> 00:28:03,776 W. Curtis Preston: So I'll answer that in two ways. 534 00:28:03,776 --> 00:28:09,361 One is probably not right, and two, they probably should have. 535 00:28:09,956 --> 00:28:10,376 Right. 536 00:28:10,796 --> 00:28:14,576 One of the things I talk a lot about in, you know, when I'm talking about 537 00:28:14,576 --> 00:28:20,646 how to design your systems and stuff is that go get the most scary person 538 00:28:20,646 --> 00:28:22,926 in your environment and have them. 539 00:28:24,396 --> 00:28:26,856 Come up with scenarios, right? 540 00:28:26,946 --> 00:28:28,326 Come up with recovery scenarios. 541 00:28:28,536 --> 00:28:31,926 This is the whole point of tabletop exercises, right? 542 00:28:31,926 --> 00:28:36,036 You get those super negative people that that interject. 543 00:28:36,036 --> 00:28:40,446 Well, what if, well, what if, you know, we're down for weeks? 544 00:28:40,596 --> 00:28:40,896 Make 545 00:28:40,971 --> 00:28:41,601 Prasanna Malaiyandi: that's like me, 546 00:28:41,796 --> 00:28:42,696 W. Curtis Preston: about this a lot. 547 00:28:42,906 --> 00:28:48,011 Make the decision upfront that if we get hit by ransomware. 548 00:28:48,591 --> 00:28:51,951 We're gonna immediately move everybody over to 365, and if 549 00:28:51,951 --> 00:28:56,241 we do that, we're going to need to do it to recover this way. 550 00:28:56,271 --> 00:28:59,811 They, they should have been able to foresee this decision. 551 00:29:00,936 --> 00:29:05,856 Right, because by doing it the way they did it, again, and I don't disagree with 552 00:29:05,856 --> 00:29:10,116 the way they did it, but by doing the way they did it, they necessitated this weird 553 00:29:10,506 --> 00:29:17,376 double, you know, double or two step, super painful restore method that put 554 00:29:17,381 --> 00:29:19,806 a lot of the work onto their customers. 555 00:29:20,346 --> 00:29:23,856 Uh, and it's, it took them months. 556 00:29:24,636 --> 00:29:25,176 Right. 557 00:29:25,206 --> 00:29:30,216 Um, before, you know, before everybody was at, at least able, we don't even know, 558 00:29:30,696 --> 00:29:35,366 um, exactly how many customers actually, were able to successfully recover. 559 00:29:35,831 --> 00:29:38,651 Prasanna Malaiyandi: Yeah, I think they said it was something like 3000 560 00:29:38,651 --> 00:29:41,261 customers were impacted in their 561 00:29:41,261 --> 00:29:42,791 hosted exchange environment. 562 00:29:42,791 --> 00:29:43,001 Right. 563 00:29:44,096 --> 00:29:44,936 W. Curtis Preston: right, right. 564 00:29:45,746 --> 00:29:50,906 They said this was interesting is that they also said that play, or remember 565 00:29:50,911 --> 00:29:52,466 play is the name of the ransomware group. 566 00:29:53,036 --> 00:29:58,466 They accessed to use their words, the email of 27 customers and said there 567 00:29:58,466 --> 00:30:00,056 is no evidence that they read it. 568 00:30:00,776 --> 00:30:04,466 Um, know, the, the, what does that mean? 569 00:30:04,526 --> 00:30:05,066 Right? 570 00:30:05,726 --> 00:30:09,416 So again, this is where you, you have to look at like legalese. 571 00:30:10,136 --> 00:30:10,766 They have. 572 00:30:11,381 --> 00:30:13,961 The fact that they have no evidence that they read it doesn't mean they 573 00:30:13,961 --> 00:30:17,411 didn't read it, it, they don't have evidence that they didn't read it right. 574 00:30:17,411 --> 00:30:19,001 You can't prove a negative, right? 575 00:30:19,661 --> 00:30:23,261 They downloaded the email of 27 customers. 576 00:30:24,131 --> 00:30:28,631 And so it's likely that there could be, you know, there could 577 00:30:28,631 --> 00:30:33,821 have been secondary attacks where play goes after the customers. 578 00:30:33,821 --> 00:30:37,391 We didn't get any news of that, so maybe it didn't happen. 579 00:30:37,451 --> 00:30:43,331 But, um, you know, just when you see messages like that, we have no evidence 580 00:30:43,331 --> 00:30:45,761 that that doesn't mean it didn't happen. 581 00:30:45,791 --> 00:30:47,621 It just means that they, they can't prove it. 582 00:30:47,681 --> 00:30:48,881 It happened right? 583 00:30:49,586 --> 00:30:52,376 Prasanna Malaiyandi: Do you think, since you're just mentioning about 584 00:30:52,381 --> 00:30:56,786 this, do you think there are possible ways they could have leveraged 585 00:30:57,296 --> 00:31:01,286 security offerings, encryption, other things to protect these emails? 586 00:31:02,351 --> 00:31:06,191 So even if play did attack their exchange server, I guess it depends 587 00:31:06,191 --> 00:31:10,481 on what level they actually were able to exploit the exchange server. 588 00:31:11,846 --> 00:31:16,481 W. Curtis Preston: I, I mean, they had admin access to exchange. 589 00:31:16,871 --> 00:31:17,651 Prasanna Malaiyandi: that all bets are off? 590 00:31:17,876 --> 00:31:19,736 W. Curtis Preston: far as all bets are off. 591 00:31:19,736 --> 00:31:24,086 So even if there was encryption, I don't know if, if exchange has the 592 00:31:24,086 --> 00:31:27,926 ability to encrypt it or if they could store the data on encrypted drives. 593 00:31:28,286 --> 00:31:30,416 It doesn't matter once you're inside the application. 594 00:31:30,446 --> 00:31:30,866 Right. 595 00:31:31,286 --> 00:31:35,366 Because the, the, the data, it doesn't have record level encryption or anything 596 00:31:35,366 --> 00:31:38,456 like that, that I'm aware of, but um. 597 00:31:39,341 --> 00:31:39,941 Yeah. 598 00:31:40,151 --> 00:31:44,261 Um, and, and the, the, the other part, and this is by the way, this is the 599 00:31:44,261 --> 00:31:49,091 part of the story where Rackspace tries to shift blame by saying, 600 00:31:49,451 --> 00:31:55,301 well, hey, this exploit that, that we got hit by was a zero day exploit. 601 00:31:55,301 --> 00:32:01,661 Which is true in that it was an unknown exploit at the time that they got hit. 602 00:32:02,171 --> 00:32:06,281 But if they had put the patch on that they should have put 603 00:32:06,311 --> 00:32:08,111 at a minimum two weeks prior. 604 00:32:08,906 --> 00:32:13,946 They wouldn't have been because that patch fixed the unknown problem at the time. 605 00:32:15,206 --> 00:32:15,446 Prasanna Malaiyandi: Yeah. 606 00:32:15,451 --> 00:32:18,116 It's, I think, I know we always talk about it on the podcast. 607 00:32:18,116 --> 00:32:19,946 It's patch, patch, patch, right? 608 00:32:19,946 --> 00:32:23,711 When something comes, especially something with this level of severity, I. 609 00:32:25,001 --> 00:32:25,241 Right. 610 00:32:25,241 --> 00:32:29,561 It's like the log four J stuff that came out, what, 2021 December 611 00:32:29,951 --> 00:32:31,541 and what a mess that was as well. 612 00:32:31,546 --> 00:32:34,781 But everyone realized how severe of a security issue it was and 613 00:32:34,781 --> 00:32:36,131 started patching their systems. 614 00:32:36,851 --> 00:32:39,251 I think something similar needed to happen here as well. 615 00:32:39,251 --> 00:32:39,344 it. 616 00:32:39,344 --> 00:32:43,034 the fact that I get it, large customers, you have hosts, you need 617 00:32:43,039 --> 00:32:47,984 to schedule downtime, but a high severity security issue like this 618 00:32:47,984 --> 00:32:52,574 that was being exploited actively, like there's no reason you should have 619 00:32:52,574 --> 00:32:54,344 waited two weeks to apply a patch. 620 00:32:56,189 --> 00:32:56,699 W. Curtis Preston: Yeah. 621 00:32:56,699 --> 00:33:00,179 And, and this, this is kind of what I wanted to go at here. 622 00:33:00,269 --> 00:33:04,079 Uh, you know, we, there's a lot of lessons that could be learned from this incident. 623 00:33:04,079 --> 00:33:05,609 One is third party backup. 624 00:33:05,759 --> 00:33:08,609 If customers had it's had third party backup, which was totally 625 00:33:08,794 --> 00:33:09,754 possible, they would be done. 626 00:33:10,194 --> 00:33:10,314 I. 627 00:33:10,439 --> 00:33:14,819 They would've, they would not have suffered this outage, um, 628 00:33:14,849 --> 00:33:18,449 just like the Salesforce outage, um, in a previous episode, right? 629 00:33:18,449 --> 00:33:20,669 If they had third party backup, they could have fixed it in minutes. 630 00:33:21,269 --> 00:33:27,659 Um, the, and the other is this really hammers home when 631 00:33:27,659 --> 00:33:30,344 there's a really severe exploit. 632 00:33:30,929 --> 00:33:34,769 That is well known and there's a patch introduced for that exploit. 633 00:33:35,099 --> 00:33:36,899 You need to put it in now. 634 00:33:37,349 --> 00:33:37,859 Not. 635 00:33:37,919 --> 00:33:39,809 Oh, we've got our patch management system. 636 00:33:39,809 --> 00:33:43,109 We do this once every two weeks or whatever the, you know, whatever it is. 637 00:33:43,379 --> 00:33:48,989 Look at this company basically, you know, a month from the, the announcement of the 638 00:33:49,049 --> 00:33:55,679 patch, two months from the announcement of, uh, the actual exploit they got hit. 639 00:33:55,889 --> 00:33:58,859 Um, and if they had simply just put the patch in. 640 00:33:59,654 --> 00:34:02,984 When it was available, and, and I, I realized that it was available 641 00:34:02,984 --> 00:34:06,434 sooner and there was a minor pa, but, but that, that issue was fixed. 642 00:34:06,704 --> 00:34:10,244 There was two weeks between the time the, the patch was fully 643 00:34:10,244 --> 00:34:11,654 fixed and fully available. 644 00:34:12,104 --> 00:34:18,644 And the, when this actually happened, and it, it, it was yet another, this 645 00:34:18,644 --> 00:34:20,444 was already a suffering company. 646 00:34:20,624 --> 00:34:23,834 And if you look at the, the stock value of. 647 00:34:24,494 --> 00:34:26,384 Um, rack Rackspace. 648 00:34:26,864 --> 00:34:28,844 It had another sharp decline. 649 00:34:28,844 --> 00:34:29,564 What, what did you say? 650 00:34:29,564 --> 00:34:30,944 Wasn't it 15% 651 00:34:31,709 --> 00:34:32,069 Prasanna Malaiyandi: Yeah. 652 00:34:32,369 --> 00:34:32,549 In 653 00:34:32,804 --> 00:34:35,294 W. Curtis Preston: on December 2nd, which is the day they announced 654 00:34:35,294 --> 00:34:36,944 that this had happened, right? 655 00:34:37,964 --> 00:34:41,024 And so, you know, they were already suffering and this created 656 00:34:41,029 --> 00:34:45,194 yet another, uh, decline in the and which did not recover. 657 00:34:45,194 --> 00:34:46,904 It did not recover from that sharp decline. 658 00:34:48,554 --> 00:34:51,779 Just think about this, think about you. 659 00:34:51,784 --> 00:34:55,634 You really have to prioritize that patch management system, right? 660 00:34:55,634 --> 00:35:00,194 You really have to make sure that you put in patches as soon as they come out. 661 00:35:00,884 --> 00:35:05,084 Uh, you know, high level, high exploit. 662 00:35:06,119 --> 00:35:07,709 Patches need to go in. 663 00:35:07,709 --> 00:35:11,009 And again, I'm gonna once again say that I think the backup system needs 664 00:35:11,009 --> 00:35:13,139 to be at the front of the line, right? 665 00:35:13,589 --> 00:35:16,169 Patch those first, because that's the last line of defense. 666 00:35:16,229 --> 00:35:20,129 And then make sure you, you know, in this case, the patch only applied to exchange, 667 00:35:20,134 --> 00:35:21,984 but this is the point that we make. 668 00:35:21,984 --> 00:35:25,644 If they had just put the patches in, this event would've never happened. 669 00:35:25,669 --> 00:35:25,959 Prasanna Malaiyandi: Yeah. 670 00:35:26,574 --> 00:35:31,494 And you would still have 3000 customers using Rackspace hosted exchange, right? 671 00:35:31,494 --> 00:35:31,524 I. 672 00:35:32,559 --> 00:35:32,979 W. Curtis Preston: Yeah. 673 00:35:33,129 --> 00:35:33,279 Prasanna Malaiyandi: Yeah. 674 00:35:33,579 --> 00:35:36,459 The other comment I was gonna make, Curtis, is 675 00:35:37,099 --> 00:35:40,609 Given that they probably had the managed exchange solution for quite a while, 676 00:35:41,329 --> 00:35:44,059 I wonder if they ever had a process. 677 00:35:44,064 --> 00:35:47,479 I know we've talked about on the podcast of going back and looking 678 00:35:47,479 --> 00:35:52,789 at their disaster recovery plans or their backup and recovery procedures. 679 00:35:53,374 --> 00:35:56,584 Or if it was just sort of, Hey, we created this once, it should be fine. 680 00:35:56,584 --> 00:35:59,344 We never have to really use it, so we'll never go back and make sure 681 00:35:59,344 --> 00:36:00,664 it's up to date and all the rest. 682 00:36:02,614 --> 00:36:07,054 W. Curtis Preston: Well, I mean, clearly they didn't, they didn't. 683 00:36:09,334 --> 00:36:11,704 Account for ransomware, right? 684 00:36:11,944 --> 00:36:16,504 A typical DR recovery scenario would've worked fine here, right? 685 00:36:16,504 --> 00:36:20,374 If they, if the building caught fire, they knew how to re, they had backups. 686 00:36:20,554 --> 00:36:22,414 They knew how to restore their exchange servers. 687 00:36:22,624 --> 00:36:25,144 I think, I still think it took them longer than it should have, but 688 00:36:25,144 --> 00:36:26,704 they had, they had a plan for that. 689 00:36:27,304 --> 00:36:30,784 But that plan ransomware breaks a lot, 690 00:36:31,039 --> 00:36:32,959 Prasanna Malaiyandi: Why would ransomware be different than 691 00:36:32,959 --> 00:36:34,429 the building going up in smoke? 692 00:36:36,049 --> 00:36:38,389 W. Curtis Preston: Because, great question. 693 00:36:38,599 --> 00:36:43,609 Because they likely were still fighting the ransomware itself, right? 694 00:36:43,639 --> 00:36:47,389 When the building goes up in smoke, they could just literally restore everything. 695 00:36:47,449 --> 00:36:52,369 It's gonna take a couple days, but at this point they're, they have an unknown time. 696 00:36:52,384 --> 00:36:52,534 Prasanna Malaiyandi: Yeah. 697 00:36:53,359 --> 00:36:55,549 W. Curtis Preston: On the moment of December 2nd, they're like, we have 698 00:36:55,549 --> 00:37:00,319 been taken down and we have no long, no idea how long we're going to be down. 699 00:37:00,634 --> 00:37:00,814 Prasanna Malaiyandi: Yeah. 700 00:37:01,234 --> 00:37:01,564 Might be a. 701 00:37:01,564 --> 00:37:02,674 day, might be a week, might be a. 702 00:37:04,294 --> 00:37:05,464 W. Curtis Preston: Exactly right. 703 00:37:05,464 --> 00:37:08,134 They, they're looking at these, at these other businesses that 704 00:37:08,134 --> 00:37:09,724 are down for months at a time 705 00:37:10,114 --> 00:37:11,764 and they're saying, we don't want to be that. 706 00:37:11,764 --> 00:37:14,374 We don't wanna stop people's email for that amount of time. 707 00:37:14,734 --> 00:37:16,054 Boom, let's do over here. 708 00:37:16,294 --> 00:37:21,364 Had they, had it been adjusted regular, a fire of flood or whatever, they 709 00:37:21,364 --> 00:37:25,294 would, they should have been able to say, this is gonna take us three days. 710 00:37:25,534 --> 00:37:27,934 Three days is gonna stink, but. 711 00:37:28,359 --> 00:37:30,969 It's not enough for us to abandon our entire business 712 00:37:30,969 --> 00:37:32,529 model and move over to 365. 713 00:37:32,949 --> 00:37:36,879 In this case, it was an unknown scenario, unknown amount of 714 00:37:36,879 --> 00:37:38,019 time that they're gonna be down. 715 00:37:38,019 --> 00:37:40,599 And so they decide to do this method that ended up making 716 00:37:40,604 --> 00:37:42,009 everything take much longer. 717 00:37:43,059 --> 00:37:43,479 Right? 718 00:37:43,719 --> 00:37:45,309 Which, and they didn't test for this method. 719 00:37:46,429 --> 00:37:49,519 Prasanna Malaiyandi: so yes, I agree with everything, but I still wanna 720 00:37:49,519 --> 00:37:51,619 go back to another clarifying point. 721 00:37:52,489 --> 00:37:54,589 Couldn't they have treated ransomware? 722 00:37:54,619 --> 00:37:58,909 Like I wanna, I wonder if they actually had disaster recovery plans in place. 723 00:37:59,014 --> 00:37:59,504 Because 724 00:38:00,499 --> 00:38:01,789 if I yeah, and this. 725 00:38:01,894 --> 00:38:03,079 W. Curtis Preston: giving them the benefit of the doubt. 726 00:38:03,259 --> 00:38:05,779 Prasanna Malaiyandi: Yeah, because because in my mind, right, if I 727 00:38:05,779 --> 00:38:10,459 had a ransomware happen, right as a company, there are two options, right? 728 00:38:10,459 --> 00:38:14,209 One is I could try to figure out what all went wrong, rebuild my servers 729 00:38:14,539 --> 00:38:19,399 in the same data center, procure hardware, all the rest, or I could just 730 00:38:19,399 --> 00:38:22,969 treat it like a fire, shoot it in the head, connect completely, disconnect 731 00:38:22,969 --> 00:38:27,799 everything, validate my DR site is still good, that there's no ransomware 732 00:38:27,799 --> 00:38:32,839 there, and then bring everything up or restore on clean hardware, et cetera. 733 00:38:34,339 --> 00:38:34,759 Like 734 00:38:34,924 --> 00:38:37,174 W. Curtis Preston: if there's not ransomware or if there is ransomware. 735 00:38:37,339 --> 00:38:38,509 Prasanna Malaiyandi: even if there was ransomware, 736 00:38:40,774 --> 00:38:44,554 W. Curtis Preston: But the problem is if you have a solid backup system, you have a 737 00:38:44,554 --> 00:38:50,404 recent backup, and then you go to restore it, you know you're restoring the The 738 00:38:50,569 --> 00:38:51,289 Prasanna Malaiyandi: bad stuff. 739 00:38:51,589 --> 00:38:55,219 Yeah, So, but then that should have just been a matter of figuring 740 00:38:55,219 --> 00:38:56,809 out what is good and what is bad. 741 00:38:58,654 --> 00:39:01,354 W. Curtis Preston: Right, which is going to take an unknown amount of time. 742 00:39:01,654 --> 00:39:02,554 That was the problem. 743 00:39:03,214 --> 00:39:03,634 Right. 744 00:39:04,324 --> 00:39:08,914 And, and, um, yeah, I, I don't know if they had a DR plan. 745 00:39:08,919 --> 00:39:13,804 I do, I do ask myself why it took them, the amount of time it took them 746 00:39:13,804 --> 00:39:15,484 to restore all the exchange servers. 747 00:39:16,169 --> 00:39:20,194 I, I just, I just go off the, I just go off the, you know, the message that 748 00:39:20,194 --> 00:39:21,784 I, or the information that I have. 749 00:39:22,024 --> 00:39:22,324 Prasanna Malaiyandi: yeah. 750 00:39:22,939 --> 00:39:23,299 W. Curtis Preston: Yeah. 751 00:39:23,599 --> 00:39:27,109 And so the end of the story is this, uh, the company continued 752 00:39:27,109 --> 00:39:30,349 to suffer, uh, additional losses in the value of their company. 753 00:39:30,349 --> 00:39:34,159 They, they, they went from 800 million, then down to 350 million 754 00:39:34,159 --> 00:39:39,529 now, uh, and they, they are close as of the taping of this episode. 755 00:39:39,534 --> 00:39:41,424 They're close to a restructuring deal. 756 00:39:42,064 --> 00:39:45,124 I don't know if this event had ever happened, if how different 757 00:39:45,184 --> 00:39:49,444 the world would be right now, but it certainly didn't help. 758 00:39:50,344 --> 00:39:54,664 Um, so please, folks, all I can say is, you know, put those 759 00:39:54,664 --> 00:39:58,204 patches in when you, you know, and test your recovery procedures. 760 00:39:58,504 --> 00:40:01,294 All of the recovery scenarios, right? 761 00:40:01,969 --> 00:40:04,969 Prasanna Malaiyandi: And if ransomware isn't one of the recovery scenarios, then 762 00:40:04,969 --> 00:40:07,219 you need to rewrite your recovery plans. 763 00:40:07,394 --> 00:40:09,709 W. Curtis Preston: You need to reconsider your recovery scenarios. 764 00:40:09,709 --> 00:40:10,339 Absolutely. 765 00:40:11,119 --> 00:40:15,079 Uh, all right, well, uh, I think this was a good episode. 766 00:40:15,289 --> 00:40:15,799 Prasanna Malaiyandi: Yeah. 767 00:40:16,699 --> 00:40:17,554 Don't be like Rackspace 768 00:40:17,749 --> 00:40:20,269 W. Curtis Preston: was, you know, ouch. 769 00:40:22,039 --> 00:40:22,519 All right. 770 00:40:22,549 --> 00:40:23,179 Uh, thanks. 771 00:40:23,629 --> 00:40:24,379 Thanks for the chat. 772 00:40:24,734 --> 00:40:25,819 Prasanna Malaiyandi: like, uh. 773 00:40:26,404 --> 00:40:30,094 It was fun, Curtis, and I am sure I will hear all about your, 774 00:40:30,094 --> 00:40:32,229 uh, your, uh, event tonight. 775 00:40:33,889 --> 00:40:35,239 W. Curtis Preston: You, you will be hearing all about it. 776 00:40:35,239 --> 00:40:37,879 You'll probably be getting some live, some live chat or some 777 00:40:37,879 --> 00:40:39,529 live, uh, texts during the event. 778 00:40:40,459 --> 00:40:44,509 Um, and, uh, I, uh, be safe out there folks. 779 00:40:44,689 --> 00:40:45,829 That is a wrap.