1 00:00:00,050 --> 00:00:04,520 W. Curtis Preston: Spoiler alert, the company in this episode ceased to exist 2 00:00:04,760 --> 00:00:06,890 due to what happened in this story. 3 00:00:07,370 --> 00:00:11,150 Disasters happen, and since the cloud is just someone else's data 4 00:00:11,150 --> 00:00:15,170 center, they sometimes happen in the cloud, and sometimes they 5 00:00:15,170 --> 00:00:16,820 take companies along with them. 6 00:00:17,390 --> 00:00:21,620 This episode is the first in a series called Cloud Disasters. 7 00:00:21,920 --> 00:00:24,080 Each episode tells the real story. 8 00:00:24,365 --> 00:00:28,955 Of a company who failed to back up their cloud data and suffered as a result. 9 00:00:29,374 --> 00:00:31,895 And these aren't podunk cloud vendors either. 10 00:00:32,165 --> 00:00:35,345 Every vendor covered in this story is a major vendor. 11 00:00:35,879 --> 00:00:39,449 I'm not kidding around when I say you should back up your cloud data. 12 00:00:39,809 --> 00:00:43,049 I don't care that there are those who think I'm just pedaling fud. 13 00:00:43,424 --> 00:00:48,044 Newsflash, the only reason we've ever backed up anything is because 14 00:00:48,044 --> 00:00:51,824 of the fear of losing it and the uncertainty and doubt we have in 15 00:00:51,824 --> 00:00:53,804 the system that we're backing up. 16 00:00:54,734 --> 00:00:59,384 Fans of the podcast know, the whole reason that I became a backup specialist is that 17 00:00:59,384 --> 00:01:05,654 in 1993, I was unable to recover a very important Oracle database for my employer. 18 00:01:06,419 --> 00:01:08,579 I didn't want that to happen to anyone else. 19 00:01:08,579 --> 00:01:12,929 So I've dedicated myself to helping others protect themselves 20 00:01:13,079 --> 00:01:14,639 from feeling that awful feeling. 21 00:01:15,359 --> 00:01:19,769 This show is aimed at you, the unappreciated backup admin, and we want 22 00:01:19,769 --> 00:01:22,349 to turn you into a cyber recovery hero. 23 00:01:22,739 --> 00:01:24,809 This is the backup wrap up. 24 00:01:38,504 --> 00:01:39,704 Welcome to the show. 25 00:01:40,574 --> 00:01:47,354 I'm your host, w Curtis Preston, and I have with me my continued and necessary 26 00:01:47,864 --> 00:01:50,404 Tesla consultant Prasanna Malaiyandi. 27 00:01:50,404 --> 00:01:51,384 How's it going Prasanna 28 00:01:51,404 --> 00:01:51,584 persona 29 00:01:51,584 --> 00:01:51,585 Prasanna Malaiyandi: Pana. 30 00:01:52,089 --> 00:01:54,879 I am doing well Curtis, and how have you been? 31 00:01:55,754 --> 00:01:58,214 W. Curtis Preston: I, well, as you know, I have been. 32 00:01:59,399 --> 00:02:00,089 Fiddling. 33 00:02:04,089 --> 00:02:05,229 Prasanna Malaiyandi: Uh, why am I not surprised? 34 00:02:05,999 --> 00:02:12,029 W. Curtis Preston: I have been, fiddling with the automations of my Tesla now. 35 00:02:12,809 --> 00:02:15,869 Tesla's come in, I'm gonna say two battery flavors, right? 36 00:02:15,869 --> 00:02:18,209 There's NMC and LFP. 37 00:02:18,563 --> 00:02:21,063 Nickel Manganese Cobalt and Lithium Iron Phosphate. 38 00:02:21,063 --> 00:02:23,130 And I have the latter. 39 00:02:23,130 --> 00:02:27,655 And the latter is not supposed to be subject to the same don't 40 00:02:27,655 --> 00:02:30,055 charge it to a hundred percent unless you need it right away. 41 00:02:30,655 --> 00:02:31,255 Issue. 42 00:02:31,405 --> 00:02:31,795 Uh, 43 00:02:31,895 --> 00:02:32,765 Prasanna Malaiyandi: In fact tell 44 00:02:33,305 --> 00:02:34,295 you to charge it to a hundred 45 00:02:34,420 --> 00:02:35,950 W. Curtis Preston: I do tell you to charge it to a hundred 46 00:02:35,950 --> 00:02:37,540 percent at least once a week. 47 00:02:37,960 --> 00:02:40,480 Um, but I was charging it to a hundred percent every day. 48 00:02:40,480 --> 00:02:43,270 And I was thinking that I, that even though it doesn't have necessarily 49 00:02:43,270 --> 00:02:46,810 the same issues, I could still get better battery life by not charging to 50 00:02:46,810 --> 00:02:48,460 a hundred percent unless I needed to. 51 00:02:48,760 --> 00:02:51,370 Uh, which, uh, for me is not all the time. 52 00:02:52,375 --> 00:02:57,505 And so, but I'm also very absent-minded, and so I couldn't like lower the 53 00:02:57,510 --> 00:03:01,855 percentage and then remember to, to, to, to raise the percentage later. 54 00:03:01,915 --> 00:03:07,165 And so, uh, I found this wonderful app called Tessie, and I've been obsessing 55 00:03:07,165 --> 00:03:09,955 over it for about a week or two. 56 00:03:10,130 --> 00:03:10,520 Prasanna Malaiyandi: A week. 57 00:03:10,610 --> 00:03:11,640 It's been nine days. 58 00:03:12,005 --> 00:03:12,295 W. Curtis Preston: Yeah. 59 00:03:12,445 --> 00:03:12,735 Yeah. 60 00:03:13,370 --> 00:03:16,940 I will say if there's any listeners that are a Tesla owner, I. 61 00:03:16,940 --> 00:03:20,900 Uh, you should just get Tessie, uh, T-E-S-S-I-E in the app store. 62 00:03:21,340 --> 00:03:22,900 I bought it for the automation. 63 00:03:22,930 --> 00:03:27,970 What I also got was this immense amount of analytics and reporting 64 00:03:27,970 --> 00:03:30,580 and, um, all this great, great stuff. 65 00:03:30,580 --> 00:03:31,750 And also reminders. 66 00:03:31,900 --> 00:03:35,380 That's another, speaking of being absent-minded, it will tell me, hey. 67 00:03:35,855 --> 00:03:38,795 Idiot, you're home and your car's not plugged in. 68 00:03:40,505 --> 00:03:41,765 Prasanna Malaiyandi: Yeah, because how many times has that 69 00:03:41,765 --> 00:03:44,315 happened to you, Curtis, that you've come home and you're like, 70 00:03:44,320 --> 00:03:47,680 W. Curtis Preston: you know, a couple, uh, enough that it was annoying. 71 00:03:47,770 --> 00:03:52,750 Uh, it, I will say it, nothing is more annoying than, you know, basically 72 00:03:52,750 --> 00:03:56,920 driving your car down to, you know, the electrical equivalent of fumes. 73 00:03:57,280 --> 00:04:00,820 And then, um, and then getting up in the morning and going, 74 00:04:00,940 --> 00:04:02,410 okay, I'm ready to drive today. 75 00:04:02,530 --> 00:04:04,690 And you're like, oh, crap, I gotta go to the. 76 00:04:05,280 --> 00:04:08,730 I gotta go the supercharger for 20 minutes before I could do anything else. 77 00:04:09,540 --> 00:04:13,980 Um, yeah, so it's got, you know, it's got that the notifications, it 78 00:04:13,980 --> 00:04:16,350 reminds me to, to rotate my tires. 79 00:04:16,400 --> 00:04:21,230 And also gives me analytics about my driving and my efficiency and, yeah. 80 00:04:21,290 --> 00:04:22,040 So, yeah. 81 00:04:22,190 --> 00:04:24,200 So happy, happy, happy, happy, happy. 82 00:04:25,580 --> 00:04:28,730 Um, but you know, who's not happy? 83 00:04:29,385 --> 00:04:36,595 The people that are gonna be in this new series, they're not happy. 84 00:04:37,225 --> 00:04:39,475 And this is a new series. 85 00:04:40,085 --> 00:04:45,635 One of the things that you've heard us say is what Bana. 86 00:04:47,670 --> 00:04:48,750 Prasanna Malaiyandi: Back up the cloud. 87 00:04:48,890 --> 00:04:50,690 W. Curtis Preston: Yes, backup up the cloud. 88 00:04:50,690 --> 00:04:52,070 The cloud is not magic. 89 00:04:52,140 --> 00:04:53,874 There is no such thing as the cloud. 90 00:04:53,879 --> 00:04:55,914 It's just somebody else's computer. 91 00:04:56,449 --> 00:04:59,454 Uh uh, the cloud doesn't magically back up itself. 92 00:05:00,034 --> 00:05:04,914 Despite what you may have been told the cloud there are parts of the cloud where. 93 00:05:05,394 --> 00:05:07,674 People generally agree with me. 94 00:05:07,819 --> 00:05:13,614 I, I think probably the best example would be something like AWS, you know, like EC2. 95 00:05:13,674 --> 00:05:17,004 They're like, yes, we know EC2 needs to be backed up. 96 00:05:17,364 --> 00:05:21,414 Um, and, and then we just argue a little bit over how that's going to be done. 97 00:05:22,164 --> 00:05:25,944 But when we get to the extreme it, the other end of that, we get the 98 00:05:25,944 --> 00:05:31,314 SaaS world, we get the Microsoft 365 lovers who say, oh, this, that, 99 00:05:31,314 --> 00:05:32,694 this doesn't need to be backed up. 100 00:05:34,526 --> 00:05:35,576 you know, or G 101 00:05:35,666 --> 00:05:45,501 Suite and, we have gone back over the last, uh, several years and selected, um, 102 00:05:47,001 --> 00:05:51,501 a number of, you know, poor victims of. 103 00:05:52,056 --> 00:06:02,436 This belief and, um, I, I can think of no better, uh, story to start this 104 00:06:02,441 --> 00:06:08,706 out than code spaces because it, I, I think it was kind of the first, 105 00:06:09,196 --> 00:06:11,896 Prasanna Malaiyandi: it was probably the first that was sort of well 106 00:06:11,896 --> 00:06:20,386 publicized as well as having very dire consequences for not backing up the cloud. 107 00:06:20,386 --> 00:06:20,416 I. 108 00:06:21,276 --> 00:06:21,846 W. Curtis Preston: Right. 109 00:06:22,866 --> 00:06:28,831 And it, it's interesting from a timing perspective, it happened in 2014. 110 00:06:29,731 --> 00:06:38,071 Which to me is the year of the beginning of the massive level of ransomware. 111 00:06:38,491 --> 00:06:41,461 Now, I know ransomware actually goes way back longer. 112 00:06:41,461 --> 00:06:43,861 You know, much longer be before that. 113 00:06:44,221 --> 00:06:48,511 But to me, 2014 is really when I started seeing ransomware kind of everywhere. 114 00:06:49,046 --> 00:06:53,336 And this was technically a ransomware attack, not in the traditional sense 115 00:06:53,336 --> 00:07:01,146 that we think of today, but, but it was, so Code spaces.com was a site, and this 116 00:07:01,146 --> 00:07:03,456 is the, the irony of all ri ironies. 117 00:07:03,456 --> 00:07:07,386 And, and I will say that unlike Alanis Morissette, I actually 118 00:07:07,386 --> 00:07:09,636 know what the word ironic means. 119 00:07:10,056 --> 00:07:11,316 Um, I love the song. 120 00:07:11,316 --> 00:07:15,366 It's just, there's so many things to that song that are not in any way ironic. 121 00:07:15,366 --> 00:07:16,146 They just suck. 122 00:07:16,926 --> 00:07:20,976 Um, code spaces.com was a site to store your code. 123 00:07:20,981 --> 00:07:22,926 It was like, um. 124 00:07:22,961 --> 00:07:23,381 Prasanna Malaiyandi: GitHub. 125 00:07:24,426 --> 00:07:27,786 W. Curtis Preston: It was like, it was like a GitHub and they had 126 00:07:27,966 --> 00:07:33,276 many, many customers and it was a safe space to store your code. 127 00:07:33,276 --> 00:07:35,256 Hence the name code spaces.com. 128 00:07:35,861 --> 00:07:39,641 Prasanna Malaiyandi: Yeah, so basically like you mentioned, it was a place 129 00:07:39,641 --> 00:07:43,631 companies could store their code and this was way back in the day. 130 00:07:43,631 --> 00:07:46,631 And so it's like, hey, if you just have your own code sitting locally 131 00:07:46,631 --> 00:07:49,841 on your system, because not everyone was comfortable with the cloud, they 132 00:07:49,841 --> 00:07:53,471 offered a service that allows you to store your code there and keep it safe. 133 00:07:54,071 --> 00:07:58,451 And now according to a cash version of their website, because of course 134 00:07:58,451 --> 00:08:03,251 you can't find anything anymore about them, uh, they had over 200 customers. 135 00:08:03,716 --> 00:08:08,996 A week or 200 companies a week using their service, which isn't 136 00:08:09,776 --> 00:08:11,156 like small beans, right? 137 00:08:11,216 --> 00:08:14,426 And yes, it's not like the thousands of millions, but there are 200 138 00:08:14,426 --> 00:08:19,616 customers who now no longer have access to their code anymore because 139 00:08:19,616 --> 00:08:20,936 of what happened in code spaces. 140 00:08:21,866 --> 00:08:26,466 The other interesting thing is according to their websites, and I will quote 141 00:08:26,856 --> 00:08:31,296 backing up, data is one thing, but is meaningless without a recovery plan. 142 00:08:31,716 --> 00:08:35,916 Not only that, a recovery plan and one that is well practiced and 143 00:08:35,916 --> 00:08:37,956 proven to work time and time again. 144 00:08:39,246 --> 00:08:43,356 Code Spaces has a full recovery plan that has been proven to 145 00:08:43,356 --> 00:08:46,146 work and is in fact practiced. 146 00:08:46,766 --> 00:08:47,096 W. Curtis Preston: That 147 00:08:47,286 --> 00:08:48,606 Prasanna Malaiyandi: do you think about that, Curtis? 148 00:08:49,451 --> 00:08:51,221 W. Curtis Preston: That sounds really good. 149 00:08:51,971 --> 00:09:00,371 Um, you know, based on what we know happened, clearly they didn't test for 150 00:09:01,446 --> 00:09:02,316 Prasanna Malaiyandi: all scenarios. 151 00:09:02,741 --> 00:09:03,191 W. Curtis Preston: Right. 152 00:09:03,281 --> 00:09:06,491 And they specifically didn't test for cyber attack. 153 00:09:07,061 --> 00:09:08,441 Um, but. 154 00:09:08,586 --> 00:09:11,496 Prasanna Malaiyandi: the time could you fault them really? 155 00:09:12,281 --> 00:09:12,821 W. Curtis Preston: I don't know. 156 00:09:14,381 --> 00:09:16,721 Well, they had poor backup design. 157 00:09:16,961 --> 00:09:18,731 Just, you know what, what we're 158 00:09:18,731 --> 00:09:20,411 gonna get to, what we're gonna find out. 159 00:09:20,501 --> 00:09:21,401 They had full backup. 160 00:09:21,401 --> 00:09:23,351 They had poor backup design. 161 00:09:23,351 --> 00:09:25,686 They failed to follow what persona? 162 00:09:25,736 --> 00:09:26,786 Prasanna Malaiyandi: The 3, 2, 1 rule. 163 00:09:26,786 --> 00:09:29,276 You know, we haven't talked about this in a long, long time. 164 00:09:29,276 --> 00:09:30,896 Do you wanna quickly mention it to some of our 165 00:09:31,141 --> 00:09:31,531 W. Curtis Preston: Yeah. 166 00:09:31,531 --> 00:09:35,821 So the 3, 2, 1 role, and by the way, many companies have said 167 00:09:35,821 --> 00:09:37,321 it's gotta be more than 3, 2, 1. 168 00:09:37,321 --> 00:09:38,191 Yes, I agree. 169 00:09:38,311 --> 00:09:38,641 Right? 170 00:09:38,641 --> 00:09:42,811 These days it has to be more than 3, 2, 1, but if it's not 3, 2, 1, there's no point 171 00:09:42,816 --> 00:09:44,641 in talking about the other ones, right? 172 00:09:45,001 --> 00:09:48,186 Having at least three versions on two different media, and the, the, 173 00:09:48,661 --> 00:09:51,961 the idea here is on, on things that are subject to different. 174 00:09:52,836 --> 00:09:54,396 Risk profiles, right? 175 00:09:54,636 --> 00:09:58,986 Maybe it's disco and tape, maybe it's, uh, on-Prem and off-Prem. 176 00:09:59,016 --> 00:10:00,846 Maybe it's, you know, on-prem and Cloud. 177 00:10:01,116 --> 00:10:04,446 Maybe it's, um, you know, a different region, et cetera. 178 00:10:04,506 --> 00:10:08,406 And then one three, the one is make sure that there's something off site. 179 00:10:08,886 --> 00:10:11,316 Um, they had neither the two nor the one, 180 00:10:11,376 --> 00:10:12,811 but we're gonna get to that in a minute. 181 00:10:12,971 --> 00:10:15,221 Prasanna Malaiyandi: and, and for those listeners who wanna know more, 182 00:10:15,221 --> 00:10:19,961 we actually had an episode with the person who coined the term 3, 2, 1, who 183 00:10:19,961 --> 00:10:22,211 comes from digital photography, in fact. 184 00:10:22,631 --> 00:10:24,881 And so you should go take a listen to that episode. 185 00:10:25,061 --> 00:10:25,241 We'll 186 00:10:25,351 --> 00:10:26,041 W. Curtis Preston: Yeah, we'll put. 187 00:10:26,461 --> 00:10:27,421 Yeah, we'll put a show notes. 188 00:10:27,421 --> 00:10:28,601 Peter Krogh. 189 00:10:28,771 --> 00:10:30,031 Um, great guy. 190 00:10:30,961 --> 00:10:32,851 So what happened? 191 00:10:32,881 --> 00:10:38,731 Well, we have that a hacker gained privilege credentials that we still 192 00:10:38,731 --> 00:10:41,401 don't know how that happened, right? 193 00:10:41,671 --> 00:10:46,411 They're saying probably through phishing or possibly through, uh, 194 00:10:46,441 --> 00:10:48,991 stored EC2 access keys in a public code 195 00:10:48,996 --> 00:10:49,891 repository. 196 00:10:50,391 --> 00:10:52,796 Prasanna Malaiyandi: Which happened a lot back then and still does today. 197 00:10:52,936 --> 00:10:53,446 W. Curtis Preston: Yeah. 198 00:10:53,476 --> 00:10:53,866 Yeah. 199 00:10:53,866 --> 00:10:59,836 The, um, there was a, uh, security researcher at Tripwire that said, that 200 00:10:59,836 --> 00:11:06,146 this is a problem for people how to, how to, manage authentication codes like this. 201 00:11:06,476 --> 00:11:10,466 And, uh, they said they, they had seen thousands of EC2 accounts 202 00:11:10,466 --> 00:11:15,386 abused after storing their EC2 keys in public code repositories. 203 00:11:15,926 --> 00:11:16,466 Ouch. 204 00:11:16,766 --> 00:11:17,216 Right. 205 00:11:17,276 --> 00:11:18,896 Um, but we don't, so we don't know. 206 00:11:19,266 --> 00:11:19,716 Someone 207 00:11:19,716 --> 00:11:20,346 got access. 208 00:11:20,526 --> 00:11:20,706 Yeah. 209 00:11:21,141 --> 00:11:24,591 Prasanna Malaiyandi: Would it be ironic if Code Spaces was using code spaces 210 00:11:24,591 --> 00:11:28,701 to store their code and they left a public repository with their EC2 key? 211 00:11:29,726 --> 00:11:31,496 W. Curtis Preston: That wouldn't indeed be ironic. 212 00:11:31,916 --> 00:11:37,286 Um, but yeah, so we don't know exactly how this hacker, uh, this, 213 00:11:37,286 --> 00:11:43,856 you know, bad actor got access to the, to the environment, but they did. 214 00:11:44,186 --> 00:11:47,576 And the first thing that they did was they started a DDoS attack. 215 00:11:47,846 --> 00:11:48,476 Uh, what is a 216 00:11:48,476 --> 00:11:49,766 DDoS attack persona? 217 00:11:49,881 --> 00:11:52,881 Prasanna Malaiyandi: This is a DI distributed denial of service, so 218 00:11:52,881 --> 00:11:59,241 you have a bunch of servers outside hammering various servers at a company. 219 00:11:59,826 --> 00:12:02,496 Causing it, flooding it with a lot of traffic, which then 220 00:12:02,496 --> 00:12:04,176 causes it to stop responding. 221 00:12:04,176 --> 00:12:08,916 So if you're code spaces, someone did a DDoS attack on you now, you wouldn't 222 00:12:08,916 --> 00:12:13,386 be able to serve and function as a service to those 200 paying customers. 223 00:12:13,916 --> 00:12:16,316 W. Curtis Preston: Yeah, the really interesting thing about this is that the 224 00:12:16,321 --> 00:12:18,626 DDoS attack was apparently subterfuge. 225 00:12:19,076 --> 00:12:19,466 Right. 226 00:12:19,556 --> 00:12:21,416 Um, you know, look over here 227 00:12:21,421 --> 00:12:23,006 while I, nothing up but 228 00:12:23,016 --> 00:12:24,936 Prasanna Malaiyandi: which I think is still common today, right? 229 00:12:25,026 --> 00:12:27,816 There are still a lot of companies who they're trying to hide their tracks 230 00:12:27,816 --> 00:12:31,206 and they're like, Hey, everyone's gonna fight, and DDoS were common. 231 00:12:31,206 --> 00:12:33,726 And so they have a plan in place and everyone's scrambling there where you're 232 00:12:33,726 --> 00:12:36,066 like, Hey, look at my left hand while I'm doing something with my right. 233 00:12:37,451 --> 00:12:41,231 W. Curtis Preston: And then I, I think the, I don't know if interesting 234 00:12:41,231 --> 00:12:46,841 is right, but the hacker left contact details for themselves in 235 00:12:46,841 --> 00:12:49,481 the customer's Amazon dashboard. 236 00:12:49,931 --> 00:12:50,351 Right. 237 00:12:50,381 --> 00:12:55,001 Uh, this was, this was, I, I think, I think maybe we haven't mentioned it. 238 00:12:55,001 --> 00:12:56,681 This was an AWS customer. 239 00:12:56,871 --> 00:12:58,731 Prasanna Malaiyandi: And this is why we call it ransomware, right? 240 00:12:58,731 --> 00:13:00,831 Even though it's not your traditional ransomware like we 241 00:13:00,831 --> 00:13:02,871 think today, it more or less is. 242 00:13:03,851 --> 00:13:04,331 W. Curtis Preston: Right. 243 00:13:04,421 --> 00:13:04,721 Yeah. 244 00:13:04,721 --> 00:13:06,521 So then, uh, I'll quote from there. 245 00:13:06,551 --> 00:13:09,011 There was a page, by the way, there was a page. 246 00:13:10,211 --> 00:13:15,356 Uh, basically code spaces.com died after this, and it became a, they, 247 00:13:15,431 --> 00:13:18,431 they replaced it with just one page that says, here's what happened. 248 00:13:18,431 --> 00:13:21,791 And we're, I'm gonna read quotes from that page that are no longer 249 00:13:21,796 --> 00:13:23,411 available because they sold the domain. 250 00:13:24,011 --> 00:13:28,331 Um, upon realizing that somebody had access to our control panel, we started 251 00:13:28,331 --> 00:13:31,901 to investigate how access had been gained and what access that person 252 00:13:31,901 --> 00:13:35,201 had, uh, to the data in our systems. 253 00:13:35,546 --> 00:13:40,586 It became clear that so far no machine had access had been achieved due to the 254 00:13:40,586 --> 00:13:43,226 intruder not having our private keys. 255 00:13:43,796 --> 00:13:47,906 Um, that's what they thought, right? 256 00:13:48,476 --> 00:13:51,086 Um, but things 257 00:13:51,146 --> 00:13:54,296 turned, uh, yeah, turned ugly. 258 00:13:54,476 --> 00:13:55,821 Uh, why don't you talk, talk about 259 00:13:55,821 --> 00:13:56,501 the next one there. 260 00:13:57,561 --> 00:13:59,481 Prasanna Malaiyandi: So what was the response? 261 00:13:59,761 --> 00:14:03,976 So Code Spaces did a smart thing and one of their first response. 262 00:14:04,576 --> 00:14:10,546 Actions was to change all of its EC2 passwords, but quickly, code 263 00:14:10,546 --> 00:14:15,766 spaces discovered that the attacker had created backup logins, which any 264 00:14:15,766 --> 00:14:17,326 sensible person's going to do, right? 265 00:14:17,331 --> 00:14:21,106 You're never gonna say, Hey, I'm only gonna have one admin in my environment. 266 00:14:21,736 --> 00:14:28,636 And so the attacker was able to create all these backup logins, and so now they 267 00:14:28,636 --> 00:14:31,066 were able to just go back into the system. 268 00:14:32,116 --> 00:14:34,246 And continue doing the attacks. 269 00:14:34,276 --> 00:14:40,096 And once they realized that Code Spaces was trying to actually recover and take 270 00:14:40,096 --> 00:14:45,106 control from the attacker, the attacker then started to go and just delete things 271 00:14:45,106 --> 00:14:48,616 from the control panel because they had super user access at that point. 272 00:14:48,616 --> 00:14:48,946 Right. 273 00:14:48,946 --> 00:14:50,506 So they could do whatever they wanted. 274 00:14:50,961 --> 00:14:53,751 W. Curtis Preston: Yeah, somewhere in here, in the version of the story 275 00:14:53,756 --> 00:14:57,981 that I have, I remember there was a d, there was a, there was an attempt 276 00:14:57,981 --> 00:15:01,466 at a ransom, basically give us this amount of money, or, or, or we're 277 00:15:01,466 --> 00:15:02,996 gonna, you know, do bad things. 278 00:15:03,566 --> 00:15:08,546 Um, but the, but you know, they, they obviously didn't want to pay the ransom. 279 00:15:09,431 --> 00:15:11,086 And then, uh, then what 280 00:15:11,286 --> 00:15:11,406 happened? 281 00:15:12,776 --> 00:15:14,846 Prasanna Malaiyandi: And so finally code spaces got their control 282 00:15:14,851 --> 00:15:17,816 panel access back, but not before. 283 00:15:17,816 --> 00:15:23,426 The attacker had caused quite a bit of damage, so the attacker had gone 284 00:15:23,756 --> 00:15:30,666 removed all the EBS syn snapshots, S3 buckets, all the amis, which are the 285 00:15:30,666 --> 00:15:36,846 Amazon machine instances, some EBS instances and several machine instances. 286 00:15:37,746 --> 00:15:42,336 And this is a quote from the same webpage that Curtis was talking about. 287 00:15:42,666 --> 00:15:47,136 In summary, most of our data backups, machine configurations 288 00:15:47,346 --> 00:15:51,396 and offsite backups were either partially or completely deleted. 289 00:15:52,316 --> 00:15:54,131 W. Curtis Preston: What I, I, you know, I did. 290 00:15:54,136 --> 00:15:56,021 I never noticed that phrase before. 291 00:15:56,966 --> 00:15:58,676 And they said And offsite backups. 292 00:15:58,676 --> 00:15:58,886 What? 293 00:15:58,886 --> 00:15:59,876 Offsite backups. 294 00:16:00,296 --> 00:16:05,486 What I mean, what be, because the hacker only had access to 295 00:16:05,486 --> 00:16:06,416 the one account. 296 00:16:07,161 --> 00:16:09,321 Prasanna Malaiyandi: I am guessing that they replicated 297 00:16:10,131 --> 00:16:12,531 to another AWS region within the 298 00:16:12,536 --> 00:16:12,956 W. Curtis Preston: Okay. 299 00:16:12,986 --> 00:16:13,016 Okay. 300 00:16:13,706 --> 00:16:14,906 Another zone within the 301 00:16:14,906 --> 00:16:15,476 same account. 302 00:16:15,476 --> 00:16:15,836 All right. 303 00:16:16,521 --> 00:16:17,541 Prasanna Malaiyandi: Or or another region. 304 00:16:18,561 --> 00:16:18,831 Yeah, 305 00:16:19,136 --> 00:16:19,586 W. Curtis Preston: Yeah. 306 00:16:19,646 --> 00:16:20,876 yeah, But still within the 307 00:16:20,876 --> 00:16:21,566 same account 308 00:16:22,491 --> 00:16:23,841 Prasanna Malaiyandi: Possibly within the same account. 309 00:16:24,146 --> 00:16:24,596 W. Curtis Preston: Yeah. 310 00:16:24,716 --> 00:16:27,026 Um, yeah, yeah, possibly you're right. 311 00:16:27,026 --> 00:16:31,796 We don't know for sure, but if it wasn't the same account, then the hacker had 312 00:16:31,796 --> 00:16:33,536 to gain access to multiple accounts. 313 00:16:33,536 --> 00:16:33,806 Right. 314 00:16:34,036 --> 00:16:34,326 Prasanna Malaiyandi: Yeah. 315 00:16:34,331 --> 00:16:34,341 Yep. 316 00:16:35,126 --> 00:16:36,206 W. Curtis Preston: Um, yeah. 317 00:16:36,206 --> 00:16:41,036 So basically this is the equivalent of blowing up somebody's data center, right? 318 00:16:41,636 --> 00:16:43,976 Um, because basically they just. 319 00:16:45,221 --> 00:16:50,381 In a matter of a few keystrokes, they just deleted essentially everything, right? 320 00:16:50,381 --> 00:16:54,491 Everything that mattered, or enough things that mattered that they, um, 321 00:16:55,061 --> 00:16:56,561 you know, uh, took out the company. 322 00:16:57,171 --> 00:16:59,241 Prasanna Malaiyandi: One of the things that I don't know if you found any 323 00:16:59,241 --> 00:17:03,741 information about Curtis is did they ever reach out to law enforcement 324 00:17:03,741 --> 00:17:10,491 or even AWS's security operations to be like, Hey, I have this issue. 325 00:17:11,151 --> 00:17:12,171 Can you help me? 326 00:17:12,941 --> 00:17:15,731 W. Curtis Preston: Yeah, we don't, I, I, I'm going to. 327 00:17:16,256 --> 00:17:21,896 Assume that once you know, the feces hit the rotary oscillator, 328 00:17:21,896 --> 00:17:23,696 I'm sure they called AWS. 329 00:17:23,726 --> 00:17:28,826 I mean, of course they called AWS, but what we know is that 330 00:17:30,116 --> 00:17:35,636 right, if you know AWS isn't magic, and if you didn't follow the architecture 331 00:17:35,636 --> 00:17:38,516 and do the things that you were supposed to do, A-W-A-W-S can't 332 00:17:38,516 --> 00:17:39,146 undo that. 333 00:17:39,726 --> 00:17:42,366 Prasanna Malaiyandi: Yeah, well, I was just wondering before it changed, its EC2 334 00:17:42,371 --> 00:17:47,076 passwords, if they had reached out to AWS, if they could have helped them in some 335 00:17:47,081 --> 00:17:50,406 way or been like, Hey, here are the best practices for locking down your account, 336 00:17:51,041 --> 00:17:54,461 W. Curtis Preston: Yeah, I'm, I'm gonna guess based on how they responded 337 00:17:54,466 --> 00:17:56,651 that they did not do that right. 338 00:17:56,751 --> 00:17:59,691 Um, I don't think their response was the best thing they 339 00:17:59,691 --> 00:18:00,711 could have done at the time. 340 00:18:01,281 --> 00:18:04,341 Um, so then we have, um. 341 00:18:05,181 --> 00:18:08,721 So here's again, I'm reading from their co their, from their quote 342 00:18:08,721 --> 00:18:12,681 here, code spaces will not be able to operate beyond this point. 343 00:18:12,681 --> 00:18:17,361 The cost of resolving this issue to date and the expected cost of refunding. 344 00:18:17,361 --> 00:18:21,081 Customers who have been left without the service they paid for, we'll 345 00:18:21,081 --> 00:18:24,876 put code spaces in an irreversible position, both financially and 346 00:18:24,891 --> 00:18:27,561 in terms of ongoing credibility. 347 00:18:28,161 --> 00:18:28,851 No kidding. 348 00:18:29,451 --> 00:18:31,911 Um, as such, at this point. 349 00:18:33,021 --> 00:18:37,041 We have no alternative but to cease trading and concentrate on supporting 350 00:18:37,041 --> 00:18:42,081 our affected customers and exporting any remaining data they have left with us. 351 00:18:42,846 --> 00:18:43,136 Prasanna Malaiyandi: Ouch. 352 00:18:45,531 --> 00:18:48,831 W. Curtis Preston: That is, um, that's a tough one. 353 00:18:49,941 --> 00:18:55,926 So basically, you know, hacker gets in Hacker, uh, you know. 354 00:18:56,586 --> 00:18:57,726 Offers are ransom. 355 00:18:57,996 --> 00:19:02,616 They try to, instead of paying the ransom, they try to lock 356 00:19:02,616 --> 00:19:05,736 the hacker out unsuccessfully, 357 00:19:06,286 --> 00:19:06,506 Prasanna Malaiyandi: Yep. 358 00:19:06,546 --> 00:19:12,426 W. Curtis Preston: and then the hacker deletes the company, the uh, right. 359 00:19:12,431 --> 00:19:16,626 I mean, they deleted basically everything, you know, as much as they could get 360 00:19:16,626 --> 00:19:21,066 access to in that, you know, in that 361 00:19:21,066 --> 00:19:21,546 account. 362 00:19:22,801 --> 00:19:24,301 Prasanna Malaiyandi: That's crazy. 363 00:19:25,311 --> 00:19:25,611 W. Curtis Preston: Yeah. 364 00:19:25,791 --> 00:19:26,151 Yeah. 365 00:19:26,551 --> 00:19:32,641 This is one of those where it's like, I, I always used this story to 366 00:19:32,646 --> 00:19:36,241 recommend backup design for the cloud. 367 00:19:36,641 --> 00:19:41,231 Even though I don't think I've ever encountered someone who says, 368 00:19:41,231 --> 00:19:44,621 well, I don't need to back up EC2, I don't think I've ever heard that. 369 00:19:45,576 --> 00:19:45,846 Prasanna Malaiyandi: Yeah. 370 00:19:46,301 --> 00:19:50,801 W. Curtis Preston: I have very commonly found people whose backup design was very 371 00:19:50,801 --> 00:19:53,561 similar to this company's backup design. 372 00:19:54,521 --> 00:19:57,521 And, um, so, well let me ask you this. 373 00:19:57,526 --> 00:20:04,841 What do you think, um, what could they have done differently to stop this? 374 00:20:05,816 --> 00:20:07,016 To prevent this from happening. 375 00:20:08,601 --> 00:20:12,681 Prasanna Malaiyandi: I would say the first could have been take your 376 00:20:12,681 --> 00:20:14,721 backups to a different account. 377 00:20:17,601 --> 00:20:20,661 In a different region, or even in the same region, that would've 378 00:20:20,811 --> 00:20:22,731 at least kept your data safe. 379 00:20:24,446 --> 00:20:27,776 W. Curtis Preston: Yeah, I, I, yeah, I, I completely agree with you. 380 00:20:27,776 --> 00:20:32,846 I would probably just say as long as you, as long as you're, I, I guess what. 381 00:20:33,986 --> 00:20:37,076 I'm, I'm trying to, I'm factor, I'm thinking in my head, like, 382 00:20:37,076 --> 00:20:41,486 from a cost perspective, does it cost extra to send to another 383 00:20:41,486 --> 00:20:42,056 region? 384 00:20:42,386 --> 00:20:42,896 Is that, 385 00:20:43,386 --> 00:20:44,511 Prasanna Malaiyandi: Yeah, usually it does. 386 00:20:44,751 --> 00:20:45,051 Yep. 387 00:20:45,326 --> 00:20:45,566 W. Curtis Preston: Okay. 388 00:20:45,571 --> 00:20:46,166 So 389 00:20:46,806 --> 00:20:48,396 Prasanna Malaiyandi: But they already have an offsite backups, 390 00:20:48,396 --> 00:20:48,726 right? 391 00:20:48,786 --> 00:20:49,206 So. 392 00:20:49,556 --> 00:20:50,126 W. Curtis Preston: right? 393 00:20:50,126 --> 00:20:50,456 Yeah. 394 00:20:50,456 --> 00:20:50,726 So. 395 00:20:52,301 --> 00:20:55,811 Um, I, I think that your backup should be copied to another 396 00:20:55,811 --> 00:20:57,671 region and another account. 397 00:20:57,866 --> 00:21:01,691 I, I actually, and, and, and, and I'll say that, you know, my, my 398 00:21:01,691 --> 00:21:05,021 opinion is somewhat peppered by having worked for a company that does this. 399 00:21:05,441 --> 00:21:11,351 But there are companies that will backup your cloud data and then get it out 400 00:21:11,351 --> 00:21:17,111 of the cloud into their cloud, and I think that's as secure as it can be. 401 00:21:17,546 --> 00:21:18,026 Right. 402 00:21:18,296 --> 00:21:24,476 And I think that it should be then stored in some type of immutable type offering. 403 00:21:25,226 --> 00:21:34,856 Um, basically get it, get, get it out of the region for security against bad things 404 00:21:34,856 --> 00:21:36,296 that might happen that aren't hackers. 405 00:21:36,656 --> 00:21:43,106 And then get it out of the account to secure it against hackers and the. 406 00:21:44,426 --> 00:21:48,086 If that costs you money, figure out a way to do that that costs 407 00:21:48,086 --> 00:21:49,886 you as little as possible. 408 00:21:50,336 --> 00:21:55,436 And, and, and I do think the companies that can back up, take the, the 409 00:21:55,436 --> 00:21:59,216 incremental data and then maybe de-dupe it before they pull it out. 410 00:21:59,246 --> 00:22:04,316 If they can do that, you can minimize the egress cost of moving it out. 411 00:22:04,976 --> 00:22:08,276 Um, the, the other thing, um. 412 00:22:09,341 --> 00:22:11,471 Yeah, so, so they didn't have it in a different account. 413 00:22:11,471 --> 00:22:13,271 They didn't have object lock turned on. 414 00:22:13,661 --> 00:22:14,411 Um, 415 00:22:15,051 --> 00:22:16,491 Prasanna Malaiyandi: Object lock did not exist back 416 00:22:16,496 --> 00:22:16,671 then. 417 00:22:16,671 --> 00:22:17,181 By the way. 418 00:22:17,471 --> 00:22:18,071 W. Curtis Preston: what's that? 419 00:22:19,101 --> 00:22:21,051 Prasanna Malaiyandi: Object lock didn't exist back then, so. 420 00:22:21,071 --> 00:22:23,141 W. Curtis Preston: Okay, so we won't, we won't blame them for that. 421 00:22:23,321 --> 00:22:28,511 But, but, but since object lock exists now, we'll say this is what you should do. 422 00:22:29,021 --> 00:22:34,271 Uh, but you know what did exist back then that they did not use multifactor 423 00:22:34,271 --> 00:22:34,991 authentication? 424 00:22:35,886 --> 00:22:39,276 Prasanna Malaiyandi: Oh, for their access into their admin account or into their 425 00:22:39,551 --> 00:22:39,851 W. Curtis Preston: Yeah. 426 00:22:40,301 --> 00:22:40,721 Yeah. 427 00:22:41,021 --> 00:22:46,721 So if somebody gains access to your admin keys and they're able to log in. 428 00:22:46,726 --> 00:22:51,401 If you don't have MFA you, you have zero protection against someone 429 00:22:51,406 --> 00:22:55,751 either stealing or accidentally, you know, inadvertently getting access 430 00:22:55,751 --> 00:23:00,371 to administrative level keys and, uh. 431 00:23:02,141 --> 00:23:04,991 I mean M-F-A-M-F-A-M-F-A mfa, I mean, how many, how many times 432 00:23:04,991 --> 00:23:06,191 do we have to say it right? 433 00:23:06,731 --> 00:23:10,781 Um, good password management, MFA and, uh, patch management. 434 00:23:10,781 --> 00:23:12,011 We, we say this all the time. 435 00:23:12,016 --> 00:23:14,891 If you just did those three things, you'd stop roughly 90% of attacks. 436 00:23:15,101 --> 00:23:18,851 And in this case, if they had had MFA, this, uh, bad actor 437 00:23:18,851 --> 00:23:19,841 would not have been able to 438 00:23:19,846 --> 00:23:20,651 gain access. 439 00:23:21,021 --> 00:23:24,261 Prasanna Malaiyandi: Yeah, and and I think it's important to say 440 00:23:24,391 --> 00:23:28,741 just because you use MFA and patch management and everything else. 441 00:23:29,581 --> 00:23:33,511 Doesn't mean that you don't need backup, you still need backup because 442 00:23:33,781 --> 00:23:37,561 that is how you are gonna recover from this, plus other issues as well. 443 00:23:37,981 --> 00:23:42,211 The security side of things just sort of helps to protect you from 444 00:23:42,216 --> 00:23:44,221 letting the hackers in to some extent. 445 00:23:44,491 --> 00:23:46,831 Not gonna be a hundred percent foolproof, but hopefully, like 446 00:23:46,836 --> 00:23:48,391 Curtis said, protects you. 447 00:23:48,391 --> 00:23:50,281 And 80, 90% of the cases. 448 00:23:50,916 --> 00:23:54,426 W. Curtis Preston: Yeah, to borrow from and totally abuse 449 00:23:54,426 --> 00:23:55,836 a quote from Shakespeare. 450 00:23:56,136 --> 00:24:00,186 There is more on heaven and earth than that is dreamt of. 451 00:24:00,186 --> 00:24:04,986 In your philosophy, there are many, many ways that your 452 00:24:04,986 --> 00:24:07,536 data can be attacked, deleted. 453 00:24:08,166 --> 00:24:10,116 Set on fire exploded. 454 00:24:10,296 --> 00:24:11,796 Sucked into a sinkhole. 455 00:24:11,916 --> 00:24:14,166 There's so many different things that can happen to your data. 456 00:24:14,166 --> 00:24:15,516 That's why you have backup. 457 00:24:15,966 --> 00:24:18,186 And backup protects against all of them. 458 00:24:18,486 --> 00:24:21,816 And, and we're saying backup and Dr and all of those things that come with it. 459 00:24:22,386 --> 00:24:28,716 Um, but uh, the other thing that they also didn't do was this idea 460 00:24:28,776 --> 00:24:32,526 of, um, you know, least privileged. 461 00:24:32,526 --> 00:24:32,676 Do you 462 00:24:32,676 --> 00:24:33,321 want to talk about that? 463 00:24:34,366 --> 00:24:38,116 Prasanna Malaiyandi: Yeah, so normally you do not want, in a company, you don't 464 00:24:38,116 --> 00:24:44,811 want the intern to have the same level of access as your CEO or your IT admin. 465 00:24:44,831 --> 00:24:49,526 And so you wanna be able to say, Hey, whatever access a person needs 466 00:24:49,526 --> 00:24:52,496 to something, that's all they should have access to and nothing else. 467 00:24:53,276 --> 00:24:57,026 And so you wanna have, make sure that you are focused on that and 468 00:24:57,026 --> 00:25:00,326 don't just say, Hey everyone, you guys have admin credentials so you 469 00:25:00,326 --> 00:25:01,496 can do anything and everything. 470 00:25:01,501 --> 00:25:05,066 Because if one person who inadvertently gets compromised, 471 00:25:05,066 --> 00:25:06,806 now everything is exposed. 472 00:25:07,136 --> 00:25:12,266 So you wanna scope down their access to only what they need and that's it. 473 00:25:13,306 --> 00:25:13,796 W. Curtis Preston: Exactly. 474 00:25:14,521 --> 00:25:19,741 Now, when, when reading the, uh, articles about this, one of 475 00:25:19,741 --> 00:25:22,156 the things that other people I. 476 00:25:22,906 --> 00:25:29,416 Uh, dinged this company for, was that they didn't have an established 477 00:25:30,196 --> 00:25:33,226 procedure for locking down the account. 478 00:25:33,616 --> 00:25:37,486 And when I thought about that, I just found myself wondering. 479 00:25:37,486 --> 00:25:37,516 I. 480 00:25:38,521 --> 00:25:38,941 Huh? 481 00:25:38,971 --> 00:25:41,101 How exactly would that happen? 482 00:25:41,701 --> 00:25:45,211 And the best that I could find, you know, how, how would you do that? 483 00:25:45,871 --> 00:25:51,691 And the best that I could find is that you would have a secondary account 484 00:25:52,381 --> 00:26:00,151 that has access to this account that you then have a procedure to do things 485 00:26:00,151 --> 00:26:04,531 like, um, disable, I think about. 486 00:26:06,481 --> 00:26:12,331 You know, what, what is, what is the cloud equivalent to blocking somebody out? 487 00:26:12,331 --> 00:26:15,421 And I, and I think the, the, the quickest way would be to 488 00:26:15,421 --> 00:26:18,361 disable particular I am profiles. 489 00:26:19,171 --> 00:26:23,281 Um, there, there was some, I, you know, and I'm not, I'm not 490 00:26:23,281 --> 00:26:26,281 an expert on this, uh, I don't think you're an expert on this. 491 00:26:26,641 --> 00:26:29,701 I would say talk to your cloud company. 492 00:26:30,091 --> 00:26:31,681 Talk to your cloud provider. 493 00:26:32,251 --> 00:26:40,501 Ask them, Hey, I am worried that one day my cloud entire environment 494 00:26:40,501 --> 00:26:42,631 might become compromised. 495 00:26:43,531 --> 00:26:47,491 How can I automate, basically locking everything out? 496 00:26:47,821 --> 00:26:48,091 Right? 497 00:26:48,091 --> 00:26:52,741 Once we determine that a hacker is in our environment, I would really like 498 00:26:52,741 --> 00:26:58,501 a button that I can press from another account that shuts everything and down. 499 00:26:59,341 --> 00:27:04,981 And everything like this can be automated and yes, that is a, uh, 500 00:27:05,341 --> 00:27:06,721 you know what, what, what's the term? 501 00:27:06,721 --> 00:27:07,951 The, you know, yeah. 502 00:27:07,951 --> 00:27:08,971 The nuclear option. 503 00:27:08,971 --> 00:27:10,351 That is the nuclear option. 504 00:27:10,861 --> 00:27:15,781 But once you have an a hacker in your account to me that that 505 00:27:15,781 --> 00:27:16,921 would be the proper option. 506 00:27:16,926 --> 00:27:18,241 Shut everything down. 507 00:27:18,526 --> 00:27:23,671 Uh, except for like, I would think create a new IAM profile that you can 508 00:27:23,671 --> 00:27:28,441 use after you've done this, and then nuke everything that isn't that, um. 509 00:27:28,861 --> 00:27:34,051 And, and that should be automated and that, and I don't think this 510 00:27:34,051 --> 00:27:35,311 is something we normally talk 511 00:27:35,311 --> 00:27:35,641 about. 512 00:27:37,121 --> 00:27:38,951 Prasanna Malaiyandi: I don't think it's something we talk about, but 513 00:27:38,951 --> 00:27:41,201 it has to be out there somewhere. 514 00:27:41,471 --> 00:27:45,731 I'm sure that AWS or Google or pick your favorite cloud provider, 515 00:27:45,731 --> 00:27:48,191 they probably have a procedure. 516 00:27:48,191 --> 00:27:53,681 I think the danger is you don't want it too automated because there's also the 517 00:27:53,686 --> 00:27:57,761 risk that a hacker or someone else could trigger that and shut down your company. 518 00:27:57,761 --> 00:28:00,881 So it's sort of one of those nuclear options. 519 00:28:00,881 --> 00:28:02,621 So you don't wanna make it too easy. 520 00:28:03,821 --> 00:28:04,631 But I'm sure that 521 00:28:04,666 --> 00:28:04,906 W. Curtis Preston: Yeah. 522 00:28:05,231 --> 00:28:05,441 Prasanna Malaiyandi: a. 523 00:28:06,516 --> 00:28:06,806 Yeah. 524 00:28:07,216 --> 00:28:08,626 W. Curtis Preston: use the nuclear option to. 525 00:28:12,406 --> 00:28:16,456 This is one of those where this, this is the nuclear button, and so you just, 526 00:28:16,786 --> 00:28:19,996 maybe you have an account that just does this and that account is like 527 00:28:20,566 --> 00:28:22,456 completely separate from everything else. 528 00:28:22,456 --> 00:28:24,436 Like we, we talk about having an account. 529 00:28:24,796 --> 00:28:26,416 That is the backup account, right? 530 00:28:26,416 --> 00:28:30,016 That is used for backups and no one ever logs into this account. 531 00:28:30,466 --> 00:28:36,016 And you create it in such a way that if anyone ever does log in, 532 00:28:36,046 --> 00:28:39,046 does log into the account, it sets off alerts everywhere and Right. 533 00:28:39,046 --> 00:28:45,616 It, it can be like a honeypot account, but this account, um, yeah, 534 00:28:45,616 --> 00:28:49,996 I'm not sure how to do, again, I'm not an expert in this, but I would, 535 00:28:50,206 --> 00:28:52,006 I would create a separate account. 536 00:28:52,666 --> 00:28:56,506 I would make that account as secure as humanly possible. 537 00:28:57,196 --> 00:29:02,746 Again, ask your cloud provider how to do that, uh, the, the best way to do that. 538 00:29:03,676 --> 00:29:08,206 But I just noticed that everybody said that almost everybody, they're like, they 539 00:29:08,206 --> 00:29:12,166 should have had procedures for what to do in this situation, and, and they didn't 540 00:29:12,166 --> 00:29:12,706 have them. 541 00:29:13,126 --> 00:29:13,786 I think 542 00:29:14,206 --> 00:29:14,806 perhaps, 543 00:29:14,906 --> 00:29:15,176 Prasanna Malaiyandi: right? 544 00:29:15,886 --> 00:29:16,366 W. Curtis Preston: what's that? 545 00:29:16,961 --> 00:29:18,456 Prasanna Malaiyandi: At least manual procedures, right 546 00:29:18,541 --> 00:29:20,341 W. Curtis Preston: At least manual procedures, right? 547 00:29:20,581 --> 00:29:27,331 Because I don't think that changing the passwords on IM profiles was 548 00:29:27,811 --> 00:29:30,061 the, the quickest way to do that. 549 00:29:30,061 --> 00:29:30,361 Right? 550 00:29:30,361 --> 00:29:35,851 I think the thing they should have focused on perhaps was kicking out, 551 00:29:35,911 --> 00:29:39,631 currently logged in sessions, uh. 552 00:29:42,121 --> 00:29:43,291 Yeah, I don't know. 553 00:29:43,291 --> 00:29:47,851 And the problem is, it's like, how, how do you, you, you have to, 554 00:29:48,061 --> 00:29:52,681 you have to build your incident response around your environment. 555 00:29:53,251 --> 00:29:59,761 And one of the things that they could have done is maintain an inventory of, 556 00:30:00,721 --> 00:30:07,381 um, basically privileged, super privileged accounts and look and see if there 557 00:30:07,381 --> 00:30:08,431 were any new ones. 558 00:30:08,966 --> 00:30:11,696 Prasanna Malaiyandi: Well, that's the thing I was going to mention is they 559 00:30:11,696 --> 00:30:13,016 should at least have had monitoring. 560 00:30:13,016 --> 00:30:17,666 When someone adds a super privileged user, they should have been flagged 561 00:30:17,666 --> 00:30:19,496 about that immediately, right? 562 00:30:19,586 --> 00:30:21,506 Because that's not a common occurrence. 563 00:30:22,066 --> 00:30:22,546 W. Curtis Preston: Right. 564 00:30:23,876 --> 00:30:26,816 Prasanna Malaiyandi: And so monitoring, alerting is also looks like something 565 00:30:26,816 --> 00:30:30,236 that people should be doing to catch these sort of issues as well. 566 00:30:31,096 --> 00:30:34,276 W. Curtis Preston: Yeah, and, and, and I'd love, by the way, I'd love 567 00:30:34,276 --> 00:30:36,166 other suggestions from listeners. 568 00:30:36,316 --> 00:30:37,756 I would love to hear from you. 569 00:30:37,966 --> 00:30:41,056 If you go to backup wrap up.com, there's actually a button on there 570 00:30:41,056 --> 00:30:42,221 that you can leave voicemails. 571 00:30:43,111 --> 00:30:47,281 You can send us notes and, uh, you know, you know, if you'd like, we can actually 572 00:30:47,281 --> 00:30:48,751 even play your response on the air. 573 00:30:48,991 --> 00:30:54,721 I would love to hear better suggestions than we have from a security perspective 574 00:30:54,991 --> 00:30:56,431 because I'm, you know, I'm Mr. 575 00:30:56,431 --> 00:30:56,671 Back. 576 00:30:56,671 --> 00:30:57,391 I'm, I'm not Mr. 577 00:30:57,391 --> 00:30:57,931 Security. 578 00:30:57,931 --> 00:31:02,131 I, I play, I play a security on tv. 579 00:31:02,731 --> 00:31:07,681 Um, the, uh, but, but, but, uh, the summary statement. 580 00:31:08,026 --> 00:31:13,216 From a backup perspective, if they had simply followed the 3, 2, 1 rule, if they 581 00:31:13,216 --> 00:31:19,696 had made another copy of their backups in another account, in another location, 582 00:31:19,936 --> 00:31:24,016 if they had used a cloud provider to do this for them so that then a copy of all 583 00:31:24,016 --> 00:31:29,116 their data was stored in, in a completely different company, if they had done any 584 00:31:29,116 --> 00:31:34,366 of those things, they would've at least had a copy so that once they got on the 585 00:31:34,371 --> 00:31:37,126 other side of the attack, they could have. 586 00:31:37,501 --> 00:31:42,091 Then recovered all the data because that's the, the true disaster here. 587 00:31:42,091 --> 00:31:46,741 As that once they've been attacked and once the attacker gained access 588 00:31:46,741 --> 00:31:49,831 to their account, they were able to delete all their data, both 589 00:31:49,831 --> 00:31:51,691 their primary and their backups 590 00:31:52,296 --> 00:31:55,716 Prasanna Malaiyandi: So I know we've been focused solely on code space as the 591 00:31:55,716 --> 00:32:01,926 company, but I think there's also blame to be placed on those 200 companies who 592 00:32:01,926 --> 00:32:09,186 were using Code Spaces for not also having a backup of their data and relying solely 593 00:32:09,186 --> 00:32:11,976 on code spaces as their service provider. 594 00:32:12,431 --> 00:32:15,041 W. Curtis Preston: You know, that is an interesting, we, you know, we 595 00:32:15,041 --> 00:32:19,421 tend to focus here on the fact that it was a provider, but what this 596 00:32:19,421 --> 00:32:21,281 really was, was a SaaS provider. 597 00:32:21,736 --> 00:32:21,956 Prasanna Malaiyandi: Yep. 598 00:32:22,571 --> 00:32:23,021 W. Curtis Preston: Right. 599 00:32:23,516 --> 00:32:23,736 Prasanna Malaiyandi: Yep. 600 00:32:23,951 --> 00:32:24,341 W. Curtis Preston: Yeah. 601 00:32:24,341 --> 00:32:26,921 So we don't know what happened to those other companies. 602 00:32:27,011 --> 00:32:29,681 And so this is, this is a double lesson, right? 603 00:32:29,681 --> 00:32:32,681 If you're, if you're running in the cloud, make sure you've got 604 00:32:32,681 --> 00:32:34,601 a backup of that, of that data. 605 00:32:34,871 --> 00:32:39,011 If you're using a SaaS provider, make sure you have another copy of 606 00:32:39,011 --> 00:32:41,861 the data that you're putting in that SaaS provider, because that would be 607 00:32:41,861 --> 00:32:46,871 another way for at least the, the, um, the, the, the thing that's difficult. 608 00:32:46,871 --> 00:32:50,531 Here, again, I agree with you, the thing that's. 609 00:32:50,841 --> 00:32:56,151 The, the difference here is that unlike many of the SaaS providers, this company 610 00:32:56,151 --> 00:32:57,921 specifically said, Hey, we got it. 611 00:32:58,371 --> 00:33:03,861 We got your backups, this data, and it's tested and it's all this stuff, right? 612 00:33:04,161 --> 00:33:08,211 Um, the, um, you know, I just had a thought. 613 00:33:08,426 --> 00:33:13,281 Um, if we go to LinkedIn and we search for code spaces. 614 00:33:14,291 --> 00:33:17,261 Find people that used to work in code spaces. 615 00:33:18,461 --> 00:33:20,921 Oh, that would be, I wish, I wish we could talk to somebody 616 00:33:20,921 --> 00:33:22,211 that was involved in this, but 617 00:33:22,841 --> 00:33:23,981 uh, I think we can 618 00:33:24,081 --> 00:33:25,551 Prasanna Malaiyandi: I'm sure there are probably NDAs. 619 00:33:26,561 --> 00:33:27,101 W. Curtis Preston: What's that? 620 00:33:27,621 --> 00:33:28,761 Prasanna Malaiyandi: There's probably NDAs. 621 00:33:29,141 --> 00:33:30,191 W. Curtis Preston: Oh, they're probably NDA. 622 00:33:30,251 --> 00:33:30,581 Yeah. 623 00:33:31,181 --> 00:33:31,631 All right. 624 00:33:31,631 --> 00:33:34,721 Well back up the cloud, I told you so. 625 00:33:35,321 --> 00:33:36,581 Any final thoughts for you, persona? 626 00:33:37,566 --> 00:33:41,286 Prasanna Malaiyandi: No, I totally agree with that, and I like this because I 627 00:33:41,286 --> 00:33:46,296 know we bring up code spaces a lot, so I think that hopefully our listeners now 628 00:33:46,301 --> 00:33:51,486 understand why we talk about it and what they should not be doing, and why we 629 00:33:51,546 --> 00:33:55,956 harp so much on things like the 3, 2, 1 rule on MFA, because you don't want your 630 00:33:55,956 --> 00:34:01,476 company to have to shut its doors because they were unable to recover their data. 631 00:34:01,561 --> 00:34:01,851 W. Curtis Preston: Yeah. 632 00:34:02,246 --> 00:34:04,916 And sadly, this will not be the last company that. 633 00:34:06,851 --> 00:34:11,201 You know, basically ceased to exist, uh, because they didn't 634 00:34:11,201 --> 00:34:12,281 properly back up their data. 635 00:34:13,086 --> 00:34:13,236 Prasanna Malaiyandi: Yeah, 636 00:34:13,871 --> 00:34:14,321 W. Curtis Preston: All right. 637 00:34:14,381 --> 00:34:19,211 Well, uh, thanks for, uh, joining me persona, as always. 638 00:34:20,046 --> 00:34:23,826 Prasanna Malaiyandi: anytime and looking forward to see your analytics on your car. 639 00:34:25,786 --> 00:34:28,436 W. Curtis Preston: I will see what I can do and I will also thank our 640 00:34:28,436 --> 00:34:30,566 listeners, we be nothing without you. 641 00:34:30,596 --> 00:34:31,586 Thanks for listening. 642 00:34:31,916 --> 00:34:34,976 And be sure to subscribe so you don't miss an episode. 643 00:34:35,186 --> 00:34:36,746 That is a wrap.