1 00:00:00,000 --> 00:00:03,630 You found the backup wrap up your go-to podcast for all things 2 00:00:03,630 --> 00:00:06,000 backup recovery and cyber recovery. 3 00:00:06,479 --> 00:00:09,480 This episode, we're doing something actually very different. 4 00:00:09,540 --> 00:00:12,480 We're calling this an emergency episode because what just 5 00:00:12,480 --> 00:00:14,670 happened is that serious. 6 00:00:14,790 --> 00:00:20,576 There's an attack on the PYPI repository targeting light LLM. 7 00:00:21,195 --> 00:00:24,465 A library that's pulled into developer environments three and 8 00:00:24,465 --> 00:00:26,805 a half million times every day. 9 00:00:27,375 --> 00:00:31,305 They took stolen credentials to publish malicious code as the real thing. 10 00:00:31,784 --> 00:00:35,175 This malware is grabbing SSH keys, cloud credentials, 11 00:00:35,175 --> 00:00:39,525 Kubernetes tokens, everything, and encrypting it and sending it home. 12 00:00:40,065 --> 00:00:42,915 We're breaking down exactly what happened, how they pulled it off, 13 00:00:42,915 --> 00:00:44,864 and what you need to do right now. 14 00:00:45,195 --> 00:00:47,055 To, uh, find out if you were hit. 15 00:00:47,535 --> 00:00:50,505 We also cover what to do to protect yourself from something 16 00:00:50,505 --> 00:00:51,735 like this in the future. 17 00:00:52,395 --> 00:00:56,145 By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr. 18 00:00:56,145 --> 00:01:00,525 Backup, and I've been passionate about backup and recovery and now 19 00:01:00,525 --> 00:01:02,925 cyber recovery for over 30 years. 20 00:01:02,925 --> 00:01:03,435 Ever since. 21 00:01:03,435 --> 00:01:05,715 I had to tell my boss that we had no backups. 22 00:01:06,075 --> 00:01:08,385 Of the production database that we had just lost. 23 00:01:08,775 --> 00:01:11,715 I don't want that to happen to you and that's why I do this. 24 00:01:11,895 --> 00:01:16,545 On that podcast, we turn unappreciated admins into Cyber Recovery Heroes. 25 00:01:16,845 --> 00:01:19,005 This is the backup wrap up. 26 00:01:35,738 --> 00:01:37,088 Welcome to the backup wrap up. 27 00:01:37,088 --> 00:01:40,748 I'm your host, w Curtis Preston, and I have with me a, I don't 28 00:01:40,748 --> 00:01:44,318 know, a, a, a cadre of joy. 29 00:01:44,408 --> 00:01:48,488 Uh, first we will start with Dr. Mike Sailor. 30 00:01:48,788 --> 00:01:49,238 How's it going, Mike? 31 00:01:50,805 --> 00:01:51,405 Doing well. 32 00:01:51,645 --> 00:01:52,330 Thanks for having me guys. 33 00:01:53,918 --> 00:01:54,698 We're glad to have you. 34 00:01:54,698 --> 00:01:58,058 I mean, it is gonna, it's a, it's a big, this is an important, I'm, I'm 35 00:01:58,058 --> 00:01:59,798 gonna call it like emergency episode. 36 00:01:59,798 --> 00:02:00,638 We're recording this. 37 00:02:00,638 --> 00:02:05,958 Unlike, normally recording this very before we, uh, publish 38 00:02:06,092 --> 00:02:06,252 District. 39 00:02:06,965 --> 00:02:08,885 course, my trustee. 40 00:02:08,885 --> 00:02:11,045 I, I was almost going, going my trustee 41 00:02:14,315 --> 00:02:14,375 stead. 42 00:02:15,135 --> 00:02:15,175 stead. 43 00:02:15,598 --> 00:02:19,078 You know, I don't think I've ever met a Prasanna or heard of a Prasanna 44 00:02:19,288 --> 00:02:20,933 racehorse, so that would be a first. 45 00:02:21,470 --> 00:02:22,940 That would be a first. 46 00:02:22,940 --> 00:02:27,470 Well, thanks for, thanks for being here early in the morning, um, for both of 47 00:02:27,470 --> 00:02:31,550 us, uh, because this is when we were available because we wanted to cover this. 48 00:02:31,970 --> 00:02:37,790 Uh, and I'm gonna start with a story, uh, a story that happened 49 00:02:37,790 --> 00:02:41,030 when Prasanna was four months old. 50 00:02:41,300 --> 00:02:43,880 It's September, 1982 in Chicago. 51 00:02:44,685 --> 00:02:45,405 got a headache. 52 00:02:45,405 --> 00:02:48,345 You walk over to your bathroom, you reach into what should be 53 00:02:48,345 --> 00:02:49,815 the safest place in your home. 54 00:02:50,085 --> 00:02:53,505 You open a bottle of extra strength Tylenol, take a capsule, 55 00:02:53,775 --> 00:02:55,425 and within hours you're dead. 56 00:02:55,875 --> 00:02:59,745 This happened to seven people, including 12-year-old Mary Kellerman. 57 00:03:00,135 --> 00:03:02,265 They weren't killed by a manufacturing error. 58 00:03:02,480 --> 00:03:05,030 They were killed because someone had tampered with the bottles 59 00:03:05,030 --> 00:03:09,230 on the store shelves lacing them with potassium cyanide. 60 00:03:09,650 --> 00:03:11,030 There was a nationwide panic. 61 00:03:11,030 --> 00:03:15,740 Johnson and Johnson did the, the best, I think a, a gold standard of response. 62 00:03:16,010 --> 00:03:20,384 They recalled 31 million bottles worth over a hundred million dollars. 63 00:03:20,714 --> 00:03:22,754 Uh, I remember this very much. 64 00:03:22,754 --> 00:03:23,924 You, you do too, Mike. 65 00:03:24,164 --> 00:03:24,434 Right? 66 00:03:26,414 --> 00:03:30,164 And this, I, I, I thought of this story because when, uh, when I was putting 67 00:03:30,164 --> 00:03:36,854 together the outline for this Mike, I came in, I came across your, uh, it was 68 00:03:36,854 --> 00:03:40,934 a term that you used in our book that we wrote together, learning ransomware 69 00:03:40,934 --> 00:03:43,394 response and recovery, uh, available. 70 00:03:43,584 --> 00:03:45,294 Uh, for everyone right now. 71 00:03:45,534 --> 00:03:49,404 Uh, and you use the term getting poisoned by your own medicine cabinet 72 00:03:49,404 --> 00:03:51,204 because you go to a trusted source. 73 00:03:51,564 --> 00:03:54,594 Uh, you know, in this case it was an actual medicine cabinet. 74 00:03:54,924 --> 00:03:58,344 In the case of the story that we're gonna talk about today, you use a 75 00:03:58,344 --> 00:04:05,694 library, uh, in this case called, uh, p pronounce it people, P-Y-P-Y-P-L. 76 00:04:06,114 --> 00:04:09,119 Uh, you ingested into your network and next thing you know. 77 00:04:09,504 --> 00:04:11,034 You have a catastrophe. 78 00:04:11,184 --> 00:04:13,374 So, um, what are we talking about, Mike? 79 00:04:16,166 --> 00:04:18,941 I don't know that I've heard anybody actually pronounce it, but I, if. 80 00:04:19,346 --> 00:04:20,906 If they did, I think it would be pipe 81 00:04:22,044 --> 00:04:22,134 Ple, 82 00:04:22,196 --> 00:04:24,506 since it's Python. 83 00:04:24,566 --> 00:04:24,926 Yeah. 84 00:04:25,314 --> 00:04:26,334 Oh, oh, that's right. 85 00:04:26,334 --> 00:04:27,299 It's pi, yeah, yeah. 86 00:04:27,659 --> 00:04:28,179 Ple, yeah. 87 00:04:29,456 --> 00:04:37,496 so, uh, in this case everybody relies, well, millions of, uh, organizations 88 00:04:37,496 --> 00:04:40,181 rely on what they felt was a trusted. 89 00:04:41,646 --> 00:04:47,226 Uh, resource, a trusted medicine, if you will, um, for their daily 90 00:04:47,406 --> 00:04:51,606 updates and, and grabs from a open source, uh, environments. 91 00:04:52,716 --> 00:04:56,676 and on the 24th, uh, they did what they always do and they, 92 00:04:56,676 --> 00:04:58,686 they go to this, uh, this library. 93 00:04:58,686 --> 00:04:59,526 They use this tool. 94 00:04:59,526 --> 00:05:01,716 They download what they thought was legitimate. 95 00:05:02,406 --> 00:05:03,006 Uh. 96 00:05:03,681 --> 00:05:08,421 Uh, data software, uh, or I, I guess a library in this case. 97 00:05:08,421 --> 00:05:13,011 And, uh, it, it, it didn't turn out to be, uh, uh, malware infected. 98 00:05:13,011 --> 00:05:17,811 So, you know, complacency and security are, are not congruent. 99 00:05:18,081 --> 00:05:22,041 Uh, you, you've, your security diminishes the, the more complacent you become. 100 00:05:22,521 --> 00:05:25,731 Uh, and in this case, you know, just going to the same place you always go 101 00:05:25,731 --> 00:05:27,561 to get the same stuff you always get. 102 00:05:28,131 --> 00:05:30,201 Uh, and this time they got a little extra. 103 00:05:32,289 --> 00:05:34,029 What, what did they get, by the way? 104 00:05:34,089 --> 00:05:38,139 I know, I know that it was very, I know it was bad and I, I, I, I read that, that, 105 00:05:38,199 --> 00:05:44,739 that this tool that was infected, uh, down is downloaded 97 million times a month. 106 00:05:44,739 --> 00:05:48,369 But what, What exactly happened to them if, if they put this 107 00:05:48,369 --> 00:05:49,084 tool in their environment? 108 00:05:51,051 --> 00:05:57,831 So what they downloaded was a, an infected version of the light, L-M-L-L-M, uh, 109 00:05:58,221 --> 00:06:00,651 which is a open source Python library. 110 00:06:01,431 --> 00:06:02,781 Um, in, in this case. 111 00:06:03,921 --> 00:06:07,101 There's a legitimate version of that, you know, with the, and we'll get into 112 00:06:07,161 --> 00:06:09,411 hash values and fingerprints of things. 113 00:06:09,471 --> 00:06:13,851 Uh, so, you know that it, it came from a trusted source versus a 114 00:06:13,851 --> 00:06:15,861 modified version of, of that, 115 00:06:17,976 --> 00:06:22,386 Which on the surface looks the same, you know, maybe even the same file size, 116 00:06:22,836 --> 00:06:27,666 uh, looks the same, smells the same, uh, did not taste, did not taste the same. 117 00:06:28,476 --> 00:06:30,636 so when they downloaded it, it came with malware. 118 00:06:30,636 --> 00:06:35,886 The malware was intended to, uh, well, primarily intended to 119 00:06:35,886 --> 00:06:37,806 harvest credentials and secrets. 120 00:06:38,106 --> 00:06:42,336 You know, API, keys and SSH, uh, tokens and, um. 121 00:06:43,791 --> 00:06:46,281 Uh, there were a couple of other things that it did. 122 00:06:46,281 --> 00:06:51,441 It did look for, um, you know, it did try to call home to see if there, if, 123 00:06:51,441 --> 00:06:56,241 if, if having identified a particular target host, if the, the threat actors 124 00:06:56,241 --> 00:06:57,711 wanted it to do something different. 125 00:06:58,371 --> 00:07:01,941 I think there were some, uh, geographic, um. 126 00:07:03,126 --> 00:07:03,876 Implications. 127 00:07:03,876 --> 00:07:08,826 So if, if the malware knew it was in Iran, for example, uh, it did something 128 00:07:08,826 --> 00:07:12,576 different than if it, if it infected something, uh, you know, a machine 129 00:07:12,576 --> 00:07:14,526 or a, an environment in, in Europe. 130 00:07:15,246 --> 00:07:19,511 Um, but yeah, it was, it was designed to, to harvest secrets in credentials. 131 00:07:21,552 --> 00:07:21,772 So. 132 00:07:22,882 --> 00:07:23,842 Thanks for that great summary. 133 00:07:23,842 --> 00:07:28,612 As you were talking, the first thing that popped into my head was in the very 134 00:07:28,612 --> 00:07:32,482 beginning was have you seen the picture or the meme where it's like, Hey, here's all 135 00:07:32,482 --> 00:07:36,382 these cool things built on top, and then at the very bottom it's like this very 136 00:07:36,382 --> 00:07:38,782 small stick holding up everything else. 137 00:07:38,962 --> 00:07:41,722 That's literally what I was thinking about the, as you were talking through 138 00:07:41,722 --> 00:07:45,442 this, how there's all this open source tooling out there, or libraries out 139 00:07:45,442 --> 00:07:48,082 there that people leverage heavily. 140 00:07:48,502 --> 00:07:48,802 Right. 141 00:07:48,802 --> 00:07:51,802 But it's like holding up everything right and. 142 00:07:52,157 --> 00:07:56,147 That's where, like you said, if you sort of attack the common source, then you 143 00:07:56,147 --> 00:08:00,617 now have access to this wide spread of people who are using that common library. 144 00:08:01,307 --> 00:08:04,847 And a lot of times like these open source developers, right, they aren't 145 00:08:04,847 --> 00:08:06,947 paid well, right, they or at all. 146 00:08:06,947 --> 00:08:11,117 And so it's sort of in their good terms and wills that they're doing this. 147 00:08:11,717 --> 00:08:15,287 Curtis, as Mike was walking through this and kind of this sort of attack, 148 00:08:15,287 --> 00:08:18,257 another one that came to mind that we talked about on the podcast before, 149 00:08:18,917 --> 00:08:21,137 do you remember the developer who. 150 00:08:21,622 --> 00:08:27,472 Was wondering why SSH was performing slightly slower and realized 151 00:08:27,472 --> 00:08:34,192 that someone had in, had sort of taken over the open SSH and had 152 00:08:34,942 --> 00:08:38,272 sort of infected it and just luckily he happened to notice because it was 153 00:08:38,272 --> 00:08:40,312 slightly different performance wise. 154 00:08:40,889 --> 00:08:41,339 Right. 155 00:08:41,662 --> 00:08:44,422 But this sort of seems like another one of those like supply chain attacks. 156 00:08:45,404 --> 00:08:45,734 Yeah. 157 00:08:45,734 --> 00:08:49,154 And, and again, I'm, I'm, you know, the, the summary I'm reading 158 00:08:49,154 --> 00:08:51,584 here, uh, there were two versions. 159 00:08:51,584 --> 00:08:55,094 It was like the initial version that sort of pushed out the second version 160 00:08:55,304 --> 00:08:57,944 and the second version, which is 1.8. 161 00:08:58,034 --> 00:09:01,424 2.8 is even worse than the first version. 162 00:09:01,754 --> 00:09:03,674 Uh, but yeah, the, it's that. 163 00:09:04,649 --> 00:09:09,359 You're, you're, you're, you're, you're, you're subject to this 164 00:09:09,359 --> 00:09:13,199 because of, you know, for those of you that aren't developers, the problem 165 00:09:13,199 --> 00:09:16,469 here is what we call dependencies Prasanna . Do you want to talk to that? 166 00:09:17,527 --> 00:09:22,117 So if every single person out there had to write every single line of 167 00:09:22,117 --> 00:09:26,162 code from scratch, right, there'd be no software really out there, right? 168 00:09:26,302 --> 00:09:29,642 And so what a lot of people do is they'll say, Hey, this, uh. 169 00:09:30,247 --> 00:09:34,057 Package, or this library does a whole bunch of things I need to do, 170 00:09:34,057 --> 00:09:37,237 instead of me writing it myself, let me leverage that library. 171 00:09:37,567 --> 00:09:41,047 And so you sort of import it into your code base, and that's what sort of 172 00:09:41,047 --> 00:09:44,857 gets built around, but you don't have access necessarily to everything in it. 173 00:09:44,857 --> 00:09:48,517 You're just sort of treating it like a black box almost and being like, 174 00:09:48,517 --> 00:09:50,137 yep, I trust whatever's in there. 175 00:09:50,137 --> 00:09:52,132 They've done their things and this is the functionality I get. 176 00:09:53,854 --> 00:09:54,074 Yeah. 177 00:09:54,074 --> 00:09:55,274 Any comments on that, Mike? 178 00:09:57,221 --> 00:09:58,761 No, I completely agree and, and. 179 00:09:59,991 --> 00:10:03,141 One of the things that, that these bad guys did, either intentionally 180 00:10:03,141 --> 00:10:06,441 or incidentally, is when they, when they pushed out the first version, 181 00:10:06,801 --> 00:10:10,461 that was kind of a test run to see how likely people were gonna 182 00:10:10,461 --> 00:10:12,801 download that illegitimate version. 183 00:10:12,801 --> 00:10:16,761 In other words, they didn't check, to see if it was legitimate, you 184 00:10:16,761 --> 00:10:18,321 know, hash values and other things. 185 00:10:18,681 --> 00:10:20,676 Uh, and then how much time would they have? 186 00:10:22,046 --> 00:10:23,396 to push a second update. 187 00:10:23,521 --> 00:10:25,436 And, and in this case it went pretty quick. 188 00:10:25,856 --> 00:10:29,456 Uh, but if you, if you don't identify the first one, uh, and it didn't get 189 00:10:29,456 --> 00:10:34,856 shut down, then pushing out a second one gives you, direct access, uh, 190 00:10:34,886 --> 00:10:39,026 through updates potentially, uh, to those that downloaded the first version. 191 00:10:39,326 --> 00:10:42,836 We saw this a lot in the, in the app stores on mobile phones. 192 00:10:43,216 --> 00:10:47,476 You know, bad guys would publish a, a seemingly legitimate and either 193 00:10:47,476 --> 00:10:51,376 entertaining or useful app, uh, to get people to download it and use it. 194 00:10:51,376 --> 00:10:54,586 And then the malware and the other nefarious things would come in 195 00:10:54,586 --> 00:10:55,996 the, in the update to the app. 196 00:10:56,959 --> 00:10:58,409 It's like a, it's like a tracer round. 197 00:10:59,956 --> 00:11:00,286 Yeah. 198 00:11:00,569 --> 00:11:00,959 Right. 199 00:11:01,709 --> 00:11:04,499 Uh, for those of you that don't know what a trace round is, it's a, it's 200 00:11:04,499 --> 00:11:08,999 a separate round that, that's not, it's not, doesn't have a payload. 201 00:11:09,179 --> 00:11:10,589 We're talking like weapons here. 202 00:11:10,919 --> 00:11:11,279 Right. 203 00:11:11,309 --> 00:11:15,179 Um, and it's a separate round that just you, you can see it, you can see 204 00:11:15,179 --> 00:11:19,469 what it does, uh, and you can use that for your, you know, the actual round. 205 00:11:19,859 --> 00:11:24,344 Uh, and in this case, the actual round was very, very, um, um. 206 00:11:25,244 --> 00:11:26,054 Dangerous. 207 00:11:26,234 --> 00:11:26,969 Um, so 208 00:11:27,522 --> 00:11:28,142 oh, one thing about the. 209 00:11:28,309 --> 00:11:28,689 Go ahead. 210 00:11:28,749 --> 00:11:29,129 Go ahead. 211 00:11:29,242 --> 00:11:32,152 One thing about the dependencies that my kid touched upon too, and I 212 00:11:32,152 --> 00:11:38,092 think is important is as someone who is leveraging this library, right in 213 00:11:38,092 --> 00:11:42,652 your code base, you have control about sort of how often you want to update. 214 00:11:42,682 --> 00:11:46,402 So say you are only developing once a month, right? 215 00:11:46,402 --> 00:11:50,092 You may not want to pull, constantly be pulling a new version of that 216 00:11:50,092 --> 00:11:52,582 library into your code base, right? 217 00:11:52,582 --> 00:11:53,812 Or if you're doing it once a year. 218 00:11:54,135 --> 00:11:58,455 you have the ability to control how often, and so like Mike was saying, 219 00:11:58,455 --> 00:12:02,265 right, the fact that they pushed out a version sort of implies that hey, 220 00:12:02,265 --> 00:12:06,195 people are sort of constantly updating to get the latest and greatest, because 221 00:12:06,195 --> 00:12:07,725 sometimes that's what people want, right? 222 00:12:07,995 --> 00:12:09,525 New functionality's always coming out. 223 00:12:09,525 --> 00:12:10,875 You want the latest and greatest. 224 00:12:12,467 --> 00:12:18,087 Mike, uh, if you could jump into how someone would know that, um. 225 00:12:18,802 --> 00:12:23,962 That they, that they had this, uh, specifically, or maybe in general, 226 00:12:23,962 --> 00:12:27,592 how would you know that you have some sort of malware in your environment? 227 00:12:27,992 --> 00:12:30,662 Do you wanna walk people through what we're talking about there? 228 00:12:30,662 --> 00:12:30,692 I. 229 00:12:32,194 --> 00:12:32,284 Sure. 230 00:12:32,284 --> 00:12:32,734 So. 231 00:12:33,159 --> 00:12:39,339 If, if you are, if you're concerned that, that, you know, you might be, uh, 232 00:12:39,339 --> 00:12:41,469 affected by this supply chain attack. 233 00:12:41,469 --> 00:12:44,753 In other words, you're, you're, uh, you often download 234 00:12:44,753 --> 00:12:46,403 these pipe Python libraries. 235 00:12:46,403 --> 00:12:54,143 Or actually, know, th this attack group, uh, team, um, it team CPC? 236 00:12:56,303 --> 00:12:56,933 Um, 237 00:12:56,979 --> 00:12:58,029 PCPI thought. 238 00:12:58,581 --> 00:12:59,181 PCP. 239 00:12:59,319 --> 00:12:59,679 Yeah. 240 00:12:59,858 --> 00:13:00,578 I was close. 241 00:13:00,998 --> 00:13:06,218 Uh, thi this attack group, uh, team, uh, PCP is, is really 242 00:13:06,218 --> 00:13:11,378 hitting every open source, uh, project, uh, repository out there. 243 00:13:11,378 --> 00:13:14,738 They, you know, they started with, uh, you know, the trivia 244 00:13:14,738 --> 00:13:16,298 scanner, check marks, GitHub. 245 00:13:16,388 --> 00:13:19,628 Uh, so, you know, the Python libraries were a, an a good 246 00:13:19,628 --> 00:13:21,488 next, uh, pivot for them. 247 00:13:21,908 --> 00:13:27,668 So if, if your organization often, uh, interacts, downloads, uploads, even, um. 248 00:13:28,103 --> 00:13:33,533 And definitely any that have, authenticated access to these 249 00:13:33,533 --> 00:13:37,193 repositories, uh, you might be concerned that you've, you've been 250 00:13:37,253 --> 00:13:38,903 affected by this, this attack. 251 00:13:39,293 --> 00:13:40,883 So common things to look for. 252 00:13:40,883 --> 00:13:44,273 And one of the reasons that this attack was identified so quickly is that there 253 00:13:44,273 --> 00:13:48,233 was a developer that went, Hey, why is my machine, uh, running so slow? 254 00:13:48,233 --> 00:13:49,223 We're doing this weird thing. 255 00:13:49,223 --> 00:13:52,943 And he dug into the services and the, the processes running. 256 00:13:53,243 --> 00:13:56,393 And that's when, um, that developer identified. 257 00:13:57,308 --> 00:13:59,408 Something nefarious is going on. 258 00:14:00,218 --> 00:14:04,148 and that's usually the, you know, the, the symptoms much like getting 259 00:14:04,148 --> 00:14:07,748 sick, the symptoms, you know, tell us that there's something off. 260 00:14:09,308 --> 00:14:14,288 so anything obvious, uh, would be a good indication that something's weird. 261 00:14:14,318 --> 00:14:15,818 You know, don't get paranoid right away. 262 00:14:15,818 --> 00:14:19,508 But definitely look into, uh, anything out of the ordinary 263 00:14:19,508 --> 00:14:21,248 running on, on your systems. 264 00:14:22,598 --> 00:14:23,768 the easiest thing to do. 265 00:14:23,846 --> 00:14:26,126 'cause you're paranoid doesn't mean nobody's out to get you. 266 00:14:26,186 --> 00:14:28,406 Mike, just, but anyway, but go ahead. 267 00:14:29,333 --> 00:14:30,743 Just because you're not paranoid. 268 00:14:31,133 --> 00:14:31,913 Uh, I don't, yeah. 269 00:14:32,156 --> 00:14:32,636 yeah, yeah. 270 00:14:32,636 --> 00:14:32,936 Okay. 271 00:14:33,146 --> 00:14:33,476 Go ahead. 272 00:14:34,328 --> 00:14:35,303 It, it goes either way. 273 00:14:35,333 --> 00:14:35,693 But yeah. 274 00:14:35,693 --> 00:14:40,763 So the easiest thing to do is go and look at, uh, in this case, if you downloaded 275 00:14:40,763 --> 00:14:48,443 the, the light LLM uh, uh, version, uh, go look at the hash value for that. 276 00:14:48,443 --> 00:14:49,793 Compare it to the. 277 00:14:50,933 --> 00:14:54,713 The light, LLLM, uh, trusted hash value. 278 00:14:54,893 --> 00:14:56,093 If they're the same, you're good. 279 00:14:56,723 --> 00:15:01,523 Uh, if they're not the same, then you know, either assume you've been 280 00:15:01,523 --> 00:15:03,803 compromised and just change everything. 281 00:15:03,863 --> 00:15:05,153 I mean, that's the least effort. 282 00:15:05,753 --> 00:15:08,393 Um, you know, fastest, fastest path to, um. 283 00:15:11,123 --> 00:15:11,993 To confidence. 284 00:15:12,773 --> 00:15:15,743 but if you wanna take the next, you know, few steps is, you know, 285 00:15:15,743 --> 00:15:17,003 start looking through your logs. 286 00:15:17,453 --> 00:15:21,023 Uh, if, if you're working in a cloud environment like AWS or 287 00:15:21,023 --> 00:15:23,333 Azure, uh, there are tools there. 288 00:15:23,633 --> 00:15:32,243 Um, so the AWS, uh, cloud uh, and log analysis tools, uh, 289 00:15:32,453 --> 00:15:33,893 one, they have to be turned on. 290 00:15:34,313 --> 00:15:36,233 Uh, and that's just good security, best practice. 291 00:15:36,233 --> 00:15:39,173 Anyway, turn your logging on, log as much as you can. 292 00:15:39,593 --> 00:15:43,853 Uh, and keep it as long as you, uh, as long as you're able to, because those logs 293 00:15:43,853 --> 00:15:47,693 help tell the story when we need to figure out if something weird has happened. 294 00:15:48,563 --> 00:15:50,603 go look at your AWS and Azure logs. 295 00:15:51,053 --> 00:15:57,353 Um, there are some specific IOCs like calls to, uh, third party libraries, 296 00:15:57,383 --> 00:16:02,633 uh, um, ex uh, data xFi, uh, volumes. 297 00:16:03,233 --> 00:16:07,073 So, you know, if, if your environment doesn't send data. 298 00:16:07,428 --> 00:16:11,958 To other places, you know, test dev environment is, uh, encapsulated and, and 299 00:16:11,958 --> 00:16:15,918 you're just working with it in that, and, and all of a sudden something has changed. 300 00:16:15,948 --> 00:16:18,078 You know, that deviation from normal behavior. 301 00:16:18,168 --> 00:16:20,718 Uh, those are other symptoms and, and things to consider. 302 00:16:22,608 --> 00:16:27,318 realize too that sometimes malware doesn't behave. 303 00:16:28,748 --> 00:16:30,878 uh, it sometimes lies dormant. 304 00:16:31,958 --> 00:16:37,088 and a lot of malware, and I think in this case too, uh, install some back doors. 305 00:16:37,088 --> 00:16:42,578 So if, you know, you simply delete the package downloaded, if you simply, you 306 00:16:42,578 --> 00:16:47,438 know, change credentials, uh, sometimes that back door can, uh, can open up 307 00:16:47,618 --> 00:16:51,488 and, and regardless of all those other things you did, uh, bad guys could 308 00:16:51,488 --> 00:16:56,018 still have access to your data and, and, uh, and create some havoc that way. 309 00:16:57,398 --> 00:16:58,028 Um, 310 00:16:58,928 --> 00:16:59,798 talked about network, 311 00:17:00,219 --> 00:17:00,969 Oh, oh. 312 00:17:00,969 --> 00:17:01,629 Go ahead, finish. 313 00:17:02,798 --> 00:17:04,478 I talked about network indicators. 314 00:17:04,478 --> 00:17:09,218 So, uh, unusual traffic to unusual destinations, IP addresses, VPN 315 00:17:09,218 --> 00:17:11,948 tunnels, uh, tour exit points. 316 00:17:12,398 --> 00:17:16,898 Um, you know, if, if you're a, a development shop in the, in Texas as 317 00:17:16,898 --> 00:17:22,658 an example and, uh, you know, you're, you're fairly isolated or, or, um. 318 00:17:24,083 --> 00:17:27,623 Domestic and you, and you're seeing all this international traffic, 319 00:17:27,743 --> 00:17:31,583 especially around the dates, uh, you know, the last couple of days. 320 00:17:31,973 --> 00:17:34,493 Uh, those are also things to be considerate of. 321 00:17:34,973 --> 00:17:41,753 Um, if you, if you're a Kubernetes shop, uh, there are some specific IOCs, uh, 322 00:17:41,753 --> 00:17:44,213 in this, in this attack for Kubernetes. 323 00:17:44,903 --> 00:17:45,623 Um, 324 00:17:45,661 --> 00:17:48,871 You wanna define IOC, although people should know, but what is an 325 00:17:48,893 --> 00:17:49,403 so. 326 00:17:50,303 --> 00:17:53,333 IOCs are indicators of compromise. 327 00:17:53,333 --> 00:17:57,263 So when we talk about attacks, there's, there's two primary acronyms we use 328 00:17:57,293 --> 00:17:59,513 IOCs indicator indicators of Compromise. 329 00:17:59,513 --> 00:18:03,203 So those are the things that we would go look for in our environment that 330 00:18:03,203 --> 00:18:05,663 are specifically or generically. 331 00:18:06,788 --> 00:18:08,048 Related to an attack. 332 00:18:08,438 --> 00:18:12,788 So known bad guy does things a certain way with certain tools. 333 00:18:13,088 --> 00:18:18,278 Those tools leave fingerprints, uh, in either logs or active behavior. 334 00:18:18,278 --> 00:18:19,418 And those are IOCs. 335 00:18:20,258 --> 00:18:23,678 so hash values, uh, certain file names, uh, 336 00:18:25,688 --> 00:18:28,928 I IP addresses, uh, URLs, things like that. 337 00:18:29,378 --> 00:18:34,688 And then we have TTPs, which are, uh, tactics, uh, techniques, and. 338 00:18:35,513 --> 00:18:36,143 Procedures. 339 00:18:36,143 --> 00:18:37,733 So what are the bad guys? 340 00:18:37,913 --> 00:18:40,973 How, what's their mo like, how do they, how do they conduct the attack? 341 00:18:40,973 --> 00:18:43,103 What they do this first and this second? 342 00:18:43,103 --> 00:18:44,303 And they use these things. 343 00:18:44,783 --> 00:18:46,163 Uh, so those are TTPs. 344 00:18:47,303 --> 00:18:53,363 Um, so reviewing those, uh, IOCs and, and they're, they've already been published. 345 00:18:53,363 --> 00:18:54,473 If you just Google. 346 00:18:55,313 --> 00:18:59,333 You know, the, the light LLM Supply Chain attack, IOCs and 347 00:18:59,333 --> 00:19:00,983 TTPs, you'll, you'll get those. 348 00:19:01,253 --> 00:19:05,183 But we're also happy to put a, a short one pager together and put, push that 349 00:19:05,183 --> 00:19:06,953 out to our stop ransomware website. 350 00:19:07,733 --> 00:19:15,023 Um, so hash value comparison of the, the known, trusted version 351 00:19:15,503 --> 00:19:19,283 or versions, um, of the light LLM. 352 00:19:20,063 --> 00:19:21,983 Um, review your logs. 353 00:19:22,643 --> 00:19:28,973 For any known deviations from expected behavior, review your network 354 00:19:28,973 --> 00:19:33,623 traffic, if you have that capability, looking for behavioral anomalies. 355 00:19:34,193 --> 00:19:39,083 Um, and if at all you suspect that you've been compromised. 356 00:19:39,413 --> 00:19:44,513 The best practice, I know it's a, it's a, it's a, it's a pain, but the best 357 00:19:44,513 --> 00:19:47,723 practice is restore from a trusted backup. 358 00:19:48,743 --> 00:19:51,803 Don't try to just onesie, twosie, you know, delete it from the 359 00:19:51,803 --> 00:19:57,173 registry, delete the image, delete the the app, turn off a service. 360 00:19:57,473 --> 00:20:01,838 Those are all I. One-offs, you know, kind of stomping out the fire 361 00:20:01,838 --> 00:20:03,638 with your, your leather moccasins. 362 00:20:03,758 --> 00:20:06,488 And I only say that 'cause it, it actually happened to me and 363 00:20:06,488 --> 00:20:07,538 that's a whole other story. 364 00:20:07,958 --> 00:20:10,358 Uh, but the fire will flame back up. 365 00:20:10,658 --> 00:20:13,958 Uh, if, if you don't take the right approach to, to truly, 366 00:20:14,078 --> 00:20:16,028 uh, starting from scratch. 367 00:20:16,268 --> 00:20:17,858 That's one of the only ways to get, 368 00:20:19,711 --> 00:20:20,011 You, 369 00:20:20,498 --> 00:20:21,068 what are you saying? 370 00:20:21,691 --> 00:20:24,091 you put out a fire wearing moccasins or you 371 00:20:24,293 --> 00:20:25,073 I did not. 372 00:20:25,193 --> 00:20:25,793 I did not. 373 00:20:25,793 --> 00:20:31,463 My former, my, my former mother-in-law started a fire on my porch by putting 374 00:20:31,463 --> 00:20:35,603 a cigarette out in a plastic, uh, pot. 375 00:20:36,143 --> 00:20:39,773 The wind kicked it up, caught the pot on fire, melted it to the house. 376 00:20:39,833 --> 00:20:41,273 She came out, saw it was on fire. 377 00:20:41,273 --> 00:20:46,163 She kicked it off the porch, out into the dead grass, caught the grass on fire. 378 00:20:47,123 --> 00:20:48,578 Uh, all the while. 379 00:20:49,313 --> 00:20:52,823 Passing fire extinguishers and a water hose and all these things. 380 00:20:52,823 --> 00:20:56,213 And she decides the best course of action is to jump out into the grass 381 00:20:56,213 --> 00:21:00,593 and try to stomp it out with her fake leather moccasins, which did not work. 382 00:21:01,013 --> 00:21:04,823 Uh, and she made several trips, uh, from the grass out into the 383 00:21:05,093 --> 00:21:08,453 back, into the kitchen with a a, a simple watering pae again. 384 00:21:08,918 --> 00:21:13,508 Bypassing fire extinguishers and watering hoses, uh, to try and put this water, 385 00:21:13,508 --> 00:21:15,578 this, this fire out with a watering can. 386 00:21:15,968 --> 00:21:17,468 Um, and she failed. 387 00:21:17,828 --> 00:21:19,928 The, the fire simply burned itself out. 388 00:21:20,408 --> 00:21:25,478 Um, but it did ruin her moccasins, uh, her, um, her Mickey Mouse t-shirt. 389 00:21:25,508 --> 00:21:25,958 Oo. 390 00:21:26,438 --> 00:21:28,178 Um, and a lot of my yard. 391 00:21:28,929 --> 00:21:29,149 No, 392 00:21:32,116 --> 00:21:32,896 Um, go. 393 00:21:33,129 --> 00:21:35,589 As you're talking through that, right? 394 00:21:35,769 --> 00:21:39,879 It's sort of helpful to know, okay, here's all the things you should be looking for 395 00:21:39,879 --> 00:21:41,709 to understand, okay, were you compromised? 396 00:21:42,639 --> 00:21:46,659 But given a lot of companies these days, right, you kind of have developers 397 00:21:46,659 --> 00:21:50,859 who go off and do things, is there a mechanism to actually figure out like, 398 00:21:51,339 --> 00:21:54,314 am I actually impacted by this breach? 399 00:21:54,909 --> 00:21:55,239 Right? 400 00:21:55,239 --> 00:21:59,274 Like, do I even, am I even using, uh, the light LLM. 401 00:22:00,939 --> 00:22:05,829 Library in my environment or across, say, all of these developers or whatever it is. 402 00:22:06,843 --> 00:22:07,043 Sure. 403 00:22:07,703 --> 00:22:13,313 Uh, so if, if you're a security team and, and the development team is 404 00:22:13,313 --> 00:22:16,523 not a, you know, they're not open and, or, or, or you're, you're, 405 00:22:16,583 --> 00:22:17,603 you're concerned about their. 406 00:22:18,908 --> 00:22:19,868 Transparency. 407 00:22:20,168 --> 00:22:25,028 Uh, you can absolutely, again, go review all of your logs, review the systems, 408 00:22:25,058 --> 00:22:30,608 uh, depending on your, your security capabilities, uh, whether it's endpoint 409 00:22:30,668 --> 00:22:36,548 like a, a good, uh, EDR in malware solution like huntress, uh, or uh. 410 00:22:37,673 --> 00:22:42,233 A complete environmental, you know, technology environment monitoring solution 411 00:22:42,233 --> 00:22:47,873 like we provide with stellar cyber, uh, where we can collect data points from 412 00:22:47,933 --> 00:22:51,863 tons of data sources and correlate that in one dashboard and go, you know, and 413 00:22:51,863 --> 00:22:56,033 we can input these IOCs and it'll look through all of that data and go, yes, 414 00:22:56,033 --> 00:23:01,343 no, if yes, it maps, you know, what endpoint, what firewall did it go through? 415 00:23:01,698 --> 00:23:02,778 When did it happen? 416 00:23:02,808 --> 00:23:04,188 What was the user involved? 417 00:23:04,188 --> 00:23:05,148 All these things. 418 00:23:05,478 --> 00:23:09,078 And then if it did trigger something, even if it, even if the system at the 419 00:23:09,078 --> 00:23:13,908 time thought it was, uh, you know, uh, uh, you know, safe, I can, I can 420 00:23:13,908 --> 00:23:19,158 tell it, I can tell the system, uh, if light element LLM was downloaded, 421 00:23:19,248 --> 00:23:22,578 uh, you know, that library and, and it did these things, map that out for me. 422 00:23:23,028 --> 00:23:24,408 Uh, and so even if it was. 423 00:23:25,163 --> 00:23:26,243 Perceived safe. 424 00:23:26,633 --> 00:23:29,633 I can now go, I can go through my, my dashboard and look at 425 00:23:29,633 --> 00:23:33,503 all of the potentially impacted systems and, uh, networks and 426 00:23:33,503 --> 00:23:35,273 users involved and all that stuff. 427 00:23:35,693 --> 00:23:40,613 Um, absent all of those things, start the firewall. 428 00:23:41,423 --> 00:23:44,123 well start with talking to your developers, uh, and 429 00:23:44,123 --> 00:23:45,083 then start at the firewall. 430 00:23:45,443 --> 00:23:48,893 'cause all your, all your traffic needs to go out through the firewall. 431 00:23:49,373 --> 00:23:49,943 Um. 432 00:23:50,723 --> 00:23:54,623 Hopefully you've restricted, you know, developer bringing his own hotspot to 433 00:23:54,623 --> 00:23:58,673 download stuff so that it doesn't get blocked by your security policies. 434 00:23:58,673 --> 00:24:03,623 But, you know, there's, there's so many different depend, it depends situations, 435 00:24:03,623 --> 00:24:07,343 but start your firewall if you're, if you feel like you're, you can't get a straight 436 00:24:07,343 --> 00:24:10,523 answer out of the, the people that might have been involved in downloading this. 437 00:24:11,746 --> 00:24:16,606 Uh, another thing that I saw in suggestions was immediately revoke 438 00:24:16,606 --> 00:24:20,086 and rotate every secret that was stored as an environment variable. 439 00:24:20,296 --> 00:24:21,916 Um, you wanna talk about that? 440 00:24:23,393 --> 00:24:26,933 So this was, this was actually a pretty, uh, uh, a, a potentially 441 00:24:26,933 --> 00:24:30,473 pretty, uh, impactful incident. 442 00:24:30,593 --> 00:24:36,233 Uh, so when, when, when the malware came down, uh, it, it 443 00:24:36,293 --> 00:24:38,693 immediately started stealing secrets. 444 00:24:38,873 --> 00:24:44,333 Uh, and so that could be a secret, within an application, uh, you know, trusted. 445 00:24:44,423 --> 00:24:48,923 Uh, uh, trust between applications, trust between servers, trust 446 00:24:48,923 --> 00:24:57,413 between network segments, keys, SSH, Kubernetes, uh, golden ticket theft. 447 00:24:57,413 --> 00:25:01,763 I mean, there's, there's so many things that this malware could have done because 448 00:25:01,763 --> 00:25:03,533 it doesn't know what it has access to. 449 00:25:03,533 --> 00:25:04,943 It was just gonna try and steal everything. 450 00:25:05,663 --> 00:25:07,733 Um, and so absolutely every machine. 451 00:25:08,828 --> 00:25:10,688 That was potentially compromised. 452 00:25:10,688 --> 00:25:14,378 You need to do an inventory of everything it had access to and everything it, 453 00:25:14,498 --> 00:25:16,208 all those secrets it could have stored. 454 00:25:16,538 --> 00:25:17,228 Absolutely. 455 00:25:17,228 --> 00:25:18,578 All those things should be changed. 456 00:25:19,238 --> 00:25:21,278 Revoke it, change it. 457 00:25:22,043 --> 00:25:27,593 And I say revoke it because open sessions are not impacted by changing secrets. 458 00:25:27,983 --> 00:25:31,913 So you've gotta revoke those open, se those open sessions first, 459 00:25:32,153 --> 00:25:37,253 and then change, uh, change all your credentials and reissue 460 00:25:37,373 --> 00:25:38,753 tickets and all that good stuff. 461 00:25:40,419 --> 00:25:41,379 It's a lot of work. 462 00:25:41,619 --> 00:25:43,719 It, or it sounds like a lot of work, you know. 463 00:25:44,993 --> 00:25:46,353 It is, it is a lot of work. 464 00:25:46,501 --> 00:25:47,611 is a huge breach. 465 00:25:47,611 --> 00:25:51,436 It's why, it's why I wanted to jump in on this and we're, I'm actually gonna. 466 00:25:52,186 --> 00:25:53,386 Publish this early. 467 00:25:53,446 --> 00:25:56,656 Uh, normally we wait till Monday to publish our episodes. 468 00:25:56,656 --> 00:26:00,316 I'm gonna publish this one early because this is, this is huge. 469 00:26:00,316 --> 00:26:04,366 I mean, when I heard that it was 97 million monthly downloads, and 470 00:26:04,366 --> 00:26:07,846 then I heard just how bad the, you know, their stealing secrets. 471 00:26:08,176 --> 00:26:12,046 Um, you know, I mean, I, I do go back to that Tylenol scare, right? 472 00:26:12,046 --> 00:26:14,596 Tylenol was such a trusted source. 473 00:26:14,686 --> 00:26:19,486 Um, and, uh, then, you know, it was literally killing people. 474 00:26:19,786 --> 00:26:21,236 Uh, and um. 475 00:26:22,846 --> 00:26:25,546 So it was, you know, everybody immediately went and got the Tylenol 476 00:26:25,546 --> 00:26:26,746 and ripped it outta their shelves. 477 00:26:26,746 --> 00:26:27,076 Right. 478 00:26:27,436 --> 00:26:32,896 Um, and, and again, the, vendor in this case did the right thing, right? 479 00:26:32,896 --> 00:26:35,866 They, they didn't hem and haw, they just said, give us back a hundred million 480 00:26:35,866 --> 00:26:39,436 dollars, um, um, you know, of, of Tylenol. 481 00:26:39,856 --> 00:26:42,526 Mike, we're gonna talk about like, action items for the future. 482 00:26:43,156 --> 00:26:46,936 Anything else that specifically regarding this attack for the moment? 483 00:26:48,413 --> 00:26:51,593 Well add, I'll add, uh, with, with regard to this attack, you know, it 484 00:26:51,593 --> 00:26:56,153 was, it was found or identified fairly quickly just, you know, within a few 485 00:26:56,153 --> 00:27:01,223 hours, uh, that that developer, you know, was, was concerned about his, 486 00:27:01,253 --> 00:27:06,173 his system not running, uh, And that's when he figured this, these things 487 00:27:06,173 --> 00:27:09,263 out, and it got communicated and they, they took, they took that, um. 488 00:27:10,493 --> 00:27:11,933 That infected version down. 489 00:27:12,893 --> 00:27:17,003 So what I'd be interested to hear is what, what is the, the true impact of this? 490 00:27:17,003 --> 00:27:20,153 You know, if it was only available for a couple hours, um. 491 00:27:21,248 --> 00:27:23,678 know, how many organizations were impacted. 492 00:27:23,768 --> 00:27:29,408 Uh, were, are there any follow on, uh, attacks based on this level 493 00:27:29,408 --> 00:27:31,058 of effort it's gonna take to fix? 494 00:27:31,058 --> 00:27:35,708 So if I was an infected or an impacted organization, much like Prasanna 495 00:27:35,738 --> 00:27:38,678 mentioned, all those things, like I talked about, is things you need to do. 496 00:27:38,678 --> 00:27:41,543 That's a lot of work, but that's not something you're gonna be able to achieve. 497 00:27:42,113 --> 00:27:45,653 In, in an hour or a couple hours, maybe not even a couple of days, 498 00:27:45,653 --> 00:27:48,593 because changing some of that stuff may impact operations, right? 499 00:27:48,773 --> 00:27:52,883 So things stop working when you change trusts and credentials, especially, 500 00:27:52,883 --> 00:27:54,743 uh, from a, an operations perspective. 501 00:27:54,743 --> 00:27:57,773 So, I'll be interested to see. 502 00:27:58,433 --> 00:28:00,683 Uh, we're here, uh, in the coming months. 503 00:28:00,803 --> 00:28:04,433 What the fallout from this, even if it was only a couple of hours. 504 00:28:04,583 --> 00:28:10,403 And if you take the 95 plus million downloads a month and you divide 505 00:28:10,403 --> 00:28:14,753 that into the hours, so I think there's 720 hours a month on average. 506 00:28:14,841 --> 00:28:15,711 I just did that. 507 00:28:15,951 --> 00:28:22,131 I, I did great Mind signal light, that's 134,722 downloads per hour. 508 00:28:23,843 --> 00:28:24,203 So. 509 00:28:24,998 --> 00:28:26,018 could be significant. 510 00:28:26,888 --> 00:28:31,238 Um, and usually, you know, it's not multiple de developers in an organization. 511 00:28:31,238 --> 00:28:34,568 It's usually like one person is in charge for updating libraries 512 00:28:34,568 --> 00:28:35,978 and do doing stuff like that. 513 00:28:36,788 --> 00:28:40,448 so that could be 130,000 environments. 514 00:28:41,198 --> 00:28:42,848 Um, so that could be huge. 515 00:28:43,328 --> 00:28:45,368 And again, the level of effort for. 516 00:28:45,526 --> 00:28:46,696 to this episode. 517 00:28:48,008 --> 00:28:51,638 I hope so, and I hope they take it seriously because once we steal those 518 00:28:51,638 --> 00:28:55,508 credentials, a lot of times those credentials now on a, on a production 519 00:28:55,508 --> 00:28:57,728 side, like with service accounts and things like that, those are 520 00:28:57,728 --> 00:28:59,708 often randomly generated some weird. 521 00:29:00,788 --> 00:29:04,538 You know, hodgepodge, you know, random, alphanumeric, upper, 522 00:29:04,538 --> 00:29:05,798 lower, all that good stuff. 523 00:29:05,798 --> 00:29:07,088 And they're usually pretty long. 524 00:29:07,928 --> 00:29:12,758 in some cases, especially from, uh, you know, developers and, and non-security 525 00:29:12,758 --> 00:29:16,238 focused people, those passwords are coincidental with other things. 526 00:29:16,688 --> 00:29:20,078 And so if bad guys stole credentials in this case, there's a good 527 00:29:20,078 --> 00:29:23,618 chance that developer uses that password or those credentials or 528 00:29:23,618 --> 00:29:25,358 those tokens for other things. 529 00:29:25,724 --> 00:29:26,894 When will we learn? 530 00:29:29,108 --> 00:29:34,028 So, um, especially if you're a, a, a man, you, you know, you're a, you're 531 00:29:34,028 --> 00:29:38,468 a development shop where you're doing development for multiple clients, very 532 00:29:38,468 --> 00:29:43,298 often, you know, your credentials are the same across multiple organizations, 533 00:29:43,298 --> 00:29:44,528 which is also bad practice. 534 00:29:44,528 --> 00:29:49,388 But, um, yeah, if, if they don't take this seriously and, and. 535 00:29:50,753 --> 00:29:54,503 You know, gonna kind of go scorched earth on rebuilding and remediating this. 536 00:29:54,953 --> 00:29:58,433 Uh, it could be, it could continuous, uh, continue to be bad 537 00:29:58,433 --> 00:29:59,783 for a lot of these organizations. 538 00:30:00,263 --> 00:30:05,543 And, and by the way, uh, even though the, this attack did attempt to, uh, 539 00:30:05,543 --> 00:30:09,953 appears to attempt to exfiltrate some data again, the fir, the primary focus 540 00:30:09,953 --> 00:30:14,543 of this was credential harvesting, which would be a type of attack that 541 00:30:14,543 --> 00:30:16,583 an initial access broker would take. 542 00:30:17,018 --> 00:30:18,188 I just want the credentials. 543 00:30:18,188 --> 00:30:21,398 I'm gonna sell those to other people that knew, uh, that know how to 544 00:30:21,398 --> 00:30:25,058 do or, or are interested in, in various other types of attacks. 545 00:30:25,651 --> 00:30:30,211 you don't know what an in initial access broker, uh, is, listen to our episode. 546 00:30:30,301 --> 00:30:31,711 What is an initial access? 547 00:30:32,161 --> 00:30:34,081 Initial access broker? 548 00:30:34,381 --> 00:30:37,711 I'll put a link to it in the, in the, uh, episode description. 549 00:30:38,731 --> 00:30:41,341 so let's talk about some action, some action items. 550 00:30:41,341 --> 00:30:43,171 And the first thing I'm gonna put on there. 551 00:30:43,991 --> 00:30:47,981 gonna say that, you know, for the future, you need an inventory of your environment. 552 00:30:48,311 --> 00:30:51,731 You need an inventory of what, what software you're using and 553 00:30:51,731 --> 00:30:53,171 what dependencies you have. 554 00:30:53,441 --> 00:30:57,761 Uh, you know, so I, I, I live in, I live in, you know, California and 555 00:30:57,761 --> 00:31:02,261 one of the things we have here, if you sell food, you have to keep a 556 00:31:02,261 --> 00:31:07,571 list of all of the suppliers, uh, of where you get the food so that when. 557 00:31:07,996 --> 00:31:11,566 Um, you know, there, there's a, you know, a what, what's that? 558 00:31:11,626 --> 00:31:11,896 What? 559 00:31:11,896 --> 00:31:16,276 An coli on spinach, uh, outbreak. 560 00:31:16,546 --> 00:31:21,016 And they say it was these suppliers you can immediately know, uh, you know. 561 00:31:21,106 --> 00:31:24,316 And so I'm gonna say that the first item that that should be 562 00:31:24,316 --> 00:31:26,116 on your list is, is an inventory. 563 00:31:26,506 --> 00:31:31,246 Um, and, um, and of course you're gonna audit the, your current versions. 564 00:31:31,606 --> 00:31:35,686 Um, for, for light, LLM, let's talk about, 565 00:31:35,724 --> 00:31:35,904 Wait, 566 00:31:36,316 --> 00:31:36,556 de. 567 00:31:36,979 --> 00:31:38,994 so, so you talked about an inventory. 568 00:31:39,354 --> 00:31:43,494 Could I also say an inventory with processes to make sure people don't do 569 00:31:43,494 --> 00:31:47,664 things outside of sort of like what's approved or like Mike had said, sort of 570 00:31:47,664 --> 00:31:53,279 like a security team who's kind of vetting the libraries before they sort of are 571 00:31:53,279 --> 00:31:55,734 allowed to be used within an organization. 572 00:31:56,606 --> 00:31:57,376 Yeah, absolutely. 573 00:31:57,376 --> 00:32:00,841 That makes a, that, that makes a, a, a or that makes a lot of sense. 574 00:32:01,081 --> 00:32:04,681 Um, and then we're gonna talk about, um. 575 00:32:05,371 --> 00:32:10,771 The dependency pending and, and hash, um, the use of hash values 576 00:32:10,781 --> 00:32:12,361 Prasanna , do you wanna talk about that? 577 00:32:13,044 --> 00:32:13,314 Yeah. 578 00:32:13,314 --> 00:32:16,224 So what happened in this attack, like Mike said, right? 579 00:32:16,224 --> 00:32:20,634 The attackers updated the version of light LLM. 580 00:32:20,634 --> 00:32:25,824 People started downloading it, and one of the reasons that happened is sometime. 581 00:32:26,004 --> 00:32:30,324 When people set up to do these pulls of these dependencies in libraries, 582 00:32:30,654 --> 00:32:31,704 they'll just say, give me the latest. 583 00:32:33,189 --> 00:32:33,609 Right. 584 00:32:33,729 --> 00:32:39,249 Rather than saying, okay, I want this particular version such that you 585 00:32:39,279 --> 00:32:41,229 know that, okay, that's the only one. 586 00:32:41,229 --> 00:32:45,729 So if they had done version pinning, which is to specify a particular version 587 00:32:46,029 --> 00:32:50,319 for my build, then they wouldn't have been able to download the latest version 588 00:32:50,319 --> 00:32:52,479 because it's only gonna pick one version. 589 00:32:53,634 --> 00:32:54,024 Right. 590 00:32:54,204 --> 00:32:56,304 And so that would've prevented that issue. 591 00:32:56,514 --> 00:32:58,974 Another thing I know, Mike, you alluded to this earlier, is 592 00:32:58,974 --> 00:33:01,254 also sort of the hashing, right? 593 00:33:01,254 --> 00:33:06,444 So when you are downloading a version, confirm that yes, this is the LA latest, 594 00:33:06,444 --> 00:33:10,224 or this is a version that I care about and here's the hash that goes with it. 595 00:33:10,224 --> 00:33:14,364 So I know that it is a valid, uh, version of that library. 596 00:33:14,364 --> 00:33:15,564 It's something that I expected. 597 00:33:15,804 --> 00:33:18,084 You wanna prevent sort of the supply chain. 598 00:33:18,564 --> 00:33:22,944 Attacks from immediately impacting you because if you can sort of delay when 599 00:33:22,944 --> 00:33:27,384 you take the latest version, it gives other people time to react and sort of 600 00:33:27,384 --> 00:33:29,364 uncover these issues before you get hit. 601 00:33:29,574 --> 00:33:32,454 It's kinda like when you download the latest, uh, software updates 602 00:33:32,454 --> 00:33:34,284 on your phone or your car, right? 603 00:33:34,284 --> 00:33:37,674 Some people are like, oh, I wanna be day one, like right as soon as it's available. 604 00:33:37,674 --> 00:33:40,824 Versus others are like, Hey, let's wait till it's baked out. 605 00:33:40,824 --> 00:33:42,234 And they sort of worked out all the bugs. 606 00:33:42,234 --> 00:33:45,924 And I'll take like the dot two or the dot three of that initial major version. 607 00:33:47,061 --> 00:33:49,241 You know, as a person who's who, you know. 608 00:33:50,161 --> 00:33:52,081 I, I am not a developer, right? 609 00:33:52,111 --> 00:33:53,761 Uh, never have been a developer. 610 00:33:53,761 --> 00:33:57,301 I have written some Pearl, I've written some pretty mean pearl in my day. 611 00:33:58,351 --> 00:34:02,311 But, uh, but I am not a developer with dependencies and such. 612 00:34:02,941 --> 00:34:07,531 That doesn't mean I don't know what app get, you know, and, uh, you know, and, uh, 613 00:34:08,281 --> 00:34:13,701 so I'm gonna be thinking about that every time I. You know, have, I'm downloading 614 00:34:13,701 --> 00:34:17,631 a tool and it says the first thing you need to do is update all your libraries. 615 00:34:17,961 --> 00:34:18,471 Right. 616 00:34:18,831 --> 00:34:23,781 Um, but the, I'm, I'm dependent on the people that wrote that tool to 617 00:34:23,781 --> 00:34:27,111 do the things you're talking about, because this is something that the 618 00:34:27,111 --> 00:34:30,141 person writing the tool has to do. 619 00:34:30,141 --> 00:34:30,411 Right? 620 00:34:30,411 --> 00:34:31,461 They have to, they have to. 621 00:34:32,236 --> 00:34:37,306 know, you said specifically call out particular versions and also, uh, uh, so 622 00:34:37,306 --> 00:34:41,746 again, because I'm not a developer, is that hash, is that going to be provided 623 00:34:41,866 --> 00:34:46,186 by that tool, or is this something you're going to create when you download the 624 00:34:46,186 --> 00:34:48,081 trusted version that you're familiar with? 625 00:34:50,283 --> 00:34:52,173 The vendor, the developer, 626 00:34:52,621 --> 00:34:52,911 Okay. 627 00:34:53,463 --> 00:34:56,403 the trusted source would provide the hash value. 628 00:34:56,971 --> 00:34:57,361 Okay. 629 00:34:57,511 --> 00:34:57,901 Okay. 630 00:34:58,143 --> 00:34:59,013 So, uh. 631 00:34:59,401 --> 00:35:03,661 speaking, speaking, as a non-developer, that sounds backwards to me. 632 00:35:03,691 --> 00:35:09,511 Like, so like some point we have to trust this, this vendor, right? 633 00:35:09,511 --> 00:35:13,891 So this is, this is, so we have a trusted version and they're going 634 00:35:13,891 --> 00:35:16,711 to, how do we get that trusted version in the first place? 635 00:35:18,721 --> 00:35:21,391 How do we determine which version is a trusted version? 636 00:35:22,328 --> 00:35:26,348 Yeah, and we, so we've gotta, we've gotta establish it as a trusted source. 637 00:35:26,618 --> 00:35:30,548 And so whether it's directly from the vendor and, you know, they go 638 00:35:30,548 --> 00:35:34,508 through any number of certifications as a trusted source, you know, 639 00:35:34,508 --> 00:35:36,278 their processes, their controls. 640 00:35:36,638 --> 00:35:41,883 So that could be an ISO certification or a, a SOC two, type two, uh, 641 00:35:41,883 --> 00:35:43,843 audit certification, or, you know. 642 00:35:44,318 --> 00:35:49,418 Something like that, that helps us as consumers, uh, feel confident 643 00:35:49,418 --> 00:35:54,098 that they're doing business in a secure and, um, you know, good way. 644 00:35:55,988 --> 00:35:57,638 But it's just a piece of paper, right? 645 00:35:57,968 --> 00:36:01,148 And it's, it's some third party that it happened at some point in time. 646 00:36:01,268 --> 00:36:05,048 You know, I could have gotten an ISO 27,001 security certification 647 00:36:05,048 --> 00:36:06,818 over everything I do yesterday. 648 00:36:07,058 --> 00:36:09,158 Well, today's a new day, and I could have changed things. 649 00:36:09,158 --> 00:36:12,548 And so there's always a level of diligence, regardless of how 650 00:36:12,548 --> 00:36:14,048 much trust you put in something. 651 00:36:14,763 --> 00:36:18,753 Um, and there are several organizations I'm I know of that whenever they 652 00:36:18,753 --> 00:36:22,503 download something new from a trusted source or not, they run it in a 653 00:36:22,503 --> 00:36:27,273 sandbox environment for a period of time, to determine operational impact. 654 00:36:27,303 --> 00:36:28,473 Is this gonna change? 655 00:36:28,593 --> 00:36:29,338 Or, you know, is it going to. 656 00:36:30,053 --> 00:36:32,843 Kill a process or is it even gonna work with our systems? 657 00:36:33,893 --> 00:36:36,443 we see this a lot with Microsoft patches. 658 00:36:36,473 --> 00:36:39,083 You know, those, those are all well known for creating issues. 659 00:36:39,833 --> 00:36:43,433 Uh, well this could, this could very well follow that same methodology. 660 00:36:43,793 --> 00:36:48,593 Whatever you download, you need to sandbox it, for a period of time 661 00:36:48,893 --> 00:36:52,373 before you implement it in your, even in your test dev environment. 662 00:36:52,973 --> 00:36:54,268 Um, but yes. 663 00:36:54,391 --> 00:36:55,711 a lot of, go ahead. 664 00:36:56,603 --> 00:37:00,593 Yes, you're the, the people that created something and they want to be, you know, 665 00:37:00,593 --> 00:37:04,433 they wanna maintain their reputation and the, the integrity around their product. 666 00:37:04,853 --> 00:37:11,033 Uh, they will often publish the hash value of that file or that object. 667 00:37:12,113 --> 00:37:15,293 and it's very difficult to, uh, 668 00:37:17,783 --> 00:37:22,523 it's very difficult to falsify a hash value. 669 00:37:23,636 --> 00:37:24,316 Right, right. 670 00:37:25,006 --> 00:37:28,936 I've done, I. a bit of work with, uh, you know, living where I live. 671 00:37:28,936 --> 00:37:31,606 I've done quite a bit of work with, uh, the um. 672 00:37:32,626 --> 00:37:37,156 Uh, biotech folks and they definitely have this concept of, you know, verified 673 00:37:37,156 --> 00:37:40,936 systems that have, uh, it's not the term verified, it's been a while. 674 00:37:41,416 --> 00:37:45,916 They have another term for the systems that have been verified, uh, and they 675 00:37:45,916 --> 00:37:49,276 very much, you change a single thing and the environment and they have 676 00:37:49,276 --> 00:37:51,796 to reverify the entire, uh, system. 677 00:37:51,796 --> 00:37:51,976 So. 678 00:37:52,911 --> 00:37:55,311 That's something, a lesson that we could take from them. 679 00:37:55,611 --> 00:37:59,181 I am of course, going to suggest that if you haven't hardened your backups, 680 00:37:59,181 --> 00:38:00,681 now's the time to harden your backups. 681 00:38:00,914 --> 00:38:05,594 We talk about this a lot in the book and the, the, and, and all of the usual things 682 00:38:05,594 --> 00:38:09,734 of, of MFA and password management, and hopefully pass keys moving forward to 683 00:38:09,734 --> 00:38:12,254 pass keys, uh, and separating, right? 684 00:38:12,254 --> 00:38:15,374 So, uh, you know, putting, uh, a different, um. 685 00:38:16,824 --> 00:38:19,314 and authorization system for your backups. 686 00:38:19,644 --> 00:38:23,334 I know it's a pain, but just like everything else in security, uh, 687 00:38:23,334 --> 00:38:27,204 you know, it secure, you know, good security and convenience are not 688 00:38:27,204 --> 00:38:31,704 necessarily in the same, uh, you know, in the same, uh, um, ballpark. 689 00:38:31,725 --> 00:38:34,185 And the number one thing here that I'm gonna, that I'm gonna be 690 00:38:34,185 --> 00:38:35,865 harping on is immutable storage. 691 00:38:36,075 --> 00:38:36,403 Right. 692 00:38:36,403 --> 00:38:41,923 So this entire time though, we talked about this library, which was supply chain 693 00:38:41,923 --> 00:38:43,693 attacked, which were stealing credentials. 694 00:38:43,693 --> 00:38:47,473 Could you help our listeners understand the link between having 695 00:38:47,473 --> 00:38:50,623 immutable backups and this attack? 696 00:38:51,185 --> 00:38:51,635 Yeah. 697 00:38:51,665 --> 00:38:52,055 Great. 698 00:38:52,055 --> 00:38:55,715 So the, the, the, the, one of the things you, if you go back earlier in the 699 00:38:55,715 --> 00:39:01,175 episode, one of the things Mike said was restore your, um, what, whatever 700 00:39:01,175 --> 00:39:04,145 this is from a trusted backup, right? 701 00:39:04,145 --> 00:39:08,675 From a backup that you trust the, the thing with the immutable backups 702 00:39:08,675 --> 00:39:12,875 is one of the things that, that just hurts my little heart when I see it 703 00:39:12,965 --> 00:39:18,155 out there is when a ransomware or a malware attack happens, and you see the 704 00:39:18,155 --> 00:39:20,225 little phrase at the end of the story. 705 00:39:20,450 --> 00:39:23,030 And the backups were also corrupted, right? 706 00:39:23,270 --> 00:39:26,240 So having immutable just means cannot be changed. 707 00:39:26,240 --> 00:39:30,470 And the standard by which I judge immutable backups is if you can delete 708 00:39:30,470 --> 00:39:36,410 them, If you as an admin can delete your old backups, then those aren't immutable. 709 00:39:37,310 --> 00:39:40,190 At least that's, that's the gold standard that I'm putting. 710 00:39:41,030 --> 00:39:44,750 So, um, configure your backups in such a way. 711 00:39:44,750 --> 00:39:45,560 Talk to your vendor. 712 00:39:45,560 --> 00:39:46,280 How do I do this? 713 00:39:46,280 --> 00:39:51,860 Configure your backups in such a way that even you, the super, super, know, 714 00:39:51,860 --> 00:39:53,990 God level access on your backups. 715 00:39:53,990 --> 00:39:58,610 If you cannot delete backups before they're supposed to expire, then 716 00:39:58,610 --> 00:40:00,350 you actually have immutable backups. 717 00:40:00,350 --> 00:40:03,530 If you're anything less than that, you're immutable ish. 718 00:40:04,235 --> 00:40:05,615 I'm not saying it's crap. 719 00:40:05,765 --> 00:40:10,745 I'm just saying the closer you can get to that level of immutability, um, you 720 00:40:10,745 --> 00:40:13,895 know, and, and Prasanna you always bring up, you know, when we start talking about 721 00:40:13,895 --> 00:40:17,615 actual immutable storage, there's like the compliance mode and the what are the two 722 00:40:17,853 --> 00:40:18,523 Governance. 723 00:40:19,370 --> 00:40:20,060 governance mode. 724 00:40:20,060 --> 00:40:20,360 Right? 725 00:40:20,540 --> 00:40:24,320 And the governance mode is the more stringent one, right? 726 00:40:24,560 --> 00:40:24,890 Yeah. 727 00:40:24,890 --> 00:40:31,040 And so, uh, basically that, that mode of, uh, and we're talking, in this 728 00:40:31,040 --> 00:40:36,530 case, we're talking about like, uh, object lock in S3, that if you enable 729 00:40:36,530 --> 00:40:41,630 the, the stricter mode, even you, the owner of the account cannot delete. 730 00:40:42,425 --> 00:40:44,855 Objects before they're supposed to expire. 731 00:40:44,855 --> 00:40:48,065 And if that's the way your backups work, then that's truly immutable. 732 00:40:48,065 --> 00:40:54,185 And if that's the case, then the bad guys can't delete or encrypt or corrupt 733 00:40:54,365 --> 00:40:59,345 your backups, which means that you can then use them to restore this library. 734 00:40:59,405 --> 00:40:59,735 Right. 735 00:40:59,735 --> 00:41:03,065 That's a, that's a great, thank you for, uh, for making me. 736 00:41:03,465 --> 00:41:06,405 Uh, get up on my soapbox and, uh, and explain that. 737 00:41:06,795 --> 00:41:09,225 and again, I, I, I, I mentioned it already, but, 738 00:41:09,225 --> 00:41:10,550 um, of course we're, you know. 739 00:41:12,215 --> 00:41:14,145 Basically in your whole environment. 740 00:41:14,755 --> 00:41:15,935 Look at MFA. 741 00:41:15,985 --> 00:41:19,115 And again, literally the last episode, Mike was a little bit 742 00:41:19,115 --> 00:41:23,795 rolling his eyes on, on MFA, but not, he doesn't think my MFA is bad. 743 00:41:24,035 --> 00:41:28,175 He just, it's not perfect, which is why we're trying to move to pass keys. 744 00:41:28,445 --> 00:41:32,015 But if you don't have MFA, if you have passwords in the wild. 745 00:41:32,240 --> 00:41:34,880 That are, that are securing things that are important. 746 00:41:34,880 --> 00:41:39,620 And you don't have MFA Mike, do you wanna explain what, what, why, again, 747 00:41:39,620 --> 00:41:45,170 why is MFA, what does it do, uh, in, in this situation when somebody does 748 00:41:45,170 --> 00:41:46,970 happen to harvest your credentials? 749 00:41:46,970 --> 00:41:48,410 What is the purpose of MFA? 750 00:41:50,937 --> 00:41:54,487 MFA is supposed to be a second, layer of security. 751 00:41:54,667 --> 00:41:57,277 And we, we consider it an out of, out of band. 752 00:41:58,087 --> 00:42:02,107 Of band means, you know, if I'm logging into my computer, the MFA doesn't 753 00:42:02,107 --> 00:42:06,037 pull up on this computer, it goes to my phone or a, or an authenticator 754 00:42:06,037 --> 00:42:08,257 app or any, another email address. 755 00:42:08,827 --> 00:42:12,547 Uh, and that's important because if, if bad guys also capture your MFA 756 00:42:12,547 --> 00:42:15,457 token and they're already at your computer, or they're already in your 757 00:42:15,457 --> 00:42:18,667 environment and they already have your credentials, then your MFA is is. 758 00:42:19,072 --> 00:42:20,122 Useless it. 759 00:42:20,122 --> 00:42:22,492 It's not providing that extra layer of security. 760 00:42:23,287 --> 00:42:26,827 MFA is also, uh, a, a good way of determining if your 761 00:42:26,827 --> 00:42:28,177 credentials have been stolen. 762 00:42:28,627 --> 00:42:32,677 Uh, so if you get a, uh, a text message or an email on your phone that says, 763 00:42:32,677 --> 00:42:37,327 here's your, here's your MFA key for Facebook or LinkedIn, you're like, well, 764 00:42:37,327 --> 00:42:39,277 I'm not logging into those right now. 765 00:42:40,387 --> 00:42:41,137 else is. 766 00:42:41,587 --> 00:42:42,997 Uh, and so that's a good indication. 767 00:42:42,997 --> 00:42:45,787 You need to go change your credentials and, and maybe even try 768 00:42:45,787 --> 00:42:47,047 to figure out how that happened. 769 00:42:47,047 --> 00:42:49,747 But the problems with MFA. 770 00:42:50,587 --> 00:42:57,487 Is if I'm on my computer and I log into something and it says, Hey, 771 00:42:57,547 --> 00:42:59,827 uh, you need to check your MFA. 772 00:43:01,057 --> 00:43:03,727 Device or your app put in that code. 773 00:43:04,207 --> 00:43:08,737 The next, very next thing that happens is usually why MFA's 774 00:43:08,767 --> 00:43:10,867 value diminishes significantly. 775 00:43:10,897 --> 00:43:15,517 And that is a popup, uh, window or a, a subsequent webpage that says, 776 00:43:16,267 --> 00:43:18,187 do you want to trust this device? 777 00:43:18,517 --> 00:43:20,077 Do you want me to remember you? 778 00:43:20,497 --> 00:43:26,647 And if you click yes, then you don't have to do MFA for that anymore. 779 00:43:27,397 --> 00:43:30,907 So at work for your bank, probably not your bank, but you know, 780 00:43:30,937 --> 00:43:33,487 LinkedIn, Gmail, whatever it is. 781 00:43:33,517 --> 00:43:37,207 If you click remember me or Trust this device, you have saved 782 00:43:37,207 --> 00:43:39,187 that MFA token in your browser. 783 00:43:40,237 --> 00:43:44,557 And so now bad guys just need to get you to go to a bad website or 784 00:43:44,557 --> 00:43:48,667 potentially even download some malware and they will harvest that MFA token. 785 00:43:49,477 --> 00:43:52,267 And if they can compromise your credentials by getting you to 786 00:43:52,267 --> 00:43:54,547 click a link, they can also. 787 00:43:55,567 --> 00:44:01,777 Create a new session as you, with that new MFA token whereby bypassing the 788 00:44:01,777 --> 00:44:04,142 value of having MFA to begin with. 789 00:44:04,960 --> 00:44:09,775 they only do that on the computer where the MFA token was, was generated? 790 00:44:10,125 --> 00:44:11,175 It's only valid there. 791 00:44:11,175 --> 00:44:11,375 Right. 792 00:44:13,232 --> 00:44:14,347 They, they cannot do it. 793 00:44:14,437 --> 00:44:16,417 They, they, they're not limited to the computer. 794 00:44:16,417 --> 00:44:17,107 It was generated on. 795 00:44:17,932 --> 00:44:21,922 They just need access to the browser or the, the computer to take the 796 00:44:21,922 --> 00:44:25,402 saved MFA token out of the browser. 797 00:44:26,182 --> 00:44:27,262 I can do that remotely. 798 00:44:27,262 --> 00:44:29,152 I can do that remotely from anywhere in the world. 799 00:44:29,962 --> 00:44:33,802 I can get you to go to a bad website, will then suck that 800 00:44:33,802 --> 00:44:35,842 MFA token outta your browser. 801 00:44:37,462 --> 00:44:41,122 Or get you to click on a phishing email or go to a website to 802 00:44:41,122 --> 00:44:42,592 download, you know, malware. 803 00:44:42,592 --> 00:44:47,242 And that malware similar to this light LLM, uh, will harvest, 804 00:44:47,482 --> 00:44:50,332 uh, those MFA tokens for me. 805 00:44:50,580 --> 00:44:53,940 And, and so just to make sure I understand, so if they get that, if 806 00:44:53,940 --> 00:44:57,870 they've got your credential, you know, your username and password and that saved 807 00:44:57,870 --> 00:45:03,480 MFA token, even though that token was created on this laptop, they can use those 808 00:45:03,480 --> 00:45:06,540 three things to log in as me anywhere. 809 00:45:08,097 --> 00:45:08,877 High Probability 810 00:45:10,858 --> 00:45:11,888 Curtis is freaked out. 811 00:45:16,115 --> 00:45:17,345 Why you always doing this to me, Mike? 812 00:45:18,022 --> 00:45:20,242 Trying to, I'm trying to get your hair to match my hair. 813 00:45:21,742 --> 00:45:25,492 There are things you can do, uh, from an organization security perspective 814 00:45:25,492 --> 00:45:30,652 to limit that in, in, in other words, uh, as a security admin for a company. 815 00:45:31,252 --> 00:45:35,782 Uh, I can go into Office 365 as an example and say, you 816 00:45:35,782 --> 00:45:38,392 know, no, no concurrent logins. 817 00:45:38,542 --> 00:45:40,132 You know, Mike can only log in one time. 818 00:45:40,552 --> 00:45:45,412 I can say, you know, uh, Mike can only log in from domestic ips. 819 00:45:45,412 --> 00:45:49,762 Or we block all, you know, bat known bad ips, you know, China, 820 00:45:50,182 --> 00:45:52,942 North Korea, um, et cetera. 821 00:45:53,842 --> 00:45:56,812 there's a list of those that's published every day of every, every week. 822 00:45:58,252 --> 00:46:00,142 there are things that we can alert on. 823 00:46:00,682 --> 00:46:03,772 Uh, and so if Mike's logged in and Mike logs in again from a 824 00:46:03,772 --> 00:46:06,832 different app IP address, and we would, uh, es especially one that's. 825 00:46:07,447 --> 00:46:08,227 Very far away. 826 00:46:08,227 --> 00:46:09,757 We call that impossible travel. 827 00:46:10,447 --> 00:46:14,527 Uh, so if you, if I've logged in from Texas and, and, you know, 10 minutes 828 00:46:14,527 --> 00:46:19,987 from now someone logs in from even, you know, Kansas, impossible travel. 829 00:46:20,377 --> 00:46:24,067 And so that should be alerted on and potentially even automatically blocked. 830 00:46:24,457 --> 00:46:27,907 And if, if we want to take a very strict approach to that. 831 00:46:28,507 --> 00:46:32,227 Uh, whenever we see that impossible travel without explanation, 832 00:46:32,227 --> 00:46:33,547 we suspend the account. 833 00:46:33,967 --> 00:46:34,777 Yeah, yeah, sure. 834 00:46:34,777 --> 00:46:37,867 Mike isn't gonna be able to work for a couple of minutes, but Mike's about to get 835 00:46:37,867 --> 00:46:41,497 a phone call and say, you know, where are you, Mike, and what are you working on? 836 00:46:41,947 --> 00:46:45,877 Um, so we can clear this up before things get bad, and that is the key 837 00:46:45,907 --> 00:46:47,497 to incident response these days. 838 00:46:47,497 --> 00:46:51,697 It is how fast can we respond to weird stuff before bad things happen? 839 00:46:53,130 --> 00:46:57,720 Speaking of alerting, Mike, uh, do you want to talk about how the kinds of things 840 00:46:57,720 --> 00:47:01,770 that people should be doing to make sure that they are aware, that they get these 841 00:47:01,770 --> 00:47:04,290 alerts when something like this happens? 842 00:47:04,560 --> 00:47:06,630 What, what should they be following? 843 00:47:06,630 --> 00:47:08,220 How and how should they be doing that? 844 00:47:09,407 --> 00:47:13,852 I'll tell you just about every tool that's out there has free training. 845 00:47:14,032 --> 00:47:15,832 We just we're too lazy to take it. 846 00:47:16,042 --> 00:47:18,502 You know, we're such a consumer driven culture. 847 00:47:18,502 --> 00:47:21,652 We just want the latest, greatest, use it now, share it with my 848 00:47:21,652 --> 00:47:23,962 friends, and move on with our day. 849 00:47:24,712 --> 00:47:28,612 very rarely set time aside to watch the video or read the manual. 850 00:47:29,422 --> 00:47:31,282 and I'm, I'm guilty of that too. 851 00:47:31,762 --> 00:47:32,272 Um. 852 00:47:33,157 --> 00:47:34,477 But it's, it's all out there. 853 00:47:34,507 --> 00:47:39,697 So if you wanna know how to secure your Gmail or your, don't be using Yahoo 854 00:47:39,697 --> 00:47:44,077 or Hotmail still, or definitely not a OL, but if you have a, whatever your 855 00:47:44,077 --> 00:47:48,787 account is, there is guidance out there from whoever that provider is to help 856 00:47:48,787 --> 00:47:53,467 you secure it and, and be more aware of when weird things happen, for example. 857 00:47:54,457 --> 00:47:57,877 In Gmail, there's a security tab where you can see all the last logins 858 00:47:57,877 --> 00:47:59,677 and IP addresses and time and date. 859 00:47:59,677 --> 00:48:01,777 And a lot of people don't know that you can do the same thing 860 00:48:01,777 --> 00:48:04,447 with iCloud, uh, for your bank. 861 00:48:05,197 --> 00:48:08,467 Very similarly, uh, there's a security tab, when were the last 862 00:48:08,467 --> 00:48:11,737 logins, what did I do, you know, uh, from an activity perspective. 863 00:48:12,487 --> 00:48:14,737 and those are just the authentication pieces. 864 00:48:15,007 --> 00:48:16,267 Well, what about the behavior? 865 00:48:16,267 --> 00:48:18,127 So what if someone was able to. 866 00:48:18,892 --> 00:48:21,232 Log in, uh, to one of these accounts. 867 00:48:21,652 --> 00:48:26,212 Uh, and like my bank, what if, what if they start to transfer money or 868 00:48:26,212 --> 00:48:28,822 they steal my credit card and they're, you know, they're buying tires. 869 00:48:28,822 --> 00:48:32,122 In Utah, there are ways of setting alerts. 870 00:48:32,122 --> 00:48:36,412 You just have to be willing to manage it. 871 00:48:36,802 --> 00:48:40,312 Uh, so for example, everything over a dollar on my credit card, my 872 00:48:40,312 --> 00:48:41,842 debit card, I get a text message. 873 00:48:42,293 --> 00:48:43,043 I do the same. 874 00:48:44,062 --> 00:48:44,422 Right. 875 00:48:44,422 --> 00:48:47,452 So, and I'm okay with that and in fact, it's kind of cool 876 00:48:47,452 --> 00:48:48,712 to see how fast that happens. 877 00:48:48,712 --> 00:48:51,442 I'm at the grocery store, I just said, please remove your car. 878 00:48:51,442 --> 00:48:52,492 And I got a text message. 879 00:48:52,642 --> 00:48:53,272 That's awesome. 880 00:48:54,142 --> 00:48:55,822 Um, and very similarly, I was 881 00:48:56,015 --> 00:48:59,580 I, I hate, I hate to cut you off, but Prasanna 's gonna turn into a pumpkin. 882 00:48:59,910 --> 00:49:01,680 The, those are great things. 883 00:49:01,770 --> 00:49:03,480 It wasn't the question I was asking. 884 00:49:03,690 --> 00:49:03,840 My 885 00:49:04,132 --> 00:49:04,672 sorry. 886 00:49:04,890 --> 00:49:06,270 is No, it's fine. 887 00:49:06,540 --> 00:49:08,170 Uh, what I'm talking about is what? 888 00:49:09,015 --> 00:49:11,655 kinds of things I, as a company should be looking for? 889 00:49:11,655 --> 00:49:15,885 Where I get these alerts that, that a security incident is, 890 00:49:15,975 --> 00:49:17,565 it's like this one is happening. 891 00:49:17,565 --> 00:49:18,360 That's what I'm talking about. 892 00:49:19,162 --> 00:49:20,872 So you gotta define the role first. 893 00:49:21,172 --> 00:49:22,912 You know, who's gonna be responsible for this? 894 00:49:22,912 --> 00:49:25,072 Nobody wants to look at logs and alerts all day. 895 00:49:25,072 --> 00:49:29,062 They've, it's usually someone's part-time job they do at lunch or at the end of 896 00:49:29,062 --> 00:49:30,202 the day, or first thing in the morning. 897 00:49:30,202 --> 00:49:30,682 And that's it. 898 00:49:30,682 --> 00:49:32,302 It's not real time all the time. 899 00:49:33,292 --> 00:49:35,212 you should have a dedicated person for this. 900 00:49:36,532 --> 00:49:39,727 you need a. Defined incident response plan. 901 00:49:40,117 --> 00:49:43,867 Uh, and so for every alert I get, I need to follow these procedures 902 00:49:43,867 --> 00:49:47,437 every alert, uh, even if it's false positive, you've gotta go through 903 00:49:47,437 --> 00:49:50,497 the process of determining it's false positive and documenting that. 904 00:49:50,497 --> 00:49:53,257 So in the future, someone goes, Hey, that thing happened. 905 00:49:53,257 --> 00:49:54,727 How come we didn't do something about it? 906 00:49:54,727 --> 00:49:58,387 Well, I looked at it and if I was false positive, you know, it wasn't, 907 00:49:58,448 --> 00:49:58,958 I think. 908 00:49:59,100 --> 00:50:01,320 do I, where do I get these alerts? 909 00:50:01,320 --> 00:50:02,400 This is my question. 910 00:50:02,857 --> 00:50:04,597 it wasn't a legit, uh, alert. 911 00:50:04,597 --> 00:50:07,117 So go into all of your systems. 912 00:50:07,942 --> 00:50:10,852 if, if you've got an environment where someone's managing your stuff, 913 00:50:11,212 --> 00:50:16,102 uh, you need to turn on event logging for as much as you can, consolidate 914 00:50:16,102 --> 00:50:18,682 all those logs into one place. 915 00:50:18,682 --> 00:50:23,512 And there's a variety of things you can do, like a SIS log server or, uh, uh, 916 00:50:23,512 --> 00:50:28,612 there's some free open source, uh, log consolidation and analysis tools like Sim, 917 00:50:28,612 --> 00:50:33,922 monster, uh, SIEM, monster, um, there are. 918 00:50:34,307 --> 00:50:38,777 Um, a variety of, uh, automated scripts like python's. 919 00:50:38,777 --> 00:50:43,487 One of them, powershells one where you can use those to, uh, to alert on specific 920 00:50:43,487 --> 00:50:45,887 event IDs and to know what those are. 921 00:50:45,887 --> 00:50:49,247 Google it, what if, what security and event ID should I be concerned about? 922 00:50:49,577 --> 00:50:54,047 And you'll get a list of those, or, uh, call, a call a managed service provider. 923 00:50:54,047 --> 00:50:58,037 If you don't have the, the skills and the staff to, to support that activity, they 924 00:50:58,037 --> 00:50:59,807 can consult with you about what you have. 925 00:50:59,807 --> 00:51:03,042 How to configure it, what to do with it internally, uh, and how 926 00:51:03,042 --> 00:51:06,792 they could help if you need, um, you know, additional skills and staff, 927 00:51:06,792 --> 00:51:08,652 especially if it's a 24 7 thing. 928 00:51:08,922 --> 00:51:12,222 Cybersecurity managed services today are so affordable. 929 00:51:12,222 --> 00:51:13,302 Everybody should have it. 930 00:51:13,842 --> 00:51:17,712 Uh, there's just no excuse and if you don't have it, it's gonna impact your 931 00:51:17,712 --> 00:51:19,752 ability to get insurance in the future. 932 00:51:19,962 --> 00:51:24,072 If you have a breach like this, your, your damages from lawsuits are gonna be a lot 933 00:51:24,072 --> 00:51:25,782 bigger 'cause you, you weren't diligent. 934 00:51:26,232 --> 00:51:27,822 Uh, but yeah, there's absolutely. 935 00:51:28,842 --> 00:51:31,692 number of ways of collecting this information, being able 936 00:51:31,692 --> 00:51:33,162 to automatically alert on it. 937 00:51:33,162 --> 00:51:36,612 You just have to have the people and the procedures available 938 00:51:36,612 --> 00:51:37,872 to, uh, to take action. 939 00:51:38,168 --> 00:51:41,498 and Mike, I guess, oh, as Curtis had asked that question, one thing I was thinking 940 00:51:41,498 --> 00:51:44,468 about is like, Hey, I work at a company. 941 00:51:44,738 --> 00:51:46,868 All these issues are constantly happening. 942 00:51:47,408 --> 00:51:53,888 Where as a person do I go to understand, Hey, where are the latest breach alerts 943 00:51:53,888 --> 00:51:55,628 or other things like that happening. 944 00:51:55,658 --> 00:51:56,618 That's kind of what I was thinking. 945 00:51:56,618 --> 00:51:58,608 I don't know, Curtis, if that was what you were intending to, but. 946 00:51:58,695 --> 00:52:01,185 is, that is the, that is the question I was asking Mike. 947 00:52:01,335 --> 00:52:04,785 Was just, I just wanted you to say like cbe.org or something. 948 00:52:05,055 --> 00:52:06,375 That's what I was looking for. 949 00:52:07,222 --> 00:52:07,867 I apologize. 950 00:52:07,867 --> 00:52:09,997 I'll start, I'll start clarifying my understanding of 951 00:52:09,997 --> 00:52:11,197 your questions in the future. 952 00:52:11,377 --> 00:52:17,587 Uh, so cisa.gov, cisa.gov, uh, is a good, uh, site. 953 00:52:18,067 --> 00:52:23,437 Um, there are, there are a ton of, uh, Twitter or X profiles. 954 00:52:24,487 --> 00:52:27,367 Uh, just search, you know, cybersecurity threat, intel and, 955 00:52:27,367 --> 00:52:29,227 and x and you'll find good accounts. 956 00:52:29,227 --> 00:52:32,857 So that's, that's people that, that's all they do, and it's very timely. 957 00:52:33,277 --> 00:52:38,137 In fact, a lot of stuff will show up there as a quote unquote proof of concept before 958 00:52:38,137 --> 00:52:42,967 ciso or some of the other agencies will actually, uh, publish it as a, a known 959 00:52:42,967 --> 00:52:45,637 vulner, uh, known exploit or an attack. 960 00:52:45,697 --> 00:52:50,197 It'll be a proof of concept that someone has, uh, observed out in the wild. 961 00:52:52,177 --> 00:52:55,957 There are vendors out there that provide free threat intelligence. 962 00:52:56,047 --> 00:52:59,617 If you're part of critical infrastructure, uh, get with your state. 963 00:52:59,707 --> 00:53:03,367 Uh, there are state information sharing and analysis centers that you can 964 00:53:03,667 --> 00:53:07,657 subscribe to for free, and you'll get daily, sometimes hourly updates on 965 00:53:07,657 --> 00:53:10,697 threats, and if you're part of the critical infrastructure working with 966 00:53:10,697 --> 00:53:14,507 the, the state ISACs, uh, they'll even help, uh, with your response. 967 00:53:14,752 --> 00:53:19,217 So, um, there, there's just too much to get into from a, from a resource 968 00:53:19,217 --> 00:53:22,157 perspective, but cisa.gov is a good one. 969 00:53:23,075 --> 00:53:27,040 I know we covered a lot of these in the book, um, you know, buy our book. 970 00:53:27,100 --> 00:53:31,870 Uh, but, uh, let, we will, I'll, if you could give me a list of those and 971 00:53:31,870 --> 00:53:33,220 we'll put 'em in the show description. 972 00:53:33,220 --> 00:53:34,960 'cause this, I think this is a big deal. 973 00:53:35,200 --> 00:53:38,440 This is an op, it's, once again, it's an opportunity for people to get scared 974 00:53:38,800 --> 00:53:41,590 to, to then go, you know, evaluate. 975 00:53:42,670 --> 00:53:43,870 Evaluate their life. 976 00:53:44,200 --> 00:53:45,460 Uh, all right. 977 00:53:45,460 --> 00:53:46,300 Well, thanks Mike. 978 00:53:46,330 --> 00:53:47,170 I This is great. 979 00:53:47,170 --> 00:53:48,910 Thanks for, thanks for getting up. 980 00:53:48,910 --> 00:53:51,950 Uh, well, it's for you, it's not, not as early, but, uh, Prasanna 981 00:53:52,480 --> 00:53:54,490 , definitely you're your early bird. 982 00:53:54,490 --> 00:53:56,890 Like you're not an early bird, but, uh, 983 00:53:57,023 --> 00:53:58,068 I am an early bird. 984 00:53:58,630 --> 00:53:58,960 Yeah. 985 00:53:58,960 --> 00:53:59,200 Okay. 986 00:53:59,200 --> 00:53:59,210 Well. 987 00:53:59,793 --> 00:54:00,063 Yeah. 988 00:54:00,285 --> 00:54:04,035 I'll just say thank you, Curtis, for, for get, for getting on the 989 00:54:04,035 --> 00:54:05,715 camera at seven 30 in the morning. 990 00:54:05,745 --> 00:54:07,035 'cause this is not, this is 991 00:54:07,233 --> 00:54:08,313 Have you had your coffee yet? 992 00:54:09,015 --> 00:54:14,145 I I've had two cups, actually, by the way, I had a cup of Java in a Java mug, 993 00:54:14,535 --> 00:54:18,525 uh, very old Java mug that you can see if you watch the YouTube version of this. 994 00:54:18,525 --> 00:54:20,865 All right, well, thanks, uh, thanks you two. 995 00:54:20,895 --> 00:54:23,153 And, uh, that is a wrap