1 00:00:00,027 --> 00:00:04,650 A credit union employee lost their logged-in laptop at a coffee shop, 2 00:00:05,116 --> 00:00:09,128 and it gave the bad guys everything they needed to phish the rest 3 00:00:09,184 --> 00:00:11,734 of the company from the inside. 4 00:00:12,525 --> 00:00:16,011 You found the Backup Wrap-Up, your go-to podcast for all 5 00:00:16,023 --> 00:00:18,287 things backup and cyber recovery. 6 00:00:18,741 --> 00:00:23,361 Prasanna and I sit down with Dr. Mike Saylor, the co-author of my book, Learning 7 00:00:23,375 --> 00:00:28,903 Ransomware Response and Recovery, to talk credential theft, how attackers 8 00:00:28,910 --> 00:00:34,034 harvest every password that they can get their hands on, uh, why shared local 9 00:00:34,041 --> 00:00:39,496 admin passwords turn one stolen device like that into the keys to the kingdom, 10 00:00:40,234 --> 00:00:44,249 and how something called Kerberoasting works once they're on your network. 11 00:00:45,170 --> 00:00:50,272 Then we get into pr- preventing credential theft with things like LAPS, just-in-time 12 00:00:50,273 --> 00:00:55,165 credentials, tiered admin accounts, and alerts that actually mean something. 13 00:00:55,940 --> 00:00:59,852 I'd also like to say Happy Diwali to all my Indian listeners. 14 00:01:00,775 --> 00:01:03,365 If this is your first time watching or listening to me, 15 00:01:03,405 --> 00:01:05,958 I'm W. Curtis Preston, AKA Mr. 16 00:01:05,968 --> 00:01:06,429 Backup. 17 00:01:06,893 --> 00:01:12,579 I've been obsessing over backup, recovery, and now cyber recovery for over 30 years. 18 00:01:12,939 --> 00:01:14,875 If that's your bag, I'm your guy. 19 00:01:15,313 --> 00:01:18,906 You're not gonna find anybody that cares about that topic more than me. 20 00:01:19,458 --> 00:01:23,786 Ever since 1993 when I had to tell my boss there were no backups of 21 00:01:23,786 --> 00:01:25,614 the database that we had just lost. 22 00:01:26,089 --> 00:01:30,083 Now I've written five O'Reilly books, a blog, and doing this podcast. 23 00:01:30,420 --> 00:01:34,489 Here, we turn unappreciated admins into cyber recovery heroes. 24 00:01:34,766 --> 00:01:36,761 This is the Backup Wrap-Up. 25 00:01:49,459 --> 00:01:50,859 Welcome to the Backup Wrap Up. 26 00:01:50,859 --> 00:01:54,059 I'm your host, W. Curtis Preston, and I have with me a guy who 27 00:01:54,059 --> 00:01:59,559 has apparently read way too many Reacher books, Prasanna Malaiyandi. 28 00:01:59,619 --> 00:02:01,759 how many Reacher books did you say? 29 00:02:02,316 --> 00:02:05,696 I think I've read 22 of them or something like that, and I think 30 00:02:05,696 --> 00:02:07,596 I still have four more to go. 31 00:02:08,946 --> 00:02:12,456 But this is… so I decided I wanted to start reading again. 32 00:02:12,486 --> 00:02:17,976 So last year I started reading and I wanted to hit 52 books, so one a week. 33 00:02:18,206 --> 00:02:21,596 I think I got to 48, so close enough I would say. 34 00:02:22,106 --> 00:02:25,766 But yes, I was reading the Reacher books because I saw the TV show. 35 00:02:25,766 --> 00:02:27,846 I was like, "Oh, this is kinda good." And this was like, I think 36 00:02:27,846 --> 00:02:28,976 they had the first two seasons out. 37 00:02:28,976 --> 00:02:32,116 I'm like, "Ah, this is interesting." And so I just started reading 38 00:02:32,116 --> 00:02:33,336 the books and I kept reading. 39 00:02:33,946 --> 00:02:39,876 And they're good, but they're not amazing book-wise. 40 00:02:40,206 --> 00:02:44,416 and so they're pretty quick reads and yeah, so I just kept reading and reading. 41 00:02:44,416 --> 00:02:48,916 And at this point, I think most of the seasons are mixed up in my head, or most 42 00:02:48,916 --> 00:02:50,296 of the books are mixed up in the head. 43 00:02:50,296 --> 00:02:50,606 So 44 00:02:50,683 --> 00:02:53,533 as you and Mike were talking about it, I was like, "I don't know which 45 00:02:53,543 --> 00:02:57,043 book that is," but apparently it's the 13th book, which is the latest season 46 00:02:58,203 --> 00:03:01,473 in, in the current season, it's the one where he beats up people. 47 00:03:01,543 --> 00:03:02,423 does that help it? 48 00:03:02,891 --> 00:03:04,361 Oh yeah, it helps out tremendously. 49 00:03:04,381 --> 00:03:05,141 Thanks, Curtis 50 00:03:07,093 --> 00:03:07,453 Yeah. 51 00:03:07,463 --> 00:03:10,093 where his physical might is a c- principal, 52 00:03:10,119 --> 00:03:10,309 Yeah. 53 00:03:10,443 --> 00:03:11,613 element of the story 54 00:03:11,779 --> 00:03:14,379 speaking of Reacher, I think we should… you and I were chit-chatting 55 00:03:14,379 --> 00:03:15,949 about this just the other day, right? 56 00:03:15,959 --> 00:03:17,399 You just finished the latest season. 57 00:03:17,399 --> 00:03:18,949 I haven't finished it, but I've read the book. 58 00:03:19,619 --> 00:03:23,949 And you were talking about how one of the bad guys in the show is, 59 00:03:23,949 --> 00:03:27,729 a 7'2" tall person because they needed to find someone who's bigger 60 00:03:27,739 --> 00:03:30,909 than the actor playing Reacher, and that's hard when you're, like, 6'9". 61 00:03:32,077 --> 00:03:32,297 yeah. 62 00:03:32,347 --> 00:03:35,087 he's massive, and the guy that they got was seven foot two. 63 00:03:35,137 --> 00:03:36,387 he is indeed massive. 64 00:03:37,097 --> 00:03:40,397 speaking of giant, massive people, how's it going? 65 00:03:40,747 --> 00:03:46,077 We have with us Dr. Mike Saylor, my co-author of Learning 66 00:03:46,077 --> 00:03:47,487 Ransomware and Recovery. 67 00:03:47,497 --> 00:03:48,327 How's it going, Mike? 68 00:03:50,120 --> 00:03:50,940 thanks for having me. 69 00:03:50,950 --> 00:03:51,290 It's good to be 70 00:03:51,341 --> 00:03:55,741 S- since I've never stood face to face with you, you could be like nine feet tall 71 00:03:57,810 --> 00:03:58,690 Only in my mind 72 00:04:00,701 --> 00:04:02,601 I, I'm with you, I'm with you, I'm with you. 73 00:04:03,041 --> 00:04:03,821 on an airplane. 74 00:04:05,284 --> 00:04:06,804 yeah, exactly. 75 00:04:07,204 --> 00:04:10,814 Mike, in this episode, we're gonna talk about, know, attacks that 76 00:04:10,814 --> 00:04:12,904 come from harvesting credentials. 77 00:04:12,904 --> 00:04:15,084 Do you have a story for us to start out with? 78 00:04:16,504 --> 00:04:17,044 Oh, I do. 79 00:04:17,294 --> 00:04:24,154 there's a small credit union that lost a laptop at a, let's call it a coffee shop. 80 00:04:24,364 --> 00:04:28,804 It was logged in, actually authenticated to their Office 365. 81 00:04:28,814 --> 00:04:29,794 Someone walked off with? 82 00:04:29,794 --> 00:04:35,304 it, which addresses some other common our hard drive's BitLocker. 83 00:04:35,344 --> 00:04:39,194 it's only encrypted when you're not using it, this one was unlocked, 84 00:04:39,724 --> 00:04:41,654 logged in, and it walked off. 85 00:04:42,364 --> 00:04:42,494 So 86 00:04:42,494 --> 00:04:47,984 not only did bad guys get access to, some authenticated systems, they had 87 00:04:47,984 --> 00:04:52,984 access to this local machine and its credentials, they were able to harvest 88 00:04:52,984 --> 00:04:56,354 not only local credentials, but then use the information that they gleaned from 89 00:04:56,354 --> 00:05:02,634 this access to now phish other employees as if they were this employee to get 90 00:05:02,634 --> 00:05:05,864 them to interact with, an attachment. 91 00:05:06,664 --> 00:05:09,274 attachment then infected one or more employees that had 92 00:05:09,284 --> 00:05:11,454 harvesting malware on it. 93 00:05:12,124 --> 00:05:15,794 and so running locally across, services and, and memory, it was 94 00:05:15,794 --> 00:05:21,154 collecting, as many credentials as it could from, users, service accounts, 95 00:05:22,256 --> 00:05:26,926 started with physically stolen credentials and that they… and this goes back to 96 00:05:26,926 --> 00:05:31,616 something that you mention a lot about not using, suspend, suspending your 97 00:05:31,616 --> 00:05:34,336 laptop, actually logging off the device. 98 00:05:34,346 --> 00:05:37,736 So had they actually logged off, this would not have been as valuable a 99 00:05:37,736 --> 00:05:39,556 device as it ended up being, right? 100 00:05:42,879 --> 00:05:47,439 It seems like most attacks, or would you say, Mike, and I know you're gonna 101 00:05:47,439 --> 00:05:51,929 say it depends, but it seems like most attacks you're probably going 102 00:05:51,929 --> 00:05:56,619 to… once the attackers are in, it seems pretty likely that the first 103 00:05:56,619 --> 00:05:59,949 thing they're want- going to want to do is, try to harvest credentials. 104 00:05:59,959 --> 00:06:04,279 Is that common in playbooks or maybe, 90% of the playbooks out there? 105 00:06:06,902 --> 00:06:09,762 And I'm trying hard, I'm trying hard, Prasanna, not to say it. 106 00:06:10,111 --> 00:06:10,571 Just say it. 107 00:06:10,601 --> 00:06:12,151 Every episode, that's my goal 108 00:06:12,272 --> 00:06:15,242 d- I'm gonna, I'm gonna take a different approach. 109 00:06:15,292 --> 00:06:18,042 their first activity, their first phase is reconnaissance. 110 00:06:18,052 --> 00:06:19,512 So what do I have access to? 111 00:06:19,512 --> 00:06:21,072 What's the value of this thing? 112 00:06:21,852 --> 00:06:25,472 if they stole a laptop and it's just grandma's cookie recipe, yeah, maybe 113 00:06:25,472 --> 00:06:28,472 there's value there, but probably not gonna spend a whole lot of time. 114 00:06:28,472 --> 00:06:29,732 I may even give it back. 115 00:06:30,202 --> 00:06:33,892 but if it turns out to be a company machine, especially one that's got, 116 00:06:34,002 --> 00:06:37,632 you know, credit union stickers all over it, yeah, I'm gonna spend 117 00:06:37,632 --> 00:06:40,032 time looking for credentials. 118 00:06:40,162 --> 00:06:43,422 but I'm also gonna spend time locally on the machine to see if there's 119 00:06:43,422 --> 00:06:49,042 any, member or account information, identities that I could also harvest. 120 00:06:49,052 --> 00:06:50,122 But, credentials for sure. 121 00:06:50,142 --> 00:06:54,752 'Cause there's two threads of concurrent thought, is what do I have access to 122 00:06:54,782 --> 00:06:56,542 And, how do I maintain access to it? 123 00:06:57,982 --> 00:07:02,662 So this machine, even though I've got physical control over it, first time 124 00:07:02,662 --> 00:07:06,962 I connect it to the internet, or if it's got a, a, a SIM card in it, they 125 00:07:06,962 --> 00:07:08,542 may be able to connect to it anyway. 126 00:07:09,182 --> 00:07:12,802 but while I'm looking at it, they could remote wipe it lock me out of it. 127 00:07:12,882 --> 00:07:16,362 So absolutely, while I'm trying to figure out what I've got, I'm gonna 128 00:07:16,362 --> 00:07:19,932 try and figure out how to keep, maintain access to it, or persistence 129 00:07:21,012 --> 00:07:25,102 and also, like you talked about with the, seeing w- where else can I put 130 00:07:25,102 --> 00:07:31,052 malware to steal other credentials to create new, avenues of a- of access. 131 00:07:31,062 --> 00:07:33,292 That's, that sound about correct? 132 00:07:34,690 --> 00:07:35,220 It does. 133 00:07:35,260 --> 00:07:40,740 And part of that, what do I have access to And, maintain that persistence 134 00:07:40,750 --> 00:07:45,940 could be someone else's laptop or some other system in this not only to 135 00:07:45,940 --> 00:07:48,170 this machine, but to this opportunity. 136 00:07:49,170 --> 00:07:53,780 and so the scenario that I explained earlier, bad guy found out really 137 00:07:53,780 --> 00:07:56,680 quickly that this, this person simply works at a credit union. 138 00:07:56,680 --> 00:08:03,580 They're not an admin, but they are involved in processes, approving 139 00:08:03,580 --> 00:08:05,920 something, submitting something. 140 00:08:06,580 --> 00:08:10,030 And so as they got to know the organization's processes, we call 141 00:08:10,030 --> 00:08:14,810 that business process compromise, they were able to legitimize, an internal 142 00:08:14,810 --> 00:08:18,610 phishing email with an attachment that would likely get someone to interact 143 00:08:18,610 --> 00:08:22,930 with it, open it with malware that would then create access to another 144 00:08:22,930 --> 00:08:27,690 device, as an alternate persistence, or even just expanding their footprint, 145 00:08:27,700 --> 00:08:29,630 but definitely additional persistence 146 00:08:31,098 --> 00:08:38,948 And, so let's move from there into this idea of the danger of having 147 00:08:39,758 --> 00:08:45,418 credentials, especially, privileged credentials that, are sitting around 148 00:08:45,428 --> 00:08:47,088 all the time And don't change. 149 00:08:47,088 --> 00:08:48,098 Do you wanna talk about that? 150 00:08:50,168 --> 00:08:52,828 Ooh, I love talking about that because that's one of the 151 00:08:52,828 --> 00:08:54,238 things that bad guys realize. 152 00:08:54,278 --> 00:08:56,448 we're all human, and we're all constrained. 153 00:08:56,798 --> 00:09:00,578 we've gotta… Even if you have a lot of different, IT people, they're 154 00:09:00,618 --> 00:09:02,508 often putting out fires every day. 155 00:09:02,938 --> 00:09:05,178 Very rarely do they get to sit in their desk and go, 156 00:09:05,178 --> 00:09:07,088 what did I forget to do today? 157 00:09:07,538 --> 00:09:11,628 What, what's that best practice thing that hasn't been done in a while?" it's 158 00:09:11,628 --> 00:09:15,408 rare that they're able to do a lot of this maintenance stuff, in light of 159 00:09:15,458 --> 00:09:19,088 everyday operational, tactical things. 160 00:09:19,828 --> 00:09:21,448 so bad guys know that. 161 00:09:21,458 --> 00:09:26,328 So bad guys know that when I install some enterprise tool, whether it's, like 162 00:09:26,328 --> 00:09:31,528 a remote management tool or a backup software or even security, one of, real 163 00:09:31,528 --> 00:09:35,828 quick reference here or thought here is, we did a red team against an organization 164 00:09:35,828 --> 00:09:39,528 and actually used the service account for a security product to gain the 165 00:09:39,528 --> 00:09:41,118 administrative access that we needed. 166 00:09:41,558 --> 00:09:42,748 so there's no immunity. 167 00:09:42,748 --> 00:09:45,948 In fact, I published a zero today on CrowdStrike, a proof of concept 168 00:09:45,958 --> 00:09:51,118 that went public, this morning called Falcon Flank, where the service 169 00:09:51,118 --> 00:09:55,508 account, related to CrowdStrike can actually provide escalated privileges. 170 00:09:56,368 --> 00:09:57,968 so it continues to happen. 171 00:09:57,968 --> 00:09:58,808 No one's immune. 172 00:09:59,298 --> 00:10:05,278 Um, but the idea is, again, one- once I have access to a system, I'm gonna 173 00:10:05,278 --> 00:10:08,638 do that reconnaissance, so that's what service accounts are running, what, 174 00:10:08,768 --> 00:10:12,958 what applications are here, especially older ones that aren't patched, what, 175 00:10:13,018 --> 00:10:15,808 user accounts, both local and network. 176 00:10:16,718 --> 00:10:19,958 And then how do I, how does this machine authenticate to other devices? 177 00:10:20,348 --> 00:10:20,758 Is that? 178 00:10:20,808 --> 00:10:22,548 a, a local active directory? 179 00:10:22,558 --> 00:10:26,518 is it Office three sixty-five, Azure environment, What is it? 180 00:10:26,878 --> 00:10:27,508 Entra? 181 00:10:28,658 --> 00:10:32,588 and so all of those things are gonna help me strategize on 182 00:10:33,008 --> 00:10:36,138 how to expand my footprint and my access in this environment 183 00:10:36,757 --> 00:10:40,217 And normally, Mike, once like in your example, right? 184 00:10:40,217 --> 00:10:44,647 Once that credit union laptop went missing, I'm guessing the bad actors 185 00:10:44,647 --> 00:10:47,737 had to act really quickly because once it gets noticed, they're 186 00:10:47,737 --> 00:10:50,157 probably gonna start shutting down access and everything else, right? 187 00:10:50,597 --> 00:10:57,597 And so how long is it normally between like laptop stolen and they start 188 00:10:57,607 --> 00:11:00,647 doing all of these like harvesting credentials and other aspects? 189 00:11:00,917 --> 00:11:01,397 I'll say it depends. 190 00:11:02,717 --> 00:11:04,837 It depends on the culture of the organization. 191 00:11:04,847 --> 00:11:08,587 Does that, does the employee that lost that laptop, go update their resume 192 00:11:08,587 --> 00:11:11,417 and clean out their desk before they tell someone they lost their laptop? 193 00:11:12,557 --> 00:11:15,227 'cause they're afraid that, don't bother coming back to work, and we'll 194 00:11:15,227 --> 00:11:20,027 ship your belongings to your house." so that, that sense of fear is gonna, 195 00:11:20,407 --> 00:11:22,537 impact the timeliness of that report. 196 00:11:22,662 --> 00:11:23,182 oh, sorry. 197 00:11:23,182 --> 00:11:25,272 I meant more from the bad guy perspective. 198 00:11:26,157 --> 00:11:26,507 I'm getting 199 00:11:26,572 --> 00:11:26,892 Okay. 200 00:11:31,815 --> 00:11:32,055 உம் 201 00:11:35,187 --> 00:11:36,767 then, all right, so I report it. 202 00:11:36,787 --> 00:11:39,427 Let's say I report it at the moment I knew it was gone, which 203 00:11:39,427 --> 00:11:40,877 in this case from a coffee shop. 204 00:11:40,907 --> 00:11:42,477 they turned around and it was gone. 205 00:11:43,477 --> 00:11:46,567 that should be within like minutes that they're calling somebody. 206 00:11:47,307 --> 00:11:50,187 they're probably gonna call their boss first. 207 00:11:50,517 --> 00:11:54,527 The boss, whether that's an IT person or some- maybe they wear the IT 208 00:11:54,537 --> 00:11:56,437 hat also in a small organization. 209 00:11:57,107 --> 00:11:59,937 right, now the IT person's gotta figure out, what do I need to do? 210 00:12:00,347 --> 00:12:04,347 Where's the playbook for how to respond to a stolen laptop?" Very few 211 00:12:04,367 --> 00:12:08,927 organizations have that, but common sense would say, "All right, I'm gonna handle 212 00:12:08,937 --> 00:12:11,887 that on this side, to the employee. 213 00:12:11,897 --> 00:12:16,797 You call the police and, 'cause we need a police report." And on the IT side, I'm 214 00:12:16,797 --> 00:12:20,987 gonna go look at, all right, so I need to revoke, I need to change their password. 215 00:12:21,817 --> 00:12:24,837 I need to find out what they had access to. 216 00:12:25,637 --> 00:12:28,077 I need to find out if that machine is still connected. 217 00:12:28,157 --> 00:12:32,407 I need to figure out how to revoke sessions related to that user and 218 00:12:32,407 --> 00:12:38,127 that device, revoke MFA tokens, do an inventory of, log, essentially 219 00:12:38,127 --> 00:12:41,567 review the logs to determine what that device has been looking at. 220 00:12:41,607 --> 00:12:43,267 Is it SharePoint files? 221 00:12:43,267 --> 00:12:45,007 Is it local files? 222 00:12:46,147 --> 00:12:50,277 All of that, e- even if they knew how to do it, that's 20 minutes 223 00:12:52,327 --> 00:12:55,177 But my experience, especially with small credit unions, 224 00:12:55,427 --> 00:12:56,617 they're having to call somebody. 225 00:12:56,627 --> 00:12:58,717 They're calling their IT support vendor, they're calling their 226 00:12:58,717 --> 00:13:02,717 security vendor, they're calling their brother, "Hey, this just happened. 227 00:13:02,717 --> 00:13:05,717 What do I need to do?" Or they're googling it, they're having to walk 228 00:13:05,717 --> 00:13:10,357 through someone else's process because they haven't documented that playbook. 229 00:13:11,037 --> 00:13:15,427 So long story short, or long res- long response, long-winded response, but 230 00:13:15,427 --> 00:13:17,117 it- they've probably got 30 minutes 231 00:13:19,269 --> 00:13:19,909 Interesting. 232 00:13:19,959 --> 00:13:25,189 and so there's a couple of things in there that I picked up, and one is, just 233 00:13:25,189 --> 00:13:27,929 go back to that, so one of the things that I've heard is this concept of 234 00:13:27,929 --> 00:13:29,989 having a cybersecurity mindset, right? 235 00:13:30,019 --> 00:13:32,609 A cybersecurity culture within a company. 236 00:13:32,999 --> 00:13:40,809 And you can't have a, an environment where if you make a single mistake, 237 00:13:40,809 --> 00:13:44,029 if you respond to a single phishing email, if you do whatever, then that 238 00:13:44,029 --> 00:13:45,759 immediately means you get terminated. 239 00:13:46,019 --> 00:13:51,819 Because all that does is it people to not report the actual, 240 00:13:51,929 --> 00:13:53,429 events when they happen, right? 241 00:13:53,429 --> 00:13:57,259 Th- this actual… you want them to, when something bad happens, you 242 00:13:57,259 --> 00:14:00,639 want them to be able to, to respond, to report the missing laptop. 243 00:14:01,039 --> 00:14:07,519 And then the other thing that I heard in there was the need to that you have 244 00:14:07,519 --> 00:14:09,219 documented your incident response. 245 00:14:10,029 --> 00:14:15,089 I'm guessing that a lot of that could be automated with the pushing 246 00:14:15,089 --> 00:14:18,669 of a single big red button that helps to automate a lot of this. 247 00:14:18,689 --> 00:14:20,509 Can you- agree there? 248 00:14:22,273 --> 00:14:23,203 Yeah, you can script. 249 00:14:23,213 --> 00:14:27,333 So that's called security orchestration automation. 250 00:14:28,093 --> 00:14:30,693 I guess the, the term is SOAR, so it stands for security 251 00:14:30,693 --> 00:14:32,433 orchestrated and automated response. 252 00:14:33,483 --> 00:14:37,823 tying together the different systems that, that, that can play together and 253 00:14:37,873 --> 00:14:43,083 creating scripted responses to known or defined incidents like a stolen 254 00:14:43,083 --> 00:14:46,293 laptop or, revoking user privileges 255 00:14:47,541 --> 00:14:52,591 And you could do that without actually a SOAR tool to t- so to speak, right? 256 00:14:52,631 --> 00:14:54,011 'cause that is a type of tool. 257 00:14:54,271 --> 00:14:57,111 But you could document things and you could start automating some 258 00:14:57,111 --> 00:15:00,901 things without having to go out and purchase a very expensive SOAR tool. 259 00:15:01,681 --> 00:15:02,101 another 260 00:15:02,311 --> 00:15:05,931 PowerShell are both, both very powerful in a Microsoft environment 261 00:15:06,229 --> 00:15:06,579 Right. 262 00:15:06,699 --> 00:15:12,139 and then another thing that I thought about when, and this just goes back 263 00:15:12,139 --> 00:15:20,019 to something that comes up a lot here, is if what we have is a local account 264 00:15:20,219 --> 00:15:26,329 on each machine, Whether we're talking laptop or server, and that we share 265 00:15:26,359 --> 00:15:31,819 passwords between those local admin accounts, that's gonna significantly 266 00:15:31,819 --> 00:15:33,959 increase the risk of a stolen, device 267 00:15:35,439 --> 00:15:35,899 It does. 268 00:15:35,899 --> 00:15:39,729 and that's one of our first tactics when we do red teams on organizations. 269 00:15:39,999 --> 00:15:44,559 we could spend a week trying to hack you from, from the hotel, but it takes 270 00:15:44,569 --> 00:15:49,009 30 minutes to walk in, to piggyback or find your way into an environment 271 00:15:49,039 --> 00:15:51,879 and just take the first laptop and go out to the parking lot, dump all the 272 00:15:51,879 --> 00:15:53,849 hashes, put it back before anybody knows. 273 00:15:54,339 --> 00:15:57,649 And by the end of the day, you've got every local account on that machine. 274 00:15:58,769 --> 00:16:02,159 and that local admin is the same, very often, the same 275 00:16:02,159 --> 00:16:03,949 credentials on every machine. 276 00:16:05,549 --> 00:16:05,709 We 277 00:16:05,817 --> 00:16:11,487 That's coincidental credentials All 278 00:16:11,658 --> 00:16:16,708 Is that because people figure that no one's ever really going to use 279 00:16:16,708 --> 00:16:20,338 the local admin account or because it's used so infrequently and they're 280 00:16:20,338 --> 00:16:22,898 managing so many different devices, they're like, "Hey, let's just use 281 00:16:22,898 --> 00:16:26,608 the same thing across everything rather than a unique per device one"? 282 00:16:28,459 --> 00:16:30,849 Yeah, it's, it really comes down to logistics and resources. 283 00:16:30,849 --> 00:16:36,249 if you have 500 machines and it's 500 different passwords, now I've gotta buy a 284 00:16:36,249 --> 00:16:39,999 tool to manage that, even though there's some automated tools where, hey, I've 285 00:16:39,999 --> 00:16:42,079 gotta go work on machine number one. 286 00:16:42,359 --> 00:16:44,619 You click a button, it tells you the password. 287 00:16:44,919 --> 00:16:48,869 You go work on the machine, that password expires, the tool automatically reset, 288 00:16:48,869 --> 00:16:53,589 regenerates, resets a, or generates a, a new password, and it's all maintained 289 00:16:53,589 --> 00:16:55,439 in the, the password management system. 290 00:16:56,109 --> 00:17:00,449 but no, n-nine and a half times out of 10, and I'm only doing that because of 291 00:17:00,449 --> 00:17:03,019 averages, organizations don't do that 292 00:17:03,955 --> 00:17:04,445 Yeah. 293 00:17:04,665 --> 00:17:06,715 it's expensive, it's difficult. 294 00:17:06,795 --> 00:17:11,075 It's the right thing to do, but it takes time, there's cost, there's management 295 00:17:11,075 --> 00:17:15,345 overhead, and then there's, the idea that, which is something we're gonna talk about 296 00:17:15,345 --> 00:17:16,985 here, is that you need to maintain those. 297 00:17:16,995 --> 00:17:21,155 You can't just put in an account, keep a password, leave it there forever, 298 00:17:21,355 --> 00:17:23,005 especially when it's a privileged account. 299 00:17:23,277 --> 00:17:29,691 A related idea, I've also worked at a company where every new user account, 300 00:17:32,201 --> 00:17:35,701 is a very predictable pattern, right? 301 00:17:36,521 --> 00:17:38,801 like something like that. 302 00:17:39,161 --> 00:17:41,551 And then they don't f- oh, wait, we put the… You know 303 00:17:41,551 --> 00:17:42,591 what we put at the end, right? 304 00:17:42,591 --> 00:17:43,541 What did we put at the end? 305 00:17:43,833 --> 00:17:44,233 Bang 306 00:17:44,574 --> 00:17:45,374 Exclamation point 307 00:17:45,381 --> 00:17:47,641 Exclamation point is exactly what they did. 308 00:17:47,921 --> 00:17:49,541 So nobody will ever guess that. 309 00:17:49,991 --> 00:17:55,011 but the… And as a result, it, you… And oh, and also tied to 310 00:17:55,011 --> 00:18:00,041 that, they did not force you to then immediately change your password 311 00:18:00,051 --> 00:18:00,251 Oh. 312 00:18:00,251 --> 00:18:00,911 no 313 00:18:01,188 --> 00:18:01,488 time. 314 00:18:01,638 --> 00:18:01,988 Yeah. 315 00:18:02,018 --> 00:18:02,998 So not only… 316 00:18:03,058 --> 00:18:03,248 it 317 00:18:04,148 --> 00:18:04,528 yeah. 318 00:18:04,528 --> 00:18:09,448 Oh, yeah, I can rem- I can remember this password, my initials, my OS, and the 319 00:18:09,448 --> 00:18:11,948 year I joined, and an exclamation point. 320 00:18:12,858 --> 00:18:17,388 that also meant I could, know, easily guess, many other people's, passwords. 321 00:18:17,388 --> 00:18:18,478 easily, exactly. 322 00:18:19,438 --> 00:18:20,078 Oh, yeah. 323 00:18:20,338 --> 00:18:23,768 I know we've been talking about, harvesting credentials, local credentials, 324 00:18:23,768 --> 00:18:25,398 remote credentials, all the rest. 325 00:18:25,398 --> 00:18:30,058 When we talk about harvest credentials in this context, are we talking about 326 00:18:30,398 --> 00:18:34,338 credentials at the organization like Mike mentioned, they discover that 327 00:18:34,338 --> 00:18:38,348 the person is responsible for work- workflow automation, so they sent out 328 00:18:38,348 --> 00:18:43,868 a bunch of attachments and got, r- harvested credentials from other users? 329 00:18:44,118 --> 00:18:47,638 Or is this mainly harvesting the credentials of that local system? 330 00:18:48,338 --> 00:18:51,998 Or just wanna make sure that we're clear on what we mean 331 00:18:52,173 --> 00:18:54,013 everything they can possibly get their hands on. 332 00:18:54,063 --> 00:18:56,663 even credentials in the browser, like for your bank or your 333 00:18:57,032 --> 00:18:57,262 Okay 334 00:18:57,293 --> 00:18:57,913 gonna get all that 335 00:18:58,562 --> 00:18:58,692 yeah 336 00:18:58,692 --> 00:18:59,842 I was gonna say yes. 337 00:19:00,450 --> 00:19:01,020 Yes to all. 338 00:19:01,140 --> 00:19:01,440 Okay. 339 00:19:01,752 --> 00:19:08,212 and so this is another example of when you don't do, proper account maintenance. 340 00:19:08,262 --> 00:19:11,632 this is another example of another way that they can, that 341 00:19:11,632 --> 00:19:13,272 they can harvest credentials. 342 00:19:13,272 --> 00:19:15,462 You, And this is via Active Directory, right? 343 00:19:18,276 --> 00:19:19,626 So you wanna talk about that Kerber? 344 00:19:19,956 --> 00:19:21,356 Sorry, carbon roasting. 345 00:19:21,426 --> 00:19:21,696 Yeah 346 00:19:21,701 --> 00:19:24,841 Kerberoasting specifically I'm saying, is via Active Directory 347 00:19:25,895 --> 00:19:25,915 Yeah 348 00:19:25,915 --> 00:19:28,655 that's how, that's an active directory authentication mechanism. 349 00:19:28,975 --> 00:19:29,705 You want to talk about that? 350 00:19:31,195 --> 00:19:31,585 Sure. 351 00:19:31,895 --> 00:19:36,515 So when, whenever you log in, you're submitting a request to an authority, 352 00:19:36,515 --> 00:19:38,205 so that's usually a domain controller. 353 00:19:38,895 --> 00:19:43,685 your request is matched against, the, the known authorization. 354 00:19:43,685 --> 00:19:47,675 So is that your username, your username and your password are good. 355 00:19:48,025 --> 00:19:50,895 now that I know you're authenticated user, what approvals do you have? 356 00:19:50,895 --> 00:19:52,075 What authority do you have? 357 00:19:52,645 --> 00:19:56,965 and so the combination of being authenticated and your authority get put 358 00:19:56,965 --> 00:20:01,135 into a ticket called a Kerberos ticket, and that's sent back to your machine. 359 00:20:01,485 --> 00:20:05,395 And now every time your machine makes a request in the environment, "Hey, I wanna 360 00:20:05,395 --> 00:20:09,635 look at that file or access that server," you've got that ticket so that you're 361 00:20:09,635 --> 00:20:11,275 not having to reauthenticate every time. 362 00:20:12,815 --> 00:20:15,815 One of the first things bad guys are gonna do in that reconnaissance phase 363 00:20:15,815 --> 00:20:17,595 is look for all the service accounts. 364 00:20:18,455 --> 00:20:19,945 So your named service accounts. 365 00:20:21,075 --> 00:20:24,715 'cause those are the ones that usually have embedded credentials 366 00:20:24,722 --> 00:20:26,512 What's a named service account, by the way? 367 00:20:29,493 --> 00:20:30,633 Like your backup… You mean the 368 00:20:30,640 --> 00:20:30,950 Okay. 369 00:20:31,093 --> 00:20:31,703 acronym? 370 00:20:31,710 --> 00:20:31,920 no. 371 00:20:31,920 --> 00:20:32,340 Yeah, like 372 00:20:32,390 --> 00:20:32,490 yeah, 373 00:20:32,727 --> 00:20:33,647 what is a name service account? 374 00:20:33,647 --> 00:20:33,827 Yeah. 375 00:20:34,360 --> 00:20:34,720 yeah, like 376 00:20:34,780 --> 00:20:34,960 Okay 377 00:20:35,073 --> 00:20:35,753 backup account. 378 00:20:36,873 --> 00:20:42,693 a Windows service account like, remote registry or something like that. 379 00:20:42,693 --> 00:20:47,403 A- and so in, in Kerberoasting, if I'm on your net- and this has to, you have 380 00:20:47,403 --> 00:20:50,393 to be on the network, on the wire, 381 00:20:53,203 --> 00:20:55,113 to conduct Kerberoasting, attack. 382 00:20:55,113 --> 00:20:59,493 So essentially you're a man in the middle on the network between someone 383 00:20:59,493 --> 00:21:05,183 submitting a Kerberos ticket or requesting one, and whatever the other end is. 384 00:21:05,213 --> 00:21:08,623 And so you're essentially, harvesting Kerberos tickets. 385 00:21:09,013 --> 00:21:13,683 and you can do that simply by, sniffing or monitoring the network. 386 00:21:14,503 --> 00:21:17,643 you can do that on a, a, a, a span port on a router. 387 00:21:17,963 --> 00:21:21,763 you can be on a machine on a core segment just running a, a network protocol 388 00:21:21,763 --> 00:21:26,713 analyzer like Wireshark, or EtherReals, what it used to be back in the day. 389 00:21:27,933 --> 00:21:32,183 and you would just capture packets and you can actually Google, what is a, what 390 00:21:32,183 --> 00:21:34,583 does a Kerberos ticket packet look like? 391 00:21:34,603 --> 00:21:38,193 And so now can search or filter on all the packets you captured for the 392 00:21:38,193 --> 00:21:41,853 ticket, then you simply take that offline and run it through tools to 393 00:21:41,853 --> 00:21:46,193 crack it, embedded in that ticket are the credentials for that account 394 00:21:47,732 --> 00:21:51,092 I'm surprised it's not encrypted over the wire. 395 00:21:52,251 --> 00:21:52,801 It is. 396 00:21:52,942 --> 00:21:53,412 It is? 397 00:21:54,181 --> 00:21:54,891 Kerberos tickets? 398 00:21:54,891 --> 00:21:55,301 Yes. 399 00:21:55,362 --> 00:21:55,742 Okay. 400 00:21:55,791 --> 00:21:57,421 That's, so you take it offline and you 401 00:21:57,512 --> 00:21:57,782 Okay 402 00:21:57,911 --> 00:21:58,901 your tools. 403 00:21:59,601 --> 00:22:02,401 similar to if I was to collect all the local, 404 00:22:03,421 --> 00:22:04,061 Band track it. 405 00:22:04,201 --> 00:22:04,481 Yeah 406 00:22:04,522 --> 00:22:08,002 the machine, I would take those offline and run it against my tools 407 00:22:09,172 --> 00:22:10,702 to get those, those credentials. 408 00:22:11,504 --> 00:22:17,584 this is a post, this is a post-compromise attack, They're already in your network, 409 00:22:17,624 --> 00:22:23,944 they have access, and this is an example of else they can do to grab additional, 410 00:22:24,044 --> 00:22:27,254 especially privileged account passwords. 411 00:22:28,764 --> 00:22:33,604 It also, depending on, if they have access to, a device with a privileged, 412 00:22:33,774 --> 00:22:39,524 user already logged in and that, that authentication happened at the 413 00:22:39,524 --> 00:22:43,044 keyboard, then a lot of times those credentials are also in memory. 414 00:22:43,094 --> 00:22:48,044 And so there's a whole other… are so many layers to harvesting credentials. 415 00:22:48,744 --> 00:22:52,964 it's not just someone's running malware that's gonna capture my keystrokes. 416 00:22:53,004 --> 00:22:54,554 there's a lot of different ways to do it 417 00:22:56,445 --> 00:23:01,135 And I don't know if we'll talk about this later, but Mike, is there 418 00:23:01,165 --> 00:23:03,465 a way to prevent kerberoasting? 419 00:23:03,625 --> 00:23:06,835 of someone doing a man in the middle or dumping… is there 420 00:23:06,875 --> 00:23:09,185 a better way to secure that? 421 00:23:11,430 --> 00:23:12,060 It's tough. 422 00:23:12,170 --> 00:23:15,420 if they're already in your network, then really your hope is 423 00:23:15,420 --> 00:23:16,990 to identify where they are and, 424 00:23:17,613 --> 00:23:18,183 Shut them down? 425 00:23:18,490 --> 00:23:19,270 that device. 426 00:23:20,000 --> 00:23:25,230 sniffing, capturing network traffic, there's no way you can really prevent that 427 00:23:25,230 --> 00:23:28,050 from a device that's on your net- that's already authenticated to your network 428 00:23:28,801 --> 00:23:31,651 But isn't it, isn't this go back to 429 00:23:33,833 --> 00:23:41,193 When they do that, they're only gonna get the credential that they can sniff, right? 430 00:23:41,463 --> 00:23:46,373 and then one that was able to, to be able to be brute force attacked. 431 00:23:47,463 --> 00:23:53,313 The, the risk then is if that credential is the same as many other credentials 432 00:23:53,313 --> 00:23:57,933 in the network, then it's gonna be more valuable, So it goes back to 433 00:23:57,933 --> 00:24:01,343 the of making sure that you're… 434 00:24:01,374 --> 00:24:02,554 a default credential. 435 00:24:03,043 --> 00:24:03,243 What's that? 436 00:24:04,664 --> 00:24:05,974 Or it's a default credential. 437 00:24:06,969 --> 00:24:07,659 There's that word again 438 00:24:07,904 --> 00:24:11,244 a lot of times even something as simple as your SNMP, community 439 00:24:11,244 --> 00:24:15,134 string is either blank, generic, or it's the name of your company. 440 00:24:15,424 --> 00:24:18,634 A lot of people are like, "It doesn't matter. It's something simple." They 441 00:24:18,634 --> 00:24:21,724 don't realize what bad guys can do with just something simple like that. 442 00:24:22,644 --> 00:24:28,124 quite similarly, I think a lot of organizations think either that password 443 00:24:28,134 --> 00:24:32,534 doesn't need to be complex or s- strong because it's this simple thing. 444 00:24:32,964 --> 00:24:36,094 But that simple thing is a service account that's running on a lot of different… 445 00:24:36,344 --> 00:24:39,034 maybe it's just a print server. 446 00:24:39,594 --> 00:24:43,954 and I'll tell you, for years I've, I have aligned with the perspective that print 447 00:24:43,954 --> 00:24:46,524 servers and print devices are not a risk. 448 00:24:47,284 --> 00:24:50,344 just recently, and I'm actually working on some research now of a threat 449 00:24:50,344 --> 00:24:54,804 vector where the main component is unprotected printers, and it actually 450 00:24:54,804 --> 00:25:00,044 escalates quite significantly and, can result in a complete network takeover 451 00:25:00,521 --> 00:25:00,741 Yeah 452 00:25:00,924 --> 00:25:01,704 of a printer. 453 00:25:03,054 --> 00:25:08,274 But simple things, a lot of… it's people that build networks and configure 454 00:25:08,274 --> 00:25:13,684 devices that aren't as, current or familiar with cybersecurity threats and 455 00:25:13,694 --> 00:25:18,354 bad guy capabilities results in these, these environments that could have been 456 00:25:18,354 --> 00:25:22,424 better if just a little more thought and effort had put in, been put into it 457 00:25:22,511 --> 00:25:26,571 And Mike, I wanna touch on that point because oftentimes I've seen 458 00:25:26,581 --> 00:25:30,511 organizations where it's like, "Hey, here's this small baby project. 459 00:25:30,511 --> 00:25:31,701 We're just gonna kick the tires. 460 00:25:31,701 --> 00:25:34,211 It might be a stealth mode thing or something else like that. 461 00:25:34,221 --> 00:25:36,621 we are not sure if it's gonna be successful," right? 462 00:25:36,651 --> 00:25:39,521 And once they prove out that success and it's like, "Okay, now let's just roll 463 00:25:39,521 --> 00:25:42,531 this out broadly, let's do this more broadly," but they don't necessarily 464 00:25:42,531 --> 00:25:45,601 have the ability to go back and rethink a lot of this architecture and 465 00:25:45,601 --> 00:25:47,111 put in place all of these controls. 466 00:25:47,111 --> 00:25:50,161 And of course, you have the executive pressure be like, "Yep, let's ship 467 00:25:50,161 --> 00:25:53,231 this," and you have mandates and timelines and costs and everything else. 468 00:25:53,731 --> 00:25:56,691 And it's always hard to go back and fix like the tech debt or take the 469 00:25:56,701 --> 00:25:58,801 time to properly plan things out. 470 00:25:59,661 --> 00:26:03,241 And so I wonder if that's also some of the issues that you're pointing out, where 471 00:26:03,241 --> 00:26:08,461 it's yeah, maybe they don't have the time or the resources to do all of that upfront 472 00:26:09,392 --> 00:26:13,402 That's a great story to tell, and I'm sure most of the IT 473 00:26:13,422 --> 00:26:15,502 guys would go, "Exactly that. 474 00:26:15,512 --> 00:26:19,592 Let's use that story." 'Cause really what happened was they did 475 00:26:19,592 --> 00:26:22,242 a proof of concept and it worked, and then they went, "All right. 476 00:26:22,242 --> 00:26:23,582 Let's roll this into production. 477 00:26:23,892 --> 00:26:25,862 Do we need to change stuff?" And they're like, "No, it's 478 00:26:26,005 --> 00:26:26,295 Okay, we're good 479 00:26:27,222 --> 00:26:28,032 don't touch it. 480 00:26:28,932 --> 00:26:33,452 a thing." we'll have to rework everything, and there goes my weekend and my vacation. 481 00:26:33,452 --> 00:26:36,772 And no, I think most of the time they don't do- they don't make changes, 482 00:26:36,772 --> 00:26:40,842 especially from test to production, because it's a lot more work 483 00:26:41,615 --> 00:26:41,745 Yeah. 484 00:26:41,772 --> 00:26:44,272 already working, and there's usually a deadline 485 00:26:44,515 --> 00:26:44,815 Yep. 486 00:26:44,872 --> 00:26:45,472 whatever else 487 00:26:46,071 --> 00:26:50,351 it's not a cyber, story, but, Prasanna, that does remind me of the Brocade 488 00:26:50,351 --> 00:26:52,351 story that I know I've told you before. 489 00:26:52,823 --> 00:26:53,293 Oh, yes 490 00:26:53,381 --> 00:26:58,111 I, I, for those that, are new to the industry, at one point Brocade 491 00:26:58,121 --> 00:27:01,571 was like the, the king of the SANs, the fiber channel, right? 492 00:27:02,071 --> 00:27:03,731 and before that they weren't, right? 493 00:27:03,731 --> 00:27:08,901 They were a, a small little company and I got to hear the story from the founder 494 00:27:08,921 --> 00:27:13,411 of, Brocade, or one of the founders, Kumar Malavalli, he was talking about 495 00:27:13,461 --> 00:27:17,201 when they were in that initial phase and they were coming up with names for things. 496 00:27:18,391 --> 00:27:21,761 and he goes, "You gotta understand, like it was a room of a bunch of Indian 497 00:27:21,761 --> 00:27:26,251 guys, right? And we're trying to come up with a name for the project, not for 498 00:27:26,251 --> 00:27:30,211 the product, but for the project, the internal name that we were gonna use." 499 00:27:30,661 --> 00:27:35,501 By the way, for those that don't know, a, a SAN is called a fabric, right? 500 00:27:35,701 --> 00:27:40,551 a- and, and so he raised his hand and he says, how about we call it a, a Silkworm 501 00:27:40,581 --> 00:27:44,051 'cause they could pull a fabric out of their ass?" and they're… Everybody 502 00:27:44,051 --> 00:27:48,221 goes, "Yes, let's do that." And his marketing person, she wr- she goes, no. 503 00:27:48,251 --> 00:27:48,941 we can't do that. 504 00:27:48,941 --> 00:27:52,781 This story will get out," da, da. And, and he goes, "Oh no, this is just, this is… 505 00:27:52,821 --> 00:27:54,581 We're just using it for the project name. 506 00:27:54,971 --> 00:27:57,801 we're not gonna use it for the product name." And of course, for those that don't 507 00:27:57,801 --> 00:28:02,981 know, Silkworm became their entire product line, all going back to that one little 508 00:28:02,981 --> 00:28:04,941 guy who raised his hand in a meeting. 509 00:28:05,751 --> 00:28:07,381 but, so all right. 510 00:28:07,721 --> 00:28:12,241 all right, let's talk about the things that we can do. 511 00:28:12,591 --> 00:28:16,831 so the first thing that we talk about in the book, Mike, is using LAPS, , local 512 00:28:16,831 --> 00:28:19,531 administrator password solution. 513 00:28:19,661 --> 00:28:24,321 I, I know we, we talked about that and d- I don't think people use it very much. 514 00:28:24,321 --> 00:28:28,231 I think that most people use, so Mike, you wanna talk about that? 515 00:28:30,080 --> 00:28:31,080 Well, we touched on it earlier. 516 00:28:31,080 --> 00:28:36,120 it's essentially a tool that allows you to manage various, and unique 517 00:28:37,540 --> 00:28:41,760 privileged, credentials across your environment so that you don't have as 518 00:28:41,760 --> 00:28:43,410 many of these coincidental passwords. 519 00:28:44,160 --> 00:28:47,920 I think at a minimum, one, we're an organization at least, and it was a 520 00:28:47,920 --> 00:28:52,190 movie theater organization, we worked with them to develop a strategy where 521 00:28:52,190 --> 00:28:55,870 they're like, "There's absolutely no way," and they have locations all over 522 00:28:55,950 --> 00:29:02,490 the world, that they're gonna be able to manage unique credentials by device. 523 00:29:02,660 --> 00:29:03,600 they have thousands of… 524 00:29:04,041 --> 00:29:04,271 Right 525 00:29:04,530 --> 00:29:05,080 I said, all right. 526 00:29:05,120 --> 00:29:10,790 what about by location?" with us, that's probably doable." Like Dallas would have 527 00:29:10,790 --> 00:29:12,920 different credentials than Las Vegas. 528 00:29:13,540 --> 00:29:13,650 right. 529 00:29:13,650 --> 00:29:17,200 what about by segment, like network segment?" we don't have good network 530 00:29:17,200 --> 00:29:22,790 segmentation, but we need to do that." what if we had a user segment and a 531 00:29:22,790 --> 00:29:26,520 production segment and a backup segment, and all those devices had different 532 00:29:27,310 --> 00:29:29,470 local and privileged credentials? 533 00:29:29,860 --> 00:29:33,250 And that's really where they settled, that they thought that was doable. 534 00:29:33,260 --> 00:29:38,430 But they still needed this tool, this, this, program to help them manage that. 535 00:29:38,870 --> 00:29:44,030 And the one that they picked, and it's been long time since w- we were working 536 00:29:44,030 --> 00:29:48,330 on this, so I can't remember what it was called, essentially you, once y- once 537 00:29:48,330 --> 00:29:52,680 you've got it deployed, and there would be an agent of this tool everywhere. 538 00:29:53,470 --> 00:29:57,180 Once you got it deployed, it already had an inventory of all your 539 00:29:57,990 --> 00:29:59,910 devices, the ones that it managed. 540 00:30:01,030 --> 00:30:05,520 you would go to the tool to check out the credentials for the device or segment, 541 00:30:05,550 --> 00:30:11,240 in this case, that you're working on, and it was good for a day or two days. 542 00:30:11,260 --> 00:30:14,020 However long you said, "I need those credentials for this long." 543 00:30:14,800 --> 00:30:17,840 when that time period expired, that tool automatically updated the 544 00:30:17,840 --> 00:30:19,480 credentials on, in that environment. 545 00:30:20,360 --> 00:30:22,620 That is an amazing approach to this problem. 546 00:30:23,401 --> 00:30:23,731 Yeah 547 00:30:23,840 --> 00:30:27,480 I don't remember the cost of the tool, but the cost of implementing 548 00:30:27,480 --> 00:30:36,500 it, across in this case, I think they had 150 geographic locations. 549 00:30:37,160 --> 00:30:42,040 and geographic would be like southern, the, the Southern US was one geographic 550 00:30:42,040 --> 00:30:44,590 location with 800 movie theaters. 551 00:30:44,920 --> 00:30:46,020 Something crazy like that. 552 00:30:46,830 --> 00:30:49,950 their deployment, their rollout, just from a labor perspective, was 553 00:30:49,950 --> 00:30:53,060 like $200,000, just for the labor. 554 00:30:54,320 --> 00:30:55,800 So it can be costly. 555 00:30:58,500 --> 00:31:00,860 you just have to understand your environment and build a 556 00:31:00,860 --> 00:31:02,490 strategy that, that makes sense. 557 00:31:02,925 --> 00:31:06,765 and you have to start with… That, that sounds like a lot of money, but 558 00:31:06,765 --> 00:31:11,155 for an organization of that size, a single cyber event is gonna cost a hell 559 00:31:11,155 --> 00:31:14,085 of a lot more than $200,000, right? 560 00:31:15,055 --> 00:31:16,245 that, I like that idea. 561 00:31:16,265 --> 00:31:18,975 I think that falls under something else we talked about. 562 00:31:18,975 --> 00:31:23,035 It's this idea of just-in-time credentials, I think is what falls 563 00:31:23,045 --> 00:31:24,375 under what you just described. 564 00:31:25,452 --> 00:31:25,612 Yep. 565 00:31:25,985 --> 00:31:26,335 yeah. 566 00:31:26,572 --> 00:31:26,812 So 567 00:31:26,885 --> 00:31:27,175 that. 568 00:31:27,292 --> 00:31:27,692 one stone 569 00:31:28,425 --> 00:31:29,415 Yeah, exactly. 570 00:31:30,645 --> 00:31:33,295 Mike, like what you were talking about with segments, I think one 571 00:31:33,295 --> 00:31:38,695 of the next things to, recommend is using tiered administration. 572 00:31:39,025 --> 00:31:44,945 So as an example, if you have workstations and servers and laptops 573 00:31:44,945 --> 00:31:49,875 and desktops, you don't need them all to have like sort of the same level 574 00:31:49,885 --> 00:31:55,365 of support or the same administrative access across everything that is there. 575 00:31:55,365 --> 00:31:59,145 And you could say, "Okay, these admins only need access to these." 576 00:31:59,805 --> 00:32:03,215 And so you're separating out your domains and the administration 577 00:32:03,215 --> 00:32:04,825 of those domains between those. 578 00:32:04,825 --> 00:32:05,945 So just like you talked about. 579 00:32:06,165 --> 00:32:08,645 And I think that's actually perfect for the network segmentation 580 00:32:08,645 --> 00:32:11,665 example you just talked about where you had your production, you had 581 00:32:11,665 --> 00:32:13,295 your end user devices, et cetera. 582 00:32:13,585 --> 00:32:15,865 Now you could also split those into domains, which I'm guessing 583 00:32:15,865 --> 00:32:20,815 is probably what they had also done at this movie theater company 584 00:32:21,995 --> 00:32:26,025 to have separate domain admins to manage different segments as well 585 00:32:27,558 --> 00:32:28,058 It did. 586 00:32:28,078 --> 00:32:31,578 So they had regional admins and then within each region they had 587 00:32:31,838 --> 00:32:34,008 desktop admins and server admins. 588 00:32:34,788 --> 00:32:38,948 And then they had a separate set of admins for non-production, so test dev. 589 00:32:39,618 --> 00:32:42,928 so the reason that they had to go to that extent was 'cause they're 590 00:32:42,928 --> 00:32:45,888 a publicly traded company, and they had to comply with Sarbanes-Oxley 591 00:32:45,888 --> 00:32:47,188 and demonstrate, all these 592 00:32:47,595 --> 00:32:48,065 Control 593 00:32:48,478 --> 00:32:49,208 governance. 594 00:32:50,938 --> 00:32:54,888 one of the things that was most important, so in addition to having layered, or 595 00:32:54,888 --> 00:32:59,538 even just different privileged accounts, so that any one admin doesn't have 596 00:32:59,598 --> 00:33:01,198 administrative access to everything. 597 00:33:02,068 --> 00:33:05,518 But one of the most important things is just making sure that your admins 598 00:33:05,568 --> 00:33:10,358 also have a normal user account, so they're not, surfing the web or checking 599 00:33:10,358 --> 00:33:13,778 their Gmail with their, while they're signed in with their admin account. 600 00:33:14,518 --> 00:33:17,248 And we saw that, we did work for a, a healthcare supply… 601 00:33:17,308 --> 00:33:20,268 In fact, the largest healthcare supply chain company in the world. 602 00:33:21,038 --> 00:33:23,918 Their… They had a database 603 00:33:26,388 --> 00:33:29,368 and she was, she had her development machine and her production 604 00:33:29,368 --> 00:33:30,978 machine both on the same desk. 605 00:33:32,018 --> 00:33:36,568 And in a, a busy, frazzled day, she thought she was deleting something 606 00:33:36,568 --> 00:33:39,178 from the test environment, and 607 00:33:39,193 --> 00:33:39,833 No. 608 00:33:39,868 --> 00:33:43,408 from the production environment because her credentials were the same as well. 609 00:33:43,428 --> 00:33:46,758 So whenever she logged into one, the screens would time out and her 610 00:33:46,758 --> 00:33:49,558 sessions would time out and, but the credentials were the same, so 611 00:33:49,558 --> 00:33:55,638 she didn't even realize logging into production with same credentials. 612 00:33:55,648 --> 00:33:58,008 if they were different, she'd have been like, "Why isn't this working?" 613 00:33:58,488 --> 00:33:58,788 Yeah 614 00:33:59,068 --> 00:34:01,468 'Cause you're trying to log into production with your testing credentials. 615 00:34:01,518 --> 00:34:05,888 But anyway, so that, that's a first step is create separate accounts 616 00:34:05,898 --> 00:34:10,618 for all your admins so that they're using user account all the time and 617 00:34:11,548 --> 00:34:13,838 account, named privileged account. 618 00:34:14,578 --> 00:34:16,068 Mike normal, Mike admin. 619 00:34:16,948 --> 00:34:19,638 We're not all using admin because there's the, there's an issue 620 00:34:19,638 --> 00:34:21,188 with repu- non-repudiation there. 621 00:34:21,188 --> 00:34:23,318 I can't tell who did… I probably 622 00:34:23,440 --> 00:34:23,720 Right 623 00:34:23,898 --> 00:34:27,208 some digging, but it's a lot easier to, to, to look at the 624 00:34:27,538 --> 00:34:31,838 activity log and Mi- and see that Mike admin did this versus admin. 625 00:34:31,848 --> 00:34:34,508 Now I gotta go track back who logged in as admin. 626 00:34:35,758 --> 00:34:40,748 but then abs- absolutely separating the different types of admins is also helpful 627 00:34:41,500 --> 00:34:43,050 also enforce that, right? 628 00:34:43,050 --> 00:34:48,830 So you first create a policy, and then you enforce it by things, 629 00:34:49,080 --> 00:34:55,700 auditing accounts, auditing access, and then seeing if the employees are 630 00:34:56,450 --> 00:34:59,470 following the policies that you created 631 00:35:00,481 --> 00:35:03,861 And then also using the SIEM and SOAR tools if you have them in your environment 632 00:35:03,871 --> 00:35:07,721 to automate your incident responses when you see things that don't happen. 633 00:35:07,840 --> 00:35:08,440 trigger 634 00:35:08,511 --> 00:35:08,681 Yeah 635 00:35:08,920 --> 00:35:11,360 one of, one of the things if you've got the, the visibility 636 00:35:11,360 --> 00:35:13,740 and the tools, set up alerts. 637 00:35:14,160 --> 00:35:17,240 back in the day that healthcare, supply chain company didn't 638 00:35:17,240 --> 00:35:19,340 have a lot of sexy, cool tools. 639 00:35:19,340 --> 00:35:23,030 They were just using like rudimentary stuff that's included. 640 00:35:23,090 --> 00:35:25,660 they set up a free Kiwi Syslog server. 641 00:35:25,970 --> 00:35:29,600 They would send all the, the Microsoft event logs to it, and then 642 00:35:29,600 --> 00:35:33,160 they had automated like i- if this event happens, send me an email. 643 00:35:33,580 --> 00:35:37,400 And that event, like there's an event ID for the creation of admin accounts. 644 00:35:38,080 --> 00:35:43,530 J- that should be triggered 100% everywhere, 'cause your admins are 645 00:35:43,530 --> 00:35:47,340 your admins, and if there's gonna be a new admin, an admin would know that. 646 00:35:47,750 --> 00:35:50,920 "I'm setting up a new admin today. There's a ticket, there's a request," whatever. 647 00:35:51,460 --> 00:35:54,230 But in the middle of the night when you get an email that says a new 648 00:35:54,590 --> 00:35:58,730 admin was created, that's probably a, n- almost 100% valid problem. 649 00:36:00,040 --> 00:36:03,570 So there's a lot you can do around authentication events, whether it's 650 00:36:03,570 --> 00:36:08,330 creation of account use during weird hours, the, changes that an, privileged 651 00:36:08,330 --> 00:36:10,271 account made, those should all be triggers 652 00:36:10,408 --> 00:36:16,518 If you're making moves in this direction, and we say this a lot, I think, with other 653 00:36:16,518 --> 00:36:20,788 things, is there will be sort of slip-ups. 654 00:36:20,878 --> 00:36:23,548 There will be false positives. 655 00:36:23,598 --> 00:36:28,078 There will be times where, if you start monitoring, you're like, "Hey, 656 00:36:28,108 --> 00:36:31,158 domain admin shouldn't be logging in at 2:00 in the morning." it turns out 657 00:36:31,158 --> 00:36:34,068 your domain admin actually does log in every day at 2:00 in the morning. 658 00:36:34,078 --> 00:36:37,918 They shouldn't be, but turns out they did, and so they're setting off alarms. 659 00:36:38,198 --> 00:36:42,978 And it doesn't matter, like, how simple the change that you're making. 660 00:36:42,988 --> 00:36:46,318 If you're going to increase logging and increase security, you will 661 00:36:46,318 --> 00:36:51,088 have some slip-ups, and you just need to be prepared for that. 662 00:36:51,138 --> 00:36:56,188 I'm, I'll just give… And again, it's not cyber related, but in my personal life, 663 00:36:56,348 --> 00:37:01,678 so I ha- I am definitely a, I have ADHD. 664 00:37:02,048 --> 00:37:06,648 And one of the things that, that, has happened a lot is, I leave 665 00:37:06,978 --> 00:37:11,408 my freezer refrigerator open, but more importantly my freezer. 666 00:37:12,718 --> 00:37:15,518 And the next thing I just throw out a bunch of food because, I left 667 00:37:15,518 --> 00:37:17,238 my freezer open for an entire day. 668 00:37:17,948 --> 00:37:24,858 And so I bought a bunch of, I bought some, monitors to, monitor my freezer 669 00:37:24,878 --> 00:37:30,658 and my fridge to tell me if the temperature is out of a particular bound. 670 00:37:30,888 --> 00:37:34,348 And I have an app that, you know, for those of you watching on the, on 671 00:37:34,358 --> 00:37:40,028 YouTube, I have an app on my phone that, that has, alerts to tell me, 672 00:37:40,138 --> 00:37:44,408 again, to trigger me for this cyber event of me leaving my freezer open. 673 00:37:44,698 --> 00:37:48,758 And what, and literally, I just put it in yesterday, and, I get this call from 674 00:37:48,758 --> 00:37:55,098 somebody at my house going, "Hey, there's this thing that's, beeping nonstop. Can 675 00:37:55,098 --> 00:37:59,388 you take a look?" And it was just that I hadn't properly tweaked, the, the 676 00:37:59,388 --> 00:38:01,728 parameters of what constituted an alarm. 677 00:38:02,008 --> 00:38:06,508 So what- whatever you do, whatever you make, just realize, I'd say that, 678 00:38:07,578 --> 00:38:13,208 again, you need the proper, attitude in your environment that we're going 679 00:38:13,208 --> 00:38:17,018 to do this, but also you need the right attitude that we're gonna make 680 00:38:17,018 --> 00:38:21,688 mistakes as we move forward, as we try to increase our security posture. 681 00:38:22,628 --> 00:38:24,978 And then when we do that, we don't chop people's heads off 682 00:38:27,960 --> 00:38:29,370 any final thoughts from you, Prasanna? 683 00:38:30,373 --> 00:38:34,523 you should go configure your alerts and make sure you don't tweak it 684 00:38:34,533 --> 00:38:38,723 so far down that you miss alerts because then it's pointless that you 685 00:38:38,723 --> 00:38:40,233 went and put in all these sensors 686 00:38:40,708 --> 00:38:44,708 what I d- what I did is I turned off the beeping part of the… 687 00:38:45,025 --> 00:38:45,395 Okay 688 00:38:45,908 --> 00:38:49,668 it was the centralized hub that I didn't know it, it did that. 689 00:38:50,008 --> 00:38:53,658 And so because I was getting these alarms, on my phone, I was like, "Yeah, 690 00:38:53,658 --> 00:38:58,098 I need to go check it out." And, it just turned out that one of the sensors was, 691 00:38:58,168 --> 00:39:00,228 its temperature is reading too high. 692 00:39:00,668 --> 00:39:03,808 the f- the fridge was actually fine, it's just it was reading 693 00:39:03,808 --> 00:39:04,948 it, higher than normal. 694 00:39:05,288 --> 00:39:09,608 I just didn't know that when I wasn't at my house, there was this really loud 695 00:39:09,709 --> 00:39:10,149 I think PP 696 00:39:10,208 --> 00:39:10,728 was going. 697 00:39:11,818 --> 00:39:13,378 Yeah, so I get this message, "Hey, 698 00:39:13,382 --> 00:39:13,452 there's 699 00:39:13,778 --> 00:39:15,718 there's this really loud beeper." "What was that, Mike?" 700 00:39:16,762 --> 00:39:20,552 There's alert fatigue and then there's alert annoyance. 701 00:39:21,302 --> 00:39:24,322 And so fatigue means I'm just getting all these alerts and I… 702 00:39:24,322 --> 00:39:25,902 There's just no way, I give up. 703 00:39:26,214 --> 00:39:29,714 And then the, the avoidance one is you hear it all the time and you're 704 00:39:29,714 --> 00:39:31,214 just like, "Yeah, I know what that is. 705 00:39:31,214 --> 00:39:34,524 I don't even need to go look at it." It's kinda like the car alarms from the '90s, 706 00:39:34,554 --> 00:39:38,714 like the chirp and the, even- initially that was all cool and when the alarm 707 00:39:38,714 --> 00:39:41,374 went off you'd be looking out the window like, "What's going on with my car?" 708 00:39:41,659 --> 00:39:42,009 Yeah 709 00:39:42,074 --> 00:39:45,214 eventually you're asleep and you hear it go off and you're just reaching 710 00:39:45,214 --> 00:39:46,627 for your keys and you're like, chirp. 711 00:39:47,327 --> 00:39:49,417 I, don't know what happened but, it just goes off. 712 00:39:49,447 --> 00:39:50,267 Maybe it was a cat. 713 00:39:50,307 --> 00:39:53,267 I don't know." but you don't even get up to look at it anymore. 714 00:39:54,127 --> 00:39:58,327 And so to Prasanna's point, and to your point too, but initially whenever you 715 00:39:58,327 --> 00:40:01,547 make this change there's gonna be a lot of noise, then you've gotta, you've 716 00:40:01,547 --> 00:40:07,437 gotta make an effort to review all that noise and tune it down to meaningful 717 00:40:07,447 --> 00:40:13,707 things, that sometimes tuning also means reviewing or assessing the data 718 00:40:13,707 --> 00:40:20,027 that's the input for the alert, because the quality of the input could have 719 00:40:20,027 --> 00:40:21,767 a result on the quality of the alert. 720 00:40:22,997 --> 00:40:24,257 So we're getting too much junk. 721 00:40:24,257 --> 00:40:25,317 why are we getting too much junk? 722 00:40:25,337 --> 00:40:26,717 'Cause we're looking at the wrong stuff. 723 00:40:27,297 --> 00:40:29,847 we're alerting on too many things or not the right thing. 724 00:40:29,877 --> 00:40:30,607 Or you know what? 725 00:40:30,637 --> 00:40:35,327 we've said, "Tell us every time an admin account's created." Great, you've 726 00:40:35,337 --> 00:40:38,377 got the, you've got the mechanism to alert you but you forgot to 727 00:40:38,687 --> 00:40:40,557 configure endpoint to start logging. 728 00:40:41,957 --> 00:40:44,567 So you gotta turn auditing on and audit the right things. 729 00:40:45,397 --> 00:40:48,167 that whole process has to be assessed whenever you do this alerting. 730 00:40:48,167 --> 00:40:51,537 All right, now we've got it tuned down to meaningful things. 731 00:40:52,267 --> 00:40:54,347 what do we do when those meaningful things happen? 732 00:40:54,357 --> 00:40:56,827 So it's not just a I see an alert or the bell went off. 733 00:40:56,867 --> 00:41:00,637 All right, where's the procedure that we follow when that happens? 734 00:41:01,807 --> 00:41:05,877 So there's a whole feedback loop there during that analysis. 735 00:41:05,877 --> 00:41:08,877 But definitely expect a large volume of noise whenever you 736 00:41:08,877 --> 00:41:09,897 turn something like that on. 737 00:41:10,265 --> 00:41:12,715 in my neck of the woods, you know the reason why the car 738 00:41:12,715 --> 00:41:14,235 alarms often go off, right? 739 00:41:16,397 --> 00:41:16,987 Tremors. 740 00:41:17,079 --> 00:41:17,399 Why? 741 00:41:19,127 --> 00:41:19,437 Yeah. 742 00:41:19,817 --> 00:41:22,687 Even though we don't get that many of them, you get one and next thing 743 00:41:22,687 --> 00:41:24,867 you know the entire neighborhood's car alarms are going off. 744 00:41:25,347 --> 00:41:25,877 Good times. 745 00:41:25,887 --> 00:41:26,557 Good times. 746 00:41:27,267 --> 00:41:27,617 All right. 747 00:41:27,617 --> 00:41:29,387 thanks again, Mike, for, joining us 748 00:41:31,171 --> 00:41:31,491 Certainly. 749 00:41:31,491 --> 00:41:32,021 Thank you 750 00:41:32,885 --> 00:41:34,035 thanks, thanks Prasanna. 751 00:41:34,315 --> 00:41:34,635 you know 752 00:41:35,050 --> 00:41:36,370 You already thanked me before. 753 00:41:37,035 --> 00:41:37,465 Did I? 754 00:41:37,715 --> 00:41:38,115 All right. 755 00:41:38,250 --> 00:41:38,970 I think you did. 756 00:41:39,115 --> 00:41:42,055 thank you, thank you, thank you, thank you. 757 00:41:42,745 --> 00:41:45,185 And thanks to our listeners, for joining us. 758 00:41:45,215 --> 00:41:46,625 you are why we do this. 759 00:41:46,905 --> 00:41:47,965 That is a wrap 760 00:41:51,112 --> 00:41:55,814 The Backup Wrap Up is written, recorded, and produced by me, W. Curtis Preston. 761 00:41:56,381 --> 00:42:01,150 If you need backup or DR consulting, content generation, or expert witness 762 00:42:01,154 --> 00:42:03,754 work, check out backupcentral.com. 763 00:42:04,458 --> 00:42:07,649 You can also find links for my O'Reilly books on the same website. 764 00:42:08,248 --> 00:42:12,226 Remember, this is an independent podcast, and any opinions that 765 00:42:12,226 --> 00:42:16,204 you hear are those of the speaker and not necessarily an employer. 766 00:42:17,069 --> 00:42:17,846 Thanks for listening