1 00:00:00,000 --> 00:00:03,653 You found the backup wrap up your go-to podcast for all things 2 00:00:03,653 --> 00:00:05,963 backup recovery and cyber recovery. 3 00:00:06,593 --> 00:00:09,713 In this episode, we're talking about something that makes me 4 00:00:09,713 --> 00:00:12,173 want to flip tables upside down. 5 00:00:12,803 --> 00:00:19,013 People using Windows shadow copies or VSS as their actual backup system, and 6 00:00:19,013 --> 00:00:23,423 then ransomware comes along and then deletes those shadow copies using your 7 00:00:23,423 --> 00:00:25,553 own built-in Windows tools against you. 8 00:00:25,973 --> 00:00:27,563 And suddenly you have nothing. 9 00:00:28,103 --> 00:00:31,943 I'm joined again by Dr. Mike Sailor, our intrepid cybersecurity 10 00:00:31,943 --> 00:00:34,643 expert and persona ndi of course. 11 00:00:35,093 --> 00:00:39,953 And we break down exactly why shadow copies are not a backup, 12 00:00:40,493 --> 00:00:44,153 how attackers are pulling off this move, and what you can do to protect 13 00:00:44,153 --> 00:00:48,683 yourself if you've been counting on VSS to save you when things go wrong. 14 00:00:49,538 --> 00:00:50,918 This is your episode. 15 00:00:51,908 --> 00:00:53,438 Uh, just a quick note about me. 16 00:00:53,438 --> 00:00:59,288 I'm w Curtis Preston, AKA Mr. Backup, and I've been obsessing about backup recovery 17 00:00:59,288 --> 00:01:02,768 and now cyber recovery for over 30 years. 18 00:01:03,338 --> 00:01:05,258 If that's your bag, then I'm your guy. 19 00:01:05,438 --> 00:01:09,188 Well, you're not gonna find anybody that cares about it more than me. 20 00:01:09,548 --> 00:01:13,063 Ever since 1993 when I had to tell my boss there were no backups of 21 00:01:13,063 --> 00:01:14,648 the database that we had just lost. 22 00:01:15,248 --> 00:01:17,978 Now I've written five books, a blog, and a podcast. 23 00:01:18,338 --> 00:01:22,448 And here we turn, unappreciated Admins and the Cyber Recovery Heroes. 24 00:01:22,748 --> 00:01:24,818 This is the backup wrap up. 25 00:01:41,658 --> 00:01:42,948 Welcome to the backup wrap up. 26 00:01:42,948 --> 00:01:45,708 I'm your host, W. Curtis Preston, AKA, and Mr. Backup. 27 00:01:45,708 --> 00:01:49,033 And I have with me a guy who couldn't bother to on time. 28 00:01:49,643 --> 00:01:51,673 Prasanna Malaiyandi, how's it going? 29 00:01:51,683 --> 00:01:52,193 Prasanna. 30 00:01:52,808 --> 00:01:54,408 You've heard about Indian standard time, right? 31 00:01:57,088 --> 00:02:01,738 I am not making any comments about anybody's stuff based on 32 00:02:01,858 --> 00:02:04,948 where they may or may not from 33 00:02:05,513 --> 00:02:09,653 so I do have an excuse because I had to change into my shirt, 34 00:02:09,653 --> 00:02:11,303 which Curtis, I see that. 35 00:02:11,488 --> 00:02:11,758 shirt. 36 00:02:11,758 --> 00:02:12,838 I should get one of those. 37 00:02:13,103 --> 00:02:13,463 Yeah. 38 00:02:13,533 --> 00:02:16,113 I think you have to just find yours is more what the. 39 00:02:16,193 --> 00:02:19,463 to God, I have turned this house upside down looking for that shirt that I 40 00:02:19,463 --> 00:02:22,763 paid good money for, I cannot find it. 41 00:02:22,973 --> 00:02:27,693 anyway, including yesterday's event where I literally took 42 00:02:27,693 --> 00:02:28,983 everything out of my closet. 43 00:02:29,403 --> 00:02:32,703 have no excuse, but maybe it's being used as a cleaning rug somewhere. 44 00:02:33,123 --> 00:02:33,333 Anyway. 45 00:02:34,318 --> 00:02:38,578 Of course have our intrepid cybersecurity expert, Dr. Mike Saylor. 46 00:02:38,698 --> 00:02:39,148 How's it going? 47 00:02:39,148 --> 00:02:39,478 Mike, 48 00:02:40,711 --> 00:02:41,431 It is going well guys. 49 00:02:41,431 --> 00:02:42,181 Thanks for having me. 50 00:02:42,568 --> 00:02:44,218 have you still have you yet? 51 00:02:44,278 --> 00:02:44,998 I have not. 52 00:02:44,998 --> 00:02:46,978 Have you seen our book yet? 53 00:02:47,956 --> 00:02:48,476 I have not. 54 00:02:49,288 --> 00:02:49,958 It's killing me. 55 00:02:50,378 --> 00:02:54,128 So we're of course referring to this book up here on my video, which if you're 56 00:02:54,128 --> 00:02:55,868 not watching us on YouTube, feel free. 57 00:02:56,148 --> 00:03:00,078 the same, this channel by the same name, the backup wrap up, and, 58 00:03:00,078 --> 00:03:05,158 learning ransomware response and recovery, available now, at, all the 59 00:03:05,158 --> 00:03:07,348 places that you buy books, although 60 00:03:07,553 --> 00:03:07,563 is. 61 00:03:07,618 --> 00:03:08,128 the. 62 00:03:09,133 --> 00:03:13,023 All the places, if you prefer Rainforest in South America, or 63 00:03:13,023 --> 00:03:17,703 if you prefer buying them directly from O'Reilly, or Barnes and Noble. 64 00:03:17,703 --> 00:03:19,473 I was in a Barnes and Noble yesterday. 65 00:03:19,903 --> 00:03:23,413 apparently it's a building in which you can buy books. 66 00:03:23,473 --> 00:03:24,553 It's a fascinating. 67 00:03:25,023 --> 00:03:26,013 Fascinating idea. 68 00:03:26,443 --> 00:03:27,463 maybe that'll take off. 69 00:03:27,733 --> 00:03:32,223 Anyway, so today, we're talking about, something a little bit up my alley. 70 00:03:32,223 --> 00:03:38,433 And I'm gonna just start out, imagine, if you will, you have developed a backup. 71 00:03:38,513 --> 00:03:41,693 I'm gonna make quotes in here, a backup system, because I don't 72 00:03:41,693 --> 00:03:43,073 think this is a backup system. 73 00:03:43,523 --> 00:03:46,643 You have developed a backup system around something called shadow copies, 74 00:03:47,003 --> 00:03:49,918 which is a ver a feature in Windows that we're gonna talk about in a minute. 75 00:03:51,413 --> 00:03:55,733 And then you just come in and you've heard that something bad has happened 76 00:03:55,733 --> 00:03:59,483 from some sort of cyber attack perspective, and then you find out 77 00:03:59,483 --> 00:04:03,663 that, you know that those previous shadow copies are not available to you. 78 00:04:04,003 --> 00:04:07,853 I like the, this reminds me of, for those of you that have been around a minute, 79 00:04:08,153 --> 00:04:12,543 this is the ultimate sweep, the leg move, that is of course a karate kid. 80 00:04:12,843 --> 00:04:13,563 Reference. 81 00:04:13,753 --> 00:04:18,523 and this also refers back to living off the Land, when we talked about living 82 00:04:18,523 --> 00:04:22,093 off the land a few episodes ago where they use your own tools against you. 83 00:04:22,393 --> 00:04:27,113 In this case, the tool would be VSS, admin, and they just delete this thing 84 00:04:27,143 --> 00:04:30,563 upon which you are counting for recovery. 85 00:04:30,783 --> 00:04:32,243 this sounds like a bad day. 86 00:04:32,243 --> 00:04:32,963 What do you think, Mike? 87 00:04:34,441 --> 00:04:35,101 I agree. 88 00:04:35,161 --> 00:04:35,521 Yeah. 89 00:04:35,611 --> 00:04:39,351 I think one of the problems, that people have is they become complacent. 90 00:04:39,351 --> 00:04:41,431 And I think that's, it's not just with your backups. 91 00:04:41,431 --> 00:04:42,646 I think in general with security. 92 00:04:43,421 --> 00:04:47,591 but for sure, whoever set up your backup strategy, whether that was you 93 00:04:47,591 --> 00:04:52,811 or your grandkid, or a volunteer or even someone been in it for a while, 94 00:04:52,871 --> 00:04:54,371 shadow copies have been around. 95 00:04:55,161 --> 00:05:00,281 and they come with, they come with the batteries, but they're, to, to 96 00:05:00,281 --> 00:05:03,321 your point, they're not, they're not a legit backup solution. 97 00:05:04,348 --> 00:05:04,638 Yeah, 98 00:05:04,713 --> 00:05:11,743 A VSS volume, shadow services, AKA, shadow copies, and the, is a service 99 00:05:11,743 --> 00:05:18,343 within Windows where you're able to create a stable, consistent, image 100 00:05:18,343 --> 00:05:20,143 through which you can view the platform. 101 00:05:20,143 --> 00:05:21,373 Let me go back in time. 102 00:05:22,258 --> 00:05:27,453 There was a time when, VSS did not exist, and when you went to go back up. 103 00:05:28,003 --> 00:05:28,783 Windows. 104 00:05:29,213 --> 00:05:31,073 the problem is you're backing up. 105 00:05:31,373 --> 00:05:34,913 When you have a large system, you're gonna, part of the backups 106 00:05:34,913 --> 00:05:37,043 are gonna be from this point in time, and part of your backups are 107 00:05:37,043 --> 00:05:38,273 gonna be from that point in time. 108 00:05:38,333 --> 00:05:41,813 And if the thing that you're backing up needs what we call referential 109 00:05:41,813 --> 00:05:45,493 integrity, meaning integrity between the different points of the system. 110 00:05:46,033 --> 00:05:49,573 You're not gonna have ref referential integrity and 111 00:05:49,573 --> 00:05:50,593 you're gonna have a bad backup. 112 00:05:50,653 --> 00:05:52,813 And there used to be a product, I don't know. 113 00:05:52,913 --> 00:05:54,173 person, do you remember St. 114 00:05:54,173 --> 00:05:54,533 Bernard? 115 00:05:54,533 --> 00:05:55,553 Is that before your time? 116 00:05:55,873 --> 00:05:56,023 That's. 117 00:05:56,293 --> 00:05:56,793 Or my time. 118 00:05:57,293 --> 00:05:57,713 Okay. 119 00:05:57,843 --> 00:05:59,073 Mike, do you remember St. Bernard. 120 00:06:00,221 --> 00:06:00,641 I don't. 121 00:06:01,518 --> 00:06:03,558 Man, this is okay. 122 00:06:03,828 --> 00:06:08,958 By the way, St. Bernard was what you bought something before VSS existed. 123 00:06:08,958 --> 00:06:11,958 And for the record, St. Bernard was a San Diego company. 124 00:06:13,368 --> 00:06:16,788 and, it came from the idea of the, that St. Bernard's, the image of St. 125 00:06:16,788 --> 00:06:20,448 Bernard's with a thing of whiskey around the, they came to save you. 126 00:06:20,448 --> 00:06:22,188 So that's where the Saint Bernard name came from. 127 00:06:23,298 --> 00:06:28,398 The, so VSS was this method through which you could create a stable 128 00:06:28,788 --> 00:06:31,348 image, and do various things with it. 129 00:06:31,348 --> 00:06:34,918 So Prasanna, why don't you talk about like the first, what, 130 00:06:35,008 --> 00:06:36,598 VSS was typically used for. 131 00:06:36,598 --> 00:06:40,288 Yeah, Because one of the things that you need to do with a Windows system, 132 00:06:40,288 --> 00:06:45,208 or if you're running say Microsoft SQL or Exchange is do your backup. 133 00:06:46,178 --> 00:06:48,668 And especially on physical machines, before we had 134 00:06:48,668 --> 00:06:50,168 virtualization and everything else. 135 00:06:50,783 --> 00:06:54,023 The only way to actually get a stable point from which to 136 00:06:54,023 --> 00:06:56,183 do your backup was to use VSS. 137 00:06:56,333 --> 00:07:00,023 And so you'd have the backup apps call into and VSS. 138 00:07:00,023 --> 00:07:01,883 It's a framework that Microsoft provides. 139 00:07:01,883 --> 00:07:05,243 So you'd call in, you'd say, Hey, I wanna do a backup of this particular app. 140 00:07:05,753 --> 00:07:06,773 Microsoft would then. 141 00:07:06,873 --> 00:07:07,103 quiesce 142 00:07:09,488 --> 00:07:10,388 the application. 143 00:07:10,388 --> 00:07:13,268 Take a volume snapshot, right? 144 00:07:13,268 --> 00:07:14,228 Shadow snapshot. 145 00:07:14,648 --> 00:07:18,748 And then it would unes, so then you can continue doing your stuff, right? 146 00:07:18,748 --> 00:07:21,508 And then what the backup app would do is it would then read from 147 00:07:21,508 --> 00:07:26,668 that shadow snapshot in order to be able to do the actual backups. 148 00:07:27,328 --> 00:07:31,408 And so this is what it was useful for, and it was more of 149 00:07:31,408 --> 00:07:36,178 like a software snap snapshot implementation, pre VMware snapshots. 150 00:07:36,528 --> 00:07:39,798 but yes, it was their mechanism to get a consistent point for which 151 00:07:39,798 --> 00:07:40,593 you could do your backups from. 152 00:07:41,943 --> 00:07:44,633 And there, like you said, it was a framework, right? 153 00:07:44,633 --> 00:07:48,533 So each application that wanted to be supported by VSS, they wrote 154 00:07:48,533 --> 00:07:51,953 what was called a VSS writer, and then VSS would, and then they just 155 00:07:51,953 --> 00:07:53,753 needed to plug into that world. 156 00:07:54,053 --> 00:07:57,563 And like you said, a backup app would just say, Hey, I'm here to do a backup. 157 00:07:57,593 --> 00:07:59,933 It would talk to V-S-S-V-S-S would talk to the writers. 158 00:07:59,933 --> 00:08:02,393 Everybody would do the right thing that worked for their application. 159 00:08:02,678 --> 00:08:03,998 Then we would take the snapshot. 160 00:08:04,258 --> 00:08:06,658 it's great for that, right? 161 00:08:06,658 --> 00:08:11,988 It is the easiest way to get a backup of SQL Server or Exchange or any, or 162 00:08:11,988 --> 00:08:14,628 even Oracle running on top of Windows. 163 00:08:14,628 --> 00:08:16,668 Just use the VSS infrastructure. 164 00:08:18,198 --> 00:08:24,528 but some people, Mike, have you seen this in the while where people use 165 00:08:24,528 --> 00:08:27,768 VSS, like as a backup mechanism? 166 00:08:27,768 --> 00:08:29,928 Have you ever actually seen that you have. 167 00:08:30,186 --> 00:08:30,486 I have. 168 00:08:30,546 --> 00:08:30,816 Yep. 169 00:08:30,843 --> 00:08:31,863 You're killing me. 170 00:08:31,953 --> 00:08:32,523 What? 171 00:08:32,533 --> 00:08:33,373 any you? 172 00:08:33,873 --> 00:08:34,383 Why? 173 00:08:35,103 --> 00:08:35,343 Why? 174 00:08:35,443 --> 00:08:36,733 Not that he's killing me, but, 175 00:08:37,083 --> 00:08:37,533 No? 176 00:08:37,643 --> 00:08:38,393 oh, you're saying why? 177 00:08:38,393 --> 00:08:38,873 Why would they 178 00:08:38,893 --> 00:08:39,343 Yeah. 179 00:08:39,413 --> 00:08:39,983 Yeah. 180 00:08:40,123 --> 00:08:40,963 And what was the purpose? 181 00:08:40,963 --> 00:08:41,203 Bike. 182 00:08:41,203 --> 00:08:42,823 Could you elaborate a little? 183 00:08:43,296 --> 00:08:47,256 I think it's just in the absence of an educated solution. 184 00:08:47,448 --> 00:08:53,388 so they, they tried to understand the manual or did a Google search or, saw the 185 00:08:53,388 --> 00:08:58,308 word, Or the idea that it was a backup of some sort, and they just went with that, 186 00:08:59,175 --> 00:09:03,375 Because you could create multiple shadow copies over time, right? 187 00:09:03,475 --> 00:09:05,395 and you could hold onto them. 188 00:09:06,025 --> 00:09:06,505 These are, 189 00:09:06,633 --> 00:09:07,078 a problem. 190 00:09:07,525 --> 00:09:08,755 which is also a problem. 191 00:09:09,655 --> 00:09:11,395 It's a problem for multiple reasons, right? 192 00:09:11,645 --> 00:09:13,955 Prasanna, I'm pretty sure these are copy on write. 193 00:09:14,005 --> 00:09:14,585 Snapshots, right? 194 00:09:15,510 --> 00:09:16,050 yes. 195 00:09:16,595 --> 00:09:20,515 Yeah, For those that don't know that there are multiple reasons why using 196 00:09:20,515 --> 00:09:23,755 VSS as a backup app are a bad idea. 197 00:09:23,755 --> 00:09:27,575 We're gonna primarily talk about one of them in the episode, but one of the 198 00:09:27,575 --> 00:09:32,765 other ones is if you've got 30 days of VSS snapshots, your performance is in the 199 00:09:32,765 --> 00:09:36,395 crapper, because of the way copy on write. 200 00:09:36,395 --> 00:09:37,235 Snapshots work. 201 00:09:37,315 --> 00:09:38,725 I'll see if I can find them. 202 00:09:38,760 --> 00:09:40,560 We have a, we have an episode where we just. 203 00:09:40,685 --> 00:09:44,245 Delve into, copy on write, snapshots and what the differences between 204 00:09:44,245 --> 00:09:46,495 them and redirect on write snapshots. 205 00:09:48,205 --> 00:09:54,355 but because of the way VSS works underneath you are going to significantly 206 00:09:54,355 --> 00:09:57,505 hurt the performance of your system if you're using it as a backup app. 207 00:09:59,395 --> 00:10:00,305 go ahead, Prasanna. 208 00:10:00,595 --> 00:10:01,465 one thing I do wanna. 209 00:10:02,740 --> 00:10:05,770 Clarify is my knowledge of VSS is slightly dated. 210 00:10:05,770 --> 00:10:08,890 They may have improved things over the years, but 211 00:10:09,420 --> 00:10:12,770 I doubt that they changed fundamental architecture from copy 212 00:10:12,770 --> 00:10:14,170 on write to redirect on write. 213 00:10:14,870 --> 00:10:15,350 You agree? 214 00:10:16,180 --> 00:10:16,750 I would agree. 215 00:10:17,465 --> 00:10:19,570 I, think it's highly unlikely, but today. 216 00:10:21,180 --> 00:10:23,250 We're talking about the cyber aspects of this. 217 00:10:24,810 --> 00:10:28,080 so Mike, why don't you talk, let me just look at the outline we have here. 218 00:10:28,130 --> 00:10:32,050 Mike, you wanna, you want to, just give a, again, a brief overview of 219 00:10:32,050 --> 00:10:36,910 the concept of living off the land and what, how that A applies here. 220 00:10:38,293 --> 00:10:38,713 Sure, yeah. 221 00:10:38,983 --> 00:10:43,943 Living off the land is a bad guy's strategy of utilizing data, 222 00:10:43,943 --> 00:10:49,563 applications and trust, already associated with a compromised host. 223 00:10:49,563 --> 00:10:55,738 So if I. I'm able to get access to any degree, to a laptop or a server, 224 00:10:56,118 --> 00:11:00,508 what things are already there that I can utilize to further my attack, 225 00:11:00,508 --> 00:11:05,698 whether it's escalating my privileges or collecting data, or moving from 226 00:11:05,698 --> 00:11:07,648 this computer to another computer. 227 00:11:08,578 --> 00:11:11,128 those are the types of living off the land. 228 00:11:11,328 --> 00:11:13,488 techniques or tactics a bad guy would use. 229 00:11:13,968 --> 00:11:17,658 So one of the first things that they will do, and this is the first phase in 230 00:11:17,658 --> 00:11:20,318 just about any attack, is reconnaissance. 231 00:11:20,588 --> 00:11:22,298 So I've got access to something. 232 00:11:22,298 --> 00:11:25,178 What is it and what do I have access to? 233 00:11:26,058 --> 00:11:26,508 what's here? 234 00:11:26,508 --> 00:11:27,558 What tools do I have? 235 00:11:28,118 --> 00:11:30,938 one of the things that they will look for are shadow 236 00:11:30,938 --> 00:11:33,218 copies, for a couple of reasons. 237 00:11:33,218 --> 00:11:35,448 One is that potentially leverage? 238 00:11:36,318 --> 00:11:38,268 for extortion or something else. 239 00:11:38,548 --> 00:11:43,688 and two, can I collect that data and potentially use that as information, 240 00:11:43,738 --> 00:11:46,108 to further understand the target? 241 00:11:46,838 --> 00:11:50,778 maybe it's, I've found out that this shadow copy is of, QuickBooks. 242 00:11:51,148 --> 00:11:55,298 so now there might be some financial information there, or, maybe there's 243 00:11:55,298 --> 00:11:57,668 a company name or a project name. 244 00:11:58,338 --> 00:12:02,568 something that's gonna make, the light bulb go off and help me determine 245 00:12:02,568 --> 00:12:04,068 if this is a valuable target. 246 00:12:04,878 --> 00:12:06,553 and then whatever my next steps might be. 247 00:12:08,810 --> 00:12:09,165 Go ahead. 248 00:12:10,045 --> 00:12:14,435 When you're talking about that, I'm actually not sure what's done today, but 249 00:12:15,035 --> 00:12:22,355 when an application is reading from a shadow copy, does that get logged in all 250 00:12:22,355 --> 00:12:27,855 the other normal processes or is this sort of something that's hidden away and so 251 00:12:27,855 --> 00:12:32,535 the normal mechanisms don't necessarily apply and so it might be harder to figure 252 00:12:32,535 --> 00:12:34,785 out if someone is exfiltrating data. 253 00:12:38,513 --> 00:12:42,703 No, you can actually monitor for, not only the store, access to where the 254 00:12:42,703 --> 00:12:48,283 files, the data is, but you can also, monitor for the processes that would. 255 00:12:49,208 --> 00:12:53,608 interact with the shadow copies, the, vs admin as an example. 256 00:12:54,448 --> 00:12:57,358 so yeah, those are all things that you can write detections for. 257 00:13:00,178 --> 00:13:04,448 and whether that's like anti-malware on the device or, much broader 258 00:13:04,448 --> 00:13:07,608 like network monitoring, cybersecurity monitoring platform. 259 00:13:08,585 --> 00:13:09,215 Yeah. 260 00:13:09,215 --> 00:13:14,455 the challenge though, and again, this goes back to the, to the, the nature of living 261 00:13:14,455 --> 00:13:20,035 off the land is that the tools in question are commonly used by other tools, right? 262 00:13:20,285 --> 00:13:25,065 VSS admin and w and WMIC, which is a, the Windows management, 263 00:13:25,135 --> 00:13:26,605 interface, on the command line. 264 00:13:27,095 --> 00:13:31,175 that's one of the challenges of the, you can monitor for it, but you might 265 00:13:31,175 --> 00:13:32,975 end up getting a lot of false positives. 266 00:13:33,415 --> 00:13:34,215 the, sorry. 267 00:13:34,245 --> 00:13:35,415 just looking at here. 268 00:13:35,750 --> 00:13:38,030 Ask another question while you're looking. 269 00:13:38,810 --> 00:13:42,040 Mike, sorry, I'm, I've never actually dug into, is it vs. 270 00:13:42,040 --> 00:13:43,175 Admin or VSS Admin, by the way. 271 00:13:44,503 --> 00:13:45,163 Two S's. 272 00:13:45,330 --> 00:13:45,600 Okay. 273 00:13:46,200 --> 00:13:48,390 For VSS admin, I've never dug into it. 274 00:13:48,420 --> 00:13:52,320 Can you access control VSS admin? 275 00:13:52,320 --> 00:13:55,880 I know you talked about you can monitor it for what files are being accessed, what 276 00:13:55,880 --> 00:14:02,510 processes can you also lock down who has access to it from a process perspective, 277 00:14:02,510 --> 00:14:05,480 or is that not really possible? 278 00:14:07,243 --> 00:14:08,233 That's a good question. 279 00:14:08,233 --> 00:14:10,783 I know you have to have a, you have to be a local admin. 280 00:14:11,513 --> 00:14:13,253 but I don't know if. 281 00:14:14,423 --> 00:14:20,333 a scripted like PowerShell, could do it, remotely or, that's a good question, 282 00:14:20,835 --> 00:14:21,125 Okay, 283 00:14:21,203 --> 00:14:21,413 but 284 00:14:21,480 --> 00:14:22,590 but Okay. 285 00:14:22,740 --> 00:14:25,950 But yeah, once you have local admin, then all bets are off right at that point. 286 00:14:26,453 --> 00:14:26,723 Yep. 287 00:14:28,565 --> 00:14:37,095 So there are ways to use, again, just so we're not anti snapshot, I 288 00:14:37,095 --> 00:14:40,965 am definitely anti any kind of backup system that is easily deletable. 289 00:14:43,215 --> 00:14:44,085 And this is one of them. 290 00:14:45,195 --> 00:14:48,375 There are ways to use snapshots and there are ways to use snapshots, 291 00:14:48,375 --> 00:14:52,635 perhaps storage array, snapshots, for example, that can be done in such 292 00:14:52,635 --> 00:14:53,985 a way that they're not deletable. 293 00:14:54,355 --> 00:14:58,885 basically if you get it to a place, if you get it to a version of the 294 00:14:58,915 --> 00:15:03,585 system that is, that is where you have worm turned on, that would be one. 295 00:15:03,615 --> 00:15:06,525 Write once read many, right? 296 00:15:06,555 --> 00:15:08,985 There are, again, not anti snapshot. 297 00:15:09,165 --> 00:15:13,425 Just this particular snapshot being used as a backup method. 298 00:15:13,835 --> 00:15:14,285 go ahead. 299 00:15:14,565 --> 00:15:18,435 I wanna challenge you is based on your definition, Curtis, of what we've been 300 00:15:18,435 --> 00:15:23,375 talking about for the last six years, is that shadow copy, even a backup, 301 00:15:24,055 --> 00:15:24,715 It's not, 302 00:15:25,265 --> 00:15:25,625 okay? 303 00:15:26,295 --> 00:15:26,535 that, 304 00:15:26,575 --> 00:15:26,725 Yeah. 305 00:15:27,525 --> 00:15:30,335 So what and why do you think it's, why do you think, I think it's not a backup. 306 00:15:30,995 --> 00:15:33,305 it doesn't follow your favorite rule in the world. 307 00:15:33,760 --> 00:15:36,040 it does not follow my favorite rule in the world. 308 00:15:36,410 --> 00:15:40,820 which of course would be the 3, 2, 1, which is now the 3, 2, 1, 1 0, but 309 00:15:40,820 --> 00:15:44,450 basically three copies of the data on two different media, one of which is offsite. 310 00:15:44,660 --> 00:15:48,790 Arguably the way I understand Peter Krogh, the person who originally coined 311 00:15:48,790 --> 00:15:53,300 this term, is the idea, he would not think of three versions as three copies. 312 00:15:53,570 --> 00:15:55,940 He would think of three actual copies, right? 313 00:15:56,000 --> 00:15:59,360 And then he would want, and those need to be on separate pieces of media. 314 00:15:59,510 --> 00:16:02,960 The two, the idea there is that you have it on different types of media. 315 00:16:03,460 --> 00:16:07,990 So that you have a disc drive and an SSD or an SSD and a tape or a disc, whatever. 316 00:16:08,315 --> 00:16:08,665 Cloud. 317 00:16:08,920 --> 00:16:10,240 that's what the two was meant to. 318 00:16:10,480 --> 00:16:12,190 Basically different risk profiles. 319 00:16:12,460 --> 00:16:17,020 And then the one definitely needs to be somewhere else offsite would be the term 320 00:16:17,020 --> 00:16:18,580 that we would typically be used, right? 321 00:16:19,060 --> 00:16:21,470 It's a difficult term in the cloud, but, but. 322 00:16:21,910 --> 00:16:23,230 The, yeah, you're right. 323 00:16:23,270 --> 00:16:26,920 this isn't any of the, this doesn't meet any of the three, parts of 324 00:16:26,920 --> 00:16:28,600 the definition of the 3, 2, 1 rule. 325 00:16:28,600 --> 00:16:32,770 So yeah, I would not consider this a backup, but based 326 00:16:32,770 --> 00:16:34,810 system can absolutely do that. 327 00:16:34,960 --> 00:16:39,190 yeah, so again, not anti snapshot, just anti using them like this. 328 00:16:39,280 --> 00:16:42,370 All right, Mike, you go ahead. 329 00:16:42,460 --> 00:16:42,670 What. 330 00:16:43,225 --> 00:16:48,775 But even if you are using a snapshot system, so I know as some of my 331 00:16:48,775 --> 00:16:53,155 past employers, they would integrate with the VSS framework to then 332 00:16:53,155 --> 00:16:56,575 trigger a hardware based snapshot because that was better performant. 333 00:16:56,995 --> 00:16:59,965 And then they would do their backup off of the hardware based snapshot. 334 00:17:00,645 --> 00:17:01,185 Yes, 335 00:17:01,645 --> 00:17:03,115 question though, is. 336 00:17:03,985 --> 00:17:07,225 I guess it depends how that snapshot is created. 337 00:17:07,255 --> 00:17:13,615 And is it possible to use the same VSS framework to manage those snapshots? 338 00:17:13,615 --> 00:17:17,275 Because I'm sure that those backup software has a retention 339 00:17:17,275 --> 00:17:18,745 life cycle that it manages. 340 00:17:19,495 --> 00:17:24,655 And so is there a risk that those snapshots could be also deleted if 341 00:17:24,655 --> 00:17:27,210 someone gains access to that Windows box? 342 00:17:28,740 --> 00:17:33,840 I don't think the way VSS admin work is it's designed to 343 00:17:33,840 --> 00:17:35,760 control third party snapshots. 344 00:17:35,760 --> 00:17:38,160 I think it's only the designed to work the other way. 345 00:17:38,160 --> 00:17:39,000 I could be wrong. 346 00:17:39,310 --> 00:17:45,910 but regardless of how off host snapshots are created, they need to 347 00:17:45,910 --> 00:17:47,545 eventually be somewhere that is worm. 348 00:17:49,100 --> 00:17:51,470 And if they're not, then I'm not a fan. 349 00:17:51,720 --> 00:17:54,210 and again, I'm not, it's not, I don't have a problem with VSS. 350 00:17:54,210 --> 00:17:55,770 I think VSS is a great tool. 351 00:17:55,950 --> 00:17:58,590 Use it for what it was designed for, which is to give you a stable, 352 00:17:58,590 --> 00:18:01,290 consistent image to create a backup from. 353 00:18:01,710 --> 00:18:02,820 so let's talk about. 354 00:18:03,140 --> 00:18:09,140 What people can do to, protect against this. 355 00:18:09,140 --> 00:18:13,570 And I, Mike, you've already alluded to this one, is this idea of, 356 00:18:14,000 --> 00:18:16,700 complaining, have your EDR tools look at. 357 00:18:17,855 --> 00:18:21,325 Things like vss, admin so let's talk about the first thing here, 358 00:18:21,325 --> 00:18:25,595 this idea of, give us a little bit more information on the configuring 359 00:18:25,595 --> 00:18:27,335 your EDR to trigger on this stuff. 360 00:18:30,083 --> 00:18:31,223 It's a process. 361 00:18:31,253 --> 00:18:36,623 And what I mean by that is, you've gotta your tools and your people to 362 00:18:36,623 --> 00:18:38,573 understand what's normal, so that. 363 00:18:39,803 --> 00:18:44,303 A normal process like this running in an abnormal fashion would 364 00:18:44,303 --> 00:18:45,693 be what you're detecting on. 365 00:18:46,393 --> 00:18:51,393 so with any good security tool or security service, security 366 00:18:51,393 --> 00:18:56,703 person, there's gonna be a lot of getting to know you the beginning. 367 00:18:57,643 --> 00:19:02,743 so you know, the VSS admin, kicking off. 368 00:19:03,448 --> 00:19:07,858 Twice a week, once a day, whatever the case is, those are gonna show 369 00:19:07,858 --> 00:19:11,648 up, as a privileged, activity. 370 00:19:12,638 --> 00:19:16,148 and so when we're getting to know our clients, we want to be very strict at 371 00:19:16,148 --> 00:19:19,878 first so that we can understand all of those things and then we start to 372 00:19:19,878 --> 00:19:22,488 flag those, or, build an understanding. 373 00:19:22,518 --> 00:19:25,828 we build a wiki, that says these things happen on this. 374 00:19:26,248 --> 00:19:28,168 on this frequency or in this manner. 375 00:19:28,598 --> 00:19:32,468 so that going forward as the, as we detect those things, we can determine 376 00:19:32,468 --> 00:19:37,948 if those are normal or not, or normal with some context like this will 377 00:19:37,948 --> 00:19:40,498 happen, but so will these other things. 378 00:19:40,778 --> 00:19:44,018 but if any of those normal things happen on their own, that in 379 00:19:44,018 --> 00:19:46,448 itself is an anomaly or abnormal. 380 00:19:46,848 --> 00:19:51,338 so creating a baseline and getting to know your, your environment is important. 381 00:19:52,038 --> 00:19:55,823 because again, living, living off the land would indicate that it's stuff that's 382 00:19:55,823 --> 00:19:57,533 already there and that should be okay. 383 00:19:59,265 --> 00:20:02,745 Mike, then, I know you've brought this up, in previous episodes as well, about 384 00:20:02,795 --> 00:20:06,855 okay, you will need to have someone come in, understand the environment, Kind of 385 00:20:06,855 --> 00:20:09,285 fine tune it for your particular use case. 386 00:20:09,285 --> 00:20:11,940 It's not like a. outta the box, everything is good to go. 387 00:20:11,940 --> 00:20:15,450 For you, how long does that process normally take? 388 00:20:15,870 --> 00:20:18,690 And I'm sure it's gonna vary significantly depending on the size of 389 00:20:18,690 --> 00:20:21,920 the organization, the number of devices, the applications, all the rest of that. 390 00:20:21,970 --> 00:20:24,160 But is this like something that'll take three years? 391 00:20:26,003 --> 00:20:26,783 It depends. 392 00:20:26,980 --> 00:20:27,310 Yeah. 393 00:20:27,483 --> 00:20:31,503 what's interesting about that is, identifying the deficiencies or 394 00:20:31,503 --> 00:20:34,973 opportunities for improvement, actually happens fairly quickly. 395 00:20:35,003 --> 00:20:38,303 It's the remediation and the follow up that tends to take time. 396 00:20:38,333 --> 00:20:43,833 as an example, and I'll preface all of this with the adoption 397 00:20:43,833 --> 00:20:45,873 of security in most companies. 398 00:20:46,373 --> 00:20:50,633 Has to be aligned with the culture or the compliance requirements. 399 00:20:51,443 --> 00:20:56,063 other words, if I'm a, if I'm a company that doesn't like security, but I 400 00:20:56,063 --> 00:20:59,393 want to do business in a regulated environment, I have to adopt security. 401 00:21:00,228 --> 00:21:03,028 how we adopt it, how we implement it is a little different. 402 00:21:03,248 --> 00:21:06,398 and may, it may take some time, but there are some companies out there, 403 00:21:07,208 --> 00:21:11,638 organizations that don't like security or their culture is security averse. 404 00:21:11,668 --> 00:21:14,068 We're too friendly to have security type of thing. 405 00:21:14,878 --> 00:21:18,478 or, it's keeping me from doing my job, so we're not gonna do it. 406 00:21:19,978 --> 00:21:24,478 there are a lot of, if not every organization has one or two or 407 00:21:24,478 --> 00:21:31,018 many, Very blatant, observable, deficiencies, even with just best 408 00:21:31,018 --> 00:21:33,018 practice, basic cyber hygiene. 409 00:21:33,708 --> 00:21:38,418 One of those is allowing users to be local admins on their computer. 410 00:21:39,228 --> 00:21:44,328 back in the day was very prevalent because it needed to be in order 411 00:21:44,328 --> 00:21:49,543 to, keep the overhead on already constrained IT support staff you have. 412 00:21:50,318 --> 00:21:52,718 let users do some of the basic stuff. 413 00:21:53,648 --> 00:21:57,098 I remember when one of the new versions of Windows came out and you had to be 414 00:21:57,098 --> 00:21:59,018 a local admin to change your wallpaper. 415 00:22:00,288 --> 00:22:01,248 that was ridiculous. 416 00:22:01,248 --> 00:22:06,378 And, but it was culturally people were causing a big stink because 417 00:22:06,378 --> 00:22:09,228 they couldn't customize their computers and they were unhappy. 418 00:22:10,038 --> 00:22:13,068 And so it got overburdened with these local ad, these, 419 00:22:13,768 --> 00:22:15,598 basic local admin requests. 420 00:22:16,828 --> 00:22:18,148 So everybody got local admin. 421 00:22:19,078 --> 00:22:25,128 today, we still see that to a large degree, people, users have local admin, 422 00:22:25,228 --> 00:22:30,938 just because, because they, there's an exception because they're an engineer or 423 00:22:30,978 --> 00:22:33,288 a designer and their tools require it. 424 00:22:35,328 --> 00:22:37,458 And there are better ways of going about that. 425 00:22:37,458 --> 00:22:37,728 Sure. 426 00:22:37,728 --> 00:22:40,728 Give them access to the local account, but don't let them run 427 00:22:40,788 --> 00:22:42,958 day to day as a local admin. 428 00:22:43,008 --> 00:22:47,158 run as administrator, not, Click on the thing and do run as administrator. 429 00:22:47,168 --> 00:22:48,818 don't log in as an administrator. 430 00:22:49,148 --> 00:22:53,028 So there's things, the more things you implement, the more burden 431 00:22:53,028 --> 00:22:56,658 you're introducing into people's workflows and how they do their job. 432 00:22:57,498 --> 00:22:59,508 And so the easy thing to do is just. 433 00:22:59,838 --> 00:23:01,068 Give them local admin. 434 00:23:01,068 --> 00:23:01,878 So there's that. 435 00:23:02,828 --> 00:23:08,888 but even in the, in those cases as a security, cybersecurity practitioner, 436 00:23:08,988 --> 00:23:12,738 so whether I'm teaching a tool to tell me what's weird or I'm monitoring an 437 00:23:12,738 --> 00:23:17,148 environment for what's weird, if I knew that about an organization, if I knew 438 00:23:17,148 --> 00:23:22,608 that somebody was a local admin, I could still, with that understanding detect 439 00:23:22,608 --> 00:23:26,238 anomalies, occurring as local admin. 440 00:23:26,868 --> 00:23:31,848 There are things that computers do, does bad guys do, that just 441 00:23:31,848 --> 00:23:34,153 are not normal user behavior. 442 00:23:35,895 --> 00:23:40,860 Yeah, the challenge I did a quick, quick look and it doesn't appear 443 00:23:41,640 --> 00:23:46,170 that with VSS admin that basically anyone with administrator rights can 444 00:23:46,170 --> 00:23:48,000 run it, and that I can't say, oh. 445 00:23:48,000 --> 00:23:49,140 Only Steve. 446 00:23:50,130 --> 00:23:50,850 Only Steve. 447 00:23:50,850 --> 00:23:52,920 Or only administrator. 448 00:23:52,970 --> 00:23:55,330 Only you know the base administrator of account. 449 00:23:55,330 --> 00:24:00,670 It doesn't appear that I'm able to restrict it by administrator, and so 450 00:24:00,700 --> 00:24:02,620 let's go back to your comment, Mike. 451 00:24:02,980 --> 00:24:04,540 I think we can agree that. 452 00:24:05,755 --> 00:24:10,755 The proper thing is to have an administrator account and not have 453 00:24:10,755 --> 00:24:12,255 everybody have admin privileges. 454 00:24:12,255 --> 00:24:14,815 But to go back to your earlier, thing, and by the way, this is the 455 00:24:14,815 --> 00:24:17,365 same in, in, in Unix world, right? 456 00:24:17,365 --> 00:24:21,615 Is that you have root and then you have, we, and you use SDU 457 00:24:21,615 --> 00:24:24,065 to, to run things as root. 458 00:24:24,065 --> 00:24:25,775 The same thing here, you have administrator. 459 00:24:26,205 --> 00:24:30,045 Then you have Curtis and Curtis maybe an administrator, but doesn't 460 00:24:30,045 --> 00:24:33,765 have administrator rights, and so I need to run it as administrator. 461 00:24:34,155 --> 00:24:36,505 but to go back to your earlier comment, I couldn't agree more. 462 00:24:36,505 --> 00:24:38,305 And it's the same in the backup world. 463 00:24:38,995 --> 00:24:41,365 If your culture is not one of cybersecurity. 464 00:24:42,475 --> 00:24:45,745 they're just gonna fight you the whole time, and I'm sure you've 465 00:24:45,745 --> 00:24:50,905 been in situations where the culture is the problem, right? 466 00:24:51,125 --> 00:24:55,175 I can think of a company, I can think of a, oh, this is a great story. 467 00:24:55,625 --> 00:24:56,130 There's this company. 468 00:24:57,875 --> 00:25:02,735 on the East Coast and I used to work for a big consulting company and they had this 469 00:25:02,735 --> 00:25:06,175 particular company, and this is a company that literally everybody would know this. 470 00:25:06,605 --> 00:25:07,745 they actually make things. 471 00:25:07,795 --> 00:25:10,375 they're the rarest of all companies is an American company 472 00:25:10,375 --> 00:25:11,965 who actually makes things. 473 00:25:13,405 --> 00:25:16,165 you have these things in your house, guaranteed. 474 00:25:16,495 --> 00:25:22,415 And we went to, they were having all these problems with their backups and I, I was 475 00:25:22,415 --> 00:25:26,045 asked to come in as the big guns and to come in and basically tell them, 'cause 476 00:25:26,105 --> 00:25:27,785 they were trying to get things done. 477 00:25:27,785 --> 00:25:31,645 And then the organization was fighting them and I came in to give 478 00:25:31,645 --> 00:25:36,015 a, presentation, which basically the title of the presentation was, 479 00:25:37,125 --> 00:25:39,705 are the 10 things that you can do to screw up your backup system. 480 00:25:39,705 --> 00:25:40,605 And congratulations. 481 00:25:40,605 --> 00:25:45,405 You guys have managed to do every single one of them, But there was a person, 482 00:25:45,465 --> 00:25:48,225 there was a person that, that had power. 483 00:25:49,920 --> 00:25:52,860 Who was 100% against everything we were trying to do. 484 00:25:53,040 --> 00:25:57,130 She just, and when you have that, when you have a person, whether they're high 485 00:25:57,130 --> 00:26:01,265 up in the organization or down low in the organization, if they've got influence 486 00:26:01,265 --> 00:26:06,665 and they're actively fighting against you there, there's just not much you could do. 487 00:26:06,725 --> 00:26:07,655 And I'm sure you've. 488 00:26:08,505 --> 00:26:12,175 been in that situation, Mike, where you're trying to do something like this 489 00:26:12,325 --> 00:26:16,735 where it's like you shouldn't be using your VSS snapshots as backups and you 490 00:26:16,735 --> 00:26:21,205 really shouldn't, and you should also be monitoring who's running VSS admin. 491 00:26:21,455 --> 00:26:22,145 What. 492 00:26:22,580 --> 00:26:24,870 What, by the way, lemme just stop for a second. 493 00:26:25,170 --> 00:26:28,980 If you ag, if you follow the suggestion from the first part of the call, which 494 00:26:28,980 --> 00:26:34,980 is don't use VSS admin as a method of doing snapshots, this concern becomes 495 00:26:34,980 --> 00:26:37,380 much, much less important, right? 496 00:26:37,470 --> 00:26:42,030 Because the worst you could do with VSS admin, if you're not using it for backup, 497 00:26:42,060 --> 00:26:46,350 if you're not using them, as a backup method, the worst thing you could do with 498 00:26:46,350 --> 00:26:49,110 VSS admin is create a snapshot, right? 499 00:26:49,260 --> 00:26:50,850 Um, so. 500 00:26:51,090 --> 00:26:57,270 Um, no, but there's no snapshots to delete if you're not holding onto your 501 00:26:57,270 --> 00:27:02,725 snapshots right then who's running VSS admin becomes much less important. 502 00:27:03,685 --> 00:27:03,955 okay. 503 00:27:03,955 --> 00:27:05,155 Let's move on to 504 00:27:05,255 --> 00:27:08,765 so Mike, I know we talked about okay, running VSS admin 505 00:27:08,765 --> 00:27:10,205 within the Windows box is bad. 506 00:27:10,205 --> 00:27:11,855 Keeping it long term is bad. 507 00:27:12,285 --> 00:27:15,735 we talked earlier about hardware snapshots are good. 508 00:27:16,575 --> 00:27:22,050 Could you talk about from a cybersecurity perspective, how this kind of 509 00:27:22,780 --> 00:27:25,575 might change with virtualization? 510 00:27:27,000 --> 00:27:34,920 And if virtualization and VSS lead to less cybersecurity concerns. 511 00:27:35,190 --> 00:27:37,380 I know you talked earlier about like auditing and looking for 512 00:27:37,380 --> 00:27:38,610 anomalies and that sort of thing. 513 00:27:42,858 --> 00:27:43,078 Huh? 514 00:27:44,310 --> 00:27:45,420 Is that a loaded question? 515 00:27:45,510 --> 00:27:46,110 Is that a hard. 516 00:27:46,218 --> 00:27:46,608 no. 517 00:27:46,708 --> 00:27:50,598 so on a virtual machine or a virtual host with multiple virtual 518 00:27:50,598 --> 00:27:52,178 machines, Again, it depends. 519 00:27:52,268 --> 00:27:54,818 It depends on what the bad guys were able to compromise. 520 00:27:54,818 --> 00:27:58,928 If I compromise the host, then I've got access to all the virtual machines. 521 00:27:59,468 --> 00:28:02,738 If all those virtual machines are doing their own snapshots, I now have 522 00:28:02,738 --> 00:28:06,128 access to all of those snapshots, because if I've compromised the 523 00:28:06,128 --> 00:28:10,418 host, I'm also privileged on this box across all those virtual machines. 524 00:28:11,438 --> 00:28:12,248 So there's that. 525 00:28:13,148 --> 00:28:18,098 one of the things I do want to add though, even though the majority of, VSS. 526 00:28:18,443 --> 00:28:29,463 Process and, data require admin privileges as a normal user on the machine, I 527 00:28:29,463 --> 00:28:31,473 can still read those shadow copies. 528 00:28:32,403 --> 00:28:37,443 So even if I'm a bad guy and have, I'm not admin yet, I can still do recon into 529 00:28:37,443 --> 00:28:41,763 those shadow copies to see what it is I have access to and potentially exfiltrate 530 00:28:41,763 --> 00:28:43,823 that leverage in an extortion attempt. 531 00:28:44,423 --> 00:28:46,793 Because it's already packaged and compressed, right? 532 00:28:46,793 --> 00:28:48,203 I don't have to redo that. 533 00:28:48,233 --> 00:28:50,963 it's a nice little nugget, that I found. 534 00:28:51,653 --> 00:28:56,103 But in a virtual environment, just as virtualization makes things more 535 00:28:56,133 --> 00:28:59,373 efficient for us, it also makes things more efficient for bad guys. 536 00:29:02,973 --> 00:29:04,593 And back to a comment. 537 00:29:05,623 --> 00:29:08,133 or a thought, thread that Curtis was on? 538 00:29:08,133 --> 00:29:10,923 As far as culture goes, I've seen it both ways. 539 00:29:10,923 --> 00:29:15,363 I've seen where majority of a user base drives culture. 540 00:29:16,143 --> 00:29:19,023 so even the leadership team may say, we need security, but then 541 00:29:19,023 --> 00:29:21,843 you've got this workforce that says, we will not use security. 542 00:29:22,468 --> 00:29:27,148 especially in a service or organization, where they're out, your user base is out 543 00:29:27,178 --> 00:29:33,278 delivering something and they're rebelling against you, you will often cave to them 544 00:29:33,278 --> 00:29:35,528 unless it's truly regulatory required. 545 00:29:36,408 --> 00:29:39,878 and then I've definitely seen the opposite where leadership always wants the, 546 00:29:39,878 --> 00:29:42,758 they're gonna push it down to everybody else, but they've got the exception. 547 00:29:42,758 --> 00:29:45,418 And then everybody sees that, or maybe they become aware of it and 548 00:29:45,418 --> 00:29:47,623 they're like, Leadership's not doing it, I'm not gonna do it. 549 00:29:48,443 --> 00:29:50,153 setting the bad example type of thing. 550 00:29:51,110 --> 00:29:51,470 Yeah. 551 00:29:51,470 --> 00:29:55,080 And this is one of those, I, I've been in environments where, I've 552 00:29:55,080 --> 00:29:59,440 got people in power that are telling me that raid is backup, right? 553 00:29:59,530 --> 00:30:00,580 We don't need backups. 554 00:30:00,580 --> 00:30:03,710 We're we have raid, and then they're actively lobbying against you. 555 00:30:03,710 --> 00:30:09,745 That's the situation where like you need to just walk away, and, let them leave. 556 00:30:10,045 --> 00:30:10,225 Go ahead. 557 00:30:10,453 --> 00:30:15,443 there, there is an example of raid that is acceptable, but you have 558 00:30:15,443 --> 00:30:17,063 to unplug it and put it on a shelf. 559 00:30:18,605 --> 00:30:23,165 Okay, that's not just saying, not what we're talking about here. 560 00:30:23,165 --> 00:30:23,495 Mike. 561 00:30:23,495 --> 00:30:24,245 You're killing me. 562 00:30:24,293 --> 00:30:25,085 am just saying it. 563 00:30:25,325 --> 00:30:25,955 I've seen it. 564 00:30:26,075 --> 00:30:27,365 They've plugged in a storage array. 565 00:30:27,365 --> 00:30:28,265 They've done a backup. 566 00:30:28,265 --> 00:30:29,750 They unplugged it and put it back on the shelf. 567 00:30:30,362 --> 00:30:33,182 That's not braid, that's, you know what I'm saying? 568 00:30:33,242 --> 00:30:33,782 Dang it. 569 00:30:34,302 --> 00:30:37,752 don't, you're trying to get my, trying to get my blood pressure up. 570 00:30:37,842 --> 00:30:37,962 It's 571 00:30:38,287 --> 00:30:45,487 so this is a little bit depressing of a podcast episode, but is the takeaway that 572 00:30:45,487 --> 00:30:48,517 VSS is evil and no one should use VSS. 573 00:30:49,332 --> 00:30:52,362 just don't use VSS as a backup. 574 00:30:52,572 --> 00:30:54,972 That's literally, we could, this could this episode could have 575 00:30:54,972 --> 00:31:00,182 been two minutes long if you're using VSS as a method of backup. 576 00:31:00,817 --> 00:31:03,547 Not to be confused with using it to make your backup. 577 00:31:04,267 --> 00:31:08,347 I, don't you, you understand what, I don't know you're using it to facilitate 578 00:31:08,797 --> 00:31:10,657 whatever actual backups you're doing. 579 00:31:10,657 --> 00:31:14,467 In other words, the way we just, where we talked about where we said, Where 580 00:31:14,497 --> 00:31:18,457 you're creating a stable, consistent image that you're then creating an actual 581 00:31:18,457 --> 00:31:20,827 snapshot of or an actual backup of. 582 00:31:20,827 --> 00:31:25,867 I have no problem with VSS if you're using VSS to create multiple shadow copies 583 00:31:26,617 --> 00:31:30,937 you're storing those on your system, it's a bad idea for multiple reasons. 584 00:31:30,937 --> 00:31:32,047 We've talked about performance. 585 00:31:32,047 --> 00:31:36,667 Mike talked about using them as a method to get historical information, and it's 586 00:31:36,667 --> 00:31:39,397 also bad because all it takes is one. 587 00:31:40,377 --> 00:31:46,477 Bad, one black hat with VSS admin rights, and then they just delete 'em all. 588 00:31:46,627 --> 00:31:47,647 So it's just bad. 589 00:31:47,947 --> 00:31:49,207 So forget all the cyber stuff. 590 00:31:49,207 --> 00:31:50,047 We said, just stop. 591 00:31:50,047 --> 00:31:51,007 Just don't do this. 592 00:31:51,757 --> 00:31:56,117 get a real, go get a real, there, remember there, there was a, I used 593 00:31:56,117 --> 00:31:58,627 to have this, I don't know, it's probably bigger in my life than either 594 00:31:58,627 --> 00:32:02,827 of yours, but there used to be this Dilbert comic strip that we had, and 595 00:32:02,827 --> 00:32:04,507 it was, it was the, where the guy says. 596 00:32:05,097 --> 00:32:09,807 There's a guy with suspenders in the beard he is oh, I recognize you. 597 00:32:10,077 --> 00:32:15,837 And he says, beard and whatever, and then smug expression on your face. 598 00:32:15,837 --> 00:32:17,187 You're a Unix admin. 599 00:32:17,367 --> 00:32:20,827 And then the guy flips him a quarter, and he says, here's a quarter kid. 600 00:32:20,827 --> 00:32:22,327 Go buy yourself a real computer. 601 00:32:23,917 --> 00:32:27,517 That's the way I feel about people that are using their VSS, as backups. 602 00:32:27,907 --> 00:32:28,327 okay. 603 00:32:28,707 --> 00:32:29,007 I don't know. 604 00:32:29,007 --> 00:32:30,567 Do we have any final thoughts on this? 605 00:32:31,355 --> 00:32:31,595 So 606 00:32:31,697 --> 00:32:32,592 I think you just said it. 607 00:32:33,042 --> 00:32:33,662 go ahead, Mike. 608 00:32:34,445 --> 00:32:38,975 VSS is often necessary as the guys talked about earlier. 609 00:32:39,125 --> 00:32:44,685 it, it helps you, quote unquote lock files so that they can be prepared for. 610 00:32:44,995 --> 00:32:45,835 a backup. 611 00:32:46,152 --> 00:32:49,062 but that backup is some other solution. 612 00:32:49,432 --> 00:32:51,712 a lot of backup solutions today do that for you. 613 00:32:51,772 --> 00:32:53,092 they put an agent on your machine. 614 00:32:53,092 --> 00:32:59,042 It does the, it does this function, in the absence of that capability, you're use, 615 00:32:59,102 --> 00:33:01,772 you're using some other backup solution. 616 00:33:02,042 --> 00:33:06,652 VSS is still some, is still often necessary, to prepare 617 00:33:06,652 --> 00:33:08,182 your data to be backed up. 618 00:33:08,852 --> 00:33:13,652 and then the last thing I'll add from a detection perspective is again, if I 619 00:33:13,652 --> 00:33:18,302 knew you were doing backups every night or certain days of the week or once a 620 00:33:18,302 --> 00:33:21,222 week, and that's often scheduled, right? 621 00:33:21,222 --> 00:33:25,672 It's either a task or, a task on the local machine or a scheduled, 622 00:33:26,992 --> 00:33:28,642 activity across the network. 623 00:33:29,002 --> 00:33:31,817 If I knew that from a cybersecurity perspective, and then I saw. 624 00:33:32,587 --> 00:33:37,347 VSS or even some other privileged, backup related account, doing things 625 00:33:37,347 --> 00:33:41,347 outside that schedule, that anomaly, would be what I would respond to. 626 00:33:42,894 --> 00:33:43,434 Absolutely. 627 00:33:43,494 --> 00:33:43,764 Yeah. 628 00:33:43,764 --> 00:33:46,324 I like by the way, there was a, it was an earlier topic. 629 00:33:47,404 --> 00:33:50,524 when you're in a new environment and you are trying to configure 630 00:33:50,524 --> 00:33:55,894 their EDR tools or XDR tools, can't you just put it in and let it watch 631 00:33:55,894 --> 00:33:58,114 for a while to see what normal is? 632 00:33:59,127 --> 00:33:59,587 You can. 633 00:33:59,967 --> 00:34:02,947 and during that period, you're gonna get a ton of alerts. 634 00:34:03,627 --> 00:34:05,157 so there's your false positives. 635 00:34:05,419 --> 00:34:05,709 Okay. 636 00:34:06,087 --> 00:34:10,877 as you're monitoring, I found a new file that's not common across other 637 00:34:10,877 --> 00:34:16,557 user bases or, Bob logs in at 2:00 AM is, I don't know if that's normal yet. 638 00:34:16,557 --> 00:34:20,997 during that baselining period, you're teaching those tools, 639 00:34:21,447 --> 00:34:23,247 tweaking them or whitelisting them. 640 00:34:23,247 --> 00:34:26,547 there's a ton of things that, that happens during that onboarding period. 641 00:34:26,977 --> 00:34:30,627 so that you're essentially creating the what's normal baseline? 642 00:34:31,569 --> 00:34:34,899 You can't just say, just watch for a month and don't tell me anything. 643 00:34:35,827 --> 00:34:38,027 you could, it's not gonna 644 00:34:38,089 --> 00:34:39,109 you don't advise it. 645 00:34:39,139 --> 00:34:39,439 Okay. 646 00:34:39,439 --> 00:34:40,189 I see what you're saying. 647 00:34:40,267 --> 00:34:40,897 no, I don't advise it. 648 00:34:41,719 --> 00:34:42,109 Okay. 649 00:34:42,449 --> 00:34:46,709 this is why you're the cybersecurity professional, and I'm Mr. Backup. 650 00:34:47,669 --> 00:34:48,149 All right. 651 00:34:48,449 --> 00:34:48,749 All right. 652 00:34:48,749 --> 00:34:49,799 this has been fun. 653 00:34:49,929 --> 00:34:52,899 somewhat depressing, but yeah, just don't do that, doctor. 654 00:34:52,899 --> 00:34:54,789 It hurts when I, it hurts when I do this. 655 00:34:54,909 --> 00:34:55,599 Don't do that. 656 00:34:56,022 --> 00:34:58,212 Yeah, I'd like to, that's a new t-shirt idea. 657 00:34:58,212 --> 00:34:59,052 Just don't do that. 658 00:34:59,992 --> 00:35:00,392 do that. 659 00:35:00,460 --> 00:35:02,020 the antithesis of, Nike. 660 00:35:02,812 --> 00:35:05,587 Yeah, don't do it. 661 00:35:06,377 --> 00:35:07,907 Thanks guys for being on the show. 662 00:35:07,907 --> 00:35:08,687 Thanks, Prasanna, 663 00:35:09,447 --> 00:35:13,017 It was fun and I'm glad we gotta talk about something that was near 664 00:35:13,017 --> 00:35:14,727 and dear to my heart many years ago. 665 00:35:14,727 --> 00:35:14,937 So. 666 00:35:16,397 --> 00:35:17,357 and thanks again. 667 00:35:17,357 --> 00:35:19,037 Mike, you're a little fuzzy all of a sudden. 668 00:35:19,037 --> 00:35:20,327 What happened with your camera there? 669 00:35:21,215 --> 00:35:23,795 it's like an auto focus thing and it's got its own mind. 670 00:35:23,795 --> 00:35:25,325 So I'm not real sure, but, 671 00:35:25,522 --> 00:35:26,132 Anointing 672 00:35:26,195 --> 00:35:29,105 a few more of these with you guys, I would expect some logo. 673 00:35:29,165 --> 00:35:30,455 I'll wear a logo at some point. 674 00:35:31,202 --> 00:35:32,492 you'll get one when I get one. 675 00:35:32,612 --> 00:35:33,362 That's what I'm saying. 676 00:35:33,440 --> 00:35:33,530 on. 677 00:35:33,662 --> 00:35:34,142 All right. 678 00:35:34,382 --> 00:35:34,672 Okay. 679 00:35:34,709 --> 00:35:36,939 thanks to our listeners, we're nothing without you. 680 00:35:37,149 --> 00:35:38,139 That is a wrap.