1 00:00:00,000 --> 00:00:04,049 You found the backup wrap up your go-to podcast for all things 2 00:00:04,049 --> 00:00:06,360 backup recovery and cyber recovery. 3 00:00:07,020 --> 00:00:11,639 In this episode, we explore how to design your backups to make 4 00:00:11,639 --> 00:00:13,080 them more resilient to ransomware. 5 00:00:13,080 --> 00:00:13,139 I. 6 00:00:14,005 --> 00:00:18,265 We'll discuss the importance of understanding dwell time, the need 7 00:00:18,265 --> 00:00:21,955 for longer retention periods, and the value of frequent backups. 8 00:00:22,525 --> 00:00:27,354 We'll also delve into innovative recovery solutions, including the use of snapshots, 9 00:00:27,354 --> 00:00:29,545 replication, and cloud-based solutions. 10 00:00:30,064 --> 00:00:35,224 We also talk about the difference between database and file system recoveries. 11 00:00:35,514 --> 00:00:39,744 With regards to ransomware, we get down in the nitty gritty this week. 12 00:00:39,744 --> 00:00:41,394 I hope you like the episode. 13 00:00:41,934 --> 00:00:46,014 By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr. 14 00:00:46,014 --> 00:00:49,584 Backup, and I've been passionate about backup and recovery for 15 00:00:49,584 --> 00:00:51,714 over 30 years, ever since. 16 00:00:51,714 --> 00:00:55,224 I had to tell my boss that we had no backups of this database. 17 00:00:55,404 --> 00:00:56,574 That was really important. 18 00:00:56,839 --> 00:00:57,859 That we just deleted. 19 00:00:58,459 --> 00:01:01,550 I don't want that to happen to you, and that's why I do this show. 20 00:01:02,149 --> 00:01:07,550 On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes. 21 00:01:07,820 --> 00:01:09,800 This is the backup wrap up. 22 00:01:25,582 --> 00:01:26,871 Welcome to show. 23 00:01:27,853 --> 00:01:30,553 I'm your host, w Curtis Preston, ak. 24 00:01:30,553 --> 00:01:30,943 Mr. 25 00:01:30,943 --> 00:01:35,143 Backup, and before we get started, if I could please ask, please, like, 26 00:01:35,203 --> 00:01:38,773 share, subscribe, so you never miss a beat when it comes to this show. 27 00:01:40,713 --> 00:01:42,298 Did, did, did that give you joy? 28 00:01:43,483 --> 00:01:45,373 I met, it's been a while since I've done it. 29 00:01:45,373 --> 00:01:46,033 You know that. 30 00:01:46,138 --> 00:01:46,738 Yeah. 31 00:01:46,828 --> 00:01:47,398 Yeah. 32 00:01:47,908 --> 00:01:53,488 Well, I get to introduce you now after you've introduced me, and I'm 33 00:01:53,488 --> 00:01:59,819 going to introduce you as my fall post-Traumatic stress, counselor. 34 00:01:59,819 --> 00:02:01,991 Prasanna Malaiyandi how's going? 35 00:02:01,991 --> 00:02:03,611 Prasanna 36 00:02:03,631 --> 00:02:06,071 I am good, Curtis, how are you feeling by the way? 37 00:02:06,701 --> 00:02:11,501 I, you know, I, I, I think tho those that watch on YouTube, right? 38 00:02:11,501 --> 00:02:14,441 You, you can watch this on the backup wrap up channel on YouTube. 39 00:02:14,441 --> 00:02:16,301 Those that watch on YouTube can see there. 40 00:02:16,301 --> 00:02:19,991 There's no, uh, this is where I would've, on the left side of my 41 00:02:19,991 --> 00:02:21,971 face, I would've expected a shiner. 42 00:02:22,571 --> 00:02:27,971 Uh, I don't have any broken bones and I, I just have a little bit of soreness left. 43 00:02:28,436 --> 00:02:30,206 What are we talking about? 44 00:02:30,551 --> 00:02:30,731 Yeah. 45 00:02:30,791 --> 00:02:32,321 Well, well, before we talk about 46 00:02:32,321 --> 00:02:36,221 what happened, for those people who may not know, a shiner is 47 00:02:36,221 --> 00:02:37,811 not someone who shines shoes. 48 00:02:38,621 --> 00:02:44,021 A shiner is sort of when someone punches you and you get like a black eye 49 00:02:44,021 --> 00:02:45,401 around, right? 50 00:02:45,401 --> 00:02:46,361 It's like a bruised. 51 00:02:47,036 --> 00:02:48,356 I would've expected a shiner. 52 00:02:48,356 --> 00:02:48,716 Yeah. 53 00:02:49,196 --> 00:02:51,956 So I kind 54 00:02:52,006 --> 00:02:52,446 happened, Curtis? 55 00:02:53,186 --> 00:02:53,426 Yeah. 56 00:02:53,426 --> 00:02:55,946 I kind of fell down an entire flight of stairs. 57 00:02:56,726 --> 00:02:57,146 Um. 58 00:02:58,646 --> 00:03:03,506 By the way, this, we can, we can blame the new office on this. 59 00:03:04,076 --> 00:03:08,336 Um, or we could blame this on the new office because I'm 60 00:03:08,336 --> 00:03:09,956 spending more time upstairs now. 61 00:03:10,646 --> 00:03:15,596 Um, yeah, I was literally, it happened like 20 feet to the 62 00:03:15,596 --> 00:03:18,746 left of me and I was on a ladder. 63 00:03:19,886 --> 00:03:25,496 And I, the, there's a hall next to a landing and I, when I came down from 64 00:03:25,496 --> 00:03:29,186 the ladder, my right foot came down in the hall, my left foot came down 65 00:03:29,186 --> 00:03:33,566 in the landing and which was about, you know, it's a stair, a stair tread 66 00:03:34,136 --> 00:03:36,086 less so that's like, what, six inches? 67 00:03:36,626 --> 00:03:41,321 And, And, and, that was all the momentum I need. 68 00:03:41,546 --> 00:03:46,316 I completely lost my balance and I gained forward momentum. 69 00:03:46,991 --> 00:03:51,911 Going straight down the stairs, so I fell down face first. 70 00:03:51,911 --> 00:03:56,501 I didn't ball up like you might typically do when you fall because 71 00:03:56,501 --> 00:03:59,441 I, I was worried that I would then tumble down the stairs and 72 00:03:59,441 --> 00:04:00,011 I just knew I 73 00:04:00,011 --> 00:04:01,061 would break everything. 74 00:04:01,631 --> 00:04:02,291 I think, yeah, 75 00:04:02,291 --> 00:04:05,351 if you had tumbled down, it probably would've been a lot worse, 76 00:04:05,471 --> 00:04:06,341 like head over heels. 77 00:04:06,731 --> 00:04:07,061 yeah. 78 00:04:07,061 --> 00:04:12,641 And so I stiffed up, stiffened, stiffened up, and then put my hands 79 00:04:12,641 --> 00:04:14,441 out to brace my fall because I fell. 80 00:04:15,566 --> 00:04:17,636 90 plus 45. 81 00:04:17,636 --> 00:04:17,966 Right? 82 00:04:17,966 --> 00:04:19,766 So 135 degrees, right? 83 00:04:19,766 --> 00:04:22,736 I was standing straight up and then, yeah, like a tree going 84 00:04:22,736 --> 00:04:25,016 down, but, but there's no crowd. 85 00:04:25,556 --> 00:04:30,656 And so I fell face first on the stairs, put my hands out, managed 86 00:04:30,656 --> 00:04:33,326 to stop, you know, whatever. 87 00:04:33,326 --> 00:04:38,096 And then I slid face first all the way to the bottom of the stairs, my face. 88 00:04:38,096 --> 00:04:40,166 This is why I said I was expecting a shiner. 89 00:04:40,586 --> 00:04:44,096 My left side of my face hit the ground first. 90 00:04:44,681 --> 00:04:47,801 And then there was enough momentum that I kept going, and then my daughter made me 91 00:04:47,801 --> 00:04:49,481 go to the emergency room and I had a whole 92 00:04:49,481 --> 00:04:53,351 bunch of X-rays and a CAT scan and uh, yeah. 93 00:04:53,351 --> 00:04:55,871 You know, and you're almost 60 years old and you fall down 94 00:04:55,871 --> 00:04:57,251 an entire flight of stairs. 95 00:04:58,151 --> 00:04:59,201 You don't really get a choice. 96 00:04:59,201 --> 00:04:59,741 You go to the 97 00:05:00,026 --> 00:05:00,926 Well, and and 98 00:05:00,986 --> 00:05:04,106 I think you should tell people what the doctor said to you when 99 00:05:04,106 --> 00:05:05,876 he came up to you to check on you. 100 00:05:06,731 --> 00:05:10,481 yeah, he walks actually ev pretty much every medical professional, like 101 00:05:10,481 --> 00:05:12,911 as they turned around the corner and then they saw me and they were like. 102 00:05:13,811 --> 00:05:15,821 Are you, are you William? 103 00:05:16,451 --> 00:05:19,241 Because that's my first name for those of, that's what the W stands for. 104 00:05:19,661 --> 00:05:21,761 Uh, are you William? 105 00:05:21,851 --> 00:05:23,231 And I'm like, yes. 106 00:05:23,231 --> 00:05:27,131 And they're like, you're the guy that fell down a flight of stairs. 107 00:05:27,131 --> 00:05:28,901 And I'm like, yeah. 108 00:05:28,901 --> 00:05:34,811 And he is like, I was kind of expecting blood and guts and you know, extruding 109 00:05:34,811 --> 00:05:35,561 bone and 110 00:05:36,101 --> 00:05:36,731 Yeah. 111 00:05:36,761 --> 00:05:37,061 Yeah. 112 00:05:37,061 --> 00:05:38,591 He is like, you look fine. 113 00:05:38,591 --> 00:05:39,731 I'm like, I know. 114 00:05:39,731 --> 00:05:41,831 And they made me come anyway. 115 00:05:42,566 --> 00:05:46,166 Um, and so they were, yeah, so they did a CAT scan. 116 00:05:46,166 --> 00:05:51,926 They did, you know, I don't know, probably like 15 x-rays of my arms. 117 00:05:51,956 --> 00:05:54,296 'cause they were, they were all, you know, whatnot. 118 00:05:55,466 --> 00:05:59,636 And, uh, and then, you know, sent me home and they, they asked me if I wanted any 119 00:05:59,636 --> 00:06:03,956 pain medicine and I, I literally said no, because I remember the last time 120 00:06:03,956 --> 00:06:06,386 I was in the ER when I broke my nose. 121 00:06:06,626 --> 00:06:11,846 For those longtime fans of the show may remember that, um, I. 122 00:06:12,866 --> 00:06:15,206 They said, they said they gave me something for the pain, they gave me 123 00:06:15,206 --> 00:06:21,236 like a Tylenol and like the lowest level, like narcotic painkiller you can give. 124 00:06:21,536 --> 00:06:23,216 And they charged me $800 for that. 125 00:06:23,276 --> 00:06:24,956 So I was like, I'm good. 126 00:06:25,136 --> 00:06:27,146 Um, I'll, I'll sug it up. 127 00:06:28,346 --> 00:06:29,276 Um, 128 00:06:29,321 --> 00:06:31,116 I got a fifth of tequila somewhere in the house. 129 00:06:33,986 --> 00:06:34,826 or some edibles. 130 00:06:34,826 --> 00:06:35,816 Maybe some edibles. 131 00:06:36,266 --> 00:06:39,296 Um, but yeah, so it was, um, 132 00:06:40,046 --> 00:06:40,701 But I'm glad to. 133 00:06:42,101 --> 00:06:42,641 Yeah. 134 00:06:42,701 --> 00:06:43,091 Yeah. 135 00:06:43,091 --> 00:06:45,671 Amazingly, no major injuries. 136 00:06:45,671 --> 00:06:49,931 I've got some sore spots, you know, but no major injuries, 137 00:06:49,981 --> 00:06:52,651 So the podcast shall continue. 138 00:06:53,041 --> 00:06:56,161 Our series on ransomware shall continue. 139 00:06:56,761 --> 00:06:59,701 And, uh, we're taking a break with Mike this week. 140 00:06:59,791 --> 00:07:03,016 Um, and we're gonna talk about. 141 00:07:03,811 --> 00:07:06,451 Specifically the backup side of things. 142 00:07:06,601 --> 00:07:13,141 We've already done an episode or two that I will summarize as follows. 143 00:07:13,381 --> 00:07:17,221 Your backup server is, is at risk, right? 144 00:07:17,646 --> 00:07:17,867 It is 145 00:07:17,872 --> 00:07:18,601 at high 146 00:07:18,601 --> 00:07:19,141 risk 147 00:07:19,531 --> 00:07:21,451 Yeah, it is, it is at high risk. 148 00:07:21,451 --> 00:07:27,901 And, and, and there are, we have numerous, um, data points to back that up. 149 00:07:28,201 --> 00:07:31,771 My favorite, I think what did Dwayne say and what, first of all, what, who was 150 00:07:31,771 --> 00:07:33,331 Dwayne and what, what, what did he say? 151 00:07:33,916 --> 00:07:35,686 yeah, Dwayne is a red teamer, right? 152 00:07:35,686 --> 00:07:38,116 So he pretends to be the bad guy in attack systems. 153 00:07:39,031 --> 00:07:39,421 Yeah, 154 00:07:39,661 --> 00:07:43,141 I think he basically said, I love the backup system. 155 00:07:43,141 --> 00:07:46,801 That's the first system I target because I get access to that. 156 00:07:46,801 --> 00:07:48,901 I get access to all your data in your 157 00:07:48,901 --> 00:07:51,421 environment because everyone backs up into a single place. 158 00:07:52,531 --> 00:07:53,581 It's the key to the kingdom. 159 00:07:54,136 --> 00:07:54,556 Yeah. 160 00:07:54,556 --> 00:07:57,286 So if a red chamber thinks that, and by the way, if you haven't heard 161 00:07:57,286 --> 00:07:59,156 the Red Team episode, go back, Maybe 162 00:07:59,156 --> 00:07:59,696 two months. 163 00:07:59,696 --> 00:08:00,056 Yeah. 164 00:08:00,356 --> 00:08:03,266 Uh, there's an episode called, uh, you Know, about Red Team. 165 00:08:03,776 --> 00:08:08,066 And, he clarified that, you know, both the backup system itself in 166 00:08:08,066 --> 00:08:11,906 terms of how powerful it is, how much you get access to it, and also. 167 00:08:12,176 --> 00:08:16,796 In terms of how poorly it often is designed from a security standpoint. 168 00:08:16,796 --> 00:08:19,136 He talked about things like service accounts, right? 169 00:08:19,136 --> 00:08:21,266 He said he loves the, the backup service account. 170 00:08:21,776 --> 00:08:22,526 Uh, do you remember what he 171 00:08:22,526 --> 00:08:22,676 said 172 00:08:22,826 --> 00:08:25,256 default With the defaults password 173 00:08:25,646 --> 00:08:26,396 or no password? 174 00:08:26,546 --> 00:08:26,936 Yep. 175 00:08:27,476 --> 00:08:29,306 But do you remember what he said about what was unique 176 00:08:29,306 --> 00:08:29,636 about it? 177 00:08:30,326 --> 00:08:30,596 Yeah. 178 00:08:30,716 --> 00:08:34,556 With the backup service account, nothing gets logged in the system 179 00:08:35,421 --> 00:08:35,541 I 180 00:08:35,696 --> 00:08:38,876 you access it using the backup service account because it assumes you're 181 00:08:38,876 --> 00:08:41,906 gonna be using it all the time and reading everything in the file system. 182 00:08:41,906 --> 00:08:44,126 So I bother logging anything. 183 00:08:44,786 --> 00:08:45,896 Yeah, exactly. 184 00:08:46,076 --> 00:08:49,436 And so, you know, that I think that was an episode where we just said, 185 00:08:49,436 --> 00:08:52,376 listen, you really need to understand your backup server is at risk. 186 00:08:52,676 --> 00:08:58,046 We also had an episode or two where we talked about how to design the 187 00:08:58,046 --> 00:09:01,466 server itself, um, in order to. 188 00:09:02,336 --> 00:09:04,646 Better insulated from that risk. 189 00:09:04,916 --> 00:09:06,561 Do you remember the kinds of things we talked about there? 190 00:09:07,376 --> 00:09:12,926 Yeah, I think these were ideas such as segmentation. 191 00:09:12,926 --> 00:09:17,786 It also included, don't have your backup server connected to your 192 00:09:17,786 --> 00:09:20,006 normal active directory instance. 193 00:09:20,011 --> 00:09:24,926 Kind of keep it isolated, separate, um, make sure you're. 194 00:09:25,586 --> 00:09:27,386 Just to also the normal stuff, right? 195 00:09:27,386 --> 00:09:30,476 Like make sure you patch your systems, including the servers, 196 00:09:30,476 --> 00:09:30,686 right? 197 00:09:30,776 --> 00:09:32,966 Keep up to date on those. 198 00:09:33,536 --> 00:09:33,896 Yeah. 199 00:09:33,986 --> 00:09:34,316 Yeah. 200 00:09:34,316 --> 00:09:34,706 Uh, yeah. 201 00:09:34,706 --> 00:09:38,726 So all the, the, usual stuff of, um, you know, obviously patch management, 202 00:09:38,816 --> 00:09:41,426 password management, and MFA obviously, 203 00:09:41,516 --> 00:09:41,876 right? 204 00:09:42,056 --> 00:09:42,386 Yes. 205 00:09:42,386 --> 00:09:46,106 I think the big one that you mentioned that I harp on a lot is 206 00:09:46,106 --> 00:09:47,816 to separate it as much as possible. 207 00:09:47,816 --> 00:09:51,296 Don't have it on the active directory domain, don't have it. 208 00:09:51,416 --> 00:09:54,746 Um, you know, don't use the same username and password there 209 00:09:54,746 --> 00:09:56,216 that you use anywhere else. 210 00:09:56,711 --> 00:09:59,921 Um, I mean, that should be true anyway, but whatever. 211 00:10:00,231 --> 00:10:02,301 This is a practice you should have everywhere, but you 212 00:10:02,301 --> 00:10:03,411 should definitely have it here. 213 00:10:03,411 --> 00:10:08,331 And that is don't ever log in as root or administrator log in as you and become 214 00:10:08,331 --> 00:10:14,151 root or administrator you use, the concept of least privilege so that you can, uh, 215 00:10:14,181 --> 00:10:16,371 minimize the damage that any one person can do 216 00:10:17,086 --> 00:10:20,056 One I wanted to add there, because I know we've talked about this on 217 00:10:20,836 --> 00:10:25,696 episodes a while ago, is don't save your password to access your backup 218 00:10:25,696 --> 00:10:27,616 management system in your web browser. 219 00:10:27,976 --> 00:10:28,876 Yeah, please don't do that. 220 00:10:30,406 --> 00:10:31,096 Please don't do that. 221 00:10:32,041 --> 00:10:37,501 Um, the, and, and, and you we're, we're big proponents of password management 222 00:10:37,501 --> 00:10:42,481 here and, and and specifically password management systems, not your browser. 223 00:10:42,871 --> 00:10:45,061 Uh, again, the browser's better than nothing. 224 00:10:45,091 --> 00:10:47,221 Perhaps, perhaps not 225 00:10:47,221 --> 00:10:48,031 in this case. 226 00:10:48,311 --> 00:10:51,252 the Thing that, that I remember when we talked about third 227 00:10:51,252 --> 00:10:53,772 party password managers, it. 228 00:10:54,132 --> 00:10:59,862 Floored me the first time I installed Dashlane and it said, Hey, should 229 00:10:59,862 --> 00:11:03,612 we go get the passwords that you stored in your browser for you? 230 00:11:04,272 --> 00:11:06,972 And I'm like, wait, you can just get them, 231 00:11:08,307 --> 00:11:08,697 Yep. 232 00:11:09,012 --> 00:11:10,812 they're not like encrypted or anything. 233 00:11:10,812 --> 00:11:14,472 You can just ask the browser, Hey, what's the password you have for this? 234 00:11:14,862 --> 00:11:16,332 And they just sucked them outta there. 235 00:11:16,382 --> 00:11:19,052 Well, it's encrypted, but it probably just has API access. 236 00:11:19,052 --> 00:11:19,592 So as long as you 237 00:11:19,592 --> 00:11:21,002 granted API access. 238 00:11:21,362 --> 00:11:21,722 yeah. 239 00:11:22,322 --> 00:11:25,652 But if, if you can get access by api, any other system, 240 00:11:25,682 --> 00:11:26,672 software running on your 241 00:11:26,852 --> 00:11:31,622 Yeah, yeah, so, so some sort, some sort of third party password management system. 242 00:11:31,622 --> 00:11:34,442 And I do believe that it should be a different password management 243 00:11:34,442 --> 00:11:35,912 system than the rest of the world. 244 00:11:36,152 --> 00:11:38,912 Again, segregate, segregate, segregate, right? 245 00:11:38,912 --> 00:11:40,202 As much as possible. 246 00:11:40,412 --> 00:11:44,822 Obviously more than any other server, shut down any services that you don't need. 247 00:11:45,062 --> 00:11:48,152 Especially my favorite, the ransomware deployment protocol. 248 00:11:48,692 --> 00:11:49,292 Um, 249 00:11:49,712 --> 00:11:52,042 Otherwise known as Windows RDP. 250 00:11:52,422 --> 00:11:55,817 For those who don't, who may not recognize that acronym. 251 00:11:56,072 --> 00:12:04,652 Um, and, um, but today I wanted to talk about how do we, um, design the 252 00:12:04,652 --> 00:12:16,562 backups themselves, uh, to be more useful in times of ransomware, right? 253 00:12:16,562 --> 00:12:19,442 Probably the first thing you need to do is actually do backups, 254 00:12:21,662 --> 00:12:21,932 right? 255 00:12:21,932 --> 00:12:23,372 That should be like your step one of your 256 00:12:23,372 --> 00:12:23,972 strategy. 257 00:12:24,227 --> 00:12:28,787 I live in this, I live in this fantasy world Prasanna where like 258 00:12:29,687 --> 00:12:31,997 everybody does backups, right? 259 00:12:32,697 --> 00:12:36,537 Actually just before this recording, I was on another. 260 00:12:37,422 --> 00:12:40,992 You know, uh, another thing, and, and, and I realize that I'm speaking 261 00:12:40,992 --> 00:12:45,102 to laypeople in this particular recording, uh, people in the legal 262 00:12:45,102 --> 00:12:50,892 profession, and I was asked like, what is a backup and what is a restore? 263 00:12:51,342 --> 00:12:55,092 And it's like, and, and I know that, you know, they're, they're catering 264 00:12:55,122 --> 00:12:57,042 to an audience that doesn't understand 265 00:12:57,042 --> 00:13:00,822 this stuff, but I just live in this world where everybody backs up their stuff, 266 00:13:01,197 --> 00:13:01,617 Yeah, 267 00:13:02,157 --> 00:13:03,687 that's why you need the 88 in the room. 268 00:13:03,687 --> 00:13:04,287 IE me. 269 00:13:04,822 --> 00:13:05,652 Exactly. 270 00:13:05,952 --> 00:13:10,002 Everybody runs a third party backup of their iPhone because they 271 00:13:10,002 --> 00:13:11,802 know that iCloud is not a backup. 272 00:13:12,462 --> 00:13:13,962 iCloud is not a backup. 273 00:13:14,562 --> 00:13:16,422 iCloud is a synchronization product. 274 00:13:17,052 --> 00:13:18,852 Uh, not a backup product. 275 00:13:19,017 --> 00:13:20,817 go listen to that episode if you're interested. 276 00:13:20,982 --> 00:13:21,402 Yeah. 277 00:13:21,402 --> 00:13:24,492 Go listen to the how to properly back up your iPhone. 278 00:13:24,592 --> 00:13:29,212 When we think about designing a backup system for the purposes of, of, uh, 279 00:13:29,242 --> 00:13:35,992 responding to a ransomware attack and then being able to recover, I think 280 00:13:35,992 --> 00:13:38,452 it's important to think about, um. 281 00:13:39,502 --> 00:13:43,612 A, a lot of things in terms of how does ransomware typically behave? 282 00:13:43,792 --> 00:13:48,472 How does a ransomware response event typically take place? 283 00:13:49,132 --> 00:13:54,742 And so I, I think the first thing to talk about is this concept of dwell time. 284 00:13:54,772 --> 00:13:55,277 Do you wanna talk about that? 285 00:13:56,717 --> 00:14:01,247 Yeah, so I think a lot of, and if you go back and listen to some of the 286 00:14:01,247 --> 00:14:04,457 previous episodes and if, if you look, listen to what Mike has talked about 287 00:14:04,457 --> 00:14:07,187 before, I think everyone thinks, oh. 288 00:14:07,822 --> 00:14:12,682 I got hit with ransomware, I just got infected, and then boom, just 289 00:14:12,682 --> 00:14:15,892 that next instance, everything in my environment is encrypted, 290 00:14:16,762 --> 00:14:17,092 right? 291 00:14:17,407 --> 00:14:19,027 That's what I think a lot of people think about it. 292 00:14:19,027 --> 00:14:22,237 It's almost, but it's not how that works. 293 00:14:22,537 --> 00:14:27,157 It's like you get infected by a disease, right? 294 00:14:27,157 --> 00:14:31,357 It might take you a day before you start to get like a cold, and 295 00:14:31,357 --> 00:14:34,987 then maybe a couple days later you start to spike a fever, right? 296 00:14:34,987 --> 00:14:38,287 And so similarly for ransomware, we talk about something called a 297 00:14:38,287 --> 00:14:41,197 dwell time, which is how long is. 298 00:14:41,557 --> 00:14:44,947 The ransomware actually in your environment, even though it may 299 00:14:44,947 --> 00:14:47,437 not be actively encrypting data, 300 00:14:47,737 --> 00:14:47,977 Right. 301 00:14:47,977 --> 00:14:51,187 Or exfiltrating data or whatever it is, but it's already got a 302 00:14:51,187 --> 00:14:53,077 foothold in your environment and it 303 00:14:53,077 --> 00:14:54,667 exists somewhere in your network. 304 00:14:55,517 --> 00:14:59,382 Yeah, and and I, I think it's important to understand, again, and we've talked 305 00:14:59,582 --> 00:15:01,582 about this on other episodes, that. 306 00:15:02,467 --> 00:15:05,347 Remember that ransomware isn't a single piece of software, 307 00:15:05,527 --> 00:15:05,917 right? 308 00:15:06,367 --> 00:15:09,637 Um, maybe the actual ransomware is a single piece of software, but the 309 00:15:09,637 --> 00:15:15,547 entire, there there is a suite of tools that ransomware actors are 310 00:15:15,547 --> 00:15:19,717 going to be using to, to get, number one, to get into your environment. 311 00:15:19,717 --> 00:15:22,207 Number two, two, spread around in your environment and to figure 312 00:15:22,207 --> 00:15:25,897 out what's going on in your environment and that it's that final. 313 00:15:26,257 --> 00:15:29,767 Uh, tool, the one that you're, you know, the actual ransomware tool 314 00:15:29,767 --> 00:15:35,857 that's doing the, the, uh, encryption and or doing extraction, right? 315 00:15:35,857 --> 00:15:36,607 Doing, um, 316 00:15:37,147 --> 00:15:38,047 exfiltration. 317 00:15:38,617 --> 00:15:42,487 But before that happens, you're right. 318 00:15:42,487 --> 00:15:47,317 There is this, this process of like going through the network 319 00:15:47,317 --> 00:15:49,687 and figuring out what is, um. 320 00:15:50,572 --> 00:15:52,432 You know, figuring out what they could do. 321 00:15:52,612 --> 00:15:57,652 There was a great story that Mike talked about where he said that they were in 322 00:15:57,652 --> 00:16:03,862 an environment and they were doing a, um, they did a tabletop, and during that 323 00:16:03,862 --> 00:16:08,632 tabletop they used the incident response plan, and they obviously shared the 324 00:16:08,632 --> 00:16:10,522 incident response plan around everywhere. 325 00:16:10,972 --> 00:16:13,762 And what they found out was they got a, they got a ransomware 326 00:16:13,762 --> 00:16:15,142 attack right after this. 327 00:16:15,262 --> 00:16:18,622 And what they found out was that, um. 328 00:16:19,447 --> 00:16:23,857 That they had already been attacked and that the attacker was in 329 00:16:23,857 --> 00:16:26,377 their system for quite a while. 330 00:16:26,377 --> 00:16:31,207 And so he got to see the, you know, the, the, uh, what do you call it, the 331 00:16:31,207 --> 00:16:32,647 incident response plan and all this stuff. 332 00:16:32,647 --> 00:16:36,247 And they got to see like how much insurance they, all this stuff, right? 333 00:16:36,967 --> 00:16:38,887 Uh, so that, that's a really big time. 334 00:16:38,887 --> 00:16:42,397 And so what's the concern when 335 00:16:42,397 --> 00:16:44,497 we start talking about restoring what? 336 00:16:44,922 --> 00:16:48,732 oh, I was just going to mention like, I know this has come up in 337 00:16:48,732 --> 00:16:53,412 the past, but one of the banks that I bank with that's a credit union, 338 00:16:53,412 --> 00:16:54,857 they were hit with ransomware. 339 00:16:56,347 --> 00:16:59,257 Basically over the 4th of July that shut down everything. 340 00:16:59,817 --> 00:17:00,237 Mm-Hmm 341 00:17:00,307 --> 00:17:00,667 Right. 342 00:17:01,177 --> 00:17:06,757 But they finally published a, a analysis of what happened. 343 00:17:07,177 --> 00:17:12,007 And they say that, so July 4th or July 1st is when everything got shut 344 00:17:12,007 --> 00:17:12,487 down. 345 00:17:12,517 --> 00:17:14,107 Everything was encrypted, right? 346 00:17:14,677 --> 00:17:17,917 They said that they were in their network starting May 23rd. 347 00:17:19,387 --> 00:17:20,827 So six weeks almost. 348 00:17:21,232 --> 00:17:23,032 That would be the, that would be the dwell time. 349 00:17:23,242 --> 00:17:23,632 Right. 350 00:17:24,382 --> 00:17:28,252 And if you look, if you do some Googling, you'll find that the average 351 00:17:28,252 --> 00:17:30,172 dwell time is actually really long. 352 00:17:30,742 --> 00:17:31,162 Right. 353 00:17:31,432 --> 00:17:36,262 Um, like the mean dwell time, last time I looked was like close to 90 days. 354 00:17:36,832 --> 00:17:38,962 Uh, which means that there's. 355 00:17:39,607 --> 00:17:41,137 Ones that are way 356 00:17:41,737 --> 00:17:42,907 beyond that. 357 00:17:42,907 --> 00:17:43,297 Right. 358 00:17:43,867 --> 00:17:46,717 Um, it's not like they're all 90 days and it works out that way. 359 00:17:46,792 --> 00:17:49,312 Yeah, because I think for these actors, right, there are 360 00:17:49,312 --> 00:17:50,362 two things they want to do. 361 00:17:50,362 --> 00:17:54,082 One, they wanna spread everywhere, so they have access to as much as possible. 362 00:17:54,082 --> 00:17:56,902 And two, they want to figure out what's valuable in your environment, 363 00:17:57,427 --> 00:17:57,787 Yeah. 364 00:17:57,832 --> 00:17:58,162 right? 365 00:17:58,342 --> 00:17:59,002 And so they 366 00:17:59,197 --> 00:18:00,517 beat in there as long as they can. 367 00:18:00,517 --> 00:18:00,757 Yeah. 368 00:18:01,042 --> 00:18:05,002 And so if they go in and immediately start encrypting, right, using the 369 00:18:05,002 --> 00:18:08,272 ransomware, you're going to notice and now they've lost that opportunity. 370 00:18:08,272 --> 00:18:09,112 So it's kind of a 371 00:18:09,112 --> 00:18:10,612 balance on their side, right? 372 00:18:10,852 --> 00:18:14,722 They want to be in there spreading, observing, but the longer they're 373 00:18:14,722 --> 00:18:17,512 in there, then the more likely it is for them to be detected as well. 374 00:18:17,512 --> 00:18:18,892 So it's kind of this balance. 375 00:18:19,417 --> 00:18:19,627 Yeah. 376 00:18:19,627 --> 00:18:20,587 It is a balance. 377 00:18:20,677 --> 00:18:24,127 Um, but the longer they're in there, the bigger the possible reward. 378 00:18:24,397 --> 00:18:27,427 The other thing that they could do, and there, there's a couple 379 00:18:27,427 --> 00:18:30,157 things that they could do while they're in there a long time. 380 00:18:30,907 --> 00:18:34,087 One is they could start encrypt, encrypting. 381 00:18:34,447 --> 00:18:38,557 And again, this is one where I'm gonna describe what I've been told has happened. 382 00:18:38,797 --> 00:18:43,717 I haven't verified this, but it seems reasonable to me, and that 383 00:18:43,717 --> 00:18:48,667 is that they start encrypting stuff that nobody's looking at. 384 00:18:49,507 --> 00:18:49,837 Like the 385 00:18:49,882 --> 00:18:52,042 Right, like older data that 386 00:18:52,042 --> 00:18:53,092 nobody's looking at. 387 00:18:53,542 --> 00:18:57,142 And, um, they do that because they could, they could do it and get away 388 00:18:57,142 --> 00:18:58,432 with it because nobody's looking. 389 00:18:59,182 --> 00:19:00,382 Uh, number one. 390 00:19:00,382 --> 00:19:03,442 The other thing, what do you, what do you think is the other thing that 391 00:19:03,442 --> 00:19:07,597 they could potentially do if they're in your system for a long time? 392 00:19:08,055 --> 00:19:08,985 Exfil trading data. 393 00:19:09,255 --> 00:19:11,025 Exfiltrating data, right? 394 00:19:11,175 --> 00:19:13,365 More than likely you will end up paying the ransom. 395 00:19:13,875 --> 00:19:15,975 Um, even though I hate the idea and all that kind of 396 00:19:15,975 --> 00:19:16,455 stuff, right? 397 00:19:16,455 --> 00:19:18,825 But it's a very different argument of like, oh no, I 398 00:19:18,825 --> 00:19:19,665 don't have to pay the ransom. 399 00:19:19,905 --> 00:19:22,815 Ah, I got good backups in a DR plan and it's a response plan. 400 00:19:22,815 --> 00:19:24,945 They're like, yeah, but we still all your data, we're gonna 401 00:19:24,945 --> 00:19:25,875 tell everybody what you did. 402 00:19:27,030 --> 00:19:32,445 Um, um, the longer they're in there, the easier exfiltration is, right? 403 00:19:32,655 --> 00:19:34,185 Because they can do it slower. 404 00:19:34,185 --> 00:19:35,290 They can, you know, send it out. 405 00:19:35,805 --> 00:19:40,005 Which again is why I continue to say, please figure 406 00:19:40,005 --> 00:19:42,975 out some way to track outgoing traffic. 407 00:19:43,155 --> 00:19:45,885 Yeah, which is actually what happened at this credit union. 408 00:19:45,885 --> 00:19:49,395 They ended up exfiltrating data from their database in addition 409 00:19:49,395 --> 00:19:51,075 to encrypting everything. 410 00:19:52,035 --> 00:19:53,445 So social security 411 00:19:53,745 --> 00:19:54,795 Are they back up by the way, 412 00:19:55,215 --> 00:19:55,395 that 413 00:19:55,635 --> 00:19:57,915 Uh, yeah, I think they are back up. 414 00:19:57,945 --> 00:20:00,615 Uh, last month they had to send out paper statements 'cause they 415 00:20:00,615 --> 00:20:03,735 weren't fully up and running, but I believe now they're up and running. 416 00:20:03,735 --> 00:20:06,735 But all customer data is out there on the dark web. 417 00:20:06,945 --> 00:20:07,305 Yay. 418 00:20:08,025 --> 00:20:15,555 Yeah, because, uh, again, fans of the podcast may remember that, uh, my medical, 419 00:20:15,765 --> 00:20:22,755 uh, group got attacked with ransomware in May, and I found out last week 420 00:20:22,965 --> 00:20:25,155 they're still not fully up and running. 421 00:20:26,490 --> 00:20:27,090 Yeah. 422 00:20:27,285 --> 00:20:28,425 maybe they don't have backups. 423 00:20:28,425 --> 00:20:29,535 Curtis is what I'm guessing. 424 00:20:29,970 --> 00:20:31,380 I, I don't even wanna know. 425 00:20:31,800 --> 00:20:32,460 Um, 426 00:20:32,850 --> 00:20:35,910 all I know is that for months, the only way I could make an appointment 427 00:20:35,910 --> 00:20:40,140 was to go into, I could, I had to physically drive into the doctor's 428 00:20:40,140 --> 00:20:41,850 office, which luckily for me is like, I. 429 00:20:42,285 --> 00:20:43,245 15 minutes from my 430 00:20:43,245 --> 00:20:47,145 office, uh, I could go, you know, drive there, make an appointment. 431 00:20:47,295 --> 00:20:49,905 And it was, it was actually kind of good because it meant it was a pain to 432 00:20:49,905 --> 00:20:52,095 make an appointment, which meant that it was easy to make an appointment. 433 00:20:52,335 --> 00:20:53,535 Is that, you understand what I'm saying? 434 00:20:53,925 --> 00:20:57,645 It was, it was logistically difficult to make an appointment, which made it 435 00:20:57,645 --> 00:20:59,325 easier to make an appointment once you 436 00:20:59,325 --> 00:21:00,315 were there because 437 00:21:00,885 --> 00:21:01,065 No 438 00:21:01,125 --> 00:21:02,295 many people, yeah. 439 00:21:02,295 --> 00:21:03,525 No one wanted to deal with it. 440 00:21:03,525 --> 00:21:03,765 Right. 441 00:21:04,335 --> 00:21:07,965 Uh, but yeah, what I found out, again, this was just last week, 442 00:21:07,965 --> 00:21:09,525 that the phone part of the. 443 00:21:09,975 --> 00:21:12,165 System is still not back up and running. 444 00:21:12,795 --> 00:21:14,625 Um, yeah. 445 00:21:14,715 --> 00:21:15,015 Yeah. 446 00:21:15,195 --> 00:21:16,395 Com completely crazy. 447 00:21:16,575 --> 00:21:17,115 What's worse? 448 00:21:17,115 --> 00:21:19,275 Is it pre It's pretending that it's up and running. 449 00:21:19,425 --> 00:21:19,605 I 450 00:21:19,605 --> 00:21:24,195 called the number, I waited on hold for like 20 minutes and 451 00:21:24,195 --> 00:21:25,575 they're like, you know, press one. 452 00:21:25,575 --> 00:21:28,095 You know, to, if you're a provider, press one. 453 00:21:28,155 --> 00:21:29,475 If you're a patient, press two. 454 00:21:29,475 --> 00:21:33,915 And if you wanna speak to you wanna make an appointment, press two. 455 00:21:34,065 --> 00:21:34,545 Okay. 456 00:21:34,875 --> 00:21:37,125 And, and then it's like for the. 457 00:21:37,560 --> 00:21:42,870 Chula Vista office, press one for the Encinitas office, press two, you 458 00:21:42,870 --> 00:21:44,280 know, and I'm like eight, like I'm 459 00:21:44,580 --> 00:21:47,250 gonna sit there with, you know, and then I finally press eight. 460 00:21:47,370 --> 00:21:48,780 And then it's like, please hold. 461 00:21:48,900 --> 00:21:53,910 And then, and it was like, it was like 20 minutes later, click. 462 00:21:54,870 --> 00:21:55,090 Ah, 463 00:21:56,310 --> 00:21:57,210 I was like, what the 464 00:21:57,380 --> 00:21:57,730 hello. 465 00:21:59,010 --> 00:22:01,290 So I was like, well, if I'd known first off, I'd have known 466 00:22:01,290 --> 00:22:02,490 it was gonna take so long. 467 00:22:02,640 --> 00:22:04,590 I would've just, I would've just drove over there. 468 00:22:05,070 --> 00:22:07,620 Let alone, if I'd have known, maybe, you know, maybe I would, should have 469 00:22:07,620 --> 00:22:10,295 done is got on the phone and then Dr. 470 00:22:10,320 --> 00:22:11,010 And then drove. 471 00:22:11,070 --> 00:22:12,000 And then drove there. 472 00:22:12,930 --> 00:22:13,320 Yeah. 473 00:22:13,440 --> 00:22:17,580 So anyway, so I made my appointment, um, and and then, and then 474 00:22:17,580 --> 00:22:18,930 proceeded to fall down the stairs. 475 00:22:19,470 --> 00:22:24,270 Um, all right, so if there's a long dwell time, what do we need to be 476 00:22:24,270 --> 00:22:27,960 talking about with regards to, um, 477 00:22:28,800 --> 00:22:29,430 backups? 478 00:22:30,150 --> 00:22:30,720 Well. 479 00:22:30,750 --> 00:22:38,160 So when you are getting to the point of recovering from your ransomware 480 00:22:38,160 --> 00:22:41,790 incident, you wanna make sure that the data you're restoring. 481 00:22:42,585 --> 00:22:44,625 Is clean, right? 482 00:22:44,625 --> 00:22:48,405 That it doesn't contain any bits and pieces of the ransomware, 483 00:22:49,005 --> 00:22:49,335 right, 484 00:22:49,365 --> 00:22:53,625 Or any of those other intrusions that have happened in the past. 485 00:22:53,625 --> 00:22:54,075 So you 486 00:22:54,075 --> 00:22:56,895 really wanna find a clean point so it doesn't sort of 487 00:22:57,015 --> 00:22:58,425 restart itself and you end up 488 00:22:59,205 --> 00:23:00,165 Right. 489 00:23:00,855 --> 00:23:06,105 So how would that affect, how would a long dwell time affect 490 00:23:06,105 --> 00:23:08,145 the design of your backup system? 491 00:23:08,475 --> 00:23:11,235 You need a longer retention because you need to make 492 00:23:11,235 --> 00:23:12,855 sure your backup is 493 00:23:12,855 --> 00:23:15,015 a clean backup before you got infected. 494 00:23:15,765 --> 00:23:21,135 I say this because it's very common, and again, I, I had a 495 00:23:21,135 --> 00:23:22,575 conversation with Mike about that. 496 00:23:22,995 --> 00:23:24,405 It's very common. 497 00:23:25,050 --> 00:23:30,150 For people to say, oh, well I only need 90 days, right? 498 00:23:30,210 --> 00:23:33,300 I only need 90 days for my, my backup retention. 499 00:23:33,300 --> 00:23:35,250 And I understand the reasonings behind that. 500 00:23:35,250 --> 00:23:35,580 Right. 501 00:23:36,120 --> 00:23:43,470 Um, and uh, and, and Mike was saying that it's very common he's seen it in 502 00:23:43,470 --> 00:23:48,180 the field where people go to restore their stuff and the retention period 503 00:23:48,180 --> 00:23:51,750 of their backups is less than the dwell time of the product, and they're 504 00:23:51,750 --> 00:23:53,310 unable to successfully restore. 505 00:23:54,030 --> 00:23:55,140 I have a question about this 506 00:23:55,140 --> 00:23:55,500 though. 507 00:23:56,610 --> 00:23:57,600 So I totally get it. 508 00:23:57,600 --> 00:23:59,610 Ideally, you want a clean backup, but 509 00:23:59,850 --> 00:24:03,000 if you go, say you have to go back three months, right? 510 00:24:03,390 --> 00:24:05,910 So you restore from something from three months, and now you 511 00:24:05,910 --> 00:24:07,500 have to sort of roll forward 512 00:24:08,040 --> 00:24:09,780 to get back to the current point in time. 513 00:24:10,275 --> 00:24:10,545 yeah. 514 00:24:10,545 --> 00:24:12,195 That's the next thing to talk about. 515 00:24:12,300 --> 00:24:12,630 yeah. 516 00:24:12,690 --> 00:24:21,390 My question though is, is that better than restoring a non clean backup and 517 00:24:21,390 --> 00:24:23,730 then surgically going and cleaning it up? 518 00:24:24,510 --> 00:24:25,110 Yeah. 519 00:24:25,440 --> 00:24:28,170 So the, that's a really good question. 520 00:24:28,230 --> 00:24:29,280 Uh, let's finish this 521 00:24:29,280 --> 00:24:31,170 point and then let's go to that point. 522 00:24:31,620 --> 00:24:36,030 So all I'm saying is you, you know, to have options, 523 00:24:36,150 --> 00:24:40,140 you need a much longer restore, uh, a much longer retention period than 90 days. 524 00:24:40,560 --> 00:24:41,070 Um, 525 00:24:41,430 --> 00:24:43,350 my, my general thing would be like a year. 526 00:24:44,040 --> 00:24:45,690 Right, like a minimum of a year. 527 00:24:45,690 --> 00:24:50,010 I would actually, I would say for a business minimum of, of 13 months. 528 00:24:50,280 --> 00:24:52,890 Because sometimes you want stuff from like the annual report from a 529 00:24:52,890 --> 00:24:53,520 year ago. 530 00:24:53,970 --> 00:24:58,200 Um, I don't see any problem with going a couple of years, any 531 00:24:58,200 --> 00:25:00,690 retention, you know, I don't want you to go much longer than that. 532 00:25:01,230 --> 00:25:04,140 I was gonna say, don't consider this an archive. 533 00:25:04,230 --> 00:25:05,880 Don't use your backups for archiving, 534 00:25:06,420 --> 00:25:07,050 but yes. 535 00:25:07,290 --> 00:25:11,880 Um, but the, you know, there, there's nothing wrong with having a couple 536 00:25:11,880 --> 00:25:14,940 of years retention for your backups, maybe even three years right now. 537 00:25:15,000 --> 00:25:16,950 That's not really a ransomware defense at that point. 538 00:25:17,520 --> 00:25:20,340 Um, but you want options. 539 00:25:20,520 --> 00:25:21,030 Okay. 540 00:25:21,900 --> 00:25:26,040 Um, what, so my, Mike and I spent a quite a bit of time talking 541 00:25:26,040 --> 00:25:28,170 about this, this issue of. 542 00:25:29,730 --> 00:25:35,490 Clean versus, you know, completely clean versus clean versus cleaning it. 543 00:25:36,450 --> 00:25:37,290 Does that make sense? 544 00:25:37,290 --> 00:25:37,620 Right. 545 00:25:38,100 --> 00:25:42,120 So he, he explained a, a couple of things. 546 00:25:42,390 --> 00:25:47,370 One, and, and it, the, the, I'll just say this is a complicated question 547 00:25:48,270 --> 00:25:53,580 and it's, and it's more complicated when we start talking about file servers. 548 00:25:54,210 --> 00:25:58,050 So first off, let's just, let's just do the, the, the, the. 549 00:25:58,410 --> 00:26:00,480 The, you know, good, better, best, right? 550 00:26:01,080 --> 00:26:03,480 the the best. 551 00:26:04,110 --> 00:26:09,390 When you ask someone that's responsible for that, that has been 552 00:26:09,390 --> 00:26:14,100 through this, they will tell you that the best thing you can do is a 553 00:26:14,100 --> 00:26:18,870 complete wipe and restore afterwards, 554 00:26:19,770 --> 00:26:22,740 a, a complete wipe and a restore, especially when we're 555 00:26:22,740 --> 00:26:24,180 talking about the system, 556 00:26:24,900 --> 00:26:25,290 right? 557 00:26:25,290 --> 00:26:26,310 The os. 558 00:26:27,225 --> 00:26:28,725 Um, that 559 00:26:28,860 --> 00:26:30,030 a clean slate, right. 560 00:26:30,045 --> 00:26:31,635 start from a clean slate. 561 00:26:31,695 --> 00:26:32,175 Okay. 562 00:26:32,235 --> 00:26:33,855 Because there, there's two different things here. 563 00:26:33,855 --> 00:26:38,055 There's restoring the systems and then there's restoring the applications. 564 00:26:38,415 --> 00:26:39,585 There's actually three, and then there's 565 00:26:39,585 --> 00:26:42,135 restoring sort of just d like unstructured data, 566 00:26:42,135 --> 00:26:42,465 right? 567 00:26:43,605 --> 00:26:49,995 So regarding the systems, uh, he feels pretty strongly that this 568 00:26:49,995 --> 00:26:54,825 should be a, a clean wipe install. 569 00:26:55,665 --> 00:27:01,425 It is possible, it is possible to do what you're talking about where you restore 570 00:27:01,425 --> 00:27:07,155 the, the system and you and you're able to find, you know, you find out what 571 00:27:07,155 --> 00:27:11,085 variant of ransomware you have, you find out what tools that variant installs. 572 00:27:11,085 --> 00:27:12,585 You uninstall those tools. 573 00:27:13,005 --> 00:27:14,145 The concern 574 00:27:14,745 --> 00:27:15,165 with 575 00:27:15,165 --> 00:27:16,065 might have missed something. 576 00:27:16,275 --> 00:27:20,115 restore and yeah, but not just that, but. 577 00:27:21,045 --> 00:27:21,495 You. 578 00:27:21,525 --> 00:27:25,485 The big thing is did it install something like in the boot block 579 00:27:26,205 --> 00:27:28,395 to basically re-enable, you know, 580 00:27:28,425 --> 00:27:28,695 Yeah. 581 00:27:28,995 --> 00:27:32,895 if you can also fix that, right? 582 00:27:33,315 --> 00:27:33,585 yeah. 583 00:27:33,645 --> 00:27:37,185 Well, and I think this is some of the challenges today too. 584 00:27:37,185 --> 00:27:41,685 I don't know if you saw, but there's a new malware variant that 585 00:27:41,745 --> 00:27:44,145 injects itself in the UEFI boot. 586 00:27:44,745 --> 00:27:45,255 Which is 587 00:27:45,435 --> 00:27:47,475 literally baked into the motherboards, 588 00:27:47,865 --> 00:27:48,165 right? 589 00:27:48,165 --> 00:27:49,575 That's supposed to be super secure. 590 00:27:49,575 --> 00:27:53,715 And so in cases like that, you're basically hosed, right? 591 00:27:54,285 --> 00:27:57,615 You don't even wanna restore that data because it's always going, like 592 00:27:57,615 --> 00:28:01,155 you said, it always keep coming back over and over, no matter what you do, 593 00:28:01,860 --> 00:28:02,190 You. 594 00:28:02,340 --> 00:28:02,790 You know what? 595 00:28:02,790 --> 00:28:04,220 I'm gonna pull a Prasanna. 596 00:28:04,820 --> 00:28:05,090 what, 597 00:28:06,670 --> 00:28:07,710 Prasanna, you 598 00:28:07,710 --> 00:28:09,060 just used an acronym. 599 00:28:09,450 --> 00:28:09,960 what is. 600 00:28:09,960 --> 00:28:11,010 UEFI? 601 00:28:12,540 --> 00:28:13,320 I know what it is. 602 00:28:13,320 --> 00:28:16,770 I just, I just realized I have absolutely no idea what that stands for. 603 00:28:17,130 --> 00:28:19,620 But first off, what is A-U-F-U-E-F-I? 604 00:28:19,620 --> 00:28:19,830 Boo. 605 00:28:19,830 --> 00:28:20,415 What are you talking about? 606 00:28:21,315 --> 00:28:28,665 So UEFI boot is how the system boots up based on, so in the past, right, 607 00:28:28,665 --> 00:28:30,135 you had the master boot record and 608 00:28:30,135 --> 00:28:33,435 it had sort of certain blocks where it knew where to go in order to load it, 609 00:28:33,825 --> 00:28:38,715 in order to make things larger because there were certain limitations and 610 00:28:38,715 --> 00:28:45,015 also more secure like windows and other things use what they call A-U-E-F-I mode. 611 00:28:45,400 --> 00:28:48,550 In order to be able to boot your operating system. 612 00:28:49,250 --> 00:28:52,150 It stands for Unified Extensible Firmware Interface. 613 00:28:52,296 --> 00:28:54,606 So it's basically the, it's the next generation. 614 00:28:54,606 --> 00:28:56,256 It's been that way for a long time. 615 00:28:56,286 --> 00:28:56,646 Right. 616 00:28:56,766 --> 00:28:57,966 Um, right. 617 00:28:58,626 --> 00:29:03,036 Uh, you know, it's been a long time since MBR was that Master boot record. 618 00:29:03,276 --> 00:29:03,366 The 619 00:29:03,366 --> 00:29:08,316 NBR was the only option, but yeah, so this is just what I'm talking about. 620 00:29:08,346 --> 00:29:12,666 I I guess, again, and this is why I'm, you know, I'm such a fan 621 00:29:12,666 --> 00:29:14,976 of having an expert in the room. 622 00:29:15,126 --> 00:29:19,206 This is why you bring in somebody like Black Swan Security, right. 623 00:29:19,326 --> 00:29:22,536 Uh, to, to to come in, which is my Mike's company. 624 00:29:23,736 --> 00:29:26,616 But the, the. 625 00:29:27,216 --> 00:29:31,416 But to go back to the design issue, there are two things that you 626 00:29:31,431 --> 00:29:32,946 want to make sure you have, right? 627 00:29:34,236 --> 00:29:37,866 I'm gonna say three things, but two things that you wanna make sure you 628 00:29:37,866 --> 00:29:43,176 have is, you know, the retention period, and also a high enough frequency 629 00:29:43,776 --> 00:29:47,376 that you know you have, again, you have options. 630 00:29:47,616 --> 00:29:50,496 What does frequency, when we talk about. 631 00:29:51,021 --> 00:29:54,021 There's a couple of, couple of acronyms that we, we actually, we 632 00:29:54,021 --> 00:29:55,521 actually haven't said it in a while. 633 00:29:55,521 --> 00:29:57,141 You know what else we haven't said in a while? 634 00:29:57,741 --> 00:29:57,981 Yeah. 635 00:29:57,981 --> 00:29:59,061 RTO and RPO. 636 00:29:59,361 --> 00:30:00,081 Which one? 637 00:30:00,141 --> 00:30:00,441 You know what? 638 00:30:00,441 --> 00:30:02,031 What else we haven't said in a while? 639 00:30:02,946 --> 00:30:03,426 What. 640 00:30:03,741 --> 00:30:04,671 The 3, 2, 1 rule. 641 00:30:05,511 --> 00:30:08,031 The three 12 hasn't come up in a while. 642 00:30:08,301 --> 00:30:09,801 We've been talking about ransomware so much. 643 00:30:09,801 --> 00:30:13,491 We haven't talked enough about backup, but so when we talk about RTO and 644 00:30:13,491 --> 00:30:17,451 RPO, right, recovery time objective, that's how fast you can, you, you, 645 00:30:17,876 --> 00:30:21,831 you, you want to be able to restore and then recovery point objective, 646 00:30:21,921 --> 00:30:24,231 how much data you're allowed to lose. 647 00:30:24,891 --> 00:30:28,071 So frequency is going to impact which of those, 648 00:30:28,386 --> 00:30:29,076 RPO 649 00:30:29,916 --> 00:30:31,536 and RTO technically 650 00:30:32,946 --> 00:30:36,366 potentially depend, depends, right? 651 00:30:36,441 --> 00:30:36,831 okay. 652 00:30:36,831 --> 00:30:37,041 Yeah. 653 00:30:37,041 --> 00:30:38,211 I guess you're, I guess you're right. 654 00:30:38,271 --> 00:30:38,511 Yeah. 655 00:30:38,511 --> 00:30:39,231 Yeah, yeah, yeah. 656 00:30:39,681 --> 00:30:44,001 Um, it, I, I, think maybe like levels and things 657 00:30:44,091 --> 00:30:49,581 combined with frequency might affect, affect your RTO, so, um, it's just so 658 00:30:49,581 --> 00:30:51,471 funny, like I've spent so much time. 659 00:30:51,861 --> 00:30:55,491 In this new world where we don't do levels right, we just do one full and 660 00:30:55,641 --> 00:30:58,101 incrementals forever, which is the way all backups should be done, but whatever. 661 00:30:58,251 --> 00:30:58,881 I'd digress. 662 00:30:59,601 --> 00:31:03,711 Um, the, um, I, I really do think the concept of like 663 00:31:03,711 --> 00:31:06,276 repeated fulls is really a, a, a 664 00:31:06,276 --> 00:31:06,861 concept that 665 00:31:06,861 --> 00:31:07,911 needs to be done away with. 666 00:31:07,911 --> 00:31:08,391 Yeah. 667 00:31:08,841 --> 00:31:13,671 But, um, so you wanna make sure you have a long enough, um. 668 00:31:14,076 --> 00:31:14,916 Retention period. 669 00:31:15,006 --> 00:31:17,766 You wanna make sure you have a frequent enough backup. 670 00:31:18,276 --> 00:31:22,026 Uh, and then what I want to talk about this is, this is my third thing that 671 00:31:22,026 --> 00:31:28,086 I'm talking on the end of my two things, and that is you need options during a 672 00:31:28,086 --> 00:31:35,046 restore because when you are, uh, when I asked Mike to sort of walk through 673 00:31:35,556 --> 00:31:42,156 what a, what he felt was like a typical restore scenario and what he described. 674 00:31:42,726 --> 00:31:49,146 Was many, many restores of the same system that then allowed you to 675 00:31:49,146 --> 00:31:53,106 pick apart, to say, okay, do you know we wanna restore this version? 676 00:31:53,766 --> 00:31:55,056 Nope, that one's infected. 677 00:31:55,056 --> 00:31:55,686 We wanna restore 678 00:31:55,686 --> 00:31:56,136 this one. 679 00:31:56,196 --> 00:31:56,466 No. 680 00:31:56,466 --> 00:31:57,276 And that's infected. 681 00:31:57,276 --> 00:32:00,666 And keep going backwards until you get to a system that is 682 00:32:01,176 --> 00:32:02,526 not infected. 683 00:32:02,616 --> 00:32:02,916 Right? 684 00:32:02,916 --> 00:32:03,666 That's clean. 685 00:32:03,676 --> 00:32:07,306 If you didn't build that into your design, a ransomware recovery is 686 00:32:07,306 --> 00:32:09,076 going to take significantly longer. 687 00:32:09,466 --> 00:32:09,616 Now, I'm 688 00:32:09,811 --> 00:32:09,871 yeah. 689 00:32:09,871 --> 00:32:09,872 Yeah. 690 00:32:10,641 --> 00:32:13,036 gonna give you, I'm gonna give you a freebie. 691 00:32:14,596 --> 00:32:15,226 What 692 00:32:15,226 --> 00:32:16,576 design element I. 693 00:32:18,151 --> 00:32:19,981 That's backup related. 694 00:32:21,481 --> 00:32:27,391 Could you be using that would allow you to have infinite recovery points 695 00:32:27,871 --> 00:32:31,141 without, no, sorry. 696 00:32:31,141 --> 00:32:33,901 You fail in, let me finish my sentence. 697 00:32:34,231 --> 00:32:41,431 Allow you to have virtually unlimited recovery points and while in RT 698 00:32:41,491 --> 00:32:44,341 with RTOs of like next to zero. 699 00:32:46,756 --> 00:32:48,016 Do I have to answer this? 700 00:32:48,016 --> 00:32:49,606 I don't like this answer. 701 00:32:50,746 --> 00:32:51,946 I know what it is. 702 00:32:52,126 --> 00:32:52,756 What is it? 703 00:32:53,446 --> 00:32:54,886 It's, three letters. 704 00:32:57,046 --> 00:32:59,836 Oh, no, wait. 705 00:33:00,196 --> 00:33:02,266 Oh, no, no, not, not infinite. 706 00:33:02,266 --> 00:33:03,736 Sorry, not infinite. 707 00:33:03,736 --> 00:33:09,106 Number of recovery points ne, nearly, nearly 708 00:33:09,121 --> 00:33:12,511 So, uh, so, so, okay, so let me, let me check. 709 00:33:12,931 --> 00:33:16,591 So, in my mind as you're describing this, I'm thinking of CDP, which 710 00:33:16,591 --> 00:33:17,896 is continuous data protection. 711 00:33:18,211 --> 00:33:19,486 Yeah, that would be infinite. 712 00:33:20,236 --> 00:33:20,526 Okay. 713 00:33:21,706 --> 00:33:25,746 Uh, there is also snapshot based replication. 714 00:33:25,746 --> 00:33:25,966 Go. 715 00:33:26,096 --> 00:33:26,366 Right. 716 00:33:27,116 --> 00:33:27,206 Okay. 717 00:33:30,596 --> 00:33:32,471 So it only took me two tries, which isn't bad. 718 00:33:34,391 --> 00:33:37,721 So I gave you, I gave you a leading question, but I guess 719 00:33:37,721 --> 00:33:38,681 I didn't lead you enough. 720 00:33:39,221 --> 00:33:39,521 I did. 721 00:33:39,551 --> 00:33:41,771 I thought that, you know, given your background, this would 722 00:33:41,771 --> 00:33:43,211 just, just jump right out. 723 00:33:43,331 --> 00:33:48,371 But maybe, maybe they beat you out of it enough at your previous employer. 724 00:33:48,801 --> 00:33:49,401 So, so. 725 00:33:49,896 --> 00:33:55,326 This is where snapshots and what I would call near CDP, right near 726 00:33:55,326 --> 00:34:00,966 continuous state of protection, this is where snapshots can be so useful. 727 00:34:01,626 --> 00:34:07,086 Because how, how I, you tell me Prasanna, how else can you 728 00:34:07,091 --> 00:34:13,026 restore hundreds of versions of the same server without much pain? 729 00:34:14,136 --> 00:34:14,736 You can't. 730 00:34:15,106 --> 00:34:20,986 Unless every copy or every backup was on a separate tape, device or tape, 731 00:34:21,226 --> 00:34:25,936 and you had infinite number of tapes connected to infinite number of devices. 732 00:34:27,046 --> 00:34:27,526 Right, 733 00:34:27,931 --> 00:34:28,201 right. 734 00:34:28,261 --> 00:34:29,161 So you could do the 735 00:34:29,161 --> 00:34:31,021 restores in parallel, basically the 736 00:34:31,126 --> 00:34:34,306 and by the way, that's so you can restore one server a hundred 737 00:34:34,306 --> 00:34:38,176 different ways, and then you've got a hundred other servers. 738 00:34:38,376 --> 00:34:38,596 Yep. 739 00:34:39,451 --> 00:34:42,031 So, yeah, you would need infinite, infinite, infinite, infinite, infinite. 740 00:34:42,571 --> 00:34:42,931 infinite. 741 00:34:43,291 --> 00:34:48,361 I, I guess what I'm saying here is think about this, right? 742 00:34:48,661 --> 00:34:55,981 Think about, um, and this is where, uh, and again, we, we get into more storage 743 00:34:55,981 --> 00:34:59,731 here than backup, but you, you know that I'm a fan of this, this concept 744 00:34:59,731 --> 00:35:05,611 of snapshots and replication and that if we think about like there, there's 745 00:35:05,611 --> 00:35:07,561 a lot of technology that allows you to. 746 00:35:08,641 --> 00:35:12,721 Store your virtualization world, and I'm a big fan of virtualization, store your 747 00:35:12,721 --> 00:35:16,561 virtualization world on, on a filer, 748 00:35:17,101 --> 00:35:17,521 right? 749 00:35:17,521 --> 00:35:21,301 That allows you to take snapshots and replicate those snapshots and 750 00:35:21,301 --> 00:35:23,641 replicate them even to an immutable, uh, 751 00:35:23,671 --> 00:35:24,721 device if you want. 752 00:35:24,781 --> 00:35:25,231 Right? 753 00:35:25,831 --> 00:35:30,211 And then you have infinite number of recovery points. 754 00:35:30,751 --> 00:35:31,081 Yep. 755 00:35:31,351 --> 00:35:32,406 Um, I 756 00:35:32,411 --> 00:35:32,581 I, 757 00:35:32,731 --> 00:35:33,661 And I, I 758 00:35:33,841 --> 00:35:35,276 I wanna add one more thing on top of 759 00:35:35,401 --> 00:35:36,211 okay, sure. 760 00:35:36,686 --> 00:35:36,926 I think 761 00:35:36,926 --> 00:35:38,326 deduplication becomes important. 762 00:35:38,326 --> 00:35:40,366 Because everything you're talking about right now is for a single 763 00:35:40,366 --> 00:35:44,506 server, but now say you have a hundred servers that are all based on like 764 00:35:44,536 --> 00:35:46,486 a similar image or whatever else. 765 00:35:46,486 --> 00:35:46,936 I think 766 00:35:47,641 --> 00:35:49,681 I I think deduplication can 767 00:35:49,681 --> 00:35:52,321 add a lot of value in terms of, it can really help 768 00:35:52,321 --> 00:35:53,551 reduce the 769 00:35:54,001 --> 00:35:56,671 amount of storage that's gonna be used. 770 00:35:56,701 --> 00:35:57,391 Yes. 771 00:35:57,691 --> 00:36:00,931 Um, I don't think it's required in terms, it, it is just gonna save you money. 772 00:36:01,411 --> 00:36:01,801 Right. 773 00:36:02,731 --> 00:36:11,701 But the idea of, I guess I just, this is where, uh, you know, this, this 774 00:36:11,701 --> 00:36:18,061 is one of the, this, this is why I am such a fan of, you know, not just 775 00:36:18,091 --> 00:36:19,411 NetApp NetApp's not the only one. 776 00:36:19,411 --> 00:36:20,881 There's so many companies, and 777 00:36:20,881 --> 00:36:21,841 it's not just filers. 778 00:36:21,841 --> 00:36:25,141 It's not just na, it's, they're also SAN devices. 779 00:36:25,351 --> 00:36:26,731 There are iSCSI devices. 780 00:36:26,731 --> 00:36:30,841 There are modern scale out storage arrays that have. 781 00:36:31,261 --> 00:36:36,181 Infinite, or, you know, short, short or close to infinite number of snapshots That 782 00:36:36,181 --> 00:36:45,271 don't impact performance, And that would offer you, um, some real choices here. 783 00:36:45,511 --> 00:36:50,131 I, I think also we can throw in the concept of copy data management. 784 00:36:50,401 --> 00:36:53,221 There are, there are CDM products, like, is it still 785 00:36:53,221 --> 00:36:55,351 called acto after it got acquired? 786 00:36:55,906 --> 00:36:57,301 I think it's still called Actifio. 787 00:36:57,481 --> 00:36:57,701 Yep. 788 00:36:58,066 --> 00:37:02,986 so products like that, basically, I guess what I'm just saying is 789 00:37:04,516 --> 00:37:07,066 this is a real problem, right? 790 00:37:07,066 --> 00:37:12,946 This is a huge problem and this is a potential, really useful 791 00:37:12,946 --> 00:37:16,486 tool towards this problem. 792 00:37:16,756 --> 00:37:22,471 Maybe it won't solve all known, you know, things, but when we start talking about. 793 00:37:24,061 --> 00:37:26,491 Uh, hey, I wanna do a hundred copy. 794 00:37:26,491 --> 00:37:29,821 You know, I wanna keep retention for this long, and I want, I want 795 00:37:29,821 --> 00:37:33,751 to potentially restore my server a hundred times and I, but I don't want 796 00:37:33,751 --> 00:37:35,701 to restore my server a hundred times. 797 00:37:35,731 --> 00:37:36,001 Yep. 798 00:37:37,861 --> 00:37:42,661 Um, it just seems like snapshots and replication would 799 00:37:42,661 --> 00:37:43,921 really be your friend here. 800 00:37:44,596 --> 00:37:44,806 Yeah. 801 00:37:45,316 --> 00:37:46,036 Oh, definitely. 802 00:37:46,036 --> 00:37:46,216 Yeah. 803 00:37:46,216 --> 00:37:49,546 I don't see how else you're gonna be able to even figure 804 00:37:49,546 --> 00:37:52,576 out if a copy is clean, right? 805 00:37:52,726 --> 00:37:54,081 Without something like this, 806 00:37:54,811 --> 00:38:00,871 And, and this is, you know, the another, so I'll throw this here. 807 00:38:02,461 --> 00:38:06,511 Another possible friend is the cloud, 808 00:38:07,411 --> 00:38:07,801 right? 809 00:38:08,281 --> 00:38:13,681 If you are using a cloud-based recovery system, and if. 810 00:38:14,071 --> 00:38:19,171 That recovery system has the ability to scale out and say, I wanna 811 00:38:19,171 --> 00:38:20,911 recover, bang, bang, bang, bang, bang. 812 00:38:21,211 --> 00:38:21,601 Right? 813 00:38:21,701 --> 00:38:21,921 Yep. 814 00:38:23,131 --> 00:38:26,641 It's just that all the ones that I've seen, the, at least the ones that 815 00:38:26,641 --> 00:38:31,591 I've seen, you know, essentially with my own eyes, when we start talking 816 00:38:31,591 --> 00:38:34,231 about recovering many, many copies. 817 00:38:35,446 --> 00:38:36,826 They can scale it out. 818 00:38:36,826 --> 00:38:37,126 Right? 819 00:38:37,126 --> 00:38:37,246 so 820 00:38:37,246 --> 00:38:40,036 they, so they, they do, you know, you remember earlier when you said 821 00:38:40,036 --> 00:38:41,836 if you had a infinite tape drives and 822 00:38:42,256 --> 00:38:43,366 all that, they have 823 00:38:43,441 --> 00:38:44,611 the cloud lets you do that. 824 00:38:44,611 --> 00:38:44,971 Yeah. 825 00:38:44,986 --> 00:38:46,576 the cloud lets you do that. 826 00:38:46,606 --> 00:38:47,056 Right. 827 00:38:48,076 --> 00:38:53,176 But, uh, the restore of the actual server will still take. 828 00:38:53,176 --> 00:38:55,006 A finite amount of time. 829 00:38:55,216 --> 00:38:55,636 Right. 830 00:38:56,266 --> 00:39:00,226 And again, build that into the design, figure that out, go to the vendor, 831 00:39:00,226 --> 00:39:01,666 say, Hey, here's what we wanna do. 832 00:39:01,666 --> 00:39:04,246 We wanna be able to do, restore the server a hundred times 833 00:39:04,246 --> 00:39:05,326 and pick which one we want. 834 00:39:05,986 --> 00:39:06,856 Can I do that? 835 00:39:07,876 --> 00:39:09,976 Well, yeah, it's gonna take you three years. 836 00:39:10,006 --> 00:39:10,426 You know, 837 00:39:10,846 --> 00:39:13,006 figure, have that discussion now. 838 00:39:14,266 --> 00:39:16,516 You know, build that into the design. 839 00:39:17,356 --> 00:39:19,426 Um, I, I guess I'm. 840 00:39:20,041 --> 00:39:23,701 I'm just really, and, and maybe this means you change vendors, 841 00:39:24,421 --> 00:39:24,811 right? 842 00:39:24,841 --> 00:39:27,151 Maybe this means you change storage systems. 843 00:39:27,751 --> 00:39:28,621 Um, 844 00:39:28,921 --> 00:39:31,291 to your vendor and see how do I solve this? 845 00:39:31,441 --> 00:39:31,951 Here's my 846 00:39:32,101 --> 00:39:33,601 Well, you, you definitely should do. 847 00:39:33,601 --> 00:39:34,201 that first. 848 00:39:34,201 --> 00:39:34,621 You know, I'm a 849 00:39:34,621 --> 00:39:37,801 fan of, I, I'm not a fan of like, uh, 850 00:39:38,341 --> 00:39:40,621 uh, steeplechase, I call it, right? 851 00:39:40,621 --> 00:39:44,521 Not a fan of just going, you know, place to place just to, you know, I 852 00:39:44,521 --> 00:39:45,931 always think you should, you should. 853 00:39:46,801 --> 00:39:49,861 Let your current, you know, give your current vendor the problem. 854 00:39:50,711 --> 00:39:52,391 Don't go to them with the design. 855 00:39:52,751 --> 00:39:53,141 Right. 856 00:39:53,771 --> 00:39:54,191 Go to them 857 00:39:54,191 --> 00:39:55,181 with your requirements. 858 00:39:55,211 --> 00:39:55,601 Right? 859 00:39:57,056 --> 00:39:58,706 this is what I'm looking to do. 860 00:39:59,066 --> 00:40:00,176 What's the best way to do this? 861 00:40:00,956 --> 00:40:02,546 Because maybe they have a mechanism that. 862 00:40:04,136 --> 00:40:06,896 Yeah, I listened to this podcast and Curtis said I need to be able to 863 00:40:06,896 --> 00:40:08,336 restore my server a hundred times. 864 00:40:08,336 --> 00:40:09,476 Like how, how do I do that? 865 00:40:09,656 --> 00:40:11,306 Is there a way to do that with your product? 866 00:40:11,336 --> 00:40:11,666 Right. 867 00:40:12,296 --> 00:40:13,676 Um, or, well, 868 00:40:13,796 --> 00:40:15,116 actually, technically what? 869 00:40:15,296 --> 00:40:19,076 What you, you would say, I need to be able to, 870 00:40:19,661 --> 00:40:20,741 Identify a clean 871 00:40:20,741 --> 00:40:21,191 copy of 872 00:40:21,821 --> 00:40:22,601 a clean image. 873 00:40:22,661 --> 00:40:24,521 And so maybe there's a way to do that. 874 00:40:24,641 --> 00:40:28,631 Um, yeah, the, the example I always give for dictating the 875 00:40:28,631 --> 00:40:29,921 requirements and not the design. 876 00:40:29,921 --> 00:40:33,761 I live in San Diego and we have Coronado, which is, it's not an island, 877 00:40:33,761 --> 00:40:35,231 but people call it Coronado Island. 878 00:40:35,711 --> 00:40:39,461 Um, it's Coronado Peninsula, but that doesn't sound as cool. 879 00:40:39,881 --> 00:40:43,811 And I always give the example of, of say, listen, I need a hundred 880 00:40:43,811 --> 00:40:46,661 thousand people to be able to go to and from that island every day. 881 00:40:47,066 --> 00:40:49,526 That's a, that is a requirement. 882 00:40:50,096 --> 00:40:51,686 And maybe it's a tunnel. 883 00:40:51,776 --> 00:40:52,916 Maybe it's a ferry. 884 00:40:53,066 --> 00:40:53,516 Maybe 885 00:40:53,516 --> 00:40:54,236 it's a bridge. 886 00:40:54,776 --> 00:40:55,226 Right. 887 00:40:55,466 --> 00:40:57,986 Or in the case of San Diego, maybe it's all three. 888 00:40:59,186 --> 00:40:59,846 Not a tunnel. 889 00:40:59,996 --> 00:41:01,016 But we do have a bridge. 890 00:41:01,046 --> 00:41:03,026 We have a ferry, and we have the long way. 891 00:41:03,236 --> 00:41:03,386 The 892 00:41:03,386 --> 00:41:04,046 long way. 893 00:41:04,556 --> 00:41:05,966 Uh, it's funny, in San 894 00:41:05,966 --> 00:41:08,516 Diego you can literally see Coronado, it's like right there. 895 00:41:08,516 --> 00:41:11,816 It's like a half a mile on the other side of the water. 896 00:41:12,326 --> 00:41:16,166 But the long way without the bridge, it's like four, like a 40 minute 897 00:41:16,166 --> 00:41:18,506 drive because you gotta go to Mexico. 898 00:41:18,686 --> 00:41:18,926 No 899 00:41:18,926 --> 00:41:19,316 kidding. 900 00:41:19,526 --> 00:41:21,596 You gotta go to Mexico, turn around and come back. 901 00:41:22,946 --> 00:41:23,846 Um, 902 00:41:24,236 --> 00:41:24,596 yeah. 903 00:41:24,656 --> 00:41:27,836 Anyway, you could just swim across. 904 00:41:29,456 --> 00:41:31,196 Anyway, I think that, I think that was good. 905 00:41:31,256 --> 00:41:31,496 It was a 906 00:41:31,496 --> 00:41:32,666 good conversation. 907 00:41:32,776 --> 00:41:35,056 We did not talk about two things though 908 00:41:35,476 --> 00:41:36,856 that you had brought up earlier. 909 00:41:37,246 --> 00:41:43,486 The one is you had talked about restoring the server, but not the databases or 910 00:41:43,516 --> 00:41:43,876 Yep. 911 00:41:44,026 --> 00:41:45,136 Yep, yep. 912 00:41:45,916 --> 00:41:46,426 Yeah, yeah, 913 00:41:46,426 --> 00:41:48,196 yeah, yeah, yeah. 914 00:41:48,676 --> 00:41:52,546 So, all right, so a couple things that we didn't talk about, right? 915 00:41:52,906 --> 00:41:56,506 Uh, this idea of restoring the server and perhaps restoring the data later. 916 00:41:57,286 --> 00:42:01,666 I, I, I'm thinking mainly about the idea of like using an image 917 00:42:01,696 --> 00:42:03,376 perhaps to restore the server, 918 00:42:03,586 --> 00:42:06,196 and then we restore the, and that that image would probably 919 00:42:06,196 --> 00:42:09,046 have a clean copy of Oracle or 920 00:42:09,256 --> 00:42:14,296 whatever it is that you wanna restore, and then potentially restoring 921 00:42:14,296 --> 00:42:16,036 the data as a secondary thing. 922 00:42:16,571 --> 00:42:18,316 I, I think this is another. 923 00:42:18,931 --> 00:42:20,941 Potential way to do this. 924 00:42:20,941 --> 00:42:23,851 And by the way, virtualization makes all this so much easier. 925 00:42:23,851 --> 00:42:24,241 Right? 926 00:42:24,871 --> 00:42:29,521 Um, but which is, you know, on the list of why I, I'm such a fan of virtualization. 927 00:42:29,521 --> 00:42:34,801 But that is a potential, again, just look at these designs and 928 00:42:34,801 --> 00:42:36,211 then, and then work with it. 929 00:42:36,751 --> 00:42:42,631 When we look at that method of doing it, the, I think this is 930 00:42:42,631 --> 00:42:48,241 a much more valid method for restoring, say, databases today. 931 00:42:48,571 --> 00:42:53,671 Because I say this, you know, I'm recording this on August 21st, 2024. 932 00:42:53,881 --> 00:43:01,411 Today, they don't tend to attack databases directly, meaning they 933 00:43:01,411 --> 00:43:06,691 don't go into the database and, and mess up the individual contents. 934 00:43:06,691 --> 00:43:09,091 If they encrypt the database, the encrypt the database file. 935 00:43:09,991 --> 00:43:10,381 Right? 936 00:43:10,501 --> 00:43:11,521 So you just wanna restore. 937 00:43:11,521 --> 00:43:12,781 You wanna restore the database. 938 00:43:12,781 --> 00:43:13,051 Yes. 939 00:43:13,051 --> 00:43:13,771 You restore the file. 940 00:43:13,771 --> 00:43:15,631 You restore the file from before it was encrypted. 941 00:43:15,751 --> 00:43:16,411 You're good to go. 942 00:43:17,251 --> 00:43:22,981 Generally speaking, so I think that's a really valid way to restore a database 943 00:43:22,981 --> 00:43:25,051 server and an application server that 944 00:43:25,051 --> 00:43:27,211 has something like a database on it. 945 00:43:28,411 --> 00:43:31,471 the the real concern I do have 946 00:43:31,711 --> 00:43:37,051 is when we start talking about unstructured data and file systems, 947 00:43:37,756 --> 00:43:39,511 because what did we start this? 948 00:43:39,841 --> 00:43:42,181 What did we start this podcast talking about? 949 00:43:43,051 --> 00:43:45,121 Do you remember the phrase that we defined in the beginning? 950 00:43:45,571 --> 00:43:46,411 The dwell time. 951 00:43:46,726 --> 00:43:51,346 So, so what, why is having a long dwell time? 952 00:43:51,436 --> 00:43:53,836 It's like, hey, I get, you said 90 days, right? 953 00:43:54,196 --> 00:43:55,276 I got, I got six months. 954 00:43:55,276 --> 00:43:55,936 I got, I got a 955 00:43:55,936 --> 00:43:56,446 year. 956 00:43:56,446 --> 00:43:57,736 Curtis said to do two years. 957 00:43:57,736 --> 00:43:58,576 I'm doing two years. 958 00:43:58,756 --> 00:44:00,136 I got two years of backups. 959 00:44:00,376 --> 00:44:02,236 So what if the dwell time is 90 days? 960 00:44:02,296 --> 00:44:02,566 What's the 961 00:44:02,836 --> 00:44:06,136 Well, well, because you, 'cause it is gonna go through and 962 00:44:06,136 --> 00:44:07,186 like we talked about, right? 963 00:44:07,186 --> 00:44:09,316 It might decide, I'm gonna start with the old data and 964 00:44:09,316 --> 00:44:11,236 slowly start encrypting things. 965 00:44:11,236 --> 00:44:14,116 And maybe you notice, maybe you don't notice, but. 966 00:44:15,091 --> 00:44:18,541 Now you have to go figure out like a needle in a haystack. 967 00:44:18,541 --> 00:44:22,351 Except the haystack is a small number of files in say, a 968 00:44:22,351 --> 00:44:24,271 billion or 5 billion files. 969 00:44:24,321 --> 00:44:28,071 So the good news is there's gonna be one of two scenarios, and 970 00:44:28,071 --> 00:44:31,641 most likely you're going to be the first of the two scenarios. 971 00:44:32,421 --> 00:44:35,451 The good news is, I think most of the time. 972 00:44:35,856 --> 00:44:39,126 You're gonna look at the server, you're gonna look at the even unstructured 973 00:44:39,126 --> 00:44:44,646 data, and you will be able to easily identify which files were encrypted, 974 00:44:45,246 --> 00:44:47,826 and you're gonna find that they were all encrypted at the same time. 975 00:44:48,216 --> 00:44:49,986 They were all encrypted, all on the same day. 976 00:44:50,466 --> 00:44:55,746 I think that this idea, again, August 21st, 2024, I think this 977 00:44:55,746 --> 00:45:01,296 idea of slowly encrypting the files over time, one, one of two things I 978 00:45:01,296 --> 00:45:03,666 think it is at, at a minimum it is. 979 00:45:04,356 --> 00:45:08,316 More rare than the other method because again, the 980 00:45:08,316 --> 00:45:10,716 moment they start encrypting files, they really set off 981 00:45:11,196 --> 00:45:12,186 alarms, right? 982 00:45:13,206 --> 00:45:17,136 So I think it's pretty rare and I even think it's possibly a boogeyman. 983 00:45:17,166 --> 00:45:17,406 I, 984 00:45:17,406 --> 00:45:23,346 I, I don't know for sure, but if you have this problem 985 00:45:23,346 --> 00:45:27,936 though, there's no good answer. 986 00:45:29,166 --> 00:45:33,126 Uh, you know that, well, the, the, the only one that I am aware of, 987 00:45:33,156 --> 00:45:36,666 right there was, you know, from, from our previous employer, they 988 00:45:36,666 --> 00:45:40,836 had a solution to this problem where they, they, they, they had this, they 989 00:45:40,836 --> 00:45:42,996 called like image curation, right? 990 00:45:43,296 --> 00:45:45,186 Where you could give them a range of time. 991 00:45:45,186 --> 00:45:49,926 They go and they would go in and automatically pick the last good 992 00:45:49,926 --> 00:45:55,326 version of every file prior to it being encrypted, doing that manually. 993 00:45:55,371 --> 00:45:55,431 Yeah. 994 00:45:56,031 --> 00:46:02,091 If you have this slow encryption, doing that manually is, is, 995 00:46:02,601 --> 00:46:02,961 you know, 996 00:46:03,396 --> 00:46:06,546 And then, and then you just need to think if it's all the old data, it's 997 00:46:06,546 --> 00:46:08,796 encrypting, do I really care about it? 998 00:46:11,751 --> 00:46:12,261 yeah. 999 00:46:12,351 --> 00:46:14,571 It's the, um, take it. 1000 00:46:14,571 --> 00:46:14,991 Um. 1001 00:46:15,771 --> 00:46:20,241 Again, I'm going back to a hundred years ago when we had this, uh, this 1002 00:46:20,241 --> 00:46:23,691 old server that we were decommissioning and it had been around so long 1003 00:46:23,691 --> 00:46:25,101 that nobody knew what was on it. 1004 00:46:25,881 --> 00:46:30,861 And so basically we got down to like the final one or two servers that were part 1005 00:46:30,861 --> 00:46:36,201 of the, it was a, it was at and t's first attempt at a multi-processing computer. 1006 00:46:36,291 --> 00:46:36,681 Right? 1007 00:46:37,101 --> 00:46:39,381 And so it had multiple computers inside the computer. 1008 00:46:39,381 --> 00:46:44,991 And so we got down to like the last one or two, and basically the, the idea was. 1009 00:46:45,336 --> 00:46:48,576 Uh, we just turn it off and then see who yells 1010 00:46:50,111 --> 00:46:51,306 Sometimes that works the 1011 00:46:51,396 --> 00:46:54,036 because we couldn't figure out, you know, we couldn't figure 1012 00:46:54,036 --> 00:46:56,136 out who was on it and what it was doing. 1013 00:46:56,136 --> 00:46:57,786 And so that's the same kind of thing. 1014 00:46:57,786 --> 00:47:02,136 Like, you know, if a, if a file gets encrypted and nobody reads 1015 00:47:02,136 --> 00:47:03,606 it, that it really get encrypted. 1016 00:47:05,556 --> 00:47:10,326 It's like, you know, like if a tree falls in a forest, if a 1017 00:47:10,326 --> 00:47:14,706 file gets attacked by ransomware and nobody wants to file, who gives a crap? 1018 00:47:15,981 --> 00:47:19,251 Uh, that's our, that's our, uh, that's 1019 00:47:19,251 --> 00:47:20,811 our final piece of advice. 1020 00:47:23,091 --> 00:47:23,451 All right. 1021 00:47:23,451 --> 00:47:24,681 Well, this, this has been fun. 1022 00:47:24,681 --> 00:47:25,131 I think it's 1023 00:47:25,131 --> 00:47:25,461 good. 1024 00:47:25,461 --> 00:47:29,751 You, again, it's, this is a little bit more far reaching in terms of 1025 00:47:29,751 --> 00:47:33,621 some of the design elements and design ideas that you would put in there. 1026 00:47:34,101 --> 00:47:37,131 Um, but I think it's one that, that people should really be thinking about. 1027 00:47:37,476 --> 00:47:37,686 Yeah. 1028 00:47:37,926 --> 00:47:40,806 And this isn't intended to be a conclusive list, 1029 00:47:41,196 --> 00:47:41,496 right? 1030 00:47:41,496 --> 00:47:45,516 But this is just initial thoughts to get you thinking and go have your 1031 00:47:45,516 --> 00:47:50,256 discussions with other experts, with your vendors, right, to see what 1032 00:47:50,256 --> 00:47:51,966 else you should be thinking about. 1033 00:47:52,386 --> 00:47:53,106 Exactly. 1034 00:47:53,106 --> 00:47:53,856 Exactly. 1035 00:47:54,336 --> 00:47:56,166 Well, thanks Pana. 1036 00:47:56,166 --> 00:47:56,591 This was fun. 1037 00:47:57,546 --> 00:48:00,666 I know this was fun and Curtis, I'm glad you didn't die. 1038 00:48:01,596 --> 00:48:02,556 'cause then I would be sad. 1039 00:48:03,366 --> 00:48:07,881 One, one final thing on that, uh, 'cause I don't think I mentioned it earlier. 1040 00:48:07,881 --> 00:48:10,221 You know, we, we, we have some people that are renting are 1041 00:48:10,221 --> 00:48:11,391 renting a room from us here. 1042 00:48:11,871 --> 00:48:15,441 And he, he happened to be the only one that was home when this happened. 1043 00:48:15,831 --> 00:48:20,421 And he heard it happen and he was very glad to hear me yell at his name. 1044 00:48:20,451 --> 00:48:22,431 'cause he's like, he was scared to go out. 1045 00:48:22,431 --> 00:48:25,551 He, he was like, he heard it happen and it sounded awful. 1046 00:48:25,971 --> 00:48:27,866 And he is like, I, I hope he is not dead. 1047 00:48:28,536 --> 00:48:31,266 And then he heard me yell at his name and he is like, oh, thank 1048 00:48:31,266 --> 00:48:31,476 God. 1049 00:48:31,671 --> 00:48:32,061 thank God. 1050 00:48:32,226 --> 00:48:32,436 dead. 1051 00:48:34,386 --> 00:48:36,666 Well, I'm glad I'm not dead too. 1052 00:48:36,986 --> 00:48:39,246 Prasanna, so that we can 1053 00:48:39,261 --> 00:48:41,121 I'm sure our listeners are as 1054 00:48:41,256 --> 00:48:43,656 Yeah, I, yeah, you don't care. 1055 00:48:45,096 --> 00:48:45,486 All right. 1056 00:48:45,576 --> 00:48:50,286 If there's anybody out there that's glad I'm not dead and they're still listening, 1057 00:48:50,676 --> 00:48:53,676 send me a note on backup wrap up.com. 1058 00:48:53,826 --> 00:48:54,786 Send a message. 1059 00:48:55,281 --> 00:48:57,261 This, uh, I'm glad you're not dead. 1060 00:48:57,621 --> 00:49:01,941 Or put it as a comment on the, uh, on the YouTube video or a comment on 1061 00:49:01,941 --> 00:49:03,586 the, you know, on the, uh, on the, 1062 00:49:03,651 --> 00:49:04,491 on the backup wrap up. 1063 00:49:04,671 --> 00:49:07,671 Anyway, well, uh, thanks to our listeners. 1064 00:49:07,761 --> 00:49:10,131 Uh, you know, we kid, but we love you. 1065 00:49:10,341 --> 00:49:11,451 You're the only reason we do 1066 00:49:11,451 --> 00:49:11,721 this. 1067 00:49:11,721 --> 00:49:14,631 Otherwise, just a couple of guys just talking. 1068 00:49:15,531 --> 00:49:17,991 Uh, and we'd probably just talk about barbecue then. 1069 00:49:18,201 --> 00:49:20,001 So that is a wrap.