1 00:00:00,000 --> 00:00:04,350 You found the backup wrap up your go-to podcast for all things 2 00:00:04,350 --> 00:00:07,109 backup recovery and cyber recovery. 3 00:00:07,650 --> 00:00:13,409 In this episode, we jump into disaster recovery testing, and trust me, you don't 4 00:00:13,409 --> 00:00:15,209 wanna learn these lessons the hard way. 5 00:00:15,690 --> 00:00:17,790 I've got some wild stories about DR. 6 00:00:17,790 --> 00:00:18,839 Tests gone wrong. 7 00:00:19,244 --> 00:00:23,564 Including one from my early days at a bank that'll make you cringe. 8 00:00:24,165 --> 00:00:28,574 My co-host persona, and I break down exactly how to approach DR. 9 00:00:28,574 --> 00:00:33,285 Testing the right way, starting with the basics and working your way up. 10 00:00:33,794 --> 00:00:39,285 We'll tell you why non-destructive testing is absolutely critical. 11 00:00:39,525 --> 00:00:45,135 Seriously, you don't wanna blow up your production environment just to test Dr. 12 00:00:45,794 --> 00:00:51,915 And how to set realistic success criteria that won't make you cry Another episode 13 00:00:51,915 --> 00:00:54,014 from The Lessons From the Trenches. 14 00:00:54,225 --> 00:00:55,245 I hope you like it. 15 00:00:55,885 --> 00:00:58,945 By the way, if you don't know who I am, you're a first time listener. 16 00:00:58,945 --> 00:01:01,645 I'm w Curtis Preston, AKA, Mr. 17 00:01:01,645 --> 00:01:06,355 Backup, and I've been passionate about backup and recovery for over 30 years. 18 00:01:06,745 --> 00:01:07,345 Ever since. 19 00:01:07,345 --> 00:01:10,945 I had to tell my boss that we had no backups of the production 20 00:01:10,945 --> 00:01:12,595 database that we had just lost. 21 00:01:13,015 --> 00:01:14,605 I don't want that to happen to you. 22 00:01:14,755 --> 00:01:16,615 That's why I do this podcast. 23 00:01:16,945 --> 00:01:21,895 On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes. 24 00:01:22,165 --> 00:01:24,385 This is the backup wrap up. 25 00:01:39,442 --> 00:01:40,492 Welcome to the show. 26 00:01:40,942 --> 00:01:43,882 Hi, I am your host, w Curtis Preston, AKA, Mr. 27 00:01:43,882 --> 00:01:47,002 Backup, and if you could take just a quick moment to subscribe 28 00:01:47,002 --> 00:01:51,622 or follow so that you'll get our great content, that would be great. 29 00:01:52,042 --> 00:01:57,352 I am sitting here with none other than the guy who's very concerned. 30 00:01:57,907 --> 00:02:02,527 About my obsession with serial killers lately, or at least one serial killer. 31 00:02:02,797 --> 00:02:03,337 How's it going? 32 00:02:03,907 --> 00:02:04,432 How's it going? 33 00:02:04,537 --> 00:02:05,047 Persona? 34 00:02:05,342 --> 00:02:06,662 I am good, Curtis. 35 00:02:06,662 --> 00:02:07,142 Yeah, 36 00:02:07,267 --> 00:02:07,717 about me? 37 00:02:10,082 --> 00:02:12,637 so I think after this you should watch Hannibal, 38 00:02:15,512 --> 00:02:16,952 the TV show, not the movie. 39 00:02:16,982 --> 00:02:17,972 They both are good, but 40 00:02:18,397 --> 00:02:18,847 Right? 41 00:02:18,847 --> 00:02:19,357 Right? 42 00:02:19,652 --> 00:02:22,142 And then we could discuss about whether people taste differently 43 00:02:22,142 --> 00:02:22,982 depending on what they eat. 44 00:02:24,577 --> 00:02:26,407 that's always something you talk about. 45 00:02:28,297 --> 00:02:28,897 Uh, 46 00:02:28,897 --> 00:02:31,507 your latest obsession is Dexter. 47 00:02:31,927 --> 00:02:32,557 Yeah. 48 00:02:32,647 --> 00:02:35,047 Which, which, for the record, I'm rewatching, right? 49 00:02:35,082 --> 00:02:40,147 I, I enjoyed Dexter when it was on and when I had to wait a week, right? 50 00:02:40,297 --> 00:02:41,462 going back to those old days 51 00:02:42,217 --> 00:02:43,057 oh Lord, 52 00:02:43,117 --> 00:02:44,737 I, you know, I'm, I'm kind. 53 00:02:44,737 --> 00:02:46,147 I think you're, aren't you? 54 00:02:46,267 --> 00:02:50,227 Like, don't you watch you, you do not watch shows when they're on. 55 00:02:50,227 --> 00:02:52,957 You wait until they're done and then you binge them, 56 00:02:53,377 --> 00:02:53,767 right? 57 00:02:54,007 --> 00:02:54,757 Yeah, for 58 00:02:54,757 --> 00:02:58,717 the most part, although these days I don't actually have like broadcast tv, 59 00:02:59,842 --> 00:03:00,292 Right. 60 00:03:00,457 --> 00:03:04,057 it's whatever's available on Netflix or Amazon or Take your pick. 61 00:03:04,462 --> 00:03:07,552 We, we, we actually have YouTube tv, so we have broadcasts and there's 62 00:03:07,552 --> 00:03:09,022 some shows that we watch on there. 63 00:03:09,172 --> 00:03:13,942 Um, but there shows where, like, you know, this episode, that episode, 64 00:03:13,942 --> 00:03:15,052 they're, they're all the same, you know? 65 00:03:15,052 --> 00:03:15,142 And 66 00:03:15,337 --> 00:03:15,547 Yeah. 67 00:03:15,562 --> 00:03:17,242 you don't, it's not like, uh, it's not like 68 00:03:17,242 --> 00:03:18,802 Dexter where there's an ongoing 69 00:03:18,817 --> 00:03:20,317 so here's the funny thing, right? 70 00:03:20,317 --> 00:03:22,687 So it was, it started off first on. 71 00:03:23,587 --> 00:03:23,947 tv. 72 00:03:23,947 --> 00:03:24,127 Right. 73 00:03:24,127 --> 00:03:25,267 And like you said, you had to wait a 74 00:03:25,432 --> 00:03:26,002 Showtime. 75 00:03:26,092 --> 00:03:26,392 Yeah. 76 00:03:27,037 --> 00:03:30,487 And then what I used to do, right, so I didn't have Showtime, 77 00:03:31,162 --> 00:03:31,312 Mm-Hmm. 78 00:03:31,357 --> 00:03:34,357 so I had to wait for it to come out on Netflix DVDs, 79 00:03:35,302 --> 00:03:35,542 Oh, 80 00:03:35,542 --> 00:03:35,632 right. 81 00:03:35,707 --> 00:03:36,937 I would request the DVDs. 82 00:03:36,967 --> 00:03:40,477 And of course I was cheap, frugal, however you want to say it, right? 83 00:03:40,477 --> 00:03:44,917 And so I had like the two DVD plan, so I'd request like two DVDs, right? 84 00:03:44,917 --> 00:03:46,507 So you'd get like six episodes. 85 00:03:46,882 --> 00:03:47,392 Yeah. 86 00:03:47,572 --> 00:03:51,082 Five or six episodes, you'd binge watch those and then you'd send them back. 87 00:03:51,082 --> 00:03:53,692 And then you'd have to wait a week to then get the next set of DVDs 88 00:03:53,827 --> 00:03:54,427 so funny. 89 00:03:55,252 --> 00:03:55,402 in 90 00:03:55,402 --> 00:03:55,942 order to watch it. 91 00:03:55,942 --> 00:03:58,882 And that's I think, how I ended up watching Dexter. 92 00:03:58,882 --> 00:04:01,222 And I think Breaking Bad might've been the same way too. 93 00:04:01,507 --> 00:04:03,397 Yeah, you're just, you're cheaper than me. 94 00:04:03,497 --> 00:04:04,967 Dexter is currently on Netflix, 95 00:04:04,997 --> 00:04:07,367 so, um, yeah. 96 00:04:07,727 --> 00:04:13,172 Anyway, so, and you could do, you can watch all these things while you do. 97 00:04:14,462 --> 00:04:15,812 Disaster recovery testing. 98 00:04:16,412 --> 00:04:19,622 'cause there's a lot of time, there's a lot of time when you do DR testing, 99 00:04:19,622 --> 00:04:21,062 there's a lot of downtime, right? 100 00:04:21,062 --> 00:04:22,652 You sit there and you stare at the screen. 101 00:04:23,132 --> 00:04:26,102 Um, and, um, I'm gonna, I'm 102 00:04:26,102 --> 00:04:27,872 gonna, I'm gonna start out a story. 103 00:04:28,052 --> 00:04:28,682 What's that? 104 00:04:28,862 --> 00:04:30,842 but can you really? 105 00:04:31,712 --> 00:04:32,042 What, 106 00:04:32,042 --> 00:04:35,432 I understand like if you're watching Dexter or pick one of these 107 00:04:35,432 --> 00:04:38,552 very, the shows that pull you in, 108 00:04:39,152 --> 00:04:39,272 uh huh. 109 00:04:39,932 --> 00:04:43,652 Would you actually be focused or would your DR testing basically balloon like 110 00:04:43,652 --> 00:04:45,872 10 times the normal amount of time? 111 00:04:45,872 --> 00:04:46,292 Because you're 112 00:04:46,472 --> 00:04:47,042 like, oh yeah, 113 00:04:47,042 --> 00:04:48,062 I forgot to get back to that. 114 00:04:48,587 --> 00:04:51,377 well, it de, you know what, it's gonna depend on the type of DR test 115 00:04:51,377 --> 00:04:55,397 you do because some DR tests, there's a lot of waiting, there's a lot of, 116 00:04:55,397 --> 00:04:59,207 I'm gonna start the restore part, and then I sit there for many, many hours. 117 00:04:59,207 --> 00:05:00,557 And if you got one of those, then. 118 00:05:00,887 --> 00:05:02,807 Doesn't really matter whether you're focused or not, as long 119 00:05:02,807 --> 00:05:05,209 as somebody's keeping an eye on the, uh, percentage done. 120 00:05:05,479 --> 00:05:08,869 And I'm gonna start this with a story from back in the day. 121 00:05:09,289 --> 00:05:15,919 The first backup, uh, the first restore test I ever did. 122 00:05:15,919 --> 00:05:20,419 The first, like, well, at least the first one I can really remember. 123 00:05:21,199 --> 00:05:25,309 And this was when we had, um, you know, I was at the bank. 124 00:05:25,729 --> 00:05:30,199 And which MBNA, which at the time was the second largest credit card corporation, 125 00:05:30,199 --> 00:05:31,369 and I was in charge of backups. 126 00:05:31,369 --> 00:05:37,219 And we had, I had talked the boss into moving to what would become 127 00:05:37,219 --> 00:05:41,059 the first of many commercial backup products that I had, uh, used. 128 00:05:41,359 --> 00:05:44,629 And that product was a product called SMarch, which as I've mentioned 129 00:05:44,629 --> 00:05:48,199 before, should have been called SM Back because it was not an archive 130 00:05:48,199 --> 00:05:49,609 product, it was a backup product. 131 00:05:49,609 --> 00:05:52,519 Well, they were out, they were out of, uh, Minnesota area and, 132 00:05:52,519 --> 00:05:57,199 and we had converted to them, but I like a good. 133 00:05:57,679 --> 00:05:59,209 Good little backup guy. 134 00:05:59,509 --> 00:06:03,379 I had done like a parallel implementation and so I was still 135 00:06:03,379 --> 00:06:08,269 running my old dump tapes and I was running this new, uh, fancy tool. 136 00:06:08,269 --> 00:06:11,689 And one of the things that this tool had was built-in, uh, compression. 137 00:06:12,199 --> 00:06:15,109 Um, I wasn't using the compression on the tape drives. 138 00:06:15,109 --> 00:06:16,309 I was using compression in the 139 00:06:16,369 --> 00:06:17,089 software, 140 00:06:17,419 --> 00:06:19,879 uh, to compress, uh, to go to the tape drives. 141 00:06:20,989 --> 00:06:24,979 So we had our first major. 142 00:06:25,564 --> 00:06:30,364 Failure, uh, file server, HP FS oh one. 143 00:06:30,454 --> 00:06:32,134 I still remember the name of the server 144 00:06:32,749 --> 00:06:34,159 just like get burned into your 145 00:06:34,384 --> 00:06:35,254 Yeah. 146 00:06:35,314 --> 00:06:35,764 Yeah. 147 00:06:36,214 --> 00:06:40,084 And, and this is related to another story that we, to a friend of mine that, 148 00:06:40,084 --> 00:06:43,774 that I've referred to before where she was a consultant and she accidentally 149 00:06:43,774 --> 00:06:48,574 basically re, this was a self-inflicted disaster where a consultant was trying 150 00:06:48,574 --> 00:06:51,814 to clean up home directories and she just did a really, really good job 151 00:06:52,084 --> 00:06:53,464 of cleaning up all the home directors. 152 00:06:53,464 --> 00:06:57,394 So I, I put my, uh, SMR tapes. 153 00:06:57,959 --> 00:07:02,039 Uh, you know, in my front pocket and I put my backup tapes in my back, my 154 00:07:02,039 --> 00:07:06,929 back pocket, and I went down there like Mighty Mouse here comes said the day. 155 00:07:08,189 --> 00:07:10,709 And um, some of you will get 156 00:07:10,774 --> 00:07:12,124 Not available on iTunes, I 157 00:07:12,229 --> 00:07:13,699 Not available at iTunes. 158 00:07:14,179 --> 00:07:18,949 And we went in there and I put, these were DDS, uh, tapes, right? 159 00:07:18,979 --> 00:07:22,399 And these, 'cause this, we were all hp, HP loved DDS. 160 00:07:22,549 --> 00:07:25,819 And so I popped into DDS tape, I pulled up my SMR software 161 00:07:26,239 --> 00:07:27,709 and I started the restore test. 162 00:07:28,069 --> 00:07:29,689 And, well, it wasn't a restore test. 163 00:07:29,689 --> 00:07:34,429 It was, it was, I was testing, I was testing in anger, uh, you know, we 164 00:07:34,429 --> 00:07:37,339 were testing like this was for real. 165 00:07:37,879 --> 00:07:40,249 And, and I had not done any actual testing. 166 00:07:40,249 --> 00:07:40,309 I. 167 00:07:40,744 --> 00:07:47,404 And so I kicked off the Restore and um, it. 168 00:07:48,334 --> 00:07:51,514 I'm, I'm watching and I, I, I, you know, like a good little Unix boy. 169 00:07:51,754 --> 00:07:53,524 I had a, for a while loop 170 00:07:53,554 --> 00:07:58,054 running and it was doing a, a DF on the, on the, um, on the, you know, to 171 00:07:58,054 --> 00:07:59,344 display the size of the file system. 172 00:07:59,434 --> 00:08:02,824 And I'm watching, and like a long time was going by and there 173 00:08:02,824 --> 00:08:05,284 was no change in the size of 174 00:08:05,284 --> 00:08:06,094 the file system. 175 00:08:07,354 --> 00:08:09,094 And I was like, this is weird. 176 00:08:09,154 --> 00:08:10,894 And then I went over and I looked, I just. 177 00:08:11,224 --> 00:08:12,934 Just outta curiosity, I looked at the 178 00:08:12,934 --> 00:08:14,254 tape drive, right. 179 00:08:14,524 --> 00:08:17,584 You know, it's kind like, it's kinda like your car dies, you know, you 180 00:08:17,884 --> 00:08:20,554 open up the hood like, I have any idea what's going on in inside there. 181 00:08:20,554 --> 00:08:21,604 You know, look in there 182 00:08:21,664 --> 00:08:24,994 and I see the, I see the light on the tape drive and, and it 183 00:08:24,994 --> 00:08:27,139 goes, B blink, B blink, blink, 184 00:08:28,054 --> 00:08:31,444 one 1002, 1003, 1004, 185 00:08:31,774 --> 00:08:32,165 blink, blink. 186 00:08:33,484 --> 00:08:34,204 Right? 187 00:08:34,264 --> 00:08:37,264 And there were these giant pauses in between. 188 00:08:37,819 --> 00:08:38,689 The blinks. 189 00:08:39,589 --> 00:08:41,389 And so I'm like, that's strange. 190 00:08:41,599 --> 00:08:45,349 It always blinks when it's, you know, when it's reading or writing data, right? 191 00:08:47,569 --> 00:08:50,569 So I called up, I called up the guys and I said, Hey, 192 00:08:50,779 --> 00:08:50,869 The 193 00:08:50,869 --> 00:08:51,439 vendor. 194 00:08:51,799 --> 00:08:52,069 Yeah, 195 00:08:52,099 --> 00:08:52,489 vendor. 196 00:08:52,489 --> 00:08:52,669 Yeah. 197 00:08:52,669 --> 00:08:53,449 Called it the vendor. 198 00:08:53,839 --> 00:08:54,649 What's going on here? 199 00:08:55,279 --> 00:08:58,519 They said, well, by any chance did you use the compression feature? 200 00:08:59,734 --> 00:09:00,904 And I said yes. 201 00:09:01,144 --> 00:09:02,014 Yes I did. 202 00:09:02,374 --> 00:09:03,124 They go, yeah. 203 00:09:03,304 --> 00:09:06,394 So let us explain to you how the compression feature works. 204 00:09:06,814 --> 00:09:14,464 So when we're backing up files, um, basically, um, we do the equivalent 205 00:09:14,464 --> 00:09:19,054 of compress minus CI think it was, to, to send the result to standard out 206 00:09:19,054 --> 00:09:20,434 and it send it straight to the tape. 207 00:09:21,844 --> 00:09:23,614 But we don't know how to do that on the way back in. 208 00:09:24,334 --> 00:09:26,284 And so what we do is. 209 00:09:27,319 --> 00:09:32,179 We, uh, we, we read the tape, we read the entire file that we're gonna restore 210 00:09:32,359 --> 00:09:40,549 into, and we restore it to temp, and then we run uncompress in place in 211 00:09:40,549 --> 00:09:43,969 temp, and then we move the res, the 212 00:09:44,029 --> 00:09:45,799 uncompressed restored file. 213 00:09:47,059 --> 00:09:48,139 To where it's gonna go. 214 00:09:48,529 --> 00:09:51,349 And we do that one file at a 215 00:09:51,784 --> 00:09:51,964 Oh, 216 00:09:52,159 --> 00:09:55,279 because we're concerned that we're gonna fill up temp. 217 00:09:55,699 --> 00:09:57,169 And I'm like, oh. 218 00:09:57,349 --> 00:10:01,249 So like if temp is, if I have a single file that's bigger than 219 00:10:01,249 --> 00:10:03,259 temp, it's just not gonna work. 220 00:10:03,319 --> 00:10:04,579 They're like, yeah. 221 00:10:05,539 --> 00:10:09,619 And they're like, if this is a concern, we suggest perhaps 222 00:10:09,619 --> 00:10:10,879 you don't use this feature. 223 00:10:11,829 --> 00:10:13,929 it would've been helpful to know before I needed 224 00:10:14,139 --> 00:10:15,244 Right, right. 225 00:10:15,574 --> 00:10:18,964 And so thank God I had the, I had the backup tape in my backup 226 00:10:18,964 --> 00:10:20,164 pocket and I pulled 'em out. 227 00:10:20,164 --> 00:10:21,784 I was like, okay, we're just using, you know, 228 00:10:22,264 --> 00:10:23,914 uh, dump here. 229 00:10:24,274 --> 00:10:27,454 And, um, and we restored and everything was basic, right? 230 00:10:27,484 --> 00:10:30,214 Um, and luckily I had backups from the previous night. 231 00:10:31,294 --> 00:10:33,154 And really the moral of this story is. 232 00:10:34,429 --> 00:10:35,209 Don't do that. 233 00:10:36,139 --> 00:10:36,319 Right. 234 00:10:37,609 --> 00:10:41,149 Don't test your backups before 235 00:10:41,209 --> 00:10:42,079 you need them. 236 00:10:42,109 --> 00:10:42,559 Right. 237 00:10:42,889 --> 00:10:45,469 Do a DR test and that's what we're talking about today. 238 00:10:45,469 --> 00:10:53,209 Do a DR test before you actually need to do DR and, and thi this May, we 239 00:10:53,239 --> 00:10:57,049 will see I the, you know, sometimes when we have these episodes, persona, 240 00:10:57,049 --> 00:10:59,179 you know, you know, we have these conversations beforehand and I'm 241 00:10:59,179 --> 00:11:01,939 like, I don't know if we can fill up an entire episode over this topic. 242 00:11:01,939 --> 00:11:03,379 We said that I think over the last one. 243 00:11:03,604 --> 00:11:03,904 Yeah, 244 00:11:04,429 --> 00:11:06,229 And it ended up not being a problem at all. 245 00:11:06,499 --> 00:11:07,879 This is not one of those episodes. 246 00:11:07,879 --> 00:11:11,239 This is one of those episodes where I think we might go along and we'll end 247 00:11:11,239 --> 00:11:16,729 up turning this into two episodes, um, because there's a lot to talk about here 248 00:11:17,089 --> 00:11:29,119 because before we do any testing, right, um, what do you think we need to do? 249 00:11:30,934 --> 00:11:34,954 Well, you, well, you need to understand, 250 00:11:35,239 --> 00:11:35,359 there. 251 00:11:35,359 --> 00:11:37,159 By the way, there's probably no wrong answer here. 252 00:11:37,459 --> 00:11:37,999 Uh, but 253 00:11:38,029 --> 00:11:40,249 well, well, I think before you could 254 00:11:40,339 --> 00:11:43,069 un unless your answer is, unless your answer is nothing, 255 00:11:43,579 --> 00:11:44,119 um, 256 00:11:44,184 --> 00:11:44,405 no, no. 257 00:11:44,929 --> 00:11:46,309 So I was, I was going through my head. 258 00:11:46,309 --> 00:11:49,579 I was thinking even before you get to testing, 259 00:11:49,754 --> 00:11:50,044 Yeah. 260 00:11:50,149 --> 00:11:54,019 you need to understand what are the business requirements for how quickly 261 00:11:54,019 --> 00:11:59,389 you need to bring up that site, which in which actually you don't 262 00:11:59,389 --> 00:12:01,939 necessarily think about as testing, but that's actually part of your 263 00:12:01,939 --> 00:12:05,599 backup system design before you even get to the testing part. 264 00:12:06,709 --> 00:12:07,159 Yeah. 265 00:12:07,159 --> 00:12:13,309 So we, we have to agree on what success would be, right? 266 00:12:13,759 --> 00:12:18,259 And we have to agree on obviously what we're gonna test, but we have to 267 00:12:18,259 --> 00:12:21,049 agree on the parameters of that test. 268 00:12:26,779 --> 00:12:28,549 And so, um. 269 00:12:30,439 --> 00:12:30,889 What? 270 00:12:30,889 --> 00:12:32,599 What do you think are going what? 271 00:12:32,599 --> 00:12:33,049 Go ahead. 272 00:12:33,424 --> 00:12:37,144 You and I was just going to ask about like your parameters, right? 273 00:12:37,144 --> 00:12:37,444 What are 274 00:12:37,444 --> 00:12:39,244 those parameters in my mind? 275 00:12:39,724 --> 00:12:47,284 Some of those are how quickly do I need to be able to bring up what the scope is, 276 00:12:47,614 --> 00:12:47,944 right. 277 00:12:47,944 --> 00:12:49,774 Am I failing over and 278 00:12:49,774 --> 00:12:56,044 trying to recover a file, an application, a data center, right? 279 00:12:56,104 --> 00:12:58,384 What am I looking to actually test? 280 00:12:58,924 --> 00:13:00,699 So it depends what you're trying to test. 281 00:13:01,369 --> 00:13:03,469 And what the extent is that you want to test. 282 00:13:03,559 --> 00:13:07,759 And also, I think the other thing is how close do you want to get to testing 283 00:13:07,759 --> 00:13:10,429 an actual disaster as part of that? 284 00:13:12,089 --> 00:13:12,439 Right. 285 00:13:12,739 --> 00:13:19,324 Um, and I, I would say that there are different kinds of DR tests and, um. 286 00:13:20,374 --> 00:13:29,704 The, there's, and, and if you haven't done any, I would say let's start small, right? 287 00:13:29,704 --> 00:13:34,624 If we've never, if, if we've never done a DR test of any kind, I would 288 00:13:34,624 --> 00:13:38,374 probably start with, what are you 289 00:13:38,464 --> 00:13:44,584 I was thinking about the test you don't wanna try is who is the guy in Alaska? 290 00:13:44,764 --> 00:13:45,064 Oh yeah, 291 00:13:47,134 --> 00:13:47,494 yeah. 292 00:13:47,704 --> 00:13:48,124 Okay. 293 00:13:48,214 --> 00:13:52,234 We definitely have to put a link to that episode. 294 00:13:52,294 --> 00:13:54,694 That was the most amazing episode ever. 295 00:13:54,784 --> 00:13:55,654 In fact, you know what? 296 00:13:55,684 --> 00:14:00,394 We'll pro we'll probably replay it over, uh, over the holiday break, Uh, 297 00:14:00,394 --> 00:14:04,714 by the way, we love him and it, and, and it had a happy ending, but oh my God. 298 00:14:04,714 --> 00:14:05,764 What, what a 299 00:14:05,764 --> 00:14:06,484 nightmare, 300 00:14:06,574 --> 00:14:09,394 think he was rebuilding a raid array, if I recall. 301 00:14:09,914 --> 00:14:11,594 He was swapping the disks around 302 00:14:11,674 --> 00:14:17,984 it was self-inflicted in that he said, I want to move the discs around. 303 00:14:18,044 --> 00:14:18,644 Because they were like. 304 00:14:18,644 --> 00:14:19,874 different sizes I 305 00:14:20,144 --> 00:14:20,624 Yeah. 306 00:14:20,684 --> 00:14:20,924 Yeah. 307 00:14:20,924 --> 00:14:24,434 Like, and he's like, the only way I can do this is to just, to just 308 00:14:24,434 --> 00:14:26,384 wipe everything and start over. 309 00:14:26,384 --> 00:14:27,614 And he is like, yeah, okay. 310 00:14:27,614 --> 00:14:28,394 So that's what I'll do. 311 00:14:28,514 --> 00:14:31,994 I'll just wipe everything and then I'll restore everything. 312 00:14:32,744 --> 00:14:34,574 My s my backup system works. 313 00:14:34,844 --> 00:14:37,004 And this is how he tested his backup system. 314 00:14:37,304 --> 00:14:38,624 Please do. 315 00:14:38,774 --> 00:14:43,964 He definitely learned some things along the way, but, um, yeah, so don't do 316 00:14:44,114 --> 00:14:44,714 Don't do that. 317 00:14:44,864 --> 00:14:45,164 Yeah, 318 00:14:45,299 --> 00:14:46,139 Don't do that. 319 00:14:46,499 --> 00:14:47,459 Uh, but 320 00:14:47,669 --> 00:14:49,169 learn from his example. 321 00:14:49,169 --> 00:14:50,549 Listen to that episode and, and, 322 00:14:50,549 --> 00:14:51,209 and have 323 00:14:51,209 --> 00:14:52,829 a heart attack as you're listening. 324 00:14:52,964 --> 00:14:56,504 because the thing that ran through my head was someone who's never done DR 325 00:14:56,504 --> 00:15:00,674 testing would've been like, yeah, I'm just gonna shoot in the head my production 326 00:15:00,674 --> 00:15:04,574 site with all the applications, and I make sure that it, or just test it out 327 00:15:04,574 --> 00:15:05,834 and see does it fail over properly? 328 00:15:06,354 --> 00:15:06,704 Yes. 329 00:15:06,704 --> 00:15:10,574 The key here is non-destructive DR testing. 330 00:15:11,234 --> 00:15:11,804 Right? 331 00:15:11,894 --> 00:15:18,944 Um, and so, yeah, so, so to go back to like setting the scope, I, if this 332 00:15:18,944 --> 00:15:23,024 is your first time doing DR testing, I would set the scope as small as possible. 333 00:15:23,024 --> 00:15:28,034 In fact, if you've never done DR testing, I would not even be doing DR testing. 334 00:15:28,034 --> 00:15:30,134 I would just be doing restore testing 335 00:15:30,764 --> 00:15:31,454 and 336 00:15:31,484 --> 00:15:32,834 What's the difference, Curtis? 337 00:15:33,059 --> 00:15:33,629 What's that? 338 00:15:33,734 --> 00:15:34,034 What's the 339 00:15:34,439 --> 00:15:39,479 the question is, the question is are we going to declare a disaster 340 00:15:39,479 --> 00:15:44,909 and say that this site is down in some way, um, and then we're 341 00:15:44,909 --> 00:15:46,739 gonna fail over to another site? 342 00:15:47,069 --> 00:15:50,999 Or are we simply just going to bring another server online? 343 00:15:50,999 --> 00:15:51,479 Right. 344 00:15:52,079 --> 00:15:55,919 De depending on how you define, uh, some, and, you know, depending on the 345 00:15:55,919 --> 00:15:59,999 day of the week and the day of the year in which year it is, I have called. 346 00:16:00,704 --> 00:16:05,474 Every time you have to restore, uh, a server of any kind, a disaster. 347 00:16:05,474 --> 00:16:07,394 It's just a disaster of different levels, right? 348 00:16:07,394 --> 00:16:12,044 So if the server, if a server caught on fire, that's a disaster, 349 00:16:12,074 --> 00:16:12,614 right? 350 00:16:12,704 --> 00:16:17,084 Um, and, or, or if a server just died, right? 351 00:16:17,084 --> 00:16:20,624 Or if, like the, the story in the beginning, I think that was a disaster 352 00:16:20,744 --> 00:16:22,964 because it took out a production workload, 353 00:16:23,564 --> 00:16:24,074 right? 354 00:16:24,254 --> 00:16:27,554 So I would say perhaps start with a single. 355 00:16:28,319 --> 00:16:31,469 Production workload and restore it. 356 00:16:31,979 --> 00:16:37,379 First off, just restore it, you know, like in place, not, not in place, not 357 00:16:37,379 --> 00:16:39,749 in place, I meant like within the data 358 00:16:39,749 --> 00:16:42,539 center or within whatever computing environment you're using. 359 00:16:43,139 --> 00:16:46,709 And then start talking about VPNs and 360 00:16:46,979 --> 00:16:48,389 you know, that sort of stuff. 361 00:16:48,419 --> 00:16:52,679 So, uh, that would be defining the scope as small as you can 362 00:16:53,099 --> 00:16:54,959 for the first test that you can. 363 00:16:55,094 --> 00:17:00,584 so maybe like, uh, like maybe it might be a directory within a file server. 364 00:17:00,584 --> 00:17:01,274 If you've never done 365 00:17:01,604 --> 00:17:03,524 any restore testing, yes. 366 00:17:03,734 --> 00:17:09,074 A simple directory within a file server, does our backup system work at all? 367 00:17:09,494 --> 00:17:09,944 Right. 368 00:17:10,364 --> 00:17:15,374 Um, and then the next, the next is going to be some type of 369 00:17:15,824 --> 00:17:18,074 recovery of an entire server. 370 00:17:18,414 --> 00:17:24,264 Hopefully you have that server virtualized because doing that is going to be, 371 00:17:24,354 --> 00:17:30,774 uh, obviously much easier assuming you're treating that VM as a vm. 372 00:17:31,554 --> 00:17:36,114 Uh, if it's not virtualized, then we we're gonna start going down the 373 00:17:36,114 --> 00:17:38,184 level of, um, bare metal recovery, 374 00:17:38,214 --> 00:17:38,514 right? 375 00:17:38,514 --> 00:17:41,394 BMR By the way, we should probably do an episode on that. 376 00:17:41,394 --> 00:17:41,454 I 377 00:17:41,454 --> 00:17:42,204 didn't think about that. 378 00:17:42,204 --> 00:17:43,254 We should do an episode on that. 379 00:17:44,304 --> 00:17:48,024 I know you keep mentioning server as you're talking about this. 380 00:17:48,474 --> 00:17:53,154 Would you also qualify that that could be server or an application? 381 00:17:55,104 --> 00:17:56,364 thanks for bringing that up. 382 00:17:58,389 --> 00:17:59,859 That's why you keep me around, you know? 383 00:18:00,724 --> 00:18:04,089 So it depends on what you mean by application, right? 384 00:18:04,689 --> 00:18:05,319 Um. 385 00:18:05,889 --> 00:18:08,229 You could, these are like the different 386 00:18:08,229 --> 00:18:08,499 level. 387 00:18:08,529 --> 00:18:10,209 You talk about restoring a directory. 388 00:18:10,569 --> 00:18:16,359 I would also look at restoring a single database within a server, right? 389 00:18:16,389 --> 00:18:18,939 If that's what you mean by application, then I would say yes. 390 00:18:19,299 --> 00:18:22,329 Sometimes when we say application, actually a lot of times when we 391 00:18:22,329 --> 00:18:23,949 say application, what we mean is 392 00:18:24,129 --> 00:18:24,969 application 393 00:18:25,164 --> 00:18:25,614 Multiple 394 00:18:25,749 --> 00:18:28,539 on multiple servers and multiple things, all interconnect. 395 00:18:28,569 --> 00:18:30,879 And if that's what you're talking about, I'm gonna say no 396 00:18:31,104 --> 00:18:31,284 yeah. 397 00:18:31,284 --> 00:18:31,524 yeah, I 398 00:18:31,524 --> 00:18:32,334 was thinking like a, my, 399 00:18:32,439 --> 00:18:33,129 time out. 400 00:18:33,144 --> 00:18:35,904 yeah, I was thinking more like restore your MySQL database. 401 00:18:36,804 --> 00:18:37,494 Exactly 402 00:18:37,674 --> 00:18:38,124 right. 403 00:18:38,514 --> 00:18:39,384 Um, 404 00:18:39,414 --> 00:18:40,854 table within your MySQL database. 405 00:18:40,854 --> 00:18:41,214 Right. 406 00:18:41,364 --> 00:18:41,934 yeah. 407 00:18:41,994 --> 00:18:44,634 Um, and again, non-destructively, 408 00:18:44,949 --> 00:18:45,069 Yeah, 409 00:18:45,864 --> 00:18:46,284 right. 410 00:18:46,794 --> 00:18:47,454 Um, I. 411 00:18:47,784 --> 00:18:53,484 Uh, just, just a, just an entity and you can try all of these things, right? 412 00:18:53,484 --> 00:18:54,954 You can try restoring a database. 413 00:18:54,954 --> 00:18:59,754 You can try restoring an entire, all of the, all of the databases on a server. 414 00:18:59,754 --> 00:19:05,244 You can try restoring a server with the databases, um, and or any other 415 00:19:05,244 --> 00:19:07,284 applications that might like a web server. 416 00:19:07,524 --> 00:19:12,024 You can try all of these things individually and make sure that 417 00:19:12,024 --> 00:19:14,244 you've got those pieces down. 418 00:19:14,709 --> 00:19:16,419 That this is about defining the scope. 419 00:19:16,809 --> 00:19:22,059 Try the all of the different pieces first and make sure you've got the, 420 00:19:22,089 --> 00:19:27,639 the recovery path for each of those parts of your infrastructure down 421 00:19:27,819 --> 00:19:31,839 before you decide, okay, we're gonna pretend we're gonna blow up the data 422 00:19:31,839 --> 00:19:32,169 center. 423 00:19:32,454 --> 00:19:37,524 and the other reason to bring that up, it's important to test these different 424 00:19:37,524 --> 00:19:42,894 types of components because there are gonna be different nuances like how 425 00:19:42,894 --> 00:19:45,054 you deal with databases and recovering. 426 00:19:45,054 --> 00:19:48,174 That is definitely gonna be different than file servers, which is probably 427 00:19:48,174 --> 00:19:49,794 also gonna be different than servers. 428 00:19:50,244 --> 00:19:55,704 And so it's important to understand the nuances of what is possible and 429 00:19:55,704 --> 00:19:57,024 the steps for each one of these. 430 00:19:57,894 --> 00:20:01,974 And also the different backup systems. 431 00:20:02,664 --> 00:20:03,989 So if you've got. 432 00:20:04,974 --> 00:20:08,604 Physical, you know, you've got physical servers that are not virtualized. 433 00:20:08,934 --> 00:20:15,204 You've got servers running in Hyper V or VMware or you know, any, any, any sort 434 00:20:15,204 --> 00:20:17,514 of on-premises virtualization set. 435 00:20:17,514 --> 00:20:18,294 What was the third one 436 00:20:18,414 --> 00:20:19,014 Broadcom, 437 00:20:19,884 --> 00:20:20,094 brought? 438 00:20:20,199 --> 00:20:20,619 VMware, 439 00:20:21,624 --> 00:20:22,674 It's, it will always be 440 00:20:22,704 --> 00:20:23,514 VMware to me. 441 00:20:23,514 --> 00:20:31,344 It will always be world, um, the, um, if you've got. 442 00:20:32,514 --> 00:20:38,514 VMs running in, uh, AWS, GCP Azure. 443 00:20:39,084 --> 00:20:43,194 If you've got basically all of the different places that you 444 00:20:43,194 --> 00:20:46,884 have infrastructure, probably have different backup and recovery 445 00:20:46,884 --> 00:20:48,474 methodologies for each of them. 446 00:20:49,014 --> 00:20:52,104 And so you should also be looking at testing all of those. 447 00:20:52,659 --> 00:20:57,909 And you should be looking at testing each of them individually as a, 448 00:20:57,969 --> 00:21:03,459 an overall process that you're working towards developing, uh, 449 00:21:03,489 --> 00:21:05,619 declaring a much bigger disaster. 450 00:21:05,929 --> 00:21:06,219 Yeah. 451 00:21:08,114 --> 00:21:12,639 I, I think it's important also to be familiar because sometimes you might have 452 00:21:12,639 --> 00:21:15,789 a real disaster that doesn't require you 453 00:21:15,789 --> 00:21:22,509 to recover every single component within that higher level application, right? 454 00:21:22,509 --> 00:21:26,559 So being familiar with the individual components is also important 455 00:21:27,099 --> 00:21:28,989 depending on what the disaster is. 456 00:21:29,844 --> 00:21:30,264 Yeah. 457 00:21:30,264 --> 00:21:30,894 Agreed. 458 00:21:31,134 --> 00:21:35,694 Um, and you know, there, there, there's an application that we haven't 459 00:21:35,694 --> 00:21:41,244 talked about in terms of including or not including it in your Dr. 460 00:21:42,594 --> 00:21:49,344 Scope and that is, what about SaaS applications like Microsoft 365? 461 00:21:50,094 --> 00:21:54,084 Um, there are a couple of different scenarios there. 462 00:21:55,704 --> 00:21:57,624 One is your. 463 00:21:58,389 --> 00:22:05,409 Um, account is damaged in some sort of logical way, meaning logical 464 00:22:05,409 --> 00:22:07,899 corruption, meaning a ransomware attack. 465 00:22:08,319 --> 00:22:09,639 You deleted it. 466 00:22:09,789 --> 00:22:10,089 Right? 467 00:22:10,089 --> 00:22:11,379 We, we've covered that 468 00:22:11,679 --> 00:22:12,159 on, 469 00:22:12,179 --> 00:22:13,409 provider damaged it. 470 00:22:13,869 --> 00:22:14,199 yeah. 471 00:22:14,199 --> 00:22:14,919 The provider. 472 00:22:14,919 --> 00:22:17,439 Well, that's, that, that was, I'm gonna list that as like a 473 00:22:17,439 --> 00:22:17,859 third. 474 00:22:17,859 --> 00:22:20,349 Well, if they damaged your account. 475 00:22:20,889 --> 00:22:22,629 And there is an example of that. 476 00:22:22,689 --> 00:22:24,879 Uh, for example, the sales for 477 00:22:25,044 --> 00:22:25,284 Yep. 478 00:22:25,284 --> 00:22:26,019 That's what I was thinking. 479 00:22:26,559 --> 00:22:32,949 story where they went and blew up everybody's permissions and everybody 480 00:22:32,949 --> 00:22:34,779 had to restore that themselves. 481 00:22:35,139 --> 00:22:38,379 Um, that man, I hate that story. 482 00:22:38,739 --> 00:22:43,809 I really do, because Salesforce, in my opinion, did not own up to. 483 00:22:44,049 --> 00:22:46,659 Uh, they, they, didn't step up to the plate 484 00:22:46,944 --> 00:22:47,094 Yeah. 485 00:22:47,834 --> 00:22:48,699 at, at the time. 486 00:22:48,699 --> 00:22:52,269 I remember writing a blog post for, uh, Druva. 487 00:22:52,269 --> 00:22:56,559 I was working for Druva at the time, and I remember writing for Blo, a blog 488 00:22:56,559 --> 00:22:59,949 post that said something like, proof that, that Salesforce should not be 489 00:22:59,949 --> 00:23:01,629 trusted with your backup infrastructure. 490 00:23:02,139 --> 00:23:04,269 Um, 'cause they clearly don't know what they're doing. 491 00:23:06,099 --> 00:23:11,319 The but there's, but, so there's your account being damaged in some way. 492 00:23:11,319 --> 00:23:12,849 And then there's OVH Cloud. 493 00:23:12,954 --> 00:23:18,219 And what happened there Where the entire infrastructure goes? 494 00:23:18,924 --> 00:23:19,374 Poof. 495 00:23:20,544 --> 00:23:20,694 Right. 496 00:23:21,024 --> 00:23:28,314 So, uh, I think you should have that as, as you, you should come up with that 497 00:23:28,314 --> 00:23:30,834 as a scenario that you need to test. 498 00:23:31,434 --> 00:23:37,734 It's going to be challenging in most cases because just let's just 499 00:23:37,734 --> 00:23:38,934 talk about the different scenarios. 500 00:23:38,934 --> 00:23:45,054 Let's say it's AWS the way most people back up AWS If AWS goes down 501 00:23:45,054 --> 00:23:46,704 and takes your backups with it. 502 00:23:47,574 --> 00:23:48,234 You're screwed. 503 00:23:48,744 --> 00:23:49,524 You're screwed. 504 00:23:50,124 --> 00:23:53,514 Um, the way that most people back up most cloud infrastructure. 505 00:23:54,714 --> 00:24:02,604 And then there's the fact that most people trust their SaaS provider for data 506 00:24:02,604 --> 00:24:04,404 protection, which they should not, right? 507 00:24:04,404 --> 00:24:05,514 I talk about that all the time. 508 00:24:05,604 --> 00:24:10,989 They should not, But, but but even if you're backing up your, your 509 00:24:10,989 --> 00:24:16,179 data, um, to a third party and, and it's not, they should not. 510 00:24:16,179 --> 00:24:16,419 Right? 511 00:24:16,419 --> 00:24:17,559 I talk about that all the time. 512 00:24:17,649 --> 00:24:18,129 They should not. 513 00:24:18,129 --> 00:24:18,489 This has been 514 00:24:18,489 --> 00:24:19,389 a day in eight. 515 00:24:19,749 --> 00:24:20,349 Problem though, 516 00:24:20,794 --> 00:24:23,974 This is an age old problem, but we're talking about testing today. 517 00:24:24,094 --> 00:24:26,734 So by the way, this is definitely gonna be two episodes. 518 00:24:26,734 --> 00:24:28,264 We haven't even gotten to the testing yet. 519 00:24:28,264 --> 00:24:30,364 All we're talking about is setting up the requirements. 520 00:24:31,309 --> 00:24:31,529 So, 521 00:24:32,344 --> 00:24:33,694 so this is definitely gonna be a second 522 00:24:33,694 --> 00:24:34,114 episode. 523 00:24:35,134 --> 00:24:40,564 Um, so we talk about, we talk about setting the scope and we 524 00:24:40,564 --> 00:24:43,204 talk about starting small first. 525 00:24:43,684 --> 00:24:50,374 Each of these individual components in your infrastructure and um, 526 00:24:50,644 --> 00:24:51,519 were you about to say something? 527 00:24:51,519 --> 00:24:54,134 that this is not destructive and 528 00:24:54,134 --> 00:24:55,454 not disruptive. 529 00:24:55,919 --> 00:24:56,039 Yeah. 530 00:24:56,699 --> 00:24:57,779 Nice, nice. 531 00:24:57,869 --> 00:24:58,379 I like that. 532 00:24:58,379 --> 00:25:00,449 Non-destructive, non-disruptive DR testing. 533 00:25:00,449 --> 00:25:00,569 I 534 00:25:00,569 --> 00:25:01,049 like it. 535 00:25:01,319 --> 00:25:03,449 because you don't wanna take down your production. 536 00:25:03,509 --> 00:25:06,479 You don't wanna affect, say, ongoing Dr. 537 00:25:06,479 --> 00:25:10,019 Resiliency that's available on the secondary site because you're 538 00:25:10,019 --> 00:25:11,459 about to do some of this testing. 539 00:25:11,699 --> 00:25:15,809 There are cases where you do want to impact that, but when you're doing sort 540 00:25:15,809 --> 00:25:19,859 of these individual component levels, you may not want to impact your overall DR. 541 00:25:19,859 --> 00:25:22,019 Posture while you're doing this 542 00:25:22,844 --> 00:25:26,564 Yeah, that that last one is probably the hardest and it 543 00:25:26,564 --> 00:25:28,484 and may actually be impossible. 544 00:25:28,484 --> 00:25:29,924 What are we talking about there? 545 00:25:29,924 --> 00:25:35,024 We're saying that if you have a DR system, I hope you have a DR system 546 00:25:35,054 --> 00:25:40,004 if you have one, you know, see if there's a way 547 00:25:40,844 --> 00:25:44,924 to test your DR without messing up your dr. 548 00:25:44,984 --> 00:25:47,144 Um, I'm not sure if that's possible 549 00:25:47,144 --> 00:25:48,764 in, in many scenarios, but. 550 00:25:49,064 --> 00:25:52,004 but one of the things I think about right is a lot of data protection 551 00:25:52,004 --> 00:25:54,374 vendors, they do test and dev, right? 552 00:25:54,374 --> 00:25:54,944 You can spin 553 00:25:54,944 --> 00:25:59,564 up a copy off of your storage that is a writeable copy that you can then use 554 00:25:59,894 --> 00:26:02,384 for testing out your recovery systems 555 00:26:02,894 --> 00:26:07,034 without impacting the actual recovery instance. 556 00:26:07,544 --> 00:26:07,964 right. 557 00:26:08,504 --> 00:26:12,104 And so that's what I'm thinking about is just like those sort of scenarios or maybe 558 00:26:12,104 --> 00:26:17,024 you're able to wheel in an extra server or beg, borrow steel, an extra server to use 559 00:26:17,024 --> 00:26:19,844 for your recovery testing or DR testing, 560 00:26:20,504 --> 00:26:20,714 Yeah. 561 00:26:20,714 --> 00:26:23,564 You know, it's, it's funny that you say that because I, I, you know, when 562 00:26:23,564 --> 00:26:26,084 you, I like this idea wheeling in a, a 563 00:26:26,084 --> 00:26:26,534 server. 564 00:26:26,534 --> 00:26:30,494 I mean, I, I, I think many of our listeners, you know, and, and I, and every 565 00:26:30,494 --> 00:26:33,404 time I, by default, I'm always talking about data center, even though I, you 566 00:26:33,404 --> 00:26:34,664 know, and then in my brain goes, Hey, 567 00:26:34,724 --> 00:26:36,074 nobody has a data center anymore. 568 00:26:36,794 --> 00:26:39,224 Um, I don't think many people are wheeling in a server. 569 00:26:39,224 --> 00:26:41,654 I think that they're, they're, you know, they're looking at. 570 00:26:41,924 --> 00:26:42,614 Cloud 571 00:26:42,644 --> 00:26:43,964 infrastructure as a way. 572 00:26:44,564 --> 00:26:46,634 I, I know that not everybody can do that. 573 00:26:46,724 --> 00:26:50,384 And that's obviously another thing that you have to decide upfront is 574 00:26:50,534 --> 00:26:53,144 how are we going to do this recovery? 575 00:26:53,144 --> 00:26:55,154 Are we going to do it in the cloud? 576 00:26:55,154 --> 00:26:55,664 Are we gonna do it? 577 00:26:55,664 --> 00:27:01,064 The alternate infras, alternate infrastructure, um, these are all 578 00:27:01,064 --> 00:27:03,134 things that you have to decide upfront. 579 00:27:03,554 --> 00:27:06,074 We have to decide what it is we're gonna restore. 580 00:27:06,494 --> 00:27:10,994 We have to decide, um, where we're going to restore and, and how. 581 00:27:11,339 --> 00:27:13,019 Right and. 582 00:27:13,409 --> 00:27:15,599 The deciding the where. 583 00:27:15,629 --> 00:27:17,189 Again, this is all about planning. 584 00:27:17,189 --> 00:27:18,509 This is all stuff that needs to be 585 00:27:18,509 --> 00:27:19,499 decided upfront. 586 00:27:19,499 --> 00:27:22,859 This is also something that's going to be part of your backup and Dr. 587 00:27:22,859 --> 00:27:29,579 Design, you will have decided up upfront, you know how we're going to do Dr. 588 00:27:29,669 --> 00:27:32,279 Uh, well, disaster recovery, how we're going to do it. 589 00:27:32,964 --> 00:27:37,374 And that place is most likely the place that you're going 590 00:27:37,374 --> 00:27:39,594 to, uh, be doing DR testing. 591 00:27:40,074 --> 00:27:42,084 And, you know, there are a lot of choices here. 592 00:27:42,084 --> 00:27:45,954 Cloud infrastructure, I think, is the best choice for most environments. 593 00:27:46,554 --> 00:27:49,494 Another choice is aging infrastructure, right? 594 00:27:49,494 --> 00:27:49,674 So 595 00:27:49,674 --> 00:27:52,344 you move, you know, you move your older stuff out and that 596 00:27:52,344 --> 00:27:54,204 becomes your DR environment. 597 00:27:54,684 --> 00:27:59,934 Um, another choice, another very common choice is basically, um, 598 00:27:59,934 --> 00:28:01,644 there, there's two ways to basically. 599 00:28:02,474 --> 00:28:07,274 Rent infrastructure for the purposes of disaster recovery. 600 00:28:07,634 --> 00:28:13,004 One way is to contract with, um, you know, a company that will provide 601 00:28:13,844 --> 00:28:16,724 what you need if and when you need it. 602 00:28:17,114 --> 00:28:19,754 Uh, and that costs, you know, let's say this much. 603 00:28:20,114 --> 00:28:23,684 And then there's a company that will provide everything you need, 604 00:28:23,984 --> 00:28:27,254 always available to you all the time. 605 00:28:27,584 --> 00:28:28,874 Even when you don't need it. 606 00:28:28,874 --> 00:28:30,494 And that will cost this much. 607 00:28:30,584 --> 00:28:31,274 Yeah, exactly. 608 00:28:31,484 --> 00:28:33,374 I can't do the fingers, I gotta do the hands. 609 00:28:33,374 --> 00:28:33,674 Right. 610 00:28:34,214 --> 00:28:35,744 It's significantly more. 611 00:28:35,774 --> 00:28:39,524 Uh, and, and this is why I push everybody to the cloud as much as you can. 612 00:28:39,524 --> 00:28:42,704 'cause that's the beautiful thing about the cloud, is that you can just literally 613 00:28:42,704 --> 00:28:47,684 snap your fingers and, um, you know, and use, and you can also use infrastructure 614 00:28:47,684 --> 00:28:51,794 as code so that you can just make all of the hardware that you need 615 00:28:51,824 --> 00:28:53,114 magically appear. 616 00:28:53,744 --> 00:28:54,224 What's that? 617 00:28:54,434 --> 00:28:55,484 and work right. 618 00:28:55,679 --> 00:28:56,309 And work. 619 00:28:56,309 --> 00:28:56,759 Yeah. 620 00:28:56,969 --> 00:28:58,979 Magically appear when you need it. 621 00:28:58,979 --> 00:29:02,429 And then the moment you no longer need it because your test is over, you can 622 00:29:02,429 --> 00:29:07,229 also snap your fingers and it all goes away and you just pay for it only when, 623 00:29:07,709 --> 00:29:10,019 uh, it's, you know, up and running. 624 00:29:10,679 --> 00:29:17,249 Um, so the, the next thing to talk about is, so we decided 625 00:29:17,249 --> 00:29:18,239 what it's we're gonna test. 626 00:29:18,239 --> 00:29:20,729 We decided where we're gonna test it, how we're gonna test it, 627 00:29:21,359 --> 00:29:23,369 what about our success criteria? 628 00:29:26,039 --> 00:29:26,609 Yeah. 629 00:29:26,609 --> 00:29:32,519 I think this is important to note upfront what it means to be 630 00:29:32,519 --> 00:29:40,349 successful, but I think it's also important to be realistic, right. 631 00:29:40,409 --> 00:29:44,129 With your success criteria, especially if this is sort of your 632 00:29:44,129 --> 00:29:48,419 first time doing this, because I remember the story you tell Curtis 633 00:29:48,419 --> 00:29:50,819 about your, your runbooks that you 634 00:29:50,929 --> 00:29:51,219 Yeah. 635 00:29:51,219 --> 00:29:51,379 Yeah. 636 00:29:51,869 --> 00:29:53,219 when you used to work at the bank. 637 00:29:53,819 --> 00:29:58,079 And how a lot of times they were not able to get through and 638 00:29:58,079 --> 00:30:02,039 actually do the recovery because they, a step would be skipped or 639 00:30:02,039 --> 00:30:03,539 something wouldn't work properly. 640 00:30:04,349 --> 00:30:07,469 And so I think it's sort of a learning process. 641 00:30:07,619 --> 00:30:09,299 So don't be too hard on yourself. 642 00:30:09,299 --> 00:30:14,969 If the first 10, 20, a hundred times you try doing your recovery testing, it's 643 00:30:14,969 --> 00:30:17,609 not like a hundred percent successful. 644 00:30:18,104 --> 00:30:18,554 Yeah. 645 00:30:18,674 --> 00:30:19,964 may not be your fault. 646 00:30:19,994 --> 00:30:22,064 Things change, environments change, hard 647 00:30:22,064 --> 00:30:23,264 work may change, right? 648 00:30:23,264 --> 00:30:24,854 There's so many other factors, 649 00:30:25,514 --> 00:30:25,784 but. 650 00:30:26,999 --> 00:30:31,439 this is, this is very closely related to the discussions we've had over, um, 651 00:30:31,619 --> 00:30:34,679 doing, uh, um, cyber recovery testing. 652 00:30:34,709 --> 00:30:35,039 Right? 653 00:30:35,129 --> 00:30:38,249 And your disaster recovery very likely will be part of an 654 00:30:38,249 --> 00:30:40,139 overall cyber recovery process. 655 00:30:40,649 --> 00:30:42,434 And I agree with you that I. 656 00:30:42,779 --> 00:30:43,829 Um, it's interesting. 657 00:30:43,829 --> 00:30:46,379 I was go mentally, I was going somewhere completely different. 658 00:30:46,379 --> 00:30:49,559 But you, you, once again, this is why we make such a good team. 659 00:30:50,189 --> 00:30:56,249 Um, you were more like, Hey, make the re make the, um, um, 660 00:30:56,879 --> 00:30:58,349 you know, the requirements. 661 00:30:58,469 --> 00:31:01,859 Uh, be nice to yourself, especially if it's the first time out. 662 00:31:02,069 --> 00:31:05,729 You know, requirement number one, no one dies, right? 663 00:31:06,984 --> 00:31:07,204 Yep. 664 00:31:07,529 --> 00:31:08,939 No fires are created. 665 00:31:09,299 --> 00:31:10,439 No one quits. 666 00:31:10,679 --> 00:31:11,339 Um. 667 00:31:11,804 --> 00:31:16,094 And, um, the, you know, set your expectations low and nobody can 668 00:31:16,094 --> 00:31:17,444 take you, take 'em away from you. 669 00:31:17,684 --> 00:31:22,064 I'm borrowing heavily from, uh, Michael p Connolly, the comedian who, 670 00:31:22,184 --> 00:31:26,234 who says that's the, the happiness to life is to lower your expectations. 671 00:31:26,594 --> 00:31:30,824 And he's like, my goal for today is to go to the bathroom outside my pants. 672 00:31:31,094 --> 00:31:31,484 Um, 673 00:31:33,974 --> 00:31:34,484 so yeah. 674 00:31:34,484 --> 00:31:38,594 Set the, uh, you know, the success criteria low in the beginning. 675 00:31:39,794 --> 00:31:45,524 Um, where I was going was of course, REO and RPO, which we talked about before. 676 00:31:45,944 --> 00:31:52,454 The, that, that is definitely going to determine and the overall success, 677 00:31:53,624 --> 00:31:58,454 once you click the stopwatch and you begin the recovery process. 678 00:31:58,799 --> 00:32:02,039 And then you click it, that everything is back up and running and, and you've 679 00:32:02,039 --> 00:32:06,629 tested that, that, that whatever it is that you destroyed or you pretended you 680 00:32:06,629 --> 00:32:11,789 destroyed, is now up and running and fully functional Again, as we've mentioned 681 00:32:12,179 --> 00:32:15,989 in the RTO and RPO, that doesn't just mean the time of the restore, it's, 682 00:32:15,989 --> 00:32:17,069 it's, you know, it's. 683 00:32:17,534 --> 00:32:18,554 ah, here's a question. 684 00:32:18,824 --> 00:32:22,364 Is it RTO and RPO or is it RTA and RPA? 685 00:32:22,619 --> 00:32:24,989 the, it's, that is the objective, 686 00:32:25,079 --> 00:32:25,499 right? 687 00:32:25,499 --> 00:32:29,609 The RTO and RPO are, are the objective that we are shooting for. 688 00:32:30,029 --> 00:32:35,999 And so the goal, um, in a recovery test of any kind is that the RTA, that's recovery 689 00:32:35,999 --> 00:32:41,699 time, actual, uh, and recovery point actual, are less than, uh, or equal to 690 00:32:41,909 --> 00:32:43,169 the RTO and RPO. 691 00:32:43,379 --> 00:32:44,579 go back and listen to our episode. 692 00:32:44,579 --> 00:32:46,799 We covered it, I think a couple episodes ago, right? 693 00:32:47,249 --> 00:32:49,379 No, it was, well, well, I don't know. 694 00:32:49,379 --> 00:32:50,849 It depends on when this one gets published. 695 00:32:50,849 --> 00:32:52,079 We just, we just did it. 696 00:32:52,229 --> 00:32:52,439 Yeah. 697 00:32:52,439 --> 00:32:53,864 That's why I'm saying we just did it not. 698 00:32:54,899 --> 00:32:56,399 Oh, was that really the last episode? 699 00:32:56,819 --> 00:32:58,139 Well, it just published. 700 00:32:58,229 --> 00:33:00,599 Um, uh, but again, I don't know, you know, 701 00:33:00,599 --> 00:33:05,639 we'll see how these things, you know, but, um, yeah, if you're, if, if RTO and RPO 702 00:33:05,669 --> 00:33:08,819 don't just roll off your tongue and you don't know what they are and, and, and, 703 00:33:08,879 --> 00:33:12,029 you know, all of that stuff, it, they literally should be in every conversation 704 00:33:12,029 --> 00:33:14,039 having anything to do with backup and Dr. 705 00:33:14,039 --> 00:33:14,489 Design. 706 00:33:16,064 --> 00:33:21,104 But that to me is the ultimate success criteria, right? 707 00:33:21,434 --> 00:33:25,034 Another success criteria is the degree, and this, this is gonna be 708 00:33:25,034 --> 00:33:32,144 a percentage, um, a per a percentage achieved, and that is the degree to which 709 00:33:32,234 --> 00:33:34,484 you were able to follow your runbook 710 00:33:35,414 --> 00:33:36,644 and just do what's in the runbook. 711 00:33:36,674 --> 00:33:36,974 This is 712 00:33:36,974 --> 00:33:39,224 what you, this is what you were alluding to before, right? 713 00:33:39,224 --> 00:33:40,994 Well, and hopefully you have a runbook. 714 00:33:41,669 --> 00:33:41,909 Yeah. 715 00:33:41,939 --> 00:33:44,429 Like you said, that, that, that assumes you have a run book. 716 00:33:44,429 --> 00:33:44,609 Yeah. 717 00:33:45,239 --> 00:33:45,929 Um. 718 00:33:46,424 --> 00:33:48,854 And, and the way we always did this at the bank, as we mentioned 719 00:33:48,854 --> 00:33:53,744 before, is that we would have someone other than the person who runs the 720 00:33:53,744 --> 00:33:55,994 backup system do the DR testing. 721 00:33:56,354 --> 00:33:56,744 Right? 722 00:33:57,014 --> 00:33:59,984 Here's the runbook, please follow it because we're gonna pretend 723 00:33:59,984 --> 00:34:01,334 that Curtis got hit by a bus. 724 00:34:01,394 --> 00:34:02,564 It was never anything nice. 725 00:34:02,564 --> 00:34:03,254 It was never 726 00:34:03,584 --> 00:34:05,774 that I won the lottery and then just flew the coop. 727 00:34:05,774 --> 00:34:10,214 It was always Curtis got hit by a bus or got swallowed up in the, in the sink. 728 00:34:10,244 --> 00:34:12,284 The great sinkhole of 2024. 729 00:34:13,004 --> 00:34:13,754 Um, 730 00:34:14,009 --> 00:34:16,709 long do you think it's gonna be until people don't even know what a bus is? 731 00:34:20,489 --> 00:34:21,569 I think, I think we're good. 732 00:34:21,629 --> 00:34:22,019 I think we're 733 00:34:22,019 --> 00:34:22,319 good. 734 00:34:22,529 --> 00:34:26,609 You know, I, um, it's funny, definitely an aside, when I was working the 735 00:34:26,609 --> 00:34:30,749 election, uh, the last two weeks we're at a school and one of the hardest 736 00:34:30,749 --> 00:34:35,609 time we're, we're, we're at a, like a multifunction building next to a school. 737 00:34:35,789 --> 00:34:40,289 And one of the hardest times was during pickup and drop off because 738 00:34:40,289 --> 00:34:42,809 there are all these parents that are picking up and dropping off their kids. 739 00:34:42,809 --> 00:34:43,499 And I was like. 740 00:34:44,099 --> 00:34:47,819 You know, if only there was like a large vehicle that we could put all the kids 741 00:34:47,819 --> 00:34:51,959 in and like we could like paint it yellow so that people could see it and then like 742 00:34:51,959 --> 00:34:55,289 have a sign that comes out and make sure, oh, traffic stops in both directions. 743 00:34:55,739 --> 00:34:57,119 If only we could do all that. 744 00:34:57,119 --> 00:35:00,509 And then all these parents wouldn't have to like, uh, take their kids 745 00:35:00,509 --> 00:35:04,409 to school and waste all of that gas and all of those cars sitting there. 746 00:35:04,589 --> 00:35:07,109 And I'm sure they're, the gas is running while they're sitting 747 00:35:07,109 --> 00:35:10,139 there for a half hour waiting for their kid to come outta school. 748 00:35:10,139 --> 00:35:11,339 Anyway, I digress. 749 00:35:11,399 --> 00:35:11,999 I don't know. 750 00:35:12,344 --> 00:35:12,424 Hmm. 751 00:35:13,739 --> 00:35:17,489 But, um, so that, I, I think that, we'll, we'll stop there because 752 00:35:17,489 --> 00:35:23,099 basically it, the number one thing, the number one success criteria that 753 00:35:23,099 --> 00:35:27,389 I'm gonna say is that you know what your success criteria are before. 754 00:35:27,629 --> 00:35:28,829 What are we gonna restore? 755 00:35:28,979 --> 00:35:30,449 How are we gonna restore it? 756 00:35:30,689 --> 00:35:36,119 Where are we going to restore it, and how long is it, you know, what, what 757 00:35:36,119 --> 00:35:37,979 timeframe are we trying to fit in? 758 00:35:38,129 --> 00:35:41,219 And also what other, what other, like the thing we talked about with the. 759 00:35:41,699 --> 00:35:45,449 With the, uh, uh, the other criteria being that, that we can 760 00:35:45,449 --> 00:35:47,549 do it without Curtis' help, right? 761 00:35:47,549 --> 00:35:50,339 Or what, whatever, you know, whatever your, your Curtis is, right? 762 00:35:51,029 --> 00:35:55,769 Um, decide on what all of those are upfront and you have a much 763 00:35:55,769 --> 00:36:00,029 better chance of being successful when you actually do the recovery. 764 00:36:00,989 --> 00:36:01,589 What do you think? 765 00:36:01,754 --> 00:36:02,774 No, that makes sense. 766 00:36:03,569 --> 00:36:03,959 Okay. 767 00:36:04,439 --> 00:36:08,879 And with that, I will thank you once again for being a great co-host persona. 768 00:36:09,254 --> 00:36:10,424 I try, I try. 769 00:36:10,424 --> 00:36:13,484 This was a fun, I I like talking about disaster recovery testing. 770 00:36:14,009 --> 00:36:15,239 Yeah, absolutely. 771 00:36:15,329 --> 00:36:18,299 And we will, uh, hope you guys enjoyed this as well. 772 00:36:18,659 --> 00:36:20,429 Uh, that is a wrap. 773 00:36:23,339 --> 00:36:28,019 The backup wrap up is written, recorded, and produced by me w Curtis Preston. 774 00:36:28,589 --> 00:36:30,059 If you need backup or Dr. 775 00:36:30,059 --> 00:36:33,689 Consulting content generation or expert witness work, 776 00:36:33,989 --> 00:36:35,969 check out backup central.com. 777 00:36:36,689 --> 00:36:39,809 You can also find links from my O'Reilly Books on the same website. 778 00:36:40,469 --> 00:36:44,459 Remember, this is an independent podcast and any opinions that 779 00:36:44,459 --> 00:36:48,359 you hear are those of the speaker and not necessarily an employer. 780 00:36:49,289 --> 00:36:49,979 Thanks for listening.