1 00:00:00,000 --> 00:00:03,780 You found the backup wrap up your go-to podcast for all things 2 00:00:03,780 --> 00:00:06,340 backup recovery and cyber recovery. 3 00:00:07,039 --> 00:00:10,980 In our final episode about DR testing, the rubber meets the road. 4 00:00:11,570 --> 00:00:16,740 Last time we talked about getting ready for your DR test, and this time we're 5 00:00:16,740 --> 00:00:18,220 talking about actually running the test. 6 00:00:18,980 --> 00:00:23,290 We'll cover what you need to do during the test, like coordinating between 7 00:00:23,300 --> 00:00:29,130 teams, documenting what goes wrong, because something always goes wrong, 8 00:00:29,950 --> 00:00:34,690 and making sure that you've got backup communication methods ready. 9 00:00:35,422 --> 00:00:39,602 By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr. 10 00:00:39,602 --> 00:00:44,602 Backup, and I've been passionate about backup and recovery for over 30 years, 11 00:00:45,352 --> 00:00:47,082 ever since I had to tell my boss. 12 00:00:47,767 --> 00:00:52,347 We had no backups of that really important production database that we had just lost. 13 00:00:53,227 --> 00:00:56,307 I don't want that to happen to you, and that's why I do this. 14 00:00:57,047 --> 00:01:02,107 On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes. 15 00:01:02,617 --> 00:01:04,467 This is the backup wrap up. 16 00:01:19,212 --> 00:01:19,987 Welcome to the show. 17 00:01:21,222 --> 00:01:23,742 Hi, I am w Curtis Preston, AKA, Mr. 18 00:01:23,742 --> 00:01:27,732 Backup, and if you could just take a couple of seconds to 19 00:01:27,732 --> 00:01:29,862 either like or subscribe or. 20 00:01:30,237 --> 00:01:34,287 Uh, follow the channel so that you can, uh, always get our great content. 21 00:01:34,287 --> 00:01:35,397 That would be awesome. 22 00:01:36,117 --> 00:01:42,747 I am once again joined by a guy who has finally put some of his car 23 00:01:42,747 --> 00:01:48,577 knowledge to use Prasanna Malaiyandi. 24 00:01:50,281 --> 00:01:54,451 I'm doing well Curtis, and yes, I am finally putting some of that car knowledge 25 00:01:54,931 --> 00:01:59,491 to use, uh, for viewers who may not, or listeners who may not be aware. 26 00:02:00,241 --> 00:02:05,101 tend to watch a lot of car YouTube stuff, um, a lot of it 27 00:02:05,101 --> 00:02:12,511 tends to be a brown fabrication engine rebuilding, drag racing. 28 00:02:12,946 --> 00:02:18,436 It's a really odd mix, but a lot of it is just YouTube knowledge. 29 00:02:18,466 --> 00:02:22,156 And so I finally decided to try something different, and I've been taking auto 30 00:02:22,156 --> 00:02:26,446 shop classes at my local community college, which has been amazing. 31 00:02:26,446 --> 00:02:31,951 And so as part of it, you actually have a hands-on lab section where you get to 32 00:02:31,951 --> 00:02:34,246 actually work on cars like your own car. 33 00:02:34,246 --> 00:02:35,986 And right now it's all basic stuff, right? 34 00:02:35,986 --> 00:02:41,506 So changes, underhood inspections, inspecting cooling systems. 35 00:02:42,376 --> 00:02:47,596 But we actually gotta do things like charging tests, uh, compression tests, 36 00:02:47,596 --> 00:02:50,296 leak down tests, replacing spark plugs. 37 00:02:50,296 --> 00:02:52,036 So excited. 38 00:02:52,126 --> 00:02:55,786 I'm actually using these hands for things. 39 00:02:55,786 --> 00:02:59,806 And you did a, you did an oil change yesterday on your wife's car, 40 00:03:00,003 --> 00:03:01,563 do an oil change on my wife's car. 41 00:03:01,593 --> 00:03:01,743 Yep. 42 00:03:01,743 --> 00:03:01,773 I. 43 00:03:02,456 --> 00:03:04,616 How filthy was the oil in your wife's car? 44 00:03:04,663 --> 00:03:06,973 Yeah, it looked almost brand new. 45 00:03:07,946 --> 00:03:11,906 Um, it didn't have many miles since the last oil change. 46 00:03:11,906 --> 00:03:17,306 I'd probably say five or 600 miles, but it was a sacrifice since I needed to 47 00:03:17,306 --> 00:03:19,016 actually do an oil change for the class. 48 00:03:19,016 --> 00:03:19,256 So. 49 00:03:20,004 --> 00:03:24,174 Right, right now, this isn't the first time you've done an oil change, right? 50 00:03:24,416 --> 00:03:24,626 no. 51 00:03:24,714 --> 00:03:25,134 Okay. 52 00:03:26,246 --> 00:03:29,066 I've done one in the past, but this is the first time I've done it on a lift, 53 00:03:29,306 --> 00:03:31,376 which oh my God, is so much nicer. 54 00:03:33,714 --> 00:03:37,044 Oh, you brought, you brought her car into the class. 55 00:03:37,136 --> 00:03:37,526 class. 56 00:03:37,526 --> 00:03:38,126 And we put it 57 00:03:38,184 --> 00:03:39,234 I see, I see. 58 00:03:39,386 --> 00:03:39,896 and I. 59 00:03:41,004 --> 00:03:41,514 Yeah. 60 00:03:41,574 --> 00:03:41,814 Yeah. 61 00:03:41,814 --> 00:03:43,194 Everything's nicer on a lift. 62 00:03:43,194 --> 00:03:43,794 Absolutely. 63 00:03:43,886 --> 00:03:48,746 When you're not like struggling underneath the car, trying not to drop the hot oil 64 00:03:48,746 --> 00:03:56,396 on you, and you're actually able to get a large, like container underneath, like the 65 00:03:56,396 --> 00:03:59,996 drum was probably like three feet wide. 66 00:04:00,489 --> 00:04:01,449 Right, right. 67 00:04:01,929 --> 00:04:02,079 Yeah. 68 00:04:02,079 --> 00:04:04,059 You had one of those that you can wheel around, right? 69 00:04:04,106 --> 00:04:04,436 Yep. 70 00:04:05,906 --> 00:04:07,616 So significantly easier. 71 00:04:07,616 --> 00:04:10,496 And then I was just thinking, I was like, do I have room in 72 00:04:10,496 --> 00:04:12,626 my garage for a two post lift? 73 00:04:12,626 --> 00:04:14,126 Even a short one, but no. 74 00:04:15,039 --> 00:04:17,919 Trust me, I have thought about it back when I was doing a 75 00:04:17,919 --> 00:04:19,479 lot more work on my cars. 76 00:04:20,049 --> 00:04:23,109 I definitely looked into it and I was like, okay, I don't, I 77 00:04:23,109 --> 00:04:24,429 can't spend that kind of money. 78 00:04:24,501 --> 00:04:27,741 So let's talk about something that we actually know a little bit of something 79 00:04:27,741 --> 00:04:32,901 about, uh, so last, so two weeks ago, for those of you that follow the, 80 00:04:32,961 --> 00:04:38,211 uh, episode, or for those of you that follow the show, uh, two weeks ago 81 00:04:38,211 --> 00:04:40,821 we did DR testing part one, and then. 82 00:04:41,196 --> 00:04:45,606 Um, we aired, uh, a great speaking of Dr. 83 00:04:45,606 --> 00:04:53,406 Testing a great episode from 2021, which was the best DR testing story ever, right? 84 00:04:53,603 --> 00:04:54,083 Yep. 85 00:04:54,143 --> 00:04:54,833 Oh yeah. 86 00:04:54,893 --> 00:04:56,903 The scariest, I would say. 87 00:04:57,576 --> 00:04:58,236 Yeah. 88 00:04:58,236 --> 00:05:04,866 Where a guy for reasons that he goes into in the show, he essentially purposefully 89 00:05:04,866 --> 00:05:10,536 destroys his production environment, not just for DR testing, but as a. 90 00:05:10,851 --> 00:05:15,111 As a matter of how everything happened, he ends up testing his DR system 91 00:05:15,561 --> 00:05:18,621 and it, it does work, but oh my God. 92 00:05:22,261 --> 00:05:26,271 There was, there was a, there was a quote in there that said something like, he had 93 00:05:26,271 --> 00:05:28,761 a long weekend that lasted like five days. 94 00:05:30,713 --> 00:05:31,003 Yeah. 95 00:05:32,721 --> 00:05:33,291 Yeah. 96 00:05:34,251 --> 00:05:34,671 So, yeah. 97 00:05:34,718 --> 00:05:39,428 things it's like, and well, and the other challenge is he was up in Alaska, 98 00:05:40,101 --> 00:05:40,611 Yeah, 99 00:05:40,748 --> 00:05:43,538 If he needed to get parts or other things like 100 00:05:43,701 --> 00:05:44,031 right. 101 00:05:44,228 --> 00:05:44,588 luck. 102 00:05:44,961 --> 00:05:45,771 Yeah, exactly 103 00:05:46,058 --> 00:05:48,428 if I'm about to do like a house repair or something else like that, 104 00:05:48,428 --> 00:05:52,208 it's like, you know not to do it on a Saturday or a Sunday or a Friday, 105 00:05:53,751 --> 00:05:54,381 right. 106 00:05:54,938 --> 00:05:57,278 if you have to call someone or you need to pick up something 107 00:05:57,278 --> 00:05:58,148 and you don't do it at night. 108 00:05:59,181 --> 00:05:59,961 Yeah, definitely. 109 00:05:59,961 --> 00:06:01,161 Definitely don't do it at night. 110 00:06:01,161 --> 00:06:01,611 Right. 111 00:06:01,613 --> 00:06:01,903 Yeah. 112 00:06:02,736 --> 00:06:06,366 Um, yeah, so that's a great episode if you didn't listen to that episode. 113 00:06:06,366 --> 00:06:07,836 That is a great episode. 114 00:06:07,926 --> 00:06:12,276 Um, and, uh, uh, yeah, listen to that. 115 00:06:12,456 --> 00:06:14,016 So this one, the, the. 116 00:06:14,571 --> 00:06:18,891 Two weeks ago, we talked essentially about getting ready to do the DR test, 117 00:06:18,891 --> 00:06:23,811 preparing for it, setting the scope for it, agreeing on what's going to be a 118 00:06:23,811 --> 00:06:28,041 success, and then this week we're gonna talk about actually executing the DR test. 119 00:06:28,041 --> 00:06:30,291 And again, this is a DR test. 120 00:06:30,921 --> 00:06:35,101 What would you say is the purpose of a DR test Prasanna? 121 00:06:35,121 --> 00:06:35,151 I. 122 00:06:35,408 --> 00:06:39,758 To make sure that you're actually in the case of an actual disaster, 123 00:06:40,088 --> 00:06:44,443 you're able to recover as agreed upon whatever your agreement was. 124 00:06:45,801 --> 00:06:48,441 Yeah, I, I think that's sort of the general, yeah. 125 00:06:48,531 --> 00:06:51,081 Obviously that's the purpose of a test in general, right. 126 00:06:51,081 --> 00:06:52,641 Is to, is to, is to. 127 00:06:53,061 --> 00:06:55,461 To test whether or not you could do it when you, when you need it. 128 00:06:56,151 --> 00:07:02,571 But since most tests fail, I'm going to say that the other purpose and 129 00:07:02,571 --> 00:07:10,941 perhaps the bigger purpose is to fix the parts of your, of the TR system that 130 00:07:10,941 --> 00:07:13,071 you discover are broken in some way. 131 00:07:13,161 --> 00:07:13,611 Right? 132 00:07:14,061 --> 00:07:19,311 Um, and, uh, so the, the probably one of the biggest. 133 00:07:20,331 --> 00:07:25,071 Outcomes of a DR test is to feed back into the DR plan, right? 134 00:07:25,311 --> 00:07:25,581 Yeah. 135 00:07:25,928 --> 00:07:29,018 just in terms of what fails, I know sometimes people are like, 136 00:07:29,048 --> 00:07:32,318 oh, it's just thinking about like, I can't restore the data. 137 00:07:32,858 --> 00:07:39,128 But a lot of times what really fails is the dependencies that you didn't consider. 138 00:07:40,371 --> 00:07:40,671 Right. 139 00:07:40,778 --> 00:07:44,438 you make sure you're able to fail over and recover your active 140 00:07:44,438 --> 00:07:49,298 directory in your DR site before you can bring your applications online? 141 00:07:50,196 --> 00:07:54,020 You, you know, um, I'm glad you brought that up because I aired 142 00:07:54,039 --> 00:07:59,513 another classic episode about the actual disaster recovery on an island. 143 00:07:59,633 --> 00:08:01,973 And, uh, again, well, it's with the islands, right? 144 00:08:01,973 --> 00:08:04,853 Because Alaska was Kodiak Island. 145 00:08:05,453 --> 00:08:08,033 Um, but this was in a Caribbean island. 146 00:08:08,378 --> 00:08:12,668 And they do an actual deal, you know, an actual recovery because there 147 00:08:12,668 --> 00:08:15,488 was a hurricane that took it out. 148 00:08:15,998 --> 00:08:21,578 And one of those dependencies that you talked about was the lack of internet 149 00:08:22,120 --> 00:08:22,180 Yeah. 150 00:08:22,478 --> 00:08:25,418 and, uh, lack, the lack of power, the lack of internet. 151 00:08:25,418 --> 00:08:31,148 These are all things that we come to expect on a normal everyday basis, which 152 00:08:32,110 --> 00:08:32,290 In 153 00:08:32,348 --> 00:08:34,508 an actual disaster is, is not, 154 00:08:34,630 --> 00:08:34,990 Yep. 155 00:08:35,018 --> 00:08:35,948 not that right. 156 00:08:36,130 --> 00:08:36,370 Yeah. 157 00:08:36,820 --> 00:08:38,560 And we also had that other episode. 158 00:08:38,560 --> 00:08:40,875 Do you remember maybe, I don't know if you want to air that or not. 159 00:08:41,255 --> 00:08:42,195 The dire show one. 160 00:08:43,748 --> 00:08:45,758 That's right, the one that talked about the derecho. 161 00:08:45,758 --> 00:08:47,768 I'm gonna have to, I have to go find that one. 162 00:08:47,768 --> 00:08:50,198 'cause it's not titled the Derecho episode. 163 00:08:50,588 --> 00:08:52,148 It was, um. 164 00:08:52,478 --> 00:08:55,448 I'll have to find, if I can find that, I'll rebroadcast it in the keeping 165 00:08:55,448 --> 00:08:57,728 of the, the disaster recovery theme. 166 00:08:57,728 --> 00:08:59,588 I'll, I'll definitely see if I can find that when I br 167 00:08:59,978 --> 00:09:01,148 'cause that was also very good. 168 00:09:01,298 --> 00:09:02,978 I didn't even know what a derecho was. 169 00:09:03,488 --> 00:09:05,648 Derecho is a land hurricane. 170 00:09:06,068 --> 00:09:07,838 Uh, a hurricane that forms over land. 171 00:09:08,228 --> 00:09:12,098 I don't know why it's called derecho, but that is what it is. 172 00:09:12,128 --> 00:09:12,548 Right. 173 00:09:12,550 --> 00:09:12,770 Yep. 174 00:09:13,748 --> 00:09:18,218 Uh, to me that just means Right, you know, to the right in Spanish. 175 00:09:18,218 --> 00:09:18,578 But I. 176 00:09:21,008 --> 00:09:22,058 You know, it is what it is. 177 00:09:22,148 --> 00:09:28,958 Um, so, uh, so, so we talk about if we're executing the DR test. 178 00:09:29,633 --> 00:09:32,813 Uh, we, we, you know, we, we, we've, we've agreed on what we're gonna test. 179 00:09:32,813 --> 00:09:35,753 We've agreed on what the success criteria is. 180 00:09:36,113 --> 00:09:40,043 It's time to actually start walking through the, the test 181 00:09:40,493 --> 00:09:41,843 we're, we're going to have. 182 00:09:41,843 --> 00:09:45,953 And, and also we created a, an environment that we're going to test in. 183 00:09:46,193 --> 00:09:49,403 We're not doing what our friend from Alaska did. 184 00:09:49,520 --> 00:09:52,660 I, I was just thinking, are you just gonna go around like the TV shows 185 00:09:52,660 --> 00:09:54,670 when they get hit with an attack and they're just like, plug gun, 186 00:09:54,670 --> 00:09:56,080 plug the cables up, plug the cables. 187 00:09:56,543 --> 00:09:58,583 Yeah, don't do that. 188 00:09:58,643 --> 00:10:00,893 We, we have some sort of test of environment. 189 00:10:01,253 --> 00:10:03,923 Generally speaking, today's, it's generally gonna be the 190 00:10:03,923 --> 00:10:09,203 cloud and we're going to start executing the, the, um, this test. 191 00:10:09,593 --> 00:10:14,393 Can you think of, uh, and, and one of the things, again, this is more of 192 00:10:14,393 --> 00:10:18,023 set up a thing, but one of the things you wanna make sure is to allocate 193 00:10:18,023 --> 00:10:21,233 enough time, uh, for this, you know. 194 00:10:21,923 --> 00:10:27,563 For this process to unfold in its natural, um, evolution. 195 00:10:27,820 --> 00:10:29,140 I would say time. 196 00:10:29,530 --> 00:10:32,950 And then also make sure you have the resources right. 197 00:10:32,950 --> 00:10:36,115 And I'm not, I don't mean compute resources, but people because. 198 00:10:36,473 --> 00:10:37,133 Right? 199 00:10:37,253 --> 00:10:39,293 Make sure that people are available, right? 200 00:10:39,535 --> 00:10:39,745 yeah. 201 00:10:39,863 --> 00:10:39,923 Um. 202 00:10:39,985 --> 00:10:43,165 don't do this at like, uh, quarter end because people may 203 00:10:43,165 --> 00:10:45,325 be firefighting other things or. 204 00:10:45,788 --> 00:10:46,328 Yeah. 205 00:10:47,018 --> 00:10:47,588 Yeah. 206 00:10:47,798 --> 00:10:51,668 The company that I, the, the bank, we did it on a weekend. 207 00:10:51,818 --> 00:10:56,468 Um, but it was a dedicated, you know, a, a dedicated weekend where we're 208 00:10:56,468 --> 00:11:01,148 going to do the DR test, and we did that because again, you're, you're 209 00:11:01,148 --> 00:11:05,198 making all these resources available for the DR test, which means they're not 210 00:11:05,198 --> 00:11:08,258 available to do their day job, and their day job would happen during the week. 211 00:11:08,258 --> 00:11:11,318 So we chose to do it on the weekend and. 212 00:11:11,683 --> 00:11:16,453 I'd say the bigger, the bigger you're going, the bigger te the bigger, 213 00:11:18,493 --> 00:11:19,663 this isn't coming out in English. 214 00:11:20,323 --> 00:11:25,573 Uh, the bigger the test, the bigger the need to prepare and to, to have, um, 215 00:11:25,603 --> 00:11:29,623 you know, to make sure you have those resources and to not do it when the 216 00:11:29,623 --> 00:11:31,123 normal production stuff is going on. 217 00:11:31,270 --> 00:11:35,920 requires buy-in from the business communication, right? 218 00:11:35,950 --> 00:11:36,430 All these 219 00:11:36,553 --> 00:11:36,823 Yeah. 220 00:11:37,510 --> 00:11:37,780 right? 221 00:11:37,843 --> 00:11:38,173 Yeah. 222 00:11:38,173 --> 00:11:39,043 Make sure you. 223 00:11:40,088 --> 00:11:43,208 Make sure you communicate to all the powers that be, that you are doing 224 00:11:43,208 --> 00:11:47,018 a DR test, especially if you're gonna do any kind of failover. 225 00:11:47,468 --> 00:11:48,188 Um, 226 00:11:48,220 --> 00:11:48,940 it too, right? 227 00:11:48,940 --> 00:11:52,390 Because you want this to be done on an ongoing basis. 228 00:11:52,568 --> 00:11:53,078 right, 229 00:11:53,140 --> 00:11:57,790 to convince 'em upfront, Hey, here's why it's valuable, such that when you go back 230 00:11:57,790 --> 00:12:01,270 and after the results, right, you're like, Hey, we now need to do another DR test. 231 00:12:01,270 --> 00:12:02,740 Maybe six months down the line, 232 00:12:02,948 --> 00:12:03,248 right. 233 00:12:03,280 --> 00:12:03,730 already bought it. 234 00:12:05,378 --> 00:12:09,008 Another thing as, as we're going through the DR test, we're documenting 235 00:12:09,008 --> 00:12:12,603 what went right, what went wrong, especially what went wrong. 236 00:12:12,933 --> 00:12:13,283 Right. 237 00:12:14,273 --> 00:12:15,693 Um, go ahead. 238 00:12:16,225 --> 00:12:16,825 so 239 00:12:19,195 --> 00:12:23,305 this is an interesting thing 'cause when we had Mike podcast, right, and 240 00:12:23,305 --> 00:12:27,055 he was talking about sort of doing these tabletop exercises, right? 241 00:12:27,055 --> 00:12:30,385 I think it's important the person documenting kind of needs to 242 00:12:30,385 --> 00:12:32,335 take an objective perspective. 243 00:12:32,553 --> 00:12:32,973 Mm-hmm. 244 00:12:34,210 --> 00:12:39,640 Right, because you may be showing some biases or the person documenting 245 00:12:39,640 --> 00:12:44,530 may not want to document certain things, or may just sort of dismiss 246 00:12:44,530 --> 00:12:46,450 it as, Hey, this isn't important, 247 00:12:47,078 --> 00:12:47,738 Right, 248 00:12:47,980 --> 00:12:53,225 Versus actually capturing what happened throughout the process. 249 00:12:54,518 --> 00:12:54,608 right. 250 00:12:55,433 --> 00:12:56,003 Agreed. 251 00:12:56,333 --> 00:13:01,163 Um, the next thing is, and, and we covered this, uh, in the previous 252 00:13:01,163 --> 00:13:06,443 episode, but once you've, you know, we talked about testing little parts 253 00:13:06,443 --> 00:13:12,083 of the infrastructure, but once we grow, once we've tested this piece 254 00:13:12,083 --> 00:13:13,758 and this piece and this piece, I. 255 00:13:13,868 --> 00:13:19,088 I do think it's important to test, you know, you look at the scenario, what 256 00:13:19,088 --> 00:13:21,068 would this scenario do to our company? 257 00:13:21,548 --> 00:13:21,788 Right? 258 00:13:21,788 --> 00:13:23,468 The scenario is a disaster. 259 00:13:23,468 --> 00:13:25,063 The scenario is a fire, a. 260 00:13:25,583 --> 00:13:30,053 A terrorist action, um, and it's gonna take out all of this infrastructure. 261 00:13:30,053 --> 00:13:31,703 What would that do to us? 262 00:13:31,883 --> 00:13:40,523 So for example, you might not need to test your ability to recover from a SaaS outage 263 00:13:40,793 --> 00:13:44,723 when your, if you have a data center and your data center goes out, right? 264 00:13:44,933 --> 00:13:48,143 It's a, it's gonna be scenario dependent. 265 00:13:48,588 --> 00:13:56,658 What you're gonna test, but, um, you, you might wanna, what would be the impact to 266 00:13:56,658 --> 00:14:00,138 our business and our ability to use the different parts of our infrastructure? 267 00:14:00,198 --> 00:14:05,963 And so speaking of dependencies, if we don't have internet of any kind, it 268 00:14:05,963 --> 00:14:08,928 is, it is kind of a SaaS outage, right? 269 00:14:09,818 --> 00:14:10,268 Right. 270 00:14:10,688 --> 00:14:15,428 Um, so, uh, we're gonna, we want to test as many of those parts 271 00:14:15,428 --> 00:14:18,458 of our infrastructure that are going to be impacted by the 272 00:14:18,458 --> 00:14:20,558 scenario that we're testing, right. 273 00:14:20,710 --> 00:14:23,235 Yeah, and sometimes it's a little bit about. 274 00:14:24,520 --> 00:14:27,970 consequences or identifying gaps. 275 00:14:28,180 --> 00:14:29,650 It's like when you're writing code, right? 276 00:14:29,650 --> 00:14:30,700 You normally do unit 277 00:14:30,903 --> 00:14:31,123 Um. 278 00:14:31,360 --> 00:14:33,550 but then when you actually test the end-to-end functionality, you're 279 00:14:33,550 --> 00:14:35,920 like, oh, I didn't realize that this interacts with this other thing 280 00:14:35,920 --> 00:14:37,270 this way, and things don't work. 281 00:14:37,270 --> 00:14:40,480 That's why we also do end-to-end testing in addition to unit tests. 282 00:14:41,183 --> 00:14:41,543 Yeah. 283 00:14:41,543 --> 00:14:45,323 And, and, and again, this is why I went, why back in the beginning I 284 00:14:45,323 --> 00:14:50,033 was saying that the purpose of the DR test is to identify these gaps, right? 285 00:14:50,303 --> 00:14:51,263 The, yeah. 286 00:14:51,263 --> 00:14:53,093 I mean we can have, um, I. 287 00:14:54,248 --> 00:14:57,638 You know, we can have that perfect test that goes well and that's great and 288 00:14:57,638 --> 00:15:02,408 everybody feels better, but it's just as valuable to find the DR test that 289 00:15:02,408 --> 00:15:09,219 had, that had a big hole or a small hole and, um, you know, the, uh, and, and, 290 00:15:09,219 --> 00:15:11,018 and to document that and address that. 291 00:15:11,258 --> 00:15:13,418 And this is why we do it on a regular basis. 292 00:15:13,525 --> 00:15:15,415 I have a question for you, Curtis. 293 00:15:15,578 --> 00:15:16,178 Yeah. 294 00:15:16,825 --> 00:15:18,145 Do you think DR. 295 00:15:18,145 --> 00:15:18,895 Testing? 296 00:15:18,985 --> 00:15:22,285 So most organizations have a risk management team, 297 00:15:22,688 --> 00:15:22,958 Mm-hmm. 298 00:15:23,695 --> 00:15:24,025 right? 299 00:15:24,055 --> 00:15:27,775 Which usually has a lot of this information in terms of, okay, 300 00:15:27,775 --> 00:15:30,415 what are the business risks and everything else like that. 301 00:15:30,745 --> 00:15:34,375 But they're also probably the ones who are coordinating across the business 302 00:15:34,375 --> 00:15:36,895 in order to say, okay, let's do a test. 303 00:15:37,098 --> 00:15:37,518 Mm-hmm. 304 00:15:37,660 --> 00:15:41,860 Right where the infrastructure, DR testing that we're talking about here 305 00:15:41,860 --> 00:15:44,200 is probably one portion of that overall 306 00:15:44,448 --> 00:15:44,868 Mm-hmm. 307 00:15:44,948 --> 00:15:45,308 Mm-hmm. 308 00:15:45,310 --> 00:15:46,185 Do you think that's fair? 309 00:15:47,333 --> 00:15:48,383 Yeah, I think that's fair. 310 00:15:48,383 --> 00:15:51,743 And you know, this is, we're going to. 311 00:15:53,033 --> 00:15:59,333 I think that if we're doing a, a real DR test, we're going to this. 312 00:15:59,333 --> 00:16:03,893 This is a business test as much as it is a technology test, right? 313 00:16:04,000 --> 00:16:04,060 Yeah. 314 00:16:04,253 --> 00:16:08,453 There is this, that overlap between business continuity planning 315 00:16:08,453 --> 00:16:09,953 and disaster recovery planning. 316 00:16:10,193 --> 00:16:14,873 And maybe for a DR test, we're not concerned so much with, um. 317 00:16:15,593 --> 00:16:20,333 Uh, like if, if it's just a DR test, we're not concerned with, let's say, 318 00:16:20,518 --> 00:16:26,123 uh, uh, buildings and people, places for people to, to work and things like that. 319 00:16:26,123 --> 00:16:29,183 We're concerned more with getting the technology back up and running. 320 00:16:29,513 --> 00:16:31,253 But I, I'm glad you brought that up. 321 00:16:31,253 --> 00:16:35,813 That is a, a separate aspect that does need to be taken into account. 322 00:16:36,055 --> 00:16:40,645 Well, and the benefit with this is if there's already a team that is looking 323 00:16:40,645 --> 00:16:42,955 at that business continuity aspect, 324 00:16:43,193 --> 00:16:43,613 Mm-hmm. 325 00:16:43,885 --> 00:16:47,515 You may not need to convince the business as much, right? 326 00:16:47,515 --> 00:16:48,085 In order to be 327 00:16:48,113 --> 00:16:48,443 Right. 328 00:16:48,565 --> 00:16:51,865 right, you should partner with people who already, like that is their job, 329 00:16:52,373 --> 00:16:52,883 Agreed. 330 00:16:52,943 --> 00:16:53,423 Agreed. 331 00:16:53,575 --> 00:16:53,725 them. 332 00:16:54,383 --> 00:16:54,893 Agreed. 333 00:16:56,083 --> 00:16:58,963 We talked about documenting things that we discover here. 334 00:16:59,233 --> 00:17:02,983 I, I think that we should be maintaining like a log of, you 335 00:17:02,983 --> 00:17:05,593 know, all of the tests and the things that we've learned from them. 336 00:17:05,593 --> 00:17:09,433 Because again, that may be helpful for, uh, you know, for 337 00:17:09,433 --> 00:17:11,233 future generations of tests. 338 00:17:11,573 --> 00:17:13,043 You know, It's important to have a Dr. 339 00:17:13,043 --> 00:17:16,283 Runbook and to, to, to have this, you know, one of the pur 340 00:17:16,343 --> 00:17:18,743 the, one of the purposes of the test is to update that runbook. 341 00:17:18,953 --> 00:17:20,243 So let's just talk about that. 342 00:17:20,633 --> 00:17:23,843 Um, the, the, the thing about having a Dr. 343 00:17:23,843 --> 00:17:27,113 Runbook, I do believe in having an electronic copy of the Dr. 344 00:17:27,113 --> 00:17:32,723 Runbook, uh, but uh, also have the ability to easily update. 345 00:17:33,113 --> 00:17:34,943 A paper copy of that runbook. 346 00:17:34,973 --> 00:17:39,173 So the way to do that is to have some sort of documentation system 347 00:17:39,173 --> 00:17:41,123 online that you can easily update. 348 00:17:41,633 --> 00:17:47,613 Um, and then if you want to have a paper copy and you want to have a paper copy, 349 00:17:48,363 --> 00:17:54,813 then um, the best way to have that is a, is a loose leaf type notebook system right 350 00:17:54,843 --> 00:17:58,383 where you can update pages of it, where you don't have to update the entire book. 351 00:17:58,745 --> 00:18:01,235 I have a comment about the electronic copy. 352 00:18:01,578 --> 00:18:02,208 Sure. 353 00:18:02,315 --> 00:18:07,535 I would recommend also keeping a copy out of your normal corporate infrastructure. 354 00:18:08,148 --> 00:18:08,868 Agreed. 355 00:18:08,958 --> 00:18:09,803 Right, right. 356 00:18:09,860 --> 00:18:13,700 in case, say you get hit with ransomware and you no longer have access to that 357 00:18:13,700 --> 00:18:18,380 infrastructure, or someone deletes your account that hosted that data, right? 358 00:18:18,380 --> 00:18:21,650 So make sure it's something completely disconnected as well. 359 00:18:22,400 --> 00:18:24,230 A copy just in case. 360 00:18:24,230 --> 00:18:29,420 And I go back to think about the Pixar story, right, where they just happen to 361 00:18:29,420 --> 00:18:34,880 be lucky with Toy Story two and have a copy offsite offline to save the movie. 362 00:18:35,973 --> 00:18:36,603 Exactly. 363 00:18:36,843 --> 00:18:41,163 Um, yeah, I, I, I think obviously we, we have to keep security in mind. 364 00:18:41,163 --> 00:18:44,343 We have to make sure that what, wherever that system, wherever that other 365 00:18:44,343 --> 00:18:46,383 copy is, it's protected by security. 366 00:18:46,863 --> 00:18:49,593 But the whole point of it is to have it outside the normal security. 367 00:18:49,773 --> 00:18:54,153 So, uh, there, there's a, there's a, um, a balance that you need to have there. 368 00:18:54,213 --> 00:18:54,573 Right. 369 00:18:55,653 --> 00:18:58,863 Um, what about communications during the DR tests? 370 00:18:58,923 --> 00:19:02,133 Um, we need to keep everyone. 371 00:19:03,048 --> 00:19:04,908 Abreast of what's going on. 372 00:19:04,908 --> 00:19:06,073 You wanna talk about that a little bit? 373 00:19:06,260 --> 00:19:12,830 Yeah, so you wanna make sure people aren't working in silos and because during a 374 00:19:12,830 --> 00:19:14,330 DR test things are gonna be chaotic. 375 00:19:15,335 --> 00:19:18,785 but since this is more of a controlled environment, you want to establish 376 00:19:18,785 --> 00:19:22,805 those patterns and say, this is a normal way that we communicate. 377 00:19:23,105 --> 00:19:26,615 It might be via phones, it might be emails. 378 00:19:26,615 --> 00:19:30,275 You might jump into a video conference, right? 379 00:19:30,275 --> 00:19:34,805 Whatever it is that you use, make sure that you have all the right 380 00:19:34,805 --> 00:19:37,115 stakeholders in that session. 381 00:19:37,588 --> 00:19:37,938 Right, 382 00:19:38,225 --> 00:19:38,975 in order. 383 00:19:38,975 --> 00:19:40,985 So, so then everyone knows what's going on. 384 00:19:41,375 --> 00:19:44,885 The other thing though, uh, to mention is make sure you also have 385 00:19:44,885 --> 00:19:48,905 alternate methods, Just like what we talked about, the runbook itself. 386 00:19:48,965 --> 00:19:49,235 Make 387 00:19:49,503 --> 00:19:49,713 right. 388 00:19:49,715 --> 00:19:56,255 case your voiceover IP phones are down in your corporate environment 389 00:19:56,255 --> 00:20:00,815 or your chat slack is down, or whatever else you're using, 390 00:20:01,113 --> 00:20:01,593 Right. 391 00:20:01,715 --> 00:20:04,745 Make sure you have an alternate mechanism to get in touch with people. 392 00:20:05,913 --> 00:20:06,303 Yeah. 393 00:20:06,303 --> 00:20:07,743 That's a real challenge. 394 00:20:07,803 --> 00:20:09,243 Um, I mean, it, it is 395 00:20:09,395 --> 00:20:09,755 Smoke 396 00:20:10,143 --> 00:20:11,643 to have communication during it. 397 00:20:11,643 --> 00:20:12,303 What'd you say? 398 00:20:12,560 --> 00:20:12,840 signals. 399 00:20:13,413 --> 00:20:18,813 So signals, it's, it's a real challenge because we depend so much on technology 400 00:20:19,143 --> 00:20:22,983 and I would say that that, um. 401 00:20:23,523 --> 00:20:27,873 Again, if it's an outage, generally the outage is for you 402 00:20:27,873 --> 00:20:29,043 and not for everything else. 403 00:20:29,043 --> 00:20:35,493 So for example, if you're relying on Zoom, um, as your mechanism, zoom will 404 00:20:35,493 --> 00:20:38,163 probably be up when you have your outage. 405 00:20:38,523 --> 00:20:41,313 You just have to need to make sure that everybody can get to zoom. 406 00:20:41,733 --> 00:20:46,083 So, um, if for example, your your, your challenge there 407 00:20:46,353 --> 00:20:49,173 will be if you are using, um. 408 00:20:49,713 --> 00:20:54,633 You know, a, a, a third party authentication mechanism to get into Zoom 409 00:20:55,083 --> 00:20:58,473 and then you don't have access to that, that could be, that could be a problem. 410 00:20:58,473 --> 00:21:00,963 So these are the things you wanna make sure, you wanna be able to make sure that 411 00:21:00,963 --> 00:21:03,663 you can communicate during the outage. 412 00:21:03,993 --> 00:21:08,193 Um, and I can definitely think of a, you know, of a multi-headed zoom call where 413 00:21:08,433 --> 00:21:11,343 everybody's just sort of keeping everybody abreast of what's going on, right. 414 00:21:12,428 --> 00:21:15,878 Um, and we wanna make sure that the stakeholders are aware of everything 415 00:21:15,878 --> 00:21:18,818 that's going on, as well as the people that are executing the, um, 416 00:21:18,848 --> 00:21:20,468 that are ex executing the test. 417 00:21:21,638 --> 00:21:26,678 Um, and then what about, um, I, I, I think, by the way, the Zoom call, 418 00:21:26,678 --> 00:21:30,458 I think is the best way to have, or something like a Zoom call to have 419 00:21:30,488 --> 00:21:34,148 coordination between the teams if there are multiple teams that are happening. 420 00:21:34,238 --> 00:21:36,248 You don't necessarily have to have everybody who's 421 00:21:36,248 --> 00:21:37,688 doing something with the Dr. 422 00:21:37,868 --> 00:21:38,348 Uh. 423 00:21:38,743 --> 00:21:41,803 To, to be on the Zoom call, but the purpose of the Zoom call, I 424 00:21:41,803 --> 00:21:45,673 think is probably to keep, keep all of the different teams aware 425 00:21:45,673 --> 00:21:47,113 of what the other teams are doing. 426 00:21:47,173 --> 00:21:47,623 Right? 427 00:21:47,930 --> 00:21:49,460 It's almost like a war room, if you will. 428 00:21:49,460 --> 00:21:49,910 Right? 429 00:21:50,143 --> 00:21:50,683 exactly. 430 00:21:50,683 --> 00:21:53,293 The big, again, the bigger the test, the bigger it is, the bigger 431 00:21:53,293 --> 00:21:58,033 the need is to have, uh, some type of communication like this. 432 00:21:58,033 --> 00:21:58,333 Right? 433 00:21:58,723 --> 00:22:01,063 Uh, and then you've also got escalation procedures. 434 00:22:01,213 --> 00:22:02,983 What happens if something doesn't go right? 435 00:22:03,193 --> 00:22:04,213 Who do we call? 436 00:22:04,393 --> 00:22:05,863 Um, yeah. 437 00:22:06,065 --> 00:22:09,845 you could throw a monkey wrench in things and be like, someone's about to 438 00:22:09,875 --> 00:22:13,085 do a, normally is part of the DR test. 439 00:22:13,085 --> 00:22:13,325 Right? 440 00:22:13,325 --> 00:22:14,975 Or would be responsible for something. 441 00:22:15,395 --> 00:22:18,605 You could be like, that person is home sick with the flu 442 00:22:18,605 --> 00:22:20,585 and cannot be in the office. 443 00:22:20,645 --> 00:22:21,365 Now what do you do? 444 00:22:22,128 --> 00:22:22,848 Yeah. 445 00:22:22,908 --> 00:22:23,778 Um, yeah. 446 00:22:23,783 --> 00:22:24,588 If your DR. 447 00:22:24,588 --> 00:22:27,018 Test says, you know, call Steve. 448 00:22:27,798 --> 00:22:33,198 Um, this, this is the, the, you know, the more you have something like that, 449 00:22:33,198 --> 00:22:36,228 the bigger that, that, that kind of thing is gonna be a problem, right? 450 00:22:36,425 --> 00:22:37,145 you bring this up. 451 00:22:37,145 --> 00:22:38,765 So I was reading the register this morning 452 00:22:39,043 --> 00:22:39,463 Mm-hmm. 453 00:22:39,845 --> 00:22:43,685 there was a call in or a write in from a, a reader, 454 00:22:43,963 --> 00:22:44,383 Mm-hmm. 455 00:22:44,675 --> 00:22:47,135 and they were saying that they had worked at a company I. 456 00:22:47,570 --> 00:22:51,290 In it and managed a bunch of infrastructure and they had built 457 00:22:51,290 --> 00:22:55,400 this system to automate all of their, uh, software deployment stuff. 458 00:22:55,533 --> 00:22:55,953 Mm-hmm. 459 00:22:56,330 --> 00:22:59,510 Um, and then they had quit the company, but no one knew how to 460 00:22:59,510 --> 00:23:05,030 operate it, and he had left his number, it's in the closet, was a machine. 461 00:23:05,330 --> 00:23:06,260 He had left his number. 462 00:23:06,260 --> 00:23:09,560 It said, do not reboot, call Steve or whatever his name was. 463 00:23:11,385 --> 00:23:14,120 And he got the call and this was like 20 years later, 464 00:23:14,853 --> 00:23:15,693 Wow. 465 00:23:16,340 --> 00:23:18,800 he got a call and he was like, I don't remember the password. 466 00:23:18,800 --> 00:23:19,250 I'm sorry. 467 00:23:19,250 --> 00:23:20,420 You gotta figure it out on your own. 468 00:23:21,498 --> 00:23:22,278 Wow. 469 00:23:22,338 --> 00:23:23,208 That's crazy. 470 00:23:24,168 --> 00:23:25,098 That's just crazy. 471 00:23:25,188 --> 00:23:26,853 So call Steve. 472 00:23:28,998 --> 00:23:29,868 That's funny. 473 00:23:29,958 --> 00:23:32,748 Um, yeah, don't, don't be like that. 474 00:23:33,138 --> 00:23:39,378 Um, so, uh, let's just say we get to the end of the test, right? 475 00:23:39,478 --> 00:23:42,868 We've successfully recovered all of the, all of the aspects 476 00:23:43,138 --> 00:23:45,268 if we're doing a full DR test. 477 00:23:46,003 --> 00:23:50,923 What needs to happen is a full sort of end-to-end functional test of the 478 00:23:50,923 --> 00:23:56,263 different parts of the business to make sure that not just that the, that a 479 00:23:56,263 --> 00:24:01,183 system was recovered or a database was recovered, but the application and the. 480 00:24:01,478 --> 00:24:05,318 The, the system around that application is able to function. 481 00:24:05,558 --> 00:24:09,038 And again, this is why we go into things like phone systems, right? 482 00:24:09,060 --> 00:24:09,360 Yeah. 483 00:24:09,398 --> 00:24:13,448 Um, you know, if, if the, the application that we're recovering is our customer 484 00:24:13,898 --> 00:24:21,968 call center, um, but we don't have phones, uh, great, uh, you know, all of that 485 00:24:21,968 --> 00:24:23,498 stuff, all of that stuff has to work. 486 00:24:23,498 --> 00:24:26,498 And you've got to do the functional end-to-end test to make sure that all 487 00:24:26,498 --> 00:24:28,718 the parts that you are pretending. 488 00:24:29,003 --> 00:24:32,483 Are, you know, were damaged, are now fully functional. 489 00:24:33,405 --> 00:24:36,945 I agree to that, but I think it's also one of the things, you have to 490 00:24:36,945 --> 00:24:38,595 be careful not to boil the ocean. 491 00:24:40,103 --> 00:24:40,853 Yes. 492 00:24:41,093 --> 00:24:42,743 Yeah, yeah. 493 00:24:42,773 --> 00:24:44,543 Well, again, this is about, 494 00:24:44,715 --> 00:24:44,985 Yeah. 495 00:24:45,053 --> 00:24:45,683 what's that? 496 00:24:45,825 --> 00:24:46,455 a balance. 497 00:24:46,630 --> 00:24:46,750 I. 498 00:24:47,183 --> 00:24:50,783 Well, what I'm saying is, whatever it is, this is, I, I think what you're 499 00:24:51,173 --> 00:24:53,483 talking about is, is more about scope, 500 00:24:53,805 --> 00:24:54,225 Yes. 501 00:24:54,863 --> 00:24:55,733 Because. 502 00:24:56,308 --> 00:25:01,018 Even if we just agreed to test this one part of the application, you 503 00:25:01,018 --> 00:25:04,588 need to do a functional test of whatever it is that you recovered. 504 00:25:04,588 --> 00:25:07,438 E even if it's just a small part of the environment. 505 00:25:07,438 --> 00:25:09,178 What I'm that, that's all I'm saying. 506 00:25:09,195 --> 00:25:09,525 Yeah. 507 00:25:09,553 --> 00:25:10,063 Right. 508 00:25:10,153 --> 00:25:16,063 That, that, that we often focus a little bit too much time on the 509 00:25:16,063 --> 00:25:19,963 recovery, the restore, and we say, okay, the application's restored. 510 00:25:19,963 --> 00:25:20,653 I can walk away. 511 00:25:20,653 --> 00:25:22,003 No, the application's restored. 512 00:25:22,003 --> 00:25:26,893 When the application is restored, when people can do the thing that whatever 513 00:25:26,893 --> 00:25:28,393 it is that application was supposed to. 514 00:25:28,395 --> 00:25:31,845 I was intent, I was thinking more about, be careful about thinking 515 00:25:31,845 --> 00:25:33,255 about all the failure scenarios. 516 00:25:33,255 --> 00:25:35,685 Like I was saying, the person gets sick with the flu, right. 517 00:25:36,253 --> 00:25:37,153 Oh, yeah, yeah, yeah. 518 00:25:37,153 --> 00:25:37,303 Yeah. 519 00:25:37,605 --> 00:25:41,235 about going down that rabbit hole because you will never come back 520 00:25:41,235 --> 00:25:45,315 out because it might be, what if the butterfly flops its wing halfway around 521 00:25:45,315 --> 00:25:48,015 the world and causes X, Y, Z, right? 522 00:25:48,015 --> 00:25:48,525 So, 523 00:25:49,453 --> 00:25:51,013 The butterfly will die. 524 00:25:53,340 --> 00:25:53,690 right. 525 00:25:54,105 --> 00:25:56,775 So don't get overwhelmed by these scenarios is 526 00:25:57,403 --> 00:25:58,003 Yeah. 527 00:25:58,093 --> 00:26:02,203 And, and speaking of not being overwhelmed when we get to the 528 00:26:02,203 --> 00:26:05,743 post, you know, when we get to the, uh, the post game analysis, right? 529 00:26:06,313 --> 00:26:10,003 Let's measure against the success criteria that we agreed to. 530 00:26:10,393 --> 00:26:13,753 Um, we, we look at the things that didn't work and the 531 00:26:13,753 --> 00:26:15,373 bottlenecks and things like that. 532 00:26:16,003 --> 00:26:20,623 The key, again here is to better the world, not to prove that 533 00:26:20,623 --> 00:26:22,723 you were the best or whatever. 534 00:26:22,723 --> 00:26:22,783 I. 535 00:26:23,083 --> 00:26:25,543 Um, I know it can be really difficult. 536 00:26:25,603 --> 00:26:26,293 Say that again. 537 00:26:26,475 --> 00:26:27,435 you were the worst. 538 00:26:28,483 --> 00:26:29,383 You were the worst. 539 00:26:30,223 --> 00:26:30,733 Yeah. 540 00:26:30,913 --> 00:26:33,703 Um, you know, we're looking for things that we can improve. 541 00:26:33,983 --> 00:26:37,313 We're looking for procedures that we can update based on, you know, 542 00:26:37,313 --> 00:26:38,543 the lessons that we learned. 543 00:26:38,993 --> 00:26:41,783 Um, any other post-game analysis? 544 00:26:41,783 --> 00:26:43,133 What can you, that you can think of? 545 00:26:43,133 --> 00:26:44,273 I would also say. 546 00:26:45,473 --> 00:26:49,163 If this is your first time doing this, I think it's also good 547 00:26:49,163 --> 00:26:51,893 to say what things went well. 548 00:26:52,223 --> 00:26:55,193 I think a lot of times we tend to focus on the negatives, 549 00:26:55,566 --> 00:26:56,106 Right. 550 00:26:56,993 --> 00:26:57,383 right? 551 00:26:57,743 --> 00:27:00,683 But if this is your first time, like this is really hard. 552 00:27:01,013 --> 00:27:02,363 This is a hard thing to do. 553 00:27:02,946 --> 00:27:03,396 Yeah. 554 00:27:03,713 --> 00:27:07,373 And you should acknowledge that and realize if you got through, like I 555 00:27:07,373 --> 00:27:10,133 know Curtis, you've always talked about the bank and your DR tests, right? 556 00:27:10,133 --> 00:27:13,673 And how I don't think you guys ever completed a hundred 557 00:27:14,001 --> 00:27:14,391 No. 558 00:27:14,393 --> 00:27:14,813 right? 559 00:27:14,901 --> 00:27:15,501 No. 560 00:27:16,851 --> 00:27:18,591 Yeah, yeah. 561 00:27:19,431 --> 00:27:21,021 So don't be too hard on yourself. 562 00:27:21,321 --> 00:27:24,621 Congratulate yourself first off on doing the test in the first place, 563 00:27:25,011 --> 00:27:28,116 and second, making it to the end of the test, even if everyone is dead. 564 00:27:31,646 --> 00:27:36,021 Um, you know, and then, and then, and then, you know, yeah, 565 00:27:36,081 --> 00:27:37,401 don't be too hard on yourself. 566 00:27:37,461 --> 00:27:37,821 Right. 567 00:27:37,851 --> 00:27:42,921 Uh, because these things, these things rarely do they go well, uh, 568 00:27:42,921 --> 00:27:44,661 unless it's like fully automated. 569 00:27:44,661 --> 00:27:47,421 And, you know, the, the more I will say, the more you can 570 00:27:47,421 --> 00:27:48,411 automate things, the better. 571 00:27:48,411 --> 00:27:48,771 Right? 572 00:27:48,833 --> 00:27:48,893 Yeah. 573 00:27:49,591 --> 00:27:51,721 So you ran the tests, 574 00:27:51,874 --> 00:27:52,294 Mm-hmm. 575 00:27:52,711 --> 00:27:54,721 things that went well, things that went wrong. 576 00:27:55,441 --> 00:27:57,271 think the next step after that is. 577 00:27:58,681 --> 00:28:01,351 Identifying how do you close the gaps, 578 00:28:02,044 --> 00:28:02,394 Right. 579 00:28:02,701 --> 00:28:04,801 And coming up with a plan, because you don't want to 580 00:28:04,801 --> 00:28:06,421 just let these things linger, 581 00:28:07,064 --> 00:28:07,574 Right. 582 00:28:07,741 --> 00:28:08,491 create a plan. 583 00:28:08,491 --> 00:28:12,601 Identify what are the most critical elements that you want to address first 584 00:28:13,064 --> 00:28:13,214 Mm-hmm. 585 00:28:13,231 --> 00:28:18,661 timeframes, and make sure you get buy-in across the board to fix those things. 586 00:28:19,514 --> 00:28:19,844 Yeah. 587 00:28:19,844 --> 00:28:20,414 Agreed. 588 00:28:20,744 --> 00:28:21,164 Right. 589 00:28:21,284 --> 00:28:25,064 Um, you, you, you have a, you have an action item list and who's responsible 590 00:28:25,064 --> 00:28:28,664 for addressing the different things, and then of course, what's the next thing? 591 00:28:28,694 --> 00:28:29,474 You do it again. 592 00:28:29,954 --> 00:28:30,344 Right? 593 00:28:30,524 --> 00:28:31,094 Um. 594 00:28:31,486 --> 00:28:31,936 When, 595 00:28:32,639 --> 00:28:33,839 Uh, soon. 596 00:28:34,499 --> 00:28:34,799 Right. 597 00:28:35,099 --> 00:28:40,469 Um, I would say I'm a fan of more frequent, smaller tests 598 00:28:40,769 --> 00:28:42,749 than like an annual huge test. 599 00:28:43,319 --> 00:28:43,829 Right. 600 00:28:43,919 --> 00:28:47,669 Um, I think the more often we do that, the more we get into the, the 601 00:28:47,669 --> 00:28:54,179 mindset of thinking about the things that can go wrong, because a, a lot 602 00:28:54,179 --> 00:28:57,059 of things are, are, you know, um. 603 00:28:57,734 --> 00:29:03,669 They're the same on different discip disciplines across the, uh, the, 604 00:29:03,764 --> 00:29:05,474 the, uh, the organization, right? 605 00:29:05,894 --> 00:29:11,534 So the more often we test, the more often we get to a recovery mindset and 606 00:29:11,564 --> 00:29:16,124 we start including those things in the system design from the very beginning. 607 00:29:16,271 --> 00:29:16,561 Yeah. 608 00:29:16,604 --> 00:29:16,994 Right? 609 00:29:17,084 --> 00:29:19,244 Um, again, that's the other purpose. 610 00:29:19,244 --> 00:29:20,504 I would add that to. 611 00:29:21,134 --> 00:29:25,604 My original question, that's the other purpose of a DR test, is 612 00:29:25,604 --> 00:29:27,884 to get people to a DR mindset, 613 00:29:28,111 --> 00:29:28,171 Yeah. 614 00:29:28,424 --> 00:29:34,604 um, to a recovery mindset of saying, um, we need to design the infrastructure and 615 00:29:34,604 --> 00:29:39,854 the processes around the infrastructure so that they are easy to recovery. 616 00:29:39,854 --> 00:29:39,884 Right. 617 00:29:39,891 --> 00:29:40,221 Yep. 618 00:29:40,221 --> 00:29:44,661 Or at least even think about it to start with rather than, oh yeah, this failed. 619 00:29:44,661 --> 00:29:45,141 Now what? 620 00:29:45,231 --> 00:29:46,426 What were you gonna do with our Dr. 621 00:29:47,354 --> 00:29:47,774 Yeah. 622 00:29:47,779 --> 00:29:54,404 And, and, and lemme just give you a, a, a silly but simple example of what happens 623 00:29:54,404 --> 00:29:56,114 when you don't have a recovery mindset. 624 00:29:57,299 --> 00:30:00,509 So I go back to the bank, right? 625 00:30:00,509 --> 00:30:03,329 I have so many good stories from the days of the bank, right? 626 00:30:03,689 --> 00:30:11,249 And when we bought a a T 1000, which was, uh, an HP server, it 627 00:30:11,249 --> 00:30:17,429 was a really big server and it had, um, it was a huge server. 628 00:30:17,429 --> 00:30:20,879 It had a hundred gigabytes of data. 629 00:30:22,261 --> 00:30:23,521 Ginormous, wait. 630 00:30:23,521 --> 00:30:25,231 Let me go grab my flash drive. 631 00:30:25,709 --> 00:30:33,689 It was a huge server for the time, and it came with a two gigabyte tape drive. 632 00:30:34,739 --> 00:30:35,189 Right. 633 00:30:35,249 --> 00:30:37,709 I think with compression it was like a four gigabyte tape drive 634 00:30:39,389 --> 00:30:41,759 that, that was a system design. 635 00:30:41,939 --> 00:30:42,964 And there, there were no changes. 636 00:30:43,424 --> 00:30:46,684 No, we, we, we added 30%. 637 00:30:47,804 --> 00:30:51,104 With one server, we added 30% to the capacity of the 638 00:30:51,104 --> 00:30:53,084 data center with one server. 639 00:30:53,294 --> 00:30:57,254 There wasn't a single discussion about what we should do from a 640 00:30:57,254 --> 00:30:58,694 backup and recovery perspective. 641 00:30:59,414 --> 00:31:03,104 That's what happens when you don't have a recovery mindset, 642 00:31:03,151 --> 00:31:03,421 Yeah. 643 00:31:03,524 --> 00:31:03,944 right? 644 00:31:04,274 --> 00:31:09,224 Is that you, you do things, you add things to the system without any thought 645 00:31:09,224 --> 00:31:14,354 to what they would, you know, how that would impact the recovery system. 646 00:31:16,004 --> 00:31:17,954 So that's why we want to have a recovery mindset. 647 00:31:18,211 --> 00:31:18,481 Yep. 648 00:31:20,414 --> 00:31:20,864 Okey dokey. 649 00:31:22,651 --> 00:31:23,851 I think we covered everything. 650 00:31:23,911 --> 00:31:24,271 Yeah. 651 00:31:24,884 --> 00:31:28,064 I think so, yeah, everything you could possibly want to know about 652 00:31:28,064 --> 00:31:34,214 Dr in, uh, four episodes with the two, the two, maybe five. 653 00:31:34,214 --> 00:31:34,514 We'll see. 654 00:31:34,514 --> 00:31:36,044 We'll see if I can find that other episode. 655 00:31:36,834 --> 00:31:41,634 Thanks Prasanna for, uh, you know, once again, uh, you know, great team. 656 00:31:42,011 --> 00:31:42,701 Woo hoo. 657 00:31:42,701 --> 00:31:43,241 Go team. 658 00:31:43,241 --> 00:31:43,631 Go. 659 00:31:44,439 --> 00:31:48,339 Team go and uh, I want to thank you once again to our listeners. 660 00:31:48,609 --> 00:31:49,959 We'd be nothing without you. 661 00:31:50,019 --> 00:31:51,279 That is a wrap.