1 00:00:00,000 --> 00:00:04,019 You found the backup wrap up your go-to podcast for all things 2 00:00:04,019 --> 00:00:06,599 backup recovery and cyber recovery. 3 00:00:07,109 --> 00:00:12,059 In this episode, we talk about some hard earned disaster recovery lessons 4 00:00:12,300 --> 00:00:14,489 from major events like nine 11. 5 00:00:15,150 --> 00:00:19,139 We talk about what we learned about DR that day, and we talk about 6 00:00:19,139 --> 00:00:21,510 those critical human elements of DR. 7 00:00:21,510 --> 00:00:25,679 That people often forget, like where your recovery team is going to 8 00:00:25,679 --> 00:00:27,900 sleep when the hotels are all gone. 9 00:00:28,529 --> 00:00:31,290 Disasters happen and they're never convenient. 10 00:00:31,770 --> 00:00:35,730 Whether it's terrorists, hurricanes, or ransomware, you 11 00:00:35,730 --> 00:00:37,739 need to think through what you do. 12 00:00:37,800 --> 00:00:41,580 If you're completely isolated from the world, the time to 13 00:00:41,580 --> 00:00:43,140 learn these lessons are now. 14 00:00:43,170 --> 00:00:44,760 So I hope you enjoy this episode. 15 00:00:45,440 --> 00:00:50,360 By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr. Backup, 16 00:00:50,690 --> 00:00:55,460 and I've been passionate about backup and disaster recovery for over 30 years. 17 00:00:55,820 --> 00:00:56,420 Ever since. 18 00:00:56,420 --> 00:00:59,660 I had to tell my boss that there were no backups of the 19 00:00:59,660 --> 00:01:01,490 big database that we just lost. 20 00:01:01,820 --> 00:01:05,150 I don't want that to happen to you, and that's why I do this podcast. 21 00:01:05,660 --> 00:01:11,000 On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes. 22 00:01:11,360 --> 00:01:13,430 This is the backup wrap up. 23 00:01:28,061 --> 00:01:28,961 Welcome to the show. 24 00:01:30,131 --> 00:01:34,631 Hi, I am w Curtis Preston, AKA, Mr. Backup, and I have with me a guy who 25 00:01:34,631 --> 00:01:38,951 is as lazy as I am when it comes to how to move rugs around the house. 26 00:01:39,231 --> 00:01:41,441 Prasanna Malaiyandi, how's it going? 27 00:01:41,441 --> 00:01:41,861 Prasanna. 28 00:01:42,630 --> 00:01:49,110 I am doing well, Curtis, by the way, isn't it amazing how quiet a room gets 29 00:01:49,180 --> 00:01:50,895 when you put something on the floor? 30 00:01:51,491 --> 00:01:55,751 Yeah, I, I, I wonder if anybody, uh, if anybody notices the difference 31 00:01:55,751 --> 00:01:59,261 in sound because it, you know, and it's been, they probably just got 32 00:01:59,261 --> 00:02:00,881 so used to the other sound, right? 33 00:02:00,886 --> 00:02:05,316 Because I. I went with LVP, you know, upstairs and um, and then in 34 00:02:05,316 --> 00:02:09,876 my office and it just got so echoy and I put the stuff on this wall. 35 00:02:09,876 --> 00:02:11,916 And this over here is a acoustic panels. 36 00:02:12,456 --> 00:02:15,426 Um, you know, behind me there are acoustic panels up on the ceiling 37 00:02:15,426 --> 00:02:16,506 and it still wasn't the same. 38 00:02:16,506 --> 00:02:20,876 I. And then I found, you know, I, I, I, this, this rug is from actually my 39 00:02:20,876 --> 00:02:26,276 living room because we bought a nicer rug and, uh, a nicer, bigger rug. 40 00:02:26,306 --> 00:02:28,706 And so we put that one down there and then I moved here. 41 00:02:28,856 --> 00:02:31,736 But when I moved it in, I, I, I did what I thought was like 42 00:02:31,736 --> 00:02:33,146 a really weird way to do it. 43 00:02:33,146 --> 00:02:34,706 I didn't actually move the furniture. 44 00:02:34,706 --> 00:02:38,156 I was like lifting it up and trying to do it without having to move everything out. 45 00:02:38,156 --> 00:02:39,686 And you said, well, that's exactly how I did it. 46 00:02:40,095 --> 00:02:40,665 exactly. 47 00:02:40,665 --> 00:02:41,055 Yeah. 48 00:02:41,621 --> 00:02:45,296 And I, I, I thought it was just me, but apparently. 49 00:02:46,256 --> 00:02:46,766 It wasn't. 50 00:02:47,190 --> 00:02:51,750 because at least with the rug, well, it depends on how your desk is too. 51 00:02:51,750 --> 00:02:55,950 But like you can at least unfurl part of it and just kind of like shove it down. 52 00:02:55,950 --> 00:02:56,820 Like you just lift 53 00:02:56,820 --> 00:02:58,140 up the front part and 54 00:02:58,140 --> 00:02:59,175 then you lift up the back part. 55 00:02:59,355 --> 00:02:59,575 So, 56 00:03:00,356 --> 00:03:00,656 yeah. 57 00:03:00,691 --> 00:03:03,296 I, I think different people might go, no, no, we're gonna, 58 00:03:03,536 --> 00:03:04,646 we're gonna move everything out. 59 00:03:04,646 --> 00:03:06,836 We're gonna put the rug down and we'll move everything back in. 60 00:03:06,896 --> 00:03:09,326 And they might think of that as easier, but I was like, I 61 00:03:09,326 --> 00:03:10,226 don't wanna move everything. 62 00:03:10,380 --> 00:03:11,970 well, it's more than moving. 63 00:03:11,970 --> 00:03:13,710 It's setting everything back up again. 64 00:03:14,321 --> 00:03:16,091 Yeah, I, I looked into it. 65 00:03:16,091 --> 00:03:20,561 I could have potentially moved the desk without disassembling everything. 66 00:03:20,591 --> 00:03:25,811 It was just, you know, but, um, yeah, so I, it was just nice to hear 67 00:03:25,811 --> 00:03:27,551 when I was talking to you for once. 68 00:03:27,671 --> 00:03:31,391 You weren't like, 'cause so many times I tell you about something that I'm 69 00:03:31,391 --> 00:03:34,091 doing and then you're like, that's the dumbest thing I've ever heard. 70 00:03:34,091 --> 00:03:35,081 Why would you do it that way? 71 00:03:35,351 --> 00:03:37,811 And you're like, oh no, that's, uh, that's how I do it. 72 00:03:38,070 --> 00:03:38,400 Yeah. 73 00:03:40,275 --> 00:03:41,595 Especially if you're space limited. 74 00:03:41,595 --> 00:03:42,135 Like for 75 00:03:42,236 --> 00:03:42,746 What's that? 76 00:03:43,275 --> 00:03:45,195 especially if you are space limited 77 00:03:45,386 --> 00:03:47,396 Yes, space limited is true. 78 00:03:47,745 --> 00:03:51,915 I was just looking, so I have my desk right here and then 79 00:03:51,915 --> 00:03:53,715 like that way, that way I 80 00:03:53,715 --> 00:03:57,375 have the door, but there's no way I could fit my desk outside of 81 00:03:57,375 --> 00:03:58,905 the door with it fully assembled. 82 00:03:59,516 --> 00:04:00,416 Oh yeah. 83 00:04:00,435 --> 00:04:03,675 I'd have to at least take the top off, which is just a giant chore. 84 00:04:03,675 --> 00:04:03,825 So. 85 00:04:04,796 --> 00:04:05,336 Oh yeah. 86 00:04:05,336 --> 00:04:10,316 Well, in my case, I don't have that excuse 'cause I have two French doors, so I 87 00:04:10,316 --> 00:04:13,316 wouldn't have had that excuse, but yeah. 88 00:04:13,316 --> 00:04:16,196 But it still, it still would've been, it still would've been annoying. 89 00:04:16,256 --> 00:04:20,276 And so, but we figured, we figured it out and now I have better sound. 90 00:04:20,906 --> 00:04:21,716 It's a beautiful thing. 91 00:04:22,106 --> 00:04:27,366 And, uh, so, uh, today we're gonna talk about DR and disaster recovery and 92 00:04:27,366 --> 00:04:33,426 lessons learned, and especially, um, some lessons learned from at least one 93 00:04:33,426 --> 00:04:36,546 major event, uh, one major disaster. 94 00:04:37,026 --> 00:04:43,326 And, you know, I, I don't want to, um, you know, the, these, these events, 95 00:04:43,326 --> 00:04:47,806 especially this event, disasters are, are always difficult on, on the people. 96 00:04:48,406 --> 00:04:51,436 Uh, and, and I don't want in any way make light of those disasters 97 00:04:51,436 --> 00:04:54,256 or I, I don't know, whatever, whatever I'm trying to say. 98 00:04:54,256 --> 00:04:54,496 Right. 99 00:04:54,496 --> 00:04:56,386 So proper respect to the people. 100 00:04:56,416 --> 00:04:58,996 'cause some of the, well, especially the one, the main one that we're 101 00:04:58,996 --> 00:05:03,616 gonna talk about, people died in these disasters and, um, you know, 102 00:05:03,916 --> 00:05:05,386 with respect to those people. 103 00:05:05,876 --> 00:05:06,866 Having said that. 104 00:05:08,606 --> 00:05:13,886 We can learn from the things that happened, uh, at that time. 105 00:05:14,246 --> 00:05:14,666 Right? 106 00:05:14,786 --> 00:05:18,626 And so let's talk about, and and the disaster. 107 00:05:18,971 --> 00:05:22,466 The, the main disaster I'm talking about at this point would be what 108 00:05:22,466 --> 00:05:23,966 we all called nine 11, right? 109 00:05:24,026 --> 00:05:26,126 So September 11th, 2001. 110 00:05:26,876 --> 00:05:28,376 Uh, I lived through it. 111 00:05:28,406 --> 00:05:29,426 You lived through it. 112 00:05:29,696 --> 00:05:33,266 Uh, so how old would you have been in 2001? 113 00:05:34,905 --> 00:05:36,345 I would rather not say 114 00:05:38,745 --> 00:05:39,735 I was very young. 115 00:05:40,496 --> 00:05:41,456 You were very young. 116 00:05:42,345 --> 00:05:44,175 I, I was actually in college. 117 00:05:44,786 --> 00:05:45,236 okay. 118 00:05:46,316 --> 00:05:54,956 I was here, um, and um, I was in this house and my kids were young. 119 00:05:55,426 --> 00:06:01,906 My oldest would've been, uh, seven, and my younger one would've been in four. 120 00:06:02,116 --> 00:06:05,896 And what I remember was she might've been in, she might've been five, 121 00:06:05,896 --> 00:06:07,936 she might've been in preschool or might've been in kindergarten. 122 00:06:09,286 --> 00:06:13,816 What I remember, and for those that don't know, I live in North County San Diego, 123 00:06:13,816 --> 00:06:17,866 which means I live just south of Camp Pendleton, which is, uh, you know, one 124 00:06:17,866 --> 00:06:20,206 of the biggest, uh, Marine Corps bases. 125 00:06:22,306 --> 00:06:27,136 The, the idea was that we were under attack because there were multiple events 126 00:06:27,136 --> 00:06:28,786 that were happening simultaneously. 127 00:06:28,786 --> 00:06:29,086 Right. 128 00:06:29,476 --> 00:06:35,026 Multiple planes hit the, the trade center, there was the plane that hit the Pentagon. 129 00:06:35,116 --> 00:06:37,156 There was the plane that went down in Pennsylvania. 130 00:06:37,366 --> 00:06:41,896 There was this feeling that, like we were being attacked 131 00:06:41,896 --> 00:06:48,226 as a country, which we were, and living near a military base. 132 00:06:49,276 --> 00:06:53,476 I don't know what I thought I was going to accomplish by keeping my kids home 133 00:06:53,476 --> 00:06:55,786 from school, but that's what I did. 134 00:06:56,105 --> 00:06:56,285 Yeah. 135 00:06:57,046 --> 00:07:00,796 I was just like, I'm gonna, I'm gonna, I'm gonna, I'm gonna, I'm 136 00:07:00,796 --> 00:07:04,336 gonna keep my kid, like, it, it, like, I, I, I don't even know what 137 00:07:04,336 --> 00:07:08,626 I was thinking, but like, you know, thinking back on it now I'm thinking 138 00:07:08,626 --> 00:07:11,566 that probably I was thinking that, um. 139 00:07:12,301 --> 00:07:14,776 I, I, I, if the world's gonna end, I want my kids near me. 140 00:07:14,836 --> 00:07:14,926 I 141 00:07:14,926 --> 00:07:17,746 mean, it was like, it was, it was, it was, it's kind of morbid. 142 00:07:17,746 --> 00:07:18,316 But that's, 143 00:07:18,856 --> 00:07:23,806 I, I just remember that I didn't, you know, that, that I kept my kids near me. 144 00:07:23,926 --> 00:07:28,306 Um, and I remember that I knew multiple people that worked in 145 00:07:28,306 --> 00:07:29,836 the Real World Trade Center. 146 00:07:30,256 --> 00:07:32,176 None of them were hurt that day 147 00:07:32,180 --> 00:07:32,750 Mm. 148 00:07:33,856 --> 00:07:35,806 and for various reasons. 149 00:07:35,926 --> 00:07:41,026 One, uh, and I know one person that was in the World Trade Center that day. 150 00:07:41,776 --> 00:07:45,436 I knew other people that worked in the World Trade Center, but 151 00:07:45,436 --> 00:07:49,366 for one reason or another, chose not to go into work that day. 152 00:07:49,820 --> 00:07:49,900 Hmm. 153 00:07:50,581 --> 00:07:54,391 I know another person that was supposed to be on Flight 11, which 154 00:07:54,391 --> 00:07:58,231 was from Boston to New York, which was the flight that ended up going. 155 00:07:58,891 --> 00:08:00,811 I don't know if that's the one that went into the Pentagon or 156 00:08:00,811 --> 00:08:04,381 if that's the one that crashed in Pennsylvania, but I knew a person that 157 00:08:04,381 --> 00:08:05,791 was supposed to be on that flight. 158 00:08:05,851 --> 00:08:08,011 That person did not get on that flight. 159 00:08:09,271 --> 00:08:13,291 And, uh, and then the other person that, that I knew was the person in 160 00:08:13,471 --> 00:08:19,321 the, in the trade center, it was, uh, Michael Hingis that was, uh, the, the 161 00:08:19,321 --> 00:08:25,381 blind person that made it, that made it down thanks to his seeing eye dog. 162 00:08:25,861 --> 00:08:27,541 Um, and he, you know, he. 163 00:08:28,156 --> 00:08:29,086 Somewhat famous. 164 00:08:29,086 --> 00:08:29,806 As a result. 165 00:08:30,196 --> 00:08:33,136 He actually went on to become a motivational speaker. 166 00:08:33,736 --> 00:08:38,956 And, um, and, uh, just to, just to lighten things up, I, I'll talk about, 167 00:08:38,986 --> 00:08:43,006 uh, how I first met Michael, and that was, and I, it seems like I've told 168 00:08:43,006 --> 00:08:46,876 this story relatively recently, but I was at my very first trade show. 169 00:08:46,876 --> 00:08:51,766 This would've been in the early nineties, and it was at, it was in 170 00:08:51,766 --> 00:08:55,336 New York, um, at the Javit Center. 171 00:08:55,891 --> 00:09:04,981 In Manhattan, and it was Unix Expo, and I saw, uh, these hot swappable 172 00:09:04,981 --> 00:09:06,421 dish drives, you know, where, where you 173 00:09:06,421 --> 00:09:08,251 could push a button and pull the drives out. 174 00:09:08,251 --> 00:09:09,811 And I just thought that was the coolest thing I'd ever seen. 175 00:09:09,811 --> 00:09:13,231 'cause at the, at the time that, that was unlike anything I'd ever 176 00:09:13,231 --> 00:09:15,811 seen it, it, that was really, really new and really, really cool. 177 00:09:15,961 --> 00:09:17,071 Now we just take it for granted. 178 00:09:17,131 --> 00:09:21,061 But back then it was really cool and he was, he was the one that was. 179 00:09:21,751 --> 00:09:26,971 The se standing in the booth demonstrating these, these, uh, disc drives and 180 00:09:26,971 --> 00:09:28,681 literally his, his shtick was. 181 00:09:29,386 --> 00:09:31,336 It's so easy a blind man can do it. 182 00:09:31,786 --> 00:09:31,996 Right? 183 00:09:33,586 --> 00:09:37,006 And um, and we were like, this is amazing. 184 00:09:37,006 --> 00:09:38,386 We're like, you guys are the only ones with it. 185 00:09:38,476 --> 00:09:40,606 And he goes, yes, we are the only ones with this product. 186 00:09:40,606 --> 00:09:43,846 And then we walked through the trade center or the trade show and we saw 187 00:09:43,876 --> 00:09:45,406 several other vendors with the product. 188 00:09:45,946 --> 00:09:48,136 And one of us said to the other. 189 00:09:48,341 --> 00:09:50,921 Well, in his defense, he can't see the other vendors. 190 00:09:53,925 --> 00:09:54,740 So true. 191 00:09:56,771 --> 00:10:00,251 Um, do you, do you remember, do you have memories of that day? 192 00:10:00,251 --> 00:10:00,791 Of course You do. 193 00:10:00,860 --> 00:10:04,760 Oh yeah, so I remember, so I went to school, I was in Pittsburgh actually. 194 00:10:04,991 --> 00:10:05,141 Mm-hmm. 195 00:10:05,510 --> 00:10:10,220 And so we had a lot of friends, or I had a lot of friends whose families were 196 00:10:10,220 --> 00:10:11,870 living in the city in New York City. 197 00:10:12,216 --> 00:10:12,566 Right. 198 00:10:12,590 --> 00:10:16,370 And so I remember everyone just was very, very concerned. 199 00:10:16,730 --> 00:10:19,700 Um, and like you had mentioned, there was sort of the plane flying towards. 200 00:10:20,585 --> 00:10:21,665 Uh, over Pennsylvania. 201 00:10:21,665 --> 00:10:25,835 And so everyone in Pittsburgh was keeping a close eye being like, Hey, 202 00:10:25,835 --> 00:10:27,755 where is that plane actually going? 203 00:10:27,755 --> 00:10:28,115 Because 204 00:10:28,115 --> 00:10:29,885 it was supposed to fly over Pittsburgh, 205 00:10:30,526 --> 00:10:30,886 right. 206 00:10:30,995 --> 00:10:31,355 right? 207 00:10:31,355 --> 00:10:36,905 So I remember everyone sort of being worried, concerned because they had family 208 00:10:36,905 --> 00:10:40,925 and relatives and they weren't able, like all the phone lines were shut down, right? 209 00:10:40,925 --> 00:10:41,495 So. 210 00:10:42,050 --> 00:10:44,960 No one was able to figure out like what was going on. 211 00:10:45,230 --> 00:10:49,550 I remember going with a bunch of other folks down to the common area 212 00:10:49,820 --> 00:10:53,660 and just kind of like just being like shellshocked as we're watching the news. 213 00:10:54,886 --> 00:10:57,016 Yeah, I, yeah, absolutely. 214 00:10:57,136 --> 00:11:01,126 Um, there's a, there's a great line in the beginning of a movie that I like. 215 00:11:01,126 --> 00:11:03,436 It's become problematic now. 216 00:11:03,916 --> 00:11:08,206 Um, maybe it was always problematic, but there's a movie called Love, actually, 217 00:11:08,536 --> 00:11:15,436 and in the beginning, which is, uh, voiceover from, um, Hugh, Hugh Grant. 218 00:11:16,066 --> 00:11:19,846 And, and, and he was saying on nine 11. 219 00:11:20,461 --> 00:11:23,251 There were a lot of phone calls that were made from the plains, 220 00:11:23,311 --> 00:11:28,771 and he said, to my knowledge, none of them were messages of hate. 221 00:11:28,801 --> 00:11:29,041 They were 222 00:11:29,041 --> 00:11:30,421 messages of love. 223 00:11:31,535 --> 00:11:31,835 Hmm. 224 00:11:33,871 --> 00:11:35,101 I just got a little of a clump there. 225 00:11:35,161 --> 00:11:36,931 Anyway, so, okay. 226 00:11:38,461 --> 00:11:47,611 So, um, when we think about that event, there is, in my world, we 227 00:11:47,611 --> 00:11:49,441 immediately started talking about. 228 00:11:50,796 --> 00:11:55,831 The, we saw the things that happened to companies, and we're gonna talk 229 00:11:55,831 --> 00:12:00,121 about, I'm gonna talk about sort of two, two different kinds of companies. 230 00:12:00,151 --> 00:12:01,141 One of them. 231 00:12:01,501 --> 00:12:05,911 So first off, let's talk about, uh, the, the difference between 232 00:12:05,911 --> 00:12:06,991 a cold site and a hot site. 233 00:12:07,591 --> 00:12:10,681 Do you, so when we talk about disaster recovery, there was this 234 00:12:10,681 --> 00:12:13,741 idea that we, that we're gonna have another site, like ready to go 235 00:12:13,930 --> 00:12:14,200 Yep. 236 00:12:14,911 --> 00:12:16,471 and we talk about cold site and the hot site. 237 00:12:16,471 --> 00:12:17,671 Do you want to talk about what that means? 238 00:12:17,705 --> 00:12:23,225 Yeah, so a hot site basically is you have a site, right? 239 00:12:23,225 --> 00:12:26,585 A disaster recovery site that is fully operational, has all the 240 00:12:26,585 --> 00:12:31,265 equipment, has everything replicated to it, and basically once you push 241 00:12:31,265 --> 00:12:32,975 the BRI big red button, right? 242 00:12:33,245 --> 00:12:34,775 Everything sort of fails over. 243 00:12:34,775 --> 00:12:39,095 It's available, operational, all ready to go, sort of ready 244 00:12:39,095 --> 00:12:43,175 to serve traffic and take over, usually minutes to hours within a 245 00:12:43,175 --> 00:12:43,565 failure. 246 00:12:44,266 --> 00:12:44,686 right. 247 00:12:45,316 --> 00:12:47,956 And then a cold site is very much the opposite of that, right? 248 00:12:48,286 --> 00:12:53,986 It's a site that's sort of ready to start a restore. 249 00:12:54,676 --> 00:12:58,816 Uh, like I suppose there'd be a, there'd be a no site, 250 00:12:58,965 --> 00:12:59,255 Yeah. 251 00:13:00,226 --> 00:13:00,496 right? 252 00:13:00,736 --> 00:13:03,286 That's where, uh, a bad thing happened. 253 00:13:03,286 --> 00:13:05,356 And we're gonna go find some hardware to restore 254 00:13:05,356 --> 00:13:07,096 to, uh, cold site, the, the. 255 00:13:07,816 --> 00:13:12,016 Implication there is that you have some hardware ready to go, but 256 00:13:12,016 --> 00:13:13,336 you haven't restored anything. 257 00:13:13,756 --> 00:13:16,126 A warm site is somewhere in between those two things, 258 00:13:16,355 --> 00:13:19,565 I've have, have you heard of this term that some people call pilot? 259 00:13:20,716 --> 00:13:21,526 A pilot light 260 00:13:21,665 --> 00:13:21,845 Yeah. 261 00:13:22,816 --> 00:13:23,206 talk to. 262 00:13:23,266 --> 00:13:23,596 Talk to me 263 00:13:23,765 --> 00:13:29,435 So it's basically not quite cold, but not quite warm or hot, right? 264 00:13:29,435 --> 00:13:29,675 So 265 00:13:29,675 --> 00:13:33,245 it's a little bit better than cold, but not quite to the extent. 266 00:13:33,245 --> 00:13:37,415 And a lot of the trade off comes from not necessarily having to 267 00:13:37,415 --> 00:13:39,215 eat all of the costs upfront. 268 00:13:40,235 --> 00:13:40,475 So 269 00:13:40,475 --> 00:13:44,765 as an example, if your pilot site is, say, in the cloud, you might have your 270 00:13:44,765 --> 00:13:47,885 data available, but not necessarily your compute and everything else ready to go. 271 00:13:49,216 --> 00:13:52,096 Yeah, I think that I, I think I would still call that a warm sight, but 272 00:13:52,186 --> 00:13:55,126 I mean, there is this concept, it comes from the concept of a pilot 273 00:13:55,126 --> 00:13:55,396 light. 274 00:13:56,176 --> 00:13:56,216 I. Right. 275 00:13:56,221 --> 00:13:58,651 Which for those of you that don't know, when you have a 276 00:13:58,651 --> 00:14:00,541 gas old school that 277 00:14:00,571 --> 00:14:04,861 nowadays we have electronic ignition, but there used to be this in inside, 278 00:14:04,861 --> 00:14:07,471 if you had a gas water heater or a gas furnace, there would be this 279 00:14:07,471 --> 00:14:09,031 little flame that would burn all the 280 00:14:09,031 --> 00:14:09,481 time. 281 00:14:09,841 --> 00:14:11,551 And that would, that's called your pilot light. 282 00:14:11,911 --> 00:14:16,111 And if the pilot light goes out, then you're gonna have to relight it 283 00:14:16,111 --> 00:14:18,091 because otherwise your, your heat won't work. 284 00:14:18,211 --> 00:14:19,411 Um, modern days we use. 285 00:14:19,936 --> 00:14:22,996 electronic electronic ignition, 286 00:14:23,806 --> 00:14:26,811 but, um, because a pilot like just wastes a lot of, 287 00:14:27,215 --> 00:14:27,575 yep. 288 00:14:27,725 --> 00:14:30,635 It's always running and you're always consuming gas. 289 00:14:30,886 --> 00:14:31,306 Yeah. 290 00:14:31,426 --> 00:14:39,376 Um, so there, so the, the best from a DR perspective, right? 291 00:14:39,376 --> 00:14:43,846 The, the, the, the Cadillac, if you will, is, um. 292 00:14:44,476 --> 00:14:49,156 The, I don't know if that's the right term anymore because nobody buys Cadillac, the 293 00:14:49,156 --> 00:14:51,526 Ferrari, the Rolls Royce. 294 00:14:51,586 --> 00:14:53,326 Does anybody buy, do they still make Rolls 295 00:14:53,585 --> 00:14:54,185 Oh yeah. 296 00:14:54,706 --> 00:14:55,096 Okay. 297 00:14:56,176 --> 00:14:59,776 Um, is the hot site 298 00:15:00,526 --> 00:15:02,986 that it's, it's, it's ready to go. 299 00:15:02,986 --> 00:15:03,406 Number one. 300 00:15:03,406 --> 00:15:08,386 It's ready to go when you need it, and, and two, it's kept and 301 00:15:08,446 --> 00:15:11,566 it's ready to go within a, a few. 302 00:15:12,331 --> 00:15:13,741 Minutes or seconds, right? 303 00:15:13,741 --> 00:15:17,581 It's kept as up to date as possible, as much as technology 304 00:15:17,581 --> 00:15:18,661 would allow you to do so. 305 00:15:19,861 --> 00:15:23,491 And one of the challenges with a hot site is, is latency. 306 00:15:23,851 --> 00:15:26,521 And so you might want to put the hot site 307 00:15:27,091 --> 00:15:27,661 as close 308 00:15:27,661 --> 00:15:32,461 as you can to, uh, the, the site that you're preparing 309 00:15:33,121 --> 00:15:36,361 and what happened on nine 11. 310 00:15:37,565 --> 00:15:43,895 So there were many companies to, for their hot site, uh, or had 311 00:15:43,895 --> 00:15:45,725 their main data center in one tower 312 00:15:45,965 --> 00:15:49,025 and had their hot site in another data in the other tower. 313 00:15:50,056 --> 00:15:52,906 which makes perfect sense. 314 00:15:54,316 --> 00:15:56,506 As long as nothing would take out both towers 315 00:15:56,765 --> 00:15:57,065 Yep. 316 00:15:58,426 --> 00:15:58,906 and. 317 00:15:59,671 --> 00:16:05,041 So unfortunately, as we know, you know, both towers, I mean, that, 318 00:16:05,461 --> 00:16:06,811 that's what I woke up to by the way. 319 00:16:06,811 --> 00:16:07,141 I, I 320 00:16:07,141 --> 00:16:13,471 woke up to my wife saying both of the, 'cause I'm on the west coast, so both 321 00:16:13,471 --> 00:16:15,001 of the towers had already collapsed 322 00:16:15,001 --> 00:16:16,801 as you know, as I was waking up. 323 00:16:17,306 --> 00:16:21,811 And so people lost or companies lost their. 324 00:16:22,411 --> 00:16:22,591 their. 325 00:16:22,591 --> 00:16:27,241 primary site and their hot site in, in the same moment. 326 00:16:27,601 --> 00:16:31,321 And so one of the things we learned, nine 11 is to make sure that you are, 327 00:16:32,371 --> 00:16:36,991 if you're doing some sort of hot site or warm site, is to put that site, I'm 328 00:16:36,991 --> 00:16:42,031 gonna say nowhere near, um, that the, the site that you're being protected. 329 00:16:42,511 --> 00:16:43,831 Now let's talk about that. 330 00:16:44,341 --> 00:16:48,631 Um, there were actually some attempts, um, I lived through 331 00:16:48,631 --> 00:16:50,041 this because I was working. 332 00:16:51,361 --> 00:16:58,391 For companies at the time, and that is there were attempts at regulation to say, 333 00:16:58,445 --> 00:17:01,655 If you're a bank or whatever you need to put, if you're financial training, 334 00:17:01,655 --> 00:17:03,305 you need to put a copy of your data. 335 00:17:04,145 --> 00:17:07,745 Uh, that's, that's hot over 200 miles away. 336 00:17:08,694 --> 00:17:08,934 Yep. 337 00:17:09,035 --> 00:17:10,325 was an attempt at regulation. 338 00:17:11,285 --> 00:17:12,395 What's the problem with that? 339 00:17:13,764 --> 00:17:20,874 Um, 200 miles away if you're, depending on the type of disaster isn't far enough. 340 00:17:22,224 --> 00:17:23,214 And so that's 341 00:17:23,375 --> 00:17:24,065 not the problem. 342 00:17:24,234 --> 00:17:25,704 But the second is latency. 343 00:17:26,255 --> 00:17:28,355 yeah, that's the problem, right? 344 00:17:28,355 --> 00:17:34,835 It's just simply not feasible because the, the, the round trip time, the 200 mile 345 00:17:34,835 --> 00:17:40,415 round trip time, uh, was just far too long that couldn't keep the data up to date. 346 00:17:41,139 --> 00:17:42,039 depending on the 347 00:17:42,425 --> 00:17:44,105 Based on the technology at the time, 348 00:17:44,229 --> 00:17:44,499 Yeah. 349 00:17:44,589 --> 00:17:45,789 Well, and I think it depends. 350 00:17:45,849 --> 00:17:47,409 So I do so. 351 00:17:48,159 --> 00:17:52,239 Many years after nine 11, I was working at a storage company and one of the 352 00:17:52,239 --> 00:17:57,249 things that they did was they also like talking to financial customers, right? 353 00:17:57,249 --> 00:17:59,739 Is many of 'em had what they would call dark fiber, 354 00:18:00,360 --> 00:18:00,580 Yep. 355 00:18:01,239 --> 00:18:01,509 right? 356 00:18:01,539 --> 00:18:06,009 Where they would basically run fiber optics, two fiber network between 357 00:18:06,009 --> 00:18:08,769 their two sites, and it would. 358 00:18:09,369 --> 00:18:09,789 You're right. 359 00:18:09,789 --> 00:18:14,229 It wouldn't be completely eliminate the latency, but it would definitely 360 00:18:14,229 --> 00:18:18,369 help versus say routing it over a public network of any type. 361 00:18:19,175 --> 00:18:20,855 Yeah, we definitely, you would. 362 00:18:20,915 --> 00:18:26,495 I, I think that there was an assumption of dark fiber, uh, at that point, but even 363 00:18:27,005 --> 00:18:32,105 200 miles on a straight piece of glass, speed of light has a speed of, speed of 364 00:18:32,105 --> 00:18:34,205 light is not, it's not instantaneous. 365 00:18:34,775 --> 00:18:36,515 It's whatever it is. 366 00:18:36,740 --> 00:18:40,460 187,000 miles a second or whatever, 367 00:18:40,793 --> 00:18:41,994 six hundred or something like 368 00:18:42,230 --> 00:18:43,040 something like that. 369 00:18:43,340 --> 00:18:43,670 Right. 370 00:18:44,180 --> 00:18:48,770 Um, the, the round trip time is gonna be measured in milliseconds. 371 00:18:48,770 --> 00:18:53,120 It's not gonna be, it's not, it's, it's, it's going to significantly increase 372 00:18:53,120 --> 00:18:58,965 latency, especially if we start talking about synchronous transfer of data. 373 00:18:59,240 --> 00:18:59,510 Right. 374 00:18:59,510 --> 00:19:01,340 Now let's talk about synchronous versus sacred. 375 00:19:01,700 --> 00:19:05,060 Synchronous versus asynchronous transfer of data. 376 00:19:05,240 --> 00:19:06,125 You want to give that a shot? 377 00:19:06,519 --> 00:19:07,959 Yeah, so synchronous is. 378 00:19:09,759 --> 00:19:13,059 Basically a right comes into your production site. 379 00:19:13,059 --> 00:19:15,279 It gets forwarded over to your DR site. 380 00:19:15,279 --> 00:19:18,159 The right gets, now there are different flavors, but typically the right gets 381 00:19:18,159 --> 00:19:21,819 committed on the DR site acknowledged back to the primary site, and then the 382 00:19:21,819 --> 00:19:25,539 primary site acknowledges the client during which, so you have to add up 383 00:19:25,539 --> 00:19:30,159 basically the latency of going over the writes on both sides, coming back before 384 00:19:30,159 --> 00:19:34,869 the client acknowledges, in which case you're guaranteed that that right has hit 385 00:19:34,869 --> 00:19:37,149 both sites and so the client can move on. 386 00:19:37,760 --> 00:19:40,910 Which is great as long as it doesn't take too long. 387 00:19:41,599 --> 00:19:42,009 Yes. 388 00:19:42,710 --> 00:19:48,410 So that's synchronous and asynchronous is something, is, is uh, different than that. 389 00:19:48,740 --> 00:19:51,770 And we, we've had some different, between you and I, we've had some 390 00:19:51,770 --> 00:19:54,620 different understandings of different kinds of asynchronous, but go ahead. 391 00:19:54,729 --> 00:19:59,739 Yeah, so for me, asynchronous is, well, what I would call semi synchronous, but 392 00:19:59,739 --> 00:20:04,869 that's a different case is where, uh, you accept the right on the production, 393 00:20:04,869 --> 00:20:09,399 you forward it over to the secondary or DR site, but, but while it's in 394 00:20:09,399 --> 00:20:12,699 process of being committed on the other side, you can acknowledge the client. 395 00:20:13,029 --> 00:20:14,559 So there is a lag. 396 00:20:15,269 --> 00:20:20,009 Um, you could decide how long that lag is, depending on technology and the vendor. 397 00:20:20,249 --> 00:20:23,879 Some allow you to specify at an IO level so you can say, I want 398 00:20:23,879 --> 00:20:25,589 10 transactions outstanding. 399 00:20:25,919 --> 00:20:28,289 Others allow you to do it in terms of seconds. 400 00:20:28,289 --> 00:20:31,859 So I'm allowing up to 10 seconds or 30 seconds, um, 401 00:20:32,120 --> 00:20:36,980 Before you start, before you start kicking back a performance issue to the client. 402 00:20:37,509 --> 00:20:37,869 Before 403 00:20:37,869 --> 00:20:38,709 you start Yeah. 404 00:20:38,799 --> 00:20:39,549 Putting back pressure. 405 00:20:39,549 --> 00:20:39,849 Now 406 00:20:40,569 --> 00:20:44,229 interestingly, there is a mode, which I'm not sure if you're aware of, 407 00:20:44,559 --> 00:20:48,669 uh, that some financial co companies requested, which is called Domino mode. 408 00:20:49,805 --> 00:20:50,295 Talk to me. 409 00:20:50,709 --> 00:20:55,089 So it's a form of synchronous replication where it, because in 410 00:20:55,089 --> 00:20:58,599 synchronous replication, typically you write to the client, it sends it over. 411 00:20:58,779 --> 00:21:02,049 If the right fails on the secondary, it'll still accept it on the primary. 412 00:21:02,649 --> 00:21:04,329 And acknowledge back to the client, right? 413 00:21:04,329 --> 00:21:08,799 So you're not guaranteed that it'll stop writes if it can't write to 414 00:21:08,799 --> 00:21:13,029 both sides At the same time, there's a mode called domino mode where if 415 00:21:13,029 --> 00:21:16,779 it can't write to both sides, it will not acknowledge the client. 416 00:21:17,049 --> 00:21:18,249 So that's why it's called a Domino. 417 00:21:18,249 --> 00:21:19,449 One takes out the other. 418 00:21:20,960 --> 00:21:22,295 I, I would think that, 419 00:21:24,425 --> 00:21:30,035 this is one of those, this is one of those things where, you know, uh, this 420 00:21:30,035 --> 00:21:32,290 is reality versus the idea, right? 421 00:21:32,550 --> 00:21:32,890 To me. 422 00:21:34,115 --> 00:21:36,725 domino mode that you described, that's synchronous. 423 00:21:36,995 --> 00:21:40,145 Anything other than that is not synchronous, right? 424 00:21:40,415 --> 00:21:43,655 And anything other than the domino mode that you described, 425 00:21:43,835 --> 00:21:45,755 I would call asynchronous, right? 426 00:21:46,235 --> 00:21:50,375 So these either synchronous, it's sort of like immutable and not immutable, right? 427 00:21:50,525 --> 00:21:52,445 It's either synchronous or it's not synchronous. 428 00:21:52,445 --> 00:21:57,095 And if it's synchronous, then it shouldn't acknowledge the right to the 429 00:21:57,095 --> 00:21:59,465 client until both writes have been done. 430 00:21:59,465 --> 00:22:01,505 And if one of them fails, then they're not done. 431 00:22:02,049 --> 00:22:02,619 yeah. 432 00:22:02,649 --> 00:22:05,739 So at least from most of the vendors I've seen, 433 00:22:05,769 --> 00:22:06,999 they've never implemented it that way. 434 00:22:07,595 --> 00:22:07,985 Yeah. 435 00:22:07,985 --> 00:22:08,585 That's interesting. 436 00:22:08,975 --> 00:22:09,635 Well, they're wrong. 437 00:22:10,085 --> 00:22:16,535 So, um, uh, yeah, so that was, so that was a lesson we thought we 438 00:22:16,535 --> 00:22:19,205 learned at the time, but we need to make sure we put it far enough away. 439 00:22:19,625 --> 00:22:21,815 But then they were like, it's gotta be synchronous and 440 00:22:21,845 --> 00:22:22,955 it's gotta be 200 miles away. 441 00:22:22,955 --> 00:22:24,365 They're like, eh, it's not gonna work. 442 00:22:24,365 --> 00:22:24,725 Right. 443 00:22:25,370 --> 00:22:29,450 Um, nowadays, you know, you hinted at it earlier, nowadays we would do this with 444 00:22:29,450 --> 00:22:33,830 the cloud and we can put it actually, because, you know, you, you did say that 445 00:22:34,100 --> 00:22:38,210 your first problem was that it wasn't far enough, and that's probably true, right? 446 00:22:38,840 --> 00:22:41,450 Because especially when we start talking about certain areas 447 00:22:41,450 --> 00:22:43,490 like Southern Florida, right? 448 00:22:43,790 --> 00:22:45,140 200 miles isn't far enough. 449 00:22:45,890 --> 00:22:50,540 Um, and um, so with the cloud, you can put it. 450 00:22:51,215 --> 00:22:52,115 Pretty much anywhere. 451 00:22:52,505 --> 00:22:55,805 Now, if we're going to do that, if, especially if we're gonna use 452 00:22:55,805 --> 00:23:00,305 public networks, we're pretty much going to have to use asynchronous 453 00:23:00,815 --> 00:23:02,345 of some sort, right? 454 00:23:02,705 --> 00:23:07,685 Uh, so we're gonna send the data and put it another place we're going to, 455 00:23:07,745 --> 00:23:10,625 you know, like you said, you can have a buffer, you can have a certain amount 456 00:23:10,625 --> 00:23:15,635 of time that it's allowed to get behind before it, uh, like you said, put, what 457 00:23:15,635 --> 00:23:16,955 do you mean when you say back pressure? 458 00:23:17,664 --> 00:23:22,944 So this is where you start to, um, elongate the time 459 00:23:22,944 --> 00:23:24,504 before acknowledging a right. 460 00:23:24,504 --> 00:23:24,984 So 461 00:23:25,314 --> 00:23:28,854 to the client, because typically your client will sort of throttle itself 462 00:23:29,184 --> 00:23:31,824 because at some point your latencies are gonna get into the seconds 463 00:23:31,824 --> 00:23:32,904 and they'll be like, no, no, no. 464 00:23:32,904 --> 00:23:33,834 Something's going on. 465 00:23:33,834 --> 00:23:34,374 I'll slow down. 466 00:23:35,080 --> 00:23:35,430 right. 467 00:23:35,589 --> 00:23:37,889 Because otherwise you're just gonna start dropping the writes. 468 00:23:38,710 --> 00:23:38,930 And. 469 00:23:39,965 --> 00:23:43,565 And, and this is a, a configuration choice on the part of the customer 470 00:23:44,015 --> 00:23:46,625 where they can say, I don't want to ever put back pressure. 471 00:23:46,655 --> 00:23:51,605 I wanna, uh, you know, that, that the data protection is less important 472 00:23:51,605 --> 00:23:53,105 than actually getting the job done. 473 00:23:53,165 --> 00:23:56,315 And then other clients would say, if I don't back it up, I don't 474 00:23:56,315 --> 00:23:57,515 wanna write it in the first place. 475 00:23:57,515 --> 00:23:57,755 Right. 476 00:23:58,385 --> 00:24:02,465 Um, I, I, obviously, I tend to be more towards the latter than the former. 477 00:24:03,024 --> 00:24:03,234 Yeah. 478 00:24:03,324 --> 00:24:07,194 Uh, from, uh, most of the companies I've seen are vendors. 479 00:24:07,554 --> 00:24:07,854 Yeah. 480 00:24:07,854 --> 00:24:09,534 They're not in line with you. 481 00:24:11,690 --> 00:24:14,840 Meaning Meaning that they would just go ahead and do it anyway? 482 00:24:14,904 --> 00:24:17,214 Yeah, because most customers Right. 483 00:24:17,274 --> 00:24:17,694 Unless you 484 00:24:17,694 --> 00:24:19,614 have very, very strict regulations. 485 00:24:20,094 --> 00:24:20,304 Right. 486 00:24:20,304 --> 00:24:21,474 They're like, best effort 487 00:24:22,550 --> 00:24:22,760 Yeah. 488 00:24:22,760 --> 00:24:23,720 I, I think it's 489 00:24:23,844 --> 00:24:24,414 because they will 490 00:24:24,710 --> 00:24:27,350 that I care about data protection more than the average 491 00:24:28,100 --> 00:24:28,610 person, 492 00:24:30,219 --> 00:24:32,739 Because the thing is, at some point it will catch back up. 493 00:24:32,739 --> 00:24:33,429 Hopefully, 494 00:24:33,793 --> 00:24:34,753 Hopefully yes. 495 00:24:34,903 --> 00:24:36,253 Depending on what the problem was. 496 00:24:37,089 --> 00:24:37,569 right. 497 00:24:37,929 --> 00:24:42,879 Um, and, and again, as long as we're okay with the potential, 498 00:24:43,599 --> 00:24:48,849 right, um, uh, I, I would think that there should still be some number. 499 00:24:50,259 --> 00:24:54,969 number might be measured in hours if we're hours behind updating 500 00:24:54,969 --> 00:24:57,279 our other copy, something. 501 00:24:58,044 --> 00:25:00,329 Might be drastically wrong that we need to look at. 502 00:25:00,718 --> 00:25:01,078 Yep. 503 00:25:01,498 --> 00:25:05,818 And the other thing to also mention is with a lot of this. 504 00:25:06,373 --> 00:25:07,273 High end. 505 00:25:07,333 --> 00:25:10,213 I normally refer to it as tier one storage systems 506 00:25:10,279 --> 00:25:10,569 Yeah, 507 00:25:11,263 --> 00:25:14,383 because these are tier one applications with very strict requirements. 508 00:25:14,773 --> 00:25:20,203 Um, usually also they provide the ability to do automatic failover. 509 00:25:20,713 --> 00:25:24,073 So it's kind of, think of it like high availability plus clustering. 510 00:25:24,714 --> 00:25:25,044 right. 511 00:25:25,423 --> 00:25:29,653 So if you take a look at a lot of the tier one storage, right, they might have two 512 00:25:29,653 --> 00:25:34,933 storage systems in both locations with the drives that are all interconnected. 513 00:25:34,933 --> 00:25:39,613 So in case one unit fails, the other unit can take over the diss of the other side. 514 00:25:39,943 --> 00:25:43,423 Um, the clients are also connected to both sides, so they don't 515 00:25:43,423 --> 00:25:45,103 have to worry about failing over. 516 00:25:45,853 --> 00:25:48,793 Um, I can't remember what it was called. 517 00:25:49,093 --> 00:25:52,603 It's like the optimized and non-optimized connectivity for fiber channel, 518 00:25:52,933 --> 00:25:55,813 which allows it to have a preferred path and a non-preferred path. 519 00:25:55,813 --> 00:25:56,383 So you still 520 00:25:56,383 --> 00:26:00,883 have connectivity, so your clients will automatically fail over, so you 521 00:26:00,883 --> 00:26:04,453 don't have to do anything, and so your writes can still continue to happen. 522 00:26:05,049 --> 00:26:05,439 Yeah. 523 00:26:06,909 --> 00:26:11,529 So that's a big thing, is like, you know, we, we, we learned that 524 00:26:12,009 --> 00:26:13,479 we should have it farther away. 525 00:26:13,719 --> 00:26:17,559 We learned that maybe we shouldn't have it too far away, but, but, 526 00:26:17,619 --> 00:26:19,179 but now with the, with the. 527 00:26:19,809 --> 00:26:24,039 With the cloud, we can potentially have it pretty much anywhere, but we 528 00:26:24,039 --> 00:26:27,459 definitely have to rely on some sort of asynchronous, uh, uh, communication. 529 00:26:27,839 --> 00:26:32,214 When, I think about our, our friends that came on the show. 530 00:26:32,904 --> 00:26:35,424 Talk about their experiences with disaster. 531 00:26:35,424 --> 00:26:39,684 I think there's some, that was a major, that was a hurricane 532 00:26:39,684 --> 00:26:40,824 that took out an island 533 00:26:41,064 --> 00:26:43,824 and they had multiple data centers on that island. 534 00:26:43,824 --> 00:26:46,164 One was more in the high ground than the other. 535 00:26:46,164 --> 00:26:47,904 So one was flooded, the other was not. 536 00:26:48,144 --> 00:26:50,454 And they were gonna recover from one data center to the other. 537 00:26:50,454 --> 00:26:53,004 And that's, and they had everything they needed there. 538 00:26:53,364 --> 00:26:53,659 They did. 539 00:26:53,754 --> 00:26:54,744 They needed personnel. 540 00:26:54,744 --> 00:26:56,934 They had to fly people to the island. 541 00:26:56,984 --> 00:27:00,014 But other things happened that we can also learn from. 542 00:27:00,014 --> 00:27:01,244 Do you, what do you remember from those? 543 00:27:02,538 --> 00:27:05,508 So from, so there were two things I remember. 544 00:27:05,508 --> 00:27:06,198 One was sort of. 545 00:27:07,023 --> 00:27:10,983 The people process stuff, which I think we rarely think about. 546 00:27:11,433 --> 00:27:13,323 And then the other was the technology piece. 547 00:27:13,713 --> 00:27:18,573 So from a people process perspective, it was, do you have the right people in 548 00:27:18,573 --> 00:27:20,043 country who have the expertise and the 549 00:27:20,043 --> 00:27:21,153 skills to 550 00:27:21,153 --> 00:27:21,813 recover? 551 00:27:22,113 --> 00:27:23,583 Where are they going to sleep? 552 00:27:23,583 --> 00:27:25,053 Where are they going to get food? 553 00:27:25,053 --> 00:27:25,413 Right? 554 00:27:25,653 --> 00:27:28,113 All the things that you kind of take for granted, right? 555 00:27:28,143 --> 00:27:29,013 They had none of that. 556 00:27:29,013 --> 00:27:30,393 Like how do they communicate 557 00:27:31,004 --> 00:27:31,274 Yeah. 558 00:27:31,274 --> 00:27:36,374 He, as I recall, he, he turned a, a, um, a conference room into a 559 00:27:36,974 --> 00:27:37,814 hotel room 560 00:27:37,874 --> 00:27:38,294 Right. 561 00:27:38,324 --> 00:27:41,144 And slept on a cot and ate rice and beans for two 562 00:27:41,144 --> 00:27:41,534 weeks. 563 00:27:41,643 --> 00:27:42,423 And he was lucky to 564 00:27:42,423 --> 00:27:43,623 get rice and beans right? 565 00:27:43,623 --> 00:27:45,318 Because there were a lot of people who didn't even get that. 566 00:27:45,914 --> 00:27:46,394 Yeah. 567 00:27:46,784 --> 00:27:47,534 Um. 568 00:27:48,359 --> 00:27:52,709 Yeah, that's, that is definitely a, a lesson learned is that, you 569 00:27:52,709 --> 00:27:57,239 know, make sure to take the human element into your DR design. 570 00:27:57,449 --> 00:27:57,869 Right. 571 00:27:58,169 --> 00:28:04,259 Um, and the one that, the one that stands out to me was 572 00:28:04,289 --> 00:28:09,149 the reliance on the mainland. 573 00:28:09,308 --> 00:28:09,598 Yeah. 574 00:28:10,319 --> 00:28:10,799 Right. 575 00:28:10,829 --> 00:28:15,749 That when you have a true disaster, whether it's on an island or just. 576 00:28:16,274 --> 00:28:23,384 You know, wherever you might not be able to get connectivity to the rest 577 00:28:23,384 --> 00:28:24,974 of your computing infrastructure. 578 00:28:25,064 --> 00:28:30,074 And in this case, their authentication and authorization, their IAM system 579 00:28:31,094 --> 00:28:34,664 relied on active directory, which 580 00:28:34,814 --> 00:28:37,304 all of which was in, was in the mainland. 581 00:28:37,934 --> 00:28:45,374 And um, so this is just, you know, the lesson there is to just make sure that. 582 00:28:45,374 --> 00:28:47,714 To just take that into consideration, right. 583 00:28:47,954 --> 00:28:52,034 Just the, the realize that in a real disaster, you may be very 584 00:28:52,034 --> 00:28:55,214 isolated from the rest, rest of the world, and you need to take 585 00:28:55,214 --> 00:28:57,314 that into consideration in your DR. 586 00:28:57,314 --> 00:28:57,794 Design. 587 00:28:58,188 --> 00:28:58,398 Yeah. 588 00:28:58,878 --> 00:29:03,168 And, but I think this becomes so difficult, right, Curtis, because 589 00:29:03,858 --> 00:29:07,428 like how many scenarios are you gonna play in your head and how much time 590 00:29:07,428 --> 00:29:08,868 are you gonna focus on some of these? 591 00:29:08,868 --> 00:29:12,438 Now granted, a hurricane on a tropical island is probably 592 00:29:12,438 --> 00:29:13,518 a high likelihood event, 593 00:29:14,879 --> 00:29:15,629 Yes. 594 00:29:16,049 --> 00:29:16,619 Right. 595 00:29:16,829 --> 00:29:17,609 Well, I, 596 00:29:17,748 --> 00:29:18,288 of that, but. 597 00:29:18,539 --> 00:29:23,999 well, I, I, I think it's, I think it's a totally, I think that there's two 598 00:29:23,999 --> 00:29:26,789 things that you cannot count on, right. 599 00:29:27,359 --> 00:29:31,514 Um, I. and and I, I would just say one thing that covers the number of 600 00:29:31,514 --> 00:29:33,194 things, and that is utilities, right? 601 00:29:33,434 --> 00:29:35,114 You cannot count on the internet. 602 00:29:35,114 --> 00:29:36,614 You cannot count on power. 603 00:29:37,004 --> 00:29:40,754 You cannot count on, um, you know, uh, water, 604 00:29:41,264 --> 00:29:41,684 right? 605 00:29:41,744 --> 00:29:43,514 You can't count on those three things. 606 00:29:43,514 --> 00:29:47,174 And so I, I'm just saying, I don't, I don't think you, you don't need 607 00:29:47,174 --> 00:29:51,194 to think of all of the reasons that one or more of those might not. 608 00:29:51,663 --> 00:29:51,963 Yeah. 609 00:29:52,574 --> 00:29:53,354 available. 610 00:29:53,684 --> 00:29:55,904 You just need to plan for them not being available. 611 00:29:56,114 --> 00:29:58,694 If you don't have internet, if you don't have whatever it is you, 612 00:29:58,724 --> 00:30:03,434 however you communicate between your sites, you are going to be isolated. 613 00:30:03,434 --> 00:30:06,914 If you don't have power, you're gonna need to supply your own power, 614 00:30:06,914 --> 00:30:07,364 right? 615 00:30:07,424 --> 00:30:10,244 Um, these are just things that you can think, you can 616 00:30:10,244 --> 00:30:13,244 think through these things, and these are all things that you can say either. 617 00:30:13,664 --> 00:30:15,494 I'm just saying, have that discussion. 618 00:30:16,004 --> 00:30:18,734 And say, you know what, if we don't have power, we're not gonna do anything. 619 00:30:18,734 --> 00:30:18,974 We're not 620 00:30:18,974 --> 00:30:22,094 gonna spend $15 billion on power generators 621 00:30:22,233 --> 00:30:22,383 Yeah, 622 00:30:23,174 --> 00:30:26,954 in case we, you know, we might, or, or you might be subject 623 00:30:26,954 --> 00:30:28,154 to regulations, or you might 624 00:30:28,154 --> 00:30:31,814 have, uh, financial reasons why downtime is enough. 625 00:30:31,814 --> 00:30:35,444 That, or the cost of downtime is enough that you're gonna pay for generators. 626 00:30:36,104 --> 00:30:39,014 Um, do you remember how they got internet over there? 627 00:30:40,203 --> 00:30:41,043 It was satellite. 628 00:30:41,534 --> 00:30:42,074 Yeah. 629 00:30:42,284 --> 00:30:42,824 satellite. 630 00:30:42,824 --> 00:30:45,824 I like this idea of making sure that you think about what would 631 00:30:45,824 --> 00:30:49,874 you do if the utilities that you're normally counting on are not 632 00:30:49,874 --> 00:30:50,294 available? 633 00:30:50,294 --> 00:30:52,634 If you get completely isolated, what would you do? 634 00:30:53,534 --> 00:30:56,144 There are a number of reasons why that might end up being the case. 635 00:30:56,793 --> 00:30:59,948 or the, or the people that you normally depend on are not available. 636 00:31:00,704 --> 00:31:01,304 Yes. 637 00:31:01,754 --> 00:31:03,974 This is why we had this little thing called documentation. 638 00:31:04,323 --> 00:31:04,533 Yeah. 639 00:31:06,254 --> 00:31:06,944 Um, 640 00:31:07,173 --> 00:31:07,533 And, 641 00:31:07,683 --> 00:31:08,013 and 642 00:31:08,744 --> 00:31:09,164 go ahead. 643 00:31:09,603 --> 00:31:15,123 and hopefully I know when we've had Mike, Dr. Mike on the podcast, right? 644 00:31:15,183 --> 00:31:19,833 Um, he's talked about sort of doing tabletop exercises, right? 645 00:31:19,833 --> 00:31:23,793 So have you done a tabletop exercise, which is like, Hey, what happens if the 646 00:31:23,793 --> 00:31:27,693 main IT person, Susie, is unavailable? 647 00:31:28,244 --> 00:31:29,594 Right, right. 648 00:31:30,434 --> 00:31:31,364 And, and that's what you do. 649 00:31:31,364 --> 00:31:33,014 You work through those various scenarios. 650 00:31:33,014 --> 00:31:36,674 You, you hire somebody to come in as an outsider is the best way to do that. 651 00:31:37,169 --> 00:31:42,599 When we start talking about ransomware, um, Dr. Mike Sailor's company would, 652 00:31:42,659 --> 00:31:44,399 would, would be a great resource for that. 653 00:31:45,219 --> 00:31:47,019 It's good to use a very negative person. 654 00:31:47,289 --> 00:31:49,809 Think the most negative person in your environment, the most 655 00:31:49,809 --> 00:31:51,369 pess, pessimistic person. 656 00:31:51,729 --> 00:31:53,139 And, uh, no, no. 657 00:31:53,139 --> 00:31:54,399 Pessimist thinks they're negative. 658 00:31:54,399 --> 00:31:55,869 They, they think they're, they're realists, 659 00:31:56,038 --> 00:31:56,638 will realize 660 00:31:56,979 --> 00:31:58,239 Um, yeah. 661 00:31:58,749 --> 00:32:03,999 Sort of the final thought that I'm thinking is, and, and I, I remember. 662 00:32:04,719 --> 00:32:08,764 Doing a disaster recovery of my own, and that is, I did it where. 663 00:32:09,714 --> 00:32:12,474 I had not yet tested the throughput of the 664 00:32:12,474 --> 00:32:15,504 backup system that we were doing and the throughput of the backup system we 665 00:32:15,504 --> 00:32:17,364 were doing, it turned out to be crap. 666 00:32:18,144 --> 00:32:18,654 Right? 667 00:32:18,714 --> 00:32:22,764 And as a result, uh, that was a really hard day. 668 00:32:22,764 --> 00:32:24,504 And in Curtis land, right? 669 00:32:25,104 --> 00:32:25,764 This is 670 00:32:25,794 --> 00:32:27,954 very early in my career, I learned it. 671 00:32:27,984 --> 00:32:28,524 What's that? 672 00:32:28,734 --> 00:32:30,174 Yes, it was the compression thing. 673 00:32:30,294 --> 00:32:30,684 Yeah. 674 00:32:31,254 --> 00:32:34,014 So make sure that when, when you make configuration changes to your 675 00:32:34,014 --> 00:32:36,954 backup system or your DR system, make sure that you test that. 676 00:32:36,954 --> 00:32:43,464 Just realize that in general, restore speed is slower than backup speed. 677 00:32:43,734 --> 00:32:44,514 It just is. 678 00:32:44,544 --> 00:32:48,684 It's the way it, you know it, you know, back in the day with tape, it 679 00:32:48,684 --> 00:32:50,394 was because we were doing multiplexing. 680 00:32:51,474 --> 00:32:53,554 Nowadays it would would with dedupe. 681 00:32:53,634 --> 00:32:54,844 It's because of dedupe. 682 00:32:55,719 --> 00:32:58,899 Um, and so just sort of plan for that. 683 00:32:58,899 --> 00:33:02,769 But don't, don't just assume how much slower it is, uh, test it and see how much 684 00:33:02,769 --> 00:33:06,759 slower it is and make sure you figure that into the, to the disaster recovery plan. 685 00:33:08,529 --> 00:33:14,019 But, um, so, um, with that, I think that's enough for now in terms of lessons learned 686 00:33:14,019 --> 00:33:16,989 from just, you know, the pains of others. 687 00:33:17,319 --> 00:33:17,829 Um. 688 00:33:18,444 --> 00:33:23,514 Just think of the scenarios that might be, you know, that you might be subject to. 689 00:33:23,784 --> 00:33:27,024 Um, make sure you've got at least one copy of your data 690 00:33:27,054 --> 00:33:29,244 that's far away from everything. 691 00:33:29,754 --> 00:33:35,394 Um, and the way to probably do that today is cloud taped can still play a role. 692 00:33:35,424 --> 00:33:38,424 In fact, our previous episode we talk about why tape is 693 00:33:38,424 --> 00:33:39,774 still not dead in backup. 694 00:33:40,134 --> 00:33:42,414 Uh, it certainly is on life support, but there 695 00:33:42,414 --> 00:33:43,044 is, you know. 696 00:33:43,404 --> 00:33:47,064 There is, there is a use for tape and backup and it's disaster recovery. 697 00:33:48,084 --> 00:33:52,314 So, um, um, especially when we start talking about disaster 698 00:33:52,314 --> 00:33:54,324 recovery from a ransomware attack. 699 00:33:54,984 --> 00:33:58,644 So, well, thanks for chatting again, my friend. 700 00:33:59,053 --> 00:34:00,073 No, it was good. 701 00:34:00,133 --> 00:34:04,003 Uh, one thing I'm surprised you, I thought you were gonna mention, but you did not, 702 00:34:04,374 --> 00:34:05,119 The 3, 2, 1 rule. 703 00:34:05,213 --> 00:34:05,623 1 rule. 704 00:34:07,753 --> 00:34:08,473 You were just 705 00:34:08,844 --> 00:34:10,224 You know what's funny is we were so close. 706 00:34:10,284 --> 00:34:11,154 We were so close. 707 00:34:11,154 --> 00:34:11,514 Yeah. 708 00:34:11,544 --> 00:34:11,754 Yeah. 709 00:34:11,754 --> 00:34:14,334 The whole 3, 2, 1 rule with, you know, three copies of your data on two 710 00:34:14,334 --> 00:34:15,834 different media, one of which is offsite. 711 00:34:15,834 --> 00:34:18,774 That, that, that's a core design concept for anything. 712 00:34:19,164 --> 00:34:23,724 Uh, backup in dr. And, and that's, yeah, you're right. 713 00:34:23,724 --> 00:34:25,959 We never, we, we didn't mention, but now we have, so you 714 00:34:25,959 --> 00:34:27,834 have corrected our oversight. 715 00:34:27,924 --> 00:34:28,854 Thank you very much. 716 00:34:29,094 --> 00:34:31,134 And thank you to our listeners. 717 00:34:31,164 --> 00:34:32,454 We'd be nothing without you. 718 00:34:32,904 --> 00:34:34,044 That is a wrap. 719 00:34:36,954 --> 00:34:41,634 The backup wrap up is written, recorded, and produced by me w Curtis Preston. 720 00:34:42,204 --> 00:34:47,004 If you need backup or Dr. Consulting content generation or expert witness 721 00:34:47,004 --> 00:34:49,584 work, check out backup central.com. 722 00:34:50,304 --> 00:34:53,424 You can also find links from my O'Reilly Books on the same website. 723 00:34:54,084 --> 00:34:58,074 Remember, this is an independent podcast and any opinions that 724 00:34:58,074 --> 00:35:01,974 you hear are those of the speaker and not necessarily an employer. 725 00:35:02,904 --> 00:35:03,594 Thanks for listening.