1 00:00:00,690 --> 00:00:04,710 You've found the backup wrap-up your go-to podcast for all things, 2 00:00:04,710 --> 00:00:07,110 backup recovery and cyber recovery. 3 00:00:07,500 --> 00:00:10,950 And this episode, we share a fascinating disaster recovery case 4 00:00:10,950 --> 00:00:12,900 study that will blow your mind. 5 00:00:13,230 --> 00:00:16,260 I'm talking with someone who had to do an actual Dr. 6 00:00:16,470 --> 00:00:19,860 In the worst possible circumstances on an island. 7 00:00:19,950 --> 00:00:22,650 After a hurricane took out everything. 8 00:00:23,040 --> 00:00:26,700 And when I say everything, I mean, everything, the data center, the 9 00:00:26,700 --> 00:00:28,620 infrastructure, even the trees. 10 00:00:28,920 --> 00:00:33,240 He had to sleep on an air mattress, eat chicken and rice for two weeks and 11 00:00:33,240 --> 00:00:37,740 figure out how to restore systems when basic assumptions like we have internet. 12 00:00:37,950 --> 00:00:39,060 Weren't true anymore. 13 00:00:39,630 --> 00:00:41,400 If you've ever wondered what a real Dr. 14 00:00:41,400 --> 00:00:43,950 Situation looks like, you're about to find out. 15 00:00:44,280 --> 00:00:48,630 Plus we get into some serious discussions about backup strategy and why. 16 00:00:48,630 --> 00:00:49,980 Assuming things will work. 17 00:00:50,160 --> 00:00:51,720 Isn't the same as testing them. 18 00:00:52,080 --> 00:00:54,330 Trust me, you don't want to miss this episode. 19 00:00:54,780 --> 00:00:59,130 By the way it is a classic episode from four years ago, it was an amazing story. 20 00:00:59,130 --> 00:01:01,080 Then it's a great story now. 21 00:01:01,290 --> 00:01:03,480 And so we're bringing it to you during the holidays. 22 00:01:03,510 --> 00:01:04,440 I hope you enjoy it. 23 00:01:20,283 --> 00:01:24,355 Welcome to the show . I'm your host, w Curtis Preston, AKA, Mr. 24 00:01:24,355 --> 00:01:27,655 Backup, and I have with me my, my meat advisor. 25 00:01:27,655 --> 00:01:28,655 Prasanna Malaiyandi. 26 00:01:28,675 --> 00:01:29,095 How's it going? 27 00:01:29,095 --> 00:01:29,455 Persona, 28 00:01:29,995 --> 00:01:31,405 I don't know if I'm your meat advisor. 29 00:01:31,405 --> 00:01:32,065 I think I am. 30 00:01:32,065 --> 00:01:33,235 Your, uh. 31 00:01:33,865 --> 00:01:35,515 Uh, what do they say? 32 00:01:35,515 --> 00:01:39,205 Sort of your apprentice or wishing to be your apprentice. 33 00:01:39,235 --> 00:01:40,225 You're, you're, yeah. 34 00:01:40,225 --> 00:01:43,405 You're not my, I'm guessing you, you have a lot of vegetarian 35 00:01:43,405 --> 00:01:44,635 dishes in your house, right? 36 00:01:44,695 --> 00:01:45,295 Yes, 37 00:01:45,295 --> 00:01:47,785 we do have a, my wife is vegetarian and Right. 38 00:01:47,785 --> 00:01:52,135 Typically we do vegetarian, and given that we eat a lot of Indian food, Indian 39 00:01:52,135 --> 00:01:55,045 food, you can make a ton of dishes without ever having to make anything meat. 40 00:01:55,405 --> 00:01:57,205 So there's a lot of variety. 41 00:01:57,710 --> 00:01:58,220 Yeah. 42 00:01:58,280 --> 00:02:02,510 Uh, so, so you're more sort of an, you're, you're, you're my meat enthusiast. 43 00:02:02,570 --> 00:02:02,990 Yes. 44 00:02:02,990 --> 00:02:03,140 How's 45 00:02:03,140 --> 00:02:03,440 that? 46 00:02:03,440 --> 00:02:05,540 You, you, you're meat curious. 47 00:02:05,630 --> 00:02:05,990 Yes. 48 00:02:05,990 --> 00:02:08,000 But meat curious. 49 00:02:09,290 --> 00:02:14,540 But I do want to hear about your latest adventure, because I've been, I've 50 00:02:14,540 --> 00:02:18,050 been asking you, I think almost every week how your dry aging is going. 51 00:02:18,110 --> 00:02:20,030 So I know that this will probably come out later, but now that. 52 00:02:20,390 --> 00:02:22,520 Technically we're recording this after Thanksgiving. 53 00:02:22,730 --> 00:02:24,830 I wanna know how was it and what happened. 54 00:02:24,860 --> 00:02:28,040 So I'm gonna, I'm gonna bring on our guest and then, uh, and then 55 00:02:28,040 --> 00:02:29,480 we'll, we'll chitchat about that. 56 00:02:29,960 --> 00:02:36,890 So, uh, we have, uh, it's a rare treat for us because I've been in it for so long. 57 00:02:36,920 --> 00:02:42,020 Rarely do I have someone who's been in it longer than me, and this is 58 00:02:42,020 --> 00:02:45,770 one of those times and I'm super excited because, uh, he started in 59 00:02:45,770 --> 00:02:48,350 it just after I was in high school. 60 00:02:48,815 --> 00:02:52,775 And, uh, started out in the hardware, uh, side of things, actually working 61 00:02:53,015 --> 00:02:56,675 and running Digital Equipment Corporation, which we call Deck. 62 00:02:56,915 --> 00:03:02,405 Uh, their internal email service went into a, uh, it, uh, has done a lot 63 00:03:02,405 --> 00:03:05,555 of things in data center operations, system administrator, data center 64 00:03:05,555 --> 00:03:08,105 manager, and he's recently retired. 65 00:03:08,165 --> 00:03:09,155 Super jelly. 66 00:03:09,230 --> 00:03:11,570 And, and now lives in Seattle. 67 00:03:11,600 --> 00:03:16,880 He is a friend of the person that we previously had on that we called 68 00:03:16,880 --> 00:03:18,710 Harry Potter to keep him anonymous. 69 00:03:18,710 --> 00:03:22,670 And so to continue that tradition, I would like to introduce to 70 00:03:22,670 --> 00:03:25,250 the podcast, uh, Ron Weasley. 71 00:03:25,880 --> 00:03:26,030 Hello. 72 00:03:26,330 --> 00:03:27,410 It's a pleasure to be 73 00:03:27,410 --> 00:03:27,740 here. 74 00:03:28,220 --> 00:03:30,655 You know, we had, we had your friend Harry Yes. 75 00:03:31,670 --> 00:03:36,290 On and um, and, and we, we had a great podcast there. 76 00:03:36,290 --> 00:03:40,070 But, um, I, I, I, I kept asking him, I was like, do you think that the guy that 77 00:03:40,070 --> 00:03:43,790 actually was the one with the fingers on the keyboard would, would talk to us? 78 00:03:43,790 --> 00:03:44,870 And he said, yes. 79 00:03:44,870 --> 00:03:45,920 And so here you are. 80 00:03:46,250 --> 00:03:47,150 Uh, but. 81 00:03:47,780 --> 00:03:52,820 So the, before we get to that, uh, we'll get back to the, to the meat conversation. 82 00:03:53,090 --> 00:04:01,880 So Ron, I have had a, a, an ongoing sort of a project of experimenting 83 00:04:01,880 --> 00:04:03,860 with dry aging meat at home. 84 00:04:04,250 --> 00:04:09,650 And I started out with these things called the umai bags, which UMAI, 85 00:04:09,650 --> 00:04:12,980 it's short for umami and, um. 86 00:04:13,690 --> 00:04:20,290 The, and so Thanksgiving was the first time I did a dry aged brisket at home. 87 00:04:20,380 --> 00:04:23,800 I, I think the dry aging process went really well. 88 00:04:23,980 --> 00:04:27,460 I didn't quite get, and by the way, if you are a brisket fan, uh, just 89 00:04:27,460 --> 00:04:32,110 go to YouTube and type in dry aged brisket and you'll see why I was 90 00:04:32,110 --> 00:04:33,550 interested in dry aging briskets. 91 00:04:33,580 --> 00:04:34,630 'cause the weird thing is that. 92 00:04:35,490 --> 00:04:39,600 A lot of people in the dry aging slash brisket community don't think 93 00:04:39,600 --> 00:04:43,290 that briskets benefit from dry aging, but these videos suggest otherwise. 94 00:04:43,290 --> 00:04:44,490 So I tried it out. 95 00:04:44,910 --> 00:04:51,000 The problem was that I had a, uh, a noon Thanksgiving we had for those 96 00:04:51,000 --> 00:04:56,130 concerned, we had a sort of COD friendly Thanksgiving gathering, so we had. 97 00:04:56,180 --> 00:04:57,380 We had 10 people. 98 00:04:57,380 --> 00:04:58,580 We did it outside. 99 00:04:58,850 --> 00:04:59,990 We were socially distanced. 100 00:04:59,990 --> 00:05:03,080 I actually rented tables and chairs so that I could do that and I, so that we 101 00:05:03,080 --> 00:05:04,400 could, you know, follow all the rules. 102 00:05:04,700 --> 00:05:08,840 Uh, but it was at noon and I had, so that meant I had to start my 103 00:05:08,840 --> 00:05:13,310 brisket at midnight and, um, it meant that part of the brisket. 104 00:05:13,310 --> 00:05:13,580 Well, 105 00:05:13,880 --> 00:05:14,870 you're very dedicated. 106 00:05:15,410 --> 00:05:15,680 Yeah. 107 00:05:15,710 --> 00:05:17,990 Most brisket cooking was while I was sleeping. 108 00:05:18,940 --> 00:05:23,680 And, uh, let's just say the critical part is towards the end when you 109 00:05:23,680 --> 00:05:25,120 need to be checking doneness. 110 00:05:25,180 --> 00:05:29,890 And I was really not wanting to get up at five o'clock in the morning 111 00:05:29,890 --> 00:05:31,270 when this thing was really done. 112 00:05:31,960 --> 00:05:33,730 And so I kind of got up at six. 113 00:05:33,730 --> 00:05:36,040 And the difference between getting up at five and getting up at six 114 00:05:36,040 --> 00:05:38,530 is the difference between a brisket that is tender and a brisket that. 115 00:05:38,600 --> 00:05:43,040 Becomes pot roast and unfortunately, I blew my 60 day experiment. 116 00:05:43,820 --> 00:05:43,850 Mm. 117 00:05:43,850 --> 00:05:44,390 For an extra 118 00:05:44,390 --> 00:05:45,140 hour's sleep. 119 00:05:45,230 --> 00:05:49,310 So I had a, the brisket was super, super tender, super, super juicy. 120 00:05:49,310 --> 00:05:53,870 EAs easily the juiciest brisket I've ever cooked, but it was slightly overdone 121 00:05:54,170 --> 00:05:57,950 and so I was disappointed in it as the brisket maker, the people that. 122 00:05:58,695 --> 00:05:59,265 Loved it. 123 00:05:59,745 --> 00:06:02,895 Um, and so I had no complaints. 124 00:06:02,895 --> 00:06:04,545 I also had no brisket left. 125 00:06:04,605 --> 00:06:07,065 Yeah, that's when, you know, it was good when 126 00:06:07,065 --> 00:06:07,755 we were done. 127 00:06:08,415 --> 00:06:11,895 Uh, I think I grabbed a handful of it, uh, just so that I 128 00:06:11,895 --> 00:06:13,455 could, you know, eat it later. 129 00:06:13,455 --> 00:06:17,115 But yeah, we, we had a, like a 16 pound brisket that was 130 00:06:17,115 --> 00:06:19,125 completely gone from 10 people. 131 00:06:19,840 --> 00:06:22,660 Who also had a Turkey and a ham to eat. 132 00:06:22,960 --> 00:06:24,820 So, you know, I don't know. 133 00:06:24,820 --> 00:06:25,720 Was that a success? 134 00:06:25,720 --> 00:06:26,095 I, I just 135 00:06:26,095 --> 00:06:29,230 did a quick lookup of it, six to eight weeks. 136 00:06:29,230 --> 00:06:31,420 It's saying to, to do that, that's, 137 00:06:32,110 --> 00:06:32,590 yeah. 138 00:06:32,590 --> 00:06:32,655 A lot of it's time. 139 00:06:32,655 --> 00:06:32,935 It's 140 00:06:32,940 --> 00:06:33,460 dedication, 141 00:06:33,460 --> 00:06:33,850 right? 142 00:06:33,855 --> 00:06:34,015 Yeah. 143 00:06:34,015 --> 00:06:34,455 Dedication. 144 00:06:34,460 --> 00:06:34,870 Yeah. 145 00:06:34,870 --> 00:06:35,290 Really. 146 00:06:35,290 --> 00:06:35,560 Yeah. 147 00:06:35,980 --> 00:06:40,630 So I'm current, I'm currently in the process of an actual dry aging experiment 148 00:06:40,630 --> 00:06:42,730 where I have a dedicated refrigerator. 149 00:06:43,185 --> 00:06:46,695 With, uh, precise temperature and humidity control that's going 150 00:06:46,695 --> 00:06:48,435 on, like literally right now. 151 00:06:48,435 --> 00:06:52,755 It started December 1st, and I'm hoping to have the results of 152 00:06:52,755 --> 00:06:57,285 that by, by New Year's and have a New Year's, uh, dry aged brisket. 153 00:06:57,795 --> 00:07:00,465 But, um, but we don't know about that yet, so, um. 154 00:07:00,970 --> 00:07:02,530 We'll talk about that on later podcast. 155 00:07:02,580 --> 00:07:07,650 Ron, you listened to the podcast where we talked about you, right? 156 00:07:08,250 --> 00:07:08,460 Yes. 157 00:07:08,670 --> 00:07:09,210 With with Harry. 158 00:07:09,210 --> 00:07:09,570 Yes. 159 00:07:10,230 --> 00:07:10,350 Yes. 160 00:07:10,410 --> 00:07:15,270 And the, the idea was that there was a hurricane. 161 00:07:15,495 --> 00:07:19,275 That took out, so this actually happened, by the way, this is, this is, you 162 00:07:19,275 --> 00:07:22,155 know, uh, this, this is a true story. 163 00:07:22,900 --> 00:07:23,120 Yes. 164 00:07:24,075 --> 00:07:26,535 The, so a hurricane took out an island. 165 00:07:26,835 --> 00:07:28,545 An island took out an island. 166 00:07:29,235 --> 00:07:33,885 You and Harry both worked for, you know, we'll call it Hogwarts. 167 00:07:34,185 --> 00:07:36,195 Someone had to go down there. 168 00:07:37,530 --> 00:07:39,420 Do the disaster recovery. 169 00:07:39,780 --> 00:07:44,040 My understanding is that it, it was sort of a toss up between you and Harry. 170 00:07:44,040 --> 00:07:48,330 And Harry couldn't get there fast enough and you could, and so you 171 00:07:48,330 --> 00:07:49,590 were on your way to, you drew 172 00:07:49,590 --> 00:07:51,900 the lucky straw, you drew the lucky, lucky straw. 173 00:07:51,900 --> 00:07:52,050 So 174 00:07:52,050 --> 00:07:53,640 you were on, you were on your own. 175 00:07:53,640 --> 00:07:54,600 Does that sound about right? 176 00:07:55,260 --> 00:07:55,890 Yes. 177 00:07:55,890 --> 00:07:55,950 Yeah. 178 00:07:56,190 --> 00:07:56,430 Yeah. 179 00:07:57,030 --> 00:08:03,060 The um, one of the things, kind of the requirements was, um, because of. 180 00:08:03,540 --> 00:08:08,100 The nature of the response was you had to be comfortable doing command line 181 00:08:08,100 --> 00:08:10,050 recovery of the backup application. 182 00:08:10,380 --> 00:08:11,250 Why was that? 183 00:08:12,180 --> 00:08:17,640 Uh, just because you, um, you, you know, the, um, you're gonna 184 00:08:17,640 --> 00:08:21,645 be, um, on the console of the server for a lot of the Okay. 185 00:08:21,645 --> 00:08:22,830 So no gooey for you? 186 00:08:22,835 --> 00:08:24,750 No gooey, no gooey for you at the beginning? 187 00:08:24,750 --> 00:08:24,990 Yeah. 188 00:08:25,260 --> 00:08:28,410 Kind of a recap for the listeners who may not fully recall 189 00:08:28,590 --> 00:08:29,880 the episode, this is where. 190 00:08:30,560 --> 00:08:32,450 The hurricane took out the data center. 191 00:08:32,450 --> 00:08:37,280 I believe that you moved the servers into a different data center to try 192 00:08:37,280 --> 00:08:40,490 to recover, and you moved some of the backup infrastructure as well, correct? 193 00:08:40,970 --> 00:08:41,540 Right. 194 00:08:41,595 --> 00:08:46,695 So the, the way that the site had been set up was, um, that from. 195 00:08:48,195 --> 00:08:51,705 Like a, a main computing, um, standpoint. 196 00:08:51,705 --> 00:08:57,825 They had two main data centers and the design was to have, you know, half the 197 00:08:57,825 --> 00:09:04,095 capacity into one half the capacity and the other with, um, backups and copies. 198 00:09:04,325 --> 00:09:05,465 Going between the two. 199 00:09:05,675 --> 00:09:05,915 Right. 200 00:09:05,915 --> 00:09:12,275 And so we had two backup systems and we replicated between the, the between 201 00:09:12,275 --> 00:09:17,165 them and so data center A would back up half of the servers and replicate 202 00:09:17,165 --> 00:09:20,615 the data center B and B would back up the other half and replicate the A. 203 00:09:20,615 --> 00:09:22,655 So each side had. 204 00:09:23,715 --> 00:09:27,165 Copies of all of the backups for the entire site. 205 00:09:27,285 --> 00:09:27,345 Hmm. 206 00:09:28,845 --> 00:09:33,405 And, um, so site A, uh, data center A was fine. 207 00:09:33,855 --> 00:09:38,235 Um, the building that data center B was in was the one that was 208 00:09:38,235 --> 00:09:43,185 damaged and the damage to the data center, um, wasn't direct. 209 00:09:43,185 --> 00:09:46,275 It was indirect in, you know, the building was damaged, but it was water 210 00:09:46,275 --> 00:09:51,765 damage that came flooding on down and, um, flooded the racks and actually had. 211 00:09:51,810 --> 00:09:55,920 Uh, you know, a foot or so of water, uh, in the, in the floor 212 00:09:55,920 --> 00:09:57,240 of the data center itself. 213 00:09:57,630 --> 00:10:01,140 Uh, the data centers were not raised floor data centers. 214 00:10:01,140 --> 00:10:07,410 They were, um, the, the everything was, um, you know, in cable trays above 215 00:10:07,920 --> 00:10:09,390 and suspended above kind of thing. 216 00:10:09,390 --> 00:10:11,010 So, uh, it was quick. 217 00:10:11,010 --> 00:10:11,760 It sounds like a bad 218 00:10:11,760 --> 00:10:13,290 combination to have with a. 219 00:10:13,770 --> 00:10:15,565 Plug Hurricane. 220 00:10:15,810 --> 00:10:16,200 Yeah. 221 00:10:16,680 --> 00:10:21,480 Um, so like in, in, in the rack that had our equipment, so we had a, um, you know, 222 00:10:21,480 --> 00:10:29,430 backup server, um, uh, and a couple of, uh, media servers and then a storage, um, 223 00:10:29,610 --> 00:10:34,380 and um, and then a tape library and the library was at the bottom and the library 224 00:10:34,380 --> 00:10:36,540 was the one that was the damage the most. 225 00:10:37,400 --> 00:10:43,430 Um, the rest of the equipment, um, was okay, although the, some of the 226 00:10:43,430 --> 00:10:47,060 damage actually was caused by the, um. 227 00:10:48,660 --> 00:10:53,040 Desire to get the equipment quickly out of the one place into the other. 228 00:10:53,370 --> 00:10:56,730 And so when they were taking it out, they weren't as careful as they could 229 00:10:56,730 --> 00:10:58,260 have been with rails and whatnot. 230 00:10:58,260 --> 00:11:02,940 So we had some problems, um, racking the equipment up at the, at the new 231 00:11:02,940 --> 00:11:08,550 location, you know, and they took a, a small little server room and quickly 232 00:11:08,550 --> 00:11:13,290 converted it into a data center to house all of the, the facilities. 233 00:11:13,290 --> 00:11:13,590 Yeah. 234 00:11:14,670 --> 00:11:15,900 And so, um. 235 00:11:16,560 --> 00:11:21,540 So that was from the backup system standpoint, the impact on it. 236 00:11:21,990 --> 00:11:27,630 Um, because the, because of the replication, we were able to start, 237 00:11:28,110 --> 00:11:32,400 um, providing restores of, of a lot of the servers that they needed. 238 00:11:33,030 --> 00:11:33,480 Um. 239 00:11:34,260 --> 00:11:37,110 From the, from the A site, right. 240 00:11:37,620 --> 00:11:41,850 Uh, while we were recovering the, the backups system for the 241 00:11:41,850 --> 00:11:46,230 B site, we, we knew that we could do the recoveries from the A side. 242 00:11:46,260 --> 00:11:50,850 We just knew that when we started to bring back online, A wasn't gonna 243 00:11:50,850 --> 00:11:52,950 have the capacity to back it all up. 244 00:11:52,980 --> 00:11:53,190 Right. 245 00:11:53,190 --> 00:11:56,340 We needed to get, be back on for going forward. 246 00:11:56,640 --> 00:11:56,820 Right. 247 00:11:58,260 --> 00:11:59,400 So, and then we were. 248 00:11:59,775 --> 00:12:04,365 We were dealing with the fact that, um, we had to have vendors come in to work 249 00:12:04,365 --> 00:12:06,585 on their equipment and do a checkout. 250 00:12:06,975 --> 00:12:11,085 And we weren't, we didn't even fire it up until the vendor came in and certified 251 00:12:11,085 --> 00:12:14,955 that, that they were gonna continue to support it after we had done what we did. 252 00:12:15,645 --> 00:12:15,885 Um, 253 00:12:16,185 --> 00:12:16,905 oh, that's interesting. 254 00:12:17,115 --> 00:12:17,355 Yeah. 255 00:12:17,355 --> 00:12:18,045 And so we had 256 00:12:18,045 --> 00:12:20,775 to wait, um, for them to be How long, 257 00:12:21,225 --> 00:12:26,505 how long, how long do you think it was between basically you arrived? 258 00:12:27,105 --> 00:12:31,395 And you could actually start doing something from, from, well, let me 259 00:12:31,395 --> 00:12:35,115 rephrase, but you could actually start doing a restore of any kind. 260 00:12:35,805 --> 00:12:37,515 Um, we were restoring. 261 00:12:40,350 --> 00:12:44,400 I was restoring, starting to restore servers the first or second day. 262 00:12:44,880 --> 00:12:46,860 Um, okay, so you have to understand that. 263 00:12:46,860 --> 00:12:52,920 Well, um, um, so the, the initial recovery team was a team of, uh, you 264 00:12:52,920 --> 00:12:55,140 know, like, like myself for the backups. 265 00:12:55,140 --> 00:12:56,460 We had a, uh, a. 266 00:12:57,230 --> 00:13:00,260 DBA there for, for handle the databases. 267 00:13:00,260 --> 00:13:01,520 They had some network people. 268 00:13:01,970 --> 00:13:05,390 Um, and then they actually had to fly in a couple of vendors for 269 00:13:05,390 --> 00:13:08,180 the, um, um, emergency generators. 270 00:13:08,240 --> 00:13:08,330 Mm-hmm. 271 00:13:09,260 --> 00:13:14,330 Um, to, to them because, um, they were, um. 272 00:13:15,900 --> 00:13:21,060 The, the generators that put in were emergency generators, short term outage. 273 00:13:21,060 --> 00:13:25,320 Well, they found themselves faced with a long-term outage, uh, power wise, right? 274 00:13:25,830 --> 00:13:28,350 It was gonna be a long time before power was brought back. 275 00:13:28,350 --> 00:13:32,430 And so they were running their emergency generators way beyond the duty cycle. 276 00:13:32,430 --> 00:13:38,760 So they had guys in there, um, babying and keeping them going while they, uh, um. 277 00:13:39,300 --> 00:13:42,150 Came up with longer term solutions for how they were gonna power it. 278 00:13:43,380 --> 00:13:44,550 Um, so, um. 279 00:13:45,795 --> 00:13:50,565 So when they, when all this team get got together, as well as this was, 280 00:13:50,565 --> 00:13:54,135 was flowing in, and then you had the local, um, IT staff and that's 281 00:13:54,135 --> 00:13:56,085 just, just the whole site staff. 282 00:13:56,505 --> 00:14:03,045 Um, they had started to pull out their, um, recovery plan and which 283 00:14:03,075 --> 00:14:07,275 pieces of equipment that, uh, and which systems and that needed to 284 00:14:07,275 --> 00:14:10,485 come back online first, second, and third and all that kind of stuff. 285 00:14:10,485 --> 00:14:10,725 Right? 286 00:14:11,025 --> 00:14:12,471 Oh, so they actually had a runbook and a. 287 00:14:12,780 --> 00:14:14,460 Plan put in place ahead of time to, to some 288 00:14:14,460 --> 00:14:14,790 degree. 289 00:14:14,790 --> 00:14:15,120 Yeah. 290 00:14:15,120 --> 00:14:15,360 Yeah. 291 00:14:15,360 --> 00:14:17,370 I mean, they, they kind of knew, you know, based on the business. 292 00:14:17,375 --> 00:14:17,505 What did 293 00:14:17,505 --> 00:14:18,300 that, what did that look like? 294 00:14:19,080 --> 00:14:20,190 Was it actually, I never 295 00:14:20,190 --> 00:14:22,350 saw it myself, you know? 296 00:14:22,980 --> 00:14:23,160 Right. 297 00:14:23,220 --> 00:14:28,590 Um, but I do know that one of the things it was kind of interesting 298 00:14:28,590 --> 00:14:35,070 is that, you know, the run book, um, that, that, uh, that they had was this. 299 00:14:35,710 --> 00:14:39,220 Somewhat abstract thing at the time because, you know, they thought 300 00:14:39,220 --> 00:14:42,640 about it and they planned and they, they tested little pieces of it, 301 00:14:42,640 --> 00:14:47,080 but they never tested it in its entirety, like the disaster presented. 302 00:14:47,320 --> 00:14:50,020 You know, you know, we, we talk about that all the time. 303 00:14:50,025 --> 00:14:50,315 Mm-hmm. 304 00:14:50,395 --> 00:14:51,355 Uh, Ron, that, that. 305 00:14:52,395 --> 00:14:54,375 That, that, that's exactly the same thing. 306 00:14:54,375 --> 00:14:58,815 Back in the day when I was firing backups and anger, when we did a DR test at the 307 00:14:58,815 --> 00:15:04,575 bank that I was at, that we never tested, because, you know, without the cloud, 308 00:15:04,575 --> 00:15:09,615 without virtualization and, you know, and, and additional hardware or whatever, doing 309 00:15:09,615 --> 00:15:13,395 a full DR test is ridiculously difficult. 310 00:15:13,395 --> 00:15:13,755 Right. 311 00:15:13,755 --> 00:15:14,385 It is. 312 00:15:14,725 --> 00:15:15,085 It's, 313 00:15:15,085 --> 00:15:18,895 and, and so no one, no one does that, no one tests their whole runbook, um, right. 314 00:15:18,955 --> 00:15:22,105 Or, well, well, now, I think now more people do. 315 00:15:22,105 --> 00:15:24,205 But, so it sounds like you had that problem. 316 00:15:24,205 --> 00:15:28,435 So they, they had this runbook, but it was primarily in theory up until, 317 00:15:29,605 --> 00:15:29,995 right. 318 00:15:30,265 --> 00:15:32,370 Because what they learned is, is, you know, as, as. 319 00:15:33,855 --> 00:15:37,995 They learned that they had made certain assumptions that they shouldn't have made. 320 00:15:38,865 --> 00:15:38,925 Hmm. 321 00:15:39,075 --> 00:15:40,875 And like one of the things, 322 00:15:40,965 --> 00:15:41,145 go ahead. 323 00:15:41,355 --> 00:15:41,445 Yeah. 324 00:15:41,475 --> 00:15:41,655 Well, 325 00:15:41,655 --> 00:15:44,175 like for instance, active directory design. 326 00:15:44,625 --> 00:15:44,925 Okay. 327 00:15:44,925 --> 00:15:49,005 So the active directory for the island was tied to the mainland. 328 00:15:50,040 --> 00:15:51,630 To the, to the corporate data center. 329 00:15:51,840 --> 00:15:55,920 And when they broke the link to the corporate data center, they were, it's 330 00:15:55,920 --> 00:15:58,290 like, oh, we can't authenticate anything. 331 00:15:58,290 --> 00:16:00,690 So they had this, that's one of the first things they had to bring up. 332 00:16:00,690 --> 00:16:00,930 Right. 333 00:16:01,410 --> 00:16:03,240 So, so help me understand there. 334 00:16:03,240 --> 00:16:07,800 So when, when the hurricane hit, basically they lost connection to the mainland? 335 00:16:07,950 --> 00:16:08,190 Yeah. 336 00:16:08,940 --> 00:16:09,360 Okay. 337 00:16:09,450 --> 00:16:13,020 So they had to, they had to do all of this locally. 338 00:16:13,530 --> 00:16:15,030 Um, interesting. 339 00:16:15,360 --> 00:16:15,510 Yeah. 340 00:16:15,930 --> 00:16:16,590 You know, it's interesting. 341 00:16:16,590 --> 00:16:20,070 You would think that they would not make that assumption being an island. 342 00:16:20,820 --> 00:16:22,350 Right, right. 343 00:16:22,650 --> 00:16:30,210 Well, and, and, and see, uh, you know, um, a lot of it was driven by the 344 00:16:30,210 --> 00:16:33,900 experience of the local, you know, the local experience on the island. 345 00:16:33,930 --> 00:16:40,710 And, um, the, the prior hurricanes that they had had, had not been as devastating 346 00:16:40,710 --> 00:16:42,210 as that particular one that hit them. 347 00:16:42,960 --> 00:16:45,960 And so they were able to make it through without. 348 00:16:46,440 --> 00:16:51,210 The kinds of, of impact and losses that they, they, um, did, I mean, it walked, 349 00:16:51,450 --> 00:16:52,830 walked right up the middle of the island. 350 00:16:52,830 --> 00:16:54,120 I mean, it just devastated them. 351 00:16:54,750 --> 00:16:57,540 Um, and so, you know, so they were dealing with that. 352 00:16:57,540 --> 00:17:01,020 And then one of the other interesting things that, it took a interesting 353 00:17:01,020 --> 00:17:03,750 how it takes a while for you to figure out what's going on. 354 00:17:04,140 --> 00:17:09,120 They had this, um, um, satellite communication hookup, 355 00:17:09,120 --> 00:17:10,560 which was their fallback. 356 00:17:11,040 --> 00:17:14,160 And so that was the way they were talking between, um. 357 00:17:15,390 --> 00:17:20,130 Uh, you know, from, from the facility there to the, to the corporate end of it. 358 00:17:20,730 --> 00:17:28,500 And every day, like around noon or so, the, um, connections 359 00:17:28,500 --> 00:17:29,820 would start dropping off. 360 00:17:30,240 --> 00:17:34,080 I was using it to try to, to phone home, you know, in the afternoons, couldn't 361 00:17:34,080 --> 00:17:36,030 get a, uh, a connection or anything. 362 00:17:36,390 --> 00:17:37,800 And they were like, what's going on? 363 00:17:37,800 --> 00:17:41,790 It was working fine in the morning, but in the afternoon and one of the 364 00:17:41,790 --> 00:17:42,930 network guys started poking around. 365 00:17:42,960 --> 00:17:43,410 Oh. 366 00:17:44,025 --> 00:17:46,485 You know, and he actually called the dish mo slightly. 367 00:17:47,235 --> 00:17:52,485 Well, what it was was that it was an emergency and um, it was supposed to 368 00:17:52,485 --> 00:17:54,825 be for emergency only, short term. 369 00:17:54,825 --> 00:17:56,835 Again, one of these short term kinds of things. 370 00:17:56,985 --> 00:17:59,865 And they had turned it into their main network link. 371 00:18:00,405 --> 00:18:05,205 Well it was, um, it was a metered thing 'cause it was shared by all, you know, 372 00:18:05,205 --> 00:18:06,885 all the other emergency equipment. 373 00:18:07,215 --> 00:18:08,925 So they would use up their full. 374 00:18:09,480 --> 00:18:13,320 A day's allotment by noon or, or even earlier sometimes. 375 00:18:13,800 --> 00:18:14,070 Right. 376 00:18:14,400 --> 00:18:16,530 Also, they had, had, they had a bandwidth allotment up. 377 00:18:16,530 --> 00:18:16,710 Yeah. 378 00:18:16,710 --> 00:18:18,780 And so then they were meter in the afternoon. 379 00:18:18,900 --> 00:18:19,050 Right. 380 00:18:19,380 --> 00:18:21,240 And so they had to, they had to work. 381 00:18:21,240 --> 00:18:22,455 That was one huge, you 382 00:18:22,455 --> 00:18:23,495 know, you know what this reminds me of? 383 00:18:23,635 --> 00:18:25,860 You know, it's, you have unlimited bandwidth. 384 00:18:26,100 --> 00:18:27,390 Just don't use too much of it. 385 00:18:27,600 --> 00:18:30,030 Just, yeah, just don't use, don't use it all. 386 00:18:30,030 --> 00:18:30,330 You hit your 387 00:18:30,330 --> 00:18:31,920 data cap and we're gonna meter you. 388 00:18:32,580 --> 00:18:32,850 Wow. 389 00:18:33,510 --> 00:18:33,570 Okay. 390 00:18:33,570 --> 00:18:33,615 So yeah. 391 00:18:33,620 --> 00:18:37,380 So they, that was a huge thing for, for the networks was to work on, 392 00:18:37,650 --> 00:18:42,030 um, and work with, uh, you know, the vendors and that to try to get, um. 393 00:18:43,050 --> 00:18:47,520 A, a, a reliable, fast enough connection and multiple connections. 394 00:18:48,000 --> 00:18:52,200 Um, I, since I'm no longer, um, working with them, I don't 395 00:18:52,200 --> 00:18:53,880 know the, the end results. 396 00:18:53,880 --> 00:18:59,850 I do know that they were, um, headed towards several, um, um, different 397 00:18:59,850 --> 00:19:02,460 microwave connections from the facility. 398 00:19:02,955 --> 00:19:08,355 To, um, you know, to, to a, a main link that would then take them to the mainland. 399 00:19:08,865 --> 00:19:12,945 Um, running their dedicated link themselves to the mainland was, 400 00:19:12,975 --> 00:19:14,565 you know, just cost prohibited. 401 00:19:15,225 --> 00:19:16,155 Uh, you know, it's interesting. 402 00:19:16,155 --> 00:19:20,895 I I, I did wanna just mention, I, I'm wondering the degree to which this 403 00:19:20,925 --> 00:19:25,605 new, so Elon Musk has now come out with this, uh, I mean, they're right. 404 00:19:25,605 --> 00:19:30,765 They're in beta right now, and it's a completely redesigned way to do. 405 00:19:31,470 --> 00:19:35,195 Satellite based internet connections, starlink, where they have starlink, 406 00:19:35,275 --> 00:19:40,440 starlink, where they starlink, where they have all of these, um, satellites in. 407 00:19:40,890 --> 00:19:45,360 Um, it's a different kind of orbit, I guess than than usual. 408 00:19:45,360 --> 00:19:49,470 And, and they're saying that they can actually get both bandwidth 409 00:19:49,535 --> 00:19:57,900 and latency equal to and or better than, um, what you can do. 410 00:19:58,275 --> 00:20:03,795 On land, and so it, it, it's, I I don't know the degree to, I don't 411 00:20:03,795 --> 00:20:06,165 know how much it scales up mm-hmm. 412 00:20:06,405 --> 00:20:10,005 For a data center connection, but it, it, it's just, it's just the interesting, 413 00:20:10,275 --> 00:20:15,735 you know, thoughts towards the future for things like islands that are, 414 00:20:15,975 --> 00:20:17,700 you know, cut off the way they are. 415 00:20:17,700 --> 00:20:17,980 Mm-hmm. 416 00:20:18,135 --> 00:20:22,215 I like, even, even, you know, the, this, the microwave connection. 417 00:20:22,950 --> 00:20:27,750 Like how reliable is the main connection that they're connecting to there? 418 00:20:27,750 --> 00:20:27,960 Right. 419 00:20:27,960 --> 00:20:32,550 It's probably pretty reliable, but what if it wasn't right? 420 00:20:32,550 --> 00:20:33,480 What if that went down? 421 00:20:33,480 --> 00:20:34,530 Then you're, you're really right. 422 00:20:34,590 --> 00:20:39,570 Well, and, and what they were running into there was like, um, remote, 423 00:20:39,840 --> 00:20:42,660 I, I'm gonna use transceivers, but maybe that's not the right word. 424 00:20:42,660 --> 00:20:46,470 But you have a, um, a, a remote tower that is. 425 00:20:47,700 --> 00:20:49,770 Relaying, you know, it's a relay. 426 00:20:50,400 --> 00:20:53,820 Um, and so, you know, it's, it's receiving a signal and you know, 427 00:20:53,820 --> 00:20:55,320 because microwave is line of sight. 428 00:20:55,320 --> 00:20:59,760 And so if you're gonna go, um, you know, over mountainous things, you 429 00:20:59,760 --> 00:21:03,300 have to send it between a series of towers to get it to where you wanna go. 430 00:21:03,780 --> 00:21:08,400 And, um, they were finding out that they would lose a tower. 431 00:21:08,430 --> 00:21:10,440 And when they go out and look, well, somebody had gone 432 00:21:10,440 --> 00:21:11,820 out and stolen all the gas. 433 00:21:12,575 --> 00:21:16,265 Outta the generator because gas was like hard to get. 434 00:21:16,265 --> 00:21:16,715 Oh, yeah. 435 00:21:17,285 --> 00:21:22,565 Um, and then, um, or, or people were ripping up, um, you know, 'cause power 436 00:21:22,565 --> 00:21:26,225 lines and all the, all the sound people were ripping out the copper, you know, 437 00:21:26,225 --> 00:21:30,515 so, so you run into that kinda stuff where that just made the recovery 438 00:21:30,515 --> 00:21:36,665 effort, you know, external to the site hard, um, which then impacted the site. 439 00:21:37,180 --> 00:21:37,470 Yeah. 440 00:21:38,765 --> 00:21:38,825 Yeah. 441 00:21:40,260 --> 00:21:43,140 You were dealing, you were dealing, in this case, you, you had 442 00:21:43,140 --> 00:21:45,630 somewhat of a perfect storm where you're dealing with the fact that 443 00:21:45,630 --> 00:21:46,800 your data centers were flooded. 444 00:21:46,800 --> 00:21:49,770 You weren't, uh, you didn't have a raised floor. 445 00:21:49,860 --> 00:21:51,690 Uh, so that makes that problem worse. 446 00:21:51,690 --> 00:21:54,450 And then you didn't, you know, the, the, the Dr. 447 00:21:54,450 --> 00:21:58,770 Design made assumptions that were no longer true. 448 00:21:58,770 --> 00:22:04,860 And then meanwhile, the, the things that you did have were being frustrated by. 449 00:22:05,270 --> 00:22:06,830 Other things that were, yes. 450 00:22:06,830 --> 00:22:10,280 It's like, Hey, can you, can you stop messing with the things that actually 451 00:22:10,280 --> 00:22:13,940 work while we're trying to put the data center back together over here? 452 00:22:14,600 --> 00:22:14,960 Wow. 453 00:22:15,200 --> 00:22:19,585 That's, you know, we, we, we live in backup land and we think of the, the, 454 00:22:19,730 --> 00:22:22,730 the restore is the part we focus on. 455 00:22:22,730 --> 00:22:26,780 But it sounds like most of the problems that you were experiencing 456 00:22:26,780 --> 00:22:30,830 had nothing to do with the actual act of getting data from. 457 00:22:31,215 --> 00:22:32,715 Storage devices to server? 458 00:22:32,775 --> 00:22:33,225 No, 459 00:22:33,225 --> 00:22:38,685 actually that part of it went, um, went well in, in the areas. 460 00:22:38,985 --> 00:22:44,835 The, um, the issues that we ran into was, um, like you said, the, um, the 461 00:22:44,835 --> 00:22:51,555 disconnect between the design or intent and the reality as it, as it unfolded. 462 00:22:52,065 --> 00:22:56,715 Um, you know, they'd say, oh, so we need, you know, this server, uh. 463 00:22:57,135 --> 00:22:58,215 Brought back online. 464 00:22:58,575 --> 00:23:01,330 Um, so find the most recent backup and you start looking through. 465 00:23:01,330 --> 00:23:01,610 Mm-hmm. 466 00:23:01,690 --> 00:23:04,755 And it's like, oh, we're not backing that up. 467 00:23:04,785 --> 00:23:06,765 You know, when did you bring that online? 468 00:23:06,945 --> 00:23:08,475 Why didn't you tell us? 469 00:23:08,475 --> 00:23:08,835 You know? 470 00:23:09,345 --> 00:23:14,050 Um, and so we ran into a few of those where, um, we couldn't give them, uh. 471 00:23:14,745 --> 00:23:15,195 Yes. 472 00:23:15,225 --> 00:23:17,595 Oh, that, that is one of the most frustrating. 473 00:23:17,595 --> 00:23:24,435 So, um, and then, um, well, and, and, and then, you know, or we're backing it 474 00:23:24,435 --> 00:23:28,605 up, but we're not including everything, you know, so, um, we're backing it up 475 00:23:28,605 --> 00:23:33,885 in the sense that we're doing an OS backup, but the data that's on it, you 476 00:23:33,885 --> 00:23:36,165 didn't include it, you know, so the, um. 477 00:23:38,025 --> 00:23:40,980 So thi this was a net backup shop, so you were not using all local 478 00:23:40,980 --> 00:23:42,600 drives, is what you're saying. 479 00:23:42,605 --> 00:23:49,170 And, and it was funny that when I first hired on there, came on board, um, 480 00:23:49,680 --> 00:23:56,730 they were in the middle of, um, of a, um, upgrade and kind of a transition. 481 00:23:57,090 --> 00:24:01,021 Uh, the, the manager in charge of the backups and, and 482 00:24:01,026 --> 00:24:04,290 whatnot at the time, um, was. 483 00:24:04,380 --> 00:24:10,470 It was pushing this, we, we should only back up what we really need. 484 00:24:10,530 --> 00:24:16,140 And so he was trying to push the, uh, I hate that responsibility onto the 485 00:24:16,140 --> 00:24:20,220 data owners and saying, you need to define what's important to you and 486 00:24:20,220 --> 00:24:21,510 let us know and we'll back it up. 487 00:24:21,780 --> 00:24:23,820 So they stopped all local drives. 488 00:24:23,850 --> 00:24:23,940 Right. 489 00:24:24,840 --> 00:24:25,890 And um. 490 00:24:28,100 --> 00:24:31,425 I, you know, here's the thing, I, I don't, I don't disagree with the idea 491 00:24:31,425 --> 00:24:33,105 of not backing up worthless data. 492 00:24:33,105 --> 00:24:34,575 I don't disagree with that idea. 493 00:24:34,755 --> 00:24:40,420 What I disagree with is the implementation of, okay, so it, what I believe is mm-hmm. 494 00:24:40,505 --> 00:24:44,565 You should identify what is worthless and then we will exclude that. 495 00:24:45,345 --> 00:24:45,735 Right. 496 00:24:45,825 --> 00:24:48,555 Not identify what is valuable that Right. 497 00:24:48,555 --> 00:24:50,445 And then we will in include that. 498 00:24:50,580 --> 00:24:53,775 I, I just, I just disagree with that, the implementation and just 499 00:24:53,775 --> 00:24:55,305 because you always forget things. 500 00:24:55,965 --> 00:24:58,395 Well, you always forget things and then you, you add things. 501 00:24:58,395 --> 00:25:00,285 It's just like the server that he talked about, right? 502 00:25:00,285 --> 00:25:05,055 You end up adding file systems and the, the, I, you know, I go back to 503 00:25:05,055 --> 00:25:10,395 when I helped redesign the, the backup system for a broadcasting company. 504 00:25:10,965 --> 00:25:14,835 Uh, they were 20 terabytes as I recall when I got there. 505 00:25:15,210 --> 00:25:17,340 And one of the things they weren't doing is that they 506 00:25:17,340 --> 00:25:18,660 weren't doing all local drives. 507 00:25:18,660 --> 00:25:21,030 And I pushed really hard that we should do all local drives as 508 00:25:21,030 --> 00:25:25,080 part of the redesign, we did it, we discovered 10 more terabytes. 509 00:25:25,170 --> 00:25:25,320 Yeah. 510 00:25:25,865 --> 00:25:29,130 Of, of data that they weren't, that they weren't backing it up. 511 00:25:29,130 --> 00:25:30,930 And yes, it was really valuable data. 512 00:25:31,680 --> 00:25:32,160 Right. 513 00:25:32,190 --> 00:25:35,220 Well what, what I found interesting, um. 514 00:25:36,180 --> 00:25:39,960 Through that whole process of, you know, coming on board with that, um, 515 00:25:39,960 --> 00:25:45,300 trying to push that idea of, of the data owners being responsible to identify 516 00:25:45,300 --> 00:25:49,230 and we back up what they say to then we have to do with this recovery. 517 00:25:49,290 --> 00:25:49,620 Right. 518 00:25:50,130 --> 00:25:50,580 Um, I. 519 00:25:52,050 --> 00:25:53,460 My experience. 520 00:25:54,000 --> 00:25:54,330 Yeah. 521 00:25:54,525 --> 00:25:54,935 Well, yeah. 522 00:25:55,230 --> 00:25:58,200 And my experiences in who, whose fault has it been Ron, over the, over 523 00:25:58,200 --> 00:26:01,770 the time that I've been involved in, you know, the system side of things. 524 00:26:02,190 --> 00:26:07,410 Um, and watching, watching the way things unfold over time is that is 525 00:26:07,410 --> 00:26:11,400 really good in theory, that approach of making the, the data owners 526 00:26:11,400 --> 00:26:13,050 responsible for identifying it. 527 00:26:13,620 --> 00:26:15,180 But what I have. 528 00:26:16,475 --> 00:26:18,095 Learned and watched has been. 529 00:26:18,095 --> 00:26:24,665 The real weakness of that is, um, you know, data owners come and go and 530 00:26:24,755 --> 00:26:27,785 you don't get the transition between. 531 00:26:28,395 --> 00:26:31,635 An outgoing data owner and incoming data owner of what's 532 00:26:31,635 --> 00:26:32,775 covered, what's not covered. 533 00:26:33,225 --> 00:26:39,855 Um, you know, I know that I struggled all through my time, um, in, in, you 534 00:26:39,855 --> 00:26:46,335 know, um, it thing of doing proper and adequate documentation so that if I were 535 00:26:46,335 --> 00:26:51,285 hit by a bus as, as the saying goes, um, somebody would know what was, you 536 00:26:51,285 --> 00:26:53,595 know, what to do when they stepped in. 537 00:26:54,075 --> 00:26:56,985 Um, but I don't think that that was. 538 00:26:57,345 --> 00:26:58,845 Happening across the board. 539 00:26:58,845 --> 00:27:03,195 And so, you know, you'd have an assistant was brought online, you know, like four 540 00:27:03,195 --> 00:27:06,915 years ago you had a very conscientious data owner who identified all this 541 00:27:06,915 --> 00:27:11,775 stuff and it's covered well Over those four years that guy moved on and 542 00:27:11,775 --> 00:27:14,415 somebody else not so dedicated, came in. 543 00:27:14,625 --> 00:27:17,175 Some changes were made that weren't documented. 544 00:27:17,175 --> 00:27:20,025 That weren't covered, you know, and so then you have, 545 00:27:20,175 --> 00:27:22,545 you fast forward to, to the. 546 00:27:22,920 --> 00:27:26,370 Disaster and you have a situation well, well, we are only backing 547 00:27:26,370 --> 00:27:29,640 up part of it, you know, and, and 548 00:27:30,690 --> 00:27:35,490 yeah, this is why, or this is why I'm a strong proponent, a, a of 549 00:27:35,490 --> 00:27:40,770 virtualization and B, B, b, the reason with virtualization, it, it helps 550 00:27:40,770 --> 00:27:44,340 to, it helps to minimize this problem because really all we have to do is make 551 00:27:44,340 --> 00:27:46,545 sure we understand, we know about new. 552 00:27:47,095 --> 00:27:51,625 You know, VMware servers or HyperV servers, and then you can, you 553 00:27:51,625 --> 00:27:55,345 can tell your backup server or your backup software, uh, back 554 00:27:55,345 --> 00:27:59,605 up all VMs that show up on this thing, unless I tell you otherwise. 555 00:27:59,875 --> 00:28:03,715 And when you're backing up those VMs, backup up everything on those VMs. 556 00:28:03,865 --> 00:28:09,265 So you, you solve this problem, uh, you know, from a more global perspective. 557 00:28:09,655 --> 00:28:12,805 Um, and, but this idea. 558 00:28:13,075 --> 00:28:16,435 The, you know, we could talk for hours on all of the things that could go 559 00:28:16,435 --> 00:28:19,555 wrong with manually selecting data sets. 560 00:28:20,035 --> 00:28:23,695 Uh, and you know what, if you, if you back up a, a few terabytes of 561 00:28:23,695 --> 00:28:29,425 worthless data, that is way, that is a much smaller problem than the one 562 00:28:29,425 --> 00:28:31,615 you that this discussion started with. 563 00:28:31,615 --> 00:28:31,735 Right? 564 00:28:31,735 --> 00:28:31,915 Which is. 565 00:28:33,065 --> 00:28:33,575 Restart this. 566 00:28:33,575 --> 00:28:33,725 Yeah. 567 00:28:33,830 --> 00:28:35,250 And I'm trying to restore the server. 568 00:28:35,320 --> 00:28:35,610 Yeah. 569 00:28:35,750 --> 00:28:36,450 And it wasn't 570 00:28:36,450 --> 00:28:36,890 backed up. 571 00:28:36,890 --> 00:28:37,130 Yeah. 572 00:28:37,145 --> 00:28:38,105 Um, yeah. 573 00:28:38,105 --> 00:28:42,545 Virtualization, I think, you know, it was, it was fun to watch that come 574 00:28:42,545 --> 00:28:43,865 in and be part of that coming in. 575 00:28:44,165 --> 00:28:48,815 And it did make backups easier from a pers you know, from the 576 00:28:48,815 --> 00:28:52,145 perspective of, of, um, you got it all. 577 00:28:52,385 --> 00:28:55,085 You know, you just, I mean, the equivalent of all local drives 578 00:28:55,085 --> 00:28:57,065 was the default for, for VMs. 579 00:28:57,515 --> 00:28:59,915 One of the things that we struggled with. 580 00:29:00,095 --> 00:29:00,395 Right. 581 00:29:00,575 --> 00:29:00,995 Um. 582 00:29:01,650 --> 00:29:10,260 Uh, was that we were never able to convince and get the, um, virtualization 583 00:29:10,260 --> 00:29:17,970 team to agree upon some kind of a scheme that they would manage their 584 00:29:17,970 --> 00:29:21,445 VMs under that would allow us to do. 585 00:29:21,735 --> 00:29:24,465 What you said, um, Curtis, this, uh, the 586 00:29:24,465 --> 00:29:26,415 automatic discovery of new vm. 587 00:29:26,715 --> 00:29:27,225 Yeah, yeah. 588 00:29:27,225 --> 00:29:31,545 It's like, you know, if you had just pick something, a folder, 589 00:29:31,545 --> 00:29:32,985 a, you know, whatever kind of, 590 00:29:33,435 --> 00:29:35,145 or tags or whatever they wanted to use 591 00:29:35,385 --> 00:29:39,855 the limiter, you could do, you could do within the, the virtualization software 592 00:29:39,855 --> 00:29:44,055 to identify, you know, production machines, development machines, 593 00:29:44,055 --> 00:29:45,435 however you wanted to break 'em up. 594 00:29:45,435 --> 00:29:50,415 And then we come in and says, you know, in this, on this server, on this. 595 00:29:50,790 --> 00:29:54,360 Cluster, whatever, back up all of these types of machines, then we wouldn't 596 00:29:54,360 --> 00:29:57,570 have to worry when they added machines or took machines away, you know? 597 00:29:58,410 --> 00:29:58,500 Right. 598 00:29:58,830 --> 00:30:04,200 So, you know, we, we still had the same problems in the, with the virtual 599 00:30:04,200 --> 00:30:07,890 servers that we did, with the physical servers of them not telling us 600 00:30:08,010 --> 00:30:08,250 Right. 601 00:30:08,515 --> 00:30:13,195 The, I, I think the, the right long-term solution there is, is tags, right? 602 00:30:13,650 --> 00:30:16,525 And, and then, and then I think that there should be a policy that 603 00:30:16,525 --> 00:30:20,785 says if you have a VM that has no tags, start backing it up, put it 604 00:30:20,785 --> 00:30:22,375 in this policy, and then yell at me. 605 00:30:22,705 --> 00:30:23,035 Right? 606 00:30:23,035 --> 00:30:25,345 Because that way you can say, Hey, there's a new vm. 607 00:30:25,525 --> 00:30:27,985 And then they, then they can come to you and go, oh, that's, that's 608 00:30:27,985 --> 00:30:30,475 test or dev or whatever, and, and you can take it out right? 609 00:30:32,235 --> 00:30:35,535 It's funny how, it's funny how, no. 610 00:30:35,715 --> 00:30:38,115 You know, the more things change, the more they stay the same. 611 00:30:38,775 --> 00:30:38,865 Oh, 612 00:30:38,865 --> 00:30:39,135 yeah. 613 00:30:40,065 --> 00:30:46,395 And, and the key, I mean that I learned in mine, uh, experience for this particular 614 00:30:46,395 --> 00:30:53,475 one, this, um, was in the, all the times I did, well, not quite true, this was. 615 00:30:54,240 --> 00:30:56,160 The biggest disaster. 616 00:30:56,160 --> 00:30:57,270 I should, let me put it that way. 617 00:30:57,270 --> 00:31:00,510 'cause I've been involved in a couple of other DR scenarios 618 00:31:00,510 --> 00:31:03,270 that were very small, um mm-hmm. 619 00:31:03,510 --> 00:31:04,650 Kinds of issues, you know. 620 00:31:05,040 --> 00:31:06,450 Um, we had 621 00:31:06,450 --> 00:31:08,460 nothing beats a completely wiped out island. 622 00:31:08,460 --> 00:31:08,940 Right. 623 00:31:09,210 --> 00:31:09,600 Though we had 624 00:31:09,600 --> 00:31:10,440 a data center. 625 00:31:10,440 --> 00:31:10,500 Yeah. 626 00:31:10,560 --> 00:31:15,240 Um, uh, that had a similar problem where, um. 627 00:31:16,005 --> 00:31:19,425 Somebody had not properly cut the holes in the ceiling of the 628 00:31:19,425 --> 00:31:22,125 data center or plugged them when they were running cables through. 629 00:31:22,125 --> 00:31:29,145 And a lab had a, um, a problem and the sprinklers were released and 630 00:31:29,145 --> 00:31:32,295 the water came down and flooded a corner of this data center. 631 00:31:32,415 --> 00:31:36,075 Um, and it happened to be in a city by the Bay. 632 00:31:36,165 --> 00:31:38,835 Um, but um, we ended up. 633 00:31:40,185 --> 00:31:45,045 Shipping their tapes up or up to our facility and then doing the, 634 00:31:45,585 --> 00:31:50,265 the, um, import to be able to then do the restore from our facility 635 00:31:50,265 --> 00:31:51,585 while they were recovering. 636 00:31:52,395 --> 00:31:54,045 Um, you know, the site down there. 637 00:31:54,045 --> 00:31:59,775 So we had, I had that one before, but one of the things that I've 638 00:31:59,775 --> 00:32:05,295 noticed that's kind of like, I think a problem, it's important to. 639 00:32:06,015 --> 00:32:10,965 The, the proper implementation of a, of a DR resolution. 640 00:32:11,475 --> 00:32:15,495 Um, as well as just the planning, the building and setting up your 641 00:32:16,395 --> 00:32:20,175 infrastructure, your IT infrastructure so that it can be recoverable. 642 00:32:20,175 --> 00:32:25,665 And that's communication among the various teams and groups that are responsible 643 00:32:25,665 --> 00:32:30,525 for all the pieces that make up, you know, an IT infrastructure and, um. 644 00:32:31,740 --> 00:32:34,950 You know, that wasn't a strong point with that company. 645 00:32:35,160 --> 00:32:40,770 Um, they, there was some regulatory reasons why they needed to make 646 00:32:40,770 --> 00:32:47,310 sure that, um, no one person had the keys to the kingdom kind of thing. 647 00:32:48,030 --> 00:32:52,890 But the way they did it tended to put walls between the groups and 648 00:32:52,890 --> 00:32:57,360 so they weren't communicating in a manner, um, that I feel that they 649 00:32:57,360 --> 00:32:59,400 should have, that would've helped. 650 00:33:00,150 --> 00:33:00,600 Um. 651 00:33:01,470 --> 00:33:03,080 You know, address like the left hand, not yeah. 652 00:33:03,080 --> 00:33:04,530 Knowing what the right hand is doing 653 00:33:04,620 --> 00:33:09,030 well and, and address some of the assumptions, you know, address some of 654 00:33:09,030 --> 00:33:12,120 the assumptions because I've learned in, in a lot of my implementations, you 655 00:33:12,120 --> 00:33:16,920 know, um, anytime doing it, you know, uh, learned a long time ago in the 656 00:33:16,920 --> 00:33:18,540 beginning that you gotta have a plan. 657 00:33:18,540 --> 00:33:21,330 You gotta kind of figure out what you're gonna do, what the steps, 658 00:33:21,330 --> 00:33:24,180 what you gonna, you know, um, how are you gonna pull it off, and what 659 00:33:24,180 --> 00:33:25,345 are you gonna do if it doesn't? 660 00:33:25,555 --> 00:33:28,945 Go the way you wanted to, you know, 'cause then you gotta back everything out. 661 00:33:29,485 --> 00:33:34,315 Um, and so I would, you know, do the planning part and then run it 662 00:33:34,315 --> 00:33:37,705 by, uh, other people in my group. 663 00:33:38,670 --> 00:33:39,810 To break the assumptions. 664 00:33:39,810 --> 00:33:42,420 'cause I know when I'm building my plan, I'm making certain assumptions. 665 00:33:42,420 --> 00:33:43,770 I just know it right off the bat. 666 00:33:43,770 --> 00:33:45,750 You know, it's just, I know that's the way we all operate. 667 00:33:46,440 --> 00:33:51,360 Um, and so I'm building my plan and then I have to run it by somebody who will 668 00:33:51,360 --> 00:33:53,820 not have my assumptions to find those. 669 00:33:54,390 --> 00:33:58,170 That reminds me, when, when we used to do the DR test, the way we 670 00:33:58,170 --> 00:34:02,490 would do them is I wasn't allowed to participate in the DR test. 671 00:34:02,490 --> 00:34:02,580 Mm-hmm. 672 00:34:02,920 --> 00:34:07,240 We had to have someone else who was an it, who was an IT person, and then 673 00:34:07,240 --> 00:34:08,890 they would follow my documentation. 674 00:34:09,070 --> 00:34:09,190 Mm-hmm. 675 00:34:09,760 --> 00:34:12,880 Um, and, and I had to pretend to, to be dead. 676 00:34:13,120 --> 00:34:16,810 Um, it was always, by the way, you talked about, you know, you got hit by the bus. 677 00:34:17,020 --> 00:34:19,510 I always hated the fact that it, like, why can't I like 678 00:34:19,840 --> 00:34:21,135 win the lottery and disappear? 679 00:34:21,135 --> 00:34:21,415 Right. 680 00:34:21,795 --> 00:34:22,215 Always 681 00:34:22,215 --> 00:34:22,775 out to be a bus. 682 00:34:22,795 --> 00:34:23,015 Why? 683 00:34:25,840 --> 00:34:26,350 Yeah. 684 00:34:26,350 --> 00:34:28,480 Uh, but yeah, I had to pretend to be dead. 685 00:34:28,570 --> 00:34:30,670 And as I, I've, as I've discussed. 686 00:34:30,880 --> 00:34:32,530 More than once on the podcast. 687 00:34:32,890 --> 00:34:40,570 The, the standard was, um, that, you know, a, a success was we got 100% recovery 688 00:34:40,720 --> 00:34:42,790 without having to have Curtis help. 689 00:34:42,790 --> 00:34:42,880 Right? 690 00:34:43,210 --> 00:34:44,140 And not once. 691 00:34:44,550 --> 00:34:45,480 Did we get that? 692 00:34:45,480 --> 00:34:50,100 Because there was always, there was always something that was left out of the 693 00:34:50,100 --> 00:34:52,260 documentation, no matter how much you try. 694 00:34:52,260 --> 00:34:52,350 Right. 695 00:34:52,410 --> 00:34:52,740 To, 696 00:34:53,400 --> 00:34:53,460 yeah. 697 00:34:53,460 --> 00:34:56,010 Well, and that's, I mean, that's why they're, they, they should 698 00:34:56,010 --> 00:34:58,020 be viewed as living documents. 699 00:34:58,020 --> 00:34:59,670 You should be, um, yeah. 700 00:34:59,700 --> 00:35:00,540 Reviewing them. 701 00:35:00,540 --> 00:35:00,541 Yeah. 702 00:35:00,546 --> 00:35:03,480 You should be testing them, you should be updating them. 703 00:35:04,020 --> 00:35:09,270 Um, but you know, there have not been many. 704 00:35:09,825 --> 00:35:13,155 Places that I've worked in the, in the 30 some years that I've 705 00:35:13,155 --> 00:35:16,815 worked in, in doing that kind of stuff, that operated that way. 706 00:35:17,145 --> 00:35:21,255 Um, you know, there were companies that I would, you know, in like I 707 00:35:21,255 --> 00:35:23,655 can remember one that I got in and if they didn't have anything, and 708 00:35:23,655 --> 00:35:29,175 it really pushed really hard to get some kind of a company-wide, uh, Dr. 709 00:35:29,175 --> 00:35:32,895 Plan pulled together and so we put a lot of effort into it and we built a 710 00:35:32,895 --> 00:35:35,235 document that then was put on a shelf. 711 00:35:35,700 --> 00:35:37,290 And it was never looked at it again. 712 00:35:37,290 --> 00:35:37,380 Right? 713 00:35:37,380 --> 00:35:39,065 And it's like, well, why do we even do it? 714 00:35:39,065 --> 00:35:39,505 Binders 715 00:35:39,525 --> 00:35:41,385 has cobwebs and everything over 716 00:35:41,385 --> 00:35:41,505 it. 717 00:35:41,885 --> 00:35:42,870 Why did we even do it? 718 00:35:42,870 --> 00:35:43,170 You know? 719 00:35:43,800 --> 00:35:52,800 Um, the last company at was one of the better ones for at least attempting to 720 00:35:52,830 --> 00:35:56,760 have that be an ongoing part of their, um. 721 00:35:57,525 --> 00:35:58,785 Their operational thing. 722 00:35:58,785 --> 00:36:02,985 'cause they, they, uh, when an, uh, application or a service was 723 00:36:02,985 --> 00:36:04,545 brought in, it had to be identified. 724 00:36:04,545 --> 00:36:05,685 Is it business critical? 725 00:36:05,685 --> 00:36:09,165 You know, what level of, of protection does it need? 726 00:36:09,165 --> 00:36:14,505 And then, um, if it is business critical, they had to have a DR plan and they 727 00:36:14,505 --> 00:36:18,855 had to do a, um, table talk one year. 728 00:36:18,855 --> 00:36:20,985 And then the following year they actually had to do. 729 00:36:21,405 --> 00:36:22,695 Execute the plan, you know? 730 00:36:23,115 --> 00:36:25,275 And so, um, a good portion, did 731 00:36:25,275 --> 00:36:26,475 they have to do the whole plan 732 00:36:26,985 --> 00:36:27,825 for the application? 733 00:36:27,825 --> 00:36:28,155 Kinda like where 734 00:36:28,155 --> 00:36:28,320 we talked earlier? 735 00:36:28,540 --> 00:36:28,760 For 736 00:36:28,760 --> 00:36:29,440 the application. 737 00:36:29,650 --> 00:36:30,000 Right? 738 00:36:30,060 --> 00:36:30,720 For the application. 739 00:36:31,125 --> 00:36:31,155 Okay. 740 00:36:31,155 --> 00:36:33,465 They had to pretend they had to have a greenfield. 741 00:36:33,470 --> 00:36:33,760 Right? 742 00:36:33,975 --> 00:36:34,185 Right. 743 00:36:34,335 --> 00:36:38,985 And so there was still some weaknesses in that part because, um. 744 00:36:39,720 --> 00:36:43,200 Uh, there, there were, there were assumptions made, right? 745 00:36:43,470 --> 00:36:44,340 Uh, that okay. 746 00:36:44,340 --> 00:36:44,430 Mm-hmm. 747 00:36:44,670 --> 00:36:47,040 We're gonna assume that there's a proper infrastructure. 748 00:36:47,040 --> 00:36:49,560 We're gonna assume this, we're gonna assume there's DNS, you know, 749 00:36:50,400 --> 00:36:51,930 from an application standpoint. 750 00:36:52,110 --> 00:36:53,280 Okay, that's fine. 751 00:36:53,280 --> 00:36:58,290 But what was never really done, and this is kind of the breakdown, um, on 752 00:36:58,290 --> 00:37:03,240 the island, was those assumptions about the infrastructure had never been. 753 00:37:03,600 --> 00:37:04,920 Tested or figured out, gone. 754 00:37:04,920 --> 00:37:06,930 And that's what caught them in the beginning. 755 00:37:07,290 --> 00:37:07,440 Right. 756 00:37:07,650 --> 00:37:09,420 Those assumptions were all underwater and, 757 00:37:09,565 --> 00:37:09,855 yeah. 758 00:37:09,930 --> 00:37:10,260 Yeah. 759 00:37:10,260 --> 00:37:15,330 The infrastructure part was the part that had broken the worst, if that's, 760 00:37:17,100 --> 00:37:17,615 yeah, you know? 761 00:37:17,615 --> 00:37:20,855 You know, it's interesting and I just wonder though, like how many times do 762 00:37:20,855 --> 00:37:22,290 people make that sort of assumption? 763 00:37:22,290 --> 00:37:28,830 Like you kind of make assumptions that power may or may not be available, but 764 00:37:28,890 --> 00:37:32,610 the building may or may not be standing, but there are certain things about. 765 00:37:33,330 --> 00:37:35,940 The outside world that you just assume will still be up and running. 766 00:37:35,970 --> 00:37:36,060 Mm-hmm. 767 00:37:36,540 --> 00:37:39,270 As you're trying to work these things through, I guess on an 768 00:37:39,270 --> 00:37:40,890 island, things get more complicated. 769 00:37:41,040 --> 00:37:41,160 Mm-hmm. 770 00:37:41,460 --> 00:37:44,520 In terms of what may or may not be working, like you said, links, 771 00:37:44,970 --> 00:37:46,440 communications links, et cetera. 772 00:37:46,440 --> 00:37:49,200 But I guess you're right, that is something you have to take into 773 00:37:49,200 --> 00:37:52,770 consideration are things outside of your data center that you have to take. 774 00:37:53,145 --> 00:37:53,565 Think about 775 00:37:53,775 --> 00:37:54,105 Right. 776 00:37:54,105 --> 00:37:59,265 But I, I was, I thought about this a lot after, you know, in the aftermath 777 00:37:59,265 --> 00:38:05,235 of that for the, um, um, for, for some time after being involved in that. 778 00:38:05,685 --> 00:38:11,475 And what I experienced there, what was experienced there to me could very 779 00:38:11,475 --> 00:38:17,475 well happen in a regional sort of way here on the mainland and, and leave, 780 00:38:17,475 --> 00:38:19,545 leave a company in the same boat. 781 00:38:20,610 --> 00:38:20,940 Right. 782 00:38:21,750 --> 00:38:26,220 Um, and so, you know, while there are some particularities to being on an 783 00:38:26,220 --> 00:38:30,360 island, um, it doesn't mean that if you're not on an island, you don't 784 00:38:30,360 --> 00:38:35,700 have to worry about the kinds of assumptions that, that, you know, that 785 00:38:36,720 --> 00:38:41,340 turned out to be big problems in their initial, um, you know, recovery attempt. 786 00:38:41,370 --> 00:38:41,520 Yeah. 787 00:38:41,520 --> 00:38:41,580 You. 788 00:38:42,300 --> 00:38:46,800 You would think that being, being on an island, they would not assume 789 00:38:47,340 --> 00:38:48,750 a connection to the mainland. 790 00:38:48,840 --> 00:38:51,060 But a apparently that was the case. 791 00:38:51,060 --> 00:38:51,476 That was the case, yeah. 792 00:38:51,960 --> 00:38:56,160 Um, and the, the, the difficulty I have when I hear this story, 793 00:38:56,160 --> 00:39:00,750 because the, the island situation, what it does is, I think it, it. 794 00:39:01,365 --> 00:39:07,305 Brings to the foreground or makes possible many of the worst case scenarios that 795 00:39:07,305 --> 00:39:11,055 could happen, that that could happen to a lesser degree on the mainland, but, 796 00:39:11,445 --> 00:39:13,485 but to a greater degree on, on an island. 797 00:39:14,085 --> 00:39:20,355 Um, but the, the, the real problem here is that many, many of the modern solutions 798 00:39:20,355 --> 00:39:25,965 that I would think of to solve these problems are based on using the cloud. 799 00:39:26,235 --> 00:39:26,355 Mm-hmm. 800 00:39:27,450 --> 00:39:30,990 Is problematic when you look at an island situation. 801 00:39:30,990 --> 00:39:31,080 Mm-hmm. 802 00:39:31,320 --> 00:39:34,620 It's not as much when you're looking at a mainland situation, but if, for 803 00:39:34,620 --> 00:39:40,050 example, you, you, you, you would have to have, you know, internet connectivity, 804 00:39:40,050 --> 00:39:41,280 which is why I go back to that. 805 00:39:41,280 --> 00:39:44,670 I, I'm really curious to see how this, how the Sea Elon Musk project 806 00:39:44,670 --> 00:39:46,170 goes away or go goes forward. 807 00:39:46,170 --> 00:39:50,160 And whether or not it's, um, you know, something that can 808 00:39:50,160 --> 00:39:51,660 help solve this problem because. 809 00:39:52,740 --> 00:39:56,670 Everything that I know that's being done right now has to do, like 810 00:39:56,670 --> 00:40:01,560 in the really cool DR perspective has to do with using the cloud. 811 00:40:01,560 --> 00:40:08,820 Nobody's talking about, you know, u using physical server DR services or you know, 812 00:40:08,820 --> 00:40:12,240 all of the, basically without using the cloud, it's so much harder and with using 813 00:40:12,240 --> 00:40:15,750 the cloud, it's so much easier to not just to test an application, but to test 814 00:40:15,750 --> 00:40:18,300 the disaster recovery of the entire site. 815 00:40:18,570 --> 00:40:20,910 But here's a question for you, Curtis. 816 00:40:21,510 --> 00:40:21,840 Yeah. 817 00:40:22,230 --> 00:40:26,610 Given the island scenario and assuming that your, the cloud 818 00:40:26,610 --> 00:40:28,290 was available on the island. 819 00:40:28,890 --> 00:40:29,220 Yeah. 820 00:40:30,540 --> 00:40:35,460 Wouldn't, and say everyone got hit by this, um, isn't the cloud provider in the 821 00:40:35,460 --> 00:40:39,420 same situation where they may not have enough resources or they might be down? 822 00:40:39,450 --> 00:40:42,480 That's why I say granted, I, I know that you probably are gonna do DR 823 00:40:42,480 --> 00:40:46,350 to the mainland in this example or somewhere else, but I'm just wondering 824 00:40:46,350 --> 00:40:47,550 if everyone starts doing that. 825 00:40:47,640 --> 00:40:49,680 Don't, like cloud provider is just a big. 826 00:40:50,340 --> 00:40:51,510 Data, it's just somebody else's server. 827 00:40:51,510 --> 00:40:51,960 Yeah, that's, 828 00:40:52,285 --> 00:40:52,575 yeah. 829 00:40:52,920 --> 00:40:53,070 Yeah. 830 00:40:53,070 --> 00:40:53,820 Agreed. 831 00:40:53,820 --> 00:40:57,780 I, I'm just saying that's why the, the, the, the, the island situation is so 832 00:40:57,780 --> 00:41:02,250 problematic because I, I completely agree with you that I, if it was, let's just 833 00:41:02,250 --> 00:41:03,840 say this, this was a pretty small island. 834 00:41:03,840 --> 00:41:06,420 Let's say it's a bigger island and there is a cloud provider. 835 00:41:06,450 --> 00:41:08,640 'cause there, there wasn't, I, I don't think cloud. 836 00:41:08,890 --> 00:41:10,630 Stuff available on the island. 837 00:41:11,050 --> 00:41:14,590 Uh, if you, if it was available, I wouldn't use that as your DR site. 838 00:41:14,590 --> 00:41:15,760 I would use the mainland. 839 00:41:15,850 --> 00:41:16,180 Right. 840 00:41:16,420 --> 00:41:20,950 And so, because Yes, you, because you're completely right if, if your data center's 841 00:41:20,950 --> 00:41:24,370 underwater and you're on an island quite ly their data center's underwater. 842 00:41:24,370 --> 00:41:24,610 Right. 843 00:41:25,060 --> 00:41:29,440 Um, the, um, it reminds me, I used to, you know, manage data 844 00:41:29,440 --> 00:41:32,110 centers in Delaware and we had a. 845 00:41:32,970 --> 00:41:35,100 Our offsite vaulting company. 846 00:41:35,100 --> 00:41:37,110 It wasn't Iron Mountain, it was this local company. 847 00:41:37,500 --> 00:41:43,920 And what they had was they had a World War II bunker, um, like, like bomb shelter. 848 00:41:44,190 --> 00:41:47,820 And that's where they stored tapes for people, which sounds really 849 00:41:47,820 --> 00:41:49,410 good until a hurricane comes. 850 00:41:49,830 --> 00:41:54,900 And so whenever a hurricane was on its way to Delaware, we had to pay money 851 00:41:55,200 --> 00:42:00,060 to have all of our tapes moved out of the bunker and up to the second floor. 852 00:42:00,900 --> 00:42:04,680 So that if our data, if our data center was underwater, our tapes 853 00:42:04,680 --> 00:42:07,170 also were not underwater anyway. 854 00:42:08,250 --> 00:42:13,080 But yeah, I, I just, it's just the, the island situation is frustrating. 855 00:42:13,350 --> 00:42:14,070 Um, and. 856 00:42:15,120 --> 00:42:17,610 Uh, so any further thoughts? 857 00:42:17,940 --> 00:42:23,010 Um, Ron on the, because we haven't even talked about the recovery yet, and I, 858 00:42:23,010 --> 00:42:26,700 I'm just gonna have to have you back 'cause I, I find this solu, I find 859 00:42:26,700 --> 00:42:28,470 this discussion incredibly fascinating. 860 00:42:28,470 --> 00:42:30,120 You're, you know, you're, you're smart. 861 00:42:30,120 --> 00:42:32,820 You know what you're doing, you're articulate, uh, and, and you 862 00:42:32,820 --> 00:42:35,370 know, I'm, I'm super glad that we, that we finally have you on. 863 00:42:35,670 --> 00:42:39,000 Can you think, uh, so what I'm gonna do is we're gonna with, on this podcast, 864 00:42:39,000 --> 00:42:40,920 we'll sort of round out the discussion on. 865 00:42:41,820 --> 00:42:45,510 Everything sort of almost not recovery, like all of the 866 00:42:45,510 --> 00:42:48,720 things that just frustrated you being able to start a recovery. 867 00:42:48,930 --> 00:42:52,170 Can you think of anything else that falls into that category that happened to you? 868 00:42:52,955 --> 00:42:54,605 Well, I was just trying to go back. 869 00:42:54,865 --> 00:42:55,085 Um, 870 00:42:56,285 --> 00:42:58,680 I mean there were no Snickers bars available, for example. 871 00:42:59,130 --> 00:43:02,430 Um, well, I was just gonna ask you about how it was on the island itself. 872 00:43:02,430 --> 00:43:04,110 I know you had a job to do, trying to get. 873 00:43:04,955 --> 00:43:07,325 The company back up and running, but I'm sure there is also. 874 00:43:07,360 --> 00:43:08,075 Yeah, yeah. 875 00:43:08,075 --> 00:43:09,035 Well, okay on that. 876 00:43:09,095 --> 00:43:13,505 Um, so I need to go back to see the island, um, because I didn't really 877 00:43:13,505 --> 00:43:14,705 get to see the island, you know? 878 00:43:15,125 --> 00:43:17,825 Um, it was an interesting, it was an interesting journey. 879 00:43:17,825 --> 00:43:21,545 We, um, we flew down in a corporate jet, right? 880 00:43:21,545 --> 00:43:23,555 That was a trip in and of itself. 881 00:43:24,185 --> 00:43:24,545 Um, 882 00:43:25,475 --> 00:43:26,165 that must have been cool. 883 00:43:26,165 --> 00:43:26,705 Was that cool? 884 00:43:26,705 --> 00:43:27,120 That cool? 885 00:43:27,120 --> 00:43:27,320 Just tell me. 886 00:43:27,320 --> 00:43:27,640 That was cool. 887 00:43:27,640 --> 00:43:27,881 Very cool. 888 00:43:27,881 --> 00:43:27,882 Uh, 889 00:43:27,995 --> 00:43:28,775 but I'll tell you what. 890 00:43:28,805 --> 00:43:29,015 Okay. 891 00:43:29,015 --> 00:43:29,360 It, it. 892 00:43:29,595 --> 00:43:32,535 Spit me in the butt because I didn't have to go through TSA 893 00:43:32,535 --> 00:43:33,645 or any that on the way down. 894 00:43:33,645 --> 00:43:37,815 And so I forgot that I had put a couple of things in my bag, and when I was 895 00:43:38,355 --> 00:43:42,285 leaving the island, I had to actually flag commercial TSA and, and it was, 896 00:43:42,345 --> 00:43:46,905 it was fruit and it, like they take it out and you can't take it with you. 897 00:43:46,965 --> 00:43:52,065 So, um, but um, it was, it was weird. 898 00:43:53,100 --> 00:43:58,320 You know, when we landed, uh, you know, at that time, this was very 899 00:43:58,320 --> 00:44:02,370 early after the, the, um, the init, the hurricane itself, right? 900 00:44:02,370 --> 00:44:09,330 And so, um, limited flights in and out, you had to, um, you had to, they, you had 901 00:44:09,330 --> 00:44:13,590 to, basically, they had to request and were given this tiny window to land in. 902 00:44:13,590 --> 00:44:16,590 And so, boy, they had to make sure they made it on time and all that. 903 00:44:17,100 --> 00:44:18,630 Um, they had a number of. 904 00:44:19,245 --> 00:44:26,265 People with the company that they were relocating to a facility on the mainland 905 00:44:26,265 --> 00:44:32,325 so that they could get online and do work in helping to bring the site back. 906 00:44:32,325 --> 00:44:32,625 Right. 907 00:44:32,985 --> 00:44:36,855 Uh, they couldn't work locally and so they were waiting when we landed and you 908 00:44:36,855 --> 00:44:42,075 know, we got off and they put us in a shuttle and then took us and there wasn't. 909 00:44:42,900 --> 00:44:48,630 I grew up in Montana and Montana in the winter, when you look out, 910 00:44:48,720 --> 00:44:52,950 um, you got the evergreens, but if there's not an evergreen, it's brown. 911 00:44:53,070 --> 00:44:54,150 It's brown, right? 912 00:44:54,150 --> 00:44:56,910 I mean, there's, there's no green, um, there. 913 00:44:57,270 --> 00:45:01,590 Um, and so this looked a lot like Montana in the winter without the snow. 914 00:45:01,595 --> 00:45:02,415 I mean, it was just. 915 00:45:02,555 --> 00:45:03,095 Brown. 916 00:45:03,095 --> 00:45:05,435 There wasn't any leaves, any green anywhere. 917 00:45:05,435 --> 00:45:06,635 It had been stripped bare. 918 00:45:06,635 --> 00:45:08,165 So it was weird seeing that. 919 00:45:09,005 --> 00:45:12,090 Oh, because it had been stripped from, from bear, yeah, from the hurricane. 920 00:45:12,090 --> 00:45:12,330 Gotcha. 921 00:45:12,330 --> 00:45:12,845 The wind just 922 00:45:12,845 --> 00:45:13,775 stripped it off. 923 00:45:14,075 --> 00:45:18,305 One of the guys that, that he, he'd worked at the facility, I 924 00:45:18,305 --> 00:45:20,075 forget, he's been like 20 years. 925 00:45:20,405 --> 00:45:23,525 He said, I'm doing this commute for the last 20 years. 926 00:45:24,220 --> 00:45:28,240 I didn't know there were houses down there on the side of the road, you 927 00:45:28,240 --> 00:45:31,540 know, because it was so lush and green, you couldn't see beyond, oh. 928 00:45:31,600 --> 00:45:33,970 And he says, I'm seeing stuff that I didn't even know was there. 929 00:45:33,970 --> 00:45:39,250 But anyways, um, and then we get there and, you know, there's, they 930 00:45:39,250 --> 00:45:40,960 basically put us up in conference room. 931 00:45:40,960 --> 00:45:44,035 They turned conference rooms into, um, um. 932 00:45:45,360 --> 00:45:47,910 Just, uh, dorm, right? 933 00:45:48,090 --> 00:45:54,390 And we had, um, um, they had these little air mattresses and, um, I brought, uh, 934 00:45:54,420 --> 00:45:57,450 they told us, you know, bring a bag and, and so I brought like a sleeping 935 00:45:57,450 --> 00:45:58,740 bag, you know, change of clothes. 936 00:45:59,340 --> 00:46:03,780 Uh, and, uh, pretty much spent, well, the entire time I was 937 00:46:03,780 --> 00:46:06,120 there, um, I spent it on. 938 00:46:06,960 --> 00:46:07,890 On the facility. 939 00:46:07,920 --> 00:46:10,260 'cause there was nothing to go, no place to really go. 940 00:46:10,620 --> 00:46:14,490 Um, for, for the locals, you know, there were long lines for everything. 941 00:46:15,030 --> 00:46:19,590 Um, and so it just didn't seem right to go getting long lines for those people. 942 00:46:19,860 --> 00:46:25,770 Um, they, they were providing us with, um, you know, the food, um, and it was 943 00:46:25,770 --> 00:46:31,830 a lot of, um, chicken and rice, but, um, you know, you'll survive on, on, on it. 944 00:46:31,830 --> 00:46:35,490 So they, they took the, the company took very. 945 00:46:35,925 --> 00:46:38,625 Good care I thought of, of the people. 946 00:46:38,925 --> 00:46:45,135 Um, and they also were taking real good care of, uh, local workers, um, who could 947 00:46:45,135 --> 00:46:50,445 not come back in, um, to the facility, you know, just because they, they weren't, 948 00:46:51,105 --> 00:46:54,375 their stuff wasn't yet recovered, you know, because there's some manufacturing 949 00:46:54,375 --> 00:46:55,072 goes on there, and so they weren't. 950 00:46:56,070 --> 00:46:57,315 It's ready to do manufacturing. 951 00:46:57,960 --> 00:46:59,160 Um, but they took care of 'em. 952 00:46:59,160 --> 00:47:05,850 They shipped in, um, brought in pallets, pallets full of, of, uh, bulk food 953 00:47:05,850 --> 00:47:09,690 and whatnot and, and, uh, had workers come in, you know, and they just 954 00:47:09,690 --> 00:47:12,840 basically parceled 'em out and helped supplemented a lot of their stuff. 955 00:47:12,840 --> 00:47:13,890 So I thought that was really good. 956 00:47:13,890 --> 00:47:17,820 Their, their response overall in taking care of their, their, their local 957 00:47:17,820 --> 00:47:21,150 employees, that I thought was really well, and, you know, and, and they did 958 00:47:21,150 --> 00:47:23,070 good by us as far as, as, you know. 959 00:47:24,210 --> 00:47:27,390 Under the circumstances, you know, we didn't go hungry and we, you know, 960 00:47:27,390 --> 00:47:29,460 we had a place to sleep, but it was. 961 00:47:29,505 --> 00:47:37,195 FI was there for almost two weeks and it was pretty much, um, you know, 12, 14, 962 00:47:37,195 --> 00:47:39,585 16 hour days in the, in the beginning. 963 00:47:39,585 --> 00:47:42,465 And then it tapered down after about the first week. 964 00:47:42,915 --> 00:47:46,755 Um, and then it was, we were down to sort of eight hour days. 965 00:47:46,755 --> 00:47:47,745 But then you just. 966 00:47:47,985 --> 00:47:49,635 Sitting around a lot of time. 967 00:47:50,175 --> 00:47:53,715 And so since there was nothing to do, we just worked 968 00:47:54,555 --> 00:47:55,005 like, what else? 969 00:47:55,005 --> 00:47:55,305 What else do you do? 970 00:47:55,305 --> 00:47:55,485 Yeah. 971 00:47:55,485 --> 00:47:57,030 When you're in that situation, you know? 972 00:47:57,030 --> 00:47:57,270 Yeah. 973 00:47:57,525 --> 00:47:58,665 Um, subsequent you're not 974 00:47:58,665 --> 00:47:59,535 watching Netflix. 975 00:47:59,595 --> 00:48:00,015 Yeah. 976 00:48:00,165 --> 00:48:05,175 Subsequent teams that came down, um, by, by that time, um, they were 977 00:48:05,175 --> 00:48:08,205 actually coming down on commercial, you know, 'cause commercial, 978 00:48:08,295 --> 00:48:09,675 um, flights had started to pick. 979 00:48:09,680 --> 00:48:12,470 Back up as they had, um, kind of cleared stuff up in that. 980 00:48:12,950 --> 00:48:16,220 Uh, and then they had, um, rented out hotels, uh, space for 981 00:48:16,220 --> 00:48:18,320 them and were shuttling them. 982 00:48:18,590 --> 00:48:23,420 Um, so they weren't staying on the, on the, on the, the campus itself. 983 00:48:23,570 --> 00:48:26,510 And, but so, but it was kind of fun in one sense. 984 00:48:26,510 --> 00:48:30,920 Um, I did a, um, the only exercise I could really get there was just a lot of 985 00:48:30,920 --> 00:48:36,050 walking and I walked around all over the campus just to look at the damage and, 986 00:48:36,620 --> 00:48:38,540 you know, different parts of it had had. 987 00:48:39,285 --> 00:48:41,685 You know, some of it unscathed. 988 00:48:42,075 --> 00:48:48,135 Um, the particular building that the, the data center B was in, um, was one of 989 00:48:48,135 --> 00:48:50,175 the oldest buildings there on the campus. 990 00:48:50,175 --> 00:48:55,515 And so, um, it, it's does, you know, not surprising that it was the 991 00:48:55,515 --> 00:48:56,810 one that suffered the most damage. 992 00:48:58,650 --> 00:49:04,680 So, um, and, uh, but then it was kind of, you know, it was a, the small group 993 00:49:04,680 --> 00:49:06,510 of us that were, were forced together. 994 00:49:06,510 --> 00:49:09,660 People that you'd never met before or didn't know were all kind of 995 00:49:09,660 --> 00:49:11,640 forced together to, to work together. 996 00:49:11,640 --> 00:49:13,650 And that I found that was fun. 997 00:49:13,650 --> 00:49:14,400 That was interesting. 998 00:49:14,400 --> 00:49:16,380 You know, you get to, you get to know people and it, did you 999 00:49:16,380 --> 00:49:16,825 have any. 1000 00:49:17,680 --> 00:49:19,300 Did you have any language barriers? 1001 00:49:19,360 --> 00:49:22,930 'cause I, I'm guessing you don't speak the language of the locals. 1002 00:49:22,935 --> 00:49:23,095 No. 1003 00:49:23,100 --> 00:49:23,230 No. 1004 00:49:23,230 --> 00:49:26,920 But, but, um, you know, um, everybody within the facility 1005 00:49:26,920 --> 00:49:29,380 is, is fluent, uh, English. 1006 00:49:29,410 --> 00:49:29,920 It's okay. 1007 00:49:29,920 --> 00:49:34,480 You know, because it's a, it's, um, a mainland company, you know, headquarters. 1008 00:49:34,480 --> 00:49:34,510 Okay. 1009 00:49:35,050 --> 00:49:37,750 Um, so, so they're all, they're all bilingual. 1010 00:49:37,885 --> 00:49:40,735 Um, which puts them way up the ladder on than me, you know? 1011 00:49:41,485 --> 00:49:43,465 Um, I'm always, I'm always, yeah. 1012 00:49:43,525 --> 00:49:47,065 Um, you know, my hat off to anybody that speaks more than one 1013 00:49:47,065 --> 00:49:48,505 language as far as I'm concerned. 1014 00:49:49,195 --> 00:49:50,155 Um, absolutely. 1015 00:49:50,155 --> 00:49:50,215 Yeah. 1016 00:49:50,515 --> 00:49:51,290 So, persona, 1017 00:49:51,750 --> 00:49:52,435 how many you got? 1018 00:49:52,435 --> 00:49:52,975 Persona. 1019 00:49:53,305 --> 00:49:54,805 I can understand other languages. 1020 00:49:54,805 --> 00:49:55,885 I don't necessarily speak a. 1021 00:49:56,415 --> 00:49:57,375 Still, that's still better. 1022 00:49:57,750 --> 00:49:58,040 Yeah. 1023 00:49:58,040 --> 00:49:58,320 Mm-hmm. 1024 00:49:58,425 --> 00:49:58,665 Yeah. 1025 00:49:58,665 --> 00:50:03,015 So yeah, so it was a very, very interesting experience. 1026 00:50:03,075 --> 00:50:06,525 Um, overall, I'll tell you, you know, it, it was, uh, not that, not 1027 00:50:06,525 --> 00:50:11,085 that I wanna have to do it again, but 'cause of the circumstances. 1028 00:50:11,085 --> 00:50:11,235 But 1029 00:50:11,775 --> 00:50:15,015 you got to really do sort of the worst case scenario of what 1030 00:50:15,045 --> 00:50:17,265 a lot of us, um, prepare for. 1031 00:50:17,890 --> 00:50:18,970 Prepare for. 1032 00:50:19,090 --> 00:50:22,420 And uh, that's why I really wanted to have you on Yeah. 1033 00:50:22,425 --> 00:50:23,050 And, and talk to you. 1034 00:50:23,050 --> 00:50:26,230 And by the way, we're gonna have you back 'cause you, you've got a lot of 1035 00:50:26,380 --> 00:50:29,410 institutional knowledge up there of what it's like to actually do this 1036 00:50:29,410 --> 00:50:31,300 in, you know, in the real world. 1037 00:50:31,450 --> 00:50:34,180 Um, so we're definitely gonna have you back, but, uh, I'm gonna end 1038 00:50:34,180 --> 00:50:38,080 this one now because we try to keep these around 45 minutes or so. 1039 00:50:38,470 --> 00:50:40,030 We're definitely gonna have you back. 1040 00:50:40,030 --> 00:50:40,720 Would that be alright? 1041 00:50:41,170 --> 00:50:42,055 Oh, sure, sure. 1042 00:50:42,055 --> 00:50:42,056 Yeah. 1043 00:50:42,935 --> 00:50:48,245 You know, having spent most of my career preparing for disasters and, 1044 00:50:48,245 --> 00:50:54,455 and running a couple of, uh, of, you know, tests, uh, to actually be. 1045 00:50:54,855 --> 00:50:58,815 Be able to participate in a full on recovery of a, of a facility. 1046 00:50:59,745 --> 00:50:59,925 Yeah. 1047 00:50:59,925 --> 00:51:06,735 That was a, a, a highlight, you know, I mean, again, if you don't want those 1048 00:51:06,735 --> 00:51:10,395 kinds of things to happen, but to be able to be a part of one was, yeah. 1049 00:51:10,395 --> 00:51:13,305 That's something that I, I'm really proud of. 1050 00:51:13,725 --> 00:51:14,925 This was fascinating. 1051 00:51:14,925 --> 00:51:19,455 I I, I, it's interesting right at the end there, we got the image of you on a, on 1052 00:51:19,455 --> 00:51:21,195 an air mattress with a sleeping bag bag. 1053 00:51:22,665 --> 00:51:25,665 Uh, that, that, that, that added like a whole, I didn't even, I didn't even 1054 00:51:25,665 --> 00:51:28,485 think about that part of like, you know, that you needed a place to sleep 1055 00:51:28,575 --> 00:51:33,405 and those places weren't Exactly, uh, in the, they were in short supply. 1056 00:51:34,035 --> 00:51:38,115 Anyway, so, um, thanks so much for, for coming on. 1057 00:51:38,445 --> 00:51:38,535 Sure. 1058 00:51:39,540 --> 00:51:40,440 Thanks, persona. 1059 00:51:40,500 --> 00:51:41,130 Thanks Curtis. 1060 00:51:41,130 --> 00:51:41,790 Thank you, Ron. 1061 00:51:41,790 --> 00:51:42,480 I, yeah, pleasure. 1062 00:51:42,480 --> 00:51:45,900 Like Curtis said, I just have that image, image of you in an air mattress 1063 00:51:45,900 --> 00:51:51,510 with a bowl of chicken and rice with, uh, lights flickering, kind 1064 00:51:51,510 --> 00:51:53,400 of like an apocalypse like scene. 1065 00:51:53,670 --> 00:51:57,600 And the data center slowly coming back, online's like Jurassic Park almost. 1066 00:51:59,005 --> 00:52:00,415 I know that system. 1067 00:52:00,865 --> 00:52:05,905 All right, and with that, I wanna thank the listeners for your attention and, 1068 00:52:05,905 --> 00:52:08,005 uh, and, and listening all the time. 1069 00:52:08,465 --> 00:52:09,875 That is a wrap.