1 00:00:00,000 --> 00:00:03,929 You've found the backup wrap-up your go-to podcast for all things, 2 00:00:03,929 --> 00:00:06,420 backup recovery and cyber recovery. 3 00:00:06,960 --> 00:00:10,440 This is another classic episode from a few years ago, and it's a 4 00:00:10,440 --> 00:00:12,960 companion to last week's great episode. 5 00:00:13,379 --> 00:00:17,640 This week, we hear a fascinating disaster recovery case study. 6 00:00:18,150 --> 00:00:21,120 We're talking with someone who had to do an actual Dr. 7 00:00:21,330 --> 00:00:23,880 In the worst possible circumstances. 8 00:00:24,150 --> 00:00:27,570 On an island after a hurricane took out everything. 9 00:00:28,020 --> 00:00:30,750 And when I say everything, I mean, everything. 10 00:00:31,020 --> 00:00:34,349 The data center, the infrastructure, even the trees. 11 00:00:34,680 --> 00:00:38,489 He had to sleep on an air mattress, eat chicken and rice for two weeks 12 00:00:38,820 --> 00:00:42,989 and figure out how to restore systems when basic assumptions like we 13 00:00:42,989 --> 00:00:44,940 have internet weren't true anymore. 14 00:00:45,510 --> 00:00:48,150 Last week we heard the state side version of this story. 15 00:00:48,480 --> 00:00:52,290 This week, we hear the story on the ground just like last week, we'll 16 00:00:52,290 --> 00:00:54,420 be anonymizing, his voice and name. 17 00:00:54,780 --> 00:00:58,140 This week we hear from Harry Potter's friend, Ron Weasley. 18 00:00:58,710 --> 00:01:01,240 By the way, if you don't know who I am, I'm W. 19 00:01:01,260 --> 00:01:01,560 Curtis. 20 00:01:01,560 --> 00:01:03,030 Preston AKA Mr. 21 00:01:03,030 --> 00:01:07,680 Backup, and I've been passionate about backup and recovery for over 30 years. 22 00:01:07,980 --> 00:01:11,460 Ever since I had to tell my boss that we had no backups of the 23 00:01:11,460 --> 00:01:13,470 production database, we had just lost. 24 00:01:13,890 --> 00:01:15,270 I don't want that to happen to you. 25 00:01:15,360 --> 00:01:18,240 And that's why I do this on this podcast. 26 00:01:18,240 --> 00:01:21,990 We turn unappreciated backup admins and to cyber recovery heroes. 27 00:01:22,290 --> 00:01:24,390 This is the backup wrap up. 28 00:01:27,550 --> 00:01:28,254 Welcome to the show. 29 00:01:28,254 --> 00:01:31,622 I'm your host, w Curtis Pressin, AKA, Mr. 30 00:01:31,622 --> 00:01:34,922 Backup, and I have with me my, my meat advisor. 31 00:01:35,102 --> 00:01:35,402 Sana. 32 00:01:35,942 --> 00:01:36,362 How's it going? 33 00:01:36,362 --> 00:01:36,722 Persona, 34 00:01:37,262 --> 00:01:38,672 I don't know if I'm your meat advisor. 35 00:01:38,672 --> 00:01:39,332 I think I am. 36 00:01:39,332 --> 00:01:40,502 Your, uh. 37 00:01:41,132 --> 00:01:42,782 Uh, what do they say? 38 00:01:42,782 --> 00:01:46,472 Sort of your apprentice or wishing to be your apprentice. 39 00:01:46,502 --> 00:01:47,492 You're, you're, yeah. 40 00:01:47,492 --> 00:01:50,672 You're not my, I'm guessing you, you have a lot of vegetarian 41 00:01:50,672 --> 00:01:51,902 dishes in your house, right? 42 00:01:51,962 --> 00:01:52,562 Yes, 43 00:01:52,562 --> 00:01:55,052 we do have a, my wife is vegetarian and Right. 44 00:01:55,052 --> 00:01:59,402 Typically we do vegetarian, and given that we eat a lot of Indian food, Indian 45 00:01:59,402 --> 00:02:02,312 food, you can make a ton of dishes without ever having to make anything meat. 46 00:02:02,672 --> 00:02:04,472 So there's a lot of variety. 47 00:02:04,977 --> 00:02:05,487 Yeah. 48 00:02:05,547 --> 00:02:09,777 Uh, so, so you're more sort of an, you're, you're, you're my meat enthusiast. 49 00:02:09,837 --> 00:02:10,257 Yes. 50 00:02:10,257 --> 00:02:10,407 How's 51 00:02:10,407 --> 00:02:10,707 that? 52 00:02:10,707 --> 00:02:12,807 You, you, you're meat curious. 53 00:02:12,897 --> 00:02:13,257 Yes. 54 00:02:13,257 --> 00:02:15,267 But meat curious. 55 00:02:16,557 --> 00:02:21,807 But I do want to hear about your latest adventure, because I've been, I've 56 00:02:21,807 --> 00:02:25,317 been asking you, I think almost every week how your dry aging is going. 57 00:02:25,377 --> 00:02:27,297 So I know that this will probably come out later, but now that. 58 00:02:27,657 --> 00:02:29,787 Technically we're recording this after Thanksgiving. 59 00:02:29,997 --> 00:02:32,097 I wanna know how was it and what happened. 60 00:02:32,127 --> 00:02:35,307 So I'm gonna, I'm gonna bring on our guest and then, uh, and then 61 00:02:35,307 --> 00:02:36,747 we'll, we'll chitchat about that. 62 00:02:37,227 --> 00:02:44,157 So, uh, we have, uh, it's a rare treat for us because I've been in it for so long. 63 00:02:44,187 --> 00:02:49,287 Rarely do I have someone who's been in it longer than me, and this is 64 00:02:49,287 --> 00:02:53,037 one of those times and I'm super excited because, uh, he started in 65 00:02:53,037 --> 00:02:55,617 it just after I was in high school. 66 00:02:56,082 --> 00:03:00,042 And, uh, started out in the hardware, uh, side of things, actually working 67 00:03:00,282 --> 00:03:03,942 and running Digital Equipment Corporation, which we call Deck. 68 00:03:04,182 --> 00:03:09,672 Uh, their internal email service went into a, uh, it, uh, has done a lot 69 00:03:09,672 --> 00:03:12,822 of things in data center operations, system administrator, data center 70 00:03:12,822 --> 00:03:15,372 manager, and he's recently retired. 71 00:03:15,432 --> 00:03:16,422 Super jelly. 72 00:03:16,497 --> 00:03:18,837 And, and now lives in Seattle. 73 00:03:18,867 --> 00:03:24,147 He is a friend of the person that we previously had on that we called 74 00:03:24,147 --> 00:03:25,977 Harry Potter to keep him anonymous. 75 00:03:25,977 --> 00:03:29,937 And so to continue that tradition, I would like to introduce to 76 00:03:29,937 --> 00:03:32,517 the podcast, uh, Ron Weasley. 77 00:03:33,147 --> 00:03:33,297 Hello. 78 00:03:33,597 --> 00:03:34,677 It's a pleasure to be 79 00:03:34,677 --> 00:03:35,007 here. 80 00:03:35,487 --> 00:03:37,922 You know, we had, we had your friend Harry Yes. 81 00:03:38,937 --> 00:03:43,557 On and um, and, and we, we had a great podcast there. 82 00:03:43,557 --> 00:03:47,337 But, um, I, I, I, I kept asking him, I was like, do you think that the guy that 83 00:03:47,337 --> 00:03:51,057 actually was the one with the fingers on the keyboard would, would talk to us? 84 00:03:51,057 --> 00:03:52,137 And he said, yes. 85 00:03:52,137 --> 00:03:53,187 And so here you are. 86 00:03:53,517 --> 00:03:54,417 Uh, but. 87 00:03:55,047 --> 00:04:00,087 So the, before we get to that, uh, we'll get back to the, to the meat conversation. 88 00:04:00,357 --> 00:04:09,147 So Ron, I have had a, a, an ongoing sort of a project of experimenting 89 00:04:09,147 --> 00:04:11,127 with dry aging meat at home. 90 00:04:11,517 --> 00:04:16,917 And I started out with these things called the umai bags, which UMAI, 91 00:04:16,917 --> 00:04:20,247 it's short for umami and, um. 92 00:04:20,957 --> 00:04:27,557 The, and so Thanksgiving was the first time I did a dry aged brisket at home. 93 00:04:27,647 --> 00:04:31,067 I, I think the dry aging process went really well. 94 00:04:31,247 --> 00:04:34,727 I didn't quite get, and by the way, if you are a brisket fan, uh, just 95 00:04:34,727 --> 00:04:39,377 go to YouTube and type in dry aged brisket and you'll see why I was 96 00:04:39,377 --> 00:04:40,817 interested in dry aging briskets. 97 00:04:40,847 --> 00:04:41,897 'cause the weird thing is that. 98 00:04:42,757 --> 00:04:46,867 A lot of people in the dry aging slash brisket community don't think 99 00:04:46,867 --> 00:04:50,557 that briskets benefit from dry aging, but these videos suggest otherwise. 100 00:04:50,557 --> 00:04:51,757 So I tried it out. 101 00:04:52,177 --> 00:04:58,267 The problem was that I had a, uh, a noon Thanksgiving we had for those 102 00:04:58,267 --> 00:05:03,397 concerned, we had a sort of COD friendly Thanksgiving gathering, so we had. 103 00:05:03,447 --> 00:05:04,647 We had 10 people. 104 00:05:04,647 --> 00:05:05,847 We did it outside. 105 00:05:06,117 --> 00:05:07,257 We were socially distanced. 106 00:05:07,257 --> 00:05:10,347 I actually rented tables and chairs so that I could do that and I, so that we 107 00:05:10,347 --> 00:05:11,667 could, you know, follow all the rules. 108 00:05:11,967 --> 00:05:16,107 Uh, but it was at noon and I had, so that meant I had to start my 109 00:05:16,107 --> 00:05:20,577 brisket at midnight and, um, it meant that part of the brisket. 110 00:05:20,577 --> 00:05:20,847 Well, 111 00:05:21,147 --> 00:05:22,137 you're very dedicated. 112 00:05:22,677 --> 00:05:22,947 Yeah. 113 00:05:22,977 --> 00:05:25,257 Most brisket cooking was while I was sleeping. 114 00:05:26,207 --> 00:05:30,947 And, uh, let's just say the critical part is towards the end when you 115 00:05:30,947 --> 00:05:32,387 need to be checking doneness. 116 00:05:32,447 --> 00:05:37,157 And I was really not wanting to get up at five o'clock in the morning 117 00:05:37,157 --> 00:05:38,537 when this thing was really done. 118 00:05:39,227 --> 00:05:40,997 And so I kind of got up at six. 119 00:05:40,997 --> 00:05:43,307 And the difference between getting up at five and getting up at six 120 00:05:43,307 --> 00:05:45,797 is the difference between a brisket that is tender and a brisket that. 121 00:05:45,867 --> 00:05:50,307 Becomes pot roast and unfortunately, I blew my 60 day experiment. 122 00:05:51,087 --> 00:05:51,117 Mm. 123 00:05:51,117 --> 00:05:51,657 For an extra 124 00:05:51,657 --> 00:05:52,407 hour's sleep. 125 00:05:52,497 --> 00:05:56,577 So I had a, the brisket was super, super tender, super, super juicy. 126 00:05:56,577 --> 00:06:01,137 EAs easily the juiciest brisket I've ever cooked, but it was slightly overdone 127 00:06:01,437 --> 00:06:05,217 and so I was disappointed in it as the brisket maker, the people that. 128 00:06:05,962 --> 00:06:06,532 Loved it. 129 00:06:07,012 --> 00:06:10,162 Um, and so I had no complaints. 130 00:06:10,162 --> 00:06:11,812 I also had no brisket left. 131 00:06:11,872 --> 00:06:14,332 Yeah, that's when, you know, it was good when 132 00:06:14,332 --> 00:06:15,022 we were done. 133 00:06:15,682 --> 00:06:19,162 Uh, I think I grabbed a handful of it, uh, just so that I 134 00:06:19,162 --> 00:06:20,722 could, you know, eat it later. 135 00:06:20,722 --> 00:06:24,382 But yeah, we, we had a, like a 16 pound brisket that was 136 00:06:24,382 --> 00:06:26,392 completely gone from 10 people. 137 00:06:27,107 --> 00:06:29,927 Who also had a Turkey and a ham to eat. 138 00:06:30,227 --> 00:06:32,087 So, you know, I don't know. 139 00:06:32,087 --> 00:06:32,987 Was that a success? 140 00:06:32,987 --> 00:06:33,362 I, I just 141 00:06:33,362 --> 00:06:36,497 did a quick lookup of it, six to eight weeks. 142 00:06:36,497 --> 00:06:38,687 It's saying to, to do that, that's, 143 00:06:39,377 --> 00:06:39,857 yeah. 144 00:06:39,857 --> 00:06:39,922 A lot of it's time. 145 00:06:39,922 --> 00:06:40,202 It's 146 00:06:40,207 --> 00:06:40,727 dedication, 147 00:06:40,727 --> 00:06:41,117 right? 148 00:06:41,122 --> 00:06:41,282 Yeah. 149 00:06:41,282 --> 00:06:41,722 Dedication. 150 00:06:41,727 --> 00:06:42,137 Yeah. 151 00:06:42,137 --> 00:06:42,557 Really. 152 00:06:42,557 --> 00:06:42,827 Yeah. 153 00:06:43,247 --> 00:06:47,897 So I'm current, I'm currently in the process of an actual dry aging experiment 154 00:06:47,897 --> 00:06:49,997 where I have a dedicated refrigerator. 155 00:06:50,452 --> 00:06:53,962 With, uh, precise temperature and humidity control that's going 156 00:06:53,962 --> 00:06:55,702 on, like literally right now. 157 00:06:55,702 --> 00:07:00,022 It started December 1st, and I'm hoping to have the results of 158 00:07:00,022 --> 00:07:04,552 that by, by New Year's and have a New Year's, uh, dry aged brisket. 159 00:07:05,062 --> 00:07:07,732 But, um, but we don't know about that yet, so, um. 160 00:07:08,237 --> 00:07:09,797 We'll talk about that on later podcast. 161 00:07:09,847 --> 00:07:14,917 Ron, you listened to the podcast where we talked about you, right? 162 00:07:15,517 --> 00:07:15,727 Yes. 163 00:07:15,937 --> 00:07:16,477 With with Harry. 164 00:07:16,477 --> 00:07:16,837 Yes. 165 00:07:17,497 --> 00:07:17,617 Yes. 166 00:07:17,677 --> 00:07:22,537 And the, the idea was that there was a hurricane. 167 00:07:22,762 --> 00:07:26,542 That took out, so this actually happened, by the way, this is, this is, you 168 00:07:26,542 --> 00:07:29,422 know, uh, this, this is a true story. 169 00:07:30,167 --> 00:07:30,387 Yes. 170 00:07:31,342 --> 00:07:33,802 The, so a hurricane took out an island. 171 00:07:34,102 --> 00:07:35,812 An island took out an island. 172 00:07:36,502 --> 00:07:41,152 You and Harry both worked for, you know, we'll call it Hogwarts. 173 00:07:41,452 --> 00:07:43,462 Someone had to go down there. 174 00:07:44,797 --> 00:07:46,687 Do the disaster recovery. 175 00:07:47,047 --> 00:07:51,307 My understanding is that it, it was sort of a toss up between you and Harry. 176 00:07:51,307 --> 00:07:55,597 And Harry couldn't get there fast enough and you could, and so you 177 00:07:55,597 --> 00:07:56,857 were on your way to, you drew 178 00:07:56,857 --> 00:07:59,167 the lucky straw, you drew the lucky, lucky straw. 179 00:07:59,167 --> 00:07:59,317 So 180 00:07:59,317 --> 00:08:00,907 you were on, you were on your own. 181 00:08:00,907 --> 00:08:01,867 Does that sound about right? 182 00:08:02,527 --> 00:08:03,157 Yes. 183 00:08:03,157 --> 00:08:03,217 Yeah. 184 00:08:03,457 --> 00:08:03,697 Yeah. 185 00:08:04,297 --> 00:08:10,327 The um, one of the things, kind of the requirements was, um, because of. 186 00:08:10,807 --> 00:08:15,367 The nature of the response was you had to be comfortable doing command line 187 00:08:15,367 --> 00:08:17,317 recovery of the backup application. 188 00:08:17,647 --> 00:08:18,517 Why was that? 189 00:08:19,447 --> 00:08:24,907 Uh, just because you, um, you, you know, the, um, you're gonna 190 00:08:24,907 --> 00:08:28,912 be, um, on the console of the server for a lot of the Okay. 191 00:08:28,912 --> 00:08:30,097 So no gooey for you? 192 00:08:30,102 --> 00:08:32,017 No gooey, no gooey for you at the beginning? 193 00:08:32,017 --> 00:08:32,257 Yeah. 194 00:08:32,527 --> 00:08:35,677 Kind of a recap for the listeners who may not fully recall 195 00:08:35,857 --> 00:08:37,147 the episode, this is where. 196 00:08:37,827 --> 00:08:39,717 The hurricane took out the data center. 197 00:08:39,717 --> 00:08:44,547 I believe that you moved the servers into a different data center to try 198 00:08:44,547 --> 00:08:47,757 to recover, and you moved some of the backup infrastructure as well, correct? 199 00:08:48,237 --> 00:08:48,807 Right. 200 00:08:48,862 --> 00:08:53,962 So the, the way that the site had been set up was, um, that from. 201 00:08:55,462 --> 00:08:58,972 Like a, a main computing, um, standpoint. 202 00:08:58,972 --> 00:09:05,092 They had two main data centers and the design was to have, you know, half the 203 00:09:05,092 --> 00:09:11,362 capacity into one half the capacity and the other with, um, backups and copies. 204 00:09:11,592 --> 00:09:12,732 Going between the two. 205 00:09:12,942 --> 00:09:13,182 Right. 206 00:09:13,182 --> 00:09:19,542 And so we had two backup systems and we replicated between the, the between 207 00:09:19,542 --> 00:09:24,432 them and so data center A would back up half of the servers and replicate 208 00:09:24,432 --> 00:09:27,882 the data center B and B would back up the other half and replicate the A. 209 00:09:27,882 --> 00:09:29,922 So each side had. 210 00:09:30,982 --> 00:09:34,432 Copies of all of the backups for the entire site. 211 00:09:34,552 --> 00:09:34,612 Hmm. 212 00:09:36,112 --> 00:09:40,672 And, um, so site A, uh, data center A was fine. 213 00:09:41,122 --> 00:09:45,502 Um, the building that data center B was in was the one that was 214 00:09:45,502 --> 00:09:50,452 damaged and the damage to the data center, um, wasn't direct. 215 00:09:50,452 --> 00:09:53,542 It was indirect in, you know, the building was damaged, but it was water 216 00:09:53,542 --> 00:09:59,032 damage that came flooding on down and, um, flooded the racks and actually had. 217 00:09:59,077 --> 00:10:03,187 Uh, you know, a foot or so of water, uh, in the, in the floor 218 00:10:03,187 --> 00:10:04,507 of the data center itself. 219 00:10:04,897 --> 00:10:08,407 Uh, the data centers were not raised floor data centers. 220 00:10:08,407 --> 00:10:14,677 They were, um, the, the everything was, um, you know, in cable trays above 221 00:10:15,187 --> 00:10:16,657 and suspended above kind of thing. 222 00:10:16,657 --> 00:10:18,277 So, uh, it was quick. 223 00:10:18,277 --> 00:10:19,027 It sounds like a bad 224 00:10:19,027 --> 00:10:20,557 combination to have with a. 225 00:10:21,037 --> 00:10:22,832 Plug Hurricane. 226 00:10:23,077 --> 00:10:23,467 Yeah. 227 00:10:23,947 --> 00:10:28,747 Um, so like in, in, in the rack that had our equipment, so we had a, um, you know, 228 00:10:28,747 --> 00:10:36,697 backup server, um, uh, and a couple of, uh, media servers and then a storage, um, 229 00:10:36,877 --> 00:10:41,647 and um, and then a tape library and the library was at the bottom and the library 230 00:10:41,647 --> 00:10:43,807 was the one that was the damage the most. 231 00:10:44,667 --> 00:10:50,697 Um, the rest of the equipment, um, was okay, although the, some of the 232 00:10:50,697 --> 00:10:54,327 damage actually was caused by the, um. 233 00:10:55,927 --> 00:11:00,307 Desire to get the equipment quickly out of the one place into the other. 234 00:11:00,637 --> 00:11:03,997 And so when they were taking it out, they weren't as careful as they could 235 00:11:03,997 --> 00:11:05,527 have been with rails and whatnot. 236 00:11:05,527 --> 00:11:10,207 So we had some problems, um, racking the equipment up at the, at the new 237 00:11:10,207 --> 00:11:15,817 location, you know, and they took a, a small little server room and quickly 238 00:11:15,817 --> 00:11:20,557 converted it into a data center to house all of the, the facilities. 239 00:11:20,557 --> 00:11:20,857 Yeah. 240 00:11:21,937 --> 00:11:23,167 And so, um. 241 00:11:23,827 --> 00:11:28,807 So that was from the backup system standpoint, the impact on it. 242 00:11:29,257 --> 00:11:34,897 Um, because the, because of the replication, we were able to start, 243 00:11:35,377 --> 00:11:39,667 um, providing restores of, of a lot of the servers that they needed. 244 00:11:40,297 --> 00:11:40,747 Um. 245 00:11:41,527 --> 00:11:44,377 From the, from the A site, right. 246 00:11:44,887 --> 00:11:49,117 Uh, while we were recovering the, the backups system for the 247 00:11:49,117 --> 00:11:53,497 B site, we, we knew that we could do the recoveries from the A side. 248 00:11:53,527 --> 00:11:58,117 We just knew that when we started to bring back online, A wasn't gonna 249 00:11:58,117 --> 00:12:00,217 have the capacity to back it all up. 250 00:12:00,247 --> 00:12:00,457 Right. 251 00:12:00,457 --> 00:12:03,607 We needed to get, be back on for going forward. 252 00:12:03,907 --> 00:12:04,087 Right. 253 00:12:05,527 --> 00:12:06,667 So, and then we were. 254 00:12:07,042 --> 00:12:11,632 We were dealing with the fact that, um, we had to have vendors come in to work 255 00:12:11,632 --> 00:12:13,852 on their equipment and do a checkout. 256 00:12:14,242 --> 00:12:18,352 And we weren't, we didn't even fire it up until the vendor came in and certified 257 00:12:18,352 --> 00:12:22,222 that, that they were gonna continue to support it after we had done what we did. 258 00:12:22,912 --> 00:12:23,152 Um, 259 00:12:23,452 --> 00:12:24,172 oh, that's interesting. 260 00:12:24,382 --> 00:12:24,622 Yeah. 261 00:12:24,622 --> 00:12:25,312 And so we had 262 00:12:25,312 --> 00:12:28,042 to wait, um, for them to be How long, 263 00:12:28,492 --> 00:12:33,772 how long, how long do you think it was between basically you arrived? 264 00:12:34,372 --> 00:12:38,662 And you could actually start doing something from, from, well, let me 265 00:12:38,662 --> 00:12:42,382 rephrase, but you could actually start doing a restore of any kind. 266 00:12:43,072 --> 00:12:44,782 Um, we were restoring. 267 00:12:47,617 --> 00:12:51,667 I was restoring, starting to restore servers the first or second day. 268 00:12:52,147 --> 00:12:54,127 Um, okay, so you have to understand that. 269 00:12:54,127 --> 00:13:00,187 Well, um, um, so the, the initial recovery team was a team of, uh, you 270 00:13:00,187 --> 00:13:02,407 know, like, like myself for the backups. 271 00:13:02,407 --> 00:13:03,727 We had a, uh, a. 272 00:13:04,497 --> 00:13:07,527 DBA there for, for handle the databases. 273 00:13:07,527 --> 00:13:08,787 They had some network people. 274 00:13:09,237 --> 00:13:12,657 Um, and then they actually had to fly in a couple of vendors for 275 00:13:12,657 --> 00:13:15,447 the, um, um, emergency generators. 276 00:13:15,507 --> 00:13:15,597 Mm-hmm. 277 00:13:16,527 --> 00:13:21,597 Um, to, to them because, um, they were, um. 278 00:13:23,167 --> 00:13:28,327 The, the generators that put in were emergency generators, short term outage. 279 00:13:28,327 --> 00:13:32,587 Well, they found themselves faced with a long-term outage, uh, power wise, right? 280 00:13:33,097 --> 00:13:35,617 It was gonna be a long time before power was brought back. 281 00:13:35,617 --> 00:13:39,697 And so they were running their emergency generators way beyond the duty cycle. 282 00:13:39,697 --> 00:13:46,027 So they had guys in there, um, babying and keeping them going while they, uh, um. 283 00:13:46,567 --> 00:13:49,417 Came up with longer term solutions for how they were gonna power it. 284 00:13:50,647 --> 00:13:51,817 Um, so, um. 285 00:13:53,062 --> 00:13:57,832 So when they, when all this team get got together, as well as this was, 286 00:13:57,832 --> 00:14:01,402 was flowing in, and then you had the local, um, IT staff and that's 287 00:14:01,402 --> 00:14:03,352 just, just the whole site staff. 288 00:14:03,772 --> 00:14:10,312 Um, they had started to pull out their, um, recovery plan and which 289 00:14:10,342 --> 00:14:14,542 pieces of equipment that, uh, and which systems and that needed to 290 00:14:14,542 --> 00:14:17,752 come back online first, second, and third and all that kind of stuff. 291 00:14:17,752 --> 00:14:17,992 Right? 292 00:14:18,292 --> 00:14:19,738 Oh, so they actually had a runbook and a. 293 00:14:20,047 --> 00:14:21,727 Plan put in place ahead of time to, to some 294 00:14:21,727 --> 00:14:22,057 degree. 295 00:14:22,057 --> 00:14:22,387 Yeah. 296 00:14:22,387 --> 00:14:22,627 Yeah. 297 00:14:22,627 --> 00:14:24,637 I mean, they, they kind of knew, you know, based on the business. 298 00:14:24,642 --> 00:14:24,772 What did 299 00:14:24,772 --> 00:14:25,567 that, what did that look like? 300 00:14:26,347 --> 00:14:27,457 Was it actually, I never 301 00:14:27,457 --> 00:14:29,617 saw it myself, you know? 302 00:14:30,247 --> 00:14:30,427 Right. 303 00:14:30,487 --> 00:14:35,857 Um, but I do know that one of the things it was kind of interesting 304 00:14:35,857 --> 00:14:42,337 is that, you know, the run book, um, that, that, uh, that they had was this. 305 00:14:42,977 --> 00:14:46,487 Somewhat abstract thing at the time because, you know, they thought 306 00:14:46,487 --> 00:14:49,907 about it and they planned and they, they tested little pieces of it, 307 00:14:49,907 --> 00:14:54,347 but they never tested it in its entirety, like the disaster presented. 308 00:14:54,587 --> 00:14:57,287 You know, you know, we, we talk about that all the time. 309 00:14:57,292 --> 00:14:57,582 Mm-hmm. 310 00:14:57,662 --> 00:14:58,622 Uh, Ron, that, that. 311 00:14:59,662 --> 00:15:01,642 That, that, that's exactly the same thing. 312 00:15:01,642 --> 00:15:06,082 Back in the day when I was firing backups and anger, when we did a DR test at the 313 00:15:06,082 --> 00:15:11,842 bank that I was at, that we never tested, because, you know, without the cloud, 314 00:15:11,842 --> 00:15:16,882 without virtualization and, you know, and, and additional hardware or whatever, doing 315 00:15:16,882 --> 00:15:20,662 a full DR test is ridiculously difficult. 316 00:15:20,662 --> 00:15:21,022 Right. 317 00:15:21,022 --> 00:15:21,652 It is. 318 00:15:21,992 --> 00:15:22,352 It's, 319 00:15:22,352 --> 00:15:26,162 and, and so no one, no one does that, no one tests their whole runbook, um, right. 320 00:15:26,222 --> 00:15:29,372 Or, well, well, now, I think now more people do. 321 00:15:29,372 --> 00:15:31,472 But, so it sounds like you had that problem. 322 00:15:31,472 --> 00:15:35,702 So they, they had this runbook, but it was primarily in theory up until, 323 00:15:36,872 --> 00:15:37,262 right. 324 00:15:37,532 --> 00:15:39,637 Because what they learned is, is, you know, as, as. 325 00:15:41,122 --> 00:15:45,262 They learned that they had made certain assumptions that they shouldn't have made. 326 00:15:46,132 --> 00:15:46,192 Hmm. 327 00:15:46,342 --> 00:15:48,412 And like one of the things, go ahead. 328 00:15:48,622 --> 00:15:48,712 Yeah. 329 00:15:48,742 --> 00:15:51,442 Well, like for instance, active directory design. 330 00:15:51,892 --> 00:15:52,192 Okay. 331 00:15:52,192 --> 00:15:56,272 So the active directory for the island was tied to the mainland. 332 00:15:57,307 --> 00:15:58,897 To the, to the corporate data center. 333 00:15:59,107 --> 00:16:03,187 And when they broke the link to the corporate data center, they were, it's 334 00:16:03,187 --> 00:16:05,557 like, oh, we can't authenticate anything. 335 00:16:05,557 --> 00:16:07,957 So they had this, that's one of the first things they had to bring up. 336 00:16:07,957 --> 00:16:08,197 Right. 337 00:16:08,677 --> 00:16:10,507 So, so help me understand there. 338 00:16:10,507 --> 00:16:15,067 So when, when the hurricane hit, basically they lost connection to the mainland? 339 00:16:15,217 --> 00:16:15,457 Yeah. 340 00:16:16,207 --> 00:16:16,627 Okay. 341 00:16:16,717 --> 00:16:20,287 So they had to, they had to do all of this locally. 342 00:16:20,797 --> 00:16:22,297 Um, interesting. 343 00:16:22,627 --> 00:16:22,777 Yeah. 344 00:16:23,197 --> 00:16:23,857 You know, it's interesting. 345 00:16:23,857 --> 00:16:27,337 You would think that they would not make that assumption being an island. 346 00:16:28,087 --> 00:16:29,617 Right, right. 347 00:16:29,917 --> 00:16:37,477 Well, and, and, and see, uh, you know, um, a lot of it was driven by the 348 00:16:37,477 --> 00:16:41,167 experience of the local, you know, the local experience on the island. 349 00:16:41,197 --> 00:16:47,977 And, um, the, the prior hurricanes that they had had, had not been as devastating 350 00:16:47,977 --> 00:16:49,477 as that particular one that hit them. 351 00:16:50,227 --> 00:16:53,227 And so they were able to make it through without. 352 00:16:53,707 --> 00:16:58,477 The kinds of, of impact and losses that they, they, um, did, I mean, it walked, 353 00:16:58,717 --> 00:17:00,097 walked right up the middle of the island. 354 00:17:00,097 --> 00:17:01,387 I mean, it just devastated them. 355 00:17:02,017 --> 00:17:04,807 Um, and so, you know, so they were dealing with that. 356 00:17:04,807 --> 00:17:08,287 And then one of the other interesting things that, it took a interesting 357 00:17:08,287 --> 00:17:11,017 how it takes a while for you to figure out what's going on. 358 00:17:11,407 --> 00:17:16,387 They had this, um, um, satellite communication hookup, 359 00:17:16,387 --> 00:17:17,827 which was their fallback. 360 00:17:18,307 --> 00:17:21,427 And so that was the way they were talking between, um. 361 00:17:22,657 --> 00:17:27,397 Uh, you know, from, from the facility there to the, to the corporate end of it. 362 00:17:27,997 --> 00:17:35,767 And every day, like around noon or so, the, um, connections 363 00:17:35,767 --> 00:17:37,087 would start dropping off. 364 00:17:37,507 --> 00:17:41,347 I was using it to try to, to phone home, you know, in the afternoons, couldn't 365 00:17:41,347 --> 00:17:43,297 get a, uh, a connection or anything. 366 00:17:43,657 --> 00:17:45,067 And they were like, what's going on? 367 00:17:45,067 --> 00:17:49,057 It was working fine in the morning, but in the afternoon and one of the 368 00:17:49,057 --> 00:17:50,197 network guys started poking around. 369 00:17:50,227 --> 00:17:50,677 Oh. 370 00:17:51,292 --> 00:17:53,752 You know, and he actually called the dish mo slightly. 371 00:17:54,502 --> 00:17:59,752 Well, what it was was that it was an emergency and um, it was supposed to 372 00:17:59,752 --> 00:18:02,092 be for emergency only, short term. 373 00:18:02,092 --> 00:18:04,102 Again, one of these short term kinds of things. 374 00:18:04,252 --> 00:18:07,132 And they had turned it into their main network link. 375 00:18:07,672 --> 00:18:12,472 Well it was, um, it was a metered thing 'cause it was shared by all, you know, 376 00:18:12,472 --> 00:18:14,152 all the other emergency equipment. 377 00:18:14,482 --> 00:18:16,192 So they would use up their full. 378 00:18:16,747 --> 00:18:20,587 A day's allotment by noon or, or even earlier sometimes. 379 00:18:21,067 --> 00:18:21,337 Right. 380 00:18:21,667 --> 00:18:23,797 Also, they had, had, they had a bandwidth allotment up. 381 00:18:23,797 --> 00:18:23,977 Yeah. 382 00:18:23,977 --> 00:18:26,047 And so then they were meter in the afternoon. 383 00:18:26,167 --> 00:18:26,317 Right. 384 00:18:26,647 --> 00:18:28,507 And so they had to, they had to work. 385 00:18:28,507 --> 00:18:29,722 That was one huge, you 386 00:18:29,722 --> 00:18:30,762 know, you know what this reminds me of? 387 00:18:30,902 --> 00:18:33,127 You know, it's, you have unlimited bandwidth. 388 00:18:33,367 --> 00:18:34,657 Just don't use too much of it. 389 00:18:34,867 --> 00:18:37,297 Just, yeah, just don't use, don't use it all. 390 00:18:37,297 --> 00:18:37,597 You hit your 391 00:18:37,597 --> 00:18:39,187 data cap and we're gonna meter you. 392 00:18:39,847 --> 00:18:40,117 Wow. 393 00:18:40,777 --> 00:18:40,837 Okay. 394 00:18:40,837 --> 00:18:40,882 So yeah. 395 00:18:40,887 --> 00:18:44,647 So they, that was a huge thing for, for the networks was to work on, 396 00:18:44,917 --> 00:18:49,297 um, and work with, uh, you know, the vendors and that to try to get, um. 397 00:18:50,317 --> 00:18:54,787 A, a, a reliable, fast enough connection and multiple connections. 398 00:18:55,267 --> 00:18:59,467 Um, I, since I'm no longer, um, working with them, I don't 399 00:18:59,467 --> 00:19:01,147 know the, the end results. 400 00:19:01,147 --> 00:19:07,117 I do know that they were, um, headed towards several, um, um, different 401 00:19:07,117 --> 00:19:09,727 microwave connections from the facility. 402 00:19:10,222 --> 00:19:15,622 To, um, you know, to, to a, a main link that would then take them to the mainland. 403 00:19:16,132 --> 00:19:20,212 Um, running their dedicated link themselves to the mainland was, 404 00:19:20,242 --> 00:19:21,832 you know, just cost prohibited. 405 00:19:22,492 --> 00:19:23,422 Uh, you know, it's interesting. 406 00:19:23,422 --> 00:19:28,162 I I, I did wanna just mention, I, I'm wondering the degree to which this 407 00:19:28,192 --> 00:19:32,872 new, so Elon Musk has now come out with this, uh, I mean, they're right. 408 00:19:32,872 --> 00:19:38,032 They're in beta right now, and it's a completely redesigned way to do. 409 00:19:38,737 --> 00:19:42,462 Satellite based internet connections, starlink, where they have starlink, 410 00:19:42,542 --> 00:19:47,707 starlink, where they starlink, where they have all of these, um, satellites in. 411 00:19:48,157 --> 00:19:52,627 Um, it's a different kind of orbit, I guess than than usual. 412 00:19:52,627 --> 00:19:56,737 And, and they're saying that they can actually get both bandwidth 413 00:19:56,802 --> 00:20:05,167 and latency equal to and or better than, um, what you can do. 414 00:20:05,542 --> 00:20:11,062 On land, and so it, it, it's, I I don't know the degree to, I don't 415 00:20:11,062 --> 00:20:13,432 know how much it scales up mm-hmm. 416 00:20:13,672 --> 00:20:17,272 For a data center connection, but it, it, it's just, it's just the interesting, 417 00:20:17,542 --> 00:20:23,002 you know, thoughts towards the future for things like islands that are, 418 00:20:23,242 --> 00:20:24,967 you know, cut off the way they are. 419 00:20:24,967 --> 00:20:25,247 Mm-hmm. 420 00:20:25,402 --> 00:20:29,482 I like, even, even, you know, the, this, the microwave connection. 421 00:20:30,217 --> 00:20:35,017 Like how reliable is the main connection that they're connecting to there? 422 00:20:35,017 --> 00:20:35,227 Right. 423 00:20:35,227 --> 00:20:39,817 It's probably pretty reliable, but what if it wasn't right? 424 00:20:39,817 --> 00:20:40,747 What if that went down? 425 00:20:40,747 --> 00:20:41,797 Then you're, you're really right. 426 00:20:41,857 --> 00:20:46,837 Well, and, and what they were running into there was like, um, remote, 427 00:20:47,107 --> 00:20:49,927 I, I'm gonna use transceivers, but maybe that's not the right word. 428 00:20:49,927 --> 00:20:53,737 But you have a, um, a, a remote tower that is. 429 00:20:54,967 --> 00:20:57,037 Relaying, you know, it's a relay. 430 00:20:57,667 --> 00:21:01,087 Um, and so, you know, it's, it's receiving a signal and you know, 431 00:21:01,087 --> 00:21:02,587 because microwave is line of sight. 432 00:21:02,587 --> 00:21:07,027 And so if you're gonna go, um, you know, over mountainous things, you 433 00:21:07,027 --> 00:21:10,567 have to send it between a series of towers to get it to where you wanna go. 434 00:21:11,047 --> 00:21:15,667 And, um, they were finding out that they would lose a tower. 435 00:21:15,697 --> 00:21:17,707 And when they go out and look, well, somebody had gone 436 00:21:17,707 --> 00:21:19,087 out and stolen all the gas. 437 00:21:19,842 --> 00:21:23,532 Outta the generator because gas was like hard to get. 438 00:21:23,532 --> 00:21:23,982 Oh, yeah. 439 00:21:24,552 --> 00:21:29,832 Um, and then, um, or, or people were ripping up, um, you know, 'cause power 440 00:21:29,832 --> 00:21:33,492 lines and all the, all the sound people were ripping out the copper, you know, 441 00:21:33,492 --> 00:21:37,782 so, so you run into that kinda stuff where that just made the recovery 442 00:21:37,782 --> 00:21:43,932 effort, you know, external to the site hard, um, which then impacted the site. 443 00:21:44,447 --> 00:21:44,737 Yeah. 444 00:21:46,032 --> 00:21:46,092 Yeah. 445 00:21:47,527 --> 00:21:50,407 You were dealing, you were dealing, in this case, you, you had 446 00:21:50,407 --> 00:21:52,897 somewhat of a perfect storm where you're dealing with the fact that 447 00:21:52,897 --> 00:21:54,067 your data centers were flooded. 448 00:21:54,067 --> 00:21:57,037 You weren't, uh, you didn't have a raised floor. 449 00:21:57,127 --> 00:21:58,957 Uh, so that makes that problem worse. 450 00:21:58,957 --> 00:22:01,717 And then you didn't, you know, the, the, the Dr. 451 00:22:01,717 --> 00:22:06,037 Design made assumptions that were no longer true. 452 00:22:06,037 --> 00:22:12,127 And then meanwhile, the, the things that you did have were being frustrated by. 453 00:22:12,537 --> 00:22:14,097 Other things that were, yes. 454 00:22:14,097 --> 00:22:17,547 It's like, Hey, can you, can you stop messing with the things that actually 455 00:22:17,547 --> 00:22:21,207 work while we're trying to put the data center back together over here? 456 00:22:21,867 --> 00:22:22,227 Wow. 457 00:22:22,467 --> 00:22:26,852 That's, you know, we, we, we live in backup land and we think of the, the, 458 00:22:26,997 --> 00:22:29,997 the restore is the part we focus on. 459 00:22:29,997 --> 00:22:34,047 But it sounds like most of the problems that you were experiencing 460 00:22:34,047 --> 00:22:38,097 had nothing to do with the actual act of getting data from. 461 00:22:38,482 --> 00:22:39,982 Storage devices to server? 462 00:22:40,042 --> 00:22:40,492 No, 463 00:22:40,492 --> 00:22:45,952 actually that part of it went, um, went well in, in the areas. 464 00:22:46,252 --> 00:22:52,102 The, um, the issues that we ran into was, um, like you said, the, um, the 465 00:22:52,102 --> 00:22:58,822 disconnect between the design or intent and the reality as it, as it unfolded. 466 00:22:59,332 --> 00:23:03,982 Um, you know, they'd say, oh, so we need, you know, this server, uh. 467 00:23:04,402 --> 00:23:05,482 Brought back online. 468 00:23:05,842 --> 00:23:08,597 Um, so find the most recent backup and you start looking through. 469 00:23:08,597 --> 00:23:08,877 Mm-hmm. 470 00:23:08,957 --> 00:23:12,022 And it's like, oh, we're not backing that up. 471 00:23:12,052 --> 00:23:14,032 You know, when did you bring that online? 472 00:23:14,212 --> 00:23:15,742 Why didn't you tell us? 473 00:23:15,742 --> 00:23:16,102 You know? 474 00:23:16,612 --> 00:23:21,317 Um, and so we ran into a few of those where, um, we couldn't give them, uh. 475 00:23:22,012 --> 00:23:22,462 Yes. 476 00:23:22,492 --> 00:23:24,862 Oh, that, that is one of the most frustrating. 477 00:23:24,862 --> 00:23:31,702 So, um, and then, um, well, and, and, and then, you know, or we're backing it 478 00:23:31,702 --> 00:23:35,872 up, but we're not including everything, you know, so, um, we're backing it up 479 00:23:35,872 --> 00:23:41,152 in the sense that we're doing an OS backup, but the data that's on it, you 480 00:23:41,152 --> 00:23:43,432 didn't include it, you know, so the, um. 481 00:23:45,292 --> 00:23:48,247 So thi this was a net backup shop, so you were not using all local 482 00:23:48,247 --> 00:23:49,867 drives, is what you're saying. 483 00:23:49,872 --> 00:23:56,437 And, and it was funny that when I first hired on there, came on board, um, 484 00:23:56,947 --> 00:24:03,997 they were in the middle of, um, of a, um, upgrade and kind of a transition. 485 00:24:04,357 --> 00:24:08,288 Uh, the, the manager in charge of the backups and, and 486 00:24:08,293 --> 00:24:11,557 whatnot at the time, um, was. 487 00:24:11,647 --> 00:24:17,737 It was pushing this, we, we should only back up what we really need. 488 00:24:17,797 --> 00:24:23,407 And so he was trying to push the, uh, I hate that responsibility onto the 489 00:24:23,407 --> 00:24:27,487 data owners and saying, you need to define what's important to you and 490 00:24:27,487 --> 00:24:28,777 let us know and we'll back it up. 491 00:24:29,047 --> 00:24:31,087 So they stopped all local drives. 492 00:24:31,117 --> 00:24:31,207 Right. 493 00:24:32,107 --> 00:24:33,157 And um. 494 00:24:35,367 --> 00:24:38,692 I, you know, here's the thing, I, I don't, I don't disagree with the idea 495 00:24:38,692 --> 00:24:40,372 of not backing up worthless data. 496 00:24:40,372 --> 00:24:41,842 I don't disagree with that idea. 497 00:24:42,022 --> 00:24:47,687 What I disagree with is the implementation of, okay, so it, what I believe is mm-hmm. 498 00:24:47,772 --> 00:24:51,832 You should identify what is worthless and then we will exclude that. 499 00:24:52,612 --> 00:24:53,002 Right. 500 00:24:53,092 --> 00:24:55,822 Not identify what is valuable that Right. 501 00:24:55,822 --> 00:24:57,712 And then we will in include that. 502 00:24:57,847 --> 00:25:01,042 I, I just, I just disagree with that, the implementation and just 503 00:25:01,042 --> 00:25:02,572 because you always forget things. 504 00:25:03,232 --> 00:25:05,662 Well, you always forget things and then you, you add things. 505 00:25:05,662 --> 00:25:07,552 It's just like the server that he talked about, right? 506 00:25:07,552 --> 00:25:12,322 You end up adding file systems and the, the, I, you know, I go back to 507 00:25:12,322 --> 00:25:17,662 when I helped redesign the, the backup system for a broadcasting company. 508 00:25:18,232 --> 00:25:22,102 Uh, they were 20 terabytes as I recall when I got there. 509 00:25:22,477 --> 00:25:24,607 And one of the things they weren't doing is that they 510 00:25:24,607 --> 00:25:25,927 weren't doing all local drives. 511 00:25:25,927 --> 00:25:28,297 And I pushed really hard that we should do all local drives as 512 00:25:28,297 --> 00:25:32,347 part of the redesign, we did it, we discovered 10 more terabytes. 513 00:25:32,437 --> 00:25:32,587 Yeah. 514 00:25:33,132 --> 00:25:36,397 Of, of data that they weren't, that they weren't backing it up. 515 00:25:36,397 --> 00:25:38,197 And yes, it was really valuable data. 516 00:25:38,947 --> 00:25:39,427 Right. 517 00:25:39,457 --> 00:25:42,487 Well what, what I found interesting, um. 518 00:25:43,447 --> 00:25:47,227 Through that whole process of, you know, coming on board with that, um, 519 00:25:47,227 --> 00:25:52,567 trying to push that idea of, of the data owners being responsible to identify 520 00:25:52,567 --> 00:25:56,497 and we back up what they say to then we have to do with this recovery. 521 00:25:56,557 --> 00:25:56,887 Right. 522 00:25:57,397 --> 00:25:57,847 Um, I. 523 00:25:59,317 --> 00:26:00,727 My experience. 524 00:26:01,267 --> 00:26:01,597 Yeah. 525 00:26:01,792 --> 00:26:02,202 Well, yeah. 526 00:26:02,497 --> 00:26:05,467 And my experiences in who, whose fault has it been Ron, over the, over 527 00:26:05,467 --> 00:26:09,037 the time that I've been involved in, you know, the system side of things. 528 00:26:09,457 --> 00:26:14,677 Um, and watching, watching the way things unfold over time is that is 529 00:26:14,677 --> 00:26:18,667 really good in theory, that approach of making the, the data owners 530 00:26:18,667 --> 00:26:20,317 responsible for identifying it. 531 00:26:20,887 --> 00:26:22,447 But what I have. 532 00:26:23,742 --> 00:26:25,362 Learned and watched has been. 533 00:26:25,362 --> 00:26:31,932 The real weakness of that is, um, you know, data owners come and go and 534 00:26:32,022 --> 00:26:35,052 you don't get the transition between. 535 00:26:35,662 --> 00:26:38,902 An outgoing data owner and incoming data owner of what's 536 00:26:38,902 --> 00:26:40,042 covered, what's not covered. 537 00:26:40,492 --> 00:26:47,122 Um, you know, I know that I struggled all through my time, um, in, in, you 538 00:26:47,122 --> 00:26:53,602 know, um, it thing of doing proper and adequate documentation so that if I were 539 00:26:53,602 --> 00:26:58,552 hit by a bus as, as the saying goes, um, somebody would know what was, you 540 00:26:58,552 --> 00:27:00,862 know, what to do when they stepped in. 541 00:27:01,342 --> 00:27:04,252 Um, but I don't think that that was. 542 00:27:04,612 --> 00:27:06,112 Happening across the board. 543 00:27:06,112 --> 00:27:10,462 And so, you know, you'd have an assistant was brought online, you know, like four 544 00:27:10,462 --> 00:27:14,182 years ago you had a very conscientious data owner who identified all this 545 00:27:14,182 --> 00:27:19,042 stuff and it's covered well Over those four years that guy moved on and 546 00:27:19,042 --> 00:27:21,682 somebody else not so dedicated, came in. 547 00:27:21,892 --> 00:27:24,442 Some changes were made that weren't documented. 548 00:27:24,442 --> 00:27:27,292 That weren't covered, you know, and so then you have, 549 00:27:27,442 --> 00:27:29,812 you fast forward to, to the. 550 00:27:30,187 --> 00:27:33,637 Disaster and you have a situation well, well, we are only backing 551 00:27:33,637 --> 00:27:36,907 up part of it, you know, and, and 552 00:27:37,957 --> 00:27:42,757 yeah, this is why, or this is why I'm a strong proponent, a, a of 553 00:27:42,757 --> 00:27:48,037 virtualization and B, B, b, the reason with virtualization, it, it helps 554 00:27:48,037 --> 00:27:51,607 to, it helps to minimize this problem because really all we have to do is make 555 00:27:51,607 --> 00:27:53,812 sure we understand, we know about new. 556 00:27:54,362 --> 00:27:58,892 You know, VMware servers or HyperV servers, and then you can, you 557 00:27:58,892 --> 00:28:02,612 can tell your backup server or your backup software, uh, back 558 00:28:02,612 --> 00:28:06,872 up all VMs that show up on this thing, unless I tell you otherwise. 559 00:28:07,142 --> 00:28:10,982 And when you're backing up those VMs, backup up everything on those VMs. 560 00:28:11,132 --> 00:28:16,532 So you, you solve this problem, uh, you know, from a more global perspective. 561 00:28:16,922 --> 00:28:20,072 Um, and, but this idea. 562 00:28:20,342 --> 00:28:23,702 The, you know, we could talk for hours on all of the things that could go 563 00:28:23,702 --> 00:28:26,822 wrong with manually selecting data sets. 564 00:28:27,302 --> 00:28:30,962 Uh, and you know what, if you, if you back up a, a few terabytes of 565 00:28:30,962 --> 00:28:36,692 worthless data, that is way, that is a much smaller problem than the one 566 00:28:36,692 --> 00:28:38,882 you that this discussion started with. 567 00:28:38,882 --> 00:28:39,002 Right? 568 00:28:39,002 --> 00:28:39,182 Which is. 569 00:28:40,332 --> 00:28:40,842 Restart this. 570 00:28:40,842 --> 00:28:40,992 Yeah. 571 00:28:41,097 --> 00:28:42,517 And I'm trying to restore the server. 572 00:28:42,587 --> 00:28:42,877 Yeah. 573 00:28:43,017 --> 00:28:43,717 And it wasn't 574 00:28:43,717 --> 00:28:44,157 backed up. 575 00:28:44,157 --> 00:28:44,397 Yeah. 576 00:28:44,412 --> 00:28:45,372 Um, yeah. 577 00:28:45,372 --> 00:28:49,812 Virtualization, I think, you know, it was, it was fun to watch that come 578 00:28:49,812 --> 00:28:51,132 in and be part of that coming in. 579 00:28:51,432 --> 00:28:56,082 And it did make backups easier from a pers you know, from the 580 00:28:56,082 --> 00:28:59,412 perspective of, of, um, you got it all. 581 00:28:59,652 --> 00:29:02,352 You know, you just, I mean, the equivalent of all local drives 582 00:29:02,352 --> 00:29:04,332 was the default for, for VMs. 583 00:29:04,782 --> 00:29:07,182 One of the things that we struggled with. 584 00:29:07,362 --> 00:29:07,662 Right. 585 00:29:07,842 --> 00:29:08,262 Um. 586 00:29:08,917 --> 00:29:17,527 Uh, was that we were never able to convince and get the, um, virtualization 587 00:29:17,527 --> 00:29:25,237 team to agree upon some kind of a scheme that they would manage their 588 00:29:25,237 --> 00:29:28,712 VMs under that would allow us to do. 589 00:29:29,002 --> 00:29:31,732 What you said, um, Curtis, this, uh, the 590 00:29:31,732 --> 00:29:33,682 automatic discovery of new vm. 591 00:29:33,982 --> 00:29:34,492 Yeah, yeah. 592 00:29:34,492 --> 00:29:38,812 It's like, you know, if you had just pick something, a folder, 593 00:29:38,812 --> 00:29:40,252 a, you know, whatever kind of, 594 00:29:40,702 --> 00:29:42,412 or tags or whatever they wanted to use 595 00:29:42,652 --> 00:29:47,122 the limiter, you could do, you could do within the, the virtualization software 596 00:29:47,122 --> 00:29:51,322 to identify, you know, production machines, development machines, 597 00:29:51,322 --> 00:29:52,702 however you wanted to break 'em up. 598 00:29:52,702 --> 00:29:57,682 And then we come in and says, you know, in this, on this server, on this. 599 00:29:58,057 --> 00:30:01,627 Cluster, whatever, back up all of these types of machines, then we wouldn't 600 00:30:01,627 --> 00:30:04,837 have to worry when they added machines or took machines away, you know? 601 00:30:05,677 --> 00:30:05,767 Right. 602 00:30:06,097 --> 00:30:11,467 So, you know, we, we still had the same problems in the, with the virtual 603 00:30:11,467 --> 00:30:15,157 servers that we did, with the physical servers of them not telling us 604 00:30:15,277 --> 00:30:15,517 Right. 605 00:30:15,782 --> 00:30:20,462 The, I, I think the, the right long-term solution there is, is tags, right? 606 00:30:20,917 --> 00:30:23,792 And, and then, and then I think that there should be a policy that 607 00:30:23,792 --> 00:30:28,052 says if you have a VM that has no tags, start backing it up, put it 608 00:30:28,052 --> 00:30:29,642 in this policy, and then yell at me. 609 00:30:29,972 --> 00:30:30,302 Right? 610 00:30:30,302 --> 00:30:32,612 Because that way you can say, Hey, there's a new vm. 611 00:30:32,792 --> 00:30:35,252 And then they, then they can come to you and go, oh, that's, that's 612 00:30:35,252 --> 00:30:37,742 test or dev or whatever, and, and you can take it out right? 613 00:30:39,502 --> 00:30:42,802 It's funny how, it's funny how, no. 614 00:30:42,982 --> 00:30:45,382 You know, the more things change, the more they stay the same. 615 00:30:46,042 --> 00:30:46,132 Oh, 616 00:30:46,132 --> 00:30:46,402 yeah. 617 00:30:47,332 --> 00:30:53,662 And, and the key, I mean that I learned in mine, uh, experience for this particular 618 00:30:53,662 --> 00:31:00,742 one, this, um, was in the, all the times I did, well, not quite true, this was. 619 00:31:01,507 --> 00:31:03,427 The biggest disaster. 620 00:31:03,427 --> 00:31:04,537 I should, let me put it that way. 621 00:31:04,537 --> 00:31:07,777 'cause I've been involved in a couple of other DR scenarios 622 00:31:07,777 --> 00:31:10,537 that were very small, um mm-hmm. 623 00:31:10,777 --> 00:31:11,917 Kinds of issues, you know. 624 00:31:12,307 --> 00:31:13,717 Um, we had 625 00:31:13,717 --> 00:31:15,727 nothing beats a completely wiped out island. 626 00:31:15,727 --> 00:31:16,207 Right. 627 00:31:16,477 --> 00:31:16,867 Though we had 628 00:31:16,867 --> 00:31:17,707 a data center. 629 00:31:17,707 --> 00:31:17,767 Yeah. 630 00:31:17,827 --> 00:31:22,507 Um, uh, that had a similar problem where, um. 631 00:31:23,272 --> 00:31:26,692 Somebody had not properly cut the holes in the ceiling of the 632 00:31:26,692 --> 00:31:29,392 data center or plugged them when they were running cables through. 633 00:31:29,392 --> 00:31:36,412 And a lab had a, um, a problem and the sprinklers were released and 634 00:31:36,412 --> 00:31:39,562 the water came down and flooded a corner of this data center. 635 00:31:39,682 --> 00:31:43,342 Um, and it happened to be in a city by the Bay. 636 00:31:43,432 --> 00:31:46,102 Um, but um, we ended up. 637 00:31:47,452 --> 00:31:52,312 Shipping their tapes up or up to our facility and then doing the, 638 00:31:52,852 --> 00:31:57,532 the, um, import to be able to then do the restore from our facility 639 00:31:57,532 --> 00:31:58,852 while they were recovering. 640 00:31:59,662 --> 00:32:01,312 Um, you know, the site down there. 641 00:32:01,312 --> 00:32:07,042 So we had, I had that one before, but one of the things that I've 642 00:32:07,042 --> 00:32:12,562 noticed that's kind of like, I think a problem, it's important to. 643 00:32:13,282 --> 00:32:18,232 The, the proper implementation of a, of a DR resolution. 644 00:32:18,742 --> 00:32:22,762 Um, as well as just the planning, the building and setting up your 645 00:32:23,662 --> 00:32:27,442 infrastructure, your IT infrastructure so that it can be recoverable. 646 00:32:27,442 --> 00:32:32,932 And that's communication among the various teams and groups that are responsible 647 00:32:32,932 --> 00:32:37,792 for all the pieces that make up, you know, an IT infrastructure and, um. 648 00:32:39,007 --> 00:32:42,217 You know, that wasn't a strong point with that company. 649 00:32:42,427 --> 00:32:48,037 Um, they, there was some regulatory reasons why they needed to make 650 00:32:48,037 --> 00:32:54,577 sure that, um, no one person had the keys to the kingdom kind of thing. 651 00:32:55,297 --> 00:33:00,157 But the way they did it tended to put walls between the groups and 652 00:33:00,157 --> 00:33:04,627 so they weren't communicating in a manner, um, that I feel that they 653 00:33:04,627 --> 00:33:06,667 should have, that would've helped. 654 00:33:07,417 --> 00:33:07,867 Um. 655 00:33:08,737 --> 00:33:10,347 You know, address like the left hand, not yeah. 656 00:33:10,347 --> 00:33:11,797 Knowing what the right hand is doing 657 00:33:11,887 --> 00:33:16,297 well and, and address some of the assumptions, you know, address some of 658 00:33:16,297 --> 00:33:19,387 the assumptions because I've learned in, in a lot of my implementations, you 659 00:33:19,387 --> 00:33:24,187 know, um, anytime doing it, you know, uh, learned a long time ago in the 660 00:33:24,187 --> 00:33:25,807 beginning that you gotta have a plan. 661 00:33:25,807 --> 00:33:28,597 You gotta kind of figure out what you're gonna do, what the steps, 662 00:33:28,597 --> 00:33:31,447 what you gonna, you know, um, how are you gonna pull it off, and what 663 00:33:31,447 --> 00:33:32,612 are you gonna do if it doesn't? 664 00:33:32,822 --> 00:33:36,212 Go the way you wanted to, you know, 'cause then you gotta back everything out. 665 00:33:36,752 --> 00:33:41,582 Um, and so I would, you know, do the planning part and then run it 666 00:33:41,582 --> 00:33:44,972 by, uh, other people in my group. 667 00:33:45,937 --> 00:33:47,077 To break the assumptions. 668 00:33:47,077 --> 00:33:49,687 'cause I know when I'm building my plan, I'm making certain assumptions. 669 00:33:49,687 --> 00:33:51,037 I just know it right off the bat. 670 00:33:51,037 --> 00:33:53,017 You know, it's just, I know that's the way we all operate. 671 00:33:53,707 --> 00:33:58,627 Um, and so I'm building my plan and then I have to run it by somebody who will 672 00:33:58,627 --> 00:34:01,087 not have my assumptions to find those. 673 00:34:01,657 --> 00:34:05,437 That reminds me, when, when we used to do the DR test, the way we 674 00:34:05,437 --> 00:34:09,757 would do them is I wasn't allowed to participate in the DR test. 675 00:34:09,757 --> 00:34:09,847 Mm-hmm. 676 00:34:10,187 --> 00:34:14,507 We had to have someone else who was an it, who was an IT person, and then 677 00:34:14,507 --> 00:34:16,157 they would follow my documentation. 678 00:34:16,337 --> 00:34:16,457 Mm-hmm. 679 00:34:17,027 --> 00:34:20,147 Um, and, and I had to pretend to, to be dead. 680 00:34:20,387 --> 00:34:24,077 Um, it was always, by the way, you talked about, you know, you got hit by the bus. 681 00:34:24,287 --> 00:34:26,777 I always hated the fact that it, like, why can't I like 682 00:34:27,107 --> 00:34:28,402 win the lottery and disappear? 683 00:34:28,402 --> 00:34:28,682 Right. 684 00:34:29,062 --> 00:34:29,482 Always 685 00:34:29,482 --> 00:34:30,042 out to be a bus. 686 00:34:30,062 --> 00:34:30,282 Why? 687 00:34:33,107 --> 00:34:33,617 Yeah. 688 00:34:33,617 --> 00:34:35,747 Uh, but yeah, I had to pretend to be dead. 689 00:34:35,837 --> 00:34:37,937 And as I, I've, as I've discussed. 690 00:34:38,147 --> 00:34:39,797 More than once on the podcast. 691 00:34:40,157 --> 00:34:47,837 The, the standard was, um, that, you know, a, a success was we got 100% recovery 692 00:34:47,987 --> 00:34:50,057 without having to have Curtis help. 693 00:34:50,057 --> 00:34:50,147 Right? 694 00:34:50,477 --> 00:34:51,407 And not once. 695 00:34:51,817 --> 00:34:52,747 Did we get that? 696 00:34:52,747 --> 00:34:57,367 Because there was always, there was always something that was left out of the 697 00:34:57,367 --> 00:34:59,527 documentation, no matter how much you try. 698 00:34:59,527 --> 00:34:59,617 Right. 699 00:34:59,677 --> 00:35:00,007 To, 700 00:35:00,667 --> 00:35:00,727 yeah. 701 00:35:00,727 --> 00:35:03,277 Well, and that's, I mean, that's why they're, they, they should 702 00:35:03,277 --> 00:35:05,287 be viewed as living documents. 703 00:35:05,287 --> 00:35:06,937 You should be, um, yeah. 704 00:35:06,967 --> 00:35:07,807 Reviewing them. 705 00:35:07,807 --> 00:35:07,808 Yeah. 706 00:35:07,813 --> 00:35:10,747 You should be testing them, you should be updating them. 707 00:35:11,287 --> 00:35:16,537 Um, but you know, there have not been many. 708 00:35:17,092 --> 00:35:20,422 Places that I've worked in the, in the 30 some years that I've 709 00:35:20,422 --> 00:35:24,082 worked in, in doing that kind of stuff, that operated that way. 710 00:35:24,412 --> 00:35:28,522 Um, you know, there were companies that I would, you know, in like I 711 00:35:28,522 --> 00:35:30,922 can remember one that I got in and if they didn't have anything, and 712 00:35:30,922 --> 00:35:36,442 it really pushed really hard to get some kind of a company-wide, uh, Dr. 713 00:35:36,442 --> 00:35:40,162 Plan pulled together and so we put a lot of effort into it and we built a 714 00:35:40,162 --> 00:35:42,502 document that then was put on a shelf. 715 00:35:42,967 --> 00:35:44,557 And it was never looked at it again. 716 00:35:44,557 --> 00:35:44,647 Right? 717 00:35:44,647 --> 00:35:46,332 And it's like, well, why do we even do it? 718 00:35:46,332 --> 00:35:46,772 Binders 719 00:35:46,792 --> 00:35:48,652 has cobwebs and everything over 720 00:35:48,652 --> 00:35:48,772 it. 721 00:35:49,152 --> 00:35:50,137 Why did we even do it? 722 00:35:50,137 --> 00:35:50,437 You know? 723 00:35:51,067 --> 00:36:00,067 Um, the last company at was one of the better ones for at least attempting to 724 00:36:00,097 --> 00:36:04,027 have that be an ongoing part of their, um. 725 00:36:04,792 --> 00:36:06,052 Their operational thing. 726 00:36:06,052 --> 00:36:10,252 'cause they, they, uh, when an, uh, application or a service was 727 00:36:10,252 --> 00:36:11,812 brought in, it had to be identified. 728 00:36:11,812 --> 00:36:12,952 Is it business critical? 729 00:36:12,952 --> 00:36:16,432 You know, what level of, of protection does it need? 730 00:36:16,432 --> 00:36:21,772 And then, um, if it is business critical, they had to have a DR plan and they 731 00:36:21,772 --> 00:36:26,122 had to do a, um, table talk one year. 732 00:36:26,122 --> 00:36:28,252 And then the following year they actually had to do. 733 00:36:28,672 --> 00:36:29,962 Execute the plan, you know? 734 00:36:30,382 --> 00:36:32,542 And so, um, a good portion, did 735 00:36:32,542 --> 00:36:33,742 they have to do the whole plan 736 00:36:34,252 --> 00:36:35,092 for the application? 737 00:36:35,092 --> 00:36:35,422 Kinda like where 738 00:36:35,422 --> 00:36:35,587 we talked earlier? 739 00:36:35,807 --> 00:36:36,027 For 740 00:36:36,027 --> 00:36:36,707 the application. 741 00:36:36,917 --> 00:36:37,267 Right? 742 00:36:37,327 --> 00:36:37,987 For the application. 743 00:36:38,392 --> 00:36:38,422 Okay. 744 00:36:38,422 --> 00:36:40,732 They had to pretend they had to have a greenfield. 745 00:36:40,737 --> 00:36:41,027 Right? 746 00:36:41,242 --> 00:36:41,452 Right. 747 00:36:41,602 --> 00:36:46,252 And so there was still some weaknesses in that part because, um. 748 00:36:46,987 --> 00:36:50,467 Uh, there, there were, there were assumptions made, right? 749 00:36:50,737 --> 00:36:51,607 Uh, that okay. 750 00:36:51,607 --> 00:36:51,697 Mm-hmm. 751 00:36:51,937 --> 00:36:54,307 We're gonna assume that there's a proper infrastructure. 752 00:36:54,307 --> 00:36:56,827 We're gonna assume this, we're gonna assume there's DNS, you know, 753 00:36:57,667 --> 00:36:59,197 from an application standpoint. 754 00:36:59,377 --> 00:37:00,547 Okay, that's fine. 755 00:37:00,547 --> 00:37:05,557 But what was never really done, and this is kind of the breakdown, um, on 756 00:37:05,557 --> 00:37:10,507 the island, was those assumptions about the infrastructure had never been. 757 00:37:10,867 --> 00:37:12,187 Tested or figured out, gone. 758 00:37:12,187 --> 00:37:14,197 And that's what caught them in the beginning. 759 00:37:14,557 --> 00:37:14,707 Right. 760 00:37:14,917 --> 00:37:16,687 Those assumptions were all underwater and, 761 00:37:16,832 --> 00:37:17,122 yeah. 762 00:37:17,197 --> 00:37:17,527 Yeah. 763 00:37:17,527 --> 00:37:22,597 The infrastructure part was the part that had broken the worst, if that's, 764 00:37:24,367 --> 00:37:24,882 yeah, you know? 765 00:37:24,882 --> 00:37:28,122 You know, it's interesting and I just wonder though, like how many times do 766 00:37:28,122 --> 00:37:29,557 people make that sort of assumption? 767 00:37:29,557 --> 00:37:36,097 Like you kind of make assumptions that power may or may not be available, but 768 00:37:36,157 --> 00:37:39,877 the building may or may not be standing, but there are certain things about. 769 00:37:40,597 --> 00:37:43,207 The outside world that you just assume will still be up and running. 770 00:37:43,237 --> 00:37:43,327 Mm-hmm. 771 00:37:43,807 --> 00:37:46,537 As you're trying to work these things through, I guess on an 772 00:37:46,537 --> 00:37:48,157 island, things get more complicated. 773 00:37:48,307 --> 00:37:48,427 Mm-hmm. 774 00:37:48,727 --> 00:37:51,787 In terms of what may or may not be working, like you said, links, 775 00:37:52,237 --> 00:37:53,707 communications links, et cetera. 776 00:37:53,707 --> 00:37:56,467 But I guess you're right, that is something you have to take into 777 00:37:56,467 --> 00:38:00,037 consideration are things outside of your data center that you have to take. 778 00:38:00,412 --> 00:38:00,832 Think about 779 00:38:01,042 --> 00:38:01,372 Right. 780 00:38:01,372 --> 00:38:06,532 But I, I was, I thought about this a lot after, you know, in the aftermath 781 00:38:06,532 --> 00:38:12,502 of that for the, um, um, for, for some time after being involved in that. 782 00:38:12,952 --> 00:38:18,742 And what I experienced there, what was experienced there to me could very 783 00:38:18,742 --> 00:38:24,742 well happen in a regional sort of way here on the mainland and, and leave, 784 00:38:24,742 --> 00:38:26,812 leave a company in the same boat. 785 00:38:27,877 --> 00:38:28,207 Right. 786 00:38:29,017 --> 00:38:33,487 Um, and so, you know, while there are some particularities to being on an 787 00:38:33,487 --> 00:38:37,627 island, um, it doesn't mean that if you're not on an island, you don't 788 00:38:37,627 --> 00:38:42,967 have to worry about the kinds of assumptions that, that, you know, that 789 00:38:43,987 --> 00:38:48,607 turned out to be big problems in their initial, um, you know, recovery attempt. 790 00:38:48,637 --> 00:38:48,787 Yeah. 791 00:38:48,787 --> 00:38:48,847 You. 792 00:38:49,567 --> 00:38:54,067 You would think that being, being on an island, they would not assume 793 00:38:54,607 --> 00:38:56,017 a connection to the mainland. 794 00:38:56,107 --> 00:38:58,327 But a apparently that was the case. 795 00:38:58,327 --> 00:38:58,743 That was the case, yeah. 796 00:38:59,227 --> 00:39:03,427 Um, and the, the, the difficulty I have when I hear this story, 797 00:39:03,427 --> 00:39:08,017 because the, the island situation, what it does is, I think it, it. 798 00:39:08,632 --> 00:39:14,572 Brings to the foreground or makes possible many of the worst case scenarios that 799 00:39:14,572 --> 00:39:18,322 could happen, that that could happen to a lesser degree on the mainland, but, 800 00:39:18,712 --> 00:39:20,752 but to a greater degree on, on an island. 801 00:39:21,352 --> 00:39:27,622 Um, but the, the, the real problem here is that many, many of the modern solutions 802 00:39:27,622 --> 00:39:33,232 that I would think of to solve these problems are based on using the cloud. 803 00:39:33,502 --> 00:39:33,622 Mm-hmm. 804 00:39:34,717 --> 00:39:38,257 Is problematic when you look at an island situation. 805 00:39:38,257 --> 00:39:38,347 Mm-hmm. 806 00:39:38,587 --> 00:39:41,887 It's not as much when you're looking at a mainland situation, but if, for 807 00:39:41,887 --> 00:39:47,317 example, you, you, you, you would have to have, you know, internet connectivity, 808 00:39:47,317 --> 00:39:48,547 which is why I go back to that. 809 00:39:48,547 --> 00:39:51,937 I, I'm really curious to see how this, how the Sea Elon Musk project 810 00:39:51,937 --> 00:39:53,437 goes away or go goes forward. 811 00:39:53,437 --> 00:39:57,427 And whether or not it's, um, you know, something that can 812 00:39:57,427 --> 00:39:58,927 help solve this problem because. 813 00:40:00,007 --> 00:40:03,937 Everything that I know that's being done right now has to do, like 814 00:40:03,937 --> 00:40:08,827 in the really cool DR perspective has to do with using the cloud. 815 00:40:08,827 --> 00:40:16,087 Nobody's talking about, you know, u using physical server DR services or you know, 816 00:40:16,087 --> 00:40:19,507 all of the, basically without using the cloud, it's so much harder and with using 817 00:40:19,507 --> 00:40:23,017 the cloud, it's so much easier to not just to test an application, but to test 818 00:40:23,017 --> 00:40:25,567 the disaster recovery of the entire site. 819 00:40:25,837 --> 00:40:28,177 But here's a question for you, Curtis. 820 00:40:28,777 --> 00:40:29,107 Yeah. 821 00:40:29,497 --> 00:40:33,877 Given the island scenario and assuming that your, the cloud 822 00:40:33,877 --> 00:40:35,557 was available on the island. 823 00:40:36,157 --> 00:40:36,487 Yeah. 824 00:40:37,807 --> 00:40:42,727 Wouldn't, and say everyone got hit by this, um, isn't the cloud provider in the 825 00:40:42,727 --> 00:40:46,687 same situation where they may not have enough resources or they might be down? 826 00:40:46,717 --> 00:40:49,747 That's why I say granted, I, I know that you probably are gonna do DR 827 00:40:49,747 --> 00:40:53,617 to the mainland in this example or somewhere else, but I'm just wondering 828 00:40:53,617 --> 00:40:54,817 if everyone starts doing that. 829 00:40:54,907 --> 00:40:56,947 Don't, like cloud provider is just a big. 830 00:40:57,607 --> 00:40:58,777 Data, it's just somebody else's server. 831 00:40:58,777 --> 00:40:59,227 Yeah, that's, 832 00:40:59,552 --> 00:40:59,842 yeah. 833 00:41:00,187 --> 00:41:00,337 Yeah. 834 00:41:00,337 --> 00:41:01,087 Agreed. 835 00:41:01,087 --> 00:41:05,047 I, I'm just saying that's why the, the, the, the, the island situation is so 836 00:41:05,047 --> 00:41:09,517 problematic because I, I completely agree with you that I, if it was, let's just 837 00:41:09,517 --> 00:41:11,107 say this, this was a pretty small island. 838 00:41:11,107 --> 00:41:13,687 Let's say it's a bigger island and there is a cloud provider. 839 00:41:13,717 --> 00:41:15,907 'cause there, there wasn't, I, I don't think cloud. 840 00:41:16,157 --> 00:41:17,897 Stuff available on the island. 841 00:41:18,317 --> 00:41:21,857 Uh, if you, if it was available, I wouldn't use that as your DR site. 842 00:41:21,857 --> 00:41:23,027 I would use the mainland. 843 00:41:23,117 --> 00:41:23,447 Right. 844 00:41:23,687 --> 00:41:28,217 And so, because Yes, you, because you're completely right if, if your data center's 845 00:41:28,217 --> 00:41:31,637 underwater and you're on an island quite ly their data center's underwater. 846 00:41:31,637 --> 00:41:31,877 Right. 847 00:41:32,327 --> 00:41:36,707 Um, the, um, it reminds me, I used to, you know, manage data 848 00:41:36,707 --> 00:41:39,377 centers in Delaware and we had a. 849 00:41:40,237 --> 00:41:42,367 Our offsite vaulting company. 850 00:41:42,367 --> 00:41:44,377 It wasn't Iron Mountain, it was this local company. 851 00:41:44,767 --> 00:41:51,187 And what they had was they had a World War II bunker, um, like, like bomb shelter. 852 00:41:51,457 --> 00:41:55,087 And that's where they stored tapes for people, which sounds really 853 00:41:55,087 --> 00:41:56,677 good until a hurricane comes. 854 00:41:57,097 --> 00:42:02,167 And so whenever a hurricane was on its way to Delaware, we had to pay money 855 00:42:02,467 --> 00:42:07,327 to have all of our tapes moved out of the bunker and up to the second floor. 856 00:42:08,167 --> 00:42:11,947 So that if our data, if our data center was underwater, our tapes 857 00:42:11,947 --> 00:42:14,437 also were not underwater anyway. 858 00:42:15,517 --> 00:42:20,347 But yeah, I, I just, it's just the, the island situation is frustrating. 859 00:42:20,617 --> 00:42:21,337 Um, and. 860 00:42:22,387 --> 00:42:24,877 Uh, so any further thoughts? 861 00:42:25,207 --> 00:42:30,277 Um, Ron on the, because we haven't even talked about the recovery yet, and I, 862 00:42:30,277 --> 00:42:33,967 I'm just gonna have to have you back 'cause I, I find this solu, I find 863 00:42:33,967 --> 00:42:35,737 this discussion incredibly fascinating. 864 00:42:35,737 --> 00:42:37,387 You're, you know, you're, you're smart. 865 00:42:37,387 --> 00:42:40,087 You know what you're doing, you're articulate, uh, and, and you 866 00:42:40,087 --> 00:42:42,637 know, I'm, I'm super glad that we, that we finally have you on. 867 00:42:42,937 --> 00:42:46,267 Can you think, uh, so what I'm gonna do is we're gonna with, on this podcast, 868 00:42:46,267 --> 00:42:48,187 we'll sort of round out the discussion on. 869 00:42:49,087 --> 00:42:52,777 Everything sort of almost not recovery, like all of the 870 00:42:52,777 --> 00:42:55,987 things that just frustrated you being able to start a recovery. 871 00:42:56,197 --> 00:42:59,437 Can you think of anything else that falls into that category that happened to you? 872 00:43:00,222 --> 00:43:01,872 Well, I was just trying to go back. 873 00:43:02,132 --> 00:43:02,352 Um, 874 00:43:03,552 --> 00:43:05,947 I mean there were no Snickers bars available, for example. 875 00:43:06,397 --> 00:43:09,697 Um, well, I was just gonna ask you about how it was on the island itself. 876 00:43:09,697 --> 00:43:11,377 I know you had a job to do, trying to get. 877 00:43:12,222 --> 00:43:14,592 The company back up and running, but I'm sure there is also. 878 00:43:14,627 --> 00:43:15,342 Yeah, yeah. 879 00:43:15,342 --> 00:43:16,302 Well, okay on that. 880 00:43:16,362 --> 00:43:20,772 Um, so I need to go back to see the island, um, because I didn't really 881 00:43:20,772 --> 00:43:21,972 get to see the island, you know? 882 00:43:22,392 --> 00:43:25,092 Um, it was an interesting, it was an interesting journey. 883 00:43:25,092 --> 00:43:28,812 We, um, we flew down in a corporate jet, right? 884 00:43:28,812 --> 00:43:30,822 That was a trip in and of itself. 885 00:43:31,452 --> 00:43:31,812 Um, 886 00:43:32,742 --> 00:43:33,432 that must have been cool. 887 00:43:33,432 --> 00:43:33,972 Was that cool? 888 00:43:33,972 --> 00:43:34,387 That cool? 889 00:43:34,387 --> 00:43:34,587 Just tell me. 890 00:43:34,587 --> 00:43:34,907 That was cool. 891 00:43:34,907 --> 00:43:35,148 Very cool. 892 00:43:35,148 --> 00:43:35,149 Uh, 893 00:43:35,262 --> 00:43:36,042 but I'll tell you what. 894 00:43:36,072 --> 00:43:36,282 Okay. 895 00:43:36,282 --> 00:43:36,627 It, it. 896 00:43:36,862 --> 00:43:39,802 Spit me in the butt because I didn't have to go through TSA 897 00:43:39,802 --> 00:43:40,912 or any that on the way down. 898 00:43:40,912 --> 00:43:45,082 And so I forgot that I had put a couple of things in my bag, and when I was 899 00:43:45,622 --> 00:43:49,552 leaving the island, I had to actually flag commercial TSA and, and it was, 900 00:43:49,612 --> 00:43:54,172 it was fruit and it, like they take it out and you can't take it with you. 901 00:43:54,232 --> 00:43:59,332 So, um, but um, it was, it was weird. 902 00:44:00,367 --> 00:44:05,587 You know, when we landed, uh, you know, at that time, this was very 903 00:44:05,587 --> 00:44:09,637 early after the, the, um, the init, the hurricane itself, right? 904 00:44:09,637 --> 00:44:16,597 And so, um, limited flights in and out, you had to, um, you had to, they, you had 905 00:44:16,597 --> 00:44:20,857 to, basically, they had to request and were given this tiny window to land in. 906 00:44:20,857 --> 00:44:23,857 And so, boy, they had to make sure they made it on time and all that. 907 00:44:24,367 --> 00:44:25,897 Um, they had a number of. 908 00:44:26,512 --> 00:44:33,532 People with the company that they were relocating to a facility on the mainland 909 00:44:33,532 --> 00:44:39,592 so that they could get online and do work in helping to bring the site back. 910 00:44:39,592 --> 00:44:39,892 Right. 911 00:44:40,252 --> 00:44:44,122 Uh, they couldn't work locally and so they were waiting when we landed and you 912 00:44:44,122 --> 00:44:49,342 know, we got off and they put us in a shuttle and then took us and there wasn't. 913 00:44:50,167 --> 00:44:55,897 I grew up in Montana and Montana in the winter, when you look out, 914 00:44:55,987 --> 00:45:00,217 um, you got the evergreens, but if there's not an evergreen, it's brown. 915 00:45:00,337 --> 00:45:01,417 It's brown, right? 916 00:45:01,417 --> 00:45:04,177 I mean, there's, there's no green, um, there. 917 00:45:04,537 --> 00:45:08,857 Um, and so this looked a lot like Montana in the winter without the snow. 918 00:45:08,862 --> 00:45:09,682 I mean, it was just. 919 00:45:09,822 --> 00:45:10,362 Brown. 920 00:45:10,362 --> 00:45:12,702 There wasn't any leaves, any green anywhere. 921 00:45:12,702 --> 00:45:13,902 It had been stripped bare. 922 00:45:13,902 --> 00:45:15,432 So it was weird seeing that. 923 00:45:16,272 --> 00:45:19,357 Oh, because it had been stripped from, from bear, yeah, from the hurricane. 924 00:45:19,357 --> 00:45:19,597 Gotcha. 925 00:45:19,597 --> 00:45:20,112 The wind just 926 00:45:20,112 --> 00:45:21,042 stripped it off. 927 00:45:21,342 --> 00:45:25,572 One of the guys that, that he, he'd worked at the facility, I 928 00:45:25,572 --> 00:45:27,342 forget, he's been like 20 years. 929 00:45:27,672 --> 00:45:30,792 He said, I'm doing this commute for the last 20 years. 930 00:45:31,487 --> 00:45:35,507 I didn't know there were houses down there on the side of the road, you 931 00:45:35,507 --> 00:45:38,807 know, because it was so lush and green, you couldn't see beyond, oh. 932 00:45:38,867 --> 00:45:41,237 And he says, I'm seeing stuff that I didn't even know was there. 933 00:45:41,237 --> 00:45:46,517 But anyways, um, and then we get there and, you know, there's, they 934 00:45:46,517 --> 00:45:48,227 basically put us up in conference room. 935 00:45:48,227 --> 00:45:51,302 They turned conference rooms into, um, um. 936 00:45:52,627 --> 00:45:55,177 Just, uh, dorm, right? 937 00:45:55,357 --> 00:46:01,657 And we had, um, um, they had these little air mattresses and, um, I brought, uh, 938 00:46:01,687 --> 00:46:04,717 they told us, you know, bring a bag and, and so I brought like a sleeping 939 00:46:04,717 --> 00:46:06,007 bag, you know, change of clothes. 940 00:46:06,607 --> 00:46:11,047 Uh, and, uh, pretty much spent, well, the entire time I was 941 00:46:11,047 --> 00:46:13,387 there, um, I spent it on. 942 00:46:14,227 --> 00:46:15,157 On the facility. 943 00:46:15,187 --> 00:46:17,527 'cause there was nothing to go, no place to really go. 944 00:46:17,887 --> 00:46:21,757 Um, for, for the locals, you know, there were long lines for everything. 945 00:46:22,297 --> 00:46:26,857 Um, and so it just didn't seem right to go getting long lines for those people. 946 00:46:27,127 --> 00:46:33,037 Um, they, they were providing us with, um, you know, the food, um, and it was 947 00:46:33,037 --> 00:46:39,097 a lot of, um, chicken and rice, but, um, you know, you'll survive on, on, on it. 948 00:46:39,097 --> 00:46:42,757 So they, they took the, the company took very. 949 00:46:43,192 --> 00:46:45,892 Good care I thought of, of the people. 950 00:46:46,192 --> 00:46:52,402 Um, and they also were taking real good care of, uh, local workers, um, who could 951 00:46:52,402 --> 00:46:57,712 not come back in, um, to the facility, you know, just because they, they weren't, 952 00:46:58,372 --> 00:47:01,642 their stuff wasn't yet recovered, you know, because there's some manufacturing 953 00:47:01,642 --> 00:47:02,339 goes on there, and so they weren't. 954 00:47:03,337 --> 00:47:04,582 It's ready to do manufacturing. 955 00:47:05,227 --> 00:47:06,427 Um, but they took care of 'em. 956 00:47:06,427 --> 00:47:13,117 They shipped in, um, brought in pallets, pallets full of, of, uh, bulk food 957 00:47:13,117 --> 00:47:16,957 and whatnot and, and, uh, had workers come in, you know, and they just 958 00:47:16,957 --> 00:47:20,107 basically parceled 'em out and helped supplemented a lot of their stuff. 959 00:47:20,107 --> 00:47:21,157 So I thought that was really good. 960 00:47:21,157 --> 00:47:25,087 Their, their response overall in taking care of their, their, their local 961 00:47:25,087 --> 00:47:28,417 employees, that I thought was really well, and, you know, and, and they did 962 00:47:28,417 --> 00:47:30,337 good by us as far as, as, you know. 963 00:47:31,477 --> 00:47:34,657 Under the circumstances, you know, we didn't go hungry and we, you know, 964 00:47:34,657 --> 00:47:36,727 we had a place to sleep, but it was. 965 00:47:36,772 --> 00:47:44,462 FI was there for almost two weeks and it was pretty much, um, you know, 12, 14, 966 00:47:44,462 --> 00:47:46,852 16 hour days in the, in the beginning. 967 00:47:46,852 --> 00:47:49,732 And then it tapered down after about the first week. 968 00:47:50,182 --> 00:47:54,022 Um, and then it was, we were down to sort of eight hour days. 969 00:47:54,022 --> 00:47:55,012 But then you just. 970 00:47:55,252 --> 00:47:56,902 Sitting around a lot of time. 971 00:47:57,442 --> 00:48:00,982 And so since there was nothing to do, we just worked 972 00:48:01,822 --> 00:48:02,272 like, what else? 973 00:48:02,272 --> 00:48:02,572 What else do you do? 974 00:48:02,572 --> 00:48:02,752 Yeah. 975 00:48:02,752 --> 00:48:04,297 When you're in that situation, you know? 976 00:48:04,297 --> 00:48:04,537 Yeah. 977 00:48:04,792 --> 00:48:05,932 Um, subsequent you're not 978 00:48:05,932 --> 00:48:06,802 watching Netflix. 979 00:48:06,862 --> 00:48:07,282 Yeah. 980 00:48:07,432 --> 00:48:12,442 Subsequent teams that came down, um, by, by that time, um, they were 981 00:48:12,442 --> 00:48:15,472 actually coming down on commercial, you know, 'cause commercial, 982 00:48:15,562 --> 00:48:16,942 um, flights had started to pick. 983 00:48:16,947 --> 00:48:19,737 Back up as they had, um, kind of cleared stuff up in that. 984 00:48:20,217 --> 00:48:23,487 Uh, and then they had, um, rented out hotels, uh, space for 985 00:48:23,487 --> 00:48:25,587 them and were shuttling them. 986 00:48:25,857 --> 00:48:30,687 Um, so they weren't staying on the, on the, on the, the campus itself. 987 00:48:30,837 --> 00:48:33,777 And, but so, but it was kind of fun in one sense. 988 00:48:33,777 --> 00:48:38,187 Um, I did a, um, the only exercise I could really get there was just a lot of 989 00:48:38,187 --> 00:48:43,317 walking and I walked around all over the campus just to look at the damage and, 990 00:48:43,887 --> 00:48:45,807 you know, different parts of it had had. 991 00:48:46,552 --> 00:48:48,952 You know, some of it unscathed. 992 00:48:49,342 --> 00:48:55,402 Um, the particular building that the, the data center B was in, um, was one of 993 00:48:55,402 --> 00:48:57,442 the oldest buildings there on the campus. 994 00:48:57,442 --> 00:49:02,782 And so, um, it, it's does, you know, not surprising that it was the 995 00:49:02,782 --> 00:49:04,077 one that suffered the most damage. 996 00:49:05,917 --> 00:49:11,947 So, um, and, uh, but then it was kind of, you know, it was a, the small group 997 00:49:11,947 --> 00:49:13,777 of us that were, were forced together. 998 00:49:13,777 --> 00:49:16,927 People that you'd never met before or didn't know were all kind of 999 00:49:16,927 --> 00:49:18,907 forced together to, to work together. 1000 00:49:18,907 --> 00:49:20,917 And that I found that was fun. 1001 00:49:20,917 --> 00:49:21,667 That was interesting. 1002 00:49:21,667 --> 00:49:23,647 You know, you get to, you get to know people and it, did you 1003 00:49:23,647 --> 00:49:24,092 have any. 1004 00:49:24,947 --> 00:49:26,567 Did you have any language barriers? 1005 00:49:26,627 --> 00:49:30,197 'cause I, I'm guessing you don't speak the language of the locals. 1006 00:49:30,202 --> 00:49:30,362 No. 1007 00:49:30,367 --> 00:49:30,497 No. 1008 00:49:30,497 --> 00:49:34,187 But, but, um, you know, um, everybody within the facility 1009 00:49:34,187 --> 00:49:36,647 is, is fluent, uh, English. 1010 00:49:36,677 --> 00:49:37,187 It's okay. 1011 00:49:37,187 --> 00:49:41,747 You know, because it's a, it's, um, a mainland company, you know, headquarters. 1012 00:49:41,747 --> 00:49:41,777 Okay. 1013 00:49:42,317 --> 00:49:45,017 Um, so, so they're all, they're all bilingual. 1014 00:49:45,152 --> 00:49:48,002 Um, which puts them way up the ladder on than me, you know? 1015 00:49:48,752 --> 00:49:50,732 Um, I'm always, I'm always, yeah. 1016 00:49:50,792 --> 00:49:54,332 Um, you know, my hat off to anybody that speaks more than one 1017 00:49:54,332 --> 00:49:55,772 language as far as I'm concerned. 1018 00:49:56,462 --> 00:49:57,422 Um, absolutely. 1019 00:49:57,422 --> 00:49:57,482 Yeah. 1020 00:49:57,782 --> 00:49:58,557 So, persona, 1021 00:49:59,017 --> 00:49:59,702 how many you got? 1022 00:49:59,702 --> 00:50:00,242 Persona. 1023 00:50:00,572 --> 00:50:02,072 I can understand other languages. 1024 00:50:02,072 --> 00:50:03,152 I don't necessarily speak a. 1025 00:50:03,682 --> 00:50:04,642 Still, that's still better. 1026 00:50:05,017 --> 00:50:05,307 Yeah. 1027 00:50:05,307 --> 00:50:05,587 Mm-hmm. 1028 00:50:05,692 --> 00:50:05,932 Yeah. 1029 00:50:05,932 --> 00:50:10,282 So yeah, so it was a very, very interesting experience. 1030 00:50:10,342 --> 00:50:13,792 Um, overall, I'll tell you, you know, it, it was, uh, not that, not 1031 00:50:13,792 --> 00:50:18,352 that I wanna have to do it again, but 'cause of the circumstances. 1032 00:50:18,352 --> 00:50:18,502 But 1033 00:50:19,042 --> 00:50:22,282 you got to really do sort of the worst case scenario of what 1034 00:50:22,312 --> 00:50:24,532 a lot of us, um, prepare for. 1035 00:50:25,157 --> 00:50:26,237 Prepare for. 1036 00:50:26,357 --> 00:50:29,687 And uh, that's why I really wanted to have you on Yeah. 1037 00:50:29,692 --> 00:50:30,317 And, and talk to you. 1038 00:50:30,317 --> 00:50:33,497 And by the way, we're gonna have you back 'cause you, you've got a lot of 1039 00:50:33,647 --> 00:50:36,677 institutional knowledge up there of what it's like to actually do this 1040 00:50:36,677 --> 00:50:38,567 in, you know, in the real world. 1041 00:50:38,717 --> 00:50:41,447 Um, so we're definitely gonna have you back, but, uh, I'm gonna end 1042 00:50:41,447 --> 00:50:45,347 this one now because we try to keep these around 45 minutes or so. 1043 00:50:45,737 --> 00:50:47,297 We're definitely gonna have you back. 1044 00:50:47,297 --> 00:50:47,987 Would that be alright? 1045 00:50:48,437 --> 00:50:49,322 Oh, sure, sure. 1046 00:50:49,322 --> 00:50:49,323 Yeah. 1047 00:50:50,202 --> 00:50:55,512 You know, having spent most of my career preparing for disasters and, 1048 00:50:55,512 --> 00:51:01,722 and running a couple of, uh, of, you know, tests, uh, to actually be. 1049 00:51:02,122 --> 00:51:06,082 Be able to participate in a full on recovery of a, of a facility. 1050 00:51:07,012 --> 00:51:07,192 Yeah. 1051 00:51:07,192 --> 00:51:14,002 That was a, a, a highlight, you know, I mean, again, if you don't want those 1052 00:51:14,002 --> 00:51:17,662 kinds of things to happen, but to be able to be a part of one was, yeah. 1053 00:51:17,662 --> 00:51:20,572 That's something that I, I'm really proud of. 1054 00:51:20,992 --> 00:51:22,192 This was fascinating. 1055 00:51:22,192 --> 00:51:26,722 I I, I, it's interesting right at the end there, we got the image of you on a, on 1056 00:51:26,722 --> 00:51:28,462 an air mattress with a sleeping bag bag. 1057 00:51:29,932 --> 00:51:32,932 Uh, that, that, that, that added like a whole, I didn't even, I didn't even 1058 00:51:32,932 --> 00:51:35,752 think about that part of like, you know, that you needed a place to sleep 1059 00:51:35,842 --> 00:51:40,672 and those places weren't Exactly, uh, in the, they were in short supply. 1060 00:51:41,302 --> 00:51:45,382 Anyway, so, um, thanks so much for, for coming on. 1061 00:51:45,712 --> 00:51:45,802 Sure. 1062 00:51:46,807 --> 00:51:47,707 Thanks, persona. 1063 00:51:47,767 --> 00:51:48,397 Thanks Curtis. 1064 00:51:48,397 --> 00:51:49,057 Thank you, Ron. 1065 00:51:49,057 --> 00:51:49,747 I, yeah, pleasure. 1066 00:51:49,747 --> 00:51:53,167 Like Curtis said, I just have that image, image of you in an air mattress 1067 00:51:53,167 --> 00:51:58,777 with a bowl of chicken and rice with, uh, lights flickering, kind 1068 00:51:58,777 --> 00:52:00,667 of like an apocalypse like scene. 1069 00:52:00,937 --> 00:52:04,867 And the data center slowly coming back, online's like Jurassic Park almost. 1070 00:52:06,272 --> 00:52:07,682 I know that system. 1071 00:52:08,132 --> 00:52:13,172 All right, and with that, I wanna thank the listeners for your attention and, 1072 00:52:13,172 --> 00:52:15,272 uh, and, and listening all the time. 1073 00:52:15,572 --> 00:52:18,722 And make sure to subscribe so that you can restore it all. 1074 00:52:21,387 --> 00:52:25,802 There was a file, but I deleted it. 1075 00:52:27,392 --> 00:52:29,462 Backup system isn't worth. 1076 00:52:29,512 --> 00:52:30,352 Speed. 1077 00:52:30,772 --> 00:52:34,912 Finally needed your backup. 1078 00:52:35,362 --> 00:52:38,062 You had a chance to fix it instead. 1079 00:52:38,062 --> 00:52:39,622 It's all jacked up. 1080 00:52:39,952 --> 00:52:44,092 See right on Facebook about you. 1081 00:52:44,512 --> 00:52:48,652 Don't underestimate the things that I'll do. 1082 00:52:49,162 --> 00:52:52,762 There was a file, but I deleted it. 1083 00:52:55,462 --> 00:52:57,262 System isn't worth space. 1084 00:52:59,782 --> 00:53:00,772 Emails from you. 1085 00:53:01,767 --> 00:53:07,552 Remind me when they keep me thinking that we could restore it. 1086 00:53:07,552 --> 00:53:09,982 All emails from. 1087 00:53:15,202 --> 00:53:16,672 You could restore it. 1088 00:53:19,727 --> 00:53:20,932 Can you rescue me from 1089 00:53:23,407 --> 00:53:24,187 rescue me? 1090 00:53:29,687 --> 00:53:29,907 You. 1091 00:53:31,682 --> 00:53:32,402 How insane. 1092 00:53:36,752 --> 00:53:40,142 I have no file to hold because you are stupid. 1093 00:53:41,252 --> 00:53:43,502 Backups didn't work it all. 1094 00:53:43,952 --> 00:53:48,122 Think of me while you backing up data. 1095 00:53:49,592 --> 00:53:50,948 It would work if it. 1096 00:53:54,792 --> 00:53:55,902 Emails from you. 1097 00:53:55,902 --> 00:53:56,082 Remind 1098 00:53:58,377 --> 00:54:02,022 when they give me thinking that you could restore 1099 00:54:04,617 --> 00:54:05,387 from you. 1100 00:54:06,367 --> 00:54:06,587 Me 1101 00:54:08,857 --> 00:54:10,467 feeling you could. 1102 00:54:26,767 --> 00:54:30,347 How you could restore it all 1103 00:54:32,927 --> 00:54:34,307 and rescue me from 1104 00:54:37,327 --> 00:54:37,547 you. 1105 00:54:42,647 --> 00:54:42,867 You 1106 00:54:48,077 --> 00:54:50,747 throw yourself into every. 1107 00:54:51,387 --> 00:54:56,487 Run, hoping that just for once it'll be completely done. 1108 00:54:57,147 --> 00:55:03,992 Maybe one day it'll all work out you and. 1109 00:55:22,117 --> 00:55:22,337 It. 1110 00:55:41,717 --> 00:55:41,937 You 1111 00:55:46,237 --> 00:55:47,217 and rescue me.