1 00:00:00,000 --> 00:00:03,890 You found the backup wrap up your go-to podcast for all things 2 00:00:03,890 --> 00:00:06,289 backup recovery and cyber recovery. 3 00:00:06,680 --> 00:00:09,890 In this episode, you'll hear the harrowing tale of what I'm 4 00:00:09,890 --> 00:00:11,809 calling the backup from hell. 5 00:00:12,110 --> 00:00:16,040 A project that started as a simple one-time backup, a 40 terabyte 6 00:00:16,040 --> 00:00:20,810 of two sonology boxes that turned into a 400 terabyte nightmare 7 00:00:20,810 --> 00:00:22,280 that took months to complete. 8 00:00:22,854 --> 00:00:27,144 We're talking hundreds of millions of files with one directory alone 9 00:00:27,144 --> 00:00:29,184 containing 99 million of them. 10 00:00:29,544 --> 00:00:33,745 I'll share how I dealt with failing tape drives ridiculously slow 11 00:00:33,745 --> 00:00:38,184 backup speeds, and ultimate solution that finally got the job done. 12 00:00:38,695 --> 00:00:41,815 If you've ever wondered what happens when everything that could go wrong 13 00:00:41,815 --> 00:00:43,675 with the backup actually goes wrong. 14 00:00:44,029 --> 00:00:48,769 This episode is for you, plus you'll learn some valuable lessons about what to check 15 00:00:48,980 --> 00:00:51,589 before starting a massive backup job. 16 00:00:52,400 --> 00:00:56,749 By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr. 17 00:00:56,749 --> 00:01:00,049 Backup, and I've been passionate about backup and recovery for 18 00:01:00,049 --> 00:01:02,120 over 30 years, ever since. 19 00:01:02,120 --> 00:01:05,389 I had to tell my boss that we had no backups of the production 20 00:01:05,389 --> 00:01:06,949 database that we just lost. 21 00:01:07,505 --> 00:01:10,745 I don't want that to happen to you, and that's why I do this show. 22 00:01:11,225 --> 00:01:16,384 On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes. 23 00:01:16,685 --> 00:01:18,695 This is the backup wrap up. 24 00:01:32,491 --> 00:01:36,471 Welcome to the show, and if I could ask you to just take one quick second 25 00:01:36,531 --> 00:01:40,521 and, uh, subscribe or follow us so you can make sure that you get all of this 26 00:01:40,521 --> 00:01:43,251 great content, that would be great. 27 00:01:44,211 --> 00:01:46,521 I'm w Curtis Preston, AKA, Mr. 28 00:01:46,521 --> 00:01:51,531 Backup, and I have with me a guy that apparently owes Ben Kingsley 29 00:01:51,801 --> 00:01:55,831 a huge apology Prasanna Malaiyandin 30 00:01:55,851 --> 00:01:56,301 how's it going? 31 00:01:56,301 --> 00:01:57,441 Prasanna, why do you owe 32 00:01:58,011 --> 00:01:58,731 an apology? 33 00:01:59,781 --> 00:02:03,291 so as everyone's probably like, who's Ben Kingsley. 34 00:02:03,291 --> 00:02:09,051 So if you don't know, he is an actor and he also played Gandhi in the movie Gandhi. 35 00:02:09,051 --> 00:02:09,831 He did. 36 00:02:09,921 --> 00:02:10,341 Right? 37 00:02:11,076 --> 00:02:17,976 And for the longest time I was a little, not upset, but like the fact that you have 38 00:02:18,636 --> 00:02:25,446 like probably one of the most important Indian people in history being played 39 00:02:25,476 --> 00:02:27,366 By a guy with the name Ben Kingsley. 40 00:02:27,906 --> 00:02:28,566 Exactly. 41 00:02:28,806 --> 00:02:29,226 Yeah. 42 00:02:29,556 --> 00:02:30,306 Ben Kingsley. 43 00:02:31,416 --> 00:02:36,696 And so today I found out that Ben Kingsley is actually Indian. 44 00:02:37,626 --> 00:02:37,896 Half 45 00:02:37,956 --> 00:02:38,916 How about that? 46 00:02:38,946 --> 00:02:39,306 should say. 47 00:02:39,516 --> 00:02:39,846 Yeah, 48 00:02:40,026 --> 00:02:40,386 what? 49 00:02:41,016 --> 00:02:42,276 he's Anglo Indian. 50 00:02:42,631 --> 00:02:43,476 Anglo Indian. 51 00:02:43,956 --> 00:02:44,436 Yes. 52 00:02:44,976 --> 00:02:45,816 It's like us. 53 00:02:45,816 --> 00:02:47,586 You and me we're Indian. 54 00:02:48,786 --> 00:02:52,866 so his paternal side is from Gujarat. 55 00:02:53,616 --> 00:02:54,066 Right. 56 00:02:54,126 --> 00:02:57,246 And his mom's side I think is European. 57 00:02:57,426 --> 00:03:00,066 His dad was a physician who was born in Kenya. 58 00:03:00,876 --> 00:03:03,726 And Ben Kingsley's name is not actually Ben Kingsley. 59 00:03:04,611 --> 00:03:07,641 It's like Krishna Bunge, I think 60 00:03:07,756 --> 00:03:08,046 Yeah. 61 00:03:09,201 --> 00:03:09,261 Yeah. 62 00:03:09,261 --> 00:03:09,262 Yeah. 63 00:03:09,321 --> 00:03:13,971 And he realized that he wasn't getting called into the right casting 64 00:03:13,971 --> 00:03:17,001 roles when he was looking for, when he was starting off his career. 65 00:03:17,001 --> 00:03:19,131 So he is like, let me change my name. 66 00:03:19,131 --> 00:03:22,161 And so we changed his name to Ben Kingsley and people started calling 67 00:03:22,161 --> 00:03:23,781 him in and he started getting roles. 68 00:03:24,651 --> 00:03:28,281 Racism in early Hollywood say, it isn't so. 69 00:03:28,851 --> 00:03:30,531 Racism in current Hollywood. 70 00:03:30,651 --> 00:03:34,731 Say it isn't So, Wouldn't be the only to do so. 71 00:03:34,731 --> 00:03:34,971 Yeah. 72 00:03:35,481 --> 00:03:41,781 yeah, so I apologize to Sir Ben Kingsley, uh, for all these years. 73 00:03:43,401 --> 00:03:43,611 Yeah. 74 00:03:43,611 --> 00:03:46,671 You were putting it in the same category as the quote unquote 75 00:03:46,701 --> 00:03:50,841 Indian guy from the Short Circuit movie, which I don't know his name, 76 00:03:50,841 --> 00:03:53,001 but he is very much not an Indian 77 00:03:53,301 --> 00:03:53,901 person. 78 00:03:54,111 --> 00:03:55,191 Do you know who it was? 79 00:03:55,611 --> 00:03:56,031 the name? 80 00:03:56,796 --> 00:03:57,816 I'm looking it up. 81 00:03:57,816 --> 00:04:03,036 Or it's also like how Apu from, uh, the Simpsons is not Indian, 82 00:04:03,801 --> 00:04:06,411 Yeah, he's, he's played by, um. 83 00:04:07,401 --> 00:04:08,091 Oh, I know that. 84 00:04:08,091 --> 00:04:10,701 I know the actor, but his name is escaping me. 85 00:04:11,421 --> 00:04:12,801 So Fisher Stevens, is that 86 00:04:12,846 --> 00:04:13,581 Fisher Stevens. 87 00:04:13,746 --> 00:04:14,856 Yeah, Fisher Stevens. 88 00:04:15,156 --> 00:04:15,576 Who? 89 00:04:15,786 --> 00:04:18,006 Those of you that watch succession 90 00:04:18,396 --> 00:04:22,866 will, uh, uh, Fisher Stevens was in succession. 91 00:04:22,866 --> 00:04:26,436 He was, he was a, a lawyer, a a smarmy lawyer, which 92 00:04:26,646 --> 00:04:28,086 always plays smarmy characters 93 00:04:28,146 --> 00:04:31,296 yeah, I was just thinking, because I remember him from the blacklist 94 00:04:31,296 --> 00:04:32,766 where he plays Marvin, the lawyer. 95 00:04:32,961 --> 00:04:36,411 Yeah, got, he's got kind of the lawyer face. 96 00:04:36,411 --> 00:04:39,531 I'm glad that you, you finally realized the error of your ways. 97 00:04:39,996 --> 00:04:41,166 But did you know he was 98 00:04:41,661 --> 00:04:43,506 No, no, I didn't. 99 00:04:43,731 --> 00:04:46,551 I guess I always brought it up just like you, like I would bring 100 00:04:47,121 --> 00:04:52,521 Ben Kingsley playing Gandhi and, um, as just another example of, uh, you 101 00:04:52,521 --> 00:04:57,551 know, what would we call it, brown face, I guess we'd call it brown face. 102 00:04:57,551 --> 00:04:57,911 Yeah. 103 00:04:58,121 --> 00:04:58,451 Yeah. 104 00:04:58,661 --> 00:05:04,271 But people taking actor and there's been a lot of those great roles throughout the 105 00:05:04,721 --> 00:05:05,021 Great. 106 00:05:05,021 --> 00:05:08,231 You know, great roles played by very not, 107 00:05:09,131 --> 00:05:11,591 you know, people that are not of that ethnic group. 108 00:05:11,591 --> 00:05:11,891 Yeah. 109 00:05:12,621 --> 00:05:16,821 and I think maybe also at the time, right, there weren't many 110 00:05:16,821 --> 00:05:19,311 Indian actors in Hollywood at all. 111 00:05:20,586 --> 00:05:26,226 And I would rather have the fact, or I would rather it like the movie be made 112 00:05:26,646 --> 00:05:30,306 with someone who is non-Indian, rather, because it's a great movie. 113 00:05:30,306 --> 00:05:30,366 I 114 00:05:30,366 --> 00:05:30,786 don't know. 115 00:05:30,966 --> 00:05:31,416 You've seen it, 116 00:05:31,461 --> 00:05:31,851 good movie. 117 00:05:31,851 --> 00:05:31,941 Yeah. 118 00:05:32,061 --> 00:05:32,301 Yeah. 119 00:05:32,706 --> 00:05:35,436 So I would rather have that rather than not having the movie at all. 120 00:05:36,141 --> 00:05:37,971 Hmm, I see what you're saying. 121 00:05:38,031 --> 00:05:38,781 I see what you're saying. 122 00:05:39,231 --> 00:05:39,651 Yeah. 123 00:05:39,681 --> 00:05:44,841 And of course, you know, we have the same challenge with, uh, Asian, uh, actors, 124 00:05:45,231 --> 00:05:45,621 right? 125 00:05:45,651 --> 00:05:48,981 Uh, there's literally only three Chinese actors in all of Hollywood. 126 00:05:49,191 --> 00:05:53,031 Like if you, if you look at like the Chinese roles, they've gone to 127 00:05:53,031 --> 00:05:55,266 literally like one, there's one guy. 128 00:05:56,161 --> 00:06:02,221 Uh, I forgot how many roles he's had, but he has had a prolific career playing 129 00:06:02,281 --> 00:06:04,141 every Chinese person that you know. 130 00:06:04,771 --> 00:06:11,761 Um, but, um, anyway, so we're gonna talk about something that we've 131 00:06:11,761 --> 00:06:14,851 alluded to a little bit on the podcast. 132 00:06:14,911 --> 00:06:20,521 Uh, sort of tell the final saga of what I'm calling the backup from Hell. 133 00:06:22,506 --> 00:06:27,486 I may maybe, uh, we should probably phrase that slightly differently. 134 00:06:27,486 --> 00:06:28,506 It's probably the, 135 00:06:31,626 --> 00:06:33,636 the backup that keeps giving. 136 00:06:34,191 --> 00:06:37,131 the back, the backup that, yeah. 137 00:06:37,251 --> 00:06:38,661 Uh, what a mess. 138 00:06:39,091 --> 00:06:40,381 The beginning of the story 139 00:06:40,771 --> 00:06:46,171 that I was asked to do a backup of two Synology boxes that they 140 00:06:46,171 --> 00:06:49,441 were, uh, repurposing, right? 141 00:06:49,441 --> 00:06:52,711 So they were, um, going to move the data. 142 00:06:52,716 --> 00:06:56,821 They, they were gonna reuse these servers, but they wanted to get a backup of the, of 143 00:06:56,821 --> 00:06:59,461 the, the data before they moved it off of 144 00:06:59,566 --> 00:07:00,376 Backup is good. 145 00:07:00,406 --> 00:07:00,676 Yeah, 146 00:07:00,796 --> 00:07:01,696 Backup is good. 147 00:07:01,726 --> 00:07:01,941 Yeah. 148 00:07:01,996 --> 00:07:05,506 Apparently they hadn't had a backup of the, of these servers before. 149 00:07:05,626 --> 00:07:12,045 And, um, then the, the, um, and, and , they said it was 150 00:07:12,045 --> 00:07:13,485 about 40 terabytes of data. 151 00:07:13,525 --> 00:07:18,215 That's the information that I was given and after I had started doing 152 00:07:18,215 --> 00:07:24,695 the backup, I very quickly realized that 40 terabytes might have been. 153 00:07:27,500 --> 00:07:28,400 An understatement. 154 00:07:30,495 --> 00:07:31,725 You, found additional data around 155 00:07:31,815 --> 00:07:33,045 right as you 156 00:07:33,120 --> 00:07:33,810 data. 157 00:07:33,930 --> 00:07:34,650 Yeah. 158 00:07:34,740 --> 00:07:38,070 Uh, so it turned out that it wasn't like 40 terabytes of data. 159 00:07:38,070 --> 00:07:40,350 It was more like 400 terabytes of 160 00:07:40,410 --> 00:07:41,640 Yeah, and 161 00:07:42,180 --> 00:07:45,900 I'm guessing because these were systems that were kind of probably off on the 162 00:07:45,900 --> 00:07:48,060 side, they hadn't been used in a while. 163 00:07:48,810 --> 00:07:52,470 Like that's, I think, the problem, and I think we talked about this in one of 164 00:07:52,620 --> 00:07:58,380 our episodes about sort of systems that kind of get stored away in the corner. 165 00:07:58,380 --> 00:07:59,640 No one worries about 166 00:07:59,640 --> 00:07:59,820 it. 167 00:07:59,820 --> 00:08:00,180 Right? 168 00:08:00,180 --> 00:08:02,670 And do you leave it powered on your old backup systems? 169 00:08:03,270 --> 00:08:03,510 Right. 170 00:08:03,510 --> 00:08:04,530 We just talked about that. 171 00:08:04,620 --> 00:08:06,780 And so I think that becomes a challenge. 172 00:08:06,780 --> 00:08:09,810 It's when you have these systems that are no longer actively being 173 00:08:09,810 --> 00:08:11,940 used, it kind of gets away from you. 174 00:08:12,495 --> 00:08:13,005 Yeah. 175 00:08:13,095 --> 00:08:13,455 Yeah. 176 00:08:13,455 --> 00:08:17,505 And so the customer really didn't have any idea just how much data that they 177 00:08:17,505 --> 00:08:21,855 were dealing with here, out to be, like I said, like close to half a petabyte of 178 00:08:21,900 --> 00:08:22,260 Yeah. 179 00:08:22,530 --> 00:08:26,100 And, and for you, that changes things significantly because 180 00:08:26,715 --> 00:08:28,785 changes the backup design like massively. 181 00:08:28,785 --> 00:08:29,115 Yeah. 182 00:08:29,400 --> 00:08:29,700 Yeah. 183 00:08:29,700 --> 00:08:32,490 because your backup target, I think you had mentioned previously 184 00:08:32,490 --> 00:08:33,870 that it was like a server, right? 185 00:08:33,870 --> 00:08:35,610 That you were backing this data up to 186 00:08:36,235 --> 00:08:39,295 I was backing it up via a server, a window server. 187 00:08:40,015 --> 00:08:43,225 And, um, and tape, right? 188 00:08:43,225 --> 00:08:50,455 Had tape, but it's sized like, um, you know, four 40 terabytes. 189 00:08:50,965 --> 00:08:56,215 And so, which is, which is basically the, the, the server and the tape 190 00:08:56,215 --> 00:08:59,005 library was the perfect size for that. 191 00:08:59,515 --> 00:09:04,685 But as I started realizing that it was figure, it was filling up. 192 00:09:05,525 --> 00:09:13,475 And again, this, this is my fault for not really looking at the size of the 193 00:09:13,475 --> 00:09:17,645 data before really jumping in there, but basically I realized very quickly 194 00:09:17,645 --> 00:09:19,655 that this was a whole lot more data 195 00:09:19,745 --> 00:09:20,345 than, 196 00:09:20,465 --> 00:09:21,065 than, 197 00:09:21,065 --> 00:09:25,655 That you expect it and and I think just kind of looking at lessons 198 00:09:25,655 --> 00:09:29,825 learned as you're a backup admin who is being told, Hey, this new 199 00:09:29,825 --> 00:09:31,625 application is coming online. 200 00:09:32,435 --> 00:09:36,995 Make sure that you understand like what is the expected growth of that application. 201 00:09:36,995 --> 00:09:42,245 Because what you size for, say, a five terabyte database with a 1% growth 202 00:09:42,245 --> 00:09:46,145 is very different than like a file server with like a 50% growth rate. 203 00:09:46,655 --> 00:09:47,915 Yeah, exactly. 204 00:09:48,095 --> 00:09:51,815 Um, and just because somebody says they have 10 terabytes of data doesn't mean 205 00:09:51,815 --> 00:09:53,165 that they have 10 terabytes of data. 206 00:09:53,165 --> 00:09:57,035 So you mentioned you had a backup server, you had a tape drive. 207 00:09:57,305 --> 00:09:59,945 Is there a reason you chose to use tape 208 00:10:01,055 --> 00:10:05,915 Well, the, I mean, tape is great for long-term retention of data, 209 00:10:05,915 --> 00:10:08,435 which is what this customer wanted. 210 00:10:08,435 --> 00:10:10,925 They wanted to hold onto this data for a long period of time, 211 00:10:10,925 --> 00:10:12,095 and that's where tape is great. 212 00:10:12,725 --> 00:10:17,525 And tape also is has, uh, you know, if you're able to properly feed it, 213 00:10:18,275 --> 00:10:20,825 tape is actually, can be quite fast. 214 00:10:21,970 --> 00:10:30,850 the challenge that I had when backing up this data that for various reasons, 215 00:10:31,810 --> 00:10:35,890 which I think I, I think by the end I sort of figured out the, the 216 00:10:35,890 --> 00:10:38,650 core reason for various reasons. 217 00:10:39,530 --> 00:10:44,570 Individual backups off of the, these filers, the, they were just 218 00:10:44,660 --> 00:10:47,360 slow, just, um, you know, they were 219 00:10:47,420 --> 00:10:48,500 Like how slow, slow. 220 00:10:48,800 --> 00:10:57,200 like, slow, was like, like, like three and a half kilobytes a second slow. 221 00:10:58,595 --> 00:11:01,115 So like slower than like a 56 K modem back 222 00:11:01,190 --> 00:11:01,595 Yeah. 223 00:11:02,060 --> 00:11:02,600 Right. 224 00:11:03,170 --> 00:11:06,590 And you can multiplex all you want. 225 00:11:06,590 --> 00:11:11,540 So first off, you know, I, I was using NetBackup, which, you know, NetBackup, it 226 00:11:11,540 --> 00:11:14,150 did a great job at what we had available. 227 00:11:15,440 --> 00:11:21,470 Um, the, challenge was that because I couldn't put. 228 00:11:22,085 --> 00:11:24,845 The client on the filers themselves. 229 00:11:24,845 --> 00:11:31,625 So the, was a way allegedly to put a, a backup client on the filer, 230 00:11:31,625 --> 00:11:33,035 but I could never get that to work. 231 00:11:33,875 --> 00:11:38,440 And so I had to back up over SMB because I'm backing up 232 00:11:38,440 --> 00:11:40,480 over SMB, I'm just, I'm just. 233 00:11:41,080 --> 00:11:42,610 I'm just limited at what that was, right? 234 00:11:42,610 --> 00:11:42,850 What, 235 00:11:43,420 --> 00:11:47,710 I could get, and because I'm backing up over SMB, the client 236 00:11:48,160 --> 00:11:49,660 is just the backup server, 237 00:11:49,960 --> 00:11:50,410 right? 238 00:11:51,040 --> 00:11:54,190 So instead of running a backup from two clients, I'm running a backup from one 239 00:11:54,190 --> 00:11:56,320 client because that's the backup server. 240 00:11:56,320 --> 00:11:57,820 I'm backing it up over SMB. 241 00:11:58,300 --> 00:12:01,510 And because of that, I'm limited to the number of jobs I can run at one time. 242 00:12:01,510 --> 00:12:07,540 NetBackup, um, says 99 99 jobs, which should say, gee, that 243 00:12:07,645 --> 00:12:08,050 sounds like a 244 00:12:08,155 --> 00:12:08,970 Nine problems. 245 00:12:09,550 --> 00:12:09,790 right? 246 00:12:11,005 --> 00:12:18,235 But, but the thing is, towards the end, as I was running a lot of these backups, 247 00:12:18,625 --> 00:12:27,625 the aggregate speed of like 99 backups was only like 30, 40 megabytes a second, 248 00:12:28,495 --> 00:12:28,645 you 249 00:12:28,645 --> 00:12:30,415 you're talking about 400 terabytes of data to 250 00:12:30,505 --> 00:12:32,755 400 terabytes of data doing the math. 251 00:12:33,085 --> 00:12:35,455 I backed up for months, 252 00:12:36,055 --> 00:12:36,255 right? 253 00:12:36,415 --> 00:12:38,635 And I tried all these different things. 254 00:12:38,635 --> 00:12:42,385 Uh, you know, num, you know, was I running too many backups at a time? 255 00:12:42,385 --> 00:12:44,185 Was I running not enough backups at a time? 256 00:12:44,185 --> 00:12:48,295 You know, it, um, you know, and then the problem is every, every 257 00:12:48,295 --> 00:12:50,185 test would take days or weeks. 258 00:12:50,585 --> 00:12:51,875 Think we should mention one thing. 259 00:12:52,505 --> 00:12:54,905 You were talking about these test taking days or 260 00:12:55,065 --> 00:12:55,485 mm-hmm. 261 00:12:56,135 --> 00:13:00,965 and then do you wanna mention sort of some of the issues you ran into with these long 262 00:13:00,965 --> 00:13:05,195 running jobs just due to infrastructure or 263 00:13:06,110 --> 00:13:06,830 Yeah. 264 00:13:07,010 --> 00:13:08,795 other issues in the environment? 265 00:13:09,050 --> 00:13:15,770 yeah, you, you backups are not made to run over weeks or months. 266 00:13:16,250 --> 00:13:19,070 Just backup infrastructure isn't made to work like that. 267 00:13:20,150 --> 00:13:24,200 And so when you do backups over weeks or months. 268 00:13:24,890 --> 00:13:32,060 Weird things happen that, cause you know, consternation, one of the things 269 00:13:32,060 --> 00:13:38,780 is LTO tape drives are great, but like we were using like the half high LTO 270 00:13:38,780 --> 00:13:43,460 drives and as far as I could tell, their duty cycle was not meant to 271 00:13:43,460 --> 00:13:45,200 be a hundred percent for two months. 272 00:13:46,085 --> 00:13:46,685 Right. 273 00:13:46,835 --> 00:13:51,245 Um, they're meant to be backed up for, you know, several hours and then give 274 00:13:51,245 --> 00:13:53,645 'em a rest and then back up several hours and then give 'em a rest. 275 00:13:54,135 --> 00:13:57,585 I was just beating the crap outta these things for weeks or months at a time. 276 00:13:57,645 --> 00:14:02,175 And what would happen is after some significant period of time, 277 00:14:02,445 --> 00:14:03,915 it would just go write error. 278 00:14:05,325 --> 00:14:08,895 And that's fine when a backup runs for a few hours and then just try again. 279 00:14:08,895 --> 00:14:12,045 But if you, but if it took you two weeks or three weeks to get to that point 280 00:14:12,045 --> 00:14:14,805 and then you get a write error, um, 281 00:14:14,805 --> 00:14:15,135 then 282 00:14:15,525 --> 00:14:18,255 it's not like you could restart these jobs either, right? 283 00:14:18,255 --> 00:14:19,365 I think you're running into 284 00:14:20,055 --> 00:14:20,325 Yeah. 285 00:14:20,325 --> 00:14:20,445 Well, 286 00:14:20,490 --> 00:14:23,565 I mean, I mean, I could restart em, but, but it's like after 287 00:14:23,625 --> 00:14:27,705 a period of time I became, I. 288 00:14:29,280 --> 00:14:33,360 I eventually got to the point where I said, tape is not my friend. 289 00:14:33,420 --> 00:14:34,770 I, anybody who 290 00:14:34,875 --> 00:14:35,925 this is coming from Mr. 291 00:14:35,925 --> 00:14:36,525 Backup. 292 00:14:36,600 --> 00:14:42,330 know anybody who listens to this podcast knows that I am, I am a friend of tape, 293 00:14:42,870 --> 00:14:43,185 right? 294 00:14:43,290 --> 00:14:50,520 I believe strongly in tape for a lot of reasons, but I don't think that, uh, 295 00:14:50,550 --> 00:14:54,990 specific and, and you know, maybe the, my LTO friends can chime in here, but I don't 296 00:14:54,990 --> 00:14:59,610 think that these tape drives were designed to be backed up to like this for weeks and 297 00:14:59,610 --> 00:15:04,460 months at a time, 24 7 with no, because as soon as one, I was multiplexing 298 00:15:04,460 --> 00:15:06,320 as many backups together as I could. 299 00:15:07,230 --> 00:15:10,710 And when one backup would finish, I would just add another backup onto it, right? 300 00:15:10,710 --> 00:15:11,070 Because 301 00:15:11,520 --> 00:15:13,815 I, I could, I could, I. 302 00:15:13,875 --> 00:15:18,255 what I couldn't do is I couldn't say, well, let's do these 10 backups, let 303 00:15:18,255 --> 00:15:20,925 them run until they're finished, and then we'll do the next 10 backups. 304 00:15:21,315 --> 00:15:24,345 And that would've given the tape drives a, a moment to breathe, I think. 305 00:15:24,795 --> 00:15:32,865 But, uh, I couldn't do that because the, because we, we just 306 00:15:32,865 --> 00:15:33,885 didn't have that kind of time. 307 00:15:33,885 --> 00:15:34,305 And so I 308 00:15:34,335 --> 00:15:36,525 was just, I was just try, you know, tagging it 309 00:15:36,645 --> 00:15:40,455 and, and I know you've always talked about like the shoe shining problem, 310 00:15:40,455 --> 00:15:44,775 given that you're not going very fast with these backups, right. 311 00:15:45,000 --> 00:15:48,615 Do you think that also led to some issues as well for the tape drives? 312 00:15:48,825 --> 00:15:49,185 yeah. 313 00:15:49,185 --> 00:15:53,385 So again, the core problem was that each individual backup was running slow. 314 00:15:54,195 --> 00:15:58,485 matter how many of them that I multiplex together, it was not enough 315 00:15:58,485 --> 00:16:00,345 speed to make the tape drive happy. 316 00:16:00,345 --> 00:16:02,025 And so, yes, the tape driver shoe shining. 317 00:16:02,385 --> 00:16:06,105 And when a tape tribe is continually shoe shining, the tape drive will fail. 318 00:16:07,800 --> 00:16:10,620 And so everything, I remember learning about tape drives was 319 00:16:10,620 --> 00:16:12,960 coming back to haunt me, right? 320 00:16:13,140 --> 00:16:17,460 Um, this is all of the design that I was, that I had done throughout 321 00:16:17,460 --> 00:16:21,570 the years on backup, um, you know, 322 00:16:22,170 --> 00:16:24,450 um, backup system 323 00:16:24,630 --> 00:16:25,290 And system. 324 00:16:26,100 --> 00:16:29,460 all of the things that, you know, what do you do when the backups, you know? 325 00:16:29,580 --> 00:16:32,850 And so I came to understand 326 00:16:33,750 --> 00:16:36,810 that the only way I was gonna finish this backup was to do it to disc. 327 00:16:37,575 --> 00:16:41,235 And just quickly before you move on, I think along the way, didn't 328 00:16:41,235 --> 00:16:43,755 you also have a tape drive that failed that you then had to go 329 00:16:43,785 --> 00:16:46,275 Oh, multiple Multiple times. 330 00:16:46,905 --> 00:16:50,715 Swap out tape drives, reboot tape drives, put in cleaning tapes and tape drives. 331 00:16:50,805 --> 00:16:54,855 And by the way, that's another thing is the way tape drives normally do 332 00:16:54,915 --> 00:16:58,245 is you run them for a certain number of hours and then there's a cleaning 333 00:16:58,245 --> 00:16:59,595 tape that goes in there and cleans it. 334 00:16:59,775 --> 00:17:03,135 And when you have a robotic library, that happens automatically. 335 00:17:03,435 --> 00:17:06,225 Well, when you just run the tape drive for. 336 00:17:06,990 --> 00:17:09,570 Two months, you know, that 337 00:17:10,530 --> 00:17:12,570 And so at some point the tape drive just fails. 338 00:17:13,090 --> 00:17:13,380 Yeah. 339 00:17:13,680 --> 00:17:14,910 um, yeah. 340 00:17:14,970 --> 00:17:21,900 And so I ultimately that the only way to get this done was to, um, you know, 341 00:17:21,990 --> 00:17:26,010 buy, uh, enough disc to back this up. 342 00:17:26,010 --> 00:17:26,975 And that wasn't cheap. 343 00:17:27,990 --> 00:17:33,180 Uh, but I, I didn't think that there was any other way that this was ever 344 00:17:33,180 --> 00:17:37,930 going to get done 'cause again, the core problem that we've had with tape 345 00:17:38,560 --> 00:17:42,520 for the last three decades has been that the backup, if the backup isn't 346 00:17:42,520 --> 00:17:46,630 too fast enough for the tape drive it's a, it's a fundamental mismatch 347 00:17:47,410 --> 00:17:47,770 right? 348 00:17:48,070 --> 00:17:50,170 And so we use to make that better. 349 00:17:50,170 --> 00:17:53,410 But if the multi, but if the speed you're dealing with is in kilobytes a second, 350 00:17:54,145 --> 00:17:54,445 Yeah. 351 00:17:55,315 --> 00:18:00,565 Well, and especially 'cause you're limited by those two, uh, Synology boxes, right? 352 00:18:00,565 --> 00:18:02,155 Which are limiting your bandwidth, right? 353 00:18:02,155 --> 00:18:02,635 It's not like 354 00:18:02,680 --> 00:18:03,130 Yeah. 355 00:18:03,130 --> 00:18:05,815 Synology boxes you can then pull from, 356 00:18:06,850 --> 00:18:11,830 Yeah, and I was, I was watching, like, I was running every kind of tool I could 357 00:18:11,830 --> 00:18:15,970 run to see, like, I wasn't overt tasking. 358 00:18:16,240 --> 00:18:18,970 The, that was the really weird part is that the, it's not like the 359 00:18:18,970 --> 00:18:22,900 Synology boxes were saying, you're really beating the crap out of it. 360 00:18:22,900 --> 00:18:23,470 You shouldn't do so 361 00:18:23,650 --> 00:18:24,550 backups at a time. 362 00:18:25,420 --> 00:18:28,790 It wasn't, it, it was, I didn't have a high I/O wait. 363 00:18:29,320 --> 00:18:32,350 I didn't have high CPU, I didn't have high ram. 364 00:18:33,265 --> 00:18:35,185 There, there was no, there was no 365 00:18:35,545 --> 00:18:39,055 rhyme or as to why we'll get to the rhyme or reason later. 366 00:18:39,535 --> 00:18:40,405 I figured it out. 367 00:18:41,605 --> 00:18:47,935 Um, but, but I knew the tape and I knew the tape and this wasn't gonna work. 368 00:18:47,935 --> 00:18:52,625 So, so I had to bring in, uh, a couple of other Synology disc arrays, by the 369 00:18:52,625 --> 00:18:57,995 way, and populate them with enough disc to handle all of this, uh, this backup. 370 00:18:58,205 --> 00:18:58,535 Right. 371 00:18:58,595 --> 00:18:58,835 Yeah, 372 00:18:59,945 --> 00:19:01,145 And, um. 373 00:19:01,535 --> 00:19:02,075 Then 374 00:19:02,165 --> 00:19:04,115 but that wasn't without its issues either. 375 00:19:04,175 --> 00:19:04,535 Right? 376 00:19:04,535 --> 00:19:07,535 When you, when you brought those in, that wasn't without its issues either. 377 00:19:07,955 --> 00:19:09,425 No, it wasn't without issues. 378 00:19:09,425 --> 00:19:15,395 And the other thing, what I needed to do was to, I felt that with, in terms of the 379 00:19:15,395 --> 00:19:21,875 number of directories that were remaining, I wasn't sure like the different sizes. 380 00:19:21,875 --> 00:19:23,015 So what I did was I split, I. 381 00:19:23,975 --> 00:19:26,195 Those jobs into many smaller jobs. 382 00:19:26,255 --> 00:19:29,315 NetBackup is really good at like running thousands of jobs, right? 383 00:19:29,315 --> 00:19:34,625 So rather than just have a hundred jobs, I turned that into like 2,400 jobs. 384 00:19:34,625 --> 00:19:35,105 Like I went, 385 00:19:35,165 --> 00:19:38,075 I went another level deep and created a policy for each of these 386 00:19:38,075 --> 00:19:45,425 directories, and then I ran those and it was running for a while. 387 00:19:45,425 --> 00:19:48,965 It was, it was, you know, again, more time. 388 00:19:49,475 --> 00:19:51,545 And what I started seeing. 389 00:19:53,390 --> 00:19:56,750 Were these jobs that were like an individual job that was running 390 00:19:57,500 --> 00:19:58,700 inordinate amount of time. 391 00:20:00,305 --> 00:20:03,305 but you also had some jobs that would finish like super fast, right? 392 00:20:03,305 --> 00:20:03,545 Like 393 00:20:03,800 --> 00:20:05,240 They'd finish five, they'd finish in 394 00:20:05,930 --> 00:20:08,780 Some of 'em, some of 'em finished in five minutes, some 'em would finish. 395 00:20:08,960 --> 00:20:13,940 But I noticed that over time there were certain policies that were running for 396 00:20:13,940 --> 00:20:20,720 really, really long periods of time, and eventually started poking around. 397 00:20:21,860 --> 00:20:26,960 when I discovered what ultimately was the, the true culprit. 398 00:20:28,070 --> 00:20:32,660 And, uh, anyone who's been around backup for a long time 399 00:20:32,660 --> 00:20:34,250 has seen this culprit before. 400 00:20:35,330 --> 00:20:41,630 It's just, this is the worst example of this culprit that I've ever seen. 401 00:20:43,475 --> 00:20:44,050 And what is that? 402 00:20:46,100 --> 00:20:49,610 We affectionately refer to it as the million file problem. 403 00:20:51,120 --> 00:20:51,240 Hmm. 404 00:20:52,685 --> 00:20:59,465 Because remember, again, going back to that, um, that client back from 405 00:20:59,465 --> 00:21:02,675 25 years ago, we had one server. 406 00:21:03,500 --> 00:21:06,620 That was going to be storing a bunch of images and it was going 407 00:21:06,620 --> 00:21:08,480 to result in millions of files. 408 00:21:08,480 --> 00:21:12,830 And we knew that back then that the million file problem is, a real problem. 409 00:21:13,430 --> 00:21:18,080 and and million file problem ev over, over the network is even worse, right? 410 00:21:18,080 --> 00:21:19,460 Because everything is, is, is a 411 00:21:19,505 --> 00:21:19,671 round trip. 412 00:21:19,711 --> 00:21:23,341 The way we fixed it back then was we used a product back then called 413 00:21:23,341 --> 00:21:26,821 flashback, which would back up at the raw level, but store the 414 00:21:27,391 --> 00:21:32,521 information, and that was not available to me. 415 00:21:32,521 --> 00:21:33,121 Why? 416 00:21:34,576 --> 00:21:36,076 Because that product no longer exists 417 00:21:36,541 --> 00:21:37,021 No. 418 00:21:37,876 --> 00:21:40,156 because it doesn't run on a Synology box. 419 00:21:40,651 --> 00:21:41,311 Right. 420 00:21:42,391 --> 00:21:44,021 Remember, I'm not the Synology 421 00:21:44,371 --> 00:21:46,291 All it was was an SMB mount to me. 422 00:21:46,561 --> 00:21:46,921 Right? 423 00:21:48,361 --> 00:21:52,051 And by the way, for those curious, yes, I tested SMB, I tested NFS. 424 00:21:52,056 --> 00:21:52,696 It didn't matter. 425 00:21:54,181 --> 00:21:54,243 It didn't matter. 426 00:21:54,991 --> 00:21:58,021 Um, the um. 427 00:21:59,371 --> 00:21:59,821 And 428 00:21:59,881 --> 00:22:03,001 by the way, this was a constant, you know, you know the phrase, never, never 429 00:22:03,001 --> 00:22:04,891 go into battle with an untested weapon. 430 00:22:05,341 --> 00:22:10,681 This was constant example of I am in the battle, I'm in the stuff, 431 00:22:10,861 --> 00:22:12,601 and now I'm trying to test stuff 432 00:22:13,201 --> 00:22:17,791 and, and I did to try to make things better, just made it take longer 433 00:22:17,791 --> 00:22:19,561 and the client just had to wait. 434 00:22:20,011 --> 00:22:23,341 And the the client was incredibly patient, honestly. 435 00:22:24,481 --> 00:22:30,511 And, and you know, I did my best to say, look, I, I've been doing this for 30 436 00:22:30,511 --> 00:22:32,461 years, I've never seen anything like this. 437 00:22:32,461 --> 00:22:32,821 Right. 438 00:22:33,481 --> 00:22:36,301 And that, that helped. 439 00:22:36,631 --> 00:22:40,081 But in the end, I was backing up. 440 00:22:40,501 --> 00:22:44,341 You know, we got down to, I, I learned a way to identify which 441 00:22:44,341 --> 00:22:45,601 were the problem directories. 442 00:22:46,441 --> 00:22:49,771 So I would kick off a policy and I would watch, and I would notice 443 00:22:49,771 --> 00:22:53,521 that had run for, let's say an hour. 444 00:22:54,646 --> 00:22:58,426 And it listed, let's say 300,000 files backed up. 445 00:22:59,476 --> 00:23:00,406 kilobytes. 446 00:23:00,961 --> 00:23:01,471 Hmm. 447 00:23:02,371 --> 00:23:04,861 Literally there's, there's a kilobyte column that 448 00:23:05,371 --> 00:23:07,711 kilobytes of byte and there's no value in there. 449 00:23:07,861 --> 00:23:10,921 We backed up 300,000 files, no kilobytes. 450 00:23:11,671 --> 00:23:14,281 so that, that helped me identify these problem 451 00:23:14,401 --> 00:23:15,421 Problem child. 452 00:23:15,481 --> 00:23:15,571 Yeah. 453 00:23:15,841 --> 00:23:19,471 it and let the other non-problem policies finish. 454 00:23:19,471 --> 00:23:19,651 And 455 00:23:19,741 --> 00:23:20,041 Right. 456 00:23:20,096 --> 00:23:20,386 Yeah. 457 00:23:20,671 --> 00:23:27,151 up getting down to like 150 policies that were the problem policies. 458 00:23:27,211 --> 00:23:30,181 And so I backed them up and I was able to get them. 459 00:23:30,631 --> 00:23:37,771 Over time, I was able to get them backed up, and then finally I got down to about 460 00:23:37,831 --> 00:23:40,051 20 policies, I think somewhere around 461 00:23:41,011 --> 00:23:41,611 policies. 462 00:23:41,611 --> 00:23:42,001 Go ahead. 463 00:23:42,391 --> 00:23:45,841 And at this point when you're down to the 20, like some of these have 464 00:23:45,841 --> 00:23:48,511 been running for a long time, right? 465 00:23:48,511 --> 00:23:49,051 Like how? 466 00:23:49,831 --> 00:23:52,411 like two months backups that have been running for two months, 467 00:23:52,411 --> 00:23:53,851 successfully running for two months. 468 00:23:54,541 --> 00:23:54,781 Yeah. 469 00:23:55,471 --> 00:23:59,731 And what was good was at this point again. 470 00:24:00,421 --> 00:24:03,391 Like this is information that would've been really helpful to have at the 471 00:24:03,391 --> 00:24:07,531 beginning, but it was information that, to get all this information at the 472 00:24:07,531 --> 00:24:11,791 beginning, it would've taken time to, like we, we just wanted to get started. 473 00:24:12,096 --> 00:24:12,296 Yeah. 474 00:24:13,411 --> 00:24:19,711 What I ended up finding was that, um, these backups, um. 475 00:24:21,191 --> 00:24:24,916 The, the, there were millions and millions and millions, like one of the, one 476 00:24:24,916 --> 00:24:28,636 of the directories that I was backing up, it had 99 million files in it, 477 00:24:28,876 --> 00:24:34,576 one directory, 99 million files, and eventually what I realized was that 478 00:24:34,666 --> 00:24:39,796 again, the problem this time was just SMB. 479 00:24:40,666 --> 00:24:44,206 So the fact that every one of these files results in a round 480 00:24:44,206 --> 00:24:47,596 trip conversation, possibly multiple round trip conversations. 481 00:24:47,731 --> 00:24:47,821 Yep. 482 00:24:49,291 --> 00:24:53,101 And I realized that the only way I was gonna back up these truly problem 483 00:24:53,101 --> 00:24:56,311 directories was to back them up locally. 484 00:24:56,851 --> 00:24:58,501 But how do I back them up locally? 485 00:24:58,651 --> 00:25:02,431 Well, luckily this is when I just, you know, basically go back 486 00:25:02,431 --> 00:25:06,241 to dumb, dumb old backup tools. 487 00:25:06,691 --> 00:25:12,631 And so I was able to run a backup using tar logged in locally 488 00:25:12,631 --> 00:25:17,011 on the filers, and then just. 489 00:25:17,686 --> 00:25:23,746 Directing the tarball across the network that finally worked. 490 00:25:23,896 --> 00:25:24,646 That's crazy. 491 00:25:24,646 --> 00:25:27,016 So you had these 20 jobs, right? 492 00:25:27,016 --> 00:25:32,206 And some of them you said were running for 60 plus days, and then you sort of 493 00:25:32,206 --> 00:25:34,546 were like, okay, let me start this over. 494 00:25:34,546 --> 00:25:37,906 And by the way, you were kind of forced to start them over 495 00:25:37,906 --> 00:25:39,376 because something happened right? 496 00:25:39,376 --> 00:25:39,646 At 497 00:25:39,736 --> 00:25:39,796 yeah. 498 00:25:39,796 --> 00:25:42,226 Something some unknown thing. 499 00:25:42,496 --> 00:25:43,961 Um, I think I. 500 00:25:44,241 --> 00:25:45,166 I, I, I don't know. 501 00:25:45,196 --> 00:25:46,246 I, I actually don't know 502 00:25:46,726 --> 00:25:48,556 what caused it, but they, they did fail 503 00:25:49,186 --> 00:25:49,816 and, 504 00:25:49,816 --> 00:25:51,466 And you were like, I'm not gonna start these 505 00:25:51,766 --> 00:25:52,006 yeah. 506 00:25:52,006 --> 00:25:52,876 I'm not gonna start 'em again. 507 00:25:52,876 --> 00:25:53,566 It's just, yeah. 508 00:25:53,896 --> 00:25:54,436 Well, Because 509 00:25:54,436 --> 00:25:58,756 like, one of jobs, the, the one with 99 fi, 99 million 510 00:25:58,756 --> 00:26:00,166 files, we were nowhere near. 511 00:26:00,166 --> 00:26:00,226 I. 512 00:26:01,186 --> 00:26:01,516 yeah. 513 00:26:02,236 --> 00:26:03,586 After 60 days you were barely 514 00:26:03,586 --> 00:26:04,156 yeah, yeah. 515 00:26:04,156 --> 00:26:05,896 We're barely, barely scratching the surface. 516 00:26:05,956 --> 00:26:10,516 so I'm like, I, I, I don't have, I don't have that, you know, I, I don't 517 00:26:10,516 --> 00:26:15,616 have the amount of time that it would take, so, so I switched to, you know, 518 00:26:15,646 --> 00:26:19,276 experimentally once again, experimentally, I'm experimenting on the fly, I'm 519 00:26:19,276 --> 00:26:21,106 doing development in production. 520 00:26:21,736 --> 00:26:25,336 Uh, I was like, well, let me see how long, how quick a tar ball would run. 521 00:26:26,191 --> 00:26:27,121 I ran a tar ball. 522 00:26:27,121 --> 00:26:29,101 I remember for like a day, you remember this? 523 00:26:29,101 --> 00:26:29,461 I ran a 524 00:26:30,031 --> 00:26:36,181 a day and it, I, I had a du of the size of the directory and after a day it had 525 00:26:36,181 --> 00:26:38,761 done like, like a half of it or something. 526 00:26:39,166 --> 00:26:39,526 Yeah. 527 00:26:40,306 --> 00:26:40,876 You're like, what? 528 00:26:40,936 --> 00:26:43,096 Once taking 66 days and barely scratch the 529 00:26:43,566 --> 00:26:43,856 yeah, 530 00:26:44,236 --> 00:26:45,106 You are mainly done. 531 00:26:45,136 --> 00:26:46,156 Almost done within a day. 532 00:26:46,261 --> 00:26:46,801 yeah. 533 00:26:46,801 --> 00:26:48,241 And so I was like, this is the way. 534 00:26:48,691 --> 00:26:49,141 Right. 535 00:26:49,801 --> 00:26:55,141 So it, it, it wasn't, it wasn't a way for everything because the, the, this 536 00:26:55,141 --> 00:27:00,241 was, um, because I, you know, I'm glad that I, that I use NetBackup for the 537 00:27:00,241 --> 00:27:03,211 bulk of it, because then I have the catalog data and, you know, and, um, 538 00:27:03,931 --> 00:27:04,471 but 539 00:27:04,816 --> 00:27:05,536 on the restore side. 540 00:27:05,581 --> 00:27:06,781 yeah, yeah. 541 00:27:06,811 --> 00:27:07,471 So this will. 542 00:27:07,816 --> 00:27:11,416 This will be the diff the restores will be more difficult for these 543 00:27:11,416 --> 00:27:13,186 like remaining 20 directories. 544 00:27:14,116 --> 00:27:15,466 I mean, not, not astronomically. 545 00:27:15,466 --> 00:27:16,036 So like, 546 00:27:16,156 --> 00:27:17,686 you know, can create a tarball, a 547 00:27:17,951 --> 00:27:19,636 list of this. 548 00:27:19,686 --> 00:27:21,036 So, you know, lessons learned, like, 549 00:27:21,276 --> 00:27:21,606 do that. 550 00:27:21,606 --> 00:27:25,566 Don't store millions of files on the other side of a, of an SMB box. 551 00:27:25,566 --> 00:27:25,956 I guess 552 00:27:26,061 --> 00:27:31,791 Yeah, so Well, and I think a couple things, even if it's not SMB, right? 553 00:27:32,241 --> 00:27:36,471 Just having that many files, because I think what people don't realize is 554 00:27:36,831 --> 00:27:40,611 even though the size of every disc has gotten significantly larger, right? 555 00:27:40,611 --> 00:27:43,311 You're talking like 18 terabyte, 20 terabyte disk 556 00:27:43,506 --> 00:27:44,076 Yeah. 557 00:27:44,601 --> 00:27:48,621 They can only handle so many operations per disc, right? 558 00:27:48,621 --> 00:27:49,791 That number hasn't changed. 559 00:27:49,791 --> 00:27:51,201 It's about a hundred per second. 560 00:27:51,891 --> 00:27:54,591 And so no matter how many, how big your disc is, right? 561 00:27:54,591 --> 00:28:00,441 If it was 21 terabyte discs, right, then you get 20 times a hundred iops. 562 00:28:01,011 --> 00:28:04,071 Versus if it's one 20 terabyte disc, you only still get that a hundred. 563 00:28:04,071 --> 00:28:08,721 So that's a big thing that people don't realize with these larger size discs. 564 00:28:09,111 --> 00:28:09,621 Yeah. 565 00:28:09,621 --> 00:28:11,091 And, and the thing was that the. 566 00:28:11,556 --> 00:28:13,386 That many files. 567 00:28:13,416 --> 00:28:18,096 So, because the problem, the, ultimately the problem wasn't disc io, the problem 568 00:28:18,846 --> 00:28:19,296 io. 569 00:28:19,386 --> 00:28:19,806 Right? 570 00:28:20,046 --> 00:28:21,036 Network latency. 571 00:28:21,396 --> 00:28:22,566 So, because 572 00:28:22,926 --> 00:28:26,556 when I actually ran, I ran two tar balls. 573 00:28:26,556 --> 00:28:26,586 I. 574 00:28:27,051 --> 00:28:28,731 Simultaneously is what I did. 575 00:28:28,731 --> 00:28:29,511 I using 576 00:28:30,141 --> 00:28:32,901 I just, I ran, I was always running two at a time. 577 00:28:33,621 --> 00:28:37,071 When I was running two at a time, I/O wait was sitting at 10, 578 00:28:37,701 --> 00:28:39,321 which is, is high, 579 00:28:39,681 --> 00:28:42,801 but I was like, well, it's got nothing else going on, so I'm, I'm 580 00:28:43,161 --> 00:28:43,431 it go. 581 00:28:43,431 --> 00:28:43,791 Right? 582 00:28:44,721 --> 00:28:48,861 The highest I/O wait ran during all of those hundreds of 583 00:28:48,861 --> 00:28:51,711 simultaneous backups was like four. 584 00:28:52,256 --> 00:28:52,546 yeah, 585 00:28:52,821 --> 00:28:54,621 So like I wasn't disc bound. 586 00:28:54,681 --> 00:28:55,041 I was 587 00:28:55,611 --> 00:28:59,421 bound, but not network bound in terms of throughput, network bound, in terms of 588 00:28:59,571 --> 00:29:00,261 Laid C, 589 00:29:00,771 --> 00:29:01,551 and 590 00:29:01,851 --> 00:29:05,151 of operations, just because SMB is very chatty. 591 00:29:05,631 --> 00:29:06,531 very chatty. 592 00:29:06,531 --> 00:29:08,421 It's probably the chattiest of the protocols, 593 00:29:09,261 --> 00:29:10,011 and 594 00:29:10,746 --> 00:29:11,346 we, you 595 00:29:11,421 --> 00:29:13,311 it was just a really combination. 596 00:29:13,326 --> 00:29:13,566 Yeah. 597 00:29:13,866 --> 00:29:19,476 And you know why this, and this is why backup vendors have their own protocols, 598 00:29:19,476 --> 00:29:22,416 like Data Domain has boost, right? 599 00:29:22,416 --> 00:29:25,146 To help alleviate and solve some of these issues. 600 00:29:25,536 --> 00:29:26,106 Yeah. 601 00:29:26,156 --> 00:29:29,636 You talked about, don't, don't do the somewhere we were talking about. 602 00:29:29,636 --> 00:29:30,596 Just don't do this. 603 00:29:31,076 --> 00:29:33,056 I, I'd like, I'd like to talk today. 604 00:29:33,416 --> 00:29:37,406 When I looked at these, these, uh, these directories that had these 605 00:29:38,246 --> 00:29:42,386 tens of millions of files, it was a structure that was very clearly 606 00:29:42,716 --> 00:29:44,546 created by some application. 607 00:29:45,386 --> 00:29:49,526 one of these directors had a common structure created by some. 608 00:29:50,096 --> 00:29:55,166 I'm gonna say stupid application that thought this was perfectly fine. 609 00:29:55,196 --> 00:30:00,356 That it was perfectly fine to create 99 million files for 610 00:30:00,371 --> 00:30:01,181 Do you know, I, 611 00:30:01,376 --> 00:30:01,916 item. 612 00:30:02,471 --> 00:30:07,481 I bet they were using the file system as a database 613 00:30:09,146 --> 00:30:09,746 I don't know. 614 00:30:10,526 --> 00:30:11,246 what it was. 615 00:30:11,891 --> 00:30:14,801 given just like the number of files and the size of those files. 616 00:30:15,641 --> 00:30:17,831 I know it was forensic type information 617 00:30:18,641 --> 00:30:21,641 and I, I don't, I clearly 618 00:30:21,806 --> 00:30:22,496 That, that's fine. 619 00:30:22,496 --> 00:30:23,156 Yeah, yeah, 620 00:30:23,471 --> 00:30:25,961 No, I'm just saying I clearly don't know enough about forensic stuff 621 00:30:25,961 --> 00:30:27,551 to know why they would want tens of 622 00:30:27,941 --> 00:30:28,541 of vials, 623 00:30:29,441 --> 00:30:29,741 but 624 00:30:30,881 --> 00:30:31,871 So where are you? 625 00:30:31,871 --> 00:30:35,621 So you talked about these 20 jobs that you were starting to do tarballs with. 626 00:30:35,981 --> 00:30:37,241 So where are you right now? 627 00:30:37,496 --> 00:30:42,056 So, so we finished all of them, but one, there was one that for some reason 628 00:30:42,056 --> 00:30:43,856 it, it, the file didn't look right. 629 00:30:43,856 --> 00:30:44,606 It was weird. 630 00:30:44,696 --> 00:30:50,336 Um, it, the, the, the backup completed, but the, some reason, the, the tarball, 631 00:30:50,516 --> 00:30:51,746 it just, it just didn't look right. 632 00:30:51,746 --> 00:30:52,676 I don't wanna go into details. 633 00:30:52,676 --> 00:30:53,276 It just didn't look 634 00:30:53,906 --> 00:30:55,196 so I'm rerunning that one. 635 00:30:55,196 --> 00:30:57,446 So it, based on its size and how well it's doing, it should 636 00:30:57,446 --> 00:30:58,736 finish in about a day or so. 637 00:30:59,396 --> 00:31:00,476 Um, and what I'm 638 00:31:00,521 --> 00:31:02,501 is a significant improvement in terms of 639 00:31:02,846 --> 00:31:07,411 A significant improvement a day versus, you know, a year, um, 640 00:31:08,441 --> 00:31:10,481 Or two, I think actually it might have been two. 641 00:31:10,726 --> 00:31:11,016 Yeah, 642 00:31:11,041 --> 00:31:11,611 Agreed. 643 00:31:12,001 --> 00:31:16,291 Um, and what I'm doing is I'm, because again, I don't have the catalog. 644 00:31:16,591 --> 00:31:18,841 What I'm currently running is I'm running a tar TVF. 645 00:31:19,816 --> 00:31:25,786 On all of those files and creating tarballs or creating, I'm sorry, text 646 00:31:25,991 --> 00:31:27,496 files, a list. 647 00:31:27,496 --> 00:31:29,026 of the, the files that are in there. 648 00:31:29,026 --> 00:31:31,936 And then I'm gonna do a count on the files that are in there and 649 00:31:31,936 --> 00:31:34,336 check it against the count of the files that are in the directory. 650 00:31:34,336 --> 00:31:36,616 And, and hopefully those numbers should be the same. 651 00:31:36,636 --> 00:31:40,436 Yeah, because I believe you are even saying that to run things 652 00:31:40,436 --> 00:31:44,036 like a find to get a list of all the files in a directory or a DU 653 00:31:44,956 --> 00:31:45,496 Yeah. 654 00:31:45,716 --> 00:31:46,796 hours, right? 655 00:31:46,916 --> 00:31:48,266 Well, it was days actually. 656 00:31:48,266 --> 00:31:48,596 In 657 00:31:48,656 --> 00:31:51,446 fact, it was why I didn't have this information in the beginning 658 00:31:51,446 --> 00:31:57,356 because everything was so big and every find, every du every command 659 00:31:57,356 --> 00:31:59,706 that I had DU is quicker than find. 660 00:31:59,726 --> 00:32:00,266 DU is. 661 00:32:00,386 --> 00:32:02,396 It just does less work than find. 662 00:32:02,906 --> 00:32:06,476 But the problem that I ultimately realized was that DU wasn't 663 00:32:06,476 --> 00:32:08,306 really being helpful in terms of. 664 00:32:08,846 --> 00:32:09,176 The 665 00:32:09,446 --> 00:32:12,866 scope of the job, what was the scope of the job was determined 666 00:32:12,866 --> 00:32:14,156 by the number of these files. 667 00:32:14,966 --> 00:32:19,676 And I couldn't get those numbers because that was the thing that took forever. 668 00:32:20,516 --> 00:32:25,616 the number of jobs dwindled down to about 20, that's when I 669 00:32:25,616 --> 00:32:27,386 was able to run these, uh, the 670 00:32:28,136 --> 00:32:29,966 and they would, they would actually complete. 671 00:32:30,416 --> 00:32:32,726 And that's when I realized just how bad it was. 672 00:32:33,431 --> 00:32:39,821 so if you had to start this over, and hopefully you never do, but I'm just 673 00:32:39,821 --> 00:32:43,691 saying, if you had to go back to day one, what would you do differently? 674 00:32:43,691 --> 00:32:46,931 I know you talked about making sure you understand the size of your backups. 675 00:32:47,801 --> 00:32:48,161 Right. 676 00:32:48,701 --> 00:32:52,421 It just feels like some of these, you just have to go through the process 677 00:32:52,811 --> 00:32:54,791 though because you don't know what to do. 678 00:32:54,791 --> 00:32:56,591 Like it's not like you could just start day one and be like, 679 00:32:56,621 --> 00:32:58,061 oh, I know I need to go to disc. 680 00:32:58,331 --> 00:32:59,711 I need to do X, Y, and Z. 681 00:33:00,101 --> 00:33:00,311 Right? 682 00:33:00,311 --> 00:33:02,921 It's sort of like a learning process. 683 00:33:03,101 --> 00:33:04,121 would say that I. 684 00:33:05,681 --> 00:33:08,741 Yeah, because the problem is you're going off into the unknown, 685 00:33:08,741 --> 00:33:11,201 you're doing a backup of something that you don't know what it is. 686 00:33:11,921 --> 00:33:17,591 And I, I would say if possible, if at all possible, get things like 687 00:33:18,071 --> 00:33:23,411 dus, uh, you know, discus it, it's a Unix command, but you can load those 688 00:33:23,411 --> 00:33:29,171 tools and windows as well get, like if you're going to back up, if you're 689 00:33:29,171 --> 00:33:30,671 gonna back up a hundred directories. 690 00:33:31,136 --> 00:33:34,436 Get a du of every one of those directories so that you have an idea 691 00:33:34,436 --> 00:33:35,696 of just what you're dealing with, 692 00:33:36,266 --> 00:33:37,706 if at all possible. 693 00:33:37,886 --> 00:33:41,036 Also, look and see if the number files and if the number of, and if you're 694 00:33:41,036 --> 00:33:45,506 trying to do a, you know, it's not that hard, you just run a fine dot dash, 695 00:33:46,046 --> 00:33:50,696 you know, I didn't even do a print just fine dot pipe to wc -l, right? 696 00:33:50,696 --> 00:33:51,386 That was it. 697 00:33:51,446 --> 00:33:51,836 Right? 698 00:33:51,866 --> 00:33:54,026 Um, to, to get the number of files. 699 00:33:55,646 --> 00:33:59,336 I'd say if again. 700 00:34:00,851 --> 00:34:04,631 If I could go back in time, I, I would say maybe do a little bit more of this 701 00:34:04,631 --> 00:34:07,661 research prior to beginning the job. 702 00:34:08,591 --> 00:34:12,071 Um, but that's diff it's, it's easy to say that now, 703 00:34:12,671 --> 00:34:13,901 um, because I know what 704 00:34:13,916 --> 00:34:14,501 I know. 705 00:34:14,861 --> 00:34:15,341 Right. 706 00:34:16,361 --> 00:34:23,081 Um, but the, you know, the core problem was that you've 707 00:34:23,081 --> 00:34:24,641 got these millions of files. 708 00:34:24,731 --> 00:34:25,541 I mean, which is all. 709 00:34:26,336 --> 00:34:29,696 Already gonna be a problem if you're backing it up in any sort of normal way. 710 00:34:29,936 --> 00:34:30,266 But if you're 711 00:34:30,716 --> 00:34:33,686 up remotely over the network, it's going to kill you. 712 00:34:34,221 --> 00:34:34,511 Yeah. 713 00:34:35,126 --> 00:34:39,086 So, um, you gotta figure out a way to do that. 714 00:34:39,086 --> 00:34:42,686 And then I would just say, see if there's anything that you can do with the, with 715 00:34:42,686 --> 00:34:44,756 the application that's created this data 716 00:34:44,756 --> 00:34:48,926 which is why it's important to get involved early on, right when an 717 00:34:48,926 --> 00:34:52,886 application is being developed or deployed, right, to get involved so 718 00:34:52,886 --> 00:34:54,926 they understand the backup requirements. 719 00:34:55,361 --> 00:34:55,871 yeah. 720 00:34:56,891 --> 00:35:01,246 And so, this backup that would never finish, I literally was, I 721 00:35:01,246 --> 00:35:03,391 was starting to think that this thing was never gonna finish. 722 00:35:04,171 --> 00:35:04,891 Um. 723 00:35:05,836 --> 00:35:08,446 It's essentially finally, I mean, it's not, at this point, it's 724 00:35:08,446 --> 00:35:11,446 not a hundred percent, but I'm, I'm now, you know, it's just, I'm 725 00:35:11,476 --> 00:35:12,316 at the finish line. 726 00:35:12,736 --> 00:35:13,126 Yeah. 727 00:35:13,246 --> 00:35:13,966 at the finish line. 728 00:35:14,656 --> 00:35:15,076 Yeah. 729 00:35:15,436 --> 00:35:16,456 Um, it's nice. 730 00:35:17,296 --> 00:35:19,786 I know one of the other things you mentioned that you were using 731 00:35:19,786 --> 00:35:24,766 NetBackup, but you had also looked at other tools out there as well, right? 732 00:35:24,766 --> 00:35:27,886 That could potentially help you with this effort. 733 00:35:27,886 --> 00:35:28,216 Right. 734 00:35:28,606 --> 00:35:33,706 So do you think that that becomes valuable, like either looking at other 735 00:35:33,736 --> 00:35:37,516 tools, um, I know you had reached out to like synology support, you 736 00:35:37,516 --> 00:35:39,466 had reached out to some experts, like 737 00:35:39,616 --> 00:35:40,306 Yeah. 738 00:35:40,846 --> 00:35:41,086 Yeah. 739 00:35:41,086 --> 00:35:48,566 The problem there, there were, there were, you could do, like with Synology, 740 00:35:48,586 --> 00:35:50,481 you can like copy the data from A to B. 741 00:35:50,971 --> 00:35:51,391 Mm-Hmm. 742 00:35:51,706 --> 00:35:54,556 They have this ability essentially like, you know, for lack of a 743 00:35:54,556 --> 00:35:55,966 better word, they have Snap Mirror. 744 00:35:56,866 --> 00:35:58,186 they have the equivalent of Snap Mirror. 745 00:35:58,641 --> 00:35:58,831 Yep. 746 00:35:59,536 --> 00:36:01,066 from onSynologygy box to another. 747 00:36:01,306 --> 00:36:06,316 But to me that wasn't really a backup like I wanted in a, in a format, you know, 748 00:36:06,436 --> 00:36:10,966 the end I was forced to not do what I wanted with the tar. 749 00:36:11,116 --> 00:36:15,286 Um, but I wanted it in a cataloged format. 750 00:36:16,186 --> 00:36:19,146 So we looked at a couple of, the problem was never NetBackup. 751 00:36:19,576 --> 00:36:19,796 Right? 752 00:36:19,796 --> 00:36:25,726 NetBackup made it, um, easy to script this whole thing because it was the 753 00:36:25,726 --> 00:36:27,076 only way I could make sense of it. 754 00:36:27,106 --> 00:36:31,996 'cause it was, it was thousands of directories and, um, and even 755 00:36:31,996 --> 00:36:34,966 more thousands of sub directories under those directories. 756 00:36:35,236 --> 00:36:37,906 And the only way I could make sense of this was to script it all. 757 00:36:38,446 --> 00:36:43,246 And, um, the, the fact that NetBackup allowed me to do that was great. 758 00:36:43,951 --> 00:36:48,751 Um, there are some other tools these days, some of the newer tools, 759 00:36:48,751 --> 00:36:50,461 they want to make it easy for you. 760 00:36:51,211 --> 00:36:54,691 But if you get into a complicated situation like this, some of the newer 761 00:36:54,691 --> 00:36:58,651 tools don't even have the ability to sort of grab it by the horns. 762 00:36:58,651 --> 00:36:58,741 The 763 00:36:59,101 --> 00:37:00,281 able to do a NetBackup, 764 00:37:00,721 --> 00:37:00,931 Yeah. 765 00:37:01,171 --> 00:37:06,331 I think the other thing also that you were doing, which I thought was interesting, 766 00:37:06,331 --> 00:37:08,341 was also your scripting, right? 767 00:37:08,341 --> 00:37:12,571 Trying to automate this, like, uh, I know like scheduling your, 768 00:37:12,691 --> 00:37:14,461 the backup policies to run, right? 769 00:37:14,461 --> 00:37:16,621 And then you were sort of doing load balancing to make sure 770 00:37:16,621 --> 00:37:18,121 that you keep the two filers 771 00:37:18,361 --> 00:37:18,991 Yeah. 772 00:37:19,531 --> 00:37:19,831 Yeah. 773 00:37:19,831 --> 00:37:21,241 I couldn't, yeah, that was the thing. 774 00:37:21,241 --> 00:37:24,631 I couldn't normally, I, I just, I believe in just throwing 775 00:37:24,631 --> 00:37:27,091 everything in the NetBackup schedule or, and let it figure it out. 776 00:37:27,631 --> 00:37:32,851 But because again, because of the limitations of the weird thing I had, 777 00:37:32,851 --> 00:37:38,251 I, I couldn't figure out a way to load balance across the two target filers. 778 00:37:39,331 --> 00:37:40,651 the NetBackup scheduler. 779 00:37:41,251 --> 00:37:45,151 Um, maybe I could have, uh, done that better. 780 00:37:45,151 --> 00:37:45,721 I don't know. 781 00:37:45,721 --> 00:37:50,041 But, uh, so the way I was doing it was I was just assigning a backup. 782 00:37:50,701 --> 00:37:55,951 a backup would finish, I would assign the next backup to that, that the 783 00:37:56,551 --> 00:37:59,461 was now had more space available to it. 784 00:37:59,491 --> 00:37:59,851 Right. 785 00:38:00,211 --> 00:38:02,071 So I just had a while loop that was running, you 786 00:38:02,101 --> 00:38:03,961 know, checking to see if a backup job was done. 787 00:38:04,261 --> 00:38:05,491 but I think that's important, right? 788 00:38:05,851 --> 00:38:08,731 You can always script some of these things that if it doesn't 789 00:38:08,731 --> 00:38:10,831 exist in the native tools, right? 790 00:38:10,861 --> 00:38:11,761 Don't be afraid. 791 00:38:12,466 --> 00:38:12,796 Yeah. 792 00:38:12,796 --> 00:38:13,666 Don't be afraid. 793 00:38:14,236 --> 00:38:16,306 you know, obviously I'm, I'm pretty good at scripting and 794 00:38:16,306 --> 00:38:17,836 I'm pretty good in the backup. 795 00:38:17,986 --> 00:38:21,586 And, um, th there are, and, and, and, and thanks. 796 00:38:21,646 --> 00:38:26,776 Thanks very much to Veritas for keeping their, uh, their documentation online. 797 00:38:27,016 --> 00:38:28,546 Uh, the number of times I Googled. 798 00:38:29,476 --> 00:38:34,396 You know, backup job, you know, how do, how do I list, uh, you know, and 799 00:38:34,396 --> 00:38:37,486 I know there's a, there's, I know there's a command to, to do this. 800 00:38:37,696 --> 00:38:38,476 How do I do that? 801 00:38:38,476 --> 00:38:40,636 And, you know, and then a man page would come up and I would read it 802 00:38:40,636 --> 00:38:42,016 and I was like, oh, yeah, yeah, yeah. 803 00:38:42,016 --> 00:38:42,226 It's 804 00:38:42,236 --> 00:38:42,736 been a while. 805 00:38:43,486 --> 00:38:43,906 Yeah. 806 00:38:44,146 --> 00:38:44,836 Um. 807 00:38:44,851 --> 00:38:47,671 you have to also thank Cygwin, of course. 808 00:38:47,791 --> 00:38:52,061 Yes, special thanks to to Cygwin Without Cygwin. 809 00:38:52,411 --> 00:38:55,471 That is the tool that you can download and run on any Windows 810 00:38:55,471 --> 00:38:58,171 server to give you Unix capabilities. 811 00:38:58,381 --> 00:39:03,391 I will say there were, there were moments where Cygwin was both helpful and 812 00:39:03,541 --> 00:39:09,121 terrorizing me because it was the whole like backslash versus forward slash thing. 813 00:39:09,616 --> 00:39:14,206 Because in Windows, you know, the file separator is a backslash, which 814 00:39:14,206 --> 00:39:16,486 in Unix is an escape character, 815 00:39:17,101 --> 00:39:17,461 Yep. 816 00:39:17,836 --> 00:39:22,156 and Cygwin wasn't consistent. 817 00:39:23,566 --> 00:39:26,206 When that escape character would be an escape character. 818 00:39:26,206 --> 00:39:28,966 Like, like if you piped it into a file, it would do one thing. 819 00:39:28,966 --> 00:39:31,786 If you piped it into a command, it would do it, it would behave differently. 820 00:39:32,686 --> 00:39:36,136 And, um, so that, that definitely l lent. 821 00:39:36,511 --> 00:39:40,531 The fact that I was doing constant file manipulation on directories 822 00:39:40,531 --> 00:39:41,881 that were seven levels deep, 823 00:39:42,691 --> 00:39:42,961 Yeah. 824 00:39:43,381 --> 00:39:44,101 did not help. 825 00:39:45,661 --> 00:39:45,931 Yeah. 826 00:39:45,991 --> 00:39:51,601 Oh, and then I couldn't, the, the, the, the one thing with 827 00:39:51,601 --> 00:39:53,026 Cygwin is that it doesn't see. 828 00:39:54,661 --> 00:40:00,571 It doesn't see the, to point the backups to NetBackup, I have to point 829 00:40:00,571 --> 00:40:03,751 'em in the backs back slash filer name 830 00:40:04,501 --> 00:40:05,251 share name. 831 00:40:06,161 --> 00:40:07,511 Cygwin doesn't see that. 832 00:40:08,071 --> 00:40:12,241 Cygwin sees only mapped drive names 833 00:40:12,841 --> 00:40:13,021 and 834 00:40:13,096 --> 00:40:14,026 have to map it using 835 00:40:14,161 --> 00:40:15,961 you have to map it to a drive name. 836 00:40:15,961 --> 00:40:16,981 Let's say you map it to, 837 00:40:17,071 --> 00:40:22,681 to letter F, and then in Cygwin you would see /cygdrive/f. 838 00:40:23,416 --> 00:40:27,256 Which would be the same as this backs slash backs mount. 839 00:40:27,676 --> 00:40:30,646 know, I was constantly having to go back and forth between 840 00:40:30,646 --> 00:40:32,866 those two and, and that was fun. 841 00:40:33,706 --> 00:40:34,426 Um, 842 00:40:34,816 --> 00:40:35,386 scripting 843 00:40:35,431 --> 00:40:36,031 here's the thing. 844 00:40:36,031 --> 00:40:40,651 After all of this experience and everything you've learned, you're probably 845 00:40:40,651 --> 00:40:41,731 never gonna use any of this again. 846 00:40:42,241 --> 00:40:43,486 I don't know about that. 847 00:40:43,501 --> 00:40:44,881 I dunno about that. 848 00:40:44,881 --> 00:40:47,581 I tell you what, I'm, I'm taking a tar, all those scripts that 849 00:40:47,581 --> 00:40:53,281 I wrote, um, because I will say this, that, that the NetBackup 850 00:40:53,281 --> 00:41:00,091 documentation while, uh, extensive, it doesn't give a lot of examples. 851 00:41:00,901 --> 00:41:06,931 And so like, I'm thinking of like, um, like the BP duplicate command, 852 00:41:06,961 --> 00:41:10,081 which is the command to copy backups from one place to another. 853 00:41:11,581 --> 00:41:16,021 I couldn't, I couldn't figure out from reading the man page how to 854 00:41:16,021 --> 00:41:19,711 actually do, to do what I needed to do. 855 00:41:19,891 --> 00:41:21,601 So I would, I would like. 856 00:41:22,591 --> 00:41:25,051 I would do, I would have to run tests, you 857 00:41:25,141 --> 00:41:31,831 know, I'd, you know, um, and, um, the, you know, not like now that Cohesity's 858 00:41:31,831 --> 00:41:34,351 acquiring them, it's not like they're now gonna rewrite their man pages. 859 00:41:34,591 --> 00:41:38,131 I just thought that they could have used some more, some more examples. 860 00:41:38,131 --> 00:41:38,371 But 861 00:41:38,786 --> 00:41:39,076 Yeah. 862 00:41:39,721 --> 00:41:40,921 I figured it out eventually. 863 00:41:42,091 --> 00:41:46,801 You know, I think someone used to have a forum that people would post on about. 864 00:41:46,906 --> 00:41:51,256 Yeah, someone used to have that and then, but people stopped posting 865 00:41:51,256 --> 00:41:52,396 on that forum, so I don't know 866 00:41:52,876 --> 00:41:53,326 You know? 867 00:41:53,476 --> 00:41:56,146 Um, where people are getting their help now, 868 00:41:56,476 --> 00:41:57,196 but, uh, 869 00:41:58,396 --> 00:42:00,706 Well, I'm glad that this is almost over, 870 00:42:00,796 --> 00:42:01,276 yeah. 871 00:42:01,336 --> 00:42:01,486 Yeah. 872 00:42:02,296 --> 00:42:05,056 nearly over and I'm glad you're still alive, 873 00:42:05,491 --> 00:42:06,481 I am alive. 874 00:42:06,511 --> 00:42:08,461 I didn't kill anyone along the way. 875 00:42:08,461 --> 00:42:09,661 I didn't scream at anyone. 876 00:42:09,661 --> 00:42:10,891 Like the, the story that 877 00:42:11,221 --> 00:42:14,011 you have heard were, were Curtis Cuss Preston. 878 00:42:14,731 --> 00:42:15,841 I didn't scream at anyone. 879 00:42:15,991 --> 00:42:16,321 yeah. 880 00:42:16,606 --> 00:42:21,946 but I really, really, really think you should do an office space on those filers. 881 00:42:23,491 --> 00:42:23,671 yeah. 882 00:42:23,671 --> 00:42:25,681 Well, that would sort of defeat the purpo of the 883 00:42:26,521 --> 00:42:29,521 but, uh, I, yeah, I, like that idea. 884 00:42:30,376 --> 00:42:30,586 Hmm. 885 00:42:31,441 --> 00:42:31,861 Anyway. 886 00:42:31,921 --> 00:42:37,111 Well, uh, thanks Prasanna for helping me, uh, sort of through this. 887 00:42:37,111 --> 00:42:39,211 You were my constant counselor through this. 888 00:42:40,021 --> 00:42:42,166 I think I learned a bunch. 889 00:42:42,166 --> 00:42:45,766 I know usually I'm all about YouTube knowledge, but in this case it was 890 00:42:46,066 --> 00:42:48,286 the Preston knowledge, so it was good. 891 00:42:48,286 --> 00:42:48,316 I. 892 00:42:48,541 --> 00:42:48,901 Yeah. 893 00:42:48,931 --> 00:42:49,351 Yeah. 894 00:42:50,311 --> 00:42:56,251 uh, thanks everybody else for, uh, uh, listening along with this sad, sad story 895 00:42:56,251 --> 00:42:57,781 with I think a decent, happy ending. 896 00:42:58,261 --> 00:42:59,371 That is a wrap. 897 00:43:02,281 --> 00:43:06,991 The backup wrap up is written, recorded and produced by me w Curtis Preston. 898 00:43:07,531 --> 00:43:09,001 If you need backup or Dr. 899 00:43:09,001 --> 00:43:12,631 Consulting content generation or expert witness work, 900 00:43:12,931 --> 00:43:14,911 check out backup central.com. 901 00:43:15,631 --> 00:43:18,751 You can also find links from my O'Reilly Books on the same website. 902 00:43:19,411 --> 00:43:23,521 Remember, this is an independent podcast and any opinions that you 903 00:43:23,521 --> 00:43:25,411 hear are those of the speaker. 904 00:43:25,476 --> 00:43:27,336 And not necessarily an employer. 905 00:43:28,236 --> 00:43:28,956 Thanks for listening.