1 00:00:00,000 --> 00:00:03,859 You found the backup wrapup, your go-to podcast for all things 2 00:00:03,879 --> 00:00:06,539 backup recovery and cyber recovery. 3 00:00:07,379 --> 00:00:11,639 You know what, no one cares about your backup speed. 4 00:00:12,179 --> 00:00:12,759 That's right. 5 00:00:12,960 --> 00:00:13,479 I said it. 6 00:00:14,109 --> 00:00:14,400 Sure. 7 00:00:14,400 --> 00:00:18,989 When designing your backup system, backup speed does matter, but 8 00:00:18,989 --> 00:00:20,269 you know what really counts? 9 00:00:21,339 --> 00:00:21,859 Recovery speed. 10 00:00:22,879 --> 00:00:24,299 That's what keeps you employed. 11 00:00:25,169 --> 00:00:29,219 I've got some wild stories about how I learned this particular lesson 12 00:00:29,719 --> 00:00:34,339 the hard way, including that time at Motorola when I discovered that my tape 13 00:00:34,339 --> 00:00:37,019 drives could write but couldn't read. 14 00:00:37,529 --> 00:00:38,219 That was fun. 15 00:00:39,039 --> 00:00:43,739 Uh, joined persona and I as we dig into why recovery testing matters 16 00:00:43,889 --> 00:00:47,739 more than bragging about your backup speeds and why you should 17 00:00:48,099 --> 00:00:49,379 probably start doing some of it. 18 00:00:50,104 --> 00:00:53,844 Before you actually need to trust me on this one. 19 00:00:54,104 --> 00:00:57,324 You don't wanna learn this lesson the way I did. 20 00:00:58,344 --> 00:01:04,044 By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr. Backup, 21 00:01:04,504 --> 00:01:10,324 and I've been passionate about this topic for over 30 years, ever since. 22 00:01:10,404 --> 00:01:13,964 I had to tell my boss that we had no backups of the production 23 00:01:14,244 --> 00:01:15,324 database that we had just lost. 24 00:01:16,394 --> 00:01:19,674 I don't want that to happen to you, and that's why I do this. 25 00:01:20,535 --> 00:01:25,555 On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes. 26 00:01:26,185 --> 00:01:28,195 This is the backup wrap up. 27 00:01:43,339 --> 00:01:44,479 Welcome to the show. 28 00:01:45,679 --> 00:01:49,579 Hi, I am w Curtis Preston, AKA, Mr. Backup, and I have with me 29 00:01:49,579 --> 00:01:56,089 a guy who wishes he had my new coffee machine Prasann Malaiyandi. 30 00:01:56,149 --> 00:01:56,629 How's it going? 31 00:01:56,689 --> 00:01:57,129 Prasanna? 32 00:01:57,469 --> 00:01:58,399 I am GI Act. 33 00:01:58,399 --> 00:01:59,389 I'm good Curtis. 34 00:01:59,629 --> 00:02:03,139 So I actually don't wish for your coffee machine 35 00:02:03,344 --> 00:02:03,764 Mm-hmm. 36 00:02:03,889 --> 00:02:04,459 reasons. 37 00:02:05,024 --> 00:02:05,314 Okay. 38 00:02:05,779 --> 00:02:08,809 One, I don't have space to put a coffee maker. 39 00:02:09,469 --> 00:02:09,919 Uh, 40 00:02:10,129 --> 00:02:12,049 Without tossing something off the kitchen counter. 41 00:02:12,619 --> 00:02:13,099 mm-hmm. 42 00:02:13,249 --> 00:02:15,379 I really need to cut back on the caffeine. 43 00:02:15,379 --> 00:02:17,419 Like at work we get coffee 44 00:02:18,109 --> 00:02:18,399 Yeah. 45 00:02:18,679 --> 00:02:21,049 I try not to have more than one a day. 46 00:02:21,079 --> 00:02:27,889 And on Wednesdays we have someone come who makes coffee and uh, I get one 47 00:02:28,129 --> 00:02:33,199 single shot latte to start with, and before they're done for the day, I get. 48 00:02:33,589 --> 00:02:39,499 An iced oat milk with like a half a shot and they laugh at me. 49 00:02:39,499 --> 00:02:41,089 They're like, how much is a half a shot this 50 00:02:41,194 --> 00:02:42,994 What is a, what a, that is so funny. 51 00:02:43,279 --> 00:02:43,639 Because I 52 00:02:43,654 --> 00:02:44,074 Yeah. 53 00:02:44,689 --> 00:02:45,079 so 54 00:02:45,154 --> 00:02:45,454 No. 55 00:02:45,454 --> 00:02:47,584 I st I still think you wish you could have it. 56 00:02:47,584 --> 00:02:51,754 You just, you, you, um, you like the taste, but you don't Yeah. 57 00:02:51,754 --> 00:02:52,774 You know, it'd be bad for you. 58 00:02:52,774 --> 00:02:53,104 Yeah. 59 00:02:53,179 --> 00:02:54,979 are you enjoying your new coffee? 60 00:02:54,979 --> 00:02:55,879 Fancy coffee machine? 61 00:02:56,944 --> 00:03:00,754 So I've, I've made like three espressos with it so far. 62 00:03:00,904 --> 00:03:02,704 Uh, I have a ne espresso, by the way. 63 00:03:02,704 --> 00:03:05,124 I've, I've kind of always thought about getting one and I, you 64 00:03:05,124 --> 00:03:06,324 know, and I finally bought it. 65 00:03:06,384 --> 00:03:10,104 Um, but with, you know how, like, you know how like your. 66 00:03:10,569 --> 00:03:13,509 Your feed is tailored to stuff that you do. 67 00:03:13,719 --> 00:03:14,019 Yeah. 68 00:03:14,169 --> 00:03:18,879 This morning the first thing that greeted me when I woke up was is Nespresso 69 00:03:18,879 --> 00:03:22,329 is changing their prices, you know, increasing the cause of their pods. 70 00:03:22,779 --> 00:03:25,119 Um, and also, 71 00:03:25,389 --> 00:03:25,689 in it. 72 00:03:26,319 --> 00:03:31,179 yeah, and also there was a thing on Reddit about, um. 73 00:03:31,839 --> 00:03:35,919 It was a picture of what A-A-U-P-S place looks like from the 74 00:03:35,919 --> 00:03:38,409 Nespresso pod recycling program. 75 00:03:38,679 --> 00:03:39,939 Let's just say it's a hot mess. 76 00:03:39,999 --> 00:03:40,299 yeah, 77 00:03:40,329 --> 00:03:46,419 Um, they are doing it, it's just that it, it's logistically a, a challenge. 78 00:03:46,569 --> 00:03:47,139 Um, 79 00:03:47,199 --> 00:03:48,369 and it's also like, I think. 80 00:03:49,074 --> 00:03:51,174 Sometimes they're like, oh, separate out your bottles and 81 00:03:51,174 --> 00:03:52,794 plastic, or your plastics and trash. 82 00:03:53,094 --> 00:03:53,634 And it's 83 00:03:53,679 --> 00:03:54,159 yeah. 84 00:03:54,294 --> 00:03:57,294 to actually recycle properly is so high. 85 00:03:57,729 --> 00:03:58,179 Yeah. 86 00:03:58,254 --> 00:04:00,834 if they're just like, yep, we'll take it and we're just gonna burn it all. 87 00:04:02,139 --> 00:04:03,789 I, I think they're doing it. 88 00:04:03,849 --> 00:04:09,969 But, uh, there is this thing about like, um, that there is the belief that 89 00:04:10,059 --> 00:04:13,359 they, they give you these bags, they give you a free bag that if you just 90 00:04:13,419 --> 00:04:16,839 put your stuff in there, you can drop it off at UPS and they'll just take it. 91 00:04:16,899 --> 00:04:17,229 Right? 92 00:04:18,819 --> 00:04:21,429 And they're saying you're supposed to fill it up to this line. 93 00:04:22,029 --> 00:04:23,769 Um, but not beyond the line. 94 00:04:24,099 --> 00:04:28,689 And the belief is that if you don't fill it up to that line, the cost to ship 95 00:04:28,689 --> 00:04:32,499 that bag is less than the, than what they're able to recycle out of the bag. 96 00:04:33,219 --> 00:04:35,379 Um, but anyway, but 97 00:04:35,529 --> 00:04:36,819 up to at least the line. 98 00:04:37,269 --> 00:04:37,869 yeah. 99 00:04:37,989 --> 00:04:41,979 Well, but if you go too far over the line, like it, there's issues, but, 100 00:04:42,429 --> 00:04:46,149 um, but I, there was an incident. 101 00:04:47,159 --> 00:04:49,944 I, I, I want to hear about, let me guess. 102 00:04:50,724 --> 00:04:52,554 let me guess what the incident is. 103 00:04:53,004 --> 00:04:53,694 Yeah, 104 00:04:53,934 --> 00:04:57,294 a pod in when a pod was already in there and everything exploded. 105 00:04:58,014 --> 00:05:01,644 no, that it, it takes care of all that, you know, all you have to do is open it 106 00:05:01,644 --> 00:05:03,474 and it automatically removes the old pod. 107 00:05:04,014 --> 00:05:08,964 Um, but the, the way that, the way the one that I bought, it comes with a tower. 108 00:05:09,924 --> 00:05:12,114 Tower O water, right? 109 00:05:12,474 --> 00:05:14,214 So, and it's detachable. 110 00:05:14,454 --> 00:05:18,474 So like, by the way, the, the, the kitchen counter problem, um, I solved 111 00:05:18,474 --> 00:05:20,124 that by having this in my office. 112 00:05:20,184 --> 00:05:23,874 So this is the tower of water for those of you watching it on YouTube. 113 00:05:24,264 --> 00:05:26,814 And it literally just, you know, I'm holding it in my hand and then 114 00:05:26,814 --> 00:05:29,994 I can put it over here and, you know, there's this little slot, 115 00:05:29,994 --> 00:05:30,294 Yeah. 116 00:05:30,474 --> 00:05:31,134 you see that? 117 00:05:31,344 --> 00:05:35,454 And so that, that tells where to go, but literally you just pop it on there. 118 00:05:36,084 --> 00:05:39,234 Um, so I might. 119 00:05:39,939 --> 00:05:42,009 Have just knocked it right over like, 120 00:05:42,984 --> 00:05:43,194 And 121 00:05:43,299 --> 00:05:44,049 just after. 122 00:05:44,184 --> 00:05:45,324 like spilled everywhere. 123 00:05:45,699 --> 00:05:45,999 Yeah. 124 00:05:45,999 --> 00:05:48,309 And that's a good what, 32 125 00:05:48,379 --> 00:05:48,669 Yeah. 126 00:05:48,849 --> 00:05:51,789 ounces at least, maybe even more. 127 00:05:51,789 --> 00:05:54,999 And I just dumped the entire thing on the floor. 128 00:05:55,539 --> 00:05:56,919 Um, literally. 129 00:05:57,279 --> 00:06:01,449 Five minutes after putting it in, because I have it, I, I, I have a 130 00:06:01,449 --> 00:06:06,399 better place for it long term, but where it's at right now is a bit precarious. 131 00:06:06,429 --> 00:06:11,199 And, um, I literally, five minutes after having it here, I, I dumped 132 00:06:11,199 --> 00:06:14,169 that entire thing on the floor and had to mop up like crazy. 133 00:06:14,169 --> 00:06:15,669 And, I mean, there's power cords down 134 00:06:15,834 --> 00:06:20,544 I, I was actually worried about your power strips and power cores and 135 00:06:20,649 --> 00:06:21,069 Yeah. 136 00:06:21,519 --> 00:06:21,879 Yeah. 137 00:06:22,794 --> 00:06:25,134 Yeah, there, there's some bricks down there, some, you know, some 138 00:06:25,134 --> 00:06:26,694 power bricks and things like that. 139 00:06:26,814 --> 00:06:27,114 Those are 140 00:06:27,354 --> 00:06:32,214 Yeah, so it wasn't, it was, it was just a big water watery mess. 141 00:06:32,214 --> 00:06:32,934 But anyway. 142 00:06:34,074 --> 00:06:37,314 So, um, let's chat a bit. 143 00:06:37,314 --> 00:06:41,424 I, I, I have, do you think, do you think the title is clickbait? 144 00:06:44,274 --> 00:06:47,274 That's the first thing I told you before we started this recording. 145 00:06:47,274 --> 00:06:50,304 I was like, ah, it's a little bit click bait, isn't it? 146 00:06:50,874 --> 00:06:51,384 Yeah. 147 00:06:51,414 --> 00:06:52,869 No one cares about your backup Speed. 148 00:06:52,869 --> 00:06:52,884 Right. 149 00:06:52,884 --> 00:06:55,345 Um, and, and let's just talk about. 150 00:06:55,554 --> 00:06:57,954 similar to what we've talked about before, which is 151 00:06:58,059 --> 00:06:58,599 Yeah. 152 00:06:58,734 --> 00:06:59,274 your backups. 153 00:06:59,274 --> 00:06:59,334 I. 154 00:06:59,769 --> 00:07:00,069 Yeah. 155 00:07:00,069 --> 00:07:01,329 No one cares about your backup. 156 00:07:01,579 --> 00:07:03,949 You know, you, you, one of the things that you've heard me say on a, on 157 00:07:03,949 --> 00:07:08,449 a repeated basis is that no one wants to be the backup admin, right? 158 00:07:08,449 --> 00:07:14,604 No one grows up to be the backup admin, um, and no one is in it going. 159 00:07:15,329 --> 00:07:17,939 I, you know, I hate my job as a network admin. 160 00:07:17,939 --> 00:07:20,129 I'd really much rather be the backup person, right? 161 00:07:20,129 --> 00:07:22,529 No one, no one has ever said that in the history of time. 162 00:07:23,219 --> 00:07:26,909 Um, many people have said, God, I hate my job in backup and I can't 163 00:07:26,909 --> 00:07:29,249 wait to become a real cis admin. 164 00:07:29,789 --> 00:07:34,799 That's, and I mean, I mean, I mean that I love all of you, my backup people. 165 00:07:34,859 --> 00:07:35,579 You are me. 166 00:07:35,579 --> 00:07:36,389 I am you. 167 00:07:36,749 --> 00:07:41,939 Uh, I'm just saying that it is a tough, tough job and one of the reasons. 168 00:07:42,409 --> 00:07:43,789 Is it No one cares about your backup. 169 00:07:44,319 --> 00:07:44,609 Yeah. 170 00:07:44,899 --> 00:07:46,699 They only care about the restore, right? 171 00:07:46,699 --> 00:07:49,279 No one cares about the millions of backups that you got, right? 172 00:07:49,609 --> 00:07:53,989 They only care about the one restore that you got wrong, and, uh, they, yeah. 173 00:07:53,994 --> 00:07:54,474 What's that? 174 00:07:55,504 --> 00:08:00,454 so going along what you're saying though, think people don't grow up 175 00:08:00,454 --> 00:08:05,079 to be backup admins, but I think there's a way to make backup sexy. 176 00:08:05,389 --> 00:08:07,119 Like I'm just imagining in my head 177 00:08:07,664 --> 00:08:08,084 Mm-hmm. 178 00:08:08,794 --> 00:08:10,744 like it's Superman, right? 179 00:08:10,744 --> 00:08:12,454 It's Clark Kent is backup. 180 00:08:12,994 --> 00:08:13,384 Uh huh 181 00:08:13,939 --> 00:08:16,849 And Superman is restore saving the day, 182 00:08:19,354 --> 00:08:20,824 Here I come to save the day. 183 00:08:20,929 --> 00:08:21,289 right? 184 00:08:21,289 --> 00:08:22,879 If you think about it, you are the hero. 185 00:08:22,879 --> 00:08:25,879 You are the one who saves the company and brings things back up. 186 00:08:27,424 --> 00:08:35,524 So, so for the YouTube audience, um, I am backup guy, right? 187 00:08:37,159 --> 00:08:37,549 Yes, Mr. Backup. 188 00:08:38,509 --> 00:08:39,079 Restore man, 189 00:08:43,279 --> 00:08:44,089 restore man, 190 00:08:45,664 --> 00:08:45,934 And, 191 00:08:46,279 --> 00:08:46,489 guy. 192 00:08:46,504 --> 00:08:50,044 and for our listeners who are listening to this via podcast, 193 00:08:50,344 --> 00:08:54,394 Curtis, between those two sentences, just took off his glasses then put 194 00:08:54,439 --> 00:08:55,669 And I'm like a totally different guy. 195 00:08:55,669 --> 00:08:57,499 I mean, it worked for Superman, right? 196 00:08:58,639 --> 00:08:59,509 That's that. 197 00:08:59,509 --> 00:09:01,819 That is just the funniest I've, I've always, I've always 198 00:09:01,819 --> 00:09:02,929 been am abused by that, but 199 00:09:02,974 --> 00:09:03,874 but here's what I'm thinking. 200 00:09:03,874 --> 00:09:04,084 Sorry. 201 00:09:04,084 --> 00:09:04,744 I know this is 202 00:09:05,239 --> 00:09:05,689 yeah. 203 00:09:05,794 --> 00:09:11,254 a bit, but I think we backup people need to start thinking differently, right? 204 00:09:11,254 --> 00:09:13,414 I think if you think I'm a backup person, 205 00:09:13,659 --> 00:09:14,079 Mm-hmm. 206 00:09:14,254 --> 00:09:14,494 right? 207 00:09:14,494 --> 00:09:18,004 I don't think you're giving yourself enough credit for how important 208 00:09:18,004 --> 00:09:19,264 you are in the organization. 209 00:09:20,014 --> 00:09:20,464 Backup 210 00:09:20,494 --> 00:09:21,064 Absolutely. 211 00:09:21,124 --> 00:09:21,754 an end. 212 00:09:22,054 --> 00:09:23,914 You are really the recovery person. 213 00:09:25,099 --> 00:09:25,429 Yeah. 214 00:09:25,429 --> 00:09:27,709 And that's why, you know, the tagline of the show. 215 00:09:27,709 --> 00:09:31,849 We say that we turn, uh, we turn unappreciated backup admins into 216 00:09:31,849 --> 00:09:33,559 cyber recovery heroes, right? 217 00:09:33,739 --> 00:09:34,429 Absolutely. 218 00:09:34,489 --> 00:09:37,429 I am on board for you or I am on board with you. 219 00:09:38,419 --> 00:09:42,649 Having said that, let's go back to the title of the episode, which is 220 00:09:42,829 --> 00:09:44,719 no one cares about your Backup Speed. 221 00:09:45,049 --> 00:09:45,469 Okay? 222 00:09:46,129 --> 00:09:48,709 And, and, and, and I say that be. 223 00:09:48,754 --> 00:09:49,834 backup speed all the 224 00:09:49,924 --> 00:09:51,634 Yeah, when do we talk about it? 225 00:09:51,634 --> 00:09:54,064 We talk about it when we're designing the backup system, when 226 00:09:54,064 --> 00:09:55,384 we're working with the vendor 227 00:09:55,459 --> 00:09:56,059 Speeds and 228 00:09:56,134 --> 00:09:59,224 design the backup system and yeah, speeds of feeds. 229 00:09:59,224 --> 00:10:06,364 And I'd say this is it, it, we don't focus on it as much today 230 00:10:06,364 --> 00:10:11,014 as we did back in the day, because back in the day you had a number. 231 00:10:11,014 --> 00:10:14,944 You're like, I have, I have 18 terabytes and I gotta get 18 terabytes 232 00:10:14,944 --> 00:10:17,734 from here to there full once. 233 00:10:18,154 --> 00:10:18,844 A week. 234 00:10:19,204 --> 00:10:24,244 Um, and, um, and, and therefore there's some math that has to happen. 235 00:10:24,244 --> 00:10:25,354 I gotta transfer. 236 00:10:25,534 --> 00:10:28,564 I'm gonna do, I'm gonna, I'm gonna spread my, I got 18 terabytes. 237 00:10:28,564 --> 00:10:29,704 I'm gonna pick a better number. 238 00:10:29,704 --> 00:10:30,784 21 terabytes. 239 00:10:31,174 --> 00:10:31,714 So I'm gonna do three Tera. 240 00:10:32,929 --> 00:10:33,889 Tag I, yeah. 241 00:10:33,889 --> 00:10:34,159 Right. 242 00:10:34,189 --> 00:10:34,429 Okay. 243 00:10:34,429 --> 00:10:39,289 So I'm gonna do three terabytes a night, and that means I need a network, I 244 00:10:39,289 --> 00:10:44,269 need, I need server, and I need, uh, you know, tape or D drives that are capable 245 00:10:44,269 --> 00:10:50,119 of receiving three terabytes a night, plus, you know, let's say 10% of the 246 00:10:50,119 --> 00:10:53,689 remaining terabytes, which would be, um. 247 00:10:54,199 --> 00:10:56,029 Another roughly two terabytes. 248 00:10:56,209 --> 00:11:00,109 So I need, I need a system that's capable of receiving five terabytes a night. 249 00:11:00,709 --> 00:11:06,049 Backup speed mattered so very much when that was the way you did backup. 250 00:11:06,739 --> 00:11:14,479 Um, now I think mainly the concern is that obviously we need enough time 251 00:11:14,479 --> 00:11:20,089 in order to get the backup done, but, um, and I will say this, so I'll, 252 00:11:20,089 --> 00:11:22,189 I'll, I'll, I'll disagree with the. 253 00:11:22,894 --> 00:11:27,424 With the title first before we get into the, the reason why I actually strongly 254 00:11:27,424 --> 00:11:36,274 believe in this statement, um, if you, if your backup is not finishing within 255 00:11:36,274 --> 00:11:39,844 a reasonable amount of time within what we refer to as the backup window, what, 256 00:11:39,904 --> 00:11:41,584 what, what do we call the backup window? 257 00:11:41,869 --> 00:11:46,129 Your backup window is the time in which you have to finish a backup. 258 00:11:46,129 --> 00:11:46,669 Typically, 259 00:11:46,684 --> 00:11:47,074 Right. 260 00:11:47,119 --> 00:11:52,399 resources are idle, or as an example, you might say, I have eight hours 261 00:11:52,399 --> 00:11:55,819 during normal business days, Monday through Friday to get a backup done. 262 00:11:56,089 --> 00:11:59,899 Maybe I have all weekend because people aren't actively using the system. 263 00:11:59,899 --> 00:12:00,379 It's 264 00:12:00,454 --> 00:12:00,814 Yep. 265 00:12:01,414 --> 00:12:02,374 Yeah, exactly. 266 00:12:02,644 --> 00:12:07,174 And historically, the reason why the backup window, and this is why, another 267 00:12:07,174 --> 00:12:11,554 reason why we focused so much on backup speed back in the day, which was I. 268 00:12:12,199 --> 00:12:15,559 Backup was really hard on the servers. 269 00:12:15,559 --> 00:12:19,099 It was a, there was a significant performance impact on the servers 270 00:12:19,609 --> 00:12:23,359 when you were running a backup, and so therefore, really the only time we 271 00:12:23,359 --> 00:12:27,439 you could do it was when no one else was using the servers, and so you 272 00:12:27,439 --> 00:12:29,479 were given a window back in the day. 273 00:12:29,479 --> 00:12:31,816 I remember that my window started at like 10 o'clock at night. 274 00:12:32,584 --> 00:12:35,914 And I needed to be done by six in the morning, which meant I had eight hours. 275 00:12:35,914 --> 00:12:39,664 That was my backup window, and that meant that I had to do those. 276 00:12:39,904 --> 00:12:42,124 Um, we came up with five terabytes. 277 00:12:42,124 --> 00:12:44,254 I had to do those five terabytes within eight hours. 278 00:12:44,464 --> 00:12:45,754 Which, what? 279 00:12:46,924 --> 00:12:47,974 Well, three plus the two. 280 00:12:48,274 --> 00:12:53,614 Remember the three, the three terabyte of full plus two terabytes from the 10% 281 00:12:53,614 --> 00:12:55,024 of the, don't you remember that part? 282 00:12:55,024 --> 00:12:55,294 Yeah. 283 00:12:55,294 --> 00:12:55,564 Yeah. 284 00:12:56,629 --> 00:13:00,409 So I need to do those five terabytes within eight hours, which need, 285 00:13:00,439 --> 00:13:04,729 which means I need to do almost a terabyte an hour, which back in the 286 00:13:04,729 --> 00:13:07,099 day was a real challenge, right? 287 00:13:07,149 --> 00:13:11,319 I remember the first time I saw a, a headline that, that, that 288 00:13:11,319 --> 00:13:16,119 somebody had, was able to achieve a terabyte, an hour of backup back then. 289 00:13:16,389 --> 00:13:19,089 Like, if you couldn't do that, you, you, you, well, you just couldn't 290 00:13:19,089 --> 00:13:23,109 get your backup done within the window and therefore, um, you 291 00:13:23,114 --> 00:13:24,084 weren't gonna get your backup done. 292 00:13:24,264 --> 00:13:26,964 but, but I think. 293 00:13:27,264 --> 00:13:27,624 Mm-hmm. 294 00:13:27,789 --> 00:13:29,889 arguing or against the title 295 00:13:30,424 --> 00:13:30,774 Uhhuh. 296 00:13:30,819 --> 00:13:38,019 is I think because backup speeds have improved technology has made 297 00:13:38,019 --> 00:13:42,939 it more efficient, which I know we haven't talked about yet, but has also 298 00:13:42,939 --> 00:13:47,889 allowed us to reduce the backup window from say, once a day, eight hours. 299 00:13:48,534 --> 00:13:48,864 Yep. 300 00:13:49,809 --> 00:13:51,549 Or your backup frequency, I should say, 301 00:13:52,014 --> 00:13:52,314 Yep, 302 00:13:52,329 --> 00:13:57,669 once a day down to say, every four hours with less impact on 303 00:13:57,774 --> 00:13:58,134 yep. 304 00:13:58,359 --> 00:14:00,309 and everything else, and 305 00:14:00,384 --> 00:14:00,774 I, I would. 306 00:14:00,879 --> 00:14:03,969 to take it more frequently and give yourself better protection. 307 00:14:04,704 --> 00:14:09,774 I would actually say that the first thing that it did was it actually blew out the 308 00:14:09,774 --> 00:14:16,704 backup window because, because we can do backups and, and they don't significantly 309 00:14:16,704 --> 00:14:18,474 impact the performance of the server. 310 00:14:18,594 --> 00:14:21,444 Hopefully, depending on your backup, you may actually, depending on the server 311 00:14:21,444 --> 00:14:25,014 and depending on the backup, you may still, if you're still, for example, 312 00:14:25,014 --> 00:14:26,484 if you're doing database backups. 313 00:14:27,579 --> 00:14:30,189 And you're still required to do full and incremental backups. 314 00:14:30,219 --> 00:14:33,339 Those could still have a significant performance impact on the server. 315 00:14:33,339 --> 00:14:35,109 But everything else where we're doing. 316 00:14:35,889 --> 00:14:39,339 Either some type of block, a level incremental or some type of source site 317 00:14:39,339 --> 00:14:45,039 deduplication where you're the, the performance impact of an individual backup 318 00:14:45,039 --> 00:14:49,599 on a server is relatively nil, which means that you could do it pretty much any time 319 00:14:49,599 --> 00:14:56,174 of day, and that means that as long as you do it within 24 hours, then that's fine. 320 00:14:56,394 --> 00:14:57,014 And then I would. 321 00:14:57,294 --> 00:15:02,214 I completely agree with you that in some cases we've then said, Hey, 322 00:15:02,214 --> 00:15:06,174 because the backup is so quick, we can actually do it more often 323 00:15:06,174 --> 00:15:09,264 than the quintessential once a day 324 00:15:09,339 --> 00:15:09,489 Yep. 325 00:15:09,624 --> 00:15:10,464 we could do it. 326 00:15:10,584 --> 00:15:14,184 Um, you know, every four hours, every hour, every five minutes, there are 327 00:15:14,184 --> 00:15:16,284 backups at some, you know, different ones. 328 00:15:16,284 --> 00:15:22,164 Take every five minutes, um, and we can therefore have a much tighter RPO. 329 00:15:22,224 --> 00:15:22,944 What is that? 330 00:15:23,049 --> 00:15:24,189 Recovery point objective. 331 00:15:24,744 --> 00:15:26,184 Right, exactly. 332 00:15:26,234 --> 00:15:31,714 So if back in the day your backup frequency was once a day, then your RPA, 333 00:15:31,714 --> 00:15:36,874 your recovery point, actual, uh, you know, a lot of people use the term RPO to mean. 334 00:15:36,994 --> 00:15:39,124 R-P-A-R-P-O is the objective. 335 00:15:39,124 --> 00:15:41,075 It's the, it's the, uh, SLA. 336 00:15:41,314 --> 00:15:41,584 Yeah. 337 00:15:41,869 --> 00:15:42,229 Yeah. 338 00:15:42,229 --> 00:15:45,769 The RPA is the actual capabilities of your system. 339 00:15:46,669 --> 00:15:53,959 So the, your, if your backup is once every 24 hours, then the best RPA 340 00:15:53,959 --> 00:15:56,629 you're going to get is likely 24 hours. 341 00:15:56,869 --> 00:16:00,949 You could actually get 48 hours if a backup fails, for example. 342 00:16:00,979 --> 00:16:01,339 Right? 343 00:16:02,059 --> 00:16:04,459 Um, but if we could do a backup in 30 seconds. 344 00:16:05,479 --> 00:16:08,809 With, with minimal impact to a server, you could do it four times 345 00:16:08,809 --> 00:16:11,929 a day, like you said earlier, and we could get a much tighter RPA 346 00:16:13,264 --> 00:16:16,294 I think the one other thing to mention is, and you brought it up 347 00:16:16,294 --> 00:16:19,744 too, is with databases you still have to do fulls and incrementals, 348 00:16:20,629 --> 00:16:21,559 Most databases. 349 00:16:21,559 --> 00:16:21,679 Yeah. 350 00:16:22,294 --> 00:16:25,804 And I think that's where backup speed still plays a part, 351 00:16:26,334 --> 00:16:26,554 Yep. 352 00:16:26,884 --> 00:16:29,464 as databases are growing in size, you 353 00:16:29,534 --> 00:16:29,954 Mm-hmm. 354 00:16:30,034 --> 00:16:31,714 be able to back that up within the window. 355 00:16:31,789 --> 00:16:32,009 So. 356 00:16:32,914 --> 00:16:37,534 Having backups that can go really fast is still a good thing. 357 00:16:38,149 --> 00:16:38,479 Yeah. 358 00:16:38,524 --> 00:16:39,994 a new application comes online. 359 00:16:40,969 --> 00:16:41,359 Yeah. 360 00:16:42,319 --> 00:16:49,039 But, but even then, like the DBA, he just, he or she just assumes that. 361 00:16:49,654 --> 00:16:52,354 You are, you are going to get the backup done. 362 00:16:52,504 --> 00:16:53,704 No one's gonna be impressed. 363 00:16:53,704 --> 00:16:56,524 Hey, I figured out how to back up your, your data. 364 00:16:56,524 --> 00:17:00,274 You know my backup's really fast and now it runs in only four hours. 365 00:17:00,424 --> 00:17:01,384 No one cares, 366 00:17:01,414 --> 00:17:01,684 Yeah. 367 00:17:01,864 --> 00:17:02,164 right? 368 00:17:02,164 --> 00:17:03,844 Going back to the title, no one cares. 369 00:17:03,844 --> 00:17:04,084 You. 370 00:17:04,174 --> 00:17:05,794 Oh, you got the backup done really fast. 371 00:17:05,794 --> 00:17:07,804 No one cares, right? 372 00:17:07,969 --> 00:17:08,209 Yep. 373 00:17:08,374 --> 00:17:12,424 Um, now I will say this, the. 374 00:17:12,529 --> 00:17:15,229 no one cares as long as they're able to meet those windows. 375 00:17:15,229 --> 00:17:15,889 But yeah. 376 00:17:16,309 --> 00:17:17,959 No one cares that you can do it. 377 00:17:18,019 --> 00:17:20,959 They will care very much if you cannot do it right. 378 00:17:21,679 --> 00:17:25,699 Um, and and that's what where I was about to, to head. 379 00:17:25,699 --> 00:17:31,339 And that is the one concern because as you've heard me say 380 00:17:31,339 --> 00:17:34,909 before, I've spent my entire career fighting the laws of physics. 381 00:17:35,059 --> 00:17:35,479 Right? 382 00:17:36,049 --> 00:17:42,199 And so, um, the, even though we have. 383 00:17:42,709 --> 00:17:48,259 Reduced down to the smallest possible increment. 384 00:17:48,259 --> 00:17:52,129 I think in terms of the size of each individual, incremental backup. 385 00:17:52,339 --> 00:17:56,579 When you, when you do block level incremental, or you do source side 386 00:17:56,579 --> 00:18:01,249 deduplication you, you're, you're basically transferring the smallest amount 387 00:18:01,249 --> 00:18:06,139 of data from the server to the whatever it is that you're backing up, that that 388 00:18:06,139 --> 00:18:09,859 could possibly be transferred even. 389 00:18:10,864 --> 00:18:15,844 The, even though we have done that, you add up all the servers, that's a 390 00:18:15,844 --> 00:18:20,704 certain number of gigabytes or terabytes or petabytes that must be transferred 391 00:18:20,704 --> 00:18:23,044 within a particular period of time. 392 00:18:23,524 --> 00:18:28,024 And so still the what matters very much is bandwidth. 393 00:18:28,589 --> 00:18:28,969 Yes. 394 00:18:29,729 --> 00:18:36,449 Um, and you, and unlike, again, back in the day, if we ran the numbers and 395 00:18:36,449 --> 00:18:40,559 the problem was the network, we could just build another network, right? 396 00:18:40,559 --> 00:18:44,039 And, and, and that's what I used to do when I, I, I remember the first 397 00:18:44,039 --> 00:18:47,219 time I did this and these were, do you remember Sun E four fifties? 398 00:18:47,219 --> 00:18:48,179 Do you remember those servers? 399 00:18:48,604 --> 00:18:48,964 Were those A? 400 00:18:50,014 --> 00:18:53,074 Those were the, weren't the sun rays, were they? 401 00:18:53,614 --> 00:18:58,234 No, they, they were like the size and shape of a, of like 402 00:18:58,234 --> 00:19:00,034 a, like a dorm refrigerator. 403 00:19:01,024 --> 00:19:01,474 Right. 404 00:19:01,534 --> 00:19:04,534 And the thing about the E four fifties is, that was amazing. 405 00:19:04,534 --> 00:19:10,024 Was that the, it was, they had the, the great PCI slots and, and they had 406 00:19:10,024 --> 00:19:13,084 multiple, like full speed PCI slots. 407 00:19:13,174 --> 00:19:15,784 And so what you could do is you could put multiple. 408 00:19:16,639 --> 00:19:20,839 Uh, you know, ethernet cards in there and, um, and each of 409 00:19:20,839 --> 00:19:22,129 them could run at full speed. 410 00:19:22,489 --> 00:19:26,989 And back then the, the dream was to get a gigabit card in there, right? 411 00:19:27,004 --> 00:19:27,304 Yep. 412 00:19:27,619 --> 00:19:32,839 And so it was super cheap and easy to build a, a super fast 413 00:19:32,839 --> 00:19:34,489 separate network that that. 414 00:19:34,594 --> 00:19:35,764 Wouldn't be impact. 415 00:19:35,854 --> 00:19:39,964 You know, the backups wouldn't be impacted by production loads and production 416 00:19:39,964 --> 00:19:41,614 loads wouldn't be impacted by backups. 417 00:19:42,184 --> 00:19:46,294 The difference now is in many cases, many of our backups are no 418 00:19:46,294 --> 00:19:47,774 longer going over the corporate LAN. 419 00:19:47,774 --> 00:19:50,584 They're going over the wan, they're going over the internet, 420 00:19:50,854 --> 00:19:51,074 Yep. 421 00:19:51,184 --> 00:19:53,884 hopefully, and I really mean this. 422 00:19:53,914 --> 00:19:55,834 Hopefully you're encrypting that data 423 00:19:56,179 --> 00:19:56,569 Yep. 424 00:19:56,824 --> 00:19:57,514 you're sending it. 425 00:19:57,514 --> 00:19:57,844 Right? 426 00:19:58,354 --> 00:20:01,209 But since you're sending out, out over public networks, but. 427 00:20:02,209 --> 00:20:08,359 But still backup speed does matter in that you've got to make sure 428 00:20:08,359 --> 00:20:09,379 that you have enough bandwidth. 429 00:20:09,379 --> 00:20:12,169 You've got to make sure that you've, you've done these calculations and 430 00:20:12,169 --> 00:20:15,169 you've figured out the, the databases that we talked about that are gonna 431 00:20:15,169 --> 00:20:17,989 need full and incremental backup, those speeds will very much matter. 432 00:20:18,769 --> 00:20:22,639 Um, this is more true with sort of traditional databases, right? 433 00:20:22,639 --> 00:20:27,799 Rdbm S'S relational database management systems, um, like 434 00:20:27,799 --> 00:20:30,229 Oracle Cybase, SQL Server. 435 00:20:30,274 --> 00:20:30,424 Yeah, 436 00:20:30,529 --> 00:20:31,699 Uh, Informix. 437 00:20:31,729 --> 00:20:34,249 I haven't heard Informix in in a minute, but 438 00:20:34,989 --> 00:20:37,204 when, when you're talking about bandwidth, right? 439 00:20:37,264 --> 00:20:37,414 The 440 00:20:37,519 --> 00:20:37,669 Uhhuh. 441 00:20:37,744 --> 00:20:42,124 that came to mind is, and your son, E four 50 example, the other thing that 442 00:20:42,124 --> 00:20:47,584 came to mind is there's also a big push these days to make servers dense. 443 00:20:48,709 --> 00:20:49,039 Mm-hmm. 444 00:20:49,219 --> 00:20:51,439 Right, virtualization, right? 445 00:20:51,439 --> 00:20:52,099 You want to take 446 00:20:52,219 --> 00:20:52,699 Yep. 447 00:20:53,329 --> 00:20:54,409 rather than leave them sitting. 448 00:20:54,649 --> 00:20:59,389 And so you may end up with bottlenecks on those servers that prevent you 449 00:20:59,389 --> 00:21:00,799 from actually sending data out. 450 00:21:00,799 --> 00:21:04,639 So instead of having 10 servers with 10 network interfaces, now 451 00:21:04,639 --> 00:21:07,519 you maybe only have two, right? 452 00:21:07,759 --> 00:21:11,359 And they can only support a certain amount of bandwidth leaving the system 453 00:21:11,959 --> 00:21:12,349 Yep. 454 00:21:12,439 --> 00:21:13,729 10 independent systems before. 455 00:21:14,389 --> 00:21:15,289 Yeah, agreed. 456 00:21:15,289 --> 00:21:15,589 Yeah. 457 00:21:15,589 --> 00:21:19,969 So that, that my bandwidth comment starts at that server and continues 458 00:21:19,969 --> 00:21:23,099 on to your LAN continues on to the WAN 459 00:21:23,194 --> 00:21:23,414 Yep. 460 00:21:23,419 --> 00:21:24,799 and continues on to the vendor. 461 00:21:24,799 --> 00:21:25,069 Right? 462 00:21:25,069 --> 00:21:27,559 It basically, there could be a bottleneck anywhere along that 463 00:21:27,919 --> 00:21:29,869 and, and a, and a bottleneck there. 464 00:21:30,319 --> 00:21:30,889 Um. 465 00:21:31,924 --> 00:21:35,194 Would mean that your backups would not get done within 24 hours, 466 00:21:35,389 --> 00:21:35,719 Yep. 467 00:21:36,394 --> 00:21:40,984 which would then impact your, your r you know, your ability to meet your RPO. 468 00:21:41,134 --> 00:21:41,434 Yeah. 469 00:21:43,084 --> 00:21:47,524 Um, so if they don't care about your backup speed Prasanna, 470 00:21:47,974 --> 00:21:48,904 what do they care about? 471 00:21:49,759 --> 00:21:53,059 How quickly does it take to restore your data? 472 00:21:53,644 --> 00:21:54,364 Exactly. 473 00:21:54,994 --> 00:21:57,364 How quickly can I get my data back? 474 00:21:57,394 --> 00:21:57,874 Right. 475 00:21:58,414 --> 00:22:06,649 Um, and I. I, you know, early on in my career, I was as guilty of this as anyone. 476 00:22:06,649 --> 00:22:13,999 And that is, it's so easy to basically spend all your time designing the backup 477 00:22:13,999 --> 00:22:15,799 system, testing the backup system. 478 00:22:15,799 --> 00:22:19,639 Does it have the capability to do what I needed to do and not 479 00:22:19,639 --> 00:22:22,249 really do any restore testing? 480 00:22:22,429 --> 00:22:27,799 Because the, so first off, let me put out a, what do you call it? 481 00:22:27,799 --> 00:22:28,459 A um. 482 00:22:29,659 --> 00:22:33,529 An axiom would, would that be the right word? 483 00:22:33,919 --> 00:22:40,999 Um, just because your system can back up at a terabyte an hour does not mean 484 00:22:41,059 --> 00:22:42,889 it can restore at a terabyte an hour. 485 00:22:42,919 --> 00:22:43,789 Why might that be? 486 00:22:44,059 --> 00:22:45,979 Oh, a thousand percent. 487 00:22:46,279 --> 00:22:46,789 Uh. 488 00:22:47,719 --> 00:22:50,719 A lot of it, it, having worked at storage vendors in the past that built 489 00:22:50,719 --> 00:22:52,849 de-duplicated appliances, a lot of it is. 490 00:22:53,929 --> 00:22:55,129 The way it writes, 491 00:22:55,414 --> 00:22:55,834 Mm-hmm. 492 00:22:55,999 --> 00:22:59,779 streams, it's very unique, and so you're able to optimize. 493 00:23:00,079 --> 00:23:04,099 But when you're reading data out, you have to read every single piece of data 494 00:23:04,339 --> 00:23:06,319 in order to restore that application. 495 00:23:06,319 --> 00:23:09,199 It's not like, oh, 99% is duplicates. 496 00:23:09,199 --> 00:23:10,459 I can just drop it and just send the 497 00:23:10,589 --> 00:23:10,809 Yep. 498 00:23:11,569 --> 00:23:14,959 I actually have to read all the data out and how it's laid out on disk 499 00:23:14,959 --> 00:23:19,099 may not be optimized, especially with deduplication and other technologies. 500 00:23:20,044 --> 00:23:26,074 And we, I, I have in the past, referred to this as the dedupe tax. 501 00:23:26,269 --> 00:23:26,539 Yep. 502 00:23:26,674 --> 00:23:30,784 difference between if we, if we lay the data down on disc, like without 503 00:23:30,784 --> 00:23:35,224 dedupe, and then we lay it down on disc with dedupe, the difference in 504 00:23:35,224 --> 00:23:38,974 restore speed between the two is often referred to as, you know, the dedupe tax. 505 00:23:39,544 --> 00:23:44,044 And I've seen it be really, really, really bad. 506 00:23:44,299 --> 00:23:44,779 me too. 507 00:23:44,824 --> 00:23:45,304 Yeah, 508 00:23:45,409 --> 00:23:45,679 Yep. 509 00:23:45,739 --> 00:23:49,174 I, I can think of one vendor and I'll let them go nameless. 510 00:23:49,174 --> 00:23:50,824 'cause they're still around as a vendor. 511 00:23:51,034 --> 00:23:55,024 They still sell, uh, dedupe appliances. 512 00:23:56,014 --> 00:23:58,984 I'm pretty sure they've addressed this issue, but when 513 00:23:58,984 --> 00:24:02,584 it first came out, it had a 90%. 514 00:24:03,224 --> 00:24:10,594 dedupe tax in that, I remember that the, the, the sy the, you know, version one 515 00:24:10,654 --> 00:24:18,154 of the system had the ability to, to, uh, write data at 400 megabytes per second. 516 00:24:19,114 --> 00:24:23,314 Its ability to read data was 40 megabytes per second. 517 00:24:23,404 --> 00:24:25,714 The aggregate throughput of the entire appliance was 518 00:24:25,714 --> 00:24:27,124 only 40 megabytes per second. 519 00:24:27,694 --> 00:24:29,404 And, um, it. 520 00:24:29,749 --> 00:24:31,879 Was not good. 521 00:24:31,939 --> 00:24:36,709 And, and if you didn't test that, then you know, you, you know, 522 00:24:36,859 --> 00:24:37,009 Do you 523 00:24:37,069 --> 00:24:38,599 you'd find it out at the wrong time. 524 00:24:38,629 --> 00:24:39,859 you wanna talk about your example? 525 00:24:41,089 --> 00:24:41,689 Which one? 526 00:24:41,779 --> 00:24:46,999 The one that you had the test with, the compression it had 527 00:24:46,999 --> 00:24:47,209 Oh 528 00:24:47,239 --> 00:24:49,429 it and you know, you know that one. 529 00:24:49,519 --> 00:24:49,819 yeah. 530 00:24:50,689 --> 00:24:51,259 Yeah. 531 00:24:51,529 --> 00:24:52,249 Um. 532 00:24:52,909 --> 00:24:58,609 So, you know, back in the day, um, back before, like when I used my 533 00:24:58,609 --> 00:25:03,979 first commercial backup product, they had a software compression feature. 534 00:25:04,459 --> 00:25:09,019 Um, that the idea was that it was, this was before I was using tape drive 535 00:25:09,019 --> 00:25:15,049 compression and they had this ability and, um, it was, it was allowing me to 536 00:25:15,049 --> 00:25:16,819 do things that I wouldn't be able to do. 537 00:25:16,819 --> 00:25:21,889 And so I chose this as a, as a setup option during, during my initial setup. 538 00:25:23,224 --> 00:25:26,284 And I didn't do restore, I didn't do restore testing. 539 00:25:27,154 --> 00:25:32,134 And then, um, there was a moment when, uh, very bad things happened 540 00:25:32,134 --> 00:25:34,264 and I needed to pull my tapes out. 541 00:25:35,104 --> 00:25:38,644 I will just say, uh, so I remember running over there and being super 542 00:25:38,644 --> 00:25:42,664 excited that I was gonna, now, you know, I was essentially going into battle 543 00:25:42,694 --> 00:25:44,884 with an untested weapon, but whatever. 544 00:25:44,884 --> 00:25:49,324 I was super excited and, um, I remember. 545 00:25:49,804 --> 00:25:52,174 Standing there and, and firing off the restore. 546 00:25:52,174 --> 00:25:56,104 And I remember watching the tape drive going like blink, blink, and then 547 00:25:56,194 --> 00:25:58,954 really long period of time then Blink. 548 00:25:58,954 --> 00:25:59,464 Blink. 549 00:26:00,184 --> 00:26:05,944 And what I found out was that, um, when they do compression, 550 00:26:05,944 --> 00:26:08,224 the way they do it is literally. 551 00:26:08,619 --> 00:26:14,199 For the Unix people in the crowd, they were doing a compressed minus C to, they 552 00:26:14,199 --> 00:26:18,429 would com, they would write it to temp and then they would write that compressed 553 00:26:18,459 --> 00:26:20,769 tape to that compressed file to tape. 554 00:26:21,459 --> 00:26:25,059 And then when they were restoring it, they would restore that compressed file 555 00:26:25,059 --> 00:26:30,549 to temp. Then they would run uncompress in place, not uncompressed minus C, like 556 00:26:30,549 --> 00:26:31,899 they weren't doing it to standard out. 557 00:26:32,079 --> 00:26:33,369 They would uncompress it. 558 00:26:34,564 --> 00:26:38,974 And then they would move that file to, uh, where it needed to go. 559 00:26:39,634 --> 00:26:44,014 Um, and so that's when I learned, uh, from the vendor, like, yeah, 560 00:26:44,014 --> 00:26:48,574 we don't recommend this feature if speed is is important to you. 561 00:26:49,714 --> 00:26:50,164 You're like, 562 00:26:50,374 --> 00:26:50,974 okay. 563 00:26:51,004 --> 00:26:51,634 anywhere 564 00:26:51,729 --> 00:26:52,144 we, we, 565 00:26:52,354 --> 00:26:52,624 Yeah. 566 00:26:53,164 --> 00:26:53,824 where was that? 567 00:26:53,824 --> 00:26:54,574 In the manual? 568 00:26:54,574 --> 00:26:54,994 Right. 569 00:26:55,534 --> 00:26:56,974 Um, and. 570 00:26:57,679 --> 00:27:01,459 So there are things that you're, you're, you're not going to know 571 00:27:01,459 --> 00:27:03,649 until you do, uh, restore testing. 572 00:27:04,409 --> 00:27:08,579 So you, you have to do restore testing, right? 573 00:27:08,819 --> 00:27:13,589 I mean, for one thing, it, uh, it, how are you gonna know? 574 00:27:13,589 --> 00:27:15,449 The restore even works, even works at all. 575 00:27:15,449 --> 00:27:19,319 Um, I'll, I'll, I'll tell, I'll tell you another story. 576 00:27:20,069 --> 00:27:20,579 Curtis is 577 00:27:20,759 --> 00:27:24,719 was, yeah, I was, what's that? 578 00:27:24,929 --> 00:27:26,339 Are you sure it's not the espresso? 579 00:27:27,014 --> 00:27:28,334 This is an espresso talking. 580 00:27:29,294 --> 00:27:36,224 I was working, I was actually, um, at the headquarters of Motorola 581 00:27:36,419 --> 00:27:36,779 Hmm. 582 00:27:37,064 --> 00:27:46,154 and um, working with the computer simulation modeling and research division, 583 00:27:46,154 --> 00:27:53,414 which was a cutting edge division at the time that was using computers to simulate. 584 00:27:53,909 --> 00:27:58,169 Damage to phones and to improve the design as a result. 585 00:27:58,259 --> 00:28:02,639 And I remember that this was the group that basically came 586 00:28:02,639 --> 00:28:04,169 up with the Star Tech, right? 587 00:28:04,169 --> 00:28:07,049 Which was the, the phone that put Motorola on the map. 588 00:28:07,139 --> 00:28:07,559 Right. 589 00:28:08,729 --> 00:28:10,199 And um. 590 00:28:10,949 --> 00:28:15,959 I was using their tape drives and I, I had their backup system and I had it going 591 00:28:15,959 --> 00:28:19,589 and everything was, you know, was going and I was, I was a full on admin, but, but 592 00:28:19,589 --> 00:28:23,039 backups was part of my responsibilities. 593 00:28:23,759 --> 00:28:28,919 And, um, I had been, I'd been there for a few months and I'd been doing backups 594 00:28:28,919 --> 00:28:33,419 and I just sort of assumed that the backup system that I inherited worked. 595 00:28:33,674 --> 00:28:33,974 Yeah. 596 00:28:34,289 --> 00:28:39,839 And I, um, I just remember the first time I went to go do a restore. 597 00:28:40,289 --> 00:28:47,369 And what I found is, uh, so these were IBM, like 34 80 something like 598 00:28:47,369 --> 00:28:49,529 that, old school IBM cartridges. 599 00:28:49,979 --> 00:28:51,569 They were really good at writing. 600 00:28:52,829 --> 00:28:54,809 They were completely incapable of reading. 601 00:28:57,594 --> 00:29:00,779 I, I really don't remember what the solution to this was, 602 00:29:01,274 --> 00:29:01,394 you 603 00:29:01,409 --> 00:29:05,339 but I remember that I was like, I, I would kick off the restore and, and 604 00:29:05,339 --> 00:29:07,379 the restore would always fail because. 605 00:29:07,874 --> 00:29:09,854 Um, the, the tape drives couldn't read. 606 00:29:10,874 --> 00:29:16,034 And I'm like, you know, like it's, it's like you have two jobs. 607 00:29:17,474 --> 00:29:18,374 It's not just one job. 608 00:29:18,494 --> 00:29:19,514 You have two jobs. 609 00:29:19,754 --> 00:29:21,974 You have write and you have read. 610 00:29:22,154 --> 00:29:25,244 You have to do one to do the other, but the other is really, really important. 611 00:29:25,244 --> 00:29:27,434 You can't do just the one and not do the other. 612 00:29:28,019 --> 00:29:32,489 Um, it reminds me of, uh, the Seinfeld episode with the, the car rental 613 00:29:32,939 --> 00:29:35,279 where he is like, where they, where he's like, they're like, we don't 614 00:29:35,279 --> 00:29:36,539 have, we don't have a car for you. 615 00:29:36,539 --> 00:29:38,009 And he is like, but I had a reservation. 616 00:29:38,009 --> 00:29:41,129 He goes, yes, we understand you have a reservation, but you don't have a car. 617 00:29:41,399 --> 00:29:45,239 And he's like, anybody can, anybody can take the reservations to, you 618 00:29:45,239 --> 00:29:46,439 know, take the reservations, you know? 619 00:29:46,439 --> 00:29:50,279 But the, the important part is the holding of the reservation. 620 00:29:51,594 --> 00:29:53,549 And that's, that's the way it is to back up. 621 00:29:53,819 --> 00:29:55,859 You know, it, it's important to back up. 622 00:29:55,919 --> 00:29:57,389 It's really important to restore. 623 00:29:57,689 --> 00:30:01,559 Um, I, I literally have, I have no memory of what the solution to that 624 00:30:01,559 --> 00:30:04,919 was, but I definitely remember the day that I went to go do a restore 625 00:30:04,919 --> 00:30:06,719 and absolutely nothing happened. 626 00:30:06,899 --> 00:30:08,999 at three in the morning with the popped in your head. 627 00:30:11,309 --> 00:30:14,519 I, you know, it was probably a really traumatic event and I, you know, 628 00:30:14,824 --> 00:30:16,499 I, I, I wiped it outta my memory. 629 00:30:17,639 --> 00:30:17,939 Um. 630 00:30:18,629 --> 00:30:20,279 And, and so go ahead. 631 00:30:20,699 --> 00:30:25,919 But I guess one question is, so I know we talked about restore 632 00:30:25,919 --> 00:30:30,179 speeds, newer technologies, newer storage vendors, right? 633 00:30:30,179 --> 00:30:31,439 They are addressing this, 634 00:30:32,069 --> 00:30:32,459 Mm-hmm. 635 00:30:32,609 --> 00:30:37,139 The speeds have improved for restorers because new use cases in 636 00:30:37,139 --> 00:30:40,889 addition to just restoring data such as instant access and other things 637 00:30:40,889 --> 00:30:42,449 where performance does matter. 638 00:30:42,809 --> 00:30:45,749 And if you look at some vendors like Vast, right? 639 00:30:45,749 --> 00:30:49,079 And other things they are leveraging, um. 640 00:30:50,669 --> 00:30:52,769 flash more heavily in 641 00:30:52,799 --> 00:30:52,889 Right. 642 00:30:53,069 --> 00:30:54,539 to help improve performance. 643 00:30:54,959 --> 00:31:00,899 And so I think some parts of that are being addressed, but you still 644 00:31:00,899 --> 00:31:04,259 also have like the reading the data out and everything else, and 645 00:31:04,259 --> 00:31:09,059 just like to get a full copy of the data out is just very expensive. 646 00:31:10,154 --> 00:31:15,264 Yeah, I, I would say when we talk about dedupe, and I'm a fan of dedupe. 647 00:31:16,454 --> 00:31:20,384 It's really important how you write the data and it's really important what it's 648 00:31:20,384 --> 00:31:22,994 like to get that data back en mass, right? 649 00:31:22,994 --> 00:31:26,774 It's really easy to just write the bits that you need for a backup. 650 00:31:27,434 --> 00:31:31,514 How you write it very much determines how you're going to be 651 00:31:31,514 --> 00:31:35,384 able to read it and um, and so. 652 00:31:36,314 --> 00:31:40,424 As a result, um, again, the only way you're going to know that 653 00:31:40,424 --> 00:31:42,314 is to is to do restore testing. 654 00:31:42,359 --> 00:31:42,599 Yep. 655 00:31:43,004 --> 00:31:47,174 The other thing is that when you do restore testing, you have to 656 00:31:47,174 --> 00:31:49,634 look at all of the different. 657 00:31:50,804 --> 00:31:57,224 Ways in ways in which, and things that you need to restore, right? 658 00:31:57,284 --> 00:31:59,774 You need to look at all of the different applications. 659 00:32:00,254 --> 00:32:02,894 You need to look at all of the different types of servers and 660 00:32:02,894 --> 00:32:08,864 VMs and, and serverless, um, applications that you might have and, 661 00:32:09,329 --> 00:32:10,109 dependencies. 662 00:32:10,604 --> 00:32:12,164 and dependencies, right? 663 00:32:12,524 --> 00:32:15,374 Well, then we start talking about disaster recovery, right? 664 00:32:15,524 --> 00:32:17,954 When we start talking about disaster recovery, we introduce 665 00:32:17,954 --> 00:32:19,814 the concept of a recovery group. 666 00:32:19,814 --> 00:32:20,264 What is that? 667 00:32:20,744 --> 00:32:24,884 It's everything that is needed in order to bring up your environment or your 668 00:32:24,884 --> 00:32:27,164 application or whatever that object is 669 00:32:27,434 --> 00:32:28,419 you are looking to recover. 670 00:32:29,279 --> 00:32:29,669 right. 671 00:32:29,699 --> 00:32:35,549 So let's say you want to restore, um, I, you know, even though no 672 00:32:35,549 --> 00:32:38,879 one has exchange anymore, you wanna restore your exchange server, right? 673 00:32:39,149 --> 00:32:40,919 Everybody, everybody's moved to 365. 674 00:32:40,964 --> 00:32:41,354 Someone's 675 00:32:41,429 --> 00:32:43,589 gonna get, I'm gonna get like a bunch of emails from people say, 676 00:32:43,589 --> 00:32:44,939 Hey man, I still use exchange. 677 00:32:45,359 --> 00:32:45,719 Listen. 678 00:32:46,589 --> 00:32:51,599 It's time to, it's time to move on people anyway, so you wanna restore 679 00:32:51,599 --> 00:32:56,789 your exchange server, um, you know, you have to restore active directory, 680 00:32:56,789 --> 00:32:58,049 for example, before you do that. 681 00:32:58,294 --> 00:33:03,224 And I. Um, I did, does exchange use, it doesn't use SQL underneath, right? 682 00:33:03,464 --> 00:33:03,734 No, 683 00:33:03,884 --> 00:33:04,154 no. 684 00:33:04,214 --> 00:33:06,914 but there may, but, but there may be an application where you need an 685 00:33:06,914 --> 00:33:08,714 application on top of a database. 686 00:33:08,954 --> 00:33:13,094 Maybe there's multiple databases, maybe there's, uh, authentication, 687 00:33:13,274 --> 00:33:15,944 authentication and authorization, and I think is probably the 688 00:33:15,944 --> 00:33:19,754 biggest one that that is a dependency for most environments. 689 00:33:19,754 --> 00:33:21,194 Can you think of anything else like that? 690 00:33:23,399 --> 00:33:26,309 sometimes even, well, depending on if anyone's still doing 691 00:33:26,309 --> 00:33:27,389 bare metal recoveries. 692 00:33:27,764 --> 00:33:28,154 Mm-hmm. 693 00:33:29,279 --> 00:33:29,639 Right. 694 00:33:29,639 --> 00:33:34,169 Bringing up your host first and your server before you can actually start 695 00:33:34,169 --> 00:33:37,769 doing your application, which might be a completely separate recovery process. 696 00:33:38,789 --> 00:33:39,149 Yeah. 697 00:33:39,154 --> 00:33:43,469 If, if you're doing bare metal recovery, uh, which is a term that we used, I 698 00:33:43,469 --> 00:33:47,039 mean, literally it means you have a piece of, you have a piece of metal, 699 00:33:47,099 --> 00:33:51,269 you have a box, and it's got nothing on it, and you're gonna recover from there. 700 00:33:52,649 --> 00:33:55,499 Most people, again, no one does this anymore. 701 00:33:56,609 --> 00:33:59,399 And the reason why they don't do it is because they're doing VMware 702 00:33:59,504 --> 00:33:59,774 Yep. 703 00:34:00,449 --> 00:34:04,559 um, you know, other, or Hyper V or, you know, uh, any of 704 00:34:04,559 --> 00:34:06,359 the other different methods. 705 00:34:06,419 --> 00:34:08,399 I don't think anybody's gonna be doing VMware anymore. 706 00:34:08,879 --> 00:34:13,559 Um, everybody, everybody I talk to that has anything to do with VMware, it's like, 707 00:34:13,559 --> 00:34:16,259 yeah, we're trying to figure out how to move off of VMware thanks to Broadcom. 708 00:34:17,219 --> 00:34:19,709 And, uh, and Broadcom will probably still make money. 709 00:34:19,739 --> 00:34:21,359 'cause basically they're like, we're gonna, we're gonna. 710 00:34:21,839 --> 00:34:26,969 We're gonna do a 10 x, you know, um, we're only gonna have five customers less, 711 00:34:26,974 --> 00:34:29,849 and we're gonna, and we're gonna charge them $10 million each and we're fine. 712 00:34:30,809 --> 00:34:35,969 Um, the, um, totally lost my train of thought. 713 00:34:35,969 --> 00:34:40,139 Oh, so, so no one does bare metal recovery because it's so much easier 714 00:34:40,139 --> 00:34:43,469 to do a bare metal recovery, or the equivalent of a bare metal recovery. 715 00:34:43,469 --> 00:34:45,859 When you have a vm, you can back up the VM whole. 716 00:34:46,139 --> 00:34:49,589 You just restore that as a file and boom, you're off, you're off to the races. 717 00:34:49,604 --> 00:34:54,314 but remember, even there, you might need to bring back your vCenter instance, 718 00:34:54,629 --> 00:34:55,259 Yes. 719 00:34:55,394 --> 00:34:55,994 And everything 720 00:34:56,049 --> 00:35:01,719 That's a perfect ex example of a, of a, um, I. Of a dependency, right? 721 00:35:01,809 --> 00:35:05,769 Uh, if you have, if, if you, you're going to have to install vCenter, 722 00:35:05,889 --> 00:35:09,969 which means hopefully have a backup of the vCenter environment, right? 723 00:35:10,029 --> 00:35:14,199 Um, and there's probably an authentication authorization system to do that. 724 00:35:14,409 --> 00:35:18,279 It, you know, it, it, it, you, you have to do all of these dependencies. 725 00:35:18,309 --> 00:35:21,999 This is why you do recovery testing and you. 726 00:35:22,434 --> 00:35:27,984 You just create these different scenarios, and I will say that they are going to 727 00:35:27,984 --> 00:35:29,844 very much care about your recovery speeds. 728 00:35:29,949 --> 00:35:30,189 Yep. 729 00:35:30,714 --> 00:35:33,024 They don't care about your backup speed, but they care very much 730 00:35:33,024 --> 00:35:33,984 about your recovery speeds. 731 00:35:34,194 --> 00:35:37,044 And they, they, they really don't care about these dependencies. 732 00:35:37,374 --> 00:35:40,374 They're like, look, all I care about is this application. 733 00:35:40,704 --> 00:35:41,819 Why is it taking me so long? 734 00:35:41,819 --> 00:35:46,344 Well, because I have to store 37 other servers before I get, before 735 00:35:46,344 --> 00:35:47,934 I get to restore your application. 736 00:35:48,534 --> 00:35:51,114 Um, the. 737 00:35:52,224 --> 00:35:54,799 Did you have a, were you about to say something? 738 00:35:56,484 --> 00:36:00,354 The one other thing to think about from uh, restore speed perspective 739 00:36:00,679 --> 00:36:01,099 Mm-hmm. 740 00:36:01,254 --> 00:36:04,584 make sure it's realistic what you're looking to test. 741 00:36:05,249 --> 00:36:05,599 Uhhuh, 742 00:36:05,799 --> 00:36:09,729 an example, you might just test, say gigabyte of files 743 00:36:10,339 --> 00:36:10,689 right? 744 00:36:10,869 --> 00:36:13,869 gigabytes of files or one application, right? 745 00:36:13,899 --> 00:36:19,599 And that may not actually reflect the system actually behaves when you're 746 00:36:19,599 --> 00:36:21,099 trying to do these massive restores. 747 00:36:21,099 --> 00:36:21,189 Yep. 748 00:36:22,089 --> 00:36:26,319 Yeah, I, I completely agree and disagree at the same time. 749 00:36:27,069 --> 00:36:28,509 You gotta walk before you can run. 750 00:36:28,539 --> 00:36:31,269 If you've done no recovery testing, I would highly recommend 751 00:36:31,269 --> 00:36:33,099 restoring 10 gigabytes of files. 752 00:36:34,329 --> 00:36:36,219 You know, you know who would agree with me right now? 753 00:36:36,384 --> 00:36:36,804 Jeff. 754 00:36:37,509 --> 00:36:38,049 No. 755 00:36:39,249 --> 00:36:39,939 Stewart. 756 00:36:40,074 --> 00:36:40,794 Oh, sorry, Stuart. 757 00:36:40,794 --> 00:36:41,214 Yes. 758 00:36:41,439 --> 00:36:41,649 Yeah, 759 00:36:41,964 --> 00:36:42,144 Yep, 760 00:36:42,189 --> 00:36:48,339 so, so Stewart very much would've liked to have done a restore of, of, 761 00:36:48,339 --> 00:36:51,579 you know, 10 or 20 gigabytes of his data back before he needed to do a 762 00:36:51,579 --> 00:36:54,129 restore of, of 800 gigabytes of data. 763 00:36:54,174 --> 00:36:55,374 go back and listen to that episode. 764 00:36:55,374 --> 00:36:56,034 I think we'll put 765 00:36:56,139 --> 00:36:56,649 Yeah. 766 00:36:56,724 --> 00:36:57,264 show notes. 767 00:36:57,579 --> 00:36:57,939 Yeah. 768 00:36:57,939 --> 00:37:04,089 It's a great episode about the a, a true, a true life restore of a, of a 769 00:37:04,299 --> 00:37:08,049 basically Prasannal data, but it's, it was 800 gigabytes of Prasannal data, 770 00:37:08,049 --> 00:37:09,849 so it was a significant, you know. 771 00:37:10,134 --> 00:37:13,344 and this is by an experienced backup admin, 772 00:37:14,019 --> 00:37:14,829 Yeah, right. 773 00:37:14,829 --> 00:37:18,759 A person who'd spent an entire career being a backup admin, but like a 774 00:37:18,759 --> 00:37:22,599 lot of people, like he's using cloud backup for his Prasannal data and 775 00:37:22,779 --> 00:37:27,249 you know, he's, he, he has zero control over how this thing works. 776 00:37:27,249 --> 00:37:29,229 He just pushes the button and magic happens. 777 00:37:29,229 --> 00:37:29,529 Right? 778 00:37:30,189 --> 00:37:31,839 And let's just say it wasn't so magic. 779 00:37:35,214 --> 00:37:38,034 So, yeah, so you've got to do your restore speed testing. 780 00:37:38,034 --> 00:37:41,754 They very much, and, and I would say start small, work your way up. 781 00:37:41,904 --> 00:37:47,304 But I completely agree with you that you want to test, um, you know as much 782 00:37:47,304 --> 00:37:51,024 as you can, as real as, and that's the beauty of the cloud, is that you can 783 00:37:51,024 --> 00:37:55,524 basically push a button, recreate your environment, test the crap out of it, 784 00:37:55,794 --> 00:37:59,784 uh, and then delete it, and you just pay for it, uh, during the duration. 785 00:38:00,084 --> 00:38:01,524 What I don't think you should do. 786 00:38:02,244 --> 00:38:06,864 Is what our friend in Alaska did, and that is delete your entire data center 787 00:38:06,864 --> 00:38:08,994 and then test your, uh, recovery speed. 788 00:38:09,594 --> 00:38:10,734 We, we do. 789 00:38:12,114 --> 00:38:17,124 We love you, Paul, but, uh, you know, you, you, you know, you know you did wrong. 790 00:38:18,984 --> 00:38:20,334 You, you made it out. 791 00:38:20,334 --> 00:38:21,924 You made it out to the other side. 792 00:38:21,924 --> 00:38:27,834 But, uh, you know, um, and we're gonna have you on again soon, um, uh, to tell 793 00:38:27,834 --> 00:38:29,244 us, you know, what else you've been up to. 794 00:38:29,319 --> 00:38:31,044 'cause, uh, that, that sounds interesting. 795 00:38:32,274 --> 00:38:34,304 So no one cares about your backup speed Prasanna. 796 00:38:35,184 --> 00:38:35,544 Yeah. 797 00:38:35,994 --> 00:38:38,424 They only care about your restores and how 798 00:38:38,424 --> 00:38:39,024 Exactly. 799 00:38:39,114 --> 00:38:40,254 for them to get their data back. 800 00:38:41,139 --> 00:38:49,479 I will also say that investigate the idea of restoring in advance backup products. 801 00:38:49,479 --> 00:38:53,349 Support the idea, you know, you, you mentioned one, this idea of instant 802 00:38:53,349 --> 00:38:57,969 recovery where you can just literally basically push a button and, and 803 00:38:58,029 --> 00:39:01,899 there's a copy of your data ready to roll that you can just mount that. 804 00:39:01,899 --> 00:39:03,039 That's a great idea. 805 00:39:03,414 --> 00:39:07,734 Um, there are also companies that essentially you create a recovery 806 00:39:07,734 --> 00:39:13,374 group and they essentially pretor it, um, you know, on an ongoing basis 807 00:39:13,374 --> 00:39:17,814 so that when you go to do an actual recovery, you can just literally 808 00:39:17,814 --> 00:39:20,634 push a button and go, um, networks. 809 00:39:20,994 --> 00:39:24,924 Um, that's, that's the, that's the, the only way to have a really fast 810 00:39:24,924 --> 00:39:28,614 restore is to have already restored it before you actually need it. 811 00:39:28,764 --> 00:39:29,124 Yep. 812 00:39:29,154 --> 00:39:30,804 And then the other way is snapshots. 813 00:39:30,984 --> 00:39:31,224 Yep. 814 00:39:31,404 --> 00:39:31,884 Right. 815 00:39:32,304 --> 00:39:37,044 If, if you're able to just mount a snapshot and roll, uh, that's awesome. 816 00:39:37,284 --> 00:39:37,674 Mm-hmm. 817 00:39:37,854 --> 00:39:42,294 And that's why so many people are fans of doing backups and recoveries that way. 818 00:39:42,474 --> 00:39:42,744 Yep. 819 00:39:43,764 --> 00:39:44,124 All right. 820 00:39:44,124 --> 00:39:46,684 Well thanks for another great chat Prasanna. 821 00:39:47,574 --> 00:39:53,604 Anytime Curtis and remember, only three espressos a day or Americanos? 822 00:39:54,834 --> 00:39:55,494 Only three. 823 00:39:55,749 --> 00:39:56,040 I've, I've 824 00:39:56,064 --> 00:39:56,604 Only 825 00:39:56,759 --> 00:39:56,919 had. 826 00:39:58,074 --> 00:40:01,434 I've had two so far, so I mean, I could, I can go get one. 827 00:40:01,494 --> 00:40:02,004 You you're 828 00:40:02,094 --> 00:40:05,694 You know, you know what, you know what's funny is like when you buy 829 00:40:05,694 --> 00:40:09,984 a thing, they send you, like this, this, um, the Yeah, they send you. 830 00:40:10,044 --> 00:40:11,934 So this is what comes with the Nespresso. 831 00:40:11,934 --> 00:40:14,244 These are all the pods and um, 832 00:40:14,249 --> 00:40:14,669 that's 833 00:40:14,724 --> 00:40:15,774 this pod right here. 834 00:40:16,284 --> 00:40:16,734 By the way, 835 00:40:16,794 --> 00:40:17,214 Yeah. 836 00:40:17,394 --> 00:40:19,194 that, when you buy a sleeve, it doesn't come like that. 837 00:40:19,794 --> 00:40:20,124 Yeah. 838 00:40:20,454 --> 00:40:23,364 Um, so this pod right here, you know what this one is? 839 00:40:23,469 --> 00:40:23,759 What? 840 00:40:24,354 --> 00:40:25,794 This is a, what's the point? 841 00:40:25,854 --> 00:40:26,364 Pod. 842 00:40:27,144 --> 00:40:27,984 You know what it is? 843 00:40:28,914 --> 00:40:29,754 It's decaf 844 00:40:30,954 --> 00:40:31,764 Oh yeah. 845 00:40:32,994 --> 00:40:33,804 like No, no. 846 00:40:33,804 --> 00:40:34,254 Thank you. 847 00:40:35,244 --> 00:40:38,064 Um, tio decaf. 848 00:40:38,934 --> 00:40:39,234 Yeah. 849 00:40:39,264 --> 00:40:39,894 No, thank you. 850 00:40:39,899 --> 00:40:40,249 I'm good. 851 00:40:41,874 --> 00:40:43,974 I'm like, what is the point of that? 852 00:40:44,514 --> 00:40:46,974 Anyway, well, thanks for listening folks. 853 00:40:47,064 --> 00:40:51,264 Uh, you're why we do this and, um, good luck out there. 854 00:40:51,504 --> 00:40:52,914 And that is a wrap. 855 00:40:55,779 --> 00:41:00,319 The backup wrap up is written, recorded, and produced by me w Curtis Preston. 856 00:41:01,059 --> 00:41:05,679 If you need backup or Dr. Consulting content generation or expert witness 857 00:41:05,789 --> 00:41:08,319 work, check out backup central.com. 858 00:41:08,939 --> 00:41:12,319 You can also find links from my O'Reilly Books on the same website. 859 00:41:13,319 --> 00:41:16,959 Remember, this is an independent podcast and any opinions that 860 00:41:16,959 --> 00:41:20,719 you hear are those of the speaker and not necessarily an employer. 861 00:41:21,499 --> 00:41:22,439 Thanks for listening.