1 00:00:00,050 --> 00:00:04,070 You found the backup wrap up your go-to podcast for all things 2 00:00:04,070 --> 00:00:06,530 backup recovery and cyber recovery. 3 00:00:07,010 --> 00:00:10,280 In this episode, we're tackling one of the biggest lies in it, 4 00:00:10,970 --> 00:00:13,190 your recovery time objective. 5 00:00:13,700 --> 00:00:17,900 I don't care what your RTO documentation says or what you 6 00:00:17,900 --> 00:00:20,060 believe you've promised your bosses. 7 00:00:20,570 --> 00:00:23,240 If you haven't tested it, you can't meet it. 8 00:00:23,630 --> 00:00:28,939 Period persona and I break down why most organizations are living in fantasy 9 00:00:28,939 --> 00:00:34,040 land when it comes to recovery time, objective, and more importantly, what 10 00:00:34,040 --> 00:00:36,950 you can actually do to address that gap. 11 00:00:37,670 --> 00:00:40,730 If you've ever felt that pit in your stomach when someone 12 00:00:40,730 --> 00:00:42,980 asks you about recovery times. 13 00:00:43,385 --> 00:00:44,914 This is your episode. 14 00:00:45,305 --> 00:00:47,224 Let's get real about RTO. 15 00:00:48,035 --> 00:00:52,415 By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr. 16 00:00:52,415 --> 00:00:57,275 Backup, and I've been passionate about backup and recovery ever since. 17 00:00:57,455 --> 00:01:01,715 I had to tell my boss that there were no backups of that production 18 00:01:01,715 --> 00:01:03,395 database that we had just lost. 19 00:01:03,667 --> 00:01:06,667 I don't want that to happen to you, and that's why I do this. 20 00:01:07,132 --> 00:01:12,412 On this podcast, we turn unappreciated backup admins into cyber recovery heroes. 21 00:01:12,712 --> 00:01:14,842 This is the backup wrap up. 22 00:01:29,154 --> 00:01:30,894 hi, and welcome to the backup wrap up. 23 00:01:30,894 --> 00:01:34,674 I'm your host, w Curtis Preston, AKA, Mr. Backup, and I have with 24 00:01:34,674 --> 00:01:37,524 me the rarest of All Beasts lately. 25 00:01:37,554 --> 00:01:40,254 Anyway, Prasanna Malaiyandi how's it going? 26 00:01:40,254 --> 00:01:40,674 Prasanna, I. 27 00:01:42,069 --> 00:01:43,629 I am good, Curtis. 28 00:01:43,629 --> 00:01:45,009 I know it's been a 29 00:01:45,209 --> 00:01:46,584 It is been, it's been a minute. 30 00:01:46,719 --> 00:01:46,929 we've 31 00:01:47,094 --> 00:01:50,664 why, that's why the listeners have been listening to like repeats. 32 00:01:51,264 --> 00:01:53,064 Uh, yeah. 33 00:01:53,124 --> 00:01:56,514 'cause you of course you're gonna blame it on me with I know, I know. 34 00:01:56,514 --> 00:01:59,844 I was working the election, I was working the, uh, I, for those who don't 35 00:01:59,844 --> 00:02:03,834 know, I'm gonna, I'm a poll worker, you know, and I'm not doing other things. 36 00:02:04,974 --> 00:02:05,304 Site 37 00:02:05,484 --> 00:02:05,754 Yeah. 38 00:02:05,754 --> 00:02:09,114 I, I am a site manager of the Yeah, the Bonsall Vote Center. 39 00:02:09,324 --> 00:02:10,254 In San Diego. 40 00:02:10,254 --> 00:02:14,544 And so we did have our special election and so I worked for 11 days, not 41 00:02:14,544 --> 00:02:16,224 including the setup and tear down day. 42 00:02:17,304 --> 00:02:18,234 So I've been a little busy. 43 00:02:18,429 --> 00:02:18,719 Okay. 44 00:02:18,804 --> 00:02:20,514 What, how many voters? 45 00:02:20,604 --> 00:02:22,974 Yeah, you've been a little busy and how many voters Yeah. 46 00:02:22,979 --> 00:02:23,119 Did you have 47 00:02:23,154 --> 00:02:28,224 Uh, we had like, like 75 over the first 10 days. 48 00:02:28,254 --> 00:02:30,354 And then on the last day we had about 400. 49 00:02:30,954 --> 00:02:31,554 Um. 50 00:02:32,039 --> 00:02:32,259 Oh 51 00:02:32,574 --> 00:02:34,224 Which was, which is, which is a lot. 52 00:02:34,314 --> 00:02:37,494 Um, and I, you know, I love, I love, I love democracy. 53 00:02:37,494 --> 00:02:38,064 I love people. 54 00:02:38,064 --> 00:02:41,154 I want everybody to, uh, to, to, to, to vote. 55 00:02:41,604 --> 00:02:44,664 Um, you know, if you don't vote, you don't get to bitch. 56 00:02:44,994 --> 00:02:45,594 That's my, 57 00:02:46,044 --> 00:02:47,064 But please don't wait 58 00:02:47,124 --> 00:02:50,964 but yeah, for the love of God, look into, look into your, your sight 59 00:02:50,964 --> 00:02:53,004 most, or your state most likely has. 60 00:02:53,724 --> 00:02:58,194 Early voting, look into early voting and early vote or vote by mail. 61 00:02:58,434 --> 00:02:58,884 Right? 62 00:02:58,974 --> 00:03:03,144 Um, those 400 people could have come any time in the previous 63 00:03:03,144 --> 00:03:05,634 10 days, and we would've, they could have voted just the same. 64 00:03:06,234 --> 00:03:07,164 Um, yeah. 65 00:03:07,254 --> 00:03:08,844 Anyway, so please vote. 66 00:03:08,994 --> 00:03:09,384 Um, 67 00:03:09,474 --> 00:03:10,344 Well, welcome back. 68 00:03:11,724 --> 00:03:12,084 Yeah. 69 00:03:12,084 --> 00:03:12,654 So, um. 70 00:03:14,859 --> 00:03:17,799 I wanted to, we're, we're gonna kind of, you know, we're kind of 71 00:03:18,399 --> 00:03:21,249 redoing things after a, you know, a couple different phases here. 72 00:03:21,279 --> 00:03:26,049 And, uh, we're gonna just try to do some, some hot topics 73 00:03:26,109 --> 00:03:29,259 that, um, I think are important. 74 00:03:29,379 --> 00:03:32,409 And one of them that we're gonna talk about this week is RTO. 75 00:03:32,559 --> 00:03:35,139 And specifically which recovery time objective. 76 00:03:35,139 --> 00:03:36,424 Of course, we're, we're gonna what? 77 00:03:37,509 --> 00:03:37,749 What? 78 00:03:38,319 --> 00:03:38,499 Yeah. 79 00:03:38,499 --> 00:03:39,279 Return to office. 80 00:03:39,279 --> 00:03:44,049 So we're gonna talk about return to office and then, um, and you know, what it is 81 00:03:44,589 --> 00:03:46,719 and why it is fantasy for most people. 82 00:03:47,049 --> 00:03:50,469 And then what, what they could do, um, you know, to, to address that. 83 00:03:51,399 --> 00:03:54,069 So first off, you want to define recovery time objective. 84 00:03:55,344 --> 00:03:58,224 Yeah, it's basically your objective, right? 85 00:03:58,224 --> 00:04:02,394 Your goal for how long it should take you to recover from some disaster, right? 86 00:04:02,394 --> 00:04:05,214 And get back to a good known spot. 87 00:04:05,244 --> 00:04:09,834 This is including things like recovering your data, reconfiguring 88 00:04:09,834 --> 00:04:12,474 your network, right, and different. 89 00:04:13,884 --> 00:04:17,814 disasters might have different recovery time objectives, so it's also important 90 00:04:17,814 --> 00:04:21,984 to remember, like recovering a file may be a lot less in terms of RTO 91 00:04:22,794 --> 00:04:25,014 say, recovering an entire data center. 92 00:04:25,014 --> 00:04:26,209 If it, uh, something 93 00:04:26,334 --> 00:04:30,114 Yeah, it's interesting you brought up that that's actually a hotly debated topic as 94 00:04:30,114 --> 00:04:32,454 to whether or not RTO should ever change. 95 00:04:32,874 --> 00:04:39,354 I agree with you that, um, the RTO is situational, situationally dependent, 96 00:04:39,834 --> 00:04:44,484 um, and that, you know, if, if you've been attacked by ransomware. 97 00:04:44,529 --> 00:04:47,919 For example, there's no way you're gonna meet sort of what 98 00:04:47,919 --> 00:04:49,869 I would call a normal RTO. 99 00:04:50,259 --> 00:04:54,459 Uh, and the same, you know, and the same with like, if it's a complete 100 00:04:54,459 --> 00:04:57,459 disaster that wipes out your entire data center and you have to physically 101 00:04:57,459 --> 00:05:01,329 build a building into which to put your servers or something like that, 102 00:05:01,779 --> 00:05:07,739 that RTO should be, um, larger than, you know, we lost a single server. 103 00:05:07,739 --> 00:05:10,409 Or like, you know, you said a lost a single file. 104 00:05:11,264 --> 00:05:15,884 Or if you go tell like your application admin who needs to recover data, oh, by 105 00:05:15,884 --> 00:05:20,504 the way, it's gonna take one week or two weeks to recover your data because that's 106 00:05:20,504 --> 00:05:22,724 the RTO you set for like site disasters. 107 00:05:22,724 --> 00:05:24,164 They're also probably gonna be unhappy, 108 00:05:24,209 --> 00:05:25,319 Yeah, absolutely. 109 00:05:25,484 --> 00:05:26,114 wait, wait, wait. 110 00:05:26,114 --> 00:05:26,474 That makes 111 00:05:26,549 --> 00:05:29,189 So that's another really important thing that you brought up right there, 112 00:05:29,189 --> 00:05:33,419 which is the, and, and this is a really important concept that goes 113 00:05:33,479 --> 00:05:39,029 through almost everything that we teach, and that is that you, meaning 114 00:05:39,029 --> 00:05:42,989 the backup admin, the sysadmin in charge of backups, whoever you happen 115 00:05:42,989 --> 00:05:45,839 to be, you do not determine the RTO. 116 00:05:46,319 --> 00:05:46,709 Right. 117 00:05:46,859 --> 00:05:51,329 The business unit determines the RTO or whatever, whatever term is appropriate at 118 00:05:51,329 --> 00:05:54,479 your governmental entity or NGO, right? 119 00:05:54,689 --> 00:06:00,569 Um, that is the, the entity that determines, uh, the the 120 00:06:00,569 --> 00:06:04,229 recovery time objective because it's based on finances, right? 121 00:06:04,289 --> 00:06:09,149 It's based on, uh, you know, like if it's a business, it's based on 122 00:06:09,179 --> 00:06:10,979 how much money are we going to lose. 123 00:06:11,399 --> 00:06:13,589 While we are down, right? 124 00:06:13,829 --> 00:06:17,129 Um, if it's a governmental organization, it's based, it, 125 00:06:17,129 --> 00:06:18,599 it's very different, right? 126 00:06:18,599 --> 00:06:23,009 It, it, it's more along the lines of how much damage to our organization 127 00:06:23,009 --> 00:06:26,249 like reputationally will happen based on how long we're down. 128 00:06:26,489 --> 00:06:31,559 And also how much more difficult will it be to redo the things that 129 00:06:31,559 --> 00:06:34,379 we, you know, to, to do the things we had to do while we were down. 130 00:06:34,679 --> 00:06:38,519 Uh, you might have to switch to, you know, to paper in the meantime. 131 00:06:39,659 --> 00:06:44,699 And, uh, so you, but, but the point is, all of these calculations are 132 00:06:44,699 --> 00:06:48,749 things that the, the business or management should be doing, not 133 00:06:49,109 --> 00:06:51,239 those, uh, in charge of backups. 134 00:06:51,539 --> 00:06:56,909 Uh, what role do you think the, the, the backup people play in determining, 135 00:06:56,969 --> 00:06:58,259 uh, recovery time objective? 136 00:06:59,729 --> 00:07:04,439 I think it is basically to figure out, okay, based on what the business has asked 137 00:07:04,439 --> 00:07:08,699 for, say they come back and say, okay, my Reto recovery time objective is one 138 00:07:08,699 --> 00:07:08,909 day. 139 00:07:09,869 --> 00:07:14,489 Based on that, here are some options that we can do technology wise, 140 00:07:14,489 --> 00:07:17,699 and I think their goal is to come back and say, okay, here's how much 141 00:07:17,699 --> 00:07:20,999 it will cost you if you want to support that recovery time objective 142 00:07:22,124 --> 00:07:22,604 Yeah. 143 00:07:22,604 --> 00:07:26,324 And, and you know, in the very beginning this is gonna be ballpark numbers, right? 144 00:07:26,384 --> 00:07:29,894 Um, well the first thing I would say is you come back and you go, okay, 145 00:07:29,894 --> 00:07:33,374 you've asked for A, we do B, right? 146 00:07:33,524 --> 00:07:35,114 We do a times four. 147 00:07:35,474 --> 00:07:36,794 Um, right. 148 00:07:37,064 --> 00:07:38,534 So, well, let let me ask you this. 149 00:07:38,534 --> 00:07:42,734 Why do you think, uh, I have, I have my opinions, uh, I'm 150 00:07:42,734 --> 00:07:44,564 curious, why do you think. 151 00:07:45,344 --> 00:07:50,714 Most organizations, if they have an RTO or even if it's poorly documented, 152 00:07:50,714 --> 00:07:51,584 et cetera, et cetera, et cetera. 153 00:07:51,584 --> 00:07:55,574 If we haven't agreed upon RTO, why are most organizations 154 00:07:55,574 --> 00:07:57,889 completely unable to meet that RTO? 155 00:07:59,909 --> 00:08:02,429 Well, the biggest thing is they probably haven't tested. 156 00:08:04,304 --> 00:08:07,424 To understand like is it actually like, and that's why I said like when you 157 00:08:07,424 --> 00:08:08,774 asked the definition of RTO, right? 158 00:08:09,014 --> 00:08:10,844 It's your desire, it's your objective. 159 00:08:10,844 --> 00:08:11,864 It doesn't mean what you will 160 00:08:11,864 --> 00:08:14,564 actually hit because there are so many other things involved. 161 00:08:14,564 --> 00:08:15,884 Like we talked about. 162 00:08:15,884 --> 00:08:19,094 Maybe part of your RTO is just bringing back the data or the 163 00:08:19,094 --> 00:08:20,604 application, but then what about. 164 00:08:20,954 --> 00:08:25,214 Like making sure I'm able to procure the servers to recover or get those 165 00:08:25,214 --> 00:08:25,964 up and running. 166 00:08:26,264 --> 00:08:26,564 Uh, 167 00:08:26,594 --> 00:08:27,044 maybe I, 168 00:08:27,044 --> 00:08:29,564 need to bring up active directory or Intra or whatever it's called 169 00:08:29,564 --> 00:08:30,614 now, whatever Microsoft calls 170 00:08:30,619 --> 00:08:33,469 it's, it was rebranded while we were on this recording. 171 00:08:34,019 --> 00:08:34,309 Yeah. 172 00:08:35,714 --> 00:08:35,924 right? 173 00:08:35,924 --> 00:08:39,164 But all of these other things, which maybe you don't necessarily have control 174 00:08:39,164 --> 00:08:42,884 of, and maybe you're only thinking as a backup admin of, Hey, I need to recover 175 00:08:42,884 --> 00:08:45,224 the application or just the data, restore 176 00:08:45,269 --> 00:08:48,659 Well, and also in addition, and again, let, let's make sure that 177 00:08:48,659 --> 00:08:54,929 we, we, we say that the recovery time objective has been met or not 178 00:08:54,929 --> 00:08:59,399 met when you, when the application. 179 00:09:00,224 --> 00:09:04,634 Is fully up and running and available for use by the user, right? 180 00:09:04,874 --> 00:09:06,674 It's not, oh, well I did my restore. 181 00:09:06,674 --> 00:09:11,384 We got a four hour R-T-O-I-I did my restore and it only took four hours. 182 00:09:11,684 --> 00:09:15,314 No, the question is, is the application back up and running? 183 00:09:15,644 --> 00:09:20,414 And that includes it, like I said, any hardware, uh, procurement, which, which 184 00:09:20,444 --> 00:09:21,854 hopefully you're doing in the cloud. 185 00:09:22,154 --> 00:09:23,984 But any hardware procurement, any. 186 00:09:24,644 --> 00:09:25,604 Stuff you gotta do. 187 00:09:25,604 --> 00:09:28,964 And if we're talking about ransomware, all of the, the, the stuff you gotta 188 00:09:28,964 --> 00:09:34,514 do to make sure that the, the server is ready to, uh, be restored and the 189 00:09:34,514 --> 00:09:38,684 restore, depending on what type of thing you're recovering from, the 190 00:09:38,684 --> 00:09:44,324 actual restore may be the smallest part of the, uh, recovery time. 191 00:09:44,324 --> 00:09:45,854 Actual is the term that we use. 192 00:09:46,664 --> 00:09:49,094 Based on your sort of consulting, 193 00:09:49,304 --> 00:09:49,664 Yeah. 194 00:09:49,664 --> 00:09:51,614 Mm-hmm. 195 00:09:51,644 --> 00:09:56,564 this right, what would you estimate is that ratio between time to 196 00:09:56,564 --> 00:10:00,074 actually restore the data or the application versus the end-to-end? 197 00:10:00,464 --> 00:10:01,094 RTO 198 00:10:01,874 --> 00:10:02,144 and I? 199 00:10:02,144 --> 00:10:02,354 I'm 200 00:10:02,579 --> 00:10:03,539 Yeah. 201 00:10:03,599 --> 00:10:08,459 If, if we're not talking about ransomware, it's like 80 20. 202 00:10:09,824 --> 00:10:10,274 Right. 203 00:10:10,514 --> 00:10:16,364 Uh, meaning 80% of the time spent doing the ba the Restore, the other 20% 204 00:10:16,604 --> 00:10:21,944 in a modern day scenario where we're probably, uh, gonna do this in the 205 00:10:21,944 --> 00:10:25,304 cloud so that we can, you know, snap our fingers and we have the hardware that 206 00:10:25,304 --> 00:10:27,674 we need, uh, we're doing the Restore. 207 00:10:28,034 --> 00:10:33,614 It's well, you know, well tested, uh, although often it is not right. 208 00:10:34,109 --> 00:10:39,299 And then there's some amount of time to do some initial functionality 209 00:10:39,299 --> 00:10:41,939 testing to make sure that all the dependencies have been met. 210 00:10:42,329 --> 00:10:45,149 And then, um, you know, and then we're, we're ready to roll. 211 00:10:45,209 --> 00:10:45,509 Right? 212 00:10:45,779 --> 00:10:49,289 So I'd say it's like 80 20 in a, in a ransomware scenario, 213 00:10:49,739 --> 00:10:52,919 it's like, you know, 10 90, 214 00:10:53,114 --> 00:10:53,414 Yeah, 215 00:10:53,579 --> 00:10:53,999 right? 216 00:10:54,284 --> 00:10:54,974 yeah, 217 00:10:54,989 --> 00:10:58,439 gonna spend most of your time making sure that you're recovering 218 00:10:58,439 --> 00:11:01,019 to a, uh, pristine environment. 219 00:11:01,719 --> 00:11:02,009 yeah. 220 00:11:02,324 --> 00:11:06,014 And I think that's one thing you just touched upon in your previous statement, 221 00:11:06,014 --> 00:11:08,649 which was like, sometimes things change. 222 00:11:10,589 --> 00:11:15,269 And we are talking and sort of like understanding why do most people's 223 00:11:15,269 --> 00:11:18,389 RTOs not meet what is expected? 224 00:11:18,389 --> 00:11:19,709 Do you wanna touch on some 225 00:11:19,724 --> 00:11:20,174 Well, 226 00:11:20,249 --> 00:11:20,489 I know 227 00:11:20,714 --> 00:11:26,744 yeah, so I'm gonna say the number one reason is that they simply don't have a 228 00:11:26,744 --> 00:11:31,184 backup or disaster recovery system that is capable of meeting that RTO just period. 229 00:11:31,404 --> 00:11:31,484 Hmm. 230 00:11:31,934 --> 00:11:34,664 They didn't do, uh, and, and this is. 231 00:11:34,914 --> 00:11:39,264 This is quite possibly they, I, I remember when, when I, you know, 232 00:11:39,414 --> 00:11:42,324 back, go back 30 years, right? 233 00:11:42,534 --> 00:11:47,874 That, that we knew we abs everyone knew that our backup system wasn't 234 00:11:47,874 --> 00:11:51,384 anywhere near capable of meeting the RTOs that we had discussed. 235 00:11:51,384 --> 00:11:53,934 Even though we didn't use that term back then, I, I'm sure the term was 236 00:11:53,934 --> 00:11:57,054 available, but I didn't use it and the. 237 00:11:57,764 --> 00:12:00,284 We just knew that it, it just wasn't, wasn't possible. 238 00:12:00,314 --> 00:12:00,704 Right. 239 00:12:01,004 --> 00:12:05,294 Um, I mean, in some cases it was laughably impossible, right? 240 00:12:05,354 --> 00:12:09,464 Um, and that we had servers that it took us a week. 241 00:12:09,689 --> 00:12:12,869 It took, took us a week to get a full backup. 242 00:12:12,929 --> 00:12:13,319 Okay. 243 00:12:13,469 --> 00:12:17,069 Like, how are we gonna meet a four hour R-P-R-T-O if it takes 244 00:12:17,069 --> 00:12:18,329 a week to do a full backup? 245 00:12:18,389 --> 00:12:18,809 Right? 246 00:12:19,379 --> 00:12:21,389 Uh, and by the way, our next episode we're gonna talk about 247 00:12:21,389 --> 00:12:23,069 RPO recovery point objective. 248 00:12:23,069 --> 00:12:26,969 So it's a, it's very much a sister episode to this, but that, I'd say 249 00:12:26,969 --> 00:12:30,389 that's the number one reason is that people's backup systems, and, and again, 250 00:12:30,389 --> 00:12:32,309 I used the term backup very broadly. 251 00:12:32,339 --> 00:12:32,939 Anything that. 252 00:12:33,189 --> 00:12:35,769 That brings the server back to the way it looked, you know, before the 253 00:12:35,769 --> 00:12:37,509 disaster is a backup system to me. 254 00:12:37,509 --> 00:12:37,719 There. 255 00:12:37,899 --> 00:12:40,569 Well, you just use 'em for different purposes, disaster recovery, et cetera. 256 00:12:41,319 --> 00:12:42,669 Um, but that's the number one reason. 257 00:12:42,669 --> 00:12:46,179 The other reason that's very closely related to that is 258 00:12:46,179 --> 00:12:47,949 that they have no idea, right? 259 00:12:47,949 --> 00:12:49,959 They, they haven't tested right. 260 00:12:50,169 --> 00:12:52,074 They, they've got a system, they've got a clue. 261 00:12:52,599 --> 00:12:53,049 Right. 262 00:12:53,379 --> 00:12:56,709 And they're like, oh, well it takes us, you know, three hours 263 00:12:56,709 --> 00:12:58,449 to, or four hours to back up. 264 00:12:58,839 --> 00:13:02,979 Therefore, we should be able to do a four hour restore. 265 00:13:03,639 --> 00:13:06,549 There's a lot of ifs in that, right? 266 00:13:06,684 --> 00:13:06,974 Yeah. 267 00:13:07,149 --> 00:13:12,579 the, the other thing is a, as you know, restores often a 268 00:13:12,579 --> 00:13:15,129 lot slower than backup, right? 269 00:13:15,129 --> 00:13:19,269 For a number of reasons that, you know, are all over the 270 00:13:19,269 --> 00:13:20,349 place that, that they, they 271 00:13:21,074 --> 00:13:21,194 the 272 00:13:21,279 --> 00:13:21,519 ahead. 273 00:13:21,644 --> 00:13:22,664 is like incremental 274 00:13:22,754 --> 00:13:23,144 Yeah. 275 00:13:23,204 --> 00:13:23,624 Yeah. 276 00:13:23,744 --> 00:13:24,644 Forever incremental. 277 00:13:24,914 --> 00:13:25,154 Right. 278 00:13:25,154 --> 00:13:26,469 That I would probably say is the biggest 279 00:13:26,639 --> 00:13:26,999 Yeah. 280 00:13:26,999 --> 00:13:30,509 That you're, that you're piecing together a restore from many, many, many stuff. 281 00:13:30,869 --> 00:13:34,979 Uh, I, I think if you're, if you, if you have a proper design that alone 282 00:13:34,979 --> 00:13:37,559 shouldn't, um, you know, impact you. 283 00:13:37,889 --> 00:13:43,079 If you are doing a, a, you know, sort of the old school full restore, followed 284 00:13:43,079 --> 00:13:46,619 by each incremental restore, and that means you're actually restoring some 285 00:13:46,619 --> 00:13:48,989 files multiple times, then Absolutely. 286 00:13:49,109 --> 00:13:49,439 Right. 287 00:13:49,889 --> 00:13:53,604 If you're doing, if you, if you have a, if you have a. A system that is 288 00:13:53,604 --> 00:13:57,174 properly de uh, developed, right? 289 00:13:57,414 --> 00:14:01,494 That fixes that issue where if we know a file has changed, then we're not 290 00:14:01,494 --> 00:14:03,624 gonna restore that file multiple times. 291 00:14:03,624 --> 00:14:05,544 We're just gonna restore the latest version of the file. 292 00:14:06,054 --> 00:14:07,674 If you have that, that's not really the problem. 293 00:14:07,764 --> 00:14:10,824 But you do have the issue of ddu, right? 294 00:14:10,824 --> 00:14:15,414 You have the, the DDU tax that quite often really rears its ugly 295 00:14:15,414 --> 00:14:16,884 head when we go to do a restore. 296 00:14:17,214 --> 00:14:17,994 Now why would that be? 297 00:14:17,994 --> 00:14:19,364 Why would that be the case Prasanna? 298 00:14:20,774 --> 00:14:26,024 Because when you're dup Deduplicating data, throwing away a whole 299 00:14:26,069 --> 00:14:26,489 Mm-hmm. 300 00:14:26,984 --> 00:14:30,644 But the problem is when you need to read it, you're basically doing random reads 301 00:14:30,644 --> 00:14:35,414 across the entire system in order to be able to recreate that single file. 302 00:14:35,414 --> 00:14:38,624 Because you might have old blocks from one part of the disc and a 303 00:14:38,624 --> 00:14:40,904 different part of the file from a different part of the disc. 304 00:14:41,114 --> 00:14:45,524 And so you end up with all these random reads, which as we know, our disc 305 00:14:45,524 --> 00:14:47,744 drives are not very good at doing random 306 00:14:47,819 --> 00:14:51,989 Yeah, it, it's the, it's the ultimate fragmented file system, right? 307 00:14:52,439 --> 00:14:55,679 Uh, you are just, you're absolutely guaranteeing that everything 308 00:14:55,679 --> 00:14:57,779 you need is everywhere, right? 309 00:14:58,349 --> 00:15:03,299 Um, and, um, the, that, that is absolutely one of the cases. 310 00:15:03,539 --> 00:15:04,529 And, and if we're coming from tape. 311 00:15:05,099 --> 00:15:05,489 Right. 312 00:15:05,549 --> 00:15:08,129 Uh, which is probably less likely for most people. 313 00:15:08,279 --> 00:15:11,489 But if we're coming from tape, then we really do start talking about that, 314 00:15:11,494 --> 00:15:15,599 the, the, the forever incremental stuff and, uh, you know, because you're having 315 00:15:15,599 --> 00:15:19,589 to load all these tapes, uh, but also there's a network can get in the way. 316 00:15:19,739 --> 00:15:21,629 There's also, depending on what. 317 00:15:22,349 --> 00:15:25,109 Raid, uh, we're using, right? 318 00:15:25,139 --> 00:15:27,389 If we're using RAID and, and we're using raid, right? 319 00:15:27,419 --> 00:15:28,274 Everybody's using raid. 320 00:15:28,509 --> 00:15:28,799 Yeah, 321 00:15:29,729 --> 00:15:32,879 depending on whether or not you opted for, does anybody 322 00:15:32,879 --> 00:15:34,769 wrap up for RAID 10 these days? 323 00:15:35,519 --> 00:15:36,119 I don't know. 324 00:15:36,409 --> 00:15:36,529 I 325 00:15:36,629 --> 00:15:37,229 I don't think so. 326 00:15:37,229 --> 00:15:38,789 I think everybody does raid six, right? 327 00:15:38,789 --> 00:15:39,839 Or, or, or something. 328 00:15:39,869 --> 00:15:44,129 So raid, dual parody or whatever, and that has a right penalty, right? 329 00:15:44,129 --> 00:15:48,599 So for a number of reasons, restores are often slower than 330 00:15:48,599 --> 00:15:51,179 backup and you will never know. 331 00:15:51,599 --> 00:15:54,379 Until you do what Prasanna. 332 00:15:54,839 --> 00:15:54,959 You 333 00:15:55,079 --> 00:15:55,859 Exactly. 334 00:15:56,249 --> 00:15:57,359 And again, go ahead. 335 00:15:57,689 --> 00:15:59,309 Well, I'm gonna bring, I'm gonna bring out a story. 336 00:15:59,309 --> 00:15:59,699 Sorry. 337 00:16:00,059 --> 00:16:00,689 Uh, going, 338 00:16:00,839 --> 00:16:01,409 okay. 339 00:16:01,739 --> 00:16:06,059 going back to going back to my first, you know, the first time that things were 340 00:16:06,059 --> 00:16:10,859 really, really bad was that time when we had a new backup system and we had 341 00:16:10,859 --> 00:16:14,759 used a compression feature on, on the way in, and it was software compression. 342 00:16:15,089 --> 00:16:19,229 And long story short, when we went to, uh, there, there was a. There 343 00:16:19,229 --> 00:16:23,699 was the, um, we went to do the first major restore after we needed it. 344 00:16:23,699 --> 00:16:23,999 Right? 345 00:16:23,999 --> 00:16:26,879 We, we didn't test restores, we only tested backups. 346 00:16:27,329 --> 00:16:32,399 And uh, when we went to do it, uh, the, it was a DD, s and it was like, 347 00:16:32,699 --> 00:16:35,759 blink, blink, long pause, right? 348 00:16:36,749 --> 00:16:37,019 Blink. 349 00:16:37,019 --> 00:16:37,379 Blink. 350 00:16:37,889 --> 00:16:41,999 And once we called into support and they were like, yeah, it's working as design. 351 00:16:42,149 --> 00:16:43,174 And basically we had not. 352 00:16:44,549 --> 00:16:46,049 Tested this at all. 353 00:16:46,109 --> 00:16:49,379 And not only was it slow, it wouldn't work. 354 00:16:49,424 --> 00:16:53,249 It, it was just literally without going, without taking too much time, 355 00:16:53,399 --> 00:16:55,889 it just literally wouldn't work. 356 00:16:56,159 --> 00:16:56,579 Right. 357 00:16:56,819 --> 00:17:00,989 And, uh, unless we like tripled the size of ram or something. 358 00:17:00,989 --> 00:17:01,379 Right. 359 00:17:01,709 --> 00:17:07,079 And, um, so yeah, you, you just do not know how your system is going to 360 00:17:07,079 --> 00:17:09,539 perform until you go to do a restore. 361 00:17:10,799 --> 00:17:11,089 Yeah. 362 00:17:11,504 --> 00:17:11,774 And. 363 00:17:12,329 --> 00:17:17,789 One thing similar to that story is you should also do a realistic restore test. 364 00:17:17,969 --> 00:17:20,339 Don't just be like, oh, I'm gonna just restore a file, or 365 00:17:20,339 --> 00:17:21,479 I'm just gonna restore a vm. 366 00:17:21,479 --> 00:17:21,809 I'm good 367 00:17:21,834 --> 00:17:22,124 Yeah. 368 00:17:22,949 --> 00:17:23,249 Right? 369 00:17:23,249 --> 00:17:26,639 Because that may not be a realistic scenario for when you have to 370 00:17:26,639 --> 00:17:30,479 recover a full application suite or your entire environment. 371 00:17:30,509 --> 00:17:32,819 So make sure you're doing the right little type of 372 00:17:32,894 --> 00:17:34,334 Yeah, absolutely. 373 00:17:34,454 --> 00:17:34,874 It should, 374 00:17:35,099 --> 00:17:35,309 any 375 00:17:35,324 --> 00:17:37,484 it should be whatever the, whatever the thing is that we're 376 00:17:37,484 --> 00:17:38,864 setting the RTO for, right? 377 00:17:39,164 --> 00:17:41,624 Uh, you don't have restore the entire environment, but you need to do 378 00:17:41,624 --> 00:17:43,814 representative restore tests, right? 379 00:17:44,234 --> 00:17:48,584 Um, entire servers, entire environments, entire recovery groups. 380 00:17:48,584 --> 00:17:50,004 What's a recovery group Prasanna? 381 00:17:51,494 --> 00:17:55,904 It is a group of things you need to restore in order for 382 00:17:55,904 --> 00:17:57,014 your application to come back. 383 00:17:57,014 --> 00:18:01,214 So it might be your database server plus your storage, plus your active 384 00:18:01,214 --> 00:18:03,884 directory or uh, system, right? 385 00:18:03,884 --> 00:18:08,144 Plus whatever else is needed in order to get that production application back 386 00:18:08,474 --> 00:18:08,964 exactly. 387 00:18:09,929 --> 00:18:11,879 And so, and, and so that becomes important too. 388 00:18:11,879 --> 00:18:14,309 I was actually gonna comment on that, because there's an order 389 00:18:14,309 --> 00:18:16,889 of operations you have to do, and so you have to account for that. 390 00:18:17,219 --> 00:18:22,049 When you calculate your RTO, it's not like, oh, I can just restore my, uh, 391 00:18:22,079 --> 00:18:25,019 database, have it up and running before I have active directory up and running. 392 00:18:25,049 --> 00:18:25,439 It's not 393 00:18:25,469 --> 00:18:25,859 Right. 394 00:18:25,864 --> 00:18:29,099 And, and by the way, that one, one of the things that prompted this episode, 395 00:18:29,129 --> 00:18:34,559 uh, Kaseya did a 2025 state of the backup industry, uh, and they said that more 396 00:18:34,559 --> 00:18:38,939 than 60% of respondents believed that they could recover under, in, under a day. 397 00:18:39,179 --> 00:18:45,329 However, only 35% could actually do that in reality, which is, that's 398 00:18:45,329 --> 00:18:47,159 quite, that's quite a, a gap there. 399 00:18:48,449 --> 00:18:49,139 Um. 400 00:18:49,484 --> 00:18:49,934 Yeah. 401 00:18:49,964 --> 00:18:53,024 Another interesting thing was that only 10% of businesses reported 402 00:18:53,024 --> 00:18:55,034 no outages in the last 12 months. 403 00:18:55,514 --> 00:18:59,534 Uh, which means that 90% tested their backup systems the hard way. 404 00:18:59,924 --> 00:19:05,084 Uh, not quite as hard as, uh, our Alaskan friend, but, uh, which for those of 405 00:19:05,084 --> 00:19:10,514 you that haven't heard that episode, uh, he tested DR system by deleting 406 00:19:10,514 --> 00:19:12,614 the entire surfer for the entire. 407 00:19:13,154 --> 00:19:16,004 Data center and then restoring, and that was his first test. 408 00:19:16,004 --> 00:19:17,624 And it was like, gee, I hope it works. 409 00:19:17,804 --> 00:19:18,764 Don't do it like that. 410 00:19:19,454 --> 00:19:20,114 Um, 411 00:19:20,779 --> 00:19:21,679 but hey, it worked out 412 00:19:21,764 --> 00:19:23,414 yeah, exactly right. 413 00:19:23,504 --> 00:19:29,024 Um, and remember, again, going back to the things that fit into the RTO, right? 414 00:19:29,234 --> 00:19:32,624 You know, you, you also have to, to include things like 415 00:19:32,624 --> 00:19:34,094 detecting that there's a problem. 416 00:19:34,484 --> 00:19:39,194 Right, because the RTO clock starts the moment the outage happens, not 417 00:19:39,194 --> 00:19:40,904 the moment the restore happens, right? 418 00:19:41,294 --> 00:19:46,244 So, uh, the moment you have the outage and then you're like, what's going on? 419 00:19:46,604 --> 00:19:46,934 Right? 420 00:19:46,934 --> 00:19:52,574 Because so many times the, the symptom that gets your attention has nothing to 421 00:19:52,574 --> 00:19:54,134 do with the thing that actually went bad. 422 00:19:54,509 --> 00:19:54,899 Right. 423 00:19:55,259 --> 00:19:57,749 Uh, I mean, it does have something to do with it, but it's not 424 00:19:57,749 --> 00:19:59,009 the thing that went bad, right? 425 00:19:59,249 --> 00:20:00,479 So you gotta figure that out. 426 00:20:00,479 --> 00:20:03,419 You gotta understand how bad it is if it's a ransomware attack. 427 00:20:03,419 --> 00:20:06,869 Again, you gotta figure out, you know, how bad this, you know, how big the scope is. 428 00:20:07,169 --> 00:20:11,969 You might have to get approvals, um, you know, all these different things, right? 429 00:20:13,244 --> 00:20:13,484 Yeah. 430 00:20:13,694 --> 00:20:16,034 And well, just one thing to add to that, because I was thinking 431 00:20:16,034 --> 00:20:17,624 about the, uh, what was the. 432 00:20:18,284 --> 00:20:18,764 Company, 433 00:20:18,794 --> 00:20:19,454 Rackspace. 434 00:20:20,474 --> 00:20:21,914 with their hosted exchange. 435 00:20:22,154 --> 00:20:22,364 Right. 436 00:20:22,364 --> 00:20:24,704 I think one of the things to also consider. 437 00:20:25,139 --> 00:20:29,669 Uh, when you're thinking about RTO is order to bring my app back up and 438 00:20:29,669 --> 00:20:31,619 running, do I need to restore all my data? 439 00:20:32,489 --> 00:20:32,999 an example? 440 00:20:32,999 --> 00:20:36,269 Maybe I only need a subset of my data in order for my application to come up 441 00:20:36,269 --> 00:20:40,679 and I can solely backfill all my old data that's archived or other things 442 00:20:40,679 --> 00:20:45,239 like that, I can still get people up and running and ready to go without 443 00:20:45,299 --> 00:20:46,709 waiting for everything to be done. 444 00:20:47,159 --> 00:20:51,059 And so there might also be slight nuances depending on the application 445 00:20:51,059 --> 00:20:52,529 of what the expectations are. 446 00:20:52,529 --> 00:20:53,009 Yeah. 447 00:20:53,009 --> 00:20:56,189 The other thing I would say regarding that Rackspace outage, if you're in 448 00:20:56,189 --> 00:20:59,489 the middle of your recovery or you're about to begin your recovery, don't 449 00:20:59,489 --> 00:21:03,839 change all the rules, right In, in their case, they're like, we tested how to 450 00:21:03,839 --> 00:21:07,469 do this recovery, but you know, just before they went to do the recovery, 451 00:21:07,469 --> 00:21:10,949 they're like, ah, what if we just move everything over to Microsoft 365? 452 00:21:11,519 --> 00:21:11,909 Right? 453 00:21:12,059 --> 00:21:14,579 And it's like, oh, well that would mean that we have to like. 454 00:21:15,089 --> 00:21:18,839 Basically you, you can't, you can no longer restore the exchange 455 00:21:18,839 --> 00:21:21,209 databases directly into the user. 456 00:21:21,869 --> 00:21:25,949 Uh, you have to, um, you'll have to restore it and then migrate 457 00:21:25,949 --> 00:21:29,429 the data over individually, which is a much bigger process. 458 00:21:29,849 --> 00:21:33,719 Much, it's gonna just take much, much longer, and it ended up taking months. 459 00:21:34,529 --> 00:21:38,519 You may recall, and there was a, uh, some lawsuits regarding that. 460 00:21:38,759 --> 00:21:44,399 So make sure that whatever scenario in which you do, uh, you do the testing, 461 00:21:44,399 --> 00:21:46,019 you, you, you have to do the testing. 462 00:21:46,499 --> 00:21:47,549 So, um, 463 00:21:47,549 --> 00:21:47,789 Yep. 464 00:21:48,419 --> 00:21:52,379 I know we talked earlier about, okay, that 24 hour RTO for some businesses, 465 00:21:52,739 --> 00:21:54,749 but there are some industries, right? 466 00:21:54,749 --> 00:21:57,899 Where even like seconds make a big difference, right? 467 00:21:58,214 --> 00:21:58,994 Yeah, definitely. 468 00:22:00,044 --> 00:22:05,674 Yeah, definitely like financial trading firms, banking organizations, the more you 469 00:22:05,674 --> 00:22:11,584 can attach a real number when you can say one hour of downtime costs us this much. 470 00:22:11,919 --> 00:22:14,284 If, if you can do that, if the business can do that. 471 00:22:14,734 --> 00:22:16,264 One, $1 billion. 472 00:22:16,804 --> 00:22:19,984 Um, yeah, I'm sorry, I, I gotta do the, the pinky, right? 473 00:22:20,254 --> 00:22:25,264 Um, if you can do that, the more you can do that, the, the, the, the 474 00:22:25,264 --> 00:22:30,604 much more equipped you will be as a, you know, backup and dr. Person. 475 00:22:30,999 --> 00:22:34,209 To be able to make enhancements to the backup and recovery system if needed. 476 00:22:34,209 --> 00:22:34,569 Right. 477 00:22:35,289 --> 00:22:37,269 So let's talk about, uh, some of the things that you 478 00:22:37,269 --> 00:22:38,919 could do to close this gap. 479 00:22:38,949 --> 00:22:43,179 Obviously, the first, the first thing is if you can have an 480 00:22:43,179 --> 00:22:46,719 iterative discussion on, uh, okay. 481 00:22:46,719 --> 00:22:50,709 You said you want one minute, we can do 10 hours. 482 00:22:51,309 --> 00:22:51,999 Right. 483 00:22:52,059 --> 00:23:00,249 Let's figure out, you know, let's get the, let's get the RTO set to somewhere near. 484 00:23:00,769 --> 00:23:06,949 Um, you know, uh, realistic that we, that we can actually meet, right? 485 00:23:07,009 --> 00:23:11,179 And you can, you can say, we're gonna set the RTO for now at this. 486 00:23:11,179 --> 00:23:16,159 We're gonna move towards, uh, a better RTO at a later, a later time. 487 00:23:16,699 --> 00:23:18,314 Um, any thoughts there? 488 00:23:20,224 --> 00:23:26,194 Yeah, no, I think that makes sense because it also takes time to implement 489 00:23:26,194 --> 00:23:30,154 new technologies because if, say for instance, your RTO is 10 hours based 490 00:23:30,154 --> 00:23:32,974 on your existing infrastructure, and now they're like, oh, we need 491 00:23:33,244 --> 00:23:35,224 an hour or 10 minutes, right? 492 00:23:35,224 --> 00:23:37,834 You're now going to need to think of something very different that's 493 00:23:37,834 --> 00:23:42,004 gonna elongate the time it takes, and so it really is important to 494 00:23:42,004 --> 00:23:43,204 ask the question, do you need. 495 00:23:45,169 --> 00:23:45,709 Yeah. 496 00:23:45,799 --> 00:23:46,039 Yeah. 497 00:23:46,039 --> 00:23:47,029 The help, just, 498 00:23:47,089 --> 00:23:47,484 start with 499 00:23:47,509 --> 00:23:51,679 yeah, everybody's gonna say zero and zero for your RTO and RPO, right? 500 00:23:51,949 --> 00:23:55,039 So it is just, you gotta justify it and you gotta say, well, if it's 501 00:23:55,039 --> 00:23:57,019 really worth $10 million every minute. 502 00:23:57,529 --> 00:24:00,589 Then you need to give us, you know, whatever the number is. 503 00:24:00,589 --> 00:24:00,889 Right. 504 00:24:01,339 --> 00:24:06,199 Um, so then if we're gonna do testing, if we can automate that testing, the more we 505 00:24:06,199 --> 00:24:09,949 can automate testing, the more the, you know, the better that things are gonna be. 506 00:24:10,279 --> 00:24:13,309 Doing it very regularly, doing it small, it's sort of like, uh, the 507 00:24:13,309 --> 00:24:18,699 same as opinions that I have on testing your, it's kind of like in 508 00:24:18,699 --> 00:24:25,164 cybersecurity where you have a company that actively tries to send phishing. 509 00:24:25,899 --> 00:24:30,879 You know, phishing tests to the users to see if, um, to 510 00:24:30,879 --> 00:24:32,259 see if they fall for it, right? 511 00:24:32,709 --> 00:24:41,169 Same thing here, where over there, more frequent, smaller bite-sized testing is 512 00:24:41,169 --> 00:24:45,309 preferred to the once a year I have to do this and it takes two hours, right? 513 00:24:45,489 --> 00:24:48,999 Keeping it on the mind, keeping a recovery mindset is really important. 514 00:24:49,899 --> 00:24:51,969 So I think regular Dr. Drills is part of that. 515 00:24:53,439 --> 00:24:57,189 And I think having the regular DR drills is important because if something 516 00:24:57,189 --> 00:24:58,809 changes in your environment, you 517 00:24:59,094 --> 00:24:59,454 Yeah, 518 00:25:00,249 --> 00:25:01,389 rather than sort of that 519 00:25:01,584 --> 00:25:02,244 exactly. 520 00:25:02,514 --> 00:25:07,164 And then there's also the concept of chaos engineering, um, whi, which, you 521 00:25:07,164 --> 00:25:08,844 know, like the chaos monkey, right? 522 00:25:08,844 --> 00:25:09,329 You wanna talk about that? 523 00:25:10,359 --> 00:25:10,539 Yep. 524 00:25:10,989 --> 00:25:11,529 Yep, yep, yep. 525 00:25:11,739 --> 00:25:11,979 Yeah. 526 00:25:11,979 --> 00:25:12,399 So. 527 00:25:13,659 --> 00:25:18,309 just try breaking things in your environment, see what happens and 528 00:25:18,309 --> 00:25:21,579 see did I miss something that I wasn't backing up as an example. 529 00:25:21,909 --> 00:25:27,099 Maybe you forgot a backup active directory, and now in order and something 530 00:25:27,099 --> 00:25:30,579 happened to it, you lost all the data there and you realize, oh, I can't recover 531 00:25:30,579 --> 00:25:34,329 my application because I don't actually have a backup of active directory. 532 00:25:34,329 --> 00:25:37,089 And so you start to understand the dependencies in your 533 00:25:37,089 --> 00:25:39,039 environment and point out sort of. 534 00:25:39,609 --> 00:25:44,169 Issues that you might not foresee different failure scenarios or like if 535 00:25:44,169 --> 00:25:47,709 the network goes down or The other thing is, it's not even just technology, right? 536 00:25:47,709 --> 00:25:48,729 It might be even a person. 537 00:25:48,729 --> 00:25:51,729 I know Curtis, you used to talk about at the bank doing 538 00:25:51,729 --> 00:25:53,889 testing with someone who did not 539 00:25:53,964 --> 00:25:54,984 Yeah, exactly. 540 00:25:54,984 --> 00:25:55,674 Exactly. 541 00:25:56,124 --> 00:25:58,374 Uh, well back then, I didn't write the book, but Yeah. 542 00:25:58,764 --> 00:25:59,004 Yeah. 543 00:25:59,994 --> 00:26:01,494 I wa I wasn't Mr. Backup yet. 544 00:26:01,854 --> 00:26:04,014 I wa I was, I was Mr. Backup junior. 545 00:26:04,374 --> 00:26:08,124 Um, and, and then, you know, the idea is if, if you do this, the whole, the 546 00:26:08,124 --> 00:26:12,294 whole idea of doing this on a frequent basis to get better at it, to create 547 00:26:12,294 --> 00:26:16,554 and, and, and improve your runbooks, to create and improve decision trees. 548 00:26:17,074 --> 00:26:18,739 What do we do when this happens? 549 00:26:18,739 --> 00:26:19,189 Right? 550 00:26:19,549 --> 00:26:24,769 Um, we also, we didn't talk, uh, at all about, um, uh, tabletop exercises. 551 00:26:24,769 --> 00:26:28,069 Those are, uh, obviously a great, uh, uh, tool here. 552 00:26:28,579 --> 00:26:29,809 Uh, you know, do 'em at lunch. 553 00:26:29,809 --> 00:26:33,469 Do 'em so that they're not like, so again, do them frequently in smaller. 554 00:26:33,619 --> 00:26:37,189 We're not where the whole world isn't, you know, don't do 'em like just before 555 00:26:37,189 --> 00:26:39,109 your, your performance review time. 556 00:26:39,629 --> 00:26:41,939 Which makes them like much more stressful. 557 00:26:42,089 --> 00:26:44,819 Do them frequently and, and, and have fun at it. 558 00:26:45,209 --> 00:26:47,609 And, and then learn from it and improve your runbooks, 559 00:26:47,609 --> 00:26:48,929 improve your decision trees. 560 00:26:49,289 --> 00:26:50,489 Cross train your teams. 561 00:26:50,489 --> 00:26:51,599 Do what we were talking about before. 562 00:26:51,599 --> 00:26:53,879 Don't use the, don't rely on one person. 563 00:26:54,179 --> 00:26:58,199 Uh, you know, you know, because that one person might not be available. 564 00:26:58,559 --> 00:26:59,339 Uh. 565 00:26:59,754 --> 00:27:01,764 You know, uh, at that time, right? 566 00:27:01,824 --> 00:27:06,444 And then measure, again, measure and report reality. 567 00:27:06,474 --> 00:27:08,034 Here's where we are. 568 00:27:08,304 --> 00:27:11,154 Make sure that everyone is on the same page. 569 00:27:11,364 --> 00:27:15,714 We've asked for this, we've agreed to this for now, we would like to get to here. 570 00:27:15,894 --> 00:27:17,094 Here's where we are. 571 00:27:17,394 --> 00:27:18,714 Report those gaps. 572 00:27:19,044 --> 00:27:23,724 Uh, and then let, let business leadership decide. 573 00:27:24,369 --> 00:27:25,479 What to do about that. 574 00:27:25,479 --> 00:27:27,099 It is not your responsibility. 575 00:27:27,399 --> 00:27:27,669 Right. 576 00:27:27,669 --> 00:27:28,779 I do remember like 577 00:27:28,884 --> 00:27:29,064 Yep. 578 00:27:29,529 --> 00:27:33,789 bad because the backup system wasn't capable, but it's like, I'm not magic. 579 00:27:33,909 --> 00:27:36,219 All I can do is make recommendations. 580 00:27:36,219 --> 00:27:36,579 Right. 581 00:27:36,699 --> 00:27:42,549 And I do remember, by the way, I do remember when I, and I had a shell 582 00:27:42,549 --> 00:27:44,079 script that was doing everything right. 583 00:27:44,079 --> 00:27:47,679 We had like, I dunno, like 50 servers and I was doing all this with like a, a Unix. 584 00:27:48,654 --> 00:27:49,854 You know, shell script. 585 00:27:49,854 --> 00:27:50,214 Right. 586 00:27:50,664 --> 00:27:54,804 And at some point I couldn't, but, and all of it was based on that each 587 00:27:54,804 --> 00:27:56,244 server could fit on a tape drive. 588 00:27:56,244 --> 00:27:59,124 And then one day we bought a server that it didn't fit on. 589 00:27:59,124 --> 00:28:00,594 50 tape drives, right? 590 00:28:00,594 --> 00:28:01,524 On 50 tapes. 591 00:28:01,944 --> 00:28:04,644 And, and it, it just, that and other servers that weren't 592 00:28:04,644 --> 00:28:06,204 quite that bad, it just broke. 593 00:28:07,059 --> 00:28:08,889 It broke my ability to do it right. 594 00:28:08,889 --> 00:28:10,899 And I said, I'm just not, I can't do that. 595 00:28:10,929 --> 00:28:14,829 And then, and I just went to the boss and I said, Hey, I can't do this. 596 00:28:15,189 --> 00:28:20,289 And she said, well, aren't there like commercial products that do this, 597 00:28:20,289 --> 00:28:21,729 that we can like spend money on? 598 00:28:21,864 --> 00:28:23,244 Oh, because you flipping 599 00:28:23,384 --> 00:28:26,864 I was, I was flipping, I was flipping out what I was doing, 600 00:28:27,344 --> 00:28:33,104 and I, I remember feeling like a failure because I couldn't fix this. 601 00:28:33,524 --> 00:28:33,854 Right. 602 00:28:33,854 --> 00:28:35,654 I wasn't that good at scripting. 603 00:28:35,849 --> 00:28:39,464 I, I, I don't think anybody could deal with the 50, you know, the right. 604 00:28:39,674 --> 00:28:43,424 But I, I remember at the time feeling like a failure, and I guess I'm 605 00:28:43,424 --> 00:28:45,704 saying don't try not to feel that way. 606 00:28:46,004 --> 00:28:46,424 Right? 607 00:28:46,994 --> 00:28:49,569 Go and give an honest assessment of where you're at. 608 00:28:50,304 --> 00:28:55,494 And, um, even if you're the one that put you in that scenario, right. 609 00:28:55,794 --> 00:29:00,504 Um, I, I remember another story of a guy that told me that he bought a 610 00:29:00,504 --> 00:29:06,504 particular vendor's DDU product that had a 90% DDU tax, meaning that the restore 611 00:29:06,504 --> 00:29:09,024 speed was 10% of the backup speed. 612 00:29:09,684 --> 00:29:12,174 And, and he's like, I don't know what to do. 613 00:29:12,174 --> 00:29:13,944 I'm like, well, you have to tell your boss. 614 00:29:13,944 --> 00:29:16,314 And he's like, I'm the one that recommended the system. 615 00:29:18,649 --> 00:29:19,284 It's okay. 616 00:29:19,314 --> 00:29:20,394 You gotta be honest. 617 00:29:20,394 --> 00:29:22,794 You get, because you can't get there from, you can't get there from here 618 00:29:22,794 --> 00:29:24,474 if you, if you don't address that. 619 00:29:24,479 --> 00:29:24,669 Right? 620 00:29:25,299 --> 00:29:27,369 Yeah, so one question I wanted 621 00:29:27,419 --> 00:29:27,709 Yeah. 622 00:29:27,729 --> 00:29:28,209 Curtis. 623 00:29:28,449 --> 00:29:29,889 There is a term though, right? 624 00:29:29,889 --> 00:29:30,189 So you 625 00:29:30,329 --> 00:29:30,749 Mm-hmm. 626 00:29:31,809 --> 00:29:33,909 and then you have like, okay, you're doing these tests, you 627 00:29:33,909 --> 00:29:35,079 actually figure out like, okay, 628 00:29:35,139 --> 00:29:35,429 Yeah. 629 00:29:35,499 --> 00:29:36,159 long it takes. 630 00:29:36,519 --> 00:29:38,199 There's a term for that though, and I don't think it's 631 00:29:38,649 --> 00:29:39,729 No, it's nowhere near. 632 00:29:39,789 --> 00:29:40,239 Yeah, 633 00:29:40,239 --> 00:29:40,419 it's 634 00:29:40,719 --> 00:29:41,979 nowhere near as As. 635 00:29:42,039 --> 00:29:42,939 Yeah, thanks. 636 00:29:42,969 --> 00:29:46,179 Nowhere near as widely used as RTO. 637 00:29:46,179 --> 00:29:46,509 Right? 638 00:29:46,509 --> 00:29:49,209 And it's RTA recovery time actual. 639 00:29:49,419 --> 00:29:52,089 Some people say recovery time reality, it doesn't matter. 640 00:29:52,089 --> 00:29:53,169 Just have a different term. 641 00:29:53,349 --> 00:29:54,099 Don't say. 642 00:29:54,459 --> 00:29:58,269 Our RTO is an hour when what you're saying is this is how fast you can 643 00:29:58,269 --> 00:30:00,879 recover your RTO is your objective. 644 00:30:01,089 --> 00:30:04,959 Your other thing, I don't care what you call it, recovery time actual is good. 645 00:30:05,259 --> 00:30:06,609 This is where we are. 646 00:30:06,759 --> 00:30:11,109 The difference between your recovery time actual and your recovery. 647 00:30:11,829 --> 00:30:17,199 Time objective is the gap that you need to address with whatever changes in 648 00:30:17,199 --> 00:30:22,539 process, documentation, or quite possibly enhancements to your backup system. 649 00:30:24,429 --> 00:30:24,729 Yeah. 650 00:30:24,939 --> 00:30:25,389 All right. 651 00:30:25,509 --> 00:30:27,039 I think, I think we've covered enough. 652 00:30:27,039 --> 00:30:30,819 And then next, next week we're gonna recover recovery 653 00:30:30,844 --> 00:30:32,444 point objective, which is. 654 00:30:33,564 --> 00:30:34,074 It It is. 655 00:30:34,074 --> 00:30:34,314 Yeah. 656 00:30:34,314 --> 00:30:34,704 It's weird. 657 00:30:34,734 --> 00:30:35,574 Like all the, yeah. 658 00:30:35,844 --> 00:30:40,314 Uh, and this is going to be basically how much data we agree that we 659 00:30:40,314 --> 00:30:44,454 can lose, which is something very different than how long the system is. 660 00:30:44,454 --> 00:30:45,894 Yeah, exactly. 661 00:30:46,164 --> 00:30:50,214 We should, we should recover in zero minutes and we should lose zero data. 662 00:30:50,424 --> 00:30:51,414 We all agree. 663 00:30:51,684 --> 00:30:52,944 That would be amazing. 664 00:30:53,274 --> 00:30:54,984 Uh, it's also not gonna happen. 665 00:30:55,614 --> 00:30:57,909 Well, thanks for, uh, thanks for joining me again. 666 00:30:59,769 --> 00:31:00,189 I am. 667 00:31:00,429 --> 00:31:01,089 Enjoy these. 668 00:31:01,089 --> 00:31:01,179 I 669 00:31:01,239 --> 00:31:02,109 Yeah, I'm glad. 670 00:31:02,109 --> 00:31:02,289 Yeah. 671 00:31:02,294 --> 00:31:02,454 more. 672 00:31:03,489 --> 00:31:06,339 I think now, now that we, you've figured out your world and I figured out my 673 00:31:06,339 --> 00:31:08,409 new world, uh, we, we should be good. 674 00:31:09,069 --> 00:31:11,529 And, uh, thanks to the listeners you're, why we do this. 675 00:31:11,769 --> 00:31:12,999 Uh, that is a wrap.