1 00:00:00,000 --> 00:00:04,079 You found the backup wrap up your go-to podcast for all things 2 00:00:04,079 --> 00:00:06,719 backup recovery and cyber recovery. 3 00:00:07,260 --> 00:00:12,029 In this episode, we're covering forever incremental backup technology and 4 00:00:12,029 --> 00:00:16,499 how that completely revolutionized the entire backup industry. 5 00:00:16,499 --> 00:00:20,609 I. Remember the old days of doing weekly fos that brought 6 00:00:20,609 --> 00:00:22,080 your systems to their knees? 7 00:00:22,080 --> 00:00:25,770 Well, those days are thankfully gone for most people. 8 00:00:26,520 --> 00:00:30,329 We're gonna walk through the evolution of, from traditional approaches with 9 00:00:30,599 --> 00:00:35,340 fools and incrementals to synthetic fos, and finally to true forever incremental. 10 00:00:35,879 --> 00:00:39,060 If you're still doing things the old way, well, you're missing 11 00:00:39,060 --> 00:00:40,620 out on some serious benefits. 12 00:00:40,840 --> 00:00:44,860 Better performance, lower storage costs, and way faster restores. 13 00:00:45,489 --> 00:00:49,300 Perhaps this episode will cause you to rethink things a little bit. 14 00:00:50,080 --> 00:00:54,580 By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr. Backup, 15 00:00:54,820 --> 00:00:59,290 and I've been passionate about backup and recovery for over 30 years, ever since. 16 00:00:59,290 --> 00:01:02,920 I had to tell my boss that we had no backups of the production 17 00:01:02,920 --> 00:01:04,330 database that we had just lost. 18 00:01:04,965 --> 00:01:07,725 I don't want that to happen to you, and that's why I do this. 19 00:01:08,145 --> 00:01:13,275 On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes. 20 00:01:13,635 --> 00:01:15,705 This is the backup wrap up. 21 00:01:30,597 --> 00:01:31,527 Welcome to the show. 22 00:01:32,517 --> 00:01:37,347 Hi, I am w Curtis Preston, AKA, Mr. Backup, and I have with me a guy that 23 00:01:37,347 --> 00:01:40,677 I bet is pretty dang happy right now. 24 00:01:40,987 --> 00:01:42,997 Prasanna Malaiyandi. 25 00:01:43,707 --> 00:01:44,997 Why are you looking like that? 26 00:01:45,022 --> 00:01:46,582 very confused about what I'm happy about. 27 00:01:47,787 --> 00:01:48,387 Really? 28 00:01:49,017 --> 00:01:54,357 Do I need to explain recent events that happened at your house 29 00:01:55,077 --> 00:01:55,317 Oh 30 00:01:55,377 --> 00:01:56,427 that, that. 31 00:02:07,567 --> 00:02:12,157 For those of you by the way, um, who are listening to this podcast via 32 00:02:12,217 --> 00:02:15,217 audio, we also provide video on YouTube. 33 00:02:15,217 --> 00:02:19,837 So if you wanted to see our expressions just then and really see what Curtis 34 00:02:19,837 --> 00:02:22,117 is laughing about, uh, go check us 35 00:02:22,167 --> 00:02:22,827 Yeah, 36 00:02:23,257 --> 00:02:23,887 Backup wrap up. 37 00:02:24,897 --> 00:02:31,047 so what events happened in the last 48 hours? 38 00:02:31,387 --> 00:02:32,587 my wife was in India 39 00:02:33,177 --> 00:02:33,717 Yeah. 40 00:02:34,437 --> 00:02:36,717 Visiting family and she just got back. 41 00:02:37,317 --> 00:02:38,127 Yes. 42 00:02:39,027 --> 00:02:39,537 it was good. 43 00:02:40,977 --> 00:02:46,227 Suddenly, you're so much harder to get on the phone between that and your new job. 44 00:02:46,227 --> 00:02:47,607 I don't even know you anymore. 45 00:02:50,337 --> 00:02:52,182 He's like, why would I be happy? 46 00:02:52,932 --> 00:02:56,847 I, I was really, I was, I thought it was like, I was thinking, I was like, 47 00:02:56,847 --> 00:02:58,647 was it something YouTube related? 48 00:02:58,647 --> 00:03:02,397 Was it something like I, we had watched or we had talked about and. 49 00:03:04,362 --> 00:03:07,092 No, the fact that your wife is back 50 00:03:07,737 --> 00:03:10,857 although she's also slightly jet lagged, so, 51 00:03:11,112 --> 00:03:14,592 so well, it's what, what, what's it is 12 hours difference. 52 00:03:15,027 --> 00:03:17,307 right now it is 13 and a half. 53 00:03:17,952 --> 00:03:18,432 Yeah. 54 00:03:18,762 --> 00:03:19,182 Yeah. 55 00:03:19,872 --> 00:03:20,592 So 56 00:03:20,772 --> 00:03:20,922 does 57 00:03:20,952 --> 00:03:21,702 of course she's, 58 00:03:21,792 --> 00:03:22,932 jet lagged than I do. 59 00:03:23,202 --> 00:03:23,472 So 60 00:03:23,742 --> 00:03:24,282 yeah. 61 00:03:24,282 --> 00:03:26,232 and come back, it takes me like three weeks. 62 00:03:26,592 --> 00:03:27,072 And, 63 00:03:27,072 --> 00:03:27,462 Yeah. 64 00:03:27,522 --> 00:03:29,952 have a dog, which I think some people have seen on the podcast. 65 00:03:29,982 --> 00:03:34,002 Um, it's funny because he gets jet lagged when I'm jet lagged. 66 00:03:34,332 --> 00:03:37,572 So he, his hours will shift to match with mine. 67 00:03:37,752 --> 00:03:37,932 So 68 00:03:38,067 --> 00:03:39,477 That's really adorable. 69 00:03:39,762 --> 00:03:41,052 and he would be like, Hey, I'm ready. 70 00:03:41,052 --> 00:03:41,742 Let's go, let's go. 71 00:03:41,742 --> 00:03:43,122 And then when I'm passed out, he's asleep. 72 00:03:43,122 --> 00:03:44,532 So he's 73 00:03:44,607 --> 00:03:45,507 Interesting. 74 00:03:46,767 --> 00:03:47,127 Yes. 75 00:03:47,132 --> 00:03:50,307 Your dog that's named after an Indian dessert. 76 00:03:50,502 --> 00:03:50,742 cream. 77 00:03:50,772 --> 00:03:51,102 Yep. 78 00:03:51,477 --> 00:03:51,837 Ice cream. 79 00:03:51,837 --> 00:03:52,257 Oh yeah. 80 00:03:52,257 --> 00:03:52,737 Ice cream. 81 00:03:52,737 --> 00:03:52,947 Yeah. 82 00:03:53,037 --> 00:03:56,997 Kulfi, which I have now had, um. 83 00:03:57,042 --> 00:03:59,782 your, did you know they have various flavors of Kulfi. 84 00:03:59,952 --> 00:04:01,182 They're not all just like one. 85 00:04:01,932 --> 00:04:03,372 Oh, I thought it was all mango. 86 00:04:03,432 --> 00:04:05,712 So it's met like mango Kulfi and then what? 87 00:04:05,712 --> 00:04:06,622 Coconut Kulfi. 88 00:04:06,642 --> 00:04:07,207 What are what? 89 00:04:07,242 --> 00:04:08,472 What are the other typical 90 00:04:09,312 --> 00:04:10,932 and pistachio. 91 00:04:11,892 --> 00:04:13,632 Pistachio and, and. 92 00:04:14,517 --> 00:04:15,682 I think there are other ones too. 93 00:04:17,052 --> 00:04:17,442 Yeah. 94 00:04:17,442 --> 00:04:18,343 I don't, yeah, I don't. 95 00:04:18,762 --> 00:04:23,352 I've used cardamom as a, an ingredient, you know, when I made the, the 96 00:04:23,352 --> 00:04:26,622 mango pie thing, but I don't think I know what it tastes like by itself. 97 00:04:26,712 --> 00:04:32,922 So I'll have to, I'll have to sprinkle some and, uh, see what that tastes like. 98 00:04:32,982 --> 00:04:33,942 Or maybe just go and 99 00:04:34,137 --> 00:04:34,377 Yeah. 100 00:04:34,572 --> 00:04:34,932 some 101 00:04:35,037 --> 00:04:36,417 next time you're at the Indian store. 102 00:04:36,417 --> 00:04:40,017 Actually, sometimes even the Costcos carry like a sampler pack 103 00:04:40,482 --> 00:04:44,442 Oh, yeah, so like when I go to the cash and carry down in Mira Mesa. 104 00:04:44,577 --> 00:04:45,927 the cash and carry will definitely have it. 105 00:04:45,927 --> 00:04:47,337 If you go check out the freezer section. 106 00:04:47,787 --> 00:04:48,027 Just 107 00:04:48,057 --> 00:04:48,837 Interesting. 108 00:04:49,017 --> 00:04:50,697 all before you get home. 109 00:04:53,037 --> 00:04:53,547 Yeah. 110 00:04:54,987 --> 00:04:55,857 I, oh man. 111 00:04:55,857 --> 00:05:00,837 I'm, you're, you're literally like, like my mouth is Yeah, yeah. 112 00:05:00,837 --> 00:05:02,907 Thinking about, thinking about it sounds yummy. 113 00:05:03,232 --> 00:05:03,352 Hmm. 114 00:05:04,437 --> 00:05:07,527 Uh, so we're gonna talk about something that's just as yummy. 115 00:05:09,987 --> 00:05:14,397 That is if you're into, into the backup thing, uh, we're gonna talk 116 00:05:14,397 --> 00:05:16,737 about forever incremental and sort of. 117 00:05:16,782 --> 00:05:21,012 Uh, just how big of a deal that is or was. 118 00:05:21,462 --> 00:05:23,142 Um, it, at, at this point? 119 00:05:23,142 --> 00:05:28,782 It's in, in most modern backup products that have come out in 120 00:05:28,782 --> 00:05:34,872 the last 10 years, I'd say, uh, ones that are actually new, right? 121 00:05:34,872 --> 00:05:41,282 So, you know, when we look at the products like Rubrik, Cohesity, or 122 00:05:41,282 --> 00:05:47,132 modern SaaS applications like Druva, Hyku, keep it, Alcion, uh, right. 123 00:05:47,132 --> 00:05:49,472 You know, these are all of these products. 124 00:05:50,687 --> 00:05:51,527 Have forever. 125 00:05:51,527 --> 00:05:53,447 Incremental in common, right? 126 00:05:53,477 --> 00:05:56,117 Nobody wants to do full backups anymore. 127 00:05:56,207 --> 00:05:59,927 Uh, or, or anything like full backups, right? 128 00:05:59,927 --> 00:06:03,287 So, um, so let's talk about that. 129 00:06:03,287 --> 00:06:07,627 Let's talk about what, you know, where were we, what was it like back in the 130 00:06:07,627 --> 00:06:14,837 day, and, um, and then sort of how we got the, from there to here and why it's 131 00:06:14,837 --> 00:06:16,697 such a big deal and why you should care. 132 00:06:16,952 --> 00:06:17,342 Right. 133 00:06:17,372 --> 00:06:24,452 And why this perhaps may give you a reason to finally, um, you 134 00:06:24,452 --> 00:06:26,232 know, what, what is it, shuck? 135 00:06:26,252 --> 00:06:27,692 The mortal coil. 136 00:06:27,692 --> 00:06:29,732 What, what, what, what is that? 137 00:06:30,392 --> 00:06:30,752 What, 138 00:06:30,877 --> 00:06:31,507 I'm just 139 00:06:31,507 --> 00:06:36,337 thinking are there company, and maybe as we're talking we can figure this out, like 140 00:06:36,487 --> 00:06:42,307 are there still vendors that exist that don't support forever Incrementals? 141 00:06:42,472 --> 00:06:47,212 That's actually a really good question and I'm gonna say, uh, absolutely 142 00:06:47,452 --> 00:06:57,282 right, because my concept of what a true forever incremental will be based on. 143 00:06:57,957 --> 00:07:00,867 Like a, a complete redesign of the, of the underneath. 144 00:07:00,867 --> 00:07:01,227 Right. 145 00:07:01,827 --> 00:07:06,387 Um, so the phrase I was looking for is shuttle off sh shuffle off the 146 00:07:06,387 --> 00:07:08,787 mortal coil of your old backup product, 147 00:07:11,067 --> 00:07:12,837 by the way, wait, quick story. 148 00:07:12,957 --> 00:07:19,587 So I'm on, I'm on a plane one time and this, this, um. 149 00:07:20,202 --> 00:07:24,282 This pilot, he was talking about the fact that we were about to take off and 150 00:07:24,282 --> 00:07:27,882 he referred to taking off as shuffling off the mortal coil of this earth. 151 00:07:28,272 --> 00:07:31,662 And I'm like, dude, that's a phrase for dying. 152 00:07:33,072 --> 00:07:33,402 What? 153 00:07:33,402 --> 00:07:37,902 Why the hell are you, why would you do that? 154 00:07:37,992 --> 00:07:38,472 Right. 155 00:07:38,602 --> 00:07:43,622 , it's a Shakespeare, it's from the to be or not to be soliloquy 156 00:07:43,817 --> 00:07:44,297 I, 157 00:07:44,852 --> 00:07:45,332 Hamlet. 158 00:07:45,407 --> 00:07:49,457 in all fairness though, I would say 97% of people probably had 159 00:07:49,457 --> 00:07:50,842 no idea what that phrase meant. 160 00:07:51,332 --> 00:07:53,237 They, they never even heard. 161 00:07:53,477 --> 00:07:53,987 Yeah. 162 00:07:54,917 --> 00:07:55,577 Yeah. 163 00:07:55,877 --> 00:08:00,707 And, and, and, and, and just to, just to, uh, like, you know, since I'm over 164 00:08:00,707 --> 00:08:04,797 here quoting Shakespeare, let me just put this, I'm not a huge Shakespeare person. 165 00:08:04,797 --> 00:08:11,667 In fact, it, it wasn't until, I dunno, several years ago, and we were in England 166 00:08:11,787 --> 00:08:17,967 where I, I, I got to actually see my first, um, uh, Shakespeare play, which 167 00:08:17,967 --> 00:08:21,477 was Henry V. Some people say Henry the VI don't, I think it's actually 168 00:08:21,477 --> 00:08:23,967 called Henry five, which is the, um. 169 00:08:24,357 --> 00:08:26,787 The one with the St. Crispin stay speech. 170 00:08:27,297 --> 00:08:30,237 Uh, once more into the breach, close the wall up with, with 171 00:08:30,237 --> 00:08:31,347 her English dead and all that. 172 00:08:31,347 --> 00:08:34,377 But anyway, sorry, I just bored everybody. 173 00:08:34,377 --> 00:08:34,737 Sorry. 174 00:08:34,797 --> 00:08:35,517 Sorry, you're asleep. 175 00:08:35,517 --> 00:08:36,057 Wake back up. 176 00:08:36,057 --> 00:08:36,597 Wake back up. 177 00:08:37,692 --> 00:08:40,767 We're, we're gonna talk about backup, which is so much more 178 00:08:40,767 --> 00:08:42,747 exciting than Shakespeare. 179 00:08:43,707 --> 00:08:50,787 Um, so, all right, so first let's talk about, do you want to define, uh, so what? 180 00:08:50,802 --> 00:08:53,022 let's talk about the old days first. 181 00:08:53,082 --> 00:08:54,132 that's what I, that's what I'm saying. 182 00:08:54,162 --> 00:08:54,762 That's what I'm saying. 183 00:08:54,792 --> 00:08:57,492 Let's, do you wanna define, uh, what a full and, and 184 00:08:57,732 --> 00:08:58,212 Yeah. 185 00:08:58,362 --> 00:08:59,442 an incremental backup is? 186 00:08:59,682 --> 00:08:59,982 Yeah. 187 00:09:00,042 --> 00:09:00,642 So. 188 00:09:01,017 --> 00:09:03,837 Normally when you think about backups, right, you talk, 189 00:09:04,197 --> 00:09:05,847 typically people talk about fulls. 190 00:09:05,847 --> 00:09:07,377 Incrementals, right? 191 00:09:07,407 --> 00:09:10,587 A full is basically your entire application, dataset, whatever 192 00:09:10,587 --> 00:09:12,117 it is, everything is backed up. 193 00:09:12,762 --> 00:09:12,912 Mm-hmm. 194 00:09:13,017 --> 00:09:17,187 then between doing every fulls, you would do incrementals, which are just the things 195 00:09:17,187 --> 00:09:19,077 that have changed since that last full. 196 00:09:19,767 --> 00:09:20,382 Right, right. 197 00:09:20,517 --> 00:09:24,207 you do these periodically, and normally you do incrementals 198 00:09:24,207 --> 00:09:25,767 because they will be smaller. 199 00:09:26,752 --> 00:09:31,732 So you could say do a full once a week and then incrementals during the week 200 00:09:32,002 --> 00:09:37,042 such that you can finish in time or during your backup window, and then the next 201 00:09:37,042 --> 00:09:39,022 weekend you would do like another full, 202 00:09:39,597 --> 00:09:43,557 Right, right now, slightly, I'll slightly change your 203 00:09:43,557 --> 00:09:45,927 definition or a pen or whatever. 204 00:09:46,377 --> 00:09:50,007 Um, so you specifically described what I would call a 205 00:09:50,007 --> 00:09:52,437 cumulative incremental, right? 206 00:09:52,437 --> 00:09:57,867 So a, uh, what is referred to as a sort of a regular, incremental 207 00:09:57,867 --> 00:09:59,157 or a differential incremental. 208 00:09:59,202 --> 00:09:59,862 Yep, 209 00:10:00,687 --> 00:10:05,877 it is one where it's the changes just since the last incremental or since 210 00:10:05,877 --> 00:10:07,857 the last backup of any kind, right? 211 00:10:07,902 --> 00:10:08,292 And then you 212 00:10:08,367 --> 00:10:09,027 Um, 213 00:10:09,312 --> 00:10:10,212 levels and all the 214 00:10:10,377 --> 00:10:10,857 yeah. 215 00:10:10,887 --> 00:10:13,107 Yeah, yeah, 216 00:10:13,152 --> 00:10:16,272 But yeah, at a high level, yeah, there are those three types, if you will. 217 00:10:16,647 --> 00:10:17,097 yeah. 218 00:10:17,457 --> 00:10:22,407 And, and, and then what really, um, what really what it really 219 00:10:22,407 --> 00:10:28,407 came down to is how things behaved when it came time for a restore. 220 00:10:28,887 --> 00:10:29,307 Right. 221 00:10:30,177 --> 00:10:36,707 So if you were going to restore something, then you needed the full, 222 00:10:37,277 --> 00:10:41,627 the latest cumulative, incremental, if you were doing such a thing, and 223 00:10:41,627 --> 00:10:44,387 then any incrementals since that day. 224 00:10:44,837 --> 00:10:47,717 And so, um, those of us, uh. 225 00:10:49,202 --> 00:10:52,547 I, I remember at some point we, we used to do weekly fulls. 226 00:10:53,297 --> 00:10:57,127 I remember my preferred backup design switching to monthly fulls, 227 00:10:57,407 --> 00:11:01,127 weekly cumulative incrementals, and then daily incrementals. 228 00:11:01,397 --> 00:11:05,347 So if you were, let's say three weeks into the month, you would need the full 229 00:11:05,347 --> 00:11:10,057 from the first of the month, the latest, uh, weekly cumulative, incremental, 230 00:11:10,057 --> 00:11:11,617 and then three or four days worth of. 231 00:11:12,232 --> 00:11:13,042 Incrementals 232 00:11:13,282 --> 00:11:13,462 Yeah, 233 00:11:13,732 --> 00:11:15,977 and, uh, in order to do a restore. 234 00:11:16,022 --> 00:11:21,962 . So what it meant was that you would often, the way the Restore worked is you 235 00:11:21,962 --> 00:11:26,702 would restore everything that was in the full, then you would restore everything 236 00:11:26,702 --> 00:11:29,972 that was in the cumulative incremental, and then you would restore everything 237 00:11:29,972 --> 00:11:31,502 that was in each of the incrementals. 238 00:11:31,712 --> 00:11:36,782 What that meant was there were many files that you were actually restoring 239 00:11:36,782 --> 00:11:40,862 multiple times and you were overwriting them, which was just inefficient. 240 00:11:41,372 --> 00:11:43,442 Um, and what that also meant was. 241 00:11:44,117 --> 00:11:51,917 That you had, uh, uh, duplicate data, the old way of doing backups created 242 00:11:51,917 --> 00:11:55,307 a ton of duplicate data on the tapes. 243 00:11:55,637 --> 00:11:58,637 You could say duplicate, or you could also say it's independent, 244 00:11:58,637 --> 00:12:00,347 depending on the glass. 245 00:12:00,347 --> 00:12:01,877 Half full glass, half empty. 246 00:12:02,507 --> 00:12:04,877 Well, so no, I, I understand what you're saying. 247 00:12:04,877 --> 00:12:06,557 What I'm talking about is the fulls, 248 00:12:06,572 --> 00:12:06,862 Yeah, 249 00:12:07,697 --> 00:12:11,387 the fulls over time created a ton of duplicate data. 250 00:12:11,537 --> 00:12:11,872 that's true. 251 00:12:11,957 --> 00:12:12,317 Right. 252 00:12:12,317 --> 00:12:13,097 I know what you're thinking. 253 00:12:13,097 --> 00:12:17,087 You're thinking, you're talking about the, within the, within the, the incrementals. 254 00:12:17,267 --> 00:12:18,587 I'm mainly talking about 255 00:12:18,792 --> 00:12:19,082 yeah, 256 00:12:19,157 --> 00:12:22,547 how many different copies of the exact same file were all 257 00:12:22,547 --> 00:12:24,047 over, you know, everywhere. 258 00:12:24,257 --> 00:12:28,217 and especially because most times, right, you look at sort of like a 259 00:12:28,217 --> 00:12:31,307 2% daily change rate, which maybe 260 00:12:31,667 --> 00:12:32,027 Yeah. 261 00:12:32,177 --> 00:12:34,397 a seven or 8% change rate per week. 262 00:12:34,397 --> 00:12:34,697 And 263 00:12:35,147 --> 00:12:35,537 Yeah. 264 00:12:35,627 --> 00:12:39,977 those weekly cumulatives or even the monthly fulls, right, you're 265 00:12:40,457 --> 00:12:43,097 basically backing up a bunch of data, which you don't necessarily 266 00:12:43,097 --> 00:12:44,537 need that haven't changed. 267 00:12:45,197 --> 00:12:46,157 Right, right. 268 00:12:46,157 --> 00:12:50,507 You, you, the reason we did it the way we did, it wasn't so much about independence. 269 00:12:50,507 --> 00:12:51,137 It was about. 270 00:12:52,157 --> 00:12:52,847 Minimize. 271 00:12:52,847 --> 00:12:55,937 So if you did a monthly full, followed by nothing but regular 272 00:12:55,937 --> 00:13:00,437 incrementals, you would have to load 30 tapes at the end of the day, right? 273 00:13:00,437 --> 00:13:02,747 At the end of, and even if it wasn't tapes, you would have 274 00:13:02,747 --> 00:13:04,907 to, you have to do 30 restores. 275 00:13:05,047 --> 00:13:05,337 Yeah, 276 00:13:05,627 --> 00:13:11,597 So, so the, the, the, the differential or the cumulative incremental, um, 277 00:13:11,927 --> 00:13:15,617 was about minimizing the level of effort during a restore, right? 278 00:13:16,097 --> 00:13:19,397 And then came along this thing called dedupe. 279 00:13:20,117 --> 00:13:26,027 Uh, which at least helped minimize, I think the, all of that duplicate data. 280 00:13:26,357 --> 00:13:29,497 because I think with deduplication it was nice because you didn't 281 00:13:29,497 --> 00:13:33,127 have to change what you were doing from like a policy perspective 282 00:13:33,247 --> 00:13:33,697 Right, 283 00:13:34,177 --> 00:13:34,747 perspective. 284 00:13:34,747 --> 00:13:37,027 But you got, like you said, it was able to. 285 00:13:37,702 --> 00:13:38,692 Dedupe, of course, 286 00:13:39,427 --> 00:13:39,937 right. 287 00:13:40,252 --> 00:13:42,802 in the backend and save you the storage costs. 288 00:13:43,702 --> 00:13:49,522 And that's why when, if you, if you were doing a, a typical weekly, by 289 00:13:49,522 --> 00:13:53,902 the way, nobody did the, the monthly full thing that I recommended, most 290 00:13:53,902 --> 00:13:55,432 people did weekly fulls, right? 291 00:13:56,122 --> 00:13:58,702 And so when you had weekly fulls, um. 292 00:13:59,017 --> 00:14:02,137 This is why, and you, and you had like 90 day retention. 293 00:14:02,347 --> 00:14:07,117 This is why just with even crappy dedupe, you got 20 to one, 294 00:14:07,207 --> 00:14:07,267 Yeah. 295 00:14:07,567 --> 00:14:09,847 uh, data reduction, right? 296 00:14:10,117 --> 00:14:15,427 And this is what helped, um, pay for, you know, the what to help 297 00:14:15,427 --> 00:14:18,457 created the, the market for all that, all that dedupe work, right? 298 00:14:18,757 --> 00:14:21,637 And then we could argue over, um, you know, well we're, we're 299 00:14:21,637 --> 00:14:24,007 20 to one, we're 30 to one, we're, you know, whatever, right? 300 00:14:24,427 --> 00:14:27,037 And, um, we, we saved this much more money. 301 00:14:27,367 --> 00:14:28,057 Um. 302 00:14:28,492 --> 00:14:30,952 You know, once you get rid of all those fulls, everything else 303 00:14:30,952 --> 00:14:32,272 was, was really kind of incre 304 00:14:32,302 --> 00:14:32,592 Yeah. 305 00:14:32,722 --> 00:14:33,262 incremental. 306 00:14:34,132 --> 00:14:42,082 Um, but the real, I think the real challenge was just the core backup design 307 00:14:42,292 --> 00:14:50,482 of when we go to do a restore, we have to restore, uh, data multiple times. 308 00:14:51,052 --> 00:14:51,472 Right. 309 00:14:51,742 --> 00:14:53,422 It just simply wasn't efficient. 310 00:14:53,617 --> 00:14:56,257 Well, and I think even before we get there, I'd like to cover 311 00:14:56,452 --> 00:14:56,782 Mm-hmm. 312 00:14:57,547 --> 00:14:57,787 uh, 313 00:14:57,787 --> 00:14:58,237 Okay. 314 00:14:58,477 --> 00:15:03,007 if you had the existing backup schedule, right, where I'm still doing 315 00:15:03,007 --> 00:15:05,287 weekly fulls and daily incrementals, 316 00:15:05,472 --> 00:15:05,892 Mm-hmm. 317 00:15:06,007 --> 00:15:08,677 still transferring all that data over the network. 318 00:15:08,677 --> 00:15:08,917 Right? 319 00:15:08,917 --> 00:15:12,427 And I think this is where client side deduplication would probably 320 00:15:12,427 --> 00:15:14,557 come next, in my opinion, right? 321 00:15:14,782 --> 00:15:15,002 Yep. 322 00:15:15,187 --> 00:15:17,587 it would help you reduce the amount of data you're actually 323 00:15:17,587 --> 00:15:18,907 sending over the network. 324 00:15:19,507 --> 00:15:19,897 Right. 325 00:15:20,302 --> 00:15:24,052 And that'll help save you cost as well as time. 326 00:15:24,877 --> 00:15:26,467 Yeah, that's actually a really good point. 327 00:15:26,467 --> 00:15:27,397 Thanks for bringing that up. 328 00:15:27,397 --> 00:15:32,677 So the, one of the other like, like basically what you're saying that one 329 00:15:32,677 --> 00:15:39,127 of the other, um, downsides to doing it the old way was both the transfer 330 00:15:39,127 --> 00:15:44,407 across the network as well as the impact, uh, to the clients, right? 331 00:15:44,827 --> 00:15:49,657 If you're doing a full, on a regular basis, um, you're, um. 332 00:15:50,977 --> 00:15:53,017 There's a, there's a cost to that 333 00:15:53,137 --> 00:15:53,257 Yep. 334 00:15:53,407 --> 00:15:55,837 in both, you know, you brought up bandwidth, it's 335 00:15:55,837 --> 00:15:57,307 bandwidth storage and compute. 336 00:15:57,307 --> 00:15:57,667 Right? 337 00:15:58,147 --> 00:15:59,767 Um, it's like all the things. 338 00:15:59,797 --> 00:16:03,997 There was also a, um, sort of a, like a perfect storm brewing. 339 00:16:04,037 --> 00:16:04,157 Yeah. 340 00:16:04,837 --> 00:16:07,477 So we're doing fulls on a regular basis. 341 00:16:07,477 --> 00:16:10,087 We're doing cumulative incrementals on a regular basis, and this is 342 00:16:10,087 --> 00:16:13,417 how we were doing things, and then along came virtualization. 343 00:16:13,417 --> 00:16:14,527 Why was that a problem? 344 00:16:14,932 --> 00:16:16,822 Because a lot of the data was 345 00:16:17,122 --> 00:16:17,242 I, 346 00:16:17,302 --> 00:16:18,952 the same, right? 347 00:16:19,132 --> 00:16:21,952 When you look across all of your VMs, they're probably spun up 348 00:16:21,952 --> 00:16:24,922 from the same golden copy, of your 349 00:16:25,267 --> 00:16:25,447 Well, the, 350 00:16:25,792 --> 00:16:26,062 and 351 00:16:26,167 --> 00:16:26,677 yeah, well, the. 352 00:16:27,322 --> 00:16:27,922 amongst them. 353 00:16:28,762 --> 00:16:33,772 Well, that, that's a, this is another example of, that's a great answer. 354 00:16:33,772 --> 00:16:35,152 It's not the question I was asking. 355 00:16:36,562 --> 00:16:36,832 It's 356 00:16:36,862 --> 00:16:37,462 Um, 357 00:16:37,492 --> 00:16:39,082 resource on the backend because 358 00:16:39,292 --> 00:16:40,822 that's what I was going after. 359 00:16:40,882 --> 00:16:41,302 Yeah, 360 00:16:41,392 --> 00:16:42,322 and the compute, right? 361 00:16:42,322 --> 00:16:42,832 You don't have your 362 00:16:42,892 --> 00:16:43,342 yeah, 363 00:16:43,462 --> 00:16:44,002 instance 364 00:16:44,692 --> 00:16:45,022 yeah. 365 00:16:45,052 --> 00:16:46,882 That's, that's what I was going after. 366 00:16:46,882 --> 00:16:47,242 Right? 367 00:16:47,242 --> 00:16:54,712 So, so that was a real problem when VMware got really popular and we just 368 00:16:54,712 --> 00:16:56,482 kept doing the same old thing because. 369 00:16:57,082 --> 00:17:00,202 Remember, for a long time we backed up VMware. 370 00:17:00,202 --> 00:17:02,632 We backed up the physical, the virtual machines, just like 371 00:17:02,632 --> 00:17:03,652 they were physical machines. 372 00:17:05,122 --> 00:17:05,572 Yes. 373 00:17:05,572 --> 00:17:08,782 Guest ba. We just basically installed an A, an agent and every guest, 374 00:17:08,782 --> 00:17:10,192 and that's how we did the backups. 375 00:17:10,192 --> 00:17:15,022 And, you know, and what happened was you'd do just randomly, you'd end up 376 00:17:15,202 --> 00:17:17,332 doing a bunch of, at the same time. 377 00:17:18,037 --> 00:17:22,867 Which worked fine when we were 20 individual machines, but now we're 20 378 00:17:22,867 --> 00:17:27,127 VMs all on the same physical machine, and that just broke, like virtualization 379 00:17:27,127 --> 00:17:28,537 just broke back up overnight. 380 00:17:28,717 --> 00:17:29,107 Right. 381 00:17:29,767 --> 00:17:33,937 Um, the latest thing to break back up overnight is, um, containers. 382 00:17:33,982 --> 00:17:34,312 Yep. 383 00:17:34,807 --> 00:17:35,707 That's a whole other thing. 384 00:17:35,707 --> 00:17:40,447 We're not talking about that right now, but, um, the 385 00:17:41,002 --> 00:17:41,392 so 386 00:17:41,497 --> 00:17:41,617 ahead. 387 00:17:41,962 --> 00:17:48,022 though, this is what sort of VMware, Plus the backup vendors to figure out 388 00:17:48,022 --> 00:17:50,122 is there something better we can do 389 00:17:50,632 --> 00:17:50,962 Yeah. 390 00:17:50,992 --> 00:17:53,872 order to help satisfy backup needs? 391 00:17:53,872 --> 00:17:55,402 Because it's not going away, right? 392 00:17:55,402 --> 00:17:55,792 You have to 393 00:17:55,822 --> 00:17:56,272 Right, 394 00:17:56,427 --> 00:17:56,647 VMs. 395 00:17:57,172 --> 00:17:57,922 right, right. 396 00:17:58,732 --> 00:18:08,392 So the the real key I think, I think the real goal, I. Like, like if we could 397 00:18:08,392 --> 00:18:10,312 go back in time and I, I don't know. 398 00:18:10,312 --> 00:18:10,762 I don't know. 399 00:18:11,602 --> 00:18:16,402 This is one of the things where like, I don't know if we, I, I wasn't in the 400 00:18:16,402 --> 00:18:22,132 rooms right when they were designing this stuff, but I'd like to think that 401 00:18:22,222 --> 00:18:28,882 somebody said, you know, if, because my memory is, we had issues with tape. 402 00:18:29,827 --> 00:18:32,677 The, the, there was the fundamental mismatch of speed tape. 403 00:18:32,677 --> 00:18:34,717 You know, people thought it was slow, it was actually fast. 404 00:18:34,717 --> 00:18:35,377 That was a problem. 405 00:18:35,617 --> 00:18:38,107 And we need, and so we started doing disc staging. 406 00:18:38,107 --> 00:18:41,227 We started doing disc backups because of that, right? 407 00:18:41,347 --> 00:18:45,907 And then DDU came along and DDU said, Hey, we can make disc better. 408 00:18:45,907 --> 00:18:47,887 We can make disc, you know, more affordable. 409 00:18:47,887 --> 00:18:50,917 And, and so, so that was, and then it just sort of happened. 410 00:18:51,697 --> 00:18:52,447 But even if. 411 00:18:53,752 --> 00:19:00,652 We if, if that wasn't the case, I think the design of backups ultimately ending 412 00:19:00,652 --> 00:19:04,582 up being, uh, what we're talking about today, which is forever incremental, 413 00:19:05,422 --> 00:19:08,002 it, it accomplishes multiple purposes. 414 00:19:08,002 --> 00:19:11,032 One is it, it, it solves the issues that we have with tape. 415 00:19:11,062 --> 00:19:15,322 Because in order for forever incremental work, you've gotta be using disc, right? 416 00:19:16,072 --> 00:19:20,332 Uh, for, for true forever, incremental work to work, you've gotta be using disc. 417 00:19:21,442 --> 00:19:25,672 Um, and then the other thing is that it, it solves the, the cost issues. 418 00:19:25,672 --> 00:19:29,842 It solves the performance issues, restore, you know, it does a lot of things right. 419 00:19:29,902 --> 00:19:30,562 Um, 420 00:19:30,607 --> 00:19:33,157 you first define what forever incremental is? 421 00:19:33,157 --> 00:19:33,457 'cause I don't 422 00:19:33,532 --> 00:19:34,342 I don't want to, 423 00:19:34,597 --> 00:19:34,747 that. 424 00:19:37,702 --> 00:19:41,452 Before, before we get to forever incremental, we gotta talk about 425 00:19:41,452 --> 00:19:47,102 something that was, that was a stop gap technology bef to get from we'd, 426 00:19:47,342 --> 00:19:50,072 we wanted to get to a point where we weren't doing fullsS anymore. 427 00:19:50,972 --> 00:19:54,272 Everybody agreed that full stunk, except especially when we're 428 00:19:54,377 --> 00:19:55,682 talking about virtualization, right? 429 00:19:56,372 --> 00:20:02,672 What was the stop gap technology that backup vendors came out with before 430 00:20:02,672 --> 00:20:05,132 we went to the whole For incremental? 431 00:20:05,312 --> 00:20:08,032 Uh, they created what they called synthetic fulls, 432 00:20:08,612 --> 00:20:08,792 Mm-hmm. 433 00:20:09,002 --> 00:20:11,732 which is basically you are. 434 00:20:12,332 --> 00:20:18,422 Creating a full on the backend, but you're sending data and stitching it together and 435 00:20:18,422 --> 00:20:22,142 piecing it together to make it look like what the full would look like, because you 436 00:20:22,187 --> 00:20:22,547 Yeah. 437 00:20:22,892 --> 00:20:24,002 backups were on that 438 00:20:24,287 --> 00:20:24,737 Right. 439 00:20:25,517 --> 00:20:30,647 So it's like we already know, we, we already know what the files are in 440 00:20:30,647 --> 00:20:34,337 the full, because, you know, we did it on we, we did another incremental. 441 00:20:35,267 --> 00:20:36,497 So we know what's deleted. 442 00:20:36,497 --> 00:20:37,607 We know what's new. 443 00:20:37,787 --> 00:20:39,737 We know what a full would look like. 444 00:20:40,367 --> 00:20:44,207 If we went and made one, why don't we just make one over here 445 00:20:44,387 --> 00:20:46,067 without having to do it over there. 446 00:20:46,697 --> 00:20:49,937 Um, and there were a couple of different ways to do that. 447 00:20:50,597 --> 00:20:55,427 Uh, there was the backup vendor way and there was the, um, the, well, the 448 00:20:55,427 --> 00:20:58,727 backup software vendor way, and then there was the backup hardware vendor way. 449 00:20:58,787 --> 00:20:58,847 Yeah. 450 00:20:59,117 --> 00:21:01,637 I'll talk about the backup software vendor way first. 451 00:21:02,447 --> 00:21:05,777 It's like, we know what all the files should be on a full, 452 00:21:05,777 --> 00:21:07,097 we know where all of them are. 453 00:21:07,127 --> 00:21:08,807 We're just gonna copy them all together. 454 00:21:09,347 --> 00:21:12,197 You know, in one image, and then we're gonna copy that over to a new 455 00:21:12,197 --> 00:21:13,967 set of tapes or a new set of disc 456 00:21:14,012 --> 00:21:14,232 Mm. 457 00:21:14,507 --> 00:21:20,147 so that we essentially create an actual full, um, without actually having to 458 00:21:20,147 --> 00:21:22,697 transfer any data around the network. 459 00:21:22,787 --> 00:21:25,727 It might be transferring it, uh, well it is transferring 460 00:21:25,727 --> 00:21:28,337 it within the backup system. 461 00:21:28,547 --> 00:21:28,787 Right. 462 00:21:29,627 --> 00:21:34,937 And then there was the, the, and, and that took a certain amount of time. 463 00:21:35,297 --> 00:21:35,687 Right. 464 00:21:36,377 --> 00:21:37,727 Um, and then. 465 00:21:39,052 --> 00:21:42,047 A, a vendor that you might be familiar with came out with 466 00:21:42,047 --> 00:21:42,947 a different way to do it. 467 00:21:42,977 --> 00:21:43,637 What was that? 468 00:21:43,862 --> 00:21:48,092 And basically, um, they basically said, just tell us how you wanna 469 00:21:48,092 --> 00:21:49,472 stitch everything together, 470 00:21:49,997 --> 00:21:50,447 Yeah. 471 00:21:50,552 --> 00:21:51,152 it all together. 472 00:21:51,917 --> 00:21:52,367 Yeah. 473 00:21:52,457 --> 00:21:54,557 So it was like, it was like an API, right. 474 00:21:54,767 --> 00:21:56,147 You know where all the stuff is. 475 00:21:56,327 --> 00:21:57,467 We have all the stuff. 476 00:21:57,707 --> 00:21:59,927 Just tell us which stuff you want, where, and we'll go. 477 00:22:00,482 --> 00:22:00,872 Yep. 478 00:22:02,987 --> 00:22:05,147 And so they could make essentially a new full. 479 00:22:07,247 --> 00:22:09,827 When in reality it's just a bunch of pointers to what was already there. 480 00:22:10,007 --> 00:22:10,457 Right. 481 00:22:10,517 --> 00:22:10,667 this 482 00:22:10,757 --> 00:22:11,747 And it was really quick. 483 00:22:11,747 --> 00:22:12,107 Right, 484 00:22:12,227 --> 00:22:15,887 it was really quick because the key is, especially on a de-duplicated system, 485 00:22:16,187 --> 00:22:17,567 you're not actually copying the data. 486 00:22:17,567 --> 00:22:18,827 It's all pointer manipulation. 487 00:22:19,337 --> 00:22:20,207 right, right. 488 00:22:20,597 --> 00:22:24,527 moving any data, you're just changing metadata, which is very quick, very fast. 489 00:22:25,442 --> 00:22:27,707 That was a great singing group from the nineties, 490 00:22:28,562 --> 00:22:28,852 What 491 00:22:28,922 --> 00:22:29,042 I 492 00:22:29,092 --> 00:22:29,652 metadata. 493 00:22:29,882 --> 00:22:30,442 manipulation. 494 00:22:33,812 --> 00:22:39,002 But I'm, anyway, so yeah, so that was sort of a stop gap solution. 495 00:22:39,002 --> 00:22:44,342 It allowed people to stop doing real fulls, solve the issue of the, the, 496 00:22:44,342 --> 00:22:47,132 the load on the client, solve the issue of the load on the network. 497 00:22:47,642 --> 00:22:51,012 But we still, we were still creating fulls. 498 00:22:52,022 --> 00:22:54,842 So the concept of forever incremental. 499 00:22:55,622 --> 00:23:01,322 And, and true forever incremental is that we first have to design a backup product 500 00:23:01,322 --> 00:23:05,102 that just no longer needs repeated fulls. 501 00:23:05,522 --> 00:23:05,972 Right? 502 00:23:06,122 --> 00:23:11,372 And, and by the way, this generally does only apply to, um. 503 00:23:12,272 --> 00:23:14,912 Uh, file system backups for the most part. 504 00:23:15,362 --> 00:23:19,292 Um, not database and application backups. 505 00:23:19,592 --> 00:23:22,332 Most of them still require repeated fullss. 506 00:23:23,102 --> 00:23:27,752 Um, you know, we'll, we'll, some of, some of them have forever incremental, but. 507 00:23:27,767 --> 00:23:31,247 I would also say, or would you also say for virtualized 508 00:23:31,247 --> 00:23:34,277 workloads, this also applies 'cause 509 00:23:34,322 --> 00:23:38,042 Um, oh, you mean, uh, the, the backup of the di of the system itself? 510 00:23:38,222 --> 00:23:38,762 Yes. 511 00:23:38,762 --> 00:23:39,272 Yes. 512 00:23:39,422 --> 00:23:44,102 So, well, so I'll, so if I, if, if I was to use a term that would apply to, 513 00:23:45,242 --> 00:23:47,047 to both of them, I'd say volume-based. 514 00:23:47,357 --> 00:23:47,657 Yeah. 515 00:23:48,182 --> 00:23:48,692 Backups. 516 00:23:48,692 --> 00:23:48,932 Right? 517 00:23:48,932 --> 00:23:50,972 So either a file system or the, or a, 518 00:23:51,197 --> 00:23:51,797 Disc. 519 00:23:52,172 --> 00:23:54,002 file or a VDK file, right? 520 00:23:55,052 --> 00:24:00,002 Um, and the idea is that we're just gonna do one full, and then 521 00:24:00,002 --> 00:24:04,802 we're generally going to be doing block level incremental backups. 522 00:24:05,462 --> 00:24:09,092 Um, not just incremental, but block level incremental. 523 00:24:09,302 --> 00:24:11,192 What's the difference between those? 524 00:24:11,667 --> 00:24:15,767 Well, a block level incremental allows you to look at changes 525 00:24:15,767 --> 00:24:18,797 within, a specific object. 526 00:24:18,857 --> 00:24:22,637 As an example, say you had, uh, 10 gigabyte PST. 527 00:24:22,637 --> 00:24:27,407 File representing your email and instead of doing a normal incremental, which 528 00:24:27,407 --> 00:24:31,457 would transfer the entire 10 gig file over when you back it up using traditional 529 00:24:31,457 --> 00:24:36,557 incrementals, block level incremental would just take the changes in within that 530 00:24:36,557 --> 00:24:38,777 10 gig PST file and transfer that over. 531 00:24:39,692 --> 00:24:45,347 Exactly, and so it, it, it's another data reduction technology like deduplication. 532 00:24:45,977 --> 00:24:49,997 And it, it, it just, it, I I think it's a, it was an order of 533 00:24:49,997 --> 00:24:53,357 magnitude reduction in the number of bytes that has to be transferred. 534 00:24:53,867 --> 00:24:59,657 And, and the way this worked in, in, uh, file system backups was that 535 00:24:59,657 --> 00:25:04,727 you had to have like an API into the actual blocks, the, the discs, right. 536 00:25:05,237 --> 00:25:06,077 Uh, which. 537 00:25:06,447 --> 00:25:07,887 Was somewhat problematic. 538 00:25:07,917 --> 00:25:12,147 The, the, the easier way to get this was in the virtualization 539 00:25:12,147 --> 00:25:17,007 world where you had an API, right, like the change block tracking API 540 00:25:17,072 --> 00:25:17,492 Mm-hmm. 541 00:25:17,547 --> 00:25:21,837 you can just, the backup app can just say to the, the virtualization 542 00:25:21,837 --> 00:25:23,847 vendor, Hey, I'm here to back up again. 543 00:25:23,997 --> 00:25:26,607 Just give me a pointer to all of the blocks that have changed 544 00:25:26,607 --> 00:25:27,867 since the last time I was here. 545 00:25:27,932 --> 00:25:28,202 Yep. 546 00:25:28,587 --> 00:25:33,867 Um, and then we can transfer just those change blocks, but. 547 00:25:34,037 --> 00:25:34,257 But. 548 00:25:34,622 --> 00:25:41,852 importantly, for a true increment, for a true forever incremental, um, to, to, to, 549 00:25:41,972 --> 00:25:46,862 for me to think that it's a truly forever incremental is that you're going to change 550 00:25:46,862 --> 00:25:49,232 how the data is stored on the backend. 551 00:25:49,232 --> 00:25:52,832 So there really isn't a concept of full, right? 552 00:25:52,922 --> 00:25:59,342 Um, the way that you're storing each new backup is that each new backup from a 553 00:25:59,342 --> 00:26:02,162 restore perspective behaves like a full. 554 00:26:02,207 --> 00:26:02,387 Yeah. 555 00:26:03,092 --> 00:26:03,572 Right. 556 00:26:03,637 --> 00:26:03,857 You 557 00:26:04,097 --> 00:26:04,397 Yeah. 558 00:26:04,502 --> 00:26:05,852 want to like put that in? 559 00:26:06,032 --> 00:26:06,512 Go ahead. 560 00:26:06,557 --> 00:26:09,317 so I was just gonna put it in my own words, which 561 00:26:09,662 --> 00:26:10,052 Yeah. 562 00:26:10,097 --> 00:26:14,507 basically you're able to send only the changes and end up with 563 00:26:14,507 --> 00:26:16,547 the full on the backend, such 564 00:26:16,652 --> 00:26:16,892 Right. 565 00:26:17,177 --> 00:26:20,297 each one is a full, they can be independently restored without 566 00:26:20,297 --> 00:26:21,797 any dependencies on others. 567 00:26:22,712 --> 00:26:22,982 Yeah. 568 00:26:22,982 --> 00:26:27,452 So you don't have this idea of restoring the full and restoring 569 00:26:27,452 --> 00:26:29,072 the incrementals individually. 570 00:26:29,072 --> 00:26:29,342 Right. 571 00:26:29,342 --> 00:26:35,222 You just restore the late, you know, it's sort of, it's, it's sort of like take what 572 00:26:35,222 --> 00:26:37,732 we used to do, take the synthetic full. 573 00:26:38,252 --> 00:26:42,122 Every time we do an incremental, we're essentially creating a new synthetic full. 574 00:26:42,182 --> 00:26:42,452 Yep, 575 00:26:43,292 --> 00:26:43,892 How's that? 576 00:26:43,892 --> 00:26:44,222 yep. 577 00:26:45,032 --> 00:26:45,422 Yeah. 578 00:26:46,337 --> 00:26:49,757 Yeah, because if you think about it, it's just kind of changing 579 00:26:49,757 --> 00:26:51,527 where the processing happened, 580 00:26:52,117 --> 00:26:52,357 Right. 581 00:26:52,397 --> 00:26:57,077 Because with Synthetic fulls, the client was telling the storage, okay, these 582 00:26:57,077 --> 00:27:00,737 are the pieces I want you to stitch together, and it might have been pieces 583 00:27:00,737 --> 00:27:02,357 across multiple different backups, 584 00:27:02,487 --> 00:27:02,907 Mm-hmm. 585 00:27:03,137 --> 00:27:03,377 right? 586 00:27:03,377 --> 00:27:07,037 In the case of an incremental forever, it's like, Hey, you know 587 00:27:07,037 --> 00:27:11,327 your last backup go apply these changes on top of that last backup 588 00:27:11,827 --> 00:27:12,117 Yeah. 589 00:27:12,307 --> 00:27:12,597 Yeah. 590 00:27:12,617 --> 00:27:13,217 offsets. 591 00:27:13,892 --> 00:27:14,312 Right. 592 00:27:15,482 --> 00:27:18,212 Um, and go ahead. 593 00:27:18,917 --> 00:27:24,497 it needs to be a system that allows you to keep those copies as independent, 594 00:27:26,222 --> 00:27:27,242 What do you, what do you mean by that? 595 00:27:27,947 --> 00:27:33,617 um, uh, backup technologies which say, which claim to be incremental forever. 596 00:27:34,017 --> 00:27:34,437 Mm-hmm. 597 00:27:34,487 --> 00:27:35,867 They only keep the latest copy, 598 00:27:37,047 --> 00:27:37,397 Right. 599 00:27:37,697 --> 00:27:37,877 for 600 00:27:38,117 --> 00:27:38,237 I. 601 00:27:38,387 --> 00:27:42,047 you have a full, you apply the changes, you now get a new full. 602 00:27:42,347 --> 00:27:43,967 That old full is gone from the system. 603 00:27:44,897 --> 00:27:45,117 Hmm. 604 00:27:45,272 --> 00:27:45,722 Yeah. 605 00:27:46,022 --> 00:27:47,912 have the most recent copy, right? 606 00:27:47,912 --> 00:27:50,972 And normally for backup, because you want to be able to go back further in 607 00:27:50,972 --> 00:27:55,022 time, you need to make sure you are preserving those previous copies as well. 608 00:27:55,112 --> 00:27:55,982 And so it 609 00:27:56,132 --> 00:27:56,192 Yeah. 610 00:27:56,252 --> 00:28:00,932 require a technology like deduplication, which allows you to efficiently store 611 00:28:00,932 --> 00:28:05,402 multiple copies, snapshots, whatever technology is available on the backend 612 00:28:05,402 --> 00:28:06,782 in order to make that possible. 613 00:28:07,607 --> 00:28:11,327 Yeah, so snapshots and replication. 614 00:28:11,417 --> 00:28:14,447 A great example of forever incremental technology, right? 615 00:28:14,927 --> 00:28:16,307 Each new, um. 616 00:28:17,207 --> 00:28:21,017 Each time you do a snapshot and a replication, you get a new, something 617 00:28:21,017 --> 00:28:25,847 that looks like a full, uh, when really all you only transferred was the, the, 618 00:28:26,177 --> 00:28:27,617 you know, the bites that have changed. 619 00:28:28,277 --> 00:28:32,897 And then, um, I can, I can immediately think of when I was, 620 00:28:32,987 --> 00:28:34,847 when you were talking, I was like, what is he talking about? 621 00:28:34,847 --> 00:28:36,137 And then I realized what you were talking about. 622 00:28:36,437 --> 00:28:40,127 So here's an example of forever incremental that isn't what we're 623 00:28:40,127 --> 00:28:41,837 talking about, and that's replication. 624 00:28:41,842 --> 00:28:41,902 It 625 00:28:42,617 --> 00:28:42,827 Yep. 626 00:28:42,922 --> 00:28:43,402 Right. 627 00:28:43,522 --> 00:28:46,582 Re just regular replication without anything 628 00:28:46,727 --> 00:28:46,967 Yep, 629 00:28:47,182 --> 00:28:47,812 beside it. 630 00:28:47,992 --> 00:28:52,882 You end up with each new, every time you do a a, an if, if you want to call 631 00:28:52,882 --> 00:28:57,802 it that, in an incremental backup, all you're left with is the latest 632 00:28:57,802 --> 00:29:00,082 version and you can't go back in time. 633 00:29:00,467 --> 00:29:03,592 You know, you know what I was actually thinking of when I made those comments? 634 00:29:03,957 --> 00:29:04,792 Oh, what were you, 635 00:29:05,252 --> 00:29:06,512 Oracle incremental merge 636 00:29:07,642 --> 00:29:08,752 oh, okay. 637 00:29:09,392 --> 00:29:12,962 because Oracle incremental merge applies to changes on top of a full, but it 638 00:29:12,962 --> 00:29:14,672 does not preserve the previous versions. 639 00:29:15,467 --> 00:29:15,977 Right. 640 00:29:15,977 --> 00:29:17,147 Yeah, that's a good point. 641 00:29:17,267 --> 00:29:22,697 Um, which, which wouldn't it, that's really just a, a type 642 00:29:22,697 --> 00:29:24,297 of, synthetic full, right? 643 00:29:25,227 --> 00:29:28,898 Yeah, yeah, I didn't even realize that, by the way. 644 00:29:29,378 --> 00:29:32,438 Um, so how do you preserve the history? 645 00:29:32,558 --> 00:29:32,948 That's why 646 00:29:33,068 --> 00:29:33,548 Do you just. 647 00:29:33,668 --> 00:29:36,728 do you need a application that, or a storage system that 648 00:29:36,728 --> 00:29:38,648 allows you to take snapshots or 649 00:29:38,843 --> 00:29:39,113 okay. 650 00:29:39,338 --> 00:29:40,868 copy of the file before it 651 00:29:40,913 --> 00:29:41,235 gotcha. 652 00:29:41,320 --> 00:29:41,560 Why do 653 00:29:41,575 --> 00:29:42,145 where was it? 654 00:29:42,250 --> 00:29:42,610 Curtis? 655 00:29:42,610 --> 00:29:44,590 Why do I still remember these things? 656 00:29:45,689 --> 00:29:47,009 It's why you're here? 657 00:29:48,189 --> 00:29:49,179 So we've defined what a 658 00:29:49,194 --> 00:29:51,174 Mm-hmm. 659 00:29:51,219 --> 00:29:52,749 a little bit about the benefits, 660 00:29:53,394 --> 00:29:53,814 Yeah. 661 00:29:54,069 --> 00:29:59,949 Being able to quickly restore because you have the full copy, uh, being independent, 662 00:30:00,309 --> 00:30:03,039 um, not having to transfer all the data. 663 00:30:03,399 --> 00:30:03,789 Right? 664 00:30:04,149 --> 00:30:05,199 Um, can 665 00:30:05,274 --> 00:30:07,104 And also also cost savings. 666 00:30:07,104 --> 00:30:07,944 Right, right. 667 00:30:08,004 --> 00:30:08,184 Yeah. 668 00:30:08,199 --> 00:30:09,399 you think of any cons? 669 00:30:11,934 --> 00:30:16,074 Well, the, the only thing you know, one of the things I said 670 00:30:16,074 --> 00:30:17,994 in the beginning was that. 671 00:30:19,179 --> 00:30:26,409 In order to do forever, incremental, successfully, you've 672 00:30:26,409 --> 00:30:28,269 got to store the data on disc. 673 00:30:28,929 --> 00:30:32,889 And, and again, not, not to be, you know, pro tape and all that stuff, 674 00:30:32,889 --> 00:30:37,479 but we did an episode relatively recently where we talked about how you 675 00:30:37,479 --> 00:30:40,779 have to, uh, protect the backup disc. 676 00:30:40,779 --> 00:30:43,749 The only con that I can think of is that your entire backup is just sitting 677 00:30:43,749 --> 00:30:45,699 there ready to be attacked, right. 678 00:30:45,729 --> 00:30:45,969 Yeah, 679 00:30:46,239 --> 00:30:46,569 Um. 680 00:30:47,019 --> 00:30:48,039 I have, I have a 681 00:30:48,234 --> 00:30:48,654 Go ahead. 682 00:30:48,654 --> 00:30:50,034 You have, you have a con, okay? 683 00:30:50,709 --> 00:30:53,619 So having worked with vendors, and I'm not saying this has happened, but 684 00:30:53,619 --> 00:30:57,159 it is a potential thing you should think about is when you have forever 685 00:30:57,189 --> 00:31:00,039 incrementals, You're always re. 686 00:31:00,924 --> 00:31:03,114 Copying over the existing backup 687 00:31:03,654 --> 00:31:05,304 Mm-hmm. 688 00:31:05,394 --> 00:31:11,004 bug that somehow makes, that corrupts a part of your backup, right? 689 00:31:11,004 --> 00:31:16,134 You've now impacted how many versions back because you're never getting a 690 00:31:16,134 --> 00:31:18,024 full copy of that data again, right? 691 00:31:18,024 --> 00:31:18,594 It's always 692 00:31:18,714 --> 00:31:19,164 Right, 693 00:31:19,644 --> 00:31:19,944 And so 694 00:31:20,274 --> 00:31:20,754 Yeah. 695 00:31:21,144 --> 00:31:23,724 be one of the cons a 696 00:31:23,784 --> 00:31:24,294 Yeah. 697 00:31:24,924 --> 00:31:25,224 approach. 698 00:31:25,344 --> 00:31:30,624 That was, that was a, um, a critique of ddu, uh, because that 699 00:31:30,624 --> 00:31:33,444 has the same issue, a critique of dedupe when Dedupe first came out. 700 00:31:34,104 --> 00:31:37,944 And all you can say is that you just do your best to ensure 701 00:31:37,944 --> 00:31:38,964 that that's not happening. 702 00:31:38,964 --> 00:31:41,514 And that's why we do fingerprinting and, and, you know, integrity 703 00:31:41,514 --> 00:31:42,744 checks and all that kind of stuff. 704 00:31:42,999 --> 00:31:43,599 your backup. 705 00:31:44,274 --> 00:31:44,664 Yes. 706 00:31:44,664 --> 00:31:45,864 And also test your backups. 707 00:31:45,864 --> 00:31:46,194 Right. 708 00:31:46,344 --> 00:31:48,264 Test your backup, test your backup, test your backups. 709 00:31:49,314 --> 00:31:52,794 Um, but go ahead. 710 00:31:52,869 --> 00:31:53,799 I do have another one too. 711 00:31:53,799 --> 00:31:54,339 If you, unless you 712 00:31:54,444 --> 00:31:54,774 Oh, you do? 713 00:31:54,774 --> 00:31:54,804 Okay. 714 00:31:55,109 --> 00:31:55,529 Go ahead. 715 00:31:56,484 --> 00:32:00,744 The other one I also wanted to mention is, right if Forever incremental backups 716 00:32:00,744 --> 00:32:03,084 are so great, why isn't everyone using it? 717 00:32:03,629 --> 00:32:04,049 Mm-hmm. 718 00:32:04,164 --> 00:32:07,794 one of the challenges is it requires vendor support. 719 00:32:08,819 --> 00:32:09,109 Yeah. 720 00:32:09,114 --> 00:32:10,554 application to support it. 721 00:32:10,554 --> 00:32:12,924 So VMware is supported for virtualization. 722 00:32:13,734 --> 00:32:15,834 companies support it for their own applications, but 723 00:32:15,834 --> 00:32:17,094 not everyone supports it. 724 00:32:17,094 --> 00:32:18,594 Not everyone makes it available. 725 00:32:19,149 --> 00:32:19,509 Yeah. 726 00:32:19,584 --> 00:32:24,024 so you sort of are limited in terms of what applications, what 727 00:32:24,024 --> 00:32:27,384 APIs are available, and can you get the changes that you need in 728 00:32:27,384 --> 00:32:29,664 order to be able to create your forever incremental in the backend. 729 00:32:30,699 --> 00:32:35,019 Exactly, and, and know, I can think of modern database applications 730 00:32:35,619 --> 00:32:37,509 that do support it, right? 731 00:32:37,569 --> 00:32:42,129 Um, I can think, for example, uh, when you back up Salesforce, right? 732 00:32:42,129 --> 00:32:44,424 You're backing up at the object level and it's very easy to. 733 00:32:45,384 --> 00:32:48,504 Uh, and, and let me rephrase that, because Salesforce uses 734 00:32:48,504 --> 00:32:50,964 the term object to mean something very different than what we mean. 735 00:32:51,474 --> 00:32:54,624 Um, because why should we all use the same terms? 736 00:32:54,789 --> 00:32:55,509 of course. 737 00:32:55,644 --> 00:32:58,764 A Salesforce object is like the user's table, right? 738 00:32:58,794 --> 00:33:05,664 Um, so I'm talking about like, I made an update to the, to a, you know, a 739 00:33:05,664 --> 00:33:10,464 field in a record in a table, and, um. 740 00:33:11,079 --> 00:33:17,439 Uh, you can back up that change as right, and, and, and you can do it incrementally 741 00:33:17,439 --> 00:33:19,479 or you can do it independently. 742 00:33:19,959 --> 00:33:23,529 And so it would be very easy to store all of those, those 743 00:33:23,529 --> 00:33:25,839 objects in a, in a way that. 744 00:33:25,839 --> 00:33:30,009 It, you know, would, would foster this idea of a forever incremental. 745 00:33:30,459 --> 00:33:37,389 Um, whereas like Oracle for example, there really isn't a way to do it. 746 00:33:37,809 --> 00:33:43,659 The, at best what we can get out of Oracle is the concept of a, um, synthetic full, 747 00:33:44,289 --> 00:33:46,834 uh, and the synthetic full that they have. 748 00:33:47,384 --> 00:33:51,524 Has the issue that you, that you mentioned before, whereas once you 749 00:33:51,524 --> 00:33:54,434 do the synthetic full, you don't have the previous incrementals. 750 00:33:54,614 --> 00:33:59,774 Um, so yeah, it, that is, that is definitely a, um, it, it 751 00:33:59,774 --> 00:34:02,594 requires your backup vendors to support it, and it requires your 752 00:34:02,594 --> 00:34:04,424 application vendors to support it. 753 00:34:04,499 --> 00:34:04,649 Yeah. 754 00:34:05,049 --> 00:34:05,649 Absolutely. 755 00:34:07,419 --> 00:34:12,024 I can't remember where I was going before you brought up that other, um, I, I, 756 00:34:12,029 --> 00:34:16,719 I will say this, that, you know, the title of this is, you know, how Forever 757 00:34:16,719 --> 00:34:18,219 Incremental Backup changed the world. 758 00:34:18,279 --> 00:34:20,289 It, it, it really has, right? 759 00:34:20,289 --> 00:34:24,519 The idea, when you look at these modern backup products, the 760 00:34:24,519 --> 00:34:28,809 idea of doing a, of a, of a full backup is just, it's just gone. 761 00:34:29,019 --> 00:34:29,439 Right? 762 00:34:29,499 --> 00:34:32,649 Um, all of the downsides to doing a regular full backup. 763 00:34:33,204 --> 00:34:34,104 Are, are gone. 764 00:34:34,314 --> 00:34:37,224 Uh, the, the, the, the, the compute considerations, the 765 00:34:37,224 --> 00:34:42,114 network considerations, the storage considerations, they're all gone without 766 00:34:42,384 --> 00:34:46,014 any negative like impact to the restore. 767 00:34:46,014 --> 00:34:47,454 In fact, it's the opposite. 768 00:34:47,454 --> 00:34:53,424 The restore is now, since we know what a full looks like when we go to do the 769 00:34:53,424 --> 00:34:58,194 restore, that is to going back to Harken, back to the original way we did restores. 770 00:34:59,229 --> 00:35:03,819 We're no longer restoring the same file multiple times during a single restore. 771 00:35:04,179 --> 00:35:05,259 Why would we do that? 772 00:35:05,259 --> 00:35:08,169 We know where all the files are, we know where all the blocks are, whatever 773 00:35:08,169 --> 00:35:09,699 it is that we're, we're restoring. 774 00:35:10,659 --> 00:35:14,169 We know what the latest version of that block here file is, and 775 00:35:14,169 --> 00:35:15,429 we're just gonna restore that one. 776 00:35:15,484 --> 00:35:16,144 I have a story. 777 00:35:17,854 --> 00:35:22,086 Um, how you do forever, incremental Matters. 778 00:35:23,376 --> 00:35:28,926 The original, the OG, forever incremental product is TSM. 779 00:35:29,151 --> 00:35:29,631 mm-hmm. 780 00:35:30,546 --> 00:35:36,846 And when, and the way it was implemented back then was to tape. 781 00:35:38,031 --> 00:35:45,761 And they, tried to bring this concept of forever incremental to the world without 782 00:35:45,761 --> 00:35:49,541 changing the underlying storage technology because back then it was, it was, uh. 783 00:35:49,721 --> 00:35:50,141 tape. 784 00:35:51,041 --> 00:35:55,391 It was, uh, if you, if you brought disc into it, it was, it was crazy talk, right? 785 00:35:55,871 --> 00:35:59,981 So the, so the, so, so they can get a lot of credit for trying to 786 00:35:59,981 --> 00:36:03,551 accomplish this, this, um, this idea. 787 00:36:04,301 --> 00:36:09,941 But I, I still think when we look back to TSM of old, I, I think it was, I 788 00:36:09,941 --> 00:36:14,889 think it was a failed attempt because my earlier statement of the only proper way 789 00:36:14,889 --> 00:36:18,669 to do forever incremental is to do it on disc when they were doing it on tape. 790 00:36:20,484 --> 00:36:27,294 What it meant was that when you, um, when you went to go to do a restore, the, the, 791 00:36:27,294 --> 00:36:32,424 the full was on like 5,700 tapes, right? 792 00:36:32,574 --> 00:36:35,589 sounds worse than just doing your traditional fulls and incrementals. 793 00:36:36,444 --> 00:36:41,244 Yeah, well, it was, and I remember, I, I remember doing a, there was a one 794 00:36:41,244 --> 00:36:46,734 particular restore that I remember doing using TSM in a production environment. 795 00:36:47,274 --> 00:36:51,654 And I, I don't remember the actual numbers, but I'll, I'll just use like, 796 00:36:51,744 --> 00:36:56,274 uh, I'll just use fake numbers just to, to, to to, to describe what it was like. 797 00:36:57,084 --> 00:37:03,264 It was like we were doing like a 10 gigabyte restore and it took like. 798 00:37:03,849 --> 00:37:10,509 Two weeks because of how many hundreds of tapes that it was. 799 00:37:10,509 --> 00:37:11,589 Thousands of tapes. 800 00:37:12,129 --> 00:37:16,059 Um, because of the number of files, the number of tapes that they were 801 00:37:16,059 --> 00:37:21,699 on, and how long it takes to load and unload an individual tape. 802 00:37:21,789 --> 00:37:22,179 Right? 803 00:37:22,629 --> 00:37:28,329 Um, and they had technologies, uh, called reclamation and they, they, 804 00:37:28,329 --> 00:37:31,899 they had a number of technologies that tried to mini minimize this issue. 805 00:37:32,814 --> 00:37:34,194 Um, but 806 00:37:34,794 --> 00:37:35,574 It still existed. 807 00:37:36,354 --> 00:37:38,214 well, yeah, it still is existed. 808 00:37:38,214 --> 00:37:42,774 And also some people, they would run out of time in the day to do reclamation. 809 00:37:42,774 --> 00:37:48,084 Reclamation is where you take a bunch of tapes that, um, like they were, that 810 00:37:48,084 --> 00:37:52,104 most of the files on those, on those tapes were no longer needed because 811 00:37:52,104 --> 00:37:55,344 they, they'd expired and then you take, yeah, I've got a bunch of files that are 812 00:37:55,344 --> 00:37:58,134 10% and then you copy them to one tape 813 00:37:58,284 --> 00:37:58,464 It's 814 00:37:58,464 --> 00:37:59,784 to that's now a hundred percent. 815 00:38:00,159 --> 00:38:01,749 It's like garbage collection, but on tape. 816 00:38:02,349 --> 00:38:08,589 And it was a very, um, labor intensive process and, um, IO intensive process. 817 00:38:08,619 --> 00:38:10,509 And as a result, it took a long time. 818 00:38:10,509 --> 00:38:12,579 And so some people would, would turn it off. 819 00:38:13,089 --> 00:38:16,539 Um, and, uh, I think that's what happened in, in, in this case. 820 00:38:16,539 --> 00:38:20,169 And so there, there were literally just, I don't rem there were so many 821 00:38:20,169 --> 00:38:23,229 tapes that we had to restore and it, you know, it sounds like I'm bagging 822 00:38:23,229 --> 00:38:25,269 on T-S-M-T-S-M is still around. 823 00:38:25,269 --> 00:38:27,159 It's now called, uh, spectrum Protect. 824 00:38:27,564 --> 00:38:32,244 Yeah, but the addition of bringing in disc, I think makes this, this whole 825 00:38:32,244 --> 00:38:34,314 issue kind of, uh, not a big deal, 826 00:38:34,329 --> 00:38:34,569 Yeah. 827 00:38:35,544 --> 00:38:40,354 but, um, but yeah, I still think that you look at all of the modern backup 828 00:38:40,374 --> 00:38:44,724 products that have come out in the last 10 plus years, and they're all like this. 829 00:38:44,724 --> 00:38:48,384 They're all one full, followed by a whole bunch of incrementals, and 830 00:38:48,384 --> 00:38:49,764 the world is a better place for it. 831 00:38:49,974 --> 00:38:53,664 And, um, you young whippersnappers have never had to deal with 832 00:38:53,679 --> 00:38:54,159 tape 833 00:38:54,954 --> 00:38:55,914 incremental backups. 834 00:38:56,559 --> 00:38:57,519 You just don't know 835 00:38:57,639 --> 00:38:57,789 what 836 00:38:57,789 --> 00:38:59,679 what you uh, yeah. 837 00:38:59,679 --> 00:39:01,509 You don't know the pain that you, that Yeah. 838 00:39:01,509 --> 00:39:02,109 That you had. 839 00:39:02,379 --> 00:39:05,079 You know, it's funny, this morning I was hanging out with some people and I 840 00:39:05,079 --> 00:39:09,189 was talking to 'em, and they're, it's this lady who shares my birthday, but 841 00:39:09,189 --> 00:39:11,169 she's about 15 years older than me, 842 00:39:11,259 --> 00:39:11,589 Hmm. 843 00:39:12,099 --> 00:39:16,749 we started talking about all of the technology that we 844 00:39:16,749 --> 00:39:17,919 remember when it came out. 845 00:39:18,009 --> 00:39:18,399 Right. 846 00:39:18,909 --> 00:39:20,259 And the first, the first. 847 00:39:20,634 --> 00:39:25,944 Piece of technology I remember being introduced in my life was the, the VCR 848 00:39:26,224 --> 00:39:26,304 Hmm. 849 00:39:26,454 --> 00:39:30,594 and it, it came, they, I remember them wheeling one into the classroom 850 00:39:30,939 --> 00:39:31,019 Hmm. 851 00:39:31,254 --> 00:39:32,124 in my school. 852 00:39:32,124 --> 00:39:33,924 And it was gigantic. 853 00:39:33,924 --> 00:39:34,314 Right. 854 00:39:34,314 --> 00:39:38,304 It was like, and it was the kind where you put the tape in and push it down, right? 855 00:39:39,174 --> 00:39:39,504 Yeah. 856 00:39:39,744 --> 00:39:43,644 Um, anyway, this is, this is like that, this is like the 857 00:39:43,644 --> 00:39:45,084 way, the way we did backups. 858 00:39:45,084 --> 00:39:47,274 And by the way, you know. 859 00:39:48,429 --> 00:39:51,459 I'm, I'm glad I, I'm glad I remembered this because you asked a question earlier. 860 00:39:51,804 --> 00:39:55,044 Why is it that not everybody does things in this new way? 861 00:39:55,044 --> 00:39:55,104 Yeah. 862 00:39:55,494 --> 00:39:58,944 Because backup is the stickiest application in the data center. 863 00:39:59,049 --> 00:39:59,269 Yep. 864 00:39:59,304 --> 00:40:02,574 So many people are still using the same backup application 865 00:40:02,574 --> 00:40:04,254 that they were 15 years ago. 866 00:40:04,674 --> 00:40:06,654 Some people change their applications like they change 867 00:40:06,654 --> 00:40:08,364 their underwear, but other people. 868 00:40:08,784 --> 00:40:10,884 They just, it, it's a pain. 869 00:40:11,214 --> 00:40:16,374 It's costly, it's risk, uh, it's fraught with risk and, and, and 870 00:40:16,434 --> 00:40:18,624 difficulty and a giant learning curve. 871 00:40:18,624 --> 00:40:21,174 And during that learning curve, your, your data's at risk. 872 00:40:21,354 --> 00:40:26,964 And so most people just, they'd rather stick with the ugly, old backup product. 873 00:40:27,174 --> 00:40:27,414 it. 874 00:40:27,684 --> 00:40:28,524 Because it works. 875 00:40:28,524 --> 00:40:28,914 Yeah. 876 00:40:29,004 --> 00:40:32,784 And, and, and so it, it, it gave birth to incrementally helpful 877 00:40:32,784 --> 00:40:34,704 technologies like deduplication. 878 00:40:35,154 --> 00:40:36,534 Don't change anything. 879 00:40:36,684 --> 00:40:39,984 Just send your backups over here instead of sending your backups over there. 880 00:40:39,984 --> 00:40:43,344 And we'll get rid of the duplicates and we'll make it sort of look like a, 881 00:40:43,584 --> 00:40:49,164 an incremental forever, but just keep doing it the dumb old way because we 882 00:40:49,164 --> 00:40:52,044 don't want you to have to change your backup product in order to get to, 883 00:40:52,044 --> 00:40:54,624 and, and I guess this is, you know. 884 00:40:55,529 --> 00:40:58,554 If, if you're one of those people that's still using the backup, the 885 00:40:58,554 --> 00:41:01,224 same backup product that you had 15 years ago and you're still using it 886 00:41:01,224 --> 00:41:04,914 the same way you were using it 15 years ago, and you're not using incremental 887 00:41:04,914 --> 00:41:12,174 forever as a, a true technology in your world, maybe time to think about it. 888 00:41:12,279 --> 00:41:14,199 but I, I would probably say. 889 00:41:15,054 --> 00:41:18,474 Newer workloads, people are probably using that, right? 890 00:41:18,474 --> 00:41:22,584 Because once again, environments are a mishmash of technologies, right? 891 00:41:22,644 --> 00:41:23,004 Yeah. 892 00:41:23,604 --> 00:41:26,994 people probably do have those old databases, old applications, 893 00:41:26,994 --> 00:41:30,864 which they are using an old infrastructure backup infrastructure 894 00:41:30,864 --> 00:41:35,064 for, and for their virtualization or other newer workloads, they're 895 00:41:35,064 --> 00:41:36,294 probably using something different. 896 00:41:37,179 --> 00:41:37,569 Yeah. 897 00:41:37,629 --> 00:41:42,459 Uh, remember I did that, I did that survey a few, uh, months ago and 898 00:41:42,459 --> 00:41:47,799 I asked them how many backup apps they had, and 30% had 30, I think 899 00:41:47,799 --> 00:41:50,529 more than 30% had three or more backup apps in their environment. 900 00:41:50,564 --> 00:41:50,784 Yep. 901 00:41:50,919 --> 00:41:52,359 And it's because of what you're talking about. 902 00:41:52,359 --> 00:41:53,619 They have, they start using. 903 00:41:53,994 --> 00:41:57,594 Containers and their current backup app doesn't know what to do. 904 00:41:57,594 --> 00:42:00,054 And so they use that for the new technology and yeah. 905 00:42:00,234 --> 00:42:01,404 Yeah, that's very common. 906 00:42:01,464 --> 00:42:05,154 And I guess what I'm saying is maybe it's time to check out the water of 907 00:42:05,154 --> 00:42:10,014 the, of, of truly incremental backups and, and experience all the beauty. 908 00:42:11,634 --> 00:42:11,964 Yeah. 909 00:42:12,204 --> 00:42:12,594 All right. 910 00:42:12,624 --> 00:42:14,514 Well thanks for the chat again. 911 00:42:14,904 --> 00:42:15,864 Anytime Curtis. 912 00:42:15,864 --> 00:42:16,554 Anytime. 913 00:42:17,094 --> 00:42:17,424 All right. 914 00:42:17,474 --> 00:42:18,854 And we wanna thank our listeners. 915 00:42:19,304 --> 00:42:22,334 Yo, you're why we're here, you know, otherwise, just two guys sitting here 916 00:42:22,334 --> 00:42:24,314 talking over a couple of microphones. 917 00:42:24,734 --> 00:42:26,324 Uh, that is a wrap.