1 00:00:00,297 --> 00:00:02,907 If you're responsible for backup and Dr. 2 00:00:02,937 --> 00:00:07,497 At some point, someone is going to tell you about their amazing product based 3 00:00:07,497 --> 00:00:11,037 on continuous data protection or CDP. 4 00:00:11,487 --> 00:00:16,947 They say they can meet an RTO and RPO of zero, which sounds great. 5 00:00:17,187 --> 00:00:19,587 Why don't we do all backups and Dr. 6 00:00:19,647 --> 00:00:20,967 Using this method. 7 00:00:21,387 --> 00:00:22,297 Hi, I'm W. 8 00:00:22,297 --> 00:00:24,417 Curtis Preston, AKA Mister backup. 9 00:00:24,417 --> 00:00:28,767 And I started this podcast to turn unappreciated, backup admins 10 00:00:28,767 --> 00:00:30,567 into cyber recovery heroes. 11 00:00:31,077 --> 00:00:35,217 This episode will answer all your questions about CDP, which 12 00:00:35,217 --> 00:00:37,047 some say is the next great thing. 13 00:00:37,047 --> 00:00:37,827 And Dr. 14 00:00:38,277 --> 00:00:40,287 This is the backup wrap-up. 15 00:01:01,852 --> 00:01:03,102 Hi and welcome to the show. 16 00:01:03,102 --> 00:01:04,772 And once again, I have a guy 17 00:01:04,772 --> 00:01:06,822 who cost me money. 18 00:01:07,002 --> 00:01:10,382 Prasanna Malaiyandi how's it going, Prasanna? 19 00:01:10,577 --> 00:01:11,497 I'm good. 20 00:01:11,497 --> 00:01:12,547 I'm worried about 21 00:01:12,617 --> 00:01:13,387 what I'm going to 22 00:01:13,387 --> 00:01:14,117 be blamed for 23 00:01:14,117 --> 00:01:14,527 now. 24 00:01:14,527 --> 00:01:16,817 Well, I think, I think that, you know, 25 00:01:16,827 --> 00:01:17,777 the, the fact that I 26 00:01:17,777 --> 00:01:21,487 have new AirPods is your fault. 27 00:01:21,997 --> 00:01:22,247 What do you 28 00:01:22,307 --> 00:01:22,807 So, 29 00:01:24,477 --> 00:01:25,187 uh, no. 30 00:01:25,867 --> 00:01:26,367 the fact that 31 00:01:26,367 --> 00:01:27,967 you lost your AirPods. 32 00:01:28,267 --> 00:01:29,677 I think that you 33 00:01:29,707 --> 00:01:30,867 manifested it. 34 00:01:31,527 --> 00:01:37,377 You were suggesting that I needed new AirPods and I think my current AirPods got 35 00:01:37,487 --> 00:01:38,227 upset 36 00:01:38,397 --> 00:01:39,817 and then they literally flew out 37 00:01:39,817 --> 00:01:40,457 of my pocket. 38 00:01:41,897 --> 00:01:43,807 They're like, doo doo doo doo doo doo 39 00:01:44,137 --> 00:01:45,467 Yeah, it was the weirdest thing. 40 00:01:45,467 --> 00:01:47,317 Like, I, I had, like I, I, I've 41 00:01:47,317 --> 00:01:47,567 done. 42 00:01:47,872 --> 00:01:49,022 really good job 43 00:01:49,022 --> 00:01:50,512 with holding on my ear pods. 44 00:01:50,512 --> 00:01:51,192 And then I was, 45 00:01:51,862 --> 00:01:52,412 I was 46 00:01:52,512 --> 00:01:58,437 at a restaurant and, um, you know, having 47 00:01:58,437 --> 00:02:01,817 a date with my lovely wife was great and 48 00:02:02,417 --> 00:02:04,237 I pulled, pulled 49 00:02:04,247 --> 00:02:05,437 the thing out of my 50 00:02:05,437 --> 00:02:08,317 pocket and literally the case, like, flipped, 51 00:02:08,847 --> 00:02:11,547 like, open and the AirPod just went flying. 52 00:02:11,567 --> 00:02:15,157 And I don't, it was, it was such a weird thing that I didn't even realize 53 00:02:15,157 --> 00:02:16,367 it happened when it happened. 54 00:02:16,787 --> 00:02:17,127 It wasn't 55 00:02:17,127 --> 00:02:21,347 until I got home and I realized that both my AirPods were no longer in it 56 00:02:22,527 --> 00:02:26,807 And, uh, yeah, so I just like, in a moment like that I lost my AirPods. 57 00:02:27,087 --> 00:02:31,597 So I think what you need is a case that has one of those clasps on it. 58 00:02:32,147 --> 00:02:32,427 Right? 59 00:02:32,427 --> 00:02:35,097 So you have to undo the clasps in order for it to open. 60 00:02:35,177 --> 00:02:35,477 oh, 61 00:02:36,777 --> 00:02:37,027 Right? 62 00:02:37,027 --> 00:02:38,237 Because just the normal 63 00:02:38,237 --> 00:02:40,867 Silicon ones, I don't think will be sufficient for you. 64 00:02:41,587 --> 00:02:41,847 Yeah, 65 00:02:41,847 --> 00:02:42,257 apparently 66 00:02:42,257 --> 00:02:42,577 not. 67 00:02:43,417 --> 00:02:44,387 But, hey, I've got the new 68 00:02:44,387 --> 00:02:48,397 the new fancy AirPods Pro Generation 2 USB C, 69 00:02:48,607 --> 00:02:50,477 which really should be called Generation 3. 70 00:02:51,127 --> 00:02:52,317 But, You 71 00:02:52,317 --> 00:02:53,457 know, because I had to 72 00:02:53,547 --> 00:02:53,837 very 73 00:02:53,837 --> 00:02:56,277 specifically make sure that I bought the one with the USB C. 74 00:02:56,637 --> 00:02:57,177 well, it's 75 00:02:57,197 --> 00:02:59,547 because the actual thing is the same. 76 00:03:01,627 --> 00:03:04,217 Yeah, well, yeah, but you know what I'm saying. 77 00:03:04,287 --> 00:03:04,577 I mean, 78 00:03:04,627 --> 00:03:05,147 I know 79 00:03:05,777 --> 00:03:08,737 like I was going to buy it at Costco, but Costco only has 80 00:03:08,767 --> 00:03:09,937 the, uh, the older 81 00:03:10,037 --> 00:03:10,577 Shantoos. 82 00:03:10,677 --> 00:03:11,057 Yep. 83 00:03:12,397 --> 00:03:12,527 Yeah, 84 00:03:13,147 --> 00:03:13,397 but, 85 00:03:13,437 --> 00:03:13,687 uh, 86 00:03:14,257 --> 00:03:16,017 Tough life you live, Curtis. 87 00:03:16,067 --> 00:03:18,577 Uh, I'm just waiting to listen to what I'll be blamed for next. 88 00:03:20,047 --> 00:03:20,707 Absolutely. 89 00:03:21,427 --> 00:03:21,847 So, 90 00:03:21,887 --> 00:03:24,367 uh, we're speaking of blaming. 91 00:03:24,697 --> 00:03:26,377 We got blame to go around. 92 00:03:26,817 --> 00:03:27,037 I, 93 00:03:27,037 --> 00:03:30,057 I, I, think we should take credit for this, for this piece of 94 00:03:30,057 --> 00:03:30,547 news. 95 00:03:30,877 --> 00:03:31,607 What do you think? 96 00:03:32,387 --> 00:03:33,017 Oh, 97 00:03:33,017 --> 00:03:33,827 Curtis. 98 00:03:33,887 --> 00:03:36,967 Yes, it's our ability to 99 00:03:36,967 --> 00:03:41,467 expose to the listeners, hey, here's what ransomware is, that... 100 00:03:41,717 --> 00:03:42,767 Yeah, I think you're right. 101 00:03:42,857 --> 00:03:43,337 We... 102 00:03:43,507 --> 00:03:44,267 You think, you think, 103 00:03:44,917 --> 00:03:46,217 But is it a good or a bad thing, though? 104 00:03:46,217 --> 00:03:47,507 That's my question, this article. 105 00:03:48,397 --> 00:03:49,387 well, I actually think 106 00:03:49,387 --> 00:03:49,977 it's a good thing. 107 00:03:50,077 --> 00:03:50,957 Let's, so let's talk about 108 00:03:50,957 --> 00:03:51,077 it. 109 00:03:51,077 --> 00:03:54,217 So the, the headline, and it's from a story in the register. 110 00:03:54,867 --> 00:03:55,507 ransomware 111 00:03:55,507 --> 00:03:57,087 attacks register, 112 00:03:58,867 --> 00:04:01,967 It's a bit, it's funny I realized that the word register was in the 113 00:04:02,187 --> 00:04:03,617 title and it messed me up there. 114 00:04:04,327 --> 00:04:05,027 Ransomware 115 00:04:05,057 --> 00:04:09,987 attacks register record speeds thanks to successive InfoSec 116 00:04:10,017 --> 00:04:10,787 industry. 117 00:04:11,227 --> 00:04:11,897 So when I 118 00:04:11,917 --> 00:04:13,367 first heard that, I 119 00:04:13,367 --> 00:04:14,567 was like Wait, 120 00:04:14,987 --> 00:04:18,407 I, you know, that one, that one literally, uh, 121 00:04:18,427 --> 00:04:18,997 threw me. 122 00:04:19,217 --> 00:04:21,677 So the subtitle here is dwell times 123 00:04:21,937 --> 00:04:24,867 drop to hours rather than days for the first time. 124 00:04:24,867 --> 00:04:25,737 So first off, 125 00:04:26,217 --> 00:04:29,127 do you want to explain what a dwell time is for those of our listeners 126 00:04:29,127 --> 00:04:29,637 that don't know? 127 00:04:29,867 --> 00:04:30,247 Yeah. 128 00:04:30,507 --> 00:04:33,567 yeah, so in the past with ransomware, I think 129 00:04:34,117 --> 00:04:34,547 before 130 00:04:34,547 --> 00:04:35,007 it used to be 131 00:04:35,007 --> 00:04:35,817 measured, like you said, 132 00:04:35,817 --> 00:04:38,067 in days, like four and a half to five and a half days in the 133 00:04:38,067 --> 00:04:38,887 last couple of years. 134 00:04:38,887 --> 00:04:39,107 Right. 135 00:04:39,137 --> 00:04:42,527 But this is basically the amount of time that 136 00:04:42,527 --> 00:04:44,267 ransomware is in your system. 137 00:04:44,267 --> 00:04:44,947 So someone 138 00:04:44,957 --> 00:04:47,967 has attacked, infiltrated your systems, they've dropped a package. 139 00:04:48,297 --> 00:04:49,377 It hasn't done anything though, 140 00:04:51,157 --> 00:04:51,487 Right. 141 00:04:51,547 --> 00:04:52,327 It's just sitting 142 00:04:52,327 --> 00:04:53,047 there and waiting. 143 00:04:54,347 --> 00:04:54,737 Right, 144 00:04:54,847 --> 00:04:55,937 and there 145 00:04:55,947 --> 00:04:58,217 is now while it's waiting it could be 146 00:04:58,217 --> 00:05:00,817 discovering other things, figure out what's important, what's 147 00:05:00,827 --> 00:05:02,117 not but while 148 00:05:02,117 --> 00:05:03,357 it's waiting there's always a 149 00:05:03,357 --> 00:05:04,137 risk that 150 00:05:04,187 --> 00:05:09,337 it could be detected, it could be destroyed, and so previously, like you 151 00:05:09,337 --> 00:05:10,857 were saying, four and a half five and a 152 00:05:10,867 --> 00:05:13,497 half days for the dwell time, that it would just sit in 153 00:05:13,497 --> 00:05:15,217 your environment, not doing 154 00:05:15,217 --> 00:05:15,327 anything. 155 00:05:16,387 --> 00:05:16,797 Yeah, 156 00:05:16,807 --> 00:05:18,197 I remember 157 00:05:18,197 --> 00:05:20,607 when, I don't know if there's a difference. 158 00:05:20,607 --> 00:05:23,727 Well, there's definitely a difference between the average and the mean, but 159 00:05:23,727 --> 00:05:28,237 I remember when the mean dwell time was measured in many days, right? 160 00:05:28,237 --> 00:05:29,097 Like, like 161 00:05:29,297 --> 00:05:30,467 it was like as high as 162 00:05:30,477 --> 00:05:31,397 45. 163 00:05:31,997 --> 00:05:32,487 Right. 164 00:05:32,742 --> 00:05:35,602 And now they're saying that the, uh, 165 00:05:35,642 --> 00:05:36,492 um, you 166 00:05:36,492 --> 00:05:36,682 know 167 00:05:36,712 --> 00:05:39,232 this time, and I don't know if they're using mean or average, 168 00:05:39,642 --> 00:05:41,972 but, uh, it says it's down to 169 00:05:42,042 --> 00:05:43,702 24 hours. 170 00:05:44,182 --> 00:05:47,042 And they're saying, and in more than 10 percent of the 171 00:05:47,042 --> 00:05:47,582 incidents, 172 00:05:47,932 --> 00:05:50,672 It was deployed within five hours The ransomware 173 00:05:50,712 --> 00:05:53,492 was, you know, the actual ransomware part was done within 174 00:05:53,492 --> 00:05:54,982 five hours of the initial attack, 175 00:05:56,037 --> 00:05:56,297 which 176 00:05:56,307 --> 00:05:57,937 is good, right? 177 00:05:57,937 --> 00:06:03,327 Because, like the title said, that means people are detecting it faster, right? 178 00:06:03,367 --> 00:06:06,417 And ransomware crews and ransomware as a 179 00:06:06,417 --> 00:06:07,807 service affiliates, right? 180 00:06:07,807 --> 00:06:10,367 They realize, yeah, we can't just let it sit there. 181 00:06:10,367 --> 00:06:10,687 We have 182 00:06:10,687 --> 00:06:12,067 to Be in and out 183 00:06:12,107 --> 00:06:13,387 as quickly as possible. 184 00:06:14,292 --> 00:06:16,982 Right, yeah, and and that's, that's why the headline, 185 00:06:16,982 --> 00:06:18,292 they're saying, well, because 186 00:06:18,732 --> 00:06:19,282 we've gotten 187 00:06:19,292 --> 00:06:20,122 better at 188 00:06:20,972 --> 00:06:22,002 detecting it, 189 00:06:22,142 --> 00:06:23,882 they've, they've basically 190 00:06:24,172 --> 00:06:25,742 had to realize they've had to, 191 00:06:26,832 --> 00:06:27,102 You know, 192 00:06:27,152 --> 00:06:27,992 once they're in and 193 00:06:27,992 --> 00:06:28,282 they got to 194 00:06:28,282 --> 00:06:29,342 do bad stuff right away. 195 00:06:29,342 --> 00:06:31,302 Otherwise, they're going to, they're going to get detected. 196 00:06:31,912 --> 00:06:33,652 Um, go ahead. 197 00:06:34,402 --> 00:06:35,602 Another interesting fact I 198 00:06:35,612 --> 00:06:37,702 saw in the article was, I know we always 199 00:06:37,702 --> 00:06:39,582 talk about like double extortion, 200 00:06:40,272 --> 00:06:40,642 right? 201 00:06:40,672 --> 00:06:41,082 Where 202 00:06:41,082 --> 00:06:43,362 someone comes in, they encrypt your. 203 00:06:43,802 --> 00:06:47,422 Environment, but they also exfiltrate data, right? 204 00:06:47,422 --> 00:06:48,062 So now 205 00:06:48,062 --> 00:06:49,232 you have to pay, 206 00:06:49,232 --> 00:06:49,502 right? 207 00:06:49,502 --> 00:06:50,122 Because otherwise, 208 00:06:50,122 --> 00:06:51,732 who wants to have their data released? 209 00:06:52,102 --> 00:06:53,522 I think, actually, as we're 210 00:06:53,522 --> 00:06:54,772 recording this, 211 00:06:55,662 --> 00:06:58,522 there is a company that 212 00:06:58,552 --> 00:07:00,772 is potentially going to have 213 00:07:00,772 --> 00:07:02,202 their data exposed 214 00:07:02,782 --> 00:07:03,582 because they 215 00:07:03,592 --> 00:07:05,662 decided not to pay the ransomware operators, 216 00:07:06,412 --> 00:07:06,952 right. 217 00:07:07,872 --> 00:07:08,152 right? 218 00:07:08,192 --> 00:07:08,732 And that's the 219 00:07:08,732 --> 00:07:09,382 double extortion. 220 00:07:09,382 --> 00:07:13,122 Now, in the article though, they said that the times, number of times that 221 00:07:13,582 --> 00:07:14,242 they're seeing double 222 00:07:14,242 --> 00:07:16,372 extortion from the people they've surveyed 223 00:07:16,742 --> 00:07:18,192 is only 13 percent of the 224 00:07:18,202 --> 00:07:18,572 time. 225 00:07:18,992 --> 00:07:20,232 That seems really 226 00:07:20,232 --> 00:07:20,872 low, 227 00:07:22,217 --> 00:07:22,757 but given 228 00:07:22,757 --> 00:07:25,087 you only have 24 hours, maybe 229 00:07:25,467 --> 00:07:26,037 it makes sense. 230 00:07:26,067 --> 00:07:26,457 They don't have 231 00:07:26,457 --> 00:07:27,757 enough time to do more damage. 232 00:07:28,522 --> 00:07:29,072 Right. 233 00:07:29,252 --> 00:07:30,112 Yeah, that, that, 234 00:07:30,962 --> 00:07:31,752 and I see that 235 00:07:31,752 --> 00:07:33,062 as good news, as I'm 236 00:07:33,062 --> 00:07:33,322 sure 237 00:07:33,322 --> 00:07:36,932 you understand, because the, the actual, 238 00:07:37,932 --> 00:07:38,872 the thing I'm worried 239 00:07:38,872 --> 00:07:43,152 most about is the Um, exfiltration because 240 00:07:43,942 --> 00:07:45,582 backup just can't help in 241 00:07:45,582 --> 00:07:46,342 that, right? 242 00:07:46,752 --> 00:07:48,782 Uh, once the data has been 243 00:07:48,782 --> 00:07:49,922 exfiltrated, 244 00:07:50,392 --> 00:07:51,302 all all bets are 245 00:07:51,302 --> 00:07:51,582 off. 246 00:07:51,882 --> 00:07:52,522 So I saw 247 00:07:52,522 --> 00:07:53,042 that as good. 248 00:07:53,042 --> 00:07:57,742 And that part came from the annual threat intelligence report from Microsoft. 249 00:07:58,502 --> 00:07:58,982 So 250 00:07:59,682 --> 00:07:59,992 that 251 00:08:00,002 --> 00:08:04,112 that is really interesting though, is that, um, uh, 252 00:08:05,832 --> 00:08:08,162 the other reason why I think this is a good thing 253 00:08:08,917 --> 00:08:12,217 is that the shorter the dwell 254 00:08:12,217 --> 00:08:12,957 time, 255 00:08:13,637 --> 00:08:14,937 the easier the 256 00:08:14,937 --> 00:08:15,747 recovery. 257 00:08:16,397 --> 00:08:18,497 So when you have a dwell 258 00:08:18,497 --> 00:08:18,837 time 259 00:08:18,837 --> 00:08:20,307 measured in days or 260 00:08:20,307 --> 00:08:21,007 weeks, 261 00:08:22,207 --> 00:08:25,757 And you're doing something along the way, 262 00:08:26,707 --> 00:08:28,787 especially if you're encrypting data 263 00:08:28,817 --> 00:08:29,757 along the way, 264 00:08:30,847 --> 00:08:32,347 how do you recover from 265 00:08:32,357 --> 00:08:32,587 that? 266 00:08:33,177 --> 00:08:33,437 Right? 267 00:08:33,437 --> 00:08:33,897 There's no, 268 00:08:34,247 --> 00:08:35,987 the, the the good point 269 00:08:35,987 --> 00:08:36,587 in time 270 00:08:36,597 --> 00:08:37,937 is three weeks 271 00:08:37,937 --> 00:08:38,327 ago, 272 00:08:38,917 --> 00:08:39,377 right? 273 00:08:39,677 --> 00:08:41,417 Do you do you really want to recover? 274 00:08:41,827 --> 00:08:43,027 your primary file 275 00:08:43,037 --> 00:08:44,017 server, for example? 276 00:08:44,017 --> 00:08:45,127 This was the one I was always 277 00:08:45,127 --> 00:08:45,517 worried about. 278 00:08:45,727 --> 00:08:46,327 If you 279 00:08:46,327 --> 00:08:49,417 encrypt VMs, if you encrypt databases, 280 00:08:49,427 --> 00:08:50,617 it's easy to notice 281 00:08:50,647 --> 00:08:53,017 the moment you encrypt anything, everything stops working 282 00:08:53,017 --> 00:08:54,107 and you know when the 283 00:08:54,107 --> 00:08:54,707 point in time 284 00:08:54,707 --> 00:08:54,987 is. 285 00:08:55,297 --> 00:08:56,727 But if you talk about a file 286 00:08:56,737 --> 00:08:57,337 server 287 00:08:57,697 --> 00:09:00,677 or someone's workstation that has a lot of files 288 00:09:00,677 --> 00:09:01,587 on it, if you're able 289 00:09:01,587 --> 00:09:01,897 to encrypt... 290 00:09:02,342 --> 00:09:03,172 data over 291 00:09:03,172 --> 00:09:03,822 time 292 00:09:04,182 --> 00:09:07,492 and not be noticed, Restoring that is 293 00:09:07,502 --> 00:09:08,872 significantly more 294 00:09:08,872 --> 00:09:10,812 complicated than restoring 295 00:09:11,632 --> 00:09:14,112 an encryption attack that takes place over hours. 296 00:09:14,122 --> 00:09:15,112 So I think this 297 00:09:15,122 --> 00:09:18,732 is a much better uh, scenario. 298 00:09:19,922 --> 00:09:21,612 It does mean we have to continue to 299 00:09:21,612 --> 00:09:22,142 stay vigilant 300 00:09:23,667 --> 00:09:26,497 and to make sure that we're continuing to detect 301 00:09:26,497 --> 00:09:28,687 so that they continue to have dwell times this small. 302 00:09:30,057 --> 00:09:30,527 And this 303 00:09:30,527 --> 00:09:33,037 also goes to the importance of backups, 304 00:09:33,037 --> 00:09:33,347 right? 305 00:09:33,687 --> 00:09:36,077 Cause if it does hit, like you were saying, you want to 306 00:09:36,077 --> 00:09:37,247 be able to restore. 307 00:09:37,597 --> 00:09:39,027 And so if you don't have a 308 00:09:39,027 --> 00:09:41,037 backup that you can restore from. 309 00:09:41,497 --> 00:09:42,607 Then you're going to lose data. 310 00:09:43,667 --> 00:09:44,167 Right. 311 00:09:44,537 --> 00:09:47,177 There, There was another thing here that they were 312 00:09:47,177 --> 00:09:51,447 saying that, you know, because of ransomware as a service uh, businesses, 313 00:09:52,057 --> 00:09:52,627 that they 314 00:09:52,627 --> 00:09:53,077 actually, 315 00:09:53,207 --> 00:09:54,417 it says in June, 316 00:09:54,417 --> 00:09:55,377 they broke the single 317 00:09:55,387 --> 00:09:57,617 month record for ransomware attacks. 318 00:09:58,047 --> 00:10:01,377 Thanks to a single exploit, uh, the MoveIt 319 00:10:01,377 --> 00:10:04,967 MFT exploit, which I actually don't know much about, but that 320 00:10:04,967 --> 00:10:07,867 single exploit allowed them to uh, 321 00:10:07,867 --> 00:10:10,207 break the record of the number of attacks in a month. 322 00:10:11,427 --> 00:10:11,817 That 323 00:10:11,817 --> 00:10:12,717 doesn't sound good. 324 00:10:13,167 --> 00:10:15,027 None of this sounds good, I guess. 325 00:10:15,267 --> 00:10:15,897 It's just, 326 00:10:16,377 --> 00:10:17,157 I do like 327 00:10:17,157 --> 00:10:18,897 a quicker attack because 328 00:10:19,577 --> 00:10:21,377 a quicker attack is, I think. 329 00:10:22,207 --> 00:10:22,877 Easier 330 00:10:22,937 --> 00:10:23,627 to 331 00:10:24,637 --> 00:10:25,737 defend against, 332 00:10:25,737 --> 00:10:26,947 or let me rephrase 333 00:10:26,947 --> 00:10:27,157 that, 334 00:10:27,697 --> 00:10:29,367 a quicker attack is easier 335 00:10:29,367 --> 00:10:30,357 to recover from. 336 00:10:32,477 --> 00:10:32,727 Yeah. 337 00:10:32,727 --> 00:10:33,857 And also a 338 00:10:36,077 --> 00:10:36,647 hundred percent 339 00:10:36,647 --> 00:10:37,517 agree with you, Curtis. 340 00:10:37,567 --> 00:10:37,977 So, 341 00:10:39,097 --> 00:10:39,477 what, 342 00:10:39,597 --> 00:10:43,177 so do you still want to claim credit because of our podcast that 343 00:10:43,907 --> 00:10:44,707 we're helping 344 00:10:44,717 --> 00:10:45,347 improve? 345 00:10:45,357 --> 00:10:46,017 to how 346 00:10:46,027 --> 00:10:46,227 much 347 00:10:46,937 --> 00:10:49,397 we have gotten the word out there that long 348 00:10:49,397 --> 00:10:50,927 dwell times are bad, 349 00:10:51,457 --> 00:10:52,737 that the attackers have 350 00:10:52,997 --> 00:10:54,787 made short dwell times. 351 00:10:55,697 --> 00:10:55,817 You 352 00:10:55,877 --> 00:10:57,237 So any attackers 353 00:10:57,667 --> 00:10:58,907 so any attackers 354 00:10:58,907 --> 00:11:02,127 out there, if you would like to come on the podcast and talk about this, 355 00:11:02,727 --> 00:11:03,877 please reach out and 356 00:11:03,877 --> 00:11:04,407 let us know. 357 00:11:05,042 --> 00:11:05,302 Can you 358 00:11:05,302 --> 00:11:05,802 imagine? 359 00:11:06,172 --> 00:11:07,002 Can you imagine that? 360 00:11:08,332 --> 00:11:10,492 Um, once 361 00:11:10,492 --> 00:11:11,582 again, another thing 362 00:11:11,592 --> 00:11:12,682 from here, once 363 00:11:12,682 --> 00:11:13,272 again, 364 00:11:13,602 --> 00:11:14,472 the two highest 365 00:11:14,482 --> 00:11:17,252 profile attacks of 2023 were the result 366 00:11:17,252 --> 00:11:20,512 of unpatched infrastructure, right? 367 00:11:21,112 --> 00:11:21,932 Um, 368 00:11:22,037 --> 00:11:23,797 we like to talk about on the podcast, right? 369 00:11:23,852 --> 00:11:24,842 yeah, yeah, 370 00:11:25,887 --> 00:11:28,147 MFA, patcher systems, 371 00:11:28,787 --> 00:11:28,967 do 372 00:11:28,967 --> 00:11:29,497 backups. 373 00:11:30,582 --> 00:11:31,282 Exactly. 374 00:11:31,372 --> 00:11:34,052 That would stop the vast majority of 375 00:11:34,052 --> 00:11:35,402 ransomware attacks that we see. 376 00:11:37,142 --> 00:11:39,882 Well, with that, that is the news of the day. 377 00:11:43,133 --> 00:11:48,603 This week's episode is a continuation of our Backup to Basics series, 378 00:11:48,673 --> 00:11:53,798 and this week, we're going to be talking about a Product category 379 00:11:53,798 --> 00:11:56,678 that at one point was red hot. 380 00:11:58,168 --> 00:11:58,878 Was it not? 381 00:11:59,058 --> 00:12:02,218 Do you remember when this product category was red hot? 382 00:12:02,228 --> 00:12:06,518 Like everybody had to have a CDP product. 383 00:12:06,528 --> 00:12:06,748 Do you 384 00:12:06,808 --> 00:12:09,238 I want to say it was like 2002, 2003. 385 00:12:10,258 --> 00:12:10,828 Yeah. 386 00:12:10,948 --> 00:12:14,568 What I remember was being at Storage Networking World and half of the 387 00:12:14,568 --> 00:12:18,168 booths were CDP products, remember 388 00:12:18,348 --> 00:12:19,968 is CDP Curtis or our 389 00:12:20,328 --> 00:12:22,578 yeah, we're, we're gonna, we're gonna talk about that in just 390 00:12:22,578 --> 00:12:23,868 a second, but just the, the. 391 00:12:25,778 --> 00:12:30,108 The sheer number, I remember thinking all of these can't succeed and little 392 00:12:30,108 --> 00:12:35,188 did I know that pretty much almost none of them, uh, would succeed. 393 00:12:35,268 --> 00:12:35,478 Uh, 394 00:12:35,668 --> 00:12:37,558 I want to say there's like four left. 395 00:12:37,918 --> 00:12:38,558 In the world. 396 00:12:38,858 --> 00:12:43,328 Yeah, there's, well, and, and most of them got acquired and 397 00:12:43,348 --> 00:12:49,178 are, are simply a checkbox on, on another product's portfolio. 398 00:12:50,258 --> 00:12:51,928 So what is CDP? 399 00:12:51,928 --> 00:12:54,378 It stands for continuous data. 400 00:12:54,718 --> 00:12:55,758 Protection. 401 00:12:56,458 --> 00:13:03,338 And this was a, you may recall in a previous episode, we talked about 402 00:13:03,338 --> 00:13:10,858 replication and what, as far as I'm concerned, what is the primary problem 403 00:13:10,858 --> 00:13:14,583 with date with replication as a community. 404 00:13:14,773 --> 00:13:19,093 Data protection or a basically a replacement for backup. 405 00:13:19,093 --> 00:13:20,793 What's the primary problem with it? 406 00:13:20,968 --> 00:13:22,808 Whatever you do here happens here. 407 00:13:23,843 --> 00:13:24,473 Exactly. 408 00:13:25,123 --> 00:13:28,653 It is very efficient in replicating stupidity, right? 409 00:13:29,033 --> 00:13:35,103 Uh, or, or, or ransomware attacks or anything in any sort of cyber attack. 410 00:13:35,473 --> 00:13:46,768 So replication is great at giving you a, An RPO of zero, right? 411 00:13:46,798 --> 00:13:53,238 A recovery point objective of zero, but it's also going to replicate 412 00:13:53,708 --> 00:13:56,858 things that happen on a logical level. 413 00:13:58,188 --> 00:14:09,148 Um, and so CDP was born and I describe CDP as replication with a back button. 414 00:14:10,188 --> 00:14:11,438 What do you think of that? 415 00:14:12,143 --> 00:14:13,163 That definition. 416 00:14:15,408 --> 00:14:19,618 I like it, but I used to think I used to, you know what I used to call CDP? 417 00:14:20,213 --> 00:14:20,663 What? 418 00:14:20,938 --> 00:14:22,758 I was like, it's TiVo for your data. 419 00:14:25,073 --> 00:14:25,403 Yeah. 420 00:14:25,403 --> 00:14:29,693 That, but that was, uh, I remember, I remember vendors describing it like that. 421 00:14:29,813 --> 00:14:32,573 Uh, the problem is now nobody knows what TiVo is. 422 00:14:34,298 --> 00:14:38,088 I know that's why I said for the five listeners who may know what TiVo is. 423 00:14:38,088 --> 00:14:41,548 And for the two of us, since we both had TiVos, right. 424 00:14:41,768 --> 00:14:43,918 We understand that name. 425 00:14:43,918 --> 00:14:46,798 And also if you do watch Psych, there is references. 426 00:14:50,213 --> 00:14:51,953 Are there TiVo references in psych? 427 00:14:51,998 --> 00:14:52,548 Oh, yeah, 428 00:14:54,233 --> 00:14:54,533 All right. 429 00:14:54,533 --> 00:14:58,823 Well, you would know better 'cause you've been, you've been binging psych lately, so 430 00:15:00,038 --> 00:15:02,258 but, but, but yes, I agree with your point. 431 00:15:02,258 --> 00:15:04,788 It is a back button for replication. 432 00:15:05,098 --> 00:15:08,668 And specifically what you mean is replication. 433 00:15:08,678 --> 00:15:11,598 Do you have that one copy with CDP? 434 00:15:11,608 --> 00:15:14,168 You can go backwards from that one copy. 435 00:15:14,583 --> 00:15:15,553 To other points in 436 00:15:16,653 --> 00:15:16,923 yeah. 437 00:15:16,923 --> 00:15:20,493 The, the reason why I call it replication with a back button is that, is that 438 00:15:21,323 --> 00:15:25,063 the process of getting the data. 439 00:15:26,653 --> 00:15:27,673 We've discussed that. 440 00:15:27,673 --> 00:15:29,583 I see all of these things as backup. 441 00:15:31,013 --> 00:15:38,873 A lot of people see backup as, well, putting something on tape or a backup 442 00:15:38,923 --> 00:15:41,113 that changes its format, right? 443 00:15:41,763 --> 00:15:49,213 A lot of people try to define sort of old school backup as something that requires 444 00:15:49,213 --> 00:15:54,123 a restore, you know, different ways to try to define what old school backup is. 445 00:15:54,883 --> 00:15:55,563 And... 446 00:15:55,908 --> 00:15:59,258 I just see that as a, that is the old way we did backup. 447 00:15:59,258 --> 00:16:00,958 This is now a new way that 448 00:16:00,958 --> 00:16:01,698 we do backup. 449 00:16:02,138 --> 00:16:06,438 Backup is just a method of putting the data in a different place 450 00:16:06,448 --> 00:16:11,078 so that we can restore it in, in time of something bad happening. 451 00:16:11,488 --> 00:16:14,018 And this is one of the newer ways. 452 00:16:14,018 --> 00:16:21,488 And the, the thing is, unlike traditional backup, CDP is not a batch process. 453 00:16:21,928 --> 00:16:24,048 Traditionally backup ran once a night. 454 00:16:24,518 --> 00:16:27,378 Sometimes you might run it multiple times a day. 455 00:16:28,168 --> 00:16:29,488 You could run it once an hour. 456 00:16:29,488 --> 00:16:31,248 You could run it every five minutes. 457 00:16:31,838 --> 00:16:34,998 Traditionally backup is a batch process. 458 00:16:35,388 --> 00:16:42,928 CDP by definition, that C is that it is happening continuously. 459 00:16:43,723 --> 00:16:46,313 All the time, just like replication. 460 00:16:46,323 --> 00:16:51,213 Although we had some, there were some finer points there where we, 461 00:16:51,213 --> 00:16:57,633 where you and I were trying to argue about on what continuous means, and 462 00:16:58,343 --> 00:17:03,223 the idea is that it is happening truly continuously every time. 463 00:17:03,678 --> 00:17:07,788 A block of data that is changed on the primary system. 464 00:17:08,128 --> 00:17:15,118 It gets replicated to the target system Now, immediately, you know, 465 00:17:15,963 --> 00:17:17,003 Yeah, we can debate that. 466 00:17:17,013 --> 00:17:17,213 That's 467 00:17:17,258 --> 00:17:21,458 this happens, but, but basically this is, it's not a batch process. 468 00:17:21,468 --> 00:17:24,408 It's happening continuously throughout the day. 469 00:17:25,068 --> 00:17:28,208 And then we can talk about how that is stored on the other end. 470 00:17:28,248 --> 00:17:31,208 Uh, how are you okay with that part of the definition? 471 00:17:33,193 --> 00:17:33,843 I'm good with that. 472 00:17:33,863 --> 00:17:37,323 And I think the one other thing we should touch on is. 473 00:17:38,548 --> 00:17:43,448 As technologies have evolved, so has CDP in the sense of we could 474 00:17:43,448 --> 00:17:49,588 talk about where in the stack you're actually triggering or forwarding I. 475 00:17:49,588 --> 00:17:49,798 O. 476 00:17:49,798 --> 00:17:50,958 and the data from. 477 00:17:51,628 --> 00:17:56,028 Typically, right, and way back in the day, right, all these CDP vendors when 478 00:17:56,028 --> 00:18:01,558 you were probably at the SNIA, right, it was all, okay, here's an appliance. 479 00:18:02,453 --> 00:18:07,893 that you put in, right, the writes might come into it, get split off, go to two 480 00:18:07,893 --> 00:18:12,883 different places, right, that's one method that some people would do to make sure you 481 00:18:12,883 --> 00:18:15,353 have two copies, continuously replicating. 482 00:18:15,823 --> 00:18:20,763 Another method that some vendors have used is you sort of write to your 483 00:18:20,773 --> 00:18:25,673 primary, the primary forwards it off to an appliance or to something else which 484 00:18:25,673 --> 00:18:27,463 then writes it on the target system. 485 00:18:28,003 --> 00:18:28,233 Right. 486 00:18:28,233 --> 00:18:29,663 That's another mechanism people did. 487 00:18:29,673 --> 00:18:32,313 All of that is sort of infrastructure level down at the 488 00:18:32,313 --> 00:18:34,043 storage array or networking level. 489 00:18:34,363 --> 00:18:37,383 Actually, some people even did it at like the storage area network level, right. 490 00:18:37,383 --> 00:18:40,833 Where they would have that appliance in the middle, right. 491 00:18:40,833 --> 00:18:43,393 And basically that's that first use case where you would write 492 00:18:43,393 --> 00:18:44,523 to two different storage arrays. 493 00:18:45,763 --> 00:18:49,453 The other thing moving up the stack, right, is with virtualization, people 494 00:18:49,453 --> 00:18:54,173 were like, hey, the same challenges you had with sort of storage level, CDP, 495 00:18:54,183 --> 00:18:55,953 let's do that at the VM level as well. 496 00:18:55,953 --> 00:19:00,903 And so you had technologies that would allow you to split right at a VM level. 497 00:19:00,963 --> 00:19:04,863 You could forward it off to another ESXI cluster in a different location and have a 498 00:19:04,863 --> 00:19:07,403 continuously replicated VM somewhere else. 499 00:19:09,253 --> 00:19:15,693 Right, basically they all, the concept was the same. 500 00:19:15,763 --> 00:19:19,173 The question is, at what point are we going to split the right? 501 00:19:19,933 --> 00:19:25,053 And then take one copy and send it where we would always send it to the 502 00:19:25,053 --> 00:19:30,143 primary storage and the other copy of that right gets sent to some magic 503 00:19:30,853 --> 00:19:38,943 process or box or whatever that will then store it for CDP purposes and. 504 00:19:39,553 --> 00:19:41,703 Sometimes it can happen in the storage array. 505 00:19:41,713 --> 00:19:45,693 There, there have been boxes that you can buy that go between your 506 00:19:45,693 --> 00:19:47,203 storage array and your server. 507 00:19:47,723 --> 00:19:50,893 Sometimes it might be an independent, you know, that box might be 508 00:19:50,893 --> 00:19:53,743 an actual appliance, it might be a piece of software, right? 509 00:19:53,743 --> 00:19:55,063 We had Datacore on here. 510 00:19:55,073 --> 00:19:59,263 Datacore was one of those vendors that you can put the box in, you know, their 511 00:19:59,263 --> 00:20:00,943 software on a box in between your. 512 00:20:01,968 --> 00:20:06,798 Uh, storage array on your server, and it might be in, like you said, it might 513 00:20:06,798 --> 00:20:10,408 be in the hypervisor, it might even be in the cloud, it might be something 514 00:20:10,408 --> 00:20:11,998 that's being done in the cloud. 515 00:20:12,438 --> 00:20:16,798 But the idea is that basically as the, literally as the data is being written, 516 00:20:17,118 --> 00:20:22,398 it gets piped off into two places, and then the second of which is the CDP copy. 517 00:20:22,953 --> 00:20:28,283 Do you consider, since we're talking about CDP, do you consider database 518 00:20:28,283 --> 00:20:36,043 level things like Oracle's Data Guard as CDP or Exchange used to have 519 00:20:36,053 --> 00:20:38,153 something like, what was it called? 520 00:20:38,163 --> 00:20:42,903 CRR and all the rest where a write comes in and they forward over the 521 00:20:42,903 --> 00:20:45,553 log, because that technically is CDP, 522 00:20:45,663 --> 00:20:49,743 That is, that is application level replication. 523 00:20:50,673 --> 00:20:57,153 It is not application level CDP because I don't think that with an active database 524 00:20:58,033 --> 00:21:00,833 that you can just go backwards in time. 525 00:21:01,343 --> 00:21:05,793 I know that if it crashes you can do, you can do media recovery against it. 526 00:21:07,553 --> 00:21:09,283 But I don't think it's built. 527 00:21:09,693 --> 00:21:12,593 So I'll just say if that's built into it, then sure. 528 00:21:12,693 --> 00:21:13,103 Right. 529 00:21:13,113 --> 00:21:17,643 But if it's just replicating the changes and doesn't have the ability 530 00:21:17,643 --> 00:21:21,253 to go back in time, then no, right. 531 00:21:21,263 --> 00:21:22,383 It's not CDP. 532 00:21:22,718 --> 00:21:25,478 That is a very crucial aspect of CDP, 533 00:21:25,848 --> 00:21:28,978 yeah, and one way to think about this is, I know with databases, we 534 00:21:28,998 --> 00:21:31,928 think about redo logs, right, which allow you to go forward in time. 535 00:21:32,198 --> 00:21:35,048 With CDP, you actually want undo logs, right? 536 00:21:35,058 --> 00:21:40,288 How do I go backwards in time from the most recent version on the target system? 537 00:21:41,578 --> 00:21:42,778 That's a really good point. 538 00:21:42,828 --> 00:21:45,128 And I don't think anybody calls them undo logs. 539 00:21:45,138 --> 00:21:48,698 So everybody calls them either redo logs or transaction logs. 540 00:21:48,738 --> 00:21:49,768 No, I mean, in, in the 541 00:21:50,158 --> 00:21:50,598 Oh, database. 542 00:21:51,258 --> 00:21:54,918 um, they call them redo logs or they call them transaction logs, because 543 00:21:54,918 --> 00:21:57,678 the idea is that you, you have a. 544 00:21:58,083 --> 00:22:02,533 It allows you to have a backup, a traditional backup from this 545 00:22:02,533 --> 00:22:06,613 point in time and then use those logs to redo the transactions 546 00:22:06,613 --> 00:22:10,263 that happened during that point in time and since that point in time. 547 00:22:10,703 --> 00:22:16,963 But with CDP, you are correct, the most important thing is to be able to go 548 00:22:17,043 --> 00:22:23,033 back in time, which is not something that a typical database replication 549 00:22:23,043 --> 00:22:24,813 scenario is going to be able to do. 550 00:22:26,098 --> 00:22:29,668 You mentioned the ability to go back in time. 551 00:22:31,228 --> 00:22:38,498 How far back in time should we be able to go with ACDP system? 552 00:22:39,683 --> 00:22:42,303 Depends on what your requirements are, right? 553 00:22:42,673 --> 00:22:48,993 I would say with the CDP system, it depends on what other environments 554 00:22:48,993 --> 00:22:50,053 or infrastructure you have. 555 00:22:50,053 --> 00:22:53,113 For instance, if you have backups, right? 556 00:22:53,798 --> 00:22:58,048 That you're taking periodically, separately, outside of the CDP system. 557 00:22:58,058 --> 00:23:01,668 Your CDP system may only need 7 days worth of data, so you can recover 558 00:23:01,668 --> 00:23:04,768 within those 7 days at, sort of, uh, I. 559 00:23:04,768 --> 00:23:04,938 O. 560 00:23:04,958 --> 00:23:05,798 granularity. 561 00:23:06,148 --> 00:23:06,418 Right. 562 00:23:06,418 --> 00:23:08,578 Or a record granular or whatever we want to call it. 563 00:23:08,748 --> 00:23:09,118 Right. 564 00:23:09,238 --> 00:23:12,918 Uh, but as long as you have that backup system, that's fine. 565 00:23:12,938 --> 00:23:17,218 Going back, say 30, 90, or trying to replace your backup system with the 566 00:23:17,228 --> 00:23:22,428 CDP system is a little crazy because I think we need to talk about what's 567 00:23:22,428 --> 00:23:27,458 required on the target system or on the target side in order to handle CDP. 568 00:23:29,138 --> 00:23:35,268 Right, because in order to be able to go back in time, I need much more 569 00:23:35,268 --> 00:23:40,348 storage at the target side than I need at the primary side, because 570 00:23:40,398 --> 00:23:44,323 if I'm doing a hundred terabytes of storage, And I'm, and I'm 571 00:23:44,333 --> 00:23:45,973 going to do CDP for that. 572 00:23:46,683 --> 00:23:52,373 How much do you, because realize at that target side, I need to store the 573 00:23:52,373 --> 00:23:55,993 hundred terabytes and every block. 574 00:23:56,398 --> 00:23:59,898 That changes in that 100 terabytes during that 575 00:24:00,498 --> 00:24:00,838 Date. 576 00:24:00,938 --> 00:24:03,538 continuum that you've set, 577 00:24:03,848 --> 00:24:04,088 Yeah. 578 00:24:04,138 --> 00:24:08,628 And so that's why you would see sort of, and I think on the target side, 579 00:24:08,628 --> 00:24:11,308 we should probably differentiate depending on what technology, right? 580 00:24:11,308 --> 00:24:15,978 Your target system itself may not need all the extra space, but maybe that 581 00:24:15,978 --> 00:24:19,948 target appliance, which is dealing with these transactions coming in or 582 00:24:19,948 --> 00:24:23,908 these change blocks coming in, that might need to hold the space, right? 583 00:24:24,338 --> 00:24:25,818 Uh, sort of as a log. 584 00:24:26,828 --> 00:24:34,958 And this is really, this problem right here is why CDP, I think, 585 00:24:34,968 --> 00:24:37,598 failed in terms of the dream of CDP. 586 00:24:37,628 --> 00:24:41,518 The dream of CDP, because I remember meeting with CDP. 587 00:24:41,843 --> 00:24:47,653 CEOs, and they were like, this solves everything, We can 588 00:24:47,653 --> 00:24:49,933 recover to any point in time. 589 00:24:50,033 --> 00:24:52,353 Why would you do it any other way? 590 00:24:52,393 --> 00:24:54,773 And the answer is cost. 591 00:24:56,138 --> 00:25:00,738 It's the cost because, because the thing you have to think about is 592 00:25:00,738 --> 00:25:03,928 you have to store the data, right? 593 00:25:03,938 --> 00:25:08,728 Not only with the metadata about what came in, the data that's there, 594 00:25:08,978 --> 00:25:12,308 but if these are undue, right, you also need to store what the previous 595 00:25:12,308 --> 00:25:15,338 data was as well, because you have to be able to go backwards in time. 596 00:25:15,758 --> 00:25:19,808 And so you have to store all of this information in that appliance and. 597 00:25:20,248 --> 00:25:23,998 Some people say that you might have like a 2 percent change rate per day. 598 00:25:24,378 --> 00:25:29,798 That doesn't mean that that's 2 percent that's 2 percent over the entire day. 599 00:25:30,028 --> 00:25:33,128 But if you're adding up every single transaction, right, that might turn 600 00:25:33,128 --> 00:25:36,428 out to be like 5 percent actual change. 601 00:25:36,788 --> 00:25:37,078 Right. 602 00:25:37,108 --> 00:25:38,358 Or 10%, right? 603 00:25:39,243 --> 00:25:42,573 if you have anything, if you have a block updated, if we're talking about 604 00:25:42,573 --> 00:25:48,243 block level CDP here, which is generally what we're talking about, if a block 605 00:25:48,273 --> 00:25:52,453 changes multiple times during the day, you have to store every version 606 00:25:52,453 --> 00:25:54,183 of that block throughout the day. 607 00:25:54,668 --> 00:25:56,358 And, uh, you're right. 608 00:25:56,388 --> 00:25:57,948 It could be a significant percent. 609 00:25:57,978 --> 00:26:02,688 And by the way, you have no idea what that number is until you deploy CDP. 610 00:26:03,448 --> 00:26:03,838 Right. 611 00:26:04,678 --> 00:26:05,218 The other, 612 00:26:05,318 --> 00:26:05,548 and 613 00:26:07,918 --> 00:26:11,298 the other thing I know you were just mentioning about sort of, you don't know 614 00:26:11,298 --> 00:26:12,958 what you'll need until you deploy it. 615 00:26:13,438 --> 00:26:17,728 You also have to deploy it on pretty fast and expensive hardware, because if 616 00:26:17,728 --> 00:26:22,818 you think about it, you're getting this constant stream of rights that you have 617 00:26:22,818 --> 00:26:26,928 to store and you have to replay it down to your target storage location as well. 618 00:26:27,238 --> 00:26:31,608 And so your destination system might need to be beefier or the infrastructure 619 00:26:31,608 --> 00:26:34,988 required might need to be beefier than what you even have on your production. 620 00:26:35,543 --> 00:26:35,813 Right? 621 00:26:35,813 --> 00:26:39,803 So going back to that cost aspect, that starts to add up pretty fast. 622 00:26:41,478 --> 00:26:46,738 yeah, this, we can go back to the episode on replication. 623 00:26:47,108 --> 00:26:50,908 The synchronous and asynchronous aspect is important to understand here. 624 00:26:50,908 --> 00:26:55,188 So generally CDP will be done asynchronously. 625 00:26:55,558 --> 00:26:57,488 Do you remember synchronous CDP? 626 00:26:59,948 --> 00:27:02,438 I think there was one vendor who did it, but yes, 627 00:27:02,788 --> 00:27:03,218 okay. 628 00:27:03,378 --> 00:27:08,348 So you could do, but I do, I think most people do it asynchronously. 629 00:27:09,448 --> 00:27:12,228 And the point is, asynchronously is fine. 630 00:27:12,228 --> 00:27:14,848 Obviously your RPO won't be zero. 631 00:27:14,858 --> 00:27:16,958 It'll be something close to zero. 632 00:27:17,608 --> 00:27:24,943 But the problem with asynchronous is if the target system gets behind 633 00:27:24,943 --> 00:27:28,883 in those rights at some point, you know, the buffer is getting back. 634 00:27:29,393 --> 00:27:29,563 At 635 00:27:29,778 --> 00:27:31,798 your back pressure is going to have to, yeah, 636 00:27:32,103 --> 00:27:32,563 Yeah. 637 00:27:32,733 --> 00:27:33,483 That's a good term. 638 00:27:33,483 --> 00:27:34,203 The back pressure. 639 00:27:34,203 --> 00:27:34,783 I like that. 640 00:27:34,823 --> 00:27:35,153 Right. 641 00:27:35,163 --> 00:27:41,353 You, you will eventually have more rights in the buffer than the size of 642 00:27:41,353 --> 00:27:45,793 the buffer, which would then essentially it then becomes a synchronous or 643 00:27:46,103 --> 00:27:47,773 you have to start dropping rights. 644 00:27:48,023 --> 00:27:48,413 Because 645 00:27:48,418 --> 00:27:49,328 which you don't want to do. 646 00:27:50,253 --> 00:27:52,503 be slowing down the primary system. 647 00:27:53,243 --> 00:27:55,583 So you'd end up having to dump the buffer and you'd end up 648 00:27:55,583 --> 00:27:57,323 losing bits along the way. 649 00:27:57,403 --> 00:28:01,773 And that's just, that's just not something that you would want to do. 650 00:28:02,888 --> 00:28:06,598 Now, one of the benefits I would say, though, with the CDP like 651 00:28:06,598 --> 00:28:14,563 approach is you can do this sort of CDP to Dissimilar systems, right? 652 00:28:14,563 --> 00:28:18,163 So you might be going from like a NetApp to an EMC, or you could be 653 00:28:18,163 --> 00:28:21,573 going from a pure to a Hitachi. 654 00:28:21,873 --> 00:28:26,823 So it gives you flexibility because the CDP applying software package, 655 00:28:26,823 --> 00:28:30,323 whatever else, just needs access to devices on both sides, right? 656 00:28:30,328 --> 00:28:32,513 It's doing all the replication, it's managing everything. 657 00:28:32,513 --> 00:28:38,023 So for cases where you're looking to deal with uh, different costs or 658 00:28:38,023 --> 00:28:40,273 availability of equipment, right? 659 00:28:40,303 --> 00:28:44,043 It is an option rather than sort of being locked into a particular vendor. 660 00:28:45,303 --> 00:28:45,693 Right. 661 00:28:45,973 --> 00:28:48,363 Most of the CDP vendors that I know. 662 00:28:49,173 --> 00:28:51,773 Uh, are, are independent of the storage, right? 663 00:28:51,773 --> 00:28:54,863 So you can use whatever storage you want on, on both sides. 664 00:28:56,073 --> 00:29:00,263 The thing is, I mean, we've, we've been, we've been harping on 665 00:29:00,263 --> 00:29:06,413 it for a little bit, but I mean, the, the idea of CDP is amazing. 666 00:29:06,983 --> 00:29:11,363 The idea that I can just go back to any point in time is amazing. 667 00:29:11,383 --> 00:29:15,093 And I don't have to do anything special on the front end. 668 00:29:15,623 --> 00:29:18,753 Um, but it does come with these downsides. 669 00:29:19,173 --> 00:29:21,883 And so there were some things that happened over. 670 00:29:22,513 --> 00:29:27,653 As CDP was deployed in more and more environments, customers, I 671 00:29:27,653 --> 00:29:29,533 think, demanded certain features. 672 00:29:29,733 --> 00:29:32,743 One of them was this term called right coalescing. 673 00:29:33,053 --> 00:29:34,523 Do you want to talk about that a little bit? 674 00:29:35,103 --> 00:29:35,423 Yeah. 675 00:29:35,533 --> 00:29:39,683 So write coalescing is, I know Curtis, you talked about before where you had 676 00:29:39,693 --> 00:29:42,983 multiple changes to a single block. 677 00:29:43,623 --> 00:29:44,573 That would happen. 678 00:29:45,163 --> 00:29:47,153 Uh, and that's great. 679 00:29:47,183 --> 00:29:49,053 But at the end, would I need to replay something? 680 00:29:49,053 --> 00:29:51,533 I don't need to know all the versions, right? 681 00:29:51,683 --> 00:29:55,353 I could just say, look, just give me this version of the data. 682 00:29:55,373 --> 00:29:56,343 That's all I care about. 683 00:29:56,343 --> 00:29:59,943 And so being able to reduce down some of that data. 684 00:29:59,943 --> 00:30:03,583 So maybe instead of having every transaction for the last 685 00:30:03,583 --> 00:30:05,653 seven days, maybe for the last. 686 00:30:06,428 --> 00:30:09,408 36 hours, I have every transaction, and then after that I'm going 687 00:30:09,408 --> 00:30:10,758 to coalesce writes down. 688 00:30:11,228 --> 00:30:14,718 So I have singular points in time rather than having every single 689 00:30:14,718 --> 00:30:16,038 point in time available to me. 690 00:30:16,778 --> 00:30:21,608 Because honestly, if I go back seven days, do I really care about this I. 691 00:30:21,608 --> 00:30:21,748 O. 692 00:30:21,748 --> 00:30:22,678 versus this I. 693 00:30:22,678 --> 00:30:22,948 O.? 694 00:30:23,708 --> 00:30:24,088 Right? 695 00:30:24,668 --> 00:30:26,848 Like, how do I even find that point in time, you know? 696 00:30:26,898 --> 00:30:28,238 That's the biggest challenge as well. 697 00:30:28,963 --> 00:30:30,623 you'll be happy to have anything. 698 00:30:31,123 --> 00:30:34,083 So you could start with true CDP. 699 00:30:34,093 --> 00:30:36,643 You could, you could always replicate every change. 700 00:30:37,223 --> 00:30:40,723 The system holds on to a certain amount of, you know, all of the 701 00:30:40,723 --> 00:30:42,413 changes for a certain amount of time. 702 00:30:42,943 --> 00:30:44,263 Configurable by the customer. 703 00:30:44,703 --> 00:30:48,743 And then it starts coalescing and saying, okay, we're just going to 704 00:30:48,743 --> 00:30:52,493 make sure we have all the blocks we need to represent this point in time. 705 00:30:52,813 --> 00:30:56,913 And you might go with hourly snapshots after they're not snapshots, but 706 00:30:57,363 --> 00:31:00,783 they're not snapshots in terms of what we traditionally think of as 707 00:31:00,903 --> 00:31:02,193 There are point in times, yeah. 708 00:31:02,393 --> 00:31:03,743 There are points in time. 709 00:31:04,123 --> 00:31:07,113 So you have hourly points in time that you can recover. 710 00:31:07,113 --> 00:31:09,783 And then maybe you go to daily and even weekly. 711 00:31:10,363 --> 00:31:15,013 And that's where some CDP systems, that's what, that's the way some 712 00:31:15,013 --> 00:31:20,673 CDP systems were trying to push out that amount of time that they could. 713 00:31:21,218 --> 00:31:23,378 Essentially replaced the backup system. 714 00:31:23,708 --> 00:31:31,018 But even then it's just not, doesn't really think the way of a regular 715 00:31:31,018 --> 00:31:36,238 backup system would, and so it still ends up storing a lot more data. 716 00:31:37,438 --> 00:31:40,758 And just being more costly in general. 717 00:31:41,818 --> 00:31:46,838 I know, I remember another challenge with CDP systems is with backup. 718 00:31:47,218 --> 00:31:52,178 I know we talk a lot about application consistency, right? 719 00:31:52,188 --> 00:31:56,788 Making sure I have a application consistent point in time that Oracle, 720 00:31:56,798 --> 00:32:00,538 for instance, can quickly recover and I don't need to worry about media 721 00:32:00,538 --> 00:32:01,948 recovery and all the other processes. 722 00:32:03,128 --> 00:32:07,538 With CDP systems, A lot of them missed out. 723 00:32:07,578 --> 00:32:11,828 Now, they've gotten better, but back then, none of them really supported 724 00:32:12,138 --> 00:32:14,728 application integration in a proper way. 725 00:32:14,728 --> 00:32:19,408 Yes, some would do VSS integration to allow you to do like poor bands 726 00:32:19,418 --> 00:32:22,798 backup, but for the most part, they were CDP systems operated 727 00:32:22,798 --> 00:32:24,038 at an infrastructure level. 728 00:32:24,693 --> 00:32:29,443 And so, it didn't have that capability that, honestly, like, as a backup 729 00:32:29,443 --> 00:32:33,253 person, you cared about the application more than the storage, right? 730 00:32:33,253 --> 00:32:36,993 You needed to make sure I had an application consistent backup that I 731 00:32:37,023 --> 00:32:38,703 knew was good that I could recover from. 732 00:32:39,848 --> 00:32:40,738 Yeah, exactly. 733 00:32:40,918 --> 00:32:44,798 One of the challenges is that you say, well, you give me 734 00:32:44,808 --> 00:32:47,618 infinite recovery points, right? 735 00:32:47,658 --> 00:32:57,118 I just want one good one, one point when I know that the the CD, 736 00:32:57,168 --> 00:33:03,578 a lot of the CDP products started integrating more with the database. 737 00:33:03,783 --> 00:33:07,323 So that while they could still give you the infinite point, they could 738 00:33:07,323 --> 00:33:11,073 say, Hey, we also put the database in backup mode at these points in time 739 00:33:11,073 --> 00:33:14,623 so that we know that that point in time is one that is truly consistent 740 00:33:14,623 --> 00:33:17,063 that you could, uh, recover from. 741 00:33:17,193 --> 00:33:20,313 You, you could also use the other points in time, but we're giving you this one 742 00:33:20,313 --> 00:33:21,593 that we know for sure that it's good. 743 00:33:21,853 --> 00:33:22,533 It's special. 744 00:33:22,663 --> 00:33:25,443 so, yeah, it's, it's special, right? 745 00:33:25,823 --> 00:33:29,523 It's still not a snapshot, but it's a point in time when we can say 746 00:33:29,523 --> 00:33:32,453 that, uh, when we can say that we know we can recover to that point. 747 00:33:33,903 --> 00:33:35,873 One other thing about backup, right? 748 00:33:35,913 --> 00:33:38,023 I know we always talk about test your backups, test your 749 00:33:38,023 --> 00:33:39,203 backups, test your backups. 750 00:33:40,043 --> 00:33:43,573 CDP becomes difficult to test in most environments. 751 00:33:44,893 --> 00:33:48,403 Unless you have a lot of additional space and storage, because you don't 752 00:33:48,403 --> 00:33:52,963 necessarily want to stop the copy being updated on the target site. 753 00:33:53,423 --> 00:33:57,753 So now the question becomes, how do I now spin up a separate copy with 754 00:33:57,773 --> 00:34:01,873 that particular point in time that I'm interested in so I can test and verify, 755 00:34:02,293 --> 00:34:05,803 is my Oracle database backup, right? 756 00:34:05,833 --> 00:34:07,703 Is that a good point in time or not? 757 00:34:09,618 --> 00:34:10,278 Exactly. 758 00:34:10,888 --> 00:34:11,608 Yeah. 759 00:34:12,638 --> 00:34:15,728 So it was like, it gave you, it, it gave you almost too much. 760 00:34:16,018 --> 00:34:16,528 Right? 761 00:34:17,438 --> 00:34:21,188 the thing that it gave you that nothing else could give you except 762 00:34:21,188 --> 00:34:23,138 for replication was that RPO of zero. 763 00:34:23,728 --> 00:34:27,238 But it did come with other op it, it came with other. 764 00:34:27,628 --> 00:34:30,828 Complications that you had to, to deal with. 765 00:34:31,208 --> 00:34:34,458 It's like, I often say in IT, we never fix problems. 766 00:34:34,458 --> 00:34:35,248 We just move them. 767 00:34:35,548 --> 00:34:36,048 Right. 768 00:34:36,128 --> 00:34:38,308 So, so we, we solved one problem. 769 00:34:38,308 --> 00:34:40,048 We created, we created some others. 770 00:34:40,498 --> 00:34:44,738 So the other thing I want to talk about is how the. 771 00:34:45,608 --> 00:34:49,598 The, how the data was stored on the target end. 772 00:34:49,948 --> 00:34:55,398 There are two ways, as I understand it, that data was stored on the other end. 773 00:34:55,398 --> 00:35:01,608 There were sort of two ways that the recovery system manifested itself. 774 00:35:01,998 --> 00:35:10,268 One was that there was a volume that we were continuously updating so that if you. 775 00:35:10,633 --> 00:35:18,393 needed to do a recovery, that volume was already replicated to the point in time, 776 00:35:18,433 --> 00:35:24,603 the most recent point in time that you wanted to restore to, and then it also 777 00:35:24,603 --> 00:35:28,373 had a log and the ability to undo that. 778 00:35:29,963 --> 00:35:35,983 Uh, that volume, undo the changes to that volume so that you could take this, 779 00:35:36,523 --> 00:35:41,193 this LUN, right, bring it back in time. 780 00:35:41,553 --> 00:35:44,253 That was the one thing that was, that was really cool. 781 00:35:44,753 --> 00:35:49,963 The, the advantage to that method was that if what you wanted was 782 00:35:49,973 --> 00:35:52,653 right now, you had it immediately. 783 00:35:53,208 --> 00:35:59,388 If you wanted to go back a little bit earlier and the farther back you wanted 784 00:35:59,388 --> 00:36:01,868 to go, the more work had to be done. 785 00:36:01,868 --> 00:36:06,638 And so the longer the recovery took, but that, uh, was the 786 00:36:06,638 --> 00:36:12,428 primary, I think that was the most common way CDP manifested itself, 787 00:36:12,703 --> 00:36:12,973 Yeah. 788 00:36:13,093 --> 00:36:15,603 And I think, like you mentioned, that's a great... 789 00:36:16,098 --> 00:36:20,368 opportunity because most of the times you're probably recovering to the latest 790 00:36:20,388 --> 00:36:25,738 or somewhere near the latest point in time, rather than, hey, I need to go back. 791 00:36:25,768 --> 00:36:30,268 Let me restore all my data from three weeks ago and now replay all 792 00:36:30,278 --> 00:36:35,108 my backups going forward, which leads to a much longer time to recover. 793 00:36:37,208 --> 00:36:37,848 Exactly. 794 00:36:37,918 --> 00:36:45,228 There was this other way where they didn't create the volume that there was no volume 795 00:36:45,238 --> 00:36:47,558 that they were continuously updating. 796 00:36:48,228 --> 00:36:53,008 They essentially had all of the bits necessary to create 797 00:36:53,008 --> 00:36:54,458 the volume at any time. 798 00:36:54,918 --> 00:37:00,988 And then, I, I, this feels very NetApp y, and, and, right, although, and by that 799 00:37:00,988 --> 00:37:04,678 I don't mean this is the way NetApp did it, it's just, you know, the way with 800 00:37:04,678 --> 00:37:10,998 NetApp is, is a given snapshot is really just a bunch of pointers to blocks at 801 00:37:10,998 --> 00:37:16,273 a particular point in time, right, and it's so, when you restore a volume To 802 00:37:16,273 --> 00:37:19,273 a particular point in time, all you're doing is moving all the snapshots. 803 00:37:19,913 --> 00:37:24,183 What they're doing is they have all of the bits and pieces that are necessary to 804 00:37:24,183 --> 00:37:26,933 represent the volume at any point in time. 805 00:37:27,263 --> 00:37:29,043 And then you, 806 00:37:29,378 --> 00:37:29,738 Stitch it 807 00:37:29,893 --> 00:37:32,243 you just had to create all the pointers, right? 808 00:37:32,243 --> 00:37:36,203 There was no, there wasn't to restore so much as there was this, 809 00:37:36,773 --> 00:37:38,153 I don't know what to call it. 810 00:37:38,173 --> 00:37:39,663 It's unlike anything I've ever seen. 811 00:37:39,823 --> 00:37:40,933 I really need a whiteboard. 812 00:37:40,933 --> 00:37:41,553 I think too. 813 00:37:42,248 --> 00:37:45,778 To illustrate this method, the real advantage of this was that 814 00:37:46,348 --> 00:37:50,948 the recovery time was always the same regardless of whether or not 815 00:37:50,978 --> 00:37:55,138 you wanted to go to the most recent point in time or three weeks ago. 816 00:37:57,048 --> 00:38:01,138 because you're just at that point, just manipulating pointers and metadata and 817 00:38:01,138 --> 00:38:03,478 not actually copying and restoring data. 818 00:38:05,428 --> 00:38:11,458 And I know that some of those systems They also, we started talking, we started 819 00:38:11,458 --> 00:38:14,388 using this term copy data management when we started talking about some of 820 00:38:14,388 --> 00:38:18,278 the systems because they could say, hey, here's this, here's this volume from 821 00:38:18,278 --> 00:38:21,248 this point in time and from this point in time and from this point in time, and 822 00:38:21,248 --> 00:38:24,838 you can have all three of them at the same time because you could not do that. 823 00:38:24,848 --> 00:38:25,658 That was the other feature. 824 00:38:26,258 --> 00:38:27,118 Of the other method. 825 00:38:27,128 --> 00:38:30,618 You could not have the same volume at multiple points in time. 826 00:38:30,958 --> 00:38:34,888 This method allows you to have as many, no, you think you could. 827 00:38:35,438 --> 00:38:39,928 There are ways with newer technologies to get that. 828 00:38:39,988 --> 00:38:40,608 One method. 829 00:38:40,608 --> 00:38:44,808 Some vendors used was to update that copy, take a snapshot 830 00:38:44,918 --> 00:38:46,028 and present the snapshot out. 831 00:38:47,183 --> 00:38:49,143 Right, so that's one method. 832 00:38:49,143 --> 00:38:51,473 Now, not always the most optimal, but yeah, 833 00:38:52,108 --> 00:38:52,518 right. 834 00:38:52,778 --> 00:38:53,108 Yeah. 835 00:38:53,118 --> 00:38:57,908 I was just thinking that like a single volume can't be presented at multiple 836 00:38:57,908 --> 00:38:58,998 points in time, but you're right. 837 00:38:59,018 --> 00:39:01,858 If you do a snapshot, then yes, you could do, you could do exactly that. 838 00:39:02,803 --> 00:39:03,973 but it's more management 839 00:39:04,018 --> 00:39:04,918 did it like that? 840 00:39:06,988 --> 00:39:10,108 I said, I wonder what vendor was really good at doing snapshots. 841 00:39:10,688 --> 00:39:18,868 So CDP, Continuous Data Protection, is the system that allows you to have an RPO 842 00:39:19,058 --> 00:39:26,418 and an RTO of zero without the risk that you have with replication, where, where 843 00:39:26,438 --> 00:39:31,108 if you have the, something bad happening to your primary data, Uh, from a logical 844 00:39:31,108 --> 00:39:35,048 basis, you, you drop a table, you do something stupid, you get a cyber attack 845 00:39:35,628 --> 00:39:41,628 that it gives you that power that you had with replication, but it also gives you 846 00:39:41,628 --> 00:39:43,538 the power to be able to go back in time. 847 00:39:43,538 --> 00:39:46,018 So it gives you basically the best of both worlds. 848 00:39:46,748 --> 00:39:50,848 It gives you an infinite number of recovery points, but an 849 00:39:50,848 --> 00:39:52,598 infinite turns out might not. 850 00:39:54,318 --> 00:39:55,188 I like it. 851 00:39:55,188 --> 00:39:56,108 Anybody living space. 852 00:39:56,193 --> 00:39:57,233 Yeah, that exactly. 853 00:39:57,233 --> 00:39:58,323 See, you know where I was going. 854 00:39:59,128 --> 00:40:00,728 yeah, exactly. 855 00:40:00,788 --> 00:40:01,778 Uh, but it. 856 00:40:03,168 --> 00:40:06,978 It turns out that infinite is, is not as amazing as it seems. 857 00:40:06,988 --> 00:40:11,588 Infinite number of recovery points comes with its own challenges, but the biggest 858 00:40:11,588 --> 00:40:15,768 challenge I think with CDP is just cost. 859 00:40:16,198 --> 00:40:28,178 That very few people were comfortable with The cost of using CDP as their only data 860 00:40:28,178 --> 00:40:31,688 protection method for a given set of data. 861 00:40:32,408 --> 00:40:37,458 And so they would, what you would most commonly see is we're only going to 862 00:40:37,458 --> 00:40:39,998 use it for our most critical apps. 863 00:40:40,668 --> 00:40:44,958 Or we're going to use it, but we're also going to use a traditional backup. 864 00:40:45,558 --> 00:40:49,758 Because that's what we're, I don't know, I don't know about you, but I, 865 00:40:49,858 --> 00:40:54,328 I'm always on the lookout for something that can do, that can give me everything 866 00:40:54,328 --> 00:40:59,118 that I want, give me that long term retention to be able to go back when 867 00:40:59,118 --> 00:41:03,958 I realized that I did something stupid three months ago and also have an 868 00:41:03,968 --> 00:41:07,378 RPO and an RTO of, of close to zero. 869 00:41:07,533 --> 00:41:08,343 on a unicorn. 870 00:41:09,473 --> 00:41:12,823 I want a unicorn, but there are, there are ways, and we're going to talk about 871 00:41:12,823 --> 00:41:16,563 some of those ways, to give you an RPO and an RTO way better than what we 872 00:41:16,563 --> 00:41:24,573 traditionally had without perhaps the cost and the, the downsides and the 873 00:41:24,583 --> 00:41:27,713 logistical challenges that CDP offered. 874 00:41:27,813 --> 00:41:32,743 I think in the end it cut, there are CDP products, and for certain 875 00:41:32,743 --> 00:41:37,813 applications, for certain environments, It's like the way to do it, right? 876 00:41:38,143 --> 00:41:42,933 It's just, I think what you're seeing is the complexities of this 877 00:41:42,933 --> 00:41:44,583 and the costs associated with this. 878 00:41:44,783 --> 00:41:49,413 This is why it's still a niche play. 879 00:41:49,933 --> 00:41:52,993 And that's why there's only a handful of these products available out there. 880 00:41:54,133 --> 00:41:54,343 What do you 881 00:41:54,443 --> 00:41:54,993 I agree. 882 00:41:57,333 --> 00:41:57,763 Yes. 883 00:41:58,283 --> 00:41:58,723 All right. 884 00:41:58,773 --> 00:42:03,473 Well, hopefully, uh, for those of you that have always wondered what CDP is, now 885 00:42:03,473 --> 00:42:08,073 you know, and now you know why it didn't solve all problems in data protection. 886 00:42:08,333 --> 00:42:15,233 But I would just say, if you want an RPO and an RTO of zero, and you don't want to 887 00:42:15,243 --> 00:42:17,353 have the issue with replication, right? 888 00:42:17,353 --> 00:42:20,603 Which means, right, we've already talked about the, if you don't want to have 889 00:42:20,603 --> 00:42:27,553 the issues that replication causes, then Uh, CDP is really the only game in town. 890 00:42:27,623 --> 00:42:34,483 So hopefully this honest assessment of CDP will allow the very small 891 00:42:34,483 --> 00:42:39,493 percentage of you that need it to know that sounds exactly like what I need. 892 00:42:40,053 --> 00:42:42,523 Uh, and with that, that's a wrap. 893 00:42:42,818 --> 00:42:47,228 The backup wrap up is a production of backup central.com where you'll find my 894 00:42:47,228 --> 00:42:49,478 blog and a list of services I can provide. 895 00:42:49,928 --> 00:42:51,758 This is an independent podcast. 896 00:42:51,758 --> 00:42:56,438 And any opinions that you hear are those of the speaker and not necessarily 897 00:42:56,618 --> 00:42:58,508 any companies that they work for. 898 00:42:58,928 --> 00:43:02,078 We'll see you next week on the backup wrap up.