1 00:00:00,000 --> 00:00:04,259 I'm not sure who needs to hear this, but snapshots are not backups. 2 00:00:04,739 --> 00:00:07,859 And in this context, I'm talking about virtual snapshots, like 3 00:00:07,859 --> 00:00:09,329 what you do with the storage re. 4 00:00:09,780 --> 00:00:12,750 We're not talking about cloud snapshots, like what you do 5 00:00:12,750 --> 00:00:15,060 with EBS volumes in the AWS. 6 00:00:15,629 --> 00:00:19,649 If you don't know the difference or you can't articulate why 7 00:00:19,649 --> 00:00:21,869 these snapshots are not backups. 8 00:00:22,140 --> 00:00:24,299 Well, have I got a podcast for you? 9 00:00:24,629 --> 00:00:25,569 Hi, I'm W. 10 00:00:25,569 --> 00:00:27,720 Curtis Preston AKA Mister backup. 11 00:00:27,959 --> 00:00:33,209 And my podcast turns unappreciated backup admins into cyber recovery heroes. 12 00:00:33,479 --> 00:00:35,670 This is the backup wrap-up. 13 00:00:46,983 --> 00:00:48,853 Hi, and welcome to the Backup Wrap Up. 14 00:00:48,853 --> 00:00:49,673 I'm your host, W. 15 00:00:49,683 --> 00:00:54,313 Curtis Preston, and I have with me my Spanish language 16 00:00:55,233 --> 00:00:57,873 encourager, Prasanna Malaiyandi. 17 00:00:57,893 --> 00:00:59,313 I gave you a positive thing 18 00:00:59,498 --> 00:01:01,458 I know, I'm impressed. 19 00:01:01,458 --> 00:01:04,708 And I'm also impressed that you are done now with, what, Spanish 2, right? 20 00:01:05,583 --> 00:01:10,613 Spanish 2, uh, Sisi termine con Span, uh, I almost said 21 00:01:12,213 --> 00:01:15,153 Spanish, Espanol 2, um, yeah. 22 00:01:15,153 --> 00:01:21,033 So now I need to, I need to do a little review because there's, it's, it's 23 00:01:21,033 --> 00:01:26,483 really, you know, for a, for an English speaker, uh, the challenges, as I know, 24 00:01:26,493 --> 00:01:30,813 I've spoken to you, the challenges are things like, we're like, it, it, the, 25 00:01:30,973 --> 00:01:32,973 the gender thing is, is not a big deal. 26 00:01:32,973 --> 00:01:34,003 You get used to that. 27 00:01:34,293 --> 00:01:38,877 What's challenging is words that n an ma like PMA you would 28 00:01:38,877 --> 00:01:41,397 think that would be feminine, but it's actually a masculine word. 29 00:01:41,747 --> 00:01:44,417 Uh, because it, it ends in an, uh, ma. 30 00:01:44,907 --> 00:01:47,827 I have to pound that stuff into my head and just say it over and over. 31 00:01:48,307 --> 00:01:51,627 I will be using, the thing that I made where I, I'm actually 32 00:01:51,627 --> 00:01:55,177 using technology to make a verbal recording that I can listen to. 33 00:01:55,807 --> 00:01:57,357 Uh, it's very, very cool. 34 00:01:57,497 --> 00:02:02,087 Uh, but I am excited to move on to Spanish three and. 35 00:02:02,487 --> 00:02:03,807 Bring you dragging with me. 36 00:02:03,927 --> 00:02:04,527 It's helpful. 37 00:02:04,527 --> 00:02:07,827 I, Used to be pretty good at Spanish because I took it in high 38 00:02:07,827 --> 00:02:10,407 school and then I lost everything. 39 00:02:10,417 --> 00:02:13,007 I can order things off a menu, but that's about it. 40 00:02:14,267 --> 00:02:14,767 Right. 41 00:02:14,807 --> 00:02:15,327 Right. 42 00:02:15,527 --> 00:02:20,257 So, uh, I think it's time for the news of the week. 43 00:02:23,409 --> 00:02:24,609 this one is frustrating. 44 00:02:24,639 --> 00:02:26,829 Uh, we'll, we'll, we'll do a frustrating one first and then 45 00:02:26,829 --> 00:02:28,299 some good news about a vendor. 46 00:02:29,089 --> 00:02:30,950 Let's talk about this one password hack. 47 00:02:31,849 --> 00:02:35,259 And it frustrates me for many reasons. 48 00:02:35,849 --> 00:02:39,079 One is, we're such a fan of password managers. 49 00:02:39,749 --> 00:02:43,269 And there are those who are not fans of password managers. 50 00:02:43,269 --> 00:02:45,759 There are also those who are not a fan of the cloud. 51 00:02:46,249 --> 00:02:50,509 And this just, uh, is both of those things. 52 00:02:50,849 --> 00:02:55,039 So, uh, do you want to talk a little bit about the 1Password slash Octahack? 53 00:02:55,429 --> 00:03:00,449 so basically what happened is 1Password, I don't know if it was 54 00:03:00,449 --> 00:03:05,899 1Password or Okta, but basically some hackers got in into 1Password. 55 00:03:05,969 --> 00:03:12,059 They were able to change some things in their Okta instance and try to get a list 56 00:03:12,059 --> 00:03:16,579 of admins, which I'm sure they were going to use to target and be able to deploy 57 00:03:16,579 --> 00:03:18,459 things so they could cause bigger damage. 58 00:03:19,549 --> 00:03:19,959 Right. 59 00:03:20,989 --> 00:03:23,729 But as they were starting to sort of peel back the onion and 60 00:03:23,729 --> 00:03:25,309 figure out, okay, what happened? 61 00:03:25,689 --> 00:03:29,899 They realized what actually happened is it started on the Okta side, 62 00:03:30,959 --> 00:03:31,539 Right. 63 00:03:31,639 --> 00:03:35,059 The reason they were able to do what, what they ended up being able 64 00:03:35,059 --> 00:03:39,179 to do was because Okta had already been, had already been hacked. 65 00:03:41,379 --> 00:03:45,634 And so it's interesting in this case because It wasn't like, oh, they 66 00:03:45,634 --> 00:03:49,704 just got into Okta and then they ping ponged and got into 1Password. 67 00:03:50,094 --> 00:03:56,764 They actually looked at support files that someone in 1Password, an engineer, 68 00:03:56,764 --> 00:03:59,874 had uploaded because they probably had some issue or something else like that. 69 00:03:59,874 --> 00:04:03,994 They were reaching out to Okta and they uploaded basically a support bundle. 70 00:04:04,364 --> 00:04:08,824 Um, and in that support bundle, they contained, in addition to sort of like 71 00:04:09,224 --> 00:04:12,814 information needed to troubleshoot, it also contained session cookies, 72 00:04:13,959 --> 00:04:14,589 Right. 73 00:04:15,479 --> 00:04:17,129 Which they were then able to use. 74 00:04:17,159 --> 00:04:17,509 Right. 75 00:04:17,704 --> 00:04:24,174 Yeah, and so they were able to basically impersonate the one password employee 76 00:04:24,244 --> 00:04:28,704 and log into the system and that's when they started causing havoc. 77 00:04:30,259 --> 00:04:30,549 Yeah. 78 00:04:30,549 --> 00:04:36,939 And, and the good news is that they did, that 1Password did see what was happening. 79 00:04:37,279 --> 00:04:41,399 Uh, they did see this weird session coming from an odd IP. 80 00:04:42,049 --> 00:04:46,229 And they, they shut it down before basically all they did was 81 00:04:46,229 --> 00:04:48,399 attempt to get a list of admins. 82 00:04:48,429 --> 00:04:51,609 They didn't actually get the list of admins is what it looks like. 83 00:04:52,119 --> 00:04:57,459 Um, but the, the whole thing was again, you know, it went back to 84 00:04:57,459 --> 00:05:04,109 this, the fact that the, the hacker initially had compromised the. 85 00:05:06,184 --> 00:05:10,614 The Okta support system, which they then got this support file. 86 00:05:11,244 --> 00:05:18,444 And, you know, in retrospect, it looks like everybody did their job. 87 00:05:18,484 --> 00:05:20,934 It, it looks like, right. 88 00:05:20,934 --> 00:05:21,724 That, that, that. 89 00:05:22,589 --> 00:05:24,479 Things were noticed, things were stopped. 90 00:05:24,569 --> 00:05:29,039 I think that in the case of Okta, maybe they weren't noticed quick 91 00:05:29,039 --> 00:05:34,349 enough because it actually, 1Password was just one company of several that 92 00:05:34,349 --> 00:05:35,959 were compromised because of this. 93 00:05:36,339 --> 00:05:39,729 It doesn't appear for those of you that are 1Password customers, it doesn't 94 00:05:39,729 --> 00:05:43,489 appear that any customer data was accessed, doesn't appear, you know, 95 00:05:43,529 --> 00:05:46,469 unlike the, what was the other one? 96 00:05:46,469 --> 00:05:51,489 The LastPass, the LastPass hack where they actually got the vault. 97 00:05:52,324 --> 00:05:58,574 And then you had to worry if you had insecure passwords, right? 98 00:05:58,574 --> 00:06:01,574 That, that, that, that there were guessable, but it doesn't appear 99 00:06:01,574 --> 00:06:03,004 that that was the case here. 100 00:06:03,004 --> 00:06:04,324 They found it pretty quickly. 101 00:06:05,254 --> 00:06:07,814 think another interesting thing that I found, by the way, this is an 102 00:06:07,824 --> 00:06:12,044 article on The Register, we'll put a link in the show notes description. 103 00:06:12,204 --> 00:06:12,624 Right. 104 00:06:14,144 --> 00:06:16,194 The one thing I found interesting is they were trying to figure 105 00:06:16,194 --> 00:06:17,754 out, okay, what actually happened? 106 00:06:17,754 --> 00:06:22,644 Like, when was This information stolen and so they went back and they looked at 107 00:06:22,644 --> 00:06:26,544 their logs on the Okta side and they were trying to figure out okay was the archive 108 00:06:26,544 --> 00:06:30,424 access before the support engineer on the Okta side accessed it and they were able 109 00:06:30,424 --> 00:06:34,114 to figure out no the support engineer hadn't opened it yet so it wasn't a 110 00:06:34,114 --> 00:06:36,094 rogue support engineer on the Okta side. 111 00:06:36,644 --> 00:06:41,814 And then they looked at the 1Pass site, 1Password site and they saw, 112 00:06:41,854 --> 00:06:46,474 okay, the person who uploaded it was on a public Wi Fi at a hotel. 113 00:06:46,934 --> 00:06:50,084 And they were like, oh, maybe it could have been stolen during the upload 114 00:06:50,094 --> 00:06:52,974 because you know, those connections are always unencrypted and all the rest. 115 00:06:52,984 --> 00:06:55,884 But they looked and they're like, no, the upload process 116 00:06:55,884 --> 00:06:57,924 had TLS end to end up to Okta. 117 00:06:57,924 --> 00:06:58,364 And so. 118 00:06:58,694 --> 00:06:59,714 Everything was encrypted. 119 00:06:59,714 --> 00:07:01,394 It wasn't stolen in the process. 120 00:07:01,394 --> 00:07:05,044 So there must've been something else that had accessed it on the Okta side. 121 00:07:05,044 --> 00:07:07,764 So it's interesting how they're able to piece this all together. 122 00:07:07,764 --> 00:07:09,044 It's almost like CSI, right? 123 00:07:09,364 --> 00:07:11,554 You're piecing together all these clues, trying to figure out, okay, 124 00:07:11,554 --> 00:07:12,784 what happened, what went wrong? 125 00:07:13,964 --> 00:07:17,394 Well, that's what you have to do in an incident response system, right? 126 00:07:17,404 --> 00:07:21,344 You've got to, you know, piece together what you can from the logs that you have. 127 00:07:21,354 --> 00:07:23,054 Logs, logs, logs, right? 128 00:07:23,124 --> 00:07:24,794 That's what it's all about is the logs. 129 00:07:25,354 --> 00:07:30,674 The, the, the only thing that's somewhat distressing is that it 130 00:07:30,674 --> 00:07:32,934 appeared that they made some tweaks. 131 00:07:33,559 --> 00:07:36,029 Some upgrades to their MFA. 132 00:07:36,209 --> 00:07:40,409 Uh, and by upgrades, I mean like they changed some of their stuff. 133 00:07:40,419 --> 00:07:45,019 They're like, maybe we shouldn't allow anyone to log into Okta who isn't 134 00:07:45,079 --> 00:07:50,259 at a, um, a one password IP, right? 135 00:07:50,279 --> 00:07:52,049 Maybe, maybe we shouldn't allow that. 136 00:07:52,109 --> 00:07:53,369 That seems like 137 00:07:53,444 --> 00:07:54,154 Uh, that's, 138 00:07:54,339 --> 00:07:57,319 you should have decided that before, but yeah, 139 00:07:57,429 --> 00:08:00,649 know about that, Curtis, because there are times, like, you might 140 00:08:00,649 --> 00:08:03,619 be traveling, and you might be the super user, and you have to change 141 00:08:03,619 --> 00:08:06,069 something, and you're not at a desk. 142 00:08:06,089 --> 00:08:10,789 Or, imagine that one password blows up, something happens to their site, 143 00:08:10,809 --> 00:08:15,469 and they need to reset passwords, and they could eventually get locked out. 144 00:08:15,874 --> 00:08:16,604 But maybe this is an 145 00:08:16,674 --> 00:08:19,884 I would think, yeah, I would, uh, I would think that that would be an 146 00:08:19,884 --> 00:08:23,574 exception case that you, you would deal with that maybe by the default 147 00:08:23,584 --> 00:08:24,734 would be that you would turn it off. 148 00:08:25,074 --> 00:08:28,554 The other thing was that it appears that now they're using Yubikeys. 149 00:08:28,924 --> 00:08:31,444 For those that aren't familiar with that, Y U B I K E Y. 150 00:08:33,624 --> 00:08:35,574 This is a hardware token. 151 00:08:35,934 --> 00:08:39,699 It's pretty inexpensive as these systems go. 152 00:08:40,089 --> 00:08:44,949 But it is a a hard, you know, it's it's an actual system that you can actually 153 00:08:44,949 --> 00:08:48,609 buy for your own personal use I took a look and it was I think it was a 154 00:08:48,609 --> 00:08:51,709 hundred dollars for two of them Um, 155 00:08:51,784 --> 00:08:52,364 you want to. 156 00:08:52,379 --> 00:08:54,489 you know, and you want two. 157 00:08:54,499 --> 00:09:00,119 Yeah, and so, um, then Uh, and, you know, and so it looks like they're 158 00:09:00,119 --> 00:09:03,389 now using YubiKeys where maybe before they weren't using YubiKeys. 159 00:09:04,239 --> 00:09:05,789 I'm glad that they made those steps. 160 00:09:05,789 --> 00:09:10,519 It's just, it's just, it's just a shame that we always make changes to 161 00:09:10,519 --> 00:09:15,329 systems after we've been, you know, after we've had an exposure, but good 162 00:09:15,329 --> 00:09:18,499 on them, good on them for finding it good on them for responding and 163 00:09:18,499 --> 00:09:20,269 saying, Hey, we could make this better. 164 00:09:20,734 --> 00:09:23,204 Uh, I'm a little dinging Okta here. 165 00:09:23,214 --> 00:09:28,514 There was a thing at the end that Okta basically said that perhaps if you're 166 00:09:28,524 --> 00:09:31,944 sending a support file, uh, what was the, what was the type of that file? 167 00:09:31,954 --> 00:09:33,014 A HAR file? 168 00:09:33,474 --> 00:09:35,604 Um, some sort of archive file. 169 00:09:35,664 --> 00:09:35,974 Yeah. 170 00:09:36,654 --> 00:09:37,044 Yeah. 171 00:09:37,074 --> 00:09:40,074 And they're like, perhaps you shouldn't, perhaps you should sanitize 172 00:09:40,074 --> 00:09:41,824 that before you send it to us. 173 00:09:41,824 --> 00:09:46,224 And I'm like, well, maybe your system that creates the HAR file 174 00:09:46,234 --> 00:09:47,924 should sanitize it for the customer. 175 00:09:48,254 --> 00:09:49,214 Before you upload it. 176 00:09:49,334 --> 00:09:53,084 Uh, there was a little bit I thought of Victor blaming, uh, towards the end there, 177 00:09:53,084 --> 00:09:59,374 but, um, anyway, a, a, a much, I think one that's gonna be a, a lot easier to 178 00:09:59,374 --> 00:10:02,904 talk about our former employer, Druva. 179 00:10:03,544 --> 00:10:06,784 Uh, there, the headline here, another, it's a tech target article 180 00:10:06,789 --> 00:10:12,274 that they've added a gen AI assistant to their cloud backup tool. 181 00:10:12,619 --> 00:10:18,639 And I watched a, like a video of a demo and it basically, it looked like, uh, 182 00:10:18,699 --> 00:10:24,479 you know, an NLM that you can use to interface with the Druva cloud platform. 183 00:10:24,869 --> 00:10:30,409 And so you could ask it things like, Hey, show me my backups, like in, in like, Hey, 184 00:10:30,419 --> 00:10:32,439 show me my backup backups have failed. 185 00:10:32,489 --> 00:10:38,129 I think the big thing is, especially with, uh, with the sort of focus on 186 00:10:38,149 --> 00:10:43,069 AI and making it easily available to a large audience without needing 187 00:10:43,069 --> 00:10:45,029 to know all the training, Right. 188 00:10:45,059 --> 00:10:49,089 And being an expert in it, I think has now made it sort of commonplace, right? 189 00:10:49,089 --> 00:10:53,149 It's easy to pick up an AI model that has been pre trained 190 00:10:53,159 --> 00:10:54,579 and use it for some purpose. 191 00:10:55,509 --> 00:10:55,899 right? 192 00:10:55,899 --> 00:10:59,139 In their case, they're using, uh, Amazon's, uh, no, no. 193 00:10:59,139 --> 00:10:59,979 Great surprise there. 194 00:10:59,979 --> 00:11:03,509 They're using Amazon's, uh, NLM model, 195 00:11:03,629 --> 00:11:04,589 you referring to? 196 00:11:05,179 --> 00:11:08,439 Sorry, are you referring to NLP or LLM? 197 00:11:09,069 --> 00:11:10,149 You're calling it NLM. 198 00:11:10,209 --> 00:11:10,929 thank you. 199 00:11:10,989 --> 00:11:11,799 Oh, yeah, sorry. 200 00:11:12,279 --> 00:11:13,329 NLP Thank you. 201 00:11:13,329 --> 00:11:14,319 NL NLP. 202 00:11:14,679 --> 00:11:19,349 So they're, they're using Amazon's, uh, product to do this. 203 00:11:19,349 --> 00:11:19,619 No. 204 00:11:19,619 --> 00:11:20,429 Great surprise. 205 00:11:20,609 --> 00:11:25,739 DVA lives in Amazon, and, uh, it, it seems like the biggest thing that 206 00:11:25,739 --> 00:11:29,159 they had to do was to make sure that it was integrated with the security. 207 00:11:29,609 --> 00:11:35,439 That's the biggest thing that I see is, yeah, you have to make sure because if 208 00:11:35,439 --> 00:11:40,579 you just pick up any random LLM, I'll use LLM because that's a large language 209 00:11:40,599 --> 00:11:43,679 model, right, which a lot of the AI models are built on top of, if you. 210 00:11:43,884 --> 00:11:46,574 Pick that and depending on what is trained, you might get back bogus 211 00:11:46,584 --> 00:11:50,964 answers, random answers, answers that aren't even safe, right? 212 00:11:51,484 --> 00:11:56,594 And so at least the fact that Druva is putting guardrails in place to make 213 00:11:56,594 --> 00:12:03,274 sure that what gets returned is sensible and safe, I think is a good step. 214 00:12:04,979 --> 00:12:05,199 Yeah. 215 00:12:05,199 --> 00:12:09,699 So we've seen, uh, we've seen AI use with Alcion. 216 00:12:09,719 --> 00:12:13,469 We saw it with, um, I'm trying to think. 217 00:12:13,589 --> 00:12:13,859 Yeah. 218 00:12:13,859 --> 00:12:15,519 Cohesity came out with one. 219 00:12:15,549 --> 00:12:16,319 Do you remember who else? 220 00:12:16,379 --> 00:12:17,039 It was Druva. 221 00:12:18,089 --> 00:12:19,809 I'm trying to, I know there's another one. 222 00:12:20,179 --> 00:12:20,869 Um, 223 00:12:21,689 --> 00:12:22,479 Was it Commvault? 224 00:12:22,594 --> 00:12:23,314 in the thing. 225 00:12:23,734 --> 00:12:27,884 Oh, Dell, Dell, Dell said, yeah, I haven't, I don't think I've seen 226 00:12:27,884 --> 00:12:31,884 anything from Commvault, but, uh, Dell is also doing, uh, yeah, I merged, 227 00:12:31,884 --> 00:12:35,354 I think, in, uh, uh, uh, NLP and LLM 228 00:12:35,569 --> 00:12:36,339 To NLM. 229 00:12:37,644 --> 00:12:39,164 It's, it's a Curtis only thing. 230 00:12:39,884 --> 00:12:41,714 So, yeah, so that's interesting. 231 00:12:41,794 --> 00:12:47,194 Uh, I think anything that makes it easier to interface with your 232 00:12:47,194 --> 00:12:48,324 backup system is a good thing. 233 00:12:48,704 --> 00:12:51,804 As long as they have those guardrails in place and 234 00:12:52,369 --> 00:12:57,189 And I know you always like to talk about how, what's the job, what's the, one of 235 00:12:57,189 --> 00:13:00,479 the most important jobs that you give to the most junior person at a company? 236 00:13:01,804 --> 00:13:02,484 exactly. 237 00:13:03,039 --> 00:13:03,999 Backup, right? 238 00:13:04,319 --> 00:13:04,959 And so, 239 00:13:05,044 --> 00:13:05,804 why would you do 240 00:13:05,899 --> 00:13:11,269 yeah, and so giving an assistant, if you will, right, to that junior backup person 241 00:13:11,419 --> 00:13:14,179 is helpful as they're learning the ropes. 242 00:13:16,034 --> 00:13:16,284 Yeah. 243 00:13:16,364 --> 00:13:19,024 Well, that is the news of the week. 244 00:13:22,618 --> 00:13:26,598 As you know, every episode of the Backup Wrap Up is going to dive 245 00:13:26,608 --> 00:13:29,268 deep into one particular topic. 246 00:13:29,388 --> 00:13:33,958 This week's topic is snapshot, snapshot, snapshot. 247 00:13:35,278 --> 00:13:35,878 Click, click, click, 248 00:13:35,918 --> 00:13:37,788 It's a topic that comes up a lot. 249 00:13:38,328 --> 00:13:41,888 This word comes up a lot on this show. 250 00:13:42,188 --> 00:13:48,818 So it's time to dive deep into a world that I, I bet at one point in your career. 251 00:13:49,283 --> 00:13:51,133 Persona, you must've heard this. 252 00:13:51,813 --> 00:13:53,883 Word a hundred times a day. 253 00:13:54,123 --> 00:13:54,743 What do you think? 254 00:13:54,778 --> 00:13:55,528 At least. 255 00:13:56,673 --> 00:13:57,543 At least, 256 00:13:57,598 --> 00:13:58,138 least. 257 00:13:59,183 --> 00:14:05,163 because there's certainly one company that thinks they do snapshots different 258 00:14:05,183 --> 00:14:07,653 and better than everyone else. 259 00:14:07,733 --> 00:14:13,333 And they're probably, they certainly, it's certainly at one point in time. 260 00:14:13,333 --> 00:14:13,853 That was true. 261 00:14:13,853 --> 00:14:18,313 I think a number of other vendors are now doing snapshots 262 00:14:18,323 --> 00:14:20,483 the way NetApp did snapshots. 263 00:14:22,023 --> 00:14:27,653 So just a quick story just sort of bring why different ways 264 00:14:27,663 --> 00:14:30,633 to do snapshots really matter. 265 00:14:32,233 --> 00:14:40,983 I was at a, I'm in my brain, uh, live translating the story because I 266 00:14:40,983 --> 00:14:48,273 was at one of the largest companies in the world at a consulting gig. 267 00:14:48,493 --> 00:14:54,463 And we were helping them to pick a new storage and backup system. 268 00:14:54,463 --> 00:14:59,443 They were looking for an integrated system that would do both storage and 269 00:14:59,513 --> 00:15:02,243 data protection as part of that storage. 270 00:15:02,803 --> 00:15:08,703 They were already a NetApp customer and they knew that that meant that 271 00:15:08,703 --> 00:15:10,423 they knew every foible of NetApp. 272 00:15:11,253 --> 00:15:14,793 So, uh, they knew every bad thing about NetApp, but they 273 00:15:14,793 --> 00:15:15,733 also knew the good things. 274 00:15:16,713 --> 00:15:21,633 And I think of all of the consulting gigs that I've done throughout the years, 275 00:15:21,653 --> 00:15:23,953 they had done the best at presenting. 276 00:15:24,563 --> 00:15:26,503 These are our requirements. 277 00:15:27,218 --> 00:15:33,038 And they are well defined and there are reasons behind every one of them. 278 00:15:33,353 --> 00:15:33,743 Mm hmm. 279 00:15:34,018 --> 00:15:42,978 And one of their requirements was that they wanted end users, just regular Joe 280 00:15:42,978 --> 00:15:46,058 and Jane person sitting in the desktop. 281 00:15:46,808 --> 00:15:49,088 To be able to do their own resource. 282 00:15:49,548 --> 00:15:49,928 Right. 283 00:15:50,588 --> 00:15:56,408 And they wanted them to have the ability to do that at points in 284 00:15:56,408 --> 00:16:01,458 time that were like an hour at a time throughout the day, going back 285 00:16:01,648 --> 00:16:03,868 to, you know, it was like 90 days. 286 00:16:03,878 --> 00:16:10,698 So specifically they said, this is why we want 90 days of user browsable snapshots. 287 00:16:11,198 --> 00:16:12,388 At the time that was like. 288 00:16:13,273 --> 00:16:15,053 Not everybody did that, right? 289 00:16:15,683 --> 00:16:22,673 Depending on how you did snapshots as a vendor, you either could or 290 00:16:22,693 --> 00:16:24,253 could not meet that requirement. 291 00:16:24,303 --> 00:16:25,463 Just end of story. 292 00:16:25,843 --> 00:16:35,033 And so, the, the, the responses ranged from, uh, no problem, right? 293 00:16:35,133 --> 00:16:36,293 Literally no problem. 294 00:16:36,653 --> 00:16:40,108 To, I remember one vendor coming in and going, That's the 295 00:16:40,118 --> 00:16:41,798 dumbest requirement we've ever 296 00:16:42,148 --> 00:16:42,998 I bet I can 297 00:16:43,278 --> 00:16:46,568 you want 90 days of user browsable snapshots? 298 00:16:46,848 --> 00:16:49,028 Yeah, I think you know exactly who that was. 299 00:16:49,240 --> 00:16:53,730 It was just chaos and by the way There was a vendor that came in and they had 300 00:16:53,740 --> 00:16:59,790 this let me restate that I'm pretty sure it was that vendor that had this 301 00:16:59,790 --> 00:17:08,020 sort of Very convoluted system that was based on like you had this block system 302 00:17:08,020 --> 00:17:11,260 and then you had this other system that had the blocks and you could, 303 00:17:11,685 --> 00:17:12,075 Stephen 304 00:17:12,270 --> 00:17:14,330 it was just really, really complicated. 305 00:17:14,330 --> 00:17:19,950 And they had this like this wizard of a presenter that 306 00:17:19,950 --> 00:17:22,130 was just an amazing presenter. 307 00:17:22,750 --> 00:17:29,100 That was very, um, you know, charming and very smart and 308 00:17:29,230 --> 00:17:31,160 just presented all of the stuff. 309 00:17:31,890 --> 00:17:36,280 Uh, and, and even though that presenter did their best, the, the 310 00:17:36,280 --> 00:17:38,720 customer just was not having it. 311 00:17:39,040 --> 00:17:41,330 And even though he was a really great presenter. 312 00:17:41,820 --> 00:17:43,240 Just didn't play. 313 00:17:43,700 --> 00:17:55,295 Anyway, the thing is that the key there is that how you do snapshots It very much 314 00:17:55,295 --> 00:17:57,725 dictates how everything's going to work. 315 00:17:58,295 --> 00:18:00,255 Going back to the requirements, right? 316 00:18:00,255 --> 00:18:02,825 You said that they had a very clear list of things. 317 00:18:03,815 --> 00:18:07,285 Do you know why they had that requirement for 90 days? 318 00:18:07,285 --> 00:18:08,460 Was it to like? 319 00:18:08,780 --> 00:18:11,180 What was the purpose behind having that? 320 00:18:11,180 --> 00:18:14,670 Because I'm sure today everyone kind of thinks oh, that's just reasonable, right? 321 00:18:14,730 --> 00:18:16,130 Oh, of course, why don't I have that? 322 00:18:16,130 --> 00:18:18,840 But back then it seemed like that was something very, 323 00:18:18,955 --> 00:18:24,145 they had very specific numbers on the number of restores that had been done. 324 00:18:24,835 --> 00:18:31,765 And I know this as a backup person, but if you look at just 325 00:18:32,925 --> 00:18:34,485 all of the restores that are done. 326 00:18:35,260 --> 00:18:43,070 Ever anywhere, you know, for the history of restores, 99 percent of 327 00:18:43,070 --> 00:18:45,290 them are done from data from yesterday. 328 00:18:46,450 --> 00:18:47,080 Right. 329 00:18:47,370 --> 00:18:52,630 And then, or, or from the most recent snapshot, and then there is this cliff. 330 00:18:53,380 --> 00:18:56,700 of usage that just gets smaller and smaller. 331 00:18:56,700 --> 00:19:01,320 And it's just this incredibly ever increasing line to zero. 332 00:19:01,700 --> 00:19:05,760 And I think in their world, what they showed was that ever increasing line to 333 00:19:05,760 --> 00:19:08,380 zero basically dropped off at 90 days. 334 00:19:08,380 --> 00:19:11,670 And so they said, we need 90 days of user browsable snapshots. 335 00:19:11,670 --> 00:19:16,280 That's what I meant was that they were really good at articulating what 336 00:19:16,280 --> 00:19:18,400 their requirements were and also. 337 00:19:20,465 --> 00:19:21,555 Why those requirements? 338 00:19:21,565 --> 00:19:22,295 So like, you 339 00:19:22,440 --> 00:19:22,650 yeah. 340 00:19:22,790 --> 00:19:27,160 And I think that's a good lesson though for backup admins, right? 341 00:19:27,180 --> 00:19:31,250 You should be looking at the metrics of these systems because if you want 342 00:19:31,250 --> 00:19:36,060 to make a case for say new technologies or new process improvements or other 343 00:19:36,060 --> 00:19:39,560 things like that, having this data to show why you need something so you could 344 00:19:39,560 --> 00:19:42,495 put it into requirements is critical. 345 00:19:44,215 --> 00:19:45,525 exactly, exactly. 346 00:19:46,045 --> 00:19:48,395 So let's define 347 00:19:48,955 --> 00:19:49,235 What's a 348 00:19:49,545 --> 00:19:50,565 what we mean. 349 00:19:50,895 --> 00:19:51,215 Yeah. 350 00:19:51,215 --> 00:19:52,745 What is a snapshot? 351 00:19:53,825 --> 00:19:55,875 Let me give you my definition, let's see how closely it 352 00:19:55,875 --> 00:19:56,885 lines with what you call it. 353 00:19:57,505 --> 00:19:57,805 Alright. 354 00:19:57,805 --> 00:19:59,055 What's your definition 355 00:19:59,055 --> 00:19:59,215 of a 356 00:19:59,615 --> 00:20:06,405 my definition of a snapshot is a point in time copy of the data that existed 357 00:20:06,415 --> 00:20:11,325 at some point in time, so it has to have been plausible, that can be 358 00:20:11,405 --> 00:20:18,135 preserved such that it's not modified when the primary copy gets modified. 359 00:20:20,475 --> 00:20:24,505 So, I would take your definition and I would insert one word 360 00:20:24,585 --> 00:20:26,265 I think to make it perfect 361 00:20:26,705 --> 00:20:27,055 Okay? 362 00:20:27,315 --> 00:20:35,135 and that is the word virtual at the beginning because It's a virtual 363 00:20:35,175 --> 00:20:41,345 copy, because that is really what differentiates a snapshot from a copy, 364 00:20:41,385 --> 00:20:43,435 because you just said a copy, right? 365 00:20:43,745 --> 00:20:48,255 So it is a, I like to use the word view. 366 00:20:48,855 --> 00:20:50,495 It's a view. 367 00:20:51,100 --> 00:20:55,160 Into your volume that you're protecting with the snapshot 368 00:20:55,230 --> 00:20:56,890 at a different point in time. 369 00:20:57,500 --> 00:21:02,760 And I like the word view because it, it, which is very much a database term, right? 370 00:21:03,290 --> 00:21:07,680 It's just a different, it's a way to look at your current volume 371 00:21:07,730 --> 00:21:09,620 at a different point in time. 372 00:21:10,260 --> 00:21:17,300 The, I think the most important thing that differentiates a true snapshot from a lot 373 00:21:17,300 --> 00:21:18,150 of other things that we call snapshots. 374 00:21:20,080 --> 00:21:31,210 Is that it is a virtual copy in that relies on the primary volume that it is 375 00:21:31,220 --> 00:21:35,390 protecting for most of the blocks of data. 376 00:21:36,010 --> 00:21:39,450 The bulk of the blocks, when you're reading that snapshot, the bulk 377 00:21:39,450 --> 00:21:43,410 of those blocks are going to come from the, the current volume. 378 00:21:45,230 --> 00:21:46,000 Are you with me? 379 00:21:46,700 --> 00:21:47,110 Right. 380 00:21:47,170 --> 00:21:50,840 That basically that the change data is going to come from 381 00:21:50,970 --> 00:21:51,470 snapshot. 382 00:21:52,630 --> 00:21:55,790 some, the snapshot, right, how that happens, that's the difference between 383 00:21:55,790 --> 00:21:59,260 copy on write and redirect on write, but the bulk of the data is going 384 00:21:59,260 --> 00:22:00,820 to come from the current volume. 385 00:22:00,960 --> 00:22:02,200 That's basically what I'm 386 00:22:02,450 --> 00:22:02,790 Okay. 387 00:22:02,960 --> 00:22:03,070 I 388 00:22:03,080 --> 00:22:03,970 we on the same page? 389 00:22:04,595 --> 00:22:04,995 Okay. 390 00:22:05,405 --> 00:22:05,765 All right. 391 00:22:05,815 --> 00:22:12,115 So we can go, I mean, only one of us actually worked at a vendor that did 392 00:22:12,115 --> 00:22:15,445 snapshot, so, you know, want to make sure I'm getting things right here. 393 00:22:16,105 --> 00:22:23,645 Um, this is really the key of the difference between a snapshot and a 394 00:22:23,645 --> 00:22:29,175 copy or a snapshot and a backup why does that matter from a backup perspective? 395 00:22:29,185 --> 00:22:29,665 Why does that 396 00:22:29,795 --> 00:22:35,635 snapshots. 397 00:22:36,435 --> 00:22:37,295 They're not independent. 398 00:22:39,035 --> 00:22:43,655 And that's why, you know, those of us that, you know, care about 399 00:22:43,665 --> 00:22:45,245 things like backup and recovery. 400 00:22:45,245 --> 00:22:49,345 We just like to scream and say, snapshots are not backup. 401 00:22:49,735 --> 00:22:50,115 Right. 402 00:22:50,525 --> 00:22:58,025 Um, Now, I will say that you can use snapshots as a way to get backup, but a 403 00:22:58,025 --> 00:23:06,665 snapshot by itself on a volume is, I like to call it a convenience copy, right? 404 00:23:08,255 --> 00:23:12,975 It is a way to go back in time as long as you don't have media failure. 405 00:23:15,595 --> 00:23:16,115 Right, right. 406 00:23:16,815 --> 00:23:20,005 You don't have a double disk failure in a RAID 5 array or a triple 407 00:23:20,005 --> 00:23:21,665 disk failure in a RAID 6 array. 408 00:23:21,900 --> 00:23:26,790 Yeah, and like you're saying, it could use a snapshot to allow you to do other backup 409 00:23:26,790 --> 00:23:31,520 mechanisms like take a copy, preserve it in a point of time, now you move that 410 00:23:31,520 --> 00:23:33,900 data and you back up that data, right? 411 00:23:33,900 --> 00:23:37,410 Which, if you're integrating with applications, you can now take an 412 00:23:37,410 --> 00:23:40,980 application consistent point in time while your database is in hot 413 00:23:41,000 --> 00:23:44,490 backup mode, take that snapshot, now you preserve that point in time. 414 00:23:45,010 --> 00:23:45,730 You can. 415 00:23:47,405 --> 00:23:52,585 Uh, Tha, the database, so it can continue operating like normal, Tha, the database. 416 00:23:54,755 --> 00:23:55,925 What word are you saying there? 417 00:23:57,435 --> 00:23:57,835 I'll follow. 418 00:24:02,505 --> 00:24:04,865 I just, I don't know what word I thought you were saying there. 419 00:24:05,265 --> 00:24:07,145 So basically you're saying, because you freeze it. 420 00:24:07,145 --> 00:24:09,175 You're saying you freeze it and now you thaw it. 421 00:24:09,195 --> 00:24:09,665 Okay, all 422 00:24:09,690 --> 00:24:11,510 Yeah, or you could quiesce and unquiesce. 423 00:24:12,705 --> 00:24:13,095 Yeah. 424 00:24:13,095 --> 00:24:13,355 Yeah. 425 00:24:13,355 --> 00:24:13,665 Okay. 426 00:24:13,665 --> 00:24:15,955 I just, I don't usually use that term. 427 00:24:15,955 --> 00:24:16,795 So it really threw me. 428 00:24:17,350 --> 00:24:17,600 right. 429 00:24:17,620 --> 00:24:20,050 And then you do your backup off of that snapshot, right? 430 00:24:20,110 --> 00:24:24,100 So you now have a copy that's frozen that you can now do your 431 00:24:24,100 --> 00:24:25,410 backup and it's all good to go. 432 00:24:26,855 --> 00:24:27,385 Right. 433 00:24:27,535 --> 00:24:34,385 Um, I'd say the most common outside of storage arrays, the most common snapshot 434 00:24:34,395 --> 00:24:38,645 that is used in that way is VSS, right? 435 00:24:38,645 --> 00:24:40,915 The Windows Volume Shadow Services. 436 00:24:41,475 --> 00:24:46,135 And it, it's basically integrated into the operating system. 437 00:24:46,135 --> 00:24:48,025 It's integrated into the applications. 438 00:24:48,465 --> 00:24:49,525 It is. 439 00:24:51,485 --> 00:24:54,225 And backup apps can integrate with VSS. 440 00:24:54,225 --> 00:24:58,055 These are all done with APIs and a backup app can show up and say, Hey, 441 00:24:58,105 --> 00:25:01,015 I am here to do a backup of this box. 442 00:25:01,475 --> 00:25:04,745 Um, please, I'm going to really simplify it. 443 00:25:04,995 --> 00:25:08,095 Please take a snapshot of everything that needs to have a snapshot taken of 444 00:25:08,095 --> 00:25:12,795 it before I take a backup, then you take a backup and then they, um, and it can 445 00:25:12,795 --> 00:25:16,885 take a backup of that snapshot, even though the volume continues to change. 446 00:25:17,425 --> 00:25:20,495 It it's given this view into the volume that is static. 447 00:25:20,960 --> 00:25:25,630 And then it can back up that volume, uh, and get that perfectly application 448 00:25:25,630 --> 00:25:28,540 consistent version of the volume. 449 00:25:29,060 --> 00:25:32,340 Uh, even if the backup takes two hours, it doesn't matter. 450 00:25:32,380 --> 00:25:36,350 It has it, all of the blocks will be from the same exact point in time. 451 00:25:36,580 --> 00:25:40,430 And then when it's done, it can tell VSS to delete that snapshot 452 00:25:40,710 --> 00:25:41,820 or it can keep it around. 453 00:25:41,820 --> 00:25:42,490 It's up to you. 454 00:25:42,670 --> 00:25:44,390 It's just, it's a configuration thing. 455 00:25:44,660 --> 00:25:46,690 One thing I do want to mention. 456 00:25:47,680 --> 00:25:52,690 That, like we said, Snapshots just gives you that point in time copy, right? 457 00:25:52,690 --> 00:25:54,310 It's a read only, point in time copy. 458 00:25:54,600 --> 00:25:57,170 Now sometimes you will also hear, and I don't think we're covering 459 00:25:57,170 --> 00:26:00,240 it later, clones being used, right? 460 00:26:00,280 --> 00:26:04,400 Where I take a virtual copy of the volume and start using it, just going 461 00:26:04,400 --> 00:26:05,900 back to the previous discussion, Curtis. 462 00:26:06,280 --> 00:26:07,870 The difference is clones are writable. 463 00:26:08,820 --> 00:26:10,380 So you're making changes to it. 464 00:26:10,805 --> 00:26:15,055 Right, a snapshot is a read only copy that you preserve that point in time, 465 00:26:15,055 --> 00:26:19,315 nothing's gonna change it, and it's always there for you to go back, you 466 00:26:19,315 --> 00:26:21,085 can pull your file, your data out of it. 467 00:26:21,085 --> 00:26:26,355 Clones, on the other hand, give you a copy of the volume at a point in time, 468 00:26:26,675 --> 00:26:31,915 but it's so you can use it for some purpose, like for testing out restore. 469 00:26:32,180 --> 00:26:32,820 Capabilities. 470 00:26:32,820 --> 00:26:34,520 Can I verify my backups? 471 00:26:34,520 --> 00:26:35,500 Those sort of things. 472 00:26:35,500 --> 00:26:39,310 And also doing, like, database recovery against that copy, the clone 473 00:26:39,310 --> 00:26:40,730 copy, and other things like that. 474 00:26:40,730 --> 00:26:44,100 So, clones are different than snapshots, even though they both 475 00:26:44,100 --> 00:26:45,730 might start from a snapshot copy. 476 00:26:47,250 --> 00:26:47,720 Right. 477 00:26:47,830 --> 00:26:51,340 Um, yeah, I, I would probably just call it a read write snapshot, but 478 00:26:51,370 --> 00:26:53,150 maybe that's a contradiction in terms. 479 00:26:53,790 --> 00:26:54,330 Um, 480 00:26:54,930 --> 00:26:56,780 Yes, a snapshot is a point in time, Curtis. 481 00:26:58,310 --> 00:26:59,380 yeah, exactly. 482 00:27:00,470 --> 00:27:03,110 THere are three ways that snapshots are created. 483 00:27:03,350 --> 00:27:07,090 The most common way, I, would you, we say it's still the most common 484 00:27:07,090 --> 00:27:08,580 way, the copy on write method? 485 00:27:09,280 --> 00:27:12,530 Uh, no, I don't think, I don't think so anymore. 486 00:27:13,250 --> 00:27:13,670 Okay. 487 00:27:13,940 --> 00:27:17,720 Well, historically what used to be the most common method before 488 00:27:18,170 --> 00:27:19,900 one vendor ruined it for everybody 489 00:27:21,420 --> 00:27:22,050 Made something 490 00:27:22,050 --> 00:27:22,450 is. 491 00:27:24,460 --> 00:27:28,030 So is called the copy on write method. 492 00:27:28,030 --> 00:27:32,120 And the reason it's called the copy on write method is that we create this, 493 00:27:32,340 --> 00:27:40,230 this storage area that is going to hold the, um, the, the snapshot blocks. 494 00:27:41,570 --> 00:27:47,360 And when we go to update a block, because we're, it's a storage volume, right? 495 00:27:47,700 --> 00:27:51,495 So we're going to update a block and we say, Hey, Uh, there's 496 00:27:51,495 --> 00:27:52,975 a snapshot for this block. 497 00:27:53,265 --> 00:27:59,315 We're going to copy that block out to the snapshot area before you write. 498 00:27:59,345 --> 00:28:01,525 So that's why it's called copy on write. 499 00:28:02,055 --> 00:28:10,015 And that is very expensive because if you think about it, there are, let me count. 500 00:28:10,065 --> 00:28:12,005 There, there, there is a read. 501 00:28:12,805 --> 00:28:16,675 And a right for every right. 502 00:28:16,715 --> 00:28:17,675 There's, there's a says, 503 00:28:18,220 --> 00:28:21,770 and it's not just the one read and write that happens on the snapshot side. 504 00:28:21,770 --> 00:28:25,980 You also have a bunch of metadata and updating indirect blocks and 505 00:28:25,980 --> 00:28:27,030 a whole bunch of other things. 506 00:28:27,030 --> 00:28:31,780 So, yeah, doing a copy on write might lead to, say, 10 additional I. 507 00:28:31,780 --> 00:28:31,910 O. 508 00:28:31,910 --> 00:28:33,110 operations or 12. 509 00:28:34,365 --> 00:28:34,765 Yeah. 510 00:28:34,950 --> 00:28:36,110 for that one block. 511 00:28:37,695 --> 00:28:46,575 And so what happens is that over time that the number of blocks, remember 512 00:28:46,575 --> 00:28:51,645 when I initially, I mentioned that the bulk of the data is going to come from 513 00:28:51,655 --> 00:28:53,955 the primary volume, but over time. 514 00:28:55,145 --> 00:28:59,485 As a snapshot has been created, more and more blocks are going to be copied into 515 00:28:59,485 --> 00:29:05,805 that snapshot area, and which means that at some point, you know, a significant 516 00:29:05,805 --> 00:29:08,225 portion of my snapshot, if I'm. 517 00:29:08,790 --> 00:29:13,800 If I'm reading it, a significant portion is going to come from the snapshot area. 518 00:29:14,420 --> 00:29:22,690 And so it, there are multiple reasons that there's a performance hit. 519 00:29:22,750 --> 00:29:25,820 I think the biggest performance hit is. 520 00:29:26,295 --> 00:29:29,855 You know, you talk about all of the IOs that have to happen every time we update, 521 00:29:30,095 --> 00:29:32,285 uh, you know, we do a copy on write. 522 00:29:32,655 --> 00:29:39,665 The other is that as time goes on, the more and more data that I have to get 523 00:29:39,665 --> 00:29:43,345 from my snapshot area when I'm doing a read, the performance goes down. 524 00:29:43,815 --> 00:29:51,325 And this goes back to that vendor that they basically suggested that if we had 525 00:29:51,335 --> 00:29:56,050 90 days of user browsable snapshots, That their performance was going to be 526 00:29:56,050 --> 00:30:00,070 like half of what, uh, what it typically 527 00:30:00,230 --> 00:30:05,850 and I would say that is Probably based on older technology, Curtis. 528 00:30:06,360 --> 00:30:11,490 I think when you had traditional RAID arrays or RAID Groups 529 00:30:11,490 --> 00:30:12,480 that you were creating. 530 00:30:12,480 --> 00:30:16,580 I think there was more of a performance impact I think now since you end 531 00:30:16,590 --> 00:30:20,370 up aggregating a bunch of disks and then carving out volumes And so 532 00:30:20,370 --> 00:30:22,020 you can share in the performance. 533 00:30:22,410 --> 00:30:27,010 I think That's not as big of a concern anymore as it used to be. 534 00:30:27,010 --> 00:30:30,290 Like I know those systems you're talking about, they now support 535 00:30:30,690 --> 00:30:34,260 a thousand snapshots, right, for 536 00:30:34,280 --> 00:30:40,250 so you think that the main hit from a performance standpoint on a copy 537 00:30:40,250 --> 00:30:46,770 on write snapshot scenario in modern technology is mainly that IO hit the 538 00:30:46,770 --> 00:30:51,630 first time you go to do a write when every time, every time you update a write. 539 00:30:51,780 --> 00:30:57,060 And by the way, remember that It has to do that for every, that's sort of 540 00:30:57,060 --> 00:30:59,090 calculate every time it does it right. 541 00:30:59,090 --> 00:31:02,990 It has to calculate, is there a snapshot that is looking at this 542 00:31:02,990 --> 00:31:07,280 block as it exists at this point in time, and then you have to copy it, 543 00:31:07,300 --> 00:31:09,200 uh, for that, for that snapshot. 544 00:31:09,560 --> 00:31:12,180 Which, there are different mechanisms you could use. 545 00:31:12,180 --> 00:31:13,970 You could use bitmaps, you could use other things. 546 00:31:13,970 --> 00:31:18,660 So, it's not the end of the world, and I think that a lot of these storage vendors 547 00:31:18,660 --> 00:31:22,360 have optimized if they are continuing to use copy on write technologies. 548 00:31:22,760 --> 00:31:25,340 But I would say a good chunk of them have moved away from 549 00:31:25,340 --> 00:31:26,890 copy on write because of the I. 550 00:31:26,890 --> 00:31:27,030 O. 551 00:31:27,030 --> 00:31:28,720 penalties that we've talked 552 00:31:28,890 --> 00:31:30,750 right, right. 553 00:31:31,930 --> 00:31:36,910 So the, the, there was this vendor that came out, a little vendor 554 00:31:36,910 --> 00:31:40,620 called, at one point it used to be called Network Appliance. 555 00:31:41,640 --> 00:31:45,090 And it had a little screw and bolt as its logo. 556 00:31:46,320 --> 00:31:46,860 right. 557 00:31:46,940 --> 00:31:52,840 Um, the, the, at one point they said, we're just going to change our 558 00:31:52,840 --> 00:31:54,360 name to NetApp because that's what 559 00:31:54,785 --> 00:31:55,945 That's what everyone calls them. 560 00:31:57,205 --> 00:32:03,985 I remember actually working at a startup and got my hands on my 561 00:32:03,985 --> 00:32:07,365 first NetApp appliance and was like, yeah, wow, these are kind of cool. 562 00:32:07,365 --> 00:32:09,435 And this was before I started working there. 563 00:32:10,195 --> 00:32:12,755 And I was like, wow, this is really amazing and simple 564 00:32:12,755 --> 00:32:15,265 for what it does and easy to 565 00:32:15,420 --> 00:32:16,770 Yeah, exactly. 566 00:32:17,470 --> 00:32:29,790 And they used a completely different way to, um, to do snapshots that at the time 567 00:32:29,790 --> 00:32:34,680 was revolutionary, which I think has now been adopted by a lot of storage vendors. 568 00:32:34,680 --> 00:32:37,900 And that is, we call it redirect on write. 569 00:32:37,930 --> 00:32:40,260 Do you want to describe how that 570 00:32:40,260 --> 00:32:40,530 works? 571 00:32:41,000 --> 00:32:43,790 so what read so before you get there? 572 00:32:43,790 --> 00:32:47,470 I think we need to talk about their right anywhere file layout 573 00:32:47,470 --> 00:32:49,220 which allows then the snapshot. 574 00:32:49,420 --> 00:32:54,330 Yep waffle Which a lot of other vendors I think do something similar as well these 575 00:32:54,330 --> 00:33:00,860 days but what it is is going back to the copy on write example If you are writing 576 00:33:00,880 --> 00:33:05,490 a block of data in copy on write sort of file system, previous file systems, 577 00:33:05,740 --> 00:33:07,250 you would always write to the same spot. 578 00:33:07,530 --> 00:33:10,040 And because you're always writing to the same spot, that's why you have to first 579 00:33:10,040 --> 00:33:12,240 copy out the data and then update it. 580 00:33:12,930 --> 00:33:17,750 With a write anywhere file layout, what you end up doing is, it doesn't 581 00:33:17,750 --> 00:33:22,710 matter which actual block you end up writing to, you basically construct the 582 00:33:22,710 --> 00:33:25,280 metadata tree to reference that block. 583 00:33:26,200 --> 00:33:31,220 Even though I might be updating block 100, block 100 might actually 584 00:33:31,230 --> 00:33:33,320 physically be at, like, block 1000. 585 00:33:33,950 --> 00:33:39,020 And because I have all my metadata that tells me exactly where that data 586 00:33:39,020 --> 00:33:42,460 exists, I just need to update the metadata to say, okay, if someone 587 00:33:42,460 --> 00:33:45,850 tries to access block 100, it's actually physically on block 1000. 588 00:33:46,420 --> 00:33:50,170 And so you can end up writing to any location in the file system 589 00:33:50,730 --> 00:33:53,530 and not having to worry about always hitting that same location. 590 00:33:54,210 --> 00:33:56,280 And so that's kind of waffle in a nutshell. 591 00:33:57,975 --> 00:34:02,415 So basically what you then is you have this metadata system that has a pointer 592 00:34:02,445 --> 00:34:05,975 to every block and it doesn't really matter where those blocks happen to be. 593 00:34:06,350 --> 00:34:09,260 And you can have thousands of these snapshots, right? 594 00:34:09,260 --> 00:34:10,310 These pointers at the top. 595 00:34:10,360 --> 00:34:19,350 right, so then when we go to do an update and we have a snapshot, it just 596 00:34:19,350 --> 00:34:23,210 means that what we're going to do is we're going to change the pointer. 597 00:34:23,670 --> 00:34:27,540 We're going to say, okay, this, there's this block that's sitting here. 598 00:34:28,085 --> 00:34:31,485 And we know we, we, we're not supposed to update that block because we 599 00:34:31,485 --> 00:34:35,085 have a snapshot that's requiring on that, that's requiring that block. 600 00:34:35,725 --> 00:34:38,575 And then we're going to just redirect. 601 00:34:38,585 --> 00:34:41,675 We're going to, we're going to write a new block for the 602 00:34:41,675 --> 00:34:43,565 new version of that old block. 603 00:34:43,835 --> 00:34:46,355 And then we're going to change the pointer, right? 604 00:34:46,355 --> 00:34:47,965 We're going to redirect the pointer. 605 00:34:48,305 --> 00:34:51,925 To this new location of that block. 606 00:34:52,255 --> 00:34:56,145 Meanwhile, the old block is still sitting there and we've got a 607 00:34:56,145 --> 00:34:58,935 snapshot that's just pointing to it. 608 00:34:59,325 --> 00:34:59,735 Right. 609 00:34:59,855 --> 00:35:03,495 Um, and so you've got this infinite number of snapshots that are pointing 610 00:35:03,495 --> 00:35:07,625 to an infinite number of, well, it's not an infinite number of snapshots, but 611 00:35:07,965 --> 00:35:13,175 you have a very high number of snapshots that are, that, that, and they're all. 612 00:35:13,355 --> 00:35:18,265 Just a whole bunch of metadata pointing to a whole bunch of blocks that are 613 00:35:18,265 --> 00:35:20,925 just sitting all around the volume. 614 00:35:21,685 --> 00:35:26,205 And the one thing, so this all sounds amazing, right? 615 00:35:26,255 --> 00:35:29,635 Because you're like, Oh, writes are, writes are super fast. 616 00:35:29,675 --> 00:35:30,635 I don't have to worry about it. 617 00:35:30,825 --> 00:35:32,645 In fact, Curtis, just one correction. 618 00:35:32,895 --> 00:35:36,545 When you're going to actually write a new block, you never actually have to 619 00:35:36,545 --> 00:35:40,135 look up the old block because you're always writing to a new location. 620 00:35:40,135 --> 00:35:43,915 So you don't care if that old block is occupied by a snapshot or not, right? 621 00:35:44,215 --> 00:35:47,945 Because this is the downside though, of snapshots, especially with the 622 00:35:47,945 --> 00:35:50,035 write anywhere file layout is. 623 00:35:50,380 --> 00:35:55,140 You now need a process to go through and say when a snapshot gets deleted, 624 00:35:55,550 --> 00:35:59,210 what blocks are no longer being used actively because they don't 625 00:35:59,210 --> 00:36:02,360 belong to snapshots, they're not currently as part of the volume. 626 00:36:02,740 --> 00:36:07,860 So you have a garbage reclamation process or different vendors call it 627 00:36:07,860 --> 00:36:12,010 different things, but some process to go through and reclaim all of those 628 00:36:12,260 --> 00:36:15,060 free blocks so they can be reused. 629 00:36:16,535 --> 00:36:18,925 I think you said the same thing I said in different words, but 630 00:36:21,455 --> 00:36:22,735 yeah, I see what you're saying. 631 00:36:22,735 --> 00:36:27,385 I guess I was saying that, that it's making a decision on what to do. 632 00:36:27,510 --> 00:36:30,840 Based on whether or not, I think if I could just change my part 633 00:36:30,840 --> 00:36:36,950 of my answer, instead of, is this block being used by anything else? 634 00:36:36,950 --> 00:36:37,490 I guess is, 635 00:36:37,515 --> 00:36:38,035 Yeah, more 636 00:36:38,420 --> 00:36:39,840 I guess that's the question I'm asking. 637 00:36:39,850 --> 00:36:40,180 Yeah. 638 00:36:40,600 --> 00:36:44,020 Uh, and actually, I guess what you're saying is it actually doesn't even make 639 00:36:44,020 --> 00:36:45,570 that decision at that point in time. 640 00:36:45,850 --> 00:36:48,980 If it's going to modify a new block, if it's going to modify a 641 00:36:48,980 --> 00:36:51,140 block, it just writes a new block. 642 00:36:51,750 --> 00:36:55,530 And then what happens to that old block is a completely separate process. 643 00:36:55,860 --> 00:36:59,100 If there's, if there is a snapshot that's pointing to that, that 644 00:36:59,100 --> 00:37:01,620 old block, then it will stay. 645 00:37:02,060 --> 00:37:05,590 If there are no snapshots that are pointing to that block, then at some 646 00:37:05,590 --> 00:37:08,350 point the garbage collection process will come and make it go away. 647 00:37:09,090 --> 00:37:10,420 Is that a better, is that a better 648 00:37:10,725 --> 00:37:12,155 Yes, that is a better description. 649 00:37:12,495 --> 00:37:13,065 Now this is 650 00:37:13,130 --> 00:37:15,680 don't want to argue with a former NetApp employee 651 00:37:15,710 --> 00:37:16,530 about how NetApps 652 00:37:16,725 --> 00:37:20,015 now I should say this is based on NetApp's technology, different 653 00:37:20,015 --> 00:37:23,525 vendors may do different things, but for the most part, most of the 654 00:37:23,525 --> 00:37:25,415 vendors do something similar ish. 655 00:37:25,710 --> 00:37:28,750 Specifically, it's based on how NetApp worked when you worked 656 00:37:28,750 --> 00:37:30,170 there, which was a while ago. 657 00:37:30,520 --> 00:37:32,140 And things may be different now, 658 00:37:32,495 --> 00:37:33,365 that is also true. 659 00:37:33,410 --> 00:37:33,690 not. 660 00:37:35,080 --> 00:37:37,740 I mean, Waffle is like at the core of their... 661 00:37:38,215 --> 00:37:39,995 You know, the core of their technology. 662 00:37:41,105 --> 00:37:47,835 And so, and that's why with redirect on right, that's why you could essentially 663 00:37:47,835 --> 00:37:53,215 have an infinite number of snapshots with, with zero performance penalty, 664 00:37:53,215 --> 00:37:57,425 you have zero performance penalty of having basically the performance 665 00:37:57,425 --> 00:37:59,065 penalty of doing a right update. 666 00:38:00,350 --> 00:38:03,250 Is the same whether you have a snapshot or you don't have a snapshot. 667 00:38:03,680 --> 00:38:10,060 The, the only penalty, if you want to call it that, is that, um, one, one 668 00:38:10,070 --> 00:38:16,060 big difference between this way and the other way is there is no snapshot area. 669 00:38:16,865 --> 00:38:17,345 Right. 670 00:38:17,495 --> 00:38:20,945 So in the other method, the snapshot area could fill up if 671 00:38:20,945 --> 00:38:22,655 you held snapshots for too long. 672 00:38:23,105 --> 00:38:27,725 In this configuration, there is no snapshot area. 673 00:38:27,725 --> 00:38:33,365 The snapshot area is the volume, and if you have too many updates and you 674 00:38:33,365 --> 00:38:37,595 keep too many snapshots, you would fill up the volume with snapshots. 675 00:38:37,805 --> 00:38:39,005 So you've gotta get rid of the older 676 00:38:39,035 --> 00:38:43,615 Well, and this is where That would apply if you're talking NetApp terminology 677 00:38:43,615 --> 00:38:47,565 traditional volumes, but most of it has moved over to virtual volumes, where once 678 00:38:47,565 --> 00:38:52,135 again, you have an aggregate, a shared pool of common data, and for each of the 679 00:38:52,135 --> 00:38:57,335 volumes, typically you also set a limit on how much space a snapshot can occupy. 680 00:38:58,165 --> 00:39:02,125 So you could say, I am allowing 20 percent for snapshots of my overall 681 00:39:02,125 --> 00:39:05,385 volume capacity, in which case it'll start 682 00:39:05,515 --> 00:39:07,705 And what happens when you hit that wall? 683 00:39:07,780 --> 00:39:11,000 I, I don't know what the current behavior is. 684 00:39:11,010 --> 00:39:11,520 Previously. 685 00:39:11,520 --> 00:39:16,110 I believe it would like let you like start automatically pruning 686 00:39:16,120 --> 00:39:17,790 snapshots and trying to free up space. 687 00:39:19,145 --> 00:39:20,005 Right, right. 688 00:39:20,695 --> 00:39:23,745 Because it, obviously, it's not going to, it's not going to prune, uh, 689 00:39:23,765 --> 00:39:25,135 production, you know, current data. 690 00:39:25,565 --> 00:39:31,385 So, yeah, I, it could, but again, we're, we're talking specifically NetApp, 691 00:39:31,405 --> 00:39:33,935 but something has to happen, right? 692 00:39:33,965 --> 00:39:35,365 If you're using this method. 693 00:39:35,605 --> 00:39:36,705 Something has to happen. 694 00:39:36,705 --> 00:39:39,785 Either we have to stop creating new snapshots, right? 695 00:39:39,815 --> 00:39:42,065 Or stop updating the snapshots that we have. 696 00:39:42,535 --> 00:39:49,075 And, uh, we need to delete older snapshots or we need to maybe delete, you know, 697 00:39:49,075 --> 00:39:50,755 certain ones in the middle, right? 698 00:39:50,925 --> 00:39:53,795 Basically you've got to do some kind of pruning or else you're going to 699 00:39:53,990 --> 00:39:56,868 Yeah, the other challenge is also figuring out what snapshot to delete 700 00:39:56,868 --> 00:39:58,278 because blocks are being shared, right? 701 00:39:58,278 --> 00:40:01,818 You might be like, hey, this snapshot is huge and you go delete it, but 702 00:40:01,818 --> 00:40:04,808 because those blocks are being shared by other snapshots, you're not actually 703 00:40:04,808 --> 00:40:06,838 going to free any space, right? 704 00:40:06,888 --> 00:40:10,258 So you need to be able to figure out like which snapshot actually 705 00:40:10,258 --> 00:40:14,748 contains unique blocks that if I delete it will actually save me space. 706 00:40:16,283 --> 00:40:17,043 complicated. 707 00:40:18,483 --> 00:40:19,593 Storage management. 708 00:40:19,998 --> 00:40:20,008 I 709 00:40:21,253 --> 00:40:23,243 I don't miss production storage management. 710 00:40:23,583 --> 00:40:26,253 Any other final thoughts on redirect on, right? 711 00:40:27,358 --> 00:40:28,298 think that covers it 712 00:40:28,513 --> 00:40:33,523 I mean, My personal, if you're going to do snapshots on a storage array, I 713 00:40:33,533 --> 00:40:35,283 think redirect on write is the way to go. 714 00:40:35,283 --> 00:40:38,293 It sounds like what you're saying, they've made copy on write better, 715 00:40:38,723 --> 00:40:41,883 but I still think redirect on write is just significantly better. 716 00:40:42,023 --> 00:40:48,363 So, um, but it might be more complicated than if you're coding it. 717 00:40:48,543 --> 00:40:48,973 Right. 718 00:40:49,753 --> 00:40:54,083 So the next one is what I'm going to call the dumbest of all snapshot methods. 719 00:40:56,993 --> 00:40:58,413 That's not what I have in the book. 720 00:40:58,413 --> 00:41:00,193 I gave it a much nicer name in the book. 721 00:41:00,868 --> 00:41:04,038 And guess who does this method? 722 00:41:04,348 --> 00:41:07,258 The leading hypervisor company in the world. 723 00:41:07,778 --> 00:41:08,608 Yes. 724 00:41:09,138 --> 00:41:10,818 I think that's a fair statement, right? 725 00:41:10,888 --> 00:41:11,848 company in the world. 726 00:41:11,968 --> 00:41:13,108 I think, I think it still is. 727 00:41:13,108 --> 00:41:13,388 Yeah. 728 00:41:14,138 --> 00:41:16,198 And that would be VMware. 729 00:41:16,558 --> 00:41:21,483 So the way VMware does snapshots is just literally the Dumbest 730 00:41:21,483 --> 00:41:25,413 implementation of snapshots that I've ever seen and I don't know how they 731 00:41:25,413 --> 00:41:27,623 haven't addressed it, but here it is. 732 00:41:28,853 --> 00:41:34,103 When you create a snapshot in VMware, it literally holds all the rights. 733 00:41:34,133 --> 00:41:40,103 Now, by the way, if I'm wrong, by the way, you know, Broadcom, don't sue me. 734 00:41:40,363 --> 00:41:45,008 This is based, this is based on my understanding of VMware snapshots. 735 00:41:45,548 --> 00:41:48,488 Uh, you know, I've, I've checked every once in a while and they, 736 00:41:48,908 --> 00:41:50,638 no one seems bothered by this. 737 00:41:52,048 --> 00:41:55,578 Uh, but if this has changed, any of you that are, you know, if anybody 738 00:41:55,578 --> 00:42:00,468 works for Broadcom slash VMware, then, you know, feel free to update 739 00:42:00,478 --> 00:42:02,108 me and I will update this episode. 740 00:42:02,818 --> 00:42:04,278 And I'll just delete this section. 741 00:42:04,288 --> 00:42:06,018 But here's the way it works. 742 00:42:06,628 --> 00:42:10,968 When you create a single snapshot on a VMware volume, it halts. 743 00:42:11,378 --> 00:42:15,658 All rights on the, on the current volume. 744 00:42:15,948 --> 00:42:20,798 And then it keeps all rights in a snapshot area. 745 00:42:21,238 --> 00:42:25,738 And then when you delete that snapshot, it replays all those 746 00:42:25,738 --> 00:42:27,648 rights against the production volume. 747 00:42:28,288 --> 00:42:31,548 And this is why when you make a snapshot. 748 00:42:31,718 --> 00:42:35,508 And then you, if you hold that snapshot for a long time and then 749 00:42:35,508 --> 00:42:39,298 you delete that snapshot, this is why it has a big performance hit 750 00:42:39,938 --> 00:42:41,528 against the production volume. 751 00:42:41,738 --> 00:42:42,368 But no one 752 00:42:42,468 --> 00:42:42,978 this is 753 00:42:42,998 --> 00:42:44,318 snapshot of a VM 754 00:42:45,628 --> 00:42:49,848 yeah, this is why you do not do this. 755 00:42:49,848 --> 00:42:56,368 You don't use snapshots on VMware level snapshots the way you do any other 756 00:42:56,368 --> 00:43:01,978 snapshots, because, and by the way, I used VMware for years before knowing this. 757 00:43:01,988 --> 00:43:03,458 That's why I want to make sure I mentioned it. 758 00:43:04,333 --> 00:43:08,253 And, and that is that if you create a snapshot and then hold it for a 759 00:43:08,263 --> 00:43:12,793 long period of time, you're going to get hit with a massive IO hit 760 00:43:12,803 --> 00:43:14,263 when you delete that snapshot. 761 00:43:14,523 --> 00:43:22,223 So if you're using VMware, VMware level snapshots, then you use them the way we 762 00:43:22,223 --> 00:43:26,923 talked about earlier, where you create a snapshot, you make a, you make a backup. 763 00:43:27,528 --> 00:43:29,188 And then you delete the snapshot. 764 00:43:29,498 --> 00:43:33,408 Maybe you take a VMware level snapshot, and then you take a storage level 765 00:43:33,408 --> 00:43:38,418 snapshot of that snapshot, and then you delete the, the VMware level snapshot. 766 00:43:38,658 --> 00:43:44,438 You should, if this is the way your snapshot system works, you 767 00:43:44,518 --> 00:43:49,228 cannot leave the snapshots around for any significant period of time. 768 00:43:49,748 --> 00:43:51,068 I was going to chime in. 769 00:43:51,068 --> 00:43:52,508 Thank you for covering that. 770 00:43:52,508 --> 00:43:57,988 The, this specifically is VM where software snapshots, 771 00:43:57,988 --> 00:43:59,068 if you wanna call it that. 772 00:43:59,523 --> 00:44:01,443 Right, that are only done at the VMware level. 773 00:44:01,453 --> 00:44:04,723 Now, there are integrations that various storage vendors offer 774 00:44:05,003 --> 00:44:07,103 by plugging into the VMware API. 775 00:44:07,103 --> 00:44:10,173 So whenever you trigger a VMware snapshot, it actually triggers 776 00:44:10,173 --> 00:44:11,833 a storage level snapshot. 777 00:44:11,833 --> 00:44:15,953 So avoiding some of these issues, but not everyone is aware of it. 778 00:44:15,953 --> 00:44:19,653 Not everyone is using a third party storage array that integrates with VMware. 779 00:44:20,023 --> 00:44:20,373 So. 780 00:44:21,623 --> 00:44:22,013 Just 781 00:44:22,338 --> 00:44:22,738 Yeah. 782 00:44:22,748 --> 00:44:23,528 So, right. 783 00:44:23,528 --> 00:44:25,048 Thanks for, thanks for clarifying that. 784 00:44:25,058 --> 00:44:29,028 This is specifically VMware level snapshots that are done by VMware. 785 00:44:29,183 --> 00:44:31,573 And without any third party storage. 786 00:44:33,123 --> 00:44:33,433 Yeah. 787 00:44:33,503 --> 00:44:37,593 And I don't know why VMware did this, but it's bonkers. 788 00:44:38,843 --> 00:44:41,823 It's just literally one of the weirdest, codest thing, weirdest 789 00:44:41,883 --> 00:44:43,943 coded things I've ever heard. 790 00:44:44,033 --> 00:44:45,313 Why would you do it that way? 791 00:44:46,113 --> 00:44:49,133 Somewhere in a meeting, this is how they decided to 792 00:44:49,193 --> 00:44:49,213 it. 793 00:44:49,213 --> 00:44:50,293 was probably easier 794 00:44:50,393 --> 00:44:53,883 and yeah, yeah, maybe it was easier, 795 00:44:54,303 --> 00:44:55,293 and they didn't talk to the 796 00:44:55,303 --> 00:44:56,533 I wonder about that. 797 00:44:57,743 --> 00:45:01,403 They didn't, they exactly, they did not talk to the backup folks. 798 00:45:02,543 --> 00:45:07,593 Well, uh, I think we have, uh, summarized the world of snapshots. 799 00:45:07,883 --> 00:45:08,163 That? 800 00:45:08,173 --> 00:45:10,513 No, I think we did a good job with that. 801 00:45:11,923 --> 00:45:15,883 So, copy on write, redirect on write, dumbest method ever. 802 00:45:16,713 --> 00:45:19,073 Those are the three types. 803 00:45:19,903 --> 00:45:23,483 I've got it officially in the book, uh, I've got this labeled 804 00:45:23,483 --> 00:45:25,233 as the hold all writes method. 805 00:45:25,883 --> 00:45:28,623 Uh, I should really just change that to the dumbest method ever. 806 00:45:29,353 --> 00:45:31,193 But, um, yeah. 807 00:45:31,213 --> 00:45:36,433 So, you know, snapshots are a great tool. 808 00:45:37,128 --> 00:45:38,998 In the backup and recovery arsenal. 809 00:45:39,248 --> 00:45:45,738 They are the great sort of basis upon which we're going to talk about one 810 00:45:45,738 --> 00:45:47,798 of my favorite ways to do backup. 811 00:45:48,628 --> 00:45:50,928 And we're going to talk about that in another episode. 812 00:45:51,098 --> 00:45:53,708 Hint, it's called near CDP, not CDP. 813 00:45:53,738 --> 00:45:54,568 It's called near CDP. 814 00:45:55,738 --> 00:46:00,558 And, uh, it's just, just the number one thing you have to understand 815 00:46:00,558 --> 00:46:05,048 about snapshots is that unless you have copied this snapshot to 816 00:46:05,048 --> 00:46:08,108 another location via some mechanism. 817 00:46:08,433 --> 00:46:09,373 Which could be backup. 818 00:46:09,373 --> 00:46:10,893 It could be replication of the volume. 819 00:46:10,893 --> 00:46:12,393 It could be a number of things. 820 00:46:13,003 --> 00:46:14,613 You do not have a backup. 821 00:46:14,833 --> 00:46:17,763 You have a picture of your volume. 822 00:46:18,183 --> 00:46:22,753 And that picture of your volume is as worthless as a picture of your 823 00:46:22,753 --> 00:46:24,593 house after your house burns down. 824 00:46:25,823 --> 00:46:30,133 It'll just be a nice memory and, uh, and a really bad day. 825 00:46:30,133 --> 00:46:32,993 So it's just, that's the really, the most important thing to 826 00:46:32,993 --> 00:46:34,163 understand about snapshots. 827 00:46:34,453 --> 00:46:37,843 And now if this is your first time, snapshots have been explained to you. 828 00:46:38,053 --> 00:46:41,293 Now you understand why I don't like it that they call. 829 00:46:41,438 --> 00:46:45,848 What AWS does snapshots because that very much does not meet 830 00:46:45,848 --> 00:46:47,268 the definition that we just had. 831 00:46:47,268 --> 00:46:50,668 And I'm glad I brought this up because it's important to what we're talking 832 00:46:50,668 --> 00:46:52,278 about is storage level snapshots. 833 00:46:52,428 --> 00:46:52,788 Darn it. 834 00:46:52,838 --> 00:46:53,148 I don't know 835 00:46:53,233 --> 00:46:54,563 You can't call it that, yeah, because 836 00:46:54,668 --> 00:46:55,038 this. 837 00:46:55,578 --> 00:46:58,628 Yeah, these are traditional snapshots. 838 00:46:58,738 --> 00:47:02,248 There are other things out there that people call snapshots 839 00:47:02,248 --> 00:47:03,378 that don't work like this. 840 00:47:03,513 --> 00:47:05,833 AWS snapshots don't work like this. 841 00:47:06,323 --> 00:47:09,223 Uh, they are an actual image copy. 842 00:47:09,223 --> 00:47:13,733 It's actually, they actually, when you make an AWS snapshot, it actually copies 843 00:47:13,733 --> 00:47:18,863 that, that point in time out to another area of storage, which happens to be S3, 844 00:47:19,033 --> 00:47:19,233 you? 845 00:47:19,283 --> 00:47:23,083 And I think specifically you're talking about an AWS EBS snapshot 846 00:47:24,253 --> 00:47:24,723 thank you. 847 00:47:24,723 --> 00:47:28,368 I am talking about an AWS EBS snapshot. 848 00:47:29,208 --> 00:47:33,798 Um, my former employer, Druva, they call what they do snapshots. 849 00:47:34,758 --> 00:47:36,688 They call their backups snapshots. 850 00:47:36,988 --> 00:47:40,328 I never liked that, but you know, nobody asked me. 851 00:47:40,758 --> 00:47:45,788 Uh, so, but what we're talking about here is traditional snapshots. 852 00:47:46,358 --> 00:47:50,338 And, um, a lot of other people will call what they do a snapshot. 853 00:47:50,508 --> 00:47:54,058 Um, the problem is like a lot of terms in the, in the backup world. 854 00:47:54,728 --> 00:48:02,888 It's a term like so many of our terms are, um, their words that are used 855 00:48:03,018 --> 00:48:06,418 just, they're just English words that are used in so many different contexts. 856 00:48:06,543 --> 00:48:11,483 when, like when we had the CDP episode, we couldn't figure out what to call 857 00:48:11,483 --> 00:48:14,063 those point in time, because a lot of the CDP vendors called them snapshots. 858 00:48:16,028 --> 00:48:17,098 Yeah, exactly. 859 00:48:17,178 --> 00:48:18,568 Yeah, exactly. 860 00:48:19,348 --> 00:48:19,738 All right. 861 00:48:19,758 --> 00:48:23,788 Well, uh, I guess the only thing left for me to say is that's a wrap