1 00:00:00,000 --> 00:00:03,993 You found the backup wrap up your go-to podcast for all things 2 00:00:03,993 --> 00:00:06,603 backup recovery and cyber recovery. 3 00:00:06,963 --> 00:00:12,243 In this episode, we tackle a confusing phrase, snapshot backup. 4 00:00:12,723 --> 00:00:14,343 I know what you're thinking, Curtis. 5 00:00:14,343 --> 00:00:15,723 Isn't that an oxymoron? 6 00:00:15,723 --> 00:00:18,513 Well, that depends on what you mean by snapshot. 7 00:00:19,173 --> 00:00:21,603 We're gonna break down the difference between what I'm calling 8 00:00:21,603 --> 00:00:25,983 traditional storage snapshots and what vendors like AWS call snapshots. 9 00:00:26,313 --> 00:00:28,083 Which are actually more like backups. 10 00:00:28,413 --> 00:00:29,953 We'll talk about copy on write? 11 00:00:29,953 --> 00:00:31,183 Redirect on write. 12 00:00:31,293 --> 00:00:33,363 VMware snapshots that are just weird. 13 00:00:33,783 --> 00:00:37,953 How to turn your snapshots into actual backups that follow the 3, 2, 1 rule. 14 00:00:38,583 --> 00:00:42,573 Uh, pretty much everything that you need to know about making. 15 00:00:42,793 --> 00:00:44,203 A backup snapshot. 16 00:00:44,833 --> 00:00:48,793 If you don't know who I am, I'm w Curtis Preston, AKA, Mr. 17 00:00:48,793 --> 00:00:54,013 Backup, and I've been dealing with this topic for over 30 years, ever since. 18 00:00:54,013 --> 00:00:58,033 I had to tell my boss that there were no backups of the production 19 00:00:58,033 --> 00:00:59,563 database that we had just lost. 20 00:00:59,953 --> 00:01:01,423 I don't want that to happen to me. 21 00:01:01,573 --> 00:01:03,133 I don't want that to happen to you. 22 00:01:03,313 --> 00:01:04,633 That's why I do this. 23 00:01:05,023 --> 00:01:09,463 And so, enjoy this episode of the backup wrap up. 24 00:01:23,523 --> 00:01:24,453 Welcome to the show. 25 00:01:25,593 --> 00:01:31,653 Hi, I am w Curtis Preston, AKA, Mr. Backup, and I have with me my QOS advisor 26 00:01:31,963 --> 00:01:33,873 Prasanna Malaiyandi how's it going? 27 00:01:33,873 --> 00:01:34,393 Prasanna? 28 00:01:34,928 --> 00:01:40,263 I am good Curtis, and I'm glad that we don't have a random five second lag 29 00:01:40,263 --> 00:01:43,683 every time we talk, so That's good. 30 00:01:45,003 --> 00:01:45,453 Yeah. 31 00:01:45,513 --> 00:01:49,413 So, uh, we figured out, we, we were, we started to try to record this 32 00:01:49,413 --> 00:01:51,393 and there was this massive lag. 33 00:01:51,873 --> 00:01:56,553 Um, and, um, I realized that basically like I counted 'em real quick. 34 00:01:56,553 --> 00:01:59,673 There are four people currently watching Netflix. 35 00:01:59,703 --> 00:02:02,823 Uh, I was able to stop one of them, but I don't control the other ones. 36 00:02:03,243 --> 00:02:06,753 And so you came up with the idea of, you know, what about QOS? 37 00:02:06,753 --> 00:02:09,333 And so I, I have a firewall. 38 00:02:10,248 --> 00:02:16,608 Um, and, uh, so I, I enabled SmartQ, I put this website on it and I 39 00:02:16,608 --> 00:02:18,738 said, give me, gimme all the stuff. 40 00:02:18,738 --> 00:02:20,118 Gimme everything I can. 41 00:02:20,658 --> 00:02:23,328 And uh, so now it's, you know, beautiful. 42 00:02:23,328 --> 00:02:27,048 So I guess that's a plug for firewall as well, not a sponsor. 43 00:02:27,453 --> 00:02:27,783 Yep. 44 00:02:28,053 --> 00:02:34,148 Or for any router software that actually supports proper QOS, not like most 45 00:02:34,218 --> 00:02:34,608 Yeah. 46 00:02:34,668 --> 00:02:36,268 consumer level ones, which don't do a great job. 47 00:02:37,443 --> 00:02:41,013 Yeah, so this is, I I've been very happy with the, with 48 00:02:41,013 --> 00:02:42,423 the firewall since I got it. 49 00:02:42,483 --> 00:02:47,043 Uh, every once in a while I use something, you know, fancy on it when this is nice. 50 00:02:47,043 --> 00:02:54,123 So, um, so, um, we haven't recorded in a while. 51 00:02:54,153 --> 00:02:55,833 You, you went away. 52 00:02:56,223 --> 00:02:57,183 I went away. 53 00:02:57,183 --> 00:03:00,753 I went on vacation for the longest vacation I've been 54 00:03:00,753 --> 00:03:02,493 on in seven years I think. 55 00:03:02,493 --> 00:03:06,633 So I was gone out of the country for three weeks, which was nice. 56 00:03:06,633 --> 00:03:10,503 I was off in India for a wedding, visiting family. 57 00:03:10,533 --> 00:03:11,673 Lots of travel. 58 00:03:12,003 --> 00:03:13,293 So much good food. 59 00:03:13,293 --> 00:03:15,933 Curtis, um, and 60 00:03:15,968 --> 00:03:16,258 Yeah. 61 00:03:16,653 --> 00:03:19,113 I might be addicted to caffeine. 62 00:03:23,808 --> 00:03:24,258 So 63 00:03:24,258 --> 00:03:28,098 In India as you've had, right, uh, they serve in the south, they 64 00:03:28,098 --> 00:03:32,178 serve filter coffee, which is sort of a small cup of coffee, which 65 00:03:32,178 --> 00:03:34,458 is very, very concentrated though. 66 00:03:35,358 --> 00:03:40,008 And, uh, I was drinking three or four a day. 67 00:03:40,278 --> 00:03:43,998 Now this is normally someone who drinks maybe one cup of 68 00:03:43,998 --> 00:03:45,378 coffee a day, other than when 69 00:03:45,378 --> 00:03:45,858 Yeah. 70 00:03:45,948 --> 00:03:47,088 office where it's maybe two. 71 00:03:47,628 --> 00:03:47,808 So. 72 00:03:47,958 --> 00:03:48,288 Yeah. 73 00:03:48,888 --> 00:03:49,278 Yeah. 74 00:03:49,828 --> 00:03:52,138 Well anyway, well welcome back to the same time zone. 75 00:03:52,168 --> 00:03:52,918 Very excited. 76 00:03:53,083 --> 00:03:53,563 Thank you. 77 00:03:53,600 --> 00:03:56,420 but It is hard with the 12 and a half hour time zone difference. 78 00:03:56,420 --> 00:03:57,440 You know, it was like 79 00:03:57,500 --> 00:03:57,860 Yeah, 80 00:03:58,160 --> 00:04:01,250 we get a brief window to chit chat, or in the evenings we get a brief window to chit 81 00:04:01,250 --> 00:04:02,450 yeah, yeah. 82 00:04:02,450 --> 00:04:06,740 I basically wake up and call you and then before I go to bed, I call you. 83 00:04:06,740 --> 00:04:07,070 Right. 84 00:04:07,460 --> 00:04:11,330 Um, and, um, uh, yeah, so a lot of people don't know that it's, 85 00:04:11,390 --> 00:04:14,870 that it's 12 and a half hours time difference, which is just odd. 86 00:04:15,350 --> 00:04:17,205 yeah, I don't know why they do the half hour. 87 00:04:19,730 --> 00:04:20,510 Yeah, just funny. 88 00:04:21,260 --> 00:04:23,750 Maybe they, they saved the half hour to drink the filter coffee. 89 00:04:28,670 --> 00:04:32,870 So, uh, all right, well, today we're gonna talk about, uh, you know, and I 90 00:04:32,870 --> 00:04:36,170 thought, you know, you thought maybe we'd covered this already, or at least 91 00:04:36,170 --> 00:04:38,090 recently, and, and we really haven't. 92 00:04:38,090 --> 00:04:44,690 So this is a, a word that comes up a lot in, uh, in the show. 93 00:04:45,020 --> 00:04:47,930 And that is this word snapshot, which. 94 00:04:48,510 --> 00:04:52,200 If, if we, if we go outside of it, it, it, it means that, you 95 00:04:52,200 --> 00:04:53,640 know, it's, it's a picture, right? 96 00:04:53,640 --> 00:04:56,385 That that's what the word snapshot and also. 97 00:04:57,005 --> 00:05:01,415 There's another word that's gonna be used in this episode, which is image, which 98 00:05:01,415 --> 00:05:03,995 is really just another word for picture. 99 00:05:04,685 --> 00:05:09,245 But in our world, an image is very different than a snapshot. 100 00:05:09,665 --> 00:05:14,135 And as you've often heard me say, words mean things. 101 00:05:14,645 --> 00:05:21,305 And um, I want to make sure that this is sort of a, I'm gonna 102 00:05:21,305 --> 00:05:23,705 call it snapshots basic episode. 103 00:05:24,635 --> 00:05:28,925 So if you're, if you're just wondering what you know, what are snapshots, what 104 00:05:28,925 --> 00:05:30,515 are the different types of snapshots? 105 00:05:30,815 --> 00:05:35,525 Both what I'm gonna call the traditional snapshot, which is the ones that you and 106 00:05:35,525 --> 00:05:41,315 I, you know, grew up on, if you will, and then something that is relatively new. 107 00:05:42,905 --> 00:05:47,015 Again, comparatively speaking, new, uh, that is also being called 108 00:05:47,015 --> 00:05:49,835 snapshots, which is actually something completely different. 109 00:05:50,465 --> 00:05:54,785 So, um, uh, and then what purpose they serve, uh, et 110 00:05:54,785 --> 00:05:55,535 cetera, et cetera, et cetera. 111 00:05:55,565 --> 00:05:59,165 Any, any other sort of introductory things you can think of before we get started? 112 00:05:59,525 --> 00:05:59,735 no. 113 00:05:59,735 --> 00:06:03,845 I also did like your analogy of a snapshot is like a picture 114 00:06:03,875 --> 00:06:05,495 that you would take right 115 00:06:05,675 --> 00:06:05,945 Yeah. 116 00:06:06,125 --> 00:06:07,535 of like tech, right? 117 00:06:07,565 --> 00:06:08,015 And, 118 00:06:08,495 --> 00:06:08,825 Yeah. 119 00:06:09,065 --> 00:06:14,375 thing to add is that's important is it captures a moment in time. 120 00:06:15,245 --> 00:06:15,785 Yes. 121 00:06:15,845 --> 00:06:16,175 Right. 122 00:06:16,265 --> 00:06:16,385 A 123 00:06:16,595 --> 00:06:18,935 One moment in time. 124 00:06:18,940 --> 00:06:19,180 Sorry, 125 00:06:19,235 --> 00:06:19,565 and 126 00:06:20,260 --> 00:06:20,600 go ahead. 127 00:06:20,735 --> 00:06:24,785 but, so we're not referring to a video or whatever Apple calls 128 00:06:24,785 --> 00:06:26,555 their eight second or six second 129 00:06:27,050 --> 00:06:27,340 Yeah, 130 00:06:27,575 --> 00:06:28,385 that they send over. 131 00:06:28,385 --> 00:06:28,595 Right. 132 00:06:28,595 --> 00:06:29,645 That's not what we're referring to. 133 00:06:29,645 --> 00:06:30,335 It's a single 134 00:06:30,635 --> 00:06:30,905 Yeah. 135 00:06:31,970 --> 00:06:32,360 Yeah. 136 00:06:33,410 --> 00:06:38,720 So first let's talk about what I'm gonna call the traditional snapshot 137 00:06:39,380 --> 00:06:46,580 and, um, which I, another word, it's funny now that I think about it. 138 00:06:46,820 --> 00:06:53,060 This other word that I'm gonna use is also sort of like, um, it's 139 00:06:53,060 --> 00:06:55,250 also a word that is used outside. 140 00:06:56,030 --> 00:06:57,980 Id to also mean. 141 00:06:58,520 --> 00:07:00,770 You know, looking at a thing, right? 142 00:07:01,040 --> 00:07:04,040 And, uh, the word that I'm gonna use is view. 143 00:07:04,150 --> 00:07:04,570 Mm-hmm. 144 00:07:04,640 --> 00:07:10,550 So, uh, if anybody has any experience in databases. 145 00:07:11,210 --> 00:07:11,720 Right. 146 00:07:12,230 --> 00:07:19,970 There is this concept called a view, which is a different view into the database, 147 00:07:19,970 --> 00:07:27,440 which gives you, it looks different than, um, than the, the, the entire database. 148 00:07:27,440 --> 00:07:27,620 Right. 149 00:07:27,620 --> 00:07:30,710 It is a, it is a view, a a particular point of view, if 150 00:07:30,710 --> 00:07:32,480 you will, into the database. 151 00:07:32,630 --> 00:07:35,000 I don't want to go too much into that for two reasons. 152 00:07:35,000 --> 00:07:36,560 One is not really relevant. 153 00:07:36,740 --> 00:07:40,220 Two, I will probably mess up the description, but, um. 154 00:07:41,430 --> 00:07:45,330 Because it's one of those areas where it's right at the edge of my, of my experience. 155 00:07:45,330 --> 00:07:49,050 I am not a DBA, um, but I've often had to pretend to be one 156 00:07:49,050 --> 00:07:50,400 in order to back up databases. 157 00:07:51,060 --> 00:07:56,250 But this goes to your concept or your, the, the point that you were 158 00:07:56,250 --> 00:08:02,910 making is that a snapshot, uh, is a view of the storage at a particular 159 00:08:02,910 --> 00:08:09,450 point in time, so it is a view of your typically file system, but also 160 00:08:09,540 --> 00:08:11,220 it could just be a volume, right? 161 00:08:11,220 --> 00:08:17,460 A virtual volume at a particular point in time and what your, when 162 00:08:17,460 --> 00:08:24,390 you look at that snapshot, the actual blocks that you are looking at, or 163 00:08:24,390 --> 00:08:26,550 you're copying or you're accessing. 164 00:08:27,825 --> 00:08:29,295 What you are looking at. 165 00:08:29,895 --> 00:08:36,915 Is going to like where the blocks come from is going to be based on how the 166 00:08:36,915 --> 00:08:38,805 snapshot is actually being delivered. 167 00:08:39,045 --> 00:08:43,845 Sometimes you are going to be looking at the blocks from 168 00:08:43,845 --> 00:08:45,705 the original storage device. 169 00:08:45,885 --> 00:08:48,915 Sometimes you're going to be looking at blocks from some 170 00:08:48,915 --> 00:08:51,945 sort of cash snapshot area. 171 00:08:52,135 --> 00:08:52,555 Mm-hmm. 172 00:08:53,015 --> 00:08:57,425 Um, and so this view that you're looking at is this sort of virtual 173 00:08:57,425 --> 00:09:02,075 view of a particular point in time and it just keeps track of everything. 174 00:09:02,075 --> 00:09:07,775 And it says, okay, at this point in time they go to ask for this block. 175 00:09:08,105 --> 00:09:10,055 It knows what was where. 176 00:09:10,055 --> 00:09:13,475 So it goes, at this point in time, block A was. 177 00:09:13,860 --> 00:09:18,300 You know, at this status, well block A is still that status over on the original 178 00:09:18,300 --> 00:09:19,830 storage device, so we're gonna go get it. 179 00:09:20,100 --> 00:09:24,570 Or if you ask for block B, you go, block B was this status, and we can 180 00:09:24,570 --> 00:09:26,460 see that that status has changed. 181 00:09:26,610 --> 00:09:30,090 And so we're gonna get that block from the snapshot area. 182 00:09:30,570 --> 00:09:31,170 Um, 183 00:09:31,905 --> 00:09:32,205 And 184 00:09:32,280 --> 00:09:32,610 go ahead. 185 00:09:32,955 --> 00:09:35,535 does a different implementation. 186 00:09:36,030 --> 00:09:36,420 Yeah. 187 00:09:36,525 --> 00:09:37,365 think most of. 188 00:09:38,400 --> 00:09:43,530 As of what I know right now is they're all kind of, all the blocks 189 00:09:43,530 --> 00:09:48,540 are shared, and like you said, as things change, new copies are created 190 00:09:48,540 --> 00:09:50,340 that contain the updated version. 191 00:09:50,730 --> 00:09:53,730 But there's all this metadata and mapping, all this stuff is what 192 00:09:53,730 --> 00:09:57,720 the storage vendors, that's really like what they focus on, right? 193 00:09:57,720 --> 00:09:58,950 Is how do they make it fast? 194 00:09:58,950 --> 00:10:03,600 How do they make sure that you have that plausible point in time view to look at. 195 00:10:07,140 --> 00:10:07,560 Yeah. 196 00:10:07,680 --> 00:10:10,890 And, uh, we're gonna talk about, uh, like a couple of different 197 00:10:10,890 --> 00:10:13,110 ways that this actually happens. 198 00:10:13,560 --> 00:10:18,840 Uh, but the, but when you think about the way I described how that works, 199 00:10:19,950 --> 00:10:25,650 the important thing to understand is I said that if you grab blocks, one 200 00:10:25,650 --> 00:10:29,400 part of the, you know, one set of blocks where you're going to get those 201 00:10:29,400 --> 00:10:31,260 blocks from is from the original. 202 00:10:32,340 --> 00:10:33,480 Thing, right? 203 00:10:33,485 --> 00:10:35,430 The, the device, the volume, et cetera. 204 00:10:36,810 --> 00:10:39,930 And the other way is you're gonna get blocks from the snapshot area, but 205 00:10:39,930 --> 00:10:44,670 the, but the thing that you should infer from that is that in order for 206 00:10:44,670 --> 00:10:49,470 the snapshot to function, you need the original device to continue to be 207 00:10:49,890 --> 00:10:50,310 Mm-hmm. 208 00:10:50,430 --> 00:10:52,710 functional and et cetera, et cetera, et cetera. 209 00:10:52,815 --> 00:10:53,145 Yeah. 210 00:10:53,940 --> 00:10:54,300 Yeah. 211 00:10:54,780 --> 00:10:55,200 Um, 212 00:10:55,395 --> 00:10:55,785 thing to 213 00:10:55,890 --> 00:10:56,040 yeah. 214 00:10:56,685 --> 00:10:58,425 that's important for snapshots Curtis 215 00:10:58,425 --> 00:10:58,845 Mm-hmm. 216 00:10:59,655 --> 00:11:04,155 is that snapshots are read only, right? 217 00:11:04,185 --> 00:11:11,955 Which is a critical fundamental, uh, the, the property of a snapshot, right? 218 00:11:11,955 --> 00:11:12,375 Just like a 219 00:11:12,555 --> 00:11:12,885 Yes. 220 00:11:13,065 --> 00:11:15,525 you take a picture, you're not modifying that picture, 221 00:11:16,155 --> 00:11:16,605 Right. 222 00:11:18,375 --> 00:11:19,515 Good point, good point. 223 00:11:19,845 --> 00:11:24,075 Um, and because otherwise it wouldn't be a snapshot of that point in time. 224 00:11:24,465 --> 00:11:27,855 Now you can take a snapshot and you can make it read, right? 225 00:11:28,125 --> 00:11:31,305 But at that point, it's really no longer a snapshot, right? 226 00:11:31,605 --> 00:11:35,115 Um, it is a completely different sort of function. 227 00:11:35,625 --> 00:11:41,265 Yeah, and typically for those ones do have that snapshot, and then 228 00:11:41,445 --> 00:11:45,135 normally when you make it read, write, you're creating a new entity. 229 00:11:45,135 --> 00:11:48,045 So the snapshot still exists, but then the all the writes go 230 00:11:48,045 --> 00:11:48,375 Correct, 231 00:11:48,495 --> 00:11:48,945 entity. 232 00:11:50,025 --> 00:11:50,475 correct. 233 00:11:50,475 --> 00:11:50,925 Correct. 234 00:11:52,517 --> 00:11:54,952 Um, so there are. 235 00:11:56,402 --> 00:12:03,392 Two different ways that snapshots, again, traditional snapshots are typically done. 236 00:12:03,932 --> 00:12:11,402 And there's what I'm gonna call, um, sort of the the traditional Traditional, okay. 237 00:12:11,462 --> 00:12:13,472 And that is this concept. 238 00:12:13,532 --> 00:12:19,652 And this is pre NetApp, basically right before NetApp started doing snapshots. 239 00:12:19,682 --> 00:12:24,542 'cause NetApp really kind of reinvented snapshots and, and how they work, right? 240 00:12:25,682 --> 00:12:31,392 Before NetApp, you had what was called the copy on write Snapshot and where 241 00:12:31,392 --> 00:12:37,892 it gets, or CCOW copy on write Snapshot and where it gets that, that name is 242 00:12:37,892 --> 00:12:40,262 that you, you have a block, right? 243 00:12:40,412 --> 00:12:43,442 When, when you make the snapshot at the very beginning of making the snapshot, 244 00:12:43,892 --> 00:12:47,102 all blocks are in the original device. 245 00:12:48,002 --> 00:12:49,322 Because nothing has changed. 246 00:12:49,772 --> 00:12:55,952 The moment you go to do something, you go to update, you know, a block. 247 00:12:56,462 --> 00:12:58,202 The question is what happens now? 248 00:12:58,292 --> 00:13:03,152 So in a copy on write situation, the um. 249 00:13:03,727 --> 00:13:09,697 We're going to, before we change that block, we're going to copy that block out 250 00:13:09,697 --> 00:13:18,007 to the snapshot area so that later when we go to, um, access that block, we say, oh, 251 00:13:18,007 --> 00:13:19,927 we need the block from this point in time. 252 00:13:20,317 --> 00:13:23,107 That block, at that point in time, is only available over 253 00:13:23,107 --> 00:13:25,147 in the snapshot area, right. 254 00:13:26,592 --> 00:13:31,232 What's important to understand is that when you go to update a, a block 255 00:13:31,532 --> 00:13:39,212 with a copy on write, um, snapshot, it has to do a right of that block. 256 00:13:39,482 --> 00:13:44,012 It has to move the data, then copy, you know, write that. 257 00:13:45,112 --> 00:13:49,852 Old block in the snapshot area, and then it's gonna write the 258 00:13:49,852 --> 00:13:51,502 new version of that block. 259 00:13:51,772 --> 00:13:56,092 So there are three IO operations for every right operation 260 00:13:56,482 --> 00:13:58,222 in a copy on write snapshot. 261 00:13:58,822 --> 00:14:01,702 And then the more snapshots you have. 262 00:14:02,542 --> 00:14:09,082 And the more, and the longer you keep them, the more, uh, of your blocks 263 00:14:09,082 --> 00:14:14,902 that you're going to have to copy every time you, uh, write new data. 264 00:14:15,202 --> 00:14:17,542 Is that, how did I do with that explanation? 265 00:14:17,782 --> 00:14:18,412 yeah, you did. 266 00:14:18,412 --> 00:14:24,322 Well, um, the other thing also is at some point you end up with a 267 00:14:24,467 --> 00:14:24,757 Yeah. 268 00:14:26,572 --> 00:14:30,112 As well, just because of how writes are being done. 269 00:14:30,472 --> 00:14:34,282 And remember all the time, these aren't just small blocks, right? 270 00:14:34,282 --> 00:14:39,802 So even if you go modify, say a hundred bytes in a block, you still 271 00:14:39,802 --> 00:14:41,902 have to write that entire block back, 272 00:14:42,442 --> 00:14:42,802 Yeah. 273 00:14:42,982 --> 00:14:43,192 Yeah. 274 00:14:43,342 --> 00:14:46,732 sort of the underlying building block for the file system. 275 00:14:46,732 --> 00:14:46,852 And 276 00:14:47,182 --> 00:14:47,452 Right. 277 00:14:47,452 --> 00:14:52,462 this could lead to a lot of wasted reads and writes and IO that you're consuming. 278 00:14:52,912 --> 00:14:56,752 Are you, did you just say that the block is the basic building block? 279 00:15:00,907 --> 00:15:02,992 It is funny, the words that we use, right? 280 00:15:03,142 --> 00:15:04,222 Um, yeah. 281 00:15:04,222 --> 00:15:10,462 And so, um, what's it, I, I think one of the most important things to understand 282 00:15:10,462 --> 00:15:16,822 about a copy on write Snapshot set up is that over time, if you have a lot of 283 00:15:16,822 --> 00:15:21,052 snapshots and you keep those snapshots for a significant amount of time. 284 00:15:21,907 --> 00:15:26,887 The performance of your primary array because it's having to do all of this 285 00:15:26,887 --> 00:15:29,657 copying, you know, on, you know, on write. 286 00:15:30,127 --> 00:15:33,487 The performance of that, both the read performance and the, and 287 00:15:33,487 --> 00:15:38,617 especially the write performance can be significantly degraded over time. 288 00:15:38,977 --> 00:15:45,907 Yeah, and I'm just thinking, don't know any systems to my 289 00:15:46,252 --> 00:15:46,672 Mm-hmm. 290 00:15:46,957 --> 00:15:48,377 that still use copy on write. 291 00:15:50,032 --> 00:15:53,482 I am sure there are some, but I, but I would agree with you that 292 00:15:53,812 --> 00:16:00,122 most newer storage vendors realize sort of the evil of copy on write. 293 00:16:00,472 --> 00:16:05,377 You know, the, the, you know, it's, it's, I. It's, it's, it's sort of like 294 00:16:05,707 --> 00:16:07,867 the famous Maya Angelou quote, right? 295 00:16:07,867 --> 00:16:10,327 Like, we did what we did when we knew what we knew, but now we know 296 00:16:10,327 --> 00:16:11,347 different and we do different. 297 00:16:11,347 --> 00:16:11,647 Right? 298 00:16:12,007 --> 00:16:16,477 Uh, that's a, that's a massive, uh, I'm sure misquote of, but I think I 299 00:16:16,477 --> 00:16:18,067 got, I think I got the concept there. 300 00:16:18,827 --> 00:16:23,867 And I remember being at a large, very large oil and gas company 301 00:16:24,467 --> 00:16:26,262 and we were, um, you know. 302 00:16:26,972 --> 00:16:30,812 Helping them do a, it was an, it was a RFP, right? 303 00:16:30,872 --> 00:16:36,422 Um, for, for just a massive, uh, storage change. 304 00:16:36,422 --> 00:16:36,782 Right. 305 00:16:37,472 --> 00:16:44,102 And, um, they knew how many snapshots they wanted to create, and they knew 306 00:16:44,102 --> 00:16:45,692 how long they wanted to create them. 307 00:16:45,692 --> 00:16:47,612 They were already a, a NetApp customer. 308 00:16:47,612 --> 00:16:51,482 The, the problem with being a NetApp customer was they knew every. 309 00:16:52,067 --> 00:16:53,747 Bad thing about NetApp, right? 310 00:16:54,262 --> 00:16:58,187 They, they, they could see right through the stuff that they said that was bs. 311 00:16:58,187 --> 00:16:58,547 Right. 312 00:16:58,547 --> 00:17:01,697 You know, every vendor has got some amount of bs, but they were a customer, 313 00:17:01,697 --> 00:17:06,017 so they knew everything that NetApp, you know, they knew every blemish about 314 00:17:06,017 --> 00:17:10,142 NetApp, but they, and they were bringing in other vendors to see if perhaps they 315 00:17:10,142 --> 00:17:11,687 could get what they get from NetApp. 316 00:17:12,362 --> 00:17:14,732 Uh, while changing the vendor. 317 00:17:14,792 --> 00:17:17,732 And one of the things that they want is they wanted 90 days of user 318 00:17:17,732 --> 00:17:20,882 browsable snapshots, and they had all this data to show how much money 319 00:17:20,882 --> 00:17:24,722 that that was saving them because of the number of user generated 320 00:17:24,722 --> 00:17:26,432 restorers that were happening, right? 321 00:17:26,672 --> 00:17:30,392 Because one of the great things about having snapshots is that if you give 322 00:17:30,392 --> 00:17:34,832 them access, your users can just go to the right area and they can see their 323 00:17:34,832 --> 00:17:37,112 directory from a different point in time. 324 00:17:37,322 --> 00:17:37,682 Right? 325 00:17:38,102 --> 00:17:41,372 And so they, they had all this data that showed that they had all these user. 326 00:17:42,152 --> 00:17:48,122 Generated restores and um, uh, and so they said we want 90 days 327 00:17:48,122 --> 00:17:49,532 of user browsable snapshots. 328 00:17:49,582 --> 00:17:53,032 I won't name the vendor, but let's just say it's a fundamental law of physics. 329 00:17:53,512 --> 00:17:53,932 Um. 330 00:17:56,767 --> 00:18:02,407 That vendor said, uh, if you do that, you know, you're, you're, you're, 'cause 331 00:18:02,407 --> 00:18:05,737 they were, you know, they were doing copy on write snapshots at the time, and 332 00:18:05,737 --> 00:18:08,977 they basically said, you're going to have a significant performance degradation. 333 00:18:08,977 --> 00:18:14,167 And they, and they asked them like, how bad, and they just sort of, you know. 334 00:18:14,992 --> 00:18:18,832 Spitball the number, and they said it was 50%, like a 50% performance 335 00:18:18,832 --> 00:18:22,372 degradation, but they're like, we actually don't know because no one does 336 00:18:22,372 --> 00:18:24,532 that, no one does that with our storage. 337 00:18:24,622 --> 00:18:24,952 Right. 338 00:18:25,522 --> 00:18:30,052 Um, and, um, uh, yeah, so it's, it's a really big deal. 339 00:18:30,052 --> 00:18:30,292 Right. 340 00:18:30,292 --> 00:18:30,623 So that is, 341 00:18:30,832 --> 00:18:31,042 Yeah. 342 00:18:31,107 --> 00:18:31,717 I. Copy on. 343 00:18:31,717 --> 00:18:32,047 Right? 344 00:18:32,047 --> 00:18:34,807 Which is the old school way of doing snapshots. 345 00:18:35,557 --> 00:18:37,717 And if you're currently having any problems with your snapshots, 346 00:18:37,717 --> 00:18:39,857 maybe you investigate and see if they're copy on write? 347 00:18:39,877 --> 00:18:41,497 That's gonna be your core problem, right? 348 00:18:42,127 --> 00:18:45,397 Um, and, and it's gonna fall under the category of 349 00:18:45,577 --> 00:18:46,987 doctorate hurts when I do this. 350 00:18:47,077 --> 00:18:47,827 Well, don't do that. 351 00:18:47,827 --> 00:18:49,837 You need to do fewer snapshots or whatever. 352 00:18:50,072 --> 00:18:53,072 I just realized there is one vendor that still does copy on. 353 00:18:53,072 --> 00:18:53,412 write. 354 00:18:53,432 --> 00:18:54,302 And do you know what that is? 355 00:18:55,682 --> 00:18:55,902 Who? 356 00:18:56,942 --> 00:18:59,192 Microsoft with VSS Snapshots? 357 00:18:59,422 --> 00:18:59,902 VSS. 358 00:18:59,902 --> 00:19:00,942 The volume shadow services. 359 00:19:00,972 --> 00:19:01,262 Yeah. 360 00:19:01,452 --> 00:19:01,742 Yeah. 361 00:19:02,252 --> 00:19:02,542 Yeah. 362 00:19:02,612 --> 00:19:03,542 I was just thinking about that. 363 00:19:03,542 --> 00:19:06,062 I'm like, there has to be someone, and I was like, oh, yep. 364 00:19:07,007 --> 00:19:07,337 Yeah. 365 00:19:07,337 --> 00:19:10,757 And, and by the way, that brings up a, a really good, uh, thing that maybe 366 00:19:10,757 --> 00:19:12,347 we should have covered earlier on. 367 00:19:12,347 --> 00:19:14,417 And why do you, why do you make snapshots? 368 00:19:14,417 --> 00:19:14,717 Right? 369 00:19:15,077 --> 00:19:16,187 There's two reasons. 370 00:19:16,187 --> 00:19:18,047 And VSS is the second reason. 371 00:19:18,437 --> 00:19:21,917 The first reason is that it acts as sort of, kind of like a backup 372 00:19:21,967 --> 00:19:22,387 Mm-hmm. 373 00:19:22,487 --> 00:19:25,907 that it gives you this view, this place that you can go get and get the file 374 00:19:25,907 --> 00:19:29,597 the way it looked three weeks ago, three months ago, six months ago, whatever. 375 00:19:29,597 --> 00:19:31,067 However long you keep your snapshot. 376 00:19:31,637 --> 00:19:31,997 Go ahead. 377 00:19:32,492 --> 00:19:34,202 can we call it little B backup? 378 00:19:36,272 --> 00:19:38,372 Because, because here, here's my 379 00:19:38,377 --> 00:19:39,182 I what you're saying? 380 00:19:39,182 --> 00:19:39,632 Yeah. 381 00:19:40,052 --> 00:19:42,662 gets, because it doesn't follow the traditional 382 00:19:42,992 --> 00:19:43,592 Yeah. 383 00:19:43,772 --> 00:19:45,842 backup of 3, 2, 1 rule and everything 384 00:19:45,887 --> 00:19:49,007 Yeah, that's why I'm saying it's kind of like a backup, right? 385 00:19:49,187 --> 00:19:49,547 Yeah. 386 00:19:49,967 --> 00:19:53,507 Um, it, it's not really a backup until you copy that snapshot 387 00:19:53,537 --> 00:19:54,827 to some other location, right? 388 00:19:55,817 --> 00:19:58,337 So, and it also needs to be managed and all that kind of stuff, 389 00:19:58,397 --> 00:20:00,227 and you need to know what's in it and all that kind of thing. 390 00:20:00,227 --> 00:20:03,137 Anyway, but to, to follow the 3, 2, 1 rule, you need to at least 391 00:20:03,137 --> 00:20:04,457 copy it to another location. 392 00:20:04,457 --> 00:20:07,037 Hopefully that location is offsite and now we got three 393 00:20:07,037 --> 00:20:08,357 copies and two locations we got. 394 00:20:08,627 --> 00:20:08,897 Okay. 395 00:20:09,527 --> 00:20:13,337 So, um, actually, if all you're doing is. 396 00:20:13,907 --> 00:20:17,627 Snapshot of replication, you're actually not following 3, 2, 1 either because 397 00:20:17,627 --> 00:20:21,617 it's not, you don't follow the two, you don't have, you know, 'cause the two is 398 00:20:21,617 --> 00:20:24,737 meant to be two different risk profiles. 399 00:20:24,767 --> 00:20:28,217 And if you, if you go from one filer to another filer and you're, yeah. 400 00:20:28,247 --> 00:20:28,547 Okay. 401 00:20:29,657 --> 00:20:30,197 Um. 402 00:20:30,432 --> 00:20:35,952 The second reason that you make a snapshot and VSS falls under this 403 00:20:35,952 --> 00:20:39,192 typically, and that is what you're doing, is you're creating a stable 404 00:20:39,432 --> 00:20:43,482 point in time that is now read only that you can then use to back up. 405 00:20:43,577 --> 00:20:43,997 Mm-hmm. 406 00:20:44,112 --> 00:20:45,402 You can point your backup. 407 00:20:45,582 --> 00:20:50,532 And most backup vendors that are, that support windows, they integrate with VSS. 408 00:20:50,712 --> 00:20:51,852 And what VSS do? 409 00:20:51,912 --> 00:20:55,842 You, you, you talk to VSS and, , basically, you know, quick 410 00:20:55,842 --> 00:20:59,112 summary there, there's, there's this concept of a, of a VSS writer. 411 00:20:59,112 --> 00:21:01,932 You talk to the VSS and you say, Hey, I'm here to do a backup. 412 00:21:02,502 --> 00:21:05,352 What kind of, um, you know, what kind of stuff do you have? 413 00:21:05,352 --> 00:21:06,492 And they're like, we have SQL Server. 414 00:21:06,492 --> 00:21:10,032 You're like, okay, do a snapshot for SQL Server, and then you, and then 415 00:21:10,032 --> 00:21:14,442 that gives you a, a stable point in time to back up this, this device that 416 00:21:14,442 --> 00:21:16,122 otherwise is moving around, right? 417 00:21:16,122 --> 00:21:18,282 So it, it gives you a read. 418 00:21:18,507 --> 00:21:21,057 Only copy that isn't changing while you're backing it up. 419 00:21:21,057 --> 00:21:24,417 And then if you're just there to do a backup, it actually immediately 420 00:21:24,417 --> 00:21:26,577 deletes the snapshot as soon as you're done with the backup. 421 00:21:26,577 --> 00:21:26,757 Right. 422 00:21:26,757 --> 00:21:29,427 So that's the second reason that we use a, a snapshot, which is 423 00:21:29,427 --> 00:21:32,637 just to create a stable point in time from which we're gonna run. 424 00:21:32,937 --> 00:21:37,347 I'll call it a big B backup to use your, to use your, your terminology. 425 00:21:37,872 --> 00:21:38,307 All right. 426 00:21:38,307 --> 00:21:39,577 So that's copy on write? 427 00:21:40,797 --> 00:21:43,497 Uh, do you wanna describe, redirect on write. 428 00:21:45,627 --> 00:21:46,077 Yeah. 429 00:21:46,227 --> 00:21:49,827 So this is basically what NetApp does 430 00:21:50,172 --> 00:21:50,562 Yeah. 431 00:21:51,012 --> 00:21:53,412 Although they would say no 'cause they say there's a slightly 432 00:21:53,412 --> 00:21:54,282 different, but that's okay. 433 00:21:54,372 --> 00:21:54,702 You know? 434 00:21:55,002 --> 00:21:55,332 Yeah. 435 00:21:55,857 --> 00:22:00,507 NetApp is probably most famous for this with its write anywhere file layout 436 00:22:00,762 --> 00:22:01,002 Yeah. 437 00:22:02,217 --> 00:22:02,577 right? 438 00:22:02,757 --> 00:22:07,767 Which basically says when you're going to actually do your writes, you don't 439 00:22:07,767 --> 00:22:09,267 need to overwrite the existing block. 440 00:22:10,002 --> 00:22:10,452 Right. 441 00:22:10,707 --> 00:22:13,737 gonna pick any other random block, and then you're just 442 00:22:13,737 --> 00:22:15,267 gonna keep track of where it is. 443 00:22:15,267 --> 00:22:18,207 So all new writes go to all new blocks, so you don't have to ever 444 00:22:18,207 --> 00:22:22,767 worry about going and updating all the metadata and copying data out, and so 445 00:22:23,067 --> 00:22:24,597 you're efficient in your write code. 446 00:22:24,597 --> 00:22:28,317 Pap and managing snapshots becomes a lot easier. 447 00:22:29,307 --> 00:22:34,707 And so the where the term redirect on write, is instead of overriding the 448 00:22:34,707 --> 00:22:38,077 block in its current location, you're just gonna redirect that, write? 449 00:22:38,097 --> 00:22:41,937 Put it in another location, and then your snapshot manager just needs to keep 450 00:22:41,937 --> 00:22:43,767 track of where the old location was. 451 00:22:44,277 --> 00:22:47,187 So the big difference between a redirect on write, so there's a, 452 00:22:47,187 --> 00:22:48,327 there's a plus and minus here. 453 00:22:48,327 --> 00:22:50,427 The plus is much better performance, 454 00:22:51,012 --> 00:22:51,232 Yep. 455 00:22:51,387 --> 00:22:51,687 right? 456 00:22:51,687 --> 00:22:52,407 Which is why. 457 00:22:52,577 --> 00:22:55,307 Pretty much most vendors have gone with that as the newer. 458 00:22:55,637 --> 00:23:01,967 Um, the downside is snapshot data and production data all 459 00:23:01,967 --> 00:23:04,097 in the same volume, right? 460 00:23:04,097 --> 00:23:05,957 There's no snapshot area. 461 00:23:06,257 --> 00:23:07,217 It's, it's everything. 462 00:23:07,217 --> 00:23:10,967 So if you keep snapshots too long, you can actually fill up your, your 463 00:23:10,967 --> 00:23:13,457 volume, um, and have a problem. 464 00:23:13,787 --> 00:23:14,087 Yeah. 465 00:23:14,847 --> 00:23:17,757 And they, that's why they do try to do things like setting up a 466 00:23:17,757 --> 00:23:20,847 snapshot reserve space so you can automatically start deleting it 467 00:23:20,847 --> 00:23:23,067 once it gets to certain too high. 468 00:23:23,067 --> 00:23:26,517 But yeah, there are cases where that could also still be problematic. 469 00:23:26,517 --> 00:23:26,847 So, 470 00:23:27,387 --> 00:23:27,777 Yeah. 471 00:23:28,107 --> 00:23:32,187 And, and this is why, because you can do snapshots essentially. 472 00:23:32,187 --> 00:23:35,787 With impunity, you can do as many snapshots as you want, as often 473 00:23:35,787 --> 00:23:39,747 as you want, without, um, and, and keeping them as long as you 474 00:23:39,747 --> 00:23:43,827 want without any zero performance. 475 00:23:44,497 --> 00:23:49,417 Issues as long as you keep enough storage around for your snapshots, right? 476 00:23:49,717 --> 00:23:52,357 Uh, 'cause then we start getting low on blocks and we start, you know, 477 00:23:52,357 --> 00:23:55,057 then we, that, that, that, that, that's a totally different problem. 478 00:23:55,057 --> 00:24:00,367 So as long as you have enough space for your, uh, for everything, 479 00:24:00,367 --> 00:24:03,587 then you shouldn't have any performance from redirect on write? 480 00:24:03,607 --> 00:24:04,207 Snapshots. 481 00:24:04,617 --> 00:24:04,977 Yep. 482 00:24:05,037 --> 00:24:09,837 And I think with it now, I think the latest NetApp snapshot limit 483 00:24:09,837 --> 00:24:11,967 is I think a thousand snapshots. 484 00:24:12,357 --> 00:24:14,307 It used to be 2 55 for a while, but 485 00:24:14,367 --> 00:24:15,012 Yeah, the, 486 00:24:15,207 --> 00:24:15,267 a 487 00:24:15,417 --> 00:24:17,397 the famous 2 55 and yeah. 488 00:24:17,427 --> 00:24:17,727 Yep. 489 00:24:18,177 --> 00:24:21,207 It's funny how that number just sort of, well, 2 56 really is the number, 490 00:24:21,207 --> 00:24:23,487 but 2 55 is, we can't do 2 56. 491 00:24:23,487 --> 00:24:24,687 We can do two. 492 00:24:24,747 --> 00:24:27,597 Well, it's 2 55 because active is always zero. 493 00:24:28,392 --> 00:24:29,022 Oh, there you go. 494 00:24:29,142 --> 00:24:29,562 There you go. 495 00:24:29,652 --> 00:24:29,952 Yeah. 496 00:24:30,102 --> 00:24:30,372 Yeah. 497 00:24:31,212 --> 00:24:33,892 Um, so that's redirect on write. 498 00:24:33,912 --> 00:24:37,062 And that, you know, and we, we really talked about it very quickly, but 499 00:24:37,062 --> 00:24:40,872 basically it means that you can have as many snapshots as you want, as long as you 500 00:24:40,872 --> 00:24:42,702 want without any performance degradation. 501 00:24:42,702 --> 00:24:47,232 So that's generally what you find in most common, uh, snapshot providers right now. 502 00:24:47,277 --> 00:24:50,197 other thing I wanna also mention, copy on write? 503 00:24:50,217 --> 00:24:51,597 Versus redirect on write. 504 00:24:51,852 --> 00:24:52,272 Mm-hmm. 505 00:24:52,512 --> 00:24:55,932 discs, it made a huge benefit For Flash. 506 00:24:55,932 --> 00:25:01,632 It makes an even bigger impact with Flash there are issues around 507 00:25:01,632 --> 00:25:03,372 right amplifications when you're 508 00:25:03,732 --> 00:25:04,752 Mm-hmm. 509 00:25:04,992 --> 00:25:10,242 um, as well as how many wear cycles you can have on your flash. 510 00:25:10,242 --> 00:25:15,072 And so redirect on write is actually much better, both from a performance 511 00:25:15,072 --> 00:25:17,592 and longevity perspective for Flash 512 00:25:18,072 --> 00:25:18,402 Right. 513 00:25:18,402 --> 00:25:19,122 was on disc. 514 00:25:19,242 --> 00:25:19,452 So 515 00:25:19,547 --> 00:25:22,097 So the prob the problem there is that you're worried that you're having 516 00:25:22,097 --> 00:25:30,317 too many writes to one cell, and each individual cell in, um, flash has. 517 00:25:31,127 --> 00:25:35,567 A limited number of times that it can be overwritten, right? 518 00:25:36,047 --> 00:25:41,237 And so the worry that you have is if you have too many writes to the same block, 519 00:25:41,237 --> 00:25:43,157 you actually end, uh, to the same cell. 520 00:25:43,727 --> 00:25:47,387 You actually end up, that cell can become, uh, no longer usable 521 00:25:49,952 --> 00:25:53,372 but basically each cell has a finite number of times that you 522 00:25:53,372 --> 00:25:56,612 can overwrite that cell and you hit that and that cell is done. 523 00:25:56,612 --> 00:25:59,882 And now you know, it's like a bad block on a, or a bad sector on a disc, and 524 00:25:59,882 --> 00:26:01,292 you gotta move on to the next sector. 525 00:26:01,472 --> 00:26:05,492 and, and one other thing to mention too is I know so far when we talk about 526 00:26:05,492 --> 00:26:08,732 writes, people are probably thinking, oh, you're just writing a block of data. 527 00:26:09,062 --> 00:26:11,822 But remember with that block, you have a bunch of metadata 528 00:26:11,822 --> 00:26:13,022 associated with it, right? 529 00:26:13,322 --> 00:26:17,492 Indirect blocks that point to other blocks that contain metadata and everything else. 530 00:26:17,732 --> 00:26:19,682 That all needs to be updated by a file system. 531 00:26:19,682 --> 00:26:22,652 So even just doing like a hundred byte right, 532 00:26:22,852 --> 00:26:23,272 Mm-hmm. 533 00:26:23,372 --> 00:26:24,392 require, say. 534 00:26:24,752 --> 00:26:27,002 20 other blocks to be updated as 535 00:26:27,152 --> 00:26:28,022 Right, right. 536 00:26:28,142 --> 00:26:28,742 Exactly. 537 00:26:29,042 --> 00:26:29,432 Yeah. 538 00:26:30,062 --> 00:26:34,772 So those, so those are the two sort of what I'd call traditional snapshots. 539 00:26:34,802 --> 00:26:35,222 Right. 540 00:26:35,402 --> 00:26:37,862 Um, and now, and the second one is now what I would call 541 00:26:37,862 --> 00:26:39,302 sort of the standard snapshot. 542 00:26:39,527 --> 00:26:39,817 yeah. 543 00:26:39,902 --> 00:26:40,532 is a third. 544 00:26:41,102 --> 00:26:43,862 Really weird way that his snapshots are done, which I'm 545 00:26:43,862 --> 00:26:44,972 just gonna cover really quick. 546 00:26:44,972 --> 00:26:46,052 And that is VMware. 547 00:26:46,802 --> 00:26:52,652 So they do a, um, I, I believe it's a copy on write, but what's real 548 00:26:52,682 --> 00:26:56,022 actually, actually they do a, I'm gonna call it a weird on write? 549 00:26:56,042 --> 00:26:59,252 So what they do when you write a new block. 550 00:26:59,437 --> 00:27:04,837 They don't write the block to the, and and by the way, this is my understanding 551 00:27:04,837 --> 00:27:07,807 of the way VMware's, um, snapshots work. 552 00:27:08,077 --> 00:27:12,787 If I am now wrong, please tell me, but this is, I've checked repeatedly and this. 553 00:27:13,427 --> 00:27:14,837 I believe is still the case. 554 00:27:15,227 --> 00:27:19,217 So what would happen is when you take a snapshot in VMware, usually for backup 555 00:27:19,217 --> 00:27:25,487 purposes, um, when you go to write a new block in VMware, it doesn't write 556 00:27:25,547 --> 00:27:28,397 the new block in the production area. 557 00:27:28,517 --> 00:27:32,987 It writes the new block over in the snapshot area, leaves the 558 00:27:32,987 --> 00:27:38,417 original version of the block on the disc, and then it stays that way. 559 00:27:38,902 --> 00:27:43,522 Until you delete the snapshot, in which case it updates all the blocks to the new, 560 00:27:43,922 --> 00:27:44,432 Yep. 561 00:27:44,812 --> 00:27:45,352 thing. 562 00:27:45,802 --> 00:27:50,632 And, and, which is why, and, and I, I administered VMware for, for 563 00:27:50,632 --> 00:27:53,722 a while at least not really in a production environment, but, but 564 00:27:53,842 --> 00:27:58,132 you know, in a lab for a while before I realized this was the case. 565 00:27:58,402 --> 00:28:02,632 And so it means, what that really means in practice is you really can't use. 566 00:28:03,047 --> 00:28:03,677 At all. 567 00:28:03,677 --> 00:28:07,457 You can't use VMware snapshots the way you can use NetApp snapshots 568 00:28:07,457 --> 00:28:11,777 or even copy on write snapshots because the performance is way worse. 569 00:28:11,987 --> 00:28:12,377 Right. 570 00:28:12,707 --> 00:28:16,187 Uh, at some point you go to delete snapshots and you think deleting 571 00:28:16,187 --> 00:28:20,087 snapshots would be a nice thing, except when you delete the snapshot, there's 572 00:28:20,087 --> 00:28:25,337 this flurry of IO activity to update the primary volume with any of those blocks 573 00:28:25,337 --> 00:28:26,867 that are now the, the current block. 574 00:28:27,467 --> 00:28:29,477 It's the messiest thing I've ever seen. 575 00:28:29,882 --> 00:28:31,262 So two things to add. 576 00:28:31,322 --> 00:28:36,182 One is for VMware, it's okay if you keep those snapshots around. 577 00:28:36,182 --> 00:28:37,292 For a short duration. 578 00:28:37,547 --> 00:28:38,417 For a short duration. 579 00:28:38,417 --> 00:28:38,687 Yeah. 580 00:28:38,942 --> 00:28:39,392 for a short 581 00:28:39,437 --> 00:28:39,677 Yeah. 582 00:28:39,782 --> 00:28:43,832 it's fine, but don't think about using it for like that 90 day user restore use 583 00:28:44,027 --> 00:28:44,417 Mm-hmm. 584 00:28:44,462 --> 00:28:45,482 that you were mentioning earlier, 585 00:28:45,707 --> 00:28:46,907 it for a day or so. 586 00:28:46,967 --> 00:28:47,417 Right. 587 00:28:47,507 --> 00:28:48,257 And then delete it. 588 00:28:48,257 --> 00:28:48,497 Right. 589 00:28:48,497 --> 00:28:50,057 Make that a normal part of your thing. 590 00:28:50,222 --> 00:28:50,672 Exactly. 591 00:28:50,672 --> 00:28:56,612 And then the second point is with all of these snapshots, VSS VMware, most 592 00:28:56,612 --> 00:28:58,832 of them support two modes of operation. 593 00:28:58,832 --> 00:29:03,302 One is sort of a software based snapshot approach where they're 594 00:29:03,302 --> 00:29:04,322 dealing with all of this. 595 00:29:04,322 --> 00:29:06,662 So like Curtis mentioned with VMware, right? 596 00:29:06,662 --> 00:29:08,072 They're keeping track of everything. 597 00:29:08,522 --> 00:29:12,272 The second thing, second way that they also support it, is they 598 00:29:12,272 --> 00:29:14,132 integrate with the storage vendors. 599 00:29:14,537 --> 00:29:17,537 So when you go say VMware, go take a snapshot. 600 00:29:17,807 --> 00:29:22,007 VMware will do some things and then it'll call into the underlying storage 601 00:29:22,007 --> 00:29:26,267 provider and say, okay, now go do a hardware snapshot and lock in that data. 602 00:29:26,297 --> 00:29:29,177 In which case then you don't have all of these issues that Curtis just 603 00:29:29,372 --> 00:29:29,912 Exactly. 604 00:29:29,912 --> 00:29:30,932 That's a really good point. 605 00:29:30,932 --> 00:29:32,672 Thanks for, uh, bringing that up. 606 00:29:32,672 --> 00:29:33,002 Right. 607 00:29:33,002 --> 00:29:38,072 So VSS and VMware and maybe other sort of software level snapshots, they 608 00:29:38,072 --> 00:29:42,062 do have this concept of talking to a storage array and then putting that 609 00:29:42,062 --> 00:29:45,482 storage array, making, making, letting the storage array do the hard work. 610 00:29:45,482 --> 00:29:45,782 Right. 611 00:29:46,022 --> 00:29:46,862 Which is, which is. 612 00:29:47,482 --> 00:29:51,322 I, I think certainly the preferred way to do snapshots is to do it in, in hardware. 613 00:29:51,962 --> 00:29:52,202 Yeah. 614 00:29:52,762 --> 00:29:56,362 . So the final way of snapshots, and this is the snapshots that most of you 615 00:29:56,362 --> 00:30:01,192 know, because most of you are very, you are your head's in the clouds, right? 616 00:30:01,192 --> 00:30:08,092 And that is, and, and I'm, I blame Jeff Bezos, um, because it 617 00:30:08,092 --> 00:30:12,082 was Amazon, I believe that first started using the term snapshot. 618 00:30:12,412 --> 00:30:14,872 To actually refer to something completely different. 619 00:30:15,622 --> 00:30:21,562 If you go into a, you know, if you go to a number, it used to just be 620 00:30:21,562 --> 00:30:24,142 like EBS, the Elastic Block storage. 621 00:30:24,142 --> 00:30:24,502 Right. 622 00:30:25,402 --> 00:30:32,182 Um, that, or you take, uh, a snapshot of a EC2 E image. 623 00:30:32,182 --> 00:30:32,362 Yeah. 624 00:30:32,362 --> 00:30:33,412 RDS Right. 625 00:30:33,412 --> 00:30:33,532 You know. 626 00:30:35,017 --> 00:30:40,147 If you take a snapshot, as I made quotes in the air, it doesn't do 627 00:30:40,147 --> 00:30:41,767 anything like what we just said. 628 00:30:42,277 --> 00:30:46,477 It creates what I would call an image copy, right? 629 00:30:47,062 --> 00:30:51,472 Of that thing that you just took a snapshot of and it creates a 630 00:30:51,562 --> 00:30:54,262 basically bite for bite copy of that. 631 00:30:54,532 --> 00:30:59,092 And typically, I think it's, I'll just speak in terms of a, you know, AWS, 632 00:30:59,542 --> 00:31:02,392 it creates that bite for bite copy in. 633 00:31:02,682 --> 00:31:03,852 S3, right? 634 00:31:03,852 --> 00:31:09,402 It stores it in a special reserve area in S3, and so when you create a snapshot 635 00:31:09,432 --> 00:31:14,142 in pretty much anything in Amazon, what you're actually creating is a backup. 636 00:31:14,172 --> 00:31:16,692 You're creating an image copy of that device. 637 00:31:16,962 --> 00:31:18,612 You're putting it in S3. 638 00:31:19,127 --> 00:31:24,197 Um, you can even, um, you can often specify where an S3, right, so you 639 00:31:24,197 --> 00:31:27,587 can specify that you want that backup to go to another region, for example. 640 00:31:28,097 --> 00:31:34,427 Um, and you can also copy these snapshots around, um, and, um, yeah, 641 00:31:35,332 --> 00:31:38,032 But you can't access it directly via S3, 642 00:31:38,657 --> 00:31:39,707 correct, correct. 643 00:31:39,797 --> 00:31:42,797 Like, yeah, that's why I was saying it's like a special, like reserve area. 644 00:31:42,797 --> 00:31:43,427 NS three. 645 00:31:43,937 --> 00:31:44,237 Yeah. 646 00:31:44,392 --> 00:31:44,782 it's 647 00:31:45,197 --> 00:31:47,297 That's for, that's for a lot of safe safety reasons, right? 648 00:31:47,422 --> 00:31:47,632 Yeah. 649 00:31:47,722 --> 00:31:53,212 It's called a snapshot, but it honors a lot of our backup 650 00:31:53,212 --> 00:31:54,832 with a big B requirements. 651 00:31:55,372 --> 00:31:59,962 Yeah, because it's an actual copy, you can specify that it be sent to a, to 652 00:31:59,962 --> 00:32:01,882 another region, to another account. 653 00:32:02,392 --> 00:32:02,602 Right? 654 00:32:02,602 --> 00:32:03,382 And that's what I wanted. 655 00:32:03,382 --> 00:32:04,642 I want it in another account. 656 00:32:04,642 --> 00:32:05,932 I want it in another region. 657 00:32:06,172 --> 00:32:10,072 I don't want you to just do an EBS snapshot, for example, to the 658 00:32:10,072 --> 00:32:13,492 same availability zone, uh, that. 659 00:32:13,852 --> 00:32:16,822 Isn't, doesn't really follow the basic 3, 2, 1 rule, which is 660 00:32:16,822 --> 00:32:17,797 something we talk about a lot. 661 00:32:18,317 --> 00:32:18,607 Yeah. 662 00:32:18,727 --> 00:32:22,177 But I do, I do think it's important to understand that a snapshot in, 663 00:32:22,237 --> 00:32:27,187 in A AWS is so much, most, so much closer to what I would call a backup. 664 00:32:27,457 --> 00:32:29,887 In fact, depending on how you do it, it definitely is a backup 665 00:32:30,547 --> 00:32:33,577 than what we typically talk about when we say snapshots. 666 00:32:33,637 --> 00:32:34,027 Right. 667 00:32:34,057 --> 00:32:38,172 And, and it's interesting because there are vendors that have made 668 00:32:38,252 --> 00:32:41,017 a, a deal of, you know, they're like snapshots are not backup. 669 00:32:41,017 --> 00:32:43,567 And they've been saying that for so long that they continue to say that. 670 00:32:43,897 --> 00:32:47,137 When what we're really talking about is AWS when it really is 671 00:32:47,137 --> 00:32:48,517 a snap, it really is a backup. 672 00:32:48,817 --> 00:32:53,887 it's interesting you bring this point up because think other vendors, other 673 00:32:53,887 --> 00:32:59,617 traditional storage vendors are also now starting to go the same route 674 00:32:59,617 --> 00:33:04,657 where they're saying our snapshots can now be offloaded into object storage. 675 00:33:04,822 --> 00:33:05,172 Right. 676 00:33:06,562 --> 00:33:06,782 Yes. 677 00:33:06,942 --> 00:33:09,217 I, yeah, I have seen some of the, some vendors do that. 678 00:33:09,217 --> 00:33:09,457 Yeah. 679 00:33:09,547 --> 00:33:13,477 Right, which is very similar to what AWS has been doing, but I think 680 00:33:13,477 --> 00:33:14,887 they still call them snapshots. 681 00:33:15,637 --> 00:33:15,967 Right? 682 00:33:15,967 --> 00:33:16,147 So 683 00:33:16,357 --> 00:33:16,717 Yeah. 684 00:33:16,837 --> 00:33:21,817 of this, you have to, I think it's important as administrator, an IT 685 00:33:21,997 --> 00:33:24,457 person to really ask the question. 686 00:33:24,727 --> 00:33:28,077 When someone says snapshot, what do they mean? 687 00:33:28,567 --> 00:33:28,927 Yeah. 688 00:33:29,077 --> 00:33:29,467 Right. 689 00:33:29,497 --> 00:33:33,667 I'm gonna, and we had, we used the phrase snapshot backup in this, 690 00:33:33,727 --> 00:33:35,977 um, in the title of this episode. 691 00:33:36,892 --> 00:33:39,322 And this is a snapshot backup, right? 692 00:33:39,322 --> 00:33:43,222 So it, it's some would see snapshot backup as like a, like a oxymoron, 693 00:33:43,222 --> 00:33:47,002 like, like military intelligence, you know, um, that, you know, 694 00:33:47,002 --> 00:33:48,232 two words that can't go together. 695 00:33:48,232 --> 00:33:53,957 But if by snapshot you mean what they do in AWS and I. Similar, uh, 696 00:33:54,017 --> 00:33:58,007 you know, other cloud vendors, then I'm fine with calling that a backup. 697 00:33:58,277 --> 00:34:01,307 Just make sure, so the question is, how do you make a snapshot backup? 698 00:34:01,487 --> 00:34:02,837 Well, there's two ways. 699 00:34:03,077 --> 00:34:07,517 One is if you're talking the traditional type of, you know, storage snapshots, 700 00:34:07,817 --> 00:34:11,957 you need to copy that snapshot to another, uh, storage array. 701 00:34:11,957 --> 00:34:13,337 Hopefully one that's offsite. 702 00:34:13,757 --> 00:34:16,907 Um, hopefully one, you know, my dream would be. 703 00:34:17,977 --> 00:34:21,877 Is to another vendor or you, then you, you, you change the 704 00:34:21,877 --> 00:34:23,947 form of the snapshot, right? 705 00:34:23,947 --> 00:34:27,637 You know, you're doing, you're doing, you know, filer to filer and then you're, 706 00:34:27,727 --> 00:34:30,457 you're backing that up to some other, uh. 707 00:34:30,802 --> 00:34:31,462 Object store 708 00:34:31,507 --> 00:34:32,257 system, right? 709 00:34:32,277 --> 00:34:32,567 Yeah. 710 00:34:32,677 --> 00:34:36,577 might do NDMP, who, you know somewhere, uh, Steven, uh, 711 00:34:36,607 --> 00:34:38,437 his, his ears are perking up. 712 00:34:38,827 --> 00:34:45,397 Um, the, um, uh, anyway, but if it's, but if it's, if it's a cloud vendor, 713 00:34:45,547 --> 00:34:48,607 typically, and again, I can really only speak with, with authority and 714 00:34:48,607 --> 00:34:52,237 AWS, but from what I've seen, the other cloud vendors are very similar. 715 00:34:52,507 --> 00:34:55,747 When you take a snapshot in AWS, you're actually taking a backup. 716 00:34:56,017 --> 00:34:57,097 And when you're doing that. 717 00:34:57,397 --> 00:35:01,867 You can use AWS backup to sort of control all of this, which is AWS backup is 718 00:35:01,867 --> 00:35:05,167 really just a control plane for all the other stuff that's going on, and 719 00:35:05,167 --> 00:35:09,247 you can make sure that you know that you're following the 3, 2, 1 rules. 720 00:35:09,277 --> 00:35:13,447 You have a different account, different availability zone, different region even. 721 00:35:13,657 --> 00:35:17,767 Um, yes, it makes it cost more, but it actually makes it a real backup. 722 00:35:18,292 --> 00:35:20,542 So, questions for you. 723 00:35:20,857 --> 00:35:21,217 Yeah. 724 00:35:22,192 --> 00:35:27,022 So you mentioned that sort of doing snapshots, replicating it from 725 00:35:27,172 --> 00:35:32,212 one filer to another filer doesn't quite meet the rules of the two. 726 00:35:32,827 --> 00:35:33,067 Right? 727 00:35:33,292 --> 00:35:34,342 In 3, 2, 1, backup. 728 00:35:35,227 --> 00:35:36,337 Uh, in the 3, 2, 1 rule. 729 00:35:36,757 --> 00:35:41,587 What if so many of these vendors now allow you to run virtualized 730 00:35:41,587 --> 00:35:43,237 instances in public clouds? 731 00:35:44,182 --> 00:35:44,602 Mm-hmm. 732 00:35:45,487 --> 00:35:49,027 So I could run a NetApp instance in AWS 733 00:35:49,472 --> 00:35:49,762 Yeah. 734 00:35:50,857 --> 00:35:51,697 My answer is no. 735 00:35:52,082 --> 00:35:52,202 I. 736 00:35:52,387 --> 00:35:54,007 because it's still the same software version. 737 00:35:54,607 --> 00:35:54,967 Yeah. 738 00:35:55,627 --> 00:35:55,987 Okay. 739 00:35:56,107 --> 00:35:56,377 Yeah. 740 00:35:56,947 --> 00:35:58,837 Now AWS 741 00:35:58,867 --> 00:35:59,467 it's better. 742 00:35:59,472 --> 00:36:01,087 Better, you know, good, better, best. 743 00:36:01,087 --> 00:36:03,217 It's better than the other thing because at least it 744 00:36:03,217 --> 00:36:04,597 isolates you from the hardware. 745 00:36:04,897 --> 00:36:07,122 But it's still, the os still the same os. 746 00:36:07,702 --> 00:36:12,712 And would you also consider the same like AWS offers file systems, right? 747 00:36:12,712 --> 00:36:15,472 So they have like support for NetApp by FSX. 748 00:36:17,662 --> 00:36:20,812 Would you consider replicate if it's, I'm not even sure 749 00:36:20,857 --> 00:36:21,787 Yeah, I don't, I don't know. 750 00:36:21,847 --> 00:36:23,167 I don't know if I can speak to that. 751 00:36:23,197 --> 00:36:23,557 Right. 752 00:36:23,647 --> 00:36:24,847 I don't know if I can speak to that. 753 00:36:24,847 --> 00:36:25,147 Right. 754 00:36:25,417 --> 00:36:26,077 And then, and then 755 00:36:26,182 --> 00:36:28,102 do you have to be for the two is my question. 756 00:36:28,742 --> 00:36:31,237 I, I, again, you're, you're just trying to, it, it's sort of a 757 00:36:31,237 --> 00:36:32,437 good, better, best thing, right? 758 00:36:32,442 --> 00:36:35,857 The, the, the more different you can be, the better you are. 759 00:36:35,857 --> 00:36:37,957 It's difficult in this cloud world, right. 760 00:36:38,437 --> 00:36:45,547 Um, the, um, and some would say, well, isn't a w aren't AWS snapshots the same? 761 00:36:45,547 --> 00:36:46,027 And it's like. 762 00:36:46,597 --> 00:36:49,777 They're actually a little bit different because it is a full image 763 00:36:49,777 --> 00:36:53,557 copy, um, and it's stored in S3. 764 00:36:53,887 --> 00:36:57,877 If something were to happen to your EBS volume because of a bug in EBS, 765 00:36:58,177 --> 00:37:01,327 you could still use that snapshot, which is stored in a completely 766 00:37:01,327 --> 00:37:05,047 different system, which wouldn't have the same bug and store that in. 767 00:37:05,077 --> 00:37:07,327 You could restore that into EBS. 768 00:37:07,942 --> 00:37:10,582 that your volume wasn't corrupted by some EBS bug 769 00:37:12,517 --> 00:37:15,667 Well, but you're, but S3 isn't EBS. 770 00:37:15,862 --> 00:37:16,042 no. 771 00:37:16,042 --> 00:37:16,792 But when 772 00:37:16,882 --> 00:37:17,962 Oh, you mean, you mean before? 773 00:37:18,382 --> 00:37:21,292 Well, if it was created, if it was created, if it was corrupted 774 00:37:21,292 --> 00:37:23,362 by the EBSB, then you know Yeah. 775 00:37:23,362 --> 00:37:25,612 You're, that's gonna happen no matter what you do. 776 00:37:25,612 --> 00:37:28,132 So, so the I, I like where you're going, Curtis. 777 00:37:28,162 --> 00:37:31,642 I think there's also sort of the practicality aspect that we have to 778 00:37:31,642 --> 00:37:36,172 also look out for, because ideally, like you said, right vendor, you 779 00:37:36,172 --> 00:37:37,612 have multiple vendors, right? 780 00:37:37,612 --> 00:37:40,852 You're using different technologies, but then you have to balance that with 781 00:37:40,852 --> 00:37:42,322 pr, sort of the practicality, right? 782 00:37:42,322 --> 00:37:44,692 Are you gonna be able to understand all these various technologies, 783 00:37:44,692 --> 00:37:48,232 build out the skillsets, integrate it, pay for it, right? 784 00:37:48,232 --> 00:37:51,922 We all know how small backup budgets are versus production budgets. 785 00:37:52,192 --> 00:37:52,462 Yeah. 786 00:37:52,522 --> 00:37:53,962 Yeah, yeah, 787 00:37:53,992 --> 00:37:54,292 just. 788 00:37:54,742 --> 00:37:57,862 And, and I've always tried, you know, throughout my career to try to be. 789 00:37:58,732 --> 00:38:02,392 To say, look, if you're asking me my opinion, you know I'm 790 00:38:02,392 --> 00:38:03,862 giving, I'm giving you the bar. 791 00:38:03,982 --> 00:38:05,272 Right, right. 792 00:38:05,302 --> 00:38:10,432 And you know, and I understand that people have to, they have to, they have 793 00:38:10,432 --> 00:38:14,002 to live in the real world where, where backup is not the most important thing. 794 00:38:14,147 --> 00:38:16,192 Ah, it hurts me to say that. 795 00:38:16,942 --> 00:38:18,232 Um, right. 796 00:38:18,712 --> 00:38:19,462 Um, 797 00:38:19,687 --> 00:38:22,327 but at the same time, maybe there are certain use cases where 798 00:38:22,327 --> 00:38:27,397 you can focus on it because of the high value of the data or 799 00:38:27,472 --> 00:38:27,772 Yeah. 800 00:38:27,937 --> 00:38:29,767 like that, where it is important to have 801 00:38:31,147 --> 00:38:31,567 Yeah, 802 00:38:31,867 --> 00:38:32,497 type solution. 803 00:38:32,737 --> 00:38:35,917 also want to add, you know, we talk a lot about the 3, 2, 1 rule, and I 804 00:38:35,917 --> 00:38:39,697 say that because the 3, 2, 1, like if it doesn't follow the 3, 2, 1 rule 805 00:38:39,697 --> 00:38:41,407 and it's just not a backup, right? 806 00:38:41,407 --> 00:38:42,097 I'll say the number one. 807 00:38:42,097 --> 00:38:47,092 Number two, I think I'd be remiss to say that also I. You know, 808 00:38:47,092 --> 00:38:49,882 we're starting to talk about things like the 3, 3, 2, 1, 1 0. 809 00:38:49,882 --> 00:38:54,262 You know, you do want at least one of your backups, your copies 810 00:38:54,472 --> 00:38:55,852 to be on immutable storage. 811 00:38:55,942 --> 00:38:56,212 Right. 812 00:38:56,212 --> 00:38:58,012 To be on truly immutable storage. 813 00:38:58,252 --> 00:39:01,132 One that even you can't delete even if you want to. 814 00:39:01,402 --> 00:39:01,762 Right. 815 00:39:02,212 --> 00:39:07,462 Um, because only then is that backup, uh, protected against a ransomware attack. 816 00:39:07,942 --> 00:39:08,092 Yeah. 817 00:39:08,122 --> 00:39:12,172 Or a, or a, or a direct bad actor. 818 00:39:12,202 --> 00:39:14,212 You know, a, um, what's the, what do, what do we call it? 819 00:39:14,212 --> 00:39:14,362 Um. 820 00:39:14,617 --> 00:39:15,127 threat. 821 00:39:15,292 --> 00:39:15,712 Thank you. 822 00:39:15,712 --> 00:39:16,282 You know? 823 00:39:16,342 --> 00:39:20,047 No, there's a. Uh, uh, a rogue admin, right? 824 00:39:20,107 --> 00:39:22,987 Uh, you know, a rogue admin, either a rogue admin, right? 825 00:39:23,047 --> 00:39:28,057 'cause there have been rogue admin stories, you know, uh, but also someone 826 00:39:28,057 --> 00:39:32,917 who is able to gain access to your administrative account and then go 827 00:39:32,917 --> 00:39:34,657 and delete all your backups you need. 828 00:39:34,987 --> 00:39:41,257 Only if even you can't delete it, even if you want to, is a backup, truly immutable. 829 00:39:41,617 --> 00:39:44,497 Um, and so you can do that with snapshots. 830 00:39:44,497 --> 00:39:46,207 You just need to copy them into something else. 831 00:39:47,027 --> 00:39:47,287 Yeah. 832 00:39:48,667 --> 00:39:48,997 All right. 833 00:39:48,997 --> 00:39:49,687 This was fun. 834 00:39:51,697 --> 00:39:52,177 Yay. 835 00:39:52,237 --> 00:39:53,227 Welcome back, Curtis. 836 00:39:53,467 --> 00:39:53,887 Yeah. 837 00:39:53,887 --> 00:39:56,377 And our, our lag wasn't too bad on our thing. 838 00:39:56,437 --> 00:40:00,517 Hopefully our QOS did its job and hopefully our recording, uh, works. 839 00:40:00,697 --> 00:40:02,382 And with that, that is a wrap.