1 00:00:00,627 --> 00:00:04,047 W. Curtis Preston: Immutable backups are a critical defense against the rising 2 00:00:04,047 --> 00:00:09,057 tide of cyber threats, and at a time when ransomware attacks are targeting not just 3 00:00:09,057 --> 00:00:11,877 your primary data, but your backups too. 4 00:00:12,237 --> 00:00:15,207 Understanding immutability is more critical than ever. 5 00:00:15,747 --> 00:00:18,687 We'll explore what makes a backup truly immutable. 6 00:00:18,897 --> 00:00:22,617 The spectrum of immutability options available and why. 7 00:00:22,837 --> 00:00:25,987 This matters so much in your data protection strategy. 8 00:00:26,287 --> 00:00:30,847 From tape backups to cutting edge cloud solutions, we'll cover it all. 9 00:00:31,537 --> 00:00:32,437 Are you immutable? 10 00:00:32,437 --> 00:00:34,267 Backups actually immutable. 11 00:00:34,357 --> 00:00:35,347 Let's find out. 12 00:00:35,347 --> 00:00:35,407 I. 13 00:00:35,982 --> 00:00:40,452 By the way, if you don't know who I am, I'm w Curtis Preston, also known as Mr. 14 00:00:40,452 --> 00:00:40,932 Backup. 15 00:00:41,322 --> 00:00:46,212 I've been passionate about this topic for over 30 years, ever since. 16 00:00:46,212 --> 00:00:49,992 I had to tell my boss that there were no backups of the 17 00:00:49,997 --> 00:00:51,822 database that we had just lost. 18 00:00:52,212 --> 00:00:54,672 I don't want that to ever happen to me again. 19 00:00:54,817 --> 00:00:57,787 I don't want it to happen to you, and that's why we do this. 20 00:00:58,057 --> 00:01:04,177 On this podcast, we turn unappreciated backup admins into Cyber recovery Heroes. 21 00:01:04,387 --> 00:01:06,487 This is the backup wrap up. 22 00:01:22,583 --> 00:01:23,453 Welcome to the show. 23 00:01:24,473 --> 00:01:27,113 I'm your host, w Curtis Preston, AKA, Mr. 24 00:01:27,118 --> 00:01:33,098 Backup, and I have with me my SMB backup Difficulty. 25 00:01:33,578 --> 00:01:36,418 Sympathizer Prasanna Malaiyandi. 26 00:01:36,443 --> 00:01:36,898 How's it going? 27 00:01:37,008 --> 00:01:37,498 Persona. 28 00:01:38,828 --> 00:01:43,898 Prasanna Malaiyandi: I, I've learned more about dis space backups than, and tape 29 00:01:43,898 --> 00:01:49,418 based backups in the last, like three weeks from you than any time before. 30 00:01:49,418 --> 00:01:52,448 So yeah, how are you doing? 31 00:01:52,673 --> 00:01:56,843 W. Curtis Preston: oh, I just like, if I could get like a, like if I 32 00:01:56,843 --> 00:01:59,723 could have a wall here, just, you know, bang my head into the wall. 33 00:01:59,723 --> 00:02:01,463 That's sort of how I feel like I'm doing. 34 00:02:02,153 --> 00:02:02,663 I do 35 00:02:02,753 --> 00:02:05,423 Prasanna Malaiyandi: the problem is that you left it for 20 years 36 00:02:05,423 --> 00:02:06,803 and now you're coming back to it. 37 00:02:06,803 --> 00:02:07,133 And so 38 00:02:07,268 --> 00:02:11,108 W. Curtis Preston: That's not the problem that that implies that I am old 39 00:02:11,108 --> 00:02:13,508 or incompetent or something like that. 40 00:02:14,288 --> 00:02:15,218 That is not the problem. 41 00:02:15,323 --> 00:02:16,013 Prasanna Malaiyandi: me going back. 42 00:02:16,043 --> 00:02:17,393 It's like me going back and coding. 43 00:02:18,383 --> 00:02:20,118 I could do it, but I'm not gonna be efficient. 44 00:02:21,158 --> 00:02:22,568 W. Curtis Preston: The, the, the, um. 45 00:02:24,318 --> 00:02:27,773 It, it, it's been, it's been really weird, right? 46 00:02:27,773 --> 00:02:30,053 Because, you know, you change something and you change this. 47 00:02:30,053 --> 00:02:31,373 You're like, well, let's try this. 48 00:02:31,613 --> 00:02:34,283 And then you try that and you're like, that just doesn't work. 49 00:02:34,793 --> 00:02:39,653 That just, you know, like the, the, the, and the really weird thing that I. 50 00:02:40,028 --> 00:02:45,038 Has been pla plaguing me over the last, like, I don't know, several days as 51 00:02:45,038 --> 00:02:48,938 I've been trying to make a major change to this backup configuration is that 52 00:02:48,938 --> 00:02:52,838 the backup starts out, you know, I'm running like 20 backups at the same time. 53 00:02:53,318 --> 00:02:56,708 Backups start out and it, and it's running like 500 megabytes per second. 54 00:02:56,798 --> 00:02:57,518 This is great. 55 00:02:57,518 --> 00:02:58,388 This is what I want. 56 00:02:58,778 --> 00:03:00,848 And then about a half hour later he goes, 57 00:03:03,213 --> 00:03:03,758 you're like. 58 00:03:04,018 --> 00:03:05,183 Prasanna Malaiyandi: To a slow drip. 59 00:03:05,258 --> 00:03:05,678 W. Curtis Preston: happened? 60 00:03:05,948 --> 00:03:09,098 This is not, this is not tape, this is, you know, this is not 61 00:03:09,098 --> 00:03:12,098 like, there's not some weird tape, you know, shoe shining thing. 62 00:03:12,128 --> 00:03:16,328 This is disc two, disc via a server in the middle. 63 00:03:16,958 --> 00:03:19,498 Prasanna Malaiyandi: It is probably some random option somewhere 64 00:03:19,678 --> 00:03:20,308 W. Curtis Preston: random, 65 00:03:20,378 --> 00:03:21,018 Prasanna Malaiyandi: haven't done this. 66 00:03:21,658 --> 00:03:23,698 W. Curtis Preston: data buffers thing. 67 00:03:24,113 --> 00:03:24,928 Oh, oh, oh. 68 00:03:24,928 --> 00:03:29,133 You're, you wanna do this on a Thursday, you need to do the, you 69 00:03:29,133 --> 00:03:32,373 need to do the setting and, uh. 70 00:03:32,383 --> 00:03:34,543 Prasanna Malaiyandi: what I said, that because you haven't done this 71 00:03:34,543 --> 00:03:36,853 day in to day out, you don't know. 72 00:03:36,943 --> 00:03:39,043 It's like you lose touch of these or you don't know 73 00:03:39,163 --> 00:03:39,583 W. Curtis Preston: Yeah. 74 00:03:39,853 --> 00:03:46,573 And I, I will say that when I was, when I was, when I was last doing this really in 75 00:03:46,573 --> 00:03:49,603 earnest, this backups were relatively new. 76 00:03:50,143 --> 00:03:55,543 Um, that, you know, that really dates me, but that is, that is the case, but. 77 00:03:56,753 --> 00:03:57,803 So, yeah. 78 00:03:58,193 --> 00:04:01,163 And I just, you know, and I, I've been thankful for you to talk to you. 79 00:04:01,163 --> 00:04:03,953 It's like, you know, so here's the thing, you know, I got this, I got 80 00:04:03,953 --> 00:04:05,633 this setting, I got that setting. 81 00:04:05,843 --> 00:04:08,063 I've been playing with jumbo frames, I've been playing with, 82 00:04:08,663 --> 00:04:10,793 with, uh, buffer settings. 83 00:04:10,798 --> 00:04:12,563 I've been playing with all these different things. 84 00:04:12,563 --> 00:04:15,653 And you know, and you're like, what about the thing? 85 00:04:15,653 --> 00:04:16,733 You know, what about this setting? 86 00:04:16,763 --> 00:04:18,443 Oh, man. 87 00:04:18,898 --> 00:04:19,378 Prasanna Malaiyandi: And you know what? 88 00:04:19,383 --> 00:04:21,838 In the end, once you figure out what's going on, it's probably 89 00:04:21,843 --> 00:04:23,668 just gonna be some simple thing, 90 00:04:23,993 --> 00:04:24,413 W. Curtis Preston: Mm-hmm. 91 00:04:25,078 --> 00:04:25,438 Prasanna Malaiyandi: right? 92 00:04:25,528 --> 00:04:27,958 That you just didn't know existed. 93 00:04:27,963 --> 00:04:30,928 And you're gonna fix it and be like, oh yeah, now it's like, boom. 94 00:04:31,423 --> 00:04:31,933 W. Curtis Preston: Yeah. 95 00:04:31,993 --> 00:04:34,663 And then it's gonna run like a thousand megabytes per second, and then, 96 00:04:34,663 --> 00:04:36,373 we'll, everything will be amazing. 97 00:04:37,843 --> 00:04:45,503 Um, so, we're gonna talk about some things that are really new in terms of when, when 98 00:04:45,503 --> 00:04:51,503 I first started with backup, we really didn't talk about this concept when I, you 99 00:04:51,503 --> 00:04:53,183 know, was doing backup back in the day. 100 00:04:53,613 --> 00:04:56,283 There's a big thing that happened in the last. 101 00:04:57,033 --> 00:04:59,318 Two decades. 102 00:04:59,323 --> 00:05:05,253 That is one of the biggest reasons why we didn't use to talk about 103 00:05:05,253 --> 00:05:09,543 cybersecurity and backups, and now we talk about it all the time. 104 00:05:09,693 --> 00:05:10,743 Do you know what that is? 105 00:05:10,833 --> 00:05:11,253 Persona? 106 00:05:11,478 --> 00:05:11,898 Prasanna Malaiyandi: No, 107 00:05:11,973 --> 00:05:12,063 W. Curtis Preston: It's 108 00:05:12,063 --> 00:05:12,813 one change. 109 00:05:14,043 --> 00:05:14,433 No. 110 00:05:16,143 --> 00:05:19,863 One seismic change in backups in the last 20 years. 111 00:05:21,348 --> 00:05:22,398 Prasanna Malaiyandi: Dis space backups. 112 00:05:22,413 --> 00:05:25,653 W. Curtis Preston: Yes, exactly right. 113 00:05:27,183 --> 00:05:28,473 Tapes weren't hackable. 114 00:05:29,163 --> 00:05:31,503 Tapes were offline by design. 115 00:05:31,833 --> 00:05:32,073 Right? 116 00:05:32,103 --> 00:05:34,173 Ooh, that rhymes anytime. 117 00:05:34,713 --> 00:05:35,673 Um, yeah. 118 00:05:35,673 --> 00:05:39,093 So tapes were, tapes were not sitting there just files ready 119 00:05:39,093 --> 00:05:40,503 to be deleted or encrypted. 120 00:05:40,503 --> 00:05:46,083 Tapes were inherently hard, even if they were online, even if, if they were in a 121 00:05:46,083 --> 00:05:48,243 tape library, they were hard to get to. 122 00:05:48,813 --> 00:05:49,593 And then. 123 00:05:49,998 --> 00:05:53,118 Uh, over, you know, and, and if they were offline, if they were sitting, 124 00:05:53,118 --> 00:05:56,478 you know, in a, a tape vaulting vendor, then they were really hard to get to. 125 00:05:57,048 --> 00:06:02,868 But the, um, what happened when we went to dis backups, we 126 00:06:03,558 --> 00:06:05,718 basically made backups hackable. 127 00:06:06,438 --> 00:06:11,058 And so that, and, and I don't think, in fact, I know, I know. 128 00:06:11,058 --> 00:06:13,308 We really didn't take that through back when we were. 129 00:06:13,323 --> 00:06:13,893 Prasanna Malaiyandi: Unintended. 130 00:06:14,988 --> 00:06:17,148 W. Curtis Preston: Yeah, we were just thinking about all the problems 131 00:06:17,148 --> 00:06:19,758 that we were having with tape, and we wanted to fix those problems. 132 00:06:20,148 --> 00:06:23,538 And so we went with this space backups to fix those problems. 133 00:06:24,078 --> 00:06:27,558 And, uh, and remember, I will, we'll, we'll always say this. 134 00:06:27,558 --> 00:06:28,908 It wasn't, the tape was too slow. 135 00:06:28,908 --> 00:06:30,138 It was, the tape was too fast. 136 00:06:30,138 --> 00:06:33,258 And we can, you know, if you don't know what I'm talking about, then you need 137 00:06:33,263 --> 00:06:34,818 to, you need to listen to more episodes. 138 00:06:35,908 --> 00:06:41,338 And we've, we've come a long way, but for a really long time backwards, were 139 00:06:41,338 --> 00:06:46,198 just sitting there as files either in a, in a regular, uh, you know, 140 00:06:46,198 --> 00:06:51,268 direct attached file system or an NFS or SMB attached file system that were 141 00:06:51,268 --> 00:06:54,988 directly deletable by a privileged user. 142 00:06:55,468 --> 00:07:00,148 Uh, as long as you, you know, as long as you had access to that drive, we've. 143 00:07:00,943 --> 00:07:04,393 Hopefully in most cases we've gotten away from that. 144 00:07:04,873 --> 00:07:09,103 I think still many people go and buy a, you know, pick your favorite backup, 145 00:07:09,343 --> 00:07:12,763 you know, disc vendor, and they just mount it as a backup drive and then 146 00:07:12,763 --> 00:07:17,023 start backing up to it, and it's, it's, it's eco back slash backups. 147 00:07:17,953 --> 00:07:20,803 And which just like screams, come delete me. 148 00:07:21,463 --> 00:07:27,913 And you know, I, I, I know people still do that and they shouldn't do that. 149 00:07:27,913 --> 00:07:30,823 They should ask their backup vendor, Hey, is there a way that I can 150 00:07:30,823 --> 00:07:34,903 transfer data to my dis space backup storage without it showing up as a, 151 00:07:35,203 --> 00:07:39,403 as a dry letter on my uh, system or a mounted drive on my Unix system? 152 00:07:40,003 --> 00:07:46,663 But that is a huge change in security of backups. 153 00:07:47,368 --> 00:07:47,668 Prasanna Malaiyandi: Yeah. 154 00:07:48,208 --> 00:07:49,168 No, I agree. 155 00:07:49,168 --> 00:07:51,748 It made life easier, right? 156 00:07:51,753 --> 00:07:55,678 You could now do all the instant access, quickly, restore your data, do all the 157 00:07:55,678 --> 00:07:59,003 test in dev, but there was a fatal flaw. 158 00:07:59,983 --> 00:08:02,563 W. Curtis Preston: Yeah, it made life easier for everybody. 159 00:08:03,123 --> 00:08:03,413 Prasanna Malaiyandi: Yeah. 160 00:08:04,783 --> 00:08:05,593 W. Curtis Preston: Yeah, absolutely. 161 00:08:05,593 --> 00:08:07,033 And then replicated backups, right? 162 00:08:07,033 --> 00:08:10,633 We had replicated backups, and then if you, you know, if you have a, a, 163 00:08:10,813 --> 00:08:13,783 you know, a screw up in your backup system, it, the replication just makes 164 00:08:13,783 --> 00:08:15,313 the screw up, uh, more effective. 165 00:08:15,673 --> 00:08:16,003 Right. 166 00:08:16,088 --> 00:08:16,378 Prasanna Malaiyandi: Yeah. 167 00:08:17,413 --> 00:08:20,683 W. Curtis Preston: Yeah, so this is why we have to talk about immutability. 168 00:08:21,233 --> 00:08:24,833 Um, but before we talk about that, we'll talk about some things 169 00:08:24,838 --> 00:08:25,793 that haven't changed at all. 170 00:08:25,793 --> 00:08:30,053 So the thing I want to talk about is this concept of immutable 171 00:08:30,053 --> 00:08:33,923 backups, and that really wasn't. 172 00:08:35,378 --> 00:08:37,718 Something we talked about back in the day. 173 00:08:38,683 --> 00:08:42,253 Prasanna Malaiyandi: Well, I think a lot of it back in the day was your backups 174 00:08:42,253 --> 00:08:44,473 were kind of already immutable, right? 175 00:08:45,343 --> 00:08:46,213 If you were, 176 00:08:47,288 --> 00:08:47,408 W. Curtis Preston: the, 177 00:08:47,413 --> 00:08:49,093 Prasanna Malaiyandi: were going off tape. 178 00:08:49,323 --> 00:08:49,613 W. Curtis Preston: Yeah. 179 00:08:50,603 --> 00:08:54,353 Prasanna Malaiyandi: Right then you didn't have to worry about the scenarios 180 00:08:54,353 --> 00:08:56,783 that led you to need immutable backups. 181 00:08:56,783 --> 00:08:57,203 W. Curtis Preston: Yeah. 182 00:08:57,203 --> 00:08:58,313 That, that's really true. 183 00:08:58,318 --> 00:09:00,413 And that, and that's one of the things that we're gonna talk about, right? 184 00:09:00,893 --> 00:09:04,013 I, I would say the biggest reason was that no one was attacking the backups. 185 00:09:04,013 --> 00:09:05,603 No one was trying to attack the backups. 186 00:09:06,083 --> 00:09:11,118 And the, the worst things that we were worried about were things like I. 187 00:09:11,288 --> 00:09:15,278 Um, we were worried about a natural disaster or a terrorist 188 00:09:15,278 --> 00:09:18,878 attack, taking out the entire place where backups were stored. 189 00:09:18,878 --> 00:09:23,558 So we needed to make sure to have a, an air gap, a physical separation between 190 00:09:23,798 --> 00:09:27,608 the, the backups and, uh, you know, and the, the copy of the backups, right? 191 00:09:28,208 --> 00:09:29,018 We weren't. 192 00:09:29,183 --> 00:09:30,353 Prasanna Malaiyandi: on a tropical island. 193 00:09:30,623 --> 00:09:31,793 W. Curtis Preston: Exactly right. 194 00:09:31,973 --> 00:09:32,693 Exactly. 195 00:09:33,203 --> 00:09:37,253 And so we weren't really, we, we, we weren't at all. 196 00:09:37,283 --> 00:09:40,433 I, I could just, just absolutely tell you that we were not at all 197 00:09:40,493 --> 00:09:44,603 ever thinking about a cyber attack. 198 00:09:45,083 --> 00:09:47,903 Trying to, I, I don't even think we had that term. 199 00:09:47,908 --> 00:09:52,313 I mean, we had the internet, but we didn't really have the, the term cyber attack. 200 00:09:53,318 --> 00:09:55,598 Um, at least that I remember. 201 00:09:55,598 --> 00:09:57,578 We certainly never used it in backup circles, 202 00:09:58,403 --> 00:10:00,233 Prasanna Malaiyandi: Or it was different, right? 203 00:10:00,233 --> 00:10:03,173 Because you weren't necessarily worried about people stealing 204 00:10:03,173 --> 00:10:05,153 your data or things like that. 205 00:10:05,423 --> 00:10:08,273 I remember you were the one who recommended me to read 206 00:10:08,273 --> 00:10:09,263 the Cuckoo's test, right? 207 00:10:09,268 --> 00:10:12,353 And so those were the types of attacks that you sort of 208 00:10:12,358 --> 00:10:14,993 saw back in those days, right? 209 00:10:14,993 --> 00:10:18,413 Stealing resources, not necessarily stealing your data 210 00:10:19,013 --> 00:10:20,573 W. Curtis Preston: Yeah, cuckoo egg, by the way. 211 00:10:21,083 --> 00:10:21,653 Prasanna Malaiyandi: Or sorry, cuckoo. 212 00:10:21,803 --> 00:10:24,353 W. Curtis Preston: which by the way, if, if, if you're out there and you don't 213 00:10:24,353 --> 00:10:26,003 know what we're talking about, great book. 214 00:10:26,243 --> 00:10:29,393 Um, uh, and it's a relatively short read. 215 00:10:29,393 --> 00:10:36,173 It's a great book to read, and, and it's a true story that, you know, involves 216 00:10:36,173 --> 00:10:38,183 a, a cyber attack against a university. 217 00:10:38,273 --> 00:10:38,813 It's great. 218 00:10:40,028 --> 00:10:43,748 So before we talk about immutable backups, I want to talk about some 219 00:10:43,748 --> 00:10:49,358 of the things that really haven't changed that are also, um, impediments 220 00:10:49,358 --> 00:10:51,128 to having a safe backup system. 221 00:10:51,848 --> 00:11:00,308 And the first is the fact that backups never were, never have been, and never 222 00:11:00,308 --> 00:11:03,488 will be the sexy part of the data center. 223 00:11:05,948 --> 00:11:09,548 Prasanna Malaiyandi: Yeah, no, I a hundred percent agree. 224 00:11:09,548 --> 00:11:11,318 No one wants to do backups. 225 00:11:11,963 --> 00:11:14,273 No one wants to spend the money on backups, right? 226 00:11:14,273 --> 00:11:17,813 Everyone wants to be like, Ooh, I need to do, given the recent trend 227 00:11:17,813 --> 00:11:21,923 these days, I wanna focus on AI and do this and that and all the rest. 228 00:11:21,928 --> 00:11:27,323 But no one wants to focus on how do I protect it other than people like Curtis. 229 00:11:28,628 --> 00:11:30,728 W. Curtis Preston: Other than a few, few weird people. 230 00:11:31,098 --> 00:11:33,498 I would say that it's a, there were a lot of reasons. 231 00:11:33,498 --> 00:11:37,908 The biggest reason is easily that it is a very difficult job. 232 00:11:37,908 --> 00:11:42,168 We, we started out this, this episode talking about how 233 00:11:42,168 --> 00:11:43,938 sometimes, how difficult it can be. 234 00:11:44,823 --> 00:11:46,983 You are invisible or you're in trouble. 235 00:11:47,073 --> 00:11:50,733 No one, no one remembers the millions of backups that you got, right? 236 00:11:50,733 --> 00:11:52,923 They only remember the one restore that you got wrong. 237 00:11:53,703 --> 00:11:58,053 And so nobody wants to have that pressure on themselves. 238 00:11:58,683 --> 00:12:04,323 And so they, nobody wants to raise their hand in, you know, in the, 239 00:12:04,323 --> 00:12:07,593 uh, in the room to say, Hey, I'll, I'll be in charge of backups. 240 00:12:07,983 --> 00:12:09,753 So this is true in terms of. 241 00:12:10,338 --> 00:12:13,638 Um, it's not the sexy part of the data center, which means nobody 242 00:12:13,638 --> 00:12:15,888 wants to do it, number th number one. 243 00:12:15,888 --> 00:12:21,318 Number two, nobody wants to give that part of the data center any money 244 00:12:21,558 --> 00:12:27,273 number, and then number three I'd say is that as a result, it goes ignored. 245 00:12:29,928 --> 00:12:31,008 Prasanna Malaiyandi: Oh, I agree with that. 246 00:12:31,008 --> 00:12:35,418 I think there was a study, I can't remember when it was that. 247 00:12:35,748 --> 00:12:39,648 What percentage of an IT budget do you think goes towards backup? 248 00:12:41,613 --> 00:12:42,993 W. Curtis Preston: It's a very small portion. 249 00:12:44,358 --> 00:12:46,218 Prasanna Malaiyandi: I think it was less than a percent. 250 00:12:46,653 --> 00:12:47,013 W. Curtis Preston: Yeah. 251 00:12:47,433 --> 00:12:47,673 Yeah. 252 00:12:47,673 --> 00:12:48,543 That sounds about right. 253 00:12:49,413 --> 00:12:49,623 Yeah. 254 00:12:49,623 --> 00:12:51,603 So it, it goes, it goes ignored. 255 00:12:51,603 --> 00:12:52,173 It goes ignored. 256 00:12:52,173 --> 00:12:54,513 From a budget perspective, it goes ignored from. 257 00:12:55,518 --> 00:12:58,128 A cybersecurity perspective, which is really what we're talking about. 258 00:12:58,128 --> 00:13:00,618 It's what we've been talking about the last several episodes is, is 259 00:13:00,618 --> 00:13:02,328 cybersecurity and, and backups. 260 00:13:02,718 --> 00:13:07,578 And it, it means that it doesn't get put front of the line when we start 261 00:13:07,578 --> 00:13:09,108 talking about putting out patches. 262 00:13:09,108 --> 00:13:12,048 It doesn't get put front of the line in terms of looking 263 00:13:12,048 --> 00:13:13,848 for trends that could be scary. 264 00:13:14,328 --> 00:13:17,988 Um, it's just, it literally just goes ignored. 265 00:13:18,528 --> 00:13:21,108 Um, and that, and that's a really scary thing. 266 00:13:21,108 --> 00:13:21,138 I. 267 00:13:22,518 --> 00:13:26,478 Prasanna Malaiyandi: Yeah, and it's not even thought of initially when you're, 268 00:13:26,483 --> 00:13:31,878 I know we've talked on the episodes that when you are deploying a new application, 269 00:13:31,878 --> 00:13:34,848 you should be thinking about backup or at least have the backup person in the room. 270 00:13:34,848 --> 00:13:37,368 But sometimes things develop so quickly that. 271 00:13:38,163 --> 00:13:39,753 Backup is an afterthought, right? 272 00:13:40,353 --> 00:13:42,363 And sometimes it's very painful. 273 00:13:43,128 --> 00:13:44,148 W. Curtis Preston: Yeah, very much so. 274 00:13:44,388 --> 00:13:50,838 I can remember, I can remember one example of that when we bought, uh, the comp, the 275 00:13:50,838 --> 00:13:54,078 bank that I worked at bought a, an HP. 276 00:13:55,293 --> 00:13:58,953 Was it 1000 I think was the HPT 1000. 277 00:13:58,953 --> 00:13:59,583 That's what it was. 278 00:13:59,883 --> 00:14:06,603 It was a T 1000 and the, it came in and it had, um, it was huge. 279 00:14:06,663 --> 00:14:07,923 It was, are you ready? 280 00:14:08,193 --> 00:14:09,753 A hundred gigabytes of data. 281 00:14:09,963 --> 00:14:11,073 It was huge. 282 00:14:11,298 --> 00:14:11,738 Prasanna Malaiyandi: Humongous. 283 00:14:11,883 --> 00:14:12,778 W. Curtis Preston: It was humongous. 284 00:14:13,053 --> 00:14:14,678 Prasanna Malaiyandi: Now your phone has more than that. 285 00:14:14,988 --> 00:14:17,868 W. Curtis Preston: Prior to this server or prior to this server showing 286 00:14:17,868 --> 00:14:21,048 up at our data center, the biggest server we had was like four, and then 287 00:14:21,048 --> 00:14:26,058 suddenly we had a single server that was 25 times bigger than the next 288 00:14:26,058 --> 00:14:28,038 biggest server, and it came with. 289 00:14:28,383 --> 00:14:35,013 A DDS drive, which was a, a, um, four gigabyte drive. 290 00:14:35,403 --> 00:14:40,323 And, and I, I just remember asking like, who, who's gonna stand here 291 00:14:40,323 --> 00:14:44,733 all night long and swap tapes 25 times to get to get it back up? 292 00:14:45,063 --> 00:14:48,243 And that was, and, and nobody had knew, nobody had even thought of that. 293 00:14:48,243 --> 00:14:50,913 That's the point that nobody had even thought, well, maybe we should 294 00:14:50,913 --> 00:14:54,693 have enough backup infrastructure to back up this new gigantic server. 295 00:14:55,418 --> 00:14:59,588 And that was, by the way, how I got the, my first justification to 296 00:14:59,588 --> 00:15:03,788 buy my first, uh, tape robot, which was the first time I sent money 297 00:15:03,788 --> 00:15:05,498 to Spectral Logic back in the day. 298 00:15:06,618 --> 00:15:08,748 Prasanna Malaiyandi: You weren't just sitting there all night swapping tapes. 299 00:15:08,748 --> 00:15:09,258 Curtis. 300 00:15:09,278 --> 00:15:10,538 W. Curtis Preston: definitely not gonna do that. 301 00:15:10,598 --> 00:15:11,438 Definitely not. 302 00:15:11,528 --> 00:15:15,608 Um, so yeah, so that, that problem's not a new problem. 303 00:15:15,668 --> 00:15:19,028 Uh, the other thing, and as a result of all of those things that we just 304 00:15:19,033 --> 00:15:23,288 talked about, a very common thing for. 305 00:15:24,123 --> 00:15:28,653 Computing infrastructure or for computing environments to do is 306 00:15:28,653 --> 00:15:32,463 to give backups to the new person, 307 00:15:34,803 --> 00:15:35,283 right? 308 00:15:35,913 --> 00:15:37,953 Uh, it was the first job I ever got. 309 00:15:38,013 --> 00:15:40,863 I, I was just editing the, the podcast. 310 00:15:40,863 --> 00:15:44,043 So with Dwayne Lalo, he said it was the first job he ever got. 311 00:15:44,263 --> 00:15:48,793 And the problem with that really is you have this really, really important, 312 00:15:48,823 --> 00:15:53,623 really secure, or what should be a really secure job, and you're handing 313 00:15:53,623 --> 00:15:58,723 it to the least experienced member of the, of the, you know, the team. 314 00:15:58,933 --> 00:15:59,383 Right. 315 00:15:59,888 --> 00:16:02,953 I, I mean, I, I had been at the bank, 316 00:16:05,053 --> 00:16:07,093 you know, essentially a week. 317 00:16:07,153 --> 00:16:08,773 I, I had been, I had been. 318 00:16:09,388 --> 00:16:12,868 In computers, like, like commercial computers. 319 00:16:12,868 --> 00:16:16,888 And I had been introduced to Unix for a few months. 320 00:16:17,003 --> 00:16:20,248 I, I, I had been at this company, I, I think I had worked at this 321 00:16:20,248 --> 00:16:23,818 company for like two months, uh, prior to going to the bank. 322 00:16:23,938 --> 00:16:27,448 And I had been, you know, I had been given some lessons in Unix 323 00:16:27,453 --> 00:16:32,638 and I had been demonstrated the importance of a really good backup. 324 00:16:32,998 --> 00:16:36,598 Uh, here's, this was the first time in my career when. 325 00:16:36,913 --> 00:16:38,623 Prasanna Malaiyandi: Theoretical versus practical. 326 00:16:38,803 --> 00:16:46,543 W. Curtis Preston: Yeah, so we were a, uh, we were a blended call management 327 00:16:46,543 --> 00:16:52,183 system, which you might refer to as an auto dialer, and that was a, that was a, 328 00:16:52,303 --> 00:16:54,433 a naughty word back in the day, right? 329 00:16:54,433 --> 00:16:55,633 We were not an auto dialer. 330 00:16:55,633 --> 00:16:58,183 We were an intelligent call management system. 331 00:16:59,383 --> 00:17:03,853 And, but it was essentially an Auto Tyler and it had this very fancy 332 00:17:03,853 --> 00:17:05,983 Unix-based computer behind it. 333 00:17:06,943 --> 00:17:12,553 And we would go out and do this, this, it was like this long involved install 334 00:17:12,553 --> 00:17:16,663 because it was a very, this was before VoIP and all that kind of stuff. 335 00:17:17,178 --> 00:17:21,588 We had, you know, we had literally cards that went in the server and those, each 336 00:17:21,593 --> 00:17:23,988 card connected to a POTS line, right? 337 00:17:24,418 --> 00:17:24,718 Prasanna Malaiyandi: Mm-Hmm. 338 00:17:25,038 --> 00:17:27,648 W. Curtis Preston: And so we had this big physical install that we had 339 00:17:27,648 --> 00:17:29,748 to do, and it took about two weeks. 340 00:17:29,838 --> 00:17:34,698 And then, um, you know, you, you sent a couple of SCS on site and I was one 341 00:17:34,698 --> 00:17:39,858 of the sc I was obviously the junior se, and this other guy had been, um, 342 00:17:39,888 --> 00:17:41,988 doing all this configuration and. 343 00:17:42,378 --> 00:17:47,238 The job that we were supposed to do before we left was to pop in a tape, 344 00:17:47,568 --> 00:17:51,258 by the way, a tape, a nine track 345 00:17:51,538 --> 00:17:52,378 Prasanna Malaiyandi: Ginormous. 346 00:17:52,798 --> 00:17:53,158 Yeah. 347 00:17:53,898 --> 00:17:57,738 W. Curtis Preston: and pop in a nine track tape and back up the OS 348 00:17:57,738 --> 00:18:00,468 configuration that we had just finished. 349 00:18:00,738 --> 00:18:04,968 That was the last thing that we were supposed to do, and just 350 00:18:04,968 --> 00:18:08,058 before doing that, my, my compadre. 351 00:18:09,223 --> 00:18:12,823 Said he was looking at the root file system and he noticed that there were 352 00:18:12,823 --> 00:18:17,173 a couple of garbage files in the root file system and one of them started 353 00:18:17,173 --> 00:18:22,723 with, um, question mark and had a bunch of gar dego stuff after it. 354 00:18:22,813 --> 00:18:27,188 So he didn't want to type all of the gar dego, so he typed, um, 355 00:18:27,808 --> 00:18:30,733 RM minus RF question mark, star 356 00:18:32,293 --> 00:18:32,773 Prasanna Malaiyandi: Oh, 357 00:18:33,793 --> 00:18:35,323 W. Curtis Preston: from the root file system. 358 00:18:37,303 --> 00:18:41,683 Wanting to clean up the OS before we took the backup. 359 00:18:43,068 --> 00:18:44,113 Prasanna Malaiyandi: oh geez. 360 00:18:44,863 --> 00:18:46,543 That must have been interesting. 361 00:18:46,753 --> 00:18:48,913 W. Curtis Preston: Yeah, so that was the, that was the day that I 362 00:18:48,913 --> 00:18:50,623 learned the value of a good backup. 363 00:18:53,218 --> 00:18:56,248 The best time to do a backup is yesterday. 364 00:18:56,518 --> 00:18:58,798 The second best time is today. 365 00:18:59,283 --> 00:18:59,633 Prasanna Malaiyandi: today. 366 00:18:59,848 --> 00:19:02,188 W. Curtis Preston: Um, this was definitely the second best time, 367 00:19:02,488 --> 00:19:04,708 so we lost a whole bunch of work. 368 00:19:04,798 --> 00:19:08,878 Uh, and luckily it was not my fault, so I just learned a very valuable lesson. 369 00:19:08,938 --> 00:19:14,098 Anyway, sorry, that was bringing me, bringing me back, back in the day. 370 00:19:14,848 --> 00:19:17,278 Um, so yeah, so. 371 00:19:18,433 --> 00:19:23,773 It's a horrible practice that we put the most junior person in charge of the 372 00:19:23,773 --> 00:19:27,943 backups and that should absolutely stop, but I don't know how to merge those two. 373 00:19:28,213 --> 00:19:33,283 How do you, how do you, how do you properly reward this job that 374 00:19:33,288 --> 00:19:37,693 is really, really important such that someone who isn't junior is 375 00:19:37,693 --> 00:19:38,893 going to want to take the job? 376 00:19:38,893 --> 00:19:39,493 I don't know. 377 00:19:39,673 --> 00:19:41,323 I don't have an answer to that question. 378 00:19:42,763 --> 00:19:43,903 Do you, do you persona. 379 00:19:44,863 --> 00:19:49,183 Prasanna Malaiyandi: Well, no, but it's No, but if you think 380 00:19:49,183 --> 00:19:54,373 about this, this really isn't any different than say A DBA, right? 381 00:19:54,373 --> 00:20:01,243 A database admin who starts off initially does some Basic things, right? 382 00:20:01,243 --> 00:20:06,103 And over time they sort of become like that database architect who sort 383 00:20:06,103 --> 00:20:09,163 of figuring out overall layouts and things like that rather than sort 384 00:20:09,163 --> 00:20:10,813 of just administering the system. 385 00:20:11,443 --> 00:20:14,653 Now the difference is in the case of the DBA, you're probably starting 386 00:20:14,653 --> 00:20:19,393 off in like test and dev instances or like non-critical applications versus 387 00:20:19,393 --> 00:20:22,693 as a backup admin, you're probably starting off on like the critical 388 00:20:22,693 --> 00:20:24,253 infrastructure that you are protecting. 389 00:20:24,598 --> 00:20:25,408 W. Curtis Preston: Yeah. 390 00:20:25,888 --> 00:20:26,218 Yeah. 391 00:20:26,218 --> 00:20:30,598 That's why I'm, I, I'm not sure I agree with your, with your comparison 392 00:20:30,598 --> 00:20:34,438 there because, well, first off, people actually wanna be DBAs. 393 00:20:35,773 --> 00:20:36,013 Prasanna Malaiyandi: Yeah. 394 00:20:36,613 --> 00:20:39,013 Well, and yeah, maybe. 395 00:20:39,013 --> 00:20:42,613 Maybe what you need to do is like a cross training program where you do 396 00:20:42,618 --> 00:20:46,603 a rotation in like a different part of the organization and then you come 397 00:20:46,603 --> 00:20:49,183 into the backup work, because that way at least you're also get more 398 00:20:49,183 --> 00:20:51,043 familiar with tools and other things. 399 00:20:51,103 --> 00:20:51,853 W. Curtis Preston: Right, right. 400 00:20:51,853 --> 00:20:52,273 Yeah. 401 00:20:53,143 --> 00:20:53,383 Yeah. 402 00:20:53,443 --> 00:20:54,583 Prasanna Malaiyandi: That way you're not the junior person. 403 00:20:55,483 --> 00:20:56,833 W. Curtis Preston: exactly, exactly. 404 00:20:56,893 --> 00:20:57,223 Yeah. 405 00:20:57,223 --> 00:21:01,903 You, you need, I, I guess the best thing we want to do here is to make sure. 406 00:21:02,863 --> 00:21:07,423 That you're, that you're making sure that your junior person, that you're giving 407 00:21:07,848 --> 00:21:13,483 the, the backups to has some really good cybersecurity, some basic, at least some 408 00:21:13,483 --> 00:21:15,223 basic cybersecurity training, right? 409 00:21:15,493 --> 00:21:18,163 Because you're handing them in order to do the backups. 410 00:21:18,163 --> 00:21:22,603 They are a very powerful person and they have the ability to 411 00:21:22,603 --> 00:21:24,733 read and write every file. 412 00:21:24,783 --> 00:21:27,873 Right, which means they have the ability to overwrite every 413 00:21:27,873 --> 00:21:29,313 file in the organization. 414 00:21:30,213 --> 00:21:34,053 Uh, even if you're using proper modern backup design that uses role-based 415 00:21:34,053 --> 00:21:37,173 administration and all of that, you're, you're still most likely 416 00:21:37,178 --> 00:21:39,153 handing them the keys to the kingdom. 417 00:21:39,723 --> 00:21:40,293 Uh, 418 00:21:40,408 --> 00:21:41,728 Prasanna Malaiyandi: Is that true though? 419 00:21:41,728 --> 00:21:47,608 Because normally don't, you have sort of help desk restore operators and 420 00:21:47,608 --> 00:21:51,778 then the more senior people are kind of those who are doing like the backup 421 00:21:51,778 --> 00:21:55,918 architecture and the design, you probably have people who are figuring out 422 00:21:56,353 --> 00:21:59,083 W. Curtis Preston: In a large environment, that would be the case, but that's not 423 00:21:59,083 --> 00:22:00,823 the case in most companies, I don't think. 424 00:22:00,853 --> 00:22:04,963 I think it's just one person, actually, in most companies it's a collateral duty. 425 00:22:05,473 --> 00:22:05,863 Right. 426 00:22:06,043 --> 00:22:12,163 So, but let's, so let's get to the meat of the episode here where we 427 00:22:12,163 --> 00:22:14,233 wanna talk about immutable backups. 428 00:22:14,623 --> 00:22:19,213 So first off, just generally speaking, what does immutable mean? 429 00:22:21,553 --> 00:22:23,953 Prasanna Malaiyandi: It basically means you can't change it, 430 00:22:24,403 --> 00:22:24,643 W. Curtis Preston: Yeah. 431 00:22:24,823 --> 00:22:28,423 It's literally, it's, it's, it's, it's a word that's used outside 432 00:22:28,423 --> 00:22:31,243 of it, but within it, yeah. 433 00:22:31,248 --> 00:22:33,253 It means that it cannot be changed 434 00:22:34,153 --> 00:22:35,563 Prasanna Malaiyandi: which also includes deletions. 435 00:22:36,073 --> 00:22:36,493 W. Curtis Preston: yeah. 436 00:22:36,763 --> 00:22:37,423 Changed. 437 00:22:37,693 --> 00:22:37,993 Yeah. 438 00:22:37,993 --> 00:22:38,983 Deletion is a change. 439 00:22:40,033 --> 00:22:40,303 Right. 440 00:22:41,383 --> 00:22:46,783 Um, and the difficulty here. 441 00:22:47,683 --> 00:22:52,693 Is that, that sounds like a binary condition, 442 00:22:53,968 --> 00:22:54,238 Prasanna Malaiyandi: Mm-Hmm. 443 00:22:54,703 --> 00:22:56,503 W. Curtis Preston: Like pregnant or dead. 444 00:22:57,493 --> 00:22:57,823 Right. 445 00:22:59,553 --> 00:23:01,198 Prasanna Malaiyandi: Not the words I would've chosen, but Sure. 446 00:23:01,768 --> 00:23:05,158 W. Curtis Preston: Both pregnancy and death are binary conditions. 447 00:23:05,158 --> 00:23:09,118 You're either pregnant or you're not, or you're dead or alive, right? 448 00:23:09,123 --> 00:23:10,738 These are binary conditions. 449 00:23:11,398 --> 00:23:17,398 Um, immutable should be a binary condition that that is what is implied. 450 00:23:18,118 --> 00:23:20,398 When we say immutable, it can't be changed. 451 00:23:20,398 --> 00:23:21,238 It can't be deleted. 452 00:23:21,238 --> 00:23:22,108 It can't be encrypted. 453 00:23:22,108 --> 00:23:23,158 It can't be modified. 454 00:23:23,163 --> 00:23:25,138 It can't be, I don't know. 455 00:23:25,138 --> 00:23:26,308 I think I covered 'em all. 456 00:23:26,758 --> 00:23:30,403 Prasanna Malaiyandi: Yeah, but there's really a spectrum though. 457 00:23:31,048 --> 00:23:33,448 W. Curtis Preston: it is definitely a spectrum, and that is the 458 00:23:33,448 --> 00:23:36,988 problem, is that immutable. 459 00:23:36,988 --> 00:23:39,418 And that's, that's the thing I wanted to just mention. 460 00:23:39,958 --> 00:23:41,938 So let me, so let me ask you this. 461 00:23:41,938 --> 00:23:48,868 Why, why has immutable backups, why has that really come to the fore, 462 00:23:49,498 --> 00:23:51,748 uh, in the last say, 10 years or so? 463 00:23:52,798 --> 00:23:54,883 Prasanna Malaiyandi: Well, like you were talking about earlier 464 00:23:54,883 --> 00:23:56,833 in the episode, cyber attacks. 465 00:23:57,733 --> 00:24:02,293 Ransomware attacks, they not only go after your production, but they 466 00:24:02,293 --> 00:24:05,293 realize that, hey, you have all this data sitting in your backup systems 467 00:24:05,293 --> 00:24:08,833 that you could use to recover your environment and not pay the ransom. 468 00:24:08,833 --> 00:24:12,763 So instead, we're gonna go attack your backup systems, delete all 469 00:24:12,763 --> 00:24:14,563 your backups so you can't recover. 470 00:24:15,013 --> 00:24:18,253 And then also because they're doing the double extortion where 471 00:24:18,253 --> 00:24:20,023 they also exfiltrate your data. 472 00:24:20,938 --> 00:24:22,198 That's a treasure trove, right? 473 00:24:22,198 --> 00:24:24,388 They don't have to go hit 20 different systems. 474 00:24:24,388 --> 00:24:25,798 If they get to your backup system. 475 00:24:26,248 --> 00:24:29,698 All your data's there and backup systems are really good at restoring data so 476 00:24:29,698 --> 00:24:34,318 they can quickly pull the data out and then blackmail you in case you decide 477 00:24:34,318 --> 00:24:35,578 that you don't want to pay the ransom. 478 00:24:36,073 --> 00:24:36,853 W. Curtis Preston: Yeah, exactly. 479 00:24:36,853 --> 00:24:40,963 Now immutability won't address that second part. 480 00:24:41,083 --> 00:24:42,553 Uh, it is important to me. 481 00:24:42,553 --> 00:24:45,403 Right tho those are, those are the two main concerns that we have when 482 00:24:45,403 --> 00:24:46,693 we talk about the backup system. 483 00:24:47,383 --> 00:24:52,273 And, uh, you should definitely listen to the episode that went live. 484 00:24:52,333 --> 00:24:59,053 Um, let's see, that would've been May 26th now, um, the one where with Dwayne Lalo, 485 00:24:59,058 --> 00:25:04,093 where he talks a bit from the Red Team, the Red Team Cybersecurity episode that. 486 00:25:04,568 --> 00:25:09,908 Um, where he talks about what a treasure trove the backup system is and how he 487 00:25:09,908 --> 00:25:11,528 loves, he's a, you know, he is a red team. 488 00:25:11,528 --> 00:25:16,208 He's a hacker, uh, that, you know, an offensive cybersecurity person. 489 00:25:16,748 --> 00:25:17,708 Uh, he's very offensive. 490 00:25:20,628 --> 00:25:23,263 And, and he loves going after the backup system, right? 491 00:25:23,863 --> 00:25:27,583 When we talk about immutability, we're primarily concerned with 492 00:25:27,673 --> 00:25:31,303 making sure that the backups themselves aren't modified, right. 493 00:25:32,053 --> 00:25:35,563 And which, you know, as we said, you know, and, and. 494 00:25:36,103 --> 00:25:38,053 Like you said, this wasn't a problem. 495 00:25:38,773 --> 00:25:43,183 I don't know, even five years ago, maybe, uh, this wasn't a problem, 496 00:25:43,183 --> 00:25:46,393 but hackers weren't going directly after the backups, but now they 497 00:25:46,393 --> 00:25:51,313 definitely are because they're the best defense we have against ransomware. 498 00:25:51,313 --> 00:25:51,883 Right. 499 00:25:51,883 --> 00:25:51,913 I. 500 00:25:52,063 --> 00:25:52,273 Prasanna Malaiyandi: Yeah. 501 00:25:52,333 --> 00:25:59,653 And and I would say before immutable backups slash immutable archiving was 502 00:25:59,653 --> 00:26:01,693 really intended as a compliance use case. 503 00:26:02,398 --> 00:26:02,668 W. Curtis Preston: Yeah. 504 00:26:02,668 --> 00:26:05,248 And that's what I, that's kind of where, where I wanted to go 505 00:26:05,248 --> 00:26:09,808 with this is let's talk about the different things that people mean 506 00:26:09,808 --> 00:26:11,848 when they say immutable, right? 507 00:26:12,268 --> 00:26:18,718 And prior to cyber security attacks, when we talked about the backup system 508 00:26:18,718 --> 00:26:23,128 or the archive system, which are different things when we talk about the 509 00:26:23,128 --> 00:26:24,868 backup system or the archive system. 510 00:26:25,333 --> 00:26:31,123 Being immutable, what we're primarily, um, wanting to be able to say in a 511 00:26:31,123 --> 00:26:37,093 court of law that this file, this document that I'm presenting in court, 512 00:26:37,723 --> 00:26:39,763 is exactly the same as it was I. 513 00:26:40,418 --> 00:26:42,248 When it was created, right. 514 00:26:42,248 --> 00:26:46,778 This, this email that is the smoking gun we can state. 515 00:26:46,808 --> 00:26:53,558 It is a, it is a faithful reproduction of the original email that the plaintiff 516 00:26:53,558 --> 00:26:59,018 made or the defendant, I get It's, it's a's a faithful reproduction of 517 00:26:59,023 --> 00:27:01,058 the email that the defendant made. 518 00:27:01,753 --> 00:27:04,243 That shows that they were doing the thing that they weren't supposed to be doing. 519 00:27:04,513 --> 00:27:04,843 Right? 520 00:27:05,203 --> 00:27:08,413 And, and so the, so that the, the defense can't say, oh, well 521 00:27:08,413 --> 00:27:10,213 that's a fake email, right? 522 00:27:10,783 --> 00:27:15,883 You're like, no, this is a faithful reproduction of the email that this, 523 00:27:15,883 --> 00:27:18,163 you know, and here's the, here's the chain of custodying, the chain 524 00:27:18,163 --> 00:27:20,533 of technology to get from A to B. 525 00:27:20,893 --> 00:27:26,263 And so, um, you know, it, it's, fine, 526 00:27:26,458 --> 00:27:27,838 Prasanna Malaiyandi: It's as good as the source. 527 00:27:27,838 --> 00:27:28,948 It's as good as the original. 528 00:27:29,308 --> 00:27:29,788 W. Curtis Preston: It's as good. 529 00:27:29,788 --> 00:27:30,328 Yeah, we are. 530 00:27:30,328 --> 00:27:34,948 We are essentially looking directly into the email system from three months 531 00:27:34,948 --> 00:27:36,628 ago or three years ago or whatever. 532 00:27:36,628 --> 00:27:37,138 It's right. 533 00:27:37,768 --> 00:27:41,128 That's what we were primarily concerned with and we are still concerned with that. 534 00:27:42,028 --> 00:27:42,328 Right. 535 00:27:42,388 --> 00:27:45,778 We are still concerned with immutability for that reason from 536 00:27:45,778 --> 00:27:48,173 a legal perspective, but, um. 537 00:27:49,153 --> 00:27:57,793 What we're now more concerned with is that someone could go in and modify the backups 538 00:27:57,793 --> 00:27:59,833 in such a way that they encrypt them. 539 00:28:00,463 --> 00:28:02,923 They, you know, they, they delete them. 540 00:28:03,373 --> 00:28:06,103 Uh, they could actually modify them. 541 00:28:07,183 --> 00:28:10,693 I, I don't, I don't, I don't know of any scenario in this case, but 542 00:28:10,693 --> 00:28:13,963 they could potentially go in and selectively modify them to mess 543 00:28:13,963 --> 00:28:16,363 with people for whatever reason. 544 00:28:17,503 --> 00:28:20,713 Um, but that shouldn't be able to be the case either. 545 00:28:20,713 --> 00:28:21,103 Right? 546 00:28:21,883 --> 00:28:25,633 Prasanna Malaiyandi: Yeah, and that just seems like a lot of work to go selectively 547 00:28:25,638 --> 00:28:30,748 modify unless you're trying to do something sneakily without being detected. 548 00:28:32,143 --> 00:28:32,833 W. Curtis Preston: Right, right. 549 00:28:33,043 --> 00:28:41,953 Um, so that's the, that's one way that backup systems and archive systems use 550 00:28:41,953 --> 00:28:43,603 the term immutability is they say, I. 551 00:28:44,263 --> 00:28:47,803 We can say that, you know, these two things were, you know, are the 552 00:28:47,803 --> 00:28:48,163 same. 553 00:28:48,795 --> 00:28:51,915 So the, the next thing that I see is that. 554 00:28:52,800 --> 00:28:58,500 I see vendors that say, well, we have a A, right? 555 00:28:58,710 --> 00:29:02,820 Only we have an append only file system that once backups are 556 00:29:02,820 --> 00:29:06,570 written into our append only file system, they cannot be modified. 557 00:29:06,570 --> 00:29:08,640 Have you seen backup folks saying that I. 558 00:29:09,430 --> 00:29:12,970 Prasanna Malaiyandi: Yeah, though I've seen some storage vendors, right? 559 00:29:12,970 --> 00:29:17,800 You have companies who do backup appliances like Rubrik who say, 560 00:29:17,800 --> 00:29:21,760 yeah, once you write your backup to our system, right, you can't 561 00:29:21,760 --> 00:29:23,350 go and overwrite that data again. 562 00:29:24,040 --> 00:29:24,390 W. Curtis Preston: Right, 563 00:29:24,610 --> 00:29:25,780 Prasanna Malaiyandi: of a right only and. 564 00:29:26,395 --> 00:29:30,015 Having worked at a storage company which was famous for a write 565 00:29:30,015 --> 00:29:34,030 Anywhere file layout, I could see the benefits of doing it that way. 566 00:29:34,830 --> 00:29:35,190 W. Curtis Preston: Right. 567 00:29:36,240 --> 00:29:36,750 Yeah. 568 00:29:37,020 --> 00:29:42,020 And, and what this says is that it stops direct attacks against 569 00:29:42,230 --> 00:29:44,420 the backup storage itself. 570 00:29:45,110 --> 00:29:48,920 That if I, if I'm able to get into the file system, if I, if I'm able 571 00:29:49,070 --> 00:29:52,700 to get into the operating system of that backup system, I wouldn't 572 00:29:52,705 --> 00:29:58,070 be able to modify or delete the backup files in order to, you know. 573 00:29:59,210 --> 00:30:00,950 Well, well essentially delete them. 574 00:30:00,950 --> 00:30:04,220 I wouldn't be able to encrypt or delete them because this is 575 00:30:04,220 --> 00:30:05,750 this append only file system. 576 00:30:07,640 --> 00:30:10,490 Prasanna Malaiyandi: Which is interesting because I do wonder 577 00:30:10,490 --> 00:30:14,630 about certain scenarios like, Hey, I'm running low on space. 578 00:30:14,635 --> 00:30:17,840 I need to go delete a whole bunch of backups or backup sets 579 00:30:18,080 --> 00:30:19,460 in order to free things up. 580 00:30:20,660 --> 00:30:21,050 How? 581 00:30:21,050 --> 00:30:21,950 What do I do? 582 00:30:21,955 --> 00:30:23,060 Am I just stuck? 583 00:30:24,620 --> 00:30:26,390 W. Curtis Preston: Yeah, that's gonna, that's gonna be 584 00:30:26,450 --> 00:30:27,770 depending on the product, right? 585 00:30:27,770 --> 00:30:29,965 So the question, my, my question is. 586 00:30:31,145 --> 00:30:34,925 With the, the product in question and that this isn't necessarily Rubrik, 587 00:30:34,925 --> 00:30:39,095 this is just, I'm, it's just the, this is the question that we're asking 588 00:30:39,665 --> 00:30:44,825 is, I, I understand you're saying that it's an append only file system. 589 00:30:45,155 --> 00:30:49,055 What about the system that's controlling the backups itself? 590 00:30:49,325 --> 00:30:53,915 What protection is there in, in that system? 591 00:30:53,915 --> 00:30:57,605 Meaning if I am a privileged user. 592 00:30:58,340 --> 00:31:01,040 Am I able to, um. 593 00:31:01,940 --> 00:31:02,510 Delete 594 00:31:02,510 --> 00:31:03,530 backups, right? 595 00:31:03,530 --> 00:31:07,460 Am I able, am I able to prematurely expire backups? 596 00:31:07,460 --> 00:31:10,160 Because even with immutable storage, they can have, it can 597 00:31:10,165 --> 00:31:11,600 have an expiration date, right? 598 00:31:11,930 --> 00:31:14,420 So you can say this, this, these backups are immutable for 90 599 00:31:14,420 --> 00:31:15,770 days and then they can expire. 600 00:31:16,310 --> 00:31:23,360 But if I am, you know, the super high cloud administrator of this backup system, 601 00:31:23,720 --> 00:31:26,600 am I able to then expire backups early? 602 00:31:26,600 --> 00:31:29,060 And if you are, then. 603 00:31:29,795 --> 00:31:33,375 That is, again, this is why I say immutable backups. 604 00:31:33,765 --> 00:31:40,935 Are a spectrum because the, the, the question is, if I am, you know, 605 00:31:40,935 --> 00:31:45,915 super high up in the echelon, if I make a phone call to this person, is 606 00:31:45,915 --> 00:31:48,075 there a way to expire older backups? 607 00:31:48,435 --> 00:31:48,945 I, I would 608 00:31:48,945 --> 00:31:54,555 also ask, is there a privilege to count in the operating system that allows 609 00:31:54,555 --> 00:31:56,655 me to essentially expire all backups? 610 00:31:57,120 --> 00:31:57,410 Prasanna Malaiyandi: Yeah. 611 00:31:57,690 --> 00:32:00,990 W. Curtis Preston: that, that, that is the case in some backup systems, that even 612 00:32:00,990 --> 00:32:05,130 though they're immutable, if I have access to this one privileged account, I can wipe 613 00:32:05,130 --> 00:32:06,750 out all backups with a single command. 614 00:32:07,725 --> 00:32:11,145 Prasanna Malaiyandi: Or reset the system back to a factory state, right? 615 00:32:11,145 --> 00:32:12,375 That's sometimes common. 616 00:32:12,870 --> 00:32:13,230 W. Curtis Preston: Right. 617 00:32:13,350 --> 00:32:13,710 Right. 618 00:32:14,805 --> 00:32:18,675 Um, so that, that is another one that would be called immutable. 619 00:32:18,825 --> 00:32:19,245 Right. 620 00:32:19,725 --> 00:32:24,195 Um, and, and I'm not saying that any of these aren't immutable, I'm just 621 00:32:24,195 --> 00:32:27,135 saying that there are a whole bunch of things that are called immutable. 622 00:32:28,575 --> 00:32:28,965 right. 623 00:32:29,115 --> 00:32:29,505 right. 624 00:32:29,610 --> 00:32:29,900 Prasanna Malaiyandi: Yeah. 625 00:32:31,155 --> 00:32:34,965 W. Curtis Preston: Um, another one, and, uh, and this is 626 00:32:34,965 --> 00:32:37,155 where, when, and, and again. 627 00:32:37,815 --> 00:32:39,375 This is like good, better, best, right? 628 00:32:40,245 --> 00:32:42,345 Anything would be better than backups. 629 00:32:42,345 --> 00:32:47,325 Sitting on just an open Windows file system would zero security checks, right? 630 00:32:47,325 --> 00:32:48,555 Anything would be better than that, 631 00:32:49,365 --> 00:32:49,875 and, 632 00:32:50,070 --> 00:32:52,110 Prasanna Malaiyandi: I think I have a worse scenario. 633 00:32:52,395 --> 00:32:52,755 W. Curtis Preston: okay. 634 00:32:53,880 --> 00:32:55,855 Prasanna Malaiyandi: A window system open to the internet. 635 00:32:57,945 --> 00:32:58,455 W. Curtis Preston: Yeah. 636 00:32:59,955 --> 00:33:00,195 Yeah. 637 00:33:00,585 --> 00:33:02,085 With, uh, with, uh, 638 00:33:02,765 --> 00:33:03,165 Prasanna Malaiyandi: RDP 639 00:33:03,255 --> 00:33:05,265 W. Curtis Preston: Um, R-D-P-R-D-P enabled. 640 00:33:05,445 --> 00:33:05,835 Yeah. 641 00:33:07,245 --> 00:33:12,975 Uh, the, um, the next, you know, one of the most surprising things when, 642 00:33:12,975 --> 00:33:19,815 when I listened to the episode with Dwayne again was when he said that 643 00:33:19,935 --> 00:33:23,205 so many times that they, they, they go in with the default password. 644 00:33:24,075 --> 00:33:24,855 And that it works. 645 00:33:24,945 --> 00:33:26,085 That's just, that just hurts. 646 00:33:26,085 --> 00:33:30,135 Prasanna Malaiyandi: The next one is really about something. 647 00:33:30,405 --> 00:33:31,845 So not the best, right? 648 00:33:31,845 --> 00:33:35,115 But better than the Windows system, right? 649 00:33:35,115 --> 00:33:35,775 Which is good. 650 00:33:35,775 --> 00:33:42,735 Which is having a server that has been hardened and locked down and secured. 651 00:33:43,590 --> 00:33:48,180 Such that you are protecting from most common issues and 652 00:33:48,180 --> 00:33:52,080 security vulnerabilities, but it may not still be perfect. 653 00:33:52,380 --> 00:33:56,550 And in this example, right, we had Dave Russell and Rick Manover 654 00:33:56,550 --> 00:34:00,630 from Veeam over talking about the hardened backup repository. 655 00:34:01,200 --> 00:34:04,920 I think is the official name, which is basically a server which they 656 00:34:04,920 --> 00:34:08,850 locked down, which allows you to write Veeam backups to it, and they have an 657 00:34:08,850 --> 00:34:13,800 immutable flag in the Linux repository such that the backups cannot just be 658 00:34:13,805 --> 00:34:15,750 normally deleted, casually deleted. 659 00:34:15,755 --> 00:34:16,110 I would say. 660 00:34:16,665 --> 00:34:17,115 W. Curtis Preston: Right. 661 00:34:17,115 --> 00:34:17,475 Yeah. 662 00:34:17,475 --> 00:34:21,855 They can't be deleted by anyone except for root, right, and they can't even 663 00:34:21,855 --> 00:34:23,625 be accidentally deleted by root. 664 00:34:24,135 --> 00:34:28,425 The root would have a person with root would have to unset 665 00:34:28,455 --> 00:34:32,085 the immutability flag on the file and then delete the file, right? 666 00:34:32,505 --> 00:34:32,895 So. 667 00:34:33,540 --> 00:34:37,950 Uh, this is what, you know, I agree with you that it, it is better than, 668 00:34:38,400 --> 00:34:39,930 you know, having a Windows file system. 669 00:34:39,930 --> 00:34:42,960 I mean, first off, it's just better in that it's Linux and not Windows. 670 00:34:42,960 --> 00:34:45,240 And I don't, I don't just mean that Linux is better than Windows. 671 00:34:45,240 --> 00:34:47,550 What I mean is it's something different, right? 672 00:34:48,000 --> 00:34:51,900 And then the, uh, 'cause I definitely do not think that Linux is. 673 00:34:52,360 --> 00:34:52,870 Perfect. 674 00:34:52,900 --> 00:34:53,230 Right. 675 00:34:53,980 --> 00:34:58,420 But the, uh, I do still, like the Linux way of everything is off 676 00:34:58,420 --> 00:35:01,270 by default versus the Windows way where everything is on by default. 677 00:35:01,810 --> 00:35:04,000 Although I think they've gotten better at that over the years. 678 00:35:04,810 --> 00:35:07,240 The um, um. 679 00:35:08,190 --> 00:35:12,720 It is important to understand that if someone gains root, then they're 680 00:35:12,720 --> 00:35:15,060 able to, uh, modify these files. 681 00:35:15,300 --> 00:35:17,460 Now they do a really good job. 682 00:35:17,460 --> 00:35:21,870 I think, uh, specifically with the Veeam repository, they do a really good job of. 683 00:35:22,540 --> 00:35:26,560 Giving you a list of things that you need to do where it's like essentially. 684 00:35:26,620 --> 00:35:30,940 But once you've configured this system, you can configure it, for example, to 685 00:35:30,940 --> 00:35:36,100 automatically update itself with security patches and that you don't need to be 686 00:35:36,100 --> 00:35:37,810 logging into it regularly as route. 687 00:35:37,810 --> 00:35:41,290 So you can like super secure that, um, thing. 688 00:35:41,290 --> 00:35:44,710 So that if, if anyone ever does log into it as rude, it can set off all 689 00:35:44,710 --> 00:35:48,630 sorts of, uh, bells and whistles so that you can know that something. 690 00:35:48,819 --> 00:35:49,600 bad is going on. 691 00:35:50,380 --> 00:35:52,420 Prasanna Malaiyandi: Yeah, and I know we've talked in the past about 692 00:35:52,420 --> 00:35:54,910 like patch management, go do your patch management all the time. 693 00:35:54,910 --> 00:35:58,510 And having an automated process and keeping it up to date I think 694 00:35:58,960 --> 00:36:01,660 is a great feature that that has. 695 00:36:02,140 --> 00:36:02,530 W. Curtis Preston: Right. 696 00:36:03,910 --> 00:36:07,330 And then I would say, what's the, the next, and I'm, I'm gonna 697 00:36:07,330 --> 00:36:12,130 put this up in terms of the best as far as I can see, is that I. 698 00:36:12,730 --> 00:36:18,310 Here's an immutable system where backups cannot be modified even by a privileged 699 00:36:18,310 --> 00:36:20,440 account, even by somebody with root. 700 00:36:21,160 --> 00:36:25,480 Um, the, and there would be two types of systems like that. 701 00:36:25,510 --> 00:36:29,320 One would be a backup, a, a storage system that could be on-prem. 702 00:36:30,190 --> 00:36:35,170 Another would be something like a cloud system like S3 with the 703 00:36:35,170 --> 00:36:36,970 immutable flag turned onto that. 704 00:36:37,330 --> 00:36:39,070 And I would, I would actually put. 705 00:36:39,430 --> 00:36:43,660 Them in terms of like the, the on-prem system isn't as secure as 706 00:36:43,660 --> 00:36:45,340 the cloud system, in my opinion. 707 00:36:45,850 --> 00:36:46,090 Why? 708 00:36:46,090 --> 00:36:46,900 And why would that be? 709 00:36:46,900 --> 00:36:49,690 Prasanna Malaiyandi: because you can physically access the system and 710 00:36:50,050 --> 00:36:51,970 yank the drives and all the rest. 711 00:36:52,180 --> 00:36:52,360 W. Curtis Preston: Yeah. 712 00:36:52,360 --> 00:36:56,110 Physical access trumps, all right. 713 00:36:56,140 --> 00:36:59,320 So it's funny, a lot of people are like, oh, the cloud is insecure. 714 00:36:59,320 --> 00:37:01,570 And I'm like, well, physical system's on-prem or insecure. 715 00:37:02,200 --> 00:37:06,130 Right, because I can easily go up to that and I can do bad things to 716 00:37:06,130 --> 00:37:07,840 it as long as I know where it's at. 717 00:37:08,995 --> 00:37:09,265 Prasanna Malaiyandi: Yeah. 718 00:37:10,285 --> 00:37:14,935 And even with the physical storage systems, I know at NetApp and even 719 00:37:14,935 --> 00:37:17,365 with AWS's Object Lock, right? 720 00:37:17,365 --> 00:37:20,245 They do have those two different modes, compliance and governance. 721 00:37:20,245 --> 00:37:20,275 I. 722 00:37:21,250 --> 00:37:23,920 For their immutability options. 723 00:37:24,190 --> 00:37:27,190 One which sort of allows an admin to reset 724 00:37:27,460 --> 00:37:27,820 W. Curtis Preston: Mm-Hmm. 725 00:37:28,450 --> 00:37:30,100 Prasanna Malaiyandi: the flag, right? 726 00:37:30,100 --> 00:37:31,330 Like we had talked about earlier. 727 00:37:31,330 --> 00:37:34,660 So it's good enough, but there are cases where, yeah, I need to unset it. 728 00:37:34,930 --> 00:37:36,580 So they do give you that option. 729 00:37:37,210 --> 00:37:40,600 And then the other one is you can't touch it unless you basically, in the case 730 00:37:40,600 --> 00:37:43,060 of AWS, blow away your account, right? 731 00:37:43,060 --> 00:37:46,630 So it's really nothing you could do as an admin will help you. 732 00:37:47,800 --> 00:37:48,160 W. Curtis Preston: Yeah. 733 00:37:48,160 --> 00:37:50,830 And you know, just realize that a, what, what I would 734 00:37:50,830 --> 00:37:52,720 call a truly immutable system. 735 00:37:53,905 --> 00:37:59,125 If you change your mind, you can't change your mind, right? 736 00:37:59,395 --> 00:38:04,135 If it's, if it's truly immutable, you should not be able to, to blow it away. 737 00:38:04,165 --> 00:38:08,395 Now, I, I still haven't done this experiment, and I, I, I really am gonna 738 00:38:08,395 --> 00:38:14,095 do this at one point where I create an AWS account, turn on immutability, put 739 00:38:14,275 --> 00:38:17,095 a bunch of stuff in there, and then delete the account and see what happens. 740 00:38:17,635 --> 00:38:21,690 Um, I don't, I don't know what happens there, and then stop paying my bill. 741 00:38:24,790 --> 00:38:27,160 No, no immutability system. 742 00:38:27,400 --> 00:38:31,390 Outlives not paying for your, your, uh, your storage 743 00:38:31,390 --> 00:38:32,740 Prasanna Malaiyandi: Yeah, they will probably reclaim that 744 00:38:32,740 --> 00:38:33,910 data as quickly as they can. 745 00:38:33,910 --> 00:38:34,390 Curtis. 746 00:38:36,010 --> 00:38:37,240 W. Curtis Preston: do garbage collection. 747 00:38:37,510 --> 00:38:39,070 Goodbye Curtis's data. 748 00:38:39,880 --> 00:38:45,370 The, I I mean, I will say that in that case it's kind of your own fault, but, um, 749 00:38:46,420 --> 00:38:49,390 so this is what we mean when we say that. 750 00:38:49,930 --> 00:38:50,860 Immutable backups. 751 00:38:50,860 --> 00:38:58,180 You wanna make sure that a black hat, a bad actor that gets into your environment, 752 00:38:58,750 --> 00:39:01,390 cannot do damage to the backup system. 753 00:39:01,840 --> 00:39:04,090 And so you want to separate. 754 00:39:04,645 --> 00:39:09,895 Backups as much as you can from the, the, the bad actor. 755 00:39:10,135 --> 00:39:14,035 And the best way to do that is to store them in an immutable system. 756 00:39:14,305 --> 00:39:20,275 Just realize that there is a, um, a spectrum of immutability and 757 00:39:20,455 --> 00:39:23,755 all of them, they all say their backups are immutable, right? 758 00:39:24,665 --> 00:39:28,025 Prasanna Malaiyandi: I think it's also important to note that you may not need 759 00:39:28,025 --> 00:39:33,695 the best solution for your use case and your needs, so also make sure you 760 00:39:33,695 --> 00:39:35,885 understand what you're protecting against. 761 00:39:37,120 --> 00:39:39,880 There is gonna be cost, there is gonna be complexity potentially 762 00:39:39,880 --> 00:39:44,980 with those solutions, and maybe it isn't needed to go to that extent. 763 00:39:45,280 --> 00:39:47,800 And maybe like better is good enough. 764 00:39:48,550 --> 00:39:48,760 W. Curtis Preston: Yeah. 765 00:39:48,760 --> 00:39:52,091 Never let, never let perfect be the enemy of better. 766 00:39:52,600 --> 00:39:52,990 Right? 767 00:39:53,290 --> 00:39:55,630 So, you know, you, you could be better. 768 00:39:56,110 --> 00:39:58,780 Um, and that's okay. 769 00:39:59,260 --> 00:40:01,630 Always strive to be better, but don't. 770 00:40:02,545 --> 00:40:06,775 Don't let say, well, th you know, th this immutable system 771 00:40:06,835 --> 00:40:08,725 doesn't work in this one scenario. 772 00:40:09,145 --> 00:40:11,515 Don't let that prevent you from doing something. 773 00:40:11,755 --> 00:40:14,215 Something is always better than nothing, right? 774 00:40:14,735 --> 00:40:21,005 One thing I do want to mention is that MFA is a big part of this. 775 00:40:21,275 --> 00:40:26,495 Some of these systems that have it's immutable except for the super powerful 776 00:40:26,500 --> 00:40:31,205 person, or it's immutable unless you make these 17 phone calls to. 777 00:40:31,580 --> 00:40:34,760 Um, you know, you, you've gotta really, really authenticate yourself. 778 00:40:35,480 --> 00:40:42,110 I have seen scenarios where companies have had their email system hacked 779 00:40:42,710 --> 00:40:47,000 in addition to their backup system being hacked, and the result was that 780 00:40:47,000 --> 00:40:52,100 they were able to defeat MFA because they were using email as their MFA. 781 00:40:52,980 --> 00:40:55,110 Prasanna Malaiyandi: Or the other thing is also phone because now they 782 00:40:55,110 --> 00:40:59,400 can impersonate someone, call in, they have all the access to the emails 783 00:40:59,400 --> 00:41:02,040 and can spoof like two factor codes. 784 00:41:02,715 --> 00:41:06,165 And talk to the person on the other end, and they don't know the difference. 785 00:41:07,200 --> 00:41:13,470 W. Curtis Preston: Yeah, the, the, the, um, the ability of AI systems these 786 00:41:13,470 --> 00:41:19,230 days to mimic the sound of a, of another person's voice, uh, in, in fact, uh, 787 00:41:19,260 --> 00:41:21,660 just as a demonstration, uh, the next. 788 00:41:22,260 --> 00:41:25,230 30 seconds of audio is completely fake. 789 00:41:26,030 --> 00:41:27,470 How are you doing, Prasanna? 790 00:41:27,851 --> 00:41:29,221 Prasanna Malaiyandi: I'm doing fine, Curtis. 791 00:41:29,651 --> 00:41:30,911 What are we talking about today? 792 00:41:31,288 --> 00:41:33,938 W. Curtis Preston: I think we're talking about immutable backups, and 793 00:41:33,938 --> 00:41:36,008 I hear that they are on a Spectrum. 794 00:41:36,394 --> 00:41:38,354 Prasanna Malaiyandi: You mean like IBM Spectrum Protect? 795 00:41:38,700 --> 00:41:40,430 W. Curtis Preston: No, that's a backup product. 796 00:41:40,831 --> 00:41:42,141 Prasanna Malaiyandi: You mean the Irvine Spectrum? 797 00:41:42,541 --> 00:41:43,671 W. Curtis Preston: No, that's a mall. 798 00:41:43,991 --> 00:41:47,291 I mean that there are many things that are called immutable, and 799 00:41:47,291 --> 00:41:50,691 they all have different levels of protection against attacks. 800 00:41:51,049 --> 00:41:53,069 Nice talking with you, fake Prasanna. 801 00:41:53,589 --> 00:41:55,049 Prasanna Malaiyandi: You too, fake Curtis. 802 00:41:57,930 --> 00:42:00,810 W. Curtis Preston: So that was neither me nor persona talking. 803 00:42:00,840 --> 00:42:04,830 For the record, that was a little tool that we use. 804 00:42:05,820 --> 00:42:09,870 That is very useful in the right scenario, but very dangerous in the wrong scenario. 805 00:42:10,500 --> 00:42:14,340 Um, yeah, you, you really have to think about that, right? 806 00:42:14,340 --> 00:42:17,430 When you, when you put all these layers, if you have all these layers of 807 00:42:17,430 --> 00:42:23,670 security between you and the ability to delete your quote immutable data, think 808 00:42:23,670 --> 00:42:29,580 about all of the modern things that people are able to do, um, before, um. 809 00:42:30,195 --> 00:42:32,475 You know, you subtle with one of those systems. 810 00:42:32,475 --> 00:42:35,835 And again, don't let perfect be the enemy of the good. 811 00:42:35,835 --> 00:42:40,575 But, but, but just think about all the things that are possible these days. 812 00:42:41,025 --> 00:42:44,085 Uh, you know, they can fake video, they can fake audio. 813 00:42:44,205 --> 00:42:48,195 Uh, they can certainly fake email and phones, right? 814 00:42:48,200 --> 00:42:50,805 SMS is a horrible way to do things, right. 815 00:42:52,245 --> 00:42:54,735 But, um, um. 816 00:42:55,260 --> 00:42:57,600 With that, that's our discussion on immutable backups. 817 00:42:57,600 --> 00:42:58,710 I hope you enjoyed a persona. 818 00:42:59,805 --> 00:43:01,335 Prasanna Malaiyandi: No, this was a good topic. 819 00:43:01,335 --> 00:43:02,595 I like immutable backups. 820 00:43:03,120 --> 00:43:03,390 W. Curtis Preston: Yeah. 821 00:43:03,390 --> 00:43:04,230 Immutable backups. 822 00:43:04,230 --> 00:43:04,590 Good. 823 00:43:05,370 --> 00:43:07,980 And, uh, hope, uh, you enjoyed that as a listener. 824 00:43:08,040 --> 00:43:09,270 Hope you learned a thing or two. 825 00:43:09,570 --> 00:43:11,220 And with that, that's a wrap.