1 00:00:34,202 --> 00:00:37,112 W. Curtis Preston: hi, and welcome to Backup Central's Restore it All podcast. 2 00:00:37,112 --> 00:00:39,512 I'm your host w Curtis Preston, aka Mr. 3 00:00:39,512 --> 00:00:43,322 Backup, and have with me a guy that just like five minutes 4 00:00:43,322 --> 00:00:45,092 ago, compared me to a Luddite. 5 00:00:46,712 --> 00:00:49,832 Prasanna Malaiyandi . How's it going, Prasanna? 6 00:00:50,102 --> 00:00:51,032 Prasanna Malaiyandi: I'm good. 7 00:00:51,362 --> 00:00:56,612 W. Curtis Preston: Did, Did you or did you not compare me to a Luddite? 8 00:00:57,212 --> 00:01:00,032 Prasanna Malaiyandi: I did, and there was a reason I did. 9 00:01:00,482 --> 00:01:02,702 Right, which was you were. 10 00:01:03,592 --> 00:01:08,507 Very used to a certain way of operating a, a piece of software, 11 00:01:08,537 --> 00:01:09,737 right, for the workflow. 12 00:01:09,767 --> 00:01:14,297 And that software completely updated, but not everything was there. 13 00:01:14,297 --> 00:01:19,097 And I think your complaint was, I just want things back the way they were. 14 00:01:19,347 --> 00:01:23,327 W. Curtis Preston: Uh, it's, you could put it that way, 15 00:01:23,597 --> 00:01:24,077 Prasanna Malaiyandi: Right. 16 00:01:24,082 --> 00:01:24,437 I'm 17 00:01:24,527 --> 00:01:30,017 W. Curtis Preston: but I'm not, I'm not averse to technology or to new technology. 18 00:01:30,287 --> 00:01:33,617 I like things to get better, but it does, It is difficult when you're 19 00:01:33,617 --> 00:01:36,977 using a piece of software and then they're like, You know what? 20 00:01:37,952 --> 00:01:40,022 We had to just rip and replace. 21 00:01:40,022 --> 00:01:44,642 Like in order to get, in order to be where we wanted to be, we had to throw 22 00:01:44,642 --> 00:01:46,232 out the baby with the bath water. 23 00:01:46,237 --> 00:01:50,222 We had to, you know, you're gonna love what we have. 24 00:01:50,462 --> 00:01:54,182 Once you get used to it, I hope you hope you don't mind throwing away everything 25 00:01:54,187 --> 00:01:55,742 you've known after all this time. 26 00:01:55,742 --> 00:01:57,002 Prasanna Malaiyandi: You know what this reminds me of? 27 00:01:57,962 --> 00:02:02,702 Every time someone updates a backup software app, like completely changes it 28 00:02:02,702 --> 00:02:05,092 or pushes you, and everyone's like, No. 29 00:02:05,732 --> 00:02:11,912 I remember, uh, at the old place I used to work that there was, I can't 30 00:02:11,912 --> 00:02:15,422 remember what version of the backup software it was, but one version, 31 00:02:15,422 --> 00:02:16,622 they decided to change the color, 32 00:02:17,762 --> 00:02:18,252 W. Curtis Preston: Yeah. 33 00:02:19,172 --> 00:02:21,662 Prasanna Malaiyandi: right, the color of the background, like what their standard 34 00:02:21,662 --> 00:02:23,372 colors were and everyone complained. 35 00:02:24,092 --> 00:02:26,162 And they basically had to undo it because they were like, We don't 36 00:02:26,162 --> 00:02:27,932 understand what these colors are. 37 00:02:27,992 --> 00:02:28,922 We don't like them. 38 00:02:28,922 --> 00:02:30,692 Go back to the way they were before. 39 00:02:31,742 --> 00:02:35,162 W. Curtis Preston: Yeah, I don't think my concern was that petty. 40 00:02:35,372 --> 00:02:40,622 I'm just saying, but I'm just saying when the buttons that I used to press no longer 41 00:02:40,622 --> 00:02:46,232 work, you know, um, I'm just saying, I just, I just wanna edit my video. 42 00:02:46,237 --> 00:02:46,382 That's 43 00:02:46,442 --> 00:02:49,652 Prasanna Malaiyandi: But, but, but this is also kind of the. 44 00:02:50,432 --> 00:02:55,232 Upside and downside of SaaS is things run really fast. 45 00:02:55,232 --> 00:03:00,002 You get great new features, but because you're building so quickly, not everything 46 00:03:00,002 --> 00:03:03,182 makes it over in those first releases, and sometimes you just gotta wait. 47 00:03:04,307 --> 00:03:06,947 W. Curtis Preston: Yeah, and I'm, I'm not good at waiting, I don't think. 48 00:03:07,037 --> 00:03:09,107 I don't know if you've figured that out yet or not. 49 00:03:09,137 --> 00:03:10,967 I'm not good at delayed gratification. 50 00:03:12,257 --> 00:03:17,207 But, uh, so, uh, speaking of places that you used to work, I'll 51 00:03:17,207 --> 00:03:18,767 throw out our usual disclaimer. 52 00:03:19,007 --> 00:03:20,117 Prasanna works for Zoom. 53 00:03:20,122 --> 00:03:21,797 I work for Druva Now. 54 00:03:21,797 --> 00:03:26,957 This is not a podcast of either company and the opinions that you hear are ours. 55 00:03:27,427 --> 00:03:29,617 And, uh, one of which is apparently a Luddite. 56 00:03:30,067 --> 00:03:31,417 Um, and if you. 57 00:03:32,477 --> 00:03:33,557 Prasanna Malaiyandi: True opinion. 58 00:03:33,647 --> 00:03:33,887 It's 59 00:03:34,052 --> 00:03:35,612 W. Curtis Preston: we would love for you to rate us. 60 00:03:35,672 --> 00:03:38,342 Uh, please rate us by going to your favorite podcatcher. 61 00:03:38,462 --> 00:03:40,922 Scroll down to where you give us the stars and give us a comment. 62 00:03:40,922 --> 00:03:41,852 We love the comments. 63 00:03:41,852 --> 00:03:43,322 It's great to hear from people. 64 00:03:43,832 --> 00:03:47,252 Um, and, uh, I really love when you reach out to me on LinkedIn. 65 00:03:47,252 --> 00:03:48,152 That's also great. 66 00:03:48,182 --> 00:03:52,472 If you wanna join the conversation, be on the podcast, just reach out to me at WC 67 00:03:52,477 --> 00:03:55,082 preston on Twitter or w Curtis Preston, 68 00:03:57,002 --> 00:03:58,292 @gmail: and, uh, we'll get you on. 69 00:03:58,757 --> 00:04:04,427 And, um, so I wanna continue and, and, and almost continue 70 00:04:04,427 --> 00:04:06,317 sort of go back to square one. 71 00:04:06,317 --> 00:04:10,337 We talked about doing a back to the basics series and for some 72 00:04:10,337 --> 00:04:11,537 reason we started the back. 73 00:04:11,597 --> 00:04:13,757 We started sort of, I thought we started 74 00:04:14,627 --> 00:04:15,827 Prasanna Malaiyandi: We didn't go far enough back 75 00:04:16,247 --> 00:04:17,657 W. Curtis Preston: We, yeah, we didn't go far enough back. 76 00:04:17,662 --> 00:04:24,797 And so I thought that a perfect place to go to, to look at the basics of 77 00:04:24,797 --> 00:04:26,897 backup would be this lovely book. 78 00:04:26,897 --> 00:04:30,572 Which I believe you helped, uh, edit sir. 79 00:04:30,962 --> 00:04:35,942 Uh, Modern Data Protection from O'Reilly and Associates, and, um, 80 00:04:36,032 --> 00:04:37,472 available at a bookstore near you. 81 00:04:38,102 --> 00:04:42,602 You can get an ebook version of this by going to druva.com/podcast 82 00:04:42,962 --> 00:04:45,782 and, um, you know, free of charge. 83 00:04:46,412 --> 00:04:51,092 In the beginning of this book, we have the, you know, what some might consider 84 00:04:51,092 --> 00:04:54,122 a fluff chapter, which is why we back up. 85 00:04:54,452 --> 00:04:59,072 And I, I thought there's no more basic question to answer. 86 00:04:59,462 --> 00:05:01,382 Then why do we do all of this? 87 00:05:02,042 --> 00:05:08,762 And I, I start the book with a story, which I know I've alluded to 88 00:05:08,762 --> 00:05:10,562 a couple of times on the podcast. 89 00:05:10,592 --> 00:05:13,082 I may have even told the whole version, but, hey, we're 90 00:05:13,082 --> 00:05:14,192 starting from scratch here. 91 00:05:14,197 --> 00:05:16,412 So I'm gonna tell this version. 92 00:05:16,412 --> 00:05:18,782 And this is basically how I got into backup. 93 00:05:19,472 --> 00:05:23,042 I was al, I was already in backup, but this is how I really got into it. 94 00:05:23,042 --> 00:05:23,162 You 95 00:05:23,372 --> 00:05:25,052 Prasanna Malaiyandi: Wait before you get there, 96 00:05:25,472 --> 00:05:25,862 W. Curtis Preston: yeah. 97 00:05:26,342 --> 00:05:28,622 Prasanna Malaiyandi: do you wanna, because I don't think many of the 98 00:05:28,622 --> 00:05:30,962 listeners really know, like, what 99 00:05:31,127 --> 00:05:31,667 W. Curtis Preston: How did I, 100 00:05:31,772 --> 00:05:33,962 Prasanna Malaiyandi: you do before you became Mr. 101 00:05:33,962 --> 00:05:34,352 Backup? 102 00:05:34,352 --> 00:05:35,642 Before you got into backup? 103 00:05:35,642 --> 00:05:38,972 Like, like where, Like how did you go from whatever it was 104 00:05:38,972 --> 00:05:39,652 you were doing into backup? 105 00:05:40,637 --> 00:05:41,117 W. Curtis Preston: Hmm. 106 00:05:41,552 --> 00:05:42,602 Prasanna Malaiyandi: I don't think a lot of people know 107 00:05:42,602 --> 00:05:43,762 that, and it's very different. 108 00:05:44,897 --> 00:05:52,727 W. Curtis Preston: Well, in, um, January of 1993, I got out of 109 00:05:52,727 --> 00:05:57,527 the US Navy and I was looking for work in the computer business. 110 00:05:57,527 --> 00:05:59,627 I had zero experience in computers. 111 00:05:59,632 --> 00:06:02,617 I had taken, I don't, I don't know if you're, if, do you remember 112 00:06:02,717 --> 00:06:04,127 the National Radio Institute? 113 00:06:05,672 --> 00:06:09,512 Do you remember that they, they had the ads back in, back in Popular Science. 114 00:06:09,512 --> 00:06:11,332 They had an ad like, Build your own 115 00:06:11,332 --> 00:06:11,612 Prasanna Malaiyandi: computer 116 00:06:11,612 --> 00:06:12,547 Oh, yeah, yeah, yeah, yeah, 117 00:06:12,692 --> 00:06:13,502 W. Curtis Preston: build your own computer. 118 00:06:13,742 --> 00:06:18,482 I did that while I was out to see, I took this correspondence course. 119 00:06:18,482 --> 00:06:20,192 This is pre-internet, right? 120 00:06:20,887 --> 00:06:24,157 Um, you got, you got a, like a book and you worked through that book and 121 00:06:24,157 --> 00:06:26,887 it was like q and a and then, you know, and then once you got all done, 122 00:06:26,887 --> 00:06:32,287 then you built a computer and it was an 8088 for those of you I remember. 123 00:06:32,767 --> 00:06:35,347 So I built my own computer and that was literally the extent of 124 00:06:35,347 --> 00:06:36,607 my computer experience, right? 125 00:06:36,637 --> 00:06:40,867 And I got outta the Navy and I managed to get a job at a company called Digital 126 00:06:40,867 --> 00:06:47,527 Systems, which was a, um, a company that it put in, um, it was called Intelligent. 127 00:06:48,142 --> 00:06:50,332 Blended call management systems. 128 00:06:50,512 --> 00:06:52,522 You might call it an auto dialer. 129 00:06:52,972 --> 00:06:53,812 Um, auto 130 00:06:53,977 --> 00:06:55,027 Prasanna Malaiyandi: Robo caller. 131 00:06:55,312 --> 00:06:56,872 W. Curtis Preston: was a naughty word or a robo caller. 132 00:06:57,082 --> 00:06:58,552 Yeah, that was a naughty word. 133 00:06:58,552 --> 00:07:00,682 That was sort of the dumb machine that didn't have the 134 00:07:00,682 --> 00:07:01,852 intelligence that ours had. 135 00:07:01,852 --> 00:07:04,972 But anyway, that's what, that was the very first job, but only like, 136 00:07:05,797 --> 00:07:09,967 Three or four months later, I secured a job via a great connection. 137 00:07:10,117 --> 00:07:14,587 That would be my wife, um, at what at that time was the second largest credit 138 00:07:14,587 --> 00:07:17,107 card company, and that would be mbna. 139 00:07:17,647 --> 00:07:24,457 And, um, I got the job of the backup guy and um, I was in charge of the 140 00:07:24,457 --> 00:07:29,467 backups for our entire data center, which at that time was 15 servers. 141 00:07:30,067 --> 00:07:33,027 Um, they were DEC. 142 00:07:33,082 --> 00:07:38,062 We also had the first computer designed for Unix, AT&T SV 3B2. 143 00:07:38,482 --> 00:07:44,422 And, um, Uh, and I started with little tiny with, uh, little tiny, 144 00:07:44,542 --> 00:07:46,612 uh, cartridges and tapes and whatnot. 145 00:07:46,732 --> 00:07:47,452 Um, 146 00:07:47,857 --> 00:07:49,042 Prasanna Malaiyandi: how big was the environment? 147 00:07:49,102 --> 00:07:52,012 I know you said 15 servers, but how much data approximately 148 00:07:52,042 --> 00:07:56,242 W. Curtis Preston: so the entire data center, when I left, i d so 149 00:07:56,362 --> 00:08:00,082 when I joined it was probably the whole data center was 50 gigabytes. 150 00:08:02,842 --> 00:08:05,002 because it was literally like 11 servers. 151 00:08:05,152 --> 00:08:09,502 In fact, it was probably smaller than that because our biggest server 152 00:08:09,562 --> 00:08:11,092 I remember was five gigabytes. 153 00:08:11,122 --> 00:08:12,082 It was huge. 154 00:08:12,142 --> 00:08:15,772 It was five gigabytes and it took us all weekend to get a full backup of it. 155 00:08:16,332 --> 00:08:21,517 Um, And, uh, and the other servers were much smaller than that, so 156 00:08:21,517 --> 00:08:24,787 it was probably, maybe even 30 gigabytes as the whole data center. 157 00:08:24,787 --> 00:08:27,307 But by the time I left, it was much larger. 158 00:08:27,307 --> 00:08:29,137 It was like 300 gigabytes 159 00:08:29,502 --> 00:08:29,992 Prasanna Malaiyandi: Yeah. 160 00:08:30,217 --> 00:08:33,487 W. Curtis Preston: and 100 of those 300 gigabytes was a single server. 161 00:08:33,877 --> 00:08:36,427 It was a, it was an HP T 500. 162 00:08:36,457 --> 00:08:37,777 It had a hundred gigabytes. 163 00:08:37,777 --> 00:08:40,687 And I was like, How in the hell am I ever gonna back all this up? 164 00:08:41,407 --> 00:08:44,917 Um, because it had a, it had a four gigabyte tape drive and I was like, 165 00:08:44,922 --> 00:08:48,952 so, Are you gonna give me an FTE to stand here and swap tapes all night? 166 00:08:49,012 --> 00:08:52,132 And that's how I got my first automated table library, by the way. 167 00:08:52,462 --> 00:08:53,052 Prasanna Malaiyandi: There you go. 168 00:08:53,782 --> 00:08:58,762 W. Curtis Preston: But, uh, yeah, so, um, in the midst of all of that, I had 169 00:08:58,762 --> 00:09:06,232 been the backup person for a few months and we lost our purchasing database. 170 00:09:06,232 --> 00:09:07,942 It was an Oracle database. 171 00:09:08,002 --> 00:09:10,012 Uh, the name of the server was Paris. 172 00:09:10,442 --> 00:09:11,732 It had been recently migrated. 173 00:09:11,732 --> 00:09:12,842 It had only been on Paris. 174 00:09:14,177 --> 00:09:17,087 I don't know, like a month or two, a couple of months. 175 00:09:18,167 --> 00:09:23,267 And the roughly, slightly less time than how long I'd been at 176 00:09:23,272 --> 00:09:24,497 the company, whatever that was. 177 00:09:25,457 --> 00:09:26,387 And. 178 00:09:28,067 --> 00:09:29,597 I did what I was told to do. 179 00:09:29,597 --> 00:09:33,347 I went, I looked at the logs to see if the dump was good, cuz we used dump in 180 00:09:33,347 --> 00:09:35,627 those days and the dump wasn't good. 181 00:09:35,632 --> 00:09:38,237 So then I looked at the day before dump wasn't good, I looked at the 182 00:09:38,242 --> 00:09:40,697 day before and so on and so on and so on and so on and so on. 183 00:09:40,697 --> 00:09:45,392 And I finally found the tape that was, um, That was good. 184 00:09:45,752 --> 00:09:50,942 And it was, uh, six weeks and one day old, and I knew that I had the transaction 185 00:09:50,942 --> 00:09:54,692 logs along that time, so I was, you know, I'm thinking that that'll work. 186 00:09:54,902 --> 00:09:59,342 Uh, the only problem was that, um, the, our retention period was uh, six weeks 187 00:09:59,947 --> 00:10:00,377 Prasanna Malaiyandi: Oh my 188 00:10:00,692 --> 00:10:03,842 W. Curtis Preston: so the last good backup was just, had just been overwritten. 189 00:10:04,622 --> 00:10:09,542 And so I remember, you know, my boss Susan Davidson. 190 00:10:09,592 --> 00:10:12,712 I remember her standing over me and saying, So let me get this straight. 191 00:10:13,042 --> 00:10:16,102 We have no backups of Paris whatsoever. 192 00:10:16,762 --> 00:10:18,622 And I said, That is what I'm saying. 193 00:10:19,642 --> 00:10:20,062 And. 194 00:10:21,277 --> 00:10:25,807 You know, to her credit, she attributed the problem to, to bad training. 195 00:10:26,227 --> 00:10:27,157 That's you Ron. 196 00:10:27,637 --> 00:10:30,157 Um, to bad training. 197 00:10:30,697 --> 00:10:33,727 And what what had happened was we had migrated the server. 198 00:10:33,727 --> 00:10:37,987 Nobody had told me that there was this, this, um, chron job that had been shutting 199 00:10:37,987 --> 00:10:40,147 the database down for the backups. 200 00:10:41,207 --> 00:10:44,747 And I learned a lot of valuable lessons, or I took, I, I. 201 00:10:45,457 --> 00:10:48,707 I don't know a lot of who I am and how I do things come all the 202 00:10:48,707 --> 00:10:50,387 way back to that first episode. 203 00:10:50,597 --> 00:10:54,917 I don't like, like a completely separated cron job that's 204 00:10:54,917 --> 00:10:56,507 doing something for my backup. 205 00:10:57,317 --> 00:11:01,157 Before I do the backup, I want the backup to kick off the cron job. 206 00:11:01,337 --> 00:11:01,667 Right? 207 00:11:01,667 --> 00:11:02,897 What, You know what I'm saying? 208 00:11:03,197 --> 00:11:05,687 the CRO job kicks off the backup that kicks off the job. 209 00:11:05,687 --> 00:11:09,587 You know, I need coordination and I've always, you know, when every, when anybody 210 00:11:09,587 --> 00:11:13,607 starts talking to me about two separate processes that are gonna handle, you 211 00:11:13,607 --> 00:11:16,367 know, that are gonna do this, if they don't have a handshake mechanism, I 212 00:11:16,372 --> 00:11:18,557 get very, very nervous because of that. 213 00:11:19,307 --> 00:11:22,757 Um, cuz I continued to back up the database all the time while 214 00:11:22,757 --> 00:11:25,547 the database was changing and not putting it into hot backup. 215 00:11:27,422 --> 00:11:27,962 Uh, 216 00:11:28,017 --> 00:11:29,817 Prasanna Malaiyandi: It's like the case where a lot of people are like, Oh 217 00:11:29,817 --> 00:11:32,397 yeah, I just take a snapshot, and then someone just dumps that off the tape. 218 00:11:32,847 --> 00:11:36,777 It's like, what happens if the thing that did the snapshot failed for some 219 00:11:36,782 --> 00:11:40,077 reason, you're now gonna think you have a successful backup, and guess what? 220 00:11:40,117 --> 00:11:40,777 You don't. 221 00:11:41,862 --> 00:11:42,672 W. Curtis Preston: Yeah, yeah. 222 00:11:43,092 --> 00:11:44,502 And I understand that that's something. 223 00:11:45,597 --> 00:11:48,927 Oracle DBAs fight over the, the DBAs want to be in charge of their 224 00:11:48,927 --> 00:11:51,957 backups, but the backup person has to be in charge of the backup system. 225 00:11:52,197 --> 00:11:54,987 So how do you get those two, you know, to, to get together? 226 00:11:55,257 --> 00:11:57,117 You know, we could have a whole episode just about that. 227 00:11:57,837 --> 00:11:59,457 Uh, we probably should have an whole 228 00:11:59,457 --> 00:11:59,817 episode 229 00:11:59,907 --> 00:12:02,817 Prasanna Malaiyandi: I spent quite a bit of my career focused on that one problem. 230 00:12:03,417 --> 00:12:03,597 W. Curtis Preston: Yeah. 231 00:12:04,617 --> 00:12:05,217 , you did. 232 00:12:05,907 --> 00:12:08,997 Prasanna Malaiyandi: Before you talk about like the reasons we back up, I 233 00:12:08,997 --> 00:12:14,427 think it's important to state that you back up not because you want to back up. 234 00:12:14,427 --> 00:12:18,477 It's like you had said, you need to make sure that you're able to recover your 235 00:12:18,477 --> 00:12:22,077 database, your application, whatever it is, whenever something happens, right? 236 00:12:22,077 --> 00:12:24,677 That's the entire purpose of doing backups is 237 00:12:24,747 --> 00:12:24,897 W. Curtis Preston: you 238 00:12:24,897 --> 00:12:26,607 only need to back up if you wanna restore 239 00:12:26,997 --> 00:12:28,617 Prasanna Malaiyandi: Yeah, exactly. 240 00:12:30,687 --> 00:12:32,247 W. Curtis Preston: You know, the only reason I had kids, 241 00:12:33,762 --> 00:12:36,732 Prasanna Malaiyandi: Because you wanted to restore some place, 242 00:12:36,927 --> 00:12:37,257 W. Curtis Preston: No. 243 00:12:37,257 --> 00:12:38,547 So that I could have a grandkid. 244 00:12:39,867 --> 00:12:41,667 They're way better than kids. 245 00:12:41,877 --> 00:12:43,317 She's sitting right outside there. 246 00:12:43,317 --> 00:12:44,307 She's adorable. 247 00:12:44,607 --> 00:12:48,507 And she's nine and I'll take her over either of my two kids. 248 00:12:48,927 --> 00:12:51,797 Uh, but the thing is, you gotta have kids before you can have. 249 00:12:52,782 --> 00:12:53,622 That's just the way it works. 250 00:12:53,622 --> 00:12:54,672 Same thing with backups. 251 00:12:55,272 --> 00:12:59,322 So I've divided the, the reasons that we back up the risks to 252 00:12:59,322 --> 00:13:01,122 your data into three groups. 253 00:13:01,512 --> 00:13:06,522 Uh, human disasters, mechanical failures, uh, and natural disasters. 254 00:13:07,122 --> 00:13:11,292 So, human disasters to me are things that humans cause. 255 00:13:12,282 --> 00:13:14,982 We can think of all sorts of categories of those. 256 00:13:15,282 --> 00:13:18,552 And I'm gonna give what I still think is the number, the 257 00:13:18,552 --> 00:13:20,562 number one reason we restore. 258 00:13:20,652 --> 00:13:21,072 What's that? 259 00:13:21,102 --> 00:13:21,892 What do you think that. 260 00:13:22,362 --> 00:13:23,902 Prasanna Malaiyandi: User did something dumb. 261 00:13:24,952 --> 00:13:26,007 W. Curtis Preston: That's exactly right. 262 00:13:26,772 --> 00:13:29,772 Prasanna Malaiyandi: I accidentally deleted a folder and 263 00:13:29,772 --> 00:13:31,812 I intended to only delete a file. 264 00:13:31,812 --> 00:13:32,202 Oops. 265 00:13:32,382 --> 00:13:34,812 Or I think I still, I can't remember. 266 00:13:34,812 --> 00:13:36,042 I think you have a story, right? 267 00:13:36,042 --> 00:13:38,502 Where someone had a script that accidentally deleted a file 268 00:13:38,502 --> 00:13:40,272 server and erased all of the 269 00:13:40,587 --> 00:13:41,097 W. Curtis Preston: in the book. 270 00:13:41,097 --> 00:13:41,607 Yeah. 271 00:13:41,637 --> 00:13:41,847 Yeah. 272 00:13:41,847 --> 00:13:45,537 They were, What they were doing is they were purging home directories 273 00:13:45,537 --> 00:13:48,687 that were no longer for users that were no longer in the password file. 274 00:13:49,622 --> 00:13:53,427 . And so they were going, you know, home one and then they went to the, the first 275 00:13:53,427 --> 00:13:56,277 directory and then they'd look up the name of the directory in the password file. 276 00:13:56,727 --> 00:13:59,007 And if it wasn't there, cuz the name of the directory was 277 00:13:59,007 --> 00:14:00,327 just name of the, of the user. 278 00:14:00,567 --> 00:14:07,317 The problem was home one slash a slash all users beginning with a. 279 00:14:07,497 --> 00:14:10,227 And so they got to home one A looked up for a, in 280 00:14:10,227 --> 00:14:10,767 the password 281 00:14:10,767 --> 00:14:12,897 file they were there and then de Yeah. 282 00:14:13,047 --> 00:14:15,807 And they were half, they had deleted half the home directories before 283 00:14:15,807 --> 00:14:17,327 we figured out what was going. 284 00:14:19,002 --> 00:14:19,802 Oh Gonda. 285 00:14:20,172 --> 00:14:20,592 Okay. 286 00:14:20,742 --> 00:14:22,542 So, um, yeah. 287 00:14:22,592 --> 00:14:23,672 Prasanna Malaiyandi: like you said, most of 'em are. 288 00:14:23,672 --> 00:14:24,032 Yeah. 289 00:14:24,492 --> 00:14:25,662 W. Curtis Preston: it's dumb stuff. 290 00:14:25,962 --> 00:14:29,802 It's, it's dumb stuff by users, dumb stuff by admins, dumb stuff by developers. 291 00:14:30,432 --> 00:14:30,852 Right. 292 00:14:31,002 --> 00:14:35,022 Um, you know, that sort of dumb stuff. 293 00:14:35,052 --> 00:14:38,512 And then we have bad stuff, right. 294 00:14:38,712 --> 00:14:39,852 That happens outside. 295 00:14:39,852 --> 00:14:43,962 This is, I think today we're starting to think of this as. 296 00:14:45,402 --> 00:14:48,672 You know, the first is, I think the number one reason you need a backup system. 297 00:14:48,672 --> 00:14:52,002 This is the number one reason you need a good backup system. 298 00:14:52,662 --> 00:14:57,462 You need a really good backup n dr system, and that is malicious attacks, 299 00:14:57,492 --> 00:15:03,072 ransomware, you know, malware, um, you know, and internal threats, right? 300 00:15:03,662 --> 00:15:04,992 Uh, what do we call those? 301 00:15:04,992 --> 00:15:05,772 Rogue admins, 302 00:15:06,492 --> 00:15:06,852 right? 303 00:15:07,872 --> 00:15:08,682 Um, 304 00:15:09,072 --> 00:15:12,702 Prasanna Malaiyandi: Deleting all your data and going out, and I actually 305 00:15:12,702 --> 00:15:14,112 just read about that recently. 306 00:15:14,502 --> 00:15:15,552 Uh, who was it? 307 00:15:15,552 --> 00:15:21,942 I think someone recently got sentenced to jail because, uh, what did they do? 308 00:15:21,972 --> 00:15:25,332 They had been fired from their company. 309 00:15:25,662 --> 00:15:29,562 They still had access to the passwords because the company had not cycled them. 310 00:15:30,717 --> 00:15:34,437 And so they logged into the company, deleted, changed the 311 00:15:34,437 --> 00:15:35,547 accounts, locked them out. 312 00:15:35,547 --> 00:15:37,437 I think they might have deleted things as well. 313 00:15:38,007 --> 00:15:44,877 And then their entire purpose was they wanted to be rehired for higher pay Right. 314 00:15:44,882 --> 00:15:46,437 And just show their value. 315 00:15:46,437 --> 00:15:49,137 W. Curtis Preston: yeah, I think there was a little problem with this person's logic, 316 00:15:50,562 --> 00:15:51,192 Prasanna Malaiyandi: And so, yeah. 317 00:15:51,192 --> 00:15:52,752 Do you have rogue admins like this? 318 00:15:52,752 --> 00:15:52,992 Right? 319 00:15:52,992 --> 00:15:56,352 Where if you have someone who has a keys to the kingdom, or you don't 320 00:15:56,352 --> 00:15:59,832 know all the places that people like, you don't have a proper onboarding 321 00:15:59,862 --> 00:16:04,062 offboarding procedure to lock down your environment and delete things, right? 322 00:16:04,062 --> 00:16:06,672 You can have people access your data, change things, delete things 323 00:16:06,672 --> 00:16:08,712 like backup systems, all the 324 00:16:08,727 --> 00:16:11,547 W. Curtis Preston: And, and this, and this isn't a backup thing, but 325 00:16:11,547 --> 00:16:13,137 what do we learn from that kids? 326 00:16:13,407 --> 00:16:15,927 Have a proper onboarding and offboarding procedure. 327 00:16:16,817 --> 00:16:20,327 . I have been let go once or twice in my career, right? 328 00:16:20,357 --> 00:16:28,427 Uh, sometimes it was layoffs, sometimes it was a difference of opinion, and the um, 329 00:16:28,727 --> 00:16:30,227 as to whether or not I should be employed. 330 00:16:31,967 --> 00:16:38,177 I've had at least once where my account got disabled and then I 331 00:16:38,177 --> 00:16:39,887 got a call like a minute later. 332 00:16:40,742 --> 00:16:41,192 Right. 333 00:16:41,402 --> 00:16:44,312 That's an off, that's an off boarding procedure. 334 00:16:44,402 --> 00:16:44,822 Right? 335 00:16:44,882 --> 00:16:46,652 And by the way, I wasn't doing anything malicious. 336 00:16:46,652 --> 00:16:49,892 They weren't that, that was the off boarding procedure was your account was 337 00:16:49,892 --> 00:16:53,702 disabled, you were told to renew your password basically is what it looked like. 338 00:16:53,702 --> 00:16:56,282 You were told to renew your password and then boom, you, 339 00:16:57,242 --> 00:16:58,232 you know, you got the 340 00:16:58,772 --> 00:16:59,432 you got the call. 341 00:16:59,462 --> 00:17:06,272 Which is why to this day when I, when my password times out, which it does, right? 342 00:17:06,272 --> 00:17:08,702 I'm like, Oh, what, what did I do? 343 00:17:10,247 --> 00:17:10,637 To this 344 00:17:10,937 --> 00:17:12,047 Prasanna Malaiyandi: And it would be bad. 345 00:17:12,347 --> 00:17:15,947 Well, and it would also be bad if that happened to just coincide with 346 00:17:16,187 --> 00:17:17,497 a one on one with your manager, 347 00:17:18,227 --> 00:17:18,707 W. Curtis Preston: Yeah. 348 00:17:18,767 --> 00:17:19,187 Yeah. 349 00:17:19,187 --> 00:17:24,092 When, when you, um, Uh, yeah, and I had that happen last week where my, where 350 00:17:24,092 --> 00:17:28,802 my manager, they sent me a thing, you know, I, I have a weekly, you know, one 351 00:17:28,802 --> 00:17:33,002 on one and, and I had a, it was at three 30 on Monday afternoon, and suddenly it 352 00:17:33,002 --> 00:17:37,022 was, it was at nine 30 on Monday morning and I was like, Ooh, that's not good. 353 00:17:37,027 --> 00:17:39,692 What It turned out to be fine. 354 00:17:39,922 --> 00:17:43,172 Nothing was wrong, but that's what happens, you know, when you've 355 00:17:43,172 --> 00:17:44,432 been let go once or twice in your 356 00:17:44,732 --> 00:17:44,882 Prasanna Malaiyandi: Yeah. 357 00:17:45,242 --> 00:17:49,862 But, but, but, I think going back to the point, more people need to start 358 00:17:49,867 --> 00:17:51,452 thinking about malicious actors, 359 00:17:51,982 --> 00:17:52,542 right? 360 00:17:53,102 --> 00:17:56,522 Especially as they're designing backup systems, thinking about 361 00:17:56,552 --> 00:17:58,292 disaster recovery, right? 362 00:17:58,292 --> 00:18:01,982 Which is a huge, uh, topic that we always talk about on the podcast, right, is 363 00:18:02,342 --> 00:18:06,272 you really should be thinking about how quickly, cuz it's not good enough to 364 00:18:06,272 --> 00:18:08,222 just recover your data anymore, right? 365 00:18:08,222 --> 00:18:13,622 It's how quickly can you recover your data to recover from like a ransomware 366 00:18:13,622 --> 00:18:15,272 attack or other things like that. 367 00:18:16,262 --> 00:18:19,652 W. Curtis Preston: And I would say that you should really, especially look at 368 00:18:19,652 --> 00:18:27,782 the backup system and ask yourself, what kind of damage could Curtis, who's 369 00:18:27,782 --> 00:18:31,682 running the whole backup system, what kind of damage could Curtis do to us 370 00:18:31,682 --> 00:18:33,602 if Curtis got pissed off one afternoon? 371 00:18:34,282 --> 00:18:36,332 Now, this isn't about Curtis. 372 00:18:36,332 --> 00:18:40,292 This is about what happens if Curtis's account becomes compromised. 373 00:18:40,307 --> 00:18:40,727 Prasanna Malaiyandi: Yep. 374 00:18:42,017 --> 00:18:44,477 W. Curtis Preston: It's not just a rogue admin, it's also someone 375 00:18:44,807 --> 00:18:46,457 masquerading as a rogue admin. 376 00:18:46,907 --> 00:18:54,407 And I know that we, uh, at Druva, we added features that, um, that were 377 00:18:54,412 --> 00:18:56,567 designed specifically to work around that. 378 00:18:56,627 --> 00:19:00,287 Even if someone compromises your account, we still won't let them 379 00:19:00,287 --> 00:19:01,697 do the things that they want to do. 380 00:19:02,267 --> 00:19:03,047 Um, 381 00:19:03,317 --> 00:19:04,217 And, 382 00:19:04,277 --> 00:19:07,877 Prasanna Malaiyandi: and, even things like mfa, like we've seen a lot of recent 383 00:19:07,877 --> 00:19:11,087 attacks with MFA and other things, right? 384 00:19:11,087 --> 00:19:13,637 That isn't a fail safe, right? 385 00:19:13,637 --> 00:19:16,637 So you need to make sure you have all these various layers to handle 386 00:19:16,637 --> 00:19:20,447 the risk rather than just putting all your eggs in one basket, be like, Yep, 387 00:19:20,507 --> 00:19:22,457 that'll protect me cuz it will not. 388 00:19:24,347 --> 00:19:29,777 W. Curtis Preston: So, uh, the second big category, uh, is 389 00:19:29,837 --> 00:19:31,277 hardware and system failure. 390 00:19:31,877 --> 00:19:34,907 And I'm gonna go out there and I'm gonna see, see what you think 391 00:19:34,907 --> 00:19:35,867 about this cuz you've lived. 392 00:19:36,422 --> 00:19:38,672 The other side a little differently than me. 393 00:19:39,272 --> 00:19:44,522 Um, I'm gonna say, so when I started my career, this was, 394 00:19:45,182 --> 00:19:46,562 I don't know, but not equal. 395 00:19:46,832 --> 00:19:50,702 But it was a significant, it wasn't equal to the dumb user, right? 396 00:19:50,702 --> 00:19:54,092 Because we would get, we had 12,000 employees. 397 00:19:54,122 --> 00:19:57,862 We would do an average of 10 restores a day. 398 00:19:58,012 --> 00:19:59,687 From users calling in. 399 00:19:59,837 --> 00:20:03,077 My favorite was a user that called in and said that they needed a file to be 400 00:20:03,077 --> 00:20:10,297 restored and it was called resume.Doc. 401 00:20:11,597 --> 00:20:12,977 And I'm like, Really? 402 00:20:13,007 --> 00:20:14,417 Is that what that file is called? 403 00:20:14,417 --> 00:20:15,707 Resume dot doc? 404 00:20:15,707 --> 00:20:15,887 Yeah. 405 00:20:15,892 --> 00:20:17,687 It's really important that I have this restored. 406 00:20:17,727 --> 00:20:18,217 Okay. 407 00:20:18,557 --> 00:20:18,827 All right. 408 00:20:19,637 --> 00:20:22,697 Um, no, you know, no judgment, but, um, 409 00:20:24,137 --> 00:20:27,257 Prasanna Malaiyandi: You're like, Shall I restore it to an alternate location 410 00:20:27,467 --> 00:20:29,567 like another, a future employer, 411 00:20:30,272 --> 00:20:30,992 W. Curtis Preston: Yeah. 412 00:20:31,572 --> 00:20:33,172 Try to grok this. 413 00:20:33,202 --> 00:20:33,682 Okay. 414 00:20:34,012 --> 00:20:38,602 Back then we had servers running on hard drives, and by that I 415 00:20:38,602 --> 00:20:41,212 mean individual hard drives. 416 00:20:41,587 --> 00:20:41,947 Prasanna Malaiyandi: Yeah. 417 00:20:42,697 --> 00:20:43,147 W. Curtis Preston: Right. 418 00:20:43,567 --> 00:20:49,897 The, the most failing thing in the data center, we would have a server running the 419 00:20:49,897 --> 00:20:53,947 whole environment, and then the data would be stored on a series of hard drives. 420 00:20:54,097 --> 00:20:55,477 There was no raid. 421 00:20:56,377 --> 00:20:56,827 Right. 422 00:20:57,757 --> 00:21:00,337 And so a single, So if you 423 00:21:00,637 --> 00:21:02,677 Prasanna Malaiyandi: Drive failures, which are very, very high. 424 00:21:02,787 --> 00:21:05,917 W. Curtis Preston: Yeah, well, if you had like 10 drives, you didn't have you. 425 00:21:06,577 --> 00:21:09,367 You had 10 single points of failure, right? 426 00:21:09,727 --> 00:21:11,167 Wasn't just one single point of failure. 427 00:21:11,167 --> 00:21:12,787 You had 10 single points of failure. 428 00:21:13,117 --> 00:21:18,337 So back then, I think mechanical failure and system failure was a much bigger. 429 00:21:18,337 --> 00:21:23,107 Today we have virtualization and we have raid, and we have failover, 430 00:21:23,112 --> 00:21:24,997 and we have vMotion and all that 431 00:21:25,027 --> 00:21:25,327 Prasanna Malaiyandi: Yeah, 432 00:21:25,327 --> 00:21:25,387 W. Curtis Preston: I 433 00:21:25,537 --> 00:21:27,817 Prasanna Malaiyandi: You have erasure coding you have? 434 00:21:28,327 --> 00:21:28,687 Yeah. 435 00:21:29,467 --> 00:21:30,427 High availability, 436 00:21:30,427 --> 00:21:31,687 clustering, all the rest, 437 00:21:32,242 --> 00:21:34,702 W. Curtis Preston: We don't, generally it can happen. 438 00:21:34,882 --> 00:21:39,952 We don't generally restore because of that today. 439 00:21:41,797 --> 00:21:42,637 Prasanna Malaiyandi: typically, Yeah. 440 00:21:42,697 --> 00:21:45,217 It happens underneath the covers. 441 00:21:46,027 --> 00:21:49,477 Unless it's an application that isn't. 442 00:21:50,887 --> 00:21:54,877 Like a modern application, like a lot of environments still are running older 443 00:21:54,877 --> 00:21:57,697 applications that may not be virtualized. 444 00:21:58,057 --> 00:21:58,477 Right. 445 00:21:58,807 --> 00:22:00,457 And yes, it could, but 446 00:22:00,532 --> 00:22:03,592 W. Curtis Preston: But it's probably still stored on disc 447 00:22:03,622 --> 00:22:04,012 that's 448 00:22:04,432 --> 00:22:04,642 RAIDed 449 00:22:04,927 --> 00:22:05,377 Prasanna Malaiyandi: Yeah. 450 00:22:05,617 --> 00:22:09,277 So the storage side, you might be okay, but maybe the server side 451 00:22:09,277 --> 00:22:11,527 you might still have issues, Right? 452 00:22:11,527 --> 00:22:14,617 That you still have to be concerned about if, assuming it's not virtualized 453 00:22:15,607 --> 00:22:16,177 or you can't 454 00:22:16,462 --> 00:22:19,012 W. Curtis Preston: might go down, but I don't have to reach for my 455 00:22:19,017 --> 00:22:21,322 backups if the storage is still okay. 456 00:22:21,532 --> 00:22:21,922 Prasanna Malaiyandi: Yeah. 457 00:22:22,432 --> 00:22:23,302 W. Curtis Preston: That's all I'm saying. 458 00:22:23,402 --> 00:22:26,172 It's generally a human being did something stupid. 459 00:22:26,302 --> 00:22:29,172 Is the, is the, I, I'm gonna say the most common reason? 460 00:22:29,642 --> 00:22:31,712 Prasanna Malaiyandi: definitely the most common, The only thing I can think of 461 00:22:31,712 --> 00:22:37,022 from a hardware failure is there are still cases where you might get silent 462 00:22:37,022 --> 00:22:38,702 data, corruption and other aspects. 463 00:22:40,237 --> 00:22:40,412 Right. 464 00:22:40,622 --> 00:22:42,722 That you still might need to worry about. 465 00:22:42,722 --> 00:22:44,522 That isn't necessarily human error. 466 00:22:45,482 --> 00:22:47,672 Well, it's the developer error potentially. 467 00:22:47,702 --> 00:22:48,062 Right, 468 00:22:48,322 --> 00:22:48,812 W. Curtis Preston: Yeah. 469 00:22:48,812 --> 00:22:49,252 Yeah. 470 00:22:49,652 --> 00:22:49,832 Prasanna Malaiyandi: Right. 471 00:22:49,832 --> 00:22:52,322 But that's the only other thing I can think of where you, Yes. 472 00:22:52,472 --> 00:22:54,632 It's not necessarily that you have to think about the hardware 473 00:22:54,632 --> 00:22:58,922 itself failing, but there are still components within it, which might 474 00:22:58,922 --> 00:23:00,722 lead to data corruption and affecting. 475 00:23:02,057 --> 00:23:04,157 W. Curtis Preston: You know, it's interesting that you mentioned that. 476 00:23:04,367 --> 00:23:09,567 I'm not sure if I brought up, if I brought up, data corruption 477 00:23:09,567 --> 00:23:10,947 just happening over time. 478 00:23:11,177 --> 00:23:13,977 I, I think I'm gonna count that as system failure, as, 479 00:23:14,457 --> 00:23:16,137 as, as a hardware failure, right? 480 00:23:16,137 --> 00:23:18,787 Because it's the underlying mechanism. 481 00:23:19,257 --> 00:23:22,887 Um, I don't know if I mentioned that in the hardware failure section or not. 482 00:23:23,187 --> 00:23:24,297 Maybe I did, maybe I didn't. 483 00:23:24,387 --> 00:23:25,647 But, um, 484 00:23:25,722 --> 00:23:26,082 Prasanna Malaiyandi: There you go. 485 00:23:26,082 --> 00:23:28,182 For our listeners, it's something that isn't in the book. 486 00:23:28,242 --> 00:23:29,022 If it's not in there, 487 00:23:29,517 --> 00:23:31,377 W. Curtis Preston: Well, it may not, may not be in the book. 488 00:23:31,457 --> 00:23:31,757 Prasanna Malaiyandi: Yep. 489 00:23:31,976 --> 00:23:35,096 Having worked in storage for so long, it's one of the things that's always on 490 00:23:35,096 --> 00:23:38,396 my mind, even though I think a lot of people don't think about it, cuz there's 491 00:23:38,396 --> 00:23:43,256 a lot of scrubbing that happens and all the rest that it's not as likely anymore. 492 00:23:43,346 --> 00:23:45,926 But it's not that it's been eliminated. 493 00:23:46,886 --> 00:23:50,696 W. Curtis Preston: There, uh, there are those who think that, that, that bit 494 00:23:50,746 --> 00:23:54,746 rot is a, is a boogeyman that people like you and me use to scare people. 495 00:23:55,826 --> 00:23:58,826 Um, those people can go pound sand. 496 00:23:59,696 --> 00:24:05,606 So the next we'll talk about again, something that's very current. 497 00:24:06,836 --> 00:24:11,156 So I am from Florida, originally from Florida. 498 00:24:11,156 --> 00:24:11,996 I live in, I live. 499 00:24:12,701 --> 00:24:13,541 Much better weather. 500 00:24:13,541 --> 00:24:15,461 Now, down in, in, 501 00:24:15,461 --> 00:24:18,071 Sango, it's relative 502 00:24:18,161 --> 00:24:19,331 Um, yeah. 503 00:24:19,331 --> 00:24:21,191 My, my mother does not like the weather here. 504 00:24:21,191 --> 00:24:22,241 She thinks it's too cold. 505 00:24:22,751 --> 00:24:25,871 The idea of getting cold, you know, cool at night is she's, 506 00:24:26,411 --> 00:24:27,461 does not like that at all. 507 00:24:28,061 --> 00:24:31,331 I think that's the single greatest feature, possibly of sun ago. 508 00:24:31,571 --> 00:24:36,221 Um, but I'm speaking of course, of, of natural disasters. 509 00:24:36,221 --> 00:24:38,711 So we're recording this as they are. 510 00:24:39,941 --> 00:24:46,151 Recovering from the fourth strongest hurricane ever to hit landfall in 511 00:24:46,151 --> 00:24:48,701 the us, which is Hurricane Ian. 512 00:24:49,781 --> 00:24:53,021 I, you know, it was, it was 150 miles an hour. 513 00:24:53,021 --> 00:24:55,151 I've been in worse. 514 00:24:55,961 --> 00:25:02,261 I was actually in Houston, and I don't know where Alicia falls, but what 515 00:25:02,261 --> 00:25:04,901 I remember was 155 miles an hour. 516 00:25:05,531 --> 00:25:06,011 Prasanna Malaiyandi: Hmm. 517 00:25:06,536 --> 00:25:09,626 W. Curtis Preston: Hurricane force winds with Alicia and I was in Houston. 518 00:25:09,626 --> 00:25:10,796 So you're basically right. 519 00:25:10,946 --> 00:25:12,176 You're right there on the water. 520 00:25:12,266 --> 00:25:12,626 Right? 521 00:25:13,166 --> 00:25:14,616 Um, and. 522 00:25:16,091 --> 00:25:17,411 That was a mess. 523 00:25:17,471 --> 00:25:17,891 Right? 524 00:25:17,891 --> 00:25:22,391 I remember I was in, I was, well, I was in Alvin, which, um, 525 00:25:22,781 --> 00:25:24,851 shout out to the folks in Alvin. 526 00:25:25,601 --> 00:25:30,551 It's a tiny little town that's a suburb of Houston, and my mother and her 527 00:25:30,551 --> 00:25:33,371 husband managed an apartment complex. 528 00:25:33,426 --> 00:25:34,461 I was a teenager. 529 00:25:34,491 --> 00:25:37,671 There wasn't enough plywood to cover the windows on both sides. 530 00:25:37,671 --> 00:25:42,171 So I remember that we covered one side based on the direction of the wind, and 531 00:25:42,171 --> 00:25:46,581 then when the, when the, uh, eye hit, we took down all the plywood and put 532 00:25:46,581 --> 00:25:47,811 it on the other side of the building. 533 00:25:48,501 --> 00:25:53,421 Um, the, just trying to minimize the, you know, the blast radius 534 00:25:53,421 --> 00:25:55,431 as we talk about in it, right. 535 00:25:56,856 --> 00:26:00,966 And the thing is, when you, you know, we've got a couple of great episodes about 536 00:26:00,966 --> 00:26:07,866 this, about recovering from a natural disaster, um, from, from people that 537 00:26:07,866 --> 00:26:13,416 actually participated in it, and maybe we'll get another one here from Florida. 538 00:26:13,836 --> 00:26:19,836 But the, the challenges when you look at. 539 00:26:20,286 --> 00:26:23,616 You know, so I'm looking here at the list that I had in the book. 540 00:26:23,616 --> 00:26:29,436 Floods, fires, earthquakes, hurricanes, uh, typhoons and cyclones. 541 00:26:29,436 --> 00:26:30,636 Tornadoes and sinkholes. 542 00:26:30,906 --> 00:26:35,106 So what's interesting is Florida has all but the earthquakes, 543 00:26:38,076 --> 00:26:40,086 so well, it doesn't have. 544 00:26:41,031 --> 00:26:41,871 Prasanna Malaiyandi: Typhoons? 545 00:26:42,471 --> 00:26:43,731 W. Curtis Preston: a typhoon goes the other way. 546 00:26:43,731 --> 00:26:44,121 Right? 547 00:26:44,181 --> 00:26:46,491 That's what it isn't that what the thing was, the typhoon. 548 00:26:47,001 --> 00:26:50,511 Um, and remember we had another name of storm I, I never 549 00:26:50,511 --> 00:26:54,681 even heard of until we had the, the Derecho, um, which is a 550 00:26:55,401 --> 00:26:55,701 Prasanna Malaiyandi: Land 551 00:26:55,821 --> 00:26:57,231 W. Curtis Preston: based hurricane. 552 00:26:57,321 --> 00:26:57,501 Prasanna Malaiyandi: Yep. 553 00:26:57,531 --> 00:27:03,141 You basically get like a, the giant pattern and the high sustained winds, so 554 00:27:03,141 --> 00:27:05,031 it basically becomes a land hurricane. 555 00:27:05,871 --> 00:27:08,751 W. Curtis Preston: Yeah, we, The guy that was on here, I remember him saying 556 00:27:08,751 --> 00:27:11,961 that he was just standing on his porch and he's like, I'm sorry, what is that? 557 00:27:12,831 --> 00:27:14,031 like just outta nowhere. 558 00:27:14,616 --> 00:27:14,886 Prasanna Malaiyandi: Yeah, 559 00:27:16,011 --> 00:27:22,221 W. Curtis Preston: Um, the, the thing about all of these is that they wipe 560 00:27:22,226 --> 00:27:25,941 out everything, you know, look at the photos, look at the videos of 561 00:27:25,941 --> 00:27:28,821 what's happened in, um, in Florida. 562 00:27:28,891 --> 00:27:32,581 Fort Myers right now is, Um, devastated. 563 00:27:33,051 --> 00:27:37,771 I think the worst pictures of videos I remember was Katrina, right? 564 00:27:37,771 --> 00:27:42,631 Because you had the combination of all of this, plus the fact that New 565 00:27:42,631 --> 00:27:44,431 Orleans was essentially underwater. 566 00:27:44,626 --> 00:27:44,956 Prasanna Malaiyandi: Yeah. 567 00:27:45,661 --> 00:27:47,811 W. Curtis Preston: They were, they were below sea level, right? 568 00:27:48,031 --> 00:27:51,331 And so you see the pictures of Fort Myers, now everything's 569 00:27:51,331 --> 00:27:52,861 destroyed, but the water's gone. 570 00:27:53,461 --> 00:27:57,091 Right now back with Katrina, it was still, the water was 571 00:27:57,096 --> 00:27:57,391 still 572 00:27:57,391 --> 00:27:57,631 up. 573 00:27:58,906 --> 00:28:01,606 You know, bodies were literally still floating around it. 574 00:28:01,696 --> 00:28:03,226 It was horrible. 575 00:28:03,706 --> 00:28:07,156 Um, and it, and it, and it made rescue very difficult cuz it meant everything 576 00:28:07,156 --> 00:28:08,596 had to be boats and helicopters. 577 00:28:08,596 --> 00:28:08,866 Right. 578 00:28:09,886 --> 00:28:15,286 Um, but I think that's the number one thing to take away when 579 00:28:15,586 --> 00:28:18,666 preparing for a natural disaster. 580 00:28:18,796 --> 00:28:21,866 Is that nothing that you're used to counting on. 581 00:28:22,856 --> 00:28:23,346 Prasanna Malaiyandi: Yeah. 582 00:28:23,836 --> 00:28:24,286 I assume it's 583 00:28:24,286 --> 00:28:24,826 all gone. 584 00:28:24,976 --> 00:28:25,426 Yeah. 585 00:28:26,776 --> 00:28:29,956 W. Curtis Preston: So that was what I think we learned with that. 586 00:28:30,286 --> 00:28:32,926 That person there that came on. 587 00:28:33,136 --> 00:28:36,376 Um, do you remember he, he talked about the internet. 588 00:28:37,906 --> 00:28:38,236 Prasanna Malaiyandi: Yeah. 589 00:28:38,236 --> 00:28:43,636 I think if I recall, they had an ISP or an msp, and they basically 590 00:28:43,636 --> 00:28:47,656 were like our employees needed access, internet access to make sure 591 00:28:47,656 --> 00:28:51,256 things were okay in supporting their customers, and they had no internet. 592 00:28:51,256 --> 00:28:55,246 So I think people drove like an hour and a half, two hours away to like a 593 00:28:55,246 --> 00:28:59,686 Starbucks, so then they could log in just so they can continue and have 594 00:28:59,686 --> 00:29:02,956 access because they needed to keep their business up and running and that 595 00:29:02,956 --> 00:29:04,246 was the only way they could keep it. 596 00:29:05,566 --> 00:29:07,756 W. Curtis Preston: That that's, that's one problem that they had. 597 00:29:07,756 --> 00:29:14,386 The other one that they had was that they used LDAP that was headquartered in the US 598 00:29:14,491 --> 00:29:14,911 Prasanna Malaiyandi: Yeah. 599 00:29:15,526 --> 00:29:18,376 W. Curtis Preston: as their authentication system in order to get into the 600 00:29:18,376 --> 00:29:20,146 system, in order to do the restore, 601 00:29:20,611 --> 00:29:22,051 Prasanna Malaiyandi: Oh, that was, Sorry. 602 00:29:22,081 --> 00:29:24,331 That was, Yes, that was the, Sorry. 603 00:29:24,331 --> 00:29:26,431 The one I was talking about was the land hurricane one. 604 00:29:26,701 --> 00:29:27,061 You're right. 605 00:29:27,106 --> 00:29:28,016 W. Curtis Preston: Oh, oh, right. 606 00:29:28,021 --> 00:29:28,126 Right. 607 00:29:28,351 --> 00:29:30,361 Prasanna Malaiyandi: a hurricane on a tropical island. 608 00:29:31,051 --> 00:29:31,501 W. Curtis Preston: Right. 609 00:29:31,501 --> 00:29:31,951 Yeah. 610 00:29:32,701 --> 00:29:36,451 And the, we, we should put links, I think in the, in the show 611 00:29:36,451 --> 00:29:37,801 description to these episodes. 612 00:29:38,041 --> 00:29:39,241 That was a really good episode. 613 00:29:39,301 --> 00:29:42,211 Like if you wanna know what it's like to actually recover from a 614 00:29:42,211 --> 00:29:46,531 hurricane, Cause he talked about that nothing was consistent, right? 615 00:29:46,741 --> 00:29:49,321 That, you know, they, they could, they didn't have consistent power, 616 00:29:49,321 --> 00:29:50,731 They didn't have consistent internet. 617 00:29:50,791 --> 00:29:53,431 They, they ended up getting consistent internet by doing, 618 00:29:53,461 --> 00:29:55,951 uh, you know, satellite-based. 619 00:29:56,836 --> 00:29:57,286 Internet. 620 00:29:57,316 --> 00:29:57,706 Right. 621 00:29:58,216 --> 00:29:59,656 And they didn't need a lot of bandwidth. 622 00:29:59,656 --> 00:30:01,006 They just needed consistency. 623 00:30:01,231 --> 00:30:01,501 Prasanna Malaiyandi: Yeah. 624 00:30:01,546 --> 00:30:02,716 W. Curtis Preston: Um, They 625 00:30:02,791 --> 00:30:04,651 Prasanna Malaiyandi: like food and where to sleep. 626 00:30:04,651 --> 00:30:04,921 Right. 627 00:30:04,921 --> 00:30:06,931 I think he slept in the office 628 00:30:07,666 --> 00:30:07,876 W. Curtis Preston: Yeah. 629 00:30:07,876 --> 00:30:10,576 He converted one of the offices into a room and they, he 630 00:30:10,576 --> 00:30:11,836 ate a lot of rice and beans 631 00:30:11,941 --> 00:30:12,271 Prasanna Malaiyandi: Yeah. 632 00:30:12,316 --> 00:30:12,616 W. Curtis Preston: weeks. 633 00:30:12,616 --> 00:30:12,676 He 634 00:30:12,676 --> 00:30:12,976 was there 635 00:30:12,991 --> 00:30:15,271 Prasanna Malaiyandi: and he, and he said he was lucky, right? 636 00:30:15,276 --> 00:30:18,631 I think a lot of people didn't have food, didn't have clean water, 637 00:30:19,296 --> 00:30:19,856 W. Curtis Preston: right. 638 00:30:20,116 --> 00:30:20,506 right. 639 00:30:20,566 --> 00:30:26,011 Well, you look at what's going on now, I mean, Puerto Rico, you know, 640 00:30:26,011 --> 00:30:27,301 we talked about a lot about Florida. 641 00:30:27,301 --> 00:30:31,261 Puerto Rico and Cuba are massively hit, right? 642 00:30:31,591 --> 00:30:34,441 Because they, cuz they're getting hit on both sides at the same time. 643 00:30:34,446 --> 00:30:35,671 The place is so small. 644 00:30:35,701 --> 00:30:36,091 Right. 645 00:30:36,791 --> 00:30:40,931 Puerto Rico especially because they had already been hit by that 646 00:30:41,031 --> 00:30:42,941 other hurricane not that long ago. 647 00:30:44,591 --> 00:30:49,961 Um, and you know, it's, it's funny, um, you know, our friends over there at actual 648 00:30:49,961 --> 00:30:54,761 Tech media, Scott Lowe and company, so they're having their 10th anniversary. 649 00:30:54,766 --> 00:30:56,531 Congratulations to them, right? 650 00:30:57,071 --> 00:31:04,631 Um, they scheduled an offsite for their 10th anniversary in October in 651 00:31:04,791 --> 00:31:08,261 Florida, and they're there this week. 652 00:31:09,481 --> 00:31:12,851 Right now they're in Central Florida, they're in Orlando. 653 00:31:12,971 --> 00:31:13,841 They're fine. 654 00:31:14,111 --> 00:31:18,191 But I just, you know, I wish Scott had reached out to me and said, Hey, 655 00:31:18,191 --> 00:31:21,971 I'm thinking about doing a thing in Florida, You know, when should I not go? 656 00:31:21,971 --> 00:31:23,711 And I would've said, September, October. 657 00:31:24,041 --> 00:31:24,521 Don't go. 658 00:31:24,521 --> 00:31:25,241 anywhere near, 659 00:31:25,721 --> 00:31:26,111 Prasanna Malaiyandi: Yep. 660 00:31:26,261 --> 00:31:27,071 W. Curtis Preston: don't go anywhere 661 00:31:27,341 --> 00:31:28,781 Prasanna Malaiyandi: Weather's unpredictable. 662 00:31:28,781 --> 00:31:29,111 You never 663 00:31:29,411 --> 00:31:29,771 W. Curtis Preston: Yeah. 664 00:31:29,771 --> 00:31:32,651 It's the, it's the, it's the, you know, it's a hurricane season. 665 00:31:33,161 --> 00:31:36,701 But yeah, that's the big thing is this, but this is why we back up. 666 00:31:39,191 --> 00:31:46,361 A, a, a natural disaster, um, can just wipe out everything 667 00:31:46,366 --> 00:31:47,471 in your company, right? 668 00:31:47,476 --> 00:31:50,381 It, it wipe out, it wipes out the building, it wipes out the 669 00:31:50,381 --> 00:31:51,461 power to get to your building. 670 00:31:51,461 --> 00:31:53,051 It wipes out the internet to your building. 671 00:31:54,581 --> 00:31:57,851 Um, and you have to bring all of that back. 672 00:31:58,223 --> 00:32:02,175 Prasanna Malaiyandi: or you don't bring it back in the current location, 673 00:32:02,175 --> 00:32:03,495 you bring it back somewhere else. 674 00:32:03,495 --> 00:32:03,735 Right. 675 00:32:03,735 --> 00:32:07,035 I know we talk a lot about disaster recovery, but even 676 00:32:07,040 --> 00:32:08,595 without disaster recovery, right? 677 00:32:08,895 --> 00:32:10,785 This is where the 3 21 rule, right? 678 00:32:10,785 --> 00:32:13,485 The one part, right, of keeping a copy off site. 679 00:32:13,490 --> 00:32:17,415 Because if your data center gets hit by a hurricane and is 680 00:32:17,415 --> 00:32:19,905 completely gone, where is your data? 681 00:32:19,905 --> 00:32:21,615 At least your data should be offsite. 682 00:32:21,945 --> 00:32:23,625 Someplace far enough away, right? 683 00:32:23,625 --> 00:32:24,195 Not down the. 684 00:32:25,410 --> 00:32:26,280 Such that you 685 00:32:26,280 --> 00:32:27,300 can recover. 686 00:32:27,375 --> 00:32:29,295 W. Curtis Preston: If you're, in Florida, I don't know where the AWS 687 00:32:29,295 --> 00:32:34,455 regions are for the southeast, but I don't know if any of 'em are in Florida 688 00:32:35,250 --> 00:32:35,490 Prasanna Malaiyandi: I 689 00:32:35,535 --> 00:32:37,845 W. Curtis Preston: But if you're in Florida, you would not have 690 00:32:37,845 --> 00:32:41,115 your recovery facility, I would think, anywhere in Florida. 691 00:32:41,115 --> 00:32:43,065 And that's what, that's what's beautiful about that cloud, right, 692 00:32:43,365 --> 00:32:47,265 is you know, this is why we back up and this is also why we don't store. 693 00:32:48,150 --> 00:32:51,150 At least one copy of our data anywhere near Right. 694 00:32:51,270 --> 00:32:56,860 Um, the, I, I, I think I flashback again, back to MBNA, 695 00:32:58,350 --> 00:33:01,650 and we didn't use Iron Mountain. 696 00:33:01,710 --> 00:33:05,160 We used a small, 697 00:33:05,730 --> 00:33:06,080 um, 698 00:33:06,280 --> 00:33:08,100 media storage company. 699 00:33:08,160 --> 00:33:08,340 Huh? 700 00:33:08,400 --> 00:33:08,850 What's that? 701 00:33:09,180 --> 00:33:11,160 Yeah, Man man, Yeah. 702 00:33:11,520 --> 00:33:14,940 We used a small, uh, company that was in Wilmington. 703 00:33:15,690 --> 00:33:19,080 The thing that we liked about this company, they were probably cheaper 704 00:33:19,320 --> 00:33:23,280 than Iron Mountain, but the thing we liked about this company is that their 705 00:33:23,280 --> 00:33:30,480 tapes were stored in what used to be a World War II bomb shelter, right? 706 00:33:30,750 --> 00:33:33,570 So we felt that they were like really, really safe, but. 707 00:33:34,780 --> 00:33:38,175 . Um, it also meant that anytime there was a hurricane or anything 708 00:33:38,175 --> 00:33:42,735 or flood threatening to come up the eastern seaboard, we would call them. 709 00:33:43,695 --> 00:33:45,135 We'd say it's time for the thing. 710 00:33:45,435 --> 00:33:49,725 And the thing was, they would take all our tapes out of the vault and 711 00:33:49,730 --> 00:33:51,405 move them up to the third floor. 712 00:33:51,945 --> 00:33:55,935 Much less physically secure, but a lot less susceptible to flooding 713 00:33:56,055 --> 00:33:58,545 Prasanna Malaiyandi: Yeah, I was just gonna say, yeah, bomb shelter sounds 714 00:33:58,545 --> 00:34:00,915 amazing until you're dealing with water 715 00:34:02,250 --> 00:34:02,730 W. Curtis Preston: Yeah. 716 00:34:02,820 --> 00:34:03,270 Yeah. 717 00:34:03,390 --> 00:34:07,830 Um, I, I highly doubt that they're still using that company, 718 00:34:07,830 --> 00:34:10,980 but, um, anyway, I mean, yeah. 719 00:34:10,980 --> 00:34:12,690 So these are the reasons, right? 720 00:34:13,020 --> 00:34:18,750 What about, you know, we, we, we really didn't mention terrorism, 721 00:34:18,755 --> 00:34:21,900 but that's up in the, you know, in the human, the human disasters. 722 00:34:22,920 --> 00:34:24,480 That, that that's a real thing. 723 00:34:24,720 --> 00:34:27,960 Nowadays, I think you're more susceptible to electronic terrorism 724 00:34:27,960 --> 00:34:29,430 than you are a physical terrorism. 725 00:34:29,610 --> 00:34:32,160 We can't completely rule that out, 726 00:34:32,370 --> 00:34:32,730 Prasanna Malaiyandi: Yeah. 727 00:34:33,360 --> 00:34:38,820 W. Curtis Preston: but I think that, I think that the place to start is you 728 00:34:38,825 --> 00:34:43,140 first have to build a DR system that works, that would allow you to recover. 729 00:34:43,800 --> 00:34:46,830 Outside of the, you know, make that part of your core design. 730 00:34:47,160 --> 00:34:53,040 And then the next would be you need to build a ransomware, you know, 731 00:34:53,040 --> 00:34:56,190 electronic attack response system. 732 00:34:56,220 --> 00:34:59,610 Because it's, it's the DR piece is, is a subset of that. 733 00:34:59,690 --> 00:35:03,830 So, you know, to, to just summarize, I think the whole thing is that there, 734 00:35:03,860 --> 00:35:12,080 there are, um, there are just so many things that can happen to your data 735 00:35:12,170 --> 00:35:15,200 that is your company's lifeblood. 736 00:35:16,790 --> 00:35:22,130 That's why we back up and, and, you know, and I'll borrow a line from Shakespeare. 737 00:35:23,405 --> 00:35:26,735 Uh, that I use every once in a while when we're talking about some, we're 738 00:35:26,740 --> 00:35:30,515 gonna talk about the, the places that we need to back up, I think next. 739 00:35:31,405 --> 00:35:35,905 When I get to arguing with people about whether or not I should back up SaaS, 740 00:35:36,655 --> 00:35:42,655 one of the phrases that I, that I, um, That I like to use is there are more 741 00:35:42,655 --> 00:35:47,065 things in heaven and earth than are dreamt of in your philosophy Right. 742 00:35:47,125 --> 00:35:48,325 I think I quoted that. 743 00:35:48,475 --> 00:35:49,945 I hope I quoted that well. 744 00:35:50,065 --> 00:35:56,365 But basically there are so many ways that your data can get, uh, uh, fubar. 745 00:35:56,485 --> 00:35:58,915 That's a, that's a military acronym. 746 00:35:58,920 --> 00:35:59,815 You're familiar with that one? 747 00:35:59,905 --> 00:36:00,175 Prasanna Malaiyandi: Yep. 748 00:36:00,175 --> 00:36:00,865 I know that one. 749 00:36:01,555 --> 00:36:05,305 W. Curtis Preston: Um, and look it up if you dunno what that means. 750 00:36:05,875 --> 00:36:08,005 Um, and. 751 00:36:08,650 --> 00:36:15,550 That's why we back up and that's why, you know, Yeah, I, I I don't know. 752 00:36:15,550 --> 00:36:17,920 I don't, I don't suffer the fools that want it, that 753 00:36:18,010 --> 00:36:21,340 Prasanna Malaiyandi: Well, well, and I know back to the basics, right? 754 00:36:21,340 --> 00:36:24,460 We've been talking a lot about in a corporate environment, 755 00:36:24,460 --> 00:36:28,330 but these same risks also apply to your personal data as well. 756 00:36:28,420 --> 00:36:30,040 W. Curtis Preston: every single one. 757 00:36:30,580 --> 00:36:30,910 Prasanna Malaiyandi: Yep. 758 00:36:32,035 --> 00:36:33,325 W. Curtis Preston: And some more than others. 759 00:36:33,655 --> 00:36:36,505 Mechanical failure, much more likely in your laptop. 760 00:36:37,015 --> 00:36:40,555 Um, less likely today than it was 20 years ago when we were 761 00:36:40,555 --> 00:36:41,965 using rotating hard drives. 762 00:36:42,865 --> 00:36:45,535 Um, and yeah, still possible. 763 00:36:45,535 --> 00:36:51,415 I will just say this, the most unreliable piece of equipment on your desk is 764 00:36:51,415 --> 00:36:57,685 that mechanical, USB hard drive that you're using as a backup right now. 765 00:36:59,350 --> 00:37:00,370 Tell me I'm wrong. 766 00:37:01,035 --> 00:37:01,525 Prasanna Malaiyandi: Yeah. 767 00:37:01,750 --> 00:37:02,860 W. Curtis Preston: Tell me I'm wrong. 768 00:37:03,340 --> 00:37:03,790 Yeah. 769 00:37:04,360 --> 00:37:07,750 That, that's why, that's why I'm, I'm much more of a fan of, 770 00:37:07,780 --> 00:37:09,070 of, you know, of cloud backup. 771 00:37:09,070 --> 00:37:12,490 And by the way, you know, you wouldn't be a Druva customer, because, you 772 00:37:12,490 --> 00:37:15,700 know, we don't do the, the consumer thing, but there are so many other 773 00:37:15,705 --> 00:37:17,290 things that you do for consumers, right? 774 00:37:17,740 --> 00:37:20,680 There are cloud services to backup your important stuff. 775 00:37:20,860 --> 00:37:22,120 Um, yeah. 776 00:37:22,390 --> 00:37:26,020 Anyway, well have we, have, we beat the drum enough 777 00:37:26,235 --> 00:37:26,380 Prasanna Malaiyandi: I 778 00:37:26,380 --> 00:37:27,220 W. Curtis Preston: Prasanna? 779 00:37:28,135 --> 00:37:32,725 Prasanna Malaiyandi: And just for our listeners, I think the next topic on 780 00:37:32,725 --> 00:37:37,765 back to the basics will be what, Curtis, what are we, where do we go from here? 781 00:37:38,005 --> 00:37:38,845 Where do we go from here? 782 00:37:39,485 --> 00:37:40,085 W. Curtis Preston: Good question. 783 00:37:40,085 --> 00:37:41,760 We talk a lot about some techy stuff. 784 00:37:42,270 --> 00:37:50,460 One of the things that you hear us say a lot is that you, um, you need to go to 785 00:37:50,460 --> 00:37:56,630 the business and you need to get your, um, you know, your service level agreements 786 00:37:56,630 --> 00:38:01,130 and things like RTO and RPO . You need to get that from the business, and you 787 00:38:01,135 --> 00:38:04,400 need to get the business to buy into this whole thing that you're gonna do, 788 00:38:05,000 --> 00:38:07,010 because you can't, you can't be, you. 789 00:38:07,640 --> 00:38:10,820 Batman is not a good backup person, right? 790 00:38:10,820 --> 00:38:15,380 You can't be alone in the night, uh, with no funding doing a backup system. 791 00:38:15,380 --> 00:38:16,250 It's a horrible way to do. 792 00:38:16,250 --> 00:38:17,540 You gotta get the backup. 793 00:38:17,870 --> 00:38:23,020 You gotta get the, the powers that be as into backup as you are. 794 00:38:23,970 --> 00:38:24,160 , Right? 795 00:38:24,290 --> 00:38:28,310 Maybe not as, as into backup as Daniel Rosehill. 796 00:38:28,370 --> 00:38:29,180 Um, 797 00:38:29,405 --> 00:38:30,155 Prasanna Malaiyandi: We love you Daniel 798 00:38:31,070 --> 00:38:31,850 W. Curtis Preston: I love you, Daniel. 799 00:38:32,300 --> 00:38:35,910 The backup, anarak, so that's what we're gonna talk about next 800 00:38:35,910 --> 00:38:38,400 is how to, how to bring, how to bring the business into this. 801 00:38:39,000 --> 00:38:40,200 Um, so good. 802 00:38:40,200 --> 00:38:41,160 So thanks for asking me that. 803 00:38:41,160 --> 00:38:42,370 It was a good, it's a good follow. 804 00:38:43,920 --> 00:38:47,940 All right, and I'll just, again, I'll mention, um, if you're interested in 805 00:38:47,940 --> 00:38:51,870 all these topics, you can get a free copy, uh, you know, an ebook copy 806 00:38:51,900 --> 00:38:59,040 of my book, latest book, Modern Data Protection by going to druva.com. 807 00:38:59,330 --> 00:39:04,890 Druva.com/podcast and, um, uh, download your own copy so you can follow along 808 00:39:04,895 --> 00:39:13,550 at Home Kids And, um, remember to subscribe so that you can restore it all