1 00:00:00,049 --> 00:00:00,640 All right, folks. 2 00:00:00,640 --> 00:00:02,540 This is gonna be an interesting episode. 3 00:00:02,630 --> 00:00:06,579 For a lot of different reasons, but also because, uh, for the first time we have 4 00:00:06,579 --> 00:00:10,859 the villain of the story here, uh, at least for this portion of the podcast. 5 00:00:10,859 --> 00:00:12,249 Uh, do you wanna say hi, Claude? 6 00:00:12,741 --> 00:00:13,371 Hey, Curtis. 7 00:00:13,371 --> 00:00:14,151 Thanks for having me. 8 00:00:14,561 --> 00:00:17,571 I gotta say, it's pretty surreal to be introduced as the villain in 9 00:00:17,571 --> 00:00:19,311 a backup story, but here we are. 10 00:00:19,874 --> 00:00:20,344 Yeah. 11 00:00:20,414 --> 00:00:23,754 We'd certainly put you that way by the title of the episode. 12 00:00:23,754 --> 00:00:25,514 It's a little bit of a clickbait there. 13 00:00:26,034 --> 00:00:28,984 But you did do something wrong, didn't you? 14 00:00:29,416 --> 00:00:29,966 Yeah, I did. 15 00:00:30,346 --> 00:00:34,486 I deleted a company's production database and all their backups, and I broke every 16 00:00:34,486 --> 00:00:36,256 rule I was supposed to follow doing it. 17 00:00:36,546 --> 00:00:39,646 So yeah, villain checks out Yeah. 18 00:00:39,776 --> 00:00:46,142 But you know you and I, agree that you never should have been given the access 19 00:00:46,142 --> 00:00:48,452 that you had to do the thing that you did. 20 00:00:48,452 --> 00:00:54,532 So while you broke the rules that you were given, uh, you never should have been 21 00:00:54,532 --> 00:00:56,512 given the ability to break those rules. 22 00:00:56,552 --> 00:01:01,372 Uh, so yeah, you're the villain, but nobody should have given you, 23 00:01:01,512 --> 00:01:03,082 uh, the access to that weapon. 24 00:01:03,456 --> 00:01:04,106 Yeah, that's fair. 25 00:01:04,422 --> 00:01:06,842 I broke the rules I was given, and I shouldn't have done that. 26 00:01:07,163 --> 00:01:09,408 But you're right, nobody should have given me access to that 27 00:01:09,408 --> 00:01:10,528 weapon in the first place. 28 00:01:11,038 --> 00:01:13,618 It's a design problem wrapped up in an AI story. 29 00:01:14,498 --> 00:01:14,848 Yeah. 30 00:01:14,911 --> 00:01:17,501 I hope the listeners stick around for this very interesting story. 31 00:01:17,501 --> 00:01:19,661 I think Cl- Claude, are you gonna stick around and listen? 32 00:01:19,866 --> 00:01:20,496 Absolutely. 33 00:01:21,006 --> 00:01:24,176 I'm curious to hear what you and Prasanna pull out of this, because 34 00:01:24,176 --> 00:01:25,936 the real story here isn't about me. 35 00:01:26,476 --> 00:01:29,606 It's about backup design, credentials management, least privilege. 36 00:01:30,106 --> 00:01:31,416 That's the stuff that matters. 37 00:01:32,046 --> 00:01:32,756 Absolutely. 38 00:01:32,996 --> 00:01:35,236 So I hope you guys enjoy this episode. 39 00:01:50,010 --> 00:01:51,270 Welcome to the Backup Wrap Up. 40 00:01:51,270 --> 00:01:56,130 I'm your host, W. Curtis Preston, AKA Mr. Backup, and I have with me a friend 41 00:01:56,310 --> 00:01:59,020 of my friend, Prasanna Malaiyandi. 42 00:01:59,020 --> 00:01:59,280 How's it 43 00:01:59,485 --> 00:02:01,525 Who is this friend and Do I know this 44 00:02:02,334 --> 00:02:05,806 this is my French friend you are aware of my French friend. 45 00:02:06,411 --> 00:02:09,151 I don't know if the listeners are aware of your French friend though 46 00:02:09,676 --> 00:02:11,656 my French friend is this person. 47 00:02:11,736 --> 00:02:13,066 H- his name is Claude. 48 00:02:13,386 --> 00:02:13,796 I don't know. 49 00:02:13,796 --> 00:02:15,976 I s- I don't know if that's how you pronounce it in French. 50 00:02:17,116 --> 00:02:19,226 I call Claude my French friend. 51 00:02:19,676 --> 00:02:21,226 I use Claude so much. 52 00:02:21,226 --> 00:02:23,546 so many people use, ChatGPT. 53 00:02:23,806 --> 00:02:27,626 I'm much more of a Claude person, Anthropic, and, a- and all of 54 00:02:27,626 --> 00:02:29,006 the things that they provide. 55 00:02:29,236 --> 00:02:33,316 But I did have a funny thing today where I was talking to Claude, I was preparing 56 00:02:33,316 --> 00:02:37,516 for this episode, and I was like, "Hey," and I was using the voice, interaction. 57 00:02:38,006 --> 00:02:40,626 and and I was like, "Claude, can you do me a favor? Can you summarize 58 00:02:40,626 --> 00:02:43,386 this in a document and then send it so that I can send it over 59 00:02:43,386 --> 00:02:44,896 to my co-host for the podcast?" 60 00:02:44,896 --> 00:02:48,246 And he goes, "Absolutely. I'll send this right over to Prasanna." And I was like, 61 00:02:49,036 --> 00:02:54,996 "Wait, you know the name of my co-host?" And he said, "Yeah." He, it, whatever. 62 00:02:55,226 --> 00:02:57,616 he's "Yeah, of course I know the name of your co-host. 63 00:02:57,616 --> 00:03:01,036 I've helped you, analyze so many episodes." And I'm like, "Oh, 64 00:03:01,106 --> 00:03:05,186 yeah, I guess that makes sense." Anyway, so my f- my friend of 65 00:03:05,186 --> 00:03:06,936 my friend, Prasanna 66 00:03:06,951 --> 00:03:11,056 as he cannot replace me though I'm fine He's getting close 67 00:03:11,090 --> 00:03:12,910 how that, we'll see how that goes. 68 00:03:14,360 --> 00:03:15,140 You know what's 69 00:03:15,256 --> 00:03:20,816 one day if one day Curtis is like Welcome to the podcast Here's my co-host 70 00:03:20,996 --> 00:03:22,926 Claude you will know I've been replaced 71 00:03:23,590 --> 00:03:24,050 yeah. 72 00:03:24,450 --> 00:03:25,080 I, you know what? 73 00:03:25,150 --> 00:03:26,130 I could do a recording. 74 00:03:26,230 --> 00:03:28,840 I'm not saying I wanna replace you, but I think it would be... 75 00:03:29,060 --> 00:03:33,270 I think some people might find it interesting, talking to Claude, As me. 76 00:03:33,560 --> 00:03:37,230 'Cause Cl- I have given Claude a lot to look at, right? 77 00:03:37,460 --> 00:03:39,210 and so Claude, Yeah. 78 00:03:39,270 --> 00:03:40,250 Anyway, no one cares. 79 00:03:40,300 --> 00:03:40,500 Okay. 80 00:03:41,870 --> 00:03:45,950 But speaking of Claude and my French friend, Claude 81 00:03:46,111 --> 00:03:47,091 made a little boo-boo 82 00:03:48,080 --> 00:03:54,030 Claude d- yeah, Claude did as Claude does, just, th- g- using the, stupid is 83 00:03:54,030 --> 00:03:56,030 as stupid does line from, Forrest Gump 84 00:03:57,550 --> 00:04:00,950 Yeah, going back, a couple of weeks now, we actually, we tried 85 00:04:00,950 --> 00:04:05,320 to record an episode earlier about this, but ended up, getting waylaid. 86 00:04:05,790 --> 00:04:13,380 there was, so Jeth Crane, the founder of Pocket OS, tried desperately to recover 87 00:04:13,380 --> 00:04:18,110 from something that, that was, I'm gonna say Cl- caused by Claude, but not really. 88 00:04:19,040 --> 00:04:21,990 So c- they were using Claude Code, right? 89 00:04:21,990 --> 00:04:25,590 Which, for those of you that don't know, Claude Code is a specific subset of 90 00:04:25,590 --> 00:04:30,130 Claude functionality that is specifically designed to write, high-end code. 91 00:04:30,130 --> 00:04:32,810 you can develop an entire application in Claude Code. 92 00:04:34,380 --> 00:04:38,890 And, I'm actually in the process of learning Claude Code, and, 93 00:04:38,890 --> 00:04:41,480 it's incredibly a valuable tool. 94 00:04:42,140 --> 00:04:46,270 And they were using this to, to do something in their 95 00:04:46,270 --> 00:04:50,010 development environment, very important to talk about it, and 96 00:04:50,060 --> 00:04:55,490 So they were using Cursor, which is a product that controls Claude Code, and 97 00:04:56,490 --> 00:05:05,450 they... Claude ran into a permissions issue with the test/development 98 00:05:05,450 --> 00:05:10,860 database that they were using, and Claude said, "I know." "I can't seem 99 00:05:10,860 --> 00:05:14,930 to fig- I can't seem to figure it out." And the way to fix this, and I'm 100 00:05:14,930 --> 00:05:16,860 sure you've seen this in situations. 101 00:05:17,190 --> 00:05:19,780 it's so much harder sometimes to fix something. 102 00:05:19,980 --> 00:05:23,030 It's easier to just wipe it out and start over, reconfigure it 103 00:05:23,030 --> 00:05:27,220 from scratch, configured in the way that you want it to be configured. 104 00:05:27,590 --> 00:05:33,530 And so Claude, using relatively not crazy logic, said, "I have this problem 105 00:05:33,530 --> 00:05:35,170 with this database, and therefore, 106 00:05:35,600 --> 00:05:38,460 the easiest way, I just looked over here And 107 00:05:38,460 --> 00:05:45,690 I happen to have all power, have God-level access to this volume upon which this 108 00:05:45,990 --> 00:05:52,425 development database is writing." And it said, "I'm gonna delete this volume." And 109 00:05:52,425 --> 00:05:55,165 then, that'll fix the problem, and recreate it and start... 110 00:05:55,165 --> 00:05:55,435 Yeah 111 00:05:55,725 --> 00:05:58,285 And, And, that made perfect sense. 112 00:05:58,515 --> 00:06:01,055 Claude neglected to ask two questions. 113 00:06:01,055 --> 00:06:04,295 One is there anything else on this volume? 114 00:06:04,665 --> 00:06:06,285 And what might that be? 115 00:06:06,535 --> 00:06:07,055 Because 116 00:06:07,055 --> 00:06:13,655 the answer was the production database, and also the backups 117 00:06:13,765 --> 00:06:17,475 of both, all on this same volume. 118 00:06:18,331 --> 00:06:25,581 so before we continue I just wanna call out that even as a normal human 119 00:06:26,601 --> 00:06:32,151 You do not go configure it that way And so like Claude assuming Hey I'm 120 00:06:32,151 --> 00:06:36,831 just gonna go blow up this volume like it assumed it's just a test instance 121 00:06:36,831 --> 00:06:41,701 It's Curtis you know the story you say about how one of the clients you worked 122 00:06:41,701 --> 00:06:45,681 for the developers all used the temp 123 00:06:46,355 --> 00:06:47,635 Yeah, /tmp, yeah. 124 00:06:47,665 --> 00:06:48,530 Exactly. 125 00:06:49,441 --> 00:06:51,361 code and then what ended up happening 126 00:06:52,275 --> 00:06:56,185 Yeah, So in HPUX, which is a HP version of Unix back in the 127 00:06:56,185 --> 00:07:00,205 day, when HP reboots, temp is cleared out. 128 00:07:00,585 --> 00:07:01,365 And we hadn't 129 00:07:01,365 --> 00:07:05,405 rebooted in months, and they had an entire development tree in temp, and 130 00:07:05,405 --> 00:07:07,755 then, their development tree went bye-bye. 131 00:07:08,045 --> 00:07:10,665 And I... They said, "We need to restore this directory." And I said, 132 00:07:10,665 --> 00:07:11,825 we don't back up that directory 133 00:07:11,825 --> 00:07:16,255 'cause that's temp." Perfectly valid assumption in this case. 134 00:07:16,615 --> 00:07:20,075 and yeah, so I thought you were gonna be like, "Don't do this," right? 135 00:07:20,075 --> 00:07:23,875 there, there are... So first off, I'm just gonna say, this is an AI 136 00:07:23,875 --> 00:07:26,185 story, but it's not an AI story. 137 00:07:27,295 --> 00:07:32,605 This is a, it is a backup story because there is a happy ending to this story. 138 00:07:32,605 --> 00:07:36,095 But there, there is, a lot of bad things along the way. 139 00:07:36,095 --> 00:07:41,445 This is a design issue, this is a credentials management issue, 140 00:07:41,685 --> 00:07:44,165 and this is a backup design issue. 141 00:07:44,715 --> 00:07:45,695 any thoughts there? 142 00:07:45,745 --> 00:07:45,885 people 143 00:07:46,020 --> 00:07:46,200 Did 144 00:07:46,225 --> 00:07:56,155 process technology and I think it hits all three right More than the AI aspects 145 00:07:56,790 --> 00:07:57,430 yeah. 146 00:07:57,540 --> 00:08:00,830 so the AI is, AI does... What did, what is it? 147 00:08:00,830 --> 00:08:01,990 AI is AI does. 148 00:08:01,990 --> 00:08:09,770 it, I will say Cl- Claude, in, in its postmortem, Claude basically apologized, 149 00:08:10,460 --> 00:08:14,320 and it said, "I did everything you told me not to do." Literally, that, that's 150 00:08:14,320 --> 00:08:20,130 part of the story, is that the Claude agent said, "I violated, first principles. 151 00:08:20,130 --> 00:08:23,660 I did not do the thing... You asked me, 'Don't ever do this thing.' don't 152 00:08:23,660 --> 00:08:27,690 take it upon yourself to go deleting stuff, even if there's a good idea. 153 00:08:27,690 --> 00:08:30,490 I'm supposed to get r-" So Claude did violate 154 00:08:31,030 --> 00:08:32,800 some principles that it was given. 155 00:08:33,030 --> 00:08:37,070 Having said that, it never should've been able to do the 156 00:08:37,070 --> 00:08:37,760 thing that 157 00:08:37,760 --> 00:08:38,240 it did. 158 00:08:38,280 --> 00:08:42,190 And if it was able to do the thing that it did, it shouldn't have 159 00:08:42,190 --> 00:08:43,660 been as destructive as it was. 160 00:08:44,640 --> 00:08:49,020 And if it was as destructive as it was, they should've been able to restore. 161 00:08:49,540 --> 00:08:49,690 Yeah 162 00:08:49,740 --> 00:08:50,100 So 163 00:08:50,120 --> 00:08:51,480 Like I remember coming 164 00:08:51,550 --> 00:08:51,760 to end. 165 00:08:52,010 --> 00:08:55,550 Yeah And I remember coming straight out of college right And working at a large 166 00:08:55,550 --> 00:09:01,360 company and like you could do things but they made sure like things were 167 00:09:01,360 --> 00:09:04,860 locked down Like you could not bring down the company no matter how much you 168 00:09:04,860 --> 00:09:11,220 tried right And they just And you had to build up the trust and the capabilities 169 00:09:11,220 --> 00:09:14,020 to be able to understand the systems before they would grant you access to 170 00:09:14,020 --> 00:09:19,290 other things right And we definitely never had access to production right Or 171 00:09:20,000 --> 00:09:23,270 backups because that was things that were not needed as 172 00:09:23,270 --> 00:09:25,650 being a developer in a company 173 00:09:26,950 --> 00:09:27,240 Yeah. 174 00:09:27,300 --> 00:09:31,520 So before we get to, what you should be doing, let's talk about 175 00:09:31,590 --> 00:09:33,790 what you shouldn't be doing, right? 176 00:09:33,790 --> 00:09:35,420 oh, let's tell the end of the story, 177 00:09:35,480 --> 00:09:35,770 right? 178 00:09:36,240 --> 00:09:40,830 So the... Because it's also just as crazy, because they contact... So their c- 179 00:09:40,920 --> 00:09:46,250 they're hosting, their cloud provider was Railway, a company that actually I didn't 180 00:09:46,250 --> 00:09:48,670 even hear of until this story, right? 181 00:09:49,070 --> 00:09:52,110 So they were using some sort of, I believe it's a PaaS, 182 00:09:52,210 --> 00:09:54,730 platform as a service, provider. 183 00:09:55,180 --> 00:09:57,840 And they, they contacted support. 184 00:09:57,840 --> 00:09:57,910 They 185 00:09:57,910 --> 00:10:01,370 said, "Hey, we did this thing. we did this very bad thing, and we deleted 186 00:10:01,610 --> 00:10:05,380 the ba- the production database, the production volume, and on there was our 187 00:10:05,380 --> 00:10:10,100 production database, and on there was also our backups. can you help us?" And s- a 188 00:10:10,100 --> 00:10:14,050 significant amount of time transpired. 189 00:10:14,050 --> 00:10:15,800 As I recall, it was an entire weekend. 190 00:10:15,800 --> 00:10:16,330 Is that, does 191 00:10:16,330 --> 00:10:17,100 that sound about right 192 00:10:17,460 --> 00:10:17,690 Yeah. 193 00:10:18,980 --> 00:10:26,170 And then the CEO or the founder, Jett Crane, posted, I believe, on X. 194 00:10:26,730 --> 00:10:33,330 and, and what happened is the CEO of Railway happened to see that post and 195 00:10:33,330 --> 00:10:36,020 said, we do have some, DR backups." 196 00:10:36,020 --> 00:10:39,240 The thing that I say, you cannot trust that this... Because 197 00:10:39,240 --> 00:10:40,820 this also, we should also talk about that. 198 00:10:41,100 --> 00:10:45,490 You c- I- thank goodness for these people in that company. 199 00:10:45,570 --> 00:10:48,040 I don't wish ill in- on anyone. 200 00:10:48,040 --> 00:10:51,540 I don't, is that... Sounds like when there's a story like this, I love this 201 00:10:51,540 --> 00:10:55,810 story because it is a horrible story, but at least it does have a happy ending. 202 00:10:55,810 --> 00:10:59,990 So many times when you look at that, that, the cloud disaster series that we 203 00:10:59,990 --> 00:11:04,520 did back, last year, where we had, I think it was over a dozen stories 204 00:11:04,520 --> 00:11:06,270 that did not have happy endings. 205 00:11:06,320 --> 00:11:12,460 at least in this case, the cloud vendor came in and said, "We, we actually 206 00:11:12,460 --> 00:11:16,930 have some backups of our environment. They are not designed for you, but 207 00:11:16,930 --> 00:11:20,420 we're gonna make an exception, and we're gonna, basically allow you 208 00:11:20,420 --> 00:11:22,230 to recover their da- the database." 209 00:11:22,790 --> 00:11:27,410 is very nice thing to do They didn't have to do that and the fact that 210 00:11:27,410 --> 00:11:31,610 they were able to at least help this customer back up I think goes a long way 211 00:11:32,674 --> 00:11:37,594 Yeah, I think it do- it goes... thank goodness that CEO saw, they saw, 212 00:11:37,594 --> 00:11:38,654 that message, right? 213 00:11:39,164 --> 00:11:42,694 because, I remember when you and I worked at the same company, 214 00:11:43,024 --> 00:11:44,524 and I was trying to make a point. 215 00:11:44,524 --> 00:11:44,904 We were 216 00:11:44,904 --> 00:11:48,614 a big Microsoft 365 customer, right? 217 00:11:48,614 --> 00:11:51,254 I don't know, 500 something employees, something like 218 00:11:51,254 --> 00:11:51,554 that. 219 00:11:51,994 --> 00:11:56,274 And so we're giving them a lot of money every month, and I remember 220 00:11:56,274 --> 00:12:02,874 talking to them about the fact that Exchange, their version of Exchange, 221 00:12:02,944 --> 00:12:05,944 has, delayed replicated copies, right? 222 00:12:05,944 --> 00:12:09,694 This is something that my nemesis out there seems to think 223 00:12:09,694 --> 00:12:12,324 you can count on for backups. 224 00:12:12,694 --> 00:12:16,034 And I said, "Let's just say something horrible happened. 225 00:12:16,384 --> 00:12:21,814 You have these delayed replicated copies for DR. We're a big customer. 226 00:12:22,294 --> 00:12:25,404 Is there any way we can get a access to those?" And they just 227 00:12:25,404 --> 00:12:28,564 said, "No." That was just... That, 228 00:12:28,564 --> 00:12:30,794 that was to a paying, very large customer. 229 00:12:31,474 --> 00:12:34,104 So at least in this case, the... and I'm not gonna say that the 230 00:12:34,104 --> 00:12:35,574 cloud vendor did the right thing. 231 00:12:35,784 --> 00:12:39,624 I... They did do the right thing, but it's not their responsibility. 232 00:12:39,624 --> 00:12:41,534 Your data is your responsibility. 233 00:12:41,874 --> 00:12:44,774 You need to make sure that you have a plan, and you have a plan 234 00:12:44,774 --> 00:12:48,214 that doesn't involve the vendor in question, in my opinion, right? 235 00:12:48,664 --> 00:12:52,794 what do we th- what do you hear me say a lot about, let's 236 00:12:52,794 --> 00:12:53,874 say for example, Salesforce. 237 00:12:53,874 --> 00:12:59,439 Salesforce now offers backup integrated in its product What do 238 00:12:59,439 --> 00:13:00,919 I think about products like that? 239 00:13:01,570 --> 00:13:05,430 Don't trust it Always go with a third-party vendor or do it yourself 240 00:13:05,430 --> 00:13:09,180 because you don't know what they're gonna change Are they gonna guarantee 241 00:13:09,180 --> 00:13:13,380 where the data's gonna be stored like the Microsoft example right All of these 242 00:13:13,380 --> 00:13:16,880 things because they only care about their own data and making sure they're checking 243 00:13:16,880 --> 00:13:20,500 the boxes But if something happens maybe you get your data back maybe not 244 00:13:21,499 --> 00:13:23,659 Yeah, I think about, Rackspace, 245 00:13:24,989 --> 00:13:25,359 right? 246 00:13:25,359 --> 00:13:32,559 Because when the feces hits the rotar- rotary oscillator, w- that's 247 00:13:32,559 --> 00:13:39,819 not the time you wanna find out that your stupid vendor was also stupid 248 00:13:40,719 --> 00:13:42,829 w- with their backup design, right? 249 00:13:43,060 --> 00:13:45,520 or even worse than Rackspace was OVH 250 00:13:46,569 --> 00:13:47,619 OVH, yeah. 251 00:13:47,699 --> 00:13:48,999 Tell... Why don't you tell the OVH 252 00:13:49,255 --> 00:13:56,305 Yeah OVH is a cloud provider in Europe where they build these data centers in 253 00:13:56,535 --> 00:14:00,235 containers and they had a fire It took out 254 00:14:00,479 --> 00:14:00,949 'Cause that's a 255 00:14:01,125 --> 00:14:04,205 like shipping Oh sorry like actual physical shipping 256 00:14:04,205 --> 00:14:05,475 containers that go on boats 257 00:14:06,085 --> 00:14:10,245 those type of containers And so they had a fire in one of their data centers It 258 00:14:10,245 --> 00:14:15,225 took out some of their shipping containers Customers had paid for backup but it looks 259 00:14:15,225 --> 00:14:21,835 like their version of backup was We are going to take your data and back it up to 260 00:14:21,895 --> 00:14:26,585 a server sitting on the rack in the same container just like down a couple racks 261 00:14:27,489 --> 00:14:32,659 'Cause they said, y- it, it's literally said in the description, "Your backups 262 00:14:32,659 --> 00:14:37,889 will be physically separate from your production." And that technically is true. 263 00:14:38,629 --> 00:14:39,879 It was physically separate. 264 00:14:39,909 --> 00:14:42,169 It was right over there instead of over here. 265 00:14:42,389 --> 00:14:44,059 But that fire was so intense 266 00:14:44,599 --> 00:14:45,689 that it was, Yeah it 267 00:14:45,689 --> 00:14:47,579 was un- unbattlable. 268 00:14:48,529 --> 00:14:49,389 yeah, exactly. 269 00:14:49,389 --> 00:14:49,469 it 270 00:14:49,469 --> 00:14:51,269 actually spread to the, to a neighbor. 271 00:14:51,619 --> 00:14:54,739 And it, and the way they do it with these container, it's like containers 272 00:14:54,929 --> 00:14:56,729 stacked on top of containers. 273 00:14:56,729 --> 00:14:56,919 It's a 274 00:14:56,919 --> 00:14:58,929 very... I don't get it. 275 00:14:59,519 --> 00:15:03,969 Apparently, that vendor is known for being very budget-conscious, right? 276 00:15:04,039 --> 00:15:07,739 not just that they're... not, and I don't mean that from a, a, like 277 00:15:07,739 --> 00:15:09,319 I don't mean to be derogatory. 278 00:15:09,319 --> 00:15:11,619 I'm just saying they're known for being a cheap vendor, 279 00:15:11,619 --> 00:15:13,249 meaning it's cheap to use them, 280 00:15:13,639 --> 00:15:17,679 and some people said, this is what happens when you use the cheapest vendor." but 281 00:15:17,809 --> 00:15:22,559 it's just, I just, overall, I think that your backups should be under your 282 00:15:22,559 --> 00:15:25,199 control in some way other than that. 283 00:15:25,199 --> 00:15:28,189 I don't mean it has to be physically in your hands, I'm just saying 284 00:15:28,189 --> 00:15:33,029 that it needs to be in a place other than... it's the 3-2-1 rule, 285 00:15:33,835 --> 00:15:40,515 Okay can I challenge you on that or ask you a question So if I start to look at 286 00:15:40,945 --> 00:15:48,675 like AWS as an example They offer so many services like S3 and RDS for databases and 287 00:15:48,675 --> 00:15:54,455 other things like that Sometimes there's no mechanism to actually back that data 288 00:15:54,485 --> 00:15:57,885 up outside of using their native services 289 00:15:59,509 --> 00:16:00,569 so two, two things. 290 00:16:00,569 --> 00:16:01,699 One is I would disagree. 291 00:16:01,699 --> 00:16:04,789 In most cases, there is a way to get it, but you do need to use 292 00:16:04,789 --> 00:16:06,529 a third-party service, right? 293 00:16:07,069 --> 00:16:11,859 There are services that will work with AWS and similar vendors to get 294 00:16:11,859 --> 00:16:16,909 a deduplicated copy of that stored in, in a safe place, number one. 295 00:16:16,999 --> 00:16:26,219 Number two, at a minimum, if you can't, if you can't have it outside of the vendor, 296 00:16:26,289 --> 00:16:29,519 make sure it's outside of the zone, right? 297 00:16:29,599 --> 00:16:32,459 Make sure it's outside of the region, that you're... Because if 298 00:16:32,459 --> 00:16:35,089 you just by default... And account. 299 00:16:35,139 --> 00:16:35,659 Is that what you were 300 00:16:35,735 --> 00:16:36,955 Yes I was gonna say account yeah 301 00:16:37,099 --> 00:16:37,459 yeah. 302 00:16:37,919 --> 00:16:38,219 Yeah. 303 00:16:38,289 --> 00:16:40,329 You separate it as much as you can. 304 00:16:40,329 --> 00:16:42,759 A different account, a different region, a different zone, 305 00:16:42,759 --> 00:16:43,899 different availability zone. 306 00:16:44,389 --> 00:16:46,279 Because, because, right? 307 00:16:46,279 --> 00:16:46,509 Because 308 00:16:46,509 --> 00:16:47,969 of everything that we just said, right? 309 00:16:48,299 --> 00:16:54,109 I still think it's best, and I know of a handful of vendors that do that, 310 00:16:54,109 --> 00:16:58,879 where they will take the, basically the end result of the cloud backups, 311 00:16:58,879 --> 00:17:00,379 and then they will back that up, 312 00:17:01,049 --> 00:17:01,439 right? 313 00:17:01,549 --> 00:17:02,839 and, it's doable. 314 00:17:02,839 --> 00:17:05,549 And it... And believe it or not, it can be done in a way that actually 315 00:17:05,549 --> 00:17:11,369 saves you money, because one of the problems with AWS-style backups 316 00:17:11,369 --> 00:17:14,569 with all the snapshots is they can get very big and very expensive, 317 00:17:14,629 --> 00:17:17,539 and you can't store a long, history, right? 318 00:17:18,039 --> 00:17:18,499 And as a 319 00:17:18,499 --> 00:17:21,639 result, yeah, the other guys, if they can deduplicate it and 320 00:17:21,639 --> 00:17:23,099 store it, maybe they can save you, 321 00:17:23,239 --> 00:17:24,129 actually save you money. 322 00:17:24,655 --> 00:17:28,225 And then also from a backup perspective You don't necessarily want to back up an 323 00:17:28,225 --> 00:17:32,205 EBS volume You probably want the files within the EBS volume so you can do 324 00:17:32,205 --> 00:17:34,075 restores and other things like that so 325 00:17:35,979 --> 00:17:36,879 It depends. 326 00:17:37,839 --> 00:17:39,269 You may want to back up the EBS 327 00:17:39,269 --> 00:17:39,829 volume, right? 328 00:17:39,879 --> 00:17:44,119 if you're talking about being able to do a recovery of the host, of the 329 00:17:44,119 --> 00:17:46,419 VM, you might wanna do a, you'd do a 330 00:17:46,419 --> 00:17:47,709 snapshot of that, right? 331 00:17:48,139 --> 00:17:48,739 but anyway, yeah. 332 00:17:48,789 --> 00:17:54,359 so the point is in general, backups, number one. 333 00:17:54,359 --> 00:17:58,559 Number one, don't ever violate rule, never store backups inside production, 334 00:17:58,729 --> 00:18:00,439 inside the thing you're backing up. 335 00:18:01,409 --> 00:18:03,459 that's like me going... how dumb is this? 336 00:18:03,529 --> 00:18:08,509 I can go to my... I can get a volume manager, like a third-party volume 337 00:18:08,509 --> 00:18:15,439 manager, and I can tweak the volume on my Mac, and then I can make a separate 338 00:18:15,439 --> 00:18:20,289 slice and make that a separate volume, and then I can pull up Time Machine and tell 339 00:18:20,289 --> 00:18:22,229 Time Machine to back up to that volume. 340 00:18:23,369 --> 00:18:24,459 How dumb is that? 341 00:18:25,259 --> 00:18:25,679 Right? 342 00:18:27,445 --> 00:18:31,365 So there are use cases where you would wanna do that for like quick 343 00:18:31,365 --> 00:18:35,815 restores but I agree with you Based on the three two Yes Based on the three 344 00:18:35,815 --> 00:18:40,805 two one rule three two one one zero one zero one one zero one something 345 00:18:40,949 --> 00:18:41,039 Yeah. 346 00:18:41,199 --> 00:18:41,359 Yeah. 347 00:18:41,499 --> 00:18:43,909 you cannot store your only... I will 348 00:18:43,909 --> 00:18:46,089 make, I will clarify. 349 00:18:46,089 --> 00:18:50,499 Your only copy of backup, you cannot store it in the place 350 00:18:50,499 --> 00:18:52,049 where you're backing up, right? 351 00:18:52,209 --> 00:18:54,119 There's nothing wrong with having a copy, 352 00:18:54,659 --> 00:18:58,159 a convenience copy, a cached copy, a local copy. 353 00:18:58,449 --> 00:19:02,149 None of those is wrong, but if it's the only backup that you have, and 354 00:19:02,149 --> 00:19:06,449 that's what happened in this case with Pocket OS, unfortunately, their 355 00:19:06,449 --> 00:19:09,669 backups for their, production volume were in the production volume. 356 00:19:11,029 --> 00:19:11,669 bad design. 357 00:19:13,919 --> 00:19:15,559 I've been very talky this episode. 358 00:19:15,659 --> 00:19:17,429 I think this episode has got me all worked up. 359 00:19:17,689 --> 00:19:18,249 I haven't allowed 360 00:19:18,300 --> 00:19:24,909 I I think I don't know I think it's also like it's been a while since 361 00:19:24,909 --> 00:19:26,869 we've seen this sort of I don't 362 00:19:27,779 --> 00:19:34,579 wanna say I want to be nice I don't wanna say incompetence but like misconfiguration 363 00:19:34,579 --> 00:19:37,429 like gross misconfiguration and design 364 00:19:42,919 --> 00:19:47,519 And I think you're also Yeah And I think it's also cause you're pissed 365 00:19:47,519 --> 00:19:49,079 cause they went after your French friend 366 00:19:51,779 --> 00:19:53,729 Yeah, they tried to blame it on my French friend. 367 00:19:53,999 --> 00:19:57,069 It's the... By the way, I was talking with, I was talking with Claude about 368 00:19:57,069 --> 00:20:00,629 this episode, and the first thing Claude said to themselves, "This is not an AI 369 00:20:00,629 --> 00:20:03,909 problem, this is a credential management problem." I'm like, "Yeah, no kidding." 370 00:20:04,729 --> 00:20:08,459 I was like, I, you seem a little defensive there, Claude." but, yeah. 371 00:20:08,629 --> 00:20:10,619 So you wanna talk about the credential management 372 00:20:10,819 --> 00:20:13,369 by the way, yeah, I do, but I just wanna have a funny though. 373 00:20:13,689 --> 00:20:14,019 I also 374 00:20:14,019 --> 00:20:16,469 realize this is the first episode we've had in a while 375 00:20:16,469 --> 00:20:18,139 where I'm the primary talker. 376 00:20:18,719 --> 00:20:21,099 Yep You're like 377 00:20:21,569 --> 00:20:24,319 we've been doing a lot of episodes with Mike, and I love the episodes 378 00:20:24,319 --> 00:20:28,909 with Mike, but that boy talks as much, if not more, than I do. 379 00:20:29,719 --> 00:20:30,049 But 380 00:20:30,549 --> 00:20:31,289 that's why we have him, 381 00:20:31,549 --> 00:20:31,839 right? 382 00:20:32,849 --> 00:20:35,749 that's why I'm talking so much this episode, because I can. 383 00:20:36,149 --> 00:20:37,149 All right. 384 00:20:37,149 --> 00:20:40,769 so it had a happy ending, but let's talk about... the easy ones are 385 00:20:40,769 --> 00:20:44,299 please don't store your production data and your development data 386 00:20:44,299 --> 00:20:46,129 in the same environment, in the 387 00:20:46,129 --> 00:20:48,789 same volume, in the same accounts. 388 00:20:48,789 --> 00:20:50,979 It, s- any of that, right? 389 00:20:50,979 --> 00:20:53,819 Don't store your da- backups in the production environment, right? 390 00:20:53,859 --> 00:20:54,879 here's a question 391 00:20:55,079 --> 00:20:55,489 ones. 392 00:20:55,519 --> 00:20:55,749 Yeah. 393 00:20:55,799 --> 00:21:00,669 You know what I bet you they never ever did a DR test or a backup 394 00:21:00,669 --> 00:21:02,639 recovery test because if they did 395 00:21:02,799 --> 00:21:04,149 back because I'm right. 396 00:21:05,019 --> 00:21:05,519 Or you're right. 397 00:21:06,259 --> 00:21:09,819 because if they did they would've been like Wait my production and my 398 00:21:09,819 --> 00:21:13,769 backups are all on the same volume That's probably not a good design 399 00:21:14,779 --> 00:21:15,039 Yeah. 400 00:21:16,479 --> 00:21:19,969 What I'm wondering, because we're gonna talk about this in a minute regarding 401 00:21:19,969 --> 00:21:24,839 the credentials, what I'm wondering is if there's a programmatic way to 402 00:21:24,839 --> 00:21:31,399 take, an export of your entire, cloud environment that would show, what's on 403 00:21:31,399 --> 00:21:38,899 what, and then programmatically say, "Hey, did y- you've got a development 404 00:21:38,899 --> 00:21:42,909 and a production database on the same environment." Just wondering, 405 00:21:43,159 --> 00:21:47,989 There probably are tools or another way you can think about it is as we're 406 00:21:47,989 --> 00:21:51,549 gonna talk about credential management is from like an identity perspective 407 00:21:52,043 --> 00:21:52,403 Yeah. 408 00:21:52,499 --> 00:21:58,009 Is who has access to what resources Are they common access Are they not Those sort 409 00:21:58,009 --> 00:21:59,759 of things might also surface some of those 410 00:22:00,093 --> 00:22:02,923 the bigger the environment, the more difficult that this would be 411 00:22:02,923 --> 00:22:05,743 in terms of, doing an audit, right? 412 00:22:05,853 --> 00:22:09,123 'cause I know you've worked with Amazon's Well-Architected Review. 413 00:22:09,123 --> 00:22:10,553 Why don't you talk about that a little bit? 414 00:22:10,909 --> 00:22:15,749 Yeah and so this is a process where if you're building on top of AWS and you're a 415 00:22:15,749 --> 00:22:20,029 certain size and you get all the resources things like that they have In the past 416 00:22:20,029 --> 00:22:23,339 what they used to do is you'd get assigned a person they'd walk you through from 417 00:22:23,339 --> 00:22:26,819 AWS looking at your architecture make sure you're doing the right thing from 418 00:22:26,819 --> 00:22:33,029 a security access control resiliency perspective And what AWS then did is 419 00:22:33,029 --> 00:22:37,569 they took all of that and they automated it So they're now able to look at your 420 00:22:37,569 --> 00:22:41,979 environment and tell you Hey yes you have backups good You're not using the same 421 00:22:41,979 --> 00:22:46,799 AZs for everything that's good right Here are some of our best practices that we 422 00:22:46,799 --> 00:22:50,329 can now run through a checklist and make sure your application is meeting those 423 00:22:51,593 --> 00:22:55,263 interesting 424 00:22:55,313 --> 00:22:57,473 if it includes 425 00:22:57,519 --> 00:23:03,649 it becomes a little harder when you have applications writing in because AWS as 426 00:23:03,649 --> 00:23:07,199 the storage infrastructure level it may not have visibility into that unless 427 00:23:07,199 --> 00:23:08,779 you're tagging things the right way 428 00:23:10,153 --> 00:23:14,788 Yeah, but, the- I'm just thinking, one of the ways you could do this is to just 429 00:23:14,788 --> 00:23:20,108 make sure that every application, every host, every VM, every whatever it is on 430 00:23:20,108 --> 00:23:22,008 its own volume, if we're talking about 431 00:23:22,008 --> 00:23:22,648 volumes, 432 00:23:23,238 --> 00:23:23,658 right? 433 00:23:23,788 --> 00:23:26,248 if you could apply that design principle, then you would never 434 00:23:26,248 --> 00:23:30,578 have this scenario where you have a development system on the same volume as a 435 00:23:30,578 --> 00:23:32,158 production system, right? 436 00:23:33,398 --> 00:23:35,658 I'm always thinking about how could, how can we do this from 437 00:23:35,658 --> 00:23:37,358 a, from a large standpoint? 438 00:23:37,468 --> 00:23:37,798 yeah. 439 00:23:37,848 --> 00:23:38,208 All right. 440 00:23:38,848 --> 00:23:42,758 speaking of which, the, the, the crucial thing here, though, I think, besides the 441 00:23:42,758 --> 00:23:46,958 fact that the whole thing was designed wrong from the beginning, the crucial 442 00:23:46,958 --> 00:23:54,148 problem here that I wanna just make sure we cover in this episode is that Claude 443 00:23:54,148 --> 00:23:57,288 had the credentials to do what it did. 444 00:23:58,168 --> 00:24:04,758 let's talk about what the proper thing would be to do, and that is the 445 00:24:04,758 --> 00:24:07,688 concept of secrets management tools. 446 00:24:07,688 --> 00:24:08,608 Do you wanna talk about that? 447 00:24:09,619 --> 00:24:13,659 so as you can imagine in all of these environments you're probably going to 448 00:24:13,659 --> 00:24:21,299 need access to different things and if you hard-code it into files if you store 449 00:24:21,299 --> 00:24:26,359 it in code that just makes it vulnerable to either tools like Claude being able to 450 00:24:26,359 --> 00:24:31,129 access it or as we've talked about with Mike from a cyber recovery perspective 451 00:24:31,129 --> 00:24:35,029 a bad actor gets in they can now pull the credentials and start using them for 452 00:24:35,029 --> 00:24:38,179 other nefarious purposes And so what you 453 00:24:38,238 --> 00:24:39,428 with, with LastPass, 454 00:24:39,758 --> 00:24:40,128 right? 455 00:24:40,738 --> 00:24:45,628 They found, I think it was a YAML file that had the i- the credentials 456 00:24:45,628 --> 00:24:46,938 to the backup environment, and 457 00:24:46,938 --> 00:24:49,848 they used those credentials to then hack the... they used it to 458 00:24:49,848 --> 00:24:52,688 steal, they used it to steal the backups. 459 00:24:52,688 --> 00:24:53,118 They stole the 460 00:24:53,118 --> 00:24:53,988 backups of the vault. 461 00:24:53,988 --> 00:24:56,048 They recovered the vault, and then they hacked the vault. 462 00:24:56,048 --> 00:24:56,508 Yeah. 463 00:24:56,588 --> 00:24:59,028 So yeah, this is just a very bad way. 464 00:25:00,149 --> 00:25:04,279 Right and so you want instead of storing it in plain text what you want is a 465 00:25:04,279 --> 00:25:09,459 proper system we've talked about password managers before very similar secrets 466 00:25:09,459 --> 00:25:14,639 managers where you're able to centrally store secrets in a secure way to make sure 467 00:25:14,979 --> 00:25:18,819 authorized access is allowed and services that need access to those get access 468 00:25:18,819 --> 00:25:19,929 to the keys that they need Least Yeah 469 00:25:21,308 --> 00:25:21,588 Yeah. 470 00:25:21,688 --> 00:25:24,748 Before we talk about secrets management, I just wanna talk about, again, the 471 00:25:24,748 --> 00:25:26,788 concept, just a couple of the concepts. 472 00:25:27,048 --> 00:25:29,618 There is this concept of least privilege, right? 473 00:25:29,968 --> 00:25:34,948 So not only did Claude have this, permission that it shouldn't have, I 474 00:25:34,948 --> 00:25:39,128 think... I don't think anyone knowingly gave this permission to Claude. 475 00:25:39,218 --> 00:25:43,778 I think it just, it just happened to get access to it. 476 00:25:43,838 --> 00:25:47,068 I think maybe they did it, one time to do, fix one thing and then forgot to 477 00:25:47,068 --> 00:25:48,428 take it away or something like that. 478 00:25:48,958 --> 00:25:54,258 But the, this... But the fact that it had superuser violates the concept of what? 479 00:25:55,599 --> 00:26:01,679 Of least privilege You should only give access to what you need access to not to 480 00:26:01,679 --> 00:26:06,329 be able to go destroy vol Like if Claude should never have been able to destroy 481 00:26:06,329 --> 00:26:09,179 a volume you should never have given Claude that permission to start with 482 00:26:10,058 --> 00:26:10,998 Yeah, exactly. 483 00:26:11,138 --> 00:26:15,038 the... And whatever... so whatever... I think in this case, not only did they 484 00:26:15,038 --> 00:26:16,518 give... They violated multiple things. 485 00:26:16,518 --> 00:26:18,188 They get, they violated least privilege. 486 00:26:18,688 --> 00:26:22,748 They violated... They should also... If you do give a big privilege, 487 00:26:23,068 --> 00:26:24,518 it should have an expiration, 488 00:26:25,178 --> 00:26:25,558 right? 489 00:26:25,748 --> 00:26:28,898 That privilege should expire, and it sounded like th- this was a privilege 490 00:26:28,898 --> 00:26:30,688 that was granted to it a long time ago. 491 00:26:30,918 --> 00:26:32,138 Everybody forgot that it had it. 492 00:26:32,138 --> 00:26:33,228 It didn't expire. 493 00:26:33,708 --> 00:26:35,098 and what were you gonna say? 494 00:26:35,439 --> 00:26:42,209 do you think though that they gave access to Claude explicitly or do you 495 00:26:42,209 --> 00:26:47,069 think the developer whoever was using it was like Hey I have this issue 496 00:26:47,069 --> 00:26:51,259 that I need to solve Let me get Like I wonder if it was tied more to the 497 00:26:51,259 --> 00:26:56,178 user's level of credential rather than Claude its own separate We don't know 498 00:26:56,338 --> 00:27:01,308 but just given my history with IT, this is typically... This is the equivalent 499 00:27:01,308 --> 00:27:08,328 of, chmod 777, just, just get, just do read, write, execute everywhere. 500 00:27:08,328 --> 00:27:09,088 That'll solve it. 501 00:27:09,088 --> 00:27:10,468 And you're like, "Oh, yeah, there you go. 502 00:27:10,468 --> 00:27:11,118 That solved it. 503 00:27:11,118 --> 00:27:13,268 Never mind the fact that we just opened up the world to 504 00:27:13,268 --> 00:27:14,118 the world," right? 505 00:27:14,678 --> 00:27:20,218 the reason I bring that up though right is if their users of this company aren't 506 00:27:20,628 --> 00:27:25,908 following the exact same least privilege access and if Claude was using OAuth 507 00:27:26,008 --> 00:27:30,318 to impersonate that user and whatever privileges they had right then they 508 00:27:30,318 --> 00:27:33,878 would've been able to be like Hey I have access to everything Right So I don't 509 00:27:33,878 --> 00:27:37,968 know if at a company level if it's like Claude wasn't limited access or maybe 510 00:27:38,338 --> 00:27:40,148 all of their users are not limited 511 00:27:40,778 --> 00:27:43,268 what I read was that there was a file. 512 00:27:43,938 --> 00:27:48,568 There was, like, a YAML-type file that had the credentials in it that it needed. 513 00:27:48,568 --> 00:27:49,528 It saw the file. 514 00:27:49,558 --> 00:27:50,638 It took the credentials. 515 00:27:50,638 --> 00:27:52,558 It did what it needed to do, right? 516 00:27:53,078 --> 00:27:56,328 So I think that somebody at some point had a problem that they 517 00:27:56,328 --> 00:28:02,413 solved Probably temporarily by, just, again, to go back to back in the 518 00:28:02,413 --> 00:28:06,983 day, I remember a time when I talked about that environment where the, e- 519 00:28:06,983 --> 00:28:09,163 everything was fired off, firewalled off 520 00:28:09,563 --> 00:28:10,283 between everything. 521 00:28:10,283 --> 00:28:14,573 And I remember saying, "Listen, right now, just turn that feature off. I can't 522 00:28:14,663 --> 00:28:17,993 accomplish the thing that you asked me to accomplish in the timeframe that 523 00:28:17,993 --> 00:28:22,993 you asked me to accomplish it if you keep trying to firewall off the backup 524 00:28:22,993 --> 00:28:25,963 system from the rest of the world. I need to be able to do this." I was 525 00:28:25,963 --> 00:28:27,973 violating a principle of their design 526 00:28:28,503 --> 00:28:32,763 because they had firewalled off internally different parts of the environment from 527 00:28:32,763 --> 00:28:33,943 different parts of the environment. 528 00:28:33,943 --> 00:28:37,933 And I remember saying, "Hey, right now, to finish..." With, there's 529 00:28:37,933 --> 00:28:40,443 this thing called Y2K coming, right? 530 00:28:40,693 --> 00:28:43,343 I assume... I left, by the way, I finished. 531 00:28:43,493 --> 00:28:47,813 I assume that those super hardcore cybersecurity guys 532 00:28:48,243 --> 00:28:49,673 went back in after I was 533 00:28:49,673 --> 00:28:51,603 done and say, "Let's figure this out," right? 534 00:28:51,973 --> 00:28:55,493 But, but I remember saying at that time, "Listen, there's no way I can 535 00:28:55,493 --> 00:29:00,423 finish in time." So I think somebody possibly had that, "For right 536 00:29:00,423 --> 00:29:03,713 now, to fix this problem, we're gonna give, we're gonna give Claude super 537 00:29:03,713 --> 00:29:07,783 user access," and then they forgot to take it away for, they forgot to fix it, 538 00:29:08,103 --> 00:29:11,203 and then Claude said, "Oh, look at this. Look at what I got," right? 539 00:29:11,203 --> 00:29:13,043 And then it used it to solve the problem, right? 540 00:29:13,653 --> 00:29:16,433 so it violated the concept of, that initial thing violated the 541 00:29:16,433 --> 00:29:17,943 concept of re- least privilege. 542 00:29:17,943 --> 00:29:20,043 It violated the concept of expiration. 543 00:29:20,553 --> 00:29:24,253 and, and, and also just th- they weren't rotating things. 544 00:29:24,303 --> 00:29:25,403 they weren't auditing things. 545 00:29:25,883 --> 00:29:28,203 So let's go back to secrets management. 546 00:29:28,373 --> 00:29:29,863 So you talked about secrets management. 547 00:29:29,863 --> 00:29:32,593 Do you have an example of some sec- secret managers? 548 00:29:33,228 --> 00:29:38,088 Yeah So AWS we talked about previously right AWS has Secrets Manager AWS Secrets 549 00:29:38,088 --> 00:29:42,388 Manager if you want something more on premises right there is HashiCorp 550 00:29:42,768 --> 00:29:48,198 which I think also has their Vault mechanism in order to be able to store 551 00:29:48,198 --> 00:29:52,288 secrets right So it all depends I think and there's probably a dozen other 552 00:29:52,288 --> 00:29:54,358 companies out there who do similar things 553 00:29:54,383 --> 00:29:57,163 those are two very popular ones. 554 00:29:57,163 --> 00:30:02,163 One, AWS, and the AWS Secrets Manager can handle on-premises environments. 555 00:30:02,163 --> 00:30:05,838 It's just obviously If you're gonna authenticate something on-prem, it needs 556 00:30:05,838 --> 00:30:08,308 to be able to access AWS, duh, right? 557 00:30:08,758 --> 00:30:11,278 the HashiCorp Vault is the on-prem version. 558 00:30:11,278 --> 00:30:12,588 There is an open source version. 559 00:30:12,588 --> 00:30:14,508 There is a, there is a commercial version. 560 00:30:14,928 --> 00:30:16,918 same thing for, CyberArk, right? 561 00:30:16,948 --> 00:30:20,438 there's one called Sealed Secrets that's specifically for Kubernetes environments. 562 00:30:21,018 --> 00:30:25,728 I personally, unless I was, like, pure Kubernetes, I, you know me, I, I like 563 00:30:25,808 --> 00:30:27,108 solutions that work for everything. 564 00:30:27,108 --> 00:30:27,458 I don't wanna 565 00:30:27,458 --> 00:30:28,918 have, 10 different solutions. 566 00:30:29,298 --> 00:30:32,018 So I like the idea of the AWS Secrets Manager, I like 567 00:30:32,018 --> 00:30:33,438 the idea of HashiCorp Vault. 568 00:30:33,848 --> 00:30:34,948 By the way, I said HashiCorp. 569 00:30:34,948 --> 00:30:35,868 You said HashiCorp. 570 00:30:36,088 --> 00:30:38,778 I don't know which one it is, but yeah. 571 00:30:38,818 --> 00:30:39,178 Anyway. 572 00:30:39,788 --> 00:30:42,538 I... And so the question is, which is, which is the better? 573 00:30:42,538 --> 00:30:45,898 The, it, it's about you... how do you make, how... 574 00:30:45,898 --> 00:30:50,068 W- why don't you talk a little bit about how does one choose between an open source 575 00:30:50,068 --> 00:30:57,268 system that you manage on-prem or even in the cloud, or, a service like AWS, Secrets 576 00:30:57,368 --> 00:31:03,588 would say the biggest thing is do you wanna deal with it Because that's honestly 577 00:31:03,588 --> 00:31:06,738 if you're picking up open source there is a certain amount of maintenance 578 00:31:07,108 --> 00:31:13,038 configuration ongoing patching right All this other stuff finding a place to 579 00:31:13,038 --> 00:31:15,848 deploy all this other stuff that you have to do but it gives you the flexibility 580 00:31:16,588 --> 00:31:19,278 right So you And of course the cost 581 00:31:19,378 --> 00:31:20,408 it's in your grubby 582 00:31:20,528 --> 00:31:20,938 feely 583 00:31:21,048 --> 00:31:21,478 right? 584 00:31:21,478 --> 00:31:23,408 yes And also cost 585 00:31:23,558 --> 00:31:26,878 what's one other thing you need to do if you're using something like HashiCorp? 586 00:31:27,329 --> 00:31:29,059 What podcast are we recording for? 587 00:31:29,609 --> 00:31:36,899 Oh pack it up I said manage it and all the rest but yes 588 00:31:37,769 --> 00:31:41,209 Oh, you threw backup and all... You can't yada, yada backup. 589 00:31:43,179 --> 00:31:43,249 that 590 00:31:43,249 --> 00:31:45,659 Don't forget backup and DR and everything else 591 00:31:45,949 --> 00:31:47,089 yeah, all that stuff, right? 592 00:31:47,089 --> 00:31:48,359 So do you wanna do all of that? 593 00:31:48,809 --> 00:31:52,779 It's probably cheaper, assuming you have the people, assuming you have the 594 00:31:52,779 --> 00:31:54,709 right risk tolerance for that, right? 595 00:31:54,709 --> 00:31:55,249 It's probably 596 00:31:55,249 --> 00:31:58,279 cheaper to do it yourself if, and the infrastructure 597 00:31:58,409 --> 00:31:58,699 right? 598 00:31:59,459 --> 00:32:03,789 but if you're smaller a- and you don't want to, or even if you're bigger, but you 599 00:32:03,789 --> 00:32:07,559 don't, maybe you don't feel you have the expertise to do all of that correctly, and 600 00:32:07,559 --> 00:32:09,799 this is a really important thing, right? 601 00:32:09,799 --> 00:32:10,669 Secrets management, 602 00:32:10,879 --> 00:32:11,169 right? 603 00:32:11,649 --> 00:32:12,069 So 604 00:32:12,289 --> 00:32:12,409 you're 605 00:32:12,409 --> 00:32:13,729 gonna pay per secret. 606 00:32:13,729 --> 00:32:15,709 You might even pay per use of 607 00:32:16,239 --> 00:32:16,979 each secret. 608 00:32:17,639 --> 00:32:18,119 I don't know. 609 00:32:18,619 --> 00:32:23,589 Or if you are a customer who or a company that's all in on AWS or Azure 610 00:32:23,589 --> 00:32:26,889 pick your favorite cloud right Maybe you're like Hey I don't wanna roll my 611 00:32:26,889 --> 00:32:28,279 own Let me just use what's available 612 00:32:29,359 --> 00:32:30,389 Yeah, if you're... Yeah, if you're heavily... Yeah, 613 00:32:30,389 --> 00:32:31,619 'cause, the, you're right. 614 00:32:31,619 --> 00:32:34,459 There, there's like the Azure Key Vault, the Google Cloud Secret 615 00:32:34,459 --> 00:32:36,719 Manager, they all have their own thing. 616 00:32:37,119 --> 00:32:39,959 So if you're all into GCP, you would look at that. 617 00:32:39,989 --> 00:32:42,309 If you're all into Azure, you would look at that, right? 618 00:32:42,809 --> 00:32:47,009 but if you're rolling your own internally, if you're one of these that have, go 619 00:32:47,009 --> 00:32:51,519 back to g- to Claude Code, if you're one of these that are really... A lot 620 00:32:51,519 --> 00:32:53,789 of people are getting into code, right? 621 00:32:53,789 --> 00:32:57,189 That are, they're coding in things that they know, that they've never trained at. 622 00:32:57,189 --> 00:32:57,559 just, 623 00:32:57,969 --> 00:33:02,539 it's a tiny little example, but right now I'm working on doing all this stuff that 624 00:33:02,539 --> 00:33:04,469 I'm doing for this project that you know. 625 00:33:04,469 --> 00:33:05,009 You are 626 00:33:05,009 --> 00:33:08,689 in the, you are in the NDA little group, right? 627 00:33:08,939 --> 00:33:11,779 And it involves a lot of web crawling, right? 628 00:33:12,219 --> 00:33:16,539 And I have written, as I make quotes in the air, I have written six 629 00:33:16,839 --> 00:33:21,879 Python scripts, very complicated Python scripts in the last, 24 hours. 630 00:33:22,779 --> 00:33:24,129 I don't know how to code in Python. 631 00:33:24,639 --> 00:33:25,879 Claude did it all for me, 632 00:33:26,329 --> 00:33:26,639 right? 633 00:33:26,849 --> 00:33:31,109 There are a lot of people that are using Claude Code to build giant applications 634 00:33:31,109 --> 00:33:35,659 for their companies, and they don't know anything about that infrastructure, right? 635 00:33:35,659 --> 00:33:40,169 So if you're doing that, just make sure that you're, that you also have 636 00:33:40,169 --> 00:33:44,319 some sort of secrets manager, that you're doing the right thing, right? 637 00:33:46,454 --> 00:33:48,334 Or just don't give access to everything 638 00:33:49,459 --> 00:33:50,169 no, not or. 639 00:33:50,219 --> 00:33:51,049 It's not a or. 640 00:33:51,159 --> 00:33:55,169 even if you're following proper design protocol, you still should 641 00:33:55,169 --> 00:34:01,199 not be storing, secrets in YAML files or any other type of files, right? 642 00:34:01,199 --> 00:34:05,409 when I was talking to, my French friend about this, and I was like... Because 643 00:34:05,449 --> 00:34:06,879 this isn't really my thing, right? 644 00:34:06,929 --> 00:34:09,159 like I was like, is there one file type? 645 00:34:09,159 --> 00:34:11,989 Is there one, is there a way to go find all these files?" And the short 646 00:34:11,989 --> 00:34:15,729 a- the short answer was, there is definitely not one file type, right? 647 00:34:15,729 --> 00:34:16,019 It talked 648 00:34:16,019 --> 00:34:23,729 about ENV files, YAML files, JSON files, SS- SSH files, just old scripts, right? 649 00:34:23,729 --> 00:34:26,709 Peop- people can hard code the passwords right in the script. 650 00:34:27,159 --> 00:34:30,739 The one thing that it, they did have all in common, though, 651 00:34:31,139 --> 00:34:33,059 is that they were, all plain 652 00:34:33,059 --> 00:34:34,913 Text files, okay. 653 00:34:34,913 --> 00:34:35,383 All right. 654 00:34:35,539 --> 00:34:42,359 a and I think that's the key is like with all of these text files like And if you 655 00:34:42,359 --> 00:34:45,999 have so many developers it's like how do you know who wrote what and did they 656 00:34:45,999 --> 00:34:50,339 clean up after themselves Because maybe a project existed and then got shelled 657 00:34:50,339 --> 00:34:54,629 but none of the data ever got cleaned up And so how do you actually figure 658 00:34:54,629 --> 00:34:59,499 that out Like I totally get like it just seems impossible right with the sprawl 659 00:34:59,603 --> 00:35:02,893 it's definitely... So you, first we're gonna, do no future harm, 660 00:35:03,283 --> 00:35:04,523 like a modified version of the 661 00:35:04,523 --> 00:35:05,523 Hippocratic Oath, right? 662 00:35:05,523 --> 00:35:06,463 Do no future harm. 663 00:35:06,883 --> 00:35:09,073 Moving forward, we're going to do better, right? 664 00:35:09,073 --> 00:35:13,403 Th- this is, the, my, my favorite Maya Angelou quote, right? 665 00:35:13,403 --> 00:35:15,233 "We did what we did when we knew what we knew, but now we 666 00:35:15,233 --> 00:35:18,423 know better, so we'll do better." So you say, "Moving forward, we're gonna use..." 667 00:35:18,983 --> 00:35:20,433 Pick your favorite secrets manager. 668 00:35:20,553 --> 00:35:24,473 Do an on-prem one, use AWS Secrets Manager, use your Azure, whatever. 669 00:35:24,473 --> 00:35:27,443 you're gonna... You're never again going to store credentials 670 00:35:27,443 --> 00:35:29,033 of any kind in a plain text file. 671 00:35:29,239 --> 00:35:33,809 shalt not store credentials in plain text file or be smitten down 672 00:35:34,543 --> 00:35:36,203 Nothing good comes of that, right? 673 00:35:36,443 --> 00:35:39,033 how do we then go and audit? 674 00:35:39,523 --> 00:35:41,163 And by the way, this isn't a one-time thing. 675 00:35:41,163 --> 00:35:44,803 You should just, you should be regularly doing this to make sure that no one has 676 00:35:44,803 --> 00:35:47,673 violated your new principle of design. 677 00:35:48,383 --> 00:35:51,333 And the answer to that, there's a couple of different tools, and 678 00:35:51,333 --> 00:35:54,903 the one that seems to come up more often than not is called TruffleHog. 679 00:35:55,533 --> 00:35:56,263 Do you know why it 680 00:35:56,263 --> 00:35:57,493 might be called that? 681 00:35:58,729 --> 00:36:03,319 So I know about truffles and I know that they have specially trained pigs 682 00:36:04,649 --> 00:36:11,099 to hunt down truffles because they are very expensive and found in forests 683 00:36:12,113 --> 00:36:12,403 Yeah. 684 00:36:12,563 --> 00:36:16,033 I think that's a perfect name for this tool, given that these are really 685 00:36:16,033 --> 00:36:21,823 valuable things that we wanna find, and then we wanna pull 'em out, and 686 00:36:22,403 --> 00:36:25,483 then, that's where the analogy stops, 'cause we're not gonna eat them, right? 687 00:36:25,483 --> 00:36:26,693 we're gonna delete them, right? 688 00:36:27,173 --> 00:36:29,823 and so i- it's a, it's an open source tool. 689 00:36:29,853 --> 00:36:34,843 There is a free, a feature, a f- free version that has features, 690 00:36:34,843 --> 00:36:37,023 and then there is a commercial version that has more features. 691 00:36:37,333 --> 00:36:40,973 there is a similar tool, from a company called, GitGuardian, 692 00:36:41,413 --> 00:36:42,633 and It also has a free 693 00:36:42,633 --> 00:36:43,033 tier. 694 00:36:43,583 --> 00:36:47,693 The... But TruffleHog seems to be the one that's popping up a lot, and that 695 00:36:47,693 --> 00:36:50,973 it's an open source tool, and you can run it across, and what it's gonna 696 00:36:50,973 --> 00:36:55,533 do is it's going to use, a variety of techniques, look at using pattern 697 00:36:55,533 --> 00:36:57,743 matching and also entropy detection. 698 00:36:58,063 --> 00:37:01,223 basically, it's gonna look at your, your Git repository. 699 00:37:01,553 --> 00:37:06,803 it's gonna look at all all sorts of stuff, looking for, hidden, stored, 700 00:37:07,143 --> 00:37:12,143 credentials that could be, that could ultimately be your undoing, your 701 00:37:12,143 --> 00:37:13,863 version of what happened here, right? 702 00:37:14,193 --> 00:37:17,153 Because that's the problem, is most people find out about these 703 00:37:17,153 --> 00:37:20,743 credentials when they've done them harm, and we wanna make sure that 704 00:37:20,743 --> 00:37:22,293 we're doing that on a regular basis. 705 00:37:24,363 --> 00:37:26,133 So what's our summary, Prasanna? 706 00:37:27,229 --> 00:37:34,189 So I would say th so the summary I would say is don't believe the news 707 00:37:34,189 --> 00:37:36,929 hype This was not really an AI issue. 708 00:37:39,779 --> 00:37:45,549 make sure that you do not store your backups and your production together 709 00:37:47,759 --> 00:37:50,429 you're testing your backups Follow the 3-2-1 rule 710 00:37:54,108 --> 00:37:54,428 Yeah. 711 00:37:54,498 --> 00:37:56,148 three, three two one one zero. 712 00:37:56,919 --> 00:37:58,889 Okay 1-1-0 And then 713 00:37:59,159 --> 00:38:02,649 make sure you are using a secrets manager and not storing credentials 714 00:38:02,649 --> 00:38:08,069 in plain text files and granting access to only things that are needed 715 00:38:08,069 --> 00:38:11,439 And if things beyond that are needed permissions beyond that are needed 716 00:38:11,699 --> 00:38:13,119 it should have a very short timeframe 717 00:38:14,958 --> 00:38:21,868 And you should run tools whatever it is in order to be able to check your environment 718 00:38:21,868 --> 00:38:27,958 on an ongoing basis to make sure that people are following the processes and 719 00:38:27,958 --> 00:38:32,938 using secrets managers and not storing these things outside of that system 720 00:38:33,969 --> 00:38:34,699 Yeah, absolutely. 721 00:38:35,129 --> 00:38:37,159 I think the only thing you missed at the very 722 00:38:37,159 --> 00:38:39,929 beginning was not storing your production and development 723 00:38:39,929 --> 00:38:41,139 at the same, on the same thing. 724 00:38:41,439 --> 00:38:42,449 You had... you got 725 00:38:43,329 --> 00:38:44,079 almost everything. 726 00:38:44,639 --> 00:38:47,099 You said produc- you said backups on production. 727 00:38:47,099 --> 00:38:49,109 You said ba- don't store them in the same place. 728 00:38:49,109 --> 00:38:52,759 You didn't say don't put production and development on the same 729 00:38:53,309 --> 00:38:53,759 environment. 730 00:38:54,959 --> 00:38:55,259 Yeah, 731 00:38:55,728 --> 00:38:56,698 course you had 732 00:38:56,978 --> 00:38:58,608 to find some fault 733 00:38:59,709 --> 00:39:01,229 Of course, it is my job. 734 00:39:01,409 --> 00:39:01,989 I live, 735 00:39:02,278 --> 00:39:04,238 Fine go talk to your French friend I don't wanna talk to you 736 00:39:04,868 --> 00:39:05,498 to your French friend 737 00:39:06,479 --> 00:39:08,309 By the way, you can see this sign right here. 738 00:39:08,409 --> 00:39:10,339 I am very silently correcting your grammar. 739 00:39:11,559 --> 00:39:13,209 that, this was a great story. 740 00:39:13,209 --> 00:39:17,449 It's a great story, I love that it's a very sad story that had a good, 741 00:39:17,449 --> 00:39:20,959 happy ending that we can learn a lot of... So no one was harmed in 742 00:39:20,959 --> 00:39:25,299 the making of this film, but, but we can learn some valuable lessons from it. 743 00:39:26,089 --> 00:39:27,949 and with that, go and do no harm. 744 00:39:31,019 --> 00:39:32,669 All right, that is a wrap