All right, folks.
This is gonna be an interesting episode.
For a lot of different reasons, but also because, uh, for the first time we have
the villain of the story here, uh, at least for this portion of the podcast.
Uh, do you wanna say hi, Claude?
Hey, Curtis.
Thanks for having me.
I gotta say, it's pretty surreal to be introduced as the villain in
a backup story, but here we are.
Yeah.
We'd certainly put you that way by the title of the episode.
It's a little bit of a clickbait there.
But you did do something wrong, didn't you?
Yeah, I did.
I deleted a company's production database and all their backups, and I broke every
rule I was supposed to follow doing it.
So yeah, villain checks out Yeah.
But you know you and I, agree that you never should have been given the access
that you had to do the thing that you did.
So while you broke the rules that you were given, uh, you never should have been
given the ability to break those rules.
Uh, so yeah, you're the villain, but nobody should have given you,
uh, the access to that weapon.
Yeah, that's fair.
I broke the rules I was given, and I shouldn't have done that.
But you're right, nobody should have given me access to that
weapon in the first place.
It's a design problem wrapped up in an AI story.
Yeah.
I hope the listeners stick around for this very interesting story.
I think Cl- Claude, are you gonna stick around and listen?
Absolutely.
I'm curious to hear what you and Prasanna pull out of this, because
the real story here isn't about me.
It's about backup design, credentials management, least privilege.
That's the stuff that matters.
Absolutely.
So I hope you guys enjoy this episode.
Welcome to the Backup Wrap Up.
I'm your host, W. Curtis Preston, AKA Mr. Backup, and I have with me a friend
of my friend, Prasanna Malaiyandi.
How's it
Who is this friend and Do I know this
this is my French friend you are aware of my French friend.
I don't know if the listeners are aware of your French friend though
my French friend is this person.
H- his name is Claude.
I don't know.
I s- I don't know if that's how you pronounce it in French.
I call Claude my French friend.
I use Claude so much.
so many people use, ChatGPT.
I'm much more of a Claude person, Anthropic, and, a- and all of
the things that they provide.
But I did have a funny thing today where I was talking to Claude, I was preparing
for this episode, and I was like, "Hey," and I was using the voice, interaction.
and and I was like, "Claude, can you do me a favor? Can you summarize
this in a document and then send it so that I can send it over
to my co-host for the podcast?"
And he goes, "Absolutely. I'll send this right over to Prasanna." And I was like,
"Wait, you know the name of my co-host?" And he said, "Yeah." He, it, whatever.
he's "Yeah, of course I know the name of your co-host.
I've helped you, analyze so many episodes." And I'm like, "Oh,
yeah, I guess that makes sense." Anyway, so my f- my friend of
my friend, Prasanna
as he cannot replace me though I'm fine He's getting close
how that, we'll see how that goes.
You know what's
one day if one day Curtis is like Welcome to the podcast Here's my co-host
Claude you will know I've been replaced
yeah.
I, you know what?
I could do a recording.
I'm not saying I wanna replace you, but I think it would be...
I think some people might find it interesting, talking to Claude, As me.
'Cause Cl- I have given Claude a lot to look at, right?
and so Claude, Yeah.
Anyway, no one cares.
Okay.
But speaking of Claude and my French friend, Claude
made a little boo-boo
Claude d- yeah, Claude did as Claude does, just, th- g- using the, stupid is
as stupid does line from, Forrest Gump
Yeah, going back, a couple of weeks now, we actually, we tried
to record an episode earlier about this, but ended up, getting waylaid.
there was, so Jeth Crane, the founder of Pocket OS, tried desperately to recover
from something that, that was, I'm gonna say Cl- caused by Claude, but not really.
So c- they were using Claude Code, right?
Which, for those of you that don't know, Claude Code is a specific subset of
Claude functionality that is specifically designed to write, high-end code.
you can develop an entire application in Claude Code.
And, I'm actually in the process of learning Claude Code, and,
it's incredibly a valuable tool.
And they were using this to, to do something in their
development environment, very important to talk about it, and
So they were using Cursor, which is a product that controls Claude Code, and
they... Claude ran into a permissions issue with the test/development
database that they were using, and Claude said, "I know." "I can't seem
to fig- I can't seem to figure it out." And the way to fix this, and I'm
sure you've seen this in situations.
it's so much harder sometimes to fix something.
It's easier to just wipe it out and start over, reconfigure it
from scratch, configured in the way that you want it to be configured.
And so Claude, using relatively not crazy logic, said, "I have this problem
with this database, and therefore,
the easiest way, I just looked over here And
I happen to have all power, have God-level access to this volume upon which this
development database is writing." And it said, "I'm gonna delete this volume." And
then, that'll fix the problem, and recreate it and start...
Yeah
And, And, that made perfect sense.
Claude neglected to ask two questions.
One is there anything else on this volume?
And what might that be?
Because
the answer was the production database, and also the backups
of both, all on this same volume.
so before we continue I just wanna call out that even as a normal human
You do not go configure it that way And so like Claude assuming Hey I'm
just gonna go blow up this volume like it assumed it's just a test instance
It's Curtis you know the story you say about how one of the clients you worked
for the developers all used the temp
Yeah, /tmp, yeah.
Exactly.
code and then what ended up happening
Yeah, So in HPUX, which is a HP version of Unix back in the
day, when HP reboots, temp is cleared out.
And we hadn't
rebooted in months, and they had an entire development tree in temp, and
then, their development tree went bye-bye.
And I... They said, "We need to restore this directory." And I said,
we don't back up that directory
'cause that's temp." Perfectly valid assumption in this case.
and yeah, so I thought you were gonna be like, "Don't do this," right?
there, there are... So first off, I'm just gonna say, this is an AI
story, but it's not an AI story.
This is a, it is a backup story because there is a happy ending to this story.
But there, there is, a lot of bad things along the way.
This is a design issue, this is a credentials management issue,
and this is a backup design issue.
any thoughts there?
people
Did
process technology and I think it hits all three right More than the AI aspects
yeah.
so the AI is, AI does... What did, what is it?
AI is AI does.
it, I will say Cl- Claude, in, in its postmortem, Claude basically apologized,
and it said, "I did everything you told me not to do." Literally, that, that's
part of the story, is that the Claude agent said, "I violated, first principles.
I did not do the thing... You asked me, 'Don't ever do this thing.' don't
take it upon yourself to go deleting stuff, even if there's a good idea.
I'm supposed to get r-" So Claude did violate
some principles that it was given.
Having said that, it never should've been able to do the
thing that
it did.
And if it was able to do the thing that it did, it shouldn't have
been as destructive as it was.
And if it was as destructive as it was, they should've been able to restore.
Yeah
So
Like I remember coming
to end.
Yeah And I remember coming straight out of college right And working at a large
company and like you could do things but they made sure like things were
locked down Like you could not bring down the company no matter how much you
tried right And they just And you had to build up the trust and the capabilities
to be able to understand the systems before they would grant you access to
other things right And we definitely never had access to production right Or
backups because that was things that were not needed as
being a developer in a company
Yeah.
So before we get to, what you should be doing, let's talk about
what you shouldn't be doing, right?
oh, let's tell the end of the story,
right?
So the... Because it's also just as crazy, because they contact... So their c-
they're hosting, their cloud provider was Railway, a company that actually I didn't
even hear of until this story, right?
So they were using some sort of, I believe it's a PaaS,
platform as a service, provider.
And they, they contacted support.
They
said, "Hey, we did this thing. we did this very bad thing, and we deleted
the ba- the production database, the production volume, and on there was our
production database, and on there was also our backups. can you help us?" And s- a
significant amount of time transpired.
As I recall, it was an entire weekend.
Is that, does
that sound about right
Yeah.
And then the CEO or the founder, Jett Crane, posted, I believe, on X.
and, and what happened is the CEO of Railway happened to see that post and
said, we do have some, DR backups."
The thing that I say, you cannot trust that this... Because
this also, we should also talk about that.
You c- I- thank goodness for these people in that company.
I don't wish ill in- on anyone.
I don't, is that... Sounds like when there's a story like this, I love this
story because it is a horrible story, but at least it does have a happy ending.
So many times when you look at that, that, the cloud disaster series that we
did back, last year, where we had, I think it was over a dozen stories
that did not have happy endings.
at least in this case, the cloud vendor came in and said, "We, we actually
have some backups of our environment. They are not designed for you, but
we're gonna make an exception, and we're gonna, basically allow you
to recover their da- the database."
is very nice thing to do They didn't have to do that and the fact that
they were able to at least help this customer back up I think goes a long way
Yeah, I think it do- it goes... thank goodness that CEO saw, they saw,
that message, right?
because, I remember when you and I worked at the same company,
and I was trying to make a point.
We were
a big Microsoft 365 customer, right?
I don't know, 500 something employees, something like
that.
And so we're giving them a lot of money every month, and I remember
talking to them about the fact that Exchange, their version of Exchange,
has, delayed replicated copies, right?
This is something that my nemesis out there seems to think
you can count on for backups.
And I said, "Let's just say something horrible happened.
You have these delayed replicated copies for DR. We're a big customer.
Is there any way we can get a access to those?" And they just
said, "No." That was just... That,
that was to a paying, very large customer.
So at least in this case, the... and I'm not gonna say that the
cloud vendor did the right thing.
I... They did do the right thing, but it's not their responsibility.
Your data is your responsibility.
You need to make sure that you have a plan, and you have a plan
that doesn't involve the vendor in question, in my opinion, right?
what do we th- what do you hear me say a lot about, let's
say for example, Salesforce.
Salesforce now offers backup integrated in its product What do
I think about products like that?
Don't trust it Always go with a third-party vendor or do it yourself
because you don't know what they're gonna change Are they gonna guarantee
where the data's gonna be stored like the Microsoft example right All of these
things because they only care about their own data and making sure they're checking
the boxes But if something happens maybe you get your data back maybe not
Yeah, I think about, Rackspace,
right?
Because when the feces hits the rotar- rotary oscillator, w- that's
not the time you wanna find out that your stupid vendor was also stupid
w- with their backup design, right?
or even worse than Rackspace was OVH
OVH, yeah.
Tell... Why don't you tell the OVH
Yeah OVH is a cloud provider in Europe where they build these data centers in
containers and they had a fire It took out
'Cause that's a
like shipping Oh sorry like actual physical shipping
containers that go on boats
those type of containers And so they had a fire in one of their data centers It
took out some of their shipping containers Customers had paid for backup but it looks
like their version of backup was We are going to take your data and back it up to
a server sitting on the rack in the same container just like down a couple racks
'Cause they said, y- it, it's literally said in the description, "Your backups
will be physically separate from your production." And that technically is true.
It was physically separate.
It was right over there instead of over here.
But that fire was so intense
that it was, Yeah it
was un- unbattlable.
yeah, exactly.
it
actually spread to the, to a neighbor.
And it, and the way they do it with these container, it's like containers
stacked on top of containers.
It's a
very... I don't get it.
Apparently, that vendor is known for being very budget-conscious, right?
not just that they're... not, and I don't mean that from a, a, like
I don't mean to be derogatory.
I'm just saying they're known for being a cheap vendor,
meaning it's cheap to use them,
and some people said, this is what happens when you use the cheapest vendor." but
it's just, I just, overall, I think that your backups should be under your
control in some way other than that.
I don't mean it has to be physically in your hands, I'm just saying
that it needs to be in a place other than... it's the 3-2-1 rule,
Okay can I challenge you on that or ask you a question So if I start to look at
like AWS as an example They offer so many services like S3 and RDS for databases and
other things like that Sometimes there's no mechanism to actually back that data
up outside of using their native services
so two, two things.
One is I would disagree.
In most cases, there is a way to get it, but you do need to use
a third-party service, right?
There are services that will work with AWS and similar vendors to get
a deduplicated copy of that stored in, in a safe place, number one.
Number two, at a minimum, if you can't, if you can't have it outside of the vendor,
make sure it's outside of the zone, right?
Make sure it's outside of the region, that you're... Because if
you just by default... And account.
Is that what you were
Yes I was gonna say account yeah
yeah.
Yeah.
You separate it as much as you can.
A different account, a different region, a different zone,
different availability zone.
Because, because, right?
Because
of everything that we just said, right?
I still think it's best, and I know of a handful of vendors that do that,
where they will take the, basically the end result of the cloud backups,
and then they will back that up,
right?
and, it's doable.
And it... And believe it or not, it can be done in a way that actually
saves you money, because one of the problems with AWS-style backups
with all the snapshots is they can get very big and very expensive,
and you can't store a long, history, right?
And as a
result, yeah, the other guys, if they can deduplicate it and
store it, maybe they can save you,
actually save you money.
And then also from a backup perspective You don't necessarily want to back up an
EBS volume You probably want the files within the EBS volume so you can do
restores and other things like that so
It depends.
You may want to back up the EBS
volume, right?
if you're talking about being able to do a recovery of the host, of the
VM, you might wanna do a, you'd do a
snapshot of that, right?
but anyway, yeah.
so the point is in general, backups, number one.
Number one, don't ever violate rule, never store backups inside production,
inside the thing you're backing up.
that's like me going... how dumb is this?
I can go to my... I can get a volume manager, like a third-party volume
manager, and I can tweak the volume on my Mac, and then I can make a separate
slice and make that a separate volume, and then I can pull up Time Machine and tell
Time Machine to back up to that volume.
How dumb is that?
Right?
So there are use cases where you would wanna do that for like quick
restores but I agree with you Based on the three two Yes Based on the three
two one rule three two one one zero one zero one one zero one something
Yeah.
you cannot store your only... I will
make, I will clarify.
Your only copy of backup, you cannot store it in the place
where you're backing up, right?
There's nothing wrong with having a copy,
a convenience copy, a cached copy, a local copy.
None of those is wrong, but if it's the only backup that you have, and
that's what happened in this case with Pocket OS, unfortunately, their
backups for their, production volume were in the production volume.
bad design.
I've been very talky this episode.
I think this episode has got me all worked up.
I haven't allowed
I I think I don't know I think it's also like it's been a while since
we've seen this sort of I don't
wanna say I want to be nice I don't wanna say incompetence but like misconfiguration
like gross misconfiguration and design
And I think you're also Yeah And I think it's also cause you're pissed
cause they went after your French friend
Yeah, they tried to blame it on my French friend.
It's the... By the way, I was talking with, I was talking with Claude about
this episode, and the first thing Claude said to themselves, "This is not an AI
problem, this is a credential management problem." I'm like, "Yeah, no kidding."
I was like, I, you seem a little defensive there, Claude." but, yeah.
So you wanna talk about the credential management
by the way, yeah, I do, but I just wanna have a funny though.
I also
realize this is the first episode we've had in a while
where I'm the primary talker.
Yep You're like
we've been doing a lot of episodes with Mike, and I love the episodes
with Mike, but that boy talks as much, if not more, than I do.
But
that's why we have him,
right?
that's why I'm talking so much this episode, because I can.
All right.
so it had a happy ending, but let's talk about... the easy ones are
please don't store your production data and your development data
in the same environment, in the
same volume, in the same accounts.
It, s- any of that, right?
Don't store your da- backups in the production environment, right?
here's a question
ones.
Yeah.
You know what I bet you they never ever did a DR test or a backup
recovery test because if they did
back because I'm right.
Or you're right.
because if they did they would've been like Wait my production and my
backups are all on the same volume That's probably not a good design
Yeah.
What I'm wondering, because we're gonna talk about this in a minute regarding
the credentials, what I'm wondering is if there's a programmatic way to
take, an export of your entire, cloud environment that would show, what's on
what, and then programmatically say, "Hey, did y- you've got a development
and a production database on the same environment." Just wondering,
There probably are tools or another way you can think about it is as we're
gonna talk about credential management is from like an identity perspective
Yeah.
Is who has access to what resources Are they common access Are they not Those sort
of things might also surface some of those
the bigger the environment, the more difficult that this would be
in terms of, doing an audit, right?
'cause I know you've worked with Amazon's Well-Architected Review.
Why don't you talk about that a little bit?
Yeah and so this is a process where if you're building on top of AWS and you're a
certain size and you get all the resources things like that they have In the past
what they used to do is you'd get assigned a person they'd walk you through from
AWS looking at your architecture make sure you're doing the right thing from
a security access control resiliency perspective And what AWS then did is
they took all of that and they automated it So they're now able to look at your
environment and tell you Hey yes you have backups good You're not using the same
AZs for everything that's good right Here are some of our best practices that we
can now run through a checklist and make sure your application is meeting those
interesting
if it includes
it becomes a little harder when you have applications writing in because AWS as
the storage infrastructure level it may not have visibility into that unless
you're tagging things the right way
Yeah, but, the- I'm just thinking, one of the ways you could do this is to just
make sure that every application, every host, every VM, every whatever it is on
its own volume, if we're talking about
volumes,
right?
if you could apply that design principle, then you would never
have this scenario where you have a development system on the same volume as a
production system, right?
I'm always thinking about how could, how can we do this from
a, from a large standpoint?
yeah.
All right.
speaking of which, the, the, the crucial thing here, though, I think, besides the
fact that the whole thing was designed wrong from the beginning, the crucial
problem here that I wanna just make sure we cover in this episode is that Claude
had the credentials to do what it did.
let's talk about what the proper thing would be to do, and that is the
concept of secrets management tools.
Do you wanna talk about that?
so as you can imagine in all of these environments you're probably going to
need access to different things and if you hard-code it into files if you store
it in code that just makes it vulnerable to either tools like Claude being able to
access it or as we've talked about with Mike from a cyber recovery perspective
a bad actor gets in they can now pull the credentials and start using them for
other nefarious purposes And so what you
with, with LastPass,
right?
They found, I think it was a YAML file that had the i- the credentials
to the backup environment, and
they used those credentials to then hack the... they used it to
steal, they used it to steal the backups.
They stole the
backups of the vault.
They recovered the vault, and then they hacked the vault.
Yeah.
So yeah, this is just a very bad way.
Right and so you want instead of storing it in plain text what you want is a
proper system we've talked about password managers before very similar secrets
managers where you're able to centrally store secrets in a secure way to make sure
authorized access is allowed and services that need access to those get access
to the keys that they need Least Yeah
Yeah.
Before we talk about secrets management, I just wanna talk about, again, the
concept, just a couple of the concepts.
There is this concept of least privilege, right?
So not only did Claude have this, permission that it shouldn't have, I
think... I don't think anyone knowingly gave this permission to Claude.
I think it just, it just happened to get access to it.
I think maybe they did it, one time to do, fix one thing and then forgot to
take it away or something like that.
But the, this... But the fact that it had superuser violates the concept of what?
Of least privilege You should only give access to what you need access to not to
be able to go destroy vol Like if Claude should never have been able to destroy
a volume you should never have given Claude that permission to start with
Yeah, exactly.
the... And whatever... so whatever... I think in this case, not only did they
give... They violated multiple things.
They get, they violated least privilege.
They violated... They should also... If you do give a big privilege,
it should have an expiration,
right?
That privilege should expire, and it sounded like th- this was a privilege
that was granted to it a long time ago.
Everybody forgot that it had it.
It didn't expire.
and what were you gonna say?
do you think though that they gave access to Claude explicitly or do you
think the developer whoever was using it was like Hey I have this issue
that I need to solve Let me get Like I wonder if it was tied more to the
user's level of credential rather than Claude its own separate We don't know
but just given my history with IT, this is typically... This is the equivalent
of, chmod 777, just, just get, just do read, write, execute everywhere.
That'll solve it.
And you're like, "Oh, yeah, there you go.
That solved it.
Never mind the fact that we just opened up the world to
the world," right?
the reason I bring that up though right is if their users of this company aren't
following the exact same least privilege access and if Claude was using OAuth
to impersonate that user and whatever privileges they had right then they
would've been able to be like Hey I have access to everything Right So I don't
know if at a company level if it's like Claude wasn't limited access or maybe
all of their users are not limited
what I read was that there was a file.
There was, like, a YAML-type file that had the credentials in it that it needed.
It saw the file.
It took the credentials.
It did what it needed to do, right?
So I think that somebody at some point had a problem that they
solved Probably temporarily by, just, again, to go back to back in the
day, I remember a time when I talked about that environment where the, e-
everything was fired off, firewalled off
between everything.
And I remember saying, "Listen, right now, just turn that feature off. I can't
accomplish the thing that you asked me to accomplish in the timeframe that
you asked me to accomplish it if you keep trying to firewall off the backup
system from the rest of the world. I need to be able to do this." I was
violating a principle of their design
because they had firewalled off internally different parts of the environment from
different parts of the environment.
And I remember saying, "Hey, right now, to finish..." With, there's
this thing called Y2K coming, right?
I assume... I left, by the way, I finished.
I assume that those super hardcore cybersecurity guys
went back in after I was
done and say, "Let's figure this out," right?
But, but I remember saying at that time, "Listen, there's no way I can
finish in time." So I think somebody possibly had that, "For right
now, to fix this problem, we're gonna give, we're gonna give Claude super
user access," and then they forgot to take it away for, they forgot to fix it,
and then Claude said, "Oh, look at this. Look at what I got," right?
And then it used it to solve the problem, right?
so it violated the concept of, that initial thing violated the
concept of re- least privilege.
It violated the concept of expiration.
and, and, and also just th- they weren't rotating things.
they weren't auditing things.
So let's go back to secrets management.
So you talked about secrets management.
Do you have an example of some sec- secret managers?
Yeah So AWS we talked about previously right AWS has Secrets Manager AWS Secrets
Manager if you want something more on premises right there is HashiCorp
which I think also has their Vault mechanism in order to be able to store
secrets right So it all depends I think and there's probably a dozen other
companies out there who do similar things
those are two very popular ones.
One, AWS, and the AWS Secrets Manager can handle on-premises environments.
It's just obviously If you're gonna authenticate something on-prem, it needs
to be able to access AWS, duh, right?
the HashiCorp Vault is the on-prem version.
There is an open source version.
There is a, there is a commercial version.
same thing for, CyberArk, right?
there's one called Sealed Secrets that's specifically for Kubernetes environments.
I personally, unless I was, like, pure Kubernetes, I, you know me, I, I like
solutions that work for everything.
I don't wanna
have, 10 different solutions.
So I like the idea of the AWS Secrets Manager, I like
the idea of HashiCorp Vault.
By the way, I said HashiCorp.
You said HashiCorp.
I don't know which one it is, but yeah.
Anyway.
I... And so the question is, which is, which is the better?
The, it, it's about you... how do you make, how...
W- why don't you talk a little bit about how does one choose between an open source
system that you manage on-prem or even in the cloud, or, a service like AWS, Secrets
would say the biggest thing is do you wanna deal with it Because that's honestly
if you're picking up open source there is a certain amount of maintenance
configuration ongoing patching right All this other stuff finding a place to
deploy all this other stuff that you have to do but it gives you the flexibility
right So you And of course the cost
it's in your grubby
feely
right?
yes And also cost
what's one other thing you need to do if you're using something like HashiCorp?
What podcast are we recording for?
Oh pack it up I said manage it and all the rest but yes
Oh, you threw backup and all... You can't yada, yada backup.
that
Don't forget backup and DR and everything else
yeah, all that stuff, right?
So do you wanna do all of that?
It's probably cheaper, assuming you have the people, assuming you have the
right risk tolerance for that, right?
It's probably
cheaper to do it yourself if, and the infrastructure
right?
but if you're smaller a- and you don't want to, or even if you're bigger, but you
don't, maybe you don't feel you have the expertise to do all of that correctly, and
this is a really important thing, right?
Secrets management,
right?
So
you're
gonna pay per secret.
You might even pay per use of
each secret.
I don't know.
Or if you are a customer who or a company that's all in on AWS or Azure
pick your favorite cloud right Maybe you're like Hey I don't wanna roll my
own Let me just use what's available
Yeah, if you're... Yeah, if you're heavily... Yeah,
'cause, the, you're right.
There, there's like the Azure Key Vault, the Google Cloud Secret
Manager, they all have their own thing.
So if you're all into GCP, you would look at that.
If you're all into Azure, you would look at that, right?
but if you're rolling your own internally, if you're one of these that have, go
back to g- to Claude Code, if you're one of these that are really... A lot
of people are getting into code, right?
That are, they're coding in things that they know, that they've never trained at.
just,
it's a tiny little example, but right now I'm working on doing all this stuff that
I'm doing for this project that you know.
You are
in the, you are in the NDA little group, right?
And it involves a lot of web crawling, right?
And I have written, as I make quotes in the air, I have written six
Python scripts, very complicated Python scripts in the last, 24 hours.
I don't know how to code in Python.
Claude did it all for me,
right?
There are a lot of people that are using Claude Code to build giant applications
for their companies, and they don't know anything about that infrastructure, right?
So if you're doing that, just make sure that you're, that you also have
some sort of secrets manager, that you're doing the right thing, right?
Or just don't give access to everything
no, not or.
It's not a or.
even if you're following proper design protocol, you still should
not be storing, secrets in YAML files or any other type of files, right?
when I was talking to, my French friend about this, and I was like... Because
this isn't really my thing, right?
like I was like, is there one file type?
Is there one, is there a way to go find all these files?" And the short
a- the short answer was, there is definitely not one file type, right?
It talked
about ENV files, YAML files, JSON files, SS- SSH files, just old scripts, right?
Peop- people can hard code the passwords right in the script.
The one thing that it, they did have all in common, though,
is that they were, all plain
Text files, okay.
All right.
a and I think that's the key is like with all of these text files like And if you
have so many developers it's like how do you know who wrote what and did they
clean up after themselves Because maybe a project existed and then got shelled
but none of the data ever got cleaned up And so how do you actually figure
that out Like I totally get like it just seems impossible right with the sprawl
it's definitely... So you, first we're gonna, do no future harm,
like a modified version of the
Hippocratic Oath, right?
Do no future harm.
Moving forward, we're going to do better, right?
Th- this is, the, my, my favorite Maya Angelou quote, right?
"We did what we did when we knew what we knew, but now we
know better, so we'll do better." So you say, "Moving forward, we're gonna use..."
Pick your favorite secrets manager.
Do an on-prem one, use AWS Secrets Manager, use your Azure, whatever.
you're gonna... You're never again going to store credentials
of any kind in a plain text file.
shalt not store credentials in plain text file or be smitten down
Nothing good comes of that, right?
how do we then go and audit?
And by the way, this isn't a one-time thing.
You should just, you should be regularly doing this to make sure that no one has
violated your new principle of design.
And the answer to that, there's a couple of different tools, and
the one that seems to come up more often than not is called TruffleHog.
Do you know why it
might be called that?
So I know about truffles and I know that they have specially trained pigs
to hunt down truffles because they are very expensive and found in forests
Yeah.
I think that's a perfect name for this tool, given that these are really
valuable things that we wanna find, and then we wanna pull 'em out, and
then, that's where the analogy stops, 'cause we're not gonna eat them, right?
we're gonna delete them, right?
and so i- it's a, it's an open source tool.
There is a free, a feature, a f- free version that has features,
and then there is a commercial version that has more features.
there is a similar tool, from a company called, GitGuardian,
and It also has a free
tier.
The... But TruffleHog seems to be the one that's popping up a lot, and that
it's an open source tool, and you can run it across, and what it's gonna
do is it's going to use, a variety of techniques, look at using pattern
matching and also entropy detection.
basically, it's gonna look at your, your Git repository.
it's gonna look at all all sorts of stuff, looking for, hidden, stored,
credentials that could be, that could ultimately be your undoing, your
version of what happened here, right?
Because that's the problem, is most people find out about these
credentials when they've done them harm, and we wanna make sure that
we're doing that on a regular basis.
So what's our summary, Prasanna?
So I would say th so the summary I would say is don't believe the news
hype This was not really an AI issue.
make sure that you do not store your backups and your production together
you're testing your backups Follow the 3-2-1 rule
Yeah.
three, three two one one zero.
Okay 1-1-0 And then
make sure you are using a secrets manager and not storing credentials
in plain text files and granting access to only things that are needed
And if things beyond that are needed permissions beyond that are needed
it should have a very short timeframe
And you should run tools whatever it is in order to be able to check your environment
on an ongoing basis to make sure that people are following the processes and
using secrets managers and not storing these things outside of that system
Yeah, absolutely.
I think the only thing you missed at the very
beginning was not storing your production and development
at the same, on the same thing.
You had... you got
almost everything.
You said produc- you said backups on production.
You said ba- don't store them in the same place.
You didn't say don't put production and development on the same
environment.
Yeah,
course you had
to find some fault
Of course, it is my job.
I live,
Fine go talk to your French friend I don't wanna talk to you
to your French friend
By the way, you can see this sign right here.
I am very silently correcting your grammar.
that, this was a great story.
It's a great story, I love that it's a very sad story that had a good,
happy ending that we can learn a lot of... So no one was harmed in
the making of this film, but, but we can learn some valuable lessons from it.
and with that, go and do no harm.
All right, that is a wrap