ATR2500x-USB Microphone & Logitech BRIO-2: If you're like the majority of people
I've talked to, you don't really have an incident response plan for ransomware.
That means you'll be flying by the seat of your pants when you get hit.
The good news is I have just the episode for you.
It's an extremely popular episode from earlier this year where we talk with
Melissa Palmer about what to put in your response plan and how to build it.
It was so popular that it makes the perfect episode for our winter break.
Even if you've heard it before, it's worth a second.
Listen.
If this is your first time listening to us.
Hi, I'm W Curtis Preston AKA Mr.
Backup.
And I've dedicated my career of over three decades to helping those of
you that have the job that I had when I first started the backup person.
This podcast is just for you.
We turned backup admins into cyber recovery heroes.
This is the backup wrap up.
ATR2500x-USB Microphone-1: Welcome to the show.
W. Curtis Preston: I'm your host, W Curtis Preston, aka a Mr.
Backup, and I have with me my super expensive vacation planner coordinator.
How's it going?
Prasanna
Prasanna Malaiyandi: I'm doing well, Curtis, how are things going?
Are you excited?
We are.
I we're having technical difficulties, as you could tell.
We're trying to keep this real, but yes, doing this for the fifth ta,
fifth time, it's a little hard, but
W. Curtis Preston: I am excited, um, uh, and my wife is starting to get excited.
I started showing her some pictures a while ago and she's
been like downplaying it.
Like she doesn't want to get excited.
She wants to be sort of, Excited, but I needed her to prep for the vacation
because this is, so this is, we're going to the Maldives, uh, which for
those that don't know, is a series of islands off the southern coast of India.
And, um, and, and I'm on one of those islands and, and it's a tiny island that
literally we could walk from one end to the other in probably about 10 minutes.
Um, and.
We're staying in one of those things over the water,
Prasanna Malaiyandi: Oh, the Villas over the.
W. Curtis Preston: villas over the water with our, we have our own
pool, and then right on the other side of the pool is the ocean.
Um, I mean, it's
really, really cool.
Prasanna Malaiyandi: Can I stow away in your luggage
W. Curtis Preston: Yeah, I mean, it looks really cool.
uh, we're very excited.
We're having our, a repeat guest and, um, we, we had her on, uh, a few
weeks ago and we got talking about ransomware, one of our favorite topics.
And we, we, we got into this phase where it was like, you know what?
That, that is a great conversation, but there's no way we could, we could
do it justice on that recording.
So it was, Hey, we're gonna have her come back.
And, uh, she is, uh, she's been in the industry for quite a while and she's been
specializing in, uh, she's done VMware.
Uh, she did.
Now she's, she's working, uh, Starting to specialize in security and ransomware.
So we're, uh, and she's the author of the vmiss.net blog, and we are
excited to have her on the podcast.
Again, Melissa Palmer, aka @vmiss.
How's it going?
you for
Melissa Palmer: having me back.
It's going good.
Prasanna Malaiyandi: I was surprised that you were like, Ooh, I'll
come back on the podcast after
Melissa Palmer: yeah, that was, of course, when I come back
Prasanna Malaiyandi: Well, thank you for
Melissa Palmer: scare.
It takes a lot more.
You said it.
I've been in around this industry for a while.
It takes a lot more than that to scare me away after all these years.
Prasanna Malaiyandi: And Curtis, I think, uh, now might be a good time
to put out our normal disclaimer.
W. Curtis Preston: Yeah, prasanna and I work for different companies.
Uh, he works for Zoom.
I work for Druva.
This is not a podcast of either company and the opinions that you hear are ours.
Also, be sure to rate us at, uh, Uh, rate this podcast.com/restore
and, um, if you wanna join the conversation, reach out to me.
By the way, I, I gotta give a bunch of ways cuz I, I got some
complaints and people say, well, I don't use Twitter anymore.
So how you give your Twitter address.
So my LinkedIn is, you know, linkedin.com/ally/mr.
Backup.
Uh, you can find me there.
Uh, you can find me on Facebook.
I'm on Facebook, Facebook Messenger, but my email is, uh, w Curtis Preston.
Uh, my Facebook is w Curtis Preston.
I'm pretty easy to find if you're looking for me.
Um, and reach out to me and we'll get you in on the, on the conversation.
Yeah.
Um, the, um, this, this thing of responding to a ransomware attack,
this, this is something I've been spending a lot of time on lately, uh,
because I've been, I'm, I'm working on writing my next book, which will be
about responding to ransomware attacks.
You know, one of the things that you said in the pre-call was that if, if
the first time you're thinking about responding to a ransomware attack is
after you got a ransomware attack,
Melissa Palmer: Um,
W. Curtis Preston: it's not so good.
Right.
, there's a lot of, yeah.
In fact, when I was looking at the, sort of the outline that I've been
working on for the book, most of the outline is the first half , right?
Everything that you need to do before, right.
Um,
Melissa Palmer: that's, it's like you can't just talk about ransomware
recovery, Right, Like, it, it, it's a hard topic to talk about because
you're like, there's all this other stuff that if you haven't done it, guess what?
You are not gonna be able to recover.
So we can't just talk about recovering.
It doesn't work that way.
W. Curtis Preston: Right.
It's sort of like I, I've made the joke, uh, a few times probably on
the pod where I've said, listen, you know, I've been in the backup
industry, you know, a long time.
I, I've decided to give up backups and I'm just gonna skip straight to restores.
Right?
You can't really , you can't really do that.
Just like I've also said that if I'd have known how great grandkids were,
I would've just gone straight to them.
Um, but not, not really
Prasanna Malaiyandi: It's not how it works.
Yeah.
W. Curtis Preston: Yeah.
Melissa Palmer: It is a really
good analogy though.
It really
W. Curtis Preston: Yeah, it is, it is.
By the way, you want a little, little sad thing.
So my granddaughter and her mother and, and her husband,
uh, are, this is their last day
Prasanna Malaiyandi: Oh, I was gonna ask you about
W. Curtis Preston: been living here for a while, and they're moving out tomorrow.
So,
Prasanna Malaiyandi: Hmm.
W. Curtis Preston: little sad moment.
Little sad moment.
Prasanna Malaiyandi: No.
W. Curtis Preston: Um, but, uh, anyway, so, you know, sorry to bring that down.
So let's talk about what, what do you think, Melissa?
Let, let's sort of go through those things that we really needed to have done before.
Melissa Palmer: Uh, well, lemme, lemme try to set the stage a little bit.
Like, does everybody remember like, the disaster recovery tests, like
back in the day, you go to the colo, you got the checkbook, the.
the
Clipboard you make, the checkbox isn't like, I don't know, you play
doom for a while and eat some food.
Someone restores a server and it's like,
well, it kind of worked and we're good.
Yeah,
that's how old I am.
Um, so and then you're like, oh, it kind of worked.
So we passed our d r test, but we can't actually recover.
Right?
So what you need to do is actually do a ransomware recovery test where
you actually recover everything.
There's a novel concept, and when you do that, you're gonna figure out all the.
but you didn't do cuz it's not gonna work or something's not gonna
whatever.
But it, it's, you know, talking from the backup lens cuz I was
at Veeam for quite some time.
Um, something I talked a lot about with Veeam customers was, you know, trying to
understand the whole recovery process.
Cuz if I'm the backup admin and we get ransomware, I don't just
go start restoring stuff all over.
Like that's not what happens.
It's not like, oh no, right somewhere tech, let me start restoring servers.
We'll
be back online in 20
minutes.
Like it doesn't work that way.
, you have to figure out what happened.
Before you can start restoring, you have to figure out what happened.
You have to figure out if the threat actors are still around.
You have to understand what was impacted.
I have heard a lot of people say, um, oh, well, we treat ransomware
different and we just recover in place.
So we're good to go.
And I'll go
back to the little VMware.
Yeah, I'll go back to the VMware ransomware thing.
Well, if your VMware environment is ransomware, guess what?
You're not recovering in place cuz there's nowhere to recover to.
Uh, so it's understanding all those different things.
You need to have some kind of understanding of what happened
before you can recover.
And that is generally driven by the incident response process, which is
gonna be driven by the security team.
So again, if you haven't talked to the security team before,
ransomware has attacked you.
You're gonna have a bad time.
Prasanna Malaiyandi: Or vice versa, if the security team hasn't talked to you about
how backup integrates into that process.
Melissa Palmer: that's really scary.
That's really, That's really, that's really, disturbing.
Those are actually
really even, I think that's
scarier.
W. Curtis Preston: I think it's, it's a, it's a combination, right?
Well, you know, uh, yesterday, I think that was yesterday, we recorded
a, a great podcast, uh, by the way, with Tom from Gestalt, um, that,
that, uh, net, uh @networkingnerd.
Yeah.
and he, uh, we were talking a lot about the networking side of the, the
response, right?
Shutting down things.
Um, and, and using a combination of technologies, many of which are easier
to use if you, if you set them up front.
Right.
And, uh, talking about things like VLANs and, uh, you know, like one of
the things we talked about was having a VLAN for all of your desktops and
laptops, so that if you want to stop everybody from doing anything, you
just shut off those VLANs and boom.
Um, there, you know, instead of having to notify 5,000 users, hey, stop doing
anything, you just shut off their network.
So they can't, they can't do anything.
And then if stuff is still happening, , um, well, it's not the users,
right?
It's, it's malware,
right?
Prasanna Malaiyandi: back to segmentation.
W. Curtis Preston: know, yeah, the, the network segmentation and the, the
security part, I think, um, What, what, what role do you think the, I'll ask you
what you think before I say what I think
So what role do you think cyber insurance companies and then the, the companies
that they can put you in touch with?
The, the
Melissa Palmer: Cyber insurance is becoming more and more interesting
cuz it gets to the point where they hand you the list of things you
need to do before they'll issue your policy and guess what you're
gonna probably be able to cover anyway.
Um, but a big part of, I've seen in a lot of policies lately is
having, um, basically an instant response from on retainer ready to go
as part of your policy.
And
I think that is invaluable.
I.
, everybody should have some kinda relationship with an IR firm
if you can't do it in house.
And uh, even if you can, right?
Sometimes you do still need that outside perspective.
I know a lot of larger orgs are like, no, no, we do our own ir, well, you do
your own ir, but you're not dealing with ransomware every day and these people are
so you might want a little bit of help.
W. Curtis Preston: Yeah.
Yeah.
Um, you know, um, I hate to do it, but a another, another movie reference.
I just saw the , the movie plane, and you know, the plane goes down in the
middle of nowhere and they brought in the guy, they brought in the incident
response guy basically once he showed up.
Right.
See, there's a movie reference for everything,
Melissa Palmer: I haven't, I
can't tell you the last movie I've watched.
I really can't.
I don't
W. Curtis Preston: I can, I can, I can pull up my app, uh,
cuz I have the Regal Unlimited.
Melissa Palmer: tell you the last thing I watched.
I can't tell you the last movie I watched, cuz I don't remember.
W. Curtis Preston: I, I, yeah, I, I saw like three this week.
So
in, in the theaters
Prasanna Malaiyandi: so back to the cyber insurance from movies.
Uh,
I, yes.
Yeah.
No, but, but, but I think, well, this is one of the points that I remember
because remember when Tony came on from SPECT Logic, Curtis, and he was like,
oh my God, they got hit with ransomware.
And he's like, just the previous month they had signed up for cyber insurance.
They had an IR firm come in, give them sort of the list of, Hey, here's
everything you need to do to help.
Right.
And he was like, that was probably the most valuable thing of that sort of
cyber insurance policy was having the experts who could walk you through.
W. Curtis Preston: And it, and it wasn't even like he, he was just
lucky enough to have already, you know, contracted with them.
Right.
But the best I think would be to
, well, not that you would know this, but to do it not a month in advance, but
obviously way in
Melissa Palmer: right.
W. Curtis Preston: to get, and to give you some time to work with the incident
response team and to make sure that you are doing the things that they want
Melissa Palmer: but that's like that's like the problem, right?
Like it's not, if it's when, and you don't know when.
It could be tomorrow, it could be next week, it could be next month.
It could be next year.
Like you don't
W. Curtis Preston: It could have been three weeks ago.
Melissa Palmer: and you just haven't realized it yet, right?
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: Do it today.
Yeah.
Melissa Palmer: my favorite.
W. Curtis Preston: Yeah.
Uh, so, which is why it doesn't matter when you invent a time machine.
Melissa Palmer: You know, I have bad news to you.
W. Curtis Preston: What
Melissa Palmer: I haven't invented a time machine
because there are certain
points I've always promised to myself.
If I invented the time machine, I would go back to this point and tell
myself I invented the time machine.
And if that hasn't happened, I haven't invented it because
time is not linear, right?
So I haven't invented a time machine.
I'm very upset about that.
W. Curtis Preston: Yeah.
Me neither.
Um, but, um, well, it's been a weird, it's been, we've been jumping in and out
of the topic here on this podcast, but,
Prasanna Malaiyandi: Incident response.
W. Curtis Preston: yeah.
So we, we, we get the cyber insurance folks because I
think in the, in the initial.
Ransomware phase, what people thought of cyber insurance was just a
company to pay their ransom for you, and that they're definitely saying
they're not interested in it anymore.
Melissa Palmer: Yeah.
And there's
more costs beyond the ransom, right?
So
you paid the ransom, but what about everything else?
Um, that's the thing.
And policies have changed over time, like, back in the day a couple years ago, right?
Like before the pandemic, uh, it was like easy to get cyber insurance.
Like, oh yeah, I'll take a cyber insurance policy for 5 million, please, whatever.
And
now it's hard.
And if you do actually use your, I've seen a lot of cases where if you
actually use the insurance policy,
guess what?
They don't necessarily drop you, but guess what Your deductible co becomes.
What they paid for your last ransomware attack, right?
So if I had to pay 2.5 million, guess what?
I now have a 2.5 million deductible for my next attack because
let's face it.
We get IR in, right?
We figured out what happened, we have to recovered, and then there's a whole
stage where we have to do a postmortem, figure out how they got in, if they're
still in and close up the gaps.
That doesn't always happen cuz people are so, like, ohms are back, we're good to go.
Happy day, happy
day.
And they get hit again
because they never fixed the way they got in in the first place.
W. Curtis Preston: What, what do you think about the idea of.
And again, this would be driven by management.
And you know, a lot of times, like you said, management isn't necessarily
at that moment thinking about the the best way to do something.
They just wanna do the fastest way to do something.
right?
So another thing I've been looking into is the idea of wouldn't the best
practice to be to figure out how they got in before you do the recovery,
before you turn everything back on.
Melissa Palmer: Yeah.
And that, that's where the IR firms come in, because.
they'll kind of get in and they'll be able to do that.
They'll be able to say like, you guys are so messed up.
You didn't have any logging unabled anywhere.
Like we, we can't tell right now.
Right?
It really depends on what happens in that first phase.
Um,
W. Curtis Preston: Yeah.
Melissa Palmer: and it comes back to kind of getting ready for the
attack and what kind of security practice you have in some places.
Yeah.
We could see, people can figure out, uh, throw in a tool and say, yeah, guess what?
They came in here.
We know we're good to go.
Other times they might not find it just
because there was never.
Prasanna Malaiyandi: they came in.
They went out before you even knew
or nothing was
W. Curtis Preston: under
Melissa Palmer: or we didn't, you know, we didn't have logging
on or whatever.
Or they turned something off or,
W. Curtis Preston: Logging is a beautiful thing and, and also
a system to get those logs off
Melissa Palmer: yeah, that's what
people like
forget about, like who cares about the logs, like whatever their logs.
No, you're, you're going to care about the
logs someday, I promise you.
W. Curtis Preston: Yeah, I mean, even if it's something as simple of making
sure that the logs are represented as text somewhere, that is then
backed up by the backup system so that you can restore all of them.
That's basic, but there are systems that you can buy that
will just automatically, uh,
exfiltrate all of those logs for you.
Yeah.
Prasanna Malaiyandi: I wanna go back to a point you made earlier, Melissa, about
sort of, okay, how do you make sure that you fix the things that broke so everyone
isn't like, Hey, my VMs are back up.
I don't need to worry about these things anymore.
Have you heard any cases where, I know sometimes executives have
sort of financial liability, right?
Melissa Palmer: I've heard of that trend, right?
Like your guess what your bonus is tied to if you get ransomware or not, and how you.
And stuff like that, that's starting to happen in some places.
Um, but a lot of it comes down to maybe the processes were
never clearly defined upfront.
Right.
And that's where a lot of the cyber insurance stuff can
actually come in and help.
Well, they'll be like, you need to show us your response process.
And they'll be like, here you go.
And they'll be like, okay, so where's the rest of it?
Or something like that, right?
Like, what, what
happened?
Like, this is it.
Like here's
a page.
Like it's not gonna work.
Um, and again, it comes back to.
the old school DR test.
Like there needs to be ransomware recovery tests and postmortems of
that ransomware recovery test, right?
Like y'all need to get in room, figure out what worked, what didn't
work.
W. Curtis Preston: Having done the old school DR test, I'm curious as to how
they do a ransomware recovery test.
Because one of the hardest parts of a ransomware recovery is that the
attacker is there is still attacking, like with a dr, you just say,
okay, those six systems are dead.
Melissa Palmer: So, yeah.
So
here's where it
gets complicated.
You need to test multiple types of recoveries, right?
So maybe I'm recovering, please.
I, I can't.
, I will vomit in my mouth if I say maybe I'm recovering in place.
I can't even like say that.
So we're not gonna say that, but like maybe I'm going to my second site.
Maybe I'm going to a warm site.
Maybe I'm going to a hot site.
Maybe I'm going to a public cloud.
Maybe I'm going to a VMware cloud.
You gotta test all those, right?
Because
you don't know where you're going until that incident response
phase starts, especially when law enforcement gets involved, right?
So let's say stuff's really bad, the FBI comes, and guess what?
We are quarantining your whole data center while we investigate.
Then what do you do?
Prasanna Malaiyandi: Yeah.
You're down for business, otherwise,
Melissa Palmer: do?
No, you go to public cloud, you go to um, a service provider, you go someplace else.
So you have to have all that ironed out ahead of time.
You have to know that there's different considerations for recovery from
ransomware attack than a traditional
disaster.
So I guess, you know, from a traditional disaster, like what if
the zombies eat both data centers,
right?
Then you would still need to go to the
Prasanna Malaiyandi: but people probably aren't thinking about that though, right?
The fact that, hey, maybe the F B I will come quarantine, right?
Do you have your backups offsite?
Do you have it in someplace that you can bring it up?
And like you mentioned earlier, Melissa, it's like things you should plan for ahead
of time before you get to the point where you are trying to recover from ransomware.
Melissa Palmer: Exactly.
And again, unless an organization, so I have a couple of examples
of, I don't wanna say Dr.
Done wrong, but uh, I worked for an uh, company when I was
an intern on Wall Street and everything was in New York City.
and nine 11 happened and they were a block from the World Trade Center.
That's what they couldn't, they couldn't do anything like they were done.
Right.
Like they were just done.
So they like rebuilt their systems in a hotel room someplace.
Right.
And that kicked off a huge project to say, we actually need a second data
center and it needs to be not around here.
Right.
Um, I'm also on the east coast, right?
So New York, hurricane Sandy, we had this hurricane roll through.
And again, like the data centers are like 20 miles from each other.
Guess.
, they both tanked.
Um, so things like that.
So until an organization actually has something happen to them, it's really,
and here's the issue, the, the, the difference between disaster recovery
and ransomware recovery, when we talk about it, traditional disaster
recovery stuff, until it happens, it's easy to accept the risk, right?
Well, you know what?
It's cheaper for us to just like recover from this disaster and be down for
two weeks than it is to actually put everything into place where we build a
second site, yada, yada, yada, yada, et.
that's because the risk is so low, right?
And there's all kinds of equations for
this in, you know, cybersecurity and stuff like that.
But when you change it to ransomware, the risk is going to, it's going to
happen like a probability of one.
It
will happen.
Um, and that's what people don't understand.
Like this is going to happen.
It's not like you can say like, well, you know, we haven't had a hundred
years storm ever, so we'll be fine.
Um, it's different like that.
And a lot of people, I've actually seen a huge uptick in people getting.
I don't think a lot of people are where they need to be.
Um, but I think as people get ready and it gets harder and harder to attack
people because they've put like some semblance of security in it, right?
You're gonna go for the low-hanging fruit, you're gonna see the people
who aren't ready get hit harder and you're just gonna see more and more
attacks and the threat actors are gonna have to get more creative.
Prasanna Malaiyandi: So here's a question for you.
Normally when we think about backup and recovery, right, it's always
about restoring your data or your application because there might be
a hardware failure, an application fault, user error, et cetera.
Sometimes people talk about ransomware in the same context as
disaster recovery and sort of those
Melissa Palmer: Ransomware is a disaster.
I
Prasanna Malaiyandi: but, but here's the question though, Melissa
is, Like you had just mentioned, it's not the same as a flood or a
hurricane or something like that.
And so are we kind of pushing ourselves and kind of giving people
the false impression that it is similar to those other disasters
and things that they shouldn't worry about versus we should be treating
it similar to like an application failure or user failure and treating it
similar.
It's like more towards that side of the spectrum than this side.
Melissa Palmer: and you know, that all falls under DR
anyway, like hardware failure
and all that kind of stuff.
Um, and again, in a lot of those cases, it's easy to say, well, you know what?
I don't really want a second site.
It's
just cheaper to deal with the hardware.
It'll take we'll rush order.
I was in a situation at a company, we'll just rush order at a new array from
E M C that will solve our problems.
Like that was the plan and that happened.
Um, so crazy stuff like that.
But the problem, why I like to make the analogy so much is the problem
is when you tell someone that you have to get ready to recover from
ransomware, they're just like, I don't.
what to do.
You have to put it in some context that kind of makes sense.
I mean, disaster recovery is definitely like not sexy, even though
I've done it most in my career.
Um, but it's something that everybody has an inkling about at least, right?
Everybody kind of knows that there is usually a DR test once or twice or year a
minimum.
Um, so it's a way, it's a starting
point, right?
It's not your final destination, but it's a starting
point.
It's a.
place to start context.
Maybe you have some playbook, some processes that we can leverage to go build
on top of that and say, okay, so how do we make sure that we can recover now under
any
W. Curtis Preston: I like to, I like to say that it's a subset, right?
A DR is a subset of a ransomware recovery, but there's so much else, right?
And the big thing, the but, and I think you said it already, Prasanna, but the
big thing to me, the difference between a DR and a ransomware attack, um, is
that the, the disaster isn't, Right.
You're, you're still right
Melissa Palmer: that the disaster never
W. Curtis Preston: a flood is gone, you're like, okay, all
these servers got wiped out.
So those are the
Melissa Palmer: because the threat is still there.
Just because you
recovered from the ransomware attacked doesn't mean they're not
gonna hit you again, or someone else
isn't gonna hit
W. Curtis Preston: Right.
Well, and, and how do you even know,
um, You know, like when you, when when a hurricane wipes out a data
center, you're like, okay, those are the servers we need to restore.
But how do, when you walk into your data center and there's a
ransomware attack going on, how do you even know which servers have
been affected or not affected?
Right.
That's, that is a big part of it.
Prasanna Malaiyandi: Yeah, and I guess the other thing is even like you
might see the active infection, like things are being encrypted, et cetera,
but it might just be lying silently.
Right.
We've talked about dwell time in the past, right.
Where it's
Melissa Palmer: chill.
They just chill in there for a while.
Like, who knows?
Um, I, I can't remember off the top of my head, but I remember reading like a big
name breach or something like that, or a big name attack, and they said they were
in the network for like six months or
Prasanna Malaiyandi: I think Solar Winds was like
Melissa Palmer: was it?
I don't remember.
But I remember reading a couple of them where they've been in
there a significant period of time and who knows what they're doing
there, right?
Like who knows
unless you catch them.
So it's about
W. Curtis Preston: yeah.
The meantime is something like 60 days actually is what I, what I read.
Um,
Melissa Palmer: be the worst ransomware person.
I'd be like, let's go, let's go.
It's like, no, you're not supposed to do that.
You gotta
take your time and traverse
through the network and get ad.
I'd be like, let's go encrypt VMware.
Let's go.
I'd be caught so fast.
Or maybe I wouldn't, maybe I.
Prasanna Malaiyandi: You're only caught if someone's monitoring and watching.
Right Melissa?
Melissa Palmer: Right.
And you need
to be looking for the right things.
W. Curtis Preston: Yeah.
As soon as you encrypt a, a vm, uh, you're gonna set off alarm or two.
Um, but I, I think you encrypt, I think you encrypt a lot of
files that no one's looking at.
Right.
But the moment you start
Melissa Palmer: Once you hit the the thing,
the only thing is you'll hit.
You'll hopefully you'll be caught as soon as you start encrypting the VMs.
You do them all at once, so it doesn't matter.
W. Curtis Preston: Yeah.
Right.
Cuz it's,
Melissa Palmer: I got all of 'em.
It doesn't matter that you caught me doing the first one, I did them all.
Um, but yeah, so generally they're in their wreaking havoc, steal maybe
exfiltrating data, doing some stuff before they go encryption habit.
Or maybe like, I've heard cases recently where they don't even
bother, like encrypting stuff.
They're just stealing data at this point and
be like, by the way, look what we have.
Prasanna Malaiyandi: Is that easier by the way, to steal data?
Because it seems that you can sort of fly under the radar if you just steal
data because people will probably, maybe they notice, maybe they don't,
but it's not as obvious as, say,
Melissa Palmer: It
is definitely not as obvious as encrypting stuff, I'm like
this weird monitoring nerd too.
I had like this monitoring fetish at Veeam.
It was very strange.
Um, so like, I would like really hone in on like what to look
for to catch that too, right?
But not everybody is crazy like me.
Um,
W. Curtis Preston: I think, I think,
yeah, I do.
To answer your question, Prasanna, I do think that exfiltration as an overall
process is easier in that if you can get any data out that there's a, there's a
much higher chance that they will respond.
That they will pay the ransom.
Right?
Because backups aren't gonna help.
Melissa Palmer: I'm looking at my black hat over there.
I'm wondering if I should like, put it on for this discussion or something.
Um,
like you would probably like see like, all right, like if I'm a bad person,
I'm not a bad person, I'm a good person.
Um, like they start small,
right?
They grab a file here and there and they see if they
Prasanna Malaiyandi: if anyone notices.
Melissa Palmer: this, grab that, right?
Like, you don't go and just be like, oh look, here's the final.
25 million gigabytes of MP3s.
I'm gonna take it all at once.
No, they're like picky and choosy.
They try to find the sensitive data.
They take a little bit here and there.
Maybe they only need to grab a couple spreadsheets.
Right?
It's not like,
I think there's this misnomer that like they get in there and I'm just gonna
start downloading massive chunks of
data.
W. Curtis Preston: well,
that's the whole point of
Melissa Palmer: so you could exfiltrate a vm, just like
download the vmd K and be like,
W. Curtis Preston: yeah, exactly.
Melissa Palmer: ad.
Have a
nice life
W. Curtis Preston: that's that whole phase of the, um, the initial phase of an attack
is trying to expand out, seeing what you can find out, seeing if you can find
a spreadsheet called customer database
Melissa Palmer: You know?
Right.
W. Curtis Preston: xls , right.
Um,
or like.
Melissa Palmer: you might not bother encrypting everything, but if you
can't find much, you say, all right, I'll steal some stuff and tell 'em I
have some files, but I won't tell them what I'll hope that'll make them pay.
And I'll just
go, you know, encrypt some stuff while.
Which is more illegal?
Is one more legal than the other?
Prasanna Malaiyandi: I think they both are pretty bad,
Melissa Palmer: is one more illegal than the other?
W. Curtis Preston: Well, they're both extortion.
Melissa Palmer: Yeah,
W. Curtis Preston: the act,
The act
Melissa Palmer: but if you're actually exfiltrating, you're stealing it.
W. Curtis Preston: yeah.
That's gonna depend on where this happens.
Uh, whether or not exfiltrating the data is a different crime.
And damaging the data.
Um, but, uh, but in the, the extortion happens on both sides, right?
And that's
definitely illegal in
Melissa Palmer: that
W. Curtis Preston: pretty much
every jurisdiction
Melissa Palmer: legal kids.
Prasanna Malaiyandi: Yeah, so we talked about, so we talked
about incident response.
You've now been hit by a ransomware attack.
in, then let's just take VMware environments, right?
So what do you see people doing like, or what are things that they
should be doing that they're not?
Like, how do they even approach
Melissa Palmer: Yeah, so he,
Prasanna Malaiyandi: VMware environment gets encrypted Now, what
Melissa Palmer: Um, to me it's trash.
I would throw it away and start over, like, I'm not even joking.
Throw it
W. Curtis Preston: No, not
Prasanna Malaiyandi: and, and, and, and how much?
And and how much would you, when you say throw it away, are you talking about
throwing away the virtual machines, throwing away the ESXi servers, the.
Melissa Palmer: the host, wipe the storage array, wipe it all and start over.
Um, and, and here's the thing, right?
So like, you know, I, I like it.
I have this weird side of me that also does like weird blogging stuff, right?
And like, I like SEO and stuff like that.
And even my career at Veeam people are like, how do I back up my VMware host?
you don't, they're like, what do you mean?
I'm like, you don't, um, you automate the build process
and the configuration, right?
You don't actually back up your host and restore it.
It's, you
Prasanna Malaiyandi: You just rebuild
Melissa Palmer: thing.
It's a clean install and you configure it.
Um, so that's what people need to be testing to is how I would
actually recover is almost misnomer.
Cuz Prasannally I would trash it.
Um, how do I re rapidly rebuild a VMware environment?
And that's something.
People don't do every day, right?
Like that stuff runs like you might have not even reinstalled.
You could have just been
upgrading for the last like 10 years and like, whatever, probably not 10, probably
four or five years, you'll get a new host.
I don't know.
It depends.
Um, so that's something that people don't practice and don't do.
Um, and you can actually do that all.
for the most part, um, in a nested virtualization environment.
Get all your processes down stuff.
So it's a pretty low co I mean, you should test on your physical hardware
at some point for any drivers and stuff, but it's actually a relatively low
cost and effort thing to figure out.
It's not rocket science.
Prasanna Malaiyandi: But when you do this testing, wouldn't you also want to
involve, say like your networking team,
Melissa Palmer: Yes, you would wanna, any of
this testing, you wanna involve anybody?
Everybody, right?
Everybody should be involved in this.
everybody.
And that's I think, one of the biggest problems we see that they're not,
W. Curtis Preston: So when you say,
Melissa Palmer: They're like, I don't have time to do this.
W. Curtis Preston: when you say rebuild the VMware environment,
um, obviously you're talking about vm, you know, wiping the hosts and,
and the storage and all of that.
When we get to the phase of actually bringing back VMs,
Melissa Palmer: Mm-hmm.
W. Curtis Preston: what way would you do that?
Melissa Palmer: Um, so most backup software these days have something
built in where it'll actually scan for ransomware as you are restoring, right?
And find the ransomware if it's there.
Cause at that point, you know what you're infected with,
so you know what to look for.
Um, so I would be either scanning it or, you know, if you have really good.
and then you can decide how you're gonna fix it, or you're just gonna go
back to an earlier point or whatever.
Um, you know, some people are really good with the IR stuff and say, we know the
ransomware came in this date, this time we are absolutely a million percent certain
because we have all these logs go back to the last known good restore point, right?
Um, so it really depends.
But the backup people gonna be a big part of that, right?
Because it's gonna be
W. Curtis Preston: Y Yeah, I,
Melissa Palmer: do they have built in?
W. Curtis Preston: this is something I put a lot of thought into lately
of if the meantime of a, of a.
Infection is 60 days, and some of them are twice that,
um, the, the idea of of saying, oh, well we got, we got infected December 1st,
so we're gonna restore to December 1st.
That's a
Melissa Palmer: That doesn't, it doesn't always work.
In some cases it might, in some cases it won't.
And then you're going
back to scanning,
W. Curtis Preston: So you've got, you've got to, I think in most
cases, if many, if not most cases, you're gonna do a restoring.
Melissa Palmer: Yeah.
I've seen kind of almost like two stage recoveries too.
Like get the bare minimum of stuff something up and run something
online up and running, right.
To restore services and then do the full recovery later.
So you're not, you might be like, all right, so you know what?
We can roll these servers back to December 29th.
We can use the newest copy of the database.
We can mash it together and make it work and serve our customers
while we're actually restoring everything the right way.
Prasanna Malaiyandi: Rackspace,
Melissa Palmer: So it did that.
W. Curtis Preston: Prasanna.
Yeah.
Melissa Palmer: you okay?
You were eating another sip of tea there.
W. Curtis Preston: It's what I thought of when you, when you, as soon as
she said that, I, yeah, I know.
Yeah.
Just make sure.
Unlike Rackspace, just make sure that you thought of this beforehand.
Right.
The only way that this is gonna work is if you identify what are the three
services that need to be up right away so that we can function as a company and
what are the other 20, 5,000 services
Melissa Palmer: That kind of, um, that ties almost more into like
the business con, you know, B C D R
W. Curtis Preston: Yeah.
Yeah,
Melissa Palmer: continuity sort.
Like what are our key applications and what level of, what do we have
to do to get those online First comes back to our RPOs and RTOs, right?
W. Curtis Preston: yeah,
Melissa Palmer: it's, it's,
the thing is, it's, such a
big discussion that unless you've had it cross-functionally with the
business owners and the app owners, and the infrastructure owners and the
security team, you're not in a good.
W. Curtis Preston: Yeah.
I, I think, I think it's, it's just, it's one thing to have a discussion,
again, going to Dr versus rr, um, is that it's one thing to go, well, what
are the servers we're gonna do first?
And what are, what are the servers that we're gonna do three hours later?
It's a whole other thing to say, what are the servers we're gonna do the
first couple of days, and what are the servers we're gonna do next week?
Right.
I,
Melissa Palmer: And that, that's the problem, right?
You don't know until it happens.
Like if,
if you, if it's your whole environment is done right.
That is very different than, oh, we know, just, they just did this
subset of servers or whatever.
It's,
and like we were, um, The company I worked for a company
that I no longer worked there.
It was a pr uh, I was a customer and they had a, a very, they were one of the first
really, really big ransomware attacks in the news, and it was like a disaster.
I was like, wow, I'm glad I'm not on the VMware team anymore
there when this is going down.
Right.
Um, , but it really depends and you don't know what's gonna happen.
The only thing you can do is be as prepared as possible, right?
Test different recovery methods.
Um, and I love RPOs and RTOs in saying that we can meet them under a testing
scenario, but in the real world, we don't know that that's gonna happen.
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: One of the things on the podcast we talked about a couple
days ago was, Like Tom was mentioning, oh yeah, you just shut down your
network and you start figuring out, okay, what was affected but in what?
And you prevent everything go from going in and out.
And I was like, but how do you communicate?
Right?
And he's like, yeah, make sure you have ahead of time, sort of use cell phones.
iMessage can work.
You can set up a separate Slack instance completely outside of
the corporate environment, right?
Whatever it is to keep that ongoing communications.
Melissa Palmer: like, uh, how am I supposed to use Microsoft Teams to
communicate with a security team?
Well, that might be Office 365.
That might be, okay, that's a bad example.
W. Curtis Preston: Yeah, as long as you have a, as long as you have a,
um, an internet connection, right?
Um, which is pretty easy
to get
Melissa Palmer: but
like who has people's
phone numbers these days?
W. Curtis Preston: people with incident response plans, that's who
Melissa Palmer: yeah, that's
Prasanna Malaiyandi: But But aren't there issues though, where ransomware
actors might still have access to your Slack instance and be monitoring
what's going on from an incident
Melissa Palmer: I've
seen that.
I've
seen that.
I've seen, I have seen that happen where like, they still had access.
It was teams.
I think
they still had access.
They were watching the IR
stuff happen as they were still in there hanging out.
It's like, oh yeah, Y again,
W. Curtis Preston: ransomware stuff is bad.
Melissa, I'm just gonna take that stance.
Melissa Palmer: bad.
It's bad, and you don't know what's gonna happen until it happens.
Which is why, and it ties back to incident response, right?
And having an incident response firm on retainer that does this every day.
Right?
Because I, I don't care how good, even if, like, okay, let's say
you drop Melissa into X, Y, Z company and you put her in charge.
W. Curtis Preston: Do are you gonna repel down a rope from a helicopter?
Because that
Melissa Palmer: Yes, I'm gonna repel down a rope from a helicopter,
drop me in, right, and say, Melissa, get ready for ransomware,
and six months later you hit me.
I would like to say that I'll be able to recover, but I don't know that.
I don't know.
That doesn't matter how good you are, you're not doing this every
day, right?
Like, so unless you're doing this every day, cuz every attack is different.
It's gonna be like, what have these people seen in the other events?
What, what ransomware gang have you been hit by?
Right?
So I can put everything into place that I think I will need
to make sure that we recover.
And yeah, honestly, we'd probably recover all our data.
I don't know if we meet our RPOs and our tails.
I, I, I'm pretty sure I could get all the data to the recoverable point,
but what was Exfiltrated, how did they get in all that kind of stuff.
you don't know, which is why you have to call the pros.
You have to call the people that do this every day.
Prasanna Malaiyandi: Is there sort of a standard ransomware recovery test, but.
That kind of outlines like, Hey, here are the thing.
Because I can imagine, say you can't afford, the pros
say you can't afford the pros.
Right?
Is there sort of a, here are the testing scenarios you should be thinking
about, or here are the things that sort of get shot in the head when a
ransomware recovery or ransomware hits.
Melissa Palmer: Um, Google tabletop exercises like ransomware
recovery, disaster recovery,
tabletop exercises.
Right?
That's a good place to start.
I've thought about doing like a dungeon and dragon style type,
like ransomware recovery thing.
I
Prasanna Malaiyandi: With the actual people.
Yeah, with like you get the networking security
Melissa Palmer: think that would be
fun and useful.
And you know what?
When you make things fun, people actually pay a.
Prasanna Malaiyandi: Yep.
Melissa Palmer: right?
So like, if I get you all in terms and be like, today we are going to talk
about ransomware recovery and have a mock simulation of what would happen.
Be like, okay, you're a Paladin, you're a warrior, uh, you're a ma.
Uh, an adult black dragon just showed up and encrypted your VMs.
What are you doing?
Right?
Like,
you're gonna have so much fun,
you're gonna remember it, and it's gonna work out a lot better.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: I like that.
Yeah.
Um, by the way, one of the things, you know, we talked a lot about prepping.
One of the things that I think also in terms of, we talked
about exfiltration monitoring.
I also, uh, like the idea, and we talked about it on a couple of
different episodes, this idea of, um, Something on your d n s side
that would notice when you start talking to really weird domain names.
Melissa Palmer: Yeah, that's a
big one.
And there's all these lists.
Um, a lot of these researchers will just like tweet like, by the way, domains
looking a little hot, a little suss.
You might wanna block that stuff.
Um, so yeah,
there's
these lists of these like known bad domains and ips and stuff like that too.
W. Curtis Preston: Right.
Yeah.
And, and the other, uh, but I, I do think that if.
If you implement exfiltration monitoring, if you have a specific exfiltration
monitoring, I think you could stop mo or, or notice it quickly and stop it.
Um, but what I'm hearing from others is that not everybody
can afford such a thing.
Right.
Um, that, that,
Melissa Palmer: lot of people can't afford it or they don't
have the skill set to build it
themselves, and you
really wanna be building and maintaining your own security systems.
Probably not.
W. Curtis Preston: No, but a lot of people do,
Melissa Palmer: Yeah, because they have no choice.
It's better than nothing.
Like I've done
some weird stuff with some weird software because it was better than nothing.
Um, it, it, it's really a difficult point to be in.
And it's kind of like, you know, you all these people put out these, um, all
these, uh, security companies will do all this research of like, here's the
top ways they're getting in and blah, blah, blah, and all this kind of stuff.
Um, there's a lot of marketing that goes into it, but
there's a lot of truth, right?
So like, I.
. The big thing was the people for a long time, the people
let it in, you know, multi.
Where was it when, when this whole Cisco thing happened?
That was like, um, mfa, right?
They
got in through their mfa cuz they kept spamming of them.
Eventually they said
yes because like, stop calling me at 11
o'clock at night.
Um, . Now they're saying, oh, it's more vulnerabilities than people, right?
So honestly, I feel like the people might be easier to deal
with in the vulnerabilities.
I don't know.
Um, because then it's gonna be like testing the patches.
Can we patch everything?
Can we remediate everything?
It's, it's just like, what are the areas that you can find within your
own organization to be quick wins because you wanna prove that you can
win to your management so you get more money and can do more projects.
So you
need like a balance of quick wins to prove progress and high.
right?
What are the things that I can implement that will have the
most impact to reduce the risk?
And you're never gonna get the risk to zero.
I, there's um, a lot of people say that, like assume breach, right?
Like assume they're gonna get in so we
can do all this security stuff.
We can do all this backup.
And backup is basically assuming they're gonna get in, right?
Like, we're
not backing this stuff up cuz we think our security is so great.
Like we're assuming that it's the last line of defense, we're gonna need it.
Um, so a lot of it is just trying to mitigate what you.
in a way that makes sense for your organization, because we can't
have everybody working 20 hour days doing this either, or they're
gonna be too fried to make mistakes
and people are a problem.
Um, it, it's difficult.
It really is hard for any organization.
It's what can I do with what resources I have and cya, right?
If I'm, I'd probably be doing a lot of cya when, you know, they tell you
it's too expensive, you can't do that.
Well, you better have that documented.
So when you get ransomware, not like, Melissa, why
didn't you put in that security system?
You told me we didn't have the.
W. Curtis Preston: You don't know what's the current hot way that they're gonna,
they're, they're gonna attack you.
You can't stop all, uh, vulnerabilities.
You can't stop all stupid user things that stupid users are gonna do.
Um, and, um, And, and so you, I do think you, you have to assume breach, right?
And so you do have to do some things in your network that are going to
tell you when the bad guys are here.
Um, and that we stop it
as quickly as we can.
Melissa Palmer: Can we make a movie about this?
Please?
Like that would be
really cool.
W. Curtis Preston: Nobody.
It'll only be
Melissa Palmer: I'm gonna watch it
I'm gonna have chat, G b T, write me a movie.
I've had to write me ransomware, hallmark movies.
I kid you not, I'm just saying
have to entertain myself.
How now?
Prasanna Malaiyandi: Wait,
W. Curtis Preston: my wife would
watch it if we make it a krama, make it a Korean drama.
Um,
Melissa Palmer: be good.
Or like a Bollywood ransomware story.
W. Curtis Preston: yeah, I, there was a ransomware attack and a
krama that, uh, I dunno if you saw, there's one called Startup.
Um, and, uh, there, there's a, there's a, a really big
incubator in Korea in this movie.
Um, and this group of people, they, they do a startup there and.
Right at the crucial moment they get, they get a ransomware attack.
Um, and, and it was because some people did some dumb stuff.
They cut some corners, you know, and so they got
Prasanna Malaiyandi: They got.
W. Curtis Preston: and the tech wasn't bad.
Right.
Um, there, I, I've actually seen a lot of, there was, uh, the good
doctor, that's the one with the guy that has, he's on the spectrum anyway.
They got, they got,
they got, they got a ransomware
attack.
Melissa Palmer: Grey's
Anatomy
W. Curtis Preston: Uh, Grey's Anatomy did one.
Uh, the good doctor did one and the tech wasn't bad.
Right.
Uh, I just, I just hate it when it's like, like, when you watch, I dunno if you
ever watch, did you ever watch the Net?
Melissa Palmer: Yeah.
Yeah.
Prasanna Malaiyandi: Yep.
W. Curtis Preston: That tech
Melissa Palmer: Look, all I know is I was, I don't know, maybe there's some
Hallmark movies going on in my house and it was on in the other room when I was
cooking dinner and my ears perked up.
Cause I heard something about an engineer and it was
the dude who was the engineer.
I was like, oh, I had hopes for this one.
So Hallmark, if you are listening to this, I would love to be your female
lead in a I think that would be so much.
Come on, come on.
Happy ending.
They, we,
we recover from
W. Curtis Preston: question is, how can you incorporate a small
town with a business that's, you know, on its last legs?
And
Melissa Palmer: Totally.
Prasanna Malaiyandi: That would
work.
Yeah.
W. Curtis Preston: instead of a ran, instead of a, uh, you know, a big
bookstore coming into town to shut down your little bookstore, it's
the ransomware attack shuts down the little, the little bookstore in
Prasanna Malaiyandi: Or it could be at a doctor's
W. Curtis Preston: And,
Melissa Palmer: Yeah.
Or local hospital.
We could
do local hospital.
That would be fine.
Small town hospital
only thing for miles.
W. Curtis Preston: It's, it's the big city girl that knows, um, that knows
about ransomware to rescue the little
Melissa Palmer: big city girl, leaves her job at a software company, goes back
to her hometown to go out on her own.
just
W. Curtis Preston: Um, can you tell I've seen a Hallmark movie or show a show
Melissa Palmer: I, it's my guilty pleasure.
I'm just gonna say that, uh, around Christmas there was a thing going around.
It was like Hallmark movie generator,
and I looked at it and I went, this is my life.
Oh my goodness.
I'm a Hallmark movie.
This is so cool.
W. Curtis Preston: They are kind of predictable as storylines, but, but yet
they've yet to have a ransomware attack.
Melissa Palmer: Come on.
W. Curtis Preston: I'm behind that.
Yeah.
Well on that note, um, speaking of disappointing, um, you
know, if you folks like this
episode, I think there's
some,
I, uh, uh, I think, no, I think this was a good episode.
Um, and I like, I think, you know, we covered a lot.
We also had a little bit of fun.
I love that.
That's actually my favorite kind of episode where we, if it's just straight
talk the whole time, it's boring.
Um, and.
This was good.
Uh, good, good.
Smattering of both.
So, um, I think the one thing we're getting away from this is the best way
to respond to a ransomware attack is to respond to it before it happens.
Melissa Palmer: Yes.
W. Curtis Preston: Right.
Talk to people, talk to, you know, talk to a incident response team.
A cyber insurance company's a good way to get one of those.
Um, you know, uh, do all the, the, those, the ransomware recovery scenarios, right?
All the different scenarios from a, the, the backup and recovery standpoint, right?
Um, and, um, and do some kind of monitoring, logging, logging.
Saving your logs, getting the logs, logging log.
I can't, I can't say that.
I can't
say it that
Prasanna Malaiyandi: lugging.
W. Curtis Preston: Yeah, log, logging.
Logging, I can't, I don't know.
My tongue doesn't do that anyway.
Um, and then also some kind of monitoring for what's going on in your environment.
That would set off alarms when a ransomware.
You know, initial phase is happening.
Uh, cuz that's the key to start to stopping it, is to stop it.
Melissa Palmer: Yep.
Get it.
Prasanna Malaiyandi: Yeah,
W. Curtis Preston: absolutely.
Well, thanks Melissa
Melissa Palmer: Thank you.
W. Curtis Preston: and uh, thanks Prasanna despite the fact that you were the
cause of all of our technical problems.
Prasanna Malaiyandi: I'm sorry.
Hopefully not.
Melissa Palmer: Sounds like a Hallmark
Prasanna Malaiyandi: I
Melissa Palmer: Sounds like a
Hallmark movie, just saying
W. Curtis Preston: We'll see this.
Prasanna Malaiyandi: Thanks Curtis, and enjoy your vacation, Curtis, and
thanks Melissa for joining us again.
Melissa Palmer: my pleasure.
W. Curtis Preston: We want to say thank you to our listeners as well.
It's been a great year, 2023.
So I hope you enjoyed this repeat episode.
That was so popular earlier in the year.
And again, we just want to thank you.
That is a wrap.