You found the backup wrap up your go-to podcast for all things

backup recovery and cyber recovery.

Today I've got another popular classic episode that you probably

haven't heard, Prasanna and I talked to Melissa Palmer, AKA @vmiss, a

ransomware resiliency architect about why virtualization environments are

such juicy targets for ransomware attackers, how they're specifically

going after vCenter and ESXI hosts.

And why your backup strategy is probably missing some critical components if

you're trying to protect from that.

If you've got VMware, you can't afford to miss our episode with @vmiss.

See what I did there?

By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr. Backup,

and I've been passionate about backup and recovery for over 30 years, ever since.

I had to tell my boss that we had no backups.

Of the production database that we had just lost.

I didn't want that to happen to me again, I don't want it to happen

to you, and that's why I do this.

On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes.

This is the backup wrap up.

Welcome to the show.

Hi.

Welcome to the backup wrap up.

I'm your host, w Curtis Preston, AKA Mr. Backup, and I have with me one of

only three people who actually know and recognized my actual birthday today.

Prasanna Malaiyandi, how's it going, Prasanna?

Good.

Curtis, how are you doing?

Happy birthday.

why is my birthday so complicated?

Why do I make it

You make it complicated.

Exactly.

I do.

But why do I do that?

I do it for a reason.

Privacy.

Privacy.

Yeah.

So my, my Facebook, LinkedIn, et cetera.

Birthday was yesterday, . Um, and then my actual birthday is today.

Uh, so

You know how I figured that out?

what's that?

Because I saw on Facebook it was your birthday and the following day I totally

forgot and I wished you happy birthday.

And that's when you

and you got it

And you're like, oh no, it's actually today's my,

got it wrong, but you got it right by getting it wrong.

You got it right.

Or by being delayed.

exactly.

Yeah.

That's kind of funny.

Um, sometimes I tell people like when they, you know, when they wish me.

You know, happy birthday on Facebook.

I'm like, yeah, thanks, you know, whatever.

Uh, you know, but if it's like work people, I'm like, Hey, just so you

know, I actually do this for a reason.

Like it's privacy and, and you know, your birthday is only one of

like, uh, two in the US only one of two pieces of private information

that are needed to impersonate you.

So, um, you know, the, the one is, you know, so the other one is

social security number, which you don't typically put that out there.

So are you sure you wanna be recording this on your, on the podcast and

I, you know,

it

you know, if, if a hacker is willing to actually follow me on the podcast

get a listen in.

yeah.

Yeah.

We should get on to the business at hand.

Um, our guest is known for her insightful virtualization comments on Twitter, so I

was very excited to see her now focusing on Public Enemy number one, ransomware.

She's been in the industry over 15 years, and in independent

technology, analyst and ransomware resiliency architect, you can follow.

At vmiss.net welcome to the podcast, Melissa Palmer.

AKA @vmiss

Hello gentlemen.

Thank you so much for having me.

how's it going?

Uh, you know, it's funny.

I knew I knew you and followed you for a long time and didn't

know you had another name,

I, I, I, same thing as well, like, I'm like, I've seen like all your tweets

and everything else, but I'm like, I didn't know your actual name either.

I was like, who is this Melissa Palmer person responding to emails?

And

I I get that a lot actually.

People don't know we're the same person.

Yeah.

I, I, um, we actually, we've had a person on the podcast that, um, they continued

to go by their Reddit handle Snorkel 42.

It's like such a random name, you know.

Uh, but yeah, he, like, he wasn't, he wasn't hiding or anything.

He just preferred to go by snorkel42.

So I'm glad to actually know and be able to use your first names.

I'm very excited.

Um, I, I, I am curious, so what, what made you sort of make that jump, right?

You know, you were doing, I see that you, you know, you had background and

backup, you know, good for you, uh, having worked at Veeam, uh, but you

know, you, you've been spending so much time with virtualization lately.

Uh, what, you know, what made you sort of jump over to ransomware.

so it's kind of funny how things work out sometimes.

I have always been, I would say, security minded.

, um, as long as I can remember.

I might have been at DEF com when I was 16 years old.

Anyway, um, so it's kind of a

thing.

Yeah.

Is that true?

it's kind of a thing that has always been, uh, throughout my education,

my master's in is in secure design.

Throughout my career, I've been bringing it in, in Drs and drabs,

but as ransomware started to pick up and I was really putting a big focus.

Disaster recovery and recovery in general from at the VE perspective.

A couple years ago, I kind of said, you know what?

I think I really.

pivot hard and focus on this cuz I, I just find it so interesting,

like all aspects of it.

Uh, and I've learned a lot and I've helped people fix a lot of things they

had going very wrong in their environment.

So hopefully they, they do not feel the impact of ransomware.

So, like I said, I've had the security minded thing throughout my whole

career and it just kind of got to the point where it was like, I'm

gonna go further down this path now.

And I think we need more people like that because there's so

much ransomware out there, right?

There's so many issues.

It's, and I think everyone's trying to figure out, okay, what

are sort of those best practices?

What are the things we should be doing to sort of help protect

ourselves from some of this?

So I'm glad at least there's someone in addition trying to focus on this.

So it helps.

I Is ransomware really happening?

I mean, is it really a thing?

I thought that was like 2020, isn't it?

So one of my favorite things is I just go to Google and I type in ransomware,

and I just see what comes up.

I was like, I, I, I, I think it's fun.

yeah.

Yeah.

have a warped idea of fun as we've established.

Um, but like I just go into Google and I type in ransomware and it, it's funny,

the stuff that does make it to like the mainstream news and you see all these

like people on all the news channels that like, I dunno, sometimes you get someone

and they're like the cybersecurity expert, but they're also like the dog walking

expert and like the cat fighting expert.

I'm like, how do you find these people?

But you'll see a lot of.

So this kind of stuff going mainstream.

So the threat is out there.

It's becoming more and more pervasive.

I don't think we're gonna see less of it.

Um, cuz people have made a lot of money this way, right?

When you have those, when you did your search though, right?

What percentage do you think, or do you even think it's scratching the

surface, like what you see publicly

Oh no.

versus like what's actually happening?

I don't think people fess up unless they have to.

, right.

Unless there's a reason.

And that's actually a problem I had at Veeam working with

the disaster recovery product.

Like no one wanted to be a customer reference.

Like, I don't wanna admit I had a disaster or a ransomware attack or something

and I use this stuff to save my behind.

Like I'm not admitting that.

Um, so that was actually a challenge getting people to like publicly fast on

say, yeah, I got ransomware and everything went to Hella, but we recovered.

Don't worry, like.

Yeah.

And by the way, uh, that reminds me to throw out our usual disclaimer.

Um, I work for Druva, uh,Prasanna, works for Zoom, uh, and this is not

a, this is an independent podcast, not a podcast of either company and

the opinions that you hear are ours.

And, um, also, uh, we'd love to have you join the conversation.

Just reach out to me, uh, w Curtis Preston gmail or WC Preston on Twitter.

Uh, as long as it's up and, um, For now.

And, uh, also please rate us, uh, just, you know, scroll down to

your, you know, you know, most of you based on the stats I'm seeing.

Most of you are on Apple Podcast.

Just scroll down to the bottom there and give us some stars.

Give us some comments.

We love comments.

You can tell us how much for, well, for those of you that

are watching it on video, which you can see@backupcentral.com,

I didn't realize, I thought you guys told me the video was gonna be.

For like outtakes and stuff.

I've been sitting here making funny faces the whole time, like as we

got started, like, cuz I thought you

This may be the best.

This may be the best recording ever.

Uh, you can comment on how much you like, you know, personas,

uh, are, are we at a tweard yet?

You will tell me when you get to a tweard, right?

I think it's a, it's a, theard right,

The, the a the, yeah.

You're, you're at a tweet, but you're not at a, the when is the, the.

Uh, two months.

Really.

Um, so that would be, I, if you don't follow Melissa, he hasn't shaved,

uh, or cut his hair since Covid.

Um, so he is at, at almost at a three year beard, otherwise known as a,

I cannot relate to that.

I'm sorry at all.

It was initially supposed to be a year, which is a year long

beard, and it just kept going.

So

It's interesting, it's been getting grayer lately.

Um,

getting grayer.

what,

it's a stress.

Curtis's stress.

in the Molly Andi household?

Yeah.

getting too stressed by your ransomware.

Um, yeah, so anyway, um, yeah, I, I agree with you of how much it's

gotten out into the, you know, the general, what, what do we call that?

Like the general mindset.

don't know the regular people like

um, yeah, the regular people.

The Normies.

I see it a lot on tv.

I'm seeing it in TV shows, right?

I, uh, the, the, you know, I don't know if you've

Undeclared.

War

the undeclared war is a great show.

Have you seen that, Melissa?

No,

Um, you, uh, so it's, I don't remember where I saw it.

Did I sit on Peacock?

Thank you.

So it's actually a B B C show and it's set in.

Um, so yeah, so, so try to, try to sort of see how crazy this idea seems.

So the bad guy in, you know, the bad.

Country in the show is Russia.

And, and the good guy in the show is, is, you know, England

and, and, and US basically.

But England is the target.

And Russia in the show is using a variety of, uh, cyber attacks

and misinformation attacks to try.

real.

Like this is, wait, this is fake.

Like,

is, this is a, this is a drama.

It's a series.

It's a series.

And, uh, to try and get to, basically to try and get England

to actually declare a war.

They, they're, they're using it, they're using this undeclared war to

get England to actually declare a war.

Um, and, and, and.

It was pretty good.

Uh, you know, they, they got a lot of the tech in there and they

even, I even learned a few things.

Um, so like I learned about, yeah.

What three words have you heard of what?

Three words?

So there's a, there's a group that has taken, uh, every three

meter segment in the world, right?

Three meter squared segment in the world and has assigned three words.

So that, so that you can, you can say, um, you know, uh, you

can go to what three words.com.

You can

this is so cool.

can enter your address and like your house will have multiple three words segments.

Right now it has two purposes.

Uh, one is meeting somebody at Coachella.

Right.

I, I'm, I'm at Squirrel Pizza, you know, tree.

And, and they can put that into, um, it's much easier than saying

I'm at 1 53 negative one genome.

Right.

. Um, and then they can, they can find you.

But also in a lot of the undeveloped world, there's a lot of people

that don't have addresses and this allows them to have an address.

Right.

And they can buy things on Amazon, uh, and have stuff delivered to

their house using what, three words.

Anyway, I learned it from.

So, um, I really don't know how we got onto this, but anyway, the Oh, oh, the

point was that it's, it's out there in the, you know, um, I mean even, is it

the, there's the doctor that has, um, Asperger's, that's, is that the good

Oh, the good doctor.

Yeah.

They had a ransomware attack, took down the

Grey's Anatomy had a ransomware

episode.

Grace Anatomy

big Grey's Anatomy fan, but then the whole Derek thing happened, and I

don't know how I feel about it, and I'm still struggling with that years later.

Um, but yes, Grey's Anatomy had a ransomware episode and I remember

sitting it, watching it just like hysterical through the whole thing.

I was like,

I didn't even have words for it.

I'm like, my favorite TV show has ran somewhere on it.

My life is complete.

yeah.

I, I get excited when shows have backup in it and it, um, my wife

showed me a show just yesterday.

Darn it.

I can't remember what it was, but back up.

Oh, oh, I remember it was, there was a, I don't remember

the show, but there was in the.

The, this woman got interrupted because her, I'm guessing teenage son

called her and saying, Hey, um, like I, my, I'm, my laptop is messed up.

I can't get in my laptop or something.

And, and so he's, and he needs the, the data and she's like, you should

have backed it up like I told you to.

And then she hung up on him and I was

I, yeah, there was a show, and this had to be years ago and I don't

remember Trump, I'm gonna have to go figure it out afterwards, where

like the ESXi shell was like in like

Oh,

really?

And I remember losing my mind.

I remember the guy and it was really hot, but that's all I remember.

Like, I'm gonna have to go figure this out afterwards.

That's funny because you know, normally when you see the sh the stuff like this

in the, in tv, it's not an actual vsx.

I shell, right?

It's some.

Total random thing.

Um, and it's complete nonsense.

Um, here's a question,Prasanna.

Have you seen any ransomware attacks in Bollywood?

I don't think I have yet.

Oh, please, please come find me one.

I love Bollywood

know what we need.

You know what we need?

We need a musical, a ransomware,

Please.

Oh, can we,

ransomware, attack, music

this?

Like, I've thought about this, I literally have thought about this.

I used to do a lot of musical theater and college and stuff like that.

Like I would be so into a ransomware musical.

Like that would be amazing.

This could be, this could

That could be awesome.

yeah.

You know, send some, send some notes.

I I might have come up with some alternate Taylor Swift lyrics

about ransomware at one point.

I'm

Oh, are you guys gonna get into a battle now?

so you, you know, um,

battle.

Yeah.

So Melissa, I've actually produced a handful of parody music videos that had

Oh no, really?

backup.

Yeah.

Um, and one about

to send me some.

I need to see these.

Um, I'll give, I'll give you a quick sample.

Um, Walk into the lab.

Have you seen my VM server?

I'm, I'm so pumped about getting VMs in my server guests on a big disc.

It's so damn freaky.

People like, man, that's downright sneaky strolling into server rooms.

VMs have some massive appeal moving on to guests.

Even database aside for real, putting in some Hyper V. Microsoft said it's free.

Should have done it sooner.

Thing my boss would agree.

Uh, the um,

That's good.

Um, the, the chorus is I'm gonna build VMs, got at least 20 gifts in my server.

I'm on virtual, getting rid of servers.

VMs are so awesome.

It's, it's, uh, what was the original, what was the original song?

Um, what was that song?

What was

Uh, We're, we're gonna go pop some uh uh, McLemore

McLemore.

Yeah.

I'm gonna pop some tags.

Yeah, yeah.

Anyway, it is available on, it is available on YouTube.

I'll throw a link for those of you that are

I've been rewriting Taylor Swift songs lately on a regular basis just because

I don't know why I do this, but I do.

And I used to do demos.

That was my sign of doing a demo.

Like, am I ready to cold do this on stage or something?

Can I sing Taylor Swift while I do the demo?

Like just sing my thing, click through all my stuff, whatever.

And that was like my sign of like, you can't get me on this nowhere.

What happens?

I'm good to go.

Like I have to be able to sing a Taylor Swift song while doing the

that's okay.

I just have to tell you a ran a random, this is, uh, so, uh, several

years ago when I was underemployed, I started doing Uber right.

And then it just turned out I liked it.

So I do it when I'm bored, like I go out and do.

Uber, right.

And, um, like, and also I'm, I'm an extrovert stuck at home,

so I, you know, it's my outlet.

But one night I picked up this couple and the woman had just

broken up with her best friend of like many years over a guy, right?

And she gets in her car, she gets in my car, and she is inconsolable like she's.

Bawling, like just, just ridiculously over the top, bawling her eyes out.

And then she goes, she's, she just, she just, uh, she touches me on

the shoulder and she goes, can you, can you play some Taylor Swift?

Can you play, play some Taylor Swift, any Taylor Swift song and just go, you

know, uh, and I was just like, oh my God.

And then I just, I just said, Hey, you.

Uh, Hey Siri.

Play, play Taylor Swift on Spotify.

Stop it.

Nope.

Nope.

I don't want it.

Sorry.

It started doing it, uh, and it picked a breakup song,

Aw.

which of course all of them are right.

And so, uh, it didn't, and it, it didn't help.

Anyway,

so we were talking about ransomware.

Um,

We were.

in the general public

yeah, because, because it is so huge, right?

And the impact too, right?

It's no longer, Hey, it's just this backend company that gets impacted.

Right?

It's like hospitals, schools, right?

Every, every company, every organization is, yeah.

Is at.

Yeah.

So what do, what do you think?

Um, it, it, it, you know, looking out there from a security, I know from a

backup perspective, um, what do you think from a security perspective,

what do you think are the things that most people get wrong when they're

They don't have their stuff backed up.

Can we

start with

Okay.

Okay.

We

like, can we just start there?

Because like there's this weird cross pollination between

backup and insecurity at

There.

There is.

There is there.

By the way, we used to be

have it backed up, we used to,

We used to be enemies, but we're over that.

Yeah.

Yeah.

it's ridiculous.

Like if you don't have your BA stuff backed up, how do you think

you're ever gonna recover it?

And the amount of people that don't have their stuff backed up still or don't have

everything backed up is still astounding.

When you do, do you run into, you don't run into corporate people that don't

have their stuff backed up, do you?

Oh.

Yeah,

It hurts me.

It hurts me.

it hurts.

Or they don't have everything backed up.

Like, well, this was too expensive to back up before, so we weren't backing it up.

I'm like, well,

how expensive is it if

Or yeah, or someone just spun up something, right?

Your shadow it use cases, right?

And they're like, Hey, corporate, it didn't know about this.

And so no backups were done.

yeah.

Okay.

I, yeah, I can, you know, I think, I think the second part Yeah.

That you said, Melissa, like they missed, they missed something that I

I like, I, I can't tell you how many times like working for a backup vendor, they

would be like, well, it's too expensive to back up this over here cuz it's

only test dev, so we don't back it up.

I'm like, okay, it's test dev.

That's where you're doing all your active development.

You're not backing it up.

So what happens if that goes away?

And they're like, but it's not production.

I'm like, it's not production until something happens.

Then you realize it's production.

My, my

that.

I think that was a common thing.

My favorite test dev story, and this, this is an old story.

Uh, by the way, this month I'll have been in the industry 30 years, Melissa.

Um, and so this is like 28 years ago.

Um, we had a developer group came to me and said, we need

to restore this directory tree.

And they handed me a directory tree that started with /tmp right?

And, and I said, we don't back up temp. Like it's well documented.

We don't back up temp, we don't back up, you know, temp, right?

And this was an HP server, which I don't know what they do

these days, but Temp was in ram.

And so what happened was they rebooted and what went away was a directory, a source

code tree that was like 15 developers.

Storing their source code tree in temp and um, for like months.

And they're like, you don't understand.

This is really important.

I'm like, you don't understand.

You were

backed it up.

source code in.

You know that song, that Beyonce, that like made really pop.

Or if you like it, then you should've put a ring on it.

Like that song.

If you like it, then you should've backed it up.

Very simple.

Yeah, I, I, I do see, uh, and Prasanna, you've run into it as well, right?

Like people not backing up, you know, either, either not having backups or,

you know, we, the, the last episode we talked about, you know, a company

that had a homegrown backups, right?

Um, that was

or, or not even backing up everything required for that application.

right,

Hey, I

it's application dependency.

Mapping's, the worst part of all this

Yeah.

Yeah.

That's why, you know, you know, going all the way back.

That's why I've always just been a fan of, you know, back up all the things.

Right.

Back up all the servers and all the directories.

I know it costs more money, but, um, what,

Ah, but how much will a ransomware attack cost you these days?

To Ching?

There's your justification.

Here's your budget.

Go protect your stuff.

Now.

Finally,

Exactly.

What, one question I have, I know we'll get to it probably at some

point, but with virtualization, does it make it easier to sort of figure

out like everything that's needed,

It depends of course, cuz everything in it depends.

Uh, if everything's hosted in the virtualization environment,

then yeah, it's simple.

But when you get into crazy stuff like well this database is on the Oracle

Rack cluster over there and that's not virtualized cuz Oracle and virtualization

we're not even gonna go there.

Um, that's when you get a little dicey with stuff like that.

Or, you know, especially with hybrid cloud now too.

If you have a app that spans like on-prem in the cloud, then.

Good luck guys.

I hope you actually know what you're doing.

But would you say though, in the virtualized environment that for those

applications which are fully virtualized,

love this question

it

we're gonna go down a dark path right after this.

it makes it a little easier where maybe it doesn't cover, like you said, a hundred

percent of your environment, but it covers some good chunk of your environment

All right, let,

you have a general solution and the rest of it you can focus

Let's go with that.

If you're an organization that's a hundred percent virtualized, which if you're

a company that was started in the last 10 to 20 years, you probably are right?

Yeah.

Just back up the whole virtualization environment and you're good to go.

But you know what else that means?

That's a really big juicy target for the ransomware actors.

They can come in, come through your virtualization environment

and ransomware you a hundred times faster and a hundred times worse.

If they get Es Xi or vCenter, yay.

Right.

I think that's one thing that isn't talked about a lot

It's not.

It's not, and it drives me up a wall.

You brought up an interesting topic there, and I don't think it's one

that's discussed enough, and that is,

environments like vCenter are being targeted as a thing that

they're not just targeting the VMs, they're targeting vCenter.

They're going after vm.

The VMware infrastructure itself, not just the VMs.

I mean, any Windows server you pop these days is probably a vm, right?

If it's OnPrem, no, no, no.

They're going after vCenter, which is a management interface, and the

S X I hosts, they are going after the VMware environment as a whole.

Yeah.

And that, that sort of hurts, right?

Because like you

go up to the backup environments too.

because, uh, yes, no, we, we talk about that a lot on this podcast.

Um, that, um, and it, you know, and I know, I know this, I know this reaches

out to your former employer, but backup environments that are exclusively

Windows based, uh, bug me, right?

Uh, right , um, because I am worried about that,

Because windows is just like the most secure thing ever.

Like how many vulnerabilities out there?

Target windows.

Like,

come on guys.

no one, no ransomware, no one has Windows, laptops that they then bring,

that get infected, and then they bring it

No.

Never.

Never.

You're talking about VMware, does sort of this ransomware angle also affect like

the VMware cloud offerings as well in your mind, or do you think it's more about the

on-prem customer deployed implementations?

would say if, if I was, so, I, I, you know, you know, you've heard the whole

red verse blue team thing, right?

So I would say I'm usually like a blue team or a defender,

recover, all that kinda stuff.

I got, like, when it comes to VMware, I got like a little bit of red team in me.

I gotta be honest, like I got some red team in there.

Um, it kind of comes down to level of effort, right?

If you've deployed VMware cloud the right way, it's probably harder to get into.

Then your traditional on-prem infrastructure, if you've done

everything right, if I have everybody, if everybody can log into my Cloud

V center anyway, and I put it on the internet, then it's a target, right?

Like that kind of thing.

Um, but I would say I've seen a lot of the easier targets are

still the on-prem kind of stuff.

So that's where people go first.

Um, but I, I, I think that everything is a target.

There's kind of a misnomer that the cloud is more secure, right?

Not, it's sometimes a little harder.

So why there's enough low hanging fruit and data centers, why not start there?

Yeah.

Well, I go after that harder target.

Yeah.

Yeah.

Do you want to, for those that don't know what a red and blue

team are, you wanna, uh, fill that?

Yeah, I will.

So if, if you think about it in two different ways, uh,

red team is more like offense.

Like I am the person penetration testing and actively trying to

break stuff and trying to figure out where the weaknesses are.

The blue team is really defense.

I'm the defender.

Um, I'm trying to make sure the red teamers can't break everything cause

I'm trying to secure it and I really feel that backup and recovery does also

fall under the blue team too, right?

Like if I'm, if everything does go to hell, we are ransomware.

We're gonna try, we're putting everything in place now so we can recover later.

Yeah.

I actually know a guy that is a physical pen tester.

Um, and yeah, his, his job is to physically like to

not, he doesn't break in.

He uses

no.

He gets someone to let him in

the door.

engineering and then his job is to get to somewhere where he's not supposed to be.

And take a picture and then, and then get, and then get the hell out.

but that's very valid.

Right?

It's, it's all, there's all different layers and levels of security.

That actually sounds fun.

I think I'd be good at something like that.

I know you can't tell how tall I am, but I'm like five feet tall.

I'm like, wait, like nothing.

So I'm like a tiny little unsuspecting, put a big smile on my face, put some pink

on, like I could probably get it anywhere.

yeah.

I, I think, I think a female physical pen tester would be a, a, a force

to be reckoned with , I think.

You know, um,

career opportunity, Melissa.

just, you know, just play the . It's a little innocent.

I'm not doing anything, you know, I'm lost.

Play, play on all our biases.

That would be mean, but very effective.

Um, so, okay, so we talked about, you know, we talked

about backing up everything.

We talked about the fact that that vCenter is a target, so you need to learn, and,

and I'm, you know, hyper V is a target.

Linux is a target as well.

Like everything's a target.

kvm.

Everything is a target.

But here's the thing that people don't do, and like I said, I'm generally a

blue teamer, but I got some red teaming.

What comes to VMware and I'm kind of thinking, okay, I'm

like a ransomware person.

What do I want?

I wanna make money.

I wanna make you pay the ransom, which means I'm gonna do as much

damage as quickly as possible before you figure out I'm.

Right.

VMware, kind of VMware.

I'm, I'm, I'm kind of like torn right now.

I don't know.

What's a better target?

VMware or your backups?

Probably both.

If you get two people in there right, hit 'em at the same time.

That way you can't recover and everything's gone.

Um, but I'm just looking for a high impact way to wreak havoc.

Hit the VMware environment, that's gonna be fast.

Um, I do nerdy stuff like read ransomware, release notes, and I can't remember

which strain it was, but they're like, oh, we redid something and now

we encrypt, you know, much faster.

We use more CPU threads, right?

So you've got this big, massive vfu host sitting there with all these CPUs in it.

Once you power everything down so you can encrypt it, boom, it's gonna go so fast.

You're probably not even gonna notice before everything is encrypted.

And this encryption, does that happen at the vCenter level or is

it literally you pop each VM one

no, you don't even have to do that.

This is cake.

Let me explain how this works.

So, a VMware cluster is usually a bunch of physical servers in a cluster.

We need shared resources so that these VMs can move around the cluster based on

load balancing and if something fails, restarted, all that kind of stuff.

So the shared resources are basically, um, network and storage,

which means if I have eight nodes in my cluster, let's just use that.

That one host is connected to all the data stores and they

all see the same thing, right?

So if I get into one host, I can see all the storage for the whole cluster.

Now, when we get to the storage level or the data store level,

in VMware, a VM is just a file.

It's a file.

They're encrypting.

It's not, it's.

at the file level, right?

They just encrypt all the files on the data store, pretty much.

It's not like I have to go VM by vm.

They're just files at that point, which is why it happens so

quick and why it's so dangerous.

yeah.

And unlike like your traditional file system, right, these data store files

are pretty large in size, right?

Yeah.

Yeah.

Regarding the, you know, or, or go, you know, go after V

center or go after backup.

Um, the, the big, the big concern that I have, not just cuz generally what

you know, if they're going after the backup system, historically it's been

to just take it out, take it out of the equation, cuz they're gonna do

damage somewhere else and they don't want the backup system used to recover.

um, you can pretty easily get at least a doomsday copy.

Like if you're, if you're doing an on-prem system, most of them have the ability

to get something in the cloud, uh, to u to use to, to, you can deal with that.

hopefully people have half a brainer putting a copy of their backup data

in the cloud, like just by default,

right?

Like hopefully, hopefully.

is some of the encryption methods used by some of the backup vendors

aren't that great and that they can also use basically the backups that,

you know, you talked about how do I get paid the most if I'm a ransomware

Yeah, exactly.

If you can figure out the, the encryption method used by the backup server.

Now, not only do you have you.

All the D, you have unencrypted copies of everything, right?

That, and then you can do an extortion attack, right?

You can say, Hey, I

I love the, I love me a good cup of extortion in the morning.

Like, come on.

That's how you, that's how you and, and like that's how you

get people to pay too, right?

Ooh, I found pictures of your ct c o doing a little something, something.

I'm gonna take

whammy.

Wow.

You go right for the, you go right for the ju.

I do.

I

I I was just thinking like, you know, the CEO's, cuz you know, the thing

is you showed me an email system and I'll show you, I'll show you

emails that shouldn't have been sent.

Right.

Um,

yeah, let's go with that.

It's a little more tamer.

Like

Yeah.

Yeah.

Uh, emails that, um, I, you know, I've known, you know, and, and like even

in places where, you know, we, you know, I've been in the corporate world

for 30 years now, and it's changed over the years when we talk about

things like sexual harassment, right?

Um, it ha it ha it has changed, right?

Um, But like, what a lot of it has done is it's just gone closeted, right?

It's like, you know, so guys still talk amongst each other, but

they still do it on email, right?

And you're

Oh, I've got some stories about

Oh, I'm, I am absolutely sure

I got stories.

I am sure you do.

Uh, but that's what, if I were, if I were a hacker, I would be going after

the backups and I would be going after backups specifically where I could

figure out the encryption mechanism.

and that I can, maybe, I can't decrypt the data directly, but what I can do is

I can get administrative access to the backup server and then I can restore

whatever I want, wherever I want.

And a lot of people, a lot of people aren't watching their backup

one.

No, they're

Right.

Um, not like, not like they should be because, well, let me ask you this.

So you, you, you've dealt with a lot of backup folk.

I have.

It, it's, it's still this thing of like, nobody wants to do it.

Right.

And so it's the junior person

I will say, I will say one of my specializations when I worked with backup

was also monitoring the backup systems.

And I was telling everybody, you realize you need to be monitoring

these two for like a number of reasons, especially like if you're

ransomware and you go to Restore and you realize your backups weren't running.

Like that's a big one too, but kind of looking at like, Hey, like why is Bob

from accounting restoring a VM at 3:00 AM.

Bob from accounting shouldn't be doing that.

Like what is going on here?

Well, someone got his credentials and he had access to the backup server.

Hello?

yeah, yeah.

Um, least privilege, right?

The

One of my favorites.

That is probably like my number one, I talk to people about

like, let's start there please.

yeah.

Yeah.

Especially when it comes to VMware, right?

Like Bob, I like Bob.

I'm gonna pick on Bob from accounting now, like Bob from Accounting

shouldn't be able to log into vCenter.

I'm just putting that out there

Yeah, I know Bob from accounting's, an idiot.

Are there other things you would recommend sort of as like best practices

to sort of reducing the risk of ransomware in a vCenter environment?

put vCenter on the internet.

If you go to Showdan, it's all over the place.

People still do this.

People put their ES x I hosts on the internet too.

Do not do this, please.

And I know, but Melissa, there's valid reason that we would do this.

And if you do it in a protected manner and blah, blah, blah, and

you think it's safe, well whatever.

Nothing's safe these days, fine.

Fight me on it.

But like, let's start there.

Let's start with the basics.

Um, that's important.

Principle least privilege is a big thing.

Um, Having a good strong E S X I root password is a good thing.

Not having it written on or in a file on your desktop.

What was it?

I, so I follow a lot of this stuff and I can't remember, oh, it was some

big hack and I can't remember which one right now, but it was really going

around Twitter and like someone found the password file that was on someone's

desktop and whoever posted on Twitter, it was all redacted with the passwords

out, but they had every password to all of the infrastructure in a notepad file.

So someone got into someone's desktop, cuz that's when a lot of it happens.

They get access to your desktop or your PC or whatever they found it.

And guess what?

Now I have the root password for E S X I. I have the keys to the whole kingdom.

Like, don't

You know, the, the thing is these things sound so stupid, but you know

that, you know, like so many of the hacks that happen, ransomware and,

and, uh, and otherwise they're, because of really stupid stuff.

Like not installing

human error.

right?

Not installing a patch, having your root passwords up on a thing, um, you know,

saved in a browser.

Right?

Your password.

Like don't do

yeah.

Um, the, so, so it's like the, these seem like really basic things, but

if everybody in the world did these really basic things, there would be

a significantly, um, smaller amount of ransomware, I think, in the

But I have a question about that though.

I agree with everything you guys have said.

. But if you got rid of all the low hanging fruits, wouldn't

everything else become much har,

Well, that's the thing, right?

Once we get through this and we

It'll be the next level.

Yeah.

that's the thing, right?

So like these threat actors are out there doing this stuff day in and day out.

Like, uh, it is like if I'm a threat actor, like.

. I bet they, I bet these gangs have like VMware specialists working

for them at this point, that all they do is go in and home.

VMware, I'm sure they have a backup specialist that they

know all the backup systems.

They just go like, you have to understand that these threat actors are specialized.

Right.

Of course there's generalists.

Um, you have the whole ransomware as a service thing where they just get in

and they kind of hand it over to the threat actors and all that kind of stuff.

So like all these people do is, and they're generally probably pretty smart

people, is like, I'm just gonna figure out every way I. Just own VMware.

And that's, that's, that's what they do day in and day out, right.

So it, it's hard to compete that with that kind of stuff.

And once we clear up the basics, yes, there's gonna be another area to target.

There's gonna be something new to exploit.

Um, those zero days are gonna come out and people aren't gonna patch 'em

and everybody's watching it, right?

Like I read, um, All the CVEs and stuff like that.

Like they're just sitting there going, oh, I can exploit this and off to the races.

Like it's, it's a big thing.

There's no, there's no silver bullet.

There's no one size fits all.

It's just

Well, I know.

mitigate the risk.

Right?

Yeah.

That, that's why my approach when talking to people has been, just assume that

ransomware is going to get into your

Assume breach.

Thank you.

let's, just, let's just stop playing around.

Assume breach.

How do you recover?

How do you stop them?

How do you recover?

And how do you, and how do you limit the blast?

Right.

How do you, you know, we, you know, I

do you, how do you limit, the amount of damage they can do and then recover.

I know,

That's where it has

And a, and a great for those that are, you know, if you're listening

to this and you're on, because you're a fan of @vmiss, that's great.

Uh, you should check out this other guy that we, we had on a podcast.

We went pretty deep into this Snorkel 42.

I'll put a link in the show notes.

Um, so we, you know, he went into things like, um, what do you call it?

Um, um, limiting.

U Rack reference?

Like how did he come up with 42?

You know what

I

Rack or is it like, what's that

know, we didn't ask, we didn't ask.

Oh, Hitchhiker's guide.

the Universe?

Yeah.

The Hitchhikers guide.

Yeah.

Yeah.

He posts on Reddit all the time on the CIS admin forum, so,

Yeah.

Um, and you know, he, he talked a lot about limit limit limiting

or stopping lateral movement within your company, period.

Cuz it's, it's, it's the kind of thing where people.

I've only been in literally one company, one company in my entire

career where lateral movement had been completely shut off.

Right.

And, and I, and I only knew that was because backup was really, really hard.

like we, we had to go in and, yeah.

And I had, there, there's a, there's a great story, which I won't retell right

now, but it ends up with me losing.

Stuff at late at night.

Um, and, uh, because of they did that.

But that's the kind of thing you have to do.

Look at it's, it's, it's like the, it's like the concept of least privilege.

Look at your network, figure out which servers need to talk to which servers

and make that happen and nothing else.

Um, what, anything else that you're, you're thinking about Melissa,

Oh, there's so much.

There's, there's so much.

It's just like, it's a ridiculous amount of stuff and it's little stuff, right?

It's like leaving s ssh on making sure it's turned off by detail fault.

That's a good way to get in.

Uh, anything, anybody who has access to vCenter, right?

We

RDP

about rdp?

Well, the good news is vCenter is a Linux-based appliance.

So you can't already p to vCenter anymore, at least if there's still

some Windows vCenters around there.

Wish they probably are

there, there.

I shouldn't say that.

See, I feel weird like saying all this stuff.

Like I hate going places and be like, well here's how you break into word.

Really screw it up.

Um, I feel like I shouldn't be doing that, but I'm sure

Yeah, I mean,

stuff.

Um, I think there's still some Windows V centers hanging around.

. Um, but the same thing with the V Center, right?

Don't, don't have SSH on there either.

Turn off all the ssh s it's really simple to do, but people like it.

It's like a thing, right?

Like, oh, it's easier to ssh and go do whatever I have to do, but you forget to

turn it off afterwards, stuff like that.

Um, VMware's actually been very good about, um, they have like a whole

ransomware page where they list everything out that they suggest and stuff like that.

And that's like a good reading starting point for anybody.

But people, people just get like sloppy and, and I get that

and I have found like being.

It's weird.

I have like two personalities, like which Melissa's gonna show up?

Is it VMware, Melissa and infrastructure VMware's infrastructure?

Melissa's gonna show up.

Or is security Melissa gonna show up?

Are they gonna show up together?

Like who knows, right?

It's like I've got these two personalities.

Um, and I've noticed that there is not a lot of cross

pollination in this space, right?

There's not a lot of VMware people doing security and there's not a lot of

security people that really understand.

and I've seen this gap for a very long time, and I'm like trying to

bridge it with some of my blog posts and my content and stuff like that.

So I'll be putting more effort into there.

But you know, you really gotta the two organ, the two teams

really just need to work together.

that's interesting that you mentioned like, yeah, security and

virtualization teams not necessarily

Like I can tell you, every time I see a VMware ransomware article

in the news, it is factually.

, like, I don't know where they're getting their information from, from, but it's

like usually wrong most of the time.

And I'm just like, people don't understand these things.

Yeah.

I wonder if it's kind of like back in the day, how backup and

virtualization teams never talked to each other and everything was broken.

Maybe if they need something like that.

I remember those days and I feel old saying that, but

I, I do remember those days.

Do you remember?

You remember?

Uh, what was it?

Uh, V C B. You remember V c b

Yeah.

I said, I said that it stood for very crappy backup.

That's what I said.

It stood

Yeah, I remember

Um, yeah, that was

More backup

1.0. Um, yeah.

So e everything you just said about VMware, I would take, and I would

use, I would say exactly the same thing about backup teams, right?

And they're often, they're often very junior.

So what happens when we have to get the VMware team, the backup team, and

the security team in the same room?

What is

And network and network team.

Don't forget that.

the network team too while we're at it.

Well, I, I mean, hopefully these attacks have become so common, right.

You know, um, Druva did a, a survey and, and half of the companies

said that they had been hit with ransomware in the last three years.

Right.

Um, and.

You know, hopefully things are become, because you know, if I back up, if I

look at traditionally backup and Dr.

Um, you could often, you could often say things like, well, if, if a meteor hits

or if, if a, you know, if the earthquake takes out, I live in San Diego, right.

If the earthquake and, and suddenly Arizona becomes beach freight property,

I'm gonna be dead and I won't care.

Right.

And the, and the odds of that are, you know, right.

But,

And that's the

but you can't say that with, with

the problem with DR. And all the traditional dr. I like to say that

ransomware is a disaster, right?

Your disaster recovery plan is a great place to start.

But here's the thing, how many organizations didn't actually bother?

Cause we're gonna accept the risk of the meteor strike cuz it's not gonna happen.

Right,

Versus ransomware, which is so much

gonna happen.

It's not if it's

Yeah.

I remember being in a, in, in a, in a meeting trying to work with a large.

Company, defense contractor and, and, and, and they were basically saying, yeah, if,

if, you know, if, if that hit, if that happens, I will be dead and I won't care.

That was literally his official position.

Let's move on.

Move on.

He said . I was like,

But one question I have, so we're saying that ransomware is common, right?

People are hit with it, but are there sort of best practices like, Hey,

here's what you should be doing, and not just in silos, like the backup

team has stuff that they talk about the VMware, like you said, VMware published

something on how to prevent it, but.

Sort of looking holistically across all these organizations, security, networking,

virtualization, backup teams, right?

To come together as, Hey, here's really what you guys should be

talking about before, letting each team sort of figure things out.

So here's the interesting thing, part interesting thing.

I think until the tail end of 2022, the number one way threat actors got

in was through phishing attacks, right?

Someone clicked a link in the email.

, that was the number one way, but I believe in the later half of the year,

and you guys might know better, it switched to vulnerabilities, right?

Vulnerabilities are now the number one way threat actors are getting in.

So I think we really need to start with.

How are they getting in and starting there?

And each piece right kind of starts with cleaning up their house,

the VMware vulnerabilities, cuz there are VMware vulnerabilities.

Like everybody likes to talk about hypervisor escapes.

Like, that's like the classic VMware hacking thing.

Like, hahaha hypervisor escape.

I'm gonna be, and I'm gonna take over the hose.

Like I, it drives me up a wall.

I'm like, that's all anybody ever thinks of when they think about virtualization

insecurity as a hypervisor escape.

And that does not.

, no one cares.

That's not what's gonna get you.

Right.

So if we start with something like vulnerabilities, right?

Everybody's gotta clean their own house, right?

All the VMware team, the network team, the storage team, the backup

team, cuz backup software has vulnerabilities sometimes too.

Like anything can be vulnerable.

So let's look at the way that the threat actors are getting in and

everybody clean up their house.

And then let's all get together and talk about how we clean up

our house and go from there.

Yeah.

I think if, if we look at like all these teams, right?

What they all have in common is let's get good passwords in a password

management system, whatever you have, let's make sure that patch management

and patch installs is, is top of the top of the priority, right?

Get MFA.

. Right.

Um, and, you know, and, and, and, and, and monitoring and, and also

the concept of least privilege.

How are you, how are you implementing these concepts in your environment?

Security team, backup team s you know,

Security team too, right?

They don't get a free pass.

It's not like I'm the security person, so I don't have to update my software.

Like it doesn't work that way.

Like you're, you're the same as everybody else,

Yeah, because I think if you, if you just, if you just put in like, so many

hacks are simply based on zero zero day vulnerabilities that came out six

months ago that have been, that have been

and no one

that no one patched, right?

You know, you look, you look at what happened at Rackspace.

The Rackspace, they're calling it a zero day vulnerability, but it was actually

fixed only because it was unknown.

Prior to that, but it was actually fixed by the patch that came

out a month before the attack,

And I think, um, I remember was it Exchange or something?

I don't remember what, but I remember seeing this go around.

It was, uh, some microsofty thing.

I don't know if it was like RDP or Exchange R d p,

ransomware Deployment Protocol.

Um,

they've,

I.

Um, so it was something that, it was like a lot of, uh, windows-based

ransomware going around, but it was the same thing, like the vulnerability

used was like six months old and no one had bothered to patch it so,

Yeah.

Yeah,

So, uh, I know we talked about like each house cleaning up.

I think though, the other thing that these four groups need coordinated with is when

they do get hit by ransomware though, what does their response look like?

I feel that a lot of organizations don't have that.

of Worm as my friend.

I know a lot of organizations don't have that plan.

In fact, Curtis, when we had Tony from Spec Spectra Logic on the call, right?

Talking through like what happened when Spectra Logic

got hit with ransomware, right?

His big thing was like, I don't even know where to start.

Right?

And luckily they had cyber insurance they had just signed

up for the month before, right?

And so they had experts who would come in and sort of guide them through that.

But a lot of these organizations like, it's almost like you have

to do that fire drill right ahead of time and be like, Hey,

have it.

That's what you have to do.

You have to practice

Honestly, uh,

DR test, ransomware recovery test.

I want us to do an entirely separate recording on that.

I, I, I agree with you.

We're already, we're already over our normal time.

Uh, and we, and I don't wanna shortchange that topic.

I think that topic is, is dead onPrasanna and, uh, and I

think Melissa should come back.

What do you think, Melissa?

Yeah.

Absolutely.

I'd love to come back.

All right.

All right.

Well, I have a birthday lunch waiting for me.

You do.

I'm gonna go do that.

And, um, Melissa, uh, this, this has been great, uh, exciting and, and I'd love to

hear, you know, uh, somebody talk about backup and security all at the same time,

I know it's fun, right?

There's like, how many of us are there out there?

I don't think there's many of us.

It's so nice to be able to have a conversation about it.

yeah, and thanks again.

Anytime.

Nice to meet you, Melissa, and looking forward to having you back on.

Absolutely.

All right, and thanks again to our listeners.

We're nothing without you.