You found the backup wrap up your go-to podcast for all things

backup recovery and cyber recovery.

This episode, we continue our Mr. Robot series by breaking down advanced

persistent threats, those sneaky attacks that stick around in your

network way longer than they should.

We're talking about how they get in, how they set up shop, operate

undetected for weeks or even months.

You'll learn about dwell time, why you need to monitor, uh, new devices,

uh, also why the scan and restore approach to ransomware recovery is

really kind of just asking for trouble.

I hope you enjoy this episode on advanced persistent threats.

By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr.

Backup, and I've been passionate about backup and recovery for

over 30 years, ever since.

I had to tell my boss that there were no backups of the

production database we just lost.

I don't want that to happen to you, and that's why I do this.

On this podcast, we turn unappreciated admins into Cyber Recovery Heroes.

This is the backup wrap up.

Welcome to the show.

Hi, I am w Curtis Preston, AKA, Mr. Backup, and I have with me a guy

that I know is 100% super jealous of the shirt I'm wearing right now

You know how many years I've been asking you to get merchandise?

So for those of you who don't catch us on YouTube, we are on YouTube

under the backup wrap up channel.

And there today for, uh, today's

first time.

see, yeah, for the first

Yeah.

you could see the new shirts, which says the backup wrap up.

It's a

Um, yeah.

Yeah.

Um, and we're, and we're not even, we're not selling the merch or

anything, at least not yet, but, um,

If

yeah,

let us know.

yeah.

us or leave a comment on your favorite pod catcher and let us

Absolutely.

Absolutely.

Um, yeah, so, um,

Five

so I finally, I finally ordered it What.

Five years.

So I finally received it and now I have it.

It got, both of 'em got shipped to my house,

Hmm.

my shirt, and your tiny, your tiny little shirt.

Medium.

He says medium.

I'm like, what is a medium?

I don't even, I don't even know they made such sizes.

Um, anyway, so, um, you'll look forward to receiving your shirt

when I decide to grace it upon you.

It's sitting in my bedroom.

Um,

that's

oh, you're

it's, whenever it's ready.

It'd be amazing.

Okay,

so, so wait,

what.

I wanna know when do I get my shirt?

When, when I decide to send it to you, it's, oh, by the way, it's right here.

There's your shirt.

There's your little, little tiny, the one with the with the M on it.

There you go.

There's your M.

You know, if you

Yeah.

if you look at it upside down, it's a w.

Yeah.

For women's, that's, uh.

So there you go.

Uh, that's the big day Today we finally have a, a branded shirt where

it's like, we're like a real podcast.

Um, so hey, we are gonna get right into it today.

Um, and we're gonna talk about, I, I I, a topic that comes up quite a bit.

We're continuing our coverage of Mr. Robot, which I'm now one episode

in the future and I now know why Mr. Robot is called Mr. Robot

tell me.

I have not

Y.

yet.

Yeah.

So, um, yeah.

Um, and.

This episode, which as you may recall, uh, we we're, we're doing two episodes,

do two podcast episodes on one Mr.

Robot episode.

Uh, last week we talked about the concept of honeypots, and this week

we're talking about, um, the, the cyber attack that happens on All Safe.

So if.

technically we're doing two episodes on two episodes.

Yeah, you're right, you're right.

That's kind of why we did it.

'cause the one episode had hardly anything in it.

Right?

Yeah.

So, um, but one of the things that happens in these two episodes, which

were episodes seven and eight, AKA 1.6, and 1.7, um, that, uh, in the episode,

the main episode that we're talking about in this episode is White Rose,

which we, we are actually introduced to the character of White Rose.

But in the midst of all of this, there is a cyber attack of Allsafe.

And it was, it was really kind of, it was kind of cool how it all went down.

So, uh, Gideon is over there visiting Tyrell and he's telling them about all

the stuff that they've done to, um.

To regarding basically the initial hack that caused the,

the email leak and all of that.

Uh, and, and in the midst of that, his phone starts blowing up with, uh, 9 1 1.

Everything is down.

Yep.

So all you know, we, we, at that point, you know, anybody who follows the

show would know that they got hacked.

yeah,

Um,

and Elliot gets paged as well.

I don't know if you caught that.

yeah, yeah, Elliot gets Paige as well.

Uh, but the thing that I liked about the scene with Gideon and, and, um.

And Tyrell is, uh, Gideon gets, you know, um, you know, e everything's down.

And then Tyrell's next line was, well, I really appreciate your transparency.

Yeah,

which of course he is not being transparent at that exact moment.

And he goes back and his first response is, um, basically

we've got to figure this out.

If this gets out, we're toast.

Right?

sorry.

Clarification.

I thought he goes and tells Tyrell that he, the DAT file was compromised.

That, but that's separate.

That's referring to the original hack.

That's right.

And then doesn't he also tell him not that they've been attacked

because they don't know that yet?

I thought he tells 'em think that we found something left behind.

He said he, he said that he's gonna continue to like look into that hack.

Okay.

Right?

Um, and, and we've got the, we, we, we put the honeypot and all of that.

That's what that was about.

But then what we have is, while this is happening, he finds out

essentially that they've been hacked.

And his first reaction is not transparency,

Yep.

his first reaction, he gets back and he's like.

We gotta figure this out.

Uh, you know, gimme all the logs and everything and, and, and

if this gets out, we're toast.

yep.

um, uh, which.

the accounts, all the sales calls,

Yeah.

That cancel all the sales calls you, you know, no one's gonna figure out anything.

Uh, and, and that of course, uh, backfires because f Society who caused the attack,

um, they take over their, uh, again, with the, the hat we've talked about it.

You know, before the hack of the, of the tv.

So they hack the TV and um, and they play the, you know, it's

the, this is like, this is the, this is what it's in the movies.

It's always in the movies where they do this, right.

Where they take over the

mask.

TV and they play the message and the remote doesn't work.

Ah, can't even the, even the electrical plug.

Yeah.

It's like, no, I wanna see it.

And what you see, see is that basically f society, and again, this is, this is all.

Whatever, but, but f Society's message to them is you have helped

protect Evil Corp and therefore we are going to take you out as well.

You are com complicit in Evil Corp's evilness and um, what?

but the entire reason they did that though was it was a misdirection.

Yes, yes.

Again, but what they're being told is right.

We're, we're gonna do this.

Right.

And, uh, and they also said that they even posted it to their website

so, so much for, you know, not, uh, not letting the world know.

Yep.

but yeah.

it was a misdirection.

So Elliot can steal the MFA from Gideon

Right.

actually send an email telling them to shut down the honeypot and they have

Yeah.

hours.

Yeah, exactly.

Um, which is, um, did I miss anything in the summary, which is, um, let,

that's relevant to this, to this

well, the one thing you did miss.

Uhhuh

Is when, uh, told Terrell about, well there are two things.

One, when Gideon told Tyrell about the dat file

Uhhuh.

and honey

Mm-hmm.

right, the first thing Tyrell did is he logged into the server,

Right, right.

right?

And

Yeah.

he sort of was like, let me hunt ter around and see

what I don't have access to.

And he noticed an F Society file before he like has to leave all of a

sudden because the police are there.

Right, right.

Because of

one.

the murder.

Yeah.

Yeah.

So the thing that happened to Ollie was they weren't letting go of him.

Remember?

And they still had access to the networks.

Right.

And

Yeah.

where they wanted,

in this

sorry.

They as a dark army and

Right, right,

Elliot.

To have this meeting with White Rose, which is what

Darlene was trying to arrange.

right.

Right?

And they literally still had access to the networks because remember Angela

had put the CD in the computer and had put the virus out there and they still

had access to the network and they were doing things, and this is where Ollie.

Comes up to Elliot in the middle of this attack, by the way,

Yeah.

is freaking out.

Everyone's in the room,

Right.

Everyone is like panicking and trying to figure out what's

going on and all hands on deck.

Elliot sits down at his desk, He's like, okay, let me start doing some research.

And then all, he just kind of strolls up and he's like the sales guy, right?

And he is like, yo, bro, I got two hard drives for you.

Take it over to the recovery place or the shredding place.

Yeah, and he's like, what about one of the IT guys, gophers?

He is like, no, man, it has to be you.

Elliot thinks, and he is like, oh, this is why Dark Army wanted me

Yeah, yeah,

all of this was because the Dark Army still had a foothold inside of allsafe.

Right, right.

Yeah.

So, yeah.

So those, those were, uh, two, uh, kind of important things that

are relevant to the topic here.

So this week we're gonna talk about, uh, I think the formal term would be

called advanced persistent threat.

Right.

Yeah.

And this idea of you've got a threat actor that is, that has, as you mentioned,

a foothold that has, you know, they've used maybe an initial access broker.

Maybe they, you know, somehow they are in the network and they're, they are

operating undetected in the network.

And, um, because they're able to do that.

They're able to, you know, to do a, a lot of things, right?

So in this case, what we see is they, they're, they're able to

control one of the employees.

They're able to control sort of a second employee in the

case of, uh, Elliot, right?

Mm-hmm.

Um, and then also you've got two sort of.

Um, two persistent threats going on because, uh, because they have

the, the connection and all safe, they're also able to, to facilitate

this, the, what do you call it?

Um, f society is able to do this hack whose entire purpose was

redirection, as you mentioned.

Right.

Um, because we want to be able to do this, this thing with the phone and, um.

So that he could get in and, and, uh, get that, get that M-F-A-M-F-A.

Again, as we've discussed, m FFA is not perfect if you let your battery run down.

I like how he run the, he ran the battery down, uh, by the way,

um, story from back in the day.

Um, so this is in my very first job and I, I don't know if I've told you

this story before, but my very first job we used to have, we were at a.

We were, it was a, a DEC shop, right?

It was, it was, it was, uh, Ultrix, which was the, the DEC, this

is before digital Unix, right?

This was Ultrix and we had, our servers were Ultrix and uh,

and our desktops were Ultrix.

Important part of the story.

Both the servers and the desktops were Ultrix.

And one of the things that.

We would do back in the day.

Um, gosh, I don't, I don't, I don't think this lasted, but if you wanted

to run a, um, an application on.

The server and have the UI of that application display

its UI onto your server.

X

had to run this command right to, to allow that to happen.

And it was common in our environment to just run that, that command as

part of like your login, like, like you always ran that because you

always wanted to, um, to do that.

That left you open.

For anyone to open up

Yep.

X display

yep.

thing on your, on your desktop.

And, uh, there was a program called Stars

Hmm.

it was a, uh, it was a screensaver

Okay.

Yeah.

Yeah.

and if you wanted to crash someone's server, you would just

run a quick for loop and fire off stars like a thousand times.

Right.

And um, one day, you know, one of the guys in the shop was gonna

do that to one of the other guys.

And so he just did it, is like in the middle of the workday,

you know, and, uh, he said, boom.

And you set your display to that workstation and then you fire it off.

And, um, he did that.

And then we're waiting for the reaction.

No reaction.

And then, and then, uh, this guy is like, Hey Joe, how's your workstation?

And Joe's like, it's fine.

Why pregnant?

Pause.

Oh shit.

And he had his display set to the server,

Oh no.

so he had just crashed a production server in the middle of the

workday as a practical joke.

Career ending move.

Uh, no, no.

That guy had, that guy was like a, like a nine lives of a cat.

'cause he definitely did some, some serious CLMs.

Um, yeah.

so

It, yeah.

Go ahead.

back, back to the,

Back, back to the scope.

Yeah.

one thing about, about this entire like F Society hack on Evil Core,

Mm-hmm.

Elliot goes, right.

He flies in the data center, right in the earlier episodes.

He fixes it.

He gets everything back.

They never follow the best practices when it comes to

recovering from ransomware, right?

Right.

they do?

They just scanned the server and they're like, yeah, it's good to go.

Let's

Yeah.

It looks good.

Yeah.

Yeah.

They didn't quarantine it really,

Right,

for like the 30 seconds, whatever.

When

right.

they didn't rebuild it.

Yeah.

still left it in production.

It was

Right.

available.

Yeah.

Yeah.

And, and the thing is, with with, you know, when we talk about APTs, the

thing to understand about this is the, the length of the dwell time, right?

So you wanna define dwell time.

Yeah, the dwell time is basically long ransomware sits on your

system before it either starts running or it's been detected.

Yeah.

And um, you know, in this case it was there for quite a long time and, um.

That's, yeah.

Yeah.

Well, and you had, you had a couple of different APTs going on here, right?

You had the one going on at Ollie's home,

Yep.

Uh, and then you had the one going on in the Evil Corp

network, by the way, isn't it?

Evil Corp? I think you're saying Evil Corp

Evil, evil

or, okay.

my, my P is very silent.

Oh, so,

Yeah.

um, the, um, go ahead.

missing one,

What's the third one?

the fourth,

What the, what?

there's a four.

So what, so I just remembered two.

There is a fourth one that you missed.

Well wait, we didn't get to a third,

Okay,

it,

first is Evil Corp. Hack by F Society.

yeah,

The second is Ollie's laptop at home.

yeah.

The third is, uh, dark Armies Hack on All Safes Network, right?

Right.

Technically the fourth is FSO Society, but I don't think it's

really a persistent threat 'cause that was like a one time hack.

Okay.

Right.

But I would say that the actual fourth one, and this is one that was very subtle,

you may not have picked up on Elliot Hack on Steel Mountain of the thermostats.

Oh, right.

Yeah.

Right.

That was definitely an a PT right there.

Right.

Um, he had a foothold in there.

Yep.

So, you know, you go back a few episodes and we talked about their plan with,

um, steel Mountain was to hack the.

The, uh, climate control system and then set off, uh, the, you know, basically

make it too hot in the, the data center for tapes and have the tapes melt.

We discussed that that was kind of bs, but that was the plan.

And their plan succeeded.

They, they had gained, they, they, you know, they put a raspberry pie in the

right place, which of course, everyone knows where your, uh, climate control

system controls are, are in the executive.

Washroom.

Everybody knows that.

Yeah,

Um, and that's, that's where they put it.

Um, and yeah.

And so they, they had this, this problem sitting there ready to go

anytime that they needed to do it.

Yeah.

Right.

and, but then did you remember also during that episode, because remember by

this time the Dark Army had an attack.

They weren't able to raise a temperature on the data center and

burn the tapes, right, ruin the tapes.

But what they did have is they had that device and still connected

Yep.

and they realized that the company that manufactures that thermostat

has networked is has a cloud,

Right,

and they're using this exact same thermostat at all the

other steel mounted facilities.

right.

And so do they, do they end up controlling all of the thermostats or.

they have done nothing yet.

Okay.

So they just know that Yeah.

And this is why Elliot asks and meets White Rose and says, Hey, need time.

I, or I need you guys to act now.

Right.

And this is

Right.

Rose tells him, I will give you 43 hours or

Yeah.

Some, some very,

think it was

some very precise time.

Yeah.

Yeah.

And so that's why Elliot's in this rush trying to get like turn

off the honey pot so then they

Right.

go do the rest of the hack and hopefully try to destroy Steel Mountain and

Yeah.

Army attack the China data center.

Yeah.

And we can play with the, the thing of the honeypot and how that, the way

he described how the honeypot is that if the honeypot is what it says it is,

that they, they wouldn't be detected because it's not on the network.

But anyway, I,

Yeah.

you know, yeah.

the fact, and even the fact that like right?

That's in the thing.

The

Mm-hmm.

it's been days, no one's recogni, like you scan the network, right?

No one's realized, Hey, what is this thing that's on

Yeah.

Yeah.

Yeah.

And, and that's, and that's kind of what we need to talk about a little bit is

there are ways to detect APTs, right?

You're looking for, and I think.

There's, there's two things, right?

There's a couple of things.

One is you are looking for devices that have IP addresses on your network

that they didn't, so how do you, you know, that they're not supposed to

have, or that, that, that they're new.

Every new IP address on your network should be registered

and should be known, right?

Uh, both from a wireless perspective and a and a wired perspective.

And certainly anything if you, if you have wireless, if you have.

Unregistered, like guest wireless access, that's one thing, right?

And they have access just to the internet, but not to the corporate network.

You could probably allow unregistered devices there, but if you're allowing

a device to gain a IP address on your corporate network and you don't know

what this device is, this is a problem.

Yeah, and just a slight correction.

You're probably referring to a Mac address, at least at the minimum.

You had said IP address of

What's, why is that a problem?

Because anyone can have an ip, sorry, the MAC address is what they would

typically use to register against the network to then get the IP address.

Uh, yeah.

Okay.

I, you know, you're,

authentication mechanisms like

yeah,

other things

but I'm just saying

secure.

a, a new device was given an IP address, right?

A New Mac address to, you know, to use, to use your, um,

Well, a device

to know.

a nuke Mac address was given an IP address and the

Okay.

not.

No.

Yeah.

Okay.

Okay.

Okay.

That, that's perfectly correct.

Yeah.

Yeah.

Tomato potato.

Um, so

but

but, but you've got a new device that got it, that's gi, but given an IP

address on the network and no one seems to notice and it's there for how long?

at least three days,

Yeah.

Yeah, I, I'd like to think that a new device, you know, if you're doing

this properly, especially if you're at a place like Steel Mountain, no

new device should be given an IP address, no New Mac address, right?

Should be given a new IP address that isn't already known to the network.

Right?

exactly.

Uh, now we know that that actually happens all the time.

We actually did an episode a while back, I dunno if you remember this,

but we did an episode a while back on like wireless hacking where people are

creating devices that, that get sent to the network to be able to get on

the wireless network to then be able to get onto the corporate network.

Um, and, and.

like what he did with the cop car.

It is a little bit like what he did with the cup card.

Yeah.

Um, and, uh, and we know that that happens.

We know that corporations all the time allow new devices to be registered

on the network, get IP addresses and communicate with the network completely

unmonitored, you know, completely unmonitored like you would think at,

at a minimum you would if you do allow.

An unregistered address, you know, you would then monitor that device like crazy.

What is this device doing?

Right?

Oh, it's communicating to all my other facilities, you know,

Yeah, that's probably not

or

idea.

Yeah, or lock it down.

Good point.

Right?

Lock it down so that it can only communicate with whatever.

I don't know.

internet.

I mean, so, uh, the other thing that, again, this would help minimize.

The blast radius of an a PT, and that is devices, servers should really only, you

should also be following, um, the concept of least privilege for them as well.

Mm-hmm.

Smart devices like, uh, thermostat controlling the network or

controlling your, your, um, cooling system should not be also be able

to talk to servers, you know.

what was it?

Was it Home Depot or Albertson?

Was Target.

the

target.

attack, right.

Yeah.

Yeah,

How did they get in?

it was.

Yeah.

commercials.

Yeah, yeah.

So again, we know what happens.

Yep.

But how, but how do we, how do we stop this, right?

Yeah.

I would say one of the first things is to detect for new devices.

That's the one to stop new, like this kind of attack.

But many times the device that's doing the communicating is an existing device

that's been, uh, compromise in some way.

And so I think.

The only way for that to work is either, either a human being that's really good at

monitoring the different types of traffic or, you know, use of machine learning,

um, to, to watch what is normal traffic, what are the servers and things that

we normally talk to, and if suddenly a thermostat that controls the, you know.

Whatever the Toronto facility is, suddenly talking to all the other thermostats,

but but

that's a problem.

also remember, I think it was when we had.

Mike, I wanna say it might have been an episode with Mike.

are Mike Saylor, by the way, who is Curtis's co-author on

the upcoming O'Reilly book,

Very excited.

Yes.

Um, he'll add one more to the shelf behind you.

Um, so.

I think it was Mike who had mentioned like, people are now starting to

exfiltrate, uh, exfiltrate data,

Right,

by communicating directly to another server, but by sending DNS packets

right.

doing lookups.

So very, very small amounts of data and that's really, really hard to catch.

Yeah.

Well, it is.

It is, and it's not right.

So.

The, the person that we had, uh, that talked about that talked about how

that you actually can recognize, you can have, you can f first off you can.

Yeah, you can.

The, the, the domains that are used for these types of things for command and

control, for, uh, you know, malware.

They look really weird.

They're not normal.

Um, so again, I, I go back to machine learning.

This is not a normal, uh, type request.

Right?

Um, he also talked about looking for communication to new domains, right?

Um, things like that, right?

Um, the, um, um,

But

but I.

is really around like the active persistent threats, right?

It's like.

Yeah.

Yeah.

And really I think the, the thing that you've got to do is you, you just

have to be monitoring your network.

You've got to be looking for behaviors, looking for behavior anomalies.

Uh, you do need an EDR system, uh, to, to watch for that sort of stuff.

But again, as we discussed, I believe in last week's episode, the

EDR is only helpful on the way in.

Yeah.

Right.

Um, the, the, the EDR might, uh, that's, um, endpoint detection and response.

It, depending on where it is, it might detect an a PT

Yeah.

that a PT tries to then, uh,

Yeah.

move outside where it currently is.

it did in the case of Darlene trying to hack the police department, right?

They

Yeah.

software run being like, Hey, you should not be loading this file.

Right.

Right.

Yeah.

Yeah.

But I think the challenge, or one thing I wanted to mention with APTs is, right,

if I look at the all, uh, the Evil Corp exercise right where they hacked

and they still left things behind.

The fact that they're not monitoring the network continually or looking

for these sort of activities

Right.

a little scary because I think I was reading.

Some statistics or some article about ransomware gangs, right?

And it's not like they come in, they attack, they steal some data,

they encrypt data, and then you pay them and then they go away, right?

They might come back into your network again to extort you for more money.

Yeah.

sell that attack, right?

Or that, uh, the initial attack to another ransomware group, right?

So now

Yeah.

you.

And

Yeah.

If you don't make sure that you've closed off all the gates, right,

you're still leaving yourself open.

Yeah.

Agreed.

Agreed.

The episodes we've done with Mike have been enlightening because he really

understands like what the recovery process looks like and how companies recover.

And it's fascinating to see, a lot of times on TV it's like,

oh, they got hit with ransomware, boom, they're back up and running.

Or what?

You imagine it's

Yeah.

it's a quick operation.

But no, these could be weeks

Yeah.

before they really know what was impacted.

And it takes time and people are careful.

Yeah, the, the, the, um, containment and eradication phase

is the single biggest phase.

The actual restoring of servers is the easy part, assuming that you,

you know, backed up all the things.

Um, well, I think that can end our coverage of advanced persistent

threats from, uh, season one, episode eight, AKA 1.7 of Mr.

Robot.

Thanks.

Thanks again, Prasanna.

No, this was a good episode and don't tell me what happens 'cause

I want to see what Mr. Robot is.

Okay.

And, uh, with that, thanks to our listeners.

That is a wrap.

The backup wrap up is written, recorded and produced by me w Curtis Preston.

If you need backup or Dr. Consulting content generation or expert witness

work, check out backup central.com.

You can also find links from my O'Reilly Books on the same website.

Remember, this is an independent podcast and any opinions that you

hear are those of the speaker.

And not necessarily an employer.

Thanks for listening.

I.