You found the backup wrap up your go-to podcast for all things

backup recovery and cyber recovery.

In this episode, we talk about a real nightmare scenario.

It was a hack of an app called Arc, GIS, and it went undetected for 12 months.

That's right an entire year.

The threat group was called Flax Typhoon, and they compromised an arc

GIS server, and turned a legitimate Java extension into a web shell.

And every time the customer backed up their system, they were actually

backing up the malware too.

Uh, we talk about how this happened and why traditional security tools

completely missed it, and what you could do most importantly to prevent

something like this from happening to you.

By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr.

Backup.

And I've been passionate about backup and recovery and now

cyber recovery for over 30 years.

Ever since.

I had to tell my boss that we had no backups of the production

database that we just lost.

I don't want that to happen to you.

That's why I do this.

On this podcast, we turn unappreciated admins into Cyber Recovery Heroes.

This is the backup wrap up.

Welcome to the show.

Hi, I am w Curtis Preston, AKA, Mr. Backup, and I have with me a guy that

started poking around in my software Prasanna Malaiyandi, How's it going?

Prasanna,

Good.

Curtis, how are you doing?

do, doing okay.

You can't just go into my thing and make changes and not tell me.

Well, this is 'cause you complained that I don't do enough stuff.

So I started doing stuff and now you complain the other way.

You just gotta, you just gotta tell me.

I, I mean, we're researching, in this show, we're talking about

software that you use all the time, being used against you.

And I log in and I, there were changes.

I didn't, I didn't know there were changes.

I got scared.

I thought, I, I thought that the, that our show about being hacked had been hacked.

So, so let's, uh, let's, let's jump in this thing, this story.

Uh, and by the way, for our listeners, we're actually gonna try something new.

We're gonna try not to talk so dang much.

Um, our, our shows have typically gone 30 to 40 minutes.

We're gonna try to keep 'em a little shorter, uh, see how that works.

I think the primary problem with that is me.

But, um.

to talk?

I do like to talk.

Yeah.

Hmm.

So, uh, we're gonna talk about this, um, this story of a customer of arc. Arc.

Is it Arc, GISI guess Is that somebody that, that, that would be pronounced?

So, so this is a geographic information service?

Is that, what, is that what that's called?

I've heard about this.

You can use it for like data analytics and other things about

different areas and gather different statistics based off of it,

Yeah.

so it helps you do analysis of a lot of different data

And what it appears is that it was an actual piece of software that was

installed on a server, and it turned out that a group called Flax Typhoon,

That's a new one.

Yeah.

It's, I I hadn't heard of it.

Yeah.

Uh, and they're a, a China based, uh, a PT. Threat group or a PT group, right.

Advanced persistent threat.

Uh, and they had managed to, um, what, what would you call it?

Um, well hack it.

I guess we, we'll just call it.

Well,

Yeah,

did

go ahead.

they hacked it, right?

They found administrator credentials to the software.

They accessed it, and then they basically deployed an extension

Yeah.

Yeah.

J they call it the Java server object Extension.

SOE.

Yeah.

Yeah, and they used that as sort of a backdoor into the system and gained

access into the network and other things.

And it all just looked normal.

And I think one of the big things was, you know how we always talk about,

okay, the best way to recover from ransomware is to have a backup that you

Right, right.

this case, because of how these extensions work, people were

actually backing up the extension.

So if you ever tried to restore from your backups, which is

Yeah, yeah,

restore the extension and therefore the malware.

yeah.

And so basically they turned this SOE into a functioning web shell

that could do whatever they want.

I think my favorite part of the story is that they put a password.

In their hack.

So they had this back door that could do whatever they want, and then

they went and put a password on it.

That, that basically said, only we can use the hack.

exactly.

Um, and, and how long, how long Prasanna were they in this customer's

environment before they were discovered or

months, I wanna say.

12? Months,

Yep.

let that sit with our listeners.

They were in their environment for 12 months.

Now, I just want to say it appears that nothing that none of this, this

was not Arc C, arc GIS was not hacked.

Right?

This was not a vulnerability of Arc G. Arc GIS.

Um, this was.

This was a compromise, uh, but a compromise that was made

possible in my professional opinion based on, uh, user error.

Right?

User, user misconfiguration, uh, and then also, uh, and we're

gonna, we're gonna talk about that.

Um, basically, and, and also it was the reason it was allowed to stay

so long is because of, I think the inherent trust that people put in.

Stuff that they install and they use all the time.

And just to add one more thing to this is the way Arc GIS is deployed

in this environment is you sort of had a public facing thing.

Right.

then they also had an internal arc, GIS instance, which would kind of do all

the work, but the public one would proxy request to the internal one, and so

they also compromised things that way.

So they were able to get access to the internal network because of

that public facing arc, GIS server.

Yeah.

Yeah.

Good times.

Uh, and so what it, what it appears is that they got, uh, they, they

somehow got a password from a, uh, an administrative account.

And then I do think it's funny that it does show in the article that they then.

Um, changed the password of the administrative account, uh,

which, which I find interesting.

Like nobody, does nobody ever log into the admin account,

Like

know?

it up and running, it's like why bother going?

It may not be one of those systems you're constantly checking and monitoring

Yeah, yeah.

Uh, so basically they had, they had this web shell that could do

literally whatever they wanted.

Uh, they poked around the customer's environment.

They tried to go after a couple of different workstations.

It was it workstations.

'cause they wanted to be able to get other credentials and

access to other resources too.

Yeah, yeah.

And you know, and I, I, so the first thing I want to talk about is the,

the, the backup issue that you stated.

So while, while you're right, we do always recommend, uh,

backups of, of everything, right?

I, and, and, and I tell people, even though I do think that you

should be restoring data and apps and, and the os like differently,

uh, just, just back it all up.

Right, just back up everything.

I'd much rather you just back up everything and have wasted space than,

than to selectively select things and, uh, and accidentally miss something important.

But I do think that, uh, if they were trying to selectively restore

this environment, um, it doesn't quite go the way I would think that

you should if you think you have.

If you've suspected an attack, um, at least, at least I know in the upcoming

book, uh, that would be learning ransomware response and Recovery.

We do recommend that for the operating system and for applications, you should

be, you should be restoring this as I make quotes in the air from, uh,

a golden image, not from a backup.

You know, that was taken anytime, anytime recently.

Right.

And, and that, I think if they had done that.

If they had restored it from a golden image, then I think perhaps

I would've addressed this issue.

Having said that, there is, there is that concern that you talked about if

they were actually backing up these, these extensions as something like

extra, then that might have been, they still might have reinfected themselves.

Ex. Yeah.

And also in the article that they, I think we'll post a link in the show description

from this article, but they also mentioned that the only way, so this company

or this customer got attacked, right?

And then they brought in this other company security company

to do the investigation.

I think they're called ReliaQuest.

Yeah.

I rely a quest.

Yeah.

Rely Quest.

And so one of the things they talked about is they actually went through to

try to figure out like which extensions are valid and which ones are not, because

Mm-hmm.

didn't even know like which ones were supposed to be on

this server to start with.

Interesting.

And so I think that becomes a challenge.

Just to what you were saying, Curtis, it's you restore it.

How do you know what was supposed to be there and what wasn't?

It's if you don't know what those are and what seems to be legitimate traffic

Right.

behavior from the application versus what is anomalous.

And, and the closest analogy that I have to, that is, you know,

I use, uh, WordPress, right?

For the website.

And WordPress has extensions.

Right.

And the, and, and it's very easy to install.

Install, and, you know, activate.

It's like, it tastes like two clicks.

Install and activate.

Install and activate.

And it's very common for you to install it, activate, play with it

for a little bit, decide not to use it, and then just leave it there.

Yep.

Bad juju, right?

Because you do not want the problem that they had.

You wanna know what, you wanna know, what your system inventory is.

You wanna know what any, uh, extensions, applications, et cetera,

that are running in your environment.

And ones that are ones that are not.

Actively being used should be either removed, like you

said, and and also patched.

Yes.

Actively patched.

Yeah.

Um, and, um, so they did ultimately get, uh, they did

ultimately get rid of this problem.

I don't know, did they talk in the article?

I didn't see, did they figure out what damage was done?

Uh, they looked to see the spread, but they didn't find

Yeah.

Okay.

Um, they don't, they don't really know, I guess is what that is.

Yeah,

Yeah, yeah.

warn that just given the behavior of, what's it called, flax, typhoon.

Yeah.

Given the behavior of flax typhoon, it's one of these groups that plans

methodically before attacking.

So they were mentioning that they've probably also already figured out what

they're going to do next, and it's just a matter of time before they act.

Hmm.

Yay.

Yay.

so let's talk about what they could have done, uh, differently.

And of course this is this, this article that we're reading is a blog

post by ReliaQuest, and of course one of ReliaQuest is strategies.

Uh, is to use, uh, ReliaQuest, uh, software, right?

Um, one of which is called Ag Agentic ai.

And, and you know, not necessarily that, but I agree with their

recommendation in that.

The problem here was that the, the tools that they were using were

typically, uh, hunting for IOCs.

You want to talk about what that is,

Yeah, this is in indications of compromise,

right?

that help you understand, okay, I was compromised because I might see a log file

or a binary with this sort of signature on it and other things to indicate

that yes, something has attacked me.

Yeah.

And so instead of ho hunting for known bad software, you can hunt

for unknown behavior, right?

So it is, this is this, because the problem is this was

essentially custom software.

Yep.

so, so anything that's hunting for known signatures is not

gonna find anything wrong.

But if they had been watching the activity of what this thing was doing,

then they would've seen these odd, uh, requests and they, they go ahead.

But one thing though that, and I don't know, maybe the customer didn't have

the best tools in place, but they were doing things like creating services on

startups, seeing if it failed, restarting, like all of these things, which I would

assume you would flag regardless as,

Right.

this is some bad behavior.

I should go look at it.

If.

almost sort of like your basics, right?

If you're watching for that sort of thing, right?

A lot of people, especially people that are not, not cyber,

Yeah.

don't know, savvy, not terribly cyber savvy, but they talked about that they

had the ability to do, uh, automated response playbooks and at the first time

the this command started running weird.

Who am I?

Yeah.

like who runs a who am I, uh, command?

Right.

As soon as they did that, they said they would've quarantined that server.

It wouldn't have been able to talk to other people.

Also, as soon as it started talking to C two servers, what are C two servers?

Uh, Prasanna.

Command and control servers.

Yeah.

Yeah.

think of it as servers out there on the internet that these malicious actors

control, that send commands to these end points telling it what to do.

Yeah.

Yeah.

Yeah.

And so as soon as it saw that it was talking to, uh, uh, command and

control servers, it would've, it would've basically blocked that ip.

So there, there, so basically, and we've talked about this, I recommend

the, the, you know, it, it, it's, it's.

AI is not the silver bullet for everything, but this is one thing where AI

and machine learning can be very helpful, where you can watch how applications

typically behave and then when they start doing stuff that they're not normally

doing, uh, you can flag it and you can go, maybe you do auto, maybe you do

it automatically, but maybe you don't.

Uh, but, but at least you flag it.

Right.

And, and at least they wouldn't have been there for a year.

Can I add something to

Yeah, sure.

Uh, not directly related to what ReliaQuest talked about, but I think

periodically you should be going and doing an inventory of your systems.

Yeah.

And figuring out what's running.

Do I have just like patch management, right?

Do I

Right.

patches on all the systems?

Are people using these extensions?

Do the right people have access to the systems that need access?

Are there people I should be kicking off?

Right?

Yeah.

these other things should be sort of like

like

hygiene.

Yeah, cyber hygiene.

Thank

Yeah.

Yeah.

people should be doing that could have prevented some of these things.

Like

I may,

is a long time.

I may or may not have been recently editing a. A, a chapter that

used to phrase cyber hygiene.

So I had it right.

I had it right at the ready.

So, uh, so they've got an action plan here of four things.

And I, you know, I, I couldn't, couldn't agree with, couldn't

agree more with, with all of them.

Right.

Uh, although.

Well, they just have, they just, they use bigger words than I would use.

So they talk about audit and hardening public facing applications.

So if you have an application that is talking to the internet, uh, this

is one that you really need to be, uh, locking down that server and

that application as much as you can, more so than a server that simply

runs inside your IT environment.

And today, what isn't a public facing application, that's what I want to say,

well, I think in this case they're also talking about things that are

visible from the internet rather than things that might need internet access.

Well, what I'm saying though is like everybody uses SaaS apps, so like all

SaaS apps are public facing applications.

That's what I'm saying.

Like the, the days of we've got, we've got, you know, three apps in

the cloud and or, or three apps that are public facing and the rest of our

apps are just inside the data center.

It's like, what's a data center?

Yeah,

Right.

So I'm saying everything seems to be public facing, but go ahead.

but yes.

SaaS apps, I agree.

But I would say from a customer perspective, the SaaS

apps are not their problem.

But I'm gonna have, I'm gonna have

vendor's problem minus basic access controls and other things they should be

ag Agreed.

Agreed.

but there's very little that they can control in a SaaS app's case

A Yes.

Agreed.

I I guess what they're saying is if you do have actual apps running in

your data center, uh, that are public facing, then, then you really should

be, um, you know, auditing and harting.

Yes.

and blah, blah, blah.

Right.

Yeah.

They, we talked about it already, but they talked about moving

beyond IOC based detection.

You need to be looking at behavioral based detection at this point.

Right.

I

And, and

you.

yeah, sure

Does your book cover any of this?

it does.

Okay.

Yeah.

Uh, I, I think so.

Hmm.

I.

It's all up in my head, like, you know, going through all this stuff.

Yeah.

I mean, we talked about, um, we do talk about, um, well, I'll just say this.

The book is focused on an assumed breach standpoint.

The focus of the book is not.

How to stop all ransomware, right?

The focus of the book is you're probably gonna get ransomware,

so here's how to stop it.

There is one chapter in the book that says, look, you

really need to do these things.

And we do, we do.

In that chapter, and I, it was literally that chapter I was

just editing in that chapter.

Um.

We do talk about like the things that you absolutely have to do,

and one of them is this next one, which is strong credential hygiene.

Right.

Uh, I pulled, I just pulled a, a recent copy of this,

um, of this, um, the, from a,

not good.

yeah.

So it, it's uh, from a company called HI Systems and they have a password.

Length guessing time table, right?

And, um, the, if you have a password length of, um, if you just have letters,

um, you know that they've got like a password length of 12 is 27,000 years.

But, uh, the, the, the, the key here is that length, length

is better than complexity.

Right.

Um, so like for example, if you've got numbers, upper and lowercase

letters and symbols, and your password length is six characters long.

So it's says complex as it could possibly be, and it's six characters long.

How long do you think it takes to guess that

One year

two weeks?

If it's seven characters long, it's two years, right?

At eight now it's 164 years.

Right?

So the, the real key is like,

can

length is better than complexity.

So, yeah.

So if, if, here's, here's the beautiful thing.

If all I do is have a long password,

Yeah.

12 character password, and all I do is use lowercase letters.

So 12 character password.

Now mind you, over there we had a six character password, but it was

as complex as it could possibly be.

That was two weeks.

I'm gonna have a 12 character password and it's lowercase letters only.

Mm-hmm.

long do you think it takes?

Uh, 172 years.

27,000 years.

So what have we learned?

Length is better than complexity, right?

yeah.

So, um,

a single word.

It could be a sentence,

yes, it can, yes.

Like Prasanna is awesome.

That's one.

But, but yeah.

But yeah, so that's the thing is like you, you need, so they had a,

they had a guessable password, or they stole password, and then what?

So they got a password.

What did they also not have?

If they had had this thing, they would've stopped this password that they guessed.

Yes.

Thank you.

MFA, please, for the love of God, everybody, can you please, if you're

still using passwords, please put MFA on everything that matters.

How hard is this?

It's, it is just killing me.

Right?

Um, you know, look at PAs keys.

If you can't do PAs keys, at least put on an MFA and if your ap, if,

if your app, the response from your support from your app is like, what?

It's MFA time to get a new app, right?

But, um, and it, so, yeah.

So if, if they, if you got password management and MFA,

uh, then this would've, this, if they had just turned on MFA, that

would've, uh, solved this problem.

And then also, yes, patch management, right?

Um, potentially if the, if this, uh, SOE was part of the overall package.

Perhaps if they had updated, uh, the, the package, it would've

actually overwritten the, the SOE.

Um, don't, don't actually know that much about that thing, but,

Yeah,

but, um, yeah, I, I don't know.

So basically, uh, get better passwords, uh, turn MFA on for

anything that matters and investigate.

Those are the things you, you have to do.

Right?

Investigate behavioral based detection that IOC based detection is.

So, uh, last year.

Yeah.

And well, and I think the other thing that I took away from the article is not just.

Don't expect that someone is just gonna have a malware EXE file running somewhere.

Yeah,

could be part of your normal software stack and tools that

you have out there that look

absolutely.

Yeah.

that they've compromised, so,

And don't trust them.

Right.

Just, uh, you should be watching to see what they normally do.

And then see when they do weird things

Yep.

and when they do weird things, I go, whoop, whoop.

But off the clocks on alert.

All right, well this has been fun, Prasanna.

Likewise, Curtis, although I do miss your stories,

I just didn't have it.

I just didn't have any this time.

We'll see, you know, we're working on this new format.

let us know what you think.

this was good.

It, it was a really complex topic to cover in, in a shorter format.

It would've been a lot easier to talk about this for 40 minutes, but I'm trying

listeners, if you like this

Yeah.

don't like, leave us a comment on your favorite pod catcher.

We have YouTube videos.

You can look at our gorgeous faces,

Yeah.

YouTube.

So leave us a comment there.

We love to hear from you, but let us know what you think of this.

One of us has a gorgeous face.

The other has long hair.

All right.

Thank you very much, Prasanna.

It's been fun.

Likewise, Curtis, you have a good one.

And thanks listeners.

You know you're, why we do this?

That is a wrap.