Speaker:

W. Curtis Preston: Spoiler alert, the company in this episode ceased to exist

Speaker:

due to what happened in this story.

Speaker:

Disasters happen, and since the cloud is just someone else's data

Speaker:

center, they sometimes happen in the cloud, and sometimes they

Speaker:

take companies along with them.

Speaker:

This episode is the first in a series called Cloud Disasters.

Speaker:

Each episode tells the real story.

Speaker:

Of a company who failed to back up their cloud data and suffered as a result.

Speaker:

And these aren't podunk cloud vendors either.

Speaker:

Every vendor covered in this story is a major vendor.

Speaker:

I'm not kidding around when I say you should back up your cloud data.

Speaker:

I don't care that there are those who think I'm just pedaling fud.

Speaker:

Newsflash, the only reason we've ever backed up anything is because

Speaker:

of the fear of losing it and the uncertainty and doubt we have in

Speaker:

the system that we're backing up.

Speaker:

Fans of the podcast know, the whole reason that I became a backup specialist is that

Speaker:

in 1993, I was unable to recover a very important Oracle database for my employer.

Speaker:

I didn't want that to happen to anyone else.

Speaker:

So I've dedicated myself to helping others protect themselves

Speaker:

from feeling that awful feeling.

Speaker:

This show is aimed at you, the unappreciated backup admin, and we want

Speaker:

to turn you into a cyber recovery hero.

Speaker:

This is the backup wrap up.

Speaker:

Welcome to the show.

Speaker:

I'm your host, w Curtis Preston, and I have with me my continued and necessary

Speaker:

Tesla consultant Prasanna Malaiyandi.

Speaker:

How's it going Prasanna

Speaker:

persona

Prasanna Malaiyandi:

Pana.

Prasanna Malaiyandi:

I am doing well Curtis, and how have you been?

Prasanna Malaiyandi:

W. Curtis Preston: I, well, as you know, I have been.

Prasanna Malaiyandi:

Fiddling.

Prasanna Malaiyandi:

Uh, why am I not surprised?

Prasanna Malaiyandi:

W. Curtis Preston: I have been, fiddling with the automations of my Tesla now.

Prasanna Malaiyandi:

Tesla's come in, I'm gonna say two battery flavors, right?

Prasanna Malaiyandi:

There's NMC and LFP.

Prasanna Malaiyandi:

Nickel Manganese Cobalt and Lithium Iron Phosphate.

Prasanna Malaiyandi:

And I have the latter.

Prasanna Malaiyandi:

And the latter is not supposed to be subject to the same don't

Prasanna Malaiyandi:

charge it to a hundred percent unless you need it right away.

Prasanna Malaiyandi:

Issue.

Prasanna Malaiyandi:

Uh,

Prasanna Malaiyandi:

In fact tell

Prasanna Malaiyandi:

you to charge it to a hundred

Prasanna Malaiyandi:

W. Curtis Preston: I do tell you to charge it to a hundred

Prasanna Malaiyandi:

percent at least once a week.

Prasanna Malaiyandi:

Um, but I was charging it to a hundred percent every day.

Prasanna Malaiyandi:

And I was thinking that I, that even though it doesn't have necessarily

Prasanna Malaiyandi:

the same issues, I could still get better battery life by not charging to

Prasanna Malaiyandi:

a hundred percent unless I needed to.

Prasanna Malaiyandi:

Uh, which, uh, for me is not all the time.

Prasanna Malaiyandi:

And so, but I'm also very absent-minded, and so I couldn't like lower the

Prasanna Malaiyandi:

percentage and then remember to, to, to, to raise the percentage later.

Prasanna Malaiyandi:

And so, uh, I found this wonderful app called Tessie, and I've been obsessing

Prasanna Malaiyandi:

over it for about a week or two.

Prasanna Malaiyandi:

A week.

Prasanna Malaiyandi:

It's been nine days.

Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

Prasanna Malaiyandi:

Yeah.

Prasanna Malaiyandi:

I will say if there's any listeners that are a Tesla owner, I.

Prasanna Malaiyandi:

Uh, you should just get Tessie, uh, T-E-S-S-I-E in the app store.

Prasanna Malaiyandi:

I bought it for the automation.

Prasanna Malaiyandi:

What I also got was this immense amount of analytics and reporting

Prasanna Malaiyandi:

and, um, all this great, great stuff.

Prasanna Malaiyandi:

And also reminders.

Prasanna Malaiyandi:

That's another, speaking of being absent-minded, it will tell me, hey.

Prasanna Malaiyandi:

Idiot, you're home and your car's not plugged in.

Prasanna Malaiyandi:

Yeah, because how many times has that

Prasanna Malaiyandi:

happened to you, Curtis, that you've come home and you're like,

Prasanna Malaiyandi:

W. Curtis Preston: you know, a couple, uh, enough that it was annoying.

Prasanna Malaiyandi:

Uh, it, I will say it, nothing is more annoying than, you know, basically

Prasanna Malaiyandi:

driving your car down to, you know, the electrical equivalent of fumes.

Prasanna Malaiyandi:

And then, um, and then getting up in the morning and going,

Prasanna Malaiyandi:

okay, I'm ready to drive today.

Prasanna Malaiyandi:

And you're like, oh, crap, I gotta go to the.

Prasanna Malaiyandi:

I gotta go the supercharger for 20 minutes before I could do anything else.

Prasanna Malaiyandi:

Um, yeah, so it's got, you know, it's got that the notifications, it

Prasanna Malaiyandi:

reminds me to, to rotate my tires.

Prasanna Malaiyandi:

And also gives me analytics about my driving and my efficiency and, yeah.

Prasanna Malaiyandi:

So, yeah.

Prasanna Malaiyandi:

So happy, happy, happy, happy, happy.

Prasanna Malaiyandi:

Um, but you know, who's not happy?

Prasanna Malaiyandi:

The people that are gonna be in this new series, they're not happy.

Prasanna Malaiyandi:

And this is a new series.

Prasanna Malaiyandi:

One of the things that you've heard us say is what Bana.

Prasanna Malaiyandi:

Back up the cloud.

Prasanna Malaiyandi:

W. Curtis Preston: Yes, backup up the cloud.

Prasanna Malaiyandi:

The cloud is not magic.

Prasanna Malaiyandi:

There is no such thing as the cloud.

Prasanna Malaiyandi:

It's just somebody else's computer.

Prasanna Malaiyandi:

Uh uh, the cloud doesn't magically back up itself.

Prasanna Malaiyandi:

Despite what you may have been told the cloud there are parts of the cloud where.

Prasanna Malaiyandi:

People generally agree with me.

Prasanna Malaiyandi:

I, I think probably the best example would be something like AWS, you know, like EC2.

Prasanna Malaiyandi:

They're like, yes, we know EC2 needs to be backed up.

Prasanna Malaiyandi:

Um, and, and then we just argue a little bit over how that's going to be done.

Prasanna Malaiyandi:

But when we get to the extreme it, the other end of that, we get the

Prasanna Malaiyandi:

SaaS world, we get the Microsoft 365 lovers who say, oh, this, that,

Prasanna Malaiyandi:

this doesn't need to be backed up.

Prasanna Malaiyandi:

you know, or G

Prasanna Malaiyandi:

Suite and, we have gone back over the last, uh, several years and selected, um,

Prasanna Malaiyandi:

a number of, you know, poor victims of.

Prasanna Malaiyandi:

This belief and, um, I, I can think of no better, uh, story to start this

Prasanna Malaiyandi:

out than code spaces because it, I, I think it was kind of the first,

Prasanna Malaiyandi:

it was probably the first that was sort of well

Prasanna Malaiyandi:

publicized as well as having very dire consequences for not backing up the cloud.

Prasanna Malaiyandi:

I.

Prasanna Malaiyandi:

W. Curtis Preston: Right.

Prasanna Malaiyandi:

And it, it's interesting from a timing perspective, it happened in 2014.

Prasanna Malaiyandi:

Which to me is the year of the beginning of the massive level of ransomware.

Prasanna Malaiyandi:

Now, I know ransomware actually goes way back longer.

Prasanna Malaiyandi:

You know, much longer be before that.

Prasanna Malaiyandi:

But to me, 2014 is really when I started seeing ransomware kind of everywhere.

Prasanna Malaiyandi:

And this was technically a ransomware attack, not in the traditional sense

Prasanna Malaiyandi:

that we think of today, but, but it was, so Code spaces.com was a site, and this

Prasanna Malaiyandi:

is the, the irony of all ri ironies.

Prasanna Malaiyandi:

And, and I will say that unlike Alanis Morissette, I actually

Prasanna Malaiyandi:

know what the word ironic means.

Prasanna Malaiyandi:

Um, I love the song.

Prasanna Malaiyandi:

It's just, there's so many things to that song that are not in any way ironic.

Prasanna Malaiyandi:

They just suck.

Prasanna Malaiyandi:

Um, code spaces.com was a site to store your code.

Prasanna Malaiyandi:

It was like, um.

Prasanna Malaiyandi:

GitHub.

Prasanna Malaiyandi:

W. Curtis Preston: It was like, it was like a GitHub and they had

Prasanna Malaiyandi:

many, many customers and it was a safe space to store your code.

Prasanna Malaiyandi:

Hence the name code spaces.com.

Prasanna Malaiyandi:

Yeah, so basically like you mentioned, it was a place

Prasanna Malaiyandi:

companies could store their code and this was way back in the day.

Prasanna Malaiyandi:

And so it's like, hey, if you just have your own code sitting locally

Prasanna Malaiyandi:

on your system, because not everyone was comfortable with the cloud, they

Prasanna Malaiyandi:

offered a service that allows you to store your code there and keep it safe.

Prasanna Malaiyandi:

And now according to a cash version of their website, because of course

Prasanna Malaiyandi:

you can't find anything anymore about them, uh, they had over 200 customers.

Prasanna Malaiyandi:

A week or 200 companies a week using their service, which isn't

Prasanna Malaiyandi:

like small beans, right?

Prasanna Malaiyandi:

And yes, it's not like the thousands of millions, but there are 200

Prasanna Malaiyandi:

customers who now no longer have access to their code anymore because

Prasanna Malaiyandi:

of what happened in code spaces.

Prasanna Malaiyandi:

The other interesting thing is according to their websites, and I will quote

Prasanna Malaiyandi:

backing up, data is one thing, but is meaningless without a recovery plan.

Prasanna Malaiyandi:

Not only that, a recovery plan and one that is well practiced and

Prasanna Malaiyandi:

proven to work time and time again.

Prasanna Malaiyandi:

Code Spaces has a full recovery plan that has been proven to

Prasanna Malaiyandi:

work and is in fact practiced.

Prasanna Malaiyandi:

W. Curtis Preston: That

Prasanna Malaiyandi:

do you think about that, Curtis?

Prasanna Malaiyandi:

W. Curtis Preston: That sounds really good.

Prasanna Malaiyandi:

Um, you know, based on what we know happened, clearly they didn't test for

Prasanna Malaiyandi:

all scenarios.

Prasanna Malaiyandi:

W. Curtis Preston: Right.

Prasanna Malaiyandi:

And they specifically didn't test for cyber attack.

Prasanna Malaiyandi:

Um, but.

Prasanna Malaiyandi:

the time could you fault them really?

Prasanna Malaiyandi:

W. Curtis Preston: I don't know.

Prasanna Malaiyandi:

Well, they had poor backup design.

Prasanna Malaiyandi:

Just, you know what, what we're

Prasanna Malaiyandi:

gonna get to, what we're gonna find out.

Prasanna Malaiyandi:

They had full backup.

Prasanna Malaiyandi:

They had poor backup design.

Prasanna Malaiyandi:

They failed to follow what persona?

Prasanna Malaiyandi:

The 3, 2, 1 rule.

Prasanna Malaiyandi:

You know, we haven't talked about this in a long, long time.

Prasanna Malaiyandi:

Do you wanna quickly mention it to some of our

Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

Prasanna Malaiyandi:

So the 3, 2, 1 role, and by the way, many companies have said

Prasanna Malaiyandi:

it's gotta be more than 3, 2, 1.

Prasanna Malaiyandi:

Yes, I agree.

Prasanna Malaiyandi:

Right?

Prasanna Malaiyandi:

These days it has to be more than 3, 2, 1, but if it's not 3, 2, 1, there's no point

Prasanna Malaiyandi:

in talking about the other ones, right?

Prasanna Malaiyandi:

Having at least three versions on two different media, and the, the,

Prasanna Malaiyandi:

the idea here is on, on things that are subject to different.

Prasanna Malaiyandi:

Risk profiles, right?

Prasanna Malaiyandi:

Maybe it's disco and tape, maybe it's, uh, on-Prem and off-Prem.

Prasanna Malaiyandi:

Maybe it's, you know, on-prem and Cloud.

Prasanna Malaiyandi:

Maybe it's, um, you know, a different region, et cetera.

Prasanna Malaiyandi:

And then one three, the one is make sure that there's something off site.

Prasanna Malaiyandi:

Um, they had neither the two nor the one,

Prasanna Malaiyandi:

but we're gonna get to that in a minute.

Prasanna Malaiyandi:

and, and for those listeners who wanna know more,

Prasanna Malaiyandi:

we actually had an episode with the person who coined the term 3, 2, 1, who

Prasanna Malaiyandi:

comes from digital photography, in fact.

Prasanna Malaiyandi:

And so you should go take a listen to that episode.

Prasanna Malaiyandi:

We'll

Prasanna Malaiyandi:

W. Curtis Preston: Yeah, we'll put.

Prasanna Malaiyandi:

Yeah, we'll put a show notes.

Prasanna Malaiyandi:

Peter Krogh.

Prasanna Malaiyandi:

Um, great guy.

Prasanna Malaiyandi:

So what happened?

Prasanna Malaiyandi:

Well, we have that a hacker gained privilege credentials that we still

Prasanna Malaiyandi:

don't know how that happened, right?

Prasanna Malaiyandi:

They're saying probably through phishing or possibly through, uh,

Prasanna Malaiyandi:

stored EC2 access keys in a public code

Prasanna Malaiyandi:

repository.

Prasanna Malaiyandi:

Which happened a lot back then and still does today.

Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

Prasanna Malaiyandi:

Yeah.

Prasanna Malaiyandi:

The, um, there was a, uh, security researcher at Tripwire that said, that

Prasanna Malaiyandi:

this is a problem for people how to, how to, manage authentication codes like this.

Prasanna Malaiyandi:

And, uh, they said they, they had seen thousands of EC2 accounts

Prasanna Malaiyandi:

abused after storing their EC2 keys in public code repositories.

Prasanna Malaiyandi:

Ouch.

Prasanna Malaiyandi:

Right.

Prasanna Malaiyandi:

Um, but we don't, so we don't know.

Prasanna Malaiyandi:

Someone

Prasanna Malaiyandi:

got access.

Prasanna Malaiyandi:

Yeah.

Prasanna Malaiyandi:

Would it be ironic if Code Spaces was using code spaces

Prasanna Malaiyandi:

to store their code and they left a public repository with their EC2 key?

Prasanna Malaiyandi:

W. Curtis Preston: That wouldn't indeed be ironic.

Prasanna Malaiyandi:

Um, but yeah, so we don't know exactly how this hacker, uh, this,

Prasanna Malaiyandi:

you know, bad actor got access to the, to the environment, but they did.

Prasanna Malaiyandi:

And the first thing that they did was they started a DDoS attack.

Prasanna Malaiyandi:

Uh, what is a

Prasanna Malaiyandi:

DDoS attack persona?

Prasanna Malaiyandi:

This is a DI distributed denial of service, so

Prasanna Malaiyandi:

you have a bunch of servers outside hammering various servers at a company.

Prasanna Malaiyandi:

Causing it, flooding it with a lot of traffic, which then

Prasanna Malaiyandi:

causes it to stop responding.

Prasanna Malaiyandi:

So if you're code spaces, someone did a DDoS attack on you now, you wouldn't

Prasanna Malaiyandi:

be able to serve and function as a service to those 200 paying customers.

Prasanna Malaiyandi:

W. Curtis Preston: Yeah, the really interesting thing about this is that the

Prasanna Malaiyandi:

DDoS attack was apparently subterfuge.

Prasanna Malaiyandi:

Right.

Prasanna Malaiyandi:

Um, you know, look over here

Prasanna Malaiyandi:

while I, nothing up but

Prasanna Malaiyandi:

which I think is still common today, right?

Prasanna Malaiyandi:

There are still a lot of companies who they're trying to hide their tracks

Prasanna Malaiyandi:

and they're like, Hey, everyone's gonna fight, and DDoS were common.

Prasanna Malaiyandi:

And so they have a plan in place and everyone's scrambling there where you're

Prasanna Malaiyandi:

like, Hey, look at my left hand while I'm doing something with my right.

Prasanna Malaiyandi:

W. Curtis Preston: And then I, I think the, I don't know if interesting

Prasanna Malaiyandi:

is right, but the hacker left contact details for themselves in

Prasanna Malaiyandi:

the customer's Amazon dashboard.

Prasanna Malaiyandi:

Right.

Prasanna Malaiyandi:

Uh, this was, this was, I, I think, I think maybe we haven't mentioned it.

Prasanna Malaiyandi:

This was an AWS customer.

Prasanna Malaiyandi:

And this is why we call it ransomware, right?

Prasanna Malaiyandi:

Even though it's not your traditional ransomware like we

Prasanna Malaiyandi:

think today, it more or less is.

Prasanna Malaiyandi:

W. Curtis Preston: Right.

Prasanna Malaiyandi:

Yeah.

Prasanna Malaiyandi:

So then, uh, I'll quote from there.

Prasanna Malaiyandi:

There was a page, by the way, there was a page.

Prasanna Malaiyandi:

Uh, basically code spaces.com died after this, and it became a, they,

Prasanna Malaiyandi:

they replaced it with just one page that says, here's what happened.

Prasanna Malaiyandi:

And we're, I'm gonna read quotes from that page that are no longer

Prasanna Malaiyandi:

available because they sold the domain.

Prasanna Malaiyandi:

Um, upon realizing that somebody had access to our control panel, we started

Prasanna Malaiyandi:

to investigate how access had been gained and what access that person

Prasanna Malaiyandi:

had, uh, to the data in our systems.

Prasanna Malaiyandi:

It became clear that so far no machine had access had been achieved due to the

Prasanna Malaiyandi:

intruder not having our private keys.

Prasanna Malaiyandi:

Um, that's what they thought, right?

Prasanna Malaiyandi:

Um, but things

Prasanna Malaiyandi:

turned, uh, yeah, turned ugly.

Prasanna Malaiyandi:

Uh, why don't you talk, talk about

Prasanna Malaiyandi:

the next one there.

Prasanna Malaiyandi:

So what was the response?

Prasanna Malaiyandi:

So Code Spaces did a smart thing and one of their first response.

Prasanna Malaiyandi:

Actions was to change all of its EC2 passwords, but quickly, code

Prasanna Malaiyandi:

spaces discovered that the attacker had created backup logins, which any

Prasanna Malaiyandi:

sensible person's going to do, right?

Prasanna Malaiyandi:

You're never gonna say, Hey, I'm only gonna have one admin in my environment.

Prasanna Malaiyandi:

And so the attacker was able to create all these backup logins, and so now they

Prasanna Malaiyandi:

were able to just go back into the system.

Prasanna Malaiyandi:

And continue doing the attacks.

Prasanna Malaiyandi:

And once they realized that Code Spaces was trying to actually recover and take

Prasanna Malaiyandi:

control from the attacker, the attacker then started to go and just delete things

Prasanna Malaiyandi:

from the control panel because they had super user access at that point.

Prasanna Malaiyandi:

Right.

Prasanna Malaiyandi:

So they could do whatever they wanted.

Prasanna Malaiyandi:

W. Curtis Preston: Yeah, somewhere in here, in the version of the story

Prasanna Malaiyandi:

that I have, I remember there was a d, there was a, there was an attempt

Prasanna Malaiyandi:

at a ransom, basically give us this amount of money, or, or, or we're

Prasanna Malaiyandi:

gonna, you know, do bad things.

Prasanna Malaiyandi:

Um, but the, but you know, they, they obviously didn't want to pay the ransom.

Prasanna Malaiyandi:

And then, uh, then what

Prasanna Malaiyandi:

happened?

Prasanna Malaiyandi:

And so finally code spaces got their control

Prasanna Malaiyandi:

panel access back, but not before.

Prasanna Malaiyandi:

The attacker had caused quite a bit of damage, so the attacker had gone

Prasanna Malaiyandi:

removed all the EBS syn snapshots, S3 buckets, all the amis, which are the

Prasanna Malaiyandi:

Amazon machine instances, some EBS instances and several machine instances.

Prasanna Malaiyandi:

And this is a quote from the same webpage that Curtis was talking about.

Prasanna Malaiyandi:

In summary, most of our data backups, machine configurations

Prasanna Malaiyandi:

and offsite backups were either partially or completely deleted.

Prasanna Malaiyandi:

W. Curtis Preston: What I, I, you know, I did.

Prasanna Malaiyandi:

I never noticed that phrase before.

Prasanna Malaiyandi:

And they said And offsite backups.

Prasanna Malaiyandi:

What?

Prasanna Malaiyandi:

Offsite backups.

Prasanna Malaiyandi:

What I mean, what be, because the hacker only had access to

Prasanna Malaiyandi:

the one account.

Prasanna Malaiyandi:

I am guessing that they replicated

Prasanna Malaiyandi:

to another AWS region within the

Prasanna Malaiyandi:

W. Curtis Preston: Okay.

Prasanna Malaiyandi:

Okay.

Prasanna Malaiyandi:

Another zone within the

Prasanna Malaiyandi:

same account.

Prasanna Malaiyandi:

All right.

Prasanna Malaiyandi:

Or or another region.

Prasanna Malaiyandi:

Yeah,

Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

Prasanna Malaiyandi:

yeah, But still within the

Prasanna Malaiyandi:

same account

Prasanna Malaiyandi:

Possibly within the same account.

Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

Prasanna Malaiyandi:

Um, yeah, yeah, possibly you're right.

Prasanna Malaiyandi:

We don't know for sure, but if it wasn't the same account, then the hacker had

Prasanna Malaiyandi:

to gain access to multiple accounts.

Prasanna Malaiyandi:

Right.

Prasanna Malaiyandi:

Yeah.

Prasanna Malaiyandi:

Yep.

Prasanna Malaiyandi:

W. Curtis Preston: Um, yeah.

Prasanna Malaiyandi:

So basically this is the equivalent of blowing up somebody's data center, right?

Prasanna Malaiyandi:

Um, because basically they just.

Prasanna Malaiyandi:

In a matter of a few keystrokes, they just deleted essentially everything, right?

Prasanna Malaiyandi:

Everything that mattered, or enough things that mattered that they, um,

Prasanna Malaiyandi:

you know, uh, took out the company.

Prasanna Malaiyandi:

One of the things that I don't know if you found any

Prasanna Malaiyandi:

information about Curtis is did they ever reach out to law enforcement

Prasanna Malaiyandi:

or even AWS's security operations to be like, Hey, I have this issue.

Prasanna Malaiyandi:

Can you help me?

Prasanna Malaiyandi:

W. Curtis Preston: Yeah, we don't, I, I, I'm going to.

Prasanna Malaiyandi:

Assume that once you know, the feces hit the rotary oscillator,

Prasanna Malaiyandi:

I'm sure they called AWS.

Prasanna Malaiyandi:

I mean, of course they called AWS, but what we know is that

Prasanna Malaiyandi:

right, if you know AWS isn't magic, and if you didn't follow the architecture

Prasanna Malaiyandi:

and do the things that you were supposed to do, A-W-A-W-S can't

Prasanna Malaiyandi:

undo that.

Prasanna Malaiyandi:

Yeah, well, I was just wondering before it changed, its EC2

Prasanna Malaiyandi:

passwords, if they had reached out to AWS, if they could have helped them in some

Prasanna Malaiyandi:

way or been like, Hey, here are the best practices for locking down your account,

Prasanna Malaiyandi:

W. Curtis Preston: Yeah, I'm, I'm gonna guess based on how they responded

Prasanna Malaiyandi:

that they did not do that right.

Prasanna Malaiyandi:

Um, I don't think their response was the best thing they

Prasanna Malaiyandi:

could have done at the time.

Prasanna Malaiyandi:

Um, so then we have, um.

Prasanna Malaiyandi:

So here's again, I'm reading from their co their, from their quote

Prasanna Malaiyandi:

here, code spaces will not be able to operate beyond this point.

Prasanna Malaiyandi:

The cost of resolving this issue to date and the expected cost of refunding.

Prasanna Malaiyandi:

Customers who have been left without the service they paid for, we'll

Prasanna Malaiyandi:

put code spaces in an irreversible position, both financially and

Prasanna Malaiyandi:

in terms of ongoing credibility.

Prasanna Malaiyandi:

No kidding.

Prasanna Malaiyandi:

Um, as such, at this point.

Prasanna Malaiyandi:

We have no alternative but to cease trading and concentrate on supporting

Prasanna Malaiyandi:

our affected customers and exporting any remaining data they have left with us.

Prasanna Malaiyandi:

Ouch.

Prasanna Malaiyandi:

W. Curtis Preston: That is, um, that's a tough one.

Prasanna Malaiyandi:

So basically, you know, hacker gets in Hacker, uh, you know.

Prasanna Malaiyandi:

Offers are ransom.

Prasanna Malaiyandi:

They try to, instead of paying the ransom, they try to lock

Prasanna Malaiyandi:

the hacker out unsuccessfully,

Prasanna Malaiyandi:

Yep.

Prasanna Malaiyandi:

W. Curtis Preston: and then the hacker deletes the company, the uh, right.

Prasanna Malaiyandi:

I mean, they deleted basically everything, you know, as much as they could get

Prasanna Malaiyandi:

access to in that, you know, in that

Prasanna Malaiyandi:

account.

Prasanna Malaiyandi:

That's crazy.

Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

Prasanna Malaiyandi:

Yeah.

Prasanna Malaiyandi:

This is one of those where it's like, I, I always used this story to

Prasanna Malaiyandi:

recommend backup design for the cloud.

Prasanna Malaiyandi:

Even though I don't think I've ever encountered someone who says,

Prasanna Malaiyandi:

well, I don't need to back up EC2, I don't think I've ever heard that.

Prasanna Malaiyandi:

Yeah.

Prasanna Malaiyandi:

W. Curtis Preston: I have very commonly found people whose backup design was very

Prasanna Malaiyandi:

similar to this company's backup design.

Prasanna Malaiyandi:

And, um, so, well let me ask you this.

Prasanna Malaiyandi:

What do you think, um, what could they have done differently to stop this?

Prasanna Malaiyandi:

To prevent this from happening.

Prasanna Malaiyandi:

I would say the first could have been take your

Prasanna Malaiyandi:

backups to a different account.

Prasanna Malaiyandi:

In a different region, or even in the same region, that would've

Prasanna Malaiyandi:

at least kept your data safe.

Prasanna Malaiyandi:

W. Curtis Preston: Yeah, I, I, yeah, I, I completely agree with you.

Prasanna Malaiyandi:

I would probably just say as long as you, as long as you're, I, I guess what.

Prasanna Malaiyandi:

I'm, I'm trying to, I'm factor, I'm thinking in my head, like,

Prasanna Malaiyandi:

from a cost perspective, does it cost extra to send to another

Prasanna Malaiyandi:

region?

Prasanna Malaiyandi:

Is that,

Prasanna Malaiyandi:

Prasanna Malaiyandi: Yeah, usually it does.

Prasanna Malaiyandi:

Yep.

Prasanna Malaiyandi:

W. Curtis Preston: Okay.

Prasanna Malaiyandi:

So

Prasanna Malaiyandi:

But they already have an offsite backups,

Prasanna Malaiyandi:

right?

Prasanna Malaiyandi:

So.

Prasanna Malaiyandi:

W. Curtis Preston: right?

Prasanna Malaiyandi:

Yeah.

Prasanna Malaiyandi:

So.

Prasanna Malaiyandi:

Um, I, I think that your backup should be copied to another

Prasanna Malaiyandi:

region and another account.

Prasanna Malaiyandi:

I, I actually, and, and, and, and I'll say that, you know, my, my

Prasanna Malaiyandi:

opinion is somewhat peppered by having worked for a company that does this.

Prasanna Malaiyandi:

But there are companies that will backup your cloud data and then get it out

Prasanna Malaiyandi:

of the cloud into their cloud, and I think that's as secure as it can be.

Prasanna Malaiyandi:

Right.

Prasanna Malaiyandi:

And I think that it should be then stored in some type of immutable type offering.

Prasanna Malaiyandi:

Um, basically get it, get, get it out of the region for security against bad things

Prasanna Malaiyandi:

that might happen that aren't hackers.

Prasanna Malaiyandi:

And then get it out of the account to secure it against hackers and the.

Prasanna Malaiyandi:

If that costs you money, figure out a way to do that that costs

Prasanna Malaiyandi:

you as little as possible.

Prasanna Malaiyandi:

And, and, and I do think the companies that can back up, take the, the

Prasanna Malaiyandi:

incremental data and then maybe de-dupe it before they pull it out.

Prasanna Malaiyandi:

If they can do that, you can minimize the egress cost of moving it out.

Prasanna Malaiyandi:

Um, the, the other thing, um.

Prasanna Malaiyandi:

Yeah, so, so they didn't have it in a different account.

Prasanna Malaiyandi:

They didn't have object lock turned on.

Prasanna Malaiyandi:

Um,

Prasanna Malaiyandi:

Object lock did not exist back

Prasanna Malaiyandi:

then.

Prasanna Malaiyandi:

By the way.

Prasanna Malaiyandi:

W. Curtis Preston: what's that?

Prasanna Malaiyandi:

Object lock didn't exist back then, so.

Prasanna Malaiyandi:

W. Curtis Preston: Okay, so we won't, we won't blame them for that.

Prasanna Malaiyandi:

But, but, but since object lock exists now, we'll say this is what you should do.

Prasanna Malaiyandi:

Uh, but you know what did exist back then that they did not use multifactor

Prasanna Malaiyandi:

authentication?

Prasanna Malaiyandi:

Oh, for their access into their admin account or into their

Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

Prasanna Malaiyandi:

Yeah.

Prasanna Malaiyandi:

So if somebody gains access to your admin keys and they're able to log in.

Prasanna Malaiyandi:

If you don't have MFA you, you have zero protection against someone

Prasanna Malaiyandi:

either stealing or accidentally, you know, inadvertently getting access

Prasanna Malaiyandi:

to administrative level keys and, uh.

Prasanna Malaiyandi:

I mean M-F-A-M-F-A-M-F-A mfa, I mean, how many, how many times

Prasanna Malaiyandi:

do we have to say it right?

Prasanna Malaiyandi:

Um, good password management, MFA and, uh, patch management.

Prasanna Malaiyandi:

We, we say this all the time.

Prasanna Malaiyandi:

If you just did those three things, you'd stop roughly 90% of attacks.

Prasanna Malaiyandi:

And in this case, if they had had MFA, this, uh, bad actor

Prasanna Malaiyandi:

would not have been able to

Prasanna Malaiyandi:

gain access.

Prasanna Malaiyandi:

Yeah, and and I think it's important to say

Prasanna Malaiyandi:

just because you use MFA and patch management and everything else.

Prasanna Malaiyandi:

Doesn't mean that you don't need backup, you still need backup because

Prasanna Malaiyandi:

that is how you are gonna recover from this, plus other issues as well.

Prasanna Malaiyandi:

The security side of things just sort of helps to protect you from

Prasanna Malaiyandi:

letting the hackers in to some extent.

Prasanna Malaiyandi:

Not gonna be a hundred percent foolproof, but hopefully, like

Prasanna Malaiyandi:

Curtis said, protects you.

Prasanna Malaiyandi:

And 80, 90% of the cases.

Prasanna Malaiyandi:

W. Curtis Preston: Yeah, to borrow from and totally abuse

Prasanna Malaiyandi:

a quote from Shakespeare.

Prasanna Malaiyandi:

There is more on heaven and earth than that is dreamt of.

Prasanna Malaiyandi:

In your philosophy, there are many, many ways that your

Prasanna Malaiyandi:

data can be attacked, deleted.

Prasanna Malaiyandi:

Set on fire exploded.

Prasanna Malaiyandi:

Sucked into a sinkhole.

Prasanna Malaiyandi:

There's so many different things that can happen to your data.

Prasanna Malaiyandi:

That's why you have backup.

Prasanna Malaiyandi:

And backup protects against all of them.

Prasanna Malaiyandi:

And, and we're saying backup and Dr and all of those things that come with it.

Prasanna Malaiyandi:

Um, but uh, the other thing that they also didn't do was this idea

Prasanna Malaiyandi:

of, um, you know, least privileged.

Prasanna Malaiyandi:

Do you

Prasanna Malaiyandi:

want to talk about that?

Prasanna Malaiyandi:

Yeah, so normally you do not want, in a company, you don't

Prasanna Malaiyandi:

want the intern to have the same level of access as your CEO or your IT admin.

Prasanna Malaiyandi:

And so you wanna be able to say, Hey, whatever access a person needs

Prasanna Malaiyandi:

to something, that's all they should have access to and nothing else.

Prasanna Malaiyandi:

And so you wanna have, make sure that you are focused on that and

Prasanna Malaiyandi:

don't just say, Hey everyone, you guys have admin credentials so you

Prasanna Malaiyandi:

can do anything and everything.

Prasanna Malaiyandi:

Because if one person who inadvertently gets compromised,

Prasanna Malaiyandi:

now everything is exposed.

Prasanna Malaiyandi:

So you wanna scope down their access to only what they need and that's it.

Prasanna Malaiyandi:

W. Curtis Preston: Exactly.

Prasanna Malaiyandi:

Now, when, when reading the, uh, articles about this, one of

Prasanna Malaiyandi:

the things that other people I.

Prasanna Malaiyandi:

Uh, dinged this company for, was that they didn't have an established

Prasanna Malaiyandi:

procedure for locking down the account.

Prasanna Malaiyandi:

And when I thought about that, I just found myself wondering.

Prasanna Malaiyandi:

I.

Prasanna Malaiyandi:

Huh?

Prasanna Malaiyandi:

How exactly would that happen?

Prasanna Malaiyandi:

And the best that I could find, you know, how, how would you do that?

Prasanna Malaiyandi:

And the best that I could find is that you would have a secondary account

Prasanna Malaiyandi:

that has access to this account that you then have a procedure to do things

Prasanna Malaiyandi:

like, um, disable, I think about.

Prasanna Malaiyandi:

You know, what, what is, what is the cloud equivalent to blocking somebody out?

Prasanna Malaiyandi:

And I, and I think the, the, the quickest way would be to

Prasanna Malaiyandi:

disable particular I am profiles.

Prasanna Malaiyandi:

Um, there, there was some, I, you know, and I'm not, I'm not

Prasanna Malaiyandi:

an expert on this, uh, I don't think you're an expert on this.

Prasanna Malaiyandi:

I would say talk to your cloud company.

Prasanna Malaiyandi:

Talk to your cloud provider.

Prasanna Malaiyandi:

Ask them, Hey, I am worried that one day my cloud entire environment

Prasanna Malaiyandi:

might become compromised.

Prasanna Malaiyandi:

How can I automate, basically locking everything out?

Prasanna Malaiyandi:

Right?

Prasanna Malaiyandi:

Once we determine that a hacker is in our environment, I would really like

Prasanna Malaiyandi:

a button that I can press from another account that shuts everything and down.

Prasanna Malaiyandi:

And everything like this can be automated and yes, that is a, uh,

Prasanna Malaiyandi:

you know what, what, what's the term?

Prasanna Malaiyandi:

The, you know, yeah.

Prasanna Malaiyandi:

The nuclear option.

Prasanna Malaiyandi:

That is the nuclear option.

Prasanna Malaiyandi:

But once you have an a hacker in your account to me that that

Prasanna Malaiyandi:

would be the proper option.

Prasanna Malaiyandi:

Shut everything down.

Prasanna Malaiyandi:

Uh, except for like, I would think create a new IAM profile that you can

Prasanna Malaiyandi:

use after you've done this, and then nuke everything that isn't that, um.

Prasanna Malaiyandi:

And, and that should be automated and that, and I don't think this

Prasanna Malaiyandi:

is something we normally talk

Prasanna Malaiyandi:

about.

Prasanna Malaiyandi:

I don't think it's something we talk about, but

Prasanna Malaiyandi:

it has to be out there somewhere.

Prasanna Malaiyandi:

I'm sure that AWS or Google or pick your favorite cloud provider,

Prasanna Malaiyandi:

they probably have a procedure.

Prasanna Malaiyandi:

I think the danger is you don't want it too automated because there's also the

Prasanna Malaiyandi:

risk that a hacker or someone else could trigger that and shut down your company.

Prasanna Malaiyandi:

So it's sort of one of those nuclear options.

Prasanna Malaiyandi:

So you don't wanna make it too easy.

Prasanna Malaiyandi:

But I'm sure that

Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

Prasanna Malaiyandi:

a.

Prasanna Malaiyandi:

Yeah.

Prasanna Malaiyandi:

W. Curtis Preston: use the nuclear option to.

Prasanna Malaiyandi:

This is one of those where this, this is the nuclear button, and so you just,

Prasanna Malaiyandi:

maybe you have an account that just does this and that account is like

Prasanna Malaiyandi:

completely separate from everything else.

Prasanna Malaiyandi:

Like we, we talk about having an account.

Prasanna Malaiyandi:

That is the backup account, right?

Prasanna Malaiyandi:

That is used for backups and no one ever logs into this account.

Prasanna Malaiyandi:

And you create it in such a way that if anyone ever does log in,

Prasanna Malaiyandi:

does log into the account, it sets off alerts everywhere and Right.

Prasanna Malaiyandi:

It, it can be like a honeypot account, but this account, um, yeah,

Prasanna Malaiyandi:

I'm not sure how to do, again, I'm not an expert in this, but I would,

Prasanna Malaiyandi:

I would create a separate account.

Prasanna Malaiyandi:

I would make that account as secure as humanly possible.

Prasanna Malaiyandi:

Again, ask your cloud provider how to do that, uh, the, the best way to do that.

Prasanna Malaiyandi:

But I just noticed that everybody said that almost everybody, they're like, they

Prasanna Malaiyandi:

should have had procedures for what to do in this situation, and, and they didn't

Prasanna Malaiyandi:

have them.

Prasanna Malaiyandi:

I think

Prasanna Malaiyandi:

perhaps,

Prasanna Malaiyandi:

right?

Prasanna Malaiyandi:

W. Curtis Preston: what's that?

Prasanna Malaiyandi:

At least manual procedures, right

Prasanna Malaiyandi:

W. Curtis Preston: At least manual procedures, right?

Prasanna Malaiyandi:

Because I don't think that changing the passwords on IM profiles was

Prasanna Malaiyandi:

the, the quickest way to do that.

Prasanna Malaiyandi:

Right?

Prasanna Malaiyandi:

I think the thing they should have focused on perhaps was kicking out,

Prasanna Malaiyandi:

currently logged in sessions, uh.

Prasanna Malaiyandi:

Yeah, I don't know.

Prasanna Malaiyandi:

And the problem is, it's like, how, how do you, you, you have to,

Prasanna Malaiyandi:

you have to build your incident response around your environment.

Prasanna Malaiyandi:

And one of the things that they could have done is maintain an inventory of,

Prasanna Malaiyandi:

um, basically privileged, super privileged accounts and look and see if there

Prasanna Malaiyandi:

were any new ones.

Prasanna Malaiyandi:

Well, that's the thing I was going to mention is they

Prasanna Malaiyandi:

should at least have had monitoring.

Prasanna Malaiyandi:

When someone adds a super privileged user, they should have been flagged

Prasanna Malaiyandi:

about that immediately, right?

Prasanna Malaiyandi:

Because that's not a common occurrence.

Prasanna Malaiyandi:

W. Curtis Preston: Right.

Prasanna Malaiyandi:

And so monitoring, alerting is also looks like something

Prasanna Malaiyandi:

that people should be doing to catch these sort of issues as well.

Prasanna Malaiyandi:

W. Curtis Preston: Yeah, and, and, and I'd love, by the way, I'd love

Prasanna Malaiyandi:

other suggestions from listeners.

Prasanna Malaiyandi:

I would love to hear from you.

Prasanna Malaiyandi:

If you go to backup wrap up.com, there's actually a button on there

Prasanna Malaiyandi:

that you can leave voicemails.

Prasanna Malaiyandi:

You can send us notes and, uh, you know, you know, if you'd like, we can actually

Prasanna Malaiyandi:

even play your response on the air.

Prasanna Malaiyandi:

I would love to hear better suggestions than we have from a security perspective

Prasanna Malaiyandi:

because I'm, you know, I'm Mr.

Prasanna Malaiyandi:

Back.

Prasanna Malaiyandi:

I'm, I'm not Mr.

Prasanna Malaiyandi:

Security.

Prasanna Malaiyandi:

I, I play, I play a security on tv.

Prasanna Malaiyandi:

Um, the, uh, but, but, but, uh, the summary statement.

Prasanna Malaiyandi:

From a backup perspective, if they had simply followed the 3, 2, 1 rule, if they

Prasanna Malaiyandi:

had made another copy of their backups in another account, in another location,

Prasanna Malaiyandi:

if they had used a cloud provider to do this for them so that then a copy of all

Prasanna Malaiyandi:

their data was stored in, in a completely different company, if they had done any

Prasanna Malaiyandi:

of those things, they would've at least had a copy so that once they got on the

Prasanna Malaiyandi:

other side of the attack, they could have.

Prasanna Malaiyandi:

Then recovered all the data because that's the, the true disaster here.

Prasanna Malaiyandi:

As that once they've been attacked and once the attacker gained access

Prasanna Malaiyandi:

to their account, they were able to delete all their data, both

Prasanna Malaiyandi:

their primary and their backups

Prasanna Malaiyandi:

So I know we've been focused solely on code space as the

Prasanna Malaiyandi:

company, but I think there's also blame to be placed on those 200 companies who

Prasanna Malaiyandi:

were using Code Spaces for not also having a backup of their data and relying solely

Prasanna Malaiyandi:

on code spaces as their service provider.

Prasanna Malaiyandi:

W. Curtis Preston: You know, that is an interesting, we, you know, we

Prasanna Malaiyandi:

tend to focus here on the fact that it was a provider, but what this

Prasanna Malaiyandi:

really was, was a SaaS provider.

Prasanna Malaiyandi:

Yep.

Prasanna Malaiyandi:

W. Curtis Preston: Right.

Prasanna Malaiyandi:

Yep.

Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

Prasanna Malaiyandi:

So we don't know what happened to those other companies.

Prasanna Malaiyandi:

And so this is, this is a double lesson, right?

Prasanna Malaiyandi:

If you're, if you're running in the cloud, make sure you've got

Prasanna Malaiyandi:

a backup of that, of that data.

Prasanna Malaiyandi:

If you're using a SaaS provider, make sure you have another copy of

Prasanna Malaiyandi:

the data that you're putting in that SaaS provider, because that would be

Prasanna Malaiyandi:

another way for at least the, the, um, the, the, the thing that's difficult.

Prasanna Malaiyandi:

Here, again, I agree with you, the thing that's.

Prasanna Malaiyandi:

The, the difference here is that unlike many of the SaaS providers, this company

Prasanna Malaiyandi:

specifically said, Hey, we got it.

Prasanna Malaiyandi:

We got your backups, this data, and it's tested and it's all this stuff, right?

Prasanna Malaiyandi:

Um, the, um, you know, I just had a thought.

Prasanna Malaiyandi:

Um, if we go to LinkedIn and we search for code spaces.

Prasanna Malaiyandi:

Find people that used to work in code spaces.

Prasanna Malaiyandi:

Oh, that would be, I wish, I wish we could talk to somebody

Prasanna Malaiyandi:

that was involved in this, but

Prasanna Malaiyandi:

uh, I think we can

Prasanna Malaiyandi:

I'm sure there are probably NDAs.

Prasanna Malaiyandi:

W. Curtis Preston: What's that?

Prasanna Malaiyandi:

Prasanna Malaiyandi: There's probably NDAs.

Prasanna Malaiyandi:

W. Curtis Preston: Oh, they're probably NDA.

Prasanna Malaiyandi:

Yeah.

Prasanna Malaiyandi:

All right.

Prasanna Malaiyandi:

Well back up the cloud, I told you so.

Prasanna Malaiyandi:

Any final thoughts for you, persona?

Prasanna Malaiyandi:

No, I totally agree with that, and I like this because I

Prasanna Malaiyandi:

know we bring up code spaces a lot, so I think that hopefully our listeners now

Prasanna Malaiyandi:

understand why we talk about it and what they should not be doing, and why we

Prasanna Malaiyandi:

harp so much on things like the 3, 2, 1 rule on MFA, because you don't want your

Prasanna Malaiyandi:

company to have to shut its doors because they were unable to recover their data.

Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

Prasanna Malaiyandi:

And sadly, this will not be the last company that.

Prasanna Malaiyandi:

You know, basically ceased to exist, uh, because they didn't

Prasanna Malaiyandi:

properly back up their data.

Prasanna Malaiyandi:

Yeah,

Prasanna Malaiyandi:

W. Curtis Preston: All right.

Prasanna Malaiyandi:

Well, uh, thanks for, uh, joining me persona, as always.

Prasanna Malaiyandi:

anytime and looking forward to see your analytics on your car.

Prasanna Malaiyandi:

W. Curtis Preston: I will see what I can do and I will also thank our

Prasanna Malaiyandi:

listeners, we be nothing without you.

Prasanna Malaiyandi:

Thanks for listening.

Prasanna Malaiyandi:

And be sure to subscribe so you don't miss an episode.

Prasanna Malaiyandi:

That is a wrap.