A ransomware attack on Rackspace in 2023 left thousands of customers without

access to their critical email data for months and led Rackspace to completely

abandon the hosted exchange business line.

On this week's episode of the backup wrap-up we discuss a detailed timeline

of this event and most important, the lessons that we can learn from it.

The incident in this episode is one of the many stories that are behind

the recommendations that you may have heard from me throughout the years.

I'm w your Curtis Preston, AKA Mr.

Backup.

And there's a reason I'm so passionate about this subject.

It's because in my first job as a backup admin, my company lost an important

database and I couldn't restore it.

Since that moment, I've dedicated my career to making sure that

would never again, happen to me.

Or anyone who bothers to listen to me?

We take unappreciated backup admins and turn them into cyber recovery heroes.

This is the backup wrap up.

W. Curtis Preston: Welcome to the show.

I'm your host, w Curtis Preston, AKA, Mr.

Backup, and with me, I have my consultant that will help reduce

my level of starstruck today.

I'm hoping

I don't think that's possible.

W. Curtis Preston: I'm gonna be,

That sound.

So yes, I think you should tell people who may not have caught that.

Who are you gonna go see today?

W. Curtis Preston: I'm gonna meet William Shatner today.

I am

Are you

W. Curtis Preston: super psyched.

Yeah, I, I actually bought it.

There's an event.

There's a, there's a premiere of this new documentary that's about William Shatner.

Um, and it's in la It's, and it's, uh, they're gonna do the screening.

They're gonna do q and a, and then there is a.

Uh, birthday party for him, his 93rd birthday party for him, uh, afterwards.

And it's being held in the original, um, in the studio where

they originally filmed the pilot.

Um, and so it, I'll also be meeting, uh, Kevin Smith and, um, so.

Hopefully I will.

My dream, if I can, if I can get a selfie with William Shatner,

that'll be, you know, um,

Prasanna Malaiyandi: You'll be over the moon.

W. Curtis Preston: that'll be, I'll be over the moon.

Yeah.

I've already met.

I met, um, deforest Kelly.

I met, uh, Michelle Nichols.

I met, uh, George Decay and this will be, um, there is one

remaining, uh, star, original Star Trek member that's still alive.

Walter Koenig.

Um, that would be the, the one person who's still possible

to meet that I haven't met.

But, uh, yeah, William, I'm super excited about that.

So.

Does it count as meeting if you go visit

the grave site of the person?

W. Curtis Preston: Oh, that's just wrong.

That's

I'm just

W. Curtis Preston: That's just wrong.

Uh, yeah.

So, uh, just help me, help me keep my, my heart pitter Pat.

I'm definitely a, definitely a fan and meeting him, uh, will be very, very cool.

Uh, this week.

so wait, what is your, if you got a chance to ask

him a question, what would it be?

W. Curtis Preston: Oh, it's definitely not gonna be one of

those, like an episode 57, you know?

Um, wow.

I'm not prepared for that question.

I'll have to think about that.

Wow.

Did I

stump Curtis?

W. Curtis Preston: You did, you stumped me.

Yeah.

I'll have to, I'll, yeah, I'll definitely, you know, I'm gonna be, I'm gonna be so

nerded out, like I'm, I'm gonna be, yeah.

Um, I just, I, if I get to say two words to him, I'll be,

you know, I'll be like, hi.

Um, you know, I, yeah.

I, I just hope I don't do, like, I've met a lot of famous people and

so many times I've been like, chill.

But I remember there was this one person that I just randomly ran into in an

airport and I literally screamed their name like a, like a 10-year-old girl.

And, um, that was very embarrassing.

I just hope I don't go, William, that would

okay.

I'm sure he is used to it, you know?

W. Curtis Preston: Yeah, I'm sure.

Yeah.

Um, so this week we're continuing our series about cloud disasters

and this one is pretty bad.

Um, you know, and again, this is yet another story that's gonna

prove the point back your stuff up.

Right.

You know, even, even if this is actually, this is a really good.

Story that basically proves that even if the vendor is backing it up

for you and the backups are included as part of the package, something

so catastrophic might happen that those backups don't come in handy.

Does that sound about right, Pana?

It does, but I have.

Two comments about that.

W. Curtis Preston: Yeah.

Okay.

the first is, this reminds me a lot about the

OVH story that we did a while ago.

So if you haven't let heard that episode, go back, give it a listen, because it

was also the case with OVH that they said they were doing backups, but

people were not able to restore their backup because they were sitting in

the same data center as a production and there was a fire, so not so good.

W. Curtis Preston: Yeah.

And, and I'm gonna say, so this, this story is about Rackspace, which I'm gonna

say I have no ill will against Rackspace.

I, I feel for the people that had to go through this, uh, the thing I struggle

with is the ways in which Rackspace tried to deflect, blame Rackspace.

The company tried to deflect blame.

Uh, and so based on that, we've got a pretty solid timeline of the

events Now, just, just for color.

And I didn't know this, this part I'm about to say.

I didn't know this until, until I was researching for the story.

Prior to this event happening, Rackspace had already suffered, uh, a sharp.

Decline in value.

At the height of their value, April, 2021, they were a, an $8 billion company.

And by the time this event happened, they had dropped,

over over 90% of their value.

They were then an $800 million company.

And, as of today's recording, they

are valued at $340 million, which is 5% of where they were with at their high.

W. Curtis Preston: Right?

Yeah.

Um, so, so they were already in sort of trouble and I, I think that may

be why they tried to deflect blame.

So, um, let's start with sort of the, before the story, right?

So before the story, there was something called the proxy, not she

exploit, uh, in September of 2022.

It was publicly announced and it basically, it allowed someone to

gain control of an exchange server.

It was announced September 30th, 2022, November 8th, Microsoft

released a security update but there was a minor issue with the patch.

And Rackspace claimed this was why they didn't install it at that time,

but by November 17th, Microsoft had fixed that, , that issue.

You know, we talk about three things, right?

Password management, patch management, and MFA.

And then if everybody just did this, then it would've stopped.

Uh, you know, it would stop so much.

And this story is so much I.

Evidence of that, uh, because November 17th that minor issue

with the patch was fixed.

So they could have, and in my opinion, should have immediately put on this

security patch because it was such a huge exploit there was a CVA attached to it.

And, uh, well-known within the industry, they should have immediately patched

all of their, uh, exchange servers.

By the way, I should mention what we're talking about is that

Rackspace had a hosted exchange service, not Microsoft 365.

They ran hosted exchange on their own servers in their own, uh, data center.

Before you continue on, I think it's important to state that

for that September 30th, right, there was a workaround that was deployed, right?

That

pretty much Microsoft was like, Hey, we haven't quite figured out the patch

yet, which will come out November 8th.

But in the meantime, here's a workaround to make sure you don't get impacted,

which Rackspace did apply, apply.

Right.

W. Curtis Preston: they, they did or they did not.

They did apply the workaround.

W. Curtis Preston: Yeah.

Okay.

Yeah.

Yeah.

So it's not a permanent fix, but at least

sort of protects you for now.

So then the other thing is, um.

around this time there were actually two exploits, Right.

So there was a proxy, not shell exploit, and then there was

another one, um, O-W-A-S-S-R-F.

I don't know what that stands for, but that's what they

called it, right?

And these two are kind of related.

And so the patch though, that came out in November would have fixed both.

W. Curtis Preston: Right.

they were applied, but the workaround that was

applied in the end of September only addressed the proxy nutshell issue.

It

did not expl address the second exploit.

W. Curtis Preston: Yeah.

By the way, the OWA most certainly stands for Outlook.

Web access would be my guess.

I don't know what SSRF but stands for, but yeah, it is kind

of complicated that basically.

That there was a patch that the, the PA had, they applied the patch, they would've

fixed at a, at that time, unknown problem.

Um, but the, but they didn't apply the patch.

And then two weeks goes by and then what happened?

And then on November 29th, Rackspace says that they

were attacked by a group called Play, which gained access to their exchange

environment using stolen credentials, and that they had access to some of Rackspace

exchange environments, which, if I

was a customer on hosted exchange, I would be kind of freaked out.

W. Curtis Preston: Yeah, exactly right.

So they, they gain privileged access of their exchange servers.

We're, not sure if they knew in November, but because they first

notified people December 2nd.

Literally at two o'clock in the morning.

Right.

Based on the, the, the stuff that we have, they, they may have at that point

realized what happened and they were able to trace it back to November 29th.

And at 2:00 AM right?

This is when they were like, yeah, here's what happened.

We noticed something, and then they just brought everything down, right?

They were

like, yep, we're not gonna allow any more access

W. Curtis Preston: response, right?

The, the next part of the story is the one that really sets it apart.

I don't know any other story like this.

The res, their response was, and, and again, if you think about now that if

you think about where they were as a company, this part maybe makes more sense.

But what they decided to do was they said, you know what?

This is gonna take us a while.

This the, I'm making up words here.

We've been thinking about shooting this thing in the head anyway, and so let's

just move everybody over to Microsoft 365.

So December 2nd at 2:00 AM is when they first started telling people that

they had this problem, and by 8:00 PM that evening, they had made the

decision to move everybody over to 365.

Yeah, that I could, I would have loved to have

been a fly on the wall in those meetings, right when they were

trying to

W. Curtis Preston: not have wanted to be in the meeting.

Yeah.

Yeah, I, that's why I said I wanted to be a fly on the wall.

Right.

W. Curtis Preston: Yeah.

and hearing these conversations because you

know, it must have been a difficult decision to come to, right?

W. Curtis Preston: Yeah, because that would've been a competing service, right?

So if, if it's not obvious, like if you used hosted exchange, you were

very consciously using hosted exchange, not Microsoft 365, and you had reasons

for doing that, and they're like, guys, this is gonna take us a while.

We're gonna move everybody over to 365.

But what did they not move?

Uh, so there were two things.

They did not move, right?

The, probably the most important thing was their emails,

right?

What people cared about with the hosted exchange service, right?

Because they basically said, we will recreate things for you, make it easy.

So you have all your stuff up and running at Microsoft 365, but we can't

get you back all your emails yet.

W. Curtis Preston: Right.

Right, so that was one thing.

I think the second thing, and I don't know if they were forthcoming with

this, but the fact that with their hosted exchange implementation, they

offered backup as part of the service,

W. Curtis Preston: Right.

right?

When they told customers, Hey, Microsoft 365 is where you should

be looking at, I do wonder if they told people, by the way, you need to

figure out your own backup solution.

W. Curtis Preston: Yeah, none of the communications that we found

between them and customers, uh, showed that they'd said anything.

Uh, but the idea that they would, in the middle of the outage basically

abandoned an entire business line.

Move everybody over to 365.

What they did do was they were able, apparently they, they did

automate the process of creating the accounts for them over on 365.

So you, you were able to, um, you know, essentially you, you were able

to start sending and receiving email.

Relatively quickly considering how long the rest of this took within

a day, it looked like uh, or so you were able to send and receive

email using your old email address.

If you were an exchange hosted exchange customer, and now you're on 365.

You just didn't have access to any of the email you had received up to that point.

Yeah, which I like.

I go back and forth on that.

It's like, great.

I could send and I could see what people are sending me, but I have a lot

of old stuff and I would be freaking out if I lost all of my old emails.

Or a lot of times if these are businesses and organizations, maybe

they have contracts which are being sent back and forth via emails.

W. Curtis Preston: Yeah, like literally stuff that you got today, right?

Stuff that you got yesterday, stuff you're actively working on.

And a lot of people use their email system as sort of a somewhat,

sometimes temporary, sometimes permanent storage system, right?

And, and they're like, I know where that contract is.

It's in the email that I got, you know, three days

Prasanna Malaiyandi: and I can search for it.

I can find it.

Yep.

W. Curtis Preston: You're not able.

Yeah.

So on one on one hand, like, like you said, I would've wanted to be a fly

on the wall because they're saying we need to do what's, we need to get

our customers up and running again.

We need them, we need to get them, be able to send and receive email.

And I think that was a good decision.

Prasanna Malaiyandi: That's probably the most

important thing to do first.

W. Curtis Preston: yeah.

The, the, the bad decisions happened way before this.

In my opinion, this was a good decision, um, because as we're going

to find out in the story, they didn't exactly have a good, uh, backup.

I.

System, um, at least not one that, that I would recognize, right?

So they actually advertise backup as part of the service.

And again, I I, I'm gonna put this out as this is why, when, when I say,

even if the SaaS vendor advertise backup as part of the service, you

might want to consider a third party.

And I'll put an asterisk, especially if they charge extra for it.

Um, I think in this case it was just included as part of the, the, well,

I'm gonna put it, I'm gonna put in a, I'm gonna put it, especially,

I'm gonna put two asterisk, especially if they charge for it.

'cause then that gives you an incentive to pay somebody else instead.

But then I'm gonna say, especially if they don't charge for it, which means they're

probably not spending any money on it.

Prasanna Malaiyandi: Enough money to do that.

Yeah.

I wanna focus on that for a second

because I did remember earlier on in the episode, I said I had two

comments and we only covered one.

So here's a second

W. Curtis Preston: Oh.

Right is, does this apply If you are using a SaaS

or whatever backup product in and of itself, that you should have a second

vendor because who knows what that backup vendor will do, and maybe you

can never get your data back out.

I understand Rackspace, they offered backup and it, the backups didn't work.

Right now, there are a whole slew of SaaS data protection companies, or you could

roll your own.

Right.

In those cases though, do you have the same recommendation that if I do decide

to use a SaaS data protection company, I should probably use two SaaS data

protection companies because I don't

know if one can get me my data back?

W. Curtis Preston: Yeah, no, that, that's not what I'm saying.

I'm saying it because I'm talking about a SaaS service and then hiring

a SaaS data protection company that puts your data in two different places.

I would, I would not argue.

Having another one, but it's gonna be really, it's a, it's already

going to be a big enough cost.

I think the idea of putting the data in two different, completely different

protection zones, risk factors, you know, earthquake and flood zones, all of

it already, diversifies it.

yeah.

W. Curtis Preston: Yeah.

I think that already diversifies it.

so now my second question.

W. Curtis Preston: I'm just saying that if you have a backup, if you

have a SaaS service, like 365, so 365 is about to start offering backup.

Um, and and I'm just saying I still like the idea of a third party copy.

Yeah.

Okay.

So now my second question follow up to that is there's a lot of cloud public,

cloud providers, right, that people hook into that leverage things like snapshots.

W. Curtis Preston: Yeah.

So they're just an orchestration layer on top.

They're doing all the data movement.

They're moving your data around, they're taking the copies.

Do you have the same concerns with those as well?

W. Curtis Preston: I do.

Um, basically the, that's why again, I li I like the orchestration

companies and the ones I like best are the ones that ultimately take

a copy of the data outside, right?

We, we use snapshots to orchestrate and to create the backup, and then we use

something else to get the data out of.

You know, your favorite cloud vendor, again, getting it out,

storing it in another place.

Second best to that would be storing it in another region, in, in another account.

But you know, this is just the, basically I see, like, I think

of like, let's say AWS as NetApp.

Right.

So I need another copy, a final copy of the data that isn't on AWS because

of rolling code concerns, right?

So you get a bug and it, and it rolls, uh, you know, and takes

out both primary and the backup.

then are you also concerned though, because a lot of

these SaaS data protection companies are built on top of the big clouds.

W. Curtis Preston: So I do, I do want to, um, then make sure that they're

stored in different regions and whatnot.

I, you know, I can only, I can only,

Prasanna Malaiyandi: you can only go so far.

W. Curtis Preston: that, yeah, you could only go so far, right?

It'd be like the same.

The same would be true if we weren't talking to cloud and we were saying

they also, your backup service also uses Soliris, right At, at some point there's

a risk that you just can't get away from.

Sorry, I, I know we never talked about that before, so

I was very curious about your take on

W. Curtis Preston: Yeah.

Yeah.

Okay.

So, uh, so we were talking about December 2nd is when, um, they, you

know, when they made this move, right?

Um, the, the, they first mentioned that they believe it was a

security incident on December 3rd.

Uh, and then, uh, December 2nd or December 6th, they say that

it was a ransomware incident.

And then finally, 14th, they revealed the attack was from a, their, their

words financially motivated threat actor.

Um, so we don't know the details, uh, of, you know, the extortion, but

there was some kind of extortion.

We also don't know whether or not they ultimately, I.

Paid that money.

Um, you know,

and, and this.

W. Curtis Preston: I, I, I advise as much as possible not

to pay the money, but go ahead.

and this goes just the earlier thing about talking

about a security incident and then changing it to ransomware incident.

I get it.

But I know you and I, we've talked in the past about vendors or companies

should be more transparent to a certain extent about what's going on in order to.

Build confidence in the public in terms of they have things handled,

they're figuring things out.

It's okay.

And so I wanted to get your take on that messaging that came from Rackspace.

Do you think that caused, like, do you think that would've caused a

lot of concern for customers or the public in terms of their ability

to handle things first, calling it a security incident and then a

ransomware, and then sort of this back

and forth on data?

I.

W. Curtis Preston: I think it's possible that they just,

they revealed what they knew.

Like you do have to be careful saying only what you know for sure.

Yeah.

W. Curtis Preston: The thing that needs to be understood at this point as they move

through the story is that be like, on one hand, I'm glad they did what they did.

They moved everybody over to 365, got everybody working, but

everybody's gonna be clamoring for their emails right from the previous.

Uh, but because they did it the way they did it, so it, they, if they

had migrated exchange into 365, they could have brought the data with them,

but it would've taken longer because they still, they had dead servers.

Right.

Yeah.

They don't have the data yet.

W. Curtis Preston: A migration takes a while too.

Right.

But because it did it the way they did it, the only option at this

point is to create PSTs of individual users and then import those PSTs

to those users on the other side.

Um, and so they, they said on December 18th that they had, um, they

had created in, um, that I, I, I.

I'm a little confused where like on one hand they said that they had, I think

they had figured out a way to start restoring the affected exchange servers,

but they hadn't yet restored all of them.

Right.

And this is when they, when they first announce, they're like, so

here's how this is gonna work.

We're gonna restore an exchange server.

If you're on that exchange server, you will then be able to export,

uh, A PST file, and then you'll then be able to download a PST file for

each user, and then you will then be able to upload that into 365.

And they first announced this on December 18th, and then December

already two weeks after they shut down the service.

W. Curtis Preston: Thank you very much.

That is two weeks after they shut down the service.

Right.

Um, and uh, it also, it's important to understand that by exporting

it as a PSD and then importing it, it's not gonna be a perfect restore.

'cause , I'm pretty sure that you're gonna lose folders and

all of that in this process.

Uh, not to mention the fact that it's just you're gonna lose metadata too.

Right.

So, you're so in my mind I'm thinking, oh, this is great.

It's a way for people to do things.

Right.

And then

on December 20th, they said that they had just begun testing

the above recovery procedure.

W. Curtis Preston: Yeah.

Yeah.

So they hadn't actually tried it out or done anything.

They're kind of shooting from the hip, and I get it.

They're urgently trying to figure out how to get the data back for the customers.

But at the same time, it doesn't inspire confidence.

W. Curtis Preston: Yeah, and the thing is that if they had prepared for this

event eventuality back in the day, they could have created a different procedure.

Because there are backup software products that allow you to,

um, directly extract PST data.

From a backup, right?

It it does.

They, they do exist.

It does happen.

So if they had tested this beforehand, they wouldn't have

done it the way they did it.

They would've figured out a way to directly, instead of restoring

exchange servers and then saying, Hey, customers, go get your PSDs.

They would've been able to just directly create the PSDs.

They would've figured that out beforehand.

Um,

and then it's just a matter of

W. Curtis Preston: think that would've been a much quicker method.

Yeah.

And they would've just had to execute on it rather than trying

to figure all this out, because I don't know if a lot of folks.

Understand when you're trying to recover from a ransomware attack, right?

You're not only trying to figure out everything that went wrong,

but also all the extreme pressure you're under lack of sleep, right?

People yelling at you possibly, right?

Worrying about, am I gonna have a job after this?

Yeah.

W. Curtis Preston: I can't imagine the number of people that were, you

know, companies that were yelling about how much money they were spending

and blah, blah, blah, blah, blah.

Right?

Um, yeah.

So December 22nd is the first day that they notified some customers that they

could start retrieving some of their mail.

This is three weeks since the incident, right?

Um, and it wasn't until January 5th, which is another, what, two weeks I.

They said they had 50% done.

I mean, this would, this took a while.

Yeah.

So here's my question, Curtis.

I know we talked earlier about, okay, they should have had

these procedures documented.

They should have thought about them, right, and tested it

out.

So then they had a process, Do you honestly think people think about

these scenarios and walk through?

Because normally when you're thinking disaster recovery or

backup and recovery, right?

It's like, oh, I lost an email, or I lost a part of something, or someone

accidentally deleted a user, right?

I think that the mind needs or people's mind, people need to change.

And start to start thinking about some of these cases.

But when they designed the system, do you think that like they were like, Hey,

I wonder what'll happen if the exchange servers all get hit by ransomware and

we have to rebuild and all our customers are gonna leave us for Microsoft 365?

W. Curtis Preston: So I'll answer that in two ways.

One is probably not right, and two, they probably should have.

Right.

One of the things I talk a lot about in, you know, when I'm talking about

how to design your systems and stuff is that go get the most scary person

in your environment and have them.

Come up with scenarios, right?

Come up with recovery scenarios.

This is the whole point of tabletop exercises, right?

You get those super negative people that that interject.

Well, what if, well, what if, you know, we're down for weeks?

Make

that's like me,

W. Curtis Preston: about this a lot.

Make the decision upfront that if we get hit by ransomware.

We're gonna immediately move everybody over to 365, and if

we do that, we're going to need to do it to recover this way.

They, they should have been able to foresee this decision.

Right, because by doing it the way they did it, again, and I don't disagree with

the way they did it, but by doing the way they did it, they necessitated this weird

double, you know, double or two step, super painful restore method that put

a lot of the work onto their customers.

Uh, and it's, it took them months.

Right.

Um, before, you know, before everybody was at, at least able, we don't even know,

um, exactly how many customers actually, were able to successfully recover.

Yeah, I think they said it was something like 3000

customers were impacted in their

hosted exchange environment.

Right.

W. Curtis Preston: right, right.

They said this was interesting is that they also said that play, or remember

play is the name of the ransomware group.

They accessed to use their words, the email of 27 customers and said there

is no evidence that they read it.

Um, know, the, the, what does that mean?

Right?

So again, this is where you, you have to look at like legalese.

They have.

The fact that they have no evidence that they read it doesn't mean they

didn't read it, it, they don't have evidence that they didn't read it right.

You can't prove a negative, right?

They downloaded the email of 27 customers.

And so it's likely that there could be, you know, there could

have been secondary attacks where play goes after the customers.

We didn't get any news of that, so maybe it didn't happen.

But, um, you know, just when you see messages like that, we have no evidence

that that doesn't mean it didn't happen.

It just means that they, they can't prove it.

It happened right?

Do you think, since you're just mentioning about

this, do you think there are possible ways they could have leveraged

security offerings, encryption, other things to protect these emails?

So even if play did attack their exchange server, I guess it depends

on what level they actually were able to exploit the exchange server.

W. Curtis Preston: I, I mean, they had admin access to exchange.

Prasanna Malaiyandi: that all bets are off?

W. Curtis Preston: far as all bets are off.

So even if there was encryption, I don't know if, if exchange has the

ability to encrypt it or if they could store the data on encrypted drives.

It doesn't matter once you're inside the application.

Right.

Because the, the, the data, it doesn't have record level encryption or anything

like that, that I'm aware of, but um.

Yeah.

Um, and, and the, the, the other part, and this is by the way, this is the

part of the story where Rackspace tries to shift blame by saying,

well, hey, this exploit that, that we got hit by was a zero day exploit.

Which is true in that it was an unknown exploit at the time that they got hit.

But if they had put the patch on that they should have put

at a minimum two weeks prior.

They wouldn't have been because that patch fixed the unknown problem at the time.

Yeah.

It's, I think, I know we always talk about it on the podcast.

It's patch, patch, patch, right?

When something comes, especially something with this level of severity, I.

Right.

It's like the log four J stuff that came out, what, 2021 December

and what a mess that was as well.

But everyone realized how severe of a security issue it was and

started patching their systems.

I think something similar needed to happen here as well.

it.

the fact that I get it, large customers, you have hosts, you need

to schedule downtime, but a high severity security issue like this

that was being exploited actively, like there's no reason you should have

waited two weeks to apply a patch.

W. Curtis Preston: Yeah.

And, and this, this is kind of what I wanted to go at here.

Uh, you know, we, there's a lot of lessons that could be learned from this incident.

One is third party backup.

If customers had it's had third party backup, which was totally

possible, they would be done.

I.

They would've, they would not have suffered this outage, um,

just like the Salesforce outage, um, in a previous episode, right?

If they had third party backup, they could have fixed it in minutes.

Um, the, and the other is this really hammers home when

there's a really severe exploit.

That is well known and there's a patch introduced for that exploit.

You need to put it in now.

Not.

Oh, we've got our patch management system.

We do this once every two weeks or whatever the, you know, whatever it is.

Look at this company basically, you know, a month from the, the announcement of the

patch, two months from the announcement of, uh, the actual exploit they got hit.

Um, and if they had simply just put the patch in.

When it was available, and, and I, I realized that it was available

sooner and there was a minor pa, but, but that, that issue was fixed.

There was two weeks between the time the, the patch was fully

fixed and fully available.

And the, when this actually happened, and it, it, it was yet another, this

was already a suffering company.

And if you look at the, the stock value of.

Um, rack Rackspace.

It had another sharp decline.

What, what did you say?

Wasn't it 15%

Yeah.

In

W. Curtis Preston: on December 2nd, which is the day they announced

that this had happened, right?

And so, you know, they were already suffering and this created

yet another, uh, decline in the and which did not recover.

It did not recover from that sharp decline.

Just think about this, think about you.

You really have to prioritize that patch management system, right?

You really have to make sure that you put in patches as soon as they come out.

Uh, you know, high level, high exploit.

Patches need to go in.

And again, I'm gonna once again say that I think the backup system needs

to be at the front of the line, right?

Patch those first, because that's the last line of defense.

And then make sure you, you know, in this case, the patch only applied to exchange,

but this is the point that we make.

If they had just put the patches in, this event would've never happened.

Yeah.

And you would still have 3000 customers using Rackspace hosted exchange, right?

I.

W. Curtis Preston: Yeah.

Yeah.

The other comment I was gonna make, Curtis, is

Given that they probably had the managed exchange solution for quite a while,

I wonder if they ever had a process.

I know we've talked about on the podcast of going back and looking

at their disaster recovery plans or their backup and recovery procedures.

Or if it was just sort of, Hey, we created this once, it should be fine.

We never have to really use it, so we'll never go back and make sure

it's up to date and all the rest.

W. Curtis Preston: Well, I mean, clearly they didn't, they didn't.

Account for ransomware, right?

A typical DR recovery scenario would've worked fine here, right?

If they, if the building caught fire, they knew how to re, they had backups.

They knew how to restore their exchange servers.

I think, I still think it took them longer than it should have, but

they had, they had a plan for that.

But that plan ransomware breaks a lot,

Why would ransomware be different than

the building going up in smoke?

W. Curtis Preston: Because, great question.

Because they likely were still fighting the ransomware itself, right?

When the building goes up in smoke, they could just literally restore everything.

It's gonna take a couple days, but at this point they're, they have an unknown time.

Yeah.

W. Curtis Preston: On the moment of December 2nd, they're like, we have

been taken down and we have no long, no idea how long we're going to be down.

Yeah.

Might be a.

day, might be a week, might be a.

W. Curtis Preston: Exactly right.

They, they're looking at these, at these other businesses that

are down for months at a time

and they're saying, we don't want to be that.

We don't wanna stop people's email for that amount of time.

Boom, let's do over here.

Had they, had it been adjusted regular, a fire of flood or whatever, they

would, they should have been able to say, this is gonna take us three days.

Three days is gonna stink, but.

It's not enough for us to abandon our entire business

model and move over to 365.

In this case, it was an unknown scenario, unknown amount of

time that they're gonna be down.

And so they decide to do this method that ended up making

everything take much longer.

Right?

Which, and they didn't test for this method.

so yes, I agree with everything, but I still wanna

go back to another clarifying point.

Couldn't they have treated ransomware?

Like I wanna, I wonder if they actually had disaster recovery plans in place.

Because

if I yeah, and this.

W. Curtis Preston: giving them the benefit of the doubt.

Yeah, because because in my mind, right, if I

had a ransomware happen, right as a company, there are two options, right?

One is I could try to figure out what all went wrong, rebuild my servers

in the same data center, procure hardware, all the rest, or I could just

treat it like a fire, shoot it in the head, connect completely, disconnect

everything, validate my DR site is still good, that there's no ransomware

there, and then bring everything up or restore on clean hardware, et cetera.

Like

W. Curtis Preston: if there's not ransomware or if there is ransomware.

even if there was ransomware,

W. Curtis Preston: But the problem is if you have a solid backup system, you have a

recent backup, and then you go to restore it, you know you're restoring the The

bad stuff.

Yeah, So, but then that should have just been a matter of figuring

out what is good and what is bad.

W. Curtis Preston: Right, which is going to take an unknown amount of time.

That was the problem.

Right.

And, and, um, yeah, I, I don't know if they had a DR plan.

I do, I do ask myself why it took them, the amount of time it took them

to restore all the exchange servers.

I, I just, I just go off the, I just go off the, you know, the message that

I, or the information that I have.

yeah.

W. Curtis Preston: Yeah.

And so the end of the story is this, uh, the company continued

to suffer, uh, additional losses in the value of their company.

They, they, they went from 800 million, then down to 350 million

now, uh, and they, they are close as of the taping of this episode.

They're close to a restructuring deal.

I don't know if this event had ever happened, if how different

the world would be right now, but it certainly didn't help.

Um, so please, folks, all I can say is, you know, put those

patches in when you, you know, and test your recovery procedures.

All of the recovery scenarios, right?

And if ransomware isn't one of the recovery scenarios, then

you need to rewrite your recovery plans.

W. Curtis Preston: You need to reconsider your recovery scenarios.

Absolutely.

Uh, all right, well, uh, I think this was a good episode.

Yeah.

Don't be like Rackspace

W. Curtis Preston: was, you know, ouch.

All right.

Uh, thanks.

Thanks for the chat.

like, uh.

It was fun, Curtis, and I am sure I will hear all about your,

uh, your, uh, event tonight.

W. Curtis Preston: You, you will be hearing all about it.

You'll probably be getting some live, some live chat or some

live, uh, texts during the event.

Um, and, uh, I, uh, be safe out there folks.

That is a wrap.