A ransomware attack on Rackspace in 2023 left thousands of customers without
access to their critical email data for months and led Rackspace to completely
abandon the hosted exchange business line.
On this week's episode of the backup wrap-up we discuss a detailed timeline
of this event and most important, the lessons that we can learn from it.
The incident in this episode is one of the many stories that are behind
the recommendations that you may have heard from me throughout the years.
I'm w your Curtis Preston, AKA Mr.
Backup.
And there's a reason I'm so passionate about this subject.
It's because in my first job as a backup admin, my company lost an important
database and I couldn't restore it.
Since that moment, I've dedicated my career to making sure that
would never again, happen to me.
Or anyone who bothers to listen to me?
We take unappreciated backup admins and turn them into cyber recovery heroes.
This is the backup wrap up.
W. Curtis Preston: Welcome to the show.
I'm your host, w Curtis Preston, AKA, Mr.
Backup, and with me, I have my consultant that will help reduce
my level of starstruck today.
I'm hoping
Prasanna Malaiyandi: I don't think that's possible.
W. Curtis Preston: I'm gonna be,
Prasanna Malaiyandi: That sound.
So yes, I think you should tell people who may not have caught that.
Who are you gonna go see today?
W. Curtis Preston: I'm gonna meet William Shatner today.
I am
Prasanna Malaiyandi: Are you
W. Curtis Preston: super psyched.
Yeah, I, I actually bought it.
There's an event.
There's a, there's a premiere of this new documentary that's about William Shatner.
Um, and it's in la It's, and it's, uh, they're gonna do the screening.
They're gonna do q and a, and then there is a.
Uh, birthday party for him, his 93rd birthday party for him, uh, afterwards.
And it's being held in the original, um, in the studio where
they originally filmed the pilot.
Um, and so it, I'll also be meeting, uh, Kevin Smith and, um, so.
Hopefully I will.
My dream, if I can, if I can get a selfie with William Shatner,
that'll be, you know, um,
Prasanna Malaiyandi: You'll be over the moon.
W. Curtis Preston: that'll be, I'll be over the moon.
Yeah.
I've already met.
I met, um, deforest Kelly.
I met, uh, Michelle Nichols.
I met, uh, George Decay and this will be, um, there is one
remaining, uh, star, original Star Trek member that's still alive.
Walter Koenig.
Um, that would be the, the one person who's still possible
to meet that I haven't met.
But, uh, yeah, William, I'm super excited about that.
So.
Prasanna Malaiyandi: Does it count as meeting if you go visit
the grave site of the person?
W. Curtis Preston: Oh, that's just wrong.
That's
Prasanna Malaiyandi: I'm just
W. Curtis Preston: That's just wrong.
Uh, yeah.
So, uh, just help me, help me keep my, my heart pitter Pat.
I'm definitely a, definitely a fan and meeting him, uh, will be very, very cool.
Uh, this week.
Prasanna Malaiyandi: so wait, what is your, if you got a chance to ask
him a question, what would it be?
W. Curtis Preston: Oh, it's definitely not gonna be one of
those, like an episode 57, you know?
Um, wow.
I'm not prepared for that question.
I'll have to think about that.
Prasanna Malaiyandi: Wow.
Did I
stump Curtis?
W. Curtis Preston: You did, you stumped me.
Yeah.
I'll have to, I'll, yeah, I'll definitely, you know, I'm gonna be, I'm gonna be so
nerded out, like I'm, I'm gonna be, yeah.
Um, I just, I, if I get to say two words to him, I'll be,
you know, I'll be like, hi.
Um, you know, I, yeah.
I, I just hope I don't do, like, I've met a lot of famous people and
so many times I've been like, chill.
But I remember there was this one person that I just randomly ran into in an
airport and I literally screamed their name like a, like a 10-year-old girl.
And, um, that was very embarrassing.
I just hope I don't go, William, that would
Prasanna Malaiyandi: okay.
I'm sure he is used to it, you know?
W. Curtis Preston: Yeah, I'm sure.
Yeah.
Um, so this week we're continuing our series about cloud disasters
and this one is pretty bad.
Um, you know, and again, this is yet another story that's gonna
prove the point back your stuff up.
Right.
You know, even, even if this is actually, this is a really good.
Story that basically proves that even if the vendor is backing it up
for you and the backups are included as part of the package, something
so catastrophic might happen that those backups don't come in handy.
Does that sound about right, Pana?
Prasanna Malaiyandi: It does, but I have.
Two comments about that.
W. Curtis Preston: Yeah.
Okay.
Prasanna Malaiyandi: the first is, this reminds me a lot about the
OVH story that we did a while ago.
So if you haven't let heard that episode, go back, give it a listen, because it
was also the case with OVH that they said they were doing backups, but
people were not able to restore their backup because they were sitting in
the same data center as a production and there was a fire, so not so good.
W. Curtis Preston: Yeah.
And, and I'm gonna say, so this, this story is about Rackspace, which I'm gonna
say I have no ill will against Rackspace.
I, I feel for the people that had to go through this, uh, the thing I struggle
with is the ways in which Rackspace tried to deflect, blame Rackspace.
The company tried to deflect blame.
Uh, and so based on that, we've got a pretty solid timeline of the
events Now, just, just for color.
And I didn't know this, this part I'm about to say.
I didn't know this until, until I was researching for the story.
Prior to this event happening, Rackspace had already suffered, uh, a sharp.
Decline in value.
At the height of their value, April, 2021, they were a, an $8 billion company.
And by the time this event happened, they had dropped,
over over 90% of their value.
They were then an $800 million company.
Prasanna Malaiyandi: And, as of today's recording, they
are valued at $340 million, which is 5% of where they were with at their high.
W. Curtis Preston: Right?
Yeah.
Um, so, so they were already in sort of trouble and I, I think that may
be why they tried to deflect blame.
So, um, let's start with sort of the, before the story, right?
So before the story, there was something called the proxy, not she
exploit, uh, in September of 2022.
It was publicly announced and it basically, it allowed someone to
gain control of an exchange server.
It was announced September 30th, 2022, November 8th, Microsoft
released a security update but there was a minor issue with the patch.
And Rackspace claimed this was why they didn't install it at that time,
but by November 17th, Microsoft had fixed that, , that issue.
You know, we talk about three things, right?
Password management, patch management, and MFA.
And then if everybody just did this, then it would've stopped.
Uh, you know, it would stop so much.
And this story is so much I.
Evidence of that, uh, because November 17th that minor issue
with the patch was fixed.
So they could have, and in my opinion, should have immediately put on this
security patch because it was such a huge exploit there was a CVA attached to it.
And, uh, well-known within the industry, they should have immediately patched
all of their, uh, exchange servers.
By the way, I should mention what we're talking about is that
Rackspace had a hosted exchange service, not Microsoft 365.
They ran hosted exchange on their own servers in their own, uh, data center.
Prasanna Malaiyandi: Before you continue on, I think it's important to state that
for that September 30th, right, there was a workaround that was deployed, right?
That
pretty much Microsoft was like, Hey, we haven't quite figured out the patch
yet, which will come out November 8th.
But in the meantime, here's a workaround to make sure you don't get impacted,
which Rackspace did apply, apply.
Right.
W. Curtis Preston: they, they did or they did not.
Prasanna Malaiyandi: They did apply the workaround.
W. Curtis Preston: Yeah.
Okay.
Yeah.
Prasanna Malaiyandi: Yeah.
So it's not a permanent fix, but at least
sort of protects you for now.
So then the other thing is, um.
around this time there were actually two exploits, Right.
So there was a proxy, not shell exploit, and then there was
another one, um, O-W-A-S-S-R-F.
I don't know what that stands for, but that's what they
called it, right?
And these two are kind of related.
And so the patch though, that came out in November would have fixed both.
W. Curtis Preston: Right.
Prasanna Malaiyandi: they were applied, but the workaround that was
applied in the end of September only addressed the proxy nutshell issue.
It
did not expl address the second exploit.
W. Curtis Preston: Yeah.
By the way, the OWA most certainly stands for Outlook.
Web access would be my guess.
I don't know what SSRF but stands for, but yeah, it is kind
of complicated that basically.
That there was a patch that the, the PA had, they applied the patch, they would've
fixed at a, at that time, unknown problem.
Um, but the, but they didn't apply the patch.
And then two weeks goes by and then what happened?
Prasanna Malaiyandi: And then on November 29th, Rackspace says that they
were attacked by a group called Play, which gained access to their exchange
environment using stolen credentials, and that they had access to some of Rackspace
exchange environments, which, if I
was a customer on hosted exchange, I would be kind of freaked out.
W. Curtis Preston: Yeah, exactly right.
So they, they gain privileged access of their exchange servers.
We're, not sure if they knew in November, but because they first
notified people December 2nd.
Literally at two o'clock in the morning.
Right.
Based on the, the, the stuff that we have, they, they may have at that point
realized what happened and they were able to trace it back to November 29th.
Prasanna Malaiyandi: And at 2:00 AM right?
This is when they were like, yeah, here's what happened.
We noticed something, and then they just brought everything down, right?
They were
like, yep, we're not gonna allow any more access
W. Curtis Preston: response, right?
The, the next part of the story is the one that really sets it apart.
I don't know any other story like this.
The res, their response was, and, and again, if you think about now that if
you think about where they were as a company, this part maybe makes more sense.
But what they decided to do was they said, you know what?
This is gonna take us a while.
This the, I'm making up words here.
We've been thinking about shooting this thing in the head anyway, and so let's
just move everybody over to Microsoft 365.
So December 2nd at 2:00 AM is when they first started telling people that
they had this problem, and by 8:00 PM that evening, they had made the
decision to move everybody over to 365.
Prasanna Malaiyandi: Yeah, that I could, I would have loved to have
been a fly on the wall in those meetings, right when they were
trying to
W. Curtis Preston: not have wanted to be in the meeting.
Prasanna Malaiyandi: Yeah.
Yeah, I, that's why I said I wanted to be a fly on the wall.
Right.
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: and hearing these conversations because you
know, it must have been a difficult decision to come to, right?
W. Curtis Preston: Yeah, because that would've been a competing service, right?
So if, if it's not obvious, like if you used hosted exchange, you were
very consciously using hosted exchange, not Microsoft 365, and you had reasons
for doing that, and they're like, guys, this is gonna take us a while.
We're gonna move everybody over to 365.
But what did they not move?
Prasanna Malaiyandi: Uh, so there were two things.
They did not move, right?
The, probably the most important thing was their emails,
right?
What people cared about with the hosted exchange service, right?
Because they basically said, we will recreate things for you, make it easy.
So you have all your stuff up and running at Microsoft 365, but we can't
get you back all your emails yet.
W. Curtis Preston: Right.
Prasanna Malaiyandi: Right, so that was one thing.
I think the second thing, and I don't know if they were forthcoming with
this, but the fact that with their hosted exchange implementation, they
offered backup as part of the service,
W. Curtis Preston: Right.
Prasanna Malaiyandi: right?
When they told customers, Hey, Microsoft 365 is where you should
be looking at, I do wonder if they told people, by the way, you need to
figure out your own backup solution.
W. Curtis Preston: Yeah, none of the communications that we found
between them and customers, uh, showed that they'd said anything.
Uh, but the idea that they would, in the middle of the outage basically
abandoned an entire business line.
Move everybody over to 365.
What they did do was they were able, apparently they, they did
automate the process of creating the accounts for them over on 365.
So you, you were able to, um, you know, essentially you, you were able
to start sending and receiving email.
Relatively quickly considering how long the rest of this took within
a day, it looked like uh, or so you were able to send and receive
email using your old email address.
If you were an exchange hosted exchange customer, and now you're on 365.
You just didn't have access to any of the email you had received up to that point.
Prasanna Malaiyandi: Yeah, which I like.
I go back and forth on that.
It's like, great.
I could send and I could see what people are sending me, but I have a lot
of old stuff and I would be freaking out if I lost all of my old emails.
Or a lot of times if these are businesses and organizations, maybe
they have contracts which are being sent back and forth via emails.
W. Curtis Preston: Yeah, like literally stuff that you got today, right?
Stuff that you got yesterday, stuff you're actively working on.
And a lot of people use their email system as sort of a somewhat,
sometimes temporary, sometimes permanent storage system, right?
And, and they're like, I know where that contract is.
It's in the email that I got, you know, three days
Prasanna Malaiyandi: and I can search for it.
I can find it.
Yep.
W. Curtis Preston: You're not able.
Yeah.
So on one on one hand, like, like you said, I would've wanted to be a fly
on the wall because they're saying we need to do what's, we need to get
our customers up and running again.
We need them, we need to get them, be able to send and receive email.
And I think that was a good decision.
Prasanna Malaiyandi: That's probably the most
important thing to do first.
W. Curtis Preston: yeah.
The, the, the bad decisions happened way before this.
In my opinion, this was a good decision, um, because as we're going
to find out in the story, they didn't exactly have a good, uh, backup.
I.
System, um, at least not one that, that I would recognize, right?
So they actually advertise backup as part of the service.
And again, I I, I'm gonna put this out as this is why, when, when I say,
even if the SaaS vendor advertise backup as part of the service, you
might want to consider a third party.
And I'll put an asterisk, especially if they charge extra for it.
Um, I think in this case it was just included as part of the, the, well,
I'm gonna put it, I'm gonna put in a, I'm gonna put it, especially,
I'm gonna put two asterisk, especially if they charge for it.
'cause then that gives you an incentive to pay somebody else instead.
But then I'm gonna say, especially if they don't charge for it, which means they're
probably not spending any money on it.
Prasanna Malaiyandi: Enough money to do that.
Yeah.
I wanna focus on that for a second
because I did remember earlier on in the episode, I said I had two
comments and we only covered one.
So here's a second
W. Curtis Preston: Oh.
Prasanna Malaiyandi: Right is, does this apply If you are using a SaaS
or whatever backup product in and of itself, that you should have a second
vendor because who knows what that backup vendor will do, and maybe you
can never get your data back out.
I understand Rackspace, they offered backup and it, the backups didn't work.
Right now, there are a whole slew of SaaS data protection companies, or you could
roll your own.
Right.
In those cases though, do you have the same recommendation that if I do decide
to use a SaaS data protection company, I should probably use two SaaS data
protection companies because I don't
know if one can get me my data back?
W. Curtis Preston: Yeah, no, that, that's not what I'm saying.
I'm saying it because I'm talking about a SaaS service and then hiring
a SaaS data protection company that puts your data in two different places.
I would, I would not argue.
Having another one, but it's gonna be really, it's a, it's already
going to be a big enough cost.
I think the idea of putting the data in two different, completely different
protection zones, risk factors, you know, earthquake and flood zones, all of
Prasanna Malaiyandi: it already, diversifies it.
yeah.
W. Curtis Preston: Yeah.
I think that already diversifies it.
Prasanna Malaiyandi: so now my second question.
W. Curtis Preston: I'm just saying that if you have a backup, if you
have a SaaS service, like 365, so 365 is about to start offering backup.
Um, and and I'm just saying I still like the idea of a third party copy.
Prasanna Malaiyandi: Yeah.
Okay.
So now my second question follow up to that is there's a lot of cloud public,
cloud providers, right, that people hook into that leverage things like snapshots.
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: So they're just an orchestration layer on top.
They're doing all the data movement.
They're moving your data around, they're taking the copies.
Do you have the same concerns with those as well?
W. Curtis Preston: I do.
Um, basically the, that's why again, I li I like the orchestration
companies and the ones I like best are the ones that ultimately take
a copy of the data outside, right?
We, we use snapshots to orchestrate and to create the backup, and then we use
something else to get the data out of.
You know, your favorite cloud vendor, again, getting it out,
storing it in another place.
Second best to that would be storing it in another region, in, in another account.
But you know, this is just the, basically I see, like, I think
of like, let's say AWS as NetApp.
Right.
So I need another copy, a final copy of the data that isn't on AWS because
of rolling code concerns, right?
So you get a bug and it, and it rolls, uh, you know, and takes
out both primary and the backup.
Prasanna Malaiyandi: then are you also concerned though, because a lot of
these SaaS data protection companies are built on top of the big clouds.
W. Curtis Preston: So I do, I do want to, um, then make sure that they're
stored in different regions and whatnot.
I, you know, I can only, I can only,
Prasanna Malaiyandi: you can only go so far.
W. Curtis Preston: that, yeah, you could only go so far, right?
It'd be like the same.
The same would be true if we weren't talking to cloud and we were saying
they also, your backup service also uses Soliris, right At, at some point there's
a risk that you just can't get away from.
Prasanna Malaiyandi: Sorry, I, I know we never talked about that before, so
I was very curious about your take on
W. Curtis Preston: Yeah.
Yeah.
Okay.
So, uh, so we were talking about December 2nd is when, um, they, you
know, when they made this move, right?
Um, the, the, they first mentioned that they believe it was a
security incident on December 3rd.
Uh, and then, uh, December 2nd or December 6th, they say that
it was a ransomware incident.
And then finally, 14th, they revealed the attack was from a, their, their
words financially motivated threat actor.
Um, so we don't know the details, uh, of, you know, the extortion, but
there was some kind of extortion.
We also don't know whether or not they ultimately, I.
Paid that money.
Um, you know,
Prasanna Malaiyandi: and, and this.
W. Curtis Preston: I, I, I advise as much as possible not
to pay the money, but go ahead.
Prasanna Malaiyandi: and this goes just the earlier thing about talking
about a security incident and then changing it to ransomware incident.
I get it.
But I know you and I, we've talked in the past about vendors or companies
should be more transparent to a certain extent about what's going on in order to.
Build confidence in the public in terms of they have things handled,
they're figuring things out.
It's okay.
And so I wanted to get your take on that messaging that came from Rackspace.
Do you think that caused, like, do you think that would've caused a
lot of concern for customers or the public in terms of their ability
to handle things first, calling it a security incident and then a
ransomware, and then sort of this back
and forth on data?
I.
W. Curtis Preston: I think it's possible that they just,
they revealed what they knew.
Like you do have to be careful saying only what you know for sure.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: The thing that needs to be understood at this point as they move
through the story is that be like, on one hand, I'm glad they did what they did.
They moved everybody over to 365, got everybody working, but
everybody's gonna be clamoring for their emails right from the previous.
Uh, but because they did it the way they did it, so it, they, if they
had migrated exchange into 365, they could have brought the data with them,
but it would've taken longer because they still, they had dead servers.
Right.
Prasanna Malaiyandi: Yeah.
They don't have the data yet.
W. Curtis Preston: A migration takes a while too.
Right.
But because it did it the way they did it, the only option at this
point is to create PSTs of individual users and then import those PSTs
to those users on the other side.
Um, and so they, they said on December 18th that they had, um, they
had created in, um, that I, I, I.
I'm a little confused where like on one hand they said that they had, I think
they had figured out a way to start restoring the affected exchange servers,
but they hadn't yet restored all of them.
Right.
And this is when they, when they first announce, they're like, so
here's how this is gonna work.
We're gonna restore an exchange server.
If you're on that exchange server, you will then be able to export,
uh, A PST file, and then you'll then be able to download a PST file for
each user, and then you will then be able to upload that into 365.
And they first announced this on December 18th, and then December
Prasanna Malaiyandi: already two weeks after they shut down the service.
W. Curtis Preston: Thank you very much.
That is two weeks after they shut down the service.
Right.
Um, and uh, it also, it's important to understand that by exporting
it as a PSD and then importing it, it's not gonna be a perfect restore.
'cause , I'm pretty sure that you're gonna lose folders and
all of that in this process.
Uh, not to mention the fact that it's just you're gonna lose metadata too.
Right.
Prasanna Malaiyandi: So, you're so in my mind I'm thinking, oh, this is great.
It's a way for people to do things.
Right.
And then
on December 20th, they said that they had just begun testing
the above recovery procedure.
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: Yeah.
So they hadn't actually tried it out or done anything.
They're kind of shooting from the hip, and I get it.
They're urgently trying to figure out how to get the data back for the customers.
But at the same time, it doesn't inspire confidence.
W. Curtis Preston: Yeah, and the thing is that if they had prepared for this
event eventuality back in the day, they could have created a different procedure.
Because there are backup software products that allow you to,
um, directly extract PST data.
From a backup, right?
It it does.
They, they do exist.
It does happen.
So if they had tested this beforehand, they wouldn't have
done it the way they did it.
They would've figured out a way to directly, instead of restoring
exchange servers and then saying, Hey, customers, go get your PSDs.
They would've been able to just directly create the PSDs.
They would've figured that out beforehand.
Um,
Prasanna Malaiyandi: and then it's just a matter of
W. Curtis Preston: think that would've been a much quicker method.
Prasanna Malaiyandi: Yeah.
And they would've just had to execute on it rather than trying
to figure all this out, because I don't know if a lot of folks.
Understand when you're trying to recover from a ransomware attack, right?
You're not only trying to figure out everything that went wrong,
but also all the extreme pressure you're under lack of sleep, right?
People yelling at you possibly, right?
Worrying about, am I gonna have a job after this?
Yeah.
W. Curtis Preston: I can't imagine the number of people that were, you
know, companies that were yelling about how much money they were spending
and blah, blah, blah, blah, blah.
Right?
Um, yeah.
So December 22nd is the first day that they notified some customers that they
could start retrieving some of their mail.
This is three weeks since the incident, right?
Um, and it wasn't until January 5th, which is another, what, two weeks I.
They said they had 50% done.
I mean, this would, this took a while.
Prasanna Malaiyandi: Yeah.
So here's my question, Curtis.
I know we talked earlier about, okay, they should have had
these procedures documented.
They should have thought about them, right, and tested it
out.
So then they had a process, Do you honestly think people think about
these scenarios and walk through?
Because normally when you're thinking disaster recovery or
backup and recovery, right?
It's like, oh, I lost an email, or I lost a part of something, or someone
accidentally deleted a user, right?
I think that the mind needs or people's mind, people need to change.
And start to start thinking about some of these cases.
But when they designed the system, do you think that like they were like, Hey,
I wonder what'll happen if the exchange servers all get hit by ransomware and
we have to rebuild and all our customers are gonna leave us for Microsoft 365?
W. Curtis Preston: So I'll answer that in two ways.
One is probably not right, and two, they probably should have.
Right.
One of the things I talk a lot about in, you know, when I'm talking about
how to design your systems and stuff is that go get the most scary person
in your environment and have them.
Come up with scenarios, right?
Come up with recovery scenarios.
This is the whole point of tabletop exercises, right?
You get those super negative people that that interject.
Well, what if, well, what if, you know, we're down for weeks?
Make
Prasanna Malaiyandi: that's like me,
W. Curtis Preston: about this a lot.
Make the decision upfront that if we get hit by ransomware.
We're gonna immediately move everybody over to 365, and if
we do that, we're going to need to do it to recover this way.
They, they should have been able to foresee this decision.
Right, because by doing it the way they did it, again, and I don't disagree with
the way they did it, but by doing the way they did it, they necessitated this weird
double, you know, double or two step, super painful restore method that put
a lot of the work onto their customers.
Uh, and it's, it took them months.
Right.
Um, before, you know, before everybody was at, at least able, we don't even know,
um, exactly how many customers actually, were able to successfully recover.
Prasanna Malaiyandi: Yeah, I think they said it was something like 3000
customers were impacted in their
hosted exchange environment.
Right.
W. Curtis Preston: right, right.
They said this was interesting is that they also said that play, or remember
play is the name of the ransomware group.
They accessed to use their words, the email of 27 customers and said there
is no evidence that they read it.
Um, know, the, the, what does that mean?
Right?
So again, this is where you, you have to look at like legalese.
They have.
The fact that they have no evidence that they read it doesn't mean they
didn't read it, it, they don't have evidence that they didn't read it right.
You can't prove a negative, right?
They downloaded the email of 27 customers.
And so it's likely that there could be, you know, there could
have been secondary attacks where play goes after the customers.
We didn't get any news of that, so maybe it didn't happen.
But, um, you know, just when you see messages like that, we have no evidence
that that doesn't mean it didn't happen.
It just means that they, they can't prove it.
It happened right?
Prasanna Malaiyandi: Do you think, since you're just mentioning about
this, do you think there are possible ways they could have leveraged
security offerings, encryption, other things to protect these emails?
So even if play did attack their exchange server, I guess it depends
on what level they actually were able to exploit the exchange server.
W. Curtis Preston: I, I mean, they had admin access to exchange.
Prasanna Malaiyandi: that all bets are off?
W. Curtis Preston: far as all bets are off.
So even if there was encryption, I don't know if, if exchange has the
ability to encrypt it or if they could store the data on encrypted drives.
It doesn't matter once you're inside the application.
Right.
Because the, the, the data, it doesn't have record level encryption or anything
like that, that I'm aware of, but um.
Yeah.
Um, and, and the, the, the other part, and this is by the way, this is the
part of the story where Rackspace tries to shift blame by saying,
well, hey, this exploit that, that we got hit by was a zero day exploit.
Which is true in that it was an unknown exploit at the time that they got hit.
But if they had put the patch on that they should have put
at a minimum two weeks prior.
They wouldn't have been because that patch fixed the unknown problem at the time.
Prasanna Malaiyandi: Yeah.
It's, I think, I know we always talk about it on the podcast.
It's patch, patch, patch, right?
When something comes, especially something with this level of severity, I.
Right.
It's like the log four J stuff that came out, what, 2021 December
and what a mess that was as well.
But everyone realized how severe of a security issue it was and
started patching their systems.
I think something similar needed to happen here as well.
it.
the fact that I get it, large customers, you have hosts, you need
to schedule downtime, but a high severity security issue like this
that was being exploited actively, like there's no reason you should have
waited two weeks to apply a patch.
W. Curtis Preston: Yeah.
And, and this, this is kind of what I wanted to go at here.
Uh, you know, we, there's a lot of lessons that could be learned from this incident.
One is third party backup.
If customers had it's had third party backup, which was totally
possible, they would be done.
I.
They would've, they would not have suffered this outage, um,
just like the Salesforce outage, um, in a previous episode, right?
If they had third party backup, they could have fixed it in minutes.
Um, the, and the other is this really hammers home when
there's a really severe exploit.
That is well known and there's a patch introduced for that exploit.
You need to put it in now.
Not.
Oh, we've got our patch management system.
We do this once every two weeks or whatever the, you know, whatever it is.
Look at this company basically, you know, a month from the, the announcement of the
patch, two months from the announcement of, uh, the actual exploit they got hit.
Um, and if they had simply just put the patch in.
When it was available, and, and I, I realized that it was available
sooner and there was a minor pa, but, but that, that issue was fixed.
There was two weeks between the time the, the patch was fully
fixed and fully available.
And the, when this actually happened, and it, it, it was yet another, this
was already a suffering company.
And if you look at the, the stock value of.
Um, rack Rackspace.
It had another sharp decline.
What, what did you say?
Wasn't it 15%
Prasanna Malaiyandi: Yeah.
In
W. Curtis Preston: on December 2nd, which is the day they announced
that this had happened, right?
And so, you know, they were already suffering and this created
yet another, uh, decline in the and which did not recover.
It did not recover from that sharp decline.
Just think about this, think about you.
You really have to prioritize that patch management system, right?
You really have to make sure that you put in patches as soon as they come out.
Uh, you know, high level, high exploit.
Patches need to go in.
And again, I'm gonna once again say that I think the backup system needs
to be at the front of the line, right?
Patch those first, because that's the last line of defense.
And then make sure you, you know, in this case, the patch only applied to exchange,
but this is the point that we make.
If they had just put the patches in, this event would've never happened.
Prasanna Malaiyandi: Yeah.
And you would still have 3000 customers using Rackspace hosted exchange, right?
I.
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: Yeah.
The other comment I was gonna make, Curtis, is
Given that they probably had the managed exchange solution for quite a while,
I wonder if they ever had a process.
I know we've talked about on the podcast of going back and looking
at their disaster recovery plans or their backup and recovery procedures.
Or if it was just sort of, Hey, we created this once, it should be fine.
We never have to really use it, so we'll never go back and make sure
it's up to date and all the rest.
W. Curtis Preston: Well, I mean, clearly they didn't, they didn't.
Account for ransomware, right?
A typical DR recovery scenario would've worked fine here, right?
If they, if the building caught fire, they knew how to re, they had backups.
They knew how to restore their exchange servers.
I think, I still think it took them longer than it should have, but
they had, they had a plan for that.
But that plan ransomware breaks a lot,
Prasanna Malaiyandi: Why would ransomware be different than
the building going up in smoke?
W. Curtis Preston: Because, great question.
Because they likely were still fighting the ransomware itself, right?
When the building goes up in smoke, they could just literally restore everything.
It's gonna take a couple days, but at this point they're, they have an unknown time.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: On the moment of December 2nd, they're like, we have
been taken down and we have no long, no idea how long we're going to be down.
Prasanna Malaiyandi: Yeah.
Might be a.
day, might be a week, might be a.
W. Curtis Preston: Exactly right.
They, they're looking at these, at these other businesses that
are down for months at a time
and they're saying, we don't want to be that.
We don't wanna stop people's email for that amount of time.
Boom, let's do over here.
Had they, had it been adjusted regular, a fire of flood or whatever, they
would, they should have been able to say, this is gonna take us three days.
Three days is gonna stink, but.
It's not enough for us to abandon our entire business
model and move over to 365.
In this case, it was an unknown scenario, unknown amount of
time that they're gonna be down.
And so they decide to do this method that ended up making
everything take much longer.
Right?
Which, and they didn't test for this method.
Prasanna Malaiyandi: so yes, I agree with everything, but I still wanna
go back to another clarifying point.
Couldn't they have treated ransomware?
Like I wanna, I wonder if they actually had disaster recovery plans in place.
Because
if I yeah, and this.
W. Curtis Preston: giving them the benefit of the doubt.
Prasanna Malaiyandi: Yeah, because because in my mind, right, if I
had a ransomware happen, right as a company, there are two options, right?
One is I could try to figure out what all went wrong, rebuild my servers
in the same data center, procure hardware, all the rest, or I could just
treat it like a fire, shoot it in the head, connect completely, disconnect
everything, validate my DR site is still good, that there's no ransomware
there, and then bring everything up or restore on clean hardware, et cetera.
Like
W. Curtis Preston: if there's not ransomware or if there is ransomware.
Prasanna Malaiyandi: even if there was ransomware,
W. Curtis Preston: But the problem is if you have a solid backup system, you have a
recent backup, and then you go to restore it, you know you're restoring the The
Prasanna Malaiyandi: bad stuff.
Yeah, So, but then that should have just been a matter of figuring
out what is good and what is bad.
W. Curtis Preston: Right, which is going to take an unknown amount of time.
That was the problem.
Right.
And, and, um, yeah, I, I don't know if they had a DR plan.
I do, I do ask myself why it took them, the amount of time it took them
to restore all the exchange servers.
I, I just, I just go off the, I just go off the, you know, the message that
I, or the information that I have.
Prasanna Malaiyandi: yeah.
W. Curtis Preston: Yeah.
And so the end of the story is this, uh, the company continued
to suffer, uh, additional losses in the value of their company.
They, they, they went from 800 million, then down to 350 million
now, uh, and they, they are close as of the taping of this episode.
They're close to a restructuring deal.
I don't know if this event had ever happened, if how different
the world would be right now, but it certainly didn't help.
Um, so please, folks, all I can say is, you know, put those
patches in when you, you know, and test your recovery procedures.
All of the recovery scenarios, right?
Prasanna Malaiyandi: And if ransomware isn't one of the recovery scenarios, then
you need to rewrite your recovery plans.
W. Curtis Preston: You need to reconsider your recovery scenarios.
Absolutely.
Uh, all right, well, uh, I think this was a good episode.
Prasanna Malaiyandi: Yeah.
Don't be like Rackspace
W. Curtis Preston: was, you know, ouch.
All right.
Uh, thanks.
Thanks for the chat.
Prasanna Malaiyandi: like, uh.
It was fun, Curtis, and I am sure I will hear all about your,
uh, your, uh, event tonight.
W. Curtis Preston: You, you will be hearing all about it.
You'll probably be getting some live, some live chat or some
live, uh, texts during the event.
Um, and, uh, I, uh, be safe out there folks.
That is a wrap.