Prasanna Malaiyandi: Hi, and welcome to Backup Centrals Restore All podcast.
I'm your host, W.
Curtis Preston, a k a, Mr.
Backup, and I have with me my medical non consultant, uh,
Prasanna Malaiyandi How's it going?.
W. Curtis Preston: Good.
Curtis, how are you?
, how are you feeling?
Prasanna Malaiyandi: a as you know, it's been a rough week or two.
Um, you know, and, and you know, and, and I'm, I'm now down to three,
technically down to two medications.
W. Curtis Preston: Okay.
That's an
Prasanna Malaiyandi: one I can take, one I can take on demand.
Uh, the other, um, or so, so two that I'm supposed to continue
taking until they're gone.
Um, it's been, it's been.
Oh, you know, I don't
W. Curtis Preston: You, you, I do have to say though, you sound a lot better than
like three days ago when we talked so.
Prasanna Malaiyandi: Yeah.
Yeah.
Well in the worry, and we'll see if it happens on the podcast is if,
is if I get to actually, um, you know, as you know, I like to laugh.
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: But if I, if I start to get an actual deep laugh, I will cough.
W. Curtis Preston: Mm.
Prasanna Malaiyandi: There's this cough that, and apparently I did a little
research that it is a common side effect of a leftover viral infection.
Yeah.
So, um, I had a, I had a, um, I got a really bad sinus infection from a tooth
extraction, and then I got the flu.
Um, and um, apparently we, you know, we went to reinvent, right?
Um,
W. Curtis Preston: Couple of weeks ago.
No, last
Prasanna Malaiyandi: yeah.
And it was last week.
Yeah.
And apparently almost the whole crew got sick when they got home.
Um, either flu or covid.
Um, so, you know, I don't know.
It, it may cause us to, I wouldn't be surprised as if, if it causes us
to have some sort of new procedure or policy or something, you know,
because I was supposed to go to another trade show right after,
W. Curtis Preston: yeah, and you.
Prasanna Malaiyandi: that didn't, that
W. Curtis Preston: Well, I know in the news recently, and once again, this
is not medical advice, but just kinda keeping people up to date, right.
In my, the county that I live in, they just actually were flagging it and
saying, yeah, the number of cases are currently on the rise, and especially
with the holidays and all the rest, they're like, be safe, be vigilant.
Mask up if you can stay home if you're sick.
Prasanna Malaiyandi: And what sucks is, you know, like I'm, you
know, I, I, I respect those who want to continue wearing a mask.
I have no issue if you wanna wear a mask.
I Prasannally, and I wore them when it was, you know, when I
thought, When it a, when it was required, and I didn't complain.
I'm currently in the face of like, I am so done with the mask
And so I wear a mask when I, you know, when I kind of have to, uh, but like, I
didn't want to be the guy walking or I didn't want to be, like, out of all of
the attendees of the 60,000 attendees
right there there were like 20 of them were wearing masks and I
didn't want to be the 21st, you
know?
W. Curtis Preston: And I think that's the hard part, right?
It's like peer pressure, societal pressure, right?
It's like, Hey, I shouldn't be any different than everyone else, and.
. I think it's one of those
Prasanna Malaiyandi: And also
W. Curtis Preston: has to sort of judge and figure out their own risk
and figure out what they want to do.
Prasanna Malaiyandi: yeah.
Um, the, um, the mask thing came up on, um, Sebastian Maniscalco,
which is a comedian that I love, and his most recent thing.
Um, He was talking about.
He said that he got the vaccine, not, he didn't get it for him.
He's like, I didn't get it for you.
He says, I got it for the same reason.
Uh uh.
All the Italians.
No, the Italians.
He said the Italians did it because he said we found out that we
couldn't taste food if we got Covid.
So he is like, so that's why he got the vaccine?
Uh,
W. Curtis Preston: But I'm glad that you're feeling better.
Prasanna Malaiyandi: I should throw out our disclaimer that, uh, this is not a
Druva podcast, not a Zoom podcast either.
Where, where, uh, Prasanna happens to work.
Uh, these opinions are ours, and, uh, none of this is medical or legal advice.
God forbid, or, you know, red, I'll tell you what, we, we give official device
advice, but, uh, you know, gadget, gadget,
W. Curtis Preston: and put the holidays around the corner.
By the way, FYI, for those, I don't know when this podcast is
going out, but Best Buy is running a sale on those Ember Smart Mugs.
If you are looking for one, now is the time to buy one.
Prasanna Malaiyandi: Yeah.
I think it's gonna be too late by the time this podcast goes out,
but maybe it'll be in time for you to buy, buy me a birthday, a gift.
It'll be coming up on my, my 57th birthday.
W. Curtis Preston: Dang.
Curtis.
Prasanna Malaiyandi: Getting, getting old up there.
Not as old as Stewart though.
Hey, Stewart
W. Curtis Preston: uh, Stewart, I think we're gonna need to have
Stewart come back on the podcast to have a conversation with you.
Prasanna Malaiyandi: Yeah, yeah.
Um, so I, I wanted to talk about this.
You know, the story that we're gonna talk about today and, um, actually
two stories, but the, the main story, it's one of these where you're like,
it falls into the, are you kidding me?
Category, right?
And it also falls into the, um, the story helps prove a
couple of points of mine, right?
And you, you will see me.
You will hear me talking about those points here on this podcast today.
So we're talking of course.
What are we talking about?
Prasanna?
W. Curtis Preston: Uh, what are we talking about?
Oh, yeah, so, uh, the recent, yeah, the recent outage that happened at Rackspace,
I want to say it was December 2nd,
Prasanna Malaiyandi: Yeah, that was the beginning of it.
W. Curtis Preston: Yeah.
and it's now December 8th and people are still not, or the
service is still not up and running.
Prasanna Malaiyandi: and Rackspace as of now, their official line is,
we don't know when this is gonna end.
We don't know when or if we're gonna be able to restore data.
W. Curtis Preston: So, so, so maybe just a quick background on Rackspace for
the listeners who may not be familiar.
Right.
So Rackspace is a, what would you call 'em?
They're kind of.
Prasanna Malaiyandi: where they have a bunch of racks,
W. Curtis Preston: Well, it's kind of like Amazon before, or AWS before aws, right.
In a sense, they were kind
Prasanna Malaiyandi: Yeah.
They're, they're, yeah.
Yeah.
They're, they're, they're closer to a colo than a, than a cloud facility, I would
W. Curtis Preston: Yeah.
But they do offer managed services, including what got hit,
which is their hosted exchange.
It.
Environment, so it's not Microsoft 365 that they're just sort of proxying
through and buying Microsoft 365 licenses for this is, they're running exchange
servers in their environment, right?
And basically giving you customers a similar sort of experience that you
would get with a SaaS service, right?
So they're managing all of the infrastructure, the email servers,
provisioning accounts, everything else.
And you as a customer, you're just.
email service provided by Rackspace.
Prasanna Malaiyandi: Let me ask you a question.
W. Curtis Preston: Mm-hmm.
Prasanna Malaiyandi: I understand why somebody would, I mean, I disagree, but
I understand , why somebody would use on-premises exchange over Microsoft 365.
What does a company gain by using hosted, uh, Microsoft Exchange?
That I don't, I don't
W. Curtis Preston: So I, I think it comes back to a couple things, right?
One is, are they using Rackspace for other services already?
And this is just yet another thing that they're just using Rackspace for, right?
That could be one.
The second is maybe they have certain compliance regulations or other things
which they feel cannot be fulfilled by.
I, I, I know it's, but it's a very
Prasanna Malaiyandi: that
W. Curtis Preston: it's a very niche.
Prasanna Malaiyandi: no, no, but that's why I ask that question because if
you, if you're not compliant sitting on Microsoft Services, why are you compliant
sitting on somebody else's services?
W. Curtis Preston: be, well, it might be that Rackspace has
found a differentiator, right?
Or provide the value add, right?
Prasanna Malaiyandi: They must, they must.
Maybe it's uptime
W. Curtis Preston: Yeah.
Maybe, well, maybe it's uptime, maybe it's replication.
Maybe they're offering backup.
We'll get to that later, right?
But there are all these other things potentially you could
be adding as a value add.
right?
In addition to just what Microsoft provides, right?
Or maybe it's like an e-discovery compliance style thing that
they're also providing in.
It's hard to tell cuz I don't know what they offer fully for the managed services.
Or it could be maybe there are certain data residency requirements
that aren't met by Microsoft today because of how they operate.
Prasanna Malaiyandi: Speaking of which I'd like to off, I'd
like to announce another sale.
W. Curtis Preston: Okay.
What's your next.
Prasanna Malaiyandi: Uh, right now Rackspace stock is on sale.
Um,
W. Curtis Preston: Oh, Curtis.
Prasanna Malaiyandi: it went from, of a high of five this
week down to three something.
Um, yeah, it's been, it's been taking a hit.
And, and, and also, um, There there's been the announcement of at least one class
action lawsuit, um, on business wire.
Um, Cole and Van note announces filing of Rackspace ransomware
data breach class action.
So let's talk about what you know, cuz there's like a half a dozen,
I don't know, plus stories or you know, various stories out there.
What do we know about the outage?
So,
W. Curtis Preston: So, so far, we know that on December 2nd
they brought down their services.
They said they noticed a security incident that mainly,
Prasanna Malaiyandi: it took, it took them a day, as far as I recall.
It took them a day to say it was a security incident.
W. Curtis Preston: Well, they noticed some.
Issue and they brought down their environment.
Prasanna Malaiyandi: right,
W. Curtis Preston: Um, I believe they only brought down their exchange hosted
environment, not all the other services.
Prasanna Malaiyandi: In fact, I think.
Right.
Yes.
W. Curtis Preston: it was limited to just that.
And so they brought everything down and they kept everything down.
Like you said, they investigated it.
They then published to people saying we had a security incident,
and I think that was December 3rd.
and then they've been doing periodic updates, I would say, of where they're at.
But it's just more of the, we're still investigating.
We're still investigating.
We don't have a time yet, but yeah.
Prasanna Malaiyandi: the comment in the lawsuit, it referred to him
as a numerous, uh, very opaque, um, you know, uh, announcements, right?
That, that, that did, they didn't offer.
We're looking into it, you know, that was basically what they offered.
W. Curtis Preston: and for companies, right?
This is their email, right?
. This is, uh, really critical for a lot of companies in order to do business and
being down for six, seven days with no e t A on when they're going to be fixed.
Or what the process is cuz they haven't even talked about what the
recovery mechanisms are either.
And I know we'll talk a lit bit later, Curtis, about sort of what they're
offering for a Band-Aid solution I guess, right now or, but yeah.
So as of right now though, all those customers are a little hosed.
Prasanna Malaiyandi: Yeah, I mean it's been, we're now on December 8th.
This thing is still ongoing.
I'm hoping that by the time this episode publishes the,
W. Curtis Preston: They'll be back up and run.
Prasanna Malaiyandi: one way or the other.
W. Curtis Preston: Yeah, but could you just imagine as a, like when
you had your company, Curtis, if your email went down for a week,
what would the impact be for you?
Prasanna Malaiyandi: I mean, it just, it is just ridiculous, right?
I, I, I suppose, well, we'd be, you know, really unable to communicate
with outsiders, which is kind of the, the point of a company, right?
Uh, maybe you'd be able to talk to a few people via chat and phone
calls and whatnot, but email.
Is such a critical part of, of a typical company that the idea of email
going down for at this point almost a week or more, uh, is just unthinkable.
I, I, I can't imagine the, the, the cost that they have, uh, that their,
that their clients have incurred.
They're going to incur costs, they're gonna inc.
Reputation costs are gonna incur in financial costs.
Uh, I think when this all comes out, this is gonna be, I think this is gonna
be very bad for the likes of Rackspace.
Now let's talk
W. Curtis Preston: oh, before you get, uh, it's also going to be interesting.
So, two things I wanna throw out there.
One is, it'll be interesting once this is all done, if they
continue to stay in the hosting.
email hosting business.
Right.
That'll be one interesting thing.
I did see a number that said that it's supposed to be a 30
million a year business for
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: Right.
Which isn't peanuts.
But at the same time, if you don't have customers who are coming, right,
I'm sure there are a lot of customers who are like, I don't think I'm gonna
stick around with Rackspace for my.
Prasanna Malaiyandi: yeah, exactly.
Um, so there was this, uh, or is this guy, his name's Kevin Beaumont.
So he is a security researcher and he was affected by the hosted exchange
W. Curtis Preston: Hmm.
Prasanna Malaiyandi: and, um, or he noticed it or something.
I don't know.
Um, I don't know if he, um,
W. Curtis Preston: If he's a customer,
Prasanna Malaiyandi: I don't know if he was effective or, yeah.
But basically he, he just started poking around and he was looking at their.
Their email services and, um, he noticed the version that the, the, the version of
Microsoft Exchange that they were running is apparently very old and is PR is, and
it's before there's, there's something called, uh, proxy Nutshell and that they
were the patches that were available.
Um, according to this, the version that they're running is from August.
W. Curtis Preston: and it was patched in September.
Prasanna Malaiyandi: yeah, which is passed in
W. Curtis Preston: I, I think I read something that said one server was
unpatched, but I believe many of their other servers were patched.
Prasanna Malaiyandi: Well, what I, what I, what he said later on in the article
was, the way this works is all it takes is
W. Curtis Preston: One.
Yep.
Prasanna Malaiyandi: In fact, all it takes is one account,
one compromised account on one.
Compromised system, uh, and then you're in.
Right.
Um,
W. Curtis Preston: which is I think how they moved horizontally across
the entire environment, right?
Prasanna Malaiyandi: exactly right.
Uh, and, and he, and he goes on to, to, to basically point out to people who are,
who actually have managed, um, Exchange to, or you know, whether they're managing
it themselves or somebody else is managing it to basically say, listen, you, you
need to be running this past version.
Right.
Um, and in fact there have been two, there have been two versions of
the software since that August 9th version that they have not patched,
W. Curtis Preston: and it's kind of scary, like you think about patch
management, especially for these critical vulnerabilities, right?
It's how do you schedule downtime to be able to apply the patches in
the right order at the time, right?
Maybe they applied it for all the other systems.
This one maybe fell through the cracks, or maybe they had a problem trying to update
this one, so they're like, ah, we'll just get to it on the next patch cycle.
You.
Prasanna Malaiyandi: Yeah,
W. Curtis Preston: I, I don't know what actually happened, by the way,
this is all hypothetical, right?
Or
Prasanna Malaiyandi: Yeah, it's all hypothetical.
Should, yeah, that, that this is all, this is what appears to happen based
on the information we have available.
And we, we also don't know.
There's no evidence.
And again, Rackspace isn't helping with its o o opacity.
Would that be the, would that be the right word?
Opaqueness.
O I think opacity seems like the right word anyway.
Uh, they're not really saying much.
Um, but we don't know.
Also, we don't know that,
like, It it appears that they had a hu that, that they had the, this
unpatched or yeah, this unpatched server.
But we don't know that that's what caused the outage.
But we do know well.
Based on the information we have, we do know that they weren't up on
their patches, which is, you know, this is one of the reasons why you
go with a hosted provider, right?
Is that they're, is that they're gonna handle all of these problems
that you don't, you know, you don't want to handle yourself.
W. Curtis Preston: Yeah, you're like, I wanna focus on my business and not set
it up and managing email infrastructure.
Prasanna Malaiyandi: It's very to, to, to put it mildly.
It's very disappointing on the part of Rackspace.
Um, I, you know, I actually did some, um, some consulting work for
Rackspace back a hundred years ago.
Uh, and they seemed like a, you know, a smart group of people.
I was in there to put in a, what would now be a competitor, a Druva.
Um, they were, They were, it was, it was a rocky install.
That's what I remember, . It was a rocky install.
Um, but it's interesting.
So, so, so we have this thing with the patching that we don't have it, it appears
that they were behind on their patches.
Why, why did they patch most of them, but, but not one of them.
Why?
I don't know.
It means that they've, that would, that would suggest that they don't
have a, a strong patch management.
Process.
Right.
Um, and then the other question is, why is it a week in and they haven't been able to
restore their services and there's really only one valid answer to that question.
And by valid,
W. Curtis Preston: I think there are two.
Oh,
Prasanna Malaiyandi: okay.
W. Curtis Preston: No, actually you're right.
No, there are no valid answers.
Really.
Prasanna Malaiyandi: We, there are no valid answers, but there
are, there are phrases that can answer the question that I posed.
They, I don't think they're valid.
I, I, I would say one, you know, they, they don't want to pay their ransom.
Right.
Um, but the, and I hope for their sake from a company, History standpoint,
I hope for their sake that this isn't an extortion ransomware situation.
Right?
W. Curtis Preston: Yep.
Where someone has exfiltrated some data and now they're like, Hey, pay up.
Or
Prasanna Malaiyandi: cuz if you're hosted exchange, you've got dozens,
hundreds of companies in there.
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: By the way, they're saying that this is, um, a portion of.
Hosted exchange environment, by the way, but apparently
the outage is affecting all of
W. Curtis Preston: Yeah.
I think another,
Prasanna Malaiyandi: yeah.
Go ahead.
W. Curtis Preston: I think another valid reason that it could be right is, do
you remember when we had Tony Mendoza from Spectralogic on when he was talking
about how they recovered from ransomware?
, right.
I think a lot of it is, do you need to bring in those experts?
Hopefully they had cyber insurance.
Prasanna Malaiyandi: They, they according well, yeah.
What we do know is that they have a quote leading cybersecurity
firm, , and they're helping them out.
Yeah.
W. Curtis Preston: So hopefully they're able, I thought it
was Mandiant or one of those.
I may have stumbled across that in some article
Prasanna Malaiyandi: Yeah.
I just saw the phrase, leading, leading company.
W. Curtis Preston: So I think.
one is it takes time to bring them up to do the investigation, to
figure out what they need to recover.
So I'm thinking that doing that and then also making sure you have, because
that's the one thing that stood out for me from talking to Tony Mendoza, was you
needed to have sort of clean machines that you could start using for restores.
Otherwise it's just gonna continue propagating, right?
And so finding the hardware, right, because this is a managed company, right?
So they probably have servers in rack, so procuring the
Prasanna Malaiyandi: You would think if they had, if, if, if a company would have.
, you know, I, I'm, I'm guessing they have an entire space just for racks.
Uh, uh, uh,
W. Curtis Preston: so, but, but finding the equipment right and then identifying
the points in time, which are valid.
Right.
Standing up the infrastructure, the networking, right, making sure that
there are no further security issues.
Granted, seven days seems kind of long, but I thought
that's kind of what Tony said.
It took 'em before they started getting up
Prasanna Malaiyandi: two weeks.
W. Curtis Preston: two weeks before, well before they were fully up and
running, but I think it was about a week before they could finally
start doing restores and bringing up
Prasanna Malaiyandi: I, I, I don't remember by, by the way,
that was a really good episode.
Uh, it was ransomware victim tells their story.
Um, the, um, I, it's funny, you, you, you, you actually gave
them a, that's an actual valid
W. Curtis Preston: that that is a valid, that's why I said yeah,
Prasanna Malaiyandi: Yeah.
Yeah.
That's pro.
Possibly the only valid answer is, hey, This is hard.
W. Curtis Preston: Yeah.
But they should be more transparent, you know?
Prasanna Malaiyandi: they should be more transparent.
Why don't companies understand that?
I don't get that.
Um, but the worry is because due to the lack of transparency, is that,
is that they don't have backups.
They're so, so what, what have they done in the meantime for
customers who just can't do.
W. Curtis Preston: Yeah.
So this is the thing that I saw in, that they were actually recommending is they're
like, Hey, we went and procured Microsoft 365 licenses for you, so you can stand
up your email and continue operating.
And they're gonna help them set up, uh, what is it, forwarding rules.
So
Prasanna Malaiyandi: Yeah, so that incoming email will go to
W. Curtis Preston: And they have like the Microsoft, uh, team on
standby to help customers, right.
Prasanna Malaiyandi: Meanwhile, there's, there's all these, there's all these
email servers out there that have been trying to send mail to cus to, to, uh,
Microsoft, uh, to Rackspace customers.
And they're we're, we're trying to send the mail.
We're trying to send the mail.
Nobody's taking it.
W. Curtis Preston: yeah.
Well, and the thing is, even with this solution, right, it's only going forward.
You're receiving new emails, right?
All your old stuff is, oh, who knows what happened to that, right?
Prasanna Malaiyandi: they did say that they provided for some
customers an archive of, of emails.
W. Curtis Preston: I thought they said they were trying to
get to that, but they don't yet.
Prasanna Malaiyandi: well, I thought that they gave it for some, but not all.
Um, by the way, I was, I was, that, that's a clue for me, the fact
that they said they had archives of the email, but not backups,
W. Curtis Preston: I was looking at the article or from
Prasanna Malaiyandi: Uhhuh,
W. Curtis Preston: we are working to provide customers with archive of
inboxes where available to eventually import over to Microsoft 365.
Prasanna Malaiyandi: Right.
W. Curtis Preston: So it may
Prasanna Malaiyandi: the word archive is a bit,
W. Curtis Preston: concerning.
Prasanna Malaiyandi: you know?
Yeah.
Uh, archive is different than backup.
You know, we have an episode on that.
Please go listen to that if you have it.
Um, the, um, Yeah, I don't know.
I'm just not, I'm not, I don't have a good feeling here on,
W. Curtis Preston: a little queasy in your tummy and it's not from being sick.
Prasanna Malaiyandi: Yeah.
Yeah.
Um, I, I don't, I don't know what they're doing over there.
Um, like I said, it, it all, it all starts with why didn't they have the
service patch in the first place?
That's, that's the first concerning thing.
Um, the fact that we're a weekend and they're not saying that, you know,
if they had said, l listen, we're we are, we have to do a server by server
scrub to, to verify that the, you know, we've identified the malware.
We now have to do a server by server scrub to make sure the malware isn't anywhere
else and we need to wipe the servers.
You know, but they're not saying anything.
They're just saying We're, we're working on
W. Curtis Preston: yeah.
The other thing I wanna know,
Prasanna Malaiyandi: any, yeah.
Go.
W. Curtis Preston: the other thing I'm wondering though is I know
you're talking about backup, right?
And restoring data.
Do they not have a DR environment?
You know, I know we, you know, we talk about this all day.
Or maybe their DR.
Environment was compromised as well.
Right, and so we always talk about, right, you need a DR environment for
situations like this where you can quickly spin up and continue operating
rather than trying to go back and restore your data, keep those backups
just in case so you can restore them.
But you really should have a DR environment so you're not
spending six plus days trying to bring up your environment.
Prasanna Malaiyandi: But if you have a DR environment and you're doing
host exchange, you're gonna have to be doing some sort of real time
replication in order to, to have that DR environment be, uh, you know, effective.
W. Curtis Preston: Yep.
And maybe they're doing
Prasanna Malaiyandi: not, I'm not saying I, I can't.
W. Curtis Preston: Yeah.
And, but even that replication, right?
If it, so assuming the ransomware did not go horizontally into that DR
environment, which is a big assumption.
If you were doing exchange level replication at the application
level, hopefully your DR site shouldn't have been compromised
Prasanna Malaiyandi: It has a delayed, there's a delayed replication
W. Curtis Preston: And you're also doing it at the application level, right?
Rather
Prasanna Malaiyandi: Right, right,
W. Curtis Preston: at the database object level.
Right.
So
Prasanna Malaiyandi: Yeah.
Yeah, that makes sense.
It's interesting, you know, it's like, well, did it, did it attack
exchange, or did it attack windows?
We don't know.
We don't know anything.
Prasanna, why don't we know anything?
W. Curtis Preston: That's how these things go.
Well, hopefully they publish more information.
I'm not holding my breath for that though, but I think it could be a good
learning opportunity because I wanna say that the US government, right, had
a big push for patch exchange servers because of these specific issues.
Like a couple months ago, I think there was like a cisa.
Prasanna Malaiyandi: well that would've been helpful a couple of months
W. Curtis Preston: I thought so.
I could be wrong, but I thought there was something
Prasanna Malaiyandi: So, uh, I want to tack on.
, um, basically put the, put, you know, on, on a related note to this, cause
I'm tired of talking about Rackspace.
It's too depressing.
Uh, I wanna talk about a company I'd never heard of before.
Um, they're referred to as, uh, south Korea's, um, Google, and that
is the name of the, I don't know if, I don't know if I'm pronouncing
it right, but it looks like Ka.
um, like, it's like, it's like the way cocoa is spelled properly, but except
with Case That's the, so I'm sort of cacao, that's how I'm pronouncing it.
Like, like, you know, like the, the Bean for, for chocolate.
Um, so they,
W. Curtis Preston: This article we found on the register,
Prasanna Malaiyandi: yeah.
On the register.
Um, so, oh, it's from si Simon.
I know Simon.
Hi Simon.
I don't know if he
W. Curtis Preston: what was it, title of the article just for
Prasanna Malaiyandi: Uh, well there's two articles.
There's Data Center Fire Takes Out South Korea's Top Two Web
Giants, and then a follow up article that was back in October.
A follow up article is it's 2022 and a Korean web Giant only now decided to
write a Dr Plan So, uh, the first part is reminiscent of, um, the O V H fire,
and that is that these guys apparently, What I'm, again, what I'm deducing there,
there were two web giants, Neve and Cacao.
They both experienced service interruptions after a data center that
hosts much of their infrastructure was shut down by a Sunday fire.
So they, they are Google-like, but they're, they're, uh, in that they
have like Facebook and messaging and a lot of different stuff like that.
I don't think they're search engine type stuff, but they.
They and this other web giant, um, were taken down by this data center fire,
which as we, again, we, we can only infer from things they, they're, they're
hosted in somebody else's data center.
It's not their data center.
And, and it's not a cloud data center
W. Curtis Preston: it's like a cola.
Prasanna Malaiyandi: a data center.
Yeah, it's like a colo and then the, the, the.
Sort of outcome from that is that this company has decided to build
their own data center and they think now that maybe they should get a DR.
Plan.
W. Curtis Preston: it's, it's surprising how often sort of backup
and, and Chris, I know we've talked about it so many times, right?
And, and we've had folks on talking about, okay, this is why it's important to have
a plan and to have stakeholders aligned.
But no one ever thinks about DR and backup until it's.
Prasanna Malaiyandi: Yeah.
Uh, so I guess what happened here is that the Korean government, south
Korean government sternly criticized cacao and its c e o resigned.
So they, um, they've unveiled a strategy to create tech teams, develop
a business continuity plan, and built disaster recovery facilities.
Um, you know, and the, the, the line in here, it's a little odd in this day and
age that a company of their size doesn't have these things in place already.
But this idea that the blaze, you know, the, the, so the impact of
this on everyday Koreans is that they all rely on this service.
This is what, this is why I saw it as very much this related.
They all rely on these services of like, , uh, communi for communications, right?
Um, it says they've assumed the status of de facto telecommunications
infrastructure, right?
W. Curtis Preston: And so when you go down, that affects so
many people day in and day out,
Prasanna Malaiyandi: yeah.
W. Curtis Preston: yep.
Prasanna Malaiyandi: Yeah.
So, um, so you have two companies where they were, they were hosting
things, they were providing services.
Many other people.
In one case it's thousands of companies.
In another case it's millions of, of everyday citizens that used them.
And then both of them lost data due to some sort of thing.
Uh, one was an attack, one was a fire, and neither of them had
a plan to, to work around that.
W. Curtis Preston: And, and, and
Prasanna Malaiyandi: What am I doing here?
W. Curtis Preston: they should listen to the podcast.
That's what they should be doing.
But,
Prasanna Malaiyandi: what it is.
W. Curtis Preston: but I think.
I want to give maybe the benefit of the doubt that maybe they had
planned for some types of outages, but they may not have planned for
these types, these specific ones.
Although in the case of
Prasanna Malaiyandi: mean like the loss of an entire data center,
W. Curtis Preston: Right.
May,
Prasanna Malaiyandi: what a Dr.
Plan is,
W. Curtis Preston: well, maybe they were relying on the other, like the host.
Data center infrastructure deal with that.
Great.
Hey, should have asked a question.
Yep.
Prasanna Malaiyandi: Right?
Like, you know, we talk about this all the time.
Right?
And, and, and by the way, you know, so I, I mentioned in the ear in
the early part of the podcast that this was gonna touch on a couple,
couple of hobby horses for me.
One is that, You should not trust your supplier of services to be
the backer of services, right?
W. Curtis Preston: Unless it's written in contract or even then
Prasanna Malaiyandi: What?
No, no, no.
I don't think so.
I, I think that, I think there's a, um,
W. Curtis Preston: separation
Prasanna Malaiyandi: there's an, yeah.
Yeah.
That, that backup.
Backup should be a backup, right?
Um, that it should be done by somebody else.
This is a, this is.
Sort of a separation of powers, um, separation
W. Curtis Preston: Or shared responsibility model.
Prasanna Malaiyandi: Shared responsibility, right?
I, I think it's just best practice to have your primary stuff by one vendor and
your backup stuff done by another vendor.
That's the way it's always been done.
, and then suddenly we start throwing that up in the air as to maybe
that's, you know, maybe we can save money if we use the services, right?
I think back to O V H and what happened there where we, where they lost the data
center, it turns out their backup stunk because they were just in the corner.
Right?
W. Curtis Preston: But I, I wanna challenge you that, on that challenge.
You on that though, Curtis, right?
I
Prasanna Malaiyandi: right.
W. Curtis Preston: I think that, yes, 90% of the time don't trust the vendor, right?
The single vendor to do the right thing.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: I feel if you can ask the right question.
, do the tests, prove it out, get it in contract,
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: the full testing, which is effort, right?
It might just be easier to go do it yourself, right?
But I think if you can get it in terms and verify it and prove it out,
I Prasannally think it's okay to go with, like to trust that single person
Prasanna Malaiyandi: Do you have an example of that?
Because I don't,
W. Curtis Preston: AWS backup.
Prasanna Malaiyandi: um, yeah, so, uh, Well, but, but that's not the same though.
That's AWS Backup is a feature that you use,
W. Curtis Preston: Yes.
Prasanna Malaiyandi: There's no, there's no AWS backup service that
you rely on, and by service, I mean, you, you understand what I'm saying?
It's not like
W. Curtis Preston: it's not a managed service.
Prasanna Malaiyandi: it is, it is a feature that you
use and you, you are going
to use.
The snapshots and you're responsible for managing it.
What I'm talking about is relying, you know, putting all
of the burden on the vendor.
I don't know of a service where,
W. Curtis Preston: But don't they like, like if you, there are companies
that do manage services, right?
Where they take over your infrastructure and they operate it
Prasanna Malaiyandi: yep.
W. Curtis Preston: case.
So, but they'll bring in multiple vendors.
Right?
Sometimes.
Prasanna Malaiyandi: That that's a different, that's a different thing.
I, I, let's just say, you know, um, I'm, I'm just, I'm just saying if,
if you're asking me my druthers,
W. Curtis Preston: Yeah.
No, no, it's no.
Prasanna Malaiyandi: phrase, I would rather you have one vendor,
dut and another vendor do B,
W. Curtis Preston: A hundred percent agree a hundred percent.
Prasanna Malaiyandi: Yeah.
Um, I, I can't think of a situation.
So for example, I was going to perhaps, Think of, so it's one reason, like, I
don't like whatever Microsoft is currently supplying regarding Microsoft 365.
First off, they don't have a backup service for Microsoft 365.
They do have an archive service.
They have an e-discovery service, but they don't have a backup service, so,
W. Curtis Preston: but if I took, uh, who is, who is, uh, Eric?
Firstly, what was the company he worked for?
Right?
Would you say that they are a service, a managed service provider?
Prasanna Malaiyandi: Yeah, they are.
Uh, well they were, yeah, they were.
Yes.
They're, cuz they do, they do private cloud, essentially infrastructure.
Right?
W. Curtis Preston: but then they're bringing in other vendors and they're
offering the entire package, right?
Prasanna Malaiyandi: yeah.
Good.
Uh, good, better, best.
Right.
Again, if you're asking me if.
Again, my choice would be to have a different vendor do backup and recovery.
Right?
Because if you find out your vendor is an idiot,
W. Curtis Preston: You're protected.
Prasanna Malaiyandi: It's, you're protected.
Well, the chances of you having two idiot vendors is less, right?
Uh, so that's, that's one hobby horse.
And then the other hobby horse is, I can't believe, um, you
know, that just this idea that.
People relied entirely on their, people just relied entirely under their backup.
Clearly, nobody pushed them on it, right?
Nobody pushed them on, you know, what's your DR plan?
All these people paying 'em all this money, and they're not pushing
them on, what's your DR plan?
W. Curtis Preston: Yep.
Prasanna Malaiyandi: You know?
You know, I don't, I don't know.
I don't know what to say about.
W. Curtis Preston: Dr.
Is Asmar.
Prasanna Malaiyandi: I don't wanna be victim blaming at the same time.
You know, this is clearly Rackspace and ca cow's responsibility.
Uh, but if you are sitting there and you are using vendors and you are , you're
using vendors for stuff, you should be inquiring as to their DR plans and
their ransomware readiness plans and
W. Curtis Preston: Was this covered in your book when we talked
about talking about stakeholders?
Prasanna Malaiyandi: I can't, I can't remember.
I can't remember if it wasn't or not, or if it was or not.
Um, mainly the book was about how to make sure you're ready.
Right?
But I don't know.
So been, it's a depressing, it's a depressing episode.
We should talk about puppies for.
W. Curtis Preston: Well, it's not a depressing episode.
I think this is one of those things, right?
The silver lining is, this is lessons that other companies can look at, and
hopefully they can be like, Hey, we really should be thinking about what we're doing,
and do we have Dr, do we have backup?
Do we have a
Prasanna Malaiyandi: Take this opportunity.
W. Curtis Preston: Yeah.
Or the other thing is if you are using a SaaS service or who, or a managed service
provider, ask them, what are your plans?
Take this opportunity because everyone's gonna be asking, Hey, what is going on?
Prasanna Malaiyandi: Yeah.
If, if, yeah, if you're one of these people that I don't understand,
by the way, that's using hosted exchange in somebody else's data
center instead of Microsoft 365, uh, by the way, feel free to contact us.
Uh, you know, I didn't mention, you know, that, that, you know, you can reach out
to us at WC presson on Twitter or w Curtis Preston at gmail as long as Twitter stays
up and, um, You know, we'd, I'd love to, if, if you've got an answer as to why you
would use hosted exchange over Microsoft 365, uh, I would love to know it.
But if you have that, yeah.
It's time to reach out to that vendor or hosted anything,
W. Curtis Preston: anything.
Exactly.
It's not just email.
Prasanna Malaiyandi: Yeah, I mean, I, you know, this, this podcast is
hosted, backup Central is hosted, but um, I know what the backup setup is.
Hello?
Kopi
Coffee always wants to get on the podcast.
Um, yeah.
So I know what the backups are on that because I, I make them happen.
Right.
But, um,
W. Curtis Preston: And then for your Backup Central's website too, right?
You have a hosted WordPress site, but you're doing the backups yourself.
Prasanna Malaiyandi: Yeah.
Well, it's part of the Sea Sea Panel Sea panel.
It's, it's run by C panel.
And then C panel allows me to configure a backup, which that backup, daily backup
is then copied to, uh, S3 actually, right?
And then, and it's automatically deleted after a certain number of.
Days.
Right.
Um, so it's not, it's not, the backup is not stored.
So that's another example.
Kind of like what you were talking about, like aw s backup.
I don't rely on C panel or my hosting provider to do the backup, by the
way, they do offer a backup service.
See, I do eat my
W. Curtis Preston: That.
That's why I was telling, that's why I brought that up, that example, right.
Prasanna Malaiyandi: They do offer a hosted backup service.
LiquidWeb is the name of my hosting provider, by the way, love.
Been with them now for a really long time and I, you know, they are a actual managed
server service provider for hosting and, um, the, they do offer a backup
service and instead I use the features of C Panel to create a backup, which
is then, um, you know, sent over to s3.
And, uh, I think I keep 90 days even though I think that's ridiculous.
And my Amazon S3.
Is, uh, like $2 and 83 cents, something like that.
W. Curtis Preston: Maybe you should be looking at S three ia.
Your cost might drop
Prasanna Malaiyandi: Yeah.
Yeah.
Cuz I, you know, I've, I've used my backups two or three times, but,
um, yeah, I don't know if it's, I don't know if it's possible, but
W. Curtis Preston: the saving of 35 cents.
Prasanna Malaiyandi: Yeah.
But, uh, yeah.
Anyway, just l just investigate your vendors.
Will you check to see if they're doing, you know, I, I
wish the best for Rackspace.
I wish the best for their customers.
I hope that by the time you hear this, this has all been sorted out.
If it hasn't, dear Lord.
Anyway, well, alright, uh, hope you enjoyed this episode.
Remember to subscribe so that you can restore it all.