Hi, and welcome to Backup Centrals Restore All podcast.

I'm your host, W.

Curtis Preston, a k a, Mr.

Backup, and I have with me my medical non consultant, uh,

Prasanna Malaiyandi How's it going?.

Good.

Curtis, how are you?

, how are you feeling?

a as you know, it's been a rough week or two.

Um, you know, and, and you know, and, and I'm, I'm now down to three,

technically down to two medications.

Okay.

That's an

one I can take, one I can take on demand.

Uh, the other, um, or so, so two that I'm supposed to continue

taking until they're gone.

Um, it's been, it's been.

Oh, you know, I don't

You, you, I do have to say though, you sound a lot better than

like three days ago when we talked so.

Yeah.

Yeah.

Well in the worry, and we'll see if it happens on the podcast is if,

is if I get to actually, um, you know, as you know, I like to laugh.

Yeah.

But if I, if I start to get an actual deep laugh, I will cough.

Mm.

There's this cough that, and apparently I did a little

research that it is a common side effect of a leftover viral infection.

Yeah.

So, um, I had a, I had a, um, I got a really bad sinus infection from a tooth

extraction, and then I got the flu.

Um, and um, apparently we, you know, we went to reinvent, right?

Um,

Couple of weeks ago.

No, last

yeah.

And it was last week.

Yeah.

And apparently almost the whole crew got sick when they got home.

Um, either flu or covid.

Um, so, you know, I don't know.

It, it may cause us to, I wouldn't be surprised as if, if it causes us

to have some sort of new procedure or policy or something, you know,

because I was supposed to go to another trade show right after,

yeah, and you.

that didn't, that

Well, I know in the news recently, and once again, this

is not medical advice, but just kinda keeping people up to date, right.

In my, the county that I live in, they just actually were flagging it and

saying, yeah, the number of cases are currently on the rise, and especially

with the holidays and all the rest, they're like, be safe, be vigilant.

Mask up if you can stay home if you're sick.

And what sucks is, you know, like I'm, you

know, I, I, I respect those who want to continue wearing a mask.

I have no issue if you wanna wear a mask.

I Prasannally, and I wore them when it was, you know, when I

thought, When it a, when it was required, and I didn't complain.

I'm currently in the face of like, I am so done with the mask

And so I wear a mask when I, you know, when I kind of have to, uh, but like, I

didn't want to be the guy walking or I didn't want to be, like, out of all of

the attendees of the 60,000 attendees

right there there were like 20 of them were wearing masks and I

didn't want to be the 21st, you

know?

And I think that's the hard part, right?

It's like peer pressure, societal pressure, right?

It's like, Hey, I shouldn't be any different than everyone else, and.

. I think it's one of those

And also

has to sort of judge and figure out their own risk

and figure out what they want to do.

yeah.

Um, the, um, the mask thing came up on, um, Sebastian Maniscalco,

which is a comedian that I love, and his most recent thing.

Um, He was talking about.

He said that he got the vaccine, not, he didn't get it for him.

He's like, I didn't get it for you.

He says, I got it for the same reason.

Uh uh.

All the Italians.

No, the Italians.

He said the Italians did it because he said we found out that we

couldn't taste food if we got Covid.

So he is like, so that's why he got the vaccine?

Uh,

But I'm glad that you're feeling better.

I should throw out our disclaimer that, uh, this is not a

Druva podcast, not a Zoom podcast either.

Where, where, uh, Prasanna happens to work.

Uh, these opinions are ours, and, uh, none of this is medical or legal advice.

God forbid, or, you know, red, I'll tell you what, we, we give official device

advice, but, uh, you know, gadget, gadget,

and put the holidays around the corner.

By the way, FYI, for those, I don't know when this podcast is

going out, but Best Buy is running a sale on those Ember Smart Mugs.

If you are looking for one, now is the time to buy one.

Yeah.

I think it's gonna be too late by the time this podcast goes out,

but maybe it'll be in time for you to buy, buy me a birthday, a gift.

It'll be coming up on my, my 57th birthday.

Dang.

Curtis.

Getting, getting old up there.

Not as old as Stewart though.

Hey, Stewart

uh, Stewart, I think we're gonna need to have

Stewart come back on the podcast to have a conversation with you.

Yeah, yeah.

Um, so I, I wanted to talk about this.

You know, the story that we're gonna talk about today and, um, actually

two stories, but the, the main story, it's one of these where you're like,

it falls into the, are you kidding me?

Category, right?

And it also falls into the, um, the story helps prove a

couple of points of mine, right?

And you, you will see me.

You will hear me talking about those points here on this podcast today.

So we're talking of course.

What are we talking about?

Prasanna?

Uh, what are we talking about?

Oh, yeah, so, uh, the recent, yeah, the recent outage that happened at Rackspace,

I want to say it was December 2nd,

Yeah, that was the beginning of it.

Yeah.

and it's now December 8th and people are still not, or the

service is still not up and running.

and Rackspace as of now, their official line is,

we don't know when this is gonna end.

We don't know when or if we're gonna be able to restore data.

So, so, so maybe just a quick background on Rackspace for

the listeners who may not be familiar.

Right.

So Rackspace is a, what would you call 'em?

They're kind of.

where they have a bunch of racks,

Well, it's kind of like Amazon before, or AWS before aws, right.

In a sense, they were kind

Yeah.

They're, they're, yeah.

Yeah.

They're, they're, they're closer to a colo than a, than a cloud facility, I would

Yeah.

But they do offer managed services, including what got hit,

which is their hosted exchange.

It.

Environment, so it's not Microsoft 365 that they're just sort of proxying

through and buying Microsoft 365 licenses for this is, they're running exchange

servers in their environment, right?

And basically giving you customers a similar sort of experience that you

would get with a SaaS service, right?

So they're managing all of the infrastructure, the email servers,

provisioning accounts, everything else.

And you as a customer, you're just.

email service provided by Rackspace.

Let me ask you a question.

Mm-hmm.

I understand why somebody would, I mean, I disagree, but

I understand , why somebody would use on-premises exchange over Microsoft 365.

What does a company gain by using hosted, uh, Microsoft Exchange?

That I don't, I don't

So I, I think it comes back to a couple things, right?

One is, are they using Rackspace for other services already?

And this is just yet another thing that they're just using Rackspace for, right?

That could be one.

The second is maybe they have certain compliance regulations or other things

which they feel cannot be fulfilled by.

I, I, I know it's, but it's a very

that

it's a very niche.

no, no, but that's why I ask that question because if

you, if you're not compliant sitting on Microsoft Services, why are you compliant

sitting on somebody else's services?

be, well, it might be that Rackspace has

found a differentiator, right?

Or provide the value add, right?

They must, they must.

Maybe it's uptime

Yeah.

Maybe, well, maybe it's uptime, maybe it's replication.

Maybe they're offering backup.

We'll get to that later, right?

But there are all these other things potentially you could

be adding as a value add.

right?

In addition to just what Microsoft provides, right?

Or maybe it's like an e-discovery compliance style thing that

they're also providing in.

It's hard to tell cuz I don't know what they offer fully for the managed services.

Or it could be maybe there are certain data residency requirements

that aren't met by Microsoft today because of how they operate.

Speaking of which I'd like to off, I'd

like to announce another sale.

Okay.

What's your next.

Uh, right now Rackspace stock is on sale.

Um,

Oh, Curtis.

it went from, of a high of five this

week down to three something.

Um, yeah, it's been, it's been taking a hit.

And, and, and also, um, There there's been the announcement of at least one class

action lawsuit, um, on business wire.

Um, Cole and Van note announces filing of Rackspace ransomware

data breach class action.

So let's talk about what you know, cuz there's like a half a dozen,

I don't know, plus stories or you know, various stories out there.

What do we know about the outage?

So,

So, so far, we know that on December 2nd

they brought down their services.

They said they noticed a security incident that mainly,

it took, it took them a day, as far as I recall.

It took them a day to say it was a security incident.

W. Curtis Preston:

Well, they noticed some.

W. Curtis Preston:

Issue and they brought down their environment.

right,

Um, I believe they only brought down their exchange hosted

environment, not all the other services.

In fact, I think.

Right.

Yes.

Yes.

it was limited to just that.

And so they brought everything down and they kept everything down.

Like you said, they investigated it.

They then published to people saying we had a security incident,

and I think that was December 3rd.

and then they've been doing periodic updates, I would say, of where they're at.

But it's just more of the, we're still investigating.

We're still investigating.

We don't have a time yet, but yeah.

the comment in the lawsuit, it referred to him

as a numerous, uh, very opaque, um, you know, uh, announcements, right?

That, that, that did, they didn't offer.

We're looking into it, you know, that was basically what they offered.

and for companies, right?

This is their email, right?

. This is, uh, really critical for a lot of companies in order to do business and

being down for six, seven days with no e t A on when they're going to be fixed.

Or what the process is cuz they haven't even talked about what the

recovery mechanisms are either.

And I know we'll talk a lit bit later, Curtis, about sort of what they're

offering for a Band-Aid solution I guess, right now or, but yeah.

So as of right now though, all those customers are a little hosed.

Yeah, I mean it's been, we're now on December 8th.

This thing is still ongoing.

I'm hoping that by the time this episode publishes the,

They'll be back up and run.

one way or the other.

Yeah, but could you just imagine as a, like when

you had your company, Curtis, if your email went down for a week,

what would the impact be for you?

I mean, it just, it is just ridiculous, right?

I, I, I suppose, well, we'd be, you know, really unable to communicate

with outsiders, which is kind of the, the point of a company, right?

Uh, maybe you'd be able to talk to a few people via chat and phone

calls and whatnot, but email.

Is such a critical part of, of a typical company that the idea of email

going down for at this point almost a week or more, uh, is just unthinkable.

I, I, I can't imagine the, the, the cost that they have, uh, that their,

that their clients have incurred.

They're going to incur costs, they're gonna inc.

Reputation costs are gonna incur in financial costs.

Uh, I think when this all comes out, this is gonna be, I think this is gonna

be very bad for the likes of Rackspace.

Now let's talk

oh, before you get, uh, it's also going to be interesting.

So, two things I wanna throw out there.

One is, it'll be interesting once this is all done, if they

continue to stay in the hosting.

email hosting business.

Right.

That'll be one interesting thing.

I did see a number that said that it's supposed to be a 30

million a year business for

Yeah.

Right.

Which isn't peanuts.

But at the same time, if you don't have customers who are coming, right,

I'm sure there are a lot of customers who are like, I don't think I'm gonna

stick around with Rackspace for my.

yeah, exactly.

Um, so there was this, uh, or is this guy, his name's Kevin Beaumont.

So he is a security researcher and he was affected by the hosted exchange

Hmm.

and, um, or he noticed it or something.

I don't know.

Um, I don't know if he, um,

If he's a customer,

I don't know if he was effective or, yeah.

But basically he, he just started poking around and he was looking at their.

Their email services and, um, he noticed the version that the, the, the version of

Microsoft Exchange that they were running is apparently very old and is PR is, and

it's before there's, there's something called, uh, proxy Nutshell and that they

were the patches that were available.

Um, according to this, the version that they're running is from August.

and it was patched in September.

Prasanna Malaiyandi:

yeah, which is passed in

I, I think I read something that said one server was

unpatched, but I believe many of their other servers were patched.

Well, what I, what I, what he said later on in the article

was, the way this works is all it takes is

One.

Yep.

In fact, all it takes is one account,

one compromised account on one.

Compromised system, uh, and then you're in.

Right.

Um,

which is I think how they moved horizontally across

the entire environment, right?

exactly right.

Uh, and, and he, and he goes on to, to, to basically point out to people who are,

who actually have managed, um, Exchange to, or you know, whether they're managing

it themselves or somebody else is managing it to basically say, listen, you, you

need to be running this past version.

Right.

Um, and in fact there have been two, there have been two versions of

the software since that August 9th version that they have not patched,

and it's kind of scary, like you think about patch

management, especially for these critical vulnerabilities, right?

It's how do you schedule downtime to be able to apply the patches in

the right order at the time, right?

Maybe they applied it for all the other systems.

This one maybe fell through the cracks, or maybe they had a problem trying to update

this one, so they're like, ah, we'll just get to it on the next patch cycle.

You.

Yeah,

I, I don't know what actually happened, by the way,

this is all hypothetical, right?

Or

Yeah, it's all hypothetical.

Should, yeah, that, that this is all, this is what appears to happen based

on the information we have available.

And we, we also don't know.

There's no evidence.

And again, Rackspace isn't helping with its o o opacity.

Would that be the, would that be the right word?

Opaqueness.

O I think opacity seems like the right word anyway.

Uh, they're not really saying much.

Um, but we don't know.

Also, we don't know that,

like, It it appears that they had a hu that, that they had the, this

unpatched or yeah, this unpatched server.

But we don't know that that's what caused the outage.

But we do know well.

Based on the information we have, we do know that they weren't up on

their patches, which is, you know, this is one of the reasons why you

go with a hosted provider, right?

Is that they're, is that they're gonna handle all of these problems

that you don't, you know, you don't want to handle yourself.

Yeah, you're like, I wanna focus on my business and not set

it up and managing email infrastructure.

It's very to, to, to put it mildly.

It's very disappointing on the part of Rackspace.

Um, I, you know, I actually did some, um, some consulting work for

Rackspace back a hundred years ago.

Uh, and they seemed like a, you know, a smart group of people.

I was in there to put in a, what would now be a competitor, a Druva.

Um, they were, They were, it was, it was a rocky install.

That's what I remember, . It was a rocky install.

Um, but it's interesting.

So, so, so we have this thing with the patching that we don't have it, it appears

that they were behind on their patches.

Why, why did they patch most of them, but, but not one of them.

Why?

I don't know.

It means that they've, that would, that would suggest that they don't

have a, a strong patch management.

Process.

Right.

Um, and then the other question is, why is it a week in and they haven't been able to

restore their services and there's really only one valid answer to that question.

And by valid,

I think there are two.

Oh,

okay.

No, actually you're right.

No, there are no valid answers.

Really.

We, there are no valid answers, but there

are, there are phrases that can answer the question that I posed.

They, I don't think they're valid.

I, I, I would say one, you know, they, they don't want to pay their ransom.

Right.

Um, but the, and I hope for their sake from a company, History standpoint,

I hope for their sake that this isn't an extortion ransomware situation.

Right?

Yep.

Where someone has exfiltrated some data and now they're like, Hey, pay up.

Or

cuz if you're hosted exchange, you've got dozens,

hundreds of companies in there.

Yeah.

By the way, they're saying that this is, um, a portion of.

Hosted exchange environment, by the way, but apparently

the outage is affecting all of

Yeah.

I think another,

yeah.

Go ahead.

I think another valid reason that it could be right is, do

you remember when we had Tony Mendoza from Spectralogic on when he was talking

about how they recovered from ransomware?

, right.

I think a lot of it is, do you need to bring in those experts?

Hopefully they had cyber insurance.

They, they according well, yeah.

What we do know is that they have a quote leading cybersecurity

firm, , and they're helping them out.

Yeah.

So hopefully they're able, I thought it

was Mandiant or one of those.

I may have stumbled across that in some article

Yeah.

I just saw the phrase, leading, leading company.

So I think.

one is it takes time to bring them up to do the investigation, to

figure out what they need to recover.

So I'm thinking that doing that and then also making sure you have, because

that's the one thing that stood out for me from talking to Tony Mendoza, was you

needed to have sort of clean machines that you could start using for restores.

Otherwise it's just gonna continue propagating, right?

And so finding the hardware, right, because this is a managed company, right?

So they probably have servers in rack, so procuring the

You would think if they had, if, if, if a company would have.

, you know, I, I'm, I'm guessing they have an entire space just for racks.

Uh, uh, uh,

so, but, but finding the equipment right and then identifying

the points in time, which are valid.

Right.

Standing up the infrastructure, the networking, right, making sure that

there are no further security issues.

Granted, seven days seems kind of long, but I thought

that's kind of what Tony said.

It took 'em before they started getting up

two weeks.

two weeks before, well before they were fully up and

running, but I think it was about a week before they could finally

start doing restores and bringing up

I, I, I don't remember by, by the way,

that was a really good episode.

Uh, it was ransomware victim tells their story.

Um, the, um, I, it's funny, you, you, you, you actually gave

them a, that's an actual valid

that that is a valid, that's why I said yeah,

Yeah.

Yeah.

That's pro.

Possibly the only valid answer is, hey, This is hard.

Yeah.

But they should be more transparent, you know?

they should be more transparent.

Why don't companies understand that?

I don't get that.

Um, but the worry is because due to the lack of transparency, is that,

is that they don't have backups.

They're so, so what, what have they done in the meantime for

customers who just can't do.

Yeah.

So this is the thing that I saw in, that they were actually recommending is they're

like, Hey, we went and procured Microsoft 365 licenses for you, so you can stand

up your email and continue operating.

And they're gonna help them set up, uh, what is it, forwarding rules.

So

Yeah, so that incoming email will go to

And they have like the Microsoft, uh, team on

standby to help customers, right.

Meanwhile, there's, there's all these, there's all these

email servers out there that have been trying to send mail to cus to, to, uh,

Microsoft, uh, to Rackspace customers.

And they're we're, we're trying to send the mail.

We're trying to send the mail.

Nobody's taking it.

yeah.

Well, and the thing is, even with this solution, right, it's only going forward.

You're receiving new emails, right?

All your old stuff is, oh, who knows what happened to that, right?

they did say that they provided for some

customers an archive of, of emails.

I thought they said they were trying to

get to that, but they don't yet.

well, I thought that they gave it for some, but not all.

Um, by the way, I was, I was, that, that's a clue for me, the fact

that they said they had archives of the email, but not backups,

I was looking at the article or from

Uhhuh,

we are working to provide customers with archive of

inboxes where available to eventually import over to Microsoft 365.

Right.

So it may

the word archive is a bit,

concerning.

you know?

Yeah.

Uh, archive is different than backup.

You know, we have an episode on that.

Please go listen to that if you have it.

Um, the, um, Yeah, I don't know.

I'm just not, I'm not, I don't have a good feeling here on,

a little queasy in your tummy and it's not from being sick.

Yeah.

Yeah.

Um, I, I don't, I don't know what they're doing over there.

Um, like I said, it, it all, it all starts with why didn't they have the

service patch in the first place?

That's, that's the first concerning thing.

Um, the fact that we're a weekend and they're not saying that, you know,

if they had said, l listen, we're we are, we have to do a server by server

scrub to, to verify that the, you know, we've identified the malware.

We now have to do a server by server scrub to make sure the malware isn't anywhere

else and we need to wipe the servers.

You know, but they're not saying anything.

They're just saying We're, we're working on

yeah.

The other thing I wanna know,

any, yeah.

Go.

the other thing I'm wondering though is I know

you're talking about backup, right?

And restoring data.

Do they not have a DR environment?

You know, I know we, you know, we talk about this all day.

Or maybe their DR.

Environment was compromised as well.

Right, and so we always talk about, right, you need a DR environment for

situations like this where you can quickly spin up and continue operating

rather than trying to go back and restore your data, keep those backups

just in case so you can restore them.

But you really should have a DR environment so you're not

spending six plus days trying to bring up your environment.

But if you have a DR environment and you're doing

host exchange, you're gonna have to be doing some sort of real time

replication in order to, to have that DR environment be, uh, you know, effective.

Yep.

And maybe they're doing

not, I'm not saying I, I can't.

Yeah.

And, but even that replication, right?

If it, so assuming the ransomware did not go horizontally into that DR

environment, which is a big assumption.

If you were doing exchange level replication at the application

level, hopefully your DR site shouldn't have been compromised

It has a delayed, there's a delayed replication

And you're also doing it at the application level, right?

Rather

Right, right,

at the database object level.

Right.

So

Yeah.

Yeah, that makes sense.

It's interesting, you know, it's like, well, did it, did it attack

exchange, or did it attack windows?

We don't know.

We don't know anything.

We don't know anything.

Prasanna, why don't we know anything?

That's how these things go.

Well, hopefully they publish more information.

I'm not holding my breath for that though, but I think it could be a good

learning opportunity because I wanna say that the US government, right, had

a big push for patch exchange servers because of these specific issues.

Like a couple months ago, I think there was like a cisa.

well that would've been helpful a couple of months

I thought so.

I could be wrong, but I thought there was something

So, uh, I want to tack on.

, um, basically put the, put, you know, on, on a related note to this, cause

I'm tired of talking about Rackspace.

It's too depressing.

Uh, I wanna talk about a company I'd never heard of before.

Um, they're referred to as, uh, south Korea's, um, Google, and that

is the name of the, I don't know if, I don't know if I'm pronouncing

it right, but it looks like Ka.

um, like, it's like, it's like the way cocoa is spelled properly, but except

with Case That's the, so I'm sort of cacao, that's how I'm pronouncing it.

Like, like, you know, like the, the Bean for, for chocolate.

Um, so they,

This article we found on the register,

yeah.

On the register.

Um, so, oh, it's from si Simon.

I know Simon.

Hi Simon.

I don't know if he

what was it, title of the article just for

Uh, well there's two articles.

There's Data Center Fire Takes Out South Korea's Top Two Web

Giants, and then a follow up article that was back in October.

A follow up article is it's 2022 and a Korean web Giant only now decided to

write a Dr Plan So, uh, the first part is reminiscent of, um, the O V H fire,

and that is that these guys apparently, What I'm, again, what I'm deducing there,

there were two web giants, Neve and Cacao.

They both experienced service interruptions after a data center that

hosts much of their infrastructure was shut down by a Sunday fire.

So they, they are Google-like, but they're, they're, uh, in that they

have like Facebook and messaging and a lot of different stuff like that.

I don't think they're search engine type stuff, but they.

They and this other web giant, um, were taken down by this data center fire,

which as we, again, we, we can only infer from things they, they're, they're

hosted in somebody else's data center.

It's not their data center.

And, and it's not a cloud data center

it's like a cola.

a data center.

Yeah, it's like a colo and then the, the, the.

Sort of outcome from that is that this company has decided to build

their own data center and they think now that maybe they should get a DR.

Plan.

it's, it's surprising how often sort of backup

and, and Chris, I know we've talked about it so many times, right?

And, and we've had folks on talking about, okay, this is why it's important to have

a plan and to have stakeholders aligned.

But no one ever thinks about DR and backup until it's.

Yeah.

Uh, so I guess what happened here is that the Korean government, south

Korean government sternly criticized cacao and its c e o resigned.

So they, um, they've unveiled a strategy to create tech teams, develop

a business continuity plan, and built disaster recovery facilities.

Um, you know, and the, the, the line in here, it's a little odd in this day and

age that a company of their size doesn't have these things in place already.

But this idea that the blaze, you know, the, the, so the impact of

this on everyday Koreans is that they all rely on this service.

This is what, this is why I saw it as very much this related.

They all rely on these services of like, , uh, communi for communications, right?

Um, it says they've assumed the status of de facto telecommunications

infrastructure, right?

And so when you go down, that affects so

many people day in and day out,

yeah.

yep.

Yeah.

So, um, so you have two companies where they were, they were hosting

things, they were providing services.

Many other people.

In one case it's thousands of companies.

In another case it's millions of, of everyday citizens that used them.

And then both of them lost data due to some sort of thing.

Uh, one was an attack, one was a fire, and neither of them had

a plan to, to work around that.

And, and, and

What am I doing here?

they should listen to the podcast.

That's what they should be doing.

But,

what it is.

but I think.

I want to give maybe the benefit of the doubt that maybe they had

planned for some types of outages, but they may not have planned for

these types, these specific ones.

Although in the case of

mean like the loss of an entire data center,

Right.

May,

what a Dr.

Plan is,

well, maybe they were relying on the other, like the host.

Data center infrastructure deal with that.

Great.

Hey, should have asked a question.

Yep.

Right?

Like, you know, we talk about this all the time.

Right?

And, and, and by the way, you know, so I, I mentioned in the ear in

the early part of the podcast that this was gonna touch on a couple,

couple of hobby horses for me.

One is that, You should not trust your supplier of services to be

the backer of services, right?

Unless it's written in contract or even then

What?

No, no, no.

I don't think so.

I, I think that, I think there's a, um,

separation

there's an, yeah.

Yeah.

That, that backup.

Backup should be a backup, right?

Um, that it should be done by somebody else.

This is a, this is.

Sort of a separation of powers, um, separation

Or shared responsibility model.

Shared responsibility, right?

I, I think it's just best practice to have your primary stuff by one vendor and

your backup stuff done by another vendor.

That's the way it's always been done.

, and then suddenly we start throwing that up in the air as to maybe

that's, you know, maybe we can save money if we use the services, right?

I think back to O V H and what happened there where we, where they lost the data

center, it turns out their backup stunk because they were just in the corner.

Right?

But I, I wanna challenge you that, on that challenge.

You on that though, Curtis, right?

I

right.

I think that, yes, 90% of the time don't trust the vendor, right?

The single vendor to do the right thing.

Yeah.

I feel if you can ask the right question.

, do the tests, prove it out, get it in contract,

Yeah.

the full testing, which is effort, right?

It might just be easier to go do it yourself, right?

But I think if you can get it in terms and verify it and prove it out,

I Prasannally think it's okay to go with, like to trust that single person

Do you have an example of that?

Because I don't,

AWS backup.

um, yeah, so, uh, Well, but, but that's not the same though.

That's AWS Backup is a feature that you use,

Yes.

There's no, there's no AWS backup service that

you rely on, and by service, I mean, you, you understand what I'm saying?

It's not like

it's not a managed service.

it is, it is a feature that you

use and you, you are going

to use.

The snapshots and you're responsible for managing it.

What I'm talking about is relying, you know, putting all

of the burden on the vendor.

I don't know of a service where,

But don't they like, like if you, there are companies

that do manage services, right?

Where they take over your infrastructure and they operate it

yep.

case.

So, but they'll bring in multiple vendors.

Right?

Sometimes.

That that's a different, that's a different thing.

I, I, let's just say, you know, um, I'm, I'm just, I'm just saying if,

if you're asking me my druthers,

Yeah.

No, no, it's no.

phrase, I would rather you have one vendor,

dut and another vendor do B,

A hundred percent agree a hundred percent.

Yeah.

Um, I, I can't think of a situation.

So for example, I was going to perhaps, Think of, so it's one reason, like, I

don't like whatever Microsoft is currently supplying regarding Microsoft 365.

First off, they don't have a backup service for Microsoft 365.

They do have an archive service.

They have an e-discovery service, but they don't have a backup service, so,

but if I took, uh, who is, who is, uh, Eric?

Firstly, what was the company he worked for?

Right?

Would you say that they are a service, a managed service provider?

Yeah, they are.

Uh, well they were, yeah, they were.

Yes.

They're, cuz they do, they do private cloud, essentially infrastructure.

Right?

but then they're bringing in other vendors and they're

offering the entire package, right?

yeah.

Good.

Uh, good, better, best.

Right.

Again, if you're asking me if.

Again, my choice would be to have a different vendor do backup and recovery.

Right?

Because if you find out your vendor is an idiot,

You're protected.

Prasanna Malaiyandi:

It's, you're protected.

Prasanna Malaiyandi:

Well, the chances of you having two idiot vendors is less, right?

Prasanna Malaiyandi:

Uh, so that's, that's one hobby horse.

Prasanna Malaiyandi:

And then the other hobby horse is, I can't believe, um, you

Prasanna Malaiyandi:

know, that just this idea that.

Prasanna Malaiyandi:

People relied entirely on their, people just relied entirely under their backup.

Prasanna Malaiyandi:

Clearly, nobody pushed them on it, right?

Prasanna Malaiyandi:

Nobody pushed them on, you know, what's your DR plan?

Prasanna Malaiyandi:

All these people paying 'em all this money, and they're not pushing

Prasanna Malaiyandi:

them on, what's your DR plan?

Yep.

You know?

You know, I don't, I don't know.

I don't know what to say about.

Dr.

Is Asmar.

I don't wanna be victim blaming at the same time.

You know, this is clearly Rackspace and ca cow's responsibility.

Uh, but if you are sitting there and you are using vendors and you are , you're

using vendors for stuff, you should be inquiring as to their DR plans and

their ransomware readiness plans and

Was this covered in your book when we talked

about talking about stakeholders?

I can't, I can't remember.

I can't remember if it wasn't or not, or if it was or not.

Um, mainly the book was about how to make sure you're ready.

Right?

But I don't know.

So been, it's a depressing, it's a depressing episode.

We should talk about puppies for.

Well, it's not a depressing episode.

I think this is one of those things, right?

The silver lining is, this is lessons that other companies can look at, and

hopefully they can be like, Hey, we really should be thinking about what we're doing,

and do we have Dr, do we have backup?

Do we have a

Prasanna Malaiyandi:

Take this opportunity.

Yeah.

Or the other thing is if you are using a SaaS service or who, or a managed service

provider, ask them, what are your plans?

Take this opportunity because everyone's gonna be asking, Hey, what is going on?

Yeah.

If, if, yeah, if you're one of these people that I don't understand,

by the way, that's using hosted exchange in somebody else's data

center instead of Microsoft 365, uh, by the way, feel free to contact us.

Uh, you know, I didn't mention, you know, that, that, you know, you can reach out

to us at WC presson on Twitter or w Curtis Preston at gmail as long as Twitter stays

up and, um, You know, we'd, I'd love to, if, if you've got an answer as to why you

would use hosted exchange over Microsoft 365, uh, I would love to know it.

But if you have that, yeah.

It's time to reach out to that vendor or hosted anything,

anything.

Exactly.

It's not just email.

Yeah, I mean, I, you know, this, this podcast is

hosted, backup Central is hosted, but um, I know what the backup setup is.

Hello?

Kopi

Coffee always wants to get on the podcast.

Um, yeah.

So I know what the backups are on that because I, I make them happen.

Right.

But, um,

And then for your Backup Central's website too, right?

You have a hosted WordPress site, but you're doing the backups yourself.

Yeah.

Well, it's part of the Sea Sea Panel Sea panel.

It's, it's run by C panel.

And then C panel allows me to configure a backup, which that backup, daily backup

is then copied to, uh, S3 actually, right?

And then, and it's automatically deleted after a certain number of.

Days.

Right.

Um, so it's not, it's not, the backup is not stored.

So that's another example.

Kind of like what you were talking about, like aw s backup.

I don't rely on C panel or my hosting provider to do the backup, by the

way, they do offer a backup service.

See, I do eat my

That.

That's why I was telling, that's why I brought that up, that example, right.

They do offer a hosted backup service.

LiquidWeb is the name of my hosting provider, by the way, love.

Been with them now for a really long time and I, you know, they are a actual managed

server service provider for hosting and, um, the, they do offer a backup

service and instead I use the features of C Panel to create a backup, which

is then, um, you know, sent over to s3.

And, uh, I think I keep 90 days even though I think that's ridiculous.

And my Amazon S3.

Is, uh, like $2 and 83 cents, something like that.

Maybe you should be looking at S three ia.

Your cost might drop

Yeah.

Yeah.

Cuz I, you know, I've, I've used my backups two or three times, but,

um, yeah, I don't know if it's, I don't know if it's possible, but

the saving of 35 cents.

Yeah.

But, uh, yeah.

Anyway, just l just investigate your vendors.

Will you check to see if they're doing, you know, I, I

wish the best for Rackspace.

I wish the best for their customers.

I hope that by the time you hear this, this has all been sorted out.

If it hasn't, dear Lord.

Anyway, well, alright, uh, hope you enjoyed this episode.

Remember to subscribe so that you can restore it all.