Boy, do I have a doozy of a story for you today?

Picture this.

You're just minding your own business, trusting your cloud provider to

keep your data safe and sound.

That's when a massive fire rips through and destroys their entire data center and

suddenly your whole digital world is gone.

That's exactly what happened in 2021 to the poor customers of OVH.

And let me tell you the more I dug into the story, the more I

realized just how messed up it was.

We're talking about a company that claim to have top notch backup practices,

but in reality, they were about as reliable as a chocolate teapot.

This is a wake up call for anyone who's ever put their faith in the cloud.

This episode will be a wild ride through the ups and downs of this whole fiasco.

We'll talk about the Shadys decisions made by OVH cloud.

Y the fire got so bad in the first place.

Why companies lost data and the crucial lessons that every single one

of us needs to learn from this mess.

So whether you're an it hot shot or just someone who wants to

keep their data out of the fire.

You better listen up.

By the end of this episode, you'll know exactly what questions to ask your cloud

provider to make sure you never end up in the same boat as those poor OVH customers.

I'm w Curtis Preston, AKA Mr.

Backup, and no one cares about your data more than I do.

I want to turn you the unappreciated backup admin.

Into a cyber recovery hero.

This is the backup wrap-up.

W. Curtis Preston: Welcome to the show.

I'm w Curtis Preston, and I have with me my election worker post-Traumatic Stress

Counselor Prasanna Malaiyandi how's it

Ah, I'm doing well, Curtis, uh, how are

you doing or have you recovered?

I know that you had a pretty hectic 10 days, of which the last day was

prob 11 days, of which the last day was probably like a lot more

effort than the first 10 days, so,

W. Curtis Preston: Yeah.

Basically we got more voters on the, the actual, the actual election day.

It was the primary, it was super Tuesday.

For those of you, which for those of you not in the US is I.

A day when many states have their primaries for the presidential election.

And, uh, California is one of those, which is where I live, and

I happen to be an election worker.

Uh, it's something that I do volunteer.

Well, I, I volunteer to do it, but I do get paid for the time that I spend there.

Not a ton, but, but I do get paid for my time but you know what,

persona, I could, I could very easily talk about this for an hour.

I think we've had three episodes.

W. Curtis Preston: You know what?

I'll put in the show notes.

If you're curious to know my thoughts on the way elections

work and how they actually work versus the way you think they work.

Um, you know, you know, go, go check those.

I'll, I'll put, I'll put 'em in the show notes and you

and we do have also the episode with Mark.

Mark.

W. Curtis Preston: Yes.

And we have the episode with Mark.

Yeah.

There there's two ways that site managers run the sites.

Mark is of the first type, which is the, I'm here to help you vote and

if you have any questions other than that, like if you want to question

the process, here's a phone number I.

I'm more the, the type that will actually take the question.

I remember one voter, I basically was grilled for

probably 45 minutes by a voter.

I feel that she went away with a little more confidence in the, in the system.

And, um, what was really funny was the moment when I had two voters

within five minutes of each other.

One who said, and it was interesting, he's really, really old gentleman.

Like, I'm, I'm old.

He was old to me, so he was old, right?

And he, he basically said, this is crazy.

The, you know, with all this technology, we could just, we could

just replay all this stupid paper with technology and then another person.

Two minutes later it came up and said, there's way too much technology here.

We should just do this with paper.

And I was like, well, we are doing it with paper.

But you know, I don't have time to go into that.

But anyway, so, uh, it is time to get into our topic of the week.

This week in our continued series on cloud disasters where people thought

that they had backups or think that they don't need backups, and then

they found out otherwise, this one is a pretty big one, wouldn't you say?

Oh yeah.

I think it was probably one of the most impactful ones.

Right.

Just in terms of the customer

W. Curtis Preston: but it both, most Im, it's most impactful.

Customers were really messed up with this.

And also I.

You know it.

So, and in another way it was not impactful at all.

We're gonna talk about that.

Uh, you know, in the story we're talking about the OVH Fire.

Of 2021.

So what's interesting is, you know, we have primarily a North American audience.

We have, you know, plenty from other countries, but, but the bulk

of our audiences in North America.

And I'm willing to bet that prior to this fire, the average person in

North America had never heard of OVH.

Prasanna Malaiyandi: I never heard about o.

W. Curtis Preston: I'm willing to bet.

Me neither.

Right.

And I, um, I'm willing to bet that the average person still doesn't

know anything about opiates.

That this sort, that this story sort of came and went and it's kind of amazing

and, um, that's a little bit sad.

Yep.

W. Curtis Preston: So, um, you want to sort of tell the story of the fire?

Yeah, so in March.

So I think before we get there, so a couple things.

First people should probably know a little bit.

I know you talked about OVH isn't

W. Curtis Preston: Oh, that is right.

Yep.

right?

But OVH is a cloud provider.

They are the largest European headquartered cloud provider.

W. Curtis Preston: Right.

And they're headquartered in France.

and one of the unique things of is, at least for this data

center, the way that they built it, is they actually use shipping containers.

Right, so they're kind of stacked on top of each other.

They have a bunch of servers in there with power and cooling and also wooden

walls and framing inside as well.

Uh, because it is a shipping container, I've actually watched a bunch of YouTube

people who do a lot of container homes.

And so yeah, you, you need to add structure to containers in order to

be able to do things like put floors and walls and other aspects and

W. Curtis Preston: There's interesting, uh, relationship

there because the videos to which you referred, they also lost data.

Yep.

Yeah, the, yep.

The one that I watched.

W. Curtis Preston: we'll come maybe.

Yeah.

Maybe.

We'll, we'll come back to that in a later date.

Uh, yeah.

So it was a pretty big company, uh, big enough that Well, we, we'll save the

Yeah.

And, and, and they have a lot of customers and they're a lot of government

contracts as well, so it's not like it's like a mom and pop cloud provider.

Right.

It is a huge cloud provider

W. Curtis Preston: also, an important thing is that they

offered a backup service.

That you could pay extra for.

Um, and, uh, it happened to be hosted Veeam, right?

Uh, and you know, Veeam did nothing wrong here, but definitely the vendor did.

They offered a, a hosted backup service.

It's a little bit different than the average cloud backup.

Problem where it's like, well, you didn't, you expected your cloud vendor

to back up and you, they didn't back up and you should have backed up.

And so it's nobody's fault but your own.

But in this case, there were customers that actually paid for the backup service.

And then what happened?

So what ended up happening in March

of 2021, there was a fire and.

So how they have it situated is there are basically four, I don't know if

you'd call 'em data centers within that single site, but they had four

data centers, if you will, within it, and they had a fire in one.

And these are shipping containers.

And apparently they did not have the right fire suppress suppression equipment.

They did not have electrical cutoff capabilities.

And so firefighters came and they tried their best to put out that fire,

but the thing just spread like crazy.

W. Curtis Preston: just, yeah, the, the, the, the craziest thing in the report that

came out a year later, the, the bass rin report, it's, it said that it started at

an electrical inverter on the first floor.

But here's my favorite phrase.

When the firefighters came, they were met with electric arcs of

more than one meter around the exterior door of the energy room.

Like that sounds something out of like a.

Like a Avengers movie, you know what I mean?

Like this is what you call in Superman.

A a one meter long, that's about a yard for those of you who live in the US right?

Uh, electrical arcs that long, and then no suppression.

It said it took 'em, uh, over two hours to get the fire or to get

the electrical cut off because they had to do it, uh, somewhere else.

They couldn't do it in the

And it wasn't just the one data center that got

affected, it also impacted the other one.

So I think one of the other ones had some damage.

Right?

It wasn't major, right?

Like the first one, but there was still quite a bit of damage.

And then the other two, they had to bring it offline because like you mentioned,

they had to cut the electricity.

They weren't sure what else was broken.

Right?

So they basically brought everything down.

W. Curtis Preston: Yeah.

It, it, it, it's like you look at the pictures of this,

this was a giant fire, right?

And the, and initially I felt that the, um, you know, that they, I.

They responded well in that they, they, you know, they created, they

did, what we, we suggest people do is they created a status page, um,

and they updated that status page.

The, the CEO actually made a number of videos, put them out.

I'm not sure if he put 'em on YouTube or, uh, whatever, but he, he made it.

He was trying to reassure people that we're doing everything we can.

Recover from this, recover with a small R there.

Um, but um,

And they were transparent, right?

The other thing is they were like, Hey, we have other data centers.

They were quick trying to bring up new servers to move these customers

over to that new equipment, right?

To get them up and running as quickly as possible.

So they were bringing up the infrastructure, but

this is a big but, right?

What about their data?

W. Curtis Preston: Yeah, so apparently the backup servers were literally, you know,

in the same data center that they were backing up, and which, you know, is not in

keeping with, you know, the 3, 2, 1 rule.

Right?

So that's just, that's just a fundamental design flaw, I would say.

If you're offering backup services, you should not be putting the

backup servers directly next to the thing that they're backing up.

Right now they're saying that they were physically isolated, but they were just

literally like over in the corner, like go, you know, go sit in the corner.

Your backups.

Yeah.

which is not what you were, and, and I don't know how many

people actually understood that, right?

Uh, if you were managing your own backup environment in your own data

center, you could physically validate these things when you are trusting and

buying a service from a vendor, right?

You just have to either ask the hard questions to really figure out

what they're doing or just trust that they're doing the right thing.

And that second part got them, got a lot of customers burned.

Do you like what I did there?

W. Curtis Preston: I, I like what you just there, so about a year later

is when the, the fire department.

Uh, report came out and that's when we learned that they had no fire suppression

systems, they had no power cutoff systems.

Uh, we learned that it damaged at least two other data centers.

Right.

And the, because while they initially started out being very open.

OVH suddenly stopped talking roughly in May of 2021.

They stopped talking and, and, and they haven't spoken publicly about the

incident since that, and that could be because they were planning for an event

it was a big event.

W. Curtis Preston: Big event.

What event might they have been planning for

Yeah, so in October of 2021, which is seven months after

the fire, they went through an IPO and went public and raised a bunch of money.

Right.

It was not a it, and it wasn't a failure of an IPO either.

Like I think the stock actually went up.

They raised a good deal of money, right?

So it was a pretty successful IPO seven months after this incident.

W. Curtis Preston: I.

That just flabbergast me, right?

Like as, as someone who worked at a pre IPO company, right?

I can't imagine something that catastrophic happening and

then seven months later saying, you know what we should do?

We should have people

Well, well, well, one thing is the IPO was

probably already planned in the works.

Right?

Started going.

And they were hoping that,

W. Curtis Preston: they didn't wanna press pause.

you gotta strike while the iron's hot because remember,

W. Curtis Preston: The was the, was.

right.

So I think that they just took the chance, rolled the dice and figured, yeah, we're a

large public cloud provider in Europe that we have sort of a dominant position here.

We might as well raise the money while we can.

I don't know how things ended up there.

And I think like, and it's not like they've gone by the wayside, right?

OVH is still popular in Europe.

There are still a lot of people using their services.

Right?

So it's not like they just crumbled up and died.

W. Curtis Preston: Well, um, I would like to point out that they went out at 20.

Euros, right?

, back in, uh, October of 2022.

Uh, their current value is, uh, half that.

Okay.

W. Curtis Preston: So basically they have, they went up a little bit,

they had a little bit of a spike for about two months, and then it has

done nothing but go down ever since.

Um, so.

You know, it, it may not have been a failure in the beginning, but it

certainly doesn't look very good now.

It's worth less than half,

Yeah.

W. Curtis Preston: which gives me a little bit of consolation, I guess.

Um, maybe it's that

But,

W. Curtis Preston: the, as the news continued to come out

yeah.

W. Curtis Preston: over the, over the years.

Right.

And there was other news as well.

Immediately after this happened, we started hearing

about a class action lawsuit.

Right.

And, uh, there was a firm, Ziegler and Associates that filed, uh.

A class action lawsuit in September of 2021, which for the record

would've been a month before the IPO.

And they, um, they fired it, you know, for clients that had lost data,

uh, to the fire or due to the fire.

And as of the, basically about a year ago, it said there were

140 customers in the lawsuit.

And the lawsuit is still ongoing.

W. Curtis Preston: Yeah, it appears that, it appears that it's ongoing.

We don't have any information on, at least I don't have any

information that it ended.

I, I didn't see any stories that it ended, but we do have a story about two

customers that were apparently not in that class action law class action lawsuit.

What did you, uh, hear about

Yeah, so there were two customers.

Um, the first basically had, uh, sued them because they had signed

up for the automated service.

And, uh, based on the contract wording, the judge sided with them

saying that, yes, physically isolated means that you keep 'em separate.

And having it in the same building goes against like what's expected,

what you would do for like a backup perspective and is wrong.

And so they basically won a nominal amount.

I would say they did not.

I think they were suing for like 3 million euros.

They got like a hundred thousand euros.

So it was a nominal win, but it still showed that, yeah.

W. Curtis Preston: Yeah.

And, and a couple things there.

While I agree with the, sometimes you have the spirit of the contract

or the letter of the contract, I agree with what the judge says.

Obviously, who does that, right?

Who does the backup design, where the backup systems are in the same data

center as the, uh, primary systems.

Uh, having said that.

It that was stated in the contract and that that was the most shocking

part to me, is that, you know, in my review I went and found, um, you

know, uh, basically the both the gotta love the way back machine, right?

The, the internet archive.

Using the internet.

Using the internet archive, I was able to find the contract that they were quoting.

And, uh, what the contract says is that for the backup service that it

says that the storage resources for the backup service will be physically,

I isolated from the servers that it's backing up, and then it's like

somebody stopped reading right?

Because the very next sentence that says the storage resources will be

located in the same data center as the resources that it's backing up.

It's, it's literally right in the contract.

And, and so as, as like a, like a legal person, like I, I just wanna

say, well, how did you not read that?

Yeah.

W. Curtis Preston: I'm on the side of the customer here in that they

paid for a backup service, but I'm not on the side of the customer in

that they should have reviewed that contract and they should have said so.

Wait, wait.

What?

Yeah.

Well, and unlike a lot of other contracts right, or like SA services where they

talk nothing about backups, this was clearly stated what they are doing.

W. Curtis Preston: stated they're, and they're paying extra, right?

They were paying something like 2 cents a gigabyte per month

for this backup service and.

And the thing is, I think what the judge said, I, I like, this is one

of those things where, you know, I'm really good at arguing, and

I can argue both sides of this.

Like on one side they should have read the contract, the customer should have

read the contract, and they should have asked about what that sentence mean.

And they should have then just not paid for the backup service

because it wouldn't be worth it.

Um, on the other hand, I agree with the judge, like, who does that?

Right.

It's not like what's the point of doing a backup if it's just

two disk drives right next door?

Right.

And the judge was basically saying, this isn't in keeping with, you know, the

state of the art of the backup system.

Agreed, agreed, agreed.

And it wouldn't allow you to meet the requirements of the contract.

And so while the, I think basically what the judge says, you know, even though

the contract may have stated that.

What are you stupid?

Right?

You know what I mean?

Which is different than the second case,

So there were two customers who ended up suing and winning, right?

So the second customer also sued, uh, OVH, and this time the judge also found

in the favor of the customer, and this one was slightly different because in

their contract they had actually written out that they had a production system

in one data center and explicitly their backup system in a different data center.

In the contract, it was worded that their production and backup systems

should be separate, and OVH agreed to it.

And what ended up happening is OVH ended up keeping both copies

of data in the same data center.

But,

W. Curtis Preston: basically they just lied.

but here's the problem, is their management console

kept telling the customer that they were in separate data centers.

W. Curtis Preston: Wow.

Yeah.

That's, that's just bad.

And, and that's probably the worst thing that I've read about this story.

Because if you are a customer that's about the best you could do is say, Hey

listen, we're demanding that you put it separately and then the, you check the

console and the console sets it separate.

It's the cloud.

Like, I can't, I can't go visit it.

Right.

Um, so like they, so that in that case, the customer did

everything that they could.

And so the judge gave, uh, fined OVH 138,000 euros,

something like that, but that's not even the worst part of OV H'S behavior.

So OVH eventually found the backup server, and

W. Curtis Preston: right.

Yeah.

This hurts.

And so they were like, Hey, yeah, customer.

Great news.

We found your backup server so you can restore your data.

And while they were giving the customer back the backup server, they accidentally

deleted the data on the discs.

W. Curtis Preston: Uh, yeah.

So, um, yeah, the,

Like, I just wanna know your OVH, you claim

to be the third or the largest EU headquartered cloud provider.

But simple processes like this, you're not able to.

They, I get some of it's probably maybe they don't have the experience

or expertise or they never had to go through this process.

Hopefully now on they have better processes in place.

It's like a learning experience, but.

I feel for the customers.

W. Curtis Preston: Yeah, I feel for those customers, and I, and I, I,

I wish the best for the, the ones in the 140 class action lawsuit.

It doesn't sound good if the two people who specifically had it in

their contract only got $150,000.

By the way, I would state I.

Well, good for them.

Like it's sort of like.

I, I'm sure what they'd rather have is the data back.

It, it is sort of like, it's like a wrongful death lawsuit.

A wrongful death lawsuit.

Can't bring your, the, the, the deceased person back.

It's just in some way trying to compensate you and 150,000 Euros is not

gonna compensate anybody for anything.

Uh, I mean, I, I wouldn't mind, just for the record, if anybody wants to send me.

Uh, I'm good.

Like, you know, I, I'll take that, but for a large company that lost, you

know, their entire company and this is all they're getting, one interesting

thing that the judge, a, a point that the judge made was that there was no

reason, there was no incentive or, or whatever for OVH to configure backups

in the way that they configured them.

Meaning that they're not a brand new hosting company

that only has one data center.

He's saying you have data centers all over the place and you have clearly

bandwidth between these data centers.

So there was no reason to configure backups the way that you configured them.

You could have just as easily made sure that, that, that the backup server is

always in another data center, but you just couldn't be bothered doing that.

So that, that's just the way the judge saw things and, uh, ruled in the, you know,

in the favor of the, of the plaintiff.

So when, when we look on this, you know.

It's difficult to talk about lessons learned

Is it though?

W. Curtis Preston: at least, well, because like, like even the customer that

did what we would've said to do, right?

So, you know, number one, review your contract.

And so the customer reviewed that contract and said, Hey, what do you mean

you're keeping in the same data center?

That's nonsense.

You need to put it in another data center.

And then OVH.

Said, sure.

but so.

W. Curtis Preston: they put it at,

So, so here's, I think, how they could have done it, which

is by doing DR testing or restore testing,

W. Curtis Preston: Okay.

All right.

So, so trust and trust, but verify the old phrase from Mr.

Reagan there.

Shoot your production in the head and try to restore your data.

W. Curtis Preston: wait, not like the Alaska, not like.

Yeah.

Not, yeah, not like

W. Curtis Preston: like that episode.

Okay.

So, so simulate shooting your server in the head and then,

uh, and then trying to restore.

Uh, but well, but, but in, how do they simulate shutting down the data center?

Or you just prevent any access at a new data center to any

IP addresses in the old data center,

W. Curtis Preston: Like how do

and then try to do your restore.

So

basically.

W. Curtis Preston: need to verify that.

It's just, it is just that there is a certain degree of trust that

you are placing in the vendor that they're going to do the things

that they say they're doing.

Um, you need to figure out a way to verify that.

But I, I would say that the easier answer here is that, um.

Is to use a third party backup service.

And because while you could contractually, you could get like,

like you can use Amazon backup, right?

And you can snapshot your way to happiness and then you can replicate those

snapshots to another region if you want.

And you could probably figure out some way to verify that those

snapshots are, are in another region.

The um.

But if you use a third party backup service, it's like you,

it's like you're guaranteeing that the backups will be somewhere else

other than what you're backing up.

Because to me that's like the fundamental, you're shaking your

head, I know what you're thinking.

We could we, you can argue it in a minute.

I'm just saying that to me, like the fundamental thing of backup.

Is to make sure the backup is somewhere else.

Uh, so you're thinking that if a customer is running in the same

region,

No, no, no.

I, I wasn't, I wasn't even, I

W. Curtis Preston: you select what, what were you thinking?

I was thinking that if I'm a customer and I'm going

to a vendor to offload my need to have infrastructure and to manage all these

resources, am I as a customer going to go pay and find a backup service and manage

a backup service to go deal with this?

Right.

I think that that bar gets pretty high depending on how complex it is.

And because you also need to find a backup service that works with

the cloud provider you're using.

Right.

Which may be harder for the non top three clouds that you have, right?

non-Amazon, Azure, or

W. Curtis Preston: Yeah.

So I.

Okay.

I, I thought you were arguing something different.

I, I guess I would argue that that's the job,

It, it is.

I agree.

It is a job when you move to

W. Curtis Preston: you're developing infrastructure, if you're developing

infrastructure backup as part of the infrastructure, if you don't

know how to do cloud vendor A without also backing up cloud vendor

A, then you shouldn't use Cloud vendor A,

Yeah.

I agree with that.

Yeah.

W. Curtis Preston: okay, so what are you saying?

No, I, I agree.

But people don't really think about backup.

Right.

W. Curtis Preston: Well, that's their problem.

There isn't, that's why we exist, right?

Um, I

I can't

devil's.

I'm just playing devil's advocate.

You got a thousand things on

W. Curtis Preston: well, you're doing a good job.

You have a thousand things on your mind and like Right.

Backup is not gonna be your top priority.

W. Curtis Preston: So that, that, that's a problem as old as time.

No one thinks about backup until, you

but, but I think there is a solution though for this.

Right, and I think the solution is as an industry, we should hold vendors

accountable for providing the bare minimum needed, at least a bare minimum

needed for what is safe backups, right?

If you are saying, I'm doing backups, here's the bare minimum, right?

Follow the 3, 2, 1 rule.

Keep it out.

Separate in a, uh, right, that ensures that your backups are isolated, right?

These sort of things.

I think there should be a bare minimum.

We should hold vendors accountable to.

And until we get to that point, I think customers should ask their vendors,

Hey, here are five questions when I'm going to go use your service, and I put

it out and ask you questions, answer these five questions related to backup,

W. Curtis Preston: I, I think that's a perfectly valid thing to do.

I just know that with, with most of the SaaS services, the answer is no.

Right?

which is okay, but at least you have the

answers.

And then now you can start to think

about, yeah.

And now you can be like, Hey, maybe if I want to go use say ServiceNow, I should

figure out, okay, how am I gonna back up that data if it's important to me?

Because they don't offer

W. Curtis Preston: Agreed.

And maybe, maybe I'll live in a utopia that where that one day, there's two

products that are really impressive and one of them has backup and the other

one doesn't as part of the service.

And the one with backup wins the

Yeah.

And then the other one eventually figures out they're losing deals

because they don't have it, and then they start offering it, right?

W. Curtis Preston: That is, that's what we call the free market.

Um, yeah, I, I, I think that's really important.

Um, and I, I just think that, um, this is a really bad story because it's

just because some people who thought they were doing the right thing weren't

actually doing the right thing because the vendor wasn't doing the right thing

and they were actually misrepresenting.

That's the worst part of the story, is that they were actually

misrepresenting what they were doing.

And, um,

It's like, Hey, I bought a car.

It's supposed to get a hundred miles per gallon, but it only gets 10.

You know?

W. Curtis Preston: It's more like,

Prasanna Malaiyandi: Or, or, Or, or, it's, or

W. Curtis Preston: like it doesn't go at all.

no, no, no.

I was gonna say, or I have a car.

I bought it.

It has airbags.

I get into a wreck.

The airbags don't deploy.

W. Curtis Preston: Well, it turns out there's no

Yeah.

There are no airbags.

W. Curtis Preston: just no airbags at all.

And they're like, well, the airbags are only in the cars that are in the factory.

Uh, the demos, we, we didn't put them in

speaking of this, so I was actually reading a Reddit article

the other day, or a Reddit post and uh.

Someone was complaining and the story goes that they were sleeping, they were

renting a place, they were sleeping, and they heard a beeping and they go look.

And it was like one of those carbon monoxide detectors that they had

bought at a previous rental and brought it, but they never installed it.

They're like, huh, that's weird.

So they open all the windows and then it keeps going off and off.

And then they called a fire department and apparently their fireplace

was leaking carbon monoxide.

So they're like, that's at least it ca.

In their rental.

So they're like, good.

At least it was caught.

And then the fire department goes around.

Apparently all their smoke detectors were just shells.

There was no actual smoke detector inside of 'em.

W. Curtis Preston: So, yeah, well, at least that one is again, that's fixable.

You know that you can go around and test the smoke detectors in an Airbnb you're

saying, or staying, but that's just wrong.

So the person had had a battery powered,

Prasanna Malaiyandi: Carbon monoxide detector?

W. Curtis Preston: carbon monoxide detector in their like luggage or

No.

So this was a rental, right?

So they were renting like long term, like a year rental.

Right.

And so they had just moved into the place and from their old place,

they had brought whatever they had.

And it just happened to be just sitting in like a closet somewhere.

W. Curtis Preston: Wow.

Saved their

Yep.

Saved their life.

W. Curtis Preston: I.

Yeah.

Um,

Go replace your check your smoke detectors, folks.

W. Curtis Preston: your smoke detectors and your carbon monoxide

detectors, um, at which for the record need to be two separate devices.

For a while I was like, this is annoying that, uh, I mean, this is totally

off the subject, but for a while I was, I was saying that CO2 detectors

and co detector, I'm sorry, not co fire detectors or smoke detectors.

It should be combined.

And, and there are companies that sell combined smoke and

fire and, and co detectors.

Do you know why they should absolutely not be in the same place

or the same unit?

Because they rise differently.

W. Curtis Preston: Yeah.

Smoke goes up, co goes down, so co detectors should be

about a foot off the floor.

Uh, and smoke detectors should be up on the ceiling and they

should never, and they should therefore never be the same device.

Um, and how it's actually legal to sell them in one thing, I don't know.

Anyway.

So, uh, maybe I just scared the crap outta somebody and they're gonna

go Now buy, uh, separate CO two

you get those plugin ones, which is, which is perfect because

the outlets are at the right height.

W. Curtis Preston: Exactly.

Exactly.

Uh, all right.

Any final thoughts on the OVH Fire?

Uh, no.

I think, yeah, like you said, it's people were trying to do the right thing or

they assumed that the right thing was being done and they got burned and yeah.

I don't know what else we could recommend for them to do.

W. Curtis Preston: I.

Yeah, verify as much as you can that your backup infrastructure is separate.

And again, I think the easiest way to do that is to use a different vendor.

Um, but you know, maybe it's because I used to work for one of those

companies that there was a different vendor, but I don't anymore.

I still think that way.

Anyway.

Well, uh, thanks for the chat persona.

anytime.

Thank you, Curtis.

W. Curtis Preston: Thank you to our listeners.

Uh, you know, you're why we do this.

That is a wrap.