Boy, do I have a doozy of a story for you today?
Picture this.
You're just minding your own business, trusting your cloud provider to
keep your data safe and sound.
That's when a massive fire rips through and destroys their entire data center and
suddenly your whole digital world is gone.
That's exactly what happened in 2021 to the poor customers of OVH.
And let me tell you the more I dug into the story, the more I
realized just how messed up it was.
We're talking about a company that claim to have top notch backup practices,
but in reality, they were about as reliable as a chocolate teapot.
This is a wake up call for anyone who's ever put their faith in the cloud.
This episode will be a wild ride through the ups and downs of this whole fiasco.
We'll talk about the Shadys decisions made by OVH cloud.
Y the fire got so bad in the first place.
Why companies lost data and the crucial lessons that every single one
of us needs to learn from this mess.
So whether you're an it hot shot or just someone who wants to
keep their data out of the fire.
You better listen up.
By the end of this episode, you'll know exactly what questions to ask your cloud
provider to make sure you never end up in the same boat as those poor OVH customers.
I'm w Curtis Preston, AKA Mr.
Backup, and no one cares about your data more than I do.
I want to turn you the unappreciated backup admin.
Into a cyber recovery hero.
This is the backup wrap-up.
W. Curtis Preston: Welcome to the show.
I'm w Curtis Preston, and I have with me my election worker post-Traumatic Stress
Counselor Prasanna Malaiyandi how's it
Prasanna Malaiyandi: Ah, I'm doing well, Curtis, uh, how are
you doing or have you recovered?
I know that you had a pretty hectic 10 days, of which the last day was
prob 11 days, of which the last day was probably like a lot more
effort than the first 10 days, so,
W. Curtis Preston: Yeah.
Basically we got more voters on the, the actual, the actual election day.
It was the primary, it was super Tuesday.
For those of you, which for those of you not in the US is I.
A day when many states have their primaries for the presidential election.
And, uh, California is one of those, which is where I live, and
I happen to be an election worker.
Uh, it's something that I do volunteer.
Well, I, I volunteer to do it, but I do get paid for the time that I spend there.
Not a ton, but, but I do get paid for my time but you know what,
persona, I could, I could very easily talk about this for an hour.
Prasanna Malaiyandi: I think we've had three episodes.
W. Curtis Preston: You know what?
I'll put in the show notes.
If you're curious to know my thoughts on the way elections
work and how they actually work versus the way you think they work.
Um, you know, you know, go, go check those.
I'll, I'll put, I'll put 'em in the show notes and you
Prasanna Malaiyandi: and we do have also the episode with Mark.
Mark.
W. Curtis Preston: Yes.
And we have the episode with Mark.
Yeah.
There there's two ways that site managers run the sites.
Mark is of the first type, which is the, I'm here to help you vote and
if you have any questions other than that, like if you want to question
the process, here's a phone number I.
I'm more the, the type that will actually take the question.
I remember one voter, I basically was grilled for
probably 45 minutes by a voter.
I feel that she went away with a little more confidence in the, in the system.
And, um, what was really funny was the moment when I had two voters
within five minutes of each other.
One who said, and it was interesting, he's really, really old gentleman.
Like, I'm, I'm old.
He was old to me, so he was old, right?
And he, he basically said, this is crazy.
The, you know, with all this technology, we could just, we could
just replay all this stupid paper with technology and then another person.
Two minutes later it came up and said, there's way too much technology here.
We should just do this with paper.
And I was like, well, we are doing it with paper.
But you know, I don't have time to go into that.
But anyway, so, uh, it is time to get into our topic of the week.
This week in our continued series on cloud disasters where people thought
that they had backups or think that they don't need backups, and then
they found out otherwise, this one is a pretty big one, wouldn't you say?
Prasanna Malaiyandi: Oh yeah.
I think it was probably one of the most impactful ones.
Right.
Just in terms of the customer
W. Curtis Preston: but it both, most Im, it's most impactful.
Customers were really messed up with this.
And also I.
You know it.
So, and in another way it was not impactful at all.
We're gonna talk about that.
Uh, you know, in the story we're talking about the OVH Fire.
Of 2021.
So what's interesting is, you know, we have primarily a North American audience.
We have, you know, plenty from other countries, but, but the bulk
of our audiences in North America.
And I'm willing to bet that prior to this fire, the average person in
North America had never heard of OVH.
Prasanna Malaiyandi: I never heard about o.
W. Curtis Preston: I'm willing to bet.
Me neither.
Right.
And I, um, I'm willing to bet that the average person still doesn't
know anything about opiates.
That this sort, that this story sort of came and went and it's kind of amazing
and, um, that's a little bit sad.
Prasanna Malaiyandi: Yep.
W. Curtis Preston: So, um, you want to sort of tell the story of the fire?
Prasanna Malaiyandi: Yeah, so in March.
So I think before we get there, so a couple things.
First people should probably know a little bit.
I know you talked about OVH isn't
W. Curtis Preston: Oh, that is right.
Yep.
Prasanna Malaiyandi: right?
But OVH is a cloud provider.
They are the largest European headquartered cloud provider.
W. Curtis Preston: Right.
And they're headquartered in France.
Prasanna Malaiyandi: and one of the unique things of is, at least for this data
center, the way that they built it, is they actually use shipping containers.
Right, so they're kind of stacked on top of each other.
They have a bunch of servers in there with power and cooling and also wooden
walls and framing inside as well.
Uh, because it is a shipping container, I've actually watched a bunch of YouTube
people who do a lot of container homes.
And so yeah, you, you need to add structure to containers in order to
be able to do things like put floors and walls and other aspects and
W. Curtis Preston: There's interesting, uh, relationship
there because the videos to which you referred, they also lost data.
Prasanna Malaiyandi: Yep.
Yeah, the, yep.
The one that I watched.
W. Curtis Preston: we'll come maybe.
Yeah.
Maybe.
We'll, we'll come back to that in a later date.
Uh, yeah.
So it was a pretty big company, uh, big enough that Well, we, we'll save the
Prasanna Malaiyandi: Yeah.
And, and, and they have a lot of customers and they're a lot of government
contracts as well, so it's not like it's like a mom and pop cloud provider.
Right.
It is a huge cloud provider
W. Curtis Preston: also, an important thing is that they
offered a backup service.
That you could pay extra for.
Um, and, uh, it happened to be hosted Veeam, right?
Uh, and you know, Veeam did nothing wrong here, but definitely the vendor did.
They offered a, a hosted backup service.
It's a little bit different than the average cloud backup.
Problem where it's like, well, you didn't, you expected your cloud vendor
to back up and you, they didn't back up and you should have backed up.
And so it's nobody's fault but your own.
But in this case, there were customers that actually paid for the backup service.
And then what happened?
Prasanna Malaiyandi: So what ended up happening in March
of 2021, there was a fire and.
So how they have it situated is there are basically four, I don't know if
you'd call 'em data centers within that single site, but they had four
data centers, if you will, within it, and they had a fire in one.
And these are shipping containers.
And apparently they did not have the right fire suppress suppression equipment.
They did not have electrical cutoff capabilities.
And so firefighters came and they tried their best to put out that fire,
but the thing just spread like crazy.
W. Curtis Preston: just, yeah, the, the, the, the craziest thing in the report that
came out a year later, the, the bass rin report, it's, it said that it started at
an electrical inverter on the first floor.
But here's my favorite phrase.
When the firefighters came, they were met with electric arcs of
more than one meter around the exterior door of the energy room.
Like that sounds something out of like a.
Like a Avengers movie, you know what I mean?
Like this is what you call in Superman.
A a one meter long, that's about a yard for those of you who live in the US right?
Uh, electrical arcs that long, and then no suppression.
It said it took 'em, uh, over two hours to get the fire or to get
the electrical cut off because they had to do it, uh, somewhere else.
They couldn't do it in the
Prasanna Malaiyandi: And it wasn't just the one data center that got
affected, it also impacted the other one.
So I think one of the other ones had some damage.
Right?
It wasn't major, right?
Like the first one, but there was still quite a bit of damage.
And then the other two, they had to bring it offline because like you mentioned,
they had to cut the electricity.
They weren't sure what else was broken.
Right?
So they basically brought everything down.
W. Curtis Preston: Yeah.
It, it, it, it's like you look at the pictures of this,
this was a giant fire, right?
And the, and initially I felt that the, um, you know, that they, I.
They responded well in that they, they, you know, they created, they
did, what we, we suggest people do is they created a status page, um,
and they updated that status page.
The, the CEO actually made a number of videos, put them out.
I'm not sure if he put 'em on YouTube or, uh, whatever, but he, he made it.
He was trying to reassure people that we're doing everything we can.
Recover from this, recover with a small R there.
Um, but um,
Prasanna Malaiyandi: And they were transparent, right?
The other thing is they were like, Hey, we have other data centers.
They were quick trying to bring up new servers to move these customers
over to that new equipment, right?
To get them up and running as quickly as possible.
So they were bringing up the infrastructure, but
this is a big but, right?
What about their data?
W. Curtis Preston: Yeah, so apparently the backup servers were literally, you know,
in the same data center that they were backing up, and which, you know, is not in
keeping with, you know, the 3, 2, 1 rule.
Right?
So that's just, that's just a fundamental design flaw, I would say.
If you're offering backup services, you should not be putting the
backup servers directly next to the thing that they're backing up.
Right now they're saying that they were physically isolated, but they were just
literally like over in the corner, like go, you know, go sit in the corner.
Your backups.
Yeah.
Prasanna Malaiyandi: which is not what you were, and, and I don't know how many
people actually understood that, right?
Uh, if you were managing your own backup environment in your own data
center, you could physically validate these things when you are trusting and
buying a service from a vendor, right?
You just have to either ask the hard questions to really figure out
what they're doing or just trust that they're doing the right thing.
And that second part got them, got a lot of customers burned.
Do you like what I did there?
W. Curtis Preston: I, I like what you just there, so about a year later
is when the, the fire department.
Uh, report came out and that's when we learned that they had no fire suppression
systems, they had no power cutoff systems.
Uh, we learned that it damaged at least two other data centers.
Right.
And the, because while they initially started out being very open.
OVH suddenly stopped talking roughly in May of 2021.
They stopped talking and, and, and they haven't spoken publicly about the
incident since that, and that could be because they were planning for an event
Prasanna Malaiyandi: it was a big event.
W. Curtis Preston: Big event.
What event might they have been planning for
Prasanna Malaiyandi: Yeah, so in October of 2021, which is seven months after
the fire, they went through an IPO and went public and raised a bunch of money.
Right.
It was not a it, and it wasn't a failure of an IPO either.
Like I think the stock actually went up.
They raised a good deal of money, right?
So it was a pretty successful IPO seven months after this incident.
W. Curtis Preston: I.
That just flabbergast me, right?
Like as, as someone who worked at a pre IPO company, right?
I can't imagine something that catastrophic happening and
then seven months later saying, you know what we should do?
We should have people
Prasanna Malaiyandi: Well, well, well, one thing is the IPO was
probably already planned in the works.
Right?
Started going.
And they were hoping that,
W. Curtis Preston: they didn't wanna press pause.
Prasanna Malaiyandi: you gotta strike while the iron's hot because remember,
W. Curtis Preston: The was the, was.
Prasanna Malaiyandi: right.
So I think that they just took the chance, rolled the dice and figured, yeah, we're a
large public cloud provider in Europe that we have sort of a dominant position here.
We might as well raise the money while we can.
I don't know how things ended up there.
And I think like, and it's not like they've gone by the wayside, right?
OVH is still popular in Europe.
There are still a lot of people using their services.
Right?
So it's not like they just crumbled up and died.
W. Curtis Preston: Well, um, I would like to point out that they went out at 20.
Euros, right?
, back in, uh, October of 2022.
Uh, their current value is, uh, half that.
Prasanna Malaiyandi: Okay.
W. Curtis Preston: So basically they have, they went up a little bit,
they had a little bit of a spike for about two months, and then it has
done nothing but go down ever since.
Um, so.
You know, it, it may not have been a failure in the beginning, but it
certainly doesn't look very good now.
It's worth less than half,
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: which gives me a little bit of consolation, I guess.
Um, maybe it's that
Prasanna Malaiyandi: But,
W. Curtis Preston: the, as the news continued to come out
Prasanna Malaiyandi: yeah.
W. Curtis Preston: over the, over the years.
Right.
And there was other news as well.
Immediately after this happened, we started hearing
about a class action lawsuit.
Right.
And, uh, there was a firm, Ziegler and Associates that filed, uh.
A class action lawsuit in September of 2021, which for the record
would've been a month before the IPO.
And they, um, they fired it, you know, for clients that had lost data,
uh, to the fire or due to the fire.
And as of the, basically about a year ago, it said there were
140 customers in the lawsuit.
Prasanna Malaiyandi: And the lawsuit is still ongoing.
W. Curtis Preston: Yeah, it appears that, it appears that it's ongoing.
We don't have any information on, at least I don't have any
information that it ended.
I, I didn't see any stories that it ended, but we do have a story about two
customers that were apparently not in that class action law class action lawsuit.
What did you, uh, hear about
Prasanna Malaiyandi: Yeah, so there were two customers.
Um, the first basically had, uh, sued them because they had signed
up for the automated service.
And, uh, based on the contract wording, the judge sided with them
saying that, yes, physically isolated means that you keep 'em separate.
And having it in the same building goes against like what's expected,
what you would do for like a backup perspective and is wrong.
And so they basically won a nominal amount.
I would say they did not.
I think they were suing for like 3 million euros.
They got like a hundred thousand euros.
So it was a nominal win, but it still showed that, yeah.
W. Curtis Preston: Yeah.
And, and a couple things there.
While I agree with the, sometimes you have the spirit of the contract
or the letter of the contract, I agree with what the judge says.
Obviously, who does that, right?
Who does the backup design, where the backup systems are in the same data
center as the, uh, primary systems.
Uh, having said that.
It that was stated in the contract and that that was the most shocking
part to me, is that, you know, in my review I went and found, um, you
know, uh, basically the both the gotta love the way back machine, right?
The, the internet archive.
Using the internet.
Using the internet archive, I was able to find the contract that they were quoting.
And, uh, what the contract says is that for the backup service that it
says that the storage resources for the backup service will be physically,
I isolated from the servers that it's backing up, and then it's like
somebody stopped reading right?
Because the very next sentence that says the storage resources will be
located in the same data center as the resources that it's backing up.
It's, it's literally right in the contract.
And, and so as, as like a, like a legal person, like I, I just wanna
say, well, how did you not read that?
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: I'm on the side of the customer here in that they
paid for a backup service, but I'm not on the side of the customer in
that they should have reviewed that contract and they should have said so.
Wait, wait.
What?
Prasanna Malaiyandi: Yeah.
Well, and unlike a lot of other contracts right, or like SA services where they
talk nothing about backups, this was clearly stated what they are doing.
W. Curtis Preston: stated they're, and they're paying extra, right?
They were paying something like 2 cents a gigabyte per month
for this backup service and.
And the thing is, I think what the judge said, I, I like, this is one
of those things where, you know, I'm really good at arguing, and
I can argue both sides of this.
Like on one side they should have read the contract, the customer should have
read the contract, and they should have asked about what that sentence mean.
And they should have then just not paid for the backup service
because it wouldn't be worth it.
Um, on the other hand, I agree with the judge, like, who does that?
Right.
It's not like what's the point of doing a backup if it's just
two disk drives right next door?
Right.
And the judge was basically saying, this isn't in keeping with, you know, the
state of the art of the backup system.
Agreed, agreed, agreed.
And it wouldn't allow you to meet the requirements of the contract.
And so while the, I think basically what the judge says, you know, even though
the contract may have stated that.
What are you stupid?
Right?
You know what I mean?
Prasanna Malaiyandi: Which is different than the second case,
So there were two customers who ended up suing and winning, right?
So the second customer also sued, uh, OVH, and this time the judge also found
in the favor of the customer, and this one was slightly different because in
their contract they had actually written out that they had a production system
in one data center and explicitly their backup system in a different data center.
In the contract, it was worded that their production and backup systems
should be separate, and OVH agreed to it.
And what ended up happening is OVH ended up keeping both copies
of data in the same data center.
But,
W. Curtis Preston: basically they just lied.
Prasanna Malaiyandi: but here's the problem, is their management console
kept telling the customer that they were in separate data centers.
W. Curtis Preston: Wow.
Yeah.
That's, that's just bad.
And, and that's probably the worst thing that I've read about this story.
Because if you are a customer that's about the best you could do is say, Hey
listen, we're demanding that you put it separately and then the, you check the
console and the console sets it separate.
It's the cloud.
Like, I can't, I can't go visit it.
Right.
Um, so like they, so that in that case, the customer did
everything that they could.
Prasanna Malaiyandi: And so the judge gave, uh, fined OVH 138,000 euros,
something like that, but that's not even the worst part of OV H'S behavior.
So OVH eventually found the backup server, and
W. Curtis Preston: right.
Yeah.
This hurts.
Prasanna Malaiyandi: And so they were like, Hey, yeah, customer.
Great news.
We found your backup server so you can restore your data.
And while they were giving the customer back the backup server, they accidentally
deleted the data on the discs.
W. Curtis Preston: Uh, yeah.
So, um, yeah, the,
Prasanna Malaiyandi: Like, I just wanna know your OVH, you claim
to be the third or the largest EU headquartered cloud provider.
But simple processes like this, you're not able to.
They, I get some of it's probably maybe they don't have the experience
or expertise or they never had to go through this process.
Hopefully now on they have better processes in place.
It's like a learning experience, but.
I feel for the customers.
W. Curtis Preston: Yeah, I feel for those customers, and I, and I, I,
I wish the best for the, the ones in the 140 class action lawsuit.
It doesn't sound good if the two people who specifically had it in
their contract only got $150,000.
By the way, I would state I.
Well, good for them.
Like it's sort of like.
I, I'm sure what they'd rather have is the data back.
It, it is sort of like, it's like a wrongful death lawsuit.
A wrongful death lawsuit.
Can't bring your, the, the, the deceased person back.
It's just in some way trying to compensate you and 150,000 Euros is not
gonna compensate anybody for anything.
Uh, I mean, I, I wouldn't mind, just for the record, if anybody wants to send me.
Uh, I'm good.
Like, you know, I, I'll take that, but for a large company that lost, you
know, their entire company and this is all they're getting, one interesting
thing that the judge, a, a point that the judge made was that there was no
reason, there was no incentive or, or whatever for OVH to configure backups
in the way that they configured them.
Meaning that they're not a brand new hosting company
that only has one data center.
He's saying you have data centers all over the place and you have clearly
bandwidth between these data centers.
So there was no reason to configure backups the way that you configured them.
You could have just as easily made sure that, that, that the backup server is
always in another data center, but you just couldn't be bothered doing that.
So that, that's just the way the judge saw things and, uh, ruled in the, you know,
in the favor of the, of the plaintiff.
So when, when we look on this, you know.
It's difficult to talk about lessons learned
Prasanna Malaiyandi: Is it though?
W. Curtis Preston: at least, well, because like, like even the customer that
did what we would've said to do, right?
So, you know, number one, review your contract.
And so the customer reviewed that contract and said, Hey, what do you mean
you're keeping in the same data center?
That's nonsense.
You need to put it in another data center.
And then OVH.
Said, sure.
Prasanna Malaiyandi: but so.
W. Curtis Preston: they put it at,
Prasanna Malaiyandi: So, so here's, I think, how they could have done it, which
is by doing DR testing or restore testing,
W. Curtis Preston: Okay.
All right.
So, so trust and trust, but verify the old phrase from Mr.
Reagan there.
Prasanna Malaiyandi: Shoot your production in the head and try to restore your data.
W. Curtis Preston: wait, not like the Alaska, not like.
Prasanna Malaiyandi: Yeah.
Not, yeah, not like
W. Curtis Preston: like that episode.
Okay.
So, so simulate shooting your server in the head and then,
uh, and then trying to restore.
Uh, but well, but, but in, how do they simulate shutting down the data center?
Prasanna Malaiyandi: Or you just prevent any access at a new data center to any
IP addresses in the old data center,
W. Curtis Preston: Like how do
Prasanna Malaiyandi: and then try to do your restore.
So
basically.
W. Curtis Preston: need to verify that.
It's just, it is just that there is a certain degree of trust that
you are placing in the vendor that they're going to do the things
that they say they're doing.
Um, you need to figure out a way to verify that.
But I, I would say that the easier answer here is that, um.
Is to use a third party backup service.
And because while you could contractually, you could get like,
like you can use Amazon backup, right?
And you can snapshot your way to happiness and then you can replicate those
snapshots to another region if you want.
And you could probably figure out some way to verify that those
snapshots are, are in another region.
The um.
But if you use a third party backup service, it's like you,
it's like you're guaranteeing that the backups will be somewhere else
other than what you're backing up.
Because to me that's like the fundamental, you're shaking your
head, I know what you're thinking.
We could we, you can argue it in a minute.
I'm just saying that to me, like the fundamental thing of backup.
Is to make sure the backup is somewhere else.
Uh, so you're thinking that if a customer is running in the same
region,
Prasanna Malaiyandi: No, no, no.
I, I wasn't, I wasn't even, I
W. Curtis Preston: you select what, what were you thinking?
Prasanna Malaiyandi: I was thinking that if I'm a customer and I'm going
to a vendor to offload my need to have infrastructure and to manage all these
resources, am I as a customer going to go pay and find a backup service and manage
a backup service to go deal with this?
Right.
I think that that bar gets pretty high depending on how complex it is.
And because you also need to find a backup service that works with
the cloud provider you're using.
Right.
Which may be harder for the non top three clouds that you have, right?
non-Amazon, Azure, or
W. Curtis Preston: Yeah.
So I.
Okay.
I, I thought you were arguing something different.
I, I guess I would argue that that's the job,
Prasanna Malaiyandi: It, it is.
I agree.
It is a job when you move to
W. Curtis Preston: you're developing infrastructure, if you're developing
infrastructure backup as part of the infrastructure, if you don't
know how to do cloud vendor A without also backing up cloud vendor
A, then you shouldn't use Cloud vendor A,
Prasanna Malaiyandi: Yeah.
I agree with that.
Yeah.
W. Curtis Preston: okay, so what are you saying?
Prasanna Malaiyandi: No, I, I agree.
But people don't really think about backup.
Right.
W. Curtis Preston: Well, that's their problem.
There isn't, that's why we exist, right?
Um, I
I can't
Prasanna Malaiyandi: devil's.
I'm just playing devil's advocate.
You got a thousand things on
W. Curtis Preston: well, you're doing a good job.
Prasanna Malaiyandi: You have a thousand things on your mind and like Right.
Backup is not gonna be your top priority.
W. Curtis Preston: So that, that, that's a problem as old as time.
No one thinks about backup until, you
Prasanna Malaiyandi: but, but I think there is a solution though for this.
Right, and I think the solution is as an industry, we should hold vendors
accountable for providing the bare minimum needed, at least a bare minimum
needed for what is safe backups, right?
If you are saying, I'm doing backups, here's the bare minimum, right?
Follow the 3, 2, 1 rule.
Keep it out.
Separate in a, uh, right, that ensures that your backups are isolated, right?
These sort of things.
I think there should be a bare minimum.
We should hold vendors accountable to.
And until we get to that point, I think customers should ask their vendors,
Hey, here are five questions when I'm going to go use your service, and I put
it out and ask you questions, answer these five questions related to backup,
W. Curtis Preston: I, I think that's a perfectly valid thing to do.
I just know that with, with most of the SaaS services, the answer is no.
Right?
Prasanna Malaiyandi: which is okay, but at least you have the
answers.
And then now you can start to think
about, yeah.
And now you can be like, Hey, maybe if I want to go use say ServiceNow, I should
figure out, okay, how am I gonna back up that data if it's important to me?
Because they don't offer
W. Curtis Preston: Agreed.
And maybe, maybe I'll live in a utopia that where that one day, there's two
products that are really impressive and one of them has backup and the other
one doesn't as part of the service.
And the one with backup wins the
Prasanna Malaiyandi: Yeah.
And then the other one eventually figures out they're losing deals
because they don't have it, and then they start offering it, right?
W. Curtis Preston: That is, that's what we call the free market.
Um, yeah, I, I, I think that's really important.
Um, and I, I just think that, um, this is a really bad story because it's
just because some people who thought they were doing the right thing weren't
actually doing the right thing because the vendor wasn't doing the right thing
and they were actually misrepresenting.
That's the worst part of the story, is that they were actually
misrepresenting what they were doing.
And, um,
Prasanna Malaiyandi: It's like, Hey, I bought a car.
It's supposed to get a hundred miles per gallon, but it only gets 10.
You know?
W. Curtis Preston: It's more like,
Prasanna Malaiyandi: Or, or, Or, or, it's, or
W. Curtis Preston: like it doesn't go at all.
Prasanna Malaiyandi: no, no, no.
I was gonna say, or I have a car.
I bought it.
It has airbags.
I get into a wreck.
The airbags don't deploy.
W. Curtis Preston: Well, it turns out there's no
Prasanna Malaiyandi: Yeah.
There are no airbags.
W. Curtis Preston: just no airbags at all.
And they're like, well, the airbags are only in the cars that are in the factory.
Uh, the demos, we, we didn't put them in
Prasanna Malaiyandi: speaking of this, so I was actually reading a Reddit article
the other day, or a Reddit post and uh.
Someone was complaining and the story goes that they were sleeping, they were
renting a place, they were sleeping, and they heard a beeping and they go look.
And it was like one of those carbon monoxide detectors that they had
bought at a previous rental and brought it, but they never installed it.
They're like, huh, that's weird.
So they open all the windows and then it keeps going off and off.
And then they called a fire department and apparently their fireplace
was leaking carbon monoxide.
So they're like, that's at least it ca.
In their rental.
So they're like, good.
At least it was caught.
And then the fire department goes around.
Apparently all their smoke detectors were just shells.
There was no actual smoke detector inside of 'em.
W. Curtis Preston: So, yeah, well, at least that one is again, that's fixable.
You know that you can go around and test the smoke detectors in an Airbnb you're
saying, or staying, but that's just wrong.
So the person had had a battery powered,
Prasanna Malaiyandi: Carbon monoxide detector?
W. Curtis Preston: carbon monoxide detector in their like luggage or
Prasanna Malaiyandi: No.
So this was a rental, right?
So they were renting like long term, like a year rental.
Right.
And so they had just moved into the place and from their old place,
they had brought whatever they had.
And it just happened to be just sitting in like a closet somewhere.
W. Curtis Preston: Wow.
Saved their
Prasanna Malaiyandi: Yep.
Saved their life.
W. Curtis Preston: I.
Yeah.
Um,
Prasanna Malaiyandi: Go replace your check your smoke detectors, folks.
W. Curtis Preston: your smoke detectors and your carbon monoxide
detectors, um, at which for the record need to be two separate devices.
For a while I was like, this is annoying that, uh, I mean, this is totally
off the subject, but for a while I was, I was saying that CO2 detectors
and co detector, I'm sorry, not co fire detectors or smoke detectors.
It should be combined.
And, and there are companies that sell combined smoke and
fire and, and co detectors.
Do you know why they should absolutely not be in the same place
or the same unit?
Prasanna Malaiyandi: Because they rise differently.
W. Curtis Preston: Yeah.
Smoke goes up, co goes down, so co detectors should be
about a foot off the floor.
Uh, and smoke detectors should be up on the ceiling and they
should never, and they should therefore never be the same device.
Um, and how it's actually legal to sell them in one thing, I don't know.
Anyway.
So, uh, maybe I just scared the crap outta somebody and they're gonna
go Now buy, uh, separate CO two
Prasanna Malaiyandi: you get those plugin ones, which is, which is perfect because
the outlets are at the right height.
W. Curtis Preston: Exactly.
Exactly.
Uh, all right.
Any final thoughts on the OVH Fire?
Prasanna Malaiyandi: Uh, no.
I think, yeah, like you said, it's people were trying to do the right thing or
they assumed that the right thing was being done and they got burned and yeah.
I don't know what else we could recommend for them to do.
W. Curtis Preston: I.
Yeah, verify as much as you can that your backup infrastructure is separate.
And again, I think the easiest way to do that is to use a different vendor.
Um, but you know, maybe it's because I used to work for one of those
companies that there was a different vendor, but I don't anymore.
I still think that way.
Anyway.
Well, uh, thanks for the chat persona.
Prasanna Malaiyandi: anytime.
Thank you, Curtis.
W. Curtis Preston: Thank you to our listeners.
Uh, you know, you're why we do this.
That is a wrap.