This week's cloud disaster story is a shocking one from 2009.
And one that really chapped my hide at the time.
Today, we dive into the turbulent history of Carbonite, a once prominent cloud
backup vendor from posting fake reviews on Amazon to a major data loss incident
and a multimillion dollar lawsuit.
Carbonite's story is a cautionary tale for both customers and
vendors in the backup industry.
Join me and my co-host as we uncovered the details of Carbonite's missteps, including
their use of woefully, inadequate storage arrays, lack of data, redundancy,
and failure to take responsibility.
For a huge customer data loss.
We'll also discuss the lessons we can take away from Carbonite's mistakes.
One big discussion is guidance on what customers should look for when they're
evaluating a cloud backup provider.
You'll learn what happens when you blindly trust vendor marketing.
And you will learn the importance of thorough due diligence when
entrusting your data to a third party.
If this is your first time listening.
Hi, I'm w Curtis Preston, AKA Mr.
Backup, this podcast is my attempt to make sure that what happened to me 30
years ago will never happen to you.
My employer lost their purchasing database and I couldn't restore it.
I dedicated my career to making sure that that wouldn't ever again happen to me.
And now I'm passing it on to you.
We take unappreciated backup admins and turn them into cyber recovery heroes.
This is the backup wrap up.
W. Curtis Preston: Welcome to the show.
Hi, I am w Curtis Preston, and I have with me a fellow possessor
of useless knowledge Prasanna Malaiyandi, how's it going?
Prasanna.
Prasanna Malaiyandi: I am good, Curtis.
Yeah, I was, uh, I, yes, I do tend to have a lot of what I
would consider random knowledge.
Not always useless.
I know sometimes it's helped
W. Curtis Preston: Okay.
You do have useless knowledge.
You do also have useful knowledge.
We, you know, I've kidded you about it in time.
I'll be doing something and I'll be like, you know, working on that timing chain on
my Prius, a car that you've never honed.
And you're like, well, did you do the thing with the thing with the thing?
And I'm like, how do you know this?
And you're like, well, YouTube, you know
Prasanna Malaiyandi: I watch a lot of YouTube XI
W. Curtis Preston: But recently you and this person to which I am married, have
been kidding me a lot about my recent,
Prasanna Malaiyandi: I don't know if it's,
W. Curtis Preston: useless knowledge.
Prasanna Malaiyandi: I don't think it's recent though.
I think you've had this in your head for a while, and I've only
discovered it recently that you have this random knowledge in your head.
W. Curtis Preston: so, so I.
Well, it's relatively recent, by the way.
It, it's within the last year, uh, that I asked myself, what are the different
kind of palm trees in California?
And I found out that, first off, I found out that there's over 3000
palm tree species in the world, and that there's something like 50 to
90 palm trees depending on who you ask that grow, you know, here in
California.
Yeah.
No, no, no, no,
no.
50 to 90 that grow, period.
There's
only one species of palm tree that is native to California.
Prasanna Malaiyandi: look at this.
You're already correcting me, the expert,
the.
W. Curtis Preston: yeah.
And that, but, but with all adding all of that up, if you can correctly
identify roughly five species of palm trees, at least in Southern California
where I live, you can cover 99% of the palm trees that you're going to see.
Right.
And I point this out to people.
I, I, I kind of, I bring it up.
That's the
weird part is I bring it up, I'm like, Hey, uh, look, it's, you know, and, and I
did this with this, this lady that I was talking to, and I was like, oh man, I got
a lot of king palms in this neighborhood.
And she's, and she said, you know, the different kind of palms
Prasanna Malaiyandi: Curtis is like, my life mission is
W. Curtis Preston: like, yes, yes I do.
Would you like to know?
And so I start sharing with, with, with this person, uh,
the different kind of palms.
And then she said, can you tell me the kind of palms that are in my yard?
And I said, yes, I can.
And for the record, she had a combination of queen palms and king palms.
Queen palms are from South America.
King palms are from Australia, and uh, only the Mexican fan
palm is native to California.
And for those of you that have looked at California, been to California,
the Mexican fan palms are the ones that grow like 150 feet tall.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: Right with the tiny little sprig of green on the top, uh,
they're all, they're also 20 feet tall, but they, they're the ones that grow
to, to the really, really tall ones.
Um, so yeah.
But you got, you and my wife have been making fun of me and
my, my possession of this random
knowledge.
Prasanna Malaiyandi: so are you sure that this woman was not
paid to ask you this question?
W. Curtis Preston: No, she was not big.
I, I am pretty confident that she expressed genuine interest
in the species of palm trees that were in her neighborhood.
Prasanna Malaiyandi: Now?
Now, now are you sure that she expressed genuine interest or now that you're
replaying the story in your head, you think she was expressing genuine interest?
W. Curtis Preston: I am convinced Sir, and I stand by my story.
Prasanna Malaiyandi: Okay, so now that you've finished
palm trees, what comes next?
W. Curtis Preston: I, I don't know.
It, it, it, it, it has to come upon me.
It has to be, I, I need a muse, you know?
I need to be inspired.
Prasanna Malaiyandi: You, you know what inspired, I think last time when you
did this, we used to walk all the time.
W. Curtis Preston: We did.
That is what happened.
We used to walk together separately, walk together
separately, right, uh, on the phone.
And I was realizing in my neighborhood the incredible diversity of palm trees.
Now that I have this knowledge, I need to redo that walk and then see, um, see how
many of them are actually, uh, you know.
The Just five.
Just of the five.
Um, yeah.
So anyway, uh,
speaking
Prasanna Malaiyandi: so, so
W. Curtis Preston: what's that?
Prasanna Malaiyandi: So listeners, if you are walking and you encounter
a palm tree and you're on Twitter slash x, post it there and tag Curtis
and he will respond.
Yep.
W. Curtis Preston: Yeah, I will respond if it's one of the five that.
Um, yeah, it was kind of funny.
Um, what is funny is when I, like if I, if I'm trying to teach somebody
the difference, for example, between a queen palm and a king palm and,
um, they don't get it at first.
And I'm like,
it's so obvious.
Like, look, and at those two trees, the difference between a king and a queen
is so obvious and it takes 'em a minute.
And then, and then they have the aha moment.
And
I'm, and I feel.
It's like, it's like the moment when somebody brings up the 3, 2, 1 rule out
of, out of nowhere.
And I say, yeah.
So speaking of the 3, 2, 1 rule,
so, So here's my question, Prasanna,
we.
Have been talking a lot about cloud outages and cloud disasters and, um,
so far they've all been like a cloud provider that is supposed to do the
thing and the, and in every one of the cases, if the customer had just
had a backup, they would've been fine.
So.
Right.
Prasanna Malaiyandi: in, but the only thing I wanna, I
was wondering though, is OVH
W. Curtis Preston: A proper backup.
Prasanna Malaiyandi: Yes.
W. Curtis Preston: You know, we, we rely on these backups
a lot and so today's story.
You know, the story that went live yesterday really gets me upset because
in that story, the vendor lied.
I.
To their customers.
They said that, you know, it's proven in court that, that the
vendor lied to their customers.
They told them that the server was, was in a fit, at least one customer.
We know that they specifically said, I want a server in another data center.
And they said, you got it.
And then it turns out that they didn't
have it.
Prasanna Malaiyandi: Because contractual contracts and
operations don't always align
two separate people, two separate org
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: Yeah,
W. Curtis Preston: One of the things that we tell people to do is to,
is I'm a fan of Cloud-based backup.
Prasanna Malaiyandi: I never would've guessed.
W. Curtis Preston: I think it's the killer app for the cloud.
I really do.
Right.
Which is why this story bothers me so much, and that is that we
have a story of a, of a backup vendor that, you know, lost.
Customer data
and it's, it's so frustrating from the story.
Right?
So
Prasanna Malaiyandi: well, well it, I would say that it's not even just lost
customer data, but also the way that they've been operating isn't necessarily
what you would look for in a company that you're trusting your data to.
W. Curtis Preston: Yeah, there are multiple incidents of, um,
I'll call it everything from, you could call it lack of candor.
You could, you know, depending on how you want to characterize
it, some of it is outright.
I don't know if I'm gonna say lie to the degree that OVH did, because that
was, you know, proven in court that they purposefully misrepresented.
Um, in this case we do see them misrepresenting things, but, you know,
do we know for a fact, has it been proven in the court that it was a lie?
No.
But the activities that we see happening.
Weren't good.
Um, and so what are we talking about?
Prasanna?
are we teasing?
What are we teasing?
Prasanna Malaiyandi: So the company,
W. Curtis Preston: a big
name,
A big name, right?
This is a name who, a, you know, say four or five years ago you couldn't go
to an airport without seeing this name.
It was everywhere,
right?
What are we talking about?
Prasanna Malaiyandi: We are talking about the company called Carbonite.
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: Right.
It was a large company that recently was acquired or merged
and is now part of OpenText, uh, which is a Canadian based company.
W. Curtis Preston: Which is gonna figure into the story.
Prasanna Malaiyandi: yes.
So maybe we should roll back things and just go way back
in time and talk about Yeah.
Mm-Hmm.
W. Curtis Preston: let's go way back in time.
So first, um, you know, I'm gonna bring up a story.
that It's not on topic, but it's relevant.
Okay.
A couple of months before the key story that we're gonna talk about happened.
Carbonite got busted by essentially a, a part-time, like internet sleuth, right?
It is just some blogger guy.
Um, and what they got busted, uh, doing was posting fake reviews on Amazon.
Right.
And the, the, this, this sleuth uncovered it, you know, he busted.
I, I remember, you know, I wrote a blog post about it back when this happened.
This was in, by the way, this was 2009.
And, um, this, this person found that, um.
Uh, that basically they had created, uh, a combination of like fake
accounts and, and in some cases just really poorly disguised accounts.
The, and all of the metadata pointed that, um, that these reviews that have been po,
these really positive reviews that have been on, on Amazon were in fact written by
either, um, Carbonite employees or by, um.
Uh, people related to Carbonite
employees, right?
And this guy, this guy uncovered it and, uh, he blogged about it.
And then New York Times covered it, uh, a blog called Pogue' s Post.
And by the way, this, this is one of the stories that's, it's a little
difficult for us to cover because some of the things that we're referring to.
ar Arnold all, they're not left like the original blog that started,
the investigation seems to be gone,
right?
Um, the, I don't know if the entire blog is gone or just the blog post.
Luckily the New York Times article is still around that refers to this.
But yeah, so they were posting, um, and, and they, what I remember was
that they tried to pass it off as nothing and then they, you know,
they just sort of got busted doing this and that, that just, and that.
Story broke just a couple of months before the big story that we're gonna talk about.
Any, any thoughts about that?
Prasanna Malaiyandi: Like I just think back to like 2009.
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: in the days where like people weren't
doing it all that much, right?
Posting fake reviews, and you didn't have the smart
algorithms to try to detect this.
You didn't have all the sort of review farms or paid reviews and all those
things happening like you do today.
And so maybe they just thought, yeah, maybe we could just get away with this,
and like that was probably, they did.
They probably didn't think that there would be a risk of get being caught.
Right.
W. Curtis Preston: Yeah.
One of the people that was posting a, a a review was the vice
president of marketing at Carbonite.
Right.
Um, so, so that's sort of the, the, the
timing under which, this happened.
What's
Prasanna Malaiyandi: by the, by the way, we're not saying it's bad to post reviews.
It's, there's nothing wrong with that, right?
We're just saying it's bad to post reviews, pretending to be other
people trying to give an unbiased or, or give an unbiased opinion.
W. Curtis Preston: Yeah, it is actually, um, you know, you can get in
trouble, like in the us you can get in trouble with the SEC if you do that.
Prasanna Malaiyandi: Later,
W. Curtis Preston: Yeah, yeah.
Like the SECC will resurface in the story.
Okay, so,
so here's what happened.
2009, a couple of months after that story basically.
Um, it's, it's interesting, so I'll tell it not in, this is sort of like Star Wars.
I'm gonna tell it in the order that it happened, not in the order
that we found out about it, right?
Because how we found out about it is.
Beyond my comprehension.
So what happened was that Carbonite had outsourced the backup part
of their backup software to an IT vendor that then purchased.
Disc drives from a company called Promise and $3 million worth,
by the way, $3 million worth of hardware from this vendor.
They then had a, uh, was it a dual disc failure?
Is that that what is that, what
we, Yeah.
They had a dual disc failure in their RAID arrays, and as a result of that, they
lost the backup data of 7,500 customers that doesn't mean that 7,500 customers
lost the data that they cared about.
They lost the most recent backup.
And then what they were able to do is they, they, they
immediately restarted the backup.
And they're saying that what we now know is that 54 customers out of that 7,500.
The data that they were backing up was damaged in some way.
Their PC crashed or they
deleted it or something.
And then they went to go restore data and they were unable to do so because
the data that they wanted to restore was in that, that data that was lost,
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: um, which is.
Formerly working at a backup vendor, that number seemed high to me.
The, the number of customers that would restore 54 out of 7,500.
Right?
That's
0.1%.
Prasanna Malaiyandi: Yes, but right.
That's just customers who encountered another error while before they
could, because who knows how long it took 'em to back up.
Right.
It might have taken them days depending on how much data they had, 'cause they
had to basically back up everything again.
W. Curtis Preston: Yeah, I, I guess I'm just saying that, you know, knowing
what I knew about, about our former employer, people just don't restore stuff.
They just don't do a lot of restores.
I would be
surprised,
Prasanna Malaiyandi: unless they needed to.
though.
Right.
And in
these cases, yeah.
yeah.
So, so, and that's their hope, right?
It's that they just hope that nothing bad will happen while everything
is getting backed up again.
And of course, for those 54 people.
W. Curtis Preston: Yeah,
Prasanna Malaiyandi: Did
W. Curtis Preston: it's sort of like how you hope that another disc drive won't
fail when you're only using raid five.
Um, you hope that that won't happen while you're rebuilding.
Prasanna Malaiyandi: So having worked for storage companies in the past, I am
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: but not surprised that they were not able
to handle a double disc failure.
Um, but.
It is a little crazy that they built a system like that and where it's
like, yeah, this is good enough.
Um
W. Curtis Preston: Let's get, let's get to that.
Let's get to that in a minute.
I wanna ask you a really big question, Prasanna.
Prasanna Malaiyandi: huh.
W. Curtis Preston: How did we find out about this?
Prasanna Malaiyandi: Oh, so it was published in some articles right after
the fact, but the only reason that this was even picked up by the news was because
Carbonite decided to sue the vendors.
W. Curtis Preston: Yeah,
Prasanna Malaiyandi: breach of contract and breach of warranty and
W. Curtis Preston: the IT vendor and the storage vendor.
Prasanna Malaiyandi: Yep.
In order to recoup some of the costs.
And that's where all this information came out as they were seeking damages.
W. Curtis Preston: Yeah.
And I remember being, you know, like backup guy at the time, and
I mean, this is what, 15 years ago
and.
I remember like to the day, it's like 15 years ago and I re my fir.
What's interesting was timing wise, 15 years ago I had my
first Mac that had Thunderbolt.
Prasanna Malaiyandi: Hmm.
W. Curtis Preston: Okay.
Why does, why does that figure into the story?
Here's why.
At the time I was running a company and I needed, I needed a decent sized
storage array for my, um, by the way, the the computer I'm talking
about is literally right over there.
It's literally sitting over there.
It's a, it's a I, um,
what, uh,
Prasanna Malaiyandi: Hi Mac.
W. Curtis Preston: The, yeah, the I, the new imacs,
right?
And it had Thunderbolt and we were doing video editing on it, and we
needed a nice sized array to buy it.
And the only arrays that you could buy at the time that were
thunderbolt were promise arrays.
And when you assert at the time, the promise arrays, they, they
were, I mean, they were great.
I, you know, I had it.
I actually still have the promise array somewhere, and.
But the thing was that they were also known for being the cheapest
arrays that you could buy at the
time.
So when this story broke, I.
I, I I sort of realized a bunch of things.
I'm like, wait, this big cloud backup company is using the cheapest storage
arrays that, and they're just using like regular old storage arrays.
Not like good ones.
They're not, you know, they're like, they're supposed to be the cloud.
They're supposed to be, they're supposed to be using advanced technology and,
and, you know, so, so there was that.
The other thought I had was.
Wait, they just have like one comp, they just have one
copy of each customer's data.
And it's not even using RAID dp, it's not using mirroring, it's not using RAID dp.
So, so all it takes is one double disc failure
And and you're done.
And, and then the other thing was I, I said to myself.
Why, if this was me first off, I wouldn't have, I, I don't think
I would've made that decision.
I think
I would've done something very differently.
Second, if I had made that decision, it's the, I'm gonna publicly
announce to the world that I made that decision and like, what, what
was going through anybody's mind?
Prasanna Malaiyandi: It
seems like a big PR
snafu.
W. Curtis Preston: big, PR snafu, a bad business decision to buy
the arrays in the first place.
I'm not saying that promise arrays are bad.
I will say that having that much data without having RAID DP bad,
right, that much data without having mirroring or something like
that, that where a double disc failure just takes your data out.
Double disc failures happen all the time.
You, you, you used
to work at a company that did this?
Prasanna Malaiyandi: That's why.
So there is a quote, and I don't know if we, yeah, and I will cover it now.
Uh, where basically, uh, the CEO David Friend
W. Curtis Preston: at the time.
he's no longer the CEO.
He was the Yeah.
Prasanna Malaiyandi: Yeah.
So he had explained sort of, okay, this is why they had crashed because we were using
Promise Technology and we're using RAID, and we suffered as single disc failure.
So he said that we switched to a popular Dell server that uses RAID six.
That allows a loss of three of the 15 drives simultaneously
before you lose any data.
The configuration is, in theory, 36 million times more reliable
than a single disk drive.
The chances of three out of 15 drives failing at the same time are almost dill.
Now, having worked at multiple storage companies, I want to call,
uh, BS on that, nearly nil, right?
W. Curtis Preston: On the three drives, failing at the same time.
Prasanna Malaiyandi: Yeah, having worked at companies, especially given
the size of discs these days, what I've experienced is that during the
times when you're actually doing a raid rebuild, that's actually when you're
putting additional stress on all the
other discs.
As you're doing a whole bunch of reads, recomputing parody, in order to write
out and recover the new disc, that's actually the time in which those
other discs, if there's any weakness in them, that's when they fail.
W. Curtis Preston: I feel a story coming on Prasanna.
Prasanna Malaiyandi: What's the story?
W. Curtis Preston: The story is my oracle, the, so this is the
first time, so I, I, I, I left.
I was no longer like an it.
I.
Regular IT practitioner.
I was now a consultant and I was at this major oil and gas company and
I was just supposed to be the admin.
I wasn't supposed to be the backup guy, but I went around and I've seen
that the backups were just broken.
And, you know, and I, and, and one of the servers that really bothered
me was, it was, it was a Solaris.
I remember it was running Solaris two three, which at the time was a
seriously buggy version of Solaris.
And, um, I was trying to get the first ever backup of this Oracle database,
which was a three 300 gigabyte,
three oh
gigabyte Oracle database.
Massive, huge, right?
And, um.
It was what was called a DSS decision support system.
It was all the rage back in the day and it was the kind of thing that took
imports nightly from the mainframe.
And, um, it had never been backed up, ever.
And I had to argue with the DBA that um, he didn't want me to do.
He didn't want, he didn't want me turning on archive log mode because AR archive
log mode corrupted Oracle databases.
That was his
Prasanna Malaiyandi: stance.
Yeah.
W. Curtis Preston: I'm like, is that the stance of Oracle or is that just
some random thing that's in your head?
And it was the latter.
Um, and then I, um, and he, uh, he said, well, if it doesn't corrupt it, it
cau, it causes performance, uh, lower, you know, it lowers the performance.
And, um, I was like, well look, we've gotta get this
thing backed up at least once.
And so either I shut down the database and back it up one night,
or, which they didn't want to do 'cause that's when they did the night,
the nightly uploads, or I need to put it in archive log mode
at least for a couple of days.
So I can do a hot backup of it.
And um, and then I went to go, I, I went and that's when I looked at
it and I saw that not only was it Solaris two three, but it was Solaris
two three, completely unpatched,
right.
It, it, it had been like a couple of years
Prasanna Malaiyandi: one wanted touch
W. Curtis Preston: Patches and, and so I, I knew because I had already
encountered it at this client, that if I put the backup software that I was using.
The features of Solaris that it used during a backup would cause it to crash.
If I kicked it off, I knew I had to put on the latest jumbo patch of
Solaris in order to do the backup.
And so, uh, I did.
And uh, I got it rebooted.
And this was like a Friday afternoon and I got my first ever backup
of this 300 gigabyte database.
And then over the weekend, five drives failed.
Right.
Prasanna Malaiyandi: Should never have rebooted the server, Curtis.
W. Curtis Preston: yeah.
Yeah.
So, um, so I came in Monday morning and, and basically the, the drives failed.
I got, and by the way, it was the only an additional wrinkle was this was a place
who didn't buy Sun Service from Sun.
They bought like pieces
and
parts
Prasanna Malaiyandi: and.
put it together themselves, and then,
W. Curtis Preston: Yeah,
yeah.
But this server was from Sun and so I was able to call sun, get sun in
over the weekend, replace the drives, you know, rebuild the file system,
restore the database, and it was up and running come Monday morning,
and I was cock of the walk, baby.
Right?
And then this, this, um, this, uh, this DBA comes up to me
and he sees me and he's like.
I bet you think you're hot shit, don't you?
And I'm like, yeah, yeah, I kind of do.
You know, I'm feeling pretty good about myself right
now.
And he said, well, I'll tell you what, you don't get any credit.
I was like, why is that?
He goes, well, he goes, I'm not exactly sure what killed the server or those disk
drives, but I think it was your fault.
He is like, I have a couple theories.
One is you did this backup and it exercised the disc drives more
than they were used to being used.
And so that's what caused the thing, uh, to fail.
Or it was, uh, the fact that Oracle was in, uh, arch archive
log mode and that's maybe what caused it or the fact that you put in that
big jumbo patch, um, and one of, one of those things killed the disc drives.
And so since you killed the disc drives you get no credit for, for saving us.
I was like, okay, whatever dude.
So
yes, multiple disc drives in this case, five disc
drives can fail all at once.
This is why we back up.
This is why we do mirroring.
This is why we do all kinds of, I, I guess it just really disappointed
me at the time, and so it, it was really surprising that a company who
this is what, this is what they do.
They, they were just using the same RAID arrays, you know,
that I'm putting on the back,
on the back of my iMac.
I was very disappointed.
Um, I, I, I spoke a lot there,
but it just bugged me when it happened.
What
Prasanna Malaiyandi: yeah.
No, I, I agree with you.
It's, and the challenge is their marketing messages and everything else at the time.
Right.
Portrayed it as, uh, high quality.
I.
Backup solution for customers.
So they don't need to worry about backup and everything else that
that comes along with it, right?
They're like, yeah, we will deal with everything for you.
And when customers put their trust and their faith in that, uh, the
company didn't quite meet the bar.
W. Curtis Preston: Yeah.
Yeah, and, and I guess it's just.
Even to this day in the subsequent articles that came out after the original
article, when the CEO David friend was going to the people say, Hey, you know,
I just wanna correct the, this, you
Prasanna Malaiyandi: It's not 7,500.
Yeah.
W. Curtis Preston: it wasn't 7,500 customers that lost data.
It was 54 customers lost data because only 54 customers couldn't restore.
Right?
But still in those further comments, he was still like blaming, promise.
Like he even, you know, um, and by the way, I do wanna say I was
able to find the, the, docket, uh, the actual filing and , the
lawsuit was settled outta court,
right?
So there's no, there's no public record of what the actual thing is,
but I highly doubt that, um, you know, they're a storage array vendor
disk drives fail and sometimes monitoring stuff fails and.
You know, I, I, I just, I could not, for the life of me fathom the idea of, of
suing your, your storage array vendor.
It's a little bit like when Musey sued
Google.
because they deleted their own data.
Right.
Um, yeah.
I, I just don't understand that.
So there was an article from Backblaze, uh, which for the record
is a competitor to Carbonite, but I still agree with their point.
And they were saying that they felt that the coverage of this.
Incident missed.
One of the crucial points of this story, and that is that a backup vendor
outsourced the backup and, and they made a, you know, that they, that
they made a point of saying that, that it's like Google Outsourcing search.
Right.
And, and I, and I have to agree with 'em, right?
They, they weren't saying they were reselling backup.
They were, they were saying they were a backup vendor, and you would
think that they would deal with that.
Right?
Um, and, uh, at the time, the, the response to that was that
the CEO said that they now, now, write their own software.
Right.
So, I don't know.
The, um, yeah, in, in that blog it says Building robust online
backup technology is difficult.
There's certainly lots of complexities involved to ensure data is
backed up, redundant and secure.
It is the role of the online backup service provider to have
the technical expertise and laser focused to work through these items.
Pushing it off to an outside company seems a bit risky.
Prasanna Malaiyandi: Yeah.
Well, and the fact that we started with this right as the 3, 2, 1 rule.
They were not doing the 3, 2, 1 rule at all, you know?
W. Curtis Preston: yeah, yeah, I mean, you know, there are, there
are probably a bunch of vendors out there that, for example, are relying
on, you know, backup vendors that are storing only one copy on S3,
right?
Yeah.
Now as three stores, three copies.
But you still are, you still are relying, but in that case, you're relying on what
arguably is like the most tested storage array service in the world, right?
Prasanna Malaiyandi: Yeah, exactly.
W. Curtis Preston: Yeah.
Uh, but as opposed to promise, which was a little known
product that, that I'm willing to bet.
The reason they bought 'em is they were the least expensive.
That's
why I bought 'em.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: That and the fact that they did Thunderbolt at the time.
It was really hard to get Thunderbolt based storage array
Prasanna Malaiyandi: I think at the time it was probably OWC and Promise.
W. Curtis Preston: Yeah.
Yeah.
And OWC was probably more expensive, so I
didn't buy it.
Um, so what happened after that story Prasanna, which I didn't even know
about until we were researching this, this story until you brought it up.
Prasanna Malaiyandi: So apparently I came across while looking
for lawsuits against Carbonite.
Uh, it looks like they just settled one, um, in January, uh, 31st of 2024.
So a couple of months ago from this recording, uh, for 27 and a
half million dollars apparently.
So Carbonite was acquired in 2019 by a Canadian, Canadian
company called OpenText,
W. Curtis Preston: So five years ago
they were acquired.
Prasanna Malaiyandi: Yep.
But before they were acquired, Carbonite apparently had made some statements that
misled investors and they were releasing a new backup solution, uh, called
Server VM edition that they promised was super strong and extremely competitive.
That launched in 2018 October, and the company said the product would
allow businesses to recover virtual machines data from a single location.
And then they got sued by investors because they said the product, it never
worked and that it never once successfully backed up a customer's data and that
the executives basically knew it did not work and still made those statements.
And so this was just settled for 27 and a half million
dollars to resolve the claims.
W. Curtis Preston: Yeah, it's interesting.
So they accused the company of violating the Securities Exchange Act of 1934.
After this news came out and the stock price fell more than 24% the next day,
uh, the judge initially dismissed the case, but the court of appeals reversed
'em in 2021 and, um, and certified the, the class as a class action.
And then it looks like they finally settled it,
um, and agreed to pay $27 million.
That's a lot of money.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: I wonder how many promise arrays that'll buy.
Sorry, that was, that was, that was just mean.
Um, I, yeah, I don't, I, you know what I, what I used to, what I used to say
about Carbonite was, you know, just Google, like if you're, when you're
looking, when you're looking at vendors, just Google their name and lawsuit.
'cause at the time, Carbonite was the only one that that would come up with.
With results.
What's interesting here, like, it's sort of like with with OVH,
why in the world would anyone use.
The, a vendor, why would they go public?
Why would somebody invest?
And it's because these stories, they just go under the, under the
radar.
I think a, I think a, a, you know, an IAS vendor like OVH, who
showed that they don't know how to build data centers that they
Prasanna Malaiyandi: they
went IPO.
W. Curtis Preston: Didn't, yeah, they didn't do fire suppression.
They didn't do, um, you know, I Why would you, why would you ever
want to do business with them?
And I have a similar feeling.
This is just my feeling, my opinion, if a company is so, so,
two incidents of clear misjudgment.
One in terms of the original design and outsourcing to another vendor, and
then the second, the decision to, to make this public by, by blaming the
vendor.
What that, what that says to me is you take no blame yourself.
If, if they had done a mea culpa, if they had said, Hey, uh, we really screwed up.
We shouldn't have trusted everything to this vendor.
Uh, maybe even trashed him.
I don't care.
Right.
Now that I think back on it, that's the part that bothered
me the most in the, in the story
is that to this day, they take no responsibility for that.
Uh, they also, they also take no, they know, um, there's no, um, wrongdoing.
They admit no wrongdoing in the SEC, uh,
the settlement.
Prasanna Malaiyandi: Yep.
W. Curtis Preston: Uh, at least they're consistent.
Prasanna Malaiyandi: So, yeah, so I, I think.
Like you said, this is useful as you're, because I don't
think people actually do this.
So when you're searching for a vendor, we always talk about go read reviews,
go talk to other people, look at, uh, analyst results and or analysts
opinions and things like that.
Look at like the IDC surveys and, uh, the Magic Quadrant from Gartner.
I don't think that we've told people, go Google the Vendor plus lawsuit.
I.
You know,
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: go look and see what has come up.
Like is this a company that you are comfortable doing business with
and trusting that they will stand beside you when things go wrong?
If things go wrong?
W. Curtis Preston: Yeah.
And, and I think it's important to say that I have no ax to grind with
these, with these folks, right?
I, I don't work for a competitor.
I don't, they didn't personally wrong me.
Uh, they've never discussed me publicly that I know of.
It chaps my hide.
It just the idea that that.
You accepted no responsibility for that.
And then again, apparently even up to recently, it, it appears
that you were doing stuff, um, that seems a little shady,
Prasanna Malaiyandi: Yeah.
Now granted that was 2018 when they made those statements, but.
W. Curtis Preston: That was not that long ago, man.
That was just before Covid.
That was like, I was like, yes.
Covid iss like a day like Covid, COVID.
The entire period of covid counts as like a day.
I, yeah, that was 2019.
Prasanna Malaiyandi: Yeah.
Now the one thing though that we have to state is since the
acquisition by OpenText, things could be different, right?
W. Curtis Preston: that is true.
Prasanna Malaiyandi: So I think we should at least give 'em that benefit
of the doubt that things could
W. Curtis Preston: That's actually a really good point.
Thanks for bringing that up.
You know, we're, we're commenting mainly about the company that was.
Carbonite before it was acquired by OpenText.
And perhaps this acquisition of OpenText will significantly change things, uh, or
has already significantly changed things.
This is really what, what we need to, um.
When I think about like the OVH situation, what can we learn from this?
What can people do to protect themselves from this when
using a cloud backup vendor?
I would, I would number one, be asking them what kind of
storage technology they're using.
Right.
Prasanna Malaiyandi: I would
ask about the availability and redundancy,
W. Curtis Preston: Yeah.
Um, you know, are you using storage arrays or are you using, are you using S3?
Are you
using, you know, what, how are you storing my data and how are you making
sure you know that it stays available?
Prasanna Malaiyandi: I, I, think though, as a user, maybe what
underlying technology is less important because I could see some vendors
maybe would not discuss that I.
But at least ensuring the characteristics that you expect of S3.
So for instance, is it geographically spread out?
Right?
Is it redundant?
Is it a replicated copy, independent copy, right.
Those sort of things.
W. Curtis Preston: I understand what you're saying.
They might not want to say, but like, you know, where we used to work,
we just say, yeah, it's, it is on S3.
Right.
If you could say that,
I think that gives you a certain amount of, right.
Just like you just said, is this designed to survive, um, you know,
a fire, like what happened at OVH?
Is this designed to survive?
Um, you know, a multiple disc failure.
What happens in a mul, in, in, in a, in a catastrophic failure of,
of, you know, because let's just, let's just, let's just say this.
'cause here's the thing.
A single fire would've done the same thing.
Prasanna Malaiyandi: Yep.
W. Curtis Preston: And then who are they, who are they gonna sue then
their fire extinguisher company.
Right?
What the core problem here is that they had a single copy of customer data.
Prasanna Malaiyandi: yep,
W. Curtis Preston: That's it,
right?
No redundancy.
No.
No geo redundancy.
Prasanna Malaiyandi: Or if they had Acho go
W. Curtis Preston: redundancy.
A derecho.
You love talking about that
derecho, don't you?
Prasanna Malaiyandi: I love talking about the DRE show.
It's an
W. Curtis Preston: I never
even heard of a derecho until we
talked about it on the show.
Prasanna Malaiyandi: Exactly.
W. Curtis Preston: Yeah, it's a hurricane that starts over land.
For those of you that don't know what a derecho is, and don't ask me why.
It's the same as the word or really close to the word for right.
In Spanish.
Um, any, can you think of any other, you know, things that
a, that a customer can do to,
Because you know, the, you know what this reminds me of?
You know, there's, you know, you remember the old thing of like, on the
internet, nobody knows you're a dog.
Right.
It's like what can a customer do to ensure that they don't go to
essentially a another vendor that behaves the way Carbonite did back
in 2009?
Prasanna Malaiyandi: I think another thing you should do, and it's not
a guarantee, but at least gives you some protections, is take a look in
their SLAs and in the contract, right?
Do they talk about disaster recovery redundancy?
Do they talk about SLAs?
Do they talk anything about geographic separation for storage?
W. Curtis Preston: Right.
Prasanna Malaiyandi: Right?
If it's there, that's good.
If it's missing, ask the questions.
W. Curtis Preston: Yeah.
Yeah.
I think ask as many questions as
Prasanna Malaiyandi: yeah, it's not foolproof.
as we found with, like we found with OVH.
Right?
It was written in the contracts, but it still didn't mean anything.
But at least it gives you a more legal standing for if there is
a lawsuit that comes from this.
W. Curtis Preston: Yeah.
Um, and, uh, I, I hope, I hope the audience has appreciated this.
This is our best attempt to, to deduce.
I'm looking up here at my browser and there are like 17 tabs open to the story,
to the different story, all of which we're trying to cover the same article.
And, and one of the reasons for that is that the, the,
the original article is gone.
The original article in the Boston.
Uh, globe.
Is it that original article is gone and I couldn't, I also couldn't
find it in the internet archive.
I tried doing that.
So we had to look at a whole bunch of different blogs that were, uh, essentially
trying to summarize the same article.
But, um, you know, this is our best attempt at trying to,
to see what we saw from all
these various
Prasanna Malaiyandi: and, and it may not be a hundred percent accurate, right?
We're just going off the information
W. Curtis Preston: Based on the articles that we have.
Yeah.
Yeah, All right.
Well, it's been not fun.
It's not a fun one.
I don't like it.
This is
my world and I This is, this is, this is people that lost data
Prasanna Malaiyandi: it's,
W. Curtis Preston: that did what we told 'em to do, and they still lost data.
Prasanna Malaiyandi: so it's not fun, but I think it is hopefully
eye-opening for folks that they need to be more inquisitive and
not just blindly trust vendors.
W. Curtis Preston: Yeah.
Never blindly trust vendors.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: Yeah.
All right.
Well, uh, thank you Prasanna for your usual
analysis.
Prasanna Malaiyandi: I, I enjoy these.
I actually like this series, Curtis.
I hope our listeners do as well.
W. Curtis Preston: I.
Yeah, I'm, I'm enjoying the series as well.
It's been nice to go back to these stories that I remember from 15 years ago, and
then we're, you know, in some cases, like the OVH Fire, we're seeing some follow up.
This one, and in this one we saw some follow up we didn't expect
from quite a bit later, which technically unrelated, but you know.
Anyway, um, I, you know, listeners, we appreciate you.
Uh, you know, we'd be nothing without you and, uh, be sure to subscribe
so that you don't miss an episode.
That is a wrap.