W. Curtis Preston: Welcome to the exciting final episode in
our series on cloud disasters.
During which we conclusively answered the question.
Do I have to back up my data in the cloud?
The answer is an unequivocal.
Yes.
Not having a good backup and disaster recovery plan for your data in
the cloud is nothing short of an existential threat to your company.
Two out of the 10 companies we covered in this series immediately went out
of business after what happened.
And another killed off an entire line of business.
The cloud is an amazing place and I consider myself an
advocate, but it is not magic.
It is just someone else's computer.
In this episode, we will summarize the lessons that we learned and the major
arguments that we settled from our coverage of no less than 10 disasters.
If you're just joining us, hopefully this episode will wet your appetite
enough to have you go back and listen to, or watch those other episodes.
We had a good time making them.
And we also learned a lot.
By the way, if you don't know who I am, I'm w Curtis press an AKA Mr.
Backup.
And I've been passionate about backup and recovery for over 30 years.
Ever since I had to tell my boss that we had no backups of the
production database, we just lost.
I don't want that to happen to you.
And that's why I do this on this podcast.
We turn unappreciated backup admins into cyber recovery heroes.
This is the backup wrap up.
W. Curtis Preston: Welcome to the show.
I'm w Curtis Preston, AKA, Mr.
Backup, and I have with me my trauma sympathizer Prasanna
Malaiyandi, how's it going?
Persona.
Prasanna Malaiyandi: I am good, Curtis.
Yeah.
Um, it's interesting of all the things you can complain about or that are
wrong in this world for you, the biggest issue is different operating
systems using back slashes differently.
W. Curtis Preston: Y.
Yeah, so.
It's because I'm being forced to operate a Windows based net backup server.
I.
And I only know how to work in like Unix land, like when I'm doing automation
and scripting and stuff, which is the only way I know how to do anything.
Right.
I, I, I don't understand.
I, I guess people, if they grew up in Windows Land, they learn PowerShell.
They learn, you know, batch programming and what I didn't learn all that stuff.
I learned shell scripting.
And so when I want to administer a Windows-based system, the first
thing I do is install siggu so that I have a Unix-based environment.
But when I'm interacting with, um.
Files.
Right.
Um, so sig one is nice in that it translates all the file directory names
and everything into slashes, right?
So it's, I just have to, um, sort of figure that out.
And basically, like Deco backslash becomes slash sig drive slash d, so
you just sort of, that's not too bad.
But in the case of net backup, I'm doing things like.
I am running commands that output file names that sometimes are in Windows
format and sometimes are in Unix format.
And, um, the, what I've recently discovered is that, and it took me,
it took me literally just banging my head into the wa into the wall
a few times to figure this out.
Is that, um.
Back slashes, behave differently.
Um, you know, like I wanna massage them and the first thing I wanna do
is change 'em and four slashes, right?
So that I can, so that they're not gonna be escaping everything in my unit.
She script, right?
Um, they behave differently on like in standard in then they do in a file.
So if I take the output of a command, I.
Such as BPPL include, which is, you know, show me the files that are in this policy.
Uh, and I pipe that directly into a, uh, you know, into a series of
pipes to massage the, the output, uh, that backslash behaves differently.
I have to escape it differently if I'm just doing a, a straight.
Prasanna Malaiyandi: Yeah,
W. Curtis Preston: if I out, if I take the, the command and actually
output it to a file and then
and then look at it.
Um, and then basically I have to es like if I'm doing it on the command
line, I have to escape the escape twice.
If I'm doing, if I'm doing it in a filing, I only have to escape it once.
And that's been driving me just absolutely bonkers.
Prasanna Malaiyandi: So I think what you do is you just add a whole
bunch of escapes in there, you know,
W. Curtis Preston: I
Prasanna Malaiyandi: slashes
W. Curtis Preston: it works.
I'm just like, does two will two work?
No.
Two, no, no.
3, 3, 3.
No.
And then three, like three, uh, tells me that I, that I have too
many and I have like, uh, it, it, it, it escapes like something
later and I'm like, no, no, no, no.
Okay, so I won't take four.
You know, and I just, I just do a bunch of them until.
I get what I
Prasanna Malaiyandi: It works.
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: It's like, it's like, uh, it's like writing code, right?
You compile it, it fails.
You're like, oh, I'll fix this, and then it fails again.
It's like, oh, I'll fix this, and then you just keep going until
eventually you get zero errors.
W. Curtis Preston: And, and, and then, so I, I find that as annoying as this is, I
find that it's actually easier to take a net backup command, output it to a file,
and then hash through that file, rather than just doing it as a series of pipes.
And as annoying as that is, it's more, it seems more consistent, um, albeit slower.
And it's also slower because the first thing I have to do on that
file, you know what I have to do?
I have to run the dos2unix command on it
because of the, the new line.
Because the new line is also different.
Prasanna Malaiyandi: Yeah,
W. Curtis Preston: It, it's been fun is what I would say.
Prasanna Malaiyandi: Oh, Curtis
W. Curtis Preston: Ah, so, uh, so speaking of fun persona, we
have now terminated our cloud
Prasanna Malaiyandi: concluded.
W. Curtis Preston: series.
Prasanna Malaiyandi: Concluded,
W. Curtis Preston: What's that?
Prasanna Malaiyandi: concluded.
W. Curtis Preston: Why did I say terminated?
Prasanna Malaiyandi: Yeah.
Terminated just seems.
W. Curtis Preston: said I'm done.
I'm done with all these people losing data.
Yeah.
We have concluded our cloud disaster series.
What did you think?
Prasanna Malaiyandi: I liked it actually.
It's one of those things, like a disaster happens.
You read it on the news and then you forget about it, right?
It says to the next day.
And or you, there are many of these that like I don't think
any of us had heard about.
Right.
And just being able to go through and just understand and kind of take a look right
and getting your perspective right and seeing, okay, what are these disasters?
And really sort of doing the research ahead of time.
It was kind of fun.
I liked them.
W. Curtis Preston: Yeah, it was fun.
It was, um, what I liked about it was that enough time had passed with most
of these disasters that we often had the other half of the story, right?
When the, when, when they happen, uh, you know, at the mo you know,
at that moment you check it and you hear like in the case of, um.
The, uh, the Musi outage, the first thing you hear is that, you know, musi has sued
Google because, you know, they lost their data and, and that makes the headlines
and everybody's, oh, you know, there's a lawsuit, you know, but you know, this
is America, so anybody can sue anything.
I mean, you may remember that there was a woman who sued.
Google Maps because she drove her car off a pier.
Right.
Because Google Maps told her to go that way.
Right.
You know, you could sue for anything, but that doesn't mean
you're gonna be successful.
And it was nice to go back and look at the, um, you know what happened
after, in some of the cases, we got some additional insight from people
that were, you know, in and around.
I, I think of the.
OOVH incident.
Right?
Uh, that was nice.
We got some insight into that and how that OVH really was known for the, you
know, being a low cost provider and so that nobody was pro, nobody was surprised
when they, uh, didn't have, didn't have.
Prasanna Malaiyandi: Yeah.
Yeah.
Well, the other thing also that I came to realize from the series is,
like you said, time had elapsed.
It's also interesting to see how the company, some companies try
to sweep things under the rug,
W. Curtis Preston: Right,
Prasanna Malaiyandi: right?
And thanks to the wonders of the internet where nothing is ever gone, right?
Being able to uncover, okay, really, what was it that was initially
said versus what do they say today?
W. Curtis Preston: Yeah.
Uh, thank you very much.
Um, internet archive, right?
That was, that was, that was very helpful.
In some cases we weren't able to get stuff, but, and there were a lot of cases
where we were able to get basically the first version of the story versus the,
the later, the, the latter version.
Prasanna Malaiyandi: Yeah.
Uh, but, but I would say in most of these cases, right, the initial
information that people saw was what was allowed to be released, right?
Whatever the company had at that time, which sometimes they're working off
incomplete information, but they still wanted to provide some information
to their users, such as like the OVH incident or Rackspace, right?
Where it's just, okay, what information is available?
Let me at least push something out there.
W. Curtis Preston: Right.
And that, and that's what we encourage, uh, companies to do, right, is to
be upfront about what's happening.
And it's okay to say you don't know yet, but acknowledge what's happening.
You know, acknowledge what you know when you know it.
Don't make stuff up.
Right.
Um.
And the, and then, and then the, the other thing that I would say is, um.
Is we like it when they don't try to pass blame.
That happened a couple of times.
Uh, when the blame is, you know, clearly on you and your design,
uh, don't try to pass blame.
So we've got, I came up with a few lessons that I wrote down, uh, and
if, and if, and if a few others pop up in our head while we're recording
this episode, just come out with 'em.
Um, and the first one is actually, uh, you put that on there.
So you wanna, you wanna go for it?
Prasanna Malaiyandi: Yeah.
Yeah.
So I would say that the most important thing for a lot of
these is if it's important to you, the data, of course, back it up,
W. Curtis Preston: Yeah.
Which is, which is basically backup lesson number one.
Is
Prasanna Malaiyandi: Or zero, I would say.
W. Curtis Preston: Yeah.
Backup.
It's Rule zero.
Uh, it's one that I absolutely cannot argue with, right?
It's something that I say to you, right?
Prasanna Malaiyandi: Yep.
W. Curtis Preston: I, I say this all the time, that if, if this data
really is important to you, then it should be, you know, backed up, right?
And we can talk in a little bit as to what constitutes a backup and,
uh, that'll be some of the lessons that we learned along the way.
But, um.
I, I'm gonna say in almost all instances, what constitutes a
backup is you making a backup.
Not, not saying, oh, I got it because I got it, because it's the cloud,
or because it's SaaS, or because it's, you know, it's not my problem.
No, it's always your problem.
Prasanna Malaiyandi: Yeah, and there were companies, right, like Musi went out
of business because they lost all their data and they couldn't recover it, and
that was the end of the company, right?
So,
W. Curtis Preston: Yeah.
And, and I'd say that that's, that's the first lesson I'm gonna say that
wasn't actually on the list, is that if you don't get this right.
Your entire company may just go away, right?
That happened, uh, in at least two instances in the, the list
of the companies that we found.
Either your company or a significant portion of your company may go away.
I think of Rackspace, right?
Rackspace is now completely out of the hosted exchange business because.
Uh, of what happened with the rec based outage, right?
So either your entire company or a significant portion of your company
can literally just go, poof, overnight.
Prasanna Malaiyandi: Yeah.
So what's the first one, Curtis, in your opinion, like what's the
most important lesson from all of the ones that we've looked at?
W. Curtis Preston: 3, 2, 1.
Rule still rules.
And, and I know that there are other versions of the 3, 2, 1 rule.
I'm mu in this case, what I'm saying is if what you are doing for backup, as I make
quotes in the air, doesn't at least comply to the 3, 2, 1 rule, it is not a backup.
Right?
There are, there are many situations when we look at the stories that we, that we
talked about here, where people had what they might have considered a backup.
Right.
Um, you know, musi is probably the best example of that, right?
Um, where they had their backups with them, but basically
they deleted their account.
Right.
Um, the, the other one would be Code Spaces.
That's probably the most infamous, I think, where Code Spaces was making
backups within the, within AWS, but they didn't follow the 3, 2, 1 rule.
They didn't separate those backups from production.
And so, when.
Um, you know, when the, the bad actor went in there, they deleted the entire account.
And again, poof, they went there, went their primary as well as their backup.
And the same would be true of the SaaS.
Um, people that, you know, when I think of specifically Microsoft 365, but this is
true in so many other cases, is that, um.
It that if you are not separating that data, if you're not making
another copy, they don't have, uh, multiple copies of your data, right?
That, that are accessible to you.
They may have, may have a DR copy, but as we saw in some of the cases, the DR.
Copy does doesn't always work.
Um, that Dr.
Copy
might not be to you, and it might not, it might not work, but.
But the point is that if you're not, if you don't have a whatever they
have, it's all in the same place.
And when very bad things happen, uh, your SOL
Prasanna Malaiyandi: And specifically around the Microsoft case, right?
They may have a DR copy, but like you said, that's for their purposes
not to allow you to do your restores.
That's to make sure if something blows up somewhere from an infrastructure
perspective, they can break things back up and they'll try their
best to give you back all your data, but there's no guarantees.
W. Curtis Preston: Yeah, exactly.
Actually we're gonna get more into detail into that in, in a latter step.
But yeah, so short version is, I, I have no problem with app
pending to the 3, 2, 1 rule.
If you wanna say 3, 2, 1, 1, 0, 7, 9, 5, I don't care.
Right.
Um, and I, and I don't disagree with any of the versions that I've heard.
Right.
Things like I.
Making sure that, you know, one of the copies, the biggest one is
making sure that one of those copies is an immutable copy, which is a
relatively new requirement, and that has to do with, uh, cyber attacks.
I don't have a problem with that.
But if what you're doing doesn't comply with the 3, 2, 1 rule, uh, three
copies of your data on two different types of media, one of which is
physically separated from the other, um, then it, you don't have a backup.
Prasanna Malaiyandi: I do wonder, and maybe it doesn't have to be for
this episode, if we should really figure out does the three or have a
discussion on an episode of does the 3, 2 1 rule really need to be changed?
W. Curtis Preston: Um, I,
I think,
Prasanna Malaiyandi: I think just because I know that there's like different,
like you said, the 3, 2, 1, 1 0.
Zero.
It may not be a zero, but you know what I mean.
Right.
W. Curtis Preston: Yeah, well there, there's various versions
out there and I'm fine with those.
I do myself, like, you know, a, you know, I add to it.
One of these copies needs to be immutable.
I am, I fully agree with that.
Does it need to be part of the 3, 2, 1 rule?
I don't know.
I, you know.
I like the 3, 2, 1 for, for what it does, because what it does is it proves what
we've been talking about here is that if you have a cloud service and all of
the copies of your data are all in the same place, you don't have a backup.
Right?
Prasanna Malaiyandi: Yep.
Speaking of which, uh, I would say that the next lesson learned is really around.
Right.
Like you mentioned, the cloud is not backing up your data.
You may think it is backing up your data.
They may be doing DR copies for their own purposes, but it's not a backup
that you as a customer, a user can go restore from, can go back to any point
in time, recover it all the rest of that.
Which I think a lot of people mistake, right?
They're like, oh, I have a Salesforce, or Take your pick and it's a SaaS service.
They host my data.
They should be doing my backups.
Or Microsoft 365, they should be doing my backups.
And I know Curtis way back in the day when we were first working together, right?
You were like, have you ever taken a look at Microsoft's
contracts, terms of service?
Do you ever see backup in there anywhere?
W. Curtis Preston: Right.
Yeah.
It's just, it is just not there, right?
That and that and that.
That's the part it is.
If you think that your SaaS service is backing up your data, please go
look for it in your contract, right?
Because you're not gonna find it.
There may be, there are some exceptions.
Interestingly enough, one of the exceptions was covered on
our, on our stories, right?
Where we had a vendor that was backing up, we're gonna talk, we're gonna,
we had a vendor who was advertising that part of their service was backup,
which almost never happens, right?
You don't see that.
Go look at what, what Microsoft advertises for 365.
Look at Salesforce.
Backup is not part of the offering either in terms of what they
advertise or what's in your contract.
And if it's not in your contract, it doesn't exist.
Right.
So the next one, and this is we, we've covered this a little bit already, and
that is if they are backing up their data center, that backup is not for you.
Right.
Um, and, and I see this a lot.
Um, there's again, this, this one guy that I interact with a lot online that,
that really believes very strongly in the Microsoft way of doing things
and in his idea that Microsoft has, for example, multiple delayed copies
of your data, like replicated, uh, delayed, replicated copies of your data.
That it's, it's gonna use that in the case of disaster to be
able to recover your environment.
And that appears to be true.
I, I've never found it anywhere documented.
Right?
Which, and if it's not documented, if it's not in my contract that I don't care.
But it appears to be true, however.
The story of KPMG.
If that story with the size of KPMG 145,000 employees, if what happened
to KPMG doesn't prove to you that you don't get to use Microsoft's backup to
save your butt, I don't know what will.
This is a, this is a giant company paying tons of money to Microsoft every month.
Even they weren't able to recover from this giant blunder because it
was their fault, and that was not enough to have Microsoft use their
Dr copy, uh, to, to save them.
Yep.
Prasanna Malaiyandi: Or, or if we go back and think about like
the Salesforce example, right?
Where they changed on permissions and everyone had
access to every record, right?
And it was like, how do we undo that?
And so they tried their best and hopefully if you had a sandbox copy, they were
able to help you out a little bit.
But they were basically like, sorry, uh, I can't help you.
Good luck.
W. Curtis Preston: Yeah, that's interesting because the Salesforce
story, it's, it was their blunder,
Prasanna Malaiyandi: Yep.
W. Curtis Preston: right?
It was their blunder that, that, um, I.
Where they messed up everybody's account.
And even then, when we know that Salesforce has a, a backup, as I make
quotes in the air, um, that you can pay for like $10,000 an instance,
and it takes like weeks to get it.
But they were like, no, we're not gonna crack open Pandora's box, even though it
was us that messed up all these accounts.
So, yeah.
So th those two incidents, if, if those don't prove to you that these DR copies
that these vendors are making aren't for your purposes, I, I don't know what will.
Prasanna Malaiyandi: Yeah, and these are large companies, right?
It's not like a mom and pop cloud vendor, right?
Doing something or a SaaS application.
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: Going alongside that, so we're saying don't trust
the cloud vendor to do your backup.
Right.
And instead you really should be using a third party backup service.
If we looked at some of the cases like Deduce where academics, people
in academics and research, they lost.
Studies, right?
Because, uh, DDU went down or they lost some data, and if they had simply done
a backup or even simply exported their data, right, they would've been fine.
They wouldn't have had an issue, but they didn't because there wasn't part
of the service, there wasn't anything that could easily back that up.
And so people lost data.
W. Curtis Preston: Yeah, exactly.
I mean the, the Deduce story, uh, the Rackspace story, where it would've been
probably much easier for companies when they, when Rackspace went down and then
they immediately migrated everybody over to, to 365 would've been much easier
for them to recover from that situation, uh, if they had their own backup.
Um, and, and of course the OVH disaster where, uh, you know, the, they, so both
in the case of OVH and um, deduce, we know that backup was something that
they advertised as part of the service.
By the way, specifically in Deduce, you looked, if you look at their
website from back then, it says they're doing nightly backups.
And, and we now know that we don't know exactly how or why, but the closest
available backup was o was a month out and the one that was, and, and there
was something a little funky with it.
So they actually went two months out.
Um, what, how, how is that a nightly backup?
Right, right.
So I, I guess what you're saying is, or what I'm saying is that when.
Bad things happen and bad things happen.
Admins do wrong things.
Software engineers do wrong things.
When you're trusting a single vendor to do both your IT and your backups
of your it, if they turn out to be incompetent, they're probably also
incompetent in your backup, right?
And so this isn't, because both of us used to work for a cloud backup vendor, right?
This isn't us trying to, you know, get money for our firm employer.
Uh, this is just common sense man.
Prasanna Malaiyandi: Yeah, and I would also say for those companies where
they are focused on both your primary IT application as well as the backup,
typically they're not going to put all the money, all the effort in the backup side
because their main goal is to sell and add new features to the primary application.
And so usually you end up with something that's either half baked, not up
to date, doesn't support all your various scenarios, that someone who is
specialized in backup is thinking about.
W. Curtis Preston: Yeah, all those things there.
Yeah.
There are gonna be lots of important features, both from a security standpoint,
a recovery standpoint, all of that, that are going to be useful to you if
you ever actually need to do a restore.
In the case of any of the events that happened to the people
in our, in the stories that we covered in the last, what was it?
Like eight weeks, something like that.
Uh, what we got next.
Prasanna Malaiyandi: So, I think you did touch on this a bit earlier,
but be careful when someone sells you a feature and tells you it's
backup, but it's not backup.
W. Curtis Preston: Yeah.
Yeah, so this one's interesting because, and I, it sounds like
we pick on Microsoft a lot.
I, it's not Microsoft.
My issue is not Microsoft.
It's the people that seem to, that seem to, like.
In the case of Microsoft 365, you will not see the documentation referring
to retention policies as backup.
You won't see it.
Retention policies are an e-discovery feature.
E discovery and archive have nothing to do with backup, right?
They behave completely differently in terms of the way they save
data, the way they pull out data.
Um, and if you, if you don't understand what I'm saying, you
don't believe what I'm saying, go and try to restore a mailbox.
Using 365 retention policies, please go try that.
Um, the, um,
Prasanna Malaiyandi: tell us if it could turn out to be a plausible point
in time that that mailbox looked like
W. Curtis Preston: doesn't do it.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: It.
It's all about retention policies and the associated E-discovery feature
is all about how to get all email that ever went into or came out
of that box over a period of time.
It is not designed to restore your mailbox to the way it looked yesterday.
Right.
Doesn't restore folders.
It doesn't restore all kinds of stuff, and it doesn't know how.
It, again, just you, you gotta, maybe you have to experience it to truly
understand this, and that is that it doesn't know how to make your mailbox
look the way it looked yesterday.
It doesn't understand that concept.
That is a backup and restore concept.
Right?
And so that's, that's
Prasanna Malaiyandi: Not an archive and retrieval.
W. Curtis Preston: Yeah, it's about arch archive and retrieval.
Right?
Which, if you don't know what we're talking about, there's a whole
episode that we talk about why arch archive and backup are different.
Go, go check that out.
Uh, put a link in the show notes.
But the, but the other reason with 365 is when we look at that story of KPMG if
you, if you have a backup software, um.
Let's say, you know, pick your favorite backup software.
If you configure it really, really, really, really wrong, like just the
worst backup configuration ever.
You know what it doesn't do,
Prasanna Malaiyandi: Blow away your production.
W. Curtis Preston: it doesn't blow away your production.
Which is another key difference between Microsoft retention
policies and backup, because that's what happened in the KPMG story.
Prasanna Malaiyandi: you almost want a separation of duties, right?
That's the roles and responsibilities.
It's like no different than a cloud provider and the customers who are
on top of the cloud provider, right?
Each person has their own responsibilities and their roles and what they do
and what is provided, and it's all clearly articulated in contracts.
W. Curtis Preston: Yeah, retention policies.
When this person went to go do what they wanted to do, which was just delete one
person's chat, it's like, if I get this right, I will delete one person's chat
and I will have a really, uh, poor.
Facsimile of backup that is really lousy at Restore.
That's the best case scenario.
If I get it wrong, I'm gonna delete the personal chat
history of 145,000 employees.
It could, by the way, it could have been worse.
It could have been worse because.
Retention policies when you're in them, when you, when you first open them.
The, by the way, retention policy, the manual is 25 pages, right?
But you can go in there and you can go for all services, right?
Meaning
Prasanna Malaiyandi: It's not just for chat.
Yeah.
W. Curtis Preston: for all services.
Please set retention to one copy.
Prasanna Malaiyandi: So,
W. Curtis Preston: copy.
Literally in a, in a few mouse clicks, you can blow away your entire 365 environment.
Prasanna Malaiyandi: could you imagine if that had happened?
And KPMG, which is a huge company, loses everything.
W. Curtis Preston: Yeah.
I mean, it's bad enough that what they lost.
People are like, oh, it's just chat.
Well, chat is often quite valuable.
Yeah,
Clearly you've never lived in a company that used chat.
Uh,
Prasanna Malaiyandi: don't use smoke signals.
W. Curtis Preston: what's that?
Prasanna Malaiyandi: They don't use smoke signals
W. Curtis Preston: Yeah, it reminds me, it reminds me there was a company.
It was, I'm pretty sure this was a Xerox commercial years ago.
And it was like, does Xero, is your, is this, describe your company, like
if your copier goes down, do people say it's okay, we'll use carbon paper.
Do you remember that?
That was
a, that was a, that was a long time ago.
Um, so what's our next lesson?
Prasanna Malaiyandi: So going alongside what you talked about with Microsoft
365 and retention policies is, that's like a specific case for Microsoft.
But also cloud providers in general.
How about their designs are just awful.
Like, I think that's actually being too kind to say awful,
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: right?
I think it's just downright pathetic.
Uh, one of the ones that comes to mind is OVH.
W. Curtis Preston: Right,
Prasanna Malaiyandi: know I hate picking on OVH 'cause we always
pick on OVH, but I'm picking on OVH.
Right?
And it's really around the fact that they offered a backup service and
they basically had the backup server in the same data center, like just a
couple rows down in a separate rack.
And so when they had a fire that damaged both production and their backup data.
W. Curtis Preston: Yeah, I mean, I mean, that's bad.
Uh, I, I don't knows some equally bad, similarly bad.
Um, when I think about the fact, you know, if we go back in time, there was a time
when Carbonite was everywhere, right?
There were advertise.
You would think that Carbonite was the single biggest
backup vendor on the planet.
And they were storing customer backup data on raid five arrays.
Right.
I'm gonna say pro prosumer quality raid five arrays with, with no
dual parity, you know, um, nothing.
And then they, you know, and it was really just a matter of time before
what happened to them happened.
Right.
Um, you know, they had a double disc failure and Right.
You know, there was that one.
And then, you know, we have the other one.
Of, um, you know, code spaces where they stored their backup in the same
place that they stored production.
We already talked about that with the 3, 2, 1 rule.
Prasanna Malaiyandi: Yeah, and I think the other one there to also talk about is.
And I don't know if it's really poor design or just not understanding
backup and backup requirements is, I would actually say Rackspace,
right?
They had a hosted exchange offering, which is great, but they didn't
really have a backup solution.
They had never tested it out.
They were sort of trying things on the fly after they ran into issues.
W. Curtis Preston: Yeah, the part of that story that was the worst was
it's like, okay, we've been working on this for a couple weeks now.
We think we have a good plan for what, you know, for how
we're gonna get your data back.
Uh, you know, and then like a little bit more time passes and it's like,
oh, we've now tested the plan.
It works.
Now we need to go do the plan.
Like weeks are going by, so you're like, okay, so what you're telling us
is you're just making this up as you go along, that you never tested this before.
Uh, yeah, not, not good.
Um, the, um, uh, this next one here is that if your cloud vendor is
the low cost leader, it will show.
This was the case of OVH.
They were well acknowledged to be the low cost leader in that space, and it means
that they cut corners in a lot of areas.
They cut corners in terms of firefighting.
They cut corners in terms of not segregating the backup
data, all of that stuff,
Prasanna Malaiyandi: Yeah.
I don't think there's anything wrong with being a low cost offering, as
long as people understand that and the expectations are there, right?
You get what you paid for, right?
Not everyone can afford the Ferrari they may buy, right?
People go buy the Geo Metro, right?
Way back in the day, so.
There are offerings for different people, and that totally makes sense.
But I think what becomes a challenge is when you're not transparent
about what people are getting or you set expectations incorrectly and
misrepresent what they are getting.
W. Curtis Preston: Yeah, agreed.
Um, and there are, and well, and, and again, if you're using a low cost
provider, we go back to rule number one.
Make sure that you're, make sure that you're backing up the data somewhere else.
I mean, you should do it like no matter what your provider is, but definitely if
you're doing a low cost provider, right.
Um, make sure that you, you know, because they're gonna, they're
gonna, they're gonna cut corners.
Prasanna Malaiyandi: So the final one, uh, in our lessons learned is.
SSDs aren't perfect.
And I know Curtis, we've talked, I think we've had multiple episodes about this.
In fact, at home when, uh, I was looking for a new sort of, not really backup
drive, but just extra storage and debating between SSD or just getting spinning
disc, and I know chatting with you, I was like, yeah, we should probably get SS or.
Probably you should get a disc drive because we're not using the data often
and just the issues that we've seen around SSDs, if you're not using it all the time,
the data isn't guaranteed to be there.
And, uh.
This is one of those things where, uh, one of my favorite YouTube channels that
I watch all the time, life Uncontained, they lost a bunch of data on SSDs
and people think they're bulletproof.
Yes, there's less mechanical spinning parts versus traditional hard drives,
but they could still lose data.
W. Curtis Preston: Yeah, I, I think we could say that they, they probably
are more reliable than hard drives, especially if you're moving them around.
Right.
Um, you know, which is the case in your, obviously in your mobile
device or any sort of camera in the field, but they are not infallible
and there are some specific.
Um, ways that they behave that are unique to them.
Some u you know, some unique problems to SSDs.
One of them, you know, you alluded to there, is that it does require, um, you
know, a, a power to ensure that all of the voltages are there and over time.
If you're not using a given, um, SSD drive, those voltages can drop
and you can just lose data there.
There, um, I, I think we're, I think based on, we got a really good
message from one of our listeners, and I want to thank them for that.
Uh, I, I wanna do some more research and talk to somebody
who is a specialist in this area.
It's not my area of expertise.
I don't think it's yours either.
Uh, but, but the short version is.
That while SSDs and, and various other, you know, not just SSDs, but,
but, but solid state devices, uh, of every kind that, while they, I
think they are more reliable than the, uh, a standard hard drive,
especially if you're moving it around.
So they're great for laptops.
Uh, they are not infallible and they still need to be backed up.
Which brings us back to Rule zero.
And what's that
Prasanna Malaiyandi: if it's important.
Back it up.
W. Curtis Preston: exactly?
So the cloud isn't perfect.
SSDs aren't perfect.
Uh, software engineers aren't perfect.
Admins aren't perfect.
All of these are why we back up and, um,
Prasanna Malaiyandi: Don't forget malicious users are out there.
W. Curtis Preston: Of course malicious users, malicious
and, bad actors in general.
Right.
So, you know, like, like, uh, cyber attacks.
Only a few of the incidents that we covered were, were cyber attacks.
The rest were, uh, mistakes.
Um, there was a fire, trying to think what else, but generally it was just mistakes.
This is why we make backups.
Prasanna Malaiyandi: Yeah.
and and I think it's also, and I think the other important thing.
Like these are just cases we found, right?
There are probably hundreds out there that aren't publicized, right?
The company's either gone out of business or no one noticed, right, that
their data was missing or other things.
These are just things that we have been able to find and at least some information
to be able to report out to our listeners on, Hey, here's what we see going on.
W. Curtis Preston: Exactly.
Well, this has been fun.
I hope that our, um, you know, listeners have enjoyed it as much as you and
I have enjoyed making the series.
Prasanna Malaiyandi: Yeah.
Yeah.
No, this has been great, Curtis.
Thank you for suggesting this.
W. Curtis Preston: All right, well, uh, thanks to our listeners
and, uh, be sure to subscribe so that you don't miss an episode.
That is a wrap