Speaker:

Our latest in our cloud disaster series is the 2014 StorageCraft outage

Speaker:

that resulted in loss customer data.

Speaker:

A server decommissioning mistake during a cloud migration resulted in metadata

Speaker:

loss and StorageCraft scrambling to help customers re-seed their backups.

Speaker:

This may remind you a bit of what happened to carbonate, but this

Speaker:

is very different because here the CEO took full responsibility.

Speaker:

And the company I think went above and beyond to help customers

Speaker:

re-seed as quickly as possible.

Speaker:

Prasanna and I analyze what went wrong, the lessons learned for

Speaker:

backup providers and customers alike.

Speaker:

And what steps you should take if you ever find yourself.

Speaker:

With your backups lost in space.

Speaker:

We'll discuss the importance of migration processes, backup resiliency

Speaker:

measures and why trust is so critical in the backup and recovery business.

Speaker:

If you're not familiar with me, I'm w Curtis Preston and there's a reason

Speaker:

I'm so passionate about this subject.

Speaker:

During my first job as a backup admin, my company lost a huge

Speaker:

database and I couldn't restore it.

Speaker:

Since that nightmare, I've dedicated my career to making sure that

Speaker:

will never again happen to me or anyone who will listen to me.

Speaker:

We turn unappreciated backup admins into cyber recovery heroes.

Speaker:

This is the backup wrap-up.

Speaker:

W. Curtis Preston: Welcome to the show.

Speaker:

Hi, I'm w Curtis Preston, AKA, Mr.

Speaker:

Backup, and I have with me my career reverse trauma specialist

Speaker:

Prasanna Malaiyandi, how's it going?

Speaker:

Prasanna?

Prasanna Malaiyandi:

I am good, Curtis, how you been?

Prasanna Malaiyandi:

Are you doing a little better today?

Prasanna Malaiyandi:

W. Curtis Preston: So for the first time in a long time, I

Prasanna Malaiyandi:

find myself at the keyboard.

Prasanna Malaiyandi:

Um, you know, in a production environment administering a

Prasanna Malaiyandi:

very big net backup environment.

Prasanna Malaiyandi:

I haven't been the guy on the keyboard in a while, and it, it was.

Prasanna Malaiyandi:

Fun.

Prasanna Malaiyandi:

Um, and doing, you know, doing performance testing and all, you know, normally

Prasanna Malaiyandi:

it's been somebody else at the keyboard and I'm like advising, but this was

Prasanna Malaiyandi:

me in a data center all by myself.

Prasanna Malaiyandi:

Um, you

Prasanna Malaiyandi:

you back 30 years.

Prasanna Malaiyandi:

W. Curtis Preston: was interesting.

Prasanna Malaiyandi:

Uh, it's, it's not quite 30 years, but, but yeah, it's been, uh, it's

Prasanna Malaiyandi:

been a while since I've been the guy and, you know, and standing in what

Prasanna Malaiyandi:

we're going to call a data center.

Prasanna Malaiyandi:

It's really just a big storage closet that happens to have a, a server rack.

Prasanna Malaiyandi:

We're doing a, a backup of this, uh, customer's environment and, uh,

Prasanna Malaiyandi:

let's just say it's, it's not small.

Prasanna Malaiyandi:

It was, it was 400 terabytes of data sitting on a couple of little NAS boxes.

Prasanna Malaiyandi:

Um, so it's been, it has been interesting.

Prasanna Malaiyandi:

I will just say that, um.

Prasanna Malaiyandi:

Uh, standing in a very noisy server room talking to somebody on the phone, I could

Prasanna Malaiyandi:

barely hear them on my, you know, on

Prasanna Malaiyandi:

just be lucky.

Prasanna Malaiyandi:

You at least have cell phones now.

Prasanna Malaiyandi:

W. Curtis Preston: I, it's nice that I have cell phones, cell phones and, and

Prasanna Malaiyandi:

chat and screen connect so that like.

Prasanna Malaiyandi:

You know, they can see what I'm seeing.

Prasanna Malaiyandi:

And, um, you know, all of that.

Prasanna Malaiyandi:

And, um, yeah, it's been interesting.

Prasanna Malaiyandi:

But, um, speaking of interesting backup scenarios, we're continuing our series on

Prasanna Malaiyandi:

cloud disasters and we have another one.

Prasanna Malaiyandi:

This is very much in the vein of what happened to Carbonite.

Prasanna Malaiyandi:

It's very similar and yet different.

Prasanna Malaiyandi:

So,

Prasanna Malaiyandi:

Same.

Prasanna Malaiyandi:

Same, but

Prasanna Malaiyandi:

W. Curtis Preston: wanna do a quick summary of what happened to Carbonite?

Prasanna Malaiyandi:

So with Carbonite, what ended up happening was they had

Prasanna Malaiyandi:

their own storage that they were using.

Prasanna Malaiyandi:

Right.

Prasanna Malaiyandi:

Uh, and people would upload their backups to

Prasanna Malaiyandi:

it, and they had a double disc failure that took

Prasanna Malaiyandi:

down their storage array, and they lost all their data.

Prasanna Malaiyandi:

And they basically told customers, yeah, don't worry about it.

Prasanna Malaiyandi:

We're just gonna re-upload and do another backup and get all your data back up.

Prasanna Malaiyandi:

And during that time, there were something like, I think it was 35 customers, 34

Prasanna Malaiyandi:

customers I wanna say was the number, um, who ended up having another fault

Prasanna Malaiyandi:

in their production data, right.

Prasanna Malaiyandi:

On their laptop, didn't.

Prasanna Malaiyandi:

Crash or whatever else happened, and they were not able to restore their

Prasanna Malaiyandi:

data because there were no backups left.

Prasanna Malaiyandi:

W. Curtis Preston: Yeah, the, I I, I think the part of their story that

Prasanna Malaiyandi:

bothers me the most is that, you know, the company's reaction was our

Prasanna Malaiyandi:

stupid storage array vendor, right?

Prasanna Malaiyandi:

They blamed it on their storage array vendor and they sued their vendor.

Prasanna Malaiyandi:

Um, and so while this story.

Prasanna Malaiyandi:

Is gonna start out sounding kind of similar.

Prasanna Malaiyandi:

Uh, it, it's very similar in terms of the impact to the customers, but the response

Prasanna Malaiyandi:

from the company is completely different.

Prasanna Malaiyandi:

And so I feel completely different about the company, even though

Prasanna Malaiyandi:

the exact same thing happened on the, on the customer side.

Prasanna Malaiyandi:

So first off, we're talking about, storage craft, which.

Prasanna Malaiyandi:

Is a company that had been around for a while and then had been acquired in

Prasanna Malaiyandi:

2021 by Arcserve, which is a company that has been around a long while.

Prasanna Malaiyandi:

Arcserve goes all the way back to my early days.

Prasanna Malaiyandi:

Arcserve was actually the first backup product I ever tested as

Prasanna Malaiyandi:

a, to, to potentially buy it.

Prasanna Malaiyandi:

And running it on NetWare and that boy that brings me back.

Prasanna Malaiyandi:

But, uh, so Archer has been around a long time and, um, you know, it's

Prasanna Malaiyandi:

been sold and, and, and I don't know what, you know, spun back out

Prasanna Malaiyandi:

equity or

Prasanna Malaiyandi:

whatever.

Prasanna Malaiyandi:

Yeah.

Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

Prasanna Malaiyandi:

A private equity firm.

Prasanna Malaiyandi:

They got bought by ca.

Prasanna Malaiyandi:

They got spun back out with a private equity firm and then, uh, they

Prasanna Malaiyandi:

acquired or they merged, Arcserve merged with StorageCraft back in 2021

Prasanna Malaiyandi:

and then this story unfortunately happens not long after that.

Prasanna Malaiyandi:

I think it was about five months after the acquisition what ended up happening.

Prasanna Malaiyandi:

Prasanna.

Prasanna Malaiyandi:

Yeah.

Prasanna Malaiyandi:

So like you had alluded to earlier, just like Carbonite, uh,

Prasanna Malaiyandi:

they lost customer data, right?

Prasanna Malaiyandi:

Um, but the way it happened was slightly different.

Prasanna Malaiyandi:

So as part of this, they were moving from their own facilities

Prasanna Malaiyandi:

and migrating to Google Cloud.

Prasanna Malaiyandi:

Um, and they accidentally decommissioned a server before

Prasanna Malaiyandi:

it was already up in the cloud.

Prasanna Malaiyandi:

And it wasn't just any server.

Prasanna Malaiyandi:

It was a server that contained metadata about the backups.

Prasanna Malaiyandi:

W. Curtis Preston: Yeah,

Prasanna Malaiyandi:

Yeah,

Prasanna Malaiyandi:

W. Curtis Preston: metadata is important,

Prasanna Malaiyandi:

Yeah, so they basically, um, had the same situation as

Prasanna Malaiyandi:

what we had talked about with Carbonite, where the customers lost their data.

Prasanna Malaiyandi:

Now, I think unlike Carbonite, I think that, like you said, they did

Prasanna Malaiyandi:

a much better job of accepting blame.

Prasanna Malaiyandi:

And trying to get their customers or their end users back up and

Prasanna Malaiyandi:

running as quickly as possible.

Prasanna Malaiyandi:

Doing things like receding and really going the extra distance to get them

Prasanna Malaiyandi:

up and running as quickly as possible, including things like, Hey, if you

Prasanna Malaiyandi:

have more than three terabytes, we're gonna ship you a drive so you can

Prasanna Malaiyandi:

upload to the drive and then send it to help speed up the process.

Prasanna Malaiyandi:

W. Curtis Preston: Yeah, so for those not familiar with this process,

Prasanna Malaiyandi:

when you are a cloud backup vendor, the, you know, the hardest part

Prasanna Malaiyandi:

is that initial backup, right?

Prasanna Malaiyandi:

You know, it's one

Prasanna Malaiyandi:

So

Prasanna Malaiyandi:

fast.

Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

Prasanna Malaiyandi:

Yeah.

Prasanna Malaiyandi:

Damn.

Prasanna Malaiyandi:

Physics.

Prasanna Malaiyandi:

Uh, yeah.

Prasanna Malaiyandi:

So when you, you know, if, if you're back, you can back up.

Prasanna Malaiyandi:

I mean, you know, I remember, uh, you know, when I worked

Prasanna Malaiyandi:

at Druva, we had really, you know, multi petabyte customers.

Prasanna Malaiyandi:

You can back up that amount of data over the internet, but that

Prasanna Malaiyandi:

first backup is gonna be something.

Prasanna Malaiyandi:

And that's why we used the snowball edge.

Prasanna Malaiyandi:

which was a, an appliance that, that you would ship to them and, you know,

Prasanna Malaiyandi:

it already had the technology on there.

Prasanna Malaiyandi:

And then they would back up to that and then that would ship and it was

Prasanna Malaiyandi:

just mu, you know, basically a very fancy sneaker net was the way to

Prasanna Malaiyandi:

get the very first copy up there.

Prasanna Malaiyandi:

And you needed an, you needed one of those snowball edges for I think about

Prasanna Malaiyandi:

every 75 terabytes, something like that.

Prasanna Malaiyandi:

And, um, then.

Prasanna Malaiyandi:

And so that's what they're saying here.

Prasanna Malaiyandi:

And that by the way, that is referred to as the seed for those that don't

Prasanna Malaiyandi:

live in that world, that initial backup is referred to the seed.

Prasanna Malaiyandi:

So when you read the story, you're right, they did accept

Prasanna Malaiyandi:

responsibility and they said

Prasanna Malaiyandi:

the first thing that we need to get going is that seed.

Prasanna Malaiyandi:

And so, you know, you were talking about the, one of the way, just

Prasanna Malaiyandi:

like I said, the, the way, uh, my former employer would do it was they

Prasanna Malaiyandi:

would ship, uh, a snowball edge.

Prasanna Malaiyandi:

They're shipping whatever they need to ship to these customers

Prasanna Malaiyandi:

in order to make them whole.

Prasanna Malaiyandi:

One additional wrinkle, I think of this was that this was

Prasanna Malaiyandi:

specifically for their MSP customers.

Prasanna Malaiyandi:

Um, and I don't know what percentage of their business, uh, is via MSPs, but

Prasanna Malaiyandi:

that does, that does make an additional wrinkle in that not only are they

Prasanna Malaiyandi:

impacting their customer, which is the Ms.

Prasanna Malaiyandi:

P, they're impacting the end customer, which is the MSPs customer.

Prasanna Malaiyandi:

Um.

Prasanna Malaiyandi:

But again, I, you know, I want to, uh, you know, give a shout out,

Prasanna Malaiyandi:

um, to the CEO, Brandon Lacey.

Prasanna Malaiyandi:

I like that he, he doesn't try to pass the blame.

Prasanna Malaiyandi:

He doesn't try to, you know, I, I don't know.

Prasanna Malaiyandi:

I.

Prasanna Malaiyandi:

Maybe it's because he didn't have a choice, right?

Prasanna Malaiyandi:

You know, when you know the other vendor, they're saying, Hey, it was our, it

Prasanna Malaiyandi:

was our storage array, uh, that failed.

Prasanna Malaiyandi:

And that's why, you know, that's why our backups died in this case.

Prasanna Malaiyandi:

They didn't really add anything to blame it on other than, dang

Prasanna Malaiyandi:

it, why is my fingers here?

Prasanna Malaiyandi:

They didn't really have anything other to blame it on other than human failure.

Prasanna Malaiyandi:

Because it was a human that prematurely decommissioned a server before

Prasanna Malaiyandi:

that server had been migrated.

Prasanna Malaiyandi:

Yeah.

Prasanna Malaiyandi:

And that's just sort of a breakdown of processes, right?

Prasanna Malaiyandi:

It's not a technology issue.

Prasanna Malaiyandi:

Right?

Prasanna Malaiyandi:

And I'm sure now they have a process in place where they're like, Hey, make

Prasanna Malaiyandi:

sure that the data validate that the server is actually up in wherever it

Prasanna Malaiyandi:

needs to be before you decommission.

Prasanna Malaiyandi:

Because.

Prasanna Malaiyandi:

W. Curtis Preston: Yeah, that is, that is probably the

Prasanna Malaiyandi:

biggest disappointment, right?

Prasanna Malaiyandi:

Is you would think you're doing this like major migration.

Prasanna Malaiyandi:

I remember there was Datto, they did an exfiltration out of, um, Amazon and they

Prasanna Malaiyandi:

built their own private data center that was a giant, and they said they believed

Prasanna Malaiyandi:

at the time, and I believe this was Datto, they believed at the time it was

Prasanna Malaiyandi:

the single largest exfiltration project ever to leave en mass, uh, from AWS.

Prasanna Malaiyandi:

And it apparently went flawlessly.

Prasanna Malaiyandi:

You, you've got to, you know, when you're doing a migration, it's like,

Prasanna Malaiyandi:

you've gotta have that check, right?

Prasanna Malaiyandi:

You've gotta have that check.

Prasanna Malaiyandi:

We're gonna kill the thing.

Prasanna Malaiyandi:

In fact, I wouldn't even kill the thing,

Prasanna Malaiyandi:

You would just sort of, yeah.

Prasanna Malaiyandi:

W. Curtis Preston: I would, I would, uh, turn it off,

Prasanna Malaiyandi:

Yep.

Prasanna Malaiyandi:

W. Curtis Preston: right?

Prasanna Malaiyandi:

I, I would, in fact, I remember, um.

Prasanna Malaiyandi:

I remember the very first migration that I had anything to do with, and

Prasanna Malaiyandi:

I'm gonna, for those of you for the fellow gray hairs in the crowd, we were

Prasanna Malaiyandi:

migrating off, uh, an at and T three B two 4,000, which was the at t's attempt

Prasanna Malaiyandi:

at a multi-processing server, right?

Prasanna Malaiyandi:

So the three B two 1000 was the.

Prasanna Malaiyandi:

First computer ever built specifically for Unix.

Prasanna Malaiyandi:

Okay.

Prasanna Malaiyandi:

I love how you just have like these model numbers.

Prasanna Malaiyandi:

Just like right off the tongue, it just rolls right off.

Prasanna Malaiyandi:

You're like, oh yeah, everyone should know about the 3D two.

Prasanna Malaiyandi:

It's like R 2D two.

Prasanna Malaiyandi:

you might as well call it that.

Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

Prasanna Malaiyandi:

What's that?

Prasanna Malaiyandi:

You might as well call it R 2D two.

Prasanna Malaiyandi:

W. Curtis Preston: Yeah, so the three B two 1000 was the first, you know, 'cause

Prasanna Malaiyandi:

I think it was the PDP 11 was the one that, uh, that Unix initially ran on.

Prasanna Malaiyandi:

But the three B two was the first computer specifically designed to run.

Prasanna Malaiyandi:

Unix and it ran, uh, system five and, um, you know, directly from

Prasanna Malaiyandi:

Bell Labs, right at and t and the three B two 4,000 was, you know, they

Prasanna Malaiyandi:

attempted a multi-processing machine and it, it just, it wasn't very good.

Prasanna Malaiyandi:

And so we were trying to migrate off of that, and I remember that.

Prasanna Malaiyandi:

Basically what we would do is we would just kill things as we

Prasanna Malaiyandi:

believed we had migrated it off.

Prasanna Malaiyandi:

We would, we would turn off a database.

Prasanna Malaiyandi:

We would turn off a server.

Prasanna Malaiyandi:

see what broke.

Prasanna Malaiyandi:

W. Curtis Preston: That's the, and well, yeah, and then see if anything broke.

Prasanna Malaiyandi:

Right.

Prasanna Malaiyandi:

You don't, I, I don't know.

Prasanna Malaiyandi:

I, I guess that's the one surprise here, is that they, they decommission

Prasanna Malaiyandi:

the server to the point of actually, you know, deleting it somehow.

Prasanna Malaiyandi:

Or, or, I, I know they talk about decommissioning.

Prasanna Malaiyandi:

I do wonder if it's sort of, um, because it's metadata, I wonder if it's like

Prasanna Malaiyandi:

a distributed system and maybe they killed the node, uh, without, and so

Prasanna Malaiyandi:

since it didn't have access to the metadata, it's sort of like maybe

Prasanna Malaiyandi:

they weren't expecting that to happen.

Prasanna Malaiyandi:

Right?

Prasanna Malaiyandi:

So even though they say

Prasanna Malaiyandi:

W. Curtis Preston: Yeah, I, yeah.

Prasanna Malaiyandi:

I don't know the details.

Prasanna Malaiyandi:

So.

Prasanna Malaiyandi:

W. Curtis Preston: Yeah, we don't know the details.

Prasanna Malaiyandi:

We're Monday morning quarterbacking, et cetera.

Prasanna Malaiyandi:

Um, but let's, but let's just talk about it.

Prasanna Malaiyandi:

When you are doing a migration, you don't kill the thing.

Prasanna Malaiyandi:

You don't, you don't kill the old one.

Prasanna Malaiyandi:

You turn off the old thing, right?

Prasanna Malaiyandi:

You turn it off and then you wait a while before you, uh, actually

Prasanna Malaiyandi:

redeploy it, delete it, whatever.

Prasanna Malaiyandi:

And then I would.

Prasanna Malaiyandi:

Then I, you know, I, I find myself wondering, well, I, I guess there wasn't,

Prasanna Malaiyandi:

there wasn't a backup of that thing.

Prasanna Malaiyandi:

There wasn't a backup of the backup, which is not that uncommon

Prasanna Malaiyandi:

Yep.

Prasanna Malaiyandi:

Backup

Prasanna Malaiyandi:

W. Curtis Preston: uh, backup.

Prasanna Malaiyandi:

Backup.

Prasanna Malaiyandi:

I.

Prasanna Malaiyandi:

Services.

Prasanna Malaiyandi:

Right?

Prasanna Malaiyandi:

Um, and as a result, when that system was deleted, it, there was no backup

Prasanna Malaiyandi:

of that backup, which I think many people might be surprised by that.

Prasanna Malaiyandi:

What do you think?

Prasanna Malaiyandi:

No.

Prasanna Malaiyandi:

Uh, I think people will be surprised because they think that backup

Prasanna Malaiyandi:

systems are highly resilient.

Prasanna Malaiyandi:

And of course someone's gonna do a backup of a backup, but you have to also realize

Prasanna Malaiyandi:

that there's additional costs associated with doing a backup of a backup.

Prasanna Malaiyandi:

And.

Prasanna Malaiyandi:

Usually backup systems, people are trying to cut costs, right?

Prasanna Malaiyandi:

Because this is like the tertiary copy.

Prasanna Malaiyandi:

And so having high availability or a replica copy of your backup system, unless

Prasanna Malaiyandi:

it's critical, which a lot of systems are, and people do replicate their backup

Prasanna Malaiyandi:

systems, but not everyone could afford to do that, and especially given that this

Prasanna Malaiyandi:

was for their MSP customers who are also trying to make money and add value, right?

Prasanna Malaiyandi:

They're looking at probably a.

Prasanna Malaiyandi:

Reasonable offering, right,

Prasanna Malaiyandi:

uh, without all the bells and whistles.

Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

Prasanna Malaiyandi:

And in a world where services are often purchased on price.

Prasanna Malaiyandi:

Not necessarily alone, but where price is the primary motivator For many

Prasanna Malaiyandi:

customers, it's, this is one of those things where no one ever bought Salesforce

Prasanna Malaiyandi:

because they had a really good backup.

Prasanna Malaiyandi:

No one ever bought Oracle because it had a really good backup.

Prasanna Malaiyandi:

They bought Oracle because it had really good features and did

Prasanna Malaiyandi:

the thing they needed it to do.

Prasanna Malaiyandi:

Nobody's nobody, there's, there's nowhere, no ad in the world.

Prasanna Malaiyandi:

Said, buy Oracle, because our backup is really good.

Prasanna Malaiyandi:

Right?

Prasanna Malaiyandi:

And so the same thing is true here.

Prasanna Malaiyandi:

Buy our backup service because we also have a tertiary copy of

Prasanna Malaiyandi:

your data just in case we screw up the copy that you gave us.

Prasanna Malaiyandi:

Nobody's gonna even know that.

Prasanna Malaiyandi:

Um, and they're, and they're cer, they certainly don't want to pay for it.

Prasanna Malaiyandi:

All they see is that this one is more expensive than that one.

Prasanna Malaiyandi:

Yeah.

Prasanna Malaiyandi:

The one thing I am a little surprised about though is given this is a backup

Prasanna Malaiyandi:

system, and it looks like they did have their metadata and data split apart just

Prasanna Malaiyandi:

based on what we read in the articles, I'm surprised they didn't have a self

Prasanna Malaiyandi:

describing format for the data itself.

Prasanna Malaiyandi:

I know?

Prasanna Malaiyandi:

there are other systems which tend to do that as sort of a fail safe, so I'm not

Prasanna Malaiyandi:

necessarily going to create a copy of the data, but if I, my, something happens,

Prasanna Malaiyandi:

my metadata or there's some corruption or whatever else, I should at least be

Prasanna Malaiyandi:

able to build back all the metadata based on the data that I still have access to.

Prasanna Malaiyandi:

I.

Prasanna Malaiyandi:

W. Curtis Preston: Yeah, that's a good point.

Prasanna Malaiyandi:

And, and that is a, another really good feature to have in a backup system

Prasanna Malaiyandi:

that if the, basically the backup catalog, the image database, the, you

Prasanna Malaiyandi:

know, the database of all the backups, if that gets messed up, it would be

Prasanna Malaiyandi:

really nice if the, if the, uh, you know, if you could rebuild that.

Prasanna Malaiyandi:

And I remember back when I was doing RFPs and the, one of the questions

Prasanna Malaiyandi:

we asked was if the backup catalog, um, you know, gets corrupt, can,

Prasanna Malaiyandi:

can we rebuild it given the backups?

Prasanna Malaiyandi:

And I do remember at least one major vendor.

Prasanna Malaiyandi:

I won't call 'em out by name 'cause it might not still be the

Prasanna Malaiyandi:

answer, but their answer was.

Prasanna Malaiyandi:

Um, you know what, they, literally, the phrase was, our backup format

Prasanna Malaiyandi:

is so proprietary that even we can't read it if we don't know what's on it.

Prasanna Malaiyandi:

And they, and they saw that as like a, like a feature.

Prasanna Malaiyandi:

Uh, by the way, it turned out, interestingly enough, it turned out

Prasanna Malaiyandi:

that, uh, it wasn't that proprietary and.

Prasanna Malaiyandi:

Other vendors have figured out how to read that backup date.

Prasanna Malaiyandi:

Yeah.

Prasanna Malaiyandi:

As long as it's not, um, encrypted.

Prasanna Malaiyandi:

Right.

Prasanna Malaiyandi:

That that's a whole other, whole other thing.

Prasanna Malaiyandi:

So I don't think we've ever talked about

Prasanna Malaiyandi:

this on the podcast before.

Prasanna Malaiyandi:

And so is this something, I know we've talked previously about, Hey, here are

Prasanna Malaiyandi:

some questions you should be asking your backup vendor when trying to figure

Prasanna Malaiyandi:

out like what their solution does.

Prasanna Malaiyandi:

Do you think this is worthwhile to throw into that list of questions?

Prasanna Malaiyandi:

W. Curtis Preston: Of, do you have a backup of the backup?

Prasanna Malaiyandi:

Either a backup of the backup or if I lose

Prasanna Malaiyandi:

part of my system, like my catalog, can I still recover my data?

Prasanna Malaiyandi:

W. Curtis Preston: I, I do think that's a perfectly valid question to ask.

Prasanna Malaiyandi:

Right.

Prasanna Malaiyandi:

And I think the other question is perfectly valid as well of

Prasanna Malaiyandi:

if, what protections do you, I would point at this story, what

Prasanna Malaiyandi:

protections do you have against this happening in your environment?

Prasanna Malaiyandi:

Right.

Prasanna Malaiyandi:

Um,

Prasanna Malaiyandi:

because it may not be malicious.

Prasanna Malaiyandi:

It's just accidental things happen.

Prasanna Malaiyandi:

It's all software.

Prasanna Malaiyandi:

People are writing software, right?

Prasanna Malaiyandi:

W. Curtis Preston: Yeah, the, you know, so this, remind this, this story to

Prasanna Malaiyandi:

me is a, is a combination of the KPMG story and the Carbonite story, right?

Prasanna Malaiyandi:

Because like the KPMG story, and by the way, we have another episode on that.

Prasanna Malaiyandi:

Please feel free to check that out.

Prasanna Malaiyandi:

The KPMG story is one of failure of process as well, where.

Prasanna Malaiyandi:

You'd like to think that at a, at a company that size and when you're

Prasanna Malaiyandi:

doing something of this magnitude, I, you know, I, when I think back

Prasanna Malaiyandi:

on that story, I don't think they.

Prasanna Malaiyandi:

Thought this was of the, the, we're, I'm just trying to fix one

Prasanna Malaiyandi:

person's, one person's account.

Prasanna Malaiyandi:

They didn't think that what they were touching was a tool in this

Prasanna Malaiyandi:

case, uh, retention policies.

Prasanna Malaiyandi:

They were touching a tool that could wipe out everybody's data.

Prasanna Malaiyandi:

They, they

Prasanna Malaiyandi:

Prasanna Malaiyandi: Hundred 40,000 people.

Prasanna Malaiyandi:

W. Curtis Preston: 140,000 people's data with no backup.

Prasanna Malaiyandi:

Uh, that's a great, that's a great story.

Prasanna Malaiyandi:

Check that one out.

Prasanna Malaiyandi:

But I, the, the big thing with processes, you've, you've got to have

Prasanna Malaiyandi:

a very, uh, you know, the bigger the project, the, the bigger the process.

Prasanna Malaiyandi:

And I do remember, I.

Prasanna Malaiyandi:

When the, the biggest one I remember being involved in it was, it, was, it, it wasn't

Prasanna Malaiyandi:

a migration, it was a physical move.

Prasanna Malaiyandi:

We moved, a bunch of servers from one data center to another data center,

Prasanna Malaiyandi:

We had built a nice new big data center and we're gonna move the servers from the

Prasanna Malaiyandi:

old data center to the new data center.

Prasanna Malaiyandi:

And we had a massive process and we had tested it from one end to the other,

Prasanna Malaiyandi:

and we outsourced whatever we could.

Prasanna Malaiyandi:

So like we hired.

Prasanna Malaiyandi:

A professional moving company to move the servers we hired, uh, and paid extra

Prasanna Malaiyandi:

to have HP come in and, and, you know, unplug the servers and go ominous dominus,

Prasanna Malaiyandi:

you know, you may touch this server.

Prasanna Malaiyandi:

Right.

Prasanna Malaiyandi:

Um, and then, and then it got handed to the movers and stuff

Prasanna Malaiyandi:

and we had this big process.

Prasanna Malaiyandi:

Um, but there was one major failing and that was from the old

Prasanna Malaiyandi:

server to the new server room.

Prasanna Malaiyandi:

The new server room had, um, Twistlock.

Prasanna Malaiyandi:

You know, plugs under, underneath the floor.

Prasanna Malaiyandi:

And the old server room had the old standard, uh, north

Prasanna Malaiyandi:

American, uh, two-prong plugs.

Prasanna Malaiyandi:

And, uh, that was the, that was one day where I truly saved the day because

Prasanna Malaiyandi:

I had, uh, oh, and the electricians, that would be the ones to help us.

Prasanna Malaiyandi:

Were busy at the CEO's house, putting up his Christmas lights.

Prasanna Malaiyandi:

Um, and so, and so they didn't think we'd need electricians during our big move.

Prasanna Malaiyandi:

And I knew how to fix this problem, right?

Prasanna Malaiyandi:

I, I'm like, look, we can do this.

Prasanna Malaiyandi:

I just need to go to a gray bar, uh, electrical store.

Prasanna Malaiyandi:

And there's one down the road and it was closing in.

Prasanna Malaiyandi:

20 minutes, uh, by the time we realized what we needed to do, and I, I got in my

Prasanna Malaiyandi:

car with, with my boss and, and he looked at me and he says, I will pay the ticket.

Prasanna Malaiyandi:

Just get us there.

Prasanna Malaiyandi:

And we drove.

Prasanna Malaiyandi:

I drove and it was raining.

Prasanna Malaiyandi:

It was, I mean, there was water everywhere.

Prasanna Malaiyandi:

I'm driving to this gray bar and I've got it.

Prasanna Malaiyandi:

It's, it's, it's closing in 20 minutes and it's 15 minutes away.

Prasanna Malaiyandi:

Right.

Prasanna Malaiyandi:

I don't have time to play.

Prasanna Malaiyandi:

And I remember hydroplaning going around a big semi, and I remember

Prasanna Malaiyandi:

like, you know, swerving around a corner and all this stuff.

Prasanna Malaiyandi:

And I remember that when we got there, there's that scene in, uh, planes, trains,

Prasanna Malaiyandi:

and Automobiles where, where Steve, Steve Martin's fingers are stuck in the dash.

Prasanna Malaiyandi:

Uh, that was the way, uh, Lou was when we, when we got there.

Prasanna Malaiyandi:

But we got, we walked up to the door and it was five minutes prior to the,

Prasanna Malaiyandi:

you know, prior to the, the closing.

Prasanna Malaiyandi:

They were, the door was locked.

Prasanna Malaiyandi:

And, uh, and, and they, they pointed, they pointed at her watch.

Prasanna Malaiyandi:

They're like, they're pointing at their watch as a time.

Prasanna Malaiyandi:

And I pointed up at the clock on their wall and I was like,

Prasanna Malaiyandi:

no, it's not one o'clock yet.

Prasanna Malaiyandi:

Open your damn door.

Prasanna Malaiyandi:

And then we, you know, uh, and then I think, I think I like, I was like, please,

Prasanna Malaiyandi:

please, you know, please let me in.

Prasanna Malaiyandi:

And we got it.

Prasanna Malaiyandi:

And we bought all the stuff.

Prasanna Malaiyandi:

And I, I non.

Prasanna Malaiyandi:

Traded non whatever electrician, but I was trained as one in the Navy, um, which at

Prasanna Malaiyandi:

that time was only like five years prior.

Prasanna Malaiyandi:

Um, and then we wired adapters from.

Prasanna Malaiyandi:

A to B and save the day anyway.

Prasanna Malaiyandi:

Yeah, you gotta test your process and you gotta be ready

Prasanna Malaiyandi:

for failures in your process.

Prasanna Malaiyandi:

But I, I think that, you know, so the two big this, this one I feel very different.

Prasanna Malaiyandi:

Like I said in the beginning, I feel very different about

Prasanna Malaiyandi:

this one versus the carbonite

Prasanna Malaiyandi:

but why

Prasanna Malaiyandi:

W. Curtis Preston: because they didn't try to, they didn't try to pass the buck.

Prasanna Malaiyandi:

is that really the only difference in your mind is,

Prasanna Malaiyandi:

well, I think also in the Carbonite case, the only reason we heard about

Prasanna Malaiyandi:

it is when they try to sue the vendor.

Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

Prasanna Malaiyandi:

Um, this.

Prasanna Malaiyandi:

Yeah, that's a good question.

Prasanna Malaiyandi:

If, would we have ever heard about the story if they, I, I don't know.

Prasanna Malaiyandi:

I, I, I'd like to think that it would still have made news, but all we

Prasanna Malaiyandi:

heard about was the fact that they store, they sued their storage vendor

Prasanna Malaiyandi:

in this case.

Prasanna Malaiyandi:

The article, right.

Prasanna Malaiyandi:

Um.

Prasanna Malaiyandi:

It starts out with this quote from the CEO, right?

Prasanna Malaiyandi:

In terms of magnitude is the only thing I'm thinking about waking

Prasanna Malaiyandi:

up and going to bed every night.

Prasanna Malaiyandi:

It's orders of magnitude more important than anything else that I have going

Prasanna Malaiyandi:

on in terms of operating this business.

Prasanna Malaiyandi:

Um, says the CEO right, uh, of this company trying to bounce back.

Prasanna Malaiyandi:

Now we haven't, I tried to find, I didn't find, this was almost two years ago now.

Prasanna Malaiyandi:

I didn't see.

Prasanna Malaiyandi:

An update on what happened.

Prasanna Malaiyandi:

They were saying, we're trying to get people up and running.

Prasanna Malaiyandi:

I didn't see any stories like we did with the Carbonite about some

Prasanna Malaiyandi:

customers that actually lost data because they had to restore something

Prasanna Malaiyandi:

while they were trying to be seated.

Prasanna Malaiyandi:

Uh, we don't have any of those

Prasanna Malaiyandi:

Now, I know you're a big Reddit fan, and I know you

Prasanna Malaiyandi:

probably were browsing threads about this.

Prasanna Malaiyandi:

Did you see any, like, what was the sentiment in those threads

Prasanna Malaiyandi:

from end users, customers that, if

Prasanna Malaiyandi:

you can

Prasanna Malaiyandi:

W. Curtis Preston: that's a, actually, that's a good one.

Prasanna Malaiyandi:

Let me, uh, let me pull up that thread.

Prasanna Malaiyandi:

'cause I didn't, I didn't look too much at the thread.

Prasanna Malaiyandi:

This is funny.

Prasanna Malaiyandi:

Seems like a backup company that loses backups is no longer a backup

Prasanna Malaiyandi:

company, but a data disposal service.

Prasanna Malaiyandi:

I see a pivot opportunity.

Prasanna Malaiyandi:

They should certify these backups as destroyed and issued certificates.

Prasanna Malaiyandi:

I'd say the Reddit response is typical of every response with a cloud outage.

Prasanna Malaiyandi:

There are people that are just, you know, saying, ah, the cloud sucks, and,

Prasanna Malaiyandi:

you know, and the cloud is just, it's just another computer, it's just another

Prasanna Malaiyandi:

service, and you have to verify it just like you verify every other system.

Prasanna Malaiyandi:

Right.

Prasanna Malaiyandi:

Um, and there, there were some comments in the thread saying that.

Prasanna Malaiyandi:

You know, one of the reasons that Arc Serve and StorageCraft, uh, merged was

Prasanna Malaiyandi:

that they had been going down for a while in terms of quality of service.

Prasanna Malaiyandi:

Again, these are pe other people's comments, um, and that they, that they

Prasanna Malaiyandi:

weren't that surprised at, at this event.

Prasanna Malaiyandi:

But, um, they're still around.

Prasanna Malaiyandi:

They're still running.

Prasanna Malaiyandi:

Um, just like Carbonite is still around and still running.

Prasanna Malaiyandi:

I, I, I, you know, I wish them the best.

Prasanna Malaiyandi:

Uh, this was a major migration fail.

Prasanna Malaiyandi:

Yeah, and it's also one of those things

Prasanna Malaiyandi:

people have, like once you lose that trust, it get, it's difficult,

Prasanna Malaiyandi:

especially in the space where you are that last line of defense.

Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

Prasanna Malaiyandi:

I mean, people are still on LastPass.

Prasanna Malaiyandi:

Prasanna Malaiyandi: Yeah, let's not go there.

Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

Prasanna Malaiyandi:

What's that?

Prasanna Malaiyandi:

not go there.

Prasanna Malaiyandi:

W. Curtis Preston: Yeah, yeah.

Prasanna Malaiyandi:

Anyway.

Prasanna Malaiyandi:

All right.

Prasanna Malaiyandi:

Well, um, not, not a ton, not some huge lessons here, but the big one

Prasanna Malaiyandi:

I think here is, you know, I, I like the Maya Culpa that the company did.

Prasanna Malaiyandi:

I.

Prasanna Malaiyandi:

Uh, and I like that they knew that their priority was getting the,

Prasanna Malaiyandi:

the next backup, but unfortunately that's gonna take a long time.

Prasanna Malaiyandi:

Um, and the, the biggest lesson here is if you're doing a migration of any

Prasanna Malaiyandi:

kind, don't go shooting the other servers until you know that it's fully migrated.

Prasanna Malaiyandi:

You know?

Prasanna Malaiyandi:

what, what?

Prasanna Malaiyandi:

Is there any lessons learned for the end users?

Prasanna Malaiyandi:

Like there's a, like what could they have done?

Prasanna Malaiyandi:

I don't think there's anything that they could have done in this situation.

Prasanna Malaiyandi:

W. Curtis Preston: yeah, the only, the only thing I, in terms of.

Prasanna Malaiyandi:

Trying to identify, I mean, what's a vendor gonna say?

Prasanna Malaiyandi:

This is the difficulty with the cloud.

Prasanna Malaiyandi:

We've discussed this in previous other things, is it's hard to actually verify

Prasanna Malaiyandi:

if you say, do you have processes in place for if somebody, if somebody over

Prasanna Malaiyandi:

there accidentally, you know, blows up a server, do you have processes in

Prasanna Malaiyandi:

place to be able to bring that server back or re of course they're gonna say,

Prasanna Malaiyandi:

yeah, that's what they're gonna say.

Prasanna Malaiyandi:

Um, the, the, the one thing you could think about, again, this is

Prasanna Malaiyandi:

expensive, but people talk about using a service that puts stuff in more

Prasanna Malaiyandi:

than one place or using more than one service, but people don't wanna pay

Prasanna Malaiyandi:

for one backup service, let alone two.

Prasanna Malaiyandi:

Could they do that?

Prasanna Malaiyandi:

Should they do that?

Prasanna Malaiyandi:

Sure.

Prasanna Malaiyandi:

Will they do that?

Prasanna Malaiyandi:

Nobody does that,

Prasanna Malaiyandi:

Yeah.

Prasanna Malaiyandi:

No one's gonna spend the money in their budget on that.

Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

Prasanna Malaiyandi:

Um.

Prasanna Malaiyandi:

I mean, it's hard enough to get people to do backups in the first place.

Prasanna Malaiyandi:

It's hard enough to get them to, to properly secure those backups.

Prasanna Malaiyandi:

Um, they're not gonna pay for a completely separate, just in case they're, you know,

Prasanna Malaiyandi:

uh, I will say having something like I.

Prasanna Malaiyandi:

StorageCraft or carbonite, die and lose all your data is not

Prasanna Malaiyandi:

the same as losing all your data.

Prasanna Malaiyandi:

Right.

Prasanna Malaiyandi:

It's, it's, it's all the backups of all your data.

Prasanna Malaiyandi:

And there will be some, could be some ramifications to your business if you

Prasanna Malaiyandi:

need to restore, but statistically speaking, you're not probably

Prasanna Malaiyandi:

going to need to restore today.

Prasanna Malaiyandi:

Um, just in terms of statistically Right.

Prasanna Malaiyandi:

Um, it's not the same.

Prasanna Malaiyandi:

So, you know,

Prasanna Malaiyandi:

So, okay, so if I was an end user, I get a

Prasanna Malaiyandi:

notification from StorageCraft that says, Hey, by the way, we had an issue.

Prasanna Malaiyandi:

We lost all your backups.

Prasanna Malaiyandi:

They're gonna say, okay, I'm gonna try to help reseed.

Prasanna Malaiyandi:

Right.

Prasanna Malaiyandi:

Get all your data up there before that happens.

Prasanna Malaiyandi:

Because like you said, it could take weeks, it could take months.

Prasanna Malaiyandi:

Right.

Prasanna Malaiyandi:

What is the first thing that the end user should do at that point?

Prasanna Malaiyandi:

Right?

Prasanna Malaiyandi:

Once they get that first notification to make sure that they don't end up

Prasanna Malaiyandi:

losing all their data in case something happens to their production copy.

Prasanna Malaiyandi:

W. Curtis Preston: Well, I, I'll say this, you got any big projects,

Prasanna Malaiyandi:

uh, in process, maybe you should pause those projects, right?

Prasanna Malaiyandi:

Any big changes, any big maintenance windows.

Prasanna Malaiyandi:

Maybe you should put a pause on those maintenance windows.

Prasanna Malaiyandi:

Uh, the other is you should go back to old school backups for the

Prasanna Malaiyandi:

time being, for anything critical.

Prasanna Malaiyandi:

Uh, you know, you should make a local backup of any kind that, a backup of any

Prasanna Malaiyandi:

kind is better than a backup of no kind.

Prasanna Malaiyandi:

Uh, that's a really good question, right.

Prasanna Malaiyandi:

Um, I would put a pause on projects and I would see what I can do to get

Prasanna Malaiyandi:

some local backups of stuff, right?

Prasanna Malaiyandi:

There are plenty of open source tools out there.

Prasanna Malaiyandi:

That can, you know, surely somewhere you've got some disc and then you've

Prasanna Malaiyandi:

got some super critical servers that you really don't wanna lose.

Prasanna Malaiyandi:

Get some local backups of those.

Prasanna Malaiyandi:

Um, you know.

Prasanna Malaiyandi:

or I know you had mentioned in the past, right?

Prasanna Malaiyandi:

You really like the iDrive service, cloud service,

Prasanna Malaiyandi:

W. Curtis Preston: Right.

Prasanna Malaiyandi:

right.

Prasanna Malaiyandi:

For just doing

Prasanna Malaiyandi:

W. Curtis Preston: Well, the.

Prasanna Malaiyandi:

backup

Prasanna Malaiyandi:

W. Curtis Preston: Yeah, the, the only problem with that is, you

Prasanna Malaiyandi:

know, again, it's physics, right?

Prasanna Malaiyandi:

So I'm just trying to, I'm just trying to think of something

Prasanna Malaiyandi:

that you can do quickly.

Prasanna Malaiyandi:

Um, you know

Prasanna Malaiyandi:

Copy it to a different machine.

Prasanna Malaiyandi:

Go buy a couple external hard

Prasanna Malaiyandi:

W. Curtis Preston: Yeah.

Prasanna Malaiyandi:

Specifically for those super high value, super highly critical

Prasanna Malaiyandi:

servers, you know, uh, things like backup servers for other customers.

Prasanna Malaiyandi:

Uh uh.

Prasanna Malaiyandi:

Yeah.

Prasanna Malaiyandi:

Okay.

Prasanna Malaiyandi:

Well, all right, well, thanks for helping me walk through that.

Prasanna Malaiyandi:

That was a really good question there at the end.

Prasanna Malaiyandi:

Prasanna, thanks for helping me walk through this.

Prasanna Malaiyandi:

No, no worries.

Prasanna Malaiyandi:

That's why uh, I enjoy doing these podcasts 'cause it helps all of us, right?

Prasanna Malaiyandi:

Just figure out what should we be

Prasanna Malaiyandi:

doing in case this ever happens.

Prasanna Malaiyandi:

W. Curtis Preston: That's what we're trying to do is to pass on these

Prasanna Malaiyandi:

lessons to you, uh, the listener so that you can learn from other

Prasanna Malaiyandi:

people's struggles, unfortunately.

Prasanna Malaiyandi:

But, uh, you're why we do this.

Prasanna Malaiyandi:

We want to turn you into a cyber recovery hero.

Prasanna Malaiyandi:

That's a wrap

Prasanna Malaiyandi:

The backup wrap up is written, recorded and produced by me w Curtis Preston.

Prasanna Malaiyandi:

If you need backup or Dr.

Prasanna Malaiyandi:

Consulting content generation or expert witness work,

Prasanna Malaiyandi:

check out backup central.com.

Prasanna Malaiyandi:

You can also find links from my O'Reilly Books on the same website.

Prasanna Malaiyandi:

Remember, this is an independent podcast and any opinions that you

Prasanna Malaiyandi:

hear are those of the speaker.

Prasanna Malaiyandi:

And not necessarily an employer.

Prasanna Malaiyandi:

Thanks for listening.