Our latest in our cloud disaster series is the 2014 StorageCraft outage
Speaker:that resulted in loss customer data.
Speaker:A server decommissioning mistake during a cloud migration resulted in metadata
Speaker:loss and StorageCraft scrambling to help customers re-seed their backups.
Speaker:This may remind you a bit of what happened to carbonate, but this
Speaker:is very different because here the CEO took full responsibility.
Speaker:And the company I think went above and beyond to help customers
Speaker:re-seed as quickly as possible.
Speaker:Prasanna and I analyze what went wrong, the lessons learned for
Speaker:backup providers and customers alike.
Speaker:And what steps you should take if you ever find yourself.
Speaker:With your backups lost in space.
Speaker:We'll discuss the importance of migration processes, backup resiliency
Speaker:measures and why trust is so critical in the backup and recovery business.
Speaker:If you're not familiar with me, I'm w Curtis Preston and there's a reason
Speaker:I'm so passionate about this subject.
Speaker:During my first job as a backup admin, my company lost a huge
Speaker:database and I couldn't restore it.
Speaker:Since that nightmare, I've dedicated my career to making sure that
Speaker:will never again happen to me or anyone who will listen to me.
Speaker:We turn unappreciated backup admins into cyber recovery heroes.
Speaker:This is the backup wrap-up.
Speaker:W. Curtis Preston: Welcome to the show.
Speaker:Hi, I'm w Curtis Preston, AKA, Mr.
Speaker:Backup, and I have with me my career reverse trauma specialist
Speaker:Prasanna Malaiyandi, how's it going?
Speaker:Prasanna?
Prasanna Malaiyandi:I am good, Curtis, how you been?
Prasanna Malaiyandi:Are you doing a little better today?
Prasanna Malaiyandi:W. Curtis Preston: So for the first time in a long time, I
Prasanna Malaiyandi:find myself at the keyboard.
Prasanna Malaiyandi:Um, you know, in a production environment administering a
Prasanna Malaiyandi:very big net backup environment.
Prasanna Malaiyandi:I haven't been the guy on the keyboard in a while, and it, it was.
Prasanna Malaiyandi:Fun.
Prasanna Malaiyandi:Um, and doing, you know, doing performance testing and all, you know, normally
Prasanna Malaiyandi:it's been somebody else at the keyboard and I'm like advising, but this was
Prasanna Malaiyandi:me in a data center all by myself.
Prasanna Malaiyandi:Um, you
Prasanna Malaiyandi:you back 30 years.
Prasanna Malaiyandi:W. Curtis Preston: was interesting.
Prasanna Malaiyandi:Uh, it's, it's not quite 30 years, but, but yeah, it's been, uh, it's
Prasanna Malaiyandi:been a while since I've been the guy and, you know, and standing in what
Prasanna Malaiyandi:we're going to call a data center.
Prasanna Malaiyandi:It's really just a big storage closet that happens to have a, a server rack.
Prasanna Malaiyandi:We're doing a, a backup of this, uh, customer's environment and, uh,
Prasanna Malaiyandi:let's just say it's, it's not small.
Prasanna Malaiyandi:It was, it was 400 terabytes of data sitting on a couple of little NAS boxes.
Prasanna Malaiyandi:Um, so it's been, it has been interesting.
Prasanna Malaiyandi:I will just say that, um.
Prasanna Malaiyandi:Uh, standing in a very noisy server room talking to somebody on the phone, I could
Prasanna Malaiyandi:barely hear them on my, you know, on
Prasanna Malaiyandi:just be lucky.
Prasanna Malaiyandi:You at least have cell phones now.
Prasanna Malaiyandi:W. Curtis Preston: I, it's nice that I have cell phones, cell phones and, and
Prasanna Malaiyandi:chat and screen connect so that like.
Prasanna Malaiyandi:You know, they can see what I'm seeing.
Prasanna Malaiyandi:And, um, you know, all of that.
Prasanna Malaiyandi:And, um, yeah, it's been interesting.
Prasanna Malaiyandi:But, um, speaking of interesting backup scenarios, we're continuing our series on
Prasanna Malaiyandi:cloud disasters and we have another one.
Prasanna Malaiyandi:This is very much in the vein of what happened to Carbonite.
Prasanna Malaiyandi:It's very similar and yet different.
Prasanna Malaiyandi:So,
Prasanna Malaiyandi:Same.
Prasanna Malaiyandi:Same, but
Prasanna Malaiyandi:W. Curtis Preston: wanna do a quick summary of what happened to Carbonite?
Prasanna Malaiyandi:So with Carbonite, what ended up happening was they had
Prasanna Malaiyandi:their own storage that they were using.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Uh, and people would upload their backups to
Prasanna Malaiyandi:it, and they had a double disc failure that took
Prasanna Malaiyandi:down their storage array, and they lost all their data.
Prasanna Malaiyandi:And they basically told customers, yeah, don't worry about it.
Prasanna Malaiyandi:We're just gonna re-upload and do another backup and get all your data back up.
Prasanna Malaiyandi:And during that time, there were something like, I think it was 35 customers, 34
Prasanna Malaiyandi:customers I wanna say was the number, um, who ended up having another fault
Prasanna Malaiyandi:in their production data, right.
Prasanna Malaiyandi:On their laptop, didn't.
Prasanna Malaiyandi:Crash or whatever else happened, and they were not able to restore their
Prasanna Malaiyandi:data because there were no backups left.
Prasanna Malaiyandi:W. Curtis Preston: Yeah, the, I I, I think the part of their story that
Prasanna Malaiyandi:bothers me the most is that, you know, the company's reaction was our
Prasanna Malaiyandi:stupid storage array vendor, right?
Prasanna Malaiyandi:They blamed it on their storage array vendor and they sued their vendor.
Prasanna Malaiyandi:Um, and so while this story.
Prasanna Malaiyandi:Is gonna start out sounding kind of similar.
Prasanna Malaiyandi:Uh, it, it's very similar in terms of the impact to the customers, but the response
Prasanna Malaiyandi:from the company is completely different.
Prasanna Malaiyandi:And so I feel completely different about the company, even though
Prasanna Malaiyandi:the exact same thing happened on the, on the customer side.
Prasanna Malaiyandi:So first off, we're talking about, storage craft, which.
Prasanna Malaiyandi:Is a company that had been around for a while and then had been acquired in
Prasanna Malaiyandi:2021 by Arcserve, which is a company that has been around a long while.
Prasanna Malaiyandi:Arcserve goes all the way back to my early days.
Prasanna Malaiyandi:Arcserve was actually the first backup product I ever tested as
Prasanna Malaiyandi:a, to, to potentially buy it.
Prasanna Malaiyandi:And running it on NetWare and that boy that brings me back.
Prasanna Malaiyandi:But, uh, so Archer has been around a long time and, um, you know, it's
Prasanna Malaiyandi:been sold and, and, and I don't know what, you know, spun back out
Prasanna Malaiyandi:equity or
Prasanna Malaiyandi:whatever.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:A private equity firm.
Prasanna Malaiyandi:They got bought by ca.
Prasanna Malaiyandi:They got spun back out with a private equity firm and then, uh, they
Prasanna Malaiyandi:acquired or they merged, Arcserve merged with StorageCraft back in 2021
Prasanna Malaiyandi:and then this story unfortunately happens not long after that.
Prasanna Malaiyandi:I think it was about five months after the acquisition what ended up happening.
Prasanna Malaiyandi:Prasanna.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:So like you had alluded to earlier, just like Carbonite, uh,
Prasanna Malaiyandi:they lost customer data, right?
Prasanna Malaiyandi:Um, but the way it happened was slightly different.
Prasanna Malaiyandi:So as part of this, they were moving from their own facilities
Prasanna Malaiyandi:and migrating to Google Cloud.
Prasanna Malaiyandi:Um, and they accidentally decommissioned a server before
Prasanna Malaiyandi:it was already up in the cloud.
Prasanna Malaiyandi:And it wasn't just any server.
Prasanna Malaiyandi:It was a server that contained metadata about the backups.
Prasanna Malaiyandi:W. Curtis Preston: Yeah,
Prasanna Malaiyandi:Yeah,
Prasanna Malaiyandi:W. Curtis Preston: metadata is important,
Prasanna Malaiyandi:Yeah, so they basically, um, had the same situation as
Prasanna Malaiyandi:what we had talked about with Carbonite, where the customers lost their data.
Prasanna Malaiyandi:Now, I think unlike Carbonite, I think that, like you said, they did
Prasanna Malaiyandi:a much better job of accepting blame.
Prasanna Malaiyandi:And trying to get their customers or their end users back up and
Prasanna Malaiyandi:running as quickly as possible.
Prasanna Malaiyandi:Doing things like receding and really going the extra distance to get them
Prasanna Malaiyandi:up and running as quickly as possible, including things like, Hey, if you
Prasanna Malaiyandi:have more than three terabytes, we're gonna ship you a drive so you can
Prasanna Malaiyandi:upload to the drive and then send it to help speed up the process.
Prasanna Malaiyandi:W. Curtis Preston: Yeah, so for those not familiar with this process,
Prasanna Malaiyandi:when you are a cloud backup vendor, the, you know, the hardest part
Prasanna Malaiyandi:is that initial backup, right?
Prasanna Malaiyandi:You know, it's one
Prasanna Malaiyandi:So
Prasanna Malaiyandi:fast.
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Damn.
Prasanna Malaiyandi:Physics.
Prasanna Malaiyandi:Uh, yeah.
Prasanna Malaiyandi:So when you, you know, if, if you're back, you can back up.
Prasanna Malaiyandi:I mean, you know, I remember, uh, you know, when I worked
Prasanna Malaiyandi:at Druva, we had really, you know, multi petabyte customers.
Prasanna Malaiyandi:You can back up that amount of data over the internet, but that
Prasanna Malaiyandi:first backup is gonna be something.
Prasanna Malaiyandi:And that's why we used the snowball edge.
Prasanna Malaiyandi:which was a, an appliance that, that you would ship to them and, you know,
Prasanna Malaiyandi:it already had the technology on there.
Prasanna Malaiyandi:And then they would back up to that and then that would ship and it was
Prasanna Malaiyandi:just mu, you know, basically a very fancy sneaker net was the way to
Prasanna Malaiyandi:get the very first copy up there.
Prasanna Malaiyandi:And you needed an, you needed one of those snowball edges for I think about
Prasanna Malaiyandi:every 75 terabytes, something like that.
Prasanna Malaiyandi:And, um, then.
Prasanna Malaiyandi:And so that's what they're saying here.
Prasanna Malaiyandi:And that by the way, that is referred to as the seed for those that don't
Prasanna Malaiyandi:live in that world, that initial backup is referred to the seed.
Prasanna Malaiyandi:So when you read the story, you're right, they did accept
Prasanna Malaiyandi:responsibility and they said
Prasanna Malaiyandi:the first thing that we need to get going is that seed.
Prasanna Malaiyandi:And so, you know, you were talking about the, one of the way, just
Prasanna Malaiyandi:like I said, the, the way, uh, my former employer would do it was they
Prasanna Malaiyandi:would ship, uh, a snowball edge.
Prasanna Malaiyandi:They're shipping whatever they need to ship to these customers
Prasanna Malaiyandi:in order to make them whole.
Prasanna Malaiyandi:One additional wrinkle, I think of this was that this was
Prasanna Malaiyandi:specifically for their MSP customers.
Prasanna Malaiyandi:Um, and I don't know what percentage of their business, uh, is via MSPs, but
Prasanna Malaiyandi:that does, that does make an additional wrinkle in that not only are they
Prasanna Malaiyandi:impacting their customer, which is the Ms.
Prasanna Malaiyandi:P, they're impacting the end customer, which is the MSPs customer.
Prasanna Malaiyandi:Um.
Prasanna Malaiyandi:But again, I, you know, I want to, uh, you know, give a shout out,
Prasanna Malaiyandi:um, to the CEO, Brandon Lacey.
Prasanna Malaiyandi:I like that he, he doesn't try to pass the blame.
Prasanna Malaiyandi:He doesn't try to, you know, I, I don't know.
Prasanna Malaiyandi:I.
Prasanna Malaiyandi:Maybe it's because he didn't have a choice, right?
Prasanna Malaiyandi:You know, when you know the other vendor, they're saying, Hey, it was our, it
Prasanna Malaiyandi:was our storage array, uh, that failed.
Prasanna Malaiyandi:And that's why, you know, that's why our backups died in this case.
Prasanna Malaiyandi:They didn't really add anything to blame it on other than, dang
Prasanna Malaiyandi:it, why is my fingers here?
Prasanna Malaiyandi:They didn't really have anything other to blame it on other than human failure.
Prasanna Malaiyandi:Because it was a human that prematurely decommissioned a server before
Prasanna Malaiyandi:that server had been migrated.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:And that's just sort of a breakdown of processes, right?
Prasanna Malaiyandi:It's not a technology issue.
Prasanna Malaiyandi:Right?
Prasanna Malaiyandi:And I'm sure now they have a process in place where they're like, Hey, make
Prasanna Malaiyandi:sure that the data validate that the server is actually up in wherever it
Prasanna Malaiyandi:needs to be before you decommission.
Prasanna Malaiyandi:Because.
Prasanna Malaiyandi:W. Curtis Preston: Yeah, that is, that is probably the
Prasanna Malaiyandi:biggest disappointment, right?
Prasanna Malaiyandi:Is you would think you're doing this like major migration.
Prasanna Malaiyandi:I remember there was Datto, they did an exfiltration out of, um, Amazon and they
Prasanna Malaiyandi:built their own private data center that was a giant, and they said they believed
Prasanna Malaiyandi:at the time, and I believe this was Datto, they believed at the time it was
Prasanna Malaiyandi:the single largest exfiltration project ever to leave en mass, uh, from AWS.
Prasanna Malaiyandi:And it apparently went flawlessly.
Prasanna Malaiyandi:You, you've got to, you know, when you're doing a migration, it's like,
Prasanna Malaiyandi:you've gotta have that check, right?
Prasanna Malaiyandi:You've gotta have that check.
Prasanna Malaiyandi:We're gonna kill the thing.
Prasanna Malaiyandi:In fact, I wouldn't even kill the thing,
Prasanna Malaiyandi:You would just sort of, yeah.
Prasanna Malaiyandi:W. Curtis Preston: I would, I would, uh, turn it off,
Prasanna Malaiyandi:Yep.
Prasanna Malaiyandi:W. Curtis Preston: right?
Prasanna Malaiyandi:I, I would, in fact, I remember, um.
Prasanna Malaiyandi:I remember the very first migration that I had anything to do with, and
Prasanna Malaiyandi:I'm gonna, for those of you for the fellow gray hairs in the crowd, we were
Prasanna Malaiyandi:migrating off, uh, an at and T three B two 4,000, which was the at t's attempt
Prasanna Malaiyandi:at a multi-processing server, right?
Prasanna Malaiyandi:So the three B two 1000 was the.
Prasanna Malaiyandi:First computer ever built specifically for Unix.
Prasanna Malaiyandi:Okay.
Prasanna Malaiyandi:I love how you just have like these model numbers.
Prasanna Malaiyandi:Just like right off the tongue, it just rolls right off.
Prasanna Malaiyandi:You're like, oh yeah, everyone should know about the 3D two.
Prasanna Malaiyandi:It's like R 2D two.
Prasanna Malaiyandi:you might as well call it that.
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:What's that?
Prasanna Malaiyandi:You might as well call it R 2D two.
Prasanna Malaiyandi:W. Curtis Preston: Yeah, so the three B two 1000 was the first, you know, 'cause
Prasanna Malaiyandi:I think it was the PDP 11 was the one that, uh, that Unix initially ran on.
Prasanna Malaiyandi:But the three B two was the first computer specifically designed to run.
Prasanna Malaiyandi:Unix and it ran, uh, system five and, um, you know, directly from
Prasanna Malaiyandi:Bell Labs, right at and t and the three B two 4,000 was, you know, they
Prasanna Malaiyandi:attempted a multi-processing machine and it, it just, it wasn't very good.
Prasanna Malaiyandi:And so we were trying to migrate off of that, and I remember that.
Prasanna Malaiyandi:Basically what we would do is we would just kill things as we
Prasanna Malaiyandi:believed we had migrated it off.
Prasanna Malaiyandi:We would, we would turn off a database.
Prasanna Malaiyandi:We would turn off a server.
Prasanna Malaiyandi:see what broke.
Prasanna Malaiyandi:W. Curtis Preston: That's the, and well, yeah, and then see if anything broke.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:You don't, I, I don't know.
Prasanna Malaiyandi:I, I guess that's the one surprise here, is that they, they decommission
Prasanna Malaiyandi:the server to the point of actually, you know, deleting it somehow.
Prasanna Malaiyandi:Or, or, I, I know they talk about decommissioning.
Prasanna Malaiyandi:I do wonder if it's sort of, um, because it's metadata, I wonder if it's like
Prasanna Malaiyandi:a distributed system and maybe they killed the node, uh, without, and so
Prasanna Malaiyandi:since it didn't have access to the metadata, it's sort of like maybe
Prasanna Malaiyandi:they weren't expecting that to happen.
Prasanna Malaiyandi:Right?
Prasanna Malaiyandi:So even though they say
Prasanna Malaiyandi:W. Curtis Preston: Yeah, I, yeah.
Prasanna Malaiyandi:I don't know the details.
Prasanna Malaiyandi:So.
Prasanna Malaiyandi:W. Curtis Preston: Yeah, we don't know the details.
Prasanna Malaiyandi:We're Monday morning quarterbacking, et cetera.
Prasanna Malaiyandi:Um, but let's, but let's just talk about it.
Prasanna Malaiyandi:When you are doing a migration, you don't kill the thing.
Prasanna Malaiyandi:You don't, you don't kill the old one.
Prasanna Malaiyandi:You turn off the old thing, right?
Prasanna Malaiyandi:You turn it off and then you wait a while before you, uh, actually
Prasanna Malaiyandi:redeploy it, delete it, whatever.
Prasanna Malaiyandi:And then I would.
Prasanna Malaiyandi:Then I, you know, I, I find myself wondering, well, I, I guess there wasn't,
Prasanna Malaiyandi:there wasn't a backup of that thing.
Prasanna Malaiyandi:There wasn't a backup of the backup, which is not that uncommon
Prasanna Malaiyandi:Yep.
Prasanna Malaiyandi:Backup
Prasanna Malaiyandi:W. Curtis Preston: uh, backup.
Prasanna Malaiyandi:Backup.
Prasanna Malaiyandi:I.
Prasanna Malaiyandi:Services.
Prasanna Malaiyandi:Right?
Prasanna Malaiyandi:Um, and as a result, when that system was deleted, it, there was no backup
Prasanna Malaiyandi:of that backup, which I think many people might be surprised by that.
Prasanna Malaiyandi:What do you think?
Prasanna Malaiyandi:No.
Prasanna Malaiyandi:Uh, I think people will be surprised because they think that backup
Prasanna Malaiyandi:systems are highly resilient.
Prasanna Malaiyandi:And of course someone's gonna do a backup of a backup, but you have to also realize
Prasanna Malaiyandi:that there's additional costs associated with doing a backup of a backup.
Prasanna Malaiyandi:And.
Prasanna Malaiyandi:Usually backup systems, people are trying to cut costs, right?
Prasanna Malaiyandi:Because this is like the tertiary copy.
Prasanna Malaiyandi:And so having high availability or a replica copy of your backup system, unless
Prasanna Malaiyandi:it's critical, which a lot of systems are, and people do replicate their backup
Prasanna Malaiyandi:systems, but not everyone could afford to do that, and especially given that this
Prasanna Malaiyandi:was for their MSP customers who are also trying to make money and add value, right?
Prasanna Malaiyandi:They're looking at probably a.
Prasanna Malaiyandi:Reasonable offering, right,
Prasanna Malaiyandi:uh, without all the bells and whistles.
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:And in a world where services are often purchased on price.
Prasanna Malaiyandi:Not necessarily alone, but where price is the primary motivator For many
Prasanna Malaiyandi:customers, it's, this is one of those things where no one ever bought Salesforce
Prasanna Malaiyandi:because they had a really good backup.
Prasanna Malaiyandi:No one ever bought Oracle because it had a really good backup.
Prasanna Malaiyandi:They bought Oracle because it had really good features and did
Prasanna Malaiyandi:the thing they needed it to do.
Prasanna Malaiyandi:Nobody's nobody, there's, there's nowhere, no ad in the world.
Prasanna Malaiyandi:Said, buy Oracle, because our backup is really good.
Prasanna Malaiyandi:Right?
Prasanna Malaiyandi:And so the same thing is true here.
Prasanna Malaiyandi:Buy our backup service because we also have a tertiary copy of
Prasanna Malaiyandi:your data just in case we screw up the copy that you gave us.
Prasanna Malaiyandi:Nobody's gonna even know that.
Prasanna Malaiyandi:Um, and they're, and they're cer, they certainly don't want to pay for it.
Prasanna Malaiyandi:All they see is that this one is more expensive than that one.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:The one thing I am a little surprised about though is given this is a backup
Prasanna Malaiyandi:system, and it looks like they did have their metadata and data split apart just
Prasanna Malaiyandi:based on what we read in the articles, I'm surprised they didn't have a self
Prasanna Malaiyandi:describing format for the data itself.
Prasanna Malaiyandi:I know?
Prasanna Malaiyandi:there are other systems which tend to do that as sort of a fail safe, so I'm not
Prasanna Malaiyandi:necessarily going to create a copy of the data, but if I, my, something happens,
Prasanna Malaiyandi:my metadata or there's some corruption or whatever else, I should at least be
Prasanna Malaiyandi:able to build back all the metadata based on the data that I still have access to.
Prasanna Malaiyandi:I.
Prasanna Malaiyandi:W. Curtis Preston: Yeah, that's a good point.
Prasanna Malaiyandi:And, and that is a, another really good feature to have in a backup system
Prasanna Malaiyandi:that if the, basically the backup catalog, the image database, the, you
Prasanna Malaiyandi:know, the database of all the backups, if that gets messed up, it would be
Prasanna Malaiyandi:really nice if the, if the, uh, you know, if you could rebuild that.
Prasanna Malaiyandi:And I remember back when I was doing RFPs and the, one of the questions
Prasanna Malaiyandi:we asked was if the backup catalog, um, you know, gets corrupt, can,
Prasanna Malaiyandi:can we rebuild it given the backups?
Prasanna Malaiyandi:And I do remember at least one major vendor.
Prasanna Malaiyandi:I won't call 'em out by name 'cause it might not still be the
Prasanna Malaiyandi:answer, but their answer was.
Prasanna Malaiyandi:Um, you know what, they, literally, the phrase was, our backup format
Prasanna Malaiyandi:is so proprietary that even we can't read it if we don't know what's on it.
Prasanna Malaiyandi:And they, and they saw that as like a, like a feature.
Prasanna Malaiyandi:Uh, by the way, it turned out, interestingly enough, it turned out
Prasanna Malaiyandi:that, uh, it wasn't that proprietary and.
Prasanna Malaiyandi:Other vendors have figured out how to read that backup date.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:As long as it's not, um, encrypted.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:That that's a whole other, whole other thing.
Prasanna Malaiyandi:So I don't think we've ever talked about
Prasanna Malaiyandi:this on the podcast before.
Prasanna Malaiyandi:And so is this something, I know we've talked previously about, Hey, here are
Prasanna Malaiyandi:some questions you should be asking your backup vendor when trying to figure
Prasanna Malaiyandi:out like what their solution does.
Prasanna Malaiyandi:Do you think this is worthwhile to throw into that list of questions?
Prasanna Malaiyandi:W. Curtis Preston: Of, do you have a backup of the backup?
Prasanna Malaiyandi:Either a backup of the backup or if I lose
Prasanna Malaiyandi:part of my system, like my catalog, can I still recover my data?
Prasanna Malaiyandi:W. Curtis Preston: I, I do think that's a perfectly valid question to ask.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:And I think the other question is perfectly valid as well of
Prasanna Malaiyandi:if, what protections do you, I would point at this story, what
Prasanna Malaiyandi:protections do you have against this happening in your environment?
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Um,
Prasanna Malaiyandi:because it may not be malicious.
Prasanna Malaiyandi:It's just accidental things happen.
Prasanna Malaiyandi:It's all software.
Prasanna Malaiyandi:People are writing software, right?
Prasanna Malaiyandi:W. Curtis Preston: Yeah, the, you know, so this, remind this, this story to
Prasanna Malaiyandi:me is a, is a combination of the KPMG story and the Carbonite story, right?
Prasanna Malaiyandi:Because like the KPMG story, and by the way, we have another episode on that.
Prasanna Malaiyandi:Please feel free to check that out.
Prasanna Malaiyandi:The KPMG story is one of failure of process as well, where.
Prasanna Malaiyandi:You'd like to think that at a, at a company that size and when you're
Prasanna Malaiyandi:doing something of this magnitude, I, you know, I, when I think back
Prasanna Malaiyandi:on that story, I don't think they.
Prasanna Malaiyandi:Thought this was of the, the, we're, I'm just trying to fix one
Prasanna Malaiyandi:person's, one person's account.
Prasanna Malaiyandi:They didn't think that what they were touching was a tool in this
Prasanna Malaiyandi:case, uh, retention policies.
Prasanna Malaiyandi:They were touching a tool that could wipe out everybody's data.
Prasanna Malaiyandi:They, they
Prasanna Malaiyandi:Prasanna Malaiyandi: Hundred 40,000 people.
Prasanna Malaiyandi:W. Curtis Preston: 140,000 people's data with no backup.
Prasanna Malaiyandi:Uh, that's a great, that's a great story.
Prasanna Malaiyandi:Check that one out.
Prasanna Malaiyandi:But I, the, the big thing with processes, you've, you've got to have
Prasanna Malaiyandi:a very, uh, you know, the bigger the project, the, the bigger the process.
Prasanna Malaiyandi:And I do remember, I.
Prasanna Malaiyandi:When the, the biggest one I remember being involved in it was, it, was, it, it wasn't
Prasanna Malaiyandi:a migration, it was a physical move.
Prasanna Malaiyandi:We moved, a bunch of servers from one data center to another data center,
Prasanna Malaiyandi:We had built a nice new big data center and we're gonna move the servers from the
Prasanna Malaiyandi:old data center to the new data center.
Prasanna Malaiyandi:And we had a massive process and we had tested it from one end to the other,
Prasanna Malaiyandi:and we outsourced whatever we could.
Prasanna Malaiyandi:So like we hired.
Prasanna Malaiyandi:A professional moving company to move the servers we hired, uh, and paid extra
Prasanna Malaiyandi:to have HP come in and, and, you know, unplug the servers and go ominous dominus,
Prasanna Malaiyandi:you know, you may touch this server.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Um, and then, and then it got handed to the movers and stuff
Prasanna Malaiyandi:and we had this big process.
Prasanna Malaiyandi:Um, but there was one major failing and that was from the old
Prasanna Malaiyandi:server to the new server room.
Prasanna Malaiyandi:The new server room had, um, Twistlock.
Prasanna Malaiyandi:You know, plugs under, underneath the floor.
Prasanna Malaiyandi:And the old server room had the old standard, uh, north
Prasanna Malaiyandi:American, uh, two-prong plugs.
Prasanna Malaiyandi:And, uh, that was the, that was one day where I truly saved the day because
Prasanna Malaiyandi:I had, uh, oh, and the electricians, that would be the ones to help us.
Prasanna Malaiyandi:Were busy at the CEO's house, putting up his Christmas lights.
Prasanna Malaiyandi:Um, and so, and so they didn't think we'd need electricians during our big move.
Prasanna Malaiyandi:And I knew how to fix this problem, right?
Prasanna Malaiyandi:I, I'm like, look, we can do this.
Prasanna Malaiyandi:I just need to go to a gray bar, uh, electrical store.
Prasanna Malaiyandi:And there's one down the road and it was closing in.
Prasanna Malaiyandi:20 minutes, uh, by the time we realized what we needed to do, and I, I got in my
Prasanna Malaiyandi:car with, with my boss and, and he looked at me and he says, I will pay the ticket.
Prasanna Malaiyandi:Just get us there.
Prasanna Malaiyandi:And we drove.
Prasanna Malaiyandi:I drove and it was raining.
Prasanna Malaiyandi:It was, I mean, there was water everywhere.
Prasanna Malaiyandi:I'm driving to this gray bar and I've got it.
Prasanna Malaiyandi:It's, it's, it's closing in 20 minutes and it's 15 minutes away.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:I don't have time to play.
Prasanna Malaiyandi:And I remember hydroplaning going around a big semi, and I remember
Prasanna Malaiyandi:like, you know, swerving around a corner and all this stuff.
Prasanna Malaiyandi:And I remember that when we got there, there's that scene in, uh, planes, trains,
Prasanna Malaiyandi:and Automobiles where, where Steve, Steve Martin's fingers are stuck in the dash.
Prasanna Malaiyandi:Uh, that was the way, uh, Lou was when we, when we got there.
Prasanna Malaiyandi:But we got, we walked up to the door and it was five minutes prior to the,
Prasanna Malaiyandi:you know, prior to the, the closing.
Prasanna Malaiyandi:They were, the door was locked.
Prasanna Malaiyandi:And, uh, and, and they, they pointed, they pointed at her watch.
Prasanna Malaiyandi:They're like, they're pointing at their watch as a time.
Prasanna Malaiyandi:And I pointed up at the clock on their wall and I was like,
Prasanna Malaiyandi:no, it's not one o'clock yet.
Prasanna Malaiyandi:Open your damn door.
Prasanna Malaiyandi:And then we, you know, uh, and then I think, I think I like, I was like, please,
Prasanna Malaiyandi:please, you know, please let me in.
Prasanna Malaiyandi:And we got it.
Prasanna Malaiyandi:And we bought all the stuff.
Prasanna Malaiyandi:And I, I non.
Prasanna Malaiyandi:Traded non whatever electrician, but I was trained as one in the Navy, um, which at
Prasanna Malaiyandi:that time was only like five years prior.
Prasanna Malaiyandi:Um, and then we wired adapters from.
Prasanna Malaiyandi:A to B and save the day anyway.
Prasanna Malaiyandi:Yeah, you gotta test your process and you gotta be ready
Prasanna Malaiyandi:for failures in your process.
Prasanna Malaiyandi:But I, I think that, you know, so the two big this, this one I feel very different.
Prasanna Malaiyandi:Like I said in the beginning, I feel very different about
Prasanna Malaiyandi:this one versus the carbonite
Prasanna Malaiyandi:but why
Prasanna Malaiyandi:W. Curtis Preston: because they didn't try to, they didn't try to pass the buck.
Prasanna Malaiyandi:is that really the only difference in your mind is,
Prasanna Malaiyandi:well, I think also in the Carbonite case, the only reason we heard about
Prasanna Malaiyandi:it is when they try to sue the vendor.
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:Um, this.
Prasanna Malaiyandi:Yeah, that's a good question.
Prasanna Malaiyandi:If, would we have ever heard about the story if they, I, I don't know.
Prasanna Malaiyandi:I, I, I'd like to think that it would still have made news, but all we
Prasanna Malaiyandi:heard about was the fact that they store, they sued their storage vendor
Prasanna Malaiyandi:in this case.
Prasanna Malaiyandi:The article, right.
Prasanna Malaiyandi:Um.
Prasanna Malaiyandi:It starts out with this quote from the CEO, right?
Prasanna Malaiyandi:In terms of magnitude is the only thing I'm thinking about waking
Prasanna Malaiyandi:up and going to bed every night.
Prasanna Malaiyandi:It's orders of magnitude more important than anything else that I have going
Prasanna Malaiyandi:on in terms of operating this business.
Prasanna Malaiyandi:Um, says the CEO right, uh, of this company trying to bounce back.
Prasanna Malaiyandi:Now we haven't, I tried to find, I didn't find, this was almost two years ago now.
Prasanna Malaiyandi:I didn't see.
Prasanna Malaiyandi:An update on what happened.
Prasanna Malaiyandi:They were saying, we're trying to get people up and running.
Prasanna Malaiyandi:I didn't see any stories like we did with the Carbonite about some
Prasanna Malaiyandi:customers that actually lost data because they had to restore something
Prasanna Malaiyandi:while they were trying to be seated.
Prasanna Malaiyandi:Uh, we don't have any of those
Prasanna Malaiyandi:Now, I know you're a big Reddit fan, and I know you
Prasanna Malaiyandi:probably were browsing threads about this.
Prasanna Malaiyandi:Did you see any, like, what was the sentiment in those threads
Prasanna Malaiyandi:from end users, customers that, if
Prasanna Malaiyandi:you can
Prasanna Malaiyandi:W. Curtis Preston: that's a, actually, that's a good one.
Prasanna Malaiyandi:Let me, uh, let me pull up that thread.
Prasanna Malaiyandi:'cause I didn't, I didn't look too much at the thread.
Prasanna Malaiyandi:This is funny.
Prasanna Malaiyandi:Seems like a backup company that loses backups is no longer a backup
Prasanna Malaiyandi:company, but a data disposal service.
Prasanna Malaiyandi:I see a pivot opportunity.
Prasanna Malaiyandi:They should certify these backups as destroyed and issued certificates.
Prasanna Malaiyandi:I'd say the Reddit response is typical of every response with a cloud outage.
Prasanna Malaiyandi:There are people that are just, you know, saying, ah, the cloud sucks, and,
Prasanna Malaiyandi:you know, and the cloud is just, it's just another computer, it's just another
Prasanna Malaiyandi:service, and you have to verify it just like you verify every other system.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Um, and there, there were some comments in the thread saying that.
Prasanna Malaiyandi:You know, one of the reasons that Arc Serve and StorageCraft, uh, merged was
Prasanna Malaiyandi:that they had been going down for a while in terms of quality of service.
Prasanna Malaiyandi:Again, these are pe other people's comments, um, and that they, that they
Prasanna Malaiyandi:weren't that surprised at, at this event.
Prasanna Malaiyandi:But, um, they're still around.
Prasanna Malaiyandi:They're still running.
Prasanna Malaiyandi:Um, just like Carbonite is still around and still running.
Prasanna Malaiyandi:I, I, I, you know, I wish them the best.
Prasanna Malaiyandi:Uh, this was a major migration fail.
Prasanna Malaiyandi:Yeah, and it's also one of those things
Prasanna Malaiyandi:people have, like once you lose that trust, it get, it's difficult,
Prasanna Malaiyandi:especially in the space where you are that last line of defense.
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:I mean, people are still on LastPass.
Prasanna Malaiyandi:Prasanna Malaiyandi: Yeah, let's not go there.
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:What's that?
Prasanna Malaiyandi:not go there.
Prasanna Malaiyandi:W. Curtis Preston: Yeah, yeah.
Prasanna Malaiyandi:Anyway.
Prasanna Malaiyandi:All right.
Prasanna Malaiyandi:Well, um, not, not a ton, not some huge lessons here, but the big one
Prasanna Malaiyandi:I think here is, you know, I, I like the Maya Culpa that the company did.
Prasanna Malaiyandi:I.
Prasanna Malaiyandi:Uh, and I like that they knew that their priority was getting the,
Prasanna Malaiyandi:the next backup, but unfortunately that's gonna take a long time.
Prasanna Malaiyandi:Um, and the, the biggest lesson here is if you're doing a migration of any
Prasanna Malaiyandi:kind, don't go shooting the other servers until you know that it's fully migrated.
Prasanna Malaiyandi:You know?
Prasanna Malaiyandi:what, what?
Prasanna Malaiyandi:Is there any lessons learned for the end users?
Prasanna Malaiyandi:Like there's a, like what could they have done?
Prasanna Malaiyandi:I don't think there's anything that they could have done in this situation.
Prasanna Malaiyandi:W. Curtis Preston: yeah, the only, the only thing I, in terms of.
Prasanna Malaiyandi:Trying to identify, I mean, what's a vendor gonna say?
Prasanna Malaiyandi:This is the difficulty with the cloud.
Prasanna Malaiyandi:We've discussed this in previous other things, is it's hard to actually verify
Prasanna Malaiyandi:if you say, do you have processes in place for if somebody, if somebody over
Prasanna Malaiyandi:there accidentally, you know, blows up a server, do you have processes in
Prasanna Malaiyandi:place to be able to bring that server back or re of course they're gonna say,
Prasanna Malaiyandi:yeah, that's what they're gonna say.
Prasanna Malaiyandi:Um, the, the, the one thing you could think about, again, this is
Prasanna Malaiyandi:expensive, but people talk about using a service that puts stuff in more
Prasanna Malaiyandi:than one place or using more than one service, but people don't wanna pay
Prasanna Malaiyandi:for one backup service, let alone two.
Prasanna Malaiyandi:Could they do that?
Prasanna Malaiyandi:Should they do that?
Prasanna Malaiyandi:Sure.
Prasanna Malaiyandi:Will they do that?
Prasanna Malaiyandi:Nobody does that,
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:No one's gonna spend the money in their budget on that.
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:Um.
Prasanna Malaiyandi:I mean, it's hard enough to get people to do backups in the first place.
Prasanna Malaiyandi:It's hard enough to get them to, to properly secure those backups.
Prasanna Malaiyandi:Um, they're not gonna pay for a completely separate, just in case they're, you know,
Prasanna Malaiyandi:uh, I will say having something like I.
Prasanna Malaiyandi:StorageCraft or carbonite, die and lose all your data is not
Prasanna Malaiyandi:the same as losing all your data.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:It's, it's, it's all the backups of all your data.
Prasanna Malaiyandi:And there will be some, could be some ramifications to your business if you
Prasanna Malaiyandi:need to restore, but statistically speaking, you're not probably
Prasanna Malaiyandi:going to need to restore today.
Prasanna Malaiyandi:Um, just in terms of statistically Right.
Prasanna Malaiyandi:Um, it's not the same.
Prasanna Malaiyandi:So, you know,
Prasanna Malaiyandi:So, okay, so if I was an end user, I get a
Prasanna Malaiyandi:notification from StorageCraft that says, Hey, by the way, we had an issue.
Prasanna Malaiyandi:We lost all your backups.
Prasanna Malaiyandi:They're gonna say, okay, I'm gonna try to help reseed.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Get all your data up there before that happens.
Prasanna Malaiyandi:Because like you said, it could take weeks, it could take months.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:What is the first thing that the end user should do at that point?
Prasanna Malaiyandi:Right?
Prasanna Malaiyandi:Once they get that first notification to make sure that they don't end up
Prasanna Malaiyandi:losing all their data in case something happens to their production copy.
Prasanna Malaiyandi:W. Curtis Preston: Well, I, I'll say this, you got any big projects,
Prasanna Malaiyandi:uh, in process, maybe you should pause those projects, right?
Prasanna Malaiyandi:Any big changes, any big maintenance windows.
Prasanna Malaiyandi:Maybe you should put a pause on those maintenance windows.
Prasanna Malaiyandi:Uh, the other is you should go back to old school backups for the
Prasanna Malaiyandi:time being, for anything critical.
Prasanna Malaiyandi:Uh, you know, you should make a local backup of any kind that, a backup of any
Prasanna Malaiyandi:kind is better than a backup of no kind.
Prasanna Malaiyandi:Uh, that's a really good question, right.
Prasanna Malaiyandi:Um, I would put a pause on projects and I would see what I can do to get
Prasanna Malaiyandi:some local backups of stuff, right?
Prasanna Malaiyandi:There are plenty of open source tools out there.
Prasanna Malaiyandi:That can, you know, surely somewhere you've got some disc and then you've
Prasanna Malaiyandi:got some super critical servers that you really don't wanna lose.
Prasanna Malaiyandi:Get some local backups of those.
Prasanna Malaiyandi:Um, you know.
Prasanna Malaiyandi:or I know you had mentioned in the past, right?
Prasanna Malaiyandi:You really like the iDrive service, cloud service,
Prasanna Malaiyandi:W. Curtis Preston: Right.
Prasanna Malaiyandi:right.
Prasanna Malaiyandi:For just doing
Prasanna Malaiyandi:W. Curtis Preston: Well, the.
Prasanna Malaiyandi:backup
Prasanna Malaiyandi:W. Curtis Preston: Yeah, the, the only problem with that is, you
Prasanna Malaiyandi:know, again, it's physics, right?
Prasanna Malaiyandi:So I'm just trying to, I'm just trying to think of something
Prasanna Malaiyandi:that you can do quickly.
Prasanna Malaiyandi:Um, you know
Prasanna Malaiyandi:Copy it to a different machine.
Prasanna Malaiyandi:Go buy a couple external hard
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:Specifically for those super high value, super highly critical
Prasanna Malaiyandi:servers, you know, uh, things like backup servers for other customers.
Prasanna Malaiyandi:Uh uh.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Okay.
Prasanna Malaiyandi:Well, all right, well, thanks for helping me walk through that.
Prasanna Malaiyandi:That was a really good question there at the end.
Prasanna Malaiyandi:Prasanna, thanks for helping me walk through this.
Prasanna Malaiyandi:No, no worries.
Prasanna Malaiyandi:That's why uh, I enjoy doing these podcasts 'cause it helps all of us, right?
Prasanna Malaiyandi:Just figure out what should we be
Prasanna Malaiyandi:doing in case this ever happens.
Prasanna Malaiyandi:W. Curtis Preston: That's what we're trying to do is to pass on these
Prasanna Malaiyandi:lessons to you, uh, the listener so that you can learn from other
Prasanna Malaiyandi:people's struggles, unfortunately.
Prasanna Malaiyandi:But, uh, you're why we do this.
Prasanna Malaiyandi:We want to turn you into a cyber recovery hero.
Prasanna Malaiyandi:That's a wrap
Prasanna Malaiyandi:The backup wrap up is written, recorded and produced by me w Curtis Preston.
Prasanna Malaiyandi:If you need backup or Dr.
Prasanna Malaiyandi:Consulting content generation or expert witness work,
Prasanna Malaiyandi:check out backup central.com.
Prasanna Malaiyandi:You can also find links from my O'Reilly Books on the same website.
Prasanna Malaiyandi:Remember, this is an independent podcast and any opinions that you
Prasanna Malaiyandi:hear are those of the speaker.
Prasanna Malaiyandi:And not necessarily an employer.
Prasanna Malaiyandi:Thanks for listening.