Several years ago, a small startup placed all their eggs in the Google

drive basket, and then proceeded to set that basket on fire.

With one fatal click by an admin over a million dollars.

Went up in smoke.

Today we investigate the tragic tale of Musey, a promising young

company building an innovative app who lost everything when their Google

account was wiped out instantly.

This is our second in a series called cloud disasters.

Organizations that thought their data in the cloud was protected.

Only to find out it wasn't, it was too late for them, but not for you.

Hi, I'm w Curtis Preston, AKA Mr.

Backup.

And my goal with this podcast is to make sure that something

like that never happens to you.

This is the backup wrap up.

W. Curtis Preston: Welcome to the show.

I'm your host, w Curtis Preston, and I have with me Persona Ana.

I'm doing well, Curtis.

We're talking about this concept of.

Things that people think don't need to be backed up.

And they definitely need to be backed up.

And we're zooming in and doing a, a deep dive into each of the

stories that we often talk about when we reference this, right?

So these are, uh, typically cloud systems, you know, cloud services that people

thought that were backed up and then.

Uh, when they accidentally screwed something up, they

found out they weren't

They were relying on it for their business and for their

livelihood and bad things happen.

W. Curtis Preston: And sometimes, many times those things are like attacks.

Uh, you know, you know, things like that.

In this case though, this, uh, one is, you know, it's a reminder that.

We spent a lot of time on the podcast, talking especially lately

about cyber attacks and things like that and ransomware, and

we don't talk that much about equipment failure because typically, uh, you

know, thanks to the advent of raid and erasure coding and things like that.

Generally, that's not the reason people restore data.

The interesting thing about this story, it that I, one interesting thing that

I found out, or that I noticed was that it was only covered by the register.

So, uh, thanks to the folks over there.

I, I don't even know.

How they, uh, you know, found it, somebody sent it to them

and then nobody else covered it.

Which is interesting 'cause I looked around, I couldn't

find any coverage of it.

So this is about a company called Musi, which also I found

out, goes by the name of Moss, M-O-S-S-S.

They do have a Facebook page.

That you can find.

But They were a company that was building an interior design app that would be

used by influencers, home decorators and other folks to sort of showcase

what you see on the inside of a house.

And, um, basically somehow someone went and deleted their entire G

Suite account for the entire company.

Which contained all of their data.

And from what I could tell, they weren't doing backups.

And that's kind of where things start now.

They try to get the data back, and I think we'll talk about that in a

little bit about what they did to try to get their data back from Google.

But yeah, like Curtis says, it's not a happy ending.

W. Curtis Preston: Yeah.

It appears that it was their admin, right?

The idea that I remember hearing was that they were trying to delete a test account.

I.

But instead of deleting a test account, they deleted the production account.

And we need to be very specific here.

They didn't delete a user, right?

You know, in, in Google Drive or Google Workspace, you could have

users underneath your account.

They deleted The entire account

The organization's account.

W. Curtis Preston: Yeah.

The, yeah, the, the entire organization, right.

According to their court filing.

They did immediately contact Google and say, Hey, this thing happened.

And I'm gonna say, sadly, the Google, uh, support rep said,

I'll get right on it, rather than

saying, yeah, you're screwed.

Not my problem.

W. Curtis Preston: yeah, not my problem.

Yeah, you're screwed.

You should have backed that up.

Uh, where's your backups?

Uh, what the, uh, the Google person allegedly said was, we'll get right on it.

Meanwhile, they're out there desperately trying to reach someone at Google for

multiple days, even multiple weeks.

Meanwhile, they're not able to contact, their customers are not able to, you know,

access any of their intellectual property.

So apparently they had built.

This entire app, uh, and everything was stored in G Drive and apparently they

weren't, uh, you know, I'm just saying based on what happened, it does not

appear that they were synchronizing.

'cause you could synchronize, uh, Google Drive down to your, uh, you know, desktop.

And if they had deleted the account up there, the desktop copy

probably would have, uh, remained.

But they didn't do that because once they had deleted the, um, you know, this.

The, the cloud version of it, they had nothing.

Right?

So they had nothing in their, of their intellectual property that they

had built, uh, that they had spent a million and a half dollars on, right?

That, that was in, in the court filings, that they had spent a million and a

half dollars on this company so far.

And they also mentioned that they weren't able to contact their customers,

they weren't able to do business.

And there is, uh, they said a few weeks later.

I feel like saying

a few moments later.

I now have that in your head, don't I?

W. Curtis Preston: it's your fault.

Um, go Google did get back to them and say, uh, they can't get their data back.

I think the weirdest part of the story that you might want to talk about is this.

They're saying that

someone at Google, and I don't see any evidence of this, but they're

saying, someone at Google said something along the lines of, maybe you

could get the data back with a court

filing.

Did you see

anything about that?

I didn't see anything in the articles about

that, but that's just weird.

the Only time I could see that happening is potentially if they

had kept some of that data because of legal hold or some legal reason.

W. Curtis Preston: Yeah.

Or if for some reason they had a copy somewhere else,

like I could imagine in the case of doing like a disaster recovery copy,

maybe they had that copy somewhere else and you can get the data back.

Right.

It's not their obligation.

It's like a best effort.

I.

W. Curtis Preston: Right.

I agree with both scenarios.

I like that first one.

The second one, you know, we talk a lot about how many of these service

providers, they do have a backup, as I met quotes in the air of like the

entire data center that they can use.

To bring the data center back if they screw it up.

Right.

Um, but the, it's not built for you.

I give an example of Microsoft 365 says that they have a delayed copy of

exchange, uh, that can be used if they screw up the entire exchange environment.

But when I asked them directly as a Microsoft 365 customer,

can I use that delayed copy of exchange to restore my environment?

And they just said, no.

Right.

So maybe that's a scenario where somebody might have been alluding to it.

I don't know if that person was, uh, you know, um,

and,

W. Curtis Preston: talking outta school.

Right.

Um.

and also just along that, like if they were a billion dollar

company, I'm sure that it would've been a different result or Google may have

been more willing to jump in and help versus a smaller company like this.

W. Curtis Preston: yeah,

I'm not sure if the result would've been different.

I think maybe the activities along the way would've been the same.

Um, but the, the, the re by the way, the, where we get this, uh, this

comment about the subpoena or civil request was that was in their lawsuit.

And, and I have to say, I feel for these guys, uh, of what happened to them.

Because when I look at the filing, like you really have to see it to, like, you

have to literally lay your eyes on and we'll put a link to it in the show notes.

And what you see is a good portion of it is typed up and sort of like normal.

And then there's all these like handwritten notes all over it.

Like at the very beginning there's this like star.

And it says, serving a civil request to have access to our data

restored or provided as outlined a document for serving civil subpoenas

and civil requests given to us.

Emailed it says by Google.

Um, IE Google sent us instructions to get our data back.

Um, you know, and then.

They just sort of lay out this story in this lawsuit, which they say in the

lawsuit, we're not suing to get money.

We're suing to get our data back.

They basically believed that Google had a copy of their data and Google, you know,

clearly did not have a copy of their data.

Yeah, and just looking at the filing, I

don't think a lawyer wrote that,

W. Curtis Preston: No.

Oh yeah.

Thanks for bringing that up.

It is actually listed as a per se lawsuit,

which means that they filed

it themselves.

Yeah.

And just like, yeah, with the handwriting and everything else, you could tell

that it was an individual who was not an attorney writing and typing.

And I know that would, like you mentioned Curtis, they wanted their data back and

they were mentioning sort of what type of data they had stored in Google Drive.

Right?

It was, uh, their ip, ui, ux, mockups, user research.

And a whole bunch of other things, algorithm decisions, how they built it.

But the other thing is they were like, yeah, in our emails, we also had medical

records about someone's nephew who was going undergoing some treatment

and they needed access to their emails

W. Curtis Preston: Right.

That was the weirdest part, I think was just reading that they had intermixed.

I, you know what?

I don't know.

That just made no, that part made no sense to me.

Like it's not like they stored, it's not like this was the only copy of their.

Their nephew's medical records, like it was their only copy of

the nephew's medical records.

I, I thought that was very odd.

But the thing in the filing is that this was a filing in the federal.

US District Court for the Northern District of California, by the way, this

was in, um, the Bay Area and they, um, and so you have to give reason why the

federal court would be the jurisdiction, and they listed multiple reasons for that.

Um, but.

What we do know from the dockets, I was able to find that the lawsuit

was withdrawn, basically just

under two weeks later,

yeah.

That's suspicious.

Yeah.

Well, I think one thing we should also mention is this

happened in what year, Curtis?

W. Curtis Preston: uh, 2019.

Okay.

Yeah, so this was a while ago.

The interesting thing, and this is all just this is all just,

W. Curtis Preston: Yeah, we're hypothesizing, but I did my best to

try to find out, uh, everything I could, but this is our best effort

to, to figure out what happened.

yeah.

And I think what, like just given the timeframes, and it was two weeks

later after it was filed, either they.

Decided, or they maybe got legal counsel and talked to someone, they

were like, yeah, you're screwed.

Based on Google's response.

Or Google might have come after them and been like, Hey, if you really

wanna go through this, you're going to fail and then you're gonna have to

pay all our court fees, attorney fees,

W. Curtis Preston: Right,

in which case that would be like, I could, I can't even

imagine a Google attorney and how much that would cost because it would be

external counsel and everything else.

So they probably were like, yeah, it's not worth it.

Or maybe they talked to someone on Google who said, look.

We understand this is tough luck, but this is what it is.

And so maybe they had some sense knocked into them and they were like, yeah, this

doesn't make sense to move forward with.

It sucks, I feel for them, but there isn't much that Google

could really do at that point.

W. Curtis Preston: Yeah.

What was another sort of interesting thing about the story?

I, I just, I'm not sure why the dates don't line up.

In the filing, uh, the, the lawsuit filing, they said that the deletion

happened on June 8th, 2019.

The domain went up for sale on June 6th, 2019.

Mm.

W. Curtis Preston: The reason why I say that is, uh, the internet

archive found that right, the, the latest update for the website.

Uh, from June from 2019 was that it went, for se it said, domain available

for purchase, uh, June 6th, 2019, which would suggest that the deletion

happened on June 5th or June 6th.

But, um, I, I

thought that was to a little bit strange.

I don't mean to throw shade at the company, but it could also

be possible that the company just realized it's not feasible, what they're trying

to do in the timeframes they're given and ran outta cash and we're like, yeah.

, the one interesting thing is the fact that they're completely gone from the web.

Right.

There's no trace of 'em on the internet anywhere.

Right.

It's just really weird.

Like you can always find traces of a company that's gone bust somewhere.

W. Curtis Preston: Right.

yeah.

like you mentioned, the Register is the

only company that held this

article.

W. Curtis Preston: Yeah.

And, and that, that's why what I was using the internet, uh,

archive for and, uh, the mu app.com was the, was the website and it.

Like I said, it, it showed that it went up for purchase and then it

stayed that, that same way, uh, for another, uh, like a year and a half.

And then it got bought by a Japanese anime artist, which is a really weird

domain for a Japanese anime artist.

But, uh, in 2021.

And the, the co-founder that was just, that was mentioned in the

article, uh, left the company a few months later, um, about.

I, I think they might have tried to make a go of it with what was in

their, you know, memory or something.

Tried to bring the, the, you know, again, that's just pure conjecture,

but the, the company ceased to exist, uh, in, essentially in 2020.

It's a sad, sad story from a single mistake of an admin, an entire company

goes bust because they didn't have a backup of their, uh, cloud data.

Yep.

that's scary, right?

When you think about

W. Curtis Preston: Yeah.

Because it's not just the company, right?

It's all the time and effort all the employees put in, right?

All the customers who were hoping and buying products based on

what they were promised, right?

All of the things that they wanted, it's all like, poof.

Gone.

W. Curtis Preston: Yeah, yeah.

And all the, all those poor employees that were hoping probably for some,

uh, IPO, you know, payout or some kind of acquisition or whatever.

Obviously all that goes by.

Um, so when we think about this, what kind of

thoughts come to your mind?

Well, the first, of course, and I know like

we always like to talk about this, is backup, backup, backup, right?

Don't trust that your cloud provider is going to be backing up your data.

Do your backups yourselves.

Keep a copy because that could easily have saved this company.

W. Curtis Preston: Yeah, it would've made it a non-event.

. It'd be a nice story to tell a cocktail parties about the time

we almost lost the company.

But, uh, and, and that admin would still have a job.

Right?

But unfortunately, they weren't backing it up.

And again, this was a SaaS service.

Right?

Let's just make sure we understand this.

This wasn't the same as the first story that we talked about.

That was an IAS service infrastructure as a service.

This was a software as a service.

This was Google Drive.

This was.

Google Workspace, which is now what it, which is what it's now called.

And, um, they had stored their entire world in that and then deleted it.

If this story doesn't prove to you that SaaS services aren't

backing up your data, number one.

Number two, that they have no legal obligation to bring your

data back when you lose it.

Um, I, I don't know what story would, right.

Yeah.

W. Curtis Preston: Uh, 'cause they were sued, right?

They did a, I mean, it was a, it was a hastily filed, uh, you know, sort

of poorly written, uh, filing, but it.

It, they withdrew it for a, a couple weeks later.

Right.

Uh, we don't have any details on that, but it, it wasn't good.

Uh, the other thing is, you know, again, I mentioned it already, but

just to remember that, uh, you know, I, I, I often like to throw out

the phrase from Shakespeare, right?

There's, there's more on, there's more in heaven and earth that

is dreamt of in your philosophy.

And I, I think sometimes we focus a little bit too much lately

on cyber attacks and things.

Just realize sometimes people just do dumb stuff.

Fact, um, in fact, there's a couple of the stories that we have coming.

It's just somebody doing something dumb.

Somebody making a, a fat finger.

The title of the headline for this was, you know, admin Fat Fingers.

This company and, and basically deletes this whole company.

So just realize it's, it's human error.

Prior to hackers taking over everything, human error was the

number one reason we did restore.

Right now it's human error and direction.

I don't know how, what category you wanna put that is, it's not always hackers.

So the next one is really around keeping personal

stuff on your corporate drive.

I know it's hard.

Everyone works remotely, right?

You have a laptop.

It's like, oh, it's too hard for me to go grab a personal device and

start using that for personal stuff.

Like who doesn't log into Gmail from their work computer, honestly, right?

Or from their phone.

And so things do intermix, but the problem is what happens on a corporate drive?

Don't expect it.

If it's your personal stuff that it will be kept forever and that it's safe, right?

Personal stuff you should really take ownership of.

W. Curtis Preston: Not to mention that by putting medical information

on a company drive, you are subjecting that company to HIPAA requirements.

Um, it, it's not something that you should be doing, uh, you know,

on your, you know, on your, uh, on, on corporate systems, right?

Just not.

Yeah.

There, there's no, not to mention, you leave the company, right?

You're laid off, you're fired.

Uh, you know, they, they have no obligation to give you that data back.

Right?

Um, the other thing is, again, that I would just wanna mention that one of the

reasons why people talk about cloud data, not needing a backup as they're like, oh,

well, there's always this recycle bin.

Some deletions aren't covered by the recycle bin.

Deletion.

W. Curtis Preston: yeah, it sounds like account deletions.

There are things that you can do in Salesforce.

I was looking at a company that is specializing in backing up intra, which

is what Microsoft Azure AD is now called.

They were listing a handful of things that.

When you mess them up in intra, they're not covered by the recycle bin.

And this is another example, is that there are a number of things, again, there

are more things in heaven on earth than you are dreamt of, uh, in Salesforce.

I know for example, when you change a record, not when you delete

a record, but when you change a record, uh, that's not covered by

the recycle bin, they don't put the old version of That record.

That would be crazy if they did

that

Could you imagine?

I think this just goes to the point that backups are critical.

You can't trust what, that, there aren't going to ever be

these types of events, right?

That just go and don't have a recycle bin or maybe even the product doesn't

even have a recycle bin or your admin disabled the recycle bin.

What are you gonna do at that point?

W. Curtis Preston: Or the a or, or the hacker is able to clear out the

recycle bin because recycle bins can be cleared out, uh, manually.

Or what they can do is they can set the versions to one, you know, all

these, you know, they're, they're just, it's just a whole bunch of things.

This is why we back up.

Right.

This is, this is why it's just so frustrating to me that

there are groups of people.

Um, some of them who specialize in the platforms that we're talking

about who say, oh, well you don't really have to bag it up, um, despite.

There's a story, at least one story for every major platform we talk

about, and we're gonna cover those stories in subsequent episodes.

So yeah, so back up, back up, back up.

Um, and, and make sure that that backup is stored somewhere other than

the thing we're talking about, right?

Because that's the other problem with the recycle bin,

is that it's stored inside the thing, right?

Um,

or

versioning is stored inside the

thing.

Or in this case, if they had backed it up to

say a Google Drive under the same account, that wouldn't have saved them

W. Curtis Preston: That wouldn't have saved them.

Yeah.

Yeah.

You need to back it up to another account, um, uh, you know, to another account.

You know, if we're talking about cloud to cloud backups, just make sure it's

backed up to another account that has different, you know, authentication

authorization and has really, really tight lease privilege turned on.

Uh.

And a different region perhaps.

Right?

Um, you know, there are other services to pull it outta the cloud if that's what

you wanna do, but just make sure you're not leaving it all on the same place.

Um, there are services to back up Google Drive.

There are services to back up all of the things that we're talking about.

And, uh, that would be my favorite way.

If we're talking about how to back up the cloud, my favorite way to back up the

cloud would be to back it up to the cloud, do cloud to cloud backups, but back it

up to another service that you're paying

to do this.

Right, right.

Um,

I'm laugh you, you know why I'm laughing

W. Curtis Preston: Why?

Because you bought into the cloud service because you didn't

wanna manage equipment, infrastructure, patching, all of that stuff.

And if you went and you're like, Hey, I'm gonna go deploy an on-premises

backup solution so I can pull my data down from the cloud because I wanna

manage it and I want to do all the other stuff, that just seems kind of backwards.

W. Curtis Preston: Yeah, I know, I know there's at least one major

company that does it this way.

Uh, and I'm not saying it's necessarily wrong.

It just doesn't,

it doesn't seem right.

Right.

Yeah, unless you have a very specific use case for

needing that particular solution, I could imagine there are cases

W. Curtis Preston: I will, I will say that there are those who

disagree with me and feel that.

We wanna get it out of the cloud for backup reasons, right?

That, that, that the cloud is somehow inherently untrustable, for example.

And so we're gonna pull it out of the cloud and, and store

it in the, in a data center.

And I, I just disagree with those people.

Um, you know,

we, we can, we could agree to disagree on that one,

Yeah.

Well, I'll give you

a perfect example is data sovereignty

requirements might dictate that you pull it out

and hold it locally.

W. Curtis Preston: Yeah, it might be, that might be, it might be a, a valid example

or a reason why you might want to do that.

So, well, uh, our second sad story from the

series,

so

thanks.

Thanks for helping me tell the story persona.

I am so depressed, Curtis, but hopefully it's

a learning lesson for other folks.

W. Curtis Preston: Yeah.

We don't want anyone experiencing schaudenfreude, right?

Uh, we don't want them taking joy in these misfortunes.

We want them learning, uh, from these misfortunes.

We can definitely do that.

And with that I want to thank our listeners.

Be sure to subscribe so that you don't miss an episode.

That is a wrap.