Several years ago, a small startup placed all their eggs in the Google
drive basket, and then proceeded to set that basket on fire.
With one fatal click by an admin over a million dollars.
Went up in smoke.
Today we investigate the tragic tale of Musey, a promising young
company building an innovative app who lost everything when their Google
account was wiped out instantly.
This is our second in a series called cloud disasters.
Organizations that thought their data in the cloud was protected.
Only to find out it wasn't, it was too late for them, but not for you.
Hi, I'm w Curtis Preston, AKA Mr.
Backup.
And my goal with this podcast is to make sure that something
like that never happens to you.
This is the backup wrap up.
W. Curtis Preston: Welcome to the show.
I'm your host, w Curtis Preston, and I have with me Persona Ana.
I'm doing well, Curtis.
We're talking about this concept of.
Things that people think don't need to be backed up.
And they definitely need to be backed up.
And we're zooming in and doing a, a deep dive into each of the
stories that we often talk about when we reference this, right?
So these are, uh, typically cloud systems, you know, cloud services that people
thought that were backed up and then.
Uh, when they accidentally screwed something up, they
found out they weren't
Prasanna Malaiyandi: They were relying on it for their business and for their
livelihood and bad things happen.
W. Curtis Preston: And sometimes, many times those things are like attacks.
Uh, you know, you know, things like that.
In this case though, this, uh, one is, you know, it's a reminder that.
We spent a lot of time on the podcast, talking especially lately
about cyber attacks and things like that and ransomware, and
we don't talk that much about equipment failure because typically, uh, you
know, thanks to the advent of raid and erasure coding and things like that.
Generally, that's not the reason people restore data.
The interesting thing about this story, it that I, one interesting thing that
I found out, or that I noticed was that it was only covered by the register.
So, uh, thanks to the folks over there.
I, I don't even know.
How they, uh, you know, found it, somebody sent it to them
and then nobody else covered it.
Which is interesting 'cause I looked around, I couldn't
find any coverage of it.
Prasanna Malaiyandi: So this is about a company called Musi, which also I found
out, goes by the name of Moss, M-O-S-S-S.
They do have a Facebook page.
That you can find.
But They were a company that was building an interior design app that would be
used by influencers, home decorators and other folks to sort of showcase
what you see on the inside of a house.
And, um, basically somehow someone went and deleted their entire G
Suite account for the entire company.
Which contained all of their data.
And from what I could tell, they weren't doing backups.
And that's kind of where things start now.
They try to get the data back, and I think we'll talk about that in a
little bit about what they did to try to get their data back from Google.
But yeah, like Curtis says, it's not a happy ending.
W. Curtis Preston: Yeah.
It appears that it was their admin, right?
The idea that I remember hearing was that they were trying to delete a test account.
I.
But instead of deleting a test account, they deleted the production account.
And we need to be very specific here.
They didn't delete a user, right?
You know, in, in Google Drive or Google Workspace, you could have
users underneath your account.
They deleted The entire account
Prasanna Malaiyandi: The organization's account.
W. Curtis Preston: Yeah.
The, yeah, the, the entire organization, right.
According to their court filing.
They did immediately contact Google and say, Hey, this thing happened.
And I'm gonna say, sadly, the Google, uh, support rep said,
I'll get right on it, rather than
saying, yeah, you're screwed.
Prasanna Malaiyandi: Not my problem.
W. Curtis Preston: yeah, not my problem.
Yeah, you're screwed.
You should have backed that up.
Uh, where's your backups?
Uh, what the, uh, the Google person allegedly said was, we'll get right on it.
Meanwhile, they're out there desperately trying to reach someone at Google for
multiple days, even multiple weeks.
Meanwhile, they're not able to contact, their customers are not able to, you know,
access any of their intellectual property.
So apparently they had built.
This entire app, uh, and everything was stored in G Drive and apparently they
weren't, uh, you know, I'm just saying based on what happened, it does not
appear that they were synchronizing.
'cause you could synchronize, uh, Google Drive down to your, uh, you know, desktop.
And if they had deleted the account up there, the desktop copy
probably would have, uh, remained.
But they didn't do that because once they had deleted the, um, you know, this.
The, the cloud version of it, they had nothing.
Right?
So they had nothing in their, of their intellectual property that they
had built, uh, that they had spent a million and a half dollars on, right?
That, that was in, in the court filings, that they had spent a million and a
half dollars on this company so far.
And they also mentioned that they weren't able to contact their customers,
they weren't able to do business.
And there is, uh, they said a few weeks later.
I feel like saying
a few moments later.
Prasanna Malaiyandi: I now have that in your head, don't I?
W. Curtis Preston: it's your fault.
Um, go Google did get back to them and say, uh, they can't get their data back.
I think the weirdest part of the story that you might want to talk about is this.
They're saying that
someone at Google, and I don't see any evidence of this, but they're
saying, someone at Google said something along the lines of, maybe you
could get the data back with a court
filing.
Did you see
anything about that?
Prasanna Malaiyandi: I didn't see anything in the articles about
that, but that's just weird.
the Only time I could see that happening is potentially if they
had kept some of that data because of legal hold or some legal reason.
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: Or if for some reason they had a copy somewhere else,
like I could imagine in the case of doing like a disaster recovery copy,
maybe they had that copy somewhere else and you can get the data back.
Right.
It's not their obligation.
It's like a best effort.
I.
W. Curtis Preston: Right.
I agree with both scenarios.
I like that first one.
The second one, you know, we talk a lot about how many of these service
providers, they do have a backup, as I met quotes in the air of like the
entire data center that they can use.
To bring the data center back if they screw it up.
Right.
Um, but the, it's not built for you.
I give an example of Microsoft 365 says that they have a delayed copy of
exchange, uh, that can be used if they screw up the entire exchange environment.
But when I asked them directly as a Microsoft 365 customer,
can I use that delayed copy of exchange to restore my environment?
And they just said, no.
Right.
So maybe that's a scenario where somebody might have been alluding to it.
I don't know if that person was, uh, you know, um,
Prasanna Malaiyandi: and,
W. Curtis Preston: talking outta school.
Right.
Um.
Prasanna Malaiyandi: and also just along that, like if they were a billion dollar
company, I'm sure that it would've been a different result or Google may have
been more willing to jump in and help versus a smaller company like this.
W. Curtis Preston: yeah,
I'm not sure if the result would've been different.
I think maybe the activities along the way would've been the same.
Um, but the, the, the re by the way, the, where we get this, uh, this
comment about the subpoena or civil request was that was in their lawsuit.
And, and I have to say, I feel for these guys, uh, of what happened to them.
Because when I look at the filing, like you really have to see it to, like, you
have to literally lay your eyes on and we'll put a link to it in the show notes.
And what you see is a good portion of it is typed up and sort of like normal.
And then there's all these like handwritten notes all over it.
Like at the very beginning there's this like star.
And it says, serving a civil request to have access to our data
restored or provided as outlined a document for serving civil subpoenas
and civil requests given to us.
Emailed it says by Google.
Um, IE Google sent us instructions to get our data back.
Um, you know, and then.
They just sort of lay out this story in this lawsuit, which they say in the
lawsuit, we're not suing to get money.
We're suing to get our data back.
They basically believed that Google had a copy of their data and Google, you know,
clearly did not have a copy of their data.
Prasanna Malaiyandi: Yeah, and just looking at the filing, I
don't think a lawyer wrote that,
W. Curtis Preston: No.
Oh yeah.
Thanks for bringing that up.
It is actually listed as a per se lawsuit,
which means that they filed
it themselves.
Prasanna Malaiyandi: Yeah.
And just like, yeah, with the handwriting and everything else, you could tell
that it was an individual who was not an attorney writing and typing.
And I know that would, like you mentioned Curtis, they wanted their data back and
they were mentioning sort of what type of data they had stored in Google Drive.
Right?
It was, uh, their ip, ui, ux, mockups, user research.
And a whole bunch of other things, algorithm decisions, how they built it.
But the other thing is they were like, yeah, in our emails, we also had medical
records about someone's nephew who was going undergoing some treatment
and they needed access to their emails
W. Curtis Preston: Right.
That was the weirdest part, I think was just reading that they had intermixed.
I, you know what?
I don't know.
That just made no, that part made no sense to me.
Like it's not like they stored, it's not like this was the only copy of their.
Their nephew's medical records, like it was their only copy of
the nephew's medical records.
I, I thought that was very odd.
But the thing in the filing is that this was a filing in the federal.
US District Court for the Northern District of California, by the way, this
was in, um, the Bay Area and they, um, and so you have to give reason why the
federal court would be the jurisdiction, and they listed multiple reasons for that.
Um, but.
What we do know from the dockets, I was able to find that the lawsuit
was withdrawn, basically just
under two weeks later,
Prasanna Malaiyandi: yeah.
That's suspicious.
Yeah.
Well, I think one thing we should also mention is this
happened in what year, Curtis?
W. Curtis Preston: uh, 2019.
Prasanna Malaiyandi: Okay.
Yeah, so this was a while ago.
The interesting thing, and this is all just this is all just,
W. Curtis Preston: Yeah, we're hypothesizing, but I did my best to
try to find out, uh, everything I could, but this is our best effort
to, to figure out what happened.
Prasanna Malaiyandi: yeah.
And I think what, like just given the timeframes, and it was two weeks
later after it was filed, either they.
Decided, or they maybe got legal counsel and talked to someone, they
were like, yeah, you're screwed.
Based on Google's response.
Or Google might have come after them and been like, Hey, if you really
wanna go through this, you're going to fail and then you're gonna have to
pay all our court fees, attorney fees,
W. Curtis Preston: Right,
Prasanna Malaiyandi: in which case that would be like, I could, I can't even
imagine a Google attorney and how much that would cost because it would be
external counsel and everything else.
So they probably were like, yeah, it's not worth it.
Or maybe they talked to someone on Google who said, look.
We understand this is tough luck, but this is what it is.
And so maybe they had some sense knocked into them and they were like, yeah, this
doesn't make sense to move forward with.
It sucks, I feel for them, but there isn't much that Google
could really do at that point.
W. Curtis Preston: Yeah.
What was another sort of interesting thing about the story?
I, I just, I'm not sure why the dates don't line up.
In the filing, uh, the, the lawsuit filing, they said that the deletion
happened on June 8th, 2019.
The domain went up for sale on June 6th, 2019.
Prasanna Malaiyandi: Mm.
W. Curtis Preston: The reason why I say that is, uh, the internet
archive found that right, the, the latest update for the website.
Uh, from June from 2019 was that it went, for se it said, domain available
for purchase, uh, June 6th, 2019, which would suggest that the deletion
happened on June 5th or June 6th.
But, um, I, I
thought that was to a little bit strange.
Prasanna Malaiyandi: I don't mean to throw shade at the company, but it could also
be possible that the company just realized it's not feasible, what they're trying
to do in the timeframes they're given and ran outta cash and we're like, yeah.
, the one interesting thing is the fact that they're completely gone from the web.
Right.
There's no trace of 'em on the internet anywhere.
Right.
It's just really weird.
Like you can always find traces of a company that's gone bust somewhere.
W. Curtis Preston: Right.
yeah.
Prasanna Malaiyandi: like you mentioned, the Register is the
only company that held this
article.
W. Curtis Preston: Yeah.
And, and that, that's why what I was using the internet, uh,
archive for and, uh, the mu app.com was the, was the website and it.
Like I said, it, it showed that it went up for purchase and then it
stayed that, that same way, uh, for another, uh, like a year and a half.
And then it got bought by a Japanese anime artist, which is a really weird
domain for a Japanese anime artist.
But, uh, in 2021.
And the, the co-founder that was just, that was mentioned in the
article, uh, left the company a few months later, um, about.
I, I think they might have tried to make a go of it with what was in
their, you know, memory or something.
Tried to bring the, the, you know, again, that's just pure conjecture,
but the, the company ceased to exist, uh, in, essentially in 2020.
It's a sad, sad story from a single mistake of an admin, an entire company
goes bust because they didn't have a backup of their, uh, cloud data.
Prasanna Malaiyandi: Yep.
that's scary, right?
When you think about
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: Because it's not just the company, right?
It's all the time and effort all the employees put in, right?
All the customers who were hoping and buying products based on
what they were promised, right?
All of the things that they wanted, it's all like, poof.
Gone.
W. Curtis Preston: Yeah, yeah.
And all the, all those poor employees that were hoping probably for some,
uh, IPO, you know, payout or some kind of acquisition or whatever.
Obviously all that goes by.
Um, so when we think about this, what kind of
thoughts come to your mind?
Prasanna Malaiyandi: Well, the first, of course, and I know like
we always like to talk about this, is backup, backup, backup, right?
Don't trust that your cloud provider is going to be backing up your data.
Do your backups yourselves.
Keep a copy because that could easily have saved this company.
W. Curtis Preston: Yeah, it would've made it a non-event.
. It'd be a nice story to tell a cocktail parties about the time
we almost lost the company.
But, uh, and, and that admin would still have a job.
Right?
But unfortunately, they weren't backing it up.
And again, this was a SaaS service.
Right?
Let's just make sure we understand this.
This wasn't the same as the first story that we talked about.
That was an IAS service infrastructure as a service.
This was a software as a service.
This was Google Drive.
This was.
Google Workspace, which is now what it, which is what it's now called.
And, um, they had stored their entire world in that and then deleted it.
If this story doesn't prove to you that SaaS services aren't
backing up your data, number one.
Number two, that they have no legal obligation to bring your
data back when you lose it.
Um, I, I don't know what story would, right.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: Uh, 'cause they were sued, right?
They did a, I mean, it was a, it was a hastily filed, uh, you know, sort
of poorly written, uh, filing, but it.
It, they withdrew it for a, a couple weeks later.
Right.
Uh, we don't have any details on that, but it, it wasn't good.
Uh, the other thing is, you know, again, I mentioned it already, but
just to remember that, uh, you know, I, I, I often like to throw out
the phrase from Shakespeare, right?
There's, there's more on, there's more in heaven and earth that
is dreamt of in your philosophy.
And I, I think sometimes we focus a little bit too much lately
on cyber attacks and things.
Just realize sometimes people just do dumb stuff.
Fact, um, in fact, there's a couple of the stories that we have coming.
It's just somebody doing something dumb.
Somebody making a, a fat finger.
The title of the headline for this was, you know, admin Fat Fingers.
This company and, and basically deletes this whole company.
So just realize it's, it's human error.
Prior to hackers taking over everything, human error was the
number one reason we did restore.
Right now it's human error and direction.
I don't know how, what category you wanna put that is, it's not always hackers.
Prasanna Malaiyandi: So the next one is really around keeping personal
stuff on your corporate drive.
I know it's hard.
Everyone works remotely, right?
You have a laptop.
It's like, oh, it's too hard for me to go grab a personal device and
start using that for personal stuff.
Like who doesn't log into Gmail from their work computer, honestly, right?
Or from their phone.
And so things do intermix, but the problem is what happens on a corporate drive?
Don't expect it.
If it's your personal stuff that it will be kept forever and that it's safe, right?
Personal stuff you should really take ownership of.
W. Curtis Preston: Not to mention that by putting medical information
on a company drive, you are subjecting that company to HIPAA requirements.
Um, it, it's not something that you should be doing, uh, you know,
on your, you know, on your, uh, on, on corporate systems, right?
Just not.
Yeah.
There, there's no, not to mention, you leave the company, right?
You're laid off, you're fired.
Uh, you know, they, they have no obligation to give you that data back.
Right?
Um, the other thing is, again, that I would just wanna mention that one of the
reasons why people talk about cloud data, not needing a backup as they're like, oh,
well, there's always this recycle bin.
Some deletions aren't covered by the recycle bin.
Prasanna Malaiyandi: Deletion.
W. Curtis Preston: yeah, it sounds like account deletions.
There are things that you can do in Salesforce.
I was looking at a company that is specializing in backing up intra, which
is what Microsoft Azure AD is now called.
They were listing a handful of things that.
When you mess them up in intra, they're not covered by the recycle bin.
And this is another example, is that there are a number of things, again, there
are more things in heaven on earth than you are dreamt of, uh, in Salesforce.
I know for example, when you change a record, not when you delete
a record, but when you change a record, uh, that's not covered by
the recycle bin, they don't put the old version of That record.
Prasanna Malaiyandi: That would be crazy if they did
that
Could you imagine?
I think this just goes to the point that backups are critical.
You can't trust what, that, there aren't going to ever be
these types of events, right?
That just go and don't have a recycle bin or maybe even the product doesn't
even have a recycle bin or your admin disabled the recycle bin.
What are you gonna do at that point?
W. Curtis Preston: Or the a or, or the hacker is able to clear out the
recycle bin because recycle bins can be cleared out, uh, manually.
Or what they can do is they can set the versions to one, you know, all
these, you know, they're, they're just, it's just a whole bunch of things.
This is why we back up.
Right.
This is, this is why it's just so frustrating to me that
there are groups of people.
Um, some of them who specialize in the platforms that we're talking
about who say, oh, well you don't really have to bag it up, um, despite.
There's a story, at least one story for every major platform we talk
about, and we're gonna cover those stories in subsequent episodes.
So yeah, so back up, back up, back up.
Um, and, and make sure that that backup is stored somewhere other than
the thing we're talking about, right?
Because that's the other problem with the recycle bin,
is that it's stored inside the thing, right?
Um,
or
versioning is stored inside the
thing.
Prasanna Malaiyandi: Or in this case, if they had backed it up to
say a Google Drive under the same account, that wouldn't have saved them
W. Curtis Preston: That wouldn't have saved them.
Yeah.
You need to back it up to another account, um, uh, you know, to another account.
You know, if we're talking about cloud to cloud backups, just make sure it's
backed up to another account that has different, you know, authentication
authorization and has really, really tight lease privilege turned on.
Uh.
And a different region perhaps.
Right?
Um, you know, there are other services to pull it outta the cloud if that's what
you wanna do, but just make sure you're not leaving it all on the same place.
Um, there are services to back up Google Drive.
There are services to back up all of the things that we're talking about.
And, uh, that would be my favorite way.
If we're talking about how to back up the cloud, my favorite way to back up the
cloud would be to back it up to the cloud, do cloud to cloud backups, but back it
up to another service that you're paying
to do this.
Right, right.
Um,
Prasanna Malaiyandi: I'm laugh you, you know why I'm laughing
W. Curtis Preston: Why?
Prasanna Malaiyandi: Because you bought into the cloud service because you didn't
wanna manage equipment, infrastructure, patching, all of that stuff.
And if you went and you're like, Hey, I'm gonna go deploy an on-premises
backup solution so I can pull my data down from the cloud because I wanna
manage it and I want to do all the other stuff, that just seems kind of backwards.
W. Curtis Preston: Yeah, I know, I know there's at least one major
company that does it this way.
Uh, and I'm not saying it's necessarily wrong.
It just doesn't,
it doesn't seem right.
Right.
Prasanna Malaiyandi: Yeah, unless you have a very specific use case for
needing that particular solution, I could imagine there are cases
W. Curtis Preston: I will, I will say that there are those who
disagree with me and feel that.
We wanna get it out of the cloud for backup reasons, right?
That, that, that the cloud is somehow inherently untrustable, for example.
And so we're gonna pull it out of the cloud and, and store
it in the, in a data center.
And I, I just disagree with those people.
Um, you know,
we, we can, we could agree to disagree on that one,
Prasanna Malaiyandi: Yeah.
Well, I'll give you
a perfect example is data sovereignty
requirements might dictate that you pull it out
and hold it locally.
W. Curtis Preston: Yeah, it might be, that might be, it might be a, a valid example
or a reason why you might want to do that.
So, well, uh, our second sad story from the
series,
so
thanks.
Thanks for helping me tell the story persona.
Prasanna Malaiyandi: I am so depressed, Curtis, but hopefully it's
a learning lesson for other folks.
W. Curtis Preston: Yeah.
We don't want anyone experiencing schaudenfreude, right?
Uh, we don't want them taking joy in these misfortunes.
We want them learning, uh, from these misfortunes.
We can definitely do that.
And with that I want to thank our listeners.
Be sure to subscribe so that you don't miss an episode.
That is a wrap.