what happens when a SAS giant accidentally grants modify all permissions to
every user in every customer org.
Spoiler alert.
It's not pretty.
Join me and my co-host as we explore the fallout from this
real world cloud catastrophe.
We'll discuss how Salesforce scrambled to restore proper permissions.
The frustration felt by locked out customers and the crucial role
third-party backups could have played.
You'll learn why relying solely on your SAS vendors, recovery capabilities
might leave you high and dry and how having your own backups can save the day
when things go sideways in the cloud.
If this is your first time listening.
Hi, I'm W.
Curtis Preston also known as Mr.
Backup.
My career in backup began over 30 years ago when my backups failed
and my company was unable to restore their purchasing database.
I vowed that would never again happen to me.
And it's my goal to do the same for you.
I want to turn you the unappreciated backup admin.
Into a cyber recovery hero.
This is the backup wrap up.
W. Curtis Preston: Welcome to the show.
I'm your host, W.
Curtis Preston, AKA, Mr.
Backup.
And with me, I have my election primary worker anxiety consultant
Prasanna Malaiyandi: How's it going, Curtis?
Yeah.
You're doing, it's that time of year, or I guess every couple years
where the election happens and
W. Curtis Preston: no, there's no word for like two years.
Is there?
It's a, it's a, that time of biannual, I
Prasanna Malaiyandi: It's weird that bi counts as both, like half as well as two
W. Curtis Preston: Yeah.
Don't get, don't get me started on English.
All right.
Um, semi or bi, right?
So I will once again be an election worker for the upcoming
California Presidential primary, and tomorrow is to set up day.
This year I am running an 11 day vote site.
Prasanna Malaiyandi: Wow.
Crazy.
You'll be a busy, busy man.
W. Curtis Preston: I will be,
well, that's a different point because ask me how many people I
think I'll see in the first 10 days.
Prasanna Malaiyandi: I am gonna say 21.
W. Curtis Preston: Yeah, I think that might be high.
It, uh, because what happens is everybody comes on election day.
I mean, I'm glad we have early voting, right?
I, I, I really am.
I, I believe in access and, and I even like the 11 day sites because
there are some people that have jobs that just really mess up a week.
So four days isn't just.
It just isn't enough for some
people.
I, I believe in access to elections.
It, it's just that, you know, everybody comes on election day and then we go,
yeah, well we've been here for 11 days.
And they're like, what?
Prasanna Malaiyandi: I would normally go vote, like when they used to
have the neighborhood polling place.
I used to go vote in person on the day of the election.
I wouldn't go ahead of time.
I would just go like early in the morning and I'd just go be done and come back.
W. Curtis Preston: Yeah.
Uh, so we're there, uh, by, and by the time this episode airs, the
primary Will Al will already be over.
I love participating in the process and I will answer any.
All election questions that anybody has, and I'll say the same thing
that I say every time this comes up.
If you have any doubt as to the integrity of your election process,
do one of, or both of two things.
One, volunteer as an election worker.
That is you.
You get so much insight into the process and how it works.
Number two, be an observer.
You
are legally allowed to observe every single portion of you
the election process, right?
You know where, where the votes are initially cast, where they are received,
how they are counted, you can view the incredibly boring way in which the,
there is this, well, not just the accounting room,
but there is this process.
The most boring part of the process is when they do a 1% manual count.
So they take 1% of the cartons.
That, uh, you know, the, the ballot cartons that, that, that are gonna contain
anywhere from 20 to 200 votes, you know, and they sit there at a table with like
four people and they read it one by one, and then those four people tally it up.
And then they compare numbers and the numbers all have to match and they have
to match what the machine said to box it.
My wife has done that process, but, oh my lord.
It's like, it's like watching paint
Prasanna Malaiyandi: I, I, I was just gonna think her, you're
probably gonna be like, okay.
One ballot, two ballot, three ballots, like counting sheep.
W. Curtis Preston: Yeah.
Yeah.
But anyway, uh, so, you know, so I'm excited to participate in the process, but
I, I do have a certain amount of anxiety as I was alluding to a certain amount of
anxiety because there are people, right.
People who need people
to yell at.
Prasanna Malaiyandi: I think the other thing to note is it's not like they're
just throwing you to the wolves, right?
So you go through training, right?
W. Curtis Preston: go through a lot of training.
Right, right.
Yeah,
Yeah.
And, and, and you know, and I'm experienced The other
people are experienced.
Yeah.
There's no, yeah, it's not wolf throwing and,
and, and they have lots of support.
Right.
So there, there, there's a, there's a, a phone number, the poll worker hotline.
Which I have
saved as a contact in my phone.
I just have them
as, I have them as R-O-V-R-O-V,
uh, the Registrar of Voters.
Right.
That's their first and last name, and I just call 'em, you know?
Prasanna Malaiyandi: yeah, but you don't need to have the experience that you did
because when you first started all this, you didn't have that experience either.
Right?
You were
W. Curtis Preston: right, Yeah, yeah,
Prasanna Malaiyandi: right, who was learning the ropes as well, and so you
W. Curtis Preston: Yeah.
So you're, you're speaking to the people that I'm saying participate?
Yes.
I I think you should participate.
Um, and, uh, by and large it is a very easygoing, peaceful process.
Every once in a while there are some challenging people
and, uh, you just pass those over to your more experienced
site manager, which is me.
So it's a little bit of anxiety.
But, um, anyway, let's get on to what you know.
This is part of this series that we have, uh, called Cloud Disasters, and
this is yet another cloud disaster.
The cloud is just computers that somebody else is running, and in this case it, it's
a database that someone else is running.
Prasanna Malaiyandi: Yep.
W. Curtis Preston: And we get people all the time that wanna argue, oh, well
I don't need to back up Salesforce.
I don't need to back up Microsoft 365.
It, it is part of the service.
It's not
right.
Um, it's just not, it's not in your, uh, service description to, to go look at it.
If you don't believe me
Prasanna Malaiyandi: And even if it was part of your service description,
you don't know if you could trust them
W. Curtis Preston: Uh,
yes.
Even if it was Yeah.
You know, and that, that, that's one of the stories.
We're gonna get to
that, right?
The, um, the O-O-O-O-V-H-O-V-H,
right?
Yeah.
The OVH story proofs.
Uh, we have a story, literally every comment that we, we don't make this,
we don't just make this stuff up.
You can't make this stuff up, right?
We, we have stories behind every one of the recommendations that
we make, and this series is about telling these stories and this.
Is a good one.
Do you wanna, do you wanna sort of, uh, look, first off, everybody should
know what Salesforce is, right?
Um, but you
know, just, just in case you don't, Salesforce is, I
think the OG SaaS app, right?
I'm sure there was another before, but they were the, the first one
that really took off their actual phone number is one 800 no software.
I don't know if you, if you knew
Prasanna Malaiyandi: No, I didn't know
W. Curtis Preston: um, yeah, I, that's, that.
I remember that from, from many days gone by.
And they are A-C-R-M-A customer relationship management software.
Right.
And I remember using one of their competitors back in the day
when I had my own company and.
Uh, oh my Lord.
Is it so much easier to use Salesforce, especially when
you have multiple salespeople
that are, um, you know, all interacting with a variety of leads?
And again, to preface this story, I'm gonna explain how this works in a big org.
I've been a salesperson and most salespeople are, uh, commissioned.
I.
And they are, they're gonna attack any lead that you give them.
And they, because they're commissioned to do so,
and the only thing that prevents them, you know, you, you give these leads to
this person, these leads to this person.
And the only thing that prevents Steve from jumping all over
Janet's leads is permissions.
In a large database like Salesforce, you assign permissions, you create
groups of leads, and you give permission to Steve or to a certain team.
There's different ways to do it, but you divvy out these leads.
By way of permissions.
Right?
And that would, that prevents Steve from going over and, you know,
stomping all over, uh, Janet's leads
and, um, uh, but then something happened.
So why don't you, so when, when, when did this happen and what happened?
Prasanna Malaiyandi: So it was back in 2019.
So it was a ways ago, and what ended up happening is Salesforce ran a script.
And what the script did is it allowed everyone in an organization
to be able to modify and access all records in that organization.
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: And so in your example of Steve and Janet, Steve could
now see everything Janet had could go stomp on it and be like, Hey, by the way,
Janet, you're actually not as far along as you said you were, or change a dollar
amount of the lead and other things like
W. Curtis Preston: Or Steve could also delete all of
Janet's leads, if that's what, if
Steve is a very bad person, if he wanted to go delete all her leads or just
delete, you know, uh, like any interaction that she had with the clients, right?
If
you were.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: A nefarious person.
Yeah, the notes, right?
If you were a nefarious person, you could have done a lot of damage to
other people in the organization, uh, or you could steal their leads.
Just
reassign those leads
to you.
Um,
Prasanna Malaiyandi: or you could be doing just some random housekeeping,
innocuous housekeeping stuff like, Hey, I'm just gonna go clear out all my old
leads older than like two years old that I haven't touched and realize that
you might be stomping on Janet's leads.
Yeah.
W. Curtis Preston: Yeah, exactly.
Um, so Salesforce, interestingly enough, Salesforce, according to, and we're
gonna put links to this, we have the, a link to the original post that was made
by Salesforce, as well as a link to a follow-up post that they made several
months later as a, uh, postmortem.
What they learned, but what we know from their posts is they did not
notice that they had done this.
A customer called and said, Hey, this is odd.
Everybody apparently can modify everybody's leads.
Right?
I.
And, and by the way, just, just to put a, a point on that, it, it's
sort of like in file permissions.
If you have modified permissions, you have all of the others,
right?
Uh, you have read, write, you know, modified delete.
Right?
So, um, the, and by the way, they, they appear, it, it appears that
they only had this privilege.
To records.
They didn't have the, they at least didn't grant this permission to be able to
modify things like configurations, right?
So they couldn't go in and basically delete Janet or change Janet's permissions
as a person, as a user, but they could go in and access and do everything.
To her data.
Right?
So it's important to, to just mention that.
Anyway, so they didn't notice that they did this.
Customers called in and then they very quickly, uh, they had,
you know, what I would call, you know, an oh shit moment, right?
And they're like, holy cow.
That maintenance script that we ran, it appears that it did.
Um, you know.
Prasanna Malaiyandi: A
W. Curtis Preston: Uh, A lot more than we had intended to do, and they
realized they had really messed up.
And so the first thing they did was just say, okay, just shut down everything.
Right.
Um, which, which I think was probably the best thing they could do at the
time, even though that would, of course immediately at Cal, all their customers.
Prasanna Malaiyandi: Yeah.
Well I do wonder if I agree that that's sort of like the nuclear option, right?
But I do wonder if maybe they could have at least, uh, I guess I was just thinking
could they have removed the modify all and just given like view only, but then
some orgs, it still might have been bad to allow Steve to see Janet's leads
W. Curtis Preston: well, I, I, the problem was, I, I don't think, you
know, based on the, the, the records and stuff that we have, I'm not sure
they even know the, knew the extent
of the damage that they had caused
Prasanna Malaiyandi: it's like a ransomware attack where
you just pulled a network.
Cable.
W. Curtis Preston: Exactly.
Yeah.
Um, there's a man, there's a great scene in, in, you know, one of my
favorite shows, alias, where, uh, he goes running into the server room
and he literally is like flipping
power switches, you know, they're downloading all the files
up the server and he is just flipping all the power switches.
Uh, it, it was pretty much like that.
And so they, they, um.
Uh, that was their first response.
And then, uh, then what did it do?
Prasanna Malaiyandi: So then after that, well, so that shut
it down, but it was only for the organizations that were impacted.
And
W. Curtis Preston: by by the way, I just want to interject.
What we now know is that the, IM, the organizations that were impacted
was any user or any organization that had used Pardot, which is their.
Marketing automation.
, it's the Salesforce equivalent to Marketo.
Right?
So this is the thing that's gonna email your customers
and things like that, right?
Um, so anyone who had ever used or was currently using Pardot, that
turns out to be, who was impacted?
Prasanna Malaiyandi: So after that, let's see, what did they do?
W. Curtis Preston: I'm not exactly sure exactly when, what happened
'cause we don't exactly have a timeline 'cause there's multiple
posts and multiple articles and,
you know, we even have a, there's a stack exchange thread that we could
follow during this, uh, uh, outage.
Prasanna Malaiyandi: Yeah, even though we don't know that timeline, Curtis,
I think the one thing we can just sort of take away from all these articles
is they did try to fix it themselves.
They weren't like, Hey users, we have nothing to do.
Good luck.
Go pound sand.
Right?
It looks like they were internally trying to do things to fix this and
looking at various technologies or resources that they might have had, but.
As we know, that takes time.
Right?
And as a user, at that time, they weren't really forthcoming about, Hey, we're
trying things internally either, right?
They were, they didn't wanna give users hope.
W. Curtis Preston: Yeah, well, I'm not sure if they were communicating,
you know, it does show that they mass emailed some users.
All we have access to is what they said publicly and publicly.
Again, around this time they had this post where they said, Hey, we messed up.
We gave modify all.
And here are a couple of ways that you can potentially fix this if
you want to fix this yourself.
And, 'cause that was obviously a question that people asked is, Hey,
can I, can I fix this on my account so that I can get my account back online?
And the you, you know, that the response just really infuriates me
because I gave them two options.
And they had to do with the sandbox.
They basically said, if, you know, if you made a sandbox, which is something
that you could do regularly, which if you don't know a sandbox is, uh,
you know, it's a place where you can play with your data and, and,
mess
Prasanna Malaiyandi: a clone,
W. Curtis Preston: then It's fine.
It's what?
Prasanna Malaiyandi: It's like a clone.
W. Curtis Preston: It is a, yeah, it's a clone that you can
automatically make with Salesforce.
You, you know, it's, some people actually treat it like a backup.
I don't because, uh, it's all in the same place, but, so it doesn't
conform to the 3, 2, 1 rule.
But it, um, the, but they said, you know, if you happen to have a recent backup.
You could go and get their permissions from there.
'cause remember, they're, they don't have to restore the data.
They didn't mess up the data.
They messed up the permissions of the, of the data of the
Prasanna Malaiyandi: And just to correct you, you meant
to say sandbox not backup in
that statement, correct?
W. Curtis Preston: oh, did I, did
I say, did I call
Ouch.
You are correct.
I meant to say samples.
So if they, if they're saying if you happen to have a recent
sandbox, copy of your instance.
It's recent.
This is the problem.
It needs to be recent enough to have the user's permissions to match your current
permissions, but it can't be too recent because if it was too recent, in other
words, if it was made in the last few hours, it's just a backup of our mistake.
Right?
It's just, it is just a copy of our mistake.
So they were saying that what, what infuriates me persona is.
Not once did in, in, in, in in any of the external, uh,
stuff that Salesforce put out.
Not once did they say, by the way, if per chance you did what Curtis tells you to do
and actually backed up your data.
W you could just go and, and easily restore the, basically
the, the user's table is
what, you know, for those of you that don't know, you know, Salesforce has,
you know, all these different tables.
It's like any other database.
They call them objects.
I.
So it would've been the user's object, uh, is what I would assume was
that.
needed to be restored and you could restore the, just restore your user's
object to any time before, you know, 12:35 AM on May 17th, 2019, and you'll
be
Prasanna Malaiyandi: you'd be fine.
W. Curtis Preston: But they never said that.
I, I just.
That I remember posting a blog at the time that basically said Salesforce
proves they know nothing about backup,
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: right?
Because it's like they never once suggested they, they sort of thought
of the, of the sandbox as a backup and never thought that anybody might
want to have backed up their, their
Prasanna Malaiyandi: Of course not.
'cause who does backups of Salesforce?
You don't need no stinking backups.
W. Curtis Preston: Nice.
Nice.
Uh, all right.
Uh, um, brownie points or extra points, if you can tell me what
movie that is referring to.
And I mean, the original movie, not the second movie that,
Prasanna Malaiyandi: Not the Rob Schneider one
W. Curtis Preston: oh, that would be a third movie.
Prasanna Malaiyandi: because of what that was.
Water Boy
W. Curtis Preston: What's it?
I, no.
Yeah, I don't know.
I'm referring to the original movie starring Humphrey Bogart.
It's called The Treasure of Sierra Madre.
Prasanna Malaiyandi: Oh,
W. Curtis Preston: The, the badges.
Yeah.
We, we don't,
we don't know nothing about no stinking badges.
Great greatvine.
Yeah.
Anyway, um, the.
Why don't you read this, uh, this part about the, the stack exchange part there,
there's a, there's an interesting comment on the stack exchange, uh, thread there.
Do you see that?
Prasanna Malaiyandi: Is this is the, that's not even the worst that is going.
Yeah.
So, yeah, so on Stack Exchange, one of the users commented.
That's not even the worst that is going on.
Apparently in an attempt to fix this, they remove the modify access all data
from all admin profiles in some instances, including standard and custom profiles.
W. Curtis Preston: So.
so they removed the, they removed the permission even from the
people that needed the permission.
Prasanna Malaiyandi: Which basically means do you end up with a read-only
copy of your data while they're trying to figure things out.
Well, because I could see that they don't want you to change anything
because it might not let them restore things back to a good state later on.
W. Curtis Preston: Yeah.
You know, in that, in that Stack Exchange thread, uh, which we'll put a link to
it in, in the show notes, in that stack Exchange thread, they were saying that as
this was going on, Salesforce was saying, please don't try to fix this yourself.
We, we got it.
Like we're gonna, we think, we think we can fix it.
Um, so let's talk about some of the things that they did.
Uh, you know, in the backend, and by the way, this is all news to me.
This was not covered in the original stories that covered this.
This was, you know, in classic, you know, news stuff.
They only covered that initial explosion.
No one
covers the, the remediation and everything afterwards,
especially given that this was,
Prasanna Malaiyandi: It's a not sexy stuff, right?
W. Curtis Preston: It's not as sexy.
Yeah.
If it
bleeds, it leads.
And this is, you know,
uh, so about seven months later, so this was May and in January
of, of 2020, um, oh, I just, I just realized like timeframe.
You, like this is a group of people that are writing, they're
just, they're just, they're just writing about this, this problem
that happened in, in, in last year.
Little did they know in two months that the world was gonna fall
Prasanna Malaiyandi: Yeah, so the first thing that they.
Yeah, so the first thing they attempted to do was run a backout script.
They were like, Hey, we had a script that ran.
We should just go undo it.
Which in my mind makes total sense, right?
You're like, Hey, the script did something.
Let me just go undo everything that I just did.
W. Curtis Preston: and it looks like the, and it looks like the script had
automatically, it basically, it, it made a backup of the permissions that.
It was supposed to change.
Right.
Which is what a script should
do.
right before I go do a bunch of stuff.
Yeah.
So why
didn't that work?
Prasanna Malaiyandi: the problem is, it did not, however, record
things that it had done that it wasn't supposed to have done.
W. Curtis Preston: Yeah.
So the, so the backup, the backup line didn't have the wild, the, the
asterisk in it and the, and the execution line did,
uh, oh.
That's, I, you know, I'm sorry.
. I'm having, uh, shun Freud at the expense of this poor person
who, you know, according to the, to the aftermath and the report.
You know, they, they said, did we follow our process?
They did follow their processes.
They did, um, uh, or most of their processes.
What they didn't do when they did the initial script run that, that
did all of this, they tested it.
But what they didn't do was they didn't do a phased rollout.
Of the script.
They were like,
we got it You know, this guy wrote it, this person, uh, you know, uh,
you know, sanctioned the script.
We've tested the script, the script runs, run it everywhere.
Prasanna Malaiyandi: Did they do it on a Friday evening
W. Curtis Preston: Let's see.
17th, 2019.
It was a Friday.
They did it on Friday.
Oh, those poor guys, you know, they didn't have a weekend.
So they did have a variety of technologies that they could possibly use.
To solve this problem.
And one of them was that they have a Dr instance.
We talk about this with 365 as well, because we know that 365
has a rolling, um, you know, uh, replicated copy of their system, right?
So, so this is, again, this is a quote from their report.
A site switch to a DR instance was not an option since the purpose of
the DR option is to replicate it near real time the state of the primary
site, which meant that the inadvertent
permission change would've been replicated in near real time to the redundant site.
We talk about this, don't we?
Prasanna Malaiyandi: Yeah.
Dr.
Isn't intended to be a backup.
That is not its purpose.
W. Curtis Preston: yeah.
Well, I would say.
Replication, but like,
because we've talked about this in previous episode replication,
which is what they're using by itself is not, is not a backup.
Right.
Because it, you know, you know, as I, as I jokingly say, it
makes a mistakes more efficient.
Right.
And that's what they, that's
what they're saying here is, yeah, it would've, they're like, well, we knew.
We knew we couldn't use that.
, and then there was a, another thing that they talked about called flashback.
You wanna talk about that?
Prasanna Malaiyandi: yeah.
So flashback.
Their database vendor has this technology which allows you to
sort of keep a point in time of the database automatically in the system.
And so you could use that to restore from in case something happens.
Now, the one challenge though is they did look to see can we go use our
flashback area to restore the database,
get everything up and running again.
Unfortunately, they only kept six hours.
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: Right?
And so that's the furthest back they could run because that makes sense.
You have some sort of database corruption or you accidentally drop
a table, you just need to go back a couple seconds, you're good to go.
W. Curtis Preston: Yeah, they don't say it, but it looks like they're
referring to an Oracle feature.
Uh, and
there are a series of features there, but it's kind of like
the, the snapshot thing, right?
You can go back to when you took the snapshot, but if you, you know,
if you want to go longer than that.
You don't have, because there's a, there's a window that they, that
they specify and six hours must
have been the window.
And there were pa they were past the window by the time they, they optioned.
So it's interesting the, the option that they chose to.
To figure out what permissions were, what to be able to, you know, to restore them.
'cause the problem, once you've granted modify all, well, how
do you know what to go back to?
Right.
You, you can't just say read all right.
Uh, 'cause even that, right.
You know.
Um, so the, what they started doing is essentially log scraping, right?
They call it log mining.
To, to, to look at customers instances of.
To see what permissions in the logs that these things were set
to, and that's what they began.
And they started going through and in, in the story, in the, in the
postmortem, there are these series of.
We think we did this, we think we did that.
You know, we've, we've restored all these instances.
There were, there were dozens of instances that were affected and they're
like, we think this instance is good.
This instance is good.
So if you're on that instance, then you're good.
But even when all that was done, there were still customers
that were not restored.
and and they said, you know, we're working with you.
And then they gave instructions on how to basically manually fix this and
Prasanna Malaiyandi: Which if you have thousands of salespeople
W. Curtis Preston: Right.
Um, you know what would've been really helpful to those customers
in this scenario, persona.
Prasanna Malaiyandi: using another vendor.
W. Curtis Preston: Yeah, a backup, right?
right?
If they, if they had backed up the data outta Salesforce into another vendor,
so, so companies you know, that use Salesforce and other CRM products and if
they used a backup, they would've been able to fix this literally like that.
Prasanna Malaiyandi: Uh, well, a backup intended for Salesforce
W. Curtis Preston: Yeah, so back well backup of Salesforce,
intended for Salesforce.
I'm not sure what
other, what other method you
Prasanna Malaiyandi: No.
So I was thinking like someone could have done a backup by
just dumping out the objects.
Right.
And I don't know if that would've necessarily kept all the permissions
as well if they hadn't backed up the, or dumped the user table as well.
W. Curtis Preston: Well, if you do, if you can do a manual backup of Salesforce,
it basically gives you all the objects.
The only problem with every manual backup is you have to manually do it.
You have to do it every
once in a while, which means, I don't know how often you're gonna be doing it.
It might be once a week.
It still would be better than nothing,
Prasanna Malaiyandi: Nothing.
Yeah.
W. Curtis Preston: you basically, it just means your permissions
would be a week old, which in this case would be a good thing.
Um, but it just drives home again to me that there are, you know, you, you've
heard me say this, I think I said it in the last episode of like, there, there
is more in heaven and earth, Horatio than dreamt of in your philosophy, there are
things that can happen to you in the cloud.
The cloud isn't magic.
There are things that, that you're not gonna
anticipate.
Prasanna Malaiyandi: be magical.
W. Curtis Preston: Yeah, it is just as magical as actual magic, which
is an illusion.
So I love magic by the way.
I'm a big fan.
Like, I went and saw, like, I saw David Copperfield in Vegas, and I was amazed.
I, I loved it, but inside I knew it was all just an illusion.
Prasanna Malaiyandi: Illusion.
Yep.
W. Curtis Preston: Um, so, you know, my advice isn't so much
to Salesforce, Salesforce.
Did as much as they could do in this scenario.
It seems like they were communicating with their users.
They had a status page like we, like we tell them to do.
Um, maybe go.
Prasanna Malaiyandi: I, I think the only thing I would fault them for is
maybe a bit more communication about what they're doing internally, right.
W. Curtis Preston: again, I think they may have been doing that
just not publicly the way we were
looking.
Right.
So maybe they were
communicating
privately
'cause ' cause they said, they said in their.
In their postmortem, they're like, what?
What should we have done better?
And, um, they listed a whole bunch of things they were
doing to communicate, uh, what
was going on.
We just outside.
I was really angry at the time because all I saw was that one page,
because I saw the one page where they said, Hey, uh, sorry, um, we
just messed up all your permissions.
And so.
Can you fix it?
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: And, and by the way, we're not gonna mention backup.
I was furious at the time.
Uh, so I'm, I'm a little less furious.
But, uh, this is just another big example of why we back up, you know,
everything and why we recommend backing up cloud vendors and es
esp and especially SaaS vendors.
Prasanna Malaiyandi: Yeah,
W. Curtis Preston: Any, any
final thoughts from you from Peanut Gallery?
Prasanna Malaiyandi: I think that's the right thing.
They did everything they could and users should have backed up their data,
W. Curtis Preston: again, we're blaming the victims.
Uh oh goodness.
Prasanna Malaiyandi: could have easily been avoided though.
W. Curtis Preston: It could have, it could have back it up or give it up people.
Uh, thanks for listening.
You know, you are why we do this.
We want to turn you into Cyber recovery Heroes.
That's a wrap