what happens when a SAS giant accidentally grants modify all permissions to

every user in every customer org.

Spoiler alert.

It's not pretty.

Join me and my co-host as we explore the fallout from this

real world cloud catastrophe.

We'll discuss how Salesforce scrambled to restore proper permissions.

The frustration felt by locked out customers and the crucial role

third-party backups could have played.

You'll learn why relying solely on your SAS vendors, recovery capabilities

might leave you high and dry and how having your own backups can save the day

when things go sideways in the cloud.

If this is your first time listening.

Hi, I'm W.

Curtis Preston also known as Mr.

Backup.

My career in backup began over 30 years ago when my backups failed

and my company was unable to restore their purchasing database.

I vowed that would never again happen to me.

And it's my goal to do the same for you.

I want to turn you the unappreciated backup admin.

Into a cyber recovery hero.

This is the backup wrap up.

W. Curtis Preston: Welcome to the show.

I'm your host, W.

Curtis Preston, AKA, Mr.

Backup.

And with me, I have my election primary worker anxiety consultant

Prasanna Malaiyandi: How's it going, Curtis?

Yeah.

You're doing, it's that time of year, or I guess every couple years

where the election happens and

W. Curtis Preston: no, there's no word for like two years.

Is there?

It's a, it's a, that time of biannual, I

It's weird that bi counts as both, like half as well as two

W. Curtis Preston: Yeah.

Don't get, don't get me started on English.

All right.

Um, semi or bi, right?

So I will once again be an election worker for the upcoming

California Presidential primary, and tomorrow is to set up day.

This year I am running an 11 day vote site.

Wow.

Crazy.

You'll be a busy, busy man.

W. Curtis Preston: I will be,

well, that's a different point because ask me how many people I

think I'll see in the first 10 days.

I am gonna say 21.

W. Curtis Preston: Yeah, I think that might be high.

It, uh, because what happens is everybody comes on election day.

I mean, I'm glad we have early voting, right?

I, I, I really am.

I, I believe in access and, and I even like the 11 day sites because

there are some people that have jobs that just really mess up a week.

So four days isn't just.

It just isn't enough for some

people.

I, I believe in access to elections.

It, it's just that, you know, everybody comes on election day and then we go,

yeah, well we've been here for 11 days.

And they're like, what?

I would normally go vote, like when they used to

have the neighborhood polling place.

I used to go vote in person on the day of the election.

I wouldn't go ahead of time.

I would just go like early in the morning and I'd just go be done and come back.

W. Curtis Preston: Yeah.

Uh, so we're there, uh, by, and by the time this episode airs, the

primary Will Al will already be over.

I love participating in the process and I will answer any.

All election questions that anybody has, and I'll say the same thing

that I say every time this comes up.

If you have any doubt as to the integrity of your election process,

do one of, or both of two things.

One, volunteer as an election worker.

That is you.

You get so much insight into the process and how it works.

Number two, be an observer.

You

are legally allowed to observe every single portion of you

the election process, right?

You know where, where the votes are initially cast, where they are received,

how they are counted, you can view the incredibly boring way in which the,

there is this, well, not just the accounting room,

but there is this process.

The most boring part of the process is when they do a 1% manual count.

So they take 1% of the cartons.

That, uh, you know, the, the ballot cartons that, that, that are gonna contain

anywhere from 20 to 200 votes, you know, and they sit there at a table with like

four people and they read it one by one, and then those four people tally it up.

And then they compare numbers and the numbers all have to match and they have

to match what the machine said to box it.

My wife has done that process, but, oh my lord.

It's like, it's like watching paint

I, I, I was just gonna think her, you're

probably gonna be like, okay.

One ballot, two ballot, three ballots, like counting sheep.

W. Curtis Preston: Yeah.

Yeah.

But anyway, uh, so, you know, so I'm excited to participate in the process, but

I, I do have a certain amount of anxiety as I was alluding to a certain amount of

anxiety because there are people, right.

People who need people

to yell at.

I think the other thing to note is it's not like they're

just throwing you to the wolves, right?

So you go through training, right?

W. Curtis Preston: go through a lot of training.

Right, right.

Yeah,

Yeah.

And, and, and you know, and I'm experienced The other

people are experienced.

Yeah.

There's no, yeah, it's not wolf throwing and,

and, and they have lots of support.

Right.

So there, there, there's a, there's a, a phone number, the poll worker hotline.

Which I have

saved as a contact in my phone.

I just have them

as, I have them as R-O-V-R-O-V,

uh, the Registrar of Voters.

Right.

That's their first and last name, and I just call 'em, you know?

yeah, but you don't need to have the experience that you did

because when you first started all this, you didn't have that experience either.

Right?

You were

W. Curtis Preston: right, Yeah, yeah,

right, who was learning the ropes as well, and so you

W. Curtis Preston: Yeah.

So you're, you're speaking to the people that I'm saying participate?

Yes.

I I think you should participate.

Um, and, uh, by and large it is a very easygoing, peaceful process.

Every once in a while there are some challenging people

and, uh, you just pass those over to your more experienced

site manager, which is me.

So it's a little bit of anxiety.

But, um, anyway, let's get on to what you know.

This is part of this series that we have, uh, called Cloud Disasters, and

this is yet another cloud disaster.

The cloud is just computers that somebody else is running, and in this case it, it's

a database that someone else is running.

Yep.

W. Curtis Preston: And we get people all the time that wanna argue, oh, well

I don't need to back up Salesforce.

I don't need to back up Microsoft 365.

It, it is part of the service.

It's not

right.

Um, it's just not, it's not in your, uh, service description to, to go look at it.

If you don't believe me

And even if it was part of your service description,

you don't know if you could trust them

W. Curtis Preston: Uh,

yes.

Even if it was Yeah.

You know, and that, that, that's one of the stories.

We're gonna get to

that, right?

The, um, the O-O-O-O-V-H-O-V-H,

right?

Yeah.

The OVH story proofs.

Uh, we have a story, literally every comment that we, we don't make this,

we don't just make this stuff up.

You can't make this stuff up, right?

We, we have stories behind every one of the recommendations that

we make, and this series is about telling these stories and this.

Is a good one.

Do you wanna, do you wanna sort of, uh, look, first off, everybody should

know what Salesforce is, right?

Um, but you

know, just, just in case you don't, Salesforce is, I

think the OG SaaS app, right?

I'm sure there was another before, but they were the, the first one

that really took off their actual phone number is one 800 no software.

I don't know if you, if you knew

No, I didn't know

W. Curtis Preston: um, yeah, I, that's, that.

I remember that from, from many days gone by.

And they are A-C-R-M-A customer relationship management software.

Right.

And I remember using one of their competitors back in the day

when I had my own company and.

Uh, oh my Lord.

Is it so much easier to use Salesforce, especially when

you have multiple salespeople

that are, um, you know, all interacting with a variety of leads?

And again, to preface this story, I'm gonna explain how this works in a big org.

I've been a salesperson and most salespeople are, uh, commissioned.

I.

And they are, they're gonna attack any lead that you give them.

And they, because they're commissioned to do so,

and the only thing that prevents them, you know, you, you give these leads to

this person, these leads to this person.

And the only thing that prevents Steve from jumping all over

Janet's leads is permissions.

In a large database like Salesforce, you assign permissions, you create

groups of leads, and you give permission to Steve or to a certain team.

There's different ways to do it, but you divvy out these leads.

By way of permissions.

Right?

And that would, that prevents Steve from going over and, you know,

stomping all over, uh, Janet's leads

and, um, uh, but then something happened.

So why don't you, so when, when, when did this happen and what happened?

Prasanna Malaiyandi: So it was back in 2019.

So it was a ways ago, and what ended up happening is Salesforce ran a script.

And what the script did is it allowed everyone in an organization

to be able to modify and access all records in that organization.

W. Curtis Preston: Yeah.

And so in your example of Steve and Janet, Steve could

now see everything Janet had could go stomp on it and be like, Hey, by the way,

Janet, you're actually not as far along as you said you were, or change a dollar

amount of the lead and other things like

W. Curtis Preston: Or Steve could also delete all of

Janet's leads, if that's what, if

Steve is a very bad person, if he wanted to go delete all her leads or just

delete, you know, uh, like any interaction that she had with the clients, right?

If

you were.

Yeah.

W. Curtis Preston: A nefarious person.

Yeah, the notes, right?

If you were a nefarious person, you could have done a lot of damage to

other people in the organization, uh, or you could steal their leads.

Just

reassign those leads

to you.

Um,

or you could be doing just some random housekeeping,

innocuous housekeeping stuff like, Hey, I'm just gonna go clear out all my old

leads older than like two years old that I haven't touched and realize that

you might be stomping on Janet's leads.

Yeah.

W. Curtis Preston: Yeah, exactly.

Um, so Salesforce, interestingly enough, Salesforce, according to, and we're

gonna put links to this, we have the, a link to the original post that was made

by Salesforce, as well as a link to a follow-up post that they made several

months later as a, uh, postmortem.

What they learned, but what we know from their posts is they did not

notice that they had done this.

A customer called and said, Hey, this is odd.

Everybody apparently can modify everybody's leads.

Right?

I.

And, and by the way, just, just to put a, a point on that, it, it's

sort of like in file permissions.

If you have modified permissions, you have all of the others,

right?

Uh, you have read, write, you know, modified delete.

Right?

So, um, the, and by the way, they, they appear, it, it appears that

they only had this privilege.

To records.

They didn't have the, they at least didn't grant this permission to be able to

modify things like configurations, right?

So they couldn't go in and basically delete Janet or change Janet's permissions

as a person, as a user, but they could go in and access and do everything.

To her data.

Right?

So it's important to, to just mention that.

Anyway, so they didn't notice that they did this.

Customers called in and then they very quickly, uh, they had,

you know, what I would call, you know, an oh shit moment, right?

And they're like, holy cow.

That maintenance script that we ran, it appears that it did.

Um, you know.

A

W. Curtis Preston: Uh, A lot more than we had intended to do, and they

realized they had really messed up.

And so the first thing they did was just say, okay, just shut down everything.

Right.

Um, which, which I think was probably the best thing they could do at the

time, even though that would, of course immediately at Cal, all their customers.

Yeah.

Well I do wonder if I agree that that's sort of like the nuclear option, right?

But I do wonder if maybe they could have at least, uh, I guess I was just thinking

could they have removed the modify all and just given like view only, but then

some orgs, it still might have been bad to allow Steve to see Janet's leads

W. Curtis Preston: well, I, I, the problem was, I, I don't think, you

know, based on the, the, the records and stuff that we have, I'm not sure

they even know the, knew the extent

of the damage that they had caused

it's like a ransomware attack where

you just pulled a network.

Cable.

W. Curtis Preston: Exactly.

Yeah.

Um, there's a man, there's a great scene in, in, you know, one of my

favorite shows, alias, where, uh, he goes running into the server room

and he literally is like flipping

power switches, you know, they're downloading all the files

up the server and he is just flipping all the power switches.

Uh, it, it was pretty much like that.

And so they, they, um.

Uh, that was their first response.

And then, uh, then what did it do?

So then after that, well, so that shut

it down, but it was only for the organizations that were impacted.

And

W. Curtis Preston: by by the way, I just want to interject.

What we now know is that the, IM, the organizations that were impacted

was any user or any organization that had used Pardot, which is their.

Marketing automation.

, it's the Salesforce equivalent to Marketo.

Right?

So this is the thing that's gonna email your customers

and things like that, right?

Um, so anyone who had ever used or was currently using Pardot, that

turns out to be, who was impacted?

So after that, let's see, what did they do?

W. Curtis Preston: I'm not exactly sure exactly when, what happened

'cause we don't exactly have a timeline 'cause there's multiple

posts and multiple articles and,

you know, we even have a, there's a stack exchange thread that we could

follow during this, uh, uh, outage.

Yeah, even though we don't know that timeline, Curtis,

I think the one thing we can just sort of take away from all these articles

is they did try to fix it themselves.

They weren't like, Hey users, we have nothing to do.

Good luck.

Go pound sand.

Right?

It looks like they were internally trying to do things to fix this and

looking at various technologies or resources that they might have had, but.

As we know, that takes time.

Right?

And as a user, at that time, they weren't really forthcoming about, Hey, we're

trying things internally either, right?

They were, they didn't wanna give users hope.

W. Curtis Preston: Yeah, well, I'm not sure if they were communicating,

you know, it does show that they mass emailed some users.

All we have access to is what they said publicly and publicly.

Again, around this time they had this post where they said, Hey, we messed up.

We gave modify all.

And here are a couple of ways that you can potentially fix this if

you want to fix this yourself.

And, 'cause that was obviously a question that people asked is, Hey,

can I, can I fix this on my account so that I can get my account back online?

And the you, you know, that the response just really infuriates me

because I gave them two options.

And they had to do with the sandbox.

They basically said, if, you know, if you made a sandbox, which is something

that you could do regularly, which if you don't know a sandbox is, uh,

you know, it's a place where you can play with your data and, and,

mess

a clone,

W. Curtis Preston: then It's fine.

It's what?

It's like a clone.

W. Curtis Preston: It is a, yeah, it's a clone that you can

automatically make with Salesforce.

You, you know, it's, some people actually treat it like a backup.

I don't because, uh, it's all in the same place, but, so it doesn't

conform to the 3, 2, 1 rule.

But it, um, the, but they said, you know, if you happen to have a recent backup.

You could go and get their permissions from there.

'cause remember, they're, they don't have to restore the data.

They didn't mess up the data.

They messed up the permissions of the, of the data of the

And just to correct you, you meant

to say sandbox not backup in

that statement, correct?

W. Curtis Preston: oh, did I, did

I say, did I call

Ouch.

You are correct.

I meant to say samples.

So if they, if they're saying if you happen to have a recent

sandbox, copy of your instance.

It's recent.

This is the problem.

It needs to be recent enough to have the user's permissions to match your current

permissions, but it can't be too recent because if it was too recent, in other

words, if it was made in the last few hours, it's just a backup of our mistake.

Right?

It's just, it is just a copy of our mistake.

So they were saying that what, what infuriates me persona is.

Not once did in, in, in, in in any of the external, uh,

stuff that Salesforce put out.

Not once did they say, by the way, if per chance you did what Curtis tells you to do

and actually backed up your data.

W you could just go and, and easily restore the, basically

the, the user's table is

what, you know, for those of you that don't know, you know, Salesforce has,

you know, all these different tables.

It's like any other database.

They call them objects.

I.

So it would've been the user's object, uh, is what I would assume was

that.

needed to be restored and you could restore the, just restore your user's

object to any time before, you know, 12:35 AM on May 17th, 2019, and you'll

be

you'd be fine.

W. Curtis Preston: But they never said that.

I, I just.

That I remember posting a blog at the time that basically said Salesforce

proves they know nothing about backup,

Yeah.

W. Curtis Preston: right?

Because it's like they never once suggested they, they sort of thought

of the, of the sandbox as a backup and never thought that anybody might

want to have backed up their, their

Of course not.

'cause who does backups of Salesforce?

You don't need no stinking backups.

W. Curtis Preston: Nice.

Nice.

Uh, all right.

Uh, um, brownie points or extra points, if you can tell me what

movie that is referring to.

And I mean, the original movie, not the second movie that,

Not the Rob Schneider one

W. Curtis Preston: oh, that would be a third movie.

Prasanna Malaiyandi: because of what that was.

Water Boy

W. Curtis Preston: What's it?

I, no.

Yeah, I don't know.

I'm referring to the original movie starring Humphrey Bogart.

It's called The Treasure of Sierra Madre.

Oh,

W. Curtis Preston: The, the badges.

Yeah.

We, we don't,

we don't know nothing about no stinking badges.

Great greatvine.

Yeah.

Anyway, um, the.

Why don't you read this, uh, this part about the, the stack exchange part there,

there's a, there's an interesting comment on the stack exchange, uh, thread there.

Do you see that?

Is this is the, that's not even the worst that is going.

Yeah.

So, yeah, so on Stack Exchange, one of the users commented.

That's not even the worst that is going on.

Apparently in an attempt to fix this, they remove the modify access all data

from all admin profiles in some instances, including standard and custom profiles.

W. Curtis Preston: So.

so they removed the, they removed the permission even from the

people that needed the permission.

Which basically means do you end up with a read-only

copy of your data while they're trying to figure things out.

Well, because I could see that they don't want you to change anything

because it might not let them restore things back to a good state later on.

W. Curtis Preston: Yeah.

You know, in that, in that Stack Exchange thread, uh, which we'll put a link to

it in, in the show notes, in that stack Exchange thread, they were saying that as

this was going on, Salesforce was saying, please don't try to fix this yourself.

We, we got it.

Like we're gonna, we think, we think we can fix it.

Um, so let's talk about some of the things that they did.

Uh, you know, in the backend, and by the way, this is all news to me.

This was not covered in the original stories that covered this.

This was, you know, in classic, you know, news stuff.

They only covered that initial explosion.

No one

covers the, the remediation and everything afterwards,

especially given that this was,

It's a not sexy stuff, right?

W. Curtis Preston: It's not as sexy.

Yeah.

If it

bleeds, it leads.

And this is, you know,

uh, so about seven months later, so this was May and in January

of, of 2020, um, oh, I just, I just realized like timeframe.

You, like this is a group of people that are writing, they're

just, they're just, they're just writing about this, this problem

that happened in, in, in last year.

Little did they know in two months that the world was gonna fall

Yeah, so the first thing that they.

Yeah, so the first thing they attempted to do was run a backout script.

They were like, Hey, we had a script that ran.

We should just go undo it.

Which in my mind makes total sense, right?

You're like, Hey, the script did something.

Let me just go undo everything that I just did.

W. Curtis Preston: and it looks like the, and it looks like the script had

automatically, it basically, it, it made a backup of the permissions that.

It was supposed to change.

Right.

Which is what a script should

do.

right before I go do a bunch of stuff.

Yeah.

So why

didn't that work?

the problem is, it did not, however, record

things that it had done that it wasn't supposed to have done.

W. Curtis Preston: Yeah.

So the, so the backup, the backup line didn't have the wild, the, the

asterisk in it and the, and the execution line did,

uh, oh.

That's, I, you know, I'm sorry.

. I'm having, uh, shun Freud at the expense of this poor person

who, you know, according to the, to the aftermath and the report.

You know, they, they said, did we follow our process?

They did follow their processes.

They did, um, uh, or most of their processes.

What they didn't do when they did the initial script run that, that

did all of this, they tested it.

But what they didn't do was they didn't do a phased rollout.

Of the script.

They were like,

we got it You know, this guy wrote it, this person, uh, you know, uh,

you know, sanctioned the script.

We've tested the script, the script runs, run it everywhere.

Did they do it on a Friday evening

W. Curtis Preston: Let's see.

17th, 2019.

It was a Friday.

They did it on Friday.

Oh, those poor guys, you know, they didn't have a weekend.

So they did have a variety of technologies that they could possibly use.

To solve this problem.

And one of them was that they have a Dr instance.

We talk about this with 365 as well, because we know that 365

has a rolling, um, you know, uh, replicated copy of their system, right?

So, so this is, again, this is a quote from their report.

A site switch to a DR instance was not an option since the purpose of

the DR option is to replicate it near real time the state of the primary

site, which meant that the inadvertent

permission change would've been replicated in near real time to the redundant site.

We talk about this, don't we?

Yeah.

Dr.

Isn't intended to be a backup.

That is not its purpose.

W. Curtis Preston: yeah.

Well, I would say.

Replication, but like,

because we've talked about this in previous episode replication,

which is what they're using by itself is not, is not a backup.

Right.

Because it, you know, you know, as I, as I jokingly say, it

makes a mistakes more efficient.

Right.

And that's what they, that's

what they're saying here is, yeah, it would've, they're like, well, we knew.

We knew we couldn't use that.

, and then there was a, another thing that they talked about called flashback.

You wanna talk about that?

yeah.

So flashback.

Their database vendor has this technology which allows you to

sort of keep a point in time of the database automatically in the system.

And so you could use that to restore from in case something happens.

Now, the one challenge though is they did look to see can we go use our

flashback area to restore the database,

get everything up and running again.

Unfortunately, they only kept six hours.

W. Curtis Preston: Yeah.

Right?

And so that's the furthest back they could run because that makes sense.

You have some sort of database corruption or you accidentally drop

a table, you just need to go back a couple seconds, you're good to go.

W. Curtis Preston: Yeah, they don't say it, but it looks like they're

referring to an Oracle feature.

Uh, and

there are a series of features there, but it's kind of like

the, the snapshot thing, right?

You can go back to when you took the snapshot, but if you, you know,

if you want to go longer than that.

You don't have, because there's a, there's a window that they, that

they specify and six hours must

have been the window.

And there were pa they were past the window by the time they, they optioned.

So it's interesting the, the option that they chose to.

To figure out what permissions were, what to be able to, you know, to restore them.

'cause the problem, once you've granted modify all, well, how

do you know what to go back to?

Right.

You, you can't just say read all right.

Uh, 'cause even that, right.

You know.

Um, so the, what they started doing is essentially log scraping, right?

They call it log mining.

To, to, to look at customers instances of.

To see what permissions in the logs that these things were set

to, and that's what they began.

And they started going through and in, in the story, in the, in the

postmortem, there are these series of.

We think we did this, we think we did that.

You know, we've, we've restored all these instances.

There were, there were dozens of instances that were affected and they're

like, we think this instance is good.

This instance is good.

So if you're on that instance, then you're good.

But even when all that was done, there were still customers

that were not restored.

and and they said, you know, we're working with you.

And then they gave instructions on how to basically manually fix this and

Which if you have thousands of salespeople

W. Curtis Preston: Right.

Um, you know what would've been really helpful to those customers

in this scenario, persona.

using another vendor.

W. Curtis Preston: Yeah, a backup, right?

right?

If they, if they had backed up the data outta Salesforce into another vendor,

so, so companies you know, that use Salesforce and other CRM products and if

they used a backup, they would've been able to fix this literally like that.

Uh, well, a backup intended for Salesforce

W. Curtis Preston: Yeah, so back well backup of Salesforce,

intended for Salesforce.

I'm not sure what

other, what other method you

No.

So I was thinking like someone could have done a backup by

just dumping out the objects.

Right.

And I don't know if that would've necessarily kept all the permissions

as well if they hadn't backed up the, or dumped the user table as well.

W. Curtis Preston: Well, if you do, if you can do a manual backup of Salesforce,

it basically gives you all the objects.

The only problem with every manual backup is you have to manually do it.

You have to do it every

once in a while, which means, I don't know how often you're gonna be doing it.

It might be once a week.

It still would be better than nothing,

Nothing.

Yeah.

W. Curtis Preston: you basically, it just means your permissions

would be a week old, which in this case would be a good thing.

Um, but it just drives home again to me that there are, you know, you, you've

heard me say this, I think I said it in the last episode of like, there, there

is more in heaven and earth, Horatio than dreamt of in your philosophy, there are

things that can happen to you in the cloud.

The cloud isn't magic.

There are things that, that you're not gonna

anticipate.

be magical.

W. Curtis Preston: Yeah, it is just as magical as actual magic, which

is an illusion.

So I love magic by the way.

I'm a big fan.

Like, I went and saw, like, I saw David Copperfield in Vegas, and I was amazed.

I, I loved it, but inside I knew it was all just an illusion.

Illusion.

Yep.

W. Curtis Preston: Um, so, you know, my advice isn't so much

to Salesforce, Salesforce.

Did as much as they could do in this scenario.

It seems like they were communicating with their users.

They had a status page like we, like we tell them to do.

Um, maybe go.

I, I think the only thing I would fault them for is

maybe a bit more communication about what they're doing internally, right.

W. Curtis Preston: again, I think they may have been doing that

just not publicly the way we were

looking.

Right.

So maybe they were

communicating

privately

'cause ' cause they said, they said in their.

In their postmortem, they're like, what?

What should we have done better?

And, um, they listed a whole bunch of things they were

doing to communicate, uh, what

was going on.

We just outside.

I was really angry at the time because all I saw was that one page,

because I saw the one page where they said, Hey, uh, sorry, um, we

just messed up all your permissions.

And so.

Can you fix it?

Yeah.

W. Curtis Preston: And, and by the way, we're not gonna mention backup.

I was furious at the time.

Uh, so I'm, I'm a little less furious.

But, uh, this is just another big example of why we back up, you know,

everything and why we recommend backing up cloud vendors and es

esp and especially SaaS vendors.

Yeah,

W. Curtis Preston: Any, any

final thoughts from you from Peanut Gallery?

I think that's the right thing.

They did everything they could and users should have backed up their data,

W. Curtis Preston: again, we're blaming the victims.

Uh oh goodness.

could have easily been avoided though.

W. Curtis Preston: It could have, it could have back it up or give it up people.

Uh, thanks for listening.

You know, you are why we do this.

We want to turn you into Cyber recovery Heroes.

That's a wrap