1 00:00:00,050 --> 00:00:05,090 what happens when a SAS giant accidentally grants modify all permissions to 2 00:00:05,090 --> 00:00:07,280 every user in every customer org. 3 00:00:07,640 --> 00:00:08,630 Spoiler alert. 4 00:00:08,660 --> 00:00:09,590 It's not pretty. 5 00:00:10,220 --> 00:00:13,340 Join me and my co-host as we explore the fallout from this 6 00:00:13,340 --> 00:00:15,380 real world cloud catastrophe. 7 00:00:16,010 --> 00:00:19,700 We'll discuss how Salesforce scrambled to restore proper permissions. 8 00:00:20,060 --> 00:00:24,080 The frustration felt by locked out customers and the crucial role 9 00:00:24,080 --> 00:00:26,750 third-party backups could have played. 10 00:00:27,390 --> 00:00:31,980 You'll learn why relying solely on your SAS vendors, recovery capabilities 11 00:00:32,070 --> 00:00:36,960 might leave you high and dry and how having your own backups can save the day 12 00:00:37,170 --> 00:00:39,210 when things go sideways in the cloud. 13 00:00:40,320 --> 00:00:41,910 If this is your first time listening. 14 00:00:41,910 --> 00:00:43,200 Hi, I'm W. 15 00:00:43,200 --> 00:00:45,240 Curtis Preston also known as Mr. 16 00:00:45,240 --> 00:00:45,810 Backup. 17 00:00:46,230 --> 00:00:50,250 My career in backup began over 30 years ago when my backups failed 18 00:00:50,490 --> 00:00:53,790 and my company was unable to restore their purchasing database. 19 00:00:54,090 --> 00:00:56,520 I vowed that would never again happen to me. 20 00:00:56,760 --> 00:00:58,890 And it's my goal to do the same for you. 21 00:00:59,400 --> 00:01:02,730 I want to turn you the unappreciated backup admin. 22 00:01:02,970 --> 00:01:05,010 Into a cyber recovery hero. 23 00:01:05,310 --> 00:01:07,380 This is the backup wrap up. 24 00:01:20,736 --> 00:01:21,786 W. Curtis Preston: Welcome to the show. 25 00:01:22,266 --> 00:01:23,106 I'm your host, W. 26 00:01:23,106 --> 00:01:24,876 Curtis Preston, AKA, Mr. 27 00:01:24,881 --> 00:01:25,356 Backup. 28 00:01:25,356 --> 00:01:32,927 And with me, I have my election primary worker anxiety consultant 29 00:01:32,927 --> 00:01:34,277 Prasanna Malaiyandi: How's it going, Curtis? 30 00:01:34,277 --> 00:01:34,547 Yeah. 31 00:01:34,547 --> 00:01:38,597 You're doing, it's that time of year, or I guess every couple years 32 00:01:38,597 --> 00:01:41,087 where the election happens and 33 00:01:41,117 --> 00:01:43,007 W. Curtis Preston: no, there's no word for like two years. 34 00:01:43,007 --> 00:01:43,397 Is there? 35 00:01:43,397 --> 00:01:47,087 It's a, it's a, that time of biannual, I 36 00:01:47,117 --> 00:01:50,357 Prasanna Malaiyandi: It's weird that bi counts as both, like half as well as two 37 00:01:50,597 --> 00:01:50,837 W. Curtis Preston: Yeah. 38 00:01:50,837 --> 00:01:52,547 Don't get, don't get me started on English. 39 00:01:52,607 --> 00:01:53,087 All right. 40 00:01:53,177 --> 00:01:55,727 Um, semi or bi, right? 41 00:01:56,077 --> 00:02:00,777 So I will once again be an election worker for the upcoming 42 00:02:00,782 --> 00:02:06,567 California Presidential primary, and tomorrow is to set up day. 43 00:02:06,567 --> 00:02:10,287 This year I am running an 11 day vote site. 44 00:02:10,797 --> 00:02:11,427 Prasanna Malaiyandi: Wow. 45 00:02:11,637 --> 00:02:12,177 Crazy. 46 00:02:12,177 --> 00:02:13,767 You'll be a busy, busy man. 47 00:02:14,112 --> 00:02:14,682 W. Curtis Preston: I will be, 48 00:02:15,132 --> 00:02:19,452 well, that's a different point because ask me how many people I 49 00:02:19,452 --> 00:02:22,392 think I'll see in the first 10 days. 50 00:02:23,577 --> 00:02:25,977 Prasanna Malaiyandi: I am gonna say 21. 51 00:02:27,492 --> 00:02:28,812 W. Curtis Preston: Yeah, I think that might be high. 52 00:02:31,362 --> 00:02:35,082 It, uh, because what happens is everybody comes on election day. 53 00:02:35,087 --> 00:02:36,912 I mean, I'm glad we have early voting, right? 54 00:02:36,917 --> 00:02:37,992 I, I, I really am. 55 00:02:37,997 --> 00:02:41,862 I, I believe in access and, and I even like the 11 day sites because 56 00:02:41,862 --> 00:02:45,222 there are some people that have jobs that just really mess up a week. 57 00:02:45,222 --> 00:02:46,572 So four days isn't just. 58 00:02:46,717 --> 00:02:48,127 It just isn't enough for some 59 00:02:48,127 --> 00:02:48,577 people. 60 00:02:48,847 --> 00:02:51,157 I, I believe in access to elections. 61 00:02:51,547 --> 00:02:55,897 It, it's just that, you know, everybody comes on election day and then we go, 62 00:02:55,897 --> 00:02:58,237 yeah, well we've been here for 11 days. 63 00:02:58,237 --> 00:02:59,197 And they're like, what? 64 00:02:59,548 --> 00:03:02,218 Prasanna Malaiyandi: I would normally go vote, like when they used to 65 00:03:02,218 --> 00:03:03,388 have the neighborhood polling place. 66 00:03:03,393 --> 00:03:06,598 I used to go vote in person on the day of the election. 67 00:03:06,628 --> 00:03:07,708 I wouldn't go ahead of time. 68 00:03:07,708 --> 00:03:10,978 I would just go like early in the morning and I'd just go be done and come back. 69 00:03:12,328 --> 00:03:12,868 W. Curtis Preston: Yeah. 70 00:03:12,958 --> 00:03:17,548 Uh, so we're there, uh, by, and by the time this episode airs, the 71 00:03:17,548 --> 00:03:20,218 primary Will Al will already be over. 72 00:03:20,718 --> 00:03:25,038 I love participating in the process and I will answer any. 73 00:03:25,508 --> 00:03:28,898 All election questions that anybody has, and I'll say the same thing 74 00:03:28,898 --> 00:03:30,368 that I say every time this comes up. 75 00:03:30,728 --> 00:03:34,208 If you have any doubt as to the integrity of your election process, 76 00:03:34,208 --> 00:03:37,358 do one of, or both of two things. 77 00:03:37,478 --> 00:03:40,688 One, volunteer as an election worker. 78 00:03:41,078 --> 00:03:42,068 That is you. 79 00:03:42,068 --> 00:03:45,908 You get so much insight into the process and how it works. 80 00:03:46,148 --> 00:03:48,188 Number two, be an observer. 81 00:03:48,368 --> 00:03:48,578 You 82 00:03:48,578 --> 00:03:52,868 are legally allowed to observe every single portion of you 83 00:03:52,868 --> 00:03:54,338 the election process, right? 84 00:03:54,338 --> 00:03:59,048 You know where, where the votes are initially cast, where they are received, 85 00:03:59,108 --> 00:04:05,684 how they are counted, you can view the incredibly boring way in which the, 86 00:04:05,714 --> 00:04:08,384 there is this, well, not just the accounting room, 87 00:04:08,384 --> 00:04:10,004 but there is this process. 88 00:04:10,004 --> 00:04:17,954 The most boring part of the process is when they do a 1% manual count. 89 00:04:18,524 --> 00:04:21,224 So they take 1% of the cartons. 90 00:04:21,524 --> 00:04:25,604 That, uh, you know, the, the ballot cartons that, that, that are gonna contain 91 00:04:25,604 --> 00:04:31,724 anywhere from 20 to 200 votes, you know, and they sit there at a table with like 92 00:04:31,724 --> 00:04:35,924 four people and they read it one by one, and then those four people tally it up. 93 00:04:36,704 --> 00:04:40,184 And then they compare numbers and the numbers all have to match and they have 94 00:04:40,184 --> 00:04:42,584 to match what the machine said to box it. 95 00:04:42,924 --> 00:04:45,444 My wife has done that process, but, oh my lord. 96 00:04:45,924 --> 00:04:47,424 It's like, it's like watching paint 97 00:04:47,469 --> 00:04:50,169 Prasanna Malaiyandi: I, I, I was just gonna think her, you're 98 00:04:50,169 --> 00:04:51,729 probably gonna be like, okay. 99 00:04:51,909 --> 00:04:57,189 One ballot, two ballot, three ballots, like counting sheep. 100 00:04:58,734 --> 00:04:59,064 W. Curtis Preston: Yeah. 101 00:04:59,814 --> 00:05:00,354 Yeah. 102 00:05:00,414 --> 00:05:04,194 But anyway, uh, so, you know, so I'm excited to participate in the process, but 103 00:05:04,199 --> 00:05:08,574 I, I do have a certain amount of anxiety as I was alluding to a certain amount of 104 00:05:08,574 --> 00:05:11,934 anxiety because there are people, right. 105 00:05:12,684 --> 00:05:14,934 People who need people 106 00:05:15,234 --> 00:05:16,254 to yell at. 107 00:05:16,454 --> 00:05:18,944 Prasanna Malaiyandi: I think the other thing to note is it's not like they're 108 00:05:18,944 --> 00:05:20,444 just throwing you to the wolves, right? 109 00:05:20,449 --> 00:05:22,709 So you go through training, right? 110 00:05:22,709 --> 00:05:23,729 W. Curtis Preston: go through a lot of training. 111 00:05:23,729 --> 00:05:24,329 Right, right. 112 00:05:24,389 --> 00:05:24,749 Yeah, 113 00:05:24,839 --> 00:05:25,199 Yeah. 114 00:05:25,469 --> 00:05:28,259 And, and, and you know, and I'm experienced The other 115 00:05:28,259 --> 00:05:29,249 people are experienced. 116 00:05:29,249 --> 00:05:29,459 Yeah. 117 00:05:29,819 --> 00:05:31,919 There's no, yeah, it's not wolf throwing and, 118 00:05:32,459 --> 00:05:33,959 and, and they have lots of support. 119 00:05:33,964 --> 00:05:34,169 Right. 120 00:05:34,169 --> 00:05:38,129 So there, there, there's a, there's a, a phone number, the poll worker hotline. 121 00:05:39,359 --> 00:05:39,899 Which I have 122 00:05:39,899 --> 00:05:41,849 saved as a contact in my phone. 123 00:05:42,569 --> 00:05:43,109 I just have them 124 00:05:43,109 --> 00:05:45,209 as, I have them as R-O-V-R-O-V, 125 00:05:45,479 --> 00:05:46,559 uh, the Registrar of Voters. 126 00:05:46,559 --> 00:05:46,739 Right. 127 00:05:46,744 --> 00:05:49,019 That's their first and last name, and I just call 'em, you know? 128 00:05:49,154 --> 00:05:51,434 Prasanna Malaiyandi: yeah, but you don't need to have the experience that you did 129 00:05:51,439 --> 00:05:53,924 because when you first started all this, you didn't have that experience either. 130 00:05:53,929 --> 00:05:54,194 Right? 131 00:05:54,194 --> 00:05:54,854 You were 132 00:05:55,049 --> 00:05:56,609 W. Curtis Preston: right, Yeah, yeah, 133 00:05:56,684 --> 00:05:59,744 Prasanna Malaiyandi: right, who was learning the ropes as well, and so you 134 00:05:59,759 --> 00:05:59,999 W. Curtis Preston: Yeah. 135 00:05:59,999 --> 00:06:04,439 So you're, you're speaking to the people that I'm saying participate? 136 00:06:04,439 --> 00:06:04,859 Yes. 137 00:06:04,859 --> 00:06:06,509 I I think you should participate. 138 00:06:06,869 --> 00:06:13,559 Um, and, uh, by and large it is a very easygoing, peaceful process. 139 00:06:13,559 --> 00:06:17,399 Every once in a while there are some challenging people 140 00:06:17,699 --> 00:06:20,549 and, uh, you just pass those over to your more experienced 141 00:06:20,549 --> 00:06:23,579 site manager, which is me. 142 00:06:23,699 --> 00:06:25,704 So it's a little bit of anxiety. 143 00:06:28,439 --> 00:06:33,419 But, um, anyway, let's get on to what you know. 144 00:06:33,419 --> 00:06:38,669 This is part of this series that we have, uh, called Cloud Disasters, and 145 00:06:38,669 --> 00:06:41,129 this is yet another cloud disaster. 146 00:06:41,369 --> 00:06:47,309 The cloud is just computers that somebody else is running, and in this case it, it's 147 00:06:47,309 --> 00:06:49,109 a database that someone else is running. 148 00:06:49,894 --> 00:06:50,114 Prasanna Malaiyandi: Yep. 149 00:06:50,429 --> 00:06:53,819 W. Curtis Preston: And we get people all the time that wanna argue, oh, well 150 00:06:53,819 --> 00:06:55,169 I don't need to back up Salesforce. 151 00:06:55,174 --> 00:06:57,119 I don't need to back up Microsoft 365. 152 00:06:57,119 --> 00:06:58,679 It, it is part of the service. 153 00:06:58,709 --> 00:06:59,309 It's not 154 00:06:59,669 --> 00:07:00,089 right. 155 00:07:00,119 --> 00:07:04,649 Um, it's just not, it's not in your, uh, service description to, to go look at it. 156 00:07:04,649 --> 00:07:05,429 If you don't believe me 157 00:07:06,239 --> 00:07:08,099 Prasanna Malaiyandi: And even if it was part of your service description, 158 00:07:08,099 --> 00:07:09,034 you don't know if you could trust them 159 00:07:09,854 --> 00:07:10,074 W. Curtis Preston: Uh, 160 00:07:10,079 --> 00:07:10,559 yes. 161 00:07:10,559 --> 00:07:12,059 Even if it was Yeah. 162 00:07:12,059 --> 00:07:14,729 You know, and that, that, that's one of the stories. 163 00:07:14,729 --> 00:07:15,299 We're gonna get to 164 00:07:15,299 --> 00:07:16,319 that, right? 165 00:07:16,619 --> 00:07:19,919 The, um, the O-O-O-O-V-H-O-V-H, 166 00:07:19,924 --> 00:07:20,249 right? 167 00:07:20,249 --> 00:07:20,519 Yeah. 168 00:07:20,954 --> 00:07:22,664 The OVH story proofs. 169 00:07:23,084 --> 00:07:27,044 Uh, we have a story, literally every comment that we, we don't make this, 170 00:07:27,044 --> 00:07:28,514 we don't just make this stuff up. 171 00:07:28,664 --> 00:07:30,704 You can't make this stuff up, right? 172 00:07:30,974 --> 00:07:34,214 We, we have stories behind every one of the recommendations that 173 00:07:34,214 --> 00:07:38,234 we make, and this series is about telling these stories and this. 174 00:07:38,699 --> 00:07:39,989 Is a good one. 175 00:07:40,259 --> 00:07:44,549 Do you wanna, do you wanna sort of, uh, look, first off, everybody should 176 00:07:44,549 --> 00:07:46,019 know what Salesforce is, right? 177 00:07:46,109 --> 00:07:47,369 Um, but you 178 00:07:47,369 --> 00:07:51,629 know, just, just in case you don't, Salesforce is, I 179 00:07:51,629 --> 00:07:54,329 think the OG SaaS app, right? 180 00:07:54,329 --> 00:07:56,879 I'm sure there was another before, but they were the, the first one 181 00:07:56,879 --> 00:08:02,159 that really took off their actual phone number is one 800 no software. 182 00:08:02,219 --> 00:08:03,209 I don't know if you, if you knew 183 00:08:03,344 --> 00:08:04,064 Prasanna Malaiyandi: No, I didn't know 184 00:08:04,349 --> 00:08:06,359 W. Curtis Preston: um, yeah, I, that's, that. 185 00:08:06,359 --> 00:08:08,849 I remember that from, from many days gone by. 186 00:08:09,029 --> 00:08:14,009 And they are A-C-R-M-A customer relationship management software. 187 00:08:14,009 --> 00:08:14,399 Right. 188 00:08:14,554 --> 00:08:18,839 And I remember using one of their competitors back in the day 189 00:08:18,839 --> 00:08:20,639 when I had my own company and. 190 00:08:21,449 --> 00:08:22,589 Uh, oh my Lord. 191 00:08:22,589 --> 00:08:25,739 Is it so much easier to use Salesforce, especially when 192 00:08:25,739 --> 00:08:27,989 you have multiple salespeople 193 00:08:28,469 --> 00:08:33,659 that are, um, you know, all interacting with a variety of leads? 194 00:08:33,664 --> 00:08:40,469 And again, to preface this story, I'm gonna explain how this works in a big org. 195 00:08:41,159 --> 00:08:47,069 I've been a salesperson and most salespeople are, uh, commissioned. 196 00:08:47,339 --> 00:08:47,369 I. 197 00:08:47,489 --> 00:08:52,679 And they are, they're gonna attack any lead that you give them. 198 00:08:53,279 --> 00:08:56,939 And they, because they're commissioned to do so, 199 00:08:57,629 --> 00:09:02,909 and the only thing that prevents them, you know, you, you give these leads to 200 00:09:02,909 --> 00:09:05,039 this person, these leads to this person. 201 00:09:05,249 --> 00:09:08,969 And the only thing that prevents Steve from jumping all over 202 00:09:08,969 --> 00:09:12,809 Janet's leads is permissions. 203 00:09:13,589 --> 00:09:18,749 In a large database like Salesforce, you assign permissions, you create 204 00:09:18,749 --> 00:09:25,559 groups of leads, and you give permission to Steve or to a certain team. 205 00:09:25,889 --> 00:09:29,879 There's different ways to do it, but you divvy out these leads. 206 00:09:30,599 --> 00:09:32,759 By way of permissions. 207 00:09:32,759 --> 00:09:33,209 Right? 208 00:09:33,239 --> 00:09:37,409 And that would, that prevents Steve from going over and, you know, 209 00:09:37,409 --> 00:09:40,199 stomping all over, uh, Janet's leads 210 00:09:40,829 --> 00:09:47,159 and, um, uh, but then something happened. 211 00:09:47,879 --> 00:09:52,259 So why don't you, so when, when, when did this happen and what happened? 212 00:09:52,649 --> 00:09:54,599 Prasanna Malaiyandi: So it was back in 2019. 213 00:09:55,124 --> 00:10:01,604 So it was a ways ago, and what ended up happening is Salesforce ran a script. 214 00:10:02,894 --> 00:10:09,104 And what the script did is it allowed everyone in an organization 215 00:10:09,134 --> 00:10:14,114 to be able to modify and access all records in that organization. 216 00:10:15,114 --> 00:10:15,404 W. Curtis Preston: Yeah. 217 00:10:16,109 --> 00:10:20,069 Prasanna Malaiyandi: And so in your example of Steve and Janet, Steve could 218 00:10:20,069 --> 00:10:24,299 now see everything Janet had could go stomp on it and be like, Hey, by the way, 219 00:10:24,299 --> 00:10:28,259 Janet, you're actually not as far along as you said you were, or change a dollar 220 00:10:28,259 --> 00:10:30,359 amount of the lead and other things like 221 00:10:30,419 --> 00:10:33,389 W. Curtis Preston: Or Steve could also delete all of 222 00:10:33,394 --> 00:10:35,759 Janet's leads, if that's what, if 223 00:10:36,029 --> 00:10:39,989 Steve is a very bad person, if he wanted to go delete all her leads or just 224 00:10:39,989 --> 00:10:45,869 delete, you know, uh, like any interaction that she had with the clients, right? 225 00:10:46,919 --> 00:10:47,399 If 226 00:10:47,429 --> 00:10:48,269 you were. 227 00:10:48,609 --> 00:10:48,899 Prasanna Malaiyandi: Yeah. 228 00:10:49,269 --> 00:10:50,324 W. Curtis Preston: A nefarious person. 229 00:10:50,324 --> 00:10:51,284 Yeah, the notes, right? 230 00:10:51,284 --> 00:10:55,094 If you were a nefarious person, you could have done a lot of damage to 231 00:10:55,094 --> 00:10:59,204 other people in the organization, uh, or you could steal their leads. 232 00:10:59,204 --> 00:10:59,414 Just 233 00:10:59,419 --> 00:11:00,554 reassign those leads 234 00:11:00,559 --> 00:11:01,244 to you. 235 00:11:01,754 --> 00:11:02,624 Um, 236 00:11:03,074 --> 00:11:06,494 Prasanna Malaiyandi: or you could be doing just some random housekeeping, 237 00:11:06,494 --> 00:11:10,214 innocuous housekeeping stuff like, Hey, I'm just gonna go clear out all my old 238 00:11:10,514 --> 00:11:15,524 leads older than like two years old that I haven't touched and realize that 239 00:11:15,524 --> 00:11:17,744 you might be stomping on Janet's leads. 240 00:11:18,344 --> 00:11:18,734 Yeah. 241 00:11:20,444 --> 00:11:21,734 W. Curtis Preston: Yeah, exactly. 242 00:11:21,944 --> 00:11:28,004 Um, so Salesforce, interestingly enough, Salesforce, according to, and we're 243 00:11:28,009 --> 00:11:31,994 gonna put links to this, we have the, a link to the original post that was made 244 00:11:31,994 --> 00:11:38,144 by Salesforce, as well as a link to a follow-up post that they made several 245 00:11:38,144 --> 00:11:41,174 months later as a, uh, postmortem. 246 00:11:42,014 --> 00:11:46,304 What they learned, but what we know from their posts is they did not 247 00:11:46,424 --> 00:11:48,644 notice that they had done this. 248 00:11:48,914 --> 00:11:52,244 A customer called and said, Hey, this is odd. 249 00:11:52,664 --> 00:11:56,234 Everybody apparently can modify everybody's leads. 250 00:11:56,264 --> 00:11:56,684 Right? 251 00:11:57,269 --> 00:11:57,389 I. 252 00:11:57,389 --> 00:12:01,754 And, and by the way, just, just to put a, a point on that, it, it's 253 00:12:01,754 --> 00:12:03,974 sort of like in file permissions. 254 00:12:03,974 --> 00:12:07,184 If you have modified permissions, you have all of the others, 255 00:12:07,184 --> 00:12:07,574 right? 256 00:12:07,964 --> 00:12:10,964 Uh, you have read, write, you know, modified delete. 257 00:12:10,969 --> 00:12:11,204 Right? 258 00:12:11,204 --> 00:12:15,974 So, um, the, and by the way, they, they appear, it, it appears that 259 00:12:15,974 --> 00:12:18,134 they only had this privilege. 260 00:12:18,339 --> 00:12:20,379 To records. 261 00:12:20,499 --> 00:12:25,449 They didn't have the, they at least didn't grant this permission to be able to 262 00:12:25,449 --> 00:12:30,009 modify things like configurations, right? 263 00:12:30,009 --> 00:12:37,299 So they couldn't go in and basically delete Janet or change Janet's permissions 264 00:12:37,419 --> 00:12:43,149 as a person, as a user, but they could go in and access and do everything. 265 00:12:43,149 --> 00:12:44,349 To her data. 266 00:12:44,409 --> 00:12:44,709 Right? 267 00:12:44,709 --> 00:12:46,329 So it's important to, to just mention that. 268 00:12:46,329 --> 00:12:49,059 Anyway, so they didn't notice that they did this. 269 00:12:49,359 --> 00:12:54,309 Customers called in and then they very quickly, uh, they had, 270 00:12:54,309 --> 00:12:57,999 you know, what I would call, you know, an oh shit moment, right? 271 00:12:58,599 --> 00:13:00,369 And they're like, holy cow. 272 00:13:00,369 --> 00:13:03,699 That maintenance script that we ran, it appears that it did. 273 00:13:04,149 --> 00:13:05,379 Um, you know. 274 00:13:06,449 --> 00:13:06,539 Prasanna Malaiyandi: A 275 00:13:06,614 --> 00:13:09,044 W. Curtis Preston: Uh, A lot more than we had intended to do, and they 276 00:13:09,044 --> 00:13:11,564 realized they had really messed up. 277 00:13:12,254 --> 00:13:17,894 And so the first thing they did was just say, okay, just shut down everything. 278 00:13:18,104 --> 00:13:18,524 Right. 279 00:13:18,584 --> 00:13:22,514 Um, which, which I think was probably the best thing they could do at the 280 00:13:22,514 --> 00:13:26,444 time, even though that would, of course immediately at Cal, all their customers. 281 00:13:26,639 --> 00:13:26,999 Prasanna Malaiyandi: Yeah. 282 00:13:27,179 --> 00:13:31,979 Well I do wonder if I agree that that's sort of like the nuclear option, right? 283 00:13:31,979 --> 00:13:37,769 But I do wonder if maybe they could have at least, uh, I guess I was just thinking 284 00:13:37,769 --> 00:13:42,329 could they have removed the modify all and just given like view only, but then 285 00:13:42,689 --> 00:13:46,529 some orgs, it still might have been bad to allow Steve to see Janet's leads 286 00:13:47,324 --> 00:13:50,294 W. Curtis Preston: well, I, I, the problem was, I, I don't think, you 287 00:13:50,294 --> 00:13:54,944 know, based on the, the, the records and stuff that we have, I'm not sure 288 00:13:55,004 --> 00:13:56,924 they even know the, knew the extent 289 00:13:57,404 --> 00:13:59,624 of the damage that they had caused 290 00:13:59,859 --> 00:14:00,939 Prasanna Malaiyandi: it's like a ransomware attack where 291 00:14:00,939 --> 00:14:01,674 you just pulled a network. 292 00:14:01,684 --> 00:14:02,034 Cable. 293 00:14:03,219 --> 00:14:03,849 W. Curtis Preston: Exactly. 294 00:14:03,849 --> 00:14:04,269 Yeah. 295 00:14:04,359 --> 00:14:08,979 Um, there's a man, there's a great scene in, in, you know, one of my 296 00:14:08,979 --> 00:14:13,769 favorite shows, alias, where, uh, he goes running into the server room 297 00:14:13,769 --> 00:14:15,749 and he literally is like flipping 298 00:14:15,749 --> 00:14:18,689 power switches, you know, they're downloading all the files 299 00:14:18,689 --> 00:14:21,239 up the server and he is just flipping all the power switches. 300 00:14:21,629 --> 00:14:23,729 Uh, it, it was pretty much like that. 301 00:14:24,299 --> 00:14:26,189 And so they, they, um. 302 00:14:26,924 --> 00:14:29,054 Uh, that was their first response. 303 00:14:29,594 --> 00:14:32,474 And then, uh, then what did it do? 304 00:14:34,019 --> 00:14:36,359 Prasanna Malaiyandi: So then after that, well, so that shut 305 00:14:36,359 --> 00:14:40,289 it down, but it was only for the organizations that were impacted. 306 00:14:41,504 --> 00:14:41,714 And 307 00:14:41,749 --> 00:14:43,814 W. Curtis Preston: by by the way, I just want to interject. 308 00:14:44,264 --> 00:14:48,794 What we now know is that the, IM, the organizations that were impacted 309 00:14:48,794 --> 00:14:55,064 was any user or any organization that had used Pardot, which is their. 310 00:14:55,734 --> 00:14:57,024 Marketing automation. 311 00:14:57,534 --> 00:14:59,904 , it's the Salesforce equivalent to Marketo. 312 00:14:59,934 --> 00:15:00,264 Right? 313 00:15:00,264 --> 00:15:03,504 So this is the thing that's gonna email your customers 314 00:15:03,504 --> 00:15:04,614 and things like that, right? 315 00:15:05,034 --> 00:15:10,284 Um, so anyone who had ever used or was currently using Pardot, that 316 00:15:10,374 --> 00:15:12,924 turns out to be, who was impacted? 317 00:15:13,199 --> 00:15:17,014 Prasanna Malaiyandi: So after that, let's see, what did they do? 318 00:15:17,969 --> 00:15:21,449 W. Curtis Preston: I'm not exactly sure exactly when, what happened 319 00:15:21,449 --> 00:15:23,789 'cause we don't exactly have a timeline 'cause there's multiple 320 00:15:23,794 --> 00:15:25,349 posts and multiple articles and, 321 00:15:25,709 --> 00:15:29,519 you know, we even have a, there's a stack exchange thread that we could 322 00:15:29,519 --> 00:15:31,889 follow during this, uh, uh, outage. 323 00:15:32,259 --> 00:15:34,419 Prasanna Malaiyandi: Yeah, even though we don't know that timeline, Curtis, 324 00:15:34,419 --> 00:15:38,319 I think the one thing we can just sort of take away from all these articles 325 00:15:38,319 --> 00:15:41,529 is they did try to fix it themselves. 326 00:15:41,739 --> 00:15:44,649 They weren't like, Hey users, we have nothing to do. 327 00:15:44,649 --> 00:15:45,249 Good luck. 328 00:15:45,369 --> 00:15:46,269 Go pound sand. 329 00:15:46,659 --> 00:15:46,929 Right? 330 00:15:46,929 --> 00:15:52,089 It looks like they were internally trying to do things to fix this and 331 00:15:52,089 --> 00:15:57,909 looking at various technologies or resources that they might have had, but. 332 00:15:58,824 --> 00:16:01,854 As we know, that takes time. 333 00:16:02,004 --> 00:16:02,304 Right? 334 00:16:02,304 --> 00:16:06,384 And as a user, at that time, they weren't really forthcoming about, Hey, we're 335 00:16:06,384 --> 00:16:08,514 trying things internally either, right? 336 00:16:08,519 --> 00:16:10,794 They were, they didn't wanna give users hope. 337 00:16:11,754 --> 00:16:14,334 W. Curtis Preston: Yeah, well, I'm not sure if they were communicating, 338 00:16:14,334 --> 00:16:17,244 you know, it does show that they mass emailed some users. 339 00:16:17,604 --> 00:16:21,804 All we have access to is what they said publicly and publicly. 340 00:16:22,134 --> 00:16:27,144 Again, around this time they had this post where they said, Hey, we messed up. 341 00:16:27,149 --> 00:16:28,314 We gave modify all. 342 00:16:28,904 --> 00:16:33,794 And here are a couple of ways that you can potentially fix this if 343 00:16:33,794 --> 00:16:35,804 you want to fix this yourself. 344 00:16:36,254 --> 00:16:39,344 And, 'cause that was obviously a question that people asked is, Hey, 345 00:16:39,344 --> 00:16:43,184 can I, can I fix this on my account so that I can get my account back online? 346 00:16:43,964 --> 00:16:49,124 And the you, you know, that the response just really infuriates me 347 00:16:49,124 --> 00:16:51,974 because I gave them two options. 348 00:16:52,479 --> 00:16:54,579 And they had to do with the sandbox. 349 00:16:54,579 --> 00:16:57,789 They basically said, if, you know, if you made a sandbox, which is something 350 00:16:57,789 --> 00:17:02,139 that you could do regularly, which if you don't know a sandbox is, uh, 351 00:17:02,319 --> 00:17:05,259 you know, it's a place where you can play with your data and, and, 352 00:17:05,259 --> 00:17:05,529 mess 353 00:17:05,529 --> 00:17:05,949 Prasanna Malaiyandi: a clone, 354 00:17:06,099 --> 00:17:06,789 W. Curtis Preston: then It's fine. 355 00:17:07,059 --> 00:17:07,599 It's what? 356 00:17:07,929 --> 00:17:08,679 Prasanna Malaiyandi: It's like a clone. 357 00:17:09,724 --> 00:17:11,529 W. Curtis Preston: It is a, yeah, it's a clone that you can 358 00:17:11,529 --> 00:17:13,089 automatically make with Salesforce. 359 00:17:13,094 --> 00:17:16,329 You, you know, it's, some people actually treat it like a backup. 360 00:17:16,329 --> 00:17:21,699 I don't because, uh, it's all in the same place, but, so it doesn't 361 00:17:21,699 --> 00:17:23,169 conform to the 3, 2, 1 rule. 362 00:17:23,559 --> 00:17:30,309 But it, um, the, but they said, you know, if you happen to have a recent backup. 363 00:17:30,684 --> 00:17:32,754 You could go and get their permissions from there. 364 00:17:32,784 --> 00:17:35,424 'cause remember, they're, they don't have to restore the data. 365 00:17:35,964 --> 00:17:37,404 They didn't mess up the data. 366 00:17:37,404 --> 00:17:40,344 They messed up the permissions of the, of the data of the 367 00:17:40,554 --> 00:17:42,384 Prasanna Malaiyandi: And just to correct you, you meant 368 00:17:42,384 --> 00:17:44,064 to say sandbox not backup in 369 00:17:44,064 --> 00:17:44,964 that statement, correct? 370 00:17:45,054 --> 00:17:46,014 W. Curtis Preston: oh, did I, did 371 00:17:46,014 --> 00:17:47,214 I say, did I call 372 00:17:47,964 --> 00:17:48,564 Ouch. 373 00:17:48,894 --> 00:17:49,794 You are correct. 374 00:17:50,154 --> 00:17:51,234 I meant to say samples. 375 00:17:51,744 --> 00:17:54,654 So if they, if they're saying if you happen to have a recent 376 00:17:54,954 --> 00:17:59,094 sandbox, copy of your instance. 377 00:18:00,174 --> 00:18:00,864 It's recent. 378 00:18:01,314 --> 00:18:02,184 This is the problem. 379 00:18:02,814 --> 00:18:10,344 It needs to be recent enough to have the user's permissions to match your current 380 00:18:10,349 --> 00:18:14,814 permissions, but it can't be too recent because if it was too recent, in other 381 00:18:14,814 --> 00:18:19,194 words, if it was made in the last few hours, it's just a backup of our mistake. 382 00:18:19,194 --> 00:18:19,494 Right? 383 00:18:19,494 --> 00:18:21,594 It's just, it is just a copy of our mistake. 384 00:18:22,674 --> 00:18:26,994 So they were saying that what, what infuriates me persona is. 385 00:18:27,744 --> 00:18:33,744 Not once did in, in, in, in in any of the external, uh, 386 00:18:34,584 --> 00:18:37,614 stuff that Salesforce put out. 387 00:18:37,734 --> 00:18:45,234 Not once did they say, by the way, if per chance you did what Curtis tells you to do 388 00:18:45,894 --> 00:18:47,904 and actually backed up your data. 389 00:18:48,564 --> 00:18:52,944 W you could just go and, and easily restore the, basically 390 00:18:52,944 --> 00:18:54,954 the, the user's table is 391 00:18:54,954 --> 00:18:58,914 what, you know, for those of you that don't know, you know, Salesforce has, 392 00:18:58,914 --> 00:19:00,264 you know, all these different tables. 393 00:19:00,269 --> 00:19:01,494 It's like any other database. 394 00:19:01,494 --> 00:19:02,664 They call them objects. 395 00:19:02,664 --> 00:19:02,724 I. 396 00:19:03,549 --> 00:19:09,669 So it would've been the user's object, uh, is what I would assume was 397 00:19:09,729 --> 00:19:10,089 that. 398 00:19:10,089 --> 00:19:14,109 needed to be restored and you could restore the, just restore your user's 399 00:19:14,349 --> 00:19:21,854 object to any time before, you know, 12:35 AM on May 17th, 2019, and you'll 400 00:19:21,939 --> 00:19:22,089 be 401 00:19:22,119 --> 00:19:22,659 Prasanna Malaiyandi: you'd be fine. 402 00:19:23,349 --> 00:19:24,489 W. Curtis Preston: But they never said that. 403 00:19:24,489 --> 00:19:25,179 I, I just. 404 00:19:25,824 --> 00:19:30,654 That I remember posting a blog at the time that basically said Salesforce 405 00:19:30,654 --> 00:19:32,694 proves they know nothing about backup, 406 00:19:32,949 --> 00:19:33,369 Prasanna Malaiyandi: Yeah. 407 00:19:33,834 --> 00:19:34,014 W. Curtis Preston: right? 408 00:19:34,044 --> 00:19:37,974 Because it's like they never once suggested they, they sort of thought 409 00:19:37,974 --> 00:19:42,954 of the, of the sandbox as a backup and never thought that anybody might 410 00:19:42,954 --> 00:19:44,699 want to have backed up their, their 411 00:19:44,919 --> 00:19:45,609 Prasanna Malaiyandi: Of course not. 412 00:19:46,479 --> 00:19:49,299 'cause who does backups of Salesforce? 413 00:19:49,749 --> 00:19:51,519 You don't need no stinking backups. 414 00:19:53,109 --> 00:19:53,589 W. Curtis Preston: Nice. 415 00:19:53,589 --> 00:19:53,979 Nice. 416 00:19:53,979 --> 00:19:54,699 Uh, all right. 417 00:19:54,729 --> 00:19:59,409 Uh, um, brownie points or extra points, if you can tell me what 418 00:19:59,409 --> 00:20:01,629 movie that is referring to. 419 00:20:01,629 --> 00:20:05,139 And I mean, the original movie, not the second movie that, 420 00:20:07,299 --> 00:20:08,529 Prasanna Malaiyandi: Not the Rob Schneider one 421 00:20:09,969 --> 00:20:11,349 W. Curtis Preston: oh, that would be a third movie. 422 00:20:13,779 --> 00:20:15,159 Prasanna Malaiyandi: because of what that was. 423 00:20:15,164 --> 00:20:15,644 Water Boy 424 00:20:17,814 --> 00:20:18,324 W. Curtis Preston: What's it? 425 00:20:18,714 --> 00:20:19,434 I, no. 426 00:20:19,494 --> 00:20:20,274 Yeah, I don't know. 427 00:20:21,264 --> 00:20:24,444 I'm referring to the original movie starring Humphrey Bogart. 428 00:20:25,824 --> 00:20:27,714 It's called The Treasure of Sierra Madre. 429 00:20:28,314 --> 00:20:28,584 Prasanna Malaiyandi: Oh, 430 00:20:29,184 --> 00:20:30,054 W. Curtis Preston: The, the badges. 431 00:20:31,014 --> 00:20:31,344 Yeah. 432 00:20:31,344 --> 00:20:32,184 We, we don't, 433 00:20:32,904 --> 00:20:35,664 we don't know nothing about no stinking badges. 434 00:20:35,724 --> 00:20:36,894 Great greatvine. 435 00:20:37,344 --> 00:20:37,614 Yeah. 436 00:20:37,614 --> 00:20:40,854 Anyway, um, the. 437 00:20:41,409 --> 00:20:47,349 Why don't you read this, uh, this part about the, the stack exchange part there, 438 00:20:47,349 --> 00:20:51,819 there's a, there's an interesting comment on the stack exchange, uh, thread there. 439 00:20:53,049 --> 00:20:53,769 Do you see that? 440 00:20:53,949 --> 00:20:56,709 Prasanna Malaiyandi: Is this is the, that's not even the worst that is going. 441 00:20:56,919 --> 00:20:57,219 Yeah. 442 00:20:57,549 --> 00:21:02,139 So, yeah, so on Stack Exchange, one of the users commented. 443 00:21:02,769 --> 00:21:04,989 That's not even the worst that is going on. 444 00:21:05,079 --> 00:21:10,779 Apparently in an attempt to fix this, they remove the modify access all data 445 00:21:10,989 --> 00:21:17,229 from all admin profiles in some instances, including standard and custom profiles. 446 00:21:19,429 --> 00:21:19,649 W. Curtis Preston: So. 447 00:21:20,309 --> 00:21:23,484 so they removed the, they removed the permission even from the 448 00:21:23,484 --> 00:21:25,074 people that needed the permission. 449 00:21:27,909 --> 00:21:30,669 Prasanna Malaiyandi: Which basically means do you end up with a read-only 450 00:21:30,669 --> 00:21:34,059 copy of your data while they're trying to figure things out. 451 00:21:34,299 --> 00:21:36,909 Well, because I could see that they don't want you to change anything 452 00:21:36,909 --> 00:21:41,139 because it might not let them restore things back to a good state later on. 453 00:21:41,964 --> 00:21:42,414 W. Curtis Preston: Yeah. 454 00:21:42,419 --> 00:21:46,584 You know, in that, in that Stack Exchange thread, uh, which we'll put a link to 455 00:21:46,584 --> 00:21:51,144 it in, in the show notes, in that stack Exchange thread, they were saying that as 456 00:21:51,144 --> 00:21:55,524 this was going on, Salesforce was saying, please don't try to fix this yourself. 457 00:21:56,214 --> 00:21:57,264 We, we got it. 458 00:21:57,294 --> 00:21:59,724 Like we're gonna, we think, we think we can fix it. 459 00:22:00,084 --> 00:22:03,479 Um, so let's talk about some of the things that they did. 460 00:22:03,964 --> 00:22:08,104 Uh, you know, in the backend, and by the way, this is all news to me. 461 00:22:08,314 --> 00:22:11,494 This was not covered in the original stories that covered this. 462 00:22:11,854 --> 00:22:16,054 This was, you know, in classic, you know, news stuff. 463 00:22:16,054 --> 00:22:19,054 They only covered that initial explosion. 464 00:22:19,059 --> 00:22:19,354 No one 465 00:22:19,354 --> 00:22:22,774 covers the, the remediation and everything afterwards, 466 00:22:22,774 --> 00:22:24,214 especially given that this was, 467 00:22:24,549 --> 00:22:25,974 Prasanna Malaiyandi: It's a not sexy stuff, right? 468 00:22:26,664 --> 00:22:27,594 W. Curtis Preston: It's not as sexy. 469 00:22:27,594 --> 00:22:27,774 Yeah. 470 00:22:27,774 --> 00:22:27,984 If it 471 00:22:27,984 --> 00:22:28,794 bleeds, it leads. 472 00:22:28,794 --> 00:22:29,664 And this is, you know, 473 00:22:30,204 --> 00:22:35,034 uh, so about seven months later, so this was May and in January 474 00:22:35,034 --> 00:22:41,124 of, of 2020, um, oh, I just, I just realized like timeframe. 475 00:22:41,129 --> 00:22:43,254 You, like this is a group of people that are writing, they're 476 00:22:43,254 --> 00:22:45,714 just, they're just, they're just writing about this, this problem 477 00:22:45,714 --> 00:22:48,174 that happened in, in, in last year. 478 00:22:48,174 --> 00:22:50,664 Little did they know in two months that the world was gonna fall 479 00:22:50,814 --> 00:22:53,424 Prasanna Malaiyandi: Yeah, so the first thing that they. 480 00:22:54,189 --> 00:22:57,489 Yeah, so the first thing they attempted to do was run a backout script. 481 00:22:58,029 --> 00:23:00,489 They were like, Hey, we had a script that ran. 482 00:23:00,519 --> 00:23:01,749 We should just go undo it. 483 00:23:01,749 --> 00:23:04,509 Which in my mind makes total sense, right? 484 00:23:04,509 --> 00:23:06,129 You're like, Hey, the script did something. 485 00:23:06,129 --> 00:23:08,349 Let me just go undo everything that I just did. 486 00:23:09,144 --> 00:23:12,804 W. Curtis Preston: and it looks like the, and it looks like the script had 487 00:23:13,429 --> 00:23:18,024 automatically, it basically, it, it made a backup of the permissions that. 488 00:23:18,684 --> 00:23:20,034 It was supposed to change. 489 00:23:20,154 --> 00:23:20,574 Right. 490 00:23:20,574 --> 00:23:22,164 Which is what a script should 491 00:23:22,164 --> 00:23:22,464 do. 492 00:23:22,464 --> 00:23:24,294 right before I go do a bunch of stuff. 493 00:23:25,284 --> 00:23:25,704 Yeah. 494 00:23:26,094 --> 00:23:26,394 So why 495 00:23:26,399 --> 00:23:27,054 didn't that work? 496 00:23:27,774 --> 00:23:31,134 Prasanna Malaiyandi: the problem is, it did not, however, record 497 00:23:31,494 --> 00:23:34,779 things that it had done that it wasn't supposed to have done. 498 00:23:36,384 --> 00:23:36,864 W. Curtis Preston: Yeah. 499 00:23:37,764 --> 00:23:42,714 So the, so the backup, the backup line didn't have the wild, the, the 500 00:23:42,714 --> 00:23:47,724 asterisk in it and the, and the execution line did, 501 00:23:48,144 --> 00:23:48,954 uh, oh. 502 00:23:48,959 --> 00:23:50,124 That's, I, you know, I'm sorry. 503 00:23:50,174 --> 00:23:54,074 . I'm having, uh, shun Freud at the expense of this poor person 504 00:23:54,794 --> 00:23:59,054 who, you know, according to the, to the aftermath and the report. 505 00:23:59,054 --> 00:24:01,304 You know, they, they said, did we follow our process? 506 00:24:01,604 --> 00:24:03,434 They did follow their processes. 507 00:24:03,734 --> 00:24:07,004 They did, um, uh, or most of their processes. 508 00:24:07,009 --> 00:24:11,504 What they didn't do when they did the initial script run that, that 509 00:24:11,504 --> 00:24:13,304 did all of this, they tested it. 510 00:24:13,874 --> 00:24:18,194 But what they didn't do was they didn't do a phased rollout. 511 00:24:19,139 --> 00:24:19,979 Of the script. 512 00:24:20,009 --> 00:24:20,609 They were like, 513 00:24:21,209 --> 00:24:26,099 we got it You know, this guy wrote it, this person, uh, you know, uh, 514 00:24:26,609 --> 00:24:27,869 you know, sanctioned the script. 515 00:24:27,869 --> 00:24:30,989 We've tested the script, the script runs, run it everywhere. 516 00:24:31,620 --> 00:24:33,575 Prasanna Malaiyandi: Did they do it on a Friday evening 517 00:24:33,847 --> 00:24:34,429 W. Curtis Preston: Let's see. 518 00:24:35,695 --> 00:24:37,995 17th, 2019. 519 00:24:38,705 --> 00:24:39,545 It was a Friday. 520 00:24:39,920 --> 00:24:41,010 They did it on Friday. 521 00:24:41,424 --> 00:24:45,444 Oh, those poor guys, you know, they didn't have a weekend. 522 00:24:46,374 --> 00:24:52,644 So they did have a variety of technologies that they could possibly use. 523 00:24:53,844 --> 00:24:55,104 To solve this problem. 524 00:24:55,344 --> 00:24:57,924 And one of them was that they have a Dr instance. 525 00:24:58,074 --> 00:25:03,264 We talk about this with 365 as well, because we know that 365 526 00:25:03,264 --> 00:25:09,374 has a rolling, um, you know, uh, replicated copy of their system, right? 527 00:25:09,824 --> 00:25:12,884 So, so this is, again, this is a quote from their report. 528 00:25:13,184 --> 00:25:17,714 A site switch to a DR instance was not an option since the purpose of 529 00:25:17,714 --> 00:25:21,584 the DR option is to replicate it near real time the state of the primary 530 00:25:21,584 --> 00:25:23,594 site, which meant that the inadvertent 531 00:25:23,599 --> 00:25:27,794 permission change would've been replicated in near real time to the redundant site. 532 00:25:27,824 --> 00:25:28,964 We talk about this, don't we? 533 00:25:29,324 --> 00:25:29,684 Prasanna Malaiyandi: Yeah. 534 00:25:30,074 --> 00:25:30,464 Dr. 535 00:25:30,464 --> 00:25:31,964 Isn't intended to be a backup. 536 00:25:32,264 --> 00:25:33,314 That is not its purpose. 537 00:25:33,374 --> 00:25:33,704 W. Curtis Preston: yeah. 538 00:25:34,394 --> 00:25:35,954 Well, I would say. 539 00:25:36,539 --> 00:25:38,039 Replication, but like, 540 00:25:38,369 --> 00:25:40,919 because we've talked about this in previous episode replication, 541 00:25:40,919 --> 00:25:46,409 which is what they're using by itself is not, is not a backup. 542 00:25:46,409 --> 00:25:46,619 Right. 543 00:25:46,619 --> 00:25:51,299 Because it, you know, you know, as I, as I jokingly say, it 544 00:25:51,299 --> 00:25:54,089 makes a mistakes more efficient. 545 00:25:55,139 --> 00:25:55,469 Right. 546 00:25:56,244 --> 00:25:57,839 And that's what they, that's 547 00:25:57,839 --> 00:26:00,719 what they're saying here is, yeah, it would've, they're like, well, we knew. 548 00:26:00,719 --> 00:26:02,219 We knew we couldn't use that. 549 00:26:02,529 --> 00:26:05,559 , and then there was a, another thing that they talked about called flashback. 550 00:26:05,559 --> 00:26:06,254 You wanna talk about that? 551 00:26:07,744 --> 00:26:08,034 Prasanna Malaiyandi: yeah. 552 00:26:08,174 --> 00:26:09,274 So flashback. 553 00:26:10,089 --> 00:26:14,589 Their database vendor has this technology which allows you to 554 00:26:14,889 --> 00:26:19,299 sort of keep a point in time of the database automatically in the system. 555 00:26:19,749 --> 00:26:23,499 And so you could use that to restore from in case something happens. 556 00:26:24,159 --> 00:26:29,859 Now, the one challenge though is they did look to see can we go use our 557 00:26:29,859 --> 00:26:31,779 flashback area to restore the database, 558 00:26:31,779 --> 00:26:33,339 get everything up and running again. 559 00:26:33,789 --> 00:26:36,069 Unfortunately, they only kept six hours. 560 00:26:36,954 --> 00:26:37,374 W. Curtis Preston: Yeah. 561 00:26:37,479 --> 00:26:37,929 Prasanna Malaiyandi: Right? 562 00:26:38,259 --> 00:26:41,139 And so that's the furthest back they could run because that makes sense. 563 00:26:41,139 --> 00:26:43,539 You have some sort of database corruption or you accidentally drop 564 00:26:43,539 --> 00:26:45,789 a table, you just need to go back a couple seconds, you're good to go. 565 00:26:46,134 --> 00:26:47,849 W. Curtis Preston: Yeah, they don't say it, but it looks like they're 566 00:26:47,849 --> 00:26:49,379 referring to an Oracle feature. 567 00:26:49,769 --> 00:26:50,429 Uh, and 568 00:26:50,429 --> 00:26:53,099 there are a series of features there, but it's kind of like 569 00:26:53,129 --> 00:26:54,809 the, the snapshot thing, right? 570 00:26:54,809 --> 00:26:57,329 You can go back to when you took the snapshot, but if you, you know, 571 00:26:57,329 --> 00:26:58,739 if you want to go longer than that. 572 00:26:58,959 --> 00:27:02,079 You don't have, because there's a, there's a window that they, that 573 00:27:02,079 --> 00:27:04,359 they specify and six hours must 574 00:27:04,359 --> 00:27:04,989 have been the window. 575 00:27:04,989 --> 00:27:08,109 And there were pa they were past the window by the time they, they optioned. 576 00:27:08,469 --> 00:27:13,659 So it's interesting the, the option that they chose to. 577 00:27:14,339 --> 00:27:18,869 To figure out what permissions were, what to be able to, you know, to restore them. 578 00:27:19,049 --> 00:27:22,649 'cause the problem, once you've granted modify all, well, how 579 00:27:22,649 --> 00:27:24,269 do you know what to go back to? 580 00:27:24,779 --> 00:27:25,229 Right. 581 00:27:25,379 --> 00:27:27,719 You, you can't just say read all right. 582 00:27:28,139 --> 00:27:29,429 Uh, 'cause even that, right. 583 00:27:29,429 --> 00:27:29,939 You know. 584 00:27:30,179 --> 00:27:37,289 Um, so the, what they started doing is essentially log scraping, right? 585 00:27:37,289 --> 00:27:38,519 They call it log mining. 586 00:27:38,969 --> 00:27:43,229 To, to, to look at customers instances of. 587 00:27:44,129 --> 00:27:47,999 To see what permissions in the logs that these things were set 588 00:27:47,999 --> 00:27:50,249 to, and that's what they began. 589 00:27:50,249 --> 00:27:54,839 And they started going through and in, in the story, in the, in the 590 00:27:54,844 --> 00:27:57,449 postmortem, there are these series of. 591 00:27:58,484 --> 00:28:01,244 We think we did this, we think we did that. 592 00:28:01,244 --> 00:28:03,374 You know, we've, we've restored all these instances. 593 00:28:03,404 --> 00:28:06,164 There were, there were dozens of instances that were affected and they're 594 00:28:06,164 --> 00:28:07,964 like, we think this instance is good. 595 00:28:07,964 --> 00:28:09,284 This instance is good. 596 00:28:09,284 --> 00:28:12,044 So if you're on that instance, then you're good. 597 00:28:12,474 --> 00:28:16,489 But even when all that was done, there were still customers 598 00:28:16,489 --> 00:28:18,829 that were not restored. 599 00:28:19,969 --> 00:28:22,549 and and they said, you know, we're working with you. 600 00:28:22,549 --> 00:28:28,579 And then they gave instructions on how to basically manually fix this and 601 00:28:29,199 --> 00:28:31,589 Prasanna Malaiyandi: Which if you have thousands of salespeople 602 00:28:32,959 --> 00:28:33,559 W. Curtis Preston: Right. 603 00:28:34,069 --> 00:28:38,659 Um, you know what would've been really helpful to those customers 604 00:28:38,659 --> 00:28:39,804 in this scenario, persona. 605 00:28:41,794 --> 00:28:42,689 Prasanna Malaiyandi: using another vendor. 606 00:28:43,159 --> 00:28:44,779 W. Curtis Preston: Yeah, a backup, right? 607 00:28:45,409 --> 00:28:45,829 right? 608 00:28:46,069 --> 00:28:50,629 If they, if they had backed up the data outta Salesforce into another vendor, 609 00:28:50,929 --> 00:28:57,319 so, so companies you know, that use Salesforce and other CRM products and if 610 00:28:57,319 --> 00:29:01,999 they used a backup, they would've been able to fix this literally like that. 611 00:29:02,254 --> 00:29:06,724 Prasanna Malaiyandi: Uh, well, a backup intended for Salesforce 612 00:29:06,739 --> 00:29:09,079 W. Curtis Preston: Yeah, so back well backup of Salesforce, 613 00:29:09,079 --> 00:29:10,069 intended for Salesforce. 614 00:29:10,069 --> 00:29:10,639 I'm not sure what 615 00:29:10,639 --> 00:29:11,959 other, what other method you 616 00:29:12,079 --> 00:29:12,319 Prasanna Malaiyandi: No. 617 00:29:12,319 --> 00:29:15,859 So I was thinking like someone could have done a backup by 618 00:29:15,859 --> 00:29:17,689 just dumping out the objects. 619 00:29:19,339 --> 00:29:19,729 Right. 620 00:29:20,089 --> 00:29:23,359 And I don't know if that would've necessarily kept all the permissions 621 00:29:23,364 --> 00:29:26,869 as well if they hadn't backed up the, or dumped the user table as well. 622 00:29:27,109 --> 00:29:31,129 W. Curtis Preston: Well, if you do, if you can do a manual backup of Salesforce, 623 00:29:31,129 --> 00:29:33,349 it basically gives you all the objects. 624 00:29:33,719 --> 00:29:37,365 The only problem with every manual backup is you have to manually do it. 625 00:29:37,365 --> 00:29:38,025 You have to do it every 626 00:29:38,030 --> 00:29:40,485 once in a while, which means, I don't know how often you're gonna be doing it. 627 00:29:40,725 --> 00:29:41,805 It might be once a week. 628 00:29:41,805 --> 00:29:43,485 It still would be better than nothing, 629 00:29:44,400 --> 00:29:44,880 Prasanna Malaiyandi: Nothing. 630 00:29:44,940 --> 00:29:45,270 Yeah. 631 00:29:45,345 --> 00:29:47,295 W. Curtis Preston: you basically, it just means your permissions 632 00:29:47,295 --> 00:29:50,325 would be a week old, which in this case would be a good thing. 633 00:29:50,825 --> 00:29:57,380 Um, but it just drives home again to me that there are, you know, you, you've 634 00:29:57,380 --> 00:30:01,100 heard me say this, I think I said it in the last episode of like, there, there 635 00:30:01,100 --> 00:30:05,990 is more in heaven and earth, Horatio than dreamt of in your philosophy, there are 636 00:30:05,990 --> 00:30:08,030 things that can happen to you in the cloud. 637 00:30:08,030 --> 00:30:09,230 The cloud isn't magic. 638 00:30:09,230 --> 00:30:11,420 There are things that, that you're not gonna 639 00:30:11,420 --> 00:30:12,290 anticipate. 640 00:30:12,315 --> 00:30:13,455 Prasanna Malaiyandi: be magical. 641 00:30:16,050 --> 00:30:20,160 W. Curtis Preston: Yeah, it is just as magical as actual magic, which 642 00:30:20,760 --> 00:30:21,870 is an illusion. 643 00:30:21,870 --> 00:30:26,100 So I love magic by the way. 644 00:30:26,100 --> 00:30:27,210 I'm a big fan. 645 00:30:27,210 --> 00:30:31,530 Like, I went and saw, like, I saw David Copperfield in Vegas, and I was amazed. 646 00:30:31,530 --> 00:30:35,880 I, I loved it, but inside I knew it was all just an illusion. 647 00:30:36,525 --> 00:30:37,125 Prasanna Malaiyandi: Illusion. 648 00:30:37,215 --> 00:30:37,395 Yep. 649 00:30:38,010 --> 00:30:42,480 W. Curtis Preston: Um, so, you know, my advice isn't so much 650 00:30:42,540 --> 00:30:44,100 to Salesforce, Salesforce. 651 00:30:45,225 --> 00:30:48,315 Did as much as they could do in this scenario. 652 00:30:48,315 --> 00:30:50,715 It seems like they were communicating with their users. 653 00:30:50,715 --> 00:30:54,435 They had a status page like we, like we tell them to do. 654 00:30:55,125 --> 00:30:56,475 Um, maybe go. 655 00:30:56,475 --> 00:30:59,055 Prasanna Malaiyandi: I, I think the only thing I would fault them for is 656 00:30:59,055 --> 00:31:04,110 maybe a bit more communication about what they're doing internally, right. 657 00:31:04,230 --> 00:31:06,600 W. Curtis Preston: again, I think they may have been doing that 658 00:31:06,600 --> 00:31:09,600 just not publicly the way we were 659 00:31:09,600 --> 00:31:09,990 looking. 660 00:31:09,990 --> 00:31:10,380 Right. 661 00:31:10,385 --> 00:31:11,160 So maybe they were 662 00:31:11,160 --> 00:31:12,180 communicating 663 00:31:12,429 --> 00:31:12,939 privately 664 00:31:12,939 --> 00:31:15,187 'cause ' cause they said, they said in their. 665 00:31:15,682 --> 00:31:17,302 In their postmortem, they're like, what? 666 00:31:17,302 --> 00:31:18,592 What should we have done better? 667 00:31:19,042 --> 00:31:22,102 And, um, they listed a whole bunch of things they were 668 00:31:22,102 --> 00:31:23,992 doing to communicate, uh, what 669 00:31:23,997 --> 00:31:24,772 was going on. 670 00:31:25,192 --> 00:31:27,562 We just outside. 671 00:31:27,952 --> 00:31:31,762 I was really angry at the time because all I saw was that one page, 672 00:31:32,182 --> 00:31:36,952 because I saw the one page where they said, Hey, uh, sorry, um, we 673 00:31:36,952 --> 00:31:38,992 just messed up all your permissions. 674 00:31:39,082 --> 00:31:39,592 And so. 675 00:31:41,047 --> 00:31:41,917 Can you fix it? 676 00:31:43,137 --> 00:31:43,427 Prasanna Malaiyandi: Yeah. 677 00:31:43,452 --> 00:31:45,607 W. Curtis Preston: And, and by the way, we're not gonna mention backup. 678 00:31:45,607 --> 00:31:47,227 I was furious at the time. 679 00:31:47,557 --> 00:31:49,597 Uh, so I'm, I'm a little less furious. 680 00:31:50,047 --> 00:31:56,257 But, uh, this is just another big example of why we back up, you know, 681 00:31:56,707 --> 00:32:01,737 everything and why we recommend backing up cloud vendors and es 682 00:32:01,737 --> 00:32:04,107 esp and especially SaaS vendors. 683 00:32:05,412 --> 00:32:05,682 Prasanna Malaiyandi: Yeah, 684 00:32:06,867 --> 00:32:07,617 W. Curtis Preston: Any, any 685 00:32:07,767 --> 00:32:10,137 final thoughts from you from Peanut Gallery? 686 00:32:10,192 --> 00:32:12,022 Prasanna Malaiyandi: I think that's the right thing. 687 00:32:12,052 --> 00:32:16,252 They did everything they could and users should have backed up their data, 688 00:32:16,547 --> 00:32:17,562 W. Curtis Preston: again, we're blaming the victims. 689 00:32:20,222 --> 00:32:21,362 Uh oh goodness. 690 00:32:21,452 --> 00:32:23,252 Prasanna Malaiyandi: could have easily been avoided though. 691 00:32:23,327 --> 00:32:26,447 W. Curtis Preston: It could have, it could have back it up or give it up people. 692 00:32:26,807 --> 00:32:28,127 Uh, thanks for listening. 693 00:32:28,187 --> 00:32:29,897 You know, you are why we do this. 694 00:32:29,897 --> 00:32:32,747 We want to turn you into Cyber recovery Heroes. 695 00:32:32,927 --> 00:32:34,067 That's a wrap