W. Curtis Preston: Spoiler alert, the company in this episode ceased to exist
Speaker:due to what happened in this story.
Speaker:Disasters happen, and since the cloud is just someone else's data
Speaker:center, they sometimes happen in the cloud, and sometimes they
Speaker:take companies along with them.
Speaker:This episode is the first in a series called Cloud Disasters.
Speaker:Each episode tells the real story.
Speaker:Of a company who failed to back up their cloud data and suffered as a result.
Speaker:And these aren't podunk cloud vendors either.
Speaker:Every vendor covered in this story is a major vendor.
Speaker:I'm not kidding around when I say you should back up your cloud data.
Speaker:I don't care that there are those who think I'm just pedaling fud.
Speaker:Newsflash, the only reason we've ever backed up anything is because
Speaker:of the fear of losing it and the uncertainty and doubt we have in
Speaker:the system that we're backing up.
Speaker:Fans of the podcast know, the whole reason that I became a backup specialist is that
Speaker:in 1993, I was unable to recover a very important Oracle database for my employer.
Speaker:I didn't want that to happen to anyone else.
Speaker:So I've dedicated myself to helping others protect themselves
Speaker:from feeling that awful feeling.
Speaker:This show is aimed at you, the unappreciated backup admin, and we want
Speaker:to turn you into a cyber recovery hero.
Speaker:This is the backup wrap up.
Speaker:Welcome to the show.
Speaker:I'm your host, w Curtis Preston, and I have with me my continued and necessary
Speaker:Tesla consultant Prasanna Malaiyandi.
Speaker:How's it going Prasanna
Speaker:persona
Prasanna Malaiyandi:Pana.
Prasanna Malaiyandi:I am doing well Curtis, and how have you been?
Prasanna Malaiyandi:W. Curtis Preston: I, well, as you know, I have been.
Prasanna Malaiyandi:Fiddling.
Prasanna Malaiyandi:Uh, why am I not surprised?
Prasanna Malaiyandi:W. Curtis Preston: I have been, fiddling with the automations of my Tesla now.
Prasanna Malaiyandi:Tesla's come in, I'm gonna say two battery flavors, right?
Prasanna Malaiyandi:There's NMC and LFP.
Prasanna Malaiyandi:Nickel Manganese Cobalt and Lithium Iron Phosphate.
Prasanna Malaiyandi:And I have the latter.
Prasanna Malaiyandi:And the latter is not supposed to be subject to the same don't
Prasanna Malaiyandi:charge it to a hundred percent unless you need it right away.
Prasanna Malaiyandi:Issue.
Prasanna Malaiyandi:Uh,
Prasanna Malaiyandi:In fact tell
Prasanna Malaiyandi:you to charge it to a hundred
Prasanna Malaiyandi:W. Curtis Preston: I do tell you to charge it to a hundred
Prasanna Malaiyandi:percent at least once a week.
Prasanna Malaiyandi:Um, but I was charging it to a hundred percent every day.
Prasanna Malaiyandi:And I was thinking that I, that even though it doesn't have necessarily
Prasanna Malaiyandi:the same issues, I could still get better battery life by not charging to
Prasanna Malaiyandi:a hundred percent unless I needed to.
Prasanna Malaiyandi:Uh, which, uh, for me is not all the time.
Prasanna Malaiyandi:And so, but I'm also very absent-minded, and so I couldn't like lower the
Prasanna Malaiyandi:percentage and then remember to, to, to, to raise the percentage later.
Prasanna Malaiyandi:And so, uh, I found this wonderful app called Tessie, and I've been obsessing
Prasanna Malaiyandi:over it for about a week or two.
Prasanna Malaiyandi:A week.
Prasanna Malaiyandi:It's been nine days.
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:I will say if there's any listeners that are a Tesla owner, I.
Prasanna Malaiyandi:Uh, you should just get Tessie, uh, T-E-S-S-I-E in the app store.
Prasanna Malaiyandi:I bought it for the automation.
Prasanna Malaiyandi:What I also got was this immense amount of analytics and reporting
Prasanna Malaiyandi:and, um, all this great, great stuff.
Prasanna Malaiyandi:And also reminders.
Prasanna Malaiyandi:That's another, speaking of being absent-minded, it will tell me, hey.
Prasanna Malaiyandi:Idiot, you're home and your car's not plugged in.
Prasanna Malaiyandi:Yeah, because how many times has that
Prasanna Malaiyandi:happened to you, Curtis, that you've come home and you're like,
Prasanna Malaiyandi:W. Curtis Preston: you know, a couple, uh, enough that it was annoying.
Prasanna Malaiyandi:Uh, it, I will say it, nothing is more annoying than, you know, basically
Prasanna Malaiyandi:driving your car down to, you know, the electrical equivalent of fumes.
Prasanna Malaiyandi:And then, um, and then getting up in the morning and going,
Prasanna Malaiyandi:okay, I'm ready to drive today.
Prasanna Malaiyandi:And you're like, oh, crap, I gotta go to the.
Prasanna Malaiyandi:I gotta go the supercharger for 20 minutes before I could do anything else.
Prasanna Malaiyandi:Um, yeah, so it's got, you know, it's got that the notifications, it
Prasanna Malaiyandi:reminds me to, to rotate my tires.
Prasanna Malaiyandi:And also gives me analytics about my driving and my efficiency and, yeah.
Prasanna Malaiyandi:So, yeah.
Prasanna Malaiyandi:So happy, happy, happy, happy, happy.
Prasanna Malaiyandi:Um, but you know, who's not happy?
Prasanna Malaiyandi:The people that are gonna be in this new series, they're not happy.
Prasanna Malaiyandi:And this is a new series.
Prasanna Malaiyandi:One of the things that you've heard us say is what Bana.
Prasanna Malaiyandi:Back up the cloud.
Prasanna Malaiyandi:W. Curtis Preston: Yes, backup up the cloud.
Prasanna Malaiyandi:The cloud is not magic.
Prasanna Malaiyandi:There is no such thing as the cloud.
Prasanna Malaiyandi:It's just somebody else's computer.
Prasanna Malaiyandi:Uh uh, the cloud doesn't magically back up itself.
Prasanna Malaiyandi:Despite what you may have been told the cloud there are parts of the cloud where.
Prasanna Malaiyandi:People generally agree with me.
Prasanna Malaiyandi:I, I think probably the best example would be something like AWS, you know, like EC2.
Prasanna Malaiyandi:They're like, yes, we know EC2 needs to be backed up.
Prasanna Malaiyandi:Um, and, and then we just argue a little bit over how that's going to be done.
Prasanna Malaiyandi:But when we get to the extreme it, the other end of that, we get the
Prasanna Malaiyandi:SaaS world, we get the Microsoft 365 lovers who say, oh, this, that,
Prasanna Malaiyandi:this doesn't need to be backed up.
Prasanna Malaiyandi:you know, or G
Prasanna Malaiyandi:Suite and, we have gone back over the last, uh, several years and selected, um,
Prasanna Malaiyandi:a number of, you know, poor victims of.
Prasanna Malaiyandi:This belief and, um, I, I can think of no better, uh, story to start this
Prasanna Malaiyandi:out than code spaces because it, I, I think it was kind of the first,
Prasanna Malaiyandi:it was probably the first that was sort of well
Prasanna Malaiyandi:publicized as well as having very dire consequences for not backing up the cloud.
Prasanna Malaiyandi:I.
Prasanna Malaiyandi:W. Curtis Preston: Right.
Prasanna Malaiyandi:And it, it's interesting from a timing perspective, it happened in 2014.
Prasanna Malaiyandi:Which to me is the year of the beginning of the massive level of ransomware.
Prasanna Malaiyandi:Now, I know ransomware actually goes way back longer.
Prasanna Malaiyandi:You know, much longer be before that.
Prasanna Malaiyandi:But to me, 2014 is really when I started seeing ransomware kind of everywhere.
Prasanna Malaiyandi:And this was technically a ransomware attack, not in the traditional sense
Prasanna Malaiyandi:that we think of today, but, but it was, so Code spaces.com was a site, and this
Prasanna Malaiyandi:is the, the irony of all ri ironies.
Prasanna Malaiyandi:And, and I will say that unlike Alanis Morissette, I actually
Prasanna Malaiyandi:know what the word ironic means.
Prasanna Malaiyandi:Um, I love the song.
Prasanna Malaiyandi:It's just, there's so many things to that song that are not in any way ironic.
Prasanna Malaiyandi:They just suck.
Prasanna Malaiyandi:Um, code spaces.com was a site to store your code.
Prasanna Malaiyandi:It was like, um.
Prasanna Malaiyandi:GitHub.
Prasanna Malaiyandi:W. Curtis Preston: It was like, it was like a GitHub and they had
Prasanna Malaiyandi:many, many customers and it was a safe space to store your code.
Prasanna Malaiyandi:Hence the name code spaces.com.
Prasanna Malaiyandi:Yeah, so basically like you mentioned, it was a place
Prasanna Malaiyandi:companies could store their code and this was way back in the day.
Prasanna Malaiyandi:And so it's like, hey, if you just have your own code sitting locally
Prasanna Malaiyandi:on your system, because not everyone was comfortable with the cloud, they
Prasanna Malaiyandi:offered a service that allows you to store your code there and keep it safe.
Prasanna Malaiyandi:And now according to a cash version of their website, because of course
Prasanna Malaiyandi:you can't find anything anymore about them, uh, they had over 200 customers.
Prasanna Malaiyandi:A week or 200 companies a week using their service, which isn't
Prasanna Malaiyandi:like small beans, right?
Prasanna Malaiyandi:And yes, it's not like the thousands of millions, but there are 200
Prasanna Malaiyandi:customers who now no longer have access to their code anymore because
Prasanna Malaiyandi:of what happened in code spaces.
Prasanna Malaiyandi:The other interesting thing is according to their websites, and I will quote
Prasanna Malaiyandi:backing up, data is one thing, but is meaningless without a recovery plan.
Prasanna Malaiyandi:Not only that, a recovery plan and one that is well practiced and
Prasanna Malaiyandi:proven to work time and time again.
Prasanna Malaiyandi:Code Spaces has a full recovery plan that has been proven to
Prasanna Malaiyandi:work and is in fact practiced.
Prasanna Malaiyandi:W. Curtis Preston: That
Prasanna Malaiyandi:do you think about that, Curtis?
Prasanna Malaiyandi:W. Curtis Preston: That sounds really good.
Prasanna Malaiyandi:Um, you know, based on what we know happened, clearly they didn't test for
Prasanna Malaiyandi:all scenarios.
Prasanna Malaiyandi:W. Curtis Preston: Right.
Prasanna Malaiyandi:And they specifically didn't test for cyber attack.
Prasanna Malaiyandi:Um, but.
Prasanna Malaiyandi:the time could you fault them really?
Prasanna Malaiyandi:W. Curtis Preston: I don't know.
Prasanna Malaiyandi:Well, they had poor backup design.
Prasanna Malaiyandi:Just, you know what, what we're
Prasanna Malaiyandi:gonna get to, what we're gonna find out.
Prasanna Malaiyandi:They had full backup.
Prasanna Malaiyandi:They had poor backup design.
Prasanna Malaiyandi:They failed to follow what persona?
Prasanna Malaiyandi:The 3, 2, 1 rule.
Prasanna Malaiyandi:You know, we haven't talked about this in a long, long time.
Prasanna Malaiyandi:Do you wanna quickly mention it to some of our
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:So the 3, 2, 1 role, and by the way, many companies have said
Prasanna Malaiyandi:it's gotta be more than 3, 2, 1.
Prasanna Malaiyandi:Yes, I agree.
Prasanna Malaiyandi:Right?
Prasanna Malaiyandi:These days it has to be more than 3, 2, 1, but if it's not 3, 2, 1, there's no point
Prasanna Malaiyandi:in talking about the other ones, right?
Prasanna Malaiyandi:Having at least three versions on two different media, and the, the,
Prasanna Malaiyandi:the idea here is on, on things that are subject to different.
Prasanna Malaiyandi:Risk profiles, right?
Prasanna Malaiyandi:Maybe it's disco and tape, maybe it's, uh, on-Prem and off-Prem.
Prasanna Malaiyandi:Maybe it's, you know, on-prem and Cloud.
Prasanna Malaiyandi:Maybe it's, um, you know, a different region, et cetera.
Prasanna Malaiyandi:And then one three, the one is make sure that there's something off site.
Prasanna Malaiyandi:Um, they had neither the two nor the one,
Prasanna Malaiyandi:but we're gonna get to that in a minute.
Prasanna Malaiyandi:and, and for those listeners who wanna know more,
Prasanna Malaiyandi:we actually had an episode with the person who coined the term 3, 2, 1, who
Prasanna Malaiyandi:comes from digital photography, in fact.
Prasanna Malaiyandi:And so you should go take a listen to that episode.
Prasanna Malaiyandi:We'll
Prasanna Malaiyandi:W. Curtis Preston: Yeah, we'll put.
Prasanna Malaiyandi:Yeah, we'll put a show notes.
Prasanna Malaiyandi:Peter Krogh.
Prasanna Malaiyandi:Um, great guy.
Prasanna Malaiyandi:So what happened?
Prasanna Malaiyandi:Well, we have that a hacker gained privilege credentials that we still
Prasanna Malaiyandi:don't know how that happened, right?
Prasanna Malaiyandi:They're saying probably through phishing or possibly through, uh,
Prasanna Malaiyandi:stored EC2 access keys in a public code
Prasanna Malaiyandi:repository.
Prasanna Malaiyandi:Which happened a lot back then and still does today.
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:The, um, there was a, uh, security researcher at Tripwire that said, that
Prasanna Malaiyandi:this is a problem for people how to, how to, manage authentication codes like this.
Prasanna Malaiyandi:And, uh, they said they, they had seen thousands of EC2 accounts
Prasanna Malaiyandi:abused after storing their EC2 keys in public code repositories.
Prasanna Malaiyandi:Ouch.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Um, but we don't, so we don't know.
Prasanna Malaiyandi:Someone
Prasanna Malaiyandi:got access.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Would it be ironic if Code Spaces was using code spaces
Prasanna Malaiyandi:to store their code and they left a public repository with their EC2 key?
Prasanna Malaiyandi:W. Curtis Preston: That wouldn't indeed be ironic.
Prasanna Malaiyandi:Um, but yeah, so we don't know exactly how this hacker, uh, this,
Prasanna Malaiyandi:you know, bad actor got access to the, to the environment, but they did.
Prasanna Malaiyandi:And the first thing that they did was they started a DDoS attack.
Prasanna Malaiyandi:Uh, what is a
Prasanna Malaiyandi:DDoS attack persona?
Prasanna Malaiyandi:This is a DI distributed denial of service, so
Prasanna Malaiyandi:you have a bunch of servers outside hammering various servers at a company.
Prasanna Malaiyandi:Causing it, flooding it with a lot of traffic, which then
Prasanna Malaiyandi:causes it to stop responding.
Prasanna Malaiyandi:So if you're code spaces, someone did a DDoS attack on you now, you wouldn't
Prasanna Malaiyandi:be able to serve and function as a service to those 200 paying customers.
Prasanna Malaiyandi:W. Curtis Preston: Yeah, the really interesting thing about this is that the
Prasanna Malaiyandi:DDoS attack was apparently subterfuge.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Um, you know, look over here
Prasanna Malaiyandi:while I, nothing up but
Prasanna Malaiyandi:which I think is still common today, right?
Prasanna Malaiyandi:There are still a lot of companies who they're trying to hide their tracks
Prasanna Malaiyandi:and they're like, Hey, everyone's gonna fight, and DDoS were common.
Prasanna Malaiyandi:And so they have a plan in place and everyone's scrambling there where you're
Prasanna Malaiyandi:like, Hey, look at my left hand while I'm doing something with my right.
Prasanna Malaiyandi:W. Curtis Preston: And then I, I think the, I don't know if interesting
Prasanna Malaiyandi:is right, but the hacker left contact details for themselves in
Prasanna Malaiyandi:the customer's Amazon dashboard.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Uh, this was, this was, I, I think, I think maybe we haven't mentioned it.
Prasanna Malaiyandi:This was an AWS customer.
Prasanna Malaiyandi:And this is why we call it ransomware, right?
Prasanna Malaiyandi:Even though it's not your traditional ransomware like we
Prasanna Malaiyandi:think today, it more or less is.
Prasanna Malaiyandi:W. Curtis Preston: Right.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:So then, uh, I'll quote from there.
Prasanna Malaiyandi:There was a page, by the way, there was a page.
Prasanna Malaiyandi:Uh, basically code spaces.com died after this, and it became a, they,
Prasanna Malaiyandi:they replaced it with just one page that says, here's what happened.
Prasanna Malaiyandi:And we're, I'm gonna read quotes from that page that are no longer
Prasanna Malaiyandi:available because they sold the domain.
Prasanna Malaiyandi:Um, upon realizing that somebody had access to our control panel, we started
Prasanna Malaiyandi:to investigate how access had been gained and what access that person
Prasanna Malaiyandi:had, uh, to the data in our systems.
Prasanna Malaiyandi:It became clear that so far no machine had access had been achieved due to the
Prasanna Malaiyandi:intruder not having our private keys.
Prasanna Malaiyandi:Um, that's what they thought, right?
Prasanna Malaiyandi:Um, but things
Prasanna Malaiyandi:turned, uh, yeah, turned ugly.
Prasanna Malaiyandi:Uh, why don't you talk, talk about
Prasanna Malaiyandi:the next one there.
Prasanna Malaiyandi:So what was the response?
Prasanna Malaiyandi:So Code Spaces did a smart thing and one of their first response.
Prasanna Malaiyandi:Actions was to change all of its EC2 passwords, but quickly, code
Prasanna Malaiyandi:spaces discovered that the attacker had created backup logins, which any
Prasanna Malaiyandi:sensible person's going to do, right?
Prasanna Malaiyandi:You're never gonna say, Hey, I'm only gonna have one admin in my environment.
Prasanna Malaiyandi:And so the attacker was able to create all these backup logins, and so now they
Prasanna Malaiyandi:were able to just go back into the system.
Prasanna Malaiyandi:And continue doing the attacks.
Prasanna Malaiyandi:And once they realized that Code Spaces was trying to actually recover and take
Prasanna Malaiyandi:control from the attacker, the attacker then started to go and just delete things
Prasanna Malaiyandi:from the control panel because they had super user access at that point.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:So they could do whatever they wanted.
Prasanna Malaiyandi:W. Curtis Preston: Yeah, somewhere in here, in the version of the story
Prasanna Malaiyandi:that I have, I remember there was a d, there was a, there was an attempt
Prasanna Malaiyandi:at a ransom, basically give us this amount of money, or, or, or we're
Prasanna Malaiyandi:gonna, you know, do bad things.
Prasanna Malaiyandi:Um, but the, but you know, they, they obviously didn't want to pay the ransom.
Prasanna Malaiyandi:And then, uh, then what
Prasanna Malaiyandi:happened?
Prasanna Malaiyandi:And so finally code spaces got their control
Prasanna Malaiyandi:panel access back, but not before.
Prasanna Malaiyandi:The attacker had caused quite a bit of damage, so the attacker had gone
Prasanna Malaiyandi:removed all the EBS syn snapshots, S3 buckets, all the amis, which are the
Prasanna Malaiyandi:Amazon machine instances, some EBS instances and several machine instances.
Prasanna Malaiyandi:And this is a quote from the same webpage that Curtis was talking about.
Prasanna Malaiyandi:In summary, most of our data backups, machine configurations
Prasanna Malaiyandi:and offsite backups were either partially or completely deleted.
Prasanna Malaiyandi:W. Curtis Preston: What I, I, you know, I did.
Prasanna Malaiyandi:I never noticed that phrase before.
Prasanna Malaiyandi:And they said And offsite backups.
Prasanna Malaiyandi:What?
Prasanna Malaiyandi:Offsite backups.
Prasanna Malaiyandi:What I mean, what be, because the hacker only had access to
Prasanna Malaiyandi:the one account.
Prasanna Malaiyandi:I am guessing that they replicated
Prasanna Malaiyandi:to another AWS region within the
Prasanna Malaiyandi:W. Curtis Preston: Okay.
Prasanna Malaiyandi:Okay.
Prasanna Malaiyandi:Another zone within the
Prasanna Malaiyandi:same account.
Prasanna Malaiyandi:All right.
Prasanna Malaiyandi:Or or another region.
Prasanna Malaiyandi:Yeah,
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:yeah, But still within the
Prasanna Malaiyandi:same account
Prasanna Malaiyandi:Possibly within the same account.
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:Um, yeah, yeah, possibly you're right.
Prasanna Malaiyandi:We don't know for sure, but if it wasn't the same account, then the hacker had
Prasanna Malaiyandi:to gain access to multiple accounts.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Yep.
Prasanna Malaiyandi:W. Curtis Preston: Um, yeah.
Prasanna Malaiyandi:So basically this is the equivalent of blowing up somebody's data center, right?
Prasanna Malaiyandi:Um, because basically they just.
Prasanna Malaiyandi:In a matter of a few keystrokes, they just deleted essentially everything, right?
Prasanna Malaiyandi:Everything that mattered, or enough things that mattered that they, um,
Prasanna Malaiyandi:you know, uh, took out the company.
Prasanna Malaiyandi:One of the things that I don't know if you found any
Prasanna Malaiyandi:information about Curtis is did they ever reach out to law enforcement
Prasanna Malaiyandi:or even AWS's security operations to be like, Hey, I have this issue.
Prasanna Malaiyandi:Can you help me?
Prasanna Malaiyandi:W. Curtis Preston: Yeah, we don't, I, I, I'm going to.
Prasanna Malaiyandi:Assume that once you know, the feces hit the rotary oscillator,
Prasanna Malaiyandi:I'm sure they called AWS.
Prasanna Malaiyandi:I mean, of course they called AWS, but what we know is that
Prasanna Malaiyandi:right, if you know AWS isn't magic, and if you didn't follow the architecture
Prasanna Malaiyandi:and do the things that you were supposed to do, A-W-A-W-S can't
Prasanna Malaiyandi:undo that.
Prasanna Malaiyandi:Yeah, well, I was just wondering before it changed, its EC2
Prasanna Malaiyandi:passwords, if they had reached out to AWS, if they could have helped them in some
Prasanna Malaiyandi:way or been like, Hey, here are the best practices for locking down your account,
Prasanna Malaiyandi:W. Curtis Preston: Yeah, I'm, I'm gonna guess based on how they responded
Prasanna Malaiyandi:that they did not do that right.
Prasanna Malaiyandi:Um, I don't think their response was the best thing they
Prasanna Malaiyandi:could have done at the time.
Prasanna Malaiyandi:Um, so then we have, um.
Prasanna Malaiyandi:So here's again, I'm reading from their co their, from their quote
Prasanna Malaiyandi:here, code spaces will not be able to operate beyond this point.
Prasanna Malaiyandi:The cost of resolving this issue to date and the expected cost of refunding.
Prasanna Malaiyandi:Customers who have been left without the service they paid for, we'll
Prasanna Malaiyandi:put code spaces in an irreversible position, both financially and
Prasanna Malaiyandi:in terms of ongoing credibility.
Prasanna Malaiyandi:No kidding.
Prasanna Malaiyandi:Um, as such, at this point.
Prasanna Malaiyandi:We have no alternative but to cease trading and concentrate on supporting
Prasanna Malaiyandi:our affected customers and exporting any remaining data they have left with us.
Prasanna Malaiyandi:Ouch.
Prasanna Malaiyandi:W. Curtis Preston: That is, um, that's a tough one.
Prasanna Malaiyandi:So basically, you know, hacker gets in Hacker, uh, you know.
Prasanna Malaiyandi:Offers are ransom.
Prasanna Malaiyandi:They try to, instead of paying the ransom, they try to lock
Prasanna Malaiyandi:the hacker out unsuccessfully,
Prasanna Malaiyandi:Yep.
Prasanna Malaiyandi:W. Curtis Preston: and then the hacker deletes the company, the uh, right.
Prasanna Malaiyandi:I mean, they deleted basically everything, you know, as much as they could get
Prasanna Malaiyandi:access to in that, you know, in that
Prasanna Malaiyandi:account.
Prasanna Malaiyandi:That's crazy.
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:This is one of those where it's like, I, I always used this story to
Prasanna Malaiyandi:recommend backup design for the cloud.
Prasanna Malaiyandi:Even though I don't think I've ever encountered someone who says,
Prasanna Malaiyandi:well, I don't need to back up EC2, I don't think I've ever heard that.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:W. Curtis Preston: I have very commonly found people whose backup design was very
Prasanna Malaiyandi:similar to this company's backup design.
Prasanna Malaiyandi:And, um, so, well let me ask you this.
Prasanna Malaiyandi:What do you think, um, what could they have done differently to stop this?
Prasanna Malaiyandi:To prevent this from happening.
Prasanna Malaiyandi:I would say the first could have been take your
Prasanna Malaiyandi:backups to a different account.
Prasanna Malaiyandi:In a different region, or even in the same region, that would've
Prasanna Malaiyandi:at least kept your data safe.
Prasanna Malaiyandi:W. Curtis Preston: Yeah, I, I, yeah, I, I completely agree with you.
Prasanna Malaiyandi:I would probably just say as long as you, as long as you're, I, I guess what.
Prasanna Malaiyandi:I'm, I'm trying to, I'm factor, I'm thinking in my head, like,
Prasanna Malaiyandi:from a cost perspective, does it cost extra to send to another
Prasanna Malaiyandi:region?
Prasanna Malaiyandi:Is that,
Prasanna Malaiyandi:Prasanna Malaiyandi: Yeah, usually it does.
Prasanna Malaiyandi:Yep.
Prasanna Malaiyandi:W. Curtis Preston: Okay.
Prasanna Malaiyandi:So
Prasanna Malaiyandi:But they already have an offsite backups,
Prasanna Malaiyandi:right?
Prasanna Malaiyandi:So.
Prasanna Malaiyandi:W. Curtis Preston: right?
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:So.
Prasanna Malaiyandi:Um, I, I think that your backup should be copied to another
Prasanna Malaiyandi:region and another account.
Prasanna Malaiyandi:I, I actually, and, and, and, and I'll say that, you know, my, my
Prasanna Malaiyandi:opinion is somewhat peppered by having worked for a company that does this.
Prasanna Malaiyandi:But there are companies that will backup your cloud data and then get it out
Prasanna Malaiyandi:of the cloud into their cloud, and I think that's as secure as it can be.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:And I think that it should be then stored in some type of immutable type offering.
Prasanna Malaiyandi:Um, basically get it, get, get it out of the region for security against bad things
Prasanna Malaiyandi:that might happen that aren't hackers.
Prasanna Malaiyandi:And then get it out of the account to secure it against hackers and the.
Prasanna Malaiyandi:If that costs you money, figure out a way to do that that costs
Prasanna Malaiyandi:you as little as possible.
Prasanna Malaiyandi:And, and, and I do think the companies that can back up, take the, the
Prasanna Malaiyandi:incremental data and then maybe de-dupe it before they pull it out.
Prasanna Malaiyandi:If they can do that, you can minimize the egress cost of moving it out.
Prasanna Malaiyandi:Um, the, the other thing, um.
Prasanna Malaiyandi:Yeah, so, so they didn't have it in a different account.
Prasanna Malaiyandi:They didn't have object lock turned on.
Prasanna Malaiyandi:Um,
Prasanna Malaiyandi:Object lock did not exist back
Prasanna Malaiyandi:then.
Prasanna Malaiyandi:By the way.
Prasanna Malaiyandi:W. Curtis Preston: what's that?
Prasanna Malaiyandi:Object lock didn't exist back then, so.
Prasanna Malaiyandi:W. Curtis Preston: Okay, so we won't, we won't blame them for that.
Prasanna Malaiyandi:But, but, but since object lock exists now, we'll say this is what you should do.
Prasanna Malaiyandi:Uh, but you know what did exist back then that they did not use multifactor
Prasanna Malaiyandi:authentication?
Prasanna Malaiyandi:Oh, for their access into their admin account or into their
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:So if somebody gains access to your admin keys and they're able to log in.
Prasanna Malaiyandi:If you don't have MFA you, you have zero protection against someone
Prasanna Malaiyandi:either stealing or accidentally, you know, inadvertently getting access
Prasanna Malaiyandi:to administrative level keys and, uh.
Prasanna Malaiyandi:I mean M-F-A-M-F-A-M-F-A mfa, I mean, how many, how many times
Prasanna Malaiyandi:do we have to say it right?
Prasanna Malaiyandi:Um, good password management, MFA and, uh, patch management.
Prasanna Malaiyandi:We, we say this all the time.
Prasanna Malaiyandi:If you just did those three things, you'd stop roughly 90% of attacks.
Prasanna Malaiyandi:And in this case, if they had had MFA, this, uh, bad actor
Prasanna Malaiyandi:would not have been able to
Prasanna Malaiyandi:gain access.
Prasanna Malaiyandi:Yeah, and and I think it's important to say
Prasanna Malaiyandi:just because you use MFA and patch management and everything else.
Prasanna Malaiyandi:Doesn't mean that you don't need backup, you still need backup because
Prasanna Malaiyandi:that is how you are gonna recover from this, plus other issues as well.
Prasanna Malaiyandi:The security side of things just sort of helps to protect you from
Prasanna Malaiyandi:letting the hackers in to some extent.
Prasanna Malaiyandi:Not gonna be a hundred percent foolproof, but hopefully, like
Prasanna Malaiyandi:Curtis said, protects you.
Prasanna Malaiyandi:And 80, 90% of the cases.
Prasanna Malaiyandi:W. Curtis Preston: Yeah, to borrow from and totally abuse
Prasanna Malaiyandi:a quote from Shakespeare.
Prasanna Malaiyandi:There is more on heaven and earth than that is dreamt of.
Prasanna Malaiyandi:In your philosophy, there are many, many ways that your
Prasanna Malaiyandi:data can be attacked, deleted.
Prasanna Malaiyandi:Set on fire exploded.
Prasanna Malaiyandi:Sucked into a sinkhole.
Prasanna Malaiyandi:There's so many different things that can happen to your data.
Prasanna Malaiyandi:That's why you have backup.
Prasanna Malaiyandi:And backup protects against all of them.
Prasanna Malaiyandi:And, and we're saying backup and Dr and all of those things that come with it.
Prasanna Malaiyandi:Um, but uh, the other thing that they also didn't do was this idea
Prasanna Malaiyandi:of, um, you know, least privileged.
Prasanna Malaiyandi:Do you
Prasanna Malaiyandi:want to talk about that?
Prasanna Malaiyandi:Yeah, so normally you do not want, in a company, you don't
Prasanna Malaiyandi:want the intern to have the same level of access as your CEO or your IT admin.
Prasanna Malaiyandi:And so you wanna be able to say, Hey, whatever access a person needs
Prasanna Malaiyandi:to something, that's all they should have access to and nothing else.
Prasanna Malaiyandi:And so you wanna have, make sure that you are focused on that and
Prasanna Malaiyandi:don't just say, Hey everyone, you guys have admin credentials so you
Prasanna Malaiyandi:can do anything and everything.
Prasanna Malaiyandi:Because if one person who inadvertently gets compromised,
Prasanna Malaiyandi:now everything is exposed.
Prasanna Malaiyandi:So you wanna scope down their access to only what they need and that's it.
Prasanna Malaiyandi:W. Curtis Preston: Exactly.
Prasanna Malaiyandi:Now, when, when reading the, uh, articles about this, one of
Prasanna Malaiyandi:the things that other people I.
Prasanna Malaiyandi:Uh, dinged this company for, was that they didn't have an established
Prasanna Malaiyandi:procedure for locking down the account.
Prasanna Malaiyandi:And when I thought about that, I just found myself wondering.
Prasanna Malaiyandi:I.
Prasanna Malaiyandi:Huh?
Prasanna Malaiyandi:How exactly would that happen?
Prasanna Malaiyandi:And the best that I could find, you know, how, how would you do that?
Prasanna Malaiyandi:And the best that I could find is that you would have a secondary account
Prasanna Malaiyandi:that has access to this account that you then have a procedure to do things
Prasanna Malaiyandi:like, um, disable, I think about.
Prasanna Malaiyandi:You know, what, what is, what is the cloud equivalent to blocking somebody out?
Prasanna Malaiyandi:And I, and I think the, the, the quickest way would be to
Prasanna Malaiyandi:disable particular I am profiles.
Prasanna Malaiyandi:Um, there, there was some, I, you know, and I'm not, I'm not
Prasanna Malaiyandi:an expert on this, uh, I don't think you're an expert on this.
Prasanna Malaiyandi:I would say talk to your cloud company.
Prasanna Malaiyandi:Talk to your cloud provider.
Prasanna Malaiyandi:Ask them, Hey, I am worried that one day my cloud entire environment
Prasanna Malaiyandi:might become compromised.
Prasanna Malaiyandi:How can I automate, basically locking everything out?
Prasanna Malaiyandi:Right?
Prasanna Malaiyandi:Once we determine that a hacker is in our environment, I would really like
Prasanna Malaiyandi:a button that I can press from another account that shuts everything and down.
Prasanna Malaiyandi:And everything like this can be automated and yes, that is a, uh,
Prasanna Malaiyandi:you know what, what, what's the term?
Prasanna Malaiyandi:The, you know, yeah.
Prasanna Malaiyandi:The nuclear option.
Prasanna Malaiyandi:That is the nuclear option.
Prasanna Malaiyandi:But once you have an a hacker in your account to me that that
Prasanna Malaiyandi:would be the proper option.
Prasanna Malaiyandi:Shut everything down.
Prasanna Malaiyandi:Uh, except for like, I would think create a new IAM profile that you can
Prasanna Malaiyandi:use after you've done this, and then nuke everything that isn't that, um.
Prasanna Malaiyandi:And, and that should be automated and that, and I don't think this
Prasanna Malaiyandi:is something we normally talk
Prasanna Malaiyandi:about.
Prasanna Malaiyandi:I don't think it's something we talk about, but
Prasanna Malaiyandi:it has to be out there somewhere.
Prasanna Malaiyandi:I'm sure that AWS or Google or pick your favorite cloud provider,
Prasanna Malaiyandi:they probably have a procedure.
Prasanna Malaiyandi:I think the danger is you don't want it too automated because there's also the
Prasanna Malaiyandi:risk that a hacker or someone else could trigger that and shut down your company.
Prasanna Malaiyandi:So it's sort of one of those nuclear options.
Prasanna Malaiyandi:So you don't wanna make it too easy.
Prasanna Malaiyandi:But I'm sure that
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:a.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:W. Curtis Preston: use the nuclear option to.
Prasanna Malaiyandi:This is one of those where this, this is the nuclear button, and so you just,
Prasanna Malaiyandi:maybe you have an account that just does this and that account is like
Prasanna Malaiyandi:completely separate from everything else.
Prasanna Malaiyandi:Like we, we talk about having an account.
Prasanna Malaiyandi:That is the backup account, right?
Prasanna Malaiyandi:That is used for backups and no one ever logs into this account.
Prasanna Malaiyandi:And you create it in such a way that if anyone ever does log in,
Prasanna Malaiyandi:does log into the account, it sets off alerts everywhere and Right.
Prasanna Malaiyandi:It, it can be like a honeypot account, but this account, um, yeah,
Prasanna Malaiyandi:I'm not sure how to do, again, I'm not an expert in this, but I would,
Prasanna Malaiyandi:I would create a separate account.
Prasanna Malaiyandi:I would make that account as secure as humanly possible.
Prasanna Malaiyandi:Again, ask your cloud provider how to do that, uh, the, the best way to do that.
Prasanna Malaiyandi:But I just noticed that everybody said that almost everybody, they're like, they
Prasanna Malaiyandi:should have had procedures for what to do in this situation, and, and they didn't
Prasanna Malaiyandi:have them.
Prasanna Malaiyandi:I think
Prasanna Malaiyandi:perhaps,
Prasanna Malaiyandi:right?
Prasanna Malaiyandi:W. Curtis Preston: what's that?
Prasanna Malaiyandi:At least manual procedures, right
Prasanna Malaiyandi:W. Curtis Preston: At least manual procedures, right?
Prasanna Malaiyandi:Because I don't think that changing the passwords on IM profiles was
Prasanna Malaiyandi:the, the quickest way to do that.
Prasanna Malaiyandi:Right?
Prasanna Malaiyandi:I think the thing they should have focused on perhaps was kicking out,
Prasanna Malaiyandi:currently logged in sessions, uh.
Prasanna Malaiyandi:Yeah, I don't know.
Prasanna Malaiyandi:And the problem is, it's like, how, how do you, you, you have to,
Prasanna Malaiyandi:you have to build your incident response around your environment.
Prasanna Malaiyandi:And one of the things that they could have done is maintain an inventory of,
Prasanna Malaiyandi:um, basically privileged, super privileged accounts and look and see if there
Prasanna Malaiyandi:were any new ones.
Prasanna Malaiyandi:Well, that's the thing I was going to mention is they
Prasanna Malaiyandi:should at least have had monitoring.
Prasanna Malaiyandi:When someone adds a super privileged user, they should have been flagged
Prasanna Malaiyandi:about that immediately, right?
Prasanna Malaiyandi:Because that's not a common occurrence.
Prasanna Malaiyandi:W. Curtis Preston: Right.
Prasanna Malaiyandi:And so monitoring, alerting is also looks like something
Prasanna Malaiyandi:that people should be doing to catch these sort of issues as well.
Prasanna Malaiyandi:W. Curtis Preston: Yeah, and, and, and I'd love, by the way, I'd love
Prasanna Malaiyandi:other suggestions from listeners.
Prasanna Malaiyandi:I would love to hear from you.
Prasanna Malaiyandi:If you go to backup wrap up.com, there's actually a button on there
Prasanna Malaiyandi:that you can leave voicemails.
Prasanna Malaiyandi:You can send us notes and, uh, you know, you know, if you'd like, we can actually
Prasanna Malaiyandi:even play your response on the air.
Prasanna Malaiyandi:I would love to hear better suggestions than we have from a security perspective
Prasanna Malaiyandi:because I'm, you know, I'm Mr.
Prasanna Malaiyandi:Back.
Prasanna Malaiyandi:I'm, I'm not Mr.
Prasanna Malaiyandi:Security.
Prasanna Malaiyandi:I, I play, I play a security on tv.
Prasanna Malaiyandi:Um, the, uh, but, but, but, uh, the summary statement.
Prasanna Malaiyandi:From a backup perspective, if they had simply followed the 3, 2, 1 rule, if they
Prasanna Malaiyandi:had made another copy of their backups in another account, in another location,
Prasanna Malaiyandi:if they had used a cloud provider to do this for them so that then a copy of all
Prasanna Malaiyandi:their data was stored in, in a completely different company, if they had done any
Prasanna Malaiyandi:of those things, they would've at least had a copy so that once they got on the
Prasanna Malaiyandi:other side of the attack, they could have.
Prasanna Malaiyandi:Then recovered all the data because that's the, the true disaster here.
Prasanna Malaiyandi:As that once they've been attacked and once the attacker gained access
Prasanna Malaiyandi:to their account, they were able to delete all their data, both
Prasanna Malaiyandi:their primary and their backups
Prasanna Malaiyandi:So I know we've been focused solely on code space as the
Prasanna Malaiyandi:company, but I think there's also blame to be placed on those 200 companies who
Prasanna Malaiyandi:were using Code Spaces for not also having a backup of their data and relying solely
Prasanna Malaiyandi:on code spaces as their service provider.
Prasanna Malaiyandi:W. Curtis Preston: You know, that is an interesting, we, you know, we
Prasanna Malaiyandi:tend to focus here on the fact that it was a provider, but what this
Prasanna Malaiyandi:really was, was a SaaS provider.
Prasanna Malaiyandi:Yep.
Prasanna Malaiyandi:W. Curtis Preston: Right.
Prasanna Malaiyandi:Yep.
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:So we don't know what happened to those other companies.
Prasanna Malaiyandi:And so this is, this is a double lesson, right?
Prasanna Malaiyandi:If you're, if you're running in the cloud, make sure you've got
Prasanna Malaiyandi:a backup of that, of that data.
Prasanna Malaiyandi:If you're using a SaaS provider, make sure you have another copy of
Prasanna Malaiyandi:the data that you're putting in that SaaS provider, because that would be
Prasanna Malaiyandi:another way for at least the, the, um, the, the, the thing that's difficult.
Prasanna Malaiyandi:Here, again, I agree with you, the thing that's.
Prasanna Malaiyandi:The, the difference here is that unlike many of the SaaS providers, this company
Prasanna Malaiyandi:specifically said, Hey, we got it.
Prasanna Malaiyandi:We got your backups, this data, and it's tested and it's all this stuff, right?
Prasanna Malaiyandi:Um, the, um, you know, I just had a thought.
Prasanna Malaiyandi:Um, if we go to LinkedIn and we search for code spaces.
Prasanna Malaiyandi:Find people that used to work in code spaces.
Prasanna Malaiyandi:Oh, that would be, I wish, I wish we could talk to somebody
Prasanna Malaiyandi:that was involved in this, but
Prasanna Malaiyandi:uh, I think we can
Prasanna Malaiyandi:I'm sure there are probably NDAs.
Prasanna Malaiyandi:W. Curtis Preston: What's that?
Prasanna Malaiyandi:Prasanna Malaiyandi: There's probably NDAs.
Prasanna Malaiyandi:W. Curtis Preston: Oh, they're probably NDA.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:All right.
Prasanna Malaiyandi:Well back up the cloud, I told you so.
Prasanna Malaiyandi:Any final thoughts for you, persona?
Prasanna Malaiyandi:No, I totally agree with that, and I like this because I
Prasanna Malaiyandi:know we bring up code spaces a lot, so I think that hopefully our listeners now
Prasanna Malaiyandi:understand why we talk about it and what they should not be doing, and why we
Prasanna Malaiyandi:harp so much on things like the 3, 2, 1 rule on MFA, because you don't want your
Prasanna Malaiyandi:company to have to shut its doors because they were unable to recover their data.
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:And sadly, this will not be the last company that.
Prasanna Malaiyandi:You know, basically ceased to exist, uh, because they didn't
Prasanna Malaiyandi:properly back up their data.
Prasanna Malaiyandi:Yeah,
Prasanna Malaiyandi:W. Curtis Preston: All right.
Prasanna Malaiyandi:Well, uh, thanks for, uh, joining me persona, as always.
Prasanna Malaiyandi:anytime and looking forward to see your analytics on your car.
Prasanna Malaiyandi:W. Curtis Preston: I will see what I can do and I will also thank our
Prasanna Malaiyandi:listeners, we be nothing without you.
Prasanna Malaiyandi:Thanks for listening.
Prasanna Malaiyandi:And be sure to subscribe so you don't miss an episode.
Prasanna Malaiyandi:That is a wrap.