as 2022 comes to an end.

This is our year-end review episode, where we talk about, I think

about a dozen episodes that really hit home for us during the year.

I hope you enjoy it.

And I hope you have a great 2023.

Hi, and welcome to Backup Central's Restore it All podcast.

I'm your host, w Curtis Preston, aka Mr.

Backup.

And I have with me the guy that can probably explain why there's a lot more

echo in this room than there normally is.

Prasanna Malaiyandi

Echo.

Echo, How's it

hear it on your end?

Cuz I really hear it on my end.

I can hear it.

It's not as bad, but you could definitely tell it's different

than our normal podcast recordings.

Yeah, although I'm in the same room, although for those of

you that are used to watching the video, these things are normally way far back.

It's because my desk is literally in the middle of the room because,

um, I have prepared for the flooring, uh, the final room, the

flooring of the final room, and the

Woohoo.

Multi-month flooring project and I decided to just leave

my desk set up and I just have to move it around as the flooring progresses.

Which isn't bad.

At least it's all set up and

yeah.

And everything else is outta here.

Um, and then, and the doors are off the closet and, uh, you know, all

it's crazy how much an empty room like, or how much

the stuff in a room helps with echoes.

Right?

Because once you remove it all, it's like, wow,

Yeah.

Well, and also the, I think the big thing is that it was carpet, right?

It was previously carpet and now I'm just looking at concrete, which

So now when you put in flooring, are you gonna put in sound

Well, we'll see how, we'll see how it goes, right?

I think that well, we'll see.

We'll see.

Who knows?

Who knows?

Does wallpaper help?

You can put wallpaper up.

wallpaper, do I look like a wallpaper kind of guy to you?

Um, we'll see.

We'll see what happens once the flooring gets in.

Um, see if it

but it is a final room, which is good.

It is the final room.

And, uh, yeah.

So I'm excited about that.

And I'm sure that the three, the three listeners, uh, that care

about that project will also be excited that I finished this.

And then the other, all the other listeners are gonna be like,

finally he is gonna stop talking about this flooring project

And then it'll just be the next project.

Don't worry listeners.

We got you.

We'll keep things interesting.

Well, speaking of look, so interesting.

So, so the, the true, the full story is that that project literally

started in the January of last year in that, that's when I started.

Ordering the, um, the flooring from Costco.

And it took months due to, I don't know if you've heard, but there

was a big supply chain problem.

Was there I didn't,

but yeah, so, so January, so I thought I was gonna

start this project a lot earlier, but I ordered it in January and I got it

like in, I don't know, April I think.

Um, and then I started it kind of in.

June, something like that.

Um, but anyway, anyway, but we are looking back on 2022, so it's natural that I look

back on what I did the entire year of 2022, which was try to put down flooring

It's okay.

It's, it was a sizeable project, Curtis, you

It was, it was, well it still is.

It still is.

Um,

the question.

Now that you're probably 90% of the way through,

Mm-hmm.

Would you have hired this out to someone to get it

done in a much shorter timeframe or.

I, I will say this is not on the same level as the solar

project, like the solar project.

I regret doing it the way I did.

I don't regret this.

It's just, it's just, it's a lot.

And I saved, like, I don't know, I think based on the pricing I saw,

I saved like four grand in labor.

So, um, Yeah.

Um, although I still, I have to pay for some labor because I

do have to pay somebody to put down the, uh, the baseboard.

But cuz I'm not doing that cuz that's just, that's detail work

that it's just better left as people that know what they're doing.

Um, but, uh, yeah, so I, I'm not regretting it.

Um, I, I will say this, I'm not gonna do it again and, um, I'm

not gonna do the second floor

Wait, I was gonna ask, what about the stairs?

I am doing the stairs.

I'm gonna do the stairs.

But by the way, when you put vinyl flooring on stairs, you

have to put non tread or, yeah.

Non, non, non-skid.

You have to put tread on it because, uh, people come down and stocking feet and

they just come flying down the stairs.

I don't, I don't want to, I, you know, I don't wanna see my family

go down the stairs like that.

Um,

the stairs with like a slide, you know?

That way it's obvious.

Um, so we're looking back on 2022, right?

And you know, I, I thought, you know, we did, we did roughly 50 episodes.

Um, 50 plus.

I don't know, could we took a week or two off there, here and there for holidays?

Um, one of which was last week, right?

We're recording this on December 29th.

Uh, when I'm looking back on 2022, I one, the first thing that pops

out to me was, do you remember when I had to restore backup central?

Prasanna Malaiyandi:

Oh, I do remember that.

Prasanna Malaiyandi:

Yeah,

That that was an episode, Mr.

Backup forced to restore backup central.com.

Um, I don't remember why.

Do you remember why ? I, I don't remember what happened.

it was, it was.

I think it was your forums got corrupted and you had an issue,

so you had to restore your message

Yeah, there was.

lose all of the back history.

Yeah.

Yeah.

I think you, I think the table, like the tables got corrupted

in the, in the MySQL database.

Yeah.

Yeah.

And so, and I, I think I didn't notice it right away, like it was corrupted for

a few days, and so I had to, I had to go back to my object storage version.

Uh, so that, and, and most importantly it worked.

Um, so I , so I liked that, that episode.

Right.

Uh, that was back in February.

I know you said that it was successful and you were

kind of a little surprised , right?

Um, but had you actually done a restore test at any point in time or

is it one of those things where I was

like, uh, I, hope this works.

It's not the first restore that I've done at Backup Central.

Um, and it wasn't the first restore it, it was maybe the first time.

No, it's not even the first time.

No, no, it's just, um, I think it had been a while since I'd done it.

That's probably what it was.

It had been a

while since I'd done it,

but, um, yeah.

But hey, it

worked.

So

It worked.

so that was a, that was a

That's all that matters in the end.

Yeah.

But speak, yeah.

Speaking of which, maybe you can talk about this, uh, the Restore

test failed, uh, podcast that we had, um, due to bad documentation.

Oh my gosh, this happens all the time.

I.

Gary Williams.

Yeah.

this happens all the time where it's like, okay, I know

that we have all this documentation.

I think in that episode, correct me if I'm wrong, but I think it was that

like they'd forgotten to include like an active directory server or some

other piece of their infrastructure.

As part of the backups that had happened.

And so when they went to restore it and do their DR test, it wasn't like they

didn't have the bits to be able to first authenticate before they can actually

do the restores and bring everything up.

And so their DR test failed.

But luckily they found it out while they were doing the actual test

and not when they actually had to do a failover to recover from an.

And that just goes back, I know Curtis, you always talk about this too.

It's like make sure you test your dr.

Make sure you have everything done right.

And make sure that it's not the same person doing it,

which is kind of what happened there.

I think he, wasn't he a new person?

He was a

new person, or it was something, he was new and he tested it

and, and it didn't work well.

Uh, but

that's right.

Yeah.

It was

they tested it because they figured it out and Yeah.

yep.

Yeah.

I

like, I liked that episode

He was new and it's like you always say it's like a poor job.

Yep.

Yeah.

Yeah.

And we, and then we had, we had a couple of great episodes with a

guy that we refer to as Snorkel 42, because that's his Reddit handle.

Right.

Um, he, he's not, it's funny, he, you know, he goes on video, uh, we know

who he is, but he just prefers to go by that by the handle, and that's fine.

Um, but we, he, he, there was the big Okta hack.

which you may recall

from, uh, when was that?

I'm looking

that was like, I wanna say

half look, looks like it was in March.

Yeah.

Yeah.

Um, and yeah, he ripped a, a new one.

That's what, what I recall.

and, and, and I think he, he was unhappy with the fact

that they weren't transparent about what happened and it took so long

before they shared any information.

Right.

I don't think he was blaming necessarily what had happened, that they had

maybe had a fault in security.

I think it was they could have done a better job of messaging,

informing people what had gone on.

But cuz I think for weeks people were in the.

They didn't know what happened and

you,

yeah, you, you, you fast forward up to another interesting episode

that we did just like last week.

The one about the, um, the.

Darn it.

Um, what's it called?

Rackspace?

The Rackspace

outage.

They, that, that's still going on, right?

It started on December 2nd.

It's still ongoing.

And, uh, they have been as transparent as milk.

Um,

I was gonna say like a black hole, but

Yeah.

Yeah.

Um, they, you know, Yeah, that's just a med.

And, and, and, and we, we talked about the same thing with the Uber, the Uber story.

That was

another great story where we talked about lessons that we can learn.

And again, it's like, Transparency matters.

Um, now unfortunately, there's a LastPass story.

We we're doing a review of 2022, but also in 2022 was the LastPass hack, which we're

gonna do an episode on, I think next week.

But, um, that, so on the one hand they're being pretty transparent.

On the other hand, it's gonna really hurt them as a company.

But, um, you know, there are people who are saying You need to drop last

pass.

Um,

but, um, so let's see what else, what else

So, so the other thing, the other thing also with snorkel,

so I know we did a couple episodes, right?

I know we talked about the Octa.

The other thing though that I think he did a really good job with is the basics

that you should be thinking about from a security perspective to try to prevent

the damage from ransomware, right?

Or bad actors,

exactly.

Exactly.

Just.

Just assume you're going to get ransomware and then do these things

in your environment so that when you get ransomware, it will be

able to do little to no damage.

Yeah, and it was a lot of common sense things

that you didn't have to s shell out a lot of money for, right?

But it was things that you should be doing from a best practices perspective,

and I really like that episode as well.

Absolutely.

That was, I, I, yeah.

And you know, it's funny, uh, just in terms of like fun episodes, the one

right after that one, the one with Virginia Nichols, the one where you,

and I said like three words, um, that was about preparing for an emergency,

not so much a disaster, but that, that was just like, in terms of when I look

back on episodes, it just, you know, made me, I don't know, made me laugh.

I guess it was a

fun episode because that woman.

A powerhouse.

Um, you know, and literally we were just like, how do you prepare for an emergency?

And then she talked for 45 minutes.

Right.

Um,

, that was, that

And, and it's things that you don't even think about, right?

It's like an emergency in your community, right?

Not necessarily like a

computer outage, right?

It was, Hey, you have a fire, you have a tornado that goes

through your neighborhood.

What are the things you need to do?

How do you prepare?

How do you make sure you have a communications plan?

So all the things that people should

That was, yeah.

And, and, and she approaches it both from a personal standpoint,

uh, as well as a, um, a business

standpoint.

Right.

Prasanna Malaiyandi:

yeah, the risk, right?

yeah.

Yeah.

And then there was an episode where we had, what I'm gonna refer to as the

most mentioned person this podcast.

Do you know who I'm talking about?

Mr.

Stuart Little, I love Stuart.

He always comes up

comes up just a lot.

Yeah, he just, yeah, he just comes up.

Uh, and so basically we finally got him on, he's retired now.

Um, and, uh, you know, and we talked about his, you know, so his backup practitioner

tells stories from the trenches.

That was a lot of fun.

Uh, cuz I've known Stuart for a long time.

He has, he has more than a few Curtis stories right?

That he likes

to tell.

Where I don't, I don't necessarily fare very well in those stories, but,

it's okay.

It's a learning experience for all of us.

Yeah.

And then well, and then speaking of that, right after that, we sort of started

talking about various technologies that are out there and how sort of.

Disc isn't great for long-term archiving, right.

Or keeping data around for a long time.

Tape is great, but there are downsides with tape as well.

And so there was, uh, this awesome new techno, or not new, but this

technology I never heard about, right?

But that was introduced by another guest of the podcast, Daniel Rose Hill.

Hey Daniel, the backup anorak right uh, talking.

Talking about M disk, right?

And we had the inventor, right, Barry Lunt on the podcast

talking about that technology.

Yeah.

That was pretty cool cuz it like, you know, he was there at the very beginning

and he was able to, Uh, he, he, he set straight what I would think of as a

few, a few misunderstandings of mine.

Right.

Um, remember I thought that like, uh, cuz he's LDS and I

thought that LDS was like part of

the foundation of it, right.

Um, and he's, you know, he is like, oh, we

would've loved to have him as a customer, but that, but that

isn't, that isn't how it happened.

Right.

Um, and, uh, but yeah, that idea of, I, I, I.

I hope that technology continues to do well.

It's one of these things where it's easy to explain and, um, uh, it's also to easy.

It's easy to, I don't know, confuse it, you know what I mean?

Um, there are some sort of knockoffs out there.

Yeah.

Um, and then this next one that, that I'm looking at here,

it was Paul Hadgy from, um,

he's the, the, CEO of Harangi.

So security, right.

I don't think enough people talk about security in the cloud.

And he, he, he has a company that basically this is what

they do is they help people do better security in the cloud.

Right.

Um, and, uh, I think that was a, that was a very interesting.

Speaking of security in the cloud, I think that there's like

a misunderstanding from some customers or some end users that when you move to

the cloud, everything is handled for you.

You don't have to worry about security, right?

You don't have to.

Everything is secure.

You don't have to worry about backups.

Yeah.

, and I think that's a fallacy because it makes the cloud is

so powerful and so flexible that you can easily shoot yourself in

the foot if you're not careful.

Right.

Even look at what Amazon used to do years ago, right?

And they fixed it finally, right?

Anytime you created an S3 bucket, it was publicly available to everyone.

And how many security issues did that cause?

And yet, even though they've changed it, where the default is no

longer enabled by default, right?

Public access enabled by default.

Still people got it wrong and people left buckets open,

which led to breaches and other

right.

It is a hard thing, especially with all these complex

services and things that you normally may not consider when dealing with

an on-premises infrastructure that you can sort of put a wall around and

be like, yep, this is good for me.

When it's in the cloud, you no longer have that ability to necessarily

put the wall in place unless you do things explicitly and different.

Exactly.

Um, and, and then the, the next thing that you know, this is one of

those where it's a topic that comes up a lot, I think, or at least I

bring it up a lot on this podcast.

And it was interesting to talk to somebody who actually does it.

And that is this idea of the, the actual legal.

Side of things, right?

We, we, we had a, a person who specializes in international law, Joseph Dayner,

and, uh, he's an international lawyer and talking about e-discovery and, you

know, the way it works in the US and the way it works in other countries.

It was interesting.

I, I knew that we were, we were different when it comes to other

countries, but it was interesting how he said that, you know, there are

some countries that find the way we.

Discovery, like hilarious.

The idea that you are required, if you've got the evidence in your

backyard, you're required to give it to the person who's suing you.

Right.

And, And, there are countries where, um, I think he said Germany was one of

them, where, where they're just like,

Uh, no, you're not

required to do that, but, but in the US you are.

And so the, the thing that I'm always warning people about is, you know,

backup and recovery and using your backups as archives and storing your

email for a really long period of time.

And then if you're in the US and many other countries that if you're

sued, you then have to provide that.

And, um,

have an example of that, right, Curtis?

You've actually experienced that

Yeah, I have.

And yeah, and it, and it's, and it's, um, you know, I saw it cost

a company millions of dollars.

Uh, I've also seen it, I've also meaning that it cost them millions

of dollars to get the data back.

I've, I've seen it cost a company billions of dollars where they

lost the lawsuit because they were so bad at retrieving their email.

Um, that they, um, the, the judge issued, and we talked about it in

the recording, the adverse inference.

Uh, which is basically the judge is inferring something from your

behavior that is adverse to your case.

And

essentially what this judge said was nobody could be this

bad at restoring their stuff.

so they clearly, they're doing it on

purpose.

And so they lost the lawsuit as a result, and it was a

multi-billion dollar lawsuit.

Uh, so let's talk about, what I think is our final normal recording

of 2022, which is the, the Rackspace, um, you know, recording.

Yeah.

If you haven't heard that one, that's, I think the one, the

episode immediately preceding this one, we just cover what we knew.

A couple weeks ago.

We know more now.

Um,

but it's not any better news though for customers.

w well, no.

Well, no, it's better.

Well, I, well, so two weeks ago we were thinking they had no backups.

, right?

It looks like now they do have some backups.

They, again, just were really bad at restoring them.

Um, and that, uh, and so, and so they really left some people out in

the, you know, uh, so it's, let's,

so they restored people's email services, but by, by moving them over to 365, okay?

Now you can send and receive.

But for weeks of time, many weeks of time, they didn't allow these

people access to their previous email because they didn't, because they

didn't have a good backup of it.

And by good backup, I mean one that you could bring up in, I don't know, a few

days, right?

Well,

we don't yet.

No.

And this is because of the opacity that they have approached

this, what we don't yet know.

the degree to which ransomware like affected their ability to

restore.

We don't know the degree to we, you know, we don't know.

We don't know because they haven't

really said anything.

Um,

the challenge, right, is they haven't said anything.

A lot of people are making some assumptions, right, about what

could go on just based on limited information out there or speculation,

but it's one of those things.

Chris, do you think we will ever really know what happened?

We're just left to just theorize right.

Is the reason they were so slow at restoring that they

were too busy trying to.

, were they still putting

Right.

Um, were, because it looks like they had backups, um, for, for most of

the customers, they had backups.

And interestingly enough, let's just bring this up because I bring this up a lot.

I say that SaaS services, and this is a SaaS service.

Hosted exchange, they are not responsible for your backups.

Right.

And, and and what I normally say is that if they're responsible for your backups,

it will be in your contract, right?

It will be in the terms of service.

Well, guess what It was in their terms of service.

Oh no,

that's what we know now that we didn't know then.

Okay.

Um, I wanna, I wanna do a whole lot other separate episode on this, but yeah, it,

it, it, that's, that's new information.

It was in

there, but what's interesting, what's interesting, you

know what I, you know what

Prasanna Malaiyandi:

We'll do it in the next

episode.

Uh, we'll do it well.

We'll, well, right now we gotta cover last pass.

The next episode will be last pass, and hopefully

after that we'll get some, but yeah, it's in their, it's in their sla,

Yeah.

Oof.

Just based on everything we've talked about though, it.

Mainly has been around sort of how do you recover from bad events that happen.

It might be natural, it could be bad actors.

How do you protect yourself or minimize the risk?

And then how do you recover in situations?

Right.

Just sort of recapping the, a lot of the cases that we've talked about.

Right.

And I don't think it's going away either.

I think 2023 is gonna be more of the.

no, it's not going away.

Join 23.

More the same.

More the same.

Same, but better, different, worse.

I don't know that , but I will

say

Before 2023 gets here, my flooring will be done It will, it will be done.

It will

be done today or tomorrow?

Probably Today.

Yeah.

Um,

well, hey, it's been fun.

We've done another year of this.

Uh, we've been doing how long?

We've been doing this now since 2019, June of 2019.

years.

Yep.

Wow, dude.

And, and, and, and the, the, um, let's see the episode.

This episode will be 184.

I think that's 184 episodes in three and a half years.

Crazy

that that we've figured out how to talk about this for such a long time.

I thank you to the listeners also for listening to us

because without you we wouldn't be here.

Right?

Doing this week in,

Yeah.

Yeah, absolutely.

Absolutely.

And we love the ones that reach out.

We love the ones that, uh, um, by the way, we, we didn't

even do our usual disclaimer.

This is, this is an independent podcast.

This is just me and Prasanna that happened to know each other cuz we

used to work at the same company.

He abandoned us for a different company, So I work at Druva, he works at Zoom.

We're not speaking for either company.

And, uh, this is an independent podcast, a couple of couple of indies.

And, um, yeah.

And so we, we, we want you to rate us, rate the podcast, scroll

down to where you're, you're at.

Uh, does that make sure to follow, you know, subscribe to the episode

or subscribe to the podcast.

Give us some stars, give us some comments, man.

We love the comments and we also would love to have you join the conversation.

Whether you wanna do it privately or publicly, send me an email, uh, w Curtis

Preston gmail or, uh, at WC Preston on

Twitter.

That

much longer.

Yeah.

Yeah.

Um, I don't know.

My recommendation right now is, uh, on Tesla stock is buy, um,

because it's down quite a bit.

We are not financial advisors.

We are not

financial advisors.

Anything you hear mentioned on the podcast is our opinion.

Please speak with your own financial advisor for any advice

not financial advisors.

We're, we're not, uh, we're not

legal advisors.

We're not doctors.

Although we occasionally give opinions regarding all three of those.

They are worth exactly what you paid for 'em.

Um, so anyway, well, thanks again for another fun year

Likewise Curtis, and here's a better 2023.

Good luck with that.

Again, thanks for listening and be sure to subscribe so that you can restore it all.