The thing is, if you want to do a team building exercise,

forget all the assault courses and other things they have, you do get

the team together and do a restore.

Some of the best...

seriously.

It's a bit like, the trust exercises where you lean

backwards and catches you .It's like that.

Hi, and welcome to backup.

Central's Restore it All podcast.

I'm your host.

W.

Curtis Preston, AKA Mr.

Backup and I have with me, my table saw safety, enthusiast, Prasanna Malaiyandi.

How's it going Prasanna?

I'm good, Curtis.

I don't know if I'd call myself a safety enthusiast, but

You don't believe in safety.

no, not at all.

Plus I think you could say I'm a bad influence on you seeing, how much

equipment you've now started to accrue.

Yeah, last night I watched, I don't know.

I'm going to say two solid hours of just table saw safety videos.

Yeah, but it is good for you to refresh your

mind on what table saw safety means.

Yeah.

you do recall that table saw is the reason that this finger is missing

the, or this hand is missing end.

I'm missing the end of the middle finger on my left hand

for those of you listening.

so it's actually really hard for me to watch some of those videos.

Is it like, when you're doing like driver's

education learning to drive, they show what is that red asphalt.

Was that the name of the movie where it's like accidents happen and.

Blood on the asphalt, I think is what that one's called.

I do remember that one, but this one, there's one where a guy actually

shows in the video,, he doesn't have the board completely clear the blade

when he takes his hand off of it.

And it, the blade grabs the board and tosses it essentially at his groin, area.

And the thing is when you watch it, he looks at it one frame at a time.

And the board goes from being on the other side of the blade to his groin

in less than a frame of the video.

And, so that's, one 30th of a second, probably.

yeah.

And he's like, don't do that.

but yeah, it's been interesting, but the thing that's got me super

excited right now, has been this new video editing or just editing tool.

It's both video and audio and, and it's this thing called,

Descript,

Descript.

Yeah.

And it's just.

you sounded so excited when you texted me.

Oh, my God.

it's hard to describe how amazing this tool is, where you input the, in my

case, I'm in, I'm actually, because we're using video clips of these episodes.

I'm inputting the video and I edit the video and then I excerpt

the audio for the audio excerpts.

, It's made mainly for talking head videos like these, right?

Or audio and you input the audio or video, it does, automated transcription,

which gets about 95% accurate.

And then you go through and you obviously, you can correct the things that it

got wrong, but the really amazing part is if you start a sentence and you

change your mind, or you have the, a lot of words going up to that sentence.

All you have to do is highlight those words in the document and

it cuts them out of the video.

It's like magic

It's like magic.

And then if that's not enough magic, the part that I'm super excited

about trying is sometimes you say one word when you meant to say another.

that never happens to you, Curtis.

Like the podcast I was editing yesterday...

. It was you and I talking about 365 and you don't want this

to happen on your worst day.

That's what I meant to say.

But for some reason I said last day, so with this tool, first

off, I train it with my voice.

I literally speak into the microphone, a bunch of stuff.

It can then synthesize my voice.

And I can select that word and change the word last to worst,

and it will put my voice there, a synthesized version of my voice.

So here's a question, Curtis, do we actually

need to have this podcast anymore?

Or can we just have not even just type it out.

Can we just have something auto-generate based on all of our past podcasts and

just have it start creating new podcasts.

It'll just be a recording that says

3, 2, 1 rule over and over.

No, but it's you know how they have, they've trained AI

to now do paintings and things like that.

I wonder if we could basically have,

AI based.

yeah.

first I get it.

I have to get all the audio and then feed that into a thing.

Yeah.

We don't need you and me anymore.

Exactly.

How hard is it to just say backup your stuff, backup all the

stuff and make sure you test your backups?

And then you just do it based off of whatever's

trending on Twitter and the data protection, security space.

And it comes up with a new podcast episode for us.

That may have already happened.

Who knows?

You don't know this is an auto-generated video and auto-generated audio, who

knows, but speaking of testing backups, I was thinking about this concept, as

long as you don't test your backups, your backup is both a complete success

and a complete failure, which reminds me of, the concept of Schrodinger's cat.

I like the former, rather than thinking

about the latter, but that's

Yeah, but,

Prasanna Malaiyandi:

rather than the realist.

So you're familiar with the concept of Schrodinger's cat, right?

Prasanna Malaiyandi:

Based on TV shows, movies,

Okay.

Yeah.

So it's just a concept, as I understand the concept that you have this cat in

a box, and as long as you don't look in the box, the cat is both alive and dead.

But once you look in the box, you will know that the cat is alive or dead.

That's the concept of Schrodinger's cat.

And the reason why this is relevant today is that we have the author of

a blog called Schrodinger's Backup:

when good documentation goes bad.

a blog called Schrodinger's Backup:

He's been in the IT industry almost as long as I have.

a blog called Schrodinger's Backup:

He comes to us from the UK.

a blog called Schrodinger's Backup:

Welcome to the podcast, Gary Williams.

Thank you and thank you for the invite.

I saw that title.

And I was like, I gotta get this guy on the podcast.

Prasanna Malaiyandi:

Curtis was so excited.

Prasanna Malaiyandi:

Gary, you have no idea.

Prasanna Malaiyandi:

This is like one of his favorite topics.

Thank you.

I don't know if I coined the term.

I have seen it used since I'd like to think I coined the term,

but I don't know for certain,

why not?

it might be something that I heard and I just copied because

it's just sounds really cool when it's perfectly accurate, I think.

It was all three or four companies ago.

The lessons we learned still definitely apply today, but this

happened about three companies back.

So about 10 years ago.

So what was your role at the time?

So my role at the time was a senior network engineer or senior

support engineer, something like that.

OK, And you had the, the gall to, to ask about backups.

No, I didn't.

I was overconfident with our backups, let's say so we had the backup

software, I think it was backup exec.

And, we had all the servers being backed up.

We had everything going to dual tapes.

The tapes were going off site.

Everything was working.

Jewel, jewel tapes?

Dual tapes.

We actually had the backups, the software was writing

effectively RAID-1 one backups.

So it was writing to two tapes.

Oh, duel, capes.

Okay.

I heard, for some reason I heard Jewel.

I don't know why.

It's the English accent.

And yeah.

So it's going to two tapes simultaneously.

So the idea was that even if a tape broke, or if something happens to the

backup and we weren't entirely sure of, or you couldn't restore from one of the

tapes, you could then get the other tape and use that tape to do the restore.

So we had all that stuff going on.

And we got all the emails and of course we're getting the emails

saying all the backups are good, everything must be absolutely fine.

Why would we test them?

Why we're busy enough with other tickets and other stuff going on and projects.

We haven't got time to test them.

What's the point?

We know they work.

And so it looks like you were doing all the

right things in terms of setting up backups, Following the 3, 2, 1 rule.

Right?

Making sure your copies were offsite and.

Yeah.

I think that's probably better than maybe

like 70% of the people out there.

Who try to do backups.

You're like doing the right things.

You're like, oh, I'm good to go.

Yeah, absolutely.

As I say, we had the emails, we even checked the emails.

I think we even had a shared folder or something like that, where all

the backups emails went, and if one of us saw that the folder had

an unread one going, we check it.

If there was an error, someone would get a ticket, it would get sorted out.

If the error went on for several days, there would be a conversation.

We will get these things fixed.

where's the problem.

We know our backups are good.

So you, you were a.

You were, I don't know.

I don't know what to call it, but so instead of being a proponent

of testing the backups, you were a proponent of oh, everything's fine.

Unfortunately at that time.

Yes, I was, sitting there and quite fat, dumb and happy going.

We've got the emails, the backups work.

We know they work.

Where's the problem.

I didn't see any issue here at all.

For what it's worth.

I had a similar point in my career and there was a time.

I remember when I was at a company, I won't give the actual name of the

company, but I will just say it's a very, well-known electronics manufacturer.

and I had helped him set up their backup system and I wasn't

there just to do the backups.

I was there to do sysadmin stuff.

And they were a mess.

th this was a, it was a small department in this bigger, electronics company.

It was an interesting department.

They called it.

Simulation modeling and research.

So it was a revolutionary idea at the time of the idea of modeling,

like in a computer, what would happen if you drop this device?

And so they were doing this in a computer.

It was a fascinating new at the time, new field of science.

So I was there to fix a whole bunch of problems.

One of which, for example, was that every workstation, it was all

Unix workstations, and every person had root on their workstation.

And that was the first thing I was going to fix.

But I also set up their backup system and, the backups worked.

So I assumed the restores would work and it was some time.

I was there long enough that I went, I actually, at some

point needed to do a restore.

And I found out that those tape drives were really good at writing data.

And they were completely incapable of reading data.

Again, I don't want.

I'm sure there was something wrong with these drives, but

they were IBM 3590 drives.

Normally IBM drives are top of the line or whatever, but there was something wrong

with these drives that I was completely.

So I guess what I'm saying is you're not alone.

even me who, I've spent my career in this, although honestly, that's

that event is on the list of things that I think back to when.

Yeah.

when I try to get other people to do it.

Absolutely same with me.

the backups that we were taking, as I say, we were only a small

team and we had all the emails.

We had everything in place.

We had the two tape libraries doing the backups.

So we thought we were in a really good position because we had

everything working the way it should.

We even had documentation for how all this stuff was put together.

we actually had to consultancy come in and help us put all this stuff together.

Because at the time I was working for a financial institution, we

had to have certain boxes ticked, and we had those boxes ticked

because we have the documentation.

We had the backups, they were going off site.

They were going off site.

They were being looked after for us.

We even recalled tapes to make sure we could do the process

and no tapes were getting lost.

So we did that level of testing, but what we never actually tested was

actually restoring the data itself.

And it was a bit of an epiphany when we actually had someone come

into the team who a brand new to IT.

Had never worked in IT before.

Always wanted to work in IT.

Was actually employed in the business in a completely different role.

And then he actually said to me, one day, I'd like to move into IT.

I thought he was joking.

It turns out no, he was actually serious.

He was an ex-finance person wanting to move into IT.

So he applied internally, he got the job and he started with us and he started

looking through some old tickets and he was saying things like, why did you

do such and such a change this way?

So there's a whole education thing going on there.

And that's when he asked the question.

When did you test the backups?

What do you mean test them.

We've got the emails.

Look, here, you can see the service.

Here's the tape drives.

Here's the tapes.

We record the tape.

Yeah, sure.

But when did you restore something?

And I, something I won't actually forget because there was this look,

there's only four of us in the IT team.

We were a really small team for a company of about 300 and there's

this look going around the whole office and everyone's going well, we

haven't actually tested them have we?

It's like a light bulb goes off and it's yeah.

It's Ooh.

We like, hang on.

yeah, we should probably test one of they shouldn't we.

Okay.

what should we test and looking back on it, it was a really insane moment

just to think that we've had easy.

I think what actually had the emails was coming in for over a year.

And yes, we'd had the odd backup failure where something a time there, or there

was a fault with one of the tape drives.

These tape drives were quite old.

So they actually had physical SCSI cables that would sometimes play up.

So you had to make sure the SCSI cables were all firmly

attached, the terminator was in.

The good old days.

And.

never had to deal with restores?

Yeah.

And course you had both active and passive, terminators as well.

Yeah, exactly.

we did actually have to do some restores, but we had, a storage array and the

storage provider let us do snapshots.

So 99% of the restores that we needed.

Just copy and paste from the snapshot.

Not a problem.

You deleted that file not a problem.

There it is.

If something was deleted from a desktop, the common response was, we

don't back things up on your desktop.

Sorry.

That's tough.

If you want it backed up, put it onto the server, put it into your

home drive or something like that.

It will get backed up.

So that was the general understood consensus because it was a small company.

Most of the time, this wasn't an issue, and as I say, people deleted a file.

I remember one time we had an Excel file.

That was a real pain because of all these financial macros.

And we restored that from a snapshot.

And it was still corrupt and we had to go back a week or so, we

managed to get the file back and it was working and we actually said

it and I remember it quite well.

We said it within the team.

that was lucky.

We might have actually asked to get the tapes on site and do a restore from

the tapes, but the snapshot worked.

Everything's fine, you know yeah.

Now you've decided, okay, we haven't tested.

Maybe we should actually try doing the test.

How did you decide what to test?

funny enough, it was a new guy.

the discussion was actually, okay, you're the person sitting

there looking through the tickets.

You're looking through the documentation.

You're new to it all.

You want us to prove to you that the restore process works.

We know it does.

Pick something.

And then he sat there and he went, How about the exchange server?

Prasanna Malaiyandi:

Swinging for the fences!

Fine.

So we thought, okay, fine.

we'll get the tapes back on site.

We'll do the restore.

We'll prove that the backups work and we can go back to what we're normally doing.

all the project work, that kind of thing.

We could spend a day on this.

It will be good for us.

Not a problem.

We even went to the documentation and got the documentation out and said,

look, we've got the documentation.

The tapes are coming in.

This is going to be easy.

And it wasn't.

Of course not.

So when you decided to do the restore.

Did you bring down your production or were you like, I'm going to

restore this into a safe spot and

Yeah, we couldn't bring down production because the nature

of the business was that we needed to keep the server up and running.

We actually had a spare server and I think we're maybe had two spare servers.

VMs were just starting to come on the scene and we actually

had a spare server racked.

And the idea was that if we had a server failure, we could take the

physical discs out of one server.

Put it into another server power it on, be back running.

this is also before the days of re replicas.

They were, again, just coming out on a lot of software was super expensive and

You're giving me flashbacks, Gary.

the good old days.

Yeah.

We had this physical server and it had plenty

of disc space to handle this.

So we said, okay, Let's we've not actually even powered this server on.

I don't even think, I think maybe it was powered on when

we bought it and that was it.

So we said we should test that server out anyway.

Yeah.

Let's power it on.

Let's get the data restored to that server and bring exchange up.

We can bring it up in an isolated network.

Do some very basic tests on it, because it was a small team.

We had access to the networking guys.

I'll say networking guys.

We did a little bit of networking age and there was one guy who did a lot

of the really key networking, tasks.

So none of that was a problem.

We didn't have to wait months for tickets or to get done or anything like that.

So we set up this isolated network, we got the tapes on site and we

started doing the restore and that's when it all went horribly wrong.

So who was doing the restore?

I, if I recall, it was actually our help desk guy.

We S we said to him, look, you came up with this.

You put a lot on this guy.

It was his idea.

And you're like, what, if you think testing backups is so

important, why don't you do it?

Pretty much .We did put it on him.

cause it was his idea.

And we said, look, this is a really good exercise for you to do again.

Unfortunately, I'm going to put my hands up to this.

It's a bad thing to have done, but we said, I'm a senior IT person.

I know the backups are good.

here you go.

Here's the tapes.

Here's the documentation.

See you later and off he goes and he comes back.

I think it was about two, three hours later, something like that.

And he went, I can't get this working.

Yeah.

What do you mean you can't get it working.

What's the problem.

And I don't actually recall what the problems, all the problems were, but

I know that the server itself didn't have enough disc space, even though

it was supposed to have the disc space, because the documentation said,

you need partition sizes like this.

And it actually changed since then.

And we didn't realize, and that was really the start of a lot of problems.

Yeah.

first off I will say that even though.

the way you got there.

I like the way you did it.

what to say, even though the way you got there was wrong, the fact that

you, the fact that you had this person.

do it, who wasn't the person, that made the documentation.

That's actually something I push pretty heavily.

And it's an idea that came from back in my days when I was at a bank

and we very much did test restores.

first off we didn't have snapshots.

We didn't have any of that stuff.

And we had 10,000 employees and any one of them was allowed to

call into the help desk and ask for a restore on any given day.

And, so we would get 10 to 15 restores a day.

So we tested pretty regular, but the thing that we buy in that degree, but

the thing that we had to test in the way that you did were these large

server restores, we did a DR test and it was an absolute imperative

from the powers that be was that.

Curtis wrote the documentation.

Curtis cannot be the person actually doing the test.

Curtis needs to be standing back there, listening closely to the problems that

are happening, but, w which, which was actually kind of nice, although

it's nerve wracking to be the person who wrote the documentation and then

sitting there watching someone, you think you've answered all the questions,

but it's not like in this case, you.

you had the classic example of the documentation might've been

correct, but it was out of date.

It was correct at the time, the irony is very similar with you.

I didn't actually write the documentation.

It was written by the contractors and consultants that came on.

Actually signed off on the documentation saying, yes, all

the version numbers are correct.

And I think I'd done a couple of updates.

And then we'd had other changes and the other people

had forgoten or I'd forgotten.

Probably I'd forgotten to update the documentation because we

were busy only a small team.

And so things very slowly on, not just that document, but on every other

document that we had about the environment become out of date and it was this

snowball of errors that had crept in.

And the thing that we realized is actually having no documentation

would have been better because the documentation was lying to us.

this poor guy is sitting there going, I followed steps three, four,

and five, but I can't do step six because step five doesn't work.

What do you mean?

It doesn't work.

And that's when we found that there was a service pack that

was missing from exchange.

So it couldn't go any further and it just kept on building and building like this.

That is an interesting problem.

How do you keep your documentation up to date as you're making these

changes and making sure everyone across the environment knows like where the

documentation is and all the rest of that.

today, we use a Wiki solution for all of our documentation.

The idea behind that of course, is the Wiki is so easy to edit.

But you still don't or sometimes you still don't.

You make a note, I'll do that tomorrow or next week.

So there is still the exact same risk.

And even in my current place, we've seen this with certain, we do testing as well.

We do a lot more testing now than, anywhere I've ever worked before.

And even with a lot of the modern systems with Amazon.

Backups to S3 and all this kind of stuff.

We still test to make sure that everything's correct,

that we know what we're doing.

That those Wiki pages are fully up to date.

we did some AD restore testing not so long ago and we found, not major errors,

but there was a couple of little issues there with the restore process, which

just needed a few corrections in the documentation, just, as like a permissions

era type of thing where we couldn't actually get access to the bucket.

So we had to make some changes there.

So even with all the modern backup software.

It's still so important.

I talked about those DR tests that we did back in the day and.

The, and the fact that we always had someone who wasn't me doing the

tests, and frequent listeners to the podcast will have heard this before.

But if we define a successful restore, as we got from A to Z without having to ask

Curtis, what does this line mean, not a single one of the restores was successful.

so if Curtis ever got, blown up and, whatever, the chances of a restore

going completely without a hitch was, zero, which is why you talked

about updating, there's always little things that you have to update.

I would suggest that original documentation.

and again, take this for what it's worth to anybody who's listening.

the first mistake was writing the documentation in a way that

it can easily get outdated.

our exchange server is 75.

Terra...

r ight.

that's a problem.

So if you're going to hand that to a restore documentation, what it should say

is before beginning the restore, go look at the size of the backups, And figure

out how big the current exchange server is, and then size the volume accordingly.

yeah, that, that line wouldn't have gone out of date as quickly.

it is a real challenge by the way.

this idea of what it's like to update documentation, by the way, back in

the day we were using Wordperfect.

Yeah.

And I remember the official company standard was WordPerfect,

because we could use it on, we had Unix versions of WordPerfect.

By the way, curses spaced WordPerfect.

Not this fancy Windows.

what you'd see is what you get editing stuff.

This was text on a screen.

and I remember getting in a fight over.

There was this one guy that was new and he wanted to use Word

because nobody used WordPerfect.

And we were like, we don't care.

We use WordPerfect here for our documentation.

And if you want your documentation to fit into our documentation,

you will use Wordperfect.

And you will like it.

I remember our first days of moving across

the world where you had the.

Word had the ability to mimic WordPerfect key presses.

So you could transition easily.

Good old days.

Good old days, but I think what you're doing now with the Wiki,

I think that's a much better approach.

It is.

There's permissions list behind it, obviously, so that not everyone

can get access to it, but it's the right people can get access.

but what it means is everyone in the team can get access.

They can all update.

It.

There's a history as well.

So the other thing that we didn't have is the backup of the documentation

was on the server we were backing up.

Oh,

Exactly.

So we, all we had was that documentation and looking back on it, we made

quite a few mistakes like this.

We had the, let's say we had the documentation on the file server.

So if the file server was lost.

How did you get your documentation?

And it was, again, something that the helpdesk guy pointed out to us.

How did you get your documentation?

That's fine, actually.

How would we.

Sometimes it's an outside perspective or

someone's Hey, how are you actually going to get this stuff done?

Something I think it's really important to know is at

the time I was a senior IT person.

There's a colleague of mine who was senior and we had a network guy.

All of us, were reasonably senior.

This guy was a junior.

He'd been working in finance for three or four years beforehand.

And then he'd just moved into IT.

And he had such a fresh perspective on everything that it really opened our eyes.

And that was the day I learned that it doesn't matter if you got 50

years IT experience or five minutes.

There's always something you can learn from someone.

And sometimes the most valuable thing you can learn is from someone

who is very new to the team, fresh eyes, fresh perspective.

It's invaluable.

100% agree.

There, there is a perspective that you can only

gain by being completely ignorant.

He could have been not junior to IT in this case.

He was, but even if he's a senior IT person, but he's joining your organization

for the first time, another way, you look at this person when they ask for things

of like, when they ask stupid questions, so how often do we, test our backups here?

And you're like, we don't do that.

with my current place, any new person we get into our IT team, we

literally do that sort of thing with them.

Now where we say, have a look through the tickets.

You've got any questions.

Ask, have a look through the Wiki again.

You've got any questions ask because.

There's so many things in there.

There's like the whole corporate culture and there's corporate acronyms.

And if they don't know what they are, we've just found a problem

because if there's one acronym we have this, I, my brain's gone.

Sorry.

there's one acronym that we have, that's very similar to an IT acronym.

I can't remember what it is off the top of my head.

Yeah.

But when you look at it, you think the, IT term because you're an IT person,

but it actually means the corporate.

so there's that kind of thing.

it's always important to spell out these acronyms at the start of any

documentation so that everyone knows this is what you are referring to.

Especially

it's Prasanna's job on, on the podcast.

If anybody ever brings up, an acronym that, they don't spell out,

Prasanna's, always making them spell it

out.

Yep.

I'm like, what does that really mean?

Please tell me.

And this is the thing.

You can walk into a meeting with all the IT acronyms and every IT

person sitting there will probably think it's something different.

I think DC is a good one because DC's direct current data center.

Things like that.

And this is the sort of thing that we've experienced several times, a few different

companies I've worked for, and it's always valuable to get that new person's insight.

Because they don't know the corporate terminology, they don't

know the corporate acronyms.

So it's worth getting them on board and going through all this stuff because

they've got this fresh insight before they learn that stuff and they can spot these

problems before they become a problem.

I just realized I haven't thrown out our

usual disclaimer, Prasanna and I work for different companies.

I work for Druva and he worked for Zoom.

And this is not a podcast of either company.

And the opinions that you hear are ours.

Please rate this podcast at ratethispodcast.com/restore.

And if you, are like our guest here today, Gary who, just you're an IT person

out there, and you want to talk about your favorite subject to, or if you know

what, maybe if you don't understand why

Come challenge, Mr.

Backup.

Some crazy person would actually like them then, come on

here related topics, cybersecurity, data privacy, a number of related topics.

We'd love to have you on as a guest and, and reach out

to me at wcurtispreston@gmail or at @wcpreston on Twitter.

And we'll get you on here.

So, um, how did it turn.

With your, with your restore.

So eventually we got there, we actually got the

exchange server fully restored with correctly, the documentation.

and I think it took three or four days, something like that.

And the thing is, if you want to do a team building exercise, forget all the

assault courses and other things they have, you do get the team together and

do a restore some of the best seriously.

It's a bit like, the trust exercises where you lean

backwards and catches you it's like that.

I've also never seen so many whiteboards being used to

describe issues and draw diagrams of how things hung together.

And it was actually really good.

And I will admit we ended up putting some projects, not exactly on pause,

but we put them to one side as all of us started getting involved in

this restore, because we realized we actually had a very serious problem.

I'll be honest.

We gave the help desk guy, this junior guy to IT the documentation.

And we did expect him to trip over a few things.

He's a new person, some of the terminology is new, fine, not a problem.

We know we're there to help.

What we didn't expect was us to trip over the same issues.

We honestly thought that, like you were saying earlier, Curtis, that he

was going to ask us some questions.

We could do some updates to the documentation, do it again,

and everything would be fine.

But we didn't expect to get stumped by our own documentation.

And unfortunately we actually did, we're sitting there going

through the documentation going well, hang on a minute.

we know the, the password is in this password safe and that password

should work, but something had changed or I think at one point would

actually, changed the security model.

So it was requiring stronger passwords.

So you couldn't actually use a password that was on the backup.

You had to go and reset an account.

And it was lots of.

It was nothing seriously, wrong with a backup as such.

And there's nothing seriously wrong with the documentation,

but it was lots of little things that just piled up and piled up.

And every time we took a couple of steps forward, we thought, that's it.

We've got this solved, we'll get this restored.

And then we got it all up and running and got the server running and

exchange server service wouldn't start.

couldn't figure out why.

I think that one took us a day to go through and we ended up having

to run some additional commands.

And finally, we got there, we got it all up and running.

And I still remember, I think it was actually like a Friday or something

we're sitting there in the office and went, yeah, that was a really good

question know, can we restore the data?

Thank you for asking it.

we had a bit of a celebration over that one.

I would say that, I like what you were saying

about, it sounded like there was a lot of collaboration.

It sounds like there's a lot of whiteboards going on

and you were learning a lot.

I would argue that the reason that was the case is that you

weren't doing it under duress.

You were doing this as a test.

if your exchange had been down for three or four days, that would have

been a very different experience.

Completely.

It's something that we actually discussed, that Friday afternoon, we've got the

exchange server up and running and the conversation was what happens if

this happens for real, because sure.

We got the backup restored.

We know that the backup is good.

Do you mean I was told to stay and it was good, but the restore process wasn't good.

And I think we focused way too much on the backup itself and

not the restore at that point.

I said we had that conversation and it was a matter of what would happen.

And we knew that we were a small company.

We knew we would have the CEO down in the office, screaming at us.

I need this back.

We can't conduct business and I'll be honest that day.

We got a healthy lot of respect, both for the backups, for documentation

and the accuracy of documentation and for the server itself.

Because we knew that the company at that point, the company relied on email so

much that if that server did disappear, and we took that long to get back up and

running the loss, the financial loss to the company and the reputational loss

to the company would have been huge.

And that also actually helped form some push forward for additional resilience

in , like, the servers and moving more towards things like virtual machines,

so that we had the ability to clone and do other bits and pieces, because

we could use that as an experience.

It's look, this is how long potentially worst case scenario it will take.

It shouldn't because we're learning and we need to do this a lot more often.

We need to allocate time to do this.

And the beauty was again, being such a small company.

We actually had the ear of a couple of directors, so you could

put this case forward and they were really receptive to it.

I want to tack on something you said there.

the fact that you and I have been in that timeframe.

young kids today, they don't understand what it was like back then, when you had

no resiliency, you had no redundancy.

You had nothing.

So we had a server that a server had a disk drive.

We didn't have mirroring.

We didn't have

RAID.

Although we had very, we had rightful life.

We got, we were really market.

we didn't, when I was back in the day, we

literally were installing data directly on individual disk drives.

I think we might've had redundant power supplies on the servers that

we were using, and that was it.

And so the loss of any one of those components could take the server.

Right.

And, and now nowadays we move forward to the days of virtualization and

that you can just, if there's a little problem with this server, you just

move your VM over to another server.

In fact, you can V motion at and storage V motion, and you can

move it while it's running, which continues to boggle my brain.

likewise.

And also the devices that would, that so

many of us have grown used to.

I, at home I pretty much live a solid state life.

My TiVo has a solid state hard drive.

and so those are so much more reliable than the moving part

drives that you and I grew up on.

and I think as a result, they don't have

the respect that you need to do to test backups the way you should.

I don't know.

Just a quick editor's note.

In the next section, Gary is going to mention something called iLO and iDRAC.

And he, we forgot to have him define it.

So I'm doing that now.

They are systems from Dell and HP, the integrated Dell remote access

controller and HP integrated lights out.

They're both systems that help increase the uptime of the server by notifying

you of potential failures or issues.

Back to your podcast.

One of the things that we still do today, and this is

probably me being paranoid coming from that environment, we didn't

get alerts on a service if a disk failed, because it didn't really know.

The ILOs and iDRACs were way too expensive for us to have at that point.

So daily server room checks go around.

Is there any flashing lights that shouldn't be flashing?

And we still do that in our data centers today.

And we still do that with some of our machines.

We've actually got this philosophy in place now where if a machine is up for

more than 30 days, it needs to be rebooted because we don't know if it's reboot safe.

So we're starting to put uptime alarms in.

Certainly on Windows.

Linux is a bit different, but with Windows, when it hits a 30-day point.

If we get an uptime alarm, it means that there's possibly

a patching issue with that.

We should get an alarm from the patching system as well.

So we go off and we check.

but the other thing we do something similar with Linux as well.

We're trying to get all the Linux servers rebooted because generally

with those, we can patch them hot, but we still want to get them rebooted.

Are they reboot safe?

Because if we do lose power or machine crashes, it's great having all that stuff

there, but if it doesn't reboot, we've got a problem and we may have a backup if

that backup is inherited that corruption or that problem we're in a bad place.

So we do try to make sure that we've got, these servers rebooted on a fairly regular

Prasanna Malaiyandi:

actually very interesting.

Prasanna Malaiyandi:

I never thought about that About the fact that you need to reboot the systems

Prasanna Malaiyandi:

and just make sure is a hardware and dos and everything else could be.

Absolutely.

The other thing that we've done is we've actually turned up time

on his head now in the old days.

Now these uptime figures of two years, three years, we'll put on the

internet and it's look at my up time.

Now it's the other way around.

It's like, yeah, there's an uptime of 45 days.

Oh, look at my uptime.

That's bad.

We need to get this rebooted.

And check it is reboot safe.

Trying to find reboot windows sometimes is a bit difficult, even with all the

resilience . Just take systems down.

but we do have some sort of bargaining going on with various teams where

we do try and reboot the systems at least once a month, just to make

sure that they are reboot safe.

So help me understand that phrase.

W what do you mean when you say reboot safe?

So reboot safe is simply that if it's potentially a change can be

made to a machine, that means a machine when it reboots is going to crash, or

there's going to be a problem where it can't complete the boot corrupted

boot loader or something like that.

We've seen issues in the past where.

Microsoft update has corrupted the bootloader.

So when you go to reboot, it doesn't restart properly.

So we've actually got the term reboot safe, which just means I know

if I have to reboot that server, I don't have to worry about it.

It will come up.

You're printing system will start all the services that need to start will start,

because we've had issues in the past where certain key services don't start.

So we get a ticket.

Can you please reboot this machine?

Sure.

Reboot it, you walk off.

You think it's done, but the services don't start.

Now, the alerting will alert on that.

But in the meantime, you potentially still down for a bit longer than you need to be.

So we do these tests where we just want something sure all the services

that need to start actually start.

And it comes up completely clean and working exactly how it should.

you're giving me, Yeah.

And by the way, I agree with you with this idea of, the occasional

reboots and I agree that it's, that it's a practice that has gone by

the wayside by a lot of people.

And I remember, I can remember the first time I left my,

this is before I got the Mr.

Backup.

moniker and I got a different moniker and I'll explain it in a minute.

I was at a large oil and gas company and no one had administered the data

center, like a real sysadmin in years.

And so I was going in there and I was doing crazy things like

installing the latest patch set.

And this was, these were a Solaris systems and, it required a reboot

in order to, to install the patches.

And what was happening was I was like, 0 for 10, in terms

of I would install a patch.

I would reboot the server and it wouldn't come back.

And so I picked up the nickname crash, because that's what I was

just, I was literally, it's like the cure is worse than the disease.

So it's we need to do this, but I was doing, I was proactively

doing damage to the environment.

By doing the things I was doing, what I did get really good at though is restoring

their environment because it kept,

so what it turned out, the things that were really.

I don't know uh in trouble were the disks themselves, because we actually

powered down the servers for some of them.

And that's when things really went awry because the disk

drives had never been turned off.

And then, yeah.

And then, they wouldn't come back on.

So I had to get all new disk drives and then, and then do the restore, but yeah.

Yeah.

Yeah, but even with the virtual machines, we still like to

reboot them and to make sure all the services that should come up do come up.

We've even in some cases taken that paranoia to the next level where we'll

do a reboot test before we install a patch or before we do something,

just to make sure that it's not, that patch that has caused a problem.

Now, we generally don't do that for the Microsoft patches, but we do

that for certain application patches.

And it's almost a sanity check.

Because that way, if there is a problem, we know that it is that patch

that has caused a problem and not something lurking from beforehand.

Going back to the article you wrote, Gary, one of the

things I liked in it was you talked about this spreadsheet, if you will, that

track sort of assets that were backed up and you had a methodology that you

called out in the article in terms of how long you would wait before something

had to be tested, Or how the longest something could go without being tested.

And there were certain things that were critical in your environment that sort

of had to be done more periodically.

Yeah.

So what we did is.

we had a spreadsheet, the list of all the backups anyway, and one of the

things we tried to do was make sure that there was no clashing backups.

So the exchange server would get backed up at say, 10:00 PM.

The file server get backed up at 11:00 PM.

That kind of thing, because otherwise we found there was a

lot of issues on the network and latency and all this kind of thing.

So we wanted to stagger the backups as much as possible.

But what we did was we actually added a column to that spreadsheet that said.

Restore last tested, documentation last updated, that kind of thing.

So that we new when the backups were tested and we knew when that

documentation was last updated.

And what we do is we actually have, there was a formula in

it that would color the cells.

And if it was all green, everything's fine.

We've done a recent test.

I think recent was like six months, 12 months, something like that.

and if anything was over outside of that window, it would go red.

So I think the exchange was every six months, the active

directory was once a year.

The file server was I think we would restore a folder or a file

every month, something like that.

and we did this quite a lot and we actually slowed down some

of the tapes going off site for things like the file server.

So we could do a backup a couple of days later, you do a restore test,

update the date in the documentation.

We know that's good.

Send the tape off-site and that's actually funny enough.

That was a financial reason as well because of the cost

of sending the tapes offsite.

but yeah, we started to do that and we started to get quite good

at being able to do these restores.

We were even able to get some additional hardware and we even

starting to do some tests where we're restoring to virtual machines.

Because doing that process.

We found we could get them up and running a lot quicker.

We had a bit more room to breathe and we could have a

much better virtual environment.

And then we've got into some other really clever stuff where we had a physical

domain controller and a virtual domain controller, and we tested fail-over

and all this, we got really advanced

So you're saying that, the green column was

actually the color of that column was automatically determined by By

the age of the last test.

That's pretty cool

Conditional formatting, Curtis in Excel.

that's it.

you're probably better at Excel that I am,

but, Gary, this has been great.

I, I love this story.

I love that it, like the other story we had, where, I don't know if you

listen to the podcast at all, Gary, but we had an episode where someone, they

tested their backups by essentially deleting their entire data center.

Paul van Dyke episode 135.

Wow.

I haven't heard that one.

I have heard some of the others and I have to say I'm a fan.

And it was that one that would just, it

hurt to, to listen to his story.

And it was, he agrees that it was a really dumb idea.

It did eventually work out, but it it took him awhile.

I can imagine.

I I just remember the pain of the exchange server and whilst I've not had

a repeat of that pain since, because.

The software is better these days, the restores are a lot quicker and you do

have a lot more options to play with.

we still have that pain from time to time when trying to do certain restores

and testing the environment out.

So I am still not that brave to do something like that, but, yeah,

I think we're getting there and.

Not brave be the word I would use, but.

Now we have talks about bringing in things like the chaos

monkey and taking down things, but yeah, that's a test for another day.

Yeah.

thanks Prasanna for your usual great questions

Always and nice chatting with you, Gary.

That was fun.

Thank you.

and, thanks to the listeners again.

this is you're why we're here.

You're why we sit here and talk to us.

Curtis.

And we'll talk to each other anyway.

It doesn't matter.

Yeah.

Yeah, exactly.

We'll probably be talking about table saws or video editing tools,

but, anyway, remember to subscribe so that you can restore it all.