Gary Williams: The thing is, if you want to do a team building exercise,
forget all the assault courses and other things they have, you do get
the team together and do a restore.
Some of the best...
seriously.
W. Curtis Preston: It's a bit like, the trust exercises where you lean
backwards and catches you .It's like that.
Hi, and welcome to backup.
Central's Restore it All podcast.
I'm your host.
W.
Curtis Preston, AKA Mr.
Backup and I have with me, my table saw safety, enthusiast, Prasanna Malaiyandi.
How's it going Prasanna?
Prasanna Malaiyandi: I'm good, Curtis.
I don't know if I'd call myself a safety enthusiast, but
W. Curtis Preston: You don't believe in safety.
Prasanna Malaiyandi: no, not at all.
Plus I think you could say I'm a bad influence on you seeing, how much
equipment you've now started to accrue.
W. Curtis Preston: Yeah, last night I watched, I don't know.
I'm going to say two solid hours of just table saw safety videos.
Prasanna Malaiyandi: Yeah, but it is good for you to refresh your
mind on what table saw safety means.
W. Curtis Preston: Yeah.
you do recall that table saw is the reason that this finger is missing
the, or this hand is missing end.
I'm missing the end of the middle finger on my left hand
for those of you listening.
so it's actually really hard for me to watch some of those videos.
Prasanna Malaiyandi: Is it like, when you're doing like driver's
education learning to drive, they show what is that red asphalt.
Was that the name of the movie where it's like accidents happen and.
W. Curtis Preston: Blood on the asphalt, I think is what that one's called.
I do remember that one, but this one, there's one where a guy actually
shows in the video,, he doesn't have the board completely clear the blade
when he takes his hand off of it.
And it, the blade grabs the board and tosses it essentially at his groin, area.
And the thing is when you watch it, he looks at it one frame at a time.
And the board goes from being on the other side of the blade to his groin
in less than a frame of the video.
And, so that's, one 30th of a second, probably.
yeah.
And he's like, don't do that.
but yeah, it's been interesting, but the thing that's got me super
excited right now, has been this new video editing or just editing tool.
It's both video and audio and, and it's this thing called,
Prasanna Malaiyandi: Descript,
W. Curtis Preston: Descript.
Yeah.
And it's just.
Prasanna Malaiyandi: you sounded so excited when you texted me.
W. Curtis Preston: Oh, my God.
it's hard to describe how amazing this tool is, where you input the, in my
case, I'm in, I'm actually, because we're using video clips of these episodes.
I'm inputting the video and I edit the video and then I excerpt
the audio for the audio excerpts.
, It's made mainly for talking head videos like these, right?
Or audio and you input the audio or video, it does, automated transcription,
which gets about 95% accurate.
And then you go through and you obviously, you can correct the things that it
got wrong, but the really amazing part is if you start a sentence and you
change your mind, or you have the, a lot of words going up to that sentence.
All you have to do is highlight those words in the document and
it cuts them out of the video.
Prasanna Malaiyandi: It's like magic
W. Curtis Preston: It's like magic.
And then if that's not enough magic, the part that I'm super excited
about trying is sometimes you say one word when you meant to say another.
Prasanna Malaiyandi: that never happens to you, Curtis.
W. Curtis Preston: Like the podcast I was editing yesterday...
. It was you and I talking about 365 and you don't want this
to happen on your worst day.
That's what I meant to say.
But for some reason I said last day, so with this tool, first
off, I train it with my voice.
I literally speak into the microphone, a bunch of stuff.
It can then synthesize my voice.
And I can select that word and change the word last to worst,
and it will put my voice there, a synthesized version of my voice.
Prasanna Malaiyandi: So here's a question, Curtis, do we actually
need to have this podcast anymore?
Or can we just have not even just type it out.
Can we just have something auto-generate based on all of our past podcasts and
just have it start creating new podcasts.
W. Curtis Preston: It'll just be a recording that says
3, 2, 1 rule over and over.
Prasanna Malaiyandi: No, but it's you know how they have, they've trained AI
to now do paintings and things like that.
I wonder if we could basically have,
W. Curtis Preston: AI based.
yeah.
first I get it.
I have to get all the audio and then feed that into a thing.
Yeah.
We don't need you and me anymore.
Prasanna Malaiyandi: Exactly.
W. Curtis Preston: How hard is it to just say backup your stuff, backup all the
stuff and make sure you test your backups?
Prasanna Malaiyandi: And then you just do it based off of whatever's
trending on Twitter and the data protection, security space.
And it comes up with a new podcast episode for us.
W. Curtis Preston: That may have already happened.
Who knows?
You don't know this is an auto-generated video and auto-generated audio, who
knows, but speaking of testing backups, I was thinking about this concept, as
long as you don't test your backups, your backup is both a complete success
and a complete failure, which reminds me of, the concept of Schrodinger's cat.
Prasanna Malaiyandi: I like the former, rather than thinking
about the latter, but that's
W. Curtis Preston: Yeah, but,
Prasanna Malaiyandi: rather than the realist.
W. Curtis Preston: So you're familiar with the concept of Schrodinger's cat, right?
Prasanna Malaiyandi: Based on TV shows, movies,
W. Curtis Preston: Okay.
Yeah.
So it's just a concept, as I understand the concept that you have this cat in
a box, and as long as you don't look in the box, the cat is both alive and dead.
But once you look in the box, you will know that the cat is alive or dead.
That's the concept of Schrodinger's cat.
And the reason why this is relevant today is that we have the author of
a blog called Schrodinger's Backup: when good documentation goes bad.
He's been in the IT industry almost as long as I have.
He comes to us from the UK.
Welcome to the podcast, Gary Williams.
Gary Williams: Thank you and thank you for the invite.
W. Curtis Preston: I saw that title.
And I was like, I gotta get this guy on the podcast.
Prasanna Malaiyandi: Curtis was so excited.
Gary, you have no idea.
This is like one of his favorite topics.
Gary Williams: Thank you.
I don't know if I coined the term.
I have seen it used since I'd like to think I coined the term,
but I don't know for certain,
W. Curtis Preston: why not?
Gary Williams: it might be something that I heard and I just copied because
it's just sounds really cool when it's perfectly accurate, I think.
It was all three or four companies ago.
The lessons we learned still definitely apply today, but this
happened about three companies back.
So about 10 years ago.
W. Curtis Preston: So what was your role at the time?
Gary Williams: So my role at the time was a senior network engineer or senior
support engineer, something like that.
W. Curtis Preston: OK, And you had the, the gall to, to ask about backups.
Gary Williams: No, I didn't.
I was overconfident with our backups, let's say so we had the backup
software, I think it was backup exec.
And, we had all the servers being backed up.
We had everything going to dual tapes.
The tapes were going off site.
Everything was working.
W. Curtis Preston: Jewel, jewel tapes?
Gary Williams: Dual tapes.
We actually had the backups, the software was writing
effectively RAID-1 one backups.
So it was writing to two tapes.
W. Curtis Preston: Oh, duel, capes.
Okay.
I heard, for some reason I heard Jewel.
I don't know why.
Gary Williams: It's the English accent.
And yeah.
So it's going to two tapes simultaneously.
So the idea was that even if a tape broke, or if something happens to the
backup and we weren't entirely sure of, or you couldn't restore from one of the
tapes, you could then get the other tape and use that tape to do the restore.
So we had all that stuff going on.
And we got all the emails and of course we're getting the emails
saying all the backups are good, everything must be absolutely fine.
Why would we test them?
Why we're busy enough with other tickets and other stuff going on and projects.
We haven't got time to test them.
What's the point?
We know they work.
Prasanna Malaiyandi: And so it looks like you were doing all the
right things in terms of setting up backups, Following the 3, 2, 1 rule.
Right?
Making sure your copies were offsite and.
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: I think that's probably better than maybe
like 70% of the people out there.
Who try to do backups.
You're like doing the right things.
You're like, oh, I'm good to go.
Gary Williams: Yeah, absolutely.
As I say, we had the emails, we even checked the emails.
I think we even had a shared folder or something like that, where all
the backups emails went, and if one of us saw that the folder had
an unread one going, we check it.
If there was an error, someone would get a ticket, it would get sorted out.
If the error went on for several days, there would be a conversation.
We will get these things fixed.
where's the problem.
We know our backups are good.
W. Curtis Preston: So you, you were a.
You were, I don't know.
I don't know what to call it, but so instead of being a proponent
of testing the backups, you were a proponent of oh, everything's fine.
Gary Williams: Unfortunately at that time.
Yes, I was, sitting there and quite fat, dumb and happy going.
We've got the emails, the backups work.
We know they work.
Where's the problem.
I didn't see any issue here at all.
W. Curtis Preston: For what it's worth.
I had a similar point in my career and there was a time.
I remember when I was at a company, I won't give the actual name of the
company, but I will just say it's a very, well-known electronics manufacturer.
and I had helped him set up their backup system and I wasn't
there just to do the backups.
I was there to do sysadmin stuff.
And they were a mess.
th this was a, it was a small department in this bigger, electronics company.
It was an interesting department.
They called it.
Simulation modeling and research.
So it was a revolutionary idea at the time of the idea of modeling,
like in a computer, what would happen if you drop this device?
And so they were doing this in a computer.
It was a fascinating new at the time, new field of science.
So I was there to fix a whole bunch of problems.
One of which, for example, was that every workstation, it was all
Unix workstations, and every person had root on their workstation.
And that was the first thing I was going to fix.
But I also set up their backup system and, the backups worked.
So I assumed the restores would work and it was some time.
I was there long enough that I went, I actually, at some
point needed to do a restore.
And I found out that those tape drives were really good at writing data.
And they were completely incapable of reading data.
Again, I don't want.
I'm sure there was something wrong with these drives, but
they were IBM 3590 drives.
Normally IBM drives are top of the line or whatever, but there was something wrong
with these drives that I was completely.
So I guess what I'm saying is you're not alone.
even me who, I've spent my career in this, although honestly, that's
that event is on the list of things that I think back to when.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: when I try to get other people to do it.
Gary Williams: Absolutely same with me.
the backups that we were taking, as I say, we were only a small
team and we had all the emails.
We had everything in place.
We had the two tape libraries doing the backups.
So we thought we were in a really good position because we had
everything working the way it should.
We even had documentation for how all this stuff was put together.
we actually had to consultancy come in and help us put all this stuff together.
Because at the time I was working for a financial institution, we
had to have certain boxes ticked, and we had those boxes ticked
because we have the documentation.
We had the backups, they were going off site.
They were going off site.
They were being looked after for us.
We even recalled tapes to make sure we could do the process
and no tapes were getting lost.
So we did that level of testing, but what we never actually tested was
actually restoring the data itself.
And it was a bit of an epiphany when we actually had someone come
into the team who a brand new to IT.
Had never worked in IT before.
Always wanted to work in IT.
Was actually employed in the business in a completely different role.
And then he actually said to me, one day, I'd like to move into IT.
I thought he was joking.
It turns out no, he was actually serious.
He was an ex-finance person wanting to move into IT.
So he applied internally, he got the job and he started with us and he started
looking through some old tickets and he was saying things like, why did you
do such and such a change this way?
So there's a whole education thing going on there.
And that's when he asked the question.
When did you test the backups?
What do you mean test them.
We've got the emails.
Look, here, you can see the service.
Here's the tape drives.
Here's the tapes.
We record the tape.
Yeah, sure.
But when did you restore something?
And I, something I won't actually forget because there was this look,
there's only four of us in the IT team.
We were a really small team for a company of about 300 and there's
this look going around the whole office and everyone's going well, we
haven't actually tested them have we?
Prasanna Malaiyandi: It's like a light bulb goes off and it's yeah.
It's Ooh.
Gary Williams: We like, hang on.
yeah, we should probably test one of they shouldn't we.
Okay.
what should we test and looking back on it, it was a really insane moment
just to think that we've had easy.
I think what actually had the emails was coming in for over a year.
And yes, we'd had the odd backup failure where something a time there, or there
was a fault with one of the tape drives.
These tape drives were quite old.
So they actually had physical SCSI cables that would sometimes play up.
So you had to make sure the SCSI cables were all firmly
attached, the terminator was in.
The good old days.
And.
Prasanna Malaiyandi: never had to deal with restores?
W. Curtis Preston: Yeah.
And course you had both active and passive, terminators as well.
Gary Williams: Yeah, exactly.
we did actually have to do some restores, but we had, a storage array and the
storage provider let us do snapshots.
So 99% of the restores that we needed.
Just copy and paste from the snapshot.
Not a problem.
You deleted that file not a problem.
There it is.
If something was deleted from a desktop, the common response was, we
don't back things up on your desktop.
Sorry.
That's tough.
If you want it backed up, put it onto the server, put it into your
home drive or something like that.
It will get backed up.
So that was the general understood consensus because it was a small company.
Most of the time, this wasn't an issue, and as I say, people deleted a file.
I remember one time we had an Excel file.
That was a real pain because of all these financial macros.
And we restored that from a snapshot.
And it was still corrupt and we had to go back a week or so, we
managed to get the file back and it was working and we actually said
it and I remember it quite well.
We said it within the team.
that was lucky.
We might have actually asked to get the tapes on site and do a restore from
the tapes, but the snapshot worked.
Everything's fine, you know yeah.
Prasanna Malaiyandi: Now you've decided, okay, we haven't tested.
Maybe we should actually try doing the test.
How did you decide what to test?
Gary Williams: funny enough, it was a new guy.
the discussion was actually, okay, you're the person sitting
there looking through the tickets.
You're looking through the documentation.
You're new to it all.
You want us to prove to you that the restore process works.
We know it does.
Pick something.
And then he sat there and he went, How about the exchange server?
Prasanna Malaiyandi: Swinging for the fences!
Gary Williams: Fine.
So we thought, okay, fine.
we'll get the tapes back on site.
We'll do the restore.
We'll prove that the backups work and we can go back to what we're normally doing.
all the project work, that kind of thing.
We could spend a day on this.
It will be good for us.
Not a problem.
We even went to the documentation and got the documentation out and said,
look, we've got the documentation.
The tapes are coming in.
This is going to be easy.
And it wasn't.
Prasanna Malaiyandi: Of course not.
So when you decided to do the restore.
Did you bring down your production or were you like, I'm going to
restore this into a safe spot and
Gary Williams: Yeah, we couldn't bring down production because the nature
of the business was that we needed to keep the server up and running.
We actually had a spare server and I think we're maybe had two spare servers.
VMs were just starting to come on the scene and we actually
had a spare server racked.
And the idea was that if we had a server failure, we could take the
physical discs out of one server.
Put it into another server power it on, be back running.
this is also before the days of re replicas.
They were, again, just coming out on a lot of software was super expensive and
W. Curtis Preston: You're giving me flashbacks, Gary.
Gary Williams: the good old days.
W. Curtis Preston: Yeah.
Gary Williams: We had this physical server and it had plenty
of disc space to handle this.
So we said, okay, Let's we've not actually even powered this server on.
I don't even think, I think maybe it was powered on when
we bought it and that was it.
So we said we should test that server out anyway.
Yeah.
Let's power it on.
Let's get the data restored to that server and bring exchange up.
We can bring it up in an isolated network.
Do some very basic tests on it, because it was a small team.
We had access to the networking guys.
I'll say networking guys.
We did a little bit of networking age and there was one guy who did a lot
of the really key networking, tasks.
So none of that was a problem.
We didn't have to wait months for tickets or to get done or anything like that.
So we set up this isolated network, we got the tapes on site and we
started doing the restore and that's when it all went horribly wrong.
Prasanna Malaiyandi: So who was doing the restore?
Gary Williams: I, if I recall, it was actually our help desk guy.
We S we said to him, look, you came up with this.
W. Curtis Preston: You put a lot on this guy.
It was his idea.
And you're like, what, if you think testing backups is so
important, why don't you do it?
Gary Williams: Pretty much .We did put it on him.
cause it was his idea.
And we said, look, this is a really good exercise for you to do again.
Unfortunately, I'm going to put my hands up to this.
It's a bad thing to have done, but we said, I'm a senior IT person.
I know the backups are good.
here you go.
Here's the tapes.
Here's the documentation.
See you later and off he goes and he comes back.
I think it was about two, three hours later, something like that.
And he went, I can't get this working.
W. Curtis Preston: Yeah.
Gary Williams: What do you mean you can't get it working.
What's the problem.
And I don't actually recall what the problems, all the problems were, but
I know that the server itself didn't have enough disc space, even though
it was supposed to have the disc space, because the documentation said,
you need partition sizes like this.
And it actually changed since then.
And we didn't realize, and that was really the start of a lot of problems.
W. Curtis Preston: Yeah.
first off I will say that even though.
the way you got there.
I like the way you did it.
what to say, even though the way you got there was wrong, the fact that
you, the fact that you had this person.
do it, who wasn't the person, that made the documentation.
That's actually something I push pretty heavily.
And it's an idea that came from back in my days when I was at a bank
and we very much did test restores.
first off we didn't have snapshots.
We didn't have any of that stuff.
And we had 10,000 employees and any one of them was allowed to
call into the help desk and ask for a restore on any given day.
And, so we would get 10 to 15 restores a day.
So we tested pretty regular, but the thing that we buy in that degree, but
the thing that we had to test in the way that you did were these large
server restores, we did a DR test and it was an absolute imperative
from the powers that be was that.
Curtis wrote the documentation.
Curtis cannot be the person actually doing the test.
Curtis needs to be standing back there, listening closely to the problems that
are happening, but, w which, which was actually kind of nice, although
it's nerve wracking to be the person who wrote the documentation and then
sitting there watching someone, you think you've answered all the questions,
but it's not like in this case, you.
you had the classic example of the documentation might've been
correct, but it was out of date.
Gary Williams: It was correct at the time, the irony is very similar with you.
I didn't actually write the documentation.
It was written by the contractors and consultants that came on.
Actually signed off on the documentation saying, yes, all
the version numbers are correct.
And I think I'd done a couple of updates.
And then we'd had other changes and the other people
had forgoten or I'd forgotten.
Probably I'd forgotten to update the documentation because we
were busy only a small team.
And so things very slowly on, not just that document, but on every other
document that we had about the environment become out of date and it was this
snowball of errors that had crept in.
And the thing that we realized is actually having no documentation
would have been better because the documentation was lying to us.
this poor guy is sitting there going, I followed steps three, four,
and five, but I can't do step six because step five doesn't work.
What do you mean?
It doesn't work.
And that's when we found that there was a service pack that
was missing from exchange.
So it couldn't go any further and it just kept on building and building like this.
Prasanna Malaiyandi: That is an interesting problem.
How do you keep your documentation up to date as you're making these
changes and making sure everyone across the environment knows like where the
documentation is and all the rest of that.
Gary Williams: today, we use a Wiki solution for all of our documentation.
The idea behind that of course, is the Wiki is so easy to edit.
But you still don't or sometimes you still don't.
You make a note, I'll do that tomorrow or next week.
So there is still the exact same risk.
And even in my current place, we've seen this with certain, we do testing as well.
We do a lot more testing now than, anywhere I've ever worked before.
And even with a lot of the modern systems with Amazon.
Backups to S3 and all this kind of stuff.
We still test to make sure that everything's correct,
that we know what we're doing.
That those Wiki pages are fully up to date.
we did some AD restore testing not so long ago and we found, not major errors,
but there was a couple of little issues there with the restore process, which
just needed a few corrections in the documentation, just, as like a permissions
era type of thing where we couldn't actually get access to the bucket.
So we had to make some changes there.
So even with all the modern backup software.
It's still so important.
W. Curtis Preston: I talked about those DR tests that we did back in the day and.
The, and the fact that we always had someone who wasn't me doing the
tests, and frequent listeners to the podcast will have heard this before.
But if we define a successful restore, as we got from A to Z without having to ask
Curtis, what does this line mean, not a single one of the restores was successful.
so if Curtis ever got, blown up and, whatever, the chances of a restore
going completely without a hitch was, zero, which is why you talked
about updating, there's always little things that you have to update.
I would suggest that original documentation.
and again, take this for what it's worth to anybody who's listening.
the first mistake was writing the documentation in a way that
it can easily get outdated.
our exchange server is 75.
Terra...
r ight.
that's a problem.
So if you're going to hand that to a restore documentation, what it should say
is before beginning the restore, go look at the size of the backups, And figure
out how big the current exchange server is, and then size the volume accordingly.
yeah, that, that line wouldn't have gone out of date as quickly.
it is a real challenge by the way.
this idea of what it's like to update documentation, by the way, back in
the day we were using Wordperfect.
Yeah.
And I remember the official company standard was WordPerfect,
because we could use it on, we had Unix versions of WordPerfect.
By the way, curses spaced WordPerfect.
Not this fancy Windows.
what you'd see is what you get editing stuff.
This was text on a screen.
and I remember getting in a fight over.
There was this one guy that was new and he wanted to use Word
because nobody used WordPerfect.
And we were like, we don't care.
We use WordPerfect here for our documentation.
And if you want your documentation to fit into our documentation,
you will use Wordperfect.
And you will like it.
Gary Williams: I remember our first days of moving across
the world where you had the.
Word had the ability to mimic WordPerfect key presses.
So you could transition easily.
Good old days.
W. Curtis Preston: Good old days, but I think what you're doing now with the Wiki,
I think that's a much better approach.
Gary Williams: It is.
There's permissions list behind it, obviously, so that not everyone
can get access to it, but it's the right people can get access.
but what it means is everyone in the team can get access.
They can all update.
It.
There's a history as well.
So the other thing that we didn't have is the backup of the documentation
was on the server we were backing up.
Prasanna Malaiyandi: Oh,
Gary Williams: Exactly.
So we, all we had was that documentation and looking back on it, we made
quite a few mistakes like this.
We had the, let's say we had the documentation on the file server.
So if the file server was lost.
How did you get your documentation?
And it was, again, something that the helpdesk guy pointed out to us.
How did you get your documentation?
That's fine, actually.
How would we.
Prasanna Malaiyandi: Sometimes it's an outside perspective or
someone's Hey, how are you actually going to get this stuff done?
Gary Williams: Something I think it's really important to know is at
the time I was a senior IT person.
There's a colleague of mine who was senior and we had a network guy.
All of us, were reasonably senior.
This guy was a junior.
He'd been working in finance for three or four years beforehand.
And then he'd just moved into IT.
And he had such a fresh perspective on everything that it really opened our eyes.
And that was the day I learned that it doesn't matter if you got 50
years IT experience or five minutes.
There's always something you can learn from someone.
And sometimes the most valuable thing you can learn is from someone
who is very new to the team, fresh eyes, fresh perspective.
It's invaluable.
Prasanna Malaiyandi: 100% agree.
W. Curtis Preston: There, there is a perspective that you can only
gain by being completely ignorant.
He could have been not junior to IT in this case.
He was, but even if he's a senior IT person, but he's joining your organization
for the first time, another way, you look at this person when they ask for things
of like, when they ask stupid questions, so how often do we, test our backups here?
And you're like, we don't do that.
Gary Williams: with my current place, any new person we get into our IT team, we
literally do that sort of thing with them.
Now where we say, have a look through the tickets.
You've got any questions.
Ask, have a look through the Wiki again.
You've got any questions ask because.
There's so many things in there.
There's like the whole corporate culture and there's corporate acronyms.
And if they don't know what they are, we've just found a problem
because if there's one acronym we have this, I, my brain's gone.
Sorry.
there's one acronym that we have, that's very similar to an IT acronym.
I can't remember what it is off the top of my head.
Yeah.
But when you look at it, you think the, IT term because you're an IT person,
but it actually means the corporate.
so there's that kind of thing.
it's always important to spell out these acronyms at the start of any
documentation so that everyone knows this is what you are referring to.
Prasanna Malaiyandi: Especially
W. Curtis Preston: it's Prasanna's job on, on the podcast.
If anybody ever brings up, an acronym that, they don't spell out,
Prasanna's, always making them spell it
Prasanna Malaiyandi: out.
Yep.
I'm like, what does that really mean?
Please tell me.
Gary Williams: And this is the thing.
You can walk into a meeting with all the IT acronyms and every IT
person sitting there will probably think it's something different.
I think DC is a good one because DC's direct current data center.
Things like that.
And this is the sort of thing that we've experienced several times, a few different
companies I've worked for, and it's always valuable to get that new person's insight.
Because they don't know the corporate terminology, they don't
know the corporate acronyms.
So it's worth getting them on board and going through all this stuff because
they've got this fresh insight before they learn that stuff and they can spot these
problems before they become a problem.
W. Curtis Preston: I just realized I haven't thrown out our
usual disclaimer, Prasanna and I work for different companies.
I work for Druva and he worked for Zoom.
And this is not a podcast of either company.
And the opinions that you hear are ours.
Please rate this podcast at ratethispodcast.com/restore.
And if you, are like our guest here today, Gary who, just you're an IT person
out there, and you want to talk about your favorite subject to, or if you know
what, maybe if you don't understand why
Prasanna Malaiyandi: Come challenge, Mr.
Backup.
W. Curtis Preston: Some crazy person would actually like them then, come on
here related topics, cybersecurity, data privacy, a number of related topics.
We'd love to have you on as a guest and, and reach out
to me at wcurtispreston@gmail or at @wcpreston on Twitter.
And we'll get you on here.
So, um, how did it turn.
With your, with your restore.
Gary Williams: So eventually we got there, we actually got the
exchange server fully restored with correctly, the documentation.
and I think it took three or four days, something like that.
And the thing is, if you want to do a team building exercise, forget all the
assault courses and other things they have, you do get the team together and
do a restore some of the best seriously.
W. Curtis Preston: It's a bit like, the trust exercises where you lean
backwards and catches you it's like that.
Gary Williams: I've also never seen so many whiteboards being used to
describe issues and draw diagrams of how things hung together.
And it was actually really good.
And I will admit we ended up putting some projects, not exactly on pause,
but we put them to one side as all of us started getting involved in
this restore, because we realized we actually had a very serious problem.
I'll be honest.
We gave the help desk guy, this junior guy to IT the documentation.
And we did expect him to trip over a few things.
He's a new person, some of the terminology is new, fine, not a problem.
We know we're there to help.
What we didn't expect was us to trip over the same issues.
We honestly thought that, like you were saying earlier, Curtis, that he
was going to ask us some questions.
We could do some updates to the documentation, do it again,
and everything would be fine.
But we didn't expect to get stumped by our own documentation.
And unfortunately we actually did, we're sitting there going
through the documentation going well, hang on a minute.
we know the, the password is in this password safe and that password
should work, but something had changed or I think at one point would
actually, changed the security model.
So it was requiring stronger passwords.
So you couldn't actually use a password that was on the backup.
You had to go and reset an account.
And it was lots of.
It was nothing seriously, wrong with a backup as such.
And there's nothing seriously wrong with the documentation,
but it was lots of little things that just piled up and piled up.
And every time we took a couple of steps forward, we thought, that's it.
We've got this solved, we'll get this restored.
And then we got it all up and running and got the server running and
exchange server service wouldn't start.
couldn't figure out why.
I think that one took us a day to go through and we ended up having
to run some additional commands.
And finally, we got there, we got it all up and running.
And I still remember, I think it was actually like a Friday or something
we're sitting there in the office and went, yeah, that was a really good
question know, can we restore the data?
Thank you for asking it.
we had a bit of a celebration over that one.
W. Curtis Preston: I would say that, I like what you were saying
about, it sounded like there was a lot of collaboration.
It sounds like there's a lot of whiteboards going on
and you were learning a lot.
I would argue that the reason that was the case is that you
weren't doing it under duress.
You were doing this as a test.
if your exchange had been down for three or four days, that would have
been a very different experience.
Gary Williams: Completely.
It's something that we actually discussed, that Friday afternoon, we've got the
exchange server up and running and the conversation was what happens if
this happens for real, because sure.
We got the backup restored.
We know that the backup is good.
Do you mean I was told to stay and it was good, but the restore process wasn't good.
And I think we focused way too much on the backup itself and
not the restore at that point.
I said we had that conversation and it was a matter of what would happen.
And we knew that we were a small company.
We knew we would have the CEO down in the office, screaming at us.
I need this back.
We can't conduct business and I'll be honest that day.
We got a healthy lot of respect, both for the backups, for documentation
and the accuracy of documentation and for the server itself.
Because we knew that the company at that point, the company relied on email so
much that if that server did disappear, and we took that long to get back up and
running the loss, the financial loss to the company and the reputational loss
to the company would have been huge.
And that also actually helped form some push forward for additional resilience
in , like, the servers and moving more towards things like virtual machines,
so that we had the ability to clone and do other bits and pieces, because
we could use that as an experience.
It's look, this is how long potentially worst case scenario it will take.
It shouldn't because we're learning and we need to do this a lot more often.
We need to allocate time to do this.
And the beauty was again, being such a small company.
We actually had the ear of a couple of directors, so you could
put this case forward and they were really receptive to it.
W. Curtis Preston: I want to tack on something you said there.
the fact that you and I have been in that timeframe.
young kids today, they don't understand what it was like back then, when you had
no resiliency, you had no redundancy.
You had nothing.
So we had a server that a server had a disk drive.
We didn't have mirroring.
We didn't have
Gary Williams: RAID.
Although we had very, we had rightful life.
We got, we were really market.
W. Curtis Preston: we didn't, when I was back in the day, we
literally were installing data directly on individual disk drives.
I think we might've had redundant power supplies on the servers that
we were using, and that was it.
And so the loss of any one of those components could take the server.
Right.
And, and now nowadays we move forward to the days of virtualization and
that you can just, if there's a little problem with this server, you just
move your VM over to another server.
In fact, you can V motion at and storage V motion, and you can
move it while it's running, which continues to boggle my brain.
Gary Williams: likewise.
W. Curtis Preston: And also the devices that would, that so
many of us have grown used to.
I, at home I pretty much live a solid state life.
My TiVo has a solid state hard drive.
and so those are so much more reliable than the moving part
drives that you and I grew up on.
and I think as a result, they don't have
the respect that you need to do to test backups the way you should.
I don't know.
Just a quick editor's note.
In the next section, Gary is going to mention something called iLO and iDRAC.
And he, we forgot to have him define it.
So I'm doing that now.
They are systems from Dell and HP, the integrated Dell remote access
controller and HP integrated lights out.
They're both systems that help increase the uptime of the server by notifying
you of potential failures or issues.
Back to your podcast.
Gary Williams: One of the things that we still do today, and this is
probably me being paranoid coming from that environment, we didn't
get alerts on a service if a disk failed, because it didn't really know.
The ILOs and iDRACs were way too expensive for us to have at that point.
So daily server room checks go around.
Is there any flashing lights that shouldn't be flashing?
And we still do that in our data centers today.
And we still do that with some of our machines.
We've actually got this philosophy in place now where if a machine is up for
more than 30 days, it needs to be rebooted because we don't know if it's reboot safe.
So we're starting to put uptime alarms in.
Certainly on Windows.
Linux is a bit different, but with Windows, when it hits a 30-day point.
If we get an uptime alarm, it means that there's possibly
a patching issue with that.
We should get an alarm from the patching system as well.
So we go off and we check.
but the other thing we do something similar with Linux as well.
We're trying to get all the Linux servers rebooted because generally
with those, we can patch them hot, but we still want to get them rebooted.
Are they reboot safe?
Because if we do lose power or machine crashes, it's great having all that stuff
there, but if it doesn't reboot, we've got a problem and we may have a backup if
that backup is inherited that corruption or that problem we're in a bad place.
So we do try to make sure that we've got, these servers rebooted on a fairly regular
Prasanna Malaiyandi: actually very interesting.
I never thought about that About the fact that you need to reboot the systems
and just make sure is a hardware and dos and everything else could be.
Gary Williams: Absolutely.
The other thing that we've done is we've actually turned up time
on his head now in the old days.
Now these uptime figures of two years, three years, we'll put on the
internet and it's look at my up time.
Now it's the other way around.
It's like, yeah, there's an uptime of 45 days.
Oh, look at my uptime.
That's bad.
We need to get this rebooted.
And check it is reboot safe.
Trying to find reboot windows sometimes is a bit difficult, even with all the
resilience . Just take systems down.
but we do have some sort of bargaining going on with various teams where
we do try and reboot the systems at least once a month, just to make
sure that they are reboot safe.
W. Curtis Preston: So help me understand that phrase.
W what do you mean when you say reboot safe?
Gary Williams: So reboot safe is simply that if it's potentially a change can be
made to a machine, that means a machine when it reboots is going to crash, or
there's going to be a problem where it can't complete the boot corrupted
boot loader or something like that.
We've seen issues in the past where.
Microsoft update has corrupted the bootloader.
So when you go to reboot, it doesn't restart properly.
So we've actually got the term reboot safe, which just means I know
if I have to reboot that server, I don't have to worry about it.
It will come up.
You're printing system will start all the services that need to start will start,
because we've had issues in the past where certain key services don't start.
So we get a ticket.
Can you please reboot this machine?
Sure.
Reboot it, you walk off.
You think it's done, but the services don't start.
Now, the alerting will alert on that.
But in the meantime, you potentially still down for a bit longer than you need to be.
So we do these tests where we just want something sure all the services
that need to start actually start.
And it comes up completely clean and working exactly how it should.
W. Curtis Preston: you're giving me, Yeah.
And by the way, I agree with you with this idea of, the occasional
reboots and I agree that it's, that it's a practice that has gone by
the wayside by a lot of people.
And I remember, I can remember the first time I left my,
this is before I got the Mr.
Backup.
moniker and I got a different moniker and I'll explain it in a minute.
I was at a large oil and gas company and no one had administered the data
center, like a real sysadmin in years.
And so I was going in there and I was doing crazy things like
installing the latest patch set.
And this was, these were a Solaris systems and, it required a reboot
in order to, to install the patches.
And what was happening was I was like, 0 for 10, in terms
of I would install a patch.
I would reboot the server and it wouldn't come back.
And so I picked up the nickname crash, because that's what I was
just, I was literally, it's like the cure is worse than the disease.
So it's we need to do this, but I was doing, I was proactively
doing damage to the environment.
By doing the things I was doing, what I did get really good at though is restoring
their environment because it kept,
so what it turned out, the things that were really.
I don't know uh in trouble were the disks themselves, because we actually
powered down the servers for some of them.
And that's when things really went awry because the disk
drives had never been turned off.
And then, yeah.
And then, they wouldn't come back on.
So I had to get all new disk drives and then, and then do the restore, but yeah.
Prasanna Malaiyandi: Yeah.
Gary Williams: Yeah, but even with the virtual machines, we still like to
reboot them and to make sure all the services that should come up do come up.
We've even in some cases taken that paranoia to the next level where we'll
do a reboot test before we install a patch or before we do something,
just to make sure that it's not, that patch that has caused a problem.
Now, we generally don't do that for the Microsoft patches, but we do
that for certain application patches.
And it's almost a sanity check.
Because that way, if there is a problem, we know that it is that patch
that has caused a problem and not something lurking from beforehand.
Prasanna Malaiyandi: Going back to the article you wrote, Gary, one of the
things I liked in it was you talked about this spreadsheet, if you will, that
track sort of assets that were backed up and you had a methodology that you
called out in the article in terms of how long you would wait before something
had to be tested, Or how the longest something could go without being tested.
And there were certain things that were critical in your environment that sort
of had to be done more periodically.
Gary Williams: Yeah.
So what we did is.
we had a spreadsheet, the list of all the backups anyway, and one of the
things we tried to do was make sure that there was no clashing backups.
So the exchange server would get backed up at say, 10:00 PM.
The file server get backed up at 11:00 PM.
That kind of thing, because otherwise we found there was a
lot of issues on the network and latency and all this kind of thing.
So we wanted to stagger the backups as much as possible.
But what we did was we actually added a column to that spreadsheet that said.
Restore last tested, documentation last updated, that kind of thing.
So that we new when the backups were tested and we knew when that
documentation was last updated.
And what we do is we actually have, there was a formula in
it that would color the cells.
And if it was all green, everything's fine.
We've done a recent test.
I think recent was like six months, 12 months, something like that.
and if anything was over outside of that window, it would go red.
So I think the exchange was every six months, the active
directory was once a year.
The file server was I think we would restore a folder or a file
every month, something like that.
and we did this quite a lot and we actually slowed down some
of the tapes going off site for things like the file server.
So we could do a backup a couple of days later, you do a restore test,
update the date in the documentation.
We know that's good.
Send the tape off-site and that's actually funny enough.
That was a financial reason as well because of the cost
of sending the tapes offsite.
but yeah, we started to do that and we started to get quite good
at being able to do these restores.
We were even able to get some additional hardware and we even
starting to do some tests where we're restoring to virtual machines.
Because doing that process.
We found we could get them up and running a lot quicker.
We had a bit more room to breathe and we could have a
much better virtual environment.
And then we've got into some other really clever stuff where we had a physical
domain controller and a virtual domain controller, and we tested fail-over
and all this, we got really advanced
W. Curtis Preston: So you're saying that, the green column was
actually the color of that column was automatically determined by By
Gary Williams: the age of the last test.
W. Curtis Preston: That's pretty cool
Prasanna Malaiyandi: Conditional formatting, Curtis in Excel.
Gary Williams: that's it.
W. Curtis Preston: you're probably better at Excel that I am,
but, Gary, this has been great.
I, I love this story.
I love that it, like the other story we had, where, I don't know if you
listen to the podcast at all, Gary, but we had an episode where someone, they
tested their backups by essentially deleting their entire data center.
Prasanna Malaiyandi: Paul van Dyke episode 135.
Gary Williams: Wow.
I haven't heard that one.
I have heard some of the others and I have to say I'm a fan.
W. Curtis Preston: And it was that one that would just, it
hurt to, to listen to his story.
And it was, he agrees that it was a really dumb idea.
It did eventually work out, but it it took him awhile.
Gary Williams: I can imagine.
I I just remember the pain of the exchange server and whilst I've not had
a repeat of that pain since, because.
The software is better these days, the restores are a lot quicker and you do
have a lot more options to play with.
we still have that pain from time to time when trying to do certain restores
and testing the environment out.
So I am still not that brave to do something like that, but, yeah,
I think we're getting there and.
W. Curtis Preston: Not brave be the word I would use, but.
Gary Williams: Now we have talks about bringing in things like the chaos
monkey and taking down things, but yeah, that's a test for another day.
W. Curtis Preston: Yeah.
thanks Prasanna for your usual great questions
Prasanna Malaiyandi: Always and nice chatting with you, Gary.
That was fun.
Gary Williams: Thank you.
W. Curtis Preston: and, thanks to the listeners again.
this is you're why we're here.
You're why we sit here and talk to us.
Prasanna Malaiyandi: Curtis.
And we'll talk to each other anyway.
It doesn't matter.
W. Curtis Preston: Yeah.
Yeah, exactly.
We'll probably be talking about table saws or video editing tools,
but, anyway, remember to subscribe so that you can restore it all.