1
00:00:00,039 --> 00:00:02,739
Gary Williams: The thing is, if you
want to do a team building exercise,

2
00:00:02,836 --> 00:00:05,836
forget all the assault courses and
other things they have, you do get

3
00:00:05,836 --> 00:00:07,126
the team together and do a restore.

4
00:00:07,636 --> 00:00:08,596
Some of the best...

5
00:00:10,186 --> 00:00:10,546
seriously.

6
00:00:10,580 --> 00:00:13,224
W. Curtis Preston: It's a bit like,
the trust exercises where you lean

7
00:00:13,224 --> 00:00:15,765
backwards and catches you .It's like that.

8
00:00:35,223 --> 00:00:36,363
Hi, and welcome to backup.

9
00:00:36,363 --> 00:00:37,953
Central's Restore it All podcast.

10
00:00:37,953 --> 00:00:38,403
I'm your host.

11
00:00:38,403 --> 00:00:38,613
W.

12
00:00:38,613 --> 00:00:40,323
Curtis Preston, AKA Mr.

13
00:00:40,323 --> 00:00:46,782
Backup and I have with me, my table saw
safety, enthusiast, Prasanna Malaiyandi.

14
00:00:46,802 --> 00:00:48,122
How's it going Prasanna?

15
00:00:48,142 --> 00:00:49,402
Prasanna Malaiyandi: I'm good, Curtis.

16
00:00:49,402 --> 00:00:53,062
I don't know if I'd call
myself a safety enthusiast, but

17
00:00:53,512 --> 00:00:54,652
W. Curtis Preston: You
don't believe in safety.

18
00:00:54,982 --> 00:00:56,062
Prasanna Malaiyandi: no, not at all.

19
00:00:56,242 --> 00:01:00,725
Plus I think you could say I'm a bad
influence on you seeing, how much

20
00:01:00,785 --> 00:01:05,235
equipment you've now started to accrue.

21
00:01:06,110 --> 00:01:09,621
W. Curtis Preston: Yeah, last
night I watched, I don't know.

22
00:01:09,621 --> 00:01:14,948
I'm going to say two solid hours
of just table saw safety videos.

23
00:01:15,083 --> 00:01:17,673
Prasanna Malaiyandi: Yeah, but it
is good for you to refresh your

24
00:01:17,673 --> 00:01:19,743
mind on what table saw safety means.

25
00:01:20,268 --> 00:01:20,508
W. Curtis Preston: Yeah.

26
00:01:20,508 --> 00:01:27,881
you do recall that table saw is the
reason that this finger is missing

27
00:01:27,881 --> 00:01:29,771
the, or this hand is missing end.

28
00:01:30,371 --> 00:01:32,951
I'm missing the end of the
middle finger on my left hand

29
00:01:33,401 --> 00:01:34,421
for those of you listening.

30
00:01:34,435 --> 00:01:37,415
so it's actually really hard for
me to watch some of those videos.

31
00:01:37,465 --> 00:01:39,318
Prasanna Malaiyandi: Is it like,
when you're doing like driver's

32
00:01:39,318 --> 00:01:42,768
education learning to drive, they
show what is that red asphalt.

33
00:01:42,798 --> 00:01:45,588
Was that the name of the movie where
it's like accidents happen and.

34
00:01:46,273 --> 00:01:48,733
W. Curtis Preston: Blood on the asphalt,
I think is what that one's called.

35
00:01:48,733 --> 00:01:53,243
I do remember that one, but this one,
there's one where a guy  actually

36
00:01:53,243 --> 00:01:59,043
shows in the video,, he doesn't have
the board completely clear the blade

37
00:01:59,073 --> 00:02:00,603
when he takes his hand off of it.

38
00:02:01,053 --> 00:02:09,043
And it, the blade grabs the board and
tosses it essentially at his groin, area.

39
00:02:09,390 --> 00:02:12,582
And the thing is when you watch it,
he looks at it one frame at a time.

40
00:02:12,582 --> 00:02:18,552
And the board goes from being on the
other side of the blade to his groin

41
00:02:18,612 --> 00:02:21,102
in less than a frame of  the video.

42
00:02:21,672 --> 00:02:24,909
And, so that's, one 30th
of a second, probably.

43
00:02:25,020 --> 00:02:25,410
yeah.

44
00:02:25,440 --> 00:02:26,733
And he's like, don't do that.

45
00:02:26,736 --> 00:02:30,095
but yeah, it's been interesting,
but the thing that's got me super

46
00:02:30,095 --> 00:02:34,290
excited right now, has been this new
video editing or just editing tool.

47
00:02:34,290 --> 00:02:38,041
It's both video and audio and,
and it's this thing called,

48
00:02:38,144 --> 00:02:38,644
Prasanna Malaiyandi: Descript,

49
00:02:39,359 --> 00:02:40,019
W. Curtis Preston: Descript.

50
00:02:40,019 --> 00:02:40,499
Yeah.

51
00:02:40,709 --> 00:02:41,999
And it's just.

52
00:02:42,724 --> 00:02:45,784
Prasanna Malaiyandi: you sounded
so excited when you texted me.

53
00:02:46,489 --> 00:02:47,599
W. Curtis Preston: Oh, my God.

54
00:02:47,648 --> 00:02:54,741
it's hard to describe how amazing this
tool is, where you input the, in my

55
00:02:54,741 --> 00:02:58,358
case, I'm in, I'm actually, because we're
using video clips of these episodes.

56
00:02:58,358 --> 00:03:01,838
I'm inputting the video and I
edit the video and then I excerpt

57
00:03:01,838 --> 00:03:03,968
the audio for the audio excerpts.

58
00:03:04,458 --> 00:03:07,248
, It's made mainly for talking
head videos like these, right?

59
00:03:07,948 --> 00:03:14,251
Or audio and you input the audio or
video, it does, automated transcription,

60
00:03:14,301 --> 00:03:16,193
which gets about 95% accurate.

61
00:03:16,193 --> 00:03:20,675
And then you go through and you obviously,
you can correct the things that it

62
00:03:20,675 --> 00:03:27,022
got wrong, but the really amazing part
is if  you start a sentence and you

63
00:03:27,052 --> 00:03:32,872
change your mind, or you have the, a
lot of words going up to that sentence.

64
00:03:32,932 --> 00:03:37,912
All you have to do is highlight
those words in the document and

65
00:03:37,912 --> 00:03:39,442
it cuts them out of the video.

66
00:03:40,487 --> 00:03:41,567
Prasanna Malaiyandi: It's like magic

67
00:03:43,202 --> 00:03:44,102
W. Curtis Preston: It's like magic.

68
00:03:44,162 --> 00:03:48,512
And then if that's not enough magic,
the part that I'm super excited

69
00:03:48,512 --> 00:03:54,662
about trying is sometimes you say one
word when you meant to say another.

70
00:03:54,857 --> 00:03:56,207
Prasanna Malaiyandi: that
never happens to you, Curtis.

71
00:03:59,012 --> 00:04:01,838
W. Curtis Preston: Like the
podcast I was editing yesterday...

72
00:04:02,648 --> 00:04:06,188
. It was you and I talking about
365 and you don't want this

73
00:04:06,188 --> 00:04:08,048
to happen on your worst day.

74
00:04:08,048 --> 00:04:09,289
That's what I meant to say.

75
00:04:09,289 --> 00:04:13,579
But for some reason I said last
day, so with this tool, first

76
00:04:13,579 --> 00:04:15,499
off, I train it with my voice.

77
00:04:15,529 --> 00:04:18,899
I literally speak into the
microphone, a bunch of stuff.

78
00:04:20,219 --> 00:04:22,139
It can then synthesize my voice.

79
00:04:22,904 --> 00:04:28,394
And I can select that word and
change the word last to worst,

80
00:04:28,484 --> 00:04:32,564
and it will put my voice there, a
synthesized version of my voice.

81
00:04:32,579 --> 00:04:35,069
Prasanna Malaiyandi: So here's a
question, Curtis, do we actually

82
00:04:35,069 --> 00:04:36,719
need to have this podcast anymore?

83
00:04:36,719 --> 00:04:38,371
Or can we just have not
even just type it out.

84
00:04:38,371 --> 00:04:43,163
Can we just have something auto-generate
based on all of our past podcasts and

85
00:04:43,163 --> 00:04:44,793
just have it start creating new podcasts.

86
00:04:44,793 --> 00:04:46,203
W. Curtis Preston: It'll
just be a recording that says

87
00:04:46,203 --> 00:04:47,613
3, 2, 1 rule over and over.

88
00:04:48,008 --> 00:04:50,235
Prasanna Malaiyandi: No, but it's you
know how they have, they've trained AI

89
00:04:50,235 --> 00:04:52,731
to now do paintings and things like that.

90
00:04:52,731 --> 00:04:55,251
I wonder if we could basically have,

91
00:04:55,406 --> 00:04:56,156
W. Curtis Preston: AI based.

92
00:04:56,186 --> 00:04:56,416
yeah.

93
00:04:56,446 --> 00:04:57,046
first I get it.

94
00:04:57,076 --> 00:04:59,686
I have to get all the audio and
then feed that into a thing.

95
00:05:00,736 --> 00:05:00,916
Yeah.

96
00:05:00,916 --> 00:05:01,956
We don't need you and me anymore.

97
00:05:01,971 --> 00:05:02,661
Prasanna Malaiyandi: Exactly.

98
00:05:02,711 --> 00:05:05,491
W. Curtis Preston: How hard is it to just
say backup your stuff, backup all the

99
00:05:05,491 --> 00:05:07,731
stuff and make sure you test your backups?

100
00:05:07,731 --> 00:05:09,796
Prasanna Malaiyandi: And then you
just do it based off of whatever's

101
00:05:09,796 --> 00:05:12,826
trending on Twitter and the
data protection, security space.

102
00:05:13,276 --> 00:05:16,186
And it comes up with a new
podcast episode for us.

103
00:05:16,366 --> 00:05:17,476
W. Curtis Preston: That
may have already happened.

104
00:05:17,956 --> 00:05:18,586
Who knows?

105
00:05:18,766 --> 00:05:24,466
You don't know this is an auto-generated
video and auto-generated audio, who

106
00:05:24,466 --> 00:05:29,155
knows, but speaking of testing backups,
I was thinking about this concept, as

107
00:05:29,155 --> 00:05:35,575
long as you don't test your backups,
your backup is both a complete success

108
00:05:35,665 --> 00:05:41,679
and a complete failure, which reminds
me of, the concept of Schrodinger's cat.

109
00:05:41,804 --> 00:05:43,934
Prasanna Malaiyandi: I like the
former, rather than thinking

110
00:05:43,934 --> 00:05:45,314
about the latter, but that's

111
00:05:46,584 --> 00:05:47,153
W. Curtis Preston: Yeah, but,

112
00:05:47,182 --> 00:05:48,442
Prasanna Malaiyandi:
rather than the realist.

113
00:05:48,692 --> 00:05:51,287
W. Curtis Preston: So you're familiar with
the concept of Schrodinger's cat, right?

114
00:05:51,697 --> 00:05:55,237
Prasanna Malaiyandi:
Based on TV shows, movies,

115
00:05:55,817 --> 00:05:56,297
W. Curtis Preston: Okay.

116
00:05:57,047 --> 00:05:57,287
Yeah.

117
00:05:57,287 --> 00:06:01,817
So it's just a concept, as I understand
the concept that you have this cat in

118
00:06:01,817 --> 00:06:06,287
a box, and as long as you don't look in
the box, the cat is both alive and dead.

119
00:06:06,354 --> 00:06:09,547
But once you look in the box, you will
know that the cat is alive or dead.

120
00:06:09,577 --> 00:06:11,677
That's the concept of Schrodinger's cat.

121
00:06:12,367 --> 00:06:18,097
And the reason why this is relevant
today is that we have the author of

122
00:06:18,157 --> 00:06:24,757
a blog called Schrodinger's Backup:
when good documentation goes bad.

123
00:06:25,147 --> 00:06:28,687
He's been in the IT industry
almost as long as I have.

124
00:06:29,027 --> 00:06:30,827
He comes to us from the UK.

125
00:06:31,097 --> 00:06:33,527
Welcome to the podcast, Gary Williams.

126
00:06:35,492 --> 00:06:37,142
Gary Williams: Thank you and
thank you for the invite.

127
00:06:38,277 --> 00:06:39,717
W. Curtis Preston: I saw that title.

128
00:06:39,717 --> 00:06:41,437
And I was like, I gotta get
this guy on the podcast.

129
00:06:41,437 --> 00:06:42,987
Prasanna Malaiyandi:
Curtis was so excited.

130
00:06:42,987 --> 00:06:44,727
Gary, you have no idea.

131
00:06:44,907 --> 00:06:46,497
This is like one of his favorite topics.

132
00:06:48,152 --> 00:06:48,782
Gary Williams: Thank you.

133
00:06:48,784 --> 00:06:51,034
I don't know if I coined the term.

134
00:06:51,034 --> 00:06:53,824
I have seen it used since I'd
like to think I coined the term,

135
00:06:53,824 --> 00:06:54,904
but I don't know for certain,

136
00:06:55,079 --> 00:06:55,569
W. Curtis Preston: why not?

137
00:06:55,576 --> 00:06:57,510
Gary Williams: it might be something
that I heard and I just copied because

138
00:06:57,510 --> 00:07:00,180
it's just sounds really cool when
it's perfectly accurate, I think.

139
00:07:00,700 --> 00:07:03,040
It was all three or four companies ago.

140
00:07:03,069 --> 00:07:06,415
The lessons we learned still
definitely apply today, but this

141
00:07:06,415 --> 00:07:08,635
happened about three companies back.

142
00:07:09,235 --> 00:07:09,985
So about 10 years ago.

143
00:07:10,360 --> 00:07:12,220
W. Curtis Preston: So what
was your role at the time?

144
00:07:13,160 --> 00:07:17,480
Gary Williams: So my role at the time
was a senior network engineer or senior

145
00:07:17,480 --> 00:07:18,950
support engineer, something like that.

146
00:07:18,950 --> 00:07:24,322
W. Curtis Preston: OK, And you had
the, the gall to, to ask about backups.

147
00:07:25,337 --> 00:07:26,267
Gary Williams: No, I didn't.

148
00:07:26,267 --> 00:07:30,577
I was  overconfident with our backups,
let's say so we had the backup

149
00:07:30,577 --> 00:07:33,727
software, I think it was backup exec.

150
00:07:34,177 --> 00:07:37,079
And, we had all the
servers being backed up.

151
00:07:37,129 --> 00:07:39,149
We had everything going to dual tapes.

152
00:07:39,539 --> 00:07:40,889
The tapes were going off site.

153
00:07:40,919 --> 00:07:42,269
Everything was working.

154
00:07:43,099 --> 00:07:44,589
W. Curtis Preston: Jewel, jewel tapes?

155
00:07:45,439 --> 00:07:45,659
Gary Williams: Dual tapes.

156
00:07:45,659 --> 00:07:48,249
We actually had the backups,
the software was writing

157
00:07:48,639 --> 00:07:50,159
effectively RAID-1 one backups.

158
00:07:50,159 --> 00:07:51,489
So it was writing to two tapes.

159
00:07:52,479 --> 00:07:53,651
W. Curtis Preston: Oh, duel, capes.

160
00:07:53,681 --> 00:07:54,071
Okay.

161
00:07:54,161 --> 00:07:55,741
I heard, for some reason I heard Jewel.

162
00:07:55,741 --> 00:07:56,471
I don't know why.

163
00:07:57,381 --> 00:07:58,261
Gary Williams: It's the English accent.

164
00:07:58,281 --> 00:08:00,831
And yeah.

165
00:08:00,831 --> 00:08:03,231
So it's going to two tapes simultaneously.

166
00:08:03,231 --> 00:08:07,219
So the idea was that even if a tape
broke, or if something happens to the

167
00:08:07,219 --> 00:08:10,959
backup and we weren't entirely sure of,
or you couldn't restore from one of the

168
00:08:11,119 --> 00:08:15,609
tapes, you could then get the other tape
and use that tape to do the restore.

169
00:08:15,969 --> 00:08:17,599
So we had all that stuff going on.

170
00:08:18,309 --> 00:08:21,924
And we got all the emails and of
course we're getting the emails

171
00:08:21,934 --> 00:08:24,924
saying all the backups are good,
everything must be absolutely fine.

172
00:08:24,954 --> 00:08:25,944
Why would we test them?

173
00:08:25,944 --> 00:08:29,814
Why we're busy enough with other tickets
and other stuff going on and projects.

174
00:08:30,024 --> 00:08:31,284
We haven't got time to test them.

175
00:08:31,284 --> 00:08:31,854
What's the point?

176
00:08:31,914 --> 00:08:32,694
We know they work.

177
00:08:33,234 --> 00:08:35,154
Prasanna Malaiyandi: And so it
looks like you were doing all the

178
00:08:35,154 --> 00:08:39,130
right things in terms of setting up
backups, Following the 3, 2, 1 rule.

179
00:08:39,130 --> 00:08:39,310
Right?

180
00:08:39,310 --> 00:08:42,160
Making sure your copies were offsite and.

181
00:08:42,360 --> 00:08:42,480
W. Curtis Preston: Yeah.

182
00:08:43,315 --> 00:08:45,565
Prasanna Malaiyandi: I think
that's probably better than maybe

183
00:08:45,565 --> 00:08:47,605
like 70% of the people out there.

184
00:08:47,623 --> 00:08:48,823
Who try to do backups.

185
00:08:48,852 --> 00:08:50,382
You're like doing the right things.

186
00:08:50,392 --> 00:08:51,612
You're like, oh, I'm good to go.

187
00:08:53,177 --> 00:08:54,107
Gary Williams: Yeah, absolutely.

188
00:08:54,167 --> 00:08:56,717
As I say, we had the emails,
we even checked the emails.

189
00:08:56,717 --> 00:08:59,508
I think we even had a shared folder
or something like that, where all

190
00:08:59,508 --> 00:09:03,648
the backups emails went, and if
one of us saw that the folder had

191
00:09:03,648 --> 00:09:05,238
an unread one going, we check it.

192
00:09:05,568 --> 00:09:08,958
If there was an error, someone would
get a ticket, it would get sorted out.

193
00:09:09,378 --> 00:09:12,918
If the error went on for several
days, there would be a conversation.

194
00:09:12,948 --> 00:09:14,448
We will get these things fixed.

195
00:09:14,498 --> 00:09:15,248
where's the problem.

196
00:09:15,248 --> 00:09:16,598
We know our backups are good.

197
00:09:17,478 --> 00:09:19,038
W. Curtis Preston: So you, you were a.

198
00:09:20,093 --> 00:09:21,353
You were, I don't know.

199
00:09:21,413 --> 00:09:24,053
I don't know what to call it, but
so instead of being a proponent

200
00:09:24,053 --> 00:09:26,941
of testing the backups, you were a
proponent of oh, everything's fine.

201
00:09:27,966 --> 00:09:29,676
Gary Williams: Unfortunately at that time.

202
00:09:29,706 --> 00:09:33,458
Yes, I was, sitting there and
quite fat, dumb and happy going.

203
00:09:33,518 --> 00:09:35,198
We've got the emails, the backups work.

204
00:09:35,228 --> 00:09:36,188
We know they work.

205
00:09:36,188 --> 00:09:36,968
Where's the problem.

206
00:09:37,388 --> 00:09:39,008
I didn't see any issue here at all.

207
00:09:39,518 --> 00:09:40,658
W. Curtis Preston: For what it's worth.

208
00:09:40,748 --> 00:09:45,998
I had a similar point in my
career and there was a time.

209
00:09:46,658 --> 00:09:52,843
I remember when I was at a company,
I won't give the actual name of the

210
00:09:52,843 --> 00:10:00,823
company, but I will just say it's a very,
well-known electronics manufacturer.

211
00:10:00,889 --> 00:10:08,989
and I had helped him set up
their backup system and I wasn't

212
00:10:08,989 --> 00:10:10,489
there just to do the backups.

213
00:10:10,489 --> 00:10:13,039
I was there to do sysadmin stuff.

214
00:10:13,051 --> 00:10:14,048
And they were a mess.

215
00:10:14,071 --> 00:10:19,333
th this was a, it was a small department
in this bigger, electronics company.

216
00:10:19,493 --> 00:10:20,863
It was an interesting department.

217
00:10:20,863 --> 00:10:21,316
They called it.

218
00:10:23,091 --> 00:10:25,641
Simulation modeling and research.

219
00:10:25,641 --> 00:10:30,653
So it was a revolutionary idea at
the time of the idea of modeling,

220
00:10:30,653 --> 00:10:34,325
like in a computer, what would
happen if you drop this device?

221
00:10:35,045 --> 00:10:36,365
And so they were doing this in a computer.

222
00:10:36,365 --> 00:10:40,084
It was a fascinating new at
the time, new field of science.

223
00:10:41,194 --> 00:10:43,024
So I was there to fix a
whole bunch of problems.

224
00:10:43,024 --> 00:10:47,044
One of which, for example, was
that every workstation, it was all

225
00:10:47,044 --> 00:10:51,364
Unix workstations, and every person
had root on their workstation.

226
00:10:51,514 --> 00:10:53,884
And that was the first
thing I was going to fix.

227
00:10:53,970 --> 00:10:59,010
But I also set up their backup
system and, the backups worked.

228
00:10:59,580 --> 00:11:07,205
So I assumed the restores would
work and it was some time.

229
00:11:07,355 --> 00:11:10,175
I was there long enough that
I went, I actually, at some

230
00:11:10,175 --> 00:11:11,615
point needed to do a restore.

231
00:11:11,885 --> 00:11:17,465
And I found out that those tape drives
were really good at writing data.

232
00:11:18,815 --> 00:11:22,305
And they were completely
incapable of reading data.

233
00:11:23,385 --> 00:11:24,045
Again, I don't want.

234
00:11:25,995 --> 00:11:27,885
I'm sure there was something
wrong with these drives, but

235
00:11:27,885 --> 00:11:29,715
they were IBM 3590 drives.

236
00:11:29,765 --> 00:11:33,663
Normally IBM drives are top of the line
or whatever, but there was something wrong

237
00:11:33,663 --> 00:11:35,373
with these drives that I was completely.

238
00:11:35,423 --> 00:11:38,733
So I guess what I'm saying
is you're not alone.

239
00:11:38,752 --> 00:11:42,783
even me who, I've spent my career
in this, although honestly, that's

240
00:11:43,053 --> 00:11:47,293
that event is on the list of
things that I think back to when.

241
00:11:49,093 --> 00:11:49,268
Prasanna Malaiyandi: Yeah.

242
00:11:49,318 --> 00:11:51,208
W. Curtis Preston: when I try
to get other people to do it.

243
00:11:52,148 --> 00:11:53,618
Gary Williams: Absolutely same with me.

244
00:11:53,741 --> 00:11:56,453
the backups that we were taking,
as I say, we were only a small

245
00:11:56,453 --> 00:11:59,303
team and we had all the emails.

246
00:11:59,303 --> 00:12:00,443
We had everything in place.

247
00:12:00,443 --> 00:12:03,263
We had the two tape
libraries doing the backups.

248
00:12:03,473 --> 00:12:06,242
So we thought we were in a really
good position because we had

249
00:12:06,322 --> 00:12:07,772
everything working the way it should.

250
00:12:08,102 --> 00:12:11,042
We even had documentation for how
all this stuff was put together.

251
00:12:11,072 --> 00:12:13,832
we actually had to consultancy come in
and help us put all this stuff together.

252
00:12:14,402 --> 00:12:17,222
Because at the time I was working
for a financial institution, we

253
00:12:17,222 --> 00:12:21,092
had to have certain boxes ticked,
and we had those boxes ticked

254
00:12:21,102 --> 00:12:22,442
because we have the documentation.

255
00:12:22,442 --> 00:12:24,272
We had the backups, they
were going off site.

256
00:12:24,782 --> 00:12:25,862
They were going off site.

257
00:12:25,952 --> 00:12:28,412
They were being looked after for us.

258
00:12:28,412 --> 00:12:31,262
We even recalled tapes to make
sure we could do the process

259
00:12:31,262 --> 00:12:32,522
and no tapes were getting lost.

260
00:12:32,912 --> 00:12:37,382
So we did that level of testing, but
what we never actually tested was

261
00:12:37,532 --> 00:12:40,352
actually restoring the data itself.

262
00:12:42,552 --> 00:12:46,536
And it was a bit of an epiphany
when we actually had someone come

263
00:12:46,536 --> 00:12:49,296
into the team who a brand new to IT.

264
00:12:49,836 --> 00:12:51,216
Had never worked in IT before.

265
00:12:51,216 --> 00:12:52,326
Always wanted to work in IT.

266
00:12:52,326 --> 00:12:55,646
Was actually employed in the business
in a completely different role.

267
00:12:55,646 --> 00:12:58,216
And then he actually said to me,
one day, I'd like to move into IT.

268
00:12:58,976 --> 00:13:00,156
I thought he was joking.

269
00:13:00,206 --> 00:13:01,836
It turns out no, he was actually serious.

270
00:13:01,836 --> 00:13:04,016
He was an ex-finance person
wanting to move into IT.

271
00:13:04,526 --> 00:13:09,056
So he applied internally, he got the job
and he started with us and he started

272
00:13:09,056 --> 00:13:12,606
looking through some old tickets and
he was saying things like, why did you

273
00:13:12,646 --> 00:13:14,816
do such and such a change this way?

274
00:13:15,266 --> 00:13:17,456
So there's a whole education
thing going on there.

275
00:13:17,726 --> 00:13:19,196
And that's when he asked the question.

276
00:13:20,036 --> 00:13:21,146
When did you test the backups?

277
00:13:21,146 --> 00:13:23,796
What do you mean test them.

278
00:13:23,796 --> 00:13:24,936
We've got the emails.

279
00:13:25,206 --> 00:13:26,910
Look, here, you can see the service.

280
00:13:26,910 --> 00:13:28,140
Here's the tape drives.

281
00:13:28,140 --> 00:13:29,040
Here's the tapes.

282
00:13:29,550 --> 00:13:30,300
We record the tape.

283
00:13:30,760 --> 00:13:31,620
Yeah, sure.

284
00:13:31,620 --> 00:13:33,570
But when did you restore something?

285
00:13:34,920 --> 00:13:39,030
And I, something I won't actually
forget because there was this look,

286
00:13:39,270 --> 00:13:40,890
there's only four of us in the IT team.

287
00:13:40,890 --> 00:13:44,490
We were a really small team for a
company of about 300 and there's

288
00:13:44,490 --> 00:13:48,750
this look going around the whole
office and everyone's going well, we

289
00:13:48,750 --> 00:13:50,250
haven't actually tested them have we?

290
00:13:51,210 --> 00:13:53,391
Prasanna Malaiyandi: It's like a
light bulb goes off and it's yeah.

291
00:13:54,381 --> 00:13:55,447
It's Ooh.

292
00:13:55,572 --> 00:13:57,062
Gary Williams: We like, hang on.

293
00:13:57,062 --> 00:13:59,014
yeah, we should probably test
one of they shouldn't we.

294
00:13:59,564 --> 00:14:00,104
Okay.

295
00:14:00,150 --> 00:14:05,634
what should we test and looking back
on it, it was a really insane moment

296
00:14:05,634 --> 00:14:07,524
just to think that we've had easy.

297
00:14:07,734 --> 00:14:11,484
I think what actually had the emails
was coming in for over a year.

298
00:14:11,914 --> 00:14:15,224
And yes, we'd had the odd backup failure
where something a time there, or there

299
00:14:15,224 --> 00:14:16,784
was a fault with one of the tape drives.

300
00:14:17,084 --> 00:14:18,584
These tape drives were quite old.

301
00:14:18,854 --> 00:14:22,154
So they actually had physical SCSI
cables that would sometimes play up.

302
00:14:22,664 --> 00:14:24,764
So you had to make sure the
SCSI cables were all firmly

303
00:14:24,764 --> 00:14:26,264
attached, the terminator was in.

304
00:14:26,684 --> 00:14:27,404
The good old days.

305
00:14:28,214 --> 00:14:28,514
And.

306
00:14:28,639 --> 00:14:29,739
Prasanna Malaiyandi: never
had to deal with restores?

307
00:14:29,739 --> 00:14:29,799
W. Curtis Preston: Yeah.

308
00:14:29,799 --> 00:14:33,503
And course you had both active
and passive, terminators as well.

309
00:14:34,198 --> 00:14:35,158
Gary Williams: Yeah, exactly.

310
00:14:35,257 --> 00:14:39,584
we did actually have to do some restores,
but we had, a storage array and the

311
00:14:39,584 --> 00:14:41,264
storage provider let us do snapshots.

312
00:14:41,444 --> 00:14:43,964
So 99% of the restores that we needed.

313
00:14:44,589 --> 00:14:46,599
Just copy and paste from the snapshot.

314
00:14:46,929 --> 00:14:47,409
Not a problem.

315
00:14:47,439 --> 00:14:48,909
You deleted that file not a problem.

316
00:14:48,909 --> 00:14:49,539
There it is.

317
00:14:49,899 --> 00:14:53,268
If something was deleted from a
desktop, the common response was, we

318
00:14:53,268 --> 00:14:54,858
don't back things up on your desktop.

319
00:14:55,338 --> 00:14:55,668
Sorry.

320
00:14:55,668 --> 00:14:56,298
That's tough.

321
00:14:56,378 --> 00:14:59,357
If you want it backed up, put it
onto the server, put it into your

322
00:14:59,357 --> 00:15:00,707
home drive or something like that.

323
00:15:00,707 --> 00:15:01,757
It will get backed up.

324
00:15:02,417 --> 00:15:05,807
So that was the general understood
consensus because it was a small company.

325
00:15:06,587 --> 00:15:10,907
Most of the time, this wasn't an issue,
and as I say, people deleted a file.

326
00:15:11,237 --> 00:15:13,337
I remember one time we had an Excel file.

327
00:15:13,337 --> 00:15:16,337
That was a real pain because
of all these financial macros.

328
00:15:16,907 --> 00:15:19,367
And we restored that from a snapshot.

329
00:15:19,597 --> 00:15:22,496
And it was still corrupt and we
had to go back a week or so, we

330
00:15:22,496 --> 00:15:25,046
managed to get the file back and
it was working and we actually said

331
00:15:25,046 --> 00:15:26,396
it and I remember it quite well.

332
00:15:26,396 --> 00:15:27,386
We said it within the team.

333
00:15:27,438 --> 00:15:27,888
that was lucky.

334
00:15:27,888 --> 00:15:30,588
We might have actually asked to get
the tapes on site and do a restore from

335
00:15:30,588 --> 00:15:33,118
the tapes, but the snapshot worked.

336
00:15:33,138 --> 00:15:34,636
Everything's fine, you know yeah.

337
00:15:34,655 --> 00:15:37,265
Prasanna Malaiyandi: Now you've
decided, okay, we haven't tested.

338
00:15:37,265 --> 00:15:39,485
Maybe we should actually
try doing the test.

339
00:15:39,905 --> 00:15:41,985
How did you decide what to test?

340
00:15:41,993 --> 00:15:44,003
Gary Williams: funny
enough, it was a new guy.

341
00:15:44,094 --> 00:15:47,244
the discussion was actually,
okay, you're the person sitting

342
00:15:47,244 --> 00:15:48,354
there looking through the tickets.

343
00:15:48,354 --> 00:15:49,974
You're looking through the documentation.

344
00:15:50,034 --> 00:15:51,054
You're new to it all.

345
00:15:51,474 --> 00:15:55,224
You want us to prove to you
that the restore process works.

346
00:15:55,224 --> 00:15:56,094
We know it does.

347
00:15:56,604 --> 00:15:57,324
Pick something.

348
00:15:57,984 --> 00:16:00,717
And then he sat there and he went,
How about the exchange server?

349
00:16:03,757 --> 00:16:05,527
Prasanna Malaiyandi:
Swinging for the fences!

350
00:16:06,047 --> 00:16:06,457
Gary Williams: Fine.

351
00:16:07,967 --> 00:16:09,987
So we thought, okay, fine.

352
00:16:10,033 --> 00:16:12,088
we'll get the tapes back on site.

353
00:16:12,118 --> 00:16:13,178
We'll do the restore.

354
00:16:13,198 --> 00:16:16,748
We'll prove that the backups work and we
can go back to what we're normally doing.

355
00:16:16,796 --> 00:16:18,566
all the project work, that kind of thing.

356
00:16:18,956 --> 00:16:20,006
We could spend a day on this.

357
00:16:20,006 --> 00:16:20,846
It will be good for us.

358
00:16:20,876 --> 00:16:21,506
Not a problem.

359
00:16:22,176 --> 00:16:25,076
We even went to the documentation and
got the documentation out and said,

360
00:16:25,526 --> 00:16:26,846
look, we've got the documentation.

361
00:16:27,116 --> 00:16:28,256
The tapes are coming in.

362
00:16:28,286 --> 00:16:29,636
This is going to be easy.

363
00:16:30,086 --> 00:16:30,596
And it wasn't.

364
00:16:31,131 --> 00:16:32,211
Prasanna Malaiyandi: Of course not.

365
00:16:33,081 --> 00:16:36,591
So when you decided to do the restore.

366
00:16:37,476 --> 00:16:40,416
Did you bring down your production
or were you like, I'm going to

367
00:16:40,416 --> 00:16:42,516
restore this into a safe spot and

368
00:16:44,401 --> 00:16:46,471
Gary Williams: Yeah, we couldn't bring
down production because the nature

369
00:16:46,471 --> 00:16:48,986
of the business was that we needed
to keep the server up and running.

370
00:16:48,986 --> 00:16:52,976
We actually had a spare server and I
think we're maybe had two spare servers.

371
00:16:52,976 --> 00:16:55,560
VMs were just starting to come
on the scene and we actually

372
00:16:55,560 --> 00:16:56,690
had a spare server racked.

373
00:16:56,690 --> 00:16:59,970
And the idea was that if we had a
server failure, we could take the

374
00:16:59,970 --> 00:17:01,390
physical discs out of one server.

375
00:17:01,600 --> 00:17:04,174
Put it into another server
power it on, be back running.

376
00:17:04,174 --> 00:17:06,588
this is also before the
days of re replicas.

377
00:17:06,588 --> 00:17:10,008
They were, again, just coming out on a
lot of software was super expensive and

378
00:17:10,048 --> 00:17:12,088
W. Curtis Preston: You're
giving me flashbacks, Gary.

379
00:17:14,178 --> 00:17:15,078
Gary Williams: the good old days.

380
00:17:15,388 --> 00:17:15,808
W. Curtis Preston: Yeah.

381
00:17:15,817 --> 00:17:17,753
Gary Williams: We had this
physical server and it had plenty

382
00:17:17,753 --> 00:17:19,103
of disc space to handle this.

383
00:17:19,313 --> 00:17:24,058
So we said, okay, Let's we've not
actually even powered this server on.

384
00:17:24,058 --> 00:17:25,498
I don't even think, I think
maybe it was powered on when

385
00:17:25,498 --> 00:17:26,518
we bought it and that was it.

386
00:17:26,938 --> 00:17:29,038
So we said we should test
that server out anyway.

387
00:17:29,068 --> 00:17:29,218
Yeah.

388
00:17:29,258 --> 00:17:30,078
Let's power it on.

389
00:17:30,388 --> 00:17:34,588
Let's get the data restored to
that server and bring exchange up.

390
00:17:34,588 --> 00:17:36,388
We can bring it up in an isolated network.

391
00:17:36,388 --> 00:17:39,133
Do some very basic tests on it,
because it was a small team.

392
00:17:39,133 --> 00:17:41,173
We had access to the networking guys.

393
00:17:41,228 --> 00:17:42,368
I'll say networking guys.

394
00:17:42,368 --> 00:17:45,608
We did a little bit of networking age
and there was one guy who did a lot

395
00:17:45,608 --> 00:17:48,929
of the really key networking, tasks.

396
00:17:49,349 --> 00:17:50,429
So none of that was a problem.

397
00:17:50,429 --> 00:17:53,309
We didn't have to wait months for tickets
or to get done or anything like that.

398
00:17:53,999 --> 00:17:58,379
So we set up this isolated network,
we got the tapes on site and we

399
00:17:58,379 --> 00:18:01,619
started doing the restore and that's
when it all went horribly wrong.

400
00:18:02,079 --> 00:18:03,689
Prasanna Malaiyandi: So
who was doing the restore?

401
00:18:03,929 --> 00:18:06,189
Gary Williams: I, if I recall, it
was actually our help desk guy.

402
00:18:06,249 --> 00:18:08,849
We S we said to him, look,
you came up with this.

403
00:18:10,044 --> 00:18:11,574
W. Curtis Preston: You
put a lot on this guy.

404
00:18:11,754 --> 00:18:12,864
It was his idea.

405
00:18:13,134 --> 00:18:16,207
And you're like, what, if you
think testing backups is so

406
00:18:16,207 --> 00:18:17,707
important, why don't you do it?

407
00:18:18,752 --> 00:18:20,672
Gary Williams: Pretty much
.We did put it on him.

408
00:18:20,704 --> 00:18:21,604
cause it was his idea.

409
00:18:21,604 --> 00:18:24,754
And we said, look, this is a really
good exercise for you to do again.

410
00:18:24,754 --> 00:18:27,034
Unfortunately, I'm going
to put my hands up to this.

411
00:18:27,034 --> 00:18:31,699
It's a bad thing to have done, but
we said, I'm a senior IT person.

412
00:18:31,699 --> 00:18:32,809
I know the backups are good.

413
00:18:32,882 --> 00:18:33,422
here you go.

414
00:18:33,422 --> 00:18:34,172
Here's the tapes.

415
00:18:34,172 --> 00:18:35,582
Here's the documentation.

416
00:18:36,032 --> 00:18:39,452
See you later and off he
goes and he comes back.

417
00:18:40,202 --> 00:18:42,662
I think it was about two, three
hours later, something like that.

418
00:18:42,812 --> 00:18:44,162
And he went, I can't get this working.

419
00:18:45,297 --> 00:18:45,357
W. Curtis Preston: Yeah.

420
00:18:45,357 --> 00:18:46,427
Gary Williams: What do you
mean you can't get it working.

421
00:18:46,427 --> 00:18:47,147
What's the problem.

422
00:18:48,467 --> 00:18:52,787
And I don't actually recall what the
problems, all the problems were, but

423
00:18:52,787 --> 00:18:56,387
I know that the server itself didn't
have enough disc space, even though

424
00:18:56,387 --> 00:18:59,627
it was supposed to have the disc
space, because the documentation said,

425
00:18:59,627 --> 00:19:01,337
you need partition sizes like this.

426
00:19:01,817 --> 00:19:04,067
And it actually changed since then.

427
00:19:04,127 --> 00:19:07,787
And we didn't realize, and that was
really the start of a lot of problems.

428
00:19:08,607 --> 00:19:09,007
W. Curtis Preston: Yeah.

429
00:19:09,038 --> 00:19:11,888
first off I will say that even though.

430
00:19:11,902 --> 00:19:13,554
the way you got there.

431
00:19:13,554 --> 00:19:15,624
I like the way you did it.

432
00:19:15,688 --> 00:19:20,332
what to say, even though the way you
got there was wrong, the fact that

433
00:19:20,332 --> 00:19:24,112
you, the fact that you had this person.

434
00:19:24,126 --> 00:19:26,923
do it, who wasn't the person,
that made the documentation.

435
00:19:27,133 --> 00:19:29,293
That's actually something
I push pretty heavily.

436
00:19:30,123 --> 00:19:35,734
And it's an idea that came from
back in my days when I was at a bank

437
00:19:35,734 --> 00:19:37,744
and we very much did test restores.

438
00:19:37,802 --> 00:19:39,521
first off we didn't have snapshots.

439
00:19:39,521 --> 00:19:40,611
We didn't have any of that stuff.

440
00:19:41,121 --> 00:19:46,481
And we had 10,000 employees and
any one of them was allowed to

441
00:19:46,481 --> 00:19:49,751
call into the help desk and ask
for a restore on any given day.

442
00:19:51,086 --> 00:19:55,045
And, so we would get 10
to 15 restores a day.

443
00:19:55,045 --> 00:20:00,034
So we tested pretty regular, but the
thing that we buy in that degree,  but

444
00:20:00,034 --> 00:20:04,654
the thing that we had to test in the
way that you did were these large

445
00:20:04,684 --> 00:20:10,068
server restores, we did a DR test
and it was an absolute imperative

446
00:20:10,068 --> 00:20:13,308
from the powers that be was that.

447
00:20:14,298 --> 00:20:16,338
Curtis wrote the documentation.

448
00:20:16,398 --> 00:20:20,028
Curtis cannot be the person
actually doing the test.

449
00:20:20,478 --> 00:20:24,798
Curtis needs to be standing back there,
listening closely to the problems that

450
00:20:24,798 --> 00:20:28,964
are happening, but, w which, which
was actually kind of nice, although

451
00:20:28,964 --> 00:20:34,454
it's nerve wracking to be the person
who wrote the documentation and then

452
00:20:34,454 --> 00:20:38,288
sitting there watching someone, you
think you've answered all the questions,

453
00:20:38,648 --> 00:20:40,328
but it's not like in this case, you.

454
00:20:40,358 --> 00:20:44,098
you had the classic example of
the documentation might've been

455
00:20:44,098 --> 00:20:45,628
correct, but it was out of date.

456
00:20:46,658 --> 00:20:51,198
Gary Williams: It was correct at the
time, the irony is very similar with you.

457
00:20:51,203 --> 00:20:52,823
I didn't actually write the documentation.

458
00:20:52,823 --> 00:20:55,973
It was written by the contractors
and consultants that came on.

459
00:20:56,065 --> 00:20:58,202
Actually signed off on the
documentation saying, yes, all

460
00:20:58,202 --> 00:20:59,372
the version numbers are correct.

461
00:20:59,372 --> 00:21:01,172
And I think I'd done a couple of updates.

462
00:21:01,212 --> 00:21:03,042
And then we'd had other
changes and the other people

463
00:21:03,042 --> 00:21:04,132
had forgoten or I'd forgotten.

464
00:21:04,152 --> 00:21:06,972
Probably I'd forgotten to update
the documentation because we

465
00:21:06,972 --> 00:21:08,352
were busy only a small team.

466
00:21:08,892 --> 00:21:12,852
And so things very slowly on, not
just that document, but on every other

467
00:21:12,852 --> 00:21:17,152
document that we had about the environment
become out of date and it was this

468
00:21:18,112 --> 00:21:21,042
snowball of errors that had crept in.

469
00:21:21,622 --> 00:21:25,582
And the thing that we realized is
actually having no documentation

470
00:21:25,612 --> 00:21:28,545
would have been better because
the documentation was lying to us.

471
00:21:28,565 --> 00:21:31,955
this poor guy is sitting there
going, I followed steps three, four,

472
00:21:31,955 --> 00:21:35,435
and five, but I can't do step six
because step five doesn't work.

473
00:21:36,035 --> 00:21:36,335
What do you mean?

474
00:21:36,335 --> 00:21:36,905
It doesn't work.

475
00:21:36,935 --> 00:21:38,915
And that's when we found that
there was a service pack that

476
00:21:38,915 --> 00:21:39,995
was missing from exchange.

477
00:21:39,995 --> 00:21:44,175
So it couldn't go any further and it just
kept on building and building like this.

478
00:21:44,965 --> 00:21:46,868
Prasanna Malaiyandi: That
is an interesting problem.

479
00:21:46,893 --> 00:21:50,193
How do you keep your documentation
up to date as you're making these

480
00:21:50,193 --> 00:21:54,393
changes and making sure everyone across
the environment knows like where the

481
00:21:54,393 --> 00:21:57,183
documentation is and all the rest of that.

482
00:21:57,277 --> 00:22:01,237
Gary Williams: today, we use a Wiki
solution for all of our documentation.

483
00:22:01,717 --> 00:22:05,067
The idea behind that of course,
is the Wiki is so easy to edit.

484
00:22:05,717 --> 00:22:08,237
But you still don't or
sometimes you still don't.

485
00:22:08,237 --> 00:22:10,217
You make a note, I'll do
that tomorrow or next week.

486
00:22:10,247 --> 00:22:12,437
So there is still the exact same risk.

487
00:22:12,887 --> 00:22:16,644
And even in my current place, we've seen
this with certain, we do testing as well.

488
00:22:16,644 --> 00:22:20,649
We do a lot more testing now than,
anywhere I've ever worked before.

489
00:22:20,649 --> 00:22:23,499
And even with a lot of the
modern systems with Amazon.

490
00:22:24,129 --> 00:22:26,289
Backups to S3 and all this kind of stuff.

491
00:22:26,319 --> 00:22:28,929
We still test to make sure
that everything's correct,

492
00:22:28,929 --> 00:22:29,829
that we know what we're doing.

493
00:22:29,859 --> 00:22:32,749
That those Wiki pages
are fully up to date.

494
00:22:32,775 --> 00:22:37,060
we did some AD restore testing not so
long ago and we found, not major errors,

495
00:22:37,090 --> 00:22:40,930
but there was a couple of little issues
there with the restore process, which

496
00:22:40,930 --> 00:22:44,040
just needed a few corrections in the
documentation, just, as like a permissions

497
00:22:44,040 --> 00:22:46,530
era type of thing where we couldn't
actually get access to the bucket.

498
00:22:46,890 --> 00:22:48,330
So we had to make some changes there.

499
00:22:48,360 --> 00:22:50,900
So even with all the
modern backup software.

500
00:22:50,920 --> 00:22:52,310
It's still so important.

501
00:22:52,363 --> 00:22:55,633
W. Curtis Preston: I talked about those
DR tests that we did back in the day and.

502
00:22:57,413 --> 00:23:00,503
The, and the fact that we always
had someone who wasn't me doing the

503
00:23:00,503 --> 00:23:05,361
tests, and frequent listeners to the
podcast will have heard this before.

504
00:23:05,361 --> 00:23:16,710
But if we define a successful restore, as
we got from A to Z without having to ask

505
00:23:16,710 --> 00:23:22,075
Curtis, what does this line mean, not a
single one of the restores was successful.

506
00:23:22,075 --> 00:23:27,631
so if Curtis ever got, blown up and,
whatever, the chances of a restore

507
00:23:27,641 --> 00:23:32,248
going completely without a hitch
was, zero, which is why you talked

508
00:23:32,248 --> 00:23:35,626
about updating, there's always little
things that you have to update.

509
00:23:36,316 --> 00:23:39,466
I would suggest that
original documentation.

510
00:23:39,466 --> 00:23:42,240
and again, take this for what it's
worth to anybody who's listening.

511
00:23:42,271 --> 00:23:47,051
the first mistake was writing
the documentation in a way that

512
00:23:47,051 --> 00:23:48,591
it can easily get outdated.

513
00:23:48,616 --> 00:23:50,566
our exchange server is 75.

514
00:23:50,566 --> 00:23:50,786
Terra...

515
00:23:50,786 --> 00:23:51,333
r ight.

516
00:23:51,353 --> 00:23:52,248
that's a problem.

517
00:23:52,278 --> 00:23:56,652
So if you're going to hand that to a
restore documentation, what it should say

518
00:23:56,652 --> 00:24:01,899
is before beginning the restore, go look
at the size of the backups, And figure

519
00:24:01,899 --> 00:24:06,369
out how big the current exchange server
is, and then size the volume accordingly.

520
00:24:06,584 --> 00:24:10,475
yeah, that, that line wouldn't
have gone out of date as quickly.

521
00:24:10,965 --> 00:24:13,271
it is a real challenge by the way.

522
00:24:13,300 --> 00:24:17,610
this idea of what it's like to update
documentation, by the way, back in

523
00:24:17,610 --> 00:24:20,520
the day we were using Wordperfect.

524
00:24:20,661 --> 00:24:21,191
Yeah.

525
00:24:21,221 --> 00:24:26,277
And I remember the official
company standard was WordPerfect,

526
00:24:26,297 --> 00:24:31,167
because we could use it on, we
had Unix versions of WordPerfect.

527
00:24:31,187 --> 00:24:33,837
By the way, curses spaced WordPerfect.

528
00:24:33,857 --> 00:24:35,550
Not this fancy Windows.

529
00:24:35,550 --> 00:24:37,624
what you'd see is what
you get editing stuff.

530
00:24:37,624 --> 00:24:39,836
This was text on a screen.

531
00:24:39,842 --> 00:24:41,602
and I remember getting in a fight over.

532
00:24:42,397 --> 00:24:46,089
There was this one guy that was
new and he wanted to use Word

533
00:24:46,089 --> 00:24:48,159
because nobody used WordPerfect.

534
00:24:48,549 --> 00:24:50,469
And we were like, we don't care.

535
00:24:50,619 --> 00:24:53,274
We use WordPerfect here
for our documentation.

536
00:24:53,634 --> 00:24:56,454
And if you want your documentation
to fit into our documentation,

537
00:24:56,454 --> 00:24:57,554
you will use Wordperfect.

538
00:24:57,774 --> 00:24:58,554
And you will like it.

539
00:25:00,164 --> 00:25:02,234
Gary Williams: I remember our
first days of moving across

540
00:25:02,234 --> 00:25:03,584
the world where you had the.

541
00:25:04,914 --> 00:25:08,654
Word had the ability to mimic
WordPerfect key presses.

542
00:25:08,654 --> 00:25:10,154
So you could transition easily.

543
00:25:12,014 --> 00:25:12,644
Good old days.

544
00:25:12,834 --> 00:25:15,364
W. Curtis Preston: Good old days, but I
think what you're doing now with the Wiki,

545
00:25:15,364 --> 00:25:17,074
I think that's a much better approach.

546
00:25:17,889 --> 00:25:18,399
Gary Williams: It is.

547
00:25:18,449 --> 00:25:20,819
There's permissions list behind
it, obviously, so that not everyone

548
00:25:20,819 --> 00:25:23,183
can get access to it, but it's
the right people can get access.

549
00:25:23,217 --> 00:25:25,467
but what it means is everyone
in the team can get access.

550
00:25:25,467 --> 00:25:26,337
They can all update.

551
00:25:26,337 --> 00:25:26,457
It.

552
00:25:26,607 --> 00:25:27,627
There's a history as well.

553
00:25:27,627 --> 00:25:32,547
So the other thing that we didn't have
is the backup of the documentation

554
00:25:32,727 --> 00:25:34,767
was on the server we were backing up.

555
00:25:35,407 --> 00:25:36,037
Prasanna Malaiyandi: Oh,

556
00:25:36,342 --> 00:25:37,092
Gary Williams: Exactly.

557
00:25:37,092 --> 00:25:42,042
So we, all we had was that documentation
and looking back on it, we made

558
00:25:42,042 --> 00:25:43,422
quite a few mistakes like this.

559
00:25:43,422 --> 00:25:47,232
We had the, let's say we had the
documentation on the file server.

560
00:25:47,232 --> 00:25:48,372
So if the file server was lost.

561
00:25:48,372 --> 00:25:49,632
How did you get your documentation?

562
00:25:50,292 --> 00:25:53,052
And it was, again, something that
the helpdesk guy pointed out to us.

563
00:25:53,232 --> 00:25:54,432
How did you get your documentation?

564
00:25:54,822 --> 00:25:56,502
That's fine, actually.

565
00:25:56,502 --> 00:25:57,102
How would we.

566
00:25:59,402 --> 00:26:01,232
Prasanna Malaiyandi: Sometimes
it's an outside perspective or

567
00:26:01,232 --> 00:26:04,146
someone's Hey, how are you actually
going to get this stuff done?

568
00:26:05,571 --> 00:26:08,481
Gary Williams: Something I think
it's really important to know is at

569
00:26:08,481 --> 00:26:10,461
the time I was a senior IT person.

570
00:26:10,821 --> 00:26:13,701
There's a colleague of mine who was
senior and we had a network guy.

571
00:26:13,731 --> 00:26:15,110
All of us, were reasonably senior.

572
00:26:15,470 --> 00:26:16,640
This guy was a junior.

573
00:26:16,670 --> 00:26:21,060
He'd been working in finance for
three or four years beforehand.

574
00:26:21,120 --> 00:26:22,500
And then he'd just moved into IT.

575
00:26:22,920 --> 00:26:27,120
And he had such a fresh perspective on
everything that it really opened our eyes.

576
00:26:27,510 --> 00:26:30,660
And that was the day I learned that
it doesn't matter if you got 50

577
00:26:30,660 --> 00:26:32,430
years IT experience or five minutes.

578
00:26:32,430 --> 00:26:34,080
There's always something
you can learn from someone.

579
00:26:34,470 --> 00:26:36,990
And sometimes the most valuable
thing you can learn is from someone

580
00:26:36,990 --> 00:26:40,320
who is very new to the team,
fresh eyes, fresh perspective.

581
00:26:40,620 --> 00:26:41,530
It's invaluable.

582
00:26:42,010 --> 00:26:43,120
Prasanna Malaiyandi: 100% agree.

583
00:26:44,250 --> 00:26:47,430
W. Curtis Preston: There, there
is a perspective that you can only

584
00:26:47,430 --> 00:26:49,693
gain by being completely ignorant.

585
00:26:49,743 --> 00:26:53,883
He could have been not
junior to IT in this case.

586
00:26:53,883 --> 00:26:59,853
He was, but even if he's a senior IT
person, but he's joining your organization

587
00:26:59,853 --> 00:27:04,779
for the first time, another way, you look
at this person when they ask for things

588
00:27:04,779 --> 00:27:10,385
of like, when they ask stupid questions,
so how often do we, test our backups here?

589
00:27:11,495 --> 00:27:12,782
And you're like, we don't do that.

590
00:27:12,800 --> 00:27:17,720
Gary Williams: with my current place, any
new person we get into our IT team, we

591
00:27:17,720 --> 00:27:19,250
literally do that sort of thing with them.

592
00:27:19,250 --> 00:27:21,023
Now where we say, have a
look through the tickets.

593
00:27:21,323 --> 00:27:22,403
You've got any questions.

594
00:27:22,403 --> 00:27:24,263
Ask, have a look through the Wiki again.

595
00:27:24,263 --> 00:27:26,293
You've got any questions ask because.

596
00:27:27,038 --> 00:27:28,508
There's so many things in there.

597
00:27:28,508 --> 00:27:31,928
There's like the whole corporate
culture and there's corporate acronyms.

598
00:27:31,928 --> 00:27:34,568
And if they don't know what they
are, we've just found a problem

599
00:27:34,658 --> 00:27:39,738
because if there's one acronym
we have this, I, my brain's gone.

600
00:27:39,738 --> 00:27:40,005
Sorry.

601
00:27:40,005 --> 00:27:43,289
there's one acronym that we have,
that's very similar to an IT acronym.

602
00:27:43,359 --> 00:27:45,479
I can't remember what it
is off the top of my head.

603
00:27:45,629 --> 00:27:45,689
Yeah.

604
00:27:46,479 --> 00:27:49,599
But when you look at it, you think the,
IT term because you're an IT person,

605
00:27:49,599 --> 00:27:51,119
but it actually means the corporate.

606
00:27:51,210 --> 00:27:52,335
so there's that kind of thing.

607
00:27:52,365 --> 00:27:55,395
it's always important to spell out
these acronyms at the start of any

608
00:27:55,395 --> 00:27:59,595
documentation so that everyone knows
this is what you are referring to.

609
00:28:00,160 --> 00:28:00,560
Prasanna Malaiyandi: Especially

610
00:28:00,560 --> 00:28:02,750
W. Curtis Preston: it's
Prasanna's job on, on the podcast.

611
00:28:02,750 --> 00:28:05,724
If anybody ever brings up, an
acronym that, they don't spell out,

612
00:28:05,724 --> 00:28:07,924
Prasanna's, always making them spell it

613
00:28:07,924 --> 00:28:08,014
Prasanna Malaiyandi: out.

614
00:28:08,014 --> 00:28:08,224
Yep.

615
00:28:08,314 --> 00:28:09,514
I'm like, what does that really mean?

616
00:28:09,724 --> 00:28:10,444
Please tell me.

617
00:28:11,869 --> 00:28:12,679
Gary Williams: And this is the thing.

618
00:28:13,079 --> 00:28:17,359
You can walk into a meeting with
all the IT acronyms and every IT

619
00:28:17,359 --> 00:28:20,009
person sitting there will probably
think it's something different.

620
00:28:20,053 --> 00:28:24,243
I think DC is a good one because
DC's direct current data center.

621
00:28:24,983 --> 00:28:26,133
Things like that.

622
00:28:26,503 --> 00:28:30,643
And this is the sort of thing that we've
experienced several times, a few different

623
00:28:30,643 --> 00:28:34,013
companies I've worked for, and it's always
valuable to get that new person's insight.

624
00:28:34,013 --> 00:28:36,803
Because they don't know the
corporate terminology, they don't

625
00:28:36,803 --> 00:28:38,003
know the corporate acronyms.

626
00:28:38,273 --> 00:28:41,093
So it's worth getting them on board  and
going through all this stuff because

627
00:28:41,453 --> 00:28:45,863
they've got this fresh insight before they
learn that stuff and they can spot these

628
00:28:45,863 --> 00:28:47,633
problems before they become a problem.

629
00:28:48,593 --> 00:28:50,213
W. Curtis Preston: I  just
realized I haven't thrown out our

630
00:28:50,213 --> 00:28:53,163
usual disclaimer,  Prasanna and
I work for different companies.

631
00:28:53,163 --> 00:28:55,169
I work for Druva and he worked for Zoom.

632
00:28:55,469 --> 00:28:57,629
And this is not a podcast
of either company.

633
00:28:57,629 --> 00:28:59,749
And the opinions that you hear are ours.

634
00:29:00,099 --> 00:29:03,733
Please rate this podcast at
ratethispodcast.com/restore.

635
00:29:04,253 --> 00:29:10,218
And if you, are like our guest here
today, Gary who, just you're an IT person

636
00:29:10,218 --> 00:29:13,308
out there, and you want to talk about
your favorite subject to, or if you know

637
00:29:13,308 --> 00:29:14,686
what, maybe if you don't understand why

638
00:29:14,716 --> 00:29:15,308
Prasanna Malaiyandi: Come challenge, Mr.

639
00:29:15,308 --> 00:29:15,728
Backup.

640
00:29:16,703 --> 00:29:20,180
W. Curtis Preston: Some crazy person
would actually like them then, come on

641
00:29:20,180 --> 00:29:25,564
here related topics, cybersecurity, data
privacy, a number of related topics.

642
00:29:25,564 --> 00:29:28,998
We'd love to have you on as
a guest and, and reach out

643
00:29:28,998 --> 00:29:34,818
to me at wcurtispreston@gmail
or at @wcpreston on Twitter.

644
00:29:35,118 --> 00:29:36,138
And we'll get you on here.

645
00:29:36,259 --> 00:29:38,737
So, um, how did it turn.

646
00:29:38,744 --> 00:29:40,219
With your, with your restore.

647
00:29:41,014 --> 00:29:43,174
Gary Williams: So eventually we
got there, we actually got the

648
00:29:43,174 --> 00:29:46,414
exchange server fully restored
with correctly, the documentation.

649
00:29:46,468 --> 00:29:49,858
and I think it took three or
four days, something like that.

650
00:29:50,428 --> 00:29:54,005
And the thing is, if you want to do a
team building exercise, forget all the

651
00:29:54,005 --> 00:29:57,185
assault courses and other things they
have, you do get the team together and

652
00:29:57,185 --> 00:30:01,235
do a restore some of the best seriously.

653
00:30:01,268 --> 00:30:03,912
W. Curtis Preston: It's a bit like,
the trust exercises where you lean

654
00:30:03,912 --> 00:30:06,522
backwards and catches you it's like that.

655
00:30:07,737 --> 00:30:10,317
Gary Williams: I've also never seen
so many whiteboards being used to

656
00:30:10,317 --> 00:30:13,497
describe issues and draw diagrams
of how things hung together.

657
00:30:13,497 --> 00:30:16,017
And it was actually really good.

658
00:30:16,017 --> 00:30:19,893
And I will admit we ended up putting
some projects, not exactly on pause,

659
00:30:19,923 --> 00:30:23,193
but we put them to one side as all
of us started getting involved in

660
00:30:23,193 --> 00:30:26,883
this restore, because we realized we
actually had a very serious problem.

661
00:30:27,693 --> 00:30:28,553
I'll be honest.

662
00:30:28,583 --> 00:30:32,183
We gave the help desk guy, this
junior guy to IT the documentation.

663
00:30:32,183 --> 00:30:34,943
And we did expect him to
trip over a few things.

664
00:30:34,943 --> 00:30:39,053
He's a new person, some of the
terminology is new, fine, not a problem.

665
00:30:39,413 --> 00:30:40,373
We know we're there to help.

666
00:30:40,973 --> 00:30:43,583
What we didn't expect was us
to trip over the same issues.

667
00:30:43,613 --> 00:30:47,513
We honestly thought that, like you
were saying earlier, Curtis, that he

668
00:30:47,513 --> 00:30:49,073
was going to ask us some questions.

669
00:30:49,073 --> 00:30:53,123
We could do some updates to
the documentation, do it again,

670
00:30:53,123 --> 00:30:53,963
and everything would be fine.

671
00:30:55,343 --> 00:30:57,743
But we didn't expect to get
stumped by our own documentation.

672
00:30:57,803 --> 00:31:00,118
And unfortunately we actually
did, we're sitting there going

673
00:31:00,118 --> 00:31:01,901
through the documentation
going well, hang on a minute.

674
00:31:01,952 --> 00:31:05,989
we know the, the password is in
this password safe and that password

675
00:31:05,989 --> 00:31:09,709
should work, but something had
changed or I think at one point would

676
00:31:09,709 --> 00:31:11,432
actually, changed the security model.

677
00:31:11,452 --> 00:31:14,072
So it was requiring stronger passwords.

678
00:31:14,462 --> 00:31:17,652
So you couldn't actually use a
password that was on the backup.

679
00:31:17,652 --> 00:31:19,112
You had to go and reset an account.

680
00:31:19,112 --> 00:31:20,632
And it was lots of.

681
00:31:21,212 --> 00:31:24,572
It was nothing seriously,
wrong with a backup as such.

682
00:31:24,572 --> 00:31:26,672
And there's nothing seriously
wrong with the documentation,

683
00:31:27,062 --> 00:31:29,482
but it was lots of little things
that just piled up and piled up.

684
00:31:29,992 --> 00:31:33,603
And every time we took a couple of
steps forward, we thought, that's it.

685
00:31:34,203 --> 00:31:36,423
We've got this solved,
we'll get this restored.

686
00:31:37,233 --> 00:31:40,233
And then we got it all up and running
and got the server running and

687
00:31:40,233 --> 00:31:42,193
exchange server service wouldn't start.

688
00:31:42,693 --> 00:31:43,488
couldn't figure out why.

689
00:31:43,518 --> 00:31:45,508
I think that one took us a day to
go through and we ended up having

690
00:31:45,528 --> 00:31:46,728
to run some additional commands.

691
00:31:47,298 --> 00:31:49,698
And finally, we got there,
we got it all up and running.

692
00:31:50,448 --> 00:31:53,298
And I still remember, I think it was
actually like a Friday or something

693
00:31:53,718 --> 00:31:57,078
we're sitting there in the office and
went, yeah, that was a really good

694
00:31:57,078 --> 00:31:59,148
question know, can we restore the data?

695
00:31:59,148 --> 00:32:00,228
Thank you for asking it.

696
00:32:01,608 --> 00:32:03,258
we had a bit of a
celebration over that one.

697
00:32:03,443 --> 00:32:05,904
W. Curtis Preston: I would say
that, I like what you were saying

698
00:32:05,904 --> 00:32:08,003
about, it sounded like there
was a lot of collaboration.

699
00:32:08,023 --> 00:32:09,963
It sounds like there's a
lot of whiteboards going on

700
00:32:09,963 --> 00:32:11,305
and you were learning a lot.

701
00:32:11,326 --> 00:32:14,625
I would argue that the reason
that was the case is that you

702
00:32:14,625 --> 00:32:16,335
weren't doing it under duress.

703
00:32:16,905 --> 00:32:18,945
You were doing this as a test.

704
00:32:18,979 --> 00:32:23,479
if your exchange had been down for
three or four days, that would have

705
00:32:23,479 --> 00:32:25,444
been a very different experience.

706
00:32:26,434 --> 00:32:27,274
Gary Williams: Completely.

707
00:32:27,274 --> 00:32:31,071
It's something that we actually discussed,
that Friday afternoon, we've got the

708
00:32:31,071 --> 00:32:35,811
exchange server up and running and
the conversation was what happens if

709
00:32:35,811 --> 00:32:38,691
this happens for real, because sure.

710
00:32:38,721 --> 00:32:40,131
We got the backup restored.

711
00:32:40,161 --> 00:32:41,631
We know that the backup is good.

712
00:32:41,631 --> 00:32:45,561
Do you mean I was told to stay and it was
good, but the restore process wasn't good.

713
00:32:46,221 --> 00:32:48,861
And I think we focused way too
much on the backup itself and

714
00:32:48,861 --> 00:32:50,001
not the restore at that point.

715
00:32:50,058 --> 00:32:53,568
I said we had that conversation and
it was a matter of what would happen.

716
00:32:53,628 --> 00:32:55,668
And we knew that we were a small company.

717
00:32:55,668 --> 00:32:58,338
We knew we would have the CEO down
in the office, screaming at us.

718
00:32:58,368 --> 00:32:59,208
I need this back.

719
00:32:59,208 --> 00:33:03,228
We can't conduct business
and I'll be honest that day.

720
00:33:03,258 --> 00:33:07,728
We got a healthy lot of respect, both
for the backups, for documentation

721
00:33:07,728 --> 00:33:10,908
and the accuracy of documentation
and for the server itself.

722
00:33:11,368 --> 00:33:16,173
Because we knew that the company at that
point, the company relied on email so

723
00:33:16,173 --> 00:33:20,823
much that if that server did disappear,
and we took that long to get back up and

724
00:33:20,823 --> 00:33:23,823
running the loss, the financial loss to
the company and the reputational loss

725
00:33:23,823 --> 00:33:25,465
to the company would have been huge.

726
00:33:25,465 --> 00:33:30,543
And that also actually helped form some
push forward for additional resilience

727
00:33:30,543 --> 00:33:34,533
in , like, the servers and moving more
towards things like virtual machines,

728
00:33:34,533 --> 00:33:38,373
so that we had the ability to clone
and do other bits and pieces, because

729
00:33:38,373 --> 00:33:39,723
we could use that as an experience.

730
00:33:39,753 --> 00:33:43,314
It's look, this is how long potentially
worst case scenario it will take.

731
00:33:43,314 --> 00:33:46,684
It shouldn't because we're learning and
we need to do this a lot more often.

732
00:33:46,714 --> 00:33:48,214
We need to allocate time to do this.

733
00:33:48,784 --> 00:33:50,794
And the beauty was again,
being such a small company.

734
00:33:50,824 --> 00:33:53,344
We actually had the ear of a
couple of directors, so you could

735
00:33:53,344 --> 00:33:55,969
put this case forward and they
were really receptive to it.

736
00:33:56,469 --> 00:33:58,779
W. Curtis Preston: I want to
tack on something you said there.

737
00:34:00,506 --> 00:34:04,526
the fact that you and I
have been in that timeframe.

738
00:34:04,541 --> 00:34:12,614
young kids today,  they don't understand
what it was like back then, when you had

739
00:34:12,614 --> 00:34:15,674
no resiliency, you had no redundancy.

740
00:34:16,214 --> 00:34:17,694
You had nothing.

741
00:34:17,720 --> 00:34:21,790
So we had a server that a
server had a disk drive.

742
00:34:21,810 --> 00:34:23,270
We didn't have mirroring.

743
00:34:23,570 --> 00:34:24,020
We didn't have

744
00:34:24,290 --> 00:34:24,380
Gary Williams: RAID.

745
00:34:24,380 --> 00:34:26,075
Although we had very,
we had rightful life.

746
00:34:26,075 --> 00:34:27,785
We got, we were really market.

747
00:34:27,865 --> 00:34:30,468
W. Curtis Preston: we didn't,
when I was back in the day, we

748
00:34:30,468 --> 00:34:35,308
literally were installing data
directly on individual disk drives.

749
00:34:35,378 --> 00:34:38,953
I think we might've had redundant
power supplies on the servers that

750
00:34:38,953 --> 00:34:40,183
we  were using, and that was it.

751
00:34:40,763 --> 00:34:45,126
And so the loss of any one of those
components could take the server.

752
00:34:45,129 --> 00:34:45,579
Right.

753
00:34:45,969 --> 00:34:51,790
And, and now nowadays we move forward
to the days of virtualization and

754
00:34:51,790 --> 00:34:54,460
that you can just, if there's a little
problem with this server, you just

755
00:34:54,460 --> 00:34:56,620
move your VM over to another server.

756
00:34:56,830 --> 00:35:00,340
In fact, you can V motion at and
storage V motion, and you can

757
00:35:00,340 --> 00:35:04,300
move it while it's running, which
continues to boggle my brain.

758
00:35:04,775 --> 00:35:05,345
Gary Williams: likewise.

759
00:35:05,412 --> 00:35:08,094
W. Curtis Preston: And also
the devices that would, that so

760
00:35:08,094 --> 00:35:09,714
many of us have grown used to.

761
00:35:09,803 --> 00:35:13,282
I, at home I pretty much
live a solid state life.

762
00:35:13,304 --> 00:35:16,083
My TiVo has a solid state hard drive.

763
00:35:16,137 --> 00:35:22,587
and so those are so much more
reliable than the moving part

764
00:35:22,677 --> 00:35:24,757
drives that you and I grew up on.

765
00:35:24,769 --> 00:35:28,589
and I think as a result, they don't have

766
00:35:29,279 --> 00:35:35,600
the respect that you need to do to
test backups the way you should.

767
00:35:35,600 --> 00:35:36,020
I don't know.

768
00:35:36,981 --> 00:35:38,511
Just a quick editor's note.

769
00:35:38,671 --> 00:35:44,451
In the next section, Gary is going to
mention something called iLO and iDRAC.

770
00:35:44,991 --> 00:35:47,991
And he, we forgot to have him define it.

771
00:35:47,991 --> 00:35:49,131
So I'm doing that now.

772
00:35:49,521 --> 00:35:54,551
They are systems from Dell and HP,
the integrated Dell remote access

773
00:35:54,551 --> 00:35:57,131
controller and HP integrated lights out.

774
00:35:57,611 --> 00:36:01,721
They're both systems that help increase
the uptime of the server by notifying

775
00:36:01,721 --> 00:36:03,791
you of potential failures or issues.

776
00:36:05,391 --> 00:36:06,531
Back to your podcast.

777
00:36:07,560 --> 00:36:09,810
Gary Williams: One of the things
that we still do today, and this is

778
00:36:09,810 --> 00:36:13,620
probably me being paranoid coming
from that environment, we didn't

779
00:36:13,620 --> 00:36:17,000
get alerts on a service if a disk
failed, because it didn't really know.

780
00:36:17,000 --> 00:36:22,320
The ILOs and iDRACs were way too
expensive for us to have at that point.

781
00:36:22,620 --> 00:36:24,600
So daily server room checks go around.

782
00:36:24,660 --> 00:36:26,520
Is there any flashing lights
that shouldn't be flashing?

783
00:36:26,910 --> 00:36:29,400
And we still do that in
our data centers today.

784
00:36:29,460 --> 00:36:32,220
And we still do that with
some of our machines.

785
00:36:32,990 --> 00:36:36,140
We've actually got this philosophy in
place now where if a machine is up for

786
00:36:36,140 --> 00:36:40,040
more than 30 days, it needs to be rebooted
because we don't know if it's reboot safe.

787
00:36:40,430 --> 00:36:42,230
So we're starting to put uptime alarms in.

788
00:36:42,650 --> 00:36:43,550
Certainly on Windows.

789
00:36:43,550 --> 00:36:48,355
Linux is a bit different, but with
Windows, when it hits  a 30-day point.

790
00:36:48,385 --> 00:36:51,835
If we get an uptime alarm, it
means that there's possibly

791
00:36:51,835 --> 00:36:52,945
a patching issue with that.

792
00:36:52,975 --> 00:36:55,075
We should get an alarm from
the patching system as well.

793
00:36:55,735 --> 00:36:57,415
So we go off and we check.

794
00:36:57,461 --> 00:36:59,405
but the other thing we do something
similar with Linux as well.

795
00:36:59,405 --> 00:37:02,115
We're trying to get all the Linux
servers rebooted because generally

796
00:37:02,135 --> 00:37:05,225
with those, we can patch them hot, but
we still want to get them rebooted.

797
00:37:05,375 --> 00:37:06,535
Are they reboot safe?

798
00:37:07,005 --> 00:37:12,640
Because if we do lose power or machine
crashes, it's great having all that stuff

799
00:37:12,640 --> 00:37:16,090
there, but if it doesn't reboot, we've
got a problem and we may have a backup if

800
00:37:16,090 --> 00:37:20,620
that backup is inherited that corruption
or that problem we're in a bad place.

801
00:37:20,920 --> 00:37:26,477
So we do try to make sure that we've got,
these servers rebooted on a fairly regular

802
00:37:26,702 --> 00:37:27,812
Prasanna Malaiyandi:
actually very interesting.

803
00:37:27,812 --> 00:37:31,604
I never thought about that About the
fact that you need to reboot the systems

804
00:37:31,604 --> 00:37:34,814
and just make sure is a hardware and
dos and everything else could be.

805
00:37:36,674 --> 00:37:37,514
Gary Williams: Absolutely.

806
00:37:37,544 --> 00:37:40,034
The other thing that we've done
is we've actually turned up time

807
00:37:40,034 --> 00:37:42,444
on his head now in the old days.

808
00:37:42,464 --> 00:37:46,202
Now these uptime figures of two
years, three years, we'll put on the

809
00:37:46,202 --> 00:37:47,658
internet and it's look at my up time.

810
00:37:47,778 --> 00:37:48,768
Now it's the other way around.

811
00:37:49,188 --> 00:37:51,436
It's like, yeah, there's
an uptime of 45 days.

812
00:37:51,586 --> 00:37:52,586
Oh, look at my uptime.

813
00:37:52,606 --> 00:37:53,176
That's bad.

814
00:37:53,176 --> 00:37:54,316
We need to get this rebooted.

815
00:37:54,316 --> 00:37:55,756
And check it is reboot safe.

816
00:37:56,086 --> 00:37:59,836
Trying to find reboot windows sometimes
is a bit difficult, even with all the

817
00:37:59,836 --> 00:38:01,874
resilience . Just take systems down.

818
00:38:01,874 --> 00:38:04,709
but we do have some sort of bargaining
going on with various teams where

819
00:38:04,709 --> 00:38:08,939
we do try and reboot the systems
at least once a month, just to make

820
00:38:08,939 --> 00:38:10,339
sure that they are reboot safe.

821
00:38:10,589 --> 00:38:12,419
W. Curtis Preston: So help
me understand that phrase.

822
00:38:12,539 --> 00:38:14,798
W what do you mean when
you say reboot safe?

823
00:38:15,723 --> 00:38:20,823
Gary Williams: So reboot safe is simply
that if it's potentially a change can be

824
00:38:20,823 --> 00:38:25,743
made to a machine, that means a machine
when it reboots is going to crash, or

825
00:38:25,743 --> 00:38:28,213
there's going to be a problem where
it can't complete the boot corrupted

826
00:38:28,233 --> 00:38:29,643
boot loader or something like that.

827
00:38:30,123 --> 00:38:31,822
We've seen issues in the past where.

828
00:38:31,822 --> 00:38:33,865
Microsoft update has
corrupted the bootloader.

829
00:38:33,865 --> 00:38:36,685
So when you go to reboot,
it doesn't restart properly.

830
00:38:37,255 --> 00:38:40,585
So we've actually got the term
reboot safe, which just means I know

831
00:38:40,585 --> 00:38:43,225
if I have to reboot that server,
I don't have to worry about it.

832
00:38:43,245 --> 00:38:43,975
It will come up.

833
00:38:44,005 --> 00:38:48,365
You're printing system will start all the
services that need to start will start,

834
00:38:48,365 --> 00:38:51,082
because we've had issues in the past
where certain key services don't start.

835
00:38:51,082 --> 00:38:52,642
So we get a ticket.

836
00:38:52,732 --> 00:38:54,112
Can you please reboot this machine?

837
00:38:54,202 --> 00:38:54,412
Sure.

838
00:38:55,187 --> 00:38:57,207
Reboot it, you walk off.

839
00:38:57,227 --> 00:38:59,477
You think it's done, but
the services don't start.

840
00:38:59,957 --> 00:39:02,057
Now, the alerting will alert on that.

841
00:39:02,777 --> 00:39:06,349
But in the meantime, you potentially still
down for a bit longer than you need to be.

842
00:39:06,769 --> 00:39:09,739
So we do these tests where we just
want something sure all the services

843
00:39:09,739 --> 00:39:11,599
that need to start actually start.

844
00:39:12,019 --> 00:39:15,649
And it comes up completely clean
and working exactly how it should.

845
00:39:15,704 --> 00:39:16,673
W. Curtis Preston: you're giving me, Yeah.

846
00:39:16,733 --> 00:39:20,042
And by the way, I agree with you
with this idea of, the occasional

847
00:39:20,042 --> 00:39:23,831
reboots and I agree that it's, that
it's a practice that has gone by

848
00:39:23,831 --> 00:39:25,661
the wayside by a lot of people.

849
00:39:25,811 --> 00:39:32,111
And I remember, I can remember
the first time I left my,

850
00:39:32,233 --> 00:39:34,123
this is before I got the Mr.

851
00:39:34,123 --> 00:39:34,613
Backup.

852
00:39:34,663 --> 00:39:38,203
moniker and I got a different moniker
and I'll explain it in a minute.

853
00:39:38,563 --> 00:39:43,123
I was at a large oil and gas company
and no one had administered the data

854
00:39:43,123 --> 00:39:46,434
center, like a real sysadmin in years.

855
00:39:46,854 --> 00:39:50,461
And so I was going in there and
I was doing crazy things like

856
00:39:50,461 --> 00:39:52,231
installing the latest patch set.

857
00:39:52,298 --> 00:39:57,059
And this was, these were a Solaris
systems and, it required a reboot

858
00:39:57,089 --> 00:39:59,237
in order to, to install the patches.

859
00:39:59,987 --> 00:40:06,827
And what was happening was I
was like, 0 for 10, in terms

860
00:40:06,827 --> 00:40:08,560
of I would install a patch.

861
00:40:08,890 --> 00:40:11,920
I would reboot the server
and it wouldn't come back.

862
00:40:12,005 --> 00:40:16,282
And so I picked up the nickname
crash, because that's what I was

863
00:40:16,282 --> 00:40:19,992
just, I was literally, it's like  the
cure is worse than the disease.

864
00:40:20,322 --> 00:40:24,657
So it's we need to do this, but
I was doing, I was proactively

865
00:40:24,657 --> 00:40:26,337
doing damage to the environment.

866
00:40:27,382 --> 00:40:33,442
By doing the things I was doing, what I
did get really good at though is restoring

867
00:40:33,442 --> 00:40:35,742
their environment because it kept,

868
00:40:35,795 --> 00:40:38,581
so what it turned out, the
things that were really.

869
00:40:39,996 --> 00:40:45,516
I don't know uh in trouble were the
disks themselves, because we actually

870
00:40:45,626 --> 00:40:48,486
powered down the servers for some of them.

871
00:40:48,486 --> 00:40:52,356
And that's when things really
went awry because the disk

872
00:40:52,356 --> 00:40:54,096
drives had never been turned off.

873
00:40:54,846 --> 00:40:56,046
And then, yeah.

874
00:40:56,136 --> 00:40:57,808
And then, they wouldn't come back on.

875
00:40:57,808 --> 00:41:03,160
So I had to get all new disk drives and
then, and then do the restore, but yeah.

876
00:41:03,220 --> 00:41:03,420
Prasanna Malaiyandi: Yeah.

877
00:41:04,270 --> 00:41:06,910
Gary Williams: Yeah, but even with
the virtual machines, we still like to

878
00:41:06,910 --> 00:41:10,090
reboot them and to make sure all the
services that should come up do come up.

879
00:41:10,450 --> 00:41:14,110
We've even in some cases taken that
paranoia to the next level where we'll

880
00:41:14,110 --> 00:41:17,860
do a reboot test before we install
a patch or before we do something,

881
00:41:18,250 --> 00:41:21,100
just to make sure that it's not,
that patch that has caused a problem.

882
00:41:21,490 --> 00:41:24,250
Now, we generally don't do that for
the Microsoft patches, but we do

883
00:41:24,250 --> 00:41:26,350
that for certain application patches.

884
00:41:26,650 --> 00:41:28,500
And it's almost a sanity check.

885
00:41:29,795 --> 00:41:33,275
Because that way, if there is a
problem, we know that it is that patch

886
00:41:33,275 --> 00:41:36,395
that has caused a problem and not
something lurking from beforehand.

887
00:41:37,120 --> 00:41:39,580
Prasanna Malaiyandi: Going back to the
article you wrote, Gary, one of the

888
00:41:39,580 --> 00:41:43,829
things I liked in it was you talked
about this spreadsheet, if you will, that

889
00:41:43,829 --> 00:41:48,622
track sort of assets that were backed
up and you had a methodology that you

890
00:41:48,622 --> 00:41:53,357
called out in the article in terms of
how long you would wait before something

891
00:41:53,357 --> 00:41:57,152
had to be tested, Or how the longest
something could go without being tested.

892
00:41:57,152 --> 00:42:00,032
And there were certain things that were
critical in your environment that sort

893
00:42:00,032 --> 00:42:02,102
of had to be done more periodically.

894
00:42:03,652 --> 00:42:04,012
Gary Williams: Yeah.

895
00:42:04,012 --> 00:42:05,332
So what we did is.

896
00:42:05,399 --> 00:42:08,203
we had a spreadsheet, the list of all
the backups anyway, and one of the

897
00:42:08,203 --> 00:42:11,593
things we tried to do was make sure
that there was no clashing backups.

898
00:42:11,803 --> 00:42:14,313
So the exchange server would
get backed up at say, 10:00 PM.

899
00:42:14,343 --> 00:42:16,273
The file server get backed up at 11:00 PM.

900
00:42:16,303 --> 00:42:19,123
That kind of thing, because
otherwise we found there was a

901
00:42:19,123 --> 00:42:22,213
lot of issues on the network and
latency and all this kind of thing.

902
00:42:22,213 --> 00:42:24,613
So we wanted to stagger the
backups as much as possible.

903
00:42:25,393 --> 00:42:28,813
But what we did was we actually added
a column to that spreadsheet that said.

904
00:42:29,193 --> 00:42:33,928
Restore last tested, documentation
last updated, that kind of thing.

905
00:42:33,928 --> 00:42:37,738
So that we new when the backups
were tested and we knew when that

906
00:42:37,738 --> 00:42:39,198
documentation was last updated.

907
00:42:39,838 --> 00:42:42,737
And what we do is we actually
have, there was a formula in

908
00:42:42,737 --> 00:42:43,877
it that would color the cells.

909
00:42:44,267 --> 00:42:46,367
And if it was all green,
everything's fine.

910
00:42:46,397 --> 00:42:47,417
We've done a recent test.

911
00:42:47,417 --> 00:42:51,287
I think recent was like six months,
12 months, something like that.

912
00:42:51,380 --> 00:42:54,480
and if anything was over outside
of that window, it would go red.

913
00:42:54,870 --> 00:42:57,870
So I think the exchange was
every six months, the active

914
00:42:57,870 --> 00:42:59,430
directory was once a year.

915
00:42:59,460 --> 00:43:04,073
The file server was I think we
would restore a folder or a file

916
00:43:04,073 --> 00:43:05,463
every month, something like that.

917
00:43:05,463 --> 00:43:08,153
and we did this quite a lot and
we actually slowed down some

918
00:43:08,153 --> 00:43:10,133
of the tapes going off site for
things like the file server.

919
00:43:10,133 --> 00:43:13,214
So we could do a backup a couple of
days later, you do a restore test,

920
00:43:13,304 --> 00:43:14,714
update the date in the documentation.

921
00:43:14,714 --> 00:43:15,464
We know that's good.

922
00:43:15,674 --> 00:43:18,659
Send the tape off-site and
that's actually funny enough.

923
00:43:18,659 --> 00:43:20,459
That was a financial reason
as well because of the cost

924
00:43:20,459 --> 00:43:21,629
of sending the tapes offsite.

925
00:43:21,643 --> 00:43:25,123
but yeah, we started to do that
and we started to get quite good

926
00:43:25,123 --> 00:43:26,773
at being able to do these restores.

927
00:43:26,773 --> 00:43:30,403
We were even able to get some
additional hardware and we even

928
00:43:30,403 --> 00:43:33,193
starting to do some tests where
we're restoring to virtual machines.

929
00:43:33,473 --> 00:43:35,173
Because doing that process.

930
00:43:35,173 --> 00:43:37,063
We found we could get them
up and running a lot quicker.

931
00:43:37,483 --> 00:43:40,153
We had a bit more room to
breathe and we could have a

932
00:43:40,153 --> 00:43:41,743
much better virtual environment.

933
00:43:42,223 --> 00:43:45,313
And then we've got into some other really
clever stuff where we had a physical

934
00:43:45,313 --> 00:43:48,583
domain controller and a virtual domain
controller, and we tested fail-over

935
00:43:48,593 --> 00:43:51,030
and all this, we got really advanced

936
00:43:52,030 --> 00:43:54,896
W. Curtis Preston: So you're
saying that, the green column was

937
00:43:54,896 --> 00:43:59,539
actually the color of that column
was automatically determined by By

938
00:43:59,539 --> 00:44:00,809
Gary Williams: the age of the last test.

939
00:44:01,149 --> 00:44:01,669
W. Curtis Preston: That's pretty cool

940
00:44:01,669 --> 00:44:03,389
Prasanna Malaiyandi: Conditional
formatting, Curtis in Excel.

941
00:44:04,219 --> 00:44:04,639
Gary Williams: that's it.

942
00:44:04,704 --> 00:44:06,594
W. Curtis Preston: you're probably
better at Excel that I am,

943
00:44:06,594 --> 00:44:08,622
but, Gary, this has been great.

944
00:44:08,652 --> 00:44:10,059
I, I love this story.

945
00:44:10,059 --> 00:44:13,946
I love that it, like the other story
we had, where, I don't know if you

946
00:44:14,036 --> 00:44:18,371
listen to the podcast at all, Gary, but
we had an episode where someone, they

947
00:44:18,371 --> 00:44:22,031
tested their backups by essentially
deleting their entire data center.

948
00:44:22,266 --> 00:44:24,386
Prasanna Malaiyandi: Paul
van Dyke episode 135.

949
00:44:24,616 --> 00:44:24,896
Gary Williams: Wow.

950
00:44:25,656 --> 00:44:26,701
I haven't heard that one.

951
00:44:26,701 --> 00:44:29,671
I have heard some of the others
and I have to say I'm a fan.

952
00:44:29,776 --> 00:44:32,709
W. Curtis Preston: And it was
that one that would just, it

953
00:44:32,709 --> 00:44:34,779
hurt to, to listen to his story.

954
00:44:35,019 --> 00:44:37,926
And it was, he agrees that
it was a really dumb idea.

955
00:44:37,926 --> 00:44:41,987
It did eventually work out,
but it it took him awhile.

956
00:44:42,932 --> 00:44:44,132
Gary Williams: I can imagine.

957
00:44:44,162 --> 00:44:48,501
I I just remember the pain of the
exchange server and whilst I've not had

958
00:44:48,501 --> 00:44:50,331
a repeat of that pain since, because.

959
00:44:50,821 --> 00:44:54,699
The software is better these days, the
restores are a lot quicker and you do

960
00:44:54,699 --> 00:44:56,139
have a lot more options to play with.

961
00:44:56,142 --> 00:44:59,480
we still have that pain from time to
time when trying to do certain restores

962
00:44:59,480 --> 00:45:00,860
and testing the environment out.

963
00:45:01,220 --> 00:45:05,517
So I am still not that brave to
do something like that, but, yeah,

964
00:45:05,517 --> 00:45:06,537
I think we're getting there and.

965
00:45:06,537 --> 00:45:09,022
W. Curtis Preston: Not brave
be the word I would use, but.

966
00:45:11,742 --> 00:45:14,142
Gary Williams: Now we have talks about
bringing in things like the chaos

967
00:45:14,142 --> 00:45:18,912
monkey and taking down things, but
yeah, that's a test for another day.

968
00:45:21,342 --> 00:45:21,642
W. Curtis Preston: Yeah.

969
00:45:21,670 --> 00:45:24,276
thanks Prasanna for your
usual great questions

970
00:45:24,371 --> 00:45:26,561
Prasanna Malaiyandi: Always and
nice chatting with you, Gary.

971
00:45:26,561 --> 00:45:27,671
That was fun.

972
00:45:29,096 --> 00:45:29,576
Gary Williams: Thank you.

973
00:45:30,236 --> 00:45:31,889
W. Curtis Preston: and,
thanks to the listeners again.

974
00:45:31,889 --> 00:45:33,072
this is you're why we're here.

975
00:45:33,092 --> 00:45:34,562
You're why we sit here and talk to us.

976
00:45:34,661 --> 00:45:34,961
Prasanna Malaiyandi: Curtis.

977
00:45:34,961 --> 00:45:36,401
And we'll talk to each other anyway.

978
00:45:36,401 --> 00:45:36,731
It doesn't matter.

979
00:45:37,421 --> 00:45:37,711
W. Curtis Preston: Yeah.

980
00:45:37,711 --> 00:45:38,681
Yeah, exactly.

981
00:45:38,681 --> 00:45:42,341
We'll probably be talking about
table saws or video editing tools,

982
00:45:42,341 --> 00:45:46,766
but, anyway, remember to subscribe
so that you can restore it all.