1
00:00:00,000 --> 00:00:04,290
You found the backup wrap up
your go-to podcast for all things

2
00:00:04,290 --> 00:00:07,080
backup recovery and cyber recovery.

3
00:00:07,620 --> 00:00:12,570
This episode contains one of the
craziest DR stories I've ever heard.

4
00:00:12,870 --> 00:00:19,350
In 2021, we talked to Paul Van Dyke and
IT supervisor in Kodiak Island, Alaska.

5
00:00:19,770 --> 00:00:25,140
He tested his DR system by intentionally
destroying his production environment.

6
00:00:25,634 --> 00:00:30,705
Spoiler alert, he lived and so did his
data, but not without significant pain.

7
00:00:31,140 --> 00:00:35,490
This is one of our favorite episodes to
look back on, so we're rebroadcasting

8
00:00:35,490 --> 00:00:38,940
it this week since we just got
done talking about DR testing.

9
00:00:39,270 --> 00:00:40,350
I'm sure you'll love it.

10
00:00:41,010 --> 00:00:45,780
By the way, if this is your first
episode, I'm w Curtis Preston, AKA, Mr.

11
00:00:45,780 --> 00:00:49,290
Backup, and I've been passionate
about backup and recovery for

12
00:00:49,290 --> 00:00:51,570
over 30 years, ever since.

13
00:00:51,570 --> 00:00:55,590
I had to tell my boss that we had no
backups of the production database.

14
00:00:55,590 --> 00:00:56,340
We just lost.

15
00:00:57,135 --> 00:01:01,005
I don't want that to happen to
you, and that's why I do this.

16
00:01:01,455 --> 00:01:06,615
On this podcast, we turn unappreciated
backup admins into Cyber Recovery Heroes.

17
00:01:06,914 --> 00:01:09,164
This is the backup wrap up.

18
00:01:24,073 --> 00:01:28,468
Welcome to the show . I'm your
host, w Curtis Preston, AKA, Mr.

19
00:01:28,468 --> 00:01:33,868
Backup, and I have with me my personal
financial advisor, Prasanna Malaiyandi.

20
00:01:33,868 --> 00:01:34,328
What's up Prasanna.

21
00:01:34,328 --> 00:01:35,198
I am good.

22
00:01:35,218 --> 00:01:37,318
Curtis, what advice
have I been giving you?

23
00:01:37,318 --> 00:01:37,348
I.

24
00:01:37,438 --> 00:01:37,528
Well,

25
00:01:38,188 --> 00:01:39,838
you know, I was talking to you about that.

26
00:01:39,843 --> 00:01:40,048
Yeah, yeah.

27
00:01:40,048 --> 00:01:44,188
That loan that I was thinking about
doing, and you've been advising me,

28
00:01:44,518 --> 00:01:49,138
you know, that there's this idea that
I have of, of doing a loan to a friend

29
00:01:49,498 --> 00:01:54,158
and it's a large enough that, um, I
was like, what do you think Prasanna?

30
00:01:54,178 --> 00:01:54,208
I.

31
00:01:54,593 --> 00:01:57,923
And, you know, you gave me
advice on moving forward, but

32
00:01:58,613 --> 00:02:00,173
doing all of the right things.

33
00:02:00,233 --> 00:02:00,443
Yep.

34
00:02:00,688 --> 00:02:02,813
And, and I was really
surprised at some point.

35
00:02:02,813 --> 00:02:05,663
I, I really expect you, you to
say, well, you know, I was watching

36
00:02:05,663 --> 00:02:10,733
this YouTube video on, on personal
personal loan administration

37
00:02:11,753 --> 00:02:12,563
and so unfortunately this

38
00:02:12,923 --> 00:02:13,076
what this guy said.

39
00:02:13,493 --> 00:02:17,783
So I don't watch YouTube videos on
personal finance, but I do read a mm-hmm.

40
00:02:18,023 --> 00:02:19,733
Forum on personal finance.

41
00:02:19,733 --> 00:02:19,853
So.

42
00:02:20,443 --> 00:02:25,723
See again, again, I, you just, you're
just a random foun of knowledge of

43
00:02:26,383 --> 00:02:33,763
random topics and once again, uh, your
knowledge came in, uh, came in handy.

44
00:02:34,093 --> 00:02:35,893
So glad I can help Curtis.

45
00:02:35,953 --> 00:02:36,373
That's what I'm here.

46
00:02:36,373 --> 00:02:37,213
It's always good.

47
00:02:37,213 --> 00:02:39,553
And by the way, welcome
back to the United States.

48
00:02:39,553 --> 00:02:40,753
Having left it for.

49
00:02:41,123 --> 00:02:42,353
A brief period of time.

50
00:02:42,443 --> 00:02:44,603
It is good to be back, I have to say.

51
00:02:44,603 --> 00:02:47,903
So I did do a long flight to
India for a very short trip

52
00:02:47,903 --> 00:02:49,463
and made a long flight back.

53
00:02:50,123 --> 00:02:50,993
What's a long flight?

54
00:02:51,353 --> 00:02:56,273
Uh, so I think flying time was
about 26 hours, but door to door was

55
00:02:56,273 --> 00:02:58,883
probably closer to like 34, 35 hours.

56
00:02:59,303 --> 00:02:59,903
Wow.

57
00:03:00,413 --> 00:03:00,563
Yeah.

58
00:03:00,563 --> 00:03:04,103
That's just, I, I've done,
I've, I've flown to India,

59
00:03:04,163 --> 00:03:05,753
but I don't, I don't think.

60
00:03:06,748 --> 00:03:07,498
I don't remember.

61
00:03:07,558 --> 00:03:09,178
I I just remember it was really long.

62
00:03:09,303 --> 00:03:09,463
Yeah,

63
00:03:09,813 --> 00:03:10,183
it's long.

64
00:03:10,183 --> 00:03:10,663
And it was

65
00:03:10,663 --> 00:03:11,743
a, it was a, yeah, long,

66
00:03:11,758 --> 00:03:14,038
and especially now with
the pandemic, right?

67
00:03:14,038 --> 00:03:16,768
They require masks on the plane
the entire time minus Right.

68
00:03:16,768 --> 00:03:18,118
When you're eating or drinking.

69
00:03:18,118 --> 00:03:18,208
Right.

70
00:03:18,478 --> 00:03:21,628
So literally, and even when you're eating
and drinking, they're like, oh, take

71
00:03:21,628 --> 00:03:23,638
your mask off in between bites and sips.

72
00:03:23,638 --> 00:03:24,448
Put your mask back on.

73
00:03:24,538 --> 00:03:27,583
Oh, they're very, they're very,
um, what, what's cautious?

74
00:03:27,718 --> 00:03:28,468
I dunno what the word is.

75
00:03:28,468 --> 00:03:29,308
Yeah, yeah,

76
00:03:29,308 --> 00:03:29,608
yeah, yeah.

77
00:03:29,608 --> 00:03:31,738
And so it was a little
bit of a hassle like.

78
00:03:31,758 --> 00:03:34,638
You take a bite, it's like, instead
of taking a bite and chewing, I

79
00:03:34,638 --> 00:03:37,458
would literally take three bites
and then put the mask on and

80
00:03:37,458 --> 00:03:38,538
then sit there and chew, right.

81
00:03:38,868 --> 00:03:41,088
And then swallow and then
be like, okay, next time.

82
00:03:41,418 --> 00:03:42,783
And I, and you took, took

83
00:03:42,783 --> 00:03:45,588
a covid test on the way out,
another Covid test on the way in,

84
00:03:45,648 --> 00:03:48,048
and then another Covid
test after we got back, so,

85
00:03:48,138 --> 00:03:48,469
right, right.

86
00:03:48,708 --> 00:03:48,948
All good to

87
00:03:48,948 --> 00:03:49,218
go.

88
00:03:49,488 --> 00:03:52,698
So everything was fine, but it
is a little bit of a hassle, but.

89
00:03:53,023 --> 00:03:56,953
Yeah, at least travel is returning
back to some form of normal, I guess.

90
00:03:56,983 --> 00:03:57,373
Yeah.

91
00:03:57,373 --> 00:03:59,503
And you, and you have your wife back?

92
00:03:59,533 --> 00:03:59,773
Yep.

93
00:03:59,863 --> 00:04:02,713
My wife came back with me as well,
so it's good to be open back.

94
00:04:02,718 --> 00:04:04,003
She's been, she's been gone for a while.

95
00:04:05,083 --> 00:04:06,163
Visiting family and stuff.

96
00:04:06,223 --> 00:04:07,153
Yeah, visiting family.

97
00:04:07,153 --> 00:04:11,053
And we were in India for Di Wally,
which if anyone, that's my first time

98
00:04:11,053 --> 00:04:12,703
I've ever been in India for di Wally.

99
00:04:12,703 --> 00:04:18,373
And I have to say, it is crazy with
the amount of fireworks going on, like

100
00:04:18,403 --> 00:04:19,843
it sounds more than July 4th here.

101
00:04:20,163 --> 00:04:23,523
Yes, more than July 4th because
it happened over seven days.

102
00:04:23,973 --> 00:04:29,343
And for the seven days it would be from
like 4:00 PM till 11:00 PM And it's

103
00:04:29,343 --> 00:04:32,973
not just like the little sparklers that
you might do here, or even just like

104
00:04:32,973 --> 00:04:38,553
the little rockets, they sounded like
full on like Roman candles and like.

105
00:04:38,553 --> 00:04:39,603
Loud gunshots.

106
00:04:39,603 --> 00:04:42,813
I think you and I were on a call
a couple times and you heard a

107
00:04:42,813 --> 00:04:43,713
little noise in the background.

108
00:04:43,713 --> 00:04:44,823
You're like, what is that?

109
00:04:44,883 --> 00:04:45,003
I was

110
00:04:45,003 --> 00:04:45,693
like, what?

111
00:04:45,698 --> 00:04:46,558
What is happening already?

112
00:04:46,558 --> 00:04:46,953
And the sky's

113
00:04:46,953 --> 00:04:50,553
like, I, I was reading an article, I think
at NPR where they were saying that they

114
00:04:50,553 --> 00:04:55,233
had like a picture before and a picture
after of like the sky after fireworks.

115
00:04:55,233 --> 00:04:58,503
And it's just like clear to
completely covered in smoke.

116
00:04:59,833 --> 00:05:02,143
That's crazy, but it's good to be back.

117
00:05:02,923 --> 00:05:03,343
Yeah.

118
00:05:03,343 --> 00:05:03,978
Good to have you back.

119
00:05:03,978 --> 00:05:04,003
You missed.

120
00:05:05,263 --> 00:05:07,483
Aw, well it's nice to be on the same.

121
00:05:07,483 --> 00:05:08,023
Likewise, you know?

122
00:05:08,623 --> 00:05:08,803
Yeah.

123
00:05:08,803 --> 00:05:10,393
It is nice to be in the same time zone.

124
00:05:10,813 --> 00:05:14,543
Uh, by the way, I should mention
our standard disclaimer Prasanna.

125
00:05:14,563 --> 00:05:15,763
And I work for different companies.

126
00:05:15,763 --> 00:05:16,633
He works for Zoom.

127
00:05:16,633 --> 00:05:17,413
I work for Druva.

128
00:05:17,503 --> 00:05:18,493
This is not a.

129
00:05:18,828 --> 00:05:22,278
Podcast of either company, the
opinions that you hear are ours.

130
00:05:22,638 --> 00:05:27,678
And also, uh, please rate this
podcast@ratethispodcast.com slash

131
00:05:27,678 --> 00:05:31,848
restore and, uh, or your favorite
podcaster if, if it's not listed there.

132
00:05:32,358 --> 00:05:37,219
And, uh, finally, if you are
interested in the topics that we talk

133
00:05:37,224 --> 00:05:38,433
about, come, come, come, come, come.

134
00:05:39,023 --> 00:05:39,313
Yeah.

135
00:05:39,383 --> 00:05:43,818
Just like Paul, just, uh,
contact me at w Curtis Preston.

136
00:05:44,563 --> 00:05:47,503
gmail.com or at WC preston on Twitter.

137
00:05:47,863 --> 00:05:49,633
And uh, we will have you on.

138
00:05:49,633 --> 00:05:51,373
And it's a friendly
environment, right, Paul?

139
00:05:51,943 --> 00:05:53,833
Absolutely, absolutely.

140
00:05:54,133 --> 00:05:55,123
Everybody should come.

141
00:05:55,513 --> 00:05:59,113
And, uh, we love to talk about
all things, uh, backup security,

142
00:05:59,113 --> 00:06:04,003
data protection, data resilience,
uh, you know, puppies, whatever.

143
00:06:04,573 --> 00:06:06,523
Um, I like puppies and movies.

144
00:06:06,733 --> 00:06:08,533
I also wanna mention our giveaway.

145
00:06:08,593 --> 00:06:11,083
We are giving away one free.

146
00:06:11,743 --> 00:06:15,883
ebook version of my new book, modern
Data Protection, published in May

147
00:06:16,393 --> 00:06:18,793
courtesy of O'Reilly and Associates.

148
00:06:18,823 --> 00:06:24,763
All you have to do to qualify for
the drawing is uh, to subscribe to

149
00:06:24,763 --> 00:06:26,983
our newsletter on backup central.com.

150
00:06:27,313 --> 00:06:30,133
Just it's right there in
the top menu, subscribe.

151
00:06:30,163 --> 00:06:34,363
And in the following week, I
will select one new listener.

152
00:06:34,528 --> 00:06:37,168
To receive a free ebook copy.

153
00:06:37,528 --> 00:06:42,268
So our, I've selected the winner from
this week, and your name is John Doherty.

154
00:06:42,268 --> 00:06:43,258
Congratulations.

155
00:06:43,438 --> 00:06:46,768
You'll get an email from me and another
one from O'Reilly with your ebook.

156
00:06:47,278 --> 00:06:47,788
So.

157
00:06:48,253 --> 00:06:49,273
Back to the podcast.

158
00:06:50,023 --> 00:06:56,053
So our, our next guest is
from a part of America that is

159
00:06:56,053 --> 00:06:57,793
connected but not connected.

160
00:06:59,293 --> 00:07:04,183
He has been, uh, in it for quite
some time, just, just about as

161
00:07:04,183 --> 00:07:07,873
long as I have, uh, short of, just
short of 30 years it looks like.

162
00:07:08,218 --> 00:07:11,743
And he's actually, this is actually,
this is the second time we've had this.

163
00:07:11,743 --> 00:07:14,953
He's had one job that entire time.

164
00:07:15,458 --> 00:07:23,528
He has been at the, uh, he is the IT
supervisor at the Kodiak Island Borough.

165
00:07:23,528 --> 00:07:25,718
That would be Kodiak, Alaska.

166
00:07:26,078 --> 00:07:29,048
He's two hours, or no,
he's one hour behind us.

167
00:07:29,438 --> 00:07:32,438
Welcome to the podcast Paul Van Dyke.

168
00:07:33,038 --> 00:07:33,788
Thank you very much.

169
00:07:33,788 --> 00:07:34,748
It's a pleasure to be here.

170
00:07:35,858 --> 00:07:38,558
So, uh, how, how did we find you, Paul?

171
00:07:39,718 --> 00:07:45,088
I have followed you on Twitter for,
um, a number of, uh, for quite some

172
00:07:45,088 --> 00:07:51,028
time, and, uh, enjoy listening to
your podcast, uh, when I, when I have

173
00:07:51,028 --> 00:07:54,088
some free time and space and, uh,

174
00:07:55,228 --> 00:07:57,508
when it gets dark and
cold in your indoors.

175
00:07:57,748 --> 00:07:58,498
Something like that.

176
00:07:58,738 --> 00:07:59,638
That's, that's right.

177
00:07:59,638 --> 00:08:00,023
Listen to that.

178
00:08:00,083 --> 00:08:01,618
Listen to us by the fire.

179
00:08:02,188 --> 00:08:02,398
Yeah.

180
00:08:02,398 --> 00:08:02,938
So outside Spliting,

181
00:08:03,388 --> 00:08:07,798
firewood or you know, any, any of these,
any of these Alaskan activities that I do.

182
00:08:08,698 --> 00:08:08,938
Yeah.

183
00:08:08,943 --> 00:08:11,068
And so then you, you
reached out to us, right.

184
00:08:11,068 --> 00:08:13,558
Saying, Hey, you know, 'cause
we, because we say this, right?

185
00:08:13,558 --> 00:08:18,598
We say, Hey, if you have, if you want
to talk about our favorite topics,

186
00:08:18,598 --> 00:08:21,208
then come on and we will bring you on.

187
00:08:21,718 --> 00:08:24,808
So we, we are so happy
to have like an actual.

188
00:08:25,933 --> 00:08:33,103
It practitioner, um, you know, does
this I I was about to say in the wild.

189
00:08:33,223 --> 00:08:33,403
I did.

190
00:08:33,403 --> 00:08:34,963
But that, that, that's true.

191
00:08:35,533 --> 00:08:40,273
That leads a whole other, uh, you know,
you know that, that, that brings up a

192
00:08:40,273 --> 00:08:42,133
whole other connotation where you live.

193
00:08:42,343 --> 00:08:45,463
What, can you describe what,
what it's like where you live?

194
00:08:45,493 --> 00:08:49,453
'cause you know, for those of us that
live down here, we have these visions

195
00:08:49,453 --> 00:08:50,893
of what it's like to live in Alaska.

196
00:08:50,893 --> 00:08:51,913
And I have no idea.

197
00:08:52,528 --> 00:08:52,858
You know,

198
00:08:52,888 --> 00:08:59,158
so I'll, uh, I'll, I'll not shatter the,
the, the stereotype of interior Alaska

199
00:08:59,158 --> 00:09:02,398
where it is dark and cold all winter long.

200
00:09:02,668 --> 00:09:05,578
We live on an island
in the Gulf of Alaska.

201
00:09:06,028 --> 00:09:11,188
It is very similar to the Pacific
Northwest, although right now

202
00:09:11,188 --> 00:09:13,288
we're in the, in the mid twenties.

203
00:09:13,788 --> 00:09:14,628
It's snow on the ground.

204
00:09:14,988 --> 00:09:19,578
We rarely, uh, we rarely see
single digits Fahrenheit.

205
00:09:20,058 --> 00:09:24,378
Um, it, we, we might see them
for a week throughout the winter.

206
00:09:24,848 --> 00:09:25,138
Okay.

207
00:09:25,138 --> 00:09:27,043
Um, mostly we're in the twenties.

208
00:09:27,373 --> 00:09:33,343
We get above freezing and it will, it
will warm and, and thaw, thaw and freeze

209
00:09:33,583 --> 00:09:35,653
throughout the winter on, on occasion.

210
00:09:35,923 --> 00:09:39,343
So we don't, we don't go into the
deep freeze like interior Alaska

211
00:09:39,493 --> 00:09:43,573
or, and what about the, what about
the, the, the, the sunlight aspect?

212
00:09:44,203 --> 00:09:45,583
We are affected by that.

213
00:09:45,793 --> 00:09:49,873
Um, in the, around the solstice we.

214
00:09:50,273 --> 00:09:55,373
Usually get dark, probably three 30 or
four in the afternoon, and it doesn't get

215
00:09:55,373 --> 00:09:58,643
light till 9, 9 30, uh, in the morning.

216
00:09:59,663 --> 00:10:05,033
But you don't have this period where
you're, where you're dark 24 by seven,

217
00:10:05,753 --> 00:10:06,323
correct?

218
00:10:06,383 --> 00:10:06,833
Correct.

219
00:10:07,403 --> 00:10:09,173
That is, uh, that is Northern Alaska.

220
00:10:11,018 --> 00:10:11,308
Okay.

221
00:10:11,603 --> 00:10:16,673
Because I, um, where was, well, well
that's actually where was Northern

222
00:10:16,673 --> 00:10:18,713
Exposure set the, that TV show?

223
00:10:18,713 --> 00:10:19,763
You remember that TV show?

224
00:10:20,443 --> 00:10:24,223
I do remember that that was, that was
more interior Alaska, I believe they

225
00:10:24,223 --> 00:10:27,763
were trying to shoot around the, around
the Fairbanks area or try to Gotcha.

226
00:10:28,033 --> 00:10:31,753
Because I, I do remember that that
was an episode where they had, you

227
00:10:31,753 --> 00:10:34,723
know, there's a period where they get
nothing buts on, and then there's a

228
00:10:34,723 --> 00:10:36,073
period where they get nothing but night.

229
00:10:36,823 --> 00:10:37,393
Um,

230
00:10:37,663 --> 00:10:42,943
so I was plot line, so I, I know
this is probably going to be my, um.

231
00:10:43,948 --> 00:10:48,928
Inexperienced or talking to people
from Alaska, but like how do you

232
00:10:48,928 --> 00:10:50,728
get supplies and stuff like that?

233
00:10:50,728 --> 00:10:52,288
Like you said, you live on an island.

234
00:10:53,128 --> 00:10:54,418
You live in Alaska, right?

235
00:10:54,988 --> 00:10:55,078
Yes.

236
00:10:56,008 --> 00:11:00,418
So everything is either, uh, is
either barged in or, or flown in.

237
00:11:00,898 --> 00:11:06,388
And, uh, it's a 45 minute flight
to Anchorage, the, the largest

238
00:11:06,388 --> 00:11:11,968
city in Alaska out of Kodiak or,
uh, or it's about a 10 to 13 hour

239
00:11:12,628 --> 00:11:14,668
ferry ride from Kodiak to mainland.

240
00:11:16,753 --> 00:11:22,468
I, I, that's, I did not expect
that part 10 to 13 hour ferry ride.

241
00:11:23,008 --> 00:11:23,518
Correct.

242
00:11:23,728 --> 00:11:27,208
So how far are you from the mainland?

243
00:11:27,898 --> 00:11:31,453
Um, it's 250 miles from
Kodiak to Anchorage.

244
00:11:31,453 --> 00:11:31,454
Okay.

245
00:11:32,113 --> 00:11:32,293
Wow.

246
00:11:32,293 --> 00:11:37,573
And then, uh, and then I'm not sure
what the, uh, what the gap is between

247
00:11:37,873 --> 00:11:42,343
Kodiak and the mainland as far as the,
uh, the nautical miles that the ferry

248
00:11:42,343 --> 00:11:42,523
takes.

249
00:11:42,523 --> 00:11:42,703
Wow.

250
00:11:42,703 --> 00:11:43,423
That's far.

251
00:11:44,083 --> 00:11:45,073
I did not expect that.

252
00:11:46,213 --> 00:11:52,513
And what, what is the island like from
a, you know, what does it look like?

253
00:11:52,513 --> 00:11:55,033
Is it, well, I'll just stop there.

254
00:11:56,083 --> 00:11:58,033
It, you know, it looks a lot like Ireland.

255
00:11:58,553 --> 00:11:59,253
Um, okay.

256
00:11:59,258 --> 00:12:00,298
We do have forest.

257
00:12:00,508 --> 00:12:02,638
We do have, uh, we do have mountains.

258
00:12:03,148 --> 00:12:05,758
Uh, we have one glacier on the island.

259
00:12:06,298 --> 00:12:14,908
Uh, we do have a lot of, uh, a lot of
shrubbery, um, alder flat land down

260
00:12:14,908 --> 00:12:16,078
on the southern end of the island.

261
00:12:16,288 --> 00:12:19,768
We're this second largest island in
the US right after the Hawaiian island.

262
00:12:21,043 --> 00:12:21,263
Wow.

263
00:12:21,268 --> 00:12:21,898
Wow.

264
00:12:22,258 --> 00:12:23,098
How big is it?

265
00:12:25,348 --> 00:12:26,008
I don't know.

266
00:12:29,308 --> 00:12:30,268
I just live here.

267
00:12:31,288 --> 00:12:35,638
Here's what I know, and that is
that the state of Alaska is so much

268
00:12:35,638 --> 00:12:37,228
bigger than most people think it is.

269
00:12:37,228 --> 00:12:37,318
Yes.

270
00:12:37,918 --> 00:12:42,988
Because of the, and I don't know, perhaps
one of, you know the, the way that maps

271
00:12:42,988 --> 00:12:45,118
are done, there's a word for, yeah.

272
00:12:45,928 --> 00:12:46,348
Projection.

273
00:12:46,348 --> 00:12:47,218
Do you know what the word is?

274
00:12:47,728 --> 00:12:48,358
The projection.

275
00:12:49,813 --> 00:12:53,833
Well, it had, there's a, there's a,
it's like a person's name, I think.

276
00:12:53,838 --> 00:12:53,898
Yeah,

277
00:12:53,903 --> 00:12:54,013
yeah.

278
00:12:54,013 --> 00:12:55,003
It's a person's name.

279
00:12:55,093 --> 00:12:55,513
mcc,

280
00:12:55,513 --> 00:12:55,903
yeah.

281
00:12:56,713 --> 00:12:57,463
Is that what it is?

282
00:12:57,463 --> 00:13:04,153
It, it just has to do with, there, there
is a way of spreading out a global map

283
00:13:04,153 --> 00:13:07,363
onto a flat surface and it's that style.

284
00:13:07,633 --> 00:13:11,953
And when you do it that way,
Alaska looks a lot smaller

285
00:13:12,013 --> 00:13:14,653
than it, than it actually is.

286
00:13:14,683 --> 00:13:16,123
'cause Alaska is actually bigger.

287
00:13:16,693 --> 00:13:20,173
Than Texas is my understanding,
or similar size to

288
00:13:20,623 --> 00:13:21,043
Texas.

289
00:13:21,433 --> 00:13:26,413
I went to college in Texas and before
I went, my dad bought me a hat that

290
00:13:26,413 --> 00:13:30,133
said if you cut Alaska in half, Texas
would be the third largest state.

291
00:13:31,918 --> 00:13:34,108
I didn't wear that hat on campus.

292
00:13:37,438 --> 00:13:38,668
Is it really that big?

293
00:13:39,178 --> 00:13:39,598
It is.

294
00:13:39,598 --> 00:13:40,948
Is that Wow.

295
00:13:41,398 --> 00:13:41,608
Yeah.

296
00:13:41,608 --> 00:13:42,598
See, I, I knew it was big.

297
00:13:42,598 --> 00:13:43,623
I didn't realize it was that big.

298
00:13:43,653 --> 00:13:43,983
That big,

299
00:13:44,123 --> 00:13:44,343
big.

300
00:13:44,348 --> 00:13:44,398
Yeah.

301
00:13:45,208 --> 00:13:45,628
Yeah.

302
00:13:45,778 --> 00:13:46,258
Yeah.

303
00:13:46,498 --> 00:13:49,168
Um, and then, and then,
because Alaska appears

304
00:13:49,168 --> 00:13:52,258
in the corner of most continental
United States Maps, yeah.

305
00:13:52,948 --> 00:13:53,248
Hawaii.

306
00:13:54,028 --> 00:13:54,748
It's far away.

307
00:13:54,748 --> 00:13:54,958
Yeah.

308
00:13:54,958 --> 00:13:55,498
Yeah, yeah.

309
00:13:56,338 --> 00:13:56,638
Yeah.

310
00:13:59,578 --> 00:14:02,308
Uh, I'm Googling Kodiak
Island size, by the way.

311
00:14:02,308 --> 00:14:03,988
12,000 square miles.

312
00:14:04,438 --> 00:14:04,948
How many people?

313
00:14:04,948 --> 00:14:05,283
There you go.

314
00:14:06,478 --> 00:14:06,698
Uh,

315
00:14:06,703 --> 00:14:10,693
uh, we are, uh, a little
over 13,000 people right now.

316
00:14:11,383 --> 00:14:14,623
So, so it's so it's a pretty
rural, rural, I can't, that

317
00:14:14,623 --> 00:14:15,643
is a word I have trouble with.

318
00:14:15,643 --> 00:14:16,003
Rural.

319
00:14:16,033 --> 00:14:19,243
Rural, uh, world.

320
00:14:19,303 --> 00:14:19,693
Okay.

321
00:14:20,293 --> 00:14:21,058
And so what do, but we do

322
00:14:21,063 --> 00:14:24,043
have, we do have the world's
largest Coast Guard base.

323
00:14:24,583 --> 00:14:26,413
We have a rocket launch facility.

324
00:14:26,743 --> 00:14:29,113
We have a, a thriving fishing industry.

325
00:14:29,533 --> 00:14:30,733
And, um.

326
00:14:31,768 --> 00:14:36,328
And, you know, we have state government
represented local government, um, and

327
00:14:36,328 --> 00:14:38,698
other, other service industries in Kodiak.

328
00:14:39,208 --> 00:14:44,038
Well, if you have rockets and you have
fish, I mean, that's really all you need.

329
00:14:45,508 --> 00:14:47,999
Curtis like sold, so That's right.

330
00:14:48,718 --> 00:14:54,508
So t tell me, so you work for the
bureau, uh, the island bureau.

331
00:14:54,898 --> 00:14:55,318
What?

332
00:14:55,963 --> 00:15:00,043
Tell me what the IT environment is
like and what, you know, what, what

333
00:15:00,043 --> 00:15:01,273
do you, what do you need it for?

334
00:15:01,273 --> 00:15:03,703
What do you, you know, what
does it look like, et cetera.

335
00:15:04,213 --> 00:15:08,863
We, we have a very small it off,
uh, it shop, the, the, the borough

336
00:15:08,863 --> 00:15:14,473
is, um, paramount or, or, uh, akin
to a county in most locations.

337
00:15:14,488 --> 00:15:14,778
Okay.

338
00:15:15,043 --> 00:15:16,303
And so, um.

339
00:15:16,888 --> 00:15:19,048
We have the functions of accounting.

340
00:15:19,348 --> 00:15:22,888
We have an assessing department, a
finance department, a clerk's office,

341
00:15:23,218 --> 00:15:26,908
an engineering facilities department,
and a community development department.

342
00:15:27,448 --> 00:15:33,988
So, um, our number one, our number
one, and it's, it's sad, uh, to, to.

343
00:15:34,073 --> 00:15:39,293
To put in this context, but our number
one goal is to assess properties

344
00:15:39,323 --> 00:15:44,933
and tax them and collect money
to pay for our school district.

345
00:15:45,653 --> 00:15:47,363
Hey, that's, that's important.

346
00:15:48,538 --> 00:15:51,148
Is the, uh, is our school
district is the number one expense

347
00:15:51,718 --> 00:15:52,498
tax revenue,

348
00:15:52,618 --> 00:15:52,918
correct?

349
00:15:53,638 --> 00:15:53,698
Yeah.

350
00:15:53,698 --> 00:15:57,868
And so then with that we have, uh,
we have other functions to support

351
00:15:57,868 --> 00:15:59,188
development of the community.

352
00:15:59,548 --> 00:16:02,818
And obviously the finance department
ensures that everything is accounted for.

353
00:16:03,388 --> 00:16:07,888
Um, and then, so as an IT department, we
support all these, all these functions.

354
00:16:08,248 --> 00:16:11,998
We, we do the, uh, the gambit from A to Z.

355
00:16:12,538 --> 00:16:13,048
Um.

356
00:16:13,813 --> 00:16:17,563
We have a virtual infrastructure onsite.

357
00:16:18,043 --> 00:16:21,643
We do a lot of things that are
line of business applications to

358
00:16:21,643 --> 00:16:24,973
support the borough that are onsite.

359
00:16:24,973 --> 00:16:27,523
And so we have a, we have
an onsite data center.

360
00:16:27,823 --> 00:16:30,253
We do have some
functionality in the cloud.

361
00:16:30,703 --> 00:16:35,473
We do use some of the cloud services
that other people use, but a lot of

362
00:16:35,473 --> 00:16:40,093
these line of business applications
require internal infrastructure.

363
00:16:41,513 --> 00:16:42,988
So we have, we have it.

364
00:16:44,338 --> 00:16:44,698
Go ahead.

365
00:16:46,318 --> 00:16:51,358
So we, we have the, we have infrastructure
here to support, um, all those things.

366
00:16:51,358 --> 00:16:52,708
And it, and it needs to be backed up

367
00:16:53,008 --> 00:16:56,308
it seems, given how far you are from
the mainland and probably given how far

368
00:16:56,308 --> 00:16:58,738
you are from any public cloud region.

369
00:16:59,218 --> 00:16:59,548
Right.

370
00:16:59,548 --> 00:16:59,608
I'm

371
00:16:59,758 --> 00:17:00,088
sure.

372
00:17:00,093 --> 00:17:02,848
Where, where would be the closest
public cloud region, do you know?

373
00:17:03,808 --> 00:17:04,648
Or do you know?

374
00:17:04,678 --> 00:17:05,098
Um.

375
00:17:05,298 --> 00:17:10,668
I believe that there are some cloud, some,
uh, public cloud providers in Alaska.

376
00:17:11,028 --> 00:17:11,058
Okay.

377
00:17:11,058 --> 00:17:14,328
I've heard that, uh, some of the
telecommunication companies have either

378
00:17:14,328 --> 00:17:19,938
partnered with Azure or AWS and have
some, some functionality hosted locally.

379
00:17:19,938 --> 00:17:19,998
Yeah.

380
00:17:19,998 --> 00:17:23,028
My guess is Equinix has some
data centers out in Alaska.

381
00:17:23,448 --> 00:17:23,568
Okay.

382
00:17:23,568 --> 00:17:25,218
That some of the public
clouds we're probably using.

383
00:17:26,158 --> 00:17:30,058
One of the challenges that we
have is Alaska is also known as

384
00:17:30,058 --> 00:17:35,758
the ring of Fire, and so we were
very seismically active in 2007.

385
00:17:36,118 --> 00:17:40,858
We, we got fiber optic communications
to the island, but before that

386
00:17:40,858 --> 00:17:42,238
we had satellite internet.

387
00:17:42,238 --> 00:17:47,098
So, um, a lot of my early IT career,
we were under satellite internet,

388
00:17:47,098 --> 00:17:49,983
and so we didn't feel comfortable.

389
00:17:51,193 --> 00:17:54,493
Uh, outsourcing or, or,
or, or cloud services.

390
00:17:54,493 --> 00:18:00,493
Were very new at that time anyway, but
we felt like we were an island and we had

391
00:18:00,493 --> 00:18:02,893
to have all of our resources on island.

392
00:18:03,523 --> 00:18:06,733
Now, with better infrastructure
and more reliable infrastructure,

393
00:18:07,213 --> 00:18:07,908
uh, we're able to.

394
00:18:09,013 --> 00:18:10,063
To look at outsourcing it.

395
00:18:10,243 --> 00:18:14,803
So when you had all your infrastructure
back in 2007 on the island, how did

396
00:18:14,803 --> 00:18:16,753
you deal with like disaster recovery?

397
00:18:16,753 --> 00:18:20,388
Was there, like, is there like
another island like nearby that you,

398
00:18:22,513 --> 00:18:23,923
that that's a, that's a good question.

399
00:18:23,953 --> 00:18:27,463
Um, fortunately we never
had to do disaster recovery.

400
00:18:27,973 --> 00:18:32,863
Uh, although I, I, I, I, I
do have a story from 2001.

401
00:18:33,373 --> 00:18:35,743
Um, this was almost a disaster,

402
00:18:36,313 --> 00:18:37,723
but, uh, yeah, we'll
get, we'll get to that.

403
00:18:37,813 --> 00:18:38,083
Yeah.

404
00:18:38,938 --> 00:18:46,408
But, uh, but really we were, uh, you, we
cataloged what we had and, and, uh, we've,

405
00:18:46,918 --> 00:18:52,228
we, we just, we haven't had to do it, but
we had a, uh, we had a, a plan in place

406
00:18:52,228 --> 00:18:54,118
that we would just acquire more hardware.

407
00:18:54,388 --> 00:18:59,128
Um, I also, as, uh, as part of
my job, I am also part of our

408
00:18:59,128 --> 00:19:04,018
emergency, um, operations center,
and I'm the logistics section chief.

409
00:19:04,888 --> 00:19:06,268
With our emergency operations center.

410
00:19:06,268 --> 00:19:10,318
So, um, you know, bringing in
supplies to our community Mm-hmm.

411
00:19:10,318 --> 00:19:12,778
Would be something that I
would be responsible for doing.

412
00:19:13,198 --> 00:19:16,198
Is that, is that a volunteer
position or that's part of your

413
00:19:16,198 --> 00:19:17,458
job working for the borough?

414
00:19:18,148 --> 00:19:20,338
Uh, it is part of my job
working for the borough.

415
00:19:20,338 --> 00:19:20,368
Okay.

416
00:19:20,608 --> 00:19:26,188
The borough and the city are jointly
responsible for emergency response.

417
00:19:26,578 --> 00:19:27,688
So we have, uh.

418
00:19:28,318 --> 00:19:34,078
A city government here in Kodiak as well,
and they have, uh, police department, fire

419
00:19:34,078 --> 00:19:36,298
department, and other, other resources.

420
00:19:37,438 --> 00:19:39,118
How many boroughs are on the island?

421
00:19:40,198 --> 00:19:41,458
There's just one borough.

422
00:19:41,463 --> 00:19:41,543
Okay.

423
00:19:41,548 --> 00:19:45,568
And the borough covers Kodiak
Island and a portion of the

424
00:19:45,568 --> 00:19:47,908
mainland across the Sheaf Strait.

425
00:19:48,913 --> 00:19:49,423
Oh, weird.

426
00:19:49,423 --> 00:19:49,843
Interesting.

427
00:19:51,523 --> 00:19:57,793
It's, um, it's a, it's a function of
the watershed that is on the mainland

428
00:19:57,943 --> 00:20:03,523
that drains into the Sheko Strait
for, I believe, as it was explained

429
00:20:03,523 --> 00:20:05,893
to me, for fisheries resources.

430
00:20:06,343 --> 00:20:10,813
So rivers and streams on that
side are part of our borough,

431
00:20:11,053 --> 00:20:12,463
uh, for That makes sense.

432
00:20:13,123 --> 00:20:13,843
Fish habitat.

433
00:20:14,593 --> 00:20:14,953
Right.

434
00:20:14,953 --> 00:20:17,563
As a, as a fishing community,
it, it, it's important.

435
00:20:17,563 --> 00:20:19,273
It, uh, we have a vested interest in it.

436
00:20:19,273 --> 00:20:19,333
Yeah.

437
00:20:20,953 --> 00:20:23,533
Um, and yeah.

438
00:20:23,533 --> 00:20:29,953
By the way, I did check there is
an, um, a us Alaska region in AWS

439
00:20:32,048 --> 00:20:37,153
So I have us, I have central Eastern,
east Indiana, Pacific, and Alaska.

440
00:20:37,483 --> 00:20:41,113
So Alaska is its own, uh, AWS um.

441
00:20:42,823 --> 00:20:45,193
According to a website
that I just looked at.

442
00:20:45,403 --> 00:20:47,083
That's literally the
extent of my research.

443
00:20:47,083 --> 00:20:49,513
But Prasanna, do you know any different?

444
00:20:49,693 --> 00:20:50,323
I do not.

445
00:20:50,983 --> 00:20:51,223
Okay.

446
00:20:51,463 --> 00:20:51,733
Alright.

447
00:20:52,153 --> 00:20:55,358
Yeah, it, it looks like
it's its own region, so, um.

448
00:20:57,178 --> 00:21:01,588
So, and, and so then the other
thing I would have is in preparation

449
00:21:01,588 --> 00:21:06,568
for any kind of disaster, which I
think, so what kind of disasters do

450
00:21:06,568 --> 00:21:07,948
you need to prepare for up there?

451
00:21:07,948 --> 00:21:11,188
Obviously fire, like a giant
fire would be a problem.

452
00:21:11,998 --> 00:21:17,218
Do you have, you know, you, you
don't have tornadoes or hurricanes

453
00:21:17,218 --> 00:21:18,088
or that sort of thing up way.

454
00:21:18,088 --> 00:21:18,178
Right?

455
00:21:18,178 --> 00:21:19,123
Probably have tsunamis.

456
00:21:20,038 --> 00:21:20,608
Tsunamis.

457
00:21:20,638 --> 00:21:20,788
We,

458
00:21:21,208 --> 00:21:22,618
tsunamis are, are one thing.

459
00:21:22,708 --> 00:21:24,148
We do have wind events.

460
00:21:24,208 --> 00:21:28,558
Um, no we don't have hurricanes 'cause
they refuse to call, uh, the windy

461
00:21:28,558 --> 00:21:32,218
day last week a hurricane, even though
it was blowing 70 miles an hour.

462
00:21:33,328 --> 00:21:33,388
Hmm.

463
00:21:34,258 --> 00:21:35,758
I wonder if they had derechos.

464
00:21:35,788 --> 00:21:35,938
Yeah.

465
00:21:37,228 --> 00:21:39,298
Did you hear our episode about Derechos?

466
00:21:40,498 --> 00:21:41,273
I don't think I did.

467
00:21:41,878 --> 00:21:42,178
Yeah.

468
00:21:42,238 --> 00:21:44,368
Derecho is a land hurricane.

469
00:21:44,428 --> 00:21:47,458
It's a hurricane that
starts over land and we had.

470
00:21:48,433 --> 00:21:54,373
A guest on who was in the middle of a
derecho with, and the thing is, unlike

471
00:21:54,433 --> 00:22:00,253
a ocean hurricane, it just, it's more
like a tornado in that it just happens.

472
00:22:00,523 --> 00:22:04,303
So he just, he just was on his porch.

473
00:22:04,333 --> 00:22:06,133
I think he was out on his
yard or something, right?

474
00:22:06,133 --> 00:22:06,653
Prasanna?

475
00:22:06,673 --> 00:22:07,033
Yeah, he was out

476
00:22:07,063 --> 00:22:09,133
and then he grabbed the dog
and ran back in, I think.

477
00:22:09,763 --> 00:22:10,063
Yeah.

478
00:22:10,063 --> 00:22:10,603
Do, uh,

479
00:22:10,663 --> 00:22:14,983
per, do you have the, yeah, it's
episode number 1 26 Stop ransomware

480
00:22:14,983 --> 00:22:16,873
attacks in seconds with Greg Edwards.

481
00:22:17,023 --> 00:22:17,473
Right?

482
00:22:17,473 --> 00:22:17,713
Yeah.

483
00:22:17,713 --> 00:22:19,813
So you wouldn't know it
from the title, but Yeah.

484
00:22:19,813 --> 00:22:20,953
There, we talked to him.

485
00:22:20,953 --> 00:22:24,163
So it's called a derecho,
uh, which is weird.

486
00:22:24,163 --> 00:22:26,383
It's like the Spanish word for right.

487
00:22:26,473 --> 00:22:33,583
But it, it, um, it means, it, it's
a land hurricane, which is just, um.

488
00:22:34,303 --> 00:22:34,513
Yeah.

489
00:22:34,513 --> 00:22:38,533
So maybe, maybe that's what, maybe you
just need to get, you know, need to

490
00:22:38,533 --> 00:22:41,443
explain to these people, Hey, we didn't
have a hurricane, we had a derecho.

491
00:22:43,663 --> 00:22:46,993
Um, no, we, we have, we have low
pressures in the Gulf of Alaska,

492
00:22:46,993 --> 00:22:51,883
and that brings about, uh, um, crazy
winds, strong winds and, uh, wind.

493
00:22:52,633 --> 00:22:54,883
Um, we did have a, uh.

494
00:22:57,223 --> 00:22:58,033
In our data center.

495
00:22:58,093 --> 00:22:59,203
Was that in the middle of winter?

496
00:22:59,323 --> 00:23:06,553
Yes, it was actually the, the room next
to our data center is the, uh, is the me

497
00:23:06,883 --> 00:23:12,493
mechanical room for the building and there
was some louvers that were stuck open.

498
00:23:12,973 --> 00:23:21,373
And so a, uh, a coil froze and then
it thawed out and so that waterline

499
00:23:21,673 --> 00:23:24,403
broke and ended up flooding.

500
00:23:24,673 --> 00:23:25,093
Um.

501
00:23:25,468 --> 00:23:29,158
Flooding our data center to, to some
extent, we had a, a couple inches of water

502
00:23:29,158 --> 00:23:32,488
on the floor and, um, it, it drained.

503
00:23:32,548 --> 00:23:33,358
It drained through the.

504
00:23:34,498 --> 00:23:38,758
Into the basement, but, uh, it
was, uh, it was a little scary.

505
00:23:38,848 --> 00:23:41,668
That's also where our electrical
connections go through the floor.

506
00:23:41,788 --> 00:23:41,848
Oof.

507
00:23:42,058 --> 00:23:43,168
Into the basement as well.

508
00:23:43,168 --> 00:23:45,928
But it was, it essentially
a non-event, though?

509
00:23:46,528 --> 00:23:47,548
It was a non-event?

510
00:23:47,878 --> 00:23:51,388
Uh, we did have, we did have some
backup tapes sitting on the counter

511
00:23:51,718 --> 00:23:55,798
and we asked the, the maintenance
guy who was wearing rubber boots.

512
00:23:56,033 --> 00:23:59,603
To walk into the water and
to grab those backup tapes.

513
00:24:02,813 --> 00:24:03,713
Very important.

514
00:24:03,713 --> 00:24:04,613
Very important.

515
00:24:04,733 --> 00:24:05,993
Paul, do you had so.

516
00:24:06,823 --> 00:24:09,223
Do you use tapes mainly for your backups?

517
00:24:09,223 --> 00:24:10,663
Because I know you mentioned
that you had the maintenance

518
00:24:10,663 --> 00:24:12,253
guy go in, grab some tapes.

519
00:24:13,633 --> 00:24:19,843
I, um, we currently, we use, uh,
we use tapes, d uh, deduplication

520
00:24:19,903 --> 00:24:25,393
appliances, offsite storage, and
uh, and then local storage as well.

521
00:24:25,393 --> 00:24:25,993
So all the things.

522
00:24:27,918 --> 00:24:29,058
We wanna be secure.

523
00:24:29,058 --> 00:24:30,348
We want to be protected.

524
00:24:30,768 --> 00:24:35,298
And, and what do you do to get, you
know, give, especially given that

525
00:24:35,298 --> 00:24:42,468
you're an island, what do you do to,
you know, separate a copy of the backups

526
00:24:42,468 --> 00:24:43,938
from the thing that you're protecting?

527
00:24:45,348 --> 00:24:48,528
We have, we have a, uh, a safe.

528
00:24:49,378 --> 00:24:53,068
In our data center, we also
have a safe across the street

529
00:24:53,128 --> 00:24:55,048
in a, uh, in another building.

530
00:24:55,708 --> 00:25:01,858
So we take our, we take our tapes offsite,
which may be, uh, 200, 250 feet away.

531
00:25:02,068 --> 00:25:03,958
You walk em across the street, right?

532
00:25:04,018 --> 00:25:04,288
Yes.

533
00:25:05,038 --> 00:25:05,488
Yeah.

534
00:25:05,818 --> 00:25:06,928
And, and that's good.

535
00:25:06,928 --> 00:25:10,378
I mean, is there, is there any concern,
you know, have you had discussions

536
00:25:10,378 --> 00:25:14,368
of, you know, if there was something
like a flood or anything like that?

537
00:25:14,368 --> 00:25:17,788
Is there any concern that you know, that
you have things too close together there?

538
00:25:18,928 --> 00:25:24,538
Um, a small tactical nuke could,
uh, could take, take out my,

539
00:25:24,568 --> 00:25:26,608
my disaster recovery plans.

540
00:25:27,028 --> 00:25:27,358
Mm-Hmm.

541
00:25:27,898 --> 00:25:30,268
Um, I think that would
take out their problems.

542
00:25:30,358 --> 00:25:30,568
Yeah.

543
00:25:31,258 --> 00:25:32,038
The whole, the whole island.

544
00:25:32,038 --> 00:25:32,428
I think

545
00:25:33,148 --> 00:25:33,628
that's right.

546
00:25:33,628 --> 00:25:33,718
That

547
00:25:35,728 --> 00:25:42,328
really, um, you know, my backups are,
are for, for the use cases that I have.

548
00:25:42,748 --> 00:25:43,258
Um.

549
00:25:44,428 --> 00:25:49,468
If, if our data center were to die, um,
if the building were to collapse in an

550
00:25:49,468 --> 00:25:53,818
earthquake, you know, I would be looking
for additional hardware to restore onto.

551
00:25:54,328 --> 00:25:54,448
Mm-Hmm.

552
00:25:54,688 --> 00:26:01,948
Um, my backups are really, uh, primarily
used for accidentally deleted files.

553
00:26:02,398 --> 00:26:03,943
Ransomware, um.

554
00:26:05,083 --> 00:26:09,253
And, and any localized, you
know, localized disasters.

555
00:26:09,733 --> 00:26:15,223
Um, we have talked about doing cloud-based
backups, and we do have more bandwidth

556
00:26:15,223 --> 00:26:17,263
available to do cloud-based backups.

557
00:26:17,773 --> 00:26:20,713
But, uh, bandwidth is expensive,
as we talked about living on

558
00:26:20,713 --> 00:26:25,453
an island, and until recently
it's been rather restricted.

559
00:26:25,453 --> 00:26:25,543
Right?

560
00:26:25,903 --> 00:26:33,253
So to do cloud-based backups would also
mean looking at cloud-based recovery.

561
00:26:33,388 --> 00:26:34,498
And Right.

562
00:26:34,498 --> 00:26:34,708
That you

563
00:26:34,708 --> 00:26:37,378
wouldn't be able to bring it
back in case we have not made

564
00:26:37,378 --> 00:26:38,338
that decision yet.

565
00:26:38,818 --> 00:26:39,178
Right.

566
00:26:39,808 --> 00:26:42,928
Well, I guess I, I guess, and I, I
wasn't even necessarily going to there,

567
00:26:42,928 --> 00:26:47,068
although, you know, I do work for a
cloud company and obviously that's,

568
00:26:47,158 --> 00:26:48,778
that's our solution for everything.

569
00:26:49,168 --> 00:26:54,778
Having said that, uh, I was just thinking
about, I don't know, an occasional.

570
00:26:55,468 --> 00:27:00,808
Copy of tapes being FedEx to Anchorage
or something, you know, even if

571
00:27:00,808 --> 00:27:06,178
it's infrequent, just just to have
a copy that's a little farther away

572
00:27:06,178 --> 00:27:08,248
than a few hundred feet, or having,

573
00:27:08,428 --> 00:27:11,939
or having like a building on the mainland
that's still part of the borough.

574
00:27:12,603 --> 00:27:13,773
You just shipped the tape sheet.

575
00:27:14,853 --> 00:27:14,913
Yeah.

576
00:27:14,918 --> 00:27:18,483
And, and we have, we have talked about,
uh, talked about some of those solutions.

577
00:27:18,483 --> 00:27:21,873
We've also looked at, uh, you know,
moving tapes to Iron Mountain.

578
00:27:22,383 --> 00:27:26,973
Um, a lot, but a lot of it depends
on, as you asked earlier, what are

579
00:27:26,973 --> 00:27:30,273
the things that we, what are the,
what are the hazards and, and how

580
00:27:30,273 --> 00:27:31,953
are we restoring from those hazards?

581
00:27:32,283 --> 00:27:32,703
Right.

582
00:27:32,793 --> 00:27:33,063
Right.

583
00:27:34,173 --> 00:27:37,053
So let's talk about, um.

584
00:27:37,468 --> 00:27:39,808
You know, you, you, you gave
us a couple of stories in

585
00:27:39,808 --> 00:27:41,668
your email when you wrote me.

586
00:27:42,208 --> 00:27:45,238
Um, I, I really like
this first one that, that

587
00:27:47,548 --> 00:27:50,608
intentionally destroying
my complete environment.

588
00:27:50,908 --> 00:27:52,678
That is.

589
00:27:53,503 --> 00:27:57,853
I, one word comes to mind,
and not everybody can say this

590
00:27:57,853 --> 00:28:00,043
word, but the word is chutzpah.

591
00:28:02,533 --> 00:28:07,843
Uh, guts my friend, intensely
destroying your complete environment

592
00:28:07,843 --> 00:28:11,323
to test your backup tapes or
to test your backup system.

593
00:28:11,713 --> 00:28:14,648
Um, you really gotta tell us about that.

594
00:28:16,543 --> 00:28:21,883
So, so, well, first off, what,
what possessed you to, to do that?

595
00:28:22,098 --> 00:28:23,473
I ha I had a purpose.

596
00:28:23,538 --> 00:28:28,178
I, I, I honestly had a purpose and, uh,
and yeah, when you put it like that, it,

597
00:28:28,183 --> 00:28:33,913
it sounds like, um, sounds like I was
missing a few IQ points on that test.

598
00:28:34,843 --> 00:28:39,673
Insight is 2020 and, and, and to
survive it is, uh, is, is really

599
00:28:39,673 --> 00:28:41,563
where the, uh, where the beauty is.

600
00:28:41,863 --> 00:28:45,343
So this was, this was post, um.

601
00:28:45,643 --> 00:28:46,753
This was about 2001 Mm-Hmm.

602
00:28:47,563 --> 00:28:52,153
And we had, we had invested
in our infrastructure.

603
00:28:52,333 --> 00:28:59,233
We had moved away from, uh, you know,
PCs as servers and custom built things.

604
00:28:59,233 --> 00:29:02,743
And we were moving into more
industrial acquired servers.

605
00:29:02,743 --> 00:29:08,023
And we had purchased through, uh, through
two fiscal years, we had purchased

606
00:29:08,233 --> 00:29:11,833
five compact ml, three 70 servers.

607
00:29:13,048 --> 00:29:15,928
Because we purchased them
through two fiscal years.

608
00:29:16,228 --> 00:29:17,818
We had three.

609
00:29:18,988 --> 00:29:25,948
We, we, we, we had, we had some that
had 9.1 gigabyte drives and some

610
00:29:25,948 --> 00:29:32,043
that had 18.2 gigabyte drives, but
they were all configured to be about

611
00:29:32,363 --> 00:29:35,203
45 gigabytes of raid five storage.

612
00:29:36,538 --> 00:29:40,048
And so we were trying
to be 100% by the book.

613
00:29:40,318 --> 00:29:42,118
We installed these five servers.

614
00:29:42,868 --> 00:29:48,148
We had two domain controllers,
an email server, a file server,

615
00:29:48,358 --> 00:29:49,618
and an application server.

616
00:29:49,858 --> 00:29:53,908
Just completed my MCSE training,
and so we were doing this as a

617
00:29:53,938 --> 00:29:55,678
standard rollout as much as possible.

618
00:29:55,828 --> 00:30:00,688
After about a year, the usage
on DISC was very asymmetrical.

619
00:30:01,783 --> 00:30:04,243
Our domain controllers
didn't use very much space.

620
00:30:04,243 --> 00:30:06,433
Our email server didn't
use very much space.

621
00:30:06,433 --> 00:30:10,003
In the early two thousands, our
file server was rapidly growing

622
00:30:10,333 --> 00:30:13,243
and we were adding applications
to our application server.

623
00:30:13,573 --> 00:30:20,053
So the 45 gigabytes was filling
up in an asymmetrical fashion, and

624
00:30:20,053 --> 00:30:21,523
I started looking at the discs.

625
00:30:21,523 --> 00:30:27,613
We had 9.1 gig drives and
18.2 gig drives, and I said.

626
00:30:28,678 --> 00:30:35,728
Well, if I just move some of these disks
around and I take four of the 9.1 gig

627
00:30:35,728 --> 00:30:43,318
drives and build a rate array in the first
three servers, and then I move the 18.2

628
00:30:43,318 --> 00:30:50,758
gig drives to the last two servers, I will
have matched my storage with my workload.

629
00:30:52,138 --> 00:30:57,178
So I ran a full backup on Friday
night, and I came in Saturday morning.

630
00:30:58,678 --> 00:30:58,888
Sorry.

631
00:30:58,888 --> 00:31:02,248
So you weren't ju you
weren't just testing backups.

632
00:31:02,248 --> 00:31:06,328
You had a, a purpose, an alternate,
like you had an extra purpose besides

633
00:31:06,328 --> 00:31:07,948
just testing your, your backups.

634
00:31:08,103 --> 00:31:08,308
I, I

635
00:31:08,308 --> 00:31:15,148
was trying to match my, my disc
space in my servers to the usage of

636
00:31:15,148 --> 00:31:17,068
the, of the demand on these servers.

637
00:31:17,488 --> 00:31:17,938
Gotcha.

638
00:31:17,938 --> 00:31:21,088
I, I just want, I, I pulled up
the stats, by the way, on a.

639
00:31:21,463 --> 00:31:27,253
On a, uh, compact, uh, ML three 70
and, uh, that comes with a maximum

640
00:31:27,253 --> 00:31:32,773
of four gigabytes of Ram a Pentium
three one gigahertz processor.

641
00:31:33,253 --> 00:31:37,213
And, and here's the best part,
an integrated dual channel

642
00:31:37,273 --> 00:31:39,758
wide ultra two SC adapter.

643
00:31:40,628 --> 00:31:40,918
Nice.

644
00:31:41,588 --> 00:31:42,438
Back in the day.

645
00:31:44,773 --> 00:31:48,793
So, so basically you, that's what you
mean by basically by pulling drives

646
00:31:48,793 --> 00:31:56,593
apart, you destroyed any rate that was
going on and you required, uh, these

647
00:31:56,593 --> 00:31:59,803
rate arrays to be completely rebuilt,
which would zero everything out.

648
00:31:59,808 --> 00:32:02,383
And then, and then you do the restore.

649
00:32:03,403 --> 00:32:03,643
Correct.

650
00:32:03,673 --> 00:32:06,913
Uh, and now you, you mentioned
two rate arrays, right?

651
00:32:07,963 --> 00:32:11,383
Well, every, every one of these
servers had its own rate array.

652
00:32:11,383 --> 00:32:11,413
Okay.

653
00:32:11,713 --> 00:32:14,563
I had two different size drives, so

654
00:32:14,563 --> 00:32:14,888
I Right.

655
00:32:15,008 --> 00:32:19,663
I moved the drives around because each
system had a mix of the drive types.

656
00:32:20,563 --> 00:32:27,253
Well, um, as I bought them, I, I had two
servers that were full of the 9.1 mm-Hmm.

657
00:32:27,398 --> 00:32:28,243
Gigabyte drives.

658
00:32:28,243 --> 00:32:30,733
And, and each server held six drives.

659
00:32:30,763 --> 00:32:30,973
Yeah.

660
00:32:31,303 --> 00:32:35,863
And then I had bought three
servers that had the 18.2.

661
00:32:36,163 --> 00:32:36,583
Gotcha.

662
00:32:37,003 --> 00:32:40,303
Gigabyte drives and I only had,
uh, four, drive four of those

663
00:32:40,303 --> 00:32:41,533
drives in, in those three.

664
00:32:42,223 --> 00:32:42,433
Gotcha.

665
00:32:42,433 --> 00:32:43,393
Each of those three servers.

666
00:32:43,933 --> 00:32:44,413
So

667
00:32:44,563 --> 00:32:44,833
yeah.

668
00:32:44,833 --> 00:32:51,613
So basically you, you, in one move,
well, a series of small moves wiped

669
00:32:51,613 --> 00:32:54,673
out the storage arrays on five servers.

670
00:32:55,303 --> 00:32:56,743
Is that Yes, yes.

671
00:32:57,283 --> 00:32:57,868
Oh boy.

672
00:32:58,033 --> 00:32:59,653
And everyone was okay with this.

673
00:33:00,673 --> 00:33:02,023
I had planned on doing it.

674
00:33:02,893 --> 00:33:04,843
I, I explained what I was going to do.

675
00:33:05,773 --> 00:33:08,653
And we trusted our backup tapes.

676
00:33:13,813 --> 00:33:16,243
And by the way, the tapes did fine.

677
00:33:17,083 --> 00:33:17,833
They just, everything,

678
00:33:18,613 --> 00:33:22,363
everything was restored over
a weekend and it only took me

679
00:33:22,363 --> 00:33:25,363
five days over a weekend in five

680
00:33:25,363 --> 00:33:25,753
days.

681
00:33:26,688 --> 00:33:26,808
What,

682
00:33:29,563 --> 00:33:31,063
so what, so what was that like?

683
00:33:31,063 --> 00:33:32,323
Uh, come Monday morning.

684
00:33:32,743 --> 00:33:36,553
And you had moved into the data
center, uh, I'm assuming, is

685
00:33:36,553 --> 00:33:38,173
that what happened, by the way?

686
00:33:38,173 --> 00:33:38,713
So I,

687
00:33:38,893 --> 00:33:44,653
I slept in my office Sunday night because
the amount of time it took to rebuild the

688
00:33:44,653 --> 00:33:49,243
rate arrays, to initialize them, and then
to start restoring data, which I didn't

689
00:33:49,243 --> 00:33:53,173
realize that it was gonna take longer to
restore data than it was to back it up,

690
00:33:54,163 --> 00:33:56,263
which was, which was my first lesson.

691
00:33:56,593 --> 00:33:57,193
Why is that?

692
00:33:57,193 --> 00:33:57,613
Why is that Paul?

693
00:33:58,258 --> 00:34:00,358
Do you know why it takes longer to back

694
00:34:01,078 --> 00:34:01,138
up?

695
00:34:01,138 --> 00:34:04,708
I have not listened to your podcast
long enough to to answer that question.

696
00:34:05,773 --> 00:34:08,263
That, by the way,
massive suck up response.

697
00:34:08,263 --> 00:34:08,953
I love it.

698
00:34:09,343 --> 00:34:12,613
I don't think we've covered this
parti this particular topic, so

699
00:34:12,613 --> 00:34:14,503
that's why I want to bring it up.

700
00:34:14,713 --> 00:34:18,343
I'm guessing based on some
numbers that you've thrown out

701
00:34:18,373 --> 00:34:20,173
that this was parody based raid.

702
00:34:20,293 --> 00:34:20,683
Yes.

703
00:34:20,713 --> 00:34:21,013
Right.

704
00:34:21,013 --> 00:34:22,453
This was a raid five.

705
00:34:22,453 --> 00:34:23,338
Probably grade five.

706
00:34:23,648 --> 00:34:23,938
Okay.

707
00:34:24,073 --> 00:34:24,493
Grade five.

708
00:34:24,913 --> 00:34:27,263
That's the answer to
the question Prasanna.

709
00:34:27,283 --> 00:34:29,893
Why does it take longer to write?

710
00:34:30,628 --> 00:34:32,368
Because it has to compute the parity

711
00:34:32,938 --> 00:34:33,748
across everything.

712
00:34:33,748 --> 00:34:33,838
Yes,

713
00:34:34,618 --> 00:34:35,368
yes.

714
00:34:35,788 --> 00:34:40,768
Um, there's also, there's also
another potential, depending on the

715
00:34:40,768 --> 00:34:42,148
backup product that you're using.

716
00:34:42,148 --> 00:34:44,153
Are you doing any kind of multiplexing?

717
00:34:45,673 --> 00:34:45,793
I.

718
00:34:45,958 --> 00:34:47,668
When you're, when you're doing backups.

719
00:34:47,788 --> 00:34:49,588
I wasn't, um, at that point.

720
00:34:50,008 --> 00:34:51,058
Okay, well, mm-Hmm.

721
00:34:51,298 --> 00:34:55,558
Because that would've made it worse
if you were, and, and so let's

722
00:34:55,558 --> 00:34:56,638
just talk about that for a minute.

723
00:34:56,638 --> 00:35:01,198
So the multiplexing is evil.

724
00:35:01,498 --> 00:35:03,448
Uh, it's a necessary evil.

725
00:35:03,508 --> 00:35:07,198
I, I always felt like if, if
you're going to tape as tape

726
00:35:07,198 --> 00:35:09,388
got faster and faster, you, you.

727
00:35:10,383 --> 00:35:13,323
You know, backup speeds that
were a few megabytes per second

728
00:35:13,323 --> 00:35:17,493
were completely un incapable of
making an LTO tape drive happy.

729
00:35:17,493 --> 00:35:20,043
Even older LTOs, let alone modern LTOs.

730
00:35:20,583 --> 00:35:24,903
And so a lot of backup vendors came
out with multiplexing where they take

731
00:35:24,903 --> 00:35:29,043
a bunch of little streams and they
enter, leave them block by block onto a

732
00:35:29,043 --> 00:35:31,353
tape, which solves the backup problem.

733
00:35:31,443 --> 00:35:34,353
But then when you go to restore,
you have to read all of that

734
00:35:34,353 --> 00:35:35,913
data and throw away most of it.

735
00:35:36,213 --> 00:35:38,643
So it makes a really crappy.

736
00:35:38,853 --> 00:35:39,723
Restore speed.

737
00:35:40,023 --> 00:35:41,463
But in your case, I, yeah,

738
00:35:41,613 --> 00:35:43,113
as long as you never have
to restore, it's fine.

739
00:35:43,353 --> 00:35:43,623
As long.

740
00:35:44,643 --> 00:35:45,033
Exactly.

741
00:35:45,543 --> 00:35:49,803
Uh, but in your case, I think what you
had was to raid the, the parody Right.

742
00:35:49,803 --> 00:35:50,433
Penalty.

743
00:35:50,793 --> 00:35:54,063
And so how long did you
think it was gonna take?

744
00:35:54,333 --> 00:35:57,693
My backups were usually
done by, by midday Saturday.

745
00:35:57,693 --> 00:35:58,143
So.

746
00:35:59,698 --> 00:36:02,578
Why should it take longer than,
uh, the, than the period of

747
00:36:02,578 --> 00:36:04,258
backing up to, to restore it all?

748
00:36:04,348 --> 00:36:08,338
I, I did get four servers up
and running by Monday morning.

749
00:36:08,788 --> 00:36:11,668
I had to sleep in my office.

750
00:36:12,058 --> 00:36:12,178
Mm-Hmm.

751
00:36:12,688 --> 00:36:13,468
Sunday night.

752
00:36:13,558 --> 00:36:18,358
So I was here to change tapes
in the middle of the restore

753
00:36:18,598 --> 00:36:20,338
to get the file server running.

754
00:36:20,853 --> 00:36:25,563
Then the application server, which
had its own complexities, um, from

755
00:36:25,563 --> 00:36:30,963
running multiple applications and
trying to get a, a good backup of live

756
00:36:30,963 --> 00:36:35,823
applications, uh, took an additional
three days to, to get up and running.

757
00:36:35,943 --> 00:36:39,453
Yeah, I was gonna ask you if you were able
to get everything up and running, but it

758
00:36:39,453 --> 00:36:43,113
looks like minus the application server,
everything was good to go by Monday.

759
00:36:43,563 --> 00:36:48,483
So you're cri you, you
prioritized critical applications.

760
00:36:48,573 --> 00:36:51,513
Applications that would get
you yelled at, basically.

761
00:36:51,993 --> 00:36:55,803
Email file server logins.

762
00:36:56,103 --> 00:36:57,033
Those were, yeah, yeah.

763
00:36:57,273 --> 00:36:59,223
Those, those all sound really important.

764
00:36:59,793 --> 00:37:02,763
Uh, you know, uh, I, I don't know.

765
00:37:02,763 --> 00:37:04,683
I, what, what's the
equivalent of CEO there?

766
00:37:05,488 --> 00:37:06,928
Uh, uh,

767
00:37:07,018 --> 00:37:07,408
the borough

768
00:37:07,408 --> 00:37:10,198
manager, the Borough manager's laptop.

769
00:37:10,198 --> 00:37:12,988
If that was part of this,
that would, that would go.

770
00:37:13,678 --> 00:37:14,488
Um,

771
00:37:15,598 --> 00:37:16,138
yeah.

772
00:37:16,498 --> 00:37:17,343
So do you remember how many.

773
00:37:17,998 --> 00:37:19,948
Tapes, like you said, you
were swapping out tapes.

774
00:37:19,948 --> 00:37:22,528
Do you remember like, because I
could imagine if you're sleeping

775
00:37:22,528 --> 00:37:25,288
in your office at like, probably
like three in the morning, a tape

776
00:37:25,288 --> 00:37:28,018
probably finishes restoring and you're
like, dammit, I gotta wake up now.

777
00:37:29,098 --> 00:37:30,598
I I only had two tapes.

778
00:37:30,628 --> 00:37:30,898
Okay.

779
00:37:30,898 --> 00:37:32,848
Two tapes backed at my entire environment.

780
00:37:33,298 --> 00:37:35,698
Um, I was running the Exabyte M two.

781
00:37:36,388 --> 00:37:37,528
Tape drive.

782
00:37:37,618 --> 00:37:39,598
So this was the mammoth drives.

783
00:37:40,018 --> 00:37:46,348
So, yes, so Exabyte had mammoth,
Sony had a IT, so this was the next

784
00:37:46,348 --> 00:37:48,508
generation of eight millimeter drives.

785
00:37:48,508 --> 00:37:52,948
Because the, 'cause my, the first
tapes I cut my teeth on were

786
00:37:52,948 --> 00:37:54,838
exabyte 82 hundreds, which were the.

787
00:37:55,568 --> 00:38:00,308
The, they were like, I don't
know, uh, one gigabyte or

788
00:38:00,308 --> 00:38:02,258
something on those, those drives.

789
00:38:02,258 --> 00:38:05,618
But the mammoth was their
attempt at large, and so

790
00:38:05,618 --> 00:38:07,808
they had 60 gigabytes native.

791
00:38:07,918 --> 00:38:13,498
120 gigabytes compressed is what the,
the, the advertised capacity was, by

792
00:38:13,498 --> 00:38:16,138
the way, exabyte best company name ever.

793
00:38:16,348 --> 00:38:16,828
Right.

794
00:38:17,068 --> 00:38:18,958
But that company is no more.

795
00:38:19,048 --> 00:38:19,198
Right.

796
00:38:19,288 --> 00:38:22,588
The company that made those
drives is, uh, is no more.

797
00:38:22,588 --> 00:38:25,948
And, and, and back when Exabyte was
named Exabyte, we were all like,

798
00:38:26,128 --> 00:38:27,193
we're never gonna have an exabyte.

799
00:38:27,193 --> 00:38:28,678
Now we're, you know.

800
00:38:29,503 --> 00:38:30,643
No, it's, it's crazy.

801
00:38:31,123 --> 00:38:31,453
But yeah.

802
00:38:31,453 --> 00:38:33,313
So this was old school.

803
00:38:33,703 --> 00:38:37,873
This was a, um, a
cassette, helic scan tape.

804
00:38:37,873 --> 00:38:41,803
We talked, we talked about
helic scan a week or so.

805
00:38:41,803 --> 00:38:43,663
Actually, you, you haven't heard it.

806
00:38:43,783 --> 00:38:47,203
Well, the listeners may have heard it
by this point, but you haven't heard it

807
00:38:47,203 --> 00:38:48,703
'cause we haven't broadcast it yet, Paul.

808
00:38:48,943 --> 00:38:51,013
Not the fastest tape drives in the world.

809
00:38:51,073 --> 00:38:54,643
So you were able, I guess after five
days, get all the data back, get the

810
00:38:54,643 --> 00:38:58,453
applications and everything else up and
running, and you still had your job.

811
00:39:00,268 --> 00:39:05,188
I'm still, I'm still here 20, 28
years plus later I'm still here.

812
00:39:05,188 --> 00:39:05,608
Yes.

813
00:39:06,478 --> 00:39:10,618
And, and, and again for, for those
who are listening who say, well,

814
00:39:10,618 --> 00:39:16,078
gee, the tapes held a maximum of
120 gigabytes with compression.

815
00:39:16,168 --> 00:39:20,608
You know, if the tapes held a maximum
of 120 gigabytes, so that's a maximum

816
00:39:20,608 --> 00:39:22,798
of 250 gigabytes, you had to restore.

817
00:39:23,203 --> 00:39:24,823
What's the big freaking deal?

818
00:39:25,273 --> 00:39:29,743
Well, the big freaking deal was that
that was a ton of data back then, right?

819
00:39:30,133 --> 00:39:36,913
That was a really big, let's see, the, the
transfer rate, I'm showing it 12, if I did

820
00:39:36,913 --> 00:39:41,743
my math right, 12 megabytes per second,
which sounds about right given the.

821
00:39:42,448 --> 00:39:44,128
The generation and timeframe.

822
00:39:44,488 --> 00:39:44,968
So yeah.

823
00:39:44,968 --> 00:39:48,988
So it's only 120 gigabytes,
but the advertised thing

824
00:39:48,988 --> 00:39:50,998
was 43 gigabytes per hour.

825
00:39:51,208 --> 00:39:54,688
But clearly you weren't getting
that, that that was the problem.

826
00:39:54,688 --> 00:39:59,158
You were not getting the advertised
transfer rate because of the right

827
00:39:59,158 --> 00:40:01,918
penalty that you were experiencing
when you were doing the Restore.

828
00:40:02,998 --> 00:40:07,528
So what, what, uh, so, so
we've already covered it.

829
00:40:07,528 --> 00:40:09,208
You, you made it through.

830
00:40:09,553 --> 00:40:10,783
Everything restored.

831
00:40:11,473 --> 00:40:18,853
Clearly you didn't meet the, the
objective, you know, the, the initial

832
00:40:18,853 --> 00:40:24,553
time objective, but you got everything
back and you got the critical things

833
00:40:24,553 --> 00:40:29,743
back by Monday morning, and so
I, I'm guessing that you didn't.

834
00:40:30,448 --> 00:40:36,298
Like there wasn't a, was was there one
of those giant postmortem sessions?

835
00:40:36,778 --> 00:40:41,248
You know, I, I don't think it was
till, uh, till maybe a, a week or

836
00:40:41,248 --> 00:40:45,883
two down the road that I realized
how incredibly stupid that was.

837
00:40:50,608 --> 00:40:53,938
So you, so you, you then had a postmortem
with yourself is what you're saying?

838
00:40:54,448 --> 00:40:54,928
I did.

839
00:40:54,928 --> 00:40:54,930
I did.

840
00:40:54,935 --> 00:40:56,668
You, everybody was happy.

841
00:40:57,178 --> 00:41:02,563
I, I mean, right back in, back in that
day, uh, you know, probably, uh, a,

842
00:41:02,683 --> 00:41:08,398
a a month or two later, I had talked
to somebody about, about storage area

843
00:41:08,398 --> 00:41:12,928
networks and, and they had given me a
quote for, oh, you want to, you want to

844
00:41:12,928 --> 00:41:18,898
add storage to your servers with a storage
area network that'll only be $30,000.

845
00:41:20,218 --> 00:41:22,408
In, in, in 2001.

846
00:41:23,158 --> 00:41:23,398
Right.

847
00:41:23,398 --> 00:41:25,858
And, uh, you know, I, I
felt rather accomplished.

848
00:41:25,948 --> 00:41:26,128
Yeah.

849
00:41:26,188 --> 00:41:30,238
I, I had done what I intended to do,
you know, granted it was a, a sleepless

850
00:41:30,238 --> 00:41:33,148
night on the floor of my office.

851
00:41:33,568 --> 00:41:38,608
But, so you're saying that because
you reorganized this data, the, the

852
00:41:38,608 --> 00:41:42,868
storage and you reallocated the storage
more efficiently, you didn't need the

853
00:41:42,868 --> 00:41:46,048
$30,000 San, is that what you're saying?

854
00:41:46,708 --> 00:41:47,098
Correct.

855
00:41:47,983 --> 00:41:51,523
So I had achieved the objective
that I was after, and I saved my

856
00:41:51,523 --> 00:41:53,743
organization money in my mind.

857
00:41:54,673 --> 00:42:00,883
Did you tweak or change any of your backup
restore plans based on this experience?

858
00:42:01,333 --> 00:42:07,993
I think I changed my expectations on my
restore plans because it all restored,

859
00:42:08,893 --> 00:42:13,903
but, but at the same time it, it was
the knowledge that when you're backing

860
00:42:13,903 --> 00:42:16,513
up live applications that are running.

861
00:42:17,218 --> 00:42:20,998
They, that's a difficult,
that's a difficult thing.

862
00:42:21,148 --> 00:42:26,098
And so now I see, you know, um,
in, in virtual environments where,

863
00:42:26,308 --> 00:42:30,178
where things are, um, and I'm
missing the right word for it, but

864
00:42:30,298 --> 00:42:32,983
where things are flushed and, um.

865
00:42:34,018 --> 00:42:36,088
We asked, brought to rest.

866
00:42:36,508 --> 00:42:36,898
Yeah.

867
00:42:36,958 --> 00:42:42,238
So, so you recognize, I mean, you,
you saw when you began telling me

868
00:42:42,238 --> 00:42:47,638
this story, you know, when you started
telling us this story, the, the, the

869
00:42:47,638 --> 00:42:54,808
idea of essentially completely deleting
your entire data center and then using

870
00:42:54,808 --> 00:42:56,878
your backups really for the first time.

871
00:42:58,588 --> 00:43:00,628
I was a bit flabbergasted.

872
00:43:00,628 --> 00:43:01,558
But you you,

873
00:43:01,943 --> 00:43:02,543
I was young.

874
00:43:03,063 --> 00:43:03,688
I was young.

875
00:43:04,018 --> 00:43:08,158
You're agreeing with me that this
was a really bad thing to do.

876
00:43:08,158 --> 00:43:08,848
It sounds like

877
00:43:08,968 --> 00:43:11,878
Paul's gotten wiser with
the longer beard and.

878
00:43:12,793 --> 00:43:13,723
Since those days.

879
00:43:13,723 --> 00:43:13,933
Yeah.

880
00:43:13,933 --> 00:43:14,623
By the way, for,

881
00:43:14,653 --> 00:43:16,658
for the, for the listeners
here, white there, in

882
00:43:16,658 --> 00:43:16,858
here,

883
00:43:17,058 --> 00:43:21,013
I, I, I, I do think it's appropriate
to mention, so, you know, we record

884
00:43:21,013 --> 00:43:25,483
this as audio, but I'm looking
at a camera version of, of, you

885
00:43:25,483 --> 00:43:27,373
know, my co-host and my guest here.

886
00:43:27,793 --> 00:43:33,023
And I am the only one who doesn't
have this long flowing beard Prasanna.

887
00:43:33,043 --> 00:43:35,113
Still has this yeared, uh, how long?

888
00:43:35,203 --> 00:43:35,743
How long now?

889
00:43:35,773 --> 00:43:36,433
It's now I think

890
00:43:36,583 --> 00:43:37,723
like 19 months.

891
00:43:38,458 --> 00:43:40,108
19 months without shaving.

892
00:43:40,108 --> 00:43:44,968
So he's got this long, uh,
much blacker beard than uh,

893
00:43:44,998 --> 00:43:47,668
Paul, but Paul has the length.

894
00:43:47,908 --> 00:43:51,598
Paul ha Paul, I'll just say this,
Paul looks like he's from Alaska.

895
00:43:53,008 --> 00:43:57,298
He looks exactly like what I would
expect from someone from Alaska.

896
00:43:57,478 --> 00:44:01,888
He has a redneck cap on and this
long, you know, beard, although

897
00:44:01,918 --> 00:44:06,598
quite a bit grayer than Prasanna and
it may have been this event Paul.

898
00:44:07,163 --> 00:44:11,153
That put that you're like, you
know, it's one of those things where

899
00:44:11,153 --> 00:44:14,183
you're like, I did this to myself.

900
00:44:14,483 --> 00:44:18,383
I have no one to blame but myself.

901
00:44:18,593 --> 00:44:23,003
I'm sure you said that many
times during that event.

902
00:44:23,393 --> 00:44:29,663
So what, so going back, what, so
your goal was laudable and your

903
00:44:29,663 --> 00:44:31,913
eventual results were successful.

904
00:44:33,083 --> 00:44:35,213
What would you have done differently?

905
00:44:35,213 --> 00:44:35,273
I.

906
00:44:36,658 --> 00:44:41,968
To accomplish the same goal, but without
perhaps the amount of pain that you had,

907
00:44:42,658 --> 00:44:45,658
I, I would've, I would've had
a safety line, uh, or, or some

908
00:44:45,658 --> 00:44:47,068
sort of, sort of safety rope.

909
00:44:47,608 --> 00:44:53,908
Um, I think that, uh, you know, if I
had a, a, a spare server or a surplus

910
00:44:53,908 --> 00:45:00,088
server, that I could have migrated each
one of my servers over one at a time.

911
00:45:01,888 --> 00:45:05,098
Planned outages, so I wasn't
destroying everything.

912
00:45:05,218 --> 00:45:10,618
I had no, I had no capacity,
no, no rate arrays, right, left,

913
00:45:10,708 --> 00:45:11,068
right.

914
00:45:11,068 --> 00:45:11,143
You, yeah.

915
00:45:11,143 --> 00:45:11,983
After pulling

916
00:45:12,083 --> 00:45:17,758
drives out, you know, there were no
servers that were functioning until I

917
00:45:17,758 --> 00:45:21,568
started restoring, restoring my domain
controller from that very first server.

918
00:45:22,223 --> 00:45:29,038
I, I mean, I, I had a, a, a
textbook Windows 2000 environment

919
00:45:29,128 --> 00:45:31,378
with two domain controllers.

920
00:45:32,308 --> 00:45:33,628
Both of those were offline.

921
00:45:34,168 --> 00:45:35,638
My email server was offline.

922
00:45:35,638 --> 00:45:36,868
My file server was offline.

923
00:45:36,868 --> 00:45:38,158
My application server was offline.

924
00:45:38,248 --> 00:45:41,068
Everything was offline
until I started restoring.

925
00:45:42,328 --> 00:45:48,148
Now it is, I would, I would just
start in with one server and, and,

926
00:45:48,148 --> 00:45:50,068
and work through it methodically

927
00:45:50,668 --> 00:45:51,423
with a, I just wonder.

928
00:45:52,573 --> 00:45:57,553
I wonder the degree to which that would've
been possible given, you know, I don't

929
00:45:57,553 --> 00:45:58,903
have the, I don't have a whiteboard.

930
00:45:58,903 --> 00:46:01,273
I don't know how much you, because
you were moving, drives around

931
00:46:01,273 --> 00:46:02,683
and reallocating resources.

932
00:46:03,223 --> 00:46:05,863
I don't know the degree to which
that would've been possible.

933
00:46:06,763 --> 00:46:08,023
It would've cost some money.

934
00:46:08,263 --> 00:46:13,333
But in, in hindsight, I mean,
how bad could I have screwed up?

935
00:46:13,948 --> 00:46:16,623
I mean, if, if my, if, if
I had, I don't think it

936
00:46:16,623 --> 00:46:17,383
could have been any worse.

937
00:46:18,743 --> 00:46:22,888
I, I mean, if I had accidentally dropped
my tapes or, uh, ran them across the,

938
00:46:23,183 --> 00:46:30,298
a, a magnet, uh, between point A and
point B and lost my backups, I, I,

939
00:46:30,628 --> 00:46:31,408
it would've been a be

940
00:46:31,708 --> 00:46:34,918
I'm so far beyond it that I don't,
I, I don't think about these things.

941
00:46:36,328 --> 00:46:40,138
Well, the, but now you're here and
you're talking to us, and so we're

942
00:46:40,138 --> 00:46:43,318
asking you to relive that horrible day.

943
00:46:44,008 --> 00:46:51,268
So I, I, I, I think if, you know, looking
back on it, this is, and again, you know.

944
00:46:51,928 --> 00:46:52,678
Love you Paul.

945
00:46:52,678 --> 00:46:57,898
Uh, thanks so much for coming on and
being, being, uh, open at, at the same

946
00:46:57,898 --> 00:46:59,368
time, I'm gonna yell at you a little bit.

947
00:46:59,968 --> 00:47:09,148
Um, to me, your core, your core failure
was failure to not test at least one

948
00:47:09,628 --> 00:47:12,268
restore prior to doing this, right.

949
00:47:12,688 --> 00:47:17,128
Um, because you blew up your entire
environment without any idea.

950
00:47:17,893 --> 00:47:22,093
What restoring even one server was
gonna be like if, even if you had just

951
00:47:22,093 --> 00:47:27,313
blown up one server and restored it
because your problem, everything worked.

952
00:47:27,433 --> 00:47:28,723
Everything worked.

953
00:47:28,993 --> 00:47:33,613
Your only problem was a failure to set
proper expectations even within yourself.

954
00:47:33,673 --> 00:47:38,713
And that was because you'd never actually
done a large restore with your backups.

955
00:47:38,713 --> 00:47:41,093
By the way, you are not alone Prasanna.

956
00:47:41,113 --> 00:47:41,773
Is he alone?

957
00:47:41,773 --> 00:47:42,823
Not at all.

958
00:47:43,913 --> 00:47:48,083
And, and, and I will also say that I
have been in this situation before.

959
00:47:48,083 --> 00:47:50,213
I'll, I'll tell you a similar story.

960
00:47:50,213 --> 00:47:55,973
A hundred years ago when I bought a, my
first commercial backup program, the pro,

961
00:47:56,003 --> 00:47:58,163
the pro, the, the product was called SA.

962
00:47:59,233 --> 00:48:03,073
For archive, even though, which now
offends me because it wasn't back,

963
00:48:03,103 --> 00:48:04,663
it wasn't archive, it was backup.

964
00:48:05,053 --> 00:48:08,743
But anyway, SMR, which was a
Minneapolis company, they're,

965
00:48:08,743 --> 00:48:11,173
they're no longer software moguls
is the name of the company.

966
00:48:11,233 --> 00:48:16,213
And I had bought this as my first
commercial backup tool, and we had had

967
00:48:16,213 --> 00:48:20,953
it for a couple of months, but I was
still running my dumps to my old tapes.

968
00:48:21,073 --> 00:48:22,423
In the meantime.

969
00:48:22,713 --> 00:48:23,103
Right.

970
00:48:23,613 --> 00:48:32,943
And then we had this, our first large
outage, we lost the dis drives on

971
00:48:33,843 --> 00:48:38,163
our primary file server H pfs oh one,
I still remember the server name.

972
00:48:38,163 --> 00:48:39,633
And that was 25 years ago.

973
00:48:40,593 --> 00:48:43,683
And I, I was so excited.

974
00:48:43,833 --> 00:48:47,463
I grabbed my SMR tapes and my.

975
00:48:47,948 --> 00:48:49,988
Dump tapes, put them in my back pocket.

976
00:48:50,288 --> 00:48:54,608
And I ran, I remember dri because it, it
was a couple miles down the road where

977
00:48:54,608 --> 00:48:56,078
the, where the other data center was.

978
00:48:56,078 --> 00:48:58,988
And I remember running down
there throwing in the tape drive.

979
00:48:59,048 --> 00:49:00,818
And I remember kicking off the restore.

980
00:49:00,818 --> 00:49:03,248
And what I remember was this was Blink.

981
00:49:03,248 --> 00:49:03,668
Blink.

982
00:49:05,138 --> 00:49:06,878
Long period.

983
00:49:07,718 --> 00:49:07,988
Blink.

984
00:49:07,988 --> 00:49:10,718
Blink long period.

985
00:49:12,928 --> 00:49:18,118
And what I did was I created a wild loop
and I was watching the, the, the, the size

986
00:49:18,118 --> 00:49:21,448
of the file system not grow totally okay.

987
00:49:21,598 --> 00:49:25,978
At least not, not by a speed that was
gonna finish anytime that millennium.

988
00:49:26,458 --> 00:49:30,298
And so I called the, the tech
support and I was like, Hey, uh,

989
00:49:30,688 --> 00:49:31,948
you know what, what's going on?

990
00:49:31,948 --> 00:49:35,308
And they go, well, by any chance did
you turn on the compression feature?

991
00:49:35,908 --> 00:49:36,898
Yes, I did.

992
00:49:37,228 --> 00:49:39,838
It was a software compression
feature, which, which.

993
00:49:40,298 --> 00:49:42,368
The way it worked was, this is old school.

994
00:49:42,548 --> 00:49:48,308
It would, it would run a compressed
minus, uh, CI think would be, uh, to,

995
00:49:48,308 --> 00:49:53,468
to compress the file, to send an input,
and then redirect it to a file in

996
00:49:53,468 --> 00:49:58,418
temp, and then back up that compressed
file During a restore, it would com

997
00:49:58,658 --> 00:50:05,648
restore the compressed file into
temp, then run Uncompress on the file.

998
00:50:06,148 --> 00:50:11,908
At in that location and then move
the file from temp the uncompressed

999
00:50:11,908 --> 00:50:13,168
version of the file from temp.

1000
00:50:13,438 --> 00:50:14,518
There's a lot, you know.

1001
00:50:14,698 --> 00:50:19,108
Anyway, long story short,
it was never gonna finish in

1002
00:50:19,108 --> 00:50:20,338
any sort of reasonable time.

1003
00:50:20,398 --> 00:50:25,318
And thank God I still had my other
tapes, but this was all because I did

1004
00:50:25,318 --> 00:50:28,918
the same thing you did and that was,
I had never tested a large restore.

1005
00:50:29,548 --> 00:50:29,698
Yes.

1006
00:50:29,818 --> 00:50:33,538
And um, you know, in your
case, thank God you had.

1007
00:50:33,538 --> 00:50:34,613
You had enough time.

1008
00:50:34,613 --> 00:50:38,633
Thank God you were able to be able to
restore the critical servers and time

1009
00:50:38,873 --> 00:50:42,833
so that you know nobody's pulling their
hair out and your, your borough could

1010
00:50:42,833 --> 00:50:45,623
continue to do its function in my case.

1011
00:50:46,468 --> 00:50:50,068
Uh, thank God I had the other tapes in my
back pocket because I just pulled them out

1012
00:50:50,068 --> 00:50:54,508
and just typed, you know, UFS Restore, you

1013
00:50:54,508 --> 00:51:00,268
know, I was going to say that,
um, one thing for testing, right?

1014
00:51:00,268 --> 00:51:04,498
I know Paul, you mentioned that you
had two domain controllers, right?

1015
00:51:04,498 --> 00:51:07,378
I think potentially you could
have taken down one of the domain

1016
00:51:07,378 --> 00:51:11,908
controllers and sort of done a
restore of that domain controller.

1017
00:51:12,298 --> 00:51:15,718
To test it out while still keeping
the entire environment up and running,

1018
00:51:16,168 --> 00:51:19,588
and then made sure, and you probably
would've noticed, hey, my backups

1019
00:51:19,588 --> 00:51:21,028
are slow, or My restores are slow,

1020
00:51:22,138 --> 00:51:26,248
so he shouldn't test it on his most
important, most critical server.

1021
00:51:28,288 --> 00:51:29,848
There are so many things that

1022
00:51:30,328 --> 00:51:31,678
could have been done way, could,
could've, shoulda, would've.

1023
00:51:31,713 --> 00:51:32,278
It's okay.

1024
00:51:32,398 --> 00:51:32,968
We all learn.

1025
00:51:33,148 --> 00:51:34,318
We all learn from these lessons.

1026
00:51:34,318 --> 00:51:34,768
Honestly, that's

1027
00:51:34,768 --> 00:51:35,248
why you're here.

1028
00:51:35,518 --> 00:51:35,668
Yeah.

1029
00:51:35,998 --> 00:51:37,168
You and there.

1030
00:51:38,188 --> 00:51:41,608
Have you ever seen there, there
there's a company called despair.com.

1031
00:51:41,668 --> 00:51:42,718
Have you seen this company?

1032
00:51:43,258 --> 00:51:43,678
I have not.

1033
00:51:43,678 --> 00:51:48,928
So they, they make, they make
de-motivation posters and, uh,

1034
00:51:48,988 --> 00:51:52,708
one of them, one of them is
a picture of a sinking ship.

1035
00:51:53,128 --> 00:51:54,688
They look like motivational posters.

1036
00:51:55,318 --> 00:51:58,738
They're DEMOTIVATIONAL posters, and one
of them is a picture of a sinking ship

1037
00:51:58,918 --> 00:52:01,378
and it says, I think it says mistakes.

1038
00:52:01,438 --> 00:52:04,678
And then it said it could be that
the purpose of your life is to

1039
00:52:04,678 --> 00:52:05,968
serve as a warning to others.

1040
00:52:06,808 --> 00:52:10,378
I would much rather people
listen to this and run.

1041
00:52:12,478 --> 00:52:17,863
From, from my, from my decisions then to
then to think that I had good decisions.

1042
00:52:18,043 --> 00:52:21,613
But one of the funny things though,
Curtis, is in your book, I remember

1043
00:52:21,613 --> 00:52:22,723
you telling me the story, right?

1044
00:52:22,723 --> 00:52:25,093
That when you're writing the book,
right, and you were sending it

1045
00:52:25,093 --> 00:52:26,233
out for all the reviewers, right?

1046
00:52:26,233 --> 00:52:26,293
I.

1047
00:52:26,683 --> 00:52:29,293
Someone came back to you and was
like, Hey Curtis, you forgot to put

1048
00:52:29,293 --> 00:52:31,813
a chapter about testing backups.

1049
00:52:31,813 --> 00:52:31,873
Yeah.

1050
00:52:31,903 --> 00:52:37,333
I completely left testing out of my
book and thanks to Stewart Guy, it's

1051
00:52:37,333 --> 00:52:41,443
like, it's like the fourth time that
Stewart gets credit on the podcast.

1052
00:52:41,833 --> 00:52:42,463
You were right.

1053
00:52:42,463 --> 00:52:43,003
Stewart

1054
00:52:45,043 --> 00:52:46,633
Guy, his name's Stuart Little like.

1055
00:52:48,238 --> 00:52:48,748
Come on.

1056
00:52:48,748 --> 00:52:49,558
He's a mouse.

1057
00:52:50,428 --> 00:52:51,178
I love you, Stuart.

1058
00:52:51,508 --> 00:52:52,048
Anyway.

1059
00:52:52,498 --> 00:52:52,768
All right.

1060
00:52:52,768 --> 00:53:01,348
Well, Paul, I, I, again, want to applaud
you for coming on, for being so honest

1061
00:53:01,348 --> 00:53:07,708
about what was clearly, clearly a very
large mistake that you made it out alive.

1062
00:53:07,768 --> 00:53:09,118
You accomplish your goal.

1063
00:53:09,118 --> 00:53:09,358
Right.

1064
00:53:09,358 --> 00:53:10,318
It, it's not like.

1065
00:53:10,808 --> 00:53:13,598
I mean it, this could have
been much worse, right?

1066
00:53:13,598 --> 00:53:14,738
It could have been.

1067
00:53:15,068 --> 00:53:18,518
I intentionally destroyed my
entire environment and then I

1068
00:53:18,518 --> 00:53:20,078
found out my backups don't work.

1069
00:53:21,343 --> 00:53:22,243
It could have been that one.

1070
00:53:22,473 --> 00:53:23,483
That could have, it could have.

1071
00:53:23,618 --> 00:53:27,218
It wasn't that, thank
God you, it wasn't that.

1072
00:53:27,878 --> 00:53:29,258
But thank you.

1073
00:53:29,468 --> 00:53:34,208
Like, just this, this is, I mean
this may be the best story we've

1074
00:53:34,208 --> 00:53:37,298
had on, you know, on the podcast.

1075
00:53:37,298 --> 00:53:39,938
We've had some other
people that have had, I.

1076
00:53:40,318 --> 00:53:41,548
Bad things happen to him.

1077
00:53:41,758 --> 00:53:45,178
This is the first time where it
was, you know, self-inflicted

1078
00:53:47,188 --> 00:53:48,298
user created.

1079
00:53:48,538 --> 00:53:50,548
Normally we're blaming the end user.

1080
00:53:50,818 --> 00:53:52,048
We don't blame ourself.

1081
00:53:52,648 --> 00:53:52,978
Yeah.

1082
00:53:52,978 --> 00:53:56,008
This is a classic p CAC situation.

1083
00:53:56,158 --> 00:53:57,598
Are you familiar with that acronym?

1084
00:53:58,003 --> 00:54:02,623
Absolutely a problem exists between
keyboard and chair for those that don't.

1085
00:54:02,713 --> 00:54:08,473
Uh, and then the entire environment
became fubar, which is, uh, look that up.

1086
00:54:08,473 --> 00:54:11,593
Fouled up beyond all recognition.

1087
00:54:11,623 --> 00:54:11,983
Yeah.

1088
00:54:11,983 --> 00:54:14,078
So wouldn't you agree,
Prasanna, this is like.

1089
00:54:15,058 --> 00:54:15,808
This has been great.

1090
00:54:15,808 --> 00:54:19,348
This has been an awesome story and
it's something that I don't think

1091
00:54:19,348 --> 00:54:21,538
end users realize what goes on.

1092
00:54:21,568 --> 00:54:25,378
Like I know sometimes we blame end users
for, oh, you did this, you did that,

1093
00:54:25,378 --> 00:54:28,228
but everyone's human things happen.

1094
00:54:28,918 --> 00:54:31,078
So don't get frustrated at your IT people.

1095
00:54:31,138 --> 00:54:31,858
Absolutely.

1096
00:54:31,858 --> 00:54:34,288
By the way, and and I, I say this
every once in a while, you know,

1097
00:54:34,288 --> 00:54:38,578
there's only two industries where they
refer to their customers as users,

1098
00:54:39,688 --> 00:54:40,858
IT and drug dealers.

1099
00:54:41,443 --> 00:54:41,773
Yeah.

1100
00:54:45,583 --> 00:54:48,853
You know, it's a thing
anyway, so, uh, tha yeah.

1101
00:54:48,853 --> 00:54:49,393
And you know what?

1102
00:54:49,423 --> 00:54:53,113
If you're out there and you have
a story like this, we'd love to

1103
00:54:53,113 --> 00:54:54,043
have you come on and tell it.

1104
00:54:54,073 --> 00:54:55,393
We'll even let you be anonymous.

1105
00:54:55,393 --> 00:54:58,783
If you're embarrassed about what happened,
you know, we'll give you a synonym.

1106
00:54:59,173 --> 00:55:02,713
Uh, you know, we, we did some Harry
Potter characters for a while.

1107
00:55:03,313 --> 00:55:07,933
We'll pick, you know, pick your favorite
book and, uh, you know, whatever.

1108
00:55:07,993 --> 00:55:11,473
We'll make you a, we'll make you
one of the eternals from the movie

1109
00:55:11,473 --> 00:55:14,893
that just came out and whatever,
you know, whatever you want to be.

1110
00:55:14,983 --> 00:55:18,913
If you just, we just love great stories
because we learn from it, right?

1111
00:55:19,513 --> 00:55:20,593
That's the key.

1112
00:55:21,043 --> 00:55:22,183
Mistakes happen.

1113
00:55:22,723 --> 00:55:24,463
Um, you know, we learn.

1114
00:55:24,613 --> 00:55:28,633
So thanks, uh, to, thanks
Paul so much for coming on.

1115
00:55:28,783 --> 00:55:29,323
Thank you

1116
00:55:29,533 --> 00:55:33,613
and thanks Prasanna for, uh,
your insight into this as well.

1117
00:55:34,033 --> 00:55:34,693
Anytime

1118
00:55:34,693 --> 00:55:35,023
Curtis.

1119
00:55:35,023 --> 00:55:35,803
And thanks Paul.

1120
00:55:35,893 --> 00:55:39,313
And, uh, thanks to the listeners
and remember to subscribe.

1121
00:55:39,894 --> 00:55:41,124
That is a wrap.