​ W. Curtis Preston: hi, and welcome to Backup Central's Restore it All podcast.
I'm a host w Curtis Preston, aka Mr.
Backup, and I have with me my software troubles consultant, Prasanna Malaiyandi.
How's it going, Prasanna?
Prasanna Malaiyandi: I'm good.
I'm good Curtis.
And like software troubles.
You should expect it.
You've been in the industry for how long now?
Like it
W. Curtis Preston: I know, but I, I, you know, that I generally don't
like, I generally, so like, I like tech and then like, I like working
on tech, but then I like using tech and the tech that I'm using.
I just want it to work.
I'm not TiVo anymore, but you know, when I had the TiVo, that's why I
opted for TiVo and not Linux box.
With hard drive in some software that will record my TV stream, right?
Um, the stuff that I actually use on a day to day basis.
I want to just, I want it to just work.
And I've broken that trend by, uh, upgrading a tool that I love, right?
I love descript, but I've decided to go on the bleeding edge with
their new version, which is amazing.
But it's in beta and I'm and I'm using a beta tool for the first time.
And I'm like, I don't know.
I push this button and it doesn't, it doesn't do the thing.
Right.
And then, yeah,
Prasanna Malaiyandi: But, but, but at least you have to hand it
to the descript support folks for actually like, taking your calls and
your chats and your emails, Right.
And be like, Hey, let me walk you through that.
W. Curtis Preston: they're good with live chat.
Although there's been a handful where.
Where at some point they're like, you know, I think it's
best to transfer this to email.
That's when you know, you've put, you've asked like one too
many questions in a support chat.
They're like, I think maybe you should put all your questions in an email.
I'm like, But that's not, that's not how my brain works.
Like I, I get a question, I get it answered.
I work for a little bit a while and then I get another question, you know?
Um, yeah.
Anyway, so
Prasanna Malaiyandi: sometimes it unblocks you and then you like hit the next
thing and you're like, Oh, what is this?
Which is why if you're a good product manager, right, you go in front
of customers and you're like, Hey, show me what you're doing and let me
figure out like what your problems are, what your pain points are,
what features I should be building?
W. Curtis Preston: Yeah.
And, and it is, it is kind of funny though that I now, I'm like
on a first name basis with like four or so of their, of their, uh,
Prasanna Malaiyandi: shout out to the descript support team
W. Curtis Preston: Yeah.
Shout out to for putting up with my, uh, my questions.
Cause it's like, it's a combination of, the new version is significantly
different than the old version.
Um, and so it's a, it's a combination between Is this the way it's
supposed to work now or is this the way it's supposed to work?
Prasanna Malaiyandi: Well, and
I
W. Curtis Preston: think it's broken.
Prasanna Malaiyandi: And I think for a lot of the features, right,
they're building the right things.
Like just talking to you like they've optimized some of your flows and some of
the things that you've spent so much time working on in the past, right?
So it's going to get there, It's just, it's gonna take a little bit of time.
W. Curtis Preston: But it's been, it's been a difficult
couple of weeks, you know?
Um, anyway, let's bring on our longtime friend.
Uh, I, I've known him for, I don't know, for forever, and he's for, he's
been on the podcast multiple times.
You may remember him from, from such hits as how does Hollywood do backup?
And, uh, he is also.
A, uh, a co-author on my book, uh, latest book, uh, Modern Data
Protection, which by the way, you can get free by going to druva.com/ebook.
And, um, he wrote the chapter that we're gonna talk about today
in our Back to basic series.
He is now the head of Technical Operations for Frame Store, a
visual effects studio in Hollywood.
Welcome to the podcast Jeff Rochlin.
Jeff Rochlin: Hey.
Prasanna Malaiyandi: Yeah.
. . Woo-hoo.
Jeff Rochlin: Hi guys.
How are ya?
W. Curtis Preston: How's it going, Jeff?
Jeff Rochlin: It's going.
It's going well.
It's going.
W. Curtis Preston: It, it's, uh, it, it is it nice to be back?
Cuz I know for a while you weren't, you weren't, you, you've always
enjoyed sort of combining the tech side of you and the, and the, you
know, the entertainment side of you.
And for a while you weren't in that space Right now it's good to be
back with, you're a full person.
You get to be in entertainment and be in tech at the same time.
Jeff Rochlin: definitely.
Um, I love the creative.
Part of it.
And I love dealing with the crazy people that make creative things.
And so it's, that's, that's why I've spent, you know, the better part
of the last 30 years working at, um, uh, some kind of movie studios,
preferably in production and not, you know, it cause anybody, this will get
me in trouble, anybody can do email.
But, you know, dealing, dealing with the, dealing with the executive
producer who's trying to get his movie out on time is a different kind of,
Prasanna Malaiyandi: Yeah, I can only imagine the pressure with that
Jeff Rochlin: Um, yeah, the, the only place I've worked in the past that
where the pressure was worse in a different kind of way was Wall Street.
Working on a trading floor in Wall Street is a different piece also,
but
W. Curtis Preston: cuz that's where like you're, you know, you're making a
trade or you're trying to make a trade or you're trying to make thousands of
trades and suddenly the system goes down.
Jeff Rochlin: Right?
And it's, and it's all time critical, right?
Because the price is not sitting static waiting for you to close the trade.
If you don't buy it right now, then five minutes from now it's gonna be, you know,
a hundred million dollars more in cost and there goes your profit margin, right?
So,
W. Curtis Preston: Yeah.
Which is why, at least in in my IT experience, and I spent a, I spent a good
time in and outta Wall Street and working with other consultants that I knew that
were working on Wall Street, and I, I got the sense, it was the one place that I've
seen where it's like money is no object.
So they do, they do what is right.
To, um, to have fully, highly available systems.
Um, and I, I'd say Hollywood might be right behind that, right?
Because you've got all these super expensive actors, um, and super expensive,
you know, both above and below the line.
People see, I learned from you, I learned that
Jeff Rochlin: You did.
you.
did.
W. Curtis Preston: Uh, and, and they're all sitting there and, and
you know, and if, and if they can't do it because the computers are down,
Prasanna Malaiyandi: what?
What's above and below the lo line?
By the way,
W. Curtis Preston: oh.
Jeff Rochlin: anybody in the creative titles, actors, directors, producers,
writers are called Above the line.
Um, anybody in the skilled trades and the crew, right?
The, the camera, the, uh, the gaffer, who's the electrician, uh, you know,
those kind of got the stage hands.
Those are below the line.
Um, and the best way to describe it is, I guess if you watch the credits of a
movie, um, at the end, you know, they always run the, the cast gets their,
their individual cards and the people who wrote the movie get their cards,
and then it goes into that long scroll.
Of everybody else that's kind of above and below the line, you know, visually.
Um,
Prasanna Malaiyandi: Interesting.
Yeah.
Jeff Rochlin: so Hollywood is kind of always walking that middle line between,
I wanna, I want perfection and I really don't want to have to pay for it.
You know what I mean?
Which is sort of what the rest of the world is doing as well.
Um, how can I strike the balance so that I know I'll get a hundred percent of what
I need and yet at the same time, not have to spend the money to, to make it happen?
Uh, backups are a, backups and storage in a production are a
perfect example of this, right?
Because artists iterate their work over and over and over again.
They create a lot of files at the end of the day that they don't need because.
They've done, you know, they've done 20 versions of the scene and the
21st is the one they're gonna use.
Um, and then you go back to them and say, We're out of space.
Go clean up.
And they look at you like, Well, I don't have time to clean up.
I'm busy making art.
If you want somebody to clean up, then I can't make art.
Right.
. And you're looking at 'em and going, Well, I can go spend a hundred
thousand dollars and buy more storage for you to put more stuff on.
But eventually somebody's gotta clean this up cuz you don't need that storage.
Um, and that's been a kind of a standard theme throughout my career in Hollywood.
Um, they, they are upset when it's broken.
They, it costs them a lot of extra money to make up for it when it's broken, right?
Because over time on skilled trade and union people is not cheap.
Um, and so we do everything we can to make sure that it doesn't break..
Prasanna Malaiyandi: And I'm guessing, like especially with Hollywood, right,
when you're producing or creating right, uh, content, right, Those are
like significantly large files, right?
Uh, like you said, if you're doing like 21 versions of it, right?
The only other industries I could really think of that are similar is
probably things in like the, uh, genome space or the mining space, right?
Where you just have a whole bunch of data all coming in at once, but.
Jeff Rochlin: And we're in, uh, from a, like a rendering, the, the final
stage of production compute process.
It's the same kind of technology being used, right?
We run supercomputing clusters on the back end to render large projects, uh,
just like oil and gas, use supercomputing clusters to analyze data to figure
out where to drill next, right?
Um, and as the, the beautiful picture that you get on your TV at home gets
better and better when it moves from two K to 4k to eight K to 16 K plugged
directly into the headset that's connected into your cerebral cortex so that you're
living in the picture , that the files get exponentially larger every time.
Right.
Um, and I think I did the math for you a long time ago where, think about a.
From the perspective of there are 23 frames, still pictures in every
second of the film, and the film averages two and a half hours.
Right.
That's a lot of, that's a lot of images,
W. Curtis Preston: Right.
And, and, you know, and, and I'm glad, I'm glad we sort of started with this
part of the discussion because what we wanted to talk about, you know, this is
our back to basic series and I'll take this opportunity to do our disclaimer,
Um, uh, Prasanna and I and Jeff, we all work for different companies.
I work for Druva, Prasanna works for Zoom, and Jeff works for Frame Store.
We're not speaking for our companies.
We're speaking, uh, our individual, um, uh, opinions.
And uh, also if you would like to rate us, we'd love you to do that.
We're.
Running our Santa Claus special . If you, if you want to see this beard get
longer and longer to turn into a Santa Claus beard by Christmas, then uh,
you know, you gotta get us a few more comments in, uh, specifically in iTunes.
So go to the iTunes, just scroll to the bottom, you know, click on and give us
a, give us a rating, give us a comment.
And, um, if you'd like to be a part of this conversation, please reach
out to me at WC preston on Twitter or w Curtis Preston on gmail.
And, um, so we, we've been doing this back to the basics, uh, series, Jeff.
And, uh, so basically I'm, I'm pulling out, you know, my favorite book, Modern
Modern Data Protection, which you may recall having, having a part in.
And, um, we already did chapter one, it was Why We back up.
Right, Right.
Risks, what are all the things that, that could do damage to data.
and what I really want, you know, we talk a lot about, in the backup space, we
talk a lot about, um, the fact that too many times, those of us in technology,
we focus only on the technology, and then we find ourself asking questions, kind of
like you were talking about the stores.
Well, what should my retention be?
Right?
How long should I keep my backups?
How long, how, how often should I do backups?
Right?
What, So, and you know, and how quickly should I backup and how
quickly should I be able to restore?
And what I'm constantly telling people is, you should not be
the one answering this question.
You should always go back to the business because the business is
why you are doing what you're doing.
And so, Chapter two is about, um, gathering and determining
your service levels.
we have to realize what our organization does, whether it's
a government organization or a business organization, and I think
you always understood that role.
Jeff Rochlin: Um, yeah.
Well, so one of the things that we do as workers at any particular company we
go to is we focus on the job we have to.
Learning everything we need to learn about the job and, and understanding
that so we could be good at it, right?
But very few people started a company and then sit down and say, I wanna
understand the product we produce.
Right?
I wanna understand what the marketing is all about.
I wanna understand everything this business does because I don't just see
myself as a worker for the company.
I see myself as part owner of this business, right?
I'm, I'm a critical cog in the wheel that makes the place produce something.
And so that's always been my approach to my jobs.
It's not good enough to just know I'm a sysadmin.
I have to be able to write scripts to run this Unix box.
Um, I wanna understand that we rent cars to people all
over the, all over the world.
And there's a process involved in how that works or.
When I worked in, in broadcasting, right, I took the time to understand what
the actual broadcast chain is, right?
We create an image, it goes into a bunch of systems before
it goes out over the air.
It passes through a dozen different things that layer different sub carriers on it.
Like your Nielsen ratings box, right?
There's a, there's a subcarrier running on the signal that tweaks
your Nielsen box that says, Hey, he's watching Channel nine, right?
Um, and I wanted to understand all of that right up to the, the
transmitter, because I'm crazy that way.
Um, and what you Yeah, exactly right.
Um, and what, what you find as you become that interested in the business
also is that the people who are running the business part of the business, they
don't know anything about technology.
It could be a technology company.
The guy who's running the business isn't thinking about
how the engine actually operates.
He just wants the engine moving fast.
W. Curtis Preston: It's kind of like, kind of like it in the earlier
part, you know, when I was saying I, I just want my technology to work.
That's the way they are with tech.
Jeff Rochlin: exactly.
Exactly.
And they, and they don't really care about the details.
It's not important to them.
What they care about is they're gonna get what they expect on the outside
of the business right now because it people being who we are, think
that the technology is the do all end, all of what makes the business
work, especially in big corporations.
IT departments tend to do it for the sake of doing it.
Right.
They make systems for everybody else to use that are complicated,
uh, because they work really well inside the machine, right?
They take advantage of faster databases, faster storage, so
that things on the back end, which are complicated work really well.
But then when you and I have to go, you know, fill out our payroll record, you get
this screen that looks like it's something from 1954 off of an IBM mainframe, right?
In a world where.
W. Curtis Preston: possibly is
Jeff Rochlin: Uh, true, but, but you're in a world where everything is,
um, you know, graphical interfaces.
There's no reason why it has to be that way, and that's why I've always
taken it, that you have to understand what the business is doing, and you
have to understand what the people in the business are expecting the
business to do in order to be able to deliver to them effective systems.
W. Curtis Preston: I, I've always had that similar opinion and, and it could
be because my very first technology job when I was at MBNA, the bank that
everybody of a certain level had to spend four hours a month as a customer
support rep talking to cardholders,
Jeff Rochlin: that's cool.
That's really great.
W. Curtis Preston: and, and it really gave you a sense of like,
this is what we were about.
Right?
We are about the cardholder out there spending money that we are lending
them because that's how a credit card company is judged, is the outstanding
balance of everybody added all up, which in that world was 35 billion.
Right?
Which is just staggering to think about it.
Jeff Rochlin: the, the old adage, the customer is always right, is absolutely
true all of the time, even when they're wrong and there are pains in your butt.
Um, because if you're not going in with that attitude, Then people are gonna
say, Well, especially nowadays, right?
When there's a big in different choices, they're gonna say, Well, screw this.
I'll just go somewhere else.
I don't need,
W. Curtis Preston: We talk about building a framework and building review
and advisory boards in that chapter.
What, what is, what is that?
Jeff Rochlin: First of all, um, think about.
A pilot flying an airplane, right?
He's got a checklist that he goes through to make sure that he's
checked all the important critical things about the system, right?
And the reason why he has it written on a checklist is because those are
things you don't wanna forget,
So that's why you build a framework with anything that's important, right?
You wanna make sure you capture all of the things you need to understand
and have them on a list that you can check off so that you don't find out
after you've built the building that, oh, you forgot to dig the foundation.
Um, and, and it just assures that, you know, it's, you build things, right?
So it's, it's, it's that same concept of you measure twice and you cut once, right?
You don't wanna have to go back and have to redo it all again afterwards.
Um,
Prasanna Malaiyandi: question I have for you, Jeff, is I'm guessing the
checklist is kind of what the, like you said, the pilot uses, right?
In a term that, in this case, it would be like the backup admin would use
and they would have their checklist in terms of, Okay, these are the things
I need to make sure that I cover.
But I, and I know we'll talk about this in a minute, but like that's
not necessarily though exactly what you're going to be like word for word.
You're not gonna be asking like the business, okay, what is your rto, Right?
Because I'm guessing that like what you understand, right?
You need some translation between like what's on that checklist
into what you're asking the
Jeff Rochlin: When you're out collecting requirements for what your system
needs to be able to do, you may have to translate the question, right?
You may not go up to the, the CEO of the company and say, What,
what rto are you looking for?
But you may go up to and say, So the building burns down tomorrow.
At what point does it become no longer effective for you to reopen the business.
At what point is it bankruptcy and we're gone versus we can
survive this and move forward.
Right?
And again, that's extreme, but it's translating the question and, and a
checklist is like collecting, uh, the, uh, the requirements for any project, right?
It's.
W. Curtis Preston: Right.
The context that I remember in the chapter was we talked about that
process that we, that we had at that company where you and I worked.
Um, that was, it was a very rigid process for developing a new product.
And it was, you know, there was the,
Jeff Rochlin: we were working together in aerospace and you have to remember
that aerospace is one of those businesses where if you miss a step
in the process, things go bad, right?
Everything has to be measured and, and has to be checked
off on to make sure it's safe.
, Um, they took a lot of that and they've translated that down into their IT
processes, which by the way, are the same things that they give you in ITIL right?
If you're an IT person, which is a 34 letter word in most certainly
creative companies to drop it l um, it teaches you the idea of
things have to be structured right?
There's a, there's a right way to do things and you just need
to try to follow the right.
In the case of the aerospace company, part of that process is make sure that to
keep you on track, check in with people all the time, and make sure that they
agreed that we're still on track, right?
I, I spend a month collecting, um, the project plan and what the
requirements are, and then I send it out the stakeholders that are
gonna need to use this later and say, Just tell me did I get it right?
Is this what you wanted?
And they come back and go, Yay.
Or they come back and go, No.
And then you.
W. Curtis Preston: Right.
Jeff Rochlin: you check in and you do the same thing, right?
You come back and say, Okay, we're moving along this way, and is this still okay?
Um, or did your plans change?
Prasanna Malaiyandi: So totally makes sense having this board that
you go check back with, right?
Figuring out process.
What are some examples of, uh, requirements or
questions you would ask Right.
For specifically around backup, right.
As you're going through this process.
Jeff Rochlin: yeah.
Um, the very first question I always ask, and it's the hardest question
for any business to be able to answer, is, what is acceptable downtime?
Right.
If, if, if something happens, we can't control it, what can you live with?
Right?
I realize it's always gonna be zero.
. But now dig down deep in your heart cuz it's gonna happen and
tell me what's acceptable on that.
Right?
Um, and remember that I can give you as close to zero downtime as humanly
possible, but it's gonna cost you a lot
Prasanna Malaiyandi: That that's, that's exactly what I was gonna ask.
It's like, do you then tell them, Hey, by the way, you won zero downtime,
it's gonna cost you 5 billion.
Right.
Or something like that.
And
Jeff Rochlin: So for me, you always present it in multiple stages, right?
There's the, this is, this is the worst.
You don't want to do this.
This is survival mode, how much it'll cost.
This is.
This is like gold plated, the greatest it'll ever be in the world.
Once you get this, you'll be the happiest person ever.
It's gonna cost you this much.
And then this is probably the answer you want.
This is the middle, this is the price, right?
Um, but you always go back to the business and the business
has to decide how critical is it?
And it, and it could be for this piece of the business, right?
For my, for my call center, I need a hundred percent uptime because
worst case, they can write the order down on paper and we can key 'em in.
Um, whereas, you know, for the database service, if it's down for an hour,
it's down for two hours, maybe that's okay, as long as I can recover it to
within a, a certain period of time,
Prasanna Malaiyandi: And would you say that that's probably
the most important question?
To always start with
Jeff Rochlin: Yes.
I, I believe it is because, um, it drives everything else right.
W. Curtis Preston: The thing is downtime is sort of measured in two D.
Ways, Right?
So there's downtime in terms of like how long the system is down.
There's also downtime in terms of how much data we lost, right?
So we translate that into RTO and rpo, which I, I know we already did a, uh,
a podcast back to basics podcast on, Um, so that, that's the big thing is,
is making sure that whatever it is that we're getting ready to design that.
We're designing it based on the requirements that business
came, you know, they gave us
. Jeff Rochlin: But that's why you have your review board meeting because
you go back and you feed it back to them every once in a while and say,
Right, this is what you asked for.
Correct.
Cause this is what we're gonna build.
Right.
We're good.
And you do it multiple times so that at the end of the day, if something,
Assuming you did everything right and the system performs the way you
expected to perform, and they come back and go, What the hell, man?
You can look at them and go, Six times you came back and said this was acceptable,
W. Curtis Preston: Yeah.
Yeah.
Six times.
you and I both agreed that a six hour RTO was acceptable because a two
hour RTO was gonna cost us 3 billion.
You and I agree, remember that I have it in writing here.
And so the, the fact that this, you know, that we met a six hour RTO is good
Jeff Rochlin: yeah,
Prasanna Malaiyandi: and it's documented.
Jeff Rochlin: yeah, yeah, exactly.
It's, it's, you know, there's a, there's some element of you're
covering your own ass when you're doing it right, But we all have to.
Be sure that we're delivering what the business needs.
And that's why you have to understand what the business does because if
you understand what the business does and somebody comes back and you know,
you're in a business where you're talking to customers constantly and
somebody comes back when you ask them.
Um, what's acceptable and they go, ah, if we're down for a day or two, that's fine.
You know, they're wrong.
You know that they're not telling you the truth.
You know that they're trying to give you the easy answer.
And if you understand what the business does, you can look back
at them and say, But wait a second.
We answer calls 24 hours a day from customers.
You're okay with not answering calls 24 hours a day for those customers?
Uh, at which point they go, Oh, no, no, I don't mean that
W. Curtis Preston: I, I did, I did work with a business once,
Jeff, that had a two week rto.
You know what, You know what it was?
They, they were a paper mill.
I.
And they were like, If the computer sit down for a couple of weeks, we don't care.
Like we're, we're gonna keep cutting down trees.
We're gonna keep turning it into pulp and we're gonna keep making paper.
If, if you can't, if you can't bill anybody, uh, what
They'll still need paper.
We'll be, that was their attitude.
And so for them, a two week RTO was fine, you know?
Uh, but yeah.
But it goes back to the first thing we started with it, it goes with
what kind of business you are.
Right.
Most businesses, you start using the word day
Jeff Rochlin: They're,
W. Curtis Preston: in your rto.
They're not gonna be okay with that.
Jeff Rochlin: If you're in a retail business, you know, and you're down,
you know, you're, you're running a big, uh, online catalog service and you've
got call centers all around the world taking calls from people or websites that
are, you know, taking, uh, selling movie tickets, let's say, which I have a certain
amount of experience within the past.
Right.
And that opening weekend for the next Star Wars movie is coming up and
your systems go down for 10 hours.
You.
Yeah, you're in trouble, right?
Cause you know that that money is going to go somewhere else.
It's not like they're gonna sit there and go, Well, I'll
keep hitting return until I can
W. Curtis Preston: yeah, it's not like they're gonna not
buy tickets for that movie.
Right.
They're gonna, they're gonna do that.
Um, gosh.
You remember when we used to stand in line, Jeff
Jeff Rochlin: Yeah, I do.
You know, there was, there was a certain charm to it, but
W. Curtis Preston: Yeah, I remember standing in line, like, you stood
in line to stand in line, right?
You, you got up there, there's a big line and you're like, you, you know, it's one
o'clock, you want the four o'clock show?
And they're like, Oh, the four o'clock show already sold out.
Okay, so I guess we're doing the seven o'clock show and uh, and
you bought tickets for the seven o'clock show at one o'clock, right?
And then you went and did something with yourself for the next six hours.
Jeff Rochlin: Yeah.
W. Curtis Preston: Anyway, Sorry, two old guys remeniscing about back in the day.
Um, so you, Yeah, you, you, Jeff, in the next section, in the chapter, you
talked about drawing up multiple designs.
I think that you hinted at that already about the, here's the
one you definitely don't want.
Here's the one you're probably not gonna pay for.
And here, you know, here's the Goldilocks zone.
Jeff Rochlin: People want to feel like they have choices.
It's part of just the way life is, right, And everybody wants gold plated, but
they don't want to pay for gold plated.
So the best way to show them is there aren't, you know, I'm not narrow minded.
There are multiple ways you can do this.
You can solve this problem.
And at different cost levels, and I'm gonna leave it to you to decide how much
money you wanna spend based on that.
Right.
Um, and then just from experience, uh, I know that you present three layers
because there, you know, it's, there's a little psychology involved in it.
It's like when you write a budget, there's always something in the budget
that you're willing to cut because you know they're gonna come back no
matter what, and they're gonna ask you to cut something out of the budget.
It's almost like a goodwill kind of thing, right?
Prasanna Malaiyandi: So far we've kind of been talking about like the requirements
are a little bit in stone, right?
Sort of, uh, the business comes, gives you the requirements, you
come back with the proposals, right?
But as we all know, like business needs and requirements change, right?
And.
How do you sort of account for those changes as you're like,
so you had these three options.
They picked the middle option, business requirements change.
Right?
How do you sort of incorporate that?
Because you can't be like, Hey, by the way, we already started on our way
and so now you have to wait 12 months before we can incorporate those in.
Jeff Rochlin: We're agile now, right?
We don't actually.
We don't build projects that we set a two-year goal and then we check
in at the end of year two and say, Here's what we've got to find out.
We got it wrong.
The biggest challenge in that space to me is when you're at
the foundation level, right?
It's like if I'm building a house and we've designed the house and we've
decided the bathroom's gonna go here, the kitchen's gonna go there, right?
I'm running pipes now out of the foundation to go to those places.
You can't come back to.
Three weeks later and say, I've decided to put the bathroom at the totally
opposite end of the house cuz I've already poured the concrete and that
pipe is not gonna move on its own.
Prasanna Malaiyandi: Yeah, well, Or well not move on its own.
And I guess that's the thing, right?
Is it's okay to go back and say, Yeah, this requires sort of like a redesign
or a rethought re-implementation, right?
It's gonna affect your timelines for sure, but if that's what
the business needs, we could.
Jeff Rochlin: So I think it, I, I think the best way you deal with this kind
of thing is in the ordering of how your project is established, right?
You've gotta get the core, the solid infrastructure that isn't
going to change, um, done first.
And you have to make sure the buy off is there and you have to keep
going back and saying, Right, we're going to build this network.
We're going to buy this kind of storage, We're going to buy these kinds of tape.
We can go back and change out systems and how they operate.
We can change schedules, we can add, but if you turn around after
that check is written and say, I don't want to use that anymore.
I can't get you that money back, right?
You've spent it.
So you need to be sure your foundation is solid in what you're trying to
build on, and you need to know that.
Um, features can change, right?
But the core foundation of it has to be once you've started laying it down.
Prasanna Malaiyandi: Yeah.
And I, I guess Curtis, this would probably be your.
Pitch that you would say.
It's like, Hey.
Yeah, but with the SaaS like service, you may not need to put
all that money up front, right.
To pay for something.
Right.
It does give you some amount of flexibility to be like,
W. Curtis Preston: That that dynamic scaling up and down is definitely a
real value of a SaaS product, but like any other product you, you have to take
into account a little bit of the future.
Right.
And you know, the one, the one challenge, I'm gonna say, a backing up to a SaaS
provider is to just make sure that we've, we've calculated in the laws of physics,
Jeff Rochlin: Mm-hmm.
W. Curtis Preston: Because if you, if you've got, if you've
got 10, you know, petabyte and a T1 line, it's not gonna work.
Right?
So as long as we've ca love, as we're not at the upper edge of your bandwidth
utilization, you should be able to grow, uh, significantly on the back end without
having to have a change on the, on the,
Jeff Rochlin: But again, but again, that's the foundation, right?
That's the, that's the core infrastructure that you have to agree on up front.
How much data do I have?
How fast do I need to get it up to that damn service in the cloud?
And do I have enough bandwidth and capability to do that?
Right?
Um, and those are the things that become immutable in the beginning.
You lay 'em down.
You understand them, and then you can build on top of it whatever
you want because you have a foundation that'll support it.
You can't turn around though afterwards and say, Well, I didn't really have
that much network bandwidth, and it turns out that in order to get that much
network bandwidth, I have to have this, pay this guy over here to dig a tunnel
underneath, you know, the, the biggest avenue in town to run a fiber optics to my
building, and I'm not gonna pay for that.
So I guess we'll have to figure out how to do it without the bandwidth.
Right?
That's when it, that's when the tower collapses and none of it's ever.
W. Curtis Preston: Yeah, so I, I'd say that, So we, so
we, we got the requirements.
We've got, um, and we, we've, we've done this iterative process of going
back to the stakeholders to make sure we understand the requirements.
We've built out theoretical system.
The, the next phase that I think is, is really important is to make sure.
The product that we, that we're supposed to be buying does what it
is that it was supposed to do, right?
So we're talking about things like POCs and pilots.
Jeff Rochlin: I, I do the POC early, right.
I do it like before I even, I go back to, if the business comes back to me and
says we need to update backups, I go out and start doing the POC on the backup of
products before I come back and propose here's what I think the solution would be.
Right.
Um, you know, There are the sales guys and there are the sales engineers, right?
Don't listen to anything the sales guys have to say to you.
Cause the sales guys are sales guys.
That's what they do.
Let them buy you lunch and then don't worry about it.
Find yourself a good sales engineer who's attached to the account.
Connect up with them.
They'll give you this straight answer, and then give them your real
world, your real world scenario, that's kind of where I come from.
Right.
If you're gonna, if you're gonna go to the trouble of selling me a service
up in the cloud somewhere, then it should do everything it needs to do.
Right.
And.
Go on a tangent, which I know you'll cut cuz you hate when I go on tangent.
So I'm keeping you from doing other
W. Curtis Preston: It depends on the value of the tangent, Jeff
Jeff Rochlin: um.
When, when people, especially the non-technical people go off
and think of the cloud, right?
They think the cloud is a solution for everything, right?
I want to go store things in the cloud, so I will just copy all of my
data up into the cloud and things will be wonderful, and it's like, No, you
copy your stuff up into the cloud and all you're doing is you're handing it
off to somebody else's data center.
That doesn't mean it's properly backed up.
That doesn't mean it's redundantly stored in multiple locations
in case there's a disaster.
You still need a systems engineer to look at the problem and come up
with a solution within the context of the cloud as if it was your own.
But, but cloud is a marketing term, right?
That we sell to people who don't understand technology about a,
here the solution is there, baby.
Just buy this and, and, and your life will be better and everything will just
W. Curtis Preston: It, You know what you need, Jeff, In that
scenario, you, you need a Prasanna.
That is what Prasanna does.
So, you know, he's actually, he's actually, you know, I'm gonna say he's
more technical than I am, and when we talk about current cloud technology,
uh, he's actually making things happen and actually understands that stuff.
So you need somebody like that.
Like, it's not, like the cloud doesn't just like, Oh,
we don't need any IT people,
Prasanna Malaiyandi: Okay.
W. Curtis Preston: We just need different kinds of IT people.
Jeff Rochlin: Right.
But you would be surprised at the number of C level and especially
financial people out there who think.
By buying into a cloud service, they can throw away their data center and they
no longer need as big an IT department
W. Curtis Preston: Yeah.
Jeff Rochlin: there.
W. Curtis Preston: might, might
need as big, but Yeah.
Jeff Rochlin: Yeah.
you're gonna eliminate certain positions, right?
You're gonna eliminate the guys that are handling your hardware.
But, um, there, there is this perception and that's why everybody
thinks the cloud is cheaper, right?
In many cases it's more expensive than running your own data
center depending on what you're
W. Curtis Preston: It, It, definitely depends on how you implemented it.
Yeah.
Jeff Rochlin: Right.
But,
Prasanna Malaiyandi: talk to Corey Quinn or follow him on Twitter, right?
That's his big thing.
Jeff Rochlin: yeah,
W. Curtis Preston: Yeah.
Jeff Rochlin: so anyway, that was, that was my tangent, but I thought it might
W. Curtis Preston: That was a good tangent, Jeff.
Um, alright, so the, so the final thing here of, of, you know, of, of,
in this chapter was about documenting and implementing the new system, right?
Jeff Rochlin: Yep.
The job, the job that nobody ever wants to have to do is documentation.
W. Curtis Preston: yeah.
Um, you know what,
Jeff Rochlin: You,
W. Curtis Preston: that about?
Right?
Do do I have to, do I have to document?
Jeff Rochlin: Yeah.
You have to.
Sorry.
You know, cuz it's like that's your proof of life, right?
You have, you have it down in writing, you have a plan that you're working from.
It's the checklist, right?
And it's visible to everybody.
You can hand a copy to somebody and say, I want you to read this, and then
I want you to sign on the bottom of it so that I know that you've read
it and you either agree or you don't.
And then if they take it and they just sign it and hand it back
to you when there's something in there that they didn't like or
that they didn't expect, you look at them and go, You signed for it.
Right?
Would you sign a contract without reading it?
Would you, You know,
Prasanna Malaiyandi: all the time.
I agree.
Jeff Rochlin: Written.
The written documentation, especially the design documentation is actually kind of
like a contract with your customer, right?
Your customer being the business.
W. Curtis Preston: right.
Prasanna Malaiyandi: this design document, right?
It, I'm guessing it also sort of restates, Hey, here are all the
requirements that I built against, Right.
That we had agreed to during the sort of the checklist and
those initial discussions.
Right?
And these are sort of the requirements that this design satisfies.
Jeff Rochlin: Yeah.
When you, when you write that initial document once, once everybody has
blessed it and signed off on it, you may never go back to it again.
Right?
You may never have to go back to the printed document, except for the
fact that it's part of your completed set of things you need to have.
That's just there for the future.
Right?
And if you've ever been involved in any kind of construction, right on the back
end of the construction, there are the as-builts, right there is the actual
design document in the blueprints of the building as it was completed by
the contractors who did all the work that you stick in a box somewhere and
only go back to when you're trying to figure out where that damn wire that's
powering your wifi, uh, device is running.
Because that's when you need your as-builts, right?
You need to understand how things are wired.
This becomes the as-built for the project.
It gives you esoteric details that you really don't care about until
for some reason you need to know about it, and then it's there.
W. Curtis Preston: When I, when I think about the value of documentation, I'm
speaking in this case more operational documentation than design documentation,
but I go all the way back to when my oldest daughter was born, right?
I was in Delaware, uh, and uh, I was at the hospital in my wife's delivery room.
Nina was born, She's, you know, brand new baby, and I'm, I'm standing in, you
know, in New Father Bliss Zone, right.
And the phone rings in my wife's hospital room and you know, this
is pre-cellphone days, right?
And, um, I, um, I picked up the phone.
I'm like, Why is this phone, It's the bank on the phone, and I.
What, And they said, Well, well we have this restore, uh, that we need to do.
I'm like, I am in my wife's hospital room.
And they're like, Well, we don't understand abc.
And I said, Well,
Jeff Rochlin: Yeah.
W. Curtis Preston: did you read the documentation?
Is there a problem with the documentation?
They said, Oh, we haven't looked at it.
Slam.
I just slammed down the phone, , I.
The, I had good documentation and that was, you know, I, I was like, I'm good.
Jeff Rochlin: I, I regularly go to developers when they're writing software
that I'm involved in having to support and telling them that, um, because
I run operations as well, right?
And telling them that, um, it is in there enlightened self.
To write good documentation for their product because I will call them at
two o'clock in the morning on Saturday.
If it breaks and there's something broken, not only will I call you, I'm
gonna make sure I have your wife's cell phone number as well, and I
will call her if you choose not to
W. Curtis Preston: I call your mom
Jeff Rochlin: Cause if the, I would, Cause if the business is flat out, down
on its ass and I have the CEO of the company who's going to come looking for
me because I'm responsible for operations.
What am I gonna tell 'em?
Sorry, the software doesn't work.
We'll get back to you on Monday.
Right?
It's like, no, it doesn't work that way.
Um, and that's why everything, the better the documentation is, the
less the operations guys have to go and bother the people who wrote it.
Prasanna Malaiyandi: It just going back to the earlier thread that
I was on, how do you ensure that cuz things will change, right?
The system will change.
The designs will change.
Is that part of the process that any time like you get new
requirements or anything else like.
Always go back and update the documentation.
Jeff Rochlin: Yeah.
Should be, should be.
Without a doubt.
And if it's happening during the development process, you take it
back to the review boards so that again, everybody's aware of it
and everybody checks off on it.
Right.
That they're okay with the change.
That it's not a surprise to finance.
Right.
And it's not a surprise to, uh, facilities operations that you're,
you know, all of a sudden gonna re.
Uh, the, the waterlines or something like that to make it work?
W. Curtis Preston: Yeah, I, I would say it's your job as the technology side
of things to watch what's happening with the business side of things and
to see if there are changes, right?
If all of a sudden to, to use your world.
Jeff, you know, you are normally doing 10 movies at a time and you happen
to be in some quarterly meeting and you hear that, They're like, Listen,
we're gonna double production.
We've, you know, we've had some great sales and we're
gonna do 20 movies at a time.
You're gonna be like, um,
Jeff Rochlin: Yeah.
W. Curtis Preston: Right.
You know, current capacity is based on 10 movies at a time.
I'm gonna need twice as much
Jeff Rochlin: yeah, I guess examples in my life of that were like, um, I was working
at Disney at the time and, uh, I sat in a meeting and they said, Hey, we're gonna,
we're gonna release this one in imax.
You know, sitting there going, Wait, I have a system that's designed to
create two K frames on the back end.
I don't even know what IMAX is at this point.
Or, Hey, we're gonna start doing 3d.
That was like, ah, right.
When it comes to things like storage, right, The transition from,
we're gonna do two k, we're gonna.
4k, we wanna do these special formats.
Oh wait, we're gonna start doing live action visual effects.
That means we've gotta shoot, you know, real film back plates,
uh, that you're gonna put stuff on top
of,
W. Curtis Preston: The point here is that you need to be keeping an
eye to going back, sort of bringing this all back to the beginning.
Your organization, what is its function, right?
This, that idea works for businesses, it works for government, uh, right?
So what is the purpose of your organization and what do you, what
purpose do you serve within that purpose?
Jeff Rochlin: Right.
I can, I can promise you that the head, the, the, the chief creative
officer at Disney is not thinking about the difference in storage capacity
between a two K movie and an 8K movie.
W. Curtis Preston: Yeah, it's not, not their job
Jeff Rochlin: right.
But you have to be listening to him to hear that he just said eight K, and go,
Well, okay, I need to go figure out what that means, and then I need to go find
the right executive who's willing to go ask him if he's willing to spend money on
that or if he was just talking about it.
W. Curtis Preston: This whole thing was about gathering the requirements,
just making sure what your company does, making sure that you are doing
the things that you need to do to, to, to meet those requirements.
Um, and, and, and that, and that you should absolutely not be
determining those requirements.
You should not be determining retention.
Recovery time, objective, rpo, any of that stuff that should
all come back from the business.
Jeff Rochlin: Yeah, be an active participant in your company, right?
It's more, be more at your job than just your job.
Understand what your business does, Um, understand why it's doing it,
what makes it, you know, special.
in whatever it is that it's doing.
And when you understand all that stuff, you'll get to know the people
in the business better because you know, like I said, the guy in
customer service doesn't care about what's going on in the computer room.
But if you get to know the guy in customer service because you understand
what he's up against, all of a sudden you can build that relationship.
That'll make it easier to get the info you need later and then document the hell out
of it and make sure everybody signs off.
So, Accountability is spread evenly across the whole
organization at the end of the day.
Cause you don't want to be the guy that everybody points to and
says, Well, we're down for a week.
Because, you know, he decided that we could be down for a week
and go, No, here's the book.
Right.
And then you document it all through an inch of its life so that a,
you have it as a reference, right?
Cause other people are going to have to operate it if you don't want to
be the only person who gets called at two o'clock in the morning cause they
can't figure out how your system works.
Then you describe to people how your system works so they can figure it out at
two o'clock in the morning without you.
Um, and that, um, all of the parameters are covered so that you can now move
up to be the president of the company and the next guy coming in behind you
will be able to just easily step into.
Past role cuz he can just open the books and it's all there for
W. Curtis Preston: I like it.
Well, thanks Jeff.
Always, always a treat.
Jeff Rochlin: My pleasure guys.
Thanks for having me.
W. Curtis Preston: Thanks.
Prasanna.
I know between Jeff and I, you didn't get a word in edge wise in
Prasanna Malaiyandi: That's okay.
I, I
Jeff Rochlin: He asks good
Prasanna Malaiyandi: to the old man, old men
Jeff Rochlin: Hello?
Prasanna Malaiyandi: By the way, if you wanna listen to the two of
them talk more, they do have their, another podcast, right, where you
W. Curtis Preston: We do, we do
the, the things that entertain us Yes.
Is the name of our other podcast.
Jeff Rochlin: May maybe, maybe Curtis will stick a link in in the, uh, show notes.
Who
W. Curtis Preston: maybe, maybe he will, Maybe he will.
Uh, all right.
And thanks again to our listeners.
We're nothing without you.
Remember to subscribe so that you can restore it all.