Speaker:

You found the backup wrap up your go-to podcast for all things

Speaker:

backup recovery and cyber recovery.

Speaker:

In this episode, we tackle something that a lot of backup

Speaker:

folks get wrong, RTO versus RPO.

Speaker:

You might think this is a simple topic, but I guarantee there's

Speaker:

more here than meets the eye.

Speaker:

We're going to break down why these aren't just backup metrics.

Speaker:

They're business decisions that should come from your stakeholders, not from it.

Speaker:

And if you're thinking, Curtis, how can you spend an entire

Speaker:

episode on these two terms?

Speaker:

Well stick around.

Speaker:

We'll show you why these two concepts drive everything from backup frequency

Speaker:

to the overall system design and why getting them wrong can cost your company.

Speaker:

Tons of money.

Speaker:

This is one you won't wanna miss.

Speaker:

By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr.

Speaker:

Backup, and I've been passionate about backup and recovery for over 30 years.

Speaker:

Ever since.

Speaker:

I had to tell my boss that we had no backups of the

Speaker:

production database we just lost.

Speaker:

I don't want that to happen to you, and that's why I do this.

Speaker:

On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes.

Speaker:

This is the backup wrap up.

Speaker:

Welcome to the show.

Speaker:

If I could ask you to take just a quick second to subscribe or

Speaker:

follow wherever you're watching us.

Speaker:

Remember, you can watch us on YouTube if you wanna see

Speaker:

our bright and smiling faces.

Speaker:

Or if you think I have a face built for radio, then uh, you

Speaker:

can use the podcast format.

Speaker:

But either way, if you follow or or subscribe to us and you'll make

Speaker:

sure you get our great content.

Speaker:

Uh, I am w Curtis Preston, also known as Mr.

Speaker:

Backup, and I have with me a guy who, for some reason today is just a little bit too

Speaker:

bossy Prasanna Malaiyandi, how's it going?

Speaker:

Prasanna.

Speaker:

I am good.

Speaker:

I think it's also a voice made for letters.

Speaker:

Boy, I've never heard that one.

Speaker:

I've heard you got a face for radio buddy.

Speaker:

Um, but, uh, yeah, I've never heard voice made for letters.

Speaker:

You know, there, there's a guy that I know that I spent some time with actually

Speaker:

just last week, and he has a voice for.

Speaker:

Like, um, you know,

Speaker:

basically people are always like, you've got an amazing like, radio voice.

Speaker:

And he's actually, um, he's around my age, maybe even a little bit older, and

Speaker:

he's actually trying to become a, um, like someone who, uh, does audio books,

Speaker:

um, uh, amazingly, you know what his name is.

Speaker:

What.

Speaker:

His name is Will Shakespeare.

Speaker:

This is actually his name.

Speaker:

His name is Will Shakespeare.

Speaker:

I introduced him to another friend of mine, uh, Robert Lewis Stevenson.

Speaker:

And, um,

Speaker:

think you're totally making this up.

Speaker:

No, these are two people.

Speaker:

They're the, I, you, I can show 'em to you on Facebook.

Speaker:

I'm friends with Facebook.

Speaker:

One of 'em I know via the IT world.

Speaker:

The other one I know via the, um, election worker world.

Speaker:

So, uh, we're gonna talk about, um, for some people, you know, when

Speaker:

you saw the title, you're like, how can you talk about this topic for,

Speaker:

you know, a, a podcast episode?

Speaker:

And you know what, this may be one of those times where we're gonna have a 10

Speaker:

minute podcast, but I don't think so.

Speaker:

Yeah, I don't think, I don't, I don't think that has ever happened.

Speaker:

I don't think that will, unless like one of us, like falls ill

Speaker:

in the middle of a podcast.

Speaker:

Yeah, yeah.

Speaker:

We do have the ability to, to Jer Actually, I'd say more me, I

Speaker:

have the ability to Jer Java and you have the ability to humor me.

Speaker:

Um,

Speaker:

Yeah,

Speaker:

so we're talking about

Speaker:

They're actually earmuffs like hearing protectors, not headphones.

Speaker:

I'm just really good at reading lips.

Speaker:

I'm gonna say that this topic, there are few foundational topics

Speaker:

that are more important than the topic that we're talking about

Speaker:

Ooh, I, I,

Speaker:

What do you think?

Speaker:

I, once you introduce the topic, I'm gonna ask about the other one

Speaker:

that may or may not tie this one.

Speaker:

Okay.

Speaker:

I mean, there are things that are very, you know, some, some of things

Speaker:

like, you know, they're like all equal.

Speaker:

But I'll just say this, if you are not factoring in this concept, when

Speaker:

you are designing your backup system and when you're maintaining your

Speaker:

backup system, you just not doing your

Speaker:

job

Speaker:

Well, and, and I think that some people might kind of implicitly be doing some

Speaker:

of this without realizing it's like, you know, sometimes you do things

Speaker:

but you may not use the exact word.

Speaker:

Well, you might not use the word.

Speaker:

And so, so today we're gonna make sure that you know the word and, and, and it's

Speaker:

actually, there's actually four words.

Speaker:

Uh, there, there's two that we're gonna start with, and then we're

Speaker:

gonna add two more on top of that

Speaker:

Can this be like Sesame Street?

Speaker:

This letter is introduced by.

Speaker:

this episode is brought to you by the letters RT.

Speaker:

O and RPO.

Speaker:

That's what we're talking about this week, is recovery time objective

Speaker:

and recovery point objective.

Speaker:

And a lot of people, it's a little bit like, um, the effect and affect.

Speaker:

Uh, people always constantly miss the two.

Speaker:

And uh, I'm one of those people that constantly, when I.

Speaker:

I, I have to think twice.

Speaker:

Yeah.

Speaker:

I have to think like, is it affect?

Speaker:

This is the, the effect.

Speaker:

Um, yeah, like affect is the verb, effect is the noun.

Speaker:

Um, and, um, yeah.

Speaker:

Anyway, welcome to English, but what, why do you think, I think

Speaker:

RTO and RPO are so important?

Speaker:

Uh,

Speaker:

you know, when designing a backup

Speaker:

Because it drives everything, right?

Speaker:

It because it's not only understanding the requirements that are coming

Speaker:

from your stakeholders who hopefully you are talking to, right?

Speaker:

It also, uh, it helps define like what your backup system looks like,

Speaker:

what technologies you use, right?

Speaker:

All the rest of that, all because of these two key phrases.

Speaker:

Now I'm gonna, I'm gonna put something out there, and I'm gonna confess

Speaker:

that in my first several years.

Speaker:

Of working in the backup world.

Speaker:

I never once thought about RTO or RPO and, and when I say that,

Speaker:

I don't mean just the term.

Speaker:

I

Speaker:

mean it's like we, we really didn't, I'd say we focused a little

Speaker:

bit on recovery time, but very little time on the recovery point.

Speaker:

Right.

Speaker:

Um,

Speaker:

you go any further, can we first define what RPO and RTO

Speaker:

are just for our listeners?

Speaker:

Yeah.

Speaker:

Just because you're throwing out these terms and so.

Speaker:

And, and, and, and for those who, for those that struggle with the two.

Speaker:

Right.

Speaker:

And give you a just my, my quick, um.

Speaker:

Thing.

Speaker:

Um, and that is so recovery time objective is the goal that we've set that how

Speaker:

long a recovery time should take, right.

Speaker:

You know, um, from beginning to end, from bad thing happening, which could be

Speaker:

anything from, I deleted a file generally.

Speaker:

Generally, we're talking more about disaster recovery here than

Speaker:

an individual file, but it could be an individual, uh, database.

Speaker:

You delete one file, you take a database down.

Speaker:

That database is behind a really big application and poof, you know,

Speaker:

we're talking about RTO right?

Speaker:

But it's the time that we agree that a system, that it should take the, the

Speaker:

backup and DR system to restore the.

Speaker:

Thing, the entity, whatever it is that has been taken down back to

Speaker:

full, um, fully functional status.

Speaker:

How's that?

Speaker:

I agree with everything except that last sentence you said, and I wanna be very

Speaker:

careful here because a lot of people like this is called a recovery time

Speaker:

objective, not a restore time objective.

Speaker:

A lot of people in backup think it's just getting the data back,

Speaker:

that, well, that's why, that's why I put that

Speaker:

thing at the end.

Speaker:

Yeah.

Speaker:

Right, but it's not just the restore, right?

Speaker:

It's not just getting the data back, right?

Speaker:

It is

Speaker:

yeah.

Speaker:

Yeah,

Speaker:

that entity, which I'm glad you put the entity word right, because

Speaker:

it's more than just the data.

Speaker:

And I think that's an important concept that backup folks should understand is

Speaker:

you're just part of the process, right?

Speaker:

What you do is you're restoring the data, but there still has to

Speaker:

be that recovery of bringing the application back online, bringing all

Speaker:

the other components back up, right?

Speaker:

Doing, um.

Speaker:

Making sure that everything is good to go before you actually

Speaker:

bring up the application.

Speaker:

And all of that is encompassed in the RTO.

Speaker:

Yeah.

Speaker:

It, it, it's literally from the moment that the, the thing is damaged

Speaker:

to the thing is fully functional.

Speaker:

Yeah.

Speaker:

Whatever the thing was,

Speaker:

right?

Speaker:

We're talking.

Speaker:

We could be a data center, could be, um, a server, it could be a database, it

Speaker:

could be a really important file system behind, uh, you know, a database, right?

Speaker:

yeah.

Speaker:

And, but this, but, sorry, I don't mean to be pedantic, but this is

Speaker:

No, it's

Speaker:

just you two.

Speaker:

You mean to be pedantic, but go ahead.

Speaker:

So let me be, so it's really important though, because I think sometimes,

Speaker:

like going back to my initial point, right, that you should talk

Speaker:

to your stakeholders to understand the business objectives, right?

Speaker:

As a backup team, you are only responsible for a piece of the entire recovery.

Speaker:

So make sure when the business says, I have four hours to bring this application,

Speaker:

to recover this application, whatever it is, you're just a part of that, right?

Speaker:

You can't say, oh, I'm gonna take three hours and 59 minutes and I'm good,

Speaker:

Right.

Speaker:

right?

Speaker:

Yeah.

Speaker:

Because that doesn't meet the business objectives.

Speaker:

That's not the case.

Speaker:

Agreed.

Speaker:

So that's the recovery time objective.

Speaker:

And the key there is recovery time.

Speaker:

Right?

Speaker:

Just, you know, when you hear that recovery time, we're talking about the

Speaker:

amount of time of the recovery, right?

Speaker:

Um, and then the next one.

Speaker:

Is a little bit less obvious, I think.

Speaker:

Um, and that is the recovery point objective, which is the amount of data

Speaker:

that we've agreed we can lose as measured.

Speaker:

By time, right?

Speaker:

Um, so we agree that in a recovery we can lose one hour's worth of data, four hours

Speaker:

worth of data, two weeks worth of data, whatever the recovery point objective is.

Speaker:

That is, it's the amount of data that we have as a business.

Speaker:

Go into those stakeholders.

Speaker:

As a business, we've agreed to the amount of data that, the acceptable

Speaker:

amount of data loss as measured by time.

Speaker:

I do agree with your definition, right?

Speaker:

And I've always thought about it as how much data can I lose?

Speaker:

Right?

Speaker:

And it might be okay, I'm good losing 24 hours.

Speaker:

It's not a high I.

Speaker:

Change rate application or it's not a mission critical application versus

Speaker:

my financial database that keeps track of all my transactions that I'm

Speaker:

probably not good with the one day loss.

Speaker:

Right.

Speaker:

That is probably minutes.

Speaker:

Um, it, it's really about.

Speaker:

The RPO is driven by the kind of business you're in, the kind of application

Speaker:

that we're talking about, and it's driven by the amount of revenue that

Speaker:

you would lose when you lose data.

Speaker:

Right.

Speaker:

Um, and there, and, and just the, just the one key thing is that as measured by

Speaker:

time, it's just that normally when we talk about the amount of something in data,

Speaker:

we're talking about number of gigabytes or

Speaker:

terabytes.

Speaker:

In this case we're saying as measured by

Speaker:

time.

Speaker:

Right?

Speaker:

So three hours worth of data,

Speaker:

not

Speaker:

be a small amount or a large amount, and I think depending on the technology

Speaker:

you choose as well, I know you can also get down to fry fine grain, like

Speaker:

outstanding iOS that you are willing to lose in some of the technologies

Speaker:

that you could possibly use.

Speaker:

right.

Speaker:

So that's a definition of RTO and RPO.

Speaker:

And we, we say that this is, um, there's a lot of questions that come up here, right?

Speaker:

But with without those two values, we can't properly design a backup system,

Speaker:

right?

Speaker:

So here's the question.

Speaker:

I'm the backup guy at TI Squad company, and I need to start

Speaker:

designing my backup system.

Speaker:

What RTO and RPO should I be using?

Speaker:

You don't decide that

Speaker:

What

Speaker:

I am the backup guy.

Speaker:

I decide all things backup.

Speaker:

no.

Speaker:

Curtis, no, no, no.

Speaker:

Because no matter what you pick, if you don't communicate and get the

Speaker:

requirements from your stakeholders, I.

Speaker:

They'll be wrong and they will always yell at you, and it's,

Speaker:

they, well they'll, they'll still probably

Speaker:

yeah.

Speaker:

Well,

Speaker:

but, well.

Speaker:

so here's, so here's, here's my version of that answer, and that is, I.

Speaker:

Your RTO and RPO is not a backup decision,

Speaker:

right?

Speaker:

It is a business decision

Speaker:

yep,

Speaker:

and you, you,

Speaker:

you might not even have input on these numbers.

Speaker:

The, the only input you can have is, uh, I can't do that.

Speaker:

Well, well there, I think you have two inputs.

Speaker:

Mm-Hmm

Speaker:

One is I can't do that because it's technically impossible.

Speaker:

The second is I can do that and it's going to cost you a boatload of money,

Speaker:

Yeah, yeah,

Speaker:

yeah.

Speaker:

I think the cost is the other thing that you can come back with,

Speaker:

which might also sway the business.

Speaker:

yeah, so the first thing is that, that the RTO and RPO need to come from on high

Speaker:

and from, you know, side to side, right?

Speaker:

They need to come from you.

Speaker:

You've, you've brought up the word stakeholders many times.

Speaker:

Why don't you define what we mean when we say stakeholders?

Speaker:

Yeah, so stakeholders are people within your company who are either running the

Speaker:

application or who are in the business side who understand the risks with this

Speaker:

data and the importance of this data, and can help quantify how much is acceptable

Speaker:

as an overall company to lose, or the amount of time it takes to recover.

Speaker:

Yeah, and I would say that a lot of times, like, so you talked about people

Speaker:

that are running the application, sometimes the people running the

Speaker:

application don't have any more, I.

Speaker:

Yeah.

Speaker:

Knowledge than you

Speaker:

do.

Speaker:

It's really like the cu the customer of that application.

Speaker:

And by customer I mean the internal customer,

Speaker:

right?

Speaker:

The person who, who, whoever's making these recommendations needs to be

Speaker:

somebody with purse strings, right?

Speaker:

Um, that knows.

Speaker:

The, that knows the business impact of that application knows the, the, the,

Speaker:

the ins and outs of the business and how you make money, how you lose money.

Speaker:

And, um, because they need to be able to say, look, when this application is down,

Speaker:

we're losing a thousand dollars an hour,

Speaker:

whatever it is, right?

Speaker:

We're losing, you know, I, I, I've worked with companies that say that, that they

Speaker:

will lose a million dollars a minute.

Speaker:

Of downtime, right?

Speaker:

That number.

Speaker:

So, so the, how much money will I lose while the system is down?

Speaker:

That's going to drive your RTO.

Speaker:

Right.

Speaker:

If, if we're, if we're losing a, a million dollars a minute that the system is down,

Speaker:

that means I can spend $10 million for a system that's gonna get me back up in

Speaker:

five minutes because, you know, there's a

Speaker:

50% RTO there, or I'm sorry, RROI,

Speaker:

there's a 50, you know, 50% ROI there.

Speaker:

Um,

Speaker:

But on the flip side,

Speaker:

gotta go ahead.

Speaker:

but on the flip side, if you're like, Hey, I'm gonna spend $5 million on

Speaker:

something that's just gonna cost a thousand dollars a day, if it goes

Speaker:

down, uh, you're probably not gonna

Speaker:

You're not exactly, you're not gonna spend, you know, $5 million to get,

Speaker:

you know, to, to get you that system that can get you a five minute RTO.

Speaker:

If, uh, you know, you're, you're only gonna lose a thousand dollars a day,

Speaker:

right?

Speaker:

You're, you're just not gonna do that, right?

Speaker:

So, RTO and RPO, um, are agreeing on the RTO and RPO are gonna

Speaker:

drive, um, the cost of the system.

Speaker:

So, so if, if the amount of money that we lose while the system is down

Speaker:

drives the RTO, what drives the RPO?

Speaker:

How much data you can end up losing,

Speaker:

Yeah, I, I think I, I was asking myself the question in my

Speaker:

head as I was saying it.

Speaker:

It's, it's a difficult question because they're very similar.

Speaker:

Yeah.

Speaker:

So if, if

Speaker:

you're losing a a thousand dollars a minute, I.

Speaker:

Right.

Speaker:

The, if basically your company is, is, um, generating revenue

Speaker:

at a thousand dollars a minute.

Speaker:

If you lost, um, an hour's worth of data, you are going to lose

Speaker:

$60,000.

Speaker:

Yep.

Speaker:

Did I do that math?

Speaker:

Right?

Speaker:

Right.

Speaker:

If you're losing a thousand dollars a minute, your company's gonna lose

Speaker:

$60,000 of business that you had.

Speaker:

Yep.

Speaker:

Um, if you, if you lose an hour's worth of data.

Speaker:

But there's also,

Speaker:

in addition.

Speaker:

RTO also applies with

Speaker:

Yeah.

Speaker:

In addition to the amount of business that your system will or that your business

Speaker:

will lose, uh, either bus business that it had or business that it could have

Speaker:

had, also, someone needs to be able to

Speaker:

come up with the intangible things like damage to brand identity,

Speaker:

right?

Speaker:

Um, also.

Speaker:

Potential future lost data due to this is, this is primarily, well,

Speaker:

it's gonna drive both of them.

Speaker:

Because if you, if you, if you took an order from somebody and

Speaker:

then an order just disappears

Speaker:

and you don't even know who they are,

Speaker:

then um, you know, not only are you gonna not get that business,

Speaker:

you're gonna anger a customer.

Speaker:

Also lawsuits potentially.

Speaker:

And potential lawsuits.

Speaker:

Right?

Speaker:

Exactly.

Speaker:

Right.

Speaker:

That's a good point.

Speaker:

So you have to think about all of these different things.

Speaker:

This is, but again, this is what a business

Speaker:

does.

Speaker:

This is not your job as a backup and Dr person.

Speaker:

This is not your job.

Speaker:

If you're thinking, you're listening to this and you're like, gee, this

Speaker:

all sounds really complicated, and the big responsibility, yes.

Speaker:

It's why it's not yours.

Speaker:

Um, it's, it's the CEO, the CFO, uh, you know, a lot of

Speaker:

people with Cs in their names

Speaker:

management

Speaker:

and then the people that they appoint, uh, under them to, to make these decisions.

Speaker:

Yeah.

Speaker:

were saying, what would

Speaker:

Oh, the risk management team.

Speaker:

Risk management team.

Speaker:

Exactly right.

Speaker:

So if RTO and RPO drive backup design, what parts of backup

Speaker:

design do RTO drive and RPO drive?

Speaker:

So let's, let's do RTO first.

Speaker:

What, What, element of backup design does RTO drive?

Speaker:

It is how you do recoveries.

Speaker:

Def define how, what do, what do you mean?

Speaker:

All right.

Speaker:

Mechanisms that you use for recovering your data.

Speaker:

For example, do you have a standby DR.

Speaker:

Site available and ready?

Speaker:

Do you have data in a replica database?

Speaker:

Are you having to retrieve data from tape?

Speaker:

Yeah, all of these are, I was looking for something maybe a

Speaker:

little higher level than that.

Speaker:

Those are all correct answers.

Speaker:

Um, what I was thinking was, it's gonna deri, it's, it's going to

Speaker:

drive the, the, the design that determines the speed of recovery.

Speaker:

Yeah.

Speaker:

Right,

Speaker:

which everything you just said

Speaker:

determine the speed of the recovery, right?

Speaker:

If you have an aging system with an aging backup server and an aging, an

Speaker:

aging storage, and you're having to do a full recovery, uh, from older systems,

Speaker:

that's going to take a really long time.

Speaker:

But if you've got a cutting edge backup system with failover, Dr.

Speaker:

All of these things, um, then you should be able to do a very quick recovery.

Speaker:

Uh, you, well, you should be able to do a very quick restore,

Speaker:

which will enable to do, enable you to do a very quick recovery.

Speaker:

Um, so RTO primarily DR.

Speaker:

Drives, I'm gonna say the, the power of the backup system, the oomph, the

Speaker:

speed, uh, its ability to restore data quickly, which sometimes the

Speaker:

quickest way to restore data quickly is to restore it before you need it.

Speaker:

Right, and, and to, and to not restore it at all.

Speaker:

There are mechanisms to either restore the data before you need

Speaker:

it or to not restore it at all.

Speaker:

What would be a way to not restore it at all

Speaker:

Do you basically have a copy sitting there that is

Speaker:

or,

Speaker:

available

Speaker:

or

Speaker:

or a

Speaker:

snapshot?

Speaker:

Oh

Speaker:

I know how much you love when I bring that up.

Speaker:

yes.

Speaker:

So if,

Speaker:

Which is like copy sitting there.

Speaker:

well, it's a virtual copy,

Speaker:

Okay.

Speaker:

right?

Speaker:

Um, it's very important to distinguish

Speaker:

that, uh, at least what I would call a real snapshot is

Speaker:

a virtual copy.

Speaker:

Yes.

Speaker:

Um, yeah.

Speaker:

Um, if you're able to, if, if it's, if it's a copy just ready

Speaker:

to go and you're able to just.

Speaker:

Restore as I make quotes in the air by simply going back to an earlier

Speaker:

point in time, um, you are going to get a very quick RTO right?

Speaker:

Yes.

Speaker:

I just wondering also is re,

Speaker:

I'm wondering though if like even fast, like what's even faster than a snapshot?

Speaker:

I don't know what's faster than the

Speaker:

Because with the snapshot you still have to have someone

Speaker:

go and restore the snapshot.

Speaker:

Well,

Speaker:

it's, you're not really doing a restore, right?

Speaker:

I mean, a, a

Speaker:

good snapshot system.

Speaker:

You're just doing pointers, right?

Speaker:

So what do, I'm not sure what

Speaker:

Oh, so I'm thinking more like, um, as an example, clustered high

Speaker:

availability with the DR Replicated copy and automatic failover.

Speaker:

Yeah, yeah, that's all fine, right?

Speaker:

That that's all I, I don't know if that's faster than a snapshot, but

Speaker:

it's very similar.

Speaker:

Basically, you have stuff that's, it's immediately, immediately,

Speaker:

available and ready to go, right?

Speaker:

So there are a number of ways that you can, number of ways

Speaker:

from, from days to seconds.

Speaker:

To, to meet different levels of RTO based on, um, based on what you're

Speaker:

willing to spend and how you're willing to design your backup system.

Speaker:

Right?

Speaker:

Can I bring up a story?

Speaker:

sure.

Speaker:

So I was listening to a podcast on my walk, and it's of this automotive company.

Speaker:

They sell bolts and nuts,

Speaker:

Okay.

Speaker:

right?

Speaker:

Um, and they were talking about how they were, they were a small shop

Speaker:

and they have like 30 people now.

Speaker:

And the guy, the co-founder or the CEO founder, he basically ran it.

Speaker:

And they had a bunch of systems and things like that, and he

Speaker:

was kind of managing everything.

Speaker:

And then they got hit with, uh, ransomware

Speaker:

Yeah.

Speaker:

and they lost all their data.

Speaker:

And then he realized he had a backup, but it was from three months ago.

Speaker:

Right?

Speaker:

So his RPO was three months,

Speaker:

Nice.

Speaker:

Well.

Speaker:

RPO.

Speaker:

No, his RPO never changed his RPA.

Speaker:

We haven't got to that

Speaker:

yet.

Speaker:

Yet.

Speaker:

Well that's what he had.

Speaker:

Yes.

Speaker:

And he basically said though, so they had a bunch of invoices and customers and

Speaker:

luckily they had everything hard copy.

Speaker:

He said it took them six months to manually enter, to manually enter

Speaker:

all the transactions that they were missing before those three months.

Speaker:

At least he was able to get, uh, to put that data back,

Speaker:

right.

Speaker:

Yeah.

Speaker:

Uh, by the way, this, this is just as good a time as any to bring up the

Speaker:

terms RPA and our, our RTA and RPA.

Speaker:

So recovery time actual and recovery point actual.

Speaker:

So a lot of people say, you know, and I do it, I do it occasionally,

Speaker:

so it's not like I'm, I'm

Speaker:

calling outy or Prasanna.

Speaker:

Um, we say, oh, we, you know, we recovered it in two hours.

Speaker:

So we had a two hour RTO.

Speaker:

The So recovery, it's recovery point objective, recovery time objective.

Speaker:

This is the goal.

Speaker:

This is the service level agreement, right?

Speaker:

The

Speaker:

SLA,

Speaker:

Oh.

Speaker:

So I guess in my, in that example case, they had no agreement because they did

Speaker:

yeah, that.

Speaker:

That's actually the most

Speaker:

common.

Speaker:

That's the most common, right?

Speaker:

No, no.

Speaker:

SLA, no, R-G-O-R-P-O.

Speaker:

We just do our best case and, um.

Speaker:

But how, how fast the system is actually able to do something is your RTA and RPA.

Speaker:

Right.

Speaker:

Your recovery to recovery time.

Speaker:

Actual and recovery point actual, uh, I've also in, in various times in my

Speaker:

career, referred to that as the RTR and R-T-R-P-R recovery time reality.

Speaker:

Yeah.

Speaker:

Um,

Speaker:

but

Speaker:

that?

Speaker:

How do you actually

Speaker:

figure out

Speaker:

the only way to calculate is to actually do it right.

Speaker:

So hopefully everyone is doing restore testing

Speaker:

or recovery

Speaker:

story testing is the only way to know for re for sure.

Speaker:

All right, so let's talk about, you talked about how, what, uh, aspects of

Speaker:

backup design are determined by RTO.

Speaker:

What this answer is much easier.

Speaker:

What aspect of backup design is determined or is Yes, is determined by RPO?

Speaker:

How frequently can you back up your data?

Speaker:

Yeah.

Speaker:

So if you're backing up your data more often, you are going

Speaker:

to have a better RPA, right?

Speaker:

Um, so if you have, if, what's the most common, uh, backup

Speaker:

frequency, would you say?

Speaker:

Probably once a day.

Speaker:

Once a day.

Speaker:

And so the most common RPA is at minimum 24 hours.

Speaker:

Well, the, the, you know, there are situations where

Speaker:

you might get better than that

Speaker:

if you, like you said earlier, if you had a failure right after

Speaker:

you just finished a backup,

Speaker:

but worst

Speaker:

um, you might, that's best case

Speaker:

scenario, worst case scenario.

Speaker:

Is more like 36 hours because you're, you're, you know, at the end of a

Speaker:

day before the, you know, and, um.

Speaker:

The, and, and that, that's assuming that last night's backup worked,

Speaker:

and I've done enough backups to know that that isn't always the case.

Speaker:

And I would also say I would apply Murphy's Law of backups, which is the

Speaker:

the degree to which, you know, a ba the, the, the possibility that a backup is

Speaker:

successful is inversely proportional to, um, how, how much you need that backup.

Speaker:

Yeah.

Speaker:

Is there also a Schrodinger's cat for backups?

Speaker:

Yeah.

Speaker:

Yeah,

Speaker:

exist, or you don't know if a backup exists or not until you actually,

Speaker:

Until you actually look at it.

Speaker:

yeah,

Speaker:

Um,

Speaker:

I think we need that on a T-shirt.

Speaker:

By the way, if you would buy a T-shirt from

Speaker:

the backup wrap up, right, with these phrases on it.

Speaker:

Let Curtis know on X or LinkedIn or somewhere, or leave a

Speaker:

Yeah,

Speaker:

on YouTube.

Speaker:

We've been getting a lot of YouTube comments lately.

Speaker:

Um, yeah, let me know on YouTube, by the way, did I tell you, you know

Speaker:

how it's how back when I was not feeling well or I hurt myself and

Speaker:

I said, no one cares.

Speaker:

Then I said, if anyone cares, please say something.

Speaker:

We got comments.

Speaker:

One of,

Speaker:

you know what?

Speaker:

One of them said,

Speaker:

What?

Speaker:

India cares about Curtis.

Speaker:

Aw.

Speaker:

That's what I got.

Speaker:

Um, so.

Speaker:

Yeah, so RPO basically determines your backup frequency.

Speaker:

If, if you've got an RPO measured in hours, you can't have a backup frequency

Speaker:

measured in, in, you know, many hours.

Speaker:

Right?

Speaker:

Um, and the, um, I, I will say that,

Speaker:

just talk about the different ways that people do backups, sort of normal.

Speaker:

Regular, what I would call full file incremental backups.

Speaker:

Pretty much the best you're gonna be able to do is daily.

Speaker:

Right.

Speaker:

If you're able to do some type of block level incremental forever,

Speaker:

where each backup, you're only backing up the blocks that have changed,

Speaker:

or even better, the deduplicated blocks that have changed, right?

Speaker:

Because that's gonna be an even smaller, uh, set of data.

Speaker:

Then you could actually do the ba It does two things.

Speaker:

One is it makes the, it makes the backup.

Speaker:

Quicker, which is good.

Speaker:

Um, which generally means that you can do it more often, uh, and then it, it

Speaker:

does allow you, it also reduces the, the impact of the backup, meaning that while

Speaker:

the backup is running, it's not having too much of a, of an impact on production.

Speaker:

And so that allows you to do it more often.

Speaker:

To do it, you know, once an hour is very common.

Speaker:

Uh, if you've got a, a modern backup system, you could do it once an

Speaker:

hour or even more often than that.

Speaker:

I.

Speaker:

Yeah.

Speaker:

While you're mentioning this about the incrementals and chain block tracking,

Speaker:

one thing I realized we didn't cover is depending on what technologies you use,

Speaker:

it will have an impact on your RTO because if you have say, multiple, uh, recovery

Speaker:

points, you have to recover from first before you can get to that final point.

Speaker:

Like if you have to recover a full plus multiple incrementals.

Speaker:

That is going to take you more time, which will impact your RTO as well.

Speaker:

Absolutely.

Speaker:

Um, I do want to talk, so what we'll round out this discussion with is that

Speaker:

idea that you brought up earlier, which

Speaker:

is the,

Speaker:

that, um, every stakeholder is gonna want an RTO and an RPO of zero.

Speaker:

Because if you ask people, if you ask it like this, how much data.

Speaker:

You know, how, how long are we allowed to be down and how much

Speaker:

data are we allowed to lose?

Speaker:

And the answer is zero and zero.

Speaker:

Yep.

Speaker:

How do we get from there to something that's more reasonable and possible?

Speaker:

$1 billion.

Speaker:

Yeah.

Speaker:

What, so yeah, so you go back, you're like, okay, all right.

Speaker:

We've taken the requirements, we've gone and we've met with the powers

Speaker:

that be, and we've decided that if we go with a fully redundant system.

Speaker:

Uh, with, you know, with, with, uh, synchronous replication and a hot site

Speaker:

that's, you know, this far away we can do, we can give you the RTO and an RPO of

Speaker:

zero and it's going to cost $1 billion.

Speaker:

Right.

Speaker:

And you do that with a straight face, right?

Speaker:

You just go, okay, okay, we got, you know, we got a thing, we got a thing, we got the

Speaker:

vendor, we got this vendor standing by to

Speaker:

take your check.

Speaker:

And uh, we can do it for $1 billion.

Speaker:

And then, um, um.

Speaker:

What I generally find is at that point, obviously I have a Plan B and a plan C,

Speaker:

and uh, I've also consulted with this vendor.

Speaker:

And this vendor says that they can do an RTO and an RPO of

Speaker:

four hours for only $1 million.

Speaker:

$1 million.

Speaker:

Sounds a lot.

Speaker:

Sounds like a lot,

Speaker:

but it's a better than a billion.

Speaker:

it to $1

Speaker:

So, so here's a question.

Speaker:

So as a backup admin, when you're going to these discussions with the stakeholders,

Speaker:

Mm-Hmm.

Speaker:

is it.

Speaker:

Worthwhile to go with a couple options, like you mentioned where you're like,

Speaker:

and what would those common options be?

Speaker:

Right.

Speaker:

Do you think it's zero?

Speaker:

Zero, right?

Speaker:

It's a four hour RPO.

Speaker:

Maybe it's a one day RPO.

Speaker:

Like

Speaker:

Yeah.

Speaker:

Um,

Speaker:

what?

Speaker:

What would make sense?

Speaker:

Yeah.

Speaker:

I think it's going to be driven.

Speaker:

Most backup designs.

Speaker:

In fact, I'm gonna say all backup designs are driven by cost and reality,

Speaker:

right?

Speaker:

So, you know, they want zero and zero.

Speaker:

Or let's say, let's say they want 15 minutes.

Speaker:

We're like, look, we can do 15 minutes, 15 minutes costs this much.

Speaker:

This is the way to do it.

Speaker:

Um, if they, if they say.

Speaker:

We wanna, we wanna, you know, a very tight RTO, but we're okay with

Speaker:

an hour's loss, data loss, right?

Speaker:

Snapshots and replication.

Speaker:

Great way to do that.

Speaker:

If they want very, you know, like as close to zero as possible, you've really

Speaker:

got to do CDP or something like it.

Speaker:

You gotta do synchronous replication.

Speaker:

And by the way, it's important to mention here that not everything has

Speaker:

to have the same RTO and RPO in fact.

Speaker:

In your environment?

Speaker:

Everything should in your environment, everything should definitely not have the

Speaker:

same RTO and RPO.

Speaker:

If that's the case, it means you've sort of settled you.

Speaker:

You've definitely got an application that needs more than that.

Speaker:

Uh, and you settled for simplicity, which is very common for people to do, but

Speaker:

that's not really the right thing to do.

Speaker:

You've got one or two applications within your environment that would really lose

Speaker:

a lot of money, and it's totally okay to have a different backup design for them.

Speaker:

Right.

Speaker:

You have the gold, silver, bronze, right?

Speaker:

Um, I, I definitely like, go ahead.

Speaker:

W, and I know you mentioned backup technology, like a different backup

Speaker:

technology that might also drive having different backup vendors as

Speaker:

well to satisfy these different needs.

Speaker:

Yes, it definitely will be.

Speaker:

I mean, my, as a

Speaker:

backup person, my preference is always one vendor if possible.

Speaker:

Um, and, um, you know, one design with options, uh,

Speaker:

because again, simplicity, um,

Speaker:

means recovery, I think.

Speaker:

Um, but.

Speaker:

but.

Speaker:

yeah, it, it is really important to understand that it's very common

Speaker:

for you to have multiple RTOs and RPOs within an environment.

Speaker:

I will also state, and this is somewhat controversial and not

Speaker:

everybody agrees with me here, it's also okay to have different RTOs

Speaker:

and RPOs based on scenario, right?

Speaker:

Like if we've got, um, if a server died or a server caught on fire or a server

Speaker:

got whatever, you know, server goes, poof.

Speaker:

seems to me that that, that, that RTO and RPO, um, would be

Speaker:

much, much shorter than if an asteroid hit Oceanside, right?

Speaker:

Um,

Speaker:

it's just there are different levels of disasters and there's different amounts

Speaker:

of understanding that you can have.

Speaker:

So

Speaker:

I think it's totally okay to have different kinds of RTOs

Speaker:

and RPOs for different kinds of

Speaker:

and, and I think this is where we kind of like from talking to Mike

Speaker:

about our ransomware recovery and other things like that, right?

Speaker:

I think this is where sort of the incident response plans come in, right?

Speaker:

Where different scenarios you may have different RTOs and RPOs.

Speaker:

Because they are different and there are things that are outside of your

Speaker:

control, like recovering from ransomware is very different because of the initial

Speaker:

steps you have to do to isolate and figure out what's going on, rather

Speaker:

than, oh, this server, the disc died.

Speaker:

Now I need to recover.

Speaker:

Or someone accidentally deleted a file.

Speaker:

Yeah.

Speaker:

And by the way, you know, speaking of recovering from ransomware, you know

Speaker:

when, when we've talked about that, when you get to the actual recovery, it's

Speaker:

very common for a person who knows what they're talking about in that world.

Speaker:

Say, listen, I want you to, you might be recovering the same data set 50 different

Speaker:

times from 50 different data points.

Speaker:

That's something you really gotta figure out in your backup design.

Speaker:

Right.

Speaker:

Because there are backup designs where that is not gonna happen anytime soon.

Speaker:

Right.

Speaker:

Um, but, uh, well, hey, once again, uh, we did not do a 10 minute episode.

Speaker:

Yay.

Speaker:

And we're all alive.

Speaker:

But hold on.

Speaker:

So going back to my initial question I had, or which is, I know you said RTO

Speaker:

and RPO are probably the most important things, foundational things in backup

Speaker:

yeah,

Speaker:

system design.

Speaker:

What did you think?

Speaker:

Do you think that second from that would be the 3, 2, 1 rule?

Speaker:

Well, the 3, 2, 1 rule is, um,

Speaker:

it is also foundational, right?

Speaker:

Um,

Speaker:

and.

Speaker:

I, again, is it more important?

Speaker:

Is it if you don't, if you don't follow the 3, 2, 1 rule and all your backups

Speaker:

are all in the same place as the production, why are they're not even

Speaker:

Back up.

Speaker:

Yep.

Speaker:

Yeah.

Speaker:

Agreed.

Speaker:

Maybe next episode, should we, we should dive down into 3, 2, 1 rule.

Speaker:

We haven't done that in a while.

Speaker:

Yep.

Speaker:

All right.

Speaker:

Prasanna, bossy mc.

Speaker:

Bossy

Speaker:

man.

Speaker:

Um.

Speaker:

my bossy pants,

Speaker:

I.

Speaker:

I hope you folks have enjoyed our episode this week on RTO versus RPO.

Speaker:

That is a wrap.

Speaker:

The backup wrap up is written, recorded and produced by me w Curtis Preston.

Speaker:

If you need backup or Dr.

Speaker:

Consulting content generation or expert witness work,

Speaker:

check out backup central.com.

Speaker:

You can also find links from my O'Reilly Books on the same website.

Speaker:

Remember, this is an independent podcast and any opinions that you

Speaker:

hear are those of the speaker.

Speaker:

And not necessarily an employer.

Speaker:

Thanks for listening.