You found the backup wrap up your go-to podcast for all things
Speaker:backup recovery and cyber recovery.
Speaker:In this episode, we tackle something that a lot of backup
Speaker:folks get wrong, RTO versus RPO.
Speaker:You might think this is a simple topic, but I guarantee there's
Speaker:more here than meets the eye.
Speaker:We're going to break down why these aren't just backup metrics.
Speaker:They're business decisions that should come from your stakeholders, not from it.
Speaker:And if you're thinking, Curtis, how can you spend an entire
Speaker:episode on these two terms?
Speaker:Well stick around.
Speaker:We'll show you why these two concepts drive everything from backup frequency
Speaker:to the overall system design and why getting them wrong can cost your company.
Speaker:Tons of money.
Speaker:This is one you won't wanna miss.
Speaker:By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr.
Speaker:Backup, and I've been passionate about backup and recovery for over 30 years.
Speaker:Ever since.
Speaker:I had to tell my boss that we had no backups of the
Speaker:production database we just lost.
Speaker:I don't want that to happen to you, and that's why I do this.
Speaker:On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes.
Speaker:This is the backup wrap up.
Speaker:Welcome to the show.
Speaker:If I could ask you to take just a quick second to subscribe or
Speaker:follow wherever you're watching us.
Speaker:Remember, you can watch us on YouTube if you wanna see
Speaker:our bright and smiling faces.
Speaker:Or if you think I have a face built for radio, then uh, you
Speaker:can use the podcast format.
Speaker:But either way, if you follow or or subscribe to us and you'll make
Speaker:sure you get our great content.
Speaker:Uh, I am w Curtis Preston, also known as Mr.
Speaker:Backup, and I have with me a guy who, for some reason today is just a little bit too
Speaker:bossy Prasanna Malaiyandi, how's it going?
Speaker:Prasanna.
Speaker:I am good.
Speaker:I think it's also a voice made for letters.
Speaker:Boy, I've never heard that one.
Speaker:I've heard you got a face for radio buddy.
Speaker:Um, but, uh, yeah, I've never heard voice made for letters.
Speaker:You know, there, there's a guy that I know that I spent some time with actually
Speaker:just last week, and he has a voice for.
Speaker:Like, um, you know,
Speaker:basically people are always like, you've got an amazing like, radio voice.
Speaker:And he's actually, um, he's around my age, maybe even a little bit older, and
Speaker:he's actually trying to become a, um, like someone who, uh, does audio books,
Speaker:um, uh, amazingly, you know what his name is.
Speaker:What.
Speaker:His name is Will Shakespeare.
Speaker:This is actually his name.
Speaker:His name is Will Shakespeare.
Speaker:I introduced him to another friend of mine, uh, Robert Lewis Stevenson.
Speaker:And, um,
Speaker:think you're totally making this up.
Speaker:No, these are two people.
Speaker:They're the, I, you, I can show 'em to you on Facebook.
Speaker:I'm friends with Facebook.
Speaker:One of 'em I know via the IT world.
Speaker:The other one I know via the, um, election worker world.
Speaker:So, uh, we're gonna talk about, um, for some people, you know, when
Speaker:you saw the title, you're like, how can you talk about this topic for,
Speaker:you know, a, a podcast episode?
Speaker:And you know what, this may be one of those times where we're gonna have a 10
Speaker:minute podcast, but I don't think so.
Speaker:Yeah, I don't think, I don't, I don't think that has ever happened.
Speaker:I don't think that will, unless like one of us, like falls ill
Speaker:in the middle of a podcast.
Speaker:Yeah, yeah.
Speaker:We do have the ability to, to Jer Actually, I'd say more me, I
Speaker:have the ability to Jer Java and you have the ability to humor me.
Speaker:Um,
Speaker:Yeah,
Speaker:so we're talking about
Speaker:They're actually earmuffs like hearing protectors, not headphones.
Speaker:I'm just really good at reading lips.
Speaker:I'm gonna say that this topic, there are few foundational topics
Speaker:that are more important than the topic that we're talking about
Speaker:Ooh, I, I,
Speaker:What do you think?
Speaker:I, once you introduce the topic, I'm gonna ask about the other one
Speaker:that may or may not tie this one.
Speaker:Okay.
Speaker:I mean, there are things that are very, you know, some, some of things
Speaker:like, you know, they're like all equal.
Speaker:But I'll just say this, if you are not factoring in this concept, when
Speaker:you are designing your backup system and when you're maintaining your
Speaker:backup system, you just not doing your
Speaker:job
Speaker:Well, and, and I think that some people might kind of implicitly be doing some
Speaker:of this without realizing it's like, you know, sometimes you do things
Speaker:but you may not use the exact word.
Speaker:Well, you might not use the word.
Speaker:And so, so today we're gonna make sure that you know the word and, and, and it's
Speaker:actually, there's actually four words.
Speaker:Uh, there, there's two that we're gonna start with, and then we're
Speaker:gonna add two more on top of that
Speaker:Can this be like Sesame Street?
Speaker:This letter is introduced by.
Speaker:this episode is brought to you by the letters RT.
Speaker:O and RPO.
Speaker:That's what we're talking about this week, is recovery time objective
Speaker:and recovery point objective.
Speaker:And a lot of people, it's a little bit like, um, the effect and affect.
Speaker:Uh, people always constantly miss the two.
Speaker:And uh, I'm one of those people that constantly, when I.
Speaker:I, I have to think twice.
Speaker:Yeah.
Speaker:I have to think like, is it affect?
Speaker:This is the, the effect.
Speaker:Um, yeah, like affect is the verb, effect is the noun.
Speaker:Um, and, um, yeah.
Speaker:Anyway, welcome to English, but what, why do you think, I think
Speaker:RTO and RPO are so important?
Speaker:Uh,
Speaker:you know, when designing a backup
Speaker:Because it drives everything, right?
Speaker:It because it's not only understanding the requirements that are coming
Speaker:from your stakeholders who hopefully you are talking to, right?
Speaker:It also, uh, it helps define like what your backup system looks like,
Speaker:what technologies you use, right?
Speaker:All the rest of that, all because of these two key phrases.
Speaker:Now I'm gonna, I'm gonna put something out there, and I'm gonna confess
Speaker:that in my first several years.
Speaker:Of working in the backup world.
Speaker:I never once thought about RTO or RPO and, and when I say that,
Speaker:I don't mean just the term.
Speaker:I
Speaker:mean it's like we, we really didn't, I'd say we focused a little
Speaker:bit on recovery time, but very little time on the recovery point.
Speaker:Right.
Speaker:Um,
Speaker:you go any further, can we first define what RPO and RTO
Speaker:are just for our listeners?
Speaker:Yeah.
Speaker:Just because you're throwing out these terms and so.
Speaker:And, and, and, and for those who, for those that struggle with the two.
Speaker:Right.
Speaker:And give you a just my, my quick, um.
Speaker:Thing.
Speaker:Um, and that is so recovery time objective is the goal that we've set that how
Speaker:long a recovery time should take, right.
Speaker:You know, um, from beginning to end, from bad thing happening, which could be
Speaker:anything from, I deleted a file generally.
Speaker:Generally, we're talking more about disaster recovery here than
Speaker:an individual file, but it could be an individual, uh, database.
Speaker:You delete one file, you take a database down.
Speaker:That database is behind a really big application and poof, you know,
Speaker:we're talking about RTO right?
Speaker:But it's the time that we agree that a system, that it should take the, the
Speaker:backup and DR system to restore the.
Speaker:Thing, the entity, whatever it is that has been taken down back to
Speaker:full, um, fully functional status.
Speaker:How's that?
Speaker:I agree with everything except that last sentence you said, and I wanna be very
Speaker:careful here because a lot of people like this is called a recovery time
Speaker:objective, not a restore time objective.
Speaker:A lot of people in backup think it's just getting the data back,
Speaker:that, well, that's why, that's why I put that
Speaker:thing at the end.
Speaker:Yeah.
Speaker:Right, but it's not just the restore, right?
Speaker:It's not just getting the data back, right?
Speaker:It is
Speaker:yeah.
Speaker:Yeah,
Speaker:that entity, which I'm glad you put the entity word right, because
Speaker:it's more than just the data.
Speaker:And I think that's an important concept that backup folks should understand is
Speaker:you're just part of the process, right?
Speaker:What you do is you're restoring the data, but there still has to
Speaker:be that recovery of bringing the application back online, bringing all
Speaker:the other components back up, right?
Speaker:Doing, um.
Speaker:Making sure that everything is good to go before you actually
Speaker:bring up the application.
Speaker:And all of that is encompassed in the RTO.
Speaker:Yeah.
Speaker:It, it, it's literally from the moment that the, the thing is damaged
Speaker:to the thing is fully functional.
Speaker:Yeah.
Speaker:Whatever the thing was,
Speaker:right?
Speaker:We're talking.
Speaker:We could be a data center, could be, um, a server, it could be a database, it
Speaker:could be a really important file system behind, uh, you know, a database, right?
Speaker:yeah.
Speaker:And, but this, but, sorry, I don't mean to be pedantic, but this is
Speaker:No, it's
Speaker:just you two.
Speaker:You mean to be pedantic, but go ahead.
Speaker:So let me be, so it's really important though, because I think sometimes,
Speaker:like going back to my initial point, right, that you should talk
Speaker:to your stakeholders to understand the business objectives, right?
Speaker:As a backup team, you are only responsible for a piece of the entire recovery.
Speaker:So make sure when the business says, I have four hours to bring this application,
Speaker:to recover this application, whatever it is, you're just a part of that, right?
Speaker:You can't say, oh, I'm gonna take three hours and 59 minutes and I'm good,
Speaker:Right.
Speaker:right?
Speaker:Yeah.
Speaker:Because that doesn't meet the business objectives.
Speaker:That's not the case.
Speaker:Agreed.
Speaker:So that's the recovery time objective.
Speaker:And the key there is recovery time.
Speaker:Right?
Speaker:Just, you know, when you hear that recovery time, we're talking about the
Speaker:amount of time of the recovery, right?
Speaker:Um, and then the next one.
Speaker:Is a little bit less obvious, I think.
Speaker:Um, and that is the recovery point objective, which is the amount of data
Speaker:that we've agreed we can lose as measured.
Speaker:By time, right?
Speaker:Um, so we agree that in a recovery we can lose one hour's worth of data, four hours
Speaker:worth of data, two weeks worth of data, whatever the recovery point objective is.
Speaker:That is, it's the amount of data that we have as a business.
Speaker:Go into those stakeholders.
Speaker:As a business, we've agreed to the amount of data that, the acceptable
Speaker:amount of data loss as measured by time.
Speaker:I do agree with your definition, right?
Speaker:And I've always thought about it as how much data can I lose?
Speaker:Right?
Speaker:And it might be okay, I'm good losing 24 hours.
Speaker:It's not a high I.
Speaker:Change rate application or it's not a mission critical application versus
Speaker:my financial database that keeps track of all my transactions that I'm
Speaker:probably not good with the one day loss.
Speaker:Right.
Speaker:That is probably minutes.
Speaker:Um, it, it's really about.
Speaker:The RPO is driven by the kind of business you're in, the kind of application
Speaker:that we're talking about, and it's driven by the amount of revenue that
Speaker:you would lose when you lose data.
Speaker:Right.
Speaker:Um, and there, and, and just the, just the one key thing is that as measured by
Speaker:time, it's just that normally when we talk about the amount of something in data,
Speaker:we're talking about number of gigabytes or
Speaker:terabytes.
Speaker:In this case we're saying as measured by
Speaker:time.
Speaker:Right?
Speaker:So three hours worth of data,
Speaker:not
Speaker:be a small amount or a large amount, and I think depending on the technology
Speaker:you choose as well, I know you can also get down to fry fine grain, like
Speaker:outstanding iOS that you are willing to lose in some of the technologies
Speaker:that you could possibly use.
Speaker:right.
Speaker:So that's a definition of RTO and RPO.
Speaker:And we, we say that this is, um, there's a lot of questions that come up here, right?
Speaker:But with without those two values, we can't properly design a backup system,
Speaker:right?
Speaker:So here's the question.
Speaker:I'm the backup guy at TI Squad company, and I need to start
Speaker:designing my backup system.
Speaker:What RTO and RPO should I be using?
Speaker:You don't decide that
Speaker:What
Speaker:I am the backup guy.
Speaker:I decide all things backup.
Speaker:no.
Speaker:Curtis, no, no, no.
Speaker:Because no matter what you pick, if you don't communicate and get the
Speaker:requirements from your stakeholders, I.
Speaker:They'll be wrong and they will always yell at you, and it's,
Speaker:they, well they'll, they'll still probably
Speaker:yeah.
Speaker:Well,
Speaker:but, well.
Speaker:so here's, so here's, here's my version of that answer, and that is, I.
Speaker:Your RTO and RPO is not a backup decision,
Speaker:right?
Speaker:It is a business decision
Speaker:yep,
Speaker:and you, you,
Speaker:you might not even have input on these numbers.
Speaker:The, the only input you can have is, uh, I can't do that.
Speaker:Well, well there, I think you have two inputs.
Speaker:Mm-Hmm
Speaker:One is I can't do that because it's technically impossible.
Speaker:The second is I can do that and it's going to cost you a boatload of money,
Speaker:Yeah, yeah,
Speaker:yeah.
Speaker:I think the cost is the other thing that you can come back with,
Speaker:which might also sway the business.
Speaker:yeah, so the first thing is that, that the RTO and RPO need to come from on high
Speaker:and from, you know, side to side, right?
Speaker:They need to come from you.
Speaker:You've, you've brought up the word stakeholders many times.
Speaker:Why don't you define what we mean when we say stakeholders?
Speaker:Yeah, so stakeholders are people within your company who are either running the
Speaker:application or who are in the business side who understand the risks with this
Speaker:data and the importance of this data, and can help quantify how much is acceptable
Speaker:as an overall company to lose, or the amount of time it takes to recover.
Speaker:Yeah, and I would say that a lot of times, like, so you talked about people
Speaker:that are running the application, sometimes the people running the
Speaker:application don't have any more, I.
Speaker:Yeah.
Speaker:Knowledge than you
Speaker:do.
Speaker:It's really like the cu the customer of that application.
Speaker:And by customer I mean the internal customer,
Speaker:right?
Speaker:The person who, who, whoever's making these recommendations needs to be
Speaker:somebody with purse strings, right?
Speaker:Um, that knows.
Speaker:The, that knows the business impact of that application knows the, the, the,
Speaker:the ins and outs of the business and how you make money, how you lose money.
Speaker:And, um, because they need to be able to say, look, when this application is down,
Speaker:we're losing a thousand dollars an hour,
Speaker:whatever it is, right?
Speaker:We're losing, you know, I, I, I've worked with companies that say that, that they
Speaker:will lose a million dollars a minute.
Speaker:Of downtime, right?
Speaker:That number.
Speaker:So, so the, how much money will I lose while the system is down?
Speaker:That's going to drive your RTO.
Speaker:Right.
Speaker:If, if we're, if we're losing a, a million dollars a minute that the system is down,
Speaker:that means I can spend $10 million for a system that's gonna get me back up in
Speaker:five minutes because, you know, there's a
Speaker:50% RTO there, or I'm sorry, RROI,
Speaker:there's a 50, you know, 50% ROI there.
Speaker:Um,
Speaker:But on the flip side,
Speaker:gotta go ahead.
Speaker:but on the flip side, if you're like, Hey, I'm gonna spend $5 million on
Speaker:something that's just gonna cost a thousand dollars a day, if it goes
Speaker:down, uh, you're probably not gonna
Speaker:You're not exactly, you're not gonna spend, you know, $5 million to get,
Speaker:you know, to, to get you that system that can get you a five minute RTO.
Speaker:If, uh, you know, you're, you're only gonna lose a thousand dollars a day,
Speaker:right?
Speaker:You're, you're just not gonna do that, right?
Speaker:So, RTO and RPO, um, are agreeing on the RTO and RPO are gonna
Speaker:drive, um, the cost of the system.
Speaker:So, so if, if the amount of money that we lose while the system is down
Speaker:drives the RTO, what drives the RPO?
Speaker:How much data you can end up losing,
Speaker:Yeah, I, I think I, I was asking myself the question in my
Speaker:head as I was saying it.
Speaker:It's, it's a difficult question because they're very similar.
Speaker:Yeah.
Speaker:So if, if
Speaker:you're losing a a thousand dollars a minute, I.
Speaker:Right.
Speaker:The, if basically your company is, is, um, generating revenue
Speaker:at a thousand dollars a minute.
Speaker:If you lost, um, an hour's worth of data, you are going to lose
Speaker:$60,000.
Speaker:Yep.
Speaker:Did I do that math?
Speaker:Right?
Speaker:Right.
Speaker:If you're losing a thousand dollars a minute, your company's gonna lose
Speaker:$60,000 of business that you had.
Speaker:Yep.
Speaker:Um, if you, if you lose an hour's worth of data.
Speaker:But there's also,
Speaker:in addition.
Speaker:RTO also applies with
Speaker:Yeah.
Speaker:In addition to the amount of business that your system will or that your business
Speaker:will lose, uh, either bus business that it had or business that it could have
Speaker:had, also, someone needs to be able to
Speaker:come up with the intangible things like damage to brand identity,
Speaker:right?
Speaker:Um, also.
Speaker:Potential future lost data due to this is, this is primarily, well,
Speaker:it's gonna drive both of them.
Speaker:Because if you, if you, if you took an order from somebody and
Speaker:then an order just disappears
Speaker:and you don't even know who they are,
Speaker:then um, you know, not only are you gonna not get that business,
Speaker:you're gonna anger a customer.
Speaker:Also lawsuits potentially.
Speaker:And potential lawsuits.
Speaker:Right?
Speaker:Exactly.
Speaker:Right.
Speaker:That's a good point.
Speaker:So you have to think about all of these different things.
Speaker:This is, but again, this is what a business
Speaker:does.
Speaker:This is not your job as a backup and Dr person.
Speaker:This is not your job.
Speaker:If you're thinking, you're listening to this and you're like, gee, this
Speaker:all sounds really complicated, and the big responsibility, yes.
Speaker:It's why it's not yours.
Speaker:Um, it's, it's the CEO, the CFO, uh, you know, a lot of
Speaker:people with Cs in their names
Speaker:management
Speaker:and then the people that they appoint, uh, under them to, to make these decisions.
Speaker:Yeah.
Speaker:were saying, what would
Speaker:Oh, the risk management team.
Speaker:Risk management team.
Speaker:Exactly right.
Speaker:So if RTO and RPO drive backup design, what parts of backup
Speaker:design do RTO drive and RPO drive?
Speaker:So let's, let's do RTO first.
Speaker:What, What, element of backup design does RTO drive?
Speaker:It is how you do recoveries.
Speaker:Def define how, what do, what do you mean?
Speaker:All right.
Speaker:Mechanisms that you use for recovering your data.
Speaker:For example, do you have a standby DR.
Speaker:Site available and ready?
Speaker:Do you have data in a replica database?
Speaker:Are you having to retrieve data from tape?
Speaker:Yeah, all of these are, I was looking for something maybe a
Speaker:little higher level than that.
Speaker:Those are all correct answers.
Speaker:Um, what I was thinking was, it's gonna deri, it's, it's going to
Speaker:drive the, the, the design that determines the speed of recovery.
Speaker:Yeah.
Speaker:Right,
Speaker:which everything you just said
Speaker:determine the speed of the recovery, right?
Speaker:If you have an aging system with an aging backup server and an aging, an
Speaker:aging storage, and you're having to do a full recovery, uh, from older systems,
Speaker:that's going to take a really long time.
Speaker:But if you've got a cutting edge backup system with failover, Dr.
Speaker:All of these things, um, then you should be able to do a very quick recovery.
Speaker:Uh, you, well, you should be able to do a very quick restore,
Speaker:which will enable to do, enable you to do a very quick recovery.
Speaker:Um, so RTO primarily DR.
Speaker:Drives, I'm gonna say the, the power of the backup system, the oomph, the
Speaker:speed, uh, its ability to restore data quickly, which sometimes the
Speaker:quickest way to restore data quickly is to restore it before you need it.
Speaker:Right, and, and to, and to not restore it at all.
Speaker:There are mechanisms to either restore the data before you need
Speaker:it or to not restore it at all.
Speaker:What would be a way to not restore it at all
Speaker:Do you basically have a copy sitting there that is
Speaker:or,
Speaker:available
Speaker:or
Speaker:or a
Speaker:snapshot?
Speaker:Oh
Speaker:I know how much you love when I bring that up.
Speaker:yes.
Speaker:So if,
Speaker:Which is like copy sitting there.
Speaker:well, it's a virtual copy,
Speaker:Okay.
Speaker:right?
Speaker:Um, it's very important to distinguish
Speaker:that, uh, at least what I would call a real snapshot is
Speaker:a virtual copy.
Speaker:Yes.
Speaker:Um, yeah.
Speaker:Um, if you're able to, if, if it's, if it's a copy just ready
Speaker:to go and you're able to just.
Speaker:Restore as I make quotes in the air by simply going back to an earlier
Speaker:point in time, um, you are going to get a very quick RTO right?
Speaker:Yes.
Speaker:I just wondering also is re,
Speaker:I'm wondering though if like even fast, like what's even faster than a snapshot?
Speaker:I don't know what's faster than the
Speaker:Because with the snapshot you still have to have someone
Speaker:go and restore the snapshot.
Speaker:Well,
Speaker:it's, you're not really doing a restore, right?
Speaker:I mean, a, a
Speaker:good snapshot system.
Speaker:You're just doing pointers, right?
Speaker:So what do, I'm not sure what
Speaker:Oh, so I'm thinking more like, um, as an example, clustered high
Speaker:availability with the DR Replicated copy and automatic failover.
Speaker:Yeah, yeah, that's all fine, right?
Speaker:That that's all I, I don't know if that's faster than a snapshot, but
Speaker:it's very similar.
Speaker:Basically, you have stuff that's, it's immediately, immediately,
Speaker:available and ready to go, right?
Speaker:So there are a number of ways that you can, number of ways
Speaker:from, from days to seconds.
Speaker:To, to meet different levels of RTO based on, um, based on what you're
Speaker:willing to spend and how you're willing to design your backup system.
Speaker:Right?
Speaker:Can I bring up a story?
Speaker:sure.
Speaker:So I was listening to a podcast on my walk, and it's of this automotive company.
Speaker:They sell bolts and nuts,
Speaker:Okay.
Speaker:right?
Speaker:Um, and they were talking about how they were, they were a small shop
Speaker:and they have like 30 people now.
Speaker:And the guy, the co-founder or the CEO founder, he basically ran it.
Speaker:And they had a bunch of systems and things like that, and he
Speaker:was kind of managing everything.
Speaker:And then they got hit with, uh, ransomware
Speaker:Yeah.
Speaker:and they lost all their data.
Speaker:And then he realized he had a backup, but it was from three months ago.
Speaker:Right?
Speaker:So his RPO was three months,
Speaker:Nice.
Speaker:Well.
Speaker:RPO.
Speaker:No, his RPO never changed his RPA.
Speaker:We haven't got to that
Speaker:yet.
Speaker:Yet.
Speaker:Well that's what he had.
Speaker:Yes.
Speaker:And he basically said though, so they had a bunch of invoices and customers and
Speaker:luckily they had everything hard copy.
Speaker:He said it took them six months to manually enter, to manually enter
Speaker:all the transactions that they were missing before those three months.
Speaker:At least he was able to get, uh, to put that data back,
Speaker:right.
Speaker:Yeah.
Speaker:Uh, by the way, this, this is just as good a time as any to bring up the
Speaker:terms RPA and our, our RTA and RPA.
Speaker:So recovery time actual and recovery point actual.
Speaker:So a lot of people say, you know, and I do it, I do it occasionally,
Speaker:so it's not like I'm, I'm
Speaker:calling outy or Prasanna.
Speaker:Um, we say, oh, we, you know, we recovered it in two hours.
Speaker:So we had a two hour RTO.
Speaker:The So recovery, it's recovery point objective, recovery time objective.
Speaker:This is the goal.
Speaker:This is the service level agreement, right?
Speaker:The
Speaker:SLA,
Speaker:Oh.
Speaker:So I guess in my, in that example case, they had no agreement because they did
Speaker:yeah, that.
Speaker:That's actually the most
Speaker:common.
Speaker:That's the most common, right?
Speaker:No, no.
Speaker:SLA, no, R-G-O-R-P-O.
Speaker:We just do our best case and, um.
Speaker:But how, how fast the system is actually able to do something is your RTA and RPA.
Speaker:Right.
Speaker:Your recovery to recovery time.
Speaker:Actual and recovery point actual, uh, I've also in, in various times in my
Speaker:career, referred to that as the RTR and R-T-R-P-R recovery time reality.
Speaker:Yeah.
Speaker:Um,
Speaker:but
Speaker:that?
Speaker:How do you actually
Speaker:figure out
Speaker:the only way to calculate is to actually do it right.
Speaker:So hopefully everyone is doing restore testing
Speaker:or recovery
Speaker:story testing is the only way to know for re for sure.
Speaker:All right, so let's talk about, you talked about how, what, uh, aspects of
Speaker:backup design are determined by RTO.
Speaker:What this answer is much easier.
Speaker:What aspect of backup design is determined or is Yes, is determined by RPO?
Speaker:How frequently can you back up your data?
Speaker:Yeah.
Speaker:So if you're backing up your data more often, you are going
Speaker:to have a better RPA, right?
Speaker:Um, so if you have, if, what's the most common, uh, backup
Speaker:frequency, would you say?
Speaker:Probably once a day.
Speaker:Once a day.
Speaker:And so the most common RPA is at minimum 24 hours.
Speaker:Well, the, the, you know, there are situations where
Speaker:you might get better than that
Speaker:if you, like you said earlier, if you had a failure right after
Speaker:you just finished a backup,
Speaker:but worst
Speaker:um, you might, that's best case
Speaker:scenario, worst case scenario.
Speaker:Is more like 36 hours because you're, you're, you know, at the end of a
Speaker:day before the, you know, and, um.
Speaker:The, and, and that, that's assuming that last night's backup worked,
Speaker:and I've done enough backups to know that that isn't always the case.
Speaker:And I would also say I would apply Murphy's Law of backups, which is the
Speaker:the degree to which, you know, a ba the, the, the possibility that a backup is
Speaker:successful is inversely proportional to, um, how, how much you need that backup.
Speaker:Yeah.
Speaker:Is there also a Schrodinger's cat for backups?
Speaker:Yeah.
Speaker:Yeah,
Speaker:exist, or you don't know if a backup exists or not until you actually,
Speaker:Until you actually look at it.
Speaker:yeah,
Speaker:Um,
Speaker:I think we need that on a T-shirt.
Speaker:By the way, if you would buy a T-shirt from
Speaker:the backup wrap up, right, with these phrases on it.
Speaker:Let Curtis know on X or LinkedIn or somewhere, or leave a
Speaker:Yeah,
Speaker:on YouTube.
Speaker:We've been getting a lot of YouTube comments lately.
Speaker:Um, yeah, let me know on YouTube, by the way, did I tell you, you know
Speaker:how it's how back when I was not feeling well or I hurt myself and
Speaker:I said, no one cares.
Speaker:Then I said, if anyone cares, please say something.
Speaker:We got comments.
Speaker:One of,
Speaker:you know what?
Speaker:One of them said,
Speaker:What?
Speaker:India cares about Curtis.
Speaker:Aw.
Speaker:That's what I got.
Speaker:Um, so.
Speaker:Yeah, so RPO basically determines your backup frequency.
Speaker:If, if you've got an RPO measured in hours, you can't have a backup frequency
Speaker:measured in, in, you know, many hours.
Speaker:Right?
Speaker:Um, and the, um, I, I will say that,
Speaker:just talk about the different ways that people do backups, sort of normal.
Speaker:Regular, what I would call full file incremental backups.
Speaker:Pretty much the best you're gonna be able to do is daily.
Speaker:Right.
Speaker:If you're able to do some type of block level incremental forever,
Speaker:where each backup, you're only backing up the blocks that have changed,
Speaker:or even better, the deduplicated blocks that have changed, right?
Speaker:Because that's gonna be an even smaller, uh, set of data.
Speaker:Then you could actually do the ba It does two things.
Speaker:One is it makes the, it makes the backup.
Speaker:Quicker, which is good.
Speaker:Um, which generally means that you can do it more often, uh, and then it, it
Speaker:does allow you, it also reduces the, the impact of the backup, meaning that while
Speaker:the backup is running, it's not having too much of a, of an impact on production.
Speaker:And so that allows you to do it more often.
Speaker:To do it, you know, once an hour is very common.
Speaker:Uh, if you've got a, a modern backup system, you could do it once an
Speaker:hour or even more often than that.
Speaker:I.
Speaker:Yeah.
Speaker:While you're mentioning this about the incrementals and chain block tracking,
Speaker:one thing I realized we didn't cover is depending on what technologies you use,
Speaker:it will have an impact on your RTO because if you have say, multiple, uh, recovery
Speaker:points, you have to recover from first before you can get to that final point.
Speaker:Like if you have to recover a full plus multiple incrementals.
Speaker:That is going to take you more time, which will impact your RTO as well.
Speaker:Absolutely.
Speaker:Um, I do want to talk, so what we'll round out this discussion with is that
Speaker:idea that you brought up earlier, which
Speaker:is the,
Speaker:that, um, every stakeholder is gonna want an RTO and an RPO of zero.
Speaker:Because if you ask people, if you ask it like this, how much data.
Speaker:You know, how, how long are we allowed to be down and how much
Speaker:data are we allowed to lose?
Speaker:And the answer is zero and zero.
Speaker:Yep.
Speaker:How do we get from there to something that's more reasonable and possible?
Speaker:$1 billion.
Speaker:Yeah.
Speaker:What, so yeah, so you go back, you're like, okay, all right.
Speaker:We've taken the requirements, we've gone and we've met with the powers
Speaker:that be, and we've decided that if we go with a fully redundant system.
Speaker:Uh, with, you know, with, with, uh, synchronous replication and a hot site
Speaker:that's, you know, this far away we can do, we can give you the RTO and an RPO of
Speaker:zero and it's going to cost $1 billion.
Speaker:Right.
Speaker:And you do that with a straight face, right?
Speaker:You just go, okay, okay, we got, you know, we got a thing, we got a thing, we got the
Speaker:vendor, we got this vendor standing by to
Speaker:take your check.
Speaker:And uh, we can do it for $1 billion.
Speaker:And then, um, um.
Speaker:What I generally find is at that point, obviously I have a Plan B and a plan C,
Speaker:and uh, I've also consulted with this vendor.
Speaker:And this vendor says that they can do an RTO and an RPO of
Speaker:four hours for only $1 million.
Speaker:$1 million.
Speaker:Sounds a lot.
Speaker:Sounds like a lot,
Speaker:but it's a better than a billion.
Speaker:it to $1
Speaker:So, so here's a question.
Speaker:So as a backup admin, when you're going to these discussions with the stakeholders,
Speaker:Mm-Hmm.
Speaker:is it.
Speaker:Worthwhile to go with a couple options, like you mentioned where you're like,
Speaker:and what would those common options be?
Speaker:Right.
Speaker:Do you think it's zero?
Speaker:Zero, right?
Speaker:It's a four hour RPO.
Speaker:Maybe it's a one day RPO.
Speaker:Like
Speaker:Yeah.
Speaker:Um,
Speaker:what?
Speaker:What would make sense?
Speaker:Yeah.
Speaker:I think it's going to be driven.
Speaker:Most backup designs.
Speaker:In fact, I'm gonna say all backup designs are driven by cost and reality,
Speaker:right?
Speaker:So, you know, they want zero and zero.
Speaker:Or let's say, let's say they want 15 minutes.
Speaker:We're like, look, we can do 15 minutes, 15 minutes costs this much.
Speaker:This is the way to do it.
Speaker:Um, if they, if they say.
Speaker:We wanna, we wanna, you know, a very tight RTO, but we're okay with
Speaker:an hour's loss, data loss, right?
Speaker:Snapshots and replication.
Speaker:Great way to do that.
Speaker:If they want very, you know, like as close to zero as possible, you've really
Speaker:got to do CDP or something like it.
Speaker:You gotta do synchronous replication.
Speaker:And by the way, it's important to mention here that not everything has
Speaker:to have the same RTO and RPO in fact.
Speaker:In your environment?
Speaker:Everything should in your environment, everything should definitely not have the
Speaker:same RTO and RPO.
Speaker:If that's the case, it means you've sort of settled you.
Speaker:You've definitely got an application that needs more than that.
Speaker:Uh, and you settled for simplicity, which is very common for people to do, but
Speaker:that's not really the right thing to do.
Speaker:You've got one or two applications within your environment that would really lose
Speaker:a lot of money, and it's totally okay to have a different backup design for them.
Speaker:Right.
Speaker:You have the gold, silver, bronze, right?
Speaker:Um, I, I definitely like, go ahead.
Speaker:W, and I know you mentioned backup technology, like a different backup
Speaker:technology that might also drive having different backup vendors as
Speaker:well to satisfy these different needs.
Speaker:Yes, it definitely will be.
Speaker:I mean, my, as a
Speaker:backup person, my preference is always one vendor if possible.
Speaker:Um, and, um, you know, one design with options, uh,
Speaker:because again, simplicity, um,
Speaker:means recovery, I think.
Speaker:Um, but.
Speaker:but.
Speaker:yeah, it, it is really important to understand that it's very common
Speaker:for you to have multiple RTOs and RPOs within an environment.
Speaker:I will also state, and this is somewhat controversial and not
Speaker:everybody agrees with me here, it's also okay to have different RTOs
Speaker:and RPOs based on scenario, right?
Speaker:Like if we've got, um, if a server died or a server caught on fire or a server
Speaker:got whatever, you know, server goes, poof.
Speaker:seems to me that that, that, that RTO and RPO, um, would be
Speaker:much, much shorter than if an asteroid hit Oceanside, right?
Speaker:Um,
Speaker:it's just there are different levels of disasters and there's different amounts
Speaker:of understanding that you can have.
Speaker:So
Speaker:I think it's totally okay to have different kinds of RTOs
Speaker:and RPOs for different kinds of
Speaker:and, and I think this is where we kind of like from talking to Mike
Speaker:about our ransomware recovery and other things like that, right?
Speaker:I think this is where sort of the incident response plans come in, right?
Speaker:Where different scenarios you may have different RTOs and RPOs.
Speaker:Because they are different and there are things that are outside of your
Speaker:control, like recovering from ransomware is very different because of the initial
Speaker:steps you have to do to isolate and figure out what's going on, rather
Speaker:than, oh, this server, the disc died.
Speaker:Now I need to recover.
Speaker:Or someone accidentally deleted a file.
Speaker:Yeah.
Speaker:And by the way, you know, speaking of recovering from ransomware, you know
Speaker:when, when we've talked about that, when you get to the actual recovery, it's
Speaker:very common for a person who knows what they're talking about in that world.
Speaker:Say, listen, I want you to, you might be recovering the same data set 50 different
Speaker:times from 50 different data points.
Speaker:That's something you really gotta figure out in your backup design.
Speaker:Right.
Speaker:Because there are backup designs where that is not gonna happen anytime soon.
Speaker:Right.
Speaker:Um, but, uh, well, hey, once again, uh, we did not do a 10 minute episode.
Speaker:Yay.
Speaker:And we're all alive.
Speaker:But hold on.
Speaker:So going back to my initial question I had, or which is, I know you said RTO
Speaker:and RPO are probably the most important things, foundational things in backup
Speaker:yeah,
Speaker:system design.
Speaker:What did you think?
Speaker:Do you think that second from that would be the 3, 2, 1 rule?
Speaker:Well, the 3, 2, 1 rule is, um,
Speaker:it is also foundational, right?
Speaker:Um,
Speaker:and.
Speaker:I, again, is it more important?
Speaker:Is it if you don't, if you don't follow the 3, 2, 1 rule and all your backups
Speaker:are all in the same place as the production, why are they're not even
Speaker:Back up.
Speaker:Yep.
Speaker:Yeah.
Speaker:Agreed.
Speaker:Maybe next episode, should we, we should dive down into 3, 2, 1 rule.
Speaker:We haven't done that in a while.
Speaker:Yep.
Speaker:All right.
Speaker:Prasanna, bossy mc.
Speaker:Bossy
Speaker:man.
Speaker:Um.
Speaker:my bossy pants,
Speaker:I.
Speaker:I hope you folks have enjoyed our episode this week on RTO versus RPO.
Speaker:That is a wrap.
Speaker:The backup wrap up is written, recorded and produced by me w Curtis Preston.
Speaker:If you need backup or Dr.
Speaker:Consulting content generation or expert witness work,
Speaker:check out backup central.com.
Speaker:You can also find links from my O'Reilly Books on the same website.
Speaker:Remember, this is an independent podcast and any opinions that you
Speaker:hear are those of the speaker.
Speaker:And not necessarily an employer.
Speaker:Thanks for listening.