Speaker:

This week, we're talking about one of the most fundamental technologies

Speaker:

underpinning today's data protection systems and that's replication.

Speaker:

What is it?

Speaker:

How is it different than other methods?

Speaker:

And if it's so great, why don't we just use it for everything?

Speaker:

And of course what's the difference between synchronous and asynchronous.

Speaker:

And why does that matter?

Speaker:

Hi, I'm W.

Speaker:

Curtis Preston AKA Mr.

Speaker:

Backup.

Speaker:

And I've dedicated my 30 year career to helping people like you keep your data

Speaker:

safe from all that would do it harm.

Speaker:

This is the backup wrap-up.

Speaker:

Welcome to the show with me, as always, as a guy who seems to never

Speaker:

be happy until I have problems.

Speaker:

Prasanna Malayandi.

Speaker:

How's

Speaker:

I'm good.

Speaker:

I'm just trying to think what problems do you have right now?

Speaker:

Talking about, I'm trying to edit the podcast and you're giving

Speaker:

me problems about my network.

Speaker:

That's,

Speaker:

what I'm talking about.

Speaker:

yeah, it's.

Speaker:

What I do, you need someone to ask the hard questions,

Speaker:

Somebody's got to do it.

Speaker:

The network's going all right.

Speaker:

Those that have followed, I do have this new tool.

Speaker:

I have, um, the, uh, Firewalla.

Speaker:

version 125 of Curtis's home network, architecture.

Speaker:

Yeah.

Speaker:

but I now have this Firewalla that gives me a lot more

Speaker:

information than I had before.

Speaker:

And unfortunately it verified, my new ISPs, thing that I use too

Speaker:

much internet and I have to buy.

Speaker:

the unlimited package, which is sad because it's another 50 bucks a month.

Speaker:

But anyway, so let's get to the news of the week

Speaker:

I'm going to let you take the first story, uh,

Speaker:

Yeah, yeah, yeah, so the first one is an article that came out recently.

Speaker:

it is about a backup compa I guess, would you call them backup or data

Speaker:

protection company that recently, Raised around a funding, it's called Alcion.

Speaker:

I actually used to work with the founder previously at an old job.

Speaker:

and they actually raised a round of funding and it was actually led by

Speaker:

Veeam, which is interesting because Veeam, as we all know, is also a backup

Speaker:

company and they are investing in Alcion, who is primarily targeting backups

Speaker:

for Microsoft 365 for SMB customers.

Speaker:

today?

Speaker:

That's their initial target market.

Speaker:

I'd say that the biggest difference between the two companies,

Speaker:

one is a software product, the other is a SaaS offering, and

Speaker:

that makes them very different.

Speaker:

And I guess Veeam felt that it was different enough.

Speaker:

I like the quote in the article.

Speaker:

It said something along the lines of that, that from Veeam, that

Speaker:

this problem of ransomware is a big enough problem that we're going to

Speaker:

need the whole industry to solve it.

Speaker:

And so they invested in what.

Speaker:

is essentially, I would call it a partial competitor, right?

Speaker:

They do compete for the Microsoft 365 space, I also know the founder as

Speaker:

well, and, I think they've taken an interesting angle with a very AI based

Speaker:

approach, a very security based approach,

Speaker:

Yeah.

Speaker:

And one of the things that I'm actually looking forward to is seeing if this

Speaker:

is going to force other vendors to reconsider how they approach the problem.

Speaker:

And.

Speaker:

Everyone in all the vendors raise a bar when it comes to ransomware

Speaker:

protection and detection and other things like that, which we all

Speaker:

need.

Speaker:

Right?

Speaker:

Anybody that's listened to this show knows that we're concerned about that problem.

Speaker:

my second news story, it's from the enterprise strategy group, which for

Speaker:

those of you that have known ESG for a while, they were, A few years ago,

Speaker:

acquired by TechTarget, and there was a, 2023 survey of IT professionals.

Speaker:

The most interesting number that I see here is that they said 65 percent

Speaker:

of the almost 400 respondents said their organization were using some

Speaker:

sort of cloud based, backup system.

Speaker:

And the way they worded it, that could be either a cloud based

Speaker:

service or they're storing a copy of their backups in the cloud.

Speaker:

That, to me, really, that's a very different number, wouldn't you say, than

Speaker:

a few years

Speaker:

Yeah, everyone would be like, I don't trust my data in the cloud.

Speaker:

How do I get my data up into the cloud and bring it back and all the rest of that?

Speaker:

In fact, I think one of the interesting things was even just

Speaker:

the title of the article, right?

Speaker:

It's like cloud backup and disaster recovery evolved toward maturity.

Speaker:

You know, that just kind of says it all, right, versus before you're

Speaker:

fighting tooth and nail with customers, trying to persuade them, Hey, it's

Speaker:

okay to store your data in the cloud.

Speaker:

But

Speaker:

Yeah, so the not so good number was saying that half of the surveyed orgs

Speaker:

recovered 75 percent or less of their cloud data per incident on average.

Speaker:

That sounds really bad.

Speaker:

that is not, I agree with, Christoph Bertrand, which said that's just

Speaker:

not acceptable for, such a mission

Speaker:

I guess the question I would have is, it would be, we probably need

Speaker:

to compare that to what's your recovery rate of on premises backups,

Speaker:

because that's not going to be 100%.

Speaker:

and then I think it's also not fair to compare it because how

Speaker:

long has traditional backups, been around and had time to evolve

Speaker:

versus cloud backups, right?

Speaker:

It hasn't had that time to evolve and mature.

Speaker:

Fully as these other products

Speaker:

Yeah, I would say that part, having formerly worked at a cloud data protection

Speaker:

vendor, I would think that the problem that I see with this number is that they

Speaker:

do conflate people storing backups in the cloud versus people using a cloud service.

Speaker:

I would be very surprised to learn that.

Speaker:

People that are using a fully cloud based service, if they, if their recovery's

Speaker:

failed, because the other, it's really, the cloud just happens to be a part of it.

Speaker:

All of the same sort of misconfiguration problems that lead to problems

Speaker:

with traditional backup are going to bleed over into a cloud.

Speaker:

Just because you're using cloud

Speaker:

but it's the cloud.

Speaker:

It's magical.

Speaker:

Come on Curtis.

Speaker:

Yeah

Speaker:

Yeah, it is magic.

Speaker:

the one other interesting number from this survey, and there were

Speaker:

some other interesting numbers, but just for brevity purposes, this other

Speaker:

interesting number, they said 16 percent of the, respondents said that they.

Speaker:

reviewed their, their cloud data protection strategies annually.

Speaker:

that is, that's a very different number than back in the day.

Speaker:

Back in the day, that number was like five years, maybe three years.

Speaker:

Certainly not annually.

Speaker:

Backup historically has been an incredibly sticky process.

Speaker:

And, this idea that they would review it annually, I think it's a

Speaker:

I think it also makes sense though, because you like with the five

Speaker:

year, three year, five year, right?

Speaker:

You have to worry about hardware depreciation, right?

Speaker:

Upgrade cycles all the rest With cloud, right?

Speaker:

There might be new technologies, cheaper ways to do things, right?

Speaker:

Your workloads might change.

Speaker:

what you had running on premises might be migrated to the cloud.

Speaker:

now you have to reconsider it.

Speaker:

How do I back it up?

Speaker:

And so I think that's probably why looking at these things more

Speaker:

frequently, like annually, probably starting to make a lot more sense.

Speaker:

Well, that is the news of the day.

Speaker:

on this episode of the Backup to basic series, we're going to be talking

Speaker:

about replication, different kinds of replication, the good things, the bad

Speaker:

things, and what they, you know, what they have to do with backup and, and

Speaker:

I, I,

Speaker:

it a backup,

Speaker:

I, I, I I thought you were going to talk about like the replication

Speaker:

technologies that they have in like Star Trek and, Right?

Speaker:

Where it's like, create me,

Speaker:

No, that's, that's not replication.

Speaker:

That's...

Speaker:

What is that called?

Speaker:

what are those things?

Speaker:

Transporters?

Speaker:

Oh, oh, no.

Speaker:

the making, the stuff, right?

Speaker:

The food

Speaker:

Do they call them replicators?

Speaker:

Yeah.

Speaker:

I think they call them replicators.

Speaker:

We are, we are not discussing replicators like in Star Trek, although that does,

Speaker:

those do sound magical, by the way.

Speaker:

And by the way, like if you could do transporters, of course you

Speaker:

could do replicators, right?

Speaker:

Because all replicators are is storing a backup copy of the food and then

Speaker:

saying this is what it is, right?

Speaker:

Do you think that they have backups of those backups?

Speaker:

The copies?

Speaker:

Yeah, no, the copies of the food because imagine you wipe out the

Speaker:

copies and then you've basically killed off humanity because they

Speaker:

wouldn't have any way to make food.

Speaker:

I

Speaker:

Or maybe they're immutable, or maybe they're immutable copies, you know?

Speaker:

Maybe they're immutable copies.

Speaker:

I would, I would hope that no one can go in and change the

Speaker:

food before it gets replicated.

Speaker:

Um,

Speaker:

On to a more serious matter.

Speaker:

let's do a more serious matter.

Speaker:

Well, so we, we've moved on the previous episodes.

Speaker:

We talked about sort of regular kind of backups and they were, they were backups.

Speaker:

That require a restore basically is, is what we were talking about.

Speaker:

We've now moved on to types of backups where the restore

Speaker:

is essentially already done.

Speaker:

Right.

Speaker:

Um, and replication.

Speaker:

And so I'm, I'm, I'm putting these and, and by the way, you know, for those that

Speaker:

don't know what I'm talking about, we are basically working our way through.

Speaker:

This book, Modern Data Protection, which is my book, which is available,

Speaker:

you know, wherever books are sold.

Speaker:

And, um, you know, I might even put a, put a link in the show description there.

Speaker:

Um, and the idea here is what we're talking about is methods of backup.

Speaker:

And again, remember when I say backup, I I'm very broad in that term, methods

Speaker:

of backup that support instant recovery.

Speaker:

What do you want to talk about?

Speaker:

What instant recovery is?

Speaker:

Yeah, so, normally when you think about, you took a backup, you

Speaker:

need to, something happens, you need to get back your data, right?

Speaker:

If you had your data stored on tape, you gotta go fetch the tape, you gotta pull

Speaker:

the data off of tape, copy it somewhere else, and now you actually have your data

Speaker:

and you can do something with it, right?

Speaker:

With Instant Recovery, It's really, your data is already available for you.

Speaker:

You don't have to have any sort of processing that you have to do before you

Speaker:

can actually start accessing your data.

Speaker:

Yeah, I mean, it's a beautiful thing.

Speaker:

I think that maybe, I dream that sometime in the future, this is

Speaker:

the way all restores will be done.

Speaker:

And the cloud really does make this possible.

Speaker:

We could talk about that all day long, but we just need to work through

Speaker:

the different ways that, um, we're going to create an always on readily

Speaker:

available copy of the data, and the first that we're going to talk about is

Speaker:

Before you get there though, I think one important point to cover is, I

Speaker:

know you said in the future maybe everything will be like this, right?

Speaker:

I think if you had a gazillion dollars, yes, Because there

Speaker:

is a cost associated, right?

Speaker:

Which I think is important to cover, right?

Speaker:

With sort of using these various technologies that might make it more

Speaker:

difficult now Like you said, maybe the cloud helps with some of these but there

Speaker:

is still a cost associated with Keeping data on something that is instantly

Speaker:

available Versus say using a tape or some sort of offline Media where the

Speaker:

cost may not be just the cost of the storage but think about from power cooling

Speaker:

Right all those other aspects as well

Speaker:

Yeah, I was speaking in the future when these problems have all been solved.

Speaker:

Um, I think even today that there are services, there are cloud storage

Speaker:

services, that if used properly and judiciously, can provide you A ready

Speaker:

to go copy of your data pennies on the dollar that costs you, you know,

Speaker:

next to nothing to, to have and to possess and to have it ready to go.

Speaker:

You will then pay through the nose if you use them, right?

Speaker:

I'm thinking of course of like Glacier, Deep Archive, Instant Recovery.

Speaker:

Right?

Speaker:

There are services that are still storing the data on disk, but storing

Speaker:

it in such a way, I don't know, we don't, you know, neither of us have

Speaker:

ever worked at AWS or any of the cloud providers, but they're storing it in

Speaker:

such a way that they've made it so cheap.

Speaker:

To, to, to have that copy, to have it offsite, to have it in,

Speaker:

you know, in another region.

Speaker:

I just think that as that continues to happen, that we will eventually

Speaker:

get to a place where all DR copies can be held like this.

Speaker:

Can I have a dream, Prasanna?

Speaker:

Can I have a dream?

Speaker:

Can

Speaker:

You can have a dream, but I think, but yes, but before, sorry, before

Speaker:

we also continue on, I think we also need to define what we

Speaker:

mean when we say instant, right?

Speaker:

Because instant,

Speaker:

when we say instant recovery, right?

Speaker:

I think it's helpful to talk about that because instant is a spectrum, right?

Speaker:

Like you were talking about for your Glacier instant restore feature, right?

Speaker:

It's not like, Oh, I want my data.

Speaker:

It's available within the Cloud.

Speaker:

Like, snap my fingers, it's there.

Speaker:

Right?

Speaker:

It does take time, but that time may be acceptable to you, right?

Speaker:

It might be, say, minutes, not days.

Speaker:

yeah.

Speaker:

So instant is kind of like continuous.

Speaker:

It's a binary condition.

Speaker:

It's sort of like immutable.

Speaker:

It's a binary condition.

Speaker:

It's like pregnancy, right?

Speaker:

You're either pregnant or you're not.

Speaker:

And so it's either instance or it's not.

Speaker:

But it's yet another one of these terms where it's not truly binary.

Speaker:

Right?

Speaker:

It's just, we're defining instant recovery in this point is, I don't need

Speaker:

to do a restore to do the recovery.

Speaker:

The data's already been restored.

Speaker:

It might take a few minutes to make it accessible to me.

Speaker:

Yeah, sorry, and, and just for people who may not have heard our previous

Speaker:

episodes, do you want to quickly cover differences between restore and

Speaker:

recovery since you just talked about it?

Speaker:

A restore is essentially the act of bringing back the, you know,

Speaker:

the data and putting it in place.

Speaker:

The recovery is the process of sort of bringing that data online and using it.

Speaker:

And, and sometimes that recovery involves massaging other things.

Speaker:

It involves massaging the data and it involves doing other things.

Speaker:

When you're done with the recovery.

Speaker:

Everything is working again.

Speaker:

When you're done with the restore, everything is ready to be working again.

Speaker:

Uh, so depending on how good or how complicated the restore is,

Speaker:

the two may be, they, they may happen at the same time, but in a

Speaker:

complicated situation, you do the restore and then you do the recovery.

Speaker:

this case, uh, the recovery would be this instant of, um,

Speaker:

you know, like I said, just,

Speaker:

Everything's ready to go.

Speaker:

Or sorry, everything.

Speaker:

has, is available.

Speaker:

Yeah, exactly.

Speaker:

Uh, we don't have to do a restore.

Speaker:

And I would say that in a true instant recovery, the size of the, of the thing

Speaker:

being recovered should not be a factor.

Speaker:

Whereas in a restore, it very much is,

Speaker:

Oh yeah.

Speaker:

right?

Speaker:

Yeah.

Speaker:

Yeah.

Speaker:

Okay.

Speaker:

So with that in mind, we're going to talk about the first method that could be used.

Speaker:

And, and is used actually, it is used quite a bit to do an instant recovery.

Speaker:

Uh, as we often do on the show, let me take you back in the day.

Speaker:

Yeah.

Speaker:

So back in the day, there was a very sharp dividing line between.

Speaker:

Backup and recovery and disaster recovery.

Speaker:

And for backup and recovery for what we also call operational

Speaker:

recovery, you had a backup system.

Speaker:

You went to tape and, and when you did a restore, you, you came back

Speaker:

up, you came back off of tape.

Speaker:

Anyone with a tight recovery time objective would never build,

Speaker:

uh, their DR system based on tape back in the day, right?

Speaker:

Even as we move to disk based backup and recovery systems, anybody with a

Speaker:

tight RTO is not going to build their disaster recovery system built on

Speaker:

having to do a restore first, right?

Speaker:

So back in the day, if you worked at like a financial trading firm or

Speaker:

something like that, and you wanted to have a disaster recovery plan because of

Speaker:

course you wanted that because you had the OCC that was looking into you and

Speaker:

Regulation,

Speaker:

could say we're not a bank, right?

Speaker:

You used replication.

Speaker:

You had a backup and recovery system that did operational recoveries

Speaker:

and file recoveries, and maybe even recovering an entire server.

Speaker:

But if you had data that truly mattered, you used replication.

Speaker:

And, and I know you worked at a company

Speaker:

Yeah, that, was my first job out of, out of

Speaker:

college.

Speaker:

Yeah, So, let's talk about what replication is, uh, how does

Speaker:

it typically manifest itself.

Speaker:

Do you want to just...

Speaker:

Sort of define what replication is?

Speaker:

so replication is basically taking data, making a copy, and sending

Speaker:

it off to some other system, right?

Speaker:

At a high level.

Speaker:

Now, there's various nuances and combinations that other system

Speaker:

could be the exact same vendor, the same model, all the rest.

Speaker:

As the original, it could be a different vendor, right?

Speaker:

A different system.

Speaker:

Right?

Speaker:

So there are different ways you could do it.

Speaker:

There are different protocols for transferring that data.

Speaker:

You could use protocols that some of these vendors have built in

Speaker:

for communicating between them to make things more efficient.

Speaker:

Because when you have to send data across from one system to another,

Speaker:

doing it over and over, because it's not like it's a one time operation, right?

Speaker:

Depending,

Speaker:

Yeah.

Speaker:

If I could interrupt, I think that's one word missing from your definition.

Speaker:

If I could insert at the beginning of your definition, the word continuously.

Speaker:

Right.

Speaker:

Because anybody can copy the data over.

Speaker:

I think replication implies that we're continuously copying it

Speaker:

over.

Speaker:

The idea is that it, that it's happening all the time, right?

Speaker:

We're going to talk about the difference between asynchronous and synchronous.

Speaker:

So sorry for the confusion for those, but basically you have some data, you

Speaker:

know, in A and you, and you have a copy of that data in B and that copy,

Speaker:

that, that B copy is continuously, for lack of a better word, updated to

Speaker:

be the same exact thing as what's in

Speaker:

Yeah, so going to Curtis's disaster recovery scenario, right?

Speaker:

Something happens on system A, right?

Speaker:

System B has a version of what system A looked like at a point in time,

Speaker:

and you can fail over to system B, and that's how you're instantly

Speaker:

up and running, because system B

Speaker:

I think one thing we should also cover, which I want to talk about

Speaker:

earlier, I know you threw out the term recovery time objective.

Speaker:

I think another point to talk about here is also recovery point objective, right?

Speaker:

Yeah.

Speaker:

Well, go ahead.

Speaker:

You brought it up.

Speaker:

So yes, in the case of when you're doing synchronous replication, yes, there's

Speaker:

still going to be an RTO on that.

Speaker:

But that RTO, you should be able to measure in seconds to minutes, depending

Speaker:

on how long it takes you to bring up system B and make it available for

Speaker:

your applications and other things.

Speaker:

So that's kind of

Speaker:

like RTO side.

Speaker:

Do you wanna talk about the RPO, Curtis?

Speaker:

Yeah.

Speaker:

So the RPO is, is basically the amount of data that you

Speaker:

agree that you can lose, right?

Speaker:

The recovery point objective is we say we can lose an hour's worth of data.

Speaker:

We can lose a day's worth of data.

Speaker:

We can only lose five minutes worth of data.

Speaker:

This is.

Speaker:

You're setting your recovery point objective.

Speaker:

What, to what point must you recover?

Speaker:

And the fastest RPOs in the business are going to come from these types

Speaker:

of products that we're going to talk about in the next couple of episodes.

Speaker:

The first of which being.

Speaker:

Uh, replication with a, with a, you know, a replication

Speaker:

system, you could get an RPO.

Speaker:

Uh, you could meet an RPO that's as close to zero as possible,

Speaker:

actually seen zero,

Speaker:

Yeah, I'm sure you can do zero.

Speaker:

It's just that

Speaker:

so

Speaker:

I'm gonna

Speaker:

could happen,

Speaker:

So it's in a very, very unique case.

Speaker:

So I don't know if you're aware of this, there is a mode on

Speaker:

some systems called domino mode.

Speaker:

It's out there in the industry, right?

Speaker:

And basically what you're saying is, every time I accept a write

Speaker:

on system, A, I'm gonna make sure it's replicated, it hit system B.

Speaker:

It's acknowledged before system A acknowledges a client.

Speaker:

Yeah,

Speaker:

you're guaranteeing every

Speaker:

basically, it's like a two phase commit for storage.

Speaker:

A two phase commit is a term from database, um, world, which makes

Speaker:

sure that it's written both to the log and it's written to the, you

Speaker:

know, before you acknowledge that the, the transaction has happened.

Speaker:

It's High

Speaker:

latency, very expensive, but if you have those requirements, there is an option.

Speaker:

But like you said, most

Speaker:

people will...

Speaker:

So the question is what matters more to you, performance or never losing any data?

Speaker:

So let's, speaking of which.

Speaker:

Let's talk about the difference between synchronous replication

Speaker:

and asynchronous replication.

Speaker:

Um, synchronous is basically what you just described that, um, well,

Speaker:

it's not necessarily what you

Speaker:

Yeah, not fully to domino

Speaker:

we are going to, we are going to replicate every byte.

Speaker:

As it's happening, right.

Speaker:

And literally every single, you know, the moment it, a change happens here, the rep,

Speaker:

the change is going to be replicated to the, to the destination, to the target.

Speaker:

Right.

Speaker:

And, um, the, the good side of synchronous replication is you're going to have.

Speaker:

The smallest RPO is possible, right?

Speaker:

Including, I guess I, I, I, I guess I was aware of it.

Speaker:

I didn't know that that, that I didn't know the term dominant mode, but,

Speaker:

but I was aware of that, that idea, the, this is going to give you your,

Speaker:

your tightest, uh, RPO as possible.

Speaker:

The challenge with, uh, this is a little thing called the speed of light.

Speaker:

The, um, it takes, so, you know, one of the things that we.

Speaker:

That we know is that we want the recovery copy to be some

Speaker:

distance away from the primary

Speaker:

in the same rack.

Speaker:

not in the same rack, not in the same town.

Speaker:

Um, there are sadly some stories from 9 11 where there were companies

Speaker:

that had their hot site, which was a synchronously replicated copy of

Speaker:

their data in the, uh, other tower.

Speaker:

And so there were companies that ceased to exist on 9 11 because they lost both their

Speaker:

primary and their secondary copy, right?

Speaker:

You don't want to have the two copies in the same place where a disaster

Speaker:

could wipe out the entire town, a flood, a earthquake, whatever.

Speaker:

And so you want to have some distance between.

Speaker:

uh, the two.

Speaker:

Now after 9 11, the, I don't know which, um, which governmental organization,

Speaker:

but after 9 11, the, the fed, the federal government, not the fed, which

Speaker:

we say the fed or you know what I mean?

Speaker:

The federal government started to dictate that A recovery that, that a, that a

Speaker:

synchronous copy of the data needed to be stored more than it was like 250 miles.

Speaker:

And, and everybody basically came back and said, that's not really feasible.

Speaker:

It's possible, but not feasible because going 250 miles.

Speaker:

At the speed of light, every single time you replicate, every single time you

Speaker:

update a block on every piece of storage, um, basically really pulls down the, um,

Speaker:

the performance of the primary system.

Speaker:

And the other thing to also mention is when you're doing synchronous replication,

Speaker:

you want to make sure that you're...

Speaker:

Destination system, right?

Speaker:

Your replica copy is as performant as your primary copy, right?

Speaker:

Because otherwise, if it starts to lag and slow down, eventually you're going to

Speaker:

start falling further and further behind.

Speaker:

And usually these technologies, after a while, they'll just stop accepting writes.

Speaker:

And they'll kind of fall back into like an asynchronous mode and

Speaker:

have to then catch back up, right?

Speaker:

Which you don't want because that affects your RPO.

Speaker:

Yeah, you don't want.

Speaker:

Yeah.

Speaker:

So yeah, so it's got to be basically the same, if not better

Speaker:

performance on the backup system, which is crazy talk to most people.

Speaker:

Right?

Speaker:

Um, but that, but that is if you want an RPO of zero.

Speaker:

Synchronous replication is literally your only choice, right?

Speaker:

Let's talk about asynchronous replication and that is basically

Speaker:

not that we're, we're not, we're going to build essentially a buffer.

Speaker:

Uh, of writes as the, know, and we try to not let the buffer get full, right?

Speaker:

The, basically the, what it does is it disconnects the, the writes on the

Speaker:

other side with the writes on this side, we may, and we'll try to keep up

Speaker:

with the pace, but writes can continue to update and you might get a spur of

Speaker:

writes, uh, on the, on the primary side.

Speaker:

And then they fill up the buffer and then, you know, the, the buffer gets emptied.

Speaker:

And that I think works as long as essentially your, you know, your bandwidth

Speaker:

and your latency is, is sufficient.

Speaker:

Um,

Speaker:

And,

Speaker:

go ahead.

Speaker:

and I just want to chime in that I know you called it async and async

Speaker:

threw me for a while because in my mind async is different, but that's

Speaker:

just based on my background, right?

Speaker:

Um, other vendors might refer to this as like semi synchronous, right?

Speaker:

Where it is not fully synchronous.

Speaker:

Right, but

Speaker:

Well, not synchronous.

Speaker:

Asynchronous means

Speaker:

know, I know, I know, but this is just some vendors.

Speaker:

So if you are looking at vendor terminology, right, and looking at

Speaker:

their vendor technical specs, what, so here's a question for you, Curtis.

Speaker:

What are some things that a user can look at for a vendor's technical

Speaker:

specs to determine if they support async replication, like what

Speaker:

we're talking about right now?

Speaker:

Well, I mean, I don't know what other words they would use to describe

Speaker:

it,

Speaker:

I know NetApp uses semi synchronous.

Speaker:

For what you're describing.

Speaker:

whatever.

Speaker:

That's just mark.

Speaker:

That's just marketing BS.

Speaker:

Okay.

Speaker:

It's either synchronous or it's not synchronous.

Speaker:

You're either replicating everything as it's happening or you're not.

Speaker:

Okay.

Speaker:

Like, so the question is, so the question to the vendor, it's not, is, can my target

Speaker:

copy ever get behind my primary copy?

Speaker:

And the answer is yes.

Speaker:

And it's like, well, well, how much are we talking?

Speaker:

Right?

Speaker:

So the question is, because that's going

Speaker:

the, so that's

Speaker:

actually the key, yeah.

Speaker:

Yeah, exactly.

Speaker:

So if you can build up a buffer of an hour of changes, you could lose an

Speaker:

hour of changes on the, on the target end if you lost your primary, Um,

Speaker:

and.

Speaker:

what if you said like, I want, so, okay, so then I'll take back my comment.

Speaker:

In your mind though, if I said I want to have a replica copy

Speaker:

updated and my RPO is 8 hours.

Speaker:

That's still async.

Speaker:

Well, what else would you

Speaker:

Okay.

Speaker:

No, just, just checking.

Speaker:

Yeah.

Speaker:

I mean, again, you're either synchronous or you're not

Speaker:

So then, so then I take back my comments, right?

Speaker:

So if I look specifically, that's why I just wanted to make sure.

Speaker:

You stand corrected,

Speaker:

Yes.

Speaker:

So if I look at, sorry, I used to, I'm more familiar with NetApp and NetApp's

Speaker:

replication technologies, right?

Speaker:

But if I look at NetApp, right, they have synchronous, which we talked

Speaker:

about, which matches perfectly with what we were talking about.

Speaker:

They have semi sync where it allows some amount of lag.

Speaker:

I think it's up to 60 seconds.

Speaker:

Right.

Speaker:

But it does like what you said, buffering, and then they have

Speaker:

sort of their async, right?

Speaker:

Where it allows you to go beyond, I think it's from five minutes to, you can set a

Speaker:

replication every five minutes to days,

Speaker:

And those are just two different, those are just both asynchronous with different

Speaker:

settings, as far as I'm concerned.

Speaker:

Um, and, and you could also, like, if you were a net app and I don't think they do

Speaker:

it like this, but if you waited until you took a snapshot and then replicated only

Speaker:

the bits necessary for that snapshot, that's still kind of asynchronous,

Speaker:

just a technology.

Speaker:

That's a means.

Speaker:

yeah.

Speaker:

But generally here, we're not talking about snapshots and things, because

Speaker:

that's what we're going to talk about.

Speaker:

In a following episode, generally what we're talking about is two,

Speaker:

you know, block devices that are being continuously updated to be

Speaker:

exactly the same with possibly a lag.

Speaker:

If there's a lag, then it's asynchronous.

Speaker:

If there's no lag, then it's

Speaker:

Does it have to be, so I'm going to ask another question.

Speaker:

So

Speaker:

you are

Speaker:

I know I am.

Speaker:

So you said block devices.

Speaker:

Would you consider anything that also replicates a VM at a VM block level

Speaker:

Yeah.

Speaker:

Yeah.

Speaker:

I mean, in the end, it's still a block device underneath, right?

Speaker:

It's not what we're replicating.

Speaker:

It's just that, um.

Speaker:

We don't, we don't have this conversation when we're talking about object,

Speaker:

Or files.

Speaker:

We don't, we don't, well, even, even file replication, we don't really talk

Speaker:

about synchronous and asynchronous when we do file replication, I don't think.

Speaker:

I think that's always asynchronous.

Speaker:

At least I don't think so.

Speaker:

I think this is generally a block discussion, right?

Speaker:

There may be exceptions to that rule, but generally this is a block discussion.

Speaker:

Even if this is a NetApp, right?

Speaker:

Really what you're replicating underneath the NetApp is file.

Speaker:

It's a file device, but underneath you're replicating blocks.

Speaker:

Um, all that, all that discussion just from one word.

Speaker:

Um, so, you can also have a hybrid of these two things.

Speaker:

Can you think of an example of that?

Speaker:

Is

Speaker:

Yeah.

Speaker:

So take your bank example, right?

Speaker:

So you're a financial customer in New York.

Speaker:

You want to make sure that your data is synchronously

Speaker:

protected to someplace nearby.

Speaker:

So maybe you go over the river to New Jersey using synchronous

Speaker:

replication, and now you need to make sure that you're protected in case

Speaker:

something happens in the East coast.

Speaker:

And so you might use async replication and send that copy off to say, Utah.

Speaker:

that when we go through the woods?

Speaker:

Yes.

Speaker:

See what I did there?

Speaker:

Over the river and through the woods?

Speaker:

Um, yeah, so you might, and that was, I remember back in the day,

Speaker:

we had a synchronous copy that was literally like right next door.

Speaker:

I mean literally, when I say next door, I mean Like it's the next

Speaker:

rack over, um, that was, uh, like an A or asynchronous replication.

Speaker:

And then actually, will you remember Symmetrics and or,

Speaker:

or, or EMC and, and NetApp?

Speaker:

They, you'd have like three.

Speaker:

You'd have like four storage arrays, right?

Speaker:

You'd have the,

Speaker:

Oh, you can have

Speaker:

a lot more.

Speaker:

Yeah,

Speaker:

you'd have the synchronous that was right next to it, the asynchronous

Speaker:

that was down the street.

Speaker:

And then another asynchronous that was the other.

Speaker:

Yeah,

Speaker:

and sometimes there are customers who could use it for like data

Speaker:

distribution use cases, right?

Speaker:

So I need to

Speaker:

make sure that my database is available locally in different countries.

Speaker:

So I'm just going to replicate it all.

Speaker:

yeah.

Speaker:

So synchronous, both forms of replication have a lot more applications

Speaker:

than just backup and recovery.

Speaker:

Um, so I'm glad you mentioned, you know, or you VMs, there is another.

Speaker:

Not file and not VM thing that gets replicated, which is very, where

Speaker:

we still have sort of the same discussion about synchronous versus

Speaker:

asynchronous, and that is databases,

Speaker:

Application level synchronous

Speaker:

right?

Speaker:

And, and, and I'd say that in that case, basically.

Speaker:

When we, again, that can definitely have two phase commit, right?

Speaker:

So in the database.

Speaker:

You're replicating a transaction or, or something.

Speaker:

I don't have a better word.

Speaker:

I know not every database has transactions, but I

Speaker:

don't have a better word for

Speaker:

It's the atomic unit, whatever it is, or sorry, the unit of

Speaker:

Yeah, an atomic unit of a thing, you changed a thing in the database, right?

Speaker:

And you're replicating that, rather than the blocks underneath.

Speaker:

And, well, you could be replicating the blocks underneath, and that's

Speaker:

what we're talking about over here.

Speaker:

But at this point, we're talking about database level,

Speaker:

application level replication.

Speaker:

And when you do that, you have the choice of either immediate

Speaker:

consistency, or eventual consistency.

Speaker:

Which is essentially analogous to synchronous and asynchronous.

Speaker:

And, and what made me think about it was when you were talking about that,

Speaker:

there are other reasons that people replicate and, um, there are databases

Speaker:

that allow replicating data, you know, across the world and the best.

Speaker:

Example that a lot of people are used to is the DNS database, right?

Speaker:

That is an eventually consistent, replicated, um, system.

Speaker:

If you've never, if you've never made changes to DNS.

Speaker:

Let me tell you, it can take a minute.

Speaker:

I recently, literally in the last week or two, I've been making changes to

Speaker:

various websites that I own to, you know, change their, uh, DNS entries

Speaker:

and CNAME entries and things like that, and they don't all immediately happen.

Speaker:

Um,

Speaker:

And yeah,

Speaker:

but

Speaker:

they will eventually get there.

Speaker:

That is essentially the database equivalent to, um, to, uh,

Speaker:

asynchronous replication.

Speaker:

Now, let me ask you a question.

Speaker:

Prasanna,

Speaker:

mm hmm.

Speaker:

if replication is so amazing and we can have the tightest RPOs, and, and, and

Speaker:

I think we can agree that Tighter RPOs are better than looser RPOs, right?

Speaker:

An RPO of zero or a minute is way better than an RPO of 36 hours, which is what

Speaker:

we typically get with a backup system.

Speaker:

Why don't we do replication for everything?

Speaker:

And by the way, money is not the issue.

Speaker:

You knew what I was going to say, uh,

Speaker:

It's not, well, it is an issue besides money, right?

Speaker:

Because replication is

Speaker:

yeah, so the

Speaker:

don't we do that?

Speaker:

so the biggest problem with replication is for most of the replication technologies,

Speaker:

when you replicate it, you only have that one copy on your target system, right?

Speaker:

And so, If you had a user who dropped a database table and you replicated that

Speaker:

command over to your replica, guess what?

Speaker:

It gets dropped on your replica as well, and you have no ability to recover.

Speaker:

And

Speaker:

so that's one of the biggest limitations of relying on replication,

Speaker:

or at

Speaker:

And same would be true of any change on the, on the primary system, which

Speaker:

includes things like ransomware, right?

Speaker:

So if there's a, if there's a hacker goes in.

Speaker:

and deletes all your tables, or just, you know, corrupts your, your data,

Speaker:

deletes all your files, whatever.

Speaker:

It just, if a virus does, whatever, whatever, you know, if there's any

Speaker:

kind of cyber attack on the primary system, it simply makes that attack more

Speaker:

efficient,

Speaker:

And because the entire purpose of replication is to create a duplicate

Speaker:

copy on your target system, right?

Speaker:

Immediately, if you ask for an RPO of zero, you're going to get an RPO of zero,

Speaker:

now, now I will say, I will put a caveat that, wait until we talk about snapshots,

Speaker:

we will come back to this specific point,

Speaker:

Right.

Speaker:

There, There, are, there are back backup data protection based systems

Speaker:

that use replication at their core.

Speaker:

And don't have this problem.

Speaker:

This is just, just replication, right?

Speaker:

It's like, why don't we use raid for everything?

Speaker:

And then we don't need backup.

Speaker:

Well, same exact reasons, right?

Speaker:

Exactly.

Speaker:

Uh, raid does help with media failure and equipment failure.

Speaker:

And, um.

Speaker:

Site failure and power failure and, and, and, and, but it is completely defenseless

Speaker:

against cyber attacks and stupidity.

Speaker:

Or logic issues.

Speaker:

Well, I'm just, you're being nice.

Speaker:

See, this is why this is,

Speaker:

Well, I was thinking software bugs, not user logic.

Speaker:

Oh, okay.

Speaker:

Well, that's still stupidity.

Speaker:

It's just, you know, Different

Speaker:

Yes.

Speaker:

person who made the bug.

Speaker:

Um, well, this has been fun.

Speaker:

Uh, but yeah, this whole thing of, um, this idea that replication is where

Speaker:

we're continuously updating the data.

Speaker:

Now, now that you see like this idea of, of, um, you know, why we were wrestling

Speaker:

over the word continuous, right?

Speaker:

Because.

Speaker:

Is asynchronous continuous?

Speaker:

I think it's still a continuous process.

Speaker:

It's just continuous with an asterisk, continuous with, with, with a buffer.

Speaker:

Um, and I think that, well, in fact, I know that today is.

Speaker:

Way better than it was back in the day.

Speaker:

You back in the day, you had to have the same vendor on both sides.

Speaker:

It, it, it sold a lot of EMC, right?

Speaker:

You had to have the same vendor because the replication was based on the box.

Speaker:

And now there are a number of products.

Speaker:

one of which we've had on here, right?

Speaker:

We had Datacore on here, which, which does replication between, you know,

Speaker:

uh, disparate, different architectures.

Speaker:

You can have a very expensive primary system.

Speaker:

And a less expensive target system.

Speaker:

You can use it as a way to upgrade and move your technology.

Speaker:

You can move your older systems to be the target system, knowing that if you

Speaker:

need to use it in a disaster, you will suffer a performance loss, but you

Speaker:

don't have to, like you did back in the

Speaker:

day.

Speaker:

You don't, you know, when you primer, you don't have to have the same, you know,

Speaker:

and so you get, you could cost savings.

Speaker:

You don't have to do it that way, but it gives you the flexibility.

Speaker:

And I think that modern day replication systems are much more

Speaker:

forgiving of things like You know, outages and, and things like that.

Speaker:

Right.

Speaker:

Um, and, but, you know, but they still don't protect you from, you know, mistakes

Speaker:

and cyber attacks, which is why we have to talk about the alternatives, which we'll

Speaker:

do on later episodes, but, uh, so you were, you were unusually, not unusually,

Speaker:

you were excessively argumentative in this episode, but I thank you anyway.

Speaker:

that, that, I try, I basically spent all of my, almost all of my career

Speaker:

working on replication technology, so.

Speaker:

Yeah.

Speaker:

And I, I, yeah, you worked at a company that was like, it was trying

Speaker:

to say, well, there's asynchronous and there's near synchronous.

Speaker:

And I'm like, those are two words for asynchronous.

Speaker:

As I was editing this episode, I realized that.

Speaker:

I was being a little too pedantic about continuous or not continuous.

Speaker:

The thing with replication that really sets it apart from other data

Speaker:

protection technologies, is , one is that the copy is an actual copy.

Speaker:

It is a, it looks exactly like the original, unlike what we do in the

Speaker:

backup world, where we tend to put stuff in a container or something like that.

Speaker:

Um, that's one big thing.

Speaker:

That's separate separates replication from other backup technologies.

Speaker:

And then also the idea is that it is, it is incrementally updating that.

Speaker:

Whenever we do it, whether it's continuous asynchronous or even periodic.

Speaker:

I really forgot about how, like there are vendors like NetApp.

Speaker:

That actually do periodic replication.

Speaker:

There's nothing wrong with periodic replication.

Speaker:

It's still replication.

Speaker:

Um, so you've got continuous asynchronous and then periodic

Speaker:

that, what, what sets replication apart from everything else is this.

Speaker:

Whenever we do the next replication.

Speaker:

What we're doing is we're transferring the, the, the things, whether it's

Speaker:

database records or blocks, we're transferring the things that have changed

Speaker:

since the last time we replicated.

Speaker:

So I hope that.

Speaker:

Helps Muddy the waters.

Speaker:

Just a little bit.

Speaker:

You're either synchronous or you're not just like in.

Speaker:

In the following episode, we will be talking about a

Speaker:

term near continuous, which I actually coined, but, uh, a lot

Speaker:

of people think isn't a thing.

Speaker:

I don't know

Speaker:

if

Speaker:

think, they're like, you can't have...

Speaker:

Then again, they'll say that continuous is a...

Speaker:

Um, is, is, is a binary, uh, condition.

Speaker:

So anyway, well, um, I want to again, uh, say thank you to our listeners.

Speaker:

You are why we do this.

Speaker:

Remember, this is an independent podcast and the opinions that you hear are ours

Speaker:

I hope this episode has been helpful.

Speaker:

That's a wrap.