There are dozens of things that people do to protect their data

from loss, but many of them are worthless when you actually need them.

In this episode, we'll learn what turns a copy into a backup so that

you can make sure that anything you think is a backup actually is one.

We'll also talk about some important backup concepts like

multiplexing, incremental backups, block-level incremental backups

and source side deduplication.

Hi, I'm W.

Curtis Preston, AKA Mister backup.

And I've been specializing in backup and disaster recovery for over 30 years.

My podcast turns unappreciated backup admins into cyber recovery heroes.

This is the backup wrap up.

. Hi and welcome to the show.

and I have with me a guy who I think is super jelly of the new

toy that I put in yesterday.

Prasanna Malaiyandi.

How's it going, Prasanna?

I'm good Curtis, and yes, I am jealous of your toy, but I will have to start

sending you a bill for consulting fees, when you inevitably, or if

you inevitably run into issues.

Huh?

Yeah, because as you recall, I didn't, I purchased this like without even talking

to you, which is rather atypical of me.

because I, we talk about so much and, and basically what we're talking about

is a Firewalla, which I've had my eye on for a while and then after I realized,

so I've switched internet service providers and now they're telling me

that I'm hitting the bandwidth limit already and which is highly possible

given that I do, this, I realized that I had no bandwidth monitoring tools.

I have this really nice, mesh router system.

Wire, the wire, the wifi mesh, but that's put been put an access

point mode, which then of course offers no bandwidth monitoring.

And then I had this Cox router, which offered me nothing.

And so I replaced the Cox router with the Firewallet Purple SE and man.

Super simple to put in there.

And now I have these like super, stats.

And I get these, I'm going to, at some point I'm going to have to disable

the notifications because it's like Curtis is playing games on his phone.

Curtis is watching YouTube videos on MacBook Pro A, right?

It's literally, it's like Curtis has downloaded 3.

56 gigabytes of video on his, and I'm like, okay, this

is going to get old pretty

I have two questions for you.

Yeah,

The first question is, Did you figure out what was consuming

your data cap or data usage cap?

not yet.

Cause it's only, it hasn't even been 24 hours, but I do have a pretty good guess

and I think it's right in front of me.

we'll see what is weird.

And again, I didn't want to talk about this too much, but what is weird is I

get these, some of the notifications it's, Mac book pro uploaded 3.

5 megabytes of data to LinkedIn.

At 3 45 a.

m.

And I'm like, what, like, why is my laptop uploading three and a half

megabytes of anything while it's just sitting here and I'm sleeping somewhere?

that's weird.

weird.

Yeah.

So anyway, so yeah, go ahead.

Yeah.

since you've decided to get rid of the Cox router, can you not just

use your Wi Fi mesh as a router?

I could have, but then I would have to completely redo my

network architecture, which as you recall, was a really big thing.

That's true.

And I actually really liked the firewall features of this.

That's, that was what would, what really drew me to it.

And I'd been thinking about it and this was that final excuse to get it.

and, I'm really enjoying the security aspects

glad.

So there, for people, if Mr.

Backup can learn networking and firewalls, so can you.

Yeah.

It's certainly not my, my forte.

but I want to talk, it's time for the news of the week.

, the big news, I think, of the entire IG world, everybody seems to be

talking about it, is this MGM hack.

I can't, can you imagine?

if you've been living in a hole, They shut down MGM and Caesars and all

of the hotels attached to MGM and Caesars, which is like half the strip.

And they shut down like card keys, slot machines,

ATMs.

everything.

ATMs,

So for, so for our listeners, Who may not know about this,

MGM is a hotel chain, right?

They have a bunch of things as well, right?

Like various hotels like Caesars and MGM, and they are in Las Vegas

and there are casinos, right?

So you can stay there, you can gamble there, right?

They make lots and lots and lots and lots of money.

but not in the last week or so,

And they got hit by a cyber attack, on the week of September

20th or so, I'm guessing.

yeah, and The I think that's the saddest part about this and by the way as of

today There's been half a dozen lawsuits attached because there's, threat of a,

PII leak, personal information leak.

And so there's been all sorts of worries about that.

So there's, a half a dozen, what do you call it?

Class action or lawsuits that are attempting.

To achieve class action status that have been filed.

I think the saddest part here and the way I like and we'll put the

link to this particular article in the show description, the

heading here targeting layer eight.

I've heard of the seven layer networking model again With my extensive

networking experience, the OSI model.

What is Layer 8,

People?

it's people.

Yes.

So layer 8 is people, which is probably the weakest part

of the entire stack, I'd say.

Yeah.

You are the weakest link!

Yeah, so what, how did they get in?

So they basically targeted an employee, right?

Who had the right level of access and They basically were able to gain access into

their Okta environment as a super admin.

But how did they do that?

That's the

Oh, so how did they do that?

They basically tripped, tricked their IT help desk?

That is so bad, right?

they somehow got...

Access to a privileged account, right?

according to the powers that be that they stole a password or they

hacked Active Directory somehow.

So they were able to attempt to log in, but they were stopped by MFA

which is a good thing, Okta, but then

They were able to convince the help desk that they were the person in

question and get them to reset MFA.

Now here's a question.

Do you think that employee is still there at the company?

And

is one of those

you be blaming the person

so I'm going to fast forward like 30 years.

Okay.

so 20, what would that be?

2053.

There's a guy he's going to be called Mr.

MFA and he's going to have a podcast dedicated to security because like my

career started with a screw up of this.

not quite this magnitude, but my career started with this.

And so I, My personal opinion, I don't know if this person is, has been fired.

I think they should only be fired if they didn't follow the processes that had been,

established and they

set out for them.

Potentially they should be disciplined.

I don't know if firing, if.

If termination is the appropriate, they should be disciplined.

If they followed the procedures that had been laid out for

them, process, people, right?

Then technology.

If they had been, we just had a podcast about that.

If they followed the procedures.

have been given to them.

Then I think some massive leniency, then you update your procedures,

et cetera, et cetera, et cetera.

I think of a massive, outage that was caused at a major, software

vendor that I worked with.

I'm trying to be, I'm trying to be very cagey here, where the backup

operator followed his Procedure that they had two parts of the app

that had to be shut down in order to do a backup because they couldn't

synchronize the two backup systems.

And so every two weeks they would shut down these apps and,

and then do a backup offline.

And this person.

The backup operator just did what they were told to do and shut down these

apps at the most critical time of the year when the apps were needed, right?

The person was just doing their job.

That person should not be fired.

That person should be You know, you changed the procedure.

So I don't know what happened here.

Yeah.

You train.

Yeah.

I hope some leniency was there.

if the person was fired and, I'd love to have them on the podcast.

But anyway, what could we learn from this, from this news here?

Prasanna?

basically that, one is even if you have the greatest technologies in

place and the greatest processes in place, people will always exist

never underestimate the power of people to do dumb things.

I do think that perhaps what's in order here is an update to process.

And the process should be when, cause you have to be able to reset MFA.

When resetting MFA, it should require many more, bells and whistles

and levels of authentication.

And we need to identify, we need to identify that this person who

calls in that says that they're Steve, we need a way to identify

that Steve is actually Steve.

so you create a process around that, that really verifies that someone who they are,

and especially,

you reset MFA.

and especially when it's someone of, with that level of privilege.

Especially, super especially, that's not a word, but yeah,

that, oh, I feel for these guys.

keep, abreast of, this story because it is going to get worse before it gets better.

And that's the news for this week.

So what I thought we would talk about this week and the backup to basic series is,

I've got it defined as, backup methods that support a traditional restore.

So basically the backup methods that I grew up with that are still,

in

Relevant.

Yeah.

in, yeah, right?

we like to live in a world where everybody's using the

latest and greatest, right?

And nobody's doing this old, full and incremental backups and stuff.

Nobody's doing that.

And that's just not right.

So we need to talk about these, these methods and see what we can get out there.

the first thing, I just have to, again, I'm, I'm, we're doing this based

on, my book, Modern Data Protection, There's a cover for those of you

watching via video, all, all three listeners that are watching via video.

I think it's 10.

There's maybe 10.

the number's actually gone up since we've been putting them on YouTube.

Oh, there you go.

the, I've got this thing in here.

so this is from chapter nine and talking about backup and

recovery software methods.

And the first thing I had in there was, is everything backup?

So there was a time when backup was well defined.

Backup was copy something to tape and then put that tape in a box,

right?

It was so simple back then.

Yeah, it was so simple back then.

Yes.

so I, as quote, Mr.

Backup, I see backup a lot broader than I think a lot of people do.

A lot of people, when they say backup, they go, Oh, this isn't backup.

This is, to me, backup is anything really that protects the data, the

way backup protects data, right?

And so I'm defining backup rather broadly as anything that is a copy of data

stored separately from the original.

that can be used to restore the original if it is damaged.

There's a lot of things that qualify for backup as backup under that

so let me just give you some examples and see if you think they qualify.

Okay.

So take a copy on tape,

Yes.

a copy in AWS S3.

A copy of the data that's in S3, which is separate from

The

your not.

Yeah.

yes.

okay?

a copy replicated from one storage system to another storage

system from the same vendor.

as long as...

there's a caveat here, because you used the word replication.

I need the ability, is it replicated in such a way that if

I damage production, so

that doesn't qualify as being stored separately.

Replicated with separate retention of the copies on the destination.

Okay.

Yes, I would call that a

Okay, snapshots on a production system, on a production storage

array that does not include AWS S3,

thank you for, yeah, so snapshots on the same array.

no, End of story, not a backup until it's copied somewhere

Okay, and then doing what you were recently doing when

editing the podcast, right?

Downloading a copy from the cloud onto your local system, copying it

to a different directory, and then copying it to yet a third directory.

On your local system.

Is that local system considered backups, each of those copies?

again, we're storing the data in a separate place that

has a separate risk profile.

Etc.

yes,

As long as the copy, the original

copy was in the, cloud.

it's also about, the purpose of why I'm doing it, right?

If the purpose of downloading that is to serve as possibly a backup, right?

Because there's a lot of times that we download data That

is not for backup purposes.

Now, it could accidentally become a backup if it's the only

copy that you have available.

But, just because I copy doesn't necessarily make it a backup.

It might be an archive.

And then the last example.

taking pictures on your iPhone and using iCloud to sync

your copies to iCloud photos.

Not a backup.

Because,

is that?

for two reasons.

One, which is really the primary.

And that is specifically in terms of Apple iCloud.

But the biggest thing is that it's synchronized.

that's the key.

That's, you're, you asked earlier, you delete a picture in your phone or some

app, delete, some like ransomware deletes a bunch of pictures in your phone.

It synchronizes that deletion up in the cloud and they go byebye, right?

It is a synchronized copy, not a backup.

it is stored separately, but if you delete it here and it gets deleted

there, that's not a backup, right?

Just like we were talking, before.

And that's one really important reason, possibly the most important reason.

But the other is that there's a feature in iPhone that...

It says we can store low res copies on the phone and the high res copies in

the cloud, which means that not only is it a synchronized copy, the only true

copy of your photo is in the cloud.

It's only one copy, which means you need to be backing up iCloud.

and by extension also, Google photos if you're an Android

person, so yeah, not a backup,

okay, no,

which we had a whole podcast episode about that.

How to properly back up your iCloud account.

yeah,

were good examples.

I think those are a lot of things, like you said, right?

It's not always easy to say, is it a backup or not?

Unless you dive into the next level of questions and ask, okay, is it really a

yeah,

Does it meet these

I think.

or not?

I think you did a good job of, the different categories, like that

thing of, if it's fully synchronized, whether synchronous or asynchronous,

if it's fully synchronized and if I delete the production and

it deletes the data, the copy,

that, That's not a backup.

right?

unless that copy has the ability to undo that.

If it does, then, I would change my answer, right?

And so like a NetApp synchronized filer, I would consider that

other copy, that would be backup,

other things that are not a backup, one that you didn't mention would be,

the recycle bin in your Microsoft 365.

That is not a backup, right?

It's not stored separately.

it's just, records in a database that have been flagged as deleted.

They haven't gone anywhere.

They're sitting right next to the production data.

So yeah,

Okay.

And then the other one is,

So in your opinion, does backup require you to always be able to go

back to a point in time that could plausibly have existed in the system?

And the reason I'm asking this is if I look at, I know email archiving comes

up a lot and sometimes people are like, oh, that's the same as backup.

But with email archive, you're just getting all the data that's there,

whether or not your mailbox actually looked like that, your inbox looked

like that or not at any point in time.

Yeah.

So backup.

requires restore, right?

For it to be a backup, you need to be able to restore it to the way it

looked at some point in time, right?

yeah, that's a really good question, Prasanna.

it's one thing to say a file, but, if you cannot, if you cannot bring

the thing that's been damaged back to its You know, back to before it

was damaged and that it comes back to the same way as it was before it was

damaged, then you don't have a backup.

You copy of the data, right?

And an email archive is a perfect example of that.

You have a copy of the data, but it was stored for a different purpose.

It was stored for archive, which means it wasn't designed to be put back into the,

the state it was in,

yeah, the state that it was in,

right?

so you might be able to restore all the email, but you won't be able to

restore folders and things like that.

A good backup should bring the thing back to the way it was before it was damaged,

however it let's go back to a time when tape drive started getting, so here,

we're going to talk about a feature that is now for many people, passe, right?

it's not really necessary because they no longer use tape as their primary target

or their initial target of backups.

and that is this concept of multiplexing.

And it goes back to, there was a time when we

Way back in the days.

right back in the day.

So multiplexing, do you want to define multiplexing or explain it?

Yeah, multiplexing.

Yeah, I, let me attempt to, I know I wasn't aware of this before we started

doing the podcast and you explained everything about tape and I know we've

had a bunch of folks, tape experts on the podcast as well, but multiplexing is...

to solve an issue where tape requires you to write at a certain speed.

If you don't, it's bad.

And tapes got faster and faster, but the problem was pumping data into the tape

device itself wasn't going as quickly as the tape speeds were increasing.

And so in order to solve that, what they decided to do was say, okay, Let's have

multiple clients feed data into the tape device at the same time, and we will

multiplex or basically write all those streams into the tape drive at the same

time, keeping the tape device happy.

While still being able to do all the backups.

Yeah, another word for it would be interleaving.

You did great.

basically putting all, chopping them up into pieces and then

putting together into one, turning a bunch of streams into one stream.

And when we first started, we used multiplexing settings of four

Which means four different

turn and.

Yeah, four different clients being combined into a stream to

make a tape drive happy, but tape drives got faster and faster.

The clients didn't get faster.

And so by the time I left, by the time I used my last tape drive in

production, we were up to 36, right?

We were up to 36 streams together to, to make an individual tape drive happy.

And the reason,

I was gonna ask why.

Yeah.

Why were clients not fast enough

yeah.

So the reason that this was bad is that, what, why is the only reason we back up,

to restore

right?

So when you

go to

do a restore,

Yeah.

yeah.

When you go to do a restore, you have to read all 36 streams

and throw 35 of them away.

So your tape drive, the speed of your restore is going to be 1 35th.

Of what it could potentially be if it hadn't been multiplexed,

But if you're never doing restore tests, it doesn't really matter.

Until you actually need to restore the data.

yeah, if you're You're killing me you're killing me yeah, so it was one

of these things where it was a Cut your nose off to to spite your face, right?

So We felt that it was But it was a necessary evil.

We, you could only restore if you've got backups done and we could only get

backups done reliably if we were using multiplexing, but we knew that it was

creating this problem and ultimately this was the undoing of tape from

a backup and recovery perspective.

We switched to destaging and.

these other things to undo this, necessary evil.

But, it, it was a mess.

But that's what multiplexing is.

So if you've heard about multiplexing, you don't need to do multiplexing

if you're backing up to disk.

Because disk can write at whatever speed you tell it to write at.

And it can write a bunch of things at the same time.

And you can give it 36 streams and it can write them all at the same time in

separate places of the disk in such a way that when you go to do a restore,

you don't, you're not, you don't have to read all of them to read one of them.

What?

That was my yes.

disk is fast enough, but

Yeah.

Well, it's not,

a disk

drive has a certain

number of IOPS it could handle.

And therefore, as long as your system is big enough.

To handle all of them in peril.

yes.

they're, disk drives are not, Unlimited bandwidth, unlimited IO,

et cetera, et cetera, et cetera.

Yes.

but the point of the way that it lays the data, you don't have to lay the,

you can lay the data however you want and then read it however you want.

there are, again, there are limits to everything depending on how

much you fragment the data and all that kind of stuff, right?

But it's still way better than tape from that perspective.

All right, next one's a whole lot easier.

What comes next?

What's the first type of, what's

let you tackle

what, no, I'll let you tackle this, Curtis.

So what's the, what is it?

The first type of backup that everyone should cut their teeth on.

what a full backup?

Is that

Yeah.

what you're saying?

Yeah.

so basically we're just going to talk about this concept of

full and incremental backups.

And probably everybody knows this, but this is a backup to basic series.

So a full backup backs up everything, an incremental backup

backs up things that have changed.

And the, there are different types of incremental backups, right?

And different people have different names for these different types, right?

terms you've probably heard, incremental, differential, cumulative incremental.

For a lot of people, cumulative incremental and

differential are the same thing.

for people that got stuck in Windows land, not necessarily so what's the

difference between an incremental and these other two things?

A cumulative incremental.

So an incremental is basically, Typically, Sunday you do a full backup, right?

Monday you need to do another backup.

Now, you don't want to do necessarily the entire full backup again,

because maybe that's too much data, you don't have enough time, etc.

So you'll do an incremental, which is basically whatever has

changed since the last full.

So since Sunday.

Sorry, since the last time you did a backup, I should say.

exactly, whatever's changed since the last time you did a

Yeah, so in that case, it was Sunday, so then Monday you get the incrementals,

now Tuesday you're going to do backup, and so you do another incremental, which

is whatever has changed since Monday,

Exactly,

and we just keep doing that, right?

Yeah, and

then if it's, yeah, if it's Sunday, right?

And now it's Saturday, how many tapes do I need to do a restore?

do you need...

The previous Sunday, plus the Monday, plus the Tuesday, plus

the Wednesday, Thursday, Friday.

You basically need to replay

by the way, by the way, I really, I really channeled the old Curtis there.

I did it without even meaning to, I said tapes, right?

Cause

that was the problem back then.

We literally had to grab for seven tapes, right?

Nowadays, we don't have to grab for seven tapes, but,

but you still have to do all those restores though, right?

So even in the case of, if a file existed Sunday, and then was deleted Monday,

and then came back on Tuesday, you would still end up having to do all of those

data, like basically you're replaying like a log, all the data that would

have existed on each of those days.

right.

The real problem is a file that was changed every single day.

You would actually restore that file seven times.

It's a lot of wasted effort.

That's just the idea of a increment or regular incremental.

Then we have a differential or a cumulative incremental.

And the difference between that is that it's going to, it's going to do

the thing that you said earlier, which is it's going to back up everything

that's changed since the fall.

And so what some people do is that they've stopped, they stopped doing

incrementals and they switched to differentials or cumulative incrementals

every day, and that way at the end of the week, I would need at most two tapes.

Right now, this whole thing has pretty much gone away in the world of.

disk based backups, right?

Because the whole reason that we did backups this way, is

that, first off, let me back up.

We used to do weekly fulls followed by daily incrementals.

Then we switched for, because when we went to automated tape libraries,

the whole process of managing the different tapes wasn't as a big.

Big of a deal.

So we went to monthly folds followed by daily incrementals or maybe

a weekly cumulative and right?

So you'd still need a maximum of seven tapes to do a restore But when

we switched to this this whole thing just became Kind of silly and moot and

whatever and you could back up, however, you wanted to back up and dedupe,

which we're going to talk about in a minute, dedupe really changed the game.

And, because it didn't matter whether you backed up full or incremental or whatever,

you still stored the same amount of data.

go

before we jump though, one thing that I think people might also hear in addition

to fulls, incrementals, differentials, and cumulative incrementals is also levels.

So maybe you could talk about levels.

I know sometimes it's specific to like Oracle.

And some databases, but maybe it might

no, that's a good point.

Yeah, thanks.

so the concept of a backup level, literally, this goes

back to the days of dump, right?

which was the command to backup Unix file systems.

A level zero was a full, a level one, And if you wanted to do increment, if you

want to do what we call the incremental backups, the way we, you would do a zero

followed by a one, followed by a two, followed by a three, followed by a four.

And, it got interesting because if you then lowered the number.

It would behave like a,

cumulative incremental, right?

so like you could do a zero and then you do a one.

If you then did another one, if you kept doing ones, you would get a differential.

You would get a cumulative incremental every day.

If you did a 0, a 1, and then a 2, and then a 1 again, it's just, it

basically, it always pointed back to the number that was the most recent

number that was lower than itself, and so it got complicated, and so

there were actually some people that

Is it they prefer

called Towers of

Hanoi, Yeah, which is based on the game, and I've got it in the book,

the Towers of Hanoi progressive thing, but I can't, it's like 0, 3, 2, 4, so

basically every backup, without doing cumulative incrementals, every backup,

every file that was changed would end up being on two tapes, which was just

an interesting way to, To minimize tape, again, this is all because we're doing

tapes, but nobody has tapes anymore.

So nobody cares.

But that's what levels were.

It was all the way up to nine.

and they still have this concept in, in things like Oracle Backup.

So the next thing to talk about is this concept called file

level incremental forever.

And the company that really put this out there was IBM with their product TSM.

And back in the day,

has been renamed,

idea is you

do one full, what's that?

Hasn't it been renamed?

It has, but I'm just saying they came out with it when they

came out, it was called TSM.

It's now like IBM spectrum protect, but, the idea was you do one full and then

everything is an incremental forever.

we never again do a full and this really saved a lot of

bandwidth and saved a lot of tape.

It came with a mess and that was over time and again, tape over time,

you could end up needing hundreds of tapes to restore a single file system.

you would need just one file from this tape and one file from that tape.

And since the hardest part of a tape is like, it was like

two and a half minutes just to get a tape in and, get it loaded and seek

to So the average point in a tape.

So I was not a fan of doing backups this way when we were talking about tape.

Was there a reason?

what was the use case at the time for that?

it was about saving tape, saving

storage.

It was about saving bandwidth.

the idea, there's nothing wrong with the idea of incremental forever.

It's just that their implementation.

Back in the day when it was all tape, even when they had disk staging.

So they would stage the disk.

So they wouldn't multiplex, by the way, they wouldn't multiplex.

They would stage the disk and then they would, do the backups to tape.

And this only applied to file system backups.

It didn't apply to database backups.

And, but literally you would need hundreds and hundreds of tapes

to restore a single file system.

And it just, I was never a fan of doing backups that way.

As long as we were backing up to tape and they had ways to they had, co location

and these various, and this thing called reclamation, because when you're doing

backups that way, you end up with a lot of tapes that have files on them that have

expired that are no longer needed, but you have other files on there that are needed.

And so you'd have to copy forward.

Yeah.

so that you could reclaim that whole tape and then reuse it.

And

That sounds like a

management nightmare.

An interesting engineering problem, but...

yeah, I was never a fan of doing backups that way.

and I'm even less of a fan now that we don't have to worry about tape.

Now we can just do incremental forever and just do it without all that

co location and reclamation stuff.

Cause on disk, to reclaim, you just delete a file, right?

On tape, you delete a file in the middle of a tape.

You have to reclaim the tape.

so that's file level incremental forever.

And then, with the advent of backing up to disk, Which finally

happened, I don't know, 20 years ago.

It's so funny.

We, we say the advent of something that happened 20 years ago.

When we finally started doing it, and once everybody finally went to, and by

the way, everybody still is not backing up the desk, it's still, there's still

a small contingent of people to back up the tape, so those people will really

enjoy the first half of this episode.

Now we have this concept of block level incremental forever.

Would you like to explain that?

Yeah, with block level incremental, I guess where I think of block level

incremental, I know there's various places you can think about it, is

when it applies to virtual machines and other sort of larger objects.

where it doesn't make sense, to back up an entire VM, doing full, or, incremental

backups away, if you think about how you would have done file level backups, right?

Why would I

Now, what, why would that be?

because I have a file which represents a disk, the entire file

doesn't change every time, right?

Parts of the file

it's, so we're talking to a VMDK file or VDK, For, for,

Hyper V, VDDK, that can't say VDDK.

I think it's VDDK.

Yeah,

I think you're right.

so you're saying if anything changes on there,

you're backing up the entire whole

do an incremental, exactly.

You're going

it's the entire file change, right?

So you're backing up the entire thing, but that doesn't make sense when

you have files which are say 10, 50, 100, 200 gigabytes and you're backing

that up every single time and so with block level incrementals What they

basically have done is say, okay What blocks have changed in this VMDK?

Let me just back those up, right?

Oracle also for databases, they do something similar, right?

Where it's hey Let me only back up the blocks within an Oracle data

file that have changed rather than backing up the entire Oracle database.

And how does the backup product know which blocks have changed?

Usually you have to rely on that vendor to tell you.

So in the case of Oracle, right?

You're usually integrating with Oracle RMAN via SBT or some other

mechanism where Oracle knows, okay, I keep track of the database blocks.

I know which ones are new.

Here is a list of blocks that you need to care about.

Same thing with VMware, when you have their, what is their SDK called?

VADP.

Yeah.

they've changed

the name.

Yeah.

They've changed the name, but basically they're, they have an API to talk to,

and they maintain a bitmap, right?

And then they just give you, here's a map of the bits that you need to go get.

These are the bits that have changed.

They maintain that.

And then the, there's an API for asking for those blocks,

now this is great for disk based systems because if you think about these are

all random spots in a file and so you can dump it out now It's up to figure

out like how you want to do this and I know we'll talk a little bit later

about deduplicated storage, but In the case of Oracle, typically you would just

dump it out as incremental blocks, and just dump it into a file, and now you

have all those blocks captured together.

In the case of VMware, they started doing that.

A lot of back up vendors would just dump it out as raw blocks, which makes sense.

but then, there are other optimizations you can do to do smarter things with

it, because with incremental block based backups, you still have to

restore from multiple files in order to stitch together the final actual image.

Yeah.

And you still have that problem.

That we talked about earlier where you may restore an individual block multiple

times if it changes multiple times, right?

the advantage is it's incredibly efficient.

And the, like when we talk about backing up VMs, I agree with you.

That's where this really shines.

Because back in the day, if we backed up VMs, And we just pretended they were,

physical machines and we were running full and incremental backups on them.

We were beating the crap out of these VMs.

So this is much more IO friendly, to the VMs, right?

So it's much friendlier on the VMs.

That's why we want to talk to the VMware API and get just

the blocks that have changed.

And it doesn't really come with any major downside compared to.

The alternative is because we're storing the data on disk.

Can I

ask one

yeah,

sure.

So we've talked about using block level incrementals for VMware, for databases.

Is there a reason it hasn't really caught on for files?

Because if I take a file and kind of split it up into blocks, right?

Could I get the same benefit?

Or is there a reason that it makes a lot more sense for

like VMs or virtual machines?

the benefit will be relative to the size of the file, right?

The bigger the file, the bigger the benefit that you're going to get.

And I would say that the reason it hasn't caught on is because of the next

thing we're going to discuss, right?

That solved that problem.

But yeah, I think about like files like PST files or maybe a big access

database or backing up like MySQL.

That's not file.

I mean, it is a file, but it's, it's actually a database.

Right.

I'd say the reason they didn't put a lot of effort is deduplication, which,

why don't we just talk about that now?

I know we've covered dedupe, just really quickly for those that don't

understand what dedupe is, the idea is that we're going to identify duplicate

segments of the data, and duplicate means that we've seen this data before.

we've done a full backup or we've done an incremental backup and we've

seen this part of the data before.

And for it to be truly considered ddu, you've gotta look at,

it's gotta be subfile, right?

It's gotta be part of, like we were talking about the V M D K

or the V D D K or a P S T file.

We've gotta be looking inside the file, slicing that up into chunks,

and then deciding this chunk.

We've seen it before, this chunk, we have not, And so there are two

different places that dedupe happens.

One is at the target, which is, like a box, like a data domain

or a quantum box or ExaGrid.

these boxes are target dedupe.

And then there's this thing called source dedupe, which.

really took off from a company that was called Avamar.

That company got sold to EMC, which I know you spent a little

time with, back in the day.

And, both of our previous employer did a source side deduplication.

Yeah, so with the target site is great because you could take it and

plug it in and place anywhere, right?

Because as long as it supports whatever the protocol your client is using, right?

You could just ingest the data and you get all the benefits of deduplication.

So data domain was.

Very popular initially for in virtual tape libraries, right?

So you had tapes, right?

People are constantly doing fulls and incremental backups.

That's perfect to deduplicate.

you plug in a data domain, it emulates the tape interface.

And now you just, your clients still continue writing to there and then all

your data gets deduplicated, right?

And so it doesn't matter if it's NFS or if it's SMB or if it's tape, right?

It just works.

yeah, it's like that firewalla box that I

bought, right?

It just, it just, it goes in and then it just works, right?

You didn't have to change anything.

With source dedupe, the idea is that, there's three parts

of the deduplication process.

There's the slicing and dicing, right?

There's the creation of a hash.

You run the chunk of data through Some sort of cryptographic algorithm, like SHA,

something, and then that gives you a value

and then that value, you have to look up that value in some

sort of hash table, right?

with target deduplication, all three of those actions happen on the

target, which is why it works so well.

You just send the backups the way you're used to sending them,

and then it does the magic.

It slices and dices, it hashes, and it does the lookup, and it figures out which

chunks of data are new based on that hash.

Source side, the first two happen on the source, right?

We slice up the data before we back it up.

We slice up the data We create a hash of the data, and then we ask

some magic person in the cloud, has this hash been seen before?

And the decision is made on the other end.

Yes, we've seen this, or we haven't seen this, and then we

send Or don't send the data.

To me, source dedupe is much more efficient than target dedupe.

The difficulty is that it is a much, it's a little bit baby in

a bathwater situation, right?

Because in order to get it.

You've got to do a forklift upgrade.

You've got to stop using, let's say, again, this is things have changed, but

back in the day, you had to stop using NetBackup and start using Avamar, right?

Stop using Networker or TSM and switch to, Druva, right?

You had to change your backup product to get this done.

Things change a little bit over time, right?

A lot of these products now support source dedupe.

But that was the main downside or still is the main downside.

If you want source dedupe, you've got to change your backup product,

uh, or you've got to change how you use your backup product,

assuming it starts supporting

yeah, and I would say at this point, probably a good chunk of products

either have their own source ID deduplication mechanism or they

work with deduplicated targets which allow for source ID deduplication.

for instance, integrating with ExaGrid or Data Domain from like TSM, Veeam,

Exactly.

Yeah, there are some that criticize it saying that, the slicing and

dicing and the creation of the hash puts a load on the client.

I have always argued that if done properly, that load created by the

slicing and dicing and hashing is offset by the significant reduction

of the load of transporting or not transporting 99 percent of the data,

right?

Yeah.

Other critiques of it have been that the restore speed wasn't great because of

how the data was stored on the other end.

And I would argue that's a implementation problem.

it's not a problem with the concept.

It's a problem with the implementation of the

And then the other thing to also mention about source side deduplication is

typically these are also using proprietary protocols, so you don't end up with a

lot of security issues you have around, say, having a target dedupe appliance

with NFS or SMB open to the world.

Yep.

Yep.

Agreed.

Yes.

Agreed that there is a security advantage to having the data sliced

and diced way before and then encrypted before you send it to the other

system instead of doing it over an unsecured protocol like NFS or SMB.

Exactly.

All right.

this episode, I think, got a little longer than we had intended for it to

get, but we covered a lot.

We covered a lot in this episode.

so basically, we learned about.

what is and is not a backup.

We learned about, multiplexing, full and incremental backups,

file level incremental backups, and source side deduplication.

Uh, it's a big episode.

what do you think?

Yeah, no, that covers a lot of what everyone talks about when you...

Do you ever refer to backup and restore, You gotta know these backup

technologies in order to be able to restore and protect your company.

These are things that you need to know.

All right.

And with that, I once again want to thank our listeners.

you are why we do this in Prasanna.

Once again.

great at your insights and questions as well.

Thank you, sir.

Thank you, sir.

Keeping me honest.

And, remember this show, the backup wrap up is an independent podcast and

the opinions that you hear are ours.

Not anyone else's, and also this is a production of BackupCentral.

com and, uh, produced and edited by yours truly.

And I just want to say, that's a wrap.