W. Curtis Preston: Hi, and welcome to Backup Central's Restored All podcast.
I'm your host, Debbie Curtis Preston, a k a, Mr.
Backup, and I with me, a guy that I'm hoping knows a little
something about plumbing.
Prasanna.
Malaiyandi, how's it going?
Prasanna Malaiyandi: I am good, Curtis.
W. Curtis Preston: how's that?
YouTube?
How's that?
YouTube, the, the YouTube knowledge that you have, does it cover plumbing
Prasanna Malaiyandi: kind of, sort of.
Unfortunately it's not something I'm as familiar with, but depends
on what you're looking to do.
W. Curtis Preston: Yeah.
I gotta replace tonight.
I gotta replace a couple of valves.
Um, Shut off valves.
Um, one of them, I, one of them is on the, the, it's the one that goes
behind my fridge and in order to like, it's like I, I can't easily,
it's a, it's a, what do you call it?
It's the, the threaded, no, it's not a compression fit.
It's the, it's a threaded pipe.
Like, uh, I really gotta get in there and get a hold of the, the upper part
of the pipe, uh, in order to hold that pipe still while I'm undoing it.
Cuz it, my first attempts at it did not, I don't want to be
cracking off a pipe up in the wall.
Prasanna Malaiyandi: Yeah, I was just gonna say that's, yeah,
the last thing you want to do.
I was thinking for the shutoff though, I'm assuming you took the handle off
W. Curtis Preston: The handle
Prasanna Malaiyandi: to give you more space
W. Curtis Preston: it?
No, there's no handle.
It's just, I mean, the handle's like integrated.
It's really small.
It's not a space problem.
It's just, uh, it's um, you know,
Prasanna Malaiyandi: Or they also say heat.
W. Curtis Preston: Yeah, give it
Prasanna Malaiyandi: So when I saw my, oh, so when we had our water heater
replaced, I was watching cuz I'm always curious to see what people do.
And our pipes were really, really bad at old.
because the last time our water heater was replaced was like 15 years ago.
And so the guy's like, you just heat it up really, really hot and then
you tighten it first to break the stuff off and then you loosen it.
W. Curtis Preston: Oh, interesting.
Yeah.
Maybe I'll see.
I'll take a look at that.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: Uh,
Prasanna Malaiyandi: took like in a settling torch or a butane torch
and then heated up nice and hot, and then he tightened it just a little.
W. Curtis Preston: what, I'm not gonna be able to do that.
You know why?
There's this
Prasanna Malaiyandi: Oh, the drywall.
Oh,
W. Curtis Preston: Plastic, worse than dry.
Prasanna Malaiyandi: yeah.
W. Curtis Preston: Um, so hey, we're gonna talk about, uh, we're, you
know, in our continuing, um, backup to basic series, you know, we talked
about, like recently we talked about a difference between backup and archive.
, in a previous episode we talked about protecting your backup data.
This one, um, you know, this one is, um, it's about backup levels, right?
And, you know, it's, honestly, I'll just say this isn't the most
exciting topic, Um, and maybe, maybe, we'll, maybe we'll make it a short
episode if it's not interesting.
You know, we're not gonna make, we're not gonna make a long episode cuz
this isn't gonna be one that's gonna be, uh, littered with stuff about
cyber attacks and things like that.
Backup levels
Prasanna Malaiyandi: really straightforward.
W. Curtis Preston: really straightforward.
Yeah.
Uh, I do want to help for people that sometimes struggle with
the different types of backups.
Um, I want them to help them understand.
I, I'd say that, you know, if you just go and I'm, uh, throughout
our usual disclaimer Prasanna, and I work for different companies.
I work for Druva, he works for Zoom, and the, this is an independent podcast.
It's not theirs.
And it's, uh, the opinions that you hear are ours.
Uh, be sure to rate us, uh, go to your favorite, um, pod catcher and
uh, scroll down to wherever they got the stars and give us all the stars.
And the best thing is if you give us a comment, those are always nice,
especially on Apple Podcasts, which is.
the primary way that people listen to us over here.
And, um, also if you wanna join the conversation, I am w Curtis Preston gmail
and also WC Preston for now on Twitter.
I dunno what's
Prasanna Malaiyandi: do you have a master on?
W. Curtis Preston: Twitter man, I don't have a Macedon account.
Uh, I am interested in that.
We'll see if, I don't know if the average person can, can
Prasanna Malaiyandi: I don't think so.
From what everything I've heard.
Yeah.
W. Curtis Preston: It took me a minute, right?
Twitter, you know, you sign up for account, you're done right here.
It's like, well, what community do you want it attach to?
I'm like, oh, it's so complicated.
I don't wanna know what community, you know?
Um, we'll see.
But, uh, anyway, at WC Preston on Twitter, maybe I should sign it
for my, at WC Preston on Mastodon.
Make sure I get that.
Prasanna Malaiyandi: I think you were throwing out the disclaimer though,
because for this chapter, if people want to actually read the chapter and
understand all these terms, right.
Druva is offering a free ebook version.
W. Curtis Preston: I, you know, that wasn't why I was saying
it, but, we'll, we'll do that.
So we are discussing my book, modern Data Protection.
There's a picture of it with the little armadillo for those of you watching on
the video version on backup central.com.
And uh, you can get a free ebook copy of it by going to druva.com/ebook.
That's d r uva.com/ebook.
And.
while supplies last, or while the contract lasts.
So, you know, this, this, this episode will live for a long time.
If, if you go there and it's not there, it's not my fault.
So, uh, first off, you know, again, this is pretty basic, but.
You know, you should know what a full backup is.
Obviously.
That's it.
It backs up everything, right?
Typically.
Well, not typically.
You always have to do a full backup before you can do any,
any kind of incremental backup.
That's the, that's the next type.
The, there are amazingly, a handful of different types of
the traditional incremental
Prasanna Malaiyandi: but before you get to incremental, so.
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: Even for fulls, are there different types of fulls?
W. Curtis Preston: Well, there's not different levels for fulls, but there
are different ways that fulls get done.
Right.
Like you, like we could be doing a file system backup or we could be doing a block
level backup, but either way you're still backing up all the things, all the bites.
Yeah.
Um, the, uh, and it's, it's the full is the thing that puts the most amount of,
of stress on the thing being backed up.
Right.
The, this is why we try to limit them whenever.
, and this is why, this is why backups broke when VMware took off because everybody was
just doing full and incremental backups and they weren't coordinating them.
And you know, it was just you,
Prasanna Malaiyandi: lot of resources being used.
W. Curtis Preston: Yeah, exactly right.
And um, if you do, if you do a nightly full backup, or let's say the weekend
full backup, and you do 'em all on Friday, And you've got 20 different
physical servers, no big deal.
But if you do 20 full backups on 20 VMs that are all on one
physical server, this is a problem.
Right.
Um, so I've got a couple different types here.
There's the, the typical incremental backup, which is, it basically,
it backs up everything since the last backup of whatever kind.
And typically what what you do, and this is with backups that have
levels, you do a full backup, then you do a series of incremental
backups, and then you do another full.
right?
That's the, that's the sort of the typical way.
There is something called a cumulative incremental backup.
Um, I prefer that to the term differential backup, and I can
talk about that in a minute.
But a cumulative incremental backup is essentially just that it backs up
everything that has changed since the last full different, um, different product.
Uh, do different things.
Uh, the differential backup is a term that you see a lot in windows and, um,
I, I, and I, I don't use it because I find the term, um, the, the, because
depending on where you ask the term differential means different things.
And so I, I don't like it.
Right?
So I like the term cumulative incremental.
Even if your backup product doesn't use it, just learn
what, whatever they call it.
What?
Prasanna Malaiyandi: wait.
So as far as I understand it, , you have the thing that's like full then.
So say I do a full on the weekend, right on Monday, I do, uh, incremental,
W. Curtis Preston: Uhhuh.
Prasanna Malaiyandi: right?
Which will be the differences for whatever happened between Sunday and Monday,
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: right?
Tuesday I do another incremental
W. Curtis Preston: Yeah,
Prasanna Malaiyandi: right?
That'll be the differences between Monday and Tuesday.
W. Curtis Preston: yeah,
Prasanna Malaiyandi: And then Wednesday I do another incremental.
That's the difference between Tuesday and Wednesday.
Now, for different, uh, for cumulative incrementals, it's, I
did my still, I'll do my Sunday, full Monday I'll do my cumulative.
Incremental, or yeah, incremental, which will just be
Sunday to Monday, differences.
Tuesday, when I do the cumulative incremental, it'll
be from Sunday to Tuesday
W. Curtis Preston: Correct.
Prasanna Malaiyandi: and Wednesday when I do a cumulative incremental, it'll
be the differences between Sunday and.
W. Curtis Preston: Correct.
And so what, what we used to do back in the day was I would do a monthly
full weekly cumulative incrementals and daily incrementals, right?
So that way, even if I'm 28 days into the month, I would only need
three backups to get the job done.
. Right.
Um, I, I don't use the term differential.
Differential is what a lot of places call accumulative a criminal backup.
But again, different products use it to mean different things, so I don't use it.
Um, but interestingly enough, what we haven't talked about is backup
levels, which is the term that, again, you don't see too much these days.
Have you, have you, are you seeing it?
Prasanna Malaiyandi: Very, very rarely do Does anyone talk.
The only place I've seen levels really is when you're talking
like database backups, but most.
W. Curtis Preston: what?
That is true.
Right?
Prasanna Malaiyandi: Yeah.
But most backups software itself no longer talks about the levels.
Right.
It's either fulls or incrementals.
W. Curtis Preston: I think it was too complicated.
Yeah.
The basically if to, to do.
To do a full on a weekend and then daily regular incrementals, you would
do a 0, 1, 2, 3, 4, 5, 6, 7, right?
Well, seven.
And then maybe, um, so then, and again, this is, this is impossible to do without
a whiteboard, but if you did, um, a zero followed by all the way up to the five, if
you did a one again following that five.
you would get all the changes since to zero.
Prasanna Malaiyandi: Hmm.
W. Curtis Preston: So it was accumulative.
Incremental, right.
But what some people would do is they would do this, um, this,
there, there's, there was this thing called towers of Hanoi, which,
um, it, it comes from the game.
Prasanna Malaiyandi: like
W. Curtis Preston: Yes.
Yes.
It's like the thing with the game.
Yes,
Prasanna Malaiyandi: three.
Yeah.
W. Curtis Preston: Comes from that game.
Um, and so the, and again, this is really, don't try to understand this, I'm just
putting it out there just for, this is for the true backup anoracks, so who's gonna
Prasanna Malaiyandi: Daniel Rosen Hill.
We're looking.
Daniel Rose Hill.
We're looking at you.
W. Curtis Preston: Exactly.
So, um, let's see.
So what I've got here is, , um, a towers of Illinois Schedule 0 3 2 5 4 7 6.
And what it does is it, is it like every file that's changed
ends up on two different, um,
Prasanna Malaiyandi: Mm,
W. Curtis Preston: two different, um,
Prasanna Malaiyandi: days.
W. Curtis Preston: backups.
Prasanna Malaiyandi: Yeah,
W. Curtis Preston: Um, and um,
Prasanna Malaiyandi: you don't run the risk of if a backup is bad,
you've lost the file, which could happen if you just had a full and you
never did another incremental again.
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: Or sorry you never did another full again, I, I
think one thing we forgot to mention Curtis though, is the benefits of doing
a a, we might have talked about it implicitly, but the benefits of doing a
cumulative incremental versus a normal incremental when it comes to restores.
W. Curtis Preston: Well, you know, here's the thing.
The benefits were much bigger when we were using tape.
Prasanna Malaiyandi: Hmm.
W. Curtis Preston: I, I think that as, and, and what we're gonna discover is
that this is why a lot of this has, I think, gone by the wayside is because,
because when we were using tape, you'd gr you'd grab the full tape, you would grab
the cumulative incremental tape, and you would grab any incremental since then.
So you've got maximum like eight tapes if you did a full,
followed by 30 incremental.
, you would, um, you would need 28 tapes, you know, at the end of the month.
So this way you only need like a handful of tapes.
And what it allowed you to do was it allowed you to do
less frequent full backups.
Because it was a, it was a, an extra cost.
You're like, well, by doing a weekly cumulative incremental,
I could do a monthly full backup instead of a weekly full backup.
Thus really reducing how big the backup system needed to be,
how many tapes I needed to use.
Um, And, and by doing that cumulative incremental, it made
the restore much, much faster.
I would argue if you have a decent backup system today and you've got a monthly
full and 30 days of incremental, or even 60 days of incremental, the difference
between restore times, um, depending on how you store the backup data, shouldn't
be as significant as they were back in the day when we were loading tapes, cuz
every tape loaded was like two minutes
Prasanna Malaiyandi: Yeah.
But,
W. Curtis Preston: to the first.
Prasanna Malaiyandi: but I do wonder though if when you are doing
those restores, depending on what you're restoring, like I could
see a file system maybe not being a big deal for doing the 60 days.
I'm just wondering like database backups, right.
To do those incremental restores.
W. Curtis Preston: Yeah.
Um, I think that's a val, I think that's a valid.
Uh, concern, right?
This is why to this day we still do more frequent
Prasanna Malaiyandi: Fulls that need it.
W. Curtis Preston: on, on database backups.
Right.
The, that's, that's a good point.
Um, and since a significant portion of the world is on, is,
is structured data then, uh, yeah, that's a really good point, right?
So, um, the.
So now we've got a just some different types of, so we're done with the levels.
Now we're talking about different types of backups.
What about a block level incremental backup?
What is that?
Prasanna Malaiyandi: Block level, incremental.
Are you referring to like database slash v.
Where you're just copying the differences in whatever the application
block size is, rather than block backing up the entire image or
the entire file potentially again.
W. Curtis Preston: Yeah, so basically this is the difference between, and
again, this really only applies if you're looking at things at the block level,
which you tend to do at the database, but there are some file systems that
are so dense that backing it up at the block level would be faster than
backing it up at the, at the file level.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: And so you can do a block level incremental, basically.
The point is, when I'm doing an incremental.
A file system in increment, a file system incremental.
I'm backing up the files that have changed since the full, if I'm doing
a block level incremental, I'm backing up the blocks that have changed since
the full or since the previous bite.
Right.
And then here's something that hopefully you'll know something about, and it's
called Source side deduplication.
This is another way to do incremental.
What can you tell us about.
Prasanna Malaiyandi: Yeah.
Source side de-duplication.
So when de-duplication appliances came out, um, what you ended up happening
is whenever you would do a full right, you were talking right Curtis, about
doing weekly fulls, daily incrementals.
Every time you did that full, probably 98% of the data was the same.
So on the source you would read it, you would send it over the network, it would
go to your de-duplicated appliance and it would throw away 98% of the data.
And so source side de-duplication is, Instead of sending all of
that data to be thrown away.
On the appliance, you basically split the deduplication algorithm
such that you're running some of the processing on the source.
So you figure out what is duplicate, usually at a smaller block level
or some other, uh, granular size and only send the unique blocks
over to the target appliance.
And the rest of the data is sort of reconstructed, if you will.
There are different methods you could look at.
It's sort of rebuilt because you still need to account for those pieces
of data, even though you haven't technically sent the data over the wire.
W. Curtis Preston: Yeah, so if we think about it, like the only reason
ever to do an incremental backup is to save something, save bandwidth,
save CPU time, save storage.
And what, you know, what you're talking about is even when you had
target site dation with like data domain and the like, you, you.
, you save storage.
You, you can do full backup every day and you save storage.
But you were still using up all that CPU in that network by doing
Source I duplication and making that decision before you send it.
You save CPU time, you save bandwidth, you save storage, you save all the things.
Right?
Um, source ID is not as prevalent as targets.
I dedup.
There's, there's only a handful of companies that,
that do it, but uh, they are,
Prasanna Malaiyandi: Yep.
W. Curtis Preston: Um,
Prasanna Malaiyandi: it does require more integration into the backup product on the
source in order to take advantage of it.
W. Curtis Preston: exactly right.
Uh, for the record, Druva is a source site duplication product.
So the next thing we talk about here is called synthetic full backups,
uh, which is and interesting.
I remember when this concept first came out, again, we're trying to
minimize the impact of the backup.
on the system, we're backing up.
So the idea was we already have all of the bites necessary to create a full backup.
Why do we need to go get, you know, we're doing a, we're doing an increment.
We're doing the next backup and 99% of what's on the system we
already have on disc somewhere.
This is really a disc thing.
Uh, we did it on tape too, but it was a lot more work.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: So we'll go get the 1% that has changed.
and then we'll create a full backup by copying the stuff we already
have and the stuff that we just got.
So we create a synthetic full.
Um, have you, have you run into those out in the wild?
Prasanna Malaiyandi: no, it's very common, uh, especially when
you think about VMware images.
W. Curtis Preston: Mm-hmm.
Prasanna Malaiyandi: depending on how it's being backed up, a lot of folks
would back up a VM as incrementals and then, because like you said, most of
the data is the same, they would sort of synthesize a full image on the target
side in order to have that full copy.
Therefore, when you need to restore, you have that.
What image you could pull from or would a lot of people start doing is you take
that one image and you can now mount.
, right.
And access it directly.
Instant access is what simple some people call it.
So you can spin up the VM from your target system and then
vMotion it over or whatever else.
So you have shorter RTOs than trying to first restore the
data before you spin it up.
W. Curtis Preston: and there, and there's a couple different ways that
you can create a synthetic full.
You can either do it by copying, which is what I was talking about before.
There's also this concept of virtual synthetic full,
where you can just create one.
, you know, magically, right?
Like the, I know that again, data domain I know supported
that on the back end, right?
That if the backup product could say, you have all the bites you need to make a
full, why don't you just put 'em together and, you know, everything's beautiful,
Prasanna Malaiyandi: You know what else does it too,
W. Curtis Preston: what's that?
Prasanna Malaiyandi: Oracle?
When you do incremental merch,
W. Curtis Preston: That's right.
Incremental merge creates.
Exactly, exactly.
Um, but here's the thing.
Wherever we can get rid of full backups, I think we should get rid
of full backups, Right, because they're just the dumbest thing.
Right?
They're just, they're just, they're, we did 'em because that's the way we did 'em.
And we still do 'em because that's the way we used to do 'em.
Uh, so I'm a fan of, go ahead.
Prasanna Malaiyandi: I, so I, I wanna be careful, right?
I think we should.
Get rid of the concept of backing up full copies of data every
single time or on some increment.
I a hundred percent agree with that statement.
W. Curtis Preston: Isn't that what I just said?
Prasanna Malaiyandi: No, no, no, no, no.
But, but I don't agree with, but there's still a necessity to have the
equivalent of periodic fulls, right?
Something that represents a full backup.
Now it could be stored on de-duplicated systems to save storage space, cuz
most of it's the same, but I'm just saying you don't wanna go the route of,
it's just a full once a year and just incrementals physically stored on disk.
So using virtual synthetics or some other mechanism to end up with full copies
periodically, I think is beneficial.
W. Curtis Preston: So I would argue that that's what Incremental Forever
is a true incremental forever system.
Right.
That basically you, which is what, what I was about to talk
Prasanna Malaiyandi: Okay.
Okay.
W. Curtis Preston: Um, and that, you know, the idea is store each backup
in such a way so that a, any restore from any backup, you just, the, it's
stored in such a way that it looks like a full right and a restore from it
behaves like a full, I, I don't think we need to keep making fulls or make or
synthetically creating an occasional full.
I think that if you design it from the beginning, so that you do
incremental forever, and then you store that data in such a way that
every backup is essentially a full
Prasanna Malaiyandi: Yep.
W. Curtis Preston: from, from a way it behaved the way it behaves
during a restore, then I think that's as good as it's gonna get.
Right.
Prasanna Malaiyandi: I, I think that works in the cases of file
systems and virtual machines.
I do think in the case of databases, it is a little database is always that oddball.
W. Curtis Preston: I had the caveat back earlier where I was
like, wherever we can, right.
Um,
Prasanna Malaiyandi: Databases have so many nuances with fulls
and incrementals and log archived, redo logs, and all the rest that I
think, yeah, for everything else, a hundred percent agree with you, Curtis
W. Curtis Preston: Yeah.
Um, and there are, there are a handful of products that back up this way
that basically, said full backups is so from the tape land, right?
Except for database backups.
Right.
Um, generally speaking, there're gonna be products that have come
out in the last 10 ish years.
Right.
Um, products that have been around for 20 years.
They're gonna have spent time in the tape land and they're gonna have
parts of their architecture that are left over from the tape plan.
Prasanna Malaiyandi: I also do wonder if some of that is just the
people who are banishing the systems.
It's something that they are used to, and so change is hard, you know?
W. Curtis Preston: is hard.
Inertia is something.
Yeah, exactly.
And I would say like if you, if you're doing source side dup with incremental
forever, I think from an efficiency perspective, assuming that the way you
store it, everything behaves like a full, an efficiency perspective, I don't see it.
I don't know how it could get any more efficient in that.
Right.
You could maybe argue the block level incremental forever where you're.
Right.
So there, there are like c d P and near CDP systems that's continuous data
protection where they're doing block level changes and they're replicating that.
That's also quite efficient.
It's not source I, it's still, it's still, so I would say,
so I'll restate my statement.
Either source iddu or block level of replication and then incremental
forever stored in such a way so that everything behaves like a full.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: you know, that includes a handful of
products , one of which is Druva.
But, um, you know, there, there are other products that behave like that.
Like, you know, your old employer, NetApp, right?
That was a block level incremental replication.
Um, yeah.
Um, so I'm gonna throw out one, um, this is gonna be one of our shorter episodes.
I think.
We'll see.
We're almost done.
I'm gonna throw out one, like, uh, what's the cranky old man?
Comment
Prasanna Malaiyandi: get off my lawn.
W. Curtis Preston: this is the archive bit and Windows.
I hate the archive bit and windows.
Um, so what is the archive bit?
It's a flag on every file that when your backup software backs up
that file, it can unset that flag.
If the file is new or it changes, the archive bit is set.
And then when the flag, uh, when the back, the, uh, backup software backs
it up, the archive flag is unset.
Prasanna Malaiyandi: Why.
W. Curtis Preston: This is the old way and it's still the current way.
Uh, not looking.
This is if you're doing a file system level backup in Windows.
Um, I think it's so dumb for multiple reasons.
The first of which it's name,
Prasanna Malaiyandi: Archive.
W. Curtis Preston: it should be the backup bit.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: They called it the archive bit anyway, but that's not
really my problem is you can't, if anyone comes along with like, any, cuz there
are a number of like third party tools that can back up your Windows system.
If you run a third party tool to create your own backup, like let's
say you, you know, a lot, some people don't trust the backup system, right?
So if you, if you being an admin go and decide to make your own backup
and you use something that uses the archive bit, it will clear the
archive bit on all the files that have changed since the last full.
and then when you go, when the real backup software comes in, it actually
Prasanna Malaiyandi: not gonna find,
W. Curtis Preston: until the next full backup.
Prasanna Malaiyandi: and I bet people don't
W. Curtis Preston: problem with it.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: Yeah.
And of course people don't notice it.
Um, and so I'm gonna go back to the story, um, that I talked about.
I don't know what the, the one about me being kidnapped, was
that in this episode or was
That
Prasanna Malaiyandi: was a previous.
W. Curtis Preston: Okay, well I hinted at this story from um, this other,
from the other episode, but here's the full story cuz we got time And you know
what, if you don't want another old backup story then you know, thanks for
joining and uh, see you next episode.
But this is a good story and if figures in a good buddy of mine, Rob Worman.
I know Rob listens to the podcast.
, he figures into this story.
So I was at this large, you know, national entertainment company and I was helping
them to redesign their backup system.
We were doing a massive redesign.
And one of the, and one of the things, you know, like we pushed them from, you
know, weekly folds to monthly folds.
We, we changed all their multiplexing setting.
We changed what, uh, we changed how.
, the schedules were, I mean, we changed everything and we were doing this
to, to do massive efficiency change.
They were running like, it was like 18 tape drives simultaneously,
and I told 'em that that was their pro, they were gonna buy, uh, two
Prasanna Malaiyandi: Oh, this is a sh when you're, you were, you're talking about the
shoe shining problem and they were trying to tape goes too fast for what they need.
W. Curtis Preston: Yeah.
And so they were gonna buy two new tape drives and that was gonna fix the problem.
And I'm like, no, it's not.
It's actually gonna make it worse.
And I explained to 'em and I'm like, listen, give me a few weeks and I'll
redesign your backups to submit.
It'll cost you less than what those tape drives were gonna cost.
Cuz they were expensive.
These were the storage tech tape drives, like the 90, what?
I can't remember the tape name.
Tape names.
But anyway, um, so we did this and.
, um, and things went really well, but so I knew about the archive bit.
This is all about the archive bit.
I knew about the archive bit and I knew that if I had, I basically
left their parallel backup system.
I left their parallel policy.
Prasanna Malaiyandi: Up and
W. Curtis Preston: I left their existing policies running in parallel.
Well, I was gonna do the regular backups, uh, during the day,
um, and something like that.
It's been a while.
And, but I knew that I would screw them up if I had the archive bid on both systems.
So I used, this was net backup.
I used a feature of net backup that said, okay, don't use the archive bit on.
On the sink and what I didn't notice or what happened over time, over about
two weeks is the arc, the incremental backups got bigger and bigger and
bigger and bigger, and they got huge
Prasanna Malaiyandi: on your new
W. Curtis Preston: and they on the new system and they got so
big that they weren't finishing.
By the way, when I got there, they were backing up 28 hours a day.
So my, my opinion was no matter what I was doing, it still wasn't gonna be that bad.
, but backup, but the backups weren't finishing and I didn't understand, right.
So, so we shut off the old backups.
We're like, okay, we we're, we're beyond the point of no return.
We shut off the old backups, backups still weren't finishing.
And uh, you know, I was working through diff, you know,
multiple levels of support.
And I actually at, at one point ended up with my buddy Rob,
Prasanna Malaiyandi: Hmm.
W. Curtis Preston: I'm like, dude, I don't understand.
And we, you know, we figured out that when that.
Doesn't use the archive bit.
What it does is a, again, this is, this is a hundred years ago.
I don't know what it does today.
If it doesn't use the archive bit, it would, it would traverse the directory
tree and then it would, as soon as it encountered a directory that whose
modification time had changed, it would back up everything in that directory tree,
Prasanna Malaiyandi: Okay.
Including data that does, yeah,
W. Curtis Preston: including data that hadn't changed, right?
It, it was really inefficient.
And so it tried to add that level of efficiency by working up the,
you know, so long story short, the back, the incremental backups were,
were getting close to full backups.
And so once we figured that out, we're like, okay, okay,
okay, okay, we figured it out.
We're gonna tell it to use the archive bit now.
And, um, and we turned on the, we turned on the archive bit.
and it was the same the next day.
Right?
Prasanna Malaiyandi: Okay,
W. Curtis Preston: Now I look, now I look, and meanwhile this, this
manager, he is looking at me, you know, looking at me over the, you know, he's
beginning to think I'm a complete moron.
And, um, he, um, I said, look, I, you know, we figured it out.
This is what, you know, he's like, but it, but it didn't fix the problem.
And that's when I was really talking to Rob.
I'm like, I don't understand.
And again, I was on the edge here.
I would, no one had done what I was doing, you know, and so, so what we
figured out was when you tell net backup not to use the archive bit,
it doesn't use the archive bit.
Now, what that means is that when it backs up a file, it doesn't
clear the archive bit, which was the whole point of me using the thing.
And so that meant that
Prasanna Malaiyandi: The
W. Curtis Preston: we,
Prasanna Malaiyandi: when you turn it on
W. Curtis Preston: to run the, we had to run the one backup.
that would use the archive bit to clear everything.
But because we had run one in a long time, it was also gonna be really long.
And so then it's like, okay, tomorrow night is gonna be all better.
And that's when the boss said, okay, smarty pants, no one's going anywhere.
He was, he was really upset at the level of instability.
And I can understand him being.
But he was really upset at the level of instability in his mind that
I had introduced into his system.
Nevermind that the new backup system was backing up 50% more
data than the old system because I changed the, the inclusion factor.
And they had, they had, they had left out a lot of really important data.
So it went from 20 terabytes to 30 terabytes and.
Nevermind the fact that we were now creating two copies, whereas before
he wasn't even getting one copy done.
So I was feeling pretty good.
But all he sees is his
Prasanna Malaiyandi: Stability.
Yep.
W. Curtis Preston: Yeah.
And so he's like, well, nobody's going home until, and literally
he just, he stayed with us.
I mean, to his credit,
Prasanna Malaiyandi: Okay.
W. Curtis Preston: stayed with us and like he ordered pizza and stuff, uh,
but he literally wouldn't let us leave.
I couldn't even say like, well, I kicked off the back house.
We're gonna go get dinner.
I ain't nobody going anywhere.
Prasanna Malaiyandi: He's like bathroom.
You already had your bathroom break.
W. Curtis Preston: but we're gonna sit here and Yeah.
And so we were there until pretty late.
Um, I don't really remember how late, but I remember like, it was like a whole
second day, you know, like it was, it was a long, long, long, long night.
And that my friends, . That's what happens when you don't understand backup levels.
Anyway, All right.
Well I think this is enough torturing people with backup level information.
Prasanna Malaiyandi: I, I a hundred percent
W. Curtis Preston: talk about, go ahead.
Yeah.
Next week we're gonna talk about metrics.
Prasanna Malaiyandi: Ooh,
W. Curtis Preston: yeah.
I think it's a little more exciting.
Prasanna Malaiyandi: yep.
W. Curtis Preston: It's backups.
Like how exciting could it be
Prasanna Malaiyandi: Come on, Curtis.
What are you talking?
Telling our audience, our listeners,
W. Curtis Preston: Woo.
I dunno what to tell you.
You know what, maybe you're excited as you're as excited
about backups as I am, maybe, to which I say, welcome to the party.
All right, well, uh, thanks for listening folks, and remember to
subscribe so that you can restore it all.