You found the backup wrap up your go-to podcast for all things
backup recovery and cyber recovery.
In this episode, we're covering forever incremental backup technology and
how that completely revolutionized the entire backup industry.
I. Remember the old days of doing weekly fos that brought
your systems to their knees?
Well, those days are thankfully gone for most people.
We're gonna walk through the evolution of, from traditional approaches with
fools and incrementals to synthetic fos, and finally to true forever incremental.
If you're still doing things the old way, well, you're missing
out on some serious benefits.
Better performance, lower storage costs, and way faster restores.
Perhaps this episode will cause you to rethink things a little bit.
By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr. Backup,
and I've been passionate about backup and recovery for over 30 years, ever since.
I had to tell my boss that we had no backups of the production
database that we had just lost.
I don't want that to happen to you, and that's why I do this.
On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes.
This is the backup wrap up.
Welcome to the show.
Hi, I am w Curtis Preston, AKA, Mr. Backup, and I have with me a guy that
I bet is pretty dang happy right now.
Prasanna Malaiyandi.
Why are you looking like that?
very confused about what I'm happy about.
Really?
Do I need to explain recent events that happened at your house
Oh
that, that.
For those of you by the way, um, who are listening to this podcast via
audio, we also provide video on YouTube.
So if you wanted to see our expressions just then and really see what Curtis
is laughing about, uh, go check us
Yeah,
Backup wrap up.
so what events happened in the last 48 hours?
my wife was in India
Yeah.
Visiting family and she just got back.
Yes.
it was good.
Suddenly, you're so much harder to get on the phone between that and your new job.
I don't even know you anymore.
He's like, why would I be happy?
I, I was really, I was, I thought it was like, I was thinking, I was like,
was it something YouTube related?
Was it something like I, we had watched or we had talked about and.
No, the fact that your wife is back
although she's also slightly jet lagged, so,
so well, it's what, what, what's it is 12 hours difference.
right now it is 13 and a half.
Yeah.
So
does
of course she's,
jet lagged than I do.
So
yeah.
and come back, it takes me like three weeks.
And,
Yeah.
have a dog, which I think some people have seen on the podcast.
Um, it's funny because he gets jet lagged when I'm jet lagged.
So he, his hours will shift to match with mine.
So
That's really adorable.
and he would be like, Hey, I'm ready.
Let's go, let's go.
And then when I'm passed out, he's asleep.
So he's
Interesting.
Yes.
Your dog that's named after an Indian dessert.
cream.
Yep.
Ice cream.
Oh yeah.
Ice cream.
Yeah.
Kulfi, which I have now had, um.
your, did you know they have various flavors of Kulfi.
They're not all just like one.
Oh, I thought it was all mango.
So it's met like mango Kulfi and then what?
Coconut Kulfi.
What are what?
What are the other typical
and pistachio.
Pistachio and, and.
I think there are other ones too.
Yeah.
I don't, yeah, I don't.
I've used cardamom as a, an ingredient, you know, when I made the, the
mango pie thing, but I don't think I know what it tastes like by itself.
So I'll have to, I'll have to sprinkle some and, uh, see what that tastes like.
Or maybe just go and
Yeah.
some
next time you're at the Indian store.
Actually, sometimes even the Costcos carry like a sampler pack
Oh, yeah, so like when I go to the cash and carry down in Mira Mesa.
the cash and carry will definitely have it.
If you go check out the freezer section.
Just
Interesting.
all before you get home.
Yeah.
I, oh man.
I'm, you're, you're literally like, like my mouth is Yeah, yeah.
Thinking about, thinking about it sounds yummy.
Hmm.
Uh, so we're gonna talk about something that's just as yummy.
That is if you're into, into the backup thing, uh, we're gonna talk
about forever incremental and sort of.
Uh, just how big of a deal that is or was.
Um, it, at, at this point?
It's in, in most modern backup products that have come out in
the last 10 years, I'd say, uh, ones that are actually new, right?
So, you know, when we look at the products like Rubrik, Cohesity, or
modern SaaS applications like Druva, Hyku, keep it, Alcion, uh, right.
You know, these are all of these products.
Have forever.
Incremental in common, right?
Nobody wants to do full backups anymore.
Uh, or, or anything like full backups, right?
So, um, so let's talk about that.
Let's talk about what, you know, where were we, what was it like back in the
day, and, um, and then sort of how we got the, from there to here and why it's
such a big deal and why you should care.
Right.
And why this perhaps may give you a reason to finally, um, you
know, what, what is it, shuck?
The mortal coil.
What, what, what, what is that?
What,
I'm just
thinking are there company, and maybe as we're talking we can figure this out, like
are there still vendors that exist that don't support forever Incrementals?
That's actually a really good question and I'm gonna say, uh, absolutely
right, because my concept of what a true forever incremental will be based on.
Like a, a complete redesign of the, of the underneath.
Right.
Um, so the phrase I was looking for is shuttle off sh shuffle off the
mortal coil of your old backup product,
by the way, wait, quick story.
So I'm on, I'm on a plane one time and this, this, um.
This pilot, he was talking about the fact that we were about to take off and
he referred to taking off as shuffling off the mortal coil of this earth.
And I'm like, dude, that's a phrase for dying.
What?
Why the hell are you, why would you do that?
Right.
, it's a Shakespeare, it's from the to be or not to be soliloquy
I,
Hamlet.
in all fairness though, I would say 97% of people probably had
no idea what that phrase meant.
They, they never even heard.
Yeah.
And, and, and, and, and just to, just to, uh, like, you know, since I'm over
here quoting Shakespeare, let me just put this, I'm not a huge Shakespeare person.
In fact, it, it wasn't until, I dunno, several years ago, and we were in England
where I, I, I got to actually see my first, um, uh, Shakespeare play, which
was Henry V. Some people say Henry the VI don't, I think it's actually
called Henry five, which is the, um.
The one with the St. Crispin stay speech.
Uh, once more into the breach, close the wall up with, with
her English dead and all that.
But anyway, sorry, I just bored everybody.
Sorry.
Sorry, you're asleep.
Wake back up.
We're, we're gonna talk about backup, which is so much more
exciting than Shakespeare.
Um, so, all right, so first let's talk about, do you want to define, uh, so what?
let's talk about the old days first.
that's what I, that's what I'm saying.
That's what I'm saying.
Let's, do you wanna define, uh, what a full and, and
Yeah.
an incremental backup is?
Yeah.
So.
Normally when you think about backups, right, you talk,
typically people talk about fulls.
Incrementals, right?
A full is basically your entire application, dataset, whatever
it is, everything is backed up.
Mm-hmm.
then between doing every fulls, you would do incrementals, which are just the things
that have changed since that last full.
Right, right.
you do these periodically, and normally you do incrementals
because they will be smaller.
So you could say do a full once a week and then incrementals during the week
such that you can finish in time or during your backup window, and then the next
weekend you would do like another full,
Right, right now, slightly, I'll slightly change your
definition or a pen or whatever.
Um, so you specifically described what I would call a
cumulative incremental, right?
So a, uh, what is referred to as a sort of a regular, incremental
or a differential incremental.
Yep,
it is one where it's the changes just since the last incremental or since
the last backup of any kind, right?
And then you
Um,
levels and all the
yeah.
Yeah, yeah,
But yeah, at a high level, yeah, there are those three types, if you will.
yeah.
And, and, and then what really, um, what really what it really
came down to is how things behaved when it came time for a restore.
Right.
So if you were going to restore something, then you needed the full,
the latest cumulative, incremental, if you were doing such a thing, and
then any incrementals since that day.
And so, um, those of us, uh.
I, I remember at some point we, we used to do weekly fulls.
I remember my preferred backup design switching to monthly fulls,
weekly cumulative incrementals, and then daily incrementals.
So if you were, let's say three weeks into the month, you would need the full
from the first of the month, the latest, uh, weekly cumulative, incremental,
and then three or four days worth of.
Incrementals
Yeah,
and, uh, in order to do a restore.
. So what it meant was that you would often, the way the Restore worked is you
would restore everything that was in the full, then you would restore everything
that was in the cumulative incremental, and then you would restore everything
that was in each of the incrementals.
What that meant was there were many files that you were actually restoring
multiple times and you were overwriting them, which was just inefficient.
Um, and what that also meant was.
That you had, uh, uh, duplicate data, the old way of doing backups created
a ton of duplicate data on the tapes.
You could say duplicate, or you could also say it's independent,
depending on the glass.
Half full glass, half empty.
Well, so no, I, I understand what you're saying.
What I'm talking about is the fulls,
Yeah,
the fulls over time created a ton of duplicate data.
that's true.
Right.
I know what you're thinking.
You're thinking, you're talking about the, within the, within the, the incrementals.
I'm mainly talking about
yeah,
how many different copies of the exact same file were all
over, you know, everywhere.
and especially because most times, right, you look at sort of like a
2% daily change rate, which maybe
Yeah.
a seven or 8% change rate per week.
And
Yeah.
those weekly cumulatives or even the monthly fulls, right, you're
basically backing up a bunch of data, which you don't necessarily
need that haven't changed.
Right, right.
You, you, the reason we did it the way we did, it wasn't so much about independence.
It was about.
Minimize.
So if you did a monthly full, followed by nothing but regular
incrementals, you would have to load 30 tapes at the end of the day, right?
At the end of, and even if it wasn't tapes, you would have
to, you have to do 30 restores.
Yeah,
So, so the, the, the, the differential or the cumulative incremental, um,
was about minimizing the level of effort during a restore, right?
And then came along this thing called dedupe.
Uh, which at least helped minimize, I think the, all of that duplicate data.
because I think with deduplication it was nice because you didn't
have to change what you were doing from like a policy perspective
Right,
perspective.
But you got, like you said, it was able to.
Dedupe, of course,
right.
in the backend and save you the storage costs.
And that's why when, if you, if you were doing a, a typical weekly, by
the way, nobody did the, the monthly full thing that I recommended, most
people did weekly fulls, right?
And so when you had weekly fulls, um.
This is why, and you, and you had like 90 day retention.
This is why just with even crappy dedupe, you got 20 to one,
Yeah.
uh, data reduction, right?
And this is what helped, um, pay for, you know, the what to help
created the, the market for all that, all that dedupe work, right?
And then we could argue over, um, you know, well we're, we're
20 to one, we're 30 to one, we're, you know, whatever, right?
And, um, we, we saved this much more money.
Um.
You know, once you get rid of all those fulls, everything else
was, was really kind of incre
Yeah.
incremental.
Um, but the real, I think the real challenge was just the core backup design
of when we go to do a restore, we have to restore, uh, data multiple times.
Right.
It just simply wasn't efficient.
Well, and I think even before we get there, I'd like to cover
Mm-hmm.
uh,
Okay.
if you had the existing backup schedule, right, where I'm still doing
weekly fulls and daily incrementals,
Mm-hmm.
still transferring all that data over the network.
Right?
And I think this is where client side deduplication would probably
come next, in my opinion, right?
Yep.
it would help you reduce the amount of data you're actually
sending over the network.
Right.
And that'll help save you cost as well as time.
Yeah, that's actually a really good point.
Thanks for bringing that up.
So the, one of the other like, like basically what you're saying that one
of the other, um, downsides to doing it the old way was both the transfer
across the network as well as the impact, uh, to the clients, right?
If you're doing a full, on a regular basis, um, you're, um.
There's a, there's a cost to that
Yep.
in both, you know, you brought up bandwidth, it's
bandwidth storage and compute.
Right?
Um, it's like all the things.
There was also a, um, sort of a, like a perfect storm brewing.
Yeah.
So we're doing fulls on a regular basis.
We're doing cumulative incrementals on a regular basis, and this is
how we were doing things, and then along came virtualization.
Why was that a problem?
Because a lot of the data was
I,
the same, right?
When you look across all of your VMs, they're probably spun up
from the same golden copy, of your
Well, the,
and
yeah, well, the.
amongst them.
Well, that, that's a, this is another example of, that's a great answer.
It's not the question I was asking.
It's
Um,
resource on the backend because
that's what I was going after.
Yeah,
and the compute, right?
You don't have your
yeah,
instance
yeah.
That's, that's what I was going after.
Right?
So, so that was a real problem when VMware got really popular and we just
kept doing the same old thing because.
Remember, for a long time we backed up VMware.
We backed up the physical, the virtual machines, just like
they were physical machines.
Yes.
Guest ba. We just basically installed an A, an agent and every guest,
and that's how we did the backups.
And, you know, and what happened was you'd do just randomly, you'd end up
doing a bunch of, at the same time.
Which worked fine when we were 20 individual machines, but now we're 20
VMs all on the same physical machine, and that just broke, like virtualization
just broke back up overnight.
Right.
Um, the latest thing to break back up overnight is, um, containers.
Yep.
That's a whole other thing.
We're not talking about that right now, but, um, the
so
ahead.
though, this is what sort of VMware, Plus the backup vendors to figure out
is there something better we can do
Yeah.
order to help satisfy backup needs?
Because it's not going away, right?
You have to
Right,
VMs.
right, right.
So the the real key I think, I think the real goal, I. Like, like if we could
go back in time and I, I don't know.
I don't know.
This is one of the things where like, I don't know if we, I, I wasn't in the
rooms right when they were designing this stuff, but I'd like to think that
somebody said, you know, if, because my memory is, we had issues with tape.
The, the, there was the fundamental mismatch of speed tape.
You know, people thought it was slow, it was actually fast.
That was a problem.
And we need, and so we started doing disc staging.
We started doing disc backups because of that, right?
And then DDU came along and DDU said, Hey, we can make disc better.
We can make disc, you know, more affordable.
And, and so, so that was, and then it just sort of happened.
But even if.
We if, if that wasn't the case, I think the design of backups ultimately ending
up being, uh, what we're talking about today, which is forever incremental,
it, it accomplishes multiple purposes.
One is it, it, it solves the issues that we have with tape.
Because in order for forever incremental work, you've gotta be using disc, right?
Uh, for, for true forever, incremental work to work, you've gotta be using disc.
Um, and then the other thing is that it, it solves the, the cost issues.
It solves the performance issues, restore, you know, it does a lot of things right.
Um,
you first define what forever incremental is?
'cause I don't
I don't want to,
that.
Before, before we get to forever incremental, we gotta talk about
something that was, that was a stop gap technology bef to get from we'd,
we wanted to get to a point where we weren't doing fullsS anymore.
Everybody agreed that full stunk, except especially when we're
talking about virtualization, right?
What was the stop gap technology that backup vendors came out with before
we went to the whole For incremental?
Uh, they created what they called synthetic fulls,
Mm-hmm.
which is basically you are.
Creating a full on the backend, but you're sending data and stitching it together and
piecing it together to make it look like what the full would look like, because you
Yeah.
backups were on that
Right.
So it's like we already know, we, we already know what the files are in
the full, because, you know, we did it on we, we did another incremental.
So we know what's deleted.
We know what's new.
We know what a full would look like.
If we went and made one, why don't we just make one over here
without having to do it over there.
Um, and there were a couple of different ways to do that.
Uh, there was the backup vendor way and there was the, um, the, well, the
backup software vendor way, and then there was the backup hardware vendor way.
Yeah.
I'll talk about the backup software vendor way first.
It's like, we know what all the files should be on a full,
we know where all of them are.
We're just gonna copy them all together.
You know, in one image, and then we're gonna copy that over to a new
set of tapes or a new set of disc
Mm.
so that we essentially create an actual full, um, without actually having to
transfer any data around the network.
It might be transferring it, uh, well it is transferring
it within the backup system.
Right.
And then there was the, the, and, and that took a certain amount of time.
Right.
Um, and then.
A, a vendor that you might be familiar with came out with
a different way to do it.
What was that?
And basically, um, they basically said, just tell us how you wanna
stitch everything together,
Yeah.
it all together.
Yeah.
So it was like, it was like an API, right.
You know where all the stuff is.
We have all the stuff.
Just tell us which stuff you want, where, and we'll go.
Yep.
And so they could make essentially a new full.
When in reality it's just a bunch of pointers to what was already there.
Right.
this
And it was really quick.
Right,
it was really quick because the key is, especially on a de-duplicated system,
you're not actually copying the data.
It's all pointer manipulation.
right, right.
moving any data, you're just changing metadata, which is very quick, very fast.
That was a great singing group from the nineties,
What
I
metadata.
manipulation.
But I'm, anyway, so yeah, so that was sort of a stop gap solution.
It allowed people to stop doing real fulls, solve the issue of the, the,
the load on the client, solve the issue of the load on the network.
But we still, we were still creating fulls.
So the concept of forever incremental.
And, and true forever incremental is that we first have to design a backup product
that just no longer needs repeated fulls.
Right?
And, and by the way, this generally does only apply to, um.
Uh, file system backups for the most part.
Um, not database and application backups.
Most of them still require repeated fullss.
Um, you know, we'll, we'll, some of, some of them have forever incremental, but.
I would also say, or would you also say for virtualized
workloads, this also applies 'cause
Um, oh, you mean, uh, the, the backup of the di of the system itself?
Yes.
So, well, so I'll, so if I, if, if I was to use a term that would apply to,
to both of them, I'd say volume-based.
Yeah.
Backups.
Right?
So either a file system or the, or a,
Disc.
file or a VDK file, right?
Um, and the idea is that we're just gonna do one full, and then
we're generally going to be doing block level incremental backups.
Um, not just incremental, but block level incremental.
What's the difference between those?
Well, a block level incremental allows you to look at changes
within, a specific object.
As an example, say you had, uh, 10 gigabyte PST.
File representing your email and instead of doing a normal incremental, which
would transfer the entire 10 gig file over when you back it up using traditional
incrementals, block level incremental would just take the changes in within that
10 gig PST file and transfer that over.
Exactly, and so it, it, it's another data reduction technology like deduplication.
And it, it, it just, it, I I think it's a, it was an order of
magnitude reduction in the number of bytes that has to be transferred.
And, and the way this worked in, in, uh, file system backups was that
you had to have like an API into the actual blocks, the, the discs, right.
Uh, which.
Was somewhat problematic.
The, the, the easier way to get this was in the virtualization
world where you had an API, right, like the change block tracking API
Mm-hmm.
you can just, the backup app can just say to the, the virtualization
vendor, Hey, I'm here to back up again.
Just give me a pointer to all of the blocks that have changed
since the last time I was here.
Yep.
Um, and then we can transfer just those change blocks, but.
But.
importantly, for a true increment, for a true forever incremental, um, to, to, to,
for me to think that it's a truly forever incremental is that you're going to change
how the data is stored on the backend.
So there really isn't a concept of full, right?
Um, the way that you're storing each new backup is that each new backup from a
restore perspective behaves like a full.
Yeah.
Right.
You
Yeah.
want to like put that in?
Go ahead.
so I was just gonna put it in my own words, which
Yeah.
basically you're able to send only the changes and end up with
the full on the backend, such
Right.
each one is a full, they can be independently restored without
any dependencies on others.
Yeah.
So you don't have this idea of restoring the full and restoring
the incrementals individually.
Right.
You just restore the late, you know, it's sort of, it's, it's sort of like take what
we used to do, take the synthetic full.
Every time we do an incremental, we're essentially creating a new synthetic full.
Yep,
How's that?
yep.
Yeah.
Yeah, because if you think about it, it's just kind of changing
where the processing happened,
Right.
Because with Synthetic fulls, the client was telling the storage, okay, these
are the pieces I want you to stitch together, and it might have been pieces
across multiple different backups,
Mm-hmm.
right?
In the case of an incremental forever, it's like, Hey, you know
your last backup go apply these changes on top of that last backup
Yeah.
offsets.
Right.
Um, and go ahead.
it needs to be a system that allows you to keep those copies as independent,
What do you, what do you mean by that?
um, uh, backup technologies which say, which claim to be incremental forever.
Mm-hmm.
They only keep the latest copy,
Right.
for
I.
you have a full, you apply the changes, you now get a new full.
That old full is gone from the system.
Hmm.
Yeah.
have the most recent copy, right?
And normally for backup, because you want to be able to go back further in
time, you need to make sure you are preserving those previous copies as well.
And so it
Yeah.
require a technology like deduplication, which allows you to efficiently store
multiple copies, snapshots, whatever technology is available on the backend
in order to make that possible.
Yeah, so snapshots and replication.
A great example of forever incremental technology, right?
Each new, um.
Each time you do a snapshot and a replication, you get a new, something
that looks like a full, uh, when really all you only transferred was the, the,
you know, the bites that have changed.
And then, um, I can, I can immediately think of when I was,
when you were talking, I was like, what is he talking about?
And then I realized what you were talking about.
So here's an example of forever incremental that isn't what we're
talking about, and that's replication.
It
Yep.
Right.
Re just regular replication without anything
Yep,
beside it.
You end up with each new, every time you do a a, an if, if you want to call
it that, in an incremental backup, all you're left with is the latest
version and you can't go back in time.
You know, you know what I was actually thinking of when I made those comments?
Oh, what were you,
Oracle incremental merge
oh, okay.
because Oracle incremental merge applies to changes on top of a full, but it
does not preserve the previous versions.
Right.
Yeah, that's a good point.
Um, which, which wouldn't it, that's really just a, a type
of, synthetic full, right?
Yeah, yeah, I didn't even realize that, by the way.
Um, so how do you preserve the history?
That's why
Do you just.
do you need a application that, or a storage system that
allows you to take snapshots or
okay.
copy of the file before it
gotcha.
Why do
where was it?
Curtis?
Why do I still remember these things?
It's why you're here?
So we've defined what a
Mm-hmm.
a little bit about the benefits,
Yeah.
Being able to quickly restore because you have the full copy, uh, being independent,
um, not having to transfer all the data.
Right?
Um, can
And also also cost savings.
Right, right.
Yeah.
you think of any cons?
Well, the, the only thing you know, one of the things I said
in the beginning was that.
In order to do forever, incremental, successfully, you've
got to store the data on disc.
And, and again, not, not to be, you know, pro tape and all that stuff,
but we did an episode relatively recently where we talked about how you
have to, uh, protect the backup disc.
The only con that I can think of is that your entire backup is just sitting
there ready to be attacked, right.
Yeah,
Um.
I have, I have a
Go ahead.
You have, you have a con, okay?
So having worked with vendors, and I'm not saying this has happened, but
it is a potential thing you should think about is when you have forever
incrementals, You're always re.
Copying over the existing backup
Mm-hmm.
bug that somehow makes, that corrupts a part of your backup, right?
You've now impacted how many versions back because you're never getting a
full copy of that data again, right?
It's always
Right,
And so
Yeah.
be one of the cons a
Yeah.
approach.
That was, that was a, um, a critique of ddu, uh, because that
has the same issue, a critique of dedupe when Dedupe first came out.
And all you can say is that you just do your best to ensure
that that's not happening.
And that's why we do fingerprinting and, and, you know, integrity
checks and all that kind of stuff.
your backup.
Yes.
And also test your backups.
Right.
Test your backup, test your backup, test your backups.
Um, but go ahead.
I do have another one too.
If you, unless you
Oh, you do?
Okay.
Go ahead.
The other one I also wanted to mention is, right if Forever incremental backups
are so great, why isn't everyone using it?
Mm-hmm.
one of the challenges is it requires vendor support.
Yeah.
application to support it.
So VMware is supported for virtualization.
companies support it for their own applications, but
not everyone supports it.
Not everyone makes it available.
Yeah.
so you sort of are limited in terms of what applications, what
APIs are available, and can you get the changes that you need in
order to be able to create your forever incremental in the backend.
Exactly, and, and know, I can think of modern database applications
that do support it, right?
Um, I can think, for example, uh, when you back up Salesforce, right?
You're backing up at the object level and it's very easy to.
Uh, and, and let me rephrase that, because Salesforce uses
the term object to mean something very different than what we mean.
Um, because why should we all use the same terms?
of course.
A Salesforce object is like the user's table, right?
Um, so I'm talking about like, I made an update to the, to a, you know, a
field in a record in a table, and, um.
Uh, you can back up that change as right, and, and, and you can do it incrementally
or you can do it independently.
And so it would be very easy to store all of those, those
objects in a, in a way that.
It, you know, would, would foster this idea of a forever incremental.
Um, whereas like Oracle for example, there really isn't a way to do it.
The, at best what we can get out of Oracle is the concept of a, um, synthetic full,
uh, and the synthetic full that they have.
Has the issue that you, that you mentioned before, whereas once you
do the synthetic full, you don't have the previous incrementals.
Um, so yeah, it, that is, that is definitely a, um, it, it
requires your backup vendors to support it, and it requires your
application vendors to support it.
Yeah.
Absolutely.
I can't remember where I was going before you brought up that other, um, I, I,
I will say this, that, you know, the title of this is, you know, how Forever
Incremental Backup changed the world.
It, it, it really has, right?
The idea, when you look at these modern backup products, the
idea of doing a, of a, of a full backup is just, it's just gone.
Right?
Um, all of the downsides to doing a regular full backup.
Are, are gone.
Uh, the, the, the, the, the compute considerations, the
network considerations, the storage considerations, they're all gone without
any negative like impact to the restore.
In fact, it's the opposite.
The restore is now, since we know what a full looks like when we go to do the
restore, that is to going back to Harken, back to the original way we did restores.
We're no longer restoring the same file multiple times during a single restore.
Why would we do that?
We know where all the files are, we know where all the blocks are, whatever
it is that we're, we're restoring.
We know what the latest version of that block here file is, and
we're just gonna restore that one.
I have a story.
Um, how you do forever, incremental Matters.
The original, the OG, forever incremental product is TSM.
mm-hmm.
And when, and the way it was implemented back then was to tape.
And they, tried to bring this concept of forever incremental to the world without
changing the underlying storage technology because back then it was, it was, uh.
tape.
It was, uh, if you, if you brought disc into it, it was, it was crazy talk, right?
So the, so the, so, so they can get a lot of credit for trying to
accomplish this, this, um, this idea.
But I, I still think when we look back to TSM of old, I, I think it was, I
think it was a failed attempt because my earlier statement of the only proper way
to do forever incremental is to do it on disc when they were doing it on tape.
What it meant was that when you, um, when you went to go to do a restore, the, the,
the full was on like 5,700 tapes, right?
sounds worse than just doing your traditional fulls and incrementals.
Yeah, well, it was, and I remember, I, I remember doing a, there was a one
particular restore that I remember doing using TSM in a production environment.
And I, I don't remember the actual numbers, but I'll, I'll just use like,
uh, I'll just use fake numbers just to, to, to to, to describe what it was like.
It was like we were doing like a 10 gigabyte restore and it took like.
Two weeks because of how many hundreds of tapes that it was.
Thousands of tapes.
Um, because of the number of files, the number of tapes that they were
on, and how long it takes to load and unload an individual tape.
Right?
Um, and they had technologies, uh, called reclamation and they, they,
they had a number of technologies that tried to mini minimize this issue.
Um, but
It still existed.
well, yeah, it still is existed.
And also some people, they would run out of time in the day to do reclamation.
Reclamation is where you take a bunch of tapes that, um, like they were, that
most of the files on those, on those tapes were no longer needed because
they, they'd expired and then you take, yeah, I've got a bunch of files that are
10% and then you copy them to one tape
It's
to that's now a hundred percent.
It's like garbage collection, but on tape.
And it was a very, um, labor intensive process and, um, IO intensive process.
And as a result, it took a long time.
And so some people would, would turn it off.
Um, and, uh, I think that's what happened in, in, in this case.
And so there, there were literally just, I don't rem there were so many
tapes that we had to restore and it, you know, it sounds like I'm bagging
on T-S-M-T-S-M is still around.
It's now called, uh, spectrum Protect.
Yeah, but the addition of bringing in disc, I think makes this, this whole
issue kind of, uh, not a big deal,
Yeah.
but, um, but yeah, I still think that you look at all of the modern backup
products that have come out in the last 10 plus years, and they're all like this.
They're all one full, followed by a whole bunch of incrementals, and
the world is a better place for it.
And, um, you young whippersnappers have never had to deal with
tape
incremental backups.
You just don't know
what
what you uh, yeah.
You don't know the pain that you, that Yeah.
That you had.
You know, it's funny, this morning I was hanging out with some people and I
was talking to 'em, and they're, it's this lady who shares my birthday, but
she's about 15 years older than me,
Hmm.
we started talking about all of the technology that we
remember when it came out.
Right.
And the first, the first.
Piece of technology I remember being introduced in my life was the, the VCR
Hmm.
and it, it came, they, I remember them wheeling one into the classroom
Hmm.
in my school.
And it was gigantic.
Right.
It was like, and it was the kind where you put the tape in and push it down, right?
Yeah.
Um, anyway, this is, this is like that, this is like the
way, the way we did backups.
And by the way, you know.
I'm, I'm glad I, I'm glad I remembered this because you asked a question earlier.
Why is it that not everybody does things in this new way?
Yeah.
Because backup is the stickiest application in the data center.
Yep.
So many people are still using the same backup application
that they were 15 years ago.
Some people change their applications like they change
their underwear, but other people.
They just, it, it's a pain.
It's costly, it's risk, uh, it's fraught with risk and, and, and
difficulty and a giant learning curve.
And during that learning curve, your, your data's at risk.
And so most people just, they'd rather stick with the ugly, old backup product.
it.
Because it works.
Yeah.
And, and, and so it, it, it gave birth to incrementally helpful
technologies like deduplication.
Don't change anything.
Just send your backups over here instead of sending your backups over there.
And we'll get rid of the duplicates and we'll make it sort of look like a,
an incremental forever, but just keep doing it the dumb old way because we
don't want you to have to change your backup product in order to get to,
and, and I guess this is, you know.
If, if you're one of those people that's still using the backup, the
same backup product that you had 15 years ago and you're still using it
the same way you were using it 15 years ago, and you're not using incremental
forever as a, a true technology in your world, maybe time to think about it.
but I, I would probably say.
Newer workloads, people are probably using that, right?
Because once again, environments are a mishmash of technologies, right?
Yeah.
people probably do have those old databases, old applications,
which they are using an old infrastructure backup infrastructure
for, and for their virtualization or other newer workloads, they're
probably using something different.
Yeah.
Uh, remember I did that, I did that survey a few, uh, months ago and
I asked them how many backup apps they had, and 30% had 30, I think
more than 30% had three or more backup apps in their environment.
Yep.
And it's because of what you're talking about.
They have, they start using.
Containers and their current backup app doesn't know what to do.
And so they use that for the new technology and yeah.
Yeah, that's very common.
And I guess what I'm saying is maybe it's time to check out the water of
the, of, of truly incremental backups and, and experience all the beauty.
Yeah.
All right.
Well thanks for the chat again.
Anytime Curtis.
Anytime.
All right.
And we wanna thank our listeners.
Yo, you're why we're here, you know, otherwise, just two guys sitting here
talking over a couple of microphones.
Uh, that is a wrap.