You found the backup wrap up your go-to podcast for all things
backup recovery and cyber recovery.
In this episode, you'll hear the harrowing tale of what I'm
calling the backup from hell.
A project that started as a simple one-time backup, a 40 terabyte
of two sonology boxes that turned into a 400 terabyte nightmare
that took months to complete.
We're talking hundreds of millions of files with one directory alone
containing 99 million of them.
I'll share how I dealt with failing tape drives ridiculously slow
backup speeds, and ultimate solution that finally got the job done.
If you've ever wondered what happens when everything that could go wrong
with the backup actually goes wrong.
This episode is for you, plus you'll learn some valuable lessons about what to check
before starting a massive backup job.
By the way, if you don't know who I am, I'm w Curtis Preston, AKA, Mr.
Backup, and I've been passionate about backup and recovery for
over 30 years, ever since.
I had to tell my boss that we had no backups of the production
database that we just lost.
I don't want that to happen to you, and that's why I do this show.
On this podcast, we turn unappreciated backup admins into Cyber Recovery Heroes.
This is the backup wrap up.
Welcome to the show, and if I could ask you to just take one quick second
and, uh, subscribe or follow us so you can make sure that you get all of this
great content, that would be great.
I'm w Curtis Preston, AKA, Mr.
Backup, and I have with me a guy that apparently owes Ben Kingsley
a huge apology Prasanna Malaiyandin
how's it going?
Prasanna, why do you owe
an apology?
so as everyone's probably like, who's Ben Kingsley.
So if you don't know, he is an actor and he also played Gandhi in the movie Gandhi.
He did.
Right?
And for the longest time I was a little, not upset, but like the fact that you have
like probably one of the most important Indian people in history being played
By a guy with the name Ben Kingsley.
Exactly.
Yeah.
Ben Kingsley.
And so today I found out that Ben Kingsley is actually Indian.
Half
How about that?
should say.
Yeah,
what?
he's Anglo Indian.
Anglo Indian.
Yes.
It's like us.
You and me we're Indian.
so his paternal side is from Gujarat.
Right.
And his mom's side I think is European.
His dad was a physician who was born in Kenya.
And Ben Kingsley's name is not actually Ben Kingsley.
It's like Krishna Bunge, I think
Yeah.
And he realized that he wasn't getting called into the right casting
roles when he was looking for, when he was starting off his career.
So he is like, let me change my name.
And so we changed his name to Ben Kingsley and people started calling
him in and he started getting roles.
Racism in early Hollywood say, it isn't so.
Racism in current Hollywood.
Say it isn't So, Wouldn't be the only to do so.
Yeah.
yeah, so I apologize to Sir Ben Kingsley, uh, for all these years.
Yeah.
You were putting it in the same category as the quote unquote
Indian guy from the Short Circuit movie, which I don't know his name,
but he is very much not an Indian
person.
Do you know who it was?
the name?
I'm looking it up.
Or it's also like how Apu from, uh, the Simpsons is not Indian,
Yeah, he's, he's played by, um.
Oh, I know that.
I know the actor, but his name is escaping me.
So Fisher Stevens, is that
Fisher Stevens.
Yeah, Fisher Stevens.
Who?
Those of you that watch succession
will, uh, uh, Fisher Stevens was in succession.
He was, he was a, a lawyer, a a smarmy lawyer, which
always plays smarmy characters
yeah, I was just thinking, because I remember him from the blacklist
where he plays Marvin, the lawyer.
Yeah, got, he's got kind of the lawyer face.
I'm glad that you, you finally realized the error of your ways.
But did you know he was
No, no, I didn't.
I guess I always brought it up just like you, like I would bring
Ben Kingsley playing Gandhi and, um, as just another example of, uh, you
know, what would we call it, brown face, I guess we'd call it brown face.
Yeah.
But people taking actor and there's been a lot of those great roles throughout the
Great.
You know, great roles played by very not,
you know, people that are not of that ethnic group.
Yeah.
and I think maybe also at the time, right, there weren't many
Indian actors in Hollywood at all.
And I would rather have the fact, or I would rather it like the movie be made
with someone who is non-Indian, rather, because it's a great movie.
I
don't know.
You've seen it,
good movie.
Yeah.
So I would rather have that rather than not having the movie at all.
Hmm, I see what you're saying.
I see what you're saying.
Yeah.
And of course, you know, we have the same challenge with, uh, Asian, uh, actors,
right?
Uh, there's literally only three Chinese actors in all of Hollywood.
Like if you, if you look at like the Chinese roles, they've gone to
literally like one, there's one guy.
Uh, I forgot how many roles he's had, but he has had a prolific career playing
every Chinese person that you know.
Um, but, um, anyway, so we're gonna talk about something that we've
alluded to a little bit on the podcast.
Uh, sort of tell the final saga of what I'm calling the backup from Hell.
I may maybe, uh, we should probably phrase that slightly differently.
It's probably the,
the backup that keeps giving.
the back, the backup that, yeah.
Uh, what a mess.
The beginning of the story
that I was asked to do a backup of two Synology boxes that they
were, uh, repurposing, right?
So they were, um, going to move the data.
They, they were gonna reuse these servers, but they wanted to get a backup of the, of
the, the data before they moved it off of
Backup is good.
Yeah,
Backup is good.
Yeah.
Apparently they hadn't had a backup of the, of these servers before.
And, um, then the, the, um, and, and , they said it was
about 40 terabytes of data.
That's the information that I was given and after I had started doing
the backup, I very quickly realized that 40 terabytes might have been.
An understatement.
You, found additional data around
right as you
data.
Yeah.
Uh, so it turned out that it wasn't like 40 terabytes of data.
It was more like 400 terabytes of
Yeah, and
I'm guessing because these were systems that were kind of probably off on the
side, they hadn't been used in a while.
Like that's, I think, the problem, and I think we talked about this in one of
our episodes about sort of systems that kind of get stored away in the corner.
No one worries about
it.
Right?
And do you leave it powered on your old backup systems?
Right.
We just talked about that.
And so I think that becomes a challenge.
It's when you have these systems that are no longer actively being
used, it kind of gets away from you.
Yeah.
And so the customer really didn't have any idea just how much data that they
were dealing with here, out to be, like I said, like close to half a petabyte of
Yeah.
And, and for you, that changes things significantly because
changes the backup design like massively.
Yeah.
because your backup target, I think you had mentioned previously
that it was like a server, right?
That you were backing this data up to
I was backing it up via a server, a window server.
And, um, and tape, right?
Had tape, but it's sized like, um, you know, four 40 terabytes.
And so, which is, which is basically the, the, the server and the tape
library was the perfect size for that.
But as I started realizing that it was figure, it was filling up.
And again, this, this is my fault for not really looking at the size of the
data before really jumping in there, but basically I realized very quickly
that this was a whole lot more data
than,
That you expect it and and I think just kind of looking at lessons
learned as you're a backup admin who is being told, Hey, this new
application is coming online.
Make sure that you understand like what is the expected growth of that application.
Because what you size for, say, a five terabyte database with a 1% growth
is very different than like a file server with like a 50% growth rate.
Yeah, exactly.
Um, and just because somebody says they have 10 terabytes of data doesn't mean
that they have 10 terabytes of data.
So you mentioned you had a backup server, you had a tape drive.
Is there a reason you chose to use tape
Well, the, I mean, tape is great for long-term retention of data,
which is what this customer wanted.
They wanted to hold onto this data for a long period of time,
and that's where tape is great.
And tape also is has, uh, you know, if you're able to properly feed it,
tape is actually, can be quite fast.
the challenge that I had when backing up this data that for various reasons,
which I think I, I think by the end I sort of figured out the, the
core reason for various reasons.
Individual backups off of the, these filers, the, they were just
slow, just, um, you know, they were
Like how slow, slow.
like, slow, was like, like, like three and a half kilobytes a second slow.
So like slower than like a 56 K modem back
Yeah.
Right.
And you can multiplex all you want.
So first off, you know, I, I was using NetBackup, which, you know, NetBackup, it
did a great job at what we had available.
Um, the, challenge was that because I couldn't put.
The client on the filers themselves.
So the, was a way allegedly to put a, a backup client on the filer,
but I could never get that to work.
And so I had to back up over SMB because I'm backing up
over SMB, I'm just, I'm just.
I'm just limited at what that was, right?
What,
I could get, and because I'm backing up over SMB, the client
is just the backup server,
right?
So instead of running a backup from two clients, I'm running a backup from one
client because that's the backup server.
I'm backing it up over SMB.
And because of that, I'm limited to the number of jobs I can run at one time.
NetBackup, um, says 99 99 jobs, which should say, gee, that
sounds like a
Nine problems.
right?
But, but the thing is, towards the end, as I was running a lot of these backups,
the aggregate speed of like 99 backups was only like 30, 40 megabytes a second,
you
you're talking about 400 terabytes of data to
400 terabytes of data doing the math.
I backed up for months,
right?
And I tried all these different things.
Uh, you know, num, you know, was I running too many backups at a time?
Was I running not enough backups at a time?
You know, it, um, you know, and then the problem is every, every
test would take days or weeks.
Think we should mention one thing.
You were talking about these test taking days or
mm-hmm.
and then do you wanna mention sort of some of the issues you ran into with these long
running jobs just due to infrastructure or
Yeah.
other issues in the environment?
yeah, you, you backups are not made to run over weeks or months.
Just backup infrastructure isn't made to work like that.
And so when you do backups over weeks or months.
Weird things happen that, cause you know, consternation, one of the things
is LTO tape drives are great, but like we were using like the half high LTO
drives and as far as I could tell, their duty cycle was not meant to
be a hundred percent for two months.
Right.
Um, they're meant to be backed up for, you know, several hours and then give
'em a rest and then back up several hours and then give 'em a rest.
I was just beating the crap outta these things for weeks or months at a time.
And what would happen is after some significant period of time,
it would just go write error.
And that's fine when a backup runs for a few hours and then just try again.
But if you, but if it took you two weeks or three weeks to get to that point
and then you get a write error, um,
then
it's not like you could restart these jobs either, right?
I think you're running into
Yeah.
Well,
I mean, I mean, I could restart em, but, but it's like after
a period of time I became, I.
I eventually got to the point where I said, tape is not my friend.
I, anybody who
this is coming from Mr.
Backup.
know anybody who listens to this podcast knows that I am, I am a friend of tape,
right?
I believe strongly in tape for a lot of reasons, but I don't think that, uh,
specific and, and you know, maybe the, my LTO friends can chime in here, but I don't
think that these tape drives were designed to be backed up to like this for weeks and
months at a time, 24 7 with no, because as soon as one, I was multiplexing
as many backups together as I could.
And when one backup would finish, I would just add another backup onto it, right?
Because
I, I could, I could, I.
what I couldn't do is I couldn't say, well, let's do these 10 backups, let
them run until they're finished, and then we'll do the next 10 backups.
And that would've given the tape drives a, a moment to breathe, I think.
But, uh, I couldn't do that because the, because we, we just
didn't have that kind of time.
And so I
was just, I was just try, you know, tagging it
and, and I know you've always talked about like the shoe shining problem,
given that you're not going very fast with these backups, right.
Do you think that also led to some issues as well for the tape drives?
yeah.
So again, the core problem was that each individual backup was running slow.
matter how many of them that I multiplex together, it was not enough
speed to make the tape drive happy.
And so, yes, the tape driver shoe shining.
And when a tape tribe is continually shoe shining, the tape drive will fail.
And so everything, I remember learning about tape drives was
coming back to haunt me, right?
Um, this is all of the design that I was, that I had done throughout
the years on backup, um, you know,
um, backup system
And system.
all of the things that, you know, what do you do when the backups, you know?
And so I came to understand
that the only way I was gonna finish this backup was to do it to disc.
And just quickly before you move on, I think along the way, didn't
you also have a tape drive that failed that you then had to go
Oh, multiple Multiple times.
Swap out tape drives, reboot tape drives, put in cleaning tapes and tape drives.
And by the way, that's another thing is the way tape drives normally do
is you run them for a certain number of hours and then there's a cleaning
tape that goes in there and cleans it.
And when you have a robotic library, that happens automatically.
Well, when you just run the tape drive for.
Two months, you know, that
And so at some point the tape drive just fails.
Yeah.
um, yeah.
And so I ultimately that the only way to get this done was to, um, you know,
buy, uh, enough disc to back this up.
And that wasn't cheap.
Uh, but I, I didn't think that there was any other way that this was ever
going to get done 'cause again, the core problem that we've had with tape
for the last three decades has been that the backup, if the backup isn't
too fast enough for the tape drive it's a, it's a fundamental mismatch
right?
And so we use to make that better.
But if the multi, but if the speed you're dealing with is in kilobytes a second,
Yeah.
Well, and especially 'cause you're limited by those two, uh, Synology boxes, right?
Which are limiting your bandwidth, right?
It's not like
Yeah.
Synology boxes you can then pull from,
Yeah, and I was, I was watching, like, I was running every kind of tool I could
run to see, like, I wasn't overt tasking.
The, that was the really weird part is that the, it's not like the
Synology boxes were saying, you're really beating the crap out of it.
You shouldn't do so
backups at a time.
It wasn't, it, it was, I didn't have a high I/O wait.
I didn't have high CPU, I didn't have high ram.
There, there was no, there was no
rhyme or as to why we'll get to the rhyme or reason later.
I figured it out.
Um, but, but I knew the tape and I knew the tape and this wasn't gonna work.
So, so I had to bring in, uh, a couple of other Synology disc arrays, by the
way, and populate them with enough disc to handle all of this, uh, this backup.
Right.
Yeah,
And, um.
Then
but that wasn't without its issues either.
Right?
When you, when you brought those in, that wasn't without its issues either.
No, it wasn't without issues.
And the other thing, what I needed to do was to, I felt that with, in terms of the
number of directories that were remaining, I wasn't sure like the different sizes.
So what I did was I split, I.
Those jobs into many smaller jobs.
NetBackup is really good at like running thousands of jobs, right?
So rather than just have a hundred jobs, I turned that into like 2,400 jobs.
Like I went,
I went another level deep and created a policy for each of these
directories, and then I ran those and it was running for a while.
It was, it was, you know, again, more time.
And what I started seeing.
Were these jobs that were like an individual job that was running
inordinate amount of time.
but you also had some jobs that would finish like super fast, right?
Like
They'd finish five, they'd finish in
Some of 'em, some of 'em finished in five minutes, some 'em would finish.
But I noticed that over time there were certain policies that were running for
really, really long periods of time, and eventually started poking around.
when I discovered what ultimately was the, the true culprit.
And, uh, anyone who's been around backup for a long time
has seen this culprit before.
It's just, this is the worst example of this culprit that I've ever seen.
And what is that?
We affectionately refer to it as the million file problem.
Hmm.
Because remember, again, going back to that, um, that client back from
25 years ago, we had one server.
That was going to be storing a bunch of images and it was going
to result in millions of files.
And we knew that back then that the million file problem is, a real problem.
and and million file problem ev over, over the network is even worse, right?
Because everything is, is, is a
round trip.
The way we fixed it back then was we used a product back then called
flashback, which would back up at the raw level, but store the
information, and that was not available to me.
Why?
Because that product no longer exists
No.
because it doesn't run on a Synology box.
Right.
Remember, I'm not the Synology
All it was was an SMB mount to me.
Right?
And by the way, for those curious, yes, I tested SMB, I tested NFS.
It didn't matter.
Um, the um.
And
by the way, this was a constant, you know, you know the phrase, never, never
go into battle with an untested weapon.
This was constant example of I am in the battle, I'm in the stuff,
and now I'm trying to test stuff
and, and I did to try to make things better, just made it take longer
and the client just had to wait.
And the the client was incredibly patient, honestly.
And, and you know, I did my best to say, look, I, I've been doing this for 30
years, I've never seen anything like this.
Right.
And that, that helped.
But in the end, I was backing up.
You know, we got down to, I, I learned a way to identify which
were the problem directories.
So I would kick off a policy and I would watch, and I would notice
that had run for, let's say an hour.
And it listed, let's say 300,000 files backed up.
kilobytes.
Hmm.
Literally there's, there's a kilobyte column that
kilobytes of byte and there's no value in there.
We backed up 300,000 files, no kilobytes.
so that, that helped me identify these problem
Problem child.
Yeah.
it and let the other non-problem policies finish.
And
Right.
Yeah.
up getting down to like 150 policies that were the problem policies.
And so I backed them up and I was able to get them.
Over time, I was able to get them backed up, and then finally I got down to about
20 policies, I think somewhere around
policies.
Go ahead.
And at this point when you're down to the 20, like some of these have
been running for a long time, right?
Like how?
like two months backups that have been running for two months,
successfully running for two months.
Yeah.
And what was good was at this point again.
Like this is information that would've been really helpful to have at the
beginning, but it was information that, to get all this information at the
beginning, it would've taken time to, like we, we just wanted to get started.
Yeah.
What I ended up finding was that, um, these backups, um.
The, the, there were millions and millions and millions, like one of the, one
of the directories that I was backing up, it had 99 million files in it,
one directory, 99 million files, and eventually what I realized was that
again, the problem this time was just SMB.
So the fact that every one of these files results in a round
trip conversation, possibly multiple round trip conversations.
Yep.
And I realized that the only way I was gonna back up these truly problem
directories was to back them up locally.
But how do I back them up locally?
Well, luckily this is when I just, you know, basically go back
to dumb, dumb old backup tools.
And so I was able to run a backup using tar logged in locally
on the filers, and then just.
Directing the tarball across the network that finally worked.
That's crazy.
So you had these 20 jobs, right?
And some of them you said were running for 60 plus days, and then you sort of
were like, okay, let me start this over.
And by the way, you were kind of forced to start them over
because something happened right?
At
yeah.
Something some unknown thing.
Um, I think I.
I, I, I don't know.
I, I actually don't know
what caused it, but they, they did fail
and,
And you were like, I'm not gonna start these
yeah.
I'm not gonna start 'em again.
It's just, yeah.
Well, Because
like, one of jobs, the, the one with 99 fi, 99 million
files, we were nowhere near.
I.
yeah.
After 60 days you were barely
yeah, yeah.
We're barely, barely scratching the surface.
so I'm like, I, I, I don't have, I don't have that, you know, I, I don't
have the amount of time that it would take, so, so I switched to, you know,
experimentally once again, experimentally, I'm experimenting on the fly, I'm
doing development in production.
Uh, I was like, well, let me see how long, how quick a tar ball would run.
I ran a tar ball.
I remember for like a day, you remember this?
I ran a
a day and it, I, I had a du of the size of the directory and after a day it had
done like, like a half of it or something.
Yeah.
You're like, what?
Once taking 66 days and barely scratch the
yeah,
You are mainly done.
Almost done within a day.
yeah.
And so I was like, this is the way.
Right.
So it, it, it wasn't, it wasn't a way for everything because the, the, this
was, um, because I, you know, I'm glad that I, that I use NetBackup for the
bulk of it, because then I have the catalog data and, you know, and, um,
but
on the restore side.
yeah, yeah.
So this will.
This will be the diff the restores will be more difficult for these
like remaining 20 directories.
I mean, not, not astronomically.
So like,
you know, can create a tarball, a
list of this.
So, you know, lessons learned, like,
do that.
Don't store millions of files on the other side of a, of an SMB box.
I guess
Yeah, so Well, and I think a couple things, even if it's not SMB, right?
Just having that many files, because I think what people don't realize is
even though the size of every disc has gotten significantly larger, right?
You're talking like 18 terabyte, 20 terabyte disk
Yeah.
They can only handle so many operations per disc, right?
That number hasn't changed.
It's about a hundred per second.
And so no matter how many, how big your disc is, right?
If it was 21 terabyte discs, right, then you get 20 times a hundred iops.
Versus if it's one 20 terabyte disc, you only still get that a hundred.
So that's a big thing that people don't realize with these larger size discs.
Yeah.
And, and the thing was that the.
That many files.
So, because the problem, the, ultimately the problem wasn't disc io, the problem
io.
Right?
Network latency.
So, because
when I actually ran, I ran two tar balls.
I.
Simultaneously is what I did.
I using
I just, I ran, I was always running two at a time.
When I was running two at a time, I/O wait was sitting at 10,
which is, is high,
but I was like, well, it's got nothing else going on, so I'm, I'm
it go.
Right?
The highest I/O wait ran during all of those hundreds of
simultaneous backups was like four.
yeah,
So like I wasn't disc bound.
I was
bound, but not network bound in terms of throughput, network bound, in terms of
Laid C,
and
of operations, just because SMB is very chatty.
very chatty.
It's probably the chattiest of the protocols,
and
we, you
it was just a really combination.
Yeah.
And you know why this, and this is why backup vendors have their own protocols,
like Data Domain has boost, right?
To help alleviate and solve some of these issues.
Yeah.
You talked about, don't, don't do the somewhere we were talking about.
Just don't do this.
I, I'd like, I'd like to talk today.
When I looked at these, these, uh, these directories that had these
tens of millions of files, it was a structure that was very clearly
created by some application.
one of these directors had a common structure created by some.
I'm gonna say stupid application that thought this was perfectly fine.
That it was perfectly fine to create 99 million files for
Do you know, I,
item.
I bet they were using the file system as a database
I don't know.
what it was.
given just like the number of files and the size of those files.
I know it was forensic type information
and I, I don't, I clearly
That, that's fine.
Yeah, yeah,
No, I'm just saying I clearly don't know enough about forensic stuff
to know why they would want tens of
of vials,
but
So where are you?
So you talked about these 20 jobs that you were starting to do tarballs with.
So where are you right now?
So, so we finished all of them, but one, there was one that for some reason
it, it, the file didn't look right.
It was weird.
Um, it, the, the, the backup completed, but the, some reason, the, the tarball,
it just, it just didn't look right.
I don't wanna go into details.
It just didn't look
so I'm rerunning that one.
So it, based on its size and how well it's doing, it should
finish in about a day or so.
Um, and what I'm
is a significant improvement in terms of
A significant improvement a day versus, you know, a year, um,
Or two, I think actually it might have been two.
Yeah,
Agreed.
Um, and what I'm doing is I'm, because again, I don't have the catalog.
What I'm currently running is I'm running a tar TVF.
On all of those files and creating tarballs or creating, I'm sorry, text
files, a list.
of the, the files that are in there.
And then I'm gonna do a count on the files that are in there and
check it against the count of the files that are in the directory.
And, and hopefully those numbers should be the same.
Yeah, because I believe you are even saying that to run things
like a find to get a list of all the files in a directory or a DU
Yeah.
hours, right?
Well, it was days actually.
In
fact, it was why I didn't have this information in the beginning
because everything was so big and every find, every du every command
that I had DU is quicker than find.
DU is.
It just does less work than find.
But the problem that I ultimately realized was that DU wasn't
really being helpful in terms of.
The
scope of the job, what was the scope of the job was determined
by the number of these files.
And I couldn't get those numbers because that was the thing that took forever.
the number of jobs dwindled down to about 20, that's when I
was able to run these, uh, the
and they would, they would actually complete.
And that's when I realized just how bad it was.
so if you had to start this over, and hopefully you never do, but I'm just
saying, if you had to go back to day one, what would you do differently?
I know you talked about making sure you understand the size of your backups.
Right.
It just feels like some of these, you just have to go through the process
though because you don't know what to do.
Like it's not like you could just start day one and be like,
oh, I know I need to go to disc.
I need to do X, Y, and Z.
Right?
It's sort of like a learning process.
would say that I.
Yeah, because the problem is you're going off into the unknown,
you're doing a backup of something that you don't know what it is.
And I, I would say if possible, if at all possible, get things like
dus, uh, you know, discus it, it's a Unix command, but you can load those
tools and windows as well get, like if you're going to back up, if you're
gonna back up a hundred directories.
Get a du of every one of those directories so that you have an idea
of just what you're dealing with,
if at all possible.
Also, look and see if the number files and if the number of, and if you're
trying to do a, you know, it's not that hard, you just run a fine dot dash,
you know, I didn't even do a print just fine dot pipe to wc -l, right?
That was it.
Right?
Um, to, to get the number of files.
I'd say if again.
If I could go back in time, I, I would say maybe do a little bit more of this
research prior to beginning the job.
Um, but that's diff it's, it's easy to say that now,
um, because I know what
I know.
Right.
Um, but the, you know, the core problem was that you've
got these millions of files.
I mean, which is all.
Already gonna be a problem if you're backing it up in any sort of normal way.
But if you're
up remotely over the network, it's going to kill you.
Yeah.
So, um, you gotta figure out a way to do that.
And then I would just say, see if there's anything that you can do with the, with
the application that's created this data
which is why it's important to get involved early on, right when an
application is being developed or deployed, right, to get involved so
they understand the backup requirements.
yeah.
And so, this backup that would never finish, I literally was, I
was starting to think that this thing was never gonna finish.
Um.
It's essentially finally, I mean, it's not, at this point, it's
not a hundred percent, but I'm, I'm now, you know, it's just, I'm
at the finish line.
Yeah.
at the finish line.
Yeah.
Um, it's nice.
I know one of the other things you mentioned that you were using
NetBackup, but you had also looked at other tools out there as well, right?
That could potentially help you with this effort.
Right.
So do you think that that becomes valuable, like either looking at other
tools, um, I know you had reached out to like synology support, you
had reached out to some experts, like
Yeah.
The problem there, there were, there were, you could do, like with Synology,
you can like copy the data from A to B.
Mm-Hmm.
They have this ability essentially like, you know, for lack of a
better word, they have Snap Mirror.
they have the equivalent of Snap Mirror.
Yep.
from onSynologygy box to another.
But to me that wasn't really a backup like I wanted in a, in a format, you know,
the end I was forced to not do what I wanted with the tar.
Um, but I wanted it in a cataloged format.
So we looked at a couple of, the problem was never NetBackup.
Right?
NetBackup made it, um, easy to script this whole thing because it was the
only way I could make sense of it.
'cause it was, it was thousands of directories and, um, and even
more thousands of sub directories under those directories.
And the only way I could make sense of this was to script it all.
And, um, the, the fact that NetBackup allowed me to do that was great.
Um, there are some other tools these days, some of the newer tools,
they want to make it easy for you.
But if you get into a complicated situation like this, some of the newer
tools don't even have the ability to sort of grab it by the horns.
The
able to do a NetBackup,
Yeah.
I think the other thing also that you were doing, which I thought was interesting,
was also your scripting, right?
Trying to automate this, like, uh, I know like scheduling your,
the backup policies to run, right?
And then you were sort of doing load balancing to make sure
that you keep the two filers
Yeah.
I couldn't, yeah, that was the thing.
I couldn't normally, I, I just, I believe in just throwing
everything in the NetBackup schedule or, and let it figure it out.
But because again, because of the limitations of the weird thing I had,
I, I couldn't figure out a way to load balance across the two target filers.
the NetBackup scheduler.
Um, maybe I could have, uh, done that better.
I don't know.
But, uh, so the way I was doing it was I was just assigning a backup.
a backup would finish, I would assign the next backup to that, that the
was now had more space available to it.
Right.
So I just had a while loop that was running, you
know, checking to see if a backup job was done.
but I think that's important, right?
You can always script some of these things that if it doesn't
exist in the native tools, right?
Don't be afraid.
Yeah.
Don't be afraid.
you know, obviously I'm, I'm pretty good at scripting and
I'm pretty good in the backup.
And, um, th there are, and, and, and, and thanks.
Thanks very much to Veritas for keeping their, uh, their documentation online.
Uh, the number of times I Googled.
You know, backup job, you know, how do, how do I list, uh, you know, and
I know there's a, there's, I know there's a command to, to do this.
How do I do that?
And, you know, and then a man page would come up and I would read it
and I was like, oh, yeah, yeah, yeah.
It's
been a while.
Yeah.
Um.
you have to also thank Cygwin, of course.
Yes, special thanks to to Cygwin Without Cygwin.
That is the tool that you can download and run on any Windows
server to give you Unix capabilities.
I will say there were, there were moments where Cygwin was both helpful and
terrorizing me because it was the whole like backslash versus forward slash thing.
Because in Windows, you know, the file separator is a backslash, which
in Unix is an escape character,
Yep.
and Cygwin wasn't consistent.
When that escape character would be an escape character.
Like, like if you piped it into a file, it would do one thing.
If you piped it into a command, it would do it, it would behave differently.
And, um, so that, that definitely l lent.
The fact that I was doing constant file manipulation on directories
that were seven levels deep,
Yeah.
did not help.
Yeah.
Oh, and then I couldn't, the, the, the, the one thing with
Cygwin is that it doesn't see.
It doesn't see the, to point the backups to NetBackup, I have to point
'em in the backs back slash filer name
share name.
Cygwin doesn't see that.
Cygwin sees only mapped drive names
and
have to map it using
you have to map it to a drive name.
Let's say you map it to,
to letter F, and then in Cygwin you would see /cygdrive/f.
Which would be the same as this backs slash backs mount.
know, I was constantly having to go back and forth between
those two and, and that was fun.
Um,
scripting
here's the thing.
After all of this experience and everything you've learned, you're probably
never gonna use any of this again.
I don't know about that.
I dunno about that.
I tell you what, I'm, I'm taking a tar, all those scripts that
I wrote, um, because I will say this, that, that the NetBackup
documentation while, uh, extensive, it doesn't give a lot of examples.
And so like, I'm thinking of like, um, like the BP duplicate command,
which is the command to copy backups from one place to another.
I couldn't, I couldn't figure out from reading the man page how to
actually do, to do what I needed to do.
So I would, I would like.
I would do, I would have to run tests, you
know, I'd, you know, um, and, um, the, you know, not like now that Cohesity's
acquiring them, it's not like they're now gonna rewrite their man pages.
I just thought that they could have used some more, some more examples.
But
Yeah.
I figured it out eventually.
You know, I think someone used to have a forum that people would post on about.
Yeah, someone used to have that and then, but people stopped posting
on that forum, so I don't know
You know?
Um, where people are getting their help now,
but, uh,
Well, I'm glad that this is almost over,
yeah.
Yeah.
nearly over and I'm glad you're still alive,
I am alive.
I didn't kill anyone along the way.
I didn't scream at anyone.
Like the, the story that
you have heard were, were Curtis Cuss Preston.
I didn't scream at anyone.
yeah.
but I really, really, really think you should do an office space on those filers.
yeah.
Well, that would sort of defeat the purpo of the
but, uh, I, yeah, I, like that idea.
Hmm.
Anyway.
Well, uh, thanks Prasanna for helping me, uh, sort of through this.
You were my constant counselor through this.
I think I learned a bunch.
I know usually I'm all about YouTube knowledge, but in this case it was
the Preston knowledge, so it was good.
I.
Yeah.
uh, thanks everybody else for, uh, uh, listening along with this sad, sad story
with I think a decent, happy ending.
That is a wrap.
The backup wrap up is written, recorded and produced by me w Curtis Preston.
If you need backup or Dr.
Consulting content generation or expert witness work,
check out backup central.com.
You can also find links from my O'Reilly Books on the same website.
Remember, this is an independent podcast and any opinions that you
hear are those of the speaker.
And not necessarily an employer.
Thanks for listening.