W. Curtis Preston: Like my friend, Jeff.
See you do cigars.
You get COVID.
That's it.
That's that's the lesson that
Jeff should learn.
Prasanna Malaiyandi: Curtis and I are not medical doctors.
We are not providing any medical advice.
Please go talk to your doctor or do your research.
Thank you.
W. Curtis Preston: please don't get your medical advice on
Backup Central's Restore it All
hi and welcome to the Backup Central's Restore it All podcast.
I'm your host.
W.
Curtis Preston AKA Mr.
Backup and I have with me, my COVID isolation grief
consultant, Prasanna Malaiyandi
Prasanna Malaiyandi: oh, no.
no.
Oh, Curtis say it ain't so,
W. Curtis Preston: So I don't have COVID, but I just spent six
hours with Jeff who has COVID
Prasanna Malaiyandi: oh,
W. Curtis Preston: and I just got told, so I now have to isolate.
For the next few days, you know, and here's the thing I had a
trip planned, uh, and purchased.
And so, you know, Druva's uh new corporate office opens up this month.
So there's this big shindig.
Marketing department has their first, uh, like social gathering since COVID.
And I will be at neither of those things.
I will be at home hoping that I don't get COVID.
Prasanna Malaiyandi: When is the event supposed to be?
W. Curtis Preston: uh, Wednesday.
Prasanna Malaiyandi: Oh yeah.
Okay.
Yeah, you're not.
Oh, I'm so sorry.
W. Curtis Preston: yeah.
Prasanna Malaiyandi: sucks.
W. Curtis Preston: See, I knew that you would, you would console me in my grief.
Prasanna Malaiyandi: I hope Jeff is feeling better or feeling OK.
W. Curtis Preston: I called him, you know, he, he texted me.
He's like, yeah.
So I have COVID and I call him.
I was like, dude, like of all the weeks, you know, and you know, and you know,
not your fault, but of all the weeks.
And he goes, well, if it makes you feel any better, you did the same thing to me.
I'm like, oh yeah.
Right.
Cause when he, he came over for Christmas dinner and then, uh, I tested positive.
Prasanna Malaiyandi: So since we're talking about, uh, isolate, not isolation,
but sort of needing grief from sympathy.
So I think I called you over the weekend and told you how I had food
poisoning, which was not a fun thing at all, but I'm happy to say that
I feel much, much, much better.
It was like 12 hours in my system felt awful.
But back to normal.
And I, think that might be the second time I've ever had food poisoning,
W. Curtis Preston: Yeah.
I've had it more than that.
Um, but, and you may recall I had it last year.
Prasanna Malaiyandi: Yeah.
Which is why I called you for advice.
I'm like, Curtis, what do I do?
W. Curtis Preston: yeah.
Yeah.
It's more like, I'm like, go get a big greasy hamburger.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: I just tried to spill the tea on my desk.
Well, technically I did spill the tea on my desk.
I just didn't spill all of it.
Um,
Prasanna Malaiyandi: So have
W. Curtis Preston: me while
Prasanna Malaiyandi: gotten tested yet or you're waiting
W. Curtis Preston: no, I haven't tested yet.
It's kind of a waste to test this early.
You know, I've got tests.
I'll, I'll wait a couple of days.
Um, and then test and I'm probably fine.
I probably don't have it.
And if I do have it, um, I will have a mild case, but, uh, like the
one I had back in December, but.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: I don't need to go to a large unmasked gathering
of people right now, which is I don't want to be typhoid Mary.
Prasanna Malaiyandi: Yes.
I'm glad you're taking the initiative though.
Not to go because I'm sure there are a lot of people in your position
or you'd be like, yeah, it's fine.
I'll just go.
W. Curtis Preston: Yeah.
Yeah.
I'm uh, well, Jeff's wife, um, actually got on a plane before he found out.
So, you know, obviously she spent lots of time with him and then she
got on a plane this morning to go visit her relatives in Florida.
And, you know, he notified her like in the air.
So he's like, well, I guess you're going to be masking up
when you're around your family.
So, you know, it stinks cause you know, and the thing is,
again, I'll be fine if I get it.
Um, but I mean, due to both the vaccine and also,
Prasanna Malaiyandi: Having had it back in December
W. Curtis Preston: Yeah, I've had it and I should be fine, but I don't
want to be the person who gives it to somebody else where it might have,
you know, much more negative effects.
So I'm going to take the precautionary stuff and it stinks.
I literally found out like, as my daughter was stopping by the house
and I'm like, yeah, by the way, I'm not going to come say hi to you.
Cause, cause I got, you know,
Prasanna Malaiyandi: Oh, well, be responsible.
Be good.
W. Curtis Preston: Be be responsible.
Nope.
No, that's not.
No, that doesn't make a good cheer.
Prasanna Malaiyandi: No,
W. Curtis Preston: No.
Well, uh, I threw out the Druva name, saw throw that out again,
that a, this is a, that Prasanna and I worked for different companies.
He works for Zoom.
I work for Druva that's opening up a new.
You know, corporate office that I won't be visiting, but this is
not a podcast of either company.
The opinions that you hear are ours, and you're going to get some opinions.
This episode, I'll tell you, you will get some opinions,
some very strong ones from me.
And then, uh, also, uh, please rate this podcast at ratethispodcast.com/restore.
If you're listening to us on iTunes, just scroll down to the bottom,
click some stars, make us a comment.
Do all the things, you know, it just helps us to, to get the word out.
And, um, and then also if you're interested in the kind of things we're
interested, we want to have you on just reach out to me at what's that.
Prasanna Malaiyandi: Come on the podcast.
W. Curtis Preston: Come on the podcast.
We love new people, you know, um, even if they say mean things like the one that,
that we had last week, the one that got published today, the backup is evil.
That was an interesting, uh, the first few minutes were rough.
Uh, cause cause he was really just slamming down on backup,
but, uh, that was a good episode.
The backup is evil episode is a good one.
Prasanna Malaiyandi: you don't have to always agree with us.
It's totally fine.
W. Curtis Preston: and you know, it would be nice to you, even if you think
that, uh, for example, I'm going to make some really emphatic emphatical
emphatic, emphatic, emphatic statements.
This, this podcast, if you think that one or more of them are wrong,
come on, we'll talk to you, right.
We'll be nice.
We promise.
It's virtual.
It's not like we can, uh, will Smith you, um,
Prasanna Malaiyandi: Oh, uh, might be too soon.
Curtis might be too soon.
W. Curtis Preston: Well, yeah, I, you know, I went the thing with Jeff.
We went to the academy museum again and saw a showing of the French connection
introduced by the director of the French connection, which was amazing.
He's 87 years old.
It was an amazing thing.
I don't have time to go into it, but I will say that he opened the thing
by saying it's so great to be here.
At the, uh, academy of motion pictures, a place known for, uh, will Smith.
Prasanna Malaiyandi: Uh,
W. Curtis Preston: He's like too soon?
Too soon?
The sense of humor and, and cognitive abilities of this 87 year old guy.
They were great.
It was just awesome.
Anyway, so I, you know, the, I was inspired by the backup is evil episode,
and I wanted to have a little bit another episode that delves into some
of the things that we covered there, but, but just, um, because he was,
he was very much heading towards.
Obviously obvi, uh, the NetApp way of doing things, which is block
level replication, incremental, incremental block level forever,
and replication not backup.
And let me just take a, take a moment to define what's different between
backup and replication and for me and I don't, and I don't mean, I don't mean
like snapshots and replication, what.
Prasanna Malaiyandi: know, what's going on in my head.
Remember you sent me, uh, the Reddit.
A meme about with, uh, Chris rock and will Smith, where it was
like, snapshots is not backup.
And there was,
W. Curtis Preston: Yeah.
That one.
Yeah,
Prasanna Malaiyandi: that's what that's what was going on in my head.
W. Curtis Preston: that's funny.
It's not so much snapshots versus backup.
It's replication versus backup.
One big difference between what he was recommending.
So he, he made this point of saying, Um, you know, the whole world based
basically he, if he had his druthers, the entire world would move all
their storage onto NetApp storage.
And then you get basically backup and replication, storage, and primary and
secondary and all that all in one thing.
And he, but he acknowledged that that's only roughly 15% of the world.
So I'm saying, all right, let's talk to the rest of the world.
You're not going to move everything to, to, to net app or you're not, you,
you're not even going to use if you've got a NetApp like architecture and you
could be using snapshots and replication, but you don't, you haven't been able
to convince your boss to do that.
You're going to use backup instead.
So I just want to, before we discuss backup, I just want to point my hands
are really going to really going today before we discuss back up.
I just wanted to say.
What's the difference between a backup and what that does because I would, I would I,
if you replicate snapshots to a secondary array, I would call that a backup.
Some would not, or at least it's a, it's an element of backup, but what I'm
differentiating here is when you make a copy into some other format, right?
It's not all within.
You know, uh, and typically when we say backup, although I do not, I am not
this, um, whatever the opposite of all encompassing, I'm not this pedantic.
If you will,
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: I'm not just talking about tape backup.
I'm not just talking about traditional.
Um, like let's say Veritas NetBackup or, or Dell networker, that style.
CommVault, it's not just that.
I would also consider, you know, our competitors like Rubrik, Cohesity,
Veeam, Clumio, these are all backup.
Um, so, but what I thought we would do is talk about things that we
should absolutely stop doing, right.
That if you're still doing these things, you should really
seriously consider a change.
That was quite the build up quite a, quite a rant.
And I haven't even said anything yet.
What do you think?
Prasanna Malaiyandi: no, I think so.
I just wanted two points of clarification from you.
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: Um, the first is, we're not saying that replication
is evil or snapshots is a place in your environment, depending on your
needs and objectives to have those mechanisms, but it does not truly
replace backup and the sorts of things
that, protects against.
W. Curtis Preston: well, that's a different discussion,
Prasanna Malaiyandi: Yeah, but I just want to be clear though, that
W. Curtis Preston: right?
That's not, that's not the point I'm trying to make.
I'm just trying to say you're using a backup product.
And I want to talk about some of the same concepts that he talked about.
But in the backup context, cause he was living in this world where you have
assued backup and in favor of net, you know, I'm not saying I don't even want
to have the discussion on, I mean, maybe we will on a different episode on
what's the difference between, I just wanted to differentiate between the two.
I'm not saying one is bad or better.
I'm just saying your you're listening to this episode because
you're using a backup product.
And I want to talk about things that you shouldn't be doing anymore.
Prasanna Malaiyandi: Okay.
I'm good with that.
W. Curtis Preston: So you really only asked one question.
You had like three.
Prasanna Malaiyandi: well, I had two.
My second question and maybe we might get to it when we're talking.
You also mentioned that you made the point about it should be in a different format,
W. Curtis Preston: Yes.
Yes.
Prasanna Malaiyandi: And
W. Curtis Preston: In order to fall into this definition, that's all I'm saying.
I'm not, I'm not making.
Well, there is a judgment call.
That is a different discussion.
I'm just trying to just delineate it.
Prasanna Malaiyandi: Yeah.
The only reason I was asking is typically a lot of backup products
that might be doing native format backups don't really modify the format.
And
W. Curtis Preston: don't modify it, but they store it in a different way.
Prasanna Malaiyandi: Okay.
Okay.
So It's but
W. Curtis Preston: not just a replicated, like, again, it's easier to say what,
what, what isn't so NetApp SnapMirror, you know, snapshots and SnapMirror that
doesn't fit this definition because it's the exact same thing on both sides,
Prasanna Malaiyandi: but if someone was scp from one Linux box to
another Linux box and copying the data that would be considered
W. Curtis Preston: That would be something different.
Generally, it will change formats, but not always.
Right.
I know for example, Veeam tends to store data in a way.
And actually I think Rubrik and Cohesity do as well.
They store data in a way that it can be accessed.
But, but it's just about, again, it's about changing the manner
in which the data is stored.
Prasanna Malaiyandi: Okay.
I just looking
W. Curtis Preston: don't have a better way.
Prasanna Malaiyandi: No, that's okay.
W. Curtis Preston: Yeah.
Okay.
So,
Prasanna Malaiyandi: should we be stopped doing
W. Curtis Preston: well, the first thing I'm going to talk about is tape,
and I'm sorry to my tape friends.
Prasanna Malaiyandi: But you're Mr.
Back up?
W. Curtis Preston: Yeah, but see backup doesn't mean tape.
It has for many, many years.
And, and, and, and it's sure meant that to, to, to Ricky on that episode,
but backup doesn't mean tape to me.
And it hasn't for a long time.
I'm not saying do away with tape, but here's what, here's
what we have to do away with.
And w, and again, everything I say has exceptions.
Right.
Everything I say has exceptions.
There are reasons why you might want to continue doing things, but as
a general rule, I don't think that most companies should be backing up
from a server directly to a tape.
Right.
That design hasn't been a good design for 15 years.
Maybe more it's because of that problem that we've talked about before, it's
the tape speed mismatch problem.
The tape wants to go too fast and you want to go, your backup wants to
go really slow and the only way to.
To address that speed mismatch is to do massive levels of
multiplexing, where you're, where you're putting data together.
Um, in order to, to generate a stream of data fast enough to
keep that tape drive happy.
Right.
And we'll talk about multiplexing in a bit.
Prasanna Malaiyandi: Yeah, it's funny as you're saying
that we should stop doing tape.
I was just going back and thinking how many episodes on this podcast
have we actually talked about tape?
And I agree with you though, right?
That there is a use case per tape.
And there is improvements in the technology, but just like you
said, using tape in, in general using tape as your first copy
W. Curtis Preston: As your initial copy is very problematic.
We can have a different discussion as to whether or not it makes a good
secondary copy or a doomsday copy.
Right.
So, Brian, right, Brian came on and Brian made a good.
Case Brian Greenberg and his, his coworker whose name I forgot, but
I've known what's that Cameron.
Thank you.
They, they, they made a good case for tape for, for a copy on tape.
I'm not saying tape is bad.
I'm just saying tape is really, really bad at receiving incremental
backups unless you've spooled them all to some fast disk first.
And then you're just copying all those backups over.
That's what I mean.
Prasanna Malaiyandi: can I.
And I know this will probably fall under that in general, but
here might be an exception case.
Um, would it change?
So if, depending on what you're backing up, for instance, if you're backing
up object store which not many people probably do, but maybe in that instance,
would it make sense to go to tape?
W. Curtis Preston: All of my recommendation is based on.
How lousy incremental backups are to tape.
If you have a scenario where you don't have this problem, I don't care.
I don't have a problem with the reliability of tape.
I have a problem with it's in compatibility with the
way incremental backups run.
Okay.
Prasanna Malaiyandi: Gotcha.
W. Curtis Preston: I I'm even okay.
Maybe for using them for full backups.
If, if you've got a scenario where you can make that happen, many environments,
they just can't even do full backups or just, you know, they've got a 10
exabyte storage system, how are you going to do a full backup on that?
Right?
You you're just not.
And so, but if you can, if you can do a full backup to tape, that's
possibly a way to use tape in your environment, but generally speaking,
but then it complicates things.
We're going to do full backups to tape, but incrementals to desk.
And we're, you
know,
Prasanna Malaiyandi: Managing.
W. Curtis Preston: I'd much rather you just simplify things
and use, use disk for everything.
Um, uh, although we're gonna, we're going to come back to disk as well.
Prasanna Malaiyandi: Yeah.
So.
since you touched on tape and continuing along that train
of thought, you brought up
multiplexing.
W. Curtis Preston: Yeah, multiplexing.
Prasanna Malaiyandi: Yeah.
And I know in the previous tape conversations, right.
It's kind of the only way to actually keep tape happy, but like, you've like, we've
also talked about in previous podcasts, It's great for doing backups, but when
you have to do restores, that's where things just go really, really, really
bad.
W. Curtis Preston: Yeah, because basically you have to read the whole
tape and throw away 90% of it, or, or maybe even 95 or more percent of it.
Now I know of at least one product, their name is escaping me.
It might be, no, I don't want to say that.
I don't want to say the name.
See, you heard that notification share the notification.
I have do not.
Oh, you, you probably did because you didn't because I got a unidirectional mic,
but slack just gave me a notification.
So what I have do not disturb for everything on.
So why is slacking beeping on my computer anyway?
I'm sorry.
I digress, but that's what happens.
Cause I got a squirrel.
What were we talking about?
Prasanna Malaiyandi: Multiplexing
W. Curtis Preston: Oh yeah.
Prasanna Malaiyandi: there's
a tape vendor.
W. Curtis Preston: Multiplexing before we talk about multiplexer.
Let me just round out about tape again.
I think tape there's an argument to be made for using it as a secondary
copy as a way for creating a backup to send off site mostly in smaller
environments though, because when they get big using tape as a DR mechanism,
I think is really problematic,
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: um, and also using it for long-term storage, there's a
really, really good argument there.
Prasanna Malaiyandi: Yep.
And archive is not backup.
W. Curtis Preston: Correct.
Is it, did you have another comment you want to make?
Prasanna Malaiyandi: No, I was just going to talk about, yeah.
Tape makes perfect sense in an archiving sense as
well.
W. Curtis Preston: Yeah.
Active
Prasanna Malaiyandi: different than
backup.
Yep.
W. Curtis Preston: Absolutely.
Um, and so, yeah, so let's go back to multiplexing .Multiplexing.
It's a bit like, have you ever heard that?
Um, I dunno if this is one of those like fake quotes, but uh, I think.
Th my memory is that it's Winston Churchill that allegedly said democracy
is the worst form of government.
It's just better than the other ones.
Um, something like that.
And multiplexing is the worst backup technology ever.
It just was really helpful in solving a particular problem.
Right?
It was, it was a, it was a necessary evil because tape drives were getting
faster and faster, faster, and we didn't, we didn't yet have multiplexing.
I'm sorry.
We didn't yet have deduplication,
which, which is what allowed us to use disk.
So stop using tape and multiplexing and use disk and deduplication.
And you know, that solves that problem.
Prasanna Malaiyandi: It solves on problem
W. Curtis Preston: solves one problem.
Yeah.
We're gonna, yeah.
disk creates another problem.
We'll get to that in a minute.
Prasanna Malaiyandi: But here's
W. Curtis Preston: The next
thing,
Prasanna Malaiyandi: before we jump.
So even with the multiplexing and tape backup, are there any products other
than tape, are there other displays products that use multiplexing?
W. Curtis Preston: No, because there's absolutely no reason
to use multiplexing on desk.
And by the way, if, if I'm wrong, uh, then that's the dumbest product
in the history of dumb products.
Right?
The, oh, what I was saying before I got interrupted by the slack thing is
there is at least one backup product that does multiplexing differently.
It sort of spools.
I don't know if it does it in Ram or if it does it in flash or disk or
whatever, but it does multiplexing in such a way that they put big
segments on the tape contiguously.
And so they're able to read, seek, read, seek rather than what most backup
products do, which is read everything and just throw away everything.
It doesn't need.
There's that, but just tape and multiplexing, that's just gotta stop.
Like if you're doing tape and multiplexing, you really have to stop.
It is a horrible, horrible way.
And if you're like, well, how am I going to make my tape drives happy?
Stop using them.
That's how right.
Use them for secondary copies.
Use them to create a copy to hand, to hand to iron mountain.
All right.
So the next
one.
Prasanna Malaiyandi: Have you caught in that all out of your system?
W. Curtis Preston: Yeah, the tape thing, the next is, and this is
what we talked a lot about with Ricky and that is full backups.
And by that, I mean, repeated full backups, occasional, regular, full
backups once a week, once a month, once a quarter, once a year, I don't
care how often you're doing them.
They're dumb.
And that's where Ricky and I, a hundred percent agree the idea
of moving everything from a to B.
'cause that's why we've always done it is the dumbest idea.
Again, it was a great idea back in the day, because we were using tape
and if you're using tape, you do need to do the occasional, full backup or
else the restore can take forever.
And you know, there are backup products that have this problem.
I won't name them specifically, although Ricky didn't have a problem.
Prasanna Malaiyandi: So I agree with you because it doesn't make sense,
especially when you also have deduplicated disk where you're going to throw away
90% of it, the full backup anyway.
Right.
Why even occupy the additional network, right.
That you need in order to be able to move all the data, plus all
of the compute and the storage
W. Curtis Preston: well, the compute and everything on, on the, on the,
on the side that you're doing it.
Yeah.
Prasanna Malaiyandi: Yep.
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: And, but I do have a question
whether it's.
W. Curtis Preston: You know that whenever you say a bunch of stuff, and
then you say, but you just dismissed everything you said before the but, right?
Prasanna Malaiyandi: So I totally agree with, we should never do full
backups,
W. Curtis Preston: Okay.
So
you have a concern, you know, you pause, you just pause, you agree.
And then you say, I have a concern.
So you, that way you don't negate everything that you said before.
Prasanna Malaiyandi: I have a
W. Curtis Preston: I'm really good at arguing.
I'm just saying, this is, this is a skill that I picked up when you
say, blah, blah, blah, blah, blah.
But, or however you just dismissed everything that you just said.
So now I've learned to say, I agree with what you said about tape, not
or about not doing full backups, the concern I have C you on that
Prasanna Malaiyandi: Yeah, much better.
Yeah.
W. Curtis Preston: I'm teaching you how to argue.
Prasanna Malaiyandi: The concern I have is for.
Applications and certain workloads.
I think you can't get away from never doing full backups.
W. Curtis Preston: That is a valid statement.
In fact, the product that Ricky mentioned, they only do forever
incrementals on file systems.
They don't do.
Forever incremental on databases.
If your database basically doesn't know how to not do a full backup, you
have to do the occasional full backup.
There are ways around.
So like for example, the most popular of those databases would be Oracle and
there is a way to do forever incremental with Oracle and hardly anybody uses it.
Prasanna Malaiyandi: Yep.
W. Curtis Preston: The idea of the image copy and,
Prasanna Malaiyandi: Oracle incremental merge.
W. Curtis Preston: Right, Oracle incremental margin.
We should cover that on podcasts, but that on the list,
we'll cover that on a podcast.
I think it's a great way to do you.
You end up with a copy of Oracle.
You end up with multiple copies of Oracle on disc, that multiple versions, if you
will, of Oracle on disk and each of them only takes up the incremental new blocks.
So it's a really good way to do incremental forever.
You don't like incremental forever with what don't you
Prasanna Malaiyandi: I don't, there are some downsides to using Oracle incremental
merge, especially when it comes to integrating with other applications.
Um, because when it comes to integrating with your native backup
applications, uh, because it's not fully supporting of the streaming.
Transferred the SBT library that Oracle has, right.
It's completely managed separately and all the rest.
So it makes it a little bit more complex to manage.
I agree with you that there are a lot of great use cases with Oracle incremental
merge and doing backups in that format and being able to just, get the incrementals.
Um, and then there is also a second downside of Oracle incremental
merge, which we can talk about in a separate podcast, which is you
actually have to read the data back in order to apply the incremental
W. Curtis Preston: right.
Prasanna Malaiyandi: But we can
say that for another
W. Curtis Preston: there are, yeah, there are exceptions, dare
exceptions, just like everything.
There are exceptions to the rule and obviously if your application forces you
to do full, so then you got to do fulls.
What I'm talking about is using a backup product that it
doesn't know how to not do it.
Prasanna Malaiyandi: Like VMs, right?
VMs.
You should never be doing full backup daily.
Right?
That's ridiculous.
W. Curtis Preston: Right.
And if the, and there are some products that say, well, we don't force a full,
we just do a sink, a synthetic full, honestly, that's not much better.
It's it's a little better, but it's the same thing.
Your backup product is not yet in the 21st century, your backup product
still thinks it needs a full backup.
And so you're going to create a full backup and by the way, synthetic
fulls have limitations themselves.
Right?
I don't want to go down that route, but maybe we add that to the list.
You know, we should talk about synthetic foals versus real fulls.
But, yeah, I'm just saying in general, if you're still using a backup product
that forces you to do an occasional full, however, often that occasional
and whether or not it's synthetic, I'm just saying that should just stop full
backups unless when you're forced to do so, should be a thing of the past.
It's all I'm saying.
Prasanna Malaiyandi: Oh,
W. Curtis Preston: And re huh.
Prasanna Malaiyandi: I agree with you.
W. Curtis Preston: Okay right after that.
And again, unless you don't have any other choice
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: is full file, incremental backups.
What do I mean by that?
There is, there are so many products that, well, there are products that.
Can do block level incremental backup on those files.
The idea of backing up an entire file, just because one byte of that
file has changed is another dumb idea whose time should go away.
And if your backup product doesn't know how to do block level
incremental backups or source side deduplication, so that you're, you
know, you're doing it that way.
And by the way, replication is a way to do it.
So if your product doesn't know how to do that, and you're still backing up the,
it's an order of magnitude difference in the size of an incremental backup.
You're looking at 10 to 15% versus 1% of the, of the size.
Like just, I mean, that's a total made up number, but it's based on my experience.
Okay.
Um, what do you think about that one full file incrementals?
Prasanna Malaiyandi: And I think just to put more context for users.
Don't think of the case where you have a word document and a byte changes.
Right?
Think about also large files that might be in your system, that you are constantly
backing up day after, day after day.
Right.
And if those, if you make a small change to it, that causes the
entire file to be backed up again,
right?
W. Curtis Preston: PST files coming to mind.
Prasanna Malaiyandi: Yep.
W. Curtis Preston: And similar files like that large files that
get a little bit of change each day.
And if you have a full file incremental, backup you back up the entire thing.
Prasanna Malaiyandi: Yep.
And I agree with you.
There's no reason that you should be copying that entire file and backing
up that entire file every single time.
W. Curtis Preston: Yeah.
All right.
Let's the next, one's going to harken back to the earlier discussion
because I was like, I want you to stop storing your backups on tape.
Now.
I want you to stop storing your backups on disk by that.
I mean,
Prasanna Malaiyandi: Turn is Curtis.
W. Curtis Preston: I mean, Directly accessible disc, an SMB
Mount, or an NFS Mount, a windows server with some DAS, right.
That's direct attached storage.
So just regular old JBOD or a disk array or whatever.
An NFS mounted data domain or quantum or whatever, whatever box you're using.
If it's, if the backups are directly accessible in the operating system
of the backup server, this is what I, what we have to stop.
And, and why is that?
Prasanna Malaiyandi: well, because it now becomes easily accessible for
ransomware and other people to go blow away, encrypt your backups, and now
you can't restore your environment.
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: Just one more point on that.
It's uh, shoot.
I lost my train of thought.
Oh, you said backup server.
I think you could also be from any client as well accessing that server.
W. Curtis Preston: Well, any, any server that has direct
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: read-write access to the backups, you are right?
That there are some backup configurations where the backup client writes directly
to the storage, not via like a protocol.
Well, unless we're talking about NFS or SMB is the protocol.
If there, if they can, if they have read, write access directly
to the backups and yes, that's exactly what I'm talking about.
Prasanna Malaiyandi: Especially with all these network protocols,
most backup vendors, right.
Have a proprietary protocol or a proprietary connection for actually
writing the data to the storage.
So you really shouldn't be using open protocols.
Or I want to, I wonder if we could even extend it and say, even if you are using
something that is open, you probably want a really, really, really good way
to secure it and limit the attack scope.
It's not ideal.
W. Curtis Preston: that's a great point.
And because.
If the backup server, I'm sorry.
If the backup software administrative accounts are compromised, it doesn't
matter how you wrote the backups on that disc, whether you used a protocol or not,
you can go in and delete those backups.
Prasanna Malaiyandi: Yep.
W. Curtis Preston: However, the exception to that would be if you've
written it to storage, that is truly.
Prasanna Malaiyandi: Immutable
W. Curtis Preston: Immutable.
So if you, uh, if you're unable to delete the backups, even if you have
administrative access, then that would, that would solve this problem.
Right.
But this is, this is the thing you have to look at now, look at the
attack surface that you have, look at how you're writing backups, find
out if there is a more secure way to store your disk-based backups.
In most cases, the answer is almost always yes.
Prasanna Malaiyandi: Yeah.
And so this goes back to one of those exception cases, right?
So you have to do your due diligence to ensure that everything is protected
and secured in the right way before sort of saying, yeah, I'm just going
to write all my backups via NFS or SMB.
W. Curtis Preston: Um, and, and.
I don't have a problem with like using disk as sort of a caching mechanism.
Right.
Especially if you're using tape.
So it's not that writing backups to directly accessible disk
is necessarily an evil thing.
It's just that if that's the only way you're writing backups, it's
a really, really insecure way.
If it's one of your many copies, I don't have a problem.
But, uh, if, uh, unless you have another option, if you have another
option, a more secure way than use that way, that's all I'm saying.
Prasanna Malaiyandi: And there are also other benefits for some of these
protocols, because it allows things like source-side deduplication and helping
you efficient or save bandwidth, right.
By not sending all the data and getting tossed on the storage side.
W. Curtis Preston: And, and, uh, you know, I'll, I'll put this out.
That yes.
The way that Druva stores, the backups meets this definition, meaning that
it never stores the backups in a, in a way that, you know, we do have a cloud
cache that you can optionally use.
It's not required, but you can, in that case, you may have a
local copy of your data, um, that is also stored to the cloud, but
the main working reference copy.
If you will.
The, the copy of record is stored in the cloud and it's stored in object
storage and is stored in an account that you don't have any access to it's as
it's as removed as it could be from you without actually putting it on a tape
and then handing it to a man in a van.
So we're not the only ones that do that.
We're not, and that's not the only way to do that, but I'm just saying.
That is a way to solve this problem is to just not use disk.
Not use regular disk, uh, regular backup servers, uh, at all.
Prasanna Malaiyandi: You know, you should get t-shirts printed with that,
W. Curtis Preston: What
Prasanna Malaiyandi: not the man in the van
or way.
W. Curtis Preston: not the man in the van.
I like it.
Um, so the last thing I wanted to talk about, and this is
another age old practice.
Is an, and it's an age old practice with even more experienced than, than some of
the other ones that we've talked about.
And that is this idea of waiting for the disaster or the attack
to start your disaster recovery.
And by that, I mean, when I think back to my early days, RDR our DR as I.
As I make quotes, DR plan was a box and a tape and a set of
instructions, and we never had to fire it in anger, as I say, right.
We never had to actually use it because we got a flood or a
hurricane or a terrorist attack, or obviously not a ransomware attack.
But I know that if we had actually done that, we would have been down
for ages because I know that it took us an entire weekend to restore the,
the, just the one or two servers that we would do on a, on a DR test.
And so what I'm saying is that it used to be that.
Only the high end companies, only the financial trading firms or whatever,
where they could put downtime measured in millions of dollars an hour.
Right.
We lose a million dollars an hour.
If we're down.
Those were the folks that had the super HA stuff and the hot, the hot standby site
and all that stuff that we used to have.
What I'm saying is that's no longer the case.
It's no, there are too many options for you to be able to have a disaster recovery
plan that is ransomware friendly so that you should be able to restore your data.
Uh, and have it essentially restored for you ready to go in a disaster.
That's what I'm saying.
If you're just waiting for the restore to happen, you say, then we're
going to do our disaster recovery.
What I'm saying is don't do it that way.
I'm saying use one of the methods that allows you to restore the
data before it's ever needed.
You, you have this hot standby copy ready to go, and this will
definitely be in the cloud.
That's the whole point of this is that.
Because of the cloud, you don't have to pay for the compute infrastructure
until you need to actually test it.
All you have to pay for is the backup copy.
That's essentially ready to go.
Not, not sitting there in backup format or anything like that.
It needs to be ready to go.
And, um, that's what I'm talking about.
Prasanna Malaiyandi: And I think because of the cloud and other
technologies that have come out, it's now become a lot more affordable
before, like you said, there weren't any options, unless you had millions
and millions of dollars in your budget.
Now you can get a DR copy.
That's good enough to deal with a lot of the ransomware
scenarios and other things, right.
That is affordable for most organizations.
So go take a look at things out there.
W. Curtis Preston: Absolutely.
And I'll, I'll throw out a few random things.
We don't need to cover them.
I mean, these are more like ma maybe, maybe, maybe.
Well, I'm going to come back, run back to you.
Maybe we'll do an episode.
Let's do an episode things you better be doing.
Prasanna Malaiyandi: Oh,
W. Curtis Preston: These are things you should definitely no longer be doing.
We'll talk about things that you better be doing.
Uh, one of them being, um, MFA.
Prasanna Malaiyandi: yeah, I have one for you should stop doing this.
W. Curtis Preston: Okay.
Prasanna Malaiyandi: Um, you should stop running your backup apps
as root and your agents as root.
W. Curtis Preston: So you mean administering them as root?
Prasanna Malaiyandi: At both administering or even having agents
being run as root in your environment,
W. Curtis Preston: Well, my only concern of that is like in a Unix environment,
it has to run as UID zero in order to get the, the power that it needs or the
Prasanna Malaiyandi: but not always, not all ways as
W. Curtis Preston: What would you do?
Other than that, it runs a system or root, like in order to get the
permissions, to be able to see everything in a Unix world in a windows world
has to run is the equivalent of Mr.
Administrator.
And what I'm saying is you could make another user ID, but in the
Unix world, it would still have UID.
Prasanna Malaiyandi: yeah.
W. Curtis Preston: And so, but, and so I'm, I agree with your recommendation.
I just want to just slightly, uh, massage it to mean if you are logging in as
root or something, equivalent to root or administrator, and you're running your
backup, you're running your backup system.
That way that is absolutely wrong.
Prasanna Malaiyandi: So I would also say though, in the case where you
don't need to be running your agent as root, that it should not be running
as root because not all cases need
W. Curtis Preston: Agreed.
I don't know what those, I don't know what those cases are.
Give me an example of a case.
Prasanna Malaiyandi: for database backups, you don't need to be running as root.
In
W. Curtis Preston: I, I, I made that up in my head, as I was saying,
I was like, oh, he's probably talking about database backup.
Yes.
If the agent doesn't have to run as root then correct, don't run it as root
Prasanna Malaiyandi: Yup.
W. Curtis Preston: Yeah, apply the concept of least privilege to your backups.
Prasanna Malaiyandi: Yup.
W. Curtis Preston: And we should put that, you know, MFA, least privilege.
We'll talk about those in are things you should be doing episode.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: do you think we've yelled at people enough?
Prasanna Malaiyandi: I think so
W. Curtis Preston: This is a very angry episode.
Stop doing this.
Prasanna Malaiyandi: well, it would be interesting to see how many people
are using tools that actually do this,
or
W. Curtis Preston: It would be interesting.
Well, I would say still the bulk of the backup world is doing, using backup
products that still do full backups.
Prasanna Malaiyandi: Yep.
W. Curtis Preston: That still force an occasional full backup,
even if it's a synthetic one.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: Think about it, right.
Net backup, networker.
Commvault.
Not TSM, uh, ArcServe, I mean, you know, pick, pick a backup product
that's been around for 20 years or more, it's doing occasional full backups.
Prasanna Malaiyandi: Yep.
W. Curtis Preston: It's only backup products that have been designed in
the last 10 years or so, which would include Veeam would include, actually
seems a little older than that, but, but the rubric Cohesity Druva.
Trying to think Clumio , these are products that, that they're
like, basically they're the, they're the disk generation.
They're, they're making an assumption.
We're going to store backups on disk or something behaving like disc.
And so therefore the idea of a full backup is just dumb.
Prasanna Malaiyandi: bonkers.
W. Curtis Preston: And so let's not do it.
Let's let's design, let's design a backup product from the beginning
to not need a full backup.
A repeated full backup.
Obviously we always
always need
Prasanna Malaiyandi: At least
W. Curtis Preston: backup,
just one, one, and done baby one and done just like, no, I was just going
to try to make a go go North Carolina.
You don't even know.
Cause you don't follow NCAA.
You follow, you don't really follow basketball.
You follow football.
Prasanna Malaiyandi: Nope.
I just follow football.
the NFL.
W. Curtis Preston: Yeah, North Carolina won over Duke this past weekend.
Prasanna Malaiyandi: It was an upset
W. Curtis Preston: It was an upset.
And you know, and the guy, the coach that, you know, the coach ended his
career on a loss, which is a shame, but he's had an amazing career.
So good for him.
Well, Prasanna, thanks for letting me rant.
Prasanna Malaiyandi: you feeling better.
W. Curtis Preston: Well, I'm still going to be hanging out in this office
for the next.
Prasanna Malaiyandi: but are you at least feeling better?
W. Curtis Preston: Yeah, I'm feeling better.
Uh, you know, I don't know if I, I thought I made a, I don't know if you've ever
been in a house where somebody had lice,
you've been in a house.
Prasanna Malaiyandi: Yep.
W. Curtis Preston: Did you start like psychosomatically
feeling lice in your hair.
Yeah.
So that's where I'm like, I mean, if, if I got infected, I I'm not
going to have symptoms yet, but still my body is like, I think is that
a fever said, you know, said so.
Yeah, good times.
Prasanna Malaiyandi: lots of tea.
W. Curtis Preston: Drink, lots of tea.
Um, not, not whiskey, although I don't know why I said whiskey.
I don't drink whiskey.
Prasanna Malaiyandi: This
W. Curtis Preston: I'm not a whiskey bourbon or scotch person or cigars.
Like my friend, Jeff.
See you do cigars.
You get COVID.
That's it.
That's that's the lesson that
Jeff should learn.
Prasanna Malaiyandi: Curtis and I are not medical doctors.
We are not providing any medical advice.
Please go talk to your doctor or do your research.
Thank you.
W. Curtis Preston: please don't get your medical advice on
Backup Central's Restore it All
we, that is not, it's
Prasanna Malaiyandi: at least.
from the two of us.
W. Curtis Preston: at least.
Yeah.
Right.
We got to have Lindsey back on and I'm just waiting for like, I don't know.
I kinda just want to bring her back on and say, is it over?
Can we, can we move on now?
That's that's the episode I want to have Lindsay back on for.
And if you guys don't know what I'm talking about, we did half a
dozen episodes in the early days of the pandemic with, uh, Dr.
Lindsey Schultz.
And she, she actually is a doctor medical doctor with a, with a, a
master's in public health as well.
And she specializes in harms reduction.
It's a kind of, kind of logic that you use when.
Wanting to say, if we're going to make this public policy.
And, um, anyway, so yeah, I want to have her back on, but I want
to, I just, I want to have her on when it's like just good news.
Yes, we're done.
We can move on with our life.
No more.
COVID scares like the one I'm in right now.
Damn it anyway.
All right.
I'm going to end this thing before, before I, before I start to despair.
So thank you Prasanna, you know, like, like I said, for letting me rant
Prasanna Malaiyandi: Anytime Curtis, and I hope you feel better.
W. Curtis Preston: and, uh, thank you to the listeners and be sure to
subscribe so that you can restore it all.