W. Curtis Preston: Hi, and welcome to Backup Central's.
Restore it all podcast.
I'm your host, W.
Curtis Preston, AKA Mr.
Backup.
And I have with me my continuing advisor on my consumer backup project
Prasanna Malaiyandi, how's it going?
Prasanna.
Prasanna Malaiyandi: I'm good.
Curtis.
You know what?
We have the expert, the backup anorak Daniel Rosehill coming
next week on the podcast, so
W. Curtis Preston: yeah.
Prasanna Malaiyandi: can definitely pick his brains.
W. Curtis Preston: Yeah.
You know, it's been interesting cuz, and it was funny, it was on the
podcast where I really, I really, I had a moment where I was like,
I'm not really backing up I photos.
Right?
Like, I'm, I'm backing up, you know, I, I use iCloud, but as, as we have
discussed, iCloud is not a backup.
iCloud is a sync.
Right.
And if something catastrophic, if you, if I ever got hacked, um, and somebody
got a hold of my, my iCloud password or my iPhone and then just decided to
massively delete everything, I, if I caught it soon enough, I would be okay.
Cuz I do have like a deleted items thing.
Right.
And as we discussed, I have, well, I don't think we'd discuss on the pod,
but as part of this project I found out I have 11,000 photos in, in iCloud.
So,
Prasanna Malaiyandi: isn't as much as a lot of other people, you know.
I'm sure there are
W. Curtis Preston: know, I am not,
Prasanna Malaiyandi: you.
W. Curtis Preston: yeah.
As, as I, as you and I were talking earlier, I'm not, you
know, on one end, I'm not Cecil b Dilla and I'm not, you know,
photographing and filming everything.
On the other hand, I'm not Prasanna because you use your phone camera,
like you use your, uh, Tesla,
Prasanna Malaiyandi: Yeah, pretty much.
Yeah.
Almost never
W. Curtis Preston: Okay.
So you've had your Tesla, how long now
Prasanna Malaiyandi: four years,
W. Curtis Preston: and how many miles do you have on it?
Prasanna Malaiyandi: I think like 11,600, 700, something like that.
And it works great because my maintenance cost has been zero.
My electricity cost is really minimal versus gas,
W. Curtis Preston: Cars are incredibly reliable when you don't use them.
Um, anyway, yeah,
Prasanna Malaiyandi: powered cars, even if you don't use them,
you still gotta change the oil.
You still gotta do everything else, you know?
So,
W. Curtis Preston: I'm not, we're not doing a, we're not
doing an e an ecar thing.
But anyway, but yeah, we've, we've been having some fun with, uh, with this
project of figuring out the various ways.
Right.
Uh,
Prasanna Malaiyandi: and, and I think, yeah.
I was just gonna say, I think you should mention to the listeners what you're
current, what you are currently trying to do for backing up your iCloud photo
W. Curtis Preston: my current, uh, uh, uh, I don't know what,
I don't know what, what, what, I don't know these different methods.
My current method that I am trying is Google Photos, and it turns
out Google photos, it, it's the only one that I've found so far.
Uh, well, the only one that I've.
Well, there's maybe one other, which is iDrive, but Google Photos, cuz
the problem is that on an iPhone you can turn on optimized storage.
And so I have like, I don't know, somewhere between sixty
and a hundred gigabytes.
We're not quite sure of photos up in, uh, iCloud and, but I only have four
and a half gigabytes on my phone because it's, I'm using the optimized storage.
But apparently Google Cloud photo, Google Photos pulls down a high res
whatever high, the original version from.
iCloud and then backs
Prasanna Malaiyandi: that, that's our theory.
That's our theory.
W. Curtis Preston: That's the theory.
Well, it's, I, that's what it says in documentation.
We shall see what we shall see.
Um, and then we will report on the results here and then,
and, and I'll blog about it.
I'll backup Central, uh, because IO iCloud is not a backup.
The number of articles that I read that told me to use iCloud to
back up my iPhone pissed me off.
Right?
Like, it, it was like 95% of the articles that I found on how to back up, uh,
my photos basically said, OI cloud.
I'm like, ah.
Prasanna Malaiyandi: Because.
Because for most consumers, right, they're probably not going to do what
you're about to do, and they don't care.
And so turn on iCloud.
At least you have something else other than whatever's on your phone,
W. Curtis Preston: Yeah.
So we're gonna, we're gonna have an answer for the three people in the world,
all of which are probably already on this recording, the three people in the
world that actually care about having an actual backup of their, of their photos.
Anyway, all right.
Well, we're gonna bring back a, a longtime friend and a
returned guest to our podcast.
Uh, he is one of the few people in this industry that, um, make me feel young.
Uh, we welcome.
We welcome.
And he's also the, uh, the technologist, extraordinary and
plenty of potentially at Vast Data.
Welcome to the podcast, Howard Marks.
How's it going, Howard?
Howard Marks: I'm really happy to be here cuz you know you guys went on your
little podcast and you said something about using flash for backup being stupid
W. Curtis Preston: Yeah.
Yeah.
I might have
Prasanna Malaiyandi: But, but, but, but wait, I wanna clarify,
Howard, that was from Curtis.
I was the one who was like, yes.
W. Curtis Preston: Oh, he's
Prasanna Malaiyandi: What about Howard?
Oh,
W. Curtis Preston: throwing me right under the
Prasanna Malaiyandi: throw you under the,
W. Curtis Preston: So, we'll, we'll, we'll
Howard Marks: you don't have to explain that to me.
I've known Curtis 35 years.
W. Curtis Preston: that's that.
We will, we will, uh, we will, we'll get to that topic.
I will give you a chance to defend your, your, your Honor.
Um, why don't we start with an update.
It's been a while since we've had you on the pod.
Why don't we start with an update on, uh, how much more vast, vast
data is, uh, since we had you on.
Howard Marks: Well, you know, from the financial side, um, we announced at
the beginning of this year that we've hit a hundred million a year in annual
recurring revenue cuz we've organized ourselves as a software company even
though the experience customers have is look, appliances on my data center
call vast when something goes wrong.
Um, we arrange for customers to buy the hardware so that we are a software
company, makes life easier for us.
Um, the other big thing is that our friends at HPE
just made announcement of a product of theirs called GreenLake Files
that will be powered by our software.
So before today, if you wanted a scale out, expandable, low cost all flesh file
and object system, we would facilitate your buying hardware from the OEMs that we
deal with, and we'd sell you the software and you'd have a system that was running.
Now you can buy that from HPE as part of GreenLake, and that includes management
through the GreenLake Cloud front end.
So you can manage the GreenLake for files along with GreenLake for Block
and the Compute and all the other servers that are part of GreenLake.
So they've taken our software, married it to their control plane
and run it on their hardware.
Prasanna Malaiyandi: And is for, sorry, for those who may
not be familiar with GreenLake.
GreenLake is more of a, I don't know, a managed or a hosted environment done by
Howard Marks: It, it, it, it's an as a servicey.
So there, there are both consumption and CapEx models the way I understand it.
But you know, you don't log into the block array that and create a LUN.
You go to the cloud website and you create a LUN and their
control plane does that for you.
And so it's got more controls and you do don't have to keep the detail.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: So then you're, you're probably paying
for what you provision then,
Howard Marks: Uh, you can do that or you either way.
And, you know, that's, that's, you know, kind of an H p E finance question.
But my understanding is they do it either way.
Prasanna Malaiyandi: And I'm sure for vast data, right.
That's a huge win, being part of that offering.
Howard Marks: Well, it's, you know, first of all, it just gets hundreds and hundreds
of boots on the ground out selling our software and our whole concept of you
can do all flash for as cheaper, cheaper than other guys can do spinning disk.
And why do you want spinning discs as opposed to flash?
Other than that, they're cheaper.
You know, I can't think of another advantage.
Um, and so we, you know, for most workloads have narrowed
that down or reversed it, and so, All flash cheaper than disk.
What a great idea.
Um, so we've got, you know, a, all those HPE sales guys going out there selling it
as an HPE product, you know, it's not like Qumulo or Scality, where HPE was reselling
those products to run on HPE servers.
Prasanna Malaiyandi: Mm-hmm.
Howard Marks: Um, and you know, who you called for support was.
Well, is this a server problem or is this a software problem?
It's HPE GreenLake for files.
HPE takes the support calls.
It's a full HPE product.
Um, it's our software underneath it.
W. Curtis Preston: Yeah, it's interesting.
So you know, you talked about the, you, you know, you said you, you.
You, you have a r r.
So basically your customers are paying an annual fee to you based
on the size of their storage
Howard Marks: we, so, so we make, we make our money on a, what
we call a Gemini subscription.
That is, you know, in capacity units, we sell it at a hundred terabytes.
HP can sell it in different ways, um, and it's per year.
And we guarantee that we'll write that agreement for any piece
of hardware for 10 years at the
W. Curtis Preston: right, right.
right.
I remember
Howard Marks: Because, you know, it's not spinning disks.
They don't start failing a lot more often in year five and six.
And so if you want to keep it for 10 years, keep it for 10 years.
If you decide you want to replace some of your hardware in your five
or six, because the new denser or faster hardware is more attractive to
you, uh, but you bought seven years of support, we'll transfer it on the,
you know, terabyte per terabyte basis.
W. Curtis Preston: Right.
Gotcha.
Um, yeah, that's a pretty good deal for you.
And by the way, I'll, I'll, um, I, I was gonna, uh, compare it to something
else, but, but it, it made me remind me of our, uh, disclaimer Prasanna.
And I work for different companies.
I work for myself, he works for Zoom.
And, uh, these are our opinions, not theirs.
And, uh, be sure to rate us at, uh, your favorite podcast or give
us all the stars and comments.
It helps other people find us.
If you think we're amazing, then maybe other people will do so as well.
Uh, reach out to me, uh, @wcpreston on Twitter, or w Curtis Preston
at gmail, and, you know, to be part of the conversation.
And we'll see.
Um, you know, we'll get you on.
So your arrangement with HP reminds me of our arrangement with Dell.
Basically it's the whole boots on the ground thing.
Uh, you get to put your product in front of a whole, you know, giant number of
other people and it's great for you.
It's good for them.
Their customers get the benefit of your, uh, technology with, with the company
that they already, you know, know and
Howard Marks: And you know, and, and we all know that there are loyal H P E
customers who you know now it's a lot more likely they'll buy this product cuz
it's got that stamp of approval on it,
all of which works for us.
W. Curtis Preston: Yeah.
Yeah.
Well, congratulations on hitting a hundred million.
Um, wish you the best of luck on your way to the, you know,
doubling that and triple in that.
Um, last time you were on, we talked about, we, we alluded to, I think,
a little bit about how you do dedupe or dedupe-like stuff that's a little
different than the rest of the world.
And, and, and that it's better, you know, these are the, you know, the,
the, you're saying it's better.
So I, I want to give you a chance to talk about that, and then
Howard Marks: we, we guarantee it's, we guarantee it's better because I'm
a vendor and without a guarantee you shouldn't believe anything I say.
W. Curtis Preston: Okay.
All right.
That sounds good.
So how so how, so first off, how is it
better, and then why?
Howard Marks: It's better cause it reduces data further.
Um, And the why is how it works.
So, you know, at the beginning it's really pretty simple.
We do variable chunk deduplication with a variation of the rock soft method.
So if there are insertions, we re re re-sync relatively quickly and the
deduplication gets more effective.
W. Curtis Preston: Mm-hmm.
Howard Marks: Um, we do z standard compression on the data.
We then throw some data specific encryption algorithms at the data for
things like, oh look, it's numeric data.
Well that means it's only gonna vary within this range.
We'll store deltas.
And so whichever of those compression methods reduces this block of data most.
W. Curtis Preston: Mm-hmm.
Howard Marks: we run.
Um, because we're doing so the, the data path is writes go to storage class memory.
W. Curtis Preston: Mm-hmm.
Howard Marks: then get act and then all of this data reduction happens
as we migrate from that writebuffer to the capacity flash layer.
And since it's after the act, as long as we're draining that buffer fast
enough, l how long in time it takes to move any piece is irrelevant.
And so we have time to go, ah, let's try five different compression algorithms.
Use whichever one works best.
W. Curtis Preston: Interesting.
Yeah.
Prasanna Malaiyandi: do you do,
W. Curtis Preston: go ahead.
Go ahead.
Prasanna Malaiyandi: do you?
And that's actually very interesting how you can, like you said, by
storing it in memory, right.
You're not impacting client latencies at all.
Right.
For them it's like, Hey, right.
Went through.
And then you have this time to do the parallel, uh, computation.
Howard Marks: Yeah, just, just accept it.
It's storage class memory, so it's an S s D, so it's persistent and
there's no batteries and protection and you know, a panic when power goes
Prasanna Malaiyandi: Yeah.
Now, when you are running these algorithms, like I know AI and ML
is all the hot topic everywhere you look these days, right?
Are you guys doing anything around that in terms of trying to smartly detect
which compression algorithms based on
Howard Marks: We we're, we're not doing that in the data path right now.
You know, frankly, the running the five doesn't use that much
compute that it's worth it.
Um, we're using AI in our cloud platform, so if you have multiple
clusters, there's a cloud site you can go to and see one dashboard.
Um, and we're using it for the capacity projections.
So it's like, oh look, here's how much capacity you're gonna need
six months from now while you're filling out your budget request.
Let me tell you what you're gonna, there's AI behind that so that it
smooths things like, oh look, every three months they do a cleanup.
And so let me factor that the AI is good enough to factor that kind of thing
in, but not in the data path.
But let's get back to the data path.
Prasanna Malaiyandi: Yep.
W. Curtis Preston: yeah, your, um, your comment when you, you know, there
was a comment you were like, as long as we're clearing the buffer quick enough.
Um, and, and I would agree with you, um, you know, how, how do you ensure that
that happens, I guess is, is one question.
Howard Marks: Well, first of all, it becomes a parallelism issue.
So we have a large number of compute nodes, all of which are
draining this buffer in parallel.
And so when the buffer hits a high water mark, more threads
to D stage, it gets spawned and allocated across the parallel system.
Now, if there's a huge influx of writes, and you know, we're talking.
Tens of gigabytes per second for hours on the smallest system.
W. Curtis Preston: Mm-hmm.
Howard Marks: Um, then we'll start introducing latency into the
writes and apply back pressure.
W. Curtis Preston: Okay.
Okay, that makes
Howard Marks: But, but you know, that's, you know, literally,
you know, it, it don't
ha it, you know, the mechanism is there just in case,
W. Curtis Preston: right.
Howard Marks: happen.
Prasanna Malaiyandi: And when you, and because you have more than more
capacity, I guess, more throughput at the capacity level than at the storage
media level, is that why you can just increase the number of parallel
threads and you don't have to worry about the backend being a bottleneck?
Howard Marks: so in, in, so our, our building block, we call a D Box
or a data box, and it's got some.
S scm SSDs.
We started with Opta.
We now mostly use K oxia FL six, and then a larger number of capacity SSDs.
And they, you know, it's whatever the cheapest we can get or the cheapest that
our OEMs use is, um, the P C I E lanes.
Feeding the small number of S C m SSDs is generally the bottleneck.
Prasanna Malaiyandi: Okay.
Howard Marks: And so we can paralyze reading data out of s C m, the
writing to cut to the capacity.
We have a lot more capacity ssd, so there's plenty of bandwidth to write
Prasanna Malaiyandi: Gotcha.
W. Curtis Preston: So what, why'd you stop using OC Octane?
Howard Marks: Um, well first we decided just to get a second
source because it's a good idea.
Um, and then I, Intel
W. Curtis Preston: turned out to be a really good
Howard Marks: of, then, then Intel decided to get out of the business.
Um,
and, you know, we have supply agreements with Intel.
They still have a warehouse full of wafers.
Um, but it, you know, the, the performance advantage wasn't worth the complexity.
So we've chunked on these variable sized 32 K average blocks and we de-dupe them.
But in addition to running a strong hash.
To validate identical, we run a series of weaker hashes against the same
data blocks, and these weaker hashes are designed to generate the same
hash value for inputs across a narrow range of cryptographic distance.
So if two blocks have a sm, so cryptographic distance is the
number of bits you have to flip to turn block A into block B.
If block A is within X bits of block B, this hash will
generate the same hash value
W. Curtis Preston: Okay.
Howard Marks: from a data reduction point of view.
If two blocks generate the same hash value and are a small cryptographic
distance part, they have long common strings between them.
And will therefore re compress with the same compression dictionary.
So the first block that generates one of these similarity hashes, we just
compress and store when the second through MTH block generates the same hash.
We recall the first one and we used the dictionary from the first
one to compress the second one
Prasanna Malaiyandi: So you get better compression
Howard Marks: we can store it compressed without the overhead of
storing the dictionary a second time.
Prasanna Malaiyandi: yeah.
W. Curtis Preston: Yeah.
Yeah.
Howard Marks: and it becomes essentially the difference,
Prasanna Malaiyandi: Yeah.
So instead of storing a bigger block, it's like just very, very small Deltas
because they are cryptographically
Howard Marks: right?
W. Curtis Preston: that,
that's
Howard Marks: similar.
The mathematicians would say it's a limited cryptographic distance.
Prasanna Malaiyandi: That's unique.
I've never heard of someone doing that.
Have you, Curtis?
W. Curtis Preston: just this guy that we had on the podcast a little
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: that looks a lot like Howard.
Howard Marks: It, it, you know, nobody else is doing it now.
Um,
W. Curtis Preston: Is it, are you patenting it or,
Howard Marks: there are patents around it.
I don't, I haven't looked to see exactly
W. Curtis Preston: gotcha.
Gotcha.
Howard Marks: to.
W. Curtis Preston: Yeah.
That
Howard Marks: Um,
cause reading patent applications makes my brain hurt.
W. Curtis Preston: I see, I thought that you would, let's
say you got two chunks, right?
And you run the really weak, but much faster, I'm assuming, uh, hashing
algorithm, and that you would say these two blocks definitely aren't the same.
And so let's not do anything else other than com.
They're not, they're nowhere.
They're, they're, they're cryptographic distance.
I think you said so far apart, there's no point in running the
stronger ddu, uh, thing on it.
Um, that's
where I
Howard Marks: it, it turns out, it turns out even with a weak hash, the
number of identical hashes that are not identical data is so small that
the cost of testing is ignorable.
Prasanna Malaiyandi: Yeah.
And especially if it's in flash, like probably
Howard Marks: the,
Prasanna Malaiyandi: is
Howard Marks: compare is so rare.
It doesn't matter that it's expensive.
Prasanna Malaiyandi: Yeah.
And the fact that you're not doing this in line, right.
So it's all been de
Or
Howard Marks: it, it's not trench.
It's in line, but it's not.
Prasanna Malaiyandi: client.
Howard Marks: act,
Prasanna Malaiyandi: Yeah, yeah.
Exactly.
Howard Marks: it's, it's it's post act, so it doesn't have any impact on latency.
But you know, the, the S CM is a one-way writebuffer.
We write new data into it, it gets demoted to the capacity flash layer
and there's so much bandwidth in the capacity flash layer that reads from
there actually faster than from the scm.
So there's no reason ever to promote it back.
W. Curtis Preston: Right,
Howard Marks: Um, but the other thing is we keep all the metadata in that s scm.
So as you expand the system, you add another enclosure that's got more s cm
and more capacity, the DUP hash table and the similarity hash tables all grow
with it.
So it's one data reduction realm regardless of how big a cluster is.
We don't have to store that DUP table in memory.
Prasanna Malaiyandi: Yep.
Howard Marks: And so you know the whole, well, flash would be great for backup,
except I can't afford it as well.
If you've got three or four conventional PBBAs,
Prasanna Malaiyandi: Mm-hmm.
Howard Marks: you know, first of all, the vendors of PBBAs charged
you a lot for that disc storage.
You know, they, that's a high margin product.
Prasanna Malaiyandi: Yeah.
Howard Marks: as soon as you have two of them, you have two deduplication realms.
W. Curtis Preston: Right,
Howard Marks: And we might talk about data duping 10 to one.
That doesn't mean all your data dupes 10 to one
W. Curtis Preston: right.
Howard Marks: 50% of your data at least is unique.
Prasanna Malaiyandi: Yep,
Howard Marks: Some of your data ddus a hundred or a thousand to one, and most
of the benefits you get is from that data that ddus a hundred or a thousand to one.
Well, when you got two boxes, it's not a hundred or a thousand, it's 50 to 500.
Prasanna Malaiyandi: yep.
And every time you add a new box, you're, you lose some of that benefit as well.
Right.
So,
W. Curtis Preston: you can dup all our episodes down to
like four or five comments.
Right?
Like back up, back up, all the things.
Howard Marks: Well that
W. Curtis Preston: 3, 2, 1.
Just 3, 2, 1.
Howard Marks: that requires the next version of, uh, AI deduplication
that can take out the idle banter.
W. Curtis Preston: Exactly, exactly.
Our episodes will be like five minutes long.
Prasanna Malaiyandi: So just to summarize or just to close on that, so
we talked about the how you guys do it.
So because of all these technologies that you're leveraging or mechanisms,
right, that's how you're able to offer that guarantee, right?
That's better than anyone else.
Howard Marks: Yeah, we, you know, we use Z Standard.
It's a slightly newer compression algorithm than anybody else
does, cuz we started a little bit later than everybody else.
So we got to pick the latest one.
Um, and then we have those, you know, the additional, well, oh, they're numbers,
let's just store the differences, tricks.
And then we do deduplication on variable block, which is
as well as anybody does it.
And then we throw in similarity as, oh, here's another unique
trick nobody else does.
And so the combination is, we are confident that as long as you're send,
you know, we guarantee as long as you're sending us unencrypted data,
that will reduce it better than the other guy, whoever the other guy is.
And if we don't, we'll provide the capacity so that you
didn't pay any more money.
Cuz
Prasanna Malaiyandi: Yeah.
Yeah.
No, that's a great guarantee for end users and customers, especially
with budgets these days, right?
It's like, Hey, I bought this system.
It doesn't quite meet my expectations.
I can't go back to my boss and ask for more money.
So
Howard Marks: Well, you know, the other side of that is, you
know, just, it's really a very simple scale out architecture.
So you don't buy today what you think you're gonna need in three years.
You buy today what you think you're gonna need in a year, and then you
can buy more when you need it later.
Or if, as some of our customers have found out much to therin of
our sales guys, their data reduces better than they expected and they
don't need anymore in the next year.
Well then you're just ahead of the game.
Prasanna Malaiyandi: So, and I know maybe you could talk in gener generalities,
but sort of like if I was a customer who had one of the competition, PBBAs, right.
And I now use Vast, right.
I buy a vast system, sort of like, is there, like what is the savings that
I normally see in terms of storage?
Like if I had like a hundred terabyte P B B A, actual
Howard Marks: if, if you, you know, a hundred terabytes is small for us.
Prasanna Malaiyandi: okay.
Or say
a
Howard Marks: So if you had, if you had a petabyte P B B A, um, then you
know, you're probably storing four or five petabytes of logical data on
it.
Um, and you bought a, you know, a petabyte of usable from us and you'd
probably store 25 or 30% more on it.
Prasanna Malaiyandi: Okay.
Howard Marks: But that petabyte, P B B A is as big as you can buy that P B B
A, there isn't a two petabyte P B B A.
Prasanna Malaiyandi: Yep.
Howard Marks: And the real difference is at restore time
Prasanna Malaiyandi: Hmm.
Howard Marks: because PBBAs are scaled.
For backup speed, not restore speed.
They don't even have restore speed on the spec sheet anymore.
And backups are not sequential operations nearly as much as
you think they used to be.
And you
know, when
Curtis, when when Curtis changed block, block tracking, incremental forever,
W. Curtis Preston: Right?
Howard Marks: of those things make the backup and the restore much more random.
Prasanna Malaiyandi: yep,
Howard Marks: And so if you're backing up to a disc based p v a,
your restore speed is like a fourth or a fifth, you're backup speed.
Prasanna Malaiyandi: yep.
Howard Marks: If you're backing up to a vast, your restore speed
is five times your backup speed.
Cause we're, cuz we are designed to serve.
Re primary storage applications where reads happen much more frequently
than writes cuz the reads come from all the capacity SSDs, the
writes have to go to the s scm.
Um, and so what, where that really starts to get important is when, when we start
talking about ransomware attack, cuz 10 years ago Curtis and I used to teach
seminars and we'd go, yeah, 90, 95% of your restorers are, you know, the file.
Somebody screwed up.
And you know, if it's on A P B B A it'll be restored in a couple of minutes.
And if it was on
tape, you'd go find the tape and then a couple of minutes.
And so, but you don't know you've been ransomware attacked till
thousands or hundreds of thousands of files have been encrypted.
Prasanna Malaiyandi: Yep.
Howard Marks: And so now you have to like use something like instant recovery to
check back, you know, is this backup good?
You gotta do three or four quick looks without restoring, which
is a great feature, but you know, requires a relatively high speed
backend to work relatively well.
And then you're gonna find, okay, this is my last non good point.
And then you have to restore and you are gonna have to restore a lot
of data and restore speed starts to become really important then.
Prasanna Malaiyandi: Mm-hmm.
Howard Marks: And then the kicker is, and the lawyers in the insurance company
won't let you use the, the system that was infected for another couple of
weeks cuz it's evidence or we have to get somebody in to clean it and certify
that it's cleaned well, if you know you can run a VMware NFS data store
on VAs, you can just restore to VASc.
Now it's a bad idea to run your primary and your backup on the same
system for more than a day or two,
W. Curtis Preston: Right,
Howard Marks: but, Compared to not running your primary and just, you
know, if your choice is backup only or primary and backup, and if this one
system dies, I'm really in trouble.
Not that part of choice for me.
I want my users back up and running.
As soon as the lawyers let me get tacked to the old system, or my
VAR gives me a new system, or I have someplace else to storage, VMO
to, I'm getting that stuff off there right away.
But that might mean I'm up a week earlier and a week earlier is a lot of time.
W. Curtis Preston: my objection to flash for backup has
been for two primary reasons.
One is, is expensive af right second.
Do I really need it?
Right?
Like, because there's, there are a lot of things that we can buy in life, right?
Uh, like I, I need to move fertilizer.
I can totally borrow Prasannas, uh, Tesla and it will do it, right?
But, but is that what I should be using for that?
Do I need, do I need a Tesla to move fertilizer or will
my Prius do
Howard Marks: need Prasannas.
Tesla to move fertilizer if you ever want prasanna to speak to you again.
W. Curtis Preston: no, that's, that's true.
By the way, the Prius has been used to move fertilizer just for the record.
Um, but so, so that's the thing.
It's like, there, there are a lot of things, like, this goes back
to the c d P, the c d P argument that I made back in the day.
It was the same thing, the same two arguments.
One was c D P was too damn expensive, right?
And then the other was, does anybody actually need.
The, the, the, the functionality that C D P provided.
And the answer is yes.
0.1% of the population needed what C D P provided.
And that's why you don't really see c D P as a choice very, very often these days.
Right there, there, there's a one or two companies that do it now, um,
and all the other products have died.
So those are my two arguments.
It's, I, I already know what your argument to the second one is gonna
be because you just gave it, I think.
Um, so
why
Prasanna Malaiyandi: about for cost?
Howard Marks: Well,
W. Curtis Preston: talk about costs?
Howard Marks: so, for cost, it depends what flash systems you're talking about.
W. Curtis Preston: Mm-hmm.
Howard Marks: Um, I will give you that most all flash systems are
designed to be fast as possible for a small amount of data, because that's
what you need to run the Oracle databases that make companies work.
And so if you, you know, it's a block, you know, it's block storage to be low
latency to support O L T P and therefore expensive because that's, you know, if
that system goes down, you count by the second how much money you're losing.
And so you have always bought expensive storage for that.
Um,
W. Curtis Preston: sort of the, sort of the true normal sort of, if I, if
I can use this word, pure flash array.
Howard Marks: Yes.
W. Curtis Preston: the, that type is designed for that, right?
Um, that's technically pure with a small p, but it works the other way as well.
Um,
Howard Marks: talking either way, you know?
Yeah.
You know, I could name half a dozen other products, but
W. Curtis Preston: And they're just too expensive.
Howard Marks: of it, you know, it's, we're gonna design a system based on
having a, a pyramidal tiered system.
this is the one at the top.
Prasanna Malaiyandi: Yeah.
Howard Marks: And if you assume you're gonna build a tier system, then you
want the one at the top to be as fast as possible, and you kind of
don't care how much it costs because
you'll just put stuff that doesn't deserve it on the next tier.
Prasanna Malaiyandi: Yep.
Howard Marks: Philosophically our idea was we're gonna make something that
delivers performance for everything but the very, very top there.
W. Curtis Preston: Mm-hmm.
Howard Marks: And goes down in cost to where Well, if you use enough
of it, you don't need those tiers.
You don't need the complexity.
Right.
So, you know, part of our story is as you consolidate workloads, you have
workloads that need performance, and you have workloads that need capacity.
When you add capacity, performance comes with because in,
you know, spindles, how many SSDs?
Yeah.
A hundred SDS is so much performance.
200 SDS is twice that much performance.
So if you take the applications that need capacity, And you put them on the
same system as the applications that need performance but don't need capacity.
The performance that the capacity creates is used by the applications
that need the performance and the cost of the performance is brought down
because you've used that much capacity and you get in a vir virtuous cycle.
W. Curtis Preston: I think I followed that.
Prasanna Malaiyandi: yeah, it, it, it's basically
by
Howard Marks: if you, if
Prasanna Malaiyandi: that's
common.
Howard Marks: if you, you're, paying 10 x for 10% and one x for
90%, then you're paying a hundred.
If you have one tier that costs a hundred,
W. Curtis Preston: Mm-hmm.
Howard Marks: why have two tiers?
Prasanna Malaiyandi: Yeah.
Howard Marks: And when you use capacity, that capacity comes with performance.
W. Curtis Preston: Mm-hmm.
Howard Marks: And that means that performance is available
to other applications that didn't need the capacity.
So you don't need to have separate systems, you just have
W. Curtis Preston: So, so if I could, if I could try to put this
in, in, in just different words, but it'll say the same thing.
If I've got a hundred QLC disks, right.
Um, and, and these are how big
Howard Marks: 15 or 30 terabytes.
W. Curtis Preston: the each, each one, right?
Howard Marks: Each one
W. Curtis Preston: So if I've got a hundred, I've got one and a half
tear, one and a half petabytes.
Did I
do that
right of raw?
Yeah.
Okay.
All right.
So I've got one and a half petabytes of raw capacity.
And what you're saying is we can just take a slice off the top, if you will.
You know, we used to call short stroking the discs.
The, obviously you don't need to short stroke a, a flash, but you're
basically saying, we're just gonna take a slice off the top, uh, of these
150 discs and we're gonna get this massive performance slice, uh, for,
for the 10% that need that performance.
And then the rest will just put wherever we need to put it.
Is that, Does that sound about
right?
Howard Marks: I'm, saying all those SSDs create one pool of
performance and one pool of capacity,
W. Curtis Preston: Right.
Howard Marks: and a workload can draw from either one as much as it needs.
W. Curtis Preston: Gotcha.
Prasanna Malaiyandi: Is it separate capacity and performance pools that you
then assign to Applic?
Okay.
It's just one
pool that includes both
Howard Marks: that, and now, you know, and you can use q o s, you can say,
okay, this workload gets a hundred thousand iops, or, or 50 gigabytes
per second, and this other one gets
different.
Prasanna Malaiyandi: yeah,
W. Curtis Preston: yeah,
And
you'll just use
Howard Marks: And so you
W. Curtis Preston: you need to
Howard Marks: performance, right?
But you know, if you've got, um, you know, your backups and you've got the developers
who wanna do run, live copies of the database, well run it all on one system.
It's, it's an all flash system.
It's fast enough to run the database.
Prasanna Malaiyandi: It's almost as if you're saying, You've built a
system that works for all workloads except that 1% or whatever, that's
like that very, very, very high end.
And you're saying you have one common architecture that allows it
to deal with, regardless of if your workload is capacity focused and not
very performance, it doesn't need a lot of performance or it's high
performance and maybe a little capacity.
It's all a
Howard Marks: and and it doesn't matter whether your definition of
performance is bandwidth or iops.
You know, it's like all but that very lowest.
You know, we, you know, we're an all flash system lightly loaded.
We deliver one millisecond latency.
Prasanna Malaiyandi: yeah,
Howard Marks: You know, some systems can deliver half that
and some rare applications care.
But you know, between that and the 10, Tencent, a gigabyte, well,
there are 20 terabyte hard drives and super micro servers and you
know, they don't do any iops, but you can write to 'em pretty fast.
You know,
Prasanna Malaiyandi: Yeah.
Howard Marks: in between we can cover.
W. Curtis Preston: So we're dancing around.
You're saying why you could be cheaper, but let me,
let me just put a, lemme just put it right, you know, sort of, I'm
assuming that you get into competitive bids with PBBAs on a regular basis.
Howard Marks: Yes, sir.
W. Curtis Preston: Okay.
How do you do there?
Howard Marks: They're easy.
Those are very high profit margin products for
the
W. Curtis Preston: so you're, so you're, saying you can come in
less expensive than the effective price of the typical P B B A, even
though you're using all this flash.
Howard Marks: Yes, sir.
W. Curtis Preston: Okay.
Because that, that's the short answer.
I like the long answer.
That's, I like the long answer.
You and I
live in long, right.
Um, yeah, but in the end, it doesn't matter if it's still more expensive.
Howard Marks: yeah, the, you know, the long answer is, you know, we use the
cheapest flash we can get because we designed the system to treat flash well
and understand how to minimize wear.
We ha our erasure codes have 3% overhead at I at large scale, so we're not wasting.
Space on raid, we reduce data better than anybody else does.
So you know, we're getting as much capacity in there.
Um, and then when you start saying, okay, it's 30 terabyte SSDs, so you get a lot
of capacity and a little bit of space and a little bit of power, and the power
and Rackspace start to add up as costs.
Um, especially when you start looking at the fact that the leading PBBAs are
still using eight terabyte hard drives because that rehydration tax of turning,
W. Curtis Preston: Hmm.
Howard Marks: making everything random, well the bigger the hard
drive gets, the worse it is cause.
One hard drive is a hundred iops.
Prasanna Malaiyandi: Yep.
Howard Marks: Doesn't matter whether it's a one terabyte hard
drive or a 20 terabyte hard drive.
And so they're just reaching the, the world.
The land of diminishing returns on IO density.
They can't go any lower.
And now the sheet metal and the power supplies and the SaaS
expanders are becoming a larger and larger percentage of their cogs.
And they mark 'em up a lot cuz there's a lot of IP in there in terms of
software and they have to make a margin.
Um, and so we just don't have most of those problems.
Prasanna Malaiyandi: Yep.
W. Curtis Preston: you're, you're, also marking up due to your ddo, right?
I mean, you
Howard Marks: Yeah.
Or you know, some of it, you know, some small portion of the difference is, you
know, compared to the guys, those the best P VBAs, we still do a little bit better.
but but when we go into a customer who says, no, no, no price, this as
if you de-dupe the same, we're still coming in with a lower selling price.
Prasanna Malaiyandi: Yeah, I, I'm not surprised about that.
The other thing, Howard, I wanted to bring up, I know you
mentioned sort of dis drives and the a hundred iops limit, right?
That each of them typically have.
The other thing that I've also seen is as the drives get larger and larger, anytime
you have to do a raid, rebuild, right?
And you're talking like a 20 terabyte drive and it just takes longer and longer,
and now there's a potential for failure,
Howard Marks: Yeah.
Well,
Prasanna Malaiyandi: becomes a lot worse.
Howard Marks: you know, I do a lot of, you know, resilience calculations
and people just don't realize how big a factor rebuild time is
in the probability of data loss.
Uh, we had one customer share with us.
They ran, you know, the leading.
Scale out system before us and the average for their rebuilds was 53 days.
Prasanna Malaiyandi: Oh
wow.
W. Curtis Preston: That's two months.
Howard Marks: yeah, that's two months during which time your data is exposed
and you know, could be slightly exposed if you are already running.
N plus three could be really exposed if you're running n plus
one, like some vendors recommend.
So it all depends.
W. Curtis Preston: right.
Okay.
So, so I think, I think, you know, you, you've definitely
covered the cost argument.
Um, the, and, and it's, I think if we just back up, you've
already covered the why, right?
The why.
would, why
is today's Restore different?
Howard Marks: Ransomware.
W. Curtis Preston: Yeah.
Howard Marks: stores.
The stores bigger.
And the restore location is less well known
W. Curtis Preston: What do you mean by that?
Howard Marks: you may not be able to restore back to the infected
system cuz it's still evidence,
W. Curtis Preston: okay.
Understood.
Okay.
Howard Marks: right?
You need someplace to restore to.
And you know, having it where the primary in the backup are duped to each other
probably gives you that in a pinch.
Prasanna Malaiyandi: Yeah.
Have you
seen.
Howard Marks: emphasize in a pinch, cuz
W. Curtis Preston: Yeah,
Howard Marks: you know, we've, we've all been trained, no, no bad idea.
Don't mix the strip, don't cross the streams.
Um, but, but that's the, if you have a backup on the primary, you
don't, you don't have a backup.
But if you have to choose between backup and primary, I'd rather have primary.
Prasanna Malaiyandi: Have you seen customers actually do this
in the field with fast systems?
Howard Marks: Oh, we have several customers doing really
large scale backup to Vasst.
Um, we had one customer who was kind of shocked cuz they were
doing encryption in net backup.
And so they expected us to not reduce data at all, uh, but they were doing encrypted
net backup backups of Oracle dumps of the same database over and over again,
encrypted with the same encryption key.
And we started seeing about 20% reduction just because even when you encrypted, if
you're backing up the same data, it looks
the same encrypted as it
W. Curtis Preston: right.
There's like sort of two questions in my head here.
One is, and, and they're, they're very much related.
One is the, the whole backup container problem, right?
Meaning that you get the net backup container and the.
Arc serve container and the backup exec container, you know, and they
all stored backup data differently.
And you have that issue.
And then you, but you, there was something that you alluded
to that I found interesting.
You said commonality between the backup and the primary, but the
backup is in some weirdo format.
So are you able to get backup or commonality between the
backup and the primary?
Howard Marks: Now in that case, it's
more likely we'll see commonality between multiple primaries.
You know, it's more like you
restored 17 windows VMs
W. Curtis Preston: Does the way that you're doing ddo make the
format problem any less problematic?
Right.
Howard Marks: O Only in that we reduce them all as opposed to if you were relying
on the data movers to do reduction.
So, you know, kind of the most common case is the storage guys like backup,
you know, com Vault or net backup or Veritas or Veeam or whatever they use.
And the Oracle DBAs don't trust them and insist on doing, doing, dumps.
W. Curtis Preston: Right.
Howard Marks: And so, you know, if you're doing both to, you know, they
just give the Oracle DBAs, okay dump to this NFS mount point on the vast.
Well then we'll reduce all of those dumps as well as anybody could reduce
all of those dumps and your data mover.
You'll do data reduction at multiple stages to manage the network traffic.
And then we'll do the final dup at the end cuz we're finer grained.
And the sim similarity works really well for things that are duped course
grain, cuz the edges all look similar.
And so when
we, when we run, you know, we have a probe you can get as a VM that
scans your data and reports back, this is how much it would reduce.
And this is how much of that comes from each of these techniques.
And so when we run, when we do that with data from a data mover,
d duper, those are usually, you know, 128 K or big blocks because
they have limited memory available.
And so we see more similarity cuz we're finding those pieces Finer
W. Curtis Preston: just to, just to make sure I understood correctly.
So the, one of the question that I didn't really ask was, you know,
when you buy a, you know, pick your favorite P V B A, they tend to support.
These five backup products, and if you buy a different backup
product, well, they're like, well, we don't understand that format yet.
And so then they have to go and do some development work to figure
out how to crack that container.
Do you not have that problem or have you done that
Howard Marks: We, We, have not optimized for any of these backup applications,
W. Curtis Preston: And yet you
still get better duped than the other guys.
Howard Marks: Our, our general case data reduction against all of these reduced
data types still gets better reduction.
You know, we are not, you know, scanning for the timestamps in Oracle rack
dumps and, you know, that level stuff.
W. Curtis Preston: Right.
Howard Marks: Not to
Prasanna Malaiyandi: agnostic, right?
Howard Marks: to say we won't in the future, but our, you know, our data
reduction was written for primary storage.
It just so happens that being an
N F S or an S3 target for a backup data mover is a simple case of primary storage,
Prasanna Malaiyandi: Yeah.
Howard Marks: it just works.
W. Curtis Preston: Right.
Right.
Hmm.
What do you think Prasanna,
Prasanna Malaiyandi: in the future?
So I, I had no complaints to start with.
Uh, the one question
W. Curtis Preston: I lost this argument?
I think I might
Prasanna Malaiyandi: I think, I think you lost this one.
Uh, Howard, the one last question I had was, I know some of these backup vendors
support the ability to do source side due duplication by integrating with
the purpose-built backup appliances.
Does VAs support that?
Are you guys planning to support that?
I know you're looking, you just
previously said, right, that you're
Howard Marks: don't, we don't, um, I've never been really comfortable with the
use of client side CPU for that cuz client side CPU is valuable for other things.
Um, I think, you know, doing a pass at some level in the data mover.
Prasanna Malaiyandi: Mm-hmm.
Howard Marks: It's like, okay, we'll we'll de-dupe at the media server at some
large grain so that we're not transferring 50 copies of windows over the network.
Um, is perfectly reasonable thing to do cuz it's a network
bandwidth management technique.
Um, things like Didi Boost are, you know, let's offload this from the,
the P B B A to the client and we'd just rather do the work ourselves.
Um, and in our architecture, since you can just add more servers at the front end
and you just have to buy the servers, we don't even charge for that software.
If you need more compute.
To do more
Prasanna Malaiyandi: you just scale out.
Howard Marks: to more dup, you just add a few more servers as opposed to stealing 5%
of the cycles of all of your VMware hosts, which means you now have to not just
buy servers, but you have to buy another VMware host, another VMware license.
All the other stuff you put on a VMware host starts to add up.
Prasanna Malaiyandi: Yeah.
Gotcha.
W. Curtis Preston: Well since, well, since you stepped into my neighborhood
now Howard, I will have to say that source side, DUP done correctly,
speeds up the backup and reduces the C P U utilization on the client.
But I, I want, I can't speak to the, to the implementations
you were talking about.
Um, I can only speak to the one that I am obviously very familiar with.
Um, cuz there, you know, there that, that is the off discussed
thing of like, well there is a,
Howard Marks: It's
W. Curtis Preston: know, there's a
pen.
Howard Marks: it's also a different case because of the assumed
bandwidth at all the stages.
W. Curtis Preston: right.
Howard Marks: You know, I'm, I'm kind of assuming that there's
a lot of bandwidth for short
distances in the data center
W. Curtis Preston: Well, all right.
I, I concede this battle, Howard, I lay down my sword.
Um,
Howard Marks: Okay.
I
W. Curtis Preston: you know, I mean, you, what's that
You expect
to what?
Howard Marks: in the mail,
W. Curtis Preston: Um, yeah, I'll send you, I'll send you something.
Um, alright, well, uh, Howard's been great.
Uh, glad to hear the update and glad to, you know, I, I remember we did,
now that I heard you describe it, I, I think we did cover it in the last one,
but I think you went deeper this time and that, that's good to hear this idea
Howard Marks: probably.
W. Curtis Preston: you can, that you have the, that you have the, the bandwidth
to, to, to, to how many different ways did you say you try each block
for
Howard Marks: there's five compression algorithms and, and then there's a strong
hash and a number of similarity hashes.
I can't remember offhand what they are,
W. Curtis Preston: Gotcha.
I got, I thought I heard you say 15 total ways.
I thought I
heard you say
that.
Howard Marks: it on that order
W. Curtis Preston: Gotcha.
So the fact that you can take each chunk and try 15 different ways to
compress it and pick the one that works the best is pretty damn cool.
Um, and, um, you
know, it's just on
Howard Marks: of it's just cuz we can parallelize it so well
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: that too, right?
Uh, the
fact that, you can, you know, that's a.
Howard Marks: one takes.
We're doing a lot.
Prasanna Malaiyandi: Yeah,
W. Curtis Preston: it's that, that beauty of the scale out architecture, right?
That you can just pass that out, um, like that.
All right, well, thanks for coming back, especially to, it's really funny
how we had you, you're like, you're listening and you're like, Hey, you said
mean things about the way I do things.
I will, I accept your challenge.
And I'm like, all right, come on back.
Uh, happy to do that.
And we'll do that with other people.
By the way, if you're out there listening and you're like, the thing that Curtis or
Prasanna said is wrong, we will be happy to have you on and have us prove to you
why you're wrong or, or in this case,
In this case, uh, right.
Yeah, we concede.
Well, I mean, I mean, I think my concerns are certainly valid and
there are certainly vendors out there that are like, yes, we can
certainly sell you this appliance for the purposes of backup, because
recovery speed is really important.
I'm like, but it costs five times the cost of this thing over here.
I don't like how much better could it possibly be Anyway,
so that's, that's where those arguments tend to come from, so.
Alright, well thanks.
Thanks Howard for
Howard Marks: and given, given the marketplace, they're
not completely unreasonable.
Uh, you know, we, we just do things sufficiently different that,
you know, if you think restore speed's important than we do,
W. Curtis Preston: right, right.
Prasanna Malaiyandi: Yep.
W. Curtis Preston: Yeah.
Yeah.
Ransomware.
All right.
Once again, ransomware, you know, I don't know.
What do you, what do you call it?
Uh, trump's all, um, although I don't enjoy that word as much
as I used to for some reason.
Um, anyways, thanks for, thanks for your questions for,
Prasanna Malaiyandi: Uh, I try.
I try.
And Howard, great to have you back on the podcast.
Hopefully you'll come again.
Howard Marks: Always pleasure.
As long as I keep winning, I'll keep coming back.
W. Curtis Preston: and we thank you to our listeners.
Uh, you know, we're nothing without you.
Remember to subscribe so that you can restore it all.