curtis: I'm pretty sure we've said smoking hole more times
than we've on this podcast.
Just for the record.
Just saying
It's getting a
lot of play today.
Hi and welcome to Backup Central's Restore it All podcast.
I'm your host, W.
Curtis Preston, AKA Mr.
Backup and I have with me, my Bhangra dance consultant, Prasanna
Malaiyandi, how's it going Prasanna?
Prasanna: I'm good Curtis, but I have to warn you.
I have not a dancer at all.
So probably the wrong person to be seeking advice about dancing from,
curtis: But you said that you knew about Bhangra dancing and that you
could advise me on these things.
Prasanna: I told you that it's like a Indian dance style, if you will.
And you had asked a question of, have I seen it because I bet I've
seen a bunch of Bollywood movies.
curtis: You expanded my horizon I bought my tickets my wife and I will be going
to see the show it's called Bhangin' it!
It's bangin' was spelled would be H so it's it's trying to like
do an homage to the Bhangra.
So it's a, new musical at the LA Jolla Playhouse, which is a very nice
Playhouse that I've actually never been.
I've lived here 20 something years.
I've never watched a show there, but a lot of like big Broadway
shows actually start out.
I've never started.
I've, always watched the Broadway shows
Prasanna: Like
gone to Broadway.
curtis: This is the kind of show that could possibly hit big on Broadway.
And so we'll see it and we'll see if it's any good and
I'll
Prasanna: waiting for
curtis: with my review.
Prasanna: Yes.
I think our listeners will be curious.
And by the way, for those in San Diego, when is it running?
Do you know how long?
curtis: It's running.
It's running until April.
Prasanna: Okay.
So that
was
folks in San Diego.
curtis: Yeah.
Yeah, depending on when this goes live, if it goes live less than a month from
now, then you have two days left to go see it because it runs until April
17th at the LA Jolla Playhouse, by which time all the tickets will be
gone and you won't be able to see it.
Sorry, I don't know what to tell you, but so we, have a longtime
friend on the podcast here today.
Prasanna.
I'm excited to bring him on I, and not just because he's one of those
people that make me feel young.
As, been in the it industry for an awfully long time, makes me feel like
a young whippersnapper sometimes.
He is now the technologist extraordinary and plenipotentiary at Vast Data.
Welcome to the podcast, Howard Marks.
Howard: Thank you.
It's very nice to be here.
I was always about Fauci guy, so I don't know much about Indian dance.
Prasanna: Curtis didn't either before he met me.
So it's fine.
curtis: Yeah I, my knowledge of Indian dance it basically includes the
reference to it in what was that movie?
Prasanna: Millionaire.
curtis: bright, the BR the bride and prejudice.
Prasanna: Oh
curtis: There's a
Prasanna: yeah.
I th I think
curtis: there's a, it's a pride and prejudice,
Prasanna: yeah.
curtis: Knock off done with what's her name?
. Prasanna: Ashwaryia Rai.
I think.
curtis: She remember she, she, in the movie she, gives two D two dance moves.
It was petting the dog and screwing in the light bulb.
I don't know if you remember that.
She says that.
That's literally the extent of my knowledge of Indian dance.
That, and the fact that I've watched a bunch of Bollywood movies, but
that's all thanks to Prasanna.
Prasanna: Yeah.
curtis: So you never know what you're going to get when you're listening to the
Backup Central Restore it All podcast.
Speaking of which, let me throw out our usual disclaimer, Prasanna
and I work for different companies.
Persona works for Zoom.
I worked for Druva.
This is not a podcast of either company and the opinions that you hear are
ours, and be sure to rate this podcast ratethispodcast.com/restore, or just
go at your on your favorite pod catcher apple podcasts and just scroll down
to the bottom and give us some stars.
And if you really want to make my day, actually put some words there.
Yeah, absolutely.
And if you are interested in the things that we're interested in, like
backups and storage and resilience and ransomware recovery and cyber
warfare and all of these things.
Then just send me a note @wcpreston on Twitter, or wcurtispreston@gmail, and
I'll be happy to get you on the podcast.
Prasanna: friendly.
We ask questions.
curtis: we even apparently, although the last episode I said, unless your
name was Stewart and apparently Stewart has now reached out to you Prasanna
Prasanna: Yes, he has.
He
curtis: and,
Howard: So even
curtis: and
Howard: name
curtis: Even Stuart can get on this podcast.
So if we're going to let you know a mouse on the podcast, then surely we can let
you, his name is Stuart Liddle for those of you that didn't get that reference
anyway.
Howard: to make me feel honored here.
curtis: We literally let anybody in the door,
including guys who always wear Hawaiian shirts.
Howard: They're comfortable.
They come in my size and at this point I'm just known for them.
I have been known to tell people I'm going to meet at the
Starbucks at some conference.
Just look for Santa Clause in an Aloha shirt.
That will be me.
curtis: much.
It pretty much
Prasanna: that's.
Howard: Yeah.
You know how many 350 pound guys with a gray beard are there walking around the
average tech show, wearing an Aloha shirt?
Two
curtis: I'm going to, yeah.
Two, yeah.
At most.
Absolutely.
And one of them is going to be you.
Howard: Yeah.
curtis: so how long have you been at Vast Data?
Howard: I've been at Vast Data three years and 15 days.
curtis: Wow.
Prasanna: And the company is fairly new as well.
Howard: I joined Vast Data the day before we came out of stealth.
My, my first official act at Vast Data was a briefing for Chris Mellor followed
the next day by Storage Field Day.
curtis: Wow.
Howard: Nothing like starting off running
Now, I joined Vast from being an independent analyst.
So there were a couple of weeks there where I was getting brought up to speed
and such before my official start date.
But yeah,
curtis: And why don't you give a for those that aren't familiar with Vast
Data, give us a, know, the elevator
Howard: sure.
curtis: and
Howard: The really short form on Vast Data is that we make very large scale all
flash file and object storage systems.
And when I say very large scale our average selling price for
our cluster is well on the north side of a million dollars.
It's multiple petabytes.
Today we're just introducing a new storage enclosure that brings
our building block down from 675 terabytes per HA enclosure to 338.
So we're taking it down by factor of two.
We're going from a two U to a one U enclosure.
We'll talk about that in a little bit, but the innovative thing
about Vast is the architecture.
If you talk about a large scale system, like we build traditionally, that's been
done with a scale out, shared nothing model where you have a lot of x86 servers.
Each of those x86 servers owns some set of media and they communicate
on a backend network and software makes it look like one big system.
But those systems start to break down at really large scale.
And so we've come up with a new model.
We call DASE the shared everything architecture instead of having a field of
peer nodes, each of which owns some media, we disaggregated the media into these HA
enclosures that I was just talking about.
So no single point of failure, 400 gig connections to an NVME fabric and
that's typically a hundred gig Ethernet.
Some of our HPC customers like to run InfiniBand so we
can do InfiniBand as well.
All those enclosures do is hold data.
There's no services there.
All of the services, everything that you would think of as the controller function
of the system runs in stateless Docker containers in the front end servers.
So when a user makes a request to a protocol server to one of
those front end servers could be NFS, could be SMB, could be S3.
That server looks in the metadata that's stored in storage class memory
in the enclosures, finds the data the user's requesting in the data in
QLC flash in those same enclosures, retrieves it over the NVME over fabric's
fabric and delivers it to the user.
So there's none of the traffic from node to node required to reassemble
data, everything's north, south across that NVME over fabrics connection.
And since the metadata is in storage class memory, it's fast enough to
directly access by all of the front end servers that they can just share it.
They don't have to cash it.
And by not having the cache, we don't have all the complexities
of keeping the cache coherent.
Prasanna: I was just going to ask about that, Howard.
So it looks like though you're dis-aggregating the actual storage
and metadata from all the front end processing, which allows,
would assume the front end to scale independently of the backend.
Howard: So each of those front end protocol servers, mounts all of the
SSDs in the cluster at boot time.
And then it looks at all of those SSDs, and at those are the SCM
SSDs that hold the metadata and the QLC SSDs that hold the data.
So everybody has access to everything.
And instead of sending messages back and forth between the front end servers,
they simply write a single of truth in the shared metadata, so that the
old so that you can place a lock on the metadata or update the metadata.
But you never have to tell everybody else you updated it because if they want
to know what the state is, they'll go look in the one place where it's true.
Prasanna: Yeah.
And because everything is stateless in the front end, you don't have to worry
about that necessarily to everyone
Howard: Right,
Prasanna: that backend
Howard: right.
curtis: So the backend has both SSDs and QLC.
Howard: What has SCM sort of storage class memory SSDs, and that can be
Optane or and it has low end QLC SSDs.
Prasanna: So
curtis: And the, the, yeah the, storage class memory is what's
holding the metadata and the
QLC is, what's holding the data.
Howard: Primarily.
It's also used as a write buffer.
curtis: Okay.
Okay.
Howard: So writes come into the storage class memory and get mirrored to two
different SCM SSDs and then get ACKd.
And then the migration from SCM to QLC happens after the act.
So we have more time to do things like compress more fully.
curtis: This is a very different game than.
This idea of all of the front end nodes, being able to mount the entire
Howard: Yes.
curtis: the background
Howard: Yeah.
We we eliminate the whole concept of ownership and all the
complexity that, that creates.
And now I'm going to blow your mind because when I say the metadata is in
the SCM, I don't mean just the element store metadata, the metadata for our
merged file system object store, but also the data reduction metadata.
And so when you add another enclosure to the cluster, you add more SCM, which
means you add more room for that metadata.
So regardless of the size of cluster, the cluster is one data reduction realm
across tens or hundreds of petabytes.
Prasanna: Because everything's looks like one cluster, if you will, or one system.
Howard: right.
And, we don't have to hold the data deduplication hash
table in memory any place.
It's all in SCM where it's fast enough we don't need that.
So we don't have the limitations of how big a deduplication realm can be
that most deduplication systems have.
curtis: right.
They typically top out around a a petabyte or so, and then you
can't get any bigger than that.
I don't know where to start on my questions!
Howard: so from that, from the backup point of view, we're discovering that
the customers are starting to demand higher restore speeds that traditionally
all a customer worried about when they were picking the storage for their
backups was it fast enough that I can make my backup within the window?
And so we got systems like Data Domain and other disk based deduplicating systems,
where there was a big write read asymmetry where you could write data faster to
them than you could read data from them.
Because reading data that caused the system to rehydrate turned
sequential IO into random IO.
And they had disks on the backend.
And as disk drives have gotten bigger, this has gotten worse
because a 20 terabyte disk drive today delivers exactly the same
number of IOPS that a one terabyte disc drive delivered 10 years ago.
So now 20 terabytes of data gets a 20th as many IOPS.
And so you discover, yes, it takes me eight hours to back this up.
It takes me 82 hours to restore it
and
curtis: Yeah.
D D dedupe has never been very friendly for, large restores, especially if
you're doing any sort of, if you want to do a live mount, forget it right.
From a directly, from a Data Domain.
It's possible in the same way, it's possible that...
Howard: That's, but that's, you can bring up the Oracle or the SQL server VM.
So that the it guys can access the passwords database, so that everybody
can start at running ERP on it again.
Prasanna: Yeah.
Don't use it as production.
That's a bad thing.
Howard: Right.
curtis: right.
Howard: And we're discovering that people's requirements are getting tighter.
You start thinking about software as a service providers where, you know, if you
run some account, some industry specific accounting as a service for a thousand
customers, that's a thousand databases.
And when something goes wrong, you want to restore those databases
as fast as you can, because your customers are going to be standing
over your shoulder, yelling at you.
And the last thing that's kicked, a couple of our potential customers over
the edge is the ransomware threat.
Because the size of the restore grows so much with ransomware.
You start off with, they need to protect my data against ransomware
and use various methods to do that.
And so we have indestructable snapshots.
So you can say snapshot this folder at 6:00 AM when the backup window
closes and retain it for 30 days.
And even if the administrator wants to delete it he can't.
Prasanna: So I
Howard: but
Prasanna: about that.
So I did read a little small blurb about that.
So
What prevents, is that locked down forever?
Like an admin can't delete it no matter what, or is it just, there
are additional safeguards in place to make sure that someone doesn't
compromise the admin password,
Howard: Anyone who ever talked to any customer of EMC Centera knows that if you
build a system where you literally can't delete data someone will get themselves in
trouble and fill it a hundred percent up with junk, and it will be a bad situation.
So you have to provide some mechanism for overriding this because customers
will paint themselves in corners.
As I said, our average selling price is well over a million dollars.
We don't have small customers who we only know third hand through VARs.
We are in relatively intimate contact with every one of our customers.
And so we don't have a fixed policy that says, if you jump through these
hoops, then we will let you delete the undeletable snapshots we, and the
customer agree what the hoops are.
Yeah, multifactor authentication must be three of the five people on this list.
They have to know the passphrase and the proper response to the passphrase.
And if they respond with this other response to the passphrase, then for
the next 24 hours, do not give anybody the secret as complicated as you want.
We'll as long as we can write it down, those are the rules.
And then once you've jumped through the hoops, we give you a time limited
token that allows you to delete snapshots for a short period of time.
And that token is a one-time pad.
So that you can't re it's not good for
Prasanna: Yeah.
Howard: an hour whenever you use it.
It is good for the time when we issue it for some limited period of time.
And then you have to know the next one.
And it's just, it was the best solution we could come up with.
Prasanna: And this is probably helps in cases where someone
attacks a company, they get access to the, to a storage system.
They start deleting back-ups or what have you, it gives you
that extra layer of protection.
Howard: I've seen ransomware , you know, we think of ransomware as being on the
order of the viruses we've dealt with.
And the ransomware reports I see are much more frequently and this ransomware
opened a door and then someone physically hacked for a long period of time.
And they took over some workstation, eventually that some
administrator logged into and they have an administrator password.
And if we're just worried about, if we're just worried about the
script kiddies in a, I can protect against the script kiddies in
building my backup infrastructure and architecture and those permissions.
But we're talking about more sophisticated attacks than that.
And frankly we talk about it as ransomware, but it's also
rogue administrator protection.
Then it's also just the guy who is disgruntled and decides his
way out the door, he's going to make life for his employer.
You're protected against that too.
curtis: Yeah.
Yeah.
And, sometimes rogue administrator is a true rogue administrator, meaning
it's a, it's someone masquerading as an administrator as well.
That hacker that you talked about.
So let me let, me ask call it a difficult question, call it
whatever you want to call it.
But when I hear about boxes that where you're not supposed to be able to
delete data, but then there is this other way where you can delete data.
I immediately ask I, I have to ask the question doesn't that suggest
that there is a this is, I'm assuming this is a, Unix-based OS and that
there's that there is a root account,
Howard: It we, run in containers under linux
curtis: So there is an account, there is a a root account and that
if someone did some sort of just the right attack against that box.
And again you've already mentioned that there is that
these are sophisticated attacks.
If someone Did a privilege escalation attack against
the CoreOS, and now they've gained access to a privileged Couldn't want
Howard: if someone
curtis: want.
Howard: administrative access to the management network, because the
ports that face users as storage
ports, can't be logged into
curtis: Okay.
they're
cause they're back.
Cause they're backend,
Howard: so if you're wondering, if you want to log into
Linux as root on one of our appliances, then you need,
then the management network has to be set, has to be compromised.
And we start saying, are you looking for protection against destruction?
Because if your data center is compromised, everything can be destroyed,
but that's not really the level of attack that we're, concerned about.
We're not talking about and someone walked into the data center because we
hadn't disabled their key card and left 20 pounds of thermite in the middle of
the floor, who would do such a thing.
I've done that on video I was being paid.
So you know, I, it is a vulnerability, but it's the
generalest of the vulnerabilities.
You're pointing out that if I have sufficient
access, I can destroy anything.
curtis: The but it sounds like you have protected from the rogue
administrator, the stupid administrator.
And and someone gaining access to those.
But let me just you to clarify something from your previous answer, when you said
that means the management network has been compromised, what do you mean by that?
Howard: So you manage the system through different ethernet ports,
then you access the system.
And so too, you're if there's a vulnerability where a user could log
into the appliance as the Linux root user that Linux root user can only
log in on the management, physical Ethernet port on the appliance, not
on the gigabit NVMe over fabric port.
curtis: Gotcha.
Okay.
Howard: so network security should keep that from being an internet
connected network and to attack.
curtis: Gotcha.
Gotcha.
sense.
Okay.
Prasanna: I had a
question.
So Howard, before we dive more into the data protection side, one thing that
was curious to me was you mentioned that vast supports file and object.
Could you talk about some of the use cases that you see
your customers using Vast Data?
And then I think maybe some of the protection stuff will
probably come alongside that.
Howard: Sure.
We have the majority of our customers use us for primary storage.
And that includes one of the biggest travel sites who uses us for their
big data analytics and are using the S3 Presto connectors to store
all of their analytic data on us.
So that we're much faster than a disk based object store, obviously.
And they can do that processing faster.
We have a lot of hedge funds who do time series analysis of trade
data against large databases to try and predict the market.
We have a lot of life sciences customers who are doing things like.
Molecular modeling and cryo electron microscopy where one microscope generates
many terabytes of data a day because we have very high resolution images.
And we have a major motion picture studio who makes movies.
Prasanna: And so it looks like they are using both sort of the file and the object
interfaces for a lot of these use cases.
So specifically around data protection and backup.
A lot of times you hear The vendor's customers say, object
store doesn't need to be backed up.
Howard: This is a subject that personally I find myself on the fence about part
of me goes I've built a huge amount of resiliency into this single system.
And for durability, if for, availability, I may need to have it in another
location, but for durability, assuming that the whole data center doesn't end
up being a smoking hole in the ground I could get away without backing this up.
I am N I remain firmly on the fence there.
But
curtis: assuming you have the second copy somewhere, you're going to
write.
Howard: may decide that it's, it is data that If, the whole data
center goes away, I don't need.
curtis: Okay.
Yeah.
Agreed.
If, yeah, if we have That, data I would argue why did we make
it in the first place, but,
Howard: That the risk of that is the risk of that is small enough that I'm
going to go once every thousand years this is going to cost me a million
dollars, but it's going to cost me a million dollars a year to protect.
So I'm going to take that risk.
curtis: okay.
So such I will agree to such data classes exist.
I don't run into them much, but I will agree
Howard: yeah.
And and then we get to the okay, so this is the object store that does a
deep dispersal coding, and they have three locations and I can lose one.
So do I need to back that up?
That starts getting really close to now I need to back it up because there could be
a bug in the software that loses my data.
'cause, that's the only thing that could cause that it's like
unprotected against one of my three data centers being a smoking hole.
what again, it's I could see you going, I want to be safe and I can
see you going, it's not worth it.
curtis: And.
Howard: Now for us, most of our users use us for primary storage.
And for someone like that, big data analytics data, they may not back it
up because it's regenerate Hubble, and it's not actually in the form
it's in on the object store, but it's extracts from other things and they
can run the ETL again and it would be really annoying, but it is replaceable.
And then we and then for other use cases this is primary data.
I gotta protect it.
And so we can do snapshots to an S3 compatible object store
and back ourselves up that way.
Or you can back us up the usual ways.
curtis: And could you use one of the, like ones that are like
glacier deep archive where I hope I don't ever have to use this.
I know it's going to cost me a crap ton of money, but it'll save me a lot of money.
In the meantime, can you use that kind of storage?
Howard: The risk reading data out of that kind of storage
requires a few manual steps.
If you just use S3 standard then data in those snapshots is available
in a .Remote folder, like the .Snapshots folder in the file system.
So users can do self-service restore, but that required, but
this, that feature means the object has to be immediately readable.
And so if you, if it went to
Glacier, then.
And it would be like your net backup
Prasanna: Okay.
Howard: this backup isn't in the catalog anymore.
So I got to put those files someplace where I can catalog it and then I got
a catalog and then I can restore it.
So if you
curtis: so it's possible.
it
doesn't sound like it's very it's the smoking hole copy, right?
Howard: It is annoying.
But if it's just, but if you're protecting against the smoking hole,
then you know, you may be willing to put up with the annoyance.
curtis: I'm pretty sure we've said smoking hole more times
than we've on this podcast.
Just for the record.
Just saying
It's getting a lot of play today.
Howard: I spent way too long as a disaster recovery planner.
curtis: Yeah.
Yeah.
So the majority of your customers use you for primary storage, but clearly
you're trying to expand your TAM,
Howard: Well, w we, we deliver all flash at a substantially lower
price than anybody else does.
We start with using the cheapest QLC flash.
We have a file system designed to treat that flash properly.
So we never do small writes that would consume a lot of write amplification.
We do very wide erasure code stripes.
So we've got under 3% overhead, and then we do guaranteed better data reduction
than anybody else in the business.
And so that combination means that on an effective byte basis, from whatever backup
data mover you're planning on using, we're going to be cheaper than a Data Domain.
When you start saying that it's, you have more than a petabyte of data
and you need multiple Data Domains.
And each one of those is going to be a separate deduplication realm.
Then the gap starts to grow substantially.
So if so for these very large customers who have five or 10 or 20
petabytes data across a bunch of Data Domains, simply the fact that we're
one reduction realm makes that makes us much more efficient that can be.
it's one system to manage.
It's one namespace it's one 20 petabytes or 50 petabytes system.
curtis: So you're saying, so let me just make sure I understood
what you said there correctly.
saying on a, regardless of the size of the system, you should
be priced competitive with a Data Domain, but then the bigger you get,
better you look.
Howard: under about 500, any pricing experiments under about 500 terabytes,
curtis: Okay.
Okay.
Howard: in the large end of the business, but yes.
curtis: Right, That is interesting though, that sort of.
into that end of the business.
And you had another there was another large, all flash competitor that's
doing very well, but they have a very different architecture, they're referring
of course, to the orange company.
And
Howard: Yeah, but there,
curtis: than you.
Howard: If you're talking about Flash Blade, that's really a shared nothing
architecture it's of being pizza box servers, they're blade servers, and each
blade has flash modules built in And they they don't scale nearly as large.
curtis: So it sounds like you, you just took, you've built an
architecture based on several new pieces of technology that simply
weren't available, say, five years ago,
Howard: Yeah.
We, are the storage system designed from a clean slate around the 2016 toolbox.
So QLC, flash,
SCM, NVMe over fabrics and other people shoe horn one or two of those technologies
into an existing architecture, but we built the whole architecture
around having those technologies.
Yeah, putting all of the metadata in SCM with no cache meant it had to be in SCM.
And it meant the connection between the compute server and that SCM had to be
fast enough that we weren't going if we cached this, it would be a lot faster.
So that meant it had to be NVMe over fabrics.
And then the QLC flash gives us the cost.
But it, really is if you look at any storage system, it's by definition built
with the parts that the industry is making when they sat down to design it.
curtis: Yeah.
Howard: And that when x86 processor when Mahalum came along and the
memory bandwidth and the number of PCI e-lanes on processors got big enough.
All of a sudden we stopped seeing FPGAs and ASICs in storage systems, we started
seeing software defined storage, cause what was available for the designers
changed and the NVMe over fabrics has been used by most of the storage
vendors for that last mile connection going well, it's going to be fast and
then fiber channel or iSCSI for the user machine to access the storage.
But it hasn't been as effectively used for the server that is the logical
controller to access the media on the back end and the way we use it, we broke the
traditional limitation that a drive had to be owned by one or two controllers.
Cause I drive a SAS drive where an NVMe drive has one or two ports.
Prasanna: Yea.
Howard: We connect that NVMe SSD to what we call a fabric module, which
is an NVMe over fabrics router.
And in fact, in the new box, it's going to be a pair of Nvidia Bluefield cards
and the Bluefield card routes, NVMe over fabrics requests from the ethernet network
to the SSDs and routes the responses back.
But that's all it does.
We don't need x86 servers in the enclosure.
We can do it on the ARMs and the offloads and the Bluefields.
Prasanna: and these are the DPUs, correct?
Howard: Yes.
Yeah.
The Bluefield is, the DPU it's the Nvidia Mellanox version of that.
And so it has an ARM some ARM cores and NVMe over fabrics and RDMA and
other built-in offloads in the chip.
And so we leverage that to do the routing of requests from the front
end servers, everything is, all the work gets done the SSDs and get that
clean fast, more cost-effective channel
curtis: Let me go back in time when you did that first presentation that
you did to the Storage Field Day folks,
Howard: Yep.
curtis: how did that go over with, with those folks?
Howard: It went over pretty well.
There was a little being from Missouri and,
you,
know, we should show you,
curtis: Cause you weren't because you were brand new.
at that point.
Howard: We We were brand new.
And now we're going, okay, look, we've sold a couple of exabytes of storage.
Now at this we, our go to market model's a little different, we sell software.
We arrange for customers to buy the pre-approved hardware at cost.
And the
software licenses are,
curtis: a little interesting.
Howard: and the software licenses are transferable.
So you license a petabyte of software.
And you upgrade the hardware when you feel like you're want to upgrade the hardware.
Cause you want the denser faster one that is always coming, but we'll write
the support contract for 10 years for any appliance from install date.
So
Prasanna: That's very different
Howard: well, a typical
vendor, you would buy an appliance, it would come with an oEM software license.
They would write five years of support.
And in year six they would encourage you very strongly to rebuy.
Prasanna: yep.
Howard: And then when you rebuy, you have to buy another appliance the
software license isn't transferable.
So you have to buy another software license.
So with us, you gotta have your VAR go to a VAR, a hundred
percent channel you go to a VAR.
your VAR, goes to Avnet, says, I want this hardware for Vast.
Now $1.2 million average selling price.
One of our sales guys is involved.
We're writing the high touch sale.
It's not somebody went on a website someplace.
Um, but essentially the VAR, writes two POs: one to Avnet for the hardware and one
to us for the actually he writes one PO to Avnet, Avnet cuts us a PO for the software
and, that's a capacity subscription.
So if you bought a 675 terabyte, enclosure and an appliance, that's got
four servers that provide the front end, which is our usual entry point.
You could license a hundred terabytes for a year.
Multiples of a hundred terabytes for multiples a year.
curtis: And so that, I think that addresses the question that I had.
Cause I listened to the Chris Evans podcasts that you guys did.
Howard: Yeah.
curtis: and there was this talk of the 10 year And, again I'm gonna, I'm gonna just
Howard: Perfect.
curtis: acknowledge that I live in a SaaS world where we preach against
large capacity licensing and capital purchases and all of that stuff.
So when I heard 10 year purchase.
I was like, what?
I gotta, I got to decide now how much I need for 10 years, but that doesn't
sound like what you're talking about.
Howard: No, No, no.
no.
You th you buy the hardware.
curtis: Right.
Howard: We will write a support contract and software license.
One agreement.
For that hardware for up to 10 years from install date at the same rate.
So if you want to keep it for 10 years, you keep it for 10 years
Bought
curtis: I could buy a smaller one and then add capacity.
Howard: Oh yeah.
Our NRR is three.
Lots of people buy small and add capacity.
We had a 300% NRR.
Prasanna: I think you meant NRR,
right?
curtis: Thanks for explaining.
Yeah.
NRR,
you said ARR.
That's why you
have me confused there for a minute.
Howard: Yeah
curtis: I was like an annual recurring revenue of three, three.
Met meant net retention rate, you're saying?
yeah.
So you're saying 300% your customers start out at X and they end up
with three X very regularly.
Okay.
Howard: You know, and you can do that.
curtis: it just grows as they need it to grow.
Howard: Yeah.
And you can do it in the hardware, so if you want to start really small, then
you can buy hardware and license it
Prasanna: oh, interesting.
Howard: So You can buy, a 600 terabyte box and a hundred terabytes software
license, and the 600 terabyte box you bought at what would be our cost.
If we were still selling hardware, we negotiate the cost with the intel
and key Aksia and those vendors.
Prasanna: so you used to sell hardware and then you
of,
Howard: started off in an appliance model.
curtis: Why would I do that?
Is that just like ease of large capital purchase thing?
Howard: Yeah.
curtis: why
would I buy a bigger box
Howard: university, we had a university had this much money in this year's budget.
curtis: Oh, okay.
Howard: We won't put more than a hundred terabytes on it before the next budget
comes around when we renew, we'll renew it as a 400 terabyte license.
Prasanna: and I think this is where at the beginning, you said Howard, that you're
looking at releasing a smaller unit.
Howard: Yeah.
So the new box is one.
You,
it uses the ESS F one L the ruler form factor as, DS.
So we can, we have 2215 terabyte SSDs for 3 38 raw bat, 300 usable.
And that's half the physical size, half the capacity, because what we
have now, it holds 56 SSDs and two U
Prasanna: Gotcha.
Howard: Yeah, the new one is, from the fabric module is those NVMe routers today.
Each one has to be a dual Xeon.
So we have enough PCIE
lanes and the processors don't do hardly anything.
So there's just there's costs there.
We don't need, if the Bluefield
thing
Prasanna: That's exciting.
curtis: right.
So let's, focus for a little bit on.
The only reason I have historically been when, I historically heard the
idea of using flash for backup, I'm like, that sounds ridiculous because
for the same for cost reasons, too expensive I'm hearing you that so I
would put it this way that, in, in this upcoming world, in this current world
in a world where we have large nation states invading other nation states
and then large ransomware organizations in those countries, we had this, was
our last th they're talking about.
So we're, talking about being retaliated against because of this other country.
It's crazy.
So you have this this, need more than ever before for large recoveries.
And I, do believe strongly that there's really only one of two
ways to be really successful in any sort of ransomware situation.
And, it's basically about fighting the laws of physics .Either you
have to have already restored it.
So you already have a hot standby ready to go to switch over to or you're
doing live mount directly from your backup and live mount directly from
your backup is only going to happen if you either aren't, deduplicating
like, the way Data Domain does, or
Howard: Right.
curtis: have flash as far
Tell.
Howard: if you're not, even if you're not, deduplicating when you start talking
about big, hard drives the IO density just
isn't there it's better
curtis: Some somewhere between you and Data Domain, I would put Exagrid,
because exa grid has that front end.
It's not de duplicated now they're there.
They're nowhere near the size of you.
Howard: right, no.
And they have some, and they, have, some flash cache.
And if you look at guys who do integrated appliances where the
software and the target are one thing, those are typically hybrids.
And, so they'll do an instant recover for one or two VMs pretty well.
Cause there's enough flash for that.
But when you start going, I need the database server behind my ERP, instant
recovered, or I need all 50 of these VMs, instant recovered, then it's then you
just, don't have enough flash and you're going to get hard drive performance,
curtis: And so
what it sounds like you've replaced the hard drives with QLC
Howard: right,
curtis: Help me because I don't live in this world QLC from
a cost perspective regular.
Howard: it's, not just QLC.
So QLC means quad level cell holds four bits per cell.
curtis: okay?
Howard: The more, bits you hold, the closer, the voltage levels
that represent the differences are, and the more sensitive the cells
become to a few electrons escaping.
If you have SLC, it's like a light switch it's on or off,
It doesn't matter if a few electrons escape, you can still
tell whether it's on or off.
QLC.
You got 16 values.
The difference between value 13 and value 14 might only be a handful of electrons.
So QLC has less endurance.
Cause every time you erase it, the insulating layers wear down
a little and a few more electrons have opportunities to escape.
And it's slower to write because you have to adjust the voltage level just right
to be one of those 16 voltage levels.
And that takes a little bit longer.
Now the slower to write, we don't really care about because
we acknowledge the writes while it's still in the SCM.
So as long as we are flushing that data out of the SCM, in bandwidth terms
fast enough, Latency is unimportant.
and the endurance we specifically do a lot of things in our
software to manage endurance.
So we write very large writes so that the SSD doesn't have to garbage collect
internally to accommodate small writes.
We erase very large erases so that we delete all of the data in an erase block
in the flash so that the SSD doesn't have to garbage collect internally.
And that means not only can we use QLC, but we can use dirt cheap QLC
SSDs that don't have a DRAM buffer in them to protect the QLC from wear.
If you have a DRAM buffer, then you can aggregate multiple small
writes, but yet, but now if power fails, it's DRAM, you lose the data.
So you need a power fail protection circuit, and you need big capacitors
to power, the power fail protection
circuit so that you can that you can dump the DRAM into flash and
right, and it all starts to add up.
So the SSDs we buy, the other customers are hyperscalers.
They put them in servers.
They only need one port they're writing long tail data.
It's not like they're overriding this stuff all the time.
It's just too many people are looking at that drunken fat frat
boy picture on Facebook it to be on disk so it's on flash.
curtis: A.
Howard: We're leveraging all of that to keep so that we can literally
use that lowest cost flash.
And do the 10 year support because the 10 year support includes if the
SSD wears out, we'll replace it.
Prasanna: cause normally QLC isn't rated for that long.
I believe.
Right.
SLC is years
Howard: S SLC SLC is the very high endurance flesh, but the typical
flash that you see for volume use today is TLC triple level cell.
So it's three bits instead of four bits.
So QLC is 30% cheaper to make because it holds more bits per cell.
And QLC has substantially less endurance.
So when you start looking at enterprise SSDs on newegg.
The 0.1 drive write per day, SSD is slightly better than the ones we use.
And the three drive write per day, SSD, you notice has less capacity because
it's got the same amount of flash.
It's just more over-provisioned so they can wear level across more of it.
And the three drive rate per day, SSD probably has a DRAM cache
and all this stuff to protect it.
Prasanna: Yeah
Howard: And that's what most enterprise storage systems need because how
they put the data in the drive dates back to when it was a disk drive.
And you were trying to keep data logically adjacent, not try and manage
the write pool inside the drive.
Prasanna: yeah,
Howard: The requirements were different.
curtis: Yeah.
Interesting.
Yeah.
So again, going back to.
the fact that you built this from the scratch with that toolbox
from 2016, and you were like we need to, manage write leveling,
Howard: And look, our founder Renen Hallak was the chief engineer at Extreme IO.
And when he got tired of working for Michael Dell, he got to talk to Extreme IO
customers and find out what they wanted.
And nobody said we want faster, Extreme IO was already all flash.
They were still adjusting to all flash.
And it was plenty fast, but everybody wanted to be able to use
that all flash for more things.
And so our whole system is designed to provide very high, random read
performance, across large amounts of flash at an affordable price.
curtis: Got it.
Howard: And so our our performance asymmetry is exactly
the opposite of data domains.
curtis: wait, explain what you just said.
Howard: Our performance asymmetry is exactly the opposite of data domains.
They don't publish restore speeds anymore.
Haven't for years we publish, read speeds and writes speeds and reads
are at eight times faster than rights.
Prasanna: That doesn't mean your rights are slow either.
Just for
Howard: No Our, smallest system does five gigabytes per second of rights.
Yeah.
Or your story system probably doesn't keep up with that, but that's the SLOs.
But what that means is if you scale a system the traditional way, and
you say, I need to move this many terabytes over this many hours, so you
have to scale it by right performance.
Your backups are going to be much faster than your restores.
Excuse me.
your restores are much
faster than your
backups.
Prasanna: Yeah,
Howard: Yeah
we read much faster than we write.
And so if you size for backups speed, you're a store.
Speed's going to be
curtis: yeah.
Howard: nice.
curtis: All right.
Consider me impressed, Howard.
you know, I,
Prasanna: do by the way
curtis: I
Howard: I I've
curtis: I, I,
Howard: time.
I've impressed him once.
this is makes twice.
I'm really, I'm happy with that,
curtis: yeah it sounds like you're, clearly you've been
in the business a long time.
You've seen those companies that have really interesting technology
and nobody's buying anything.
You're not that you,
Howard: but
curtis: the really interesting technology, but you're also actually selling it,
right?
Howard: I decided it was time to get a job.
And I talked to the folks at Vast, who were still in stealth.
And I said to myself, look, Howard, you're a storyteller.
And this is a really good story.
And it doesn't matter whether it succeeds or not.
You're going to have a good story to tell.
and low and behold, it's one of those cases where it was a good
story and the market requirement fit.
And
curtis: don't have to create the need.
Howard: we are selling we have, for the past couple of years
done comparisons, all the storage companies have gone public you.
Yeah.
We're growing faster than all of them put together
curtis: all right Howard thanks a lot for coming on.
We might have to have you back.
Cause I, I know that I know we've, just begun to scratch the surface and but
sounds like you got a good gig over there.
I'm glad.
Both of us could be
employed.
Howard: Ed
curtis: well.
Howard: for the people have known us a long time.
It really must be shocking to you and I both the same job multiple years, but
I'm still having fun at Vast.
And there's lots of interesting stuff still to come.
Having taken a fresh eye to the market.
We got all sorts of good stuff coming.
curtis: Cool.
All right.
I wish you the best.
And thanks Prasanna.
This is one of those cases where your background was very helpful.
I think,
Prasanna: Oh, I try.
I try,
Yeah Yeah.
Having spent a bunch of time building storage arrays.
It helps, but
no, it's still interesting problems though, and, yeah.
Thank you, Howard, for sharing some of the details and indulging in my questions.
So.
Howard: problem