I'm not sure who needs to hear this, but snapshots are not backups.
And in this context, I'm talking about virtual snapshots, like
what you do with the storage re.
We're not talking about cloud snapshots, like what you do
with EBS volumes in the AWS.
If you don't know the difference or you can't articulate why
these snapshots are not backups.
Well, have I got a podcast for you?
Hi, I'm W.
Curtis Preston AKA Mister backup.
And my podcast turns unappreciated backup admins into cyber recovery heroes.
This is the backup wrap-up.
Hi, and welcome to the Backup Wrap Up.
I'm your host, W.
Curtis Preston, and I have with me my Spanish language
encourager, Prasanna Malaiyandi.
I gave you a positive thing
I know, I'm impressed.
And I'm also impressed that you are done now with, what, Spanish 2, right?
Spanish 2, uh, Sisi termine con Span, uh, I almost said
Spanish, Espanol 2, um, yeah.
So now I need to, I need to do a little review because there's, it's, it's
really, you know, for a, for an English speaker, uh, the challenges, as I know,
I've spoken to you, the challenges are things like, we're like, it, it, the,
the gender thing is, is not a big deal.
You get used to that.
What's challenging is words that n an ma like PMA you would
think that would be feminine, but it's actually a masculine word.
Uh, because it, it ends in an, uh, ma.
I have to pound that stuff into my head and just say it over and over.
I will be using, the thing that I made where I, I'm actually
using technology to make a verbal recording that I can listen to.
Uh, it's very, very cool.
Uh, but I am excited to move on to Spanish three and.
Bring you dragging with me.
It's helpful.
I, Used to be pretty good at Spanish because I took it in high
school and then I lost everything.
I can order things off a menu, but that's about it.
Right.
So, uh, I think it's time for the news of the week.
this one is frustrating.
Uh, we'll, we'll, we'll do a frustrating one first and then
some good news about a vendor.
Let's talk about this one password hack.
And it frustrates me for many reasons.
One is, we're such a fan of password managers.
And there are those who are not fans of password managers.
There are also those who are not a fan of the cloud.
And this just, uh, is both of those things.
So, uh, do you want to talk a little bit about the 1Password slash Octahack?
so basically what happened is 1Password, I don't know if it was
1Password or Okta, but basically some hackers got in into 1Password.
They were able to change some things in their Okta instance and try to get a list
of admins, which I'm sure they were going to use to target and be able to deploy
things so they could cause bigger damage.
Right.
But as they were starting to sort of peel back the onion and
figure out, okay, what happened?
They realized what actually happened is it started on the Okta side,
Right.
The reason they were able to do what, what they ended up being able
to do was because Okta had already been, had already been hacked.
And so it's interesting in this case because It wasn't like, oh, they
just got into Okta and then they ping ponged and got into 1Password.
They actually looked at support files that someone in 1Password, an engineer,
had uploaded because they probably had some issue or something else like that.
They were reaching out to Okta and they uploaded basically a support bundle.
Um, and in that support bundle, they contained, in addition to sort of like
information needed to troubleshoot, it also contained session cookies,
Right.
Which they were then able to use.
Right.
Yeah, and so they were able to basically impersonate the one password employee
and log into the system and that's when they started causing havoc.
Yeah.
And, and the good news is that they did, that 1Password did see what was happening.
Uh, they did see this weird session coming from an odd IP.
And they, they shut it down before basically all they did was
attempt to get a list of admins.
They didn't actually get the list of admins is what it looks like.
Um, but the, the whole thing was again, you know, it went back to
this, the fact that the, the hacker initially had compromised the.
The Okta support system, which they then got this support file.
And, you know, in retrospect, it looks like everybody did their job.
It, it looks like, right.
That, that, that.
Things were noticed, things were stopped.
I think that in the case of Okta, maybe they weren't noticed quick
enough because it actually, 1Password was just one company of several that
were compromised because of this.
It doesn't appear for those of you that are 1Password customers, it doesn't
appear that any customer data was accessed, doesn't appear, you know,
unlike the, what was the other one?
The LastPass, the LastPass hack where they actually got the vault.
And then you had to worry if you had insecure passwords, right?
That, that, that, that there were guessable, but it doesn't appear
that that was the case here.
They found it pretty quickly.
think another interesting thing that I found, by the way, this is an
article on The Register, we'll put a link in the show notes description.
Right.
The one thing I found interesting is they were trying to figure
out, okay, what actually happened?
Like, when was This information stolen and so they went back and they looked at
their logs on the Okta side and they were trying to figure out okay was the archive
access before the support engineer on the Okta side accessed it and they were able
to figure out no the support engineer hadn't opened it yet so it wasn't a
rogue support engineer on the Okta side.
And then they looked at the 1Pass site, 1Password site and they saw,
okay, the person who uploaded it was on a public Wi Fi at a hotel.
And they were like, oh, maybe it could have been stolen during the upload
because you know, those connections are always unencrypted and all the rest.
But they looked and they're like, no, the upload process
had TLS end to end up to Okta.
And so.
Everything was encrypted.
It wasn't stolen in the process.
So there must've been something else that had accessed it on the Okta side.
So it's interesting how they're able to piece this all together.
It's almost like CSI, right?
You're piecing together all these clues, trying to figure out, okay,
what happened, what went wrong?
Well, that's what you have to do in an incident response system, right?
You've got to, you know, piece together what you can from the logs that you have.
Logs, logs, logs, right?
That's what it's all about is the logs.
The, the, the only thing that's somewhat distressing is that it
appeared that they made some tweaks.
Some upgrades to their MFA.
Uh, and by upgrades, I mean like they changed some of their stuff.
They're like, maybe we shouldn't allow anyone to log into Okta who isn't
at a, um, a one password IP, right?
Maybe, maybe we shouldn't allow that.
That seems like
Uh, that's,
you should have decided that before, but yeah,
know about that, Curtis, because there are times, like, you might
be traveling, and you might be the super user, and you have to change
something, and you're not at a desk.
Or, imagine that one password blows up, something happens to their site,
and they need to reset passwords, and they could eventually get locked out.
But maybe this is an
I would think, yeah, I would, uh, I would think that that would be an
exception case that you, you would deal with that maybe by the default
would be that you would turn it off.
The other thing was that it appears that now they're using Yubikeys.
For those that aren't familiar with that, Y U B I K E Y.
This is a hardware token.
It's pretty inexpensive as these systems go.
But it is a a hard, you know, it's it's an actual system that you can actually
buy for your own personal use I took a look and it was I think it was a
hundred dollars for two of them Um,
you want to.
you know, and you want two.
Yeah, and so, um, then Uh, and, you know, and so it looks like they're
now using YubiKeys where maybe before they weren't using YubiKeys.
I'm glad that they made those steps.
It's just, it's just, it's just a shame that we always make changes to
systems after we've been, you know, after we've had an exposure, but good
on them, good on them for finding it good on them for responding and
saying, Hey, we could make this better.
Uh, I'm a little dinging Okta here.
There was a thing at the end that Okta basically said that perhaps if you're
sending a support file, uh, what was the, what was the type of that file?
A HAR file?
Um, some sort of archive file.
Yeah.
And they're like, perhaps you shouldn't, perhaps you should sanitize
that before you send it to us.
And I'm like, well, maybe your system that creates the HAR file
should sanitize it for the customer.
Before you upload it.
Uh, there was a little bit I thought of Victor blaming, uh, towards the end there,
but, um, anyway, a, a, a much, I think one that's gonna be a, a lot easier to
talk about our former employer, Druva.
Uh, there, the headline here, another, it's a tech target article
that they've added a gen AI assistant to their cloud backup tool.
And I watched a, like a video of a demo and it basically, it looked like, uh,
you know, an NLM that you can use to interface with the Druva cloud platform.
And so you could ask it things like, Hey, show me my backups, like in, in like, Hey,
show me my backup backups have failed.
I think the big thing is, especially with, uh, with the sort of focus on
AI and making it easily available to a large audience without needing
to know all the training, Right.
And being an expert in it, I think has now made it sort of commonplace, right?
It's easy to pick up an AI model that has been pre trained
and use it for some purpose.
right?
In their case, they're using, uh, Amazon's, uh, no, no.
Great surprise there.
They're using Amazon's, uh, NLM model,
you referring to?
Sorry, are you referring to NLP or LLM?
You're calling it NLM.
thank you.
Oh, yeah, sorry.
NLP Thank you.
NL NLP.
So they're, they're using Amazon's, uh, product to do this.
No.
Great surprise.
DVA lives in Amazon, and, uh, it, it seems like the biggest thing that
they had to do was to make sure that it was integrated with the security.
That's the biggest thing that I see is, yeah, you have to make sure because if
you just pick up any random LLM, I'll use LLM because that's a large language
model, right, which a lot of the AI models are built on top of, if you.
Pick that and depending on what is trained, you might get back bogus
answers, random answers, answers that aren't even safe, right?
And so at least the fact that Druva is putting guardrails in place to make
sure that what gets returned is sensible and safe, I think is a good step.
Yeah.
So we've seen, uh, we've seen AI use with Alcion.
We saw it with, um, I'm trying to think.
Yeah.
Cohesity came out with one.
Do you remember who else?
It was Druva.
I'm trying to, I know there's another one.
Um,
Was it Commvault?
in the thing.
Oh, Dell, Dell, Dell said, yeah, I haven't, I don't think I've seen
anything from Commvault, but, uh, Dell is also doing, uh, yeah, I merged,
I think, in, uh, uh, uh, NLP and LLM
To NLM.
It's, it's a Curtis only thing.
So, yeah, so that's interesting.
Uh, I think anything that makes it easier to interface with your
backup system is a good thing.
As long as they have those guardrails in place and
And I know you always like to talk about how, what's the job, what's the, one of
the most important jobs that you give to the most junior person at a company?
exactly.
Backup, right?
And so,
why would you do
yeah, and so giving an assistant, if you will, right, to that junior backup person
is helpful as they're learning the ropes.
Yeah.
Well, that is the news of the week.
As you know, every episode of the Backup Wrap Up is going to dive
deep into one particular topic.
This week's topic is snapshot, snapshot, snapshot.
Click, click, click,
It's a topic that comes up a lot.
This word comes up a lot on this show.
So it's time to dive deep into a world that I, I bet at one point in your career.
Persona, you must've heard this.
Word a hundred times a day.
What do you think?
At least.
At least,
least.
because there's certainly one company that thinks they do snapshots different
and better than everyone else.
And they're probably, they certainly, it's certainly at one point in time.
That was true.
I think a number of other vendors are now doing snapshots
the way NetApp did snapshots.
So just a quick story just sort of bring why different ways
to do snapshots really matter.
I was at a, I'm in my brain, uh, live translating the story because I
was at one of the largest companies in the world at a consulting gig.
And we were helping them to pick a new storage and backup system.
They were looking for an integrated system that would do both storage and
data protection as part of that storage.
They were already a NetApp customer and they knew that that meant that
they knew every foible of NetApp.
So, uh, they knew every bad thing about NetApp, but they
also knew the good things.
And I think of all of the consulting gigs that I've done throughout the years,
they had done the best at presenting.
These are our requirements.
And they are well defined and there are reasons behind every one of them.
Mm hmm.
And one of their requirements was that they wanted end users, just regular Joe
and Jane person sitting in the desktop.
To be able to do their own resource.
Right.
And they wanted them to have the ability to do that at points in
time that were like an hour at a time throughout the day, going back
to, you know, it was like 90 days.
So specifically they said, this is why we want 90 days of user browsable snapshots.
At the time that was like.
Not everybody did that, right?
Depending on how you did snapshots as a vendor, you either could or
could not meet that requirement.
Just end of story.
And so, the, the, the responses ranged from, uh, no problem, right?
Literally no problem.
To, I remember one vendor coming in and going, That's the
dumbest requirement we've ever
I bet I can
you want 90 days of user browsable snapshots?
Yeah, I think you know exactly who that was.
It was just chaos and by the way There was a vendor that came in and they had
this let me restate that I'm pretty sure it was that vendor that had this
sort of Very convoluted system that was based on like you had this block system
and then you had this other system that had the blocks and you could,
Stephen
it was just really, really complicated.
And they had this like this wizard of a presenter that
was just an amazing presenter.
That was very, um, you know, charming and very smart and
just presented all of the stuff.
Uh, and, and even though that presenter did their best, the, the
customer just was not having it.
And even though he was a really great presenter.
Just didn't play.
Anyway, the thing is that the key there is that how you do snapshots It very much
dictates how everything's going to work.
Going back to the requirements, right?
You said that they had a very clear list of things.
Do you know why they had that requirement for 90 days?
Was it to like?
What was the purpose behind having that?
Because I'm sure today everyone kind of thinks oh, that's just reasonable, right?
Oh, of course, why don't I have that?
But back then it seemed like that was something very,
they had very specific numbers on the number of restores that had been done.
And I know this as a backup person, but if you look at just
all of the restores that are done.
Ever anywhere, you know, for the history of restores, 99 percent of
them are done from data from yesterday.
Right.
And then, or, or from the most recent snapshot, and then there is this cliff.
of usage that just gets smaller and smaller.
And it's just this incredibly ever increasing line to zero.
And I think in their world, what they showed was that ever increasing line to
zero basically dropped off at 90 days.
And so they said, we need 90 days of user browsable snapshots.
That's what I meant was that they were really good at articulating what
their requirements were and also.
Why those requirements?
So like, you
yeah.
And I think that's a good lesson though for backup admins, right?
You should be looking at the metrics of these systems because if you want
to make a case for say new technologies or new process improvements or other
things like that, having this data to show why you need something so you could
put it into requirements is critical.
exactly, exactly.
So let's define
What's a
what we mean.
Yeah.
What is a snapshot?
Let me give you my definition, let's see how closely it
lines with what you call it.
Alright.
What's your definition
of a
my definition of a snapshot is a point in time copy of the data that existed
at some point in time, so it has to have been plausible, that can be
preserved such that it's not modified when the primary copy gets modified.
So, I would take your definition and I would insert one word
I think to make it perfect
Okay?
and that is the word virtual at the beginning because It's a virtual
copy, because that is really what differentiates a snapshot from a copy,
because you just said a copy, right?
So it is a, I like to use the word view.
It's a view.
Into your volume that you're protecting with the snapshot
at a different point in time.
And I like the word view because it, it, which is very much a database term, right?
It's just a different, it's a way to look at your current volume
at a different point in time.
The, I think the most important thing that differentiates a true snapshot from a lot
of other things that we call snapshots.
Is that it is a virtual copy in that relies on the primary volume that it is
protecting for most of the blocks of data.
The bulk of the blocks, when you're reading that snapshot, the bulk
of those blocks are going to come from the, the current volume.
Are you with me?
Right.
That basically that the change data is going to come from
snapshot.
some, the snapshot, right, how that happens, that's the difference between
copy on write and redirect on write, but the bulk of the data is going
to come from the current volume.
That's basically what I'm
Okay.
I
we on the same page?
Okay.
All right.
So we can go, I mean, only one of us actually worked at a vendor that did
snapshot, so, you know, want to make sure I'm getting things right here.
Um, this is really the key of the difference between a snapshot and a
copy or a snapshot and a backup why does that matter from a backup perspective?
Why does that
snapshots.
They're not independent.
And that's why, you know, those of us that, you know, care about
things like backup and recovery.
We just like to scream and say, snapshots are not backup.
Right.
Um, Now, I will say that you can use snapshots as a way to get backup, but a
snapshot by itself on a volume is, I like to call it a convenience copy, right?
It is a way to go back in time as long as you don't have media failure.
Right, right.
You don't have a double disk failure in a RAID 5 array or a triple
disk failure in a RAID 6 array.
Yeah, and like you're saying, it could use a snapshot to allow you to do other backup
mechanisms like take a copy, preserve it in a point of time, now you move that
data and you back up that data, right?
Which, if you're integrating with applications, you can now take an
application consistent point in time while your database is in hot
backup mode, take that snapshot, now you preserve that point in time.
You can.
Uh, Tha, the database, so it can continue operating like normal, Tha, the database.
What word are you saying there?
I'll follow.
I just, I don't know what word I thought you were saying there.
So basically you're saying, because you freeze it.
You're saying you freeze it and now you thaw it.
Okay, all
Yeah, or you could quiesce and unquiesce.
Yeah.
Okay.
I just, I don't usually use that term.
So it really threw me.
right.
And then you do your backup off of that snapshot, right?
So you now have a copy that's frozen that you can now do your
backup and it's all good to go.
Right.
Um, I'd say the most common outside of storage arrays, the most common snapshot
that is used in that way is VSS, right?
The Windows Volume Shadow Services.
And it, it's basically integrated into the operating system.
It's integrated into the applications.
It is.
And backup apps can integrate with VSS.
These are all done with APIs and a backup app can show up and say, Hey,
I am here to do a backup of this box.
Um, please, I'm going to really simplify it.
Please take a snapshot of everything that needs to have a snapshot taken of
it before I take a backup, then you take a backup and then they, um, and it can
take a backup of that snapshot, even though the volume continues to change.
It it's given this view into the volume that is static.
And then it can back up that volume, uh, and get that perfectly application
consistent version of the volume.
Uh, even if the backup takes two hours, it doesn't matter.
It has it, all of the blocks will be from the same exact point in time.
And then when it's done, it can tell VSS to delete that snapshot
or it can keep it around.
It's up to you.
It's just, it's a configuration thing.
One thing I do want to mention.
That, like we said, Snapshots just gives you that point in time copy, right?
It's a read only, point in time copy.
Now sometimes you will also hear, and I don't think we're covering
it later, clones being used, right?
Where I take a virtual copy of the volume and start using it, just going
back to the previous discussion, Curtis.
The difference is clones are writable.
So you're making changes to it.
Right, a snapshot is a read only copy that you preserve that point in time,
nothing's gonna change it, and it's always there for you to go back, you
can pull your file, your data out of it.
Clones, on the other hand, give you a copy of the volume at a point in time,
but it's so you can use it for some purpose, like for testing out restore.
Capabilities.
Can I verify my backups?
Those sort of things.
And also doing, like, database recovery against that copy, the clone
copy, and other things like that.
So, clones are different than snapshots, even though they both
might start from a snapshot copy.
Right.
Um, yeah, I, I would probably just call it a read write snapshot, but
maybe that's a contradiction in terms.
Um,
Yes, a snapshot is a point in time, Curtis.
yeah, exactly.
THere are three ways that snapshots are created.
The most common way, I, would you, we say it's still the most common
way, the copy on write method?
Uh, no, I don't think, I don't think so anymore.
Okay.
Well, historically what used to be the most common method before
one vendor ruined it for everybody
Made something
is.
So is called the copy on write method.
And the reason it's called the copy on write method is that we create this,
this storage area that is going to hold the, um, the, the snapshot blocks.
And when we go to update a block, because we're, it's a storage volume, right?
So we're going to update a block and we say, Hey, Uh, there's
a snapshot for this block.
We're going to copy that block out to the snapshot area before you write.
So that's why it's called copy on write.
And that is very expensive because if you think about it, there are, let me count.
There, there, there is a read.
And a right for every right.
There's, there's a says,
and it's not just the one read and write that happens on the snapshot side.
You also have a bunch of metadata and updating indirect blocks and
a whole bunch of other things.
So, yeah, doing a copy on write might lead to, say, 10 additional I.
O.
operations or 12.
Yeah.
for that one block.
And so what happens is that over time that the number of blocks, remember
when I initially, I mentioned that the bulk of the data is going to come from
the primary volume, but over time.
As a snapshot has been created, more and more blocks are going to be copied into
that snapshot area, and which means that at some point, you know, a significant
portion of my snapshot, if I'm.
If I'm reading it, a significant portion is going to come from the snapshot area.
And so it, there are multiple reasons that there's a performance hit.
I think the biggest performance hit is.
You know, you talk about all of the IOs that have to happen every time we update,
uh, you know, we do a copy on write.
The other is that as time goes on, the more and more data that I have to get
from my snapshot area when I'm doing a read, the performance goes down.
And this goes back to that vendor that they basically suggested that if we had
90 days of user browsable snapshots, That their performance was going to be
like half of what, uh, what it typically
and I would say that is Probably based on older technology, Curtis.
I think when you had traditional RAID arrays or RAID Groups
that you were creating.
I think there was more of a performance impact I think now since you end
up aggregating a bunch of disks and then carving out volumes And so
you can share in the performance.
I think That's not as big of a concern anymore as it used to be.
Like I know those systems you're talking about, they now support
a thousand snapshots, right, for
so you think that the main hit from a performance standpoint on a copy
on write snapshot scenario in modern technology is mainly that IO hit the
first time you go to do a write when every time, every time you update a write.
And by the way, remember that It has to do that for every, that's sort of
calculate every time it does it right.
It has to calculate, is there a snapshot that is looking at this
block as it exists at this point in time, and then you have to copy it,
uh, for that, for that snapshot.
Which, there are different mechanisms you could use.
You could use bitmaps, you could use other things.
So, it's not the end of the world, and I think that a lot of these storage vendors
have optimized if they are continuing to use copy on write technologies.
But I would say a good chunk of them have moved away from
copy on write because of the I.
O.
penalties that we've talked
right, right.
So the, the, there was this vendor that came out, a little vendor
called, at one point it used to be called Network Appliance.
And it had a little screw and bolt as its logo.
right.
Um, the, the, at one point they said, we're just going to change our
name to NetApp because that's what
That's what everyone calls them.
I remember actually working at a startup and got my hands on my
first NetApp appliance and was like, yeah, wow, these are kind of cool.
And this was before I started working there.
And I was like, wow, this is really amazing and simple
for what it does and easy to
Yeah, exactly.
And they used a completely different way to, um, to do snapshots that at the time
was revolutionary, which I think has now been adopted by a lot of storage vendors.
And that is, we call it redirect on write.
Do you want to describe how that
works?
so what read so before you get there?
I think we need to talk about their right anywhere file layout
which allows then the snapshot.
Yep waffle Which a lot of other vendors I think do something similar as well these
days but what it is is going back to the copy on write example If you are writing
a block of data in copy on write sort of file system, previous file systems,
you would always write to the same spot.
And because you're always writing to the same spot, that's why you have to first
copy out the data and then update it.
With a write anywhere file layout, what you end up doing is, it doesn't
matter which actual block you end up writing to, you basically construct the
metadata tree to reference that block.
Even though I might be updating block 100, block 100 might actually
physically be at, like, block 1000.
And because I have all my metadata that tells me exactly where that data
exists, I just need to update the metadata to say, okay, if someone
tries to access block 100, it's actually physically on block 1000.
And so you can end up writing to any location in the file system
and not having to worry about always hitting that same location.
And so that's kind of waffle in a nutshell.
So basically what you then is you have this metadata system that has a pointer
to every block and it doesn't really matter where those blocks happen to be.
And you can have thousands of these snapshots, right?
These pointers at the top.
right, so then when we go to do an update and we have a snapshot, it just
means that what we're going to do is we're going to change the pointer.
We're going to say, okay, this, there's this block that's sitting here.
And we know we, we, we're not supposed to update that block because we
have a snapshot that's requiring on that, that's requiring that block.
And then we're going to just redirect.
We're going to, we're going to write a new block for the
new version of that old block.
And then we're going to change the pointer, right?
We're going to redirect the pointer.
To this new location of that block.
Meanwhile, the old block is still sitting there and we've got a
snapshot that's just pointing to it.
Right.
Um, and so you've got this infinite number of snapshots that are pointing
to an infinite number of, well, it's not an infinite number of snapshots, but
you have a very high number of snapshots that are, that, that, and they're all.
Just a whole bunch of metadata pointing to a whole bunch of blocks that are
just sitting all around the volume.
And the one thing, so this all sounds amazing, right?
Because you're like, Oh, writes are, writes are super fast.
I don't have to worry about it.
In fact, Curtis, just one correction.
When you're going to actually write a new block, you never actually have to
look up the old block because you're always writing to a new location.
So you don't care if that old block is occupied by a snapshot or not, right?
Because this is the downside though, of snapshots, especially with the
write anywhere file layout is.
You now need a process to go through and say when a snapshot gets deleted,
what blocks are no longer being used actively because they don't
belong to snapshots, they're not currently as part of the volume.
So you have a garbage reclamation process or different vendors call it
different things, but some process to go through and reclaim all of those
free blocks so they can be reused.
I think you said the same thing I said in different words, but
yeah, I see what you're saying.
I guess I was saying that, that it's making a decision on what to do.
Based on whether or not, I think if I could just change my part
of my answer, instead of, is this block being used by anything else?
I guess is,
Yeah, more
I guess that's the question I'm asking.
Yeah.
Uh, and actually, I guess what you're saying is it actually doesn't even make
that decision at that point in time.
If it's going to modify a new block, if it's going to modify a
block, it just writes a new block.
And then what happens to that old block is a completely separate process.
If there's, if there is a snapshot that's pointing to that, that
old block, then it will stay.
If there are no snapshots that are pointing to that block, then at some
point the garbage collection process will come and make it go away.
Is that a better, is that a better
Yes, that is a better description.
Now this is
don't want to argue with a former NetApp employee
about how NetApps
now I should say this is based on NetApp's technology, different
vendors may do different things, but for the most part, most of the
vendors do something similar ish.
Specifically, it's based on how NetApp worked when you worked
there, which was a while ago.
And things may be different now,
that is also true.
not.
I mean, Waffle is like at the core of their...
You know, the core of their technology.
And so, and that's why with redirect on right, that's why you could essentially
have an infinite number of snapshots with, with zero performance penalty,
you have zero performance penalty of having basically the performance
penalty of doing a right update.
Is the same whether you have a snapshot or you don't have a snapshot.
The, the only penalty, if you want to call it that, is that, um, one, one
big difference between this way and the other way is there is no snapshot area.
Right.
So in the other method, the snapshot area could fill up if
you held snapshots for too long.
In this configuration, there is no snapshot area.
The snapshot area is the volume, and if you have too many updates and you
keep too many snapshots, you would fill up the volume with snapshots.
So you've gotta get rid of the older
Well, and this is where That would apply if you're talking NetApp terminology
traditional volumes, but most of it has moved over to virtual volumes, where once
again, you have an aggregate, a shared pool of common data, and for each of the
volumes, typically you also set a limit on how much space a snapshot can occupy.
So you could say, I am allowing 20 percent for snapshots of my overall
volume capacity, in which case it'll start
And what happens when you hit that wall?
I, I don't know what the current behavior is.
Previously.
I believe it would like let you like start automatically pruning
snapshots and trying to free up space.
Right, right.
Because it, obviously, it's not going to, it's not going to prune, uh,
production, you know, current data.
So, yeah, I, it could, but again, we're, we're talking specifically NetApp,
but something has to happen, right?
If you're using this method.
Something has to happen.
Either we have to stop creating new snapshots, right?
Or stop updating the snapshots that we have.
And, uh, we need to delete older snapshots or we need to maybe delete, you know,
certain ones in the middle, right?
Basically you've got to do some kind of pruning or else you're going to
Yeah, the other challenge is also figuring out what snapshot to delete
because blocks are being shared, right?
You might be like, hey, this snapshot is huge and you go delete it, but
because those blocks are being shared by other snapshots, you're not actually
going to free any space, right?
So you need to be able to figure out like which snapshot actually
contains unique blocks that if I delete it will actually save me space.
complicated.
Storage management.
I
I don't miss production storage management.
Any other final thoughts on redirect on, right?
think that covers it
I mean, My personal, if you're going to do snapshots on a storage array, I
think redirect on write is the way to go.
It sounds like what you're saying, they've made copy on write better,
but I still think redirect on write is just significantly better.
So, um, but it might be more complicated than if you're coding it.
Right.
So the next one is what I'm going to call the dumbest of all snapshot methods.
That's not what I have in the book.
I gave it a much nicer name in the book.
And guess who does this method?
The leading hypervisor company in the world.
Yes.
I think that's a fair statement, right?
company in the world.
I think, I think it still is.
Yeah.
And that would be VMware.
So the way VMware does snapshots is just literally the Dumbest
implementation of snapshots that I've ever seen and I don't know how they
haven't addressed it, but here it is.
When you create a snapshot in VMware, it literally holds all the rights.
Now, by the way, if I'm wrong, by the way, you know, Broadcom, don't sue me.
This is based, this is based on my understanding of VMware snapshots.
Uh, you know, I've, I've checked every once in a while and they,
no one seems bothered by this.
Uh, but if this has changed, any of you that are, you know, if anybody
works for Broadcom slash VMware, then, you know, feel free to update
me and I will update this episode.
And I'll just delete this section.
But here's the way it works.
When you create a single snapshot on a VMware volume, it halts.
All rights on the, on the current volume.
And then it keeps all rights in a snapshot area.
And then when you delete that snapshot, it replays all those
rights against the production volume.
And this is why when you make a snapshot.
And then you, if you hold that snapshot for a long time and then
you delete that snapshot, this is why it has a big performance hit
against the production volume.
But no one
this is
snapshot of a VM
yeah, this is why you do not do this.
You don't use snapshots on VMware level snapshots the way you do any other
snapshots, because, and by the way, I used VMware for years before knowing this.
That's why I want to make sure I mentioned it.
And, and that is that if you create a snapshot and then hold it for a
long period of time, you're going to get hit with a massive IO hit
when you delete that snapshot.
So if you're using VMware, VMware level snapshots, then you use them the way we
talked about earlier, where you create a snapshot, you make a, you make a backup.
And then you delete the snapshot.
Maybe you take a VMware level snapshot, and then you take a storage level
snapshot of that snapshot, and then you delete the, the VMware level snapshot.
You should, if this is the way your snapshot system works, you
cannot leave the snapshots around for any significant period of time.
I was going to chime in.
Thank you for covering that.
The, this specifically is VM where software snapshots,
if you wanna call it that.
Right, that are only done at the VMware level.
Now, there are integrations that various storage vendors offer
by plugging into the VMware API.
So whenever you trigger a VMware snapshot, it actually triggers
a storage level snapshot.
So avoiding some of these issues, but not everyone is aware of it.
Not everyone is using a third party storage array that integrates with VMware.
So.
Just
Yeah.
So, right.
Thanks for, thanks for clarifying that.
This is specifically VMware level snapshots that are done by VMware.
And without any third party storage.
Yeah.
And I don't know why VMware did this, but it's bonkers.
It's just literally one of the weirdest, codest thing, weirdest
coded things I've ever heard.
Why would you do it that way?
Somewhere in a meeting, this is how they decided to
it.
was probably easier
and yeah, yeah, maybe it was easier,
and they didn't talk to the
I wonder about that.
They didn't, they exactly, they did not talk to the backup folks.
Well, uh, I think we have, uh, summarized the world of snapshots.
That?
No, I think we did a good job with that.
So, copy on write, redirect on write, dumbest method ever.
Those are the three types.
I've got it officially in the book, uh, I've got this labeled
as the hold all writes method.
Uh, I should really just change that to the dumbest method ever.
But, um, yeah.
So, you know, snapshots are a great tool.
In the backup and recovery arsenal.
They are the great sort of basis upon which we're going to talk about one
of my favorite ways to do backup.
And we're going to talk about that in another episode.
Hint, it's called near CDP, not CDP.
It's called near CDP.
And, uh, it's just, just the number one thing you have to understand
about snapshots is that unless you have copied this snapshot to
another location via some mechanism.
Which could be backup.
It could be replication of the volume.
It could be a number of things.
You do not have a backup.
You have a picture of your volume.
And that picture of your volume is as worthless as a picture of your
house after your house burns down.
It'll just be a nice memory and, uh, and a really bad day.
So it's just, that's the really, the most important thing to
understand about snapshots.
And now if this is your first time, snapshots have been explained to you.
Now you understand why I don't like it that they call.
What AWS does snapshots because that very much does not meet
the definition that we just had.
And I'm glad I brought this up because it's important to what we're talking
about is storage level snapshots.
Darn it.
I don't know
You can't call it that, yeah, because
this.
Yeah, these are traditional snapshots.
There are other things out there that people call snapshots
that don't work like this.
AWS snapshots don't work like this.
Uh, they are an actual image copy.
It's actually, they actually, when you make an AWS snapshot, it actually copies
that, that point in time out to another area of storage, which happens to be S3,
you?
And I think specifically you're talking about an AWS EBS snapshot
thank you.
I am talking about an AWS EBS snapshot.
Um, my former employer, Druva, they call what they do snapshots.
They call their backups snapshots.
I never liked that, but you know, nobody asked me.
Uh, so, but what we're talking about here is traditional snapshots.
And, um, a lot of other people will call what they do a snapshot.
Um, the problem is like a lot of terms in the, in the backup world.
It's a term like so many of our terms are, um, their words that are used
just, they're just English words that are used in so many different contexts.
when, like when we had the CDP episode, we couldn't figure out what to call
those point in time, because a lot of the CDP vendors called them snapshots.
Yeah, exactly.
All right.
Well, uh, I guess the only thing left for me to say is that's a wrap