I'm not sure who needs to hear this, but snapshots are not backups.

And in this context, I'm talking about virtual snapshots, like

what you do with the storage re.

We're not talking about cloud snapshots, like what you do

with EBS volumes in the AWS.

If you don't know the difference or you can't articulate why

these snapshots are not backups.

Well, have I got a podcast for you?

Hi, I'm W.

Curtis Preston AKA Mister backup.

And my podcast turns unappreciated backup admins into cyber recovery heroes.

This is the backup wrap-up.

Hi, and welcome to the Backup Wrap Up.

I'm your host, W.

Curtis Preston, and I have with me my Spanish language

encourager, Prasanna Malaiyandi.

I gave you a positive thing

I know, I'm impressed.

And I'm also impressed that you are done now with, what, Spanish 2, right?

Spanish 2, uh, Sisi termine con Span, uh, I almost said

Spanish, Espanol 2, um, yeah.

So now I need to, I need to do a little review because there's, it's, it's

really, you know, for a, for an English speaker, uh, the challenges, as I know,

I've spoken to you, the challenges are things like, we're like, it, it, the,

the gender thing is, is not a big deal.

You get used to that.

What's challenging is words that n an ma like PMA you would

think that would be feminine, but it's actually a masculine word.

Uh, because it, it ends in an, uh, ma.

I have to pound that stuff into my head and just say it over and over.

I will be using, the thing that I made where I, I'm actually

using technology to make a verbal recording that I can listen to.

Uh, it's very, very cool.

Uh, but I am excited to move on to Spanish three and.

Bring you dragging with me.

It's helpful.

I, Used to be pretty good at Spanish because I took it in high

school and then I lost everything.

I can order things off a menu, but that's about it.

Right.

Right.

So, uh, I think it's time for the news of the week.

this one is frustrating.

Uh, we'll, we'll, we'll do a frustrating one first and then

some good news about a vendor.

Let's talk about this one password hack.

And it frustrates me for many reasons.

One is, we're such a fan of password managers.

And there are those who are not fans of password managers.

There are also those who are not a fan of the cloud.

And this just, uh, is both of those things.

So, uh, do you want to talk a little bit about the 1Password slash Octahack?

so basically what happened is 1Password, I don't know if it was

1Password or Okta, but basically some hackers got in into 1Password.

They were able to change some things in their Okta instance and try to get a list

of admins, which I'm sure they were going to use to target and be able to deploy

things so they could cause bigger damage.

Right.

But as they were starting to sort of peel back the onion and

figure out, okay, what happened?

They realized what actually happened is it started on the Okta side,

Right.

The reason they were able to do what, what they ended up being able

to do was because Okta had already been, had already been hacked.

And so it's interesting in this case because It wasn't like, oh, they

just got into Okta and then they ping ponged and got into 1Password.

They actually looked at support files that someone in 1Password, an engineer,

had uploaded because they probably had some issue or something else like that.

They were reaching out to Okta and they uploaded basically a support bundle.

Um, and in that support bundle, they contained, in addition to sort of like

information needed to troubleshoot, it also contained session cookies,

Right.

Which they were then able to use.

Right.

Yeah, and so they were able to basically impersonate the one password employee

and log into the system and that's when they started causing havoc.

Yeah.

And, and the good news is that they did, that 1Password did see what was happening.

Uh, they did see this weird session coming from an odd IP.

And they, they shut it down before basically all they did was

attempt to get a list of admins.

They didn't actually get the list of admins is what it looks like.

Um, but the, the whole thing was again, you know, it went back to

this, the fact that the, the hacker initially had compromised the.

The Okta support system, which they then got this support file.

And, you know, in retrospect, it looks like everybody did their job.

It, it looks like, right.

That, that, that.

Things were noticed, things were stopped.

I think that in the case of Okta, maybe they weren't noticed quick

enough because it actually, 1Password was just one company of several that

were compromised because of this.

It doesn't appear for those of you that are 1Password customers, it doesn't

appear that any customer data was accessed, doesn't appear, you know,

unlike the, what was the other one?

The LastPass, the LastPass hack where they actually got the vault.

And then you had to worry if you had insecure passwords, right?

That, that, that, that there were guessable, but it doesn't appear

that that was the case here.

They found it pretty quickly.

think another interesting thing that I found, by the way, this is an

article on The Register, we'll put a link in the show notes description.

Right.

The one thing I found interesting is they were trying to figure

out, okay, what actually happened?

Like, when was This information stolen and so they went back and they looked at

their logs on the Okta side and they were trying to figure out okay was the archive

access before the support engineer on the Okta side accessed it and they were able

to figure out no the support engineer hadn't opened it yet so it wasn't a

rogue support engineer on the Okta side.

And then they looked at the 1Pass site, 1Password site and they saw,

okay, the person who uploaded it was on a public Wi Fi at a hotel.

And they were like, oh, maybe it could have been stolen during the upload

because you know, those connections are always unencrypted and all the rest.

But they looked and they're like, no, the upload process

had TLS end to end up to Okta.

And so.

Everything was encrypted.

It wasn't stolen in the process.

So there must've been something else that had accessed it on the Okta side.

So it's interesting how they're able to piece this all together.

It's almost like CSI, right?

You're piecing together all these clues, trying to figure out, okay,

what happened, what went wrong?

Well, that's what you have to do in an incident response system, right?

You've got to, you know, piece together what you can from the logs that you have.

Logs, logs, logs, right?

That's what it's all about is the logs.

The, the, the only thing that's somewhat distressing is that it

appeared that they made some tweaks.

Some upgrades to their MFA.

Uh, and by upgrades, I mean like they changed some of their stuff.

They're like, maybe we shouldn't allow anyone to log into Okta who isn't

at a, um, a one password IP, right?

Maybe, maybe we shouldn't allow that.

That seems like

Uh, that's,

you should have decided that before, but yeah,

know about that, Curtis, because there are times, like, you might

be traveling, and you might be the super user, and you have to change

something, and you're not at a desk.

Or, imagine that one password blows up, something happens to their site,

and they need to reset passwords, and they could eventually get locked out.

But maybe this is an

I would think, yeah, I would, uh, I would think that that would be an

exception case that you, you would deal with that maybe by the default

would be that you would turn it off.

The other thing was that it appears that now they're using Yubikeys.

For those that aren't familiar with that, Y U B I K E Y.

This is a hardware token.

It's pretty inexpensive as these systems go.

But it is a a hard, you know, it's it's an actual system that you can actually

buy for your own personal use I took a look and it was I think it was a

hundred dollars for two of them Um,

you want to.

you know, and you want two.

Yeah, and so, um, then Uh, and, you know, and so it looks like they're

now using YubiKeys where maybe before they weren't using YubiKeys.

I'm glad that they made those steps.

It's just, it's just, it's just a shame that we always make changes to

systems after we've been, you know, after we've had an exposure, but good

on them, good on them for finding it good on them for responding and

saying, Hey, we could make this better.

Uh, I'm a little dinging Okta here.

There was a thing at the end that Okta basically said that perhaps if you're

sending a support file, uh, what was the, what was the type of that file?

A HAR file?

Um, some sort of archive file.

Yeah.

Yeah.

And they're like, perhaps you shouldn't, perhaps you should sanitize

that before you send it to us.

And I'm like, well, maybe your system that creates the HAR file

should sanitize it for the customer.

Before you upload it.

Uh, there was a little bit I thought of Victor blaming, uh, towards the end there,

but, um, anyway, a, a, a much, I think one that's gonna be a, a lot easier to

talk about our former employer, Druva.

Uh, there, the headline here, another, it's a tech target article

that they've added a gen AI assistant to their cloud backup tool.

And I watched a, like a video of a demo and it basically, it looked like, uh,

you know, an NLM that you can use to interface with the Druva cloud platform.

And so you could ask it things like, Hey, show me my backups, like in, in like, Hey,

show me my backup backups have failed.

I think the big thing is, especially with, uh, with the sort of focus on

AI and making it easily available to a large audience without needing

to know all the training, Right.

And being an expert in it, I think has now made it sort of commonplace, right?

It's easy to pick up an AI model that has been pre trained

and use it for some purpose.

right?

In their case, they're using, uh, Amazon's, uh, no, no.

Great surprise there.

They're using Amazon's, uh, NLM model,

you referring to?

Sorry, are you referring to NLP or LLM?

You're calling it NLM.

thank you.

Oh, yeah, sorry.

NLP Thank you.

NL NLP.

So they're, they're using Amazon's, uh, product to do this.

No.

Great surprise.

DVA lives in Amazon, and, uh, it, it seems like the biggest thing that

they had to do was to make sure that it was integrated with the security.

That's the biggest thing that I see is, yeah, you have to make sure because if

you just pick up any random LLM, I'll use LLM because that's a large language

model, right, which a lot of the AI models are built on top of, if you.

Pick that and depending on what is trained, you might get back bogus

answers, random answers, answers that aren't even safe, right?

And so at least the fact that Druva is putting guardrails in place to make

sure that what gets returned is sensible and safe, I think is a good step.

Yeah.

So we've seen, uh, we've seen AI use with Alcion.

We saw it with, um, I'm trying to think.

Yeah.

Cohesity came out with one.

Do you remember who else?

It was Druva.

I'm trying to, I know there's another one.

Um,

Was it Commvault?

in the thing.

Oh, Dell, Dell, Dell said, yeah, I haven't, I don't think I've seen

anything from Commvault, but, uh, Dell is also doing, uh, yeah, I merged,

I think, in, uh, uh, uh, NLP and LLM

To NLM.

It's, it's a Curtis only thing.

So, yeah, so that's interesting.

Uh, I think anything that makes it easier to interface with your

backup system is a good thing.

As long as they have those guardrails in place and

And I know you always like to talk about how, what's the job, what's the, one of

the most important jobs that you give to the most junior person at a company?

exactly.

Backup, right?

And so,

why would you do

yeah, and so giving an assistant, if you will, right, to that junior backup person

is helpful as they're learning the ropes.

Yeah.

Well, that is the news of the week.

As you know, every episode of the Backup Wrap Up is going to dive

deep into one particular topic.

This week's topic is snapshot, snapshot, snapshot.

Click, click, click,

It's a topic that comes up a lot.

This word comes up a lot on this show.

So it's time to dive deep into a world that I, I bet at one point in your career.

Persona, you must've heard this.

Word a hundred times a day.

What do you think?

At least.

At least,

least.

because there's certainly one company that thinks they do snapshots different

and better than everyone else.

And they're probably, they certainly, it's certainly at one point in time.

That was true.

I think a number of other vendors are now doing snapshots

the way NetApp did snapshots.

So just a quick story just sort of bring why different ways

to do snapshots really matter.

I was at a, I'm in my brain, uh, live translating the story because I

was at one of the largest companies in the world at a consulting gig.

And we were helping them to pick a new storage and backup system.

They were looking for an integrated system that would do both storage and

data protection as part of that storage.

They were already a NetApp customer and they knew that that meant that

they knew every foible of NetApp.

So, uh, they knew every bad thing about NetApp, but they

also knew the good things.

And I think of all of the consulting gigs that I've done throughout the years,

they had done the best at presenting.

These are our requirements.

And they are well defined and there are reasons behind every one of them.

Mm hmm.

And one of their requirements was that they wanted end users, just regular Joe

and Jane person sitting in the desktop.

To be able to do their own resource.

Right.

And they wanted them to have the ability to do that at points in

time that were like an hour at a time throughout the day, going back

to, you know, it was like 90 days.

So specifically they said, this is why we want 90 days of user browsable snapshots.

At the time that was like.

Not everybody did that, right?

Depending on how you did snapshots as a vendor, you either could or

could not meet that requirement.

Just end of story.

And so, the, the, the responses ranged from, uh, no problem, right?

Literally no problem.

To, I remember one vendor coming in and going, That's the

dumbest requirement we've ever

I bet I can

you want 90 days of user browsable snapshots?

Yeah, I think you know exactly who that was.

It was just chaos and by the way There was a vendor that came in and they had

this let me restate that I'm pretty sure it was that vendor that had this

sort of Very convoluted system that was based on like you had this block system

and then you had this other system that had the blocks and you could,

Stephen

it was just really, really complicated.

And they had this like this wizard of a presenter that

was just an amazing presenter.

That was very, um, you know, charming and very smart and

just presented all of the stuff.

Uh, and, and even though that presenter did their best, the, the

customer just was not having it.

And even though he was a really great presenter.

Just didn't play.

Anyway, the thing is that the key there is that how you do snapshots It very much

dictates how everything's going to work.

Going back to the requirements, right?

You said that they had a very clear list of things.

Do you know why they had that requirement for 90 days?

Was it to like?

What was the purpose behind having that?

Because I'm sure today everyone kind of thinks oh, that's just reasonable, right?

Oh, of course, why don't I have that?

But back then it seemed like that was something very,

they had very specific numbers on the number of restores that had been done.

And I know this as a backup person, but if you look at just

all of the restores that are done.

Ever anywhere, you know, for the history of restores, 99 percent of

them are done from data from yesterday.

Right.

And then, or, or from the most recent snapshot, and then there is this cliff.

of usage that just gets smaller and smaller.

And it's just this incredibly ever increasing line to zero.

And I think in their world, what they showed was that ever increasing line to

zero basically dropped off at 90 days.

And so they said, we need 90 days of user browsable snapshots.

That's what I meant was that they were really good at articulating what

their requirements were and also.

Why those requirements?

So like, you

yeah.

And I think that's a good lesson though for backup admins, right?

You should be looking at the metrics of these systems because if you want

to make a case for say new technologies or new process improvements or other

things like that, having this data to show why you need something so you could

put it into requirements is critical.

exactly, exactly.

So let's define

What's a

what we mean.

Yeah.

What is a snapshot?

Let me give you my definition, let's see how closely it

lines with what you call it.

Alright.

What's your definition

of a

my definition of a snapshot is a point in time copy of the data that existed

at some point in time, so it has to have been plausible, that can be

preserved such that it's not modified when the primary copy gets modified.

So, I would take your definition and I would insert one word

I think to make it perfect

Okay?

and that is the word virtual at the beginning because It's a virtual

copy, because that is really what differentiates a snapshot from a copy,

because you just said a copy, right?

So it is a, I like to use the word view.

It's a view.

Into your volume that you're protecting with the snapshot

at a different point in time.

And I like the word view because it, it, which is very much a database term, right?

It's just a different, it's a way to look at your current volume

at a different point in time.

The, I think the most important thing that differentiates a true snapshot from a lot

of other things that we call snapshots.

Is that it is a virtual copy in that relies on the primary volume that it is

protecting for most of the blocks of data.

The bulk of the blocks, when you're reading that snapshot, the bulk

of those blocks are going to come from the, the current volume.

Are you with me?

Right.

That basically that the change data is going to come from

snapshot.

some, the snapshot, right, how that happens, that's the difference between

copy on write and redirect on write, but the bulk of the data is going

to come from the current volume.

That's basically what I'm

Okay.

I

we on the same page?

Okay.

All right.

So we can go, I mean, only one of us actually worked at a vendor that did

snapshot, so, you know, want to make sure I'm getting things right here.

Um, this is really the key of the difference between a snapshot and a

copy or a snapshot and a backup why does that matter from a backup perspective?

Why does that

snapshots.

They're not independent.

And that's why, you know, those of us that, you know, care about

things like backup and recovery.

We just like to scream and say, snapshots are not backup.

Right.

Um, Now, I will say that you can use snapshots as a way to get backup, but a

snapshot by itself on a volume is, I like to call it a convenience copy, right?

It is a way to go back in time as long as you don't have media failure.

Right, right.

You don't have a double disk failure in a RAID 5 array or a triple

disk failure in a RAID 6 array.

Yeah, and like you're saying, it could use a snapshot to allow you to do other backup

mechanisms like take a copy, preserve it in a point of time, now you move that

data and you back up that data, right?

Which, if you're integrating with applications, you can now take an

application consistent point in time while your database is in hot

backup mode, take that snapshot, now you preserve that point in time.

You can.

Uh, Tha, the database, so it can continue operating like normal, Tha, the database.

What word are you saying there?

I'll follow.

I just, I don't know what word I thought you were saying there.

So basically you're saying, because you freeze it.

You're saying you freeze it and now you thaw it.

Okay, all

Yeah, or you could quiesce and unquiesce.

Yeah.

Yeah.

Okay.

I just, I don't usually use that term.

So it really threw me.

right.

And then you do your backup off of that snapshot, right?

So you now have a copy that's frozen that you can now do your

backup and it's all good to go.

Right.

Um, I'd say the most common outside of storage arrays, the most common snapshot

that is used in that way is VSS, right?

The Windows Volume Shadow Services.

And it, it's basically integrated into the operating system.

It's integrated into the applications.

It is.

And backup apps can integrate with VSS.

These are all done with APIs and a backup app can show up and say, Hey,

I am here to do a backup of this box.

Um, please, I'm going to really simplify it.

Please take a snapshot of everything that needs to have a snapshot taken of

it before I take a backup, then you take a backup and then they, um, and it can

take a backup of that snapshot, even though the volume continues to change.

It it's given this view into the volume that is static.

And then it can back up that volume, uh, and get that perfectly application

consistent version of the volume.

Uh, even if the backup takes two hours, it doesn't matter.

It has it, all of the blocks will be from the same exact point in time.

And then when it's done, it can tell VSS to delete that snapshot

or it can keep it around.

It's up to you.

It's just, it's a configuration thing.

One thing I do want to mention.

That, like we said, Snapshots just gives you that point in time copy, right?

It's a read only, point in time copy.

Now sometimes you will also hear, and I don't think we're covering

it later, clones being used, right?

Where I take a virtual copy of the volume and start using it, just going

back to the previous discussion, Curtis.

The difference is clones are writable.

So you're making changes to it.

Right, a snapshot is a read only copy that you preserve that point in time,

nothing's gonna change it, and it's always there for you to go back, you

can pull your file, your data out of it.

Clones, on the other hand, give you a copy of the volume at a point in time,

but it's so you can use it for some purpose, like for testing out restore.

Capabilities.

Can I verify my backups?

Those sort of things.

And also doing, like, database recovery against that copy, the clone

copy, and other things like that.

So, clones are different than snapshots, even though they both

might start from a snapshot copy.

Right.

Um, yeah, I, I would probably just call it a read write snapshot, but

maybe that's a contradiction in terms.

Um,

Yes, a snapshot is a point in time, Curtis.

yeah, exactly.

THere are three ways that snapshots are created.

The most common way, I, would you, we say it's still the most common

way, the copy on write method?

Uh, no, I don't think, I don't think so anymore.

Okay.

Well, historically what used to be the most common method before

one vendor ruined it for everybody

Made something

is.

So is called the copy on write method.

And the reason it's called the copy on write method is that we create this,

this storage area that is going to hold the, um, the, the snapshot blocks.

And when we go to update a block, because we're, it's a storage volume, right?

So we're going to update a block and we say, Hey, Uh, there's

a snapshot for this block.

We're going to copy that block out to the snapshot area before you write.

So that's why it's called copy on write.

And that is very expensive because if you think about it, there are, let me count.

There, there, there is a read.

And a right for every right.

There's, there's a says,

and it's not just the one read and write that happens on the snapshot side.

You also have a bunch of metadata and updating indirect blocks and

a whole bunch of other things.

So, yeah, doing a copy on write might lead to, say, 10 additional I.

O.

operations or 12.

Yeah.

for that one block.

And so what happens is that over time that the number of blocks, remember

when I initially, I mentioned that the bulk of the data is going to come from

the primary volume, but over time.

As a snapshot has been created, more and more blocks are going to be copied into

that snapshot area, and which means that at some point, you know, a significant

portion of my snapshot, if I'm.

If I'm reading it, a significant portion is going to come from the snapshot area.

And so it, there are multiple reasons that there's a performance hit.

I think the biggest performance hit is.

You know, you talk about all of the IOs that have to happen every time we update,

uh, you know, we do a copy on write.

The other is that as time goes on, the more and more data that I have to get

from my snapshot area when I'm doing a read, the performance goes down.

And this goes back to that vendor that they basically suggested that if we had

90 days of user browsable snapshots, That their performance was going to be

like half of what, uh, what it typically

and I would say that is Probably based on older technology, Curtis.

I think when you had traditional RAID arrays or RAID Groups

that you were creating.

I think there was more of a performance impact I think now since you end

up aggregating a bunch of disks and then carving out volumes And so

you can share in the performance.

I think That's not as big of a concern anymore as it used to be.

Like I know those systems you're talking about, they now support

a thousand snapshots, right, for

so you think that the main hit from a performance standpoint on a copy

on write snapshot scenario in modern technology is mainly that IO hit the

first time you go to do a write when every time, every time you update a write.

And by the way, remember that It has to do that for every, that's sort of

calculate every time it does it right.

It has to calculate, is there a snapshot that is looking at this

block as it exists at this point in time, and then you have to copy it,

uh, for that, for that snapshot.

Which, there are different mechanisms you could use.

You could use bitmaps, you could use other things.

So, it's not the end of the world, and I think that a lot of these storage vendors

have optimized if they are continuing to use copy on write technologies.

But I would say a good chunk of them have moved away from

copy on write because of the I.

O.

penalties that we've talked

right, right.

So the, the, there was this vendor that came out, a little vendor

called, at one point it used to be called Network Appliance.

And it had a little screw and bolt as its logo.

right.

Um, the, the, at one point they said, we're just going to change our

name to NetApp because that's what

That's what everyone calls them.

I remember actually working at a startup and got my hands on my

first NetApp appliance and was like, yeah, wow, these are kind of cool.

And this was before I started working there.

And I was like, wow, this is really amazing and simple

for what it does and easy to

Yeah, exactly.

And they used a completely different way to, um, to do snapshots that at the time

was revolutionary, which I think has now been adopted by a lot of storage vendors.

And that is, we call it redirect on write.

Do you want to describe how that

works?

so what read so before you get there?

I think we need to talk about their right anywhere file layout

which allows then the snapshot.

Yep waffle Which a lot of other vendors I think do something similar as well these

days but what it is is going back to the copy on write example If you are writing

a block of data in copy on write sort of file system, previous file systems,

you would always write to the same spot.

And because you're always writing to the same spot, that's why you have to first

copy out the data and then update it.

With a write anywhere file layout, what you end up doing is, it doesn't

matter which actual block you end up writing to, you basically construct the

metadata tree to reference that block.

Even though I might be updating block 100, block 100 might actually

physically be at, like, block 1000.

And because I have all my metadata that tells me exactly where that data

exists, I just need to update the metadata to say, okay, if someone

tries to access block 100, it's actually physically on block 1000.

And so you can end up writing to any location in the file system

and not having to worry about always hitting that same location.

And so that's kind of waffle in a nutshell.

So basically what you then is you have this metadata system that has a pointer

to every block and it doesn't really matter where those blocks happen to be.

And you can have thousands of these snapshots, right?

These pointers at the top.

right, so then when we go to do an update and we have a snapshot, it just

means that what we're going to do is we're going to change the pointer.

We're going to say, okay, this, there's this block that's sitting here.

And we know we, we, we're not supposed to update that block because we

have a snapshot that's requiring on that, that's requiring that block.

And then we're going to just redirect.

We're going to, we're going to write a new block for the

new version of that old block.

And then we're going to change the pointer, right?

We're going to redirect the pointer.

To this new location of that block.

Meanwhile, the old block is still sitting there and we've got a

snapshot that's just pointing to it.

Right.

Um, and so you've got this infinite number of snapshots that are pointing

to an infinite number of, well, it's not an infinite number of snapshots, but

you have a very high number of snapshots that are, that, that, and they're all.

Just a whole bunch of metadata pointing to a whole bunch of blocks that are

just sitting all around the volume.

And the one thing, so this all sounds amazing, right?

Because you're like, Oh, writes are, writes are super fast.

I don't have to worry about it.

In fact, Curtis, just one correction.

When you're going to actually write a new block, you never actually have to

look up the old block because you're always writing to a new location.

So you don't care if that old block is occupied by a snapshot or not, right?

Because this is the downside though, of snapshots, especially with the

write anywhere file layout is.

You now need a process to go through and say when a snapshot gets deleted,

what blocks are no longer being used actively because they don't

belong to snapshots, they're not currently as part of the volume.

So you have a garbage reclamation process or different vendors call it

different things, but some process to go through and reclaim all of those

free blocks so they can be reused.

I think you said the same thing I said in different words, but

yeah, I see what you're saying.

I guess I was saying that, that it's making a decision on what to do.

Based on whether or not, I think if I could just change my part

of my answer, instead of, is this block being used by anything else?

I guess is,

Yeah, more

I guess that's the question I'm asking.

Yeah.

Uh, and actually, I guess what you're saying is it actually doesn't even make

that decision at that point in time.

If it's going to modify a new block, if it's going to modify a

block, it just writes a new block.

And then what happens to that old block is a completely separate process.

If there's, if there is a snapshot that's pointing to that, that

old block, then it will stay.

If there are no snapshots that are pointing to that block, then at some

point the garbage collection process will come and make it go away.

Is that a better, is that a better

Yes, that is a better description.

Now this is

don't want to argue with a former NetApp employee

about how NetApps

now I should say this is based on NetApp's technology, different

vendors may do different things, but for the most part, most of the

vendors do something similar ish.

Specifically, it's based on how NetApp worked when you worked

there, which was a while ago.

And things may be different now,

that is also true.

not.

I mean, Waffle is like at the core of their...

You know, the core of their technology.

And so, and that's why with redirect on right, that's why you could essentially

have an infinite number of snapshots with, with zero performance penalty,

you have zero performance penalty of having basically the performance

penalty of doing a right update.

Is the same whether you have a snapshot or you don't have a snapshot.

The, the only penalty, if you want to call it that, is that, um, one, one

big difference between this way and the other way is there is no snapshot area.

Right.

So in the other method, the snapshot area could fill up if

you held snapshots for too long.

In this configuration, there is no snapshot area.

The snapshot area is the volume, and if you have too many updates and you

keep too many snapshots, you would fill up the volume with snapshots.

So you've gotta get rid of the older

Well, and this is where That would apply if you're talking NetApp terminology

traditional volumes, but most of it has moved over to virtual volumes, where once

again, you have an aggregate, a shared pool of common data, and for each of the

volumes, typically you also set a limit on how much space a snapshot can occupy.

So you could say, I am allowing 20 percent for snapshots of my overall

volume capacity, in which case it'll start

And what happens when you hit that wall?

I, I don't know what the current behavior is.

Previously.

I believe it would like let you like start automatically pruning

snapshots and trying to free up space.

Right, right.

Because it, obviously, it's not going to, it's not going to prune, uh,

production, you know, current data.

So, yeah, I, it could, but again, we're, we're talking specifically NetApp,

but something has to happen, right?

If you're using this method.

Something has to happen.

Either we have to stop creating new snapshots, right?

Or stop updating the snapshots that we have.

And, uh, we need to delete older snapshots or we need to maybe delete, you know,

certain ones in the middle, right?

Basically you've got to do some kind of pruning or else you're going to

Yeah, the other challenge is also figuring out what snapshot to delete

because blocks are being shared, right?

You might be like, hey, this snapshot is huge and you go delete it, but

because those blocks are being shared by other snapshots, you're not actually

going to free any space, right?

So you need to be able to figure out like which snapshot actually

contains unique blocks that if I delete it will actually save me space.

complicated.

Storage management.

I

I don't miss production storage management.

Any other final thoughts on redirect on, right?

think that covers it

I mean, My personal, if you're going to do snapshots on a storage array, I

think redirect on write is the way to go.

It sounds like what you're saying, they've made copy on write better,

but I still think redirect on write is just significantly better.

So, um, but it might be more complicated than if you're coding it.

Right.

So the next one is what I'm going to call the dumbest of all snapshot methods.

That's not what I have in the book.

I gave it a much nicer name in the book.

And guess who does this method?

The leading hypervisor company in the world.

Yes.

I think that's a fair statement, right?

company in the world.

I think, I think it still is.

Yeah.

And that would be VMware.

So the way VMware does snapshots is just literally the Dumbest

implementation of snapshots that I've ever seen and I don't know how they

haven't addressed it, but here it is.

When you create a snapshot in VMware, it literally holds all the rights.

Now, by the way, if I'm wrong, by the way, you know, Broadcom, don't sue me.

This is based, this is based on my understanding of VMware snapshots.

Uh, you know, I've, I've checked every once in a while and they,

no one seems bothered by this.

Uh, but if this has changed, any of you that are, you know, if anybody

works for Broadcom slash VMware, then, you know, feel free to update

me and I will update this episode.

And I'll just delete this section.

But here's the way it works.

When you create a single snapshot on a VMware volume, it halts.

All rights on the, on the current volume.

And then it keeps all rights in a snapshot area.

And then when you delete that snapshot, it replays all those

rights against the production volume.

And this is why when you make a snapshot.

And then you, if you hold that snapshot for a long time and then

you delete that snapshot, this is why it has a big performance hit

against the production volume.

But no one

this is

snapshot of a VM

yeah, this is why you do not do this.

You don't use snapshots on VMware level snapshots the way you do any other

snapshots, because, and by the way, I used VMware for years before knowing this.

That's why I want to make sure I mentioned it.

And, and that is that if you create a snapshot and then hold it for a

long period of time, you're going to get hit with a massive IO hit

when you delete that snapshot.

So if you're using VMware, VMware level snapshots, then you use them the way we

talked about earlier, where you create a snapshot, you make a, you make a backup.

And then you delete the snapshot.

Maybe you take a VMware level snapshot, and then you take a storage level

snapshot of that snapshot, and then you delete the, the VMware level snapshot.

You should, if this is the way your snapshot system works, you

cannot leave the snapshots around for any significant period of time.

I was going to chime in.

Thank you for covering that.

The, this specifically is VM where software snapshots,

if you wanna call it that.

Right, that are only done at the VMware level.

Now, there are integrations that various storage vendors offer

by plugging into the VMware API.

So whenever you trigger a VMware snapshot, it actually triggers

a storage level snapshot.

So avoiding some of these issues, but not everyone is aware of it.

Not everyone is using a third party storage array that integrates with VMware.

So.

Just

Yeah.

So, right.

Thanks for, thanks for clarifying that.

This is specifically VMware level snapshots that are done by VMware.

And without any third party storage.

Yeah.

And I don't know why VMware did this, but it's bonkers.

It's just literally one of the weirdest, codest thing, weirdest

coded things I've ever heard.

Why would you do it that way?

Somewhere in a meeting, this is how they decided to

it.

was probably easier

and yeah, yeah, maybe it was easier,

and they didn't talk to the

I wonder about that.

They didn't, they exactly, they did not talk to the backup folks.

Well, uh, I think we have, uh, summarized the world of snapshots.

That?

No, I think we did a good job with that.

So, copy on write, redirect on write, dumbest method ever.

Those are the three types.

I've got it officially in the book, uh, I've got this labeled

as the hold all writes method.

Uh, I should really just change that to the dumbest method ever.

But, um, yeah.

So, you know, snapshots are a great tool.

In the backup and recovery arsenal.

They are the great sort of basis upon which we're going to talk about one

of my favorite ways to do backup.

And we're going to talk about that in another episode.

Hint, it's called near CDP, not CDP.

It's called near CDP.

And, uh, it's just, just the number one thing you have to understand

about snapshots is that unless you have copied this snapshot to

another location via some mechanism.

Which could be backup.

It could be replication of the volume.

It could be a number of things.

You do not have a backup.

You have a picture of your volume.

And that picture of your volume is as worthless as a picture of your

house after your house burns down.

It'll just be a nice memory and, uh, and a really bad day.

So it's just, that's the really, the most important thing to

understand about snapshots.

And now if this is your first time, snapshots have been explained to you.

Now you understand why I don't like it that they call.

What AWS does snapshots because that very much does not meet

the definition that we just had.

And I'm glad I brought this up because it's important to what we're talking

about is storage level snapshots.

Darn it.

I don't know

You can't call it that, yeah, because

this.

Yeah, these are traditional snapshots.

There are other things out there that people call snapshots

that don't work like this.

AWS snapshots don't work like this.

Uh, they are an actual image copy.

It's actually, they actually, when you make an AWS snapshot, it actually copies

that, that point in time out to another area of storage, which happens to be S3,

you?

And I think specifically you're talking about an AWS EBS snapshot

thank you.

I am talking about an AWS EBS snapshot.

Um, my former employer, Druva, they call what they do snapshots.

They call their backups snapshots.

I never liked that, but you know, nobody asked me.

Uh, so, but what we're talking about here is traditional snapshots.

And, um, a lot of other people will call what they do a snapshot.

Um, the problem is like a lot of terms in the, in the backup world.

It's a term like so many of our terms are, um, their words that are used

just, they're just English words that are used in so many different contexts.

when, like when we had the CDP episode, we couldn't figure out what to call

those point in time, because a lot of the CDP vendors called them snapshots.

Yeah, exactly.

Yeah, exactly.

All right.

Well, uh, I guess the only thing left for me to say is that's a wrap