if you're using a hybrid cloud storage system that keeps a cache

of your data on premises and then stores the primary copy in the cloud.

You may be wondering if you need to back up this data.

If you're familiar with me, you might think you already know what

I'm going to say at this point.

And my answer might surprise you.

If you're not familiar with me.

Hi, I'm W.

Curtis Preston.

AKA Mister backup.

For over 30 years I've had a singular passion for helping

others protect their data.

With backup and disaster recovery systems.

Every episode of this podcast starts with a brief overview of backup

related news, followed by a deep dive, into a single area or lesson that

you can apply in your environment.

To protect you from horrible things like ransomware.

Welcome to the backup wrap up.

W. Curtis Preston: welcome to the show.

I'm your host, w Curtis Preston, a k a, Mr.

Backup, and I have with me my T 5 6 8 B consultant Prasanna

Malaiyandi, how's it going?

Prasanna.

I'm good, Curtis, and it took me a second

to realize what you talking about.

W. Curtis Preston: I knew you would figure it out though.

'cause you know you got those little nerd bits up in your brain

that just get activated when I say certain things, what is the T?

5 6, 8 B and a.

it is the standard for how your ethernet cables are

run, whether which pairs go together.

And there are apparently two standards

W. Curtis Preston: Yeah, I think the A is the old B is the new.

they all work as long as, you do the same thing on both ends of the cable

. Prasanna Malaiyandi: unless you're building

a crossover cable, in which yeah.

W. Curtis Preston: Don't do that.

Yeah.

I, the thing is I'm, I recently discovered that I have, what do you call it?

That the phone jacks in my house actually have RJ there, that

they have Cat six run to them.

And so I'll be wiring them as ethernet ports and, so hence

my interest in the subject.

But, let's get off of that nonsense and get right onto the news of the day.

You know what we need?

We need a, we need like a letter, like a wow.

News of the day.

let's start with our little story from a small, it's a startup.

I believe it's a startup company.

It's called T to Toyota

It's a small startup

I don't believe that they pronounced the second T

as a T

W. Curtis Preston: Really Toyota.

how else would you pronounce

Yeah.

I think the, I think the official name was Toyota,

but they, I think they figured that Americans wouldn't

be able to pronounce that.

oh,

W. Curtis Preston: interesting.

this is a Japanese thing.

Yeah.

So what in the world happened over

at Toyota Prasanna

Yeah, just something about them trying to make a bunch

of cars then for some reason a whole bunch of their manufacturing plants.

I think

14 Yeah.

W. Curtis Preston: shut down

Yeah.

as they're trying to crank out.

So if you were waiting for the new

W. Curtis Preston: Prius or the

new Prius, or the new R four,

you might have Yeah.

W. Curtis Preston: If you're on the li, if you're on the list of people

desperately trying to get a Prius Prime instead of just a Prius, this is why.

This is the worst, is they were doing Scheduled maintenance on

their database, and they neglected to take into account the amount

of space necessary to do the task.

And they ran out of storage space on their, production system, and it just

haled the production at a dozen factories.

I don't know what to say.

Yeah.

and the crazy thing is it's not like, oh, I've just got a free up space and I'm

boom, everything's gonna come back online.

Like restarting a production

W. Curtis Preston: Yeah.

is time consuming and very

hairy.

W. Curtis Preston: you know, it reminds me, uh, when I was at the

bank a hundred years ago, when.

The E p O button was pressed.

I won't say who it was me, . So shutting it down was easy.

Bringing the bag, the bank, nobody knew how to bring everything back online.

It's probably the same thing here.

And here's the part that, just this sentence in this article from

bleeping computer.com, which I'll link to in the show description.

Toyota explains that its main servers and backup machines

operate on the same system.

Due to this, both systems face the same.

Failure making a switchover impossible, inevitably leading

to a halt in factory operations.

So they don't have

high availability

W. Curtis Preston: so

or are they running like two virtual machines on the

W. Curtis Preston: Here's what I think that means.

They do have standby machines.

Maybe what this means, what they don't have is standby storage.

The storage was the thing that had the outage.

And so since both the backup and the primary systems used the same storage,

that's not very highly available, right?

a single point of failure.

yeah.

But Toyota, you're a big company.

spend some money and hire a proper IT consultant to come in, or a storage

consultant to come in and plan your

storage.

I'm sure there are numerous companies who would like to

come talk to you about buying

their products.

W. Curtis Preston: exactly.

yeah,

Howard would love to talk you.

Howard

Marks

W. Curtis Preston: would, would this is a classic example of single point of failure

so I, I really don't know what to say.

And what do we have is our second story, This one is actually,

I think, more depressing.

So it's from a, I don't know if it's a

company or a news source called

Beta News, and they recently published an article by Ian Barker saying

that two out of three companies lose data due to failed backups.

And Curtis backups never fail.

Right,

W. Curtis Preston: Uh, so it was based on a survey from a

company called aorn Acorn aorn,

Yeah, I

W. Curtis Preston: think so.

an encrypted

drive

W. Curtis Preston: maker.

Prasanna Malaiyandi: And it was a survey done

in the uk, right.

So

W. Curtis Preston: I'm sure things are so much better over here.

yeah.

W. Curtis Preston: not Here's the thing that I, when I looked at this

article, and again, we'll link to it in the show description, that.

They said that this was a marked difference from the previous year.

They said a quarter of respondents say the ransomware has been the main cause

of a data breach and increase for 15%.

32% is the people that, they had an unsuccessful recovery and

that was up from only 2% in 2022.

So a huge change.

The thing that I think says everything is they said that this

sharp . Increase un recoverability.

it was basically at the same exact time as there was a sharp

decrease in the level of automation.

What, you know what I'm talking about there?

Yeah.

which makes sense, right?

Yeah.

Because it's oh, you don't have automated processes in place, tools, et cetera.

If you don't have those and how can you really verify that

your backups were successful?

I know Curtis, you like to talk about, verify your backups, make sure they

work before you actually need them.

And so if you don't have it, then that makes sense.

The other interesting thing in the article, which might go hand

in hand with is they were also talking about how.

Like with everyone working remotely, There was a lot of sort of self-service.

Let employees do their own backups, copy the data they want.

And we all know how that goes.

come on Curtis.

How often would you back up data on your

If someone wasn't sitting there backing it up automatically.

W. Curtis Preston: Yeah, I've never been a fan of backups that

require manual intervention.

I understand that in some cases, that may be the only choice, but I think backups

should be a hundred percent automated that new systems that come online, for

example,, So like when you should be using auto selection and auto discovery

and all of these things so that let's say you're backing up VMware and you

add a new vm, that VM will automatically be protected by the backup system.

You don't need to add it.

I think that anything that you can automate

is possible.

At least a base level, right?

It doesn't have to be like, oh, everything is like super duper protected, but

at least you have something going.

And just going to the example they gave about trusting

employees to copy their data.

if you don't wanna trust your employees, you could do things like Give them

a one drive or a Google Drive and have them put their content there

and that's where they're creating it.

And then you as a central admin, you now go back up that one repository rather

than trusting your users are doing

W. Curtis Preston: Yeah.

Yeah.

we know how

that

W. Curtis Preston: goes.

exactly.

So I think that the thing that we can learn from this sort of story

is an increase in manual backups is a decrease in recoverability.

And if you walk away with nothing else, right?

Walk away with that piece of information.

Is automation good, manual, bad when it comes to backups?

The more we can get the human outta the better.

And I also wanna chime in on one thing, Curtis, I know

you probably cut your teeth doing this way back when you started, but

even things like scripts, right?

People may think scripts are automated, but scripts are not

W. Curtis Preston: They're not bulletproof.

I wrote a lot of scripts back

in my day, Prasanna.

I remember, yeah, I remember that one year.

I remember writing 150 custom shell scripts to make a

particular configuration work.

And it was just meant, and I'm sure every one of those broke at some point,

, Prasanna Malaiyandi: Yep,

W. Curtis Preston: I wonder if any of 'em are still in use.

That was 20 years ago.

Probably not 23 years ago.

so I think it's time for our main topic today.

Our main topic today is going to be whether or not we need to back up

what I'm calling hybrid cloud storage.

how would you, how do you think we should define that?

It's funny, I know we recently just talked about

like the public cloud, and if you ask Five years ago what the hybrid

cloud is, you'd probably get very different answers from different people.

I think though what you're starting to see is people coming to the realization that

hybrid cloud is Where you own some of the resources, maybe it's running in your own

data center or somewhere like that, and you're also leveraging the benefits of

the cloud, but you're not fully embracing the cloud like we talked about with the

public cloud in the last episode, right?

It's you're sending some bits and pieces of data leveraging services

where possible, where it makes sense, and otherwise you have a lot of your

applications still running in your data

center.

Now if we dig into what hybrid cloud storage is though, right?

That's the capability where you basically are leveraging all the

benefits of having cloud storage.

Like we talked last time about infinitely available storage

with AWS's SS three, right?

Google has their own.

but in order to access that, you have a component running on premises.

in your data center, which your applications can talk to.

That way they don't need to rejigger themselves to talk to the cloud.

They talk to that one appliance, and that one appliance is able to spread

the data out amongst the cloud.

Sometimes it keeps a local copy cached on that appliance, so then

you get almost instant access, just as if the system, the storage, was

local

so this is companies like

Nasuni Panera CTERA..

W. Curtis Preston: The idea is you get a box or multiple boxes.

You put the data on that box, and then they use the magic of the cloud to

make sure that that data is protected.

everything that goes on that box is also copied into the cloud, but the

box essentially acts as a cache.

the record of authority, I think is the one up in the cloud.

But you're accessing a local cache of what's in the cloud, and then it's,

through cloud magic, they're making sure that the current version, as well

as historical versions of those files or blocks, are stored in the cloud.

Does that sound about right?

Yeah.

Yeah, that sounds good.

And right.

And the other benefit that you see typically with this sort of hybrid cloud

storage is there are use cases where you have multiple remote offices all

who need to collaborate on a document.

Like you see this in

CAD design.

or CAD firms, I should say.

And so this is where you sort of have multiple people all collaborating and

being able to synchronize and get access to the same files without necessarily

having multiple copies all spread out.

And they all have their own CAD for performance reasons,

especially with CAD files.

They're very

W. Curtis Preston: Yeah, so it's in that sense, it's like a big, much

more expensive, say Dropbox, right?

It's both a, it's a collaboration tool as well as a on-prem storage solution.

Yeah.

And the one thing I wanted to correct, Curtis, I know you said they give you like

a box, right?

I think we need to be careful these days.

Like it's not always

a physical box if that's what our listeners think, right?

Many times it's a software package that you install in a hypervisor, right?

And that gives it the functionality it needs.

It doesn't have to be a physical server,

W. Curtis Preston: the hypervisor's on a box.

So I still say . That's, I know what you're saying.

It's not

yes it is.

W. Curtis Preston: a physical appliance that you're getting from this company.

Yeah, it's just an on-prem component.

That may be a physical box, it may be a VM running in your, virtualization world.

So the

question Prasanna, as is often

the question

How about I asked you the

W. Curtis Preston: Oh, okay.

I'll ask you.

And Curtis, I know that your stance has always been backup

W. Curtis Preston: yeah.

Back up.

All the things.

Now in the case of, yeah.

Back up.

All the things in the case of hybrid cloud storage, do I need to back it up?

I know we talked about public cloud last episode.

Right.

And I know in the past we've talked about traditional data sources.

What's your take on hybrid cloud storage?

W. Curtis Preston: So first off, let's start with the concept to

back up all the things, right?

assuming that the data that you're putting on there has value to the, your

organization, it needs to be backed up.

Okay.

That the, there's just no question about that.

The question I think that is appropriate to this scenario is whether or not.

it is being backed up, right?

I think the first thing to think about is are there two copies, right?

And then are there historical copies?

And the answer is, it depends.

, right?

on

W. Curtis Preston: For, For, the recent data, there are definitely

two copies, right there is the copy on your local device.

There is the copy that is up in the cloud because by design, everything you put on

the device is copied up into the cloud.

So there you have two copies.

Depending on what you're syncing to, you could also argue that there are typically

three copies up there, because if you're syncing to SS three or something like

S three, you are using object storage that is automatically replicating

its data to multiple locations.

So you have multiple locations that . Are subject to different risk profiles,

assuming they're far enough apart.

But here's my thing.

Typically what you're storing though, in that case is probably

something that is in a native format for that stor storage appliance.

And so you are at a single point of failure at that instance, right?

Because if say something gets corrupt with metadata, Or if it's

for some reason unable to piece together what's been stored in the

backend, like how do you know that?

Like the file table or the entire file system

with all the metadata has been backed up as well?

If something gets corrupt, how do you restore that?

If someone deletes a file, right?

How do you do that?

And Or if you need to restore, right?

Does that require you to completely spin up a new software appliance

somewhere and connect it in, and what does that look like?

So I think while I agree with you that yes, you do have your copy in the cloud,

and so it meets all those principles, I think it depends what you're trying

to recover from, which will determine if it's really a backup or if you

really need to back it up or not.

W. Curtis Preston: Yeah, so I think we're in agreement there.

I was just one thing at a time, right?

I was thinking first, let's think about the files.

Now we have to think about the file system and the configuration that you're right.

That in this case, you, you're using essentially a cloud gateway.

The now, if it's the appliance itself that dies, or you're talking about the

metadata, this sort of fancy file system, that metadata, is there a facility

within that to rebuild that metadata?

based on what's ever, whatever's already up there, is there a way to rebuild

that, assuming that gets corrupt?

the other thing is that file system, maybe the data is up there is fine

and maybe the box is fine, but I, I I got a ransomware attack, right?

This is the most common thing, right?

I got a ransomware attack and it attacked a Windows box that was s m b mounted

to . or the box is s and b mounted to the windows box, and I was able to go in

and, corrupt slash encrypt all the files connected to a project or a work group.

And now what we need is not just we haven't corrupted the file system

per se, as much as what we've corrupted is we've corrupted the

current version of the file system.

Right.

We need to be able to go back in time.

So that's the E, that's the other aspect is, do you have it in multiple

locations and then do you have different versions of it over time?

What is the company's answer to that?

Yeah.

And that's where I think each company's probably gonna do something different.

And you gotta dig into the documentation, but it is, like

you said, an important question to ask as you're evaluating vendors.

W. Curtis Preston: Yeah.

how do we recover from, fat fingering, deleting a file, fat fingering,

deleting a directory, or accidentally deleting a whole bunch of files.

And then of course, the,

the, what do you call it, the, the ransomware attack.

Thank you.

Yeah.

The other thing to add to that list, Curtis, is also what happens when metadata

in the file system gets corrupted.

W. Curtis Preston: So what happens if the Yeah, exactly.

Whatever this magic is that's taking all those objects and

pretending they're a file system.

'cause that's essentially what we're doing, right?

we're taking a bunch of objects stored up in object storage, and we're making

it look like it's a file system.

What happens if that gets messed up?

How do you recover from that?

I think one thing though, that it would be interesting to get your take on this is I.

,when you're doing these backups, you're now locked into

that vendor's format, right?

Because it's a vendor's file system that is being versioned, right?

Let's assume they have versioning, right?

They're able to protect again.

So all those scenarios that we talked about, right?

There may still be something to consider that you may want it to be

agnostic to that vendor's format,

W. Curtis Preston: It's a very good point.

The best analogy that I have to, this is NetApp, right?

We go back to NetApp, right?

Yep.

I knew you were going there.

W. Curtis Preston: So they had all this stuff, they got

all this replication, right?

What did people want?

They wanted the ability to back up that data and put it in another format.

And which for a record or for, for the record.

That's what I didn't like about N D M P.

Right.

Um, I mean, I, I always saw N D P as like a necessary evil because it was

the better way to back up a NA box.

But one of the things I never liked about N D M P was that

. Prasanna Malaiyandi: History.

W. Curtis Preston: the backup in a format that was only usable on that platform.

So that was a, it was, by the way, it was the only way they

were able to make it happen.

Because they were able to say to all the NAS vendors, of which at

the time they were probably four or five, they said, look, you can use

whatever backup format you want.

This is a protocol.

This is a, this is an IO protocol.

It's not a backup format protocol.

You don't have to go.

Yeah, but that's what I didn't like about it, is that it wasn't portable.

Until you got a company like Avamar who was able to crack

the code and allow the

restores to different file

systems.

W. Curtis Preston: Even there, like they were able to do that for

NetApp, but they didn't necessarily do it for other For other vendors.

I'm assuming that the code for U F S dump is available out there somewhere.

And

so it probably wasn't rocket science to crack that particular backup format,

but yeah, you make a real, that's a really valid point, is to make sure

that you at least discuss that question.

What do we, what is our plan if and when, The feces hits the

rotary oscillator for this product.

What is our plan?

And because if you're a hundred percent tied into them and you're,

it's the whole vendor, tie in and I'm sure they're very happy with that.

Yeah.

Vendor lockin,

Yeah.

W. Curtis Preston: I'm sure they're very happy with that.

The question is, what is your plan if and when you ever choose,

to do something different?

Yep, yep.

It's important to at least ask that question upfront because you may not

ever consider another vendor for Ever right, But it's at least an important

question to ask upfront because if there's a huge switching cost, that's

something you should take, into

W. Curtis Preston: Yeah, so quick answer is yes.

Every piece of data needs to be backed up.

The longer answer is you need to make sure that you've got the location dealt with.

You've got to make sure that you have a recovery plan for when you

get attacked by a ransomware bot, and it just encrypts your entire

System, does that encrypt your history?

Does that encrypt all the versions?

Does that all that encryption automatically replicate to the cloud?

'cause the answer is yes.

So the question is how do I go back to before the encryption happened,

what's, what's that process?

And then I think, you know your point about whether or not

you've got backups that are.

Have the ability to go back to a different vendor, assuming you want to do one.

my dru would also be in the backend to have that, to have the data also

replicated to a, an inexpensive copy in another cloud provider.

That would be my dream.

That may actually be possible with some of these vendors.

If you're replicating both to.

Let's just assume, for example, that the only thing you care about is the

active data, at least per the only.

One where you care about performance is the active data.

You want to keep the historical stuff, but you don't wanna

spend a lot of money on it.

So what would be nice is if the Cloud copy could be something like

Glacier Deep Archive with Instant Restore, pay the instrument, restore

fee, and then also copy that to the equivalent on Azure or Google.

And now you've got two independent copies of the data, with history.

That's cool.

Yeah, and I think what they would probably end

up doing, what the vendor would do is they might support two cloud

targets rather than trying to copy data just because cloud egress costs are

W. Curtis Preston: Yeah.

Oh, I see what you're saying.

You're saying that it'll just send it to both, uh, vendors.

Yeah, exactly.

Yeah.

Cloudy res, , cloudy, gress costs are very real.

and I know Curtis, we like to talk about it

a lot on this podcast, but test your recovery scenarios If you are

using a hybrid cloud storage, right?

Test it with some of these, test what happens when you delete a file, right?

Can you restore it, do it before you actually need

W. Curtis Preston: Yeah.

Yeah.

And don't do it like our friend in Alaska did it by deleting all

your data and then testing it.

this is, this is what the cloud is for, right?

Uh, te you could probably spin up a copy, in the cloud, right?

I can't agree with you more, and we can't say that enough.

Is that, the only backup that is truly valid is one that's

been tested in recovery.

yeah.

Exactly.

Well, I think we covered that topic and hopefully answered the question.

If you disagree or strongly agree, we'd love to hear from you.

You can reach me atCurtis@backupwrapup.com.

I look forward to hearing from you.

After recording this episode, I had a chance to see one of these

vendors present a tech field day.

CTERA talked about their edge filer and how it can replace your on-premises

Nass infrastructure with a global file system that is automatically stored in

his many cloud accounts and vendors.

As you ask it to use.

It also automatically stores all versions of every file and stores them in an area

that they say is air gapped and immutable.

I will say I didn't have enough time to dig deep on those claims.

So I would suggest you do so.

From a data protection standpoint, the most exciting feature they

talked about was ransom protect.

They monitor all of file activity using AI after the number of suspicious

events that you specify, they can alert the appropriate number of people.

In any monitoring tools that you have automatically blocked

the offending end point.

And allow the admin to roll back any encrypted files.

They claim to be able to detect and stop the ransomware

attack within about 30 seconds.

And then they can roll back any files and only a few minutes.

That sounds a whole lot better than anything else I've seen.

I did ask them about monitoring for suspicious reads as well as rights.

If they were able to detect suspicious reads, they could also

stop an exfiltration attack, which is quickly becoming the standard.

In fact, most cyber attackers are performing exfiltration

before they begin any encryption.

It seems like they have the tools and logic in place to be able to do this.

And they did say that they were looking into it.

I will eagerly await their update.

Although I stand by our podcast from two weeks ago where we said that

perhaps we get a little too excited about new tools and not enough emphasis

on things like process and people.

I do think it's still okay to get excited about new tools.

So this is, I think very interesting.

I hope you've enjoyed this, uh, first actual episode with our

new name, the backup wrap up.

This has been a production of backup central.

Edited and produced by yours truly.

Any opinions that you hear are those of that speaker and not

necessarily their employer.

This has been the backup wrap-up.