if you're using a hybrid cloud storage system that keeps a cache
of your data on premises and then stores the primary copy in the cloud.
You may be wondering if you need to back up this data.
If you're familiar with me, you might think you already know what
I'm going to say at this point.
And my answer might surprise you.
If you're not familiar with me.
Hi, I'm W.
Curtis Preston.
AKA Mister backup.
For over 30 years I've had a singular passion for helping
others protect their data.
With backup and disaster recovery systems.
Every episode of this podcast starts with a brief overview of backup
related news, followed by a deep dive, into a single area or lesson that
you can apply in your environment.
To protect you from horrible things like ransomware.
Welcome to the backup wrap up.
W. Curtis Preston: welcome to the show.
I'm your host, w Curtis Preston, a k a, Mr.
Backup, and I have with me my T 5 6 8 B consultant Prasanna
Malaiyandi, how's it going?
Prasanna.
Prasanna Malaiyandi: I'm good, Curtis, and it took me a second
to realize what you talking about.
W. Curtis Preston: I knew you would figure it out though.
'cause you know you got those little nerd bits up in your brain
that just get activated when I say certain things, what is the T?
5 6, 8 B and a.
Prasanna Malaiyandi: it is the standard for how your ethernet cables are
run, whether which pairs go together.
And there are apparently two standards
W. Curtis Preston: Yeah, I think the A is the old B is the new.
they all work as long as, you do the same thing on both ends of the cable
. Prasanna Malaiyandi: unless you're building
a crossover cable, in which yeah.
W. Curtis Preston: Don't do that.
Yeah.
I, the thing is I'm, I recently discovered that I have, what do you call it?
That the phone jacks in my house actually have RJ there, that
they have Cat six run to them.
And so I'll be wiring them as ethernet ports and, so hence
my interest in the subject.
But, let's get off of that nonsense and get right onto the news of the day.
You know what we need?
We need a, we need like a letter, like a wow.
News of the day.
let's start with our little story from a small, it's a startup.
I believe it's a startup company.
It's called T to Toyota
It's a small startup
Prasanna Malaiyandi: I don't believe that they pronounced the second T
as a T
W. Curtis Preston: Really Toyota.
how else would you pronounce
Prasanna Malaiyandi: Yeah.
I think the, I think the official name was Toyota,
but they, I think they figured that Americans wouldn't
be able to pronounce that.
oh,
W. Curtis Preston: interesting.
this is a Japanese thing.
Yeah.
So what in the world happened over
at Toyota Prasanna
Prasanna Malaiyandi: Yeah, just something about them trying to make a bunch
of cars then for some reason a whole bunch of their manufacturing plants.
I think
14 Yeah.
W. Curtis Preston: shut down
Prasanna Malaiyandi: Yeah.
as they're trying to crank out.
So if you were waiting for the new
W. Curtis Preston: Prius or the
Prasanna Malaiyandi: new Prius, or the new R four,
you might have Yeah.
W. Curtis Preston: If you're on the li, if you're on the list of people
desperately trying to get a Prius Prime instead of just a Prius, this is why.
This is the worst, is they were doing Scheduled maintenance on
their database, and they neglected to take into account the amount
of space necessary to do the task.
And they ran out of storage space on their, production system, and it just
haled the production at a dozen factories.
I don't know what to say.
Prasanna Malaiyandi: Yeah.
and the crazy thing is it's not like, oh, I've just got a free up space and I'm
boom, everything's gonna come back online.
Like restarting a production
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: is time consuming and very
hairy.
W. Curtis Preston: you know, it reminds me, uh, when I was at the
bank a hundred years ago, when.
The E p O button was pressed.
I won't say who it was me, . So shutting it down was easy.
Bringing the bag, the bank, nobody knew how to bring everything back online.
It's probably the same thing here.
And here's the part that, just this sentence in this article from
bleeping computer.com, which I'll link to in the show description.
Toyota explains that its main servers and backup machines
operate on the same system.
Due to this, both systems face the same.
Failure making a switchover impossible, inevitably leading
to a halt in factory operations.
Prasanna Malaiyandi: So they don't have
high availability
W. Curtis Preston: so
Prasanna Malaiyandi: or are they running like two virtual machines on the
W. Curtis Preston: Here's what I think that means.
They do have standby machines.
Maybe what this means, what they don't have is standby storage.
The storage was the thing that had the outage.
And so since both the backup and the primary systems used the same storage,
that's not very highly available, right?
a single point of failure.
Prasanna Malaiyandi: yeah.
But Toyota, you're a big company.
spend some money and hire a proper IT consultant to come in, or a storage
consultant to come in and plan your
storage.
I'm sure there are numerous companies who would like to
come talk to you about buying
their products.
W. Curtis Preston: exactly.
yeah,
Prasanna Malaiyandi: Howard would love to talk you.
Howard
Marks
W. Curtis Preston: would, would this is a classic example of single point of failure
so I, I really don't know what to say.
And what do we have is our second story, This one is actually,
I think, more depressing.
Prasanna Malaiyandi: So it's from a, I don't know if it's a
company or a news source called
Beta News, and they recently published an article by Ian Barker saying
that two out of three companies lose data due to failed backups.
And Curtis backups never fail.
Right,
W. Curtis Preston: Uh, so it was based on a survey from a
company called aorn Acorn aorn,
Prasanna Malaiyandi: Yeah, I
W. Curtis Preston: think so.
Prasanna Malaiyandi: an encrypted
drive
W. Curtis Preston: maker.
Prasanna Malaiyandi: And it was a survey done
in the uk, right.
So
W. Curtis Preston: I'm sure things are so much better over here.
Prasanna Malaiyandi: yeah.
W. Curtis Preston: not Here's the thing that I, when I looked at this
article, and again, we'll link to it in the show description, that.
They said that this was a marked difference from the previous year.
They said a quarter of respondents say the ransomware has been the main cause
of a data breach and increase for 15%.
32% is the people that, they had an unsuccessful recovery and
that was up from only 2% in 2022.
So a huge change.
The thing that I think says everything is they said that this
sharp . Increase un recoverability.
it was basically at the same exact time as there was a sharp
decrease in the level of automation.
What, you know what I'm talking about there?
Prasanna Malaiyandi: Yeah.
which makes sense, right?
Yeah.
Because it's oh, you don't have automated processes in place, tools, et cetera.
If you don't have those and how can you really verify that
your backups were successful?
I know Curtis, you like to talk about, verify your backups, make sure they
work before you actually need them.
And so if you don't have it, then that makes sense.
The other interesting thing in the article, which might go hand
in hand with is they were also talking about how.
Like with everyone working remotely, There was a lot of sort of self-service.
Let employees do their own backups, copy the data they want.
And we all know how that goes.
come on Curtis.
How often would you back up data on your
If someone wasn't sitting there backing it up automatically.
W. Curtis Preston: Yeah, I've never been a fan of backups that
require manual intervention.
I understand that in some cases, that may be the only choice, but I think backups
should be a hundred percent automated that new systems that come online, for
example,, So like when you should be using auto selection and auto discovery
and all of these things so that let's say you're backing up VMware and you
add a new vm, that VM will automatically be protected by the backup system.
You don't need to add it.
I think that anything that you can automate
is possible.
Prasanna Malaiyandi: At least a base level, right?
It doesn't have to be like, oh, everything is like super duper protected, but
at least you have something going.
And just going to the example they gave about trusting
employees to copy their data.
if you don't wanna trust your employees, you could do things like Give them
a one drive or a Google Drive and have them put their content there
and that's where they're creating it.
And then you as a central admin, you now go back up that one repository rather
than trusting your users are doing
W. Curtis Preston: Yeah.
Yeah.
Prasanna Malaiyandi: we know how
that
W. Curtis Preston: goes.
exactly.
So I think that the thing that we can learn from this sort of story
is an increase in manual backups is a decrease in recoverability.
And if you walk away with nothing else, right?
Walk away with that piece of information.
Is automation good, manual, bad when it comes to backups?
The more we can get the human outta the better.
Prasanna Malaiyandi: And I also wanna chime in on one thing, Curtis, I know
you probably cut your teeth doing this way back when you started, but
even things like scripts, right?
People may think scripts are automated, but scripts are not
W. Curtis Preston: They're not bulletproof.
I wrote a lot of scripts back
in my day, Prasanna.
I remember, yeah, I remember that one year.
I remember writing 150 custom shell scripts to make a
particular configuration work.
And it was just meant, and I'm sure every one of those broke at some point,
, Prasanna Malaiyandi: Yep,
W. Curtis Preston: I wonder if any of 'em are still in use.
That was 20 years ago.
Probably not 23 years ago.
so I think it's time for our main topic today.
Our main topic today is going to be whether or not we need to back up
what I'm calling hybrid cloud storage.
how would you, how do you think we should define that?
Prasanna Malaiyandi: It's funny, I know we recently just talked about
like the public cloud, and if you ask Five years ago what the hybrid
cloud is, you'd probably get very different answers from different people.
I think though what you're starting to see is people coming to the realization that
hybrid cloud is Where you own some of the resources, maybe it's running in your own
data center or somewhere like that, and you're also leveraging the benefits of
the cloud, but you're not fully embracing the cloud like we talked about with the
public cloud in the last episode, right?
It's you're sending some bits and pieces of data leveraging services
where possible, where it makes sense, and otherwise you have a lot of your
applications still running in your data
center.
Now if we dig into what hybrid cloud storage is though, right?
That's the capability where you basically are leveraging all the
benefits of having cloud storage.
Like we talked last time about infinitely available storage
with AWS's SS three, right?
Google has their own.
but in order to access that, you have a component running on premises.
in your data center, which your applications can talk to.
That way they don't need to rejigger themselves to talk to the cloud.
They talk to that one appliance, and that one appliance is able to spread
the data out amongst the cloud.
Sometimes it keeps a local copy cached on that appliance, so then
you get almost instant access, just as if the system, the storage, was
local
so this is companies like
Nasuni Panera CTERA..
W. Curtis Preston: The idea is you get a box or multiple boxes.
You put the data on that box, and then they use the magic of the cloud to
make sure that that data is protected.
everything that goes on that box is also copied into the cloud, but the
box essentially acts as a cache.
the record of authority, I think is the one up in the cloud.
But you're accessing a local cache of what's in the cloud, and then it's,
through cloud magic, they're making sure that the current version, as well
as historical versions of those files or blocks, are stored in the cloud.
Does that sound about right?
Prasanna Malaiyandi: Yeah.
Yeah, that sounds good.
And right.
And the other benefit that you see typically with this sort of hybrid cloud
storage is there are use cases where you have multiple remote offices all
who need to collaborate on a document.
Like you see this in
CAD design.
or CAD firms, I should say.
And so this is where you sort of have multiple people all collaborating and
being able to synchronize and get access to the same files without necessarily
having multiple copies all spread out.
And they all have their own CAD for performance reasons,
especially with CAD files.
They're very
W. Curtis Preston: Yeah, so it's in that sense, it's like a big, much
more expensive, say Dropbox, right?
It's both a, it's a collaboration tool as well as a on-prem storage solution.
Prasanna Malaiyandi: Yeah.
And the one thing I wanted to correct, Curtis, I know you said they give you like
a box, right?
I think we need to be careful these days.
Like it's not always
a physical box if that's what our listeners think, right?
Many times it's a software package that you install in a hypervisor, right?
And that gives it the functionality it needs.
It doesn't have to be a physical server,
W. Curtis Preston: the hypervisor's on a box.
So I still say . That's, I know what you're saying.
It's not
Prasanna Malaiyandi: yes it is.
W. Curtis Preston: a physical appliance that you're getting from this company.
Yeah, it's just an on-prem component.
That may be a physical box, it may be a VM running in your, virtualization world.
So the
question Prasanna, as is often
the question
Prasanna Malaiyandi: How about I asked you the
W. Curtis Preston: Oh, okay.
Prasanna Malaiyandi: I'll ask you.
And Curtis, I know that your stance has always been backup
W. Curtis Preston: yeah.
Back up.
All the things.
Prasanna Malaiyandi: Now in the case of, yeah.
Back up.
All the things in the case of hybrid cloud storage, do I need to back it up?
I know we talked about public cloud last episode.
Right.
And I know in the past we've talked about traditional data sources.
What's your take on hybrid cloud storage?
W. Curtis Preston: So first off, let's start with the concept to
back up all the things, right?
assuming that the data that you're putting on there has value to the, your
organization, it needs to be backed up.
Okay.
That the, there's just no question about that.
The question I think that is appropriate to this scenario is whether or not.
it is being backed up, right?
I think the first thing to think about is are there two copies, right?
And then are there historical copies?
And the answer is, it depends.
, right?
Prasanna Malaiyandi: on
W. Curtis Preston: For, For, the recent data, there are definitely
two copies, right there is the copy on your local device.
There is the copy that is up in the cloud because by design, everything you put on
the device is copied up into the cloud.
So there you have two copies.
Depending on what you're syncing to, you could also argue that there are typically
three copies up there, because if you're syncing to SS three or something like
S three, you are using object storage that is automatically replicating
its data to multiple locations.
So you have multiple locations that . Are subject to different risk profiles,
assuming they're far enough apart.
Prasanna Malaiyandi: But here's my thing.
Typically what you're storing though, in that case is probably
something that is in a native format for that stor storage appliance.
And so you are at a single point of failure at that instance, right?
Because if say something gets corrupt with metadata, Or if it's
for some reason unable to piece together what's been stored in the
backend, like how do you know that?
Like the file table or the entire file system
with all the metadata has been backed up as well?
If something gets corrupt, how do you restore that?
If someone deletes a file, right?
How do you do that?
And Or if you need to restore, right?
Does that require you to completely spin up a new software appliance
somewhere and connect it in, and what does that look like?
So I think while I agree with you that yes, you do have your copy in the cloud,
and so it meets all those principles, I think it depends what you're trying
to recover from, which will determine if it's really a backup or if you
really need to back it up or not.
W. Curtis Preston: Yeah, so I think we're in agreement there.
I was just one thing at a time, right?
I was thinking first, let's think about the files.
Now we have to think about the file system and the configuration that you're right.
That in this case, you, you're using essentially a cloud gateway.
The now, if it's the appliance itself that dies, or you're talking about the
metadata, this sort of fancy file system, that metadata, is there a facility
within that to rebuild that metadata?
based on what's ever, whatever's already up there, is there a way to rebuild
that, assuming that gets corrupt?
the other thing is that file system, maybe the data is up there is fine
and maybe the box is fine, but I, I I got a ransomware attack, right?
This is the most common thing, right?
I got a ransomware attack and it attacked a Windows box that was s m b mounted
to . or the box is s and b mounted to the windows box, and I was able to go in
and, corrupt slash encrypt all the files connected to a project or a work group.
And now what we need is not just we haven't corrupted the file system
per se, as much as what we've corrupted is we've corrupted the
current version of the file system.
Right.
We need to be able to go back in time.
So that's the E, that's the other aspect is, do you have it in multiple
locations and then do you have different versions of it over time?
What is the company's answer to that?
Prasanna Malaiyandi: Yeah.
And that's where I think each company's probably gonna do something different.
And you gotta dig into the documentation, but it is, like
you said, an important question to ask as you're evaluating vendors.
W. Curtis Preston: Yeah.
how do we recover from, fat fingering, deleting a file, fat fingering,
deleting a directory, or accidentally deleting a whole bunch of files.
And then of course, the,
the, what do you call it, the, the ransomware attack.
Thank you.
Prasanna Malaiyandi: Yeah.
The other thing to add to that list, Curtis, is also what happens when metadata
in the file system gets corrupted.
W. Curtis Preston: So what happens if the Yeah, exactly.
Whatever this magic is that's taking all those objects and
pretending they're a file system.
'cause that's essentially what we're doing, right?
we're taking a bunch of objects stored up in object storage, and we're making
it look like it's a file system.
What happens if that gets messed up?
Prasanna Malaiyandi: How do you recover from that?
I think one thing though, that it would be interesting to get your take on this is I.
,when you're doing these backups, you're now locked into
that vendor's format, right?
Because it's a vendor's file system that is being versioned, right?
Let's assume they have versioning, right?
They're able to protect again.
So all those scenarios that we talked about, right?
There may still be something to consider that you may want it to be
agnostic to that vendor's format,
W. Curtis Preston: It's a very good point.
The best analogy that I have to, this is NetApp, right?
We go back to NetApp, right?
Prasanna Malaiyandi: Yep.
I knew you were going there.
W. Curtis Preston: So they had all this stuff, they got
all this replication, right?
What did people want?
They wanted the ability to back up that data and put it in another format.
And which for a record or for, for the record.
That's what I didn't like about N D M P.
Right.
Um, I mean, I, I always saw N D P as like a necessary evil because it was
the better way to back up a NA box.
But one of the things I never liked about N D M P was that
. Prasanna Malaiyandi: History.
W. Curtis Preston: the backup in a format that was only usable on that platform.
So that was a, it was, by the way, it was the only way they
were able to make it happen.
Because they were able to say to all the NAS vendors, of which at
the time they were probably four or five, they said, look, you can use
whatever backup format you want.
This is a protocol.
This is a, this is an IO protocol.
It's not a backup format protocol.
You don't have to go.
Yeah, but that's what I didn't like about it, is that it wasn't portable.
Prasanna Malaiyandi: Until you got a company like Avamar who was able to crack
the code and allow the
restores to different file
systems.
W. Curtis Preston: Even there, like they were able to do that for
NetApp, but they didn't necessarily do it for other For other vendors.
I'm assuming that the code for U F S dump is available out there somewhere.
And
so it probably wasn't rocket science to crack that particular backup format,
but yeah, you make a real, that's a really valid point, is to make sure
that you at least discuss that question.
What do we, what is our plan if and when, The feces hits the
rotary oscillator for this product.
What is our plan?
And because if you're a hundred percent tied into them and you're,
it's the whole vendor, tie in and I'm sure they're very happy with that.
Yeah.
Vendor lockin,
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: I'm sure they're very happy with that.
The question is, what is your plan if and when you ever choose,
to do something different?
Prasanna Malaiyandi: Yep, yep.
It's important to at least ask that question upfront because you may not
ever consider another vendor for Ever right, But it's at least an important
question to ask upfront because if there's a huge switching cost, that's
something you should take, into
W. Curtis Preston: Yeah, so quick answer is yes.
Every piece of data needs to be backed up.
The longer answer is you need to make sure that you've got the location dealt with.
You've got to make sure that you have a recovery plan for when you
get attacked by a ransomware bot, and it just encrypts your entire
System, does that encrypt your history?
Does that encrypt all the versions?
Does that all that encryption automatically replicate to the cloud?
'cause the answer is yes.
So the question is how do I go back to before the encryption happened,
what's, what's that process?
And then I think, you know your point about whether or not
you've got backups that are.
Have the ability to go back to a different vendor, assuming you want to do one.
my dru would also be in the backend to have that, to have the data also
replicated to a, an inexpensive copy in another cloud provider.
That would be my dream.
That may actually be possible with some of these vendors.
If you're replicating both to.
Let's just assume, for example, that the only thing you care about is the
active data, at least per the only.
One where you care about performance is the active data.
You want to keep the historical stuff, but you don't wanna
spend a lot of money on it.
So what would be nice is if the Cloud copy could be something like
Glacier Deep Archive with Instant Restore, pay the instrument, restore
fee, and then also copy that to the equivalent on Azure or Google.
And now you've got two independent copies of the data, with history.
That's cool.
Prasanna Malaiyandi: Yeah, and I think what they would probably end
up doing, what the vendor would do is they might support two cloud
targets rather than trying to copy data just because cloud egress costs are
W. Curtis Preston: Yeah.
Oh, I see what you're saying.
You're saying that it'll just send it to both, uh, vendors.
Yeah, exactly.
Yeah.
Cloudy res, , cloudy, gress costs are very real.
Prasanna Malaiyandi: and I know Curtis, we like to talk about it
a lot on this podcast, but test your recovery scenarios If you are
using a hybrid cloud storage, right?
Test it with some of these, test what happens when you delete a file, right?
Can you restore it, do it before you actually need
W. Curtis Preston: Yeah.
Yeah.
And don't do it like our friend in Alaska did it by deleting all
your data and then testing it.
this is, this is what the cloud is for, right?
Uh, te you could probably spin up a copy, in the cloud, right?
I can't agree with you more, and we can't say that enough.
Is that, the only backup that is truly valid is one that's
been tested in recovery.
yeah.
Exactly.
Well, I think we covered that topic and hopefully answered the question.
If you disagree or strongly agree, we'd love to hear from you.
You can reach me atCurtis@backupwrapup.com.
I look forward to hearing from you.
After recording this episode, I had a chance to see one of these
vendors present a tech field day.
CTERA talked about their edge filer and how it can replace your on-premises
Nass infrastructure with a global file system that is automatically stored in
his many cloud accounts and vendors.
As you ask it to use.
It also automatically stores all versions of every file and stores them in an area
that they say is air gapped and immutable.
I will say I didn't have enough time to dig deep on those claims.
So I would suggest you do so.
From a data protection standpoint, the most exciting feature they
talked about was ransom protect.
They monitor all of file activity using AI after the number of suspicious
events that you specify, they can alert the appropriate number of people.
In any monitoring tools that you have automatically blocked
the offending end point.
And allow the admin to roll back any encrypted files.
They claim to be able to detect and stop the ransomware
attack within about 30 seconds.
And then they can roll back any files and only a few minutes.
That sounds a whole lot better than anything else I've seen.
I did ask them about monitoring for suspicious reads as well as rights.
If they were able to detect suspicious reads, they could also
stop an exfiltration attack, which is quickly becoming the standard.
In fact, most cyber attackers are performing exfiltration
before they begin any encryption.
It seems like they have the tools and logic in place to be able to do this.
And they did say that they were looking into it.
I will eagerly await their update.
Although I stand by our podcast from two weeks ago where we said that
perhaps we get a little too excited about new tools and not enough emphasis
on things like process and people.
I do think it's still okay to get excited about new tools.
So this is, I think very interesting.
I hope you've enjoyed this, uh, first actual episode with our
new name, the backup wrap up.
This has been a production of backup central.
Edited and produced by yours truly.
Any opinions that you hear are those of that speaker and not
necessarily their employer.
This has been the backup wrap-up.