You found the backup wrap up your go-to podcast for all things

backup recovery and cyber recovery.

In this episode, we examine one of the most misused terms in our industry.

Air gap.

I'm talking with persona about what air gap really means, where it came from

back in the day, and why it's become so critical again, with ransomware attacks.

We'll explain the difference between a true air gap and what I would call a

virtual air gap or a logical air gap.

Trust me.

If you've ever wondered whether your backup system is really air

gapped or if you're tired of hearing vendors, throw this term around

without knowing what they're actually talking about, this is your episode.

We're going back to the original definition and

explaining why context matters.

When evaluating modern backup solutions.

By the way, if you don't know who I am, I'm w Curtis Preston, also known

as Mr. Backup, and I've been passionate about this topic for over 30 years.

Ever since.

I had to tell my boss that there were no backups.

I. Of the production database that we had just lost.

I don't want that to happen to you.

I certainly don't want it to happen to me.

That's why I do this.

On this episode, we will turn unappreciated backup admins

into Cyber Recovery Heroes.

This is the backup wrap up.

Welcome to the show.

Hi, I am w Curtis Preston, AKA, Mr. Backup, and I have with me the one guy

that got me to actually start doing my taxes this year, way ahead of schedule.

By way, I might actually finish it before April, which will be

the first time in, I don't know,

So

a few years.

listeners who there are quite a few of you out there,

Wait, I haven't introduced you.

You're not allowed to talk yet.

hi Curtis.

So

Hey.

listeners who may not be aware in the US we have to submit our taxes

or file our taxes by April 15th.

Yeah,

deadline.

You can of course ask for an extension and file in October

and blah, blah, blah, blah,

but if you owe taxes, you have to pay before the April 15th.

Yeah.

Yeah, Curtis normally starts first week of April,

Yeah.

Yeah.

this time he started five days before the first week of April.

Well, and, and in my defense, in my defense, my taxes are much more

complicated than the average person, which you could say, well, that should

mean you should start them earlier.

Uh, but it's like, but the, the procrastination, you know,

I, if you've ever heard this, uh, hard work always pays off.

Eventually procrastination always pays off immediately.

Yes it does.

And but here's the other thing is I think in the past you've put off.

Doing taxes because like you said, your taxes are complicated and

Yeah.

all the data

Yeah,

90% of what everyone struggles with, which is why they

yeah,

But

yeah.

And I,

now with the in with technology,

yeah.

has now made your life easier, which is why I think you actually

started a little earlier.

Um, yeah, the technology definitely helped.

Um, and, um.

You know.

Yeah.

But, and I'm, I'm at, I'm at like 85% is where I think I am.

Uh, I just need to dot the, i's across the T's, so to speak.

Um, and uh, I, the hardest part for me is the going through the

thousands of transactions and QuickBooks and making sure that

they're all in the right categories.

Right.

You

And making sure that I include, 'cause I'm.

I'm not always, the thing I do that's just bad is I'm not always like, I don't just

have one card for business and then always use that card when I'm doing something.

Sometimes I'll be somewhere and I don't have that card or

whatever, and I'll just buy it.

or whatever

Yeah.

Yeah.

So,

yeah.

there is a way to solve this.

So,

What's that?

what I tell, which is what I recommend for a lot of people, which is.

Do it along the way.

Don't wait until March 24th

Yeah,

through and

yeah.

QuickBooks or whatever software thing you're using.

Like if you get a physical receipt, just put it in a box.

If you're downloading transactions, just periodically, maybe once

a month, spend like 15 minutes

That's a fascinating idea.

right now.

Right?

The next thing you're gonna tell me is to clean up my shop in between

projects instead of waiting until I do 10 projects and then have

giant mess and clean that up.

you know what I'm really what you should be doing in your shop when you use a tool.

When you're done with the tool, put it away.

Oh, that's just crazy talk.

That's just crazy talk.

God, God,

see

just can't handle that level of whatever.

Yeah.

Anyway.

Well, uh, speaking of putting away tools, we're gonna be talking about

putting away backups, uh, putting 'em away in a, in a, in an area that

you know, that you can't get to.

Not

gonna be talking about air gap, which, what was that?

Not getting rid of them.

Right.

Because that

Yep.

a way

No, no.

Not get, no, I said putting, putting, putting him away, you

know, putting him away in a,

yeah.

place.

Yes.

Uh, we're gonna talk about this, this, this term that, um, you

know, is bantied about quite a bit.

Uh, a couple of terms that have become in vogue for reasons that we will

discuss in the last couple of years would be this term and what other term?

Immutability.

Yeah, yeah, exactly.

So I think we need to start though, before we dive into this

Yeah,

I think we need to start with the history because

yeah, back in the day

Yeah.

Well, I don't think

Mm-hmm.

folks understand the context.

They're like, oh, they use the term, but they don't know like

where it necessarily came from.

Right.

And I think that becomes

And this, this is one of those things where context really matters, right?

Uh, so we have to go back to the days when everybody made tapes, right?

That was just how backups were made, all backups were on tape.

Um, and that, that's just the way it was.

And when you.

If you were doing it right, in my opinion,

Hmm,

right there, there you always put a gap of error.

A a, you know, between

physical

an actual gap of error between, uh, your backups and the thing you were backing up.

There were two ways to do that.

One would be to send the original offsite.

The other and more proper way to do it would be to make a copy and

then send one of those offsite.

I actually prefer that you send the original offsite

So just a couple clarifications.

When you talk about a physical gap, you're not just talking about like good

friends at OVH and how they had a fire.

Right.

Where technically

was air gapped.

that.

That's why I'm asking, right, is there was technically a

Well, no.

those systems.

Well, no, there wasn't because.

racks.

No, that, yeah.

I'm glad you brought that up.

So if you are electronically connected to the thing that is not air gapped,

That's

right?

It, yeah.

Good.

Yeah.

Good, good point.

What I'm talking about is an offline, very important an offline.

Both in terms of electronically, offline and physically offline.

Copy that.

There is a, like I said, a literally a gap of air.

There is no way to get it, get to it electronically,

Yeah.

right?

The only way you could get to it was a person you needed to call.

Uh, you know, iron Mountain or whoever is, you know, you had,

and you needed to bring it back.

And then what you could do, and you would do is you would put all

different levels of, of, of, you know, what we would now call IAM Right.

Between you and your data and that copy of your data.

Right.

It wasn't even really possible, but you wouldn't have the concept

of like using your, your same login to go get your tapes, right?

You would go in, you would, you would do it, you would have some sort of

physical identification method., You would have an id, you would have a

process, uh, to, to authenticate yourself.

And we're talking about actually physically showing up, going in,

getting tapes, or you would have an, you would have a protocol for contacting.

The people and having them bring it to you.

And if you deviated from that protocol, for example, if you called up and said,

I don't want you to take my tapes to the bank where you always took them.

I want you to take them and meet me at a Walmart.

That would be, that would be a problem.

Right.

Um, uh, and, and, and obviously back then we weren't worried about like

AI voices, uh, faking that stuff.

but this is kind of like, I would say the Why it was needed from

like an enterprise perspective, but

Mm-hmm.

smaller companies, right?

You could still achieve air gaps, if you will, without requiring like

Iron Mountain or another service.

I know we've brought up the case, uh, multiple times, right?

With someone doing a backup and then shipping a tape in a

box to a different facility and

Yeah.

Uh.

back

Uh,

or.

for, for smaller companies, another thing that you would do is, uh, if

you were, uh, you know what I call A TSB, a truly small business, you

might take the tape out and put it in your trunk and take it home, right?

Uh, and then you gotta just worry that you're not leaving the tape in a hot

car and all of that kind of stuff.

But you, the, the biggest thing that we were worried about back then

was, was natural disasters and fires and floods and things like that.

So you wanted to physically separate.

That copy as much as possible from the thing that it was protecting.

We weren't so much worried about cyber.

Uh, you know, it wasn't, it wasn't completely outta the question, but mainly

what we were worried about was actually, um, besides the natural disasters and

things, we, we actually were more worried about, like social engineering, uh, you

know, like the movie sneakers, which if you haven't seen the movie sneakers.

Please go watch the movie sneakers.

There's a few things in there that are obviously very silly and,

and over the top, but honestly, some of the best depiction of

social engineering I've ever seen.

Not to mention a lot of fun and a lot of stars.

Robert Redford, um, uh, Dan Arod, uh, James Earl Jones.

Um, the what, what's the actor that you now know to be Indian?

Ben Kingsley.

Been Kingsley.

Yeah.

'cause we discussed you thought for a long time.

He wasn't Indian because of that name.

Uh, yeah, he, he he's in it, uh, anyway, great movie.

We were more concerned about that sort of thing.

And so we would actually, uh, and again, yes, this is for an

enterprise, but we actually had a process where we would, on a regular

basis, try to defeat the security.

Right.

We would go over there and we would, um, have a very involved, like story

as to why, for example, I needed to be left in the vault alone.

you were

W with my tapes.

Huh?

You were

We

teaming,

were red teaming the, uh, the, yeah, the um, yeah, which is what, which

is what sneakers is about as well.

Please watch that movie.

Just, just a good movie.

is it a strict requirement that the copy be

for it to be air gapped, or would you consider a case

where someone takes a backup

I.

removes it from the tape drives, puts it on a shelf or in a safe in the

location, or a tape robot as an example?

That

those are two very different things that you just said.

Yeah.

but I'm

These, these are great questions.

You're so good at asking these questions.

Prasanna.

Um, so technically no the term air gap, right?

Like in it parlance.

Generally refers to a computer that isn't connected to the network.

Right.

Then there's, there's computers that are like almost air gapped,

Yeah.

right.

Where, where they're like massively firewall walled off or, you know.

Right.

Yeah.

Yeah.

Uh, I can think back to, um, being at, uh, Amazon.

When I worked at Amazon, I had a contract there in.

98 and they had a computer that was a payment processing computer and it

was basically one, one way traffic.

So, so to go to your question, yeah.

Technically.

The term air gap could apply to a tape that has been taken out and

is in a, in a safe, you know, in a, in a safe place on premises.

But that would just, if that's your only copy, that would violate what?

3, 2, 1, rule.

Yes, that would violate the 3, 2, 1 rule.

Right?

Three copies of data on two different media, one of which is offsite and, yeah.

that works.

For the first scenario I described

But, but let me, you, you talked about, you also talked about a,

a second scenario, which is, a tape in a tape library that is

accessible to the system air gaped.

And I'm gonna say absolutely not, right?

Because why?

Because I can electronically control that tape library.

I can put that tape in the tape drive and I can erase that

tape so that is not air gapped.

Even though there is technically

That

a gap of air between the thing.

Yeah,

like what

yeah,

I think as we talk about modern air gaps and

yeah,

have changed the words, I think it's good to understand the history.

yeah.

Absolutely.

And so that is the gold standard by which anything that wants to call

itself air gap, uh, is measured.

Back then we were predominantly concerned with, um, fires, floods,

terrorist attacks, et cetera.

And we still are concerned with those things.

It's just that the likelihood of those is much less than the likelihood

of something else, which would be

Ransomware

ransomware or, or any kind of a cyber attack.

But yeah, ransomware definitely.

I agree that cyber attacks make it difficult, but I wonder if

that's sort of a, what do you say?

just kind of like a secondary effect, right?

But do you think that it's because having disconnected systems is

nearly impossible these days

yeah, we'll get to that.

I, I'm just saying that back then we were predominantly concerned with physical

things that needed a physical separation.

Now we're concerned, primarily, while we're still concerned with those things,

Yeah.

somewhat dealt with many of those things with a lot of technologies, a

lot of replication, a lot of highly available systems, especially for,

um, uh, mission critical systems.

But we have not.

Necessarily kept up with this relatively new threat, which is, you

know, the idea of, of a cyber attack and, and, and a ransomware attack.

And there is also, so that's the new reason why air gap is even

more important than it was before.

What is the reason that backup systems don't.

Really have air gaps as I define them earlier.

because.

,They're connected, right?

And typically these backup systems aren't, are no longer using tape, right?

You're now

using things like de-duplicated storage.

Uh, you might be using the cloud, right?

You have all these other options, and so you're not really using tape anymore.

And

Yeah.

the new backup technologies that we talked about, like incremental forever, which

aren't, which tape isn't so friendly with.

No.

No, they're not.

Yeah.

Uh, our previous episode we talked about the concept of incremental

forever and what a great thing it is, but it would not work very well.

So, you know, I. Like, I wanna make sure people understand me here.

Like I, I definitely am on the record of being a, a friend of tape.

Right.

And tape has a lot of uses and, and it's got a lot of life left in it until someone

invents something that offers everything, that tape offers at the price point

to tape offers it, tape will continue.

You know, there, there's a, there's a guy that, um, um, storage

Zillow was his Twitter name.

I don't know if it still is, but.

Uh, mark Toomey, and he used to say that like there will be, like, there'll be a

nuclear apo apocalypse, and, and somewhere in the world there will be somebody

with a mainframe and some tape, right?

Yeah.

It's not going anywhere anytime soon.

There's all sorts of possible other things that we're talking about to

maybe supplant tape, but, but it's not going anywhere anytime soon, but.

It definitely is not.

Uh, yeah.

Let's go back to my concept.

I'm definitely a friend of tape.

Having said that, there are a lot of reasons why we moved off the tape

for, um, on-premises backup, right?

There were a lot of really good reasons.

I mean, there were.

There were re like downsides of tape, right?

Specifically having to do with how fast tape was and how slow backup was.

And it was a fundamental mismatch.

And that's really the core reason why we moved off of tape.

And then we got a lot of really cool things like incremental forever, the

idea being able to replicate backups.

Now we can have an onsite backup and an offsite backup.

Without touching a tape, without doing anything, it's just magic.

And we have on-prem and off-prem, we can back up using deduplication, we

can back up across the, the internet,

Yep.

would've thought.

Right?

Um, but

But

as you mentioned already,

Yeah.

the big drawback from a cybersecurity perspective is that there's no air

gap, not an air gap as defined.

Previously and.

we talked about in the beginning.

and I think this is why many vendors have now sort of modified the

Yeah,

to now call it a virtual air gap.

A logical air gap.

Like to, basically nuance it versus

yeah.

definition.

My.

My experience has been, they don't even, they don't even nuance it.

They just say air GAed and they only use those terms when pushed.

Right.

Um, and uh, they'll just say it's, it is air GAed.

Yeah.

Right.

And they'll, and, and I'm gonna say no, it's not

Yeah.

for

let's talk about what some of vendors now call air gap, quote

unquote air quotes, right?

But

Yeah,

use the word virtual air gap just for the

sure.

podcast, just to refer

Yeah,

definition

yeah,

using?

yeah.

So, um, basically

I would, I would put it into like two categories.

One is that if it's in a storage system that refers to itself as worm, right?

That's right.

One's read many, in other words, a, a. Immutable storage system,

meaning that the data cannot be

Yep.

deleted there.

And, and these are two different things I I want to definitely separate.

There is there is storage that is damn near immutable,

Yep.

right?

Even tape that's immutable isn't really immutable because you can take, you

can take a hammer, a what'd you say?

A what?

fire to it,

Yeah, yeah, yeah.

Flame thrower, right?

Um, same thing with optical media and, and anything that claims to be immutable.

Nothing is a hundred percent immutable.

Um, but there are storage systems that even if you, um, root it,

basically, you can't overwrite the data

Yep.

that's there.

Now, having said that, again, you could set fire to it.

Right.

Um, which is why when we, we start talking about cloud copy, so that's one way

in which that they say it's something.

get to sort of the immutable right, versus the other one.

Maybe we should talk about like if we kind of distill down what the

air gap provided, right, or what they're looking for air gap to

Yeah, sure.

really about making sure that the data doesn't go away

Yeah.

there is a cyber attack or something else like that.

Exactly.

That's pretty much the whole reason of an air gap.

And so how can we do that

Yeah.

an electronic world, in an online world, in a cloud world?

How can we ensure that when the feces hits the rotary oscillator, that you

know that it it, that you will have at least one copy that is somewhere

that you know is available to you.

And so this is a great technology that a lot of vendors offer and

it's been around for a long time.

I wanna say I first heard about this almost two decades ago

The concept of immutable storage.

Yeah, yeah, yeah.

right.

Um.

so it's been around for a while.

It does offer some great capabilities, but also some drawbacks.

I think the biggest being, you write to it, if you ever want to

delete it for whatever reason,

Yeah,

needed.

sorry.

accidentally wrote something I shouldn't have.

I needed to reduce my retention periods for legal reasons.

Whatever it is,

Yeah,

delete it.

yeah.

Sorry, not sorry on that one.

Right.

Um, and so, uh, and, and, and when we talk about immutable and, and

we've done an episode or two on that.

There are various levels of immutable immutability right there.

There are, you know, basically append only file systems.

There are, um, but many of them have kind of a back door, uh, that then you need to

look into what does it take to use that back to get to the back door, and, you

know, all of that, that different stuff.

But, um, the, the best ones are.

Storage level, immutable, so that even if you got super all powerful,

you know, uh, then you, you wouldn't be able to do anything.

to get to delete the data is basically to physically break

Right.

Right, right.

And um, so that's one thing.

And then the other thing is basically I am.

Right.

So, uh, identity and access management.

So we have a copy that's not technically immutable in the cloud, but it's

just impossible to get to it, right?

Um, and the idea is that you use, you use a different, uh, IAM system for this copy

than you do for the rest of the world.

And you also, um, you, you basically put many levels of protection, right?

Obviously you, you have MFA, you have, um, you know, pass keys.

You have all of the best security that you have available to you.

And,

you know, and you just, you just follow all of the best practices to.

Ensure as best as possible that even if somebody got a username

and password, they wouldn't be able to do, you know, the, you, you,

you protect it as much as you can.

I agree with that.

And I think another thing you can consider is this could either be

something like when you said IAM, right?

And putting access controls, right.

Another mechanism is you.

There are two ways I look at it.

Either you do it yourself, so

Mm-hmm.

creating separate AWS

Right.

Right.

uh, putting it in.

If you run primarily on Amazon, maybe you're backing up to Microsoft Azure,

Mm-hmm.

Mm-hmm.

So you're kind of segregating your, and isolating your environment.

Another mechanism is maybe you end up using a cloud service provider.

Yep.

service provider that actually provides these services for you and kind of

gives you that separation that you

Yeah.

Essentially they have the key to the vault.

Yeah.

And, um, speaking of cloud service providers, I was thinking about

with the first one with immutable.

There are backup vendors, cloud backup vendors, that while the

backups as they're writing it, are, they're not using immutable storage

because immutable storage and dedupe

Geez, don't

really go well together very well.

Um.

At least in terms of the way cloud does

yeah,

costing and everything.

So they don't use immutable storage on the backend because there is cloud, there is

immutable storage available in the cloud.

If they don't use that in the backend, but then what they do is they have

software level immutable storage built into their product that would basically

says even if you were a cloud admin of this particular backup product, you would

not be able to using the backup product, um, you know, delete the data, uh, to

basically prematurely expire your backups.

And then also built into the configuration is that there's, again, with the IAM,

there's all these protections so that there's no way to get to the data in the

cloud except through their application.

And then they make the application, you know, uh.

Immutable as I may call.

And so this is the way they make their, their virtual air gap.

So I wanted to ask you, so I agree with these two approaches for how you

achieve protection against cyber attacks,

Yeah.

right?

Which is what, uh, air gap is supposed to provide,

Right.

air gap.

So we talked about controls.

I know some vendors, they offer sort of a mechanism to create a

secure vault for their backups.

Mm-hmm.

Um, these vendors though though mechanisms that they use this because

they are using storage level replication

Right,

is they actually do things like kill the network connection when not needed

right.

and have separate management.

Uh, domains between source and destination and the vault,

Yeah.

You know?

you that

Yeah.

so you're not

I,

this connected all the time

yeah, I, I, yeah.

That's really good.

I, I think this falls into basically a third category, which is a simulated.

A simulated air gap.

Right.

Um, even, even more virtual than the, well, it's, it's, it's

simulated in that they're doing their best, like, like you said, like

shutting off the, the connection.

Uh, so that at least when they're not actively replicating that

the, that there is literally no connection to that, to that device.

Yeah.

That is, that is another topic.

That is another method.

Yeah.

I like that.

I do wonder, so just since we're talking about that, I wonder how it is if you go

through like Tor to then connect to your backup infrastructure, your vault, right?

The onion router, if you will.

Yeah.

Yeah.

Well, I know some, some companies you, you open up a Tor browser and

you're fired, but, um, um, yeah, i'm not a dark web guy, so, uh, and,

and I think that's a good thing.

But, but because of that, I, I, I have no, I have no opinion on that.

Yeah.

Yeah.

But so regardless though, I think it is important people to understand when

companies today talk about air gaps, I.

of the time, it's probably referring to virtual air gaps where it's relying

on something like immutability or IAM or network connectivity order

to provide some form of isolation.

It may not be as perfect as what we had initially with tapes, but it still

satisfies some of those use cases that are needed to, uh, handle cyber attacks.

Yeah.

Uh, yeah.

So unless they're using the word tape, then yeah.

It's, it's, it's gonna be, it's gonna be virtual.

but it's even hard though because as we talked about in one of

the previous episodes, right, about tapes and is it dead?

Right?

Hyperscalers are using tape.

You don't

Yep.

they're like, what is actually being done.

And so for you to figure out is something truly air gapped or not,

may be even more difficult these days.

Yeah, it's the, you know, the old phrase of on the internet, nobody

knows if you're a dog, you know, nobody knows if you're a tape.

Yeah.

Um, but yeah, so hopefully that helps understand like, what,

what, what was an air gap?

What, why was it there?

And.

Just, you know, when you're comparing it, just don't any

more than the term immutable.

Don't just take a term and go, oh, it's air gapped.

Right?

What?

How is it air gapped?

How is it air gabbed as I make air quotes?

Air quoted air gap, um, ask questions, understand what they're doing, and then,

you know, and, uh, unless they're making a copy and putting it on tape and then

handing it to a man in a van, uh, it's not really air gapped, at least in terms of

not the, not the OG as the kits would say.

I think maybe that's the thing to think about too, is.

you really need that level of protection?

Yes.

I'm sorry.

I got, I got excited.

Or what are you looking to solve?

And a lot of that comes down to what are the needs of your business and

what is the impact from a cyber attack and how you're protecting yourself.

Exactly.

Exactly.

All right.

And on that note, I will say thank you once again, sir. For, uh, well first for,

you know, goading me to get my taxes done.

And second, we're doing another good episode of the.

I, I like air gaps.

Uh, yeah.

It's definitely something that comes up and think I'm getting to that

point in my career where when people misuse a phrase, it kind of irks me.

I'm like,

Yeah, yeah,

So.

yeah.

Uh, I'll just say this.

It is because it is because you're older, right?

Even though you're not as old as me, you're still older than,

you know, these whipper snapper.

And I got, I had this image.

I was, I was at Lowe's yesterday and I was checking out.

I was just checking, you know, I just had some, I was buying

some screws and then, uh, I.

Then I, I got in, I got in line and there was this guy that was

older than me and they only had the self-checkout lines open.

There were, there was no actual checkout.

He was like, can't even, can't even hire anybody to run the registers anymore.

I'm like, oh my God, dude.

Like,

it was just, it made me laugh.

Um, anyway.

All right, well that is a wrap.

The backup wrap up is written, recorded, and produced by me w Curtis Preston.

If you need backup or Dr. Consulting content generation or expert witness

work, check out backup central.com.

You can also find links from my O'Reilly Books on the same website.

Remember, this is an independent podcast and any opinions that

you hear are those of the speaker and not necessarily an employer.

Thanks for listening.