What actually is an air gap isn't even possible with backups

stored on disk or in the cloud.

Your backup product says that your backups are air gapped.

By what standard can you even judge that statement?

The answer to all of these questions goes back to my earliest days as a

backup admin back in the early nineties.

You know, back when we had an actual air gap.

Once, you know, what a true air gap was like.

I think I can explain how to use that as a standard to judge the

virtual air gaps we have today.

Hi, I'm W.

Curtis Preston an AKA Mr.

Backup.

I've been a backup admin, consultant, analyst, and even

a backup product evangelist.

Backup is kind of my thing.

And backup folks are my people.

This podcast turns unappreciated backup admins into cyber recovery heroes.

This is The Backup Wrap-Up.

Welcome to the show and thanks for joining.

Con, how's it going, Prasanna?

I've been practicing my Spanish as you know.

so, so for those that don't know, what I said, I just said with me as always, is

my friend Prasanna Malaiyandi, who's been encouraging me in my language attempt.

I'm spending a lot of time on the past tense, uh, and also on.

Birthdays and ordering food

Ordering food is important, you know.

ordering.

Yeah.

The next, the next lesson is about, um, uh, renting a car.

Ah, there you go.

also very important.

As you can see, they, they tend to, uh, focus on things that you

might do while, while traveling,

is good, right?

Because it's important to have those skills.

Exactly.

Uh, so I wanna jump right into our news section and I, and I, I want to thank you.

You found this story and it's from the National Cybersecurity Center.

That center with an re in, uh, the uk.

Um,

That should have given it away.

It's either gonna be the UK or Canada.

So one of the two.

yeah, uh, well, it could, could be Australia, we could, you know,

could be that, but they put out, um, they have some new principles to

make cloud backups more resilient.

Why do you, why do you think they would've done that?

Uh, because there's a lot of issues going around right now with, uh,

people storing data in the cloud.

Uh, but it's still being exposed either due to security issues or

just improperly securing it, such that when you get hit by ransomware.

They go and trash your backups and now you have nothing you can restore from.

I think it goes back to what we've always talked about, Curtis, which is everyone

thinks the cloud is magical and it's just gonna alleviate all their problems.

I actually think that the cloud is like the best place where you can put your

backups, and we could, I, I'd love to have somebody on that thinks that, that,

well, I, I'm st I'm sticking with that.

Uh,

a good place to put it.

Yes.

I think it's the best place, uh, you know, with, with caveats.

But, but, but it is not magic.

Um, it does allow you to do things it simply aren't possible in a data center,

which is why I have that opinion.

But it, I'm talking about like cloud DR and stuff like that, but it's not magic

and you have to, and I'm, I'm really glad to see them sort of acknowledge that and

to give specific guidance on the use of,

cloud.

Cloud four backups now, you know, throw, throw out your, what?

What's your thing?

What, what are you saying?

You like, you don't, you don't agree with me that it's the best place for backups?

I agree as long as you're okay if you need to pull the data back down,

for example, in your home, right?

I know we've talked about this.

You have issues running into your data cap

mm-Hmm

right at home.

Imagine if you had something happened.

You need to download all your data from the cloud because that's the

only place you have your data.

One, how long would it take you?

Two.

How many times would you end up hitting your data cap?

Limit data usage limit

mm-Hmm

And is that reasonable for keeping your only copy in the cloud?

I, that's why I think there are cases where cloud is perfect for keeping

a copy because it is offsite, right?

You don't have to worry about it.

It's all there.

It's really low cost.

I think for important data and other things like that, it might be beneficial

to keep something local as well.

That was my only point.

You said the best place for backup?

After everything you just said, I'm sticking by my statement.

I never, and I never said, don't have another copy.

I.

I'm just saying if I, if I was picking one and only one place, which I don't

think you should do, but if I was picking one and only one place, I would much

rather have it in the cloud than on a device sitting next to my computer.

I.

Uh, or in a dish drive, right?

Plugged in, all that, all that stuff, right?

I would just, and, and I, and I am thinking a lot about home users there.

I really feel this way for home users.

And, uh, the bigger you get, the the more challenging it becomes.

But the, and again, I don't have any issue with having a

local copy for quick restores.

I just really think, like, again, if I only had one, only having a

local copy is a really bad idea.

I

So having a, we want a remote copy, and I think that that remote

copy should be in the cloud.

I do not think it should be discs in a dis array, you know, in your data center.

We can talk about, you know, again, I, I'm pro tape.

I, I like tape.

Uh, again, I wouldn't mind it being one of the copies.

Yeah.

I hope you don't need to use it because it's gonna take a while to get it back.

But, uh, so, so, so

Okay.

I'm good.

Yeah, yeah, yeah.

I'm good now.

Just wanna make sure that anyone listening does not think that Mr.

Backup is not Yeah.

Is just saying just go to the cloud for everything.

yeah.

I'm not, I'm not, um.

I'm not against local copies.

Um, I just, I'm against that as your only option and when, yeah.

Okay.

Um, so, and of all the options, I still prefer the, the remote again,

I still think it's the best option.

All of the options have downsides.

Right.

But the downside to the cloud, if done right, which

is what this article is about.

Is that at least it wouldn't get deleted on you.

Right.

But, but like the, the local one, it could also be done poorly, which

is what this article is about.

So this is a, this is an interesting, uh, so let's just review real quick.

'cause it's, it's, it's kind of, we're, we're gonna come back to that.

They have five principles for ransomware resilient cloud backups.

The first principle is it should be resilient to destructive actions.

Right?

I I like that.

Right?

Um, that, yeah, that, that you should, that deletions

shouldn't really be deletions.

There should be soft delete and things like that.

A backup system shouldn't, should be configured so that it's, it's not

possible to deny all customer access.

So if you lose your internet service, right.

We'll come back to that.

Yeah.

We'll come back to that.

But basically when your infrastructure is down, um, I.

It shouldn't rely on your infrastructure to get in that this, we have talked

about this in that I do think that your authentication authorization system for

your backup system should be separate.

Uh, principle three, the service allows a customer to restore from a backup version

even if later versions become corrupted.

This, I mean, this should be, uh, this is what backups are.

You should always go back to previous versions, but I think, again,

this is talking about bad design.

In previous episodes, we've talked about the difference between, uh,

um, replication, and one of the problems with replication is that

the, the backup becomes corrupted.

I think what they're saying here is, yeah, it's nice to have a copy in the

cloud, but if that copy gets corrupted, hopefully you have other versions of that.

Uh, four robust key management for data at rest.

Uh, yes, I'm with that.

Encryption, encryption, encryption.

Three rules of cloud backups, uh, and then also alerts, uh, triggered

if significant changes are made.

I like that a lot.

So we're gonna, we're gonna back to this.

What, go ahead.

Yeah, no, I, I was reading it.

I was like, that's a lot of what we talked about, but also some bits that we

don't always talk about on the podcast.

I know maybe when we had snorkel 42, we might have touched on some of these.

But

Yeah.

yeah, it was a good list.

And the other thing I liked about this article is they also split

it out into sort of, sort of how do you deal with ransomware?

Like how do you build your infrastructure?

And then also.

Yeah, A lot of times we struggle because sometimes it's, Hey, for

enterprises, you have all the budget, you have all the tools.

You have all the resources like people and expertise, but

smaller businesses you don't.

And so as part of the set of articles, they also publish one for like SMBs

and how to protect your environment.

Right.

Yeah.

And speaking of which, um, they said that this article, it specifically said,

we're not gonna tell you how to back up.

Right?

They're just saying there are a bunch of different ways to get a copy in the cloud.

They're just saying if you're gonna have a copy in the cloud,

you need to make sure that it.

Is protected, it's resilient against ransomware attacks because as we have

often mentioned, the, your backups are just as big a target, if not a bigger

target than the your primary copy.

In fact, the, you know, I, I mentioned it on a blog I wrote the other day,

was that, that that Veeam uh, survey that, you know, I call it the Veeam

survey, but they actually, uh, and I don't think we explained enough when

we did the episode on it that it was a.

Double blind survey, uh, of like a thousand companies, I think.

And they said that, uh, and they weren't.

I, I, I think I may have assumed that it was Veeam customers.

It wasn't Veeam customers that it was something like I.

Uh, that backups were targeted in like 85% of the attacks.

So you've got to do this.

And that is, we're gonna make a, a quick switch.

Uh, we're, we're gonna, we're gonna call that the news section.

And that's the news.

And then we're gonna just move right into what I wanted to talk about.

It just, it, it's, you know, I, we were looking for news articles and

you found this article and it's absolutely perfect for what I wanted

to talk about on this episode.

We've talked about this before, but I want to talk about it in a different way

now, and that is this concept of air gap.

So I wanted to take people, especially people like you, persona,

Back in the day.

that never, that never touched the tape.

Uh, you know, you never fired a tape in anger to, to back

when we had an actual air gap.

That the term has a very specific.

Definition, a history, a connotation.

And it comes from back in the day.

So again, for those of you that are, that are NIT, and you're, you know,

maybe you've grown up in this world of disc only backups, that is a lot of you.

And so I wanted to just give you a, like a, a lesson of what

it was like to do backups back.

In, in the, in the mid nineties.

Well,

What were you, what were you doing in the mid nineties?

Persona

I might have been in elementary school.

element Elementary school.

No, probably middle

That's great.

That's great.

Uh, well, it's okay.

I'm, I'm, I'm feeling, I'm actually feeling young today

because I was talking to.

Uh, our friend of the pod Stewart, and he was telling me how he joined

the Air Force, like he enlisted in the Air Force the day before

his draft number was called.

Um, and so, which means he is a lot older than me, is

You just need to hang around with people who are older than you all the time,

and that way you always feel young.

That's why I hang out with

to feel young.

Yeah, yeah, exactly.

Nice.

Nice.

I see what you did there.

let me go back to the, you know, back to the time and what we had at our,

at our sort of technological height in that data center was we had a bunch of.

Spectra Logic Tape Libraries.

They were, they were, their, their, I think they call 'em, they

started calling 'em Tree Frogs.

We didn't call 'em that, but basically they were like, I

don't know how many u was that?

Like three U High?

Something like that.

And inside was a carousel and they were, I think four DDS

tape drives in that carousel.

Then the robot rotated like in the middle.

It didn't have to move, it just rotated.

And it would take a, it would take a tape from a slot on that carousel

and then slightly rotate and then put that tape in the tape drive.

And then there was a door on the front that was basically the ejection port.

Right?

Uh, similar.

Uh, you know, fancier more expensive units like from storage tech.

You again, you, you had a robot that was in the middle that would

turn around and then we grab the tapes and put 'em in the tape drive.

But then you actually had like, um, you had a, a, a cartridge, I dunno what the,

I dunno what we called that, but basically there was this unit that sat in the door

that the robot could actually put like 10 tapes in that, in that removable.

Carrier,

I don't, I don't remember what

we called it.

And we'd take it out and there would be like 10 tapes in that, and then you could,

you could move that around as a unit.

Um, but, but basically the principle was the same that you, you had a robot that

moved the tapes around and then you had the ability to eject specific tapes.

And the way we did it, again, proper backup design, you always have two copies,

so we would make a backup copy two.

Tapes that were in that tape library, and then we would copy those tapes or those

backups to other tapes, which then at the end of the night, we would then spit

those tapes out into that ejection port.

And then we would have, uh, if I could, I don't remember how many tapes it was,

but it, I, I know that it fit in like a, a storage bin that was like six inch.

What's that?

like I was gonna say, like a cardboard box.

No, no, it was a plastic, it was like one of those plastic things

with the lids that, that, that, that, yeah, the totes that the,

the lid is like part of the unit.

Right.

So you, you'd open it up, you'd put the tapes in.

Uh, I don't even think we, um, I mean, you would put the tapes inside.

There was a, a holder that each tape was held, so that tape, the tape

itself physically secure, and then you put those tapes inside a, um.

This, this tote and then you would, um, we had a barcode scanner and I, I don't

know, so I'll just stop there 'cause this is, you know, it's kind of a long story.

So does any of that, did you learn anything from any of that?

Yeah, you basically had two copies.

And they were never always in the device itself that was easily accessible.

It was kind of stored separately, and then you had one copy separated out

that you could keep somewhere else if you decided to move it off site or

whatever else you wanted to do with it.

Right, right.

So again, the key is again, separate these two copies as much as you can.

So we would put those copies and, and every tape had a barcode

and we had a barcode scanner, and we had a database that.

Um, it was an Informix database.

We had a custom built app where I could scan all of the barcodes of all of the

tapes that were going into today's tote.

And then that would create like a pick list.

And, um, that would, well it's not really a pick list, but it,

it was a list of the tapes that, that were going in today's tote.

And then that list would go in the, the, the, the tote.

And then we had a guy from Iron Mountain.

Um, sometimes the guy was, was a girl, right.

Sometimes it wasn't always, it

It was a man in the

Sometimes it was a woman.

Yeah, it was a, but yeah, we used the term man in the van.

There was a man in the van that came and picked up our tapes and they

would, um, they would scan the tapes, like as they're picking them up.

So we had like receipts.

We had like an electronic receipt that they had picked up our

tapes, and then when they got to the actual physical location.

By the way, here's a little piece of trivia.

How do you, it's a, it's a trick question really.

How do you spot Iron Mountain Vans with tapes in them?

The giant logo on the side.

No.

See, I told you it was a trick question.

If Iron Mountain, if I, if you have vans that have the Iron Mountain

logo that has paper inside.

The, the tape, the, the vans that transported media did not have,

they were not branded at all.

And then they would go to the Iron Mountain facility and then

the, the, you could, there, there were two ways to do it.

We did it the more secure way you could put a barcode on the tote and

just not let them open the tote.

But what we did was we had them open the tote and then scan each barcode.

Into a shelf that it had a slot.

Right?

And they would scan, like there was a barcode next to the slot, you know,

And then there was a barcode on the tape and they would scan it.

So they could, they could, we could say we need, we need tape

number, you know, A, B, C 1, 2, 3.

And they had a computer that would tell 'em exactly where that was in the vault

and they could send us just that tape.

'cause the other way is if you needed a tape that was in a particular tote,

you had to bring back the entire tote.

So they would scan in each of these tapes, and then we got an elect.

We had an electronic connection to, it was very cutting edge really

for, for, for early nineties.

But we had, uh, an electronic connection to Iron Mountain.

We would get notification that tapes had been scanned in, and then

we had a system that would, um.

Basically double check their list of tapes that have been scanned in to our list of

tapes that they should have scanned in.

And one time out of a thousand

There's a

might be a discrepancy, there would be a missing, uh, tape.

And it was always found, but it was, it was, you know, we, we were

notified immediately that they did not know where one of our tapes were.

So again, the principle there is to find out.

The problem before you need the tape.

Exactly.

Right.

Um, that, that's pretty cool.

Don't you think of like all that stuff that we did back in the day.

Yes, and I would not wanna be the person maintaining that system of

just data transfer back and forth and trying to keep these things in sync.

I.

Yeah.

Um, and then of course they had the reverse process when they would, they

had to scan the tapes out of their vault into a tote, and then, and then

we would scan them back into our system.

Right.

And then they just, the, the barcodes didn't change.

The barcode was part of the actual tape.

And then, um, it would just go into the, essentially back into a drawer.

Um.

or whatever else.

Then reused.

Right.

And the, the, the key, and then the other part, and I think, I think I've

talked to you about this before, is we then did pen testing against our own

Oh yeah.

Right?

So that's, that's penetration testing.

So we did physical penetration testing, we.

Uh, would send people that weren't authorized to be in the vault, to go over

to the vault to see if they could get in.

They always, they always had some crazy story.

Um, and we, you know, we had rules.

It was like, you, you just can't, you can't ever let someone who

isn't on the list into the vault.

Right.

Um, and we would concoct stories to,

they need to be in

um, of why we need to be there.

So we would either send.

A person who wasn't authorized to be there, or we would send somebody that

they knew like me and I'm over there with this massive inconvenient like,

pick list and it's gonna take hours.

Uh, you know, and it's one tape, you know, out of, you know, and the

idea was what I'm trying to get them to do is to leave me alone in the

vault with other people's media.

Um, that never happened.

I'm,

I'm glad, I'm glad to tell you that not that never happened.

Um, and we didn't ever, I, I, I, as I recall, like our, our, none of

our pen tests ever actually worked.

Right.

But we, but we, but we

You tried?

on a semi-regular basis.

Yeah.

Um, but the, um, I'm just trying to think if there's anything, any

element of that, that you know, and so when we needed a tape.

We sent an electronic request, I think we, we could probably call, we could

probably call and we could say, Hey, we need tape, we need tape A, B, C, 1, 2, 3.

And they would, um, they would then bring that, that, that tape back.

Um, I will say that this process was not perfect.

I, I think our process was as good as it could have been.

We knew when our tapes got scanned in, we knew when our tapes got scanned out.

We knew when they were in transit.

We knew, um, you know, we, we just basically knew and, and there were

glitches where sometimes there would be a tape that would be in

limbo and it always got found right.

Um, not everybody had that level of, um, what's

Integration sophistication.

Yeah, sophistication, I think would be a great word because

sometimes, especially people that went by the tote method, right?

They just put a bunch of tapes in the tote, they don't really have

any tracking for individual tapes.

I remember, um, at a consulting company that I used to work at that

they got a box of tapes, um, from.

A leading media management storage company, and it, it wasn't their tapes.

And, and then they called their rep for this company and the rep, like, typed

and said, you know, looked up the, like, the barcodes of the tapes and whatnot.

And, uh, the rep said.

Uh, yeah, I don't know who those tapes are.

Go.

You can go ahead and keep 'em and

What.

Wow.

like I said, it wasn't perfect.

Uh, so you could misconfigure things back then just like

you misconfigure things now.

But the key takeaway here is that when the feces hits the

rotary oscillator, the, the.

The tapes, the backups were in a physically separate location using a

completely different authentication and authorization system.

Right?

You couldn't hack them to save your life.

There was no way, there was literally no process to like there, send an

electronic request to have those tapes to anything to be done with those tapes.

It was always a human in the way.

could, could they not?

Sorry?

Could a mischievous hacker

Mm-Hmm.

fake up an electronic request to request all your tapes back?

They could, but those requests were always verified in person.

They were very, they were very, very rare.

Anything, anything outside of the tapes coming back at

their normal expiration date.

'cause that, that's what I was describing earlier was, you know,

we had like a six week retention.

At six weeks, a box would come back and we would send them a new box.

So we had six weeks worth of, of tapes over there.

Anything outside of that was really, really rare

and, uh, had all kinds of controls put around it to make sure that a single rogue

employee, uh, can't do what, you know, uh, again, you had, you know what, what?

You know, we call it, um, like four eyes, um, authentication, right?

You had to have two people do it and, and things like that.

Um, but the, the key here it is just that we talk about this phrase, air gap

just gets thrown around, uh, so much.

And so I just, I thought it would be interesting to just say

that is the standard by which I.

Measuring something that is calling itself an air gap.

Which I think totally makes sense.

Now I have a question though, for, so imagine that you weren't shipping

the tapes off to Iron Mountain,

Mm-Hmm

right?

You had.

Your tape library, it was creating tapes.

It would pull the tape out, put it in a separate spot.

Do you consider that air gap or based on

your

because it's in the same, because it's in the same place as the primary.

If it's just pulled out.

If it's just like, uh, if let's say, 'cause I, I've seen people do this.

They've got two robots and, uh, I remember like having, um, some people would have

a tape library in this building and a tape library in this building, and they

had a fiber channel san, and so they had enough bandwidth that they could copy from

this tape library to that tape library, and they thought of that as an air gap.

And I'm like, I can sit here on my keyboard and delete every one of

those tapes over in that tape library.

That's not an air gap.

The, yeah, so the reason I bring this up is I think going back to what you're

saying, a lot of people think air gap just means no physical connectivity.

It's offline, it's not accessible.

Right?

Where, and I think what you're saying is that's part of the definition,

but really the other part is you have the controls in place.

You have a separate sort of communications channel and con to be able to pull the

data back and other things like that.

Right.

Yeah.

So, you know, you and I were joking about little Indian and Big Indian, the, uh,

which is spelled with an E by the way, for those that don't know what that is,

uh, this is like, that, it is like a little air gap and a big air gap, right?

So like a, a true air gap means that it's somewhere else, right?

Um, not just a six inch gap of air sitting in a drawer.

Again, I would have no problem.

Having tapes sitting there in a drawer.

Like if you don't have a big enough tape library and you've got tapes that are,

that's your on-prem copy, but your other copy needs to be in another location.

I'm just wondering though, like industry de, I know this is Curtis's definition of

what an air gap is to solve the problems that existed back in the nineties.

Um.

Industry terms though today, do you think that the industry follows that

same terminology or that same thought when they think about an air gap?

Because I would say that most people, at least like when I heard the term air gap,

right, it was really around that there's no direct access connectivity to the data.

So if you are, so I've seen some vendors who would say,

look, I don't have any like.

One, uh, thing is like a skiff, Right,

Right, right.

infrastructure framework, whatever it is, right?

It's basically a secure environment where there's no connectivity outside, right?

And they call that

you know, in terms of, yeah, in, in terms of, you know, industry

definition, there's basically two groups of people in the industry.

There's vendors and then there's people like me, right?

No one like me would define an air gap the way that you're talking about, right?

I mean, maybe, you know, we, we, we could talk about it.

An air gapped system that is sitting there in the data center that isn't

physically connected to anything.

but that, what's the point of it?

I, I, well, I, I knew an air gap system.

Well, it wasn't actually air gapped, it was just, again,

electronically, air gapped.

Uh, when I did work, uh, for the large internet retail company, uh,

their, their payment processing system was air gapped in a lot of ways.

Right.

But it still, in order to work it, it wasn't actually air

gapped, it was just Right.

But I couldn't talk to it in any, in any other way.

Well, and I think that's important because as our listeners are probably

trying to read vendor brochures and other things and trying to understand

like what is air gap versus what's not.

Because a lot of vendors, like you said, throw out the term,

oh, we're air gaped, right?

So I think it's important to understand why you need air gaping.

And what problems you it is solving for.

So then you can evaluate is it truly an air gap or not?

And I think you gave a good example earlier on in this

episode about, Hey, here's why.

What Air Gap solved for me?

Yeah, so I, I think that a properly designed backup and DR system, one of

the copies needs to be in, um, another physical location, and it needs to

be air gapped and separated from the primary in as many ways as possible.

I'm not asking anybody, and I know some people still make tape copies

and I, I have no problem with that, but I'm not asking large companies

to start going out and buying big tape libraries and, and copying it,

although I'm sure our friends at, at, you know, IBM and Spectra Logic and all

these companies, and Fujifilm would be very, very happy for you to do that.

Um, I'm just saying that we, we take that as a standard, this physically

separate place where I have to go through a different process.

And again, the, the principles to take away from that are that normal

tape rotation was fine, that just

Normal retention, right?

yeah, normal.

Which basically in modern day term would be normal retention backups being

deleted automatically by your backup system after your retention period

expires should just happen where.

The alarms should go off and the protection, uh, goes up is when you

are transferring those backups back or deleting those backups prior to

any other normal, uh, timeframe, and we have to protect against.

That in as many ways as possible and as many ways as, as you can

that we're like what we used to do.

Right.

Um, and again, you look at, so I, I, so I thought it'd be great to

revisit this, this, the, the news here from the, the uk and again,

this is from the national cyber.

Security center, by the way, cyber spelled with an e, ER, but center is with an RE.

I don't know what's that about.

Anyway, so, uh, the, you, do you want to tackle the first principle?

so the first principle is backups should be resilient to destructive actions, which

I think is what you just said, right?

So.

You wanna make sure that anytime you are trying to delete the backup

after it's been created, before the retention policy goes off, right?

You wanna make sure that that's not allowed.

Um, and then the other thing that they also mentioned is offering a

soft delete mechanism where it's sort of, it goes away, the system thinks

it goes away, but it still exists.

And this allows you to recover in case you actually need it.

Um.

And then if you are doing any deletion or alteration request, right, you

delay the implementation of it.

So if someone says, Hey, I want to delete everything older than 30 days, you don't

allow that to happen for say, two weeks, as well as alerting when that happens.

The, the other thing that's part of that is the, the forbidding destructive

requests, right from customer accounts.

What I like here is.

All.

So here's a phrase I'm reading from this.

All exceptional destructive requests.

Again, going back to what I said before, things out of the norm.

All exceptional destructive requests must be authorized out of band using

a pre-agreed upon mechanism between the customer and the backup service.

So it's okay to create a backup system that allows, that allows for this, but.

It needs to not just be somebody pointing and clicking, right

Something, issuing an API call and then the, the data just gets deleted.

It needs to be a conversation between two people that know each other,

and you can, you can put all of the protections, again, just like the

protections that I talked about back then.

You can say, Hey, I have this, I have this security phrase.

I, you know, you can have multiple security phrases, and I need to

give you the name of my dog that's named after a Indian sweet treat.

Or you could say, Curtis is Guapo

curtis is guapo.

Yeah.

Um, I remember, uh, I remember my, uh, when we had, when we had

a a, an arm, a security company for the, for the house here.

I remember that our passcode was lumpia.

Um, that was, that was our security pass phrase.

Like when, if you had a a false alarm.

This was the, everything's fine.

And they're like, what is your passcode?

And you're like,

Hmm.

okay.

Um, and yeah, there, there could also be a distress code, which our, our passcode is.

I don't know.

I don't know some other thing, but yeah, it's out of band.

And again, because of ai, because of the ability to mimic speech and, uh, you need

to have, you know, multiple, you need to basically do things that can't, that

aren't stored digitally anywhere, that can't be stolen and then used against you.

So you need a, you need a passcode, right?

don't leave the passcode in your email box.

Exactly right.

Well, don't ever, don't ever put it, don't ever put it in your email box.

Right.

Um,

just have, you need to have a conversation with a real person.

Yeah.

So I know we've talked about password managers in the past.

Would you put that pass phrase in your password manager?

Hmm.

Um, I might,

I would say no.

I.

Yeah, you, uh, yeah, we could have a separate conversation about

that, but you, you shouldn't.

It's just a question of, it's the whole, like, this is like that episode that

we had of like, how do you do things when you, when you've lost everything.

Right.

Um, you need a, you need a fail safe place.

Um, all right, we, we could talk about that all day.

So the next thing is a backup system shouldn't be configured so that it's

possible to deny all customer access.

Um, and what this, the way I'm interpreting this is making sure

that if active directory in your site goes down, you don't lose,

um, access to your backup system because it's using active directory.

Yeah.

Or if the customer is able to compromise your policies, right?

You wanna make sure it's not all tied to a single account.

Exactly, and we've talked about this quite a bit.

Please don't use active directory, um, you know, as your password

management system for, for critical infrastructure like this, the, um.

We, we had, by the way, you, you may recall when, when we had that, that

person that had a DR scenario, right?

And they were in an island and the, you know, um, remember

we, we didn't say the island.

It was the island in the Caribbean that they went there after hurricane.

And one of the problems they had was that their backup systems

relied on active directory.

That was where?

In the

Yep.

Which they had no

Yeah.

So, yeah.

So don't do that, right?

That's principle two.

You wanna talk about

principle

principle three is making sure that you could restore from a backup version,

even if other versions are corrupted.

And this is like we've talked about, right?

You get hit with ransomware, it's gonna start corrupting data.

And you might not notice it for 20 days.

And so you wanna make sure that you can go back and restore from a version,

even though the newer versions are old.

So making sure that you have a way to.

Keep those backup versions.

I know in a previous episode we talked about replication and why replication

is not great for backups, right?

So making sure that you have that, making sure there's a mechanism to test.

I know that's been one of the things that we always talk about

is verify your backups because a non verified backup is useless

. This is one that to me, as a backup person, I'm saying, well, yeah,

duh, but, but, but not everybody has versions in their backup.

Your backup has to have versions, right.

It, it can't be just a replicated copy of the most recent transactions.

It's got to have the ability to go back in time.

And, and this is more, more true now than ever before.

You've got to be able to go because they could, they could

corrupt both your primary.

And your backup

copy.

And so just make sure you have that.

And then also make sure you have a retention period, right?

Or say that you are gonna store a fixed number of backups based on time,

rather than number of backups, right?

So

Rather than just number of versions.

Yep.

and just be flexible in having different storage policies.

Not everything needs to be kept for the same amount of time.

And just say, okay, I don't need all my copies to be stored for

six years or six months even.

Maybe I only need dailies for a month, and then after that I can do weeklys.

So allow these flexible policies because that'll make it more flexible

and allow you to keep data for longer periods of time as well.

Exactly.

Uh, the next principle is robust key management for data at rest.

Protection is in use.

So yeah, if it's backups, it needs to be encrypted and you need a

robust key management system.

I.

That allows you to do things like rotate keys, delete keys.

Um, also they talk about offering an out ofAnd, uh, key backup option right there.

There are companies that will actually do key escrow for you, and this is again,

that you need a doomsday copy of that.

You need a you, you need the, the way to basically bring

in a key management system.

Again, think about everything going wrong, and again, when your primary

goes down, you don't want your cloud backup system, for example, to rely on

the key management system in your data

center.

that

Right.

Um, yeah, that would be bad.

So that, that's a relatively easy one.

Go

So going back to the previous story you told about that company

you worked for that ended up getting the wrong tote of tapes,

Yeah.

did they ever check the tapes to see if they were The data was encrypted.

I didn't get to, probably not back in that timeframe.

Right.

So yeah, so even if you are using tapes, make sure you encrypt your data.

It shouldn't just be for the cloud copies.

And I will say encryption done properly, um, doesn't slow down

your backups, so, um, so by the way, oh, what I will say this.

Dedupe, then encrypt, don't encrypt, then ddu, because dedupe

works by looking for patterns.

Encryption works by getting rid of them.

So, uh, you gotta do that in the right order.

All right, final principle

Is alerts, right?

This is super important that whenever significant changes are made, that

you have some alerting mechanisms so you understand what's going on.

Significant could be things like someone went and added a new user, or they're

trying to change the retention policy.

You wanna make sure that you can catch these as early as you can

to make sure that there's nothing funky going on in your environment.

the, the key here is, you know, just when something.

Out of band or out, out of the norm is happening, especially

when it's a high risk thing like deleting backups or restores.

I, I don't know how, you know, how you've seen it, but I, nobody restores anything.

Right.

I mean, I mean, it's like we make all these backups and they very,

very rarely restore data and.

So when a restore kicks off, that should be a high risk

alert that is going off saying,

Hey, there is this restore going on.

And you're all like, oh, yeah, yeah, we're doing the restore the thing.

It's, everything's fine.

But if you see this big alert that's going on, there's a big old restore going on.

And no one knows who's kicked off the restore.

You can do something about it at that point.

Right.

Um, there was, um, you know, we had a, a cyber expert on the podcast a couple

months ago and he talked, remember how he said he loves backup systems?

'cause 'cause he loves to use them to, to steal data.

We're like, oh, that's really depressing.

I think, I think another one that they call out that I think doesn't get

enough focus is people stopping backups.

Yes.

Right.

In addition.

Right, because

yeah.

Agreed.

Yeah.

A lot of ransomware actors that'll stop your backups and you may not realize

it for 15 days, and by then you don't have any good backups left because

your old backups have already expired.

Yeah, that's a really good point.

I'm glad you brought that up.

The be because a lot of the reporting that's built in is they

tell you when a backup is done.

They don't tell you when a backup didn't happen.

So yeah, you wanna have, you wanna have, um, uh, reporting kickoff when something

like that happens, stopping your backups.

'cause they could stop your backups for let's say a week if

they, if they're able to do that.

And, uh, and then they can corrupt your data that, you know, and

your oldest copy is a week ago.

You know, you're gonna, you're gonna lose

yeah, you're gonna lose data or you're probably gonna be

more willing to pay the ransom.

right, right.

So again, I, I am not only am I not against cloud copies of the data, I

really like cloud copies of the data.

Um, I, I, I, I want us to be careful with the term air gap.

I wanna make sure, are you doing all of these things?

How are you mimicking the question?

The overarching question is, how am I mimicking?

I.

What Curtis did with a box of tapes in Iron Mountain back 30 years ago,

that is the standard by which your backups should be measured in terms

of protecting them because they were protected without doing it on purpose,

we were, we were protecting it both from natural disasters as well as hacks.

It's just, back then the hacks were very, very uncommon.

Uh, but now the hacks are the primary reason that we're doing restores.

so this might be a controversial question and we don't have to answer it.

Based on everything you've said and what you're looking to solve

with air gaps, would you call cloud data protection vendors?

Air gap.

I think that, again, have they separated their data from your data?

So, I'll, I'll just say this at best.

At best, I'm gonna call all of these guys electronically, air

gapped or virtually air gapped.

None of them are actually air gapped.

Okay.

Um, because they're all running in a computer that's connected to

something that's, that's a, that's, that's the only real air gap.

So at best, I'm gonna call them virtually air GAed.

And so it's, it's a standard by which we measure something.

And so my question is, how close to you are that, do you have

a separate authentication and authorization system, right?

Do you have, do you have the ability to, to delete backups?

Like, like, like does the, does the hacker have the ability to delete backups?

Do you have the, you know, all the, all the principles

that they talked about here?

I think if you're following, if they're following the principles found in here.

I think they could be called virtually air gapped.

The, the, the problem is not all of them

do.

Right.

Um, and the, if you look at the, the question is, can I electronically, you

know, delete a bunch of stuff without,

Any checks in place?

Yeah.

Right.

Um, and, and if that stuff gets deleted.

Is it really deleted?

Uh, can I get it back?

Uh, what kind of multi-factor authentication system

do you have in place?

Do you have things like multi-person authentication for big actions?

Like, I like the multi-person.

Some people call it four eyes.

Uh, I like the multi-person.

The multi-system authentication.

Um, you know, again, speaking of standards, it's like the

missile key thing, right?

It's not possible for one person.

To turn both missile keys.

So you, you, you add in all those things and if you've got all those

protection, I think I'd be fine with calling them virtually air gapped.

Um, but some of them don't have those systems just because their cloud doesn't

mean they're doing all these things.

In fact, there's a reason that the UK government came out with these principles,

and that's because they're not always

Yeah.

so well.

Hopefully that was helpful to some people.

Uh,

links to the article in the show description.

yeah.

Yeah, that's a great article.

Uh, by the way, I think, I think what we should do next is

what, what actually immutable

is.

Um, that's another one that we talk about.

All right.

Well, uh, thanks Prasanna for your, uh, your wisdom as always.

I try Curtis, and thank you for the.

Educational lesson on, uh,

From the, from back in the day and always, thanks to our listeners.

That's a wrap

The Backup Wrap-up is written, recorded and produced by me w Curtis Preston.

If you need backup or Dr.

Consulting content generation or expert witness work,

check out backup central.com.

You can also find links from my O'Reilly Books on the same website.

Remember, this is an independent podcast and any opinions that you

hear are those of the speaker.

And not necessarily an employer.

Thanks for listening.