Hi, and welcome to Backup Central's Restore it All podcast.

I'm your host, w Curtis Preston, aka mr.

Backup, and have with me my cloud backup experiment,

collaborator, Prasanna Malaiyandi.

How's it going, Prasanna?

Good, Curtis, and how much have you spent on cloud these days?

Um, I don't know.

I don't know.

I don't know.

I'm up, I'm, I'm probably up to about $20 I think so far, just

because I keep buying little things and, you know, um, and, uh, the.

The, the, the quickest one.

You know, I tried, I did like, I, I tried the rsync.net cuz I,

I know we had those guys on here.

Um, and I will say it was the simplest, right, cuz basically you just get it,

it's a, you know, it's a destination.

You can send stuff to it.

And it was really very straightforward.

Um, the,

But for the amount of data you

had, it didn't make sense.

For the amount of data I had, it was overkill because they have a

minimum charge of 10 bucks a month, which is for 680 gigabytes, which, you know,

it's 10 bucks a month isn't a lot, but.

Um, it just, there are other options that are less expensive than that and, um,

it was just the experiment of, of trying to, I, I'm still, the, the main project

here is trying to figure out what's the best way to properly back up my iphotos.

This all started with a conversation on the podcast and

me re just really realizing.

My, for my own self that iphotos, you know, that I'm not backing up my photos

because, or my, my photos in my iPhone and, uh, it used to be called iphoto

and now they just call it photos and it makes for weird internet posts.

Because Apple or iCloud is not a backup.

It is a synchronization.

Right.

And so, um, I'm, I, I'm on I think experiment number three.

Um, And the one that I think will be the ultimate winner.

I'm not gonna reveal my my thing yet, but I think it'll be the ultimate winner.

And in the midst of this, I tried out a new product called, um, what is it called?

I should have it here on my desktop.

It's called System.

Uh, it's, it's it's system, but it's, it's like, C I S D E M, duplicate finder.

Um, and it has some pretty cool technology in it where it will

actually find similar photos,

hmm.

Not just identical photos, but similar photos and like low

and high-res versions of the same photo.

Um,

you took a burst.

Yeah, well, a b a burst is, it would be, yeah, it

would be nearly identical photos.

Uh, but this will find, so it ended up finding 1500 similar photos, and

then it can, it can smart select all of the dupes for you and delete them.

Uh, and I did that.

Um, anyway, it's been, it's been, uh, it's been a journey, you know, as they

So are your experiment going to be

finished by the time we have Daniel Rosehill on the podcast?

Um, yeah, probably, uh, when, what are we having him on?

How and how many weeks

week,

early next week?

next Wednesday.

Mm.

Might not be.

Cause you know, like everything I try takes about a day.

So we'll see.

We'll see, we'll see, we'll see.

Um, but speaking of modern, you know, hashtag first world problems.

I'm gonna record our usual disclaimer, uh, Prasanna and I

work for different companies.

And, uh, this is not a podcast of either company.

And these are our opinions, not theirs.

Uh, please reach out to us at Curtis Preston at gmail,

or at WC Preston on Twitter.

And, um, And Owen LinkedIn, yes, linkedin.com/in/mrbackup.

And you can find us.

And please also rate us, uh, go to your podcast.

That helps other people find us.

If you like the show, help other people find it by giving us,

you know, stars and whatnot.

So we're in our continued backup to basic series here.

We're in part three, which, uh, the first part was about traditional data sources.

The second is about, uh, is about databases.

But in that episode, we focused mainly on the architecture, the

different types of databases.

The different ways that they sort of exist.

And, uh, this week we're gonna talk about basically the, you know, the

meat and potatoes, which is how you actually back up, uh, databases.

So, um, the.

What's that?

You don't,

said no one ever.

up the databases said no one.

Uh, yeah.

Yeah.

I mean, you know, the, the only ones I, I'd say the only data

databases that I would consider valid not to be backed up are

Test.

database or something like that.

Test, yeah.

Test tester dev.

An actual dev.

Not, you know.

Sometimes dev is not

Yeah.

Yep.

Um, so, and, and I'll say that a lot of the things, in

fact, many of the options that we're gonna talk about really only apply

to if you are hosting the database on a server or VM that you control.

Mm-hmm.

Right.

Um, because, you know, it'll become obvious.

And, and so those of you that, that, you know, you're using RDS

or something like that, um, then.

Um, you know, you're gonna be like, well, that doesn't work for me.

Well, you know, if it doesn't work for you, it doesn't work for you.

Right?

So, do you remember what what we said was the, the main pro, you know, most

databases are residing on files, right?

There are a handful of people out there that like to use

raw devices for databases.

What?

especially with

Right.

Yeah.

Yeah.

Okay.

Most databases that are running as an application inside servers

are really just a bunch of files.

You know, they're stored as

a pile.

Yeah,

system.

Right.

And we talked about last week that you can't just go and back those up.

You were the one that, you know, you brought that up that, um, I might have

just edited that so it's fresh in my mind.

I was like, how do you know

were the one that brought, yeah.

You were the one that brought that up about um, that you can't just

go back up those files, right?

Yeah.

Yeah, because if you did, right, given how databases operated and they're

constantly modifying files, right?

You might get a file at a different point in time than something else, or even

within a file at different points in time, cuz it's not like you're taking a

snapshot and then doing a backup, right?

When you do a

normal file

exactly.

And.

And even if you did take a snapshot, um, that might not be consistent.

Right.

Um, you

Uh,

because

Do

you wanna talk about that now?

might not.

I do.

Yes.

So, so, so yeah.

Having worked

at storage vendors, I, yeah.

Having worked at storage vendors, I know some database companies

are very particular to say that we do not guarantee recovery.

From a snapshot based copy.

However, in based on experience, 95, 90 9% of the time it works.

But there's no guarantee

because of

not.

I know.

I know, I

Well, uh, again, you're, wait, wait, wait.

I'm specifically talking about taking a snapshot without doing

anything special with the database.

Yes.

That's what I'm talking

That's, that's the scenario that I'm, I'm talking about

what is referred to as, uh, well, darn it, we're getting ahead of ourselves.

Okay.

Um, so we're getting, we're getting ahead of ourselves.

All right.

So the, the first, uh, type of database backup is called a cold backup.

Do you want to talk about that?

Yeah, so for a cold backup, it's.

Probably the safest way to do a backup, right?

Because you're basically shutting down the database completely.

You're guaranteed to have no writes happening to your database at all

during this time, and then you're basically taking your copy once you're

done or you take your snapshot and then you back up from there, whatever

mechanism you want to use, right?

Once that's done, then you bring the database back online and then

writes start again until you're.

Hundred percent guaranteed that nothing should be going and

modifying that database file while you're doing your backup.

It has

The downside is

downside,

Prasanna Malaiyandi:

big, yeah, it's great.

Prasanna Malaiyandi:

As long as no one needs to use the database during this process, right?

Prasanna Malaiyandi:

Because you literally are bringing the database down, which means

Prasanna Malaiyandi:

any applications, any clients who are connecting to that

Prasanna Malaiyandi:

database, they're like, bye-bye.

And you know, I've been doing backups a long time and I can't

remember a single client that did this have, have you ever seen anybody do this?

I've never seen anyone do this.

Yeah, because it, it means, it's just, it, it

goes back again back in the day.

Um, the, the Unix, so you, so, you know, I grew up around Unix systems, right?

And the, the command that we used to back up the file systems back then was dump.

Mm-hmm.

Do, do you know what the Dump Man page recommended

when you dump a file system?

no.

This.

It recommended that you unmount the file system

Tit

and then run dump on it.

No one did that.

No.

No one did that.

But this is a very safe way and if you have a, um, you know, if you have a world

where nothing is happening, It at night.

This is a completely viable way to get a solid backup of the database.

Right.

Um, it's just like, you know, like you said, it comes with, with, uh, one

Just a, a small, small curve, a small downside.

Yeah.

Small, small.

Now here.

Now this next one, um, it, it is, should be one that you might remember from.

Your former employer, and I'm generically calling it Split Replica,

you wanna talk about

you know, to which I'm referring to

This is

former employer

my former, former, former, former employer,

starts with an e.

Yes, so this is, yeah, where you basically have a replica

copy, which of course everyone is gonna have a replica of their database because

you need that for availability, or they might be using for availability.

And whenever you need to do a backup, right, you basically stop replicating.

And you basically split the replica.

So the replica is at a point in time you do the backup off of the replica.

Once the backup is done, then you resynchronize and then

continue doing your availability

yeah.

The.

E

EMC had the term b c V for this, right?

The business continuance volume.

Um,

Business

is a, another great way.

Yeah.

It's, yeah.

What did I say?

Business.

Oh, business con.

Yeah.

Um, yeah, I thought it was continuance on, on the BBC V, but,

uh, whatever, it doesn't matter, you

On the symmetric?

Yes.

on the symmetric, yeah.

Been a while.

It's been, it's been a minute since I've done this.

Um.

And this is essentially the same thing as a cold backup, but just

how you get that cold backup, right?

You stop writes for a second.

You, you, you split the replica, et cetera, right?

Um, that's

though, Curtis?

viable way.

Well, the downside, uh, is cost, right?

You know, you actually said everybody's got a replica.

Everybody doesn't have a replica, right?

Everybody has a replica maybe in Salesforce.

Uh, so I'm sorry.

Uh, maybe in.

EMC Dreamland sales, you know, EMC salesperson, dreamland.

But, uh, that's the big, that's the big cost.

And also complexity, right?

I remember lots of scripts.

Um, you know, if the, the split can fail, the rejoin can fail, you know, all

these kind of things, just complexity.

but isn't also another downside, the fact that when

you're, do you now have to decide, do I want a replica copy or do I want a backup?

Right.

For a period of time, it could be one or the other, right?

Because when you split it, you're no longer getting updates, which

means that your protection of that, of your production is now

restricted until your backup is done.

Yep.

Yeah, exactly.

Um, so you're, so, you're,

Prasanna Malaiyandi:

What's more important?

you're, yeah.

Yeah, I, I'd say at that point you're no worse off than people

that don't have a replica, but, which is I think most people.

But, um, I, so I really just go back to the, the downside

of that is cost and complexity.

Most people, at least most people that I've worked with, didn't have

the money to pay for A, B, C V just for the purposes of backup.

Right?

Because then also they would show you another, Right.

Another symmetric to replicate that too.

And, um, yeah.

And, and if, if you needed synchronous and you could, you

could end up with three symmetric by the time you were done, right?

Four copies of the data.

You had the local, the, the primary copy, the B c v, which

was physically attached to that.

Then you had another symmetric that you were replicating to,

uh, symmetrical synchronously.

Yeah, thank you.

Replicating too synchronously.

And then you could replicate asynchronously to another one.

Uh, man that sold a lot of disk back in the day.

srd, F s, and Srd.

S a.

Yep.

Yeah, exactly right.

Um, so the next one is what I would say, my usual way that

I'm backing up databases.

Um, at least in a case of, um, Like Oracle,

Hmm.

Oracle and Informix, which is still around, um, Sybase

to a lesser degree, not SQL server it didn't have this functionality.

But that is this concept of hot backup

Does SAP support it?

I don't know, um, I never had to admin sap, so I don't

know the answer to that question, but

I stumped Backup.

So I'm glad I'm I'll, I'll make that a social post.

Um, the, um, You know, if, if you've ever Oracle admin and Oracle database, there

is this, there is this command, it used to you, it used to be you had to say

alter table space, begin backup, and you had to do this a table space at a time.

And if you don't know what a table space is, it's literally the space

where you put the tables, right?

Um, you know, you have, you have some data files that form a table space and

you put tables in those table spaces.

We talked about that on last week's episode.

Um, But now there's an alter database.

Uh, begin backup command.

And what it does in the case of Oracle, I can speak specifically,

it doesn't halt writes.

I actually thought, I used to think that it halt halted writes.

It doesn't halt writes.

What it does is it changes what it logs in the redo logs and it,

it, it logs, the changed blocks rather than the vector, right?

Because basically it's, it's storing all of the blocks that are changing while the

backups are happening so that when you, uh, go to try to use these oracle knows

that, oh, this, this backup occurred during a ti a period when it was, um,

Inconsistent.

backup mode.

And so it can use those BA blocks to then, um, to then fix the blocks that

change during the backup in case you got them before they were changed.

Right.

Um, and it's a, it's a really cool way in that you just need a pretty basic script.

That you run at the time of backup, you just say, alter database, begin

backup, and then you can back up that database however you want.

We've talked about a couple ways.

You can, you know, just split shot or snapshots.

You can do a regular, uh, file system backup, which is the way

that I've historically done it.

Um, and, and the other nice thing about this is historically you, you quite

often would pay extra for database backup

functional, yeah.

Um, that's, that's become a little bit

less in vogue, um, these days.

But we've kind of gone with capacity base, um, pricing.

But the, it used to be that you would pay a lot extra.

The other, um, The, the di, the disadvantage of this was that the DBAs

couldn't really control backups, right?

So, uh, this was always a war between, you know, those of us that wanted to

have the backup system control the backups or the DBAs control the backups.

Yeah.

Oh man, it's so, I never administered databases, but I visited many, many

customers and it would always be interesting getting the backup folks

and the DBAs into this room at the same time, and as we talk about solutions or

as I would talk about solutions, they'd be like, no, I want control of this.

No, I want control because I don't trust the other guy.

Right.

It's just that back and forth that goes on, and I would say it

Yeah.

It was notorious, especially for Oracle deployments, right?

I think.

It was, I dunno what that was about, but it was, yeah.

And, and, and yet Oracle has, I think, the best answer that makes both sides happy.

Right.

Uh, we're, we're gonna get to that in a minute, but

the one thing

I think that you need to consider as a disadvantage, or just to be careful

of when you are using Hot backup is, I know Curtis, you mentioned sort of

logging those change blocks, right?

This causes additional load on your database, so you wanna make sure that.

Right, that you're not, that you've sized your storage systems

to account for this as well.

So wherever you're storing your logs, your archive, do we do logs?

Yeah, that's a good point.

And it brings up a story.

Um, I know I've told this story on here before, but it's apropo to this moment.

So I was at a very large oil and gas company.

This was very early in my career.

I.

And they had this, this database server that had never been backed up.

And, uh, it was a d s s system decision support system.

Uh, I don't even know what it did, but that's what it was called.

And it got a nightly upload of data from the mainframe.

And then the new data would be there the next day.

Well, because the upload occurred at night, we couldn't do a cold

backup, so I, I found out that it had never been backed up and I'm like,

guys, we have to get this backed up.

And the D B A, he had a couple of really, Weird conceptions

about the way hot backups worked.

Basically, the, the, the, in order to do hot backups in Oracle, the database has

to be running in archive log mode, right?

He had all his databases running in no archive log mode.

Don't even get me started.

So he had this weird belief that if I put the database in archive log

mode, that it would cause corruption.

I'm like, can you please go call Oracle support cuz you know you're crazy talk.

Um, and Oracle assured him that this feature that everyone uses is fine.

But then he had this other thing, what, what you were alluding to was

that putting the database in backup mode was going to cause an increase.

A, a performance decrease on his database.

And so he was going to, uh, the, the, the nightly loads, which were occurring

at the same time as the, the backup would, uh, take significantly longer.

And I ended up betting him that, um, For, for some reason what I'm reme

remembering was that it was 5%, right?

I, I said the backups won't, or the, the, the uploads or ingest of

this data wouldn't take more than 5% longer than it, cuz it was a very

documented process that happened.

Uh, and we, we did it, we put it, we put it in archive log mode.

We ran the first ever backup.

Um, oh, and by the way, um, the, the, the backup product that I was using,

Used, um, some features of Solaris that were well known to be buggy in

the version of the OS that we had.

So there were some patches.

So we had to, I had to load these patches, uh, so I had to load the patches.

I had to put the, uh, it was the, the jumbo patch, the Solaris jumbo patch.

Right.

We had to load the latest one, or else when I ran the database, when I ran the

backup, it would crashed the server.

that would be

I had to put on the jumbo patch.

Yeah.

And, and I, and I had to put it on archive log mode.

And then, um, I got the first ever backup.

And we did the first ever, uh, uh, ingest of this mainframe

data while running a backup.

And it took like 5.2% longer.

And so I ended up buying lunch for this guy, and it, it was

on like a, like a Thursday.

And then over the weekend the system crashed and we lost five disc drives and

I had a backup and, you know, and so I was like, I'm cock of the walk, right?

So I came in Monday morning and, and he's like, I bet, I

bet you think you're something.

I'm like, yeah, I kinda do.

Right?

And he said, well, you get no credit for saving our ass.

I'm like, why?

He goes, well, for two reasons.

He goes, I don't know what caused, uh,

caused the dystrophy.

Yeah.

So I think maybe the extra, the extra, you know, stuff you were

doing was like exercising the disk more than they were used to.

And so they failed because of that.

Or maybe it was those patches you put in either way, I think you caused a failure,

so you get no credit for saving it.

And I was like, whatever dude.

Oh man.

Uh, good times.

Um, so, um, All right, so that's the hot backup mode.

Oh, by the way, the, the other downside, any backup method that involves

writing and maintaining custom scripts,

Oh

it, it is not to be preferred over backup

methods that don't require, right.

Uh, I would prefer you not use a system that that doesn't use custom

because just imagine that someone forgot to add a

database table into the backup script.

And all this time, and it runs for a year and everyone thinks, oh yeah, everything's

backed up until a failure happens.

And it's like, oops, we forgot about that cuz no one told me.

see, this is why my backup scripts queried the tables.

And I got a list of all the table spaces and then I put each table space and

back 'em up cuz I knew what I was doing.

But you are correct.

Not everybody, not

not everyone

does.

Yeah.

Exactly.

All right.

Uh, and then we have snap and sweep and dump and sweep.

All right.

So, um, snap and sweep we've kind of covered, but basically the

idea is, uh, hopefully put your database in backup mode then.

Take a snapshot.

And you're right, a lot of people don't, and it works 99.9% of the time.

And I would say I'm even okay with that as long as we got all the transaction

logs going far back as we need them to go.

Right?

So if last night's backup doesn't work, then right?

We can go to yesterday's backup.

But remember, remember, remember, remember, remember the story

that I told last episode?

This is, this launched my career, so when I hear people go, it works 99%

of the time, I'm like, well, it didn't for me and it didn't for six weeks.

Right.

So this is what I'm saying.

Right.

Anyway, but I didn't use a snapshot.

I was using dump and I didn't unmount the file system.

But yeah, so snap and sweep.

And what is the, what's the sweep Prasanna

Basically taking it and putting it somewhere else, right?

Snapshot just gives you the point in time, but it's on the same system,

which as we all know, is not a backup.

So therefore take it and then put it somewhere else, right?

Move it off box, move it to a different system, right?

Whatever it is, whatever mechanism you use, do that.

So when I, so one of the first projects I worked at, at my former

employee, which starts with an ee.

Was actually, how can you make backups more efficient for these large databases?

Because when you started talking about 20 terabyte or 200 terabyte

database instances, right, at the time, there was no great mechanism to

back that up without sort of causing lots of pain for a lot of people.

And so one of the mechanisms that we created was a product or

a feature called Protect Point, which basically took a snapshot.

An application consistent snapshot on your primary storage.

Use different data movers to actually move that copy from storage to at

the time a data domain appliance.

Right.

So being able to efficiently move it, because the other problem

is once you take that snapshot in, say a normal case, right?

You have your snapshot, now you need to sweep it somewhere else.

It's normally your client who's moving it.

You're now impacting your storage system.

I know we talked about this in a previous podcast, right?

About N D M P and having it done at a storage system makes a lot more sense.

That's what we used to do.

So we actually built a solution to move the data from.

Primary storage.

Write the EMC snapshot to a data domain system directly without

requiring a client in the picture.

Hmm.

It's both pieces of that arch architecture were sold by emc.

That's fascinating.

Um, so, so we'll get the next one is what I think is probably

the most common backup method for databases, which is dump and sweep.

You wanna talk about that?

Yeah, so this is, so most database applications support

a functionality called dump, right?

Which is take my database files and dump it to a location, right?

So rman supports this, right?

Uh, which we'll talk about in a little bit, right?

Other tools do it, and similar to sort of a snapshot and sweep, right?

Dump and sweep is you do a database dump somewhere, and then you sweep it off

and copy the files off somewhere else to get that backup copy, if you will.

Right.

And so this is another mechanism, and you don't need any fancy tools.

You don't need, like this works everywhere, right?

So you don't need to have a storage system that supports snapshots, right?

You could be running it on a server, right?

With local disk, do a local dump and then sweep it off

somewhere else to get your backup.

Yeah, the upsides of this is that it's very

flexible and anybody can do it.

Um, you know, the, and it gives the DBAs that control that they're looking for.

Uh, the downsides are many.

Unfortunately, no one cares, right?

So you have to have twice as much storage.

Right.

Um, you know, you have to have a dump location that you can back it up to.

Now what some people have done is they've backed up directly to a

deduplication appliance, right?

Like a data domain.

They get an NFS mount and they back up to that.

I do not like that method either.

I don't like the backups being visible directly as on an NFS mount

point, but, you know, whatever.

Uh, ransomware cuz of ransomware.

Right?

Um, but the, um, Uh, the other problem is again, that this is going to be done by

some sort of custom script, some sort of scheduling system, which it may or may not

have any decent reporting to it, right?

One of the, one of the main concerns I have with Dump and Sweep in

general is that, um, the, the two sides don't talk to each other.

Right.

So the, the, the sweep is happening.

The backup person is running the sweep and the DBA is running the dump.

What happens if the dump fails?

The backup just keeps running.

Thinks it's all fine.

It's great.

Yeah.

Yeah.

So what I've done, again, it's, it is just complexity, but, you know, I liked it.

Um, was I, I built into the dump script.

I would work with the, the DBA to build into the dump script, a file that they

would touch that would say that the backup worked or didn't work or whatever.

And then I would look at that file from the other side and I would know

that the backup ran recently or not.

And if it didn't run recently, then I would error and then I could go to

the DBA all that stuff is all custom

Well, well, and what percentage of DBAs

and backup people do you think actually do that mechanism, Curtis?

You know?

Yeah.

Right.

Most people, and I, and I've actually seen this, right?

So when I've talked to customers, right?

Sometimes they're like, yeah, our DBA was doing dumps.

We were doing sweeps, and then something happened.

It wasn't coordinated.

And we lost some data.

Yeah.

Yeah.

Uh, so that's why I don't like it, even though it's the most popular method.

So the, the next, um, is the second, I think the second most popular method,

which is the stream to backup product.

Right?

So this is, I'm gonna get the, the agent, uh, For the thing, right?

I'm gonna get an in, you know, in the case of Oracle, it's Rman, right?

Our Rman by the way, RMAN can do a dump, right?

Uh, RMAN can do a dump to disc.

Rman can also basically send data to a pipe that is then read by the

backup product, and it can then be sent directly to wherever that

backup product stores its backups.

Whether it's a DDU device, regular disc tape.

The cloud, right?

Whatever it uh, does, but hardly anybody.

Does this right?

What?

I hope you're being sarcastic.

saying No, no, no, no.

I'm just saying I'm just, it's the second most popular, but

it's, but it's a distant second.

It's, it goes back to that.

To that control, right?

The, the, the, the DBAs just want to control it.

Now, in the case of RMAN, you can properly configure RMAN so that the DBA can run

a backup whenever they want, and you can run the backup whenever you want.

And then you don't have to talk to each other.

But I don't, I don't mean that the, the DBAs the SAs shouldn't talk to

each other, but you know what I'm

Yeah.

It's, it's, it's a control.

Everyone still has control and visibility, right?

I think that's the key is people want the control.

They don't want to give it up because it's their job on the line.

Like if the database fails, DBA is like, do I trust the backup team or am I

going to take control and do it myself?

And this is why even with these tools, Right, that the backup team

and the DBAs can work together.

I would still see DBAs on expensive primary tier one primary storage,

take five, six database copies.

Yeah, and keep it locally on production storage because they

did not trust a backup team.

Yeah.

Good times.

Yeah.

Yeah.

The, the biggest advantage to this is, well, the two, one is that you don't

have to have that extra storage, but a lot of people end up having it anyway.

Uh, but the biggest advantage here is no custom scripting, right.

Put the agent where it's supposed to be, uh, connect it.

It's usually like a DLL or something, and then it talks to the, the backup

system, and then you get all the reporting, everything end to end.

That's why I like it.

I, I think it's a better design, but I know why the

other guys do The other thing.

Um, and then just real quick, I'm just gonna cover 'em real quick

cuz we're, we're gonna be too long.

Uh, transaction log backup, making sure that you know, if you have a transaction

log database, making sure that that's included in the backup, because

especially if you're doing a hot backup.

You need those transaction logs during the time of the backup where you're

not gonna be able to, to get it.

And also you need those transaction logs to roll your backup forward

to the current point in time.

Right.

And then the master file, right?

The control file and Oracle, the, the master database and sql, uh, et cetera.

Yeah.

There are all these other things that you need as well.

Yep.

Yes, yes, yes.

Um, so the next we have is backing up PaaS and serverless databases.

Um, this is.

Um, I, I think that things times have changed, even as, as I, you

know, since I wrote the book.

There's dump and sweep there too, right?

Uh, there's also integrated backup as a service, right?

Um, so like, w when you think about like RDS or DynamoDB, how,

how do people back those up today?

I think they just predominantly use AWS's native tools.

Right?

If I look at that, it's a cloud provider, whoever I'm using,

right, gives me the tools I need to do the functionality to backup,

yeah, yeah.

I, I don't even think it's, many of them don't even offer the dump feature.

Right.

I don't think you can dump, uh, DynamoDB

think Dynamo DB now supports dumping to s3, I believe.

Oh, okay.

All right.

Maybe, um, yeah, so quite often that feature's not available.

Uh, sometimes it is, it sounds like maybe, uh, Amazon was sort of

probably forced to come out with that.

They, they had customer demand for it.

Um, I think, I agree with you.

I think the most common method is the, basically the, the integrated.

Backup method, right?

In the case of aws, you take snapshots, those snapshots

end up getting stored in s3.

Um, you can ha you can have those snapshots replicated to other

places, uh, which you should, right?

That's, that's the only concern that I have.

If you have RDS and you're just like, RDS by default has snapshots enabled, and they

just run, like, I think they run once a

They run once a day,

but.

all in the same, go ahead.

but the thing you should be careful of and read

carefully what they offer because each one does something different.

And even RDS gives you different flavors depending on what database

type you're using within RDS.

Um, so most of 'em, when they're doing that automated snapshot, like

you're talking about, Curtis, They typically only keep it for 30 days max.

That's it.

You can't extend it beyond 30 days.

That's all you get.

So if you have retention requirements, you now have to manually manage the snapshots

on your own or the backups on your own.

Um, the other

Yeah.

oh, go ahead.

go ahead.

The other point I was gonna make is that be careful

also because some tools allow you to do a backup, but they don't

actually allow you to do the restore.

Yeah.

OuRMAN on RDS perfect example of

Yeah, so make sure

The last time I checked anyway.

So carefully read what it offers.

Don't assume.

I am gonna throw an idea out there, Prasanna,

and tell me what you think.

Why in the hell do you need backups Over 30 days of a database?

Of a database?

What good is a backup of your database from two months ago?

This not the file system,

CA.

right?

I can.

I have a, so here's where I think it could be useful.

Now, I've never administered backup systems.

I've never actually had to deal with recoveries like you have.

Mm-hmm.

But one potential I could see is, say there was a corruption,

you have all the transaction logs.

Going back, say 60 days, you could theoretically, Recover the

database back to 60 days and replay all the transactions to recover.

I think that is the edge condition of all edge conditions.

I said, but, but did you notice that I did

not say anything about archive?

Because that's typically why people should be

keeping it

Well, they, they're like, well, they, they're like, well,

we keep it for, um, for, um, um, For retention or compliance or whatever.

And I'm like, yeah, but you're keeping a backup for compliance reasons.

If you actually need that in three years, you're gonna be in a world of hurt.

Cuz you're, we're gonna be on Oracle version 78 and you're gonna be on version

13 and you're gonna be in a world of hurt if you need stuff for compliance reasons

that should be archived, which would be in like full text, not in a database format.

Uh, you know, that's a, that's a whole other discussion.

So, um, the, the final section and, and it, and it's gonna be admittedly

much shorter than the previous, right?

And, and that is, and it's, it's equally as important, which

is recovering the databases.

But there's, um, there are, um, h Hang on, sorry, I'm just looking here.

Um,

um, I, I do in the book, go into.

Preferred, at least my preferred backup methods for, for a

handful of different databases.

If, if, if you want those, you know, those are available in the book.

Uh, but as far as recovering traditional databases, There's

essentially four steps, right?

I'm just gonna go through 'em real quick.

Right?

Identify what's wrong, restore the data files, apply media recovery,

and then start the database, right?

So the, this is for regular, you know, databases hosted on a, on a server.

Um, you, you have to figure out what's wrong because you might

not have to do a full recovery.

Right?

And hopefully you can avoid that.

Yeah.

Like in the case of, again, I'm gonna use Oracles a lot cuz

it's where most of my experience is.

You might just find out that the control file is missing, right?

You might just find out that one data file is, was deleted or corrupted and

you could restore just that data file.

Um, anything you can do like that, um, you know, you know to figure out what's wrong.

Before you do any recovery, it, it's just like, you know, we talked about ransomware

recovery, just like ransomware recovery, the more you can do upfront to figure

out the, the more time it'll save you.

Yeah.

Yeah.

Every, you don't need a hammer for everything.

Full recovery.

You don't need to full restore everything.

Anytime something happens with your database.

A lot of these database backups, right, and the database applications

themselves are smart enough to recover with just smaller granularities.

So once you figured out what caused the issue, right, most of these

database, then you can actually restore whatever needs to be recovered, right?

So like you mentioned, Curtis, your second step was recover the data

file or whatever that object is.

Most of the database recovery tools sort of deal with a lot of that for you.

So say you've.

Found Oracle has a corrupt data file, right?

You can go to Oracle RMAN and say, recover this particular data file, and

it'll just pull back that one data file.

Right.

And if you used, if you used dump and sweep, it's a little bit different.

But if you used, uh, snap and Sweep or you know, one of the others where

basically you have the, the data files right there, you can just grab

'em and restore 'em, or you can, you know, use your database product.

The, the really important step, the database won't open at that point because

now you've got, You've either got a, all the files are from one point in time from

last night or maybe even two days ago, or you've got some files from right now

and some data files which are behind, and then you have to apply media recovery.

So this is essentially using those transaction logs to redo

logs to, to redo everything that happened since that backup.

And you, you basically bring all the data files up to the same point in time.

And then you can open the database.

And I,

go ahead.

and I remember just going back talking about it, right?

Only once I started talking to like the application experts who are really good at

backups for databases, and I realized it's a two step process, like you said, right?

One is restore and the other is recovery.

You're restoring your data files or whatever the media is, you're

recovering the database into a well-known point in time so it can, in

the footstep, be brought back online.

And it is possible.

For example, if the reason that you're restoring the database is that somebody

dropped a table, uh, or did a DBA or, or a threat actor did something

to your database, deleted a bunch of records, deleted some tables, whatever.

When you do media recovery, you can typically say, do media recovery.

Up to this point in time, you know that the date that the table

was dropped at, you know, 1745.

And so you, you just restore, you know, recover up to 1744.

Um, you might even do 17 44, 59, what, you know, whatever it is that, you

know, when that table was dropped.

You can do media recovery up to that point and it'll bring it up to a consistent

point in time at that point in time.

Right.

Um, Yeah, it, it, it's not an easy thing and, and uh, you know, hopefully

it's something that you practice on a regular basis, just like everything else.

Hopefully it's something you test and you practice, um, et cetera.

So I've got so recovering modern databases, um, I.

You know, I, I've got a handful of different ways, right?

So I've got the point in time recovery without media recovery.

So, um, some of them just don't provide it, right?

So, like when we talked about backing up, um, What are we talking

about backing up DynamoDB, right?

So it doesn't really have the concept of media recovery.

If you want better, if you want more granularity in the restore, you just take

More uh, snapshots more often, right?

That gives you a better rpo, right?

Recovery point objective.

Uh, so that, that could be one option.

The other option would be table based recovery.

Um, why might I want to do that?

because you only care about one particular thing in the

tape or in the entire database instance.

Not everything.

Well, I hint, I mentioned something

that could happen a few minutes

Oh, someone dropped a table

or uh, or you have a malicious person who goes and deletes things.

Yes.

Yes, exactly right.

So it may be that the only thing wrong with your, with your, um,

system is, uh, is ta is is the table.

But the other reason that you might want to do this is that

this is the only way to do this.

Some, some of the backup agents or some of, I'm sorry, some of the

database products, the only way to back up is at the table level, right?

Um, there's also node level recovery.

Right.

you're talking

a multi node database, go ahead.

I was thinking like when you think about like Cassandra

or MongoDB or other things like this where you have multiple nodes, which.

Then make up the database.

Right.

You might have a node fail.

Right.

And so what's the process for bringing up and recovering a particular node?

Right, and and most of them, it looks like you don't really

have to recover the note as much as you just have to create a new replica.

Say, Hey, uh, this one's dead.

We're gonna do this one now.

And they just sort of figure that out.

That is the beauty of those type, you know, of a scale out database, right?

Um, the, the, the more common I think is gonna be the, you know, the one

that we, it, it, I'm, I'm saying it twice, but the, I have cloud level

snapshot as a backup and restore method.

That's a super easy restore method though, right?

Yeah, can you just bring it back?

Yeah.

assuming that it's supporting, assuming it's supporting,

Yeah.

Assuming, supported.

Yeah.

Um, don't assume any of this stuff right.

To make

go test it.

you're testing this upfront.

Yeah, exactly right.

Um, the.

So, uh, I'm just gonna read this because it's, it's outside, really

outside of my world here, but, so there's some restored, uh, scenarios.

Depending on how the data is restored will need to be brought consistent

with the rest of the database.

So you may have to restore.

This is, again, this is with an eventually consistent database.

That's why we wanted to mention that term back in the last, uh, recording.

Um, if you've got an eventually consistent database, You may restore

only part of that database and that it then needs to be brought consistent

with the rest of the database.

It's a bit, honestly, it's a bit, uh, outside of my realm of experience.

So I, um, so I just have, I just have to say that is a type of restore and

when you talk about like Cassandra and MongoDB and these, these giant, uh, you

know, massive databases, That I, I talk to, um, like the folks over at datos.io,

that, that they, you know, they back up these type of things and they're

the ones that talk, that, talk to me about this, where that some of the stuff

that's documented in the, in the manual that I, if you follow it, it could end

up having to do an eventual, like an eventual recovery and that it can take

a really, really long time to do that.

So that, that, that's sort of what I'm alluding to there.

Honestly, I barely understand what, what that is myself if you don't understand it.

But here's what I'm saying is if you've got a, if you've got one of these massive

scale out databases, the, the big thing, the big takeaway for me from those is

please do not confuse availability.

Yep.

durability, right?

Um, that, uh, someone can still drop a table, you can still lose a number

of a, a number of nodes, right?

So the question is, how do we recover?

If somebody lost, if somebody dropped a table, how do we recover?

If we lost whatever the number of nodes is, that would take out the system.

What's that solution?

Because if you don't have a solution for that, Uh, then you're gonna be in

a world of herd if that ever happens.

Right?

Some final thoughts on backing up databases.

Just a few things that come to my mind.

You, you've mentioned one of 'em, I, if you are scripting things, make sure you

built some intelligence into that script so that it will identify new databases,

new table spaces, new data files, whatever it is that you're don't, please don't

have a configuration file that you have to update when you make a new database

Well, because things are changing and no one's

gonna reach out to you either.

Yeah, exactly what I used.

Uh, shoot, it's been a while.

Was it the, the, is it like.com?

Like it was the, it was the configuration file for Oracle itself.

Right?

So if you had a new database in Oracle, you would had, you

would add a new instance, right.

And that instance would have an instance Id, that instance Id would

go into the Oracle configuration file.

It's basically the Oracle version of the, of the file system table, right?

And so I would start there.

I would read the Oracle configuration file, and I would say, what, you

know, what instances are on this?

And then I would put all those instances in backup mode back in the day.

I had to put the table spaces in backup mode.

So I would, log into that instance, query the instance of what the table

spaces were, you know, then loop through and, and, and do all of

tho put all those in backup mode.

And then of course, very importantly, take them out of backup mode when you're done.

Um, and, uh, and then add some reporting, add lots of error

checking for, for things.

You know, like, you know, the, the or.com file isn't where

I thought it was gonna be.

Right.

Um, or the, the sequel, I remember the, they would change.

Yeah.

Stuff would change.

So just have error conditions built into when the, when the script doesn't do

what you think you're gonna do, have some kind of way of letting other people know.

Back in the day, the only way I had was, was email.

Um, And we would just, you know, uh, send out a note, say, Hey dude, you

know, something happened over here.

Come take a look at it.

What, what other sort of

general things to can you think of?

I think the other thing to take away is, Whatever

solution you're using for backup, right?

Even if you do have these automated abilities in the script, make

sure the backup and the database people talk to each other.

You guys don't have to be enemies.

You don't have to be best buds either, right?

But at least talk to each other and make sure you're coordinated.

Because when you start to talk about recovery and the process,

right, it all starts from what did we decide upfront for backup?

Um, man, we covered a lot in this episode, dude.

What do you think?

Yeah.

No, I think

this is a, I think

Yeah.

I always learned something new thinking, and I think

I talked about this on the last database chapter or episode as well.

It's databases are complex.

If some of these things go over your head, don't worry.

Right?

There's a lot of

Hey, it goes over my head sometimes.

Right, right.

Because I'm trying to talk about, you know, we talked about if you go

to db-engines.com, we talked about that there are, um, it was like

30 different database products.

There's

I think

it's more than

different types.

Yeah.

30

well, no, it was like 30 types and then like a couple

of hundred different databases out there, and they're all different.

And they all.

You know, they all back up different and restore different, so just

make sure you know how yours works.

Right.

Um, and please don't let the vendor go, oh, this doesn't need to be backed up.

Please don't.

It hurts my ears when I hear that.

Uh, so hopefully this was helpful.

Um, you know, if you've got other database questions, throw 'em out.

Throw it out to us.

I hope you enjoyed the episode.

Um, it's always fun, um, talking about my favorite subject

Yeah, and no, I, I'm excited to hear about your

experiment, Curtis, more than anything, honestly, I wanna see what happens.

Yeah, it'll be interesting.

It'll be interesting and, uh, thanks everybody for listening and remember to