This week on the backup wrap-up we dive into the disastrous data loss
event that hit dedoose a cloud-based research data management platform.
Back in 2014.
Dedoose lost access to their primary data and their backups simultaneously
resulting in the loss of over three weeks worth of customer research.
We discussed what caused the crash as well as some questionable backup practices.
These practices resulted in a ton of data loss for doctoral researchers who
lost countless hours of painstaking work, they will never get back.
We discuss what dedoose did right, and what they could have done
better so that we can all learn.
If you're new to the pod, I'm w Curtis Preston, AKA Mr.
Backup.
And there's a reason I'm so passionate about this topic.
When I first started in backups over 30 years ago, my company lost an important
database that I couldn't restore.
Since then I've dedicated my career to making sure that would never again happen
to me or anyone who will listen to me.
We turn unappreciated backup admins into cyber recovery heroes.
This is the backup wrap-up.
Welcome to the show.
W. Curtis Preston: Welcome to the show.
I'm your host, w Curtis Preston, AKA, Mr.
Backup, and I have with me my tax non-ad advisor, Prasanna Malaiyandi.
How's it going?
I, I've needed you a lot over the last few days is
Prasanna Malaiyandi: It is that time of the year, Curtis, you know,
I wouldn't expect anything else.
For, for our international listeners who don't know , april
15th IS a US' day that you have to file your yearly tax income tax
with the government, with the IRS
W. Curtis Preston: You can file an extension, but any money that you
owe has to be paid by that time.
This was, I'm gonna say one of the, if not the most complicated years that
I've ever done tax, because halfway through the year I was laid off.
And so then I went, you know, self-employment and I went like nine
different ways of self-employment.
And um, you know, and I had, I had some carryovers from previous years and I had
some, I had some of this, some of that, and I'm like, I think I'm gonna be okay.
I might not be okay.
Turned out I was way okay.
I was very okay.
But you may recall that there was a day there or so where I was not having any.
Prasanna Malaiyandi: I recall that.
And, and for people who may not understand the complexities of the US
tax system, they're probably thinking, oh, don't you just like take your
income and they just take a check for based on a certain percentage.
Yeah.
That's not how the US system works.
W. Curtis Preston: Yeah.
And well, and then when you add in the complexities of, um, some of the
different types of self-employment that I did, there's some very specific
deductions that you have to take and then you have to really record stuff
to, to, to make those deductions.
And, and, uh, there was, um.
It was just,
Prasanna Malaiyandi: I remember the picture you sent of just like, Hey, here
are the things I have to be tracking in order to be able to do this deduction.
I was like, oh my gosh, Curtis, there must be a better way to
W. Curtis Preston: Yeah.
I, and, and I, I will say I did figure out a better way for that one.
I've only done that, that particular one for like seven years.
And, uh, I never had figured out a better way.
And actually this year, for the first time, I figured out a better way.
Prasanna Malaiyandi: So 2024 will be better for
W. Curtis Preston: 2024.
And also, uh, I, you know, uh, you helped me, uh, you know,
come up with a spreadsheet.
I mean, I did it on my own, but, you know, there was some,
definitely some advice in there.
And man, you're right, the US tax code is super complicated.
Um, you know, because, you know, we have a progressive tax system, so like
they tax this percentage on that amount and this percentage on that amount and,
and, uh, so that, so it's not just.
Figure out how much you made and take X percentage of it.
It is so not that.
And of course, I live in California and we have a state income tax,
so we have that in addition.
And just in my tax bracket from the lowest, you know, there
are, what, what did we find it?
Six, six different layers of that.
It starts at 1%, then 2%, 3%, 4%, you know, all the way up to like 9%.
I think, um, you know, once you go.
Prasanna Malaiyandi: 12.3.
Yeah.
Is
W. Curtis Preston: Oh, well, no.
Well, I, well, I know it goes past that, but like, but like, you know,
if you make over a certain amount, you end up paying 9% on that final, uh,
on the, on the marginal, that's what's called the marginal tax rate, uh,
Prasanna Malaiyandi: So it's like every dollar you make
additional gets taxed at that rate.
And yeah.
And then the other thing challenge is you have kids.
They have different deductions, credits and deductions change
every year, depending on what gets passed, what doesn't get passed.
Your brackets don't stay fixed because some are adjusted based on the rate
of inflation, but some are based on one inflation rate, others is
based on a different inflation rate.
So it's a giant mess.
You basically need a PhD in order to be able to do this stuff and keep up with
W. Curtis Preston: Yeah.
Yeah, but through, but through it all, I have contacted my
financial non-ad advisor.
Prasanna Malaiyandi
Prasanna Malaiyandi: let's just say you are not the only person
who talks to me about, uh, taxes.
W. Curtis Preston: Is that right?
I, that, that does not surprise me.
'cause you're a, you're a smart, you're a smart little person.
Prasanna Malaiyandi: It's just random bits of knowledge that I have in my head.
W. Curtis Preston: yeah.
I love, I love how you can just, you can just quote the marginal tax rates for
different, for different levels of income.
Um, yeah.
Yeah.
And you're like, oh, yeah, with the self-employment tax, you know,
you could deduct half of it, right?
I'm like, no, I didn't know that.
Yeah.
And of course, I, I bet a significant portion of our listeners go, yeah,
no, that's why I have a guy, right?
I just give the money to the guy and the guy just tells me, um, you know, uh, yeah.
Um.
Prasanna Malaiyandi: useful to learn, right?
Because then you could figure out the impact of certain things that
you're gonna be doing or like how you should be withholding.
And honestly, like it's great to have a guy, but the guy
charges money and it's expensive
W. Curtis Preston: charge money.
Yeah.
I, I think one meeting with the guy, and by the way, it
could also be a girl, right?
Just, you know, one meeting with the guy and you, uh, you've spent
like the entire thing of just what I spent with TurboTax for the year.
And that's just one meeting, right?
Um, yeah.
So.
My guy's free, except he is not allowed to be my guy.
He's, he is my non-ad advisor, legally is not giving me financial advice.
Um, yeah, our financial advice is get a guy.
Um, anyway.
Prasanna Malaiyandi: Well, at least it's done.
W. Curtis Preston: At least.
Yeah, it is done both last year and this year.
I have filed my estimated taxes for Q1, like a week early.
I
Prasanna Malaiyandi: Look at that, Curtis.
And I'm hoping that a lot of people who have complicated
taxes don't wait till the end.
W. Curtis Preston: Yeah.
Yeah.
I'm just so super happy to be done with my taxes, both for the last
year and for this, this quarter.
I'm so super happy, super excited.
Uh, but we're gonna talk about something not so exciting.
We have yet another episode of, you know, cloud disasters and this one.
This
Prasanna Malaiyandi: I don't know about you.
W. Curtis Preston: what, go ahead.
Prasanna Malaiyandi: I don't know about you, but I've kind of enjoyed this series.
I know it's sort of taking joy in other people's misery, but
I've found it to be kind of, I.
Like interesting, right?
To kind of look and figure out, hey, what did people do?
What happened?
Because these are things that could happen if you do something wrong or
misconfigure something, or maybe it's not even your fault and you just ended up
getting attacked by a malicious user and
W. Curtis Preston: Yeah, I, I.
I will say that this story, there's one particular part in the middle of it that
is the crux of what happened that is so bad backup design that we're gonna,
we're gonna get to right at the core.
At the core, they had some other mistakes that, you know, the, the same mistakes
that everybody else makes and, um, and, and sadly it appears that this was
one where they stepped on their own.
Toe or whatever, right?
Um, but this was not an attack.
This was just, uh, you know, some of the headlines, you know, use phrases
like fat fingering and stuff like that.
Uh, we are of course talking about a company who I'd never even heard of prior
to this story, and it's called Dedoose.
At least I think that's how it's pronounced because they are a, um,
uh, you know, what do they describe?
They're, they're a cloud-based, qualitative and mixed methods research,
data management and analysis application.
That's a mouthful.
I.
And they were in Manhattan Beach, California, which for the record is
right up the road from me and right down the road from our friend, uh,
Jeff Rochlin, uh, that's literally like right next door to him.
And they, they were designed to help researchers organize, analyze,
and collaborate on qualitative and mixed methods research data.
And one particular group of people that used this platform a lot were people
doing their doctoral dissertation.
I.
And so this is gonna
Prasanna Malaiyandi: a bunch of research.
W. Curtis Preston: bunch of research and that, that you're just getting
ready to file to, to, to, you know, to what, what do you call, do you,
what do you do with your dissertation?
You file it, you.
You defend.
Yeah.
Um, present and defend.
Right.
Um, and you're just at that moment where you're about to do that.
And, um, the um, uh, and, and this, then this bad thing happened.
So they were, they had some competitors, uh, I'll just finish this part.
They had some competitors, Atlas Ti NVivo.
And Max does, they all have these, these interesting names, but they were
apparently the only SaaS based, um, you know, web-based, uh, portal, which made
them the only game in town for this, for doing it the way that they were doing it.
Prasanna Malaiyandi: Yeah.
And my guess is people, like if you're an academic, you're probably
like, yeah, I'll just throw it into spreadsheets or throw it into Microsoft
Access and just run it locally.
Um, but that doesn't really scale.
And then you have to manage it, and then you have to worry about how
do I protect it and back it up, and what happens if my laptop crashes?
How do I sync it over to OneDrive or wherever else it has to go?
And yeah, so I could see the appeal of this company.
W. Curtis Preston: Yeah, absolutely.
Prasanna Malaiyandi: You mentioned it was a cloud provider.
They are hosted on top of Azure,
W. Curtis Preston: Yes, they are hosted on Azure.
And this was one of the reasons that we wanted to cover this because we were
trying to get a story from each of the major cloud providers, and this was the
biggest story that we could find on Azure.
But I don't know if the fact that they were on Azure was
really part of the story.
It almost was as we're gonna cover it in the story.
But, um, but in the end, I think this was all them.
Prasanna Malaiyandi: Actually, I take it back.
It's Azure, not Azure.
W. Curtis Preston: Oh, it depends on who you ask.
Um, we could go on, we could go on Microsoft copilot and
ask them how to pronounce it.
Do you want to describe what, what actually happened?
Prasanna Malaiyandi: Yeah, so everything was going fine, right?
They had the service up and running, and then sometime in 2014
they encountered an event, right?
And it was basically a failure of a service.
Running on top of Azure, which initially they said was a Microsoft service that
had an issue, but turns out it was just their service that they had been
operating that just leverages Microsoft component infrastructure to run on.
W. Curtis Preston: right.
Prasanna Malaiyandi: And so, uh, what ended up happening is that failed and they
basically lost access to their databases, which hosted the data for the end users.
They also happened to lose access to their data itself and
the backups at the same time.
And so a lot of bad things happened all at once.
W. Curtis Preston: Yeah.
I mean, this is why we back up, right?
Right.
Because bad things happen.
Um, what I, what I don't, just don't fundamentally understand is
they go, they describe it as we were in the middle of our database
encryption and backup process.
What does that mean to you?
Prasanna Malaiyandi: I read that as they're encrypting the database
before they write out a backup.
W. Curtis Preston: Right.
But
Prasanna Malaiyandi: And so my,
W. Curtis Preston: place
Prasanna Malaiyandi: no, I
W. Curtis Preston: as an event.
Prasanna Malaiyandi: my guess is, well, it's almost like streaming, right?
So I think as they're reading data out, they're encrypting it and writing
W. Curtis Preston: Okay.
Okay.
It's just that it's just that the event damaged the database.
That's why, that's what I don't understand is why would a backup
event damage the database?
Prasanna Malaiyandi: Well, it's like the old days of Oracle, right?
And I know you've been in this situation so many times, right, Curtis, where
it's like, Hey, you did a backup, you took down my production database, right?
They might have been an issue.
And I know that in all your scenarios, Curtis, that's
always been not the case, right?
W. Curtis Preston: Right.
Prasanna Malaiyandi: That backup doesn't take down production, right?
Maybe they just didn't code for it properly, or maybe there
was just some race condition,
W. Curtis Preston: Yeah.
Prasanna Malaiyandi: some reason while they were doing a backup.
Our production database also had a failure.
W. Curtis Preston: Right.
And I think this was good that they were encrypting the backup, but
based on what I'm hearing, what I'm inferring from this is that this was
a homegrown backup system, right?
Because they, they, they seem to.
Make a big deal about the fact that they're encrypting and, and
then, and then, you know, and then backing it up, it's like, okay,
well that's sort of standard.
You know, that's SOP for most people, right?
But, um, so they had a failure of the primary database as they were backing
it up, they were backing up and this, this was the crucial part that I
was talking about earlier, is that.
It's apparently they didn't have enough space or whatever that they
ba they're overriding the last good backup of the database as they're
backing it up and slash or they're only backing up like once a month because,
Prasanna Malaiyandi: a bunch of incrementals
W. Curtis Preston: well, but if they were, but if they were doing incrementals,
they would've been able to restore to a better point in time because.
Prasanna Malaiyandi: So maybe we should talk about that point in time that they
actually were able to restore four.
W. Curtis Preston: Yeah.
So after, after, yeah.
I guess we, we kind of jumped ahead, so we're doing the research afterwards.
Yeah.
So, um, yeah.
So let's back up to, you talked about the event.
So after that event, they, they went out and they did what we tell people to do.
They put out a blog post and, uh, they're like, here's the situation.
Uh, we believe we can go back to, this was May of 2014 . We know for a fact
we can, that the March backup is good.
We think we can also restore the April backup.
We unfortunately believe that we are going to be unable to restore.
The, the, the May backup.
And so every, all of the changes and anything that you put in the system for
the last two to three weeks will be gone.
So I am, I am inferring from that, that they only backed up once a
month because otherwise they would've been able to get a lot closer
Prasanna Malaiyandi: and it wasn't just the database, like I was reading Reddit
posts where they're like, Hey, we talked to a service person or a support person.
They basically said if you created a new account, even in that May
timeframe, you're basically out of luck.
Right.
So it
W. Curtis Preston: Like the entire application, right?
So it's possible that they weren't overriding the last good backup.
It's just that the last good backup was from over a month ago.
Prasanna Malaiyandi: Or it could be that like their more
recent backups were corrupt.
W. Curtis Preston: Uh, well, we don't have any information to suggest that,
Prasanna Malaiyandi: don't have any information.
Yeah.
But
W. Curtis Preston: I I also don't understand why if it, if all backups
were equal, why didn't they immediately try restoring the one from April?
They, they, they said, we know for a fact we can get the one from March.
We think we're gonna get to one.
And they go and, and, and they made a comment about, you know, we realize
that this is, it reminds me a little bit of the Rackspace event where.
What we can do, the quickest is we can get the March one up and running.
Why wasn't the April one also just as good as the March one?
Right.
So I'm not, I'm not sure what happened there, but there was something that
was not as good about the April backup.
Uh, but, but it did appear that they eventually were able to get
that up, but they lost two to three weeks worth of, of effort.
Prasanna Malaiyandi: Yeah.
Yeah.
And even for the April backup, they didn't actually restore it.
So using the way back machine right, we were able to go back and
find the original blog post for
W. Curtis Preston: Right.
Why did we have do that Prasanna?
Why did we have to use the, the, the internet archive?
Prasanna Malaiyandi: because the company took down that post and created a
new post at that same, for that same date, that shed different light on
what had happened and their steps.
W. Curtis Preston: Interesting, right?
Um,
Prasanna Malaiyandi: so, so from that or may, uh, blog post, they basically
said, or soon thereafter, right, I think it was a couple weeks later, they
were like, Hey, we're able to get you back up and running for your March and
for April, we were able to restore the data, but they didn't wanna merge the
data back into the original database.
So they actually created a staging site where people had to go to manually
in order to be able to pull down their data if they want to access it.
W. Curtis Preston: this is so much like the Rackspace event, right?
And so what that, what that tells me, you know, and again, we're, we're, we're not
taking joy in this, but what that tells me is that they didn't have a regular
tested procedure because they didn't, they didn't know that the April one was good.
If they knew the April one was good, they would've just restored the April
one, because by restoring the March one.
They may, you know, you couldn't just do an incremental restore
on top of the March database.
So, um, they created this extra work for
themselves and their
Prasanna Malaiyandi: and especially their entire thing was we want to get people
up and running as quickly as possible.
Right.
That's why they did the March one.
But like you said, Curtis, if April was good, they should have just used April and
it would've been the same amount of time.
W. Curtis Preston: There had to be something up with the April one.
You know, you could, you'd sort of have to read between the lines.
Um, there was something up with the April one and then,
we've talked about this before.
We urge people not to do homegrown backup and when I said this last time,
you said sometimes you have no choice.
Prasanna Malaiyandi: Yeah,
W. Curtis Preston: I would like to, I would like to.
Prasanna Malaiyandi: say it again.
W. Curtis Preston: Well, I'm, say it again, but I would like to point out the
fact that they were running on Azure, and Azure has backup features built into it.
Right?
They could have taken a snapshot of their entire environment every day, every hour.
They could have done that.
They would've had essentially, you know, you know, a copy of
their database from two hours ago.
You know, they could have, and that wouldn't have been homegrown.
I'm just saying they could have done that.
Prasanna Malaiyandi: Well.
The other thing they could have done is why don't they
have a disaster recovery copy?
Like if this is a SaaS service, wouldn't you have expected a Dr.
Copy to be available somewhere where even if the production instance went down for
some reason, they should have been able to come back up even if it wasn't a backup.
W. Curtis Preston: Yeah, so this, this is where, you know, we often talk about
that, that if your data is sitting on a SaaS service, I think it's your
responsibility to back it up, or at least to make sure that it's backed up.
You can, you can make an argument as to whether or not you should use
their backup service if they have one.
I think we've got three stories now that show that sometimes their backup
service is as good as their it, um, and.
The, so you, I think you should be backing up your own data, but
generally speaking, some of these SaaS services, they will have a dr.
Copy that isn't any good for you if you do something stupid, but, but if
they do something stupid or there's a fire, or there's a, an attack,
they've got a copy that they can restore the entire environment that
that appears to be what they had here, but it was just woefully out of date.
Um, and so the vast majority of their customers did get their data back, but
there were a few, and I wanna, I want, I want to just point out a few that
they were, uh, quoted in the story.
There was, uh, Margaret Fry, a postdoctoral researcher at Harvard
University, lost about 60 annotated texts and over a hundred hours worth of coding
work related to her research on AIDS and sexual attraction Southern Africa.
Jason Richardson, an associate professor at the University of Kentucky and his
colleagues lost around a hundred hours of work on a project, analyzing job
advertisements for school principles to assess how well Kentucky prepares
aspiring school administrators.
Uh, there's one comment on Dedoose Facebook page that the company had managed
to destroy their wife's $10,000 research project, so there were real people.
And these weren't just companies, these were just, these were just regular
people trying to get their doctorates done and uh, they lost, you know,
Prasanna Malaiyandi: yeah, but, but to their credit though, dedoose actually
had no, I don't know if it was before this issue or after the event, but
they did say they have functionality.
Actually, no, it was even before because for staging, this is
how you get your data out.
There is an export functionality.
W. Curtis Preston: Right.
Prasanna Malaiyandi: That a user could use to pull all their data
out of Dedoose for doing, like what you talked about, a local backup.
So it is possible that these users could have been using that to protect
themselves from these situations, and Dedoose is still around, and so I highly
recommend people who are using Dedoose.
Maybe you should periodically export your own data out.
W. Curtis Preston: Can I, can I edit you and take the word maybe out there?
Prasanna Malaiyandi: Yes.
W. Curtis Preston: Yeah, so you should periodically back up your work no
matter where it's residing, you know, a, a periodic, you know, inefficient
backup that has a poor recovery point.
Actual is still better than, than nothing.
Right.
Um, yeah.
I remember, I, I have a, a niece that came to me with her, she had a laptop
and she had her masters on this laptop.
And the laptop, the drive was making those noises.
You know
Prasanna Malaiyandi: Oh, the click, click, click, click,
W. Curtis Preston: Yeah.
The hard drive should not make.
We were able to get off her master's thesis, uh, off of that
drive before it just went up in smoke and it was killing me.
'cause I was like, you don't have a backup of her.
That's really important work.
Uh, you know, I feel for these people At the same time, you know, uh, my,
my statement is gonna remain the same.
That if your data is sitting in, in a SaaS service somewhere, it's
your responsibility to, to make sure that it's being properly backed up.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: and and I still say even if the SaaS vendor offers
backup, which this one apparently did, um, I still think that you
should do it some other way.
Um, yeah.
Prasanna Malaiyandi: Well, do we know if they offered backup or
was this just sort of their own
W. Curtis Preston: right.
It this was their, this was their internal dr.
Right.
Um, what I don't know is whether or not they advertise this as part of the, um.
You know, now the CEO of Dedoose said this is a horrible moment for our company.
We have never lost data or had a breach.
It's impossible to say how many customers have lost data because
of the firm doesn't monitor how customers are using their accounts.
Only users can access, can assess losses in their projects, not Dedoose.
Yeah, it was a, it was a
Prasanna Malaiyandi: Yeah, if I
W. Curtis Preston: a.
Prasanna Malaiyandi: Oh, oh yeah.
Like, and especially researchers, like coming back and getting
that data is so difficult.
And then just being like, yep, it's gone.
Just like that puff of
W. Curtis Preston: And, and some of that data is irreplaceable.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: It's interviews of subjects.
Um, you know, I don't even want to try to, when you're in the middle of
something that complicated, there is some data that is simply irreplaceable.
Um, and, um, so it, you know, it's, it's a real shame what happened.
Prasanna Malaiyandi: Did you wanna talk about what they said they're going
to do and what they've actually done?
W. Curtis Preston: yeah.
That's a great, yeah, that's a great segue.
So they said they're now going to be doing seven things and when, when I
say now we, we mean, uh, 10 years ago.
Right.
Um, after the event they said they were going to, it sounds like they
went from not much to way overboard.
Right.
Of course, I would never say that any backup system is overboard.
But here's what they said they were gonna do.
They were gonna develop a database mirror, slave, and Azure.
A database mirror slave in Amazon S3, keeping a mirror copy of the entire
blob storage, you know, including all the stuff in an encrypted volume
storing nightly database backups on the V-H-D-V-H-D, virtual hard drive, I guess,
and Azure blob storage and Amazon three.
So they're gonna keep three nightly backups in three separate places mirroring
all Azure file data into an S3 bucket.
Carrying out weekly restore exercise, uh, for the database backups and
a monthly bare bones restore.
That all sounds great.
Now I do, I do wanna comment, um, that anything that refers to mirroring to me.
Isn't a backup, right?
Um, that is a dr.
Copy that will only be useful if what happens is like a fire, right?
If there's any sort of attack, any of those mirrors are totally worthless
because, or if, if you do something stupid like drop a table, all those
mirrors are gonna immediately, uh, you know, get corrupted as
Prasanna Malaiyandi: but it would've helped in their current,
in the event that it happened.
W. Curtis Preston: Uh, I'm gonna say it's unclear as to whether
or not it would help in it,
Prasanna Malaiyandi: Okay.
Sorry.
It could help.
Yeah.
W. Curtis Preston: It could have helped.
Yes, yes.
Um, mirroring is not bad just because they're putting it here and
they're listing all these things.
Um, you know, it's a shame that they, that they had to wait.
Uh, after the thing.
What I also would've liked to seen in that list is I would've liked
to have seen that one or more of these copies, uh, was immutable.
Right?
Um, and, uh, and maybe they are, I don't know, you know?
Prasanna Malaiyandi: It's a great list, but it's also one of those
things like once a horse has left the barn, like what do you do?
You know, it's sort of like, Hey, uh, we're just gonna throw
everything at this and just say, this is everything we're going to do.
W. Curtis Preston: yeah, it
Prasanna Malaiyandi: than what
W. Curtis Preston: yeah.
Prasanna Malaiyandi: what is actually makes sense from a business perspective,
because sure, you could do all these things, but just imagine the cost
you would be spending for backup.
Right?
Your backup copies are probably more than your production
copies at that point, right?
W. Curtis Preston: Yeah.
It depends on, depends on the way they're doing them.
Yeah.
Prasanna Malaiyandi: Yeah.
Right.
W. Curtis Preston: Yeah.
To take, to take your, to take your analogy, you know, the horse is laying
out dead in the field, and you're like, you know what we're gonna do?
We're gonna lock the, we're gonna lock the barn door now, and then we're
gonna put another padlock on top of that lock, and then we're gonna put
a fence around the area in front of the barn, just in case the lock fails.
And then, uh, you know, and then we're gonna have a guy
standing, standing there with a.
With a tran gun, uh, to shoot the horse.
If the horse comes outta the bar, it's like, yeah, that's all great, but your
horse is dead, so you need another horse.
Prasanna Malaiyandi: Yeah,
W. Curtis Preston: we're some, I'm, I'm sorry that was a little
morbid, but I apologize to the horse lovers in the, in the group.
Prasanna Malaiyandi: but the other thing we found though, so that's what they
had promised 10 years ago, but they do have in their, what is this called in
their security article on Dedoose, sort of what they've actually done, right?
So.
What they have done is they are using redundant storage volumes on Azure.
W. Curtis Preston: mm-Hmm.
Prasanna Malaiyandi: All project data backed up in full on, on a nightly basis.
A volume is onsite with two being offsite and replicated across geographic regions.
Um, and then the storage volume is encrypted and mirrored in
real time to Amazon S3 storage.
They've automated everything such that on a weekly basis, it'll download the most
recent backup from each storage volume.
Verify they're using the correct version of the backup file, do a full
test restoration of the database, and email reporting to all backup
and restore process results to key members of the Dedoose team.
W. Curtis Preston: Yeah, it's, it's still, to me, it still reads
like a homegrown system, but, um.
You know, because like one of the things like, you know, I, I did, I did a lot of
homegrown systems over the years, right?
Like one of the things like with email reporting is.
D does your system report if it doesn't run right, is is there,
is there a thing that, that's, you know, like what happened?
You know, 'cause sometimes you don't notice that you're not getting an email.
Right?
Hey, has anybody noticed that?
We don't get the, we stopped getting the backups, you know,
uh, like three weeks ago.
Um, usually when you, when you discover that is right after the, um, the
bad thing happens, whatever, it's
Prasanna Malaiyandi: but even though it might be sort of homegrown from a
automation process perspective, like we don't have enough information.
I think the fact that there are at least.
Doing nightly backups, right?
Verifying it on a weekly basis like these are, and also making sure that
the data is stored offsite, right?
I think maybe it's going a bit for like, they may not have to do like multi-cloud.
Maybe that's going a stretch too far, but I could see that they really are
worried about data integrity, so,
W. Curtis Preston: There's no such thing as a stretch too
far for backup and recovery.
My friend.
Prasanna Malaiyandi: well,
W. Curtis Preston: They're doing the thing that I wished everybody did, but
nobody does because it's too expensive,
Prasanna Malaiyandi: Yes.
W. Curtis Preston: right?
Um, so I, I can't fault 'em for doing that,
Prasanna Malaiyandi: Yeah.
So it looks like they are doing the right things,
W. Curtis Preston: mm-hmm.
Prasanna Malaiyandi: um, from what we can gather.
So it looks like they have learned and they're still around, right?
People still continue to use 'em.
They do have an active subreddit.
They are continuing to release new features.
W. Curtis Preston: Mm-Hmm.
Yeah, and I think, I think what was in their favor is that they
were the only SaaS game in town.
And for a lot of people that, that, that's probably a significant.
Reason to use their service.
Right.
Um, I would also hope that this is a significant reason for anyone to do a,
a regular export of their data, right.
When I'm using a service like that, where I tend to do is I do, I do an export.
Like if I'm doing a manual export, I'm doing it after significant events
like finishing my taxes, right?
I did, you know, I did an export.
I, I did a PD, I didn't do exports of.
My intuit data as I was doing it right.
But once I finished my return, I printed a PDF of that.
Um, I have a paper copy.
I have a PDF copy that's on my laptop.
It's backed up using my backup software.
And uh, I put it in, um, Google Docs, right?
So it's all over the place, right?
Um, but I don't do that every day.
It's just you do it when you, when you get to that moment where you're
like, ah, oh, this is really good.
I hope I don't lose
Prasanna Malaiyandi: a good stopping point.
W. Curtis Preston: It's a good stopping point.
Yeah.
Um, it's like also I use voice recognition software, right.
And sometimes it crashes and you lose whatever you said, uh, in that thing.
And so I just got into the habit of anytime I finish, like
a page, I just hit, you know?
I just say, um, yeah.
Uh, 'cause I, I can actually say.
Um, I almost said, Hey, Siri, you know, click file save.
That's what I say.
Click file, save.
And, um, and I can do that or you can just type control s right?
Uh, just do you know, this is, I guess this is another lesson to learn
is, is that should be a part of your mental process when you do stuff and
it's storing that data in some place.
At the end, at that stopping moment, do you, do you think to yourself, Hey,
is this data backed up in a way that protects me from all stupid things?
Prasanna Malaiyandi: You know
W. Curtis Preston: are you
Prasanna Malaiyandi: funny when you're, when you're talking
about the control save, right?
Or Control S to save it.
So I'm so used to working on Excel spreadsheets that I would always do
like Control S, control S, to save
W. Curtis Preston: Right, right,
Prasanna Malaiyandi: Even when I use Google shoots, I still do it anyway,
W. Curtis Preston: right.
Prasanna Malaiyandi: just out of the force of habit.
W. Curtis Preston: Yeah.
That's funny.
, Prasanna Malaiyandi: So, lessons learned.
Lessons
W. Curtis Preston: Yeah.
Yeah.
So we, you know, so obviously we talk about, you know, it's still your
responsibility to backup up SaaS.
The other thing is a month RPO?
A month backup frequency.
That's, that's unacceptable.
That's just, I can't imagine that being acceptable, uh, to
in, in any world, by the way.
That is determined by your business, right?
I, I have met at least one company where a month long RPO was fine.
They were a paper mill, right?
They were, they were like, we don't, the, the, the, the computers, they just, they
don't have anything that we care about.
Right?
They were fine with a month.
That should be determined by your business.
But a month seems really long to me.
Um, and the other thing is, if your backup system.
Is this is, we don't know if this is actually the case,
but I'm inferring from it.
If your backup system overwrites the last good backup as part of
the process because you're so starved for storage, that is bad.
That is bad.
Bad, bad, bad, bad.
That is basic
Prasanna Malaiyandi: storage is cheap.
W. Curtis Preston: Bad.
Well, it's cheap, but I'm just saying it's cheap.
Right?
So much.
So much we do or don't do is because it's, it's inexpensive.
Right?
Prasanna Malaiyandi: Well, given that they are keeping all these
copies and dealing with cross site replication and everything else,
W. Curtis Preston: Mm-Hmm.
Well, they, yeah, they're not doing that now.
Right.
I'm just saying there are people, there are people that their backup system
is overriding the last good backup, and I just, that, that is a, that is
a really, basically, it means that if you're in the middle of the backup and
something happens, you have no backup.
Prasanna Malaiyandi: Yep.
W. Curtis Preston: Unless you're backing up the backup.
If you're backing up the backup right, then, then, then, right.
But if this is your only backup and you're overriding it with the next
good backup because you're too cheap to buy enough storage for multiple
backups, um, I dunno what to tell you.
Prasanna Malaiyandi: Yeah.
Please do not please step away from the keyboard and have someone
else design your backup system.
W. Curtis Preston: step.
That
Prasanna Malaiyandi: is true though.
W. Curtis Preston: perfect.
It is true.
Yeah.
Prasanna Malaiyandi: Yeah.
W. Curtis Preston: Yeah.
Uh, can you think of any other, um, lessons from this one?
Prasanna Malaiyandi: I know that we talked about how Dedoose may not have
had the best backup systems in place,
but this is also one of those things like it was.
Unfortunate that they had these multiple issues that led to such a big failure.
But it is good that they did see, okay, here are gaps in our system.
Here's how we're gonna fix it.
They were transparent with the users and it looks like they've gotten
their act together, which is good.
Right?
So I think it's just a matter of making sure they're continuing to stay up to
date and reevaluating their architecture and infrastructure of their backup.
Especially like, I don't know how they deal with ransomware.
W. Curtis Preston: Right, right,
Prasanna Malaiyandi: Like are they really using separate accounts,
like when they're replicating to different buckets, different regions,
or is it all like the same username, password, or whatever across the two?
Right.
W. Curtis Preston: You're killing me
Prasanna Malaiyandi: Right.
Because hopefully they're keeping their Azure or their AWS system isolated.
W. Curtis Preston: Right, right.
And hopefully again,
Prasanna Malaiyandi: than backups.
Yep.
W. Curtis Preston: somewhere, along the way they're using immutable storage.
Right.
And we, and we do like the way that they, that they had the blog.
They were, they were immediately, um, you know, transparent with what was going on.
They were, they were a little bit of, uh, you know, mea culpa, right?
They're like, Hey, this is our fault.
You know, this is all us, you know, uh, we're so, so sorry.
Uh, I'm a little disappointed that they deleted the blog after the fact.
Um,
Prasanna Malaiyandi: Put a link to the.
W. Curtis Preston: referenced in mo.
What's that?
Yeah, we will put a log to a, a link to the archive version of the blog.
'cause the internet never forgets.
Um, yeah.
So.
All right, well thanks for helping me work our way through
another cloud disaster Prasanna.
Prasanna Malaiyandi: No, thank you Curtis.
This was fun.
W. Curtis Preston: Yeah.
Fun, fun, fun and a sad way that unfortunately involved
a, you know, a deceased
horse.
Prasanna Malaiyandi: but.
W. Curtis Preston: Um,
Prasanna Malaiyandi: Uh, poor horsey.
W. Curtis Preston: yeah.
Um, well subscribe folks so that you don't miss all of this amazingness.
Uh, and, uh, that is a wrap.