Can you imagine finding out that a Chinese state-sponsored hacker group
spent over a year inside your network and your MFA didn't do a single thing?
That's exactly what happened to some academic and medical research
institutions, and why phishing-resistant MFA isn't optional anymore.
Today, Prasanna and I break down exactly how attackers harvested credentials,
hijacked REDCap, uh, which is a thing for research and education, Google
Workspace with fake compliant rules, and exfiltrated sensitive research data
for months without anybody noticing.
We'll also talk about what you need to do right now so that
this doesn't happen to you.
Here we turn admins into cyber recovery heroes.
This is the Backup Wrap Up.
Welcome to the Backup Wrap Up.
I'm your host W. Curtis Preston.
I have with me a guy who is super jealous of the fact that in the
last couple of weeks I have used, I think, every tool that I own.
Isn't that true, Prasanna?
the one that I had you buy
It, that is true.
a significant amount of use, of the one that you basically really talked me
into it, and I was like, "What do I need one of those for?" And now I have it.
We're… What?
I found you the deal
You did find me a deal.
of course that, that, the tool we're talking about at
this moment is the planer.
and if you've never had a planer, it's a, it's an amazing device, but it's
if you've never used one before, you don't know why you would want one.
And then once
use
get one,
"Oh my gosh."
you're like, "How did I ever live my life without this?" like the air fryer.
It's like the air fryer of tools.
I don't know about an air fryer
meaning you don't know if an air fryer is awesome?
Or if I would justify having an air fryer
that's because you've never had an air fryer.
See?
This is what I'm talking about.
That and you know what else?
The West Wing.
if you would just watch The West Wing.
so wait, so here's another interesting thing about Prasanna.
Prasanna owns, because you got a deal on Apple, like Apple, what is it?
Apple Plus?
Yeah
TV, and you bought the entire series for some ridiculous price, right?
Like, how much was it?
It was like $30 or
something
yeah.
and that was, like, a year ago at least?
it might be going on a year and a half
Yeah, and you've yet to see a single episode because you know that the
moment you do, you're gonna get hooked and you're gonna, you're gonna
here's my problem though, Curtis, is this is not atypical for me where I
will buy something and it will sit
Yeah.
for
years
this is something you and I share
Yes.
I w- so recently I've gotten back into watching physical media, right?
Yeah.
and all the rest.
Yeah
I had some which were still sealed for the last, 15 years that had never been
Wow.
Yeah.
that was kinda like me in the wood shop with,
Yeah
some of the, my dust collection, and now that's my new obsession,
is my dust collection.
Yeah.
Anyway, people don't want… They won't, they don't wanna hear about this.
They wanna hear about, Google.
Are you sure?
yeah.
so this is a, this was, this is another one of those stories where you're like,
"Man," like 'cause it, it's another, living off the land attack where
Wh- what's living off land for our
Yeah.
So basically, living off the land is leveraging, i- is an attack that leverages
y- the tools that you have in your arsenal, against you basically, right?
and, the, what happened here and, so first off, who are we, talking about?
Google's, their threat intelligence group, they have published a really
big report which we'll link, in the show notes, that really detailed, I
have to give them props, that they really detailed what happened here.
It's, it's actually a Chinese, PRC-sponsored attacker,
threat actor called UNC6508.
That is
Is
worst name ever
sometimes they have these Titles, nerd titles, I don't know, labels
yeah.
of these attacks.
But then the same organization, like what we might know externally
as LockBit or we'll take whatever
Yeah.
what's the name of…
ones
Yeah, what's the name of the one, the one that did the last attack that we covered?
Shiny hunters
It was the Canvas.
Go
the, so we didn't cover it, I'm sorry.
It was the last attack you and I talked about.
Yes
and it was the attack of Canvas, which is a learning platform, and, there,
because I saw an article that said it was the same, it was the same group.
I didn't check that, but, Google is calling them UNC6508.
and they targeted mainly, North American institutions, academic,
medical, things like that.
And via this tool, called REDCap, which is a Research Electronic Data Capture,
it's a web-based platform, and, they, th- which has a unique attribute that
allows you to run multiple versions of the software simultaneously.
And so even though you may be upgrading and patching more recent versions
of the software, if there's a, an exploit against an older version of the
software, they, they would be able to do it, and that's what happened here.
And once again, it started with harvested credentials.
we don't know where…
do they get?
they were, it was an admin.
it was an admin's credentials, but the, but again, this is they broke
some of the rules, and we're gonna talk, the meaning the victims.
they, broke some of the rules that, that we talk about, and we're gonna cover
them in the, that, that had they not done that, it would've been a different,
the attack would've been different.
but the scary thing about this is that they put in this, this malware
that's called Infinite, Infinite.red.
it sounds to me like that sounds like it's a s- a specific malware that
they wrote just for REDCap, and it sat there for how long, Prasanna?
It looks like from what I could tell this happened like November 2023
Good.
Yeah, it looks like literally they got in, and then they…
this is we talk about dwell time.
We talk about, just to go back to backups here for a minute, one of the real
problems with these types of attacks is that sometimes having a really good backup
isn't going to be enough because, what…
If they've been in there for over a year, like a year and a half, maybe even more,
probably isn't that far back.
no, right?
You're not gonna, you're not gonna have backups that don't have this.
And they… And what they did was they leveraged these, what do you call it?
v- vulnerable versions of RedCap to harvest their credentials, and
then, and then they somehow…
But wait, go ahead.
Go ahead
But even when they harvest the cred- credentials, it's not
like they wrote it to a file.
They wrote it back to a red capped database,
Yeah.
end product database.
Yeah,
they wrote it to
talk about living off the land, right?
and then they intercepted RedCap's upgrade process, so it then injected this
malware into every new version, right?
even if you were trying to patch things, you would get the new version, right?
It's an interesting question.
I know we talk about patching and, making sure everything's up
to date and having central IT.
Do you think a lot of these issues stem to the, from the fact that these were,
like, IT or software packages targeted at academia and researchers who sometimes
pull things together on their own?
It reminds me of, the shadow IT phenomenon, right?
n-
they're
no, actually, yeah, I don't think so.
From what I'm seeing i- is that, again, we're g- and we're gonna get
to the what could they have done better and what you can do better.
I think just, i- it's not the upgrade process.
it goes back to the harvested credentials.
It goes back to limiting the ability for products to do what they did,
and then we're gonna get to the next thing, which is once they had…
They had been harvesting credentials for a year, right?
for over a year, right?
And then they took some of those credentials, and they tried to log
into other systems, and this is where Google Workspace comes in.
And so what they had was, once they got… They were able to take those
harvested creden- they started with stolen credentials, then they used those
credentials to harvest other credentials.
Then they took those credentials, and they logged into Google Workspace,
Yep.
into, again, valid admin, as a domain admin, and they created
something called compliance rules.
You wanna talk about what those are?
Yeah.
So in an organization, you might have certain compliance requirements which
are, hey, if an email is from this person or to this person, or if it contains
these types of words, then forward it off somewhere else so then it can be secured
and can't be deleted and is protected for compliance auditing other purposes.
And so they created a compliance rule that said if it matched certain keywords
to send it out to a certain email address.
And this was ingenious because it just looked like normal email traffic.
So even if you were trying to detect something, it was like, oh yeah,
this is just part of your normal systems operations and all the rest of
Yeah
address was controlled by the attackers
And they would, and then it would BCC forward the, to a Gmail
address that they owned, right?
The keywords, very interesting.
The keywords, they're talking about geostrategic policy, military strategy,
advanced tech, and specific pathogens.
And by the way, that one was interesting because one pathogen that
they had on the list was chikungunya.
I don't know how to … Chika- You can say that?
Okay.
checking
I've never heard of that before.
So they're saying it's a mosquito-borne viral disease that's
responsible for an outbreak in China.
Yeah.
so interesting, right?
Oh, okay.
Gotcha.
hence your ability to pronounce that word.
I've never even heard of that word before.
anyway.
so they were… So on one hand, you're saying it just looks like normal
email traffic, but, I do think that perhaps somebody could have noticed.
By the way, the rule that they, the compliance rule they created was called
Patriot, which is interesting, right?
Didn't they misspell it?
You're right.
So they did sp- so it's spelled Patriot.
which again, might have been a, might have been a red flag, right?
but, and so the, basically anything that, that met these, the
filters that they were looking for gets forwarded to the bad guys.
And, d- Google, the Google Threat Int- Threat Intelligence
Group, is that what it's called?
Yeah, GTIG
Yeah, GTIG.
they, it says they disrupted the infrastructure, they disabled the
Gmail account, that was being used for exfiltration, and they notified everybody
and published, rules on how to stop this.
But, I always go back to, what could they have done differently, right?
and I don't wanna focus on just this particular account, but I don't think
Red Cap is alone in this idea of having legacy versions being, run side by side.
Can you think of any other products that work like that?
VMware
Oh, yeah.
Yeah.
Right?
'Cause when you were describing RedCap,
Yeah
I was like, "That sounds exactly like a hypervisor."
yeah.
you're right.
Is
and,
thinking or were you gonna give something
else?
no, I wasn't thinking.
My brain was blank.
but yeah, no, that's… I think that's a per- that's a perfect example, right?
where you leave, you leave many versions running.
and so really what they could have done in this case is, again, obviously
general password… we can start with general password, hygiene, right?
The, that if they had been doing normal password hygiene, the
passwords wouldn't have been harvested out there somewhere in the wild.
And then because the initial attack started with, passwords that have
been harvested out somewhere.
they don't know where that happened.
and if that means that they were… somebody was using a username and
password externally, potentially that, that then was used internally.
so if they had good password hygiene, that wouldn't have happened.
The next thing is to not…
Go ahead
I'm surprised you didn't start with the number zero.
What's the first thing that you do before password?
list of three things, if you did these three things,
Oh, patching?
You talking about
And so for patching, I was gonna say inventory management,
Yeah, absolutely.
and you knew what was out there and then you were able to patch
it, you probably may, or you may have avoided some of these issues
Yeah.
and then the big thing with the legacy version, support is to not do that, right?
you don't run things that you don't, at a minimum you disable them.
the best practice is actually to remove the old version completely.
don't leave it sitting around.
so another thing is, to not trust application-level
authentication by itself.
to use something like SSO, this is an important enough thing that you don't
trust the IAM system of just any old, piece of software that you use SSO to
log into important things like REDCap.
and had they done that… A- and by the way, this had, when we go back to
the, we talked about the Canvas hack, same thing there that, I'm currently
talking with a client that they were hit by the Canvas hack, but it was very
minor because they use SSO for the vast majority of accounts when logging into it
One other thing, and I don't know if we've necessarily touched on
it in past episodes, using SSO or a central identity provider,
you can enforce requirements.
So if a password needs to be rotated, do you use MFA, minimum number
or, minimum number of characters?
All the rest of these things you can enforce versus if Redcap had their own
password system and, say, didn't have this functionality, kinda limited.
Maybe someone hasn't changed their password in seven years.
Who
Yeah, or even if it had that functionality, that means you're
managing that functionality in every, SaaS app that you use, right?
Yep.
yeah.
Good point.
you're an organization, you really should have a central password system,
Yes.
system
There's an, entire, there's an entire industry built around
that and, thumbs up to that.
the next one… Go ahead.
What?
I touched on something which I think might be the next one
Okay.
Touch away.
one MFA?
Is it MFA?
It's always
the next…
like it's always DNS
specifically phishing-resistant MFA.
yes.
You wanna talk about what that is?
Yeah.
So like we've talked about, someone can guess your password or steal your
password, like what happened here.
But with MFA or multi-factor authentication, just having
the password isn't enough.
It should also send you a way to authenticate in addition to just knowing
the password, the username and password.
Typically, you see this as, a code that shows up on your phone.
email is not great, SMS is not great, but it's better than nothing.
but
you really
should be using like a one-time password
Yes
there.
Many versions are out there.
So that helps.
And then the other thing that you mentioned, Curtis, is phishing-resistant,
So you wanna make sure that someone doesn't go and steal your, MFA
token and then start to use it, that you do have … I know we've
talked about this in past episodes.
There's this notion of MFA fatigue, right?
Where people constantly keep pinging you, being like, "Is this you? Is
this you?" And then you click yes because you're tired of saying
no and responding all the time.
So you want
something that
is more resilient to those.
I think the big thing with truly phishing-resistant MFA is that MFA
that is tied to something, right?
The ph- it's tied to a security key, it's tied to a particular location,
it's tied to a particular device, so that even if, a- again, just
the… It's MFA for the MFA, right?
That, that, that if they steal a session cookie, which is what was happening here,
they were stealing session cookies, then they wouldn't be able to just replay
that because the system would notice that it was coming from other place.
Yeah.
and then of course you can and should investigate, passkeys, right?
and again, the beautiful thing about passkeys is that it solves all of this and
it's, they're tied to a location, right?
What's a passkey?
thank you.
So a passkey is basically a complete replacement for passwords
and, MFA, and it's basically a…
Think of it as a locally stored, I was gonna say password, but it is a key that
is stored locally with the device that you have, that is tied to that device and
tied to that account, and it's basically played on your behalf, whenever you
need to log into that, that account.
So passkeys fall under what's called FIDO, which is Fast Identity Online,
and the common thing that is stated…
There are multiple, there are many known attacks for passwords in MFA,
and you taught, you touched on one of them, which is the concept of,
becoming, an MFA fatigue attack, where they send you so many things
that you accept one of them, right?
And then they take the accepted one, and then they go do bad things, right?
there are known, there are no known attacks against FIDO-compliant passkeys.
So if you're not doing them everywhere you believe you can be doing them, you
should be investigating that right now.
if y- if you're familiar with that and you're working your way, I would just
say prioritize that wherever you can.
if you don't know anything about it, then, you should be looking into passkeys,
One of the things that GTIG pushed, in terms of what could have helped here
is the concept of device-bound session credentials and context-aware access.
And w- we talked about this a little bit.
Context-aware access, if you enable that on an account, it's going to do things
like, why is this coming from a different place than it was 30 seconds ago, right?
and why is it coming from this IP address?
Why is it coming from this country?
I thought this person was in California.
Whatever.
It's a context, right?
A con- and every time you go to access, it's gonna check that context.
And then you have the device-bound session credentials, or DBSC, and
what they do is, they're tying each…
'Cause when you authenticate with a, a browser, that creates a session,
and if you're using DBSC, those credentials only work in that session
and they tie it to that device.
It's a little bit like passkeys in that regard, and Google is just
talking about these especially for highly sensitive accounts.
If you turn on these two features, it would tie, once you authenticate,
it would tie that authentication to that session, and then context-aware
access would check that, so the two work together hand-in-hand.
And what that meant was if and when someone, like this malware, steals
your session credentials, they wouldn't be able to use those anywhere else
Yeah.
And specifically, this is the case where they're not just stealing
your password, but they're actually
stealing, say, your browser cookie
your cookie and trying to replay and reuse your cookie on a different device.
Right
By using device-bound session credentials, you're guaranteed that
even if they stole that session cookie, it's not gonna be usable anywhere else
again, it's specifically something that, that they, talked about,
or that Google talked about.
so the next is this idea of compliance rules, and they're not… Th- this is,
that's what Google Workspace calls it.
Every system has something like this, and the idea is look at any
standing rules that are there.
In a large company, you may have tons of them.
This is where I think AI can be helpful here.
Look for things that are created that are forwarding email out to
especially external accounts, right?
Didn't you
that's what's-
anytime something gets touched in these compliance rules?
yeah.
A- agreed, right?
Ha- have a compliance rule for the compliance rule.
that anything that's a new compliance rule, th- it should trigger some sort of
event so that you can then go check that.
That's a great, that's a great point.
So then the next one they recommend is something you might think
is obvious, but I guess a lot of people get this incorrect.
It's separating your credentials across security domains.
So as we looked at this case, RedCap, one security domain, Google Workspace, another
security domain, the two streams shall never cross, and yet someone used the same
credentials across the two, and that's what sort of allowed a breach of RedCap to
now impact their Google Workspace account.
Right
really should be keeping credentials separate if they need, if they don't need
to be common, or using SSO or an identity provider which deals with all of this for
you so you don't have to worry about it.
Yeah, or also PAM, so privileged access management, right?
and go ahead
Would… Here's a question for you, Curtis.
Yeah
this comes up a lot of times when we talked about backup systems, how keep
backup systems separate than your production systems, from a credential
management, active directory perspective.
When a company is using SSO, say O- Okta or someone else like that,
do you think it is important to still think about these security domains and
so have a different SSO user for Red Cap versus their Google Workspace account,
or is that overkill since you have Okta providing some of that control?
W- yeah, when it comes to admin accounts, I don't think there is
such a thing as overkill, right?
I think you, you need to treat admin accounts differently.
perhaps you do one thing for, regular users.
And again, going back to the client that I was talking about, that they,
the reason why they were only, they used SSO for all the normal users,
but admin accounts were separate.
so I think that is, that is important.
You can use SSO to layer on top of that, right?
So that they work together.
But I think, I still think you need to, keep that separate for the, for all
the reasons that we just talked about.
and then the next thing, we talk about here is get some logging, man.
Some sort of XDR, some sort of SIEM, some sort of, some sort of monitoring going on,
looking for the kind of things that were happening here, because they had to be
sending a butt ton of sensitive e- email out to external accounts, for a really
long time, and, nobody seemed to catch it.
I don't, that, that's quite disconcerting
did they say when they actually breached Google Workspace?
'Cause I know
I, it was towards the end.
It was towards the end of the attack.
Yeah.
It was the last thing they did after they'd been harvesting
credentials for over a year
Yeah.
Crazy
Crazy.
So here's a question.
So we talked about this, right?
And things you could do to prevent attacks.
Given the name of the podcast, is there anything that you would think
Did we forget something?
from a data protection, data recovery perspective, from a
cyber recovery perspective they could have, should have done?
Or is this even if you had the most amazing
cyber recovery plan in place, you didn't do any of these basics, you're hosed
I don't think, this is one of the situations where I'm not sure how much
backups would have actually helped, right?
Because it, you do need, obviously, you, you're never gonna be saying
you don't need backups, right?
one of the things we talk a lot about is just please, for the love
of everything, please make sure that you have regular, automated backups
of everything that are on a sec- separate security domain, and also that
use truly immutable backups, right?
That i- you, 'cause the worst thing is when you hear something
like this, and then you see that the backups were also impacted.
The downside here is that backups quite possibly aren't going to
help you because d- two things.
One is, many people don't store their backups longer than the amount of time
that this attack took, number one.
Number two, what are you going to recover?
Are you going to literally, like you found out that they first compromised
your system a year and a half ago.
Are you going to restore your database to a year and a half ago?
You're not going to do that.
so the related topic here is, from a system standpoint, one of the
things that we came to in the book, which we haven't mentioned the book.
if you're not familiar with the book, that should be over my,
right shoulder in your view.
if you're watching this, it's my left shoulder, but it's
right in the video, right?
I think I'm pointing right.
Yeah, it's to my right in the video.
Anyway, the, is, Learning Ransomware Response and Recovery,
which, I wrote with, Dr.
Mike Saylor, who is a frequent podcast, guest.
that, Dang it, I lost my train of thought.
We were talking about the… Oh.
So you really have to approach backup a- as two things.
We really came to the r- reality that th- there, there were three ways to
restore after something like this.
One is to just restore back to a point in time before you, you know that you
were attacked, and that's like the last thing we want you to do, actually, i- in
terms of restoring the operating system and the applications, because it's really
hard to know when that point is, and it's really easy to reinfect your systems.
So that's one method.
The other is this idea of, restore and then go in and surgically
clean each system before you release it to the public, right?
And that is better than just knowingly restoring, without doing any cleaning.
But again, it still could be really difficult to find some
of this very pesky malware.
You talk about the malware here that could- kept reinfecting.
Sometimes the malware will be, in the firmware, and it will
reinfect you after a reboot.
So really, I think the only real method here isn't just, I guess it's
technically backups, but it's really an automated, like infrastructure as
code type environment, where what…
you know what your… when we talk about VMs and applications, you know what
version of the OS you're running, you know what version of the app you're running.
You should be able to push a button and boom, you have a new
VM of that from a trusted source
yeah
that is, that was created when you first made that system.
This isn't something you're backing up.
So what you're backing up is the application data, the database and,
and the documents and all of that stuff, and then you, when it's time
to, to rebuild, you rebuild the OS and the application, and then you restore
the actual data for that application.
And generally speaking, the data itself won't be infected.
Go ahead.
I have a question for you.
Yeah
Do you know, and maybe this is also a Dr. Mike question, any ransomware actors
who have attacked the Golden Images?
I am not aware of any, right?
I would I think I would argue… You mean the actual s- so
where the images are stored?
in the sense of if they infected the golden image, right?
The, like you said in your
Like a supply side type attack, basically.
Yeah
right?
Or it during the attack, yeah.
If they, if they infected those images as you're trying to recover
and rebuild, like you said, right?
Rebuild the OS and everything else, you're basically reinfecting yourself.
Now, hopefully
Yeah.
testing those as well before they
Yeah
them during rebuild, but maybe that's something
yeah, I, that's, I, it's a good risk to think about and I guess, in my mind,
any golden image is stored either on worm media or on, immutable… Like
worm media like a DVD-type situation or on truly immutable storage.
But because again, you gotta think of it like a backup.
you gotta make sure that there's no way they can attack it, right?
And
so this is something that you're creating right in the very beginning.
Yeah.
ahead
do wonder how many people actually think about that, 'cause
I don't think they do
until you were actually just talking through it and
Yeah.
Yeah, you have to think about it like a backup, and that backup has to be stored
in a way that no one can ever get to it.
And again, my standard there is if you can't delete it even
if you want to, then I'm happy.
If it's anything less than that, I'm not so happy.
and there are a bunch of ways to do that, right?
You c- there are truly immutable storage platforms that even you can't delete.
There's different modes.
sometimes you'll see compliance mode and retention mode.
you want the more restrictive of the two.
Again, the standard is i- if you can't delete it, even if you want to, then,
then that's what you want, right?
and, you could also do that with worm tape, and you could
do it with worm, optical media, one of which we'd covered here.
M-Disk, right?
the media that's meant to be around forever.
I wanna see hackers attack that.
good luck with that.
It's a, called a
blowtorch.
Which is what I was saying.
Unless, like the only way you can this media is, like you said, if you
have physical access, all bets are off
Yeah, you, yeah.
That, that's true with any element of cybersecurity, right?
so you, that's why physical access is, paramount.
and I go back to the best that I've ever seen there, where it
took multiple layers to, to get, to just to get in the building.
I remember
Yeah.
I did this a couple years ago with a client where it took a half hour just to
get in the building, and then once you were in the building, you were monitored,
because physical access is king, right?
all thanks again for, chatting about, one of my favorite topics
I know.
I love these topics.
Come on.
I was talking about woodworking
Oh.
I'm glad to see at least you're using the tool which I
guilt-tripped you into buying, so
Yeah, it was a lot of fun.
I made, for a preschool that I work with, I made these, plexiglass
windows that were… I m- I made window frames, out of two-by-threes.
Yeah
They were, yeah, two-by-four plexiglass window frames.
I made seven of them, and they were incredibly inexpensive because I
did them with all of my own tools.
And, yeah, it was very cool.
All right.
folks, thanks for listening.
if you didn't listen, why would we even do this?
that is a wrap
The Backup Wrap Up is written, recorded, and produced by me, W. Curtis Preston.
If you need backup or DR consulting, content generation, or expert witness
work, check out backupcentral.com.
You can also find links for my O'Reilly books on the same website.
Remember, this is an independent podcast, and any opinions that
you hear are those of the speaker and not necessarily an employer.
Thanks for listening