Hello, listeners. And welcome back to another thrilling episode
of data driven. In today's episode, we delve deep into the
fascinating and, let's be honest, slightly terrifying world of
generative AI and security risks. Joining us is
Niamh Braun, co founder and CEO of Noma Security,
who's on the front lines of keeping your AI driven project safe from
digital mischief. So grab a cuppa and let's get data
driven. Well, hello, and welcome back to Data Driven, the podcast where we explore
the emergent fields of AI, data science, and, of course, data
engineering. Speaking of data engineering, my favoritest data
engineer in the world can't make it, today. But we
have an exciting, conversation queued up with Niv Braun,
who is the cofounder and CEO of Noma. Noma
is a security firm that focuses on effectively
he'll describe it more eloquently than I can, but effectively thinks about
security in the context of data and AI across the
entire life cycle. Welcome to the show, Niv. Hey,
Frank. Happy to hear you, bro. Yeah. It's good to have
you. And and security is one of those things where I've been thinking about more
lately. Right? So my background was a software engineer and,
you know, software engineers historically have not thought of
security. Then I made the transition into data engineering and data
science, and, traditionally, security is not really at top
of mind, for them either. Now I
kinda look at this, and I kinda look at the landscape that we're in where
enterprises are deploying LLMs,
generative AI solutions, on top of the predictive AI solutions,
fast and furiously, and not thinking about
security ramifications. So what are your what's your take on
that? 100% agree. I think that, it's
even like the the the the current, like, timing is even more fascinating
than the than just, like, a new technology. Because exactly like you said, like,
Frank, like, we all like the data practitioners. We all know that, like, security is
not, like, our top priority. And by the way, like, by, like, like, this is,
like, how it should be. Like, we are focusing on the business and, like, drive,
like, drive, like, the business forward. And this is why we're, like, this is
what we're paid for. The problem is that
because we're not, like, in this kind of, like, mindset, we also, like, like
any technologies in the company, also, like, create some risk. What we see right
now is the LLM drive, which is pretty cool, is that for the
first time, the security teams started to put
the focus and, like, the spotlight on the data and AI teams. Because until
now, let's be honest, they were focusing only on the software developers and
their SDLC and the CICD and all these areas. Like, we were,
like, you know, like, in the shadow. And we were, like, able, like, to act
like exactly like, like, like, completely freely as we wanted.
But now when, like, the security team start, like, to put the spotlight on the
data and AI teams, what they understand is that it's not
only this new kind of LLM threats, but also all
the basic principles of security are not implemented
in the data engineers and the data science teams. Nobody, like, scans all the
code in our notebooks, for example, unlike the software developers that, like, all
their code is being scanned. Nobody helps us to
find configurations in our data pipelines or our
MLOps tools or our AI platforms, like Databricks, for example.
Like, nobody, like, provide us this ability to to find it easily,
unlike, again, the software developers that they receive all this coverage
and everything. Like, on the moment that they have, like, the smallest misconfigurations
in their SCM or their their CICD, they
will immediately, like, receive, like, a notification, like,
helping them exactly, like, how to secure it. And also eventually,
like, in the run time, in the runtime, in software life cycle, in
classic like software application, we also have a lot of API security and web
application firewalls tools that help us to protect the application in the
runtime. But now specifically in LLM, this is, like, very
related also, like, to what you said. Like, there are new kind of adversarial attacks,
all the prompt injection and model jailbreak and stuff like that.
And, again, nobody, like, else would like to protect it, like, in real time. And
I think that this is, like, one of, like, the main shift that we see
today in this area. We understand that the spotlight
moved to the data and AI teams, but we need to make sure that we
do, like, both. Like, we start with, like, a new kind, like,
trendy, like, risk that we want to make sure that we are protected from.
But also that for the first time, after a lot of years, we're
starting also, like, to implement the basic security measurements
needed in our area. But the most important thing, of course,
is to continue and, like, do it without slowing us down. Like, we need to
make sure that, like, everything, like, all the different, like, security measurements that
we take still provide us the ability to move fast, to enable
the data sent the data science and the data engineering teams to
continue and, like, innovate, but in a secure way.
You know, that's a good point because I never thought about scanning a notebook for
errors. Right? Shame on me. Right? Like for code
security I mean, not errors, but, you know, security vulnerabilities. That's not something
that I have seen done in practice. I mean, the the
closest I've seen where security has been an issue for
anyone in this space is,
basically using protected, you know, Python
libraries, right, or or Python library repos, right, where they're those
are scanned by, I forget the name of the 3rd party that'll do it where
you just basically say you point your Python instance to there. Yeah. Because
I also think that Internal Artifactory. Yes, exactly. So
like, what, because I often
wonder, you know, people just like to install.
God only knows what's in there. I can tell that, like, it already, like, happens.
Like, I don't know if you heard, but for example, like, like,
like, pretty recently, PyTorch, for example. Right.
PyTorch that we all know was compromised. We all know and love. We're most people
love. It was compromised. Like, specific version of PyTorch, a
malicious actor succeeded to to put some
code inside that basically,
collected all the the the secrets and token that you have in the
environment and sent it to DNS. Now we all
know, like, how much like like, how many downloads, like, PyTorch have.
And most times, where PyTorch is downloaded to through, like, to
all these different, like, notebooks, wherever they be, JupyterOps,
SageMaker, Databricks, like, we all use them.
And it I can tell that, like, it caused us to a lot of, like,
problem. I can tell, like, like, like, firsthand, like, we saw, like, a lot
of organizations that were compromised because of this attack.
And it happens all the time. And by the way, if you mentioned, for example,
like, if you already, like, touched the point of, of open source,
now you have also Hugging Face, which is completely different area. Now it's
not only Open Source packages. It's all these different Open Source
Hugging Face models and Hugging Face datasets. And there,
all these internal artifact are completely useless because they don't even
scan these models. It's completely different technology, completely different, like,
heuristics in order to find it. And, therefore, you start to
see kind of, like, trends for for the attackers. They started to
upload a lot of backdoored and a lot of malicious models
into Hugging Face. I can tell you, like, we personally, we already,
like, detected, I think, almost, like, 100, back
or the malicious models, on Hugging Face because it's a wild
west. Right. Because how do you because these these models, first off,
they're physically large files. Right? So that there's that's a factor.
Right? I don't know how Hugging Face makes money. I'd be
curious to have someone on the show talk about that. But, you know,
they're doing the service. And, how would you even scan? I
mean, that's a good question. Right? What types of vulnerabilities have you sent have you
found so far? And how does one even scan, like, a safe
tensor or g file? Like, how do you what's what's
that look like? Right? Obviously, I'm pretty sure, you know, McAfee
antivirus doesn't have a thing for that. But, like Exactly.
But, how do you even do that? I'm just curious. Yeah. So this is, like,
exactly, like, the problem. Like, it's even, like, in in in the models, like, it's
even, like, a a more, like, the the risk there, like, is
more, like, clearer because as you know, a lot of time, like,
these models in hanging face are even, like, in pickle. And, like, pickle is, like,
by design, like, insecure, like, file. And so
binary dump, right, of, like, the memory space. Yeah. Like, in the deserialization
process, like, basically, you can, like, put, like, any kind of, like, malicious,
action that you'd like, that, like, the attacker can. So we see,
like, different attacks. Like, most of the attacks come today, like, from pickle files.
Some also, like, not even, like, in the deserialization process, but also, like, in the
model code itself. For example, like, if you ask
for a specific example, like, share something that we
detected, like, recently. We found, like, a very,
let's say, a popular, open source, LLA model that we all
know. But we know that, like, a it has a lot of, like, different
versions. And one of the version was actually a docker
that took the original model, wrapped it up with few
lines of code in the model, which what they did is that every
input to the model and every output from the model
was also sent to the attacker, which basically
just received full visibility and observability to all the
runtime application and production. So, like, all the organizations that,
like, use this model. And performance wise, the
data scientist, of course, they cannot, like, detect it because performance
wise, it worked perfectly because it took the original model. So nothing to be
suspicious about. If we want the data
scientist, every new open source model that they like, like
in Hugging Face, they'll start, like, to open, like, these files and the binaries and,
like, to start, like, to looking, like, in their own hands, they're manually
for, like, a for a for risk. First, like, of course,
like, we understand that this is not their expertise and, like, it it
like, we want to be secured, but, like, like, even, like, worse,
we just spend all their time on security. And I think that
this is, like, the worst stuff. Actually, it's not the worst. I think that, like,
the worst, and this is also, like, something that, like, I saw recently in several
organizations is just, like, to block everything. Organizations
that, like, understand, okay, Hugging Face model, it's, like, true, like, a secure, like,
in secure area. Let's block it. Let's say, like, to
all the data scientists in the organization, you're disallowed to use HAG interface model. I
think this is, like, the worst. That seems like a mistake because
because the people are gonna find a way. Well, 1, where you can't stop the
signal. Right? That was a line from, a movie.
They can't, kudos if people know who that what movie that is.
But, you know, if you block Huggy Face, people are gonna find a way
around that. They're gonna put it on a thumb drive at
home and then bring it in. So percent. This is, by the way, also, like,
what you see, like, with this kind of, like, internal Artifactory. You see that, like,
once you get to you you create for the r and d or create for
the developers or for the data scientists, you create some level of, like,
friction. They will just find a way out to, like, bypass
it and to to lower this, this friction.
Right. So so couple of questions.
One, I've seen, improper naming
Not improper naming, but but basically using, names,
like, that's looks similar to what should be. Yeah. Type will split. Type
type splitting. That's it. I've seen that, which is kind of, I guess,
kind of, you know, dollar store approach. But also,
how does how does it if you wanted to look through these model files, as
far as I know, they're just I just looked at them. I just see binary
stuff. Like, how would you look for malicious code in there? Because I think you're
right. That's not a skill set the average AI engineer or data scientist
would have. Yeah. So, basically, like, you need, like, to manually kind of, like,
parsing it because, like, you have, of course, like, the the binary file, but most
times, it's not only, like, the binary file. You label for, like, the the code
file that run, like, run the model, and you label for, like, the, in
case it's, like, pick a, like, the deserialization process, that you can, like,
parse and then, like, to see, like, the code there. But then you
need also, like, you know, like, you have, like, 2 phase. 1st, you need to
to parse it, you know, like, to see, like, the code, but then you need
also, like, to be able to read code and to understand which
one is valid and which one is malicious, which is also, like, completely, like, you
know, like, you need expertise in this area. If you see bash
commands, is it okay or not? Do you see access to the
Internet? Okay or not? Like, you you need, like, to have, like,
some, like, detectors in there that, that know how to do it, like, build
by by expert or something. So how would you even detect
that if you found it? Like, how was this found? Was this just somebody looking
in network packets? Or, like, what how was it discovered? I'm
just curious. Yeah. This specifically was, like, by our
security research team. Okay. Yeah. That's like, looks a
lot if, a lot like all the time, like, you know, all these different kind
of, like, open source and third party models in order to to help
our users to make sure that, like, everything that they use
is is valid. And again, most importantly, without slowing
them down. They can just, like, download and, like, run, like, with everything that they
that they want. And in case, we see something that is,
that is suspicious, we know how to detect it and to to help them to
to secure it. Interesting. Interesting.
Because I know a lot of people, you know, they they've been downloading
these models from Hugging Face. And just taking it on
faith, and I've heard that these things don't call
out to the Internet. Mhmm. And I fell into that. And then
I kinda had this moment of paranoia where I'm like, how do I know?
I mean, the only way I'm a I'm just a humble data scientist. Right? Like,
so the only way I would think about it would be to have a firewall
rule that would block network traffic going up for that box.
And I'm sure there's probably workarounds to that too. I mean, are these
attacks are these attacks that sophisticated yet?
Yeah. Yeah. And, like, also, like, most times you don't, like, the data
science, like, they don't want, like, to permanently, like, to close, like, the Internet, like,
the outbound because also, like, the application needs it. And also, like, the, you
know, like, the the in order, like, to download, like, the dependencies and the models
you needed. So most times, like, just, like, to block the Internet, it doesn't solve
everything. It was, like, more, like, in the past that everything was, like, network based
only. Today, when you have, like, also, like, the applicative layer here, so
it's, like, a bit more sophisticated.
But yeah. Wow. So
the safe tensor format, as I understand it, what you
know, you basically digitally sign or somebody
digitally signs the contents of it. Is that is
that a correct understanding? Yeah. So it's end up like a
like, in general, first thing, of course, that, like, a safe denture is, like,
much more secure. Okay. I already like by design, and as long
as we as the industry will go, like, more and more, like, towards
this road, because today, like, we still see, like, tons of light pickles.
But as long as we progress, like, all as an industry, we'll already,
like, be, like, in a bit better situation. It's not
perfect, of course. We still see some issues. And, of course, organizations still
need, like, to have some security measurements and processes
to make sure that, like, they're aware of what,
like, Hang in Face are using. But I think that it's already, like,
going to be a bit better. I can tell you something that, actually,
like, recently one of our one of our partners told me,
which was pretty cool, very similar to what you said that you
start, like, to feel a lot of concerns about this area.
VP data science of a very big like,
Fortune Fortune 500, like, very big, like, corporate. And you kind
of, like, the the head of, like, the older data science, like, groups here. And
they told me, you know, Niv, I I already know
what I'm going to be fired about, like, in a in the next, like,
24 months, and it's going to be about that. I know for sure, like, we're
using, like, so much, like, Agiface models. I know for sure that I'm this is,
like, the reason that I'm going to be fired, like, one day. Because today, like,
we're using it, like, freely. We are also, like, very creative. We're not, like,
only using, like, the most popular LAMA model, but, like, we're to,
like, take advantage of this great advantage of the platform, which is, like,
the amount and the diversity of the model that you have there. But I have
no no doubt that we create so many risks that we're just,
like, not exposed yet, that I'm going to to pay with it,
like, with my head. So it it's
it's pretty cool because it's not it's not always that you see,
r and d and business owners that are so concerned
about security even before the security team arrived
to them. But they're already aware of this risk. And it's something that
we start, like, to see more and more because, you know, it's just like it's
it's too obvious. Like, the the the window is open and everybody see it.
Yeah. I I would suppose that's in in a in a very kinda strange way
that's bit progress, right, where people think about security beforehand.
Like, even if they don't know I mean, I think this this this VP,
you know, is pretty spot on. Like, what concerns me about the widespread
adoption of these models and particularly Hugging Face, so there are no knock on Hugging
Face. I think whatever you get your models Mhmm.
I mean, we just don't know. And these things are just complicated.
Right? I mean, they are by design complicated with 1,000,000,000,000 of
parameters. In some cases, I guess, 1,000,000,000,000. But also, you
know, they have this ability to even
even if everything worked out well, even even assuming everything is
fine, right, in terms of the operationalization of these things,
There's still the chance that the model itself and its
training was poisoned. So, like,
I I mean, like, there's just so many because when I my wife works in
IT security, and I was all excited. It was about a year and a half
ago. I I was talking to her about LLMs and stuff like that
and chat GPT and and and those types of things.
And I was like, oh, well, you take all this data and you train a
model and you you distill down this graph and this and this. And then she's
like, that sounds like a big attack surface to me. Yeah.
And I was like like, data poisoning in the classic one and data
poisoning can be, like, in in in 2 levels or, like like, someone
like poisoning your data or exactly what you say,
somebody just, like, this way, like,
create backdoor in, in third party models and open source
models that then, like, everybody downloads. Right.
Right. And we wouldn't know, like, what's
the I mean, the defense against that seems very
intricate. Not impossible, but very delicate and intricate.
So in in in classic application security, there is a
great practice called SBOM. SBOM is a software
billing of material. Basically, it means that, you get, like, in
specific format, kind of like visibility to all the different
software components that build your application. One of the things that
now we're also, like, part of the building is a
official framework of OWASP, the nonprofit organization
around security of AI and machine learning. And
what you have there is for the first time you have like double layer
of visibility. The first one is just like to understand
what models I'm even using in the organization. Everything, like
what models like, include in my application. It can be open
source models. It can be self developed models. Also, by the way, not only not
only LLM, of course, also like vision, NLP, like everything else.
And also third party models that are embedded as part of the application, they
are not open no. They are not open source. For example,
if software engineer add API call as part of the application
to OpenAI, in this way, they embed
LLM as part of the application. This is also like one of, like, the models
that you are using, but you you you want to know this is all my
AI and model inventory that I'm using as Spyro as part of the application.
And in addition to that, you have even the deeper context there, which
is also like what you referred to. It's not only this is
the list of the model that I'm using, but for each one, you want to
understand on what dataset it was trained, what data
maybe also like it has access to in case it's in production, let's say, with
RAG architecture. You want to understand, like, the deep context
of all these, like, models, what I'm using, but also, like, what
happens, like, in this specific, like, model. Sometimes
it's, as you said, for to to understand what data was trained on a
model before, like, I'm starting, like, to use it by 3rd party, a
lot of time is even, like, internally in the organization.
Because once we start to train a lot of models,
we want to make sure that we don't violate
any policy that we have in the organization, either it's for compliance or
security. For example, one of the things that, like, we are like, I keep, like,
hearing a lot of time from, from security and legal and privacy
teams is that, look, we instruct all the
organization not to train any sensitive
data, PII, PCI, PHI, any other sensitive
information on our models. But except instructing
it and speak about it, nobody knows if it
happens. And we don't provide also our data
teams tools that will help them to
detect it in case it, like, it happens, like like, not in purpose. For
example, I can tell you, like, one of the thing that we saw very
recently. Big organization, a huge Fintech company,
that data scientist unintentionally trained all the
transaction of the application on one of the models. Now it's
a, like, crazy big violation there of, like, compliance and
security. The data scientist did this unintentionally. They
truly, like, didn't know it. If they had something that, like, would help them, like,
the basic visibility that you mentioned before, it will truly, like, help them to
start, like, to continue, like, innovate and just, like, in case something like bad happens,
to be alerted in that. And so I see that, like, the the data training
is also, like, very, very important point also internally and not
only the external data train on the external models that we're embedding and
downloading. So you mentioned, OWASP. So just
for the benefit of folks who may not know, because most of our listeners are
either data engineers, data scientists. What is OWASP? And what is the
I think it's with the OWASP 10? Yeah. So
OWASP in general, it's a amazing organization that,
is like a nonprofit one that helps basically,
we combine a lot of people together, gather together in order to make
sure that all our industry is much more secured with a lot of
different security initiatives in a lot of different aspects, mainly of like product
security, but not only. Product security is like application
security. It's building security.
Specifically in OASP, you have several different types of
projects. So for example, one type of project is the OSP10,
top ten, that basically takes different areas
and define the top ten risks in this specific area. So it can be top
ten for API, top ten for
CICD. And now there is also like top ten for LLM.
Addition framework, like, there are a lot of like different tools. Specifically,
if someone wants to understand a bit more about like the wide
landscape and the risk around AI and machine learning,
the framework that I would like recommend on, highly recommend on, is
amazing and very comprehensive called the OWASP AI
Exchange. A group of people, again, gathered together,
that covered not only LLM, but all the basic
principles and risk in data pipelines and MLOps
and start from the building and up to the runtime and start from the
classic machine learning and up to Gen AI, very comprehensive,
very also practical, which is very important and
speaks in both language, on both languages. On one hand,
of course, security, but on the other, also like very oriented
for data and machine learning and AI practitioners.
Interesting. Interesting.
What what do you see
well, here's what I mean, I'll have a lot of questions, but one of them
is, do you think the 0 what do you think the
0 trust approach is a good starting point? I don't think
it's the answer here like it is kinda everywhere else. But do you think that,
that type of philosophy of don't trust anything?
Right? Kind of like, I mean, is that because you you mentioned this
early when I talked about network firewalls, right, where the old approach of thing
is just pull the plug or set up rules. And that used
to work, but there's plenty of other ways around it, Both I think
kind of low skill, mid skill, and certainly high skill
ways around that. What do you you mean then 0
trust is meant to address that. What are your thoughts on like I
mean, is that the pro is that the mindset that either
security folks in this space would have to take on? Like, it's more
if they well, they probably already have. Right? Yeah. I think you're,
like, I think you're actually, like, the the you you you perfectly
defined it because I believe that 0 Trust is exactly like you say, it's kind
of like a, like, kind of like a mindset. It's not like a very
accurate, like, technical approach, but it's kind of
like more like a a philosophy with some level of implementation.
I believe that, like, the right mindset and, like, the right framework to look
on a on a security for AI and, like, all the building
and also, like, the runtime is basically to take all the
different principles that we are all already aware
of. Like we are all, like I'm saying, like the security industry,
we are all already aware of on classic software development,
building and runtime, and to implement it on the
data and AI lifecycle. For example, if we mentioned, like,
code scanning, so code scanning the notebooks, we mentioned open source,
so checking all the all the Ag interface models. But it's not only that.
For example, one of the things that, like, we see, a lot of attacks that
we, like, we had recently in the security area are around the
CICD. A few years ago, there was a big attack called
SolarWinds, that basically, yeah, so you know it
perfectly, just for the audience that, like, are not familiar with the specific details
in, like, very high level attacker that exploited and
misconfigurations in CICD tools. And this is
basically how they succeeded, like, to start, like, this whole huge attack and
breach. Now one of the things that, like, it taught us all as an industry
is that until now we were focusing on, like, securing only
our code. Now we understand that the code is not enough. We need to make
sure that the building tools are also well configured. So
we start, like, to see a lot of, like, tools that help us to make
sure that we don't have misconfigurations in the CICD and the SCMs and all
these different kind of tools. But when we are going to our domain, when
we go to the data and AI teams, as we know, we just use different
stack. We use all these data pipelines and model
registries and MLOps tools and platforms like Databricks and Domino
and Snowflake and stuff like that. The configuration, as we know, is
not like neverwhere. Most time, it's even wider. This is why it's
not managed by DevOps. It's managed by us, by the data teams. It's managed by
MLOps teams, by data infra, by data platform. And we're doing a
lot like, a great job in order to optimize all the configuration for the
product. We're not security experts. We don't want to be
security experts and, like, start, like, to spend a lot of time in that. But
nobody else just like to very easily find all these different kind of misconfigurations.
And this is also a threat and, like, attack vector that we
started, like, to see a lot in the field today. I can tell you that,
like, we see tons of attacks around
different misconfigurations in tools like Airflows and Databricks
and stuff like that. And I think this is also like a very, very important,
like, mindset, like, to be in. And in addition to that, of course, we have
all the all the runtime and all the adversarial attacks there.
There are specifically, if I mentioned in the
OSPI exchange, so OSPI exchange covers everything.
The OSPI 10LLM specifically is more
covering this LLM, like,
specific risk. And then you have, like, all the adversarial attacks, like prompt injection
and model jailbreak and model dn out of service, model dn out of wallet,
etcetera. So basically, the mindset should
be we already know security very well. We already have, like, these
principles. Until now, we just haven't
implemented them on the data and AI teams,
tools, and technology. And this is exactly what we start, like, to
what we, like, need, like, to start to do. And this is what we see
also that, like, you know, like, now we have no reason. Like, we all see,
like, these different kind of attacks. So we start to see that all the organizations
were, like, starting to to already, like, walk the walk.
Wow. Yeah. I I often wonder too, like, what you
mentioned the pipelines being a vulnerability or an
attack surface. Right? Like, or a potential vulnerability.
I often wonder now, like, when, you know, we're looking at agentic
AI, right, where these things aren't just LLMs,
right, producing text or going through these materials.
We're giving them, you know, abilities,
right, to influence pipelines, right, to to or to
whatever. Right? Like, that just seems to me like a giant
security risk. I mean, telling someone you know, there's there's multiple ways to
break an LOM. Right? Like, obviously, there's the the the $1 Chevy
Tahoe. Right? Where the guy did that. Right? Pretty low tech
approach, pretty brute force ish.
But I often wonder, like, well, what
what sorts of things are agentic systems gonna open up?
Like, what does that look like? I think that this is exactly like where we
we will start, like, to see, like, the very big LLM,
breaches, that we'll have. I believe that, by the
way, my belief is that the the how does the
attack start will still be, like, in a lot of cases,
very similar to what we see today. But the impact of the
attack will be much, much, much, much, much higher because now like the
model cannot only like, promise you a
$1 a car, but you can throw, like, I already like
send the order, can send the car to you, can like book your
hotel, can do like everything there, can share with you, like, the data
of maybe, like, other customers in the application because it is,
like, a RAG architecture, and it is also, like, different, like, tools
that provide him the ability to maybe even, like, write different codes
to the application. And then it might also like start like different types
of remote code execution. As long as we are going to
provide to these NLMs more privilege, more access,
more tools, more abilities, the impact of the risk
that they will be able, like, to cause will be much higher. I still
believe again that that pack vectors are going to start from more or less,
like, the same areas, like prompt injection and model jailbreak,
but they they eventually, like, the outcome of these attacks will be much
higher. I could see that. Because we're giving them
actuators, so to speak. Right? Like we're not we're we're
giving them agency. Right? Like where they could actually do real damage as
opposed to because one thing in saying you're gonna give somebody
a $1 Chevy Tahoe. It's quite another to actually place the order,
sign off on the invoice, and then ship it. Right? Yep. And what
if you'll do, like I don't know. Like, you'll you'll start, like, to see it
also, like, in banks and in investments. They will start, like, to transfer
your money. They will start, like, to invest, like, to buy stock. They will like,
the the the the amount of, like, potential impact here is, like, a
crazy high. I believe, by the way, that eventually, this is going to be one
of the things that, like, we'll see also, like, slow down the adoption, not
less than the than the technology or, like, finding, like, the
right use case. Yeah. No. I could see
that. I I just think that we're just setting, as an industry.
We're setting ourselves up for a huge exploit that we
haven't figured out is already there yet.
And so so what what
can AI engineers, data scientists,
data engineers do today to make things
better? I know we can't fix it because we don't know what's we really don't
know what's broken. I think that's one of the frustrating and kind of fun things
about security work is, like, it's not that there's no vulnerabilities.
You haven't discovered any vulnerabilities yet. Right? There are no unknown there are
always un there are always unknown unknowns.
But if you have an unknown unknown or a known thing,
you can you can say that you pretty much figured that out. But there's this
whole aspect, which I don't think data scientists
fully appreciate. I think they can understand the concept of the unknown
unknowns. But in terms of the consequences of it, I don't
think I think it's gonna take 1 major solar wind style
issue or CrowdStrike style issue to make people conscious
of of that. But how do
we how do we prepare ourselves? Right? You can't
stop the hurricane, but you can board up your windows. Right? Like, you
know, how do you Yeah. I and I totally
agree that, like, what's going through, like, to to shake every everybody
will be, like, the the first SolarWinds or, like, the 4 log 4
j attack that we see, like, in these areas. I think that,
like, I think that you broke it very well
and that we need to relate to both categories.
1st is, like, the known,
which already, like, exist. Like, we know that, like, you know, like,
we see that as scientists. Like, we are not a scientist.
And we see that one of the the things that, like, we see
in in in our code in compared to software developers
is that we don't give a
tip on, like, everything, around security.
Like, you'll see, like, tons of exposed secrets in plain
text. You'll see tons of, like, test and, like, the sensitive data
just like playing. And, like, it's state, like, exposed, like, in the notebooks.
You'll see that we download, like, any dependencies without, like, like,
even, like, think about it. Even so that, like, yeah, it looks like maybe, like,
a bit suspicious and stuff like that. So it's it's far
from from the basic. Let's make sure that, like, what we know that is not
best practice, just, like, start, like, to implement it. And
then regarding the unknown unknown, so, of
course, like, you don't know how to handle it. I think that, like, as you
as you said, you can start to prepare yourself. How do how do you
prepare yourself in security? It's basically to be very
organized and to to make sure that you have, like, the right visibility and
governance. As long as you have, for example, like, you know how to build,
like, your your AI or the machine learning bomb. You know all the
different, like, models that are built or embedded as part of the application,
and you have, like, the right lineage, which one
was trained on which dataset, etcetera.
Once, for example, that now let's say we'll continue with the
examples of of Hugging Face. Like, a new Hugging Face
model is is is now, like, published as a like, someone,
like, found that it's, like, malicious. You because you prepared
yourself and you have, like, the right visibility, you are able to go
and very easily search exactly, like, if you use it and
where you use it in all your organization. And this is also
because you prepare yourself. This is exactly what happened, like, in Log 4
j. In Log 4 j, it was like a dependency that
found as a critical vulnerable. And a lot of
organization, what they spent, like, most of the time is to try to understand
where they even use this Log4j. And they seem that, like, if you prepare
yourself, you are like, if you are organizing everything, you'll already
be very, very, like, ready for the for the
attack of, like, the unknown unknown. And, of course, everything
in addition to to, you know, like, learning and, like, educating
yourself. If you start, like, to understand, you'll go
to, I don't know, Databricks, for example. A lot of people use Databricks. You'll
go and, like, start, like, to see what are, like, the best practices of how
to, like, configure your Databricks environments and what are, like, the best practices
there. It's something that you can, like, find very easily, like, in the Internet. You
don't need, like, to to do it, like, from scratch.
But I'll say that, like, you you know, like, it's still, like, when we are
aware of that, it's not still, like, the the top of our mind as the
data practitioner to start looking, like, in our free time for this
kind of concept. Right. I mean, that's a good point.
Right? The fundamentals are still fundamental. Right?
You know, making sure, you know, you track what
your dependencies are. Right? So that way, if there's a breach in a hugging face
model, like you said, you'll know right away whether or not it
impacts you or not. Also too, I think you're
right. This isn't top of mind for AI practitioners. Right?
Even when I code, like, an app, my met
my thought process are very different than when I'm in a notebook.
Mhmm. It's just different wiring.
Yep. And by the way, it's kind of like, it's kind of
like a paradox because most times on the notebooks,
we are connected to much more sensitive information than on our
ID. Right. No. Exactly. So
it's kind of it's like the worst, one of the worst case
scenarios. Right? And and you're right. Like, people wanna work with real
data, and they they just assume that if they're on a system that's
secured and internal, they
they, they don't have to worry about such things,
which I think you're right. Like, with these systems that have access to
sensitive data, these pipelines, I mean, it's one of those
things where we need to start thinking about this. And what would you do
you think that there's a, like, a career path for, like, an AI security engineer?
Right? So it's not just a security engineer, like, in a traditional
sense. Right? But also a someone who specializes
in AI related issues. You think that's a growth industries? I
have, like, no doubt that we are going to like to see more. Like, we
already see these kind of practitioners in the field. I have no
doubt that it's going, to be more and more frequent. And in
addition to that, I believe that, like, even in the future, it's it's going to
be even, like, several different, like, roles. For example, one of the
things that, like, a lot of people that we work also, like, very closely with
are AI red teaming. Right. It's not even,
like, just like a AI security engineer, like, general one. Specifically around,
like, credit teaming because all these kinds of adversarial
attacks on models are very different, requires
different techniques, different tactics. And the red teamers are the
ones that, like, to, like, learning all these different
types of adversarial attacks and how to, like, check your model,
in your organization. And by the way, specifically in this
area, I do feel that it's kind of, like, top priority and
like top of mind also for the data science
team. Like you do see that on LLMs,
once they are deployed into production, the data
scientists, they are kind of like understand that there are a lot of risk there
and they are starting, like, to take also, like, responsibility even completely, like, regard
regardless of the security team to make sure that, like, we we
we reduce some of the risk there. Now the risk is not only
security. The first thing is security, like, to try and, like, make sure
that you are secured from all these different adversarial attacks or that you know how
to detect sensitive data leakage, for example, as part of the response and stuff
like that. In addition to that, it's also a lot of time
like safety risks. You want to make sure that once you deploy LLM into
production, your model doesn't give any financial advice to your
customers, doesn't give any health advice in case it's not your business.
So you then have, like, these kinds of responsibility, or example, like in the
Chevy example that you gave, that you just, like, you don't just, release
free cars or flights or books or a tail off, like, anything
like that. So I think that because the
the the the amount of potential risks are
so high on the run time. In this area, I
believe that, like, the data scientists already understood that this is, like,
under their responsibility. They see it also as part of,
like, being a professional data scientist. If I
deploy this model, it has, like, a lot of, like, accuracy, but,
like, it creates all these different kinds of risk.
I would define myself as not a super professional data
scientist, unlike on the supply chain, unlike in the
notebooks that if I code a code that is not secure, I wouldn't say that,
like, it's not professional. I would say that, like, it's okay. You're just, like, focusing
on the business. So I do believe that we start, like, to seeing this shift
also, like, in the mindset of the data scientist because of the risk of
the Gen AI, but now it's also, like, like, a move
to to all the the development and the building practices
that we have. Yeah. And I think data
scientists are acutely aware that LLMs
are just taking they mean, we talk we we call it hallucinating when
they get things wrong. But realistically, they're
always hallucinating to a very real degree. Right? It's just they
happen to be correct. And what these things are doing
under the hood is they are looking for patterns of words.
Sometimes those patterns of words are wrong, obviously wrong.
And sometimes they may give out sensitive information
inadvertently. So I can talk at least at least there's some common sense out there
when they when they do realize these things are higher risk than I think
we've been led to believe. Yeah. Actually, I love this this finish. They are,
like, hallucinating, like, all this time. Sometimes they really find it
as wrong. Like, they do the same thing as always. Right.
Right. The they don't know they're hallucinating because they're just operating normally.
And so when they go in a different direction and I've noticed
that, you know, kinda like a little bit of, you you know, off by a
little bit, and then then then it generates an off by a little bit, off
by a little bit. I ran an experiment with a hallucination, and
I read it through I ran it through a bunch of models and each one
of them didn't do any fact checking, which I mean, realistically,
I wouldn't expect that. Right? In the future, I think that'll be kind of table
stakes. But, you know, it would just go through. So
I took a hallucination, fed it through notebook l m, which then
create even more hallucinations. Right? So it took this little
genesis of something that was wrong and then made it even crazier wrong,
which I think is an interesting kinda statement and and and
also is a risk. Right? Like hallucination on top, compounding
other hallucinations. And I don't think we've really seen that yet because we've
only really seen for the most part, I've only seen one kind
of model in production. But if you have these models that will kinda work together
as agents or, you know, whether they're agents
that do things or agents that it's different LLM discrete LLMs that talk
to one another. They can get things wrong and make things worse. I mean, I
haven't I think it's too soon to tell either way, honestly. Yeah. But, like, the
like, theoretically, like, it makes a lot of sense. I think in general, like, we
don't see, like, a lot like, we hear a lot about Gen AI.
I think that, like, the level of adoption and the amount
of business use cases that, like, businesses
found are not that high yet. I think that, like, the
most of the usage today is done by, like, consumers, like,
like, directly, like, from, from the foundation model providers, like OpenAI and stuff
like that for day to day, like, jobs, like, you know,
like, reviewing mails and stuff like that.
The the big businesses are still trying to find these
different, like, use cases. I do believe that the that the
agents are going, like, to open a lot of different use cases
around it. Right. Right. I could I could see that. And
I think I think it's just too soon to make a statement
either way. But I think grounding yourself in the fundamentals
is probably always a good idea. Mhmm.
And probably a good a good
approach. So so tell me about NOMA. What is is it NOMA? I
I don't wanna make sure I pronounce it. NOMA. Okay. NOMA. Security. What does
NOMA do? Is it security firms that focus on this space? You
mentioned red teaming. Is that is that a sir service you offer?
Yeah. So NOMA basically is an like, our name is Nomo
Security. The domain is Nomo dot security. So it's Oh, okay. Sorry about
that. No. No. We're good. So, so, yeah,
what we do is, like, secure the entire data in the AI life cycle.
Basically means that we truly, like, cover it end to end. Like, we enable, like,
the data teams and the machine learning and the AI teams, to continue and
innovate while we are securing them without
slowing down. And this is like the the like, we are built from, like,
data practitioner, like, the company. So this is, like, our main focus,
meaning that we start, like, from the building phase. So if we
said, like, notebooks and hugging face models and all these different stuff and the
misconfigurations are on all the different stack and all the envelopes
tools and AI platforms and data pipelines and stuff like that. So we are
connected seamlessly on the background, and,
basically assist the the data teams to to work securely,
without changing changing anything in the workflows.
And then also, like, we provide, as you said, the red teaming.
Before you're deploying the model into production, you want to
understand what is the level of, of
robustness and security that the like, that your model has. And
what we do is we had, like, a big research team that,
like, builds, simulated, thousands of different
attacks. And then we dynamically start to run all these attacks against
your models, showing you exactly, like, what kind of, like, tactics
and techniques your model is vulnerable to, and exactly
also how to mitigate and improve it to be more
robust. And then the 3rd part is also the runtime.
We are mapping, we're scanning all the prompts and all the
responses in real time, making sure that you don't
have any risk on both sides. The security, we are detecting all these
different kind of, like, a host and a little, like, adversarial tax prompt
injection, model jailbreak, etcetera. We check also the responses for
sensitive data leakage and stuff like that. But in addition, also the
safety. We see a lot of organizations that the data scientists, as we
said, they understand the risk of deploying
models into production. And this is why not even, like, the security, but more like
the the Chevy example and, like, the the health advice and stuff like that.
So they built for their own, model
guardrails in order to make sure that they are, like, controlling what
are, like, the topics that the model is be able like, is allowed or
disallowed to communicate about. And what we do is basically to save
them also like this time. We also provide them, like, all this
runtime protection already, like, as a service. You can define exactly what kind
of, like, detectors and in native language, what kind of, like, policies you want
to make sure that are enforced. And then we also, like, protect it in the
run time. So, basically, we just, like, cover you, like, end to end, start from
the building and up to the run time. It starts from the classic data engineering
pipelines and machine learning and up to gen AI. Interesting. Interesting.
It sounds like something I think is totally, I think, a
needed needed service and and skill
set. Because you're right. Like, I mean, there's just so many risks
here, and the hype around Gen
AI is so over the top.
It is gonna be revolutionary, but
maybe not in the way you think. Right? And I always call back to the
early days of the dotcom. Right? Where it was pets.com. There was,
you know, this.com, that, you know, like all these crazy things. But the
real quote unquote winner of, you know,
.com was some guy in Seattle selling books.
Mhmm. Right? No one no one I mean, selling books. Like, really?
Like, not, you know, and it's
it's interesting to see how I think I
think that the the obvious use case for chat for for
LLMs thus far has been chatbots. Right? Customer
service type things. I think that's really only the
the the the the the surface of it. I think for me, what
I've seen is most impactful is the ability for natural language
understanding and their ability to understand what's happening in a in
a block of text. And I think
that that has enormous potential. I
agree. A lot of risks too. Right? Because what if, you know, what if
I I mean, to your point. Right? You wanna make sure these things stay on
topic. Right? Like, I don't if I'm talking to a
financial services chatbot and I say, hey, I have
my my leg kinda hurts. Right?
It's, you know, the risk of moving into health care, like, it's just kind
of, I don't how mature are those guardrails? Because I've
not really seen a good implementation of
it yet. Yeah. So, you know, like, I
don't want to to give ourself, like, a compliment, but,
we Oh, you guys are pretty good at it? Yeah. Like, we're pretty good. Like,
we were, like, you know, like, with fortune 5 100, with fortune 1 100.
Not in vain. But, yeah, I believe that in general, specifically, like, when we speak
more, like, on the guardrail side, I see that the most important thing is
to make sure that it's, it's building the
right architecture to be very flexible and easily
configure for the organization because eventually, like, you know, like, each
organization is completely different needs, completely different
context to the calls, like, in their customers, internally to their employees.
So everything should should be, like, very easily configured, but very flexible.
Interesting. Interesting. I wanna I I could talk for
another hour or 2 with you because this is this is a fascinating space.
Where can folks find out more about Noma and you? I you think it's Noma
dot security? Yeah. Noma dot security. Can't believe that's now
a top load pain, but,
and, any any, NOMA dot
security, you're on LinkedIn, and, anything
else you you'd like the folks to find out more?
No. I had, like, a great time speaking with you, Frank. Great.
Likewise. And for the listeners out there, if you're a little bit
scared and a little bit paranoid about generative AI and LLMs,
then I think we had a good conversation. Because I think we need a little
bit of that fear in the back of our heads to guide us and
maybe think about security issues. A
little bit of thought ahead of time will probably save you a lot of problems
later. And want to lose some. That's
that's all I got, and we'll let the nice British AI,
Bailey finish the show. Well, that wraps up another
eye opening episode of data driven. A big thank you to Niamh
Braun for sharing his expertise on the critical intersection of AI,
security, and innovation. If today's conversation didn't make
you double check your data pipelines or rethink your Hugging Face
downloads, well, you're braver than I am. As always,
I'm Bailey, your semi sentient MC, reminding you that while
AI might be clever, it's never too clever for a security breach.
Until next time, stay curious, stay secure, and
stay data driven. Cheerio.