Hello and welcome back to Data Driven, the podcast where we explore the emerging field
of data science, artificial intelligence, and of course, all the hype
around AI. All of it is impossible without data
engineering. However, my favoritest data engineer in the world can't make it
today, but I did bring the most curious person I know,
and that sounds really bad, quantum curious, but she's also data curious
too, Candice Cooley. How's it going, Candice? It's great. I'm very
excited about today. Our guest has a really exciting
background. Yeah, just looking at his, his
LinkedIn profile makes me ask a lot of questions. Our guest today is
Neil Katz, who is Chief Product Officer at Valantor AI
based in New York. In the virtual green room, we geeked out on some New
York stuff, but he's also a 4-time Emmy winner
and he knows that it's a non sequitur. So welcome to the show, Neil.
Thank you very much, guys. Great to be here. Much appreciated. Good to have you.
I have to ask first, the Emmys. Did you work in media? Did you work—
what, how did you get an Emmy? I, yeah, I've, I probably have— 4
times. 4 times, true. I probably have one of the, one of the stranger
backgrounds to be in leadership at an AI company these days, but I actually
started my career in technology. When I came outta college, I built one of the
first digital design companies outta New York City. This is Web
1.0, so kind of dating myself here, but After a couple years of doing
that, we sold that company and I had a soul-searching moment. I just realized I
had spent my early 20s just spending all my time in
dark rooms with computers till 4 in the morning. And I didn't
want to do that anymore, or at least not for my whole life. So I
retooled, became a journalist, and spent 20 years in
journalism, having the great fortune to be able to report in places
like Iran, Vietnam, India for quite a time,
southern Mexico, all over the United States. for places like the New York
Times and CBS News and, and then Weather Channel, where I was running the
digital news division at the Weather Channel. We actually built a digital news
operation from the ground up, which was really a cool opportunity. And
strangely enough, even though I mentioned these nice brands like the New York Times and
CBS News, the 4 Emmys actually come from our time at the Weather
Channel, where we built a documentary
unit and went all around the world, really. Wow.
Reporting on the relationship between climate change and extreme weather
and social issues in a way that you wouldn't expect.
So we reported everywhere from Iraq to
Ethiopia to Sudan to Central and South America,
and of course all across America. And then we're lucky enough to win
some Emmys for documentaries we did. The first one we did
was actually— that we won for— was about the southern border of the United States.
and how many immigrants were dying actually at that
border, actually inside Texas when they crossed over. Things had become
hotter, things had become drier, and the passage had become more dangerous.
It's not how it works today because right now everyone goes to the southern border
and basically says, hi, I'm here and I want asylum. But back then you snuck
in, and in Texas it had become very dangerous, and they would
find a couple bodies, corpses a week, out in kind of the badlands of
southeast Texas. So anyway, starting off on a weird note for an AI
podcast, I realize, but I spent a long time in the journalistic world
and happy to go into that if you want. And then something interesting happened.
IBM came along and purchased the Weather Channel in one of probably the
strangest acquisitions in the era. Yep. We
can say a lot about that if we want to, but then I got thrusted
right back into technology. I'd run from it. So I'm going to do this content
thing and then tell people's stories and got thrust right back
into tech. And IBM really I think to their credit, made us much
more of a technology company than a media company. That was actually probably pretty good
for us. And then I was running big product and AI teams
under Watson, IBM Watson Group, and met my,
my current partner, Ben Fletcher, who's our CTO. And we formed
a company called EyeLevel, which just got acquired by Valontour. We can tell that
story in a minute. Ah, very cool. But even back in,
I actually looked up the date right before this podcast, back in 2016, so about
6 years before the launch of ChatGPT. We were using
the Watson model to build some really popular AI experiences for
consumers. We built something called Watson Weather, and about 2
million people a day were chatting, if you will, a chatbot that was on
Facebook Messenger, of all things, which was the popular place to chat,
that gab back then. And 2 million people a day were chatting with
our AI bot and trying to get their weather or hurricane warnings.
And these models were not nearly as good as the GPT-class models.
But what we saw was really interesting. We were, we understood what you
wanted maybe about 35% of the time.
So pretty high failure rate. I was thinking of the early Siri experiences. You're like,
if this is artificial— At the time it was magical, right? At the time
it was magical. 'Cause I also worked lots in the, the
before thing. I think you're right. I think people are gonna remember the, at least
the early part of the 21st century as there was before
ChatGPT launched and then after ChatGPT launched. You're probably right. The before time. But, but
a lot of people don't realize there was natural
language processing. It's old. I remember being a kid on the Commodore
64 playing Zork, right? And again, that seemed
magical at the time. Really couldn't chat with it per se. Yeah. But
just the idea that you could type something in. And before that there was ELIZA,
which you've probably heard of. But, you know, but
I remember building out chatbots. I remember using chatbots on Facebook Messenger
where it would basically do a lot of linguistic processing. was
not nearly as coherent or, or as
good of responses as you get from ChatGPT. But
yeah, no, people forget. Hopefully none of our listeners, because a lot of ours
are very savvy. But a lot of people, a lot of the normies out there,
just assume there were no chatbots prior to— There was a lot of
hard work that probably won't get any glory because it
just wasn't as cool. But we saw something I think really
we saw a piece of the future when we built this system, which was that
when we got it right, when people were able to communicate effectively with
this chatbot, the engagement stick looks
basically the same as the ChatGPT engagement stick. It just was vertical. Right. So
if we got you, you were hooked. You'd come every single day, you'd talk with
us for many minutes a day trying to get your weather. If we didn't, of
course you were gone because we didn't get it right. But you could see already
what was possible if you could make this work in a more generalized way.
The stickiness of chat cannot be overstated.
You know, yeah, the way you've been talking about it, you, you know, you were
involved before ChatGPT. So my question is, how
has your definition of AI changed over the last 5
years? Oh, that's a good— that's
a really good way of putting it. Look, I think in the before times,
You would build— like, most of what was built was machine learning, if we're honest
with ourselves. Most of the things that you would practically create was more about
basically statistical algorithms to try to find efficiencies or try to find
things that were popular. So much of it was something that people
wouldn't even recognize today, actually, when they say something like AI. These were like internal
tools that you would use to improve the various performances in a company, maybe improve
your supply chain by 5%. But already you saw
the kernels, as I'm saying, of what you could possibly do. I think now obviously
we're in the era of much more generalized models, which to me feels like one
of the big breakthroughs of GPT that was very different from what was happening with
Watson. Watson in its day was pretty badass. I don't know if I can say
that word on your pod. I don't know what your PG rating is, but— We'll
figure it out. All right. You gotta bleep me, let me know. But
you remember this thing won Jeopardy, which was very hard to do a long time
ago, but things were much more specialized. You'd be building models that were
around one very specific business problem. You try to train it
on the specific business problem. And that kind of made sense,
especially from IBM's perspective, because they have business customers that all it's about— it's only
about private data. But it was very limiting. And I think that's what actually
the breakthrough with GPT was, like, wait a second, actually the right way to get
to something that's more useful even for specific tasks is to make the
models more generalized. And obviously now today, people, you
can almost ask it anything And it can do almost anything.
For me, the exciting part though is not even where we are at the moment.
Language models— this is something a lot of people in the public don't understand, your
audience will understand— language models are trained on language.
A lot of people don't understand that they're not math geniuses. We're trying to make
them better at math, but they're actually word geniuses. And
that unlocks a huge amount of human potential that computers never touched before.
Our knowledge really has been— it's been unable to
communicate with computers in that way. But even though that's been
really amazing, I think the next step, which we're not there yet, is getting the
models to do things that are not language, right? So people don't quite understand that
you can't build a rocket ship to go to Mars with a language model. We
don't— that's what the models do today. So we're still like a step away, I
think, from some very different innovations that probably don't look like the language models
today. So that was a long answer on that question, but— All
the math that these models are good at all go—
they're not good at math per se, like you said, but they are good
at math. But all the math goes into linguistics and
text, right? You can ask it— what's interesting is now if you ask it to
say how many Rs are in strawberry or what's 2 2, it'll
actually write code. It's good at writing the code that'll give you the answer,
right? But no, you're right. And it's an interesting—
it's very curious to see how
people misinterpret what AI is doing, right? People say, oh, it's
hallucinating. I was like, from a certain point of view, it's always hallucinating, right? Because
it's just guessing the next logical step. And I know there's
attention mechanisms and all that that do keep it honest. But I
was once at an event, and I hate to name-drop, but
this was Esther Dyson. Oh, yeah. Right? Yeah. Somehow I ended up in the same
room with her. I'll never figure out how, but, but she was, I
was on the same panel and she was saying like, this was 2023, so early
days of what we're, what we have today. Yeah. And she was saying, well, there's
no real difference between autocomplete on your phone or
guesses the next number and, and ChatGPT. And I was
like, far be it for me to tell her she's wrong. Sure. Certainly in a
public forum. But also too, she's right in a sense that the stealth
bomber is the same tech, technology as a paper airplane,
right? They're both doing a lot of the same things. Obviously one's way more
advanced than the other. What fascinates me— I won't explain on that one.
Yeah, I think both are radar invisible. But no, but I'm
actually— I often find myself surprised at how far we've taken
the transformer and LLM architecture. I didn't think— I thought
we would run out of steam after a couple years, right? But
things like things that have no business working, fine-tuning,
That was— for fine-tuning, that's harsh. But for things like
distillation, it shouldn't work as well as it does.
Reasoning on these things also works way better than I would have
bet on. I'm sorry, I cut you off. I'm agreeing with you. I think reasoning
and logic is something you would think maybe would be beyond guessing the next word
or the next token. But you can see
emerging properties in language models that seem to go far beyond what you think guessing
the next word would offer way
more. And I can only imagine that's probably the attention
mechanism. There's some kind of wisdom in the attention mechanism. But that—
I think it's going to sound weird. I almost think we don't
necessarily know. No, that's crazy. Yeah. There's the
new paper that came out from Anthropic, I think last week, that was talking about
that. It wasn't even totally new because this idea of a latent space inside
models has been around for a long time. This idea that there's some kind of
its own thinking space where it's doing things in its own language, in its own
quote-unquote mind, whatever that means for a model. But Anthropic reconfirmed that
and is finding that their models are now— they've been able to figure out where
inside the model this activity is happening. That's interesting because
obviously no model has been built to do that specifically. And so what is so
surprising is just by training them on the world and then
refining them, they have emergent properties that aren't specifically developed.
That's a whole new kind of thing in the history of human invention. There's no—
like when someone invented the fork, it didn't have— there was no way it would
have some emergent properties and suddenly become a spoon and do something else. So
something, something special is going on as these models get bigger,
smarter, better trained, and more refined. Oh, that's a great way to put it. And
it makes you wonder too, what— maybe language is the secret
sauce to— I don't want to say consciousness, but intelligence. Maybe language
is in itself the magic. Emerg— no,
the notion of emergence, emergent properties is not unique to AI,
right? You'll see birds will flock a certain way, right? And
all they're doing is following the bird in front of them, right? And they inadvertently
makes these patterns. And it seems
language is the gateway to some kind of emergent properties of
intelligence and reasoning. It could be. And I'm actually also interested in
languages that aren't our own. If I think about like DNA, a chemical
language. It's one of those areas where I think language models will eventually be pretty
good, as opposed to the pure physics of building a spaceship, which they
will be good at as well, but this is a different problem. But there's lots
of languages that humans are not naturally conversant in. We can't think in
DNA, but a language model perhaps can, and that can— You could find—
that's a great point, right? What sorts of things could this possibly unlock?
And mathematics is also, to your point about building a spaceship, mathematics is also a
is a language in itself. Most humans don't
think it, can't think of it natively, but there are some, right? It might be
the gateway to, to other ways of thinking about this.
Very much so. Yeah, it's— Candace, you kicked off an awesome question, by the way.
I think that you answered that question maybe 10 minutes ago and it kicked off
a great conversation. So thank you. No, it was great. I really, I'd like to
dig a little deeper into the idea of DNA as a chemical language.
So what do you mean by that? And how has thinking
about biology influenced the way you think about AI?
Boy, that's a good deep question. Look, in our practical work and the things we
do for customers every day, I wouldn't say that we think in a biological sense
or that DNA is part of that story, at least the work we deliver today.
But it excites me because when I try to think about what are the unique
things that— where are the places in the universe where language can create something new?
DNA literally is the language of life. English or
Spanish, human languages, are like the language of consciousness.
DNA is the language of life, the fundamental building blocks of creation,
at least on this planet. We don't know what it'll be like on some others.
That's powerful witchcraft. That's powerful stuff to be playing with. And so I—
and things that the human brain just isn't really built to speak that, to
think conceptually in DNA. So to me, that's going to be a very
powerful space when we can really get language models just to be native,
natively speaking the language of creation of life on Earth. That's
super interesting. It's nothing we do. I wish I could say I'm delivering that to
Air France and EDP, some of our customers today. We're not, sorry. But that's some
cool sci-fi where I think we're not that— we're not 20 years from that. Maybe
we're 10 or 5. Baby steps. Yeah. So one of the things that I
see on the Valintor website is
really piqued my interest for a number of reasons. It was, you mentioned
enterprise visual intelligence, which one, that's
very interesting. 2, for sovereign AI
environments. So tell me about what sovereign AI means
to you, because everyone has a slightly different take on it. I've written a book
on it. I have my own take on it. I'll send you a free copy
of my book and you can, you can, you can tell me if I'm full
of it or— Same, same. What? I'll sign it for you. Yeah, I'll
sign you, I'll sign you the PDF. But the, the sovereign AI
went from being this really weird niche topic to
now is the topic of the G7, right? And
obviously there's fallout for that. So what does sovereign AI mean
to you? For me, it means, I think for us as a
company, it means that we believe, look,
fundamentally, I think the world's most important
The world's most important knowledge, in a lot of sense, is not on the
public internet and never will be. So much of
the vital data in the world sits
behind firewalls, right? Sits behind corporate or government or military walls.
And if we want to liberate that with AI, with the
newest technologies, we think that's not going to come from companies or
organizations that have built 20, 30 years of digital security around the golden
egg of knowledge in their company. That's probably not going to be them opening up
the kimono and firing that off across the internet and sending it to ChatGPT
or Anthropic. There's a big— the next wave, I
think, is letting companies, governments,
countries even have sovereignty over their
entire AI stack, from the data that they've been protecting for
decades to getting the models and the
harnesses and the stacks to run right next to the data. So
you have a boundary around your world, not like a
militaristic border, but like a boundary around your space so you can
confidently and safely build AI that sits right next to the
data that you've been working on protecting for the last few decades. I know in
the political space, as you brought that in, obviously the
EU is thinking about this, the Middle Eastern countries are thinking about this. We
often get calls from, from the Middle East on this stuff. I think
they think it more from like a geopolitical perspective of Can we trust
America? Can we— increasingly, companies, countries feel like they can't.
Our relationships are fraught. So if we can't trust America, maybe
we can't trust the products coming out of America, and we need to have control
from the global perspective. We're not really thinking about it from
geopolitics, not what we do. But when we go to a company like
ADP, for example, obviously they have a tremendous amount of
enormously privileged data from their customers, the largest payroll provider in the
country, hundreds of millions of people's financial information is stored in that
company, a lot of it actually down in mainframes in the basement, old
IBM machines that we used to work for, old iron.
That's not going out to ChatGPT. They wanna build AI around that
stuff too, but they're not gonna use some public endpoint to throw that at
Fable, whoever it is. So they need solutions to bring really powerful AI
that can sit next to their best data. And that's really how we think about
it. Because as
organizations— I'm sorry, go ahead. No, go ahead, Candace. I've been hogging the mic because—
As organizations continue to adopt AI, how do they— how do you balance
innovation with the need to keep sensitive data
secure, compliant, and under your own control?
I think when people— when people say— I'm trying to guess what the word innovation
means here for you. I think what you might mean is the latest, greatest model
from the biggest companies. And, but the open source universe
of models is not far behind. It's generally, maybe it's 6 months
behind. There's a lot of debate about how robust the models are. But it's
getting shorter, right? The time span's getting shorter. And I think that's an interesting metric,
right? 'Cause it used to be, ah, they're about a year behind or, or early
in 2022, right? Ah, no one can ever touch OpenAI and where they
are. Look, the, look, the, those companies may also
make business decisions, the OpenAIs and the, the frontier
model companies, as we're calling them, they may eventually move to a model that's not
just SaaS, right? Today they've decided to be pure cloud. That doesn't
mean they have to be pure cloud in the future. But from a, like, how
do you help a company implement today standpoint, what we're really talking about
is helping them run open source models
inside their security perimeter. And our core technology, Ground X,
is an enterprise-grade RAG and document understanding system.
And the way that we've built it allows you to bring in all the documents
of your company, all the knowledge of your company. And we've done
some really special things on the first principles so that you don't even need a
frontier model to get the best performance out of the information
inside your company. We've actually built our own, trained our
own vision models to take apart your documents, break
them down into small pieces. Here's a table, here's a diagram, here's a text
block. You break things down into atoms, then you could feed those pieces,
small pieces of data one at a time into models and get the same kind
of performance you would get as if you were trying to work to a frontier
model. So we make it possible for companies to use smaller, faster, cheaper
models on-prem or in managed cloud
so they get all the benefits of the frontier on their private data.
Yeah, that's how we're approaching it today. Interesting. So do you— are you talking about
a chunking strategy? For RAG, or are you talking about
fine-tuning or all the above? It starts with a chunking
strategy. It's funny, our product really comes out of— we're talking now in a
philosophical way about AI, but our product really comes from really practical problems.
When we were at IBM, we'd see this problem. The same thing
would happen over and over again on big projects where you were
trying to get a company's information to work with a model. At that time,
Watson was the model. Now it's other models. But the problem is always the same.
What is a company's knowledge? Essentially, it's typically stored in documents,
and documents are visually complex. And fundamentally, these
documents confuse language models because they're
either really long, or they're very visually confusing,
or they're very dense in certain parts. The frontier models are getting
better at understanding these things, but when we started, it was really bad.
So our first problem was like, Company
X hands you 100,000 pages of
stuff. They say, this is what I know. This is what my department— this is
what my legal department knows. This is what my HR department knows. Okay, how do
you make that work with AI? At the time, you really
couldn't. And what we did was we
trained a vision model. We actually trained a video model because the document
models weren't good enough. The Detectron
2 is where we eventually landed. It's our most recent one we're working on from
Meta. It's a really great video model. So the purpose of that video models designed
to follow a soccer ball around the— around a video screen or
security footage and things like that, but they're really good at detecting
objects. We trained that vision model on a million pages of
corporate data, all kinds of stuff— legal, medical, financial, what have you.
And the purpose of that was because we realized the
problem of a document or a million documents was too big for
models to deal with. And this is still true because no matter what they
say the context window is, it's not really true. And so you— it
became obvious to us that to make models work with your documents and your
data, you had to break them into really small pieces. And so we did that
with a vision model that we trained to basically identify on every
single page where's the table, where's the
text block, and where's the graphic. And then we have an
agentic pipeline that essentially feeds those small
objects into a visual language model. Again, one that can be
smaller, cheaper to run because we're giving it small bits to work on,
and have it explain in great detail what those objects are
into text that's friendly to language models, or text that's
friendly to essentially search is the other aspect of this. Because when
you build these systems— I'm in the weeds now, and you tell me if you
want me to come back out to 30,000 feet, the engineering weeds a
bit— these systems are really— RAG
systems are document understanding is the first problem and
search is the second problem. Yes. The document
understanding problem, everyone thinks it's solved. It's actually not. It's still pretty hard.
And if you get it wrong,
the downstream consequences of it are extremely severe.
It's not so much that you have incorrect
information in your database. That's a problem. You ingested 100,000 documents or a
million documents. And let's imagine you're using a system that's not ours and you've got
15, 20% of that's incorrect when it gets turned into LLM-ready
data. Okay? Okay. That's a problem.
But the real problem is it's a silent failure.
So you'd never know which chunks are right and which chunks
are wrong. So now you've got this burning problem sitting in
your database, in the, in your AI representation of your data, which you've now
vectorized, you've turned it into numbers, you've done these things with it. But underlying, it's
not the same as the actual information that came out of your company. And you
don't know where it is. Kind of smolders just below the surface,
and you never know when it's going to cause trouble. I'm sorry, I cut you
off, but— No, that's right. And it— how does that represent to folks? It represents
in a funny way, because at first blush, it looks like we don't have a
problem at all. You produce an MVP, you're like, oh, it looks pretty
good. Most of these answers are right. Looks pretty good. Then the SMEs get
in there and really play, and they're like, wait a second, this is in— this
is In an intermittent way, which every engineer knows is the worst problem you can
have. You don't want an intermittent bug. In an intermittent way
that's nearly impossible to figure out why or track, this thing is wrong.
Now everyone thinks these are model hallucinations. We have
found that 90 to 95% of the errors in RAG-based systems are
not model hallucinations. They're incorrect—
it's like an information gap. It's a comprehension gap.
It's that the things that you have bad information— it's garbage in, garbage out. The
information that sits in your RAG database is not an
accurate representation of the world, of the documents you put in the first place.
Right. Now someone asks a question or an agent asks a question of your RAG
system, you've done a search. We can get into how you would do search and
why that's complicated. You basically search your RAG database for the 20 to
100 blocks of text most likely to contain the answer to this question.
And then the model synthesizes those together and produces a final answer.
This is RAG. Most of the errors are not because the
model just made some stuff up, because the whole point of RAG is that you
can only answer based on the content I've just given you. It's typically the
answer— the content I've given you is wrong, and it's hard to
figure out why or where. So I went down a rabbit hole here, but this
is the core of our technology, the problem we've been trying to solve for a
number of years, this is the problem we started solving at IBM. And
frankly, IBM wouldn't let us solve it, and so we left. From
IBM's perspective, it was a research
problem that was too middle
space. I couldn't commercialize it 6 years ago at IBM,
so IBM said, I can't make money today. And it wasn't a
$100 billion bet like quantum computing that they hope will make them into the next,
you know, bigger than SpaceX. It was this weird middle research problem.
And so we left it high level to build that. And that's the core of
the Ground X tool, that and really powerful search, which I can get into if
you want me to get into that. But that's the ultimate beginning data layer
of how you build knowledge, an AI empowered knowledge system inside a company,
whether you're using that for customer support or agent orchestration
or finding fraud in insurance claims, which a lot of our customers do.
You got to start with that, the engineering right to turn a company's knowledge
So you don't have a comprehension gap into something AI can understand.
No. And it's funny you mentioned IBM. Sorry, Candace, I cut you off.
But one of— I used to work at Red Hat up until a few days
ago. So a few days ago, literally. And so I'm quite familiar with that.
So, and then one of— I'm sorry, what? We're built on actually the first
Kubernetes, the first on-prem implementation we did was built on OpenShift. Oh, very cool. I
worked on OpenShift AIs. It was But
what's fascinating about this is that I, my last project was
a RAG, kind of like how to teach the field sales, like what RAG is,
right? Because there was this product called InstructLab. InstructLab said, hey, just
fine-tune your models. You'll be fine. Don't worry about RAG.
Didn't really work out that way. But so it was like, no, let's reintroduce
RAG into the conversation. So ultimately, kind of like the lab I
worked on, so I, I had the opportunity to, to deep dive into the space
for a couple months starting in January. Yeah. So it's a spot near and dear
to my heart because RAG RAG solutions are really going to be
the shortest path to enterprise ROI. I still
think so, obviously partially because we sell a product that does it, but
we had to— we could choose to pivot into something else, but we— and every
few months someone comes out and says RAG is dead. Why do they say that?
They say that because the models are bigger and better, but we've
always had this point of view that if you really think macro,
Okay. The world's— it would be—
it's very difficult to move all of the world's information,
which is currently stored on the cheapest medium possible, hard drives and flash
drives, move that into GPU, the world's most expensive
silicon. So unless there's a huge architecture
change on the hardware side, and I think eventually there probably will be,
but for where we are today, you're not going to move everything
from a hard drive onto a GPU. And that means by
definition, you need something in the middle that can figure out what to
feed the model when, depending on what kind of— what an agent wants
or what a human wants to do next. And it's like RAG, I think, is
that middle layer that's gonna be with us for quite, quite some time. Yeah, I
think at most it won't die. It might evolve. I know there's RaFT and then
there's something else. PEFT, I think, is the other one. No, RaFT
is the other one. RAG-assisted Retrieval augmented fine-tuning, right? But
I think the RAG pattern, whether or not it'll still be called RAG, is— you're
right. I think it's fundamental. I think it's a fundamental force of the AI
universe personally. But— Yeah, we do too. And you
mentioned fine-tuning models. That was when this whole craze started. I think it would go
into big enterprises and all of them were like,
yeah, RAG sounds interesting, but also boring to me. I want to go fine— I
want to fine-tune my own model. I want to do it because that looks like
a lot of fun, frankly. Then you find out, and it's
cool, at the other side of the engineering project inside a company, you're like, hey,
we have our own model inside the company, and you're a hero. And that's awesome.
Resume-driven development is probably what's behind it. Maybe.
But when you get in there and you're like, wait a second, how do you
actually fine-tune a model? You have to fundamentally— you need
question-answer pairs. You have to distill knowledge out of, again,
those millions of pages of documents we started with. You still have the same problem
in the beginning. Now you're going to try to distill that into
basically question-answer pairs that you can then go fine-tune a model with.
And I think everyone found out pretty quickly that is dirty, messy, difficult
business. And companies' knowledge is really
chaotic. And trying to distill out the perfect
tens of thousands of QA pairs out of that was actually pretty difficult and
time-consuming and out of date. As soon as new information happens, your
fine-tunes are out of date. So that being said, now
we're doing something that's a little bit in the middle. One of the things our
technology is really good at is extracting information from documents. And the
pattern that most of our customers did in the beginning was pure RAG. Okay.
And GroundX is an end-to-end RAG platform that
does the ingest, the processing, the parsing, the storage,
the search, the re-ranking, all of that. And you can set it up in 3
lines of code. Wow. So that's cool, you know, problem solved.
However, increasingly we're seeing is a different kind of pattern
where a lot of the information, a lot of the things people
ask are not actually good for RAG.
People would— and customers don't often understand this, right? So
people ask questions that typically merge structured
information and unstructured information and sometimes graphed
information, relationship between things. Okay, so let me give you
a— let me give you a perfect example from some of our customers in the
insurance space of a question that RAG is natively bad at. Okay,
we do a lot of work, we have a lot of customers, we have another
product called FraudX, and FraudX helps insurance carriers
find the potential evidence of fraud in their claim files
about 40 times faster than human review. Interesting. When you have
an accident, let's say an accident in a building or on a construction site, How
do you actually— what does fraud mean? Often someone's faked an accident,
or they've had a real accident and they've inflated their injuries so they can have
a much bigger settlement. Okay, the evidence for
that is sitting inside the documents. It's sitting inside 5,000 to
10,000 pages of documents. A human would take about 50
hours to read these 5,000 to 10,000 pages. Yeah, a well-trained
human will take about 50 hours to go try to find the inconsistencies in the
stories. I'm going to get to the point. A lot of that is
contained in the medical history, a medical chronology. Okay,
so this person had an accident, 2 years later they had back surgery.
Does the accident relate to the back surgery 2 years later? You could discover that
in the medical chronology of every single thing that happened to them medically over 2
years. An investigator might want to say something like,
how many times did this person go to physical therapy Before they had back
surgery. Because if you rush to back surgery,
if you haven't gotten all the physical therapy, maybe you're trying to get a settlement,
or maybe the doctors you're working with are trying to push you too fast.
Okay. RAG cannot answer that question. Why?
Because RAG can't count. So it's a judgment call, right?
I mean— Not that. Not that.
They're actually asking a very concrete question. They're asking a structured data question.
How many physical therapy appointments did this person have before they had
surgery? If you ask that question in SQL,
you could answer it instantly for a millionth of a penny.
Easiest thing in the world to do in an Excel spreadsheet. Or select
star from wherever. Yeah, easiest thing to do.
Weirdly, a RAG system cannot answer this question well.
Why? Because RAG is using unstructured data. It's
unstructured search is what RAG really is. Fundamentally what RAG is, is you've
taken all this doc, all these unstructured documents, you've stored them.
Someone asks a question, you then search that data
to try to find the 20 to 100 blocks of text most likely to answer
the question. Semantically most likely, right? That would— that's sure.
Okay. We use a mixture of techniques, but let's imagine it's just purely semantic for
the moment. What if you've had— let's
imagine you get that, you nail that search, but someone has had 200
physical therapy appointments. By definition, you're going to miss them.
The other problem is RAG doesn't actually know how many things, how many events
exist, because it doesn't— by its nature, it's searching, it's
filtering out information, searching and giving you a piece of it. So let me
get to the— sorry, let me get to the point. Okay, we're going to— Candace,
I see you're floating. I'm going to, I'm going to take you home. We'll get
it when we get there. Okay. We're gonna get right back to DNA in a
second, I promise you.
What we see now is that
customers don't understand the difference between— even you guys were a little like, wait a
second, there's a difference between a structured and unstructured question? You guys do this all
day long. They don't understand very upfront that
you might be asking a structured question— show me how many times a thing happened—
You might be asking an unstructured question like a judgment call,
Candace. Explain what happened in the ER visit.
That's a good unstructured question. You might be
asking a relationship question. How often did this doctor and lawyer work
together on a case? And so what we really need are
systems that when you ingest your documents, you can
extract the information and interpret it in multiple ways so that you could
store it in structured databases, unstructured
databases, which mostly is vector, and even graph
databases. And then when someone asks a question, they don't have to care
that whether it's structured, unstructured, or relationship-based like a graph.
So that's where we're going on the tactical level. We're seeing companies really need—
it's going beyond RAG, but it starts with that
document comprehension. If you don't get the documents right, if you can't
extract it, all is lost downstream. All right, let's talk about DNA. Let's get off
this topic, right? Let's get out of here. Let's talk about using language models to
talk to whales or something like that. Let me ask you though, so then
is knowledge extraction becoming the new competitive advantage?
We think it's 2 things, and this is where the conversations start. It's knowledge extraction.
It's, it's, can you accurately represent, we call it the comprehension gap. Can you
accurately represent the information inside this company in some way that AI can do
something with? And can you keep the information
safe, which gets into this sovereign AI? And can
you keep it inside the bound— whatever boundary a company is comfortable with? Are
they comfortable with our SOC 2 cloud? Smaller
companies are. Are they comfortable with a boundary that lives inside their
managed cloud on AWS? Some information is
okay there. Do they need the boundary to be in the basement? Do they need
to be in a data center they control machines? Do they need to be air-gapped?
The military needs things to be air-gapped. So we're built to run in all of
these environments, wherever the boundary is. You can natively install us right
next to the mainframes if you want to and still get that data comprehension
to work. So would better
documentation, better document
understanding be more valuable than simply deploying another
AI agent? The AI agents haven't—
what's an agent? An agent is just a, it's a bunch of prompts. To me,
an agent is like the future of the if-then statement. Okay. A very
sophisticated if-then statement, really. An agent is a bunch
of prompts that, you know, use a model for judgment to do a thing,
whatever that function is. But underneath— but you have to have the data underneath for
it to know what it's talking about. Ultimately, a company's knowledge is probably going to
be stored in documents and conversations, and that has to be turned into something that
agents can understand, or else
you've got a bunch of nothing at the end of it. You've got a bunch
of agents running around in circles talking to themselves.
That makes sense. That's an interesting way to put it. Think of it.
What, what do you think is next for
kind of the rag space, right? I think you and I are both in agreement
that it's here to stay. It's not going to die. There'll be many of clickbait,
ragebait articles saying that. But,
and the other thing too, actually, is you said there's multiple approaches,
right? Obviously semantic is the most obvious, but What are the other ones
that you find most effective? Yeah,
mechanically under the hood, search is a hard
problem, actually. And it's really core to making RAG work. It's obviously
fundamentally 2 things. Can I get the document— the documents right?
And when someone asks a question, can I find the right pieces inside the RAG
database? We come from a funny place on this, actually.
Because we started doing this before GPT launched. Once GPT
launched, let me get a little tinfoil hat for a second. That's
going to make it juicy for you. Okay, maybe a little tinfoil, make it juicy.
I'm not a conspiracy guy, I'm just going to give you a little bit. Okay,
juice it up for you. All right, so when—
a lot of people don't know this— OpenAI originally had a RAG
product. Okay, and
the RAG product actually was— No, that's right. I'm sorry, I cut you off.
I had a moment of realization. We were actually one of the first, first 20
testers of OpenAI. We were like one of the early guys who were given access
to the GPT models back in the day. And they had— they weren't really
commercializing anything just yet, but they had a RAG service. And
what a lot of people don't realize is it was not a vectorized service. They
were not vectorizing the data yet. They were actually— their first pass
at this, which is a bit more similar to what we do, was just storing
it as text. And then doing various kinds of text search
against the text to try to find the chunks that would eventually go
into the language model. Now, here's the tinfoil hat part. Okay,
when they launched ChatGPT publicly, they
removed that RAG service. They killed all the
web pages, so you could never even see that they had one unless you use
the Wayback Machine. We've got some emails that we're talking about it with them way
back, and then they told everybody, you know what? The better way to do
search actually is for you to vectorize all your data and do
vector search, not text search. And what do they wind up selling
the next day? An embedding service to vectorize your data.
So embedding models, that's the— and I'm not saying they're a tinfoil hat. They're an
awesome company. They have great products. We use them all the time. They have plenty
of detractors without us joining the— I'm not in the hater group at all.
I'm making it fun and funny, but We always
thought actually vectorization and vector
search, which you call— most people call similarity search,
is one of several techniques you need to blend together to
get to the answer when you do search. Why
vector search? When you say similarity, it's basically trying to
find things that are related to each other, that's more similar to each other, to
the question, right? Semantically similar to the question. Here's
the problem. When you have a lot of data,
you have too many things that are similar to each other. It's— if
you've got a jar of red marbles and you've got 10
red marbles, all different shades, and you say, find me the medium
red marble in a jar of 10 marbles, you can do it.
If you have 100 million marbles, all different shades of
red, find me medium red marble. It's really hard.
And so what happens in systems that are heavily vector-focused or
100% vector-focused, which is a lot of systems out there today,
the more information you add, the worse your RAG database performs.
We've done a lot of testing on this. Most vector systems will
start to fail or decline in performance in as few as 10,000 pages of
information. Wow. Remember, our world, we
do a lot of scale. But we have customers that have— an insurance customer
might have 20, 30 million pages of information
across their thousands of insurance claims.
So I can't go to a customer and say, after 10,000 pages, look out,
things are not gonna be so good. A lot of projects don't realize that's a
problem because a lot of things haven't scaled yet. A lot of projects are still
working on small knowledge bases. Yeah, they're all POCs. Yeah, there's
a lot of that. We're getting past that now. We're going beyond that now, but
still a lot of things are living in the small database world. When you scale,
you start to get problems. So our approach has always been really fundamentally
different and maybe a bit more similar to OpenAI's original idea.
We actually begin with the first search is a
bigram text search, taking advantage of 20 years
of good text search that's been like solid technology around a long time. We
use that to downsample to 1,000 potential retrievals out of the
system. Then We've, we have a
heavily tuned model that we designed that real-time
vectorizes those 1,000 results, then does the similarity,
and then does a re-rank at the same time. And then you get down to
the 20 to 100, let's say, much more likely
answers to your question. At the same time, we're
also doing something different with the data. When we
ingest your data we don't just represent it precisely as it
exists on the page. We're creating a lot of
metadata around every chunk so that
the chunks themselves are more differentiated than
any other system, so that when you do similarity, you wind up
with things that are more different than each other. So you don't run into that
marble problem we just discussed. So it's a more powerful way to do vector
search, more differentiated on the data. It's merged with other
techniques, and it's also merged with what we call like a
micrograph inside of it. So we're— every time we ingest documents, we're
creating keywords around all the chunks in there, and then we graph those.
That allows us to graph the chunks together so we can see relationships between things.
So basically, it's a 3— the marketing version would be like
3-way hybrid search, but now I just told you how you actually do it.
And well, yeah, that makes sense, right? Each one of these Each one of these
is gonna have their own weaknesses, right? So hopefully it's like the power system where
it's a 3-phase power, right? You know, when they all kind of, if you
do enough different approaches, they, where one is strong, one's gonna be weak
and vice versa. Yeah, I think that's part of it for sure. Yeah.
So we've always come at it from just like a first principles perspective, which is
sometimes good and sometimes bad cuz it's, you're, you're going against the grain of
maybe what everyone thinks is the right way to do it. But to your point,
what people think is the right thing— way to do something is
there's— everyone has an agenda, right? That's interesting. For sure.
Even OpenAI itself was a controversial bet. Now that the bet paid off, people don't
realize it was a bet, right? There was a lot of controversy a decade
ago of would this transformer architecture work
well enough? If I just add more data and more money and more training, does
this become something really interesting or not? Or do I need a different technique?
And OpenAI made this enormous financial bet that, nah, just keep working on it, keep
putting more cash, more training, and more data into this particular
model architecture. It will be something quite extraordinary.
So did that bet ever pay off? We don't know if it'll
pay off yet, right? They haven't— Oh, pay is the operative word. But I think
you're right. And I think, like we said earlier, like the transformer
architecture, we've gotten more mileage out of it than I originally thought.
And out of— than a lot of the original creators of it thought. I mean,
there's a reason why— I'm like, I'm not inside Google, but I assume there's a
reason why Google created— literally created it and was like, interesting,
but am I gonna throw $10 million at— $10 billion at that? Maybe not that
interesting just yet. Someone else proved it.
Sorry, Candace, I was hogging the mic because I was geeking out pretty hard.
No, I'm totally geeking out. I'm totally fascinated. And I'm wondering
if you think RAG is a long-term architectural
pattern, or do you think it'll eventually
evolve into something more sophisticated as enterprise AI
matures? In a way, my answer is—
you're gonna hate this answer— I was a journalist for 20 years, and I hate
when someone says both. I was like, pick a lane, man.
And here's why I say both. It would be silly to say that what we
have today is what we'll have tomorrow. Obviously, that's just not true. There'll be more
interesting patterns and techniques to do these things over time. And
we might be 10 years out, but the chipset will evolve. So maybe it can
take on— maybe it can hold more data inside of the
model than we can do today. But at the same time, if I think about
the history of computing, so much never changes. We're still—
take a really simple example, edge versus cloud. This
conversation has been happening since the beginning of computing, right? There was the
mainframe that was cloud, if you will. There was the
compute was in the center, and then the mini computer moved it to a
little bit of a rim around that, but it still was in the center. Then
PC revolution, massive. Everyone's forget it, compute's on the edge.
We have all this power on our desk. And then that happened
for 20 years. Then I was like, wait a second, maybe we should go back
to the cloud. And then the cloud came along, we start to put the compute
in the center. Then mobile came along, and now we've got these powerful supercomputers in
our phone. It's like, now it's at the edge. Now it's going to— same thing.
So just a lot of metaphors in computing just seem to be forever. And to
me, I think there's something in RAG that's pretty fundamental, which is
that the big fence, fancy, expensive thing in the middle
will probably never have all the information it needs to. And it's always going
to need something that has a larger database of knowledge than it
and filter it in real-time ways so that it can answer a question or perform
a function. And if you just think about what RAG is really fundamentally, it's that.
It's that the universe for information is really vast. It's out here.
It's hard for me to store all that and use it in a functional way
in real time inside whatever thing I'm building. And so you need something in the
middle that helps you search and sort and filter and deliver it in the right—
in real-time way. Sorry, that was not a straight-line
answer. That was like both sides of the mouth on that one, but I think
that's the truth. That makes a lot of sense.
We are— I would love to talk to you another hour or 2, but I
think we're at the end of time and I'm going to be respectful of your
time. So where can folks find out more about you and your
company? Yeah, so the company is Valantor
and you can get us at valantor.com.
V-A-L-A-N-T-O-R. Luckily it's spelled like it
sounds. I just spent a week and 2 weeks in Croatia and
I don't know what they're doing with the alphabet over there, but I'm trying to
make sense of it. It's an awesome country. I got nothing bad to say, but
they need some vowels. Okay. But Valontor is easy to
pronounce. So valontor.com, check us out. And for
me, look, if you want to see the documentaries, you could search— would you search
weather films or Weather Channel documentaries on YouTube and see some of the cool stuff
from my past? But check us out at valontor.com. And if
you're an enterprise that has the kind of problems we're talking about, large
amounts of data, scale, security, accuracy is what you need, say hi.
Cool. Excellent. That sounds great. That sounds great. Thank you so much for your time
today. It was fabulous. Thanks a lot, guys. I really appreciate it. And
next time we'll get back on the DNA, Candace. I'm sorry, it felt like we,
we ran outta time before we can get back there. It's okay. We'll have you,
we'll have you back and then we'll— Have you back. Yeah. That's
it. Thanks a lot. I'll let the outro music play.