1
00:00:00,096 --> 00:00:03,926
Hello and welcome back to Data Driven, the podcast where we explore the emerging field

2
00:00:04,102 --> 00:00:07,804
of data science, artificial intelligence, and of course, all the hype

3
00:00:07,804 --> 00:00:11,570
around AI. All of it is impossible without data

4
00:00:11,570 --> 00:00:15,240
engineering. However, my favoritest data engineer in the world can't make it

5
00:00:15,240 --> 00:00:18,718
today, but I did bring the most curious person I know,

6
00:00:18,846 --> 00:00:22,660
and that sounds really bad, quantum curious, but she's also data curious

7
00:00:22,708 --> 00:00:26,202
too, Candice Cooley. How's it going, Candice? It's great. I'm very

8
00:00:26,282 --> 00:00:29,728
excited about today. Our guest has a really exciting

9
00:00:29,840 --> 00:00:33,601
background. Yeah, just looking at his, his

10
00:00:33,714 --> 00:00:37,403
LinkedIn profile makes me ask a lot of questions. Our guest today is

11
00:00:37,435 --> 00:00:40,788
Neil Katz, who is Chief Product Officer at Valantor AI

12
00:00:41,323 --> 00:00:44,474
based in New York. In the virtual green room, we geeked out on some New

13
00:00:44,571 --> 00:00:47,850
York stuff, but he's also a 4-time Emmy winner

14
00:00:48,437 --> 00:00:51,404
and he knows that it's a non sequitur. So welcome to the show, Neil.

15
00:00:53,148 --> 00:00:56,974
Thank you very much, guys. Great to be here. Much appreciated. Good to have you.

16
00:00:57,119 --> 00:01:00,828
I have to ask first, the Emmys. Did you work in media? Did you work—

17
00:01:00,828 --> 00:01:04,232
what, how did you get an Emmy? I, yeah, I've, I probably have— 4

18
00:01:04,280 --> 00:01:07,925
times. 4 times, true. I probably have one of the, one of the stranger

19
00:01:08,134 --> 00:01:11,907
backgrounds to be in leadership at an AI company these days, but I actually

20
00:01:12,148 --> 00:01:15,600
started my career in technology. When I came outta college, I built one of the

21
00:01:15,600 --> 00:01:19,437
first digital design companies outta New York City. This is Web

22
00:01:19,437 --> 00:01:23,275
1.0, so kind of dating myself here, but After a couple years of doing

23
00:01:23,307 --> 00:01:27,050
that, we sold that company and I had a soul-searching moment. I just realized I

24
00:01:27,066 --> 00:01:30,824
had spent my early 20s just spending all my time in

25
00:01:30,920 --> 00:01:34,694
dark rooms with computers till 4 in the morning. And I didn't

26
00:01:34,694 --> 00:01:37,521
want to do that anymore, or at least not for my whole life. So I

27
00:01:37,521 --> 00:01:41,022
retooled, became a journalist, and spent 20 years in

28
00:01:41,119 --> 00:01:44,877
journalism, having the great fortune to be able to report in places

29
00:01:44,893 --> 00:01:48,266
like Iran, Vietnam, India for quite a time,

30
00:01:48,539 --> 00:01:52,177
southern Mexico, all over the United States. for places like the New York

31
00:01:52,177 --> 00:01:55,968
Times and CBS News and, and then Weather Channel, where I was running the

32
00:01:55,968 --> 00:01:59,455
digital news division at the Weather Channel. We actually built a digital news

33
00:01:59,551 --> 00:02:03,153
operation from the ground up, which was really a cool opportunity. And

34
00:02:03,603 --> 00:02:07,383
strangely enough, even though I mentioned these nice brands like the New York Times and

35
00:02:07,383 --> 00:02:10,796
CBS News, the 4 Emmys actually come from our time at the Weather

36
00:02:10,796 --> 00:02:13,566
Channel, where we built a documentary

37
00:02:13,631 --> 00:02:17,098
unit and went all around the world, really. Wow.

38
00:02:17,488 --> 00:02:21,111
Reporting on the relationship between climate change and extreme weather

39
00:02:21,303 --> 00:02:24,558
and social issues in a way that you wouldn't expect.

40
00:02:25,311 --> 00:02:28,613
So we reported everywhere from Iraq to

41
00:02:28,613 --> 00:02:31,932
Ethiopia to Sudan to Central and South America,

42
00:02:31,996 --> 00:02:35,795
and of course all across America. And then we're lucky enough to win

43
00:02:35,827 --> 00:02:39,098
some Emmys for documentaries we did. The first one we did

44
00:02:39,659 --> 00:02:42,849
was actually— that we won for— was about the southern border of the United States.

45
00:02:43,682 --> 00:02:47,337
and how many immigrants were dying actually at that

46
00:02:47,369 --> 00:02:50,896
border, actually inside Texas when they crossed over. Things had become

47
00:02:50,896 --> 00:02:54,599
hotter, things had become drier, and the passage had become more dangerous.

48
00:02:54,663 --> 00:02:58,030
It's not how it works today because right now everyone goes to the southern border

49
00:02:58,030 --> 00:03:01,156
and basically says, hi, I'm here and I want asylum. But back then you snuck

50
00:03:01,156 --> 00:03:04,923
in, and in Texas it had become very dangerous, and they would

51
00:03:04,923 --> 00:03:08,369
find a couple bodies, corpses a week, out in kind of the badlands of

52
00:03:08,369 --> 00:03:11,817
southeast Texas. So anyway, starting off on a weird note for an AI

53
00:03:11,817 --> 00:03:15,602
podcast, I realize, but I spent a long time in the journalistic world

54
00:03:15,602 --> 00:03:19,291
and happy to go into that if you want. And then something interesting happened.

55
00:03:19,500 --> 00:03:23,285
IBM came along and purchased the Weather Channel in one of probably the

56
00:03:23,285 --> 00:03:26,717
strangest acquisitions in the era. Yep. We

57
00:03:26,717 --> 00:03:30,181
can say a lot about that if we want to, but then I got thrusted

58
00:03:30,181 --> 00:03:33,213
right back into technology. I'd run from it. So I'm going to do this content

59
00:03:33,213 --> 00:03:36,998
thing and then tell people's stories and got thrust right back

60
00:03:36,998 --> 00:03:40,749
into tech. And IBM really I think to their credit, made us much

61
00:03:40,749 --> 00:03:44,435
more of a technology company than a media company. That was actually probably pretty good

62
00:03:44,451 --> 00:03:48,025
for us. And then I was running big product and AI teams

63
00:03:48,458 --> 00:03:52,240
under Watson, IBM Watson Group, and met my,

64
00:03:52,320 --> 00:03:56,167
my current partner, Ben Fletcher, who's our CTO. And we formed

65
00:03:56,167 --> 00:03:59,692
a company called EyeLevel, which just got acquired by Valontour. We can tell that

66
00:03:59,692 --> 00:04:03,090
story in a minute. Ah, very cool. But even back in,

67
00:04:03,442 --> 00:04:07,112
I actually looked up the date right before this podcast, back in 2016, so about

68
00:04:07,112 --> 00:04:10,929
6 years before the launch of ChatGPT. We were using

69
00:04:10,929 --> 00:04:14,455
the Watson model to build some really popular AI experiences for

70
00:04:14,455 --> 00:04:18,042
consumers. We built something called Watson Weather, and about 2

71
00:04:18,042 --> 00:04:21,455
million people a day were chatting, if you will, a chatbot that was on

72
00:04:21,551 --> 00:04:25,345
Facebook Messenger, of all things, which was the popular place to chat,

73
00:04:25,426 --> 00:04:28,872
that gab back then. And 2 million people a day were chatting with

74
00:04:28,953 --> 00:04:32,791
our AI bot and trying to get their weather or hurricane warnings.

75
00:04:32,840 --> 00:04:36,316
And these models were not nearly as good as the GPT-class models.

76
00:04:37,110 --> 00:04:40,930
But what we saw was really interesting. We were, we understood what you

77
00:04:40,930 --> 00:04:44,740
wanted maybe about 35% of the time.

78
00:04:45,350 --> 00:04:48,510
So pretty high failure rate. I was thinking of the early Siri experiences. You're like,

79
00:04:48,620 --> 00:04:52,350
if this is artificial— At the time it was magical, right? At the time

80
00:04:52,350 --> 00:04:55,670
it was magical. 'Cause I also worked lots in the, the

81
00:04:55,670 --> 00:04:59,450
before thing. I think you're right. I think people are gonna remember the, at least

82
00:04:59,450 --> 00:05:02,690
the early part of the 21st century as there was before

83
00:05:02,690 --> 00:05:06,510
ChatGPT launched and then after ChatGPT launched. You're probably right. The before time. But, but

84
00:05:06,510 --> 00:05:10,223
a lot of people don't realize there was natural

85
00:05:10,239 --> 00:05:13,702
language processing. It's old. I remember being a kid on the Commodore

86
00:05:13,702 --> 00:05:17,212
64 playing Zork, right? And again, that seemed

87
00:05:17,212 --> 00:05:20,883
magical at the time. Really couldn't chat with it per se. Yeah. But

88
00:05:21,139 --> 00:05:24,698
just the idea that you could type something in. And before that there was ELIZA,

89
00:05:24,746 --> 00:05:28,449
which you've probably heard of. But, you know, but

90
00:05:28,513 --> 00:05:32,344
I remember building out chatbots. I remember using chatbots on Facebook Messenger

91
00:05:32,360 --> 00:05:35,774
where it would basically do a lot of linguistic processing. was

92
00:05:35,919 --> 00:05:39,483
not nearly as coherent or, or as

93
00:05:39,660 --> 00:05:43,353
good of responses as you get from ChatGPT. But

94
00:05:43,353 --> 00:05:47,078
yeah, no, people forget. Hopefully none of our listeners, because a lot of ours

95
00:05:47,078 --> 00:05:50,739
are very savvy. But a lot of people, a lot of the normies out there,

96
00:05:50,819 --> 00:05:54,577
just assume there were no chatbots prior to— There was a lot of

97
00:05:54,593 --> 00:05:58,237
hard work that probably won't get any glory because it

98
00:05:58,237 --> 00:06:01,721
just wasn't as cool. But we saw something I think really

99
00:06:02,154 --> 00:06:04,786
we saw a piece of the future when we built this system, which was that

100
00:06:05,668 --> 00:06:09,454
when we got it right, when people were able to communicate effectively with

101
00:06:09,454 --> 00:06:12,984
this chatbot, the engagement stick looks

102
00:06:13,064 --> 00:06:16,786
basically the same as the ChatGPT engagement stick. It just was vertical. Right. So

103
00:06:17,011 --> 00:06:20,477
if we got you, you were hooked. You'd come every single day, you'd talk with

104
00:06:20,525 --> 00:06:23,140
us for many minutes a day trying to get your weather. If we didn't, of

105
00:06:23,172 --> 00:06:26,349
course you were gone because we didn't get it right. But you could see already

106
00:06:27,005 --> 00:06:30,638
what was possible if you could make this work in a more generalized way.

107
00:06:31,141 --> 00:06:33,801
The stickiness of chat cannot be overstated.

108
00:06:35,745 --> 00:06:38,811
You know, yeah, the way you've been talking about it, you, you know, you were

109
00:06:38,811 --> 00:06:42,522
involved before ChatGPT. So my question is, how

110
00:06:42,602 --> 00:06:46,176
has your definition of AI changed over the last 5

111
00:06:46,176 --> 00:06:50,006
years? Oh, that's a good— that's

112
00:06:50,006 --> 00:06:52,908
a really good way of putting it. Look, I think in the before times,

113
00:06:54,137 --> 00:06:57,183
You would build— like, most of what was built was machine learning, if we're honest

114
00:06:57,183 --> 00:07:00,983
with ourselves. Most of the things that you would practically create was more about

115
00:07:01,111 --> 00:07:04,783
basically statistical algorithms to try to find efficiencies or try to find

116
00:07:04,799 --> 00:07:08,647
things that were popular. So much of it was something that people

117
00:07:08,647 --> 00:07:12,270
wouldn't even recognize today, actually, when they say something like AI. These were like internal

118
00:07:12,270 --> 00:07:16,118
tools that you would use to improve the various performances in a company, maybe improve

119
00:07:16,118 --> 00:07:19,902
your supply chain by 5%. But already you saw

120
00:07:19,902 --> 00:07:23,736
the kernels, as I'm saying, of what you could possibly do. I think now obviously

121
00:07:23,736 --> 00:07:27,540
we're in the era of much more generalized models, which to me feels like one

122
00:07:27,573 --> 00:07:30,848
of the big breakthroughs of GPT that was very different from what was happening with

123
00:07:30,864 --> 00:07:33,464
Watson. Watson in its day was pretty badass. I don't know if I can say

124
00:07:33,464 --> 00:07:36,274
that word on your pod. I don't know what your PG rating is, but— We'll

125
00:07:36,274 --> 00:07:39,501
figure it out. All right. You gotta bleep me, let me know. But

126
00:07:40,271 --> 00:07:43,819
you remember this thing won Jeopardy, which was very hard to do a long time

127
00:07:43,819 --> 00:07:47,351
ago, but things were much more specialized. You'd be building models that were

128
00:07:47,351 --> 00:07:51,076
around one very specific business problem. You try to train it

129
00:07:51,172 --> 00:07:54,416
on the specific business problem. And that kind of made sense,

130
00:07:55,219 --> 00:07:58,719
especially from IBM's perspective, because they have business customers that all it's about— it's only

131
00:07:58,719 --> 00:08:02,476
about private data. But it was very limiting. And I think that's what actually

132
00:08:02,829 --> 00:08:05,527
the breakthrough with GPT was, like, wait a second, actually the right way to get

133
00:08:05,559 --> 00:08:09,397
to something that's more useful even for specific tasks is to make the

134
00:08:09,397 --> 00:08:13,202
models more generalized. And obviously now today, people, you

135
00:08:13,202 --> 00:08:16,508
can almost ask it anything And it can do almost anything.

136
00:08:17,173 --> 00:08:20,270
For me, the exciting part though is not even where we are at the moment.

137
00:08:21,880 --> 00:08:24,624
Language models— this is something a lot of people in the public don't understand, your

138
00:08:24,720 --> 00:08:28,368
audience will understand— language models are trained on language.

139
00:08:29,095 --> 00:08:32,124
A lot of people don't understand that they're not math geniuses. We're trying to make

140
00:08:32,124 --> 00:08:35,522
them better at math, but they're actually word geniuses. And

141
00:08:35,878 --> 00:08:39,536
that unlocks a huge amount of human potential that computers never touched before.

142
00:08:40,548 --> 00:08:44,146
Our knowledge really has been— it's been unable to

143
00:08:44,438 --> 00:08:48,149
communicate with computers in that way. But even though that's been

144
00:08:48,246 --> 00:08:51,883
really amazing, I think the next step, which we're not there yet, is getting the

145
00:08:51,883 --> 00:08:54,823
models to do things that are not language, right? So people don't quite understand that

146
00:08:55,340 --> 00:08:59,183
you can't build a rocket ship to go to Mars with a language model. We

147
00:08:59,183 --> 00:09:02,362
don't— that's what the models do today. So we're still like a step away, I

148
00:09:02,362 --> 00:09:06,206
think, from some very different innovations that probably don't look like the language models

149
00:09:06,239 --> 00:09:09,930
today. So that was a long answer on that question, but— All

150
00:09:09,930 --> 00:09:13,670
the math that these models are good at all go—

151
00:09:13,670 --> 00:09:16,990
they're not good at math per se, like you said, but they are good

152
00:09:16,990 --> 00:09:20,690
at math. But all the math goes into linguistics and

153
00:09:20,690 --> 00:09:24,470
text, right? You can ask it— what's interesting is now if you ask it to

154
00:09:24,470 --> 00:09:28,150
say how many Rs are in strawberry or what's 2 2, it'll

155
00:09:28,150 --> 00:09:31,860
actually write code. It's good at writing the code that'll give you the answer,

156
00:09:32,090 --> 00:09:35,439
right? But no, you're right. And it's an interesting—

157
00:09:36,762 --> 00:09:39,522
it's very curious to see how

158
00:09:41,039 --> 00:09:44,666
people misinterpret what AI is doing, right? People say, oh, it's

159
00:09:44,666 --> 00:09:48,048
hallucinating. I was like, from a certain point of view, it's always hallucinating, right? Because

160
00:09:48,048 --> 00:09:51,817
it's just guessing the next logical step. And I know there's

161
00:09:51,817 --> 00:09:55,578
attention mechanisms and all that that do keep it honest. But I

162
00:09:55,627 --> 00:09:59,340
was once at an event, and I hate to name-drop, but

163
00:09:59,340 --> 00:10:02,729
this was Esther Dyson. Oh, yeah. Right? Yeah. Somehow I ended up in the same

164
00:10:02,729 --> 00:10:06,424
room with her. I'll never figure out how, but, but she was, I

165
00:10:06,424 --> 00:10:10,247
was on the same panel and she was saying like, this was 2023, so early

166
00:10:10,279 --> 00:10:13,796
days of what we're, what we have today. Yeah. And she was saying, well, there's

167
00:10:13,812 --> 00:10:17,346
no real difference between autocomplete on your phone or

168
00:10:17,346 --> 00:10:21,089
guesses the next number and, and ChatGPT. And I was

169
00:10:21,105 --> 00:10:23,771
like, far be it for me to tell her she's wrong. Sure. Certainly in a

170
00:10:23,771 --> 00:10:27,353
public forum. But also too, she's right in a sense that the stealth

171
00:10:27,385 --> 00:10:31,076
bomber is the same tech, technology as a paper airplane,

172
00:10:31,157 --> 00:10:34,970
right? They're both doing a lot of the same things. Obviously one's way more

173
00:10:34,970 --> 00:10:37,896
advanced than the other. What fascinates me— I won't explain on that one.

174
00:10:39,130 --> 00:10:42,646
Yeah, I think both are radar invisible. But no, but I'm

175
00:10:42,695 --> 00:10:46,421
actually— I often find myself surprised at how far we've taken

176
00:10:46,421 --> 00:10:50,240
the transformer and LLM architecture. I didn't think— I thought

177
00:10:50,369 --> 00:10:53,638
we would run out of steam after a couple years, right? But

178
00:10:54,057 --> 00:10:56,779
things like things that have no business working, fine-tuning,

179
00:10:57,890 --> 00:11:01,331
That was— for fine-tuning, that's harsh. But for things like

180
00:11:01,331 --> 00:11:04,913
distillation, it shouldn't work as well as it does.

181
00:11:06,270 --> 00:11:09,094
Reasoning on these things also works way better than I would have

182
00:11:10,079 --> 00:11:13,581
bet on. I'm sorry, I cut you off. I'm agreeing with you. I think reasoning

183
00:11:13,581 --> 00:11:16,970
and logic is something you would think maybe would be beyond guessing the next word

184
00:11:16,970 --> 00:11:20,520
or the next token. But you can see

185
00:11:20,520 --> 00:11:24,280
emerging properties in language models that seem to go far beyond what you think guessing

186
00:11:24,280 --> 00:11:27,998
the next word would offer way

187
00:11:27,998 --> 00:11:31,528
more. And I can only imagine that's probably the attention

188
00:11:31,560 --> 00:11:35,165
mechanism. There's some kind of wisdom in the attention mechanism. But that—

189
00:11:36,891 --> 00:11:40,489
I think it's going to sound weird. I almost think we don't

190
00:11:40,489 --> 00:11:44,247
necessarily know. No, that's crazy. Yeah. There's the

191
00:11:44,247 --> 00:11:47,401
new paper that came out from Anthropic, I think last week, that was talking about

192
00:11:47,401 --> 00:11:50,912
that. It wasn't even totally new because this idea of a latent space inside

193
00:11:50,960 --> 00:11:53,679
models has been around for a long time. This idea that there's some kind of

194
00:11:54,985 --> 00:11:58,453
its own thinking space where it's doing things in its own language, in its own

195
00:11:58,517 --> 00:12:02,273
quote-unquote mind, whatever that means for a model. But Anthropic reconfirmed that

196
00:12:02,273 --> 00:12:05,693
and is finding that their models are now— they've been able to figure out where

197
00:12:05,902 --> 00:12:09,642
inside the model this activity is happening. That's interesting because

198
00:12:09,738 --> 00:12:13,190
obviously no model has been built to do that specifically. And so what is so

199
00:12:13,190 --> 00:12:16,866
surprising is just by training them on the world and then

200
00:12:17,139 --> 00:12:20,767
refining them, they have emergent properties that aren't specifically developed.

201
00:12:21,554 --> 00:12:25,153
That's a whole new kind of thing in the history of human invention. There's no—

202
00:12:25,169 --> 00:12:27,949
like when someone invented the fork, it didn't have— there was no way it would

203
00:12:27,965 --> 00:12:31,693
have some emergent properties and suddenly become a spoon and do something else. So

204
00:12:31,741 --> 00:12:35,452
something, something special is going on as these models get bigger,

205
00:12:35,468 --> 00:12:39,131
smarter, better trained, and more refined. Oh, that's a great way to put it. And

206
00:12:39,131 --> 00:12:42,923
it makes you wonder too, what— maybe language is the secret

207
00:12:42,923 --> 00:12:46,586
sauce to— I don't want to say consciousness, but intelligence. Maybe language

208
00:12:46,683 --> 00:12:50,307
is in itself the magic. Emerg— no,

209
00:12:50,467 --> 00:12:54,250
the notion of emergence, emergent properties is not unique to AI,

210
00:12:54,458 --> 00:12:57,904
right? You'll see birds will flock a certain way, right? And

211
00:12:58,321 --> 00:13:01,607
all they're doing is following the bird in front of them, right? And they inadvertently

212
00:13:01,607 --> 00:13:05,117
makes these patterns. And it seems

213
00:13:05,133 --> 00:13:08,820
language is the gateway to some kind of emergent properties of

214
00:13:08,820 --> 00:13:12,506
intelligence and reasoning. It could be. And I'm actually also interested in

215
00:13:12,506 --> 00:13:16,353
languages that aren't our own. If I think about like DNA, a chemical

216
00:13:16,353 --> 00:13:19,961
language. It's one of those areas where I think language models will eventually be pretty

217
00:13:19,961 --> 00:13:23,794
good, as opposed to the pure physics of building a spaceship, which they

218
00:13:23,794 --> 00:13:26,552
will be good at as well, but this is a different problem. But there's lots

219
00:13:26,552 --> 00:13:30,353
of languages that humans are not naturally conversant in. We can't think in

220
00:13:30,385 --> 00:13:34,057
DNA, but a language model perhaps can, and that can— You could find—

221
00:13:34,057 --> 00:13:37,826
that's a great point, right? What sorts of things could this possibly unlock?

222
00:13:37,986 --> 00:13:41,546
And mathematics is also, to your point about building a spaceship, mathematics is also a

223
00:13:42,349 --> 00:13:46,186
is a language in itself. Most humans don't

224
00:13:46,186 --> 00:13:49,895
think it, can't think of it natively, but there are some, right? It might be

225
00:13:49,895 --> 00:13:53,652
the gateway to, to other ways of thinking about this.

226
00:13:54,920 --> 00:13:58,067
Very much so. Yeah, it's— Candace, you kicked off an awesome question, by the way.

227
00:13:58,115 --> 00:14:01,519
I think that you answered that question maybe 10 minutes ago and it kicked off

228
00:14:01,519 --> 00:14:04,473
a great conversation. So thank you. No, it was great. I really, I'd like to

229
00:14:04,473 --> 00:14:07,829
dig a little deeper into the idea of DNA as a chemical language.

230
00:14:08,640 --> 00:14:12,265
So what do you mean by that? And how has thinking

231
00:14:12,313 --> 00:14:15,632
about biology influenced the way you think about AI?

232
00:14:18,198 --> 00:14:21,083
Boy, that's a good deep question. Look, in our practical work and the things we

233
00:14:21,083 --> 00:14:24,629
do for customers every day, I wouldn't say that we think in a biological sense

234
00:14:24,629 --> 00:14:27,530
or that DNA is part of that story, at least the work we deliver today.

235
00:14:28,963 --> 00:14:32,426
But it excites me because when I try to think about what are the unique

236
00:14:32,442 --> 00:14:35,840
things that— where are the places in the universe where language can create something new?

237
00:14:36,916 --> 00:14:40,495
DNA literally is the language of life. English or

238
00:14:40,800 --> 00:14:43,480
Spanish, human languages, are like the language of consciousness.

239
00:14:44,556 --> 00:14:48,199
DNA is the language of life, the fundamental building blocks of creation,

240
00:14:48,953 --> 00:14:51,184
at least on this planet. We don't know what it'll be like on some others.

241
00:14:52,404 --> 00:14:56,191
That's powerful witchcraft. That's powerful stuff to be playing with. And so I—

242
00:14:56,432 --> 00:15:00,091
and things that the human brain just isn't really built to speak that, to

243
00:15:00,460 --> 00:15:04,104
think conceptually in DNA. So to me, that's going to be a very

244
00:15:04,120 --> 00:15:07,699
powerful space when we can really get language models just to be native,

245
00:15:08,790 --> 00:15:12,481
natively speaking the language of creation of life on Earth. That's

246
00:15:12,481 --> 00:15:16,269
super interesting. It's nothing we do. I wish I could say I'm delivering that to

247
00:15:16,333 --> 00:15:20,089
Air France and EDP, some of our customers today. We're not, sorry. But that's some

248
00:15:20,089 --> 00:15:23,042
cool sci-fi where I think we're not that— we're not 20 years from that. Maybe

249
00:15:23,074 --> 00:15:26,846
we're 10 or 5. Baby steps. Yeah. So one of the things that I

250
00:15:26,846 --> 00:15:29,863
see on the Valintor website is

251
00:15:30,842 --> 00:15:34,527
really piqued my interest for a number of reasons. It was, you mentioned

252
00:15:34,623 --> 00:15:38,022
enterprise visual intelligence, which one, that's

253
00:15:38,425 --> 00:15:41,355
very interesting. 2, for sovereign AI

254
00:15:41,644 --> 00:15:45,468
environments. So tell me about what sovereign AI means

255
00:15:45,581 --> 00:15:49,100
to you, because everyone has a slightly different take on it. I've written a book

256
00:15:49,100 --> 00:15:52,102
on it. I have my own take on it. I'll send you a free copy

257
00:15:52,102 --> 00:15:54,790
of my book and you can, you can, you can tell me if I'm full

258
00:15:54,839 --> 00:15:58,474
of it or— Same, same. What? I'll sign it for you. Yeah, I'll

259
00:15:58,474 --> 00:16:02,194
sign you, I'll sign you the PDF. But the, the sovereign AI

260
00:16:02,483 --> 00:16:06,011
went from being this really weird niche topic to

261
00:16:06,187 --> 00:16:09,619
now is the topic of the G7, right? And

262
00:16:09,619 --> 00:16:13,356
obviously there's fallout for that. So what does sovereign AI mean

263
00:16:13,388 --> 00:16:17,237
to you? For me, it means, I think for us as a

264
00:16:17,237 --> 00:16:20,348
company, it means that we believe, look,

265
00:16:20,364 --> 00:16:23,459
fundamentally, I think the world's most important

266
00:16:23,973 --> 00:16:27,747
The world's most important knowledge, in a lot of sense, is not on the

267
00:16:27,747 --> 00:16:31,554
public internet and never will be. So much of

268
00:16:31,586 --> 00:16:34,606
the vital data in the world sits

269
00:16:35,296 --> 00:16:38,477
behind firewalls, right? Sits behind corporate or government or military walls.

270
00:16:39,376 --> 00:16:43,071
And if we want to liberate that with AI, with the

271
00:16:43,071 --> 00:16:46,781
newest technologies, we think that's not going to come from companies or

272
00:16:46,781 --> 00:16:50,620
organizations that have built 20, 30 years of digital security around the golden

273
00:16:50,620 --> 00:16:53,844
egg of knowledge in their company. That's probably not going to be them opening up

274
00:16:53,844 --> 00:16:57,658
the kimono and firing that off across the internet and sending it to ChatGPT

275
00:16:57,658 --> 00:17:01,215
or Anthropic. There's a big— the next wave, I

276
00:17:01,215 --> 00:17:04,660
think, is letting companies, governments,

277
00:17:04,724 --> 00:17:08,538
countries even have sovereignty over their

278
00:17:08,570 --> 00:17:12,223
entire AI stack, from the data that they've been protecting for

279
00:17:12,271 --> 00:17:15,668
decades to getting the models and the

280
00:17:15,781 --> 00:17:19,564
harnesses and the stacks to run right next to the data. So

281
00:17:19,564 --> 00:17:22,921
you have a boundary around your world, not like a

282
00:17:22,921 --> 00:17:25,957
militaristic border, but like a boundary around your space so you can

283
00:17:26,230 --> 00:17:29,989
confidently and safely build AI that sits right next to the

284
00:17:29,989 --> 00:17:33,427
data that you've been working on protecting for the last few decades. I know in

285
00:17:33,459 --> 00:17:36,511
the political space, as you brought that in, obviously the

286
00:17:36,511 --> 00:17:40,206
EU is thinking about this, the Middle Eastern countries are thinking about this. We

287
00:17:40,206 --> 00:17:43,900
often get calls from, from the Middle East on this stuff. I think

288
00:17:43,900 --> 00:17:47,450
they think it more from like a geopolitical perspective of Can we trust

289
00:17:47,515 --> 00:17:51,322
America? Can we— increasingly, companies, countries feel like they can't.

290
00:17:51,370 --> 00:17:55,128
Our relationships are fraught. So if we can't trust America, maybe

291
00:17:55,128 --> 00:17:58,566
we can't trust the products coming out of America, and we need to have control

292
00:17:59,064 --> 00:18:02,148
from the global perspective. We're not really thinking about it from

293
00:18:02,196 --> 00:18:05,665
geopolitics, not what we do. But when we go to a company like

294
00:18:05,665 --> 00:18:09,376
ADP, for example, obviously they have a tremendous amount of

295
00:18:09,520 --> 00:18:13,118
enormously privileged data from their customers, the largest payroll provider in the

296
00:18:13,118 --> 00:18:16,874
country, hundreds of millions of people's financial information is stored in that

297
00:18:16,874 --> 00:18:20,590
company, a lot of it actually down in mainframes in the basement, old

298
00:18:20,590 --> 00:18:23,573
IBM machines that we used to work for, old iron.

299
00:18:24,791 --> 00:18:28,413
That's not going out to ChatGPT. They wanna build AI around that

300
00:18:28,413 --> 00:18:32,213
stuff too, but they're not gonna use some public endpoint to throw that at

301
00:18:32,213 --> 00:18:35,463
Fable, whoever it is. So they need solutions to bring really powerful AI

302
00:18:36,013 --> 00:18:38,703
that can sit next to their best data. And that's really how we think about

303
00:18:38,703 --> 00:18:42,262
it. Because as

304
00:18:42,262 --> 00:18:45,441
organizations— I'm sorry, go ahead. No, go ahead, Candace. I've been hogging the mic because—

305
00:18:45,553 --> 00:18:49,278
As organizations continue to adopt AI, how do they— how do you balance

306
00:18:49,407 --> 00:18:52,072
innovation with the need to keep sensitive data

307
00:18:52,779 --> 00:18:56,279
secure, compliant, and under your own control?

308
00:18:57,692 --> 00:19:01,385
I think when people— when people say— I'm trying to guess what the word innovation

309
00:19:01,385 --> 00:19:04,532
means here for you. I think what you might mean is the latest, greatest model

310
00:19:04,532 --> 00:19:08,371
from the biggest companies. And, but the open source universe

311
00:19:08,371 --> 00:19:11,905
of models is not far behind. It's generally, maybe it's 6 months

312
00:19:11,905 --> 00:19:15,681
behind. There's a lot of debate about how robust the models are. But it's

313
00:19:15,681 --> 00:19:19,280
getting shorter, right? The time span's getting shorter. And I think that's an interesting metric,

314
00:19:19,296 --> 00:19:22,493
right? 'Cause it used to be, ah, they're about a year behind or, or early

315
00:19:22,590 --> 00:19:26,204
in 2022, right? Ah, no one can ever touch OpenAI and where they

316
00:19:26,285 --> 00:19:30,076
are. Look, the, look, the, those companies may also

317
00:19:30,076 --> 00:19:33,691
make business decisions, the OpenAIs and the, the frontier

318
00:19:33,691 --> 00:19:36,838
model companies, as we're calling them, they may eventually move to a model that's not

319
00:19:36,886 --> 00:19:40,659
just SaaS, right? Today they've decided to be pure cloud. That doesn't

320
00:19:40,659 --> 00:19:43,004
mean they have to be pure cloud in the future. But from a, like, how

321
00:19:43,004 --> 00:19:46,825
do you help a company implement today standpoint, what we're really talking about

322
00:19:46,905 --> 00:19:50,341
is helping them run open source models

323
00:19:51,016 --> 00:19:54,629
inside their security perimeter. And our core technology, Ground X,

324
00:19:54,950 --> 00:19:58,081
is an enterprise-grade RAG and document understanding system.

325
00:19:58,723 --> 00:20:01,999
And the way that we've built it allows you to bring in all the documents

326
00:20:01,999 --> 00:20:05,661
of your company, all the knowledge of your company. And we've done

327
00:20:05,709 --> 00:20:09,468
some really special things on the first principles so that you don't even need a

328
00:20:09,468 --> 00:20:13,258
frontier model to get the best performance out of the information

329
00:20:13,258 --> 00:20:17,016
inside your company. We've actually built our own, trained our

330
00:20:17,016 --> 00:20:20,790
own vision models to take apart your documents, break

331
00:20:20,790 --> 00:20:24,340
them down into small pieces. Here's a table, here's a diagram, here's a text

332
00:20:24,340 --> 00:20:28,090
block. You break things down into atoms, then you could feed those pieces,

333
00:20:28,266 --> 00:20:31,968
small pieces of data one at a time into models and get the same kind

334
00:20:31,968 --> 00:20:34,820
of performance you would get as if you were trying to work to a frontier

335
00:20:34,820 --> 00:20:38,602
model. So we make it possible for companies to use smaller, faster, cheaper

336
00:20:38,602 --> 00:20:41,903
models on-prem or in managed cloud

337
00:20:42,383 --> 00:20:46,117
so they get all the benefits of the frontier on their private data.

338
00:20:47,158 --> 00:20:50,699
Yeah, that's how we're approaching it today. Interesting. So do you— are you talking about

339
00:20:50,699 --> 00:20:54,148
a chunking strategy? For RAG, or are you talking about

340
00:20:54,308 --> 00:20:58,080
fine-tuning or all the above? It starts with a chunking

341
00:20:58,080 --> 00:21:01,772
strategy. It's funny, our product really comes out of— we're talking now in a

342
00:21:01,772 --> 00:21:05,351
philosophical way about AI, but our product really comes from really practical problems.

343
00:21:06,250 --> 00:21:10,086
When we were at IBM, we'd see this problem. The same thing

344
00:21:10,118 --> 00:21:13,794
would happen over and over again on big projects where you were

345
00:21:13,794 --> 00:21:17,389
trying to get a company's information to work with a model. At that time,

346
00:21:17,421 --> 00:21:20,615
Watson was the model. Now it's other models. But the problem is always the same.

347
00:21:21,337 --> 00:21:24,483
What is a company's knowledge? Essentially, it's typically stored in documents,

348
00:21:25,365 --> 00:21:28,992
and documents are visually complex. And fundamentally, these

349
00:21:29,040 --> 00:21:32,844
documents confuse language models because they're

350
00:21:32,908 --> 00:21:36,246
either really long, or they're very visually confusing,

351
00:21:37,000 --> 00:21:40,659
or they're very dense in certain parts. The frontier models are getting

352
00:21:40,691 --> 00:21:44,350
better at understanding these things, but when we started, it was really bad.

353
00:21:44,687 --> 00:21:48,440
So our first problem was like, Company

354
00:21:48,440 --> 00:21:52,080
X hands you 100,000 pages of

355
00:21:52,080 --> 00:21:55,620
stuff. They say, this is what I know. This is what my department— this is

356
00:21:55,620 --> 00:21:59,420
what my legal department knows. This is what my HR department knows. Okay, how do

357
00:21:59,420 --> 00:22:02,860
you make that work with AI? At the time, you really

358
00:22:02,860 --> 00:22:06,420
couldn't. And what we did was we

359
00:22:06,420 --> 00:22:09,920
trained a vision model. We actually trained a video model because the document

360
00:22:09,920 --> 00:22:13,560
models weren't good enough. The Detectron

361
00:22:13,560 --> 00:22:16,740
2 is where we eventually landed. It's our most recent one we're working on from

362
00:22:16,740 --> 00:22:19,980
Meta. It's a really great video model. So the purpose of that video models designed

363
00:22:19,980 --> 00:22:23,251
to follow a soccer ball around the— around a video screen or

364
00:22:23,478 --> 00:22:27,154
security footage and things like that, but they're really good at detecting

365
00:22:27,170 --> 00:22:30,809
objects. We trained that vision model on a million pages of

366
00:22:31,168 --> 00:22:33,956
corporate data, all kinds of stuff— legal, medical, financial, what have you.

367
00:22:35,637 --> 00:22:39,343
And the purpose of that was because we realized the

368
00:22:39,343 --> 00:22:43,129
problem of a document or a million documents was too big for

369
00:22:43,129 --> 00:22:46,964
models to deal with. And this is still true because no matter what they

370
00:22:46,964 --> 00:22:50,559
say the context window is, it's not really true. And so you— it

371
00:22:50,559 --> 00:22:54,251
became obvious to us that to make models work with your documents and your

372
00:22:54,251 --> 00:22:57,862
data, you had to break them into really small pieces. And so we did that

373
00:22:57,862 --> 00:23:01,489
with a vision model that we trained to basically identify on every

374
00:23:01,489 --> 00:23:05,084
single page where's the table, where's the

375
00:23:05,084 --> 00:23:08,519
text block, and where's the graphic. And then we have an

376
00:23:08,519 --> 00:23:11,761
agentic pipeline that essentially feeds those small

377
00:23:11,809 --> 00:23:15,579
objects into a visual language model. Again, one that can be

378
00:23:15,579 --> 00:23:18,450
smaller, cheaper to run because we're giving it small bits to work on,

379
00:23:19,508 --> 00:23:23,309
and have it explain in great detail what those objects are

380
00:23:23,613 --> 00:23:27,125
into text that's friendly to language models, or text that's

381
00:23:27,125 --> 00:23:30,910
friendly to essentially search is the other aspect of this. Because when

382
00:23:30,910 --> 00:23:33,139
you build these systems— I'm in the weeds now, and you tell me if you

383
00:23:33,139 --> 00:23:35,961
want me to come back out to 30,000 feet, the engineering weeds a

384
00:23:35,961 --> 00:23:39,553
bit— these systems are really— RAG

385
00:23:39,553 --> 00:23:43,343
systems are document understanding is the first problem and

386
00:23:43,360 --> 00:23:47,014
search is the second problem. Yes. The document

387
00:23:47,014 --> 00:23:50,685
understanding problem, everyone thinks it's solved. It's actually not. It's still pretty hard.

388
00:23:51,920 --> 00:23:54,096
And if you get it wrong,

389
00:23:55,736 --> 00:23:58,901
the downstream consequences of it are extremely severe.

390
00:24:00,470 --> 00:24:03,932
It's not so much that you have incorrect

391
00:24:03,932 --> 00:24:07,670
information in your database. That's a problem. You ingested 100,000 documents or a

392
00:24:07,670 --> 00:24:10,950
million documents. And let's imagine you're using a system that's not ours and you've got

393
00:24:12,576 --> 00:24:16,203
15, 20% of that's incorrect when it gets turned into LLM-ready

394
00:24:16,203 --> 00:24:19,509
data. Okay? Okay. That's a problem.

395
00:24:20,440 --> 00:24:23,457
But the real problem is it's a silent failure.

396
00:24:24,613 --> 00:24:28,400
So you'd never know which chunks are right and which chunks

397
00:24:28,400 --> 00:24:32,140
are wrong. So now you've got this burning problem sitting in

398
00:24:32,236 --> 00:24:35,702
your database, in the, in your AI representation of your data, which you've now

399
00:24:35,702 --> 00:24:39,410
vectorized, you've turned it into numbers, you've done these things with it. But underlying, it's

400
00:24:39,410 --> 00:24:42,523
not the same as the actual information that came out of your company. And you

401
00:24:42,523 --> 00:24:46,341
don't know where it is. Kind of smolders just below the surface,

402
00:24:46,518 --> 00:24:49,807
and you never know when it's going to cause trouble. I'm sorry, I cut you

403
00:24:49,823 --> 00:24:53,433
off, but— No, that's right. And it— how does that represent to folks? It represents

404
00:24:53,433 --> 00:24:56,481
in a funny way, because at first blush, it looks like we don't have a

405
00:24:56,481 --> 00:25:00,251
problem at all. You produce an MVP, you're like, oh, it looks pretty

406
00:25:00,251 --> 00:25:04,006
good. Most of these answers are right. Looks pretty good. Then the SMEs get

407
00:25:04,006 --> 00:25:07,118
in there and really play, and they're like, wait a second, this is in— this

408
00:25:07,214 --> 00:25:10,781
is In an intermittent way, which every engineer knows is the worst problem you can

409
00:25:10,781 --> 00:25:14,428
have. You don't want an intermittent bug. In an intermittent way

410
00:25:14,541 --> 00:25:17,510
that's nearly impossible to figure out why or track, this thing is wrong.

411
00:25:18,986 --> 00:25:22,811
Now everyone thinks these are model hallucinations. We have

412
00:25:22,811 --> 00:25:26,373
found that 90 to 95% of the errors in RAG-based systems are

413
00:25:26,373 --> 00:25:30,048
not model hallucinations. They're incorrect—

414
00:25:30,161 --> 00:25:33,214
it's like an information gap. It's a comprehension gap.

415
00:25:34,485 --> 00:25:38,017
It's that the things that you have bad information— it's garbage in, garbage out. The

416
00:25:38,017 --> 00:25:41,694
information that sits in your RAG database is not an

417
00:25:41,694 --> 00:25:45,066
accurate representation of the world, of the documents you put in the first place.

418
00:25:45,404 --> 00:25:48,615
Right. Now someone asks a question or an agent asks a question of your RAG

419
00:25:48,615 --> 00:25:52,003
system, you've done a search. We can get into how you would do search and

420
00:25:52,003 --> 00:25:55,680
why that's complicated. You basically search your RAG database for the 20 to

421
00:25:55,680 --> 00:25:58,538
100 blocks of text most likely to contain the answer to this question.

422
00:25:59,736 --> 00:26:03,504
And then the model synthesizes those together and produces a final answer.

423
00:26:03,600 --> 00:26:07,368
This is RAG. Most of the errors are not because the

424
00:26:07,368 --> 00:26:11,082
model just made some stuff up, because the whole point of RAG is that you

425
00:26:11,082 --> 00:26:14,655
can only answer based on the content I've just given you. It's typically the

426
00:26:14,720 --> 00:26:18,357
answer— the content I've given you is wrong, and it's hard to

427
00:26:18,357 --> 00:26:22,010
figure out why or where. So I went down a rabbit hole here, but this

428
00:26:22,010 --> 00:26:25,711
is the core of our technology, the problem we've been trying to solve for a

429
00:26:25,711 --> 00:26:29,419
number of years, this is the problem we started solving at IBM. And

430
00:26:29,419 --> 00:26:33,041
frankly, IBM wouldn't let us solve it, and so we left. From

431
00:26:33,041 --> 00:26:36,437
IBM's perspective, it was a research

432
00:26:36,502 --> 00:26:40,013
problem that was too middle

433
00:26:40,061 --> 00:26:43,491
space. I couldn't commercialize it 6 years ago at IBM,

434
00:26:43,781 --> 00:26:47,533
so IBM said, I can't make money today. And it wasn't a

435
00:26:47,533 --> 00:26:50,692
$100 billion bet like quantum computing that they hope will make them into the next,

436
00:26:50,708 --> 00:26:54,030
you know, bigger than SpaceX. It was this weird middle research problem.

437
00:26:55,098 --> 00:26:57,088
And so we left it high level to build that. And that's the core of

438
00:26:57,104 --> 00:27:00,394
the Ground X tool, that and really powerful search, which I can get into if

439
00:27:00,394 --> 00:27:04,214
you want me to get into that. But that's the ultimate beginning data layer

440
00:27:04,278 --> 00:27:07,922
of how you build knowledge, an AI empowered knowledge system inside a company,

441
00:27:08,034 --> 00:27:11,773
whether you're using that for customer support or agent orchestration

442
00:27:11,950 --> 00:27:15,433
or finding fraud in insurance claims, which a lot of our customers do.

443
00:27:16,203 --> 00:27:20,007
You got to start with that, the engineering right to turn a company's knowledge

444
00:27:20,472 --> 00:27:24,000
So you don't have a comprehension gap into something AI can understand.

445
00:27:25,283 --> 00:27:28,731
No. And it's funny you mentioned IBM. Sorry, Candace, I cut you off.

446
00:27:29,213 --> 00:27:31,987
But one of— I used to work at Red Hat up until a few days

447
00:27:32,083 --> 00:27:35,724
ago. So a few days ago, literally. And so I'm quite familiar with that.

448
00:27:36,205 --> 00:27:39,701
So, and then one of— I'm sorry, what? We're built on actually the first

449
00:27:39,717 --> 00:27:43,438
Kubernetes, the first on-prem implementation we did was built on OpenShift. Oh, very cool. I

450
00:27:43,470 --> 00:27:46,869
worked on OpenShift AIs. It was But

451
00:27:47,014 --> 00:27:50,829
what's fascinating about this is that I, my last project was

452
00:27:50,941 --> 00:27:54,275
a RAG, kind of like how to teach the field sales, like what RAG is,

453
00:27:54,468 --> 00:27:58,074
right? Because there was this product called InstructLab. InstructLab said, hey, just

454
00:27:58,074 --> 00:28:01,264
fine-tune your models. You'll be fine. Don't worry about RAG.

455
00:28:02,082 --> 00:28:05,865
Didn't really work out that way. But so it was like, no, let's reintroduce

456
00:28:06,410 --> 00:28:10,209
RAG into the conversation. So ultimately, kind of like the lab I

457
00:28:10,241 --> 00:28:13,624
worked on, so I, I had the opportunity to, to deep dive into the space

458
00:28:13,624 --> 00:28:16,653
for a couple months starting in January. Yeah. So it's a spot near and dear

459
00:28:16,653 --> 00:28:20,377
to my heart because RAG RAG solutions are really going to be

460
00:28:20,506 --> 00:28:24,120
the shortest path to enterprise ROI. I still

461
00:28:25,100 --> 00:28:27,863
think so, obviously partially because we sell a product that does it, but

462
00:28:29,035 --> 00:28:32,697
we had to— we could choose to pivot into something else, but we— and every

463
00:28:32,714 --> 00:28:36,312
few months someone comes out and says RAG is dead. Why do they say that?

464
00:28:36,504 --> 00:28:40,295
They say that because the models are bigger and better, but we've

465
00:28:40,295 --> 00:28:43,315
always had this point of view that if you really think macro,

466
00:28:43,828 --> 00:28:47,438
Okay. The world's— it would be—

467
00:28:47,518 --> 00:28:50,999
it's very difficult to move all of the world's information,

468
00:28:51,497 --> 00:28:55,202
which is currently stored on the cheapest medium possible, hard drives and flash

469
00:28:55,202 --> 00:28:58,732
drives, move that into GPU, the world's most expensive

470
00:28:58,732 --> 00:29:02,261
silicon. So unless there's a huge architecture

471
00:29:02,261 --> 00:29:05,951
change on the hardware side, and I think eventually there probably will be,

472
00:29:06,753 --> 00:29:10,493
but for where we are today, you're not going to move everything

473
00:29:10,525 --> 00:29:13,906
from a hard drive onto a GPU. And that means by

474
00:29:13,906 --> 00:29:17,658
definition, you need something in the middle that can figure out what to

475
00:29:17,690 --> 00:29:21,502
feed the model when, depending on what kind of— what an agent wants

476
00:29:21,567 --> 00:29:24,330
or what a human wants to do next. And it's like RAG, I think, is

477
00:29:24,362 --> 00:29:28,010
that middle layer that's gonna be with us for quite, quite some time. Yeah, I

478
00:29:28,058 --> 00:29:31,402
think at most it won't die. It might evolve. I know there's RaFT and then

479
00:29:31,499 --> 00:29:35,117
there's something else. PEFT, I think, is the other one. No, RaFT

480
00:29:35,519 --> 00:29:39,285
is the other one. RAG-assisted Retrieval augmented fine-tuning, right? But

481
00:29:39,318 --> 00:29:42,768
I think the RAG pattern, whether or not it'll still be called RAG, is— you're

482
00:29:42,768 --> 00:29:46,523
right. I think it's fundamental. I think it's a fundamental force of the AI

483
00:29:46,523 --> 00:29:50,294
universe personally. But— Yeah, we do too. And you

484
00:29:50,294 --> 00:29:54,082
mentioned fine-tuning models. That was when this whole craze started. I think it would go

485
00:29:54,082 --> 00:29:56,874
into big enterprises and all of them were like,

486
00:29:57,837 --> 00:30:00,822
yeah, RAG sounds interesting, but also boring to me. I want to go fine— I

487
00:30:00,822 --> 00:30:03,229
want to fine-tune my own model. I want to do it because that looks like

488
00:30:03,229 --> 00:30:07,066
a lot of fun, frankly. Then you find out, and it's

489
00:30:07,066 --> 00:30:09,555
cool, at the other side of the engineering project inside a company, you're like, hey,

490
00:30:09,571 --> 00:30:12,108
we have our own model inside the company, and you're a hero. And that's awesome.

491
00:30:12,204 --> 00:30:15,255
Resume-driven development is probably what's behind it. Maybe.

492
00:30:16,379 --> 00:30:19,349
But when you get in there and you're like, wait a second, how do you

493
00:30:19,429 --> 00:30:23,074
actually fine-tune a model? You have to fundamentally— you need

494
00:30:23,122 --> 00:30:26,799
question-answer pairs. You have to distill knowledge out of, again,

495
00:30:27,169 --> 00:30:31,012
those millions of pages of documents we started with. You still have the same problem

496
00:30:31,012 --> 00:30:34,308
in the beginning. Now you're going to try to distill that into

497
00:30:34,695 --> 00:30:37,271
basically question-answer pairs that you can then go fine-tune a model with.

498
00:30:38,383 --> 00:30:42,104
And I think everyone found out pretty quickly that is dirty, messy, difficult

499
00:30:42,104 --> 00:30:45,549
business. And companies' knowledge is really

500
00:30:45,840 --> 00:30:49,075
chaotic. And trying to distill out the perfect

501
00:30:49,625 --> 00:30:53,045
tens of thousands of QA pairs out of that was actually pretty difficult and

502
00:30:53,045 --> 00:30:56,634
time-consuming and out of date. As soon as new information happens, your

503
00:30:56,651 --> 00:31:00,419
fine-tunes are out of date. So that being said, now

504
00:31:00,484 --> 00:31:03,071
we're doing something that's a little bit in the middle. One of the things our

505
00:31:03,071 --> 00:31:06,872
technology is really good at is extracting information from documents. And the

506
00:31:06,872 --> 00:31:09,859
pattern that most of our customers did in the beginning was pure RAG. Okay.

507
00:31:10,584 --> 00:31:14,383
And GroundX is an end-to-end RAG platform that

508
00:31:14,431 --> 00:31:17,716
does the ingest, the processing, the parsing, the storage,

509
00:31:18,360 --> 00:31:21,242
the search, the re-ranking, all of that. And you can set it up in 3

510
00:31:21,242 --> 00:31:24,958
lines of code. Wow. So that's cool, you know, problem solved.

511
00:31:25,824 --> 00:31:29,094
However, increasingly we're seeing is a different kind of pattern

512
00:31:30,232 --> 00:31:33,743
where a lot of the information, a lot of the things people

513
00:31:33,759 --> 00:31:36,405
ask are not actually good for RAG.

514
00:31:37,703 --> 00:31:41,054
People would— and customers don't often understand this, right? So

515
00:31:41,936 --> 00:31:45,735
people ask questions that typically merge structured

516
00:31:45,799 --> 00:31:49,647
information and unstructured information and sometimes graphed

517
00:31:49,647 --> 00:31:53,466
information, relationship between things. Okay, so let me give you

518
00:31:53,563 --> 00:31:57,033
a— let me give you a perfect example from some of our customers in the

519
00:31:57,033 --> 00:32:00,873
insurance space of a question that RAG is natively bad at. Okay,

520
00:32:00,905 --> 00:32:03,909
we do a lot of work, we have a lot of customers, we have another

521
00:32:03,925 --> 00:32:07,363
product called FraudX, and FraudX helps insurance carriers

522
00:32:08,022 --> 00:32:10,978
find the potential evidence of fraud in their claim files

523
00:32:11,814 --> 00:32:15,589
about 40 times faster than human review. Interesting. When you have

524
00:32:15,589 --> 00:32:19,350
an accident, let's say an accident in a building or on a construction site, How

525
00:32:19,350 --> 00:32:23,150
do you actually— what does fraud mean? Often someone's faked an accident,

526
00:32:23,230 --> 00:32:26,670
or they've had a real accident and they've inflated their injuries so they can have

527
00:32:26,670 --> 00:32:30,450
a much bigger settlement. Okay, the evidence for

528
00:32:30,450 --> 00:32:34,270
that is sitting inside the documents. It's sitting inside 5,000 to

529
00:32:34,270 --> 00:32:37,910
10,000 pages of documents. A human would take about 50

530
00:32:37,910 --> 00:32:41,610
hours to read these 5,000 to 10,000 pages. Yeah, a well-trained

531
00:32:41,610 --> 00:32:45,150
human will take about 50 hours to go try to find the inconsistencies in the

532
00:32:45,150 --> 00:32:48,925
stories. I'm going to get to the point. A lot of that is

533
00:32:48,942 --> 00:32:52,468
contained in the medical history, a medical chronology. Okay,

534
00:32:53,302 --> 00:32:56,300
so this person had an accident, 2 years later they had back surgery.

535
00:32:57,598 --> 00:33:01,317
Does the accident relate to the back surgery 2 years later? You could discover that

536
00:33:01,686 --> 00:33:04,587
in the medical chronology of every single thing that happened to them medically over 2

537
00:33:04,587 --> 00:33:08,355
years. An investigator might want to say something like,

538
00:33:09,701 --> 00:33:13,155
how many times did this person go to physical therapy Before they had back

539
00:33:13,205 --> 00:33:16,996
surgery. Because if you rush to back surgery,

540
00:33:17,580 --> 00:33:20,418
if you haven't gotten all the physical therapy, maybe you're trying to get a settlement,

541
00:33:20,661 --> 00:33:23,240
or maybe the doctors you're working with are trying to push you too fast.

542
00:33:25,287 --> 00:33:28,516
Okay. RAG cannot answer that question. Why?

543
00:33:29,532 --> 00:33:33,356
Because RAG can't count. So it's a judgment call, right?

544
00:33:33,485 --> 00:33:36,534
I mean— Not that. Not that.

545
00:33:37,196 --> 00:33:41,043
They're actually asking a very concrete question. They're asking a structured data question.

546
00:33:41,918 --> 00:33:45,561
How many physical therapy appointments did this person have before they had

547
00:33:45,610 --> 00:33:48,966
surgery? If you ask that question in SQL,

548
00:33:50,030 --> 00:33:53,760
you could answer it instantly for a millionth of a penny.

549
00:33:54,603 --> 00:33:58,260
Easiest thing in the world to do in an Excel spreadsheet. Or select

550
00:33:58,260 --> 00:34:01,458
star from wherever. Yeah, easiest thing to do.

551
00:34:02,443 --> 00:34:05,462
Weirdly, a RAG system cannot answer this question well.

552
00:34:06,318 --> 00:34:09,627
Why? Because RAG is using unstructured data. It's

553
00:34:09,643 --> 00:34:13,387
unstructured search is what RAG really is. Fundamentally what RAG is, is you've

554
00:34:13,419 --> 00:34:17,066
taken all this doc, all these unstructured documents, you've stored them.

555
00:34:17,741 --> 00:34:20,874
Someone asks a question, you then search that data

556
00:34:21,517 --> 00:34:24,666
to try to find the 20 to 100 blocks of text most likely to answer

557
00:34:24,666 --> 00:34:28,297
the question. Semantically most likely, right? That would— that's sure.

558
00:34:28,683 --> 00:34:32,121
Okay. We use a mixture of techniques, but let's imagine it's just purely semantic for

559
00:34:32,121 --> 00:34:35,730
the moment. What if you've had— let's

560
00:34:35,730 --> 00:34:38,990
imagine you get that, you nail that search, but someone has had 200

561
00:34:38,990 --> 00:34:42,360
physical therapy appointments. By definition, you're going to miss them.

562
00:34:43,930 --> 00:34:47,650
The other problem is RAG doesn't actually know how many things, how many events

563
00:34:47,650 --> 00:34:51,310
exist, because it doesn't— by its nature, it's searching, it's

564
00:34:51,310 --> 00:34:55,050
filtering out information, searching and giving you a piece of it. So let me

565
00:34:55,050 --> 00:34:57,810
get to the— sorry, let me get to the point. Okay, we're going to— Candace,

566
00:34:57,810 --> 00:35:00,030
I see you're floating. I'm going to, I'm going to take you home. We'll get

567
00:35:00,030 --> 00:35:03,716
it when we get there. Okay. We're gonna get right back to DNA in a

568
00:35:03,716 --> 00:35:05,138
second, I promise you.

569
00:35:08,320 --> 00:35:10,320
What we see now is that

570
00:35:11,965 --> 00:35:15,450
customers don't understand the difference between— even you guys were a little like, wait a

571
00:35:15,450 --> 00:35:18,595
second, there's a difference between a structured and unstructured question? You guys do this all

572
00:35:18,595 --> 00:35:22,384
day long. They don't understand very upfront that

573
00:35:22,384 --> 00:35:25,899
you might be asking a structured question— show me how many times a thing happened—

574
00:35:26,630 --> 00:35:30,448
You might be asking an unstructured question like a judgment call,

575
00:35:30,496 --> 00:35:32,646
Candace. Explain what happened in the ER visit.

576
00:35:33,882 --> 00:35:37,716
That's a good unstructured question. You might be

577
00:35:37,748 --> 00:35:41,519
asking a relationship question. How often did this doctor and lawyer work

578
00:35:41,519 --> 00:35:45,129
together on a case? And so what we really need are

579
00:35:45,129 --> 00:35:48,803
systems that when you ingest your documents, you can

580
00:35:48,803 --> 00:35:52,574
extract the information and interpret it in multiple ways so that you could

581
00:35:52,574 --> 00:35:56,288
store it in structured databases, unstructured

582
00:35:56,288 --> 00:35:59,903
databases, which mostly is vector, and even graph

583
00:35:59,903 --> 00:36:03,390
databases. And then when someone asks a question, they don't have to care

584
00:36:04,402 --> 00:36:07,744
that whether it's structured, unstructured, or relationship-based like a graph.

585
00:36:09,370 --> 00:36:12,346
So that's where we're going on the tactical level. We're seeing companies really need—

586
00:36:12,346 --> 00:36:15,844
it's going beyond RAG, but it starts with that

587
00:36:15,844 --> 00:36:19,352
document comprehension. If you don't get the documents right, if you can't

588
00:36:19,368 --> 00:36:23,101
extract it, all is lost downstream. All right, let's talk about DNA. Let's get off

589
00:36:23,149 --> 00:36:25,881
this topic, right? Let's get out of here. Let's talk about using language models to

590
00:36:25,881 --> 00:36:29,465
talk to whales or something like that. Let me ask you though, so then

591
00:36:29,899 --> 00:36:33,431
is knowledge extraction becoming the new competitive advantage?

592
00:36:35,476 --> 00:36:38,880
We think it's 2 things, and this is where the conversations start. It's knowledge extraction.

593
00:36:38,928 --> 00:36:42,778
It's, it's, can you accurately represent, we call it the comprehension gap. Can you

594
00:36:42,778 --> 00:36:46,418
accurately represent the information inside this company in some way that AI can do

595
00:36:46,418 --> 00:36:50,173
something with? And can you keep the information

596
00:36:50,285 --> 00:36:54,070
safe, which gets into this sovereign AI? And can

597
00:36:54,070 --> 00:36:57,863
you keep it inside the bound— whatever boundary a company is comfortable with? Are

598
00:36:57,863 --> 00:37:01,587
they comfortable with our SOC 2 cloud? Smaller

599
00:37:01,587 --> 00:37:05,425
companies are. Are they comfortable with a boundary that lives inside their

600
00:37:05,473 --> 00:37:09,104
managed cloud on AWS? Some information is

601
00:37:09,104 --> 00:37:12,426
okay there. Do they need the boundary to be in the basement? Do they need

602
00:37:12,426 --> 00:37:14,990
to be in a data center they control machines? Do they need to be air-gapped?

603
00:37:15,978 --> 00:37:19,114
The military needs things to be air-gapped. So we're built to run in all of

604
00:37:19,114 --> 00:37:22,862
these environments, wherever the boundary is. You can natively install us right

605
00:37:22,910 --> 00:37:26,566
next to the mainframes if you want to and still get that data comprehension

606
00:37:26,797 --> 00:37:29,999
to work. So would better

607
00:37:29,999 --> 00:37:33,801
documentation, better document

608
00:37:33,817 --> 00:37:37,660
understanding be more valuable than simply deploying another

609
00:37:37,773 --> 00:37:41,499
AI agent? The AI agents haven't—

610
00:37:41,580 --> 00:37:44,348
what's an agent? An agent is just a, it's a bunch of prompts. To me,

611
00:37:44,365 --> 00:37:47,931
an agent is like the future of the if-then statement. Okay. A very

612
00:37:47,979 --> 00:37:51,730
sophisticated if-then statement, really. An agent is a bunch

613
00:37:51,730 --> 00:37:54,407
of prompts that, you know, use a model for judgment to do a thing,

614
00:37:55,738 --> 00:37:58,928
whatever that function is. But underneath— but you have to have the data underneath for

615
00:37:58,928 --> 00:38:02,727
it to know what it's talking about. Ultimately, a company's knowledge is probably going to

616
00:38:02,727 --> 00:38:06,029
be stored in documents and conversations, and that has to be turned into something that

617
00:38:06,142 --> 00:38:09,909
agents can understand, or else

618
00:38:09,909 --> 00:38:11,544
you've got a bunch of nothing at the end of it. You've got a bunch

619
00:38:11,544 --> 00:38:13,419
of agents running around in circles talking to themselves.

620
00:38:16,094 --> 00:38:18,846
That makes sense. That's an interesting way to put it. Think of it.

621
00:38:19,817 --> 00:38:23,410
What, what do you think is next for

622
00:38:24,025 --> 00:38:27,476
kind of the rag space, right? I think you and I are both in agreement

623
00:38:27,476 --> 00:38:30,683
that it's here to stay. It's not going to die. There'll be many of clickbait,

624
00:38:30,941 --> 00:38:34,483
ragebait articles saying that. But,

625
00:38:34,563 --> 00:38:38,407
and the other thing too, actually, is you said there's multiple approaches,

626
00:38:38,503 --> 00:38:42,003
right? Obviously semantic is the most obvious, but What are the other ones

627
00:38:42,999 --> 00:38:46,131
that you find most effective? Yeah,

628
00:38:46,629 --> 00:38:50,147
mechanically under the hood, search is a hard

629
00:38:50,147 --> 00:38:53,889
problem, actually. And it's really core to making RAG work. It's obviously

630
00:38:54,066 --> 00:38:56,893
fundamentally 2 things. Can I get the document— the documents right?

631
00:38:57,841 --> 00:39:00,427
And when someone asks a question, can I find the right pieces inside the RAG

632
00:39:00,427 --> 00:39:03,896
database? We come from a funny place on this, actually.

633
00:39:04,202 --> 00:39:07,853
Because we started doing this before GPT launched. Once GPT

634
00:39:07,918 --> 00:39:11,662
launched, let me get a little tinfoil hat for a second. That's

635
00:39:11,662 --> 00:39:14,141
going to make it juicy for you. Okay, maybe a little tinfoil, make it juicy.

636
00:39:14,271 --> 00:39:16,117
I'm not a conspiracy guy, I'm just going to give you a little bit. Okay,

637
00:39:17,475 --> 00:39:21,001
juice it up for you. All right, so when—

638
00:39:21,276 --> 00:39:25,108
a lot of people don't know this— OpenAI originally had a RAG

639
00:39:25,156 --> 00:39:28,687
product. Okay, and

640
00:39:28,978 --> 00:39:32,767
the RAG product actually was— No, that's right. I'm sorry, I cut you off.

641
00:39:33,597 --> 00:39:37,113
I had a moment of realization. We were actually one of the first, first 20

642
00:39:37,226 --> 00:39:40,420
testers of OpenAI. We were like one of the early guys who were given access

643
00:39:40,420 --> 00:39:44,145
to the GPT models back in the day. And they had— they weren't really

644
00:39:44,193 --> 00:39:47,982
commercializing anything just yet, but they had a RAG service. And

645
00:39:47,982 --> 00:39:51,482
what a lot of people don't realize is it was not a vectorized service. They

646
00:39:51,674 --> 00:39:55,463
were not vectorizing the data yet. They were actually— their first pass

647
00:39:55,463 --> 00:39:58,915
at this, which is a bit more similar to what we do, was just storing

648
00:39:58,915 --> 00:40:01,996
it as text. And then doing various kinds of text search

649
00:40:02,831 --> 00:40:06,585
against the text to try to find the chunks that would eventually go

650
00:40:06,602 --> 00:40:10,132
into the language model. Now, here's the tinfoil hat part. Okay,

651
00:40:11,095 --> 00:40:14,176
when they launched ChatGPT publicly, they

652
00:40:14,946 --> 00:40:18,701
removed that RAG service. They killed all the

653
00:40:18,701 --> 00:40:21,044
web pages, so you could never even see that they had one unless you use

654
00:40:21,044 --> 00:40:23,964
the Wayback Machine. We've got some emails that we're talking about it with them way

655
00:40:24,414 --> 00:40:28,121
back, and then they told everybody, you know what? The better way to do

656
00:40:28,121 --> 00:40:31,943
search actually is for you to vectorize all your data and do

657
00:40:32,039 --> 00:40:35,780
vector search, not text search. And what do they wind up selling

658
00:40:35,812 --> 00:40:39,505
the next day? An embedding service to vectorize your data.

659
00:40:40,147 --> 00:40:43,808
So embedding models, that's the— and I'm not saying they're a tinfoil hat. They're an

660
00:40:43,905 --> 00:40:46,538
awesome company. They have great products. We use them all the time. They have plenty

661
00:40:46,586 --> 00:40:49,814
of detractors without us joining the— I'm not in the hater group at all.

662
00:40:50,520 --> 00:40:53,960
I'm making it fun and funny, but We always

663
00:40:54,073 --> 00:40:57,854
thought actually vectorization and vector

664
00:40:57,854 --> 00:41:00,909
search, which you call— most people call similarity search,

665
00:41:01,619 --> 00:41:05,439
is one of several techniques you need to blend together to

666
00:41:05,471 --> 00:41:08,004
get to the answer when you do search. Why

667
00:41:09,975 --> 00:41:13,786
vector search? When you say similarity, it's basically trying to

668
00:41:13,786 --> 00:41:16,451
find things that are related to each other, that's more similar to each other, to

669
00:41:16,451 --> 00:41:20,296
the question, right? Semantically similar to the question. Here's

670
00:41:20,296 --> 00:41:23,196
the problem. When you have a lot of data,

671
00:41:24,158 --> 00:41:27,988
you have too many things that are similar to each other. It's— if

672
00:41:27,988 --> 00:41:31,657
you've got a jar of red marbles and you've got 10

673
00:41:31,673 --> 00:41:35,230
red marbles, all different shades, and you say, find me the medium

674
00:41:35,455 --> 00:41:38,996
red marble in a jar of 10 marbles, you can do it.

675
00:41:40,342 --> 00:41:43,964
If you have 100 million marbles, all different shades of

676
00:41:44,012 --> 00:41:47,715
red, find me medium red marble. It's really hard.

677
00:41:48,678 --> 00:41:52,160
And so what happens in systems that are heavily vector-focused or

678
00:41:52,160 --> 00:41:55,161
100% vector-focused, which is a lot of systems out there today,

679
00:41:56,365 --> 00:41:59,879
the more information you add, the worse your RAG database performs.

680
00:42:00,954 --> 00:42:04,758
We've done a lot of testing on this. Most vector systems will

681
00:42:04,822 --> 00:42:08,577
start to fail or decline in performance in as few as 10,000 pages of

682
00:42:08,577 --> 00:42:12,365
information. Wow. Remember, our world, we

683
00:42:12,365 --> 00:42:16,040
do a lot of scale. But we have customers that have— an insurance customer

684
00:42:16,040 --> 00:42:19,400
might have 20, 30 million pages of information

685
00:42:19,400 --> 00:42:23,070
across their thousands of insurance claims.

686
00:42:23,410 --> 00:42:27,220
So I can't go to a customer and say, after 10,000 pages, look out,

687
00:42:27,220 --> 00:42:30,560
things are not gonna be so good. A lot of projects don't realize that's a

688
00:42:30,560 --> 00:42:34,120
problem because a lot of things haven't scaled yet. A lot of projects are still

689
00:42:34,120 --> 00:42:37,900
working on small knowledge bases. Yeah, they're all POCs. Yeah, there's

690
00:42:37,900 --> 00:42:40,580
a lot of that. We're getting past that now. We're going beyond that now, but

691
00:42:40,580 --> 00:42:43,856
still a lot of things are living in the small database world. When you scale,

692
00:42:43,969 --> 00:42:47,723
you start to get problems. So our approach has always been really fundamentally

693
00:42:47,739 --> 00:42:50,899
different and maybe a bit more similar to OpenAI's original idea.

694
00:42:52,006 --> 00:42:55,391
We actually begin with the first search is a

695
00:42:55,455 --> 00:42:59,193
bigram text search, taking advantage of 20 years

696
00:42:59,242 --> 00:43:02,899
of good text search that's been like solid technology around a long time. We

697
00:43:02,980 --> 00:43:06,589
use that to downsample to 1,000 potential retrievals out of the

698
00:43:06,589 --> 00:43:10,058
system. Then We've, we have a

699
00:43:10,510 --> 00:43:13,936
heavily tuned model that we designed that real-time

700
00:43:14,001 --> 00:43:17,849
vectorizes those 1,000 results, then does the similarity,

701
00:43:19,322 --> 00:43:22,591
and then does a re-rank at the same time. And then you get down to

702
00:43:23,206 --> 00:43:26,102
the 20 to 100, let's say, much more likely

703
00:43:26,684 --> 00:43:30,236
answers to your question. At the same time, we're

704
00:43:30,317 --> 00:43:33,659
also doing something different with the data. When we

705
00:43:33,804 --> 00:43:37,411
ingest your data we don't just represent it precisely as it

706
00:43:37,411 --> 00:43:40,763
exists on the page. We're creating a lot of

707
00:43:40,779 --> 00:43:43,970
metadata around every chunk so that

708
00:43:44,387 --> 00:43:48,060
the chunks themselves are more differentiated than

709
00:43:48,172 --> 00:43:51,941
any other system, so that when you do similarity, you wind up

710
00:43:51,941 --> 00:43:54,667
with things that are more different than each other. So you don't run into that

711
00:43:54,667 --> 00:43:58,516
marble problem we just discussed. So it's a more powerful way to do vector

712
00:43:58,516 --> 00:44:02,037
search, more differentiated on the data. It's merged with other

713
00:44:02,037 --> 00:44:05,351
techniques, and it's also merged with what we call like a

714
00:44:05,351 --> 00:44:09,195
micrograph inside of it. So we're— every time we ingest documents, we're

715
00:44:09,260 --> 00:44:12,946
creating keywords around all the chunks in there, and then we graph those.

716
00:44:13,477 --> 00:44:16,890
That allows us to graph the chunks together so we can see relationships between things.

717
00:44:16,939 --> 00:44:20,550
So basically, it's a 3— the marketing version would be like

718
00:44:20,550 --> 00:44:23,815
3-way hybrid search, but now I just told you how you actually do it.

719
00:44:25,649 --> 00:44:28,312
And well, yeah, that makes sense, right? Each one of these Each one of these

720
00:44:28,344 --> 00:44:32,134
is gonna have their own weaknesses, right? So hopefully it's like the power system where

721
00:44:32,134 --> 00:44:35,939
it's a 3-phase power, right? You know, when they all kind of, if you

722
00:44:35,971 --> 00:44:39,664
do enough different approaches, they, where one is strong, one's gonna be weak

723
00:44:39,664 --> 00:44:43,197
and vice versa. Yeah, I think that's part of it for sure. Yeah.

724
00:44:43,775 --> 00:44:47,259
So we've always come at it from just like a first principles perspective, which is

725
00:44:47,371 --> 00:44:50,438
sometimes good and sometimes bad cuz it's, you're, you're going against the grain of

726
00:44:51,064 --> 00:44:53,874
maybe what everyone thinks is the right way to do it. But to your point,

727
00:44:54,902 --> 00:44:57,856
what people think is the right thing— way to do something is

728
00:44:58,595 --> 00:45:01,902
there's— everyone has an agenda, right? That's interesting. For sure.

729
00:45:02,288 --> 00:45:05,756
Even OpenAI itself was a controversial bet. Now that the bet paid off, people don't

730
00:45:05,756 --> 00:45:09,465
realize it was a bet, right? There was a lot of controversy a decade

731
00:45:09,465 --> 00:45:13,238
ago of would this transformer architecture work

732
00:45:13,367 --> 00:45:16,482
well enough? If I just add more data and more money and more training, does

733
00:45:16,514 --> 00:45:19,741
this become something really interesting or not? Or do I need a different technique?

734
00:45:21,089 --> 00:45:24,862
And OpenAI made this enormous financial bet that, nah, just keep working on it, keep

735
00:45:25,247 --> 00:45:28,874
putting more cash, more training, and more data into this particular

736
00:45:29,260 --> 00:45:32,775
model architecture. It will be something quite extraordinary.

737
00:45:32,984 --> 00:45:36,660
So did that bet ever pay off? We don't know if it'll

738
00:45:36,660 --> 00:45:40,271
pay off yet, right? They haven't— Oh, pay is the operative word. But I think

739
00:45:40,271 --> 00:45:44,091
you're right. And I think, like we said earlier, like the transformer

740
00:45:44,108 --> 00:45:47,805
architecture, we've gotten more mileage out of it than I originally thought.

741
00:45:48,769 --> 00:45:51,710
And out of— than a lot of the original creators of it thought. I mean,

742
00:45:51,759 --> 00:45:54,636
there's a reason why— I'm like, I'm not inside Google, but I assume there's a

743
00:45:54,636 --> 00:45:58,330
reason why Google created— literally created it and was like, interesting,

744
00:45:58,362 --> 00:46:02,140
but am I gonna throw $10 million at— $10 billion at that? Maybe not that

745
00:46:02,140 --> 00:46:03,989
interesting just yet. Someone else proved it.

746
00:46:06,988 --> 00:46:10,239
Sorry, Candace, I was hogging the mic because I was geeking out pretty hard.

747
00:46:10,657 --> 00:46:13,939
No, I'm totally geeking out. I'm totally fascinated. And I'm wondering

748
00:46:14,881 --> 00:46:18,251
if you think RAG is a long-term architectural

749
00:46:18,364 --> 00:46:21,588
pattern, or do you think it'll eventually

750
00:46:21,929 --> 00:46:25,753
evolve into something more sophisticated as enterprise AI

751
00:46:26,043 --> 00:46:29,650
matures? In a way, my answer is—

752
00:46:29,827 --> 00:46:32,178
you're gonna hate this answer— I was a journalist for 20 years, and I hate

753
00:46:32,178 --> 00:46:35,006
when someone says both. I was like, pick a lane, man.

754
00:46:37,210 --> 00:46:40,882
And here's why I say both. It would be silly to say that what we

755
00:46:40,882 --> 00:46:44,367
have today is what we'll have tomorrow. Obviously, that's just not true. There'll be more

756
00:46:44,415 --> 00:46:47,996
interesting patterns and techniques to do these things over time. And

757
00:46:48,509 --> 00:46:51,544
we might be 10 years out, but the chipset will evolve. So maybe it can

758
00:46:51,544 --> 00:46:55,060
take on— maybe it can hold more data inside of the

759
00:46:55,060 --> 00:46:58,689
model than we can do today. But at the same time, if I think about

760
00:46:58,689 --> 00:47:01,708
the history of computing, so much never changes. We're still—

761
00:47:02,543 --> 00:47:06,316
take a really simple example, edge versus cloud. This

762
00:47:06,316 --> 00:47:09,761
conversation has been happening since the beginning of computing, right? There was the

763
00:47:09,761 --> 00:47:13,366
mainframe that was cloud, if you will. There was the

764
00:47:13,526 --> 00:47:17,211
compute was in the center, and then the mini computer moved it to a

765
00:47:17,211 --> 00:47:20,720
little bit of a rim around that, but it still was in the center. Then

766
00:47:20,752 --> 00:47:23,860
PC revolution, massive. Everyone's forget it, compute's on the edge.

767
00:47:24,838 --> 00:47:28,587
We have all this power on our desk. And then that happened

768
00:47:28,587 --> 00:47:31,407
for 20 years. Then I was like, wait a second, maybe we should go back

769
00:47:31,407 --> 00:47:34,050
to the cloud. And then the cloud came along, we start to put the compute

770
00:47:34,050 --> 00:47:37,839
in the center. Then mobile came along, and now we've got these powerful supercomputers in

771
00:47:37,839 --> 00:47:41,566
our phone. It's like, now it's at the edge. Now it's going to— same thing.

772
00:47:41,742 --> 00:47:44,553
So just a lot of metaphors in computing just seem to be forever. And to

773
00:47:44,553 --> 00:47:48,151
me, I think there's something in RAG that's pretty fundamental, which is

774
00:47:48,183 --> 00:47:51,926
that the big fence, fancy, expensive thing in the middle

775
00:47:53,387 --> 00:47:57,226
will probably never have all the information it needs to. And it's always going

776
00:47:57,258 --> 00:48:00,455
to need something that has a larger database of knowledge than it

777
00:48:01,275 --> 00:48:04,254
and filter it in real-time ways so that it can answer a question or perform

778
00:48:04,254 --> 00:48:08,085
a function. And if you just think about what RAG is really fundamentally, it's that.

779
00:48:08,278 --> 00:48:11,616
It's that the universe for information is really vast. It's out here.

780
00:48:12,797 --> 00:48:15,789
It's hard for me to store all that and use it in a functional way

781
00:48:16,355 --> 00:48:19,670
in real time inside whatever thing I'm building. And so you need something in the

782
00:48:19,670 --> 00:48:23,252
middle that helps you search and sort and filter and deliver it in the right—

783
00:48:23,446 --> 00:48:27,043
in real-time way. Sorry, that was not a straight-line

784
00:48:27,076 --> 00:48:29,480
answer. That was like both sides of the mouth on that one, but I think

785
00:48:29,480 --> 00:48:33,043
that's the truth. That makes a lot of sense.

786
00:48:33,916 --> 00:48:36,439
We are— I would love to talk to you another hour or 2, but I

787
00:48:36,439 --> 00:48:39,124
think we're at the end of time and I'm going to be respectful of your

788
00:48:39,172 --> 00:48:42,886
time. So where can folks find out more about you and your

789
00:48:42,886 --> 00:48:46,720
company? Yeah, so the company is Valantor

790
00:48:47,010 --> 00:48:48,380
and you can get us at valantor.com.

791
00:48:49,980 --> 00:48:53,477
V-A-L-A-N-T-O-R. Luckily it's spelled like it

792
00:48:53,477 --> 00:48:56,683
sounds. I just spent a week and 2 weeks in Croatia and

793
00:48:57,470 --> 00:49:00,920
I don't know what they're doing with the alphabet over there, but I'm trying to

794
00:49:00,920 --> 00:49:04,180
make sense of it. It's an awesome country. I got nothing bad to say, but

795
00:49:04,180 --> 00:49:07,820
they need some vowels. Okay. But Valontor is easy to

796
00:49:07,820 --> 00:49:11,380
pronounce. So valontor.com, check us out. And for

797
00:49:11,380 --> 00:49:14,440
me, look, if you want to see the documentaries, you could search— would you search

798
00:49:14,440 --> 00:49:17,980
weather films or Weather Channel documentaries on YouTube and see some of the cool stuff

799
00:49:17,980 --> 00:49:20,940
from my past? But check us out at valontor.com. And if

800
00:49:20,940 --> 00:49:24,540
you're an enterprise that has the kind of problems we're talking about, large

801
00:49:24,540 --> 00:49:27,440
amounts of data, scale, security, accuracy is what you need, say hi.

802
00:49:29,755 --> 00:49:33,316
Cool. Excellent. That sounds great. That sounds great. Thank you so much for your time

803
00:49:33,365 --> 00:49:37,135
today. It was fabulous. Thanks a lot, guys. I really appreciate it. And

804
00:49:37,183 --> 00:49:39,958
next time we'll get back on the DNA, Candace. I'm sorry, it felt like we,

805
00:49:40,087 --> 00:49:42,958
we ran outta time before we can get back there. It's okay. We'll have you,

806
00:49:42,974 --> 00:49:46,006
we'll have you back and then we'll— Have you back. Yeah. That's

807
00:49:46,985 --> 00:49:48,329
it. Thanks a lot. I'll let the outro music play.