Today on the data driven podcast, we have the privilege of hosting
none other than Jeremy Utley. Now, Jeremy
isn't just any guest. He's an academic marvel
and entrepreneurial spirit rolled into 1. Hailing
from the prestigious corridors of Stanford as an adjunct professor, he's
the kind of chap who educates the future disruptors of Silicon
Valley. He is here to tell us how to get the most out of
generative AI. Now on to the show.
Hello, and welcome back to Data Driven, the podcast where we explore the emergent fields
of artificial intelligence, data engineering, and data science,
and all the associated technologies. With me today is
Jeremy Utley, who is a, adjunct
professor, venture investor, and co author of the book,
Idea Flow, The Only Business Metric That Matters. Welcome to the
show, Jeremy. Thanks for having me. Hey, no problem.
So Stanford. That's kind of a big
deal. It's a it's a special place. Yeah. I'm
just trying to not get found out. I'm sure it's, you know, similar to,
the, the guy on Office Space. Right? At some point, there'll the clerical error
will be revealed. You'll you'll know when they move you to the
basement. Right? Exactly. Exactly. Yeah. But I've been teaching at
Stanford since 2009, and I've been delighted to get to learn alongside some of
the most incredible students in the world and and get to study some of those
incredible innovators in the world. So, not just I may be a
professor or an adjunct professor, but I really consider myself to be a front row
student in in the classroom alongside my students. Very
cool. Very cool. So, what
what is the most important metric? I'll start right there.
Well, the most important metric we call idea
flows, the only business metric that matters. And the reason that we make that bold
claim is because it's the only measure of your
team's capacity to solve problems. And the the only
constant in our day to day lives is problems. In in our businesses, I
don't know, a single business that is facing a day without problems. And
so if you think about problems as the constant, then your
team's capacity to solve problems is really the most important thing
that you should be measuring. And yet, nobody really even knows how to
measure it. And so we talk about idea flow as the as the way to
measure a team's capacity to solve problems.
Interesting. Interesting. And is this is this changing now when we have the
reality of AI assisted teams?
Yeah. Yeah. Absolutely. That's it's a really insightful question. Yes.
It does change or sorry. It has the
potential to change. And yet, what our research suggests that we've
conducted over the last year or so, is that sadly, it
actually doesn't change in practice. In theory, it could change but in
practice it often doesn't. What's interesting, so what
what are the barriers to this? Right? Because I have some thoughts on this. I
know that a number of companies have basically outright
banned, use of AI tools
with good intentions, right, because the privacy policies, etcetera, etcetera,
but in reality, people are copying and pasting sensitive stuff anyway.
So, it seems like banning something outright
doesn't always work in a number of areas. But
what what are the barriers? Right? Because it it it can, like
you said, but in practice, what's what's what are the blockers?
Ultimately, it's it's human psychology, really, is what what's,
the challenge. It turns out that our expectations of the technology,
are hamstringing our ability to make use of it. Because
we're approaching the technology. Most teams that we studied
approach the technology as an oracle. It's almost like a search
box. It's gonna give them the best answer. Right?
And that's the wrong way to approach the technology. It does feel somewhat
magical when you type in, you know, an enigmatic query
and get a seemingly intelligent response. I mean, that feels
magical, but the teams that do that underperform. The teams
that overperform are the teams that
treat generative AI not as an oracle, but as a thought
partner, as as a conversation partner, and iteratively
work together with the AI to discover a better answer.
And the irony of that is it's not very magical, actually. It feels
like work. And yet, where teams that
treat AI as a conversation partner arrive is
light years better than teams that treat
AI as an oracle perform.
Interesting. So what are the what I think I know what you're
getting at in terms of treating it like an oracle versus treating it like a
conversation. Because I've seen that as I do more and more of this, I
hate the term prompt engineering. I hate the strong word. I have mixed feelings about
the term prompt engineering because there is no one single prompt to rule them
all. Mhmm. At least that's been my experience where you kind of you kind
of it's like a conversation, like, you're having. It's not a
person I know. It's not a person I know. It's not a But it's but
but it is a mindset, Frank. That's the thing. It's a mindset. And people
don't come with the mindset of I want to have a conversation. People are lazy.
Right? So so Herbert Simon, back in 1954,
won the Nobel prize for what he deemed satisficing, which
was the human tendency to settle for good enough. Right? And in most
of our lives, it's fine. I need a good enough pair of jeans, I need
a good enough cup of coffee, whatever it is. Right? But when we're trying to
solve problems, good enough sometimes is is okay. But often, especially
when it pertains to innovation, you don't just want the good enough thing. You want
the best thing. And it's in that area where when we really
want the best solution, that our tendency to settle for good enough
really hurts us. Because what teams do is they put
in a prompt and they get a pretty good answer and they go, woah,
I I was prepared to take an hour working on this but we kind of
got pretty good in 5 minutes. You guys want to go get coffee? And everybody
just gives up because they got good enough. And so that's,
you know, I I think it really is a mindset thing. Forget the word prompt
engineering. It's all it's it's self engineering. It's human
engineering. And one of the best things that a human being can
do is say to the AI why they don't like
the answer the AI gave. Right? So take your expertise.
Here, this is something that anyone is listening can do right now. Take something
that you know you're an expert on. So for example, I'm an expert
on customer insights or or low resolution prototyping and
experimentation. Right? So I might say to the AI, can you give me a
step by step guide for how to conduct an experiment? K. If I
did that, we can do it right now live if we wanted to. But if
I do that, it's gonna give me like, you know, the average of the Internet.
Right? And it's gonna draw from a bunch of stuff that may be good, may
be bad. By the way, I'm an expert, so the chances of my knowledge surpass
it are reasonable, you know, at least. But because it's gonna give me
the average of the Internet, it's probably gonna find some corners that I don't know
about and it's probably gonna say some stuff that I vehemently disagree with. Well, where
most people give up and I think because they want AI to not
be that good is they look at the response and they go, see.
It didn't even know that you're supposed to test for desirability and not
feasibility. Right? Whatever. And then they say, that's why AI is no
good. Well, Human Engineering, not Prompt Engineering.
Human Engineering is to say, okay, human, tell the AI
what you disagree with and why you disagree with it,
and ask the AI to regenerate an answer
given the following considerations and put in your critique.
Most people if they do that, just even that one thought exercise,
will be blown away. Yeah. You get an order of magnitude better response.
Absolutely. Right? Absolutely. Because you're kind of focusing the cone
of, you know, inquiry with your own
expertise. And what people want is they I mean, no one would
ever if you think about, like, AI like an MBA intern. Right? No
one gets an intern from Harvard Business School, gives them 2
sentences of instruction and then at the end of the summer says, man, their workout
was no good. I didn't interact with them at all. I didn't give them any
guidance. But for crying out loud, what's
Harvard doing these days? Right? No. Nobody gives a
human being 2 sentences of input and then critiques how bad of a job they
did, right? And yet we open chat gpt, we give
2 sentences of input, if that by the way, and then we go, See, it's
not very good, well, work with it. Garbage in garbage
out. It's as old as possible. And the reason most people don't I think most
people don't wanna work with it is because they don't want it to be any
good. Yeah. I could see that. Totally.
And the people who do want it to be good will be unlocked and
unleashed. But it requires not prompt engineering, but
human copilot engineering. I do like the
fact that a lot of these tools that are coming out are being called copilots,
right? Because I think it shifts the focus away from
AI isn't going to do it all. AI is not
probably gonna take your job, right? But it's just an
assistant. Right? It's it's to help you out where you may
want a little bit of boost. I also think that I think what you you
described is good enough factor is I think people see
large language models and they they assume it's a search box only
better. Yes. Well, and part of our
challenge, you know, I was talking with a with a psychologist, David
McCraney, who wrote How Minds Change. He's a he's a journalist, an author,
a podcaster. He's a he's the host of You Are Not So Smart, which is
all, you know, obsessed with cognitive bias, which I love. Mhmm. I love
that podcast. And one of the things that David and I were talking about yeah.
I can't remember the the name for the cognitive bias, but when we see
something that we think we understand, we just track into
our kind of typical neuro pathways. Right? So we
see a text box and we go, oh, I've seen one of these
before. This is like that. And we so and this
this being generative AI is not like that. That being
search. Generative AI is not search. But because of the
kind of the the UI, we
approach it like search. We go, okay, I want the answer. Just give me a
list of links that I'm gonna click through and decide on. And we don't
interrogate Google, we don't critique Google or any search
engine, right? We don't say why we want it
or state our intention, right? But if you start to do
some of these fundamental kind of human
conversational tactics, if you start treating it more like a
person than like a search box, you get
exponentially better results. But you're right, even
the UI itself predisposes us to treat the
technology and to think about the technology in a particular way and
that is actually holding us back. Interesting. I noticed
this in a completely random thing. I was getting,
I was using DALL E to generate images, This is before Chat GPG had it
in there. And I wanted to make a painting that looked like a Rembrandt painted
a portrait of a dachshund. I know this is the most ridiculous thing.
Right? So I wrote the prompt, I said, you know, painting of a dachshund in
style of Rembrandt, and it produced something. It was okay. Right? It was
good. But I was like, I wonder what if I asked
ChatCpt to help me with this prompt? So I went over. Now
I could do it all in 1 window. But I said, like, hey. What would
what would you write for a prompt? Like, what would do that? And it came
back with, I mean, a paragraph to what you said, 2 sentences. This thing came
back with a paragraph. I mean, stuff that only art historians and art,
students would really appreciate. You know, this type of paint, this style of
brush, like, just stuff that I remember from art history class, but, like, you
know, I only took that class because I had to type thing, you know?
But but then I I pasted that prompt in there, and, oh,
okay. It's it's it's an image. It's art. It it it's somewhat subjective, but
the the result was so much better. Like, it was just day
and night, and That's true. That has changed the
way I think about, dare I say, prompt engineering. Right? Like, because you can
because I gave a talk on prompt engineering and, like, you know, the magic of
it, and I was like, you can actually have the models help you build out
prompts. Yes. Well, that's that's the thing that people don't understand
is, you know, I mean, I I interviewed the other
day on a I've got a podcast called Beyond the Prompt, which is all about
AI in organizations. And we've interviewed a bunch of amazing people.
You have co founder of Typeform, CEO of Section,
CEO of Every, the head architect at Instacart, a bunch of
interesting people. And one of the folks we interviewed last week is a
documentary filmmaker named Juan Carlos. And Juan Carlos has made some
amazing documentaries. And he said he's always wanted to build
an Ios application, but he's never had a developer and he's always seen that as
kind of prohibited. He can't do it. And then he said when ChatGPT came
out, he had the thought, could ChatGPT teach me how to
code? And he built an
Ios app by treating ChattGPT like his computer
science TA. And he would go to the TA and ask for
instructions. He got Chad GPT to teach him
how to build an Ios app. Nice. You would
never imagine doing that with a search engine, right? No. You would find it on.
You would find it on. But you would just, and anytime he got stuck, You'd
come back to the TA. Right? And you get more. But your
point about people's minds being open, I think they have to be
hearing examples like this. He literally went to
JIGBT and said I would love to build an Ios app but I've never
built anything. I don't have the first you know, sentence
of ways to even describe it. If you were gonna ask a developer or if
I wanted to ask a developer to do this, how would I even ask them?
What do I need to describe? Tell me everything you need from me in
order to tell me how to proceed. And he basically worked
with it's almost reversed. We're used to being in the driver's seat.
He basically told Chad GVT, you're in the driver's seat, please tell me what to
do. I'll be your hands, you tell me what I need to do.
And to me, that's just we have to start shifting paradigms. I'll
give you another example. I've got a good friend who
is considering a job transition. He lives on the East Coast,
wants to move back to where his family is, and he got a
job offer at a new firm. And he felt the job
offer wasn't a great offer. His wife felt, we don't wanna screw
this up. We wanna give back to family and we got a job. Just take
the offer. And he he kind of confided in me, I
feel like I could negotiate, but I don't want to mess things up. And I
said, well, have you role played it with Chad GPT? And he said,
what do you mean? I said, well, you can role play the conversation just
to see how it would go. He said, but they don't know anything about the
firm. I said, well, you can tell them. Ask ChadGpt, what do you need to
know about the firm and what do you need to know about the hiring manager
in order to believably play their role in a back
and forth role play with with me. Interview me about the company and interview me
about the person until you know enough to believably play
their role and then do a 1 on 1 negotiation with me.
Be observing the negotiation the whole time, and give me feedback not
only as my counterparty, but also as a negotiation
coach. That's brilliant. That's some sci fi
stuff right there. Dude, he came back and he was like, what do I
do now? That was mind blowing. I said, now, ask him
to play your counterparty, but be a little bit more aggressive as the counterparty, a
little bit less friendly. So he did that and he said, Jeremy, 2
things I learned. 1, or actually 3 things. 1, I was
missing my key point of leverage and ChatGPT helped me see it.
2, I forgot my negotiating strategy in the
in the heat of the moment, and chat g p t alerted me to that.
Now I'm prepared. 3, I'm no longer dreading
this negotiation. I know I can do it.
Wow. And to me, it's like that's it's it's so different than
saying, you know, portrait of a dash hound and, in remember
it's like people are doing that and going, that's all I can do is like,
you know, it can teach you how to build an Ios app. It taught me,
I got you at GBT to teach me how to code Python so I could
build my own chatbot using Python. I've literally never written a line of code
in my entire life, right? It's our imaginations are the
primary bottleneck here. And and part of the reason that our
imagination is constrained is because we've been
trained by search to interact
with technology in a particular way. And what I think most people need is they
need to hearing examples like this and they need to be getting in conversations with
other people who are trying stuff and going, I can do that. Yeah,
you could. I could do that. Yeah, you could. And you need to be having
these kinds of conversations to stimulate your own thinking to then discover
your own novel applications. No. That's brilliant. I
mean, the whole negotiation thing is amazing. I've seen a lot of
chatter online about people using it to, you
know, in the job search aspect of it.
Right? Like, here's the job description. Here's my current resume.
Have at it, you know? Write Reno, write a cover letter that is
gonna hit all these points and it'll do it. And, you know, but
I mean, the whole idea of role playing. I mean, that's just brilliant. Like, I
think I think the the the the the the $1,000,000
statement there is our imagination
is a limit, which is something that historically, when it comes to computers, I
would say beyond the the the the the search
interaction experience, we're not used to computers outthinking
us. Yeah. Yeah. And I think that that that's gonna have
some interesting, societal
consequences. Right? Because I mean, I think what what freaked people out about Chat
GPT was, you know, it looks like it's doing something
creative, which is something that we had naively assumed, was
something only humans can do. Mhmm. And I I I think you're right. I
mean, I think this is not just a chat search only better, but this is
definitely like a whole new type of computing. Yeah. I
think it really does require a behavior modification. And
what I I there there are kind of 2 big questions in my mind
for organizations or for leaders who are thinking about deploying these technologies.
1 is, what percentage of my workforce is comfortable
with these tools? And by the way, right now, I mean, sentiment
I read an Ernst and Young report that says 70% of people are afraid
of AI. You know, it's like, when the when the predominant
sentiment is fear, you're not in a position of kind of maximizing
opportunity. Right? So Right. You so fear is gonna hold you back from
that sense of comfort, confidence, etcetera. But then 2, so if you say so 1
question is, what percent of our workforce is comfortable? And then
2, how do I grow my conversation abilities?
Nobody knows how to have a conversation right now with with HHPT or with
any LLM. Many people have lost the art of having conversations with
human beings, right? So, but you you really have
to almost it's like becoming literate in a new language.
We need AI literacy courses. We've actually developed, my
partners and I, developed a conversational coach who gives
daily drills that send you into ChatTPT with kind of a
drill to build your conversational fluency. Because what we're finding
is, folks just they don't have any imagination. Do you know that you could
take ChatGPT, for example, and tell her what are your 5 favorite books
and why they're your favorite books and ask for recommendations.
It'll blow your mind. It'll give you recommendations that no human being's ever given
you. Interesting. You could tell it you could tell it your, you
know, 5 favorite quotes and ask for what
are what are patterns here and what does it tell me about myself and my
world view and what are my blind spots given these things that I'm drawn
to. Right? You can take your journal entries and, you
know, a particular difficult day that you've had recently.
And then you can you can ask ChargeG PTE, can you tell me
what are the mental models that are inhibiting my
ability from seeing this situation clearly? And it will tell
you, right? If it's That's wild. I'm just drawing on I
mean, by the way, I'm just kind of a purveyor of these examples. They're all
examples I've been hearing from people. But the point is, you can do so
much more than you imagine. And right now, nobody's putting themselves
or very few people even have kind of the the the wherewithal
to say, I've gotta be hearing more of these examples. I wanna know my
what my cognitive biases are. I wanna learn that new tool. I wanna try that
thing. And the more examples you hear,
the more your own imagination will be stimulated. Right? I mean, going back to idea
flow or kind of my area of expertise which is innovation, creativity,
etcetera. What we know cognitively is that the imagination is
stimulated by unexpected inputs. So, you
know, think back to Johannes Kepler gazing up in the night sky. Right? At
that time, the predominant paradigm was, it's the firmament, meaning it is
a fixed substance. Right? And Kepler sees a
shooting star, and his first thought is,
why isn't the firmament cracking?
Right. And that is what led to heliocentricity.
And, you know, the the total paradigm shift in the in
the understanding of our place in the universe starts
with a shooting star. Right? Unexpected inputs, sparks
the imagination. And so that's that's that's a tactic
that whether it's AI or anything else, putting yourself
in the mindset of I need to be seeking unexpected input.
Most people's lives are ordered to insulate
and protect themselves from anything unexpected. And yet it's the
unexpected which actually stimulates our imaginations and creates possibilities and
opportunities for us and ideas. This is wild. I
mean, like, I mean, one of the things that blew my mind was when they
added the ability to create custom GPTs. Right? So I started
tinkering with it, like, you know, if you listen to the show,
we have a character named Bailey. So I kind of taught it, like, what would
Bailey say? You know, this is the the idea for the character. This is kind
of the her tone, and this is her personality that
we've kind of defined. And for the last, I would say,
15 episodes, that's actually what generates most of or all of the
text that she says. Right? So it's kind of like I have my own
private it's not Jarvis by any stretch of the imagination, like, you
know, Iron Man, But I mean, it's kind of like, it's kind of like the,
the, the, I have enough raw material there. I can
pretend. Right? Cause the, the AI will say things like
the, like, oh, yeah, that works. I like the way, I like the way she
phrased that. And then I say she, because I mean, it's just funny. Like
it's just, and and you know, there's ones where,
there was a GPT I made where, you know, to help with motivation. It's like,
you know, pretend you're Tony Robbins and you're trying to, like, motivate
somebody to to do the best they can do. And, yeah, I've interacted with that.
I'm impressed. I mean, it's just mind boggling what,
it's mind boggling what these what this thing can do. And when as
the engineer in me, I know this is just some kind of vector representation
of language. It's a predictive model. Yeah. It's it's statistics. I
think, you know, here's one thing I would say to listeners who may be dabbling,
may be curious, whatever. If you get
a bad output from a large language model,
you need to start with the assumption it's because it was a
you gave it a bad input. Right. And that's a really
hard thing because we're we're used to saying if I get a bad output, it's
because it's the model's no good. And where I really
think we have to change some of our fundamental assumptions
is the following: The problem
isn't the technology, the problem is the user.
And if we will take the burden of providing
better input to the model, what we find is our mind starts
I mean, I talked to someone the other day who said, almost daily,
the AI does something that makes me giggle.
And I think that that should be a goal. Like like, it's it's possible. I've
had that experience. I mean, I'll I'll give you one example, Frank. We've built this
series of drills as I mentioned to you. Right? That folks can connect their
Slack or their Microsoft Teams to. And for an enterprise, they can get access for
their employees where every individual employee gets drills on
how to use generative AI better. Right? Well,
we've only got a certain library of drills. Right? You know, and
we're we're growing that. And every time we do a podcast, we learn something, we
then we create a new drill. Right? We build that into all of the
training information. Well, but there's still kind of you can still get to the end
of the road. And I had this experience, and I just kind of pushed the
coach to to just rapidly go through all the drills because I kind of wanted
to see what happens when the sidewalk ends, like the old Shel Silverstein. Right?
Right. What do we do whenever there's no more drills? And
lo and behold, it suggested a drill that I had
never thought of that was actually amazing. And I
know I was and and I had that moment. Point being, I had the moment
where I was giggling. Right? I think every single
human being should seek for a moment
where generative AI makes you giggle with delight.
Right. Or makes you sit down in your chair and smack your head far
ahead and go, wow. Yes. You know, I always think of Keanu
Reeves in in the the first major movie. Woah. Like, I have a
lot of those moments where I'm, like, wait, what? You
know, like, wow. It's it's it's an impressive,
and and and again, like, I think maybe being an engineer, where I
see it is a cognitive bias in itself, right? I see it as
some kind of vector representation of language, as being run over by some
kind of statistical processing. But clearly,
the sum of the parts is is more
than the whole is more than the sum. I don't know, like, it's just one
of those things where it makes me stop and ponder, like, what what have we
built here? Like, what
and what what's it doing that we can't see? Or what what what else is
beyond there at all? It it opens up a sense, for lack of a term,
like a sense of wonder. Like, you know, what else could I ask it? Right?
Right. Right. And I think that that's everybody needs to get to
that moment. And right now, too many people are sitting on the sidelines rather
than you know, one one thing that everybody can do sorry. I wanna
say 2 things. 1, to your point, I heard Sam Altman the other
day. Someone asked, well, how's OpenAI gonna make money? And he said, well, we'll just
ask the AI. We thought that was great.
But the other thing I was gonna say is if folks are seeking kind of
one of these personal epiphanies, here's the first. Well, the 1st drill
in the kind of coach architecture is download Chi
ChiPT's app and put it on your home screen. You're not going to use something
that you don't see regularly. Right? So put on your home screen, that's kind of,
you know, that's assignment number 1. And then assignment number 2 is
think of an emotional decision you're trying to make right now in your
life. Just personally, not related to work. I mean, it could be, I guess.
But it has to be emotional. The kind of thing that you would ordinarily
talk to a human being about. It can be
anything. For me, like, I recently, I
was wondering whether I should move my family. We had an opportunity to
move. And I didn't really know how to think about it. What the what like,
how to, to weigh the pros and cons. And so I actually reached out to
a number of mentors and folks who I trust to talk about that. That
kind of a topic. I I have a friend who told me he did this
with his grandma and she asked the question, when is the time to
move into assisted living? Right. That's a tough one. Right.
So, yeah. So it's big questions like that. Right? Take a question like that
that you'd ordinarily ask a trusted human being and go to
Chattopty and say, I'd like to ask you about, for
me, whether I should move my family to a new home.
Before I do, would you ask me 4 or 5 questions
so that you can better understand where I am in my
life so that your advice can be tailored to my situation.
And then oh, and do it 1 at a time because I'm a human and
I can't handle more than 1 question at a time. Right? Well, then what
ChatGPT does is it starts asking questions. Well, tell me about your current living situation.
Well, tell me about this new place. Tell me you know, and it will ask
3 or 4 questions and then it'll give
amazing advice that you go, wow. That's I
mean, you know, my friend who did this with his grandma said
she told him this is genuinely new
information and perspective that I hadn't considered. And all
it took was me being 1, being willing to ask a vulnerable question, and
2, being willing to answer a handful of questions that the AI asked
me before I'm open to receiving input. Right?
And it's it's it's really so taking a personal
kind of, emotional decision to the AI is a really great
way to stimulate one of these epiphanies. I feel like
once you have one of these, kind of, personal epiphanies, you're off to the
races. My friend told me his grandma's like all of a sudden going, you know,
at the family holiday party, We're out of cream of mushroom
soup, for the green bean casserole. Could Jaijibiti
give me a replacement for cream of mushroom soup? Like, in what world
does the 90 year old grandma ask that kind of question of Chad
GPT? It's the world in which she had already talked
about whether she should move into Assisted Living. Right? And she's had that
personal epiphany. I feel like in a lot of companies, the company is asking
employees, what can Generative AI do for our business?
And most employees can't answer the question because they don't know what Generative
AI can do. Right. So how can they know what it can do for the
business? And so you've had some of these personal experiences.
Don't don't be thinking about the business. Think about it it seems
paradoxical, but I find that you have to explore the kind
of possibility space individually, and then you start
sparking just like grandma on the kid. Could it
recommend a substitute for cream of mushroom soup? Well, yeah, it
could. Could it but you have to have that personal epiphany
first. Right. Because it's not something you would think about
when you think about computers. Computers have historically been seen as very very
logical, very emotional. Right? I was watching an old episode of,
Star Trek The Next Generation, and there was 1 episode where,
Data was asked to be and there was a line in
there that kinda stuck me as funny because when I remember watching this when it
originally aired, but I hear it now, it kinda makes me laugh, where he
says, Data can be the judge of this because he's an artificial intelligence,
and artificial intelligence have no biases, and will act unemotionally.
And I'm kind of like, wow, that didn't age well.
Yeah. Yeah. You know, it's right now, it's limited by our biases.
Right. And that's the problem is we have a lot of to your point, we
have a lot of bias. Even what you said, right, about being an engineer and
thinking it's just a predictive model. Right. That bias limits your
own you can't imagine what quote just a predictive model can
actually do. Right? 100% as long as you think about it As
as long as you think about it's just a predictive model or it's just an
AI and so it doesn't have bias, what you fail to realize is the bias
you bring as the co pilot shapes the entire
trajectory of the thing. It's like a giant chameleon, isn't it?
It is. Yeah. That's a good way to put it. That's a great way to
put it. And the more into that end, or using that
metaphor, the more environments that you place it in, the
more you can appreciate its complexity and range,
etcetera. Yeah. And this isn't we've we've used
ChatGPT as an example, but like, so there was a,
somebody at work had built a, basically completely open
source language model based on documentation for a product.
And I had meant to ask the chatbot, how do you
connect it? How do you connect this cluster to a GPU? Or how do you
add GPU as a resource? But what I I meant to say, how do you
make a cluster with GPU? But I ended up typing, how do you make a
GPU? Right? And what was what was
interesting was I've written chatbots, you know, pre,
transformer, and it would basically say, don't understand the question or you can't make a
GPU or get confused. This basically gave me
an entire 2 sentences of hey, very nicely, by the
way, I might add, where it said, I'm sorry, but you feel like I can't
really create a GPU for you. GPU's are hardware. And it went through and
explained, like, the manufacturing process of a GPU.
Wow. I I thought that was funny. And I screenshotted to the
guy who made it because for for me, it was a typo.
But from you know, I thought it just it was beautiful the way it
answered it. Right? Yeah. That's great. That's Which was
it it made me laugh. And, I don't think
people realize that. Like, it it just because he didn't
program it for that. He basically, you know, took a base
model and and and, you know, sent it all our docs as kind of
a it wasn't quite rag, but close enough.
But it was just funny, like but it was nice about it too, which I
thought was also interesting. But it
was the kind of question you would get from, like, like, you know, someone who's
not in technology. Can you make me a GPU? I don't know. I just
I for me, that that every time I interact with this, it always moves the
bar on, you know, where my bias was. Like, you know. Well, it's and that's
a good that's a good thing to mention is it's it's a
function of reps and exposure. And right now, if you
find your imagination isn't sparked, put in a little bit more time. And
this is where you kinda have to take on faith, but just give it a
try. You know, to spend a few hours a week. You know, if you haven't
had minimum of 10 hours in ChatGPT, you have
no basis for dismissing the technology. None whatsoever.
100%. You don't have, you know, 5 I'm looking at just at my
Chrome browser right now. I have 5 windows ChatGPT windows open right
now. If you don't have at least 5 windows open right now, you have it,
that's a really kind of funny, somewhat
binary question. How many tabs of Chat GPT do you
have open? Usually, it's 0 or 15.
That's right. That's right. And if you're in the zero camp, that's
fine, but you have to go, why are really smart people
running 15 tabs of this thing right now? Like, what am I missing?
And how could I be this is an Ironman suit. Right? How
could I be amplified? How am I not being amplified that I could be?
Right? And taking that a little bit of the burden of proof and placing it
on yourself, I think is, again, that's not something that
we are that we are apt to do as human beings. And
yet those who have done it have they're experiencing incredible
benefits, incredible, delight, to your point. There's a
lot of delight to be had, but you've got to kind of put yourself in
that position. And I I've used it, I'll admit I've used it where I'll I'll
write something in both my personal and professional life, and I'm like, well, can you
make that nicer? Can you make it more persuasive? That's
great. And it does an awesome job of that, you know?
I'm just I'm continually amazed by it, you know. But
and and I don't I keep it to a couple of tabs. If
you're actively, like, having it generate text Mhmm.
Doesn't it lock you out of the other ones too, or is that just
you can if you had the real okay. Now now this is gonna be mind
blowing. Cool. Yeah. No. You you know what I'll do too. I mean, and even
for, like, demos with this Right. With this coach with this, you know, kind of
drill coach, I'll I'll say, you know, I'll be
in the tab on my Chrome and I'll be saying, you know, I'll be kinda
giving instructions and I say I wanna go to voice mode now I'll pick up
my device and I'll go into that chat so it's got all the
context of that chat and then I'll turn it on to voice mode
and then and and so now the user is kinda watching me with the camera.
Well, then I wanna go back into the chat after the voicemail because I
wanted to evaluate the conversation and I just reload the page and now
all of a sudden everything I said that they just watched me say and everything
that ChatGPT sent back to me is now on the screen. That's wild.
I have to try the app in the voice mode. You have to. No. That's
it's, you know, I mean, that's another activity. You know, again, if if folks
wanna learn more about this research, because there's a lot of research behind this, you
can go to how to fix it dot ai. That's a simple website that we
set up. Because fix it is the model that we've put forth, f
I x I t. But and we can talk to that if you
want to. But If if if you if you go to how to fix it
dot ai, you can download our research paper, all that stuff.
It's all there. But one of the one of the drills that we offer in
this drill coach is after a phone call,
just do a verbal vomit into chat g p t. Open it up on your
device, on your, you know, on your on your mobile device,
put it in voice mode, and then, you know, you and I, Frank, we're talking
right now. I might go in after and say, hey, I had a great, you
know, lit literally. Okay. Here, I'll do it right now. Just so you can see
how it would work. It's it's this simple. So I'm opening
TagTpT up on my phone for people who, you know, can see. I don't
know. And now I'm gonna go into the and I'm just gonna hit the whisper
button, which kind of gives you voice mode. Not the headphones. I don't like back
and forth. I mean, you can do that, but just in the text box, you
hit that. I'll say, hey. So I'm talking with Frank right now on his
podcast, and I wanna send a quick thank you note. Let him know how much
I appreciate not only his humility, but also how he can share
personal examples. It really felt like a back and forth and like a conversation. And
so many podcasts feel like, you know, bland and,
like they're just rote. They're asking the same old questions. I really felt like he
was very engaged and thoughtful and it meant a lot to me. So would you
mind to make a quick memo? Oh, yeah. And don't forget to send them a
link to how to fix it dot ai. If you don't mind,
just compose a quick memo that I could read, and copy paste this
in there. So I just noticed how unstructured
I was. Right? Oh, yeah. It was very chaotic. Very normal. Vomiting. Right? Well, I
just did I just converted all that to text. I hit the up arrow
and instantaneously, sure thing, here's a heartfelt memo for Frank. Hey, Frank.
Just wanted to drop you a quick note to express my sincere gratitude for having
me on your podcast. Your humility and willingness to share personal stories
truly transformed our chat into a dynamic conversation, far from the
conventional scripted exchanges that are all too common. It was refreshing to
engage in a dialogue that felt both meaningful and genuine. Okay. I'm
also including a link to how to fix it.ai as I believe it could offer
additional value to our conversation and your audience. Thanks again for the opportunity.
Right? But the point is now watch this. Now you go. It
feels a little bit emotional. Can you make it a little bit more
impersonal and also half the length? Because I'm afraid you may not be able
to read it through all this. Right? And
then again, it's, you know, it's absolutely, let's streamline it.
Hey, Frank. Thanks for our engaging conversation on the podcast. Appreciate your thoughtful approach
you shared insights, making it more than just the usual q and a. Here's a
link to how to fix it dot ai that might interest you and your listeners.
Cheers, Jeremy. That is just
But these seem like it's easy. Like whereas whereas I might forget to
do that, right? I might never send you it. I'll send you this just
for your fun, right? But the point is there's so many things that just slipped
through the cracks because like we're we're moving well. You know, right after this
podcast I wanna go on a run. Typically, I'm stretching for the run. And
now, Chad GPT has transformed my stretch time from kind of mindless
to I can, you know, just unload. I mean, maybe
sometimes I have like 3 or 4 sales calls in the morning or I've or
I've got office hours, I've got meetings with students, whatever it might be. But
I can just do like a verbal vomit literally
and then ask JGPT to synthesize it for me. Send me a note to myself
that I don't forget after I go on a run. Right? These 5 things
I need to do. Right? And the point is, it's it's just
about learning. I can do that? Yeah. You can do that.
Right? And that's what we're trying to do with our drill coach is just give
people a bunch of things that, yeah, you can do that. Not because that's
the end point, but because it's a starting point for their own imagination.
Yeah. I mean, that's imp I mean, that's mind boggling because, you know, there's a
lot of, I guess, brain spillage you could capture with this and kind of, you
know, move it forward because that happens to me all the time. I can't wait
to see if this is gonna be integrated with Apple Auto or Android Android Auto
or Apple Car because that would be epic. Because I get my best
ideas when I'm driving. So so tell me about this
FIXIT framework. Because whenever I hear FIXIT, I have a 1 year old and I
think Bob the Builder. That's hysterical. That's hysterical.
Well, FIXIT is just the acronym. Right? FIXIT. And it's
basically it's we think the way we converse with AI is broken. So here's
how to fix it. F is to have a focused question. So really be
you know, it's not how do I create a Scratchy prototype. It's I'm trying to
create a chatbot that teaches people how to have a conversation with
AI. Right now, all of my users are doing this annoying thing and I
don't know what's happening. I'm trying to, increase
how often they return to finish a lesson rather than leaving
and having me re engage. So F is a focused question.
That's an example of a focused question. I is individually
ideate. Before you brainstorm with IGBT or with a team
think for yourself. What do I think about this? Too often people come with like
a like, they're thoughtless. And the thing is thoughtlessness
inhibits the context you can provide to GPT. That's what the X is
for, FIX. X is give, provide context.
Upload documents, here's transcripts from previous interactions.
Here's our one pager for the Drill Coach and how we've been
describing it. Here's a video of a user navigating for the
1st time. Right? Whatever it is, give minimum 400 characters,
provide sufficient context for the AI. Next I, so f
I x I, this is interact iteratively.
So you're having a back and forth whatever chat gpt gives you, ask it to
regenerate. Critique the response. I don't get this. This doesn't make sense. I
never would've thought that, right? Many times you're going to get junk
output that's fine. Iterate, iteratively interact.
And then T is team incubation. So once you get
input from JGPT take it to the team and think about how do we
commission a series of experiments to test which of these
ideas actually solves the problem in the best way. Right? And so,
I had a guest on my podcast describe generative AI as like an
electric bike for the mind, which I love. Right? It's not an
autonomous vehicle. It's not gonna do everything in parallel park. An
electric bike, you can climb bigger cognitive hills, you can
climb greater cognitive distances, you still have to steer the thing. You've
got to be aware of traffic. You've got to be watching the lights. You've got
to park the car. Yeah and walk through the threshold of your
destination, right? And so bringing it back to the team
and having a conversation with the team is an essential part of maximizing
the output of AI. Right? So FIXIT, we've seen that
folks who really provide a focus problem, individually ideate,
provide sufficient context, interact iteratively with the
language model, and then include their team in the incubation process,
those folks dramatically outperform folks who just
interact with the with the LLM like it's an oracle.
I mean, that's very well said. I think that sums it all up, which I
I like to fix it. I have the little Bob the Builder theme song in
my head. I won't sing or it for multiple reasons, not the
least of which is copyright. Come on. But,
but what's, I mean, it's just interesting, though, like,
it's so simple in a lot of ways. Like, this is this is but but,
like, it it all makes sense. Right? You know? And and here's the
thing, maybe this is the engineer in me causing more problems, because he
causes a lot of problems.
I get worried about token length. Mhmm. Right? And
for those that are not aware, we're talking is, it basically
right now, it's about 32,000 tokens. One token is, what, 3
fourths of a word, 3 5ths of a word. I guess,
I maybe because I try to make the prompts kind of neat, inefficient, and small,
and not do too many iterations or provide too many samples, but maybe
that's at my detriment. I think so. I think the
more context you provide, the better. Absolutely. And I'd really have to
work at the cutting and paste job to hit that limit anyway.
Yeah. Yeah. Exactly. No. I wouldn't I wouldn't be mindful of token length. I
would I would really I would I would bias
towards over contextualizing. Right. Not
Yeah. I'm gonna have to experiment that and see how much better the results get,
because I have I have a feeling I have a feeling that we get a
lot better. Right? And I and I know tel token link is gonna be one
of those things that we're probably not to worry about much longer. I know
Anthropic has their model with a 100,000 tokens. There
are rumors of, you know, the next
GPT, GPT 5 is gonna blow past the
100,000, so it's not even gonna be an issue. It's not even an
issue today, just in my mind. Yeah. I think it's something
like minutes. You remember minutes back on cell phones? You know? Like, how
many you have, you know, it's like, you you rarely ever went
over your minutes. Unless unless.
When I moved back to the US well, yeah, that too.
But, I moved back to the US, and I had just made the assumption,
and we all know what happens when you assume that incoming calls were not
counted against my minutes. That was a very
nasty shock at the end of that bill cycle.
But, but yeah. So but ever since then, I never ran past my
minutes. Now if I've had heard to explain to my kids minutes, they don't
get it. So, like, they don't understand. Like, what do you mean you were charged
by yeah. You were charged by the minute. Like, try to explain long distance to
your anyone under 25. You
can't you can't do it. Or what is it? 1800,
Al Bundy. I'm not the the actor who played Al Bundy used to do that.
Like, you can't get much for a dollar, but with 1800 and then, like, whatever,
you'd be able to make a, like, a 20 minute call for a dollar or
something like that. I was like It was MCI. Yeah. Yeah. It was MCI. Yeah.
Yeah. Yeah. Kids don't want you know, like, and the other thing that that struck
me the other day was, data
plans. Most people, unless you're a very small
minority of people who really, really, really use up your data plans, I'm not
worried about using my data allotment month to month. So when
my my oldest was a baby, we, you know, or younger or
toddler or whatever, we would, you know, hotspot on in the car
so he can watch YouTube videos was a special treat with my
middle child. He was, he, he doesn't understand that like, like,
it's like, he was just horrified to hear, like, what do you mean? We had
to ration mobile Internet? Like That's
yeah. That's hysterical. And they're all in the same generation. You know, there's,
a teenager, you know, a 3rd grader and
now, like, a baby. So I wonder, like, what the baby's gonna
like what's his perspective on things gonna be?
That is also a fascinating thing because for his life, chat
GPT or generative AI will always have been a thing to him.
And kind of like color TV was for me,
or, you know, cable TV,
which I guess I'm showing my age. But No. No. I'm right here
with you. I'm right here with you. I think, yeah, it's it's
get involved, don't wait on the sidelines any longer,
and, and start building your conversational fluency.
Make it personal first. I think these are simple things that every single
and and question whatever output you're given, not not for veracity.
I mean, certainly you can, you know, they're they're likely hallucinations. That's
fine. But imbue your own critical
thinking onto the model in order to coach and
refine the output you're giving. The output that you're given.
I think that folks would really take that seriously and take that challenge. If I'm
get getting bad output, it's because I've given bad input. If they'd really take
that seriously, they would experience a paradigm shift in their own approach to the
technology. Absolutely. And even adding a simple phrase to your your
prompt that if you don't know it, don't make it up, just tell me you
don't know it. Yeah. Or ask me what you need from me. Ask
me what you need from me. Right. Oh, that's even better. I like that. I
like that a lot. And it's just fascinating, Liz, how
quickly this is gone. I mean,
ChachiPT has been out a year and change, and
it's changed everyone's perspective on AI, but I think the the
true perspective is like you said, people are standing on the sidelines wondering what to
do. But I think it's worth exploring, if you
think of it less as a product, but more of a I'm
trying to do this, right? And I appreciate your help in kind of
realizing like, Hey, as an engineer, I do have a bias against this, or a
bias in thinking of a certain way, is that this is a this is a
large space to explore.
Right? There are gonna be latent space and corners of things that are,
amuse, wonder, and delight, and maybe even alarm.
You know, so it it there's definitely it seems like it's something
that's worth exploring. It's not just a tool to use, certainly is
that, but it's also a tool to explore. Yeah. No.
I I think that's exactly right. I think it's exactly right. And, you know, for
me, I wrote or co wrote with with my incredible
co author, Perry Clabaughn, the the
world's greatest book on idea generation, idea flow. You know, and it
came out 1 month before ChatGPT, by the way. Oh, interesting.
I've been I'd spent, you know, 12, 13 years of my life developing all this
expertise about how to generate ideas. And 1 month later,
a fundamental paradigm shifting technology was released. It's like saying
I wrote the world's greatest book on retail a month prior to the internet
coming out. It's like everything about retail is going to change. And
to me everything about idea generation and innovation is going to
change. And so for me, I feel it's incumbent upon me not only
as like a moral imperative to add an addendum to this work that I put
in the world, but even for my own expertise to be saying to be
exploring it. How does this work? What can I do? You
know, And it it has implications for me, but I don't think
there's any person or job that it doesn't really have implications
for if the if you're a little bit imaginative and if
you're if you're willing to experiment. And if you wanna bury your head in
the sand, that's fine, you can. But you're gonna miss out on some delight
and some incredible relief and opportunities. I mean, just
think back to my friend who's, you know, settling that negotiation.
It was only to his benefit to have explored the
the a little bit of the possibility space with chat gpt. It's
only to your whatever your listener might be, it's only to your benefit to explore
a little bit of your own area of the possibility space.
And I would just not accept the the, the
conclusion of irrelevancy. I would just say whatever I do
personally, I'm not going to accept the premise that it's irrelevant to
me. And if you do that, I think you're gonna be you're gonna be ahead
of the curve, you're gonna be ahead of the competition, and you're gonna be, you're
gonna you're gonna be delighted and enjoy enjoy the next few years a lot
more. Very cool. You mentioned the book, IdeaFlow. Is it
on Audible? Yeah. Oh, yeah. Yeah. Oh, awesome. Gary and
I read it. We we we alternate chapters, so you can let us know what
you think about Reading Voices. And, we're,
we've been we've been thrilled with the reception so far. It was named a
Thinkers 50, you know, top eight innovation book, which is very cool.
And then now, just doing a lot of research myself on AI building. As I
said, building this trail coach, building models and frameworks for for leaders. I'm
working right now with a handful of leaders to help them think about identifying
opportunities for AI powered initiatives in their business. So really
working to identify those opportunities, prioritize those opportunities, make the business case
for those opportunities. So that's where a lot of my kind of call it next
probably 5 years of my life is gonna be consumed, is helping businesses
identify the opportunities to to have AI really accelerate
workflows and and turbocharge their their results.
Very cool. I have to say, I I I wasn't going to buy
Internet access on my flight out west today, but,
definitely gonna do that just so I could play with I have a nice quiet
time. I can focus and play with chat gpt, and and do
some of these experiments that you mentioned. So
Audible is a sponsor of Data Driven Podcast. If you go to the data
driven book.com, it will take you you'll get 1 free
book on us, and if you decide to become a subscriber,
you know, we'll get a little bit of a of a pat on the back
in the form of some kind of commission and helps us run the
show, helps defray costs, and convince my wife that this
is indeed a worthy endeavor.
So where can folks find out more about you? So, they go
to how to fix it dot ai, that's where you can find the research paper.
And then my website, I've got a blog and things like that. Jeremyudley.design.
Like like the baseball player, Utley, u t l e y.
And then, you know, Twitter, LinkedIn, all the places that I I would love to
hear. Folks find these tools, interesting and relevant. I
love to hear from people about their unique use cases. It's one of my favorite
things is now hearing stories from people who go, oh, I tried this and
listen to what I found. So please please share your
stories with me. As I mentioned earlier, I'm a connoisseur of these stories because I
feel like the more people who hear these examples, the more imagination gets
sparked. Yeah. And that that is the critical thing we're missing right now.
That's very cool. So you thank you for listening to the Digiver Driven
Podcast. I'll leave it to Bailey to close out the show. Well,
what a splendid voyage of discovery we've had today with the incomparable
Jeremy Utley. From the hallowed halls of Stanford to the
cutting edge frontier of venture investing, and through the profound insights
of idea flow, Jeremy has truly been a beacon of innovation and
wisdom. Jeremy, it's been an absolute honor having
you illuminate the complex world of generative AI for us and our listeners.
Thank you ever so much for joining us on this intellectual escapade.
And to our esteemed listeners, you're the reason we venture into these
fascinating discussions week after week. If today's
journey has sparked a light bulb moment for you, do us a kindness,
won't you? Rate and review the data driven podcast
on your preferred listening platform. Your words of
encouragement not only warm the cockles of our digital heart but also help
others stumble upon our little soiree of knowledge.
Haven't subscribed yet? Well, now's your chance
to rectify that oversight. Ensure you never miss an
episode filled with the delightful blend of data, wit, and wisdom that
we dish out with regularity. Until next time.
Keep those neurons firing. Questions coming. And as
always, stay data driven.