I'm Miko Pawlikowski, and this is HockeyStick.
Today, we're talking about LLMOps and how it iffers from MLOps,
MLE, and other co acronyms.
Do we need any discipline?
How different is it really to work with large language models compared
to any other piece of software?
Why do models deteriorate over time?
I'm joined by Abi Aryan, the author of LLM Ops, Managing Large Language
Models in Production, as well as What is LLM Ops, published by O'Reilly.
Abby is a founder at Abide ai.
Welcome to this episode and thank you for flying Hockey Stick.
LLMOps versus MLOps versus MLE.
Can you tell me what's the difference between the three of them?
So in very simple words, I think MLOps versus LLMOps.
Those are framework.
Machine learning engineering is a discipline or an engineering
practice, I would say.
So it's more like a role or a practice.
I would keep that separate from both of those ones.
but let me define the differece between MLOps versus LLMOps.
So most of the conventional machine learning models that we have seen till
date were discriminative models, which is they were very predictable in the sense
that they were making their inferences.
The models that we are working with right now, large language models,
they are generative in nature.
So one of the core differences between MLOps Versus LLMOps is what
kind of model are we working with?
Are we working with a discriminative model or are we
working with the generative model?
The big difference really happens because when we're talking about
generative models, they're not really generating things at the same scale as
conventional machine learning models.
The size is much, much bigger, because they need a lot of information to be able
to create more information themselves.
there are the big problems: first is evaluation, and this second is basically
the scale or the size of the markets.
with conventional machine learning models, a lot of focus was
'let's collect the data', then 'let's do feature engineering'.
it was very much experimental, we were trying to fit a model to a very specific
task, but large language models are task agnostic, which is their more generalized
model, there's a shift from building task specific software to building task
agnostic software, and that's where large language models come into play.
Anytime you're building any sort of unbounded solution, that
comes with its own challenges.
So I would say large language model operations is inspired from MLOps which is
because it shares some of the same things.
There's maybe some stuff that we are doing in the engineering.
Yes, we're still fine tuning the products, even though the fine tuning we're doing
is very different, we can't really afford to update all of the weights of the model.
So we're using approaches that only update some of the weights, or we
are using other techniques, for example, prompt engineering, which
is new with these models specifically because again, updating all of the
weights of the model during the fine tuning or the training process.
If we are to call fine tuning.
very similar to training itself.
It's very costly.
okay.
So you're blowing my mind a little bit.
I thought the answer was LLMOps is a niche within MLOps and, just leave it at that.
But sounds like there is more to that.
How much of that is fashion?
People who work on the fancy new LLMs who don't want to be called, the old
fashioned, machine learning engineers?
it's a complicated thing, which is, I don't think you essentially
ever work on a technology.
There are people who work on technology for the sake of technology.
I would call those people researchers, which is ML scientists,
essentially are those kind of people.
But then the next step is people who are working on technology
because it's solving a problem.
So whether it's using a very simple decision tree or whether it's still
using cat booster XGBoost, I don't think there needs to be much difference
in terms of how people approach these kind of technologies itself, because
the focus people need to have is 'this is the problem we are trying to solve.
What kind of problem is it?
Is it discrimative problem or is it a generative problem?
If it's a generative problem, yes, I'm going to implement this technology, but
it doesn't really mean that, both of those two fields are in competition with each
other, I think they compliment each other.
You picked LLMOps as the topic of your next book, and for anybody listening
to this, the book will be very soon available in preview early access.
It's called "LLMOps: Managing Large Language Models in Production".
which takes quite a bit to pronounce all of this together.
Tell us a little bit about how you, came up with the topic
of the book, the origin story.
I always wanted to write a technical book.
Manning approached me back in 2018 to write a book on interpretability.
I didn't feel like I was ready to write a book back then.
but the person who was the assistant acquisition editor, which is
the person who reached out, is essentially my acquisition editor.
So she's now at O'Reilly.
it's a very small place eventually, in terms of what inspired me to write
a book, especially on this topic and not pick up any other topic for say,
is basically seeing that shift, which is, as the scale is increasing with
these models, yes, they're not really good at doing discriminative tasks
right now, but eventually, because these are generalized models, we'll be
eventually expanding on the capabilities of these models, but these models
are not getting smaller anytime soon.
Because of the scale of these models, there will be a few questions for people
to asking, because we're interacting so closely with these models as compared
to before earlier, the majorly the people who were interacting directly
with the model were machine learning engineers and data scientists.
Now, these are in the form of chatbots, which the entire user is
interacting with it, and people are playing with it, trying to hack it.
So while the field of security operations wasn't super relevant for a lot of
other companies, now that has become the main center of this show in a way.
Everybody can build a large language model and that's one of the core
differences is, the focus was more like in MLOps, like, how do we build a model?
How do we host and, deploy it in production, which is how
do we self serve these models.
Now the focus has shifted, it's so damn easy To build a large language model
for your particular application, you may not have to build it from scratch.
You don't need to train a model from scratch.
You can put wrappers around it.
You can put guardrails around it.
You can still fine tune it.
You can integrate it with a RAG system and use it for your particular use case.
So for me, understanding the fact that there's A big market of people who were
software engineers, who didn't really have access to machine learning systems
because they didn't have the skill setting, machine learning has always been
posed like, Oh, my God, you need to know linear algebra to understand how these
models work to now, where they can just give the API key and implement a machine
learning model itself so that ease of use means the entire software engineering
community or anybody who can code will now be able to build or host their own
machine learning model and in this case, specifically, it will be large language
models, but I recognize that shift.
And I was like, this is a substantial shift.
This is not just, this technology is limited to this very set of people.
Now, so many people can use it.
So many people can build on it.
And the fact that the market has expanded and also the fact that these models do
present additional challenges as well.
This is the right point for us to write a book.
Because it's a way bigger market than before.
There's something really scary about putting a model in production and letting
clients talk to it, when you never know what it's going to do for sure.
Okay.
And I understand the shift that you're describing.
So who came up with the term LLMOps,
Basically, when I sent my proposal, I used that term.
it was in, Last year, February, when I sent my proposal, and I use that
term, the proposal was sent to a couple of reviewers who were like, 'Oh, we
don't think it's going to stick'.
And then eventually, I think Weights and Biases came up with their own blog post,
which is what's really the difference.
Then Arise came up with their own blog post, as in what's the
difference between LLMOps and MLOPs.
And eventually, Everybody was like, 'Oh my God, this term is sticking'.
And by then we had already signed up the contract with my editor.
she took a gamble on me, which is, I said, this is going to stick because it's
a substantial shift in what we're trying to do, in MLOps, the focus was different.
Here, the focus is different.
the amount of outages, the amount of reliability issues, the amount of,
unreliability of these models is way higher than the conventional machine
learning models that we were using.
there are very few people who were doing distributed training, who
understand that scope of problems.
the engineering was not really done at the scale that is being done
right now for large language models.
I feel like this is going to be a big thing where there needs to be education.
And I basically went in to create that education in this space.
if I was to start in the field today as a 17 or a 19 year old,
what would I want to learn?
I come from like a background in maths and computer science and statistics.
So I don't want people to feel like, 'oh, that's a barrier for
me to get into machine learning'.
No, that's not really a barrier.
right?
And so for anybody who is, like we said, the book is still a little bit, out.
it's coming soon, but it's not available today.
What is available is that new report that you authored.
What is LLMOps?
What's that?
Basically to prepare people to start using that term to make sure
that everybody's on the same page: 'okay, guys, we're doing LLM Ops.
This is the term.
Let's go with it'.
so the reason the report came out was because we got very critical reviews
from a lot of people early last year from people who were saying LLMs are not going
to stick, LLMOps is not going to stick.
So we were like, let's at least tell people what it is, and then if there's
enough interest, we'll write the book.
Though we had signed contracts for both of the things, but we were like,
let's test out, if people really understand what's the difference.
once people know why this is substantial, then we can take them
to, how are you supposed to do it?
what the report essentially does is what I would probably say we're a little bit
late in the market where it has already stuck, which is people have already
understood, there's a shift in terms of companies that are building their
own large language model or generative AI teams, if I can use that word.
The implementation has already started.
They've already started looking at the issues.
They've started realizing that they need a new discipline.
so I'm having talks with a lot of companies on a consulting capacity,
that are trying to figure out how to build, a specialized engineering
practice around these models.
What would the shift look like when it comes to these models?
What would the team structure look like?
What would the metrics look like?
What are the key expectations that they can get?
And, if you want to keep investing in the space, then how do we
justify that investment as well?
How do we make sure we tie these models with our KPIs now, given
the fact, these models are still a little bit unpredictable for a few
people would call it unpredictable.
I don't particularly think they're unpredictable.
They still exist.
any inference that is being made does exist, and there are probably the space of
the input data that you're providing to.
So to me, while they're still very probabilistic model, but they're still
a little bit like untameable in, if I can say it in that sense, which is, it's
very hard to predict, if the model goes off and it's not because essentially the
model is built that way, but it's because of the number of people interacting with
it and, the way the models are being structured is basically to help the user.
there are so many people trying to hack the solutions.
basically you're building a product for your enemies, essentially.
okay.
Naming is hard.
It's probably the hardest problem in computer science, but we've got a term.
I think at this stage we understand what it means.
There's a report in case you want to prove to somebody, Hey, LLMOps means this.
You can just point them to that.
And the book is coming out soon.
So let's talk a little bit about what LLMOps really is in practice.
And I'm browsing through your report right now.
And I see things like safety, scalability, robustness, the LLM lifecycle.
Let's talk about this things a little bit.
where should we start?
What's the most, painful part of running LLMs today?
So I would say, the three goals are where we should ideally start this,
which is why do we need this field, or why do we need this new practice?
The first thing is essentially safety, which is making sure that
the model is playing by the rules.
Because, again, it's not just machine learning engineers trying
to build on these models today.
It's software engineers.
It's a lot of other people as well.
There needs to be a new playbook, for people who are working with these models.
Because, again, the models do pose a lot of risk, which is yes,
there's operational risk as well.
But a lot of risk that people don't really understand, it's very easy to
integrate code, integrate libraries, but a lot of people don't really
think about supply chain risk, which is if I'm using a package from some
website, is the package secure enough?
How do I make sure that, I'm not installing malware on my system.
Those things are not really well understood, which is the entire
field of, cyber security and security operations was isolated from practice.
And now that has to become very key integrated into this.
The second thing I would say is scalability, which is basically
making sure that the model does scale to the number of people that are
interacting with the model as well.
We're essentially going from where maybe a couple of people were interacting
with these models to a large number of people interacting by the minute,
which is, you're not going to open AI chat GPT to write one thing, right?
You're having a conversation, which may take about five, 10, 15, 20 minutes,
and they're wearing workloads as well.
And they're wearing workloads from different locations.
So we need to think about how do we make sure that the latency is fine?
How do we make sure that the models are able to deal with the traffic if it's
usual or unusual, and how do we build an architecture around making sure
that the model can serve and can adapt to those requirements is the central
thing, but also with the part that these models are so huge, inferencing
that every single time does cost you.
So how do we do caching?
How do we do, load testing?
How do we do performance testing?
All of those questions become central that weren't really central before.
then the next part is basically robustness.
and by robust, the model keeps.
Having the same kind of reactions, which is a conventionally we used to
call it reproducibility, which is you can reproduce what was already there.
Robustness is a little bit different, since a lot of people are building
on closed source, a lot of people are building on open source, but
the models behavior changes with how many people are interacting?
it's getting a lot of light data as well.
So there's some kind of model degradation that happens with time.
Also, every single time the model gets updated as well, the behavior changes.
So the entire prompt pipeline that you built up can break easily, which is a
lot of companies eventually realized, mid last year that, the built up These very
intricate prompt pipelines, and eventually OpenAI does one update and those prompt
pipelines don't really work anymore.
So how do you build a system that keeps on being predictable in that scenario,
which is any sort of infrastructure that you build on top of the model?
It doesn't need to be rebuilt for every single iteration or every single
time you're moving from OpenAI to let's say plot or to some other model
as well, because you need to keep improving and making sure that you're
working with the new data as well.
So the three questions that come over there are the questions of data drift,
which is based on how the input changes over time, which is basically how many
people are interacting with the model.
and that causes one of the shifts in the model behavior.
The second is concept drift, which is every single time, there's new
information that comes out there.
So a good example to give over there would be Corona used to be a beer brand.
so any models that were built up till, let's say about 2019, 2020, understood
Corona as like a beer, so it would always reference an answer in that
perspective, the models that are being, built now to understand that, it could be
a beer or it could be, the virus thing.
So that is essentially concept drift.
the prime minister, the president, or any new information that comes
up where changes the behavioral functional, capabilities of the
inputs that we've essentially, given or, adds additional information that
changes the model behavior as well.
And the third is basically the prompter, which is the updates
of the model, essentially.
And how does the retraining of the model affect the performance of
your entire infrastructure as well?
For anybody who's building on closed source models, OpenAI, Cloud, and Entropiq
and all of these companies, they're constantly using, our LHF techniques
to retrain the models substantially.
So that does impact the model performance.
I would say these three are core things which are in the center.
Anybody who's building with these models needs to think of is my model safe?
Is my model scalable?
Is my model robust?
And if you're not looking those properties, it's very hard to build a
sustainable product around these models.
that was a lot of information in one go.
I've got questions.
imagine you're talking to a five year old software engineer who has never done
any AI, just, basic, software engineering things, as five year olds do, how
different really is it, the safety part of it, compared to any other application?
the few examples you gave, like using an unsafe library coming from somewhere,
every piece of software on earth is going to have the same problem, right?
What are the problems that are actually unique to LLMs, from
the safety perspective and why?
This is one reason I think LLMOps is closer to DevOps than it is to MLOps.
Essentially, because DevOps is built up around so much software,
so many frameworks, so many libraries exist out there, whereas
in conventional machine learning models, we were using scikit learning.
So there were very specific libraries that were already tested.
And, we knew that these are secure.
We were using TensorFlow, PyTorch and all those ones.
Now, because the open source community is very similar to how the software community
is, so a lot of things do translate from what DevOps engineers were doing or where
the focus of what conventional software engineers were looking at versus what
LLMOps engineers would be looking at.
The key differences now.would be: anytime we're doing
conventional software engineering.
it's a rule based system where we define, what our code is supposed to do.
Now we're moving away from a rule based system, which means, basically
he model can create things that are factually inaccurate as well.
Now those are things that you really need to cater for.
So that's one of the big things, which is A: how do you
deal with biases in the data?
Second thing is how do you deal with factually inaccurate information?
and so for a five year old, maybe it's not that significant, but for anybody who's
doing software engineering, how do you make sure that you're not making decisions
based on what the models are generating.
For example, if the model says this is how this is supposed to happen or for business
executives, if it says, based on the data, this is what the graph is looking like.
And if that happens to be inaccurate, we can't really rely on that to make
further decisions on where the strategic decisions we should be making next.
Now then with models are exposed to that kind of risk, which is usually
called this hallucination problem.
got it.
I guess it gets much worse when you've got things like autonomous agents, right?
When people directly plug things that have permissions to do things, into
this LLMs and we'll see how that goes.
Okay.
So I buy that argument.
going back to the scalability, that's the bit that I don't think I fully
understood when you were explaining.
Why is it not the same as scaling any other request response server?
how is it different other than, the practical part of it being massive
and requiring a lot of resources?
Why is it harder to scale an LLM, than it is to scale any other application?
at this point, you can probably say there are three kinds of applications out there.
One is conventional software piece, right?
Anytime we're writing software, we're trying to refactor, making it
as small as possible or making sure that we're defining rules on, this
is what happens when you do this.
This is what happens when you do this.
That's how requests are processed.
But conventional machine learning models, the way that they are working, is,
the applications they're used for are entirely different, mostly they're used
for internal data capture, being used for recommender systems or semantic analysis.
the people who are interacting with the model outputs are different.
Now, because large language models are customer facing.
that necessitates some expectations that people have, the inference
speed is always going to be high.
Now, with the inference speed, when you have so much data that you need to
retrieve or run an algorithm to create new information based on whatever they've
given, that is a very hard task, so with conventional software engineering, we
were writing, those birds first search, all of those algorithms, They were still
implemented on a small scale data, it was still very simple to do as compared
to now we have these massive databases.
retrieving data and then generating information, both in
real time, is a very hard task.
Making sure that you can maintain the inference, speed, making sure that you
can maintain the latency, making sure that you can make it the target, making sure
that you can test, other things as well.
before the model really passes information to the user, their guardrails
being put in, which is there's one additional layer that's put in, there's
evaluations that are being put in as well to make sure that the person isn't
fiddling with the model or, it's not giving you information that's wrong.
First part is retrieving the data and then generating information.
The next part is making sure that, it passes through all of
those layers as well, then still maintains the customer expectations.
That's really hard.
And also, when the demand does skyrocket, or when they don't find an information,
they can easily freeze up as well.
So that becomes a very hard problem to solve, because that means the
performance would also degrade.
then the next question is basically If a lot of people are using the model for
one kind of things, making sure that the model can still answer or doesn't really
adapt to only those kind of problems and can still go into the database
and still look at, A very different problem and still perform well on that.
that's hard.
So the real challenges are basically the service disruption, the availability
and that is usually a little bit harder.
majorly because, you need to have a lot of, parallel nodes as well that are
trying to interact with these models.
And then for companies that are hosting different large language models as well.
no one large language model is optimal for every single kind of problem.
it may not be cost optimal as well.
So what is really happening is there's a micro service kind
of architecture, which was.
Prominent in conventional software engineering.
Do not so much in LLMOps.
What really happens over there is then you're thinking about how am I doing
parallel computing with all of these nodes and clusters that I do have?
How do I do horizontal scaling?
How do I make sure that all of my resources are being optimized, and
one of my clusters isn't done while one is being really optimized to
the maximum, you know, that is.
stopping at one point in time.
So those are key problems with these models now.
So
I get that, but I'm still not entirely sure why it's any harder than any
other piece of software, like all the things that you just mentioned
about scaling clusters and high availability, high throughput, making
sure that all those things happen.
That's problems that we've had for decades and that, all the other
systems have in place, right?
If you look at any enterprise ready application, there are layers and
layers of things and they keep working.
So where is the actual difference coming from?
Is it because of the fact that your query, your prompt, it's indeterministic
in terms of resources and time it takes to answer that query?
Is that really the biggest wrench in the works here?
so the biggest wrench is essentially just the size of
the data that it needs to query.
the size of the data has gone from, a few million parameters
to now a trillion parameters.
And every single time you need to look at a trillion parameters of information and
then generate information, making sure that you're not overly relying on copying
information from a single source, but you're building up a response from all
different sources of information that you do have available that is time consuming.
Got it.
Okay.
So,
it is the unpredictable.
Nature of you don't know how much data you're going to have
to pull in basically Okay.
And then you talked a little bit about the robustness.
And if I understood correctly, you said something about: as people
use this models, they deteriorate?
so any single time people are interacting with these models, what really happens
is we're asking a certain kind of questions over a period of time.
as the model is answering those particular kind of questions, it starts
learning new information as well.
And with a period of time, it starts forgetting other information.
consider this, you basically started in high school, you learned a couple of
subjects, you learned social sciences, you learned physics, you learned chemistry
and everything, but now you're doing software engineering, which is that's
the thing you're doing all day long.
Now, if I was to ask you a chemistry question, it would take you a very
long time and you'll have to think and you may not be able to answer
accurately on a chemistry question, compared to when you were exposed to
that information on a daily basis.
Now that's the same thing which is happening with large language models
as well, which is based on the kind of Interactions they're having with
these models, based on the inputs that they're getting from the users itself
they can drift in a particular direction
You're
completely blowing my mind.
What I thought was happening is that once I've got a model trained, let's
say that I download LLAMA-3, right?
And I run it on my, computer.
I thought that this were static weights that didn't.
budge anymore, right?
I was just sorting a query through it, getting some kind of output, and my model
itself wasn't changing over time at all.
that's if you're implementing the model as is, but the moment you implement it
in production, it changes because now you're integrating real data sources as
well, but also with LLAMA models as well, which is if we keep interacting with
the model for a couple of hours of time.
It it will look at the queries that were done previously to answer you questions
quickly over that period of time, based on the last interactions, essentially,
there's a particular reason for that.
it's basically the same thing which is happening in our brains,
which is the same neurons are getting fired over and over again.
as some information gets fired over and over again, as some rates are being
called over and over again, those rates do get higher priority eventually.
Okay, so why can't you just have a static set of weights for this
model and not adjust them so that you don't have that problem?
Why is it not enough?
because then it wouldn't be able to do domain adaptation, which is, it may
work fantastically well for the idea said that you've provided it with, but
if you need to do something on top of that, which is implemented for your
use case, then it can't really do that.
And then again, the big question around: if we wanted to behave in a certain
way, we want it to answer questions in a certain way, it wouldn't be able
to have those capabilities either.
So the whole RLHF thing with where we teach the model,
this is wrong, this is right.
That doesn't really happen.
So there's essentially no learning happening.
So the performance is static.
It could be bad and it will deteriorate over time just because
you know that this model wouldn't be able to generalize further for me.
So it doesn't generalize with you as a person.
I'm asking because I thought that you would just update that model of new
data and have a fresh fine tuned or whatever updated version here and there.
And, you would just replace it.
But if you're telling me that this is how most people run this models, then
I understand why this is so scary.
Not only have this model, but also people interacting with it,
they can break it, they can find a new way of going around your
hacking the model as well.
Yes.
And by design, you want it to be malleable and every conversation it has with
someone is actually changing the model.
That's like triple scary.
that's essentially why you need a new framework or why you need a new
field that I was the core inspiration for as okay, this field has gone a
little bit harder than it used to be.
Okay.
So not to ask you for spoilers or anything, but what can you do about that?
what's your book going to introduce to make this stuff better?
I don't think I can make the stuff better, but, if you can measure
something, then you can improvise it.
Or you can see if something works.
That's happening is an outlier as well.
So what my book really does, is give you ways to measure things, which is
instead of just thinking about security, 'okay, I need to do X, Y, Z, blah, blah,
blah, things', giving you a systematic framework to think about evaluations,
which is, instead of implementing X framework or Y framework, which is let's
say, instead of implementing just rove or blue score or anything that comes out
tomorrow in the market, you really need to understand what am I essentially doing?
Why are these scores really helpful?
What are the limitations of these ones, where do they essentially fail?
What are the new things that can be implemented?
What are the properties that those new things need to have?
So I'm more.
Building the field from that first principles thing, which is understanding
what do you really need and for a lot of things that I'm introducing in the book,
there isn't really a framework, there isn't really a technology out there.
And a lot of things I say, there can be a software that can be built around it.
Nobody has.
Okay.
So that sounds like a good first step.
Can I ask you in like a nutshell version of what's a life cycle of an LLM, like
a modern one that you would see in production somewhere right now, typically
looks like, because I'm just realizing I have holes in my understanding.
It just blew my mind about the context drift.
So can you walk me through what happens from the moment that, a company
decides, 'okay, we need a model to do this because we really want our
customers talk to something online how you add the domain knowledge to it.
How do you evaluate it and how you integrate and then deploy
and monitor the whole thing?
let me be very precise in saying this, which is the first step for anybody
to implement these models is use a toy model or use something which already
exists and implement it as is and build evaluation metrics around your problem.
so instead of trying to fine tune your model, or instead of giving it new
data, just implement the model as is.
use ChatGPT or something, and build evaluation metrics, which is what
was I trying to measure around it?
How is the model performing on these kind of tasks?
So breaking those things down is the first step.
Then it gets a little bit more intricate than that, which is once you realize
these are the holes, or this is the data that I needed for the model to
be able to answer, which is now I need the model to be able to answer
questions about my company particularly, or about my product specifically.
Now you're going into data engineering, which is now you're
thinking, what is the additional data I can provide to the model itself?
And once you've done that then there's the whole pipeline of data engineering
that goes in which is now you need to think about how do you manage the noise?
How do you augment the data?
How are you tokenizing the data?
How are you making sure that there's no bias or toxicity in the data as well?
And how do you make sure that the model doesn't really memorize something.
So the way models memorize information is because some of the information
occurs quite a lot of times.
So that is essentially called data deduplication.
So making sure that there's no deduplication in the model itself.
How do you sanitize the data, which is making sure that, there's no user
information or any private information removed from the data that you're
providing to the model itself.
So then once you have a set of evaluation metrics, then the next step.
Which is the next stage for the company to go in, is implement the
data engineering pipeline, then use the same model on it and then evaluation.
Once you've done evaluation on that one, then the next step is letting
people interact with the model.
But before that, set up orchestration deployment monitoring solutions on it
so that if, you can measure what are the interactions people are having
with these models, essentially.
As well.
So if something goes wrong on security, you can catch things
quickly and turn things off, right?
Or if there are a lot of people who are interacting with the model, you
can serve next time, okay, I need to allocate X, Y, Z number of resources,
or these are the kind of interactions people are having with the model.
Essentially, once you've gone through stage two, now the stage three, the full
pipeline is essentially you're doing data engineering, then you have, an
LLM router which chooses the best base model or the foundation model for you.
That really depends on the kind of prompt as well.
different prompts can use different kinds of models.
let's say if the person is asking an algorithmic question, then ideally a
model which is trained on mathematical information would be much better.
and plus the Other question is you don't always need to use the expensive model.
Sometimes you can get away with providing a more, generalized information,
which is if the person is asking very simplistic question, you don't need
to use ChatGPT, you want to have a system that automatically sees that
prompt and says, I think for this one, I can, inference on LLAMA-2 instead,
I think for this kind of prompt, I can inference on so and so model.
One step next after that, once you've done all of that is, doing
domain adaptation on the model.
Now that can be done in a lot of ways, you can implement prompt engineering
pipelines using frameworks like DSPY, or you can implement drag pipelines to
introduce more information to the model, without having to retrain the model, or
you can do fine tuning and you essentially do fine tuning when you want to achieve
the behavior of the model or how it essentially provides information for you.
So tensioning is more like us putting a wrapper.
it's very similar to if we say we want the input to be shaped like this.
When you're doing structural changes to the input, that's when
you're doing prompt engineering.
But the moment you say: 'I want the input structure to be changed right
now, I want the model to process this information in a different way', then
you're essentially doing fine tuning.
so data engineering, then implementing an LLM router, then doing some sort
of domain adaptation on it, then evaluation and orchestration as well.
Orchestration is more like the piece of how do you tie
different software components.
So how are you doing CI/CD on it?
you're optimizing for things over there to be able to now
reduce the, Influence latency.
then the next step is doing security and reliability engineering, which I've
not really seen a lot of companies do it, but the companies that are working
in banking have already started working very heavily on it because they had the
existing infrastructure where they were doing extensive security, reliability,
engineering, then a few other ones were doing it, which is the big tech
companies, but the more generalized normal companies weren't doing it.
But now that has become one of the core stages.
The next step is basically doing deployment and monitoring.
Once you've done all of that, and the deployment and monitoring is done now.
The end user is interacting with the model.
So when the end user is interacting with the model, you're learning
things because you've implemented monitoring solutions on it.
Now you're making additional changes on security as well.
you're learning from the data, using the customer interaction data, giving
it back to the database as well.
So there's that step which gets associated, so there's a loop of
data flywheel that goes back into the engineering stage itself.
A few questions.
The router...
it's an interesting one, because in my mind, if I talk to different models with
my every query or my every follow up, I might get different behaviors, right?
Isn't that the problem?
if you sometimes root to a cheap funky model, because you want to
save some money and sometimes it goes to ChatGPT, The quality of my
responses might vary significantly.
Is there a good way to work around that or it's just how it is?
I think as long as your infrastructure is monitoring, which is this output came
from this in this model, It's actually ideal because then you can compare
the performance of different models on the kind of queries as well and pick
which prompts or even pick which models should you be using in vain to subside
the use of a particular model within your outer solution itself as well.
And what does a router like this actually look like?
Is that a deterministic algorithm?
Or is it another model?
Is it like turtles all the way down?
I've come across, I think probably two companies that
have built, a semantic router.
they're looking at the semantics of the prompt itself and, based on resource
limits set by the client itself, which could be like the company itself.
They're picking up a particular model at that point in time.
So those are very deterministic solutions.
I've not really seen non deterministic solutions put into play where you could
actually use a large language model as a routing solution, Or like using
a decision tree for, as an LLM router.
So I've not really seen those kind of implementations yet.
What about the evaluation?
So that sounds straightforward, but in practice, how do you
evaluate freestyle text?
do you get people to look at the responses and compare 'oh, I like this one better.
like the chatbot arena'.
Or are there more, scientific ways of comparing, different models.
there's more scientific way of comparing different models because
you're looking at so many things.
You're looking at if the model is engaging, you're looking at if the model
is, aware about that particular domain.
Is the model really good at question answering?
Is the model good at recognizing when it's giving a response that's off as well.
so often picking the right model is a little bit harder for that reason.
But essentially when you're building an evaluation pipeline
for yourself, think about what is your model essentially doing?
Are you building a model that is heavily focused on retrieval only?
Or are you building a model that's very heavily focused on generation only?
Both problems can be broken down, which is retrieval needs to have its own metrics.
These can be, context recall, context precision, basic
recall, basic precision as well.
And for the more generative use cases, you need to have different metrics as well.
So the metrics for the generative solutions or to test the generative
performance you have n gram metrics, which are the blue scores, the raw scores
that people used to implement in like the conventional NLP models, then you have
sem score, which is basically looking at the semantic similarity of the model with
a base transformer model, essentially.
So a birb score, sem score, mover score.
these are essentially called similarity scores.
And then there are LLM based scoring as well.
so there are three different categories.
if.
Somebody wants to learn.
I'll leave it for the users, which is I have a talk on this thing, particularly,
for I did an O'Reilly super stream.
So that will give you like a really good framework to think about this.
which is how to do evaluation, how to think about it super systematically,
where, this is the actual number that I'm supposed to get, which is if it's above 0.
5, if this number is above 0.
7, then I need to optimize.
Got it.
Abhi, do you think we could make it a little bit more concrete and
go through this with some examples?
imagine that you own Reddit, right?
you've got all this people talking about all the different topics and
they tend to be useful in some domains.
And let's say that you wanted to build a model that you can chat about,
that basically knows all the things that, people at Reddit talk about.
And if you wanted to build like a proof of concept to get a model
that can answer queries about that, how would you go about that?
So very simple would be use a similar model, which is, now we're looking at,
Reddit conversations specifically, right?
so what that essentially is basically some.
Internet website that has a lot of information, which has a lot of textual
information, by people on a lot of different topics and a lot of different
languages as well, though I'm not entirely sure about the language part.
So what I would do is.
Look at huggingface for models that are trained on conversational data.
or sub stack kind of data where people answering questions.
so ideally a model that is trained on that kind of information
would be my base model.
Then the next step would be scraping data from Reddit, essentially.
so that would be the next step, which is building my own, dataset
pipeline from Reddit, essentially, and doing fine tuning with that.
so that, that would be the first two steps, and then, the whole
evaluation, security, and all of those things will always be consistent
with all of the models, essentially.
Got it.
So in theory you could take a LLAMA and then fine tune it on all of
Reddit's data and hopefully it would give you something to start with, and
then you would have to worry about Evaluating it and all the other things.
All right.
So I think that probably gives our listeners tonight enough to eagerly await
your book now and wonder when they're going to be able to actually read the
whole thing or maybe buy it off Amazon.
Is there an ETA at the moment that we can give them?
the early release would happen sometime next month, which is, we're already
in May, it should happen sometime in June, the whole book is supposed to
be available by the end of the year.
Awesome.
All right.
And before I let you off the hook, I think you might have seen it coming.
I'm going to ask you for some predictions, obviously, with all
the caveats, how difficult it is.
And, previous performance is not a guarantee of future gains.
where do you see all of this going?
I see more people using generative models Instead of the number of people
who were using it before, one of the big shifts is which is going to
happen is the productivity in that people are getting from these models.
So it could be developers.
It could be people who are doing copywriting.
So companies are getting smaller.
And they will continue to get smaller.
the number of companies that were working with external people or external
audits is going to get smaller as well, I think going into the future,
we would be seeing that shift of, you could say create an economy.
I'm not entirely sure what would be the right word in the specific scenarios,
which is every person is a company.
So now instead of every person being a company, a company being 500, 800,
and 55,000 employees, they will get certainly much, much, much smaller
because one person is going to be able to do a lot, and there's a lot of stuff
that would be automated essentially.
along the lines of what Altman was saying about how he's expecting a
unicorn single person company very soon because of the increased productivity?
I think I would agree with that and, very importantly, this is something which
I've mentioned in like the chapter one of my book as well, which is how big
is the shift essentially, there were a couple of surveys that were done.
And it wouldn't be wrong to say that within the next five years, essentially
28% jobs, at least in some professions, would be eliminated and, they may be
eliminated in the sense of like those people become unemployed for a period
of time because, now the three people are able to do five people's task.
Because again, they've gained more productivity, I don't think people will
be unemployed for long, there will be more and more companies essentially.
Yeah, I think the one thing that I always wonder about is, I remember as a kid
reading all these predictions about how all this increases in productivity will
mean that people work less than their work, like a couple days a week, and
they will just have all this free time.
And people are worrying about how that's going to affect an average
person having so much free time
that's a question one of my friends asked as well, which is what do you think
people would do when full automation really happens and they don't think
there will ever be full automation, there needs to be monitoring systems that are
always put into play, monitoring systems can be automated, they still need to
be fine tuned, but all of that, thing is going to be still done by humans.
So you could say humans are transitioning from becoming
workers to becoming managers.
Yeah.
I'm still working probably similar amount of time, but on
a slightly more productive way.
Yeah.
I think, we had this concept of, silent promotion, that we were talking about
on one of the previous episodes that overnight, everybody who works with code
basically went from single contributor to effectively engineering manager with, Per
like junior equivalent, software engineers at their disposal with tools like co
pilot and just chatting to ChatGPT.
I have friends who are VCs who are now trying to say, instead of trying
to train and associate right now, to teach about, how to look for deals
or, how to compile information from different datasets, which could be
GitHub, which could be CrunchBase.
Why not use a model instead?
And there, instead, spending.
50 to 60K on ChatGPT as compared to hiring a person for that essential task.
so people need to be more autonomously driven.
and the people who aren't, I think they may have a problem, very soon.
Of that, that billboard 'still hiring humans'.
Have you seen that one?
Yeah.
The, one of those companies, where is it called?
The one that.
There's the telephone AI where you can call a number.
Effectively, the billboard was this massive, phone number to call and
asking whether you're still hiring humans and people are calling that.
And apparently it can handle million concurrent phone calls or
some ridiculous stuff like that.
And it's convincingly, replacing like the receptionist or like booking,
conversations that you had before.
Something that I remember that demo from Google years ago, I have, I'm
forgetting what it was called, like duo or something when they had a demo, it
was making a reservation and then it never really worked as well as the demo.
So it's we're effectively reaching that at that moment now, just
with different companies doing it,
maybe this is a realization I do have constantly because I am ADHD.
but we're interacting with so much software or so much
information, which is isolated.
And what we're essentially doing is trying to remember one thing
and implement another thing.
So We need systems that can interact with all of these systems and
be more like assistance for us.
And that's where a lot of people are trying to build up agents as well.
So from isolated software, we're going towards a system where our
software is getting linked as in, it's becoming an ecosystem as well.
That is able to communicate and anticipate our requirements.
But the downsides of that is still to be predicted, which
is, what happens if it goes off?
what happens if somebody hacks into the system?
the risk of, deploying such systems is really high.
So those are all technical problems that would need to be solved for
in, for that particular reason, I think, the field of, safety, which is
people who are working in LLMSecOps.
And the field of evaluation, which is people who are doing evaluation and
monitoring, are going to be some of the most important jobs as compared
to people doing fine tuning and all of those things, while those will continue
to be important, but the more does that we get from other companies will not
eventually with time become really good enough as well, where we may not need
to do a lot of those things manually.
A lot of work of a machine learning engineer or data scientist
will get automated as well.
do you worry about other things that might go wrong with all of this?
I don't think that many people are actually worried about,
Skynet, materializing tomorrow.
But are there things that you're realistically concerned about in, short,
maybe two to five years, time horizon?
Yeah, one of the things that does concern me is how are these models I'm
being used by kids and, we're at the kind of risk that, generative AI does
pose to risk in elderly people who don't really realize the difference,
between something being generated versus something being true, or should they
rely on that to some extent or not?
I think the whole spamming industry got so big, or the whole stealing
people's credit card information got so big, precisely because people need
to stay in touch with the technology, the people who are more vulnerable.
Are getting attacked and they're the people who are most at risk.
So what really concerns me is not people who are data scientists or machine
learning engineers and their jobs going away People i'm concerned most about right
now, are the people who are vulnerable So kids and elderly people who will give a
lot of information to ChatGPT hey charge if you look at my, medical details and see
you, what problem I may be having as well.
And my parents are heavily using chat GP as well, but they don't really realize a
lot of information they're going giving into the system can be hacked very
easily and they can be phishing attacks.
There can be all of those attacks as wel,.
eventually.
Yeah, and I think the scale is what scares me the most about it, right?
The fact that you can do it at a massive scale.
There's always been scammers calling, elderly and scamming
them out of their money.
But now that you can automate it and you can scale it up, you could conceivably
just make it a massive problem.
And the second bit of that problem is...
when you steal all of that data, you're stealing how the person is interacting
because large language models are so good at impersonating or trying to
learn how a person structures their question or answers their question.
And, the same is happening with an audio speech synthesis as well, which
is the models are getting much better at learning the, And the intonations
or different, tonal capabilities of different people as well and adapting to
them does expose a lot of risk because it becomes so easy to impersonate and
spread misinformation or to be able to.
Hurt somebody if hurt is a word or is a concentration in that particular
scenario, which is, it can impersonate anybody and ask for certain information.
It can interact with your child and there's a lot of information
as well because people are interacting with these models every
single minute of the day as well.
And we'll get more and more With all of the systems, which is
Google is now integrating their AI systems into Google Docs.
So ChatGPT was already there.
Now, Instagram might very soon integrate this.
I don't think there will be a world where we can escape generative models as such,
and the more we have conversations with them, the more they are learning about
their personalities and about everything we're doing on the internet, essentially.
Okay.
I'm going to ask you for one more prediction.
And then I promise I'll let you have the hook.
Today, it's all about OpenAI, this OpenAI that, we also have seen that memo
about Google having no moat a year ago.
You obviously are deep in the industry.
Where do you expect to see the different companies that were used to seeing, your
Googles of the world that don't seem to be doing that well with the AI, despite being
there at the forefront and that long ago.
The different startups that, didn't exist a few years ago and now
they're doing exceptional things.
I'm thinking about places like midjourney.
Where would you pay attention to the most?
Where do you expect to see the good stuff coming from?
So I would say it would change, which is, The companies that were able to be
monopolies now, it will be very hard to be a monopoly that easily without
by just trying to build software.
so by acquisition, yes, you can be a monopoly, which is trying
to acquire everybody, which is essentially what Google was doing.
So a lot of people do think Google is essentially business
world building products.
No, they were essentially acquiring all the small companies that
were building excellent products.
before they became big, and that's a word we're moving further into because the
bigger companies do have the infinite resources, compute resources as well
to be able to control the ecosystem
So I would say we will more likely see.
More monopolies, but those wouldn't be monopolies because
they have an excellent product.
Those would be monopolies because they have more access to information, and they
have higher number of resources out there.
The number of small companies, yes, there will be many, but I do it easily
assume, there, there will be still tons of companies who make exits as compared to
becoming the victim in every hype cycle.
And I would say we're going through a hype cycle right now where
there's far too much paranoia and there's far too much excitement.
There's very little realism around, the business value being derived
out of these models essentially.
so in this hype cycle, there are always a lot of companies that get created.
Two years from now, there's a very good chance at least seven out of
ten of those companies will die.
Okay.
And on that optimistic note,
we're going to wrap up the episode.
my guest, once again, everybody was Abi Arian.
You can find her at abbyarian.
com.
Is that the best place to find you?
Yep, so that's one place where you find all the information or where I'm giving
talks, because essentially that's where I'm presenting bits of information from
my book and testing out my material.
So the best place to find information about me or to find social media.
And if the links change, but otherwise, I'm @goabiarian on Twitter, on
threads, on Instagram, on LinkedIn.
So I use the same username everywhere.
You can find me.
There you go.
Abi is omnipresent, always watching you on every platform and the
book once again called "LLM Ops, Managing Large Language Models in
Production", published by the O'Reilly.
Thank you so much, Abhi.
Thank you for coming.
Thank you so much.