Welcome to Data Driven, where we dive into the thrilling world of data,
AI, and on occasion, misbehaving chatbots suggesting
glue for your pizza. This episode features Bar Moses,
CEO of Monte Carlo. Not the casino, not the car,
but the company keeping your data from quietly wrecking your business.
We talk observability, the chaos of unreliable data,
and why one tiny schema change cost a company
$100,000,000. Ouch. So buckle
up. Because if your AI bots are making decisions without
reliable data, well, hope you like eating rocks for the
minerals. Hello, and
welcome back to Data Driven, the podcast where we explore the emergent
fields of data science, artificial intelligence, and, of course, data
engineering. And with me today is my favorite data engineer in the
world, Andy Leonard. How's it going, Andy? It's going well, Frank.
How are you? I'm doing well. I'm doing well. I was in Raleigh last
week, drove down, rented a car actually,
to save mileage on, on ours, and,
spoiled because it's been a while since I bought a new car. And
this is the second time I rented a car, and I'm getting tempted. I ain't
getting tempted. It was a Chevy. It was
a Chevy Malibu. Not a Monte not a Monte Carlo.
See what I did there? I don't even know if they still make them. I
I was driving, the little one off and dropping the little one off at daycare,
and I was behind a Chevy Monte Carlo, like, a two early
two thousands vintage. But that is actually quite relevant
to our discussion today because with us today, we have Bar Moses, who is the
CEO and cofounder of Monte Carlo, the data
and AI reliability company, not the casino
or the car, I would assume, or the town. Monte Carlo
is the creator of the industry's first end to end data and
AI, observability platform with
$236,000,000 in funding from Accel
Iconic Growth and others. They are on a mission to bring
trustworthy and reliable data and AI, to
companies everywhere. The company was recently recognized as
a enterprise tech 30 company, a CRN
emerging vendor, and an inc.com,
best workplace and accounts Fox, Roche,
Nasdaq, and PagerDuty, among others, as their customers. Welcome
to the show, Bar. Thank you so much. Great to be here, Frank and
Andy. Awesome. An intro. No problem. Do you drive a
Monte Carlo? Because that would be epic. You know, I really should
be driving a Monte Carlo. I do not, and I've never actually been to
Monte Carlo either. So I will tell you if you're into cars,
like, I'm like a recovering car, nerd. Oh,
very cool. It looks like a car show. Like, honestly, I went to Monte
Carlo, and we had rented, like, a Saab convertible. And I felt like we were
driving. We were driving driving, like, the low end
of the car thing. I mean, there were I mean, I've never
seen Bentleys in the wild, like, just parked on the street,
like, no big deal. Wow. Like, I mean, every
luxury car if you're in a Saab and you feel like you're slumming it
Yeah. It is clearly a high money area.
But, so welcome to the show. So Monte Carlo
why'd you get the name? I I'm assuming it might have something to do with
Monte Carlo simulations, but that's in the Great question. Yeah. The
unofficial story is that, one of our CO, founders is a fan
of formula one and, you know, as, you know, formula one crisis.
So right. That's, you know, clearly the, the, that's the
unofficial story. The official story is that, you know, we
had to we had to name the company. We started working with customers when we
started the company, and we we had to choose some name.
And, I studied math and stats in college, and so I sort
of opened my my stats book and sort of looked through and,
you know, reviewed my option and, you know, Markov,
chains didn't seem like a great name. And next up was
Bayes' theorem, which was similarly kind of not great. And
and then, you know, I was reminded of Monte Carlo and Monte Carlo simulations. I
actually I actually did some work with Monte Carlo simulations earlier in my career.
And it seemed like it seemed like a great name, a name that would speak
to, you know, data engineers, data analysts, folks that have been the space.
And, you know, I think naming a company is a very difficult
thing to do. We decided to go with it. And the spirit of Monte Carlo,
One of our values is ship and iterate. And so, the
name has sort of stuck with us since. And, it's quite memorable. People either
love it or hate it. So I think it works for us. I think it
it works. Like, I think of the car. I think of the casinos. It has
a certain amount of, high class, maybe more so than Markov
chains, Markov chains. Although I did for a time flirt with the
idea of of also starting a company called Markoff Chains, but,
like, have see if we could see if we can get money for mister t
to be the spokesman. That would
have been epic. Yeah. Jeez. He did you. Ideas, Fran. I was the
only one I was the only one that thought that was a good idea, but,
you know, I was a big fan of mister t as a kid. Marketing. Yeah.
That's funny. That's what I do in my day job now. Oh, yeah.
I swear, folks, I didn't pay her to say that.
So so you you talk about data and I AI
reliability. And to me, when when I hear that,
a slew of things come to mind. Like, there's security, there's the
veracity, like, the five v's and all that or four v's or whatever it
was. What exactly is kind of Monte Carlo's, like,
wheelhouse there? Yeah. Great question. I'll
actually sort of anchor ourselves in in kind of the metaphor or sort of a
corollary that we like to use here, which is really based on software engineering.
So we didn't reinvent the wheel when we say data and AI observability.
We really take concepts that work for engineering and adapt them.
So, you know, when we started the company, the idea, the
hypothesis, the the thesis that we started the company on was data
was going to be as important to businesses as applications, as online
applications. And, they were data was going to
drive the most critical sort of, you know, lifeblood of companies through
decision making, internal products, external products.
And, while software engineers had all the solutions and tools in the
world to make sure their applications were reliable, and so some, you
know, some off the shelf solutions like Datadog, New Relic, Splunk might be
familiar to you, data teams were flying wide. So there was literally
nothing that they could use to know that their data was
actually accurate and trusted. That's sort of, like, the the problem the core problem that
we started. Fast forward to today, you know, we created the data observability
category. We're continuing to create it. AI is making this problem just
infinitely bigger, harder, more important. Why? Because
data and AI products are now you know, there's a proliferation of those.
An AI application is only as good as the data that's powering it,
and the AI application itself can be inaccurate, can be
unreliable. Right? And so at a very high level
I know this is, you know, very vague, but at a very high
level, the idea was the same diligence that we treat software
applications, we should be treating for data and AI applications. Now,
what does that actually mean? How do we do that? Enter the concept of
observability. Observability is basically understanding or
assessing a system's health based on its output.
And so basically, the thesis was, can we observe end to end the
data and AI estate, learn what the patterns
are in the in the data, bring together metadata and context,
lineage, for example, about the data, derive insights
based on that to understand and determine what the system should
behave like, and alert if that gets violated. So that's sort
of the first part. The first is actually being being able to help data teams
detect issues. The second part is actually being help,
helping data teams resolve issues. Now here's the interesting thing
that we sort of learned over over the years. We've worked with hundreds of of
enterprises. So, you know, we mentioned a few. We real really work with the top
companies in every single industry. So,
you know, in in, in health care, in retail,
in manufacturing, in, technology, in each of these
areas, the, data in the state
obviously varies, but there are actually interestingly commonalities. And the
commonalities is that every single issue can be
traced back to a problem with the data, problem with the code,
problem with the system, or problem with the model output. Can go
into detail into more each of those, but that's sort of the high level,
framework. We basically provide end to end coverage to help data teams
understand what the issues are and help them trace them back to data issues,
code issues, system issues, or model output issues. So when did
you get the idea that I'm sorry, Andy. I cut you off. Okay. When
did you get the idea when you realized that data is gonna be as important
as applications are to businesses? Oh, great question.
Yeah. Great question. So so we started the company in 2019.
And, actually, what's interesting, it was pretty clear to us then, but we
had to prove that or we had to convince that of people. Definitely.
Yeah. It was not obvious. It's it's still there's still a
lot of people that are kind of, like, I guess, they'd be in the quadrant
of laggards where they realize, oh, I guess this is important.
A %. I would imagine in 2019, that would have
been you would have sounded insane. Like We we sound I
sounded insane a %. People are like, what? Data is
gonna be important? Are you sure? Now a couple of things happened
since, which I think helped. First is,
there were some large acquisitions in the data space, like Tableau and
Looker earlier on, and then Snowflake IPO'd. Snowflake was the
largest software IPO of all times. It was quite interesting that the
largest software IPO of all time is a data company. So I think those
things sort of help kind of convince that this you know,
convince, at least, externally, you know,
to the market that data will continue to be will will be
important and critical. I think the things that I noticed is, you know,
before we even started the company, we spoke to hundreds of data leaders, and I
speak to dozens of data leaders every single month. They continue
and I think what you hear from them is more and more
data teams and software engineering teams are building products hand in hand.
So they're actually they're side by side building. Right? And so, actually,
almost more and more critical business
applications, revenue generating products are based off of
data, and they're being powered by data. I'm not even talking
about generative AI, which is a whole whole other story why that matters, but just
data products by itself. Think about reports that people look at internally.
You know, just give you an example. You know, we work with with, many
airlines, for example. Airlines have a lot of data that goes to internal
operations. Like, what's the connecting flight? What's your flight number? How
many flights left today? What time did they leave? How many passengers were on
the airplane? Where is your luggage? Right? That
information is powering internal and external products. You know, it's powering the application
that you're using in order to onboard the the plane, in order to connect
to your next flight. If that data is inaccurate, like,
you're screwed. Right? And that hurts tremendously. Your brand
is an as an airline, your reputation, it leads to
reduced revenue, increased regulatory risk that you're putting
yourself. Right? So so the data,
what we see from our customers is powering critical use cases like
airlines. I'll give you another example. You know, we work with a,
you know, a Fortune 500 company, perhaps your your favorite cereal.
I don't know if you're you guys are big cereal. I I, like, eat cereal
for breakfast, lunch, and and dinner. It's, like, my go to.
You'd be surprised into how much data optimization, machine learning,
and AI goes into actually optimizing the number and
location of cereal on the shelf. So there's a lot of
data that goes into supply chain management to make sure that you're
actually, like, fulfilling the right warehouse,
demands on time and, you know, making sure that everyone gets
their serial on time. There's actually a lot of data that goes into all of
that. So I think what gave me conviction was in speaking with
so many companies across so many industries, data was
actually allowing data teams, allowing
organizations to build better products, to build more
personalized products, and to make better decisions about the organization.
So I think that really sort of made it clear that the future was going
to be based on on data. Well, I I like that
you pointed out, the importance of observability.
My career path winding as it was,
I made a a leap from being a software developer to being
a data really a database developer. When I made that
transition, one of the things I had noticed, this was two two and a half
decades ago, I had just started in software development
doing test driven development and it had just
come out, it was called fail first development. I remember thinking
this was perfect. It was a big deal. Yeah. It was. Yeah. Twenty five
years ago. And I remember thinking this is perfect because I'm always failing.
So this this will work nothing ever runs the first time and if it does,
it's suspect. But when I got over into data, I had just
become, you know, kind of a a big believer in the power
and and and really the the confidence that
test driven development gave me. And I was like, we need that
over here. And so it was, just a
field that's fascinating me. I have an engineering background, and so it kind of flowed
right through. Instrumenting the data engineering,
was a big deal so that, again, you could achieve what we now call
observability. But being able to watch that data flow
and when I would mention this to people kinda like you in 2019, I
I would get all sorts of responses. Most of them kinda raised
eyebrows. And I would, some of the more interesting ones
were things along the lines of, well, the data is sort of self
documenting. I mean, it's it's just there. And I'm
like, no. No. It's not. It's I especially when you've moved it through
a bunch of transformation to put it into a business intelligence solution or data
warehouse or or any of that. And that now feeds,
you know, modern LLMs, AI, and and the like, those
same sorts of, I guess, old school processes, I
do. Or at least that's my my understanding. Maybe I'm reading too much into
that, but I love the idea of having observability go
all the way through. You mentioned lineage. That's huge. You wanna make sure that when
you, you know, you make this one change, that's not gonna affect anything
else. Usually, it does affect other things, and having
that lineage view is huge. That is spot on.
That's exactly how we've we've thought about this as well. So, you know, I
think there are specific things that you can test for in data. Like, for
example, you know, specific thing that you can declare, you can say, like,
you know, you know, a T shirt
size should only be, you know, small, medium, large, extra large, whatever.
Right? But then there are some specific things that, you
know, you you don't necessarily know. Like, for example, if there's a particular,
you know, pattern that the data is being updated,
you can actually use machine learning to automatically learn that pattern and then forecast
when it should get up updated again. So it's not necessary for someone to
manually write a test for that. Right? And so
I actually think it's a combination of both of those things which really
give confidence to to data teams over time. So there there's sort of a
couple components to it. The first, I think it really starts with visibility,
sort of call it end to end observability, but it really includes, like, you know,
you mentioned a few of these parts, but, the data
lake, the data warehouse, an orchestration,
BI, ML, AI application that can include the agent,
the vector base if you have a prompt. Right all of those
components you have to have visibility. The first thing is actually to to
your point, like, having lineage into what are the different components that can cross
this. So all the way from. You know, sort of ingestion of the data to
consumption of it. And the second is to start observing.
And and, you know, you there are some specific things that you can declare
and test and based on your business needs, and there are some things that you
can do in an automated way. And and, actually, I think this is an area
where AI can help. So for example,
what what oftentimes teams end up doing is spending a lot of time
trying to define what are data quality rules. And,
actually, you can use LLMs to profile the data,
Make some make some, yeah, make some inference,
based on the semantic meaning of data and then make recommendations.
So for example, I I love this example. We work with lots
of, sports teams. And so you can imagine that,
you know, you have a particular field called, like, let's say this is
in baseball, a baseball team and sort of, like, you know, pitch type.
And and then, like, the the speed that matches that. And
so you can imagine that, like, an l m can recommend or infer that
a fastball should not be, you know, less than
70 miles per hour or whatever it is. Even though I don't know what
the real number is. I just made that up. But there is, like, some you
you can infer based based on that and make a recommendation. And
so, actually, it's a I find that AI and LM is a really cool
application of how to make observability faster and and and
easier for for teams. So, yeah, I'm I'm
very excited about about what you just shared, Andy. Well,
I I love what you brought up about machine learning being able to to
make basically make predictions about things.
And and one of the terms that, you know, as a practitioner
of, business intelligence is especially the data engineering that supports
it Mhmm. Is data volatility. Mhmm. So if I'm
especially if I'm looking at an outlier. So I'm consuming this
data day in and day out, And let's
say, you know, 10% of the data is new stuff,
and maybe another 10 or 15% are things that are have
been updated, old stuff that's been updated, and the rest of it's relatively
stable. If I see those numbers go crazy out of bounds,
you know, and machine learning would be able to pick that up right
away and say, there may be a problem with the data we're
reading today. You know, I would I that that sounds like one of
the problems that would solve is that volatility,
expected ranges of volatility of data. That's exactly
right. Yeah. Cool. Interesting. I think there's
also something you said was, you know, when you have LLMs, because, obviously, we have
to talk about GenAI because it's 2025, and I think you're in
Silicon Valley. I think if you don't mention GenAI every twenty five
minutes, the cops come and knock on your door and check it out. Welfare check.
Could get in trouble. Or they make sure you're okay. Make
sure you're okay. But I think one of the things that really
kind of makes me worry about GenAI is that it's not
immediately obvious. Like, if you're at the airport, obviously, it's not a good look for
you. Like, if the if the and this has happened to me where the app
says one thing, the screen says something else, and my ticket says yet a
third thing. So I'm not really sure where I'm supposed to go.
Generally speaking of those, the app tends to be more accurate.
But, that depends on the airline.
But with with LLMs, it's a the latency
between you seeing the data where the cons the bad
consequences of the data tends to be a lot more
I'll use a $10 word today. I can't even say
it, but it's not it's not immediately obvious. Right? There goes my
my fail and my $10 word. But, like, it's not like it there's a lot
more steps in labyrinthine. I'll go with that one because I can say that.
But, like, what so how do you provide
observability in something like LLMs where
the, the input and the output time tends to not
be quite as straightforward as a data as an old school data pipeline?
Yeah. Such a great question. And maybe I'll just share some of my favorite
wonders if that's helpful. And and I think I'll share them
because it's helpful to explain the gravity
of these issues. So, for example, you know, if you're in an airport and, you
know, the app doesn't say the same as what you have,
hopefully, you arrive early at airports, Frank. I don't know if you have enough time
to, like, figure out the discrepancy and you won't miss your flight. Right?
But oftentimes, those things can lead to to really big disasters.
Even three gen AI. So so I think this was in 2020.
Unity, which is a gaming company, they had one schema
change, resulting in a hundred million dollar loss.
Their stock dropped 37%. Oh my gosh. Pretty
meaningful. Right? Fast forward, I think this was
2023 or 2024,
but not so much related to AI yet.
Citibank was hit with a $400,000,000 fine for
I remember that. For data quality practices for lack
of data quality practices. So think about all the regulatory
industries like health care, financial services,
like, you know, wherever there's, like, PII and and,
And and the, like, you know, the the
implication there are pretty grave. Some fun examples for more recently.
I don't know if fun. I shouldn't call them fun. Some other examples from
yeah. You mentioned Chevy. So I think there was a user
that convinced a chatbot to sell the Chevy Tahoe
for $1. I I commend the user from being able to
do that, but that is terrible. Right? That's terrible
that, that happened. And that chatbot went down
the next day. They they took it offline the next day. I think it was
in Fremont, California, so not that far from the bay.
Yeah. So right. So that's pretty pretty consequential.
I'll just give another, like, example. This is my favorite example. This is what
it went viral on x couple months ago. Someone googled, what should I
do when cheese is slipping off my pizza? And Google responded,
oh, you should just use organic superglue.
Great answer. They they had some really good gaps.
There was the, eat eat one rock a day to get your,
minerals and stuff like that. Yeah. So I I
love that because that's an example of where, like, the prompt was
fine, the context was probably fine, the model was
fine, but the model output was totally not fine.
Right? Right. And so and by the way, maybe Google can get away with it
because it's Google, but, like, 99.9% of brands can't get
away with with the mistakes. Right? And so what, you know, what
do you do? How do you provide observability in in that world? What does that
look like? First, I'll just say, I think
there's still human in the loop, and there will be. So, actually, you know,
it's interesting going back to 2019 when we started the company. People would tell us,
oh, you know, I have this important report that my CEO looks at.
But before they look at it, I have, like, six different people looking at the
report with, like, you know, sets of eyes to make sure that the data is
accurate. So, like, people use manual stuff back then. Today, what I
hear is I was just speaking with this head of AI, Silicon Valley,
and I was like, how do you make sure the answers are accurate? And they
were like, well, we have someone sifting through dozens, hundreds of
responses every single day to make sure they're accurate. So I don't think human in
the loop evaluation is going anywhere. There's more advanced techniques, you know,
comparing to to to ground truth data, using LLM
as a judge. There's various sort of, things that we can do, but but I
think human isn't going away. In terms of observability,
I talked before I'll explain a little bit about this sort of framework
of, you know, data issues can be really traced back
to these four core root causes, and I think it's
important to have observability for each in in sort of this world.
So the first I mentioned is data. And so by that, I mean,
you know, let's use another example. Credit Karma, for example,
has a financial advisor chatbot where, basically, they take in information
about you that they have, you know, like, what kind of car you
have as being of cars and, you know, where you live and whatnot, and then
they make financial recommendations based on that. If the
data that they are ingesting from third party data is late or isn't
arriving or is incomplete, that messes up everything downstream. So one
root cause can be the data that you're ingesting is just wrong. Maybe it's all
null values, for example. The second can
be due to change in the code. So the code could be like a a
bad like a schema change, like in the Unity example. It could be a change
in the code that's actually, being used for the
agent. Really, code change can happen every anywhere. And, by the
way, not necessarily by the data team. It can happen by an engineering team or
someone else. It has nothing to do with the with the data state. Right? So
code changes can contribute. The third is system.
A % of systems fail. What what do I mean by system? I
mean system is, like, basically the infrastructure that sort of runs all these jobs.
So this could be, like, an airflow job that fails or a DDT job
that that fails. You know, again, a % of systems fail,
and so you would definitely have something that goes wrong in systems.
And then the fourth is you could just have the model output be wrong, kinda
like with the cheese in in Google, example. And
so when we think about sort of having what does it mean,
what does observability mean in this in this age, I think it has to
have coverage for all four of those things. And here's the problem. It oftentimes
includes all four together. So I don't know if it you know, it's typically on
a Friday at 5PM. You're just about done, and then
everything breaks at the same time. That's an
interesting point. Like and and it's you also use the a term
a couple of times, which, you're I can count on one hand how many
non Microsoft people have used this term,
data estate. And I'm just curious about I know where I pick from
Microsoft. No. No. No. Like, I'm like I mean, I always
thought it was a, you know, Microsoft invention. I don't think it is.
But, like, where did you pick up that term? Because I've only like, seriously, you
were, like, the third or maybe fourth person who is not
never worked for Microsoft, never worked with Microsoft. I I mean, I don't know if
you work with Microsoft, but, like, I I always whenever I hear someone say
data to state publicly, I'm like, so who'd you work for at Microsoft? What division?
Like, like Oh, wow. Yeah. It's like that. And at first, I
didn't like I'll be honest. I didn't like the term at all, but eventually, I
kinda grew to like the term because it there's a lot behind it, and I'd
be curious to get, like, one, where'd you where'd you where'd you
pick that up? Like, I'm just, like and then two, what does it mean to
you? Like, what does that term data state mean to you? Great question. For
what it's worth, I actually didn't like it either. For the record, I didn't even
like data observability to begin with Mhmm. To be totally Really? English is
yeah. English is my second language, and observability was such a difficult word to
pronounce. When we started the when we started the, you know,
the company and and the category, we had to give it a name. So we
didn't really know is this you know, we used we we coined the term data
downtime, you know, as a corollary to application downtime. We thought maybe
data reliability. There are lots of
options. At the end of the day, I always try to get gravitate towards where
my customers are, so whatever language my customers use. And so customers
started using the word observability, so I started using that too. And same with the
state, they started using the data state sort of as a language. And so
Interesting. Full disclosure, have not, have no
ties to Microsoft, but but just have heard
mostly enterprises sort of think about that. I I think my understanding,
you know, for for what they mean is, you know, wherever
you store aggregate process data. And so that, you know, can
include, you know, you know, upstream
sources or upstream, data sources. But, you know, it could be,
like, an Oracle or SAP database. It could be data
lake house, data warehouse like Snowflake, Databricks,
AWS, Redshift, s three, all the
way to wherever you're consuming that. That could be a BI report. You know, Power
BI. Sorry, Microsoft.
Right, Looker, Tableau, you know,
various, various options. And,
honestly, the, you know, the most common enterprise has all of
the above in some shape or forward fashion. And so to sort
of include all of that, I think
the some of the thesis that we have around observability is that, by the way,
each of those by themselves has some concept of observability.
Right? Like, you
can, for example, with Snowflake, you can set up some basic,
sort of checks, if you will, like a sum check or whatever. Right?
You you could do that in Snowflake. However, we think that observability
needs to be sort of third party and to be end to end. And,
again, that draws on on software corollary. So,
you know, like, AWS has CloudWatch, for example,
but that's probably not sufficient for whatever you're building. You're probably
gonna use, again, like, New Relic or Datadog to connect
across the the board to, you know, variety of of,
integrations. Right? They have hundreds. So that's what I think about when I
say data estate. But it's a great question. It's definitely not my
word. No. I was just curious. Like like, you know,
because whenever because first, I hated the term too. Right? And I can't maybe it's
Stockholm Syndrome. I don't know. But,
the more I kind of sat on it and kind of digested it, I was
like, I like it because it explains, like, you know, you know, historically.
Right? Like, a state is, you know, whoever
owned the land got to call the shots and whoever called the shots owned the
land. Like, there was a very, you know, you drew the food, you you cut
down the trees, you, you know, you mined for, I think the Minecraft
movie is coming out. So you mined for all these things. Right? My kids are
into it. But, like, and it's
really kinda like it's just the idea of seeing it, like, it's land. It's kinda
like land. It's kinda like a natural resource. It's not really natural, but it is
a resource. Right? And if I say unnatural resource, that's really weird. But it's a
resource. Right? And if you you can either you have it. You already have
it. You either develop it or you don't. And, you know, do
you, you know, do you grow food on it? Do you, you know, like so
see, I I liked it because it was the idea that it's already there. Right?
Mhmm. And it's it might be in forms you don't really think about. Right? Like,
you know, PDFs in a in a SMB share somewhere.
Right? Mhmm. I mean, that's part of your data to state. Yep. Right?
And it's that's how I kinda, like, came to terms with it. And,
like, I really kinda like it because it helps you to think holistically about data
because I think a lot of business decision
makers and even technical decision makers don't see data as a
as a as a as a resource. I think that's changed
over the last maybe five, six years.
But it really became something that they don't see
it as a resource they could mine, they can get value out of. Right? The
smart people did. But, for the most part That's
right. Yeah. You had to convince them. Right? Exactly.
It sounds like based on what you say because, like, you know, my wife works
in IT security. Right? So, so we're a two engineer
household. So the kids are super nerds. But, like, I was telling
her after chat CPT came out, I was all excited about it. And I was
telling her about how this works. I was like, you give it this big corpus
of data, and they chews through it, and it comes up with these these vectors
and stuff like that. And then she looked at me and it's like, so all
the training data is now a massive attack surface.
And Yep. When that's just why I love my wife. So I
I'm wronged. She's never wronged. Well, that's true. But at
first I was like I was thinking but but you're missing and then I was
gonna say you're missing the point which one is never a good thing to say
but Like midway through I was like, oh my gosh,
she's right. Oh my gosh. She's right. So then
when I started talking to other data science and AI types, and I was like,
but but don't you think this could be, like, a big attack surface? I look
like that meme with the guy from It's Sunny in Philadelphia with, like, it's
always sunny where he had, like, the conspiracy thing. Like, I swear I will
like that meme. Yeah. And, you know, and if you
look at the I think OWASP has, like, the top 10 vulnerabilities of LLMs
that is either two or three. Right? So it's
kinda like there's a fine line between,
like, thinking too much about problem, but also kind of thinking ahead of the
problem. I don't know. No. Oh, I think you
cut off a little bit, Frank, but, Andy,
to me, that resonates a lot, and I think it's sort of really the overlap
between data and engineers. And, by the way, like, we didn't even talk
about security. Like, all these concepts also exist in security.
Right? And I think in the same way that we sort of manage, like, you
know, sub zero, sub one issues in security engineering, data
issues should be treated the same way. You should have a framework to understand what's
a sub zero, what's a sub one for data issues. You should it should be
connected to pager duty. Like, people should wake up in the middle of the night
when you have data issues. I think I think that's right. It's
improving, but, we're not quite there. It'll
happen. No. You're right, though. Like, they don't think about this in
terms of they don't does it I wouldn't say it's not disciplined. Sorry,
Annie. I cut you off. No. But my experience we talked to data engineers. Sorry,
Andy. And I I I I am a former data engineer
myself. Like, I thought of it in terms of schema structures and pipelines.
Mhmm. Not necessarily securing those pipelines. Right? Mhmm. Sorry,
Andy. I'll go. No. I was curious. I wanted to to shift back
to you. You mentioned the four areas that your software,
looks over your AI and the observability software does. What
happens when it detects something amiss?
Great question. So not even talking about Monte Carlo specifically, but rather
an observability solution. I think an observability solution needs to
have coverage or an observability approach, by the way. Like, some people build this
in house. An observability approach should take into consideration
your data estate, should take into consideration, right, your
entire data estate. I think, oftentimes, the mistake is people will even if they
build it in house or do anything else, they'll really just focus on, like, the
data and their data lake or the data in a particular report. Like, that's
not sufficient. Right? It it just isn't. And so people waste
a ton of time trying to understand, like, what's wrong and where. So I think
the first is, like, you need you need visibility across the data
state, which hopefully we've defined an unnatural resource that should be
managed securely. And and I think that's right because I
I by the way, Monte Carlo doesn't doesn't do the security
part, but I similarly believe that in the same kind of diligence
that we apply to data as engineering, you want data products to
be reliable but also secure, scalable,
like all those concepts should adapt. By chance, we happen to
focus on the reliability and observability part, but all the other,
principles of software engineering should apply.
We specifically don't do it, but very much believe that should be
the case. But back to your question, you
know, so so what happens when there is an issue?
Very similar to workflow that you might find in Datadog,
New Relic, and and PagerDuty. So there is an alert that goes out,
often you know, in whatever flavor of choice. If you're an enterprise that has a
data state, this is likely Microsoft Teams. If not, this would mean
Slack or an email or what you know, some teams like to have it connected
to to Jira and and pager duty for for sev zeros or sev
ones. And, you know, the first thing
that people will do is start, you know, typically an analyst.
I was I was in, you know, prior an analyst. The first thing you start
asking yourself is, why the hell is the data is wrong?
Right. Yeah. You're like, well, was the report on time?
Was the data accurate? Was it complete? You start going through all
and then you start you basically come up with hypothesis. And then you start
researching those hypothesis, and you're like, well, let me let me
trace the data all the way all the steps of the transformation
and start looking. Was the data okay here? Yes. Check. Okay. Move on. Was it
data right? You literally you started this, like, recursive process. Gotcha.
Before we started the company, I used to do this all manually. So I remember,
like, I would go into a, you know, into a room. Maybe you did this
too. And, like, on a whiteboard, I would start, like, basically mapping out
the lineage. Okay. This broke here. Was the data here okay? Let's let
let's sample the data and make sure it's okay. Okay. Move on. Let's like, literally,
we have this, like, very every morning, actually, you know, that this
became such such a problem because we were so reliant on this particular day
dataset that every morning, me and my team would wake up, and we would basically
go step by step and diligently, like, make sure that the data is accurate,
which I felt like was I was like, this is, like, total, you know, crazy.
So, you know, I think, particularly in Monte
Carlo or, like, what observability does is provides the
information that you need in order to troubleshoot and understand where the issue is. And
so we can surface you information like, hey. There was at the same time that
this dataset you know, maybe the the percentage of null values in
particular field was inaccurate. And then at the same time, there was a full
request that happened. Maybe those are correlated, actually. Gotcha.
Maybe, you know and maybe, actually, you can use you can also
do a code analysis. So you can, like, basically, you know, analyst
what we used to do is, like, sift through lines of code and try to
see what the change. Hey. Why did few surface to you that, like, there was
a particular change in the, you know, name of a field,
at the same time as an example. So bringing all that data into one
place can help you sort of troubleshoot that. And
sorry for another LLM plug, but you can actually have
an LLM do this for you, which is pretty sick where it's like an early
beta test for us. We haven't released it yet. But, basically, what we're
testing internally is for every like, for data incidents,
there's basically, like, an in like, a troubleshooting agent that
spawns agents for each of the hypothesis. So there's, like, an agent that
statement. Yeah. I it's really cool. There's an agent that
looks into, like, the code change, the data change, the system
change, and then and then it does it recursively on
all those tables. So you can actually run up to a hundred agents in under
one minute. And then there's a larger LLM that takes all that information
and summarizes it and synthesizes it. So, again, early days, this is like we're still
building it. Very cool. But the early results are really cool. Yeah. It's
like basically turbocharging your your data analysts and your data
stewards. Sorry. I got all excited. No. It's it is That's really
cool. Fascinating, and I love that you're excited about it. And what one of the
jokes that I make when I'm I'm working with my kids on something, if
they nail something, I'll I'll say to them, you know,
something similar to this. It's like, if you can only, you know, if you
can only run a hundred in one minute, I guess that's if that's the best
you can do, we'll just have to live with it. Yeah. Exactly.
That's that's an amazing stat. Yeah. Yeah. That is interesting. And I
also think too I also think too that, like, observability could help
with secure the security story. Right? Because if, you know, you're looking at a
pipeline and it's like, hey. Weren't there a bunch of
sketchy looking IPs, like, poking around our system about the time that this
pipeline ran? Maybe the rest of the data that goes out of that pipeline
run is a little bit suspicious too. Yeah. A
%. Like, we we you know, for example, you work with a,
call it delivery service, and there was a very
suspicious tip very suspicious
amount of tip that was given. Like, you
know, you can imagine, you know, the range of tips can be between x
dollars and y dollars, and suddenly that's, like, you know,
10,000 times y, like, 10,000 times the upper limit.
Yeah. You know, triggers off a suspicious alert. It's
not a normal tip, and it's not a mistake. It's actually, you know, security
issue. So that's an example. Yeah. Interesting. Yeah. I
love the anomaly detection aspect of that. I mean, it just it
it's it's something that we've been doing for a long time,
but then at wrapping it with automation and then
combining that automation with what you just described with all the
agents running down all of the permutations, that
that just sounds amazing. Yeah. It's really cool. I can't
take credit. This isn't me. It's it's it's my team. But,
but I I was like, woah. It's like a hundred bars
running at the same time under one minute. That's amazing. There you go. It's really
cool. Probably smarter than me. But yeah.
That is so awesome. That is cool.
So we we generally have is, we have kind of our
our stock questions that we ask, if you're interested in doing them.
They're not we're not Mike Wallace. We're not trying to I don't even think
anyone gets that reference anymore, but we're not trying to catch you in a,
I gotta come up with a new one, in a thing. But it's mostly, like,
how'd you find your way in the first one is I'll get the rest of
them, up for you in a second. But the first one is, how'd
you find your way into data? Did did the data did you find the data
life or did data life find you? Oh, that's such a great
question. You know, it's funny.
I grew up you know, my my, my mom is a meditation and dance
teacher and my dad is a physics professor. And so,
yeah, and so I, I, you know, grew up with very sort of like, yin
yin yang in my family, if you will.
At a very early age, I used to, like, hang out in in my dad's
lab and, like, do scientific research and stuff like that. So or, you know,
like, very at a very young age, my memories are, like, sitting in a
cinema, watching a movie with my dad and trying to, like, guesstimate how
many people are sitting in the in the audience.
Right? Yes. Just like, you know, I think for, like, a five year
old, it's sort of like a fun fun thing. But, you know, throughout my my
adulthood, like, always sort of had that in in the background. And,
you know, I I think later on in life, I sort of always gravitated towards
data. And when I decided to start a company,
I was actually debating between various areas
like IT and actually blockchain, or, you know,
crypto for a little bit and and data. I think at the end of the
day, like, my heart was really in in data. If I look at, like,
the next ten, twenty years, it's pretty clear to me that data is
gonna be I think it still is the coolest party, and I think it
will be the coolest party to be in. And I personally,
like, you know, it's it's it's funny. Like, throughout my my
career, I've I've also learned the limitations of data. Right? So so data can
tell you whatever story you want. It could tell you, you know, for every question,
it give can give you a yes, and you can also tell a no story.
Right? So so there's also limitations to data,
but but I always have been fascinated,
by by data and space. So can I say both? That's
Yeah. I mean, that's fair. That's fair. Good answer. That's fair. Yep. So
what what's your favorite part of your current job?
Oh, that's hard to choose. I love my job.
I just love it. I think, you know,
the ability to work with customers and actually, like, change the way they
work, I I think that's probably the biggest gratification that I
get, you know, from from my my career. Like, the fact that you can
actually work on something that matters is pretty insane. You know? And when I think
about, like, the future, I'm like, what? So data is gonna be wrong? Like, we're
just gonna be, you know, making decisions off of wrong like, what? I don't
wanna live in that world. You know? And so Yeah. I think
there's something that's, like, really fulfilling and helping, you know, drive a mission that
I believe in that has an impact on customers. And, you know, when customers will
tell me, you know, I started sleeping at night because I
know that, like, I have some coverage for my data. I'm like, yeah. Oh, wow.
I'm glad you're sleeping. You know? Like, good for you. I love
sleeping. So What a cool thing to hear. Yeah. Exactly. I
think that's that's probably, you know, maybe one part. And then the second is, like,
just working with an amazing team. You know, I I spend most of my my
day maybe kinda like, you know, you guys, like, hang out having fun,
laughing. So, you know, I I I'm very
grateful that I get to work with the smartest people on on
worthwhile challenges. Oh, very cool.
We have, three complete these sentences. When I'm not
working, I enjoy blank. Sleeping.
I yeah. I I have a I we recently
have added we we had two kids, and we adopted a cousin. And
I forgot how draining a toddler can be. And I'm
I'm eight to 10 years older since the last time I had a toddler, so
it's like I, I have two
kids, on two under four. So I,
respect the sleep even more. I I can't even I can't
even wrap my head around that. It gets it gets better. I can say
that. It's my own role. I appreciate that.
So our second one is I think the coolest thing in technology
today is blank. The coolest thing in
techno I think the pace of innovation. I think that's really
freaking cool. You know, you can, like, work at a problem today and you're like,
you can't solve this. Two days two days later, a new model will come out.
Boom. You're done. So it's harder. Right? The bar is
higher in order to, like, actually like, it's it's harder to it's
harder to know what to bet on. It's harder to know what the future will
look like, but it's a lot more exciting. So I'm in it.
Cool. Our third and final complete sentence is, I look forward
to the day when I can use technology to blank.
I was always a big fan of teleportation. I think teleportation is really
freaking cool. That would be nice. Can't wait for that. That would be cool.
That would be cool. You know, you're not the first person to answer with them.
Oh, really? Yeah. It's pretty cool. Pretty cool. Sorry.
Number six is share something different about
yourself. Something different.
Yeah. Something different. Let's
see. I mentioned I have two kids. I
meditate when I don't sleep. I like to meditate.
I, what else? I'm married to
my cofounder. Oh, wow. So we,
yeah, we're fortunate to share our lives both at work and at
home. That is cool. Yeah. I can
imagine that would work out really well or not. Like, there's not a lot of
middle ground there. High risk, high reward. High risk, high reward. I
get, like you know, my wife is, you know, she's a
federal employee, and she's, you know, reevaluating what her career
futures look like, you know, and she's like, you
know, I was like, well, you know, you could help. You can start
a new podcast. I can help you with that. She's like, yeah. But then I
have to work with you. And, like, I know what she meant. I know how
it sounds. I know how it sounds, but I know what she means. Like, so
when she did work from home, like, there was literally a, like, an entire floor
between us because Yep. Like, it's too loud. I'm too loud. Yeah. Yeah. Yeah.
Yep. We're very loud too. So
where can folks find more, learn more about, Monte
Carlo and, and and what you're up to?
Probably, I'm the place where I hang out is LinkedIn. So,
I know we just got connected on LinkedIn. That's great. Probably follow me
on LinkedIn or, honestly, reach out to me directly, me,
Moses@MonteCarlodata.com. I hope I don't get a lot of phishing now because
of that. But Well, hopefully, make sure it's the right account because we found out
in the process that there's there was another suspect in
suspicious looking account. And I also think that for our
listeners, it's worth pointing out that I think that people have realized that LinkedIn is
a is a major security vector because I've been getting a lot of
weird a lot more lately. Now I don't think it's related to
the, the refrigerator scandal. Andy and I will do a whole show on that
later because there's there's actually an interesting AI component to that. Okay.
Good to know. And finally, last but not least, Audible
is a sponsor of the podcast. Do you do audiobooks? If
so, recommend one. Otherwise, just recommend a good book you recommend.
A good book. Let's see.
Thinking in bets by Annie Duke.
Professional poker player. Interesting. In in how
lessons from poker can be applied in, in life
and in business. Interesting. I
once worked at a financial services company, and one of the
big shots used to play online poker. And
they're on company, not on company money, but on company time. And a
lot of people Not a lot of people took a dim view of that.
Rightfully so. But he was
making so much money. You know, people that matter didn't take a damn view to
it. When he stopped making so much money, people everyone took a damn view to
it. And it they don't that does end the the story. It
is on I don't see if it's an audio oh, it is
an audio book. It is an audio book. Awesome. I'm gonna add that to my
list. I'm done. Okay. And if you you know, they are a sponsor.
So if you go to, the datadrivenbook.com, you know,
you'll get a free audio book on us. And, you know, if you sign up,
we'll get enough to, you know, buy a coffee.
Maybe not tip them $8,000, but, you know,
we'll get enough for a Starbucks maybe. Maybe. Yeah.
I just tested the link, Frank. Every now and then, we had trouble early on
with the link coming and going. So I just when you saw me turn away
a minute ago when Frank started to this question, that was me typing
in. It worked. It worked.
It's always DNS. That's the Always. It's interesting
you mentioned that. I read an article. Actually, it was a newsletter recently that talked
about, betting being the first stage
in, kind of the path to minimally viable products. And
I thought, now that's curious, and I don't know again, I haven't
read the book. I will listen to it. But the idea of
engaging your team I I manage a team, as well.
And engaging the team by having them do
interesting things and making taking these very large bets
that look nearly impossible,
perhaps. And it's like you said, the the the problem
comes up, and you're thinking this is this is unsolvable. And two days
later, it's solved. And over and over again, I've had that
experience, but I never tied it to the concept of
bets. And I saw this this newsletter that talked about do
that first, And it reminded me a little bit
of Collins talking about, the the big hairy
goals, you know, back in the day. It's very
similar to that maybe in concept. I don't know. I'll have to listen to the
book and check it out, but I was intrigued by the newsletter. Yeah.
There's interesting concepts. Like, I think some of the ideas is, like I mean, even
when you start a company or sort of, you know, start working on a team,
like, you basically have you have a set of cards, which are, like, your strengths,
your weaknesses. And so how how do you play your cards? Like, you can't you
know, if you wanna win around, you can't play with someone else's cards.
You are what you are. And so the best thing you can do is play
with your card. I think that's true for a team solving a problem or startup
or whatever it is. I love that. Yeah.
Interesting. Any final thoughts? This was so fun. Thanks for
having me. Thank you. Thanks for, and you did mention kinda offhand early
on. I don't remember if it was in the green room or not. You have
a podcast yourself? I do not have a podcast myself.
Alright. That was my mistake. Maybe I'll end it tomorrow. Okay. All
good. Life goal one day. There
you go. There you go. And with that, we'll let our AI finish
the show. And that wraps up another data packed episode of
data driven. A massive thank you to our brilliant guest, Bar
Moses, for taking us deep into the world of data observability,
sketchy LinkedIn impersonators, and the dark arts of tipping
anomalies. Who knew a dodgy schema change could cost more than
a luxury sports car? Now, dear listener, if you've made
it this far, you clearly have excellent taste. So why not
put that good judgment to work and leave us a rating and review on
whatever platform you're tuning in on? Apple, Spotify,
Pocket Casts, Morse code, however you get your fix, would love
your feedback. And dare I ask, are you subscribed?
I mean, you wouldn't want to miss out on future episodes filled with more
wit, wisdom, and the occasional fridge based conspiracy,
would you? Until next time, stay curious, stay
observant, and for heaven's sake, keep your data tidy.