Welcome to Data Driven, where we dive into the thrilling world of data,

AI, and on occasion, misbehaving chatbots suggesting

glue for your pizza. This episode features Bar Moses,

CEO of Monte Carlo. Not the casino, not the car,

but the company keeping your data from quietly wrecking your business.

We talk observability, the chaos of unreliable data,

and why one tiny schema change cost a company

$100,000,000. Ouch. So buckle

up. Because if your AI bots are making decisions without

reliable data, well, hope you like eating rocks for the

minerals. Hello, and

welcome back to Data Driven, the podcast where we explore the emergent

fields of data science, artificial intelligence, and, of course, data

engineering. And with me today is my favorite data engineer in the

world, Andy Leonard. How's it going, Andy? It's going well, Frank.

How are you? I'm doing well. I'm doing well. I was in Raleigh last

week, drove down, rented a car actually,

to save mileage on, on ours, and,

spoiled because it's been a while since I bought a new car. And

this is the second time I rented a car, and I'm getting tempted. I ain't

getting tempted. It was a Chevy. It was

a Chevy Malibu. Not a Monte not a Monte Carlo.

See what I did there? I don't even know if they still make them. I

I was driving, the little one off and dropping the little one off at daycare,

and I was behind a Chevy Monte Carlo, like, a two early

two thousands vintage. But that is actually quite relevant

to our discussion today because with us today, we have Bar Moses, who is the

CEO and cofounder of Monte Carlo, the data

and AI reliability company, not the casino

or the car, I would assume, or the town. Monte Carlo

is the creator of the industry's first end to end data and

AI, observability platform with

$236,000,000 in funding from Accel

Iconic Growth and others. They are on a mission to bring

trustworthy and reliable data and AI, to

companies everywhere. The company was recently recognized as

a enterprise tech 30 company, a CRN

emerging vendor, and an inc.com,

best workplace and accounts Fox, Roche,

Nasdaq, and PagerDuty, among others, as their customers. Welcome

to the show, Bar. Thank you so much. Great to be here, Frank and

Andy. Awesome. An intro. No problem. Do you drive a

Monte Carlo? Because that would be epic. You know, I really should

be driving a Monte Carlo. I do not, and I've never actually been to

Monte Carlo either. So I will tell you if you're into cars,

like, I'm like a recovering car, nerd. Oh,

very cool. It looks like a car show. Like, honestly, I went to Monte

Carlo, and we had rented, like, a Saab convertible. And I felt like we were

driving. We were driving driving, like, the low end

of the car thing. I mean, there were I mean, I've never

seen Bentleys in the wild, like, just parked on the street,

like, no big deal. Wow. Like, I mean, every

luxury car if you're in a Saab and you feel like you're slumming it

Yeah. It is clearly a high money area.

But, so welcome to the show. So Monte Carlo

why'd you get the name? I I'm assuming it might have something to do with

Monte Carlo simulations, but that's in the Great question. Yeah. The

unofficial story is that, one of our CO, founders is a fan

of formula one and, you know, as, you know, formula one crisis.

So right. That's, you know, clearly the, the, that's the

unofficial story. The official story is that, you know, we

had to we had to name the company. We started working with customers when we

started the company, and we we had to choose some name.

And, I studied math and stats in college, and so I sort

of opened my my stats book and sort of looked through and,

you know, reviewed my option and, you know, Markov,

chains didn't seem like a great name. And next up was

Bayes' theorem, which was similarly kind of not great. And

and then, you know, I was reminded of Monte Carlo and Monte Carlo simulations. I

actually I actually did some work with Monte Carlo simulations earlier in my career.

And it seemed like it seemed like a great name, a name that would speak

to, you know, data engineers, data analysts, folks that have been the space.

And, you know, I think naming a company is a very difficult

thing to do. We decided to go with it. And the spirit of Monte Carlo,

One of our values is ship and iterate. And so, the

name has sort of stuck with us since. And, it's quite memorable. People either

love it or hate it. So I think it works for us. I think it

it works. Like, I think of the car. I think of the casinos. It has

a certain amount of, high class, maybe more so than Markov

chains, Markov chains. Although I did for a time flirt with the

idea of of also starting a company called Markoff Chains, but,

like, have see if we could see if we can get money for mister t

to be the spokesman. That would

have been epic. Yeah. Jeez. He did you. Ideas, Fran. I was the

only one I was the only one that thought that was a good idea, but,

you know, I was a big fan of mister t as a kid. Marketing. Yeah.

That's funny. That's what I do in my day job now. Oh, yeah.

I swear, folks, I didn't pay her to say that.

So so you you talk about data and I AI

reliability. And to me, when when I hear that,

a slew of things come to mind. Like, there's security, there's the

veracity, like, the five v's and all that or four v's or whatever it

was. What exactly is kind of Monte Carlo's, like,

wheelhouse there? Yeah. Great question. I'll

actually sort of anchor ourselves in in kind of the metaphor or sort of a

corollary that we like to use here, which is really based on software engineering.

So we didn't reinvent the wheel when we say data and AI observability.

We really take concepts that work for engineering and adapt them.

So, you know, when we started the company, the idea, the

hypothesis, the the thesis that we started the company on was data

was going to be as important to businesses as applications, as online

applications. And, they were data was going to

drive the most critical sort of, you know, lifeblood of companies through

decision making, internal products, external products.

And, while software engineers had all the solutions and tools in the

world to make sure their applications were reliable, and so some, you

know, some off the shelf solutions like Datadog, New Relic, Splunk might be

familiar to you, data teams were flying wide. So there was literally

nothing that they could use to know that their data was

actually accurate and trusted. That's sort of, like, the the problem the core problem that

we started. Fast forward to today, you know, we created the data observability

category. We're continuing to create it. AI is making this problem just

infinitely bigger, harder, more important. Why? Because

data and AI products are now you know, there's a proliferation of those.

An AI application is only as good as the data that's powering it,

and the AI application itself can be inaccurate, can be

unreliable. Right? And so at a very high level

I know this is, you know, very vague, but at a very high

level, the idea was the same diligence that we treat software

applications, we should be treating for data and AI applications. Now,

what does that actually mean? How do we do that? Enter the concept of

observability. Observability is basically understanding or

assessing a system's health based on its output.

And so basically, the thesis was, can we observe end to end the

data and AI estate, learn what the patterns

are in the in the data, bring together metadata and context,

lineage, for example, about the data, derive insights

based on that to understand and determine what the system should

behave like, and alert if that gets violated. So that's sort

of the first part. The first is actually being being able to help data teams

detect issues. The second part is actually being help,

helping data teams resolve issues. Now here's the interesting thing

that we sort of learned over over the years. We've worked with hundreds of of

enterprises. So, you know, we mentioned a few. We real really work with the top

companies in every single industry. So,

you know, in in, in health care, in retail,

in manufacturing, in, technology, in each of these

areas, the, data in the state

obviously varies, but there are actually interestingly commonalities. And the

commonalities is that every single issue can be

traced back to a problem with the data, problem with the code,

problem with the system, or problem with the model output. Can go

into detail into more each of those, but that's sort of the high level,

framework. We basically provide end to end coverage to help data teams

understand what the issues are and help them trace them back to data issues,

code issues, system issues, or model output issues. So when did

you get the idea that I'm sorry, Andy. I cut you off. Okay. When

did you get the idea when you realized that data is gonna be as important

as applications are to businesses? Oh, great question.

Yeah. Great question. So so we started the company in 2019.

And, actually, what's interesting, it was pretty clear to us then, but we

had to prove that or we had to convince that of people. Definitely.

Yeah. It was not obvious. It's it's still there's still a

lot of people that are kind of, like, I guess, they'd be in the quadrant

of laggards where they realize, oh, I guess this is important.

A %. I would imagine in 2019, that would have

been you would have sounded insane. Like We we sound I

sounded insane a %. People are like, what? Data is

gonna be important? Are you sure? Now a couple of things happened

since, which I think helped. First is,

there were some large acquisitions in the data space, like Tableau and

Looker earlier on, and then Snowflake IPO'd. Snowflake was the

largest software IPO of all times. It was quite interesting that the

largest software IPO of all time is a data company. So I think those

things sort of help kind of convince that this you know,

convince, at least, externally, you know,

to the market that data will continue to be will will be

important and critical. I think the things that I noticed is, you know,

before we even started the company, we spoke to hundreds of data leaders, and I

speak to dozens of data leaders every single month. They continue

and I think what you hear from them is more and more

data teams and software engineering teams are building products hand in hand.

So they're actually they're side by side building. Right? And so, actually,

almost more and more critical business

applications, revenue generating products are based off of

data, and they're being powered by data. I'm not even talking

about generative AI, which is a whole whole other story why that matters, but just

data products by itself. Think about reports that people look at internally.

You know, just give you an example. You know, we work with with, many

airlines, for example. Airlines have a lot of data that goes to internal

operations. Like, what's the connecting flight? What's your flight number? How

many flights left today? What time did they leave? How many passengers were on

the airplane? Where is your luggage? Right? That

information is powering internal and external products. You know, it's powering the application

that you're using in order to onboard the the plane, in order to connect

to your next flight. If that data is inaccurate, like,

you're screwed. Right? And that hurts tremendously. Your brand

is an as an airline, your reputation, it leads to

reduced revenue, increased regulatory risk that you're putting

yourself. Right? So so the data,

what we see from our customers is powering critical use cases like

airlines. I'll give you another example. You know, we work with a,

you know, a Fortune 500 company, perhaps your your favorite cereal.

I don't know if you're you guys are big cereal. I I, like, eat cereal

for breakfast, lunch, and and dinner. It's, like, my go to.

You'd be surprised into how much data optimization, machine learning,

and AI goes into actually optimizing the number and

location of cereal on the shelf. So there's a lot of

data that goes into supply chain management to make sure that you're

actually, like, fulfilling the right warehouse,

demands on time and, you know, making sure that everyone gets

their serial on time. There's actually a lot of data that goes into all of

that. So I think what gave me conviction was in speaking with

so many companies across so many industries, data was

actually allowing data teams, allowing

organizations to build better products, to build more

personalized products, and to make better decisions about the organization.

So I think that really sort of made it clear that the future was going

to be based on on data. Well, I I like that

you pointed out, the importance of observability.

My career path winding as it was,

I made a a leap from being a software developer to being

a data really a database developer. When I made that

transition, one of the things I had noticed, this was two two and a half

decades ago, I had just started in software development

doing test driven development and it had just

come out, it was called fail first development. I remember thinking

this was perfect. It was a big deal. Yeah. It was. Yeah. Twenty five

years ago. And I remember thinking this is perfect because I'm always failing.

So this this will work nothing ever runs the first time and if it does,

it's suspect. But when I got over into data, I had just

become, you know, kind of a a big believer in the power

and and and really the the confidence that

test driven development gave me. And I was like, we need that

over here. And so it was, just a

field that's fascinating me. I have an engineering background, and so it kind of flowed

right through. Instrumenting the data engineering,

was a big deal so that, again, you could achieve what we now call

observability. But being able to watch that data flow

and when I would mention this to people kinda like you in 2019, I

I would get all sorts of responses. Most of them kinda raised

eyebrows. And I would, some of the more interesting ones

were things along the lines of, well, the data is sort of self

documenting. I mean, it's it's just there. And I'm

like, no. No. It's not. It's I especially when you've moved it through

a bunch of transformation to put it into a business intelligence solution or data

warehouse or or any of that. And that now feeds,

you know, modern LLMs, AI, and and the like, those

same sorts of, I guess, old school processes, I

do. Or at least that's my my understanding. Maybe I'm reading too much into

that, but I love the idea of having observability go

all the way through. You mentioned lineage. That's huge. You wanna make sure that when

you, you know, you make this one change, that's not gonna affect anything

else. Usually, it does affect other things, and having

that lineage view is huge. That is spot on.

That's exactly how we've we've thought about this as well. So, you know, I

think there are specific things that you can test for in data. Like, for

example, you know, specific thing that you can declare, you can say, like,

you know, you know, a T shirt

size should only be, you know, small, medium, large, extra large, whatever.

Right? But then there are some specific things that, you

know, you you don't necessarily know. Like, for example, if there's a particular,

you know, pattern that the data is being updated,

you can actually use machine learning to automatically learn that pattern and then forecast

when it should get up updated again. So it's not necessary for someone to

manually write a test for that. Right? And so

I actually think it's a combination of both of those things which really

give confidence to to data teams over time. So there there's sort of a

couple components to it. The first, I think it really starts with visibility,

sort of call it end to end observability, but it really includes, like, you know,

you mentioned a few of these parts, but, the data

lake, the data warehouse, an orchestration,

BI, ML, AI application that can include the agent,

the vector base if you have a prompt. Right all of those

components you have to have visibility. The first thing is actually to to

your point, like, having lineage into what are the different components that can cross

this. So all the way from. You know, sort of ingestion of the data to

consumption of it. And the second is to start observing.

And and, you know, you there are some specific things that you can declare

and test and based on your business needs, and there are some things that you

can do in an automated way. And and, actually, I think this is an area

where AI can help. So for example,

what what oftentimes teams end up doing is spending a lot of time

trying to define what are data quality rules. And,

actually, you can use LLMs to profile the data,

Make some make some, yeah, make some inference,

based on the semantic meaning of data and then make recommendations.

So for example, I I love this example. We work with lots

of, sports teams. And so you can imagine that,

you know, you have a particular field called, like, let's say this is

in baseball, a baseball team and sort of, like, you know, pitch type.

And and then, like, the the speed that matches that. And

so you can imagine that, like, an l m can recommend or infer that

a fastball should not be, you know, less than

70 miles per hour or whatever it is. Even though I don't know what

the real number is. I just made that up. But there is, like, some you

you can infer based based on that and make a recommendation. And

so, actually, it's a I find that AI and LM is a really cool

application of how to make observability faster and and and

easier for for teams. So, yeah, I'm I'm

very excited about about what you just shared, Andy. Well,

I I love what you brought up about machine learning being able to to

make basically make predictions about things.

And and one of the terms that, you know, as a practitioner

of, business intelligence is especially the data engineering that supports

it Mhmm. Is data volatility. Mhmm. So if I'm

especially if I'm looking at an outlier. So I'm consuming this

data day in and day out, And let's

say, you know, 10% of the data is new stuff,

and maybe another 10 or 15% are things that are have

been updated, old stuff that's been updated, and the rest of it's relatively

stable. If I see those numbers go crazy out of bounds,

you know, and machine learning would be able to pick that up right

away and say, there may be a problem with the data we're

reading today. You know, I would I that that sounds like one of

the problems that would solve is that volatility,

expected ranges of volatility of data. That's exactly

right. Yeah. Cool. Interesting. I think there's

also something you said was, you know, when you have LLMs, because, obviously, we have

to talk about GenAI because it's 2025, and I think you're in

Silicon Valley. I think if you don't mention GenAI every twenty five

minutes, the cops come and knock on your door and check it out. Welfare check.

Could get in trouble. Or they make sure you're okay. Make

sure you're okay. But I think one of the things that really

kind of makes me worry about GenAI is that it's not

immediately obvious. Like, if you're at the airport, obviously, it's not a good look for

you. Like, if the if the and this has happened to me where the app

says one thing, the screen says something else, and my ticket says yet a

third thing. So I'm not really sure where I'm supposed to go.

Generally speaking of those, the app tends to be more accurate.

But, that depends on the airline.

But with with LLMs, it's a the latency

between you seeing the data where the cons the bad

consequences of the data tends to be a lot more

I'll use a $10 word today. I can't even say

it, but it's not it's not immediately obvious. Right? There goes my

my fail and my $10 word. But, like, it's not like it there's a lot

more steps in labyrinthine. I'll go with that one because I can say that.

But, like, what so how do you provide

observability in something like LLMs where

the, the input and the output time tends to not

be quite as straightforward as a data as an old school data pipeline?

Yeah. Such a great question. And maybe I'll just share some of my favorite

wonders if that's helpful. And and I think I'll share them

because it's helpful to explain the gravity

of these issues. So, for example, you know, if you're in an airport and, you

know, the app doesn't say the same as what you have,

hopefully, you arrive early at airports, Frank. I don't know if you have enough time

to, like, figure out the discrepancy and you won't miss your flight. Right?

But oftentimes, those things can lead to to really big disasters.

Even three gen AI. So so I think this was in 2020.

Unity, which is a gaming company, they had one schema

change, resulting in a hundred million dollar loss.

Their stock dropped 37%. Oh my gosh. Pretty

meaningful. Right? Fast forward, I think this was

2023 or 2024,

but not so much related to AI yet.

Citibank was hit with a $400,000,000 fine for

I remember that. For data quality practices for lack

of data quality practices. So think about all the regulatory

industries like health care, financial services,

like, you know, wherever there's, like, PII and and,

And and the, like, you know, the the

implication there are pretty grave. Some fun examples for more recently.

I don't know if fun. I shouldn't call them fun. Some other examples from

yeah. You mentioned Chevy. So I think there was a user

that convinced a chatbot to sell the Chevy Tahoe

for $1. I I commend the user from being able to

do that, but that is terrible. Right? That's terrible

that, that happened. And that chatbot went down

the next day. They they took it offline the next day. I think it was

in Fremont, California, so not that far from the bay.

Yeah. So right. So that's pretty pretty consequential.

I'll just give another, like, example. This is my favorite example. This is what

it went viral on x couple months ago. Someone googled, what should I

do when cheese is slipping off my pizza? And Google responded,

oh, you should just use organic superglue.

Great answer. They they had some really good gaps.

There was the, eat eat one rock a day to get your,

minerals and stuff like that. Yeah. So I I

love that because that's an example of where, like, the prompt was

fine, the context was probably fine, the model was

fine, but the model output was totally not fine.

Right? Right. And so and by the way, maybe Google can get away with it

because it's Google, but, like, 99.9% of brands can't get

away with with the mistakes. Right? And so what, you know, what

do you do? How do you provide observability in in that world? What does that

look like? First, I'll just say, I think

there's still human in the loop, and there will be. So, actually, you know,

it's interesting going back to 2019 when we started the company. People would tell us,

oh, you know, I have this important report that my CEO looks at.

But before they look at it, I have, like, six different people looking at the

report with, like, you know, sets of eyes to make sure that the data is

accurate. So, like, people use manual stuff back then. Today, what I

hear is I was just speaking with this head of AI, Silicon Valley,

and I was like, how do you make sure the answers are accurate? And they

were like, well, we have someone sifting through dozens, hundreds of

responses every single day to make sure they're accurate. So I don't think human in

the loop evaluation is going anywhere. There's more advanced techniques, you know,

comparing to to to ground truth data, using LLM

as a judge. There's various sort of, things that we can do, but but I

think human isn't going away. In terms of observability,

I talked before I'll explain a little bit about this sort of framework

of, you know, data issues can be really traced back

to these four core root causes, and I think it's

important to have observability for each in in sort of this world.

So the first I mentioned is data. And so by that, I mean,

you know, let's use another example. Credit Karma, for example,

has a financial advisor chatbot where, basically, they take in information

about you that they have, you know, like, what kind of car you

have as being of cars and, you know, where you live and whatnot, and then

they make financial recommendations based on that. If the

data that they are ingesting from third party data is late or isn't

arriving or is incomplete, that messes up everything downstream. So one

root cause can be the data that you're ingesting is just wrong. Maybe it's all

null values, for example. The second can

be due to change in the code. So the code could be like a a

bad like a schema change, like in the Unity example. It could be a change

in the code that's actually, being used for the

agent. Really, code change can happen every anywhere. And, by the

way, not necessarily by the data team. It can happen by an engineering team or

someone else. It has nothing to do with the with the data state. Right? So

code changes can contribute. The third is system.

A % of systems fail. What what do I mean by system? I

mean system is, like, basically the infrastructure that sort of runs all these jobs.

So this could be, like, an airflow job that fails or a DDT job

that that fails. You know, again, a % of systems fail,

and so you would definitely have something that goes wrong in systems.

And then the fourth is you could just have the model output be wrong, kinda

like with the cheese in in Google, example. And

so when we think about sort of having what does it mean,

what does observability mean in this in this age, I think it has to

have coverage for all four of those things. And here's the problem. It oftentimes

includes all four together. So I don't know if it you know, it's typically on

a Friday at 5PM. You're just about done, and then

everything breaks at the same time. That's an

interesting point. Like and and it's you also use the a term

a couple of times, which, you're I can count on one hand how many

non Microsoft people have used this term,

data estate. And I'm just curious about I know where I pick from

Microsoft. No. No. No. Like, I'm like I mean, I always

thought it was a, you know, Microsoft invention. I don't think it is.

But, like, where did you pick up that term? Because I've only like, seriously, you

were, like, the third or maybe fourth person who is not

never worked for Microsoft, never worked with Microsoft. I I mean, I don't know if

you work with Microsoft, but, like, I I always whenever I hear someone say

data to state publicly, I'm like, so who'd you work for at Microsoft? What division?

Like, like Oh, wow. Yeah. It's like that. And at first, I

didn't like I'll be honest. I didn't like the term at all, but eventually, I

kinda grew to like the term because it there's a lot behind it, and I'd

be curious to get, like, one, where'd you where'd you where'd you

pick that up? Like, I'm just, like and then two, what does it mean to

you? Like, what does that term data state mean to you? Great question. For

what it's worth, I actually didn't like it either. For the record, I didn't even

like data observability to begin with Mhmm. To be totally Really? English is

yeah. English is my second language, and observability was such a difficult word to

pronounce. When we started the when we started the, you know,

the company and and the category, we had to give it a name. So we

didn't really know is this you know, we used we we coined the term data

downtime, you know, as a corollary to application downtime. We thought maybe

data reliability. There are lots of

options. At the end of the day, I always try to get gravitate towards where

my customers are, so whatever language my customers use. And so customers

started using the word observability, so I started using that too. And same with the

state, they started using the data state sort of as a language. And so

Interesting. Full disclosure, have not, have no

ties to Microsoft, but but just have heard

mostly enterprises sort of think about that. I I think my understanding,

you know, for for what they mean is, you know, wherever

you store aggregate process data. And so that, you know, can

include, you know, you know, upstream

sources or upstream, data sources. But, you know, it could be,

like, an Oracle or SAP database. It could be data

lake house, data warehouse like Snowflake, Databricks,

AWS, Redshift, s three, all the

way to wherever you're consuming that. That could be a BI report. You know, Power

BI. Sorry, Microsoft.

Right, Looker, Tableau, you know,

various, various options. And,

honestly, the, you know, the most common enterprise has all of

the above in some shape or forward fashion. And so to sort

of include all of that, I think

the some of the thesis that we have around observability is that, by the way,

each of those by themselves has some concept of observability.

Right? Like, you

can, for example, with Snowflake, you can set up some basic,

sort of checks, if you will, like a sum check or whatever. Right?

You you could do that in Snowflake. However, we think that observability

needs to be sort of third party and to be end to end. And,

again, that draws on on software corollary. So,

you know, like, AWS has CloudWatch, for example,

but that's probably not sufficient for whatever you're building. You're probably

gonna use, again, like, New Relic or Datadog to connect

across the the board to, you know, variety of of,

integrations. Right? They have hundreds. So that's what I think about when I

say data estate. But it's a great question. It's definitely not my

word. No. I was just curious. Like like, you know,

because whenever because first, I hated the term too. Right? And I can't maybe it's

Stockholm Syndrome. I don't know. But,

the more I kind of sat on it and kind of digested it, I was

like, I like it because it explains, like, you know, you know, historically.

Right? Like, a state is, you know, whoever

owned the land got to call the shots and whoever called the shots owned the

land. Like, there was a very, you know, you drew the food, you you cut

down the trees, you, you know, you mined for, I think the Minecraft

movie is coming out. So you mined for all these things. Right? My kids are

into it. But, like, and it's

really kinda like it's just the idea of seeing it, like, it's land. It's kinda

like land. It's kinda like a natural resource. It's not really natural, but it is

a resource. Right? And if I say unnatural resource, that's really weird. But it's a

resource. Right? And if you you can either you have it. You already have

it. You either develop it or you don't. And, you know, do

you, you know, do you grow food on it? Do you, you know, like so

see, I I liked it because it was the idea that it's already there. Right?

Mhmm. And it's it might be in forms you don't really think about. Right? Like,

you know, PDFs in a in a SMB share somewhere.

Right? Mhmm. I mean, that's part of your data to state. Yep. Right?

And it's that's how I kinda, like, came to terms with it. And,

like, I really kinda like it because it helps you to think holistically about data

because I think a lot of business decision

makers and even technical decision makers don't see data as a

as a as a as a resource. I think that's changed

over the last maybe five, six years.

But it really became something that they don't see

it as a resource they could mine, they can get value out of. Right? The

smart people did. But, for the most part That's

right. Yeah. You had to convince them. Right? Exactly.

It sounds like based on what you say because, like, you know, my wife works

in IT security. Right? So, so we're a two engineer

household. So the kids are super nerds. But, like, I was telling

her after chat CPT came out, I was all excited about it. And I was

telling her about how this works. I was like, you give it this big corpus

of data, and they chews through it, and it comes up with these these vectors

and stuff like that. And then she looked at me and it's like, so all

the training data is now a massive attack surface.

And Yep. When that's just why I love my wife. So I

I'm wronged. She's never wronged. Well, that's true. But at

first I was like I was thinking but but you're missing and then I was

gonna say you're missing the point which one is never a good thing to say

but Like midway through I was like, oh my gosh,

she's right. Oh my gosh. She's right. So then

when I started talking to other data science and AI types, and I was like,

but but don't you think this could be, like, a big attack surface? I look

like that meme with the guy from It's Sunny in Philadelphia with, like, it's

always sunny where he had, like, the conspiracy thing. Like, I swear I will

like that meme. Yeah. And, you know, and if you

look at the I think OWASP has, like, the top 10 vulnerabilities of LLMs

that is either two or three. Right? So it's

kinda like there's a fine line between,

like, thinking too much about problem, but also kind of thinking ahead of the

problem. I don't know. No. Oh, I think you

cut off a little bit, Frank, but, Andy,

to me, that resonates a lot, and I think it's sort of really the overlap

between data and engineers. And, by the way, like, we didn't even talk

about security. Like, all these concepts also exist in security.

Right? And I think in the same way that we sort of manage, like, you

know, sub zero, sub one issues in security engineering, data

issues should be treated the same way. You should have a framework to understand what's

a sub zero, what's a sub one for data issues. You should it should be

connected to pager duty. Like, people should wake up in the middle of the night

when you have data issues. I think I think that's right. It's

improving, but, we're not quite there. It'll

happen. No. You're right, though. Like, they don't think about this in

terms of they don't does it I wouldn't say it's not disciplined. Sorry,

Annie. I cut you off. No. But my experience we talked to data engineers. Sorry,

Andy. And I I I I am a former data engineer

myself. Like, I thought of it in terms of schema structures and pipelines.

Mhmm. Not necessarily securing those pipelines. Right? Mhmm. Sorry,

Andy. I'll go. No. I was curious. I wanted to to shift back

to you. You mentioned the four areas that your software,

looks over your AI and the observability software does. What

happens when it detects something amiss?

Great question. So not even talking about Monte Carlo specifically, but rather

an observability solution. I think an observability solution needs to

have coverage or an observability approach, by the way. Like, some people build this

in house. An observability approach should take into consideration

your data estate, should take into consideration, right, your

entire data estate. I think, oftentimes, the mistake is people will even if they

build it in house or do anything else, they'll really just focus on, like, the

data and their data lake or the data in a particular report. Like, that's

not sufficient. Right? It it just isn't. And so people waste

a ton of time trying to understand, like, what's wrong and where. So I think

the first is, like, you need you need visibility across the data

state, which hopefully we've defined an unnatural resource that should be

managed securely. And and I think that's right because I

I by the way, Monte Carlo doesn't doesn't do the security

part, but I similarly believe that in the same kind of diligence

that we apply to data as engineering, you want data products to

be reliable but also secure, scalable,

like all those concepts should adapt. By chance, we happen to

focus on the reliability and observability part, but all the other,

principles of software engineering should apply.

We specifically don't do it, but very much believe that should be

the case. But back to your question, you

know, so so what happens when there is an issue?

Very similar to workflow that you might find in Datadog,

New Relic, and and PagerDuty. So there is an alert that goes out,

often you know, in whatever flavor of choice. If you're an enterprise that has a

data state, this is likely Microsoft Teams. If not, this would mean

Slack or an email or what you know, some teams like to have it connected

to to Jira and and pager duty for for sev zeros or sev

ones. And, you know, the first thing

that people will do is start, you know, typically an analyst.

I was I was in, you know, prior an analyst. The first thing you start

asking yourself is, why the hell is the data is wrong?

Right. Yeah. You're like, well, was the report on time?

Was the data accurate? Was it complete? You start going through all

and then you start you basically come up with hypothesis. And then you start

researching those hypothesis, and you're like, well, let me let me

trace the data all the way all the steps of the transformation

and start looking. Was the data okay here? Yes. Check. Okay. Move on. Was it

data right? You literally you started this, like, recursive process. Gotcha.

Before we started the company, I used to do this all manually. So I remember,

like, I would go into a, you know, into a room. Maybe you did this

too. And, like, on a whiteboard, I would start, like, basically mapping out

the lineage. Okay. This broke here. Was the data here okay? Let's let

let's sample the data and make sure it's okay. Okay. Move on. Let's like, literally,

we have this, like, very every morning, actually, you know, that this

became such such a problem because we were so reliant on this particular day

dataset that every morning, me and my team would wake up, and we would basically

go step by step and diligently, like, make sure that the data is accurate,

which I felt like was I was like, this is, like, total, you know, crazy.

So, you know, I think, particularly in Monte

Carlo or, like, what observability does is provides the

information that you need in order to troubleshoot and understand where the issue is. And

so we can surface you information like, hey. There was at the same time that

this dataset you know, maybe the the percentage of null values in

particular field was inaccurate. And then at the same time, there was a full

request that happened. Maybe those are correlated, actually. Gotcha.

Maybe, you know and maybe, actually, you can use you can also

do a code analysis. So you can, like, basically, you know, analyst

what we used to do is, like, sift through lines of code and try to

see what the change. Hey. Why did few surface to you that, like, there was

a particular change in the, you know, name of a field,

at the same time as an example. So bringing all that data into one

place can help you sort of troubleshoot that. And

sorry for another LLM plug, but you can actually have

an LLM do this for you, which is pretty sick where it's like an early

beta test for us. We haven't released it yet. But, basically, what we're

testing internally is for every like, for data incidents,

there's basically, like, an in like, a troubleshooting agent that

spawns agents for each of the hypothesis. So there's, like, an agent that

statement. Yeah. I it's really cool. There's an agent that

looks into, like, the code change, the data change, the system

change, and then and then it does it recursively on

all those tables. So you can actually run up to a hundred agents in under

one minute. And then there's a larger LLM that takes all that information

and summarizes it and synthesizes it. So, again, early days, this is like we're still

building it. Very cool. But the early results are really cool. Yeah. It's

like basically turbocharging your your data analysts and your data

stewards. Sorry. I got all excited. No. It's it is That's really

cool. Fascinating, and I love that you're excited about it. And what one of the

jokes that I make when I'm I'm working with my kids on something, if

they nail something, I'll I'll say to them, you know,

something similar to this. It's like, if you can only, you know, if you

can only run a hundred in one minute, I guess that's if that's the best

you can do, we'll just have to live with it. Yeah. Exactly.

That's that's an amazing stat. Yeah. Yeah. That is interesting. And I

also think too I also think too that, like, observability could help

with secure the security story. Right? Because if, you know, you're looking at a

pipeline and it's like, hey. Weren't there a bunch of

sketchy looking IPs, like, poking around our system about the time that this

pipeline ran? Maybe the rest of the data that goes out of that pipeline

run is a little bit suspicious too. Yeah. A

%. Like, we we you know, for example, you work with a,

call it delivery service, and there was a very

suspicious tip very suspicious

amount of tip that was given. Like, you

know, you can imagine, you know, the range of tips can be between x

dollars and y dollars, and suddenly that's, like, you know,

10,000 times y, like, 10,000 times the upper limit.

Yeah. You know, triggers off a suspicious alert. It's

not a normal tip, and it's not a mistake. It's actually, you know, security

issue. So that's an example. Yeah. Interesting. Yeah. I

love the anomaly detection aspect of that. I mean, it just it

it's it's something that we've been doing for a long time,

but then at wrapping it with automation and then

combining that automation with what you just described with all the

agents running down all of the permutations, that

that just sounds amazing. Yeah. It's really cool. I can't

take credit. This isn't me. It's it's it's my team. But,

but I I was like, woah. It's like a hundred bars

running at the same time under one minute. That's amazing. There you go. It's really

cool. Probably smarter than me. But yeah.

That is so awesome. That is cool.

So we we generally have is, we have kind of our

our stock questions that we ask, if you're interested in doing them.

They're not we're not Mike Wallace. We're not trying to I don't even think

anyone gets that reference anymore, but we're not trying to catch you in a,

I gotta come up with a new one, in a thing. But it's mostly, like,

how'd you find your way in the first one is I'll get the rest of

them, up for you in a second. But the first one is, how'd

you find your way into data? Did did the data did you find the data

life or did data life find you? Oh, that's such a great

question. You know, it's funny.

I grew up you know, my my, my mom is a meditation and dance

teacher and my dad is a physics professor. And so,

yeah, and so I, I, you know, grew up with very sort of like, yin

yin yang in my family, if you will.

At a very early age, I used to, like, hang out in in my dad's

lab and, like, do scientific research and stuff like that. So or, you know,

like, very at a very young age, my memories are, like, sitting in a

cinema, watching a movie with my dad and trying to, like, guesstimate how

many people are sitting in the in the audience.

Right? Yes. Just like, you know, I think for, like, a five year

old, it's sort of like a fun fun thing. But, you know, throughout my my

adulthood, like, always sort of had that in in the background. And,

you know, I I think later on in life, I sort of always gravitated towards

data. And when I decided to start a company,

I was actually debating between various areas

like IT and actually blockchain, or, you know,

crypto for a little bit and and data. I think at the end of the

day, like, my heart was really in in data. If I look at, like,

the next ten, twenty years, it's pretty clear to me that data is

gonna be I think it still is the coolest party, and I think it

will be the coolest party to be in. And I personally,

like, you know, it's it's it's funny. Like, throughout my my

career, I've I've also learned the limitations of data. Right? So so data can

tell you whatever story you want. It could tell you, you know, for every question,

it give can give you a yes, and you can also tell a no story.

Right? So so there's also limitations to data,

but but I always have been fascinated,

by by data and space. So can I say both? That's

Yeah. I mean, that's fair. That's fair. Good answer. That's fair. Yep. So

what what's your favorite part of your current job?

Oh, that's hard to choose. I love my job.

I just love it. I think, you know,

the ability to work with customers and actually, like, change the way they

work, I I think that's probably the biggest gratification that I

get, you know, from from my my career. Like, the fact that you can

actually work on something that matters is pretty insane. You know? And when I think

about, like, the future, I'm like, what? So data is gonna be wrong? Like, we're

just gonna be, you know, making decisions off of wrong like, what? I don't

wanna live in that world. You know? And so Yeah. I think

there's something that's, like, really fulfilling and helping, you know, drive a mission that

I believe in that has an impact on customers. And, you know, when customers will

tell me, you know, I started sleeping at night because I

know that, like, I have some coverage for my data. I'm like, yeah. Oh, wow.

I'm glad you're sleeping. You know? Like, good for you. I love

sleeping. So What a cool thing to hear. Yeah. Exactly. I

think that's that's probably, you know, maybe one part. And then the second is, like,

just working with an amazing team. You know, I I spend most of my my

day maybe kinda like, you know, you guys, like, hang out having fun,

laughing. So, you know, I I I'm very

grateful that I get to work with the smartest people on on

worthwhile challenges. Oh, very cool.

We have, three complete these sentences. When I'm not

working, I enjoy blank. Sleeping.

I yeah. I I have a I we recently

have added we we had two kids, and we adopted a cousin. And

I forgot how draining a toddler can be. And I'm

I'm eight to 10 years older since the last time I had a toddler, so

it's like I, I have two

kids, on two under four. So I,

respect the sleep even more. I I can't even I can't

even wrap my head around that. It gets it gets better. I can say

that. It's my own role. I appreciate that.

So our second one is I think the coolest thing in technology

today is blank. The coolest thing in

techno I think the pace of innovation. I think that's really

freaking cool. You know, you can, like, work at a problem today and you're like,

you can't solve this. Two days two days later, a new model will come out.

Boom. You're done. So it's harder. Right? The bar is

higher in order to, like, actually like, it's it's harder to it's

harder to know what to bet on. It's harder to know what the future will

look like, but it's a lot more exciting. So I'm in it.

Cool. Our third and final complete sentence is, I look forward

to the day when I can use technology to blank.

I was always a big fan of teleportation. I think teleportation is really

freaking cool. That would be nice. Can't wait for that. That would be cool.

That would be cool. You know, you're not the first person to answer with them.

Oh, really? Yeah. It's pretty cool. Pretty cool. Sorry.

Number six is share something different about

yourself. Something different.

Yeah. Something different. Let's

see. I mentioned I have two kids. I

meditate when I don't sleep. I like to meditate.

I, what else? I'm married to

my cofounder. Oh, wow. So we,

yeah, we're fortunate to share our lives both at work and at

home. That is cool. Yeah. I can

imagine that would work out really well or not. Like, there's not a lot of

middle ground there. High risk, high reward. High risk, high reward. I

get, like you know, my wife is, you know, she's a

federal employee, and she's, you know, reevaluating what her career

futures look like, you know, and she's like, you

know, I was like, well, you know, you could help. You can start

a new podcast. I can help you with that. She's like, yeah. But then I

have to work with you. And, like, I know what she meant. I know how

it sounds. I know how it sounds, but I know what she means. Like, so

when she did work from home, like, there was literally a, like, an entire floor

between us because Yep. Like, it's too loud. I'm too loud. Yeah. Yeah. Yeah.

Yep. We're very loud too. So

where can folks find more, learn more about, Monte

Carlo and, and and what you're up to?

Probably, I'm the place where I hang out is LinkedIn. So,

I know we just got connected on LinkedIn. That's great. Probably follow me

on LinkedIn or, honestly, reach out to me directly, me,

Moses@MonteCarlodata.com. I hope I don't get a lot of phishing now because

of that. But Well, hopefully, make sure it's the right account because we found out

in the process that there's there was another suspect in

suspicious looking account. And I also think that for our

listeners, it's worth pointing out that I think that people have realized that LinkedIn is

a is a major security vector because I've been getting a lot of

weird a lot more lately. Now I don't think it's related to

the, the refrigerator scandal. Andy and I will do a whole show on that

later because there's there's actually an interesting AI component to that. Okay.

Good to know. And finally, last but not least, Audible

is a sponsor of the podcast. Do you do audiobooks? If

so, recommend one. Otherwise, just recommend a good book you recommend.

A good book. Let's see.

Thinking in bets by Annie Duke.

Professional poker player. Interesting. In in how

lessons from poker can be applied in, in life

and in business. Interesting. I

once worked at a financial services company, and one of the

big shots used to play online poker. And

they're on company, not on company money, but on company time. And a

lot of people Not a lot of people took a dim view of that.

Rightfully so. But he was

making so much money. You know, people that matter didn't take a damn view to

it. When he stopped making so much money, people everyone took a damn view to

it. And it they don't that does end the the story. It

is on I don't see if it's an audio oh, it is

an audio book. It is an audio book. Awesome. I'm gonna add that to my

list. I'm done. Okay. And if you you know, they are a sponsor.

So if you go to, the datadrivenbook.com, you know,

you'll get a free audio book on us. And, you know, if you sign up,

we'll get enough to, you know, buy a coffee.

Maybe not tip them $8,000, but, you know,

we'll get enough for a Starbucks maybe. Maybe. Yeah.

I just tested the link, Frank. Every now and then, we had trouble early on

with the link coming and going. So I just when you saw me turn away

a minute ago when Frank started to this question, that was me typing

in. It worked. It worked.

It's always DNS. That's the Always. It's interesting

you mentioned that. I read an article. Actually, it was a newsletter recently that talked

about, betting being the first stage

in, kind of the path to minimally viable products. And

I thought, now that's curious, and I don't know again, I haven't

read the book. I will listen to it. But the idea of

engaging your team I I manage a team, as well.

And engaging the team by having them do

interesting things and making taking these very large bets

that look nearly impossible,

perhaps. And it's like you said, the the the problem

comes up, and you're thinking this is this is unsolvable. And two days

later, it's solved. And over and over again, I've had that

experience, but I never tied it to the concept of

bets. And I saw this this newsletter that talked about do

that first, And it reminded me a little bit

of Collins talking about, the the big hairy

goals, you know, back in the day. It's very

similar to that maybe in concept. I don't know. I'll have to listen to the

book and check it out, but I was intrigued by the newsletter. Yeah.

There's interesting concepts. Like, I think some of the ideas is, like I mean, even

when you start a company or sort of, you know, start working on a team,

like, you basically have you have a set of cards, which are, like, your strengths,

your weaknesses. And so how how do you play your cards? Like, you can't you

know, if you wanna win around, you can't play with someone else's cards.

You are what you are. And so the best thing you can do is play

with your card. I think that's true for a team solving a problem or startup

or whatever it is. I love that. Yeah.

Interesting. Any final thoughts? This was so fun. Thanks for

having me. Thank you. Thanks for, and you did mention kinda offhand early

on. I don't remember if it was in the green room or not. You have

a podcast yourself? I do not have a podcast myself.

Alright. That was my mistake. Maybe I'll end it tomorrow. Okay. All

good. Life goal one day. There

you go. There you go. And with that, we'll let our AI finish

the show. And that wraps up another data packed episode of

data driven. A massive thank you to our brilliant guest, Bar

Moses, for taking us deep into the world of data observability,

sketchy LinkedIn impersonators, and the dark arts of tipping

anomalies. Who knew a dodgy schema change could cost more than

a luxury sports car? Now, dear listener, if you've made

it this far, you clearly have excellent taste. So why not

put that good judgment to work and leave us a rating and review on

whatever platform you're tuning in on? Apple, Spotify,

Pocket Casts, Morse code, however you get your fix, would love

your feedback. And dare I ask, are you subscribed?

I mean, you wouldn't want to miss out on future episodes filled with more

wit, wisdom, and the occasional fridge based conspiracy,

would you? Until next time, stay curious, stay

observant, and for heaven's sake, keep your data tidy.