I grew up in a mining town, and I don't recommend
that actually, as a place to grow up. Got
partially homeschooled by my mom and dad because the quality
of the local schools was not what they wanted to see. I
saw resource production up close and personal. I saw data
coming in up close and personal before anybody knew what to do with it
or how to use it or analyze it at all.
Statistics on operations, on interruptions, what
causes machines to go down and be unavailable for all
these kinds of things. Data can help us understand that.
Hello, and welcome back to Data Driven, the podcast. We explore the
emerging industry of AI, data science, and, of course, data
engineering. You may notice that my favorite data engineer in the world,
Andy Leonard, is not here. However, I brought the most quant,
most quantum curious person I know. I don't like calling her curious, although
one could argue. And Candace Cooley. How's it going,
Candice? It's great. Thank you so much. I'm really excited to be part of Data
Driven today. Awesome. We're happy to have you. And we're also very
happy to have Mr. Dale Nesbitt, who is,
in addition to being at Arrowhead Economics, he's also a
lecturer at Stanford, and I guess he has his first summer class today,
and we're happy to have him. Welcome to the show, Dale.
Thank you. I appreciate it. And thanks for the opportunity to speak with you.
No problem. No problem. What exactly does
Arrowhead Economics do? It's implicit in the name
we do economics. We named after
ourselves after the second Nobel laureate in economics,
Kenneth Arrow, who was riding his bike around Stanford
happily until his 96th birthday. And then. Then we
lost him about five years ago. So we do
economics in the energy patch, critical materials
patch, any commodity that's. That's
produced or traded. Interesting. And that's a.
That's a very large field. I don't think. If were it
not for Eddie Murphy's movie Trading Places, most
people probably wouldn't know a thing about it. I was a kid when that came
out, and I was just fascinated. So sorry. I'm sure that's not the first
time you've heard that, and I hate to tell you, it's probably not gonna be
the last time you've heard that. Well, yeah, I've heard that.
I had an auspicious start. I grew up in a mining town.
Okay. I don't recommend that, actually, as a place to
grow up. Got partially homeschooled by
my mom and dad because the quality of the local schools was not what
they want. I saw resource production up
close and personal. I saw data coming in up close and personal
before anybody knew what to do with it or how to use
it or analyze it at all. Statistics on
operations, on interruptions, what causes
machines to go down and be unavailable for all these kinds of
things. Data can help us understand that.
Interesting. So you were the data. Life really found you? It sounds.
Yeah. Yes and no. Yes. My background is actually in
probability and decision analysis. That's where I did my research.
And what. One of the things that you're going to find the data is really
good for is developing probability distributions over
phenomena that you don't understand the uncertainty about.
We want to understand the uncertainty about certain phenomena.
Data is a way to do that. Interesting. Sorry,
Candice, I cut you off. No, I was curious. So many organizations, they
collect enormous amounts of data, but why
do so few of them seem to not be making better
decisions based upon their data? As a professor of decision
analysis, you need more than data to make decisions.
You need alternatives. You need information, probability
distributions, not data. And you need a notion of your
values, your objectives, what do you like and what do you not like. So if
you're coming like profits, there's a lot of uncertainty
in the middle. If you're producing oil, there's a probability distribution
over oil price, over supply chain costs and those things.
You can't make a good decision unless you have those probability
distributions. I get them in part from data.
The data is out there telling you what these probability distributions
are, if you know how to process it. The reason is because
the notion data driven decisions is a non
sequitur. It really is. Data is not
enough to drive your decisions. You need intelligence,
you need an understanding of uncertainty. What's the on
average value of these variables that you're going to see?
How much spread is there in those variables? You can't make a decision under
uncertainty unless you understand the uncertainty.
So data driven is not data driven, it's analytic driven. And
the data helps you. If you, some of the techniques that, that
we use in practice get what those uncertainties are
to feed into a decision analysis
framework, then you make good decisions.
Yeah, I mean, that makes sense, right? Because data would be the raw
commodity, if you will, the raw oil. But the thing that
you put in your car, gasoline, petrol, whatever you want to call it,
is the refined product from the raw material, you know, Frank?
Absolutely. Data is the crude oil and
the probability distribution of the finished product, gasoline,
distillate, so forth that you put into your machinery that makes it go
absolutely Data is the raw material.
And one of the problems there, a Candace, pursuant to your remark,
is this is Nesbitt's maxim number three, people
only gather data that's easy to gather.
And maximum number two is the process of gathering
data is itself stochastic.
Watch how data is gathered. It's a noisy, noisy process
to observe anything and to write down the
correct, say, operation of machine. We
don't even know what the price of crude oil is because everybody reports
it differently. So data itself isn't
gathering itself is noisy. You have to take that into account when you
analyze it. So this idea that data is. It's like
Rumpelstiltskin, right? Spin straw into gold. You can't.
The data is not straw. It has to be scrupulously
worked. Frank, just exactly what you alluded to to get
proper decision making, to get proper
forecasting and those things.
Well, then doesn't that explain where a lot of data projects or
AI projects go wrong, is because they don't take into account the random.
The randomness. Because you said data collection is stochastic. Right. So
it is. It's inherently unpredictable.
Right. And doesn't that kind of echo throughout the training of these models
that we have and cause weird outliers?
Or does is there magic that can happen that can cancel that
out? Yeah, there's. Well, there's magic. You try to measure
through observation the intrinsic uncertainty in the data, like
the difference between predicted and actual, maybe due
to measurement error, just observing the data. And
the great statistician Fisher told us that data was
generated by a stochastic process. And what
you're trying to do is figure out what that stochastic process must
have been very subtle. So you,
you, at the same time you're estimating, say, coefficients for your
model or something you want to estimate, you're also estimating what
stochastic process was at work when the data were
gathered. And you can infer that
it's very sophisticated. AI doesn't do any of that.
Regression analysis does that. AI
doesn't deal with uncertainty at all. No,
it actually gets really wonky when it's uncertain.
Yeah, wonky being a technical term, of course.
But I think it's really important when you think of data analysis. You want to
be careful to try to measure intrinsically what sort of
stochastic variation, what sort of process by which people
observe data. What was it? When we
do regression analysis, simple linear regression is a term,
sigma in there, and that sigma has to do with the random nature
of the data. So
interesting. So, yeah, it really goes back to the fundamentals of
statistics, doesn't it? Yeah. I mean, think of the, you know,
the clock's always right. Suppose you're measuring the time
and you fail to notice that the clock broke.
And it gives you the same time all the time. Your sensor
broke and it gave you the same reading, no matter what the state of the
machinery was. Your data set ain't too good with
that in there. And yet, wow. No matter what, I get the
same answer. That's really predictive. Fact is, it's a broken
data machine. Right. So you get just
industrial measurements. Your, your process of measuring,
be it manual or sometimes be it automated, you have to
be very, very cognizant of
that and the stochastics of that
and take that into account when you try to understand what that data
implies about your probability distribution over something
interesting. Candice looks like she's
going to say something. I'm thinking about uncertainty,
and I'm also thinking about risk. And I'm trying to understand
what's the distinction that matters between risk and
uncertainty. Well, it's, it's kind of interesting. So suppose
that. Let's take the price of gold. You're going to open a gold mine. Price
of gold right now is $4200 an ounce. It's
infinity minus three. It's really, really high.
Wars and things like that caused that to happen.
But you're uncertain about it. But if you're going to open
that gold mine and it's profitable at a gold price of
fifteen hundred dollars an ounce, there's uncertainty, but there'
risk has to do with loss. So suppose you were
uncertainty. One day I grew up in Reno. One day I walked into the
nugget, I was underage, and I inadvertently pulled the handle on the
slot machine and it played. You didn't have to put money
in it and it would just play. It was broken.
That's a pretty good lottery. I played it until the security guard came and
kicked me out. Right. So there was
uncertainty. I didn't know if it was going to give me cherries or clowns or
whatever, but there was no risk. I didn't have anything at risk.
Risk happens when you have losses that have to be balanced against
gains. Now all of a sudden, your elementary tech
gets a lot tighter when you have losses. I had to put
a quarter in that machine. And so if it didn't come up, cherries
are better. My quarter was gone. So risk has to do
with the probability of loss and how you,
you trade that off with Your preferences against probability of
gain and uncertainty just has to do with how
sure are you. If you had to do an over under on your
profitability, what's the 50, 50 point
risk estimate? That makes a lot of sense. Yeah, that makes a lot of
sense. That's interesting. But
what's your take on. Obviously the commodity that everyone
is most impacted by, at least in the obvious sense, is oil.
Crude oil. Yep, Crude oil. So
I've noticed obviously there's been some, you know, the elephant in the room, there's been
a lot of instability in that, that space, but somebody had said something and
I didn't quite get it, is that the oil
contracts are down a decade out or five to ten years
out. Yeah. So the fluctuation in
prices that we see at the gas pump have more
to do with
speculation. Is that true? No, Did I mishear it?
I don't think it's true. This has to do with the short term supply demand
balance. When the strait of Horamuz was closed, you had 10
million barrels a day less supply at the gate of
the straight of horror moves. So draw yourself a supply
curve and a demand curve and shift that supply curve 10 million
barrels of the day to the left. Your price is going to go high.
Yeah. No matter what we have. And I do this and
there's a different kind of data I'd like to chat about. We have a world
oil model, multi regional world oil model. We forecast this stuff for
the industry short term and long term. Okay. And
so short term is different from the long term. All the oil that's going to
be produced is sitting right there at the wellhead and you have to have
logistics to get it to market. So the supply curve and the demand curve and
the short term are fixed. In the long term they're not fixed.
People can invest capital and go produce some tar sands in
Venezuela or produce some more oil in the Middle East.
And so longer term price effects tend to look a lot different
than shorter term price effects. But both are uncertain.
Both are. So if we gather data on some
phenomenon, if it affects long term prices,
then that data will have a stochastic effect. Right.
It helps you understand them better, helps you reduce your, your
uncertainty of what those prices are going to be. The more data that you gather,
the more definitive, I. E. The lower variance your look.
So a lot of these comments that you hear, they give me
hives. I thought I was back in the 1970s.
The stupidity in these comments, number
one, the biggest stupidity in the world, is that the future price
is an extrapolation of the past price. And people do
that statistically, it's dead wrong. Economics teaches us that
the future occurs because of future
supply and demand has absolutely nothing to do with the past.
So AI is going to have a really hard time with any economic
problem because the future price that you're going to
see, say right at the gate of the straight of horror moves, it's going to
be a function of what's happening then, not what happened now.
People will speculate into the future. They will solve. They will sign
a long term buy or sell contract. I'll give you a million
barrels a day in 2040, okay, I
better have that crude oil in 2040 for you and
you'll sign a contract that I'll buy it. They sell, buy and sell
futures. That just makes the price more and more available
and visible to everybody.
Speculation is a good thing. It's a very, very good thing.
The best thing is speculation trading.
Because it takes risk away from the producers, right? Or no,
it shows you the price. If you watch how people make these
trades, if all of them are made at an implicit price of $80 a
barrel, there's your best guess at the price. Speculation
shows us the price. So this idea that speculation
is bad is really stupid. It's so stupid.
You want more and more and more and more trading. So everybody knows the
price. So the price in the fair, the free market is known to all.
Interesting. That's why people
trade, right? So you don't have to, you
don't have to be data driven about the future price. It's transactional.
Right? Okay. We have short
term disruptions, there's not enough trading. And
until a point, because you're trading on the probability
that the straight remains open versus vibrates
between open and closed versus closed.
And people will speculate that. Thank God for
speculators. Interesting.
What's your take on these poly market type
marketplaces? What's your take on that?
People are speculating. You do? Okay, I love it, I love it.
I think people are betting their probability assessment against yours. They
think they got a better probability distribution of something over yours and
they put their money where their mouth is, they trade on it. And if you
look at the volume of trades that people lay down, that gives
you some notion of the cons, consensus probability of those events
occurring. So the what like on the election?
Speculating on the election. I love those things. I don't play them, but I love
them. Sportsbook is that sports books are
great. They're so much fun to play too.
Bad they rake off so much of the pot as their fee.
But you can watch the data on those and I think
the data suggests that they're very pretty. Pretty
predictive of what happens in a probabilistic sense.
No, I remember one of the. There's a paper from a long
time ago and basically discussed how if you remember the show
who Wants to Be a Millionaire, the crowd, when you ask the crowd a question,
the crowd was more accurate than any of the other mechanisms
that they had, which was phone of friends pass
and something else. But the
crowd actually got the answer right. Far and above.
Yeah. That book, the Wisdom of Crowds. That was it.
Yeah. And it's very good. See, the market is the ultimate wisdom of
crowds. The crude oil market is so heavily traded. There's
no misinformation in that price. There's even a
theory which I subscribe to, that there's nothing the matter with insider
trading. It gets more information into the price and
more correct information into the price. It's
illegal because people say it's unfair. It's.
It's capitalizing on. On insider knowledge and all that.
I'm not so sure I buy that. I want the. When I buy somebody's
stock like SpaceX, I want all the insider and
outsider knowledge involved in my decision whether or
not to purchase that st. And it wasn't all
the insider knowledge was concealed from the market by law.
Yeah, no, that. It's funny you mentioned that because insider trading
has only been illegal since the Great
Depression. Yeah, I remember one of the
most interesting books. It was called the Patriarch and it was
basically about Joe Kennedy. Oh yeah. And
FDR put him in charge of that. But he was a notorious insider
trader. Oh yeah. As you would point out later on, it wasn't
illegal then. Right? It wasn't illegal. It's not even. It's not
illegal in Congress today. Well, yeah,
exactly. I mean there are people who are financial
geniuses. This is crazy. I mean, really. Yeah. These
guys have confident insider information, it's
confidentially delivered. Confidentially. And you can go off
and trade on it. Come on, guys. It's
not a practice that be condoned. Yeah. Shutting down insider
information is about fairness. Then they should be included here.
But we're a little far afield. Getting back to data science, there's a couple of
other things that. One of my. One of my favorite
themes. I saw this during COVID go back to the beginning of COVID
How much data was there on what was curative
or what was ameliorating of symptoms about any treatment
we had, there was no data. Not a lot, but
there was a lot that was suppressed. If it's suppressed, it
doesn't exist. Okay. If you can't analyze with it,
then there is no data. It's like if I flip a coin and put a
piece of paper over it and ask you to call the coin,
it's still 50, 50 to you. I can look under that paper, it's 100 to
me. I know what it is. Your state of information is still 50,
50. You don't have any information when I have it.
So you're going to have to take that into account. So
people were coming out and saying, oh, hydroxychloroquine is going to have
prophylactic effects and curative effects. That's my
subjective judgment. And my brand of data analysis is called
Bayesian, where you come to it with a judgment in
advance and then what data does is
revise your judgment. So,
so people were treating people with hydroxychloroquine and they were
on TV to the extent they could get on. This is great.
60 year old medicine is going to cure it all. As the data began
to came in, come in. If you'd have been a Bayesian,
you'd have said, my initial statistics on
the curative powers of high gloxychloroquine are being
totally not supported by the data. I would have changed
my judgment. That's what data analysis is for,
to get you to change your judgment.
So valuable. You know, you start off an AI
program with judgment as to what all the parameters are. Data
stops the judgment process and makes those parameters
consistent with observations. That's powerful, isn't it?
Yeah, very powerful. And so
hydroxychloroquine fairly quickly started
the, the curative and the, and the amelioration
properties of that drug were pretty quickly
contradicted by the data. Okay, Same with
Ivermectin, the horse tranquilizer. All right. Horse
tranquilizer is going to cure everybody. And, and
pretty quickly contradicted when they tried a little bit of that in small
case studies. It, you know, you might as well have been taking a sugar pill.
And so then we get to the shots and all the people said,
these shots, man, they're going to be magic. They're going to stop transmission.
That's my judgment. We're going to stop transmission. We began to gather
data. The shots were extremely valuable. They didn't stop
transmission, but they cut the severity of the symptoms by three
orders of magnitude. And so they changed our judgment.
The point of data science is get the data to give you
the best possible judgment of probability. That's
what data science is all about. Gather the right data that
can have the most effect on your judgment. That's what we want
from data, is our judgment as to how these
processes work. That's cool, isn't
it? So very valuable raw
material to making me a smarter person.
Having better judgment is consistent with a gazillion observations,
each of which is stochastic in its own right, but they're not
completely stochastic. It's not random. There's a lot of systematic
stuff hidden in there. So the idea of data science is, is
gathering data. If you don't just go out and gather the data, that's easiest to
gather. Gather data that you really need to have
to figure out whether the shot was going to
be strictly prophylactic or whether it's going to cut the
transmission. Well, we figured it out after a while, got enough data that
said it's not going to change the transmission. And you've heard all the political
bitching and moaning about that. Oh God, it's a lousy shot. It doesn't cut
the transmission. It saved lives by cutting the severity.
And that's statistically valid. And so
big success. The data gathering in places like Israel where shots were
mandatory was really interesting. Really showed the prophylactic effect.
God, that's valuable data, isn't it? That's
trillion dollar data. Is it allowed people to
make better decisions? Candace, you're right on the mark. Once you know
that, you can make a lot better decisions both collectively
and individually. I decided to take the COVID
shot because I did a decision analysis of it using my
subjective probabilities. It was a no brainer. How could you be an
anti vaxxer when you made that kind of assessment?
I've done that with allergy shots. Great example. I take
allergy shots, I'm a severe allergy patient. I went in to take
these things. Well, you know, they know what you're allergic to and they stick it
directly into your bloodstream. That's what an allergy shot is. If you're allergic to
peanuts, you get peanuts stuck in your body. And so I asked,
well, what are the, what are the problems here? Well, you know, you can die
of anaphylactic shock. You can drop off the bench and be dead in 30
seconds. Good. That's really a good thing to do. And
they might not work. So I looked at the statistics
and as near as I could tell, about 70% of the time
they Provided significant relief and it was about
1 in 10 to the 7th. So one 10
millionth probability that was going to drop dead and so
more likely to win the lottery or find another. You know
I looked at my values and I rolled back my decision tree. It was a
no brainer. 50 years I've been stepping up to the lady and
having her stick peanuts in my arm which I'm deathly allergic to.
And so you there also other things too that you're
in the doctor's office when that happens. So like if you do like they could
administer isn't that kind of alter the risk profile too? It's not
like you're. I carry an amputee. You wouldn't. I carry. Yeah, like so
you wouldn't go on a rowboat in the middle of the Ocean without your EpiPen
and then take that shot. I did but you know you're not supposed to. All
right, but, but I mean that would also. Yeah but in that
simple example the statistics of 10 to the -7
and the statistics of 70 probability of
amelioration of symptoms, that's the billion dollar data
right there. Simple minded data but
it helps you make can the right decision. I got that out
of the data but I also had to analyze it to find out
how that interacted with my preferences. So it
data analysis is critical to getting the information to
make a good decision. That's what I think. I didn't even
do any analysis. I just looked in books and saw those probabilities ain't
that great. So every time I go in there, hey, I might never walk out
of here again but the odds are pretty low.
That's a great point. So we want to be respectful of your time. It's
almost 40 after the hour. Where can folks find out about more about
you? And I know Stanford does a lot of these lectures. They post them online
or any of yours online. They haven't
been. No, they haven't been but that's why I'm talking to you. Maybe I'll do
that. And if you want to continue. I haven't told you much about data
analysis per se. If you want to fire up tomorrow, I'm
glad to do that. Yeah, let's. Let's talk again soon. We
want to know the uncertainty we're looking at. And the more data you
get, the less uncertainty you have. It gets pretty definitive if you
got 60 terabytes of data but we don't have that
candice. The other thing we'll talk about next time is in real decisions you don't
have any or very little data that's relevant to the decision you're
making. You get the data that somebody gathered on a government
grant, like crime statistics. There's
nothing in it. It's just data that's easy to gather. Okay.
You are awesome. I look forward to speaking with you
again. Yeah. With that, we'll let the
outro music play.