On this episode of data driven Frank and Andy interview
stephen Oren, the CTO of Intel Federal
yes. Intel, the computer chip company. Because if you want
to train your AI models in a reasonable amount of time, you need better
hardware. Well, it turns out that intel has developed new
CPU instructions to accelerate AI workloads
FPGAs allow for faster development in custom
applications with specific needs. Speaking of intel,
you have to check out an upcoming intel and Red Hat webinar
link in the show notes. Tell them Bailey sent you.
Now on with the show.
Hello and welcome to Data Driven, the podcast where we explore the emergent fields
of data science, data engineering, and of course,
artificial intelligence. As with me, I always have Andy
Leonard, my most favorite data engineer in the world.
And today we have a special guest, Steve Oren, who is the federal
CTO of intel. Yes, that's right, intel, the chip
company. And although they do a lot more stuff
now. So welcome to the show, Steve.
Thank you and glad to be here. Frank and Andy cool.
So one of the things that I think people have not realized, people
think that AI is a software story, right?
Primarily. But quickly, once you get into it,
everyone goes gaga for things like Chat GPT or
well, no one's really gone gaga for Barred just yet. We're going to give that
a few more time for the paint to dry on that.
But quickly, I think when people start
becoming builders of AI tools, the
number one restriction, aside from kind of what your data engineering pipeline looks
like, is how quick you can train these models. And
obviously, I'm pretty sure intel has a thing or two to say about
hardware. Absolutely. And as you've as you've
alluded to AI, and all the things that make up
AI rely heavily on the infrastructure that you're
training you're inferencing. But even before you get to the fun stuff, how do you
do the data curation? How do you suck in the data? The ingestion get the
large multi node data sets that these large language models are
trained against. There's a lot of hardware and infrastructure that has to make
that happen. And then when you get to the important phase with how do
you train those in a timely fashion, hardware is
the answer. And what we're seeing in a lot of these spaces,
especially we start looking at things like large language models and transformers
as well as looking at other approaches that are coming out,
is that not only does the hardware matter, but the type of hardware
matters. If you think about it, it's not a one size
fits all. It's a heterogeneous architecture to make sure you have the right
hardware for your workload. One great example. So
large language models in graph analytics requires not just
heavy duty hardware but the right memory architecture to keep those nodes
in place while you're training. And what you find is that often
doesn't fit well. Intel just a classic GPU only kind of mode,
which is what the classic AIS leveraged, just the sheer number
of cores that you would have in a GPU. And so what we're seeing is
optimizing the hardware for the kind of workload is the answer to getting
timely training. And especially when you start doing more. That sort of iterative. And
feedback training, it's not a one and done, it's an ongoing process. So you need
that to be quick enough and powerful enough and robust enough to handle those
workloads. And then the other side where hardware really starts to matter is on the
inferencing, you want to be able to ask the question and get a response fairly
quickly, if not near real time. If you're in a car and it's
autonomous driving, you want it real time. You want to know that's a tree and
not a shadow. If you're talking about online and doing some
fun stuff with chat GBT, you still don't want to wait 20 minutes for your
response. And so inferencing matters, training matters, and
the kind of hardware and infrastructure that support it. And that's why intel
and our ecosystem are looking at providing a
heterogeneous set of architectures. So our classic CPU, so the Xeon and
the server and CPU and the client core, but also FPGA
based logic AI accelerators like our Habana chips in
the cloud and our targeted edge AI
chips like Movidius for video processing and the like. But then
really, besides the hardware, it's that software infrastructure layer. How do you
optimize your code? Because most AI developers are not hardware
experts, nor do I want them necessarily to be. So a lot of it is
about building out those abstraction layers that optimize your code, that's
doing your hugging face or whatever, to take full advantage of the
hardware underneath you, without you having to know what hardware is underneath you so
that you can provision your workload where it needs to go and not have to
worry about the hardware infrastructure. And that's part of our overall strategy. And working
with the broader ecosystem, the open source community, the
commercial providers, and the software frameworks to give them the
tools to get the best performance out of their AI and their
data science, right? And I think you hit the nail on the head. I
think we're at an inflection point. Not so much in engineering,
right, but more in the perception, right? Because whenever you think, oh,
we have a large workload we got to do, let's throw some GPU at
it, right? And it's a little more nuanced than that. I think
people are finding out that you need more than just a
bunch of GPU. And I was on a call
and I want to get your thoughts on this, because he said something very similar
to what you said. You ever have these moments
when you're on a call and somebody smart says something, you're like, I don't know
about that, right? And it's kind of like what they did
in World War Z and where there was like the 10th Man Rule, where no
matter how ridiculous it sounds at first, you kind of want to
investigate it. And that's why I was glad when your
name popped up in the feed because I'm like, yeah, I want to talk to
you about this. Because he was basically saying that
GPU usage is
overrated and that where the real advantage is going to be
is going to be in software acceleration and on
CPU kind of optimization too, which sounds
a lot like what you said. And when I first heard that, my first thought
was, I don't know about that, but this guy's plugged in. He's a
big shot at Red Hat, right? He's plugged in, he knows a lot. And I
was like, I didn't want to just dismiss that. Like, if my cousin said that,
I'd be like, yeah, okay, but if this guy says it,
whether or not he's right, maybe yet to be determined,
but the fact that he believes it means that there's a trail there to follow.
So I've been kind of poking around at stuff. Tell me
about that. It sounds like there's some weight
behind that opinion. So Frankie, you hit it on
the head there. It's not that GPUs aren't important, it's just GPUs
aren't the only and best solution for all aspects of AI. And there are
certain vendors that want, again, for a variety of reasons, want GPU to be the
foundation for all of your AI activities. Like if you're a GPU based
hardware company. Exactly makes sense. But
when you actually go look at the benchmarks across multiple and here's the key thing,
across multiple AI types. So different
algorithmic models as well as the flow, so there's different stages. So the
inference versus training, ingestion and curation
versus the training, versus the feedback training, what you'll find is
that GPUs will rock for certain things and they are important for certain things,
both from that vendor as well as from a variety of other vendors. GPUs do
play a key role, but when you look at the breadth of AI activities
and the benchmarks associated, you actually find that a lot of really
good work just happens on standard commercial off the shelf CPU. And
actually most of the inferencing, I mean, we're talking in the 70% to 80%
of inferencing happens best on CPU
and areas like large language model and graph analytic based
approaches. The numbers really show very
clearly that it's not a core bound problem,
it's a memory bound problem. And so having efficient in
and out of memory, which is what you get from a CPU or an accelerator
with ample memory on board, is actually much more powerful
for training those types of data sets because the GPU you're dealing with that
latency across the bus. And that actually starts to matter when you're
talking about billions or trillions of node graph
analytics. So I wouldn't say that GPUs
are a dying breed. That is absolutely not the case. And there's going to be
a huge market for GPUs or GPU like
functionality. I want to be careful about that because you don't have to have a
discrete card. The reality is you can have GPU capabilities embedded in your
processor. We've already seen from intel and from other
architectures. The real interesting thing is making sure that
whatever your workload is can be optimized, like your friend said, optimized
through software to that hardware. So that if you are
running a large language model, that you're actually
running it on the right hardware, and that the hardware and your software know how
to work together to give you the best performance if you're working
on. I'm seeing a lot of really cool things right now around graph based
approaches in the memory intensive side of that
and the switching back and forth between that. Those
latencies can really come to bear when you're talking about cross bus
kind of communication. So having high amount of memory available directly to
the CPU to be able to do those training, keep all that data in flight
so you can train, is going to be one of the key
differentiators of how you can take those large angle models, apply them to
more than just writing cool essays by Shakespeare.
I think what we're going to see is things like chat, GPT, and that whole
category of transformer based approaches applied to just about everything, not
just chat, but prediction
approaches. And it's really about getting it the training sets to become
smart on those very vertical domains.
That's going to be a resource intensive process and it's not going to be throwing
a bunch of GPU or it's going to be a lot of cloud scaling and
it's going to be a lot of memory intensive activities. And like your friend
highlighted, the software is going to really matter, that it's taking full advantage
of the hardware to get you those performance report. Well, this reminds me a
lot of just patterns I've seen over the decades of
being in computing as a hobbyist and then a profession
is you see a lot of things come into the
fore as being very monolithic, and then people
realize, wait, that's really a team effort.
And I think about it as a baseball team, right? You don't want to put
the pitcher, the person who's skilled at pitching in center field, can they
perform there? Well, gosh, yeah, but you're wasting them,
right? They are tuned their whole body, their
desires, their motivations. They love being pitchers.
So put that person on the pitchers mound and you see this
happen. And it's in all sorts of places. We saw it, frank and I have
seen it over the years when the unicorns were the big
deal, the data science unicorns who could do data engineering and everything
that we've kind of broken out now into other fields.
And we're seeing it now in the hardware
and in the distribution of the separation of
concerns and the distribution of concerns, getting every component to do
what it's best at. And along with that, and I'll shut up after
this, is this whole idea that it's moving so
fast that the hardware that's going to perform
the task first sometimes isn't even identified
yet because some new approach popped into the equation. If
somebody tested something and went, this is great. Now whether I run it
and you just see that and it's on a scale now where it
used to be measured in years and moved to months, it's now weeks
and sometimes days. It's just amazing how fast this
is going. And not that long ago, people were predicting
an AI intel. Right.
I think Dolly kind of and the whole generative artwork
stuff, I think kind of like, wait a minute, there's something here. Then Dolly came
out and then OpenAI did the one two punch of here's
Dolly a couple of months later, here's Chachi BT. Now you're just seeing like
it's on fire. Like it's not just AI summer, it's an AI heat
wave. Yeah, exactly. It is. It's a full El
Nino. I like that. That's the
quotable, for sure.
I think one of the things I think people realized is,
and a lot of the thinking was that AI
winter was coming because we're hitting processor or
hardware kind of upper barriers. And I think we're
finding out, I think much to what you said is that it's not just about
throw this many GPUs at it. It's right. The entire story, the entire
bus matters. Right. So the shortstop matters using the
baseball analogy. Right. The outfielders. Right. You can't really win
a lot of baseball games if not everybody on the team is
playing at their best. Absolutely. And just to take that metaphor
all the way, the turf matters, too. The infrastructure that you're running
those specialists on, you're going to play better in different
fields. That's true. That's a good point.
I love that you took the metaphor to the next level. That's awesome.
I think you mentioned whether it was in the virtual green room or here something
called habanero. And I know you're not talking about just
cooking. Right. Spicy habana. Yes, habana. I'm
sorry. I had food on my mind, as is
often. What is habana? Because I've
heard whispers of it. I know we're recording this middle of
May. There's going to be some announcements at the Red Hat Summit. Well, they'll
probably already happen by the time this goes live, but what is
it? So Havana is an architecture, an AI
accelerator, and it's a specialty chips specifically
designed for accelerating AI. And it's actually two
chips. And the reason it's two chips is that you want, again, going back
to what we were talking about, you want the right hardware for the AI workload.
So you want to be able to have the right hardware to opt optimized for
training flows and a separate set of hardware
for cloud scale and hyperscale inferencing
workloads. And so that's actually what Habana is. It's a two
chip strategy. So habana gowdy which is out available.
V two is available. V One has been out for some time. If
you go to the Amazon cloud, you can get it today. It's also available in
data centers, and a lot of universities have them in their high performance computing
environments. And it's geared to doing that sort of scale,
large data set training that you would find
whether it be in a cloud kind of environment, a chat GPT level
of analytic, or in the case of high performance computing.
Whether you're doing climate modeling or flow dynamics, those kind of big
training model sets that you want to be able to do at scale. And
what's nice about it is that like your cloud scale, it scales with your architecture.
So it allows you to be able to scale up your training based on
the compute needs with an AI accelerator specifically tuned to
that. The other chip, the Goya chip, is an inferencing
chip. So it's again tuned for that inference. But the reason,
again, this is for high end cloud scale hyperscale or things like high
speed training, where you want to be able to do large amount of inference in
as near or close to real time as possible against really
complex kind of data flows that you're trying to do
the analysis of. And again, looking at the right
hardware, we wanted to make sure to not just meet what we call the sort
of the normal scale. So the kind of things you would interact with when
you're going to do fraud detection, but you also want to be able to handle
really large scale inferencing because you're dealing with ingestion of multi data
sets across multiple different domains and having to be able to do that
inferencing in a streaming kind of mode. And that's really where the Goya chip
shines, is an inferencing platform that can scale
with the cloud. And that's really the Habana strategy is about giving you the
hyperscalers and high performance computing, the equivalent of
an AI custom chips. And that's really where Habana
sits. And then when you look at sort of the majority of what most
people will leverage in a cloud or on prem, what we've been
doing there is adding new instructions to the CPU. So
VNNI was the first really big one in AVX 512,
which really accelerates the math that you're doing behind
inferencing and training and give you those
instructions. That software, whether it be Intel's OpenVINO software
or TensorFlow or other frameworks can take advantage of
that math to use hardware offload to accelerate the math that you're
doing in your training and your inferencing workloads for most of your normal
kind of AI. A lot of the AI we deal with, not the high performance
computing style. And so you get the balance. And again, it goes back to what
we talked about in the beginning, the right compute for the right AI. We've also
introduced data center graphics because again, there are workloads that absolutely
make sense for a GPU besides fun gaming. And
that's really where you'll see GPU shine on, those kind of specialty
workloads that take full advantage. And a lot of the deep learning object
recognition ones work well on GPUs. They actually work well on other
kind of platforms as well. And one of the things we're seeing in the Edge
is a shift towards more customized approaches, whether that be using
an FPGA as sort of a hardware platform that you can code
in your algorithms to do inline inferencing, do feedback loop
training. And you see this a lot of times in the image processing, video
processing side, also in the signals processing. So whether it's five
G and being able to do signal quality testing or signal acquisition
and being able to do RF signal analysis, FPGAs
actually really shine for that kind of workload. Where you want to put in your
custom algorithm that you're going to actually test against or
use as part of your conditioning. And then we get to the idea
of what we call an ASIC. And that's where you know your workload, you
know you're going to be doing this kind of inference. You can actually code that
into a custom chip that will do just
audio AI inferencing or
do certain aspects of video coded. And this way you get the most
performance in a low swap. And that's the idea here
is you want to be able to handle everything from the pointy end of the
spear, the Edge sensor and give it the ability to do AI as
opposed to waiting for it to send the data to the cloud and get a
decision. You want to be able to give it something, but it also has to
operate at the size, weight and power that
you'd expect from an Edge sensor. You obviously don't have a data center power
system for your car, for your drone, or for
your camera on the streetlight. Right. That would be a very heavy to
fly that drone. That's okay.
I'm curious how you kind of manage what
I'm just going to make up words here, but like an innovation chain,
I'm thinking about like supply chain management. And I know
I've got experience in electronics engineering, and I
know some of how much it takes to go into mind you my
work was decades old, but this whole idea of getting
ahead of the curve or at least being able to predict where the
curve is going and how steep and when. That
sounds like a huge challenge for figuring out what
will be needed next. So what you're talking
about is how does a company that's building out both the hardware and the infrastructure,
stay ahead of, like you said, the week to week turnaround
in the AI world. Part of that is having a diverse team
of specialists. So the Intel Labs,
which is our team that looks five to ten years out, is over 1000
people who full time looking at process node technology,
security, AI data science. They're across multiple domains
and within each domain we have specialists in different areas.
One of the really I'll give you a great example. Before Chat GPT blew up,
I had two different of my AI specialists, one on the
government side and one on the performance side. Start talking to me about this thing
called Transformer. Like, oh, there's this really cool thing that we're seeing here, it's called
a Transformer. And I'm like, okay, that's interesting, and tell me more. And they explain
sort of how it worked. And then fast forward, six months later,
Chat chips BT shows up and I'm like, I know what that is because that
has the word Transformer. I've seen this. And again, it's about giving
your people the ability to go out and look. I think one of the
advantages of being at intel, and it's really why I've been here so long,
is everyone knows intel inside.
But there's something to that. Our chips are inside the
edge. Clients are inside the financial services, healthcare,
manufacturing, oil and gas. They're in the government system, they're in the cloud,
we're in the network. Which means we see workloads both current
and coming from all those different domains. So in some
respects we're on the cutting edge because we're seeing what people do because they come
to us, say, hey, I've got this software, I want to optimize on your hardware.
What does it do? Well, it does blah, blah blah blah. I'm like, okay, let's
help you. And then eventually that becomes open AI.
That's the kind of thing because ultimately every startup, every big company
wants to get the most out of their software and our teams. And one of
the things people don't realize is intel has over 19,000 software engineers
and a large majority of those do you know, they really divide up into three
areas sort of research and pathfinding, ecosystem
enabling, and then software development for
compilers, software services, software tools. That ecosystem enabling team
is a very robust team, it's been around for a very long time. Whose job
is to make Microsoft Windows rock on intel, make Oracle
rock on intel, make red hat rock on intel, make open source. We have
over 1000 open source software developers whose full time job is committing
to open source. We're actually one of the largest committers to open source
community and a lot of what they do is build the optimized
version of those Linux kernel libraries or to
that AI model running on intel and give it away and open source
it. We've created whole model zoos optimized for the variety of intel
architecture because we know if you can run it best on intel, you will run
it, and that consumes resources. We like that. But ultimately
it gives us they call them bell cows, if you will.
We're seeing those bell cows of what's coming next because they come to us and
they say, hey, help us. And very few see us as competition because
we're not going to go build the Chat GPT. We're not going to build a
new operating system or a new sort of predictive maintenance
solution. We're going to give you the architecture for you to run it
best. And even our OEM, whether you buy from Dell or
HP or from Lenovo, we don't care. You're buying intel hardware
inside. And so let's help you take the best advantage of those platforms. And that's
really been the approach from intel, is we want everyone's software
to work. And even with the GPU vendors, they still run on a CPU
platform. And so we want to make sure that that code runs best. So that,
again, you're driving the overall consumption. We raise the bar for everybody. We
raise the bar for everybody. Nice. Yeah. I
think there's a lot to unpack there. Right. And I think one of the things
you brought out, which is something that people don't, I don't think people have
widely realized yet that Edge is probably going to be the next
frontier in just
computing. Right. Obviously the last ten years have all been about cloud. Right.
But I think we're swifting as companies kind of take a look at the bills
and realize that lift and shift was not a
financially great decision. Right. Whether or not cloud is a good
thing or not, I think it always goes back to those two
words that every consultant and every It person always says it depends.
Whereas previously it was last ten years was
oh, definitely was the two words. But I think now we're realizing it depends.
And I think one of the drivers for this are things like autonomous systems
or drones or self driving cars, right. No matter how good
5G is, and I can tell you I know all the dead spots
in the DC area, but
if you're driving along at 60 miles an hour, 100
miles, 100 km/hour for our friends overseas,
and like you said, is that a tree? Is that a shadow? Is that
a person? Is that a grandma? Right. You don't want to wait on
the latency to come back. You want the inference or the decision to
be made on device. So you're really bumping up against the
speed of light and you're talking nanoseconds, not
milliseconds. Right.
What do you see? Because you mentioned you want there to be
sensors, but obviously these things have to be relatively low power. I guess in
a car it doesn't matter as much, but certainly on a drone that
matters.
What sorts of challenges does intel see in that regard in terms
of you want the most performance, but you want the most
energy efficiency. That seems like two
opposing forces. You would think that way, but if you
look at Moore's Law and you look at what's really behind that, it's about
reducing the size. And really that means the
power and increasing the performance, increasing the amount of
transistors. And that's really been what's driving compute all along, is how do we get
to lower power per density. Now, where it
becomes interesting is in the cloud. It's a cost measure. It's about getting
more for your dollar in a car or in a
drone or even in a factory floor. It's about being able to
operate closer to where the decision needs to be made
without having to, again, to have to power it and have that immense
cost. Or in the case of a drone, the weight of the battery pack and
so forth. So lower swap actually enables those edge use
cases. And again, one of the things that people realize is that Edge can mean
different things to different people. You talk to the cloud providers and Edge is just
a couple of racks closer out of the cloud. On
Prem, you look at Azure Stack or Snowball or these kind of
approaches. It's really about pushing pieces of the cloud closer to the edge through like
the core or they called it the
fog back in the day. You look at the edge and
you take a look at a Tesla, it's like a driving data center.
There's compute capabilities in there. A plane is a flying data
center. Your drones are getting to be more
computing. And when you move from a
discrete mode to a logical mode, and I've seen these already, where you have a
drone who actually has one processor but multiple containers, so actually running
multiple functions that could be thought of as different
applications on different nodes, but now they've all been collapsed with either virtualization
or container. So you can have navigation being one, you can be
doing object detection and mapping with another, and then be able to do sort
of other kinds of sensing like temperature
or barometer and things like that and doing analysis in
real time. One of the best examples that we demonstrated
at our last year's Fed summit was a set of drones out
mapping a region. They were going about their business, but they had a policy that
if somebody walked into a specific area of interest, let's say in front of an
embassy or in front of Lloyd or too long, that one of the drones would
be retasked and go over and investigate and do facial
recognition. All the things you want to do to make sure, hey, is this person
up to no good? And it didn't require a reprogramming
of a drone. It didn't require a special drone that was just the investigator. It
would basically retask itself with a new. Mission in real time
and go investigate. And when the person left that zone, it go back to its
day job of mapping the environment. That's just sort of the tip of
that simple prototype to show that even a very
small autonomous system and these were like sort of my mini drones
here, is capable of the compute necessary to
do multimission kind of use cases. So the edge absolutely is
that new frontier. And it's again similar to the cloud. When you say cloud,
everyone thinks, oh, public cloud, really? Cloud is all those architectures
all the way down to the edge. It's the way we develop those cloud native
apps that can flow back and forth. So from a cloud provider, it's moving
more of their cloud infrastructure closer to the edge. And what the
edge, folks, whether it be the actual device or sensor manufacturers
are looking at, is bringing some of those cloud
capabilities to their device to operate
independently. And there's a reason for that is that, number one, latency, like you
mentioned, Frank, but also the cost of shipping all that
data. No one wants to ship Raw 4K video feeds to the
cloud just to be able to tell me, is that a tree?
You want to be able to send the results that I saw a tree
here with the longitudinal latitude, which is a small data
packet, and let the sensor do the AI, do the inference
at the edge. Right. And then you have the case
where you're talking about planes or vehicles, right?
Like the whole time it's tracking, did the wheel fall off? Did the wheel fall
off? Did the wheel fall off? Right, but at one point when you get to
your destination, the wheel either fell off or it didn't. Right.
So you collapse that entire thing
to one integer level or really not even an
integer. Like a bit. Right, a bit. And then if the wheel does
fall off, I'm sure there's plenty of other stuff you can pick up too,
but hopefully nobody gets hurt. But I mean,
ultimately you're right. The problem with data is so much
that there's value, but there's a certain
amount of we've gotten to the point where
just because we can, we've done it. Right. Yeah, sure. Bring up that
4K. If I'm a salesperson for one of those cloud
providers. Yeah, man, bring in all that 4K data you want,
we'll take it all. We'll be happy to charge you for it too. Right,
but I think as we get to the point where
there might just be too much data, I think people organizations are going to start
thinking like, where can we scale back on the storage? Because
we don't really need it unless there's some kind of regulatory reason for
it. Now, one thing I want to double click on,
because this is a fascinating conversation, we'd love to have you back
on the show at some point. What's the
deal with FPGA because you mentioned
that and this was a huge deal. So a couple of things that are
interesting is that I first heard about Transformers at
the Microsoft has this internal data science conference
MLADS, and they first talked about Transformers. I went into
the talk and ten minutes, my head went
boom, right? I didn't quite follow it. Somebody later on in the
day in the reception area was kind enough to explain it, how it
works. And one of the other things that came out of that conference was talking
about the importance of FPGAs and what they're going to be like in the future.
Now, again, I'm a data scientist. I really don't focus on
hardware so much until when I need to buy new
hardware, like a new desktop or laptop.
What are FPGAs? And I remember hearing a lot about them and then
they kind of went dark for a while and then now they're kind of coming
back into vogue. Can you talk to us about, one, what they are and then
two where you see they're going? Sure. So Ed and FPGA are a field
programmable gate array. They've been around for forever. I mean, computer
science engineers going back, electrical engineers going back to the
80s played with FPGA. They were very early FPGA, but
basically they're programmable hardware. That's really the way to think about it.
You think about a CPU or an Ace or any chip it's
laid down with its transistors, and the flow of those transit is
fixed. CPU can do multiple software
flows, but the instruction flow is the instruction
flow. What makes FPGAs interesting is that you
can create new RTL, new layouts of flows, what
they call netlist of those instructions going across those transistors
each time. You can go in and customize it after. So the
manufacturing builds you a clean slate of a bunch of think about a bunch of
rows, and then you program them to your specific need
at a hardware style abstraction layer. So it gives you a much
faster capability because you're now really writing in hardware. It's a lot more
complex of a coding. It's not like doing Python,
but what you get is a very optimized piece of
hardware for your specific use case. And what's nice about that
is one of the great examples is in signals conditioning. When
you're doing like 5G research or testing signal amplitudes and
things like that, as you put in your algorithm actually into hardware, you go out
and test it. It works sort of here. I need to tweak it well, instead
of going and spinning a new piece of hardware, you just upload new code and
you go right in. So it's a much faster time of development for doing
those custom things. What people have found when we start looking at sort of
AI use cases and machine learning and pattern matching
is that FPGA really lend themselves well
to be able to create different kinds of architectural approaches to how
you process that data flow. If you think about a GPU
or CPU or even an ASIC, it's a fixed data flow. It's good for the
things it was designed for. What FPGA allows you to do is to customize
your flows based on what the data is or based on what your algorithm are.
And so a lot of the FPGA work they were seeing in AI is people
coding their AI algorithms or the machine learning algorithms right into
hardware and then deploying it. And so it allows you to be able to deploy
your thing quicker and you get pretty good performance. It's not as
good as say, as a custom ASIC for your algorithm. And it's not as
scalable really as like a software abstraction on running on a
cloud set of CPUs. But for a lot of these training and
inferencing use cases, one of the areas where it shines is in the whole
area of neuromorphic processing. So a whole part of the AI machine learning
space is modeling after brain activity or how our
brains process. It's a whole field. FPGAs are actually
well designed for those kind of algorithms that X 86 and
other CPU style Arctic just aren't yet.
And that's why FPGAs really shine in those environments, because you can create
these linear sort of permutation flows that you find in neuromorphic
algorithms. You just code those into the path for the
FPGA. They're really good. You'll see, FPGAs are very often used
in cellular and RF communications that are really good at those sort of
channelizer and signal optimization and
be able to do those kind of algorithms that you do on RF and
Comps, again, really good for those kind of workflows. And so why we
see the resurgence of FPGAs, although they've never gone away, you find them
everywhere. Open up your big screen flat screen TV, you'll find a couple of
FPGA in there. Where they're shining is because it
allows you to do some rapid prototyping on AI. And because we're seeing
now FPGAs come to the cloud. So you go to Azure has an FPGA
cloud. You can now deploy those algorithms at cloud scale,
or you can deploy an FPGA into your edge sensor and be able
to do that real time, sort of. Let's go try this inferencing model. Oh, we're
going to change the inferencing model. Let's go do that one. And where this becomes
really interesting in those low slop environments is a modern FPGA is
reprogrammable in milliseconds, which means you can go from one
program to another by just pushing a firmware, if you will,
update. And now you go from a 5G communications
system to LTE or to a six G
without actually going and swapping out the hardware. That's wild.
That's wild. Yeah, it's exciting times. So
with that, the updatable part of it,
how do you secure that? Because I can easily see that being like particularly
you work in the in the federal space, right? Like security
is top of mind in that work. It should be top of mind everywhere,
but in the near term it's top of mind, at
least in the federal spaces. FPGA
sounds like awesome, but it also sounds like that just seems
dangerous in a lot of ways. You can reprogram it in milliseconds.
There's got to be some kind of security story there. Oh absolutely. And
Fpjs have actually in many cases led as far as the kind of security
mechanisms built into the hardware for that very reason.
At its core, at the core level, it's the same kind of approach you do
for verifying your firmware on your system. It's signed
by hardware so that basically you're verifying
your load and if you're going to do an update, you're going to verify a
signature against a hardware rooted key so that you make sure that only
legitimate folks can do the update and that it's only be able to be done
by someone who's got the permission. From a cryptographic
perspective, what we find in the current FPGA that are out in the market
is that they've built in a whole suite of security
capabilities. Things like Puff Provably, unclonable
functions, which is basically a hardware root key that is
really secure as that hardware route of trust, signing in
cryptography functions, anti tamper functions to make sure someone can't go
pop open the lid or put in a jumper and try to try to change
the code. So those kind of mechanisms have been in place for a long time
because FPGAs have been used in such critical places. We find them in
radar stations, we find them in systems and so they've been building security
in for a very long time. And it's part of the workflow that when you
build your code you're going to take advantage of these implicit, let's call them IP
blocks that do security for your RTL, for your code that you're putting
in place. The other important thing is that the way that the code works
is once you lay it out, once you translate your software into that
layout, the layout is you can't just sort of go and reverse engineer
back. And so it's really a very powerful
mechanism as opposed to say firmware. When you're it's software.
If you think about the BIOS update, it's software that you're loading just deeper in
your platform and if anyone wants to go inspect, you'll find
there's a lot of software in the hardware that you don't realize is actually
software. The same kind of security mechanism we did there. You verify it against a
hardware of trust, you make sure it's signed before you run it
and then you apply cryptography to make sure that it can't be changed or it's
integrity protected. You find those same capabilities built into the
hardware of an FPGA and the software development tools, the
dialogue, the cordis and so forth have the mechanisms to take advantage.
So again, programmers don't have to be security gurus. They basically say,
I'm going to push this, and it's auto going to take advantage of those features.
It's good because programmers historically are very bad security
people. I say that. It says, yeah,
it's its own specialty. And yeah, you can't be
good at everything these days. There's too much. So I'm going
to echo what Frank said earlier. Steve, we got to have you back.
I really appreciate you being here. We could talk and geek out on
hardware stuff forever, but we want to
pivot and go to our questions and if that's
okay, we want to start with unless Frank, unless you had anything else you wanted
to do before. Let me
rephrase. No.
In the virtual green room, you talked about some things that are going on and
kind of operationally and
wow, we didn't even get there. I mean, I
think the important thing I took from this conversation is
that one, GPUs, they are important, but
they're not the whole story. And two,
at the end of the day, chat
GPT, any of these magical looking AI
models, magical seeming, right. They're all mass,
right? Yeah. And being beneath the math are electrons
bouncing around inside these microscopic chips. And
there's all sorts of things you could do to tweak and improve that, even if
it's like a billionth of a second, right? A billionth of a second times
a billion adds up.
And that adds up in terms of whether you're driving a car
or you're flying a plane or
you're a company like AWS or Microsoft,
where, hey, if I save one compute second per
transaction, I do trillions of those a day. And that's real
money. Exactly. And that's the thing that blew my mind. But yeah,
let's switch because we could geek out for hours. Because this is very
true. Yeah. Amazing.
It really is. So how did you find your
way into not so much data, but it how did you find your way into
data? Did you find it or did
it find you or hardware specifically? So ring. It's a really good
question and going back to the very beginning, actually, I started
out in the molecular biology
bioresearch side of the camp, going all the way back. I was going to be
a research biologist and probably still be there today,
except for a couple of key life events early in
the early ninety s, I was a hacker as a kid.
I loved seeing how things fell apart and how to code and break code
and things like that. But in the late 80s, there really wasn't a
career other than a COBOL programmer, which
wasn't an exciting career at the time. So I went the bio route,
which was my, the love. And right after I graduated and was going to start
med school, I had a year off and
someone had some money, wanted to do a startupy thing and they knew I was
a hacker and say, hey, why don't you help me get this thing running? And
I'm thinking, well, med school is expensive. This would be a good way to help
pay for it. And so I started my first company in
95 and after three months just fell in love with everything that was
going on. It was the exciting time to be in the internet. Got to apply
some of my security hacker background in an interesting way
and had some really good mentors. People like Bruce Schneier,
the writer of Applied Cryptography sort of took Zebru Schneider.
Zebrus Schneider was one of my mentors and took me under his wing.
And like I say, I sucked his brain dry as best as I could. But
really it just sort of got the opportunity to get on the ground floor
right before Netscape went public. So really early days on
a startup in the email encryption space and then one thing led to another and
I just felt this was what I was going to do. And for the next
sort of several years, I did multiple security startups throughout
the then in 2005 got acquired
by intel. I like to joke, I'm still trying to figure out
how I ended up here for 18 years. But I think what intel
has provided me and provides a lot of our folks is the ability to sort
of innovate in an environment where a, you've got a big company
behind you helping you do that. But one of the best
reasons why I think intel has been fun for me, my most
successful startup, we had 500 of Fortune Thousand companies using
our product. The first project I worked on in intel went to 40 million
PCs. So the impact is just
unbelievable. Now from the data
side again, at the end of the day, like you mentioned earlier, underneath the data,
underneath the machine learning, underneath the AI, and even before we were talking about AI
was machine learning and advanced pattern matching. There's electrons
moving around it's running on hardware. And so a lot of what my
job has been before I came to the federal team was looking for ways to
innovate or take advantage of new use cases in software, to
take advantage of hardware in interesting ways. And so we call that
pathfinding. So you think about our labs or thinking about the next generation
hardware five to ten years out, I ran the team, the security
pathfinding team that was looking at the two to five year horizon. I
knew this was the hardware platform that was going to be there next year. What
would be some interesting things I could do with it to either advance security or
increase security, that was my area domain. And so things like
antimalware technologies, cloud security, before they knew how to spell
cloud. We called it virtualization security first and things like that.
Web security, that was the fluffy stuff. That was Steve's world while
the hardware engineers are figuring out low level cryptography and hardware
roots of trust. And we sort of worked in tandem to innovate.
And so as things like data science started to take off, it was like,
this is a key area both from a security and perspective. How do I secure
that data? How do I secure the algorithms? How do I use that? I mean,
one of the really cool things is being able to use machine learning and AI
and apply it to the cyber problem.
And when you start doing things like that, you immediately run to, well, we've
got too much data flowing in. I mean, the classic example is streaming
analytics on network at network speed. Well, how do you do
deep packet inspection at gigabit or higher
speeds without losing data? That's a big problem. That's where hardware can
help save you, that you just can't do in software.
And then when I transitioned to the federal team and took over and
drove our federal technology practice, you really opened the door to
all the different use cases. And one of the things I like about the federal
government is that it's a macrocosm of all verticals. You want to
talk finance, you've got IRS and CMS, some of the largest
processing of financial data. You want to talk healthcare, the VA is the
largest provider of healthcare, the largest insurer in the world. You want to talk
logistics, DoD logistics is huge. So
you sort of look at it, every kind of use case you'll find in government.
So it's really a good way of looking at all the different verticals. And they
all have unique or interesting data problems. There's some
commonality. And one of the things I really like about the federal government is that
you get that commonality across the divisions. They all are having trouble doing data
ingestion. That is just fundamental. It doesn't matter if you're the federal government or
Citibank or startup in Silicon Valley. Data ingestion is hard
and doing it at scale and being able to then do something
once you've got the data. And I like to use the analogy
of an iceberg. So AI, Chat, GPU, all these are the tip of the
iceberg. That's the cool, sexy stuff you can do, the hard work,
the data curation, data wrangling is all the work that has to be done before
you ever get there. And that's data ingestion, it's labeling, it's curation,
it's data set management, it's all that stuff. And then layer in things like
removing bias or dealing with bias and securing and integrity, protecting your
data. Like all those things have to happen before you ever start having
the fun math that happens towards the end of that curve.
That's where you find that coming out. Everyone is challenged with those things, and I
think that's where the excitement is today. No, you definitely hear in your
voice, sorry, Andy. Yeah, definitely. No, it's okay. We refer to
that as kind of a joke that's been going on
for seven years now. We say, first you get the data,
and that's 90% of the work. We know
that and your iceberg analogy fits that, Frank.
We need a shirt that has a picture of an iceberg against us. First you
get the data under the I like that. I'm definitely going to do that.
We launched a magazine, actually, yesterday as we record this, and
the cartoon segment is called First You Get the Data. And it
kind of like cringy things that you'll hear about data, and one
of them was like, yeah, first we get the data. My
favorite was how
to prep and clean the data. And they were like, oh, no, our data is
already in the normalized database. We don't need to clean it or prep it. It's
already ready. Like, oh, boy.
You need you need a picture of someone throwing data into a washing machine.
That's a good shirt. We could do that. Yeah,
no, that's cool. And I think you bring up something that I think,
folks, we don't know our exact age demographic. We have a rough
idea, but if there's anyone, let's say, under the age of 30,
right in the car with the parents
or they're listening, it's hard to imagine the time because we're about the same age.
I think you're a little older.
If this was not seen as a good career path, like, coding was not the
whole learn to code movement is a modern
phenomenon. I started my college career to be a
chemical engineer because
I had to convince my parents that software engineering was a
viable career path. And my mom, God rest her
souls, was like, I don't want my baby to be one of those weird
people in the basement. Right?
And then my dad, God rest his soul, was like because when
they came to visit me, I had a Sunday print out of the New York
Times, which of course had the job section, which was
at one point like a book. Right. And look at all these
jobs for computer programming. This is a thing. And my
dad looked through it, and he saw all the starting salaries, and it was like
seven or eight pages of near six figure
salaries in the early 90s, which was a lot of money back then, right?
Yeah. Like, looking through, like, on Wall Street stuff. And
he's like, I'm sold. And it's like
and my mom was like, no.
That is literally, like, my experience as well. When I told my parents that I
was going to not go to the research biology route and do the MD
PhD, I was going to go into the security thing. They wanted to do an
intervention. They thought something was wrong.
About two years. In 96, after I'd done the start, for about
a year and a half, there was an article in the New York Times, Paul
Cotcher, had done the timing attacks against RSA, and it
was front page news. And when you read down the first blurb, it says, 22
year old bio student from Stanford cracks RSA encryption. So
I cut that out and faxed it to my parents because they have an email
yet and said, look, another bio student doing security. It can
happen. Right? That's funny. One of
the best web developers I ever worked with, his degree was in biology
as well. And I think there's something to be said about understanding natural
systems, and I think there's some pattern matching gifts
that go along with that. I know my friend was that way as well. And
Frank, when your mom said she didn't want you to be one of those
weirdos in the basement that flew through my head, but I
maintained discipline was too late.
And I could say the same for me as well. Too late.
In her defense, my mom stayed with us in a house that my
wife also works in technology too.
She had an entire suite in our basement of our
house, which was not
windows, walk out yard, everything.
It worked out well. Sometimes
your parents my mother encouraged it without realizing. She allowed me to buy
the haze modem and connect it to our phone. And I did get
disciplined when I had that $1,000 phone bill from dialing into BBS's overnight.
But they should have seen it coming. Yeah,
my mom freaked out when I wanted a modem. She's like, no, absolutely
not. And my dad was like, yeah, you probably should stay out of trouble.
It's easy to stay out of trouble. Then. I think I was lucky
that my parents didn't know what a modem was, so I didn't know what
they were getting me. Right. This
is awesome. But I want to jump to question too sure. And ask, what's your
favorite part of your current gig? Favorite part of my good
gig? I think honestly, I thrive on being challenged,
on trying to solve big hairy problems. I think that's what has always
excited me is present to me with something that isn't being done well today and
trying to figure out how to do it. And I think one of the things
that I love about my job is meeting with government customers who
have big hairy problems and looking at a variety
of technologies. And I think what makes my role somewhat unique at intel, so we
have like a CTO for memory and a CTO for various
architectures is my role is pan intel so I can look
across FPGAs server parts,
networking, and sort of see that collective of where do the bits can
come together to solve big hairy problems. And that's really, I find
keeps me very excited is that every day I could be talking about an
IoT problem today with an edge sensor, and they're
talking about petabytes of data being processed in the cloud tomorrow.
It's looking across the technology domains and again, coming
from a background of cybersecurity, which again looking at various different domains from a security
perspective, but then adding to that AI, high performance computing,
it's a technology playground, right? And the federal
government, when I first joined Microsoft,
I was in the public sector, part of doing basically
technology developer evangelism for the federal government. And a lot
of my commercial sector colleagues were like, wow, it must be really boring
there. I might be like, you know,
we see things that you don't see
and what it is, is like there's interesting work going on, but the folks doing
interesting work for many reasons do not want
a lot of attention. Indeed. So you see
some things that like, wow, see, I hadn't really
thought of that type moments. Well, decades
ago I spent just a little bit of time in a really odd shaped
building up that way. Just a touch of
time. So I can have five it did. So
I can go yes and amen everything
you both have shared about. So now we have three. Complete
the sentences. When I'm not working, I enjoy blank.
Spending time with my kids. I have two small children and they keep me young
and full of fun and keep
me trying to stay in shape to keep up with them.
Very cool. Both Frank and I have
children as well. Frank has the younger kids. I'm
probably the old guy in this conversation now that I think about it.
But number two, complete this sentences. I think the
coolest thing in technology today is blank.
One thing that is a tough question,
I would have to say. So the two things that I think are really cool.
Number one, again, not the chat GPT, but
what the future will do with that capability is one
area. And then again, because I'm a security geek at heart, post quantum
crypto is going to be fun. Figuring out the next generation of algorithms
and how robust they'll be once quantum computing comes online.
I think that's an exciting area of math that is going to
spurn a lot of mathematic. Academia is
excited because it's a renewed interest in that space
and the algorithms are really interesting. The lattice
space structures are fun area of math to look at. Nice.
Interesting. The third and
final, complete the sentence. I look forward
to the day when I can use technology to
blank. So I'm going to give you two answers. I look
forward to the day when I can draw something on a
whiteboard and it turns into code. That's one thing I'm looking forward
to. Oh, nice. I can totally and that's not that
far off. It's not, I think a little bit of sort of the
image to text, image to code. I think
building box, you have to be able to read my horrible handwriting. That's going to
take an AI in its own right. But I would love a day. When I
can start draw my design like I like to do I'm a whiteboard kind of
guy, and then have it create a prototype. I think that's one thing
I'm looking forward to. And then I think
the other thing is I'm looking forward to the day when
augmented reality becomes reality, where it's not just
a cool toy, but where we actually see it integrated
into our daily lives. And I'm not talking to glasses and all that. I'm talking
about having the digital world and our physical world actually start to make
sense instead of it being a throwaway toy and I think we're seeing
pockets of it, but I think that the future is going to hold a lot
more of that immersive experience that we only see in movies today. I think
those are the two things from a technology perspective, I'm looking forward to.
Although I have to say, if I can get that, the code from the whiteboard
is going to make me a lot more efficient. No, that's true. And
it's funny because things that once seemed impossible
are now possible and even mundane. So I remember
when I was a kid, there was a story, there was like a story we
read about a kid who wrote a built a homework machine, right? And this was
like first or second grade and a bunch of us kids were like, yeah, how
do we do this? We got to make one of those. Now you look at
Chat GPT, obviously we abandoned the effort
because it just wasn't possible at the time. But you look at how kids
are using Chat GPU today, that machine exists
not in the way or the shape or form we could have imagined, but
it's definitely here. So to have that whiteboard to code
thing, it's totally
within sight. Whether it'll be within reach, only time will
tell. Probably a few weeks. If there are VCs out there listening, this is an
idea to invest in, for sure. I would love to see
especially for you, Steve. I'd love to see whiteboard
two FPGA code. That'd be even
better. We're just combining ideas. There you go.
I know that would make some of my engineers happy. There you go. Really
cool stuff. So we ask all of our guests to
share something different about yourself. But we caution
everyone to be fair that remember, we're trying to keep
our clean rating at itunes, so please keep that in
mind. So something different about me.
Well, I guess one thing we've already talked about that I have a bio
background, but the other thing I like to do is I play
tournament poker. I am an avid
poker player when not in COVID Lockdowns and things like
that. I played in the World Series back in 2013.
Really? That's something I like to do as a
past. It's a different use of my skills, of sort of social
engineering, if you will. And I like the tournament play
because it's sort of a long game. Right? Well, I have a
stack of money and I'd love to learn more about
is that the joke? All you need is you're always
welcome to my table. I'm lying about the
money. My wife is
actually a pretty good poker player, and when she was pregnant with our second,
she's short and she would carry a stool with her because she would have
to set up and her feet didn't reach the floor. And I think I gave
her like $100 in seed money and said, go knock yourself out.
And she came back like she was spending money. I think she turned that into
something like two grand before she had to quit and go have
Emma. I
would love to see you, because I don't think she's
your level by any stretch, but she did okay. We should have
a data driven poker tournament. We should. There we
go. That's an idea, Frank. The other time we had an
idea of somebody on the live stream said we should do like an ATV
race or something because we always go off track. That's kind of the joke.
Very true. But no, that's cool. Audible is a sponsor
of data driven can you recommend a good book? Ideally
audiobook if you do, audiobooks if not. Sure. Absolutely. Actually, I just
finished one that I think would be perfect sort of summation of this. So
Chips is an excellent book.
You think it's talking about today, but it gives you the history of how we
got here. And even one of the things I thought was really interesting is
some of the decisions that were made early on from the
policy, the government policies that we've seen and how it
affects where we are today. Fascinating reading. So, yes, absolutely.
Chips wars, it's available on Audible because I literally just finished reading
listening to it on Audible. So that would definitely be a book I would
recommend. Cool. I watched a show called Halt and Catch
Fire a few years ago when it was at, and it was similar. It was
in that vein of when things were developing and trying basically
the laptop development story. And of course it was
fiction, but I know enough about it to
know there were some true parallels in there. So this
would be very appealing to me. I'm going to get it. I hadn't heard of
it. Thank you for recommending and our listeners can go to
thedatadedrivenbook.com I didn't test it today, Frank.
Some days it's moody, but if you go there, it should
redirect you to Audible. And if you decide you get a free book on us.
And if you decide later to sign up, then it buys
Frank a cup of coffee. So when
you do that, we get a little bit out of it. It's a great way
to support the show and we really appreciate it.
Awesome. And where can people find out more about you and
what the federal team at intel is doing. So find out more about
me, go to my LinkedIn page. That's S-O-R-R-I-N on
LinkedIn. And then to find out more of what intel is doing in public sector,
just go to Intel.com public sector and it will redirect you to our
Government Solutions page. It covers everything from AI
data science to Cybersecurity to Edge, with lots of white
papers. Use cases podcasts with folks like myself and
others that are recording content on how intel is helping our
ecosystem. So definitely come check us out. Awesome.
And with that, I'll let Bailey finish the show. Now that was some
show. Is it me or are the shows getting better? It could be my
bias that leads me to say that, but I figured I would ask to get
more input. After all, what's an AI without good
input and a feedback loop? Speaking of feedback, have you
checked out Data Driven magazine yet? We are looking for writers
for the Autumn 2023 issue.