This transcription is provided by artificial intelligence. We believe in technology but understand that even the smartest robots can sometimes get speech recognition wrong.

Interview In Action: Surviving and Understanding the AI Feature Flood with Joseph Evans

Drex DeFord: [00:00:00] If you're still paying maintenance fees on legacy systems, you can't shut down, you're not alone. Health systems are drowning in cost and risk just to store old data. Harmony Health IT Migrates that clinical and financial data into a secure archives so you can finally decommission those systems and keep every record compliant and accessible.

Find out more@harmonyit.com.

Bill Russell: Today on the 2 29 podcast.

Joe Evans: i'm not sure that we see a lot of hallucinations, which is what I think originally there was a lot of concern about. But it's more the emissions that I worry about, you know, what is it missing? And as we think about summarization, what some of our incumbent models do, um, are we sure we're getting the whole picture?

Bill Russell: My name is Bill Russell. I'm a former health system, CIO, and creator of this Week Health, where our mission is to transform healthcare one connection at a time. Welcome to [00:01:00] the 2 29 Podcast where we continue the conversations happening at our events with the leaders who are shaping healthcare.

Let's jump into today's conversation.

Another episode of the 2 29 podcast. And today we're joined by Joe Evans, CMIO, for Sentara healthcare. Joe, welcome to the show.

Joe Evans: Phil, thanks for having me. A pleasure and a privilege to be here. Appreciate it.

Bill Russell: you know, Joe, the, the, so we ran into each other in LA and I think you saw me after about four hours of sleep and I was moderately coherent. We're recording this Thursday following vi and you've had about, a little bit more than that in sleep, but not much.

Joe Evans: That's correct. Yeah, I am about.

Bill Russell: Perfect.

Joe Evans: eight hours of sleep and two meetings in between and, uh, or not eight hours of sleep, eight hours home, uh, a little bit of sleep, couple meetings. But, uh, yeah, great to be here and, uh,

Bill Russell: Yeah. That,

Joe Evans: as always, good to [00:02:00] talk.

Bill Russell: having it in LA is so tough. I mean, that flight, that flight out is, isn't bad, but the flight back, you, you, you lose an awful lot of hours on that flight back.

Joe Evans: all the storms we had, the, the, you know, all the people catching up, uh, definitely impacted,

Bill Russell: all right. Hey, Joe, You're just back from the vi event, so I'm gonna take advantage of this. Uh, I mean, what was the, what were the key themes? What were you hearing at, at the event?

Joe Evans: The key themes were, of course, all around AI and you know, what agents are gonna do what, what our major platforms are gonna do around ai. but yeah, that was, I think one of my big takeaways. It was a great opportunity to meet with some of our partners, uh, talk to them about their roadmap, and then also explore, um.

Some other opportunities around how we think about how we can quantify the value of, um, new AI solutions versus how we can um, also look at some of our AI [00:03:00] solutions that are already in production, how we can monitor their use and even look at how, um, kind of what some refer to as shadow AI is being used, uh, in your system that might.

Include people dropping in P-H-I-P-I-I, um, ip, et cetera.

Bill Russell: Well, if, if, if that was the theme there, we're gonna continue that theme on this show. Uh, Uh, oddly, oddly enough. That's, uh, what seems to be top of mind for everybody these days is, uh, AI governance, shadow ai, um, how we're gonna bring AI into the organization. We have our existing vendors, which are. Um, how do I, say this kindly? They're throwing AI features at us. Like, there're, there's so many of them. It's like, alright, slow down, slow down. What, what does this one do? How does it process information? How, what are the guardrails? Um, so let's, let's start there. Let's start with the existing partners that we have in the industry.

I, you guys are an Epic shop, I assume.

Joe Evans: A hundred percent [00:04:00] we're an Epic shop. You know, as I think about, um, AI solutions, you know, we're, we're definitely a problem first shop, so we need to think about. What our business partners are bringing forward as what's a problem they're trying to solve. And then we look at our incumbent vendors, um, with Epic being one of them.

Uh, Microsoft, another. Um, and then we have a couple of other platforms to think about. Um. You know, whether they have a solution for it or where it is on the roadmap. Um, and, uh, I mean, I think that is our starting point, but as, as you referenced, um, you know, Epic has a lot of. Generative AI solutions that are, you know, out there, you know, we are still on the, um, a la carte.

We have, we've not jumped over to the All You Can Eat Buffet menu of pricing for, uh, Epics. Models. So we're still being pretty thoughtful about, um, what we take, what we don't take. You know, I think we'll get there pretty quickly, um, to that, so we might have to be a little bit [00:05:00] less discriminative, um, about, about that.

But, but yeah. Um, it is, it's a, a crowded space.

Bill Russell: I guess the better question for you is from a clinical standpoint, how, what problems are you potentially looking to AI to solve or to do a little bit better than the, the previous generation of tools that were out there.

Joe Evans: so from a strategy standpoint, the way we're thinking about this is we're looking at some big opportunities around rev cycle management, around what we can do better in care management, being A IDN, and also from a level of care length of stay standpoint. But from a purely clinical standpoint, looking at all the tools in the Epic Box, um.

As well as other solutions out there, you know, what we can give to our clinicians to make their day better, um, to make [00:06:00] it easier for them to have a whole picture of the patient, um, simply from a summarization standpoint, and then pull insights out of all that information to make it easy to deliver the best high value evidence-based care, um, really to improve that.

The health of the.

Bill Russell: The ambient solutions, those are well documented. We have, have some prior auth solutions that are starting to emerge, which are, are interesting as well. And clinical summarization seems to be. The, the, the next big, uh, hill to, to or hurdle. Uh, if you think of these complex patients, if we're able to really summarize that, that massive chart into the, uh, reliably and, uh, in, in a matter that the doctors trust, that's always the thing that always struck me was the number of conversations I had with physicians where they would say. I don't trust what's in the EHR and I'd be like, okay, we've, we've gotta get to the bottom of this. [00:07:00] You have to trust what's in the EHR. They're like, well, you're bringing information from these third parties and, and whatever. I'm like, so we started marking all that information. So they, they, they knew, hey, this was from, uh, someone outside of our network that was putting information in here and whatever.

'cause they wanted to, to know that in order to trust it. What's it gonna take for, for us to trust clinical summarization at scale within a health system?

Joe Evans: Yeah. I mean, Bill, I mean, I think for any of this, um, to work, trust is the foundation and I think to build that trust. There has to be a good governance process in place so that any solution we bring into our production environment, uh, is well vetted. Uh, that's. The very foundation of it. The next layer, uh, for our clinicians is to be able to, you know, create citations.

So if somebody looks at something in a summary, um, they're like, oh, wow, I would've expected that they could hover over it, click on it and see [00:08:00] exactly where it came from. And then they can, uh, you know, with the human in the loop standpoint. Be able to say, yeah, that makes sense, that doesn't make sense. Um, uh, either integrate that into their care plan or, um, discard it.

Bill Russell: It's, I are, are you concerned about the models themselves that are being used? I, I know we're, we're looking for as much visibility, um, dare I say, transparency into these models and, and how they're processing information or how they're. their work, but I, I know that some, some partners push back and they're like, look, that's proprietary.

And then some are using these foundation models and we're never gonna have clear visibility into these, um, non-deterministic models, how they make decisions. Just 'cause we know we can ask them the same question twice and get two different answers. I mean, anyone can. Try that out today with chat GPT ask it the same question, two different sessions, you'll get two different answers.

And uh, obviously there's, there's enough documented [00:09:00] proof of that. And that's, I think what sort of lends itself to the, to the, um, I don't know what's, what's the right con concern on, on the clinical side.

Joe Evans: as you said, the foundation or frontier models, um. From an AI oversight or governance process, it's not like a machine learning model where you can measure its outputs, see if it drifts, as you said, you might ask it the same question Exactly. Or with slightly different, um, input, um, but get a different output.

But, you know, the thing that I worry about in the summarization standpoint is, i'm not sure that we see a lot of hallucinations, which is what I think originally there was a lot of concern about. But it's more the emissions that I worry about, you know, what is it missing? And as we think about summarization, what some of our, at Sentara, incumbent models do, um, are we sure we're getting the whole picture? Um, you know, maybe what our EHR vendor [00:10:00] is bringing us might not take into account all of, all of the components versus what some of the other, uh, incumbents in the marketplace. Um, brain, uh, as far as, you know, unstructured data in the media tab.

You know, we all know that. In our EHR 80%, at least 80% if not more, of our, um, information is unstructured data, you know, scan, PDFs, et cetera. You know, is it pulling those pearls out of there that are gonna influence what I'm gonna do to in the clinic? Right?

Bill Russell: you know, the context windows are getting larger, but I don't even think they can account for. Or the size that some of these medical records get to, and, and you, I, I can understand how it's like, did it read? Every file in the context to do this summary? Or is it just saying, Hey, I I took the five best from 2007 to five, best from 2008 and you know, I mean we literally, I, we needed to go [00:11:00] through every single one of those files. 'cause there, there could be that, that one thing that needs to be documented and needs to be brought forward. and then there's probably some things that need to drop off as well. I mean. this is, I mean, this is the promise and the challenge, isn't it? So the promise is spend an awful lot of time, especially on these really complex care patients, we spend an awful lot of time summarizing those charts and, and teams do that, and nurses do that, and it, it just takes a lot of time.

That's the promise and the opportunity. but it has to be built on a foundation of trust in order for it to, uh. To, to really be effective. The, you know, I think the next thing I, I, I'd wanna talk about is, so you have your traditional vendors and you know, we'll look to Epic to use Cosmos and summarization and do the things that they do and do effectively and do well. Um, but there's a whole, I mean, you were just on the floor, right? So that floor is loaded with people that are like, Hey, we can solve this one-off problem that nobody else [00:12:00] can solve. How do you determine. Uh, I I How do you determine if you know that vendor's worth talking to? They're worth bringing in the, I mean, what, what criteria sort of did you use as you walked around that five floor to say, Hey, this is somebody we should probably take a look at?

Joe Evans: I think about our kind of incumbent platforms. Where they are in that space. And then, you know, I think one of the crucial things,

Bill Russell: So,

Joe Evans: from,

Bill Russell: your, if, if one of your existing vendors, your platform vendors does something, you just keep walking

Joe Evans: if they do something well. Um, you know, if, if they have maybe a 50 or 60% solution. Now, I think one of the keys is, um, you know, maybe we explore someone else to fill that gap and have a short contract with them. And then when one of the incumbents catch up, you know, then maybe we end that contract and keep open, uh, for exploration.

But I think, Especially important is [00:13:00] capturing what additional value they're gonna bring. You know, whether it is true hard ROI, which is, you know, if it's dollars, if it's a coding related solution around do we get more CCC or MCC capture or is it a time back to care, uh, for our clinicians, um, which is.

Sometimes a little bit harder. But if there's a significant gap there, I think they're worth, uh, talking to, uh, at least for the interim. Um, because it is a rapidly changing space.

Bill Russell: let's Lets take one of the problems. Let's take length of stay. That's an easy one to solve, right? Length of stay's an easy one to solve. So, uh, um, you know, what, what kind of, what kind of solutions, uh, are, are you looking at to, reduce the length of stay and, you know, what kind of solutions, what kind of technologies are they, are they using and, and what kind of outcomes are they promising today?

I guess.

Joe Evans: Yeah. I mean, again, there's a lot in that marketplace that is, that is one of the areas where we've set up a kind of. [00:14:00] Agile pod that is specifically focused just on, you know, level of care. Are they, you know, inpatient obs, how can we optimize that length of stay? And, uh, they're working in a agile format with regular sprints focused on looking through everything in the marketplace. But yeah, that, that is as many know, kind of one of those, things along world peace, world hunger, hard thing to solve.

Bill Russell: Well there, there's only so far you can push that, right? I mean, they have to stay as long as it's necessary. But no longer than is necessary. I, I, I remember, uh, having an interview with somebody and they were talking about the fact that one of, one of their findings on length of stay was they couldn't get the doctor to discharge them.

Like they just couldn't find the doctor at the time that they were ready to discharge the, the, the patient. And, uh, you know, some patients were sitting an extra. know, [00:15:00] 12, 14 hours because we just couldn't, uh, do, the process wasn't there, the communication wasn't there, the whatever. I mean, so I

Joe Evans: Sense?

Bill Russell: an agile team identifies those things, but a lot of that is organizational change more than a technology solution I would imagine.

Joe Evans: Yep. Yeah, a hundred percent. You know. Kind of process redesign, thinking about it. 'cause as you said, it's such an orchestration. It's, you know, care management. Thinking about what those discharge milestones are. Um, you know, what social determinants, what does it take to get them home or to, what does it take to get them to a skilled facility?

Um, I think that's why it is so, not only complicated but complex. Um, because there's so many. So many variables.

Bill Russell: How, how are doctors thinking about these, these recent movements, uh, specifically, you know, the, the chat GPT for healthcare, that they're really, your patients and they're saying, Hey, come to us when your doctor's not available. Come to us, talk to us about, [00:16:00] about your situation. And then they're, they're touting these stories of, you know, this person had. know, this diagnosis, they wanted to learn more. They went to chat, GPT, they were comforted and, and those kinds of things. I, how do, how are doctors perceiving that, that push by, by open ai right now?

Joe Evans: I think it is just a little bit of an advance. I mean, we all had Dr. Google before this, right? I think this is just the next iteration and I worry more about people that would reach out to GPT Health or. Healthcare and be relieved and say, oh, well they said it's, you know, that burning in my throat is probably just reflux and I should take some malo and I'll be fine, but it's something more significant.

Um, and that is what I worry about is not. How we as physicians think about it, but how patients, uh, think about it and if they are more comforted and [00:17:00] feel that they trust it more, that they wouldn't seek care. Mm-hmm.

Bill Russell: You know, it's so I take my blood pressure every morning, I'm pointing over there because that's where I take my blood pressure. Um. And for a while there I was logging it and then I took the log and I put it in the chat, GBT, and it must've said three times, consult a physician. I mean, they, they are, they're making sure that it's, it's, uh, they are, are. Making recommendations or, or giving you an an analysis is probably, they're not giving you a diagnosis. They're giving you an analysis of what you're telling it they're saying, Hey, look, your blood pressure's within norms, one or two of these are outside the norms. Um, it starts with, you probably should talk to a doctor.

It ends with, you should probably talk to a doctor in the middle. It says, this reading probably means you should talk to a doctor. Uh, so they're, they're try really trying to cover their bases there. Um. I think the big difference between chat GBT and Dr. Google was Dr. Google didn't do [00:18:00] analysis. It said, oh, you're asking about this. Well, WebMD has something for you and, and this, uh, Mayo Clinic site has something for you and you can go to, you know, here's some more information for you. Um, and so. Patients would come in with these references and maybe even stacks of paper. I've heard stories of people coming in with, like, handing these to their doctor and say, you know, this is what I read about my thing.

And the doctor's looking at them like, I have, I have 15 minutes to see you. What do you think I'm gonna do? Read all this and, and talk to you. It's like, uh, they would come in with that, but it, it, Dr. Google didn't necessarily, you know, make. Uh, I'm not gonna say diagnosis again. I'm gonna say do analysis based on the information you gave it.

Whereas and chap, GPT, are actually making analysis of, Hey, this is, you know, you, you, you could have this or this, or this. and, uh, again, they're very careful, but they're, they're telling people, Hey, you know, consider these things. It's more of an answer system. Then it is [00:19:00] a, uh, a, a card catalog of, Hey, you could look here, here, and here to find information. I, I don't know if that distinction, resonates with physicians or not, but, it, it almost seems to me to be almost a little more dangerous. 'cause it's, it's getting into the prescriptive, uh, territory.

Joe Evans: Well, I mean, it's good to hear that open AI is heavily on the conservative side with saying, Hey, see your position. Um, but, um,

Bill Russell: But

Joe Evans: but yeah.

Bill Russell: see the physician.

Joe Evans: Well, you know, not always, uh, right. I mean, there's some things that are simple problem focused. Hey, I stepped on this sea urchin. Um, is it gonna be something that I need to worry about?

Or is it benign? You know, you take a picture of it and, uh, tells you what it is and say, put some peroxide on it and you'll be fine.

Bill Russell: I can see that I'm on vacation. I don't wanna call my doctor. And,

Joe Evans: Exactly.

Bill Russell: how do we, you know, there's, there's, uh, a lot now that's been written about [00:20:00] the nurse shortage and about, uh, primary care doctors, not, not enough primary care doctors and whatnot. How are you looking at technology to, uh, to, uh, assist with that or to alleviate that in some ways?

Joe Evans: Not only nursing, but as you said, physicians apps. Definitely short, um, leveraging technology to help us triage or orchestrate the patient journey, um, to get them to the right level of care that they need. I think it's a tremendous opportunity. If we think about our complex specialties, like we'll take cardiology where, you know, a cardiologist is not just a cardiologist.

They might be, you know, ep, they might be a structural heart person, um, but be able to use technology to help see if they get a referral. You know, does this need, number one, does it need an appointment at all or is there a, uh, a PP that can review the chart and say, Hey, you know what? I think everything looks good.

Let's just [00:21:00] have a quick telehealth visit and I'll walk you through this and see if there's any need for escalation or, um, you know, orchestrate to the right person to get them to, because I think access is a huge problem and I think leveraging technology in a smart way, uh, to get them to the right person, the right, um.

Kind of level of clinician, I think is a, uh, tremendous opportunity for us.

Bill Russell: And, and is that the kind of thing you bake into the portal or how does, where, where does that get baked into the, to the process? Is that, is that in the EHR? Is that in the portal? Is that in, um, you know, some other systems that you have that.

Joe Evans: Yeah, I mean, I, I think it could be the portal. It could be, um, you know, we have a, uh. Single call center for our ambulatory sites that, uh, with which we call our clinical access center, um, it could be baked into there to help them orchestrate, getting them to the right person. That might go beyond what we would get through a decision [00:22:00] tree, uh, through our typical, you know, EHR vendor.

Um. That would be the first step. But then once it's, building on the cardiology example, once it goes down that pathway, customizing, what is the next best step? Is it a A PP? Is it a centralized a PP? Is it our, um, remote telehealth team? Uh, or do they get a in-person visit with one of our, um, cardiology specialists?

Bill Russell: Outta curiosity, just to give people a little flavor. Um, outside of work, what's, what's the, what's the coolest thing you've done with, uh, with one of the, uh, one of the, uh, frontier models?

Joe Evans: So outside of work, um, last year my wife and I went to France for, um, 15 days, went to three regions. Uh, I'm a, uh, Gemini. Fan, uh, use deep research and, uh, basically it. Planned our entire trip. Um, you know, you know, based on some key prompts, I did, um, a prompt for each region. Um, and it was remarkably accurate.

Um, it had one fall down in the South France [00:23:00] where it recommended a restaurant that had a fire, you know, a few months before. And hadn't reopened, but outside of that, it was on Target and you know, for each region probably hit, you know, had at least 400 references. Um, so that would've spent, you know, taking me a long time in Rick Steve's books.

Bill Russell: But, but that's part of the fun of, uh, fun of travel, isn't it?

Joe Evans: Yeah.

Bill Russell: and all that stuff.

Joe Evans: Uh, I still have them, but it, it did, it gave us, you know, in less than a half an hour, um, a good starting point. I still bought the books though, so Rick Steves is safe.

Bill Russell: Yeah, it's, uh, it's, it's really interesting to think, I think travel is one of the things I hear the most. When I ask that question, I, I tend to ask that question at 2 29 meetings just to hear what people are doing. 1, 1, 1 guy for his, uh, four or 5-year-old did a scavenger hunt and, uh, he, he, you know, put all the information in there and then it came up with a scavenger hunt for his kids.

Uh. Fifth birthday party, I think it was. And they, he [00:24:00] said it was a huge hit. It was, you know, a lot of fun. Had riddles in it and had all this stuff. They had the, the party had to solve together. And, uh, he said that was, that was pretty cool. But the number one use case I've heard over and over again is, is travel.

It's like, okay. Um. Especially international travel. 'cause it's, it's not something you do every, you know, every year or even every couple of years. I mean, you'll, my guess is you're not going back to France next year,

Joe Evans: We actually may go back. We'll see. Uh, we felt like, uh, we had been to Paris before, but, uh, yeah, I feel like we missed some Aaron de that, that we might like to look deeper at, but we'll see.

Bill Russell: I don't think there's a more relaxing country than, than France. Like, it, it just, there's something about it that just. I love, uh, obviously Paris is, is to be loved, but when I get into the countryside, I mean, it is, it is just a calming, I I, I think if I took my blood pressure while in the countryside of France, it would, be almost perfect 'cause it's just, it's just wonderful out there.

Joe Evans: you know, my [00:25:00] favorite was probably our time in Provence. We, we had just an amazing time. It was beautiful. It was sweltering in the US and it was cool. We had, um, typical winds that are from the winter, uh, through there. So it was, you know, 18, 20 knot wins every day. Beautiful.

Bill Russell: Well, Joe, I Hey, I appreciate your involvement in the uh 2 29 project coming to our events City tour dinners. I love coming to Charlottesville. You guys have, uh, a, a great group there and, good conversations. And, uh, the thing that's distinct about the Charlottesville event is people will drive two hours to come into that, that city tour dinner, which, which is, which is kind of fun.

It's, it's good to hear from, uh, all over Virginia when, uh, when we do that. So I wanna thank you for that and I wanna thank you for your time today.

Joe Evans: Yeah, Phil, thank you and always great to talk and really appreciate the opportunity to be here.

Bill Russell: Thanks for listening to the 2 29 podcast. The best conversations don't end when the event does. They continue here with our community of healthcare leaders. [00:26:00] Join us by subscribing at this week health.com/subscribe.

If you have a conversation, that's too good not to share. Reach out. Also, check out our events on the 2 29 project.com website. Share this episode with a peer. It's how we grow our network, increase our collective knowledge and transform healthcare together. Thanks for listening. That's all for now.