AI is everywhere in healthcare now. Doctors are conflicted

Medical AI tools are transcribing patient visits, responding to doctors' questions, and analyzing scans. What are the benefits—and the risks?

• 0:02Speaker 00

  Hey, it's Flora and you're listening to Science Friday.

• 0:06Speaker 01

  I am a endocrinologist and professor of medicine.

• 0:10Speaker 02

  I'm a psychiatrist. I am a hospital medicine physician.

• 0:14Speaker 00

  I am a medical student. We're headed to the doctor's office and we're asking, how are doctors using AI in their practice? We asked our physician listeners to call in with answers and many of you did.

• 0:28Speaker 01

  I use artificial intelligence every day in my work.

• 0:32Speaker 03

  I've been using it for less than half a year and I find it to be extremely helpful.

• 0:38Speaker 04

  One of the ways I use it is when a patient asks me a question directly in their electronic chart and it automatically generates a reply that I could potentially send back to the patient. Sometimes the AI version of the answer is much more polite than my version would be of the test.

• 1:02Speaker 00

  What we heard from doctors mirrored broad trends, that more and more doctors are using AI tools for diagnosis, paperwork, and getting up to speed on patients. So how is AI changing medical care and how's it affecting patients? Here to sort fact from hallucination is Dr. Jonathan Chen, Director for Medical Education in Artificial Intelligence at Stanford University. He's studied the accuracy of these models and how healthcare workers use them. Jonathan, welcome to the show.

• 1:32Speaker 05

  Thanks so much for having me, Flora. Looking forward to a very dynamic conversation of this changing under our feet.

• 1:38Speaker 00

  Me too. I mean, can you give me a sense of how dynamic this is? I mean, how much the daily workflow has changed for doctors in the past year or two?

• 1:48Speaker 05

  I mean, I liken it to, it's like the internet got invented three years ago or rapidly trying to adapt to it. I worked in the space, AI and medicine, 10 years ago, right before it was, before it was cool. And six years ago, I had students wearing this. This stuff was unusable. You wouldn't even bother talking about it. Three years ago, ChatGPT blows up on the scene. It's like, whoa, this can kind of barely pass a medical exam, but it's barely passing. That thing's not smarter than me. And now it's like, no, it's basically as good or better than most doctors be answering medical questions. And we're still figuring out how to integrate that in a responsible and effective way while managing very predictable harms at the same time.

• 2:26Speaker 00

  How many doctors in the US are using AI for their work? Do we know?

• 2:30Speaker 05

  I mean, I bet a lot of them are using without even realizing it, right? If you just go into Google search, it is starting to integrate AI into those responses, whether even intended to for not. So more and more, I think the majority are starting to do it. Two years ago, when we did a study, maybe a third had never touched it before, a third used it once or twice, maybe a third of docs had used it two or years ago. But now it's clear the majority are starting to figure out, at least have touched it. But that doesn't mean they know how to use it effectively. That's still an ongoing battle to figure out.

• 2:58Speaker 00

  What are the tools or platforms that are most often used by doctors?

• 3:04Speaker 05

  The reality is just the generic chat platforms, right? Your ChatGPT, your Claude, your Geminis. Within medical specific ones, there are several have started to blown up. How do you combine this great chat capability but with actual medical specific knowledge references? So now, we just had a study recently, there's, you know, there's Amboss came out, there's this German company, Glass Health on there, but also Doximity has a product and Open Evidence is certainly the company that kind of blew up out of nowhere in the past several years. It became a multi-billion dollar company, trying to create this interface in the form that doctors are used to. And it is pervasive all over the place, when three years ago, you've never even heard of these things before.

• 3:41Speaker 00

  What are doctors using this chat feature for? What kinds of questions are they asking? And are there other uses?

• 3:48Speaker 05

  I mean, for one, there are also these Ambien scribe tools, like, let's just listen to our conversations. Your doctor's not spending their time writing into their note. They could just be talking to you, looking into your face. So there's that kind of thing. A lot of these ones, I think are more compelling because it's really like answering questions. They're really almost like a second consultation. We used to, as doctors, look up an article in a, you know, a medical encyclopedia and try to figure out if it applies to you. I see a lot of doctors and trainees I work with, they don't want to look at an article and then read it and figure out if it applies to you. They just, you just want an answer now, right? Which means you ask an AI system, it'll read the article for you and say, I read the article, this looks like the answer. And that's a different dynamic and a different interface. And I think why people are really gravitating toward it and really kind of

• 4:32Speaker 05

  getting hooked on it. I don't really see how we're going back.

• 4:35Speaker 00

  And what are the problems that these tools are solving for doctors?

• 4:41Speaker 05

  Oh gosh, solve is actually a, is a higher bar. It's certainly assisting with a lot of things, but making sense of and organizing medical information, keeping track of things is actually very powerful. A very common human behavior, doctor behavior, it's like, we're all very smart, but there's unlimited medical knowledge. So we'll often run into our colleague, hey, what's the latest cholesterol guidelines? I literally had a patient ask me, if a patient has a hip fracture and there's some blood around the joint, does that need surgery? Like, I'm not a surgeon. I don't really know. But could I run into someone? Or a curbside consult is a classic phrase, where I run into you on the curbside, can I ask you a quick question? That's the kind of thing, the reality is, AI is pretty freaking good at this kind of thing. And is it perfect? No, but it's also way more accessible in

• 5:26Speaker 05

  ways that just wouldn't be practical for doctors or patients.

• 5:30Speaker 00

  How is, so if a doctor was looking for help with a curbside consultation and they opened Open Evidence, how is that query or that process different from my experience in ChatGPT?

• 5:45Speaker 05

  The reality is it's grounded in the same underlying technology, the large language model technology that's reading your phrase and digesting the internet, autocomplete and steroids, guesses the next word. What a lot of these medical ones are focusing on is, I mean, if a chatbot says, how do you prescribe morphine? I mean, who knows where it got that from the internet? You don't really want to trust that. But here they combine it with RAG, Retrieval Augmented Generation. What does that mean? They'll find an article, a medical guideline, ah, this seems to be talking about your question, and it will read sections of that article and tell you what the answer is, which is nice because then you can trust but verify. We feel a lot better if we can point back to a source and we can say, ah, I can go back and look at this directly.

• 6:26Speaker 00

  What about hallucination? I mean, is that a problem you have to worry about, making up citations?

• 6:32Speaker 05

  It is a very real issue. I won't say that's a solved problem. It's a lot better than it was two, three years ago. Two, three years ago was a huge issue. These chapters would confidently make up citations. And that's worse than if it's wrong, because it's so disarming. It's so, has the appearance of credibility and hallucinations and confabulations could very easily lead you the wrong way. And there have been studies to show people are off, sometimes worse off using AI, because they are confidently following a wrong path. This mail looks so believable. I won't say it's a solved problem, but these RAG-based systems, the ones that look up articles for you and really can, you can click on the link and go to the source. It helps a lot. I would say now you have different kinds of issues. What if the question doesn't have an answer in the literature? Many different kinds of studies are higher quality than others. What if it cites something and doesn't

• 7:15Speaker 05

  really interpret in the right way or interprets a low-quality study? These are very real tensions that it's not an AI problem, it's a fundamental medical knowledge and reasoning problem, that this is a danger because it looks so good. Oh my gosh, it looks so good. People treat these things like they're the oracle, like they're God. And I had to tell my trainees, this thing is not God. It is very powerful, it's really cool, but just because it says something does not mean you can blindly trust it.

• 7:39Speaker 00

  I want to get back to trainees in a second, but we asked our listeners, you know, as patients, how do they feel about their doctors using AI? And we got a variety of responses.

• 7:50Speaker 06

  They ask before they use it, is it okay? And I think that's pretty important.

• 7:55Speaker 03

  I kind of feel a lot of times some big skepticism when I see things, AI will fix this, but maybe this is a good thing. Let's wait and see. I don't think it's ready for the medical system.

• 8:07Speaker 00

  Okay, so here's my question, Jonathan. Have we done the research to know if these tools, which more and more doctors, you say the majority of doctors are using, are helping patients, are doing more good than harm?

• 8:24Speaker 05

  Ooh, that's a, that's a deep question. You know, there are some very interesting studies, not mine, others who study regular people using ChatGPT or other things to answer medical questions. And the AI made patients actually worse. In our studies, it sometimes helps doctors, sometimes it doesn't help them enough. But in patient cases, sometimes it made them worse, because it's not just having the knowledge, right? With a great podcast interview, it's how do you elicit the information, frame the question, interpret the answer the right way? If you don't do that well, it'll go way off the rails.

• 8:54Speaker 00

  Yeah, well, I wanted to ask you about this. I mean, we know prompts matter so much in what you get out of AI. Are doctors getting trained to use these tools?

• 9:04Speaker 05

  At the high level, no. You know, when we submitted one of our studies a couple of years ago, showing what happens when doctors use ChatGPT in this case, a peer review said, that's not realistic. No one would unleash some computer system for doctors to use without properly training them first. I'm like, what are you talking about? That is exactly what they do. That happens all the time. It's happening right now. So most clinicians, patients, are not getting any formal training on how to use these things. One of my jobs now is this newly created role, Director for Medical Education, AI, Stanford, was specifically because we need more of that here and broadly. And it is letting the trainees and the doctors know, these are real tools, you need to understand a little bit how they work, because they're

• 9:45Speaker 05

  clearly going to become a part of your life and your work. Know what the caveats are so you can use them safely, effectively, and responsibly.

• 9:53Speaker 00

  What about procedures? I mean, we know hospitals and medical systems have tons of procedures, often to cover their butts in terms of, you know, liability. Are hospitals or medical centers actively writing new AI usage policies?

• 10:08Speaker 05

  They are, and it's a tough thing, let alone scientific peer review. Usage policies, they're lagging, right? The technology's moving so fast. It's crazy, right? It's moving every month. It's like, shoot, another update. And your policy you wrote six months ago is already out of date. A couple of years ago, a lot of people are trying to ban the technology. We don't know how to deal with it. It's kind of scary. So just ban it. Nobody's allowed to use it. That's completely impractical, even if it were a good idea, which I don't think it is. That's my opinion. And there's not consensus. You cannot effectively ban it because someone could just pull out their phone, right, and work around you. And now they're probably using a non-secure, non-privacy compliant tool to do what they would have done anyway. So more of these policies, more of these frameworks are coming out, but it's

• 10:51Speaker 05

  happening so haphazard, so distributed, because the technology has just moved so fast, that human institutions cannot keep up pace with how fast it's moving.

• 11:01Speaker 00

  After the break, I want to talk to you about medical students and trainees and how they should be using these tools. You down to stay with us?

• 11:09Speaker 05

  Absolutely.

• 11:10Speaker 00

  Don't go away.

• 11:19Speaker 07

  WNYC is supported by Bear. Every day in America, science delivers, not just in the headlines, but in the moments that matter most. Science is a journey, one of curiosity, persistence, and progress. It's about asking questions, testing, learning, and improving again and again. Science delivers harvests that feed families, tools that help farmers grow more with less, and new ways to protect land for generations to come. Let's keep asking, keep testing, keep discovering, so science can keep delivering. Learn more at sciencedelivers.com.

• 11:54Speaker 00

  Hey, everybody. If you like Science Friday, I want to recommend another podcast I know you'll love, too. Science Versus. From peptides to protein-maxing, Science Versus takes on fads and trends and the opinionated mob, I know you know what I mean, to find out what is fact, what is not, and what's something in between. And they're the only podcast with better pun game than us. So check them out, Science Versus, wherever you get your podcasts. You know, one thing that was interesting in the calls we got from doctors is that even though, you know, almost everybody who called us said they were using it, they also were

• 12:38Speaker 00

  conflicted about it for kind of interesting reasons.

• 12:43Speaker 08

  It is definitely something that we are learning about in medical school because it's something that we'll be using in our futures as physicians.

• 12:51Speaker 01

  I do worry that learners will become too reliant on generative AI. And as a result, they may lose some of their clinical reasoning skills. And I also worry about biases that are inherent in the models that AI is trained on.

• 13:09Speaker 00

  Are you worried about this, that med students are going to be overly reliant? They won't be able to vet the information they're getting from these tools?

• 13:16Speaker 05

  It is a real issue. It's a real issue. I say this, a very huge dilemma, and we have absolutely not consensus on this. We recently did a poll, like 60% in favor, 40% against. We had a debate, and then it went to 55% in favor. And yet, you still need a policy on what to do. I'll give this brief anecdote to really punctuate the point. In our medical reasoning class at Stanford, you know, here's a case, what do you think the diagnosis is? What do you think the treatment and management should be? On the homework, on the homework two years ago, the students are killing it. They're killing it. But when it came time for the closed book exams, like twice as many students failed that exam compared to usual. Like, what happened? It's obvious what happened. They're using the AI to do their homework, and then they miss the point. No homework, you're supposed to struggle through homework so that you actually then

• 13:58Speaker 05

  actually learn and are ready for the closed book exam or the live patient interaction, where you can't have AI do it for you. So that's definitely a real danger. On the other hand, when we cautioned them last year, this is a very powerful tool, the best tutor you've ever had if you use it right, but like a chainsaw, it's a powerful tool that you could really hurt yourself with if you don't use it properly. This time, the students actually did better on the closed book exam, and they learned to use it in practice when they're understanding how to use it with the right guardrails.

• 14:27Speaker 00

  We've, you know, we've been talking a lot about chatbots and sort of querying for medical questions, but what about AI for analyzing radiology? Where are we with that kind of use of AI in medicine?

• 14:38Speaker 05

  Sure. Computer vision, actually, I would say is much more mature. That was a very hot thing five, six, seven years ago. It was, here, look through this x-ray, you know, tell me if I have a lung infection. There's a nice study where doctors using a thing to help it find polyps on a colonoscopy, it made them better. And when they turned off the AI computer vision tool, the doctors were worse than before they started, or they got used to it. They started to depend on the technology. So very powerful capabilities. But the famous thing here is Jeffrey Hinton, Nobel Prize winner, kind of father of a lot of neural networks. He famously said about seven, eight years ago, it's so obvious all radiologists should like just stop working, right? They're all going to be replaced by computers within five years. He said this like eight, eightish years ago. Clearly, he's wrong, because now there's more radiologists needed than ever before.

• 15:22Speaker 05

  But it was a shifting and understanding of what the jobs and needs are. Spotting things on x-ray, maybe that's not the task anymore. It's like it's synthesizing, organizing that and managing that for increasing demand for these services, too.

• 15:34Speaker 00

  Hmm. Do you worry about privacy concerns? Like, are these tools compliant? Do we know how the patient data is getting used?

• 15:44Speaker 05

  So it's a huge issue, and that's actually why, you know, one of the key trainings we give our people, doctors and trainees, FYI, you cannot put in real patient information into ChatGPT or Claude or Gemini, whatever. You have totally just uploaded private patient information to a public server if when you do that. And some tech company now knows all of that. It's very easy for that to be invisible and people to not notice that. So that's been a lot of the guardrails in place to manage this. But the companies also have to be responsible for where it can go wrong. Right now, there's too thin of, really fake disclaimer, FYI, you should not use this for actual medical advice, even though I know you're actually doing that. And literally, companies will put out press releases, look at this cancer patient, we're like saving their life.

• 16:28Speaker 05

  Like, they're really talking out of both sides of their mouth, right? It's like, okay, if you want to provide the benefit, you got to take the responsibility. But if something goes bad, literally, somebody can sue me for medical malpractice.

• 16:39Speaker 00

  Are you liable, or is the tech company liable? Like, if you use, if you use one of these tools, like Open Evidence or something else, you rely on it for, you know, medical advice, you pass it along, something goes really wrong, who gets sued?

• 16:53Speaker 05

  Yeah, it's a very thorny issue. Well, if you're a malpractitioner, what you do is you sue everybody and you see where it sticks. Right now, it, in theory, sticks to the doctor. All these tools, all this, at the end of the day, I signed the order, and so in theory, I'm the one who gets sued. If like, I use 20 different tools, I don't even know how all of them work. This magical reference library, this AI, it seemed to have reasonable information. So that's actually is an awkward thing that they have very limited responsibility. The FDA, Health and Human Services, many entities are trying to figure out how to wrap their heads and their brains around this, realizing it's a, it's a very difficult problem, and we have to figure out how to adapt to it.

• 17:32Speaker 00

  Well, speaking of a magical reference library, I read that you're also a magician. Do you see any relationship between this work and that work?

• 17:43Speaker 05

  It is. That's somewhat was a coincidental hobby that spanned out of control, but is really aligned. And being a good magician is actually about having good empathy. You have to understand what another person is thinking so that you can trick them with the other thing and get them to think another way. And so much of AI is like, that's a really, it's so believable, but it can't be real. This thing looks like it expresses emotion. It's right, the caller said, it's more polite. It's not polite. It's not kind. It doesn't think. It's, it's a computer, it doesn't do anything. But man, is that illusion really convincing? And if you want to operate in the world effectively, you have to be able to distinguish. You still need your judgment, not your knowledge, you need your judgment to distinguish what

• 18:23Speaker 05

  is real and what is not, so that you can do some good without hurting yourself or others.

• 18:29Speaker 00

  Dr. Jonathan Chen is an associate professor of medicine at Stanford University. Thank you for taking the time to talk to us today.

• 18:35Speaker 05

  Great to talk to you. Look forward to catching up.

• 18:38Speaker 00

  This episode was produced by D. Peter Schmidt. Speaking of questions, we're working on a segment about peri-menopause, and if, like me, this topic is dominating your group chat, I want to hear a story about how you feel like peri-menopause is affecting you. Brain fog, rage cleaning, a desire to run away from your family and make a new life in a motel room. Whatever it is, I want to hear the story. We'll have an expert to sort out the science behind the symptoms we hear so much about, at least on social media. 8774 sci-fright is our number. Thanks for listening. We'll catch you next time. I'm Flora Litzman.