The AI Will See You Now: Exploring Biomedical AI and Google’s Med-PaLM2 With Karan Singhal

What if AI could revolutionize healthcare with advanced language learning models? Sarah and Elad welcome Karan Singhal, Staff Software Engineer at Google Research, who specializes in medical AI and the development of MedPaLM2. On this episode, Karan emphasizes the importance of safety in medical AI applications and how language models like MedPaLM2 have the potential to augment scientific workflows and transform the standard of care. Other topics include the best workflows for AI integration, the potential impact of AI on drug discoveries, how AI can serve as a physician's assistant, and how privacy-preserving machine learning and federated learning can protect patient data, while pushing the boundaries of medical innovation. No Priors is now on YouTube! Subscribe to the channel on YouTube and like this episode. Show Links: May 10, 2023: PaLM 2 Announcement April 13, 2023: A Responsible Path to Generative AI in Healthcare March 31, 2023: Scientific American article on Med-PaLM February 28, 2023: The Economist article on Med-PaLM KaranSinghal.com Sign up for new podcasts every week. Email feedback to [email protected] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @thekaransinghal Show Notes: [00:22] - Google's Medical AI Development [08:57] - Medical Language Model and MedPaLM 2 Improvements [18:18] - Safety, cost/benefit decisions, drug discovery, health information, AI applications, and AI as a physician's assistant. [24:51] - Privacy Concerns - HIPAA's implications, privacy-preserving machine learning, and advances in GPT-4 and MedPOM2. [37:43] - Large Language Models in Healthcare and short/long term use.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:05Sarah Guo?

  Welcome to No Priors. Today, we're speaking with Karen Single, a researcher at Google where he is a leader on medical AI, specifically on MedPom 2, where he and a team are working on a responsible path to generative AI in healthcare. Google just announced the launch of its next-generation language model, Palm 2, with improved multilegal reasoning and coding capabilities, which is behind MedPom 2. So it's a great time to be speaking with Karen about everything he and his team are working on. Karen, welcome to No Priors.

• 0:32Speaker 01

  Hey, guys.

• 0:33Sarah Guo?

  So you've been working in this field for a long time. Tell us about how you ended up working on medical AI at Google. I think I saw you started a fake news detector for using AI as a 19-year-old.

• 0:46Speaker 01

  Yeah, that was one of my first AI projects. I really got into AI thinking about how it could be used in socially responsible ways. And for me, I was thinking around the time of the 2016 election that maybe a little bit naively, that we could, you know, AI-based solutions could be a bit of help for, you know, things like misinformation and detecting that. I think in the longer run, I mean, I've thought of that as kind of a more naive project. And I think in the longer run, I've been thinking more about, you know, how it can help shape the trajectory of AI to be more beneficial and more broadly. And I think for me, thinking about the medical setting has been motivated largely by thinking about the fact that, you know, it's a great place to think about concerns around safety, reducing hallucination and misinformation as well here.

• 1:31Speaker 01

  You know, thinking about how we can produce medical question answers that are less likely to be harmful and all these kinds of things. And, you know, that motivation, I think, has driven us to this point where really going for the jugular in terms of thinking about how to train these models, make them better in the setting. And so very excited about that kind of work.

• 1:49Sarah Guo?

  Have you been working on the medical domain your entire time with Google?

• 1:54Speaker 01

  No, I mean, for me, this is just something I've gotten to the last, like, year and a half. So I've been new to it. I've been learning from an excellent team, and it's been an amazing journey so far.

• 2:02Sarah Guo?

  What else has been the most interesting in your work at Google so far?

• 2:07Speaker 01

  Yeah, I started working out in representation learning and federated learning. So this is kind of the technology, representation learning in particular is kind of the technology underlying a lot of the, you know, deep neural networks of today, including G P three, G P four and so on. And so this is largely about learning representations of text, of images, of other modalities, such that you can efficiently encode them, you can learn from them in the future, you can generalize to new texts and images and so on. So work for this really started, you know, back in the beginnings of the deep learning era, like in 2013 with convolutional neural networks and scaling those up and work to back around 2015 and glove and all these

• 2:48Speaker 01

  things. And I think since then, you know, we've been working on technologies around self-supervised learning, around, you know, doing that in a privacy preserving way. And so, you know, after a couple of years of working on that at Google, I had the opportunity to kind of quickly grow and start to lead a team. I kind of got to the point where I was thinking like, okay, I've up scaled in a lot of ways. I've gone to the point where I can mentor many other researchers in a lot of ways. And now it's a great time to be thinking about my next thing and, you know, going for something ambitious in terms of shaping the trajectory of AI. And so, you know, about a year and a half ago, a few of us had the idea to think about this medical setting as kind of a setting in which, you know, these concerns are especially important. And there was a ripe opportunity to think about this paradigm of foundation models and

• 3:31Speaker 01

  medical AI. And so within Google, we had the opportunity to pitch what's called a brain moonshot, which is a kind of like an internal incubator program for ambitious research projects. And this is, you know, a lot of cool research projects that you've heard of in Google have eventually come out of this program. As we pitched that, we got it accepted and funded, we got the ability to kind of get a bunch of compute to bring other folks on board with the sponsorship of a bunch of leaders. And our first thing together was really met palm. And so that was, you know, a really amazing thing for us to be able to work on together.

• 4:01Speaker 02

  Can you talk a little bit about Palm and how that's related to met Palm and what Palm is to begin with? And then how met Palm is different?

• 4:09Speaker 01

  Yeah, absolutely. I mean, so the original MedPom work built on this model called POM, which stands for Pathways Language Model. And so this is really an infrastructure that Google has built to be able to scale up large language model training that is kind of Google wide. And so the first POM model was released in 2022, which was kind of this 540B decoder-only transformer model at the time, the largest densely activated model. And, you know, it kind of realizes breakthrough achievements in code and multilingual capabilities and reasoning. And so I think a lot of the work with respect to kind of improving benchmarks specifically

• 4:50Speaker 01

  that we're seeing with like POM, MedPom, GPT-4 recently, I think all comes down to a lot of the improvements that were made during POM, during the training of POM. And so, you know, shortly after POM, there was this Minerva work where maybe like a few months after the POM work itself, people were able to show that on STEM benchmarks, that there was this kind of zero to 100 or zero to 60 at least effect where, you know, you went from random chance to, you know, solid performance across a bunch of benchmarks. And that laid the foundation for a lot of the work that Jason Way and others have had on, you know, thinking about immersion abilities in large language models. And so for us, that was part of the motivation

• 5:30Speaker 01

  for looking at multiple choice benchmarks as well for MedPom. And so for MedPom in particular, what we did was we took MedPom, this kind of general large language model, trained on web scale data, and then kind of further aligned it to the medical domain. We evaluated it base, but also thought about like, given its limitations in long form medical question answering, thinking about things like safety, factuality, low likelihood of, you know, outputting an answer with bias. What do we need to do to kind of better align that model with this domain? And so really MedPom was an attempt to do that.

• 6:05Speaker 02

  Yeah. So basically, it sounds like you started off with POM and POM was tested against a bunch of different types of tests, right? And so You could take the MCAT or you could take other types of effectively tests for professional accreditation or for knowledge understanding. And then it sounds like you then said, hey, this this seems really interesting. Right. We're starting to get really good performance here. And so can we do something that's in the medical domain specifically? And that was MedPOM. And so how did you do that alignment that you mentioned? Was it some form of RLHF? Was it some other form of fine tuning? Was it how you train the model to begin with? Like, what was the difference in terms of MedPOM versus POM?

• 6:41Speaker 01

  Yeah, absolutely. I mean, when when we tried evaluating POM in the medical setting, we noticed that out of box on multiple choice questions, performing pretty well. And when we took a variation of POM, the flanPOM model, which was, again, worked from Jason Way and team. You know, this is an instruction to a model, a model that's been trained to follow instructions better. You know, again, it was able to perform quite well out of the box. And this was the first model that was able to perform above the pass mark on the MedQA set of USMLE style questions. But then what we noticed is that when we evaluated it on long form medical question answering, like actually getting the model to generate response, there was a lot of limitations. And we compared that to clinician performance, it actually didn't do super well.

• 7:25Speaker 01

  And so really, that was the motivation for that MedPOM specific alignment. And so what we did there was really thinking about instruction prompt tuning, which was this technique, which we explored in that in that MedPOM paper, which is kind of a data efficient technique and a technique that doesn't require too much data to work because, you know, getting labels from doctors is expensive, which took a bunch of expert demonstrations of good behavior from doctors and then use that to tune the parameters of the model and do that in a way that's a little bit more learned than prompting, but also less expensive than full fine tuning.

• 7:58Speaker 02

  And so you did that. And then I guess if you start looking at this now shift from POM to POM 2 and from MedPOM to MedPOM 2, did you basically just reproduce that same approach for MedPOM 2? Or did you do anything different there?

• 8:11Speaker 01

  Yeah, this is the work of many folks other than myself. So just to preface it with that. I mean, I think a few things that have been important have been. One is better objectives for pre-training and using something like a mixture of objectives, training objective. And so that's been something that's been crucial. And so this is work that started with UL2, a paper that was released also last year. And then two other things that ended up being super important. One is following the optimal scaling laws that were empirically evaluated again in this work. And there's been a few works that have tried to do this from OpenAI and DeepMind. And again, this work tried to understand in this context, what are the optimal scaling laws with respect to data and compute? And how do you trade those things off?

• 8:55Speaker 01

  And so this paper, again, found something similar to the Chinchilla paper, which was that the total amount of data being used for these models was relatively low compared to the number of parameters. And that if we wanted to add in more data, we could do so. We could train a better model in a more compute efficient way. So this model also did that. So that's an important improvement as well. And the third thing was improvements in the data that were used to train the model. And so this especially focused on multilingual data, including more multilingual data and more code data in a bunch of different coding languages as well.

• 9:29Sarah Guo?

  Maybe just zooming out a little bit in terms of when you might apply some of these different techniques to align a model to a specific domain. Like, do you have a framework in your mind for why you might do full, like, pre-training from scratch? Why you might do fine-tuning, why you might do a more efficient form of fine-tuning? And when you can just get away with prompt tuning or prompting? Like, how do you think about that?

• 9:55Speaker 01

  Yeah, this is a great question. I think it really comes down to the data that's available, both in quantity and relevance to a particular topic. I think if you have an infinite supply of data that's relevant for the specific problem that you're trying to solve, then probably the best thing to do is pre-train everything from scratch and do everything end-to-end. And if you don't mind compute and money as well. If you are working on a task in which, you know, general pre-training data in the web confers general advantages to that task. And so that could be domain knowledge, it could be general abilities like reasoning, you know, which is very applicable across many tasks, which I think is, you know, the case for medical reasoning as well. Um, then I think it makes a lot of sense to build on top of an existing model,

• 10:39Speaker 01

  especially if you're sensitive to things like cost or compute, you know, which most people are these days. And so, um, you know, I think on that spectrum between things like prompting and um, prompt tuning, uh, all the way to like full fine tuning. I think it largely comes down to, um, so, so given an existing pre-trained model, you know, which is, I think, a big hurdle for, for most uh, teams and most people uh, to train a large scale, a pre-trained model, then the question is, do you prompt it? Do you prompt tune it? Do you full fine tune it? I think that largely comes down to data. If you have three to five examples, let's say, then I would prompt it. If you have, um, maybe 10 or 50 examples, it would either be prompt tuning or fine tuning.

• 11:23Speaker 01

  I think generally in that realm, prompt tuning and, and fine tuning performs similarly. And I would prefer prompt tuning if you're at all sensitive to things like compute or cost. Um, if you care about the best performance and you have more than a hundred examples, then probably fine tuning is your best bet. And it's not as expensive as full pre-training, um, if you're doing it with a model that's been pre-trained, of course.

• 11:45Sarah Guo?

  When you thought about evaluation of this model, um, you must have been surveying the landscape for the other sort of medical, um, probably, you know, science-specific and then medical-specific models, like wha- what's out there? And how did you guys think about evaling and changing eval?

• 12:04Speaker 01

  Yeah, absolutely. And this is not the first work to explore the potential of a large language model in science or biomedicine. And so I think it's, you know, important to acknowledge the, all the work that's come before us. What we saw when we, you know, first came into this work and tried to understand what other models existed, what other evaluation has been done was that, one, there was a few exciting works from other teams, uh, like Lactica or BioGPT and so on, that we thought we could learn from and, you know, benefit from. And so that was, you know, a really exciting thing to be able to see. And the second thing we saw was that there was a bit of a shortage of kind of a systematic way of doing evaluation of these models. And so it didn't feel like there was a systematic way to think about automated evaluation of the clinical knowledge of these models.

• 12:49Speaker 01

  So for example, via multiple choice benchmarks, there were a few popular benchmarks like the MedQA benchmark, but you know, it varied across paper, what benchmarks they were studying. In some cases, we felt like these benchmarks were not high quality. And so that was one thing that we saw. And another thing that we saw, which was more acute, I think, was kind of a lack of detailed human evaluation across many of these works. And so there was some steps in this direction that we were able to build on, but I think for the most part, a lot of these models that have already existed didn't have kind of detailed human evaluation given a use case like medical question answering. And so I think that to us was a significant limitation as we think about, you know, the real world potential of these models, because you know, when it comes down to it,

• 13:33Speaker 01

  we had to make sure that it actually serves humans and is beneficial to humans. And so for us, that was like a significant motivator for the MedPalm work being relatively evaluation forward and thinking carefully about human evaluation with both physicians and laypeople.

• 13:48Speaker 02

  How do you think about where that bar is? Because I think it's one of those things that, you know, having started a medical centric company before, on the one hand, you really want to be cautious in terms of providing people back information that's accurate, right? And so when I was working actively as on the operating side of color, we spent a lot of time agonizing over the ensuring that the results that we provided back to patients were as accurate as possible, particularly in the context of anything that had to do with, you know, core genetic or other information. The flip side of it is, you know, I remember I took my son to the emergency room when he was younger and the doctor said, I'm going to go research this case and I'll be right back. And I had to go ask him another follow up question and I go around the corner and he's

• 14:33Speaker 02

  in his cube, literally Googling the symptoms, right? And so it wasn't like he had some deep, accurate source. He was just making things up, right? Effectively, right? I mean, I've seen Google results and you're kind of clicking around and he was just clicking around, I was like, oh my gosh. And I could see the query, right? So I knew he was looking at my kid's symptoms. He had no idea, right? And so there's this bar from, hey, it needs to be incredibly accurate and correct on through to, well, the state of the art actually isn't that amazing in many circumstances. And so how do you think about the right quality bar for these sorts of things? In terms of real use, application or practice?

• 15:06Speaker 01

  That's, that's an amazing, great question. I think, I think as you said, there's two competing forces here, right? Obviously, the stakes are high in the medical setting. And counterfactual, you want to make sure that the information you provide versus the information they would have otherwise gotten is actually high quality. And so that's, you know, very, very careful as you think about, you know, any informational use case for these models. At the same time, I think it's useful to recognize that people are searching for health information online and indecision is a decision as well. And so, you know, a large percentage, roughly 10% of searches on the internet are for health information. And some of these are coming from physicians themselves, as you, as you mentioned in a lot. And so, you know, I think that there is a responsibility to think about how to shepherd

• 15:50Speaker 01

  this technology carefully and safely towards that real world impact for patient health information. And I think that, you know, is crucial as well. And I think one thing that has been missing from our work so far is really grounded evaluations in a specific use case in a workflow to show that there is a benefit, both in terms of safety in the short term and in terms of kind of long-term patient outcomes as well. And so, you know, I think that could be a health informational use case. It could be other clinical workflows, but you know, I think that's one thing that we have to really make sure we do and, you know, are careful about before any kind of real world use case here.

• 16:27Speaker 02

  Yeah, that makes sense. Yeah, it definitely feels like in the medical world, the importance of safety is paramount. And at the same time, there's very little cost benefit being done anymore. And so there's, you know, interviews with Janssen and other sort of giants of the industry basically saying, you know, we need, we need to think about the benefit side, not just the cost side or the safety side. And what you're working on, I think, is so important in terms of, if you think of the really big areas of societal impact, it's what you folks are doing, right? If you could provide amazing health equity globally for everyone in terms of this information, how powerful is that? I mean, that's fundamental. And maybe education is the other one, right? And it feels like AI really has a promise in both of these areas. And so I always worry about, you know, how do you make sure that this can get to market? Because it's so valuable, but there's going to be all these regulatory or safety obstacles

• 17:12Speaker 02

  that in some cases are merited, but in some cases may actually prevent. The emergence of really important applications. So I think it's awesome that you folks are working on all this and are being so thoughtful about it. How do you think about what workflows this is going to be most useful for? So, you know, if you look at a lot of the bio or biomedical AI companies, for some reason, they keep doing drug development. A, why do you think that is? Like, why? Because this seems like such an important part of healthcare and probably the bigger driver of healthcare efficacy. And so, A, why is everybody just going and building another protein folding model or, you know, molecular company? And B, where do you think are the best applications of what you've been working on?

• 17:52Speaker 01

  Yeah, these are great questions. I think on the drug discovery front, there are, there's a bit of a playbook here, which any new company here looking for some revenue in the short term can follow. And that, you know, that could be a safe option. Like, there are, for example, existing AI augmented pipelines for doing things like giving small molecule chemistry, predicting things like absorption or toxicity. And it's kind of relatively easy to see that, you know, some of the more modern models, if placed into these pipelines, could perform better. And so there's like a relatively safe bet there. And so I think that probably accounts for a lot of the popularity of that as a use case. I totally agree that like there is a kind of a chance to go for the jugular here in terms of health information, for example.

• 18:37Speaker 01

  And so, you know, I think this is something that is going to be crucial, but I think it is also something where a lot of the big players are more risk averse. And so, you know, the people who get access to health information or provide access to health information are also thinking, not maybe super counterfactually, about the positive benefits of things. And they're thinking more about the risks. And so, you know, I think that is also, you know, a concern that's been slowing folks down, both in terms of big companies and smaller companies. And, you know, I think there is an opportunity to kind of think more about that and what that could look like. And I think the company that, you know, gets that right, or, you know, the set of companies that get that right, I think will also have a seat at the conversation when it comes to policy and regulation and things like that.

• 19:22Speaker 01

  And so they have the chance to shape, you know, what this looks like for the future. And so, you know, I think that's going to be potentially quite impactful.

• 19:29Speaker 02

  Yeah, it seems very exciting because if you look at healthcare, it's 20% of GDP. Pharmaceuticals are about 20% of that. And then drug development is a fraction of that, right? So really what you folks are focused on in terms of the types of models that you're building is at least, you know, 16% of GDP. You know, maybe it's more than that if some of the pharma stuff is, you know, more clinical decision making around who gets a certain pharmaceutical. Do you view this as a technology that's initially a physician's assistant? Do you view it as something that helps with adjudication of medical claims and billing? Like, there's so many places where this can sort of insert. I'm just sort of curious, like, you know, where do you think you'll see this technology popping up first?

• 20:10Speaker 01

  Yeah, I think we're already starting to see it in some clinical workflows when it comes to documentation and building. I think there are a lot of companies and people thinking about taking models like GPT-4 and applying them in that setting. And I think that that is definitely going to be something. I think that is also going to be something where players like Epic are going to be able to partner with existing models and I think potentially deliver real value there. And I think that's very exciting. I think that's something that also, you know, general domain models will be potentially quite good at as well. I think where there might be more of a need for specialized models is when it comes down to kind of higher stakes workflows.

• 20:52Speaker 01

  And I think that might look in the short term more like a physician's assistant. And so imagine, for example, an agent that can work with the radiologist, help them interpret a scan and leverage the benefits of AI to kind of help contextualize, you know, a patient's medical record or any previous scans or different angles of scans that a patient has had to help a radiologist write a more accurate report. I think that's something, that's the kind of thing which I think, you know, is in the sweet spot of both feasible today, you know, leverages the benefits of AI in terms of taking in additional context and, you know, potential multimodality and all these kinds of things. And it's also potentially in a sweet spot with respect to regulation as well.

• 21:32Speaker 01

  And so I think that's something that could happen in the shorter, medium to short term.

• 21:38Speaker 02

  How do you architect a model or workflow in this context to deal with things like HIPAA or patient privacy? So I feel like healthcare data is unique from the context of what you're allowed to do in terms of who you send it to, with what permissions from users. So is it just you have to get the right user opt-in and then it's fine? Or is there extra work that you need to do in terms of blinding data or doing other things relative to the prompts or queries you're sending in?

• 22:01Speaker 01

  Yeah, it's a great question. I mean, I think this is something that people are just trying right now and just seeing what happens. And it's kind of interesting. People are just putting in patient information into GPT-4. Sometimes they're redacting information and all these kinds of things. I mean, I think the like, the ideal way to do this obviously is is more privacy forward, I think, in terms of building trust with the relevant stakeholders and all these kinds of things. You know, I think a starting point is just models that are able to automatically redact very sensitive information from, you know, you know, being sent further down a pipeline. I think that's something that's like a very low hanging fruit that, you know, many people can do. There's also potential for HIPAA compliance within organizations.

• 22:43Speaker 01

  So I know some organizations working in the space are partially HIPAA compliant or are kind of trying to make that claim. And I think that's, that's something that's useful. And I think that's something that we should work towards as well. You know, I think in the in the longer run, I think a lot of these concerns, I think, will are actually unclear in terms of how things will work out. Like, I think there is kind of a bigger question about software of unknown provenance and how that will be used and regulated, you know, in the future. There could be some kind of situation in which like these things actually end up being very hard to scale up and apply in the real world for, you know, high stakes settings. But I think we'll probably end up with a scenario where it'll become obvious that we need to and that we must, and that doing so will improve patient outcomes. And

• 23:28Speaker 01

  so then I think it'll be, it'll be time to have like a serious conversation about what regulating these models and making sure privacy concerns are mitigated looks like. And I think, you know, I think we have, we have yet to have that discussion.

• 23:40Speaker 02

  Yeah, HIPAA is kind of interesting from the context of it was an incredibly well intentioned piece of legislation. But the flip side of it is, it's really backfired in all sorts of ways in terms of actual patient good. And you see that sometimes as well, and in terms of some of the things that, that as you sign up for a clinical trial or other that you can actually do with your own data, where sometimes you're constrained from accessing it. I know of one example where somebody had brain cancer. They had a glioblastoma and it was a researcher at MIT. And he participated in a small clinical trial. And then they wouldn't, they were unable because of compliance to give him his own data. So that he could try and discover drugs against his own glioblastoma, his own brain cancer. Right. And so sometimes you see these very well intentioned approaches in terms of the

• 24:24Speaker 02

  protocols around a clinical trial or on HIPAA or other things that are very well intentioned in terms of what they want to do, but then sometimes they may backfire as you start to enter the modern data world, since I think that legislature is now almost 30 years old, right? And so I just think it was set up for a world that's very different from what we have now in terms of, you know, the liquidity influency of your ability to interact with information and, you know, patients driving their own diagnoses and things like that. So I, you know, my hope is that some of these things get rebalanced in the AI world, since it could be so valuable to things like what you're doing.

• 24:54Sarah Guo?

  I was, I was just gonna say, that is the status quo. And you've also worked on the, like, the areas of, you know, privacy preserving machine learning and federated learning. Those areas have broadly taken a backseat to, let's say, like, scaling and aligning these more centralized models. Like, do you see a place for that technology in this field?

• 25:15Speaker 01

  Yeah, that's a great question. So, I mean, as I mentioned before, the first couple of years of my career, we're really thinking more about privacy preserving machine learning and federated learning and scaling that up and coming up with new algorithms that can learn new things without, you know, sending all the data to a centralized place. And so in, in a lot of ways, that has a very, very natural fit with the setting. And, you know, part of my motivation, when I first started working on the setting, was bringing in a lot of that expertise and bringing into that setting. My sense is that, I, I think one hesitation I have there is that I think a lot of, you know, the most impactful work that's going to happen in this setting is going to happen with the largest and most capable models, at least for the next few years, it seems

• 25:57Speaker 01

  like. And I think that, like, one, one thing that we're seeing is that even without any patient health information put into these models, like, for example, MedPom and MedPom 2 are trained without any patient health information. They, they're just kind of taking all the knowledge of POM and POM 2 and then just kind of. Aligning them and making them behave in a certain way. I think in the short term, there is this kind of thing that we see where models like G P T four and met palm, met palm two are able to do surprisingly well without any patient health information. And so it seems like we can get fairly far with that. I mean, in the longer run, I do think that like, you know, coming back to that question of of data and how do you think about how to train a model depending on how much data you have and how relevant you have, how relevant that data is.

• 26:42Speaker 01

  The ideal thing would be to have access to all of the data, but in a privacy preserving way, in a way that people are in control of their data, are able to revoke access to that data and are able to kind of benefit from that shared understanding of their data. And so that's the kind of the ideal world. But I think there are like real world obstacles to doing federated learning on health data, which actually kind of increase the activation energy to the point where in the next few years, I doubt that like the biggest advances are going to come from using federated learning approaches. But I think there are kind of intermediate solutions, which people often sometimes refer to as federated, but maybe are not technically federated, which are things like trusted execution environments or other environments in which models are running, but you know, don't have

• 27:25Speaker 01

  the folks at Google don't have access to the to the data, direct access to the models. And so there is this ability to kind of silo that from, you know, silo any patient health information in the future potentially. Or, you know, any other data that's quite sensitive from engineers or other folks at, you know, big companies or small companies.

• 27:46Sarah Guo?

  Yeah, going back to perhaps more promising near term areas of of research. You've had this idea of building a medical assistant as a sort of laboratory for safety and alignment research. Can you talk about that?

• 28:01Speaker 01

  Yeah, absolutely. I mean, this is a lot of what got me thinking about the setting, especially coming into the setting as somebody. Who, you know, didn't have much of a medical background in terms of expertise. I was really thinking about, you know, what are the, what are the big things that I could do to help shape the trajectory of AI or nudge it in a more beneficial direction? And thinking about AI safety seriously in terms of both short term and longer term risks, I think was important to me. And so, you know, one thing I've, I've become more convinced of about over time is this idea that, you know, many organizations right now, Google, DeepMind, Anthropic, OpenAI, are right now looking at the idea of a general chat assistant.

• 28:41Speaker 01

  And kind of instead of like doing alignment research in a vacuum, are looking at that setting as a way in which we can think about kind of better refining these models and better aligning them to human values. I think there's a good chance that this setting, this medical setting, for example, medical question answering, or maybe more broadly, I think ends up being a better scenario to study concerns about technical safety and to mitigate concerns like misalignment with human values or hallucinations or things like that. And so, I mean, I think this comes down to things like making sure the incentives are aligned with respect to releasing products. So for example, I think if any organization wants to release products in the space, it actually needs to work on these problems more so than I think ChatGPT.

• 29:22Speaker 01

  I think it also comes down to kind of the stakes of the setting. I think everybody feels like the stakes of the setting are high enough that everybody feels like these issues are, you know, especially important. Hence, no debate about that. And I think there's also like some more subtle technical points, like, I think one issue that, you know, alignment researchers are now working on is idea of scalable oversight, which means how do you give human feedback to a model when human feedback might not be super well informed or it might be unreliable? Because AI capabilities are starting to reach human level. And so, when we start to get to that point, like things like RLHF start to fail and starts to become unclear what to do.

• 30:04Speaker 01

  And so, I actually think the medical setting is a scenario in which this is already more obvious. So you're already in a setting in which you need experts to be able to evaluate answers. And one thing we're seeing with MedPom too, is we get closer to physician level performance on medical question answering is that. It's hard to tell the difference anymore. It's hard to tell the difference between different models. It's hard to tell the difference between models and physicians. And when you're at that point where it's uninformed oversight, then it becomes very tricky to think about aligning to human values. And so that problem is super well motivated in the setting. And that's something I'm very excited about.

• 30:39Speaker 02

  What do you think is a solution to that? Because if you look at the gaming analog, which is probably a bad analog here, right? Once machines were better than humans at things like Go or chess or other things, people started learning off of the things that the machines were doing that were unique or creative or different, or the problem solving was very different. And if we really want this technology to be incredibly valuable for medical applications, in some cases, we may end up with these suggestions that will really work well, but that, to your point, people may misinterpret or misunderstand. And so what, how do you think about evaluating things when the AI will be better than a person at medical adjudication or better than an expert?

• 31:19Speaker 01

  Yeah, this is, I mean, this is a really, really interesting question. I don't think I have all the answers, but I think there are approaches that people at Google and other organizations have been looking at. I think a couple of ideas here that I think are interesting and useful. One is the idea of kind of self-refinement or self-critique of these models. And so this is the idea that these models can take their own responses, give critiques, often guided with human feedback. And so that's where the place where human feedback comes in. But some of these techniques, there is no human feedback. And in that case, I'm not sure that's as valuable. Give critiques guided by human feedback and then use that to produce better answers. That's one line of approaches. I think a second line of approaches is around debate.

• 32:02Speaker 01

  And so the idea here is that it's easier for a human to judge a debate between two different answers than to judge the answer itself. And so the kind of standard for verification is a bit lower here. And so there's that ability for humans to be able to judge a response that potentially they wouldn't be able to judge otherwise via things like debate. And so that's that's another thing. I mean, another thing which is people are working on as well is thinking about how we can take AIs that are less capable and use that, use them to kind of supervise other AIs that are more capable. And so this is kind of the motivation. And this is partly the motivation of RLHF as well, even though it's about human feedback.

• 32:44Speaker 01

  It's about training a reward model that takes into account human feedback. And then at that point, it's AI feedback from then on. And then you use your RL algorithm and then you get rewards from your reward model. RL AIF or Constitutional AI, you know, kind of builds on that idea. But there's also limitations to that approach as well. I mean, I think if you ask, you know, researchers across all these organizations, have we solved this problem? Do we know what we're supposed to do? I think most of them would say no. It seems like a pretty consequential problem. So I'm excited for more folks to work on it.

• 33:14Speaker 02

  Yeah, one thing that I feel like would also sort of be generated as a side effect of all this is just you end up with these really interesting closed loop data sets over time that may be unique outside of an EMR or somewhere else, or a really robust medical record system, because if you have a faculty physician's assistant or something else, and then you have the endpoint of what happened based on treatment, you actually have a really interesting retrospective data mining training set.

• 33:39Speaker 01

  Yeah, I mean, I think that's like another opportunity for feedback for these models to show could have a huge impact on the world.

• 33:45Speaker 02

  Yeah, it'll actually be data driven medicine, which I think, you know, sometimes happens, but sometimes doesn't. So it's very exciting. I guess one more question is just, you know, there's there's amazing potential here. And if I look at the history of medical technology, you know, in the 1970s, there was something known as the Mycin project at Stanford where they built an expert system, which was an old computer program of its time or a computer program of its day that was sort of a precursor to some of the things that eventually happened in AI. They had a expert system that outperformed all of Stanford's medical staff on the prediction of the infectious disease that somebody had. So 40 years ago, almost, we had a machine that outperformed people in terms of diagnosis. But it never got adopted.

• 34:30Speaker 02

  And so often, when I look at medical technologies, there's this almost like anti adoption curve. In some cases, for the things that may be most impactful. How has the medical field embraced or not embraced these AI models? Is it different this time? Are people excited about it? Are they not excited? Does it really depend on the type of physician? I'm just sort of curious like what the reaction has been from the medical community today.

• 34:54Speaker 01

  Absolutely, that's a really great question. You know, I think when we started this, this brain moonshot, which we call it within, within Google, that was actually our motivation. It was really to think about the fact that these models had already, kind of already exist, and there was this opportunity to catalyze the medical AI community to really think about them carefully, think about the promise there, and to catalyze the AI community to think about how we can resolve any remaining limitations that would prevent real-world uptake. And so this was really our goal. And I think when we started this, there was much less conversation about the potential for large language models and foundation models for healthcare. And I think, I mean, partly because of, I think largely also because of other work that's gone on, you know, with GPT-4 and excitement around that.

• 35:39Speaker 01

  I think there's much, much more conversation about, you know, how these models can be used in the setting in a productive way. I think that's really, really exciting, and I think there's a lot of optimism, I see. But there's also a lot of justified concern about, you know, the potential limitations of these models and how we can, how we can get over them. Personally, I mean, from what I've seen from, from giving talks to different groups and chatting with different folks and different stakeholders, I think there's like a, you know, a wide, widely-held optimism about this technology and about the potential. But I think there's also kind of a little bit of fear that I think, you know, people have seen in other domains, like, I think programmers often feel a little bit of fear when they see GPT-4, for example. And I think it's not, not necessarily a fear that like, jobs will be replaced in the short term

• 36:23Speaker 01

  or things like that, but it's more of a fear of like, look how fast things are moving. This is, this is not like, think about just an improvement from MedPom 1, GPT-4, MedPom 2 in three months, like, it's, it's absolutely crazy. And I think we, you know, it's definitely an inflection point for AI, as you guys know. And I think it's definitely a good time to think about, you know, what are the most important, important problems we need to solve versus like getting caught up in the hype wave and, you know, forgetting to solve the most important problems as well.

• 36:52Sarah Guo?

  I think back to a lot sort of point earlier, thinking about the actual benefits of these technologies at scale, if adopted, even at human and at some, you know, defined superhuman level, should we come to some sort of agreement as a democratic society about what eval looks like, is really important in that if you if you just think about what the status quo is for somebody who has a complex case in a median background in America, what do they know about the error matrix of their doctor and what, you know, in a field that's also advancing in parallel to AI, like the specific rare condition that

• 37:37Sarah Guo?

  they have, it's it's not super encouraging, right? And so in terms of leverage for a field where the status quo is not sufficient, not not as a comment on, you know, the class of physicians and researchers, but just in terms of the the quality of care that we want to be able to offer every person, that seems it seems like we want to set a reasonable safety case, not a unlimited safety case, right? Which is, I think, is one of one of the things that has held back other sort of mission critical AI applications in the past. Maybe on that note, like, one last ask for you in terms of encouraging some optimism, you know, you're working on the state of the art in this field and thinking about

• 38:19Sarah Guo?

  the the barriers to the applied use, like, five years from now, like, how do you hope we are using large language models in the in the medical field?

• 38:31Speaker 01

  Yeah, I guess I think about this in two broad buckets. I think there there are two broad types of things that we can do for large language models in the medical field. I think the first is increasing the standard of care very broadly. And so that looks a lot like, you know, increasing access to health information, providing assistance to physicians, so the radiology example I gave earlier, potentially clinical decision support, like double checking a doctor's decision or quality assurance for a radiologist's report. So if, you know, a radiologist is dictating a report, they say no plural effusion scene, but then it's written down as plural effusion scene, then maybe an AI double checks that and just make sure that that's what was intended. I think augmenting telemedicine,

• 39:14Speaker 01

  I think, is is kind of a short term opportunity that I think in the next five years is very achievable. I think the other big bucket of things that is very much achievable is augmenting scientific workflows. And I think this could be a longer term thing than five years, but I think there's also short term things that we can do as well. So thinking about looking at correlations across modalities in existing data to find novel biomarkers for existing diseases that we know about, or kind of using large language models as research assistants. So I think there's already a lot of work on the idea of literature search and augmenting literature search with large language models. I think there's a lot of opportunity there. And that goes a little bit beyond what Metpom is likely going to do. But I think that's

• 39:56Speaker 01

  something that I think is going to be really promising with respect to the future of AI. Because I think the long term, when things go really well with AI, it's going to be because we've solved a lot of the most pressing scientific problems of today. And I think that's going to be because it augmented scientists, it helped scientists, it helped us figure out what are the things that we're missing. And I think there's a lot of potential there. So I'm also really excited about that in the long term.

• 40:21Sarah Guo?

  Awesome. Wrapping up, is there anything else you think we should touch on?

• 40:25Speaker 01

  Yeah, absolutely. I mean, I think for real world uptake of these models, there are a few large language model capabilities, in some cases that already exist, but we need to figure out the right way to do them. And I think a few of them are just multimodality, which is something that we were working on, we kind of previewed last week at I O. And grounding and authoritative sources, I think is important as well, thinking about how these models can use tool former like approaches to, for example, query authoritative medical information like a human would, but potentially better. I think that's also one way of getting around the risk averseness that you see in this area with respect to health information. If you're able to attribute information to an authoritative source, I think that has

• 41:07Speaker 01

  been something that has progressed this area in big companies before. And so where, for example, Google is doing that with health information is largely because it can attribute things to the Mayo Clinic and other organizations. And so I think that's going to be really important for moving this forward. I think also solid research, thinking about better ways to Improve the ways we are taking in human feedback. I think, you know, the jury's still out with respect to how to best, you know, collect human feedback even. I think people are still debating things like whether or not, you know, pairwise comparison versus rewrites are the best things to do. And, you know, that's a valuable thing to think about. I think another thing to think about is how to actually use that human feedback

• 41:50Speaker 01

  in the most valuable way, especially given all the scalable oversight concerns that you guys mentioned. I think that's, you know, a significant limitation of met bomb as it is today. And I think there's a lot of exciting things to do. And I think a lot of these questions are like foundational questions for AI more broadly, but, you know, become more acute and more relevant in this setting.

• 42:06Sarah Guo?

  It's been great to have you on No Priors. Thanks for doing this.

• 42:08Speaker 02

  Yeah, thanks so much for joining.

• 42:09Speaker 01

  Thanks, guys.