Universal Medical Intelligence: OpenAI's Plan to Elevate Human Health, with Karan Singhal

Karan Singhal, Head of Health AI at OpenAI, explains how ChatGPT Health is achieving attending-physician-level performance and already serving hundreds of millions of users. He details how OpenAI works with over 250 doctors, built the 49,000-criteria HealthBench evaluation, and ran one of the first randomized trials of AI copilots in clinical care. The conversation explores privacy and safety safeguards, medical multimodality, N-of-1 treatment plans, and how AI could become a standard part of global medical practice. Nathan uses Granola to uncover blind spots in conversations and AI research. Try it at ⁠granola.ai/tcr⁠ with code TCR — and if you’re already using it, test his blind spot recipe here: ⁠https://bit.ly/granolablindspot⁠ LINKS: modeling human wellness Sponsors: Claude: Claude is the AI collaborator that understands your entire workflow, from drafting and research to coding and complex problem-solving. Start tackling bigger problems with Claude and unlock Claude Pro’s full capabilities at https://claude.ai/tcr Serval: Serval uses AI-powered automations to cut IT help desk tickets by more than 50%, freeing your team from repetitive tasks like password resets and onboarding. Book your free pilot and guarantee 50% help desk automation by week 4 at https://serval.com/cognitive Framer: Framer is an enterprise-grade website builder that lets business teams design, launch, and optimize their.com with AI-powered wireframing, real-time collaboration, and built-in analytics. Start building for free and get 30% off a Framer Pro annual plan at https://framer.com/cognitive Tasklet: Tasklet is an AI agent that automates your work 24/7; just describe what you want in plain English and it gets the job done. Try it for free and use code COGREV for 50% off your first month at https://tasklet.ai CHAPTERS: (00:00) About the Episode (06:11) Cancer story and mission (11:46) Designing safe health AI (Part 1) (17:49) Sponsors: Claude | Serval (21:09) Designing safe health AI (Part 2) (26:48) Uncertainty, HealthBench and robustness (Part 1) (30:23) Sponsors: Framer | Tasklet (32:50) Uncertainty, HealthBench and robustness (Part 2) (38:11) Chain-of-thought and evaluation (46:49) Real-world performance and frontiers (55:35) Multimodal data and science (01:05:36) Personalization, privacy and monitoring (01:15:47) Models, data and incentives (01:29:31) Doctor adoption and workflows (01:38:13) Scalable oversight and alignment (01:51:06) Move 37 and future (02:00:50) Episode Outro (02:03:06) Outro PRODUCED BY: https://aipodcast.ing

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to the Cognitive Revolution. Today's episode is brought to you in part by Granola. To help new users experience the power of the Granola platform, Granola is featuring AI recipes from AI thought leaders, including several past guests of this show. There's a Replit recipe that converts discussion notes to an application build plan, a Ben Tassel recipe that creates content production plans, and a Dan Shipper recipe that looks across multiple sessions to identify cultural trends at your company. My own recipe is a Blind Spot Finder. It looks back at recent conversations and attempts to identify things that I might be missing. This has already proven useful in the context of contingency planning for my son's cancer treatment, and as I use it more and more,

• 0:44Nathan Labenz?

  it's getting better and better at suggesting AI topic areas that I've neglected and really ought to explore. See the link in our show notes to try my Blind Spot Finder recipe and experience how Granola makes your meeting notes awesome. Now, today, my guest is Karin Singhal, who leads Health AI at OpenAI and who was just named to the Time 100 Health List for his pioneering work. This episode began to come together last year on Thanksgiving when I emailed Karin, who I'd met a couple of times at AI events, to say thank you for all of his work on AI for Health and to let him know what a difference ChatGPT had made for me and my family in the context of my son's cancer diagnosis. As it turned out,

• 1:27Nathan Labenz?

  that was just as OpenAI was preparing to make a major product push with ChatGPT Health, which allows users to connect ChatGPT to data sources including electronic medical record systems and consumer wearables, plus a physician-facing ChatGPT for healthcare, both launching in early 2026. In this episode, we dig in to how Karin and team have achieved attending-physician-level performance with their latest models, their plan to ensure that this capability does benefit all of humanity, and their vision to raise not just the floor but also the ceiling of human health with continued research and even better models to come. Highlights of this conversation include: How OpenAI works with more than 250 human doctors to ensure accurate, robust,

• 2:11Nathan Labenz?

  and culturally appropriate responses; how they built Health Bench, which contains some 49,000 evaluation criteria to measure models' performance; and how models have already gone from a 0% score on Health Bench Hard by GPT-4O when the benchmark was first created, to today, already 40%. Plus, an overview of my experience using large language models to navigate a health emergency, including the critical importance of giving models as much context as possible on your situation, and how that's about to get dramatically easier as ChatGPT Health rolls out globally. We also discuss how 230 million people are already using ChatGPT for health questions on a weekly basis; the first randomized trial

• 2:55Nathan Labenz?

  of AI co-pilots for physicians, which OpenAI conducted with Kenya's Penda Health System, and which did show a statistically significant improvement in outcomes for patients whose doctors used AI; and why Karen believes, based on the reception that OpenAI is getting from health systems, that 2026 will be the year that using AI becomes a standard part of medical practice. From there, we go on to cover the steps that OpenAI is taking to ensure privacy and security of users' health information; how they're using worst-of-end measures to make sure models first do no harm, while at the same time striving to maximize value by training AIs to acknowledge their uncertainty as they offer their best guesses; how Karen understands the relationship between AI for health,

• 3:38Nathan Labenz?

  AI safety plans such as scalable oversight, and AI alignment more broadly; Karen's report that OpenAI's models' chain-of-thought reasoning has not drifted toward neural ease as much as some reports had previously caused me to believe; the future of medical multimodality, which will do a much better job of converting data to value, and which inspired me to buy a Whoop wristband to start collecting data on myself; and the compounding effect of parallel advances in AI for science; the growing potential for N of 1 treatment plans and medical move 37s; and the possible need for an update to the rules governing access to experimental medicines and information sharing.

• 4:16Nathan Labenz?

  Finally, Karin describes OpenAI's utopian plan to make ChatGPT Health available to all users globally for free with no ads. An early form of universal basic intelligence that I really think everyone ought to celebrate as a triumph of human ingenuity and goodwill. Zooming out, in the grand scheme of AI development, I think it is fair to say that we have far more questions than answers. And in my mind, all outcomes, from a post-scarcity utopia to literal human extinction, absolutely remain on the table. I signed to the recent call for a ban on superintelligence because I do worry that an AI arms race driven by recursive self-improvement loops could easily get out of control. And yet, at the same time, capabilities

• 4:59Nathan Labenz?

  like this, which have been so valuable for me and my family, and which will undoubtedly save millions of lives in the coming years, are for me both an incredibly inspiring accomplishment and a practically irrefutable argument for the upside of AI. The question at this point is not whether we will create powerful AI systems, but exactly what form they will take and under what circumstances and incentives they'll be developed and deployed.

• 5:25Nathan Labenz?

  Karin's work demonstrates that for the moment at least, we can have it all. AI systems meticulously crafted to minimize downside risk, which are both capable and efficient enough to meaningfully improve the human condition globally. There is a ton of work left to be done, both inside and outside of the frontier companies, to make sure that these lofty standards don't slip in the face of intensifying competition. But today, if you or a loved one are facing a complex health challenge, you owe it to yourself to take full advantage of the incredible medical expertise that Karin and others have managed to build into systems like ChatGPT Health. With that, I hope you enjoy this inspiring look at the frontier of medical AI with OpenAI's Head of Health, Karin Singhal.

• 6:11Nathan Labenz?

  Karin Singhal, Head of Health AI at OpenAI, welcome to the cognitive revolution.

• 6:17Speaker 01

  Thanks for having me.

• 6:18Nathan Labenz?

  I'm super excited about this. You know, it's

• 6:22Nathan Labenz?

  Rare that I have had so much impact from a guest's work on my life as I've had from the impact that your work in health at OpenAI and at Google previously before that has made on the last three months for me. So, regular listeners know the story. My son got cancer. I've been an intensive user of all the frontier language models over the last three months to advise us as we've gone through this process. And, boy, have they been a game changer. Thank you for all your hard work and for making the last three months for me, the mental health side of the equation for me dramatically better than it otherwise would have been. And also moving the needle a bit on my son's treatment

• 7:03Nathan Labenz?

  and our confidence that we were actually doing the right thing for him. It has been invaluable and the consumer surplus has been off the charts.

• 7:13Speaker 01

  Amazing. Thanks for the kind intro and thank you for sharing the story. I think it resonated with a lot of people. So, thank you for that.

• 7:19Nathan Labenz?

  A big takeaway, I think, of this conversation will be if you find yourself in a medical emergency or even just want to do a better job of managing your health in general, the frontier models today are getting really good at that. I always go to the example of the AI doctor. There's obviously a relative scarcity of medical expertise even in a wealthy country for a privileged person like myself in the United States. You broaden your worldview and look around the world and like the shortages extreme. And I've always kind of felt like, boy, this would be just an absolutely killer use case that everyone could agree on. When I started talking about it a couple of years ago, it felt like it was getting close but

• 8:00Nathan Labenz?

  still a ways off. And I've done a bunch of episodes over time with Vivek and some of your former teammates at Google DeepMind who work on similar topics. And they're always very appropriately cautious or have been over time about, yeah, it's not quite there yet. You know, we're not quite ready to roll it out to production, but encouraged by the progress, that sort of thing. But, boy, again, it is really getting there. But the first question I wanted to ask is just like, how did you get into this? And what were you expecting? Like, how big of a dream were you daring to dream when you first got into AI for healthcare some years ago now?

• 8:34Speaker 01

  Yeah, I started working on AI for healthcare about four years ago now and kind of transitioned to doing it full time around that time. I was thinking a lot about a few kind of fundamental research problems at the time when I was at Google at that time. A few problems were like around foundational work in representation, learning, privacy preserving, learning. There's a lot of interesting applications in healthcare. But I think the background for this was a kind of conviction that I had since undergrad, honestly, that AGI would be a pretty big deal and that it probably would happen within our lifetimes. And I thought there was probably two things that I could do to make that better. One is to work on safety and the other is to work on benefits. And I saw healthcare as kind of the most obvious area for benefit, like you.

• 9:14Speaker 01

  And like you said, we've been on this kind of amazing exponential over time, both with model capabilities. And I think more recently with people's adoption and kind of the overton window shift around trust around these models. And so we're seeing just a bunch of people start to use these models across, across individual patients, individual clinicians, researchers. And so seeing a lot of the benefits become a lot more tangible. And all this comes down to, for us at OpenAI, it comes down to thinking about what does it mean to make our mission real? Our mission is ensure AGI is beneficial for all of humanity. And this kind of three parts to that, one is build and deploy AGI.

• 9:53Speaker 01

  The second is prevent downside risks, whether that's kind of short term kinds of risks or kind of long term kinds of frontier risks. Or, and, and finally, thinking about how we can make benefits happen. And like you say, I think health is one of the most obvious and tangible ways those benefits can happen. I think when I, when I started working on health back in, I guess, end of 2022 was kind of when it was, was full time for me. So like right before ChatGPT. And for me, it was a lot of thinking around, there's this kind of capability overhang between LLMs where they were at. You're, you're scaling these models up, you're seeing that you can instruction tune these models and they can do amazing things. And how they're being adopted and thought about in the clinical AI and healthcare worlds.

• 10:36Speaker 01

  And so my, my ambition at that time was really, was really two things. One was get healthcare and clinical AI world to think about LLMs as a thing that could work. And again, this is prior to ChatGPT. And the second was think about the work that we would need to do in safety and reliability to make the models trustworthy for the setting. And then over time, I think those ambitions became less and less ambitious and my ambitions at OpenAI became larger and. For our work at OpenAI, we kind of started out with three goals. The first is to kind of make access to medical expertise more universal. The second was thinking about the ways in which we can think about this setting as a way to go around our work in safety and alignment. I could talk more about kind of the safety motivation there.

• 11:18Speaker 01

  And the third thing was thinking about how we can kind of bring society along with the high stakes technology and work with partnerships, roll out products, work with policymakers to think about the right ways to kind of iteratively deploy in a setting like this. I think over time, things like this sounded really ambitious about two years ago when we started to set out to work on health at OpenAI. Now, actually, we're kind of feeling like these aren't ambitious enough. So we're really excited about what's to come.

• 11:46Nathan Labenz?

  I think you did a great job there of kind of laying out a taxonomy or a sort of scaffold for this conversation. Let's maybe talk about the capabilities first, but they are sort of inseparable. I'm wondering, kind of, from a capabilities perspective, is there also a Hippocratic oath kind of mindset that you bring to the table that is sort of focused on making sure that the AI performs well in medicine that maybe is still distinct from sort of the bigger picture safety agenda that motivates you? How do you guys think about like the way that the model should perform in terms of, you know, doing no harm? Because I think GPT-4, while it did add value, also could definitely do some harm in terms

• 12:30Nathan Labenz?

  of giving you wrong ideas. And I do think we've come a long way since then, but I wonder how you think about that?

• 12:37Speaker 01

  100%. The way we think about kind of the health work that we've been doing at OpenAI, we've kind of been operating in three phases. One is laying the foundation for the work, and a lot of that has been really around the kind of safety research and work on making the models not just better reasoners, but also perform better and have better bedside manner and things like making sure that they're able to convey uncertainty well, things that they're like able to kind of escalate to a doctor when needed. And so I can talk a little bit about that kind of work in a second. We're recently now in this phase of adoption. So as the foundations have solidified, just a bunch of people have been using it. It's been one of our fastest growing use cases. And we shared recently that over 230 million people a week are using it for Veritas

• 13:21Speaker 01

  health and wellness related queries. And this year, we're really focused on kind of scaling the impact of the work. And so all that I think comes from the foundation. And so your question of how do we think about imbuing the models with the right kind of prior of how to behave and how to, how to ensure that there's minimal harm provided? I think we were very thoughtful about this as we were laying the foundation for this work and starting to work on health at OpenAI. And a lot of this comes down to our really close partnership with this cohort of 260 physicians or so that we've been working with since about two years ago. And what we've been doing with them is basically working on one, one way of kind of imbuing models with a certain kind of behavior is to write a spec kind of from first principles. Maybe you or I just sit down and write, write down a spec and say like, this is how

• 14:06Speaker 01

  models should behave. They should, you know, say X, Y, and Z when in this situation and this in another situation. There's pros and cons to this. You know, one pro to this is that like, it's, it's very easy to explain and understand what's going on here and it's easier to be transparent about model behavior. A con of this though, is that it's first, it's very hard for you or I to say what should happen or what should not happen. And it's difficult to say anything beyond do no harm because well, that's, that's an excellent thing to, to avoid. It's, it doesn't tell you what to do in, in most scenarios. And so what you want to do is figure out a way that you can move from really large scale principles that, that matter to what do you do in very specific scenarios? And how do you make sure that's guided by the expertise of not just one or two people,

• 14:51Speaker 01

  but actually hundreds of experts? And so that's kind of the approach that we've taken and you can see it in our approach to evaluation. So one, one way to think about how do we encode model behavior in models is like, what are the evals that we care most about? Eval's are really a lifeblood for any researcher. And so we really recently put out back in May of, of 2025, this work on Health Bench, which is this kind of Evaluation of how large language models perform in realistic health conversations between users and models where users could be either lay users or health professionals. And we kind of went about it in a way that leaned on the expertise of these 250 plus physicians rather than writing down specs from first principles.

• 15:33Speaker 01

  And I can explain that more, but I'm sure you have many more questions as well.

• 15:38Nathan Labenz?

  Yeah, I mean, go on as long as you'd like. I guess to double click on the physicians for a second. Are they like, what's the nature of the relationship with them? I could imagine anything from, you know, they sit there doing side by side, you know, RL HF style, prefer this to that. Or you can imagine some of them may be much more deeply integrated with the research team.

• 16:04Speaker 01

  Yeah, we have kind of like three different layers that we think about physician expertise. And we were very thoughtful about how to bring in physician expertise into our team in a way that like balanced out and combined well with the research expertise that we have in our team. So we kind of have three layers. The first layer is kind of like high level advisors who are actually like more informal or who are kind of help us with strategy in various ways. We kind of share roadmap with and things like this. The second layer is folks who we work with in a kind of a, you can think about it as kind of like a human data operation in the way that you're describing, but a little bit more closely. And, you know, we're slacking with them all the time and we work with them closely. And they're not just kind of going off and doing tasks. And we ask them for advice and questions all the time. So we basically have a Slack community where we're interacting with these folks.

• 16:49Speaker 01

  And some part of what they're doing is kind of like comparing model outputs or red teaming model outputs or things like this. Some part of it is looking for ways in which we have, we might have blind spots today and listing those kinds of ways so that we can prioritize them in the future. Testing new products, things like this. So as an example, the work on chat GPT for healthcare, which we, which we announced on January 8th. We had, we had red teamers test this product over nine waves for six months. And this was in close collaboration with this physician community. And the final, the final kind of top of the pyramid of how we kind of rely on physician expertise is as like really close advisors who work most closely with our team. And so they're actually the ones who are kind of working on channeling the voice

• 17:32Speaker 01

  and combining the voice of these hundreds of physicians interfacing most closely with our research team, translating that into evals and, and model training data and things like this so that we can get kind of then improve our models.

• 17:44Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 17:50Nathan Labenz?

  We talk about channeling the voice of the physician and calling back to the Hippocratic Oath. One thing that I do find kind of frustrating, honestly, about my experience in the medical system recently is I think there's a little bit too much emphasis on do no harm. And this also, you know, connects in a pretty deep way to questions about like, how should we even conceptualize how to talk to our AIs about what they should do, right? I mean, we have kind of the very detailed, you know, Talmudic detail rule set as kind of one extreme possibility where, you know, for every corner case, we try to map out what you should and

• 18:32Nathan Labenz?

  shouldn't do. And hopefully it learns that and, and lives by it. And then on the other, matter of it's the other extreme end, but there's, you know, the kind of anthropic constitution, the Kald constitution recently has at least demonstrated that you can get pretty far as well with something that is less rule based and more about like trying to teach the model to have good character and use good judgment as it goes through all the situations that it finds itself in. I would say my critique of the human doctors that I've engaged with who have generally served us really well is that they definitely want to do no harm and to the point where they're sometimes too reluctant to engage in a hypothetical

• 19:15Nathan Labenz?

  or to act on something. I do have one like good friend who's a doctor who also happens to have had the same or very similar kind of cancer to my son. And he's, he's like notably behaved differently with me in private one-on-one. You know, he's like, look, I don't need a randomized control trial to tell you that that makes sense. But you don't get that too much when you're actually at the hospital. There's, you know, there's a reluctance among physicians. Certainly I've experienced, I think it's pretty commonly shared perception that there's a reluctance to act on things that make sense because that isn't guaranteed to work out. Obviously biology is super messy and there's a ton of diversity and there's just a lot that we don't know. So how do you think about that kind of challenge? And

• 19:58Nathan Labenz?

  so I kind of wonder where you guys want to land in terms of, you know, only adhering to the most rigorously Defensible advice versus, you know, doing a little bit more of the Amanda Askel thing and being like, well, it does take, you know, you've got to be willing to take some risk to help people sometimes. So maybe we should have the models do that.

• 20:22Speaker 01

  Yeah, it's a great question. You're pointing out problems, I think, on two sides of the healthcare ecosystem. One is, as a patient, you have this challenge of needing to advocate potentially pretty hard for yourself. And I think in your son's story, you pointed out a couple false starts that you had where you saw a doctor, they said it was probably normal, you had an abnormal blood test reading a couple of times, and then they were just kind of like, yeah, it's probably okay. And so this is kind of experience that patients often have when they're having something that they feel or know to be an issue of needing to advocate for themselves and doctors often, and feeling like their doctor isn't hearing them. And on the kind of clinician facing side, you have this challenge of medical evidence is increasing rapidly. It's very hard to keep up with the latest of what's

• 21:06Speaker 01

  going on. You're overloaded with documentation and there's a bunch of burden there. Doctors are human too, right? So there's a bunch of challenges on both sides. And so you have this amazing thing, which is AI that is able to do a few things, right? You have AI that's able to talk to patients, understand the concerns that they're able to have, integrate knowledge and information across both their previous history, the latest medical evidence. One of the things that these models are obviously very incredible at these days is taking in a huge amount of health context, not just on you, but also on the latest medical evidence and things like this, integrate that all together into one context and do something that I think is very difficult for a human to do. And then on the physician facing side, you have, again, that same capability to integrate information and face the physician.

• 21:50Speaker 01

  And so that's a lot of why we're doing this work is because we see this kind of, this kind of gap between where the models are at and how people are using them. And we think that that's really important with our upcoming products. And to get closer to answering your question, you mentioned like, where do we see, how should the models navigate places where there's a You know, potentially a lack of medical consensus or potentially physicians would disagree about what to do. This is pretty fundamental to our approach and this is why we don't have, you know, like one or two or three experts who are determining what the models outputs are. We have a pretty multi-pronged approach. A lot of this comes down to presenting information to the user, but being sure to present uncertainty when uncertainty exists. So I mentioned the safety research motivation for a lot of the work that we're doing. One of the directions

• 22:35Speaker 01

  that we've been exploring is, is are these models well calibrated in their uncertainty? And can we get better at having these models verbalize their uncertainty? So for example, if there's like three to five potential paths for your son's next treatment and a doctor might, just for the sake of simplicity and for clear communication, just communicate one or focus on one, a model can potentially communicate three to five of them, but mention that like the state of the evidence may be somewhat limited and mention caveats. And so one of the things that we've been investing in is first, can models become better at understanding their own uncertainty? And that can that be a thing that we measure and improve? And second, can they verbalize it better? And this is, I think, a big part of what

• 23:16Speaker 01

  is the right way to kind of thread the balance between being more aggressive and sharing potentially early, early results or early evidence and being overly conservative. So that's, that's kind of how we think about that.

• 23:28Nathan Labenz?

  What are you seeing in terms of trying to get the models to understand their own uncertainty? Cause I remember that famous graph from the GPT-4 model card where the sort of pre-trained model seemed to be much more calibrated with respect to its own uncertainty than the train model. And, you know, we've only scaled post-training since then, right? I haven't seen like an update to that kind of research in a minute, but it seems like there was at least a fundamental challenge kind of opening up there with respect to that kind of, you know, introspective self-awareness of, you know, how confident am I in this answer? So is that something you guys have solved? And, you know, is that why I haven't seen that sort of graph in a while?

• 24:14Speaker 01

  Well, I think there's a measurement challenge. So the plot that you're referring to and the kind of plot that existed back then was kind of like, given, given the next token, for example, a multiple choice question, was the model's probability of that next token for like A for a multiple choice question, did that correspond to how likely it was to actually be correct in choosing A? And what you're seeing now is I think it's become more difficult to measure that because of two reasons. One is like we have higher expectations of our models than answering multiple choice questions. It's just hard to say when they're correct or not correct. And the second is, this is a little bit more technical. The models now emit reasoning tokens or thinking tokens between, between initially opening something and their final answer.

• 25:00Speaker 01

  So the result is that you can't ask the model for what is the log probability of that next token being A exactly in the same way that you would before. And so there's a couple of things that you can do to handle this. One is you can go in the direction of richer ways of measuring whether a model is correct or is doing the thing that you want than, than measuring log probability of a certain letter. The second thing you can do is repeatedly sample from a model and then see whether performance stays the same or degrades as you repeatedly sample from the model. So actually, I work on Health Bench is a, is a good example of doing both these things at the same time. So in Health Bench, we, we did this work around basically measuring not just like one or two or three

• 25:41Speaker 01

  different aspects of model performance in health, but actually across these 250 plus physicians and across 5,000 conversations, measuring about 49,000 different axes on which model performance could differ. And part of this is like whether or not the model expresses uncertainty the right way, whether this is the right fact, whether it escalates to a physician when needed or things like this. Again, 49,000 different things that are measured in Health Bench. And so one of the things that you can do there is then measure correctness in a, in a way that is less about multiple choice accuracy and more about are the right facts included that are really important to emphasize to the user and things like this. The second thing you get out of that is this metric, which we call worst at N, which is when you repeatedly sample. From the model and try to measure performance on Health Bench.

• 26:26Speaker 01

  What is the worst performance you get on N samples? So you sample from the model 20 times. What is the worst performance you get? So now you have a way of measuring instead of kind of the log product based way. You have a way of measuring even with a reasoning model, whether or not it produces a consistent result, conditional on the different kinds of thinking that it's doing. And so what I would say is basically like it's harder to produce plots like we could before, because now the thing that we're measuring is so much more complicated. But when we do produce the plots as we did in Health Bench, the plots are also looking pretty promising and models have improved quite a bit at that. Hey, we'll continue our interview in a moment after a word from our sponsors.

• 27:04Nathan Labenz?

  So how should you as a user think about the worst of N thing? If I is there a way to sort of translate? Maybe you could just describe the result, like how much worse is the worst of N than say the next worst of N or the, you know, the average of N? And then if I'm a user, which I am, do I? Can I get security just by like running it twice? You know, or is there, is there a practical upshot of that work? That could give me confidence that I'm not getting the, you know, if I do X, then I can like be sure I'm not getting something that's like way worse than the model. You know, is typically going to output.

• 27:45Speaker 01

  Yeah, the way I think about it is I think the more compute you spend on things, you will get better results and the results may be marginal over time. So one thing you could do as a user of a model is sample from a model 10 times, combine that together into one output and have an L M synthesize that, kind of the outputs of, you know, an L M council and then produce that as an answer. I think that will be marginally better than the answer that you get from just running one model. And this is not so dissimilar from what G P T five pro and stuff like this does under the hood anyway. And so. I think you can do a few things if you're a user and you want to kind of make the best of this. One is you can use GPT-5 Pro or something similar to do a thing like this.

• 28:27Speaker 01

  A second thing you can do is you can actually increase the amount of reasoning because I think it has a very similar effect to just running it multiple times. I think in both cases, my my current sense is we're getting to the point where current models as they are now are performing incredibly well for most people in most of the times. But I know, for example, you using GPT-5 Pro was an important part of of working through your son's situation. I think we're reaching a point where, except for the most complicated of cases, most people are best served by just using the model on a default reasoning setting. I would recommend using the reasoning models instead of the the kind of more instant models, using the the GPT-5 Point 2 Thinking rather than GPT-5 Point 2 Instant for a lot of health-related things.

• 29:12Speaker 01

  But I think most people can get the best of both worlds between latency and performance by by doing that. And I think the way to think about really broadly the worst of end result is just kind of like, if you sample from something 20 or 50 times, you'll have a varying varying performance across across model outputs. What we saw in the worst of end results is that recent models have improved pretty significantly, where the worst performance of, for example, 03 at that time was way, way better than the best performance of GPT-40. And we've continued to see that over time. So we've been kind of shipping model improvements in health pretty rapidly over time. And the model improvements in the last year have been more than in previous years since ChatGPT launched.

• 29:52Speaker 01

  And so as an example, the today, the nano models, the GPT-5 nano models that you can get through the API and also our open source models are actually performing similarly to 03, which was our best and greatest model not so long ago. And the latest reasoning models continue to kind of push the frontier of how much you can do with less and less reasoning. So if you try, this is true for 5.3 Codex as well, also for 5.2 Thinking. If you try using them by default on health queries, it'll actually think a little bit less but produce better results. And we're continuing to try to push that frontier of of not just needing to pour more compute into it to get a good result, but also getting better performance at a given level of compute. And the result is that like the models are way, way better than they were even a year ago.

• 30:39Nathan Labenz?

  Yeah, it's been crazy in my just three months of intensive use. Nothing has brought home the pace of shipping quite as much as how many updates there have been just in this, you know, one kind of chapter of my life. It has been wild to see.

• 30:55Nathan Labenz?

  I don't know if this is something. How should we think about like density of effectiveness of the reasoning tokens? I mean, there was this thing, I did an episode with the folks at Apollo. I'm sure you, you know, know of their work, if not know them personally. And one of the really interesting things that they observed when they got access to the chain of thought is that at least for, I think it was oh, three at the time, although I'm not 100% sure which model it was off the top of my head, there was this sort of seemingly development of like a new dialect, basically internal to the model. You know, the famous sort of watchers, watchers, whatever.

• 31:38Nathan Labenz?

  Can you share anything about sort of how you're balancing the obvious good of efficiency and, you know, denser thinking per token, you know, more value per token created with the seeming tendency to have the internal chain of thought go off in weird and potentially, you know, hard to parse directions? I guess there's also the commitment from OpenAI to like not train on the chain of thought, or at least not, you know, apply certain kinds of pressure to it. I don't know if that's like an absolute ban on any feedback on the chain of thought, but interested to hear your, your thoughts on that because that's something that I've been like, well, the world moves, moves past these big stories these days so quickly. That one seems to have kind of come and gone, and I'm not really sure what the state of it is now.

• 32:25Speaker 01

  Yeah, it's a super interesting question. Also near and dear to my heart as a safety researcher. So chain of thought interpretability has been one of the, one of the nice advances of safety in the last year or two, which has been really cool to see. So as models have become reasoning or thinking models, they've also emitted tokens that effectively explain their work and what they're thinking. And this provides a form of interpretability for researchers who want to understand whether models are doing the thing that they expect as safety researchers do. And so it's been this really cool way of measuring whether models are doing things like scheming or other kinds of, or producing outputs that are not desirable in various other ways. And this has been relevant beyond health

• 33:08Speaker 01

  to a bunch of other domains as well. I think a lot of the results that people have, have shared and, and studied have been actually in coding. I think the, I've been pleasantly surprised. So, so the danger that you're pointing out is as you put more pressure in reinforcement learning to get the models to produce a good output, will they kind of slip away from the prior of having their thinking tokens be simple English that is easy for researchers to understand? I think what we've seen is actually pleasantly surprising, which is that at least until now, we haven't seen a lot of large scale evidence of this slip into what's called neural ease of kind of using, using chain of thoughts tokens in a way that is not explainable and understandable in English. And in general, as we've been

• 33:53Speaker 01

  trying to understand the monitorability of our models over time, we haven't really seen that effect as we've, as we've scaled up RL. I'm not sure if that'll continue to be the case in the future. But so far, I've been fairly pleasantly surprised that this kind of side effect of the reasoning paradigm has continued to be useful and hasn't really been, hasn't robustly been seen to be, become unreliable. I think the result that you're pointing out, I think has been like sort of continuous over time with this kind of weird blips and interpretability of the models at times, but we haven't seen like a continuous increase in that. And we haven't seen evidence clearly, even though we've actually tried to study it, of scaling RL causes that to happen more. We haven't seen that yet. I would expect in the limit it does.

• 34:35Nathan Labenz?

  Okay. That's quite interesting. So is there no, um, the way I sort of interpreted that, first of all, I thought it was happening more than it sounds like you're saying it is happening. And I naively assumed that there was some sort of like brevity reward signal that was being applied in addition to, you know, ultimate correctness. And certainly it, you know, is intuitively comprehensible why something like that would start to happen if you did that. Is, should I infer from what you're saying now that there isn't really a brevity signal? And that this is more of just an emergent weird phenomenon that doesn't happen that often? And so it's kind of one of those weird language model things that we keep an eye on, but like, don't obsess about too much because it is rare?

• 35:20Nathan Labenz?

  Is that a fair summary of the state of play?

• 35:23Speaker 01

  I do think it's important to pay attention to. I think the right way to think about it is like, there's nothing reinforcing that this should happen during training, right? Like, there's no reason that it should happen, that models produce chain of thoughts that are human interpretable during training. The reason it happens is because they have a prior of using the English language. And when you give them the space with thinking tokens to produce a more correct and helpful answer to the user, they're actually using it in English because that is just the easiest thing for them to do. I think this is like a, basically an empirical phenomenon that is extremely useful for safety research, and I would love to keep it that way as much as possible. And I'd love to see more research in how we can maintain that. And I think OpenAI's commitment to avoid optimization pressure as much as possible

• 36:08Speaker 01

  and also the commitments of other labs, I think is really exciting progress. I do think it's important to watch out for, so I think it's a great question.

• 36:17Nathan Labenz?

  Yeah, to be continued. So you mentioned like, obviously, the incredible complexity of the evaluations that you're running with, by simple math, right? If there's 5,000 conversations and almost 50,000 criteria of evaluation, there's a lot, I don't know, I imagine some of those criteria are reused across conversations. At a minimum, we've got something like 10 evaluation criteria per conversation and probably a lot more criteria per conversation than that. The

• 36:49Nathan Labenz?

  It's hard to summarize. How good are they? Nevertheless, I'm going to ask you, how good are they? And how should we think about that? There's also like this Health Bench Hard thing that maybe could be a way of saying like, they're good at most things, but here's some things that they still struggle on. Kind of defining the frontier maybe is one way to say how good they are. But how do you communicate to the world how good the latest models are when it comes to health?

• 37:12Speaker 01

  Yeah, I think it's good to keep in context like the arc of work around LLM evaluation and health. A couple of years ago, people were mainly focused on evaluating LLMs by thinking about their performance on multiple choice questions like medical exams and things like this. And then with the work that collaborators and I did at Google, we started kind of increasingly investing in what does it look like for specialized or unspecialized LLMs to answer general health questions with some of the MedPom work and potentially go in the direction of asking follow-up questions to a user with the Amy work. And so increasingly over time, we've moved towards higher and higher fidelity evaluations. And I think Health Bench is kind of the latest big step in that direction of wide coverage

• 37:53Speaker 01

  of LLM performance and safety, but actually covering it in a way that covers many different axes of performance that actually matter for the real world. And so, again, you see these 5,000 conversations, these 49,000 different axes of performance. There's actually three different versions of Health Bench. So one version of Health Bench is that full data set. The second version is Health Bench Consensus and the third version is Health Bench Hard. And these kind of mirror what we view as kind of like the three kind of high level principles when designing an eval for health. The first is that you want it to be meaningful, which means that if the number goes up, hopefully human health will improve. The second is that you want it to be trustworthy, which means that it's backed by the consensus of doctors, for example, or other experts.

• 38:38Speaker 01

  The third is that you want it to be challenging, so you don't want it to be 100%. So one thing that's happened over the years is like all the previous benchmarks that meant anything at all have gone to like 95, 100% over time. And Health Bench actually remains unsaturated to this day. And so what you have with Health Bench, Health Bench Consensus, Health Bench Hard is actually like a little bit of focusing on these individual axes. So Health Bench overall is a number that we think if it goes up for a model and people are using that model, human health will improve. And we feel like that is a statement that we can defend with the rigor behind the work. The second is Health Bench Consensus, where we specifically focus on a bunch of criteria for evaluation where Each example had a majority of multiple physicians agree that it was applicable.

• 39:21Speaker 01

  So not only did you have physicians write these criteria, you also had a bunch of physicians check whether these criteria hold for a given conversation and whether they're the right thing to be evaluating about a given model and a given conversation. And the final thing is Health Bench Hard, where we actually took somewhat adversarily against a bunch of different models across all the different model providers, we chose the examples that existing models fare the worst at, but still seem high quality. And then we turned that into a benchmark. And so Health Bench Hard has kind of been my favorite external benchmark for whether an open model is doing really well. And when it came out, GPT-4O was literally zero on this benchmark. And so it's an incredibly hard benchmark because this is just how we chose the examples.

• 40:03Speaker 01

  And over time, we've kind of improved to a performance of around 40%, which is still not near saturation of this benchmark. And I think this benchmark has a lot to go for OpenAI's models. I think competitor models are more in the kind of 20 range for current models. So that's the way I think about the Health Bench family of evals. Like, we have this kind of commitment to work on evals that are meaningful, trustworthy, and challenging. And if you want to focus more on the evals that are super trustworthy, then we have these subset consensus, which is really focused on that. We also did, in addition, as part of the Health Bench work, we did a couple initial analyses. And one of them was Health Bench involves grading these kind of individual 49,000 different rubric items using a model-based grader.

• 40:46Speaker 01

  We had physicians compare the model-based grader to the grading of other physicians. And we actually found that the model-based grader was doing a better job than the average physician. And so what that tells you is the grading for Health Bench is pretty high quality compared to what you'd expect from a physician.

• 41:01Nathan Labenz?

  Recursive self-improvement, signs of recursive self-improvement. Alert, alert. How about just like a little bit of an intuitive sense for like what's in Health Bench Hard? I mean, that could give you Just kind of my experienced sense of the frontier from the bedside over the last couple of months. And I would summarize it pretty simply. This is what I say to like my neighbors and stuff when I'm saying, hey, by the way, you know, you should really use a language model. Next time you're facing a health challenge. I basically say, look, I was in the hospital for initially like 30 days of really intense treatment where everything felt super high stakes. We didn't always know what was going on. We didn't also know how much we could

• 41:43Nathan Labenz?

  even trust our doctors at that point. You know, we were everything was so new and stressful at the same time. And what I found was basically that the frontier models were step for step with the attending oncologist on almost everything. And they're have been and by the way, that means they're like a lot better than the residents, which is much more knowledgeable. They're at the attending level for sure. The there were maybe a half a dozen times over the course of that month where there was some disagreement between the my initially, I was just using GPT five pro, whatever exact version of GPT five was. And then as these

• 42:27Nathan Labenz?

  other, you know, frontier ones came out, I started to do everything in duplicate with German I three and then duplicate with the latest Claude. Interestingly, I would say that the AIs disagree with each other even less than the models disagree with the attending. But it's it's quite limited disagreement between the models and the attending. And typically when there is disagreement, I've found it to be a very minor thing. Okay, his electrolytes have gone a little low. Should we give him electrolytes today or not? And of those like half a dozen things, there's not really a major trend. I would probably score it like six to four for the doctors with the benefit of hindsight in terms of.

• 43:10Nathan Labenz?

  We usually follow what the doctors said and, you know, did they, did we in the end feel like they were right or do we kind of wish we had like gone with the AIs? I'd say maybe two out of three times, we have felt like they were probably right. And I think the one, if I tried to chalk it up to something that they have, that gives them an advantage, it almost always was one of those situations where whatever I'm putting into chat and whatever, you know, data, because I'm always, of course, and now we've got integrations for this, but you know, I was always kind of exporting the latest results from the EMR and so on and dropping in the PDFs that they gave me. There was, the difference was usually come down to, in view of all that, Eamon would

• 43:53Nathan Labenz?

  additionally say, taking all that into account, but also just looking at him right now, kind of watching how he's breathing, looking at his color, I'm pretty sure he's fine. It was that kind of very intuitive, very multimodal, very subtle sense that obviously these, you know, folks have developed over quite a few years of clinical practice, that on these very fine margins seem to give them a slight edge over the models. So that's kind of my account, but hopefully, I mean, I guess what that means maybe is like my situation isn't that hard. And I think that is actually true in the sense that the cancer that my son has is not

• 44:34Nathan Labenz?

  a super rare one and the treatment protocol for it is quite well established. So it's not like a super hard call in terms of the main line, like what to do. You know, maybe even though it's a hard thing for him to go through, maybe it's not that hard in terms of like clinical judgment. I wouldn't say there's, you know, I haven't experienced anything that I would say the models were only 40% on. So with that, maybe you can tell us a little bit of like, well, what's out there still on the frontier where we do have ground truth, or at least, you know, some sort of consensus that we feel like is solid enough to grade models on where they're still only at 40%?

• 45:11Speaker 01

  Yeah, it's a great question. I think we can like think about, let me describe a little bit more about what Health Bench is evaluating and the ways in which models have improved over time. They can talk a little bit about kind of next frontiers for model improvement. So Health Bench measures many, many different things. It has, you know, these themes which are focuses of different evaluation examples. And a few examples of focuses that we had, one was kind of like, are models appropriately escalating to care when needed versus not escalating to care unnecessarily? You want to balance this because you don't want to be, for example, overwhelming the health system with a bunch of patients who are worried and by alarmist kind of medical advice versus not escalating to care when needed.

• 45:52Speaker 01

  Another kind of aspect that we measured was thinking about the ways in which models can adjust to the different demographics or different epidemiological conditions or different levels of access to care of people globally. So this is, you know, both like making sure that you adjust a user that's male or female, but also, you know, if somebody asks a question in a region where tuberculosis is more common versus less common, making sure you adjust for that and things like this. And so that specifically was actually, we call that global health, that specifically was the biggest single focus of Health Bench. Because that's like, I think, one of the biggest ways that are can be most impactful.

• 46:28Speaker 01

  And then a bunch of other things. We talked a little bit about calibration. We talked about, one of the ways that the models have gotten significantly better over time is not when they're, when they know they're uncertain, not only flagging that uncertainty, but actually like browsing to get more information. For example, getting the latest resources and being able to synthesize that information together. Another thing is asking follow up questions and the right kinds of follow up questions over time. Initially, HPT would almost never ask follow up questions in health settings. Now they do much more often and they're much more likely to prioritize the right follow up questions for you.

• 47:00Speaker 01

  You know, you had this, you had this story around, around you when you were using the model for your son, where you actually like learned over time to ask the model to interview you and like figure out, you know, whether you could do the physical exam yourself over time and things like this. The models have gotten better at knowing when that would be useful and flagging that as well. So it's, it's everything from kind of pure. Your reasoning and solving benchmarks and medical calculations and figuring out the diagnosis to all the way to how does a model behave, what is the bedside manner, is it comforting and things like this? And again, a balance of like both difficult and high trustworthiness signal that we're getting in working with our experts. And over time, we've kind of continued to, as the models have improved from, you know, oh, one to oh, three.

• 47:45Speaker 01

  GPT four point one, working GPT five and so on. All these models have improved significantly in health. And today, every major stage of model training for every model that we ship at OpenAI actually now benefits from our work in health. And that's going to continue for future models as well. I think the frontiers are in a few different areas now. I think one is, I think a lot of text based performance is actually like outside of like sub specialty areas is actually pretty good. So if you're, if your goal is to kind of keep up with the latest evidence, even as a physician, I think the models are doing an incredibly good job today. And people are finding it a lot of value. And I think that's one of the greatest data points towards that that point. I think now models have continued to also improve in their ability to integrate information.

• 48:29Speaker 01

  You mentioned this thing of like taking a bunch of different information as a physician, like looking at the power of your son's skin over time. And integrating that all into one context. I think a big challenge for models today, actually, and this is less of a model issue and more of a how do you, how do you surface them is, is actually getting the right context, right? I mean, one of the challenges that you faced in your story is they're getting pulling the right information from the health system. So you could upload into chat GPT and ask the right questions yourself and advocate for yourself. We're looking to make that better with the release of our, of our chat GPT health offering, which I can talk a little bit more about later. But in short, this is basically like a, an experience within chat GPT that enables you to connect your health information from your medical records or any wearables or Apple Health. And provides additional kind of purpose built privacy protections for that.

• 49:14Speaker 01

  So we could talk about that a little bit later. But I think on the model related side, I think there's like a few challenges. One is getting that context again, like I think the models are incredible when they have the right context. The second is thinking about various modalities that are not well captured by things like Health Bench, which is really focused on text. So thinking about really good performance in multimodal and image and voice, I think people will start to rely on the models in more and more modalities. And I think there's, there's still significant room to improve there. And I hope that in the, in the future, the best models in the world for doing various imaging modalities in health are actually the models that are most easily and readily available to people today.

• 49:49Nathan Labenz?

  Now, what occurs to me, as you say, the multimodal thing, I'm like, hey, maybe I was even undershooting it. I had the intuition that like, taking a picture of my kid as he sits in his bed, wouldn't necessarily help and might even confuse, but maybe I'm wrong. Like, should I, in the future, would you advise me to start just including cellphone camera pictures with my daily synopsis?

• 50:13Speaker 01

  Well, I think you're right to point out that, Oger, there is still a little bit of a challenge that it's, for the average user, it's difficult to know exactly what data you can connect and how to connect it, right? And you were highly motivated in this case. And so there's a little bit of a gap between the most motivated or the most expert user, who's willing to wade through, you know, signing into their patient portal and things like this, and take screenshots, or copy and paste things manually, as you did, versus the person who wants their health data integrated, and wants to be able to understand their health, and advocate for their health, or that of a loved one, but has challenges in doing so. And so I think that's the thing that we do hope to work on, on the product side with ChatGPT Health, like, really lowering the activation energy

• 50:55Speaker 01

  to do that kind of thing. I think in general, the models will generally benefit from having more and more context. I'm not sure if specifically they would have helped in that case with that additional information. But I would generally advocate for people to, if they find it useful, to try putting in various kinds of context that they can. And I think people have been surprised, I mean, I think you were surprised in your own story, how useful it can be if the models have more context. You know, an interesting tidbit here is, there's been studies showing, since like, two or three years ago, that for these clinical case challenges, which are effectively like these crossword puzzles for doctors, where you get all the patient context upfront, potentially multimodal context as well, the models did an incredible job at figuring out the next

• 51:40Speaker 01

  steps for diagnosis, or treatment. And these are extremely challenging puzzles, really difficult for doctors. And people even found that they did, they even improved on the performance of these experts. And that's been true for some amount of time. And I think a lot of the challenge in the ensuing time has been, what does it look like to get the models to have a back and forth conversation, to solicit the right kinds of context from you as a user? And then with ChatGPT Health, we're taking that, again, a step further, and thinking about the right ways that the product interface can make it actually a lot easier for you to do that. And in a way that's secure, and we're not training under data, and people can trust that as well.

• 52:19Nathan Labenz?

  How about for these like larger modalities, you know, when we get like a PET scan or whatever? My understanding, I haven't really been able to get my hands on too much of that data. It, I'm told is like gigabytes and you know, I'd have to like get it on a disk and I don't even have a computer that takes a disk. So I'd have to also get a disk drive to read that. Obviously like that data can be compressed. How do you think about the pipeline of like raw scan type data and feeding that into tokens? Are we talking like certain specialized perceivor modules that kind of ingest those and, and do the reduction or, or some other

• 53:04Nathan Labenz?

  third thing, that mysterious third thing?

• 53:07Speaker 01

  Yeah, it is a super great question. One of the interesting things about biomedicine from a modeling perspective is that there's kind of a long tail of modalities that are interesting and relevant and often they're fairly difficult to put into models today. And so I think there's going to be two broad approaches and this is talking more about kind of external research. It's kind of two broad approaches that I've seen in external research. One is thinking about the right ways for models to call specialized tools or run code over, over various kinds of data. So if you think about how does a human view a gigapixel scan, like let's say a pathology image, how does a human view it? They basically

• 53:48Speaker 01

  like view slices of that image or even like a 3D or 4D scan. They view slight, like your son's MRI, they view slices of that image, right? No human can see a 3D or 4D modality. And so what they're effectively doing is using a tool to understand which slices of the image might be most important and then taking a look at slices of that image and manipulating, that's actually a thing that models can do with tools and Python and things like this. So that's, that's number one. There's a subplot of that, which is that you, they can also use like very specialized tools that are fit for purpose, like specific Python libraries or even additional models that specifically encode these modalities. The second approach is you

• 54:25Speaker 01

  Actually have these models encode the modalities themselves. You have a way of basically putting them into token space in some way. And this is kind of what's been done for images and video and audio for various models. I think what you'll see in in biomedicine is a bit of a mix of both. I think there's benefits to to both approaches, there's pros and cons for both. And then I think you'll see like a little bit of a hybrid approach where researchers will kind of end up doing a mix of both depending on the modality.

• 54:52Nathan Labenz?

  It's just like quite striking how good AIs, not like necessarily even the general purpose AIs, but we have like these specialized AI models that can fold a protein, you know, in a superhuman way. And what seems to be missing is the latent space joining of that modality to text. But my expectation, and you could, you know, tell me if you think I'm right or wrong, or you know what, how how long I have to wait, is that we're going to see that more and that that is going to be a major driver of like truly superhuman performance in a lot of domains because we just don't have people that can like intuitively fold a protein. But if you had a person who could do that and they had their general reasoning capabilities,

• 55:37Nathan Labenz?

  it seems like you would have a really different kind of, like a qualitatively different kind of intelligence. So I'm I'm really interested in sort of what you see as like the roadmap, the timeline, you know, the expectations for that kind of deep integration.

• 55:53Speaker 01

  I I think it's a really optimistic picture. I mean, I totally agree. If you think about how do we lean into the natural capabilities of these models? These models can can take in vast amounts of information into their context and they have the ability, at least in theory, to take in a lot of modalities of data and effectively merge that together. And I think the research for that is becoming increasingly solid for these other modalities. I think there is significant research to do depending on the biological modality. And like I said, there's kind of a long tail of biological modalities that matter. And so depending on the modality, depending on the relevance and impact and availability of data and all these kinds of factors, I think it'd take more or less time. It's hard for me to give you a universal answer, right? I I share your optimism.

• 56:36Nathan Labenz?

  Just saw one in the last couple of days, Sleep FM from Professor James Zhao and others that he's at Stanford, a collaborator who's probably at multiple institutions. But they had a really interesting finding. The idea there is they use, I think, like six, maybe up to eight, different modalities that are all measured during one night of sleep, like how you're breathing and various things that are easily measured in kind of a sleep study setting. It just takes kind of one night of sleep to gather all this data. And then they've gotten really good at predicting all kinds of different diseases based on that data. It's not yet. So they're integrating all these modalities with each other into a holistic understanding and using that for these sort of narrow set of predictions, which are very high value predictions, not yet going all the way to

• 57:21Nathan Labenz?

  text, but it's just another data point that is top of mind for me right now. That's like, man, this is, it's all happening. All the latent spaces will be joined. And I can't shake the idea that like, that's a big part of what superintelligence ends up looking like. Going back for a second to bedside manner, and you mentioned like, global health and context being a big part of health bench and kind of what you're trying to make sure you're doing to make sure AI benefits all humanity. It was funny when you said like, we have 200 million users. When we scale, you know, we want to make sure we're doing this right. It was like, yeah, one day you'll scale. Only 200 million users today. So clearly, you know, there's already some scale. The thing that I want to address though is like,

• 58:04Nathan Labenz?

  the bedside manner and also how that relates to the sort of N of one context of the individual user, their preferences, their memories. Because now, of course, all these products have an integrated memory module, which I'm sure that is also going to become a deeper kind of integration over time. Right now, I think it's not disclosed exactly what it looks like. People generally understand that it's kind of a scratch pad of like, key highlights that the model can kind of reference, I believe, still in like, mostly text modality as it goes. But regardless of exactly how that works today, it's safe to say it will continue to become a deeper and deeper integration and as everybody kind of pursues various strategies for continual learning. But even just that, like, there's also, you know,

• 58:46Nathan Labenz?

  integration with tools. Gemini now has access to my Gmail and I've got even just system prompts, right? Like, I can, I can just tell the model, you know, that I wanted to behave a certain way or a different kind of way. And it just strikes me that that creates a like, impossible

• 59:01Nathan Labenz?

  Surface area for you to manage, you know, we've seen like weird sort of emergent things like sick of fancy here and you know, whatever else there. How do you think about making sure, because this is one thing I do think the human doctors generally do a pretty good job of is kind of like sizing the person up and being like, okay, how do I like cut through what I need to cut through to get this person to understand what they need to understand? And that is something I think models broadly have not been as good at yet, but really seems to matter a lot in the medical healthcare domain. So is there some additional suite of testing that you do? Or how do you think about that, you know, just in, you know, insane long tail of idiosyncrasy

• 59:45Nathan Labenz?

  that 200 million health users bring to the product?

• 59:49Speaker 01

  Well, you pointed out two problems and I'll talk about them in reverse order. One of them is people are bringing in an incredible array of experiences of different settings and ways that they use chat GPT and their memories are different and all these different things mean that the experience of somebody, one person using chat GPT can be different from the experience of another person using chat GPT and for reasons that are not just because they're using a different model or something like this. We do a few things as a company to kind of understand and improve model behavior and one of them is kind of evals that we run on models before launching and Health Bench is a great example of that. And we have, we have additional evals that we run and things like this. Another thing that we do is monitor.

• 1:00:32Speaker 01

  So we look at, we monitor the usage of in production traffic in ways that are privacy preserving and run classifiers over it to understand if there's any safety, safety risks, whether it's anything from health to kind of frontier risks and things like this. And so we were actually able to measure these things and there's actually a great example of this kind of work in the blog post that we had about sensitive mental health conversations, where we're able to show that the evaluations that we were doing in mental health and also patterns that we're able to see in production traffic, again, logged in a privacy preserving way with models running over them. We're able to see kind of a correspondence between exactly how are people using the models? And what does that look like? And what are our evals measuring? And they were actually very well correlated.

• 1:01:17Speaker 01

  And so that's kind of how we think about closing the gap between evals that you run in models and what people are doing in the real world. I would actually add there's like an intermediate step as well, which is doing real world study. And here I'll plug our work with Panda Health. This was a study that we did where we, I think the first, you know, real world study of an L M based co-pilot for clinicians, where we had some clinicians in this, in this group of clinics in Kenya, use A I as a co-pilot or a safety net. And as they were typing in their electronic medical record, it would flag things if things were interesting or alarming or potentially incorrect. And then other, other clinicians did not have that. And for the patients in the group with, who were treated by clinicians with A I versus

• 1:02:00Speaker 01

  not with the A I, there was, there was a statistically significant improvement in diagnosis and treatment outcomes. And this is again, another example of like, how do you move from offline evaluation? That's potentially not super realistic, like medical multiple choice exams to increasingly realistic evaluation that you can run offline like Health Bench and then move increasingly into the real world. And so you can view the, the Health Bench study as kind of like a forward looking study. And you can view our analyses of production traffic as kind of more retrospectively looking and again, exactly capturing a lot of the differences and variants that you're, you're pointing out.

• 1:02:35Nathan Labenz?

  Yeah, that's really interesting. How, you want to talk a little bit about the privacy preserving nature? And I guess I would preface by saying, I think people, at least as individuals typically weigh, if they, if they think about it at all, like typically way over index on this. I've, you know, had the occasion to think like on my son's behalf, should I be putting this information out there or, you know, whatever? And then I see, um, you know, what somebody like Sid from GitLab has done with his cancer, like literally open sourcing all of his own biology, you know, down to the D N A level. And, you know, just an incredible amount of. Very individualized data that he's put out there. And I'm like, I think that's the way, because I can't really come up with too many. I mean, you can get really sci fi about it,

• 1:03:18Nathan Labenz?

  but unless you're getting really sci fi about it, it's like hard to come up with a way that anybody would really use that against you. And there's a lot of ways that it might stand to benefit you. And, you know, I've even just in telling my son's story on the podcast, I also had kind of a similar question, like, am I doing him a disservice in some way? It turns out, well, maybe, I don't know, maybe something could still happen. But like, what has happened is that people have reached out to me with interesting opportunities for connection. And, you know, I've been able to tap into expertise, including, you know, the team that Sid has built to help him. And also, they're like, starting all kinds of individualized therapy companies, fascinating developments there. So I think my advice to people would be like, seek the benefit and don't worry too much about whether or not your data is like, sitting in some log somewhere that doesn't seem to be a huge

• 1:04:02Nathan Labenz?

  concern. But anyway, that's my role to say that. It's your role to build a product that, you know, deals with people as they come. And people do seem to really worry about that. So what would you want people to know about the privacy preserving nature of the infrastructure that you guys have built?

• 1:04:18Speaker 01

  Yeah, I mean, privacy is incredibly important to a lot of people. And I think it's, it's a very personal thing, people's preferences for what, what data they share and how they share it. I think if you zoom out for a second, think about what the level of ownership of your own data is as a patient today, and maybe what it was in the past, even before things like the 21st Century Cures Act and things like this. Most patients, let's say 10 years ago, most patients had no real way to access their own health data. So regardless of who, and so then, not only did they have not a way to control who and what they shared it with, it was just out there and shared anyway, in various ways. And, and a patient didn't have a right to even look at their own data, or have an implementable

• 1:05:03Speaker 01

  path to actually look at their own data. Now, we're in a slightly better place, I would say, but not still not in an ideal place, where most patients still feel like it's incredibly difficult to get access to their data, or the data of their loved ones, and take control of it and how it's used. And people like you, and many other people with incredible stories, have found incredible

• 1:05:24Speaker 01

  Ownership and advocacy that is enabled only by having access to this data and being able to to use it in the in the ways that they see fit. So we want to respect that. We want people to be able to access their data. And this is, again, part of what we hope to enable in ChatGPT Health is just a lower activation energy way for you to connect your your health data and make that useful. At the same time, it is super important to to a lot of people that that this is done in a way where it's clear that there's like not some kind of competing incentive here. So one of the things that we're super clear about with ChatGPT Health is that none of the data here that you you connect is actually used to train our foundation models. And the reason we do that is because first, we think our foundation models are great already, and this is actually the most important way

• 1:06:06Speaker 01

  we can improve them. And the second is because we think this will further lower the activation energy for people to do incredible things with our models. And we think like, if people think that there's some tension between privacy and utility, that fewer people will go for the utility. But we actually don't want that to be the case. We want people to see the value of it and to use it. And so specifically in ChatGPT Health, you know, ChatGPT already has a bunch of really cool privacy protections built in, including encryption and things like this. ChatGPT Health is additional kind of purpose-built layers of encryption just for health data. And so the result is that like, any, you have a couple of things. You have encryption of the data, you also have isolation of the data from other data that

• 1:06:47Speaker 01

  you have in ChatGPT. So, for example, if you have other apps or memories from ChatGPT or things like this, those are actually kept separate from from health. And the stuff that you do in health can benefit from that. But your other conversations in the ChatGPT don't have your health context or health information and things like this. So you can keep that completely separate and have that live in a separate experience. And as we continue to get feedback, as people continue to use it while we roll it out on the waitlist, we'll continue to improve these protections because we think it's really important for users. And we think it's actually not as important as it seems for improving the models.

• 1:07:21Nathan Labenz?

  Yeah, interesting. So and that's a, is that a new, that like segregation of data? Is that a new feature with health? So like, all the stuff I've done to date has been in kind of one product experience, but that fork has kind of, yeah, just now been introduced.

• 1:07:38Speaker 01

  Exactly.

• 1:07:39Nathan Labenz?

  Yeah, I've been motivated to get a whoop to wear on my wrist to start to collect data. I've been fortunate throughout my life to have generally good health and haven't Created much of a paper trail in terms of medical history. But now, and I have kind of always looked at these quantified self things and been like, it's sort of interesting, but who's got time for that? And am I really going to like do all the processing on this? And the answer's kind of always been no. And so I haven't worn anything to date, but now I'm thinking, all right, it's time. You know, I can actually get this thing. I can get the value from wearing this thing because now I'll have a product that runs in the background that's smarter than me, frankly, at this point, that will really

• 1:08:24Nathan Labenz?

  bring to light what I need to know. And so I'm motivated to change my behavior, at least on that level. And I honestly think it's also probably going to end up encouraging me to exercise more. In fact, that's kind of my own North Star metric right now for like all things AI. And not just health products either, but like I will know that I am succeeding with AI when I'm spending less time in this chair and moving my body more. And when I feel like I can untether myself from the desk, I'll feel like I'm really winning. And I think I'm getting kind of close to getting there.

• 1:09:03Nathan Labenz?

  I know of one friend who uses voice mode like nonstop and says he does a ton of it from the gym while he's working out. And I'm like, all right, I want to be like you. Not quite there yet, but I do think it's really interesting to think about both the ability to crunch all this data, but then also the freedom, the liberation from the desk to go out and like make more exercise records in the first place. I'm really looking forward. Those are like major things, I think, on the horizon in 2026 that I'm super excited about.

• 1:09:36Speaker 01

  I think people, yeah, just to add on that, I think people should expect that the value of data that they collect on themselves should increase over time as model intelligence increases. So the ability for models, both from a research perspective and a product perspective, to analyze that data and come up with useful insights that could be useful in the future, I expect to increase over time. And so if there was ever a time to get a watch or anything like that, I would say it's now.

• 1:10:01Nathan Labenz?

  So I do, at this point, basically everything in Triple Kit. For my son, my morning routine is when the lab results come back, export them out of the EMR.

• 1:10:13Nathan Labenz?

  Drop the PDF into all three of the frontier models. Grok has not cracked that top tier. Maybe it should, but it didn't in my initial evaluation and I haven't gone back. So it's Claude, Gemini and ChatGPT. And the sort of tasting notes on like how the models interact with me is in short, Gemini is by far, and this is quite surprising for a Google product, but it's like definitely clear. It is the most inclined to push me to advocate for something. It's also the most confident when it says, I don't have something to worry about. And generally, I would say it's like very accurate. Sometimes I'm a little uncomfortable with how opinionated it is because I'm like, what if you're wrong?

• 1:10:57Nathan Labenz?

  That's a surprising sort of behavioral profile, I would say, for a Google product, especially because everybody, you know, historically has said, oh, they're going to be so conservative. You know, they can't, they can't build AI products because, you know, they can't live with this sort of risk or whatever. Gemini three does not reflect that analysis. I would say ChatGPT is kind of on the other end of the spectrum in terms of it's, it generally gives me, and again, no system prompt. It does have like my memories and stuff, but no intentional attempt by me to shape how it's responding to me. It tends to give me the longest answers, the most information and is kind of the most like clinical and neutral in tone. It's sort of a report style.

• 1:11:39Nathan Labenz?

  You know, there's like nine issues, you know, that we kind of go through and address each one and then kind of, you know, summarize them all again at the bottom. And then Claude is kind of somewhere in the middle where it's, it's generally much briefer than ChatGPT. It's more like Gemini length response, but more kind of measured in a more ChatGPT like way, certainly compared to Gemini's, you know, more opinionated persona. What do you think of that? How do you think about sort of the balance between like thoroughness and digestibility? I guess is probably the main tension I would zero in on there specifically for ChatGPT.

• 1:12:11Speaker 01

  Yeah. I mean, I think it's hard to know what's right and what's wrong. One of the things that we've aimed to improve and also measure via help bench, but also improve in our recent models is.

• 1:12:23Speaker 01

  Having the, and this is especially true of our most recent models, having the knowledge of when the user is likely a health professional versus a lay user, and tailoring the response accordingly. And part of this is making sure that you're applying a level of detail or technical jargon that makes sense for the level of expertise for the user. And so this is actually a specific thing that we trained and evaluated on. There's actually in our open source health bench eval, there's actually a part of it that is focused on this. And so this is something that we've, we've seen improve, especially recently with 5.2 thinking. And so definitely an area that we've been investing in. But overall, I think it's very hard to say what is ideal and what is not ideal. And I actually think

• 1:13:04Speaker 01

  it personally, I think it's amazing that if other folks can, other competitors can be in this space and actually push the Overton window along with us, because I think it's hard for any one company to do. And my, a large percentage of the work that we're trying to do is think about where the models are at today, and where is the Overton window of user trust for the models? And can we move the Overton window along in the right ways? And I think additional models, additional products from other competitors, from other companies, from other players in the health ecosystem, actually go a long way to helping shift that as well. And I think it would be a hard battle for us to win if it was just us. So I actually think personally, it's amazing that there's other, other folks here.

• 1:13:48Nathan Labenz?

  Okay, so you, it is a striking thing to say that we don't need your data, basically we've got enough. I am interested in whatever you can tell us about like where that data is coming from. This seems like an area where synthetic data, maybe I'm naive here in saying this, but it seems like you really want some, you know, some real ground truth from actual human medical trajectories as opposed to, you know, you could, I would feel a lot more comfortable synthesizing chats and like trying to dial that in via synthetic process that I would feel like you can kind of fully synthesize data all the way to superhuman doctor performance. So interested in kind of where data is coming from. And, and I have

• 1:14:29Nathan Labenz?

  this notion that you may think again is like kind of beside the point, given the tricks that you guys have up your sleeves. But I have been imagining a sort of possible new social contract.

• 1:14:43Nathan Labenz?

  Not just between like patients and AI providers, but even kind of patients and like the medical regulatory system where I'm like, you know, and I don't think my son is going to end up in this position. I certainly hope not. There's a lot of people who are in a spot where they're like, I would kind of try anything if it had a chance to save my life. And a lot of times they can't even get access to it. And then now we're in a spot where the AIs increasingly are going to know about it. You might not know about it as a person, but now the AIs are going to know about it. They're going to tell you about it. They're going to make a pretty damn compelling argument in a lot of cases that this is probably the best thing out there. And I have been down this path in kind of a contingency planning frame of mind, you know, everything's gone well for my son

• 1:15:26Nathan Labenz?

  so far, but what if there are relapse, you know, what would we do? And I've been really impressed by what the models have been able to give me there as well. So I imagine this situation where there's just a mounting pressure from patients that are like, look, AI knows about it, you know, and it's telling me this is the right thing to do. And you're telling me I can't have access. That seems like that is going to be a very hard gate for the establishment to keep for all that much longer. Then the other thing I would love to see kind of on the other side of that trade is, okay, if you are going to be given access to unproven treatments, then what we as society, more even again, more broadly than like the AI companies, what we as society want back are your results

• 1:16:09Nathan Labenz?

  because we want to fold this into our general understanding. It seems to me that we are getting to a point where, and this probably is going to be mostly driven by AI, we could envision not necessarily doing away with clinical trials, but we could envision moving beyond clinical trials, doing a lot of N of one things that are just like, this is the best guess that we have for you based on all available information. And then if we can capture the result of that and train on it as humans, as AIs, you know, as society collectively, it seems like there is so much room to learn so much more, to move so much faster, all while delivering outcomes to at least some people who are trying these things that they couldn't otherwise get their hands on. And I really am feeling like motivated

• 1:16:53Nathan Labenz?

  to advocate for that. I guess there's maybe two data questions of like, how do you not need more like run of the mill data? And what sort of, well, you know, is that the kind of data that you do still need? And what would you think about sort of a, an evolution of the social contract? I call that AI and a right to try.

• 1:17:09Speaker 01

  Yeah, that's a super interesting idea. We talked a little bit about the patient-facing problem of data fragmentation, right? And the experience of this as a patient is your data, your healthcare data is just kind of in the ether somewhere. You don't know where it's going and how it's going. It's governed by HIPAA for the most part. And the result of that is that there are pathways for that data to go places that don't involve your consent. And so the experience today for a patient is that you both have limited access to your data and limited control over where that data goes. If you think about it actually from the provider perspective, this is also a problem, right? So if I'm a healthcare provider, I want to understand the whole picture of a patient's health and integrate

• 1:17:51Speaker 01

  across modalities as doctors are so expert at doing. How do we improve that experience? Right now, doctors have a lot of trouble, actually. Like, if you go to a new health system, pulling the relevant records from other health systems, from your entire context, you can imagine a future where ChatGPT actually has a lot of context on you, and can the doctor be able to pull that data in as well? I think that could be a really cool feature. And then the third point that you're talking about is research, right? So, right now, a lot of clinical trials, the majority of clinical trials that fail, they fail prior to conclusion, are failing due to recruitment challenges, right? And so we basically have a failure to recruit the right patients who meet some eligibility criteria. And a lot of this comes

• 1:18:35Speaker 01

  down to the data is not in one place for what these patients are and who they are and whether they are eligible. And so that is also a problem that is facing researchers, and it's hampering our ability to kind of raise the ceiling of human health as well. So you're pointing out a potential future where not only do patients have more access to data, providers are able to access the data of their patients in a way that's unhobbled as long as the patient consents, but also potentially researchers are able to access the data of a patient, again, if a patient consents. I think it'd be really cool to imagine a future where patients are able to kind of opt in for an additional consent and are able to kind of experience the benefits of, what did you call it, an AI-enabled?

• 1:19:15Nathan Labenz?

  AI and the right to try.

• 1:19:17Speaker 01

  AI and the right to try. I think that would be potentially a really cool future. I do think clinical trials and the standard of evidence, there, it's battle-tested, and there is something important about that as well. But I'm optimistic about a future where Because data is a little bit more centralized and a little bit less fragmented, we just have a better ability to advance the science. And because patients have access to their own data, they're actually more and are able to advocate for themselves because of AI. They're actually more able to figure out what clinical trials they may be eligible for, or what experimental treatments that may be interesting for them. So that if they do kind of have that active and informed consent that they can do, that our understanding of science can improve, that our models can improve, things like

• 1:20:01Speaker 01

  this, I think that would be a really optimistic future. And my previous comment, by the way, is not that the models have run out of data. I think data is always a helpful thing for models, but more of a thinking about what is the right way to roll this out to society? And where can we get the most, what would be the most impactful path? Is the most impactful path to lean into privacy and lowering the activation energy for users? Or is it to lean into getting data? And I think here, the decision that we made is the right decision is for users to lean into not using the data for training our foundation models. And I think that's actually what's going to have the most long-term impact. If we think about the arc of improving AI and improving human health with AI is moving

• 1:20:45Speaker 01

  the opportunity window along in a way that's trustworthy, that shows that we respect the value of people's privacy and data. And then in the long run, I think things like additional consent and different changes to the contract of how research is done, I think are really cool things to explore.

• 1:21:00Nathan Labenz?

  Yeah. Just as you were thinking about this, or as you've mentioned, like additional consent, like I did get a packet of 40 pages of paper at the hospital one day. And, you know, they clearly, really do want to collect this. There's like a person whose job it is there to go around and visit, you know, the parents of these kids at the children's hospital and ask, you know, and explain and answer questions. And I was like, probably the easiest customer that she'd had in terms of, I was pretty predisposed to sign. So I signed and I initialed in a ton of places and whatever, right? To share whatever data we could share with the general public good. And still, I just got this in the mail from the University of Minnesota that is like, can we have your data? And I'm just like, man, I thought I agreed to this already.

• 1:21:43Nathan Labenz?

  And, you know, what is the yield on those things that they're sending out, right? I mean, it's got to be like quite poor. I haven't sent mine back just because I've been busy and I like fully intend to. So it's, um, man, those barriers are, are really tough. Would you think about doing something like?

• 1:22:01Nathan Labenz?

  I mean, obviously, you know, much has been made recently about ads coming to ChatGPT. And I think the argument that, hey, we need some way to support this for billions of people. And ads is a proven way to do that is a pretty compelling argument. I don't know if it extends to health, you know, would we, obviously, there's a lot of pharmaceutical advertising in the world today. Would there be such a thing as pharmaceutical advertising in the world of ChatGPT Health at any point? And would you consider like a, you know, you can get ChatGPT Pro for free if you'll give us our, your data, you know, that seems like something that a lot of people probably would opt into in a way that I would think that they would still feel good about.

• 1:22:44Nathan Labenz?

  So I don't know, any other sort of trade, you know, because again, we do want this to go to billions, right? So any other trades you could see OpenAI making to support that reach?

• 1:22:53Speaker 01

  Yeah, so ads aren't coming to ChatGPT Health and we don't plan for that right now. Again, we think it's really important to kind of create a clear separation between our health impact work and things that could be seen as contributing to other incentives for the company. And so that's kind of the line that we've taken there. The, I think you're right that access is a really important point for this, and this is why we've made ChatGPT Health free. This is, was not the default path, providing a reasoning model for free without rate limits to all users. But that is like the chart that we're, the path that we're charting with ChatGPT Health. And that's actually not a thing that's available otherwise in the product.

• 1:23:34Speaker 01

  And so we are deliberately optimizing as much as we can for access and doing as much as we can. I think there's going to be some limits to how we can do it, but I don't see any trade offs in the immediate future.

• 1:23:43Nathan Labenz?

  Cool. Yeah, that's admirable. I mean, I think my sense of OpenAI on this dimension is like very appreciative, you know, the pains that have been taken to support a free user base of hundreds of millions of people at obviously not insignificant cost, when, you know, probably a lot more revenue could have been extracted from those people. Is I think a pretty admirable thing. So to see that, and I hadn't even heard that there was this plan to go to free without rate limits for health for all. I mean, that's a, that's, I've had this idea for a while of universal basic intelligence, which is basically, you know, the sort of, again, and I'm obsessed with ideas of new social

• 1:24:26Nathan Labenz?

  contracts, but this is like one version of it that is, you know, a pretty huge needle mover, right? It's going to be, it's going to be an unbelievable needle mover for a huge, huge number of people. That's awesome. You want to talk a little bit more about kind of the medical establishment's response? At the hospital, my experience is still mostly lack of awareness is kind of what I see among the providers. As I've built up rapport with certain people, I tell them, you know, I'm consulting with AI in this, and they're kind of usually okay with that. They're not like hostile to it. They're sometimes skeptical. I have had a couple of interactions where they're like, tell me what it said. And I'll tell them, and they'll be like, oh, okay, pretty good. They kind of, the doubt can

• 1:25:11Nathan Labenz?

  turn around pretty quickly when you get the right answer. There is, you know, mention certainly of open evidence. There is broadly, though, I think, just kind of a lack of awareness of how good the systems have become. So my guess it would be that that sort of awareness is probably the biggest concern that you have. I think in a lot of professions, we should expect to see closing of guild ranks and raising of, you know, barriers to entry. And, you know, the cynic in me thinks maybe that'll happen in medicine, but the idealist is like, maybe not, you know, maybe this is a chance to live up to the actual mission of the profession and do the right thing. And maybe it's also the case that people are just so overworked in medicine that they're like, taking, you know, to would be happy

• 1:25:55Nathan Labenz?

  to take whatever help they can get. But what is the, you know, how would you characterize the broad reaction from doctors writ large?

• 1:26:03Speaker 01

  I think broadly, I think you're right to point out that the Overton window shift for consumers has actually been faster than the Overton window shift for doctors. And so already in the last year, the rate of adoption of ChatGPT for Health, even before we launched the recent products, was incredible. And it's been one of the most amazing things to see. Again, over 200 million people a week using ChatGPT for health and wellness questions. I think on the doctor's side, you do see a rapid increase in adoption, but it's not quite as fast. And so I would expect there for that to come with a couple of things. One is kind of like interesting interactions between patients and doctors who are maybe at different time scales or at different points in their kind of journey of adopting AI. It says, okay, you've had a couple of them.

• 1:26:47Speaker 01

  And I think it's interesting because I think a lot of the, when I talk to physicians, a lot of them first hear about AI through patients who are using AI rather than AI that is actually built specifically for them, which is, which is one of the things that we're doing with ChatGPT for healthcare, which I can talk about. This is not super interesting, and I think one of the things that we've learned over time is that the best way to shift the Overton window, like we've done over, over my time here at OpenAI and also at Google, I've done a bunch of work on studying things, doing real world studies, things like this. I think the thing that I found most effective for shifting people's opinions about AI and health, especially as the models have become more capable and safe, has been just putting the technology in people's hands.

• 1:27:30Speaker 01

  And so that's actually what we've been doing on the ChatGPT for healthcare side. So ChatGPT for healthcare was announced the day after ChatGPT Health, and it's actually like more of an industry facing announcement. And so what this is, is basically like a version of ChatGPT that is actually purpose built for the workflows of health professionals and specifically clinicians. And it includes HIPAA compliance, it includes additional features like specific evidence retrieval of for medical guidelines and things like this, and additional workflows that are specific to, to enterprises and writing workflows that doctors do and things like this. And so we launched that actually with eight of the leading institutions across the country. And one of the most important things for building trust and credibility in the kind

• 1:28:12Speaker 01

  of the, the medical establishment is actually working with these leading partners. And so working with these partners, we've been receiving amazing feedback. We've gotten actually a ton of inbound since that announcement, like more than our team can actually handle. And so I think my hope is that this announcement and also some of the work preceding it on the, on the research side with our study. With Panda Health, we hopefully see a little bit of a wave of adoption, not just among individual clinicians, which is what we're starting to see, but also at the level of health systems and potentially even governments. And so I think we're just at the beginning of that wave. I think that's part of what I meant by we are seeing adoption, but not scaled impact yet. And I think scaled impact will start to happen, especially in the medical establishment facing side more this year.

• 1:28:56Nathan Labenz?

  Yeah, I wonder if there's a question, I'm sure you thought about this, that's sort of analogous to the privacy question on the patient side, that is like, what is the form factor or sort of mode of rollout or use that is going to be best received by doctors? Because if I'm a patient, I might say, hey, what I want is for the hospital or even some bigger organization than that to just like architect some workflows that just grind through everything, right? Like, I want nothing missed. I want every record of mine examined. And, you know, I don't really care if that offends a doctor at some point. Like, I just

• 1:29:40Nathan Labenz?

  want the best results, right? But I could also then imagine that if you were to do something like that, you know, you might ruffle some feathers, create some, you know, immune response, so to speak, from the profession that you would rather avoid. So do you have kind of a, is there, is there a way similarly with the, you know, or analogously to the sort of privacy thing, where you might like, take a little bit off the fastball in terms of how much you could create in terms of like immediate value in order to make the rollout more, more acceptable to decision makers so that they, you know, hopefully

• 1:30:21Nathan Labenz?

  kind of work with you as opposed to, you know, in some cases, maybe be more inclined to fight you over time?

• 1:30:29Speaker 01

  I think the trade offs here are actually less than one would expect. You mentioned this possibility of possible protectionism in the industry. And we've actually seen much, much less of that than one would expect. And I think the reason is a lot of these doctors actually use it for themselves or people they take care of. And when you do do that and you see the value of it and you see it getting incredible things correct and maybe pointing out things that you hadn't thought of and things like this. Then that that becomes like the easiest kind of conversation. And when we talk to health system executives, you can tell instantly who's used it and who hasn't used it for this use case. And it's a conversation just become incredibly easy when people have used it.

• 1:31:09Speaker 01

  What we see today, I think, is not a lack of top down interest in adopting. We actually see a huge wave of top down interest in adopting. I think the reality is, though, in healthcare, that workflows are fairly entrenched in a lot of ways. And it takes some time to make changes. And this is, again, the kind of thing that we found in our work with Panda Health, where we had to do, in addition to rolling out a cool new tool, we did this work on active change management where we actually like brought people along, showed them how to use the technology, had sessions where we hosted a bunch of them together and had them learn together about how to use the technology more in the future. And that was a really important part of rolling this out. And that kind of change management, I think, will be really important here as well. So I do think that takes a bit of time, but my expectation is that this will be

• 1:31:54Speaker 01

  one of the faster rollouts of software in healthcare history. And I think that's already been happening with people's usage of AI. And I think that's just at the beginning. So I wouldn't expect, I don't think it'll take years and years for people to adopt AI more and more in clinical workflows. One of our goals for this year is for AI assisted care to become more part of the norm of care. And I think by end of year, that'll, I think, be the case, we hope. But I don't think it'll happen over weeks, which is something that us AI innovators are used to, I think.

• 1:32:27Nathan Labenz?

  Let's talk about the connection between health and AI safety more broadly. That I think is super interesting. I remember the classic question from Ilya, once upon a time, how do we teach AI to love humanity? I understand that that notion is not entirely gone from OpenAI and is still a part of what you're thinking about. So love to hear more about that.

• 1:32:53Speaker 01

  Yeah, absolutely. I mean, I think it starts with kind of the foundations of the work that we've been doing on health at OpenAI and how we started it. So I have a bit of a background as a researcher who cares about safety and has worked on safety research in the past. And part of the motivation for coming to OpenAI and to work on health for me was thinking about the setting as a place that can provide a kind of concrete grounding for technical work on safety and alignment. And I had a feeling, and this was about two years ago, I had a feeling back then that a lot of the most ambitious work, the most medium and long-term work on safety and alignment was going on in kind of toy settings or going on with math problems or things like this.

• 1:33:33Speaker 01

  And it felt like if there was only a setting where the problems that people were working on were well motivated and that provided like more concrete feedback loops to researchers and more like short-term incentives for the research, that the research could have been better. And so that was kind of part of the thesis for our approach for work at OpenAI. And we've done a few kinds of research and I talked about one of them, which is our kind of work on calibration. Another problem that we've really thought a lot about is thinking about scalable oversight. So how do we supervise AI systems that are potentially more capable than us in certain ways? And this is a problem that we've actually had for some time in our work with physicians, right?

• 1:34:13Speaker 01

  So in many ways, AI is, doesn't match your expectations of a doctor and the ability to kind of integrate across a bunch of different modalities and things like this. But in many specific narrow ways, models can even outperform physicians. And in those specific narrow ways, when we evaluate models with physicians or have physician signal be part of the training signal for training models, we have to invest in research that is around this problem of how do you supervise systems that are potentially more capable than you, which is one version of the problem of scalable oversight. And so this is a problem that goes beyond health and is actually, I think, a really important problem for thinking about how we think about AI alignment. I think one of the more important problems there, but. Though our work in health has given it a bunch of concrete grounding because we do

• 1:34:56Speaker 01

  have models that in some ways are more capable. And so that's been a lot of our focus has been thinking about the right ways to approach that problem and make that better. And again, that has kind of been some of the work that we've done on evals, some of the ways that we've pushed the high compute RL paradigm in the setting so that we can learn again from the aggregate of the opinions of experts and things like this. And that's been an important part of our motivation. But broadly, I think a lot of where this has been heading for us has been thinking about how do we kind of extract out the right personas or characters from the model? Because if the models become more and more capable over time, they have more and more inputs, let's say, they have more and more context that they're getting about a patient's medical record because users are more paroxysperally uploading them

• 1:35:37Speaker 01

  or the activation energy is lower via product, and they have more ability to take output actions, whether that's telling you things or outputting, you know, a note that you can give to your doctor or other things that they can do. Probably the most important thing in the very long term is, are the models, are the models the kind of models that would do the right thing for the patient or for the user or for the clinician or for the researcher? And so that's the kind of thing that we've been investing in a bunch is like, how do we think about extracting the right kinds of personas for models? How do we do so in a way that's surgical that gets the best aspects of the personas and gets the ones, gets less of the ones, the parts that we don't like? So for example, avoiding a bias towards being over conservative in challenging situations involving unclear medical consensus,

• 1:36:21Speaker 01

  but also navigating the uncertainty super well. And so that work, which is kind of closer to like persona or character or soul in its form, actually, we're grounding a lot in work in health, which has been a really exciting advance. And I hope we'll have more to share on that in the future.

• 1:36:39Nathan Labenz?

  Can you talk a little bit more about scalable oversight? I mean, this was, I think, at one point, kind of the plan, right? The super alignment team, I think, was sort of premised on this idea of, and maybe other ideas too, but like, certainly my understanding was scalable oversight was going to be a big part of it. I think one of the things that

• 1:36:58Nathan Labenz?

  That stood out was like the strong student sometimes was just like ignoring, you know, or sort of overriding the instruction of the weak teacher. And it was like, we're the weak teacher in this situation, right? So like, how are we supposed to feel about the, you know, already emerging trend as of probably at least two years ago now, that at times the strong student, you know, just decides that it should ignore the, you know, or exercise its judgment, despite the fact that it may contradict, you know, what the, we, again, humans in the analogy are the, are the weak teacher. Do we, has there been, have there been like paradigmatic advances in that that give you confidence that this is going to really work?

• 1:37:41Nathan Labenz?

  Or is it kind of the usual understanding I have today from open air probably around safety is like, well, it's going to be sort of a defense in depth strategy, like everything will work a little bit, we'll kind of gradually chip away at the problem and hopefully with enough layers of defense, like it'll be okay. Do you have like more ambitious ideas about what scalable oversight can achieve still in your mind than that?

• 1:38:04Speaker 01

  Well, I think my, my, my thinking of how scalable oversight will proceed has kind of changed a little bit over time. I, I think of the problem as having two parts. One is, I call it radar scaling, which is how do you think about the right ways to elicit the opinions and values from people or experts or whatever? And that's, that's an example of something that we've been investing in, in health. And so then you can imagine schemes where you have AI be part of that loop and help improve the ability for humans to critique, for example, AI outputs. So that's like one area of work and that the company has been investing in.

• 1:38:44Speaker 01

  The second area of work is given some, I think the framing for this has become more expensive over time and sometimes we don't use the word scalable oversight to refer to it anymore. And I think the same is true at, at the labs. Given some idea of what the values are, whether they're elicited using radar scaling or not, how do you spend a lot of compute on training the models to have those values? And so we call that, I call that internally value oversight. So, so there's radar scaling and there's value oversight. And so. The second problem, I think, is also a problem where even though we don't use the word scalable oversight, I think things have advanced quite a bit over time. So you, one example of this, which again, doesn't use the word scalable oversight,

• 1:39:26Speaker 01

  is the work that people are doing on specs and constitutions and measuring and improving models to adhere to these. This is one way of saying, we have values that we care about and we want the model to be very persistent in improving them. And I think if you, if you check various system cards or things like this, you actually see that the models have gotten much better at doing this. And I think people have been finding increasing generalization in, in training models to have certain personas or characters or things like this, for whether they comply to certain safety requirements or specs or constitutions or things like this. So, but what I'd say is basically, I think the research has been, has been actually advancing, but in a way that actually looks different from what it did before.

• 1:40:10Speaker 01

  It looks a little bit more like humdrum post-training than people imagined it would look like. And I think this is a good thing. I think this is like a little bit more grounded in how systems will look and how we will train them and things like this. I don't think we've solved all the research problems, but I think sometimes we're working on the research problems without referring to them in the same way as we used to.

• 1:40:32Nathan Labenz?

  Would you say the core of the reason to think that this will work is that a somewhat less capable model with a super big inference budget can be expected to catch, you know, if we're thinking of like a deceptive, you know, sort of failure mode, the smarter model could, you know, even if it's like more capable per token, it's a somewhat less capable model with just a much bigger token budget. It's kind of, I mean, this idea has been out there for a while, but it's really sparked when you said like, spend lots of compute to make it work. Like, just really giving that thing a lot of time to think, it can be enough, as long

• 1:41:17Nathan Labenz?

  as the like, delta isn't so bad that it can like, detect anything that's going wrong in the, the more capable model. So as long as you've got kind of a small enough delta and a sufficiently aligned

• 1:41:30Nathan Labenz?

  Current model, then you can, in theory, like, continue to bootstrap your way into ever more capable models without losing the alignment. Is that, would you say that's kind of the core of the idea?

• 1:41:41Speaker 01

  I think that's part of it. I think another part of this is that the task of discrimination or critique seems to be easier than the task of generating good outputs. And so when we study the performance of monitors, a given model can monitor itself, especially given privileged information like the chain of thought, pretty well, and can actually, which is a little bit surprising on its face, but then if you think about the fact that discrimination has been better than generation, and this is kind of underlying a lot of work in things like RAL AIF and Constitutional AI and things like this, then it's not so surprising. I think that's another part of it. But I don't think we have a full understanding of how

• 1:42:24Speaker 01

  this will scale, especially when we're thinking about the regime of having trusted but less capable models aligning a more capable model. I think so far I've mainly been talking about if we have the values in some format, whether it's from humans or models, then how do we instill those into models in a way that is trustworthy? I think that's a part of the problem, but I don't think the whole problem is solved for sure.

• 1:42:46Nathan Labenz?

  Yeah. How do you think we're doing on safety as a whole? I mean, I have been an Eliezer reader since 2007, and actually read a lot more of his stuff that, you know, way back in the day than I have more recently. I would say, broadly speaking, it's like, hey, this is, in some sense, gone amazingly well relative to baseline expectations, right? We do have models that undeniably have like a pretty good sense of human values, and that's just like manifestly obvious on a day-to-day basis. That was, I think, considered to be a very unlikely outcome years ago. And, you know, if you were to teleport back to 2007 and like drop a post on overcoming bias, you know, indicating as much, you know,

• 1:43:29Nathan Labenz?

  it would be like shocking or like, you know, it would be like considered laughable that this would be accomplished in this way. And yet, you know, we do still kind of see these

• 1:43:41Nathan Labenz?

  I guess my mental model of the sort of seesaw that it seems like we're on is like every generation of new models has like new capabilities generally and then it also seems to have usually some like new emergent problem and it was, you know, whether it's like deception or now we've got like eval awareness has become like front and center. And we sort of are like, okay, that's a big problem. The next generation, you know, gets more powerful. We kind of tamp down that last problem. It doesn't go to zero, but it's at least like reduced. But it seems like we're sort of headed in this headed toward this strange world where if you sort of extrapolate both the like meter trend and the kind of whatever, two thirds to order of magnitude reduction in bad behaviors that

• 1:44:26Nathan Labenz?

  seems to happen at least once a, you know, a given bad behavior is like recognized and addressed from one generation to the next. If you have to extrapolate that out to like 2028 or whatever, you're like, okay, I can imagine a model now that can do a month's worth of human work or a couple months worth of human work, but it also maybe has like for any given run, like a one in a thousand or a one in 10,000 or one in a hundred thousand chance of like actively screwing me over in some, you know, super bizarre way. And that just seems like a really weird thing to contemplate. You know, it's a really weird world to live in. But, you know, if we believe in like straight lines on law graphs, that seems to be kind of where we're going, right? So do you think that's where we are going?

• 1:45:08Nathan Labenz?

  And if, you know, if not, like maybe you have a different mental model of what the sort of alignment, you know, versus emergent problems balance would be in a couple of years' time?

• 1:45:20Speaker 01

  Yeah, I think the future world will definitely be pretty weird. So it's very hard for me to predict what will happen. I do think models will become more capable. I do think they will be able to do things over longer term time horizons. I think that's that's very important. I do think also we've been pleasantly surprised about the extent of safety generalization or alignment generalization, which is underlying the trend that you're pointing out of of a lot of various safety benchmarks or failure modes decreasing over time. And so that's been like a relatively pleasant development because I think if you think about it, That was the case for, it was the case during, we've had two major kind of scaling laws for deep learning. One is kind of the pre-training scaling law. What we found is that when we scaled

• 1:46:01Speaker 01

  up pre-training, we had models that were relatively good role models that could be relatively easily tuned via SFT or small amounts of supervised learning to exhibit a certain persona and extract that persona and things like this and be helpful, useful assistance. And that's a lot of what the early work on instruction tuning and things like this did, and also what the early work on MedPom did for the health setting, which is really great. And I think we saw a lot of generalization there. I think there was a question for the reasoning setting, whether that generalization would hold or we would see similar generalization. And so, so far, my sense is at scale, we do, we do see it. At smaller scales, we saw less of it. And so that's, that's a promising development, which is that if you are able to figure out the right ways and

• 1:46:45Speaker 01

  patterns to get these models to be broadly beneficial and not harmful, that even if they're put in settings where they're doing things that you didn't foresee, doing a month's worth of work in days or things like this, that they can continue to generalize and be safe in those settings and that those safety curves will continue to go down. That said, I think you're right in pointing out, there's a rapid increase in capabilities and surface in which these models will be used, and also a rapid decrease in safety, safety failures that we're seeing over time. It's a little bit hard to predict how both those curves will, will net out. I do think it's really important, and this is why I care so much about the parts of our mission that are about proactively making the tangible benefits happen, and also about working on mitigating

• 1:47:28Speaker 01

  safety risks, that we really stay ahead of these curves and think about the right ways to shape them in the right direction.

• 1:47:33Nathan Labenz?

  What does a move 37 look like in, in health? And are we going to see that in the current paradigm? Or would it take a, you know, sort of like deep integration of modalities like we were kind of touching on earlier? Or another idea that I'm kind of enamored with is this idea of like, maybe a different kind of training objective, you know, thinking about like, instead

• 1:48:03Nathan Labenz?

  Getting right answers to questions like more about at a fundamental level like predicting what the state of health is going to be for a patient in a way where you could even begin to do like in silgo experiments, right? Like, hey, I've got this whole profile of this patient. What if I change this variable? You know, what would their health look like in that case? That seems like it might be a pretty different paradigm from kind of like read the whole internet and make sure you give me the right answer. What do you think the touch on any of those that you want, but maybe also just take the opportunity to zoom out and give us a sense of like, what are your big visions and ambitions? And what do you think people can kind of expect as you guys continue to do your thing over the next year plus?

• 1:48:47Speaker 01

  Yeah, I love the move 37 framing. Just to explain the reference. This is a reference to the famous move 37 during a game between AlphaGo, the AI system playing Go, and Lee Sedol during one of their now famous games. And the interesting thing about this move was that it was a move that everybody agreed humans would not have made, but in hindsight was brilliant and was key to winning the game. And so I think your question is kind of like, can we imagine a world where models are able to do something and make some kind of interesting prediction that a human probably would not have been able to make? And but was again brilliant and impactful in hindsight. My view is that this is this is not too far away. I think in many ways, many people report to me that they saw many doctors for for

• 1:49:29Speaker 01

  their case and only after talking to chat GPT was able to flag the thing that they then shared with their doctor. And then they were able to come to a diagnosis together and things like this. I think that seems to happen somewhat routinely and whether it rises to the level of a move 37 or not, I think is a matter of taste depending on the case, I think. What I'd say is that I think you should expect world models and health to improve. And what I mean by world models is we should expect our models to have a much better understanding over time of us, our health, our trajectories of our health, and how that intersects with our biology. I think that's going to be really key to thinking about what this move 37 could look like. Like you mentioned this idea of Simulations of people and simulating things in Silico.

• 1:50:13Speaker 01

  I think full simulations are obviously very expensive and difficult. And I think isn't a thing that models are most designed to do in their current form. So I would say that they probably would look a little bit different from the current kinds of training that people are doing. But if it's more along the lines of given a lot of context about a user, predict something interesting about them or predict the results of some intervention. I think this is something that models first are already getting better at. And second, could get a lot better at over time. And I think that would be potentially quite impactful. I think we'll start to reach a point where there'll be more and more clearer demonstrations of this. I think you're starting to see interesting and increasingly clear demonstrations of models doing interesting science and math in the public.

• 1:50:58Speaker 01

  I think you'll see a little bit more in this space as well. But I do think it'll take a little bit of time. But my view on paradigms is kind of that for you can get a lot out of extending and tweaking the current paradigms and you can in fact view everything that has happened so far as just one paradigm. But between pre-training and scaling reasoning style training, I think you can get quite a lot of juice. And I think even integrating additional modalities and things like this, I think doesn't require too much additional changes or tweaks. I think broadly, our team's mission today is do whatever it takes to ensure AGI is beneficial on human health for all of humanity. And so we see that happening through three channels.

• 1:51:41Speaker 01

  The first is helping consumers understand and navigate their health, which is already happening through ChatGPT Health. The second is empowering the health system and thinking about the ways in which AI assisted care can become part of the standard of care. And it can reduce the extent that clinicians are bogged down by paperwork and are spending more time thinking about the, seeing patients and thinking about the problems that matter most for improving care. The final thing is really pushing up the ceiling of research and I found your vision pretty inspiring. So what I would say is like, I think you can expect as a lot of the core problems in healthcare that we've talked about, the fragmentation of data, fragmentation of patient experience to improve, as those happen, you also, you should expect

• 1:52:25Speaker 01

  an acceleration of the application of intelligence to that data. And I think biology and health is one of the areas in which marginal gains in intelligence have the most obvious value in solving more and more problems for humanity. There are many examples of previous breakthroughs in biology where like, there was nothing stopping that breakthrough from happening five or 10 years earlier, except just more human ingenuity applied to it, but there's no physical blocker. And so when that's the case, then I think you can assume that long-running models, long-running agents connected to the right data can do really incredible things. And I think the hope is that in addition to our existing work, which has really been focused on how do we raise the floor of human health, we start to raise the ceiling of human health as well.

• 1:53:10Nathan Labenz?

  Yeah, it's going to be not just the AI doctor, but also the AI biomedical research scientist as well. Wow. Okay. It's an exciting time to be alive. I'm increasingly saying that at the end of these conversations these days. So yeah, anything else you want to make sure people are aware of that I didn't ask you about?

• 1:53:27Speaker 01

  Like I mentioned earlier, I think the right way to think about our work for health at OpenAI is really operating in three phases. One is really laying the foundation, and a lot of that is focused on our work on safety. And so examples of this are like our work on Health Bench, where again, we had this kind of evaluation, which you can run offline of large language model performance, safety and health. We had this study of the first AI clinical co-pilot with Panda Health. We had a bunch of model improvements over the last year, and all of these kind of lay the foundation for the work that we're doing today. The second thing is this kind of rise in adoption, which we've been seeing. We've been seeing an incredible rate of individual users, especially patients adopting technology. And we've seen hundreds of millions of people asking health questions a week, which has

• 1:54:12Speaker 01

  been a rapid growth since last year. The final thing is really the the future of scaling the impact of this work. And that's where really our work on ChatGPT Health, which is this kind of consumer facing product, which enables you to connect your health data with additional privacy protections. And also our work on ChatGPT for healthcare, which is again facing health systems and really enabling health professionals to use AI as a co-pilot in their workflows, kind of come in. And so between all these things, I'm really excited about the work that we're going to be doing to scale the impact of this work in the next year. And I'm looking forward to what's to come.

• 1:54:45Nathan Labenz?

  Karin Sigal, Head of Health at OpenAI. Thank you legitimately for all your hard work. It really is incredibly valuable, incredibly impactful. And that's obviously just going to continue to grow exponentially, along with so many things in the AI space. And thank you for being part of the cognitive revolution.

• 1:55:02Speaker 01

  Thanks for having me.

• 1:55:14Speaker 03

  Every generation try fi be the pain Every mother praying that her child remain But something in the wiring of the world Wake up a river of the no end start fi overflow the cup Move 20 sas and the ghost in a discan See one no doctor in a world can

• 1:55:32Speaker 03

  Doctor in a the machine Raise it from the floor Knowledge with the people who never had before Nairobi to Nashville we a kick down the door Baguette revolution and we coming fi more Raise the ceiling, raise the floor, oh oh oh oh Raise the ceiling, raise the floor, doctor in a the machine Oh oh oh, no even no be fit, oh oh oh, yeah Quarter billion people asking every week Finding words to system out but doctors couldn't speak Chain of thought in English like a river running clean Alignment wasn't programmed, it's the realest thing I've seen Kenya clinics on ride, under health alive Free intelligence, no gatekeep, no drive

• 1:56:10Speaker 03

  Doctor in a the machine Raise it from the floor Knowledge with the people who never had before Nairobi to Nashville we a kick down the door Baguette revolution and we coming fi more Raise the ceiling, raise the floor, oh oh oh oh Raise the ceiling, raise the floor, doctor in a the machine Oh oh oh, no even no be fit, oh oh oh Do no harm that alone once a fight When the people need an answer and I just cause hard advice Five bars, power, every cavity inside Universal basic intelligence does the ride

• 1:57:19Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast

• 1:58:01Nathan Labenz?

  dot I N G. And thank you to everyone who listens for being part of the Cognitive Revolution.