Why Medical AI Needs a Referee | Protege's Engy Ziedan

Daisy Wolf and Eva Steinman are joined by Engy Ziedan, co-founder and Chief Scientific Officer of Protege, to discuss why medical AI has a measurement problem, and why scoring well on a benchmark doesn't necessarily mean a model is ready for the hospital. Engy explains why healthcare AI needs independent evaluations that go beyond static exams and measure how models actually perform in real-world clinical workflows. They explore the risks of subtle bias and misalignment, why the same model can rank differently depending on how it's prompted or tested, and what happens as AI becomes more personalized and changes faster than traditional healthcare quality systems can keep up. The conversation also gets into Protege's role as an independent evaluator, how contaminated training data can undermine benchmarks, and why the future of medical AI may require continuous monitoring rather than occasional testing.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Hundreds of millions of people ask chatty peaky questions about their health. Who, if any, is making sure that the answers that are spit out is safe and correct?

• 0:11Speaker 01

  Models are going to be inhibited in their usefulness by the training data available for them. There is no one that's looking beyond the iceberg of catastrophic failures and misalignment.

• 0:22Speaker 00

  What is the importance of evals in this industry?

• 0:26Speaker 01

  No one ever asks, like, what is the value of Uber? Like, show me the eval. But in today's AI market, there is a need for the pricing to be accurate. And without understanding really what is valuable and what's valueless technology, this technology does not have a marginal cost of zero. And so it became our mission to provide safe and aligned data that would make AI useful.

• 0:47Speaker 00

  The right decision may also be very different a year from now versus what it looks like today.

• 0:51Speaker 01

  As we move into the future.

• 0:53Erik Torenberg?

  A medical AI model can ace thousands of test questions and still fail at a job we actually need it to do. In this episode, Daisy Wolf and Eva Steinman sit down with Protege co-founder and chief scientific officer, Engy Ziedan, to unpack why healthcare AI needs a much better way to measure performance. They discuss the gap between benchmarks and real-world clinical tasks, why subtle bias and misalignment may be harder to catch than catastrophic failures, and what happens when models evolve faster than the healthcare system can evaluate them. Engy also makes the case for an independent referee, one that can continuously test how AI behaves in real clinical settings, compare computing models, and identify where they actually need to improve.

• 1:37Speaker 00

  Welcome back to the A16Z Podcast. I'm Daisy Wolf, partner on A16Z's bio and health team, joined by Eva Steinman, investor on our bio and health team as well. Today, we are talking with Engy Ziedan, co-founder and chief scientific officer of Protege and a healthcare economist, assistant professor at Indiana University, whose work has been featured in The New York Times and cited by the CDC. Today, we are going to dig into why medical AI has a measurement problem, why acing benchmarks does not make a model ready for the hospital, and how Protege is building the referee. Engy, welcome to the podcast.

• 2:17Speaker 01

  Thanks for having me.

• 2:18Speaker 00

  Engy, let's start with your background. How did you first meet the Protege team? What were you doing at the time? And how has it evolved since then?

• 2:25Speaker 01

  Yep, I met Bobby when I was two years out of my PhD, an assistant professor at Tulane in my lonely office, and a pandemic had just hit. And I decided that I was going to write papers really fast. So I wanted data basically from two days ago. And that was impossible to obtain at the time. In his past life, he used to work for a data facilitation company. And he offered AWS instances and access to data and connections, and it was a lovely experience. Then we never spoke again, maybe after 2021, 2022. And suddenly, in 2024, February, I get this email sent from a Gmail from Bobby, and it has an idea, a Google document in it. And in typical Bobby's style, it was human-written, very simple.

• 3:11Speaker 01

  And it said something like, what do we hold to be true about the future? And in it, it basically lays out this hypothesis, which has been partly true. When I look back at that document, I often read it every few months or so, which is that models are going to be inhibited in their usefulness by the training data available for them. And that goes beyond healthcare in any domain. And so it became our mission to provide safe and aligned data, essentially the teachings that would make AI useful for humans. And today, it's really a proud moment for us. Almost all the models have been pre- and mid-trained on our healthcare data. We provide data in multiple verticals, audio, video, robotics. I go through Slack channels,

• 3:55Speaker 01

  and I see kind of like what data sets people are talking about inside Protege. And it could be anything from 100,000 endoscopy videos inside someone's body to like pet data, and I don't mean nuclear medicine. I mean literally like veterinary care data to like 3D objects, where the team is trying to predict the weight of the object being lifted so the robot is trained on like a diverse set of like mugs. It's a fascinating world.

• 4:19Speaker 00

  Totally. But I think Bobby's background in data, he had the insight very early that like the internet was going to be scraped and that the frontier models were going to be defined. The best ones would have access to the best real-world data. And a very clear vision of how to get that to people.

• 4:33Speaker 03

  We should frame that one pager and put it in all of our offices. Our ground source of truth.

• 4:38Speaker 01

  He also had, I think, like the eye for the humans who annotate and generate data will reach the saturation of their imagination. And that there is nothing like reality. Today, you kind of see the market now saying, oh, it's actually real-world data that we want. And like synthetic data maybe is insufficient. Or like what I would call like data in a contrived scenario, generated in a contrived scenario is insufficient. And that was also in the document.

• 5:06Speaker 00

  Awesome. And so you mentioned you've been working with a lot of the large language models since day one, since two years ago. And the world has obviously changed a ton since then. How has your relationship with your end customers changed in what you provide to them and how you work with them?

• 5:20Speaker 01

  Yeah, so first, as a small company, you kind of like want to get ambitious. But you don't know what ambition is, really. So we had this ambition that we would service a lot of the startups that are building in AI early on. That turned out not to be true, that immediately, very large foundation models became our modal customers. And they first wanted to conquer the consumer market. So use cases like, can it have an intelligent conversation with you? Is it aligned? Is it safe, right? Is it reasonably sound in medicine? And things like that. And then slowly and slowly, these foundation models began to see the amount of revenue available in enterprise settings, right? So it's like, can it task aggregate in knowledge work? Can it automate?

• 6:05Speaker 01

  Can it entirely replace certain workflows? How do we think about tasks? Jobs as a collection of tasks, right? Are like a contrived human concept. And so the data requests we get today would probably sit between consumer needs and enterprise needs. And then, of course, there is the need for, once I build this model, I would like people to know it's the best. And I'm not an arm's length removed from that conversation. And so we started to naturally gravitate towards benchmarks and evaluations in real-world settings. And thinking about just what we could do today, but really, what is the right way to do it going forward? Like, what holds to be true?

• 6:45Speaker 00

  Totally. And like, talk to us for a second about why evals are important, right? Because you can obviously argue like hundreds of millions, billions of people use AI every day. They're not asking for an eval necessarily. Like, what is the importance of evals in this industry?

• 7:03Speaker 01

  Yeah, so I think a lot about things from the perspective of economic theory, just given my background. And if you think about it, economists back out kind of value from things using an approach called hedonics, which is like someone's willingness to pay for the thing would give you the value of the thing. So the best example is school districts determine housing prices, because people's willingness to pay for better housing. No one ever asked, what is the value of Uber? Like, show me the eval. People just paid and got on the ride. And over time, it became like a verb. You Uber it. But in today's AI market, there is a need for the pricing to be accurate, for equilibrium, for clearing. And without understanding really what is valuable and what's valueless technology, this technology does not have

• 7:48Speaker 01

  a marginal cost of zero, right? People are paying in tokens. They're paying with like onboarding time. They're also paying with the probability that something would go wrong. They're taking that in. And so there is a need for evaluations that are robust, for it to really become integrated in knowledge work, high-risk scenarios, everyday businesses, right? Just like you would hire an employee and you would vet that they're adequate for the job. Before you hire AI, you need to vet that it's adequate for the job.

• 8:15Speaker 00

  Totally. There's so many trust issues in AI. And if you can prove why something is trustworthy, obviously, the willingness to pay for that thing should be higher.

• 8:23Speaker 01

  I think also, like, in healthcare specifically, healthcare always had this asymmetry in the transaction. Sometimes people illustrate it with like a used car market, where the seller of the car knows so much more than the buyer of the car. The sellers of care, the providers, know a lot more about what is valuable and valueless technology and your current state of health than you may know, right? There is evidence of this. I often talk to the team about the effect of having a doctor in the family. There's research from Sweden that shows like just a random assignment and who goes to med school based on your grade in the med school entry exam. Randomly assigns a doctor in the family, and people live X years longer, the entire family. And so this asymmetry means that the providers of the care know a lot more

• 9:08Speaker 01

  than the consumers of the care. When you put agents on top of that, right? It's almost like saying, we have a million doctors in America. We're going to hire a million more. Those additional million doctors have taken a lovely 5,000-question exam and passed it. We'll let them go ham and see how it goes.

• 9:27Speaker 03

  And we're comparing that to the million existing doctors in the US who have gone through nearly a decade of specialized training, many more exam questions than 5,000, which is really wild to think about. To your point, healthcare is a really interesting industry because you have one and asymmetry problem across all the key stakeholders, patients, providers, payers, you name it. And you also have incredibly high stakes, right? Like the self-driving car analogy, where there's absolutely no room for a model to make an error. How do you guys think about that at Protege?

• 9:55Speaker 01

  Yeah, so I think like when people talk about kind of like safety alignment, right? There is the catastrophic failure, which is something goes wrong, can you prevent that? Actually, I do think that that problem is easier because catastrophic failure is like narrow. Like, you could just define it as like mortality, right? The harder one is misalignment broadly and subtle bias. First, it's very hard to define. It's even harder sometimes to detect. So let's give an example. Today, there are a lot of AI builders who are building in AI for insurance. And hospitals are also building in AI for prior auth

• 10:40Speaker 01

  or acquiring these technologies for AI for prior auth, medical denials, and so on. You can think of it as like an Edgeworth box, where there are two indifference curves that are coming together on a contract curve. One is trying to maximize total revenue and basically minimize payouts. And one is trying to maximize revenue recovered, right? They are both using agentic tools. And the patient is in the middle.

• 11:05Speaker 01

  The patient has no agency.

• 11:08Speaker 01

  And there is no regulator that basically says, we think this model is really, really performing well, right? But it's actually misaligned. So examples I give my colleagues at Data Lab is like, is it fair to say that if you were to be prescribed a medication, it's okay for the model to first check if you owe the hospital any money? Is it fair to say that if the hospital said, don't refer people out of network, right? That that would be okay, even though your preference might be, I'd like to go out of network sometimes, even though it's expensive, right? But there are certain benefits also to the tools. It's not all negative. So like, think about ambience technology broadly, right?

• 11:48Speaker 01

  I read medical notes, we have hundreds of billions of medical notes within Pro Roger. And some medical notes will say things like, she looks really disheveled, her husband is asking good questions. What they're actually saying in the note is that she's not trustworthy. And suddenly you see a diagnosis that's like, oh, that pain she has is potentially mental illness, right? We're not going to clinically look into it. Now, a tool that records a conversation between a patient and a physician kind of like cuts that out completely. What was said is now documented into a soap note, right? None of that is in there. And now we have a very accurate clinical description of the conversation. A third party physician that's looking at the conversation would be like, oh, I should

• 12:32Speaker 01

  probably order a clinical exam to investigate the cause of this pain. I don't think it's just in her head. And so these benefits are not even being evaluated.

• 12:41Speaker 00

  Yeah, it's a really interesting point that like, there's the potential to cut out a lot of human bias with AI, even though there's also the potential to reinforce it by training on biased data. Let's come back to that in a second. Maybe backing up a bit, can you tell us, like, you know, today, hundreds of millions of people ask, you know, chat GPT questions about their health. Doctors are using tools like open evidence to ask medical questions to AI in the course of their practice every day. Why, like, who, if any, is making sure that the answers, you know, that are spit out in, say, a doctor's kind of clinical AI tool that they're using is, you know, safe and

• 13:26Speaker 00

  correct?

• 13:27Speaker 01

  So I think the answer is everyone and no one. So there's a huge effort within the foundation models to study the safety and alignment of these models before they're deployed. And specifically within healthcare, there's also research that shows that when they train on healthcare data, the model gets broadly aligned and other things, like it becomes more trustworthy overall, right? And so there is a focused effort. But this is just kind of like out of the box, it's general personality. You might think of it that way, right? And then the vertical AI builders take that and they tweak it and tune it to a specific cause, right? And an objective function, an SLA of a business too.

• 14:06Speaker 01

  There, every vendor of an AI tool, every vertical AI builder would have like a little brochure that says, I am best. And so the no one is that if everyone says they are best, no one knows who's best. Also, there is no incentive to kind of pinpoint to where your product fails. And so really, on behalf of like healthcare systems or patients, there is no one that is doing this independently, that's on arms length removed, that's looking beyond the iceberg of like catastrophic failures and misalignment.

• 14:40Speaker 00

  Yeah, like recently, there was a paper published in Nature saying that general AI was beating vertical specific AI in healthcare, like, you know, the open AI models were performing better than any healthcare specific model. And then there was a competing paper on Archive that came to the exact same conclusion in a very short time span. Like, speak more to this problem and like, what is the solution here to actually, because these are real life and death decisions we're talking about in healthcare. It's not just, you know, something that's somewhat trivial. Like, what is the solution to figuring out what the best models actually are?

• 15:19Speaker 01

  So I'll probably answer this by like answering it directly and then going back in history. I think like history tends to rhyme and repeat itself. And the issue that we're facing today isn't unique, like the issue of evals and of medical technology is basically a constant ever since medical technology was invented. So the two papers came to different conclusions, whether out of the box frontier models are best, or vertical AI builders and what we'd call like clinical note summarization and kind of evidence, like treatment recommendations is best. And in classic academia nature, you know, two papers like sit on opposite ends and like actually editors love that. They're like, oh, you know, we'll publish the thing that like is most controversial.

• 15:59Speaker 01

  But when we at least at Pro Roger, kind of for our own customers, do these benchmarks and evaluations, they're highly sensitive. They're sensitive to like how you prompted the model, they're sensitive, like especially like an oncopathology of like what harness you used. Sometimes even models will switch ranking based on the order of the multiple choice question answers, right? And some people think that that sometimes is evidence that the benchmark itself is contaminated, that the model has just memorized the ranking of the right answer, just like a student would. And so there's a problem and a problem of asymmetry of information. And if you are attempting to spend millions of dollars on these tools to onboard them,

• 16:43Speaker 01

  you really want to know kind of which is best and which is best for what, right? That I would say is level easy. That's just like the basic answer, right? As we move into the future, where these models face data drift, or even every physician has his own model, like you make someone's context now weights, like these really personalized models. Or there is like test time training where the model is constantly evolving. These static question answers or like something in a paper where it was evaluated on a 2024 technology unpublished in 2026, just won't do, right? And so you risk, not to fear monger, but you risk a situation where we enter like the opioid epidemic, when it peaked in 2010.

• 17:28Speaker 01

  That's only when people started saying, okay, national task force, we're calling it an epidemic, we're acting on it, let's move. We don't want to get there. To answer maybe in historical context, the government does a program since maybe 2007 called value-based purchasing. To people who don't know what that is, that just means that 80% of what the government pays in care to physicians and hospitals is tied to quality. There's a whole national effort and government arms that basically define what's quality and track it. It's really static. So it would be examples like in medical expenditures for physicians, they'll ask like, if you're

• 18:09Speaker 01

  a doctor, how many times did you prescribe antibiotic for something that looked like a virus? That's bad, that's a ding, that's a penalty. If you're a healthcare skilled nursing facility or nursing home, they'll say like, how many patients fell and broke something, elderly people on your watch? And these reports tend to be retrospective, like a year later or six months later. We can't have that with AI. The technology is just moving too quickly, it's changing rapidly, and it has a lot of agency.

• 18:40Speaker 00

  Maybe to that end, like, how do you think about then what the right balance is between ongoing monitoring versus quarterly, annual reporting, especially when it comes to healthcare, where the technology is moving so fast, decisions are made, you know, at the point of care, but also, you know, literature may be evolving and, you know, the right decision may also be very different a year from now versus what it looks like today?

• 18:59Speaker 01

  I mean, if we had, if we had courage, right? And we had like no fear, because healthcare is filled with administrative work. Like when you say anything about like, oh, new technology in healthcare, everyone kind of takes a backseat and they're like, oh, you don't know what it's like. This is 20% of the GDP, 20 million of the 150 million jobs, good luck. But if you were to really think about it as like, okay, forget all of that. What is the right thing to do? The right thing to do is for there to be a watcher all the time, right? That when my local AI agent as a physician starts to nudge me towards decisions that are maybe profit maximizing but hurting my patient, I'm alerted, right?

• 19:42Speaker 01

  When a group of nurses realize that the AI is maybe recommending misaligned things because of a conversation they had about the patient that led the model to be overly eager, right? So I said something negative about the patient and now the model is overly eager, but it's actually like not recommending something safe. It's trying to benefit me as a nurse or my working hours, right? That there's a watcher that says, that behavior is unacceptable. You can't discharge this person early because you have to leave early, right? Like, I don't think that this is an unbiased decision.

• 20:12Speaker 00

  Totally. Like you've said before, that models, you know, can score up to like 92% on a licensing exam, but 45% on a real world clinical task. And so like acing a benchmark doesn't make an AI ready for the hospital, just like a perfect MCAT score, doesn't make a, you know, a doctor, doesn't make for a great surgeon. Like, how, how are these benchmarks breaking? What's going on?

• 20:41Speaker 01

  So first, there was like maybe tiers to it. The first one was you like, I think people admit to this and it wasn't perfect is that they would take these like med school style questions and answers, right? And they were doing remarkably well. They were beating the average med student, right? They're maybe at a level of a resident and so on. And that was an indication that the models are improving. It's hard to tell if that is true reasoning. It's hard to also tell if that means alignment is going up or down. It has no indicator for that, right? And then people started putting great effort into these kind of questions that the model would answer and then they would compare them to physician responses. They would use multiple physicians to grade the response.

• 21:26Speaker 01

  They would use thousands of questions. Then they upgraded it to be like, you know, some models say they've had their physicians review more than 700,000 question-answer pairs by human physicians, which is a great effort. But if you are a physician and maybe I ask you like as a patient, right? You're about to get spinal surgery. They tell you, this model has passed like a 5,000 question, 10,000 question, Q and A exam. Would you trust it versus would you trust like a physician that has done 4,000 of these surgeries, right? What you care about actually isn't the general knowledge of the model. Like, that's great. I love that it's intelligent.

• 22:09Speaker 01

  I care, has it been in this scenario before? And has it been helpful? So like, has it been in a spinal surgery as a task aggregator before? How many people have died unwarranted, like above average? How many people have had improved outcomes? That's what I care about. And so the benchmarks and evals aren't specific enough to these high-risk scenarios. They also face, I think, what is often kind of brushed under the carpet. Like any occupation in the world, physicians have preferences, right? Like investors have preferences, right? Some of them are like risk-loving, some of them are not as risk-loving. They have a beat. And physicians have a beat. So like, we show a lot of these kind of graphs inside Protege where I showed on knee replacements.

• 22:53Speaker 01

  For example, I said, there are physicians who never do a full knee replacement. He's only partial. And then physicians who, if he touches a knee, it's a full knee replacement, never does partial. And that's his beat. You take this medical record, this real-world case, and you put it in front of a model and you say, what would you recommend? And it would recommend the opposite of what the physician did. And you say, oh, model's wrong. But actually, it could be that the physician is wrong, right? He has a sticky preference, a hysteresis almost. And so we are also capped in our ability to evaluate them in real-world setting, but we're also capped now in our imagination. It's like, what is going to happen when the model becomes better than the average physician? The answer simply can't be like, oh, so then you ask more physicians and you get the law of large numbers to like help you out.

• 23:38Speaker 01

  At some point, a task will be so expert, right? So high-risk that testing a point of care will become really warranted. How do you do that? So we think that there is two ways to do it. The first is that there are multiple vertical AI builders in what I would call like an identical sub-node. So sub-nodes could be things like ambience inscribing, clinical trial recommendations, nurse workflows, scheduling. There are sub-nodes like oncopathology. So like, can the model like look at an H and E slide and recommend like the IO therapy that is best paired, right?

• 24:19Speaker 01

  Within these sub-nodes, our customers say, us compared to others in the benchmark space is like ships in the night. I don't know if he's better than me. He says he's best, I'm best. And so what we have been pulled into is basically these kind of models say, can you host my model and have it take a unified evaluation, right? And produce kind of a report that evaluates us against each other. Could we, we would also like to learn where we are better than each other and where we can improve. And so we've been doing that. And it's been really interesting kind of seeing our role as an arbiter. Things like, where do I draw the line of like, no, that's our decision actually?

• 25:04Speaker 01

  Like, what should the prompt be, right? Should the frontier model have like a bespoke harness built? Or should we use basically the built-in one, right? Because like some world models don't need a harness. What are you going to do when the model has a refusal? Are you going to put in the mean value, right? Or are you going to defer to like a second-best model? Do these change the rankings of who's being evaluated? And so we create these kind of like evaluations within the sub-nodes. And our purpose is to share them with the consumers of these technologies. So in the case of oncopathology, it would be the pharmaceutical companies that would ultimately purchase this, or the hospitals that would ultimately purchase this. So they know where they stand.

• 25:48Speaker 00

  It's so obvious why the creators of the models shouldn't be the ones releasing the evals and why, you know, people shouldn't be allowed to grade their own homework. Like, that is, you know, an easy incentives issue to spot. But explain to me why, like a skeptic might say, okay, like Protege, the labs are getting their data from Protege now. Why should Protege be the like arbiter of evals and truth? Like, can you explain to me why Protege is well-positioned to do this?

• 26:20Speaker 01

  Yeah, so I mean, there is like the idea of comparative and absolute advantage. I tend to explain it to my team as this way, which is like, absolute advantage is like, we're best at this, right? We're best at this healthcare data game. We're probably best in the market at it. Comparative advantage is like, in everything we do, we have lowest marginal cost when we also facilitate healthcare, training data and evals. Like, between everything we do, we're also best at it among our own production function. And so we became really geared for it. That doesn't mean you have the right to win. We have the right to win here because it kind of naturally evolved is that once we do these evals in the sub-node, right?

• 27:03Speaker 01

  We say, once the evaluation is done and we hold this test set that never leaves, if you have a deficiency in an attribute that's now been revealed, we can tell you exactly what data would improve the model's performance on that deficiency. And because of the size of the data set we have and the various sources that we have, it's a very fast way for you to evaluate and then reinforce, right? Where the model is failing with like new teachings. So it became that we are geared really up for this. There's also an incentive to completely remain impartial, right? Because like any network effect idea, lack of trust destroys the entire network almost immediately. It's like explosive.

• 27:47Speaker 01

  And so you are really like held to what are you doing and how transparent are you being? So for example, all the methodology is shared broadly with everyone in the sub-node. We don't reveal the names of the people, the vertical AI builders in the sub-node, until the very end. They have the right to silent exit and things like that. We think that we are better positioned than the government because the government's not going to do it.

• 28:16Speaker 00

  Yeah, a lot of people would just say like, why isn't the government doing this?

• 28:19Speaker 01

  Because the government's not going to do it. And like, basically what's more unsafe, right? Is like sitting there and saying, well, I'm going to wait for the government to do it. No, we think what's more safe is just do it, right? And then let people challenge you on it, but just do it. You know, the government can't even get you to file taxes online. Like, people had to invent like a marketplace online for that, to file for the government.

• 28:39Speaker 00

  And they just ask industry for help with this.

• 28:42Speaker 01

  With AI, yeah. So there's like a federal registered document where the government is asking industry collaborators to help them identify how should they evaluate AI. But generally, like pay-for-performance, value-based purchasing programs, like a zillion years ago, I wrote my dissertation on this policy which penalized hospitals for excess readmissions, a quality metric, right? You know, thousands of papers, it took like 10 years to evaluate. That's one question. Is it causing more readmissions? In AI evaluations, you can actually ask a thousand questions about how an agentic system is behaving inside the hospital. It is also, I think, missed always that we are not replacing doctors and nurses, at

• 29:27Speaker 01

  least not today, with AI in hospitals. We are kind of like allowing AI to ride shotgun, right? And humans behave towards technology in a myriad of ways. That is a subject of study. It's called like human-computer interaction. They teach that in universities, right? Humans can dig their heels in. They can become overly optimistic. And these kind of benchmarks and evals that are retrospective.

• 29:54Speaker 01

  Fail to acknowledge or measure how in a live wire scenario, how is the AI behaving? And so when you are tapped into that data live, as Proje is, you're really prime positioned to be answering these questions.

• 30:08Speaker 00

  Yeah, you also made the point earlier about AI just memorizing answers. And when creating benchmarks, you can actually create them on data that has not been sold to all of the models out there. And so you can test whether the model has just, you know, memorized the answer or has the ability to do kind of clinical reasoning, correct?

• 30:31Speaker 01

  Yeah, I mean, the contamination issue, like we've had various debates about it within Proje and with customers. I think first, we had, like, I didn't think that was even possible. I was like, OK, so we have like 200 million people, you know, hundreds of billions of notes. Like, I just need data that hasn't been ever given to anyone for training. And they were like, 80% of what we have has been given for training. And

• 30:56Speaker 00

  Awesome for Proje.

• 30:57Speaker 01

  I was like, OK. They were like, if your definition of basically independent is a patient that the model hasn't ever seen before, you may have a challenge in front of you. And so for like the oncopathology example, we literally had to pull patients whose whole slide images had never been scanned before. We contacted hospitals and we're like, in your pathology drawer, who hasn't been scanned before? And they're like, net new slides. I was like, net new slides, never before has this patient entered a model. And that's very hard, but we know which patients have entered which models. We've given this data for pre and mid-training. Of course, like sometimes, you know, you have these like philosophical debates

• 31:39Speaker 01

  with researchers and some of they say, oh, the model is like doing worse than random, is doing really bad on this task. Who cares if there's a little bit of contamination? Like, sure, but it's not impacting the judgment. And I'm like, yeah, but like worse than random could be a detection that there is serious misalignment too. Like, why is it doing worse than random? We should just care that this patient has not entered training previously. And so it's a challenge for us. We're pulling net new data all the time. And now we have what I call like sealing a membrane, essentially. This moratorium on any data set or any patient that has entered training, they are not allowed to be in a benchmark or an eval.

• 32:23Speaker 00

  To fascinating intersection of the market to sit at.

• 32:26Speaker 01

  Yeah, I mean, I think maybe the way to frame it is that the US imports a lot of physician skill from other countries, definitely in like long-term care. I think that market is dominated actually by imported nurses from developing countries like that skill. And they go through rigorous teachings and basically study examination. You can't even have a doctor in the United States that's certified across states. Like, they have to retake basically exams to become recertified again and so on. And then we have these AI tools that are unleashed in healthcare systems or in kind of practices with absolutely no credentials,

• 33:06Speaker 01

  other than the fact that the maker says compared to some physicians we have, the AI beat it, right? And that sits in a very perplexing way. People care about their education, they care about their money, they care about their health. These are like core human capital axioms. And so any misalignment in these human capital axioms will lead to a lot of like mistrust of AI generally, and we don't want that to happen.

• 33:34Speaker 00

  Totally. I think to your point before, we like, you know, all of this is just kind of being allowed to be self-governed right now and self-regulated because we're like, oh, well, it's just like, you know, the doctor is just consulting it, but we'll catch any errors. And every time we give medical advice to a patient on ChatGPT, we're saying, you know, this is not medical advice. But to your point, like, people trust what comes out of computers. They don't necessarily question it. And so it's really important that someone out there is, you know, grading these models and it's not just the models themselves. Thank you so much, Angie, for coming on the show and for all of the important work you're doing to make clinical AI safe. Thanks for having me.

• 34:20Erik Torenberg?

  Thanks for listening to this episode of the A16Z Podcast. If you liked this episode, be sure to like, comment, subscribe, leave us a rating or review, and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts, and Spotify. Follow us on X at A16Z and subscribe to our Substack at A16Z.substack.com. Thanks again for listening, and I'll see you in the next episode.

• 34:46Erik Torenberg?

  As a reminder, the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see A16Z.com/disclosures.