Bad Medicine, Part 2: (Drug) Trials and Tribulations (Rebroadcast)

How do so many ineffective and even dangerous drugs make it to market? One reason is that clinical trials are often run on "dream patients" who aren't representative of a larger population. On the other hand, sometimes the only thing worse than being excluded from a drug trial is being included.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:01Stephen J. Dubner?

  Freakonomics Radio is sponsored by Progressive. Whoever says saving money is hard doesn't know that new customers who bundle and save with Progressive save hundreds because switching your home and auto insurance is better than cutting down on the things you love, like those iced lattes that help you get ready to face the day. So switch your home and auto to Progressive and see if you could save and keep sipping away. Get your quote today at progressive.com. Progressive Casualty Insurance Company affiliates and other insurers. Not available in all states. Freakonomics Radio is sponsored by PNC Bank. Boring. Great word, bad rep. Yet, time and time again, here at Freakonomics, we find that when something culture-changing happens,

• 0:43Stephen J. Dubner?

  boring things like calculations and careful planning are usually behind it. Which begs the question, why not apply it to other aspects of life, like banking, for example? Well, PNC Bank did since its inception, proving time and again that the best banking strategy isn't the exciting type, but the steady, reliable, definitely not YOLO type. PNC Bank, brilliantly boring since 1865. Brilliantly boring since 1865 is a registered mark of the PNC Financial Services Group, Inc. PNC Bank, National Association, Member FDIC. Freakonomics Radio is sponsored by hotels.com. Hotels.com helps you find and book the right

• 1:25Stephen J. Dubner?

  stay for work faster. With business preferences, save your travel must haves, like gym access or breakfast, just once, then get tailored results whenever you search for work travel. Plus, with a free membership, you can save up to 20% on hotels and earn rewards on every stay for future travel. Book now at hotels.com. Visit hotels.com for details.

• 1:54Stephen J. Dubner?

  Freakonomics Radio is sponsored by ChatGPT. ChatGPT Work is a new way to work in ChatGPT. ChatGPT Work can pull info from across your apps and connect to your tools to help you create finished work. You can even automate work in ChatGPT with the Scheduled Tasks feature, and the whole time, you stay in control. Put ChatGPT to work on your most ambitious ideas and projects. Get started at chatgpt.com by selecting Work Mode, available on Plus and Pro plans.

• 2:29Stephen J. Dubner?

  Hey there, it's Stephen Dubner. Last week, we brought you part one of the rebroadcast of our Bad Medicine series. Today, part two. It's called Drug Trials and Tribulations, and it starts right now.

• 2:49Stephen J. Dubner?

  In the mid-20th century, an exciting new drug hit the market.

• 2:53Speaker 01

  It's a small molecule that was produced in West Germany in the late 50s and early 60s.

• 3:00Stephen J. Dubner?

  It was a sedative, but not a barbiturate, so it wasn't addictive, didn't clash with alcohol or other drugs, and, according to its manufacturer, was entirely safe. They based this claim on the fact that no matter how much of it they fed the lab rats, the rats did not die. Once this new sleeping pill was made available, doctors discovered it did more than help people sleep.

• 3:23Speaker 01

  It would combat, for pregnant women, morning sickness.

• 3:26Stephen J. Dubner?

  And so, pregnant women all over the world were given the drug. It was called thalidomide.

• 3:32Speaker 01

  The problem was the thalidomide would actually cross the placenta and impact the baby, and it would cause a whole series of malformations and probably a lot of fetal death.

• 3:45Stephen J. Dubner?

  Fetal deaths were thought to number at least 10,000. Among the babies who survived, there were serious birth defects.

• 3:53Speaker 01

  Children that survived were deaf and blind, had a number of disabilities. They had shortened or lacked limbs.

• 4:02Stephen J. Dubner?

  Babies born with horribly malformed limbs, with missing or malfunctioning organs, because of the putatively "super-safe" drug their mothers took to prevent morning sickness. Thalidomide was on the market for roughly five years before it was banned. Its German manufacturer, Chemie Grunenthal, first denied the disastrous side effects before ultimately accepting blame. The history of medicine is full of tragic missteps, but thalidomide, coming as it did during a boom in global mass media, made more noise than most.

• 4:36Speaker 02

  The problem of tighter controls to prevent the distribution of dangerous drugs such as thalidomide is a matter of concern to the president at his news conference.

• 4:45Speaker 03

  Concern over the tragic effects of the new sedative thalidomide prompts President Kennedy to...

• 4:51Speaker 04

  Already, more than 7,000 children have been born with some or all of their arms and legs missing.

• 4:56Speaker 05

  Every doctor, every hospital, every nurse has been notified.

• 5:00Stephen J. Dubner?

  Although a few million thalidomide tablets had been distributed to doctors in the United States for trial use, it was never approved for sale here. That was thanks to a doctor at the Food and Drug Administration named Frances Oldham Kelsey. She did not believe that the application from the American distributor offered complete and compelling evidence of the drug's safety. President Kennedy later hailed Dr. Kelsey as a hero.

• 5:26Speaker 06

  The alert work of our Food and Drug Administration, and particularly Dr. Frances Kelsey, prevented this particular drug from being distributed commercially in this country.

• 5:37Stephen J. Dubner?

  Even though the US was an outlier in blocking thalidomide, the disaster had a number of lasting effects on American drug regulation. For one, the FDA established much more stringent rules for drug approval. It also rewrote the rules on what kind of people should be included in clinical trials.

• 5:56Speaker 01

  Because of the effects on young women and on the fetus, it suggested that women shouldn't be included in clinical trials because of the potential adverse events to the fetus.

• 6:07Stephen J. Dubner?

  Meaning, women were summarily excluded from early clinical trials for new drugs.

• 6:14Stephen J. Dubner?

  On one level, this might make sense. It's a protective impulse. But this impulse had a downside.

• 6:20Speaker 01

  The study of women in general became part of the collateral damage of that pregnancy conversation. So there certainly are young women who are not pregnant, who could be included in clinical trials, and women in general could be included in clinical trials, to really understand some of the effects of drugs on their own health. And they were labeled as broadly vulnerable because of the potential to become pregnant. And I think that was part of a very rapid response to a very, very visible tragedy.

• 6:52Stephen J. Dubner?

  We see this all the time. Something terrible happens, and we rush to introduce laws, or regulations, or just mores that respond to the terrible thing, but often wind up over-correcting. Think about the Three Mile Island nuclear reactor accident in 1979. No one was killed, and the lasting health and environmental effects were negligible. But it was so frightening that it essentially killed off the nuclear power expansion in the US. Even as other countries embraced nuclear as a relatively clean and safe way to make electricity, often using American technology, by the way, the US retreated.

• 7:36Stephen J. Dubner?

  What'd we do instead? We burned more and more coal to make electricity. Now, from an environmental and health perspective, coal is almost indisputably worse than nuclear. But that's where the correction took us. That's where the fear took us. And the fear of another thalidomide took us to exclude most women from early-stage drug trials, and also to under-represent women for a time in Phase II and III trials, even if the drugs market included women.

• 8:09Stephen J. Dubner?

  And, as you'll hear today on Freakonomics Radio, that had some severe unintended consequences.

• 8:16Speaker 01

  It's just heartbreaking to know that so many women had to wake up in the morning, and they still got up, but they went out and drove into the side of a mailbox because we didn't have sex as one of the variables that we would study.

• 8:31Stephen J. Dubner?

  Also, when the only thing worse than being excluded from a medical trial was being included.

• 8:37Speaker 07

  The use of vulnerable populations of African Americans, people in prison, children in orphanages, vulnerable populations like these have been used for medical experimentation a fairly long time.

• 8:51Stephen J. Dubner?

  And what happens when a new class of drugs comes to market with great clinical trial results?

• 8:58Speaker 08

  But none of them have got evidence showing that they reduce your risk of heart attack, or renal failure, or any of the actual real stuff that patients actually care about.

• 9:19Speaker 09

  From WNYC Studios, this is Freakonomics Radio, the podcast that explores the hidden side of everything. Here's your host, Stephen Dubner.

• 9:33Stephen J. Dubner?

  This is the second episode in a three-part series we're calling Bad Medicine.

• 9:39Stephen J. Dubner?

  It's about the many ways in which the medical establishment, for all the obvious good they've done in the world, has also failed us. Last episode, we talked about how much we still don't know from a medical perspective about the human body. I would say maybe 30%, 40% that we don't know. We talked about the fact that medicine hasn't always been, and often still isn't, as empirical as you might think.

• 10:03Speaker 10

  You know, medical practice was based on bits and scraps of evidence, anecdotes, bias, preconceived notions, and probably a lot of psychological traps.

• 10:12Stephen J. Dubner?

  We went over some of medicine's greatest hits and its worst failures. You take a sick person, slice open a vein, take a few pints of blood out of them, and think that that was a good thing. On any list of medical failures, thalidomide is near the top. Although we should point out that long after it was found to have disastrous side effects on pregnant women, it's had a productive renaissance. Thalidomide and its derivatives have been used to successfully treat leprosy, AIDS, and multiple myeloma. That said, its effect on pregnant women, as we heard, contributed to women being excluded from many drug trials. Thalidomide and another good-seeming drug that went bad, called DES.

• 10:54Speaker 01

  Diethylstestrol, or DES, was manufactured in the early part of the 1900s.

• 11:00Stephen J. Dubner?

  That's Teresa Woodruff, who's been telling us the thalidomide story.

• 11:04Speaker 01

  I'm the Watkins Professor of Obstetrics and Gynecology at Northwestern University.

• 11:09Stephen J. Dubner?

  Woodruff also founded and directs the Women's Health Research Institute at Northwestern, and she's an advocate for something called onco-fertility.

• 11:17Speaker 01

  It's a word that was coined only about 10 years ago. So what we did was to bring together both oncologists and fertility specialists. So many young people are surviving that initial diagnosis of cancer that we've really converted it over the last 20 years from a death sentence to a critical illness. Many of the young people will actually survive that initial diagnosis and live long lives. And so when they return from that cancer experience, many of them are sterilized by those same life-preserving treatments. And so we want to provide fertility options to both males and females.

• 11:58Speaker 01

  So we developed not only the corridors of communication between oncology and fertility, but we also created new technologies that could provide new options for young women and for pediatric males and females.

• 12:13Stephen J. Dubner?

  So for Teresa Woodruff, as for many in the medical community, the future holds great promise. But so many decisions are informed by mistakes of the past. Like thalidomide and DES, which first became available in the 1930s.

• 12:29Speaker 01

  So DES, it's an estrogenic compound, was being prescribed to pregnant women to prevent miscarriage. Miscarriage was thought at the time medically to be caused at some level by low estrogen. And so supplying this estrogenic-like factor was thought to correct a really difficult problem.

• 12:51Stephen J. Dubner?

  Makes perfect sense.

• 12:53Speaker 01

  Yeah.

• 12:53Stephen J. Dubner?

  Was miscarriage in fact caused by an estrogen shortage?

• 12:56Speaker 01

  It's probably not. It's multifactorial. There may be some cases where low estrogen would have a modest effect, but in general, that's not the case.

• 13:06Stephen J. Dubner?

  DES, as it turned out, wasn't very effective in preventing miscarriage. Worse yet, it sometimes produced side effects that would become manifest only years later in the offspring of the women who'd taken DES. It affected boys and especially girls.

• 13:21Speaker 01

  Well, the physicians just started reviewing the medical records of these young women who were now coming up with this very, very rare vaginal cancer. The onset of that disease is clearly estrogen-dependent, probably a very narrow window during pregnancy when estrogen would have that effect. You know, DES and thalidomide are both tragedies, but it wasn't that the physicians were going out to try and create an adverse problem for women who were pregnant. But as you look back across medicine, across science, we're always learning.

• 13:53Stephen J. Dubner?

  In 1977, because of the tragic consequences of DES and thalidomide, the FDA made a big change. It recommended excluding from early clinical trials all pre-menopausal females capable of becoming pregnant unless they had life-threatening diseases, which meant that many of the drugs that later came to market had been tested only on male subjects, which could cause some real trouble for women.

• 14:22Speaker 01

  A great example of this is the drug Ambien, which was just the latest of the large number of drugs that had adverse events in females.

• 14:31Stephen J. Dubner?

  Ambien is a sleeping pill whose main ingredient is a drug called zolpidem. Americans love their sleeping pills. About 60 million prescriptions are written each year for roughly 9 million people. Some 2/3 of these medications contain zolpidem, which was approved by the FDA in 2007. But as it turned out, men and women metabolize the drug differently.

• 14:54Speaker 01

  The drug maker actually have in the FDA filing the metabolism of this drug in males and females. And in fact, knew that it cleared the circulation of males faster than it did females. But they only studied the efficacy on males, had no females in that efficacy study.

• 15:13Stephen J. Dubner?

  When you say the clearance, it means how quickly the body is metabolizing?

• 15:17Speaker 01

  Yes.

• 15:17Stephen J. Dubner?

  Yes, essentially?

• 15:17Speaker 01

  That's right. How long that drug is available in the body.

• 15:20Stephen J. Dubner?

  Can you just explain that a little bit? Let's say it's a 150-pound male and 150-pound female. I assume those will be different clearance rates. Can you explain why that is?

• 15:30Speaker 01

  Right, so it's going to depend on individuals. And so some drugs will go into the fat and will be available for longer. So how much fat exists? And what kind of fat can take up some of the drugs? But probably the most important part of drug metabolism is the liver. And so males and females have different enzymes and different P450s that are on the liver. And so that can alter the way drugs get cleared. For example, you know, women wake faster from sedation with anesthetics. So they recover much more slowly and have more reported pain events in hospital.

• 16:07Stephen J. Dubner?

  Talk for a few moments about the differences between females and males in medicine and/or medical science.

• 16:15Speaker 01

  Well, so I think one is hormones. And that's what we often think about. Males have testosterone and females have estrogen and progesterone. And so those hormones influence a lot of the biology of males and females in a very distinct and different way. But the fundamental way males and females differ is that every cell in a male's body is XY and every female cell is XX. And the sex chromosomes actually also inform, just like the other chromosomes within the cell, the overall function of that particular cell. And so understanding how chromosomal sex informs the biology of kidney cells or of eye cells or of muscle cells

• 16:59Speaker 01

  is really important. In addition, there are anatomical differences between males and females. So heart size might differ. And that's relevant to cardiovascular disease. And then the environment, the microbiome. We now know from a variety of studies that there is a sex to the gut microbiome that inhabits all of us.

• 17:21Stephen J. Dubner?

  So I would think, therefore, if I want to be a doctor or medical researcher or running the FDA or anywhere up and down the ladder, I would like to think that for the past 100, if not for the past 1,000 years, I've been very careful to consider any treatment and how different people would accept it differently based on their biology.

• 17:43Speaker 01

  Right. And I think that's the surprise to everyone, that in fact, sex has not been a fundamental part of the way we look at biological systems. And, you know, at some level, this is just the way biology has always been done. And then science keeps building on what was done in the past. And I think that's the really critical question. Are there real adverse events that occur when you only use one sex? And the answer is, of course, yes. So something like eight out of the last 10 drugs pulled from market by the FDA were because of this profound sex difference.

• 18:24Stephen J. Dubner?

  In the case of Ambien, the FDA was getting complaints for years from users who were sleepwalking, even sleep driving.

• 18:31Speaker 01

  It's just heartbreaking to know that so many women had to wake up in the morning, and they still got up, but they went out and drove into the side of a mailbox because we didn't have sex as one of the variables that we would study.

• 18:47Stephen J. Dubner?

  Eight hours after taking an Ambien, 10 to 15% of women still had enough Zolpidem in their system to impair daily function compared to 3% of men. The FDA's ultimate recommendation? Women should take a smaller dose than men. The federal government had acknowledged for years the problem of excluding women from medical trials. In 1993, Congress required that women be included in all late-stage clinical trials funded by the National Institutes of Health, unless it was a drug taken only by men. But what that didn't do-

• 19:20Speaker 01

  What that didn't do was include males and females in the animal studies and the cell studies that are the precursor to all, that's the engine to all of medicine.

• 19:32Stephen J. Dubner?

  Meaning, drugs that might be useful for women, but not for men, might not even get to the earliest stages of testing.

• 19:39Speaker 01

  It wasn't that they were thinking, "Well, let's make it hard on women to have this drug down the line." I think they were thinking of trying to do the most clean study they could imagine, and the study group that they imagined were the simplest, were the males.

• 19:54Stephen J. Dubner?

  Males were considered simple because they don't have menstrual cycles that change hormone levels. They don't get pregnant. They don't go through menopause. As one researcher puts it, "Studying only men, quote, 'reduces variability and makes it easier to detect the effect that you're studying.'" But ultimately, the exclusion of women was deemed inappropriate. In 2014, the NIH spent $10 million to include more women in studies, and in 2016, they decreed that all studies had to include sex as part of the equation.

• 20:24Speaker 01

  And that date, January 25, 2016, to me, there's a before and there's an after. And before that time, sex wasn't a variable in the way time, or temperature, or dose has always been. And I think we're going to see an enormous number of new discoveries, simply because science now has an entirely new toolbox to work with.

• 20:47Stephen J. Dubner?

  So, that's progress. But, as we'll hear later, drug companies still like to use very narrow populations for their drug trials, the better to prove efficacy, of course. So, exclusion still exists. On the other hand, it wasn't so long ago that exclusion from a certain kind of medical trial would have been a blessing.

• 21:08Speaker 07

  The use of vulnerable populations, of African Americans, people in prison, children in orphanages, vulnerable populations like these have been used for medical experimentation a fairly long time.

• 21:24Stephen J. Dubner?

  That is Evelyn Hammonds, a professor of the history of science and African-American studies at Harvard, and this is Keith Weilu, an historian at Princeton.

• 21:32Speaker 11

  You see it in the era when the birth control pill is being tested in Puerto Rico in the 1950s, and you see it in things like the Tuskegee syphilis study, which extended from the 30s into the 1970s.

• 21:48Stephen J. Dubner?

  The Tuskegee study of untreated syphilis in the Negro male, as it was called, is one of the most infamous cases in US medical history. Its goal was...

• 21:58Speaker 11

  Trying to understand the long-term effects of venereal disease as it developed through its various stages. And the study was being conducted on a group of really poor African-American men.

• 22:12Speaker 07

  White government doctors working for the US Public Health Service found approximately 400 African-American men presumed to all have syphilis.

• 22:22Speaker 11

  The problems emerge after penicillin is discovered and more widely used. And the question that should have been asked is, now that we have a series of effective treatments for venereal disease, ought we to continue a study of untreated syphilis? Or ought we to provide treatment?

• 22:43Stephen J. Dubner?

  So even though a syphilis treatment became available, it was withheld from the men in the study. Put aside for a moment the short-term elements of this maneuver, the cruelty, the ethical failure, consider the long-term implications. What happens when one segment of the population is so willfully exploited by the mainstream medical establishment? Well, that part of the population might develop a deep mistrust of said establishment. A recent study by two economists found that the Tuskegee revelation seriously diminished African-Americans' participation in the healthcare system. They were simply less willing to go to a doctor or a hospital. The result?

• 23:26Stephen J. Dubner?

  A decrease in male African-American life expectancy of about 1.4 years, which at the time accounted for roughly 1/3 of the life expectancy gap between blacks and whites.

• 23:44Stephen J. Dubner?

  Coming up on Freakonomics Radio, with such a fraught history of inclusion and exclusion in medical studies, who does end up in clinical trials?

• 23:54Speaker 08

  When you look at the evidence, what you often find is that trials are conducted in absolutely perfect dream patients.

• 24:01Stephen J. Dubner?

  Also, how good are the new drugs that typically make it to market?

• 24:06Speaker 10

  I think if we're honest with ourselves, we'll have to admit that the majority of new cancer drugs offer sort of very small gains at tremendous prices.

• 24:15Stephen J. Dubner?

  And what happens if you write about conflicts of interest among oncology researchers and then you go to an oncology conference?

• 24:25Speaker 10

  I always wear a bulletproof vest.

• 24:38Stephen J. Dubner?

  Freakonomics Radio is sponsored by Granola. You know that moment when a meeting ends and someone says, "Let's circle back on this next week," and everyone nods, but no one writes it down? Then next week rolls around and it's radio silence? That's why Granola exists. Granola is an AI-powered notepad built for the way real people actually meet. You take rough notes like you normally would, and in the background, Granola securely transcribes the meeting. Then it turns everything into clean, structured, actually useful notes when the meeting ends. And the best part? Granola works through your device's audio, which means it integrates seamlessly into the video conferencing tools you already use. No setup, no awkward bots. It's just your normal meeting with superpowers. If meetings are eating up your day, Granola is a no-brainer. You can try it totally free.

• 25:23Stephen J. Dubner?

  Just head to granola.ai/freakonomics. That's granola.ai/freakonomics to get your time back. Try it for free at granola.ai/freakonomics.

• 25:41Stephen J. Dubner?

  Freakonomics Radio is sponsored by Progressive. Whoever says saving money is hard doesn't know that new customers who bundle and save with Progressive save hundreds because switching your home and auto insurance is better than cutting down on the things you love, like those iced lattes that help you get ready to face the day. So, switch your home and auto to Progressive and see if you could save and keep sipping away. Get your quote today at progressive.com. Progressive Casualty Insurance Company, affiliates and other insurers. Not available in all states.

• 26:13Stephen J. Dubner?

  Freakonomics Radio is sponsored by PNC Bank. Boring. Great word, bad rep. Yet, time and time again, here at Freakonomics, we find that when something culture-changing happens, boring things like calculations and careful planning are usually behind it. Which begs the question, why not apply it to other aspects of life? Like banking, for example? Well, PNC Bank did since its inception, proving time and again that the best banking strategy isn't the exciting type, but the steady, reliable, definitely not YOLO type. PNC Bank. Brilliantly boring since 1865. Brilliantly boring since 1865 is a registered mark of the PNC Financial Services Group, Inc. PNC Bank, National Association, Member FDIC.

• 27:04Stephen J. Dubner?

  My name is Stephen Dubner. This is Freakonomics Radio, and this is the second of a three-part series we're calling "Bad Medicine." We don't mean to be ungrateful for the many marvels that medicine has bestowed upon us, nor do we mean to pile on or to point out the avalanche of obvious flaws and perverse incentives, but, well, it's just so easy.

• 27:27Speaker 10

  Doctors do something for decades. It's widely done. It's widely believed to be beneficial. And then, one day, a very seminal study contradicts that practice.

• 27:37Stephen J. Dubner?

  That's Vinay Prasad. He's an oncologist and an assistant professor of medicine at Oregon Health and Science University. He also co-authored a book about what are called "medical reversals," when an established treatment is overturned. Which happens how often?

• 27:53Speaker 10

  It's widespread, and it's resoundingly contradicted. It isn't just that it had side effects we didn't think about. It was that the benefits that we had postulated turned out to be not true or not present.

• 28:04Stephen J. Dubner?

  How can it be that so many smart, motivated people, physicians and medical researchers, come up with so many treatments that go all the way through the approval process and then turn out to be ineffective or even harmful? A lot of it simply comes down to the incentives.

• 28:22Speaker 10

  So much of the research agenda, even the randomized trial research agenda, is driven by the biopharmaceutical industry. And that's not necessarily a bad thing. I think there's many good things about that that really drives many, many trials. It drives a lot of good products. It also drives a lot of marginal products, or products that don't work. And the people who design those trials are, I think, very clever. You can sort of tilt the playing field a little bit to favor your drug, and the incentive to do so is often tremendous. Billions of dollars hinge on one of these pivotal trials. And to some degree, that's because it's a human pursuit. But to some degree, we could have policy changes that could more align the medical research agenda with what really matters to patients and doctors.

• 29:03Stephen J. Dubner?

  Let me ask you, in your own field, in oncology and in the particular cancers that you treat, how much more effective generally would you say the new cancer drugs are than the ones that they are replacing or augmenting?

• 29:15Speaker 10

  Let me say that there are few cancer drugs that have come out in the last two decades that are really wonderful drugs, great drugs. One drug came out of work here in the Oregon Health and Science University by Dr. Drucker, Gleevec, and that's a drug that transformed a condition where maybe 50 or 60% of people are alive at three years to one where people more or less have a normal life expectancy. So that's a really wonderful drug. But I think if we're honest with ourselves, we'll have to admit that the majority of new cancer drugs are marginal, that they offer sort of very small gains at tremendous prices. And to give you an example of that, among 71 drugs approved for the solid cancers, the median improvement in overall survival, or how long people lived, was just 2.1 months.

• 29:57Speaker 10

  And those drugs routinely cost over $100,000 per year of treatment or course of treatment.

• 30:02Stephen J. Dubner?

  But that points to one of the tricks that works so well, which is if it's 2.1 months extra, and if the expected lifespan was, let's just pretend for a moment it was six months, then on On a percentage basis, that's a massive improvement. And so I don't, as the patient or as the pharma representative, I'm not talking about that length of time, which might be lived under physical duress and financial duress, but rather I'm thinking about, goodness gracious, a huge 33% life expectancy extension.

• 30:33Speaker 10

  Right, and new drug improves lifespan 33% longer.

• 30:36Stephen J. Dubner?

  And who doesn't want that, especially when you're sitting there with your loved one in a horrible situation facing the end?

• 30:42Speaker 10

  The other thing I'd point out is, those two point one months, these clinical trials that are often conducted by the biopharmaceutical industry, they really choose sort of the healthiest patients, the people who are the fittest of the patients. On average, the age is almost 10 years younger in pivotal trials for the FDA drug approval than in the real world. And then when you start to extrapolate drugs that have real side effects in very carefully selected, and small benefits in carefully selected populations, to the average patient that walks into my clinic, who's older, who has other problems, who's taking heart medicine. There was a paper that came out about one of those costly, expensive drugs for liver cancer, and in the pivotal trial, it had a benefit of about two, three months, something like that.

• 31:25Speaker 10

  But in the real world, in the Medicare data set, it had no improvement in survival over just giving somebody good nursing care and good supportive care. And I think that's the reality for many of these marginal drugs, that when you actually use them in the real world, they start to not work so well, and maybe not work at all.

• 31:40Stephen J. Dubner?

  You've written and spoken out about cronyism and conflicts of interest between drug makers and the doctors who prescribe drugs. I'm curious, what happens when you go to an oncology conference? Are you an unpopular person there?

• 31:54Speaker 10

  Steve and I always wear a bulletproof vest when I go to the, no. But this has really been sort of the way medicine has operated for many years, that to some degree, practicing doctors in the community, having ties to the drug makers, that's one thing. But increasingly, we see that the leaders in the field, the ones who design the clinical trials, who write up the manuscripts, who write the review articles, who sort of guide everyone on how to practice in those fields, they have heavy financial ties to drug makers. And there's a large body of evidence suggesting that that biases the literature towards finding benefits where benefits may not exist, towards more favorable cost-effective analyses when drugs are really probably not cost-effective. It's a bias.

• 32:39Speaker 12

  Yes, well, we have a great deal of empirical data showing that funding sources and author financial conflicts of interest are associated with over-optimistic data.

• 32:49Stephen J. Dubner?

  That's Lisa Barrow. She's a professor of medicine. She's also co-chair of the Cochrane Collaboration, which is a global consortium of medical professionals and statisticians. Cochrane promotes evidence-based medicine by performing systematic reviews of medical research.

• 33:06Speaker 12

  And in fact, we have a Cochrane review on this very question. And this finding shows that if a study, a drug study, is funded by a pharmaceutical company whose drug is being examined, they're much more likely to find that the drug is effective or safe.

• 33:24Stephen J. Dubner?

  How much more likely?

• 33:25Speaker 12

  It's about 30%.

• 33:27Stephen J. Dubner?

  Did you catch that? An industry-funded study is 30% more likely to find the drug is effective and safe than a study with non-industry funding.

• 33:38Speaker 12

  And they're likely to find this even if they control for other biases in the study. So by that, what I mean, it could be a really well-done study, it could be randomized, it could be blinded, but if it's industry-funded, it's still more likely to find that the drug works.

• 33:53Stephen J. Dubner?

  But if a study is well-done, how can the results be so skewed?

• 33:58Speaker 12

  So it's everything from, I mean, the question they're actually asking, to how they frame the question, the comparators they use, how they design the study, how it's conducted behind the scenes.

• 34:08Speaker 08

  Trials are very often flawed by design in such a way that they're no longer the gold standard, no longer a fair test of which treatment is best.

• 34:18Stephen J. Dubner?

  That's Ben Goldacre.

• 34:19Speaker 08

  I'm an academic in Oxford, working in evidence-based medicine, and I also write books about how people misuse statistics.

• 34:27Stephen J. Dubner?

  He's also a doctor.

• 34:28Speaker 08

  Yeah, that's right. So I qualified in medicine in 2000, and I've been seeing patients on and off in the NHS for 15 years now.

• 34:37Stephen J. Dubner?

  One of Goldacre's books is called Bad Pharma. He echoes what Vinay Prasad was telling us about the people who are chosen for clinical trials.

• 34:46Speaker 08

  When you look at the evidence, what you often find is that trials are conducted in absolutely perfect dream patients, people who are, by definition, much more likely to get better quickly. Now, that's very useful for a company that are trying to make their treatment look like it's effective. But actually, for my real-world treatment decisions, that kind of evidence can be really very uninformative.

• 35:11Stephen J. Dubner?

  Imagine you're a doctor who's treating a patient with asthma. Not hard at all to imagine.

• 35:16Speaker 08

  Now, asthma is obviously a very common condition. It's about 1 in 12 adults.

• 35:20Stephen J. Dubner?

  With such strong demand for asthma treatment, there's been a bountiful supply from drug makers with dozens of clinical trials. A 2007 review of these studies looked at the characteristics of real-world asthma patients and how they compared to the people who'd been included in the trials.

• 35:36Speaker 08

  And they said, "Okay, let's have a look and see, on average, what proportion of those real-world asthma patients would have been eligible to participate in the randomized trials that are used to create the treatment guidelines, which are then in turn used to make treatment decisions for those asthma patients?" And the answer was, overall, on average, 6%. So 94% of everyday real-world patients with asthma would have been completely ineligible to participate in the trials used to make decisions about those very patients.

• 36:17Stephen J. Dubner?

  Of course, it isn't only with asthma patients where this happens.

• 36:21Speaker 08

  It's very common for randomized trials of antidepressants, for example, to reject people if they drink alcohol. Now, that sounds superficially sensible, but actually, I can tell you, as somebody who's prescribed antidepressants to patients in everyday clinical practice, it's almost unheard of to have somebody who is depressed and who warrants antidepressants who doesn't also drink alcohol. So you need trials to be done in people who are like the people that you actually treat.

• 36:52Stephen J. Dubner?

  If you look at the overall efficacy rate of most antidepressants, you'll find it to be very, very low, if there's any efficacy at all. And of course, there's the opportunity cost to consider.

• 37:05Speaker 08

  Because you tend to prescribe one antidepressant at a time.

• 37:08Stephen J. Dubner?

  Which means while a patient is on one drug that may not be working, they can't try another that might. Plus which, there are the side effects to consider.

• 37:21Stephen J. Dubner?

  So a lot of drugs that look great on paper don't do very well in the real world. Why? Part of it is what Ben Goldacre and Vinay Prasad were talking about, cherry picking subjects for clinical trials. But Goldacre says there are plenty of other ways to manipulate trial numbers in the drug maker's favor. What do you do, for instance, when research subjects quit a trial because of the treatment's side effects?

• 37:47Speaker 08

  And what you see is people inappropriately using a statistical technique like last observation carried forward to account for missing data from patients who dropped out of a study because of side effects.

• 38:00Stephen J. Dubner?

  Last observation carried forward is a statistical extrapolation, pretty much what it sounds like, and worth looking up if you're interested in that kind of thing. You can see how an inappropriate use of such a technique would tilt things in the drug maker's favor. There's also the widespread use of what are called surrogate outcomes, as opposed to real world outcomes. Consider the recent drugs approved by the FDA to treat diabetes.

• 38:25Speaker 08

  All of those drugs have been approved onto the market with only evidence showing that they improve your blood sugar.

• 38:32Stephen J. Dubner?

  But, Goldacre says, none of those drugs showed evidence that they reduced the risk of heart attack, renal failure, or eye problems, the actual consequences of having diabetes. There are a couple new drugs on the market that seem promising, but it's hard to test for these outcomes in a clinical trial. And by hard, what I really mean is time consuming and expensive. So instead, the researchers go for the simple surrogate outcome of whether their pill lowers blood sugar.

• 39:00Speaker 08

  But actually, it's not correlated as well as you might hope. And the history of medicine is absolutely littered with examples of where we have been given false reassurance by a treatment having a good impact on a surrogate outcome, a laboratory measure, and then discovered that actually it had completely the opposite effect on real world outcomes.

• 39:22Stephen J. Dubner?

  As in the case of the infamous CAST trial that we covered in part one of the series, in which the drug that suppressed aberrant heart rhythms actually worsened survival outcomes.

• 39:36Stephen J. Dubner?

  Now, we should point out that Ben Goldacre and everyone we've been speaking with for our Bad Medicine episodes, fully appreciates that medicine is science and that failure is part of science. The human body is an extremely complex organism with lots to go wrong. Diagnosing and treating even a simple problem can be very difficult. So it's easy to take potshots from the sideline at good ideas that went bad. It's even easier to criticize pharmaceutical companies who seem much more intent on making money than on making good medicines. But as Goldacre points out, those companies are simply responding to the incentives that are placed before them. Incentives that don't necessarily encourage them to do the right thing.

• 40:22Stephen J. Dubner?

  Goldacre points to a massive eight-year study called the Allhat Trial, in which academic researchers compared various drugs from a number of drug makers that were intended to lower blood pressure and cholesterol. Two of these drugs were made by the American pharmaceutical company Pfizer.

• 40:39Speaker 08

  Pfizer came along and they said, "Look, we've got this fantastic new blood pressure lowering drug. And we've got various grounds for believing that it's going to be better than old-fashioned blood pressure lowering drugs. But at the moment, all we can tell you is it's roughly as good at lowering blood pressure."

• 40:57Stephen J. Dubner?

  So Pfizer asked the Allhat researchers to test whether their drug actually reduced the real-world outcomes that really matter: heart attack, stroke, and death.

• 41:07Speaker 08

  So, the researchers said what all academic researchers have said to drug companies since the dawn of time, which was, "Thank you very much. That sounds like a fabulous idea. That'll be about $175 million, please."

• 41:21Stephen J. Dubner?

  Actually, Goldacre misspoke. It was only $125 million, and Pfizer's share was just $40 million. But still, $40 million.

• 41:30Speaker 08

  And it was so expensive simply because measuring real-world outcomes like that, especially before the era of electronic health records, was extraordinarily expensive.

• 41:41Stephen J. Dubner?

  So Pfizer pays in and the trial begins.

• 41:45Speaker 08

  Run for a very, very long time, many, many, many years. But it was stopped early because the Pfizer treatment, which was just as good at lowering blood pressure, was so much worse at preventing heart attack, stroke and death, that it was regarded as unethical to continue exposing patients to it.

• 42:03Stephen J. Dubner?

  The Pfizer drug we're talking about was called Cardura. So where does Pfizer come out in all this?

• 42:09Speaker 08

  It's really important, I think, to recognize that Pfizer did nothing wrong here. Pfizer did exactly what we would hope all companies should do. They didn't just say, oh, that's fine. We've got some surrogate endpoint data. We've got laboratory data showing it lowers blood pressure and that's all we need. Instead, they went out and they did the right thing. They exposed themselves to a fair test. They said, we want to see if this treatment improves real world outcomes that matter to patients, heart attack, stroke and death. And they were unlucky and it flopped.

• 42:43Stephen J. Dubner?

  The real problem, Goldacre says, is when drugs aren't subjected to the real world test.

• 42:48Speaker 08

  The real bad guys here are the people who continue to accept weak surrogate endpoint data, like, for example, on the new diabetes drugs. It may well be that they lower the laboratory measurement on a blood test, but that doesn't necessarily mean that they reduce your risk of heart attack, stroke and death. And to find that out, we need to do proper randomized trials, which are admittedly longer and more expensive.

• 43:18Stephen J. Dubner?

  But there's yet another problem. What happens if a proper randomized trial doesn't show the efficacy a drug maker was hoping to show? Well, there's a good chance the world will never know about it because of

• 43:32Speaker 08

  Publication bias.

• 43:34Stephen J. Dubner?

  Ian Chalmers, a co-founder of the Cochrane Collaboration, is a major player in the evidence-based medicine movement.

• 43:41Speaker 13

  About half of the clinical trials that are done never see the light of day. They don't get published. Isn't that outrageous?

• 43:49Stephen J. Dubner?

  Which trials do get published?

• 43:51Speaker 13

  Trials that show results that are so-called statistically significant are more likely to get published than those that don't have those results.

• 44:01Stephen J. Dubner?

  Now, you might think, well, yes, it makes sense to publish trials where a medicine seems to work. And if it doesn't seem to work, why is that important to publish? Ben Goldacre again.

• 44:12Speaker 08

  So if you cherry pick the results, if you only publish or promote the results of trials which show your favoured treatment in a good light, then you can exaggerate the apparent benefits of that treatment.

• 44:25Stephen J. Dubner?

  As Chalmers tells us, there are all kinds of reasons why the results of an unsuccessful trial might not get published.

• 44:32Speaker 13

  It may threaten a commercial enterprise's interests to publish a trial which is disappointing. It may be something which someone who has had a favourite hypothesis and been known for writing and speaking about it for years finds out that the first really good study to test the hypothesis doesn't find any support for it. There's laziness.

• 44:58Speaker 08

  And that's the real scandal here, is that you are allowed to legally withhold the results of these trials. And so people do. So the results of trials are routinely and legally withheld from doctors, researchers and patients, the people who need this information the most. That is a systematic, structural failure.

• 45:18Stephen J. Dubner?

  The structure, that is, imposed by government regulators, by funding sources, by the markets themselves, all of which can be very hard to change. So how does Ben Goldacre see the situation improving?

• 45:31Speaker 08

  We set up something called the All Trials Campaign a couple of years ago. And the All Trials Campaign is a global campaign to try and stop this problem from happening. So asking companies, research institutes, academic and medical professional bodies, patient groups and all of the rest to sign up and to say, All trials should be registered, so you publicly post on a publicly accessible register the fact that you've started a trial, because that means we know which trials are happening. So at least we can see if some of them aren't being published.

• 46:04Stephen J. Dubner?

  The All Trials campaign also urges the publication of what's called a clinical study report.

• 46:09Speaker 08

  And a clinical study report is a very long, very detailed document, hundreds, sometimes thousands of pages long, that describes in great detail the design of the study and the results of the study. And that's really important, because often a trial can be flawed by design in a way that is sufficiently technical that it is glossed over in the brief report that you get in an academic journal article about a trial. And those design flaws can only be seen in the full-length clinical study report.

• 46:44Stephen J. Dubner?

  There's also a growing momentum to curb conflicts of interest in medical research.

• 46:49Speaker 12

  Well, I think we've already had great improvements in transparency. And what's really pushed the disclosure of funding sources has been the journals.

• 46:58Stephen J. Dubner?

  That's Lisa Barrow again, from the Cochrane Collaboration.

• 47:01Speaker 12

  So if you publish something, you are required to disclose the funding source. And, you know, this is still not 100% enforced, but it's getting pretty close.

• 47:10Stephen J. Dubner?

  On the other hand, Barrow says, a given researcher or investigator may have undisclosed biases or conflicts of interest.

• 47:18Speaker 12

  And one sort of loophole is that the investigator themselves has to decide if something's relevant to the particular study. And so they may say, well, I just don't think it's relevant.

• 47:28Stephen J. Dubner?

  There's another quirk in the medical industry that probably doesn't serve the public good.

• 47:33Speaker 12

  Drug companies evaluate their own products, whereas in software, you usually get somebody external to check the quality of your project. Engineers get people to do the earthquake checks for them, who are independent from the people who built the bridge. So it's a very odd system that we have, where the companies with an interest or stand to gain financially from testing a product are testing it themselves. So I think we need to change that.

• 48:00Stephen J. Dubner?

  And finally, there are the doctors themselves, the endpoint in this complicated, conflicted infrastructure that's meant to deliver better medicine. Ben Goldacre, the gadfly physician who knows so much about bad pharma and bad medicine, acknowledges that the entire system is due for reform.

• 48:19Speaker 08

  And it's a structural failure that persists because of inaction by regulators, by policymakers, by doctors and researchers as much as because of industry. And none of us can let ourselves off the hook.

• 48:35Stephen J. Dubner?

  So next time on Freakonomics Radio, in our third and final episode of Bad Medicine, what's a doctor to do?

• 48:42Speaker 11

  I see the opioid story as part of the recurring sense of of hope and despair associated with these drugs that are supposed to solve problems, but they end up being problems in themselves.

• 48:56Stephen J. Dubner?

  What to do about the troubling finding that more experienced doctors have worse outcomes than young doctors? So I would think that you are a downright danger to your patients. How is it that you're not?

• 49:10Stephen J. Dubner?

  No comment. And finally, yes, finally, lots of reasons to be optimistic, at least cautiously so, about the future of medicine.

• 49:21Speaker 01

  Where science and medicine is going in the future is to more and more precision medicine so that we can get closer to an autonomous and individualized diagnosis.

• 49:31Stephen J. Dubner?

  That's next time on Freakonomics Radio.

• 49:37Stephen J. Dubner?

  Freakonomics Radio is produced by WNYC Studios and Dubner Productions. This episode was produced by Stephanie Tam. Our staff also includes Alison Hockenberry, Merritt Jacob, Greg Rosalsky, Eliza Lambert, Emma Morgenstern, Harry Huggins, and Brian Gutierrez. You can subscribe to Freakonomics Radio on Apple Podcasts, Stitcher, or wherever you get your podcasts. You should also check out our archive at Freakonomics.com where you can stream or download all our episodes. You can also read the transcripts and find links to all the research and books we've mentioned in this episode. You can also find us on Twitter, Facebook, or via email at [email protected]. Thanks for listening.

• 50:32Stephen J. Dubner?

  Freakonomics Radio is sponsored by Ozempic. Innovation happens through rethinking what's possible. And when it comes to GLP-1s, Ozempic pill does just that. Learn more about Ozempic semaglutide tablets, 4 and 9 mg, by calling 1-833-OZEMPIC or visit ozempic.com to view the medication guide. And ask your doctor what's possible with FDA-approved Ozempic pill.

• 51:03Stephen J. Dubner?

  Freakonomics Radio is sponsored by Southern Company. The world and its energy needs are always changing. Southern Company's commitment to meeting this demand stays the same. That's because Southern Company believes energy is more than a utility. It's what powers possibilities. So they are looking ahead and investing $80 billion in infrastructure upgrades and are committed to fueling growth in ways that benefit all customers so that reliable and affordable energy is accessible for generations to come. Go to southerncompany.com to learn more. Southern Company, building the future of energy.

• 51:41Speaker 14

  Pandora makes it easy for you to find your favorite music. Discover new artists and genres by selecting any song or album, and we'll make you a personalized station for free. Download on the Apple App Store or Google Play and enjoy the soundtrack to your life.

• 51:53Speaker 15

  This episode is brought to you by Charles Schwab. Is there a right time to sell a stock? Are you taking the right risks with your portfolio? Financial decisions can be tricky, and often your own cognitive and emotional biases can lead you astray. Financial Decoder, an original podcast from Charles Schwab, can help. Join host Mark Reapy as he offers practical solutions to help overcome the cognitive and emotional biases that may affect your investing decisions. Listen at schwab.com/financialdecoder.