Special Episode: Adam Kucharski & Proof

Why do we believe what we believe? Is what we believe the truth? How can we convince others of our beliefs? If you’ve ever found yourself pondering these questions, you know that the answers are rarely clear-cut. We need to form beliefs in order to navigate the world, but how skilled are we at evaluating evidence for those beliefs or weighing new data that contradicts them? In this week’s TPWKY book club episode, Adam Kucharski https://kucharski.io/, Professor at the London School of Hygiene and Tropical Medicine joins me to discuss latest book, Proof: The Art and Science of Certainty https://bookshop.org/a/3175/9781541606692. With this book, Dr. Kucharski presents a compelling and thoughtful examination of the concept of proof, delving into topics ranging from the justice system (what’s a reasonable doubt?) to infectious disease, clinical design to the founding of this country. And he leaves us with a powerful lesson: what convinced you of something might not convince someone else. Tune in for a fascinating conversation! Support this podcast by shopping our latest sponsor deals and promotions at this link: https://bit.ly/3WwtIAu See omnystudio.com/listener https://omnystudio.com/listener for privacy information.

• 0:01Speaker 00

  This is exactly right.

• 0:06Speaker 01

  Dry hands can be uncomfortable and frustrating, especially when frequent hand washing, changing weather, and everyday wear and tear take their toll. O'Keefe's Working Hands is here to help keep dry hands feeling hydrated and comfortable. O'Keefe's Working Hands creates a soothing barrier on the skin that locks in moisture and prevents further dryness. It's concentrated and non-gracey, and you start to feel relief in one use. Go to O'Keefe'sCompany.com to get yours today. That's O-K-E-E-F-F-E-S company dot com for guaranteed relief. O'Keefe's Working Hands, yeah, it works.

• 0:37Speaker 00

  Breathe in and reset with Lincoln Rejuvenate, an available multisensory feature in the 2027 Lincoln Navigator SUV. Experience the luxurious massage, visuals, and scents to help you relax in the driver's seat. It's your time for a reset. Visit Lincoln.com to learn more. Lincoln Rejuvenate does not operate unless the vehicle is in park, when the vehicle is off, or if any doors open. Do not use the feature in a closed garage or in other enclosed areas. See owner's manual for important operating instructions. Lincoln and Navigator trademarks of Ford or its affiliates. All rights reserved.

• 1:52Speaker 02

  Hi, I'm Erin Welsh, and this is This Podcast Will Kill You. You're tuning in to the latest episode of the TPWKY Book Club, where I chat with authors of popular science and medicine books about their latest work. Since starting this series a few years ago, I've gotten to cover some Amazing books, and I appreciate so many of you reaching out with your suggestions for books to feature. Keep those recommendations coming, please. And if you'd like to take a look at the full list of books that we've covered in this series, as well as get a sneak peek at ones that are coming up in future episodes, head on over to our bookshop.org affiliate page, which you can find on our website, thispodcastwillkillyou.com, under

• 2:33Speaker 02

  the Extras tab. On the Bookshop page, you'll find several podcast-related lists, including one for This Book Club and the TPWKY Kids Book Club, which, if you're not following us on social media, you absolutely should be, because Erin Updike has been putting together videos reviewing children's books. It is such a great resource for sciency kids' books for all ages. And if you want to share your thoughts on these episodes, make topic suggestions, submit a first-hand account, you can get in touch with us using the Contact Us form on our website. Two last things before moving on to the Book of the Week, and that is to please rate, review, and subscribe. It really does help us out. And second, you can now find full video versions of most of our newest episodes on YouTube.

• 3:18Speaker 02

  Make sure you're subscribed to the Exactly Right Media YouTube channel, so you never miss a new episode drop. Belief is a powerful force. It shapes every facet of our lives and transforms perception into reality. What we believe to be true is not always what is actually true, something I'm sure we can all relate to. Maybe you've debated with a friend over the answer to a trivia question, like you both know the right answer, but your answers are somehow different. Or maybe you've had a heated exchange with a relative who firmly believes that the moon landing was faked. How do we decide what we believe? How can we know that what we believe is the truth? And how can we convince others of that?

• 4:04Speaker 02

  These are precisely the questions that Adam Kucharsky, who is professor at the London School of Hygiene and Tropical Medicine, asks in his latest book, "Proof: The Art and Science of Certainty." Kucharsky, who is a mathematician that works on infectious disease outbreaks, explores how we are inundated with information, and increasingly misinformation, that we have to evaluate to determine whether or not we should incorporate it into our decision-making. This extends beyond personal decisions, which route is best to take to work, what to make for dinner. Our world is built upon structures of proof with varying degrees of support. That car that you drive to work is manufactured under rigorous safety testing,

• 4:49Speaker 02

  meaning there are established guidelines for what is considered "safe" and how to test that. Same thing with the food we eat, the medicines we take, the buildings we spend time in. We don't question so many of our beliefs. To do so would leave you frozen, uncertain of which direction to move in, what to trust. You'd have no time to actually live your life. But when we do scrutinize our certainty, we might find a gulf between our beliefs and someone else's, and those beliefs, and the objective truth. Where does that incongruity originate? Why are we skeptical about some things and not others? What does it take to make up our mind? And what does it take to change it?

• 5:34Speaker 02

  That answer might not be the same for everyone. An enlightening blend of philosophical musings, political commentary, statistical exploration, and personal reflection, "Proof" is a fascinating read, particularly as this unceasing flood of information, both good and bad, shows no sign of stopping. Let's take a quick break and then get into things.

• 6:23Speaker 02

  Professor Kucharski, thank you so much for joining me today.

• 6:26Speaker 03

  Thanks for having me.

• 6:27Speaker 02

  I am thrilled to talk with you about your latest book, Proof: The Art and Science of Certainty. And before we dig into the various forms of proof and how we determine a threshold for proof, or what different types of proof exist for certain situations, I want to start at the very beginning. What is proof? Is there a standard definition?

• 6:48Speaker 03

  Yes, I think that's a great question. And I think my background's in maths, so I think a lot of my training was around this idea that you can have this definitive knowledge that something is true. And I think it's something that people grappled with across fields. I mean, one of the stories that really struck me was Abraham Lincoln, when he was training to be a lawyer, came across this word demonstrate. And you had this kind of beyond reasonable doubt, this certainty. And he's like, I don't really understand what this is as a concept. And he actually went back to all of these ancient Greek mathematical texts to understand how can we take the knowledge we have, build on that, prove new theorems, use that to prove subsequent knowledge. But I think one of the things that was really

• 7:33Speaker 03

  the motivation for the book, and something that I think anyone who works with information and decision making and evidence happens across very often, is it can become quite a shifting concept. I mean, even in mathematics, things that people thought were proven turned out had some hidden assumptions or human judgments that were lurking there, and caused a lot of that to collapse. So I think it's a fascinating concept, because it's something that's so important in life, not just having knowledge that we gradually accrue, but for many of the things we care about, whether it's dealing with emergency, whether it's a legal case, whether it's even just a minor business decision in our day, we have to

• 8:15Speaker 03

  work out where we set the bar and how we evaluate what we've got. And I think for me, that was really the launching off point to explore this. How do we converge on certainty? And what happens when it goes wrong?

• 8:27Speaker 02

  Thinking about the difference between proof and certainty and truth, like, what is the relationship between those concepts?

• 8:35Speaker 03

  Yeah, I think that's a, that's a great question. And without going down the kind of philosophical rabbit hole, it could have been a book on it. What is reality?

• 8:43Speaker 02

  Yeah.

• 8:44Speaker 03

  But I think the way that I approached it is just to look at how people have thought about this in, in different fields. And again, even going, going back to, to Lincoln and much earlier, there was this, this appeal of this certainty, this idea that there could be this universe truth. And it's why a lot of fields ended up borrowing from mathematics. You see it in the US Declaration of Independence. You know, we hold these truths to be self-evident. The original draft was, we hold these truths to be sacred and undeniable. But Benjamin Franklin didn't like that because it sounded like they were kind of appealing to some divine authority. And self-evident is, is just borrowed directly from, from maths. It's just a given truth. And unfortunately, it turned out a lot of these things about equality weren't self-evident. But I think that, that story of how you think about these things, and even when we see

• 9:28Speaker 03

  in, in the legal world, a lot of it was originally derived from concepts around maths, around probability. If you talk about, you know, some of these thresholds, preponderance of evidence, you're saying it's more likely than not, and you're kind of borrowing a lot of these kind of probability based ideas. And even in the world's kind of more experimental design, as that kind of developed, a lot of it was about, I mean, actually, some of these early studies were almost trying to discount some of the influences of religion, you know, wanting to understand cause of effect in the world, rather than just appealing to some other influence. And then it, for lots of people, it became this question of how do you take the evidence you have and how do you link that to a conclusion that you want to make? And where do you set the bar for that? Do you try and get ever closer to certainty?

• 10:13Speaker 03

  And there was actually a lot of statistical tension about 100 years ago. I know statistical debates kind of sound a bit boring, but it was actually this, this real, you know, people just, you know, almost like wouldn't talk to each other because it was this tension between, do you just try and get ever closer to the truth? Or do you have a framework that allows you to make decisions? And I think a lot of times in life, we don't get to do the academic, I'm just going to sit on the fence, you know, I just, I don't know, and I'm just not going to do anything with, you know, with, with life or actions. But often we have to decide, we, we do something or we don't do something. Or, you know, we say someone's guilty or we let them go free. Or there's these decisions we have to make. And so. That process of interacting with evidence is much more pressured.

• 10:55Speaker 03

  I think that was one of the real big tensions that never fully got resolved, actually. Even how we teach statistics at school, we kind of smushed together these two very different philosophies. One of this ever higher bar for evidence and one where we're sort of outlining a framework to make a decision based on the knowledge we have.

• 11:13Speaker 02

  When it comes to public health and medicine, there's a lot more pressing need to make decisions. And yet, this decision is often dragged out for long periods of time. And sometimes that is at the urging of someone who has an incentive to drag out a decision. So one of the examples that you talk about is Austin Bradford Hill, who is talking about this relationship between cigarettes and lung cancer and saying, oh, we have some evidence and there's still a lot of skepticism, but we have enough to make a decision. We cannot use uncertainty as an excuse for inaction. Do you feel like we've ever truly learned that as a society?

• 11:56Speaker 02

  Or has it been players like the tobacco industry saying, oh, no, this uncertainty, we need to push for more and more and more evidence?

• 12:07Speaker 03

  Yeah, I think that's a really good question. I think that's a really good example of almost weaponized certainty, that you can always set the bar high in any aspect of life. You can set the bar high and high and high to the point where you just won't do anything. And inaction, of course, is in itself a decision. And I think Bradford Hill's work, he was extremely thoughtful in how he approached this. Because something like smoking, you can't really design it like a trial. You can't get people to randomly take up smoking and see if they get cancer. There's obviously ethical reasons around. There's also just timeline reasons. If you look at the timescale of the intervention versus what happened, you might have to wait decades to have that clear signal.

• 12:46Speaker 03

  And so he did a lot of pony work with others, linking together the various non-random data sets you had available. Because one of the criticisms, of course, is any data is yes, smokers are more likely to get cancer, but maybe there's a genetic reason that makes them more likely to smoke and get this. And he outlined a lot of the ways we can think about cause and effect. And I think that's A very useful set of concepts, even some of it's the obvious ones of the cause needs to become before the effect, or that if you have this strength of association, more cigarettes makes you more likely to get cancer. If you see that across multiple countries, or if you can start to think about the biological plausibility, we see carcinogens in other kind of situations as well. None of those things on their own is conclusive, but you

• 13:28Speaker 03

  can start to build this evidence base. And he made this really good point that any knowledge we have, even if it's very confident knowledge, is always subject to further refinement. But we still have that knowledge at that point in time, and we can seek further information. There's been lots more studies of smoking since the early ones. But also, that's information that we have to do something with. And I think we often, particularly in the situations with emerging threats, or kind of early concerns about things, whether it's a health intervention we think might be harmful. I mean, one of the examples are given in the book is the work at the FDA around thalidomide, which was this treatment for sickness in pregnancy.

• 14:10Speaker 03

  And there was actually a lot of concerns about safety for babies, and the FDA blocked it as a result. But on the other hand, you get things where there might be a lot of value, for example, in reducing smoking for health outcomes. And even if there's that uncertainty, and Bradford Hill made this nice point of actually, the standard you should apply for taking action kind of depends a bit on the situation you're dealing with. If it's a fairly cheap action to take, if it's not too disruptive for people. But actually, in his argument, he said, smoking is something people really enjoy. So we need a kind of higher barrier. And I think it's a reasonable point. If you're going to tell a lot of people to change how they live their lives, that the evidence you need is perhaps different

• 14:54Speaker 03

  for something where you can take some action and you can unwind that. So it is those kind of trade offs that you have available that obviously needs to play in as well.

• 15:04Speaker 02

  Let's take a short break. And when we get back, there's still so much to discuss.

• 15:10Speaker 01

  What is one thing many of us deal with every single day but rarely talk about? Dry hands. Whether you frequently wash your hands or deal with harsh weather, your hands can be left feeling dry, rough, cracked, and even uncomfortable. That's where O'Keefe's Working Hands Hand Cream comes in to help. The fast-absorbing formula goes on smoothly and it won't leave behind a greasy residue, so you can get back to your busy day without ever missing a beat. And because a little will go a long way, O'Keefe's Working Hands Hand Cream is effective, dependable, and economical, too. Whether you're dealing with harsh weather outside, demanding work, or everyday wear and tear, O'Keefe's Working Hands works just as hard as you do. So keep your hands hydrated, smooth, and feeling comfortable all season long.

• 15:55Speaker 01

  Visit okeefescompany.com today. That's O-K-E-E-F-F-E-S company dot com for guaranteed relief. O'Keefe's Working Hands. Yeah, it works.

• 16:08Speaker 04

  Hey, I'm Niall Rogers, producer, songwriter, guitarist, and co-founder of the group Sheik. A great song has a way of defining a moment, a moment of joy, heartbreak, or anything in between. This fall, I'll be driving across the country, dropping in on some of the coolest, most fascinating artists working today. We'll go deep into their lives and careers and hear what it was like before the moment that made them. From hitmakers like Miguel...

• 16:41Speaker 05

  That last year, finishing that record was the hardest experience of my life. Regardless of how everyone else buys into this, I believe in this and I'm gonna push.

• 16:53Speaker 04

  To rising stars like Alex Warren.

• 16:56Speaker 06

  You know, a song is a story, and the verse is setting the scene. And for me, at least, it's allowing me to be personal and vulnerable with the audience, whether it's about my wife and I'm being vulnerable about her, or it's about, you know, my parents and I'm being vulnerable to them. It's almost like a cinematic piece in a way where each song tells that beautiful story.

• 17:14Speaker 04

  It's gonna be one hell of a trip. Experience Before the Moment, a Lincoln Original Series on the iHeartRadio app, YouTube, or wherever you get your podcasts.

• 17:42Speaker 02

  Welcome back, everyone. I've been chatting with Dr. Adam Kucharsky about his book, Proof: The Art and Science of Certainty. Let's get back into things. Right, the thresholds for certainty is, it can be different depending on the situation. And then there's also these personal thresholds for certainty or evidence, you know, how much information do we need? And, and one of the things that you discuss in your book as well is sort of what happens when evidence flies in the face of our personal beliefs and how sometimes even despite a mountain of evidence, we can just still feel like that's not possible, we can't reject it, it's not an intuitive truth, you know, what, what happens? Like,

• 18:24Speaker 02

  what does this show us about sort of the personal nature of proof and certainty?

• 18:29Speaker 03

  Yeah, I think that's one of the things that, that really kind of struck me in researching that. I mean, even in, in some of these kind of mathematical puzzles, examples, it's things that, you know, I'd come across as, as a kid and convince myself, oh, that's just, that's the answer to the puzzle. And it was only years later when I was explaining it to someone else, or someone else had asked me about it, and I sort of went through the thing that convinced me, and it just didn't convince them at all. And I think that's a really interesting, I think we focus a lot on, you know, how science works, how methods work, what convinces us. And you see this in even a lot of studies around political beliefs, that people will often try and convince others with arguments that convince them. And then you get this gap, and it's almost like that just fails. And I think that's a really interesting step

• 19:12Speaker 03

  to explore. But why does that fail? And one of the things that I find even just kind of in some of the modern tools we have in the modern era, quite striking is where we have this desire to explain things. So, yeah, a few years ago, I was talking to a bunch of people working on AI, and there's a lot of- So concern about things that self-driving cars, that we don't understand why they make mistakes. We need that explainability. We can't have things we don't trust. And actually, in medicine, we have all sorts of things that we know work, we know how often they work. We don't fully understand the physics and biology, something like anesthesia, for example. You know, you can control the effect it's going to have, but actually, all the underlying biology and kind of physics mechanisms, there's still more work to be done. Things like defibrillation, you know, if you give a heart a shock, you can kind of reset it.

• 19:54Speaker 03

  Again, some of that's understood, but there's, there's still those kind of gaps in knowledge. But we know that these are useful tools and even if you run a clinical trial, you can assess how effective a treatment is. But that on its own will just tell you the effect. It won't tell you necessarily all the mechanisms that are going on to explain it. But there's these tools that we've got and we've got the evidence to take action and use these things we're very happy with. And there's other areas of life where actually that inability to explain something kind of really bothers us. You know, if even if self-driving cars were much safer than humans, and humans, when I started looking into a book, humans are not good at driving. You know, it's not a massively high bar. But I think it would still make people uncomfortable, even if they were, say, twice as safe.

• 20:35Speaker 03

  In cities where it's well, very well defined, I think it would still bother people if every now and again, there was just an accident that we had no, no real idea what was happening. And I think that's really important to bridge because I think that, you know, particularly when you get that gap in understanding, that's room for other explanations to, to kind of creep in. And I think that's where we start to see emergence where things like conspiracy theories, whether it's things with kind of incorrect logic. Often it is that gap between what we're seeing and the understanding of why that's happening. I think humans have this very, in many ways, very powerful desire to explain what they're seeing. But in some cases where the explanation is very hard to untangle, it can lead us astray.

• 21:15Speaker 02

  That's fascinating to think of the, the gap between understanding and what is happening. We don't understand how anesthesia works or how Tylenol or acetaminophen truly works. But we do understand how vaccines work, for instance. And yet there's so many conspiracy theories and misinformation surrounding this thing that we do know how it works. Well, I guess what good does evidence do if we do not take it into account and are not open to it?

• 21:42Speaker 03

  Yeah, and I think for me, a lot of it is is just understanding at what point that breaks down. I mean, even if you if you look at some of the the Covid vaccines, for example, you know, or even some of the kind of other debates around climate intervention, other things, you know, often it gets very into debating some element of the technology. And I think often it's actually just people disliked some of the the control that was exerted over them through mandates or for other things. And actually, you know, if you've got an intervention that you're unhappy with, you can disagree with the intervention and say, look, yeah, for example, we know that intervention works, but I disagree with how you're implementing it. Or you can disagree that the intervention actually has an effect.

• 22:24Speaker 03

  Or you can even one step down and just say, you know, actually, I think there's there's sort of deeper problems, or maybe the disease isn't a threat. And I think often those kind of leg levels get get tangled up. And I think in a lot of conversations I've had with people, often they're they're sort of deep down concerned, or the thing that they're approaching it with isn't necessarily that they've just out of nowhere decided that this isn't a threat or that that technology doesn't work. It's actually, in some of these instances, things are a bit more marginal. And you could say, you know, you can make an argument either way, even if the underlying intervention is is effective or is going to have this. You know, you can make kind of this moral and this is not just about an sort of epistemological question.

• 23:06Speaker 03

  And so I think kind of understanding where those drivers are, and also just in our own arguments, I think sometimes, you know, I have the conversation with people and I think I'm just arguing about the kind of the nuances of whether intervention is a good idea or not, and they're actually arguing whether it's a problem in the first place. We see it, vaccines are, I guess, examples more polarized, but even something in climate, you know, you can have a lot of people who just agree on the nature of the threat of climate change. They agree on the different levers that we probably have available as society, but they might strongly disagree about actually how we prioritize those and all of the trade offs. And I think it's just understanding what level we're on and where the evidence might stop and where it might then just be other things that are filtering in on a personal level.

• 23:49Speaker 02

  This idea of proof and certainty and truth, that seems very intuitive in a lot of ways today. But this maybe wasn't always the case. Like, when did the concepts of truth and the need for these self-evident truths, or certainty, or proof, when did these come to be? And then, you know, in what fields or what areas were they initially applied?

• 24:12Speaker 03

  Yeah, I think that's a great, it's easy just to think of like the world and sort of science and evidence just always was, as it is. I mean, even in mathematics, this idea that we had a universal truth wasn't the same throughout history. If you go back to the ancient Egyptians, ancient Babylonians, they were much more focused on problem solving. A lot of their texts are kind of these, these kind of puzzles and very much things around kind of practical, everyday problems. And even if you look at, you know, their formulas for an area of a circle, they're quite approximate. And if you're, you know, if you're building something that needs quite a large circle, you're probably going to be okay using those. But it's not going to give you that really precise truth, no matter what problem you're working on. And that's something where the ancient Greeks, mathematicians like Euclid came in and tried to put things on a much more solid footing. So you've got these concepts

• 24:57Speaker 03

  like pi, that if you want the area of a circle, that will just be universally true. And you won't have this issue of your kind of approximation breaks down. And it was then, I think, that as it sort of came into the Enlightenment era, was very appealing for people, that you could have these undeniable truths about the world. And I think that's where a lot of other fields started borrowing them as well. But even in, yeah, in medicine, if you look at this, this study of cause and effect, a lot of that, it was the medieval Arabic world that a lot of that started to emerge. So a lot of the, the kind of superstition, this idea that disease or conditions just kind of come out of nowhere and it's, you know, bad people, or, you know, someone's a witch, or this kind of stuff that was going around in, in much of Europe at the time.

• 25:41Speaker 03

  There was a lot of early writings even around sort of the 11th century saying these aren't supernatural, they're natural causes and we can study them, yeah, we can study them, we can work out what the cause of effects were. A lot of early attempts to try and think about concepts that we would now call things like having a control group or thinking about how we kind of, you know, would divide and treat some people and not treat some people and then compare the difference. The conclusions didn't always work out. I mean, there was, I think, one of the earlier studies was someone who'd identified correctly the symptoms of meningitis, but then concluded that blood letting was really effective for it, which probably something in their study design had gone astray. But again, it just kind of really in, and it's one of the things you look back on and you think it's just,

• 26:23Speaker 03

  it's pretty obvious that we should be doing it that way. But even coming into the 20th century, if you look at something like analysing a medical treatment, a lot of the early studies did an alternation method. Because if you think about it, rather than randomise patients, you could just say, well, the first patient that comes in, I'm going to treat, the second I won't, the third I will, fourth I won't. And on average, you should get something that any other sources of variability should balance out and the difference between those groups should be on average down to the treatment effect. But Bradford Hill, actually, who did a lot of the pioneering work in the early sort of clinical trial space, noticed that the groups were often imbalanced. Because what was happening is patients were coming in and doctors were maybe subconsciously that, oh, maybe that person looks a bit ill, I'll enrol them, or maybe they don't meet the diagnosis. And it, and actually, a lot

• 27:08Speaker 03

  of the early randomisation wasn't statistical. It was just, it was to sort of keep humans from themselves because we couldn't trust subconscious judgement. So a lot of the early randomisation in medicine wasn't about the statistical properties of the trial design. It was just about making sure humans didn't muck things up, basically, with their internal biases.

• 27:27Speaker 02

  Well, I mean, we'll find a way, I'm sure, somehow, some way. But it's, it's interesting to think about this idea of like self-evident truths, thinking back to, okay, yes, there's superstition and this person is a witch based on these signs or whatever. Was that also viewed as proof?

• 27:44Speaker 03

  The story of those trials by ordeal is a fascinating one as well because they were used for a long time. You could have trial by ordeal, like by water or by fire, whatever. And then you could also choose trial by duel. So basically, the big criminals always pick that. And people started to notice like, oh, you know, if God is deciding which one's innocent, God tends to pick the bigger one, like pretty much every time, which is, I think there was that that came in. But actually, one of the reasons they stopped using them is a lot of the religious scholars became concerned that they were basically trying to, by running those trials, you're essentially trying to get God to do your work for you. And that felt for them a bit awkward because you're sort of on demand saying, hey, can you come and make a decision for us, which they they sort of got quite uncomfortable

• 28:25Speaker 03

  with. But even those early systems, I mean, early juries in England were kind of fascinating because they they weren't the structure that we had today. They kind of did their own investigations. Often someone was accused and then they went off and accused someone else and kind of did their own thing. And it was it was only over time that system kind of evolved of having that way and converging to something. And I think that's, you know, we talk a lot about the problem of black boxes, but to some extent, juries, and talking to legal scholars was kind of interesting with this, that it's not so much about getting to the truth, it's having a system where you can reach a decision and you've got kind of that finality or semi-finality to that decision and having a system that works rather

• 29:07Speaker 03

  than, you know, you're 100% convinced of that. And I think we see that kind of across different fields of that emergence of truth, and as you said, what's kind of obvious and what's self-evident. I mean, one of the other things I found kind of interesting was how many mathematicians were deeply influenced by religion. So Newton, for example, Isaac Newton, driving all these equations and theories about planets and planetary motion, he saw that it was God keeping the universe in balance. And he was essentially just observing divine influence. So for him, although he was doing a lot of this scientific work, he saw that there was this external influence keeping it all in place along the way. So even quite far through history, you had these kind of other baseline explanations going

• 29:52Speaker 03

  on. I think even in the modern era, I think the way the way sometimes we tell the story of science, I think is sometimes almost a bit disingenuous. If you read a scientific paper, it's kind of, yeah, there's this problem, and I decided to run this experiment, and I got these results. But I think there's also just that element of like, why, what was the hunch that made you think that that might be an interesting thing to investigate? Or was that spark of inspiration? I think even in this era of AI, it's a really interesting question because AI can kind of process and mimic human decisions as we write them down. But I think there's often that kind of spark or that idea that would lead you to do something that just people wouldn't have tried before. And that's much, much harder to articulate. So it's not necessarily that

• 30:35Speaker 03

  kind of obviousness that we might have had in another era. But I think there still are those things which are quite hard to explain in where that evidence might have initially sparked from.

• 30:46Speaker 02

  One of the things that you mentioned was the use of proof and evidence in the legal system. And I feel like this was a really fascinating discussion in your book as well, where this is employed as like, you know, proof beyond a reasonable doubt, or innocent until proven guilty. What does this show us about like, the variable level of evidence needed to make a decision? And I guess like, the different forms that proof can take in this setting?

• 31:14Speaker 03

  It's a really interesting question about how different societies have even set that balance. Essentially, in a legal case, there's two main errors you can make. That someone can be guilty and you can let them go free, or they can be innocent, you can convict them. And William Blackstone, who was a legal scholar in the 1760s, came up with what's known as Blackstone's ratio. He said, it's better for 10 guilty people to go free than one innocent to be convicted. And Benjamin Franklin, actually, and sort of was even more cautious. He said, it's better for 100 guilty people to go free than one innocent to be convicted, seeing that as the kind of balance. Other cultures, particularly some communist regimes in 20th century, set it the other way. It's like, it was better for 10 innocents to be imprisoned than one guilty to go free. Because there's this kind

• 31:54Speaker 03

  of trade off on where they're seeing it as the worst error. And actually, some analysis looking at US legal cases, obviously, they don't try and target these error rates, but you can sort of infer. How people are valuing this, a lot of them seem to land between that kind of Blackstone and Franklin ratio of error. But there's of course, yes, the different evidence and how it makes its way into the courtroom, particularly some of the examples historically of kind of things like early probability. And again, one of the challenges here is what one scholar I talked to called the weak evidence problem. And I think a lot of how we navigate life is around probabilities that are quite likely. You know, a lot of probability theory was originally developed around like dice games

• 32:37Speaker 03

  and things, you know, you can study and you can quantify. But in legal cases, we often have this weak evidence problem where, you know, someone ends up in some extremely bad looking situation from a guilt point of view. And you're like, well, it's extremely unlikely this is just a coincidence. But then if you think about it, you're like, well, this person, you know, might just be a normal, everyday person. Well, it's extremely unlikely to that they're guilty. So you have these two extremely unlikely events and a lot of statistics just isn't equipped to handle that. And so there's this notion, it's called the prosecutor's fallacy, where people say, well, this is the probability that that will all be a coincidence and therefore that's the probability that innocent. But of course, you've got to weigh it against the fact that it's extremely unlikely they're guilty as well.

• 33:19Speaker 03

  And we see this even in other areas, so the work we do dealing with like emerging health threats and, you know, pre-COVID, there were some studies and actually we did a TV show where you sort of say, oh, it's, you know, a pandemic could just be around the corner. Or there was another study that the World Bank, I think, put it at 1% and you're like, well, what is that? That's, is that, are we, was that a good prediction? Was that a bad? And it's these, these very unlikely events. I think in legal cases, again, for that weak evidence problem, it's less about, do we definitively work out with high probability which this is true? And it's more just, we have to converge on the best explanation for what we've seen, given those two possibilities. And in reality, we, we may never have certainty about where we are.

• 34:04Speaker 03

  And I think it's something that kind of struck me, both thinking about that and then also thinking about a lot of people who, you know, have to plan for emergencies and very unlikely events, thinking a lot of the way we traditionally think about probability can very quickly lead us astray. Because I think we're so used to having this idea, well, I can just be 99% sure that, that this happened. But actually, it's much more about that balancing act that we have to perform.

• 34:28Speaker 02

  Let's take a quick break here. We'll be back before you know it.

• 34:32Speaker 01

  What is one thing many of us deal with every single day but rarely talk about? Dry hands. Whether you frequently wash your hands or deal with harsh weather, your hands can be left feeling dry, rough, cracked, and even uncomfortable. That's where O'Keefe's Working Hands Hand Cream comes in to help. The fast-absorbing formula goes on smoothly and it won't leave behind a greasy residue, so you can get back to your busy day without ever missing a beat. And because a little will go a long way, O'Keefe's Working Hands Hand Cream is effective, dependable, and economical, too. Whether you're dealing with harsh weather outside, demanding work, or everyday wear and tear, O'Keefe's Working Hands works just as hard as you do. So keep your hands hydrated, smooth, and feeling comfortable all season long.

• 35:17Speaker 01

  Visit O'Keefe'sCompany.com today. That's O-K-E-E-F-F-E-S company.com for guaranteed relief. O'Keefe's Working Hands. Yeah, it works.

• 35:30Speaker 04

  Hey, I'm Niall Rogers, producer, songwriter, guitarist, and co-founder of the group Sheik. A great song has a way of defining a moment: a moment of joy, heartbreak, or anything in between. This fall, I'll be driving across the country, dropping in on some of the coolest, most fascinating artists working today. We'll go deep into their lives and careers and hear what it was like before the moment that made them. From hitmakers like Miguel...

• 36:03Speaker 05

  That last year, finishing that record was the hardest experience of my life. Regardless of how everyone else buys into this, I believe in this and I'm gonna push.

• 36:15Speaker 04

  To rising stars like Alex Warren.

• 36:18Speaker 06

  You know, a song is a story, and the verse is setting the scene. And for me, at least, it's allowing me to be personal and vulnerable with the audience, whether it's about my wife and I'm being vulnerable about her, or it's about, you know, my parents and I'm being vulnerable to them. And it's almost like a cinematic piece in a way where each song tells that beautiful story.

• 36:36Speaker 04

  It's gonna be one hell of a trip. Experience Before the Moment, a Lincoln Original Series on the iHeartRadio app, YouTube, or wherever you get your podcasts.

• 37:01Speaker 02

  Welcome back, everyone. I'm here chatting with Dr. Adam Kucharsky about his book, Proof. Let's get into some more questions.

• 37:09Speaker 02

  Thinking about this in the context of COVID, when, you know, things were evolving very rapidly, the situation was evolving rapidly, and the general public and, you know, of course, government officials wanted answers and wanted decisions, you know, what is the best thing to do? Wear masks, not wear masks, sanitize groceries, all these things that were just constant questions and people wanting hard answers, like just yes, period, end of. As someone who was on the informational front lines of the COVID pandemic, what was your relationship with uncertainty like at that time? Did you struggle with feeling like we don't have enough information yet? You know, how did that feel, I guess, in your, in your position?

• 37:52Speaker 03

  Yeah, I think, I mean, those, those kinds of situations were enormously tant, both in terms of evidence generation and communication, and then obviously the political decision-making that comes off the back of it. I think in many of those situations, I found it useful to, you know, kind of convert, in some cases, uncertainty around the exact estimate to just, just to kind of broadly what situation we're in. So, for example, when, I think it was the Delta variant emerged, and we did a lot of the work identifying the early advantage it had. And it really wasn't, you know, was it 30%, was it 40%, was it 60%? But essentially, all of those were a big problem. And it's kind of arguing like, is this, you know, is this a disaster, or just a catastrophe, or just very, very bad? And it's like,

• 38:33Speaker 02

  Right, right.

• 38:34Speaker 03

  If, for a policy, you don't need to kind of necessarily communicate, you can just say like, we're very confident that it's going to take off. A couple of things, I think, that jumped out for me, I think, were, one was, The need to triangulate across data sources. I think sometimes people have this idea of science that you go out and you run a study and that study gives you the answer or it doesn't give you the answer. And there were quite a few, yeah, of the early skepticism were saying, well, actually, this study wasn't definitive and this study wasn't definitive. But once you start to look at all of those, you know, you start to look at evacuations flights, you start to look at the taste testing data and the contact tracing and the big testing of, you know, some of the cruise ships, you start to look at the clinical data, all of those signals start to drag you in the same way. And again, e- each bits of those evidence

• 39:18Speaker 03

  on their own might have problems, but you can start to bring together and draw that into a conclusion. I think we saw that across the pandemic that if you, if you view it very much as like, I'm going to get the perfect study and it's going to give me the answer, you'll struggle. But often you can actually find a lot of complementary data sources that all, you know, for variants or a lot of that early severity were all pointing in the same direction. I think it's harder, obviously, when they're, they're pointing in different directions, as we saw with, you know, some of the interventions where it was less clear because different countries, different economies, certain things did affect behaviour and other things in different ways. But I think the other challenge that kind of jumped out, and I think a lot of the health issues we deal with in US, UK and the modern era are non-contagious. So they're very much kind of individual, you know, things like

• 40:01Speaker 03

  cancer, things like heart disease. So it's very much individual focus. So you, you have someone who's ill, do you treat them? Do you not treat them? If you don't treat them, that's some, someone who's, one person who's going to get worse. But contagious health threats have this dependence where, you know, a problem can get worse and that problem can then accelerate in, in very different ways. I think that was something that was, was quite a challenge to communicate. So I think a lot of people had this notion of you've got normal life and then you could do something else that's not normal life and we'd obviously just prefer it to be normal. But I think as we saw globally, you didn't get that, that, that status quo. I mean, that was, that was gone. And, you know, no country had, they had varying levels of normality, but no country had like, just, you know,

• 40:43Speaker 03

  pretend absolutely nothing happened. You either had, in varying degrees, depending on the, the structure of society and advantages they had in kind of like terms of demography and healthcare and other stuff, big changes in behaviour or borders, whatever, or you saw a huge amount of death. And I think that's something that's. That can be, from an evidence point of view, much more challenging because I think just in life, we're much more used to those kind of linear problems where, you know, like with cancer or something, these are tragic events that happen, sort of distributed across the population, rather than something that the worse it gets, the worse that worseness accelerates.

• 41:18Speaker 02

  You mentioned how we have these different data sources, these different, you know, studies that are all leading us in a certain direction. And we have, by this point in time, developed ways to measure both the quantity and quality of evidence. I really enjoyed your discussion on randomized controlled trials because this quote unquote gold standard of medical studies, that might not always be the gold standard. And I was hoping you could tell me a little bit about the times when the true gold standard might not be, for instance, an RCT, but it might be something else entirely, or it might be unethical to do a randomized controlled trial in that situation.

• 41:57Speaker 03

  I think we've seen quite a lot of examples where treating it as that kind of cookie cutter, this is the only method we can use, kiles into problems. I mean, smoking and cancer is a very well-known one that we couldn't just have in action because you can't get that level of perfection. I mean, actually, even the first randomized controlled trial in modern medicine, which is 1947, so streptomycin, a trial for TB, Austin Bradford Hill, who led that, made the point that actually streptomycin had some very promising looking lab data and kind of early signals. And he suggested it would have been unethical to withhold it from patients if it was available. But actually, it's 1947, there were currency controls. The UK and its post-war state couldn't get enough dollars to buy streptomycin. There

• 42:41Speaker 03

  wasn't enough to go around. So in that situation, they said it would be ethical to randomize because there's not enough of it. So if there's not enough of it, you might as well randomize and just learn something along the way. And I think we've seen that in other situations, I mean, in other sort of examples that you see where things are very difficult to randomize, you can think about natural experiments. A lot of the, you know, well-known ones is the Vietnam draft, where people were essentially randomly assigned to go to war based on their birthdays. A lot of economists have done Nobel Prize winning work. Using that to understand the effects of war on subsequent life outcomes, because it's not something where you can fully design that experiment, but you can then make use of what you have available. So I think a lot of it just comes down to this issue of

• 43:25Speaker 03

  we want to understand cause effect. And the benefit of randomisation is a lot of the other things that would influence whether or not, you know, someone's getting a vaccine, someone's getting the disease, because you're randomising on the vaccine, on average, those will cancel out as effect. So it gives you that that quite neat benefit. But of course, you've also got the challenge that you might run a population in one group and one population that doesn't generalise to somewhere else. You've also got the time issue. So for diseases that evolve, you know, you might run a trial now against flu or Covid or something, a year later, that's going to be a different variant. And to what extent can you carry over those conclusions? I think we see a lot of examples in the literature where, for instance, someone might run a trial in one population for one disease, for flu, for

• 44:08Speaker 03

  example, and then see a very different result when people look at population patterns elsewhere, because it's a different immune structure, it's a different strain, it's a different time period. And, yeah, I think we can't just say, well, that study from a few years ago is the gold standard, we're only going to use that one. We have to think about how these things move along. I mean, that being said, though, I think in Covid, there were missed opportunities, I think, to gather much stronger data. I think it's it's very hard to justify running those kinds of studies as a threat increases. I think when epidemics going up, you know, taking your time to kind of try and randomise. I mean, I think essentially countries have to take that threat as the evidence suggests. But I think particularly as countries lifted measures, that was often just done in quite an ad hoc way. And we

• 44:51Speaker 03

  could have done much more kind of staging. In the UK, there were some early studies, for example, of can we use rapid tests? So people test themselves every day rather than quarantining for like a week or two. And then in practice, a lot of people just didn't bother. But apart from that, you know, I think there's there's a lot of these debates we're still having, and we probably could have got better answers for that with some higher quality studies. So not necessarily even an RCT, just just making use of what we had with more observations.

• 45:19Speaker 02

  One thing that I feel like during the COVID pandemic, especially the early months, was this desire from the public to have the answers. And I feel like there's a lot of variation in how willing someone is to say, I don't know. And I'm wondering your feelings on this. Do you feel that scientists in particular have a difficult time saying that they don't know the answer to something? Like, do we need to embrace uncertainty more in as scientists? Or do you feel like there's that we are embracing it, but just not communicating it well?

• 45:52Speaker 03

  Yeah, I think that's a really good question. It's kind of how, I guess, how personality and politics and all these things. And I think, I mean, there's been good reviews of evidence showing that the overstated certainty just undermines trust and confidence, whether it's kind of vaccines or it's other things. So saying, yeah, this is 100% safe, there's absolutely no risk. And if there's even a tiny risk, you then kind of undermine that. Yeah, one of the challenges with kind of that overstated certainty, I think particularly once you make that public statement, it's very hard to back that. And we saw that with some of the airborne, you know, some even health organizations say it's not airborne, fact. It's very difficult to then walk that back. But I think it's fine line because you don't just want to say, we have no idea. You want to try and communicate the way that

• 46:35Speaker 03

  evidence. I think some countries did that better than others, particularly Denmark, Singapore spring to mind on their reopening, where they said, this is the data we're looking at to do this, that might change. And this is kind of how we have to work things through. But I think one of the difficulty, I think, because any emergency that goes on for that long is, you know, you have some people who have very loudly said something's 100 times less severe than it is. And then they're kind of very nailed on to having to keep promoting that. And I think it is, there was one of the government advisory committees I sat on, you know, so a lot of early Alpha variant, early Delta variant, a lot of this early severity came out of this group. And there was a phrase that became used quite a lot, which was, tell me why I'm wrong.

• 47:19Speaker 03

  If you have that discussion where you want to get criticism, if you present stuff and especially people more senior and say, is this is this correct? It's very hard for people to kind of come in and say, oh yeah, actually I spotted a problem, especially if there's power dynamics or seniority and other things. So I think there was a lot of that thing where people present work and be like, right, tell me why I'm wrong. Tell me what I'm missing. And I think that's quite a healthy attitude in that kind of environment to be much more, you know, looking for weaknesses and be able to kind of lay out. And I remember, actually, I think it was when, was it the gamma variant is sort of emerging in Latin America? And I gave a immediate interview and when they wrote it up, it was basically, you know, Dr. Katschaski doesn't really know was the kind of, but in that situation,

• 48:03Speaker 03

  we we didn't. And it is hard to do because I think, you know, especially people asking you questions around your area of expertise. I think in terms of how to balance that, not just saying, I don't know, but saying, well, we do know this. And we can make some judgment. And there was this this wonderful study in the, it was nineteen fifty one, it's by the CIA analyst. And it was about words we use when we're unsure and words about judgment. And he basically realised that people use probable and possible to mean all sorts of things and they all, you know, had kind of different notions. And he said, yeah, humans will go out of their way to making a judgment about something. Um, that will often, you know, the risk is you get the uncertainty where we're like very hazy and like, oh, it's, you know, it's a definite possibility.

• 48:46Speaker 03

  And actually, in some cases, like with, you know, if you've got an emerging threat and you've got experts, you do actually want them to put a number on it. You know, even if there's uncertainty, you want them to say, I am sixty percent sure that this is the case. Um, and there's been a lot of nice work, you know, even around things like super forecasting where people make those predictions. And you can go back and then look, because, you know, if people are well calibrated in their uncertainty. Yeah, if you say you're fifty percent sure about a list of things. About fifty percent of the time, those things should happen. So about half the things on that list should occur. So there are these situations where I think we can get better just about thinking about our own uncertainty. And one of the things that I actually tried to do, I've tried to do in a lot of kind of emerging threats, is even just just writing

• 49:29Speaker 03

  down what you think's gonna happen. Because I think we're we're great, you know, the human mind at like, kind of rationalizing, oh, yeah, maybe I did think that. And so, yeah, I did quite, quite a lot of like, where, where you could state, I actually think the vaccine's gonna be pretty good. Or, you know, I think this, and like, this is where social media, when it was maybe slightly less polarized, was quite helpful, because you could just put a post out. And I think I was always very careful. I didn't delete any of my tweets during COVID, because I was like, I actually want that record. And the worst time I got wrong, you know, I was in Singapore in Feb 2020, and their policy was, don't wear a mask unless you have symptoms. And I think I tweeted, I was like, yeah, that seems like a sensible policy. That seems quite evidence based. And now we'd probably, you know, with

• 50:12Speaker 03

  some of the studies, not look back on that as as being the best post. But so, yes, I think it's almost that, as well as overstating certainty, I think it's also holding ourselves to account, even if it's just, you know, privately, about how confident we were and what played out.

• 50:28Speaker 02

  I wanna to close out by asking you about the subtitle of your book, which is The Art and Science of Certainty. And I want to know about the art part of this. What is, what is the, the art aspect?

• 50:42Speaker 03

  So I think for me, it was, the more I dug into this, the more I saw these other elements beyond kind of pure logic, pure observation coming in. I mean, even if you look at what was was essentially a bit of a mathematical civil war in the late 19th century, where a lot of these ancient Greek theorems, you know, things about the properties of triangles, started to break down, because people started to draw shapes on spheres and other structures, and come up with functions that these supposedly proven theorems didn't hold. And I think one of the reasons that was that was really controversial was there was this idea that there's a universal truth out there about the world. And actually, in this situation,

• 51:24Speaker 03

  it it kind of depended on what assumptions humans were making and what we were willing to to kind of define. And you're even in this supposedly pure pure subjects, there's still these these debates around, well, it kind of depends on which one you want to pick, and that will change the answer. I think even in in science, there's a lot of these these situations where, you know, we can accumulate the evidence, but then you have disagreement about where you you set the threshold. I mean, this kind of five percent cutoff has become very popular, this this sort of P value, or that, you know, the chance that you get a result that extreme, um, if there was nothing going on, or your no hypothesis was wrong. But that was kind of arbitrary. I mean, it was partly picked just for convenience, that this was, you know, a hundred years ago, the calculation was a bit easier if they picked a value, um, one official did a lot of this work,

• 52:09Speaker 03

  just easier to pick a value around, um, point zero five. And others who were more pragmatic, you know, working in in business on something, thinking, well, actually, the evidence is a bit weaker, but that's still useful to it. So there's this kind of human balancing act. And, you know, we saw, again, the things like legal cases, where how much you value different types of errors depends a lot on the individuals. I mean, one of the, um, examples that that I found fascinating in the book was Einstein, when he moved to the US, got very angry about peer review, because he sent something to a sent something to a journal, and it came back, it's like, oh, we've got another opinion on it. And he was like, whoa, whoa, whoa, like, why haven't you just accepted accepted my work? And actually, Max Planck, who published some of his, like, amazing early papers, Planck made that point

• 52:53Speaker 03

  that actually, I would rather kind of publish a few things that are a bit, you know, nonsense, that this is me paraphrasing, than miss a really important idea. So for him, his threshold was like, I want to set the threshold low, admittedly mainly amongst kind of physicists he he knew, because I don't want to set it too high and miss a good idea. And I think we all we all have those this kind of, that's where the art, I think, creeps in, that that kind of subjectivity in not just the evidence. I think one of the, for me, the real difference with something like proof is it's not just generating data, it's how that data interacts with the world and the decisions we make. And I think that's where things get really interesting. It's like, where do we actually set the bar for evidence? And then both to convince ourselves, but then go out and convince others to.

• 53:37Speaker 02

  Well, Professor Kucharski, thank you so much for joining me today. This was such an enlightening conversation, and I really did, I loved your book, Proof. So I appreciate you coming on to the show.

• 53:48Speaker 03

  Thanks. Great to talk.

• 54:08Speaker 02

  A big thank you again to Dr. Adam Kucharski for taking the time to chat with me. If you enjoyed today's episode and would like to learn more, check out our website, thispodcastwillkillyou.com, where I'll post a link to where you can find Proof: The Art and Science of Certainty, as well as a link to Dr. Kucharski's website, where you can also find his other book, The Rules of Contagion: Why Things Spread and Why They Stop. And don't forget, you can also check out our website for all sorts of other cool things, including, but not limited to, transcripts, Quarantini recipes, show notes and references for all of our episodes, links to merch, our bookshop.org affiliate account, our Goodreads list, a firsthand account form, and music by Blood Mobile.

• 54:51Speaker 02

  Speaking of which, thank you to Blood Mobile for providing the music for this episode and all of our episodes. Thank you to Liana Scolaci and Tom Bryfogle for our audio mixing, and thanks to you listeners for listening. I hope you liked this episode and are loving still being part of the TPWKY Book Club. A special thank you, as always, to our fantastic patrons. We appreciate your support so very much. Well, until next time, keep washing those hands.

• 55:46Speaker 01

  What is one thing many of us deal with every single day but rarely talk about? Dry hands. Whether you frequently wash your hands or deal with harsh weather, your hands can be left feeling dry, rough, cracked, and even uncomfortable. That's where O'Keefe's Working Hands Hand Cream comes in to help. The fast-absorbing formula goes on smoothly and it won't leave behind a greasy residue, so you can get back to your busy day without ever missing a beat. And because a little will go a long way, O'Keefe's Working Hands Hand Cream is effective, dependable, and economical, too. Whether you're dealing with harsh weather outside, demanding work, or everyday wear and tear, O'Keefe's Working Hands works just as hard as you do. So keep your hands hydrated, smooth, and feeling comfortable all season long.

• 56:31Speaker 01

  Visit O'Keefe'sCompany.com today. That's O-K-E-E-F-F-E-S Company dot com for guaranteed relief. O'Keefe's Working Hands. Yeah, it works.

• 56:44Speaker 07

  With ChatGPT, you can do more than get answers. You can get things done. Whether you're leading a big project, planning an elaborate trip, or trying to balance work and life, ChatGPT can help. Give it a goal, plus things like files, notes, or project plans, develop a trip itinerary with daily updates, or create a prioritized schedule. So you can spend less energy sorting out the details and more on bringing your biggest ideas to life. Get started at ChatGPT.com.

• 57:11Speaker 08

  This is Colin Cowherd from the Colin Cowherd Podcast. Buy smart, drive for less, and trade for more. That's Toyota affordability. All starts with a great deal on an off-road Ready Tacoma or a half-ton Tundra, two of Toyota's most capable and longest-lasting trucks. Built on a reputation for quality and reliability, Toyota trucks are known for holding their value, which could mean big savings when it's time for your next Toyota. For a great deal today to a great value tomorrow, shopToyota.com. Toyota, let's go places.