Daniel Kahneman: Algorithms Make Better Decisions Than You

Daniel Kahneman won the Nobel Prize for proving we're not as rational as we think. In this timeless conversation we discuss how to think clearly in a world full of noise, the invisible forces that cloud our judgement, and why more information doesn't equal better thinking. Kahneman also reveals the mental model he discovered at 22 that still guides elite teams today. Approximate timestamps: (00:36) – Episode Introduction (05:37) – Daniel Kahneman on Childhood and Early Psychology (12:44) – Influences and Career Path (15:32) – Working with Amos Tversky (17:20) – Happiness vs. Life Satisfaction (21:04) – Changing Behavior: Myths and Realities (24:38) – Psychological Forces Behind Behavior (28:02) – Understanding Motivation and Situational Forces (30:45) – Situational Awareness and Clear Thinking (34:11) – Intuition, Judgment, and Algorithms (39:33) – Improving Decision-Making with Structured Processes (43:26) – Organizational Thinking and Dissent (46:00) – Judgment Quality and Biases (50:12) – Teaching Negotiation Through Understanding (52:14) – Procedures That Elevate Group Thinking (55:30) – Recording and Reviewing Decisions (57:58) – The Concept of Noise in Decision-Making (01:01:14) – Reducing Noise and Improving Accuracy (01:04:09) – Replication Crisis and Changing Beliefs (01:08:21) – Why Psychologists Overestimate Their Hypotheses (01:12:20) – Closing Thoughts and Gratitude Thanks to MINT MOBILE for sponsoring this episode: Get this new customer offer and your 3-month Unlimited wireless plan for just 15 bucks a month at MINTMOBILE.com/KNOWLEDGEPROJECT. Newsletter - The Brain Food newsletter delivers actionable insights and thoughtful ideas every Sunday. It takes 5 minutes to read, and it’s completely free. Learn more and sign up at ⁠⁠⁠⁠⁠⁠fs.blog/newsletter⁠⁠⁠⁠⁠⁠ Upgrade — If you want to hear my thoughts and reflections at the end of the episode, join our membership: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠fs.blog/membership⁠⁠⁠⁠⁠⁠⁠⁠ and get your own private feed. Watch on YouTube: ⁠⁠⁠⁠⁠⁠@tkppodcast Photograph: Richard Saker/The Guardian Follow Shane Parrish X ⁠⁠⁠⁠⁠⁠@ShaneAParrish⁠⁠⁠⁠⁠⁠ Insta ⁠⁠@farnamstreet⁠⁠ LinkedIn ⁠⁠Shane Parrish⁠ Learn more about your ad choices. Visit megaphone.fm/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Delay your intuition. Don't try to form an intuition quickly, which is what we normally do. Focus on the separate points. And then when you have the whole profile, then you can have an intuition and it's going to be better.

• 0:20Shane Parrish?

  Welcome to The Knowledge Project. I'm your host, Shane Parrish. In a world where knowledge is power, this podcast is your toolkit for mastering the best what other people have already figured out. So you can use their insights in your life.

• 0:36Shane Parrish?

  Before we get into the interview, I want to tell you about a moment that didn't make it into the episode. I first came across Daniel Kahneman's work in the early 2000s. His impact on me and so many people around the globe has been unbelievable. By the time I sat down with him in his New York City home in 2019, I had so many questions for him. Kahneman won a Nobel Prize in Economic Sciences in 2002, yet he never took an economics course. His central message was very simple. If we want to make better decisions, we need help. Danny died last year on March 27, 2024. He was 90. This conversation is now one of the final opportunities to hear directly from one of the most influential

• 1:20Shane Parrish?

  thinkers of our time. I get messages about this episode every week. People come away with new insights on everything from life to decision making. I re-listened to it recently and it's timeless. That's exactly why I'm republishing it. Consider loss aversion one of his most important discoveries. Why does losing $100 hurt twice as much as gaining $100 feels good? The asymmetry affects everything. It affects your stock portfolio, your golf game. Check your portfolio when it's down and you'll start making emotional decisions. A golfer putts better for power than for birdie. But here's what happened near the end of our interview. Danny's phone rang and it was loud. He'd forgot to turn it off.

• 2:04Shane Parrish?

  And we're almost done the interview at this point, but he answered and someone obviously wanted him to give a talk. Or review a book. He ended the call with words that have stayed with me since then. My rule is I never say yes on the phone. I'll get back to you tomorrow. I wanted to discuss that on air, but we ran out of time. As I packed up my gear, I asked him about that. This rule was a trick to avoid saying yes intuitively. It gave him time to think. He's always bombarded with requests and he often says yes when he didn't want to. At first, he would try saying no, that date doesn't work, that timeline doesn't work. But what happened in those moments was it turned into a negotiation. What about another date, another timeline? So he hit on this rule.

• 2:47Shane Parrish?

  And to me, this is his most practical discovery. Most people don't even know about it. This rule lets you reprogram your unconscious mind. Your desired behavior becomes your default behavior, and that's incredibly powerful. It changed my life. I now exercise every day. It's actually easier than three times a week. The activity, duration and scope can change, but working out and exercising doesn't. I think I've missed five days in five years at this point. And I talk about this in my book, Clear Thinking, and the concept has changed so many lives, including my great friend Brent B. Shore. In episode 196, we talk about this a little. Several parts of this conversation stuck out when I was re-listening to it. First, we talk about happiness versus satisfaction.

• 3:32Shane Parrish?

  Happiness is feelings. It's mostly social. Am I with the people who love me and whom I love back? Satisfaction, on the other hand, is how you feel about your life, your job, your career, conventional aspects. Danny argued people want satisfaction more than happiness. Second, changing behavior. Make good behavior easier and bad behavior harder. The insight, all behavior is equilibrium. Rather than pushing people to change, ask why they aren't doing it already. Third, behavior is situational. Want to understand behavior? Look at the situation. When someone acts in ways that don't make sense, ask yourself, what would the world have to look like for that behavior to make sense?

• 4:17Shane Parrish?

  Fourth, agents making decisions on your Be half, beat you at certain types of decisions. They have no sunk costs. They have no emotions. Brian Johnson talks about this in episode 188. He turned his health decisions over to effectively an algorithm, because that algorithm makes better decisions than he does. Fifth, our beliefs are formed by people more than facts. We agree with people we like, despite the facts. It's easier to believe a lie from someone you like than a truth from someone you dislike. We form identity beliefs, liberal, conservative, Democrat, Republican. They can do no wrong. If they're wrong, we're wrong, and we can't handle that. Finally, intuition.

• 5:02Shane Parrish?

  Danny had talked about this so much, his answers sounded repetitive. So I framed my question on this to include his typical answer in the question, forcing him to think a little deeper. Whether this is your first listen or your third, you'll come away with ideas that you can use in life.

• 5:38Shane Parrish?

  Daniel, I'm so happy to get a chance to talk to you.

• 5:41Speaker 00

  Well, I'm happy to have you here.

• 5:44Shane Parrish?

  What was your childhood like? What were you like as a child?

• 5:48Speaker 00

  Oh, my God. That was a long time ago. I was, I was a nerdy child, as you might expect, I suppose. I was, I thought I'd be a professor when I was like three or four years old, because people told me I would be, because I probably spoke with long words and stuff like that. So, and then the rest of my childhood, I mean, I was five when World War II began. So, and I was a Jew in France. So I've had a difficult childhood, but from that point on. But, but I was, was I like? Yeah, I was, I was a nerdy child. I was quite inept physically.

• 6:32Speaker 00

  Very fortunately for me, when I'd finally moved to Israel at age 12, they held me up a grade and then, and that was all right. But that's. That's what I was like.

• 6:44Speaker 02

  Are there any particular lessons or memories that stand out for you?

• 6:47Speaker 00

  There are two of them that I speak about. So one is that I was, I was a psychologist very early on. That was, that was very clear. I, I wrote an essay, um, before I was 11. I remember where because it was, it was a German counterattack. It was during that period, we were in Paris. And I wrote an essay about faith and religion. And it was a very pompous essay. I had a little book that was, that was titled, What I, What I Write About What I Think, something pompous like that. But the essay started with another pompous thing that I quoted Pascal.

• 7:32Speaker 00

  My, my sister had passed her exams and I had read, now she's studied some Pascal and I had read it. And Pascal had said that faith is God made sensible to the heart. And, you know, little me, I said, how true. That's what my essay said. And then, but then I said, but faith is really hard to get. You don't say it's God all the time. So that's what religious pomp is for. Cathedrals, organ music, they give you, and I call that Ozat faith.

• 8:13Speaker 00

  Sort of a substitute faith because it's a, it's a similar feeling. It's got to do with God and that's what you must do with. That's, that's a psychologist. So it, it's clear that, you know, that was my calling. And so that's one significant memory of my childhood.

• 8:35Speaker 02

  So you wanted to be a psychologist?

• 8:37Speaker 00

  I think so. I think so. I mean, I, you know, it's always had that point of view that when later, as a teenager, I was, you know, interested in all the philosophical issues like, you know, does God exist? And what's good and bad and stuff like that. And why shouldn't we masturbate? You know, serious questions. Um, but, but I discovered that actually.

• 9:04Speaker 00

  I was less interested in the question of whether or not God exists than in why do people believe that he exists? That I thought was interesting. And I wasn't particularly interested in the question of what's good or bad, but I was really interested in what makes people angry and indignant. So, you know, I've had the psychological point of view since, turns out, since my childhood.

• 9:27Speaker 02

  Was there anybody that sort of influenced you to go on to study this? I mean, it's one thing to have these dreams as like a 12, 13, 14 year old boy. It's another to turn this into, you know, probably the most eminent career that's ever happened for a psychologist.

• 9:43Speaker 00

  No, not the most eminent career. You know, and I wasn't sure actually that I would do psychology. When I took a vocational exam to tell me what I was good at and psychology and economics turned out, but you know, that was unexpected. I was, and then I took psychology as an undergraduate and mathematics at which I was not particularly good. So, and no, it's not that I knew at the time that, you know, I had that calling to be a psychologist. Didn't occur to me. I thought, you know, I thought I'd be a professor in one thing or another. I mean, that's, I thought I'd be an academic, but not psychology specifically.

• 10:25Speaker 02

  You worked for, with Amos Tversky for a long time. Are there any particular stories that you remember about working with him that bring a smile to your face?

• 10:35Speaker 00

  Almost everything about working with him brings a smile to my face. You know, he was a very unusual person. Most people who knew him thought that he was the smartest person that ever met. And in fact, famously, psychologist Nick Nesbitt said that it's sort of an intelligence test. And he said that when you are with Amos, how long does it take you to figure out that he's smarter than you are? And the faster you figure that out, the smarter you are. So, you know, he was, he was super bright and very, very funny. He joked a lot. He laughed a lot at his own jokes, and that was infectious.

• 11:18Speaker 00

  When I was with him, I was very funny, too. More than half of the laughs of my life, of my lifetime I've had during the 10 years I worked with him.

• 11:29Shane Parrish?

  You know what doesn't belong in your summer plans? Getting burned by your old wireless bill. While you're locking in itineraries, your wireless bill should be the last thing holding you back. That's why I recommend the switch to Mint Mobile. No gimmicks, just a smarter play most people overlook. You make smart trade-offs everywhere else in life. Why not here? All plans come with a high-speed data and unlimited talk and text, delivered on the nation's largest 5G network. Use your own phone with any Mint Mobile plan and bring your phone number along with all your existing contacts. It's a small switch that compounds over time, just like any great decision. Most people leave money on the table. This is how you stop doing that. If I had needed this product, it's what I'd use.

• 12:13Shane Parrish?

  The savings are unbeatable. This year, skip breaking a sweat and breaking the bank. Get this new customer offer and your three-month unlimited wireless plan for 15 bucks a month at mintmobile.com/knowledgeproject. That's mintmobile.com/knowledgeproject. Upfront payment of $45 required, equivalent to $15 a month. Limited time new customer offer for first three months only. Speeds may vary. About 35 gigabytes on unlimited plan. Taxes and fees extra. See Mint Mobile for details.

• 12:44Speaker 03

  The new LinkedIn Hiring Pro can't undo your last hire. The human postponer. They were the master of one phrase. I'll circle back on that. But three months later, you are the one doing all their work and wondering how big that circle is. But LinkedIn Hiring Pro can take the hiring load off your plate by automating the hiring busywork from the initial job post to scheduling interviews. Hire right the first time with LinkedIn Hiring Pro. Post a free job today at LinkedIn.com/quality.

• 13:14Speaker 02

  You have an interesting distinction between happiness and satisfaction. Can you walk us through that?

• 13:20Speaker 00

  Yeah, sure. I mean, the word happiness is so ambiguous and it means so many things to many people. But one sensible interpretation of it is that it's got to do with your emotions, with how you feel, with the emotional tone of your life, whether it's a happy life, that, you know, it's pleasant to be you. Life satisfaction is a completely different thing. I mean, life satisfaction is how do you feel about your life when you think about your life? And most of the time, we don't think about your life, you just live. But, you know, sometimes you sort of look and that's, that's when you determine how satisfied you are. That's life satisfaction.

• 14:05Speaker 00

  It's not satisfaction, it's life satisfaction.

• 14:08Speaker 02

  Should we balance the two? Or how would you think about them? Should we be more happy when we're younger, more satisfied when we're older?

• 14:15Speaker 00

  That thought had never occurred to me when I began to work on this. So I started out thinking that happiness, in that sense, that's how you feel when you live. But that was reality and that life satisfaction was just stories that people tell themselves. And the, the important thing was to be happy in real time. But later, when we did more research, it turned out that the circumstances that make people happy and the circumstances that make them satisfied with their life are not the same. So happiness is mostly social. It's, you know, it's being with people you love and who love you back.

• 14:58Speaker 00

  That's, that's a lot of what happiness is. Life satisfaction is much more conventional, is to be successful. And, you know, so it's money, education, prestige, that sort of thing is what life satisfaction is about. So those are two very different things. I thought that life satisfaction is irrelevant. You know, that's how I began. And we, we had a research program where we were, we were trying to, you know, to show that this is the case. But then, after a few years, I realized that what people really want in their life is they don't seem to care about how happy they'll be. They seem to want to be satisfied with their life.

• 15:43Speaker 00

  They seem to want to have a good story about their life. And then I was in the position of saying that to define well-being in a way that people didn't seem to care particularly about. So that was not a tenable position. So I I dropped back into saying that I had no idea how to deal with it.

• 16:02Speaker 02

  Was this a result of the research? You did some research that was, I think it said above seventy thousand, you don't become happier, but do you become more satisfied?

• 16:11Speaker 00

  No, the research I did with Angus Deaton at Princeton, famous economist, we showed that in terms of happiness, in terms of emotional tone, positive and negative, having a lot of money doesn't make you happier, but being poor makes you miserable. So that's above a threshold that was like seventy thousand dollars approximately in the US. Then extra money didn't make you emotionally happier. But with life satisfaction was a different story. With life satisfaction, that doesn't satiate. So it's always good to have more. Because basically, I think

• 16:55Speaker 00

  money is a proxy for success. And it's a proxy for subjective success in many cases.

• 17:02Speaker 02

  So it's not necessarily about spending it or doing something with it. It's just a measure.

• 17:06Speaker 00

  Just getting it. I mean, you know, you look at all those people, those billionaires working their heads off. And they're clearly not doing this because they need more money. They're trying to get more money. And they're trying to get more money because that will be an indication that they're good at what they do. I think mostly it's a proxy.

• 17:24Speaker 02

  Do either of those variables correlate to longer living? Happiness or satisfaction?

• 17:30Speaker 00

  Both, apparently. But, you know, it's it's hard to separate. And I haven't been followed. You know, shortly after this deciding that I didn't know what well-being was, I sort of stopped doing research on this. So I haven't been following. But I think. There's clear evidence that being effectively happy, you know, is very good for you. And you do live a little longer and you make better and so on. And life satisfaction works in the same direction. Whether it's separable, which of them, you know, is more important, that I don't.

• 18:06Speaker 02

  I want to switch gears a little bit and talk about behavior. And I'd love your insider expansion upon the idea of we can change behavior. And how do we go about changing our behavior?

• 18:19Speaker 00

  Well, you know, I'm not sure I buy the premise. I think changing behavior is extremely difficult. The, there are a few tips and, you know, a few guidelines about how to do that. But anybody who is very optimistic about changing behavior is just deluded. It's, it's hard to change other people's behavior. It's very hard to change your own, not simple.

• 18:42Speaker 02

  This is what marriage is all about, right?

• 18:44Speaker 00

  No, the things, you know, people, when, when, you know, married people try to change each other's behavior.

• 18:52Speaker 02

  It's a lot of dissatisfaction.

• 18:53Speaker 00

  They are not on their way to a good marriage, I think.

• 18:56Speaker 02

  We'd all be happier with lower expectations.

• 18:58Speaker 00

  Yes, I mean, and, and even if you have expectation, don't try to change. Because, you know, it's very unlikely to work and in a significant way.

• 19:08Speaker 02

  I can think of the common ways that we would sort of go about behavior change. And it would be, uh, you know, making good behaviors more easy or negative behaviors harder.

• 19:19Speaker 00

  I think that's the main, the main insight. You know, when you want to influence somebody's behavior, that's a very big insight. I've always thought that this is the best psychological idea ever, you know, so far as I'm concerned. But it's that when you want somebody to move from A to B in terms of their behavior, you can think of it that there are two ways of doing it. You can push them or you can ask the question, why aren't they doing B already?

• 19:50Speaker 00

  Which is an unusual question, but you know, why? So then, when you ask, why, why not? Why aren't they doing B they ought to? As they think they ought to. Then you get a list of what Koudlouin, that's the psychologist who, my guru on this, my hero, and many people's hero, he spoke of restraining forces. I mean, so there are reasons why they're not where you want them to be. So he spoke of behavior as an equilibrium. There are forces that are pushing you one way, forces that are pushing you the other way. So how loud you speak, how fast you drive. It's easy to think of it as, as an equilibrium.

• 20:33Speaker 00

  And what we tend to do when we want to move people from A to B is we push them. We add to the driving forces. And Koudlouin's insight was that this is not what you should do. You should actually work on the restraining forces and try to make them weaker. And that's a beautiful point. And he showed, he had that image that, you know, I've had since I was an undergraduate. And I'm not sure actually whether it was his image or something that I drew from reading him. But it's like you have a plank and it's being held by two sets of springs. You know, you want it to move one direction.

• 21:16Speaker 00

  And so you could add another spring that would push it that way. Or could remove one of the springs that are holding it back. And the interesting thing, and that's the striking outcome, is when it moves, if it moves because of the driving force, you've added to the driving force, then at equilibrium, it will be in a higher state of tension than it was originally. That is, because you've compressed one spring and such pushing back harder. But if you remove a restraining force, at equilibrium, there'll be less tension in the system. Must have been 20 years old, I thought, that's just so beautiful.

• 21:56Speaker 02

  What do you wish that everybody knew about psychology that you don't think that they do? If that was class one, what's class two?

• 22:04Speaker 00

  You know, class two, which is a development from class one, you know, it's, it's the same idea extended. Class two is that behaviors don't necessarily reflect.

• 22:18Speaker 00

  The personality, but behaviors have a lot to do with the situation. And so if people behave in strange ways, look at the situation they're in and what are the pressures in the situation that make them act this way. So there is a bias that the social psychologist, well-known social psychologists call the fundamental attribution error. And that means that when you see people acting in some way, you think that it's because of their personality that they do it. That may not be the case. It's quite likely that the situation is making them do it. I'd like people to know that motivation is complex and that people do good things for a mixture of good and bad reasons,

• 23:02Speaker 00

  and they do bad things for a mixture of good and bad reasons. And I, I think that there is a point to educating people in psychology is to make them less judgmental. Just have more empathy and more patience, and being judgmental doesn't get you anywhere.

• 23:21Speaker 02

  When you talk about situational, one of the things that comes to mind is it's so easy for us to give our friends advice. But if we were in that situation, we might not necessarily see it. Why, why is that the case? Why is it so much easier to give other people advice?

• 23:34Speaker 00

  I mean, feelings get in the way of clear thinking. There is a phenomenon that we call the endowment effect, which is that when I've asked for more money to sell you my sandwich than I've paid to get it. I mean, that's essentially the endowment effect. And our explanation of it, there are many explanations, but the story I like to tell about it is that it's more painful to give something up than to get something. But there is an interesting result that if you have an agent making decisions on somebody's behalf, that agent doesn't have loss of those. So that agent sells and buys at the same price, which is the economically rational thing to do. Where this goes into policy and government and really important things.

• 24:20Speaker 00

  That governments are like agents or people who think about the good of society. And agents, they take the economic view. They take the view of what things will be like at the end. They don't figure out that there are some people who are going to be losing because of the reform that they make. And it turns out that you can really expect losers, potential losers, to fight a lot harder than potential winners. And that's the reason that we form so frequently fair. And that when they succeed, they're almost always way more expensive than anticipated. And they're more expensive because you have to compensate the losers. And that frequently is not anticipated.

• 25:02Speaker 00

  So that's an example of a story about that incorporates behavior change. And the difference between perspectives, between being, you know, in the situation, feeling the pain of giving up the sandwich and not feeling the pain of giving up the sandwich.

• 25:23Speaker 02

  That would have huge public policy sort of implications too, right? That we don't tend to think about or discuss. That's a really interesting angle there. I'm going to come back to sort of situational decision making based on sort of like what we see is all there is. And we have these feelings that we can't sort of disassociate with. How does environment play a role, like the physical environment, in sort of what we decide or does it?

• 25:48Speaker 00

  I mean, you know, there are sort of obvious things that we know. If people are hot and bothered and distracted and there is a lot of noise and so on, then they'll think as well. And that we know that's... But even there, there are puzzles. I mean, many people think and work a lot better in cafes, you know, where there is actually ambient noise and activity around them. And it helps them concentrate better. So there isn't a very simple story of the environment. But certainly you can make the environment tough enough so that people won't be able to think properly. That's feasible.

• 26:28Speaker 02

  Are there things that we could do to, I guess, push the environment to be more conducive to clearer thinking? The physical environment in this case?

• 26:39Speaker 00

  Oh, there are all sorts of, you know, odd findings, you know, the color of the color of the room. Some colors are better than others. And you would expect that some colors are more calming than others. So you wouldn't want to be in a red room.

• 26:56Speaker 02

  Making decisions.

• 26:57Speaker 00

  Making, making decisions. But, you know, those are extreme and minor effects.

• 27:03Speaker 02

  I want to come to intuition and noise later. Is there anything else that stands out that gets in the way of clear thinking that we can sort of bring to the surface now?

• 27:13Speaker 00

  Well, you know, what, what gets in the way of clear thinking is that we have, we have intuitive views of almost everything. So as soon as you present a problem to me, I have, you know, I have some ready made answer. And what gets in the way of clear thinking are those ready made answers. And we can't help but have them. So that's one thing that gets in the way. Emotions get in the way. And I would say that independent clear thinking is to first approximation impossible. I mean, in the sense that, you know, we believe in things most of the time, not because we have good reasons to believe them. If you ask me for reasons, I'll explain you.

• 27:58Speaker 00

  I'll, I'll always find a reason, but the reasons are not the causes of our beliefs. We have beliefs because mostly we believe in some people and we trust them and we adopt their beliefs. So we don't reach our beliefs by clear thinking. Some, you know, unless you're a scientist or doing something like that.

• 28:20Speaker 02

  But even then, it's probably a very narrow.

• 28:22Speaker 00

  But that's very narrow. And there is a fair amount of emotion in your scientists as well that gets in the way of clear thinking. You know, commitments to your previous views, being insulted that somebody thinks he's smarter than you. I mean, lots of things get in the way than when you're a scientist. So I'd say there is less clear thinking than people like to think.

• 28:46Speaker 02

  Is there anything that we can do at the belief formation stage? Like, it sounds almost as though when you say that we're reading a newspaper, we read this op ed and it's well constructed and fits with our view of the world. Therefore, we adopt that opinion and we forget the context that we didn't learn it through our own experience or reflection. We learned it sort of from somebody else. So we don't know when it's sort of likely to work or not work, but we just proffer that as our opinion, is there?

• 29:18Speaker 00

  That's how I believe in climate change. You know, I believe in the people who tell me there is climate change and the people who don't believe in climate change, they believe in other people. So.

• 29:30Speaker 02

  But similarly, there's like fake news and all this other stuff that we would have the same reaction to.

• 29:35Speaker 00

  You know, but, but I'm much more likely to believe fake news on my side than the fake news on, on the other side. I mean, it's true that there is a huge degradation in, in public discourse in, in the recent 10, 15 years in the United States. I mean, there used to be an idea that facts matter.

• 29:59Speaker 02

  What would be your hypothesis as to why that, that is playing it? Without getting into politics, because I don't want to talk politics, but like, why is that?

• 30:06Speaker 00

  Well, I mean, it's hard to, it's hard to answer that question without, without politics because it's a general political polarization has had a very big effect. And the fact that people can choose the sources of information.

• 30:25Speaker 02

  Let's switch gears a little bit and talk about intuition. I think one of the, the things that strikes me the most about some of the work that you've done is the cases where we're likely to trust our intuition and when we're not. And so if I'm, correct me if I'm getting this wrong. So it's sort of like a stable environment, repeated attempts and rapid feedback. It strikes me that most decisions made in organizations do not.

• 30:59Speaker 02

  Fit that environment. And yet, we're making a lot of these decisions on judgment or experience. What are the ways that we can sort of make better decisions with that in the context?

• 31:14Speaker 00

  Well, in the first place, I think, you know, you shouldn't expect too much.

• 31:18Speaker 02

  And that's a low expectation.

• 31:20Speaker 00

  Think through, yeah, should have low expectations about improving decisions. I mean, there is, you know, one basic rule is slow down, especially if you if you have that immediate conviction, slow down. There are procedures, you know, there are ways of reaching better, better decisions, but reaching better judgments, and we can talk about them.

• 31:43Speaker 02

  I would love to hear.

• 31:44Speaker 00

  If you really want to improve the quality of decision making, use algorithms. I mean, whenever, wherever you can, if you can replace judgments by by rules and algorithms, they'll do better. And there's big social costs to trusting, allowing an algorithm to make decisions, but but the decisions will likely to be better. So that's one thing. If you can't use algorithms, then you slow yourself down. And then there are things that you can do for certain types of problems, and there are different types of problems. So one class of problems, like forecasting problems.

• 32:25Speaker 00

  A friend, Phil Tetlock, you know, has that book on super forecasters, where he identifies with people who are good at forecasting the future. What they do that makes them good, and, you know, it tries to train people and it can improve people. So that's one class of problems. I'm interested specifically in another kind of problem, judgment problems. Where basically you're considering options or you're evaluating a situation and you're trying to give it a score. There, there, there is advice, I think, on how to do it. For me, it goes back to something I did in the Israeli army when I was like 22 years old.

• 33:09Speaker 00

  So that's a long time ago, like 63 years ago. I was a psychologist in the Israeli army and I was assigned.

• 33:18Speaker 00

  The job of setting up an interviewing system for the army. It's ridiculous, but you know, this was the beginning of the state of Israel. So people were improvising all over the place. So I had a BA and I was, I think I were the best trained psychologist in the army. My boss was a chemist. Brilliant, but anyway. And the existing system was one where people would interview and try to form an intuitive global image of how well that recruit would do as a combat soldier. It was the objective, the object of the interview. And because I had read a book by Paul Neill, I took a different tack.

• 34:00Speaker 00

  And the different tack was I identified six traits that I sort of made up. And I had them ask questions and evaluate each of these traits independently and score it and write down the score, then go on to the next trait. And they had to do it for all six traits. And that was, that's all I asked them to do. And the interviewers, who were about one year younger than I, all recruits, but very, very smart, selected for being good at it. They were furious with me and they were furious with me because they wanted to exercise their intuition. And I still remember that one of them said, you're turning us into robots.

• 34:42Speaker 00

  So I compromised with them and I said, okay, you do it my way. And I told them, you try to be reliable, not valid. You know, I'm in charge of validity, you be reliable, which was pretty arrogant, but that's, that's how I presented it. But then when you're done, close your eyes and just put down a number. How good a soldier has that guy going to be? And when we validated the results of the interview, it was a big improvement on what had gone on before. But the other surprise was that...

• 35:20Speaker 03

  The new LinkedIn hiring pro can't undo your last hire. The human postponer. They were the master of one phrase. I'll circle back on that. But three months later, you were the one doing all their work and wondering how big that circle is.

• 35:34Speaker 03

  But LinkedIn Hiring Pro can take the hiring load off your plate by automating the hiring busywork from the initial job post to scheduling interviews. Hire right the first time with LinkedIn Hiring Pro. Post a free job today at LinkedIn.com/quality.

• 35:50Speaker 04

  College football is back. So Hilton called and me, the superstition concierge, to make your fan rituals a reality. Need a room to match your lucky number? We got you. Want to make sure our team doesn't wash your lucky jersey? Oh, that smells lucky. Hilton's unmatched hospitality can keep up with any superstition. Even a marching band wake-up call at 5:55 in 55 seconds. Hit it! When you need a team that will do whatever it takes on game day, it matters where you stay. Hilton, for the stay.

• 36:20Speaker 00

  The final intuitive judgment added, it was good. It was as good as the average of the six straight and not the same. It added information. So actually, we ended up with a score that was half, was determined by the specific ratings, and the intuition got half the way. And, and that, by the way, stayed in the Israeli army for well over 50 years. I don't know whether it's, I think it probably, some version of it was still being forced. But around 15 years ago, I visited my old base and, and the commanding officer of the research unit was telling me how they run the interview.

• 37:02Speaker 00

  And then she said, and then we tell them, close your eyes. So that, that had stayed for 50 years. Now, the close your eyes and that whole idea is now the basis of the book that I'm writing. So actually, I have the same idea, really, that when you are making decisions, you should think of options as if they were candidates. So you should break, break it up into dimensions, evaluate each dimension separately, then look at the profile. And, and the key is, delay your intuition.

• 37:37Speaker 00

  Don't fight the form and intuition quickly, which is what we normally do. Focus on the separate points. And then when you have the whole profile, then you can have an intuition and it's going to be better. Because people make form intuitions too quickly. And the rapid intuitions are not particularly good. So if you delay intuition until you have more information, it's going to be better.

• 38:02Speaker 02

  I'm curious how we delay intuition.

• 38:04Speaker 00

  You delay intuition by focusing on the separate problems. So our advice is that if you have, you know, a board of directors making decisions about an investment, we tell them, you do it that way. Take the separate dimensions and really think about each dimension separately and independently. And don't allow, you know, if you're the chair, don't allow people to give their final judgment. Say, we'll wait until we cover the whole of it. I mean, if you find a deal breaker, then you stall. But if you haven't found a deal breaker, wait till the end and look at the profile.

• 38:47Speaker 00

  And then your decision is almost certainly going to be better.

• 38:52Speaker 02

  Does that include weighting the different aspects of the problem differently? Or do you highlight that in advance?

• 38:57Speaker 00

  Yeah, I mean, it makes you see the trade offs more clearly. Otherwise, when we don't follow that discipline, there is a way in which people form impressions. Very quickly you form an impression. And then you spend most of your time confirming it instead of collecting evidence. And so if accidentally your impression was in the wrong direction, you're going to confirm it. And you don't give yourself a chance to correct it. Independence is the key. Because otherwise, when you don't take those precautions, it's like having a bunch of witnesses to some crime

• 39:41Speaker 00

  and allowing those witnesses to talk to each other. They're going to be less valuable if you're interested in the truth than keeping them rigidly separate and collecting what they have to say.

• 39:53Speaker 02

  What have you seen work in a repeatable way, maybe a particular organization or cross organizations, to not only reliably surface disconfirming evidence, but then place a value on what is surfaced instead of being dismissive? Is there a framework for that? Is there?

• 40:16Speaker 00

  Well, yeah, there are many, you know, there are many procedures like red team, blue team, a devil's advocate. I mean, there have been, you know, many attempts. In, in general, you know, if you're, if you're the head of a group that makes decisions, one of your missions would be to protect the dissenters because they're very valuable and you should make it painless to dissent or as painless as possible because it's hard to be said. It's painful and costly. So protecting dissenters is important.

• 40:53Speaker 02

  I'm curious about the distinction between intuition and judgment. You, you had mentioned intuition, judgment, intuitive judgment. Can, can you walk me through some of like how those differ?

• 41:08Speaker 00

  It's a bit hard to separate. And judgment is, is what you do when you integrate a lot of information informally into a score of some kind. We speak, we being my co-authors, you know, in the book we're writing, we speak of judgment as measurement, but it's measurement where the measuring instrument is your mind, but you do it informally. And because you do it informally, people are going, are not necessarily going to agree. So wherever we say it's a matter for judgment, we're allowing for differences, for variability.

• 41:48Speaker 00

  Now judgment can be more or less slow, more or less systematic. So at one end, you have pure intuition, where you allow the judgment to go very quickly and so on. And at the other end, you try to delay intuition. But ultimately, if you're making it by judgment, you're going to have a judgment and it's going to be like an intuition. And you're going to go with it. So the more or less deliberate judgment, intuition is always involved at one point or another.

• 42:20Speaker 02

  You're either sort of like listening to it or fending it off?

• 42:23Speaker 00

  Yeah, and our recommendation is fend it off.

• 42:28Speaker 02

  Are there ways to judge the quality of somebody's judgment?

• 42:31Speaker 00

  Yeah, sure.

• 42:33Speaker 02

  I mean, some of them would be unique to the actual scenario, but what are the sort of other ways that we could?

• 42:38Speaker 00

  Well, I mean, you you may require people to explain their judgments and evaluating the quality of the explanation is, you know, whether it's logical, whether it uses the evidence, whether it uses all the evidence, whether it is strongly influenced by wishes, whether the conclusion was reached before the judgment supposedly is made. You know, there are lots of, there are lots of ways for judgment to fail that can be recognized. So it's harder to recognize very good judgment, but it's fairly easy to see, you know, what goes wrong. And there are quite a few ways for judgment to go wrong.

• 43:22Speaker 02

  And I think some of those ways are the cognitive biases, like overconfidence and sort of using small or extrapolating from small sample sizes. And one of the interesting things that I've heard you say in interviews before, so correct me if I'm off here, is that you've studied cognitive biases effectively your whole life and you're no better at avoiding them than anybody else.

• 43:48Speaker 00

  Yeah, certainly, not much better, no.

• 43:51Speaker 02

  What hope do the rest of us have?

• 43:53Speaker 00

  Not much. I mean, I never, you know, I, I think, you know, the quality of people's judgment is affected by, by education, but.

• 44:04Speaker 00

  So in general, you know, more educated people make better judgments, I think, on average. But people decide, "I'm going to make better judgments." I don't think that's very hopeful. I'm much more hopeful about organizations. Because organizations think more slowly, and they have procedures for thinking. And so you can control the procedures. Individual judgment is really hard to fix. Not impossible.

• 44:34Speaker 02

  One of the things that I see people do in response to cognitive biases and trying to account for them is to sort of make a list of them, almost like a checklist, and then go through that checklist and explain or rationalize why those things don't apply in this situation. It also strikes me that the more intelligent you are, the more stories you'd be able to conjure up about why you're avoiding this.

• 44:59Speaker 00

  I really think that's not very hopeful, because there are so many biases. And the biases work in different directions anyway. So sometimes you can recognize a situation as one in which you're likely to be wrong in a particular way. So that's like illusions. If you recognize a particular pattern as something that gives rise to a visual illusion, then you don't trust your eyes. You know, you do something else. And the same thing happens when you recognize this is a situation where I'm likely to make an error. So sometimes you can recognize the importance,

• 45:44Speaker 00

  for example, of what we've called an anchor. So you're going to negotiate a price with somebody. They start very high, and that has an effect. So you know, or you should know, that the person who moves first in a negotiation has an advantage. Because the first number changes everybody's view of what is considered plausible. So it moves things in that direction. That's a phenomenon. People can learn that. And they can learn to resist it. So when I was teaching negotiations, I would say, somebody does that to you,

• 46:28Speaker 00

  comes up with a number that's absurd. I would say, lose your temper, make a scene. Say, I will not start the conversation from that number. It's an absurd number. I don't want to. Let's erase that number. So that's something that, you know, you can improve if you recognize it. I think people are aware of the fact that you shouldn't make a decision about road safety within a short interval of a terrible accident. You know, so you should allow things to settle down and cool down. There is a more subtle error and harder, harder to fix.

• 47:08Speaker 00

  But that, the best prediction, the best guess, is always less extreme than your impression. Intuitive prediction is, as we say, not regressive. It doesn't recognize regression to the mean. But statistics is, statistics, in statistics, things are less extreme. Should I give you my favorite example of a bias?

• 47:32Speaker 02

  Yeah, please.

• 47:33Speaker 00

  Okay. I have been unable to think of a better one. But the story is about Julie. It's, that's part of the story. That's her name. She is a graduating senior at university. And I'll tell you one fact about her, that she read fluently when she was four. What's her GPA? And, and the interesting thing here is that everybody has a number. As soon as I told you that thing, her number came to mind. Now, we know where that number came from. We really, that's one of the few things that I'm reasonably sure I understand perfectly. And this is that when you hear she read fluently at age four, you get an impression of how smart she is, of how precocious she was at age four.

• 48:19Speaker 00

  And you could put that in percentiles. You know, where does that put her? On a percentile for sort of aptitude, ability. And it's high. It's not, you know, if she had read fluency at age two and a half, would be more extreme, but age four is pretty high. So say it, the ninetieth percentile. And, and then the GPA that comes to your mind is around the ninetieth percentile in the distribution of GPA. So you pick something, your prediction is as extreme as your impression.

• 48:53Speaker 02

  Mm hmm.

• 48:54Speaker 00

  And it's idiotic statistically, completely stupid. Because clearly, the age at which a child learned to read is not all that diagnostic with respect to GPA. So it's better than nothing. If you didn't know anything, you would predict the mean GPA, whatever it is, three point one, three point two. Now, she's bright, so probably a little higher, but not three point seven. You don't want to. So that's cool. That's a bias. That's non-regressive prediction. And that's very hard to resist. Sometimes I'm able to resist it,

• 49:37Speaker 00

  but never when it's important. You know, when I'm really involved in something, I don't think about it. But sometimes I will vaguely, oh, you know, that's a situation I should moderate my prediction.

• 49:50Speaker 02

  And if you're conscious of it, that's an example of one you can sort of talk yourself out of.

• 49:55Speaker 00

  Yeah, you can talk yourself into. Although, you know, you, you usually will find a way to cheat and end up with your intuition. Uh, it's, that's remarkable, you know, when you've been in academic life a long time. So you've been at, in many situations where people discuss a job candidate. And absurdities of that kind are very common. So somebody, a job candidate gives a talk and people evaluate the talk. And

• 50:29Speaker 00

  This is something happened, you know, at Berkeley when I was teaching there, that somebody in a talk wasn't a very good talk, stammered a bit. Now, that person had teaching prizes. And yet, what was said about him in the discussion, he can't teach. You know, we heard the talk. So that's a mistake. But the funny thing is, you can point out to people that that's a mistake. They still don't want to hire him because he gave a lousy talk. So it's hard to resist.

• 51:02Speaker 02

  It's interesting. One of the, I think, one of the ways I probably got my job is using psychology in the interview, which is asking why I was there and then reinforcing those beliefs through the inner feed. I want to come back just one second to the immediacy of sort of having a stimulus and then making a decision. So we use the example of roads and a tragic accident happens and you're rethinking sort of policy or laws around the roads. How much of that do you think is social pressure? And I'm wondering if we could even extrapolate that a little more to, we're taught to answer questions on a test right away, right? So we see this question and we answer it.

• 51:46Speaker 02

  We're taught that we, or maybe it's reinforced, taught is probably the wrong word, that politicians need to have a response, an immediate response to. And even if they know the best thing to do is like, okay, like let this settle, take some time, it's, society writ large seems to demand it. Like the environment is not conducive.

• 52:08Speaker 00

  I think it's pretty clear that people prefer leaders who are intuitive and were overconfident. Leaders who deliberate too much are viewed with suspicion. So I think Obama was at a certain disadvantage relative to George Bush.

• 52:29Speaker 02

  Because he was seen as more deliberate.

• 52:31Speaker 00

  Yeah, he was more deliberate. And then when you're very deliberate, you look as if you don't know what you're doing. But when you act with confidence, so people want leaders who are intuitive, I think, by and large.

• 52:46Speaker 00

  Provided they agree with.

• 52:48Speaker 02

  Just working my way back through some of these rabbit holes that we've gone down. You taught negotiations. I'm curious what would be in your your sort of syllabus for negotiations that everybody should learn about negotiations when it comes to your work in psychology.

• 53:06Speaker 00

  Well, you know, that goes back to a theme that we started with, the essence of teaching negotiations, that negotiations is not about trying to convince the other guy. It's about trying to understand them. So again, it's slowing yourself down. It's not doing what comes naturally because trying to convince them is a prime pressure. Arguments, promises and threats are always a prime pressure. And what you really want is understand, you know, what you can do to make it easy for them to move your way. Fairly non-intuitive, that's a surprising thing when you deep negotiation. It's not obvious.

• 53:50Speaker 00

  You know, we are, we are taught to apply pressure in social lives up there.

• 53:55Speaker 02

  You had mentioned that there was procedures for thinking in organizations. Are there any that stand out in your mind that we could use to elevate thinking? And if not elevate, but give feedback on the quality of thinking to improve it?

• 54:10Speaker 00

  Well, I think one of the ideas that people like the most is an idea by Gary Klein, that he calls the pre-mortem. And that's a, that's a universal winner. People really like that idea. And this is that when you're about to make a decision, a group, not quite, because if you've made it, it's too late, but they're approaching you. And then you get people in a room, can be the people who are making the decision. And you said, suppose it's two years from now. And we made the decision that we're contemplating. And it turned out to be a disaster. Now, you have a page in front of you.

• 54:53Speaker 00

  Write the history of that disaster in bulletins. That's the pre-mortem. And, and it's beautiful as an idea. It's beautiful because when people are coming close to a decision, it becomes difficult to raise doubts or to raise questions. People who are slowing the group down when the group is nearing a decision are perceived as really, you know, annoying, you know, there's, you want to get rid of them. And the pre-mortem legitimizes that sort of dissent and that sort of doubts. Not only legitimizes it, you know, it rewards it. And so that's a very good idea. I don't, you know, I don't think that it's going to

• 55:37Speaker 00

  prevent people from making mistakes, big mistakes. But it could, certainly it will alert people to possible loopholes, to, to things that they ought to do to make a safer decision. So that's a, that's a good procedure. And there are many others.

• 55:55Speaker 02

  What comes to mind?

• 55:56Speaker 00

  What comes to mind is, is to make intelligence, I mean, the collection of, of the information independent of the decision maker's wishes. And you really want to protect the independence of the people collecting the evidence. And I would add to, you know, a procedure that really people don't like, but if it were possible to implement it, I think would be good. And that's, that when you're going to be discussing a topic and it's done in advance and people incent the material to think about the topic, that you may want them to

• 56:38Speaker 00

  write down their decision, the decision they are in favor of before the discussion starts. That has many advantages. It's going to give you a broader diversity of points of view. Because people tend to converge very quickly in a in a group discussion. And it forces people to be better prepared. It's except people don't want this. So I I don't know whether it's even possible to implement. But clearly, if you could, it would be a good idea.

• 57:13Speaker 02

  What are the reasons people don't want it?

• 57:15Speaker 00

  Too much work.

• 57:16Speaker 02

  Right. Forces you to do a lot rather than the signaling you can sort of get away with.

• 57:21Speaker 00

  Yeah, and then, you know, there's somebody who is going to prepare the case. And so I glanced at the material and then, you know. So a lot of meetings are tremendous sink for wasted time. And improving the quality of meetings would be a big thing.

• 57:41Speaker 02

  Do you have any insights on how to do that?

• 57:43Speaker 00

  Keeping them short. You know, another professional at fixing meetings. So I I have I have a few ideas, but not a complete view. The the question of structuring the meetings to be discussing topics one at a time. That I think is is really useful. I'll give you an example. I mean, it's something that I suggested when I was consulting. But for some reason, people didn't buy that suggestion. So you when an investment is being discussed, say, by an investment firm. Some staff people, if it's a big investment, staff people will prepare a briefing book with chapters.

• 58:28Speaker 00

  Now, our recommendation would be that the staff should end each chapter with a score. How does that chapter taken on its own independently of anything else affect the likely decision? And then you could structure the meeting that discusses this and the meeting of the boards. To discuss these scores one at a time. That has the effect that I was talking about earlier. Making the decision, making the judgments about the dimensions. We call them mediating assessments is a dragon too. The mediating assessments come first. And then you have a profile of them and then you make a global judgment.

• 59:13Speaker 00

  And you can structure it. So if the staff has presented a score and you discuss in the board, do we accept their score? You're forcing people to have a look at the evidence.

• 59:29Speaker 02

  And think about why they would just have to reject. And then they they feel like they have to construct an argument that might be less intuitive.

• 59:35Speaker 00

  That's it. So, you know, there are ways of doing this. But if you're going to be too rigid about it, it won't work either.

• 59:46Speaker 02

  So. I'm curious, what other advice you gave as a consultant that nobody follow?

• 59:50Speaker 00

  Oh, I mean, virtually all the advice I gave people don't follow. I mean, you know, I think that's that's not you shouldn't, you know, you're not going to be a consultant if you expect your advice to be taken. You have to give the best advice you can.

• 1:00:05Speaker 02

  What would be other examples of something you think could be widely applicable? That you would advise you would have advised people and you just sort of like saw them drop the ball?

• 1:00:15Speaker 00

  Well, I mean, you know, I would advise people who make a lot of decisions to keep track their decisions and of how they turned out. So that later you can come and and evaluate your procedures and and and see whether there is anything that is in common with those decisions that turned out well and didn't not so well and so on. People hate doing this.

• 1:00:42Speaker 02

  Why do you think people hate doing it?

• 1:00:45Speaker 00

  Oh, because because retrospectively, they may look foolish, some of them or all of them, or in particular, the leader. So they really don't like keeping track. I mean, there are exceptions. Ray Dalio and his firm and where everything is explicit.

• 1:01:03Speaker 02

  Bridgewater, yeah.

• 1:01:04Speaker 00

  Yeah, Bridgewater. But in general, in my having consulted with Bridgewater, they don't need me. But but in general, then I suggested that never went anywhere.

• 1:01:16Speaker 02

  What are the variables that you would recommend people keep track of? Like, what would your decision journal look like?

• 1:01:22Speaker 00

  Oh, I mean, my my decision journal would be a mess. I don't I'm not putting myself as an example, but.

• 1:01:32Speaker 02

  So obviously the outcome, but you've got to do that post after.

• 1:01:35Speaker 00

  Yeah, but no, no, you, you would want to say, what were the main arguments pro and con? What was the alternative that were considered? You know, it doesn't have to be very detailed, but it should be enough so that you can come later and debrief yourself.

• 1:01:55Speaker 02

  Should you have a calibration, like what degree of confidence you are?

• 1:01:58Speaker 00

  That would be good. Then, you know, it would depend on something that you could evaluate later.

• 1:02:04Speaker 02

  It strikes me that decision journals and pre-mortems are a way to identify people that are sort of perhaps suppressed by their manager, where you have somebody who's actually a better, better at exercising judgment than the person that is, you know, that they're working for. And this would be a pain-free sort of way to calibrate that score over time and identify the quality of judgment in a consistent way.

• 1:02:35Speaker 00

  Oh, yeah. I mean.

• 1:02:37Speaker 02

  That strikes me as worth a lot of money to an organization.

• 1:02:40Speaker 00

  Yeah, but, but also very costly. And you, you will see that certainly anything that threatens the leader is not going to be adopted. And, and leaders may not want something that threatened their subordinates either. People are really very worried about embarrassment.

• 1:02:58Speaker 02

  You're writing a book now on noise.

• 1:03:00Speaker 00

  Yeah.

• 1:03:01Speaker 02

  Tell me about noise and decision making. Can you explain the concept?

• 1:03:05Speaker 00

  Yeah, I can really explain it by saying what, you know, was the beginning of it. And which was a consulting assignment for an insurance company where we, I had the idea of, of running a test to see whether people in a given role who are supposed to be interchangeable. Agreed with each other. So, you know, when you come to insurance company and an underwriter gives you a premium, the underwriter speaks for the company. And so it's, you expect that any underwriter, that it doesn't matter which underwriter you get to for the premium. And the company has that expectation. Shouldn't make much difference.

• 1:03:49Speaker 00

  So we tested that and they constructed some cases. And then we had some like fifty underwriters assess a premium for the case.

• 1:03:59Speaker 02

  With the same information?

• 1:04:01Speaker 00

  Hmm? Yeah, with a really very realistic, we didn't construct it. They constructed the case. So, and they conducted the experiment. But now, the interesting question is, how much variation do you expect there to be? So, we asked the executives the following question. Suppose you take two underwriters at random. By what percentage do they differ? I mean, you look at the difference between their premium, divide that by the average premium, what number do you get? And people expect ten percent. By the way, it's not only the executives in that company. For some reason, people expect ten percent. And it was roughly fifty percent, five zero.

• 1:04:43Speaker 00

  So that's, you know, that's what made me curious about noise. Uh, that and the fact that the company was completely unaware that it had noise. It took them completely by surprise. So now we're writing a book because there's a lot of noise. So wherever a rule is that wherever there is judgment, there is noise and more of it than you think. So that's the pattern.

• 1:05:07Speaker 02

  Are there procedures to reduce noise? And conversely, i- is noise a, it strikes me that the variation would be good, but maybe only in an evolutionary concept?

• 1:05:16Speaker 00

  Well, we call that noise is useless variability.

• 1:05:21Speaker 00

  I mean, variability can be very useful if you have a selection mechanism and some feedback. So evolution has built a variability, but of course it's useful. But noise among underwriters is useless, is nothing. Nothing gets learned, there's no feedback. It's just noise and it's costly. The first advice, of course, would be algorithms, as I said earlier. So algorithms are better than people and judgment. That's not intuitive, but it's really, it's really true. And, and after that, then, you know, the procedure that I mentioned earlier

• 1:06:03Speaker 00

  for making decisions in an orderly way by breaking it up into assessments. That's the best that we can do. And, and there is one very important aspect that I haven't mentioned, but, and this is training people in, but the scale is. So there is one piece of advice that you'd have for underwriters, that they should always compare the case to other cases. And if possible, if you can have them share the same frame of reference with other underwriters, you're going to cut down on the, on the noise.

• 1:06:41Speaker 02

  Oh, that's a good idea, yeah.

• 1:06:43Speaker 00

  Controlling the scale. And that exists in, in human resources, where performance evaluation, which is one of the scandals of modern commerce, how difficult it is, but performance evaluation, they have the thing that's called frame of reference training, which is teaching people, you know, how to use the scale. There's a lot of variability in the scale. And a part of what the super forecasters do, they make judgments in probability units and they teach them to use the probability scale. So learning the scale is a very important aspect of the new signals.

• 1:07:25Speaker 02

  I know we're, we're coming up to the end of our time here. What have you changed your mind on in the past 10 years?

• 1:07:31Speaker 00

  Oh, a lot.

• 1:07:32Speaker 02

  Anything big?

• 1:07:33Speaker 00

  Yeah. There's been a replication crisis in psychology. And some of the stuff that I really believed in when I wrote Thinking Fast and Slow, some of that evidence has been discredited. So I've had to change my mind.

• 1:07:49Speaker 02

  What are the, what's the?

• 1:07:51Speaker 00

  Some of the sexiest stuff, priming and unconscious priming. And so it just hasn't held up in replication. And I believed it and I wrote it as if it were true because, you know, the evidence suggested it. And in fact, I thought that you had to accept it because that was published evidence. And, and I should have, I blame myself for having been a bit gullible. That as I should have known that you can publish things even if they're not true. But I just didn't think that through. So I changed my mind. I'm now much more cautious about spectacular findings.

• 1:08:34Speaker 00

  I mean, very recently, I've come, I think I have a theory about why psychologists are prone, or social scientists generally are prone to exaggerate, to be overconfident about their hypotheses. So I've done quite a bit of learning.

• 1:08:52Speaker 02

  What's, what's the theory?

• 1:08:54Speaker 00

  Well, the theory, one element of the theory is that all these hypotheses are true. In what sense? That, you know, if I, there's a famous study that you mentioned wrinkles to people. And then you measure the speed at which they walk and they walk more slowly. Turns out that hasn't held up in replication, which is very painful. It's one of the favorite studies. But actually, you know, that if you mention wrinkle and it's going to have any effect on, on the speed of walking, it's not making to make, it's not going to make people faster. If it has any influence, it's going to make them slower. So directionally, all these hypotheses are true.

• 1:09:37Speaker 00

  But what there is, is what people don't see is that the huge number of factors that determine the speed at which individuals walk and the differences in the speed of walking between individuals. And that's noise, and people neglect noise. And then there is something else, which is touches on both philosophy and psychology. When you have intuitions about things, there are clear intuitions and there are strong intuitions. Another thing. So a clear intuition is if I offer you a trip to Rome, a trip to Rome and an ice cream cone, you know what you prefer,

• 1:10:21Speaker 00

  it's easy. But it's very weak, of course. I mean, the amount of money you would pay to get a trip to Rome and a trip to Rome and an ice cream cone, nothing. But when you are a philosopher, and I should add one thing, to see a clear intuition, you have to be in this kind of situation that psychologists call within-subject, that you have both, you have both with the ice cream cone and without the ice cream cone. So in a within-subject situation, that's an easy problem. In a between-subject situation, it's an impossible problem. But now, if you're a philosopher, you're always in a within-subject situation. And but people live in a between-subject situation.

• 1:11:06Speaker 00

  They live, you know, in one's condition. And the same thing is true for psychologists. So psychologists live in a, when they cook up their hypotheses, they're in a within-subject situation. But then they make guesses about what will happen between subjects. And they're completely lost between clear intuitions and strong intuition. We have no way of calibrating ourselves. So that makes us wildly overconfident about what we know and reluctant.

• 1:11:39Speaker 00

  To accept that we may be wrong.

• 1:11:43Speaker 02

  That's a great place to end this conversation, Danny. Thank you so much.

• 1:11:48Shane Parrish?

  Thanks for listening and learning with us. Be sure to sign up for my free weekly newsletter at fs.blog/newsletter. The Farnham Street website is also where you can get more info on our membership program, which includes access to episode transcripts, my repository, ad-free episodes, and more. Follow myself and Farnham Street on X, Instagram, and LinkedIn to stay in the loop. Plus, you can watch full episodes on our YouTube channel. If you like what we're doing here, leaving a rating and review would mean the world. And if you really like us, sharing with a friend is the best way to grow this community. Until next time.