A Numbers Game: With Guests Linda Chang & Stephen M. Colbert

Why do we tend to fixate on numbers even when words or graphs represent the same information?

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:08Speaker 00

  In the beginning, there was a vision, a bold plan to turn DC Comics and its characters into larger-than-life mythical heroes on film. An operatic saga of gods amongst men, complete with darkness and hope, sacrifice and death. The monumental task of shaping this cinematic universe fell to visionary director Zachary Edward Snyder. An ambitious serialized arc began to build around their central hero, Superman. But up, up high in the ranks of Warner Brothers Studios, a battle of ideals was raging. A war that would ultimately shake the very foundation of what Snyder and his

• 0:50Speaker 00

  team were crafting: The DC Extended Universe.

• 0:58Katy Milkman?

  A lot of the time, the drama in Hollywood extends beyond the screen. From production budgets to box office earnings, Hollywood movies are a high-stakes business gamble. Onscreen, the characters tell the story, but offscreen at the box office and in critical reviews, the numbers tell the story. In this episode, we'll look at our very human tendency to fixate on numbers. Why we often value hard numbers over words or graphs or other equivalent representations of the very same information, and what we should be aware of when we face a decision involving numbers. Numbers that might not tell the whole story.

• 1:44Katy Milkman?

  I'm Dr. Katie Milkman, and this is Choiceology, an original podcast from Charles Schwab. It's a show about the psychology and economics behind our decisions. We bring you true and surprising stories about high-stakes choices, and then we examine how these stories connect to the latest research in behavioral science. We do it all to help you make better judgments and avoid costly mistakes.

• 2:21Speaker 02

  The idea was, let's bring in these filmmakers who have their own unique takes on things and have their own vision and voice, and give them these big budgets to make these huge blockbuster movies.

• 2:31Katy Milkman?

  This is Stephen M. Colbert, not to be confused with comedian and late-night host Stephen Colbert.

• 2:37Speaker 02

  My name is Stephen Colbert. I'm a writer and podcaster.

• 2:44Katy Milkman?

  Stephen is a commentator on and close observer of the film industry. One of his podcasts breaks down the movie Batman v Superman minute by minute, so he pays very close attention to superhero films. In this case, from the DC Extended Universe. DC is the comic book publisher, which is now a subsidiary of Warner Brothers, and it's home to superheroes like Wonder Woman, Superman, and Batman.

• 3:09Katy Milkman?

  Before the Batman v Superman movie, there was the first DC Extended Universe film, Man of Steel, a film that was greatly overshadowed before it was even released.

• 3:19Speaker 02

  Man of Steel was going to come out in 2013, so it was already mostly produced and in post-production. And The Avengers came out. That was the first Marvel Cinematic Universe crossover movie directed by Joss Whedon.

• 3:32Katy Milkman?

  A quick aside for anyone who's not familiar: Both Marvel and DC started out as competing comic book companies. Now, they're also competing movie companies. And they each have their own roster of superheroes: Spider-Man, Black Panther, Iron Man. They're on the Marvel side. Flash, Aquaman, and Superman are all DC characters. It's okay if you can't keep them all straight, though. This story will still make sense. So, Man of Steel, a Superman movie from DC Comics, was released after the blockbuster hit Marvel movie The Avengers. And while Man of Steel did pretty well at the box office, it paled in comparison to The Avengers.

• 4:15Speaker 02

  The Avengers made $1.5 billion at the global box office. It was a huge deal at the time. Everyone wanted their new Avengers.

• 4:22Katy Milkman?

  So now Warner Brothers Studios, with its DC Extended Universe movies, was aiming for Marvel numbers.

• 4:29Speaker 02

  When you're trying to make a film like that, that has that big market appeal, you're looking for something that's harder for audiences to dislike. You're not taking big swings. That means you need all demographics to be appealed to in order to fill seats, as opposed to saying like, what's the most interesting, unique, best movie we can make?

• 4:49Katy Milkman?

  The filmmaker involved, Zack Snyder, was not a big-market-appeal kind of director. His work is darker in tone, more mature.

• 4:58Speaker 02

  Zack Snyder is really interesting. People use the word visionary, which once again isn't necessarily a term of quality. It's just a term of approach, where he's got a movie in his head. He'll have a story and he'll know what it looks like. He knows what it smells like. You'll see a shot and there'll be an alleyway. He could tell you what's around that alley, even if the camera never sees it.

• 5:21Katy Milkman?

  He creates unique and striking worlds. But his approach is not for everyone, which means it's not necessarily a recipe for broad appeal.

• 5:30Speaker 02

  Zack Snyder's kind of polarizing in nature. Rotten Tomatoes isn't very friendly to that. And so the reviews that liked it did really like it, and a lot of fans really liked it. But because it was polarizing, that gets punished by Rotten Tomatoes scores.

• 5:44Katy Milkman?

  You're probably familiar with the Rotten Tomatoes rating system. Look up any film online and you'll see ratings as part of the overview. The rating appears as a number. And for Rotten Tomatoes, that number is a percentage. Search "Man of Steel" and you'll see a 57% rating. Now, 57% doesn't mean it averaged 5.7 out of 10 as a score. It means 57% of critics' reviews were positive. And positive can range from feeling mostly OK about the movie to absolutely loving it. It's a big range.

• 6:19Speaker 02

  Despite the praise that it was getting, it wasn't dominant enough in its score or in its reviews for anyone to say, "Let's not mess with the formula."

• 6:27Katy Milkman?

  So the formula was tested.

• 6:30Speaker 02

  Zack Snyder was pressured to make Batman vs Superman funnier. There was a big focus on adding jokes and on making it lighter and not so dark. He still had enough leverage at the time to not completely cave to that. Warner Brothers did step in to make some changes. It was almost a three-hour movie. For the theatrical version, they made him cut out 30 minutes. There was some significant plot developments and other details that got lost in that stylistically. Warner Brothers wasn't really exerting any real effort there yet.

• 7:03Katy Milkman?

  Not yet. Batman v Superman: Dawn of Justice was set to release in March of 2016. But a week before it premiered, the film's Rotten Tomatoes score became available. And it wasn't good.

• 7:18Speaker 02

  It got a 28% on Rotten Tomatoes. That was a huge moment where it's following up a somewhat controversial movie. And then a week before it comes out, it's like this bomb drops of 28%, which is really low.

• 7:34Speaker 03

  What was Zack thinking?

• 7:35Speaker 02

  Oh, no, it is bad. All my friends were like, "No way." Henry Cavill rocked it.

• 7:38Speaker 03

  Well, our review is in two and it's not much better.

• 7:42Speaker 02

  For something that's taking Batman and taking Superman and putting them in the same movie at the same time, a lot of people would make the argument that this clearly is an example of a movie that should have done over a billion dollars. And so considering the low Rotten Tomatoes score that was seen as that's the reason why. If only this had the larger audience appeal, this is something that would have made over a billion dollars. And so therefore, we're missing out on hundreds of millions of dollars because of how polarizing this is.

• 8:14Katy Milkman?

  The next film in the series, Justice League, was just about to start production right as this low Rotten Tomatoes score for Batman v Superman came out. The studio understandably panicked.

• 8:25Speaker 02

  They were going to leave to go start shooting on location, I think in Iceland, like the day after the Batman v Superman premiere, which is before the box office results are even in. But because of that Rotten Tomatoes bomb drop, they actually immediately decided to do some rewrites on the script.

• 8:42Katy Milkman?

  The studio delayed production, and Zack Snyder and his team were asked to modify some of the more controversial elements in this next film, Justice League. And again, they were asked to make it funnier.

• 8:54Speaker 02

  I think there's a lot of humor in his movies, but it's not the poppy, kind of quippy jokes that you would get from a lot of other popcorn superhero movies, like especially like the Marvel style that they're known for now. And so there's about two months there where they ended up with a script that was similar, but kind of got this workover to try to make it something that would be more appealing, ideally get a higher Rotten Tomatoes score.

• 9:17Katy Milkman?

  It wasn't just rewrites. Throughout the filming, executives were on set, essentially to babysit Zack Snyder, to remind him of what makes a commercial hit. The pressure was rising.

• 9:29Speaker 02

  He started to second guess himself a little bit because of the pressure. And so there's just a lot of stress. And also, there's a lot riding on it now because instead of riding high on this universe that they're building out, there's now a sense of this is kind of a make it or break it moment, not just for the franchise, but also it's a big deal for Zack Snyder. Because if this pressure has been growing movie by movie by movie, and now not only is the franchise at stake, but this is going to be a big deal for his career also if this doesn't work out.

• 9:55Katy Milkman?

  Snyder eventually stepped aside. It was in part due to the studio pressure, but mostly for personal reasons. There'd been a tragedy in his family. But the film was nearly done at that point.

• 10:09Katy Milkman?

  In a twist, instead of working with what they had, Warner Brothers hired director Joss Whedon. Yes, the same director of the Avengers movies. He came in and he did some rewrites and even some reshoots. They had to bring the cast back and all of this reportedly cost an extra $30 million. Justice League was finally released in 2017.

• 10:33Speaker 02

  It's been compared to like a Frankenstein's monster. You have the Zack Snyder style scenes with the high contrast colors and the serious tones. And then you have these other scenes where Superman's face is weirdly CGI'd and the tone is totally mismatched. And there'll even be dialogue points that are like really on the nose exposition that seems to almost be apologizing to the audience to like undo controversial plot points.

• 10:56Katy Milkman?

  With a 37% on Rotten Tomatoes at the time, Justice League did a bit better than Batman v Superman, but worse than Man of Steel. Not a total failure, but also not a success. And fans eventually started piecing together that this movie was not Snyder's vision. From posts he made on social media during production, they determined that Zack Snyder's version of Justice League was out there, somewhere, and they weren't going to rest until they saw it.

• 11:24Speaker 02

  Fans started campaigning online. Release the Snyder Cut is the famous hashtag for that. Whenever there was an interview with someone from the caster crew, the interviewer would have to ask, well, what about this? Does this exist? Cast and crew started being more open about, you know, they'd say, yes, it does exist and yes, it was better.

• 11:41Speaker 04

  You overwhelmed us with Snyder Cut questions. Snyder Cut. Give it to me.

• 11:46Speaker 02

  Eventually, the volume got so loud that the studio called up Zack Snyder and said, so about that? What if we were to finish that up? And then we can use that as a big flagship product to launch HBO Max.

• 12:01Katy Milkman?

  Justice League: The Snyder Cut ended up getting a 71% from critics on Rotten Tomatoes. And it did even better with audiences.

• 12:11Speaker 02

  It's definitely the highest score of any of the movies that he directed for DC so far. It also got a 92% audience rating on Rotten Tomatoes.

• 12:21Katy Milkman?

  That's a score regular users on Rotten Tomatoes gave the film, not the official critics.

• 12:26Speaker 02

  Finally, kind of checked all the boxes for what the studio was supposedly looking for with the other ones.

• 12:35Katy Milkman?

  Snyder's Cut better reveals the planned arc for this franchise, to the point where most recent reviews of Man of Steel and Batman v Superman are more positive. Choices in those movies make more sense, but the Rotten Tomatoes numbers, accounting for hundreds of professional reviews, are mostly unchanged. Those first Rotten Tomatoes scores weigh it down.

• 12:57Speaker 02

  Creatives have this big struggle against these people who are trying to make decisions from numbers. They're trying to make business decisions and ultimately there is no connection there between what's a good creative choice versus a business choice.

• 13:15Katy Milkman?

  Stephen M. Colbert writes about movies and the film industry. He co-hosts the Batman v Superman By the Minute podcast, as well as Screen Rant's Ring of Power podcast. He thinks the least interesting way to talk about a movie is by saying it's good or bad. You can find more details about Stephen's work in the show notes and at schwab.com/podcast.

• 13:49Katy Milkman?

  There's more than one way to rate a film. Film critics make careers out of writing in-depth reviews of movies, for columns, or TV shows, or social media. And you may remember "two thumbs up" from Siskel and Ebert, and of course, critics pick from The New York Times or five-star scales from many reviewers. But dominant in the current zeitgeist are aggregated number ratings from websites like Rotten Tomatoes and Metacritic. But do we put too much faith in numbers? Audiences weighing what to watch have to balance their taste for action versus comedy, films versus TV shows, and much more. And they have to consider those attributes alongside a movie's "quality", according to a number on Rotten Tomatoes.

• 14:33Katy Milkman?

  I've invited my friend and collaborator Linda Chang on the show to talk about why we tend to focus excessively on numbers, attending more to them than other ways of representing the very same information when we make choices with trade-offs. It's a bias we call quantification fixation. Linda Chang is a behavioral scientist at the Toyota Research Institute and a former MindCorps postdoctoral research fellow at the University of Pennsylvania.

• 15:00Katy Milkman?

  For several years, Lynda worked with me and Morton's Department of Operations, Information and Decisions. And together with two former guests on the show, University of Chicago Professor Erica Kyrgios and MIT Professor Sendhil Mullainathan, Lynda and I studied quantification fixation. Hi, Lynda. Thank you so much for joining me today.

• 15:19Speaker 05

  Hi, Katie. It's so great to be here. Thank you for having me.

• 15:21Katy Milkman?

  Well, I am thrilled to have the chance to ask you some questions about quantification fixation. And I actually just want to start by asking you to define it. What is quantification fixation?

• 15:32Speaker 05

  Yeah, absolutely. So when people face a decision where there's a trade-off, so they're sort of bombarded with different types of information, maybe some represented numerically, some represented with like star ratings or with words, we find that they tend to favor options that are more attractive on the numeric dimension. So it's the sort of tendency to overemphasize or focus more on the numeric dimension.

• 15:56Katy Milkman?

  Could you say a little bit more about what kinds of options you have in mind where we would face trade-offs? Like, what are these kinds of decisions that you're thinking about? And what would be a numeric component of that decision?

• 16:07Speaker 05

  If you're on the job market and you're facing decisions between two different job options, you might evaluate any number of attributes, right? So something like your salary, your PTO, but also the company's culture, the day-to-day of what you're actually doing. Only two of the dimensions that I've mentioned just now, salary and PTO, are more naturally represented via numbers, where something like a company's culture, what you're doing day-to-day, that's more qualitative, something that is a little bit richer, can't be distilled to a specific number. And you might overweight the salary, the PTO, the sort of numeric benefits of the job, rather than thinking potentially deeply about the day-to-day experience of what it might be like to take a particular option.

• 16:47Katy Milkman?

  Okay. And just for listeners who aren't familiar with that acronym, PTO is Paid Time Off, and it would normally be a number of days that comes with a job offer. So what are the kinds of situations where you think people should be aware that they're at risk of letting quantification fixation bias their choices? In other words, could you talk about some of the types of situations where this tendency might matter?

• 17:09Speaker 05

  For anyone who's browsing the web, I think we're increasingly contending with a mix of both qualitative and quantitative or numeric information. And so something even as important as deciding what healthcare plan to enroll in, we get a ton of information about these sorts of different options. Some of that information is represented numerically, some of it with words, others with sort of pictographs, so maybe like four out of five stars or bar charts. And all of this information we're taking in and evaluating and then hopefully making a good decision for ourselves. And so I think whether you're making decisions about your healthcare plan, about like a raincoat online or camping equipment, all of these things were often not presented just with one type of information, but a whole slew of different kinds of information.

• 17:54Speaker 05

  And so when you have that mix of information, I think this is when quantification fixation can potentially come into play.

• 18:00Katy Milkman?

  Yeah, I love that. And as you were describing, and I was sort of thinking about all the websites I use that have that mix of information, like Yelp, where they give me maybe a numeric rating on average or an average price, but also a bunch of user reviews and stars, which you're saying are different than a number. And then I'm thinking about, you know, buying a house or shopping for an apartment or what have you. Constantly, we're looking at a bunch of attributes lined up for a number of choices that might suit our needs. And some of them are presented as numbers and some are presented in different ways. So you're basically saying, in those situations, the number will loom larger, even if it's not the most important component of the choice.

• 18:45Speaker 05

  Exactly. So I think some deliberation probably needs to be had upfront in terms of what actually matters in terms of the decision at hand, so that we're not swayed by the format, let's say, of the information.

• 18:58Katy Milkman?

  Could you describe some of the details of the research showing that people exhibit this bias, that they show quantification fixation?

• 19:08Speaker 05

  Yeah, participants in one experiment were presented with an option to choose between two distinct hotels, Hotel A and Hotel B. And these two hotels varied on whether they had a higher price and a higher rating, or a lower price and a lower rating. And so there's a trade-off between these two dimensions of price and rating. What we varied between different participants in conditions or scenarios is how price and rating was represented. So for some participants, they saw Hotel A and Hotel B, where price was given as a number, and rating was given as star ratings. And other participants saw ratings represented as numbers, and price represented as these filled cash icons.

• 19:51Katy Milkman?

  So what did you find?

• 19:52Speaker 05

  We find that people tend to favor the option that is more attractive on the numeric dimension.

• 19:58Katy Milkman?

  So if I'm trying to decide between going to the Hyatt and going to the Marriott, and the Hyatt is pricier but also nicer, I am more likely to pick the Hyatt when you tell me its rating, not in stars, but you say it's a 4.8. So you make that numeric, because now I'm more fixated on the rating. And now if you change things and you tell me the rating in stars, but the price is numeric, now I actually want to go to the Marriott, because the Marriott is cheaper and I'm fixating more on price. Is that the right way of thinking about this?

• 20:34Speaker 05

  That's exactly it.

• 20:36Katy Milkman?

  You came up with some really clever designs for these studies, Lynda. And one of the things I liked in particular was not only did you present people with scenarios, but you had people make real choices that mattered, and that had financial consequences for them. Could you talk a little bit about one of the studies, which I particularly love that you did, involving charitable giving decisions?

• 20:58Speaker 05

  Yeah, absolutely. So I think the question of how we make charitable donations, which charities we choose to donate to, is a really interesting and meaningful one. There is a website online called Charity Navigator, which is, sort of, audits charities and rates them on these different dimensions. And so we pulled two of these dimensions out, one being accountability. And finance and another being culture and community. And we found two environmental charities, so same particular topic area that trade off on these two dimensions. So one had a higher accountability and finance rating and lower culture and community score. Whereas the other had a higher culture and community score and a lower accountability and finance score.

• 21:39Speaker 05

  And so we told participants, please choose between these two charities and we'll make a donation on your behalf. A $1 donation based on the decision that you make. So this is a meaningful decision, not sort of a hypothetical out in the wild. And so we represented these scores either numerically or with these filled bar graphs to look at whether this sort of shifts people's decisions about which charity to donate to. And we found that it does. So when accountability and finance is represented numerically, people are more likely to want to donate to the charity that has the higher accountability and finance score.

• 22:16Katy Milkman?

  Got it. So it's really a fixation on whichever of these ratings provided by an outside audit is described with numbers that is going to drive the preference for the charity is what you're finding.

• 22:30Speaker 05

  Exactly.

• 22:31Katy Milkman?

  Why is it that people show this bias? What causes it?

• 22:35Speaker 05

  One thing I think that surprised both you and I is that we tested actually a number of different explanations for why this might be the case. And one mechanism that we've identified is what we're calling comparison fluency. So this sort of felt and actual ease of using information to make judgments about differences. We're constantly bombarded with numbers. We feel very comfortable in making comparisons between numbers, right? I think when we think back to grade school, one of the first things that we learn is math, right? Addition, subtraction, all sorts of operations that underlie our sort of comparison framework, let's say. So we find that that numbers are more comparison fluent than non numbers in terms of at least the kinds of formats that we've, we've looked at in our experiments.

• 23:19Speaker 05

  And because of this, people rely on the numeric attributes more when making decisions.

• 23:24Katy Milkman?

  Because we're just so, it's so easy for us basically to say, oh, three is 0.2 higher than 2.8. And so that's, that's got to be the right choice.

• 23:33Speaker 05

  Exactly. So put another way, I think quantification fixation isn't driven necessarily by whether it's possible to evaluate choice attributes, but the ease with which we do so.

• 23:42Katy Milkman?

  I'm curious, what made you excited to study this topic? What, what made you decide to devote a couple of years of your life to this research project?

• 23:50Speaker 05

  I think in having those conversations with Erica Sendl and with you, I think I really latched on to this idea that we were noticing more and more, we seem to be attaching numbers to all sorts of things. And it doesn't seem like that is slowing down at all with, yeah, the sort of migration of the qualitative to the quantitative. I think a slightly different phenomenon is, is the want or the desire to attach numbers to so many different things, which I think is a whole other, probably fascinating research project that I hope someone out there is doing. And so it felt like everywhere that I turned, this kind of question was relevant. As someone who studies judgment and decision making, I felt like I was studying decisions that I was making so many times in a day.

• 24:34Speaker 05

  And so I really wanted to understand what are the consequences? Why might this matter?

• 24:38Katy Milkman?

  So what advice would you give our listeners about how they can improve their own everyday decisions?

• 24:45Speaker 05

  I think we have to be aware that when we're making decisions, not just what information is presented, but how is the information presented? And I think before any decision is made, we also need to be careful. So we need to maybe take a step, step back and decide for ourselves what actually matters here. Should all attributes be weighted equally? Does one matter more than the other? Often we're not just dealing with two attributes, but many more than that. And so thinking carefully, I think about what matters and whether our decisions are aligning with the things that we say matter, I, I think is my takeaway.

• 25:19Katy Milkman?

  I really appreciate you taking the time to talk to me today about this. Thank you so much. And thanks for being an amazing collaborator and leader on this project.

• 25:26Speaker 05

  It was, it was a true pleasure. Thank you.

• 25:31Katy Milkman?

  Lynda Chang is a behavioral scientist at the Toyota Research Institute and a former MINECORPS postdoctoral research fellow at the University of Pennsylvania in Wharton's Department of Operations, Information and Decisions. You can find a link to her work, work that I was so happy to be part of, with University of Chicago professor Erica Kyrgios and MIT professor Sendhil Mullaneathan, in the show notes and at schwab.com/podcast.

• 26:03Katy Milkman?

  Setting financial goals can be hard, and the temptation to compare ourselves to others, especially in the social media age, can be an impediment to creating and sticking with a financial plan. But a good financial plan is based on your goals and starts with important considerations like your personal values that go beyond the numbers. Check out the recent Financial Decoder episode, "How Can You Set the Right Financial Goals", to learn more about goal setting. You can find it at schwab.com/financialdecoder or wherever you get your podcasts.

• 26:38Katy Milkman?

  Quantification fixation is a bias that many people find unsurprising when they first hear about it. "Oh, sure, that makes sense that we overweight numbers." That's a reaction I hear a lot. What I find most interesting about this tendency, though, is that we overweight quantified dimensions of our choices, even when we could get the very same information in a different format. Seeing that a stock got a 4 out of 5 for its environmental, social, and governance risk makes you think differently about it. You'll weight its ESG score more in your decision about whether to invest in that stock than if you'd instead seen a graphic depicting the same 4 out of 5 rating visually. There's something about literally assigning a number instead of showing a graph or an icon or a word that means the

• 27:23Katy Milkman?

  same thing that can make you care more. Like most biases we cover on this show, this one is likely to trip you up more when you don't plan ahead to dodge it. Being aware of quantification fixation gives you a big opportunity. You can combat it by thinking carefully about the attributes of a choice that really matter to you before you say, "Visit a website like Redfin, or Amazon, or Morningstar," where the information you care about most may or may not be quantified. And you can choose to convert all information you collect when making trade-offs into the same format. Say, quantifying not only the critical reviews for two movies you're considering watching tonight, but also those film's runtime, how much you like their genre, and anything

• 28:08Katy Milkman?

  else you've decided should really matter for your choice. Just because counting changes what counts doesn't mean you can't fight back. You can control the things you count.

• 28:33Katy Milkman?

  You've been listening to Choiceology, an original podcast from Charles Schwab. If you've enjoyed the show, we'd be really grateful if you'd leave us a review on Apple Podcasts, a rating on Spotify, or feedback wherever you listen. You can also follow us for free in your favorite podcasting app. And if you want more of the kinds of insights we bring you on Choiceology about how to improve your decisions, you can order my book, How to Change, or sign up for my monthly newsletter, Milkman Delivers, on Substack. Next time, I'll speak with University of Delaware professor Jackie Silverman about the power of tracking streaks when we're trying to build habits, and how you can avoid becoming demotivated if you break a streak.

• 29:17Katy Milkman?

  I'm Dr. Katie Milkman. Talk to you soon.