Choiceology with Katy Milkman
Explore the high-stakes friction between human intuition and algorithmic guidance.
5,318 words · 8 speakers
Okay, what am I supposed to make with this?
Imagine you're cooking dinner on a Thursday night. You open your fridge and you see a random collection of groceries. Butter, ketchup, a bag of shredded cheddar cheese, a jar of salsa and some chicken.
So you text your foodie friend, asking for ideas about what to do with the ingredients you have on hand.
While you wait, you also type a list of your refrigerator's limited offerings into ChatGPT to see what it recommends. Which cooking advice are you more likely to follow? And why?
Honestly, probably ChatGPT because it gives you something concrete right away.
I think I'd trust my friend. She knows what I like and is a good cook.
I don't know, I wouldn't ask a friend.
A foodie friend over ChatGPT because I trust the authenticity of a foodie friend's advice.
I think I would ask ChatGPT. I find their recipe suggestions are usually pretty good.
I guess the algorithm because it's faster and, you know, at that point, I just want dinner on the table.
Today, we'll dig into what happens when people face choices between trusting another person's expertise versus the advice produced by a computer. And we'll look at why it matters in the kitchen, at the hospital, and on the basketball court.
I'm Dr. Katie Milkman, and this is Choiceology, an original podcast from Charles Schwab. It's a show about the psychology and economics behind our decisions. We bring you true and surprising stories about high-stakes choices, and then we examine how these stories connect to the latest research in behavioral science. We do it all to help you make better judgments and avoid costly mistakes.
The Boston Celtics are iconic. They have won a very significant percentage of all the NBA championships in history, over the 80-year history or so of the NBA.
But the team had been in a rut for years and hit a low point in 2006, back when I was a graduate student living in nearby Cambridge, Massachusetts. So I remember it well.
They were among the worst teams in the NBA. They had the beginnings of a young core. They had Paul Pierce, who was already an established star, who missed a good part of the year. And then some of their young players got hurt. I remember going to one of their games that year and the Celtics people apologizing that they really didn't have their full roster that night. So their devoted fans were rather critical. And they had to kind of reinvent themselves.
This is Dean Oliver. He's a data scientist at ESPN, and he helped create the Sports Network's analytics department.
In 2006, the Celtics needed to rebuild. And to do that meant a big shakeup of their roster, which required ruthless scouting. These recruiting decisions traditionally relied heavily on scouts' experience, on their intuition.
An old-school scout is someone who goes and watches players play and uses their eye. They're watching the game. They have the responsibility of contributing to making a judgment on whether a team should acquire that player, whether through the draft, or whether through trades or free agency.
Scouts were respected because they'd been around the game forever. Some of them believed they could size up a player just by looking at him.
I heard this from scouts, actually. There are guys I can see for five minutes and I never have to see them again because I know right then. And that's obviously not the way a lot of data goes. There are players that have great games and there are players who have bad games that go against their typical pattern.
But the Celtics were starting to question whether a scout seeing was the same as a scout knowing. Going into the 07-08 season, the Boston Celtics knew they needed a new approach, even if they weren't entirely sure what that would look like yet.
They hired Mike Zarin, who came out of law school, to work on legal aspects in the MBA. But Mike came in with a knowledge of analytics and trusting numbers by showing this interest in using numbers, some of what came out of Moneyball. He was hired about the same time that Moneyball came out.
You might be familiar with Moneyball. It's the famous book by Michael Lewis about the struggling Oakland A's baseball team and how they used advanced analytics in 2002 to hire new talent and turn their team around. Well, his on-base percentage is all we're looking at now. The Moneyball story served as inspiration for other baseball teams, and eventually other sports leagues, to adopt a more analytical approach to scouting. With Mike Zarin's influence, analytics became a key part of the decisions that would be used to rebuild the 07-08 Celtics roster.
He is pretty close-lipped about the details of it, but I know generally from our conversations on how he thinks about players and what they've had to do to build, say, a draft model. And they're trying to predict how good a player is going to be in year four, or year three, or throughout their career. They have metrics for that.
The predictive player model that Zarin used offered the Celtics insight that wasn't visible to the public, and it helped them make key trade decisions to build a stronger team. As the season approached, Boston started making moves, big ones.
The two new players that came to the team were Kevin Garnett and Ray Allen. These are players who had established careers. They were good, for sure. They hadn't played necessarily quite as well in the last season.
These players weren't unknowns. They were stars, but aging ones.
There was some sense that they may decline, but the Celtics were able to identify that they had a long way to go before they would decline to a poor level or even an average level. Kevin Garnett, who was not a very big point scorer, but was a great defender by all the metrics. He was a genius on the defensive side of the ball. And if you watch highlights around basketball, you will see that 90-plus percent of the highlights are about offense. They don't show what Kevin Garnett does when he's off the ball. And even the old-school scouts, old-school scouts are supposed to look for that, and they're supposed to see it. But it is not as prominent as offense.
Mike Zarin and the Celtics' general manager, Danny Ainge, brought together players whose reputations weren't flashy, but the data said they were undervalued. Paired with longtime Celtic Paul Pierce, Allen and Garnett became a force to be reckoned with, known as the Big Three.
Celtic buzz is back in a big way, with the additions, of course, of Kevin Garnett and Ray Allen, along with All-Star Paul Pierce.
That Celtic season felt like a season of inevitability. They were so good on a regular basis. It just felt like, okay, they start off the year at the top, it's going to be a mountain for everybody else to climb up to touch them.
The Celtics went 66 and 16 in the regular season. That's 66 games won and 16 games lost over the 82-game season. They shut down teams that usually run up the score. They looked composed, even when things got tight. By spring, the Celtics had one of the best records in NBA history, and they cruised into the Eastern Conference playoffs. But of course, regular season success doesn't guarantee anything. And the playoffs exposed some weaknesses.
They struggled a bit in their playoffs. It became clear that, okay, teams may have figured some of their juice out. They all of a sudden had to play teams that were generally better. There were no easy ones for the Celtics anymore when you get to the playoffs. When you're playing the top half of the league, okay, every night is work.
In the Eastern Conference semifinals against LeBron James and the Cleveland Cavaliers, the Celtics pushed it to a game seven and won by just five points. And the Cavaliers fall just short in an absolute game seven thriller. And the Celtics win.
Then, for just the second time since 1988, Boston reached the Eastern Conference Finals. They faced the Detroit Pistons, a team that had dominated the East for much of the decade. After a back-and-forth series, the Celtics prevailed 4-2, becoming Eastern Conference Champions and NBA finalists. But the work was just beginning. They'd have to defeat the L.A. Lakers in the playoff finals, and no one was calling this a sure thing anymore.
The Lakers hate the Celtics, and the Celtics hate the Lakers. Those two teams, they faced each other enough over time, and the Celtics, with their most titles on record, they were getting challenged by the Lakers. Lakers were getting close, so the rivalry was very real.
The Lakers were the Western Conference Champions and the Celtics' biggest rivals, meeting them in the Finals for the 11th time in NBA history. They were led by superstar coach Phil Jackson and the incredible Kobe Bryant. But unlike the Celtics, the Lakers hadn't relied on analytics in crafting their roster.
What the Lakers, frankly, had as an advantage is that they were the Lakers. They were showtime. Players wanted to play for them. If you are working for a team in a smaller market and you think you're going to get a player that the Lakers want, you're not going to get them. Basically, all the rest of the teams realized that they needed the analytics weapon to deal with someone who stood at the top with their branding like the Lakers.
The series was exhilarating to watch. It started at home in Boston with over 18,000 fans in TD Bank North Garden, decked out in green and white and cheering through the rafters.
NBA Finals games, they are incredibly loud. If you were sitting up near the top of a dome and such like that, you can't hear. Your ears will be ringing at the end, especially in a place like Boston. People waving the signs, wearing the green when they were facing the Lakers. It was loud.
It was the first time the Celtics had been this close to the championship since 1986.
Their best players, their big three definitely played well in those games, and they ended up winning by around 10.
They went all the way across country to face the Lakers in game three. The Lakers kind of turned it around. They were able to slow down some of the big three. So at that point, what they say is that the series only becomes real when a road team wins. And so the Celtics were facing game four in LA, and that was when they were able to kind of take the close win. They were able to win by six points in game four. So they had at that point a 3-1 lead. So finally, you had a road team win. The Lakers were then desperate for game five.
Game after game, the series swung back and forth. Momentum shifted. Hope wavered. By the time the Celtics returned home for game six, they led three games to two. They were desperate to close out the series.
The Lakers, actually, the Lakers did come out a little bit flat. By halftime, the Celtics had a 23-point lead. And so there was a sense that, OK, we've done a lot of the work that we need to do. We just need to maintain.
So that lead then kind of grows more in the second half. They didn't take their foot off the pedal.
In the final minutes, the energy in the Garden was erupting. Fans envisioning the NBA championship banner hanging from the rafters. The Celtics were winning 131 to 91 with just 30 seconds left. Courtside, Paul Pierce doused coach Doc Rivers with orange Gatorade, splashing news cameras.
They end up winning by 39 points.
Green and white confetti filled the stadium. Players cried and gave passionate interviews. Garnett shouted "Anything is possible" and bowed down on center court to thank Lucky the Leprechaun, the Celtics' logo.
Finally, after 22 years, the Celtics had won the NBA championships. But what few people in the arena could see was that many of the most important choices leading to this remarkable victory were made long before the opening tip.
The 07-08 Boston Celtics were lauded with awards. Kevin Garnett was named NBA Defensive Player of the Year. Paul Pierce was named NBA Finals MVP. General manager Danny Ainge was named NBA Executive of the Year. And all of Boston's 41 regular home games sold out. For Mike Zarin, who led the charge on analytics, the win meant everything.
I know what it meant for Zarin. Zarin, he got the big ring, he was so proud. Being a fan for all his childhood and then being able to add a ring onto a finger.
Zeran doesn't brag about a lot, but he definitely was able to show off how happy he was. That's the thing, right? When you're a fan, you feel like you earned it in the sense that you supported your team. You were loud and things like that. When you're Mike Zeran, you're proud because you actually made a decision that shaped that franchise. And he earned it.
For Dean, the Celtics' win was a watershed moment in demonstrating how useful analytics could be in building a championship NBA team.
I don't think it's recognized as much as it should be for how it changed the league. There were enough circumstances, oh yeah, those players were already good. But it was a big analytical decision behind the scenes. Certainly, if they hadn't changed their path, if they hadn't decided to go after Ray Allen and Kevin Garnett, they would have been in a rebuilding period for quite a while. They might have competed, they might have been playoff teams, but it's extremely unlikely without having made another deal at some point to get better players. It would have been practically impossible to win a championship.
Dean Oliver is a pioneering sports statistician at ESPN and the author of the books Basketball on Paper and Basketball Beyond Paper. You can find more details and links to his writing in the show notes and at schwab.com/podcast.
The Boston Celtics' 2008 NBA win is legendary. Of course, many factors contributed to the team's success, in addition to the choice to rely on algorithms. Factors like the individual talent of the players, the chemistry of the big three, a relentless focus on defense, and Coach Doc Rivers building a culture of trust and teamwork. But without insights from Zeran's player prediction models, it's unlikely the team would have succeeded in the same way. Today, algorithms are ubiquitous in the NBA. Every team has an analytics department, and they all strive to project how good players will be in this season and in the future.
Algorithms have shaped the game in other ways, too. They've guided teams towards shooting more three pointers, which were previously thought of as high risk and low reward compared to two pointers. Defensive switching has also become more common, where players don't guard the same person all the time. Of course, coaches and scouts still second guess algorithms, especially when a pick that looked perfect goes wrong. We explored people's tendency to lose greater faith in algorithms than in humans for equivalent missteps in an earlier episode of Choiceology. That was back in February of 2019 with my Wharton colleague, Cade Massey. The episode looked at "Algorithm Aversion," which describes our tendency, after seeing
algorithms and humans make the same mistakes, to trust flawed human judgment over superior machine logic. The question of when and how much we'll embrace algorithmic guidance and when we'll turn away from it is the focus of my next guest's research. Jennifer Logg is an assistant professor of management at Georgetown University's McDonough School of Business, and she joins me to talk about her research on "Algorithm Appreciation." Hi, Jen. Thank you so much for joining me today.
Thanks so much for having me. I'm excited to be here.
Well, I'm excited to talk about your work on "Algorithm Appreciation." Could you start by just defining what that term is, that you and your collaborators coined to describe a consistent pattern you are noticing in human behavior?
So, we find that people incorporate advice more from an algorithm than from people. What we found was that the people who received advice from an algorithm incorporated more of that advice into their final prediction than the people who thought that advice came from a person. And that's the result that we call "Algorithm Appreciation."
I really love this work, and I was hoping you could give us some examples of the kinds of algorithms that offer people advice, just so that we're all on the same page about what we're talking about here when we say an algorithm or a human could offer someone advice.
I always kind of think of Netflix. Netflix is known for its algorithm generating recommendations, but...
I could also text my friend if it's a rainy day and ask her, hey, what TV shows have you been binging?
And in this era where LLMs are everywhere, would you consider an LLM like a ChatGPT or Claude giving advice to be an algorithm category that could be compared with, you know, calling my doctor or phoning my neighbor?
Definitely. Algorithms from 10 years ago versus Gen AI today, it can do so many more things.
I'd love it if you could describe one or two of your favorite studies showing that algorithm appreciation is an important phenomenon.
So in one of our studies, our participants made predictions about the popularity of songs that came out. For example, what rank will Perfect by Ed Sheeran place on the Billboard Magazine Hot 100 this week? So how popular will it be? Then they gave their numeric answer. On the next page, they received advice that half of our participants read that the advice came from an algorithm, half of our participants read it came from a person, and they made their final prediction. So they could decide how much they wanted to listen to or disregard that advice.
And then we measured how much people updated from their time one prediction to their time two prediction. So we found that when people were making these predictions about songs, they listened more to advice when they thought it came from an algorithm than from a person.
So the same advice labeled differently as coming from an algorithm versus a person basically produces different decisions. Do you find that experts and novices are similar when they are choosing how to incorporate advice from an algorithm or a human advisor? Because I think you maybe have another study on that, right?
Yeah. So we also asked people to make business predictions, but in this experiment, we compared how non-experts, people who didn't necessarily make forecasts or predictions for a living. Compared to a sample of national security professionals, people who on a regular basis for their jobs make predictions. And what we found was that our non-expert sample, seeing the same exact questions responded quite differently than our national security professionals. We found that our non-experts were quite happy to listen to algorithmic advice relative to advice from people.
However, when the experts, these national security professionals, were deciding how much they wanted to listen to the advice that they received in the study, it actually didn't matter if they were the group of people that read that the advice came from the algorithm or advice came from other people. What the national security professionals did was discount advice from everyone. So I expected that maybe the experts knew something about these predictions that I could never expect. And maybe them discounting advice was a good thing for their accuracy. But we found that when we scored all of these predictions on accuracy,
our sample of non-experts actually made more accurate predictions than our expert sample. And that's because they were willing to listen to the algorithmic advice while the experts discounted all advice.
Yeah, the implications of that are a little scary, right? As two experts speaking to one another in a moment when algorithms are taking over. It sounds like maybe don't listen to us, but listen to the algorithms and you'll make better judgments.
It's a pretty humbling result.
Yeah. Yeah, that's really interesting. Could you tell us a little bit about why it is that people listen to advice from algorithms more than people when the advice is the same? It's not totally intuitive to me why getting an estimate and that is labeled, "This came from an algorithm," would lead me to say, "Oh, wow. Okay, well, the probability forecast I just made of whether or not this geopolitical event will happen, I should really update now, now that I've got an algorithm telling me something more than if I'm told, 'Here's a piece of advice from a group of wise humans,' or from one wise human."
So my answer might not be very satisfying.
That's okay.
One of the things that I found was when we- Gave our participants seven simple questions related to just simple mathematical questions. That was our way of capturing their comfort with numbers. So researchers have called it their numeracy. But the more comfortable people were with numerics, the more they relied on algorithmic advice. Now, why my answer might not be totally satisfying is because I tested a lot of other things and they did not actually drive the effect. So one of the things that I tested that didn't explain the effect was age.
My prediction was that people who were older might have less experience with algorithms, say Netflix algorithms, compared to younger people. So age I used as a proxy for familiarity with algorithms. And it turns out in our data, age did not correlate with people's responsiveness to algorithmic advice, their willingness to update to algorithmic advice.
Is there any sense in which giving people a deeper understanding of how an algorithm works can affect the degree of algorithm appreciation? If we sort of unpack the black box, does that exacerbate algorithm appreciation? Or actually, does it reduce it?
Yeah, so some of the first studies that I ran, I was kind of obsessed with the idea that you just brought up. And I tried to present the algorithm, which ultimately, for a lot of our lives, the algorithms in G N A I we deal with, they are black box. But I wanted to try to unpack that and understand if people knew different pieces of information about the algorithm, would knowing more encourage them to trust it more than they already seem to do? Or might knowing more actually make them more hesitant? Because now that they understand it more, they might kind of consider, you know, how things could go wrong. So I compared if people were shown calculations of what was kind of behind the algorithm.
Algorithm, the gears of the clock, if you will. And if it was presented in a simpler way, people didn't respond differently than if the algorithm was presented in a more complex way. And I would like to kind of mention Dan Goldstein's work. So Dan has a paper presenting people with different information about an algorithm, also simple and complex. And he similarly found no difference. People were quite happy to listen more to the algorithm than the person. And it didn't matter if people thought that algorithm was complex or not. And I kind of think that's a nice bridge from thinking about algorithms to Gen AI, too.
So hopefully, the research that we've done on algorithms, to some extent, also can speak to how people think about Gen AI.
What are some situations where you think people would be wisest to privilege algorithmic advice?
Kind of feels like a tightrope walk. So where and when is this useful? One of my favorite results is actually from the medical domain. And they found, this is before Gen AI. They found that when a biopsy was shown to an algorithm versus a pathologist, the algorithm was better at predicting the severity of breast cancer than pathologists. And the reason why I love this result so much is that they unpacked why it was better. The algorithm that they used was actually identifying new cues that pathologists just didn't even have in their textbook. So there were cues that were otherwise overlooked.
So regardless of how you feel about algorithms, even in the abstract, if we can learn from algorithms in Gen AI cues that we've otherwise overlooked that are predictive of forecasts that we care about, well, then we can update textbooks, even if you say, I want a person to have ultimate control.
Are there situations, though, where you would say you want people to privilege human judgment over algorithmic judgment that are sort of an edge case to what we've been talking about, where algorithmic advice? Is so powerful. So I kind of keep thinking about what Daniel Kahneman wrote in before his book on Noise came out, that algorithms are a great antidote or solution to human judgment, which can be inconsistent, because by definition, algorithms are consistent. I think the boundary conditions are, which Kahneman notes, where the data is lower quality. So if the data
you're using is based on historic data made by people that might be biased, it might be missing data, which could lead to bias. So you could end up with a really low quality algorithm and trust too much in it. Right. That's really helpful. What advice do you have for our listeners about how they can make better decisions now that they're familiar with your research on this topic? The way I think about how people can improve their lives with generative AI is really to go back to classic research on the wisdom of the crowd. We know that getting multiple predictions and judgments from groups of people is really beneficial for
improving our accuracy. And we can think of Gen AI as another source to add to that collective wisdom. Jen, what do you do differently in your own life as a result of having studied algorithm appreciation? I think a lot more about data. And if I'm getting advice that seems to be coming from an algorithm, I think more questions come up for me. Is this data complete? You might think you have data that's useful to make predictions for the future. But if you
are missing data on so many other demographics, or it could be about people's preferences, you are going to be making the same mistakes over and over again. The algorithm is going to magnify whatever patterns in the input data. So I think a lot more about where does this data come from? Is this data representative of the things that my organization and I care about? And that is kind of the driving factor behind why I
Created a class called the psychology of big data, like thinking carefully about measures and what we're feeding into algorithms in Gen AI.
That's so helpful. Thank you so much for taking the time to talk to me today, Jen. I really appreciated it.
Thank you. This was so much fun.
Jennifer Logg is an assistant professor of management at Georgetown University's McDonough School of Business. You can find her paper, "Algorithm Appreciation: People Prefer Algorithmic to Human Judgment", in our show notes, or just visit schwab.com/choicology.
To hear more about the intersection of behavioral science and sports, check out the Financial Decoder episode, "Does Your Inner Scorekeeper Skew Your Judgment?" You can find it at schwab.com/financialdecoder, or just search for Financial Decoder in your podcast app.
In the past few years, sophisticated algorithms that can offer us guidance on nearly every type of decision have gone from a futuristic concept to a normal part of modern life. Jen Logg and her collaborators have shown that unless we've just watched an algorithm make a bad decision, we tend to exhibit "algorithm appreciation", trusting and incorporating the very same advice more when we're told it came from an algorithm than when we're told it came from another person. This unquestionably reflects our appreciation that algorithms can process far more data faster and more reliably than we can. And research by John Bogard and Suzanne Shu
has shown that the more common it's become for people to rely on algorithms in a given setting, be it on the basketball court or in the operating room, the more we see algorithm appreciation arise in that setting. Our trust in algorithms is exciting because it suggests a safer, healthier future is around the corner. Self-driving cars will soon be able to prevent tens of thousands of annual traffic deaths if we trust them to take over. And algorithm-powered scans will catch cancer, heart failure, and other treatable diseases earlier and more often than doctors ever could. But it's also important that we not blindly hand over our futures to algorithms without recognizing that they aren't all created equal. While many deserve our trust,
biases like sexism and racism can be baked into algorithms if they're trained on flawed data. And like humans, recent research shows that large language models exhibit many biases. For instance, Stephen Laren colleagues have shown that they exhibit cognitive dissonance, persistently adopting views they've been randomly assigned to articulate after the fact to maintain a semblance of internal consistency. What other biases have LLMs learned from us or evolved independently? Research on this is still in its infancy, but there is some evidence that prompting LLMs to make "rational decisions" reduces biases.
Another risk of placing too much trust in algorithms is de-skilling. Recent medical research found that doctors with many years of experience who performed colonoscopies for just three months with algorithmic assistance were less effective when they returned to performing unassisted operations. If we outsource learning and thinking to algorithms, that may be okay in some arenas, but catastrophic in others. So we should lean into our algorithm appreciation, but with care and calibration, and with a recognition that de-skilling is a risk, that model quality is limited by training data and algorithmic biases, and that blind trust in any powerful tool can lead to missteps.
There are limits to the quality of any advice in a world without crystal balls.
You've been listening to Choiceology, an original podcast from Charles Schwab. If you've enjoyed the show, we'd be really grateful if you'd leave us a review on Apple Podcasts, a rating on Spotify, or feedback wherever you listen. You can also follow us for free in your favorite podcasting app. And if you want more of the kinds of insights we bring you on Choiceology about how to improve your decisions, you can order my book, How to Change, or sign up for my monthly newsletter, Milkman Delivers, on Substack. Next time, I'll speak with Richard Thaler, Nobel laureate and emeritus professor at the University of Chicago's Booth School of Business. We'll explore a bias that helps explain why most of us polish off our plates at restaurants, even when we're too full to enjoy what we're eating, and why we feel compelled to finish
boring books. I'm Dr. Katie Milkman. Talk to you soon.
For important disclosures, see the show notes or visit schwab.com/choysology.
Page updated October 31, 2026 · transcript and summary are machine-generated