686. Why Is It So Hard to Find an NFL Quarterback?

For decades, teams have tried (and mostly failed) to predict which college quarterbacks will thrive in the pros. Is the science finally catching up? Stephen Dubner speaks with a scout, a psychologist, a data warrior, and a Hall of Fame quarterback who nobody wanted.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:01Stephen J. Dubner?

  Freakonomics Radio is sponsored by Angie. Angie has been connecting homeowners with skilled pros for over 30 years, and they've made it easier than ever to tackle home projects. Their nationwide network has experts in everything from plumbing and landscaping to roofing, remodels, and more. You can read verified reviews and request and compare quotes to find your best price. So join the millions of homeowners who use Angie to get the job done well. Download the Angie app today or visit a n g i dot com.

• 0:37Stephen J. Dubner?

  Freakonomics Radio is sponsored by Southern New Hampshire University. The best investments are the ones you make in yourself, and no one knows that better than Southern New Hampshire University. With more than 200 online degree programs, no set class times, and low online tuition, Southern New Hampshire University is built to fit your life today and set you up for success tomorrow. Education is an investment. Why not make it a smart one? Apply for free at s n h u dot edu slash Freakonomics.

• 1:10Stephen J. Dubner?

  Freakonomics Radio is sponsored by Amica. You know what they say, if you want to go fast, go alone. If you want to go far, go together. So go with Amica and get coverage from a mutual insurer that's built for their customers, one that looks after what's important to you together. Auto, home, life, and more, coverage that fits your unique needs. Amica helps you protect what matters most. Visit amica dot com and get a quote today.

• 1:42Stephen J. Dubner?

  Freakonomics Radio is sponsored by Chat GPT. Chat GPT work is a new way to work in Chat GPT. Chat GPT work can pull info from across your apps and connect to your tools to help you create finished work. You can even automate work in Chat GPT with the scheduled tasks feature, and the whole time you stay in control. Put Chat GPT to work on your most ambitious ideas and projects. Get started at chat g p t dot com by selecting work mode, available on plus and pro plans.

• 2:22Stephen J. Dubner?

  If you have ever been responsible for hiring people, you may have developed a checklist of some kind, something to help you identify the perfect person for the job. So as you assess your candidates, you want to be sure that they are good at problem solving.

• 2:38Speaker 01

  They can see what's happening. They can understand what's going right. They can understand what's going wrong.

• 2:42Stephen J. Dubner?

  You will want to know if they are good communicators. X, blue, Z, nine, sevens, blah, blah, blah, blah, et cetera, et cetera. You may also want to know whether they can handle chaos.

• 2:52Speaker 02

  How much can you take in? How much can you see? How much can you handle? Can you respond when it's getting tougher?

• 2:59Stephen J. Dubner?

  In some jobs, some very specialized jobs, may even require certain key physical attributes.

• 3:07Speaker 01

  We look at height. We look at speed. We look at armor strength. We look at accuracy.

• 3:16Speaker 01

  You can't be a good quarterback and be bad at those things.

• 3:20Stephen J. Dubner?

  And in the National Football League, you can't win unless you have a good quarterback, or maybe a great one. It is the most important position in the NFL, and since the NFL is the world's biggest commercial sports operation, this means that hiring a new quarterback is a big dollar, high-risk decision. So how does that decision get made? Every year, there is a fresh crop of college quarterbacks who enter the NFL draft. College football would seem to be the ultimate audition for the pros. It generates mountains of evidence, statistical and otherwise. And then the NFL holds Combine, where all 32 teams get to poke

• 4:00Stephen J. Dubner?

  and prod the new talent, part meat market, part beauty pageant, part job fair. These teams are simply trying to predict which of the best college quarterbacks will succeed in the pros. How hard can that be?

• 4:14Speaker 01

  Quarterback evaluation is probably one of the most studied and still most uncertain problems in all of professional sport.

• 4:20Stephen J. Dubner?

  If you are a football fan, you already know this. You know that NFL draft history is littered with can't-miss quarterbacks who miss big time. You also know that some of the very best quarterbacks were overlooked in the draft. Tom Brady was the 199th player taken in his draft season, and in 2022, the San Francisco 49ers chose Brock Purdy with the very last pick of the draft, earning him the nickname Mr. Irrelevant. Purdy was the ninth quarterback taken that season, but he has turned out to be easily the best quarterback from that class. This quarterback prediction problem has been discussed for a while. The 2011 research paper found that college and combine statistics have little

• 5:04Stephen J. Dubner?

  value in predicting whether a quarterback will be successful in the NFL. Why is this such a hard problem? And what does it take to thrive as an NFL quarterback? To find out, we will hear from a Hall of Fame quarterback, a professional scout, a pioneering sports psychologist, and a Silicon Valley entrepreneur who wants to fundamentally change how athletic potential is evaluated. Also, you'll hear me take a very specialized IQ test.

• 5:36Speaker 03

  Now you're going to see more designs.

• 5:39Stephen J. Dubner?

  That's brutal.

• 5:40Speaker 03

  boxes to make the designs you see.

• 5:41Stephen J. Dubner?

  That's brutal. Whoa, what happened there? Mama, brutal.

• 5:50Stephen J. Dubner?

  Today on Freakonomics Radio, how to pick, or in my case, how not to pick, a great NFL quarterback.

• 6:09Speaker 04

  This is Freakonomics Radio, the podcast that explores the hidden side of everything with your host, Stephen Dubner.

• 6:26Stephen J. Dubner?

  The rookie quarterback that everyone has their eyes on this year is Fernando Mendoza. The Las Vegas Raiders took him with the first overall pick in the draft. Last year, Mendoza led Indiana University to an undefeated season and its first ever national championship. Will Mendoza thrive in the NFL? No one knows for sure, but the expectations are huge. Mendoza got very high grades from most draft analysts. Although we should say, these analysts are often dead wrong. They make sins of commission and sins of omission. Consider this former quarterback who wasn't even invited to the NFL Combine.

• 7:04Speaker 02

  I was told that I had a grade that was good enough to go to the Combine, but they passed me up to take a guy from a bigger school.

• 7:11Stephen J. Dubner?

  That is Kurt Warner. He grew up in Iowa and played his college football at Northern Iowa, which is not a football powerhouse. What's he do now?

• 7:20Speaker 02

  NFL analyst on ESPN NFL Network.

• 7:24Stephen J. Dubner?

  In 1994, after Warner's final season with Northern Iowa, he entered the NFL draft, but not a single team was interested enough to use a draft pick. The Green Bay Packers invited Warner to join their training camp, but he was released before the season started. He kept himself in shape in case he got a call, and he took a job stocking shelves at a Hy-Vee grocery store.

• 7:47Speaker 02

  That was probably the most difficult time of my career, and not just because you go from inches to accomplishing your dream to working for 15 hours stocking shelves in a grocery, not just because of the distance you can see between those two situations, but it was more because no one had ever gone from aisle seven at Hy-Vee to the NFL.

• 8:09Stephen J. Dubner?

  That's the story we hear, because you didn't make it in the NFL in the normal route. Is that a story you're sick of, or do you take pride? I'm curious.

• 8:19Speaker 02

  I do celebrate it a little bit more because I think my story is way more like real life than most professional athletes. Most guys that get their scholarship and play great in college and get drafted and have great careers, that's not what normal life is for most people. Most people have their supermarket moment along their journey as they're trying to chase some dream and trying to get to the mountaintop.

• 8:45Stephen J. Dubner?

  Kurt Warner wasn't just disappointed that he had a hard time making the NFL. He was also surprised.

• 8:51Speaker 02

  Every time I had played, all I knew was, I'm successful. You can't show me a time when I was between the lines, ball in my hands, and I needed to play well, and I didn't. And so that's where my mindset was when I was going through all this is like, why can't somebody see that?

• 9:09Stephen J. Dubner?

  But Warner stuck with it. He joined the Iowa Barnstormers of the Arena Football League, and after three seasons there, he was offered a futures contract by the St. Louis Rams of the NFL. They sent him to play in Amsterdam for NFL Europe. After leading that league in passing yards and touchdowns, Warner was summoned back to St. Louis to play for the Rams. And he got his chance when, in a preseason game, the Rams' starting quarterback was badly injured.

• 9:37Speaker 02

  I was put into the game, and I only played one drive. They were trying to figure out, okay, we don't want Kurt to get hurt as well. So, you know, we're gonna give him one drive. And they called a pass play. It was a 20-yard in-breaking route. I threw it and completed it, didn't think anything of it. And as we were going back and watching the film of that game, Mike March was my office coordinator at the time, and he stopped that play. And he goes, okay, Kurt, thinking about making you the starter, and it's gonna be your team. And we watched the play, and it looked really tight. And I know he was kind of alluding at like, hey, if you're gonna be our starter, I'm not sure you can take chances like this. You just go back and throw it. And I remember thinking in the game, like,

• 10:20Speaker 02

  That was easy. Like it wasn't tight. It wasn't a tough throw. I really didn't understand why he wanted to make a big deal of this particular play. And so he goes, okay, Kurt, I want you to walk me through why you threw this. I want you to take me through your thought process. I remembered it vividly. I still do to this day. When I came back to throw that, there was a big in route, as I was talking about over the top. It was like a 20 yard route. And then we had something underneath it at like five or six yards. And so the whole goal for me in simple terms, right, was I got a high and I got a low, and there's one defender in between those two guys. And my job was to read that defender. And if he stays low, then I throw it high. If he goes high, I throw it low, like, simple,

• 11:02Speaker 02

  right? And so I remember telling him, well, the key was that linebacker. When he started to drop, I saw his hips turn to the inside, and his shoulders just shifted enough that I knew because of that shift, he could not stop the big in route. Once I saw that, I threw the end because I knew he had no chance to get there. And so we watched it on film, and I'm like, right there. There you see it. And he just looked at me and he goes, okay, I can live with that. So it would ultimately become the route that built my house is what I say, because we threw that more than anybody else.

• 11:45Stephen J. Dubner?

  And so it was that Kurt Warner overlooked, undervalued, but undeterred, wound up becoming one of the best quarterbacks in the NFL. He appeared in three Super Bowls, two with the Rams and one with the Arizona Cardinals. The Rams won one of those Super Bowls, and Warner was voted the game's MVP as well as MVP of the league that year. Now he's in the Pro Football Hall of Fame.

• 12:08Speaker 02

  I believe we can get better at just about anything if we immerse ourselves in whatever that is, and we take the time to learn and grow and subject ourselves to it.

• 12:18Stephen J. Dubner?

  Once you were established in the league, and the opposing coaches could see what you were good at, how did they try to counteract that? And how did you then counteract what they were counteracting?

• 12:30Speaker 02

  Yeah, I think the biggest thing, first and foremost, that teams would try to do is they would try to blitz me. It's like, okay, if we're going to sit back here and just play coverage, we don't have a chance. Kurt's really good at that. He sees it. So we got to speed this up, right? Okay, it's four to five seconds on a field. Well, let's see if I can make that three seconds. He's got to take all that information in, and he's got to respond correctly in three seconds. Maybe we've got a chance if we speed him up, and he'll make bad decisions. I thought that was the greatest thing ever. Like, the faster, the better. I played three years of arena football, smaller field, everything was faster, decisions had to be made faster. When I got to St. Louis, we played on what

• 13:13Speaker 02

  they called the old turf. It was, you know, a baseball turf. There was no padding underneath it. And it was just a faster game. There is no doubt in my mind, you try to speed it up, you're going to make more mistakes than I can. The second thing that they had to try to do was, it became more about scheme. I talked about the route that built my house. So now teams had to just kind of cheat to go, we got to try to take away what they do well, and get the advantage before the play even starts. Otherwise, we don't really have a chance. So that became the cat and mouse of who we were and how we got really good. And then how do you stay good is, okay, we know teams are going to try to stop us from doing what we've really built this offense on. Now can we complement that, right? So

• 13:56Speaker 02

  for a quarterback, hey, can you throw outbreaking routes as well as you throw in breaking routes? Can you see these particular coverages? Can you connect the mental and physical of something different, as well as you did to the stuff that you were very familiar with? And so that became the challenge for me and for us was, I want to be great at everything.

• 14:16Stephen J. Dubner?

  I'm starting to understand why there's only a handful of guys on the planet who can do this on a high level every week. It's a really hard job, isn't it?

• 14:24Speaker 02

  It is a hard job, and there's a lot that goes into it.

• 14:26Stephen J. Dubner?

  When I think about the cognitive and processing speed, it's really hard to comprehend, I think, for a layperson. So do you feel a bit like you're in a kind of separate layer of humanity when you have those abilities?

• 14:40Speaker 02

  Yes, I do. Just in my realm. When it comes to my job, our job as an NFL quarterback, yes, I do think that I see and understand the game differently than most people.

• 14:54Stephen J. Dubner?

  Some people say that when you're watching professional athletes, you're basically seeing the guys who, unlike everybody that fell away, they are the guys who perform under pressure. You can be great, but if you don't perform under pressure, you don't become a pro. So I'm curious to know how that developed, or how that came about for you.

• 15:14Speaker 02

  I don't know the answer to that. I know a lot of people talk about, hey, the greatest to ever play are the ones that elevate their game in the biggest moments. I don't believe that. What I believe is that most people shrink under pressure. They don't embrace that. They don't play to that level. And so I always felt like, when the game was on the line, I was the best version of myself. I can't put my finger on why that was. I know that I played hundreds of Super Bowls in my front yard. I played hundreds of NBA championships in my driveway. I pitched a number of World Series championships into

• 15:57Speaker 02

  taped off wall. I do believe when you talk to the great specifically in sports, that seems to be a thread with all of them, is I want the ball, I believe I'm going to be successful, and whether I'm successful or not, that doesn't affect me from believing I'm going to be successful the next time. I really felt like, very early on in my career at Northern Iowa, that I was the most talented guy there. But I still couldn't get on the field. I had a friend on the team, and I just asked him if he would mind talking to the coaches on my behalf, because he had a better relationship. Just asking him, you know, what's Kurt missing? So this was after practice one day, and I'm sitting in the

• 16:39Speaker 02

  locker room, and I'm all by myself, just waiting for him to come back in. And finally shows up, and he goes, all right, Kurt, here's what they told me. They told me, the reason that you're not starting is because you're not very good in practice.

• 16:49Stephen J. Dubner?

  Wow.

• 16:50Speaker 02

  And I was, you know, I was disappointed, and I was angry because I thought it was just an excuse. And the more I got to think about it, the more I started to realize, okay, if I'm in high school, I play nine or 10 games a year. If I'm in college, we'll play 12, 13 games a year. And then when I got to the NFL, it was 16. And so as I started to put all of this into my mind, I started to realize, wow, about 95% of our life is lived in practice. So everything changed because I started to think about it differently. And it became something that I applied to every area of my life. What kind of practice player are you going to be? My viewpoint is, man, if I get all the little moments,

• 17:35Speaker 02

  if I practice well every day, man, I'm going to get this whole thing right.

• 17:39Stephen J. Dubner?

  So if you're a talent evaluator, let's say you're a GM for an NFL team, or a head coach, or someone involved in the drafting, how well can you measure what you're talking about now, this intense level of self-belief?

• 17:53Speaker 02

  I don't think you can. That's one of the reasons that this position is so difficult to evaluate, is that you can see guys, and I've seen a lot of them, that are great in college, play at the biggest schools, play in the biggest moments, and they're great. And they get to the NFL, and they never get any better. And then there's other guys, they're able to elevate themselves, they're able to think faster, they're able to acclimate to the speed of this new game. I think we have to throw people into the deep water and go, show us, can you handle the complexity of this game? Can you handle the speed? Can you handle the pressure? It's going to be faster. It's going to be more intense. There's

• 18:36Speaker 02

  going to be more on the line. And we're not going to know how somebody handles that until they show us how they handle that.

• 18:49Stephen J. Dubner?

  There are a lot of morals to the Kurt Warner story. For our purposes today, I think the main one is this. When you draft your quarterback, what you're betting on is potential. Past results, as the mutual fund people like to say, do not guarantee future performance. So how are you supposed to figure out that bet? Historically, this job has been the domain of NFL scouts, like this one.

• 19:14Speaker 05

  My name is Dan Hatman, and I am the Director of Scouting Development at the Scouting Academy.

• 19:19Stephen J. Dubner?

  Hatman worked in the scouting department of three NFL teams, and he left in 2014 to launch his own firm to teach others what he had learned.

• 19:27Speaker 05

  When I first started, I assumed it was going to be only people that wanted to work in the NFL. And we have had those. In fact, we've had 93 people go on to work in the NFL. But we have had people that wanted to be in media. We've had people that want to be agents. We've had people that want to be in fantasy football. My favorite of all time was a gentleman out of Georgia. He was 62, the pediatric surgeon, and he wanted to be smarter than his sons on Sunday. So he came and took the class just so he could run circles around his boys on Sunday.

• 20:03Stephen J. Dubner?

  I would think, and I think most football fans would think, that with the amount of data that's produced by a top tier college quarterback, and the amount of scrutiny, that choosing a great quarterback for the NFL would be pretty easy.

• 20:18Speaker 05

  It is very much harder than it looks.

• 20:20Stephen J. Dubner?

  For someone from the outside who might say, well, I don't understand. I mean, these people are doing essentially the same thing in two different environments that are fairly similar. What's your best explanation for that sort of philosophical problem?

• 20:35Speaker 05

  I've been exposed to this concept from a physicist, what's called the N-body problem. And that is, the more worlds and objects that are interacting, the more complicated the interactions. In football,

• 20:49Speaker 05

  You're not isolated, right? He is one with 10 in front of and around him, facing 11, being coached by some, playing against those that are coached by others. We can reduce these players in a very detailed way, but that runs into a wall. You have to understand enough about why he did what he did in the last place and enough about what you're going to need him to do in the NFL and then make that gap close.

• 21:18Stephen J. Dubner?

  Name five or 10 or however many things you need to have to have a chance to be really good as an NFL quarterback.

• 21:29Speaker 05

  My favorite way to say it is the physical sets the floor. So if you're really big, really fast, really strong arm, really accurate, that's the floor. The ceiling is the mental. That is the processing, the decision-making, the toughness, the intelligence, the coachability. Those are the delta and we get kind of sucked in by that floor, right? The big and the fast and the strong. And then we, we buy the hope, we buy the potential upside.

• 21:57Stephen J. Dubner?

  So there's one paper by Wolfson, Donna Schmicker from 2011 called "The Quarterback Prediction Problem." This was published in the Journal of Quantitative Analysis in Sports. These authors write, "College and combine statistics have little value for predicting whether a quarterback will be successful in the NFL." Let's start with that. Agree or disagree with that statement?

• 22:18Speaker 05

  Agree. I think the predictive nature is thin. The data can allow you to have a triage mechanism. If a player is not able to complete passes in college, there's very little probability they're going to improve on that. But being able to complete in college doesn't predict completing in the NFL.

• 22:38Stephen J. Dubner?

  Furthermore, they write, "The NFL teams aggregate pre-draft information, including qualitative observations, quite effectively, and their inability to consistently identify college quarterbacks who will excel in the professional ranks is a consequence of random variability in future performance due to factors which are unlikely to be observable." So talk to me about those unobservable factors, what they are and how you try to think about them.

• 23:04Speaker 05

  I could watch a college player go out and his receivers run the routes. He delivers, they catch, touchdown, right? That's an objectively good outcome. And that objectively good outcome may lead me to believe that, well, he did a good job. But I have to make an assumption on, well, what was he supposed to look at? And then did he process that well? If he did process that well, did he make the right decision? If he made the right decision, how does that fit in the larger context? And there are times in there where you're going to see a player who has really good process have a bad result and you're going to have those opposites of when that player is going to have a great result and it was really bad process.

• 23:44Stephen J. Dubner?

  Which NFL teams historically draft better than average and why?

• 23:49Speaker 05

  Classically, it's been Green Bay, Baltimore. More recently, the Rams have done a good job. Philadelphia has done a good job. Quarterback wise, I mean, I'd put Andy Reid on the Mount Rush more and I don't know how many other people I could put up there.

• 24:08Stephen J. Dubner?

  Well, don't you have to have three more to have a Rush more?

• 24:10Speaker 05

  Yeah, that's what I'm saying. It's, you know, I would hope that I could and I'm not sure I would be able to.

• 24:15Stephen J. Dubner?

  Talk to me for a bit about how cognitive abilities are measured. Football fans have heard for years about the Wonderlic, which I assume is either not used much or at all anymore. So just talk to me about that style of testing and how valuable they are.

• 24:29Speaker 05

  So it was the Wonderlic for a long period of time. I think the teams have known for a long time, it wasn't predictive. It was more for if a player scored low, maybe that's a flag that we need to do some more research and figure out how do they learn as opposed to being a disqualifier. I think when we got to the HRT period of time, HRT is Human Resource Tactics. These are things taken from a military context to look at, not a straight intelligence, but more of a performer under pressure. That started to elevate things. And then with things like AIQ, what Scott and those guys are doing, AIQ is really smart stuff.

• 25:10Speaker 06

  My name is Scott Goldman. I am a sports psychologist. And then I'm also the owner of a company called the Athletic Intelligence Quotient, or the AIQ.

• 25:21Stephen J. Dubner?

  Coming up after the break, the sports psychologist who is trying to build a better prediction engine. I'm Stephen Dubner. This is Freakonomics Radio. We will be right back.

• 25:34Stephen J. Dubner?

  Freakonomics Radio is sponsored by Mint Mobile. Summer means freedom. As a kid, no school, no homework. Summer still means freedom for adults, thanks to Mint Mobile's summer sale. All of Mint's plans are $15 per month right now, even unlimited. Say no to the gimmicks and gotchas of big wireless and switch to Mint for high-speed data and coverage on the T-Mobile 5G network. Choose from three, six, or 12-month plans and say goodbye to a monthly bill. You can even bring your current phone and your number. To get your new wireless plan for just 15 bucks a month, go to mintmobile.com/freak. That's mintmobile.com/freak. Cut your wireless bill to 15 bucks a month at mintmobile.com/freak. That's it. There's no

• 26:19Stephen J. Dubner?

  catch. Upfront payment, $45 for three months, $90 for six months, or $180 for 12 months. Plan required, $15 per month equivalent. Taxes and fees extra. Initial plan term only. Greater than 50GB may slow when network is busy. Includes up to 20GB hotspot. Capable device required. Availability, speed, and coverage varies. See mintmobile.com.

• 26:45Stephen J. Dubner?

  Freakonomics Radio is sponsored by Granola. You know that moment when a meeting ends and someone says, "Let's circle back on this next week," and everyone nods, but no one writes it down? Then next week rolls around and it's radio silence? That's why Granola exists. Granola is an AI-powered notepad built for the way real people actually meet. You take rough notes like you normally would, and in the background, Granola securely transcribes the meeting. Then it turns everything into clean, structured, actually useful notes when the meeting ends. And the best part? Granola works through your device's audio, which means it integrates seamlessly into the video conferencing tools you already use. No setup, no awkward bots. It's just your normal meeting with superpowers. If meetings are eating up

• 27:26Stephen J. Dubner?

  your day, Granola is a no-brainer. You can try it totally free. Just head to granola.ai/freakonomics. That's granola.ai/freakonomics to get your time back. Try it for free at granola.ai/freakonomics.

• 27:47Stephen J. Dubner?

  Freakonomics Radio is sponsored by hotels.com. Hotels.com helps you find and book the right stay for work faster. With business preferences, save your travel must haves, like gym access or breakfast, just once, then get tailored results whenever you search for work travel. Plus, with a free membership, you can save up to 20% on hotels and earn rewards on every stay for future travel. Book now at hotels.com. Visit hotels.com for details.

• 28:27Speaker 06

  To really do good science, it takes a tremendous amount of training. We talk about it being a 25-year overnight sensation. It's like, we had to make sure the science was sound.

• 28:38Stephen J. Dubner?

  That is Scott Goldman, the founder of Athletic Intelligence Quotient, or AIQ. Before that, he was a practicing psychologist.

• 28:47Speaker 06

  I have two PhDs, one's in clinical psychology, one's in school psychology.

• 28:51Stephen J. Dubner?

  At the University of Arizona, Goldman became one of the first sports psychologists embedded within a Division One program. His work there caught the attention of the NCAA, which in 2013 asked Goldman to join a new task force on mental health. By then, he had been thinking about the quarterback prediction dilemma for a while. One classic example came from the 1998 NFL Draft.

• 29:15Speaker 06

  The big debate was between Peyton Manning and Ryan Leaf. Ryan Leaf was known as being physically dominant, but he had kind of a relatively short exposure career. Peyton Manning, on the other side, had less of the physical attributes, but, you know, his dad was a legend, Archie Manning. So as I was watching the debate between these two quarterbacks, who goes one, who goes two, I said, "I think we could build a better mousetrap regarding, at least, understanding the intelligence part."

• 29:44Stephen J. Dubner?

  The Wonderlic test that was used for years by NFL teams was created in 1936 by a Northwestern University graduate student named Eldon Wonderlic. It was meant to help an employer assess the problem-solving and reasoning abilities of prospective employees. Peyton Manning and Ryan Leaf had gotten nearly identical scores on their Wonderlic. The Indianapolis Colts took Manning with the number one draft pick, and he wound up playing in four Super Bowls, winning one with the Colts and another with the Denver Broncos. When Manning retired, he was a Hall of Fame pick. Ryan Leaf, meanwhile, was drafted number two overall by the San Diego Chargers. But Leaf got hurt in his second season and within

• 30:29Stephen J. Dubner?

  a couple more years, he was out of the league. By then, he had earned a reputation as a bad teammate and a malcontent. How could two college quarterbacks who received nearly identical pre-draft grades have turned out so differently? And what if there was a better way to figure that out before the draft? Scott Goldman and his partner Jim Bowman set out to build a test that might help.

• 30:54Speaker 06

  We spent 15 years studying unsolvable puzzles, not just in sports, but we also looked at military, first responders, so firefighters, police officers, et cetera. So anytime like a firefighter has to kick a door down and scan looking for threats and danger, as well as victims to save, that's the same as a quarterback.

• 31:12Stephen J. Dubner?

  Goldman found that a comprehensive athletic profile consists of four domains.

• 31:18Speaker 06

  First domain is physical. Are you big? Are you strong? Are you fast? Bucket two is knowledge or experience, what they already know from the systems they came from. Bucket three is personality or character, and then bucket four is intelligence.

• 31:31Stephen J. Dubner?

  And what is intelligence in a quarterback context?

• 31:34Speaker 06

  It's things like decision-making and processing speed and visual spatial understanding. So the quarterback or the play caller calls the play in, which is some super complicated, like learning a whole nother language, like trying to understand French for me, which I can't speak French, right? X, blue, Z, nine, sevens, blah, blah, blah, blah, et cetera, et cetera. So first part of intelligence, right, is can he translate that? Can he recall what that play is? Some quarterbacks can do that easier than others. The ones that struggle, they have the wristband, right? So the play caller goes, hey, it's 17 on your wristband. And then the guy looks up, oh, hey, 17, this is what we do. So after the play is called, he's got to scan and identify another intellectual property. So he's got to identify the defensive coverage.

• 32:19Speaker 06

  They're getting talked to in the headset. So even while they're multitasking, the headset's also kind of chirping in their ear. And so imagine like you're trying to figure out this puzzle. You've got your coach giving you advice and guidance. All of a sudden, the defense shifts. And you're like, shoot, was the instruction that was just given meant for the defense that we were both looking at or the defense that we thought was coming?

• 32:41Stephen J. Dubner?

  The only thing I can sort of equate it to, and I have no experience at this either, but it feels a little bit like you're doing air traffic control and you're flying the plane at the same time.

• 32:50Speaker 06

  Yeah.

• 32:51Stephen J. Dubner?

  Can you describe generally the attributes of a person that's really good at that?

• 32:57Speaker 06

  You need someone who's strong enough to make the throw. You hope for somebody who's got a sophisticated knowledge of the game. You hope that their ability to handle the pressure, the stress, and also the leadership qualities to keep everybody else on track. And then you also hope that they have intelligence. And that's where the AIQ comes in.

• 33:19Stephen J. Dubner?

  Goldman and Bowman built out the original AIQ database using thousands of college athletes. Today, the AIQ is used by NFL teams as well as teams in all the other major pro sports leagues in the US and in many other countries. So how good is the AIQ at predicting whether an athlete will thrive?

• 33:39Speaker 06

  I mean, not to get too nerdy, but the place that I start with is we're the only instrument that has five published papers. And when I say published papers, I mean professors from universities took our data and they found statistically significant correlation with on-field performance.

• 33:57Stephen J. Dubner?

  I'm looking at one, it's called Head in the Game: The Impact of Cognitive Abilities on Performance of NFL Quarterbacks. This looked at data from 42 quarterbacks who had taken your test at the NFL scouting combine. The authors tested whether AIQ subscales visual spatial processing, reaction time, decision-making, and learning efficiency, whether they added predictive value beyond draft position alone. And the answer was basically a big fat yes. The numbers are kind of remarkable to me, so you must feel good about them, yes?

• 34:28Speaker 06

  Well, it just means that we're on the right track, right? And it was QBR, quarterback rating. It was higher AIQ scores tended to make less turnovers. It was passing yards, passing TDs, time efficiency, career approximate value, career longevity. So we found pretty meaningful stuff.

• 34:45Stephen J. Dubner?

  Can you just describe the test for me, including how it's administered?

• 34:49Speaker 06

  It's all done on an iPad. So what it looks like to the player is it looks like you're kind of playing a lot of games, kind of like Candy Crush or Tetris, depending on how old you are.

• 35:00Stephen J. Dubner?

  How long does it take to take the test?

• 35:03Speaker 06

  35 minutes.

• 35:04Stephen J. Dubner?

  What is the range of scores?

• 35:05Speaker 06

  100 is the mean score. And then you have a 15-point standard deviation in each direction. Anything between 90 and 104 is considered an elite athlete's normal profile. If you get 105 to 114, that's a strong score. And 115 or higher is an elite score, meaning like this is something that's really a gift and makes life easier for you. And then on the flip side, 85 to 89 is like an area to start to pay attention to. And 85 and lower is, hey, let's come up with a strategy that can help offset this.

• 35:39Stephen J. Dubner?

  The one thing I don't understand, Scott, is how a non-athlete is expected to do on the AIQ. In other words, how geared is it toward intelligence within the construct of the sport? I, for instance, not a professional athlete, unfortunately.

• 35:57Speaker 06

  Well, I think that's just the beauty of it, right? Is there's all kinds of aspects and forms of intelligence, right? So how would you do on the AIQ? I don't know, but I'm willing to let you take it and we can find out.

• 36:12Stephen J. Dubner?

  Can't wait. Is there anything I can do to prepare?

• 36:15Speaker 06

  Get some good rest. This is what I tell the players. I just say, give your best effort, work as quickly as you can without making any errors, and enjoy the process. It's a challenging test, but most people do better than they think they do because science has often shown that we tend to remember the questions we got wrong more than the questions we got right.

• 36:33Stephen J. Dubner?

  You don't have like an age-adjusted grading curve, though, do you?

• 36:36Speaker 06

  We do not. How old are you?

• 36:38Stephen J. Dubner?

  Damn it, 62.

• 36:39Speaker 06

  You're going to be a little bit outside of our optimal curve.

• 36:43Stephen J. Dubner?

  Oh, I would think I'm a lot outside your optimal curve.

• 36:46Speaker 06

  Well, I was trying to be polite, but yes. And sorry to say this to anyone over a certain age, but our cognitive abilities do start to decline at around age 35. What we'll do is we'll take your scores and we'll just put a little asterisk by it and note.

• 36:59Stephen J. Dubner?

  No, no, no, I don't want an asterisk. But you're saying there's probably not a lot of chance that my AIQ score is going to be so good that the Pittsburgh Steelers, for instance, are going to want me to come and replace Aaron Rodgers?

• 37:11Speaker 06

  Well, let's see what the results are before we come to that kind of decision. And look, if it's as promising as we hope it to be, I know that there's more than just the Steelers that are looking for a good quarterback.

• 37:28Stephen J. Dubner?

  Scott Goldman set me up to take the AIQ. So I sat down in my office with an iPad and our production assistant, Alaria.

• 37:37Speaker 07

  Okay, there are 10 subtests. Each is two to three minutes long. Have fun and do your best.

• 37:43Stephen J. Dubner?

  No problem. Here we go.

• 37:45Speaker 08

  In a moment, you will be asked to do a number of things like rotating shapes in your mind and making quick comparisons between numbers.

• 37:51Stephen J. Dubner?

  I can feel my heart racing. Is that what it's supposed to do?

• 37:54Speaker 08

  To begin, good luck.

• 37:56Stephen J. Dubner?

  Thank you. Okay, start. Shape rotations.

• 38:01Speaker 08

  Paired associative learning. The first pair is already there.

• 38:04Stephen J. Dubner?

  Oh, man. No way.

• 38:07Speaker 08

  Object scanning. In this activity, you'll be asked to search the screen, looking for certain objects. Route finding. Now you will be asked to find the shortest. Simple reaction time. Memory for shapes.

• 38:17Stephen J. Dubner?

  Whoa, whoa, come on, seriously?

• 38:21Speaker 08

  Number matching. Choice reaction time. Design matching. Continue to touch each box until it matches the design.

• 38:26Stephen J. Dubner?

  Oh.

• 38:27Speaker 08

  Now you're going to see.

• 38:29Stephen J. Dubner?

  That's brutal.

• 38:30Speaker 08

  Boxes to make the designs you see.

• 38:32Stephen J. Dubner?

  That's brutal.

• 38:33Speaker 08

  Each time you have finished a design, press begin.

• 38:36Stephen J. Dubner?

  Really hard. I'm going to do very poorly at this one.

• 38:42Stephen J. Dubner?

  All right, Scott, I took your fiendishly difficult test the other day. I think I know the sections that I did really badly on and then some sections I did better on, but I don't know, you're the one that wrote the test and you scored it. So how did I do overall? I'm about 5'11", so I need to add at least a couple inches, a bunch of pounds, and have a much stronger arm, and I need to be 40 years younger. But other than that?

• 39:05Speaker 06

  You're ready. Other than that, you're ready. So the first question that most players will ask is, so how did I do? Did I do good or did I do bad? What I always try to remind them is two things. There really is no good or bad. It really is just describing how you do things. My guess is as I go over this with you, you're going to go, I did know that, or I knew that about me, or oh, hey, that kind of explains a little bit about what I do. And then the other thing is remember that there's no one element or piece of data that perfectly defines your capabilities. We have a section on the report, and I don't know if you're looking at it right now, but on this first page, you'll see something called positional focus. I went and I talked to coaches that I've worked with over the last 30 years in the

• 39:50Speaker 06

  NFL and collegiate space for every position, as well as offensive coordinators, defensive coordinators, et cetera. And I just said, what are some of the intellectual abilities that you most value in a quarterback? So I collected all this data of what each coach is looking for for every position. My partner, Jim, he looked analytically at some of the highest performers, MVPs, Josh Allen, Lamar Jackson, Patrick Mahomes, those folks. And he looked at where they were outperforming their peer group on the AIQ. We did this independently. We came together and we had an 87% alignment. And that's what we call the positional focus. So the most important element that we have

• 40:34Speaker 06

  found to be of value is something we call learning efficiency, which is the ability to kind of like download and absorb a playbook. It's the ability to recall it. That was an area that you scored in the strong range on. The other thing that I sent you all was our AI bot. We call it the AIQ plus. We fed it over 297 documents of just sports specific intelligence theory. Then we added to it everything we've ever done, like every development book we ever made for a professional team, every player profile. I asked it, what style of offense is Steven best suited for? What's the best way to coach them? And one of the things it picked up on was your strong learning efficiency. So it talked about things like you're being able

• 41:18Speaker 06

  to run more complex playbooks that you could absorb the playbook faster than your peers. The second area that you scored really well on was called spatial awareness, which is really about pocket presence. It's knowing where you are in relation to teammates, opponents. When you think about the pocket and stepping up in the pocket to avoid a last second hit, that was another area. And then the next one that was one of your highest scores was something called multiple target search, and that's the ability to scan. So it's kind of like you go high, low, you do full field scans, you look at coverage based answers. What this says is you're less likely to make errors and you're more likely to be a great game manager.

• 41:59Speaker 06

  So fundamentally, you scored in the strong range on one, two, three, four, five, six, seven, eight, eight of the 16 abilities, which I think is pretty impressive.

• 42:13Stephen J. Dubner?

  So first of all, plainly, I made a bad career choice. I should not be podcasting. I should have been a quarterback, obviously.

• 42:20Speaker 06

  Yeah, I mean, who knew?

• 42:21Stephen J. Dubner?

  Let's say you got this report on this quarterback named Dubner, and you don't have a scheme that fits. What do you make of this prospect then? Do you say, okay, this guy's got talent, he's got cognitive ability in a certain direction, but he's not right for us?

• 42:38Speaker 06

  Here's the luxury of the role that I sit in. I'm an information provider, but I'm not a decision maker. So what I do is I try to tell the head coach and the GM, I think you've got a choice. Do you love his physical talents? Or do you want to go let him be good for someone else if you know he can't be great for us? And sometimes they say, we still like him. We like his arm strength. We like his mobility. We like his throwing angles. We like the coach that he came from. We still like him, even though he might not be a great system fit. How can we bridge that? And that's where I'll ask the AIQ plus. I'll say, hey, what's the best way to coach this guy? And that's one of the things that I asked the AIQ plus about you. You want to know what the first thing that really

• 43:23Speaker 06

  popped to me that the AIQ plus said that I loved?

• 43:26Stephen J. Dubner?

  Yes, please.

• 43:27Speaker 06

  It said that you should have predefined panic answers.

• 43:34Stephen J. Dubner?

  That doesn't sound good.

• 43:35Speaker 06

  No, it's actually, it makes perfect sense to me based off the profile. What it's saying is if the situation goes to [bleep], I don't, sorry.

• 43:44Stephen J. Dubner?

  No, you can say it's okay.

• 43:45Speaker 06

  If the situation goes awry, let's be sophisticated, right? If the situation goes awry, some people are really great at improvising. And your ability to download such a high volume of information, we're better off teaching you that the situation could go awry. And if and when it does, here are some predetermined solutions or scripts that you can activate. I think you could be kind of like a Brock Purdy. I did ask the AIQ plus out of curiosity, how to compare you to Patrick Mahomes and Josh Allen.

• 44:19Stephen J. Dubner?

  And?

• 44:20Speaker 06

  It said that they were better able to handle the chaos and the unpredictable. And you had some real strengths in the ability to manage a game. Just generally speaking, intelligence is one of the greater predictors of success. People with high levels of intelligence, they have the ability to discard a calcified way of doing something. When things go awry, your ability to improvise is very much based off of intelligence. So sports, firefighters, police department, and currently, AIQ is being used by many of these types of professions, military, special forces, airline pilots. There are other aspects of intelligence that we do not measure that might have to do more with, say, analytics or math or language.

• 45:04Speaker 06

  But for us, what we've talked about was high pressure, chaotic, constantly mutating puzzles.

• 45:10Stephen J. Dubner?

  Let's say my AIQ score came in and I'm a current prospect in the draft. Which teams would benefit from a quarterback like me?

• 45:20Speaker 06

  Ooh, I think there's a lot. I would go more with the play caller than the team because that's the thing you really want to align is the play caller and the quarterback. So play callers that I think are amazing, that you would fit wonderfully with, and again, assuming everything below the neck is of an elite, superior level. I think you'd be great in Minnesota with Kevin O'Connell. Unbelievable, brilliant mind, brilliant play caller.

• 45:44Stephen J. Dubner?

  You know, I look pretty good in purple.

• 45:47Speaker 06

  I didn't know we were adding that as one of the variables of consideration. How do you feel about colder states versus warmer states?

• 45:54Stephen J. Dubner?

  Minnesota, first of all, they're in a dome, so I'm not going to wear. And also, like the downtown in Minneapolis, they have those skyways that are heated and I'm okay with that.

• 46:02Speaker 06

  All right, so we'll put in a word with the Vikings for you to potentially be an undrafted free agent.

• 46:07Stephen J. Dubner?

  What's the youngest player you've tested with AIQ?

• 46:11Speaker 06

  16.

• 46:11Stephen J. Dubner?

  What if someone came to you and said, I've got a 12 year old quarterback, or maybe I run an academy where I've got 100 12 year old quarterbacks. And I really believe in what you're doing, Scott. I think it's meaningful and the data show that it's predictive, right? That people who do better on their AIQ do better in football. And it's more predictive than a lot of the more traditional yardsticks. And even though I understand that AIQ is measuring who they are, I also wonder if we could use it a little bit the other way around to coach them up cognitively while they're young. What do you think of that idea?

• 46:47Speaker 06

  That's what we created it for, was not to be a predictive model, but to be a descriptive tool. I've never met a player that loves to get mother effed. And I've never met a coach that really loves to do that. But the frustration between the player and the coach happens because there's some kind of disconnect. And if you look at the way that we designed the AIQ reports and the AIQ plus to talk, it's very much a descriptive element. We are describing what the player can do and how the player processes the game, where it was like, okay, so this is what we would do with this guy. This is how we would get him, you know, to really handle the rigors and the pressures of the game.

• 47:33Stephen J. Dubner?

  We will be right back with the final act in our story about how to pick an NFL quarterback. I'm Stephen Dubner. This is Freakonomics Radio. As always, thanks for listening.

• 47:49Stephen J. Dubner?

  Freakonomics Radio is sponsored by Granola. You know that moment when a meeting ends and someone says, let's circle back on this next week and everyone nods, but no one writes it down? Then next week rolls around and it's radio silence? That's why Granola exists. Granola is an AI powered notepad built for the way real people actually meet. You take rough notes like you normally would, and in the background, Granola securely transcribes the meeting. Then it turns everything into clean, structured, actually useful notes when the meeting ends. And the best part? Granola works through your device's audio, which means it integrates seamlessly into the video conferencing tools you already use. No setup, no awkward bots. It's just your normal meeting with superpowers. If meetings are eating up your day, Granola is a no brainer. You can try it totally free.

• 48:33Stephen J. Dubner?

  Just head to granola.ai/freakonomics. That's granola.ai/freakonomics to get your time back. Try it for free at granola.ai/freakonomics.

• 48:51Stephen J. Dubner?

  Freakonomics Radio is sponsored by Advantech. No one likes a creaky floor and builders know that the subfloor sets the tone, which is why Advantech subflooring is engineered for strength, stiffness, and moisture resistance. Advantech products are designed to stand up to the elements and give you a bond so strong it's backed by the industry's first squeak free guarantee. No squeaks, fewer callbacks, no problem. When the schedule is tight and performance matters, head to huberwood.com/advantech to lay the foundation of a solid build.

• 49:28Stephen J. Dubner?

  Freakonomics Radio is sponsored by PNC Bank. Boring, great word, bad rep. Yet time and time again, here at Freakonomics, we find that when something culture changing happens, boring things like calculations and careful planning are usually behind it. Which begs the question, why not apply it to other aspects of life? Like banking, for example? Well, PNC Bank did since its inception, proving time and again, that the best banking strategy isn't the exciting type, but the steady, reliable, definitely not YOLO type. PNC Bank, brilliantly boring since 1865. Brilliantly boring since 1865 is a registered mark of the PNC Financial Services Group, Inc. PNC Bank, National Association, Member FDIC.

• 50:22Stephen J. Dubner?

  When a seemingly intractable problem meets a potentially disruptive technology, Silicon Valley usually isn't very far behind. In this case, Silicon Valley by way of Ireland.

• 50:33Speaker 01

  I'm Stephen Smith. I'm the CEO and founder at Kitman Labs.

• 50:37Stephen J. Dubner?

  Smith was a strength and conditioning coach for an Irish pro rugby team before he moved into data science. His firm now works with all kinds of sports organizations, including teams, leagues and governing bodies across more than two dozen countries. So what is Kitman exactly?

• 50:55Speaker 01

  Kitman Labs is an athlete intelligence platform. We help sports organizations capture, manage and derive insight from data across the full athlete lifecycle. Everything from recruitment and development through to performance optimization, welfare, and even the operations of major leagues themselves.

• 51:13Stephen J. Dubner?

  And why the name Kitman?

• 51:15Speaker 01

  It doesn't really translate to the US very well, but in Ireland, the kit man or equipment manager almost is always the lifeblood of the organization. They do the hundreds and thousands of tiny things that nobody sees. But without them, the entire organization falls apart. We had a kit man called Johnny O'Hagan, this grouchy old guy who who'd just been there for about 30, 40 years. If a clip popped off, of course, he had the exact one ready for them. The jerseys were set up, the water was set up, the fruit was set up, the whistles were there. He got sick at one point in time, ended up in hospital, and the entire organization nearly fell apart. He just did the thousands of unseen jobs that nobody really appreciated.

• 51:57Speaker 01

  And I loved that concept and for the company and what I wanted us to do. I wanted us to be the lifeblood and power behind great decisions in an organization, but not the person that's picking up the trophy, taking the accolades, doing the interviews. I wanted us to be behind the scenes, you know, the hidden gel within the organizations.

• 52:16Stephen J. Dubner?

  The reason we are talking to Smith is because we heard about a model that Kitman has built, a model that uses, here, I'll read you a little bit. A model using AI trained on more than 15 years of NCAA performance and NFL Combine data to identify the statistical profiles most associated with NFL success. So does this mean that Kitman can predict which college quarterbacks will become great pro quarterbacks?

• 52:43Speaker 01

  So the role of the model is not necessarily to be predictive. We don't have a crystal ball. The problems that we're solving are really complex. The role is to reduce uncertainty. So the model draws on a very broad set of inputs. Physical and physiological data, trading history, match performance metrics, biographical factors, and increasingly psychological behavioral indicators. And then we're looking at, of all of those characteristics, which are most indicative of long-term success? And then we're going back and sweeping the data, looking for the next factor in line, and the next, and the next, and the next. And we're building that breadcrumb trail behind it. And then what we're doing is trying to see, okay, who fits that profile?

• 53:17Stephen J. Dubner?

  The Kitman model found that the top quarterback in this year's draft class with an AI success prediction of 51% was, not surprisingly, the number one overall pick.

• 53:29Speaker 01

  The strongest quarterback signal was Fernando Mendoza. Undefeated season produced exactly the kind of statistical fingerprint that the model rewards most highly.

• 53:39Stephen J. Dubner?

  In the old days, statistical fingerprint was mostly about physical skills, size, speed, arm strength, et cetera. And now?

• 53:48Speaker 01

  What the data increasingly points to is the importance of processing speed, how quickly a quarterback reads a defense, how they go through the progressions and makes decisions on their pressure. There's also growing interest in resilience metrics, how a player responds to adversity, but within a game across the season. And frankly, coachability, right? The rate at which a player absorbs and applies new information, how quickly they're developing. They are some of the most predictive long-term variables that we see today.

• 54:13Stephen J. Dubner?

  Do you have a relationship at all with the NFL or with NFL clubs?

• 54:16Speaker 01

  The relationships that we have with some of our larger leagues and sporting bodies are highly confidential. I can't actually speak directly about them.

• 54:25Stephen J. Dubner?

  I'm going to take that as a probably.

• 54:28Speaker 01

  Yes, we operate in professional football.

• 54:31Stephen J. Dubner?

  You also operate in the other professional football, what we call soccer, in the English Premier League, MLS in the US. Tell me about that.

• 54:40Speaker 01

  The core and backbone of what they're trying to do is to produce better players. So in the Premier League, they have a thing called the E-triple-P, which is the Elite Player Performance Plan. And essentially, that is a set of rules and regulations for how you develop talent. So from the age of U9 all the way through the first teams, they have a set of standards and regulations for everything from tactical development, technical development, physical development, and even like medical care, resilience, robustness, and even some aspects of education. Our platform is designed to help them to collect all of the relevant data in each of those different strands and/or operationalize the workflows for specific users. So for example, if I'm a coach within an academy setting in somewhere like Chelsea Football Club,

• 55:24Speaker 01

  I'm the U9 coach, I come in and I build out my session plan. What athletes are involved in each one of those drills? How long are we going to spend on that? And then what's the physical load or toll that that's going to take on our players? Then the fitness staff members are coming in and they're pulling in all of the data from, let's say, wearable technology. If they're using performance tracking data to objectify some of the tactical and technical pieces of information that's coming in. If there was an injury in that practice session, the medics would come in and log the injury within the system. And then the next day, as the athletes are recovering, they may be collecting recovery indexes from them or things like, how's their neural system, how have they slept? How are they moving? So what we're trying to do is basically build a 360-degree view of, what are we doing to the athletes?

• 56:09Speaker 01

  What are the load and stimulus we're placing on them? And then how are they responding?

• 56:12Stephen J. Dubner?

  So U9, we should say, is, this is a concept that Americans will largely be unfamiliar with. That's under nine years old, correct?

• 56:19Speaker 01

  Absolutely.

• 56:20Stephen J. Dubner?

  So these are kids. What you're describing now is capturing data of the athletes that are already in your building, essentially. Talk about data that you use to analyze for recruitment and predicting. Let's say you're an NFL team that's looking at potential draftees from college.

• 56:37Speaker 01

  Typical models for scouting or recruitment. If we use the NFL as an example, they're looking at your college stats from that year. Who is the best quarterback? Who are the best running backs? Who are the best receivers, et cetera? And they look at key attributes that are important for each of those positions. What we're trying to do is slightly different in that the time horizon we're looking at is much longer. So regards to like the E-triple-P, what we're trying to do is look at, what do players that make it and play once in the Premier League look like? What do players that play 100 times? What do players that play 10 seasons? And when you invest in these types of league-wide initiatives that span this type of time horizon, you can really start to basically reverse engineer. What did that person look like? In today's landscape, the Premier League are the highest producing league

• 57:19Speaker 01

  from a professional talent basis than any other league in the world. And they have the benefit of now being able to look back on players like Bukayo Saka or Cole Palmer, and they can basically follow them back to when they're U9. And when we look at the NFL today, we don't have that luxury.

• 57:36Stephen J. Dubner?

  That's because you don't have these future NFL players from the time they were eight or nine and all the data on them. Is that why?

• 57:42Speaker 01

  Yeah, number one, I think we need a connected pathway. Since I've moved to America, I've realized, well, there is somewhat of a disconnect between the professional game, the collegiate game, and the youth game. Chelsea's academy are recruiting at the age of eight. Major League Soccer have now made a huge investment in both Next Pro, their semi-professional league, and then Next, which is their elite development league. And they're getting all the way down to U13 right now.

• 58:05Stephen J. Dubner?

  That's a tradition in soccer that we don't have in American football. Let's assume that that tradition doesn't start in American football. In other words, let's assume that the Pittsburgh Steelers and Green Bay Packers and Dallas Cowboys aren't starting youth academies where children are starting from the age of seven, eight, nine. What can you do to establish a firmer pathway like the one you're describing for the NFL?

• 58:28Speaker 01

  Well, you look at a partnership model. I know that that's more complicated in the world of the NFL because high school football has its own situation, collegiate football is a powerhouse. But I think there's an opportunity to create some level of partnerships. And ultimately, this will benefit players and it will benefit the game and it will benefit fans. I mean, listen, when you look at predictive modeling or you're thinking about scouting recruitment, you're thinking about performance optimization. We've seen these superstar athletes that come through. But then going from being a superstar in college to being a superstar in the NFL, it can crush people if they don't have the mental resilience and the tools to be able to deal with that. We've often seen players that have come through who weren't a first round, weren't a second round, sometimes weren't even a third round draft pick,

• 59:13Speaker 01

  become like the player. Like, look at Brock Purdy. People would laugh at you when it was a draft year if they had have suggested that you would put Brock Purdy as a first round draft pick. But like, form has shown.

• 59:23Stephen J. Dubner?

  So in your view, if you could gather as much data as is possible on Brock Purdy's life and career, physical, strategic, behavioral, cognitive, et cetera, and go back as far as you could, what do you suspect that you'd find in the data that was essentially overlooked by 31 teams?

• 59:41Speaker 01

  I think you'd see a lot more consistency than you sometimes see in other players. If we have the data and the inputs about what makes success, what drives success, how we create success, we can replicate that, we can scale that.

• 59:55Stephen J. Dubner?

  I see that Kitman Labs also does some work with the US military, more than $4 million in defense contracts over the past several years. Tell me about that.

• 1:00:04Speaker 01

  It was kind of an area that we stumbled into, quite honestly. We've been very heads down in the world of sport. But they reached out to us and asked us, will you come in and look at a particular body of work for them, they're trying to recruit, just like we are in sports. We think about that as a really simple thing in the US. Well, you go, you sign up and you're recruited. No, that's not really exactly what's happening. What it takes to be a fighter pilot is completely different to what it takes to be a power rescue jumper, is very different to being a Marine or very different to being a SEAL. So how we recruit for those and the types of characteristics that are needed is the same thing as, well, the characteristics to be a quarterback versus a running back versus a receiver versus a lineman, all of those are different. Once they have them in, they care about development. So how do we give them the right characteristics? How do we develop all of these things to put them in the right position? Just like we do in sport.

• 1:00:48Speaker 01

  And then when we have them at the other end of a training pipeline and they're ready for work, how do we keep them healthy? How do we keep them performing well? They're the same underlying principles. So whether you're playing football, whether you're playing rugby, whether you're playing tennis, whether you're, you know, running around in a training environment to be an airmen, we care about measuring the load. We care about understanding what you're doing. We care about understanding how you're responding to that. And that's about people, not about sport.

• 1:01:14Stephen J. Dubner?

  I learned a lot today from Stephen Smith, Scott Goldman, Dan Hatman, and Kurt Warner, and I'm guessing you did too. But still, there are some big questions, like will NFL teams finally learn to identify which college quarterbacks will become good pro quarterbacks? It's one thing to over invest in a college superstar who turns out to be a bust in the NFL, but what about the other scenario, the Kurt Warner scenario, where a quarterback who turns out to be world class was passed over by every team in the league? As it turns out, history may be repeating itself within the Warner family. Kurt's son, EJ, is also

• 1:01:56Stephen J. Dubner?

  a quarterback. He just finished his college career, and he too went undrafted. Here's Kurt.

• 1:02:02Speaker 02

  EJ, he played at a little smaller school. He didn't play at an SEC school or a Big Ten school. Fresno State was where he finished up. He has done it really well for a long time, but there are some things that he doesn't stack up with what's prototypical.

• 1:02:20Stephen J. Dubner?

  You said as a player, you could understand that and process it. What about as the father of a player, though? It's got to be hard.

• 1:02:28Speaker 02

  It is hard. And I think the thing that's hard for me with EJ is because I know what he does really well. And I've watched it. And he was in some situations in college very much like me. I've watched him play in so many different scenarios. And I understand that he's not the most physically gifted player to play. But these mental things that I'm talking about, I think he's very rare at those things.

• 1:02:54Stephen J. Dubner?

  Do you think he's going to get a chance?

• 1:02:56Speaker 02

  He's only 22 years old. I do believe that he's better than some of the guys that are out there. And I do believe he's going to get a chance. EJ is better than me coming out of college, and it's not even close. But I do believe he is an NFL quarterback and could have a long career in the NFL if given a shot.

• 1:03:12Stephen J. Dubner?

  EJ Warner did have tryouts with two NFL teams, the Kansas City Chiefs and the Denver Broncos, but he didn't make either roster. We will keep an eye on him and let you know if there's any news. In the meantime, my big thanks to Kurt Warner, as well as Smith, Goldman, and Hatman. Let us know what you think about this episode. Our email is [email protected]. Coming up next time on the show, prediction markets have been around for a long time, but they've suddenly become famous because they will let you place a bet on just about anything. So is that all they're good for? Kalshi is the most accurate way to predict the future.

• 1:03:55Stephen J. Dubner?

  We'll hear from the CEO of Kalshi, as well as the economist who pioneered the prediction market idea 40 years ago.

• 1:04:03Speaker 09

  When we do head to head pairwise comparisons of betting markets and some other institution at the same time, same topic, similar resources, the markets just do about as well or substantially better in terms of accuracy and similar cost. That's just a remarkable fact.

• 1:04:19Stephen J. Dubner?

  And we'll hear from a regulator about prediction market risks. What I think is the cost, and a very real, serious cost, is trust in the markets. That's coming up next time in the first of a two-part series on prediction markets. Until then, take care of yourself. And if you can, someone else, too. Freakonomics Radio is produced by Renbud Radio. You can find our entire archive on any podcast app. It's also at freakonomics.com, where we publish transcripts and show notes. We will be sure to post my AIQ scores there if you want to take a look. And a special thanks to listener Peter Wynn, who suggested this episode on the quarterback prediction problem. This

• 1:05:01Stephen J. Dubner?

  episode was produced by Pete Madden and edited by Ellen Frankman. It was mixed by Jake Loomis with help from Jeremy Johnston. The Freakonomics Radio Network staff also includes Dalvin Abouaji, Eleanor Osborne, Elsa Hernandez, Gabriel Roth, Alaria Montenacort, and Teo Jacobs. Our theme song is "Mr. Fortune" by The Hitchhikers, and our composer is Luis Guerra. Thanks again for listening. Should I leave my phone on in case the Vikings or Jaguars or Bears call?

• 1:05:31Speaker 06

  I think Lloyd from Dumb and Dumber said it best.

• 1:05:35Stephen J. Dubner?

  You're saying there's a chance.

• 1:05:37Speaker 06

  I'm saying there's a chance. That's right.

• 1:05:39Speaker 04

  The Freakonomics Radio Network, the hidden side of everything.

• 1:05:51Stephen J. Dubner?

  Freakonomics Radio is sponsored by Ozempic. Innovation happens through rethinking what's possible. And when it comes to GLP-1s, Ozempic pill does just that. Learn more about Ozempic semaglutide tablets, 4 and 9 milligrams, by calling 1-833-OZEMPIC or visit ozempic.com to view the medication guide and ask your doctor what's possible with FDA-approved Ozempic pill. Freakonomics Radio is sponsored by Southern Company. The world and its energy needs are always changing. Southern Company's commitment to meeting this demand stays the same. That's because Southern Company believes energy is more than a utility, it's what

• 1:06:34Stephen J. Dubner?

  powers possibilities. So they are looking ahead and investing $80 billion in infrastructure upgrades and are committed to fueling growth in ways that benefit all customers so that reliable and affordable energy is accessible for generations to come. Go to southerncompany.com to learn more. Southern Company, building the future of energy.

• 1:06:55Speaker 10

  With the Discover Cash Back Card, it's payback time when you earn cash back on everyday purchases. Activate and earn 5% cash back at different categories each quarter on up to $1,500 in purchases. That's 5% cash back at different places each quarter, like grocery stores, on gas, and at restaurants. It pays to discover. Terms apply. See discover.com/five for details.

• 1:07:26Speaker 11

  This episode is brought to you by Charles Schwab. Is there a right time to sell a stock? Are you taking the right risks with your portfolio? Financial decisions can be tricky, and often, your own cognitive and emotional biases can lead you astray. Financial Decoder, an original podcast from Charles Schwab, can help. Join host Mark Reapy as he offers practical solutions to help overcome the cognitive and emotional biases that may affect your investing decisions. Listen at schwab.com/financialdecoder.