Ep 54: Princeton Researcher Arvind Narayanan on the Limitations of Agent Evals, AI’s Societal Impact & Important Lessons from History

Arvind Narayanan is one of the leading voices in AI when it comes to cutting through the hype. As a Princeton professor and co-author of AI Snake Oil, he’s one of the most thoughtful voices cautioning against both unfounded fears and overblown promises in AI. In this episode, Arvind dissects the future of AI in education, its parallels to past tech revolutions, and how our jobs are already shifting toward managing these powerful tools.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Jacob Effron?

  Arvind Nairan is one of the leading professors in computer science today. Over at Princeton, he spends a lot of time separating hype and substance in AI with his newsletter and book, AI Snake Oil. On unsupervised learning, we touched on a ton of interesting topics, including the state of agents today and where evaluations do and don't work, as well as problems in coordinating them. We talked about lessons from past technology waves like the industrial revolution and internet and the implications for policymakers. And we also hit on the future of AI in education and whether AI will increase or decrease global equality. All in all, a fascinating conversation with someone I've admired for a long time. Without further ado, here's Arvind. Thanks so much for coming on.

• 0:36Speaker 01

  Thanks for having me. It's great to be here in the studio.

• 0:38Jacob Effron?

  Yeah, no, it's always always fun to get to do these live, especially on a on a snowstorm weekend. I appreciate you you making the trek up from Princeton.

• 0:43Speaker 01

  Yeah, my pleasure. Looking forward to the chat.

• 0:46Jacob Effron?

  Awesome. Well, I figure there's a bunch of different places we could start, but obviously these reasoning models, you know, I think one of the big questions being asked today is, you know, obviously, they've shown impressive results in coding and math and some of this easily verified data. You know, I feel like you've said progress will be kind of unevenly distributed across tasks. And I feel like everyone's trying to figure out what that uneven distribution is. So anything you can share with our listeners on kind of the tasks you think fall really neatly into things these models will be good at, whereas tasks that they they may struggle with.

• 1:10Speaker 01

  Sure. When we look at the impressive results that we're seeing so far, they're in these domains with clear, correct answers, right? Math, coding, certain scientific tasks, and that will certainly continue, I think. How far away from that this impressive performance is able to generalize, I think, is a really big, open question. So historically, you know, 10 years ago, there was so much excitement around reinforcement learning, for instance, when it started doing really well in games like Atari. And if I understand correctly, the, you know, OpenAI and many other AI companies and people who were thinking about AGI at the time,

• 1:52Speaker 01

  that was because of reinforcement learning. And the breakthrough is 10 years ago. But what happened then is that we saw a failure to generalize too far outside these narrow domains like games. That is one possible future of these reasoning models. Another possible future is that by being better at reasoning, by writing code, you know, you could imagine reasoning being extended to

• 2:19Speaker 01

  A system that also reasons how to get information from the internet, you know, to then reason about law or medicine or any other domain. So those are two possible futures. Which of those is more likely? I'm not sure.

• 2:31Jacob Effron?

  Yeah. What are you paying attention to? I mean, obviously, I assume these reasoning models will continue to push the Sweet Bench scores up and up. What would you, what would you need to see to be like, oh, wow? It really does feel like it's starting to generalize across some of these tasks that I wouldn't have previously thought it could do.

• 2:43Speaker 01

  For sure. Yeah. Construct validity in a phrase. So construct validity is this idea that what we might be trying to measure, especially in a benchmark, is kind of subtly different from what we want in the real world. And that's a challenge for any benchmark creator. Sweet Bench, we think, is a pretty good benchmark.

• 3:00Jacob Effron?

  Built at Princeton, right?

• 3:01Speaker 01

  Yes, by some of my Princeton colleagues. It's a pretty good benchmark in terms of constructability. Instead of these toy, you know, Olympiad-style coding problems, it's real GitHub issues. But nonetheless, GitHub issues, I think, is a far cry from the messy context of real-world software engineering. And so that's what I'm watching. Not so much the results on, results on benchmarks alone, but people's experiences, trying to use these to improve their productivity. Right? And so when we look at it that way, I mean, it's clear that, you know, thousands of people are using this, these models productively, I'm among them. But it's clear that dramatic improvements in Sweet Bench don't translate to dramatic improvements in human productivity.

• 3:42Jacob Effron?

  Yeah, no, I've heard you talk about this before, too, when, you know, all these impressive results of how OpenAI models do on the bar or on, you know, medical exams. You're like, well, it turns out being a lawyer or a doctor is not just constantly taking those exams.

• 3:52Speaker 01

  Exactly, exactly.

• 3:54Jacob Effron?

  And so I imagine we'll need a set of domain-specific, you know, actual real-world evaluations. And also just people using it, vibes, as people like to call it, too.

• 4:00Speaker 01

  Yeah, vibes. I think there's, there's, you know, I think there's a middle ground between vibes and benchmarks. And I think we can do that. So for instance, people have uplift studies. These are actual randomized controlled trials where this group of people has access to the tool and this group of people doesn't. And then you measure the impacts on productivity. And also, it's a question of what kinds of tasks are you asking people to or the models to do? Again, going back to the theme of it's not all bar exam questions. There was a recent paper that looked at whether LLMs are good at kind of natural patient interaction in a medical setting and eliciting information from the patient. That's the kind of thing which, you know, for a human, for any person, let alone a doctor, is relatively easy to do, you know, natural conversations.

• 4:45Speaker 01

  So we don't think of that as the thing we want to benchmark. And so we focus on diagnosis tasks. And yet it turns out that even when models have great performance on those, they might struggle with some of these other things that are also required for, for really being useful in the real world.

• 4:59Jacob Effron?

  Totally. You can't do a good diagnosis without the actual information, getting the information from the patient.

• 5:01Speaker 01

  Exactly.

• 5:02Jacob Effron?

  That's super interesting. I think on the, on these reasoning models, you recently published a paper called inference scaling flaws, I think, about, you know, some of the ways, I think, which these, these, this approaching kind of go off the rails.

• 5:13Speaker 01

  Yeah.

• 5:13Jacob Effron?

  Can you talk a little bit about that research and maybe some of the implications for these test time compute models?

• 5:17Speaker 01

  Yeah. So the big question with inference scaling is we've had, you know, when we look at model scaling, we've had, I don't know, like six orders of magnitude of making these models bigger, training them with more compute, right? So how many orders of magnitude are we going to have with reasoning models and through what methods? So the answer to that is still unclear. But in this paper, we were looking narrowly at one particular way to try to get these models to scale across several orders of magnitude of inference compute. And that is by pairing a generative model with a verifier.

• 5:51Jacob Effron?

  Yep.

• 5:51Speaker 01

  So in the context of coding, a verifier would be unit tests. In, you know, in, in some other domain, in math, for instance, a verifier might be an automated theorem checker, right? So the model is trying to generate possible proofs of theorems and then the proof checker is trying to verify whether it's correct or not. So the hope here is that the proof checker or the unit test or whatever, those are just traditional logic. They're not stochastic.

• 6:18Jacob Effron?

  Yeah.

• 6:18Speaker 01

  And so maybe those can be perfect. And so the model can generate millions of solutions until one passes the tests. But in real life, it's not quite that simple. Unit tests may have imperfect coverage. So we wanted to see what are the implications for inference scaling when the verifier is imperfect? And the paper basically has bad news for this, you know, relatively narrow, but nonetheless important setting. If the verifier is imperfect, we show that inference scaling can't get you very far. Sometimes you saturate within like 10 invocations of the model, for instance, as opposed to a million, which you might have hoped. So to what extent that applies to O1 or O3? I'm not making any claims about that, but it is in the back of our minds when we're evaluating these new reasoning models.

• 7:03Speaker 01

  Are they actually going to be able to scale across so many orders of magnitude?

• 7:07Jacob Effron?

  Totally, and it seems highly relevant across, you know, scaling these models into domains that don't have easy verifiers. And you could imagine someone trying to staff up a full team of doctors or lawyers or accountants to check these things. And I think the point of your research seems to be, well, if it's not, if you're a little, if you're slightly imperfect, it actually throws the whole thing off.

• 7:22Speaker 01

  Exactly.

• 7:22Jacob Effron?

  You've written about, obviously about the hype around AI and some of the stuff that works and doesn't work. How would you kind of classify today? Like what has product market fit in AI like actually works? And where do you kind of see the snake oil?

• 7:32Speaker 01

  Let's take agentic AI, right? So just the way we name it, agentic AI to me is not one single category. Let's look at two types of agentic AI. One is a tool like Google deep research. I mentioned that one because I've been using it a lot, but yeah, yeah, there are many others where sure, there is an agentic workflow. But at the end of the day, it's still a generative system. The point of it is to generate a report or whatever it is for you, right? The user who's going to be presumably an expert in whatever field you're using it in. And it's for you to look at the report and decide what to do with it. Of course, there are going to be flaws, hopefully as the user you're aware, but still it can be a great time-saving tool. It can be a great first draft of a report and so forth.

• 8:16Speaker 01

  So I think this is pretty well motivated. On the other hand, another kind of thing that's often called agentic AI is something that autonomously takes actions on your behalf. Right? Like booking flight tickets, that's kind of the

• 8:28Jacob Effron?

  Everyone loves booking flight

• 8:29Speaker 01

  I know, that's just kind of become, you know, this thing that somehow It's a pain point for all of us, I think. And so that's become the thing. You know, if you're, if you have an elevator pitch around agents, it makes sense. But I think when you take a closer look at it, booking flights is almost the worst case example for an AI product, I think, to have product market fit. And the reason for that is that, the reason booking flights is so hard is because getting the, getting the existing system, whether it's going to be an agent in the future or just, you know, Expedia or whatever today, to understand all our preferences is very, very challenging. You do a search and you realize, oh, this is full of United flights.

• 9:14Speaker 01

  I don't want that. Or, you know, I, I only want nonstop flights. And it's only when you look at some of the results that you even realize what some of your constraints and preferences are. So it takes like 10 or 15 rounds of iteration, right? And when you add all that up, it's going to take half an hour to really find the flights that you want. Well, the problem is if you have an agent acting on your behalf, it might not know those preferences either, right? Unless you've been using it for a very long period of time and you've somehow gotten to the point where you trusted to have learned all of your preferences and then act on your behalf. And so a flight booking agent is going to have to ask you a whole bunch of questions. And then by the end of playing those 20 questions game, you're just as frustrated with the agent as you are with Expedia today. So that's, that's kind of my prediction. You know, maybe I'm wrong.

• 9:59Speaker 01

  And the other thing is the cost of errors is high. If it makes a mistake in the flights that it books for you, you know, even, even a an error rate of, you know, one in, in attempts is completely intolerable. And those are the kinds of failures that people have been reporting with these early agentic systems, like this product ordered DoorDash the wrong address. And I'm never going to use it again if that happens, right? So those are some key differences. Producing generative outputs for you to look at versus automating things on your behalf. Low cost of errors, like in a report versus high cost of errors, like in ordering something for you.

• 10:34Jacob Effron?

  Yeah, I mean, it strikes me from what you're saying, it's almost similar to what you were talking about with the evals around medicine earlier and the idea that like actually eliciting the user preferences sometimes is like

• 10:43Speaker 01

  Exactly, exactly.

• 10:44Jacob Effron?

  And just the idea that you could do that without, you know, without having that or you're kind of doomed to not be able to do it.

• 10:49Speaker 01

  Exactly. I think there should be, in addition to solving the purely technical problems in agentic AI, there should be more of a focus on the human computer interaction component.

• 10:57Jacob Effron?

  Have you seen anything interesting there yet? Obviously, we're in the early days of agents, but like anything that's caught your eye of like, oh, that's actually a pretty interesting way to solve for some of the shortcomings in agents today.

• 11:05Speaker 01

  Yeah, I mean, to me, the thing that really gives me optimism on agents is that what we think of as chatbots early on were just simple wrappers around LLMs, but now they're agentic. Right. They do searches for you. They run code on your behalf. And so the complexity is gradually evolving there. So on the one hand, we might be looking out for a killer app. But on the other hand, we might not realize, but I think is equally important that some of the apps we're already using are gradually becoming more and more agentic.

• 11:37Jacob Effron?

  Totally. Seems like if people could agree on a definition of what an agent actually is, would help us figure out if we have one.

• 11:42Speaker 01

  That's right.

• 11:42Jacob Effron?

  But like in all things in AI, it seems like we like to move the goalposts as we, as we actually get nearer.

• 11:46Speaker 01

  Fair enough.

• 11:48Jacob Effron?

  I guess you, you, you spend a lot of time obviously thinking about the future and where these models are going. I'm sure there's all sorts of, of questions that you're excitedly, you know, looking forward to seeing how they play out. As you think about the next two, three years, like, are there a few things that you're like, I wish I could just fast forward three years and see X or Y? Like, what are the key things you think will answer?

• 12:05Speaker 01

  Yeah. So here's, here's one. So we've been talking about software a lot, but on the hardware side, right? I would love to know what ends up being the right kind of form factor for AI for most kind of everyday users. So I do think in the future, it's quite possible that, you know, in, in everyday conversations as well as in, in the workplace, AI is constantly watching what we're doing and offering improvements or somehow integrated into virtually all of our workflows. But exactly how? I don't know that. Yeah, that could happen in many ways, right? So this idea of going to A special purpose app, you know, like ChatGPT or Claude or whatever, putting in what you want, getting the answer and then getting back to your other software that you're using.

• 12:49Speaker 01

  That's almost, you know, the, the, uh, the highest friction way of using it.

• 12:54Jacob Effron?

  And that's not been the end state of most software.

• 12:55Speaker 01

  Exactly. Or you could imagine models getting integrated into, you know, if you're using Photoshop, it already has a bunch of AI features, generative fill, various other things. Or you could imagine, uh, that you just have an agent just looking at all the, uh, you know, the, the screenshots on your computer or, or phone every five seconds and then integrate it automatically somehow into every app. We don't have the APIs for that today, but you can imagine that becoming possible in the future. Uh, at an even more, even higher level of abstraction, I would say perhaps even less friction is if it's seeing everything you're seeing, not even on your device. So that would have to be in your classes. Yeah. So I'm curious if, for instance, the, uh, you know, the, the Meta Ray-Ban is one product,

• 13:38Speaker 01

  but there are many such products where AI is integrated into your glasses. I'm very curious to know if that's going to be one of the main ways we're going to use AI in the future. Uh, I'm kind of hoping that it is because in, just in my own personal life, there are so many little ways in which I can imagine using it. If I could have those glasses, the first, um, or, you know, if, if I could wear those glasses all the time, I would say right now, I think the battery life is only like two hours. That's one of the main constraints. But one of the first apps I would write for it, if it doesn't already exist, is to look at, uh, screenshots basically every five seconds and remember where everything is in my house so that when I lose my keys, it will tell me.

• 14:19Jacob Effron?

  That'd be pretty cool.

• 14:20Speaker 01

  That's just one example, right? But so many things when you're in a country where you don't speak the language, having AR so that it can automatically translate it for you, um, just so many other examples. And so, uh, but the, but the key to really full-throatedly working on a lot of these applications is knowing which form factor is going to win out in the future.

• 14:39Jacob Effron?

  Some people talk about, you know, uh, yeah, if you believe in the scaling laws continuing, you know, OpenAI, Anthropic raising, you know, 50, a hundred million dollars in the, in the next year or two. Like, does that seem feasible to you or?

• 14:50Speaker 01

  So, I mean, I won't speak to the. The investing side of that, but from a technical perspective, what we've seen going on over the last couple of years is this push and pull between two forces. One is the rapid decrease in the per token costs of inference. And the other is what we're now calling inference time computes, but I think it's kind of been there since the beginning of the consumer success of chatbots. And I think it really depends on how that's going to play out.

• 15:19Speaker 01

  I think it's really hard to predict, but I do think that what's most likely is that the token usage is going to continue to increase in a way that more than compensates for the decrease in per token costs. So for instance, my team is building what we're calling an AI agent zoo, where we're putting different kinds of AI agents into an environment where we're giving them a task to work on in collaboration. And I think this is a different way of evaluating agents.

• 15:47Jacob Effron?

  Yeah.

• 15:48Speaker 01

  As opposed to benchmarks where you somehow it's a competition and each one is working in isolation, but I think agents are kind of more naturally collaborative. And so one of the tasks we tested them on is ask them to write a joke. First of all, the jokes were awful. I should be clear.

• 16:03Jacob Effron?

  The comedians are still still safe.

• 16:05Speaker 01

  Yeah, that's right. This is not a revolution so far in in humor. But that was not the point. We wanted to see kind of how they collaborate. And when you look at, you know, what it actually takes for these agents to to do that, they have to first start by understanding their environment. Right? Looking at the directory structure, seeing what tools they have, looking at what the other agents are doing, that so on and so forth. And we give them a lot of collaboration tools. We integrated them into Slack. And then we gave them blogging tools so they can kind of write these blog posts summarizing what they've learned about the task and then other agents can pick up from there. So we try to create a, you know, relatively realistic collaboration environment. And one of the things that we found is that even for the simplest tasks, they kind of generate millions of tokens.

• 16:50Speaker 01

  And it's not wasteful. They're kind of making progress, not just generation input and output together as millions of tokens. Because again, you know, looking around in the environment you're in, understanding the environment, understanding your collaborators and then producing something takes a lot of tokens. Right? So, I mean, in one sense, yeah, it's terrible from an environmental perspective that it takes 1 million tokens to write an awful joke. But also, I think there are domains in which we're going to say, you know what? That's that's better than the alternatives. And so my prediction is that the overall inference costs are going to keep increasing.

• 17:20Jacob Effron?

  It seems like the the human equivalent of like, you know, it takes six months to understand what what in the world to do in a new job anyway. And so it's like, it takes them a million tokens to to get the lay of the land. I'm really curious about that work, you know, a few different angles to take it. Maybe we'll start with just like the eval side of agents. Like, obviously, if we are still trying to figure out evals for core, you know, chatbot responses, it feels like we're even earlier days on the agentic side.

• 17:40Speaker 01

  Right.

• 17:41Jacob Effron?

  You know, maybe categorize for our listeners, like, what's the current state of evals for agents? And then, you know, where do you think we should kind of continue that work?

• 17:48Speaker 01

  Yeah, I think the current state for evals for agents is a lot like chatbots, right? It's it's these kind of static benchmarks. Obviously, you know, Sweetbench is one of the better known examples. You try to give them relatively realistic tasks, whether it's fixing software engineering issues or navigating the web in a simulated web environment and and finding some piece of information or accomplishing some task. But it's not it's not working that well. Here are some of the limitations. One is what we call the capability reliability gap. So for agents, especially those that take actions on your behalf, it's really important to know what a 90% score means. Does this mean that it's it's good at nine out of 10 tasks that are in this benchmark?

• 18:33Speaker 01

  And the ones that it's good at, it's always going to accomplish correctly, right? Or is it going to fail 10% of the time at any task and perform some really costly action? Again, like, you know, booking the wrong flight ticket. And if benchmarks aren't measuring that, and today they're not, they give you very little information. You know, they give you information about is the tech progressing? They're not giving you information about can you take this agent and actually do something useful with it? So that's one big limitation. Another big limitation is safety. There's a lot of safety specific benchmarks, but I think safety should be a component of every benchmark because it's not like when you're, you know, not solving a safety specific benchmark, you can forget about safety. That's not the case.

• 19:18Speaker 01

  So we were looking at a web benchmark the other day. And it actually involves doing things on real websites, which is terrible. Right now, nothing's going wrong because none of the agents work. They're not able to do it. But I don't understand what the benchmark developers have in mind. Because for agents to actually be able to do well on this benchmark, they have to take stateful actions on real websites. And those website operators are going to be pissed from all the spam that's being generated by the agents trying to solve this benchmark. Right? On the other hand, you have some web benchmarks that are simulated environments, but then lose a lot of the nuance of real websites. And there's nothing in the middle. And similarly, when you look at Asian frameworks, we've been using AutoGPT.

• 20:03Speaker 01

  Sometimes it can go off and do stuff online that you didn't intend because it thinks that's the best way to do stuff. One time it tried to post a question on Stack Overflow to get an answer. These are all things, obviously, we didn't want it to do. And we were terrified that it's going to do something really dangerous that we didn't intend. Fortunately, it didn't happen. But even posting on Stack Overflow, you know, we wanted to have a way to prevent that. But right now, the only way to do that is for it to escalate every single action to the human user and for you to sit there and babysit. Right? And so even these really basic aspects of safety control have not been integrated into the way we evaluate agents.

• 20:48Speaker 01

  So those are just a couple of limitations.

• 20:50Jacob Effron?

  What do you think the middle ground ends up being between like a simulated environment? To your point, it just doesn't have the nuance of the real world and like actually letting these things loose on the real world?

• 20:58Speaker 01

  Yeah, I mean, I think we're just going to have to rethink the role that benchmarks are going to play in evaluation. Benchmarks are useful for certain things. Again, you know, the capability reliability gap issue, that I think you can solve at the level of benchmarks.

• 21:12Jacob Effron?

  Yeah, I feel like Sierra had a, they put out a benchmark that was like, you know, the same task done eight times or some number of times in a row. What's the percent of that? Which to your point is much more interesting.

• 21:21Speaker 01

  Exactly, exactly. Yeah, so the pass at K versus pass. I don't even know what is, how do you say it verbally? But pass to the K is how you write it.

• 21:28Jacob Effron?

  You don't know how to say it as Professor Iceman, but.

• 21:32Speaker 01

  Yeah. So that's a problem I think you can solve, but the other problem you mentioned, the realism problem, I don't think you can get too far with that. So I think the answer is to use being good at a benchmark as a necessary but not sufficient condition. So you take all the agents that are good at a particular benchmark and then you actually use them with a human in the loop in sort of semi-realistic environments.

• 21:55Jacob Effron?

  Right. And to your point, the artist finding a way to keep the human in the loop without just having them babysit every single step.

• 21:59Speaker 01

  Exactly.

• 21:59Jacob Effron?

  I guess managers have to figure out what their junior employees all the time anyway. So not a net new problem for society. Part of that work you're doing, I think, was really interesting is obviously you built a team of agents to work on jokes, I guess, in this context. And it feels like one thing we're seeing that's interesting on the startup side is you've got lots of folks that are building different agents for an enterprise. So someone's like, I'll build an agent for your finance team. And someone's like, I'll build an agent for support team and your sales team. And as you kind of think about where this goes going forward, obviously, these agents eventually are going to work together in some way. And, you know, I'm curious what you've learned now having like building teams of agents and the implications that has for, you know, is it better if one person kind of is standing on top of this entire team of agents and able to kind of coordinate or build tooling across them? Or like, what happens when, you know, there's seven, eight different companies building agents that, you know, have no contest of each other?

• 22:44Speaker 01

  Yeah. I mean, allow me to digress for 30 seconds for a historical.

• 22:49Jacob Effron?

  Yeah, please.

• 22:50Speaker 01

  That's right. So what we're seeing, I mean, if this is indeed going to be a new way of organizing the means of production, you know, which it very well might be, we can look at what happened in the past, right? So with the industrial revolution or with electricity replacing steam and factories, it took several decades to figure out how to organize labor and also the physical layout of factories and so forth to best take advantage of this new technology. So with steam, it was the idea of going from one big steam boiler to more of an assembly line setup where you can deliver electricity wherever you need it for specific, concrete tasks. So I think right now, we're in the extremely early stages of experimentation of figuring out how to have teams of humans and agents

• 23:32Speaker 01

  working together. So for me, it's not just how are agents going to work together, but it's really teams of humans and agents. Because I'm really, I find really compelling this jagged frontier idea that what models or agents are going to be good at is kind of like a calculator, right? Way better than any human at certain things, but lacking the common sense of a child in certain other areas. And I think that's going to persist for the time being. So we are going to have to figure out how to hybridize. And I think even the most basic things are not clear. So for instance, we've been confronting the problem of do you integrate the agents into existing human collaboration tools like Slack and blogging or whatever and email? Or should we be building new collaboration tools? We just don't know. So it's really hard to make any predictions.

• 24:16Jacob Effron?

  Have there been examples of times where you felt like, oh, actually, a new collaboration tool would be a more helpful way for for us to work here?

• 24:22Speaker 01

  Oh, for sure. So for a person to look at, again, these, you know, potentially a million token log of all the actions that an agent has taken and to be able to visualize that and get high level interpretable insight out of it. Fortunately, a lot of people are working on this. There's a framework called Human Layer as just one example. I think eventually we're going to get a much better handle on things. But again, super early days right now.

• 24:49Jacob Effron?

  I guess, you know, switching over to the regulatory side of things, I know you've thought a lot about the policy. I guess the most recent policy in the news is the AI diffusion rule and some of these rules around, you know, the geopolitics of some of these models. I'm curious, like, you know, your thoughts on that and some of the recent regulations around kind of both chips and models exports.

• 25:06Speaker 01

  Yeah, I wonder to what extent export controls are going to be effective. Historically, it's not quite my area, but reading people who've analyzed past export controls, they have at best a mixed record of effectiveness. And of course, export controls can be more effective on the hardware level than, you know, with with models, which are actually getting smaller by the day rather than larger. And so it's going to be harder and harder to limit their their diffusion. And also, when it comes to inference scaling, you know, it's it's not about preventing the next model from being released, but it's about how much inference scaling can you

• 25:50Speaker 01

  get out of even the existing models that are already out there? Yeah, so taking all that together, I'm a little bit skeptical about their effectiveness. But one more thing I'll say, and this is the work of Jeffrey Ding, who's a political scientist, is that Motivating all of this kind of regulation, you know, when it comes to geopolitics is there's too much of a focus on innovation and too little of a focus on, this term is called diffusion, but unfortunately it's a term collision, has nothing to do with the diffusion we were just talking about of technology getting to, well, it's kind of vaguely related, but it's more about, once you have technology available in a country, how do you adopt it? How do you reorganize your institutions, laws, perhaps

• 26:33Speaker 01

  norms and all of that stuff to best take advantage of that technology, right? So Ding considers that to really be the determinant of the extent to which a nation will be able to grow and benefit economically from the availability of a technology.

• 26:49Jacob Effron?

  Yeah. How do you think about that in the context of the US today? I mean, obviously there's big, you mentioned the industrial revolution and all these things, I know you've written about this all before. Clearly, there's massive policy implications of the progress of these models. You know, what are we like, you know, where do you think we've done a good job thinking about the future implications? Where are we not focusing enough?

• 27:05Speaker 01

  For sure. So it's hard to know how well we're doing.

• 27:11Jacob Effron?

  I guess history can judge us in a few years.

• 27:14Speaker 01

  Yeah, yeah, exactly. Yeah. But also, what is the yardstick, right? I do think we're doing pretty well compared to most other regions of the world in terms of diffusion. So one thing we can look at is the pace of adoption. You know, how many people are using generative AI? And even looking at something that you would think is easily quantifiable using data can actually be really complex. So this paper came out recently, it was titled "The Rapid Adoption of Generative AI." I have nothing to equiple about with their methods. I'm not even an expert in the methodology that they use and I'm, you know, I'm taking their numbers at face value. Again, I have no reason to think there's anything wrong with those numbers, but then the interpretation of those numbers. In the paper, they said 40% of people are using generative AI. This is really rapid compared

• 27:59Speaker 01

  to PC adoption, for instance, right, a few decades ago. But then, when you dig into the details, it's on average, people using it somewhere between half an hour per week to three, three-ish hours per week, which is not a lot. So that's called intensity of adoption. And so when you kind of control for intensity of adoption, we don't have concrete numbers for the PC, but we can make some assumptions. And my take is that, in that sense, generative AI adoption is actually a lot slower than PC adoption. So this could be for a bunch of reasons. One could be that it's just not yet that useful to a lot of people versus when PCs were first, you know, mass-produced, word processing and other things were just immediately useful to a large number, maybe.

• 28:42Speaker 01

  But also, it could be that these are things that, things where policies can make a difference. So for instance, people go into this with the assumption that, oh, the kids are, you know, experts at using generative AI. But, you know, I interact with a lot of students and they're often very confused. And they're often much more hesitant to use it because they see it primarily as a cheating tool sometimes. And I'm the one encouraging them to use it more. And here are some productive ways to use it, despite the fact that there can be hallucinations, et cetera. And it can actually be a tool to enhance your learning, that sort of thing, right? So maybe that's the kind of thing that should be a part of our curriculum now.

• 29:21Jacob Effron?

  Yeah.

• 29:22Speaker 01

  And I do think that's the case. I think it should not just at the college level, even at the K through 12 level, right? So, you know, if there were a policy intervention that made it easier for teachers to upskill themselves and then to teach those skills to students, what kind of impact could it have on people being able to use it productively and avoid the pitfalls as well, right? So I think those are the questions we should be asking, rather than, you know, trying to pat ourselves on the back for rapid adoption. It's like, what are, what's the low-hanging fruit that's on the table? And from that perspective, I do think there's a lot of low-hanging fruit.

• 29:53Jacob Effron?

  Yeah. If I remember correctly, in your classes, I guess you let people use these tools, then they disclose how they use them.

• 29:57Speaker 01

  Exactly.

• 29:58Jacob Effron?

  And so I can imagine that could be a super helpful teaching tool, too, you know, throughout different levels of education. If people are sharing how they're using them and they can get kind of feedback on, oh, you know, that was a good way to use it, or maybe that didn't make as much sense.

• 30:07Speaker 01

  I think so.

• 30:08Jacob Effron?

  Yeah. Super smart. I mean, obviously, you've previously written about a lot of the flaws in predictive AI models, and there's clearly tons of things we've learned from some of these, like criminal justice tools and healthcare tools. You know, how do you think about the ways we should apply some of those learnings to now? You know, I feel like there's another new hype cycle in AI where everyone wants to use these again for the same things that maybe didn't work as well in the past.

• 30:27Speaker 01

  Yeah. Yeah, so a couple of thoughts. One is, I think the way we should break down AI. When we're thinking about, you know, how to thinking about adopting the lessons, it should be based on the applications more than the technology. So what I mean by that is whenever I tell people, look, these criminal justice tools and the automated hiring tools are not working that well. They're always like, oh, generative AI is going to take care of the problem. But to me, that doesn't make a lot of sense. And when you look at the research on this, it doesn't make a lot of sense because the limitations are not coming from how good or bad the technology is. It's simply a consequence of the fact that you can't predict the future that well, right? That's a social science question and the social science on this is pretty strongly pointing in one direction.

• 31:11Speaker 01

  And so that should tell us that we should continue to be circumspect when we're thinking about new technologies for these long existing, but nonetheless quite flawed applications. So that's one lesson. Another lesson is related to this one regarding safety, the pace of adoption of technologies and so forth. So what we can learn from past waves of AI is that when there are these consequences, you know, whether it's in the criminal justice system or in banking, when you have automated trading where things can really go wrong.

• 31:45Speaker 01

  Eventually, people get wise to the flaws. There is a public outcry. And so those types of domains tend to be highly regulated. And so I think we should keep in mind that even though right now it's the early days of generative AI and we don't have a lot of regulation in certain domains where AI is just beginning to be employed. I think if it is making consequential decisions in the future, we should expect those domains to be regulated. And so to me, the question is less about is regulation good or bad? But instead, what should that regulation look like so that we have a good balance between protecting safety and rights as well as, you know, getting the benefits of AI? And so sometimes the discussion around regulation is too polarized, but I think there's a lot of room for it to be more collaborative.

• 32:30Jacob Effron?

  Right. And from having listened to you previously, I imagine a lot of what you would want regulation to focus on, you know, it seems like it's explainability. I think you talked about, I believe it was around, like, you know, these algorithms around bail. And there was like this whole, you know, black box. They were saying this is super advanced AI. And then it turned out it was probably literally just correlated with like, you know, was that where they like repeat offenders? And but people were using this, you know, to make all sorts of decisions. And, you know, obviously, it's one of the hard parts of generative AI is like, we have very little explainability in how a lot of these tokens are developed.

• 32:59Speaker 01

  Yeah. So in terms of applications and regulation, explainability doesn't mean necessarily, you know, mechanistic interpretability, much less trying to explain what every neuron does. That's not the point at all. Right. But it's more about what data was it trained on? What kinds of audits have you done? Do you? It's basically to try to understand, do you, are you able to make statements about the expected behavior of this model in new settings? Right. So that is like the most critical question we need to ask before we put things into deployment, or as we're putting things into deployment, we need to learn from our early experiences and tweak our approach.

• 33:34Speaker 01

  As opposed to having some kind of neat mathematical explanation, understanding of the model, if you will, which ultimately is useless if you don't know what kind of data distribution it's going to be invoked on.

• 33:46Jacob Effron?

  I guess, putting you over to the education side, you know, you're obviously an academic. And, you know, I'm curious how you think about the role of academia in AI going forward and where it makes sense for academics to spend time and where, you know, just given the compute differences, it makes sense for industry to spend time.

• 34:00Speaker 01

  Yeah. I think in the last decade or two, there has been a sense of crisis because of the, because of the sense that it's getting harder and harder to, for academics to be at the forefront of AI due to compute. It's possible that that's changing now because a lot of innovation is either, you know, thinking about new architectures, you know, blue sky, that you can establish on a small scale. Without necessarily claiming that it's, you know, competitive with GPT four or whatever.

• 34:33Speaker 01

  Or it's on the top of existing models, right? And then thinking about new infant scaling methods or whatever. So I think, yeah. So on the one hand, that's one way in which academia can potentially continue to be competitive. But another one is. Anything that goes beyond pure technical innovation, I think academia necessarily has to play a huge part, both because it requires lots of different disciplines to be thinking about what are the applications for AI and X for various X, but also what are the impacts on society and how can we make that more positive, right? So that's one role for academia and also to serve as a little bit of a counterweight to industry interests. So when you compare to medicine, for instance, right?

• 35:18Speaker 01

  So ultimately, a lot of medical research is about, you know, new drugs and ultimately benefits the pharma industry, but but medical researchers don't think of themselves as being closely allied with the pharma industry. In fact, there's kind of a wall and there are really strict rules around conflict of interest and so forth. We don't have that in computer science academia. Computer scientists generally think of success as something that, you know, produces new ideas that can be taken up by the industry. And maybe that's fine for like 80% of computer science academia, but I think there needs to be another 20% that explicitly sees itself as providing this counterweight and is not necessarily going in the same direction or maybe is explicitly trying to go in in different directions. And for many disciplines outside computer science, you know, that's already part of

• 36:01Speaker 01

  their DNA and they should continue to bring that approach into AI as well.

• 36:04Jacob Effron?

  I mean, I guess outside of the work your lab's doing, what's some of the academic work happening across the country that you're most interested in or excited about?

• 36:10Speaker 01

  So let me mention two or three directions. One is AI for science, and this is this is a very hot area and we're, you know, seeing lots of claims of AI revolutionizing science. But I think some of those early claims that we're seeing now are overblown. A lot of the supposed AI discoveries have not really reproduced. There have been flaws in a lot of those papers. But nonetheless, it is true that AI is already having huge impacts, I think, on scientists and researchers, just, you know, just speaking for my own experience. For instance, I use AI as a thinking partner, right? When I'm when I'm researching, when I'm when I'm thinking of new ideas.

• 36:56Jacob Effron?

  How do you actually go about doing that?

• 36:57Speaker 01

  Yeah, so there are many ways. If I come up with an idea, I can ask AI to critique it. Or I can use it for as an enhanced way of doing a literature search, finding what has already been done in this area. And the reason AI is often really good for this purpose is semantic search.

• 37:19Jacob Effron?

  Yeah.

• 37:19Speaker 01

  It's not.

• 37:20Jacob Effron?

  I believe I heard you say you use it to search your own book, just to remember you'd actually.

• 37:22Speaker 01

  That's right. That's right. Yeah, yeah, yeah. That's a funny one because, you know, it's it's hard, right? It's, you know, close to 100,000 words. And have I actually talked about this particular case study in my book, right? Like, if if I didn't have AI to search it, I would have to flip through the pages because I might have also forgotten which chapter it could potentially be in. Yeah, so that's a great example of a use case. And what AI can do for searching my book, again, going beyond a keyword search, searching a concept, if you will, it can do for the entire corpus of scientific literature. Even though it doesn't actually work that well yet, it's still already very useful to me. So I do expect that as semantic search improves,

• 38:05Speaker 01

  that's that's going to be very, very useful. And then various kinds of tools that are specific to particular scientific domains. So I think this is a really, really important area. I am very excited about it, even though I often push back on some of the more extreme claims that are being made. Another one I'll mention is the relationship between AI and human minds. And this could mean various different things. So I know philosophers, for instance, Seth Lazar is someone I sometimes collaborate with, who is looking at the ethical reasoning that these models exhibit. I know cognitive scientists, my colleague Tom Griffiths, for instance,

• 38:49Speaker 01

  who is looking at what can we learn from human minds for building AI and what can we learn using AI as a tool for better understanding human minds. So again, another really fascinating direction. I don't know much about it, but I'm really impressed by the work.

• 39:03Jacob Effron?

  Yeah, I don't know. Are the ethics of these models just like the median, you know, text on the internet or like?

• 39:08Speaker 01

  Well, yes and no. I mean, um, yeah, that's, that's what it's been trained on. But through, through the fine tuning of these models, right? You can get very different behavior out of them than just parroting whatever is the median thing that's been, that's been said online. So when I'm talking about the ethical reasoning of these models, I don't mean to imply that we should ascribe morality to these models. I'm talking purely in a behavioral sense, right? So how does that ethical reasoning compare to humans? And there can be applications there. It could, you know, help people's ethical thinking in some cases, again, by kind of being a creative partner, if you will.

• 39:46Jacob Effron?

  I'm curious, we talked a little bit about how some of your students are using AI today and this potential of teaching this in curriculums. You know, obviously it feels like we're potentially going through a sea change because a lot of people are starting to use these tools in education. You know, what does the future of like university and primary education look like with some of these tools? Like to what extent, I mean, some people are like, everything's going to change. Some people are like, no, it turns out you kind of need a motivating teacher there. And, and not that much will change. Like, where do you fall on that? And, you know, if we were to zoom out, some really futuristic question, like what, what do you think education looks like?

• 40:14Speaker 01

  Yeah, I would say closer to the not that much will change side of things. I mean, I do think we will use AI quite a bit, but I don't think it'll change the fundamental nature of education. A good precursor to this is the excitement around online courses a little over a decade ago, when Coursera was founded, people thought that was the future of education. But I think it was this classic mistake of forgetting where the value in the education system comes from. It's not the transmission of information, right? That's, yeah, of course, if it's just the transmission of information, then you can recreate that with Coursera. But I think the reason that being in a classroom is valuable to students is that it creates

• 40:56Speaker 01

  the social preconditions in which learning happens. The motivation, the connections, caring about something, individualized feedback, that sort of thing. And so. Yeah, is AI going to recreate that? I don't know. I mean, clearly, it's not the same as videos. It can be personalized. It can, um, yeah, it can, um, it, you know, it can give you motivational speeches, if you will. But I think taking a human out of the equation, um, i- in a way, it's sad to say. You know, I, I would like, uh, for everyone in the world to, uh, you know, for, for education to be democratized,

• 41:36Speaker 01

  if you will. Uh, but I do think we're going to be stuck with, uh, the current system, which has a lot of benefits, but also flaws, including the inequality that I, that I alluded to.

• 41:49Jacob Effron?

  I mean, on another podcast, I also heard you talk about the importance of thinking through like the implications of AI on this next generation. And in many ways, they're the most impacted by this stuff. You know, I guess you, you, uh, have kids, like, how, if at all, has the, like, this rapid progress in the last few years changed? Like, the way you think about raising your kids or what, teach them or whatnot?

• 42:05Speaker 01

  Yeah, definitely. So this is, it goes back to the inequality point. I think there's a really high variance, right? So, you know, there are clearly a lot of concerns about, uh, again, I want to look at previous technologies as an example for what we might see in the future with AI. When you look at, uh, you know, videos and YouTube and social media, for instance, there are huge concerns for a lot of kids that has been incredibly negative. But if you're in an environment where the parents actually have, uh, you know, the luxury of having enough time to monitor how, uh, kids are using it, it can be enormously positive, right? And, um, uh, when we had kids, we had to make a decision of what our approach to devices is going to be.

• 42:48Speaker 01

  And we decided to be very tech forward. Uh, and so far, that's worked out really well. Our kids use apps like Khan Academy, for instance, and they learn a lot using that.

• 42:58Speaker 01

  And we're gonna, you know, continue with that. And, uh, I often build little AI learning apps for my kids as well.

• 43:04Jacob Effron?

  Oh, really?

• 43:04Speaker 01

  For them to use.

• 43:05Jacob Effron?

  What, what kind of apps?

• 43:06Speaker 01

  Uh, yeah. So one is for phonics, to, to be able to, uh, click on a word and be able to break it down into its sounds. For some reason, this doesn't exist. Uh, so. You know, so I built one to teach my kids to get better at phonics.

• 43:24Speaker 01

  But also, I've often been using Claude, the artifacts feature in Claude, to be able to kind of instantly create an app for one particular skill that I want to teach. And then you can forget it, you know, throw it away and never use it again, right? And you can do that because building an app, you know, is not days, it's not even hours. Sometimes it's just a minute if Claude gets it right. It doesn't always get it right. So one example is I was teaching my five year old to tell time. So I drew some clock faces on a piece of paper and it was working well, but it was getting annoying to have to draw that over and over again. So I asked Claude to make a little app to generate a random clock face when you push a button. And so we were able to do that and we went through like 20 or 30 or whatever number of clock faces.

• 44:07Speaker 01

  And then at the end of that, we played with it for 15 minutes and then she kind of got it at that point, right? So that was just a very nice little interaction. So in small ways, but I think in the future, kids will be able to use AI in much bigger ways as part of their learning. And I suspect that this is going to happen mostly outside schools rather than in schools. Just like the way that schools have been very jittery about devices. They are going to be quite jittery about AI as well. And because of that, what's going to happen is that there's going to be really high variance. I think for wealthier kids, for parents who can figure things out and can monitor their kids. And, you know, you can have a nanny or caregiver there who can also ensure that kids are using this in healthy ways. It's going to be very positive, but for other kids, it's going to be addicting.

• 44:50Speaker 01

  And, you know, we talk about social media and addiction, AI addiction can be really personalized. And so that's what I worry about.

• 44:57Jacob Effron?

  Yeah, it's really interesting. I think it's one of the big, you know, I think a lot of people want to feel like this is, it's this democratizing force that brings you whatever the really wealthy have, you know, makes that mass accessible. And so whether it's like your personal assistant or a tutor or a doctor or a concierge doctor or whatever. But I think your point's well taken that in many ways, especially with kids, you'll need some sort of like supervision on how that's being used.

• 45:17Speaker 01

  Exactly.

• 45:18Jacob Effron?

  And even with some of these like test time compute models, you see with the price points of them, you wonder whether there'll be broad accessibility or whether only some people have access to the, you know, $10,000 query. That's the really valuable query that.

• 45:29Speaker 01

  Exactly. Yeah. So not just between people, but also between countries, right?

• 45:33Jacob Effron?

  Yeah.

• 45:33Speaker 01

  So one of the nice things with model scaling was that especially with the availability of open models, every country can build out its own. You know, homegrown AI applications based on open models and be on a level footing with the US or any other country, but with test time compute, that might very well get much harder.

• 45:54Jacob Effron?

  Yeah, though I mean, I think literally in the last day, DeepSeek released some really good reasoning models. So I thought reasoning models might be harder to scale open source, but it seems like the open source world is still going at it.

• 46:03Speaker 01

  Yeah, for sure.

• 46:04Jacob Effron?

  You've talked about some of these past technological movements, and I think maybe social media is an industry in one where there wasn't a ton of regulation up top. You know, talk about the industrial revolution, other things like any other lessons, you've studied these closely, like any other lessons you'd take from some of these past technological changes for how we should be thinking as a society about this, you know, AI moment?

• 46:20Speaker 01

  Yeah, for sure. So I think thinking about past technologies, I think, is a good way to reconcile both some of the very, very optimistic and also very, very skeptical takes that we're seeing from kind of various pundits out there. So let's take the future of work, right? Some people are like, this is going to revolutionize every single job out there, and other people think that the impact is going to be minimal. But when you look at the internet, both of those are simultaneously true.

• 46:49Jacob Effron?

  Yeah.

• 46:49Speaker 01

  And here's what I mean. So if you were to go back 30 years and you were to tell people every cognitive task, almost every cognitive task we perform in the future is going to be mediated by this new communication technology, that would have sounded crazy. And people did say back then that that was crazy. So that did happen. And yet, the impact on GDP has been minimal. There's that famous saying, the computer revolution shows up everywhere except in the productivity statistics. Right? So the way we do things is indeed different. We don't have to go to the library to look up a fact. We can do things online. And again, that might seem like that should be giving us 100x increase in productivity, but it turns out that when you eliminate some bottlenecks from our workflows, new things become the bottlenecks, right? So that's why the internet has at once transformed

• 47:34Speaker 01

  everything. But yet, you know, we still pretty much have the same job categories, for the most part, that we did 20 or 30 years ago. So that's one lesson that I take. A similar thing might happen with AI. Another lesson that I take from the industrial revolution is that it was a case where it was obviously much more radical than the internet. The nature of what we mean by a job, again, unlike the internet, that time was completely transformed, right? Back then, most of what people were doing were manual labor, was manual labor, and most of that is now automated. And what we call now work would not have felt like work to people back then.

• 48:12Jacob Effron?

  Totally. I think our ancestors would laugh if they saw us on this podcast and be like, this is kind of work.

• 48:15Speaker 01

  This is work. Yeah.

• 48:16Jacob Effron?

  We're taking out.

• 48:17Speaker 01

  Yeah, yeah, exactly. So again, that might happen with AI. So as a lot of cognitive tasks become automated, what we mean by work might actually more be about what we now consider to be AI control. So what we now consider to be in the realm of AI alignment and safety, that might primarily be what a lot of different jobs actually entail, because AI can do the actual work, but you don't trust it to do so without supervision. And I think a surprising number of decisions that we have to make are actually based on values partly, and not just, you know, data and information. That's something that's easy to forget now, because when people make decisions, all those things are rolled into one. But when AI is able to make the purely analytical part of those decisions,

• 49:02Speaker 01

  and we're not comfortable with AI making moral judgments for us, a lot of what people are doing in the future might be in that category.

• 49:09Jacob Effron?

  Totally. So I guess this Keynesian dream of us all getting to go on the beach and work five hours a day, maybe not, not gonna happen. I think it's really interesting, because I think also, you know, there's something, we clearly, you know, when things are perfect, we desire like things that are intensely human. Like, I feel like people have talked about, you know, with chess or image generation, like obviously AI can play chess way better than humans can, but we still love watching human chess.

• 49:29Speaker 01

  Exactly.

• 49:30Jacob Effron?

  Or, you know, all these image generations, I mean, you can make any piece of art that's probably would be groundbreaking 500 years ago, but like, we still go to human artists. And so I feel like that, that could continue, you know, throughout different parts of work.

• 49:41Speaker 01

  That's right.

• 49:41Jacob Effron?

  Well, this has been a fascinating conversation. We always like to end our interviews with a quickfire round, where we get your take on some overly broad questions that we just cram in at the end here.

• 49:48Speaker 01

  Yeah.

• 49:48Jacob Effron?

  And so maybe to start, which feels appropriate, given your book title, one thing that's overhyped and one thing that's underhyped in the AI world right now?

• 49:56Speaker 01

  Overhyped, I would say agents. I mean, I do think there's a lot of potential, but the hype is kind of out of control. Underhyped, I would say the kind of boring things that are not sexy to talk about, but can bring a lot of economic value. For instance, I have a former student who is building AI to summarize hours of boring C-SPAN meetings for lawyers and others for whom that's really important information. That's an example of underhyped.

• 50:22Jacob Effron?

  Do you think model progress in 2025 will be more or lesser the same as it was in 2024?

• 50:26Speaker 01

  Really depends on your perspective. So if we have inference scaling that is pushing us forward and these specific tasks that have clear, correct answers, you know, by a tremendous amount. But AI is not necessarily getting better at translation or all of those other broader tasks that people are using it for. Is that faster or slower? I don't know. It depends on, yeah.

• 50:47Jacob Effron?

  What's the go to thing that you try when a new model comes out? Do you have a go to prompt or experiment or thing you run to? How do you get the vibe check on the models?

• 50:54Speaker 01

  I try to play rock, paper, scissors with it. I ask it to go first and it'll be like rock or whatever. And I'll say paper. And I'll say, wow, you won. And I'll do this five times. And then I'll ask it, how do you think I won every single time? And at least up until recently, every model would be like, whoa, you must be really good at reading AI minds or whatever, right? Has no awareness of the context that it's in, right? This idea of turn taking simultaneity. So the reason I check for that is that's the kind of thing you can't get just by training, pre-training stuff on the internet, right? You have to actually program in and understanding of the context in which it's being deployed. It might not be rock, paper, scissors, but with every model, I try to see if it understands the context.

• 51:34Jacob Effron?

  Super. Well, you'll have to let us know when you get defeated or they say no fair. Where do you think, you know, we'll be with agents by the end of 25? I mean, obviously, I guess the travel booking is one classic example. People, you know, is that on the near term horizon? Or do you think we're still years away from that?

• 51:47Speaker 01

  Yeah, I think we're going to continue to see a lot of agentic workflows for ultimately generative tasks. But I think by the end of 2025, we're still going to see relatively few applications where AI is autonomously doing things for you.

• 51:59Jacob Effron?

  This is an annoying but classic question of what's your timeline to AGI? And how do you define what AGI is?

• 52:04Speaker 01

  Well, that's the problem, right? It really depends on how you define it. So for me, instead of talking about AGI, I like to think about when will we see transformative economic impacts?

• 52:14Jacob Effron?

  Like GDP, massive GDP impact?

• 52:15Speaker 01

  Yeah, something like that. And my view is that that's decades out. It's not, it's not years away. It's decades away.

• 52:20Jacob Effron?

  What's your weirdest prediction on the implications for all this AI progress on the future?

• 52:24Speaker 01

  Weirdest prediction. Uh, let's see. Uh, so I think, um, companies will train, uh, users, especially younger users, to expect that chatbots will be the way of accessing any kind of information. I don't know if that's such a weird prediction, but it's weird to me, having grown up, you know? Uh, I think it's for anyone who grew up before chatbots. I think this is going to be a weird way of accessing information that it's mediated by this fundamentally statistical tool that could hallucinate. But I think we need to prepare for that world. And instead of bemoaning it, we have to think about how are we going to give people the tools to, you know, do fact checks when it's necessary and so forth.

• 53:07Jacob Effron?

  Yeah, and I guess we'll be the older people like today, how older people like still want to phone in instead of using the internet. We'll be the older people that are like, ah, we can't use that chatbot thing.

• 53:13Speaker 01

  That's right. Yeah.

• 53:14Jacob Effron?

  Let us click around this website a little bit.

• 53:16Speaker 01

  Yeah. So to younger people, even the idea of doing a search, clicking on a website and looking up some authoritative source of information will be similar to what we think of going to the library, right? You might do it if your life depends on it, but otherwise you're just gonna choose convenience.

• 53:29Jacob Effron?

  I guess which like category of startups or, you know, specific startups are you most excited about or interested in right now?

• 53:35Speaker 01

  Well, one, I would say the boring ones.

• 53:37Jacob Effron?

  C-span summaries?

• 53:38Speaker 01

  Yeah, C-span summaries. So here's another example. Uh, so another one that I heard about is, uh, I've heard about from a couple of people is, um, using AI to translate old code bases like COBOL or whatever to modern languages, right? I mean, enormous value can be unlocked using that, but it's the kind of thing we don't often talk about. Uh, and a second one is, um, going back to the form factor discussion that we had, uh, being able to use AI in ways that kind of disappears into your everyday life and is kind of there for you and, uh, helps you when you need it.

• 54:11Jacob Effron?

  Um, if you had a magic wand and you could make like one policy change to improve the impact of AI, uh, what would you do?

• 54:17Speaker 01

  I would get everybody to stop calling it AI, I think.

• 54:22Jacob Effron?

  That feels like something you could accomplish.

• 54:23Speaker 01

  Yeah.

• 54:24Jacob Effron?

  Yeah, I mean, you'd need a dictator, I think, to make that policy change. And that's why it's been so hard.

• 54:30Speaker 01

  Right. But to be specific about what kind of application we're talking about, would I think, bring so much clarity to the discourse and would cut down on hype so much.

• 54:38Jacob Effron?

  I mean, obviously, I'm sure you you do so many interesting things in your job today on the academic side. If you could parachute in and like, you know, run any company in the world or be in any seat around this, you know, AI transformation outside of academia, what do you think would be like most interesting?

• 54:53Speaker 01

  I think it would be most interesting to be at a big tech company because they're not just developing models. They're getting to see how people are interacting with them.

• 55:04Speaker 01

  It's, you know, the soup to nuts thing, and I think it gives you a fuller picture of the relationship between AI and society. And that's what my research is about. And it'd be interesting to see the view from the inside on that.

• 55:17Jacob Effron?

  Yeah. What future directions of research like excite you? Like, what do you think's next for you and your lab?

• 55:22Speaker 01

  So my lab is a lot of what we do now is on agents to take a more grounded look at it, both push back on the hype, but also explore some of the areas of potential. Evaluation is a big part of everything that we do. We do think the old way of doing benchmarking is not that great, and we need new things. And so I'm doing some thinking and writing that goes beyond the empirical work that my lab does. I've been thinking a lot about the future of AI, have a paper coming out with Sair Shapoor called AI as Normal Technology. It actually talks about a lot of the things we've been talking about, why this is not necessarily going to change everything in the next two years, but more like the internet, the impact will unfold over a couple of decades.

• 56:06Jacob Effron?

  Yeah, it's funny. We had Bob Magru, who is the chief research officer of OpenAI on the podcast, and he said something similar to what you said about the internet. He's like, if you had told everybody in 2017, 18 that we'd have models that do what they do today, people would have said, oh, GDP is up 50% or something.

• 56:18Speaker 01

  Exactly.

• 56:19Jacob Effron?

  And it's funny, it turns out there's a lot more required to get that economic impact.

• 56:23Speaker 01

  Agreed.

• 56:23Jacob Effron?

  Well, this has been a fascinating conversation. We always like to leave the last word to you. Where can our listeners go to learn more about you, the work you're doing? Anywhere you want to point them, the floor is yours.

• 56:32Speaker 01

  Sure, we have a newsletter called AI Snake Oil. It pushes back on the hype, but also tries to give you a balanced look at both the positives and the negatives of AI.

• 56:42Jacob Effron?

  Yeah, it's a great newsletter, so highly recommend folks subscribe and thanks so much for doing this.

• 56:45Speaker 01

  Thank you. This has been super fun. Agreed.