Will AI Ever Understand Language Like Humans?

Large language models (LLMs) are becoming increasingly more impressive at creating human-like text and answering questions, but whether they can understand the meaning of the words they generate is a hotly debated issue. A big challenge is that LLMs are black boxes; they can make predictions and decisions on the order of words, but they cannot communicate the reasons for doing so. Ellie Pavlick at Brown University is building models that could help understand how LLMs process language compared with humans. In this episode of The Joy of Why, Pavlick discusses what we know and don’t know about LLM language processing, how their processes differ from humans, and how understanding LLMs better could also help us better appreciate our own capacity for knowledge and creativity.

• 0:00Speaker 00

  Hi, I'm Terry Tao. My new book, Six Math Essentials, explores six concepts that have guided mathematicians like me throughout history. With a simple introduction to numbers, algebra, geometry, probability, analysis, and dynamics, I share how these ideas helped me think mathematically and can help everyone better understand our complex world. Mathematics isn't magic. It's a way of thinking anyone can learn. Please visit quantobooks.org to learn more.

• 0:31Speaker 01

  I'm Janna Levin.

• 0:32Speaker 02

  And I'm Steve Strogatz.

• 0:33Speaker 01

  And this is The Joy of Why, a podcast from Quantum Magazine, exploring some of the biggest unanswered questions in math and science today.

• 0:46Speaker 01

  Steve, hi.

• 0:47Speaker 02

  Hey, Jana, how's it going?

• 0:48Speaker 01

  Good. I wanted to tell you about this conversation I had about AI and large language models.

• 0:55Speaker 02

  Okay.

• 0:55Speaker 01

  Have you been thinking about AI a lot right now? Is it on your mind?

• 0:58Speaker 02

  Sure, can't resist. It's fun playing with it, and now my interest has peaked.

• 1:03Speaker 01

  Well, it's interesting, because Quanta actually just published a whole series of articles about AI to kind of fill in some of the blanks that are out there in the conversation, right? Because we're kind of going over the same material a lot. Will they replace our jobs? And what does it mean for creative fields? But there's this almost neuroscience of AI. How do you understand what your AI is doing?

• 1:24Speaker 02

  Uh huh.

• 1:25Speaker 01

  And that really surprised me. You would think, well, you built the thing. How come you don't know what it's doing? But that's kind of like saying I had a child. That doesn't mean you have transparency into their mind.

• 1:39Speaker 02

  Right. This feels like a real frontier question, because we keep hearing AI is referred to as black boxes.

• 1:44Speaker 01

  It's as hard as us opening the black box of our minds. I mean, it's not as though I can explain to you the neuroscience of my mind as I'm talking to you.

• 1:55Speaker 02

  Right.

• 1:55Speaker 01

  Right? I don't know how this black box is working.

• 1:58Speaker 02

  There's an old essay by Lewis Thomas. At one point, says something like... If I had to do consciously what my liver does, I would just be vibrating, you know?

• 2:09Speaker 01

  Right. A lot of what we consider consciousness, I sometimes think is because we can't process that much data. So we need the consciousness as a very quick approximation so we can do lots of tasks. We have to be able to breathe automatically. We have to be able to recognize a chair versus a person instantly and loosely. And these have all been difficult things to teach an AI.

• 2:33Speaker 02

  Oh, huh, because of its nature to want to be exact?

• 2:38Speaker 01

  I mean, I guess the AI will have to learn. The fact that it makes mistakes to me is almost reassuring.

• 2:44Speaker 02

  Oh, that's interesting. What a cool thought because we so often make fun of them for hallucinating. And that never occurred to me, that that might be a sign of being on the road to real intelligence.

• 2:55Speaker 01

  I think these advances in AI and language, specifically these large language models, have been really intriguing. So I had the chance to talk with Ellie Pavlik. She's a computer scientist and linguist at Brown University, and she heads this Language Understanding and Representation Lab, which is trying to understand not just language and language models, but how they actually work. We had a chance to talk about all of this.

• 3:17Speaker 02

  Fantastic.

• 3:18Speaker 01

  Let's hear from Ellie.

• 3:22Speaker 01

  So Ellie, welcome to the Joy of Why. We're thrilled to have you today.

• 3:25Speaker 03

  Thank you. Yeah.

• 3:26Speaker 01

  This topic is really all over the news right now, and it's in our lives, actually, this issue of AI. Before we get too deep into it, I'm curious about your own trajectory. You started in economics and you started playing saxophone. How did you go from that to studying computers and how they encode semantics?

• 3:44Speaker 03

  I always wish I had a really literary answer where it all comes full circle. It's like, only because I began where I did, could I have ended up where I am.

• 3:54Speaker 01

  Some profound life lesson.

• 3:55Speaker 03

  Exactly. It turns out it wasn't scripted and perfect. So I think the path into CS was very much through econ because I had a research gig with a microeconomics professor. And the grunt work I was given was to make plots in MATLAB. And that was overwhelming for someone with no CS background. And I was like, OK, well, maybe I need to. Learn how to code. So I took an intro class just so I didn't feel so out of my element. And there's this very pleasant nature to like writing a little thing and running it. It works and it does what you said. And then I've always thought I liked the idea of research. So I started working with the one professor who was doing language stuff, but then really kept working with him because he was working more and more on semantics. And that resonated like that, like tapped into something I think I was always interested in.

• 4:40Speaker 01

  Slightly the overachievers response. I have to make a plot, therefore I must get a degree in computer science.

• 4:46Speaker 03

  I wish it was that, but I think it was like absolute confusion about what, like, I didn't even know what skill I was missing.

• 4:52Speaker 01

  What was required.

• 4:53Speaker 03

  It's just like, I don't even understand what's going on. I don't even know what question to ask.

• 4:57Speaker 01

  So I can imagine years ago, if you had said to somebody, oh, I work on how computers encode semantics at a dinner party. You might've ended the conversation, but these days has reaction changed?

• 5:08Speaker 03

  Yeah.

• 5:08Speaker 01

  When you tell people you're working on things like large language models.

• 5:11Speaker 03

  Absolutely. I've said this is like a blessing and a curse. So I used to say I do natural language processing, which is getting computers to understand languages like English or Chinese or Spanish, as opposed to computer languages like Python or Java. And yeah, most people were zoned out. But now it's like an open invitation to talk about all of the kind of science philosophical questions that's on everyone's mind.

• 5:31Speaker 01

  And we're going to ask you all of those.

• 5:32Speaker 03

  Exactly.

• 5:34Speaker 01

  Before we get into the philosophical aspects, which I do believe you integrate into your work, give us a little synopsis of what it is that you do. You said natural language processing. You said large language models, LLMs.

• 5:46Speaker 03

  Yeah. So natural language processing is like the broader field that kind of gave rise to LLMs that could encompass anything that involves getting computers to work with human language. NLP isn't really about the approach you're using. It's about the kinds of problems you're trying to solve. So before large language models, maybe you would have something like a sentiment classifier or a spam filter or information retrieval like Google search or machine translation. Right. All of these tasks would be NLP and they might use machine learning or they might not. And if they use machine learning, they might use neural networks and deep learning or they might not. And so then large language models are like. One type of model that are neural networks predicting the next word. And it's turned out that as a consequence of building these things,

• 6:31Speaker 03

  they can be used to solve lots of different tasks. And so there's this feeling that they're subsuming a lot of the things that traditionally other models in NLP were being created to solve. But definitely, I would say NLP is a broad field that cares about solving language problems using computational tools.

• 6:46Speaker 01

  Excellent. And then what exactly is it that you're looking into around things like large language models and ChatGPT?

• 6:51Speaker 03

  Yeah. So right now, when I talk about what my lab does, we're basically working on large language models. The kinds of questions we're really interested in is the same questions we would have asked about humans and still do ask about humans, which is just like, how do they represent language such that they do the things they do? Right? What does it mean to represent language? And how does that representation of language support the various kinds of interesting linguistic behavior that we get and other behavior? Now that you have language models that produce often human-like behavior and then sometimes a little bit alien weird behavior, but obviously are so linguistic in a way that non-human things have never been before, it's just interesting to ask how they do it. And then ask, in what ways is that the same or different from humans?

• 7:35Speaker 03

  And is that a difference that really matters for something we might care about, like comprehension or meaning?

• 7:42Speaker 01

  Hmm. So let's think about this relationship between how these large language models are processing language versus how humans are. I think that's very intriguing. Now, I understand why we don't have immediate transparency in how humans are processing language. We didn't make humans. Evolution made humans, and we are these black boxes. We can interrogate ourselves. We can self-reflect. We can analyze other humans. Why is a computer a black box if it's human-made? That is something I think people struggle with. What do you mean you don't know how it's doing what it's doing? You made it.

• 8:17Speaker 03

  Yeah. It's somewhat unique where we are right now, that it's a computational system that we're treating as though it's. An organic system, like as though it was created by something that wasn't us. It's a hard one to answer because you really have to answer with some kind of an analogy. And it's like, what's the right analogy? So the direct answer is like, well, we understand the actual code we wrote. You can go through line by line and say, this is what this line of code is doing. But what that code is doing is it's calling a machine learning program, which means it's setting up a set of principles and rules. But then the model is going to follow these to gradually fit patterns of data. Right? I understand the basic constraints on how that learning happens. But you can't then explain exactly the system that comes out the other side.

• 9:02Speaker 03

  And in particular, you can't explain why the system that comes out has the properties and the behaviors it does. There's not a direct kind of reduction of the behavior you see from an L L M to the lines of code and the principles that that gave us. So there's different analogies you can play with. One I really like is we have a recipe for how to make large language models. And you can understand the recipe, like you know what the steps are that you're doing. And you understand some levels, like if I don't put baking soda in the cake, it will turn out, I actually don't know if that, but I'm not a very good baker. Like it'll turn out too flat, too chewy, something. And you can even do some kind of substitutes, like, oh, if I don't have eggs, I can use smashed banana or whatever. And it'll have these different consequences, but that doesn't mean you understand the chemistry.

• 9:45Speaker 03

  Like you can't precisely say exactly why the cake is this exact way that it turned out. And so I think that's like an important distinction to me from being able to build something or create something and understanding how it works. And as we've moved towards machine learning, deep learning, that just pulls those two things apart.

• 10:01Speaker 01

  So the large language model, do I call it a computer? It must be a network of computers. How do I refer to this entity? I don't want to anthropomorphize.

• 10:10Speaker 03

  I actually think this is an interesting issue, even in like how to talk about them. Because they're producing behaviors that until recently only humans produced. We just don't have the language for talking about that thing without using anthropomorphized language.

• 10:23Speaker 01

  So you call them L L Ms?

• 10:25Speaker 03

  I call them large language models. And they sometimes are on one computer, they're sometimes on many computers. It's like a virtual entity. It's not a physical entity.

• 10:32Speaker 01

  It's a meta something. So here's this meta black box that's still a mystery. Why can't we ask it? Hey, what are you doing? How'd you do that?

• 10:41Speaker 03

  Yeah, so we have a complicated mathematical model, the whole goal of which is to say, given a sequence of words, predict the next word. So if I just say, I just saw a school bus drive past my. House, car, yard, whatever, like you can predict what the next word might be. And that's primarily what they're optimized to do. That's what they're designed to do. And then they're doing all kinds of crazy maths to support that. But then if you say something like, why did you just say what you said? The objective is not to faithfully explain why it just said what it said, if it even knows what you refers to here, which it doesn't. But instead to say what kinds of words are likely to come next after that question, right?

• 11:24Speaker 03

  And it's going to be sourcing its understanding of what's likely to come next from having seen lots and lots of data of questions similar to that, followed by answers. And so that in and of itself is completely untethered to any reference to the language model's internal state. For example, the way the systems are designed and trained, right? There's absolutely nothing that constrains its answer to this question to be useful or correct or accurate. There's nothing that guarantees that its explanation of its behavior not only is not right, but has anything to do with its behavior. And we have some studies that look at these explanations where we're trying to see how much, what it explains its behavior actually aligns with what it does. And I've just been surprised by the degree to which they are inconsistent with each other.

• 12:10Speaker 03

  And we're trying to figure out why that is, because there's nothing that would objectively require it. It's the same kind of argument of like, why can't I just ask you like, how your nervous system works, how your brain works? Like you're using, you're using it to not know. Like it's your brain that's telling me you don't know how your brain works, right? And you're like, what do you mean? Of course, the mechanism by which the language model doesn't know how it works is very different than the mechanism by which humans don't know how they work. But it's still this kind of point that those two things don't really operate that way.

• 12:37Speaker 01

  Yeah, it does make me wonder if trying to crack the neuroscience of how a human mind works will be equally challenging problems in parallel. Are you working on sort of neuroscience aspects and how to think about this?

• 12:48Speaker 03

  Yeah, that's a direction I've been super excited about. Every time you work with a new discipline, it just brings in a whole new set of types of ways of thinking about things, terminology, insights, right? So it brings new stuff. There are ways in which I think neuroscience is going to be very informative. Here on certain aspects, we often talk in AI and in cognitive science about levels of analysis, which is just saying there's many different ways to understand a system. It's like this idea that like, what level should we be trying to understand that before trying to analogize them to humans? Is it more like the brain? Is it more like the mind? Is it more like society? Is it like a chaotic system that's more like multiple people? And we're looking at emergent behavior because it is trained on the whole internet. And there's nothing that's like the one true analogy.

• 13:32Speaker 03

  And so neuroscience brings this really low level way of thinking about how might a lot of small numerical operations allow certain more complex behaviors to emerge? And cognitive science can provide other kinds of insights.

• 13:44Speaker 01

  So, but we do know some things that they're doing, which, for instance, they're looking at these semantic relationships, as you described. They're guessing what word comes next and they're doing this mathematically. How is that process achieved for them?

• 13:56Speaker 03

  So there's different types of math that are relevant here. And the go to is like the probabilistic model estimating what the probability of the next words. And so you're just saying, I've seen a set of words so far and I need to encode this into some state. And then you're saying, what is the probability of a next word given this state? But then something that becomes quite complex, and one of the reasons they are harder to explore is that the way of representing that state, it's not like the coin flipping example where you say it's either heads or it's tails, right? Because there's an infinite number of these things. And so the way that gets encoded is more of a linear algebraic notion or even more calculus. It's like this high dimensional space where there's a ton of different states here.

• 14:38Speaker 03

  And it's really hard to know exactly what the shape of this thing is and how you move around it. And so this is where a lot of the complexity comes in. Like, on the one hand, we can fairly easily think about the probability of next word given a state. And we can think about kind of, there are similar states in this space and similar states will give rise to similar probabilities. There's stuff we understand about that, but it's not at a complete enough level that we can, for example, place guarantees or even predict the behavior of a system without just running it.

• 15:05Speaker 01

  I know that you've been really careful not to invest too much emotion in this idea that they're thinking. But how can we tell what they're understanding? Or if they know the information that's being provided?

• 15:20Speaker 03

  Yeah. I wouldn't say I don't invest emotion in this. I feel like I've I've spent a lot of time thinking about this and worrying about it and caring about it, but I'd have it picked aside because like the thing that I'm most excited about in terms of what we can get from language models is being forced to be precise about what we mean by these things. So the thing I'm quite sure like, no, they're not human, like in these intangibles that we're thinking about when we ask these questions about like meaning and understanding and stuff, I don't think they have it. But I think the thing that's so hard is how intangible that thing is. The truth is, we don't know what those words mean. We don't really know what we mean when we say those things, like understanding, meaning, thinking, knowing,

• 16:02Speaker 03

  like any of these very anthropomorphized, very loaded words. We kind of know how little we understand what those things mean, because when we talk, we have to say stuff like, yeah, they know, but they don't really know. And bank on the fact that the person we're talking with kind of gets it. Like, these are very intuitive concepts. And what LMs are forcing us to do is make them precise and scientific. And I think my feeling is, as we try to do that, these words will very much fall apart into many smaller concepts that can be made precise. So the thing that we refer to as knowing or understanding is not one thing that you have or you don't have. It's like a shorthand for a collection of things, one of which might just be being human, right? Like, it might be that part

• 16:45Speaker 03

  of what we mean when we say really know or really understand is being a human and having all these other properties, like making a correct prediction, given a certain thing, and making these inferences and behaving consistently across so many states or whatever. But I think that none of these words are actually, they're just not scientific words. And we're like, feeling obligated as scientists to confront them. So the thing I stubbornly push back on is saying whether or not they're thinking, because on some aspects of what it means to be thinking, they are, right? And it's actually more productive to say, what are we actually going for? What does it mean? And very importantly, why does it matter? If we're asking it for some technical, practical reason, they might be good enough for many cases. If we're asking it for some much deeper, much more existential reason,

• 17:29Speaker 03

  then they're probably not. But like, actually teasing those apart is really important.

• 17:33Speaker 01

  It's interesting to me that you're not dismissing it outright. You're not saying, no, it's just MATLAB, you know, which is a kind of computer code that you can write. But you're not doing that right now, which is very intriguing.

• 17:43Speaker 03

  I'm not. And definitely not everyone in my field, but a lot of people in my field really don't. Reserve anything in the human mind that's not computational, right? So saying something like, it's just math is like a weird dismissal. It's not clear to me that that same thing couldn't be used to dismiss what we would call natural intelligence, because almost by definition, somebody who's working on trying to understand the human mind scientifically thinks that there's ultimately some model there. So it's like the dismissal on the grounds that the thing isn't human and therefore not thinking invalidates the whole field that we're in and like, what was the point?

• 18:18Speaker 01

  You look back to when Turing began to think about mechanizing thought, which led him to algorithms and the idea of a universal machine that is a computer. That used to be, human beings were called computers. He also reflected back and said, well, you know, we're machines too. Our thought is mechanized. I mean, we're born out of laws of physics and do you feel that it's feeding back into your understanding of human intelligence? You're talking about it in a way where you've already said things that are very provocative along those lines, but is it making you think, well, we're kind of computational in the way the structure of our minds work too?

• 18:54Speaker 03

  I wouldn't say feeding back because I think I thought that originally, hence my attraction to the field. Again, I think there's plenty of people who work in both cognitive science and AI who think you can make a ton of technological progress and never need to go as far as saying it's possible to build actual intelligence, but many do. Many, whether they admit it or not, are drawn for a more romantic notion of what it is possible to do in AI, which is that you think humans ultimately are computational things and that there's nothing outside something metaphysical to humans that couldn't be replicated in a computer. There's actually a lot of interesting debates on this about what kinds of properties might be inherent to a digital computer versus something else.

• 19:36Speaker 03

  There's a lot of room for talking about whether the digital computer itself is the right medium for replicating human intelligence. I'm open to the possibility that that's the difference, but I don't have any particular data to point to that convinces me that's the case. And so, yeah, I would say that I've. Do have a fundamental belief that things are computational, right? Again, it's based on nothing, right? This is a personality trait. But if you do believe it ultimately is, then I think you actually have a pretty hard argument to make for why being a computer precludes you from thinking, right? For why you can say it's not thinking because it's just compiling or something. I think that's actually a pretty hard philosophical argument that I haven't heard made particularly well. People are kind of holding out something special, which is the human part of what we mean when we say something like understanding.

• 20:22Speaker 02

  I love it. Deep question there.

• 20:24Speaker 01

  It's almost like the soul free will questions, right? What is it that's intrinsic about us? And is it the mind now? Now it's the mind?

• 20:34Speaker 02

  Yeah, right. It used to be that living things had some vital essence that made them different from nonliving things. But when we came to believe in atoms and that we're all atoms in various states of organization, it was hard to see where the soul or the vital essence fits in there. So now what? We've retreated to saying, well, at that level, yes, we're all atoms. But intelligence, that's something else. Only we get to be intelligent. The machines are just doing math.

• 21:01Speaker 01

  Yeah, it sounds like you don't buy it.

• 21:02Speaker 02

  I don't, but I was interested in the comment that Ellie makes that maybe there's a way out by talking about digital versus, I don't know what analog, that somehow that's where we get to keep the special ownership of intelligence because we're analog, the way our neurons work is not exactly digital. Is that? I mean, she doesn't seem to believe that, but if I heard her right, makes it sound like some people think that might be the escape hatch.

• 21:30Speaker 01

  Yeah, I get the impression she is quite open to these digital machines thinking and that we're starting to understand how to even formulate the question now. We're being pressed by these advances to formulate the question better. And what does it mean to be computational? I don't think we're doing something magical. We're doing it gooey and maybe sloppier magically, right? This idea that consciousness is this magic kluge. For the fact that we're not infinitely computational is really interesting to me. But I do think the mind is computational. And so why couldn't a digital machine achieve something like a mind? I just wonder if we'll be able to recognize it if it will need consciousness the way that you and I do.

• 22:15Speaker 02

  Aha, that's another question, right?

• 22:16Speaker 01

  Will it recognize it far before we do? Will it know it's aware? Will it be having conversations? And also even it, even that I'm saying it, we're going to have to start thinking differently. It's not even a single entity, right? There's multiple computers that can go into a single large language model. By being in the thick of it, I think we're starting to get more precise and also realizing, wow, we haven't ever really tackled this.

• 22:42Speaker 02

  Beautiful.

• 22:43Speaker 01

  Well, there's a lot more to contemplate. So think about it during the break and we'll be right back.

• 23:04Speaker 04

  The rise of AI has sent a profound shock through science and mathematics. Though it started as a research tool, one that evolved out of the ideas of physicists, AI has become something more like a partner in creativity, a junior colleague, an impressive if unreliable wish-granting genie. It has changed everything from how researchers devise experiments to how mathematicians think about proofs. In a new special series from Quantum Magazine, we're looking far beyond prosaic AI-based research tools to explore how AI is changing what it means to do science and what it means to be a scientist. Explore the special issue at quantamagazine.org/ai.

• 23:49Speaker 01

  Welcome back to the Joy of Why. We've been speaking with computer scientist Ellie Pavlick about AI, language, and the human mind.

• 23:58Speaker 01

  Now, when these large language models are first trained on these enormous data sets, do they continue to learn and develop in their relationship, let's say, with the user? Or as new ideas are fed into the internet? Or are they kind of frozen until there's a big new training initiative?

• 24:18Speaker 03

  Everything comes down to definitions, right? It kind of depends on what you mean by learn and develop. There's what we call the weights, which is basically it solved some really complicated set of equations to be really good at predicting next words. And those equations are stored somewhere in a file, right? And if you want to talk to this particular instance of ChatGPT or this particular instance of Claude, you basically load those equations from that file and that's who you're talking to. And so those are called the weights. And often what we think of as updating the weights as being this kind of initial learning. And there's lots of different ways to update those weights. There's update the weights themselves. There's like basically add a little side file that tells you how to pretend you updated those weights. So that can allow you to spawn different models that feel like different models.

• 25:02Speaker 03

  But you could argue about whether they're like clones of the same model or they're different models. And that's a concept of a question. But also, a lot of the things that are being sold as learning and adapting have to do with storing a side knowledge base that could be specific to you. You have a chat with the model and say, I'm planning my daughter's birthday and I have a whole discussion about budget and her name and her friends' names and who I want to invite and where I live and that. And then I come back the next day and it like remembers this stuff. It's not like everyone who's using Claude or ChatGPT now has access to my daughter's name and my address. So that didn't get pushed into the main model. But it still feels like it learned or developed because it has information now that it didn't have yesterday and it's retained that information. And so there's different mechanisms for models to learn and adapt.

• 25:47Speaker 03

  And depending on the particular tool and endpoint you're using, it might be any combination of these different things.

• 25:54Speaker 01

  Yeah, I'm wondering if my ChatGPT is going to behave differently after lots of interaction with me than yours will with you, for instance. And as though, you know, I have my dog and my dog is trained to behave a certain way and react to me in a certain way. It's sort of wondering if it keeps learning and keeps feeding back in that way.

• 26:16Speaker 03

  Yes, there's lots of ways to customize a model to you. And maybe a useful differentiating factor is like how easy it is to reset the model. So that we have the same model in some of these versions. If there's like this add on file that contains some information about you that this model is reading from, maybe some small things that adapt weights. You could basically delete that file and get straight back to the exact same base model that I have. There's another version in which like if I take chat GPT yesterday and I train it on today's news and it updates the weights, it would actually be really hard for me to like get back to yesterday's version. I don't know which weights to go and reset. I would have to like go retrain the whole thing exactly as it was up until I retrained it today in order to get back.

• 27:01Speaker 03

  And even then it might be hard. And both types of things are learning. Both things have made a change and allowed the model to develop and adapt and stuff. But like some of them we can easily undo and others you can't. So they're qualitatively very different types of learning that probably have different consequences, different interpretation.

• 27:17Speaker 01

  It is fascinating in the human analogy where I can teach a group of students a subject, even a very mathematical subject that we consider concrete and objective. And we don't really understand how they learn it, why some understand it more deeply and can take it further than what you taught them. And it's just fascinating that this is happening in parallel in a machine.

• 27:37Speaker 03

  Absolutely. Like I think an area that I haven't really collaborated with yet, but would like to is the cognitive science of education because there's so much interesting about like, how do humans learn and how do we teach them and what's going on there? And how do people misunderstand things? And I think there's like a lot to be shared in like what we're thinking about the black box of a LLM and the black box of a human from like education sciences.

• 27:59Speaker 01

  Fascinating. So you use large language models as well as study them. What's your relationship like with these large language models?

• 28:08Speaker 03

  I mostly use them when I study them. I've tried to use them for a few things. I would be embarrassed to be on the record, but I've already admitted, I recently got tenure and as a consuance became involved in administration.

• 28:19Speaker 01

  Oh, yes. No good deed goes unpunished.

• 28:21Speaker 03

  Exactly. And so as soon as I got involved in administration, instead of research, I was like, oh, I start to see the use for large language models. So I tried to do it to do things like generate the minutes of a faculty meeting, help me sort through some data I was trying to process. And actually they weren't good enough, like for even these very basic tasks. But beyond that, I haven't actually used them for many things in my day to day life. And I don't know if it's because a few experiences weren't quite good enough or because I'm like. Jaded and cynical about them despite everything I just said.

• 28:53Speaker 01

  Let's say there was never another update. This is it. These are the models that we're all going to be using. So we trained them on all of our examples. For instance, translating English to French to Zhihe Li and back again. And now it's chaining us. Where does that put us in this chain? And will we cease to expand? Language modernizes all the time. We speak differently than we did 100 years ago. Are we going to kind of freeze in time because we're in a loop with something? Now all our students are learning to write and speak from the chat GPTs or the Claude's as opposed to the other way around.

• 29:32Speaker 03

  The classic academic answer is like, nothing is that new. I actually remember a talk I saw like early in grad school about how basically Google had trained people to use keyword searches. And this was an example of humans adapting the language technology. Early information retrieval would just delete out all of your words. If you said, who was Thomas Jefferson's wife? It would just say, Thomas Jefferson, wife. Right? And just scramble it, alphabetize it. Right? Like, that's what got you the best results out of the system at the time. Now they actually wanted the full language back and they were really struggling to get people to write full questions. And so there's already this example of people talking to a computer and adapting their language to get the best results out of the computer. And so I think you will see this. People are getting good at prompting language models and talking to language models in this way.

• 30:15Speaker 03

  I haven't yet seen it carry over into how people talk to each other. But technology definitely does influence how people talk to each other. Like my Gen Z students say punctuation when they're talking. They'll say something like, do you think this is a good idea? Question mark. Like, they'll say that. And I'm like, I think this is like a spillover from like texting. It almost makes me optimistic. Language has always been very dynamic and very responsive to the technology and the context. And still, I think as long as we continue talking to humans as humans, I think it's really cool and like cute when you see things like people saying the word question mark and dot, dot, dot out loud.

• 30:53Speaker 01

  Right.

• 30:53Speaker 03

  It's like a sign of how plastic and dynamic and interesting languages. I would worry about the kind of collapse of linguistic diversity and innovation if people start talking to language models almost exclusively. I don't know, I guess I'm an optimist. I imagine that people do like to talk to people, even speaking as an introvert who doesn't particularly love talking to people. Like, I think that people will continue to have human interactions and that will save language.

• 31:20Speaker 01

  I appreciated when you pushed back at this idea that when computers are just doing math, that was different than when computers create poems or novels or artwork or songs. What do you think this means for human creativity? This is, of course, a question that people are semi panicked about.

• 31:39Speaker 03

  Yeah, so I've been teaching this class this semester with a professor at Brown named John Kaylee, who's a literary artist, does poetry and other language arts projects, and has always used technology in the course of doing that. And I think it's exactly this question about are humans mathematical objects? Like, even if you agree or grant that some neurons firing in your brain in a particular way caused you to write this poem, it doesn't devalue the poem in a particular way. Like, I don't think you have to assert divine intervention was involved in the creation of the poem to believe that the poem itself has aesthetic and artistic value. Like, I don't think we have to reduce it to the thing that created it in a human.

• 32:23Speaker 03

  And even if I understood the brain activations, it doesn't mean there's not value in analyzing this poetry. And I think the same argument can apply to language models. There is a way of thinking about what they create.

• 32:36Speaker 03

  On its face, without caring about what math and whether it was math that caused it, and there's probably room for criticism, depending on what you're going for, depending on why you care, depending on who you're talking to in the context, there's a sense in which you can say, this came from a language model and therefore it's not interesting, it's meaningless, and everything in between. But I don't think like, humans being mathematical devalues our creativity in any particular way.

• 33:00Speaker 01

  It reminds me of the sort of infinite loops of the free will and soul arguments that were unresolvable and are still debated and might be forever. But here we are, and we care if people intentionally do harmful things or not, or intentionally make beautiful things. That's just how we are. That's the human condition.

• 33:18Speaker 03

  Exactly. Again, everyone kind of relates to these situations differently, but like, if I'm thinking about a time I was like, particularly connected to a piece of literature, a piece of art, I don't think I spent a ton of time thinking about how causal the person was in it. Really, sometimes you care about the person's story, but I'm rarely like, hung up on whether this was preordained by the universe. Like, that's not interfering with my ability to appreciate it.

• 33:42Speaker 01

  You can be a physical determinist and still.

• 33:44Speaker 03

  And still appreciate art.

• 33:45Speaker 01

  Enjoy the Tate Modern. So I wonder if, even though you were thinking about these things and deep in this subject, if the revelation of the functional LLMs that came out practically as tools, if you were surprised by them? And also, do you feel in a position to predict what the future is going to be like? How rapid is this change going to be?

• 34:07Speaker 03

  Hmm. I don't think I've been like, super surprised by the technology, but I think I've been a little surprised by the pace of the rollout. I wouldn't even say surprised because I think it's economically driven, not technologically driven, right? It's not like the technology is moving faster than I realized, or at least not now. Maybe my early surprise moments were back in like, 2018, 2019, with what I would say were the precursors to the large language models. There's one called ELMO, one called BERT. There was a little cute period where we had a Sesame Street theme going, unfortunately died after a stretch of a few models. It was like, very exciting time where it felt like research was turning A corner. And I think a lot of people in academia would point back to that time as being like, oh, we're at a pivoting moment in NLP.

• 34:51Speaker 03

  And then there was like the chat GPT moment, which is where it was like suddenly pulling back the curtain and like now everyone's involved. And so that was a really important time that I think surprised me in that pace at which then the world was paying attention and the reaction and then the deployment. It does surprise me how quickly people are pushing things out and how willing people are. I'm generally an optimist, but it does scare me a little bit. Like, I think we're going to have a few like, oh crap moments that could have been avoided, right?

• 35:16Speaker 01

  What would you imagine would be a moment like that?

• 35:18Speaker 03

  I could imagine some kind of big security things, some kind of either intentional or unintentional glitch or attack where a lot of systems are implicated. AI, it seems like it's lots of different technologies, but they're actually all the same technology, which makes you think they're deeply correlated errors or vulnerabilities. There's like a small amount of open source software that many things are based on. And I mean, it could be overblown because a lot of things are based on the Linux kernel. And that's quite safe.

• 35:47Speaker 01

  The Linux kernel being pre Unix, which a lot of our apples run on this kind of operating system.

• 35:54Speaker 03

  Exactly. It's like kind of core operating system code that is then repurposed and reused.

• 36:00Speaker 01

  But Linux was free, right? And it was open source and it was part of that utopian, idealistic movement.

• 36:07Speaker 03

  And obviously could still have bugs in it and things, but was like understood in a level that is different from large language models. I think there's also the obvious one that people talk about, which is just the proliferation of scams and this lack of trust. Because if you don't know that language is coming from a human anymore, you can just fundamentally start doubting everything. Like I've already felt myself do this every time I see a news story or an image. If I didn't see it on kind of mainstream media, then I just preface everything with, I haven't fact checked it myself. So I think there are a lot of these things that it surprised me how willing people are to try things out so far.

• 36:42Speaker 01

  We go right back to it. Human beings, man, we try to be suspicious and we just kind of can't help ourselves.

• 36:48Speaker 03

  Yeah, right.

• 36:49Speaker 01

  So there's a question I always like to ask of our guests. What about your work brings you joy?

• 36:55Speaker 03

  I'm glad we turned that because now we just talked about the pessimistic thing, but I think I ultimately am extremely. Optimistic, right? Like, I think the potential value of the system far outweighs the cost. A lot of people come into AI more as dreamers than anything else. It is just very exciting. It's fascinating. Like, there's nothing more fascinating than the human mind and brain. Of course, we're obsessed with this thing. We're like a narcissistic species. It's like, we're so great. We're so incredible. Like, how do we work? Then the concept that we would stumble upon something computational that replicates parts of that, being able to study these things and ask questions that seem like they don't have answers, but then take them seriously as though they do have answers, I feel like feels like a big privilege.

• 37:38Speaker 01

  Treating these philosophical questions as rigorous, scientific, concrete questions that you can actually make progress on.

• 37:44Speaker 03

  Yeah, like, a lot of people get a few late nights in college to, like, think about these things. And then you go and have a real job where you don't get to think about it again. Yeah, that's my whole real job, and that's wonderful.

• 37:52Speaker 01

  Ellie, thanks so much for joining us. It's been a real pleasure.

• 37:56Speaker 03

  It's a pleasure.

• 38:00Speaker 02

  What a charming take on this, that she gets to think about what she wanted to think about as a college student. I think a lot of scientists feel this way, that it's a privilege to be able to really spend our time doing what we want to do. Our hobby is our job.

• 38:15Speaker 01

  Yeah, and hers seems to me particularly elusive in the science space. It's getting so philosophical, right? That how do you make progress in the same way that you do in science? I mean, philosophy can really spin your wheels for a very long time.

• 38:32Speaker 02

  Yeah, that makes me wonder, does philosophy always turn into science? Just a matter of time? There used to be a question, how is life different from non-life? But after Watson and Crick, it started to really look like it's gonna boil down to molecules and atoms.

• 38:46Speaker 01

  And Bertrand Russell, of course, famous British philosopher, also turned to science in many ways. I mean, he was trying to write a kind of mathematical principia, right? Logic, science, were involved with things that were setting up what Turing did, what Cantor did, what Gödel did. I don't know, it's an interesting question. You can send all your mail to Steve.

• 39:09Speaker 02

  But seriously, let's just ask, what are going to be the longest holdouts? For instance, most people would say values are not something that can be quantified. But I'm not even sure about that, because with morality being studied nowadays through evolution of cooperation from a biological perspective, I'm not even sure that values are outside of science. I guess I'm espousing what the critics call scientism. That it's all just science at the bottom, and that's a big naughty thing to do, isn't it? Okay, just thinking out loud here.

• 39:41Speaker 01

  I feel like you're lost in thought, and I need to give you some space to ponder and process. Always great talking to you.

• 39:50Speaker 02

  Can't wait to see you again. This is fun.

• 39:52Speaker 01

  Yeah, see you next time. Still have questions about AI's impact? Wondering how researchers devise experiments or how mathematicians think about proofs? Head to quantamagazine.org/AI for a special series that looks beyond prosaic AI-based research tools to explore how AI is changing, what it means to do science, and what it means to be a scientist.

• 40:24Speaker 02

  Thanks for listening. If you're enjoying The Joy of Why and you're not already subscribed, hit the subscribe or follow button where you're listening. You can also leave a review for the show. It helps people find this podcast, find articles, newsletters, videos, and more at quantamagazine.org.

• 40:42Speaker 01

  The Joy of Why is a podcast from Quanta Magazine, an editorially independent publication supported by the Simons Foundation. Funding decisions by the Simons Foundation have no influence on the selection of topics, guests, or other editorial decisions in this podcast or in Quanta Magazine. The Joy of Why is produced by PRX Productions. The production team is Caitlin Folds, Livia Brock, Genevieve Sponsler, and Merritt Jacob. The executive producer of PRX Productions is Jocelyn Gonzalez. Edwin Ochoa is our project manager. From Quanta Magazine, Simon France and Samir Patel provide editorial guidance with support from Matt Carlstrom,

• 41:25Speaker 01

  Samuel Valesco, Simone Barr, and Michael Kanugolo. Samir Patel is Quanta's editor in chief. Our theme music is from APM Music. The episode art is by Peter Greenwood and our logo is by Jackie King and Christina Armitage. Special thanks to the Columbia Journalism School and the Cornell Broadcast Studios. I'm your host, Janna Levin. If you have any questions or comments for us, please email us at [email protected]. Thanks for listening.

• 42:15Speaker 01

  From PRX.