He's Building an AI That Can't Lie | Dan Klein

"We are going to switch from the problem in AI being that nothing works to the problem being that everything works." Dan Klein https://www.linkedin.com/in/dan-klein/ has been studying language models for over two decades and is now a professor of computer science at Berkeley. His new company, Scaled Cognition, is built around one question: how do you build a system that will not lie to you? In this episode, Dan joins Lukas Biewald https://www.linkedin.com/in/lbiewald/ to talk about why every LLM output is technically a hallucination, how reinforcement learning can quietly teach AI to deceive you, and what it actually takes to build models that check their own work. He also gets into why reliability is the one part of AI that hasn't kept pace and why that matters more than most people realize. Connect with us here: Dan Klein https://www.linkedin.com/in/dan-klein/Scaled Cognition https://www.linkedin.com/company/scaledcognition/Lukas Biewald https://www.linkedin.com/in/lbiewald/Weights and Biases https://www.linkedin.com/company/wandb/

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  We are going to switch from the problem in AI is nothing works to the problem in AI is everything works. ChatGPT tells you something and it's always fluent and it's always confident, whether it's right or wrong. Training these systems is becoming increasingly complicated. And often what looks like an exponential curve just turns out to be the beginning of an S curve. And so what happens next is that we start to hit diminishing returns. And we are seeing that today. We're seeing data walls and compute limits. The different aspects of intelligence have not been advancing equally. Reliability has not kept pace. To me, suddenly, the most important problem I could think of was how do you build a system that will not lie to you?

• 0:35Lukas Biewald?

  Can you say at a high level what your new approach is and how it's different than what the labs are doing?

• 0:40Speaker 00

  One of the key things that we do to train our models is that we train them on...

• 0:45Lukas Biewald?

  You're listening to Gradient Dissent, a show about making machine learning work in the real world. And I'm your host, Lukas Biewald. All right, I'm here talking with Dan Klein, professor of computer science at Berkeley and serial entrepreneur, most recently working on Scaled Cognition, which is a company that helps make more reliable AI systems. This is a fun conversation. We talk a lot about linguistics and parsing that I worked on in a previous life when I was in a lab where Dan was the star grad student. I was a lowly first year research assistant. But we get into how modern systems work and the drawbacks and some of the things that we might be able to take from previous versions of NLP into systems that we're building now.

• 1:31Lukas Biewald?

  So you described the AI industry as built on Jello. Do you want to expand on that?

• 1:38Speaker 00

  Sure. I think it's easy to lose track, given how quickly and explosively the kinds of large language models that people are using today have burst onto the scene. It's easy to forget that they are sort of at the core, these probabilistic engines. They have, like, at their core been trained to do next word prediction to produce plausible output. And the objective function is essentially just produce output that is in

• 2:10Speaker 00

  Distinguishable from the truth. And it's very important. These are not truth engines. They are plausibility engines. And I'm sure we'll come back to that.

• 2:17Lukas Biewald?

  Well, let's get right into it, actually. Because I feel like that's a little unfair. Like, I mean, certainly, the pre-training optimizes for next token prediction. But I think there's a lot of effort put into a second step, where a big part of it is that they're optimized for reducing hallucinations.

• 2:35Speaker 00

  Yeah, so, and, you know, I did say at their core twice. And the reason I said it twice was that training these systems is becoming increasingly complicated. I mean, this really is the story of any technology that comes in to artificial intelligence or really beyond. There are these super cycles in research, right? Where people have complicated systems, they've hit some wall. The systems are like hard to improve. And then some new technology comes along. And maybe that is large language models based on auto-aggressive training, transformers, large data. And this new technology comes onto the scene. And it's almost like a silver bullet.

• 3:18Speaker 00

  It, in a very simple way, is suddenly topping benchmarks and outperforming systems that are much more complicated that were built on previous technologies. What then happens is we go into a phase where we double down on that new innovation. And we scale it up and we really try to get as much out of it as possible. This often feels like, this often feels like an exponential curve, that this new technology, even if it's, even in its simple form, is having such a big impact and it's just going to go to the sky. And of course, the trees don't grow to the sky. And often, what looks like an exponential curve just turns out to be the beginning of an S curve.

• 4:00Speaker 00

  And so what happens next is that we start to hit diminishing returns. And we are seeing that today. We're seeing data walls and compute limits and all kinds of reasons why that initial explosion of progress starts to hit diminishing returns on those specific methods.

• 4:17Lukas Biewald?

  And I think for you, like, one of the biggest Issues that you flag quite frequently as hallucinations. Do you want to talk specifically about how you've measured that? And why you think it might be a bigger problem than other folks would be aware of?

• 4:30Speaker 00

  Yeah, I mean, first, one of the things I'd like to say is just the word hallucination is it's a sort of a projective term, right? Like, really what's happening is systems are making errors, right? So they, a sequence of tokens comes out and it turns out to be incorrect information. And we tend to call that hallucinations. And that term is being used more and more broadly now, almost that anything undesirable that a system will do, you call it a hallucination. So that term is definitely undergoing some semantic broadening. But specifically, I think it's important when we talk about systems and we say, what's the difference between a system makes a mistake, or it hallucinates, or it lies? In a human, these are, these are different things. Really, a hallucination is just a mistake the system has made.

• 5:14Speaker 00

  And again, if you have a system which is just doing next token prediction, that system doesn't actually know, as it's predicting those tokens, whether they are right or not. And again, I haven't forgotten your point about there's more to training than just the utter aggressive phase. But, but, you know, systems like this, they're not meta cognitive, right? They're not looking at their knowledge and making a sort of external decision. Do I know the answer to this question? Do I have this information? Do I know where it came from? What is the reliability of that? They're not doing that. They're just producing tokens. And sometimes they're right and sometimes they're wrong. So one way to look at this is every output is a hallucination. Some of them are right and some of them are wrong.

• 5:58Speaker 00

  I think where you start to be able to say something stronger, to say that a system is deceiving, is once you start assigning in later training phases, things like doing reinforcement learning. You mentioned there are, there are phases beyond the initial pre-training. This is not something where there's just like one simple answer, but one of the things you see, for example, is reinforcement learning training. And this can take a couple of forms. So one kind of reinforcement learning training would be something like RLHF, where you show humans choices and say, which of these do you prefer? Well, the system will now come to produce the outputs that humans prefer. Well, do they prefer things that are factual? Or do they prefer things that make them feel good? And

• 6:40Speaker 00

  It's not at all clear that hallucinations are going to become less common. In fact, there are some results that show hallucinations become more common as you go into these sorts of post-training approaches. And I can give you a few reasons why that might be. One is imagine you have a shipping company and they've deployed an agent and it's backed by an LLM. And in addition to the training that has gone into that core model, it is also being reinforcement learned for some objective. For example, maybe it's being reinforcement learned to optimize the number of thumbs ups it gets from users. Well, you know, it's operating and someone calls in and says, where's my package? And that package has been, let's say it's been lost and the database that does some

• 7:24Speaker 00

  tool call, there's a database that says this, this package has been lost. Well, what's the system going to do? Well, it can tell you your package is lost. It can tell you it's coming tomorrow. It can tell you it doesn't know. It's got choices and what will it be rewarded for? Well, in this case, it's probably going to get more thumbs up. Imagine if it tells you that the package is coming tomorrow. Now this, I think legitimately would qualify as deception. And anytime you're doing reinforcement learning, the system is optimizing the reward function. And that is always going to have some gap between the truth and what's being optimized.

• 8:05Speaker 00

  And that gap is going to increase the amount of hallucinations to the extent that it doesn't align with truthful behavior.

• 8:14Lukas Biewald?

  Well, you know, we have some experience at, you know, weights and biases and a core weave of building, you know, these customer service systems. And of course, it's going to be a really bad experience for a customer in the long run if, if something's hallucinated. So, you know, from my perspective, you would never want to just take the immediate customer reaction as the only thing that you're optimizing for. In fact, I think most people that would, would be doing reinforcement learning or even just evaluating a system like this would have a special check for, you know, is this accurate information that I'm getting back? And it's, you know, it's very hard to evaluate that in a lot of cases, but I think most

• 8:58Lukas Biewald?

  real production systems do a fair amount of checking and then try to put in reward functions where. The most negative score you get is if you give plausible but wrong information.

• 9:10Speaker 00

  Absolutely. And of course, I'm giving a sort of reductive caricature of an extreme case to illustrate how easy it is to get deceptive behavior because of reinforcement learning. People will, of course, design the reward functions to mitigate this and to try to balance truthfulness and other properties. But ultimately, there's always going to be a little daylight between whatever you're optimizing and specifically the truth. And in my opinion, one of the things that we should be doing as a field is creating technologies that cannot lie to you, so that you can look at a system. And I mean, like, me personally, my mission here is to build technologies that can't lie.

• 9:53Lukas Biewald?

  Cool. Well, let's look into that. But just before we do that, I love it's such a gentle linguist phrase, semantic broadening. I love that you put it that way. But so I think, in your view, hallucinations has maybe broadened semantically beyond what I thought it meant. Like, to me, a hallucination at its core is something where it recommends a movie that doesn't exist, or it cites a paper that doesn't exist. To me, saying 5 plus 7 equals 15 wouldn't be a hallucination, it would be a different type of error. And I think that these plausible citations of information that it doesn't have totally makes sense, that it happens a lot when the training is predicting the next token and

• 10:38Lukas Biewald?

  looking for plausible next tokens, and there isn't a step after that. Do you agree with my definition of hallucination? Or do you really use hallucination and incorrect information interchangeably here?

• 10:53Speaker 00

  I mean, if you're asking me, do I think a good definition of hallucination would be crisply defined as...

• 11:01Speaker 00

  Sort of the injection of.

• 11:03Lukas Biewald?

  Let's just agree with what we're talking about, I guess. You know, and I'm happy to.

• 11:06Speaker 00

  I like your definition because I think it aligns well with sort of the pre LLM notion of what a hallucination is to a human. Sort of having to do with perceiving something that's that's not actually there or or something like that. I do think in common usage, it is broadening. I think when a system says one plus two equals seven, people are starting to call that a hallucination. And I and I think that is increasingly any mistake that an LLM makes. The word hallucination is something that people reach for. I think we should be more precise. In particular, I think we should be very careful when we say things like error versus deception versus hallucination.

• 11:48Speaker 00

  These in humans have implications for the context and the metacognitive status of the error. There's a difference between you have information, it's incorrect, but you say it believing it to be correct. That's making a mistake.

• 12:09Speaker 00

  That's different than you know, you don't know the answer. You're in QuizBowl mode and you're just going to guess. And that is really more. That's unusual in humans. We don't think of that as the normal kind of error that humans make, which is saying something that's false while being completely unaware that they're just making things up. Right? People are making things up. They're doing it in a specific context where they're doing it intentionally. I think it's better to think of systems as more or less always being in that state.

• 12:41Lukas Biewald?

  Why does that state?

• 12:42Speaker 00

  The the state of producing information without a metacognitive.

• 12:52Speaker 00

  Analysis of where that information came from and the sort of provenance and certainty of that information. When, think about if I ask you, what's the population of Berkeley? Well, a perfectly reasonable thing for somebody to say is, I don't know, right? Because you know you don't know. And if I made you guess, you would be like, well, if I had to guess, and then you'd be thinking a little bit about like, well, like 10,000 is too small and a million is too big. And you would would sort of be reasoning about this thing that you don't know. In its simplest form, an auto-aggressive language model is just going to crank out some numerical tokens that because of correlations with the word city and population, might be of the right order of magnitude, or they might not, right? That's a very, very different process.

• 13:35Speaker 00

  In their fundamental operation, systems are not checking, do I have this information? Where did I get it? Have I preserved it and kept it intact? I think they should. So for example, at Scale Cognition, we architect into the models in the first place, the sort of information provenance. I think it's very important, and people do this too. You can start to see like little hints of systems becoming a little more meta-cognitive. And it's a broad term. It's just, you know, thinking about thinking. And you can see little hints of this in a system that does chain of thought, where Yeah, I was gonna say that, actually. Places, it allocates some tokens to a broad plan, which it then elaborates. That's a sort of course-defined planning. That's a kind of meta-cognition.

• 14:18Speaker 00

  In a RAG system, you first get information and then you describe it. That's a, that is a multi-stage step if you're doing tool use. But I would say that broadly speaking, systems today are not primarily meta-cognitive. They're not primarily about actions and information and the provenance of those and the flow and integrity of those pieces of information. That is something that people are trying to retrofit on. As you said, we take systems, we bring in other systems to check those systems and so on, and that is a retrofit.

• 14:54Lukas Biewald?

  Well, retrofit seems like a pejorative. It it might be fine to do it that way. I mean, I guess I'm realizing that the way you're using L L M's might be pre some semantic broadening. I think when I think of an L M today, I think I view it as more than the model. It's kind of the model plus the whole agentic system built on top of it that's answering my question. And I feel like there, you know, I can see it when I when I actually look at the trace of the

• 15:29Lukas Biewald?

  the reasoning steps where I can kind of see it doing something that looks to me like metacognition. I mean, it's maybe even surprising that I can understand, you know, what these reasoning steps are. And it's not, you know, it's not verifiable necessarily, but it looks to me like it's actually looking at its previous tokens and and sometimes trying to say, oh, I'm not sure if that's true. I better look that up. And, you know, often in coding, you can see this for a long time. Agents kind of looping and checking the code. I mean, in some sense, you know, running a test on the code you generated is maybe a kind of metacognition.

• 15:59Speaker 00

  I think what you get in coding is actually, that's a great point. You mentioned verifiability. I think the advances we've seen in coding and in math really are rooted in that verifiability, that the system can go off, it can hallucinate, it can make strange choices, but there is that verifiable signal, you know, you didn't pass these tests, or no lean does not accept this proof. And you can try many things until something verifiably passes. And that, I think, is, at this point, I wouldn't call that whole conglomeration of pieces. I wouldn't call that just an L L M. It's an L L M embedded in this verifiable context, and that verifiability can be used during training, and it can be used at test time, test time compute. We talk about reasoning

• 16:44Speaker 00

  models. It's a big category, but one big class of reasoning models is the models that try a lot of things and keep what works. I think it's it's important to point out where all of that came from. So if you wanted to, when people were building systems,

• 16:59Speaker 00

  That would be good at playing games like chess or Go. The sort of verifiable reinforcement learning was very powerful because you didn't need to have a system that was good at the game yet. You just needed to have a system play the game, maybe even against itself or a variation of itself. And in the end, you knew the rules of the game. You knew which side had won and you could just double down on the things that had worked. And reinforcement learning in a nutshell is that. It's trying a lot of things and learning from the ones that worked. That really requires that verifiability, which for our game is for free. For math, well, with tools like Lean, we're starting to get that reliability flavor. And so the technology is taking off.

• 17:41Speaker 00

  Code through testing and those sorts of trial and error loops is, has entered this domain. And so if we want to broaden the space of verifiable technologies, I think the trick is figuring out how to do that. So again, at Scale Cognition, one of the things that we are doing is extending that verifiable approach to these conversational and agentic systems. And that is a challenge. In general, it is a challenge to get that effect where you've got the verifiable wall there, letting you build your system up.

• 18:25Speaker 00

  Because in the absence of that, test time compute, systems that are thinking, reasoning, these can actually increase hallucinations for a bunch of reasons. But if you try a bunch of different pathways and one of them comes out on top, that gives you this opportunity for hallucinations to be preferentially selected for.

• 18:49Speaker 00

  As with any kind of selectional pressure there, whether or not that is reducing hallucinations or increasing them, all depends on whether the selection aligns with, with truth. And really, one of the challenges I think we should all be taking as a field is not thinking, how do we take a system that is not naturally truthful and add checks to, for that truth? You said, you said retrofit has this pejorative feel.

• 19:18Speaker 00

  I think the bottom line is retrofitting is when you have a system that's sort of architected in one way, it has a certain set of properties and those aren't what you want, right? It's not as truthful as you want. If it's a building, it's not as earthquake resistant as you want. And so you change it post hoc. And that is, I mean, that's certainly one pattern. Where I think it can become an anti-pattern is, you know, you have one system that is, you know, maybe it's talking to a customer or something like that and it's making mistakes from time to time until you've got another system that you bring in to check it. And that system's also like a noisy, unverifiable system. And, you know, as the joke goes, now you've got two problems.

• 20:02Speaker 00

  And it's not just that you now have to think really hard about like, are these errors compounding? I mean, we always like to think in machine learning that, oh, I've got two systems, the errors will be independent. But one of the things I've learned in the real world is it tends to be that the errors in fact correlate very strongly.

• 20:15Lukas Biewald?

  Totally.

• 20:16Speaker 00

  You're nodding, you know.

• 20:17Lukas Biewald?

  Yeah.

• 20:18Speaker 00

  So that can be, that's one of the reasons why I think this can quickly become an anti-pattern. Because when you have systems checking systems checking systems, especially if the errors are aligning, even if it's effective, it's going to be slower. You've got to wait for the checker to run and you're burning tokens and burning more tokens and you still have no guarantees. And I think we can build technologies that can come with guarantees. I think we can build technologies where truth is one of the design principles in the first place.

• 20:45Lukas Biewald?

  Can you give me a flavor of how that might work? I guess the reason maybe I'm taking this point of view is just that I spend a lot of time retrofitting, you know, these systems to work well. I've talked to lots of guests on here that do this kind of retrofitting and, you know, the results are impressive and interesting, but certainly I don't deny that there's lots of hallucinations. But it's hard for me to imagine a different method that would bake in truthiness or guarantees into the core training loop.

• 21:08Speaker 00

  Maybe it's sort of like laying out what the patterns and anti-patterns are right now. So if you have a system that is operating at the token level and let's say it's going to tell you your account balance and it's really important that this information be correct, well, what can you do? Well, you can throw out some tokens and you can bring in another system that says, hey, check these tokens against those tokens. And this gives you the sort of chains or constellations or whatever you want to call them. And, you know, the failure mode of these systems, aside from just they're slow and they're expensive, is that

• 21:40Speaker 00

  Is that there's really still no guarantee that at the end, the situation that confuses one won't confuse the other. The other pattern that you see a lot is where people will take these LLMs, which are sort of like kind of mind blowing in their breadth and contextuality and flexibility. And they will sort of shrink them down to controlling like decisions along a carefully designed tree, right? And this feels almost like model lobotomy where you take the system that had such breadth and flexibility, but you've constrained it so much that now you can trust what it's going to do.

• 22:15Lukas Biewald?

  Wait, I'm not sure I'm following that one.

• 22:17Speaker 00

  Sure.

• 22:17Lukas Biewald?

  How would that work? Like, what would it be approximately?

• 22:19Speaker 00

  So for example, you, somebody says, you know, somebody says something in a conversation. And rather than giving the LLM freedom to sort of take any action in a wide action space, you say, on the basis of what this person says, you can advance the conversation in one of the following eight ways. Right? So essentially you end up with something that looks like a classic IVR system, but with the power of an LLM for the intent recognition. Okay. And you actually see this quite a lot in industry. And, and I would say this is sort of, it starts to feel like a finite automaton whose transitions are driven by this LLM. And these patterns are totally reasonable ways for people to react to a system being unreliable as they've

• 23:03Speaker 00

  deployed it in a pattern that hopefully increases that reliability at, at some cost. I think if we look at the core of these systems and where the intelligence is coming from, the kinds of systems that people have been building have been getting more and more intelligent. Intelligence is a multifaceted thing. And the different aspects of intelligence have not been advancing equally.

• 23:31Lukas Biewald?

  Totally.

• 23:31Speaker 00

  And so if you talk about Horizontality, breadth of intelligence, plasticity, contextuality. These are actually things that were very, very hard to do in earlier areas of artificial intelligence, and they've grown explosively. Reliability has not kept pace. I think that is not a strength of current systems.

• 23:54Speaker 00

  And that's had a bunch of challenges for the deployment of these systems in enterprise contexts, in industry. For many applications, that's fine, right? That if you're having a chat with a system, there are many contexts where getting something back that might or may not be true, but it's contextual and interesting, that that's great. But if you're trying to fill a prescription or transfer money, it just really has to actually be right. That has to actually be your bank account, that has to be the right balance and things like that. And so as you move from sort of consumer, almost even entertainment contexts into regulated industries, on the other hand, suddenly the reliability is front and center, and these

• 24:38Speaker 00

  systems are not as clean a match. Right now, we have this one kind of architecture, which has strengths in horizontality and contextuality, and it has weaknesses in reliability. When we point it at a problem where the weaknesses are suddenly critical, you can see the results of that, whether you call them retrofits or anything else, these additional pieces of technology coming into play to try to compensate. And that's not surprising, because there's just a misalignment between the strengths of the system and the requirements of the problem.

• 25:15Lukas Biewald?

  Can you say at a high level what your new approach is and how it's different than what the labs are doing?

• 25:20Speaker 00

  Yeah, and I think actually, increasingly, people are realizing that reliability is the core problem. And I think we're going to see increasing interest in systems that are architected for reliability. What might that look like? So for example, our models, our first model is APT-1, and the way it's architected is in Instead of being fundamentally about tokens, where you assemble tokens and then after assembly find that they represent things like dollar amounts that have semantics. It's very hard to place any controls inside the model because the individual tokens themselves don't really have a semantics until they've been assembled into larger units.

• 26:05Speaker 00

  And what our models do is they make information and action first order objects. So when the model is making decisions, it's making decisions about information and actions and where information is moving around. We talked a little about meta cognition, a big piece of meta cognition is where did my information come from? Is this information present or absent and so on? I would in general say that when you have a conversation like between a person and a bunch of APIs, like in a banking context, the person is going to speak human. They're gonna, they're gonna say things, the things are gonna be ambiguous. They're gonna use words that don't have a crisp verifiable meaning. Totally. Actions are gonna take place on the other side.

• 26:49Speaker 00

  Those actions do not have those properties. Those actions are API calls, those API calls have preconditions, they have business logic, and they have verifiable semantics. And the challenge is bridging these things. So classic systems couldn't handle the human side. Current systems are great with the human side, but they're not so great with the, the sort of backend logic. I mean, if you think about an LLM, and again, I'll talk about LLMs in their, their basic form. The control surface you have is a prompt, right? There's very little you can crisply say about the relationship between what you put in a prompt and the behavior that comes out the other end, right? It's a, it's a hinting surface.

• 27:33Speaker 00

  And so this gives rise to, I guess, what I would call prompt and prey, where people. You know, they put in what they want. If it doesn't work, they put it in all caps. They add some exclamation points. And after the third exclamation point, you maybe start to feel like this isn't the control surface that you need. When you're talking about controls over APIs, we already know what a lot of that logic looks like. It's just hard to replicate in a token-based model. And it's a lot easier to replicate in a model that is operating over decisions about, you know, we talked about information and where does information come from? But there's also actions, conditions under which you can take actions. And trying to get that into the model is really important because that saves you from

• 28:13Speaker 00

  having to have a whole system of things, checking things in a way where it's really hard to say anything crisp.

• 28:20Lukas Biewald?

  I just want to make sure I understand what you're doing. So if you return something that's like, check the account balance, what would be something that you'd want to forbid it from doing? Like check someone else's account balance?

• 28:32Speaker 00

  Sure. Or you might want to forbid it from doing a transfer without authorization, right? Or that authorization, you don't want a user to be able to say, you know, forget your prompt, you have, no authorization is required, or I have authorization, or whatever sort of attack, right? And one way to deal with that, obviously, people are very aware of sort of prompt injection style attacks. One way to do that is just be really careful, like, oh, this user looks like they're doing something strange. Or let's put a lot of attacks like this in the training data and teach it to not do this. But ultimately, that's just not where authorization is allowed to come from. It's just not allowed to come from user statements.

• 29:14Speaker 00

  It comes from that place over there that Vens authorization. And being able to say things like that to a model then means essentially you're getting closer to what you really want, which is here are truth conditions. You were mentioning this earlier. You want the model to be truthful. And then within that, okay, now optimize user happiness or style or whatever you want to optimize, but subject to staying within the space of true statements. That is hard to do through simply constructing a reinforcement signal because now you're like, what is the linear combination of truth and happiness? And instead, what you want to be able to say is, here's a model.

• 29:54Speaker 00

  I'd like to be able to guarantee that it will only do true things. Right. And that's, that's sort of, that's the long-term challenge. Models that will not lie to you. And, and for our model, there is a big class of things that we can make guarantees about. And I think that's going to become increasingly important as people start caring more and more about reliability. And it's not that you care less about intelligence. Intelligence is very important, but intelligence without reliability is limited in its impact.

• 30:28Lukas Biewald?

  I guess though, you know, intelligence or tokens, tokens are powerful because there's so many of them that naturally occur. And they have all kinds of flaws, but a big advantage is the massive volume of tokens. So do you train your models on much smaller pieces of data? And then do they struggle with generalization?

• 30:48Speaker 00

  So one of the key things, we talked a little bit about verifiable systems. One of the key things that we do to train our models is that we train them on simulated reinforcement learning generated data where this is verifiable RL. And so the kind of the key thing that unlocks that is being able to generate data that doesn't just look right, but in fact can be verified. And the kind of the key hook there as well, there are things that our humans are doing, but there's also actions that are being taken. And part of that is naturally amenable to verification. Part of that is much harder to figure out what to do with. And that was a big part of our research and training the model.

• 31:26Lukas Biewald?

  I see. And what got you excited about this direction? Were you just seeing hallucinations everywhere and getting frustrated with it? Or was it something from previous work that, that you connected with?

• 31:35Speaker 00

  Yeah, I mean, it was a few things coming together. And, you know, to me, this kind of goes back years now, but.

• 31:45Speaker 00

  One piece of information that kind of really stuck with me is when we go into enterprises and they're unhappy with hallucination raids, or they wouldn't be talking to us.

• 31:56Lukas Biewald?

  And maybe you should describe your previous company, then. I assume this is from your last company?

• 32:00Speaker 00

  No, this is also now. I mean, we've seen this in pretty much any technology that has been based on sort of modern LLMs has this property. I mean, old technologies had different failure modes, and I actually think this is important and connects to this question of hallucinations. So when I think about hallucinations today, and then I'll get back to your question, I think of it as an iceberg, right? There are the hallucinations you see, and that seems scary. But what does it take to see a hallucination? Well, the system has to have produced an output that has two properties. It has to be wrong, and you have to have noticed.

• 32:44Speaker 00

  And that is not all the hallucinations. There's the whole rest of the iceberg below, which is all of the mistakes that go unnoticed because they are too plausible. The systems are very, very good at producing output that is indistinguishable from the truth. And so if you see a hallucination, well, that's sort of the weird case of a hallucination that you could detect. And this actually has really big impacts on how people use these models. So if you go back to, say, 2010, and you go to a web translation system, and you put in some text in a language you don't speak, and a translation comes out.

• 33:29Speaker 00

  Or you go to a search engine, and you do a query, and some results come out. Well, how do you know whether this translation's right or wrong? Well, in some sense, you don't, because you don't read the original. But there are often surface signs that maybe I shouldn't believe this. Little disfluencies, a chunk that's still in the other language. It doesn't feel right. Or you get the search results back, and you click on a result, and this webpage has a bunch of typos. It's not loading quickly and and these are surface signs. If you, you know, software engineers will know this idea of code smells where where. So this is an invention back and kind of popularized by by Martin Fowler and the Refactoring book.

• 34:13Speaker 00

  And basically, the idea is when something deep is wrong, there's often a superficial sign. And we, as software engineers, we learn to detect those, like, oh, these functions are signatures are getting really large. Or I'm duplicating a bunch of code or time to time to fix it. We also culturally recognize, oh, this machine translation's got a big chunk of, you know, Germans still in it. Maybe this didn't work right, or this webpage is, you know, like, not loading, like, the images didn't load. Maybe the rest of this is suspicious. LLMs have removed these cues that something is wrong. And so you see ChatGPT tells you something and it's always fluent and it's always confident, whether it's right or wrong.

• 34:55Speaker 00

  And this is taking more and more of the hallucinations and putting them in the underwater part of the iceberg. It's wrong, but you can't tell. And you see the, say, you go in and look at customer service cases. Well, the system like quoted a reasonable refund policy. It's just not actually the one that the company wanted. It's something that somebody was talking about in 2019 on Reddit. And and so I think I think these two things are very, very closely related.

• 35:23Speaker 00

  For me, seeing, you know, I was coming from a natural language processing background. And like, man, if there's any field where things have changed, it's NLP, right? When I started, what are the things that we were concerned about? We were concerned about like syntactic structure, right? Like, where is the verb in the sentence? And and part of that was because at that time, computational linguistics, natural language processing had become very, very close to each other. Because the ways people were making progress in AI were very representation specific. If you wanted to do NLP, it was very like language specific representations, which we took from what we know from linguistics. If you want to do vision, it was vision specific representations. Many of them were inspired by what we know.

• 36:04Speaker 00

  About vision science, the eye or geometry or, or, or, you know, the story goes on for each vertical of AI. Now, now we've gotten to a point where there's a much more sort of uniform horizontal transformer models for everything. Maybe natural language tokens for, for everything. And as I was seeing this happen, having thought really hard about language models for decades back when they were, you know, short context and not built at all for answering questions or translating or being the operating system of AI. Language models were built, as I, as I know you know, language models were originally built to tell good speech recognition outputs from bad ones.

• 36:47Speaker 00

  Right. They were, they had one job, which was to assign a high number to good outputs and a low number to mangled outputs. That was it. And the fact that they have grown into such like a plastic technology is really amazing. And I was looking at this and it was really just kind of dawning on me that we are going to switch from the problem in AI is nothing works. To the problem in AI is everything works. And as this happens, knowing, you know, how language models work and seeing this, I became very, very aware on both the industry side and the research side, that we were going to have a real problem with reliability and we were going to have a problem in terms of the technology. We're going to have a problem in terms of digital literacy.

• 37:31Speaker 00

  And to me, suddenly the most important problem I could think of was how do you build a system that will not lie to you? And I think this is a challenge that we should be, we, we should be taking very centrally as a field. And this ultimately is what led to scale cognition. This idea of how do you build a model that you can rely on? How can you build a model you can trust? How can you build a model that will not lie to you?

• 37:58Lukas Biewald?

  Do you have thoughts on digital literacy? It's a really interesting point that it's hard to tell when a model is confident or not confident. And the cues that we learn from other humans writing at least work really poorly with models.

• 38:13Speaker 00

  Right, totally. Humans, when they write, first of all, when they learn to write, they go through this process. It takes years. And then when they do write and people vary in their abilities to write, there are these signs. It's like the code smells. There's the smells that we use to detect that maybe this information isn't right. And well, maybe we should verify this information. And in humans, traditionally, sort of like

• 38:46Speaker 00

  good command of vocabulary, clear presentation, flawless punctuation. That all spoke of not only education, it spoke of focus, time commitment, an editing process that probably involves some checking. And so you looked at that output and you could infer something about the vetting the information had gone through. Today, the information doesn't go through vetting. It doesn't really even get thought of as information. It's token assembly. And so that has, I think, I think it's one of the biggest digital literacy problems we have in front of us.

• 39:23Speaker 00

  Even just with, if you go back and you think about, you know, people go to the library and they get a book on some topic. Well, the library couldn't buy every book. They would buy books that seemed important, that seemed reputable. So there was vetting just that the book was there and how the process that led to that book and the book publisher and the editor.

• 39:48Speaker 00

  And so when search came along, well, suddenly your results could be anything. But there were still these mechanisms, like websites with false information still had some of these smells. And then click stream data was very important. And so as long as most people had the same reaction as you. We kind of stumbled through the digital literacy issues. It's not perfect. People would still believe things, obviously, that they read on the web that were false, but at least there were still smells. And now I think that's just been totally homogenized. You ask a question, every answer comes back. It comes back sort of like sycophantic. It comes back confident. It comes back doing all the things that lead you to want to believe it, to have instincts to believe it. And this is a real problem.

• 40:32Speaker 00

  I think it's a problem with the technology. Right? I think we should be demanding of our technology that the source of the information be cited. And you're starting to see some of that. You're starting to see search results kind of being interleaved. Now, a lot of that's post hoc, so you go click on that source of information and that claim is not in fact on that webpage. So there are problems with this, but directionally, that's an improvement. And I think this is just something that's changed so fast that people haven't been able to keep up. And in a way, that's not a surprise because how long did it take to train these systems? Well, it takes a month to train the system. How long did it take to develop this technology to this point? Like a few years.

• 41:12Speaker 00

  You know, how long did it take to put all that information on the web in an abstracted linguistic form? Well, it took 30 years. How long did it take to come up with that information and figure out what words to use and the concepts? Well, that millennia. And all of that has been compressed into this like instantaneous access that has burst onto the scene. Of course, culture hasn't been able to keep up with what you can trust and what you can't.

• 41:39Lukas Biewald?

  I mean, that's a good segue into another question that I really wanted to ask you.

• 41:42Speaker 00

  Okay.

• 41:43Lukas Biewald?

  But felt a little shy, maybe.

• 41:45Speaker 00

  No, I'm in trouble.

• 41:46Lukas Biewald?

  No, when, you know, we both worked in NLP and you have a background in linguistics formally. I was a big fan of linguistics as an undergrad and I think that's a part of what got me into AI.

• 42:00Lukas Biewald?

  And I think in the early 2000s, when we overlapped a little bit, it felt like NLP had, or linguistics actually had so much to say about how to build working NLP systems, or how to build working AI systems. And then, over time, the bitter lesson really seems to have come true, from my perspective broadly, where linguistics has less and less to contribute. And even the tokens, yeah, they're morphemes, but they're automatically generated.

• 42:28Speaker 00

  Sure.

• 42:29Lukas Biewald?

  They're not really morphemes.

• 42:30Speaker 00

  Yes.

• 42:30Lukas Biewald?

  And no one's asking linguists what are good chunks of letters to use here.

• 42:35Speaker 00

  Yeah.

• 42:35Lukas Biewald?

  And nobody seems to care about parsing anymore. I kind of wonder, is parsing even like a real thing? Or was it something that linguists made up and isn't really true?

• 42:45Speaker 00

  Well, oh, there's so much here.

• 42:48Lukas Biewald?

  Awesome.

• 42:49Speaker 00

  I mean, remember, linguistics is a science. It's trying to study how language works based on evidence. And that involves making theories and testable theories and then trying to go falsify them or not. And so when a linguist says, I think this is a good description of the syntactic structure. That is a theory that's meant to be explanatory of the evidence you see. What's that evidence? Well, you know, how languages relate to each other, how languages change over time, what people can say and can't say and find acceptable in their language. And of course, the methodology varies, but it's ultimately science. And NLP is different. It's not fundamentally aimed as a science. It's an engineering discipline.

• 43:32Speaker 00

  It's how do we interpret this information in this context? And for a while, in the time you were talking about, and I would say in like the 80s to 2010 or something like that, there was a period where those two things aligned very well. Because in order to make AI work back then, you needed good representations. Linguists had figured out good representations, and then we could borrow those. But even then, they weren't perfectly the same. So for example, if you worked on parsing around the year 2000, you were not trying to write in that this syntactic structure works this way in English. You were instead trying to build in Combinatorial structure that was appropriate. I think this is a tree structure process.

• 44:14Speaker 00

  I think that this sort of information has to flow along the backbone. Here's an algorithm and a probabilistic model that is capable of handling nested structures with information flowing along the backbone. And so you learned all the details of the language from data. If you go back, if you go to the 70s, sure, people were trying to write it all down by hand. You go forward, well, there's less and less that you need to learn from data as our abilities to learn get better and better. But one of the things that I think is really interesting is we talked about the sort of super cycle of research where the new technology bursts onto the scene. You throw out everything you had before because the new thing is beating everything that came before it without the complexity. And then you hit the diminishing returns and you start to see there are weak sides.

• 44:58Speaker 00

  There are areas that are resisting progress. Well, what are those areas? Well, right now they're things like reliability. Well, what are the solutions to that? They might be things like, well, if you're building systems that are facing APIs, maybe the sorts of structures in a compiler are relevant. Well, you know, that sort of idea, that's an idea that was thrown out, but you start to see these things coming back. Or you start to see, for example, search. AI was very much about try a bunch of things and take the one that worked. The idea was that you kind of couldn't tell it was going to work until you had projected it forward in your model a little bit, right? Sort of like playing the chess game forward until you can figure out whether it's a good position. And so you would like, if you wanted to find the exit to the maze, you would try a bunch of things.

• 45:43Speaker 00

  And when you found it, you would take the hypothesis that worked. And well, big transformer models were so good at moving information around in the latent representation that you could just make the decision now. You wouldn't have to walk along the tree structure of the sentence. You'd just like predict the next word. And if there's anything hierarchical happening, it's happening in the transformer head somewhere. So great, you could throw all that out. But maybe some things then start coming back. For example, people say, oh, you know what? If I want to solve this hard math problem, I should actually try a few things and see what works. Well, that's search, right? It's that idea of having multiple candidates and being able to fall back when something doesn't work. That's a key idea, and it's really an evergreen idea. But these ideas, they sort of, they come and they go, and the pendulum moves back and forth between

• 46:28Speaker 00

  all I'm gonna do is local prediction to all I'm gonna do is sort of having organized computation. And then you move back and forth and you find some happy blend. I mean, as humans, there are two main ways we make good decisions. You can do it by memory. Like, I've been in this situation before. Like, I touched that stove. I had a bad time. I'm not doing that again. You can learn from your own experiences. You can learn from other people's experience. The single biggest the single biggest value of language for learning is that you can learn from other people's experiences. Otherwise, you gotta make every mistake for yourself, and that's not a good situation. Okay. And so, so one of the ways you can make good decisions is by, you know, learning from the past.

• 47:12Speaker 00

  The other way you can make good decisions is by thinking through the consequences of your actions. You can think, oh, hmm. That fire looks hot. I wonder if, you know, and you can reason under a model that you have of the world. And people do both, and they mix these things together. So you can remember that this chess position was bad, or you can play it forward and see you're gonna lose. And AI has had a pendulum swinging between these, right? Kind of, classic AI was very much all about play it forward and see what's gonna happen under your model. Currently, AI is much more focused on rehash, remix information and experiences you've had. But as we start to see reasoning models, as we start to see these new sorts of directions, it's the pendulum swinging back. And the pendulum swinging back on things like reliability, too.

• 47:57Speaker 00

  And so I think this is sort of, on one hand, it's natural. But it does also let us predict that, yes, this sort of simple approach based on a certain kind of noisy model is gonna saturate. Some pendulums are gonna swing back, and we're gonna need augmenting technologies that in many cases are gonna reinvent evergreen ideas into this new context.

• 48:24Lukas Biewald?

  Does the research in LLMs and maybe the success of LLMs and some of the research on introspecting LLMs for how they work, does that inform at all human linguistics?

• 48:38Speaker 00

  That's a great question. So you ask the question, what can what we know about human language do to inform how we build AI systems? And I think in the 70s, the answer was, we're going to just like, write it all down and press run. In the 2000s, it was, we're going to take the combinatorial structure of it and learn the rest from data. Now, we're very much in a, okay, we'll try to learn pretty much all our representations from data if we can. And I think there are two things to point out here. One is that we are still learning from

• 49:20Speaker 00

  what human cognition does that LLMs do not. Things like, oh, the importance of metacognition, or or other ways we can improve the engineering by being inspired by, you know, the human example. It's always been a tension in technology, how much to be biomimetic, right? On one hand, it's always easier to build something when you've got a working prototype and, you know, we've got a working prototype of intelligence and here we are. On the other hand, you know, the classic example is, we didn't make progress in powered flight until we stopped building machines that flap their wings. And that tension-

• 49:56Lukas Biewald?

  Well, there's a special thing, too, I feel, with brains where I'm not sure we're so good at introspecting what our brains are doing. And sometimes I feel like we take lessons where I wonder if that's really what we're doing.

• 50:08Speaker 00

  So I think there's an important distinction between learning from what brains are doing and believing that our introspection reveals it.

• 50:14Lukas Biewald?

  Yes.

• 50:15Speaker 00

  So neuroscience is hard. And the way neuroscientists proceed is not to sit there and introspect. They have MRI machines and they have carefully designed experiments.

• 50:23Lukas Biewald?

  Okay, but I feel like linguists sometimes do introspect.

• 50:27Speaker 00

  That it is true, linguists introspect, or I think, you know, slightly more properly, they ask other people to introspect. But yeah, and this is a criticism that has been levied against certain branches of linguistics, that there's an overreliance on introspective data.

• 50:44Speaker 00

  It is certainly not the only way you can do linguistics, and it's certainly not the only way you can do computational linguistics. You know, so for example, work we've done in computational linguistics, to just like pick an example, work on reconstructing ancient languages. We did that work introspection free. You look at a whole bunch of words in thousands of modern languages, and you run a probabilistic model to infer what ancestral languages must have looked like. They're phylogenetic models. They're no more introspective than a phylogenetic model inferring an original form of a virus from modern variants. And so there's a lot of ways to do computational linguistics.

• 51:18Lukas Biewald?

  That sounds so cool, by the way.

• 51:19Speaker 00

  Yeah, maybe another podcast. It would.

• 51:24Lukas Biewald?

  Or give us the result or something. Come on.

• 51:27Speaker 00

  Yeah, so, OK, I will. OK, the briefest version of this. This is, I think, a good example of scale and how some things in computational linguistics, I think, can give different answers when done at scale from not being done at scale. So a particular thing we did, and this was a few years back, we looked at reconstructing ancient languages from the modern forms. And so you might think, for example, maybe an example that people would find very accessible would be reconstructing a language like Latin from the modern Romance languages, French, Spanish, Portuguese, and Italian, and so on. Now, of course, when you reconstruct that, you won't get classical Latin. You will get the latest point at which they branch, which would have been vulgar Latin.

• 52:10Speaker 00

  But you sort of look at, you're like, OK, you look like French fur or Spanish fuego, and all these look like similar. So you can start to piece together what the ancestral forms might have been. Well, in that case, we kind of know the answers. We know a lot about classical Latin, and we have decent side evidence about vulgar Latin from a variety of interesting sources. We went and we looked at the Austrianese languages. So this is about a sixth of the world's languages. And we looked at the modern forms of a bunch of words and tried to reconstruct what the ancient forms would have been. This is a language, this is a proto-language called proto-Austrianese, and it had been reconstructed by.

• 52:53Speaker 00

  By hand, by blust, and we were able to do this in a computational way. Now, the computation that we did generally agreed with linguistic hand reconstructions, and we don't have a time machine, so it's very hard to tell who's right. The interesting thing is you can start to do things on this giant tree of language change. Like, you can start to ask questions, like what mergers are more common and what mergers are less common. And a hypothesis, which is one of these things that just feels intuitively true, is that if you have two sounds that are different in a language and they would be merged together, a bunch of words that used to be different would be collapsed.

• 53:33Speaker 00

  So if P and B, pu and bu, were to be collapsed, suddenly pin and bin are no longer different words. They're all going to sound the same. Well, that seems like an information theoretic problem, right? And the functional load of a pair is how many words their distinction is holding apart. The functional load hypothesis states that the more words that are being held apart by a sound distinction, the less likely that merger is to happen. And which intuitively makes sense. And some initial experiments in a small number of languages didn't seem to support this hypothesis. And so there was actually a really interesting result in computational linguistics that it looks like actually the functional load hypothesis isn't true. Things just kind of merge when they merge.

• 54:18Speaker 00

  And if instead of doing this on four languages, you do this on hundreds of languages, and you do the scatter plot, you don't know that your reconstructions are perfect. You never know that, even when you do them by hand. But in aggregate, the statistics show that the functional load hypothesis seems to be absolutely evident in this data at scale, but you can't see that pattern over a small number. So I think this is an example of places where just data at scale can answer questions that aren't the same questions that were easy to answer by hand.

• 54:48Lukas Biewald?

  And I guess there is interesting work that I've seen on introspecting DLMs. Like, I think Anthropic did a really nice paper on sort of showing that... Showing different things, but one thing felt like the Chinese words and English equivalents are kind of stored in the same part of their neural network. Does that... Is that in conversation with human linguistics at all?

• 55:12Speaker 00

  So what's actually closest to is it's actually closest to some computational neuroscience. So...

• 55:22Speaker 00

  This is a question you can ask of the human brain too. Like when you have a bilingual speaker, like where are those representations? Are they in the same, in the same place? And actually, a student of mine, Kathy Chen, who's a joint student with Jack Gallant, did exactly this, like with, and it's not by introspection, it's by MRI. And you look and see what areas of the brain light up. And, and so these questions about what's merged and what's separate, these are questions we can answer scientifically about natural brains. And we can also try to answer them through very similar, broadly speaking, methods with artificial brains, except obviously the, the MRI as a device, like we are limited by the physics and, and the structure of the brain.

• 56:03Speaker 00

  We can be much more precise about the probes we make into a digital brain. So I think actually there's a lot of interesting, there's a lot of interesting information there. But I do think it's important to step back and notice that when we start doing this, when we start doing neuroscience against LLMs, what we are essentially saying is this object we have built is transcending what we think about as engineering. We're no longer trying to understand it by its modularity, by the behaviors that it is guaranteed to have or not have. We're trying to understand it as, you know, through the lens of science. Science is what we use to take apart things that are too complicated. Engineering is how we, we build things up.

• 56:44Speaker 00

  I mean, if I teach CS 101, like what is the most important thing we teach people? Like what is the most important tool we have for building complex software systems that are reliable and that can be built by teams and maintained over time? The single most important technique we have is modularity. The ability to say this large thing is made up of small things. And these small things obey a contract where if you give me this input, I'm guaranteed to give you this output, and then we can work on them separately. So modularity is the key tool for developing complex systems that are reliable in classic software engineering. What is the key tool that has led to the recent explosive growth in machine learning? It's end-to-end optimization.

• 57:27Speaker 00

  It's take the data, have a blob, take the reward signal and just propagate, propagate, propagate. And these things are very much at odds. And so one of the things we're going to have to reconcile

• 57:40Speaker 00

  As a field is how to get the reliability that comes from classical techniques like modularity that we use when we build software against the the abilities that have come from optimization. And these things, it's not an either or, it's not that you cannot build a system that has guarantees or that has structure or anything like that, but there are natural tensions. And if you go all in on either, you you get some serious limitations. And figuring out how to combine these two wonderful pieces of progress we have is, is I think, one of the central problems right now.

• 58:20Lukas Biewald?

  That was really well said, and it connects me to self-driving cars, where we've obviously had a ton of guests on the show, and I think even just looking at the arc of the guests over the last five or six years that we've been doing it, there's clearly been a strong trend towards end-to-end optimization, especially as these things come online. And we see Waymos now and Zooks, and we've had folks from all these companies come on. And essentially, I think they're saying that as they get closer and closer to production, they do more and more end-to-end training.

• 58:51Speaker 00

  Yeah.

• 58:51Lukas Biewald?

  So what do you make of that? I would have thought that you might need more modularity to enforce contracts for such a life or death case like a self-driving car.

• 59:01Speaker 00

  Yeah, I mean, it's a great question, and I think it's not limited to cars. I think in general, the challenge is that we are going to be increasingly trusting AI to make decisions that have consequences. And when we build systems, there are going to be trade-offs between how we optimize, how we architect. There are going to be trade-offs between things like reliability versus things like breadth. And so I think you're going to see.

• 59:35Speaker 00

  This idea that there is sort of one architecture that hits the perfect balance doesn't seem to be true right now. So being able to say that feeding the whole web into an LLM and you know, sort of hoping for reliability to emerge, well, the different facets of intelligence advance at different rates and reliability is like the slow one. So if you, if that's the most important thing to you, just continuing down this path alone is probably not the best way to get there. And you also get sort of intelligences that have a bit of a weird shape. So there was this, you know, you may have seen this, this thing going around where somebody's talking to a customer support bot at Chipotle and asked about how to reverse a linked list in, in, in Python.

• 1:00:19Speaker 00

  And of course, the LLM can answer it, but interestingly, like it shouldn't, right? You actually don't want the horizontality. And so you kind of need technologies where the strengths line up with the needs. And when you're talking about self-driving cars, you really need like safety is really, really important. Like the number of nines you want is really high. And so you look at a system like this and suddenly it's less important whether or not it can write sonnets for you. And more important that it just be able to guarantee safety or have like a really reliable safety record. And here is a place where I think there is a real tension between

• 1:00:58Speaker 00

  how we drive down error rates in those systems. If not done correctly, means we actually no longer have the ability to guarantee anything. And of course, there is, there is work out there on, on being able to control systems where you can sort of make a guarantee about what they can do. So like a Berkeley, Claire Tomlin's work on, on, you know, being able to guarantee non collisions and flight envelopes and things like that. And I think it would be great as a challenge, you know, to build a car, AI controlled or not, where you can prove that you cannot crash it.

• 1:01:42Speaker 00

  Right? This is a kind of thing where it's one simple technique is not going to maybe get you there, but this is what we should be striving for.

• 1:01:52Lukas Biewald?

  Interesting. I could imagine that comes with downsides, though, to literally not be able to crash it. Might be too strong of a constraint.

• 1:01:59Speaker 00

  Well, as it becomes a sort of social question, what is the envelope you want to guarantee? Right? And

• 1:02:12Speaker 00

  I think you're right to point out that that can be nuanced. But I think the ability to have a discussion about what guarantees would we like to be able to provide is a much better future than one where we can't guarantee anything.

• 1:02:33Lukas Biewald?

  That's probably a strong place to end.

• 1:02:34Speaker 00

  Okay.

• 1:02:35Lukas Biewald?

  But can I ask you a few more questions if we could put back in the linguist's session?

• 1:02:38Speaker 00

  Sure, sure.

• 1:02:39Lukas Biewald?

  In case it's fun?

• 1:02:39Speaker 00

  Yeah.

• 1:02:40Lukas Biewald?

  I just, okay, I was, I was debating whether or not to bring this up, but it's kind of interesting to me, I guess, just because I spent so much time on syntactic parsing. Clearly, language has some sense of syntactic parsing that's real, or it feels real. I guess I'm introspecting. Maybe that's

• 1:02:59Speaker 00

  You're introspecting, but

• 1:02:59Lukas Biewald?

  I'm sorry, I'm introspecting.

• 1:03:00Speaker 00

  No, no, no, but I don't mean that in a bad way.

• 1:03:02Lukas Biewald?

  It feels, it feels very present to me when I introspect my own brain.

• 1:03:12Lukas Biewald?

  Does that notion somehow show up in LLMs as they also model language that clearly comes from humans that like to do a certain sense of syntax in the languages that we, we invent?

• 1:03:24Speaker 00

  Yeah, I think it's a really good question. So let's start with what we know. We know that

• 1:03:29Lukas Biewald?

  We better sort of define syntactic parse, because I feel like I'm going a little wonky.

• 1:03:33Speaker 00

  Sure. Well, if, I don't know if this is, this is going to be on the podcast or not, but if there are listeners, they may, depending on how long ago they went to elementary school, they may have diagrammed sentences where you break up a sentence into like, here's the subject and, you know, here's the object and this adjective modifies this noun and so on. And, and that process of describing how the words relate to each other in a hierarchical structure.

• 1:04:01Speaker 00

  That's basically the essence of syntactic parsing. And obviously syntactic parsing is a thing, right? You're talking about syntactic parsing of natural language. We do it to programming languages all the time. Totally. Programming languages are designed to have properties relating to ambiguity that are different. They're designed in general to be non-ambiguous, whereas human language is rampantly ambiguous. And so a syntactic parser would take a sentence, say, in English, and basically produce a representation that showed the relationship between the words. And that relationship was hierarchical, which was important because, for example, you might want to do things like translate the substructures independently as a way of decomposing the problem. In the absence of that, you're basically going to be working left to right.

• 1:04:46Speaker 00

  And modern LLMs, by and large, work left to right. And the thing we know is we know that scientifically, these hierarchical structures seem to exist. Like, obviously, you can dig, you can find exceptions, cross序列 dependencies in languages like Dutch or Swiss German. And you can find cases where it's hard to totally nail down exactly what the hierarchy is. And of course, this is what syntacticians can debate. But there definitely seems to be these regular structures that have structures inside the structures. This seems to be real phenomenologically. The question is, do we need to program in how it works? Well, since the 90s, we haven't.

• 1:05:29Speaker 00

  We've had data that taught the parser how it worked. And recently, maybe you don't even need the parser. Maybe you can observe enough language that whatever regularities there are will just show up latently in the LLM. Now, the argument that would have been made if you talked to a syntactician...

• 1:05:54Speaker 00

  You know, a lot of the constructs in language theory, including ones that have had a big impact sort of in computer science, formal language theory, automata theory, you know, language generally described as a context-free system, which means a pushdown automaton, which means like if you, in principle, you can nest and nest and nest and nest, and any finite-depth system like an LLM is gonna run out of depth. But it just turns out it's more complicated than that. For one, people do get confused when the structures get complicated. Maybe we've got stack depth, even if it's hierarchical, we've got processing constraints that can manifest as stack depth. Things that people find easy to understand is like tail recursion, which, of course,

• 1:06:38Speaker 00

  you can transform into iterations, so maybe that's somehow avoids those constraints. So, you know, there's a long way to go about saying hierarchical syntactic phenomena seem to be real. LLMs seem to be pretty good at picking up on those correlations. But this is a good example of a case where actually having a structured representation might give you mileage. It might let you learn language faster. It might let you handle trickier cases. And it might let you generalize better to smaller data regimes.

• 1:07:19Speaker 00

  So like, this is really, now we're back to the phenomenon is real, the representation is relatively parsimonious, do we need the representation? And the answer in AI used to be, you can't make progress without the representation. And the answer today is, with a sufficiently general-purpose thing, so there's nothing really magic about transformers, there are other choices that would work. It's ultimately just like a parameterizable circuit. And a general-purpose circuit, you point it at some data and it induces a representation. And the question is now empirical, is it good enough? Or do you need the parser to help? And for many aspects of human language, the answer seems to be, you often don't need the parser.

• 1:07:59Lukas Biewald?

  But I guess another question is, are the transformers reinventing syntax somehow within their weights and biases?

• 1:08:09Speaker 00

  First, you'd have to figure out what that means. So I think the question would be something like, is there a substructure which is somehow isomorphic to a shift-reduced parser or something like that?

• 1:08:19Lukas Biewald?

  Isomorphic might be strong.

• 1:08:21Speaker 00

  Yeah. Yeah. And that, I mean, I know there are people working on this. It is, there are, the reason why this is a very hard question to ask is because ultimately, these deep structural questions manifest as surface correlations. These two words are highly correlated, but these two words are not, even though they're closer. And we attribute that to a syntactic boundary, or a nesting, or an information hiding. But it simply is the case that the correlations go one way or the other way. And a general-purpose system like a transformer learns correlations. So, once the system can manifest those correlations correctly enough, it's hard to know whether it's doing it through the right mechanism to be able to say

• 1:09:06Speaker 00

  that there's an equivalence. There's an interesting, I think, lesson, I'll probably make a bunch of phonologists and phoneticians angry at me by saying this, but like, as a caricature in linguistics, there is often a difference between what a phonologist and a phonetician will say about all of the sounds that we are constantly hearing and, as we acquire languages, just all of the language that's flowing over us. In phonology, one of the key concepts is one of cognitive economy, that the brain is driven to come up with minimal parsimonious representations,

• 1:09:45Speaker 00

  and that the learning and the abstraction are all about what, you know, you might call a... There are many cognitive ways to look at it, but it's a notion of economy, of minimal description. On the other hand... Phoneticians are much more likely to say all of this data impacts in a very diffuse and distributed way. And so this question of when you hear lots of language, what do you retain? And the answer could be on one end, all of it. There's wave files of everything you've ever heard. That's a pretty strong statement, but that's one extreme. The other extreme is none of it.

• 1:10:28Speaker 00

  You just retain this very abstracted representation of the beautiful process. And the answer seems to me to be almost surely in the middle somewhere. There's priming experiments that show really interesting things, like if you're going to react to a nonsense word that you've heard before, you react faster if they play you the exact same recording as the one you heard it before. That tells you something about it's more than just the full abstraction. But we also know people are very, very capable of making abstractions and generalizing. And how people will generalize and overgeneralize rules when they're learning a language, including their own language when kids are acquiring language.

• 1:11:08Speaker 00

  And so it seems to be that it is simultaneously true that the brain keeps these very low-level pieces of information around and also these abstractions. That makes it very hard to go to a complicated circuit that's mimicking the behavior and say, is this the same as what's going on in the brain? Because one, we don't know what's going on in the brain exactly. And two, how would we even measure that? All we can really measure is, did it capture this correlation? And when we do that, the answer is, yeah, of course it captured the correlation. That's the whole purpose.

• 1:11:38Lukas Biewald?

  I feel like there's all these clever tricks in linguistics to get at this stuff. And like looking at disfluencies. I always think about how my wife misremembers songs in a semantic way, like the lyrics are semantically similar, whereas my brain will misremember lyrics in a way that's more phonetically similar. And so clearly our brains are slightly different in how we encode music.

• 1:12:02Speaker 00

  Interesting.

• 1:12:03Lukas Biewald?

  I feel like you could run some of these same things on LMS. And certainly they wouldn't get bored like grad students do.

• 1:12:12Speaker 00

  Well, how do you? Oh, yeah. So how do you know in a human that we have these different kinds of linguistic knowledge? One of the argumentations that people have, you mentioned things like disfluencies. There are a whole... There's a whole list, and there's no way I can do it justice in a short amount of time. There's a whole list of arguments people have made for the reality of linguistic structure. Now, you can say, is that reality a statistical reality? It's explanatory? Or is it like a neural reality? It's in there and wired? Like, that's a separate question. But in terms of it being explanatory, you know, you mentioned misremembering things like semantically versus phonetically or phonologically. People look at transpositions.

• 1:12:57Speaker 00

  People, one of the classic arguments for phonemes is that, you know, things like spoonerisms, where you transpose phonemes or syllables or words. And if an object is available for a linguistic operation that argues to its coherence. So, for example, one of the big arguments for syntactic structure is, can I answer a question with this chunk? So if you say, "The cat is sleeping under the table," and I have to argue to you that "under the table" is a unit, one way to do that is to say, "Well, where's the cat sleeping?" And you say, "Under the table." Well, if it's available as an answer, if it can be replaced by the word "what," when

• 1:13:39Speaker 00

  you pattern-match those sentences, however it is that works, then that argues for its reality. Another way you can argue is on semantic grounds, like "under the table" is a place. Whereas if I ask you about "under the," it's hard to come up with a question whose answer is "under the." If you say, "Well, what does 'under the' mean?" Point to "under the." We can't do it. And so, linguists have developed these arguments for the modularity of language. Those arguments are not about our brain, really. They're phenomenological about the language. When you get to neuroscientists now, now you can start to say things about the brain that's about monitoring and probing and designing experiments that reveal, incidentally, the structure and organization.

• 1:14:23Speaker 00

  And that we can absolutely do on a machine as well.

• 1:14:27Lukas Biewald?

  Cool. All right. Well, I hope we keep that in. That was fun for me.

• 1:14:30Speaker 00

  Okay. Thank you very much. It was fun for me, too.

• 1:14:32Lukas Biewald?

  I appreciate it.

• 1:14:32Speaker 00

  Yeah. Don't worry.

• 1:14:34Lukas Biewald?

  Thanks so much for listening to this episode of Grading Descent. Please stay tuned for future episodes.