How Do AI Models Actually Think? - Laura Ruis

Laura Ruis, a PhD student at University College London and researcher at Cohere, explains her groundbreaking research into how large language models (LLMs) perform reasoning tasks, the fundamental mechanisms underlying LLM reasoning capabilities, and whether these models primarily rely on retrieval or develop procedural knowledge. SPONSOR MESSAGES: *** CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. https://centml.ai/pricing/ Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on o-series style reasoning and AGI. Are you interested in working on reasoning, or getting involved in their events? Goto https://tufalabs.ai/ *** TOC 1. LLM Foundations and Learning 1.1 Scale and Learning in Language Models [00:00:00] 1.2 Procedural Knowledge vs Fact Retrieval [00:03:40] 1.3 Influence Functions and Model Analysis [00:07:40] 1.4 Role of Code in LLM Reasoning [00:11:10] 1.5 Semantic Understanding and Physical Grounding [00:19:30] 2. Reasoning Architectures and Measurement 2.1 Measuring Understanding and Reasoning in Language Models [00:23:10] 2.2 Formal vs Approximate Reasoning and Model Creativity [00:26:40] 2.3 Symbolic vs Subsymbolic Computation Debate [00:34:10] 2.4 Neural Network Architectures and Tensor Product Representations [00:40:50] 3. AI Agency and Risk Assessment 3.1 Agency and Goal-Directed Behavior in Language Models [00:45:10] 3.2 Defining and Measuring Agency in AI Systems [00:49:50] 3.3 Core Knowledge Systems and Agency Detection [00:54:40] 3.4 Language Models as Agent Models and Simulator Theory [01:03:20] 3.5 AI Safety and Societal Control Mechanisms [01:07:10] 3.6 Evolution of AI Capabilities and Emergent Risks [01:14:20] REFS: [00:01:10] Procedural Knowledge in Pretraining & LLM Reasoning Ruis et al., 2024 https://arxiv.org/abs/2411.12580 [00:03:50] EK-FAC Influence Functions in Large LMs Grosse et al., 2023 https://arxiv.org/abs/2308.03296 [00:13:05] Surfaces and Essences: Analogy as the Core of Cognition Hofstadter & Sander https://www.amazon.com/Surfaces-Essences-Analogy-Fuel-Thinking/dp/0465018475 [00:13:45] Wittgenstein on Language Games https://plato.stanford.edu/entries/wittgenstein/ [00:14:30] Montague Semantics for Natural Language https://plato.stanford.edu/entries/montague-semantics/ [00:19:35] The Chinese Room Argument David Cole https://plato.stanford.edu/entries/chinese-room/ [00:19:55] ARC: Abstraction and Reasoning Corpus François Chollet https://arxiv.org/abs/1911.01547 [00:24:20] Systematic Generalization in Neural Nets Lake & Baroni, 2023 https://www.nature.com/articles/s41586-023-06668-3 [00:27:40] Open-Endedness & Creativity in AI Tim Rocktäschel https://arxiv.org/html/2406.04268v1 [00:30:50] Fodor & Pylyshyn on Connectionism https://www.sciencedirect.com/science/article/abs/pii/0010027788900315 [00:31:30] Tensor Product Representations Smolensky, 1990 https://www.sciencedirect.com/science/article/abs/pii/000437029090007M [00:35:50] DreamCoder: Wake-Sleep Program Synthesis Kevin Ellis et al. https://courses.cs.washington.edu/courses/cse599j1/22sp/papers/dreamcoder.pdf [00:36:30] Compositional Generalization Benchmarks Ruis, Lake et al., 2022 https://arxiv.org/pdf/2202.10745 [00:40:30] RNNs & Tensor Products McCoy et al., 2018 https://arxiv.org/abs/1812.08718 [00:46:10] Formal Causal Definition of Agency Kenton et al. https://arxiv.org/pdf/2208.08345v2 [00:48:40] Agency in Language Models Sumers et al. https://arxiv.org/abs/2309.02427 [00:55:20] Heider & Simmel’s Moving Shapes Experiment https://www.nature.com/articles/s41598-024-65532-0 [01:00:40] Language Models as Agent Models Jacob Andreas, 2022 https://arxiv.org/abs/2212.01681 [01:13:35] Pragmatic Understanding in LLMs Ruis et al. https://arxiv.org/abs/2210.14986

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Tim Scarfe?

  If I understand correctly, you created queries that resemble reasoning and ones which resemble some kind of fact retrieval.

• 0:09Speaker 01

  Yeah. And after that paper, I was left with this question. Like, to what extent is the scale that makes these models get better at these tasks? To what extent is that driving the performance? Or how is it driving the performance? Is it just that the model is seeing more similar stuff and therefore can memorize more? Or is it really doing something more interesting and learning something sort of qualitatively different from more data or with more parameters?

• 0:33Tim Scarfe?

  If language models are doing something which is akin to approximate reasoning, what's the difference between that and formal reasoning?

• 0:40Speaker 01

  So I believe they're in very controlled setups. We have already shown that connections models can do formal reasoning. So I think empirically and theoretically, we have shown that they can do a form of systematistic or symbolic computation, although it's still limited. But the question with my most recent paper was, can it also learn to do something in that direction approximately from data and the wild? And I think it can. And my paper doesn't show that exactly. It just shows that it's doing something generalizable, that it can apply to many different questions. But intuitively, I think it would be possible.

• 1:13Tim Scarfe?

  I guess you would agree that agency could emerge, even if we're not explicitly trying to make it emerge.

• 1:20Speaker 01

  Yeah, I think that's the interesting case. So there's this definition from Zach Kenton from DeepMind. They also have a safety interest in agency. And a couple of years ago, they made this definition of agency that an agent is something that changes its policy when its actions affect the environment in a different way. You can kind of trivially make a system of LLMs in an environment or something where the environment is also an LLM, such that it adheres to this definition. So I think the important thing is, like, when does something like that emerge from something as simple as next token prediction? And that's kind of what I'm interested in.

• 1:59Speaker 02

  So Tufa Labs is a new AI research lab I'm starting in Zurich. In a way, it is a Swiss version of DeepSeek. And first, we want to investigate LLM systems and search methods applied to them similar to O1. And so we want to investigate, reverse engineer, and explore the techniques ourselves.

• 2:23Tim Scarfe?

  MLST is sponsored by SentML, which is the compute platform specifically optimized for AI workloads. They support all of the latest open source language models out of the box, like Llama, for example. You can just choose the pricing points, choose the model that you want. It spins up, it's elastic auto scale. You can pay on consumption essentially, or you can have a model which is always working, or it can be freeze drive when you're not using it. So what are you waiting for? Go to sentml.ai and sign up now. Laura, it's amazing to have you on MLST. Welcome.

• 2:57Speaker 01

  Thank you. Amazing to be here.

• 2:59Tim Scarfe?

  Can you tell us about yourself?

• 3:00Speaker 01

  Yeah, sure. So I'm Laura. I'm a PhD student at University College London, supervised by Tim Rocktash and Ed Krupfenstetter. And I'm also part of my co-here. And I'm broadly interested in understanding language and its relation to human cognition, and how we can evaluate that in artificial intelligence. To what extent can pillars of human intelligence also show up in artificial intelligence? Things like reasoning, both mathematical reasoning, social reasoning, that kind of stuff. And also specifically trying to understand how state-of-the-art models are doing what they're doing.

• 3:37Tim Scarfe?

  Very cool. I'm a huge fan of co-here, Ed and Tim.

• 3:40Speaker 01

  Nice.

• 3:40Tim Scarfe?

  So this is very cool.

• 3:42Speaker 01

  Me too.

• 3:43Tim Scarfe?

  Okay, so you've just written a paper. Now, there's this huge controversy. You know, I've been speaking with Subar Al, for example. You know, he calls LLMs approximate retrieval engines and O1 approximate reasoning engines. So he is saying they're doing a little bit of reasoning, whatever that means. But you've written this paper. It has generated loads of interest on socials. Procedural knowledge in pre-training drives reasoning in large language models. Give us the elevator pitch.

• 4:11Speaker 01

  Yeah, so I was doing evaluation in language models, trying to understand how they were doing social reasoning before. And we designed a benchmark and evaluated models on their social reasoning skills. And after that paper, I was left with this question, like, to what extent is the scale that makes these models get better at these tasks? To what extent is that driving the performance, or how is it driving the performance? Is it just that the model is seeing more similar stuff and therefore can memorize more and seems to be, uh, have more capabilities? Or is it really doing something more interesting and learning something sort of qualitatively different from more, uh, data or with more parameters?

• 4:51Speaker 01

  And of course, the way we, um, evaluate machine learning methods in the past is we separate test from train, but that's not possible anymore these days because models are just trained on everything. Test is in train now. So, um, we wanted to understand when, uh, language models are producing zero-shot reasoning traces. So for example, for simple arithmetic, um, the steps you, it, it can produce the steps to reach an answer. Is it kind of relying on having seen those exact steps before in training? Or is it doing something generalizable? Is it sort of taking the steps itself and getting to the answer? And that was like the motivation for this paper.

• 5:35Tim Scarfe?

  Very cool. So, um, you used influence functions to do this analysis. Can you explain what they are?

• 5:41Speaker 01

  Yeah, yeah. So I was very happy when I, uh, like stumbled across that tool because it's, uh, this, it's this method from, uh, robust statistics that tries to answer a counterfactual question about the model. So the question it tries to approximate is what if I have to take this pre-training document out of the data set and I retrain the entire model? How does the behavior change? How do the model parameters change? And with that, the log likelihood of completions? And that's what influence functions estimate, and that's the tool we use to, um, Determine how pre-training data determines reasoning steps by models.

• 6:21Tim Scarfe?

  Very cool. So if I understand correctly, you created queries that resemble reasoning and ones which resemble some kind of like, you know, fact retrieval.

• 6:31Speaker 01

  Yeah.

• 6:32Tim Scarfe?

  And you kind of compared what the influence functions did on those queries.

• 6:36Speaker 01

  Yeah, exactly. So we used this factual task as a sort of grounding because influence functions are very approximate. We don't actually retrain the model for every data point because that's going to be too expensive. So you wanna have some kind of idea that what you're finding actually makes sense intuitively and factual retrieval is a natural task for that because for those factual questions, the only way to answer them is to retrieve the relevant documents. And we compare this to the influence scores for the reasoning traces. And those tasks are simply like the zero-shot reasoning kind of prompts and the model generates

• 7:17Speaker 01

  generates the reasoning steps itself. So if the model were to be doing retrieval for those types of reasoning, it would really have to retrieve each reasoning step from the pre-training data because it outputs zero-shot itself, the reasoning traces. I don't give it any examples.

• 7:34Tim Scarfe?

  So if I understand correctly, like the way, the intuition behind your work is that when we are doing, you know, fact retrieval, it seems quite focused. So it's just going to a document and it's retrieving the fact. And when it's doing reasoning, it seems very diffused. It's looking at loads and loads of documents that have reasoning-like processes in them.

• 7:55Speaker 01

  Yeah. Yeah, that's sort of the abstraction you can take from it. But of course, in reality, even when it's doing factual retrieval, there's much more going on because it needs to adhere to syntax. There's all kinds of like stylistic elements that are going on. But still importantly, like, I think the most striking finding from this paper to me was that for factual retrieving. Retrieval, whether a document is influential for a factual question, is not predictive of its influence for another factual question. So they rely on very distinct sets of documents. Whereas for a reasoning question, if it underlies the same task, if it's both, for example, calculating the slope between numbers, but for completely different numbers,

• 8:40Speaker 01

  the influence over the documents is very similar. So the same documents can influence these questions in the same way. And that we didn't see for factual retrieval, and that is really the basis for why we call this procedural knowledge.

• 8:53Tim Scarfe?

  Very interesting. So for the folks at home, like an example of a reasoning task, it'd be like two-step arithmetic, calculating slopes, solving linear equations.

• 9:01Speaker 01

  Yeah.

• 9:02Tim Scarfe?

  And factual retrieval might be something like, what is the tallest mountain?

• 9:05Speaker 01

  Yeah. Yeah, so what is the tallest mountain? What is the largest ocean? In which year did the Bynacca Library open, which is the Yale Library? Those are examples of factual questions, and then we have three different reasoning tasks. One is simple two-step arithmetic, so you can imagine seven minus four times eight. That's like two-step arithmetic, you first have to calculate seven minus four, and then do three times eight. Calculating the slopes requires, I think, more steps, three steps. You have two different points in a 2D space, and you have to calculate the difference between the y points and the x points, and divide them by each other to get the slope between two points.

• 9:48Speaker 01

  And, yeah, and then the linear equations task is you have a linear equation and you have to solve it for x, which also requires three simple arithmetic steps.

• 9:56Tim Scarfe?

  So what we are observing then is that when we're doing reasoning tasks, the models are synthesizing knowledge in some kind of abstract way from all of these documents.

• 10:09Speaker 01

  Yeah.

• 10:10Tim Scarfe?

  Is that reasoning?

• 10:11Speaker 01

  I would say yes, first of all. But I'm not so restrictive in what I call reasoning. As in, I don't think only formal, step-by-step logical reasoning is reasoning. I think deep neural networks can do that kind of reasoning, but our paper doesn't show that that is what's going on here. But I think the important point is that it is seemingly taking knowledge from many different documents and applying it to the same task, so.

• 10:45Speaker 01

  That's a generalizable strategy. And it's using that to generate step by step knowledge that that's um solve some kind of problem. And that that is reasoning to me. But that doesn't mean that it has a lot of um any bearing on other forms of reasoning, like inductive reasoning. For example.

• 11:06Tim Scarfe?

  Yeah, I mean, it's it's I suppose we can go into what knowledge is um. Dargar and I had this discussion. It's said that it's a justified true belief, but he said it's a justified useful belief. So you you could you could say in some sense that the templatization of the information in these documents is kind of like creating useful knowledge.

• 11:25Speaker 01

  Yeah. And what is this distinction between useful and true?

• 11:30Tim Scarfe?

  Well, I think it's about whether we can know something is true just based on a bunch of data in a corporate. There's always this epistemic gap, isn't there, that like, can we? Can we have models that actually give us facts?

• 11:43Speaker 01

  Yeah. Yeah, true.

• 11:45Tim Scarfe?

  The really interesting thing from reading your paper is that you found that when doing reasoning, like the documents that were, you know, things like Stack Overflow and code and stuff like that. That was really like almost had had a lot of influence on on the reasoning process.

• 12:02Speaker 01

  Yeah.

• 12:03Tim Scarfe?

  And that's weird, isn't it? Because it's code, it's it seems different.

• 12:06Speaker 01

  Yeah.

• 12:07Tim Scarfe?

  How how do you think about that?

• 12:08Speaker 01

  That's a good question, and I spent a lot of time looking into those results. I really spent days trying to understand what was going on there because I think importantly, we find a lot of evidence for documents influencing similar uh reasoning questions, like one document influencing many slopes questions and another document influencing many linear equation questions. But the only documents that seem to be influential, both positively and negatively for all types of reasoning, is code. And I tried to look into what what about code is makes that it's so influential? And I couldn't find any patterns. And importantly, we we don't only find that it's both that it's like good for reasoning, but also bad for certain in certain cases. So

• 12:53Speaker 01

  it was, of course, conventional wisdom that code helps downstream capabilities. Um OpenAI knows that, Anthropic knows that, they uh um. initialized our models with purely co-trained models, but we don't really know what's going on there. And and that's essentially what I'm working on now, trying to understand that better because. Yeah, I couldn't clearly find patterns in the data that we found in this paper.

• 13:19Tim Scarfe?

  It's weird, isn't it? Because code feels like the perfect materialization of human cognitive processes, right? You know, we're we're solving problems and then we manifest that in code. Does that have implications for how we design data sets for training these models?

• 13:35Speaker 01

  Yeah, yeah, I think it does, like. The the trend is adding more and more code into the pre-training corpus, right? For models to be trained on, so um I think it definitely has implications for that. I think importantly, a thing that we find in this paper that is that it seems like the model can learn to do these step-by-step reasoning traces to output them from uh descriptions of procedures in code that are really purely descriptive. So a piece of Python code that calculates the slope between two points is highly influential for actual questions, um prompts, asking the model to do that in math in in text. And if that is something that

• 14:19Speaker 01

  generalizes, if you can train a model on procedures and it can from that learn to execute those procedures. I think that can be pretty influential for how we should um how we should, for example, synthetically generate data. It could be helpful to generate lots of procedures and instead over like step-by-step applications of those procedures or you know, uh focus a bit more on both.

• 14:44Tim Scarfe?

  I suppose because of this diffused nature, so there are many, many examples in code of solving slopes and stuff like that. Um in a way, that that that's a form of robustness. Do you see what I mean? So in a way that that gives us many, many ways of doing that type of problem.

• 15:01Speaker 01

  Yeah, yeah, I see what you mean. That if you don't just see like the application, the step by step reasoning, but you also see the procedure that's gives you more robustness to different ways of expressing it or?

• 15:12Tim Scarfe?

  Yeah, or even just in terms of just having more redundancy or having many, many expressions of the same thing. It would it would make it robust to, you know, potentially different selections of the data set. It would still work, whereas perhaps fact retrieval, if the fact isn't in the data set, it's just not going to work.

• 15:31Speaker 01

  Yeah, yeah, and that's definitely true. I mean, in that sense, it's a form of abstraction that's can generalize better.

• 15:37Tim Scarfe?

  So on this abstraction thing, I was I was speaking with the guy who wrote the GSM symbolic paper earlier. And Douglas Hofstadter says that an abstraction is a bag of analogies. Right? So, you know, we have these concepts in our mind, like the concept of a chair. It's really difficult for us to describe what a chair is, because I can give you like a million different descriptions, or even the letter A. There was a book he wrote called Surfaces and Essences, where he was talking about like how all the different ways that A's could be written. So it might be the fact that the case that our brains don't really have these high level abstractions the way we think they do. Like actually, all of these neuro circuitry activation pathways are firing,

• 16:22Tim Scarfe?

  and we sort of like know an abstraction via a million different perspectives.

• 16:26Speaker 01

  Yeah.

• 16:26Tim Scarfe?

  And do you think that could actually be analogous in some way to the way a neural network works?

• 16:31Speaker 01

  Yeah, and I think it's that's also how language works. I think, I mean, I didn't come up with this like Wittgenstein did, but he he wrote this whole book, where he was just like page after page trying to show that you you can't define a thing. There will always be a situation where it doesn't exactly apply like that, right? It's all fuzzy and and meaning is used essentially, and it can change based on context and stuff like that. And I think that's the strength of language, this kind of abstraction that is not.

• 17:08Speaker 01

  Formal or like purely symbolic, but very fuzzy. Um. In that there's no clear boundaries of meaning or concepts of abstractions.

• 17:19Tim Scarfe?

  Yeah, we were speaking about Montague. So he he argued that, you know, we should we should model language like it's a formal language. And of course, it's gnarly and it's very, very kind of constructive and whatnot.

• 17:32Speaker 01

  Yeah.

• 17:33Tim Scarfe?

  Do you think L L M's are actually, you know, an appropriate tool, given that natural language is not a formal language?

• 17:39Speaker 01

  Yeah, I think that is what we have seen in the past couple of years, because like Montague tried to formalize language and that famously, I mean, didn't lead to the most simple uh formalization, right? It's it's very hard to formalize language. It's, I think Montague came up with like this very strict form of compositionality. And that has been very useful, because there there is definitely something uh in language where the meaning is composed from the parts. That that's definitely true. But this the very strict way in which Montague defined this is probably also not right.

• 18:18Speaker 01

  Probably there is, like, if you wanna make this strict form of compositionality work in language, you have to come up with really like roundabout functions where the meaning of a word is a function of the whole sentence or something and goes back into the the word itself, whereas if you take a more lenient form of compositionality or systematicity, like Föder came up with, um that roughly just says like, there is something predictable about the way we use language. If you teach someone a new word like flips and you say, um, I had a really good flip slide last night, then you can immediately um sort of estimate that this this word is probably food or something and it was at night, so maybe it was a dessert.

• 19:03Speaker 01

  Um. And you can use it in many different sentences and that's. And that is a form of compositionality and systemat化that almost seems like it's it's it's formal and predictive and like it there you can describe that formally but actually we've tried and that hasn't really worked. That's probably precisely why language models work better because they can approximate such systemat化but they are not pure formal systems.

• 19:29Tim Scarfe?

  So you've said that language models could develop a causal understanding of the world and this is really interesting. I suppose it comes back to semantics in general. So you know, John Searle said that the reason why humans have semantics is basically because we're physically and causally embedded in the world, right? And lots of linguists like Pianta Dosi are talking about things like concept role semantics. And you know, there's a whole intellectual school of thought now around how we could build semantics just in language models. What say you?

• 19:57Speaker 01

  Yeah, I I love Pianta Dosi's work and it has inspired me in many ways. And I agree with him, but I I would say he probably also, or I don't know, I shouldn't speak for him, but like there is of course a role for reference in the world. When children learn language, they start off by, this again, something I've heard from Pianta Dosi is there's there's phases of language learning where initially a princess is just like a nice woman that that is nice dress, has nice dresses and is always kind to you and the child can point to it in the world and it has a clear reference.

• 20:39Speaker 01

  But as language evolves and as the child becomes an adult language speaker, this reference becomes less and less important and it becomes more and more abstract. And now, as an adult language speaker, I can talk to you about the Covid vaccine, but I would not be able to pick it out. If you give me a bunch of substances and you ask me which of these is the Covid vaccine or what is it made of, I have no idea. And there's many examples, of course, of.

• 21:09Speaker 01

  Of things we discuss that don't have any reference in the world. But the COVID vaccine is just one where it does have a reference. But I don't know how to pick it out. And I still think I understand what a COVID vaccine is and have some sense of its meaning. But my meaning could also be further developed if I would know how to pick it out in the world, right? That means I understand it better and I have a better world model.

• 21:30Tim Scarfe?

  How much of a sharp boundary do you think there is between these kind of facts that you're talking about and reasoning?

• 21:37Speaker 01

  Probably not really, because when I was thinking about factual retrieval. And I was building these tasks, I often struggled to come up with pure factual questions. So you can imagine if you ask someone like, what is the largest ocean in the world? Maybe this person is retrieving all the oceans in the world and retrieving their sizes and comparing them. And then saying like, oh, it's the Pacific Ocean. And then they did some reasoning, so there's no clear boundary. And I really tried to make these questions very factual, like in what year does the Barnacle library open? Again, you could come up with a way in which you could reason about the answer. But you really need to have some atomic knowledge to answer this question.

• 22:21Speaker 01

  But yeah, it's all fuzzy.

• 22:25Tim Scarfe?

  So coming back to cell, you know, there were so many replies to the Chinese room experiment, like the robot reply, the systems reply, you know, this kind of stuff. And I guess at some point, when does mimicry, functional mimicry, just becomes so good that it's a distinction without a difference?

• 22:41Speaker 01

  Yeah, that's a good question. And I think then you're, that's why it's important that people like Francois Choulet come up with things like ARC, right?

• 22:51Speaker 01

  His definition of intelligence is really about acting in novel ways and using your knowledge in novel situations. And a system that's just mimicking could never do that.

• 23:05Tim Scarfe?

  Could we design a way of measuring the depth of understanding, whatever that would mean?

• 23:10Speaker 01

  We're trying, and I think evaluation is one of the hardest parts of the field. And we're, we're like, I, I, this week I heard a funny characterization of moving the goalposts. Someone characterized it in a positive way, and I totally agree with that. They said, um, people are constantly moving the goalposts and people are saying that as something that's, that's, uh, bad, but actually what we're doing is collectively refining our definitions. So first we're saying like, oh, if a system can do chess, it must be intelligent, right? But then it could do chess and then we're like, oh, that's not all we meant actually. Wait, let it, let me move the goalposts. And that's, that's not a problem. It helps us refine our definitions and it helps us, we, no one knows what intelligence exactly is.

• 23:55Speaker 01

  But designing more and more complex benchmarks and keeping, keep on moving the goalposts,

• 24:02Speaker 01

  makes for a clearer view of what it actually is and what we're all talking about.

• 24:06Tim Scarfe?

  Yeah, I think experience is helping us carve the space up a little bit better in our minds. Like, for example, I think we used to have quite a puritanical view about, you know, understanding and reasoning that, you know, you, you either you're reasoning or you're not reasoning. And I think what we're starting to see with these models is, you know, there's this Swiss cheese problem that sometimes you're in a hole in the Swiss cheese and it goes bananas. And sometimes it's retrieval and sometimes it's reasoning. It's almost like there are these different modalities of function. And sometimes it's doing more reasoning and sometimes it's doing less.

• 24:36Speaker 01

  Yeah, yeah, exactly. I mean, that's also how I, I think about them. That's, there's this, this view that if you can show that a model trips up, it necessarily means it cannot reason, but I don't think that's true. I think it's such a complex system. And, um, and if you prompt it in a, it's in a certain way, it might use a completely different sort of function or program or whatever, how you want to conceptualize what it's doing. Um, than if you prompt it in another way. And if you give it tokens that are so foreign to it that it fails to reason over them, that doesn't mean that it's, it cannot do those actual reasoning patterns and the rules that underlies that kind of reasoning. But it's just a limitation of the system and it is, it is a statistical model.

• 25:20Tim Scarfe?

  So you focused on, um, specific types of mathematical reasoning. Do you think that they would transfer to other forms of reasoning, like, you know, solving ethical dilemmas or something like that?

• 25:31Speaker 01

  Yeah, that's a good question.

• 25:34Speaker 01

  I think they do, um but of course there is a lot, there is, reasoning is such a multifaceted concept that mathematical reasoning cannot nearly cover it all. So mathematical reasoning is very formal, it has rules, um that's why we chose it. It's, uh it's the one, this, the type of reasoning we look at is so simple that you can actually find the answers in the pre-training corpus. But there are forms of reasoning for which we, like inductive reasoning, you can't find the answers if you, if you only see, uh if you only observe white swans, can you deduce from, or in- induce from that, that black, black swans don't exist? I don't know, it's, that's a form of, uh reasoning that actually underlies most of

• 26:20Speaker 01

  science and that is more difficult to see if a language model can do that. But I think fundamentally, probably, probably can. Um it becomes just in such cases, much more important to do some kind of verification of like what's going on, how, why is it making this, uh induction, um and can we like do experiments to verify it?

• 26:40Tim Scarfe?

  If language models are doing something which is akin to approximate reasoning, what's the difference between that and formal reasoning? And do you believe in principle that connectionism on its own could scale up to formal reasoning?

• 26:53Speaker 01

  Yeah, I think it can, and I think my recent paper, um gave, so I believe they're in very controlled setups, we have already shown that connectionist models can do formal reasoning, that they literally can learn to apply systematic rules, um in a way such that they achieve one hundred percent accuracy on novel, um problems. There is a good paper by Lake and Veroni in Nature that does this. Um there are other papers that, that show that, for example by Andrew Lampinen, passive learning of active causal strategies, that show if you set up the problem in such a way that you can, um that the model can learn to do the task as opposed to latch on to sort of, um.

• 27:36Speaker 01

  On important things in the data, it can prob- it can learn to apply tasks in novel situations. So I think empirically and theoretically, we show, we have shown that they can do a form of systematistic or um um symbolic computation, although it's still limited for sure, like it can't handle completely novel tokens. Um but the question with my most recent paper was, can it also learn to do something in that direction approximately from uh data in the wild? Because language models are not trained on data that is so carefully curated that the only way you can um make the laws go down is learn the underlying rules, because that's what these papers often do.

• 28:18Speaker 01

  Um and the question is, can it also then learn to do something that's that's a formal formal reasoning or symbolic reasoning? And I think it can um.

• 28:28Speaker 01

  And my paper doesn't show that exactly, it just shows that it's doing something generalizable, that it can apply to many different questions, but intuitively, I think it would be possible.

• 28:37Tim Scarfe?

  So there there's always been this notion of a gap that people talk about, um especially in respect of creativity, um ad- adaptability, dealing with novelty and so on. In fact, many people think that the definition of intelligence is is, you know, dealing with novelty. So there's there's always this thing that um we can do combinatorial creativity, right? So we can do reasoning by like recomposing bits we already have, but people say that this inventive creativity, like being able to, you know, train on all the data up to nineteen forty-five and then invent some new theorem that came after that. People feel intuitively that the models wouldn't be able to do that.

• 29:12Speaker 01

  Yeah.

• 29:12Tim Scarfe?

  What do you think?

• 29:13Speaker 01

  That's really like the goal, right? Like that kind of stuff, it would be really cool. And I don't feel like current language models can do that, but I don't feel it's it's technically impossible. Um even in the current regime, like if we were to be find so much data that the model can learn the causal underlying data generating process of the of that is relevant to come up with novel information, then it can do that. But of course, like, we we have used most of the data that were created over the past couple of thousand years, or at least we're trying to. Um and it's probably not feasible to scale up to such intelligence in in this way, but

• 29:54Speaker 01

  Yeah, I I I think it's not not theoretically theoretically impossible. It kind of gets at at whether Einstein came up with some like stroke of genius that doesn't compose stuff he has seen before, or whether he actually also stands on the shoulder of other scientists and reasoned about things for a long time and and use that to come up with new knowledge. And I think it's probably the latter. And that's not so special that we cannot. I mean, OK, I don't want to say that Einstein is not so special he is, but

• 30:30Speaker 01

  probably we can in some form recreate that process.

• 30:34Tim Scarfe?

  So Tim Rocktashal, he's he's got some great work on open-endedness and creativity and and whatnot. And it's interesting because Ilya Sutskevar, he gave a talk at this conference, and he said that we are hitting a data wall.

• 30:47Speaker 01

  Yeah.

• 30:47Tim Scarfe?

  And and to me, that doesn't pass the sanity test, right? Because if you think about it, there are an infinite number of ways you can make more data. You you can transform the data we already have, and you can generate lots of data. But this is where it gets into Tim Rocktashal's domain, which is that it's not just about generating more data, it's about generating interesting data.

• 31:06Speaker 01

  I find genie and stuff incredibly interesting, and I agree that the intelligence of the system is purely limited by by the complexity of its environment. So I think that's an interesting approach. And I also think that, yeah, so I I think scaling up data helps because it makes it less and less possible for a model to latch on to spurious correlations. It's going to be more and more useful to learn like the causal world model that generates this data, the more data you get, because it's likely going to be less semantically similar than what you've seen before. But if you were to somehow be able

• 31:47Speaker 01

  to select from this all this data that we have, data in a way that's like sufficiently deferred diverse.

• 31:56Speaker 01

  For you to learn this causal mechanism quicker without seeing, I don't know, trillions of tokens, I think that that should maybe also be possible. And I think that is informed by these controlled studies that show that you can train a model to do something systematic in one task. But how do we train a model to do something systematic in as many tasks as we want as we want language models to do?

• 32:19Tim Scarfe?

  What's your philosophy on scaling in general? So do you think that if we just scale current approaches up that we will get dramatically better results? Or do you think we're missing something significant?

• 32:30Speaker 01

  I am not going to bet against scaling because that seems scary. It has worked pretty well. But yeah, so I think scaling is cool. I think there is issues with it and there's probably more data efficient ways to do it. Just because theoretically you can train a model to do many different complex tasks with next token prediction doesn't mean it's the best way to do it. And maybe there's something about intervening on an environment and generating your own data that can help there and that can make models more data efficient. And I could see how that that could be important in the future.

• 33:12Speaker 01

  And I think Ilya also mentions not specifically that, but agency or agents. And maybe that's getting at the distinction between sort of passive learning and active interventional learning.

• 33:25Tim Scarfe?

  We should save the agency discussion for a bit later because we've got lots of things to say on that. But why don't we just talk a little bit about that FODA and Polition paper from 1988? So this was their famous connectionist critique. And they said that the way that humans think is very formal. You know, we have these rules and we have this compositionality. So we can generalize Mary loves John to Mary loves Jane. And we can also take a sentence and we can kind of invert it. We can decompose it back to all of the constituent parts and we can, you know, figure out what things mean. And neural networks on their face, they don't do that explicitly, but perhaps they might do it implicitly. What are your reflections on that?

• 34:08Speaker 01

  Yeah, um, I think Fodor and Felician's argument has definitely stood the test of time. Um, although there's been some theoretical work showing that it's not impossible in a connectionist regime to learn symbolic functions, like Smolensky's work in the nineties, uh, in tensor product representations. Um, that that was a theoretical work showing that you can do some symbolic computation in the in the sub symbolic regime that connectionist networks represent. But the the argument nonetheless stood the test of time because this systematicity that we also spoke about uh earlier is definitely something that's present in language and that is

• 34:50Speaker 01

  necessary to explain, um, if you want to understand how humans can produce something that's so varied with so such little, um, examples or, you know, memory. And it was a challenge for, I don't know, thirty years or something like that. And it it probably still is, uh, it's probably still relates to this concept of intelligence as being able to process novel information. But I think there's been a lot of empirical work showing now that actually sub symbolic models like neural networks can can do symbolic computation, albeit not explicitly. I mean, yeah, explicitly in in the sense that they can maybe

• 35:31Speaker 01

  uh, output some symbolic computation in form of language and and explicitly reason over that. That's probably a good idea. But probably they can also do it implicitly.

• 35:40Tim Scarfe?

  Yeah, I suppose that there was a bit of a a theme of um just having strong theoretical tools, especially around that time. So, you know, this idea of productivity, being able to generate an infinite number of sentences. I mean, Chomsky said the probability of a sentence is an oxymoron. It just doesn't make sense to to say that. And um, it certainly feels, as you say, that our language is compositional. And again, Chomsky said it's a language of thought. So if our language is compositional, then surely our mind must be compositional. So it might it might have just been like almost an intuition pump to reason about how our brains work.

• 36:14Speaker 01

  Yeah, um, I do think though that's.

• 36:19Speaker 01

  So this gets at this question of whether language is thought, right? And I think that has been pretty rigorously debunked at this point. And I think maybe language is as useful to us precisely because our thought is not compositional. Because we can use it as a compositional tool. Um, that is maybe a bit harder for us to do systematically in our brain. Um, and I mean, there's been work by F. Fedorenko in 2020, for example, showing that people with F. Asia can still reach as grand masters. So when your language system is completely um messed up, you can still reason perfectly fine. Which kind of, in my view, debunks the theory that language is thought.

• 37:03Tim Scarfe?

  You said to me earlier, like, well, what's the big deal? Why do we need to have invertibility? You know, and and when I say invertibility, I think I'm kind of saying um decomposition. So, you know, they were talking about compositionality, but I think decomposition is is really important. That's being able to go back to the constituents. And it's not only about being able to, like, explain what I'm thinking, it's also about parsimonious reuse. So we see in in Mechinterp, for example, on that um scaling mono semanticity paper, that like, the representations for the Golden Gate Bridge, it was scattered throughout all of these circuits throughout the neural network. And it it feels like, certainly at a psychology level, it feels like our brain doesn't work that way. But maybe that's just a bit of an illusion.

• 37:47Speaker 01

  Yeah, um, it's hard for me to say, as I mean, I don't want to comment on neuroscience because I have no idea about that. But what I could say about this is that it seems to me that it's pretty useful that the model is representing it in this way. And maybe that's also, as in, it's doing a very distributed representation, right? And that that's essentially the core reason why people in the 90s believed in connectionist models, this distributed representation that can where all neurons can essentially light up for all different tasks, as long as there's some shared structure. And that makes them so flexible and that makes them so good in novel situations, actually.

• 38:28Tim Scarfe?

  I suppose there's a bit of a broad theme as well. So, um, certainly 20 years ago. We used to design AI systems with explicit strategies. So, planning was an explicit thing, reasoning was an explicit thing. Even certain architectures like Dreamcode, Kevin Ellis's Dreamcode, had an explicit wake dream state. So, when you dream, you kind of expand your hypothesis space. And then like, you know, when you're, when you're awake, you kind of select the ones that work. And neural networks do this expansion and collapse all of the time. And, but what we're seeing though with the newer architectures is that they kind of do the same thing, but they do it more and more implicitly, like we don't, we don't hard code it in.

• 39:10Speaker 01

  Exactly, and that, that's what we've learned the past couple of years. That's, that's the way to go probably. Because that's what we've learned from the LSTM, the LSTM to transformers, which is quite funny actually. I, one of my first papers is, is on compositionality. We, we designed this benchmark together with Brendan Lake and others, where we, we held out systematic experience in the data. And we showed that a human can easily do this, but a LSTM couldn't. And this was all pre like transformers, LSTMs, LLMs and ChatGPT and that kind of stuff. And someone this week told me that actually a transformer gets almost one hundred percent,

• 39:54Speaker 01

  one hundred percent performance on most of the tests we designed in that paper. Not all, but most. And there, that's just one example of the transformer being a much better fit for compositional tasks than LSTMs. And the lesson maybe we can take from that is that LSTMs have this explicit recurrence, which seems very useful, right? Because there is clearly a recency bias. Clearly what we have just talked about is more relevant than what we, I don't know, you and I talked about last time we saw each other. But if this recency bias is so obvious, why would you build it in? Because the model can easily learn it from language. And this is, this is what we've learned in the past couple of years.

• 40:37Speaker 01

  That if, if something can be learned, don't build it in.

• 40:42Tim Scarfe?

  Or use XLS TMs.

• 40:43Speaker 01

  Yes.

• 40:44Tim Scarfe?

  Yes.

• 40:44Speaker 01

  You spoke to Seb, so.

• 40:46Tim Scarfe?

  The other day, yeah, he was saying about their new exponential gating scheme allows them to kind of like, you know, overwrite their memory.

• 40:52Speaker 01

  Yeah, very cool.

• 40:53Tim Scarfe?

  It's it's kind of weird though, isn't it? Because there is this notion, I was saying to him, like, when are we going to see industry adoption of of XLS TMs? And I think in industry, the perception is it kind of doesn't matter, like, it's just about scale.

• 41:06Speaker 01

  Exactly. But that's that's the thing also is OpenAI, like, they they don't care about these, like, compositional, like, is it out of distribution? Are they are they uh? Are we holding out the right things? Has it seen this before? No, they're just like, we're gonna make it in distribution, and we're gonna scale it up and and that's kind of their genius, essentially. They, no matter what the architecture is, no matter how much it looks like the brain, or or what's the, why it should theoretically work better than something else, if you can, you know, uh, use more flops, it's better, it's better.

• 41:39Tim Scarfe?

  Let's quickly talk about um Smelensky. I always mispronounce his name, so I'm gonna say it very slowly. So around nineteen ninety, he, I I guess it was a response to this photopolition thing. And and he said that with these, you know, connectionist models, you can still implement the essential capacities of symbolic processing, you know, such as representing variable bindings and structured data and compositional operations. What did he propose?

• 42:03Speaker 01

  So he proposed a mathematical, it was a mathematical uh framework for variable value binding. And that's like this very intuitively uh symbolic computation, right? No matter uh what the value is, the variable can take it, and you can do processing on it, and the results will be uh reliable and the same. And um yeah, further inflation says uh uh connectionist models can't do that, and that's produced a decade-long back and forth between connectionists and symbolicists.

• 42:34Speaker 01

  And Smolensky gave this answer with a tensor product representation to say now like, "Look, you can actually represent variable value binding in a purely sub-symbolic connectionist way." And that's what he showed in tensor product representations, where you represent the variable and the value both in a distributed sub-symbolic way, and you can do processing on them, and they all become embedded in this continuous space, this distributed space, but then you can still extract the value from the variable after processing, unbinding is what they call that.

• 43:08Tim Scarfe?

  So what were the drawbacks with that approach? And also, I think there's a bit of a leap of faith here that neural networks could in some way approximate what he was talking about.

• 43:20Speaker 01

  Well, and I didn't read this paper, but Tom McCoy published a paper together with Smolensky, I think, that's titled "RNNs implicitly learn tensor product representations." So that seems to indicate that they can, but that's just, they can, but that's just based on the title. I think that one has been on my reading list since I found out about the tensor product representations. But you're right, like, what's the limitations of this method? It's a purely theoretical argument, right? He's saying to failure inflation, "Look, actually, you can do this." That doesn't mean that it's practical, that doesn't mean that it scales. Tensor products, the way he proposed in the 90s, doesn't scale at all, because it explodes in the number of variables that you're representing.

• 44:02Speaker 01

  So let's say variables are positions in sequences, and values are the tokens. Then the tensor product representation will, I think, is squared in the, or no, like, explodes in the number of positions that you're trying to represent and in the number of tokens. So that's not feasible. I think Smolensky is working on this at Microsoft, so I'm sure he's working on making it more scalable. But another thing that I took away from reading that paper is that to get actually this value back from this distributed representation, the, there, something needs to be linear

• 44:42Speaker 01

  independent, like the rows in a matrix or something like that, needs to be linear independent. And that seems like a very...

• 44:52Speaker 01

  Hard restriction to me that probably won't naturally arise that. Or maybe it would, because I think people also tell me that if you randomly sample is almost always linearly independent in in high dimensions. So maybe actually that's not the big of a limitation. But yeah, the way he proposed it back then wasn't that scalable.

• 45:10Tim Scarfe?

  Yeah, so these tensor out of products composed of these roles and fillers. And apparently the roles required quite a lot of hand engineering. And yeah, we've got this combinatorial explosion problem. But but anyway, it's it's interesting. It's it's it's a potential, it's a potential way forward.

• 45:28Speaker 01

  Yeah.

• 45:29Tim Scarfe?

  Okay, Laura, where does agency fit into all of this? And and just to frame the question a little bit, some people are really worried about agency. I was speaking with Benjio the other day, and he said that agency is is really bad, you know, it it's going to lead to these things controlling their own goals and it could be very dangerous and whatnot. And we should strip away all agency.

• 45:52Speaker 01

  Yeah, no, I totally agree with that. It's like if you think about an intelligence system that's also an agent or just like a random human can be very dangerous, right? And probably agency is a large part of that. So if you if you have two systems that are otherwise completely identical in capabilities and one is an agent and the other is a tool, I would prefer the tool. The thing is just that I'm not so sure if it's possible to reach an interesting form of intelligence without the notion of agency. Um, so my interest in this question has just been like, how can we define this concept? And how can we detect whether it's present in a in a system? Um, and that's a pretty difficult question, I think.

• 46:36Tim Scarfe?

  It certainly is, it certainly is. Do you think L L M's to any meaningful extent have agency?

• 46:43Speaker 01

  Yeah, that's that's a question that I've been thinking about. Um, I think like there's many definitions uh of agency. And what to me, it's just a kind of goal, goal directed intentionality, and we can get into what that exactly means. And doesn't L L M have that? I I think you could in some way see it as modeling uh agents and maybe it also models their goals.

• 47:11Speaker 01

  So, of course, they're uh they're trying to model the text and this text is they're trying to yeah uh predict the next word uh efficiently or the next token and decrease the loss there and this text has been generated by agents. And probably it's useful to decrease the loss if you also understand what goal overarching goal this agent has. So if this agent is trying to persuade you, maybe uh that that informs or not not you that L M, but if the agent is the text is trying to persuade something or someone, then maybe it's useful to model that goal um to sort of decrease the number of possible tokens that can be show up in that text.

• 47:53Tim Scarfe?

  When we say the L L M is trying to persuade someone.

• 47:57Speaker 01

  Yeah.

• 47:58Tim Scarfe?

  There's this there's this weird thing, isn't it? Because you know, to a certain extent, agency is is observer relative. It's like it's a thing that we say that another thing has.

• 48:05Speaker 01

  Yeah.

• 48:05Tim Scarfe?

  So it feels like at the bottom of the spectrum, it could be as if the thing has this goal. Because the L L M probably isn't thinking, oh, um Laura's an agent and Laura's got this goal and in order for me to control Laura, I need to do this and it's in service of that. You know, it feels like there's an unwitting form of agency first.

• 48:25Speaker 01

  Which might be even more dangerous, right? Like if it's accidentally persuading you and it doesn't understand the the things that can happen when you when you do that, then that might be even more dangerous. So this kind of gets at the distinction between simulating something and actually coming up with it with it yourself. And I don't know what's how you can this, yeah, find the distinction between the two.

• 48:47Tim Scarfe?

  I guess you would agree that agency could emerge even if we're not explicitly trying to make it emerge.

• 48:55Speaker 01

  Yeah, I think that's the interesting case. I that's I've been thinking about this a lot recently and I think the interesting case is when it's emerged. So there's this definition from Zach Canton from Deep Minds. They also have a safety interest in in agency and a couple of years ago, they made this definition of agency that's um about how.

• 49:17Speaker 01

  If you, an agent is something that it changes its policy when its actions affect the environment in a different way. And that's a nice definition, and I think that definitely captures something that I also find important about agency. But you can kind of trivially make a system of L L M's in an environment or something where the environment is also an L L M, such that it adheres to this definition. So I think the important thing is like, when does something like that emerge from something as simple as next token prediction? And that's kind of what I'm interested in.

• 49:49Tim Scarfe?

  How might we measure that?

• 49:51Speaker 01

  Yeah, that's a good question, and I have no answer to that, but I've been thinking a lot about that. I I've been even speaking to some psychologists. Um, uh, Ellen Su at N Y U, actually, who works on intent detection uh in in uh in A I. So there there is methods we can learn from in psychology that can help us inform here, but I think the thing I've been thinking about is what makes agency potentially um interesting and and complex is is planning. So if an agent, if it can't plan, it's probably not super uh um useful or dangerous. So and planning seems like somehow an important

• 50:36Speaker 01

  aspect of a of an agent that's able to achieve complex goals. So I've been thinking more about planning and trying to detect when a model can be doing planning and when a next token predictor can actually be said to do planning.

• 50:49Tim Scarfe?

  Yeah, it's so interesting that so many people are converging on the same idea. I mean, certainly in in active inference, Carl Friston would say that the planning horizon is basically the measure of the degree of agency that that a that a thing has.

• 51:02Speaker 01

  Okay.

• 51:02Tim Scarfe?

  Even Eliza Yudkowski, he basically said that, you know, an intelligent thing is defined by its planning horizon, pretty much.

• 51:09Speaker 01

  Yeah.

• 51:09Tim Scarfe?

  And uh, Yoshua Bach told me that agency is the ability to control the future and and the future, of course, implies a planning horizon. Um, but so you definitely think of agency, though fundamentally, as about this um kind of cybernetic.

• 51:25Tim Scarfe?

  Information exchange with the environment, can you tell me about that?

• 51:28Speaker 01

  Yeah, so you just said that someone called it the ability to control your future, that maps on to what I think. I think an agent is something that takes actions in order to control its own future inputs, which is essentially the same thing said differently. And I also think importantly, it is able to do this in uncertainty, under uncertainty, in uncertain environments. Because you kind of wanna get at this distinction between reflexes and maybe deterministic environments where nothing changes and environments where there is uncertainty and the system can still control the future.

• 52:09Tim Scarfe?

  In the kind of, you know, the biological world, we are decomposed into all of these autonomous cells. And agency is just something which emerges through the sheer complexity of interaction.

• 52:22Speaker 01

  Yeah.

• 52:23Tim Scarfe?

  Yet we still talk about L L M's as having a type of agency. What's the difference between the two?

• 52:31Speaker 01

  Oh, that's a difficult question.

• 52:34Speaker 01

  I think it's just an abstraction that we use to describe a complex behaviour. And we can get at that abstraction in a way that it applies both to the, you know, balls of cells that we are and to the other types of cells that the L L M's are in a very, composed in a very different way. I think what you can't get at with this view is this, you know, it feels like something to be an agent kind of thing, like it, like there is something that's, you know, less explained by this abstraction that I was describing earlier that maybe doesn't describe what it is like to be an agent or something like that.

• 53:17Speaker 01

  Or whether or not it feels like I'm setting my own goals or whether they're induced by the environment. And I don't know how to make a definition that can distinguish between those two things.

• 53:30Tim Scarfe?

  I suppose the world model comes into it as well, that in order to do planning into the future, you have to have a very good representation of the world.

• 53:37Speaker 01

  Yeah, definitely. The more like causal your world model is, the better you can plan. I mean, you also need other things, like some way to represent the possible futures that you're rolling out. But definitely, yeah.

• 53:51Tim Scarfe?

  And even that seems to suggest that causally embedded agents, we have this active sense-making, continual learning. So we're always doing experiments, right? We're kind of learning about the micro-causal patterns in the world, which makes us more, you know, it makes our world model higher fidelity. Language models seem to have a very globalized version of that, but that still works quite well.

• 54:15Speaker 01

  What do you mean by globalized?

• 54:17Tim Scarfe?

  Well, as in, even though they're learned on all of these patterns from many, many data sources that have been mixed together, they can learn powerful representations that can respond well.

• 54:27Speaker 01

  Oh, yeah, yeah, yeah.

• 54:28Tim Scarfe?

  But we're like in the situation continually learning, active sensing, like finding out about our environment. So it feels like we understand our world that we're in even better.

• 54:38Speaker 01

  Yeah, no, definitely. That's true. I think we have, I mean, we have these sort of core knowledge systems that our intelligence is built upon, right? And that are present in all animals on the world to some extent. And that shows that they are just so useful for surviving in the world that they just emerge for everything. Whereas language models are trained on language and they have probably some sense of all these kind of things, but they're not constrained in the same way that we are. Language is inherently able to describe impossibilities and things that are physically not possible and imaginize and imagine and stuff like that. So it's also not so surprising that they show some different behavior and hallucinate

• 55:22Speaker 01

  and produce impossibilities. But humans are learning in a very, very different environment. And we have also learned to talk about impossible situations through language and to imagine a future that is possible or not possible and reason about these things, but we're still constrained by. You know, the physical reality.

• 55:45Tim Scarfe?

  So I often have disagreements with my co-host, Dr. Duggar. So he has a real no-nonsense definition of of agency. He thinks it's basically just an automaton, right? So it's, I mean, I can give you the definition. It's a machine that receives input S from an environment E, performs a computation C that depends on a non-empty subset of S, and takes action A that depends on C to modify E. So it's basically like, you know, you have an environment.

• 56:14Tim Scarfe?

  Pretty much. And to be honest, you could use this rough definition even to describe active inference and many other things like that. But the thing I don't like about it is, you know, it's basically describing a kind of state machine. And of course, like, for him, the environment could mean like any ambient things in the environment. And for him, computation is very important. So he's a big fan of like the Chomsky hierarchy. And he thinks there's something special about Turing machines. So he thinks that we as strong agents, we must be able to do this kind of recursive nested iterative form of computing, which is what allows us to do planning and whatnot. But to me, that just seems a little bit like, a little bit weird, right? I love this philosophical notion of agency.

• 56:56Tim Scarfe?

  And I realize this is a bit wishy-washy because I'm using words like emergent self-organization, autonomy, learning, adaptability, intentionality, you know, degrees of agency and all of this kind of stuff. And it kind of feels like, how can a computer program that maps from an input to an output? How could that be an agent?

• 57:17Speaker 01

  Yeah, that's really the question, right? And I think that I agree with you that this definition that your co-host gives is, I mean, it's a fair definition. I just think it puts the emphasis on the wrong thing. I think it exactly doesn't explain what I find interesting about agency, which is this like acting under uncertainty kind of idea, right? It doesn't get at that. And there's something very intuitive about agents to us. And it would be useful if we were able to describe that in a way that's more.

• 57:51Speaker 01

  More abstracted away from computation than this definition. It somehow gets at the difference between a thermometer, that you could also describe in this system, and an agent. Because that's what we're trying to do, right? Maybe there is, maybe there is no distinction, but humans perceive a distinction. Like, it's, it's actually one of the chronology systems, agency. And this is very nicely uh shown by this Hyder and uh simul- simulation, this video from the nineteen forties, where you have a big triangle and a small triangle moving around in a two D environment. And uh, there's a little box with an opening and the small triangle is trying to escape from the big triangle. And it's going into the box and a big triangle is like bumping against the box.

• 58:36Speaker 01

  And, and these are just moving uh shapes. But we immediately assign agency to them. And we say the big triangle is mean and the small triangle is scared. And um, maybe this is a failure of our application of the agent chronology system, because they are not agents, probably. Someone programmed them to be. We intuitively pick out an agent from a thermometer. And that's the distinction I wanna get at. And the definition by your co-host doesn't really get at that.

• 59:04Tim Scarfe?

  To what extent is agency just, just the way we think?

• 59:09Speaker 01

  Yeah. You mean it's not, it's just not real or something?

• 59:13Tim Scarfe?

  It could be, it could be both. It could be because it's real. It's such an important way of dividing the world up that it's become embedded in us as a core cognitive primitive. But it, it seems fundamental to the way we recognize things.

• 59:26Speaker 01

  Yeah, yeah, it definitely seems fundamental. I think it's just important in the sense that an agent can uh be of use to us or in a different way than a non-agent can, or can be dangerous to us in a different way than a non-agent can. And, and whether or not that's um just, you know, something we perceive that's not sort of fundamentally there, doesn't really matter then, I think.

• 59:52Tim Scarfe?

  Yeah, one thing I guess that's.

• 59:56Tim Scarfe?

  Sometimes people say, oh, we just philosophize everything. And certainly when we talk about consciousness, you know, there's people like David Chalmers, who says, you know, it might, we might be philosophical zombies, it might just be a little bit extra. And even with free will, which is almost like a stronger form of agency, which is that in the situation, you could have done differently. So we're kind of like imagining how things could have been differently. And it's a similar thing with intentionality. We think that intentionality is like something on top of what a language model might do, or what an automaton might do.

• 1:00:27Speaker 01

  Yeah.

• 1:00:27Tim Scarfe?

  And do these philosophical properties, are they useful?

• 1:00:33Speaker 01

  I think they are, because again, they get at, so it definitely feels like something to be conscious to me. And people have talked about that a lot. So it must get at something interesting, I'd say. And therefore it's useful. And I think similarly with intentionality,

• 1:00:54Speaker 01

  it's, I just view it as a useful abstraction of behavior that can guide us towards understanding better, maybe how cognition has emerged, or how animals, certain animals are different from other animals, and how, and can also help us evaluate an artificial intelligence in whether or not they are doing something that can be seen as intentional and goal-directed. And yeah.

• 1:01:24Tim Scarfe?

  So I think the other thing I don't like about the automaton view, or maybe even reinforcement learning as an extension, is that it's a form of behaviorism, which is that we only look at what the thing does, and we don't have rich cognitive models of like what the mental states are. And it feels to me, and maybe this is like an interesting departure for you, because like, in a way, like with the language model discussion, it felt like you were arguing that we don't really, you know, it doesn't, it doesn't matter if we convolve functions together into this big soup. But with agency, it feels like you are saying that we need to have an explicit structure of how an agent thinks.

• 1:02:02Speaker 01

  Yeah, so I don't think it doesn't, representations don't matter, right? I think they matter a lot. I think there, there is a distinction in

• 1:02:12Speaker 01

  Um pragmatic representations that are purely goal directed. And the representations that are somewhat divorced from the current situation you're in or the current goal you have. And both are important and I think both exist in the real world, in animals and also in language models and um so. I don't think it matters that we, whether or not we convolve or how we do it, but I do think. It's important to reason about what kind of representations have been learned and whether they are. Ref nicely reflective of a causal world model that we want, the models have learned. So I think this behavior is, what you said, that this definition is a bit behaviorist, I think that.

• 1:02:52Speaker 01

  Kind of gets at what my problem is with it, I think, yeah, because it's it's it's sort of like, sure, yes, this definition applies. But it doesn't explain to me why I care about this system and what is interesting about this system and um. Yeah, often behavior can, you know, explain a lot and you can say a lot about the behavior, but if you. Know something about the representations that produce that behavior, you can describe a system in a more useful way.

• 1:03:20Tim Scarfe?

  I know you're a big fan of the Simulators article by by Janus. And he said, I I guess you can you can interpret this in an agential way, that there's some kind of agential decomposition of of a language model into these. Role players.

• 1:03:34Speaker 01

  Yeah.

• 1:03:34Tim Scarfe?

  What do you think about that?

• 1:03:36Speaker 01

  Yeah, I I think that was like, so first of all, I'm a huge fan of the article, but I I um. Became a big fan because of Jacob Andre's language models as agent models paper, because somehow. He describes it in in a type of language that I find easier to follow. But the simulator is supposed to has, of course, been hugely influential and also for me in my conceptualization of. Of language models, this is essentially the reason why I think like, I think about them maybe modeling. Human intent and the intent of the agents of the text that they have been uh learning from. And this view of them as a sort of superposition of many different agents just is such a rich conceptualization that.

• 1:04:19Speaker 01

  Really explains many things, both their successes and their failures. And I think that's what's cool about it.

• 1:04:25Tim Scarfe?

  One interesting thing about the article is is this notion of coherence. So when an when an agent is, you know, like a a role player is selected, then that role player will stick around for a little while. And certainly, it feels like our intuitive notion of agency in the real world is that we maintain ourselves and we also stay kind of coherent over time.

• 1:04:44Speaker 01

  Yeah, a bit, maybe.

• 1:04:46Tim Scarfe?

  Yeah, a little bit.

• 1:04:47Speaker 01

  Definitely changed my views over time. I think that's also the sign of, yeah, that's important. But no, you're right. Like, I think there's also been a paper here and that shows that, or here at NERIPs, uh, that shows that language models are not, don't stay in character as long as as actual agents do or humans do.

• 1:05:06Tim Scarfe?

  Tell me more.

• 1:05:07Speaker 01

  To be honest, I didn't I didn't read it, but I I saw it. It's I saw it as something that like I I I should look into, but I I mean, definitely, they're probably not as coherent and they don't stick to their role as as clearly as humans do. That's probably the nature nature of being an approximate sort of agent or a superposition of agents and you can't really disentangle one agent from the other agents and.

• 1:05:31Tim Scarfe?

  What do you think of um non-physical agency? So I'll give you an example of that. Um, we as a collective form a kind of agency, you know, like a meme is a type of agent maybe, and I know Dargard doesn't agree with me about this. Um, even the Covid virus, I mean, I I heard that flu rates are dramatically up in in the UK and there there's a weird kind of symbiotic relationship between flu and Covid. So when flu is up, Covid is down. And it's almost as if they are these virtual agents that are sort of like interacting with each other through the hosts.

• 1:06:03Speaker 01

  Yeah. No, I think that makes total sense. I think probably it will be really hard to say that a sort of collection of agents is not an agent, but an agent itself is. And I think it can be a useful. Um useful way to represent something, for example, a company can be seen as a group of agents, right? And how how they behai- how it behaves, but at the same time, at the company level, there's something that seems sort of extra uh that you can't exactly explain from the parts, which might be some kind of emerge, I don't know, some kind of emergence, I don't. Yeah, it's hard to describe, but that many people, of course, have thought about.

• 1:06:43Speaker 01

  But I think it makes sense that a collective of agents can also be seen and abstracted as an agent in some sense. But there's probably also something to uh sort of single agent that understands, you know, the actions you're taking that are uh that you guide, whereas in a collections of agents, maybe maybe that's that becomes different or more difficult or something.

• 1:07:10Tim Scarfe?

  When we look at a super agent like a company or a country or a religion or something like that, do you think the purpose bubbles up or down?

• 1:07:21Speaker 01

  Oh, um, let's see, both, I think both.

• 1:07:28Tim Scarfe?

  Yeah.

• 1:07:29Speaker 01

  Yeah, I think the purpose of a company is probably definitely some combination of the people that work there. Um, and then probably the company as a whole forms some values or something that then inform the agents individually or something again as well. Um, yeah.

• 1:07:48Tim Scarfe?

  On this subject that I was discussing with with Benjo about AI safety.

• 1:07:52Speaker 01

  Um hum.

• 1:07:53Tim Scarfe?

  Is that is that something you're concerned about?

• 1:07:55Speaker 01

  Yeah.

• 1:07:56Tim Scarfe?

  Tell me more.

• 1:07:57Speaker 01

  Um, yeah, I think just if you think about and if you philosophically think about a system that's intelligent, that is just can be dangerous. So, um, as a society, we don't even really know how to control humans, but we have set up a pretty okay system to do it that fails at different levels, right? At the individual level, at the between-country level, at all kinds of level, it fails sometimes. That's scary and dangerous. And I think intelligence is, um.

• 1:08:30Speaker 01

  Not so special that we can never build it. Um, therefore, that can be dangerous, right? Um, but I I've I've struggled talking about my timelines, as in, when when will this happen? I have no idea. Um, I don't see it happening in the next three years. I see, I feel like a lot needs to change. I think society um moves slow as well. I think there's massive issues in adoption, like these these systems are not reliable. So I think as sort of philosophically about an an intelligent agent is dangerous. And a whole separate separate thing of A I safety that I found even more compelling

• 1:09:10Speaker 01

  is that if we slowly give over control to dumb agents or dumb dumb A I, that's that can also be dangerous in a society like ours. And that's also something I worry about. So I think that's um. I think understanding how everything works and how these systems works is important. Um, because I also think it can, it, you know, I'm I'm not purely a pessimist. I think it could bring a lot of um great things to the world. There's a lot of things that should probably be automated because, or at least it would be great if we can get certain professions some help because, you know, we're all getting older and um a lot of people are working in care and

• 1:09:54Speaker 01

  uh if we don't do something, I'm not saying A I is gonna help there, but it it would be great if if if A I could alleviate some of the things that are gonna become more difficult in the future. Um, like healthcare, if they could make uh doctors more productive there, for example. Um, but it's really non trivial to think about how it can have a positive impact, I think. And it's it's good that lots of people are thinking about it.

• 1:10:19Tim Scarfe?

  I love agency as a kind of meta model to think about this, because if it is the ability to control the future, then to me it's analogous to power.

• 1:10:26Speaker 01

  Yeah.

• 1:10:26Tim Scarfe?

  And certainly, talking about power dynamics is is the language of talking, you know, how how we should govern this.

• 1:10:32Speaker 01

  Yeah.

• 1:10:32Tim Scarfe?

  And I can see many arguments. I can see how this kind of technology actually takes away our agency. It it can also dramatically give us agency, because all of a sudden people can build chemical weapons and bombs and and stuff like that.

• 1:10:46Speaker 01

  Yeah.

• 1:10:47Tim Scarfe?

  Um, but but the other the other concern, of course, is that it itself will, you know, adopt a form of agency and through instrumental goals or whatever.

• 1:10:56Speaker 01

  Yeah.

• 1:10:57Tim Scarfe?

  You know, so so on on those three, where do you see the the most significant risk is?

• 1:11:02Speaker 01

  I think all risky. I think the the thing I am most worried about is um skewed access. So if A I becomes very useful and and makes us more productive, it would be great if we can distribute that in society in a way that helps people um like. I think techn- technological improvements have maybe not have not like in the right proportion uh help the right people and that's you know, a a result of the system we live in and a result of our politics. So I think it's it's really important in the future to think more about that and to think more about how we can uh give access to the right people. And I think

• 1:11:48Speaker 01

  the way to go about that is policy and um think about how our system works, how how our economy works and and be prepared for um massive in in uh improvements in A I capabilities.

• 1:12:02Tim Scarfe?

  If this starts to go bad, what do you think? Because obviously there there should be some kind of a harbinger. You know, what would be the early warning signal for you?

• 1:12:09Speaker 01

  So what's harbinger?

• 1:12:10Tim Scarfe?

  As in like um a harbinger is like a some like a signal that something bad is about to happen.

• 1:12:16Speaker 01

  Okay. Yeah, I I think probably that's like, that's not what's gonna happen. I think we are gonna slowly build something and then at some point we're gonna be like, oh wait, you know, remember back then and when there were elections and and Facebook apparently uh may have influenced them and we all built this tool and we didn't realize how it would affect us. And I think that's that's will probably also work like this with A I. We don't understand what intelligence is and probably we won't recognize it immediately if we see it.

• 1:12:45Tim Scarfe?

  That's fascinating. Yeah, I love this notion of kind of undermining our weaknesses. So it's it's actually a very sort of alien diffused thing that we might not even be fully cognizant that it's happening.

• 1:12:57Speaker 01

  Yeah, exactly. Yeah, I think that's more likely to happen uh than that we're all of a sudden gonna be like, oh, wait a minute, this is this is dangerous. Though I mean, there are also examples of that happening, right? Like Chat G P T. Um. In NURIPS twenty twenty, twenty twenty one, twenty nineteen.

• 1:13:16Tim Scarfe?

  Twenty two.

• 1:13:17Speaker 01

  Twenty twenty two, a couple of years off. Yeah, so I remember ChatGPT dropping and that was for me the first time that I was like, oh my god, language models are crazy. But ChatGPT was a very in. So OpenAI is like, they made much more than incremental process, but ChatGPT itself that dropped that that day. Was maybe just somewhat incremental in the sense that it was just like a usable interface to a model that was already pretty powerful. And it helps us understand that GPT three was actually really powerful with uh some instruction tuning on top and actually a chat interface. So that that was like a sort of slow um.

• 1:14:00Speaker 01

  Change of things that immediately made people aware of like, wow, this this is pretty, pretty crazy. So maybe there can be something similar where where an an AI does something that we all didn't expect it to do. And that makes us collectively think like we now have to like pay attention and and, you know. Change things, but.

• 1:14:24Tim Scarfe?

  Yeah, I think the um the locus of AI is is something that we're hinting at as well because certainly no one at Meta. Intended for these issues of social media to to happen. They built these algorithms, they they just kept going one step, you know, let's build like an an advertising system, let's do collaborative filtering and. All of these stuff are just externalities that I don't think that, you know, so it's like this unwitting agency.

• 1:14:48Speaker 01

  Yeah.

• 1:14:48Tim Scarfe?

  Then what is the agent? So like the whole system, including us, we're we're we're the agent. And it's the same thing with AI that we're in a way, we're looking for agency inside the large language model, but like we as a system. We're actually a a weird form of new collective intelligence that no one really even understands.

• 1:15:04Speaker 01

  And that's pretty scary, isn't it? Like, yeah, if eh. Yeah, if if Facebook can be seen as an agent in and of itself and we have built legal structures around who to blame for what, right? But that doesn't mean that they these people that we blame intended that to happen. Um. So that's pretty scary and that can become even scarier when you build a bunch of intelligence, artificial agents that that you cannot. Um. Subject to the same level of societal control that we do to ourselves.

• 1:15:33Tim Scarfe?

  So we last um spoke at NURIPS twenty twenty, uh twenty twenty two.

• 1:15:37Speaker 01

  Yeah.

• 1:15:38Tim Scarfe?

  And I feel that you have. You've moved your position a little bit since then. Can you, can you talk me through that?

• 1:15:45Speaker 01

  Yeah, um, I, at the time, I, I, so I took a while accepting that ChatGPT is cool. Uh, like many others, um, yeah, I was skeptical at first and especially like the amounts of data that it has been shown. And that's also like my recent paper has again moved my opinion, right? I, I for a while also thought they're doing a bit more like, um, less generalizable retrieval than the kind of approximate, um, generalizing they are doing now. And over time, I've just changed my view of, of, um, how promising this approach is.

• 1:16:25Speaker 01

  And I can pinpoint it to, um, a specific thing also that happens is I put out this paper on L L M's are not zero-shot communicators. And at the time, I thought like zero-shot communication is pretty important, right? All of us can do it. We don't need five examples. So I thought, okay, we need to, we need to make sure these models can, um, respond zero-shot to these questions. But later I, I developed this view of them being multitask learners and general learners that you, you do need to find the right way to interact with them. Right? And, and one very salient, um, memory for me was that Andrew Lampinen described it as, I think even on this show.

• 1:17:07Speaker 01

  He said, zero-shot prompting a language model is like walking down the street and shouting to someone like, what is fifteen times thirty-two? And they're gonna be like, you know, who are you? Far off. And that was sort of his analogy for zero-shot, um, zero-shot reasoning, and, and that makes total sense to me. Like just because they can't do something in which your specific zero-shot prompt doesn't mean that they can't do it at all. And it's important for them to do it zero-shot, definitely. It's a limitation. But if they can't do it, you need to try a few shots and you need to find the right prompt. You also don't need to go overboard. You don't wanna do, um, you know, prompt engineering on the test set essentially. But, uh, there's a, the middle ground.

• 1:17:51Tim Scarfe?

  Laura, thank you so much for joining us today. It's been amazing.

• 1:17:53Speaker 01

  Thank you.