What Biological & Social Systems Can Teach us About AI with Nora Ammann, Cofounder of PIBBSS Research

In this episode, Nathan sits down with Nora Ammann, cofounder and CEO of PIBBSS (Principles of Intelligent Behavior in Biological and Social Systems), a research initiative focused on the principles of intelligent behavior in biological and social systems. If you need an ecommerce platform, check out our sponsor Shopify: https://shopify.com/cognitive for a $1/month trial period. We're hiring across the board at Turpentine and for Erik's personal team on other projects he's incubating. He's hiring a Chief of Staff, EA, Head of Special Projects, Investment Associate, and more. For a list of JDs, check out: eriktorenberg.com. SPONSORS: Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries. From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off. X/SOCIAL @labenz (Nathan) @AmmannNora (Nora) @eriktorenberg (Erik) @CogRev_Podcast TIMESTAMPS: (00:00:00) - Introduction to Nora Amman and PIBS (00:03:25) - Nathan's view that society is not ready for rapid AI development (00:05:55) - Nora's framing of AI safety, alignment, and risks (00:09:31) - The problem of epistemic access to future AI systems (00:12:36) - Categorizing different research approaches to AI safety (00:15:46) - Seeking epistemic robustness through plural perspectives (00:19:00) - Expecting AI systems to continue getting more powerful and different (00:21:13) - Nora's "third way" research approach using biological and social systems (00:25:00) - Defining intelligence broadly across systems that transform inputs to outputs (00:27:29) - Avoiding analogies and anthropomorphism when studying AI systems (00:30:23) - Using multiple perspectives and assumptions to triangulate understanding (00:35:13) - Translating concepts between domains more multifaceted than just "borrowing" (00:38:15) - Learning from interdisciplinary practices like bioengineering (00:42:05) - Key questions about the nature of intelligence and agency (00:46:38) - Understanding hierarchical levels of agentic behavior (00:49:32) - Aligning goals between individuals and organizations (00:53:27) - Assessing the validity of assumptions and analogies about AI (00:58:57) - Informing AI interpretability and evaluation with theory (01:02:12) - Modeling interactions between AI systems (01:04:56) - Using ecology as a framework for AI interactions (01:07:46) - Developing a computational ecology theory (01:11:45) - Learning from historical examples of societal transformations (01:16:59) - Unprecedented speed and autonomy of AI development (01:19:00) - PIBS programs for connecting researchers across domains (01:23:39) - Seeking researchers to tie theory to concrete AI applications (01:26:45) - Need for productive disagreement on AI risks and solutions This show is produced by Turpentine: a network of podcasts, newsletters, and more, covering technology, business, and culture — all from the perspective of industry insiders and experts. We’re launching new shows every week, and we’re looking for industry-leading sponsors — if you think that might be you and your company, email us at [email protected].

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  The question then arises, okay, like, how can we at all make systematic progress towards understanding the future of AI systems where we do not have empirical access? We still want to make progress somehow, and ideally, we want that progress to be systematic and not just, you know, getting lucky. You should always be suspicious of the analogies and proxies you use. But if you use a bunch of them and they sort of like converge or sort of like agree on some aspect, you should be more confident about those aspects. The sort of like pluralistic perspective and seeking for sort of like at this systemic robustness through plural perspectives is what I'm after.

• 0:33Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life and society in the coming years. I'm Nathan Labenz, joined by my co-host, Eric Torenberg. Hello and welcome back to The Cognitive Revolution. Recently, I had the honor of being invited by Jan Taulen, leading AI investor and philanthropist and former Cognitive Revolution guest, to serve as a recommender in the Survival and Flourishing Funds' fall 2023 grant making process. Over the course of two months, five

• 1:18Nathan Labenz?

  other recommenders and I worked our way through roughly 150 grant applications. And yes, I did read every single one, conducting in-depth research and interviews with the groups that we considered most promising. And ultimately, we used a formal deliberation known as the S Process to officially recommend grants to Jan and the fund's other donors. In the course of that project, I had the opportunity to meet the leaders of a number of AI safety-focused nonprofits. And now that the grant process is complete and the checks have been sent, I'm planning to invite a number of them on the show to talk about their various approaches to advancing our understanding of and preparedness for AI risks.

• 2:00Nathan Labenz?

  Contrary to some recent conspiracy theories you might have seen online, I have found this space to be remarkably transparent, with funders regularly publishing their grants, and this was the case for the Survival and Flourishing Fund grant process, and also the grantees themselves, often publishing their plans and status reports. So today, to kick off this series, and we will be weaving this in with other types of guests over the next month or two, my guest is Nora Aman, co-founder and director of PIBS, a research initiative focused on the principles of intelligent behavior in biological and social systems. Regular listeners to the show will know that I often worry that most of us, myself included at times, are still thinking too small about AI.

• 2:44Nathan Labenz?

  Because not only are the AI systems fast becoming more powerful, but we're also deploying them at a faster pace than any previous technology has ever been deployed. What this will ultimately entail, I of course don't know, but I do expect all kinds of surprises. And so with that in mind, I was excited to meet Nora and learn more about her work. She really impressed me as one of few people who is thinking appropriately big about the future of AI, while also maintaining an impressive level of epistemic humility and rigor. Her work, as you'll hear, is an attempt to create an entirely new and highly interdisciplinary approach to understanding the future of AI. And while she's still early in that process, and the research is fundamentally challenged

• 3:29Nathan Labenz?

  by the fact that the AI systems of greatest interest don't yet exist, I'm certainly among those who believe that timelines to some form of transformative AI could end up being quite short. And thus, I consider her work to be both important and urgent. As always, if you're finding value in the show, I'd ask that you take a moment to share it with friends. This episode in particular, I'd suggest sending to a brilliant friend currently working in a more mature field whose brain power might be more valuable if they applied it to the study of AI. And with that, I hope you're inspired by this conversation about new angles on AI safety research with Nora Aman of PIBS.

• 4:11Nathan Labenz?

  Nora Aman, co-founder and director of PIBS and research associate at the Alignment of Complex Systems Research Group at Charles University. Welcome to the Cognitive Revolution. Thank you so much. So I'm really excited to talk to you because I think you have a really unique and interesting perspective on all things AI safety and AI alignment. And, you know, one of my big ideas, and honestly, the reason I do this show is because I think broadly, we as a society are not really ready for what is coming at us with the rapid development of AI systems.

• 4:54Nathan Labenz?

  And I think we need a lot more people bringing their unique perspectives to the challenge of figuring out what is it that we are creating and, you know, what is likely to happen as a result of all this? So, you know, you have such a unique angle on that, and I want to kind of unpack it in depth. But hopefully, you know, one of the things I hope we can accomplish here is to maybe inspire some people to think that, hey, you know, I don't have necessarily a background in machine learning, but maybe I still actually could bring a useful perspective to this grand society-wide, you know, defining challenge of our time of figuring out like what's up with AI.

• 5:38Nathan Labenz?

  So hopefully, you know, we will, as we get into all the different angles, you know, that you approach the problem from and all the work you've done to start to bring people from various backgrounds into the study of AI, hopefully will inspire some others to come along. For starters, how about maybe just giving us a introduction to the way that you think about AI safety, AI alignment, you know, let's get the Nora take on this framing question.

• 6:11Speaker 00

  I think there exists quite a few different frames on like what AI risk is about, or maybe what AI safety and alignment is about. And I don't want to frame this as sort of like the right or authoritative way of thinking, but I think it's actually productive to also play with different frames on this. But there is the one way I like to think about it that is like sort of erring on the more like pragmatic or deflationary sense. So I'm like, okay, cool. Like in very simple terms, what's the main thing here? And then, and then we can still build on, you know, sophisticated theory of what exactly is the worry. But like, yeah, in the most simple terms, I think the way I look at the space is something like,

• 6:51Speaker 00

  first, well, it seems like we are building these systems that just like becomes sort of increasingly powerful, increasingly capable, increasingly sophisticated in how they can act in like complex environments. So like, that's the thing that seems going to be going on, you know, chat GPT coming out, et cetera. And then I think on the other hand, there's like also like a tendency for these systems to like come to be in positions or come to be used in ways that like would like make increasingly consequential decisions. And then as sort of a third premise here is like, we don't actually like, in a very technical sense,

• 7:34Speaker 00

  have not figured out how to like tell these systems to do anything we want reliably. We don't, we don't know how to do that. We, you know, like they, they do sometimes what we want and, and, but sometimes they don't do what we want. And I think the combination of these three things is, is really worrying. So like the more we sort of like automate big, important, critical decisions away to systems, we don't know how they act. We don't know how to like. Make do things that we want reliably, that seems problematic. And I think that's sort of how I like look at AI risk at the very sort of like faraway perspectives.

• 8:14Speaker 00

  And then I think from there, we can sort of dig more deeply into like, cool, like, why in the first place is it hard to let get these systems to what we want reliably? Like, why is this hard and not easy? Like, yeah, why is this not not easy? And then I think there you can get into like a much more fine range or like detailed sort of discussion about what are the reasons for this? Is this in fact the case? And then if so, what are the reasons for this, which can maybe provide one frame towards like, what sort of, what are the obstacles to be overcome? So that we could have AI systems that we can like trust in how they act or or understand in how they how they act.

• 8:49Nathan Labenz?

  You know, when people talk about AI safety, and unfortunately, I think the the discourse around these topics is kind of growing coarser and more polarized at the moment in unfortunate ways. And one of the things that people often say is like, oh, you're just kind of making everything up. And, you know, this is all super speculative and these systems don't even exist yet. And, you know, therefore it's all fanciful and we shouldn't worry about it. And I think you have a great way of reframing that that is both a lot more prudent in my mind,

• 9:28Nathan Labenz?

  but also invites new and more productive ways of thinking about the problem. And that's your framing of epistemic access. So tell us how you frame this problem of thinking about, you know, trying to get a handle on these systems that we haven't even invented yet.

• 9:48Speaker 00

  Yeah, so the key idea here, and this is also sort of maybe the like main motivating sort of premise for like PIPs, like one of the projects I'm directing. It's just the idea that. Cool. So like, we are, we are like building increasingly advanced AI systems. Maybe in some important way, the like, the systems we might be most worried about. Like the systems we like least understand and also might be like most powerful. Those systems don't necessarily exist yet and or sort of like. We maybe don't just wanna do like trial and error um to like figure out how they work. Because because they on the premise here like would be um like really powerful or like.

• 10:32Speaker 00

  Could potentially already have like a lot of um like potentially harmful impact. And what what this describes, I think, is like, yeah, what I call an epistemic access problem. And this refers basically to just like looking at what is the the like normal typical way for like. Science to make progress in understanding like. Science in its core sort of like depends on empirical access. And to a phenomenon phenomena in order to be able to falsify your hypothesis. Or in engineering, sort of that trial and error approach, you like, you try something, it doesn't work. You try a slightly different thing, it still doesn't work. You try yet another thing and it works. But that's kind of like, that's like in some sense, the core of the machinery of our like collective societal.

• 11:16Speaker 00

  Sense-making, knowledge-building machinery. And um the premise here is that like which with questions regarding the future of AI. We don't have access to sort of the basic gears of that machinery. Nevertheless, like we wanna, we wanna understand something about what what's gonna, what's gonna happen and. How to like make what's gonna happen go well. The question then arises, okay, like, how can we at all make systematic progress towards understanding the future of AI systems? Where we do not have empirical access. And and then I think from there, sort of like that, having that as the problem statement, from there we can be like. Well, cool, like, we still wanna make progress somehow and and. I really want that progress to be systematic and not just, you know, getting lucky.

• 11:58Speaker 00

  And then I like think about maybe the like research base on uh that that is like. Trying to tackle about these questions and like.

• 12:07Speaker 00

  Something we can we can divide up in in different and spaces depending on how they they tackle that epistemic access problem.

• 12:15Nathan Labenz?

  So tell us how you categorize the approaches to this today. And I think you're in a sense creating your own like third way. But to to kind of contrast your, you know, field building approach against those that have emerged already. Why don't you start by telling us how you kind of conceptualize the approaches to trying to understand this AI future that others have been pursuing so far. And then you can, you know, tell us about the the new paradigm that you're developing.

• 12:47Speaker 00

  I mean, this is again, is not the like, the only way of of thinking about or carving up the the research space. But I think it's like one productive way. The idea is roughly, okay, so we have this epistemic access problem and different research or like epistemic approaches. And try to tackle this epistemic access problem in different ways. So for example, I think maybe the most prominent way that is like going on is what I would roughly refer to just as like the most typical sort of like ML technical safety types of approach. Well, here, the key, the key assumption that goes into this epistemic approach is to say that the sort of systems we are most worried about, like the sort of future AI systems we were interested in, they will in important ways be similar to the state of the art ML systems we have

• 13:32Speaker 00

  at the moment, such that understanding more about these state of the art ML systems, understanding how they like the ways in which they may be unsafe, understanding how to make them safe, all of these sort of insights that will translate to these these future systems we are interested in. So assumption being, in in some important ways, state of the art ML is similar to the future systems, thus our insights about current systems will translate to future systems. Another way, another research approach that is also used makes maybe another makes another basic assumption. One I often talk about is mostly being about. Sort of trying to think about, okay, let's imagine sort of like an idealized rational agent.

• 14:14Speaker 00

  Let's assume that highly advanced AI systems will act pretty much like these idealized systems. You might argue for this either because you think the future advanced systems we are thinking of, they are just very powerful, so they're like, just in practice, will come very close to like acting like that, these idealized systems, or because you think this is just like a good way to get started. Like, I think, like, idealizing is something that is like happening across the sciences, right? You like, you start with like an idealized, like simplified model, and then you like, maybe over time, try to like nuance and, and make it more realistic, but like, that can also be a productive epistemic approach.

• 14:54Speaker 00

  So I think that's, that's like research directions that are often informed by, like, maybe formal epistemology, decision theory, rational actor models in, in economics, and then making arguments from this. So like, a bunch of the sort of very classical discussion on AI risk in particular, where we're like talking about, uh, basic AI drives, um, something that Omongro introduced, I think in twenty twelve, or like instrumental convergence, power-seeking arguments. These are often conceptual arguments that sort of like, look at, okay, like, an idealized, or like, going towards an idealized powerful agent, they would, you know, see these constraints to like acting these in these ways, and like, what risks does that propose?

• 15:38Speaker 00

  Or again, what would it look like for these systems to be set? So I think these are like, these are like, these characterize as like, too important, like high-level epistemic approaches.

• 15:46Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 15:51Nathan Labenz?

  But obviously, that's not all that one could do. So I guess another big, just brief commentary for me is that I always kind of keep coming back to this thought that we're all still thinking too small, and You know, these, there are certainly plenty of big thoughts to be found in, in both of these different approaches that you've described. But it does seem still like there's a lot out there, you know, that is not captured by either kind of school of, of thought so far. You know, for one thing, I, I'm always repeating to this mantra that neither the human brain nor the transformer are likely to be the end of history.

• 16:33Nathan Labenz?

  And so, you know, I don't know how different the AI systems of the next few years are likely to be as compared to what we have today. But I would be very surprised if it's, you know, just a bigger transformer, you know, that is continues to drive progress. And I certainly think we're seeing even just, you know, over the last few months, as far as I can tell, an acceleration of results where new architectures that have pretty profound differences from transformers, especially as it relates to like, you know, essentially kind of bringing recurrence, you know, just, just ways of bringing information

• 17:17Nathan Labenz?

  forward through time that do not depend on this like super finite attention window. That seems to be where like a huge frontier is. And that's such a weakness of the current systems that they have these like very brittle kind of finite memories that that alone, you know, seems like it could really change how the AIs of 2025, you know, look compared to and behave compared to the, the ones of today. You know, on the, on the pure reason side, who knows what will happen there? You know, we, I, I'm old enough to remember when the thought was like, yeah, we're going to get these like hyper rational agents, you know, that are going to, you know, never make mistakes and that's going to be really powerful, but problematic, whatever.

• 18:01Nathan Labenz?

  That doesn't seem to be like so much the kinds of things that we're getting. But then you look also at some of the, the fundamental research that hasn't quite hit the, you know, the deployment phase yet. And you, you do see some like proto agents that look like they, you know, may also have a, a real role to play in the future. So I do think those are, you know, again, both like. There's a lot there for both of them, but even then I would be like, yeah, it still feels like it's a little small. This is what has attracted me so much to to your work. I'm always kind of like, but how is the world going to change in response? You know, there's like, there's so many, all of, you know, everything we've talked about so far, kind of tries to look at the AI

• 18:45Nathan Labenz?

  largely in isolation, you know, and figure out like, what is it? What does it do? Can we look at the internals? Can we, you know, change around how it how it works internally? Can we can we detect what's going on inside? Whatever. And and very little of the work that I see gets into what are the dynamics going to look like as things get deployed? And that is at least what that's one of, not the only, but that's one of the things that I think is really interesting about the the line that you are taking on this. So with that setup, tell us about your kind of third way to approaching the understanding of AIs that don't yet exist.

• 19:23Speaker 00

  Yeah. So like, first, I'm going to do sort of the very broad framing, and then we can get into some of the more more details. So like, what concretely this this might produce in terms of research? I think the very broad framing on like this, like broad epistemic approach is something like kind of the the premise or the underlying assumption that like maybe intelligent behavior as like a phenomena in the world is is governed to some extent by the same or similar principles, irrespective of like what system or specific substrate or specific scale it's implemented in. And if this this assumption is true, at least to some extent, then it's a warrant the idea that like, oh, like, can we go look at currently existing

• 20:07Speaker 00

  systems that implement intelligent behavior in the wild that we can epistemically access and can like insights about? Well, how intelligent behavior is implemented in the wild? Can that help us inspire or actually just transfer insights to the like AI question specifically? So that's and obviously that's like a very broad space, right? Like here, I'm talking about intelligent behavior being like implemented in the brain. And so that's sort of a very standard example. I'm talking about like in sort of biological systems, whether that's like in an evolutionary sense, in an ecological sense, in the sense of like what do my cells do compared to like my entire body?

• 20:48Speaker 00

  And how do like things scale up into tissue and into sort of like this like larger organism? And I'm also talking to some extent about social systems, right? Like several human agents, economic agents interacting with each other, producing sort of complex, sophisticated behavior dynamics that again, we can like look at from this intelligent behavior perspective and like ask like, are there parallels or are there like insights we can use from there?

• 21:16Nathan Labenz?

  Is there a definition of intelligence that you could extract from that? Because, you know, it's really interesting. I'm like, I feel like I'm somewhat intelligent. I feel like cells typically people don't think of as being intelligent. But then you look at some, you know, video of an amoeba running around and it's like clearly doing something, you know, pretty sophisticated. And I don't think people usually think of like the economy as necessarily being, you know, I think we maybe just kind of couple

• 21:48Nathan Labenz?

  the notion of intelligence to like our subjective experience. But you're obviously working with a somewhat broader definition. Do you have a definition of intelligence that you use?

• 21:58Speaker 00

  Yeah, I'm definitely working with a broader notion of intelligent behavior here. And I think in particular, I'm like, in general, I'm coming at it less from a place where I'm like, oh, I like know what the definition is and this is the definition, this is what I mean by it. And more like, cool, so there's definitely stuff going on in the world. That's like complex and sophisticated and doing, doing things. And like, some of it looks quite like intelligence as I would naturally think of some, some a bit less. But like, what is the difference between these things really? Is there a bright line? Insofar as there isn't like, what just really is going on under the hood? And maybe with the sort of like aspiration of like, hopefully, or possibly finding, what I often refer to as sort of like a naturalized account of like intelligent behaviors.

• 22:41Speaker 00

  Like, I'm interested in, yeah, like, how how is this implemented, right? Like, like, insofar as I'm, as a human, can be described as doing intelligent behavior. Like, what is it that I'm doing? And then, how is this similar or different from what other systems are doing? And I'm like, not trying to say, well, all these systems are doing exactly the same thing, or intelligent in the same way. But I am interested in similarities and dissimilarities of like, just like the genetics they're implementing. And it seems like a lot of these systems are able to do what we might describe as like systematic work. Like, an economy is able, to some extent, to like allocate resources efficiently. Like, that's like, if you would give me the task to do that, I would have to like think

• 23:26Speaker 00

  really hard and be pretty bad at it. Like, there's sort of work involved. And like, um, like an organism sort of like developing from like a a single cell into like a fully developed organism, like, has to figure out which which parts have to go where and like, you know, like, what what were them proteins to produce, etc. Like, that's, you know, that that is something, there's like work happening. How is this implement, how how is this actually happening? Um, and is there, yeah, is there like shared principles to how this how this occurs?

• 23:55Nathan Labenz?

  Sometimes I just think of like inputs and outputs, you know, I that's kind of how I I always have a habit when I'm talking to somebody who's developed, you know, whether it's a new research product or a project or a a commercial product or whatever, I try to ground the discussion a lot of times and just saying like, okay, what are the, you've made some AI thing, what are the inputs and what are the outputs? And, you know, then we can kind of get into how it's doing that transformation. Maybe that's one, you know, kind of not too presumptuous way of thinking about, you know, there there's something in the middle between inputs and outputs that seems to be doing some like, you know, not always predictable, but, you know, useful for some purpose, um, transformation of these inputs to outputs, whether it be, you know, that

• 24:38Nathan Labenz?

  it's obviously can be like behaviors or literal, you know, literal products. But I like the way you kind of resist some of these.

• 24:50Nathan Labenz?

  Labels or, you know, familiar terms that that people, you know, just kind of use perhaps without fully, you know, grappling with what they contain or what they imply.

• 25:00Speaker 00

  And and maybe if I could just like mo- motivate that a bit more like I think I think this is quite crucial to do insofar as like the very thing we're confused about is like what sort of thing will future AI even be like what sort of like process will this instantiate? And if we start with like, you know, pretty like anthropomorphic concepts to start with, like, I'm just, I'm just not particularly confident that this will be like, like, it might turn out right, but like, I would like to be right for the right reasons, right? And not not by luck. And I think there's some arguments just coming from a place of sort of like, cool, like, our minds have evolved in an evolutionary context that like,

• 25:40Speaker 00

  make me good at noticing phenomena at a certain scale, temporal and spatial scale, but not a different one. And like, there's like phenomena that happen at much slower time scale that they're just not, you know, they don't get propagated into my attention or at much faster scales and I can't see them. And I think if we have them like, and I think so that there's sort of like anthropomorphism creeping in here, like there's some things my mind is trained to to see and be able to understand or like notice. And yeah, like, what if an AI system is will be sufficiently alien or instantiate so really alien behavior that I won't by default see it? I like will actually need to like enhance my ability

• 26:23Speaker 00

  to like sense the world around me with these sort of scientific theories or or like ways of measuring things that will allow me to like see beyond my sort of like evolutionary prime sort of special and temporals scales.

• 26:40Nathan Labenz?

  Yeah, it's quite, it's quite a high-wire act, really. I find, you know, for myself, I I've definitely put myself into the mostly the first camp to date, right, of in the in the three buckets that we've used here, the first one being based on the assumption that future systems will look like current systems. I'm not always necessarily fully conscious of of that, but I guess I'm also just trying to wrestle sometimes with what are the current systems and what can they do, you know, before even trying to get into the future. And in that line of work, I really try to avoid analogies and.

• 27:18Nathan Labenz?

  You know, because I find often, and you know, again, I have the benefit of mostly focusing on currently existing systems that I can actually bring, you know, the normal procedures of experimentation and science to not always with full access to the weights, of course, but you know, with at least some access to things that really exist. And I find it more often than not, when people try to make analogies or use metaphors, that that ends up confusing the situation more than it helps. But again, with this problem of epistemic access, where you're really specifically saying, I want to try to figure out, you know, what the future might look like. And I'm not, you know, going to assume that it's

• 28:03Nathan Labenz?

  going to look like the present in these particular ways, then you sort of have no choice but to make some leaps. So I think it's, that's why I say it's a high wear act, because you're, you're really challenged to figure out what parts of current ways of thinking are useful. And, you know, at what point do we maybe cross over into, you know, deluding ourselves or confusing ourselves? You can maybe just unpack that at a conceptual level more. And then I definitely want to get into some of the particulars because you know, your, your big project is essentially recruiting talent, you know, from different

• 28:43Nathan Labenz?

  backgrounds and, and starting to refocus their energy on the study of AI. And it's been cool to see how many different types of people you've been able to attract already.

• 28:54Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 28:59Speaker 00

  Let me just give one more sort of clarification, where I think like so far, we've like framed the epistemic approach quite sort of century around, like what will future systems look like? And I think this is like an important motivator, but I also want to like, I want to make some argument about how I think it is very important to look at current systems and actually how some of these approaches are also meant to provide productive insight into current systems. There might be several ways this might happen, right? This might either happen by, so like one sort of more mundane example might be, hey, like neuroscience has.

• 29:34Speaker 00

  Several decades of attempts behind itself, trying to like figure out what's happening in the brain. And what methodologies have been developed, what methodologies have been shown to work more or less well? Are any of those importable to AI interpretability, ML interpretability? And like, so we had like some fellows in the past, basically just making that translation. But that translation, you know, happening to like, like is going towards trying to understand current systems, but taking inspiration from how we have studied intelligence behavior in in other types of systems. And yet another example might just be cool. Like, yes, we will want to look at current ML systems, but like, what are different frames of theories or models through which to like conceptualize ML systems, right? And like, yes, there is

• 30:18Speaker 00

  a low level matrix multiplication thing going on, but what are other things, right? Like active inference, FEP, like free energy principle minimization frames might be like, it might be one example here of like, it's it's a trying, it's self conceiving as being like a theory of cognition. Is this a productive frame to use to look at state of the art ML systems? And if so, like, what does it suggest about their behavior, how they will scale, right? Like, what they will do if they like come into touch with with the world around them? So yeah, I just want to sort of clarify, it's not just about future systems that don't exist yet, but maybe also what are useful frames to like, understand current, currently existing systems?

• 30:59Nathan Labenz?

  And also even, you know, between those two, there's the question of as current systems get, you know, more broadly deployed and begin to kind of shape the environment, you know, then you've got all kinds of dynamics, you know, that I think are very much not yet in existence, but, you know, probably more predictable. And I think that's also a really interesting middle ground that your work touches on in various ways. So when we were preparing for this, I had organized my thinking and the way that I was going to approach the the questions around.

• 31:34Nathan Labenz?

  The different fields, there are different, you know, kind of academic backgrounds that people might be coming from. And I use the term borrowing and I was kind of inspired a little bit by Tyler Cowen, who's, you know, always going around saying, where are the models? You know, we, we, nobody has written down an explicit model of like, what's gonna, what's gonna happen, what's gonna go wrong? And now there's starting to be a little bit of that. And I think you're, you know, again, building the, the field that's going to bring more of that. But you challenged my thinking there a little bit and said like, okay, first of all, borrowing, if I understand it correctly, you're kind of saying borrowing is often

• 32:15Nathan Labenz?

  a little too narrow. It, it sort of suggests that there's like a key idea from a particular field that we can just kind of quickly snap over. And it's unfortunately not usually that simple. I guess in the analogy frame would be like assuming a tighter analogy than in fact exists. So tell me how you prefer to think about the move from, you know, one kind of background to studying AI.

• 32:44Speaker 00

  It's really, really multifaceted, right? Like I don't, I don't, I was thinking a bit about it and I don't have like one single answer to like, this is the thing that happens or something like actually a bunch of different things can happen. Um, but it, but it can try, like there's a, there's a few different things I can give examples of, right? So like, I mean, I think this can range from, you know, very simple, like there's some facts about the world that might be interesting, right? Like size of a million brains, like that, that can be a fact that sort of informs some of our priors around dating about, of course, customer on AI. Or like that might be a fact we take from one discipline towards that's helpful, but it's sort of like, sort of kind of small scale. And I think from there, we can really extend to

• 33:25Speaker 00

  like theoretical frameworks, models that live at like different levels of abstraction. And for each of which we can sort of ask, is this a productive way of like asking this specific question in AI land? And whether you're like, hey, should we use like games theoretic frames to think about what are the strategic implications of like several? AI agents interact with each other and can we like, you know, pour like principal agent concept from economics to like conceptualize what we mean by the alignment problem. Can we use methodology, so like sort of mathematical often methodology, say from like the complex systems to like model complex attractions. The example I gave before, right, like what methodologies existed in neuroscience has then have been used to like

• 34:11Speaker 00

  try to understand what's what's happening in these like highly connected networks. So, so yeah, with the model theories, concepts, methodologies. And then I think there's also something like, like slightly more metal, like epistemic practices. So like, I've been recently reading up on something that I've been finding quite inspiring. And this sort of philosophy of science, history of science work on the field of bioengineering. And I feel like it's a pretty productive sort of metaphor or example case study for some work in AI alignment. And there, I guess, like the main idea here is something like in bioengineering, why do you have on the one hand, you have sort of like high level theoretical work on like, you know, like how different organisms, how different, how cells work, what what is happening within a body.

• 34:57Speaker 00

  And then on the engineering side, you're interested in like being able to do specific things with your knowledge, right? Like, um, control specific things, like create specific products that help us to do something specific that we have in mind. And and bioengineering is sort of like the combination of the two, right? Like, how can the theory help us produce the sort of like pharmaceutics or something that like do what we want them to do? And what you see in bioengineering is that you have to, like, real world organism is really complex, really really high detail, and also sometimes it may be like unethical to just like, you know, fiddle around with it. So like, you rely a lot on building sort of like simulations or models, which you build

• 35:42Speaker 00

  specified by your understanding of the theory. And on which you then do sort of a bunch of engineering work and then sort of only occasionally go like into like actual organism and and um uh like see whether like this worked out. And I think this like conversation between applied and theory that we can let study in in this like case of bioengineering, for example, is a pretty apt um metaphor as well for like, I think some of the like um AI alignment or AI safety work that I'm most excited about, where like, I want I want that to be applied work that like actually has like a concrete bat like helping us make these systems safer, but I also want that work to be like well informed and sort of conceptually rigorous, like we're using a lot of terms

• 36:25Speaker 00

  from intelligence to agency to goal directedness to deception, they don't like really vague and um in order for you to like come up with an experiment that like actually, you know, uh whether the results that you get are actually what you like think they are or something like to get results that you can actually trust, we need for these experiments to be like well motivated and conceptually well rounded. And I think that's like sort of where this like theory and applied um work comes together. And so yeah, that might just be a third example, which more like what was the practice of different fields that help us, you know, do do different types of interdisciplinary work in in AI safety?

• 37:04Nathan Labenz?

  If I understand your approach correctly, I guess there's the sort of intellectual level of the approach and then the, you know, practical approach. Instead of saying, hey, AI is a big deal and we need, you know, a lot of people to think about it. Let's go get some people from here and there and, you know, try to assemble them. It seems like you're starting with questions that you think are important about the future of the AI timeline. And then

• 37:38Nathan Labenz?

  Working from those out to go figure out, okay, well, what are the best, most applicable, you know, most likely to produce useful insight, fields, I guess, is a word, you know, ways of thinking, you know, methodologies, perhaps the better term that we could try to pull in. So do you want to kind of run down some of the top questions that you've been focused on? And then you can like, for each one, you can kind of unpack, like, you know, I guess, first of all, why do you think that's important? And then, you know, where have you gone to find people who might have the right toolset to bring to bear? And then this is probably a good time also to mention, like, you are actually doing

• 38:20Nathan Labenz?

  this, like, you're bringing people together. This is what PIBS is, right? It's a program that that actually helps facilitate people making this sort of leap. So you can also get into specific individuals, if you want, and like projects that they've done that kind of show, you know, some of the early fruits of this approach.

• 38:39Speaker 00

  Cool. So I'm gonna go through sort of different clusters, which is like partially how I'm thinking about this. I guess like one cluster that is quite fundamental is something like, just in sort of quite fundamental terms, understanding what is intelligent behavior, what is like agentic goal directed behavior, what is autonomous behavior, which is which is relevant because part of the premise is that we might be coming to like instantiate these types of behaviors artificially. And of course, if you like get really powerful agentic autonomous behavior, well, that might be dangerous if you don't understand it well. So like, but then also at the same time, right, like these terms are really fuzzy. What do we even mean by them? Can we can we actually do science on them? Like, can we have some sort of grounded mechanistic understanding of what's going on here? How

• 39:23Speaker 00

  do these behaviors come about? And also, what's the logic of how they how they work? And like, so that that is a big cluster. And just like some examples, I guess, to invoke here, I think, I'd say there's an entire cluster on sort of like complex systems, which which itself is like a very diverse cross disciplinary field, but like various methods in complex systems that are, I think, essentially trying to ask, right, like, how does it, how do we like, how do we get agents out of atoms, right? Like, at some point, we just had a world full of atoms, and they seem kind of easy to understand. And then at some later point, we had a bunch of Agents doing things, what's going on? Um, uh, so, so I think these are examples.

• 40:07Speaker 00

  And then like a bit more concretely, like, I guess, like recently we've been, um, talking more with people from the field of artificial life, which like, you know, has like, this can be sort of more applied mathematical work. This can be more like simulation based work, but a, a field that is like essentially asking like, what is agency? How does it come about? Can we like implement it? Do we, can we understand? Can we like thoroughly understand how it works? So Chevy could come to implement it. What does that field has to offer to AI language? Um, insofar as like a lot of the risk phenomena would be worried about have to do with these like complex sophisticated behaviors. Uh, but, but then I think another, another way of going about this would be sort of more from the biological angle, right? So like biological systems have developed these behaviors.

• 40:52Speaker 00

  There is more and more fields like trying to bridge maybe this like biological cognitive angle. Um, where like, well, how do biological systems come to exhibit these like cognitive tendencies? So I've, I've mentioned at the entrance earlier, there's sort of like basal cognition type work that is trying to ask, right? Out of these like very simple systems, how do we, how do we sort of get complexified and, and how quickly scaled up such that to like, um, that they eventually exhibit, um, so complex sophisticated behaviors. Um, and then also from sort of, um, the social sciences, like an example here might be,

• 41:31Speaker 00

  there's a, there's a like a longstanding question, the social sciences asking or in sociology, asking like sort of agency versus structure debate, right? Like if you, if you're trying to explain social phenomena, when should you explain them in terms of like individual human agents? Um, taking action and when should you explain them in terms of like higher level structures? Um, having like causal force. And there's just sort of like, this is essentially in the end, the question of causality, right? Like do, you know, does the market do things to like political institution do things? Or is it at the end of the day, always human individuals? There's like a philosophical question here and there's some extent also sort of like a mathematical question of like, how do you model this? And, and where do emergent, um,

• 42:16Speaker 00

  Causality emerging phenomena, um, trying to understand this. Yeah. So, so yeah, that's sort of like, that's definitely a field where like a, a, a cluster, where like a lot of different fields, I think, have potentially productive insights. Yeah. Understanding like how sort of multi-scale systems uh work. And I guess like maybe one other example, if in here would be something like, how are these complex systems structured? So like, so there's like, in biology, we see a lot of examples of modularity. Um, so you have like modularity brains, um, developing like modularity, not like some sort of systems to be something that uh evolution gets, gets selected for.

• 42:54Speaker 00

  You also see that to some extent in social systems. So like, there's some work asking, where do like the boundaries of firms come from? Like, why are firms, why do firms have boundaries in this way and not some other way? Why are we not just like contracting everyone? Like, that's weird. And like, what are the answers to that? Which I think is like, in some sense, looking as, as at a similar thing, like how does structuration um and, and in the situation happen in different systems?

• 43:20Nathan Labenz?

  Are, are there any like examples of projects that you have brought people together to do in, within that cluster?

• 43:27Speaker 00

  They're all quite detail heavy. So like, um, how do best talk about them? So like, maybe one example would be someone sort of like coming in with the, the premise of like, you know, like there was a time when we were really confused about what temperature was. And we weren't sure whether temperature was a thing or whether there were subtle things going on or like, and eventually we came to be able to measure temperature and like have a sense for like what that phenomenon is. And like, what if the same is true for agency? Like, what, what if like agency is actually a thing, agency, agentic behavior? Like, what, what if that's a thing that could be measured um across systems? It would just be understood properly. And then going into um, like often sort of like physics-inspired questions, right?

• 44:12Speaker 00

  Like, what would it, what would it mean to, to come up to with a solid, solid measure of, of agentic behavior? There might be one angle. Uh, there's another line of work that I'm interested in, which is sort of like related to hierarchical agency or like hierarchical alignment. So like, it seems like we see a lot of systems in the wild that they actually have this hierarchical structure, where there's maybe like, Uh to some extent, agentic behavior happening at different levels of abstraction. Um and you might wonder what level of abstraction is the one you should be looking at? Or like, what's the relationship between these levels of abstraction? So like, one example here might be, you know, there's like individual cells, there's like tissues in my body and then there's me overall.

• 44:53Speaker 00

  And all of this have like, depending on what frame I take, like some claim to be like, this is like a relevant level of abstraction to look at. Um in particular, like usually my cells kind of like are nicely aligned with what I wanna do. But like occasionally you get things like cancer, but it's like individual cells sort of getting like information decoupled from the overall systems. And then like starting to kind of like act against the the goal of the the higher level system. So that might be an example here of like a hierarchical like misalignment. And then you can start to ask, right, like, how A in the first place, do you scale up these levels of interactions or shit to like? Get some like something coherent at the at the higher levels?

• 45:37Speaker 00

  And then also, what are ways that like the alignment gets kept in place or breaks? Um and obviously you like see this in other places, right? You could look at the economy and be like, there's individual economic agents. There's like, you know, high level abstractions that like usually in economics, we think a lot about high level abstractions, like organizations, et cetera. Uh so so so that's a line of work um which you can tackle both conceptually, both from the like biological angle of like, how does this happen in biology? And from the like math complex systems angle, like, how do we even model this um this sort of phenomena?

• 46:14Nathan Labenz?

  Is it uh too soon to ask if there are any insights that um, you know, are simple enough to impart to people?

• 46:22Speaker 00

  Yeah, I I think like very clearly results are like. Struggle just like in simple terms pointing at them. I do, I do think this is like a frame on even like formulating what we might mean by air safety or air alignment, that I think is a productive frame among other ones. So like, if some people are maybe familiar with this, like, um, some of the arguments for like why air risk might be concerned, for example, what is called an inner optimizers or meta optimization. So like you, you sort of a dynamic where you have like an, an, an outer sort of optimization process, but that sort of spinning off, like an inner optimization process that can come to have goals that are like misaligned from each other. And sort of a simple example of this might be potentially sort of like

• 47:06Speaker 00

  uh, evolution overall, trying to optimize inclusive fitness. And then like that's spinning out and agent like humans or a species like humans that have goals of their own. So like that would be like a simple examples with some nuance uh neglect this year, but that seems to be like an example of a hierarchical thing in a similar way I've described it before happening. So like getting an answer to like, what are the like keys to alignment between hierarchical like levels? Um, will potentially be productive here. Or like, I guess another frame on this could be something like we, we have developed in end of the twenty, twentieth century like language to

• 47:46Speaker 00

  talk about the strategic implications of multiple agents interacting with each other at the same level of interaction, which is game theory. Which we can now use like, talk about what might, what is happening in different situations where we get multiple agents uh interacting, you know, what, what would something like vertical game theory look like here? Like what can we, can we come to say something principled is about what are the strategic causal interactions between agents at different levels of abstraction? Um, can we have like a mathematical framework that allows us to think about this more productively?

• 48:20Nathan Labenz?

  So this is all under the, the cluster of questions around like, where does agency come from? You know, is it something that we could potentially have a quantity measure for? Is it inherently hierarchical or whatever? And then if you were to take this same question of AI agents to somebody who is an application developer, they would say, well, I know what an AI agent is. It's a language model that I give a goal and put in a loop and kind of allow it to take a step and then get some feedback from the environment and then just have it take another step. And I just kind of keep looping until I either hit my goal or I get stuck long enough that I abort.

• 49:04Nathan Labenz?

  And I guess I wonder, like, how do you develop an intuition for which assumptions or which sort of, where are the valid analogies and where are the invalid analogies? You know, I also look at a transformer and I'm like, in some sense, you could say this is hierarchical, but in other ways, it's like kind of one directional. And it's certainly not that much like the hierarchy of our bodies, right? You know, where there's sort of these clearly different scales.

• 49:44Nathan Labenz?

  And it seems like the higher level scales largely can control or direct, but also take feedback from the lower. It doesn't seem like that kind of stuff is going on in a transformer. Maybe it is, and we just can't see it or it's happening in some sort of weird smeared out way over the course of the forward pass. But all these questions are so interesting. And then I'm still at the end, I'm like, how do I know how much of this I should believe? How would you kind of coach me in developing my sense for that?

• 50:14Speaker 00

  The whole point about analogies or sort of proxies is that no single analogy or proxy is true. Like, that's what we mean when we say analogy or proxy. Like, instead, I think the thing to go for is that when we're very confused about a phenomenon, we have sort of limited immediate access to it. And it's like, you know, the territory is very detailed and we only have maps to try to think about it. And they're never, they're never perfect.

• 50:43Speaker 00

  Instead of being like, what's the perfect map that will give me all the answers I want? What I'm interested in is Something one might describe as like a sense of like epistemic pluralism, where if I get a bunch of different proxies, a different maps, different perspectives on the same phenomena, and I can sort of triangle it between them. And then I can come to form what are hopefully more robust guesses about what actually holds about this system. So the more sort of like independent from each other perspectives I have on this thing, but like, say, all of the perspectives do suggest that like feature A is something and like only one of those perspectives that's feature B is also something. I have more confidence that feature one is in fact the thing. And because I think we sort of like

• 51:26Speaker 00

  lack the sort of like, this is the one true map for understanding this this phenomena. And the sort of like pluralistic perspective and seeking for sort of like epistemic robustness through plural perspectives is is what I'm after. So that's one met chance, right? Like you should always be suspicious of the analogies and proxies you use. But if you use a bunch of them and they sort of like converge or sort of like we on some aspect, you should be more confident about those aspects than about others. So that's one. And then I guess there's like another thing to say here, which is like,

• 51:59Speaker 00

  there's already different types of analogies, right? It could be like, well, we can like loosely talk about, oh, in what ways is like a language model like a human sort of like speculated about that. And then there's like another way in which I'm like, oh, we could like

• 52:13Speaker 00

  have some sort of mathematical model and be like, how would we, how would we like describe this phenomenon in question through this mathematical model? Different levels of precision, uh, different, different sort of epistemic statuses to both of these things. One, the former maybe like more geared towards inspiring new hypotheses, which is like a valid thing to do, but like you're just, you're just generating hypotheses at this point. Um, and another maybe more geared towards like falsifying whether like a specific hypothesis is productive or not.

• 52:47Nathan Labenz?

  Maybe another way is I'm trying to put this into my own words. Another way I might frame it is a lot of this work kind of takes the form of if this, then that. And we don't know what value of this. Is going to pertain at various points in the future. But by like doing a bunch of if this, then that sort of work, we can hopefully be more prepared and be ready as new things start to emerge and say, okay, what is this? And hopefully have a bunch of ready analysis for a bunch of different, you know, possible this is that we may encounter.

• 53:28Nathan Labenz?

  Um, so, you know, better to go push all those frontiers in advance, even not knowing which one will will actually pertain. But it's it then just becomes really important to get clear on like, what are the assumptions? You know, and and what would we need to see in a future AI system to have confidence that this way of thinking about it is going to be productive?

• 53:57Speaker 00

  That's right. And like, just maybe giving a bit more, like one slightly more applied example still. Like I am definitely on the more concrete end. One of the things I'm most hopeful about is this sort of theory being able to like productively inform both interpretability and sort of like eval evaluations type work. So like in evaluations type work, what you what you're doing essentially is be like, okay, here is a risk phenomenon. Here is like some sort of dynamic this this AI systems could produce that we would be worried about. Can we find it in the AI system? And for that to like happen in a way that I would, you know, would be somewhat confident it actually succeeded, we need to have a good characterization of of what we mean.

• 54:42Speaker 00

  And if you're like trying to look for, does the system, you know, act autonomously, pursue goals over like long horizons, is deceptive, like you need to have like a good account of what you mean by deception. And it's actually pretty fraught. It's actually pretty hard to like specify what we mean there. So I think that's like. Can we come up with frames, accounts here that can help us do evaluation work better? And the other, the other similar application is something like in interpretability, you also to some extent, I mean, you can do very bottom up interpretability and just be like, what do I find? But you can also be like, what am I even looking for, right? Like, am I looking for planning? Okay, cool. Like, what would that even look like?

• 55:27Speaker 00

  And these questions is just like, gets hard and, and having sort of like an epistemic plural way of informing how to interpret what you're finding, seems, seems productive. And I think that's like at the more, at the more like applied side of, of the spectrum.

• 55:42Nathan Labenz?

  So you want to move on to some other clusters of questions?

• 55:46Speaker 00

  Yeah, I guess an important other cluster, it's, it's in a similar way of just like pretty fundamental, trying to understand what we're dealing with. But like a bit more focused on like, I guess like minds and cognition in general. What I had in mind here is like, most of the like, brain sciences, right? Like neuroscience, cognitive sciences. And then to some extent, I guess like philosophy of mind, um, comparatives. And studies between like animal, human cognition, right? Like how is like cognition evolved, evolution either of like some work on this? Can this inform our understanding of what we even mean by doing higher level cognition? So should again, we have sort of a grounded way of thinking about this or a more grounded way of thinking what is in the air case? This can then go to different places, right? From I've, I've mostly mentioned, applications for interpretability.

• 56:31Speaker 00

  How do humans come to be pretty good at deductive reasoning and arithmetics? How does that, how is this similar and different from like transformers? What about the ways in which transformers are brittle, like or kind of brittle in this, this way? Do, should we just expect they're like, stop being brittle in this entirely? Because like, human brains also are just big connectoms. What's going on here? Where do motivations come from, right? Like, where do intrinsic motivations come from? Could we understand this process well enough to just engineer motivations, intrasystems? That then sort of are reliable or like, what are the limitations to that? How does theory of mind work, right? Like, higher order animals all seem to have this capacity to like model other agents. That that's kind of a wild thing.

• 57:16Speaker 00

  AI systems with theory of mind, we're getting into sort of like maybe higher order capabilities around deception or situational awareness. Like, what's that like? Again, human minds seem to do a lot of this. What is it? How do we identify it? How might it come about in AI systems? There's also some work happening, Steve Bryant is like working on like brain inspired like alignment proposals, right? Like taking this like brain analogy really seriously and being like, what would it now look like to like actually make AI systems safe based on that? So like, not just interpretability, but safety proposals. Um, so that might be another, another application. One other, another cluster, and you have actually alluded to this several times, so I'm glad I can actually, and finally mention it as well. I think there's something really important about multiple agents interacting.

• 58:01Speaker 00

  Definitely classically, the AI risk discourse has focused a lot on like the way a single, um, advanced AI systems could be pretty harmful. And I think this is like relevant in many ways. But I also think there's like important reasons to expect that what we will see in coming years is a bunch of different AI systems interacting with each other or AI systems interacting with a bunch of humans. Can we come to understand what's gonna happen here in a more principled way? Um, one, one frame of this I kind of like is something like, you know, interpretability basically is like trying to look into the system and understand based on its internals, what does it do? How does it behave and why?

• 58:42Speaker 00

  Um, what I want is kind of like interpretability, but for interacting agents. And I think we're currently extremely far away from that. We're currently even like this is like very, so we had some fellows last summer working on this and I was very excited about this. But it's also very evident how sort of pre-paradigmatic we are, right? Like, it's, it's already hard to study complex interactions because in order to make it more tractable, you need to simplify. And then you need to be like, am I simplifying away something that's important to understand what's actually going on? So that's already hard. But then now in the AI case, you're trying to model how different AI systems interact. But like, we're not even sure. You know, like what we mean by AI system, right? Do we mean L L M's? Okay, cool. Like, we can

• 59:27Speaker 00

  put in G P T four and be like, how does G P T four interaction with each other, with like several instances of each other, work out? Or like, different, different like characters within the same language model, or different like architecture for AI, right? Like, it's, we're even confused about what even AI, what we mean by AI, what sort of mind is it? We're like very low, like, very weak theory on what, how even to like, understand its behavior. Well, if we model humans interacting with each other, we're also very complex. But I think sort of our like, full psychological assumptions about how we might interact with each other while sometimes problematic, we still have better reasons to expect that they are sometimes valid. Um, but

• 1:00:12Speaker 00

  our intuitive assumptions about AI systems and language models, I think they're just like even more fraught and even more likely to be unjustified. So like, I think this is really hard to, to make the progress on this. But um, there's various reasons why there might be a lot of risks just coming from the emergent dynamics between interacting models. Um, and like, some work has been done on characterizing the risk from this. So yeah, can we, you know, either like use game theoretic um, models, can we use like, something like ecology, right? Ecology is interested in different types of species interacting over a set of resources, what are the dynamics, what are the reproductive dynamics, et cetera. So that could be a frame, or obviously like economic frames are looking into this.

• 1:00:56Nathan Labenz?

  I don't know a lot about this, but I've always kind of thought that ecology seems like a really good frame of reference, just because the different forms of life are very different, you know, and whatever ecology is doing, it is at least forced to find some way of integrating into some, you know, single account that all these different things. And I mean, you think like, good God, you know, you've got. An octopus and a human and an elephant and you know, and that's not even that far flung in terms of how different things can be.

• 1:01:36Nathan Labenz?

  And so maybe, you know, you could kind of fit pretty weird AIs into those frames as well. I know you have had some folks who have come to the PIBS program from ecology type backgrounds. Any kind of, again, like results or sort of frames that they have developed that you think have been, you know, useful for developing your own intuitions for what might happen in the future?

• 1:02:00Speaker 00

  It's definitely early days and I think there's not that many examples of what we've had so far. So like, I have one good friend who's starting a PhD in ecology, who's like interested in this, so I'm like, kind of interested in that and I think has gained some insights from that. And then we had some fellows last summer coming from a sort of evolutionary and ecological perspective. So I think those people have mostly been working on like, okay, can we even come to find good, like, experimental setups that we then can eventually run and give us good insights? But like, one of the main high level insights is like, it's really hard to like come to make good choices here, right? Like, if you let LLM systems interact, what's the analogy here for like the environment?

• 1:02:44Speaker 00

  Why, like, what are the resources they, like, in typical ecological models, you often think about space, right? And like, you want to have like, control over different space, control of different resources, what's the analogy here in like textual space? What are, what are we, what are we trading off? What are, what are they like coordinating over? How do you, how do you run these experiments? Just all very early days. I think another meta-level insight here is something like, a challenge to translation here is that, I mean, I think this is definitely coming, but traditionally, I think ecology still is very, you know, informed by going into the field and being like, looking at the actual animals and tracking and what's happening there. And I think

• 1:03:29Speaker 00

  This is like not where we at um for what we're trying to do here is we're sort of more talking about as a computational ecology. Um type direction and that doesn't really exist. Um and I think it needs sort of like. Bulls and individuals with like who happen to find the freedom in their academic context to be like, hey, like. I want to combine these insights from ecology with these questions in in physics and math and and AI and computer science. And then you need a supervisor or a faculty or so that's like, you should do that. So like, that's that's that's not a given. Yeah, a a string here that is like a bit.

• 1:04:09Speaker 00

  Theoretical, but I think that I'm interested in is specifically better understanding methe evolutionary um dynamics. So like, I think we have like, people in general have a pretty good sense for like. Evolution and like what sort of like pressures evolution creates and what that predicts about. Or like how that shapes our expectations for, you know, how different systems get shaped. But there is even in the in the field of biological evolution, sort of like a lot of still open and unclear work on like. Are evolutionary pressures themselves subject to change? And on the what logic and what does that mean for then the organism or the like evolution of the organisms?

• 1:04:49Speaker 00

  Um and and is that relevant to understanding future minds to do understanding like the sort of. And pressures that act on what motivations and type of behavior we will see emerge from AI systems? Um, yeah, I think I think this is conceptually very interesting, but also still on the on the on the conceptual side and not very applied yet.

• 1:05:13Nathan Labenz?

  What kind of techniques are you seeing people explore? You know, I can imagine.

• 1:05:20Nathan Labenz?

  Word-based analysis, you know, just like a language level. I can imagine pure math, you know, kind of pure, pure analysis. And I can imagine simulation, I guess, just to name three. What sorts of approaches are you seeing people take? And, you know, what seems to be kind of most fruitful so far?

• 1:05:41Speaker 00

  I've seen some early examples of all of these. They all seem potentially relevant. I think it's, it's hard. And so like textual analysis, right? Like let different instances of L L M's interact with each other in, in prompt them in some way iteratively. And, and see how, you know, this, the sort of thing they answer evolves. And that is, you know, a thing to look at. And you like, in principle, seem like you could get at what are like the, the dynamics of interaction that come out of this. But I definitely have not in so far seen an experimental setup here that like has given me much new, you know, Bayesian evidence towards something being true versus not.

• 1:06:26Speaker 00

  So like, I think I'm conceptualizing this mostly as like, this seems clearly work that's warranted. This seems clearly something we need to look into, but we're not really at a place at the moment where I'm, I'm seeing results that like really make the update drastically. Mathematical modeling, most, I guess, of what I've seen is something like formalizing world would even look like to like, use a specific model on AI interactions or on the like modeling AI systems through that. And then sort of discussion about, you know, how you would apply it and what, what are maybe the, the confusions or shortcomings of that. And that again seems productive and I feel excited about it.

• 1:07:08Speaker 00

  Not very concrete things to point at in terms of results. Simulation, yeah, I think this is one of the directions I'm pretty excited about specifically to start to inform evaluations. I think a lot of this is about how do we set up? How do we like design experimental setups for current or like? State of the art M L systems to like with any meaningful confidence device. Are they capable of X? Are they not capable of X? Um. Under what conditions are they? What sort of like capabilities can you listen from them? When they start to interact, not just in some artificial sandbox, but like with lots of environmental affordance from the wider worlds.

• 1:07:50Speaker 00

  And how much do their capabilities um shift when deployed in in the wide world? That has like a lot of, you know, detail and affordances that they might not have. And so like, what would what would it look like to have a good environmental setup? Eh experimental setup to elicit this? That's something I think on the like shorter time lines front, I'm like most excited about. And work is in progress there. I mean, from sort of like evaluations group like our our key ball. Trying to like, see whether like these models can reliably self replicate, right? Like testing this um to like trying to, you know, like, what would it look like to test whether uh and an A system evolves evolves something like situational awareness.

• 1:08:32Speaker 00

  I'm like kind of excited about um someone working with us who's like think has sort of like early stage uh stages of the theory of what? How to, how how we might conceptualize situational awareness sort of mathematically? So that can then inform what sort of like environment eh eh experiment would you have to design to like? Posibly illicit this this phenomena in A I systems.

• 1:08:57Nathan Labenz?

  Are there any historical moments that you think we could learn anything from? In terms of just other times in history where we were confronted by something very new and, you know, had to kind of figure out or, you know, were challenged to figure out what it might imply? I mean, there's a lot of analogies that people make between, you know, A I and the industrial revolution. And, you know, I call this show the cognitive revolution kind of on that basis. But I I don't really know much about my sense is that our our track record isn't great. You know, if you like looked at the predictions from the beginning of the industrial revolution. You know, hopefully we have better tools now, you know, and better frames to bring to it. But are there any historical moments where?

• 1:09:44Nathan Labenz?

  People collectively were able to see around the corner and get a handle on what was to come?

• 1:09:50Speaker 00

  I guess an answer will come in degrees, like not shakily. So I guess what I'm interested in here is maybe something like, I think industrial revolution and some other examples, electricity, whatever, seem like pretty good examples of like transitions of innovation that has like shaped society pretty transformatively. And I think there's things to learn from this. A transition that I'm mostly think we haven't faced so far as a society in which I think is like one way of thinking about what we're facing at the moment is us actually becoming able to artificially instantiate highly complex intelligent in the sense of like autonomous, goal-seeking behavior,

• 1:10:30Speaker 00

  which is like in some important senses, different from inventing electricity, inventing specific tools. That like, that really forces us to gripe with these like very complex phenomena of like, there's some recursive thing going on where an atom, like where a bunch of atoms start to like act in some coherent ways over long time horizons, are able to self-maintain and, you know, against dissipative pressures and sort of like act in such a way that presses through the nose, right? They like seem to like pursue things despite, you know, facing environmental obstacles, like

• 1:11:12Speaker 00

  something really complex is going on here, something that's like on its own terms, able to like shape the world around us. And I don't think we have like meaningfully dealt with this sort of thing. I mean, like, you know, there's like non-human animals around us that have some of these capacities, but our approach here was to control them pretty reliably. And so like, not sort of like in a way that they like clearly spin out of, spin out of control or understanding. And so I think that's like a historic, like one way I'm thinking about the current historic moment or potentially the current historic moment that I struggle to find analogies to.

• 1:11:53Nathan Labenz?

  Yeah, unfortunately, as I listen to you, my mind goes to like, you know, the Neanderthals didn't necessarily see us coming. And, you know, we could even interbreed with them, but still somehow.

• 1:12:07Nathan Labenz?

  We were different enough that, you know, it didn't, didn't go so well for them in the, in the grand soup of history.

• 1:12:14Speaker 00

  And, and I think a lot of this is about, you know, the speed or like the sort of like the, the iteration loop. If you say, you know, like how much time you have to react to this, when things happen really quickly or otherwise in such a way that humans can't like easily understand what's happening around them. Our ability to intervene and course correct or shape the trajectory of the development becomes smaller. And I think that's rather than wanting to say that, like, I think what we're like looking at at the moment is like qualitatively different. I think what we're looking at at the moment is different in, on the continuum, on this dimension, where we are more confused about what we're building.

• 1:12:55Speaker 00

  And this is like happening, this like, the development is happening much faster than whatever evolution has produced so far. Such that it's just a much more drastic case of, of, of something that has been going on to some extent. From economic improvements, right? Like industrial revolution has also like sped up a lot of things. But like, and already we were struggling with adapting to what, what should the institutional framework be for like making sure people are so well in this, in under these new conditions. But I, I think what we're facing is just like this, but much faster. And, and then potentially also just with this like autonomous dimension. You know, I'm not sure how much civilization capacity you have to, to deal with this particularly well.

• 1:13:41Nathan Labenz?

  Yeah, I mean, it's an intimidating challenge and I commend you for taking it on in such a direct way. And, and in such a, you know, unique way from, from all the things I've seen. I, I really think your approach of trying to.

• 1:13:58Nathan Labenz?

  Start with these questions and then go figure out, even if it's not perfect or not even close to perfect, like who might have the best chance at getting traction on these questions? I think that is a really inspired approach, you know, and maybe in the last few minutes, we can kind of talk about like what sort of people you are looking for. You know, you're actively like running this program and bringing people in, but also you have kind of a reading list, which is a place that people can go to like, get up to speed on what you deem to be the fundamentals. And obviously, this is something that people are going to be increasingly, I think, open to, and you know, as you mentioned earlier, like if you're a PhD student or whatever,

• 1:14:42Nathan Labenz?

  and you need a mentor who can be open minded enough to encourage you to go in this direction. I think that is probably going to become increasingly common. So I'd love to just, you know, get a little bit more on kind of the very practical, like what sorts of people do you think should get into this? What do they need to do to start to get, you know, to build their foundation, you know, from which they can really start to hopefully make progress? What kinds of people are you specifically looking for, you know, in the program that you're running?

• 1:15:11Speaker 00

  Yeah, just very briefly, a quick outline on like a few different programs we run, where like the answers are like very slightly different. So like, we have been running for the last two years, the summer fellowship, which is just a three months, you know, summer fellowship, research fellowship program where we pair. And this is like mostly sort of PhD level and sometimes a bit more senior than that with researchers in AI alignment, where the idea is to just match people with expertise in one of these source domains, one of these domains, and that study complex intelligent behavior, natural systems with AI alignment researchers. This is three months, so like not always enough time to like get to really groundbreaking insights, but definitely get to build traction and that mutual surface area between the fields.

• 1:15:55Speaker 00

  And then much more recently, we started a new program where we're taking on research affiliates. They tend to be more research experienced, where the aim is really to support them in a more tailored and longer term fashion to just like much more sort of like get them to like pursue their individual personal research agendas. And then we have some sort of more, you know, low key, lower key programs like with the World of Speaker events from time to time and the reading group that was curated by TJ on. Like, I think a pretty good frame of just like thinking through, why is AI element hard and not easy? If it is, but like, why don't we not get this right by default?

• 1:16:39Speaker 00

  What are the obstacles we're meeting? Why is this hard? Um, so you had to potentially also to some extent characterize like, this is the sort of conceptual obstacles to be overcome. If we are meant to to get this right. So this is meant to sort of provide some surface area in a way that's hopefully pretty um, digestible to people from fairly different backgrounds. To just build surface error with like, what is, what are the risk phenomena we are grappling with in AI safety? To then hopefully provide the sort of surface area such that someone with a specific domain expertise can notice. Oh, like maybe this approach we are using in my domain, maybe this like, these sort of mathematical tools, these like conceptual tools.

• 1:17:23Speaker 00

  Are maybe a productive angle and untapping one of these, this risk phenomena. So yeah, going a bit more into, okay, cool, like, what is this sort of like, how maybe we're particularly excited about? And I think it's like, I guess maybe the most like three things, like, um, something that seems important is surface error with AI risk, AI safety. What are, what are the, the sort of obstacles to, to getting this right? What are the sort of intricacies to where the, where risks come from or like, uh, how, how maybe safety should? We can achieve safety. Um, so just like some, some sort of a surface error with, uh, the AI safety discourse seems very useful.

• 1:18:04Speaker 00

  Among others, and this is maybe the second point to say here is like, this is all pretty, pretty early stages, right? There is no good one paradigm telling you this is how to think about it correctly. A lot of open questions, a lot of confusions, and especially also in this sort of like, knowledge transfer question. I don't have all the, like, all the answers for, like, if you have a background in the true theory, this is what you should work on. Like, I'm like, I don't have some ideas, some are more speculative than others, but a lot is really needing to grapple with. Yeah, but could it be? What, what seriously could it be? And I think for that, you wanna well understand your source domain really well, have like expertise. But also understand what would it, what would a useful application actually look like?

• 1:18:50Speaker 00

  And, and yeah, I think in my experience is just involves both just knowledge, um, skill. Um, but also some sort of maybe research experience or just like willingness to, to dive into study open ended questions. Um, I think I'm also at the margin, particularly excited about people who are willing to, even if they don't come from like an L L engineering and to try to really tie in their domain experience with like, cool, like, how could this inform concrete applied, applied work, either conceptually, in terms of experimental design, in terms of evaluation or input, input ability. I think this edge of like really tying theory up with, with applied is, is a really difficult one.

• 1:19:34Speaker 00

  Um, and what I'm particularly excited in trying to foster. But yeah, that is not to, that is neither to undermine the like value of theory, nor like the importance to like actually do things that have like some concrete application or like feedback loops in, in a concrete application.

• 1:19:51Nathan Labenz?

  It's going to be a challenge. You know, it sounds like there's, um, there's certainly a lot more questions than answers, but I guess my pitch to, um, the audience is that the study of A I is also super dynamic. Um, still full of lots of low hanging fruit, it seems. You know, it's like we're, we're both pre paradigmatic and still picking the low hanging fruit at the same time. In terms of impact, you know, the, the move from a more mature field where you may be refining techniques, you know, to a field of A I where, you know, it is just wide, wide open.

• 1:20:34Nathan Labenz?

  I suspect for many people could feel like quite liberating and, you know, an opportunity to really ask big and important questions. You know, some fields are for, less affording, uh, of certainly as they mature. So I do think this is something. You know, the more we've talked about just how early it all is and how sort of how much remains to be done to to figure out how to map these different approaches to trying to figure things out onto the AI challenge, the more I'm just like, man, we need more people to look at themselves and say like, maybe I should be one of those people. So

• 1:21:19Nathan Labenz?

  my kind of call to action again is don't count yourself out just yet. You know, there's nobody, there's no like anointed group right now that is got it under control or that is going to, you know, that we can say with any confidence is going to, you know, answer the big questions. So pile in and, you know, sign up for perhaps a bit more of a unpredictable and wild ride than you were perhaps first thinking, you know, when you got into a PhD. But know that the the sense of adventure, the impact that you can make, the scope of the the questions that are on the table to be tackled is just incredible. And, you

• 1:22:04Nathan Labenz?

  know, for the right personality, that is super invigorating, you know, and can be a lot of can be a lot of fun as well as, you know, potentially extremely valuable to society because this this is coming at us pretty quick, it seems. And, you know, again, I just think we we need every every bit of brainpower that we can bring to bear on it. We will definitely need that. Any closing thoughts on your end?

• 1:22:31Speaker 00

  Yeah, there aren't established answers to any of this. And in particular, we don't have established answers to the solutions. And also, there's like confusion and room to really clarify, how should we best think about the risk? And I think I mean this in a like, in a productive sense, like where we need to wanna like, you know, maybe all of this is like, not worth worrying about, nor, you know, we're gonna die in this very specific way. Like, I think productive and plural perspective on like, what could go wrong? That that seems important and and not over indexing on one specific way. So I think like, Both room for pluralism on what different solutions could we find and also what is the right way of really

• 1:23:14Speaker 00

  understanding what is up here, what is on sick, where do the risks come from? Productive rational disagreement should be had on each side.

• 1:23:22Nathan Labenz?

  Love it. Well, more questions and answers, but again, I invite everybody to join Nora Aman of PIBS and the Alignment of Complex Systems Research Group in jumping in and trying to tackle them. For now, Nora, thank you for being part of the Cognitive Revolution.

• 1:23:40Speaker 00

  Thank you so much.

• 1:23:42Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.