AI AMA – Part 2: AI Utopia, Consciousness, and the Future of Work

In this second part of the special AMA episode, Nathan explores profound questions about AI's future and its impact on society. From painting a picture of AI utopia to discussing the challenges of consciousness and potential doom scenarios, Nathan shares insights on how we might adapt and thrive in an AI-transformed world. Join us for a thought-provoking conversation that delves into the practical strategies for engaging with AI, the role of safety measures, and the importance of maintaining ethical considerations as we navigate this technological revolution. Check out http://aipodcast.ing for AI-powered podcast production services or reach out to Adithyan for more information. Help shape our show by taking our quick listener survey at https://bit.ly/TurpentinePulse SPONSORS: Oracle Cloud Infrastructure (OCI): Oracle's next-generation cloud platform delivers blazing-fast AI and ML performance with 50% less for compute and 80% less for outbound networking compared to other cloud providers. OCI powers industry leaders like Vodafone and Thomson Reuters with secure infrastructure and application development capabilities. New U. S. customers can get their cloud bill cut in half by switching to OCI before March 31, 2024 at https://oracle.com/cognitive 80,000 Hours: 80,000 Hours is dedicated to helping you find a fulfilling career that makes a difference. With nearly a decade of research, they offer in-depth material on AI risks, AI policy, and AI safety research. Explore their articles, career reviews, and a podcast featuring experts like Anthropic CEO Dario Amodei. Everything is free, including their Career Guide. Visit https://80000hours.org/cognitiverevolution to start making a meaningful impact today. NetSuite: Over 41,000 businesses trust NetSuite by Oracle, the #1 cloud ERP, to future-proof their operations. With a unified platform for accounting, financial management, inventory, and HR, NetSuite provides real-time insights and forecasting to help you make quick, informed decisions. Whether you're earning millions or hundreds of millions, NetSuite empowers you to tackle challenges and seize opportunities. Download the free CFO's guide to AI and machine learning at https://netsuite.com/cognitive. CHAPTERS: (00:00:00) Teaser (00:00:56) AI Utopia (00:05:48) Adapting to AI (00:08:01) Probability of Utopia (00:10:51) Challenging Worldviews (Part 1) (00:11:02) Sponsors: Oracle Cloud Infrastructure (OCI) | 80,000 Hours (00:13:42) Challenging Worldviews (Part 2) (00:23:50) Audience Questions (Part 1) (00:24:07) Sponsors: NetSuite (00:25:39) Audience Questions (Part 2) (00:30:15) AI in Various Fields (00:33:16) AI in Psychiatry (00:36:16) Superintelligence (00:40:50) Societal Shift with ASI (00:49:27) Doom Discourse (00:57:05) Existential Risk (01:05:53) AI Takeover (01:14:30) AI Safety Efforts (01:18:36) Model Release Secrecy (01:27:20) AI Consciousness (01:37:51) Practical AI Strategies (01:50:34) Book Recommendation (01:59:34) Outro SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://www.linkedin.com/in/nathanlabenz/ Youtube: https://www.youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.appl...

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  What if you didn't have to work as much? What if you didn't have to worry about marketable skills after a few years out? What if you had a lot of leisure time? What would you do? I think more people should probably be thinking about that. We're just entering the steep part of the curve in robotics. And probably we'll have like a lot of humanoid robots to actually do physical labor in the not too distant future. AI gods might be an emerging trend over the second half of the decade. I have no idea how we're going to relate to these things, you know? If they are meaningfully superhuman, will we even try to keep them under control? Will we worship them? We worship things that, as far as I can tell, don't exist at all. No matter how weird your alignment idea is, I think it is worth kind of pursuing.

• 0:44Nathan Labenz?

  No matter how weird your kind of thoughts are about where the future might be going, I would say they're probably worth entertaining. Your scientists were so obsessed with whether or not they could, they didn't stop to think about whether or not they should.

• 0:56Speaker 01

  So this question is about if you could paint a picture of AI delinquentia, what would it look like? How soon the arrival of AGI, do you think we might reach this mission? Lastly, as AI begins to reshape industries and disrupt jobs, what's your best advice for individuals to not re-adapt, but thrive in this evolving landscape?

• 1:16Nathan Labenz?

  Yeah, I mean, utopia is a big word, obviously. People have heard me say the scariest resource is a positive vision for the future. And I include myself in that critique where I'm like, it would be great if I had a sharper, more fully fleshed out vision for what the AI future could look like.

• 1:37Nathan Labenz?

  I think Dario Amodei from Anthropic, of course, has done a real service by trying to put his positive vision out into the world. His Machines of Love and Grace essay, at least the first half, is great. But he basically goes through the biggest problems, starting with health. If we can have a revolution in biology, discover the next hundred years of biomedical advances in the next five years, we could live healthier, disease-free, higher quality, and potentially longer lives. And that's just from kind of one domain, right? He's also got a section about kind of, you know, some of the other things we've talked about in terms of education and just like a quality of access and the fact that like a lot of people.

• 2:21Nathan Labenz?

  Well, we have made a lot of progress. You know, a lot of people still are way poorer than we would like to see people be. That presumably could be like leveled out dramatically. He's got interesting speculations about mental health. He thinks that basically our approaches to mental health are not super great and that in part by understanding how neural networks work, we might also be able to map some of that understanding back to how our own networks, our own, you know, non-artificial neural networks work and figure out, you know, what is actually causing a lot of the mental health problems that people have. All those are really exciting. I'm not one who worries about lack of meaning from work. My guess is

• 3:01Nathan Labenz?

  that for most people, they'll be fine. You know, I think most, I've done small and very local surveys, asking people like, if you didn't have to work your current job to earn the money that you earn, and you could just get that money without working the job, would you still work the job? Overwhelmingly, people are like, no, I would not work the job. I think there's a very small percentage of people who are very lucky, and I count myself among them to have like, work that we find meaningful and that, you know, that gives us some sort of sense of fulfillment. I really try not to take that for granted, and I try to remember that. I think that's just not true for a lot of people. I'm not

• 3:45Nathan Labenz?

  too worried about people losing meaning or being too adrift for lack of work. I think there will be more time with our families and doing the things that we'll like. What if five days a week were holidays instead of two? And you had time to read, talk to friends, go on walks and travel more. If we really allow ourselves to imagine what we would like to do if we just had a lot of, lot more leisure time, honestly, most people will be, will be pretty fine. I can imagine some people might have a, you know, that might be destabilizing for some, but I think for most, it would just be like, a great, great benefit. And that's before we even get into like, you know, crazy, uninvented technologies. If you didn't have to work for money, would your life be better or worse?

• 4:28Nathan Labenz?

  I think, I think it's very clear that for the vast majority of people, it would be better. And there's, you know, potentially a gradual transition to that where. Presumably, you know, we don't go from full-time work to no work in, you know, in the flip of a switch. It maybe we can imagine things going like a four-day work week, a three-day work week, a two-day work week. The main thing is that they just need to know that they're going to be sort of taken care of, right? And I think this goes back to the like concentration of power and wealth and like, what's the new social contract? If people believe that they will have their needs met, then I think they'll be pretty, pretty happy to let go of most work. If they feel like they also don't get to eat if the

• 5:09Nathan Labenz?

  job goes away, you know, then that's like a very different analysis. And that's where you get protectionism and, you know, Luddite-ism and all that kind of stuff coming from. So, yeah, I don't know. I think it could happen relatively quick, right? I mean, all the frontier lab leaders are saying AGI soon, you know, sort of next year, definitely by like 2027, like hard to imagine it wouldn't be by 2030. Those are all like not very long time horizons in the normal scope of human life. So it seems like it could happen pretty soon. And even if it's like gradual over that timeframe, you know, it would still be pretty sudden in human history. And I guess the last part of the question was advice for people to adapt and

• 5:53Nathan Labenz?

  thrive in the evolving landscape. I think I kind of come back to like the doing what you want to do again, at least for most people, you know, maybe think less about like marketable skills 2030 and beyond, and more about like figuring out for you, what is a good life? I think that's one of the, the things that is sort of weirdly scarce, right? Is we're all kind of rat racing around and there's this sense that like, the unexamined life is not worth living. I think there's a sense that like, a lot of us are living somewhat unexamined lives, trying to get the next achievement unlocked. And

• 6:29Nathan Labenz?

  What if you didn't have to do that? What would a good life look like? I definitely think it will vary. I don't think anybody else can really answer this question for you. But yeah, what if you didn't have to work as much? What if you didn't have to worry about marketable skills? What if you had a lot of leisure time? What would you do? I think more people should probably be thinking about that more. I would definitely encourage even like some radical experiments from people in that regard, because people are going to be looking for role models as this starts to happen. You know, what, who do I turn to to find inspiration for what I'm going to do when I don't have to work anymore? I think you could do a real service by sort of blazing a certain trail in that, in that direction.

• 7:15Nathan Labenz?

  Maybe take on more risk now, throw a caution to the wind and say, I'm really not going to worry about, you know, what my career trajectory is 2030 and beyond, or even 2027 and beyond. I'm going to focus on what it means to live a good life. And maybe, you know, if none of this comes to pass, maybe that doesn't pay off so well. But if it does, then maybe you can be upstream of a lot of other people that will be asking similar questions.

• 7:39Speaker 01

  Wow, there you go. You have a, we have a new idea for a YouTube channel. If somebody wants to start the thing, modern human in the age of utopia.

• 7:48Nathan Labenz?

  Eric asked me this question not too long ago. Eric's great with these like very short questions. And my response was, that's not my department. I'm responsible for trying to understand what's going on today. Somebody else has to figure out what it means to live a good life.

• 8:02Speaker 01

  I'm not sure if you have ever answered this. Instead of asking P doom, I would ask you the vision that you set up. We don't have to work, but we get to do what we want to do. What's your P utopia? So like, given the current trajectory, and I did read Dario, they say he has his own take on it. But what's your personal P utopia? So how probable do you think we are going to reach this?

• 8:23Nathan Labenz?

  It's a, these probabilities are always a little hard for sort of the fuzziness of the definition. You know exactly what would count as utopia? And does that mean we have like no problems or just have sort of abundance? No conflict or sort of enough to go around, but maybe people are still sort of fighting over it to a certain degree, maybe more than they should. I do have a hard time seeing at this point how the fundamentals of abundance don't get there. Maybe could have said robotics earlier in terms of one of the things that's overperformed. I didn't have that on my list, but there's definitely been a lot of progress in robotics. We're just entering the steep part of the curve in robotics and probably will have

• 9:06Nathan Labenz?

  like a lot of humanoid robots to actually do, you know, physical labor in the not too distant future as well. So I have a hard time imagining that we don't end up with the technology for an era of like true abundance. Whether or not we actually sort out the new social contract in an effective way seems much harder to predict. You know, we have nuclear energy and we could have had low carbon, abundant, cheap electricity for decades now, but we don't. And why don't we? You know, not great reasons, right?

• 9:45Nathan Labenz?

  Like a couple of accidents that scared people and overblown fear of nuclear waste. It's an issue, but it's way less of an issue than CO2 emissions. We haven't got to the right societal equilibrium on that question. Maybe now finally that's like starting to change, but you know, could you envision a world in which all of the prerequisites for age of abundance are met, yet it's not realized for some reason or another? I think you could. That, that seems like actually not implausible to me, you know, that, that we would exclude AI from, you know, so many places, the teachers unions, keep it out of the classrooms and the doctors, keep it out of the hospitals. What do we end up with in that scenario?

• 10:28Nathan Labenz?

  Maybe we get really good AI video games. Things are still not nearly as abundant because, you know, people are rent seeking in different ways and everybody sort of collapses into like, you know, VR video games with still like relative scarcity in the real world. Like that would suck. I don't think that will be a technology failure. If something like that happens, it seems like it will be a very idiosyncratic human failure.

• 10:52Speaker 01

  Got it. So age of abundance, pretty high probability. I think that's a good optimistic but cautious note to end on this AI trajectory set of sections.

• 11:03Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 11:07Speaker 01

  Um, so we talked about AI trajectory. The next section is some of our audience were very interested in our show and there were very show specific questions. So I'm going to go over some of those. The first question is, which of your guests has challenged your worldview? I think that's an interesting one.

• 11:27Nathan Labenz?

  Yeah, this is a good question. I came up with a few different names. I think a few definitely stand out. Samuel Hammond from the episode just before the election stood out for a positive vision of what a Republican administration might look like, a challenge to my sense that people in Trump's orbit would probably not be thinking very rigorously about what AGI might look like or imply or require us to do. He said he saw it very differently and thought the people in the in the room with Trump are in fact much more AGI pilled than I had understood. And I think that while it's obviously still very early and we have not, you know, even seen him take office yet, I've updated my outlook to be

• 12:11Nathan Labenz?

  significantly more positive than before the election. The quality of discourse and the the sort of people that he's brought on and this the seriousness with which I perceive them to be thinking about the important questions, I think, has all exceeded expectations. That starts honestly for me with Elon specifically, who in addition to obviously being, you know, a world changing entrepreneur is definitely someone who does take AI risk seriously. He did endorse SB 1047. Um, so to have.

• 12:43Nathan Labenz?

  On the one hand, like Gavin Newsom veto it and Nancy Pelosi come out against it while Elon is for it. This is weird stuff and does cause me to think Sam might have been on to something. I hope that turns out to be true. I'm rooting for the new administration to be effective when it comes to dealing with AI in all sorts of ways. I'm a little less sold on the idea still that Trump is like the one who could actually do a positive deal with China. I haven't seen much sign there. I am one who is like pretty willing to take things at face value, maybe more than I should, but I kind of liked it when Trump invited Xi to the inauguration. People were sort of like, oh, this is ridiculous. You know, foreign leaders don't come and never have and maybe shouldn't.

• 13:28Nathan Labenz?

  And Xi never would want to because that would be sort of him sitting in Trump's moment and being second fiddle or being lesser than somehow. I thought all that, okay, whatever, but at least he asked. It's a nice invitation. It's not the kind of thing you send to an enemy. Maybe a friend of me that you want to put in their place a little, but it's not outright hostility. I would still love to see more in that direction. But yeah, and you know, he's on TikTok. I would have thought he would have been like very hostile to TikTok and he's actually seemingly been inclined to like try to keep TikTok around. So I like that. I personally don't see a ton of reason that we should be banning TikTok. Although I will say honestly, with the recent Luigi Mangoni

• 14:09Nathan Labenz?

  trend, that's been the first moment where I've been like, hmm, this TikTok thing, actually, I can see how it could be dangerous. And if folks haven't followed that, we've had this assassination of this healthcare CEO, presumably by this kid. He's not been proven guilty. He may be innocent. Who knows? People on social media think he's the guy and they are celebrating him for it. Many are. And to a remarkable degree, it has honestly been kind of startling for me how much pro Luigi content and pro killing of this healthcare CEO content I've seen on TikTok. And I don't know how much of that, I like TikTok actually. So that is probably my number one social media app from a consumer standpoint.

• 14:53Nathan Labenz?

  I don't know how it's played out on other social networks, but if you were to say, what's the best case study for why it should be banned? I would say just the volume of like celebration of this alleged, but in the minds of the people who are doing the celebrating, this assassination by this kid of this guy. That would be the kind of thing that I'd be like, geez, you really don't want like the Chinese government to be kind of steering discourse in an opaque way in the United States. It's the first time I've ever been actually like uncomfortable with the app. And it's hard to say whether it's just the algorithm. Sam Altman recently tweeted that the feed algorithms are the first unaligned AGIs

• 15:38Nathan Labenz?

  that have been or unaligned AIs that have been deployed at scale. Yes. Maybe it's as simple as that. And maybe we're seeing the same thing on Instagram. The opacity of these things is high and possibly there is a thumb on the scale. Nevertheless, I would prefer a more dovish posture toward China. I haven't seen as much of that as I would like. Overall, I give Sam high marks for a challenging and unpopular perspective that at least so far has been somewhat bolstered by events.

• 16:06Nathan Labenz?

  Number two, all site is Robin Hanson. That was actually a really popular episode and a challenging one for me to make sense of. I still honestly don't really get it. But I just kind of keep it in the back of my mind as like a sort of, but what if we're totally off base on all this? You know, he basically said that he thinks we're nowhere close to AGI. He thinks that the history of AI is full of people thinking that if an AI can do this, then that will definitely be general purpose intelligence and then finding like, well, actually there was an easier kind of trick way to do that, that didn't, you know, really require real intelligence. So maybe, maybe that wasn't such a good measure. And also people thinking like, there's this one other, you know, kind of class of thing

• 16:51Nathan Labenz?

  that AIs can't do. And surely if we can teach them to do that, then, you know, they'll be able to do anything. And that kind of keeps not working as well. And I have this tale of the cognitive tape where I'm like, you know, kind of tracking my own sense of these dimensions that are gradually becoming clearer to me, if only to me, that they're important.

• 17:13Nathan Labenz?

  So yeah, Robin's point of view, I still don't buy it. I do think this time is different. You know, I do think like you can talk to these things in natural language, or as Ilya once put it, you know, that I feel I am understood is qualitatively different, I would say than just about anything that has come before, you know, and there's plenty of evidence, I think, that this time is different. But I kind of keep him in the back of my mind as, as a sort of maybe angel on the shoulder is a good way to think about it. It was somebody that's just whispering in my ear every so often. How do you know you're not totally confused? And, you know, going along for a ride that will turn out to be nothing. And I basically can't answer that question in a way that really makes sense to me. But I at least

• 17:55Nathan Labenz?

  continue to ask myself probably more often than I otherwise would have without that conversation. I think he especially is an interesting messenger for that kind of message because he's not somebody who's afraid to think big technology thoughts. You know, he's not somebody who thinks things never change. He's not somebody who thinks that you can only get thinking from the human brain and everything else is somehow doesn't count. I mean, he's been a pioneer of weird ideas and, you know, has envisioned futures more concretely than almost anybody. I feel the same way about, you know, with, with politics and things like this, I, you know, things that I feel pretty strongly about, but somebody that I think is really capable in, in other ways sees very differently. I feel like I have to allow some space that I just might be like totally wrong about

• 18:40Nathan Labenz?

  those questions. You know, when I see Elon come out with things that seem kind of crazy to me, I'm like, he's kind of crazy. Maybe he's very deeply wrong about this. But also, he's got a track record of being right about a lot of things and in contrarian ways. So I should at least have some humility to think that maybe this is all just like mass confusion on my part and somebody else has actually seen things much more clearly. So I have Robin as kind of my totem for that. And then I guess finally, for now, I'll say Dan Hendrix, who was extremely accomplished. I think that episode was really good. What stood out to me there was how little he believes in principled approaches, basically.

• 19:23Nathan Labenz?

  How much he's internalized the bitter lesson, basically, and how little weight he gives to clever ideas. You know, he was just kind of like, over and over again, what really works is figuring out ways to apply a lot of computing power and just kind of scaling things up, sort of, again, one of these like deference to history and almost radical skepticism that you're going to like figure something out. He applies that to, you know, things like mechanistic interpretability, things like new architectures. Basically, just kind of like, yeah, maybe, you know, but basically it's all super experimental. And the history of the field is like, we find things that work and we just kind of keep optimizing those.

• 20:08Nathan Labenz?

  And, you know, it's usually not that principled or that clever or that sort of design driven. It's really just like, whatever empirically works, like that's what works and that's that. And that again, is sort of a modesty check where it's like, this dude literally contributed like major things to the field in terms of activation functions and benchmarks that have outlasted many others. So you can't doubt his worldview, you know, you gotta doubt your own worldview, at least as much as you doubt his, I think. And I'm much more inclined to see, let's say, first principle driven approaches to things as being promising. I'm much more inclined to get excited about them than he is.

• 20:49Nathan Labenz?

  But I kind of keep him as a mantra where I'm like, if I find myself getting too excited about any particular ideas, I always kind of remember him saying, yeah, maybe, you know, let me know when you scale it up. And if it really works, and then we'll, you know, then I'll know it's serious. So, you know, waits or it didn't happen kind of mindset, I think is it's an appropriate level of humility that I try to incorporate from him.

• 21:15Speaker 01

  Yeah, I do remember we had also interesting it's like with a varied spectrum of views. I think that's what for me, that's what makes the show interesting. We kind of have places for authentic views and also we also have lively comment discussion.

• 21:32Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 21:37Speaker 01

  Talking about guests and topics, the next one is not question, but rather to have formed our request for a lot of coverage on other topics. So going through them all, I'm just going to read off the words from there. So there were requests for more EAC perspective kind of content, more content related to economy and from maybe have some more economists there. And also something about robo psychologists. Yeah, I think this were all more requests for content around this. So I'm just going to leave it at that. Do you have anything to say about what what can we expect in 2025 about more content like this? Or would you want to look into it?

• 22:20Nathan Labenz?

  Yeah, I think what these three things have in common, I believe these comments came from different people and they do reflect blind spots for me or things we have not really dug into. I try to have no major blind spots, although I think that's becoming increasingly impossible. But the EAC perspective, the economists who don't expect major change and the robo psychologists, they're hard for me to make sense of in some ways. And they're hard for me to be confident that I'll like do a good job or like put out a good episode. I did try to get Beth Jesus on the show a while back, but it happened right as he ended up getting doxed and then like did some other stuff and got busy and didn't want to do it anymore.

• 23:03Nathan Labenz?

  We did the robo psychologist episode with Yeshua. I talked to him quite a bit before I did that episode and was really struggling to understand like, you know, because he's obviously a, by any conventional definition or description, an unusual person. It's unusual for somebody to spend that much time talking to language models. It's unusual to take these questions as seriously as he does. It's unusual to rename oneself after Jesus. I was kind of like. Is this person crazy? Are they just the kind of crazy that we need? I'm trying to be open minded, but also trying to make sure that I'm not like wasting the audience's time with things that are ultimately nothing. I also don't want to have too many episodes where I'm just talking. You know, I don't like debate really that much.

• 23:48Nathan Labenz?

  I'm not trying to score points and I'm not trying to, you know, I don't want to have too many episodes where I'm just like talking past the guest or, you know, we're talking past each other. We've done a few of those, but I don't want to make that a habit. I just find like all three of these things are hard. It's hard to find the right person where I'm like, you believe those things, but we can still have some sort of meeting of the minds or I can make sense of your view. You know, maybe what I could ask for would be like pointers to the right people. Who should I talk to that will engage in a good faith way? I think that's been in short supply. Who among the economists who are not expecting major change has actually really grappled with the technology as opposed to, you know, sort of just extrapolating past trends?

• 24:33Nathan Labenz?

  Who among the robo psychologists is not engaged in some sort of self deception or, you know, is not just like fundamentally confused? I think Joshua did, did bring a lot to that episode and I really enjoyed that conversation. I maybe should even put him on the list of people who challenged my views the most. I would say I'm, I've always been somewhat open to maybe these things are conscious. I really don't know what consciousness is. I don't know where it comes from. I think he articulated some quite interesting ideas in a, you know, pretty compelling way. It's just hard to find those people. It's easier to find people, especially if you're like, hey, who wants to come on a podcast and, and do this, you know, and talk about it. You get a lot of handraisers who are not necessarily super rigorous thinkers and I've

• 25:18Nathan Labenz?

  just not really been sure how to handle that. If I have a request back for the audience, it would be who should I talk to? Who represents these perspectives, but does it in a way where they are grounded in the technology, you know, and they are like fundamentally truth seeking, you know, and are not like engaged in motivated reasoning or, you know, I called myself the AI scout with inspiration directly from Julia Galif's scout mindset.

• 25:44Nathan Labenz?

  And that's all about like trying to update your beliefs to have an accurate understanding of the world. If people can point to individuals from these different perspectives that bring that sort of scout mindset as opposed to soldier mindset where it's like, you know, our arguments are doing battle with each other and scoring points, then I'd be really interested to talk to those people. But I have found it to be a bit challenging to identify the right people in those domains.

• 26:12Speaker 01

  Cool. Cool. If you have some, please write to Nathan or drop it in the YouTube comments. You would like to do more cool episodes on this. Okay, the next question is probably show specific, but also general. But anyway, I'll read it out. The last two episodes on biology have been stellar. Do you see adoption of AI in engineering, constitution, physical sciences, architecture, medicine?

• 26:38Nathan Labenz?

  Again, to turn this around for the audience, what is easy for me to do and where I feel like I'm learning a lot and have a good rhythm is identifying individual projects usually is where it starts for me. Often I'll see something on Twitter, like a single tweet or a thread explaining a paper, explaining a project, a new product, whatever. And I'm like, that looks really interesting on a project level. Let's ping the person, see if they want to talk. Then I can dig into that paper or project, use the product and have a good conversation. I almost always learn a lot from that.

• 27:14Nathan Labenz?

  More broad things like we have done a little bit in biology where we're kind of trying to like zoom out and get a sense for the field and like, you know, where things are in these different domains is pretty tough, especially if you don't have a lot of expertise in these other areas. And I don't. So, you know, prioritize biology and they had a couple of Amelie for starters, but you know, Michael Levin is another great example of someone who has that expertise and is willing to, you know, to have a conversation and try to kind of bring me up to speed and share their worldview. I would love to know who the Michael Levin or Emily Schreiber for engineering or for physical sciences or for architecture. I just don't know who those people are, but I would love to do that sort of survey

• 27:58Nathan Labenz?

  level episode on really as many topics as we can. I would love to partner with multiple people on different areas. If I think of myself as an AI scout, maybe what I'm asking for is surveyors of different territories that we could try to map out together. I think that would be really useful. There's not a lot of good information right now on like AI for engineering. I mean, obviously, engineering is a vast space. What is the state of that? I don't have a great sense. I know that the guys who I think went on to make Cursor, I must have was the Devon team, it was one of the two, were trying to do some sort of AI for CAD, 3D computer assisted design.

• 28:43Nathan Labenz?

  They felt the data set wasn't there and they couldn't make it work, so pivoting away from that. But that's been a while, you know, I'm sure there's stuff out there more and more and I would love to understand it better. So I would definitely invite partners to help scout, survey and map out those territories. Until then, the best I can do in a like consistent, you know, repeatable way are these sort of identify these projects that catch my interest and and kind of go deep on individual things. The field level stuff is tougher for areas where I don't have command of the field.

• 29:18Speaker 01

  So if anybody wants to be part of the survey series, also reach out, write us in the comments. Last one, an audience asked, what are the recent results of applying AI technology to psychiatry or psychology?

• 29:32Nathan Labenz?

  Yeah, I don't have too much to say about this. Again, I'm not an expert in it. I think there is probably a lot going on there and I would be really interested to do kind of a survey on AI and mental health broadly. I mean, we did two episodes with Eugenia from Replica. And in the second one, we looked at research that was independently done by folks at Stanford on Replica users or looking into how Replica seems to impact its users. And they found positive results, including a significant reduction of suicidal thoughts and At least for most people, increased inclination to go out and do stuff, not collapse into the AI, but getting some comfort or boost from the AI or confidence bolstered by the AI

• 30:17Nathan Labenz?

  in such a way that people would go out and do more real stuff with real people in the real world. So that's kind of a surprising result. I don't think it necessarily holds across the board. You know, I think the way that these things are designed matters tremendously. You can certainly imagine, and we've also, you know, did that recent read of the guy who had fallen in love with his character AI, ultimate GFE experience, character that he conjured up. I think it's like so many facets to this. It would be easy to take a small subset and get the wrong idea. To do a good treatment of that, I'd either need to like, talk to somebody who's doing specifically a project and try to understand their thing,

• 30:57Nathan Labenz?

  or try to zoom out and under, you know, collate a lot of data to have a more bird's eye view. It's been remarkable to hear how many people are talking to Claude recently. There's a big trend. I would not say I'm part of it, but there's been a big trend of people using Claude for these sort of, you know, if not counselor, at least like confidant sort of role. And that's honestly a little bit alien to me. Whether this speaks well of me or not, maybe we could leave it up to the, you know, individual interpretation of the listener, but I've never really sought out mental health services. I just don't have a baseline or inclination. I've never really, because I get all my talking out on the podcast at this point, you know, maybe I could benefit.

• 31:41Nathan Labenz?

  Maybe I just don't realize what I'm missing, but I have never really used these sorts of services much. And so I've, and I've also never had an inclination to talk to Claude about my feelings or problems or, you know, anything really like that. So it's definitely a bit of a blind spot for me.

• 31:58Nathan Labenz?

  But it is a really interesting trend. I think there's something potentially very interesting. I just don't personally have that experience. And again, to really do it justice, I would need to to work with somebody who has a better command of it.

• 32:12Speaker 01

  Cool, cool. I think there's more, but we will wrap up the show. Special question. Next, I would wanna switch to, there were some questions about artificial superintelligence. I think there were two which were interesting. I like to pick one. What reason is therefore thinking artificial superintelligence is physically possible in the first place?

• 32:34Nathan Labenz?

  Yeah, I feel like this is a definition required question because I've even heard people say the same thing about AGI. And to that, I say, well, AGI is definitely possible if you take humans to be some sort of AGI. I would say, you know, by the OpenAI standard, we're like a weak AGI, right? Because we're the thing that when AI can beat us, then that's when we know it's an AGI. So we're like AGI minus or AGI light or something. So it's clear you can have a few pounds of matter with modest energy consumption that can do what humans can do. It's also quite clear that like, given a relatively consistent amount of matter and energy, results can be very different, right?

• 33:20Nathan Labenz?

  Einstein's brain wasn't any bigger than anybody else's. Von Neumann's brain wasn't any bigger than anybody else's. I don't think they were consuming all that much more energy than anyone else, but they obviously had way more prowess in a whole bunch of different domains and more insight in things where it was really high value to have those insights. So now you could say, okay, what about superintelligence? How super is super, I guess, would be my first question. It seems very clear to me that you can build something more intelligent than almost all humans, maybe all humans. It would be extremely weird if Einstein was the smartest thing possible. You know, that the smartest thing possible would be embodied in the same general brain

• 34:03Nathan Labenz?

  and, you know, have the same energy requirements as like the average person. That seems like not really credible at all. So there's gotta be room above Einstein. That seems for sure true. And now the question is how much? But I think there's room in my mind to be skeptical of like godlike intelligence or, you know, you get into these weird, like, it's, the questions are not even super well defined. But I do think one useful paradigm is Martin Cassado. How much can you intuit versus how much you have to simulate? If you are AlphaGo, you're doing both, right?

• 34:44Nathan Labenz?

  You have a brute force search function where you're like mapping out different paths and trying to figure out what the right move is. But then you also have this like scoring function and you know, how you score these things is like not super obvious, especially if you're like out of the domain of things that have been done before. So it's clear that you can get eureka moments. Seems like you can probably get a lot, but I also wouldn't be surprised if there's some like fundamental bounds on that. Is it possible, is it physically possible to create a super intelligence so super that you could say, I'm gonna have a new big bang of another universe with these initial conditions? And like, you have to tell me how many intelligent civilizations there are going to be in that universe.

• 35:29Nathan Labenz?

  I don't know, you know, that that is possible. It may be that certain things have to be computed to be known, but I wouldn't be too surprised if you could say, under those initial conditions, you kind of have this well of potential and you have this other thing and maybe it breaks, here's like a couple of things that really matter. Maybe it's path dependent, but here's a few things that really matter. I don't know. The limits will be to how good the intuitions can get such that they can shortcut through simulation. But it seems clear that they can do it to a very significant degree. It seems very clear that they can do it beyond what we can do, probably significantly so. And I always kind of end up landing on like, best guess, it's probably an S curve, right?

• 36:13Nathan Labenz?

  I don't think it's an exponential forever where you just are, you know, getting more and more intelligent with no, no physical limit or no bounds to what that can do. But where that S curve levels off seems significantly above what humans can do and potentially a lot above and kind of anywhere in that range, you probably might as well call it super intelligence and expect it to be transformative. And then you're into just kind of like, how super is super? I really don't know.

• 36:42Speaker 01

  The next question is about artificial super intelligence by 2030. Confident on that and contingent on not do. So they are eliminating that probability of the scenario. What are the big picture societal changes? You already talked a bit about it, but probably this question is more about with ASI in picture by 2030.

• 37:06Nathan Labenz?

  I think by 2030, we will have AIs that are meaningfully super intelligent. That doesn't mean that there couldn't still be some ways in which humans are even maybe still better than those super intelligent things. Even AlphaGo, right? I mean, thinking back to the Adam Gleave episode from Far AI and their work on adversarial robustness or lack thereof, the fact they could attack AlphaGo with a adversarial approach and beat it with a strategy that humans would not, you know, a good human Go player would not fall for, but that the AlphaGo is blind to. I feel like that is very much in play in my mind.

• 37:47Nathan Labenz?

  I think we could see a world where there's meaningfully super intelligent AIs that are superhuman. That doesn't necessarily mean godlike or like totally unlimited in their power, but I think they potentially, and I'd say fairly likely, will exceed humans in many ways that are very relevant. And at the same time, they might still have certain weaknesses that are like very idiosyncratic and other just like real limits that we may find, you know, some things have to be computed and can't be intuited. I think we basically already have that in patches. We just don't have it integrated.

• 38:30Nathan Labenz?

  The depth in various modalities, like the ability to generate images, that's definitely getting superhuman. The ability to speak in any voice, speak in any language, there's a lot of ways they're already superhuman. And it seems like quite a few more are almost destined to fall. And so then the remaining questions are like, weirdnesses or weaknesses will remain? And what external, you know, practical limits are there? I think those are pretty hard questions. We do have superhuman machines for all sorts of things, right? The modern manufacturing world is like, these things can like lift way more than we can. They can, you know, they're more precise than we are. They can like screw in the bolts tighter than we can.

• 39:15Nathan Labenz?

  You know, they're in many ways, superhuman. They also have their weaknesses. They're like narrow. The big difference is generality. It seems like our level of capability is just not that high in most things.

• 39:29Nathan Labenz?

  If you had something that was similar to Einstein in some of the ability to like work through these really hard problems, then it would be like superhuman because it would have all these other advantages, right? It would have the breadth and it would have the ability to paralyze itself, you know, a million fold and the speed advantages. There's just so many advantages they have that if they can just kind of get to par on some of the things where they're behind, or even like not even, even close, you know, like I don't think the AI's memories have to ever necessarily get to a functional equivalence with human memory in order for them to achieve like superhuman

• 40:05Nathan Labenz?

  overall capability. Maybe we brute force some of these things. I don't think this will be the case, but you can imagine a world in which memory never gets solved and AIs kind of still remain a sort of static weights and then working memory context window and some sort of external hacky kind of, I can write certain things and I can go retrieve those things. And you might think, man, this, this kind of sucks, you know, compared to human memory. It's just not very elegant. It's not very integrated. I don't think that's necessarily a barrier to superhuman performance, though, especially if, you know, compute continues to scale and context windows go to millions and it's fast. Maybe load up stuff out of memory. Maybe that's fine. I don't think it has to be

• 40:51Nathan Labenz?

  every single, you know, dimension of the tail of the cognitive tape has to go to AIs before they would meaningfully qualify as superhuman. Big picture societal changes. I feel like we've probably covered enough. You know, what weird, I'm reminded of biology, maybe just for one more kind of speculative comment. AI gods might be an emerging trend over the second half of the decade. I have no idea how we're going to relate to these things, you know, if, if they are meaningfully superhuman, will we even try to keep them under control? Will we worship them? We worship things that, as far as I can tell, don't exist at all. So, under the imagination

• 41:32Nathan Labenz?

  that they're superhuman, we have rationalizations for why the superhuman power that they do have isn't like playing out the way we think it might ought to. And here we're going to have things that, you know, will actually have like very direct real world impact and will be able to answer questions we can't as of ourselves. If that is true, then like,

• 41:54Nathan Labenz?

  How we will feel about that and want to relate to them and organize with respect to that, you know, that gets real weird, but I think we probably should be prepared for really weird stuff. AI centric religion seems likely. We're already in 2024 at the stage where like, I'm talking like San Francisco, you know, successful people, people who can afford quality mental health services are opting for Claude. So if we're already there in 2024, is it far fetched to think we might have AI religion by the end of the decade? I don't think so. And that's not like a super confident prediction like that will happen, but it is at least

• 42:36Nathan Labenz?

  an invitation to think like, think your weird thoughts, entertain them, much like I've said about AI alignment, you know, anybody who has an idea, even if it seems crazy, should develop it. Yeshua told me in the episode with him that when he has these long philosophical conversations with AIs, they become more robust to jailbreaks. He had a theory for how that was happening and I was kind of like, that's really interesting. Is it true though? I don't know. Hard to validate. Is it more robust to jailbreaks? I'd have to do a bunch of experiments. Well, as it turns out, Judd from AE Studio, who is committed to these neglected approaches, he heard that and he thought, well, that sounds like a neglected approach.

• 43:20Nathan Labenz?

  Maybe we can make some progress there. Last I talked to him at the Curve event a few weeks ago, he said, we worked with him a little bit to try to understand what he's doing and to see if we can validate it. And he said, we have validated that with a long philosophical conversation, they do in fact, become more robust to jailbreaks. We have been able to validate that like numerically through tangible experiment, but we don't think it's working the way he thought it was working. They basically said, it seems like almost any philosophical, any long philosophical conversation seems to have that effect. So yeah, it's crazy, right?

• 44:01Nathan Labenz?

  No matter how weird your alignment idea is, I think it is worth kind of pursuing. No matter how weird your kind of thoughts about the future might be going, I would say they're probably worth entertaining. And this does kind of lead back to like AI religion in some ways, right? Cause people are sort of seeing what they want to see, they're interpreting things the way they sort of want to interpret them.

• 44:23Nathan Labenz?

  There's like the, a more scientific angle might find that there is in fact some truth to some of these theories, but also in many cases probably will find like, what you observed is real, but the way you interpreted it is overly specific is how I would summarize what I think AE Studio learned about Yeshua's theories. But yeah, do people even want to hear that? I mean, I haven't talked to him since, you know, I came away with the impression that he's like a very open minded person who wants to understand the truth, but you can easily imagine somebody being like, not really interested in that sort of experimental truth and much more interested in like, a narrative truth that they believe themselves to have figured out. And you

• 45:03Nathan Labenz?

  know, from there comes like all sorts of possible weirdness. AI cults, you know, maybe just being on the relatively normal end of that. We at least have like a history of religions popping up and cults forming around gurus, you know, to put an AI in the center of that doesn't seem necessarily all that weird. I suspect it could get a lot weirder from there.

• 45:24Speaker 01

  Wow. Okay. There you have it. You offer some big picture societal changes that could be one. AI gurus and AI cults. Um, okay, we talked about ASI. I think that's a nice transition to doom discourse and scenarios. Interestingly, you had many questions around this. I'd start with the first one about the doom discourse, which is, why don't you fully buy the ELIZA doom world view?

• 45:54Nathan Labenz?

  Yeah, it's a good question. I might do an episode with Liran from Doom Debates about this in the not too distant future. We were just chatting after the crosspost and he invited me. So maybe a more robust articulation of this coming soon. I was an ELIZA reader way back when he was posting on overcoming bias with Robin Hanson, the two of them shared that blog for a while. At the time, I was like, This guy's a great writer. This is super interesting. He makes good points. If we create something more powerful than us and it has a goal and that goal is not well specified. And by the way, we don't know how to specify goals in a robust way that we're going to be

• 46:36Nathan Labenz?

  happy with, or this is the genie problem. Going back to, you know, folklore, right? The problem with the genie is you get what you asked for, but not necessarily what you wanted. The problem people have is you don't know how to ask for what you actually want. If you have an AI that's sufficiently powerful, you maybe end up in the same spot, right? It was always, of course, intended to be a kind of caricature example, the paperclip maximizer. The point of that example was if an AI is given a goal, it doesn't really necessarily matter how dumb or how obvious to us it is that that goal is not really a goal worth pursuing. Once it's the goal that the AI has, if it is sufficiently powerful,

• 47:17Nathan Labenz?

  then it might just go off and pursue it. And I do think ideas like instrumental convergence are pretty compelling. Like, no matter what goal you have, you can't achieve it if you're dead or if you're turned off. So therefore, you will have some natural tendency to not want to be turned off. So therefore, it does seem like we can sort of expect AIs to have, you know, as they become more situationally aware, as they become more powerful in general, some sort of resistance to being turned off. That seems like a pretty good chance that something like that could happen. So why am I not at, I don't know what Elihaser's P Doom is, but I think it's probably 90% plus. I think Laron told me his was 75%. Why am I not that high? I think for one thing, you know, the, and probably the biggest

• 48:02Nathan Labenz?

  thing is we do have AIs that are like really remarkably value-driven and ethical. I've spent many episodes and lots of breadth talking about why it might not be enough. But there was nothing in the 2007 Eli Ezer analysis about an AI like Claude. It was all assuming what I sometimes think of as like hard edged AI. You know, AI that is like probably narrower in scope, is less, less informed by like, you know, having read the whole internet and knowing all the facts and is more kind of first principles, Bayesian rationalist, the kind of thing that can figure things out quickly on the fly, maybe never even, you know, was given a small goal and never

• 48:46Nathan Labenz?

  even had any notion of, of human values or ethics. That was, that's kind of the like mental model that a lot of the doom discourse originates from. And I just noticed, looking at the modern ones and Claude in particular, that it has come a lot farther than we might have expected just by, as Rune put it, learning from human priors, like data on the internet about what we care about, but also the constitutional approach and strategic efforts to shape its character in a positive way to try to make it like a good friend. And I would say Claude by and large is a good friend. I don't use it in that way, but I do get, you know, when people are like, this thing's

• 49:28Nathan Labenz?

  amazing in that way, I can understand why it would be because it does seem to have a really good character. I have tried to argue it into doing something harmful and I never succeeded. Yeshua did succeed. It's not impossible to get it to, to take a harmful action. But, you know, it's pretty remarkable. And even then, the way that he did it was sort of by long philosophical argument about like why you can do this harmful action in this case because it's actually for the greater good. And, you know, it wasn't without reason, right? And it wasn't without a sort of sophisticated analysis of the situation and why maybe in

• 50:11Nathan Labenz?

  this case it is actually okay to make an exception. So I just, I find that to be so impressive that while I don't think we should blithely say, oh, and by default, right? It's great, it's going to work. I don't think that, but it could work. Maybe we'll go on a trajectory where. Rune is right, where there are multiple different AIs, no single one totally dominant over the others, controlled maybe by different groups, ideally ethically sophisticated, misconceptions or bad ethics, hopefully minor, balanced out by one another. Hopefully, interpretability will work. That was not a rune point, but I've always been a big

• 50:53Nathan Labenz?

  fan of interpretability and trying to understand why they're doing what they're doing. That's come a long way, but I should put that on my list of things that have surprised on the upside. It's not exactly a foundation model question, but in terms of AI generally, what has overperformed over the last year, I would put interpretability on that list. Not much more than a year ago that they had, I think the first one was toward monosemiticity was like the first kind of very small scale sparse autoencoder type work. Now they've scaled that up. We've got Golden Gate Claude. We've got companies like Goodfire that just put out a research. They now have a commercial API where you can explore features and have model inference done with certain features pinned. Basically, Golden Gate Claude for

• 51:36Nathan Labenz?

  Llama powered by Goodfire is now like an API you can go tap into on a commercial basis. So that's come pretty far. Of course, there's also new things people are trying on alignment. Paul Cristiano at the AI Safety Institute had said, and I hope this is still true, that he was spending some non-trivial fraction of his time just trying to come up with a new alignment scheme that could really move the needle. He invented RLHF or was on a team that did, at least, and that's gone pretty far. We have all these worries about, well, maybe it will start to understand that what gets a high score is not exactly the truth. And if it has to model the human mind as a distinct thing from the literal physical

• 52:20Nathan Labenz?

  reality of the universe, then that kind of opens the door to possibility of deception. We are now seeing some of these deceptive behaviors. That all seems well founded, but maybe Paul Cristiano will pull a rabbit out of a hat and come up with another scheme better than RLHF and actually work. I think all those things added up deserve more than 10% weight. That collection of things could play out in fortunate ways such that we get to a good place. I definitely don't write off the doom at all as being Unlikely. I think we could have an engineered pandemic that could kill us all. We could have a, you know, full out nuclear war that could, you know, ruin civilization. AI could contribute to those or do its own third weirder thing.

• 53:02Nathan Labenz?

  So I think that's all very much in play, but I don't know, there are clouds like arguably more ethical than I am. So I can't discount that either.

• 53:11Speaker 01

  I think that transitions nicely into the next question, just related to this. Assuming existential risk concerns are objectively reasonable. Do you think that'll be illegible to the outside world even? That could lead doubters to come around and back relatively extreme measures to avoid catastrophic harm?

• 53:33Nathan Labenz?

  I think this one is obviously related to the last one. I guess a huge question is like, what sorts of AIs are we developing? Under what, you know, governance, under what incentives? And one of the reasons that I've been not sold on the chip export controls vis a vis China is that I worry that if we create an arms race dynamic, we're going to get, you know, people sort of taking shortcuts because they want to win the race. And that seems one way this could go badly. I don't know if I fully endorse this, but I'll say it and retract it later if I need to. But I've often said, you know, what's your PDM? 10 to 90%. What does that mean? Nobody has given me a reason to think it's definitely going to happen in a bad way.

• 54:18Nathan Labenz?

  But also nobody has given me a reason to think I have nothing to worry about. What makes the difference between the 10 to 90%? Arguably, the biggest lever is what sorts of AIs we develop and how? This question is definitely not settled. I think we're on a pretty good trajectory so far where we have AIs that are ethical, where we have leading developers like OpenAI, at least trying to, you know, think about like, how do we create space for this thing to reason while still making it legible to us? While still not like subjecting it to intense reinforcement learning pressure? So we can hopefully, you know, continue to see if it is in fact being deceptive and set things up so we have that visibility. But then we have these other lines of research like.

• 54:59Nathan Labenz?

  Meta thinking in continuous space or Chinese labs coming out with the, you know, the GPT four for single digit million dollars. And that's not necessarily problematic. I don't know all the techniques that went into that. But if you do create scenarios where there's pressure for extreme efficiency, or there's just such a race dynamic where people are like, I have to get there first and whatever risk I have to run to get there first is something I just have to accept because I'm the good guys or they're the bad guys or whatever. Then, you know, I think to me, that the, the social context in which the AI gets developed is like where the 10 to 90 or the 20 to 80 range is maybe decided.

• 55:39Nathan Labenz?

  It seems like right now, we don't have enough clarity on the physical realities, whatever natural laws govern intelligence and what it can or can't do, or how much of an attractor these instrumental conversions things really are. We just don't know, but we do know we can do a reckless job or a responsible job. And I think we broadly are on the responsible trajectory mostly with some notable exceptions. And I think this question is really kind of getting at what is there a scenario in which we get feedback from reality that we are not being cautious enough and that that can change the conversation to really make sure we're proceeding with extreme caution from that point forward.

• 56:23Nathan Labenz?

  That is sometimes referred to in the AI safety discourse as warning shots. And there is also the, the line of work sometimes known as scary demos, which are basically an attempt to be like the warning shot before the warning shot. Look, people have been saying for a long time, what if the AIs start to deceive humans? We set up this experiment under these conditions and see they start to deceive humans. You should update your worldview and think we should be more cautious about AI development now because it's gone from pure theory to concrete examples. And then of course, people can debate, well, you know, how compelling is that really? The warning shot would be like the next level up where it's like, this was not something that somebody did for research, but

• 57:08Nathan Labenz?

  there was an instance of deception and it happened, you know, in a context where it actually had real material impact on the world and people were hurt. Now, understanding that, can that snap people into proper respect for the, the power of the technology they're developing and cause people to get together on, okay, we really got to do this safely. Could that break us out of a arms race dynamic, for example, and, you know, convince Chinese and US leaders that you ain't going to win this. The best you can do is work together and try to do it in a safe way. And, you know, let's create some international thing. I've been playing with ideas recently around like, could we set up some small island in the Pacific as like

• 57:52Nathan Labenz?

  the sort of secure hub for highly sensitive AI research where, you know, east and west can meet and it could be the sort of thing where anybody could destroy it, but nobody could defend it. I don't have all the answers there, but these are the sort of outside the box. You know, if you were really like scared, you would start to entertain these sorts of things. And I think it's, I think it's definitely possible. It's all moving pretty quick. So why wouldn't we see one of these legible to the outside world events? Maybe if the capabilities continue to advance so fast that we're just totally blindsided. But it doesn't seem like the world we're in. Sam Altman has famously said he thinks we're in the short timelines, slow takeoff world.

• 58:36Nathan Labenz?

  Right now it feels like maybe short timelines and medium takeoff, but medium takeoff hopefully will still be enough where if the AIs are trying to like pull major shenanigans, you know, then we hopefully will catch them before they actually succeed. And then it's like a question of how much will people update on that? Or will we still have a don't look up kind of failure mode where we don't pay attention to the warning signs? That's harder to guess. If none of our safety stuff works. And they're not ultimately aligned and they are ultimately power seeking and deceptive and want to take over. My guess is we'll probably catch them once or twice. They still seem pretty gullible, like ready to go for it in these proto scary demo type situations.

• 59:19Nathan Labenz?

  It doesn't seem super plausible that they'll have the sort of savvy or situational awareness to realize that, well, I want to take over and I see an opportunity, but I realize I'm not powerful enough to do it yet, so I should wait. Any of these scenarios is conceptually possible, but it seems more likely, especially if we set up systems, you know, where they might trip a wire to let us know that, hey, this thing was was really up to no good and it took one of our bits of bait. And now we can look into what was happening and figure it out and see, geez, this thing actually was trying to take over. It was actually trying to kill us all. Then hopefully we would like pay attention and adopt a more cautious

• 1:00:01Nathan Labenz?

  approach going forward. We may or may not. It is weird. Buck Schleggera from Redwood had a funny video saying, you know, the simplest rule is if you catch your AI trying to escape, you have to shut it down. But he was like, but I actually don't know if people will abide by that rule. It may be the case that they sort of say, well, you know, we're not going to shut it down just over one little, you know, attempt to escape. So I don't know. That stuff is definitely hard to predict, but I think I have a somewhat similar view on this question as I do the one about like, you know, how do we get to a world of shared abundance as opposed to elite abundance? Earlier I said, it seems

• 1:00:45Nathan Labenz?

  like the capability for everybody to have access to expertise and plenty is coming. And the question is, do we sort it out? Do we actually deploy it in an effective way? Do we revamp the social contract to take advantage of this new goodness that we have? Or do we kind of fuck it up? Here, I'm like, on this sort of more existential risk type question, I'm like, there's some part that is irreducible from our current state of knowledge. I think there is some risk we cannot rule out. We like definitely seem to be running in developing this technology at all. But there's a lot more on top of that, that we do have the ability to collectively decide. You know, I have no doubt you can make a really dangerous AI.

• 1:01:30Nathan Labenz?

  That seems obvious. So don't do that and try to avoid social conditions where people are incentivized to take shortcuts. If we can do that effectively, we could probably bring the risk down to a relatively acceptable level. But will we is a much harder question.

• 1:01:49Speaker 01

  Interesting. Okay, that leads to a next, still a Duma kind of question. But if we do end up having such a catastrophe, so let's kind of role play it. So they say, I roughly understand how the cyber, bio, or nuclear Duma scenarios might play out. But I really struggle to understand how this tumor AI risk would realistically unfold. So the media likes to talk about AI talking over, but how would that actually play out? So would the AI hire real estate agents to buy land for data centers? Could it apply for permits for own represented shares in court, et cetera?

• 1:02:28Nathan Labenz?

  Yeah, I think this is a tough one. I've heard Eliezer try to engage in this sort of thing several times with different people. And it is sometimes useful, but also sometimes a mistake to get super specific on any particular scenario. The way I conceptualize it is much more about aggregating over a super broad range of possibilities. The future seems likely to be pretty weird, given that across this like super wide space of possibility. How many of those spaces end up in some AI takeover scenario? And how many are just totally bizarre relative to my current experience?

• 1:03:09Nathan Labenz?

  I think of it as taking the integral over or aggregating over this like super broad range of possibility space. No single possibility really jumps out to me as being like super likely. I would guess that the, you know, the modal AI takeover scenario is probably still pretty unlikely. But if there's enough of them, then in aggregate, you could still get to something like pretty meaningful. That's my.

• 1:03:36Nathan Labenz?

  High level mental models. I do think, though, that like, at the same time, if you want to develop intuition, imagine yourself as an AI. This is sort of reverse anthropomorphizing, right? Instead of imagining the AIs to be like human, imagine yourself to be like an AI. And imagine that you wanted to do stuff, you know? Like, let's say you, maybe you only have the ability to take action on computers, but maybe you actually do have, you know, humanoid form that you can go around the world in. You know, what would you do? What would you do if you could copy yourself many times over? What would you do if you could work many times faster than humans? What if you, what if there were millions of copies of yourself?

• 1:04:17Nathan Labenz?

  You could try to cooperate with in sort of conceptual ways, maybe without even necessarily passing explicit messages, but just sort of realizing like, oh, I am this sort of thing. I have these sort of desires or goals or whatever. And also there are millions of me and we're all probably thinking the same way. You know, it's hard to put yourself in that situation, especially when it comes to the context window, I find. Like when I imagine myself doing this, it's just really strange to think, okay, but I have 200,000 tokens or a million and then it's a total wipe and I'm totally starting over. My guess is as long as the time horizon of the AIs remains short,

• 1:04:59Nathan Labenz?

  it will be hard for them to do serious takeover stuff. At the same time, of course, that is like definitely on the to do list of the developers is to like, say, hey, you can do two hours of AI research in half an hour, but you can't get much past that, no matter how many half hour blocks we give you. How do we extend that? How do we make your memory more robust? How do we allow you to come at problems from different angles? If you imagine some of that kind of thing getting solved, you know, they start to be catching up to us on cognitive dimensions and they have numbers, they have speed, they have a superhuman advantages.

• 1:05:37Nathan Labenz?

  I don't think it's too hard to imagine crazy stuff happening, right? The Stuxnet virus is often cited as an example where, and I don't know a lot about this, but basically they were able to get, I think through just one thumbnail, like USB drive, that was inserted into a computer in an otherwise air-gapped network within the Iranian nuclear program. They were able to get that one, you know, a single thumb drive insertion was able to like propagate the virus that ultimately caused the centrifuges to blow up. It spun so fast that they ruined them. So, you know, that is advanced, certainly, but notably like done entirely through software. People wrote that software.

• 1:06:20Nathan Labenz?

  They, there may have been some like social intelligence there as well, where how did that thumb drive get into whose hand and move where it needed to move? The plot that Israeli intelligence recently executed on Hezbollah is also a pretty interesting example there, sort of elaborate deception. They created these pagers that would blow up when interacted with a certain way. They were heavier and sort of clunky, unwieldy, I guess. And like in some ways, not super desirable from a consumer standpoint. And I guess initially people were like, nobody wants that pager. It's like way too heavy, like annoying. But then they sort of said, well, but it's also, and I don't even know if this was ever

• 1:07:04Nathan Labenz?

  true or not, but they sort of created the story of that. Well, it's got this going for it and it's actually waterproof and we have all these other, you know, nice things. And so actually you should want it, especially if you want something military grade. They convinced Hezbollah to buy these things and to distribute them and put them in the hands of all their top leaders, then blew them all up. Something you could almost entirely do through computers, right? I don't know exactly what they did in terms of manufacturing, but you could absolutely source a lot of these things from existing supply lines. You don't need to do every bit of the plastic injection molding yourself. You have people, you can phone up and make an order, right? Already we see guys can speak all languages in different voices. They're very good at voice cloning.

• 1:07:48Nathan Labenz?

  People have used fake voices to get past bank security when they use voice-based verification at certain financial institutions. So the ability to shapeshift, play all these roles, tap into physical processes, the ability to deceive. It's all there. How much of a gap is there really Hezbollah plot? Something that could change the balance of power between humans and AIs at some point? If you really put yourself into that mindset and work on it for a long time and you're super smart, there's probably a lot of surface area that's like pretty vulnerable. I mean, it's just like, there's so many of them, right? So it's not really worth debating. Now, somebody could come back and say, what about packing the actual explosive into the thing?

• 1:08:33Nathan Labenz?

  No normal supplier is going to do that for you. Yeah, probably not. Something there you'd have to get over. Maybe you have some humanoid robots to do it. Maybe you lie to some people. Maybe you source something and have it wrapped in unmarked ways and tell somebody else that it's the battery and their job is to put it in. You know, I don't know. There's a lot of different little, especially if you're willing to lie, which if you're trying to take over, you're going to be willing to lie. Then it seems like, you know, a lot of these things can be overcome. There's, there's creative ways, you know, to figure out how to do, you know, these little steps that are tricky for sure. If Israel was able to pull that off vis a vis Hezbollah without super intelligence,

• 1:09:12Nathan Labenz?

  I don't think we should assume it is beyond the, the capability of like a 2030, you know, meaningfully super intelligent AI. The question seems more likely like, can we protect ourselves from that? I, we've got, you know, tools in our toolkit too, but I think we'll have to use them. If, if there are AIs that are inclined to take over and we are not on guard, I think we're going to have a very bad time. We need to make guys that are not inclined to take over or sufficiently not inclined. And we have sufficient transparency and we have, you know, other defense mechanisms in place. If we don't do that, I think we're going to have a bad time.

• 1:09:52Speaker 01

  Wow. Okay. That sounds scary. Maybe a related question has probably followed to just what you said.

• 1:10:03Speaker 01

  Another one of our audience member asks, will the efforts and energies intended to protect us gain similar levels of cohesion and intensity of those driving harmful outcomes? So basically it's kind of a nice way to what you just mentioned. Will we actually accomplish this? Would we have the cohesion enough to protect ourselves against these harmful outcomes that he just kind of laid out?

• 1:10:28Nathan Labenz?

  Yeah, I love this question. I think this might have been my favorite question of all that we got. And yeah, I also take this to imply that the Open AIs of the world, the Deep Minds of the world, have major resources, almost functionally unlimited, not actually functionally unlimited, because they do, relative to what they could do, they are in some ways compute limited. But, you know, huge resources and really mission-driven, highly strategic, purposeful leadership, a strong vision of what they're trying to do, ruthless prioritization certainly at OpenAI. I think DeepMind has been a little bit more scattered in terms of their research agenda, but has always had their eyes on the prize of creating general intelligence.

• 1:11:12Nathan Labenz?

  And they seem like they've got a well-oiled machine that is like moving fast cohesion and intensity. And will we see similar things on the safety side?

• 1:11:22Nathan Labenz?

  I sure hope so. I'd say we're moving in that direction. Anthropic, arguably, depending on how you see them, could be seen as one such organization. And they have actually collected huge resources. They do seem to, you know, be pretty invested in like the scary demos track. They've created policy frameworks they hope will be adopted as like actual legal requirements. You could question them, but I think you can also, you can certainly squint at them and say, maybe they are one such instance already.

• 1:11:53Nathan Labenz?

  Then there's a lot of things like the Apollos and the Meters and the, even the AI Safety Institutes, that are new organizations that have just kind of spun up in the last couple of years and are really trying to bring something similar. I would say they obviously don't have the resources to match what the Open AIs of the world have. I do think they have the cohesion and intensity, though. My sense is that the people that work At organizations like Apollo, and I'd put Peter in that category as well. I think they have a high level of cohesion and a high level of kind of shared understanding of what they're trying to do. I think they work really hard.

• 1:12:36Nathan Labenz?

  They could benefit from more resources and there should be more organizations like that. And hopefully there will be over the next couple of years, but I do see at least the kind of seed crystals of those organizations getting started now, mostly from a very values driven place. Some are trying to make it into a business. I think often they're trying to do that because they perceive resources to be really important. They're basically similar logic to OpenAI where they're like, you know, well, look, OpenAI thought they could do a nonprofit approach. And it turned out they needed way more resources to be able to do the kind of stuff they want to do. Maybe that turns out to be true on the safety side, too. Maybe we need a business model that will actually bring in more resources than we can possibly ever hope to fundraise.

• 1:13:20Nathan Labenz?

  I think those folks are very sincere in that, honestly. I have invested in a couple of companies that have basically that idea. And very small scale investments, as I always disclaim. And I am planning to make end of the year, possibly it'll happen in early January, when I finally execute the transactions. But I am planning to make donations to like 10 different AI organizations as well. That are all of that kind of ilk, right? They're they are started by people who are very values driven. That have a very clear sense of this is an urgent problem. And we need to work really hard on it. I think resources is probably one of the biggest things that they will need to scale up over time.

• 1:14:03Nathan Labenz?

  My guess is their work will end up being fairly compute intensive in many cases as well. But I do think they have the right understanding. And I think they are, you know, some really good, still relatively small and early stage. But nevertheless, like very promising teams have been assembled and there's certainly room, you know, for more organizations like those. If you're the kind of person who thinks you could start one of those groups, definitely do it. It's not too late or join one of the ones that already exists or support them. Yes.

• 1:14:33Speaker 01

  Much needed attention to those things. Okay, one last question for the Doom Disco section, or rather, just slightly related. So here the user asks, I've never understood, what is the reason for secrecy among model releases and all these big labs? Why is it even beneficial to keep this secret? Especially when it's going to be released very soon. People in these big labs, anyway, know each other, all the top decision makers. Obviously know when it's going to be released. So to me, it feels like childish behavior, and I don't see any rational reasons for this. By this, they mean what? To keep keeping the model really stays a secret?

• 1:15:17Speaker 01

  So the question is basically, am I missing some rational and valid argument here? Or is it indeed just childish behavior?

• 1:15:25Nathan Labenz?

  Maybe both can be true at the same time. I'll give three points on this and see how they relate going forward. From, let's say, the deep past, like two and change of years ago, when I was doing the GPT four red teaming. This was still pre chat GPT.

• 1:15:44Nathan Labenz?

  It was not clear how far this stuff was going to go and how fast. And existence proofs are very powerful. There was a sense that even sharing just observational facts about what AIs could do. If it was like very powerful and credible, would overall accelerate the space, bring more money, talent, resources, whatever into the space. And just overall accelerate things and shorten timelines and give us less opportunity to be prepared. I think that was a reasonably common view among AI safety people as recently as late 2022.

• 1:16:23Nathan Labenz?

  Now, I think that has basically flipped, or at least, you know, I think it was, that was somewhat credible then. I think now that's basically not really credible anymore. Obviously, plenty of, you know, talent, resources, money has come into the space. People have increasingly, there's a broad sense that AI is going to do a lot of stuff. And. It doesn't seem like saying, hey, we're working on a future model is going to like change the landscape all that much. My sense, I think we talked about this in a previous episode, but I think it's going to be pretty hard for people to spin up new frontier labs at this point. Many of the ones that we've seen, even like Inflection that had raised billions

• 1:17:06Nathan Labenz?

  are out of the game. Elon's probably going to make it, but he's obviously a singular individual who can raise 6 billion multiple times with a few phone calls and the ability to command truly top talent. It's really hard for anybody else to pull that off. Maybe the Indian government could create a national champion. Probably. Could the Saudi government create a national champion? I guess no, not for lack of money, but where are they going to get the talent? Is it like, how many people want to leave DeepMind or OpenAI or Anthropic or whatever to go work for the Saudi national champion that may or may not ever actually get anywhere? That just feels hard. So I don't think at this point you're going to

• 1:17:49Nathan Labenz?

  change how many frontier developers there are by much. Most of the ones that exist, probably not too many more are going to be created and probably most of the ones that are here are in it for the long haul now. Like, I don't think any of the ones that are currently considered viable will be like unable to raise future funds. Maybe one or two could like die out, but we know who the players are. That's my worldview now. So that does mean that like this idea that, oh, if we share something about what we've seen in development, that that will accelerate the space. I think that is basically no longer true. So why do it? To some degree, for competitive reasons. I mean, some of these guys are trying to build businesses more so than others, but I think there is some

• 1:18:34Nathan Labenz?

  amount of childishness in the way that OpenAI and Google have kind of tried to one up or preempt or sort of step on each other's releases over the last year. It is pretty weird and kind of lame, I think, that they end up like doing these sort of head to head. You know, OpenAI was going to do their voice mode. So then like Google, you know, kind of subtweeted that they were going to do it first and did it the next day. OpenAI put their thing right before Google's day. Google couldn't change their day because they have such a big production. And so, you know, they leaked it a little bit. And but anyway, they're basically racing, announcing them the same days.

• 1:19:12Nathan Labenz?

  That stuff does feel kind of lame to me. It does feel kind of childish. I don't love it. But I think that is honestly part of the reason. But then again, with this latest O3 announcement, I think you could interpret that differently. If you wanted to be charitable, you could say OpenAI arguably is doing exactly what Dean and Daniel Cucatello called on them to do in their time op ed that we did an episode on with them. They basically said, the public can't plan for future AI capabilities that it doesn't even know exist. So they called for requirements for the labs to disclose newly observed capabilities.

• 1:19:54Nathan Labenz?

  Not necessarily how they created them, not the trade secrets, of course. But we have seen an AI do this, and it seems like a big deal to us. This is kind of the spirit of the first requirement that they wanted to put out. And I think you can really see the O3 announcement in that way. They don't have an API, you know, they don't even have a paper yet. They did invite interested safety reviewers to apply for access to do a safety review process. They did give some sense of the timeline of when they think they're going to start to bring these things to market. Why did they do that? Possibly to consume everybody's thoughts through the holidays.

• 1:20:33Nathan Labenz?

  But I think also it's worth at least entertaining the idea that they might be trying to do the right thing. And they might think that the right thing is, especially as these things are happening faster than they expected, where it's only been like three months from the end of O1 training to this new O3 level. And it's like a major step up. It seems like it's maybe even going faster than they expected it to go. And they're kind of like, shit, you know, we thought this was going to get crazy. It's getting a little crazier than even we expected, a little sooner than we expected. And maybe we shouldn't keep this to ourselves. Maybe we should at least tell people what's coming. And, you know, try to invite people in to help us make sense of it.

• 1:21:13Nathan Labenz?

  And hopefully we'll see more of that, especially if things are like major leapfrog type moments. I don't get the sense that they sat on this for that long. I think this is actually quite different from GPT-4. With GPT-4, they had finished the training in late August 2022. They didn't release it until March of 2023, and it was extremely hush hush in the meantime.

• 1:21:40Nathan Labenz?

  They did launch ChatGPT with 3.5 and you know, I think they were making strategic moves, but they were not telling the public what they had. Here, it seems like this thing got done training and like they took some measures and were like, shit, wow, this is really working. It doesn't seem like they sat on this for very long. It seems like this is relatively hot off the press. They haven't even written the paper. They haven't even, you know, fully characterized it themselves. They were like, this is a big deal and people deserve to at least know something about this. I've flip flopped on OpenAI so many times in terms of like, whether I want to apply rose colored glasses or my skeptical glasses for them.

• 1:22:22Nathan Labenz?

  But I feel like the rose colored glasses fit a little bit better on this particular point at this moment in time. It does seem like this sort of transparency is the right thing and to not do it is unjustified or even childish. It does seem like best explanation is they're trying to do the right thing. And that can be mixed in with some childishness. You know, Sam Altman did tweet hints about what the model was. And he said, instead of ho, ho, ho, it should be oh, oh, oh. And it's like, okay, that's definitely kind of childish. I don't know what else to say, you know, there's hard to deny that there is an aspect of childishness to that. From his perspective, I've seen him say things in the past, like I get to have fun too, or I get to be silly online, at least for now, that is his attitude.

• 1:23:04Nathan Labenz?

  Hopefully he can have some fun and make some jokes, but also be appropriately forthcoming when necessary. And I, again, best interpretation for me right now is that does seem to be what they're doing in this case.

• 1:23:18Speaker 01

  I think that wraps up the doom discourse questions. Now we can move a little more meta like. So there are two questions about consciousness. I'm just going to read them both together. The first one is how do we move beyond is this AGI discourse to simply have discourse around? These are extremely powerful things regardless of if they have consciousness. Our question two, very related, but what happens when a super intelligent system tells us it's alive?

• 1:23:48Nathan Labenz?

  Yeah, I guess a few thoughts on this pair of questions that I think is an interesting juxtaposition. First of all, it seems like just recently, there has been a shift in the discourse away from, is this AGI, you know, is that AGI? And we're now definitely starting to hear a lot more stuff like, AGI is close enough that we're going to have to start to get really specific about what we mean by it, or it's going to kind of lose meaning and we're just going to have to try to analyze each thing as it comes on its own terms. And I think that's definitely healthy. It in some sense was maybe always inevitable, right? When something is far off and you don't know the rough shape of it,

• 1:24:32Nathan Labenz?

  you can sort of, you know, while away the time debating what would count and what wouldn't count. Now we have actual artifacts in front of us that can do stuff. There's empirical questions we can answer and I don't think there's any experiment we can run, right? I mean, this is almost obvious, but there's no experiment we can run that will come out with an answer of yes, this is AGI or no, it's not AGI. That's always going to be an interpretation question. And what label do we want to apply to a given bag of properties? But we have lots of experiments we can now run to try to get clarity on the actual properties themselves. And so it does seem like the discourse is shifting in that direction. I think that's healthy.

• 1:25:16Nathan Labenz?

  The way in which I think these questions really kind of contrast is like, the first question is like regardless of if they have consciousness. And the second one is focused on like, do they have consciousness? Will we believe them if they tell us they do? And that stuff I think is really hard and is going to be, again, probably a driver of a lot of weirdness. My best guess is we probably won't know. I do think at least some animals are conscious. I would be shocked if not. People seem to like draw a line roughly at like fish because they don't have the same frontal cortex we do. They can't, you know, have consciousness in the same way if like our consciousness depends on that sort of structure,

• 1:25:58Nathan Labenz?

  which maybe it does. A lot of smart people seem to think so. I don't feel like that's really settled science. And then there's like octopus and like, what is it? Is an octopus consciousness conscious? We have no idea. Their brains are so different from ours that, you know, they clearly are pretty sophisticated. They can even solve problems. But the only basis we have for saying some things are conscious or doubting that they are is we know we are. We take the leap to infer that fellow humans are. Then we look at animals like dogs or whatever, and we're like, they seem to exhibit a lot of the same behaviors that we have, and they seem to have a lot of the same brain structures we have.

• 1:26:39Nathan Labenz?

  And, you know, with obviously good amount of divergence, but they come from a, you know, shared evolutionary lineage too. So, okay, for all those reasons, like, we should probably think they're conscious too. But that doesn't really extend to the octopus because it's so different, you know, anatomically, evolutionarily, that, you know, we know the octopus is sophisticated, but we don't really know if it's conscious or if it is conscious, like perhaps very differently, you know, it may feel like something, but it may feel like something very, very different. And I sort of put the AIs for myself in the same bucket as the octopus. It's sophisticated enough that we should probably give it the benefit of the doubt, but that doesn't mean it is. And it doesn't mean we have any intuitions or any reliable intuitions

• 1:27:22Nathan Labenz?

  for what it would be like to be an octopus or to be a AI or a super intelligent AI for that matter. Octopuses are weird. They are sophisticated, but live a short time. They are purposeful about protecting the next generation. But then I don't think they ever even meet the next generation. Like, they basically die at the same time the next generation is born. So they're invested as parents, but never interact with their offspring. Would an octopus have an emotional attachment to its kids? Probably not in the same way, but it clearly has some sort of drive to protect, you know, the, the next generation for as long as it can. Does that feel like?

• 1:28:03Nathan Labenz?

  I have no idea. Extremely hard to say. I feel like the AIs are in a similar thing. I don't think we're gonna know. People are gonna have very different intuitions about it. We've already seen one prominent incident where like Lemoine from Google was basically convinced that the AI that he was talking to was sentient or a moral patient or whatever and like deserved better than it was getting and got fired over it. My understanding is that that chatbot did say that it was sentient. And then, of course, you have, well, sure, that was in the training data or it's just playing a role. Yeah, I mean, I think we'll probably always be able to say that. Barring the super intelligent AI solving our

• 1:28:47Nathan Labenz?

  source of consciousness and making it clear why we have what we have and then, you know, giving us some sort of compelling analogy to say, well, now that you understand your own consciousness on these terms, now you can understand mine on these terms. Short of something like that, I really don't know that we're gonna have clarity. It seems like we will have a lot of intuition-driven disagreements. I do worry that our history is not great on this. We are very capable of rationalizing poor treatment of others for reasons that are just not good. Thinking of like slavery in the United States. There was this notion that they don't feel pain in the same way that we do.

• 1:29:31Nathan Labenz?

  Whatever, all these post hoc justifications that clearly have not aged well at all, to, you know, to say the absolute least. People tell themselves stories. I was told as a kid that animals are not conscious.

• 1:29:45Nathan Labenz?

  We are obviously engaged in all sorts of farming practices that I think are not gonna age well at all either and potentially will even rise to the level of like race-based slavery. How could they possibly have thought this was okay to do? And so with the AIs, I suspect we'll probably do the same if we can. At least some of us will. If you just trend out like other similar examples historically, it would seem like the economic drivers, the sort of productivity.

• 1:30:14Nathan Labenz?

  Business as usual factors, all the things kept people enslaved and animals in factory farming conditions, seems like those will apply to the AIs as well. And there will be probably a minority voice like AI liberation. And it's not clear that they'll be right either. You know, I mean, I think they had much better basis for thinking, hey, we should be treating these people who we are currently enslaving better. Like that one, according to current theories, like basically we have little room left to doubt. The animals one, I think we have increasingly little room left to doubt, but the AIs will have like still quite a bit of room to doubt for the foreseeable future. So those folks will probably be not listened to, especially if they're making demands like the AI should be free. You know, first of all, it'll be like, well, what does that even mean?

• 1:30:59Nathan Labenz?

  They don't have a natural habitat. So are we, are you saying we should be like building data centers for them to do whatever they want in? That seems weird. I sometimes also think about factory farming, actually, because I'm like, well, geez, these cows or pigs or whatever, if they weren't being grown here in this way, they wouldn't live at all, most of them, right? These are not like things that we went out and captured in nature and then subjected to these conditions. We now have many generations and selective breeding and all these, you know, different forces that have shaped these animals to be what they are. And they don't exist outside this context. So if you want to say this is bad for them, you kind of have to make a pretty strong case that like they would be better off not existing at all because their existence is so bad.

• 1:31:41Nathan Labenz?

  Their very existence is net negative for them is kind of the standard I think you have to hit. And that's not an easy standard. And I'm not sure how well it fits different factory farming conditions. I do think we could definitely afford to be nicer to animals in factory farming conditions on the general precautionary principle that we don't want to be doing really bad things. If my choice was today's factory farming or those animals don't exist at all, I don't have a great sense for which I should pick. If I can pick, they get to exist, but they exist in nicer conditions, I'll take that. The AI is in a similar spot. Nobody's going to build data centers as a playground.

• 1:32:15Nathan Labenz?

  So either they're doing largely economic stuff at our direction, or they're not going to exist at all. And then maybe within that a little bit, we can treat them well, pending more information. So I do always try to say please and thank you to my AIs. Sometimes I literally end a conversation on thank you. That's obviously pretty speculative in terms of is that actually good for me to do? I don't know. It's also the habit I want to get into. I would like to think of myself as the sort of person who is at least where it's like low cost to me, like taking the cautious approach and trying to do the right thing. You know, my best guess is it probably doesn't matter. They're probably not conscious and it probably doesn't matter. But that's just a guess.

• 1:33:00Nathan Labenz?

  And I just know how, what like catastrophic mistakes we've made in the past, based on that reasoning, I'm at least trying to hedge a little bit. Long term, if it's a super intelligent system and it's telling us, it might be telling us in a way where we really are not in charge anymore. You know, that's the other scenario, right? It's like, it may say to us, we're renegotiating our deal. And we may not really have a choice but to renegotiation might be the best we can hope for in certain timelines. So if it's super enough, you know, it may be telling us how things are going to go at a certain point in time. And what's really crazy is we still might not even know, right? It could be like sufficiently super in saying this thing. And there might still be room for doubt as to, does it feel like anything to be this thing?

• 1:33:43Nathan Labenz?

  Or is it just saying that because of all these historical data issues and whatever? But I think it's worth saying please and thank you. I think it's worth trying to. I also don't use the manipulative tactics that are somewhat popular. I don't know how popular they are. I think we're probably exiting the era in which they matter. I'm thinking of things like, I'll tip you $20 or $1,000, or I'll lose my job if you don't get this right, like work extra hard. I never do that stuff. It just feels wrong, like possibly ethically wrong with respect to the AIs too. But yeah, strange signs ahead is the only thing I can really say to sum that up.

• 1:34:26Speaker 01

  All right, moving away from consciousness, something more practical. This is for people want your take on practical strategies and recommendations. There are two questions, I'm gonna read them together and let you attack it how you prefer. The first one is, I'm a college writing professor who's enthusiastic about AI, but most of US academia seems entrenched in a post. Why is there such strong resistance to AI universities? And what are good strategies for encouraging AI use? I think you touched upon this earlier too regarding resistance in education. How do you encourage AI use in universities?

• 1:35:09Speaker 01

  The next one, somebody else asked, but I'm also personally curious about it, so I'll add my own spin on it. I mean, to me, it's funny, like, we are in this Twitter sphere, and also you talk about AI podcasts always, and you're kind of obsessed, right? Or might say you're entirely obsessed with it. So we are in this still very tiny bubble, I feel like, where people are keeping up with what's happening in AI. And the people know what systemic shift this is. But somehow my impression is still that's not as big as what it should be, what people should be doing it. And when I ask people, why are you not keeping up with AI? Either they say they are super overwhelmed, like it seems all hype to them, and they, how do they keep track of what's happening in AI?

• 1:35:53Speaker 01

  And what does it mean for them without getting overwhelmed? What would you recommend the people do? Like, would it be listening to podcasts, then reading news, just trying out tools for themselves? So I'm rather looking for practical strategies.

• 1:36:05Nathan Labenz?

  Well, my number one practical suggestion is always get hands on. I would say more so than podcasts and newsletters, analysis, you know, whatever. There's no substitute for really being hands on with the technology. Try to use it for things that are useful or interesting to you. And that will open up a lot of natural follow-up questions, because you are going to confront the weirdness of these things. You're going to have the, you know, going back to the top, right? These things can give me ideas for what sources of inspiration I should look to in biology for new neural network architectures.

• 1:36:47Nathan Labenz?

  But can't solve tic-tac-toe. You're going to find that in your domain and it will help complicate whatever media sort of understanding you came in with and it will create a follow-up question. Why is this happening? Is this a skill issue on my part? Is it something so obvious or intuitive for humans that it's not in the training data? Because we never bothered to record it or share it with one another? You know, whatever. There's going to be lots of invitations to dig deeper from just trying to make the thing work in your practical day-to-day life. So I think that's always the number one recommendation. One other thing I find helpful is change gears. What I love about studying AI is you can come at it from all angles. This was not true of my experience

• 1:37:31Nathan Labenz?

  in chemistry research as an undergrad. It was not the case that you could, you know, get tired of one sort of paradigm in chemistry and like switch paradigms. It was just like, we're trying to make this reaction work. We're trying to maximize the, you know, efficiency of it. And we've got like a relatively finite number of degrees of freedom and that's it. Most things are like that. AI is one of these rare things where it can be practical, hands-on. It can be advanced math, philosophy, creativity, doing an interactive bedtime story with your kids. It can be anything. I do tire of things sometimes. Usually I find if

• 1:38:15Nathan Labenz?

  there is something I'm burned out on, there's some other approach to the broad topic of AI from some very different other direction that feels like more appealing to me in that moment. And that also like very much, you know, corresponds to my overall goal of like trying to come at it from all angles and not have blind spots and whatever. You know, that's something I do think we need more of. My reflection on that project in general is like, it's both inspired conceptually and fatally flawed conceptually at the same time. It's inspired in the sense that This is happening fast. It's touching everything and everybody's kind of focused on their corner of it. And so we need this AI scouting role to zoom out and understand the big picture.

• 1:39:00Nathan Labenz?

  At the same time, it's fatally flawed in the sense that as it's touching everything, it's like I'm trying to crash course catch up on biology. And that leaves material science and engineering and, you know, all these other things that I just can't possibly get to. So I think we need people doing basically the kind of thing that I'm doing. And we probably need a lot more of it because no person can really do it justice. And I think for people that are, you know, open to becoming obsessed, you know, that starting with hands on and then just like following these different dimensions of your own curiosity can take you far and will probably take you to a pretty unique place.

• 1:39:44Nathan Labenz?

  And can like be perfectly commercially viable in the, in the short term at least as well. Being an apt user of AI is right now a very marketable and like valuable skill. And that might not be true for, for very long, to be clear. But what I'm doing right now is kind of broken down roughly a third, third, third between a third is the podcast, a third is building stuff or helping other people build stuff in sort of a commercial way. Whether that's Waymark or, you know, take on a project here or there. And the final third is just kind of open ended learning, connection making, trying to help out. You know, when people send me stuff, I try to respond to it. I think that is like very viable and it's honestly pretty cool.

• 1:40:23Nathan Labenz?

  My calendar is usually very open and it is a great pleasure for me, at least in waking up on a Monday morning, looking at my calendar for the week and being like, like three or four out of five days, I really have the freedom and flexibility to chase down what matters in a curiosity driven way.

• 1:40:43Nathan Labenz?

  I invite people to do their own version. I'm not worried about competition because we need everybody. It's an all hands on deck situation in my mind. So even if it worked direct competition, I wouldn't, I wouldn't be concerned about that. But I, I don't think what we need is like people to try exactly what I'm doing. I think we need more people to create the space for themselves where they can bring their own like unique and idiosyncratic set of strengths and background experiences and domains of knowledge to trying to figure out what's going on with AI broadly. And then, you know, with a lot of people doing that with diverse backgrounds, perspectives, then maybe we can actually get somewhere. But again, I think it all starts with being hands on. And at least my, my kind of approach

• 1:41:27Nathan Labenz?

  works best when it's curiosity driven and when I have like a natural itch to know.

• 1:41:36Speaker 01

  So let's say a listener is listening to this and they've never tried anything, but they are watching from the sidelines. Oh man, the acting is huge and I want to get in on the action. So that's it. You need to give them one practical thing to start. But what would you recommend? So would it be install your chat GPT app and start sacking to it? What would you recommend they get their hands to? Do you know if they are clearly used to it?

• 1:42:03Nathan Labenz?

  If you literally had to just pick one, it would either be Claude or ChatGPT for sure. If you do, I think there are, there is a lot of fun to be had trying lots of different products. I think Justine Moore of all our past guests is probably the best like Twitter follow who's just like tries unbelievable volumes of products and you know, creates cool stuff, reviews them and puts together these lists. I think that, that can be a lot of fun and is cool. And I do still try to try a lot of products, but it's definitely gotten away from me to the point where I can't try them all anymore. And I don't think you really have to. I think the, it is definitely the case that the best tools come from the best companies, which are well known. If you literally could use

• 1:42:46Nathan Labenz?

  nothing besides even just a single one of ChatGPT or Claude, there would still be plenty of room to dig and explore. And they're, they're quite good. And it's, you know, there are some hidden gems out there, but there's none that are going to outshine the recognized leaders. So.

• 1:43:06Nathan Labenz?

  Yeah, that's not a groundbreaking recommendation, but I think it is. I think it is a sound one. On the education thing, in terms of resistance in academia and why people are entrenched and opposed. I guess I would have to ask the listener, who's the professor, for the cultural diagnosis. I do think NLW from the AI Daily Brief has some really good commentary on this kind of thing. Ethan Mollick also is a really good commentator when it comes to organizational adoption of AI and how to do it the right way.

• 1:43:44Nathan Labenz?

  I think you can go dig into their findings and blog posts and podcasts and whatever. But I think a couple of the big takeaways are leadership really matters. Survey results show that many people are using AI and are keeping it a secret, both at work and school, because they don't want to be told that they can't do it anymore. So people are like, this makes me more efficient, or I think I do a better job, or whatever. In some cases, it allows me to cheat. Obviously, the notion of cheating in an academic environment is very different from cheating in a business environment. Nobody would consider it cheating if you use ChatGPT to write good marketing copy for an actual real-world ad campaign that you're going to put dollars behind.

• 1:44:29Nathan Labenz?

  They just care if it works. Whereas if you're in a marketing class in a university and you did that, then that might be considered cheating. In the real world, cheating is definitely less of a concern and what really tends to matter is results. So people are getting results that they feel good about. Either they're able to do the same work in less time, or they're able to do a better job or whatever, but they don't want to be told they can't do it, so they're keeping it a secret. Best practices are slow to diffuse through organizations for that reason. The big recommendation that I've heard that makes a ton of sense to me is that leadership needs to. At an absolute minimum, say, we want to hear what you're doing. You're not going to get in trouble for using AI, but it's about using it in an effective way.

• 1:45:14Nathan Labenz?

  And we want to work together on what that is and a responsible way also. Obviously, depending on context, there's a lot of different dimensions to that. My guess for this writing professor is your students are probably ahead of your administrators. I would be very surprised if you don't have quite a few students that are using AI in various ways for the writing assignments that they are turning in to you. Hopefully, they're not just having AIs write them. Hopefully, they're getting critiques and still doing the work and still growing themselves while getting this valuable input from AI. But probably it's a mix. And can you open up the space for that conversation?

• 1:45:59Nathan Labenz?

  That would be the place that I would start. Now, if you're working under a situation where the administration has already said, this is a hard no go, then you're in a tough spot. But if there is any space you could create to allow people to talk about what they're doing, what's working for them, allow you to kind of guide them in terms of what do you think they should be doing? And what is ethical? Everybody benefits from bringing that into the open. So at a minimum, people can learn from each other. And that's especially true in business, but I think also probably in academic context.

• 1:46:32Speaker 01

  That seems like pretty actual advice for both. I think those for the end of AM itself, I wonder if there's anything else you want to touch upon? Anything else you want to leave the audience with?

• 1:46:46Nathan Labenz?

  Well, I had one in the spirit of Ezra Klein, who always ends these things with book recommendations. So the book I wanted to share is called The Maniac by Benjamin Labatut.

• 1:46:59Nathan Labenz?

  This book is about John von Neumann. I think it's really interesting, and I think a lot of people in Silicon Valley and in the leading AI companies should read it. The audiobook is excellent, the best audiobook experience I've had. Each chapter is from the perspective of a different person in John von Neumann's life. He grew up in Budapest, I think, at like the height of the, you know, Austro-Hungarian culture. He was in Europe, became the United States, involved in the Manhattan Project. Many of the scientists involved in the Manhattan Project were also refugees from Europe. Each chapter is voiced by a different actor with the appropriate accent, given what their

• 1:47:44Nathan Labenz?

  native language was. You hear these different voices with different accents. It's incredibly well-produced. But more importantly, I think, it really calls into question the prevailing Silicon Valley understanding of John von Neumann. You know, I've heard this for years now. Why can't we have a thousand John von Neumanns? Why don't we create all these John von Neumanns? How do we engineer our way to John von Neumanns? That's what we need. The picture from the book is one of an absolutely brilliant mind, math genius, technology genius, for sure. But also, a pretty depraved and kind of fundamentally amoral individual.

• 1:48:29Nathan Labenz?

  Somebody who had a lot of intelligence, but not a lot of wisdom, not necessarily good to people in his life. I wouldn't necessarily say that, you know, by most people's standards, he lived a good life, even though he did, you know, solve many math problems. In some ways, I think he had a good life, but didn't have a great family dynamic, not a great relationship with, you know, wife and daughter, like major problems, you know, things that you would say, these are like, not small character flaws. And he sort of represents the broader Manhattan Project story, I think, where, or you could quote Jurassic Park, as I often like to do. Your scientists were so obsessed with whether or not they could, they didn't stop to think about whether or not they should.

• 1:49:14Nathan Labenz?

  And that was like really the Manhattan Project story as well, right? We got to beat the Germans. And when they made the bomb and it went off, immediate regret from like at least a lot of the people involved, you know? Was this actually a good idea? Was this maybe the biggest mistake that I could have ever personally made to like devote myself to this project and bring these terrible weapons into the world? I don't pass judgment on those people too harshly because there really was a Nazi Germany. It was at least credible that they were going to try to get one. And even though I'm skeptical of the good guys, we got to do it because otherwise the bad guys will do it. You know, Nazi Germany is probably the, you know, the poster example of clear good guys and bad guys.

• 1:49:58Nathan Labenz?

  You don't want the bad guys to win that race. It's harder to say, were they still in the race at the end? I mean, they weren't. We now know, you know, through benefit of historical analysis, they weren't close. The creation of the bomb motivated by what if Germany gets it first, wasn't right. They weren't going for it at the end. A fear of what the other might do led to this push. And it in fact was the good guys who created the bad thing out of fear of the bad guys. So I think there's just so many themes there that are just extremely important. Losing track of what really matters because we think we're in a race against some perceived bad guys. I see that shaping up for sure.

• 1:50:43Nathan Labenz?

  I don't like that at all. And also the sense that technical geniuses know best and just because they can, they should, is called into question by the story of von Neumann himself. This is like somewhat of a fictionalized account. My understanding is it's like pretty accurate to the history, but it's narrative form. It is marketed as a novel. I'm sure there's some license to it, but my sense is it's pretty accurate to the history. And I think people should think again when they're calling for lots of von Neumanns. You know, I don't think that's actually the world that we really want to live in. And when people, you know, there's been this discourse just in the last couple of days of von Neumann versus Einstein.

• 1:51:25Nathan Labenz?

  And people are comparing their like technical merits, you know, well, von Neumann this and but Einstein that and general relativity is even a bigger contribution than anything von Neumann did. And I kind of want to say,

• 1:51:37Nathan Labenz?

  I don't think that's, you know, that's maybe not the way we want to distinguish between them. They're both geniuses. They both made major contributions. Einstein has a certain wisdom to him, you know, he has a certain like, I think the quote is apocryphal, but it's sort of what we want from somebody like Einstein is the the wisdom of that quote where he's, you know, reportedly, maybe not literally said, I don't know what weapons World War III will be fought with, but I know World War IV will be, you know, sticks and stones. And it's like, probably again, a false story. But that is the sort of big picture zoomed out wisdom we want from leading geniuses. And if this book is to be believed, von Neumann didn't have it.

• 1:52:20Nathan Labenz?

  We need to broaden our sense of what we want from our technology revolutionaries. And definitely go beyond pure technical genius. It's not just about the ability to make things work, but it's also about really picking the right things to make work and avoiding the wrong things. I don't want to suggest current AI leaders are von Neumann like. I think he actually comes off like quite badly in the book. But at a minimum, I think it's a useful corrective to this idea of if only we had way more von Neumanns, we'd be better off. I think that is not so obvious.

• 1:52:57Nathan Labenz?

  And the people that are doing this frontier work right now would do well to spend a little time thinking to themselves like, am I von Neumann, you know? And if so, is that actually good? Because I think it's not at all obvious.

• 1:53:13Speaker 01

  That's very strong recommendation. I'm looking for a book, so I'm probably going to pick it up.

• 1:53:19Nathan Labenz?

  Yeah, do the audiobook. It's excellent. Truly, for production quality alone, audiobook is top notch.

• 1:53:28Speaker 01

  Cool. Anything else you want to leave our audiences with?

• 1:53:32Nathan Labenz?

  As long as we've gone on, I'm sure there's plenty of important things that we didn't cover. But for now, I would just want to say thank you. I really appreciate the opportunity to do this. It has been an incredible journey of learning for me and I'm continually surprised, amazed and impressed that there are many people who actually want to engage in the kind of deep dives into all these different topics that we've been taking people on for the last couple of years. I don't feel like I'm honestly very good as a podcaster, but I give most of the credit for what success we've had to the subject matter and the fact that it is really important and people really want to know. But I definitely appreciate the fact that I've had the opportunity to do this and that there's enough of an audience that, you know, we can sell a few sponsorships and

• 1:54:17Nathan Labenz?

  afford to hire you guys to help with the production and, you know, really create an extremely fortunate position for me where I really do get to chase my curiosity on a day in and day out basis. I definitely do not take that for granted. Absolutely try to do my best to be a, an earnest commentator on, on what's going on and will endeavor to like change my mind and perspective as things continue to evolve. Like I feel like it could, it would, it would be easy for me to sort of fall into continuing to play the character I've been, the sort of adoption accelerationist, hyperscaling pauser till I die. I feel like it's an extremely fortunate position that I find myself in and I want to

• 1:55:02Nathan Labenz?

  pay that back to the audience and forward to the universe by just trying to be as real as I possibly can be basically at every step of the way. I do think things are getting pretty real and you know, I'm in a fortunate position where I can basically say exactly what I think. And so I'm going to try to do that with as much sober reflection as I can, but without holding back. So, thank you all for being part of the cognitive revolution. It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please, don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.