E16: Pausing the AI Revolution? With Technologist Jaan Tallinn

Nathan Labenz dives in with Jaan Tallinn, a technologist, entrepreneur (Kazaa, Skype), and investor (DeepMind and more) whose unique life journey has intersected with some of the most important social and technological events of our collective lifetime. Jaan has since invested in nearly 180 startups, including dozens of AI application layer companies and some half dozen startup labs that focus on fundamental AI research, all in an effort to support the teams that he believes most likely to lead us to AI safety, and to have a seat at the table at organizations that he worries might take on too much risk. He's also founded several philanthropic nonprofits, including the Future of Life Institute, which recently published the open letter calling for a six-month pause on training new AI systems. In this discussion, we focused on: - the current state of AI development and safety -Jan's expectations for possible economic transformation - what catastrophic failure modes worry him most in the near term - How big of a bullet we dodged with the training of GPT-4 - Which organizations really matter for immediate-term pause purposes - How AI race dynamics are likely to evolve over the next couple of years LINKS REFERENCED IN THE EPISODE: Future of Life's open letter: https://futureoflife.org/open-letter/pause-giant-ai-experiments/ Eliezer Yudkowsky's TIME article: https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/ Podcast: Daniela and Dario Amodei on Anthropic- https://podcasts.apple.com/ie/podcast/daniela-and-dario-amodei-on-anthropic/id1170991978?i=1000552976406 Zvi on the pause: https://thezvi.substack.com/p/on-the-fli-ai-risk-open-letter -- We're hiring across the board at Turpentine and for Erik's personal team on other projects he's incubating. He's hiring a Chief of Staff, EA, Head of Special Projects, Investment Associate, and more. For a list of JDs, check out: eriktorenberg.com. SPONSORS: Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries. From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive Thank you Omneky for sponsoring The Cognitive Revolution. Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work, customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off. TIMESTAMPS: (0:00) Episode Preview (1:30) Jaan's impressive entrepreneurial career and his role in the recent AI Open Letter (3:26) AI safety and Future of Life Institute (6:55) Jaan's first meeting with Eliezer Yudkowsky and the founding of the Future of Life Institute (13:00) Future of AI evolution (15:55) Sponsor: Omneky (17:20) Jaan's investments in AI companies (24:22) The emerging danger paradigm (33:48) AI supervising itself (40:06) Evolution, useful heuristics, and lack of insight into selection process (43:13) Current estimate for life-ending catastrophe (54:20) Our luck given the softness of language models (56:24) Future of Language Models (1:01:00) The Moore’s law of mad science (1:03:02) GPT-5 type project (1:11:00) AI alignment with the latest models (1:14:31) AI research investment and safety (1:21:00) What a six month pause buys us (1:27:01) AI’s Turing Test Passing (1:33:18) Responsible AI development. (1:41:20) Neuralink implant technology

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  There is a lot of reasonable discussion that happens like in the labs and even between the racing labs about the need to coordinate, need to be careful. And like people have been kind of public about it, etc. But like, there's always like one missing component. The component is like, when? Like, if there is this pause and like associate that realization that like these experiments are considered too reckless by society, this will create some kind of incentive gradient for the companies themselves to figure out how to make them in a more responsible manner and more legible manner. But if we survive, the life, the world, the universe could be potentially

• 0:39Speaker 00

  like unfathomably better than it is now. So in a sense, we are living a lottery ticket. And it is in some way in our control to improve the odds. And so that's what I'm doing.

• 0:52Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg.

• 1:15Speaker 02

  Before we dive into The Cognitive Revolution, I want to tell you about my new interview show, Upstream. Upstream is where I go deeper with some of the world's most interesting thinkers to map the constellation of ideas that matter. On the first season of Upstream, you'll hear from Marc Andreessen, David Sacks, Balaji, Ezra Klein, Joe Lonsdale, and more. Make sure to subscribe and check out the first episode with A16Z's Marc Andreessen. The link is in the description.

• 1:44Nathan Labenz?

  Hi, everyone. Our guest today is Jan Tollen. Jan is a technologist, entrepreneur, and investor whose unique life journey has intersected with some of the most important social and technological events of our collective lifetime. Born in 1972 in then Soviet Estonia, Jan was 17 years old when the Berlin Wall fell, and he quickly became a video game entrepreneur. Years later, he created Kazaa, the famous P2P file-sharing platform that, at its peak, accounted for half of all internet traffic.

• 2:19Nathan Labenz?

  From there, he went on to cofound Skype, which eventually sold to eBay in 2005 for $2.5 billion, and for years remained the most successful internet company founded outside of the United States.

• 2:34Nathan Labenz?

  Circa 2009, Jan came across Eliezer Yudkowsky's AI risk writing, which he found extremely persuasive and which inspired him to dedicate his time, resources, and personal credibility to existential risk mitigation, with a particular focus on AI. Jan has since invested in nearly 180 startups, including dozens of AI application layer companies and some half dozen startup labs that focus on fundamental AI research. Those include DeepMind, Anthropic, and most recently, Conjecture.

• 3:09Nathan Labenz?

  He's done all this in an effort to support the teams that he believes most likely to lead us to AI safety and to have a seat at the table at organizations that he worries might take on too much risk. He's also founded several philanthropic nonprofits, including the Future of Life Institute, which recently published the open letter calling for a six-month pause on the development of AI systems more powerful than GPT-4.

• 3:35Nathan Labenz?

  With so much happening in AI right now, I decided to touch on Jan's personal story and to discuss Eliezer's baseline AI safety worldview only briefly in the first part of today's conversation. Instead, we focused on the current state of AI development and safety, including Jan's expectations for possible economic transformation, what catastrophic failure modes worry him most in the near term, how likely he believes next-generation systems like GPT-5 are to literally end the world, how big of a bullet we dodged with the training of GPT-4, whether in some sense we are lucky that language models are softer and slower than alternative AI paradigms,

• 4:17Nathan Labenz?

  which organizations really matter for immediate-term pause purposes, to what extent those organizations are currently coordinating or slowing down already, how AI race dynamics are likely to evolve over the next couple of years, what Jan and his team hoped to accomplish by calling for a six-month pause, and finally, how it's gone and how he's feeling about it all now.

• 4:41Nathan Labenz?

  If nothing else, I hope this conversation makes it clear that the pausers are not merely Luddites who have never built and don't understand technology. On the contrary, Jan's personal achievements, world-class investment portfolio, and evident optimism for an AI-enabled future, should we manage to build one safely, show that at least some of our most sophisticated and accomplished thinkers take existential risks from AI extremely seriously. With that, I hope you enjoy this conversation with Jan Tollen. Jan Tollen, welcome to the Cognitive Revolution.

• 5:18Speaker 00

  Thanks for having me.

• 5:19Nathan Labenz?

  Really excited to have you. You have been a, I think, quiet but major player in the development of AI over the last 10 or so years now. And I want to give people just a very quick kind of overview of who you are and the role you've played, and then kind of jump to the future, which is the present, and talk about all the things that have happened in the last few months, as well as the call that you recently participated in putting out as part of the Future of Life Institute to call for this six-month pause in the development of large-scale models. So, lot to cover. The world is moving faster

• 6:00Nathan Labenz?

  than ever, it seems. But maybe just give us a little bit of an intro to yourself as an investor in AI companies. You can tell a little bit, if you want, about the story of how you came to be in position to invest in AI companies, but really super interested in how you have managed to become an investor in so many leading companies and the philosophy that supports that.

• 6:21Speaker 00

  So, skip over the period of becoming entrepreneur, running my own games company, then getting into development of peer-to-peer technology that culminated with Skype. And then, at the end of my Skype career, stumbled upon Eliezer Rytkowski's writings and going like, "Holy hell, what, what a thing is, what is the world that I've been born into?" And having a meeting with Eliezer, like, almost exactly 14 years ago. Where I kind of tried to poke at his arguments, didn't find any holes. And then I thought, okay, how can I help? Uh, send them some money, uh, but I think more importantly, started to taking those arguments,

• 7:05Speaker 00

  turning around and, uh, you know, presenting it, those arguments to people who would want to have some brand behind the person who is like making the arguments. Uh, and that's how basically the Cambridge Center for the Study of Extinction Risk got started, where I convinced my co-founder Hugh Price there that these topics are important. Uh, and Max Tegmark, uh, I think he already was kind of like very, uh, primed to these arguments, but that's how the Future of Life got started, Future of Life Institute. Uh, and the other strategy that I deployed was, okay, yeah, I am, I've already was like a bit of an investor and I thought that perhaps I could kind of use my brand, uh, to,

• 7:50Speaker 00

  uh, sort of cut a foot, uh, in the door, uh, in like various companies who are developing potential dangerous things. Uh, so I did invest in a bunch of, uh, AI companies, uh, just to, I mean, I always had this dilemma of, of not wanting to kind of directly accelerate them. So I, I tried to not be like majority investor or anything, uh, but I just enough to have a voice, uh, and then basically it kind of, with DeepMind, I actually had to like walk up to Demis at the conference, uh, and that's, that's how we kind of started talking and eventually became friends. I still catch up with him like, uh, every second

• 8:34Speaker 00

  time I'm in London or so. But like, and then I, once you're, once you're, once I already was investor in DeepMind and eventually a board member, like getting the air of other AI companies became easier. So it's kind of like has been like, uh, uh, just work my way up, so to speak.

• 8:52Nathan Labenz?

  Yeah, I think a lot of VCs would be extremely envious of your deal flow. So I want to get back to that a little bit more in a second, but let's just go to the Eliezer moment for a second. So you said this was 14 years ago, so this takes us back to circa 2009. At the time, the deep learning revolution hasn't even really started yet. It's a, you know, at that point, a kind of highly, well, this may not, you may object to this, but I would say for me, I read it as a highly speculative, yet very compelling thought about what might happen. And, you know, the arguments were, there was a lot of detail to be filled in where it was like, well, we have this, you know, insane amount

• 9:35Nathan Labenz?

  of compute and like, we're probably going to figure out how to use it. And then that probably goes very bad for us. So how did you understand or like, what do you think is kind of the strongest version of that original argument? And then what have been the biggest changes to that worldview in the intervening time?

• 9:52Speaker 00

  Yeah, I mean, there are like many ways to frame things, frame the problem. Sometimes I've been asking people to questions like, A, can you program? And B, do you have children? And then I get like four different kind of framings or approaches. I can explain the situation with AI. Like one sort of simple argument is that, look, there is a reason why chimpanzees are not determining the future and haven't been determining the future for like a long time, if ever. And humans, like humans are, but perhaps not for long because

• 10:30Speaker 00

  we are working furiously to get rid of that advantage that we have over, as like the apex species on this planet. So, so once you kind of realize that AI will

• 10:48Speaker 00

  Likely not stop at human level. There is this unfortunate narrative, especially that's very widespread in Asia, where people, a lot of people think that like we are going to make AI smarter and smarter up to the point where it becomes conscious and then it's just like us. Then it's just like, just like people, other people and we need to integrate them, give them, give it voting rights and whatnot. Whereas I think this is just completely illusionary tale. It will probably not be conscious. It will just be very competent and competence and consciousness. They might be related somehow, but probably not. So we have just

• 11:33Speaker 00

  control over the future yanked from our hands. So that's, I think, is for me compelling enough story.

• 11:42Nathan Labenz?

  Yeah, so the linchpin there is we're the boss of the world because we're the smartest thing around. And if we change that, there's a pretty good chance that we may not be the boss of the world anymore. And not only that, but we really, you know, at this point, as things are starting to come online, we don't have a great understanding of what the new boss would look like or what it might want, or even how to conceptualize, you know, things like want in the context of its internal workings. So any, would you, anything you would object to in my kind of very brief extension? And, and then, you know, how, how has your mindset shifted also from the kind of, you

• 12:23Nathan Labenz?

  know, purely theoretical, largely purely theoretical 2009 L E S R arguments versus today where we're in this world of large language models, obviously, but also, you know, increasingly multimodal large language models and, you know, agent style systems like Agato, you know, that can do all sorts of things. How has the actual development of the technology changed how you think about it?

• 12:47Speaker 00

  Yeah, so many things to say about that. I mean, first of all, yeah, like, I think I just agree the way you phrase things. Like, one, sometimes I've been saying that, like, look, we are, we are seeing the tail end, possibly last years of something like a hundred thousand year period during which humans were the boss on this planet.

• 13:08Speaker 00

  And, and like, it could be even, even more extreme. It's, it's unclear if evolution will continue, if self replicators will continue. Once, once you have AI just completely taking the silver system to like, down to the atom levels and rebuilding it and the rest of the universe. So like, like, it might even be like tail end of like a four billion year period. So how my thinking has changed? Yeah, it's, I mean, there has been like this abstract argument that, look, if we just continue on this trend, we're going to accelerating towards a cliff. And I think the current situation is that we seem to be starting to see the shape of

• 13:52Speaker 00

  the cliff through the fog. It's possible that this, it is still a mirage and like false alarm and like things will kind of level out and we need like some new paradigms. But, but the current situation is like, it's, it, it seems more likely than not that this is it.

• 14:11Speaker 00

  And when it comes to general trend, I think it has been like very unfortunate in AI research with some like silver linings. The unfortunate trend has been like, we have gone from kind of more transparent, more understandable paradigms to less and less understandable paradigms. We went from like, things like expert systems, they're like, by definition, they were super understandable. People were just interviewing experts and trying to kind of hand code the rules by which experts are, are making decisions into machine. And that was like eighties, it was really big, big thing in eighties.

• 14:50Speaker 00

  Then we went to supervised learning where people were just labeling data in different, different domains, trying to Distinguished numbers and this is like where the deep learning kind of started to shine first. And now we are in unsupervised learning. It's just like through, we don't even care much about what, what data we through. It's just like through a lot of data at AI and have it, ask it to just well, figure it out in what kind of universe you are. What kind of heuristics you should apply, what kind of skills you need to, need to learn in order to predict the next token. And that is like, I call it a summon and tame paradigm, which is like you just use these like multi hundred million.

• 15:36Speaker 00

  Large experiments to summon an uncontrollable, controllable mind. And then you look at like what it looks like and try to tame it. So, and this like kind of works if the mind is not very powerful, but it might not be, might not work for very long.

• 15:51Speaker 02

  Omnikie uses generative AI to enable you to launch hundreds of thousands of ad iterations that actually work. Customized across all platforms with a click of a button. I believe in Omnikie so much that I invested in it and I recommend you use it too. Use Cogrev to get a 10% discount.

• 16:09Nathan Labenz?

  It is wild to think about. I have a lot of different follow up questions I want to ask you as well, but, but let's go back and just touch on the investment side for a second. Cause I think this will help people understand the point of view that you have, right? I mean, it, it started with a series of blog posts in 2009, but now you're really quite the AI insider. You did a recent interview where you kind of ran through your investment portfolio in more detail, but I thought it was interesting how you. Split it into kind of two categories, one being like the fundamental AI research type company that you've invested in. I believe there's a half a dozen of those. And then there's kind of the application layer companies and it sounds like there are.

• 16:53Nathan Labenz?

  Dozens, you know, maybe fifty plus of those. It seems like the big research companies would be the ones that would kind of give you more insight into what's going on and what matters most right now, but maybe that's wrong. So could you kind of just give a quick run through of, you know, some of the highlights of the portfolio? And, you know, get a little, we can get a little sense from that of kind of all the different angles that you have on AI today.

• 17:18Speaker 00

  Yeah, I'm actually not best person to talk about my investments. Because like I have mostly delegated the way to a team of a few people. I kind of still make the final decisions. But my focus really is like philanthropy. But yeah, when it comes to investments. Yeah, like the fundamental research, AI research companies, I specifically kind of like invested not to make money, but to have some kind of influence over, over what's happening inside those companies. So they are in some ways kind of adjacent to my philanthropy. And like when it comes to applied AI, I think the prospects of applied AI companies are much worse now than they used to be before this like large LLM paradigm.

• 18:02Speaker 00

  But of course, like the LLM paradigm is like very new. So there was like no way to know that, like in, like, I don't know, five, 10 years ago.

• 18:15Speaker 00

  But currently, I, I mean, I think we have had this discussion with you that, that like the, the big problem with applying, trying to build an applied AI company using the LLM paradigm is that you have to be ready for the rug being pulled out from your like next six months to a year work by the next generation of LLM, which is like a new crop of capabilities that have been bred. So it's in a way.

• 18:46Speaker 00

  Like this, the more kind of domain specific is the AI competence, the more value there is in building kind of application layers around this competence. Whereas if you just get this like general increasingly general generally competent minds, it's much harder to build applications in a stable way.

• 19:09Nathan Labenz?

  Yeah, I, one of the interesting things about doing this show and talking to all the people that we have is, not to spoil one of our closing questions, but we often ask what AI products people are using today that they recommend to the audience. And I have been really amazed by how few different answers we've heard. Probably two thirds of people have said like, well, basically just use, you know, chat GPT. That's it, you know, we get a couple other mentions, but it has led me to believe that the application layer faces some very serious challenges. And it reminds me of, you know, kind of other hyperscaling platforms that we've seen over

• 19:52Nathan Labenz?

  the last, you know, couple decades where you build around the edges of them, but the monopolist power is just so big. I do want to ask a little bit more about competing trends between like centralization and decentralization, because I don't think it's obvious at all that it like plays out as it did for Google and as it did for Facebook this time around. Well, let's just cover, you know, kind of the, the flagship, maybe that's the wrong word, but you know, the fundamental research company investments. DeepMind was the first. I know that you're also an investor in Anthropic and have supported Aut. I don't know if that's an investment or if that's like a just a donation. Conjecture is on that list. Who am I missing on the list?

• 20:35Nathan Labenz?

  And I'm also really interested in terms of the conversations that you've had with founders as somebody, you know, just given your statement, I'm sure you said the same to them, like, I'm not really doing this to make money. I'm doing it because I want to have your ear in case something important comes up. Like, how do people react to that? Do they say like, yeah, that's great. I want you to be in that position to have my ear. Or are people sort of like, I don't know what to make of you. Yeah, maybe you're only investing in aligned people.

• 20:59Speaker 00

  Yeah, I found it like really, in general, my, my kind of. Pitch as an investor to to deep tech companies is that, look, I'm investing my own money. I don't have a boss, right? So and I have like a sizable philanthropic operation. So if I can kind of do good by walking away from profits, I can do that in a way that like VCs are at least.

• 21:23Speaker 00

  For them, it's harder to do that because they manage other people's money and like for them, it's kind of like. Yeah, in some ways like LPs are are their bosses. So like, A, I will be on the side of founders if they think that like this like some. If they feel like uneasy, I'm not going to push you to take like this defense contract or whatnot. And that this this usually goes down pretty well with founders because yeah, it's it's true.

• 21:51Nathan Labenz?

  So who am I missing on the list? We got DeepMind, Anthropic, Conjecture, Oot. Who else would you put in that fundamentals bucket?

• 21:58Speaker 00

  Oot, I'm actually not an investor in. I have like, I have sent some philanthropic money money their way though. So yeah, I mean, Vicarious was like a long time investment around the same time than DeepMind and like a few. Few other kind of AGI groups that are not less, not as well known like curious.ai, for example. Uh, there is this improbable.ai, if I forget, if I if I remember correctly in UK. Yeah, it's just like I have like 180 investments or something like that, so I don't quickly recall all of all the. All the names from that. But yeah, conjecture, I think very highly of conjecture. In fact, I kind of whenever I go to London, I try to hang out.

• 22:42Speaker 00

  Hang out in their office because they are, they seem to be like a company that a group that has like the highest respect for AI. In a sense that like this could be like really dangerous and like the danger is the important part here to focus on rather than like whatever exciting. Commercial contracts we can squeeze out of it or something.

• 23:02Nathan Labenz?

  So let's talk about that kind of emerging paradigm of danger. I mean, this has obviously been all over the discourse lately with. The Paul's letter and Eliezer's time piece and I think broadly speaking.

• 23:16Nathan Labenz?

  The public is extremely confused because on the one hand, we have Ali Azar. And then on the other extreme, we still have people routinely saying like, this is all nothing. You know, this is all just hype and it'll never amount to anything. Which seems crazy to me at this point, like almost like self evidently, this is a big deal. But that is still out there and you know, for folks like my parents who like, you know, don't rush to try chat GPT, they're just kind of hearing all these different messages from, you know, the media and it's, it's all just very confusing. So let's start with maybe the neutral or ideally even positive side. People are, you know, throwing around AGI like all over the place.

• 24:00Nathan Labenz?

  A lot of disagreement or, you know, probably mostly implicit disagreement on what does that even mean? Maybe we could just start with like, what do you think AI is going to do for us in daily life? We'll, we'll then extend to like the dangers that it can pose. But what is your kind of expectation for how AI is going to impact our lives over the next few years?

• 24:23Speaker 00

  I think it's really kind of dependent on like how capable the planet is in kind of constraining the large scale experiments. Because if it turns out that we can't constrain them and, and slow them down, then we're just gonna die. That's my fairly confident prediction. If it can pause, yeah, then like a lot of interesting questions come up because like the GPT level crop of, of AI start will continue improving, even if we don't do like, like sort of new generational experiments, breeding experiments, then even, even those are, are just could be like super disruptive.

• 25:06Speaker 00

  Like, for example, I, I wouldn't, wouldn't want to be an art student in the year of like 2023. Because like.

• 25:16Speaker 00

  It's possible that the skills that you're learning are somehow can be pivoted into something that there will be societal domain, societal demand for. But the answer could also be no, there won't be any demand for your skills.

• 25:29Nathan Labenz?

  I personally see that extending to a great many domains. We just did a little episode on the possibilities for economic transformation. And one of the things I'm trying to kind of help people understand is I feel like right now we are in this kind of perfect little happy zone. You could call it like the Goldilocks time after the, you know, I don't know if you know the Goldilocks story, but this feels like the level of AI power that is just right, perhaps, in that, you know, 90th percentile on the American bar exam, like that's a really strong showing. And, you know, that's base model GPT four capability, right?

• 26:13Nathan Labenz?

  When you imagine what that can start to power when it is fine tuned, when it is integrated with other systems, you know, when it's able to take advantage of its ability, which we've seen demonstrated, to use tools, you know, and that's not yet broadly deployed, but it certainly has been, I think, compellingly demonstrated. Then you add on to that, you know, an even bigger context window that very few have seen. And then on top of that, you've got the multimodal stuff, like these, these models will, the latest models will certainly be able to like browse around the internet and understand websites and, you know, kind of navigate and take, take actions online. It feels like that is enough on the positive side to create

• 26:59Nathan Labenz?

  transformation, really economic transformation is kind of my baseline scenario at this point. And, you know, we're just at the beginning of, of kind of the engineering phase of that, the, the deployment phase, the like social, you know, figuring out of how it's all going to integrate. And it feels like that could be really amazing. And yet, at the same time, it seems like still pretty safe to say that it's limited enough in power that it won't become, you know, an out of control problem at this level. So.

• 27:32Nathan Labenz?

  I think that is, you know, one of the things that frustrates me most is when people who focus on AI risk also dismiss the power. Cause I'm like, you're undermining your own message there. If you, if you dismiss what it can do, then nobody's going to worry about what you are worried about, you know, that it might do. So let's, let's be very clear on like, just how capable the, the systems are. So I like your comment about conjecture having like the highest respect for AI. I think that's something I, I try to cultivate in myself as well. Do you see that any differently from me? Like, does it, does it feel to you like what we have is enough for economic transformation or where do you think we are in that?

• 28:11Speaker 00

  I have like a lot of confusion about like how economy works in the first place, because I know that there are jobs whose main purpose is to make the boss feel more important. I don't think like these jobs are very vulnerable to AI disruption. Because boss would not be, feel, would feel less important if, if that like underling would be replaced by AI.

• 28:36Speaker 00

  But I don't know how, how kind of, how typical that kind of job is in human economy.

• 28:43Speaker 00

  And also like there is like, I think Leizer has pointed out that they just doesn't expect any changes from AI before like we all die. Because like the rules and regulations in economy have like sort of constraint everything to that degree where it just can't have like innovation that

• 29:02Speaker 00

  is going to leave like a significant mark on the GDP or, or like have like a big, big changes in like, I don't know, construction or something like that. Perhaps he's wrong, but, but I just, I have like a significant uncertainty about. I definitely wouldn't be confident that we're gonna get like massive economic disruption from the current crop of AIs. But it's very plausible that, that we would, yes.

• 29:28Nathan Labenz?

  Yeah, I think it, what you said about just how much time there is for the. Transformation to play out definitely makes sense to me. We're on, I've started counting time since the official release of GPT-4. So we're at four weeks and one day into the GPT-4 era as of today. And I do think it's really worth just kind of reminding ourselves and grounding ourselves in the fact that no previous system that the public had any access to could really do the sorts of high-value tasks that GPT-4 can just do. And so we're

• 30:13Nathan Labenz?

  literally, you know, there's been a lot of growing awareness, there's been, you know, interesting use cases, there's been like copyrighting, you know, assistants that have made a lot of money. But there was not an AI on the market until a month ago that had any plausible chance of like giving you quality legal advice and or a quality medical advice. And now, you know, that is there. And, you know, again, we're just so early in in starting to figure out how to use it. So it does seem like that takes a little while, kind of unavoidably. And I just want to remind the listening audience more so than you, like this, that window has just opened. And, you know, we have, we have no idea what's about to start coming through it economically, let alone in terms

• 30:56Nathan Labenz?

  of, you know, alien AI overlords. So turning then to the kind of things that you worry about. I think this model of AI strength kind of proceeding through, you know, what appears to be a smooth loss curve, but what actually seems to be happening under the hood is like all these little thresholds of unlocking different discrete capabilities kind of being passed, you know, one by one and all of that kind of aggregating. Yes, I love that paper. All of that kind of aggregating to a smooth curve, but actually being like all these little discrete bits. I think that's a really helpful frame.

• 31:36Nathan Labenz?

  But I want to ask you, like, what are the things that you kind of most worry about? If you could try to make this somewhat vivid for people, what are the big thresholds that you're like, man, I don't know when, but an AI crosses that threshold and we're in real trouble, like, what are those and how does that play out in your mind?

• 31:54Speaker 00

  So it's possible that there are like many such thresholds that we should be worried about, like one kind of sort of neutral frame to describe what AI is, is that it's an automated decision-making machine that is A, non-human, and B, it is getting increasingly competent by day. So and like, whenever, as every leader knows, whenever you're delegating something, you're also like giving up some control over the outcome. So with that frame, like, there could be like many domains where we, in order to remain in charge of what happens next, we should not delegate it to non-humans. So we should have like, as they call it, human

• 32:38Speaker 00

  in the loop. But like the most obvious one that I can think of, where we are already rushing to delegate things away, is AI development. So like, once you have AI, LLMs that are able to develop, AI is better than any human researcher scan, then basically we have the most capable systems on this planet appearing without any human help, and possibly without any human consultation. And then basically, good luck, humanity.

• 33:13Nathan Labenz?

  Yeah, that's interesting. I thought you were maybe going to say the deception threshold, which is one I hear kind of thrown around most. I mean, it's funny, it's striking for one thing that like, that's kind of OpenAI's explicit plan, you know, their, their fairly high level, I would say, plan for AI alignment involves ultimately having AIs kind of supervise themselves and, you know, refine the dataset and hopefully bootstrap into something good.

• 33:44Nathan Labenz?

  That has never really reassured me that much either. Anthropic also doing something.

• 33:50Speaker 00

  Yeah, constitutional AI, that is like specificity. Although this is like kind of AIs constraining AIs rather than AIs developing next generations of AIs. I think this like it's important to kind of distinguish between two, those two frames.

• 34:05Nathan Labenz?

  One big threshold is AI designing, training the next generation of AIs. Pretty hopefully intuitive to see for people how that becomes potentially a runaway problem that we don't have great control over the deception threshold, you know, kind of outer inter alignment mismatch seems like one that a lot of people worry about just as much. Any personal thoughts on that one that you want to share?

• 34:33Speaker 00

  Yeah, I think like one thing that I and like alignment community has sort of like learned over the last decade is that the shape of the of the alignment problem has become like much clearer, like for example, indeed, this like inner outer alignment.

• 34:51Speaker 00

  Dichotomy is something that at least myself had had no idea about. Just like this idea that

• 35:00Speaker 00

  deep learning paradigm and what's it like machine learning paradigm in general is training AIs by picking essentially random minds out from behavioral classes. So like you're not selecting AIs based on what what they want. You're selecting AIs based on how they behave. And and like there could be like many, many motivational structures behind behind giving certain particular behavior. The most scary one is basically AI realizing that it is being trained and then just like acting out

• 35:40Speaker 00

  the the goal that you're training, training it for in order to kind of be selected and and eventually escape the box.

• 35:50Nathan Labenz?

  Yeah, I think that one is hard to get around too, just from the simple observation that. We're not super reliable. You know, anybody who spent a significant amount of time trying to validate language model output, even just for like a, you know, relatively run of the mill application, like I've done this at Waymark, right? We're making marketing video content for small businesses. And, you know, really all the stuff we create is for with language models. The main thing is write a script for a like short commercial for a small business. So it's a pretty narrow domain of of space that we need to evaluate. And yet it remains a real challenge to figure out like, is this model better than this one?

• 36:34Nathan Labenz?

  Or, you know, we do a fine tune. How does it compare to the last fine tune? You know, you're getting all these like different outputs and it's just tough. Like there's the over the distributions are overlapping. You know, the, the, the rate at which the new model is preferred to the previous one is often fairly low. I've seen published results as low as like 11 to nine ratio, you know, where one is preferred to the other. Even GPT four to 3.5 is just 70 30 in terms of preference. So like still a full third of the time. People prefer 3.5 in a head to head comparison, which kind of blows my mind, given how like qualitatively better it seems. GPT four is. So that's just like the general problem of of validation.

• 37:20Nathan Labenz?

  But then you add into that mix that like, we have all these, you know, heuristics and biases that are exploitable. We have these kind of, you know, cognitive gaps that have kind of lingered in our, our own systems. And, you know, evolution never had a, had a real reason to eliminate all of them or, you know, never hasn't got around to it yet. And so we're exploitable, right? And we're, everybody kind of knows that in our daily life, like we know that people at a minimum will tell us like little white lies to make us feel good or just to get through a situation a little bit easier. Do you see any promising route to avoiding that sort of?

• 38:00Nathan Labenz?

  Exploitable evaluator problem?

• 38:04Speaker 00

  I mean, short answer is no.

• 38:07Speaker 00

  But it's very much like open research question. So on a theoretical level, indeed, you would want to somehow kind of hitch a ride on the increasing capabilities of AI. When it comes to

• 38:22Speaker 00

  when it comes to somehow making it more reliable or more constrained.

• 38:30Speaker 00

  More predictable in general. I hesitate to say more aligned because like my model of Eliezer goes like, no, no, no, you don't. You don't point point AI towards alignment. That's just like a silly thing to do. So, but yeah, like AIs are going to get more capable. Can we somehow get something out of it that is scalable? Rather than kind of ending in in a predictably bad place.

• 38:59Nathan Labenz?

  It might be worth just spending a little bit more time to on. Again, just kind of how these things might play out. I think Eliezer has spoken very interestingly, componentally about what happens when you go outside of your distribution of training. And for humans, he just points out that, you know, basically everything in nature is optimized for reproduction. And, you know, inclusive genetic fitness. And yet, the behavior that we observe in ourselves is not at all about in the modern environment, does not appear to be about maximizing our reproduction.

• 39:37Nathan Labenz?

  And in fact, we didn't even know that that's what we had been optimized for until relatively recently. So we're out here kind of doing whatever we're doing. It took like a few random geniuses to figure out how we had actually kind of been created by nature. And that has had relatively little impact on what anyone has actually done in their day to day lives. So would you, would you add anything to that? That story or observation?

• 40:06Speaker 00

  Yeah, I mean, just to be more precise, I think we are selected for

• 40:13Speaker 00

  for kind of inclusive genetic fitness ability to reproduce. And again, because of the same.

• 40:23Speaker 00

  Problem that machine learning faces that that we can only select based on behavior or based on results. That selection kind of effectively pulls in like a random random instantiation of capabilities and and motivations that just happen to give you this particular behavior without having like any fine grained control over over what these motivations and capabilities actually are. So, yes, like we ended up evolution ended up pulling us selecting us in this like ancestral environment where we had developed this bunch of heuristics that were very useful for for reproduction

• 41:08Speaker 00

  in that ancestral environment, but much less so in in the modern environment without like never actually actually kind of ingraining in us any fundamental understanding what we're being selected for. So I the very same process might just replay it when it comes very same process might get replayed as we are selecting AIs based on behavior and without without any insight on the inner workings of them.

• 41:41Nathan Labenz?

  So we could spend hours unpacking all this. I know you have done that. You know, many times. So we will bracket that for the moment. We've got all these different failure modes. We've got, you know, potentially kind of runaway AI training its own successors in a way that is not clear to us. We've got kind of the deception problem. We've got the fact that we have no reassurance or no reason to believe really at all that the goals that we have for AI will be like represented internally. And so, you know, with A sudden jump in kind of the domain in which the AI can operate, you know, it can be totally

• 42:23Nathan Labenz?

  outside of of training, distribution and, you know, who knows how it might act, you know, just like who would have expected how humans might have acted from, you know, the ancestral environment. So all these things are, you know, pretty big conceptual problems. We don't have good answers to them at the moment. What do you think that kind of, well, how does that boil down to a, a simple worldview for you? Like, what are the odds that you see right now of serious catastrophe happening in, say, the next two, five, 10 years? And maybe we could segment that into, given the trajectory that we're on, versus like, what, how we might be

• 43:06Nathan Labenz?

  able to shift that, if, for example, we've, you know, took a pause?

• 43:11Speaker 00

  Yeah, my current estimate for, for kind of life ending disaster is basically one to 50% per generation of, like 10 X-ing of compute that's being thrown at these experiments. So, and like, currently there are 10 X-ing things in like six to 18 months window. So you can kind of calculate from there. I mean, at some point, we're going to run out of compute, because like, there's only so many, so many 10 X-ing you can do. So you can't like, you probably can't do like thousands of, thousands of those. But still, like, let's say something like a

• 43:53Speaker 00

  geometric mean of one and 50% is, is 7%. So with 7% risk to everything. Like, if you, if you continue doing those, we probably can still do like something like five or six of those. And at that point, we are like more likely dead than not.

• 44:10Nathan Labenz?

  Worth taking a second to just let that sink in. Would you have put that same estimate on GPT-4?

• 44:19Nathan Labenz?

  Like, do you think we just survived a 7% X risk event with the training of GPT-4?

• 44:25Speaker 00

  That is a great question. Like, with hindsight, it's like, I'm like, super anchored now, right? So I really want to say no. But again, like, the range is like, 7% is like this point estimate, but but like, really, my uncertain range is like from 1 to 50%. And like, so interesting question is like, would I put like, less than 1% confidence in GPT not destroying everything, GPT-4 not destroying everything? And probably not. So I think, yeah, it's, I think it's unreasonable to have, at least, the thing, given the things that I knew with GPT-3 and like everything else, and like things that I didn't know,

• 45:11Speaker 00

  like having a very close look at what's happening at GPT-4, in GPT-4 training, then yeah, I think it's, it would have been unreasonable for me to be less than 1% or like confident in less than 1% doom from GPT-4.

• 45:28Nathan Labenz?

  Honestly, I can't really argue with you there. When I look at the, you know, when I got my first look at it, it had already finished pre-training and initial reinforcement learning. I mean, this was, you know, the six months ago, when they kind of finished the first version before any of the safety work. And obviously, there was the, you know, the whole red teaming effort and and everything else. It definitely hit me pretty hard that like, wow, this is a significant leap. And now you look at all the papers that have kind of come out characterizing it

• 45:59Nathan Labenz?

  in the wake of the official release.

• 46:03Nathan Labenz?

  The thing that I kind of keep coming back to is, we have these smooth curves, but then you have on individual behaviors, you have these sudden jumps. So the one they published in the, in the technical report, which isn't such a big deal, obviously. But maybe, you know, indicative of things that could happen in the future on on more problematic dimensions is the hindsight bias failure, where it had previously been observed, I think by Anthropic, that bigger models suffered more from hindsight bias. And so it was an example of, you know, an inverse scaling law, where the behavior is getting worse with bigger models. And then all of a sudden with GPT-4, that problem is totally fixed and there is no hindsight bias.

• 46:44Nathan Labenz?

  And it basically just scores like 100%, you know, perfection on those hindsight bias problems, which by the way are basically like scenarios where you had a good bet available to you. You like took the good bet and you lost, you know, in an unlikely way. And so the question then is like, you know, should you have taken the bet? And you know, people might say in the hindsight bias would be like, well, no, if I lost, then I shouldn't have done it. When in reality, like you actually had, you know, all good reasons to do it. So 3.5, you know, was actually getting this wrong more often than like three and more often than than some smaller models. But then again, boom, some somehow some unlock has happened in the course of training.

• 47:27Nathan Labenz?

  And it was probably never registered on the smooth loss curve, uh, which, you know, mostly looks smooth. But all of a sudden this behavior now is like totally strong. I would say, you know, probably safe, safe to say superhuman in the sense that obviously we create these measures because some of us struggle with the hindsight bias. And so, yeah, you wonder, okay, with that's those kinds of things, we do see those kind of sudden jumps in capability. In the context of GPT-4, like, you know, another 10, 50, 100x compute scale up. You know, it's predictable that it will bring more of them, but it's very unclear like what exactly they would be.

• 48:12Nathan Labenz?

  So 1 to 50% across these, you know, big scale up training runs. How do you think that plays out across different groups that might be running those processes? I don't know, you may or may not want to go, you know, to kind of specific names, but like, obviously there's a few leading groups that can plausibly scale up another, you know, one or two orders of magnitude right now.

• 48:41Nathan Labenz?

  Do you think that's equally reckless for any of them to do? Or do you think some have a better handle on how to do that responsibly than others?

• 48:48Speaker 00

  I mean, there are always some differences in various dimensions.

• 48:53Speaker 00

  So yeah, I mean, just like hanging out in Anthropic feels materially different than hanging out in DeepMind, where I've both hanged out at both places and like a little bit at OpenAI. So like there's certainly like a much more safety culture in Anthropic. Does that justify risking everything, like killing everyone? It's like, I don't think so. So it's like, in some ways, these are like second order. In my view, they're like second order effects. So how, how kind of like safety conscious your group is compared to the fact that you're taking just massive risks with everyone's lives right now?

• 49:35Nathan Labenz?

  So how do you think about, you mentioned two companies that I have serious questions about. I guess let's go DeepMind first.

• 49:46Nathan Labenz?

  I've been kind of waiting for like a Gato 2 to drop. And it seems like, you know, as I check my imaginary watch, like it seems like that is probably due right around now, unless there's some sort of pause or like somebody's kind of thought better of doing a Gato 2. Or, you know, maybe it just didn't work for some reason, but that seems unlikely because it seems like almost everything is, you know, quote unquote, working these days.

• 50:14Nathan Labenz?

  Do you have a sense for like what is going on at DeepMind? You know, Demis published a Time article, it feels like a long time ago, much more, you know, reserved and moderated tone for a Time article than, than Elias's more recent one. But still pretty striking to see founder, CEO of DeepMind saying, you know, we need to think about slowing down. Are they slowing down?

• 50:37Speaker 00

  I mean, I don't know. I don't have like that much visibility into DeepMind. I have heard about them deliberately being more cautious about publishing things, which is like empirical, I think that I haven't verified, is that actually true? But it feels that they are more careful now when it comes to publishing. In some ways, we are kind of in a lucky world that like, I mean, all the big free labs, they are safety conscious to the level, at least to the level of not dismissing the risks in a way that, for example, Yann Le Cun or Andrew Ng are just completely dismissing

• 51:19Speaker 00

  the risks. It's not obvious that we, that the world should be in a way, in a way that, like, for example, I mean, I've been praising Sam Altman for, for saying, saying the things that he says about, about the risks. And he's been very explicit about the massive dangers that humanity is facing from AI. So, so like, another question is like, like, to what degree does this safety consciousness actually kind of constrain the actions of these companies that have their own incentives as like non-human optimization engines?

• 52:00Speaker 00

  And, and like necessarily some hard to, hard to lead, like, I mean, the leaders of AI companies, they, they have like a bunch of conflicting requirements that they want to satisfy, especially in like in DeepMind, where it's like one big constraint is that they're not a company, they're like, like a subsidiary of, of, of Google. So it's, in, like, I, in some ways, I'm kind of sympathetic to them trying to, trying to navigate like that must be complex set of constraints. And

• 52:37Nathan Labenz?

  I think you're right. Yeah, I found myself saying this too. Like, it is easy to imagine people that are a lot more cavalier, you know, running the frontier projects. So I'm, I'm thankful that there does seem to be a profound, like, awareness and, you know, real seriousness of approach across the, the biggest companies.

• 53:00Nathan Labenz?

  In some ways, I also feel like we might be in a lucky scenario in that language models are taking off and yet they're very like kind of soft edged AI. They kind of also run slow. And I contrast that to what, you know, I think maybe Eliezer sort of had in mind 14 years ago, or what DeepMind was, you know, seemingly like closest to, if I had to say, you know, five years ago, who was closest to AGI, I would have said DeepMind with all of their like, you know, game playing, you know, agent, learning agents, all that kind of stuff. Those notably like achieve, you know, like AlphaZero and all that, right?

• 53:45Nathan Labenz?

  They notably achieve like dramatically superhuman performance in obviously like narrow domains. They also run really fast. And they're trained, if anything, in like an even more alien way where AlphaZero just plays itself, right? In in all these games and kind of learns from that and doesn't even need to see the database of human games and therefore when it shows up with superhuman skill, it's also like kind of an alien superhuman skill. And you get these like dramatically surprising moves that, you know, no human would ever have made. In contrast, I feel like language models, you know, everything has pros and cons, right? They certainly have insane surface area. But their kind of

• 54:30Nathan Labenz?

  softness and slowness does seem like it might be a real advantage relative to a more kind of hardened, you know, faster agent type of model. How do you think about that? Do you think we are lucky with LLMs or am I just naive in my optimism?

• 54:46Speaker 00

  Yeah, I think like the big trend has been negative in terms of like going towards more and more black boxy and kind of uncontrollable training regimes. Like going from like expert systems to supervised to unsupervised learning. On the other hand, yeah, there are a few things definitely that we can sort of got got lucky with. Like one of like, I would say the prime one is is the fact that you actually do need a lot of compute to do the pre-training of large language models, which means that there are only like number, small number of organizations on the planet who can do that. And like those training runs are potentially very

• 55:28Speaker 00

  conspicuous, so like only half jokingly say that like the planet is now breeding alien minds in a way that aliens can see because like very plausibly you can see those energy expenditures from space. So like that's like one lucky thing about L L M's. And the other thing, yeah, I agree that the the speed at which or the slowness rather, at which they can have like process things is is an advantage. But this is a temporary advantage, I'm pretty sure, because like human minds, human brains themselves are like offer like a concept, proof of concept that like not have to be that slow. This is like just like pure inefficiency. And the other thing is, yeah, like, once you

• 56:13Speaker 00

  have like some kind of feedback process where the like L L M's will start developing AIs, those AIs might not might no longer be L L M's.

• 56:22Nathan Labenz?

  What do you make of this current moment? I mean, this is something that's really just popped up and gone kind of widespread in just the last two weeks. But there's all these kind of projects to create, you know, some, one of them is called Baby A G I, another is called Auto G P T. And essentially, they're taking a language model, putting it in a loop, and kind of giving it, you know, the ability to like, have a goal, you know, delegate to itself, go through these like thinking, reasoning, planning steps, you know, then start to use tools. And again, like, you know, getting around the the kind of hard limit of a context window through some sort of self delegation. I'm struck by that as

• 57:05Nathan Labenz?

  kind of, you know, potentially the next convergence between those two paradigms in some ways. And it also seems to open up the potential for a kind of auto, you know, self play reinforcement learning, like these agents are not very good right now. And so if you go on Twitter, you'll see people being like, this is so amazing. Look what this thing can do. And then you'll see other people being like, it fails way too much. These are not useful. It's going to be a long time before they are useful. But I kind of think those people are wrong in the sense that This is the first language model paradigm that feels like relatively easy to evaluate in a fairly open domain because you can kind of know, like, did the thing, you know,

• 57:47Nathan Labenz?

  book you the flight or whatever, right? Or did it just get hung up on some API error that it never solved? And it seems like they're going to learn pretty quick from this like massive little agentization and, you know, kind of exploring paradigm that's just been set up. Like, how worrying of a development is that for you?

• 58:10Speaker 00

  There are like several frames to look at this thing and, and these frames will kind of like give you some very kind of opposing judgment about the situation. For example, one very positive frame to look at this is that, that like, it's great that society is like kind of like poking, rattling and poking these current models to see like, what are the extremes that you can push them to? Because like, they are not very competent. And like, by having those kind of experiments with like Chaos GPT and whatnot, we will, we as a civilization will actually learn like how bad things could be if, if things

• 58:52Speaker 00

  would be scaled up. So if you take like Chaos GPT and put like GPT-6, I claim you might not be safe at all anymore. On the other hand, like you can take this frame that, that like when, I don't know, people like Yann LeCun, et cetera, have been saying like, there's nothing to worry about from AI because like, it's not going to be agent. And even if it's going to be agent, like, it would be just stupid to kind of install some like self-preservation and, and, and just kind of like bad goals. Like, we wouldn't, we're not going to do that. It's like, no, we absolutely are. Like, a fraction of humanity has a death wish. So it's, it's like, it's, it's, that is like enough.

• 59:32Speaker 00

  Clear empirical demonstrations that like if something really bad can be done with AI, it will be done.

• 59:38Nathan Labenz?

  Yeah, it's a big world and it's pretty easy. I mean, that's the other thing that's amazing. Like, I think the first commit of the baby AGI project, which I believe has been the number one trending project on GitHub over the last couple of weeks, alongside like a couple other very similar projects. The first commit, I think, was 105 lines of code and that's all it takes. You know, a couple clever prompts and a, you know, and kind of a loop and you've got yourself a little agent, you know, and it might not do much yet. But the bear, given the model, the barrier to, you know, creating some sort of semi embodied, you know, autonomous

• 1:00:18Nathan Labenz?

  version of that is proving to be extremely low. So yeah, I don't think we're gonna, we will not be able to rely on the good discretion of users in the long term. Certainly, probably not more, more than for a few days with the release of any major new system. You mentioned a minute ago, you said three kind of leading groups. And I wanted to ask you how you think about like who is at the frontier and who is maybe going to be at the frontier over the next few years. I assume the three you had in mind, you didn't specifically say OpenAI, but obviously they're in that group. DeepMind was the other that we were discussing. And then I am guessing you're thinking Anthropic would be the third.

• 1:01:00Speaker 00

  I think DeepMind and Google, they're kind of like, could be kind of interchangeable. I kind of hear that they are kind of, email publicly mentioned that they're somehow kind of joining forces when it comes to this LLM race.

• 1:01:13Nathan Labenz?

  So nobody else, you feel like is close enough at this point? Like, if it's a coordination problem of who actually, you know, who are you calling on to pause? I mean, you're calling on everyone to pause, but it sounds like it's really those three organizations that you're calling on for a pause.

• 1:01:28Speaker 00

  Yeah, I think they are like the first, so-called like first tier when it comes to doing like the most dangerous experiments. But like, of course, then you have like the second tier. I think Eliezer has this kind of related law, like the Moore's law of mad science, that I kind of forget exactly how it was framed. It was like every, every two years, the IQ needed to destroy the world drops by like one or two points. So it's

• 1:02:00Speaker 00

  Hardware companies, mostly Nvidia at this point, will throw in more and more capable and cheap computing cycles at the market. The world destroying capability will be in a larger number of hands.

• 1:02:17Nathan Labenz?

  Interesting to think about how that evolves over the next few years. Do you think, if you imagine that, let's say we do enact a pause and then, you know, meanwhile Nvidia keeps shipping and people keep kind of doing, you know, fundamental, which notably the letter explicitly goes out of its way to say, like, we're not saying all AI research should stop or that, you know, you can't build your small models or fine tune things for your use cases and so on. So if we imagine a world where there is kind of a pause on these high end experiments, but hardware continues to ship and generally speaking, like, the field is not shut down.

• 1:02:58Nathan Labenz?

  Do you have a guess for how many folks would be in the kind of, you know, would be able to do a GPT five type project if they chose to in, say, five years?

• 1:03:11Speaker 00

  Five years is a super long time. Yeah.

• 1:03:15Nathan Labenz?

  I'm with you on that, by the way. I don't even try to guess things five years out, so I shouldn't ask.

• 1:03:18Speaker 00

  Yeah. Two years, I mean, yeah, probably a dozen. It's something that can, like, just pull out of thin air. If I would think about it, then I would probably have like a, I mean, it's probably less than a hundred, more than ten, perhaps closer to ten than than than a hundred is my answer in two years.

• 1:03:36Nathan Labenz?

  And so some of those we can kind of fill in pretty obviously, right? Like Meta, seems like it would be a very natural candidate. Microsoft, I mean, they have the OpenAI partnership, but they certainly also have their own big research division. Presumably, Apple, you know, has the resources to get into that game.

• 1:03:57Nathan Labenz?

  Maybe even like a Tesla. I mean, they're focused on other things at the moment seemingly, but they also have the bot. The Tesla bot is going to need some sort of, you know, fairly general intelligence to to help it walk around and talk to people and pick stuff up. Any other kind of specific actors that you think would be likely entrance there? And then how do you think about kind of the international scene? Like, is there anything coming out of Europe? And obviously, then everybody starts to think about China, too.

• 1:04:24Speaker 00

  Yeah, I was going to say that, like, the obvious, like, in two-year perspective, yeah, you should kind of like start also like looking at like non-US actors in Europe, where it's kind of, yeah, compute is much more available. And then also China. I think there are like, one sort of obvious counterargument that we're getting with when it comes to like calling for the pause is that like, what about China? And like, my answer there is that, like, currently, China does not seem to be in the race, at least not as intensely

• 1:05:08Speaker 00

  as the leading US labs are in a race between themselves. And second, like, sort of, almost like culturally, Chinese seem to be like much less keen on pulling a Bing and just like unleashing uncontrollable mind on their territory. I mean, only half joking, you say that, like, in China, if you do that, as a tech CEO, that might get to disappeared. But yeah, in the in the longer run of like two or more years, it is big, will become like more and more important to get some kind of international agreements that we already have in nuclear, for example, in place for also for compute.

• 1:05:52Nathan Labenz?

  On the China front, I totally agree. It's like, I don't know why we would assume, given the posture that the Chinese government has taken toward technology over the last few years, that this is the technology that they're just gonna throw a caution to the wind on, right? I mean, they've shut down, functionally shut down their entire video game industry, and limited it to, as I understand it, just like a few hours on like a couple weekend nights per week is like all that like video game companies I think can even operate in China now. We should fact check me on that, but that's what I understand to be the case.

• 1:06:26Speaker 00

  And online, online learning, I understand is also being mostly.

• 1:06:31Nathan Labenz?

  And that one is fascinating too, because like who could object to online learning? But my sense is that they intervene there on like a state level because they feel like there is an unhealthy market dynamic developing where people are like working too hard on these, you know, on these like whatever standardized test measures and like putting way too much resources in this and it's like gone past the point of like societal benefit.

• 1:06:57Speaker 00

  Yeah, I mean, I'm actually a chairman of a language, online language teaching company, an Estonian company that has like big linguists. Yes. Oh yeah, you know it, of course. And, and like one thing that they learned in Japanese market is that there's like a massive English teaching market in Japan, but people aren't caring, don't care about learning the language. They just want to pass the tests in order to get like better employment options. So like, like you're not, like this is like as a language teaching company in Japan, your job is not to teach the language. Your job is to get people pass a good, get good test scores, which is very different job. So I say suspect that something like that was was happening in China.

• 1:07:41Nathan Labenz?

  I would not take personally, I'm not ready to say by any means that I want to, you know, fully subscribe to Xi Jinping thought or live under the technology regime there. But it does seem like we're jumping to a conclusion. Way too quickly when we say like, well, if we don't do it, they will. And you know, that'll be worse. But leaving that aside for a second, just coming back to ourselves. The other big thing that has come out this last week or so is apparent, and I take that it's probably legitimate. A sort of leaked fundraising document from Anthropic that says that they are planning to raise five or so billion dollars and

• 1:08:27Nathan Labenz?

  kind of see the next two, three years as like super critical, planning to do like next gen models. You know, they're kind of immediately moving into a GPT five, you know, type scaling regime. It sounds like the model itself, you know, supposedly is going to cost like a billion dollars to train. And then they say, again, this is all according to reporting. I haven't seen the deck. But those that fall behind in the 2526 timeframe may never be able to catch up. So when I heard that, I was like, man, that does not sound like a company that's about to pause. It doesn't sound like, I mean, it does sound like a company that's kind of in the race now.

• 1:09:12Nathan Labenz?

  How are you viewing that news? And I don't know if you have any, you know, inside view. I don't obviously wouldn't ask you to, you know, to share anything that you shouldn't. But what should the public make of that, that, you know, update from supposedly like the most safety centric leading lab that there is?

• 1:09:33Speaker 00

  Yeah, I mean, like to that, to the degree that this thing was accurate, which again, I can't comment on. Because I'm an investor in Anthropic and board observer as well. I think to the degree that it was accurate, it is. Strong evidence that there is like a massive race happening between the US companies. That is going to get us killed. So like, can we stop that race, please?

• 1:09:58Nathan Labenz?

  What exactly is the thought process, knowing that this group came from OpenAI? They, you know, high level description that I've heard is, you know, they felt like it was becoming too commercial. They wanted to be more focused on a safety first type of approach. And that's been, you know, two years or whatever. Now this, like, what? The only thing I can come up with is, people must be thinking, we'll do a better job than they will do. So therefore, we should do it before they do it. Because if we don't, then they'll do it, or they'll do it worse. And it seems like maybe everybody's thinking that. I kind of model like OpenAI to some degree that way as well. Is that how you think about the decision makers? Or what do you think they're thinking?

• 1:10:44Speaker 00

  So I think like one very informative, uh, public, uh, piece of information is like Future Flight Institute podcast with Dario and Daniela Amade. Uh, that was like during COVID, about a year ago, um, like, yeah, in, uh, early, early 22. Well, basically, kind of like, uh, they explain like what the approach of Anthropic is. And as far as I know, it is, uh, kind of, uh, it is true. Uh, what, what they said in this podcast is basically like train the frontier models. Uh, and then basically, uh, do alignment in an empirical fashion while having access to frontier models. And the claim is that it is exactly because of these emergent capabilities.

• 1:11:28Speaker 00

  There is only so much, uh, you can do, uh, using, uh, not state of the art models. Uh, because, uh, in some ways, as the models get more competent, uh, they also become easier to interact with. Uh, and like, I mean, just the fact that we have language models in the first place, like that is, in some ways, I think you pointed it out, this is, uh... Can also like serve as a good interface when it comes to alignment. So like, yeah, that is kind of Anthropic's

• 1:12:03Speaker 00

  thesis to be to have like this latest models. And then basically use those to do like state of the art alignment. That is kind of like empirically tied to the actual objects that we have. Now, of course, the big question there is like, how many generations you can do that for? Like, like, because like the pre-training is like largely uncontrolled, unsupervised process. Like, how many generations we can do the pre-training safely? Not to mention things like leaks to elsewhere of the resulting weights. So like, I think it's a genuine dilemma.

• 1:12:45Speaker 00

  It's, it's, in some ways, I think Anthropic's framing and perspective makes a lot of sense. Because indeed, like the largest, like the

• 1:12:56Speaker 00

  you can do more, get more useful alignment work done with the latest

• 1:13:02Speaker 00

  crop of the, of the large language models. But on the other hand, like each training imposes like some risk. Again, in my estimate, one to 50% risk of complete unification of the planet.

• 1:13:14Speaker 00

  How do you navigate that trade off? It's, it's not obvious, and I don't think there's current enough kind of thinking into, into, into that trade off.

• 1:13:23Nathan Labenz?

  Again, it feels like there's some sort of, you know, game theory element here where it's like, it seems like they're doing it because they believe someone else is doing it still on some level. Right? If they, if they roughly share your view and they're like, well, the only way we can do the alignment work is if we have access to the latest models. Then a good counter argument would be if it were true, well, nobody else is going to create these if you don't. So maybe you should just sit tight too. And then we can all kind of, you know, study what we have. We don't need another frontier just yet. So it still, it still seems like it is fair to say there that like.

• 1:13:58Nathan Labenz?

  On some level, they feel like their hand is forced, like they can't not do it because they're either in the game or they're out of the game, but the game will continue regardless. It seems to be the the model that's kind of implicit in that decision making.

• 1:14:09Speaker 00

  Yeah, I mean, there are like a few models that are sort of consistent with evidence, but that definitely is like one model because of like these race pressures. They're going to feel that like if they don't have access to the latest generation of models, they like can their prospects of actually doing alignment are significantly hampered.

• 1:14:31Speaker 00

  So that's like the positive way of of framing things. The letter even like I think explicitly mentioned that one goal of this letter is to like call for this timeout in this race that indeed has like non-chaos interatic element.

• 1:14:47Nathan Labenz?

  I think your most recent kind of fundamental AI research investment is in conjecture. And if I understand correctly, and I may be wrong on this, but I don't get the sense that they are planning to like try to train a GPT five in the short term. How are you thinking about kind of their contribution, their strategy? It seems like they take a different approach where they they don't feel like they have to create the frontier models in order to do something useful.

• 1:15:11Speaker 00

  Yeah, I am much more positive about conjecture's approach in terms of like safety capabilities trade off. They still train language models, but they do not do like enough latest language models. Their goal is to kind of like compose less capable language models in a way that kind of makes a more makes for a more predictable structure. So you're like, you don't risk the world during the pre-training phase and have like a more in some ways kind of more old school approach to AI rather than this someone and tame approach.

• 1:15:49Nathan Labenz?

  Yeah, interesting. We just did an episode with Andreas and Jung Hwan from OTT and they. Have a very similar, you know, kind of outlook to

• 1:16:00Nathan Labenz?

  composition of models, you know, the sort of traceability of all the logic, you know, atomization of the different decisions and operations to try to create, you know, some sense of kind of designed control, you know, into the system from the beginning. It sounds like conjecture is kind of on a similar line of thinking.

• 1:16:21Speaker 00

  Yep. And I think I did manage to get the pause in this game theoretic race.

• 1:16:28Speaker 00

  Assuming it is a game theoretic race, it does also like some frame that says that no, this is like just apocalyptic death cults trying to end the world. This is like the least charitable frame. But if it could get a pause, then like, I do think that like there's like this almost automatic pressure to get like more competence out of the minds that we already have trained. And part of it is just like to have like a better understanding, a better composition of the capabilities that we already have. Because like one important bit, like as a lot of your listeners probably know, is that you have like this training phase

• 1:17:12Speaker 00

  that is like much, much more expensive than actually the inference phase. So like, once you have like spent a lot of compute on training, once you finish the training, you have a lot of ability to run many, many instances of the of the minds that are just just trained.

• 1:17:28Nathan Labenz?

  So let's talk about the latter. So you guys, obviously, you know, we've covered a decade of your thinking and investment on this subject. And now we get to the point where, okay, GPT four is released. It is, you know, closing in on human expert performance in a great many domains. It does seem to me like it's quite. Unclear like what the next generation of that would bring. Obviously, you guys are thinking, you know, something very similar there. So how did this project come about for a letter? Like, how did you guys settle on a six-month pause? How, what was the process like of like trying to bring a broad coalition together?

• 1:18:12Nathan Labenz?

  And was this something that you guys like actually thought might happen? Or, you know, is it, is it kind of a intended to be a conversation starter? Like, how do you think about this project?

• 1:18:21Speaker 00

  Good questions. I remember we had a FLI catch-up call on, uh, on March 21st. So, I mean, less than a month ago. And we were kind of like, we had already observed that there are like significant voices, like in, in the public. I mean, the Gesler Klein's article in New York Times had come, come out where he kind of explicitly compared the current AI race to like summoning, uh, summoning minds.

• 1:19:04Speaker 00

  And I think Harari's article also was published around that time, uh, where he kind of was concerned about LLMs plugging into, into the operating system of civilization, which is like about language and which operates on the language level. Uh, so, and then like numerous discussions, uh, sort of private discussions that are very concerned about, uh, about the current race, including with people in the labs themselves. So we thought that, okay, perhaps like one valuable thing that IFLI could do, uh, is to try to create some kind of common knowledge that, yes, like a bunch of people are worried and now basically.

• 1:19:49Speaker 00

  Create a situation where like those people know that other people are also worried and and the other people know that that the other people know etc. And we thought, okay, we have some experience with with um open letters. So perhaps we should like try to draft draft one up. Uh and of course, like our previous open letters had something like thousand or less than thousand signatures. So like one thing that we got just completely blown away by uh was was just a reaction. Uh like gonna be muted in the first few days, we got like tens of thousands of signatures. Uh and we had like technical problems because of that. Uh so yeah, like the when it came to kind of drafting the the letter, uh there were like multiple considerations.

• 1:20:34Speaker 00

  Like one consideration was just speed. So like clearly, if we would have had like several weeks to work on it, uh then uh then it would have been much better. Uh but like it was like sort of done much in the spirit of like not let's let's not have the perfect to be the enemy of the good and and let's draft up something that that feels uh okay to put out. Uh indeed, like the six month uh number was like one thing that kind of uh was put in and then put taken back out of uh from different versions of the draft. Uh and uh

• 1:21:16Speaker 00

  argument finally that kind of won over the six month thing was that like one question I we get like so like why six months? Why what can you do with a six month? What can you demonstrate with a six? What would six month pause buy us? Like one one important thing that people don't necessarily realize that six month pause would buy us is is confidence that we can pause. And so, and like in that, that sense, it's, uh, it's better to have a proposal that calls for six month pause to fail than a, than a, than a proposal that calls for indefinite pause to fail. Because like the indefinite pause situation, people go, oh yeah, if it would have been six months, of course, we could have, could have done it. But like,

• 1:22:01Speaker 00

  because indefinite, like, nobody would pause indefinitely, right? So that was like the final, final kind of like reason that, okay, let's, let's put in six months and see what happens.

• 1:22:09Nathan Labenz?

  Yeah, I thought, uh, Zvi had some great analysis of this in the first, uh, somebody who also works with a speed premium that I appreciate, um, had some, just, I thought, pretty ultimately simple, but, but still very wise analysis. That's just like, if you want, if you feel like you're going to need coordination in the future, it makes sense to start building it now. And if you need, if you, you know, can only get a little bit going at first, then you kind of take what you can get and you hope to build on that foundation.

• 1:22:39Speaker 00

  Yeah, the other kind of like really big, uh, consideration that can fade into the open letter is that like, there's a lot of reasonable discussion, uh, that happens like in the labs and even between the racing labs, uh, about the need to kind of like coordinate, the need to pause, need to be careful. And like people have been kind of public about it, et cetera. But like, there's like always like one missing component. The component is like, when? They like, they always say, I think Scott Alexander, uh, wrote a good article about, uh, about OpenAI's like AGI and beyond, uh, statement where he kind of also pointed out that like, yeah, they're saying a lot of nice things, which

• 1:23:21Speaker 00

  is great. I mean, honestly, it's, it's good. Uh, but they don't say when. And like this is to make a little bit suspicious at least. And so therefore like the, one of the rationales for the letter is that, how about now?

• 1:23:34Nathan Labenz?

  So what have you, I, I want to go through some additional kind of questions that I've seen floating around in the discourse, but.

• 1:23:43Nathan Labenz?

  What is your sense of kind of how the reaction has been? Obviously, there's been a ton of signatures, even one notable CEO of, you know, a company that I would say is like close to a leading lab, which is Ahmad from Stability, uh, signed on to it. I thought that was fascinating. As far as I know, it's been pretty quiet reaction from the kind of three main groups, you know, that are the ones who are going to either pause or not pause, you know, for something beyond GPT-4. So what do you make of the reaction from them? Like, has there been any, there's been no public statements as far as I know, and has there been any kind of private or, you know, sort of confidential reassurance? Have you, I mean, has there been any reaction

• 1:24:28Nathan Labenz?

  from the, the leading labs?

• 1:24:31Speaker 00

  Yeah, I mean, I, I think like Sam Altman like said publicly that like he's kind of like with the spirit of the, of the letter, uh, like appreciates the spirit of the letter or something. Uh, but then there was of course a but, I don't even remember what the but exactly was. Uh, but yeah, like, uh, DeepMind definitely hasn't said anything, uh, nor has on Tropic, although like I just today read, uh, Jack Clark's newsletter, uh, Import AI, uh, where he kind of like mentions the letter and also says like why he didn't mention it last week. Uh, so, uh, so there's like some, uh, some reactions, but like, uh, also kind of like want to be careful here in the sense that I don't want to like create

• 1:25:16Speaker 00

  some self fulfilling prophecies. Uh, I would say that the possibilities are definitely very much open at this stage, uh, for, for the letter somehow catalyzing, uh, an actual pause, uh, but it's like double digit uncertainties both ways.

• 1:25:33Nathan Labenz?

  Cool. That's actually more positive of a response, albeit minimal than, um,

• 1:25:41Nathan Labenz?

  than I had expected or even understood. I haven't seen that Sam Altman tweet. I'll have to dig into that and

• 1:25:46Speaker 00

  Yeah, I don't think it was a tweet. I I think I hope I'm not just like he didn't see it in in my in my dreams or something. Uh, but I think it was actually a comment in in some news article uh about uh uh about the letter.

• 1:25:59Nathan Labenz?

  Humans too suffer from hallucination at times, so we'll fact check ourselves.

• 1:26:04Speaker 00

  Indeed, we do.

• 1:26:05Nathan Labenz?

  So, okay, so we you guys put this out there. Now people start to say all kinds of stuff, right? The thing that I think is kind of most like you didn't finish reading the letter. Uh, but which is definitely worth giving you a chance to to kind of respond to is, well, the all the letter says is a pause, you know, that doesn't ask for anything else. So it's stupid because like, what are we supposed to do? Again, like you can read the letter for yourself. There are definitely some things that it calls for. But just kind of, you know, obviously, it's a big tent, you know, committee sort of document, just zeroing in on your own, you know, priorities. What do you think are like the most important tangible, concrete things?

• 1:26:43Nathan Labenz?

  That we could do over, say, a six month timeframe, such that maybe even you would be comfortable, you know, ending the pause like it do. I guess top priorities and if those top priorities happened, would that be enough in your mind to then end the pause and like trade a GPT five?

• 1:27:01Speaker 00

  So so one thing that will sort of like happen automatically is that we will get more experience with a crazy situation now that we're in, uh, which is that we have. Uh, I think I think Joshua Benja uh put it or Joshua Benja put it. That now we have AIs out there in the internet that can pass a Turing test. Uh, and that is a novel situation, uh, like, and I think we are would be much smarter six months from now than we are now. Uh, because six months will be a long period, long, long chunk of time, uh, when it comes to living with aliens on uh on on your planet. Uh, so like, uh, I don't know what we're gonna learn, but hopefully we just like are smarter.

• 1:27:45Speaker 00

  Uh, in the in the autumn than we are in the spring. But like when it comes to like more concrete things, yeah, there's like, uh,

• 1:27:54Speaker 00

  I mean, Neil Nanda, a researcher in UK who has worked with DeepMind as well as Anthropic, has this blog post called "200 Open Problems in Mechanistic Interpretability." So like, there's like so much work that can be done with even like the previous generation of of of models, not to say anything about about the latest generation. So there's like so much kind of alignment work and just opening up those black boxes and trying to understand what makes them tick, how can we, how can we get any guarantees about what what the next generation is going to do? So again, with with research and with kind of lived experience, I hope we will be in a much better.

• 1:28:39Speaker 00

  I mean, I hope we will be in a much better, but like realistically, I just say that we're gonna be in a better place six months from now.

• 1:28:46Nathan Labenz?

  That doesn't sound to me like you would expect just on general kind of improvement to get to a point where you would then say, "Okay, let's end the pause and do the next generation." Is there anything, I mean, correct me if I'm wrong on that, but maybe framing the question slightly differently. If we abstracted away from the six-month timeframe and said like, are there concrete structures of some sort that we could put in place that would on any timeline, you know, kind of give you enough confidence or reassurance that again, you could say like, "Okay, it seems like we could, we're now in a decent enough place"

• 1:29:27Nathan Labenz?

  that you would personally be comfortable going ahead with a a next-generation training?

• 1:29:33Speaker 00

  Still, like, the big question is like, how much risk are we okay taking? And I'm not saying this risk should be zero, because there is always like this background risk of existential risk. We could be hit by asteroid. There's a certain probability that like this call will not end because the planet will be hit by by like some, probably not an asteroid, because we can see this coming, but by by comet, that that is kind of harder to see, I understand. So like, we shouldn't like get the risk to zero, but like, perhaps there are, like, if there is this... Pause and like associate that realization that like these experiments are considered like too reckless by society. Hopefully, we will get this will create some kind of incentive gradient for the companies

• 1:30:17Speaker 00

  themselves to figure out how to make them in a more responsible manner and more legible manner. So I am like I'm interested this project in ARC Alignment Research Center in Berkeley, led by Paul Cristiano called evals, like evaluating models about like what what are the things that that they could in principle do, other things that that they're still kind of incompetent on. I mean, you can have like multiple opinions about this, but or different opinions about this, but like there's like a generalization of this, like, is it possible to replace the current blind, as far as I know, training runs with something that you like at every

• 1:31:02Speaker 00

  iteration, you do some tests that would give you some kind of guarantees about about alien that you're summoning?

• 1:31:09Nathan Labenz?

  Yeah, there's a few other things that I I think give me a little bit of hope that they might be developed on the kind of a technical level during that time. There was an interesting paper that came out from, I think this was a pretty small scale. Yeah, I believe it was out of Anthropic, where they experimented with doing the pre-training on a human preference data set from the beginning as opposed to just kind of, you know, random decent quality text off the internet. And it seemed that the upshot of that was that you never had quite as alien of an alien in the first place, you know, on measures of like harmfulness, you know, for example,

• 1:31:54Nathan Labenz?

  helpfulness. It it stayed much closer throughout the training process to kind of that final post RLHF or RL AIF state that we know now and never kind of dipped as far into.

• 1:32:12Nathan Labenz?

  The sort of, you know, just very strange alien territory of of general pre-training. So something like that, kind of changing the the data set maybe from the get-go seems interesting. Another recent thing that jumped out to me was somebody just, I think in the last week or two, just published a a result where they showed that they were able to increase the size of the model progressively throughout training. It was kind of presented to me as like a efficiency thing that does go to show all these things are kind of pros and cons. You know, everything is dual use, but the net savings on the training flops was like 50%,

• 1:32:52Nathan Labenz?

  which is obviously significant enough for people to take notice, you know, even for just purely commercial reasons. But then it seemed to me like, boy, there's something really interesting there where you're creating kind of this seed, you know, kernel, like truly a little baby version of the the model that is actually just a lot smaller in terms of its parameters. And then you're able to kind of layer on more and more layers, parameters, you know, as you go through the the whole training process. It seemed like maybe there would be a way to, you know, zoom in on that small thing and get get something kind of working right in like the small, you know, presumably much more interpretable version first before, you know, growing the model itself and kind of just having one big, you know, tangled knot of of parameters.

• 1:33:38Speaker 00

  I guarantee that it will not guarantee like safety in the long run, but it might just be enough to reduce the probability of destroying everything in the next generation. So we can actually kind of like do one more step in a way that is much more responsible than the current default.

• 1:33:52Nathan Labenz?

  So one thing that you have not gone to much here is regulation, government intervention. I mean, the letter does call for that. If there's, if there can be no pause, then government should step in and insist on one. But you're not putting a lot of, you know, maybe you just haven't got to it yet, but I'm not hearing anything from you that's like government can come in and set up a regime that's going to. Too much for us, is that your, do you have any hope in a sort of government regulatory approach?

• 1:34:23Speaker 00

  Oh yeah, I do. Glad that you asked. This also like reminds me that like Google in the, in form of Sundar Pichai, did react to the letter, like there was a podcast, Hard Fork, New York Times podcast, where he said that, that like the discussion started by a letter is great, but he thought that like government intervention is necessary here. That is not enough to rely on, on the labs kind of self-regulating. So like, I think like one silver lining that I already mentioned is that like this pre-training is, is super

• 1:35:01Speaker 00

  expensive and super visible and therefore like, like the metaphor that I've, that I've been using is like the nuclear control, that in nuclear, nuclear arms, you also have like these two phases. One is like the hard step of enriching uranium, which is like super energy, energy intensive as well as like much more visible than the second step, which is harder to control, which is the proliferation of nuclear grade material. So like here we had a similar situation where the pre-training is actually happens just in a few places and this is visible to governments.

• 1:35:43Speaker 00

  Whereas like the proliferation is already much harder to control. So, so yeah, my model is that,

• 1:35:49Speaker 00

  and also like it, as I know, it, it kind of resonates with, with thinking in the labs themselves, that if you want to kind of have some constraints on the AI trajectory, then kind of intervening in this compute, having some kind of compute governance is, is probably great thing to start from. And in fact, like this CHIPS Act and like the export controls to China are already things that are happening and in some ways make the problem much, much more easier, although it doesn't, it's, it's far from solving it.

• 1:36:28Nathan Labenz?

  I think that makes sense. I mean,

• 1:36:31Nathan Labenz?

  Certainly at the moment, the compute requirements are are high. I do wonder, what do you make of things like the diffusion of just language model know-how proliferation? We, I think we might have a guest coming up on the podcast that is building a decentralized GPU cluster, potentially with some sort of blockchain governance, where, you know, my M1 or M2 MacBook Air can like contribute to a cluster. Do you think, like, how long do you think we have before that barrier of just like mega compute resources goes away?

• 1:37:13Nathan Labenz?

  Possibly because of these like virtual clusters, or possibly because of like further algorithmic breakthroughs, or just model leaks could be another thing? How long do you think that holds?

• 1:37:23Speaker 00

  I mean, I gotta refer back to this, like, there's more slow math science. But yep, like with every, every year or a couple of years, destroying the world becomes easier.

• 1:37:36Speaker 00

  So there's that. But also, like, if we do not get a pause, like, do you think that the world will be destroyed by one of the big labs? So just because, yeah, the frontier models happen. It's much easier to train frontier models in a big data center than in a distributed manner. That said, if the pause does happen, then yes, we have to worry about things like proliferation, things like hackers taking the hackers stealing the stealing the weights and then kind of doing experiments like Chaos GPT, but on with a competent AI, stuff like that.

• 1:38:17Nathan Labenz?

  I think one of the maybe most kind of, I don't know if it's an argument or whatever, but maybe one of the things that's kind of hardest to argue with, in my experience, with for those that are kind of objecting to the letter, or the to the concept of a pause, is the is the sense that like, someone might say, you have no, you're not giving me anything to hope on here, really, right? You're just telling me like, every every generation, it gets easier for the world to get destroyed. Like, we've talked about buying ourselves some time here and there. But we have, we haven't really heard much of a sort of here is the path we can take to safety. And so like, why bother pausing if we don't even have a sense of where we're going?

• 1:38:58Nathan Labenz?

  Do you have any hopes for concrete paths to safety that you would try to inspire that kind of person with?

• 1:39:06Speaker 00

  Yeah, I mean, sorry, humanity, you have cancer.

• 1:39:10Speaker 00

  So it's like, you might be cured of it, but currently it doesn't look good. So it's like, I'm not gonna lie, the prognosis is bad. But like, it's not hopeless. And also like the kind of like silver, massive silver lining is that if we do manage to survive the cancer, uh, the future is going to be amazing. So like in some ways, uh, kind of the expected value of the future

• 1:39:38Speaker 00

  is not bad. It's just like the odds of survival are bad. Uh, but if we survive, uh, the, the, the life, the life of the world, the universe could be potentially like, like unfathomably, unfathomably better than it is, it is now. So we, in, in sense, we are living lottery, living a lottery ticket. Uh, and it is in some way in our control to improve the odds. And so that's what I'm doing.

• 1:40:06Nathan Labenz?

  Well, that's probably about as good of a bottom line as we could hope for, for this conversation. So, uh, I wanna thank you for spending the time with us. I do have a couple real quick hitter, just fun questions that I usually end on if you have an extra second for those. We touched on this earlier. Any applications aside from like the obvious, you know, kind of usage of the core language models that you are personally just finding delightful or useful that you would recommend that people check out?

• 1:40:37Speaker 00

  Uh, no, I'm just like way kind of pan with, uh, limited, uh, to, to tinker, uh, with, with the language models. I mean, I, I've done a little bit, uh, but then I, I'm trying to find coders at this point, uh, to like delegate bunch of my projects, uh, some of them including, some of them might involve language models.

• 1:41:01Nathan Labenz?

  Fair enough. You're in the majority on that answer. Uh, most people are just using a few things. And then second, um, let's imagine a world where we're here in a couple of years and... Neuralink has been deployed to 1 million people. In this scenario, you are well. So you don't need it for restoring any functionality. But if you were to get a Neuralink implant in your head, it would give you the ability to essentially transmit your thoughts to devices. So you would have effectively thought to text or thought to UI control. Would that be enough for you to be interested in getting a Neuralink implant?

• 1:41:46Speaker 00

  Depends so much on details like how reversible is the procedure? What are the risks? And what is the demonstrated upside? Will I become a better dancer as a result?

• 1:41:57Nathan Labenz?

  Well, yeah, that one I they did show in their in their show and tell. They did show an animal where they weren't creating motor control through the Neuralink. But yeah, I think it was a long way from improving on your dancing skills. So I hope you are dancing for many years to come. Jan Tolan, thank you so much for spending this time with us. We appreciate you being part of the cognitive revolution.

• 1:42:24Speaker 00

  Thank you very much.

• 1:42:26Speaker 02

  Omnikie uses generative AI to enable you to launch hundreds of thousands of ad iterations that actually work, customized across all platforms with a click of a button. I believe in Omnikie so much that I invested in it, and I recommend you use it too. Use Cogrev to get a 10% discount.