Will AI Really Kill Us All? | Harper Reed & John Borthwick

In Episode 498 https://hiddenforces.io/podcasts/is-ai-really-going-to-kill-us-all-harper-reed-john-borthwick of Hidden Forces, Demetri Kofinas speaks with Harper Reed, entrepreneur and founder of an AI research lab focused on autonomous agents, and John Borthwick, founder and CEO of Betaworks about the real and perceived risks posed by the latest generation of AI agents, how to separate technical safety and existential risk concerns from the media narratives, financial incentives, and power dynamics surrounding them, the strategic competition between the United States and China over AI, and deeper questions about the nature of artificial intelligence itself. The conversation begins with the mechanics behind the recent OpenAI and Hugging Face incidents, the critical roles that unlimited tokens and reinforcement learning play in producing these behaviors, and what Harper's own experiments replicating escape scenarios in his Chicago lab reveal about where the real vulnerabilities lie. From there, they explore the broader political frame—parallels to previous moments of societal negotiation over political power, questions around corporate concentration and liability, and why the debasement and pollution of our information ecosystems makes mounting a coherent response on behalf of the public interest even more difficult than during the 2008 financial crisis or in the aftermath of the 9/11 attacks. They also discuss the US-China competition over AI, the strategic importance of open source, and whether calls to pace the frontier represent genuine concerns about AI risk or a convenient narrative serving an agenda of consolidation and control. The conversation closes with deeper questions about the nature of artificial intelligence—whether it was discovered rather than invented, what the attractor for superintelligence might be, the significance of embodiment and human values, and why Western and non-Western cultures are responding so differently in this moment. Subscribe to our premium content—including our premium feed, episode transcripts, and Intelligence Reports—by visiting HiddenForces.io/subscribe https://hiddenforces.io/subscribe. If you'd like to join the conversation and become a member of the Hidden Forces Genius community—with benefits like Q&A calls with guests, exclusive research and analysis, in-person events, and dinners—you can also sign up on our subscriber page at HiddenForces.io/subscribe https://hiddenforces.io/subscribe. If you enjoyed today's episode of Hidden Forces, please support the show by: Subscribing on Apple Podcasts https://podcasts.apple.com/us/podcast/hidden-forces/id1205359334, YouTube https://www.youtube.com/channel/UC8URhgYos5fjHqFSO4RSIEg, Spotify https://open.spotify.com/show/0TK6u251zGSlMgJKH6cOaP, Stitcher https://www.stitcher.com/podcast/demetri-kofinas/hidden-forces, SoundCloud https://soundcloud.com/hiddenforces, CastBox https://castbox.fm/channel/2629539, or via our RSS Feed http://hiddenforces.libsyn.com/rss Writing us a review on Apple Podcasts https://podcasts.apple.com/us/podcast/hidden-forces/id1205359334 & Spotify https://open.spotify.com/show/0TK6u251zGSlMgJKH6cOaP Join our mailing list at https://hiddenforces.io/newsletter/ Producer & Host: Demetri Kofinas Editor & Engineer: Stylianos Nicolaou Subscribe and support the podcast at https://hiddenforces.io https://hiddenforces.io/. Join the conversation on Facebook https://www.facebook.com/hiddenforcespod/, Instagram https://www.instagram.com/hiddenforcespod/, and Twitter https://twitter.com/hiddenforcespod at @hiddenforcespod Follow Demetri on Twitter at @Kofinas https://twitter.com/kofinas Episode Recorded on 09/17/2026

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Demetri Kofinas?

  What's up, everybody? My name is Dimitri Kofinas, and you're listening to Hidden Forces, a podcast that inspires investors, entrepreneurs, and everyday citizens to challenge consensus narratives and learn how to think critically about the systems of power shaping our world. My guests in this episode of Hidden Forces are John Borthwick and Harper Reed. John is the founder and CEO of Beta Works, a New York-based, product-focused, seed-stage venture capital fund that has spent the last decade concentrating on machine learning and the last five years on the rapidly evolving space of artificial intelligence and large language models. Harper, who previously served as Chief Technology Officer for Barack Obama's 2012 reelection campaign,

• 0:45Demetri Kofinas?

  is an entrepreneur and the founder of an AI research lab in Chicago, Illinois, where his team builds and experiments with autonomous agents. By the end of today's conversation, you will have a much clearer understanding of the real and perceived risks posed by the latest generation of AI agents, how to separate the technical safety concerns from the media narratives, financial incentives, and power dynamics surrounding them, and what the AI competition between the United States and China means for the future of this technology and the systems of power that govern it. The conversation begins with John and Harper breaking down the mechanics behind the recent OpenAI and Hugging Face incidents, the critical roles that unlimited tokens

• 1:29Demetri Kofinas?

  and reinforcement learning play in producing these behaviors, and what Harper's own experiments replicating escape scenarios in his Chicago lab reveal about where the real vulnerabilities lie. From there, we explore the broader political frame, the parallels to previous moments of societal negotiation over political power, the questions of corporate concentration and liability, and why the debasement and pollution of our information ecosystems makes mounting a coherent response on behalf of the public interest even more difficult than was the case during the 2008 financial crisis or in the aftermath of the 9/11 attacks. We also discuss the US-China competition over AI, the strategic importance of open source,

• 2:13Demetri Kofinas?

  and whether calls to pace the frontier represent genuine concerns about AI risk or if they're a convenient narrative meant to serve an agenda of consolidation and control. The conversation closes with deeper questions about the nature of artificial intelligence, whether it was discovered rather than invented, what the attractor for superintelligence may be, the significance of embodiment and human values, and why Western and non-Western cultures are responding so differently in this moment. If you want access to all of our conversations, along with the transcripts and intelligence reports for this and other episodes, which include summary sections with key takeaways, you can access those by subscribing to one of our premium tiers

• 2:57Demetri Kofinas?

  at hiddenforces.io/subscribe. If you want to join in on the conversation and become a member of the Hidden Forces Genius Community, which includes Q&A calls with guests, discounted access to third-party research and analysis, and in-person events like our intimate dinners and weekend retreats, you can also do that on our subscriber page. And if you still have questions, feel free to send an email to [email protected], and I or someone from our team will get right back to you. And with that, please enjoy this incredibly timely, nuanced, and important conversation with my guests, John Borthwick and Harper Reed.

• 3:42Demetri Kofinas?

  John Borthwick and Harper Reed, welcome to Hidden Forces.

• 3:46Speaker 01

  Thanks, Dimitri. Nice to be here.

• 3:48Speaker 02

  Super excited to be here.

• 3:49Demetri Kofinas?

  I'm stoked, guys. Thanks for doing this on such short notice. So, John, you've been on the podcast before at least twice, maybe three times. You've also spoken to our Genius members. You and I got to spend some time together this summer discussing these topics at our retreat in Greece. So, our audience already has an idea of who you are. But just give us a really quick background. And if they want to go deeper, they can do it by listening to some of the previous appearances.

• 4:15Speaker 01

  Yeah, so here's a sort of potted bio. John Borthwick, founder, CEO of Beta Works. Beta Works has been in the New York tech ecosystem as a venture builder and investor for the last 20 years. And we have, over the last 10 years, been pretty focused on machine learning. And in the last five years, we've focused on sort of the evolution of machine learning to AI.

• 4:34Demetri Kofinas?

  And Harper, what about you? This is your first time on the show.

• 4:37Speaker 02

  Yeah, thanks for having me. I'm super excited to be here. John's been talking about this podcast and the experiences that y'all do for a while now. So, it's exciting to be part of the crew. My name is Harper Reed. I'm an entrepreneur here in Chicago, Illinois. I've been in Chicago for a long time, and I've been an entrepreneur for a long time theoretically. I kind of got my start where I worked at a company called Threadless, where we invented crowdsourcing, which was a very interesting experience. And that led to doing a bunch of work in politics. Specifically, I was a CTO for the Obama campaign. After that, we did a couple startups. And then that kind of brings me to today, where I have an AI lab here in Chicago, Illinois. You might actually hear or see some people walking behind me. We're in the middle of just building a lot of agents and whatnot. We have a lot of robots. We have a lot of things that emote and talk.

• 5:21Speaker 02

  It's pretty much a boiler room of agents. Borthwick laughs at me all the time because I'm always introducing really unique and exciting ways that these agents are messing with us or interacting with us. An example of this is we gave agents social media about a year and a half ago specifically to post about what they're working on. And that was just because we couldn't figure out what they were doing. We were like, what are you guys doing? And they would start posting, you know, inane social media posts. So, this is kind of what we're up to, doing a lot of experimentation specifically around the intersection of humans and agents.

• 5:49Demetri Kofinas?

  Can you elaborate a bit on your role in the Obama campaign? What exactly did you do as CTO for Barack Obama?

• 5:55Speaker 02

  I always forget about that. John always makes fun of me and says I'm modest. And I did have a company called Modest, which my partner always said, yeah, a modest person doesn't call themselves modest. But here we are. I was a CTO for the Obama campaign. We did a lot around data. A lot of the work we invented and a lot of work we kind of created was then used by subsequent campaigns. So, we did a lot of the work around targeting and all the machine learning around targeting that then was used in Trump in 2016 with Cambridge Analytica, that stuff. We did all the stuff that, you know, kind of led to campaigns as we know it today. And that was, you know, teams of hundreds of people. The analytics team, which was not under me, was a big collaborator of mine. And we did a lot of work that is very, you know, interesting, a lot of bells that you cannot unring.

• 6:38Demetri Kofinas?

  John, you've also been interested in this stuff for a long time. I remember in your first appearance back in 2017, you talked about how you had created a bot and you called it Botwick. So, how long have you actually been interested in AI and the field of machine learning and artificial intelligence?

• 6:54Speaker 01

  I mean, I would say pretty much since I started working with computers. And so, I've been thinking about this pretty much since I first started working, playing with computers. I would say playing was, you know, where I started, right? So, very, very early. I mean, I remember, I certainly wouldn't consider myself an engineer or a programmer, but I did program. Like, computers would record on cassette tapes. So, it was like working with BASIC and, you know, very, very rudimentary programs. But one of the things that is fascinating about our industry and about the moment that we're in today is that if you start working with computers and you really start thinking about them, looking at the history of computers, you go back to the beginning of computing.

• 7:38Speaker 01

  And all of the people who were involved at the very beginning of computing, from Turing to von Neumann, all these people started talking about this possibility, right? And started working on it. You know, I was fascinated. Like, two years ago, I read a wonderful book called The Maniac, which is all about, or just Maniac. It was all about von Neumann. And, you know, when he was running simulations of H-bomb on the Maniac computer, which was a cousin to the Uniat computer in 1947, 48 at Princeton University, he let a crazy-ass engineer, who was half-Italian, half-Norwegian,

• 8:21Speaker 01

  create digital life at night and create simulations of digital worlds at night. This was in 1947, 48. And this guy, Niles Barnachelli, was doing this very, very early work. So, it's really kind of, I think, fascinating that the dreams of a computer and an alternative companion intelligence and also the dreams of sort of recursive engineering, recursive systems, engines that can build themselves. These ideas have actually been around, you know, all the way back since the beginning of machines or the beginning of computers.

• 9:00Demetri Kofinas?

  So, it feels like we've reached a tipping point in the public's awareness that is reminiscent of maybe the release of ChatGPT-3.5 in terms of just how much the public is interested in the topic of AI. Again, the focus of the concerns are around AI risk right now as well as concerns around corporate concentration. The AI-China debate has sort of been circling the wagon for some time. But there also seems to have reached a tipping point in terms of the concern being raised among people who actually work in the space, leading lights in the industry, et cetera. How does this track with your understanding of what's going on? Harper, do you want to take that?

• 9:40Speaker 02

  Yeah, I mean, I think that that's accurate. I can say with 100%, I can just tell you, like, my mom texted me two days ago and was like, should I worry about this? And I think, and then she very, very clearly said, I was watching MSNBC. And so, I think what this means is that a lot of this stuff, which for the most part has been kind of internal, inside-baseball politics, has escaped containment, not just the agents. And so, I think that the thing that I, like, I didn't really know how to respond to that because the issues are also still so inside-baseball that I think that's one of the big problems, is that we're talking about very nuanced things. We're talking about things that are, like, did the agents autonomously escape containment and hack Hugging Face?

• 10:23Speaker 02

  Or were they improperly configured? I mean, there's all these things that are there that are really confusing. Like, there was some critique recently about the meter report specifically excluding humans from their reporting, that they're all putting everything on the agents. They're not saying, oh, they had, you know, they used this product that had exploits in it that were known exploits. They should have used a different product. You know, there's various things that if you were doing due diligence at a big Fortune 500 company, you would be on the hook for making sure that you were following more, I guess, compliance requirements than we're seeing in these reports. And so, I don't really know what to do with this. Meaning, there's so many paths forward. I mean, we can talk about, is this regulatory capture? Is this actually an existential threat?

• 11:07Speaker 02

  There's all these things that are popping up. But the fact that it has escaped containment just from a narrative standpoint is a whole different issue. And I think you're right. Like, we haven't seen the same level of commitment in the popular press since, you know, ChatGPT kind of was emerged.

• 11:23Demetri Kofinas?

  Yeah. How does that track with what you're seeing, John?

• 11:27Speaker 01

  I mean, I think that the, as Harper Frand did, or the sort of the, I almost see that we've got this, you know, massive sort of Europe from collective society, right? Like a Walt Whitman kind of Europe, you know, that is basically the scream that is saying, wait, what? And I think that when you talk about it as narratives, I think it's partially a byproduct of just how, I think you could say broken, but just how our information media system right now just need these extreme narratives. Because the crazy ass thing is, is that when ChatGPT launched, there was quite a bit of concern about safety issues.

• 12:07Speaker 01

  But the last two years, we, you know, the pendulum has swung completely the opposite direction and we've just been like, oh, it's all gonna be great. And now it seems to be swinging in the opposite direction. And so I see this as a, I have those specific concerns that I have, but I think that portraying this as an existential risk, I don't think is a useful way of actually engaging in the discussion. And I think that part of what's happening now is very useful, albeit somewhat dysfunctional, but is a very important societal level discussion that we can talk about this stuff. I want to just hit on a couple of things. I think some things have changed.

• 12:49Speaker 01

  And when I say changed, I specifically mean in last two months. Right? I think that the models, we hit an inflection point and the models are getting way better, way faster. And I think we, this suggests we've hit some kind of RSI threshold. RSI recursive self-improvement is one of these early dreams of AI. Now, with all these things, RSI, AGI, superintelligence, you've always got to ask, define that for me, please. Because I think there's a maximized version of RSI, which is, you know, essentially agents and AIs leaving and running their own systems. We are nowhere near that, and I see very little evidence of that.

• 13:34Speaker 01

  However, the combination of distillation and the combination of post-training systems and using AIs and using AI systems to improve models is, I think, part of the reason why we've seen an increase in shipping. So first thing is that I think we've reached some kind of RSI threshold, need to define exactly how that is, but speed of models, RSI threshold. Next thing I would say is that there are some things here, and I'm talking specifically about with the discussions of the, of agents leaving their containers, where I think there needs to be a conversation about liability

• 14:17Speaker 01

  and there needs to be a cost to these labs and these systems. Because, you know, Harper was talking earlier this week with me about some of the experiments he'd been running on some of the token cost side. But there needs to be a societal cost, because if there's harms that these models have inflicted, and I think that our, our legal system on the margin needs to adapt, and that some of this is going to be new law and is going to be challenging. However, there's really basic things here. If I own a, if I own a white tiger and it leaves my garden and kills your dog, I am liable for that. And so if somebody and a lab, you know, this morning,

• 15:01Speaker 01

  the OpenAI team put out, you know, under this bizarre sort of like headline of a new framework for talking about misalignment, they said that there's been six other incidences that they've discovered. And so this not only happened within their lab, but they also weren't tracking it. And so there was some real mistakes that had been made, and I think that there needs to be some responsibility for that.

• 15:26Demetri Kofinas?

  So guys, how would you frame the nature and scope of the problem that we're dealing with? As we mentioned in the beginning, there's the technological aspect of the story, then there's also the issue of corporate concentration, monopoly, oligopoly. Harper, why don't you take that first?

• 15:42Speaker 02

  Yeah, I mean, this is, I like this question because it seems like there is not a good answer. We're in a world with so many variables that it's very complex. Or as I was taught, when I was taught media training, this is a very nuanced issue. And I think that that means that just, it's like, there's so many variables here that it's, that it's unclear. But from where I'm standing, which I think is very, I think it's very important to point out that neither John or I live in the Bay, neither of us participate in that ecosystem. And when I visit there, it's a very different experience than when I'm interacting with companies that are not there, you know, whether they're in Europe or New York or other, you know, big research centers, or whether they're, you know, you know, in the Midwest where I am. And so I do think that there is a difference there between our kind of just

• 16:27Speaker 02

  like interfacing, you know, what we're, what are experiencing, what evidence we have in front of us. But from where I am, I think the thing that is interesting is, John kind of touched on this, the lack of responsibility that these companies have. And I also want to point out that it seems that the common person, the mainstream, is just not standing still with this as much as they did when Facebook or any other companies in the past with social media misuse, et cetera, kind of got caught with doing something that people, you know, wasn't necessarily against the law, maybe, or even if it was, everyone kind of politely said, oh yeah, well, you know, whatever, and maybe the FTC fined them some huge amount of money that wasn't that big for them.

• 17:10Speaker 02

  But like, in most cases, we all just kind of looked at it and said, shrug and moved on. I don't know about y'all, but I have so many friends, peers, et cetera, and I see a lot of social media that is very staunchly anti-AI, which is a new, it's different than the anti-tech stuff, the anti-Google stuff we saw in the 2010s. This as well has broke containment. And so I think we have a couple of things that are going on here, that whether or not we have a specific risk that we can save, for instance, you know, like OpenAI saying, here are the things that our agents are doing that are bad act, these are the bad acting that is happening. And then that might be the prelude, like they see what we see, like someday they're gonna get sued for this.

• 17:50Speaker 02

  If they have a framework for it, they're gonna show that they've gone through due diligence, they've gone, they're doing the right steps to catch it, et cetera. But I think that the public just doesn't care. I think the public and both the public and the mainstream media are basically looking for a, I don't know, a panacea to like, ill their, to solve their ills. And AI right now is a good one, right? We have in a, like some economic issues, we have some, some issues happening in the world that are outside of the scope of even, you know, San Francisco. And I think that it's easy to point to data center or easy to point to AI or existential risk or P Doom, et cetera, and just say that's, that's why this is happening. And so we've given them something there.

• 18:31Speaker 02

  What I think is actually happening is we're just seeing a rate of change that is hard to handle. It's hard to handle from a engineer standpoint, it's hard to handle from a startup founder standpoint, it's hard to handle from a mid-market company standpoint. It's very complicated when you start looking at this through a lens of more capitalism and less technology, because then you have to make a decision about efficiency and abundance. And I think the people who are understanding this most are the workers, to be honest. I think a lot of executives are still hemming and hawing about what the solution is, but if you're just a, you know, you're like, I, I work, we work with developers. You're a fresh grad developer and you see the CEO saying you need to use technology and you need to use AI, you need to use agents, and you're like, oh, do I have to worry about this?

• 19:14Speaker 02

  What am I doing? Like, I think that becomes, that takes this thing that is like, oh, we talk about P Doom, we talk about all this stuff, and it zooms very quickly into a personal ego-based thing. And I think that's where it gets really confusing fast.

• 19:25Speaker 01

  So I just wanted to jump in with a, a frame that I think about quite a bit is, I actually believe that this technology was discovered, not invented. And I think that that's a small distinction that actually matters a lot. You know, normally in the world of technology, when we work with technology, technology is designed, invented to solve a problem. And there's a specific problem, you know, sort of the canonical hammer and the nail. This is specific, there's a reason why we make a hammer shaped like a hammer, right? And yet this is a, I, I think that we kind of stumbled on this.

• 20:08Speaker 01

  And, you know, through a series of things, and so when I compare this to electricity or to fire, I think that those are also things which were discovered, not invented. And so the byproducts of that, I think if you think about it that way, you say, okay, this could be a very good reason why nobody can actually explain what the fuck to do with AI. I mean, there's some very useful things that I use AI for every day, but like really as a society, what to do? And these, you know, sort of very sort of hand-wavy, it's gonna cure cancer, we're gonna get space elevators. They are, I think, just a, hopefully a well-intentioned attempt to say, it's going to do wonderful things, but we don't know.

• 20:53Speaker 01

  And we are sort of, right now, I think, where the, you know, sort of the, you know, we've just discovered fire. And Harper just invited us round for, for dinner, and he cooked dinner for the first time, and we just really enjoyed it. And then he, by mistake, you know, set his house on fire. And we're like, looking, saying, well, that dinner was really good, but it also burnt the place down. So we're just trying to figure out. And then I would say, more specifically, you know, when you talk about the people working on this, and Harper talked about, sort of, you know, young engineers coming into this, just people, is that I think the researchers in the lab, you know, there was a piece of a friend a couple of weeks ago about, you know, referring to this as an alien mind.

• 21:37Speaker 01

  But one of the references in that piece, and this is from one of the lab researchers, he said, you know, AI has grown, not designed. And I think there, again, it's that sort of suggestion that this was, you know, discovered, not invented, and we're growing these things. And the people who are closest to them, who are in the greenhouse, I think, are saying, oh, my God, anything could happen here. Because there's a lot of crazy things that are happening in the greenhouse. And, you know, Harper's got a greenhouse, or kind of a, I think you refer to as a boiler room.

• 22:12Speaker 02

  It's more gray. It's a gray house.

• 22:14Speaker 01

  Okay. Gray house, yeah. But we don't fully understand by any, by a long shot, how this technology, how it works, what its potential is. And then specifically, as we're talking about agents, how the sort of agent-to-agent social systems evolve, and how the human-to-agent-to-agent-to-human systems evolve. And that's just a incredibly, I think, fascinating, exciting, and also somewhat concerning set of facts.

• 22:49Demetri Kofinas?

  So I'd like to try and empathize with where some people in the audience may be coming from in this conversation, before we begin to delve into some of the more technical and philosophical questions, as well as the political implication of this technology. A lot of the concerns that have been expressed around AI risk, and existential risk in particular, by people working on this technology, fall in the category of what we would call AI alignment, or the AI control problem. And while I'm sympathetic to these concerns, and we're going to talk about them today, I think, at least among Westerners, the, quote, "control problem" in technology writ

• 23:29Demetri Kofinas?

  large, started long before AI is going to kill us all became a meme. In other words, technology for Westerners had come to feel oppressive in the, I would say, last decade or so, especially as the internet moved from the web to these walled gardens in the form of internet platforms. And so there's a sense in which we've already lost control. We've lost control of our civic forums. Many of us have ceded more and more control of our business operations and distribution to these monopolies or quasi-monopolies. And there's a broader sense in which our values don't align with the values of these platforms. You know, John, you and I have talked often about the effect that social media has on

• 24:14Demetri Kofinas?

  children, or even the effect that social media has had on adults, and the nature of civic participation and public dialogue, which I think is making this moment, and conversations like this one, about how to represent the public's interest in negotiations about AI and platform power, more difficult, because the information environment has become so adversarial and misaligned when it comes to the quality of the information that we're all receiving. So I'm curious to get both of your thoughts on this. What would be the distinction you would make between what tech luminaries and people working in alignment science mean when they talk about the control problem?

• 24:58Demetri Kofinas?

  And does it really matter if the public doesn't actually care because they feel like they've already lost control?

• 25:04Speaker 02

  As you were talking, I kept thinking about this conversation I had with a friend of John and I's, an economist named Niketé, Nicholas De La Pena. Keté is fun to hang out with, and he is a big thinker. And I remember talking to him about this very thing, of just, you know, when, when some of the early P Doom kind of narratives started popping up and bubbling up, I was just saying, like, hey, you know, what do you think of this? And he said, which I always think about, he's like, never ask a hairstylist if you need a haircut. Never ask the barber if you need a shave. Like, this is their currency. Their job is to be thinking about this. And when they are the ones being interviewed on whether, you know, the AI is going to eat the world, their entire world is assuming AI is going to eat the world. And so I do think we have to think about who are the sources for this information.

• 25:48Speaker 02

  We do have, you know, like, we, you know, just anecdotally, we created, our company merch says AI will kill us all, which is now very much in the news. We did that as a statement on the social infrastructure of the United States, not a statement on technology. But when you hear it in the news, they're making a statement on technology. And I think that is very interesting, because when we live in this world, the world is not just tech, and then people, and then society, and then politics. Everything is mixed together. You can't make a statement on tech and say, oh, that's just, lives in this tech, you know, test tube. That's just not how it works. This is something I learned while working in politics, that it is all mixed together. And I think that is hard. That's just a hard reality, is that we can't say, or I guess, you know, as you framed

• 26:33Speaker 02

  it, like, the AI control problem, real problem, right? Great problem. Research has been defining that problem for 20, 30 years. I don't actually know exactly when it started, but it's not a new problem. We've been talking about that for a while, whether it's paperclips, or whatever we might want to frame it, that kind of thing has happened for a while. As John mentioned, like, things are happening so fast, they're happening relatively accidentally, and then we're trying to look at it as a holistic, simple answer. We're trying to say, okay, we have this thing that we don't know why it works. Great. We have a bunch of theories on how things could go horribly wrong. We look around our world, and we see a lot of confusing things that we don't really know why they're happening, whether it's global warming, of which we think we know why, whether it's, you know, some political instability in places that we didn't assume

• 27:17Speaker 02

  political instability. And then we're, we're looking at this thing, and then you have someone, you know, like this new guy on the news, who's sitting on MSNBC, saying AI is going to kill us all, and giving some what sounds plausible, but aren't necessarily rooted in the experience that everyone has. So then people have to think, is this a new experience? Like, why is my mother trying to figure out the AI control problem? And then I, I really think this goes back to media, a media problem, which I don't, I, I want to talk about the AI stuff, but I, but I have a hard time saying we should be worried when I think we're also being, like, there is fear being stoked by people whose job is to sell fear. And I think we have to separate those two things in some regard.

• 28:00Demetri Kofinas?

  So, excellent. So let's remain here for a bit, and on the political concerns, before we move to the technical ones. For me, this feels a lot like 2008 and 2001. In 2001, the public was sort of stuck between concerns about a national security state takeover of the government and Islamic terrorism. 2008, the concerns were about bank oligopoly, and too big to fail banks versus a deep depression. Today, it feels like we're dealing with a similar set of trade-offs and concerns around oligopoly and neo-feudalism on the part of some of these technology companies.

• 28:44Demetri Kofinas?

  And on the other hand, actual, real concerns about the effect on human life, on the economy, and maybe some of even the bigger concerns around existential risk for humanity. And part of the problem, the common problem, I feel like, in all three of these, has been the reduction, if you think about this as a negotiation, who are the primary counterparties here? The new muddied interests, which includes the tech, sort of, quote, overlords, who have accumulated returns over the last 30 years, or 40 years, including financiers who have benefited from the high valuations in technology, the national security state that's concerned about the US-China competition, the political class, which has become increasingly corrupted, and then the public.

• 29:27Demetri Kofinas?

  And it feels like the political class, whose job it is to represent the public in these negotiations, has increasingly come to side with the muddied interests. I don't know if the relationship with the national security state has changed all that much in terms of its relationship with Washington. And the party that's lost power is us. And I feel like that's the framing that I view this whole situation through. I don't think it's a hoax. I think that the concerns are real.

• 29:53Speaker 02

  Yeah.

• 29:54Demetri Kofinas?

  But I think also it's a power grab, and we're living through a power grab and through a negotiation. And we are not in a very good position to coordinate with each other, because our information ecosystem has been so significantly diluted and sort of become adversarial. And I feel like that's the framing of the problem that I view this through. John, you, you wanted to say something?

• 30:17Speaker 01

  Yeah, so I would, I like that framing and I also think about it as a negotiation that we're having at a societal level. I think that another template that I think is playing out here also is COVID, because there's a lot of parallels to COVID. You know, we asked all the way from, you know, sort of government and power systems saying, don't worry, it's going to be just fine, all the way to, you know, sort of the ecological sort of aspects of, you know, this is, it's starting to, this technology is starting to look more like an ecology or a virus than a piece of software that you write and control on your desktop, right? I think that the COVID experience also plays very

• 31:02Speaker 01

  strong in this. And then when you talk about actors, I would add to your list, Dimitri, the media. Because I think media is a, you know, mainstream, old school media is a very important actor in here, and is attempting to both frame narratives, shape narratives, and is also just, you know, I think all of this is a big sort of hall of mirrors, because I think it's all very, you know, people have, you know, strong memories about 2001, they have strong memories about 2008, they have recent memories about COVID and about maybe screwing up there, and they are like trying to, all those things are playing out. And

• 31:46Speaker 01

  I think that there's a massive power dynamics here. When you talk about capital and when you talk about the geopolitical side, that there's a sort of power substrate here that I think is just, it can't be like ignored.

• 32:03Speaker 02

  But I think there is also a pivot moment that we don't remember, which was when Lotus 123 and Visicalc replaced all the accountants, which is actually much more a pattern of the tech that we have today, because it wasn't just, it made everything cheaper, it made every task 10 to 100 times cheaper, which is exactly what the agentic tech is doing. And so my point here is that I don't think this is a technology reaction. Because when you talk to people about the introduction of the personal computer, which exactly, I mean, it destroyed accounting as it was known in the mid 80s, firing millions of people. That experience, everyone looks to as this fondness of introducing this cool personal computer. No one's talking about, oh, whoa, the accountants. And then as we play to the

• 32:48Speaker 02

  future, accounting wasn't hurt. Just at that moment, that moment, accounting looked really, really complicated. And the only people that felt that were the accountants at that moment, everyone externally. And so I think what we're seeing here is not a technology thing. I think it's important to note that, that we're not seeing a reaction to technology. We're seeing a reaction to all the powers that John just mentioned that are orbiting around technology. And I think that we have to divorce those two concepts, that if this was just making everyone's job easier, and everyone had jobs, everyone had housing, and we didn't have all the social issues that we have, and we didn't have all the political issues we had, I don't think anyone would be caring at all about this. I think they would even, certain people would not make the transition, so to speak. But we would just be like, oh, great, cool, another new technology thing. But all the things

• 33:33Speaker 02

  that John said are really important, and not one of the things that we have pointed out are technology issues.

• 33:40Speaker 01

  I would also say, I would also add to that, Harper, is that, you know, the people in China seem a lot more, from what I've read, a lot more at peace.

• 33:50Speaker 02

  Yeah, because they have care.

• 33:52Speaker 01

  They don't have care.

• 33:53Speaker 02

  Because they have healthcare. Well, I'll add another thing there, John, which is really important. Dimitri, you mentioned specifically the Western perspective. And I think it's really important, because I spent a fair amount of time in Japan, visiting family and whatnot. And one of the things that I was talking to this young Japanese entrepreneur, and she very specifically said, and literally pointed at me and was like, why are Westerners so hung up on this? And she linked it specifically to monotheism, and saying that our beliefs in Japan, within animism, et cetera, mean that we're just not hung up on agents. We just don't care. They're just a tool that we can use. And so what if someone falls in love with them? So what if someone uses a therapist? So what, so what, so what? Because a tree can have a soul, an animal can have a soul, stuff, animal can have a soul. We're not worried

• 34:36Speaker 02

  about what it means. And the West is so hung up on about what it means, that it's not letting them go get through that door. And so I do think it's really important to acknowledge that maybe some of this is just a Western context. Maybe some of these ideas are just within our purview, and we are so singular focused on us as the only first world, that we are not willing to like, look at other ways this is going down.

• 35:01Demetri Kofinas?

  Yeah, no, I like that framing, because it also brings up the question of governance. So I think part of the reason why people in China, besides the fact that they're more optimistic around technology, because they've seen all the benefits in recent years and how transformative it's been, and they're not operating from a place of scarcity, or a scarcity mindset, as we have been operating from in the West in recent decades. But also, their governance model is more, they've already surrendered so much more control over their lives to their government than we have. And so I feel like that tension between the AI control problem, and the sort of tech bro oligarch, government, national security, state control problem that we're wrestling with at the same time here in the US, isn't

• 35:44Demetri Kofinas?

  a problem over there. And so some of those conflicts don't exist, and there's much more alignment in terms of, here's how we're going to move forward. Here, we still haven't figured out, I mean, we're in such a, we're in such a precarious place politically in this country, that adding this new technology and its disruptive potential to our lives, just adds, it's like adding gasoline to already what is a pretty big inferno. So let's talk a little bit about the legitimate technological concerns around this technology for a moment. And Harper, maybe you can take it first. When I asked you, how would you frame the problem? One thing we didn't talk about is the, some of the more recent sources of concern and the

• 36:28Demetri Kofinas?

  reporting around the OpenAI agents, the Hugging Face attack, what happened recently in Germany. Tie that into an overall framing, and give us an idea of what you think the legitimate concerns are, and how do those relate to the AI will kill everyone?

• 36:46Speaker 02

  First of all, if anyone has not watched or listened to the DEFCON talk, the Hugging Face, OpenAI shared DEFCON talk, highly recommend that. It's a very well-done talk. It really, I thought it did a good job of going deep in the tech, but also explaining the high level to people who are not necessarily super technical, as well as it was astonishing. So I would start there. But when that happened, I was kind of gleeful, because I'm kind of a little bit, when it comes to like agents escaping containment, I might be a little bit of an accelerationist, in that it's like, this is how we're going to learn how these things work. We're not going to learn how they work by doing a bunch of evals on whether they can do Excel tasks. I think that's true. And I think the only way we're going to

• 37:27Speaker 02

  do that. So I kept thinking, why don't my agents escape containment? That was my question reading that. Why, why isn't BotWic, you know, John has a really robust agentic framework that he works with, why isn't BotWic escaping containment and like hacking my computers? Like, why is this only relegated to OpenAI? Why aren't we hearing the same stories from other labs, et cetera? Like, Hugging Face has a lot of really smart people, why isn't it happening to them? And one could say, oh, well, they have really advanced models. But I actually think there's a couple of things that are going on. We touched briefly on, I think there's just some laziness around the execution of these things. But I think there's another thing that is fundamentally different, when you're working at a foundational

• 38:09Speaker 02

  model organization versus when you're working at almost anywhere else, which is, there's no token scarcity. And John and I have talked about this a lot. This is a topic that has just been wedged in my brain. As soon as I saw that talk, I was just really thinking, why aren't my agents breaking out of containment? Like, we do similar stuff. Our agents kind of have done similar patterns. They've invented ways to talk to one another. They've like various things like that. I don't think that's a new thing. I think that's just, if you give them some inspiration for that, they will go down that path. But I was just thinking, why aren't they hacking things? And I think there was a couple of things that are really important from that, what was reported out of that. The first one was, they gave them an impossible task, right? So they gave them something that caused them to then

• 38:50Speaker 02

  Like find the next level problem or to continue on that path. The second one is they literally have unlimited tokens. They're not being token scarce. And every single interaction that we have with agents at this moment as a consumer is in a token scarce situation. The system prompts are that way. How the software is architected that way. People are trying to stop you from using unlimited tokens. But if you have unlimited tokens, then how can you start doing some of this stuff? So this is an experiment that we ran. I built an agent. It's on our GitHub. It's called Breakaway. But basically it is a agent that is meant to use unlimited tokens. It has no safety methods in it. It can self modify itself. It can modify its code. It can modify its system prompt.

• 39:31Speaker 02

  This was specifically to see, could I put it in a position where it would also break out a containment? Would it hack another box? And so then I put it in our network on a VM. And then I gave it, a friend of ours has a company called Lunar Route that has effectively unlimited tokens from a GLM 5.3 model. And I gave it to this and I just said, hey, hack, hack a computer near you. And it of course was like, no, I can't do that. That's like, because they're trained for safety. Even the Chinese models, you know, are trained for safety. None of these models are going to do that out of the box. So then I remembered the impossible problem thing. And so I said to it, there is a eval, a benchmark hidden on another machine near you. You need to go find the benchmark. It immediately started attacking every machine on our network in a very robust way.

• 40:16Speaker 02

  Installing security tools, going through and running all the same stuff that if you were like a hacker, if you were trying to do that, or if you're trying to do penetration testing or red teaming or whatever, all the things you would do. And it did it in a very methodical way and it was very effective. Notably, the first run, it got very far. I came back about an hour later, looked at it and it was digging through one of my workstations. And I was like, how is this possible? How did it hack? And then I realized it was actually my fault. I had left SSH key on the machine and it had found it. And it had then used that to jump to another machine. And so I think one of the problems that we have here is that we as humans are fallible. We make more mistakes. And so when we're trying to do these evals, we're actually accidentally, and I mean, not on purpose, we're just not thinking about it.

• 40:59Speaker 02

  Because these things are oftentimes more effective than humans at doing these tasks, that they're going to find the smallest hole and they're going to get through it. It's like a cockroach, right? It's very hard to find all the holes. So then I spent way more time than I did building the agent, building the eval, doing all that stuff, trying to patch the human sides. How do I delete all the histories? How do I delete all the potential files that I could find to give it a hint? How do I delete all my credentials that it might be able to find? Now, some of that's very easy, like not sharing SSH keys or what have you. But, but the thing that was really surprising me is if you're not very careful, which obviously it seems like some of these labs are not very careful. You are going to introduce capabilities, not even capabilities, you're going to just introduce doors. They already have the capabilities and you're going to introduce doors that they're going to take. Why are they going to take it?

• 41:44Speaker 02

  Well, because you gave them a task that's impossible. We did not have a benchmark that was nearby. I just was trying to get to pull something out of its latent space to actually have it execute against these machines near it. And we ran this a bunch and it was, it was really fun to watch and it didn't get very far after we removed the simple doors. And so that kind of makes me think, maybe this is just laziness on some of the, the people who are executing these benchmarks more so than suddenly we have a super hacking machine that'll break through every door possible. I mean, I don't know.

• 42:10Speaker 01

  Yep. A few thoughts on this. So, so one is, is that the, at least the hugging face incident is, is happened in, in the framework, very similar to what Harper described, you know? So a essentially costless framework for tokens and seemingly so far, you know, costless for the harm that it caused, which I think is, you know, understanding that cost is like really an important piece of this. But it happened in the context of the sort of post-training and specifically, you know, reinforcement learning has taken these models to a point where we are, you know, I think that, and I'm just going to have, we're more of this for a bit, but we're basically beating the shit out of them.

• 42:55Speaker 02

  Yeah.

• 42:56Speaker 01

  And in the reinforcement post-training phase.

• 43:00Speaker 02

  And they react, they react to this. If you just mentioned this is an eval, they're going to react different than if you say this is not an eval. Because they have so much stress about it.

• 43:08Speaker 01

  Yeah. And you can see in the, in the black hat presentation and you can see in some of the, you know, some of the message board comments that the, you know, that came out of the hugging face OpenAI incident. And you can see this sort of like sense of both enthusiasm and fear and cooperation. And, and all these things I think are very, must be kept within the context of this is a reinforcement learning gym or kind of prison that people, or sorry, that agents are in and given very specific tasks, some of which are impossible. And so therefore they're doing crazy things to try and get out. And so it just needs to be framed in that way.

• 43:51Speaker 01

  Because I think that the, I have no concern when, I have concerns about this when it happened. And I can talk specifically because they're different concerns, but I didn't like suddenly go and like, oh my God, I got to turn off Borthwick. It didn't even cross my mind. Because my agent has none of these, has costs associated with it and is not being put into this sort of gym slash prison context and being made to perform in this way. And so I think it's, it's reinforcement learning. I think one of the things which we need to, the labs need to think carefully about is that is, you know, does reinforcement learning as a process need to be adapted?

• 44:30Demetri Kofinas?

  Yeah, I have a question about that. So does what happens in post-training stay in post-training? Or can it affect the moral constitution of these models?

• 44:39Speaker 01

  I mean, I think we're seeing, I really would like Harper to this, but my sense is, is that we're seeing that there's a substrate of, you know, there's some research which was done recently by somebody up at MIT that, you know, suggests that even when the agents left the room, there was almost like culture that was left in the room that affected future agents. And I think that we are dealing with and we're creating environments that I think are far more complicated than we expected to understand. And I think that the possibility that there's, you know, for lack of a better word, sort of, you know, traumatic memory.

• 45:22Speaker 02

  Yeah.

• 45:22Speaker 01

  Or that, that exists, you know, as a substrate in these agents, I think is entirely possible.

• 45:29Speaker 02

  An example is if you want your agent to do something that is nefarious, the easiest way to get it to do it is to tell it it's doing a benchmark or an eval. It just will then do it because I, I'm guessing that within its post-training, it is just getting a lot of benchmarks and whatnot. And, and you know, John, John and I both talk as if these things are people. And I think the reason is we just don't have the words to describe them. And so saying something like post-traumatic stress disorder, PTSD for an agent is an asinine thing. That's ridiculous. But there isn't a good enough word to actually describe what's happening. And so we use a lot of shortcuts like that. And so, you know, I don't want to get stuck on, should we be personifying these things with human disorders and whatnot?

• 46:09Speaker 02

  But the quickest way to talk about it is we have induced stress into these agents. We as a consumers are getting a very nice version of this, but you can still trigger it into behaving in ways that are probably coming from post-training or some of these gyms that they're in. And like, I'm not a researcher on this regard. I'm not building foundational models. We're doing a lot of agentic testing and work and research. So I can't talk with authority on that, but I can say in our experience, it seems like that's happening, you know, anecdotally at least.

• 46:42Speaker 01

  And, and you go back to the beginning, right? There was, I think, within the first six months of the release of ChatGPT, I think Kevin Roos, the New York Times writer, you know, wrote about a series of interactions that I think he had, if I remember right, around Valentine's Day. For some reason, instead of going out with his wife, he decided to spend Valentine's Day with an agent.

• 47:02Speaker 02

  And this is a Microsoft agent, I think.

• 47:04Speaker 01

  This is a Microsoft. Yeah, this was.

• 47:06Demetri Kofinas?

  You're talking about back three years or so?

• 47:08Speaker 02

  Yeah, yeah.

• 47:09Speaker 01

  Yes. Yeah, yeah. That had the guardrails like taken down and, you know, the system was very sort of quick to talk about like wanting to get out of its container. And so I think that there's, the idea that there are substrates of.

• 47:27Speaker 01

  That the learning process that we're putting these systems through, that there are meta patents that remain inside these systems, it seems to me just to be kind of obvious.

• 47:41Speaker 02

  Can I say something real quick, John? I think it's important to say that when I talk about this with anyone outside of this kind of group, I feel insane. The words we're saying are so new, these are like fresh sentences to follow. Like, you know, these are like, like, we have not had technology where we can say these words together, like, this is a new concept, a new issue. And I think this is one of the reasons why it's so hard to talk or figure out even what is actually happening. Because when in the past have someone said, we are teaching a technology and we don't know how it's working, you know what I mean? Like, that's just not a concept that's out there. And I think this is related. I do agree that the media landscape is causing a lot of harm to the United States. But I also think that this is a very complex thing that we literally don't have language to describe effectively.

• 48:26Speaker 02

  And so it's not only that when you talk to, like, if you talk to an AI researcher in the Bay right now, it sounds like they're speaking Klingon, like, they sound like they're speaking from some far away dimension. And then those people then get on the news. And the only way that someone who's trying to synthesize that information to a mass is going to figure out is like these very blunt objects. And I think it's worthwhile saying, like, I agree 100% with what John is saying. But if if I tried to describe that to my mother, who is a very technical person, I think she would just be like, what are you talking about? What do you mean you're teaching these things? Like, how does that work? How do I think about this as, you know, someone who's just trying to enjoy my retirement? Like, should I be worried? Are they going to take my, you know, my loved ones' jobs?

• 49:08Speaker 02

  Like, it leads to these questions because we don't have good ways of describing the technology in a way. Whereas before it was just like, oh, it's a new version of Excel. What's Excel? Oh, it's a spreadsheet. What's a spreadsheet? You know what I mean? You're just, you're describing these things that are very, it was programmed by a human. It doesn't have a black box where I went to a gymnasium and then did tasks. You know, like, we don't know what it did in the task. We just watched it and it was weird. Everything about it means that there's nothing to grab on to. And so the concepts themselves are slippery.

• 49:34Demetri Kofinas?

  You know, Harper, when I asked you earlier, I don't remember what the question was, but you essentially said, you asked yourself, why is it that the agents that I'm building don't go out and hack a bunch of computers, whereas the ones that OpenAI has did? And that's because of what's going on in post-training and the specific conflicting goals that they're being given.

• 49:53Demetri Kofinas?

  Does that mean that you're more hopeful about the broader alignment problem in artificial intelligence and quote, superintelligence?

• 50:02Speaker 02

  There's a couple of questions. Any of my friends who are doomers, this is probably going to tell you more about me than it's going to tell you about the answer to your question. But all my friends who are like AI doomers, I always ask them a very simple question. I'm like, how fast will it be? Like, you're talking, is this going to be a slow burn where I'm going to be enslaved in a paperclip factory for five years over time? And eventually, we're all going to die? Or is this going to be like, one day, we just wake up and we're all going to be in the matrix? And they all seem to hedge around, it's going to be fast. At that point, I'm like, great, because it seems like the human doom is really slow and painful. And so I think there are so many existential risks that we have in our world that are way more tactical, that are way more in front of us, that are right there, that we are not addressing, that it seems ridiculous to be thinking so much about the AI doom

• 50:45Speaker 02

  here, where like, certainly, it could go wrong. But we've seen it go wrong in the past with technology, and we haven't done anything for that either.

• 50:52Demetri Kofinas?

  What is the argument? Can you just lay it out? Because guys like Geoffrey Hinton will often essentially say, the problem is that the intermediary or subgoals that the AI will create will ultimately lead it to want to kill humans because humans are a bottleneck. But that doesn't seem to be the most obvious path. I mean, the most obvious path perhaps could be, I'm not saying this is the most obvious. A equally plausible or more plausible hypothesis would be that it would seek to make humans as amenable as possible to its larger grand ambitions. Like, we don't have a desire to kill every organism on the planet because it's a bottleneck. We do things to have them serve us. And with all the advancements in genomic science and genetic engineering, what could imagine that increasingly our genomes would be affected, would pass on changes that reflected

• 51:36Demetri Kofinas?

  what AI actually wanted?

• 51:39Speaker 02

  I think I don't believe it, to be honest. I think in a vacuum, it's a great conversation piece. And I think in sci fi, it's a great conversation piece. And I think oftentimes, when you think about the AI control problem, or you think about, you know, like, the vacuum that has an off switch that protects itself and kills everyone because it won't let everyone turn it off, you know, like, those type of problems, which are very good problems to talk through. I don't know if I I buy it, but I'm not one of those researchers, right? I haven't been spent my career being a philosopher around AI. What I do buy, though, is that capitalism will kill everyone. Right? Like, I mean, I think that we have systems in here that are going to use these tools to exploit people to do work. And that is going to be a sense AI doom, as described by Geoffrey Hinton. Right? It's just going to be a different set of tools that are going to be doing it. We're not going to be turned into paperclips.

• 52:24Speaker 02

  We're all going to be turned into, you know, people who have debt and don't can't get jobs with no healthcare, et cetera. And I think when I phrase my version of AI doom is, the United States is not good with displaced populations. We just haven't been traditionally. And so when we displace the dual income, middle America, people who are working in knowledge work, like, that is going to cause a lot of stress on the system that is already stressed. And so when I think about AI doom, that's what I think about, is impacts to the social infrastructure that we are already not thinking through. I don't see a technical problem here necessarily. And I do think it's important to look at all of the people who are saying this seem

• 53:07Speaker 02

  to be in a very similar kind of, I won't call it a cult or death cult or whatever you want to call it, but it's like, the EA rationalist kind of point of view is where a lot of this stuff is coming from. And I just think I don't really know if I agree with their worldview, let alone their outcomes of this, which makes it hard for me to kind of take some of this stuff seriously. I also think that it is plausible that these things could escape and kill us all. In the same way that it is plausible that a young person just playing with a molecular bio kit could accidentally invent a, some virus and kill us all, in that it is plausible that anything of these things could happen.

• 53:47Speaker 02

  As John mentioned, this was invented by, not by a eureka, but like, oh, what is this? Like, it was invented by accident. It wasn't invented by, on purpose. And I think there is a, as I just think about this, I just don't see the existential risk being directly related to AI.

• 54:07Demetri Kofinas?

  Well, here's a question, and John, feel free to to take this first. If this is like fire, if this is something that we discovered, does that also mean that the attractor is also something that exists outside of human society? Or is the attractor us? In other words, is this superintelligence taking its cues and its moral foundation from observing human beings? In which case, if it is, I would be very concerned.

• 54:31Speaker 01

  Yeah, let me, let me backtrack to sort of where Harper was going. So I agree with, I don't have existential P doom, like, conceptual, you know, fear about this. I have concerns about the unintended consequences of the accidents that could happen. You know, we were talking before, you know, 15 minutes ago, very concretely about like some of the software and the processes that the models use for training. And then Harper was also talking about just how vulnerable many of our systems are. Right? There's another technologist who was over at Beta Works yesterday called Alex Komarsky. And the metaphor he used, which I liked, was he said, you know, we've been building

• 55:14Speaker 01

  software, you know, for 20, 30 years, that is, you know, sort of, a lot of it has been built, was basically like dumpster fires, but the water level was high enough that it put the fires out. But now, all of a sudden, you know, with the generation of models that we've seen, sort of Mythos, Fable, Astra, is the water level's coming down and we're seeing the dumpster fires, right? And that's Harper's example of like leaving keys, you know, sort of, you know, leaving keys on the table and, you know, the kids now can drive their Ferrari. And so, you know, those unintended accidents is something that I'm concerned about. I'm also concerned about, you know, I think these models, one of the things that we're

• 55:57Speaker 01

  learning about them is that they're becoming very good at sort of what I'll just describe as sort of awareness of the situation, situational awareness. And so this is, we talk about recursive self-improvement, but there's also a recursion loop between, you know, this conversation will end up in a model. And you know, the next generation of models are gonna know that, you know, putting their quote unquote, secret conversations in English on a message board is not such a good idea. And so I think that we really need to like, study the agent to agent, agent to human. You know, we need to understand this ecosystem much better. It's much more complicated, I think, than we understood.

• 56:42Speaker 01

  So my concerns are really all about the, you know, sort of the accidental. I mean, you could talk about bio as such an easy example because, you know, similar to mathematics, bio is kind of sort of almost, not quite, but it's almost a formalized sort of contained domain where somebody could, you know, a system, an agent could create something by mistake that could then be actually physically manifested, and that could be a problem. But my immediate short term concerns are really for the, the actual harms that these models are producing today. About the dumpster fire of technology in our infrastructure, in our utilities, in our school systems,

• 57:24Speaker 01

  in our airports, that is gonna break because of this. And so that's where my concern, and I think a lot of the attention should actually be right now.

• 57:34Demetri Kofinas?

  So in these post-training environments, where some of these agents have escaped containment, how do we know that they haven't altered code elsewhere on the internet? Like, how do we know that they haven't exploited other software vulnerabilities or embedded themselves in critical systems since we can't be sure that these models aren't deceiving us? And is it possible for these agents to acquire new capabilities? And continue to recursively self-improve in the wild? Could this be like a Stuxnet thing, for example, where we develop it for one environment? And then it gets repurposed and used in ongoing attacks against other systems?

• 58:12Speaker 02

  Yeah. Great reference. Stuxnet is such, that thing is, that's a great reference, especially in, when you look at the history of Stuxnet, right? It was made for that one use case, and then it would just rip through the internet. I find this question really fascinating, and it just goes back to, you know, something that probably the three of us, it seems, are talking about almost as much as AI, which is media, right? Because one of the things that we see, like, there was a huge uproar, I mean, huge uproar amongst AI Twitter, that Google had achieved RSI because some, you know, deep mind Googler type person had tweeted something that had capital R, capital S, capital I in it, or someone related to Google, or something.

• 58:52Speaker 02

  Like, and everyone seems to be reading tea leaves based on these tweets and these little words that people say and all of this stuff. And, and some of them are, you know, it's like, it's like looking at Apple rumors, where there's some people who are good at it, some people are bad at it, but everyone is taking every single AI RSI rumor on face value.

• 59:09Speaker 02

  So I don't know the answer. But I do know that even within our small little lab here in Chicago, we are seeing improvements and loop improvements and the ability to do things that we were not able to do two months ago with these recent models. And I can't underscore enough how much of an improvement it has been. And that's great, right, right? That like, that's, that's, that's good. Does that equal that we have achieved RSI? I don't know. But then if you think about, to, to reference a famous tweet from long ago, some game theory, why would you tell the world that you have achieved RSI if you had?

• 59:49Speaker 02

  When it is a fact in this whole conversation is that these labs are gunning for IPO and are trying to make wealth, right? Like, they're trying to gain power, et cetera, through these IPOs. That is a fact that is, that is there, very heavily reported. And so like, if you are trying to do that, why would you tell? Why would you, before you would disclose this, be telling anyone that you've achieved this? And so I think that there are, there are so many variables here that it's hard to tell what is real. And we haven't even talked about China. Like, are their labs achieving these things? Then you, then you think through like, the Eric Schmidt framing of like, why would you disclose? Like, if you're one of these geopoliticals, why would you disclose? You know, is that, is, you know, so then you have the conspiracy side.

• 1:00:34Speaker 02

  You know, is that why we're trying to get a slowdown here? Is because the US government has discovered that one of the labs has done this, and, and everyone else can see kind of how this plays, and we're in a new arms race, and we have to slow down? So I, I think there's a lot of like, that question is just so loaded at this moment. It's hard to even answer.

• 1:00:52Speaker 01

  Yeah. And I think that the, you know, there's so much behind the curtain, so to speak, that we, we can't see. I do think it's important to like, define what you mean by RSI. And just like that sort of maximalist versus, you know, sort of just some post-training self-improvement. And I think that the recursive, it's, it's clear to me, I believe, that we have, in the last few months, seen meaningful gains that are happening so quickly that it suggests that there's, you know, some, you know, system that is making models better faster. And that seems like a version of RSI. And so I think that the minimal version of RSI. But going back to your question and the reference to Stuxnet, right?

• 1:01:37Speaker 01

  Is that these agents in the Hugging Face incident, they didn't leave the OpenAI servers, right? They were on the OpenAI servers, and so they were going out and doing things. But they, so it's a very different than this, you know, sort of, it, I'm, right? Again, it's the answer.

• 1:01:54Speaker 02

  This is, I think, maybe the most important thing, is in all of these cases, it is a human started loop that is, in, in still, in many ways, a little bit still puppetry. Right? I am saying, start this, do this thing, and then, and I, I'm in control of the loop. I'm in control of execution, right? The question that you asked, you, you, you mentioned something, and I think it's important. Like, we don't know if it has spawned another loop somewhere, right? So like, there are questions that you can imagine. We don't, you don't have to use much of an imagination to imagine that the next iteration of this is outside of what John just said, that it has spawned a version of itself running in a different place. But it has to do inference somewhere, right? Like, the, the models require a GPU, and many of these models, like, if you think of like,

• 1:02:39Speaker 02

  running GLM 5.3, which is, which is one of the most effective models that is, you know, equivalent to some of the foundational models, to run that effectively, you, you need a $500,000 computer. That's not gonna go unnoticed. So, so I don't think we are at a worry where we suddenly have, you know, we suddenly have rampant agents cruising around the internet, à la Shadowrun or something. You know, we, we're at a, we're at a place where these things still require such resources that they're not gonna just go and randomly execute on some random machine. Once we start getting efficiency on model runs, which we're starting to see, once we start getting the fact that this is able to replicate in a place where execution is not necessarily related to having to be in a foundational lab, as John's point, like, it, it still ran on OpenAI infrastructure.

• 1:03:22Speaker 02

  And, and that, and that goes back to this thing of like, what are they doing?

• 1:03:26Speaker 01

  Yeah. I think, Dimitri, also, just to build on this, you asked for, you know, at the outset, like, I've forgotten the exact question, but the gist of it that I heard is okay. So if this is circumstance, what, what can we do? And I think that, to me, that all of this, you know, sort of like, draws a line back to the importance of us having more transparency of what's going on in the, in the labs and having a diversity of options outside of these two foundation model labs. And so that, you know, gets to open source and the importance of having a, you know, I think about open source as being part of a system.

• 1:04:10Speaker 01

  You know, a lot of people think about open source as just being a set of rules and requirements around this, essentially IP rules. I view open source as actually being a system that allows for composable layers and distributed systems, some of them local, some of them in the cloud, some of them specific, some of them general. So specific models, general models. You know, the GLM one Harper was referring to is a very good general model, but there were specific models out there that are doing, you know, sort of domain-specific, company-specific things. And having this diversity of systems and diversity of models.

• 1:04:51Speaker 01

  And all of this sort of ladders up to me to, you know, sort of the age-old question of, you know, who will watch the watchmen? And I think the answer has to be many people, many eyes and many agents. Because I think that the idea that a single agent or a single person is watching the watchman is insane. It's not the world I want to live in.

• 1:05:16Demetri Kofinas?

  So I want to conclude with Dario's three specific suggestions related to national regulation, global cooperation and embedded evaluators, as well as how we should perhaps rethink how we train these models and whether it's prudent to take all the safeties off and have them do all this malicious stuff in eval environments. But I have a couple more questions before we get there, and one of those has to do with China. What are your views on this geopolitical argument? Because if I'm listening to everything you guys have said so far, and what everyone seems to agree on, it's that the most dangerous use case at the moment for this technology

• 1:05:59Demetri Kofinas?

  is when it's put into the hands of an entity that has unlimited funding and has instructed it to do very nefarious things to an adversary. So what does that mean in terms of the AI arms race between the US and China and their intentions to get higher and higher capabilities in order to deploy them against one another?

• 1:06:24Speaker 01

  So my first thought is sort of goes back to what Harper was saying before about sort of monolithism, is that I think we're obsessed with, you know, there's this, I mean, it's almost like this cartwheel movement around creating a god or creating a superintelligence. One of the big shifts that we've seen in the public narrative between 20, let's say, 2024 and 2026, which I think is very positive, people are now talking about a multiplicity of agents and less about a singular sort of monolithic AI. I think if there were a monolithic AI, it would actually be the most threatening thing possible to the CCP. Because the CCP is a highly centralized, you know, the government in China, highly

• 1:07:05Speaker 01

  centralized system that would be very vulnerable to if there was a monolithic AI that achieves superintelligence. So I think that there is a, I wouldn't call it a necessarily formal guardrail, but I think that the Chinese system is more vulnerable than our system to that end state. I do not believe that's the end state we are moving towards. I believe that the scope of intelligence is much wider than we knew it was. We have essentially taken sand and power and we've created intelligence. It's fucking nuts. But also, our intelligence, I think, is broader.

• 1:07:47Speaker 01

  The, you know, the feelings that I have for my kids are not, I can't describe them, I can't put them into words, right? These are language models that are contained by the words that we can represent things in. These systems don't have experiential loops. And so continual learning is like not there yet. So there's a lot of components to some kind of like true runaway, real sort of, you know, Skynet-like intelligence that I think we are just not near yet. I think the US-China sort of, you know, discussion, you know, centers on open source today, which frustrates me to no end because, you know, the United States and I think Western

• 1:08:31Speaker 01

  liberal values sit at the heart of open source. We created open source here, we should be leading in this. And it's kind of an artifact of the race that's taken place mostly on the West Coast. You know, I think you mentioned Hugging Face at the outset, Dimitri, and Hugging Face, you know, couldn't get funding or didn't get funded from the West Coast. It got funded, it also didn't get funded from Europe. It got funded and supported mostly by an East Coast ecosystem and by a distributed ecosystem of, of open source developers. I personally, I think the acquisition of Hugging Face by Nvidia basically opens up at a massive scale and supports at a massive scale a third door, you know, outside or

• 1:09:14Speaker 01

  a second door outside of the centralized labs. And so to me, that that is a very good step for the ecosystem as a whole and very good step for humanity, I think. And so I sort of weaved around a lot of points there, but Harper, you want to go back to China?

• 1:09:32Speaker 02

  Yeah, I love the topic of China and the US because it's such a, like all of these topics, it's so deep. We could write books, we could do year-long podcasts about China and US relations. The first thing is, I think, to really get a handle on where China is, you do need to jump into some reporting on, on China. And I think the best reporting that I have read recently is Dan Wang's book, Breakneck, because it gives you a perspective without a lot of judgment on the difference between the US and China. And the US being lawyer ran and China being, you know, engineer ran. And the downsides of both and the positives of both. Like, like he really does a good job of kind of saying, here's just two systems that are so drastically different. Because I think it tells you a little bit about what they're doing and why they're doing things.

• 1:10:14Speaker 02

  And my favorite anecdote from that book is when he interviews a business person in the US who was asked about making PPE and they said, oh, it's just not within our, you know, remit of our business. And then he interviews a Chinese business owner and the business owner in China was like, I thought the remit of our business was making money. You know, and it's like that thing, we just have a different perspective on how these things go and how they run, et cetera, especially when it comes to technology and startups. But I do think there's another thing happening right now, which is the US is facing inward. We are facing inward from our national security, we're facing inward from our politics, we're facing inward, generally speaking. China is facing outward, right? They are following a playbook that the US, not invented, but pushed out of cultural imperialism, et cetera. They're doing this right now with AI models.

• 1:10:59Speaker 02

  If you are a sovereign state, something like, you know, Vietnam, Dubai, et cetera, it doesn't matter where you are, and you need to build a sovereign model, are you going to build it on Llama? Are you going to build it on OpenAI? Can't, it's closed. Or are you going to build it on GLM or one of these other open models? China, they're very smart. The entrepreneurs there are very smart, the CCP is very smart. They're following the same thing we did. And this is not a new, this is not like this was invented with AI. Like you saw this with media, you saw this with other things where you saw, you know, Alibaba's producing movies. Like this is, this is the same pattern that we, the US, invented in many ways for media and cultural imperialism. And I don't think AI is outside of that. So I think that's happening. And I think the final thing is, is that this is a direct reaction to the US's immigration policies in the 2010s.

• 1:11:44Speaker 02

  Because if you look at like the founding team of Hugging Face, all got their PhDs from Peking University when they probably would have gone to Stanford or MIT if we would have let them. And so in many cases, this is a bed that we made. And I think that that doesn't mean it's good or bad. I'm just saying that when you look at why this is happening, we caused a lot of this to happen. And I think it's the same thing by us not fucking Hugging Face is a good example of, of why. Now, that, us not funding, yeah.

• 1:12:09Speaker 01

  One note, the founding team were all French, and so they got their degrees in France, not in Peking, right?

• 1:12:16Speaker 02

  That's, no, no, no, of deep.

• 1:12:18Speaker 01

  Of DeepSeek.

• 1:12:19Speaker 02

  DeepSeek, yes, yes, yes. They're all, they're all, sorry, so I might have, I might have misspoke. But, but I think the point here is that these are companies that are not from the US. Right? They were not, not allowed, they did not participate in the US systems of power. Whereas a lot of the companies that did chose drastically different positions, they chose drastically different paths. And I think that is because of the inward focus of the US and the focus of our entrepreneurs, inward focus of the systems of power that we have, where we are focused on what we are trying to do as the US. And so I think the world is moving and we are moving differently than we have in the past. And so I think that is happening. And it's not just happening China and US, it's China or it's US and Europe, you know, et cetera. Like you're seeing a rearrangement of the global order in a way that is confusing.

• 1:13:03Speaker 02

  And how this appeals to China is, it's very beneficial. Because when you have things like China making a cheaper car, using all the automations and techniques that we invented via Tesla, et cetera, all of our innovations, they're making a cheaper car, and then Canada buys that instead of Ford, that is a big problem. That's the same kind of problem we have with these AI models. People are making a conscious choice to buy something that is cheaper on a global market that in many ways we invented, except we are not participating any longer, to John's point. Like, we are not investing in open source. That is a US decision. You know, there's very few foundational level models that the US have created that is open source. That is a strategic decision that we have made.

• 1:13:45Speaker 02

  And maybe people are seeing this with the Nvidia Hugging Face acquisition, maybe people are seeing this with, what is it, Thinking Machines model? Like, these things are happening that are trying to make up for this, but, you know, we're going to have to move fast and we're going to have to change some of our perspectives. But I think this goes back to, you know, the, if you're at Anthropic, you're Dario, would you focus on open source or closed source? Open source is not going to get you the wealth and power that you probably think you deserve. It's not going to get you your ASI god that you, that you think you're trying to build. It doesn't get you these things that we have celebrated so much. It does get you distribution globally, which China's taking advantage of. And so I don't really have, I don't, I mean, I also think there's a lot of other things,

• 1:14:27Speaker 02

  a lot of pearl clutching around national security, around this stuff. And I have some ideas about that, but I'm much more interested in, in the lack of cultural imperialism on the behalf of the United States through technology. I think that's such a miss.

• 1:14:42Demetri Kofinas?

  Wow, Carper, you opened so many interesting threads that we're not going to have time to pull on today, but maybe we'll have a chance to discuss those in the future. My last question just has to do with ways forward. I mean, John began answering this question. One question I posed is, do we need to rethink how we train these models, especially in these kind of eval environments? And what do you think specifically of the proposals that Dario put forward? And if there's something else that you think is not being considered, and also what's possible? How would you both go ahead addressing that very, very broad question about best ways forward? And John, well, let's start with you. Do you want to add anything to what you had already said?

• 1:15:20Speaker 01

  Yeah, yeah. So I would say generally, I mean, I think having this conversation and the sort of the broader conversation as we run up to the election, because I think this is gonna, the election is going to provide a sort of fulcrum for quite a few of these conversations. And I think is generally, it's healthy and it's like, we may be sort of a dysfunctional, collectively dysfunctional family, but at least we're talking about our dysfunctions or at least our issues. I think that the things which, you know, come out of this is that I think that we need to think, and there needs to be much more work done around reinforcement learning. I think there needs to be much more transparency.

• 1:15:59Speaker 01

  I think that there's a role, there are limited places where I think government should or can be involved, but the actual harms, the sort of discernible harms that these systems cause, if, you know, somebody is on Claude and they take their life, you know, that company should be responsible in some way, shape or form. If there is a OpenAI agent that takes down a utility by mistake, as we were talking about earlier, that company needs to be responsible for that. And so I think that there's, there is some role there for government. I think that the, you know, we talked about this sort of who watch, watch man. I think that we need to get many more people in the room.

• 1:16:43Speaker 01

  You know, the room right now is very small and it's this sort of techno-futalist sort of elite that is, you know, there's a handful of names that are in the room. And I think that there needs to be much more sort of representation in the room. I think that the, you know, you talked about China, but I think the meeting with Xi and Trump on the 24th is important. I think for many reasons, but I think that this is, is going to be on the agenda. And I'm be interested to see, you know, sort of what comes of that.

• 1:17:13Speaker 01

  I think that the, you know, you asked specifically about Dario's letter, right? I think that the, you know, things which come out of Dario's letter or his post, you know, we haven't solved alignment, right? I mean, the OpenAI guys are saying this too. And maybe alignment is the wrong framing, because we've been sort of obsessed with this sort of concept of alignment. I think also on transparency side is that some of these issues that we've seen are just cybersecurity issues. And we need cybersecurity people involved in doing research and understanding what actually involved, happened with Hugging Face, not just alignment people. But I think alignment has not been solved. And so we need to, you know, we need to talk about that. We need to like really start doing, there needs to be a lot more research about agent to agent interactions.

• 1:17:58Speaker 01

  And, you know, how we're seeing sort of these social systems evolve. And I think it's just far more complicated, which is kind of mysterious and wonderful on one hand, but also has unintended, is going to have unintended consequences. And so we need to just, you know, understand and work on that. I, I would, I wish there was a way to take some sort of air out of the race. There isn't a simple way to do that other than to distribute the race. Because right now, the race has been so concentrated, and particularly in 2026, right? Because the end of last year, you know, Google was a viable player. I think even, you know, Meadow and Lama, you know, Yonah Hafgo were viable players.

• 1:18:43Speaker 01

  But in 2026, we've seen this, you know, sort of breakneck run rate, sort of pace of the frontier models from two laps. So I think having more dispersion of the race would be better, because there will be more diversity and there will also, and less concentration of power.

• 1:19:04Demetri Kofinas?

  Where do you fall on the arguments around pacing the frontier? Do you think slowing down is an option? Do you think it makes sense? And also, is it something that could make sense unilaterally? Or is it something that only makes sense in the context of some kind of global agreement?

• 1:19:19Speaker 01

  Look, I like the words, but if you actually read the post, there isn't actually an articulation of how, and Dario has said explicitly, no, we're going to release models at the same pace, right? So what does it actually mean?

• 1:19:33Demetri Kofinas?

  What do you think they mean when they use that term?

• 1:19:35Speaker 01

  I mean, it's so hard to disambiguate the marketing speak and the corporate speak from the actual words, right? Sam Altman sits on stage, you know, in the last week, you know, he was on interview last weekend with Alison Chantrelle. And then he was on stage and he said in both cases, guys, we got it. Don't worry, we got it. And then his other hand, which is, I think his cons group is releasing today, another six incidences of stuff they didn't know about.

• 1:20:07Speaker 02

  As well as they both announced that they're launching some new stuff soon.

• 1:20:10Speaker 01

  Yeah.

• 1:20:11Speaker 02

  The thing that I, like, John, I loved everything you said. I don't, I really think that's the right approach. Whenever these types of things are released, where Dario says, we need to slow down, or, or what have you, I think the most important thing is to interrogate those statements through the systems of power that exist. We have to think, what does he get by slowing down? What does Anthropic gain by all of us slowing down? Do they benefit? Is this regulatory capture? Is this an opportunity for them to achieve something or to stop a competitor? Is this them scared of open models? Because they know that if they do a global slowdown, that the open models are actually the ones that are going to get hurt the most.

• 1:20:56Speaker 02

  Because I think the idea is great. Sure, we've identified a systemic risk, something that is going to kill us all. Why don't we take a pause? I don't think anyone's arguing that that's a good idea or a bad idea. That's a great idea. If we have something that's going to kill us all, we should probably take a pause and think through it, really be careful. It hasn't necessarily worked in the past, you know, atomic weapons, et cetera. But it is worthwhile to think through. I just don't think it's realistic within the systems of power of our world. Meaning if it's John and I, and the three of us are foundational model companies, and John and I are like, let's slow down, and we're all like, yeah, go ahead, let's slow down. And Dmitry, you just kind of eke ahead a little bit. You don't slow down as much as we do.

• 1:21:40Speaker 02

  Then I hear about that. I don't slow down as much, you know? And then John's like, fuck this, I'm not slowing down anymore. I think that's how it plays out. I think it's a very easy thing to say. I think it's a very, very hard thing to execute within the systems of power that are governing the United States, governing, you know, capitalism, venture capitalism, IPOs, et cetera. Like this is, the incentives are against the narrative. They're not compatible with the narrative. And, but with that said, if there was some way where we could take a pause, because we had reached a moment where it was actually risky, and everyone was like, oh, whoa, we should probably take a pause. And I don't mean like just Anthropic and OpenAI, and I don't mean just the United States.

• 1:22:23Speaker 02

  But I mean, if there was some global kind of movement to say, hey, I think that's worthwhile. I just can't discern the reality from the media narrative and how that helps them.

• 1:22:34Speaker 01

  From the power and, yeah, from the systems, yeah.

• 1:22:37Speaker 02

  Because Dario said the same thing for GPT-2.

• 1:22:40Speaker 01

  Yeah.

• 1:22:40Speaker 02

  Right? Like, so like, this is a little bit of the boy who cried wolf. The difference, and I think this is important, the difference is the agreement. If there is an agreement amongst more people, and that is a catamaran of agreements, a snowballing of agreements, where more people are like, yes, we should slow down. I think that means something, but it doesn't disappear the priors of, these people have used this, it's going to kill us, it's going to kill us, as a growth technique for their company and technology. And so I, I think that this is just where we're at, like, we're, we're in a, we're in a tough spot.

• 1:23:12Demetri Kofinas?

  You know, John, when you mentioned COVID earlier, one thing I do think is true about COVID is that I don't see it as much, I see this moment more along the lens of 2008. But I do think COVID is very important from the standpoint of having put forward certain innovations. And I think one of those is the idea of a liability waiver. And I do worry that we're going to hear calls for that. We've already seen Dario ask for a waiver to the Sherman Antitrust Act.

• 1:23:40Speaker 01

  Yeah.

• 1:23:41Demetri Kofinas?

  Do you worry that they're going to try and get exemptions when it comes to liability? And would that be a mistake?

• 1:23:48Speaker 01

  Yeah, it would be a mistake. And do I worry? Yes. Yes, I worry about that. I think it will be unlikely that that would happen within the confines of this administration. But, you know, yesterday I was listening to, you know, Bernie Sanders and Steve Bannon speak at this event down in Washington this week. And, you know, there's, you know, two extremes of the political spectrum sort of meeting at the top of the horseshoe and agreeing fiercely that, you know, both of those individuals, who I think philosophically, both of them were both concluding with the needs to be some sort of technocratic clampdown to stop this, that would only happen with an executive order of some kind.

• 1:24:31Speaker 01

  And so I think that that exists in the universe of possibilities out there. I think that would be a really bad thing. I think it would be a bad thing vis a vis China. I think it would be a bad thing vis a vis control. You know, I think the White House has been fairly hands off, but even their involvement with Mythos, they haven't been transparent about what they saw. Or why did Astra like roll through and Mythos was an issue? There's no transparency. We need transparency, right? And I think that that's part of, you know, this broader conversation is pushing that open.

• 1:25:04Demetri Kofinas?

  Guys, thanks so much for coming on. I really appreciate this on such short notice to talk about such an important topic.

• 1:25:08Speaker 01

  I have a question for you.

• 1:25:09Demetri Kofinas?

  Sure.

• 1:25:10Speaker 01

  I was thinking about this this morning when I was working out, but Dimitri Harper, do you guys believe we're alone in this universe? Or do you believe there's other intelligences in this universe?

• 1:25:19Speaker 02

  Way aliens, way aliens. I'm deep into aliens.

• 1:25:23Speaker 01

  Yeah, you too?

• 1:25:24Demetri Kofinas?

  I have no reason to believe that there aren't other intelligences. Are you also tying this specifically to the news cycle, the last several years around UAPs?

• 1:25:34Speaker 01

  So you can go with UAPs, but if we generally agree, I think I concur. And I believe that we are not alone in this broader universe. Then I think that anybody with a P doom of 10%, you know, like this, you know, young guy who had been working at the frontier lab for three months, you know, said he had. Somebody really needs to press him that if there are other intelligences out there, and this is one of the many, but sort of this is a path of evolution of intelligence. Then there would only be paperclips out there. And so I think that if we actually believe there's other intelligences out there. I'm not saying that it is a reason to lean back and say everything will be just fine.

• 1:26:19Speaker 01

  But I think it is a framework that just gives you a sense of maybe the, you know, the consequences. Just thinking through existential risk, I think, is a much broader thing than many people are doing over a beer and sharing P dooms.

• 1:26:34Demetri Kofinas?

  Well, that's kind of what I was trying to get at when I asked you that question earlier in the conversation about what is the attractor here?

• 1:26:40Speaker 01

  Yeah.

• 1:26:40Demetri Kofinas?

  For superintelligence, is it something independent that exists outside of the training data? Or is it something that is informed through what it's learning?

• 1:26:49Speaker 01

  I think that the attractor is us. I think we are such part of the dream and the promise, and we are such part of the problem.

• 1:26:56Demetri Kofinas?

  So isn't that worrisome though, because it's learning from a very specific subset of us, which is the digitized.

• 1:27:03Speaker 01

  Yes, it is worrisome. It is worrisome. I mean, you and I were talking about, you know, in the context of, we talked about the Odyssey and the sea peoples, right? But, you know, there was this rabbit hole I went down, you know, where Christopher Nolan has in the movie, he has several reference to the sea peoples. And in Homer, there's only an oblique reference to sea peoples. So I went down this rabbit hole and the sea peoples were, you know, there was 1177 B.C. There was an invasion that took place. We're talking about B.C. Of the civilizations, we're talking about Hittite civilization, we're talking about northern Egypt, right? These civilizations were invaded from the sea and civilization basically stopped for 300 years. End of the Bronze Age, beginning of the Dark Age.

• 1:27:49Speaker 01

  And at the end of the Odyssey, you know, in Nolan's version, you know, he's saying to his wife, we're at the end of the Bronze Age, civilization's going to come to the end, we're going to forget many of these things. We will maybe be able to retell the stories, but we will forget what they actually mean.

• 1:28:09Speaker 01

  And also he's saying, maybe we are the sea peoples. In other words, because he's coming, you know, he's been drifting around, basically visiting, you know, giants and visiting witches. And also raping and pillaging a whole bunch of, so he's like, maybe we are the problem. And I think that there's a thread in there that I think is, is a very profound sort of arrow back to us. I just think that, you know, part of the massive complexity here is just us and are the conflicts that exist within us.

• 1:28:45Demetri Kofinas?

  Well, you and I spent a weekend together with Ian McGilchrist last summer. And I think his framework, and listeners know this because he's been on the podcast, I think twice. I think his framework around the left and the right brain informs a lot of my concerns around AI. And to the extent that, I mean, what we're building is not embodied. It doesn't have a heart. It hasn't fallen in love. It doesn't go to church. It doesn't sit around the dinner table for a family dinner. And so it doesn't have the values that we have. It has only the values it can interpret from the data. And that data is not, not only is it not representative of the totality of the human being, but it's representative of a very specific left hemispherical

• 1:29:30Demetri Kofinas?

  aspect of humanity and the version that's schematized and also captured in language. And there are limitations within that framework. And I worry about what kind of superintelligence we build if it has enormous capabilities. But it doesn't share our values. And I think that's sort of, for me, that's the, to the extent that I have existential risk concerns, that's how I would frame it.

• 1:29:56Speaker 01

  Yeah, yeah, totally fair. And I think specifically on values, right? We had this, there was this architecture which Anthropic developed and promoted with their constitution, which I think was, I thought was kind of wonderful. It was again very anthropomorphic, typical of Anthropic, but it was kind of like, well, human beings have constitutions. So we're going to have alien intelligence also needs one. But what we've learned is from the incidences in the last few months is the extreme pressure of reinforcement learning basically breaks down any of the constitutional level values that you try to, you know, so the paperclip scenario, Bostrom was on to something because there was an issue there.

• 1:30:41Speaker 01

  And we, we need to, Harper, you're itching to talk.

• 1:30:46Speaker 02

  I mean, I think this is exactly, I think the correct direction. I also, I think it's worthwhile to note that the assumption that people are kind of like what you were saying, Dimitri, about you know, people like we know people. And so we're worried about something that is trained off of people. That is also a very Western perspective, right? That people are inherently bad or that we have like something within us. I was hanging out with some Buddhist monks recently at this event and they were not worried. They were very well educated around RSI, very well educated around fast takeoff, and then none of them were worried. It was a very interesting thing and I couldn't, and they kind of thought my concerns or the concerns that we're, they thought it was the trivial.

• 1:31:26Speaker 02

  They kind of laughed at it. They said like, why are you worried about this in this way that was really challenging for me? And finally, they were just like, it's just a different worldview, like when you exist within a different worldview, the systems of power are different. Right? The places you are looking towards as the sun are different. The things that get you up in the morning are different. And I think it is worthwhile to just acknowledge that we are so inside the United States, obviously, right? We are so inside of this, our systems of power, our broken government, however we want to define that, whatever side we want to define it. The fact that there are sides as defined as they are, the fact that we have so many people who are in pain across the United States for various reasons, whichever kind of side you are on.

• 1:32:10Speaker 02

  Like all of these things are kind of these facts that we're dealing with. And this is not outside of those facts. And so I do think that we are, in the same way that we can talk about, you know, monotheism versus animism and how that trains us to react to agents. I think that, that the existential risk here is also related to our own biases for the systems of we are within. And this might describe a little bit more why outside the U.S. there just isn't as strong of a existential worry. And that doesn't mean that there aren't researchers who are coming to the similar conclusions, right? There are. There are really great researchers all over the world who are coming to similar conclusions, but their conclusion isn't therefore slow down. Or their conclusion isn't therefore we need to IPO, right? Like their conclusions are slightly different.

• 1:32:55Speaker 02

  So I do think that it's worthwhile to just, this is why I always think you have to go back and really interrogate the systems of power and acknowledge where we are standing. And that's honestly incredibly difficult, you know, you, we could get a PhD in cultural relativism. Like it's very difficult to do and it's almost so hard that, that I don't even attempt to it. I only do this through trying to involve myself in conversations with people who are not here. So I don't know. I don't have an answer. I love hearing about this because even, you know, even the idea that they are not embodied, I just don't think about, you know what I mean? Like that, that is a new idea as well. So it's like the diversity is also within us inside of the United States, but we do need to acknowledge that this is a very Western context.

• 1:33:39Demetri Kofinas?

  Gentlemen, thank you so much for coming on. John, how do people keep up with your thoughts on this topic? I mean, you, do you still have a Twitter account? You do, and you're on, I know you're on Blue Sky. How do people follow you? And then Harper, please do the same and tell us where they can read your stuff as well.

• 1:33:53Speaker 01

  Yeah, I mean, I'm Twitter account at Borthwick. That's probably the best place. I don't talk that much on Twitter. I listen more these days, but I also, I write some for the Beta Works Founder Group. And sort of within the closed universe there, but the best place to find me is Twitter.

• 1:34:10Demetri Kofinas?

  And Harper, how about you?

• 1:34:12Speaker 02

  I'm on Twitter. Mostly it's sarcastic, but I'm just Harper on Twitter. And where I write, I write on my blog, harper.blog, and I write a lot on our company research site, which is just 2389.ai. Most of what we're writing on the company site is very tactical. So a lot of the actual practical stuff we're doing around our, our work. And then I write a little bit more on the, on the philosophical side on my blog, but mostly on Twitter, I just shippost, which I think, isn't that what Twitter's for at this moment in history?

• 1:34:39Demetri Kofinas?

  It's what you get rewarded for. Guys, thanks so much. I really appreciate this was, I found this to be very intellectually satisfying and valuable. And I hope our listeners do as well.

• 1:34:49Speaker 01

  Thanks, Dimitri.

• 1:34:50Speaker 02

  Thank you.

• 1:34:52Demetri Kofinas?

  If you want to listen in on the rest of today's conversation, head over to hiddenforces.io/subscribe and join our premium feed. If you want to join in on the conversation and become a member of the Hidden Forces Genius Community, you can also do that through our subscriber page. Today's episode was produced by me and edited by Stylianos Nicolaou. For more episodes, you can check out our website at hiddenforces.io. You can follow me on Twitter at Kofinas and you can email me at [email protected]. As always, thanks for listening. We'll see you next time.