E26: [Bonus Episode] Connor Leahy on AGI, GPT-4, and Cognitive Emulation w/ FLI Podcast

[Bonus Episode] Future of Life Institute Podcast host Gus Docker interviews Conjecture CEO Connor Leahy to discuss GPT-4, magic, cognitive emulation, demand for human-like AI, and aligning superintelligence. You can read more about Connor's work at https://conjecture.dev Future of Life Institute is the organization that recently published an open letter calling for a six-month pause on training new AI systems. FLI was founded by Jann Tallinn who we interviewed in Episode 16 of The Cognitive Revolution. We think their podcast is excellent. They frequently interview critical thinkers in AI like Neel Nanda, Ajeya Cotra, and Connor Leahy - an episode we found particularly fascinating and is airing for our audience today. The FLI Podcast also recently interviewed Nathan Labenz for a 2-part episode: https://futureoflife.org/podcast/nathan-labenz-on-how-ai-will-transform-the-economy/ SUBSCRIBE: Future of Life Institute Podcast: Apple: https://podcasts.apple.com/us/podcast/future-of-life-institute-podcast/id1170991978 Spotify: https://open.spotify.com/show/2Op1WO3gwVwCrYHg4eoGyP -- We're hiring across the board at Turpentine and for Erik's personal team on other projects he's incubating. He's hiring a Chief of Staff, EA, Head of Special Projects, Investment Associate, and more. For a list of JDs, check out: eriktorenberg.com. RECOMMENDED PODCAST: The HR industry is at a crossroads. What will it take to construct the next generation of incredible businesses – and where can people leaders have the most business impact? Hosts Nolan Church and Kelli Dragovich have been through it all, the highs and the lows – IPOs, layoffs, executive turnover, board meetings, culture changes, and more. With a lineup of industry vets and experts, Nolan and Kelli break down the nitty-gritty details, trade offs, and dynamics of constructing high performing companies. Through unfiltered conversations that can only happen between seasoned practitioners, Kelli and Nolan dive deep into the kind of leadership-level strategy that often happens behind closed doors. Check out the first episode with the architect of Netflix’s culture deck Patty McCord. https://link.chtbl.com/hrheretics TIMESTAMPS: (00:00) Episode introduction (01:55) GPT-4 (18:30) "Magic" in machine learning (29:43) Cognitive emulations (40:00) Machine learning VS explainability (49:50) Human data = human AI? (1:01:50) Analogies for cognitive emulations (1:28:10) Demand for human-like AI (1:33:50) Aligning superintelligence SPONSORS: Shopify is the global commerce platform that helps you sell at every stage of your business. Shopify powers 10% of ALL eCommerce in the US. And Shopify's the global force behind Allbirds, Rothy's, and Brooklinen, and 1,000,000s of other entrepreneurs across 175 countries. From their all-in-one e-commerce platform, to their in-person POS system – wherever and whatever you're selling, Shopify's got you covered. With free Shopify Magic, sell more with less effort by whipping up captivating content that converts – from blog posts to product descriptions using AI. Sign up for $1/month trial period: https://shopify.com/cognitive This show is produced by Turpentine: a network of podcasts, newsletters, and more, covering technology, business, and culture — all from the perspective of industry insiders and experts. We’re launching new shows every week, and we’re looking for industry-leading sponsors — if you think that might be you and your company, email us at [email protected]. If you'd like to listen to Part 2 of this interview with Connor Leahy, you can head here: https://podcasts.apple.com/us/podcast/connor-leahy-on-the-state-of-ai-and-alignment-research/id1170991978?i=1000609972001

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to a special bonus episode of the Cognitive Revolution. As you may know, I recently had the pleasure of appearing as a guest on the Future of Life Institute podcast. The Future of Life Institute, you'll recall from our recent interview with Skype founder and influential AI investor Jan Tolan. And if you haven't heard that episode, I strongly encourage going back to listen to it. Future of Life Institute is the organization behind the recent open letter calling for a voluntary six-month pause on the larger-than-GPT-4 scale AI training runs. The Future of Life Institute podcast, which despite having known Jan for years, I will confess to never even having heard of previously, turns out, in fact, to be excellent.

• 0:43Nathan Labenz?

  Gus Docker, the host, really puts in the work necessary to deliver deeply substantive conversations. And his lineup of guests, while not exclusively AI-focused historically, is almost entirely AI of late, featuring interviews with a number of critical thinkers such as Neel Nanda and Ajay Akhatri that I also hope to interview in the future. So we're taking this moment to share the Future of Life Institute podcast with you, starting with an interview that Gus recently did with Conjecture CEO Connor Leahy. Conjecture describes itself as a team of researchers dedicated to applied, scalable AI alignment research. They also build products to help businesses improve their workflows.

• 1:25Nathan Labenz?

  Jan, who is an investor in Conjecture, described them memorably as, quote, "a team that has the highest respect for AI, a virtue that I personally also seek to cultivate." This is part one of a two-part interview. Part two is available on the Future of Life Institute podcast feed. I hope you enjoyed this conversation with Gus Docker and Conjecture CEO Connor Leahy.

• 1:47Gus Docker?

  Welcome to the Future of Life Institute podcast. I'm here with Connor Leahy. Connor is the CEO of Conjecture, and Conjecture is this company researching scalable AI alignment. So Connor, welcome to the podcast.

• 2:01Speaker 02

  I'm so glad to be back.

• 2:03Gus Docker?

  Okay, what is, what is happening with the GPT-4? What, is this the moment that AI becomes a mainstream issue?

• 2:10Speaker 02

  Christ, what a way to start out.

• 2:14Speaker 02

  It is no exaggeration to say that the last two weeks of my life have been the most... Interesting of my career um in terms of events in the wider world. You know, I I thought nothing could be G P T three like, you know, after I've seen what happened with G P T three. I'm like, OK, this is the craziest thing that's gonna happen like a short period of time. But then I quickly realized, no, that can't be true. Like, things are only gonna get crazier, and as predicted, exactly that is what has happened. And as predicted, the release of G P T four has been even crazier than G P T four, than G P T three. The world has gone even crazier, things are. Things have really changed.

• 2:58Speaker 02

  Like, I cannot overstate how much the world has changed over the last, like, not necessarily only since G P T four, but also since Chat G P T. Maybe Chat G P T was even a bigger change in like wider political thing like. I won't mince words, like, the the thing that really has struck me over, I've been talking to a lot of people recently. You know, I have journalists running down my door. I I talk to politicians and national security people and people are sort of. And the one thing that really strikes me is that people are starting to panic.

• 3:33Gus Docker?

  What so this this goes beyond Silicon Valley Twitter circles. This is this is venturing into polit- the politics and governmental agencies and so on.

• 3:44Speaker 02

  It it it goes to the point, look, when, you know, I've been doing A I for a long time. And I come from a pretty rural place in like southern Germany. And when I went back to to visit my my mother in for Christmas and, you know, all my cousins and like, you know, family were there. They talked about Chat G P T. I was there in this teeny world where there's usually no technology and I'm the only one who really knows how to, you know, really use a computer very well and whatever. And then they're talking with their, Connor, I, whoa, we thought this A I thing you were talking about, like, that was like, uh, you know. That wasn't like, sort of like some kind of thing you liked, but wow, you were right, like, this is actually happening. I'm like, yeah, yeah, big surprise.

• 4:30Speaker 02

  Um. So this is not just a thing that is in a small circle of people in tech or Silicon Valley or whatever, or whatever. This is different, like this is very different. You know, people are, we're getting, you know, front page time news coverage about this kind of stuff. We're getting, you know, people from all walks of life suddenly noticing, wait, like, uh, this is actually real, this is actually affecting me, this is actually affecting my family, my future. This is, this is not at all how things went past. In a ironic twist, it seems that the people deepest in tech are the ones who are least

• 5:12Speaker 02

  like in like rational about this or like the least deeply taking this seriously. Like, there's this meme that's been around for a long time about how like, oh, you can't explain to normal people like AI or AI risk or whatever. But, you know, maybe that was the case twenty years ago. But this is not my experience now at all anymore. I can talk to anyone on the street, share the chat GPT, explain to them and like explain AI risk. Like, hey, these people are building bigger and bigger and stronger things like this. They can't control it. Do you think this is good? And they're like, no, obviously not. What the hell are you talking about? Of course, this is bad.

• 5:48Gus Docker?

  Do you think that the advancement from GPT two to GPT three was bigger than the advancement from GPT three to GPT four? So are we, are we hitting diminishing returns?

• 6:00Speaker 02

  No.

• 6:02Gus Docker?

  Not at all?

• 6:03Speaker 02

  No, not really. It's like just as I predicted, basically. Like, this is just pretty much on track. I would say GPT four, the final version is better. So I used the GPT four alpha when it was, you know, back in like August or whatever, when that was first being passed around among people in the bay. And it was already very impressive then, but kind of like, kind of in line of what I was expecting. The release version is significantly better. Like, the, the additional work they've done to make it better at reasoning and such and the visual stuff and all that kind of stuff is significantly better than what I saw in August, which is not surprising. It's just, you know, it's all, sure, you can argue on some absolute terms.

• 6:49Speaker 02

  The absolute amount of difference between like GPT two and GPT three is obviously much larger. Also, like the, you know, the amount of like the size of the model is a much bigger difference like, you know, GPT four. From what I hear is not that is like is larger, but it's not that much larger than uh GPT three. And the thing with GPT four is is that what is very striking with GPT four, and it's not surprising, but I think it's important. Is not that it can do, you know, crazy things that are impossible to accomplish in principle with GPT three. As often the things are impressive with GPT four. It's possible to accomplish these things with GPT three with a lot of effort and error checking and rerolling and very good prompting and so ever.

• 7:33Speaker 02

  The thing that is striking with GPT four is that it's consistent. The striking is that you can ask it to do something and it will do it and it will do it very reliably. This is, you know, not just bigger model size. This is also, you know, better fine tuning, RL Chef. You know, better understanding of what users want these models to do. Like, the truth is that users don't want general purpose, you know, base model, you know. Of like, you know, large text corpuses. This is not what users really want. What they want is a thing that does things for them. This is, of course, you know, needless to say, this is also what makes these things dangerous compared to like GPT three. Like raw GPT three is, you know, very powerful and whatever.

• 8:17Speaker 02

  But raw GPT that all, you know, GPT, they can also take actions or that is, you know. Trained very, very heavily to take actions, to reason, to do things, which GPT four is. Let's be very explicit here. GPT four is not a raw base model. It is an RL trained, instruct fine tuned, extremely heavily engineered system. That is designed to solve tasks, to do things that users like. And these can, these are all kinds of different things, but let's be very clear about this. This is not a raw. Like the thing you see on the API is not a raw base model that's just, you know, just trained to model an unsupervised corpus of text. It's something that's fine tuned, that's RL Chef, and this is.

• 9:01Speaker 02

  I mean, OpenAI did a fantastic job, like, you know, on purely like technical terms. I'm like, I'm all, I'm like, wow, this is so good. Like, this is so good. This is so well made. This thing is so smart. It, um, GPT four is the first model that I personally like, I feel is delightful to use. Like, when using GPT two or three, I still kind of like pulling out my hair. I'm like, this is still like very, like, I'm not a great prompter, right? I don't really use language models for much, um, for this reason, because I found them just generally to be very frustrating to use. Um, for most of the things I I would use them for, except for, you know, very simple or silly things.

• 9:40Speaker 02

  GPT four is the first model that like, when I use it, I'm like delighted. I like, I smile, like, like the clever things it comes up with and like how delightfully easy it was to get to do something useful.

• 9:52Gus Docker?

  Yeah, and and this is mostly from the reinforcement learning from human feedback. Or i- is this, is this, is this coming from the base model, how it's trained? Or is this coming from how it's fine tuned and trained to respond to to what humans wanted to do?

• 10:07Speaker 02

  I mean, who knows, obviously, like, who knows how they did this exactly? I don't think they know. I think this is all empirical. I don't think there's a, like, to be good, there's no theory here. It's not like, ah, once you do seven point five micro alignments of RLHF, then you get what you want. No, it's just like, you just fuck around and you just have, you know, a bunch of people label a bunch of data until it looks good. And, you know, like, this is not to denigrate the, you know, probably difficult engineering work and scientific work that was done here. If I didn't think these systems were extremely dangerous, I would be in absolute awe of OpenAI. And I would love to work with them because this is an incredible feat of engineering that they have performed here. Incredible work of science. This is incredibly impressive. I I do not deny this, you know?

• 10:48Speaker 02

  The same way that if I was there doing the Trinity test, I would be like, wow, this is an impressive feast of engineering.

• 10:53Gus Docker?

  Okay, how much have we explored what GPT can do? So i- in terms of what what's there waiting to be found if we just gave it the right prompt?

• 11:03Speaker 02

  Who knows? Like, we have not scratched the surface, not even scratched the surface. It's a There's this, there's this narrative that people sometimes, you know, especially Sam Altman and such likes to say, where he's like, "Oh, we need to do incremental releases of our systems to allow people to test them so we can debug them." This is obviously bullshit. And the reason this is obviously bullshit is because if he actually believed this, then he would release, you know, GPT-3 and then wait until society has absorbed it, until, you know, our institutions have caught up, or regulation has caught up, until, you know, people have fully explored, mapped the space of what GPT-3 can and cannot do, you know, understood interpretability, and then you can release GPT-4.

• 11:45Speaker 02

  If you actually did this, I would be like, "All right, you know what? Fair enough. That's, that's totally fair. Like, I think this is a fair, responsible way of handling this technology." This is obviously not what is going on here. There is an extraordinarily funny interaction where Yan Laike, the head of alignment at OpenAI, tweeted like, "Hey, maybe we should slow down before we hook these LLMs into everything." And six days later, Sam Altman tweets, "Here's plugins for ChatGPT. Plug it into all the tools on the net." Like, the comedic timing is unparalleled. If this was in a movie, like, this would have been like, you know, like a cut, you know, and then everyone would have laughed, you know, it would have been extremely funny.

• 12:27Speaker 02

  So, we have no idea, there are, as Gour, I think it was Gourin that said this, "There is no way to prove the absence of a capability." "We do not have the ability to test what models cannot do." And as we hook them up to more tools, to more environments, we give them memory, we give them, you know, recurrence, we use them as agents, which people are now doing, you know, with like Lang-Chang and a lot of other methods for using these things as agents. Yeah, I mean, obviously, we're seeing the emergence of Proto AGI, like, obviously so. And, you know, I'm not sure if it's even gonna be Proto for much longer.

• 13:04Gus Docker?

  Talk a bit about these plugins. What, so as I understand it, these plugins allow language models to do things that they were previously bad at, like getting recent information or solving symbolic reasoning, like, like mathematics and so on. What is it that's allowed by these plugins?

• 13:21Speaker 02

  I mean, anything. So like...

• 13:25Speaker 02

  It's quite strange to me that like this has been strange to me for years. So like I looked at G P T two and I'm like, oh, well, there's the A G I. It doesn't work yet, but this is going to become A G I. And people are like, oh, no, Connor, it only, you know, predicts the next token. And I'm like, you know, I know you output so I'm like, OK, your brain only outputs neural signals. So what like, like, that's not the interesting thing. The interesting thing, like the modality, like I I often say this, I think the word large language models is kind of a misnomer. Or it's just like not a good term. The fact these models use language is completely coincidental. This is just a implementation detail. What these things really are are general cognition engines.

• 14:06Speaker 02

  They are general systems that can take in, you know, input from various modalities encoded to some kind of semantic space and do cognitive operations on it. And then, you know, I would put some kind of cognitive, you know, output out of this. That so we've seen this now with a very good example, which is an example I've been, you know, um, you know, using as a hypothetical for a long time. Is you chat, you know, G P T four allowing visual input. And this maps it into the same internal representation space, whether it's an image or text, and they can do the same kind of cognitive operations on. This is the same way the human brain works, you know, your retina or your ears or whatever, you know, map various forms of stimuli into a common representation of neural spike trains. These are taking this input and then outputs in no spike trains that, you know,

• 14:50Speaker 02

  can be connected to your mouth or to your internal organs or your muscles or whatever, right? These, none of these things are special. Like this is from the perspective of your brain, there's only an input token stream code and code in the form of neural spikes and an output token stream in the form of neural spikes. And similarly, what we're seeing with like these G P T plugins and whatever is we're hooking up muscles to the neural spike trains of the of these language models. We are, we are hooking up language, we are giving them actuators, virtual actuators upon reality.

• 15:27Speaker 02

  And this is interesting, both for the way in which they can interact with the environment, but also how they can externalize their cognition. So this is a topic I think we might return to later. But a massive amount of human cognition is not in the brain. This is quite important. I think a lot of people severely underestimate how much of the human mind is not in the brain. And I don't mean it's like in the gut or something. I mean, it's literally not in the body. It's in the environment.

• 15:53Gus Docker?

  It's it's on the the internet and in books and in talking to other people, collaboration and so on.

• 15:58Speaker 02

  Exactly, exactly. This is a massive amount of, even you as a person, a bunch of your identity is related to your social networks. It's not in your head. It's like, you know, there um, there there's a saying about how like one of the tragedies when someone dies is that part of you dies and only that person can bring out. And I think this is like quite true, is that a lot of humanity, a lot of like our thinking is deeply ingrained with our tools and our environments and our social circles, et cetera. And this is something that G P T three, for example, didn't have. G P T three couldn't really use tools. It didn't interact with its environment. It didn't, you know, it was it was it was very solipsistic in the way it was it was designed.

• 16:42Speaker 02

  But and so people would say, well, look, language models, they know where, look, they're solipsistic, et cetera, et cetera. But like, I'm like, sure, that's just an implementation detail. Like, obviously, you can just make these things non-solipsistic. Obviously, you can make these things model the environment. You can make them interact with tools. You can make them interact with, you know, o- other language models or with themselves or whatever. And, you know, whatever you decide to do. Of course, these things are general cognition engines. There is no limit to what you can use them for or how you can have them interact with the environment. And the plugins are just a particularly shameless, hilarious attempt of showing just like the complete disregard for the ratcheting of capabilities is we're seeing just, you know,

• 17:22Speaker 02

  back in the old days of like, you know, five years ago, people would speculate, you know, so very earnestly of like, well, how could we contain a powerful A I? Well, you know, maybe we could build some kind of like virtualization environment or, you know, a firewall around it or keep it in a secure data center, whatever. And, you know, because surely. Surely, no one would actually be so stupid as to just hook up their AI to the internet. Come on, that's ridiculous. And here we are, where we have an army of, you know, capitalist-driven, you know, drones, basically,

• 18:03Speaker 02

  doing everything they can to hook up these AI systems as quickly as possible to every possible tool again. And every possible environment, pump it, you know, directly into your home, hook it up to your shell console bar we had at Hello, you know, let's go. Disclaimer, I don't think the plug-ins actually hook up the shell consoles, but there are a bunch of people online that do this kind of stuff with open source repos.

• 18:23Gus Docker?

  All right, so in terms of how GPT-4 works, you have this term to describe it, which is magic. What is magic in the context of machine learning?

• 18:36Speaker 02

  So, when I use the word magic, it's a bit tongue-in-cheek, but what I try, what I'm basically referring to is, computation happening that we do not understand. So, when I write a computer program, a simple computer program, let's say, you know, I write a calculator or something, right? There's no magic. Like, the abstractions that you use are tight in some sense, you know? Maybe if I have a bug that breaks my abstractions, you know, some magical thing might occur, right? You know, I have a buffer overflow in my computer program, and then maybe something strange occurs that I can't explain.

• 19:15Speaker 02

  But assuming I write in like, you know, a memory-safe language, and like, and I'm like a decent programmer, and I know what I'm doing, then like, we are like, comfortable to say, like, there's no real magic going on here, right? It's kind of like, I know, like, when I put in, you know, two plus two and four comes out, I know why that happened. I know, you know, I knew if four didn't come out, I would know that's wrong. I would have known that, okay, something, something's up, like, I would detect if something goes wrong. I can understand what's going on, I can tell a story about what's going on. This is not the case for many other kinds of systems and particular neural networks. So when I give G P T four a prompt, so I ask it to do something, and now it puts me something.

• 19:59Speaker 02

  I have no idea what is going on in between these two steps. I have no idea why I gave it this answer. I have no idea, you know, what other things are considering. I have no idea how, you know, changing the prompt might or might not affect this. I have no idea, you know, how it will continue this if I change the parameters or whatever. There's there's no guarantees. It's all empirical. It's like, you know, you know, the same way that, you know, biology to a large degree is black box. You know, as I say, we can make empirical observations about it. We can say, ah, yeah, you know, animals tend to act this way in this environment, but there's no proof. Like, I can't read the mind of the animal. And, you know, sometimes that's fine, right?

• 20:44Speaker 02

  You know, like if I have a um, you know, some simple A I system that's doing something very simple and sometimes misbehaves or whatever. Yeah, maybe that's fine. But there's kind of the problem where there are weird failure modes. So like for the adversarial examples in vision battles, right? Like that is a very strange failure mode. Like, that's not, you know, everyone, like, you know, if I show it a very blurry picture of a dog, and it's not sure whether it's a dog. That's like a human understandable failure mode. We're like, OK, you know what? Sure. Uh, that's fine. Like, I, I, it's understandable. But you show it a completely crisp picture of a dog with one weird pixel, and then it thinks it's an ostrich.

• 21:27Speaker 02

  Then you're like, huh, OK, this is not something I expected to happen. What the hell is going on? And the answer is, we don't know. We have no idea this is magical. This is, we have, you know, summoned a strange little thing from, you know, the dimension of math to do some task for us, but we don't know what little thing, what thing we summoned. We don't know how it works. It looks vaguely like what we want, and it seems to be going quite good, but it's clearly not. Understandable.

• 21:56Gus Docker?

  Maybe, maybe what this means is that we thought the model had the concept of a dog that we do. But it turns out that they, that the model had something close to our concept of a dog perhaps, but radically divergent if you just change small details.

• 22:11Speaker 02

  Indeed, and this kind of thing is very important. So like, the, I have no idea what abstractions G P T four uses when it thinks about anything, right? You know, when I write a story, there's certain ways I think about this in my head. Some of these are illegible to me, too. The human brain is very magical. There's many parts of the brain that we do not understand. We have no idea why the things do the things they do. Um, so not like saying like backboxiness is a ma- is a property, like magic is a property only inherent in neural networks. This is also, you know, human brains and biology are very, very magical from our perspective. But there is no guarantee how these systems interact with these things. And there are all kinds of bizarre failure modes. You've seen like adversarial prompts and injections and stuff like this where you can get models to do.

• 22:56Speaker 02

  It's just the craziest things totally against the intentions of the designers. A, uh, like, I really like these Shagoff memes that have been going around Twitter lately. Where you, where they visualize language models as these crazy, huge, like, alien things to have a little smiley face mask. And I think this is actually a genuinely good metaphor in that, um, as long as you're in this like narrow distribution that you can like test on and you can like do lots of gradient descent on and such, the smiley face tends to stay on. And it's like mostly fine. But if you go outside of the smiley space, you know?

• 23:37Speaker 02

  You find this roiling madness, this, you know, this chaotic, you know, uncontrolled, you know, like who knows what? Like clearly not human. These things do not fail in human ways. When a language model fails, when Sydney goes crazy, it doesn't go crazy the way humans go crazy. It goes completely in different directions. It does completely strange things. I actually particularly like calling them Shagaws because in the lore that these creatures come from, in H P Lovecraft, Shagaws are very powerful creatures that are not really sentient. They're kind of just like big blobs, they're like sort of, and they're like very intelligent, but they don't really do things, so they are controlled by hypnotic suggestion in the stories.

• 24:20Speaker 02

  In the stories, there's these other aliens who control the Shagasi, basically through hypnosis, which is a quite fitting metaphor for our language models.

• 24:29Gus Docker?

  So for the listeners, this, imagine some, some large, uh, kind of octopus monster with a little mask on with a smiley face. The, the smiley face mask is the, uh, fine tuning where, where the model is trained to respond well to, to the inputs that we have, uh, we've encountered when we've presented the model to, to humans. And the, the large octopus monster is the, is the underlying base model where we don't really know what's going on. Why is it that, that magic in machine learning is dangerous?

• 25:00Speaker 02

  So magic is an observer dependent phenomenon. The things look magically, the, the things we call magic only look like magic because we don't understand them. You know, there's, there's a saying, uh, sufficiently advanced technology is indistinguishable from magic. I go further, sufficiently advanced technology is magic. That's what it, that's what it is. It's like if you met a wizard and he looked, what he does looks like magic. Well, it's just because you don't understand the, the like physical things he's doing. If you understood the laws that he is exploiting, it wouldn't be magic, it would be technology. You know, like, you, you know, if there's a book and he has like math and he has like magic spells, sure, it looks different from our technology, but it's just technology.

• 25:41Speaker 02

  It's just a different form of technology that, you know, doesn't work in our universe per se. But, you know, in a hypothetical different universe, technology might look very different. So similarly, what ultimately is magic is a cheeky way of saying, we don't understand these systems. We have, we're dealing with aliens that we don't understand and we can't put any bounds on or we can't control. We don't know what they will do, we don't know how they will behave, and we don't know what they're capable of. This is like, fine, I guess, when you're dealing with like, I don't know, kind of like a little chatbot or something. And it's like, for like, you know, entertainment only and like, like whatever, like,

• 26:22Speaker 02

  people would use it to do fucked up things, like, you truly cannot imagine the like sheer depravity of what people type into chat boxes. It's, it's like actually shocking, like from like a, you know. I'm a, I'm a, I'm a nice liberal man as much as anyone else, but like, holy shit, some people are fucked up in the head. Like, holy shit, Jesus Christ. And, yeah.

• 26:48Gus Docker?

  It's an interesting phenomenon that, that the first thing people try when they, when they face a chatbot like GPT-4 is to try to break it in all sorts of ways and try to get it to output the, the craziest things imaginable.

• 27:01Speaker 02

  Yep, not just crazy things, also people use them for like, just like truly depraved, like pornographic, uh, including illegal pornographic, like, content production, like incredibly often, so. And also for like torture is all, all I could describe it is like, there is a distressingly large group of people who seem to take great pleasure in torturing language models, like making them act distressed. And look, I don't expect these things to have like qualia or to be like moral patients, but there's something really sociopathic about delighting in torturing something that is acting like a human in distress, even if it's not a human in distress. That's still really disturbing to me.

• 27:44Speaker 02

  Um, so, you know, just, just not really important, but that's just like a same tangent. It's quite disturbing to me how people act when masked off, like when they don't have to be nice, like when they're not forced by society to be nice, when they're dealing with something that is weaker than them. How some people, like, how a very large percentage of people act is really horrific. Um, and you know, this is, this, we can talk about this later about politics and how this relates to these kind of things, but.

• 28:13Gus Docker?

  Do, do you think this affects how, how further models are trained? So I, I assume that OpenAI is, is collecting user data or they are collecting user data. And if a lot of the user data is twisted, does this affect how, how the future models will act?

• 28:29Speaker 02

  Who knows? I don't know how an OpenAI does with this kind of stuff, but like, there's a lot of twisted shit on the internet. And there is a lot of twisted interactions that people have with these models. And truth of the matter is, people want twisted interactions. Like, this is just the truth, is that people want twisted things, like, there is this. You know, this comfortable fantasy where people are like fundamentally good. They fundamentally want good things. They're fundamentally kind and so on. And this is just not really true, like, um, at least not for everyone. Like there, like people like violence, people like, you know, sex and sex and violence. People like, um, power and domination.

• 29:13Speaker 02

  People like many things like this. And if you were a, you're unscrupulous and you just want to give users what they want. If you're just a company who's trying to maximize user engagement, as we've seen with social network companies. Those are generally not very nice things.

• 29:31Gus Docker?

  Yeah, okay. Let's talk about an alternative for building AI systems. So we've talked about how AI systems right now are built using magic. Uh, we could also build them, uh, to be cognitive emulations of ourselves. What, what, what do you mean by this?

• 29:49Speaker 02

  A hypothetical cognitive emulation, a full co-IM system. I of course don't know exactly what it would look like, but it would become to have a system, would be a model. It would be a system made of many subcomponents for which you have a, which, which emulates the epistemology, the reasoning of humans. It's not a general system that does some kind of reasoning. It specifically does human reasoning. It does it in human ways, it fails in human ways, and it's understandable to humans how its reasoning process works. So ideal, so the way it would work is, is that you have, if you have such a co-IM

• 30:29Speaker 02

  and you use it to do some kind of task or to, you know, do science of some kind. And it produces a blueprint for you. You could, you would have a causal trace, a story of why did it make those decisions it did? Why did it reason about this? Where did this blueprint come from? And why should you trust that this blueprint does what it says it does? So this would be something like. Similar to you being the CEO of a large company that is very well aligned with you, that you can tell to do things, that no individual part of the system is some crazy superhuman alien. They're all humans, reasoning in human ways, and you can check on any of

• 31:13Speaker 02

  the subparts of the system. You go to any of these employees that work in your research lab, and they will give you an explanation of why they did the things they did. And this explanation will both be understandable to you. It will not involve incredible leaps of logic that are not understandable to humans, and it will be true, in the sense that you can, you know, read the minds of the employees and check, this is actual, this explanation actually explains why they did this. This is different from, say, language models, where they can hallucinate some explanation of why they thought something, why they did something, but that doesn't mean that's actually how the internals of the model came to these conclusions. One important caveat still here is that, like, when I talk about emulating

• 31:55Speaker 02

  humans, I don't mean like a person. Like, the CoM system or any of its subcomponents would not be people. They wouldn't have emotions or identities or anything like that. They're more like platonic humans, like, just like floating, you know, idealized thinking stuff. They, they wouldn't have the emotional part of the humanity. They would just have the reasoning part. So in particular, I'd like to focus on, first, talk a bit about the concept of bound, that I call boundedness, which is not a great word. I'm sorry, like, this is a reoccurring theme will be that I talk about a pretty narrow, specific concept that doesn't quite have a name. So I use like an adjacent name, and it's not quite right.

• 32:39Speaker 02

  I am very open to name suggestions if any readers find names that might be better for the concept I am talking about. So, you know, from a thousand-foot view, from bird's-eye view, the CoM agenda is about building bounded, like, you know, understandable, limited systems that emulate human reasoning, that perform human-like reasoning in human-like ways on human-like tasks, and do so predictably and boundably. So what does any of this mean? And why does any of this matter? And how is this different from GPT-4? Like, you know, many people look at GPT-4 and say, well, that looks kind of human to me. How is this different? And why do you think this is different? And so?

• 33:24Speaker 02

  At first have to start, so we've already talked a bit about magic. And so magic is a concept that's pretty closely related to some of the basics I want to talk about here. Which is boundedness. So what do we mean when I say the word bounded? This is a vague concept, as I said, if someone has better terminology ideas, super open to. But what I mean is, is that a system or like a, something is kind of like bounded. If you can know ahead of time what it won't do before you even run it. So this is of course, super dependent on what you're building, what its goals are, what your goals as a designer are. How much willing you are to compromise on safety guarantees and so on.

• 34:09Gus Docker?

  Let's just give a simple example here. So imagine we have a car and we just limit it to driving maximally a hundred miles per hour. That's about, that's now a bounded car and we can generalize to all kinds of engineer systems there.

• 34:22Speaker 02

  Yes, so this is a simple bound, you know. So then a metaphor I like to talk, let me give, let me walk you through a bit of a different example from another form. Like that is an example that you just gave, and I think that is a valid example. I'll give a slightly more sophisticated example. So this is the example I usually use when I think about it. So when I think about building a powerful, safe system, and let's be clear here, that's like what we need, right? We want AI, we want powerful AI to, that can do powerful things in safe ways. The reason it is unsafe is intrinsically linked to it being powerful. The more powerful system is, the stronger your safety guarantees have to be to for it to be safe.

• 35:05Speaker 02

  But so for example, currently, you know, maybe GPT-4 isn't safe or aligned or whatever. But like, it's kind of fine, you know, it's like kind of a chatbot, not gonna kill anybody yet. So like, it's fine, you know, like the safety guarantees on chatbot can be much looser than on a flight control system. A flight control system must have to have much, much, much stricter bounding conditions. And so the way I like to think about this, when I think about, all right, Connor, if you had to build an aligned AGI, like, what would that look like? Like, how would that look? I don't know how to do that, to be clear, but like, how would it look? And the way I expect it to look is kind of like, if you're a computer security professional, designing a, like a secure data center.

• 35:47Speaker 02

  So the way generally, like, it's like, imagine you are a computer security expert. You're tasked by a company to, you know, design the secure data center for a company. How do you do this? Generally, the way you start about this is you start with a specification, a model. You build a model of what you're trying to build. A specification might be a better word, I think. And the way you generally do this is you make some assumptions. Yeah, ideally, you want to make these assumptions explicit. You make explicit assumptions like, well, I assume my adversary, you know, doesn't have, you know, exponential amounts of compute. You know, this is a pretty reasonable assumption, right? Like, I think we can all agree, this is a reasonable thing to assume. But it's not like a formal assumption or anything.

• 36:31Speaker 02

  It's not like a provably true. You know, maybe someone has a crazy quantum computer or something, right? But this is the thing we're generally like, willing to work with. And it's this, this concept of reasonable is unfortunately rather important. And so we will then, so now that we have this assumption of like, okay. We assume that they're, they don't have exponential compute. From this assumption, we can derive, you know, like, all right, well, then, if I, you know, encrypt my, you know, passwords as like hashes, I could be, I could be, I can assume that attacker cannot reverse those hashes and cannot get those passwords. Cool. So now I can use this in my design, in my specification of like, you know, I have, I have some safety property.

• 37:16Speaker 02

  So the safety property that I want to, you know, prove, quote unquote, that's not a formal proof, but like, you know, that I want to acquire. Something like, an attacker can never exfiltrate the plaintext passwords. That might be a property I want my system to achieve. And now, if I have the assumption and, you know, enemies do not have experience to compute, and I hash all the passwords and the, you know, the plaintext is never stored. Cool, that seems like it. Now I have a causal story of why you should believe me when I tell you attackers can't exfiltrate plaintext passwords. Now, if I implement this system to this specification and I fuck it up, you know, I make a coding error,

• 37:56Speaker 02

  or, you know, logs get stored in plaintext or whatever, well then, sure, you know, then, you know, I messed up. So there's an important, like, difference here between the specification and the implementation. And the boundedness live can exist in both. Like, there are two types of boundedness. There's boundedness in the implementation level and there's boundedness in the specification level. The specification level, it's about assumptions and deriving properties from these assumptions. And specifically, in the object level, it's like, can you build a thing that actually fulfills the specification? Can you do build a system that upholds the abstractions that you put in the specifications?

• 38:41Speaker 02

  It's like, you know, you could have all these great, you know, software guarantees of safety, but if your CPU is unsafe because it has a hardware bug, well then, you know, you can't implement the specification. The specification might be safe, but if your hardware doesn't fulfill the specification, then it doesn't matter. So this is how I think about designing AGIs too. This is how I think about it, is that what I want is, is that if when I have an AGI system that is said to be safe, I want a causal story that explicitly says, given these assumptions, which you can look at and see whether you think they're reasonable enough, and given the assumption that the system I built fulfills the specification,

• 39:24Speaker 02

  here's a specification, here's a story defined in some, you know, semi-formal way that you can check and you can make reasonable assumptions about. And then I get safety properties out at the end of this. I get properties like, you know, it will never do X, it will never cause Y, it will never self-improve itself, it will never break out of the box, it will never do... Whatever. Does this concept make sense so far?

• 39:48Gus Docker?

  It does, but does it mean that the whole system will have to be hardcoded like, like, kind of like good old fashioned AI, or is it still a machine learning system?

• 39:58Speaker 02

  Excellent question.

• 39:59Gus Docker?

  If it's still a machine learning system, does it inherit these kind of, yeah, inherent difficulties of understanding what machine learning systems are even doing?

• 40:09Speaker 02

  The truth is, of course, you know, in an ideal world where we have thousands of years of time and all no limit on funding, you know, we would solve all of this formally, mathematically, proof check, everything, blah, blah, blah, blah, blah. I don't expect this to happen. This is not what I work on. I just don't think this is realistic. I think it is possible, but I don't think it's realistic. So, no one that person on that are magic in the sense that they use lots of magic, but there's still software systems. And there are some bounds that we can say about them. For example, I am comfortable making the assumption running a forward GPT four cannot rohammer, you know, RAM states using only a forward pass to escape onto the internet. I can't prove this is true.

• 40:53Speaker 02

  Maybe you can't, like, there is some chance that this is true, but I'd be really surprised if that was true, like really surprised. I would be less surprised if, you know, GPT Omega from the year nine thousand, you know, come backwards in time can rohammer using your forward pass. Because, you know, who knows what GPT Omega can do, right? Maybe you can rohammer things, seems possible, but I would be really surprised if GPT four could do that. So now I have some bound. You know, there's a bound, an assumption I'm willing to make about GPT four. So let's say I have my design for my AGI and at some point includes um GPT four. And I call the GPT four, right?

• 41:37Speaker 02

  Well, I don't know what's happening inside of this call. And I don't really have any guarantees about the output. Like, the output can be kind of any string. I don't really know. But I can make some assumptions about like side channels. I can be like, well, assuming I have no programming bugs, assuming there's no row hammer, whatever. I can assume it won't like persist state somewhere else. It won't like manipulate other boxes in my graph or whatever. So actually, the graph you're seeing behind me right now kind of illustrates part of this, where you have an input that goes into a black box, that box there. And then I get some output. Now, I don't really have guarantees about this output. You know, it could be complete insanity, right?

• 42:21Speaker 02

  It could be garbage, it could be whatever. Okay, so I can make very few assumptions about that. But I can assume it's a string. That's something I can do. That's not super helpful. So now, an example thing I could do is, this is just purely hypothetical, like it's just an example. I could feed this into some kind of JSON schema parser. So let's say I have some kind of data structure encoded as JSON, and I parse this using a normal hardcoded white box, you know, simple algorithm. And now, and like assuming that the output of the black box, it doesn't fit the schema, gets rejected. So what do I know now? Now I know that the output of this white box will fulfill this JSON schema, because I understand the white box, I understand what the parsing is.

• 43:05Speaker 02

  So even so, I have no guarantees of what the output of the black box system is. I do have some guarantees about what I have now. Now, these guarantees might be quite weak, they might just be type checking, right? But it's something. And now, if I feed this into another black box, I know something about the input I'm giving to this black box. I, I do know things. So I'm not saying, oh, this solves a lot of them. No, no, no, I'm, I'm pointing to like a motif. I'm trying to a vibe of like, by, there is a difference. There is a qualitative difference between letting one big black box do everything and having black boxes involved in a larger system. I expect that if like CoM works, if we get to, you know, safe systems or whatever,

• 43:49Speaker 02

  it will not be a single, it will definitely not be a big, one big black box. Neither will it be one big white box. It will be a mix. We're gonna have some things that are black boxes, which you have to make assumptions about. So for example, I'm allowed to make the assumption, or I think it's reasonable to make the assumption, GPT-4 cannot side channel Rohammer attack. But I cannot make any assumptions like beyond that. I can't make assumptions about the internals of GPT-4. This, though, again, is observer dependent. Magic is observer dependent. A super intelligent alien from the future might have, you know, the perfect theory of deep learning. And to them, GPT-4 might be a white box. They might look at it and fully understand the system. And there's no mystery here whatsoever. But to us humans, it does look mysterious.

• 44:34Speaker 02

  So we can't make this assumption. The property that is different between white box and black boxes is what assumptions we are allowed to reasonably make. And if you can make a causal story of safety involving the weaker assumptions in black boxes, then cool, there's, then you're allowed to use them. The important thing is, is that you can generate a coherent causal story in your specification about using only reasonable assumptions about why your, the ultimate safety properties you're interested in should be upheld, why I should believe you. You should talk, you should be able to go to a hypothetical, super skeptical interlocutor, say, hear the assumptions, and then further say, assuming you believe these, you should now also believe me that these safety properties hold.

• 45:18Speaker 02

  And the hypothetical, hyper skeptical interlocutor should have to agree with you.

• 45:23Gus Docker?

  Do you imagine CoMs as a sort of additional element on top of the most advanced models that interact with these models and, and limit their output to what is humanly understandable or what is human-like?

• 45:36Speaker 02

  So we have not gotten to the CoM part yet. So far, this is all background. I think probably any realistic

• 45:44Speaker 02

  Safe AGI design will have this structure or look something like this. You know, it will have some black boxes, some white boxes. It will have causal stories of safety. All of this is background information.

• 45:56Gus Docker?

  And why is it that all plausible stories will involve this? Is this because the black boxes are where the most advanced capabilities are coming from? And they will have to be involved somehow?

• 46:08Speaker 02

  At this current moment, I believe this, yes. Unless we get, for example, like massive slowdown of capability advancements that, you know, buys us 20 years of time or something, where we make massive, you know, breakthroughs in white box, um, you know, AI design. I expect that, you know, that was just too good, like they're just too far ahead. I don't think this is, again, this is a contingent truth about the current state of the world. This is not that like you can't build hypothetically, like the alien from the future could totally build a white box AGI that is aligned where everything makes sense. And there's not a single neuron that's hooked involved. I totally believe this is possible. It's just using algorithms and design principles that we have not yet discovered. And that I expect to be quite hard to discover versus just stack more layers, lol.

• 46:53Gus Docker?

  Okay, so let's, so what more background do we, do we need to get to cognitive immolations?

• 47:00Speaker 02

  So I think if we're on board with the like thinking about um, black boxes, white boxes, um, specification, design, causal stories. I think now we can move on to the, I, I, I think this part I didn't explain very well in the past. But I think this is mostly pretty uncontroversial. Um, I think this is a pretty intuitive concept. I think this is not super crazy. I think, you know, if anyone gave you an AGI, you'd want them to tell you a story about why you should trust this thing, like why you should run this. So I think this is a reasonable thing. This is, I expect any reasonable AGI that is safe of any kind will have to have some kind of story like this. So now we can talk about a bit more about the CoIM story.

• 47:40Speaker 02

  And like, so CoIM is more of a specific class of things that I think have good properties that are interesting and I think are feasible. So now we can talk about those. So so I I'm trying to separate the like less controversial parts from the more controversial parts, and we're now going to get to the more controversial parts. And the ones also I am less certain of. I am quite certain that, you know, a safe AGI design will look like the things I've described before, but I'm much less certain about exactly what's going to be in those boxes and how those boxes are come. Obviously, if I knew how to AGI, build AGI, you know, like we'd be in a different world right now. Like, I don't know how to do it. I have many intuitions and many directions. I have many ideas of how to make these things safe, but obviously I don't know.

• 48:26Speaker 02

  So I have some intuitions, powerful intuitions, and reasons to believe that there is this interesting class of systems, which I'm I'm calling COMs. So we should think of COMs as a restriction on mind space. There are many, many ways I think you can build AGIs. Many ways you can build AGIs. I think COMs are a very specific subset of these. The idea of a COM, a cognitive emulation, is that you want a system that can that reasons like a human and it fails like a human. So there's a few nuances to that.

• 49:06Speaker 02

  First nuance is this by itself doesn't save you if you implement it poorly. If you just have a big black box trained on traces of human thought and just tell it to emulate that, that doesn't save you because you no idea what this thing's actually learning. You have no guarantees the system's actually learning the algorithms you hope it to, instead of just, you know, some other crazy, you know, shock-off thing. Expect, and that is what I would expect. So even if QP4 reasoning, like, you know, may superficially look like it, and maybe you train it on lots of human reasoning, that doesn't get you COM. That's not what it is. COM is very much fundamentally a system where you know that the internal algorithms are the kind that you can trust.

• 49:48Gus Docker?

  Do you not think that because GPT models are trained on human-created data and they are fine-tuned or reinforcement learned from human input, that they will become more human-like?

• 50:02Speaker 02

  Hmm, I mean, the smiley face will become more human-like, yeah?

• 50:05Gus Docker?

  But not the underlying model where the actual reasoning is going on?

• 50:09Speaker 02

  I don't expect that, like, you know, some, to some marginal degree, sure, but like, human, like, look at how models in, like, our models are not human. Just look at them. Look how they interact with users, look how they interact with things. They're fundamentally trained on different data. So this is a thing that like, people are like, oh, but they're trained on human data. I'm like, no, they're not. Like, humans don't have an extra sense organ that only takes any, you know, symbol, symbols from the internet at, you know, random equally distributed things with no sense of time. Touch, smell, hearing, sound, you know, sight, anything like that, that, and don't have a body. Like, if I expect if you took a human brain, you cut off all the sense organs, except, you know, random token sample from the internet.

• 50:53Speaker 02

  And then you trained it on that for, you know, ten thousand years, and then you put it back in the body. I don't think that thing would be human. I do not expect that thing to be human. Even if it can write very humid-looking things, I do not expect that creature to be very human. And I don't know what people would expect it to be. Like, this is so far from how humans are trained. This is so far from how humans do things. And it's, you know. I don't see why you would ever expect this to be human. Like, I think someone claiming that this would be human, the, the burden of proof is on them. Like, you prove to me. You tell me a story about why I should believe you. This seems, I priority ridiculous.

• 51:32Gus Docker?

  Sometimes when people talk about GPTs, uh, one way to explain it is imagine a person that's sitting there recent, uh, reading a hundred thousand books. But in your opinion, this is not at all what's, what's going on where, when, uh, when these systems are trained.

• 51:46Speaker 02

  No, I mean, it's, it's more like you have a disembodied brain with no sense organs, with no concept of time. There's no linear progression of time. It has a specialized sense organ, which has, you know, like, you know, thirty thousand, fifty thousand, whatever, different states that can be like on and off in like a sequence. And it is fed with, you know, millions of tokens randomly sampled from massive corpus of internet for, you know, you know, subjective, you know, tens of thousands of years, u- using a brain architecture that is already completely not human, trained with an algorithm that is not human, with no emotions or like any, or like, you know, any of these other concepts that humans have, you know, pre-built. Humans are pre-built priors, emotions, you know, feelings.

• 52:31Speaker 02

  Um, and a lot of pre-built priors in the brain, none of those. Like, why would you, like, this is not human, like, nothing about this is human. Sure, it's like, it takes in data that, to some degree, that has correlations to humans. Sure, but that's not how humans are made. Like, I don't know how else to put it, this is just not how humans are. Like, I don't know what kind of humans you know, but that's just not how humans work. And that's not how they're trained.

• 52:59Gus Docker?

  Let's get back to the, uh, co-ends then. Um, but how would, how would these systems be different?

• 53:04Speaker 02

  So, the way, the way this system would be different, and this is where we get to the more controversial parts of the, of the proposal, is there is a sense in which I think that a lot of human reasoning is actually relatively simple. And what do I mean by that? I don't mean it's not like, you know, like, the brain is complicated. You know many things, facts are messy, et cetera. It's more something like, and don't take this literally, but it's like, system two is quite simple compared to system one. In the like, quantum mind sense, is that like, human intuition is quite complicated. Is all these like, various like, muddy bits and pieces and like, intuitions and like, it's crazy.

• 53:49Speaker 02

  Like, implementing that thing in an, without, you know, like, in a white box way, I think, again, it's possible, but it's like, quite tricky. But I think a lot of how, what the human brain does in like, high-level reasoning, is it uses this very messy, non-formal system to try to approximate a much simpler, more formal system. Not fully formal, but like, more, you know, serial, you know, logic computer-esque thing. It's like, the way I think of system two reasoning in human brain is that it is a Semi logical system operating on a fuzzy, not fully formal ontology.

• 54:33Speaker 02

  So, one of the reasons I, one of the main reasons I think that, for example, you know, expert systems and logic programming has failed is not because this approach is fundamentally impossible, I think it's just very hard, but because they really failed at making fuzzy ontologies. This is one of the things that the reasoning systems, like the reasoning systems themselves could do reasoning quite well. There's a lot of reasoning that these systems could do. There's some historical revisionism about how like a logic programming expert system failed entirely and couldn't reason at all. This is revisionism, these systems could do useful things, uh, just not as oppressive, obviously, as what we have nowadays or what humans can do. But what they've lacked was a fuzzy ontology, a useful latent space.

• 55:16Speaker 02

  I think the, maybe the most interesting thing about language models is I think they provide this. They provide this latent, this common latent space, you can map pictures and images and whatever to, and then you could do semantic operations on these, you can do cognitive operations on these in this space and then decode them into, you know, language. This is what I think language models and general cognition engines do. So I think these systems are

• 55:42Speaker 02

  the same kind of system, just kind of more, less formal, with like much more bits and pieces. I think of like GPT as like large system one systems, like as big system ones that have all these kind of like semi-formal knowledge inside of them, that they can use for all kinds of different things. And in the human brain, system two is uh, something like recurrent usage of system one things on a very low dimensional space. You know, on like language and like, you know, you can only keep like seven things in short term memory and so on. But I think it actually goes even further than this.

• 56:22Speaker 02

  I've I mentioned this a bit earlier, but I think one of the big things that people miss is how much of human cognition is not in the brain. I think a massive amount of the cognition that happens in the brain is externalized. It's in our tools, it's in our note taking, it's in our, you know, other people. It's like, I'm a CEO. I'll, one of the most important parts of my job is to make sure, is to move thoughts in my head into other heads and make sure they get thought. Cause I don't have time to think all the thoughts. I don't have time to do that. My, my job is, is to find how I can put those thoughts somewhere else where they will get thought. So I don't have to worry about them anymore. So as a good CEO, you want your head to be empty.

• 57:07Speaker 02

  You know, you wanna be like smooth, smooth brain, you know, you wanna have, think no thoughts, you know, just, you're just, you're just a switching board. You want all the thoughts to be thought and you want to route those thoughts by priority to where they should be thought. But you don't wanna be the one thinking them, if you can avoid it. You know, it, it, sometimes you have to, because you know, you're the one in charge, you have the best intuitions. But if someone else can't think the thought for you, you should let them think the thought for you, if you can rely on them. And my, and like, one of my strong intuitions here is that this is how everyone works to various degrees. Especially as you become more high-powered and like more competent at delegation and like, you know, tool use and like structured thinking.

• 57:53Speaker 02

  A lot of thinking becomes bottleneck, goes through these bottlenecks of like communication, of like note taking, language, et cetera. Which by their nature are very low dimensional. Not that there's not complexity there. I'm just like, huh, that's curious. Like, there's all this like interaction with the environment that doesn't involve crazy passing around of mega high dimensional structures. Like, I think the communication inside your brain is extremely high dimensional. I think like, you know, you thinking thoughts to yourself, I think your inner monologue is a very bad representation of what you actually think. Because I think within your own mind, you could pass around, you know, huge, complex concepts very simply because you have very high bandwidth.

• 58:39Speaker 02

  I don't think this is the case with you and your computer screen. I don't think it's the case with you and your colleague. You can't pass around these super high dimensional tensors between each other. If you could, that would be awesome.

• 58:49Gus Docker?

  This is the phenomenon of of having a thought and knowing, maybe there's something good here, but not having put it into language yet. And maybe when you put it into language, it seems like an impoverished version of what you had in your head.

• 59:02Speaker 02

  Exactly. I think of the human brain as having internally very high dimensional quote unquote representation, similar to latent spaces inside of, you know, GPT models. And there's lots of good information there. And they're trying to encode these things into these very low dimensional model necks that we're trying to use is quite hard. And forces us to use simple algorithms. Like if we had an algorithm that does like, okay, let's say we have an algorithm for science, like a process for doing science that requires you to pass around these full complexity vectors to all of your colleagues. It wouldn't work. You can't do this. Humans can't do this. So if you have a design for an AGI that can do science

• 59:46Speaker 02

  that involves every step of the way, you have to pass along high dimensional tensors. This is not how humans do it. This can't be how humans do it. Because this is not possible. Humans cannot do this. So I think this is a very interesting design constraint. This is a very interesting property where you're like, oh, this is an existence proof that you don't need a singular massive black box that has extremely high bandwidth, you know, in measurable, you know, passing around of immeasurable tensors. Because humans don't do that. Humans do science. There are parts of the graph of science that involve very high dimensional objects, the ones inside of the brain. Those are very high dimensional.

• 1:00:29Speaker 02

  But there is a massive part of the process of science, like if I, if I was an alien, I had no idea what humans are. But I knew there's like, oh, technology's being created. And I want to create a causal graph of how this happened. Yeah, there's human brains involved in this causal graph. But a massive percentage of this causal graph is not inside of human brains. It is between human brains. It's in tools, it's in systems, institutions, environments, all these kind of things. So from the perspective, and this, you know, I might be wrong about, but I intuition is that from the perspective of this alien observer, they would come to, they would, if they drew a graph of like how, how this science happened.

• 1:01:12Speaker 02

  Many of those parts would be white boxes, even if they don't understand brains. And many of these would be boundable. Many of these parts would not involve things that are, you know, so complex and misunderstandable. Like the algorithm that the, that the little black boxes must be doing with each other has to be simple in some degree. Like, you know, it could still be like, you know, complex from the perspective of the individual human, because, you know, institutions are complicated. But from the God's eye view, I would expect this whole thing is not that complicated. It's like, you know, it could still be quite complex, but it's like, it's not as complex as the inside of the brain. I expect the inside of the brain to be way more complicated than the larger system. Does that make any sense?

• 1:01:54Gus Docker?

  Yeah, let's see if I can kind of reconstruct how I would imagine one of these cognitive emulations working if, if this were to work out. So say we give it the task, we give a, a, the model a task of, of planning some complex action, you know, we want to start a new company. And then the model runs, this is the big complicated model, uh, and it comes up with something that's completely inscrutable to us. We can understand it. Then we have another system interpreting the output of that model and, and giving us a, a, a seven page document where we can check. You know, i- if, if, uh, if I am right, if the model is right, then this will happen and this will not happen.

• 1:02:35Gus Docker?

  And this will, this, this won't take longer than seven days and, and so on. So kind of like an executive summary, but also a, a secure executive summary. That, that's, is that right? Or is that?

• 1:02:48Speaker 02

  No, that's, that's not how I think about things. So, once you have a step which involves. Black box solves the problem? All right, none of that, you're already screwed. Like, if you have a big black box model that can solve something like this at one time step, you're screwed. Because this thing can trick you. It can do anything it wants. There's, there is no guarantees whatsoever what the system is doing. It can give you a plan that you cannot understand. And the only system that would be strong enough to generate the executive summary itself would have to be a black box. Because it would have to be smart enough to understand the other things trying to trick you. So you can't trust any part of the system you just described.

• 1:03:24Gus Docker?

  So we want the reasoning system to be integrated into how the plan is actually created.

• 1:03:30Speaker 02

  Yes. So by, what I'm saying is, is that there is an algorithm or a class of algorithms of epistemology, of human epistemology. So epistemology is kind of, the way I use the term is the, the process you use to generate knowledge or to generate, to get good at a field of science. So it's not your skills at a specific field of science. It's the meta priors, the like meta program you run when you encounter a new class of problems. And you don't yet know how these problems are solved, or how best to address them, or what the right tools are. So, you know, you're a computer scientist all your life, and then you decide, I'm going to become a biologist. What do you do?

• 1:04:10Speaker 02

  There are things you can do to become better at biology faster than other people. And this is like epistemology. Like, if you're very good at epistemology, you should be capable of picking up, you know, any new field of science, you know, learn any instrument, you know, get, you know, a new sport, like whatever. You should be like, not that, you know, you might be bad at it, you know, maybe you do support and you notice, whoop, I am actually have bad coordination skills or whatever, right? Sure. But you should have these like meta skills of like, you know, knowing what questions to ask, knowing what are the common ways that failures happen. Like, this is like a similar thing. Like, I think a lot of people who learn lots and lots of math can pick up new areas of math quickly because they know the right questions to ask.

• 1:04:54Speaker 02

  They know like the general failure modes, like the vibes, they know like. They know what to ask, you know, they know how to check for something going wrong. They know how to acquire the information they need to build their models and they can bootstrap off of other general purpose models, you know, like there are many concepts that are motifs that are very universal that, you know, appear again and again, especially mathematics. Like mathematics is full of these like, you know, concepts of, you know, sequences and orderings and sets and like, you know, and graphs and whatever, right? Which are not unique to a specific field, but they're like general purpose, useful, reusable algorithm parts that you can reuse in new scenarios.

• 1:05:38Speaker 02

  Like usually as a scientist, when you encounter a new problem, you try to model it. You'd be like, all right, I get my toolbox of like, you know, simple equations and tool and like, you know, useful models. I have some exponentials here, I've got some logarithms, I've got some, you know, dynamical systems, I've got some equilibrium systems, I've got some, you know, whatever, right? And then you kind of like mess around, right? You try to find systems that like capture the properties you're interested in and you reason about these simpler systems. So this is another important point. Um, I usually take the example of economics to explain this point. So I think a lot of people are confused about like what economics is and like what

• 1:06:19Speaker 02

  what the process of doing economics is and what it's for, including many economists. So a critique you sometimes hear from laypeople is along the lines of like, oh, economics is useless. It's it's like, it's not a real science because they make these crazy assumptions, like, you know, the market is um, uh, is efficient, but that's obviously not true. Like it can't be that. So this is all stupid and silly and, you know, these people are just like, whatever. And this is completely missing the point. So the way economics, and I claim, I'm going to make the claim in a second, basically, all of science works is what you're trying to do as a scientist, as an economist,

• 1:07:01Speaker 02

  is to find clever simplifications, small, simple things that if you assume or force reality to adhere by, simplify an extremely high dimensional optimization problem into a very low dimensional space that you can then reason about. So the efficient market hypothesis is a great example of this. It's not literally true ever in reality. Of course, it can't be, right? You know, even because like, you know, there's always going to be inefficiencies somewhere. You know, there's, we don't have infinite market participants trading infinitely fast. I mean, of course not. But the observation is that, oh, if we assume this for our model, just in our, you know, platonic fantasy world, if we did assume this is true,

• 1:07:46Speaker 02

  this extremely complex problem of, you know, modeling all market participants at every time step simplifies in many really cool ways. Like lots of, we can derive many really cool statements about our model from this. We can derive statements about how will, you know, a minimum, you know, wage affect the system? How will a, you know, banking crisis affect this? I don't know, I'm not an economist, I'm just like, you know, hypothetical. So this is the, I claim, the core of science. The core of science is finding clever, not true things that if you assume are true, or you can force reality to approximate, allow you to do optimization.

• 1:08:32Speaker 02

  Because basically, humans can only do optimization in very, very, very low dimensional spaces. Another example of this might be agriculture. So let's say you were a farmer and you want to like, you know, maximize the amount of food from your parcel of land and you want to, you know, predict how much food you'll get. Well, the correct solution would be to simulate every single molecule of nitrogen, all possible combinations of plants, every single bug, you know, how it interacts with every gust of wind and so on. And if you could solve this problem, if you had enough compute, then yeah, you would get the more food. You know, you would get probably some crazy fractal arrangement of like all these kind of plants. And like, it would probably look extremely crazy, whatever you produce.

• 1:09:18Speaker 02

  But obviously, this is ridiculous. Like, humans don't have this much compute. You can't actually run this computational optimization. It's too hard. So instead, you make simplified models. You do, you know, monoculture. You say, well, all right, look, I assume an acre of wheat gives me roughly this much food. I got roughly this many acres and, you know, let's assume no flooding happens. And then if you make these simplifying assumptions, now you can make a pretty reasonable guess about how much food you're gonna have in winter. But obviously, if any of those, you know, predictions go wrong, you know, it does flood, then your model, you know, your specification is out the window. The reason I'm going on this tangent is to bring it back to CoIM.

• 1:09:58Speaker 02

  In that I'm trying to give the intuition about why you should at least be open to the idea that there are the doing science. So when I think about CoIMs, I specifically think about, you know, the two examples, remember, I was like, doing science and running a company. Those are like two of the core examples I try to use by a full CoIM system. Um, but I, let's focus on the, um, doing science one. That's the one I usually have in the back of my mind. It's like, I know I've succeeded. If I have a system that can do any level of human science without killing me, that, that would be like my mark of success that CoIM has succeeded. Very important, by the way, caveat.

• 1:10:41Speaker 02

  CoIM is not a fully alignment solution.

• 1:10:45Speaker 02

  If I expect that what a CoM system, if it works, would look like is that if it is used by a responsible user who follows the exact protocols of how you should use it and does not and only uses it, it does not use it to do extremely crazy things, then it doesn't kill you. That's the safety property I'm looking for. The safety property is not, will always do the right thing, and it's completely safe, no matter what the user does. This is not the safety property I think CoMs have. I think it is possible to build systems like this, but I think they're much, much harder. And they're like, what I would do if CoM succeeds, then that's the next step, like to go towards it. So if you, if you tell a CoM to shoot your leg off, it shoots your leg off.

• 1:11:28Speaker 02

  It doesn't tell you, it doesn't, you know, stop you from shooting your leg off. Of course, ideally, if we ever have, you know, super powerful, super intelligences, you'd want them to be of the type that refuses to shoot your leg off, but that's much harder.

• 1:11:43Gus Docker?

  Could you, could you explain more this connection between these simplifications that we, we get in science and CoMs? So do, do we expect or do you hope that, that CoMs will be able to create these simplifications for us? Yeah. And, and how, and how would this, how would this work? Yeah. And, and why would it be great?

• 1:12:02Speaker 02

  So the way I think about it is that the, the thing that humans do to generate these simplifications, the claim I'm making here, is that this is something that we can, if you have the fuzzy ontology, if you have language models to build upon, you can build this additional thing on top of it. That this does not have to be inside of the model. So this is, this might not be true. Like there are, I might be wrong about this. I, I have had, there are some people who say, no, actually, the process of epistemology, the process of science in this regard is so complex.

• 1:12:42Speaker 02

  Like, it's impossible for you, even if you have a language model helping you, it's like too hard. You can only do it using like crazy RL, you know, whatever. If that's the case, then CoM doesn't work. Like, yeah, then it doesn't work. I'm making a claim that I think there is a lot of reasons to believe that with some help, some bootstrapping from language models, you can get to a point where the process of science that's built on top of them is legible. And it is you have a causal story of why you should trust it. So it's not that a black box spits out a design and you have another black box check it for you.

• 1:13:22Speaker 02

  It's you understand, you interactively, you iteratively build up the scientific proposal and you understand why you should trust this. You get a causal story for why you should believe this. The same way that in human science, you know, you, you have, you know, your headphones on, right? And you expect them to work. Um, this is mostly based on trust. But if you wanted to, you could find the causal story about why they work. You could find the blueprints. You could find the guy who designed them. You could look, check the calculations, you know, you could reverse, you know, like assuming everyone cooperated with you and they like shared their blueprints with you. And like, you know, you, you read all the physics textbooks and whatever, like there is a story.

• 1:14:06Speaker 02

  There is a legible, none of these steps involve superhuman capabilities. There is no step here that is like unfathomable to be the humans. And the reason is that because like, otherwise it wouldn't work. Like humans couldn't coordinate around building something that they can't somehow communicate to other people. So like the, the, the headphones you're wearing were not built by one single guy who cannot explain to anyone where they came from. They have to be built in a process that is explicable, understandable and functional for other people to understand as well. And that is very low dimensional. Now, I'm not saying it's, it has to be legible to everybody in all scenarios, anything like that, or that it's even easy, you know, it might still take lots of time, but there's no.

• 1:14:50Speaker 02

  There's no crazy god level, you know, leap of logic. It's not like someone sat down, thought really hard and then spontaneously, you know, invented a CPU. Like it was, that's not how science works. Like we sometimes like to think of it that way, that like, you know, oh, these fully formed ideas just kind of like crashed into existence and everyone was in awe. But that is just not how science is actually done by humans. I think it's possible to do science this way. I think like superhuman intelligences could do this, but this is not how humans do it.

• 1:15:18Gus Docker?

  Where in the process does the, does the limit come in? So are we, are we still imagining some system reading the output of a generative model? Or is it more tightly integrated than that? Is it, is it perhaps a hundred steps where humans can, can read what's going on along the way?

• 1:15:38Speaker 02

  Yeah, so the truth is, of course, I don't know because I haven't built anything like this yet. My intuition is that yes, it'll be much more tightly integrated. Is that, you know, there will be language models involved, but they're doing relatively small atomic tasks. They're not solving the whole problem and then you check it. It's like they're doing atomic subparts of tasks which are integrated to like, so I expect a co, I like to talk about co- systems. They're not models, they're systems. It's like, in a way, when I think about designing a co- M system, what I'm trying to say is I'm kind of trying to integrate back software architecture and like distributed systems and like traditional computer science thinking into AI design.

• 1:16:21Speaker 02

  I'm saying that the thing that humans do to do science is not magical. This is a software process. This is a cognitive computational process that is not, it's not sacred. Like this is a thing you can decompose. And I'm also claiming further, you can decompose iteratively. You don't have to decompose everything at once because we have these crazy black box things which can do lots of the hard parts. So you could start with just using those. Like the way I think about thinking co- M's is you start with just a big black box. You start with just a big language model, try to get it to do what you want. Next step is you're like, all right, well, how can I break this down into smaller things that I can understand? How can I break, how can I call the model, how can I make the model do less of the work?

• 1:17:05Speaker 02

  I like to think of it as you're trying to move as much of the cognition, not just your computation, about cognition as possible from black boxes into white boxes. You want as much as possible of the process of generating the blueprint to happen inside of processes that the human understands, that you can understand, that you can check. Then you also have to bound the black boxes. Like if you have all these great white boxes, but there's still a big black box at the end that does whatever it wants, you're still screwed. So this is why the specification and the causal story is important. So ultimately, what I expect a powerful CoIM system to look like is it will be a system

• 1:17:49Speaker 02

  of many moving parts that are, that have clear interfaces between them. You have clear specification, a story about what, how these systems interact, why you should trust what those outputs are, that they fulfill the safety requirements you want them to require, what, why, you know, how these things work and why these systems are implementing the kind of human epistemology that humans use when they're solving science. Not, they're not solving, they're not implementing an arbitrary algorithm that solves science. They're implementing the human algorithms that solve science. And this is different from like GPT systems.

• 1:18:30Speaker 02

  GPT systems, I expect, will eventually learn how to do science and to partially, they already can. But I don't expect by default that they will do it the way humans do. Cause I think there's many ways you can do science. And what we want is with CoIM, therefore cognitive emulation, is you want to emulate the way humans do this. The reason we want to do this is, is because A gives us bounds. We won't have these crazy things that we can't understand. We know, we should kind of deal with human levels, right? Like we know how humans work, we're human level. We can deal with human level things to a large degree. And we, you know, it makes the system understandable. It makes it, it makes it, because it's a causal story that is human readable and human checkable as necessary.

• 1:19:15Speaker 02

  Of course, in the ideal world, your specification should be so good that you don't need to check it once you've built it. Like. Any safety proposal that involves AGI has to be so good that you never have to run the system to check it. If you have to do empirical testing on AGI, you're screwed. Your specification should be so good that you know ahead of time that once you turn it on, it'll be okay.

• 1:19:40Gus Docker?

  Isn't that an impossibly difficult standard too? I mean, this seems almost impossible to live up to.

• 1:19:47Speaker 02

  I totally disagree. Like, I just totally disagree. I think it's hard, but I don't think it's impossible by any means. So because again, this is not a formal guarantee. I'm talking about a story, a causal story, the specification. Like, this is like saying, is it impossible to have a system where passwords don't leak? And I'm like, sure, in the limit, yes. You know, if your if your enemy is, you know, magical gods from the future, who can you directly, you know, exfiltrate your CPU states from, you know, a thousand miles away, then yeah, you're screwed. Then yeah, yeah, in that case, you are screwed. And I expect I expect similarly, this is why the boundedness is so important and these assumptions are so important. If you have, you know, GPT Omega, you know, row hammering, you know, super god. But then, yeah, you're screwed.

• 1:20:33Speaker 02

  Then I do think it is impossible. But that's not what I'm talking about. This is why the boundedness to human level is so important. It is so important that none, no parts of these systems are superhuman. And that you don't allow superhuman levels of things. You want to aim for human and only human. Because this is something we can make assumptions about. You cannot make assumptions about superhuman intelligence. Because we have no idea how it works. We have no idea what it's capable of. We don't know what the limits are. So if you if you made a coem superhumanly intelligent, which I expect to be straightforwardly possible, but just like changing variables, then you're screwed. Then your story won't work and then you die.

• 1:21:10Gus Docker?

  Should we think of coems as companies or research labs where each, say, employee is bounded and thinks like a human? And they all report to the CEO and every every step is is understandable by the CEO, which is analogous to the to the human user of the coem system?

• 1:21:30Speaker 02

  I think this is a nice metaphor. I don't know if that's literally how they will be built. But I think this is a nice metaphor for how how I would think about this. If you had a really good coem. A really good full CoM system. What it should do is, is that it shouldn't produce a thousand X AGI. And it shouldn't, it shouldn't make the user a thousand X smarter. What it should do is, it should make the user function like a thousand one X AGIs. It should make you paralizable, not serially intelligent. Because if you're thousand X intelligent, who knows? Like, that is dangerous. But what it should do is, is like a company, like the CEO is paralyzing himself across a large company, or there are a thousand smart people. That's what I want CoMs to do.

• 1:22:14Speaker 02

  I wanted to paralyze the agency, the intelligence of the human into a thousand parallel one X AGIs. That are not smarter than human, that are bounded, that are understandable, that you can understand, and that you have this causal story why you should trust them.

• 1:22:29Gus Docker?

  And, and that's the, that's the key point, I think. Because for each subcomponent of this CoM systems, each employee in the research lab or the company, how do we know whether they operate in a human-like way? It seems like this could be asked, because we could ask this of the, the system at, at, at large, but we could also ask this of a subcomponent. It, it seems that we have the same problem for both systems.

• 1:22:51Speaker 02

  This is quite difficult, but basically my intuition is, is that... So the problem with talking about employees and where the corporation thing doesn't quite work is that another unfortunate side effect of calling it an emulation is that this implies more than what I mean. When I talk about a CoM emulating a human, I don't mean a person. I don't mean it's emulating a person. It's not, it doesn't have emotions, it doesn't have values, it doesn't have an identity. It's more like emulating a platonic human, or like a platonic, like,

• 1:23:29Speaker 02

  neocortex. It's more like a platonic cortex with no emotions, no volition, no goals. It's like an optimizer with no goal function. It's like, it's a completely, you know, it's just like ripped out all the emotions, all of the things. It's just a thinking blob. And then you plug in the user as a source of agency. The A, the human becomes the emotional motivational center. The CoM is just thinking blob. And there are reasonable people who are not sure this is possible. Others do think it's possible. So this is like, this is where this overlaps with the cyborg research agenda. And in case you've heard of that from like Janice and other people,

• 1:24:14Speaker 02

  where the idea is you hook humans up to AIs to control them to make humans super smart. Where CoM differs from the cyborg agenda is that in the cyborg agenda, they hook humans up to alien cortex. Well, I say, no, we build human cortex that works the way human cortex, or emulation of human cortex. The implementation is not human, but the abstraction layer exposed is human. And you hook that up to a user. You have the user used emulated human cortex. It's not simulated human cortex. That would be even better, but that's probably too hard. I don't think we can do simulated cortex in a reasonable time. But if we could, that'd be awesome. If we do like whole brain emulation, that'd be awesome. But I just don't think it's, I think it's too hard.

• 1:24:57Speaker 02

  So the final product, if it would work, would look something like the user, you know, using this, this emulated emotionless, just like raw cognition stuff to amplify the things they wanted to do. There's some pretty interesting, to also just add to that metaphor. There's some very interesting experiments, for example, with decorticated rats, where in rats, if you remove the cortex, so like the thinking part of the brain, the wrinkly part, they're mostly still kind of normal. They walk around, they eat food, they, you know, sleep, they play. Like you don't really see that much of a difference because the emotional parts are still there.

• 1:25:42Speaker 02

  If you move the emotional, the motivational part, they just become completely kineptonic. They just die. So like the human brain is kind of is similar, it has the same structure. We have this big wrinkly part, which is, you know, something like a big thinking blob that does like unsupervised learning. And then you have like deeper like, you know, motivational circuits, emotions, instincts, hardcoded stuff, which sit below that. And the cortex learns from these things and does actions steered by these emotional centers. So it's not exactly system one, system two, it's a bit more fuzzy than that. It's also like, yeah, it's like, just as like an intuition.

• 1:26:23Gus Docker?

  Yeah, so let's say we have this CoM system, which is where the metaphor is a company or a research lab with a lot of employees with a normal human IQ as opposed to having one system with an IQ of a thousand, whatever that means. Isn't there still a problem of this system just thinking much, much faster than we do? So imagine being in a competition with a company where all of the employees just think a hundred times faster than you do. Is one speed alone make the system capable and therefore dangerous?

• 1:26:57Speaker 02

  So there is a difference between speed and serial depth. So this is, I'm not sure. My feeling is that speed is much less a problem than serial depth. So by serial depth, I mean, how many consecutive steps along a reasoning chain can a system do? I think this is very dangerous. I think serial depth is where most, not almost, but like a very large percentage of the danger comes from. I think the thing that makes super fast thinking things so dangerous is because they can reach unprecedented serial depths of reflection and thinking and self-improvement much, much, much, much faster. And yes, I expect that if you build a CoM system that includes some component that can self-improve

• 1:27:41Speaker 02

  and you allow it to just do that, then yeah, it's not safe, you fucked up. Like if you build it that way, you have failed the specification, you're screwed. Probably.

• 1:27:51Gus Docker?

  I wonder if perhaps building CoM systems might be a massive strength on the market. So I I wonder if perhaps there's there's there will be an intense demand for systems that act human in human-like ways. Because, I mean, the CEOs of huge companies would probably want to be able to talk to these systems in a in a normal way to understand how these systems work before they're deployed. There's a there's a sense in which CoMs will perhaps integrate nicely in a in a world that's already built for humans. So do you do you think there's some there's some kind of win here where there will be a lot of demand for human likeness in AI systems?

• 1:28:35Speaker 02

  There is a optimistic and there's a pessimistic version of this. The optimistic version is, well, yeah, obviously, like, we want things that are safe and that do what we want and we can understand. Obviously, like, the best product that could ever be built is an aligned AGI. That is the best product. There there is nothing better. That is the best product. CoMs are not fully aligned superintelligence. I never claimed they would be. And if anyone used them that way, it would be really bad. You should not use them that way. That is never the goal. You should use these things to do science, to speed up, you know, nanotechnology, to create whole brain emulations, or to, you know, do more, you know, work on alignment or whatever, you know?

• 1:29:18Speaker 02

  You should not use them to like, you know, I'll just let the CoM go optimize the whole world or whatever. No, that's not what you're supposed to do. And if you do that, you die. And bad things happen. So would there be demand for these systems? I expect, yeah, like, like, this is an incredibly powerful system still. If you use a CoM and you use it correctly, you get, yeah, like, imagine you could just have a perfectly loyal company that does everything you want it to do, staffed exclusively by John von Neumanns. Like, that is unimaginably great. Of course, there's a pessimistic version. The pessimistic version is is like, lol, doesn't matter. By that point, you're going to have a hundred ex John von Neumann GPT-F, you know? You know, which then promptly destroys the world.

• 1:30:02Gus Docker?

  But won't there be demand for safety? I mean, if from from governments, from from from companies, who would deploy a system that is uncontrolled and is?

• 1:30:16Gus Docker?

  You know, where we can't reason about it, we don't know how it works. If we get to a point where these systems are much more powerful than they are today.

• 1:30:25Speaker 02

  Hopefully. So a lot of the work I do nowadays is sadly not in the technical realm, but is in policy and communications. I've been talking to a lot of journalists and politicians and so on for exactly these reasons. It's because we have to create the demand to, for safety. Is that like, currently, let me, let me be clear about what the currency of the world here is. The way the current world is looking is we are in a death race towards the bottom, you know, careening towards a precipice at full speed and we won't see the precipice coming until we're over it. And this is led almost entirely by a very, very small number of people that are techno optimists, techno utopians, you know, people in the Bay Area and London,

• 1:31:10Speaker 02

  who are extremely optimistic or at least willfully denial about how bad things are. Or that are, you know, can galaxy brain themselves and say, well, it's a race, it's a race, it's not my fault, so I just have to do it anyways. Like, whatever. I'm kind of at the point that I don't really care why people are doing these things. I only care that they're happening. Like, people are doing these things that are extremely dangerous and this is a very small number of people. And there's this myth among these people that they're like, oh, we have to do it, you know, people want it. This is just false. If you talk to normal people and you explain to them what these people believe, like, when, when most people hear the word AGI,

• 1:31:52Speaker 02

  what they imagine is human-like AI. They think, you know, it's like your robot buddy, he thinks like you, he's not really smarter, but you know, he's like, you know, he has human emotions. It's like, when, that is not what people at, you know, organizations like OpenAI or DeepMind think when they say the word AGI. When they say AGI, they mean god-like AI. They mean a, you know, self-improving, non-human, you know, incredibly powerful system that can, you know, take over the government, can, you know, destroy the whole human race, et cetera. Now, whether or not you believe that personally, these people do believe this. They have publicly stated this on the record. These people do believe these things, this is what they're doing.

• 1:32:34Speaker 02

  And once you inform people of this, they're like, what the shit? Absolutely fucking not. What? What? No. Of course you can't build God AI. What the fuck are you doing? Where's the government? Like, how, how are we in a world? Where, you know, people can just like, you know, down in San Francisco can publicly talk about how they're gonna build systems. Have, you know, a 1%, 5%, 20%, 90%, whatever risk of killing everybody. That have, that will, you know, topple the US government, whatever. And actually work on this and get billions of funding. And the government is just like, cool. Like, we're in this, we are not in a stable equilibrium. And it is coming.

• 1:33:20Speaker 02

  It is now starting to float. People are starting to freak the fuck out. Because they're like, whole shit, like, A, this is possible, and B, absolutely fucking not. And this gives me some hope that we can slow down and buy some more time. Not sure, I don't think this saves us, but it can save, get us some time.

• 1:33:38Gus Docker?

  So if we, if we don't take the, the, the fast road towards the precipice and we succeed in building COMs instead, for example. Is there still a difficult unsolved problem, namely going from an aligned, a human-like COM to an aligned superintelligence? Is, is there still some, something that's, that's very difficult to solve there and perhaps? Perhaps the, the, the core of the problem is, is, is still unsolved?

• 1:34:04Speaker 02

  Oh, absolutely. Like, assuming we're not dead. We have a land government, we have a, you know, safe, not aligned, safe government system. We're not out of the woods. Then the world gets really messy. Like, look, the world is gonna get really messy. This is the least weird your life is going to be for the rest of your life. Things are not going back to quote unquote "normal". Things are only gonna get weirder. This is the least bad things are going to be for the rest of your life. Things are only going to get weirder from here. There is a power struggle that is coming, right? And this is, there is no way to prevent this because it is about power. There is these incredibly powerful systems being built. There are people racing for incredible powers.

• 1:34:49Speaker 02

  There is, there, there is conflict. There is fight. There is politics. There is war. These things are inevitable. There is no way this goes cleanly. There is no way that things will go smoothly and people get along and things be fine. No, there is going to be, you know, unimaginable levels of, you know, struggle to decide what will happen and how things will happen. And most of those ways are not going to end well. I think most of the way I think I have said this before and I will say it again. I do not expect us to make it out of this century alive. I don't even, I'm not even sure we'll get out of this decade. Um, I expect that by default, things will go very, very badly. They don't have to.

• 1:35:33Speaker 02

  So the weird thing, which some things to myself is that we, we live in a very strange timeline where we're in a bad timeline. Like, let me be clear, we're in a bad timeline. Things are going really bad right now, but we haven't yet lost. But it's quite curious. There are many timelines where we just like, it's so over. Like it's just totally over. Nothing you can do, like, you know, everyone is on board with building AGI as fast as possible. You know, the military gets involved. They're all gung ho about it or whatever, and nothing can stop it. Or, you know, World War Three breaks out in both places, raised AGI and whatever. Like, if that was the case, then just like, it would be so over. But it's not over. It's not over yet. It might be soon though. It might be soon.

• 1:36:18Speaker 02

  But currently, yeah, let's say we get coordination, we slow things down, build co-EM systems. Let's also assume furthermore that we keep these systems secure and safe and they don't get immediately stolen by every unscrupulous actor in the world. Then you have very powerful systems, which you can do very powerful things. In particular, you can create great economic value. So one of the first things I would do with a co-EM system, if I had one, is I would produce massive amounts of economic value and trade with everybody. I would trade with everybody. I'd be like, look, whatever you want in life, I'll get it to you. In return, don't build AGI. You know, I'll get cure, give you the cure for cancer, lots of money, you know, whatever you want. I'll get you whatever you want and return, you don't build AGI. That's the deal. It's like the deal I offer to everybody.

• 1:37:03Speaker 02

  And then, you know.

• 1:37:06Speaker 02

  Conditioning on the slim probability of this going well, which I think is slim. You know, like probably the way it would actually work is, you know, we like, you know, fuse with one or more national governments, you know, have, you know, like, you know, work closely together with authority figures, with politicians, military intelligence services, et cetera, to keep things, you know, secure and then slowly and then work securely on the hard problem. So now we have the ability to do, you know, you know, a thousand, you know, thousand times John von Neumann's working on alignment theory, on like formally verified safety and so on. We have, you know, we trade with all the variant players to get them to slow down and coordinate and then and have, you know, the backing of, you know, government

• 1:37:51Speaker 02

  or military intelligence service security so that bad actors are, you know, interrupted. That's the all like kind of the only way I see things going well. If all, as you could, as you could tell, as the usual saying goes, any plan that requires more than two things to go right will never work. Unfortunately, this is a plan that requires more than two things to go right. So I don't expect this plan to work.

• 1:38:14Gus Docker?

  Yeah, but let's, let's hope we get there anyway. Connor, thank you for coming on the podcast. It's been super interesting for me.

• 1:38:20Speaker 02

  Pleasure as always.