#371 – Max Tegmark: The Case for Halting AI Development

Max Tegmark is a physicist and AI researcher at MIT, co-founder of the Future of Life Institute, and author of Life 3.0: Being Human in the Age of Artificial Intelligence. Please support this podcast by checking out our sponsors: – Notion: https://notion.com – InsideTracker: https://insidetracker.com/lex to get 20% off – Indeed: https://indeed.com/lex to get $75 credit EPISODE LINKS: Max’s Twitter: https://twitter.com/tegmark Max’s Website: https://space.mit.edu/home/tegmark Pause Giant AI Experiments (open letter): https://futureoflife.org/open-letter/pause-giant-ai-experiments Future of Life Institute: https://futureoflife.org Books and resources mentioned: 1. Life 3.0 (book): https://amzn.to/3UB9rXB 2. Meditations on Moloch (essay): https://slatestarcodex.com/2014/07/30/meditations-on-moloch 3. Nuclear winter paper: https://nature.com/articles/s43016-022-00573-0 PODCAST INFO: Podcast website: https://lexfridman.com/podcast Apple Podcasts: https://apple.co/2lwqZIr Spotify: https://spoti.fi/2nEwCF8 RSS: https://lexfridman.com/feed/podcast/ YouTube Full Episodes: https://youtube.com/lexfridman YouTube Clips: https://youtube.com/lexclips SUPPORT & CONNECT: – Check out the sponsors above, it’s the best way to support this podcast – Support on Patreon: https://www.patreon.com/lexfridman – Twitter: https://twitter.com/lexfridman – Instagram: https://www.instagram.com/lexfridman – LinkedIn: https://www.linkedin.com/in/lexfridman – Facebook: https://www.facebook.com/lexfridman – Medium: https://medium.com/@lexfridman OUTLINE: Here’s the timestamps for the episode. On some podcast players you should be able to click the timestamp to jump to that time. (00:00) – Introduction (07:34) – Intelligent alien civilizations (19:58) – Life 3.0 and superintelligent AI (31:25) – Open letter to pause Giant AI Experiments (56:32) – Maintaining control (1:25:22) – Regulation (1:36:12) – Job automation (1:45:27) – Elon Musk (2:07:09) – Open source (2:13:39) – How AI may kill all humans (2:24:10) – Consciousness (2:33:32) – Nuclear winter (2:44:00) – Questions for AGI

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Lex Fridman?

  The following is a conversation with Max Tegmark, his third time on the podcast. In fact, his first appearance was episode number one of this very podcast. He is a physicist and artificial intelligence researcher at MIT, co-founder of Future Life Institute and author of Life 3.0: Being Human in the Age of Artificial Intelligence. Most recently, he's a key figure in spearheading the open letter calling for a six-month pause on giant AI experiments like training GPT-4. The letter reads, "We're calling for a pause on training of models larger than GPT-4 for six months. This does not imply a pause or ban on all AI research and development,

• 0:45Lex Fridman?

  or the use of systems that have already been placed on the market. Our call is specific and addresses a very small pool of actors who possesses this capability." The letter has been signed by over 50,000 individuals, including 1,800 CEOs and over 1,500 professors. Signatories include Joshua Benjio, Stuart Russell, Elon Musk, Steve Wozniak, Yuval Noah Harari, Andrew Yang, and many others. This is a defining moment in the history of human civilization, where the balance of power between human and AI begins to shift. And Max's mind and his voice is one of the most valuable and powerful in a time like this.

• 1:28Lex Fridman?

  His support, his wisdom, his friendship has been a gift I'm forever deeply grateful for. And now, a quick few-second mention of each sponsor. Check them out in the description. It's the best way to support this podcast. We've got Notion for project and team collaboration, InsideTracker for biological data, and Indeed for hiring. Choose wisely, my friends. Also, speaking of hiring, if you want to work with our amazing team, we're always hiring, whether it's through Indeed or otherwise. Go to lexfridman.com/hiring. And now, on to the full ad reads. As always, no ads in the middle. I try to make this interesting, but if you must skip them, please still check out our sponsors.

• 2:12Lex Fridman?

  I enjoy their stuff. Maybe you will, too.

• 2:16Lex Fridman?

  This show is brought to you by Notion. I've spoken endlessly about how amazing Notion is, how everybody, all the cool kids are recommending it for just basic note taking, but there's so, so much more. It's the collaborative aspect of it, the project management aspect of it, the wikis, the document sharing, all of that, all in a simple, powerful, beautifully designed solution. What can I say? On top of this, there's the Notion AI tool. This is the best integration of large language models into a productivity note-taking tool. There are so many amazing features. I mean, it's just endless. Go to their website. You can generate entire presentations and reports based on a to-do list. You can summarize stuff. You can shorten stuff.

• 3:01Lex Fridman?

  You can generate tables based on the description. You can write a summary. You can expand the text. You can change the style of the text. You can fix spelling and grammar. You can translate. You can use simpler language, more complicated language, change the tone of the voice, make it shorter, longer. Like I said, everything is just so easy to play around with and all of it, no matter what you're doing, will challenge you to think how you write. It will challenge you to expand the style of your writing. It will save you a lot of time, of course, but I just think it makes you a better thinker and productive being in this world. And I think that's such a great integration of AI into the productivity workflow.

• 3:42Lex Fridman?

  To me, it's not enough for a large language model to be effective at answering questions and having good dialogue. You have to really integrate it into the workflow and Notion, better than anybody else I've seen, has done that. So if that's interesting to you, Notion AI helps you work faster, write better and think bigger, doing tasks that normally take you hours in just minutes. Try Notion AI for free when you go to notion.com/lex. That's all lowercase, notion.com/lex to try the power of Notion AI today.

• 4:14Lex Fridman?

  This show is also brought to you by InsideTracker, a service I use to track biological data. It's really good to do that kind of thing regularly, to look at all the different markers in your body and to understand what could be made better through lifestyle, through diet changes. It's kind of obvious that decisions about your life should be made based on the data that comes from your own body, not some kind of population study, although those are good, not some spiritual guru, although those are good, not some novel, whether it's Harry Potter or Dostoevsky, although those are sometimes good, not your relative who says, "I heard a guy say that a guy does this thing" that is very bro-soundsy sounding,

• 4:58Lex Fridman?

  although sometimes it turns out to be pretty effective. Overall, the best decisions about your life should be based on the things that come from your own body. InsideTracker uses algorithms to analyze your blood data, DNA data, fitness tracker, all that kind of stuff to give you recommendations. You should be doing it. You should be doing it regularly. So it's not just a one-time thing, but regularly, over time, you see what changes led to improvements in the various markers that come from your body. Get special savings for a limited time when you go to insidetracker.com/lex. This show is also brought to you by Indeed, a hiring website. I think the most important thing in life, not to quote Corin the Barbarian, because that would be very inappropriate

• 5:43Lex Fridman?

  to quote at this moment. And it's not actually accurate at all as a reflection of what's important in life. It's only, has comedic value. What I really want to say about what's important in life is the people you surround yourself with. And we spend so much of our time in the workplace, seeking solutions to very difficult problems together, passionately pursuing ambitious goals, sometimes impossible goals. That is a source of meaning, a sort of happiness for people. And I think part of that happiness comes from the collaboration with other human beings. The...

• 6:22Lex Fridman?

  Sort of professional depth of connection that you have with other human beings of being together through the grind and surviving and accomplishing the goal or failing in a big epic way, knowing that you have tried together. And so doing that with the right team, I think, is one of the most important things in life. So you should surround yourself with the right team. If you're looking to join a team, you should be very selective about that. Or if you're looking to hire a team, you should be very selective about that and use the best tools of the job. I've used Indeed many, many times throughout my life for the teams I've led. Don't overspend on hiring. Visit indeed.com/lex to start hiring now. That's indeed.com/lex.

• 7:05Lex Fridman?

  Terms and conditions apply. This is the Lex Freedom Podcast. To support it, please check out our sponsors in the description. And now, dear friends, here's Max Tagmark.

• 7:34Lex Fridman?

  You were the first ever guest on this podcast, episode number one. So first of all, Max, I just have to say, thank you for giving me a chance. Thank you for starting this journey. It's been an incredible journey. Just thank you for sitting down with me and just acting like I'm somebody who matters, that I'm somebody who's interesting to talk to. And thank you for doing it. That meant a lot.

• 7:58Max Tegmark?

  Thanks to you for putting your heart and soul into this. I know when you delve into controversial topics, it's inevitable to get hit by what Hamlet talks about, the slings and arrows and stuff. And I really admire this. It's in an era, you know, where YouTube videos are too long and now it has to be like a 20 minute TikTok, 20 second TikTok clip. It's just so refreshing to see you going exactly against all of the advice and doing these really long form things and that people appreciate it, you know? Reality is nuanced and thanks for.

• 8:32Max Tegmark?

  Sharing it that way.

• 8:33Lex Fridman?

  So let me ask you again, the first question I've ever asked on this podcast. Episode number one, talking to you. Do you think there's intelligent life out there in the universe? Let's revisit that question. Do you have any updates? What's your view when you look out to the stars?

• 8:50Max Tegmark?

  So when we look out to the stars, if you define our universe the way most astrophysicists do, not as all of space, but the spherical region of space that we can see with our telescopes, from which light has had time to reach us since our big bang. I'm in the minority. I estimate that we are the only life in this spherical volume that has invented internet, radios, gotten our level of tech. And if that's true,

• 9:23Max Tegmark?

  then it puts a lot of responsibility on us to not mess this one up. Because if it's true, it means that life is quite rare. And we are stewards of this one spark of advanced consciousness, which if we nurture it and help it grow, eventually life can spread from here out into much of our universe. And we can have this just amazing future. Whereas if we instead are reckless with the technology we build and just snuff it out due to stupidity or infighting, then maybe the rest of cosmic history in our universe was just going to be a place for empty benches. But I do think that we are actually very likely

• 10:07Max Tegmark?

  to get visited by aliens, alien intelligence, quite soon. But I think we are going to be building that alien intelligence.

• 10:15Lex Fridman?

  So we're going to give birth to an intelligent alien civilization.

• 10:22Lex Fridman?

  Unlike anything that human, the evolution here on Earth was able to create in terms of the path, the biological path it took.

• 10:31Max Tegmark?

  Yeah, and it's gonna be much more alien than a cat or even the most exotic animal on the planet right now, because it will not have been created through the usual Darwinian competition where it necessarily cares about self-preservation, afraid of death, any of those things. The space of alien minds that you can build is just so much vaster than what evolution will give you. And with that also comes a great responsibility for us to make sure that the kind of minds we create are this kind of minds that it's good to create, minds that will share our

• 11:14Max Tegmark?

  values and be good for humanity and life, and also create minds that don't suffer.

• 11:22Lex Fridman?

  Do you try to visualize the full space of alien minds that AI could be? Do you try to consider all the different kinds of intelligences, sort of generalizing what humans are able to do to the full spectrum of what intelligent creatures, entities could do?

• 11:40Max Tegmark?

  I try, but I would say I fail. I mean, it's very difficult for a human mind to really grapple with something still completely alien, even for us, right? If we just try to imagine, how would it feel if we were completely indifferent towards death or individuality?

• 12:05Max Tegmark?

  Even if you just imagine that, for example, you could just copy my knowledge of how to speak Swedish, boom, now you can speak Swedish. And you could copy any of my cool experiences and then you could delete the ones you didn't like in your own life, just like that. It would already change quite a lot about how you feel as a human being, right? You probably spend less effort studying things if you just copy them and you might be less afraid of death, because if the plane you're on starts to crash, you'd just be like, oh, shucks, I haven't backed my brain up for four hours, so I'm gonna lose all these wonderful experiences of this flight.

• 12:49Max Tegmark?

  We might also start feeling

• 12:52Max Tegmark?

  more like compassionate, maybe with other people, if we can so readily share each other's experiences and our knowledge and feel more like a hive mind. It's very hard, though. I really feel very humble about this, to grapple with it, how it might actually feel. The one thing which is so obvious, though, it's, I think, is just really worth reflecting on is because the mind space of possible intelligences is so different from ours, it's very dangerous if we assume they're going to be like us or anything like us.

• 13:27Lex Fridman?

  Well, there's a, the entirety of human written history has been through poetry, through novels, been trying to describe through philosophy, trying to describe the human condition and what's entailed in it, like Jessica said, fear of death and all those kinds of things. What is love and all of that changes.

• 13:47Max Tegmark?

  Yeah.

• 13:48Lex Fridman?

  If you have a different kind of intelligence.

• 13:50Max Tegmark?

  Yeah.

• 13:50Lex Fridman?

  All of it, the entirety, all those poems, they're trying to sneak up to what the hell it means to be human. All of that changes. How AI concerns and existential crises that AI experiences, how that clashes with the human existential crisis, the human condition.

• 14:09Max Tegmark?

  Yeah.

• 14:09Lex Fridman?

  That's hard to, hard to fathom, hard to predict.

• 14:12Max Tegmark?

  It's hard, but it's fascinating to think about also. Even in the best case scenario where we don't lose control of, over the ever more powerful AI that we're building to other humans whose goals we think are horrible and where we don't lose control to the machines and AI provides the things we want, even then, you get into the questions you touched here, you know, maybe it's the struggle that it's actually hard to do things is part of the things that gives us meaning as well, right? So, for example, I found it so shocking that this new Microsoft GPT-4 commercial that

• 14:52Max Tegmark?

  they put together has this woman talking about, showing this demo of how she's going to give a graduation speech to her beloved daughter and she asks GPT-4 to write it.

• 15:05Max Tegmark?

  If it's frigging 200 words or so, if I realized that my parents couldn't be bothered struggling a little bit to write 200 words and outsource that to their computer, I would feel really offended, actually. And so I wonder if eliminating too much of the struggle from our existence,

• 15:26Max Tegmark?

  do you think that would also take away a little bit of what?

• 15:31Lex Fridman?

  Means to be human, yeah.

• 15:33Max Tegmark?

  Yeah.

• 15:34Lex Fridman?

  We can't even predict. I had somebody mention to me that they used, they started using ChatGPT with a 3.5 and not 4.0

• 15:46Lex Fridman?

  to write what they really feel to a person. And they have a temper issue and they're basically trying to get ChatGPT to rewrite it in a nicer way, to get the point across, but rewrite it in a nicer way. So we're even removing the inner asshole from our communication. So I don't, you know, there's some positive aspects of that, but mostly it's just the transformation of how humans communicate. And it's scary because so much of our society is based on this glue of communication. And if we're now using AI as the medium of communication that does the language for

• 16:28Lex Fridman?

  us, so much of the emotion that's laden in human communication, so much of the intent that's going to be handled by outsourced AI, how does that change everything? How does it change the internal state of how we feel about other human beings? What makes us lonely? What makes us excited?

• 16:46Max Tegmark?

  Yeah.

• 16:47Lex Fridman?

  What makes us afraid? How we fall in love? All that kind of stuff.

• 16:50Max Tegmark?

  Yeah. For me personally, I have to confess, the challenge is one of the things that really makes my life feel.

• 16:58Max Tegmark?

  Meaningful, you know, if I go hiking mountain with my wife Maya, I don't want to just press a button and be at the top. I want to struggle and come up there sweaty and feel wow, we did this in the same way. I want to constantly work on myself to become a better person. If I say something in anger that I regret, I want to go back and really work on myself rather than just tell an AI to from now on always filter what I write. So I don't have to work on myself because then I'm not growing.

• 17:35Lex Fridman?

  Yeah, but then again, it could be like with chess and AI. Once it significantly, obviously supersedes the performance of humans, it will live in its own world and provide maybe a flourishing civilizations for humans. But we humans will continue hiking mountains and playing our games, even though AI is so much smarter, so much stronger, so much superior in every single way, just like with chess.

• 18:00Max Tegmark?

  Yeah.

• 18:01Lex Fridman?

  So that, that, I mean, that's one possible hopeful trajectory here is that humans will continue to human. Uh, and AI will just be a kind of

• 18:17Lex Fridman?

  a, a medium that enables the human experience to flourish.

• 18:23Max Tegmark?

  Yeah.

• 18:26Max Tegmark?

  I would phrase that as rebranding ourselves from Homo sapiens to Homo sentience. You know, right now it's sapiens, the ability to be intelligent. We've even put it in our species name. So we're, we're branding ourselves as the smartest.

• 18:43Lex Fridman?

  Yeah.

• 18:43Max Tegmark?

  Information processing entity on the planet. That's clearly going to change if AI continues ahead. So maybe we should focus on the experience instead, the subjective experience that we have Homo sentience and, and say, that's what's really valuable, the love, the connection, the other things.

• 19:06Max Tegmark?

  Get off our high horses and get rid of this hubris that only we can do integrals.

• 19:14Lex Fridman?

  So consciousness, the subjective experience is a fundamental value to what it means to be human. Make that, make that the priority.

• 19:24Max Tegmark?

  That feels like a hopeful direction to me, but that also requires more compassion, not just towards other humans, because they happen to be the smartest on the planet, but also towards all our other fellow creatures on this planet. And I personally feel right now we're treating a lot of farm animals horribly, for example, and the excuse we're using is, oh, they're not as smart as us. But if we admit that we're not that smart in the grand scheme of things either in the post-AI epoch, you know, then surely we should value the subjective experience of a cow also.

• 19:57Lex Fridman?

  Well, allow me to briefly look at the book, which at this point is becoming more and more visionary that you've written, I guess, over five years ago, Life 3.0. So first of all, 3.0, what's 1.0, what's 2.0, what's 3.0? And how's that vision sort of evolved? The vision in the book evolved to today?

• 20:19Max Tegmark?

  Life 1.0 is really dumb, like bacteria, in that it can't actually learn anything at all during the lifetime. The learning just comes from this genetic process from one generation to the next. Life 2.0 is us and other animals which have brains, which can learn during their lifetime a great deal. And so, and, you know, you were born without being able to speak English. And at some point, you decided, hey, I want to upgrade my software. Let's install an English-speaking module.

• 20:56Lex Fridman?

  So you did?

• 20:58Max Tegmark?

  And Life 3.0, which does not exist yet, can replace not only its software the way we can, but also its hardware. And that's where we're heading towards at high speed. We're already maybe 2.1 because we can, you know, put in a an artificial knee, pacemaker, et cetera, et cetera. And if Neuralink and other companies succeed, we'll be life 2.2, et cetera. But what the company's trying to build AGI or trying to make is of course full 3.0 and you can put that intelligence in something that also has no biological basis whatsoever.

• 21:41Lex Fridman?

  So less constraints and more capabilities, just like the leap from 1.0 to 2.0. There is nevertheless, you're speaking so harshly about bacteria, so disrespectfully about bacteria. There is still the same kind of magic there that permeates life 2.0 and 3.0. It seems like maybe the thing that's truly powerful about life, intelligence and consciousness was already there in 1.0. Is it possible?

• 22:12Max Tegmark?

  I think we should be humble and not be so quick to make everything binary and say either it's there or it's not. Clearly, there's a great spectrum and there is even controversy about whether some unicellular organisms like amoebas can maybe learn a little bit, you know, after all. So apologies if I offended any bacteria here. It wasn't my intent. It was more that I wanted to talk up how cool it is to actually have a brain.

• 22:39Lex Fridman?

  Yeah.

• 22:39Max Tegmark?

  Where you can learn dramatically within your lifetime.

• 22:43Lex Fridman?

  Typical human.

• 22:44Max Tegmark?

  And the higher up you get from 1.0 to 2.0 to 3.0, the more you become the captain of your own ship, the master of your own destiny, and the less you become a slave to whatever evolution gave you, right? By upgrading your software, we can be so different from previous generations and even from our parents, much more so than even a bacterium, you know, no offense to them. And if you can also swap out your hardware, take any physical form you want, of course, really the sky's the limit.

• 23:15Lex Fridman?

  Yeah, so the, it accelerates the rate at which you can perform the computation that determines your destiny.

• 23:23Max Tegmark?

  Yeah, and I think it's worth commenting a bit on what you means in this context also. If you swap things out a lot, right? This is controversial, but my.

• 23:36Max Tegmark?

  Current

• 23:38Max Tegmark?

  understanding is that, you know, life is best thought of not as a bag of meat or even a bag of elementary particles, but rather as a system which can process information and retain its own complexity, even though nature is always trying to mess it up. So it's all about information processing. And

• 24:05Max Tegmark?

  that makes it a lot like something like a wave in the ocean, which is not, it's water molecules, right? The water molecules bob up and down, but the wave moves forward. It's an information pattern. In the same way, you Lex, you're not the same atoms as during the first interview you did with me. You've swapped out most of them, but it's still you. And the information pattern is still there. And if you could swap out your arms and whatever, you can still have this kind of continuity. It becomes much more sophisticated sort of wave forward in time where the information lives on. I lost both of my parents since our last podcast.

• 24:49Max Tegmark?

  And it actually gives me a lot of solace that this way of thinking about them, they haven't entirely died because a lot of mommy and daddy's, sorry, I'm getting a little emotional here, but a lot of their values and ideas and even jokes and so on, they haven't gone away, right? Some of them live on, I can carry on some of them. And they also live on a lot of other, and a lot of other people. So in this sense, even with life 2.0, we can, to some extent, already transcend our physical bodies and our death. And particularly if you can share your own information,

• 25:32Max Tegmark?

  your own ideas with many others, like you do in your podcast,

• 25:38Max Tegmark?

  then

• 25:40Max Tegmark?

  you know, that's the closest immortality we can get with our bio-bodies.

• 25:45Lex Fridman?

  You carry a little bit of them in you in some sense.

• 25:48Max Tegmark?

  Yeah.

• 25:50Lex Fridman?

  Do you miss them? You miss your mom and dad?

• 25:52Max Tegmark?

  Of course, of course.

• 25:53Lex Fridman?

  What did you learn about life from them? If it can take a bit of a tangent.

• 25:59Max Tegmark?

  On so many things.

• 26:03Max Tegmark?

  For starters, my fascination for math and the physical mysteries of our universe. You got a lot of that from my dad, but I think my obsession for really big questions and consciousness and so on, that actually came mostly from my mom. And what I got from both of them, which is very core part of really who I am, I think, is this

• 26:38Max Tegmark?

  just feeling comfortable with

• 26:42Max Tegmark?

  not buying into what everybody else is saying.

• 26:48Max Tegmark?

  Doing what I think is right.

• 26:52Max Tegmark?

  They both very much, you know, did their own thing. And sometimes they got flak for it and they did it anyway.

• 27:01Lex Fridman?

  That's why you've always been an inspiration to me, that you're at the top of your field and you're still, you're still willing to, uh, to tackle the big questions in your own way. You're one of, one of the people that represents MIT best to me. You've always been an inspiration in that. So it's good to hear that you got that from your mom and dad.

• 27:22Max Tegmark?

  Yeah, you're too kind, but, but yeah, I mean, the real, the good reason to do science is because you're really curious. You want to figure out the truth. If you think

• 27:34Max Tegmark?

  this is how it is and everyone else says, no, no, that's bullshit. And it's that way, you know,

• 27:40Max Tegmark?

  you stick with what you think is true. And, and even if everybody else keeps thinking it's bullshit, there's a certain,

• 27:50Max Tegmark?

  I always root for the underdog when I watch movies. And my, my dad once, I, one time, for example, when I wrote one of my craziest papers ever, our math, talking about our universe ultimately being mathematical, which we're not going to get into today. I got this email from a quite famous professor saying, this is not only bullshit, but it's going to ruin your career. You should stop doing this kind of stuff. I sent it to my dad. Do you know what he said?

• 28:14Lex Fridman?

  What'd he say?

• 28:15Max Tegmark?

  He replied with a quote from Dante. Segui il tuo corso e lasci dire la gente. Follow your own path and let the people talk.

• 28:25Max Tegmark?

  Go dad.

• 28:26Lex Fridman?

  Yeah.

• 28:26Max Tegmark?

  This is the kind of thing, he's dead, but that, that attitude is not.

• 28:32Lex Fridman?

  How did losing them as a man, as a human being, change you? How did it expand your thinking about the world? How did it expand your thinking about, you know, this thing we're talking about, which is humans creating another living sentient, perhaps, being?

• 28:51Max Tegmark?

  I think it mainly did two things. One of them, just going through all their stuff after they had passed away and so on, just drove home to me how important it is to ask ourselves, why are we doing this, things we do? Because it's inevitable that you look at some things they spent an enormous time on and you ask, in hindsight, would they really have spent so much time on this? Would they have done something that was more meaningful? So I've been looking more in my life now and asking, you know, why am I doing what I'm doing? And I feel it should either be something I really enjoy doing

• 29:34Max Tegmark?

  or it should be something that I find really, really meaningful because it helps humanity. And if it's none of those two categories, maybe I should spend less time on it, you know? The other thing is dealing with death up in person like this. It's actually made me less afraid of, even less afraid of other people telling me that I'm an idiot, you know, which happens regularly. And just live my life, do my thing, you know? And it's made it a little bit easier for me to focus on what I feel is really important.

• 30:18Lex Fridman?

  What about fear of your own death? Has it made it more real that this is, that this is something that happens?

• 30:27Max Tegmark?

  Yeah, it's made it extremely real. And I'm next in line in our family now, right? Me and my brother, my younger brother. But

• 30:37Max Tegmark?

  They both handled it with such dignity. That was a true inspiration also. They never complained about things and you know, when you're old and your body starts falling apart, it's more and more to complain about. They looked at what could they still do that was meaningful? And they focused on that rather than wasting time like talking about or even thinking much about things they were disappointed in. I think anyone can make themselves depressed if they start their morning by making a list of grievances. Whereas if you start your day with a little meditation and just things you're grateful for, you basically choose to be a happy person.

• 31:18Lex Fridman?

  Because you only have a finite number of days. You should spend them.

• 31:21Max Tegmark?

  Make it count.

• 31:22Lex Fridman?

  Being grateful.

• 31:23Max Tegmark?

  Yeah.

• 31:26Lex Fridman?

  Well, you do happen to be working on a thing which seems to have potentially some of the greatest impact on human civilization of anything humans have ever created, which is artificial intelligence. This is on the both detailed technical level and in a high philosophical level you work on. So you've mentioned to me that there's an open letter that you're working on.

• 31:53Max Tegmark?

  It's actually going live in a few hours. I've been having late nights and early mornings. It's been very exciting, actually. In short, have you seen Don't Look Up, the film?

• 32:09Lex Fridman?

  Yes, yes.

• 32:10Max Tegmark?

  I don't want to be the movie spoiler for anyone watching this who hasn't seen it, but if you're watching this, you haven't seen it, watch it. Because we are actually acting out. It's life imitating art. Humanity is doing exactly that right now, except it's an asteroid that we are building ourselves. Almost nobody is talking about it. People are squabbling across the planet about all sorts of things which seem very minor compared to the asteroid that's about to hit us, right? Most politicians don't even have their radars, this on their radar, or they think maybe in a hundred years or whatever. Right now, we're at a fork in the road. This is the most important fork humanity has reached in its over a hundred thousand

• 32:54Max Tegmark?

  years on this planet. We're building.

• 32:58Max Tegmark?

  Effectively, a new species that's smarter than us. It doesn't look so much like a species yet because it's mostly not embodied in robots. But that's a technicality which will soon be changed. And this arrival of artificial general intelligence that can do all our jobs as well as us. And probably shortly thereafter, superintelligence, which greatly exceeds our cognitive abilities. It's going to either be the best thing ever to happen to humanity or the worst. I'm really quite confident that there is not that much middle ground there.

• 33:33Lex Fridman?

  But it would be fundamentally transformative to human civilization.

• 33:37Max Tegmark?

  Of course, utterly and totally. Again, we branded ourselves as Homo sapiens because it seemed like the basic thing. We're the king of the castle on this planet. We're the smart ones. If we can control everything else, this could very easily change. We're certainly not going to be the smartest on the planet very long if AI, unless AI progress just halts. And we can talk more about why I think that's true because it's controversial. And then we can also talk about reasons you might think it's going to be the best thing ever. And the reason you think it's going to be the end of humanity, which is of course, super controversial.

• 34:19Max Tegmark?

  But what I think we can, anyone who's working on advanced AI can agree on is it's much like the film, Don't Look Up, and that it's just really comical how little serious public debate there is about it, given how huge it is.

• 34:42Lex Fridman?

  So what we're talking about is a development of currently things like GPT-4 and the signs it's showing of rapid improvement that may in the near term lead to development of superintelligent AGI, AI, general AI systems, and what kind of impact that has on society.

• 35:04Max Tegmark?

  Exactly.

• 35:04Lex Fridman?

  When that thing achieves general human level intelligence and then beyond that, general superhuman level intelligence.

• 35:15Lex Fridman?

  There's a lot of questions to explore here. So one, you mentioned halt. Is that the content of the letter?

• 35:23Max Tegmark?

  Yeah.

• 35:23Lex Fridman?

  To suggest that maybe we should pause the development of these systems?

• 35:27Max Tegmark?

  Exactly. So this is very controversial. From when we talked the first time, we talked about how I was involved in starting the Future Life Institute. And we worked very hard on 2014, 2015 was the mainstream AI safety. The idea that there even could be risks and that you could do things about them. Before then, a lot of people thought it was just really kooky to even talk about it. And a lot of AI researchers felt worried that this was too flaky and could be bad for funding. And that the people who talked about it were just not, didn't understand AI. I'm very, very happy with how that's gone. And that now, you know, it's completely mainstream.

• 36:10Max Tegmark?

  You go in any AI conference and people talk about AI safety. And it's a nerdy, technical field full of equations and blah, blah.

• 36:19Max Tegmark?

  As it should be. But there is this other thing, which has been quite taboo up until now, calling for slowdown. So what we've constantly been saying, including myself, I've been biting my tongue a lot, you know, is that, you know, we, we don't need to slow down AI development. We just need to win this race, the wisdom race between the growing power of the AI and the growing wisdom with which we manage it. And rather than trying to slow down AI, let's just try to accelerate the wisdom. Do all this technical work to figure out how you can actually ensure that your powerful AI

• 36:59Max Tegmark?

  is going to do what you want it to do and have society adapt also with incentives and regulations so that these things get put to good use. Sadly, that didn't pan out. The progress on technical AI capabilities has gone a lot faster than, than many people thought. Back when we started this in 2014, turned out to be easier to build really advanced AI than we thought. And on the other side, it's gone much slower than we hoped with getting.

• 37:40Max Tegmark?

  Policymakers and others to actually put the incentives in place to make steer this in good directions. We can, maybe we should unpack it and talk a little bit about each. So why did it go faster than we, than a lot of people thought them?

• 37:56Max Tegmark?

  In hindsight, it's exactly like building flying machines. People spent a lot of time wondering about how, how do birds fly, you know? And that turned out to be really hard. Have you seen the TED Talk with a flying bird?

• 38:10Lex Fridman?

  Like a flying robotic bird?

• 38:11Max Tegmark?

  Yeah, flies around the audience. But it took a hundred years longer to figure out how to do that than for the Wright Brothers to build the first airplane, because it turned out there was a much easier way to fly. And evolution picked the more complicated one because it had its hands tied. It could only build a flying machine that could assemble itself, which the Wright Brothers didn't care about. They can only build a machine that used only the most common atoms in the periodic table. Wright Brothers didn't care about that. They could use steel, iron atoms, and it had to be able to repair itself, and it also had to be incredibly fuel efficient. You know, a lot of birds use less than half the fuel of a remote controlled plane flying the same distance.

• 38:55Max Tegmark?

  For humans, throw a little more, put a little more fuel in a roof, there you go, a hundred years earlier. That's exactly what's happening now with these large language models. The brain is incredibly complicated. Many people made the mistake of thinking we had to figure out of the brain, does human level AI first before we could build in a machine? That was completely wrong. You can take an incredibly simple computational system called a transformer network and just train it to do something incredibly dumb. Just read a gigantic amount of text and try to predict the next word. And it turns out, if you just throw a ton of compute at that and a ton of data,

• 39:36Max Tegmark?

  it gets to be frighteningly good, like GPT-4, which I've been playing with so much since it came out, right? And there's still some debate about whether that can get you all the way to full human level or not. But yeah, we can come back to the details of that and how you might get the human level AI, even if.

• 39:58Max Tegmark?

  Large language models don't.

• 40:00Lex Fridman?

  Can you briefly, if it's just a small tangent, comment on your feelings about GPT-4? It's just that you're impressed by this rate of progress. But where is it? Can GPT-4 reason? What are like the intuitions? What are human interpretable words you can assign to the capabilities of GPT-4? That makes you so damn impressed with it?

• 40:23Max Tegmark?

  I'm both very excited about it and terrified. It's interesting mixture of promotions.

• 40:30Lex Fridman?

  All the best things in life include those two somehow.

• 40:33Max Tegmark?

  Yeah, I can absolutely reason. Anyone who hasn't played with it, I highly recommend doing that before dissing it. It can do quite remarkable reasoning. I've had to do a lot of things, which I realized I couldn't do that myself that well even. And it obviously does it dramatically faster than we do too. When you watch it type and it's doing that while servicing a massive number of other humans at the same time. At the same time, it cannot reason as well as a human can on some tasks. Just because it's obviously a limitation from its architecture. We have in our heads, what in Geek Speak is called a recurrent neural network. There are loops.

• 41:17Max Tegmark?

  Information can go from this neuron to this neuron to this neuron, and then back to this one. You can like ruminate on something for a while. You can self-reflect a lot.

• 41:26Max Tegmark?

  These large language models that are, they cannot, like GPT-4. It's a so-called transformer where it's just like a one-way street of information, basically. In Geek Speak, it's called a feed-forward neural network. And it's only so deep. So it can only do logic that's that many steps and that deep. And it's not. And you can sure you can create problems which will fail to solve, you know, for that reason.

• 41:56Max Tegmark?

  But the fact that it can do so amazing things with this incredibly simple architecture already is quite stunning. And what we see in my lab at MIT, when we look inside large language models to try to figure out how they're doing it, that's the key core focus of our research. It's called mechanistic interpretability in geek speak. You know, you have this machine that does something smart. You try to reverse engineer, see how does it do it? I think of it also as artificial neuroscience. That's exactly what neuroscientists do with actual brains. But here you have the advantage that you can, you don't have to worry about measurement errors. You can see what every neuron is doing all the time. And a recurrent thing we see again and again,

• 42:41Max Tegmark?

  there's been a number of beautiful papers quite recently by by a lot of researchers, some of them here, even in this area, is where when they figure out how something is done, you say, oh, man, that's such a dumb way of doing it. And you immediately see how it can be improved. Like, for example, there was a beautiful paper recently where they figured out how a large language model stores certain facts, like Eiffel Tower is in Paris. And they figured out exactly how it's stored. And the way the proof that they understood it was they could edit it. They changed some of the synapses in it. And then they asked it, where is the Eiffel Tower? And it said, it's in Rome. And then they asked you, how do you get there? Oh, how do you get there from Germany?

• 43:24Max Tegmark?

  Oh, you take this train to Roma Termini train station, and this and that. And what might you see if you're in front of it? Oh, you might see the Colosseum. So they had edited. So they literally moved it to Rome. But it, the way it's storing this information, it's incredibly dumb for any fellow nerds listening to this. There was a big matrix. And roughly speaking, there are certain Rowan column vectors which encode these things. And they correspond very hand waverly to the principal components. And it would be much more efficient for sparse matrix just store in the database, you know, and. But and everything so far, we've figured out how these things do are ways where you can see

• 44:08Max Tegmark?

  they can easily be improved. And the fact that this particular architecture has some roadblocks built into it is in no way going to prevent crafty researchers from quickly finding workarounds and making other kinds of architectures sort of go all the way. So, so it's, in, in short, it's turned out to be a lot easier to build human, close to human intelligence than we thought. And that means our runway as a species to get our shit together has shortened.

• 44:44Lex Fridman?

  And it seems like the scary thing about the effectiveness of large language models, so Sam Altman, I recently had a conversation with, and he really showed that the leap from GPT three to GPT four has to do with just a bunch of hacks, a bunch of little explorations with a smart researchers doing a few little fixes here and there. It's not some fundamental leap and transformation in the architecture.

• 45:13Max Tegmark?

  And more data and more compute.

• 45:16Lex Fridman?

  And more data and compute. But he said the big leaps has to do with not the data and the compute, but just learning this new discipline, just like you said. So researchers are going to look at these architectures and there might be big leaps where you realize, wait, why are we doing this in this dumb way? And all of a sudden, this model is 10x smarter. And that, that can happen on any one day, on any one Tuesday or Wednesday afternoon. And then all of a sudden, you have a system that's 10x smarter. It seems like it's such a new discipline. It's such a new, like, we understand so little about why this thing works so damn well, that the linear improvement of compute or exponential, but the steady improvement of compute, steady improvement of the data, may not be the thing that even leads to the next leap. It could

• 46:00Lex Fridman?

  be a surprise little hack that improves everything.

• 46:02Max Tegmark?

  Or a lot of little leaps here and there because. Because so much of this is out in the open also. So many smart people are looking at this and trying to figure out little leaps here and there. And it becomes this sort of collective race where a lot of people feel if I don't take the leap, someone else will. And it is actually very crucial for the other part of it. Why do we want to slow this down? So again, what this open letter is calling for is just pausing all training of systems that are more powerful than GPT-4 for six months. Give a chance for the labs to coordinate a bit on safety and for society to adapt,

• 46:46Max Tegmark?

  give the right incentives to the labs. Because you know, you've interviewed a lot of these people who lead these labs and you know just as well as I do that they're good people. They're idealistic people. They're doing this first and foremost because they believe that AI has a huge potential to help humanity. And but at the same time, they are trapped in this horrible race to the bottom. Have you read Meditations on Moloch by Scott Alexander?

• 47:19Lex Fridman?

  Yes.

• 47:19Max Tegmark?

  Yeah, it's a beautiful essay on this poem by Ginsberg where he interprets it as being about this monster. It's this game theory monster that pits people into against each other in this race to the bottom where everybody ultimately loses. And the evil thing about this monster is even though everybody sees it and understands, they still can't get out of the race, right? Most, a good fraction of all the bad things that we humans do are caused by Moloch. And I like Scott Alexander's naming of the monster. So we can, we humans can think of it as a thing. If you look at why do we have overfishing? Why do we have more generally the tragedy of the commons?

• 48:04Max Tegmark?

  Why is it that, so Liv Borey, I don't know if you've had her on your podcast?

• 48:09Lex Fridman?

  Yeah, she's become a friend, yeah.

• 48:11Max Tegmark?

  Great, she made this awesome point recently that beauty filters, that a lot of female

• 48:19Max Tegmark?

  Influencers feel pressured to use, or exactly Moloch in action again. First, nobody was using them. And people saw them just the way they were. And then some of them started using it. And becoming ever more plastic fantastic. And then the other ones that weren't using it started to realize that. If they want to just keep their. Their market share, they have to start using it too. And that and then you're in a situation where they're all using it. And and none of them has any more market share or less than before, so. Nobody gained anything, everybody lost.

• 48:55Max Tegmark?

  And they have to keep becoming ever more plastic fantastic also, right? And. But nobody can go back to the old way because it's just. Too costly, right? Moloch is everywhere. And. Moloch is not a new arrival on on the scene either. We humans have developed a lot of collaboration mechanisms to help us fight back. Against Moloch through various kinds of constructive collaboration. The Soviet Union and the United States, they had signed a number of our of arms control treaties. Against Moloch, who is trying to stoke them into unnecessarily risky nuclear arms races, et cetera, et cetera. And this is exactly what's happening on the AI front. This time.

• 49:41Max Tegmark?

  It's a little bit geopolitics, but it's mostly money. Where there's just so much commercial pressure, you know, if you take any of these. Leaders of the top tech companies.

• 49:52Max Tegmark?

  If they just say, you know, this is too risky, I want to pause. For six months, they're going to get a lot of pressure from shareholders and others. We're like, well, you know, if you pause, but those guys don't pause.

• 50:07Max Tegmark?

  We're. We don't want to get our lunch eaten. Yeah. And shareholders even have the power to replace the the executives in the worst case, right? So.

• 50:17Max Tegmark?

  We did this open letter because we want to help these idealistic tech executives to do. What their heart tells them by providing enough public pressure on the whole sector. Just pause so that they can all pause. In a coordinated fashion. And I think without the public pressure, none of them can do it alone.

• 50:39Max Tegmark?

  Push back against their shareholders, no matter how good hearted they are. Because Moloch is a really powerful foe.

• 50:47Lex Fridman?

  So the idea is to, for the major developers of AI systems like this, so we're talking about Microsoft, Google, Meta, and anyone else?

• 51:00Max Tegmark?

  OpenAI is very close with Microsoft now, of course, and there are plenty of smaller players, for example, Anthropic is very impressive, there's conjecture, there's many, many, many players, I don't want to make a long list to leave anyone out. And for that reason, it's so important that some coordination happens, that there's external pressure on all of them, saying you all need to pause, because then the people, the researchers in these organizations, the leaders who want to slow down a bit, they can say to their shareholders, you know, everybody's slowing down because of this pressure, and it's the right thing to do.

• 51:41Lex Fridman?

  Have you seen in history their examples where it's possible to pause the model?

• 51:47Max Tegmark?

  Yes, absolutely. And even like, human cloning, for example, you could make so much money on human cloning.

• 51:56Max Tegmark?

  Why aren't we doing it?

• 51:59Max Tegmark?

  Because biologists thought hard about this and felt like this is way too risky. They got together in the seventies in the Silmar and decided even to stop a lot more stuff also, just editing the human germ line, gene editing that goes in to our offspring, and decided, let's not do this because it's too unpredictable what it's going to lead to, we could lose control over what happens to our species, so they paused. There was a ton of money to be made there, so it's very doable, but you just need, you need a public awareness of what the risks are, and the broader community coming in and saying, hey, let's slow down.

• 52:45Max Tegmark?

  And, you know, another, another common pushback I get today is, we, we can't stop in the West because China.

• 52:53Max Tegmark?

  And in China, undoubtedly, they also get told, we can't slow down because the West, because both sides think they're the good guy. Yeah. But look at human cloning, you know.

• 53:05Max Tegmark?

  Did China forge ahead with human cloning? There's been exactly one human cloning that's actually been done that I know of. It was done by a Chinese guy. Do you know where he is now? Where? In jail.

• 53:16Max Tegmark?

  And you know who put him there? Who? Chinese government. Not because Westerners said, China, look, this is. No, the Chinese government put him there because they also felt they like control. The Chinese government, if anything, maybe they are even more concerned about having control than than Western governments have no incentive of just losing control over where everything is going. And you can also see the Ernie bot that was released, I believe, by Do recently. They got a lot of pushback from the government and had to rein it in, you know, in a big way.

• 53:51Max Tegmark?

  I think once this basic message comes out that this isn't an arms race, it's a suicide race. Where everybody loses if anybody's AI goes out of control. It really changes the whole dynamic. It it's not. It's. And I'll say this again, because this is this very basic point, I think a lot of people get wrong. Because a lot of people dismiss the whole idea that AI can really get very superhuman, because they think there's something really magical about intelligence, such that it can only exist in human minds, you know, because they believe that, I think it's gonna kind of get to just more or less GPT four plus plus, and then that's it.

• 54:34Max Tegmark?

  They don't see it as a super as a suicide race. They think whoever gets that first, they're going to control the world, they're going to win. That's not how it's going to be. And we can talk again about the the scientific arguments from why it's not going to stop there. But the way it's going to be is if if anybody completely loses control and, you know, you don't care if if some, some, if someone manages to take over the world who really doesn't share your goals, they, you probably don't really even care very much about what nationality they have. You're not going to like it, much worse than today.

• 55:09Max Tegmark?

  If it's, if you live in Orwellian dystopia, who, what do you care who's created it, right? And if someone, if it goes farther and and we just lose control even to the machines.

• 55:23Max Tegmark?

  So that it's not us versus them, it's us versus it.

• 55:28Max Tegmark?

  Why do you care who created this this underlying entity which has goals different from humans ultimately and we get marginalized, we get made obsolete, we get replaced.

• 55:41Max Tegmark?

  That's why what I mean when I say it's a suicide race. It's kind of like we're rushing towards this cliff. But the closer the cliff we get, the more scenic the views are and the more money there is there and so we keep going. But we have to also stop at some point, right? Quit while we're ahead and it's

• 56:06Max Tegmark?

  it's a suicide race, which cannot be won. But the way to really benefit from it is to continue developing awesome AI a little bit slower. So we make it safe, make sure it does the things that humans want and create a condition where everybody wins.

• 56:24Max Tegmark?

  Technology has shown us that, you know, geopolitics and politics in general is not a zero sum game at all.

• 56:32Lex Fridman?

  So there is some rate of development that will lead

• 56:37Lex Fridman?

  us as a human species to lose control of this thing. And the hope you have is that there's some lower level of development, which will not, which will not allow us to lose control. This is an interesting thought you have about losing control. So if you have somebody, if you're somebody like Sunde Perchay or Sam Altman at the head of a company like this, you're saying if they develop an AGI, they too will lose control of it. So no one person can maintain control, no group of individuals can maintain.

• 57:06Max Tegmark?

  If it's if it's created very, very soon and is a big black box that we don't understand, like the large language models, yeah, then I'm very confident they're going to lose control. But this isn't just me saying, you know, Sam Altman and Demis Hassabis have both said themselves acknowledge that, you know, there's really great risks with this and they they want to slow down once they feel it gets scary. It's but it's clear that they're stuck in this again, Moloch is forcing them to go a little faster than they're comfortable with because of pressure from just commercial pressures, right?

• 57:42Max Tegmark?

  To get a bit optimistic here, of course, this is a problem that can be ultimately solved. It just to win this wisdom race. It's clear that what we hoped that was going to happen hasn't happened. The capability progress has gone faster than a lot of people thought than and the progress in the public sphere of policymaking and so on has gone slower than we thought. Even the technical AI safety has gone slower. A lot of the technical safety research was kind of banking on that large language models and other poorly understood systems couldn't get us all the way. That you had to build more of a kind of intelligence that you could understand. Maybe it could prove itself safe, you know, things like this.

• 58:23Max Tegmark?

  And I'm quite confident that this can be done, so we can reap all the benefits, but we cannot do it as quickly as this out of control express train we are on now is going to get the AGI. That's why we need a little more time, I feel.

• 58:41Lex Fridman?

  Is there something to be said, what like Sam Altman talked about, which is while we're in the pre AGI stage, to release often and as transparently as possible to learn a lot. So as opposed to being extremely cautious, release a lot. Don't, don't invest in a closed development where you focus on AI safety while is somewhat dumb, quote unquote, release as often as possible. And as you start to see signs of human level intelligence or superhuman level intelligence, then you put a halt on it.

• 59:20Max Tegmark?

  Well, what a lot of safety researchers have been saying for many years is that the most dangerous things you can do with an AI is first of all, teach it to write code.

• 59:30Lex Fridman?

  Yeah.

• 59:31Max Tegmark?

  Because that's the first step towards recursive self improvement, which can take it from AGI to much higher levels. Okay, oops, we've done that. And another thing, high risk is connected to the internet. Let it go to websites, download stuff on its own and talk to people. Oops, we've done that already. You know, Eliezer Yudkowsky, you said you interviewed him recently, right?

• 59:53Lex Fridman?

  Yes.

• 59:54Max Tegmark?

  He had this tweet recently, which gave me one of the best laughs in a while, where he was like, hey, people used to make fun of me and say, you're so stupid, Eliezer, because you're saying, you're saying. You have to worry of obviously developers, once they get to like really strong AI. First thing you're gonna do is like never connect it to the internet, keep it in a box. Where you know, you can really study it.

• 1:00:19Max Tegmark?

  So he had written it in the like in the meme form. So it's like then.

• 1:00:22Lex Fridman?

  Yeah.

• 1:00:23Max Tegmark?

  And then that. Now.

• 1:00:27Max Tegmark?

  LOL, let's make a chatbot.

• 1:00:31Lex Fridman?

  Yeah.

• 1:00:32Max Tegmark?

  And the third thing is Stuart Russell.

• 1:00:34Lex Fridman?

  Yeah.

• 1:00:34Max Tegmark?

  You know. Amazing AI researcher. He has argued for a while that we should never teach AI anything about humans. Above all, we should never let it learn about human psychology and how you manipulate humans. That's the most dangerous kind of knowledge you can give it. Yeah, you can teach it all it needs to know about how to cure cancer and stuff like that, but don't let it. Read Daniel Kahneman's book about cognitive biases and all that. And then. Oops LOL, you know, let's invent social media. I'll recommend their algorithms which do exactly that. They they get so good at knowing us and pressing our buttons.

• 1:01:18Max Tegmark?

  That we've we're starting to create a world now where we just have ever more hatred.

• 1:01:25Max Tegmark?

  Because they figured out that these algorithms, not for out of evil, but just to make money on advertising that the best way to get more engagement. The euphemism. Get people glued to their little rectangles, right, is just to make them pissed off.

• 1:01:38Lex Fridman?

  That's really interesting that a large AI system that's doing the recommender system kind of task on social media. Is basically just studying human beings because it's a bunch of us rats giving it signal. Nonstop signal. It'll show a thing and then we give signal on whether we spread that thing, we like that thing. That thing increases our engagement, gets us to return to the platform. And it has that on the scale of hundreds of millions of people constantly. So it's just learning and learning and learning. And presumably if the parameter, the number of parameters in your neural network that's doing the learning. And more end to end the learning is. The more it's able to just basically encode how to manipulate human behavior. How to control humans at scale.

• 1:02:23Max Tegmark?

  Exactly, and that is not something I think is in humanity's interest.

• 1:02:27Lex Fridman?

  Yes.

• 1:02:28Max Tegmark?

  Right now, it's mainly letting some humans manipulate other humans for profit.

• 1:02:34Max Tegmark?

  And power, which already caused a lot of damage. And eventually, that's a sort of skill that can make AIs persuade humans to let them escape whatever safety precautions we have. But, you know, there was a really nice article in the New York Times recently by Yuval Noah Harari and two co-authors, including Tristan Harris from The Social Dilemma. They have this phrase in there I love. It said, humanity's first contact with advanced AI was social media, and we lost that one. We now live in a country where there's much more hate, in a world where there's much more hate, in fact.

• 1:03:19Max Tegmark?

  And in our democracy that we're having this conversation, and people can't even agree on who won the last election, you know? And we humans often point fingers at other humans and say it's their fault. But it's really Moloch and these AI algorithms. We got the algorithms and then Moloch pitted the social media companies against each other. So nobody could have a less creepy algorithm because then they would lose out on revenue to the other company.

• 1:03:45Lex Fridman?

  Is there any way to win that battle back? Just if we just linger on this one battle that we've lost in terms of social media? Is it possible to redesign social media, this very medium in which we use as a civilization to communicate with each other, to have these kinds of conversations, to have discourse to try to figure out how to solve the biggest problems in the world, whether that's nuclear war or the development of AGI? Is it possible to do social media correctly?

• 1:04:14Max Tegmark?

  I think it's not only possible, but it's necessary. Who are we kidding that we're going to be able to solve all these other challenges if we can't even have a conversation with each other that's constructive? The whole idea, the key idea of democracy is that you get a bunch of people together and they have a real conversation, the ones you try to foster on this podcast, where you respectfully listen to people you disagree with. And you realize, actually, you know, there are some things, actually, some common ground we have and that's, we both agree, let's not have a nuclear war, let's not do that, et cetera, et cetera.

• 1:04:47Max Tegmark?

  We're kidding ourselves, thinking we can face off the second contact with ever more powerful AI that's happening now with these large language models if we can't even have a functional conversation in the public space. That's why I started the Improve the News project, improvethenews.org. But I'm an optimist fundamentally in

• 1:05:14Max Tegmark?

  and that there is a lot of intrinsic goodness in people. And that what makes the difference between someone doing good things for humanity and bad things is not some sort of fairy tale thing that this person was born with an evil gene and this one is not born with a good gene. No, I think it's whether we put, whether people find themselves in situations that bring out the best in them or that bring out the worst in them. And I feel we're building an internet and a society that brings out the worst.

• 1:05:52Lex Fridman?

  But it doesn't have to be that way.

• 1:05:54Max Tegmark?

  No, it does not.

• 1:05:55Lex Fridman?

  It's possible to create incentives and also create incentives that make money. They both make money and bring out the best in people.

• 1:06:03Max Tegmark?

  I mean, in the long term, it's not a good investment for anyone, you know, to have a nuclear war, for example. And, you know, is it a good investment for humanity if we just ultimately replace all humans by machines? And then we're so obsolete that eventually there are no humans left? Well, depends against how you do the math. But I would say by any reasonable economic standard, if you look at the future income of humans and there aren't any, you know, that's not a good investment.

• 1:06:31Max Tegmark?

  Moreover, like, why, why can't we have a little bit of pride in our species? Dammit, you know, why should we just build another species that gets rid of us? If we were Neanderthals.

• 1:06:43Max Tegmark?

  Would we really consider it a smart move if we had really advanced biotech to build Homo sapiens? You know, you might say, hey, Max, you know, yeah, let's build, build these Homo sapiens. They're going to be smarter than us. Maybe they can help us defend us better against predators and help fix up our caves, make them nicer. We'll control them undoubtedly, you know. So then they build, build a couple, a little baby girl, little baby boy, they're

• 1:07:14Max Tegmark?

  and then you have some wise old Neanderthal elder is like, hmm, I'm scared that we're opening a Pandora's box here and that we're going to get outsmarted by these

• 1:07:28Max Tegmark?

  super Neanderthal intelligences and there won't be any Neanderthals left. But then you have a bunch of others in the cave, right? Are you such a Luddite scaremonger? Of course, they're going to want to keep us around because we are their creators. And, and why, you know, the smarter, I think the smarter they get, the nicer they're going to get. They're going to leave us, they're going to, they're going to want us around and it's going to be fine. And, and besides, look at these babies, they're so cute. Clearly, they're totally harmless. That's exactly, those babies are exactly GPT-4. It's not, I want to be clear, it's not GPT-4 that's terrifying. It's the GPT-4 is a baby technology. You know, and Microsoft even had a paper recently out

• 1:08:11Max Tegmark?

  with a title, something like Sparkles of AGI. Well, they were basically saying this is baby AI, like these little Neanderthal babies.

• 1:08:21Max Tegmark?

  And it's going to grow up. There's going to be other systems from, from the same company, from other companies. They'll be way more powerful and, but they're going to take all the things, ideas from these babies. And before we know it, we're going to be like those last Neanderthals who are pretty disappointed. When they realized that they were getting replaced.

• 1:08:44Lex Fridman?

  Well, this interesting point you make, which is the programming, it's entirely possible that GPT-4 is already the kind of system that can change everything by writing programs.

• 1:08:57Max Tegmark?

  Like three, it's, yeah, it's because it's life 2.0. The, the systems I'm afraid of are going to look nothing like a large language model and they're not.

• 1:09:07Max Tegmark?

  But once it gets, once it or other people figure out a way of using this tech to make much better tech, right, it's just constantly replacing its software. And from everything we've seen about how, how these work under the hood, they're like the minimum viable intelligence. They do everything in the dumbest way that still works, sort of.

• 1:09:27Lex Fridman?

  Yeah.

• 1:09:28Max Tegmark?

  And so they are like 3.0, except when they replace their software, it's a lot faster than when you, when you decide to learn Swedish.

• 1:09:39Max Tegmark?

  And moreover, they think a lot faster than us too. So when, you know, we don't think on how one logical step every nanosecond or a few or so, the way they do. And we can't also just suddenly scale up our hardware massively in the cloud, because we're so limited, right? So they are, they are also life, can soon be, become a little bit more like life 3.0 in that if they need more hardware, hey, just rent it in the cloud, you know? How do you pay for it? Well, with all the services you provide.

• 1:10:23Lex Fridman?

  And what we haven't seen yet, which could change a lot, is an entire software system. So right now, programming is done sort of in bits and pieces as, as an assistant tool to humans. But I do a lot of programming and with the kind of stuff that GPT-4 is able to do, I mean, is replacing a lot what I'm able to do. But I, you still need a human in the loop to kind of manage the design of things, manage like what are the prompts that generate the kind of stuff, to, to do some basic adjustment of the code, to do some debugging. But if it's possible to add on top of GPT-4, kind of. Uh, feedback loop of, of, uh, of, uh,

• 1:11:08Lex Fridman?

  self debugging, improving the code. And then you launch that system onto the wild on the internet because everything is connected and have it do things, have it interact with humans and then get that feedback. Now you have this giant ecosystem.

• 1:11:22Max Tegmark?

  Yeah.

• 1:11:22Lex Fridman?

  Of humans. That's one of the things that, um.

• 1:11:24Max Tegmark?

  Yeah.

• 1:11:25Lex Fridman?

  Elon Musk recently sort of tweeted as a case why everyone needs to pay $7 or whatever for Twitter.

• 1:11:31Max Tegmark?

  To make sure they're real.

• 1:11:32Lex Fridman?

  That make sure they're real. We're now going to be living in a world where the, the bots are getting smarter and smarter and smarter to a degree where you can't, uh, you can't tell the difference between a human and a bot.

• 1:11:45Max Tegmark?

  That's right.

• 1:11:45Lex Fridman?

  And now you can have, uh, bots outnumber humans by.

• 1:11:49Max Tegmark?

  Yeah.

• 1:11:49Lex Fridman?

  One million to one.

• 1:11:51Max Tegmark?

  Yeah.

• 1:11:51Lex Fridman?

  Which is why he's making a case why you have to pay.

• 1:11:54Max Tegmark?

  Yeah.

• 1:11:54Lex Fridman?

  To prove you're human, which is one of the only mechanisms to prove, which is depressing.

• 1:11:59Max Tegmark?

  And I, yeah, I feel we have to remember as individuals, we should from time to time ask ourselves, why are we doing what we're doing? Right? And as a species, we need to do that too. So if we're building, as, as you say, machines that are outnumbering us and more and more outsmarting us and, and replacing us on the job market, not just for the dangerous and, and boring tasks, but also for writing poems and doing art and things that a lot of people find really meaningful, gotta ask ourselves, why? Why are we doing this? We are, the answer is Moloch is tricking us into doing it. And it's such a clever trick that even though we see the trick, we still have no choice

• 1:12:44Max Tegmark?

  but to fall for it, right? Come also, the thing you said about you using, uh, Copilot AI tools to program faster. How many times, what factor faster would you say you, you code now? Does it go twice as fast or?

• 1:13:00Lex Fridman?

  I don't really, uh, because it's such a new tool.

• 1:13:04Max Tegmark?

  Yeah.

• 1:13:04Lex Fridman?

  It's, I don't know if speed is significantly improved, uh, but it feels like I'm a year away from being, uh, five to 10 times faster.

• 1:13:15Max Tegmark?

  So if, if that's typical for programmers, then, uh, you're already seeing another kind of self recursive self improvement, right? Because previously, uh, one like a major generation of improvement of the code would happen on the human R and D time scale. And now if that's five times shorter. Then it's going to take five times less time than otherwise would to develop the next level of these tools and so on. So this, these, these are the, this is exactly the sort of beginning of an, of an intelligence explosion. There can be humans in the loop a lot in the early stages and then eventually humans are needed less and less and the machines can more kind of go along. But you, what you, you said there is just an exact example

• 1:13:57Max Tegmark?

  of, of these sort of things. Another thing which, which um, I was kind of lying on my psychiatrist, imagining I'm on a psychiatrist's couch here saying, what are my fears that people would do with AI systems? Another, so I mentioned three that I had fears about many years ago that they would do, namely teach it the code, connect it to the internet and teach it to manipulate humans. A fourth one is building an API where code can control this super powerful thing, right? That's very unfortunate because one thing that systems like GPT-4 have going for them is that they are an oracle

• 1:14:39Max Tegmark?

  in the sense that they just answer questions. There is no robot connected to GPT-4. GPT-4 can't go and do stock trading based on its thinking. It is not an agent. An intelligent agent is something that takes in information from the world, processes it to figure out what action to take based on its goals that it has and then does something back on the world. But once you have an API, for, for example, GPT-4, nothing stops Joe Schmo and all, and a lot of other people from building real agents, which just keep making calls somewhere in some inner loop somewhere to these powerful oracle systems, which makes

• 1:15:22Max Tegmark?

  them themselves much more powerful. That's another kind of unfortunate development, which I think we would have been better off delaying. I don't want to pick on any particular companies. I think they're all under a lot of pressure to make money. Yeah. And again, we, the reason we're, we're calling for this pause is to give them all cover to do what they know is the right thing, slow down a little bit at this point. But everything we've talked about, I hope we'll, can, we'll make it clear to people watching this, you know, why.

• 1:15:58Max Tegmark?

  These sort of human level tools can cause gradual acceleration. You keep using yesterday's technology to build tomorrow's technology.

• 1:16:05Lex Fridman?

  Yeah.

• 1:16:05Max Tegmark?

  And when you do that over and over again, you naturally get an explosion. You know, that's the definition of an explosion in science, right? Like, if you have two people, they fall in love. Now you have four people and then they can make more babies. And now you have eight people and then, then you have sixteen, thirty-two, sixty-four, et cetera. That's, we call that a population explosion, where it's just that each, if it's instead free neutrons in a nuclear reaction, that if each one can make more than one, then you get an exponential growth in that. We call it a nuclear explosion. All explosions are like that. And an intelligence explosion, it's just exactly the same principle.

• 1:16:47Max Tegmark?

  That some quantity, some amount of intelligence can make more intelligence than that and then repeat. You always get exponentials.

• 1:16:55Lex Fridman?

  What's your intuition why it does? You mentioned there's some technical reasons why it doesn't stop at a certain point. What's your intuition? And do you have any intuition why it might stop?

• 1:17:06Max Tegmark?

  It's obviously going to stop when it bumps up against the laws of physics. There are some things you just can't do, no matter how smart you are, right?

• 1:17:12Lex Fridman?

  Allegedly.

• 1:17:15Lex Fridman?

  And Because we don't know the full

• 1:17:17Max Tegmark?

  Yeah.

• 1:17:17Lex Fridman?

  Laws of physics.

• 1:17:18Max Tegmark?

  Yeah.

• 1:17:18Lex Fridman?

  Yet, right?

• 1:17:19Max Tegmark?

  Seth Lloyd wrote a really cool paper on the physical limits on computation. For example, if you make it, put too much energy into it and the finite space, it'll turn into a black hole. You can't move information around faster than the speed of light, stuff like that. But it's hard to store way more than, than a modest number of bits per atom, et cetera. But, you know, those limits are just astronomically above, like thirty orders of magnitude above where we are now. And so.

• 1:17:50Max Tegmark?

  Bigger, different, bigger jump in intelligence than if you go from a, from an ant to a human. I think, of course, what we want to do is have, have a controlled thing. Your nuclear reactor, you put moderators in to make sure exactly it doesn't blow up out of control, right? When we do experiments with biology in cells and so on, you know, we also try to make sure it doesn't get out of control. We can do this with AI too. The thing is, we haven't succeeded yet. And Moloch is exactly doing the opposite, just fueling, just egging everybody on faster, faster, faster.

• 1:18:34Max Tegmark?

  Or the other company is going to catch up with you, or the other country is going to catch up with you. We do this, we have to want this stuff. We have, and, and I don't believe in this, just asking people to look into their hearts and do the right thing. It's easy for others to say that, but like if, if you're in a situation where your company is going to get screwed, if you, by other companies that are not stopping, you know, you're putting people in a very hard situation. The, the right thing to do is change the whole incentive structure instead. And, and this is not an old, maybe I should say one more thing about this, because Moloch has been around as humanity's number one or number two enemy since the beginning of

• 1:19:19Max Tegmark?

  civilization. And we came up with some really cool countermeasures, like first of all, already over a hundred thousand years ago, evolution realized that it was very unhelpful that people kept killing each other all the time. Yeah. So it genetically gave us compassion and made it so that like if you get two drunk dudes getting into a pointless bar fight.

• 1:19:45Max Tegmark?

  They might give each other black eyes, but they have a lot of inhibition towards killing each other. That's a gen- and similarly, if you find a baby lying on the street when you go out for your morning jog tomorrow, you're gonna stop and pick it up, right? Even though it maybe make you late for your next podcast. So, evolution gave us these genes that make our own egoistic incentives more aligned with what's good for the greater group we're part of, right? And then, as we got a bit more sophisticated and developed language, we invented gossip, which is also a fantastic anti-Maloch, right?

• 1:20:25Max Tegmark?

  Because now, it really discourages liars, moochers, cheaters, because their own incentive now is not to do this, because word quickly gets around and then suddenly people aren't gonna invite them to their dinners anymore or trust them. And then, when we got still more sophisticated and bigger societies, you know, invented the legal system, where even strangers who didn't, couldn't rely on gossip and things like this, would treat each other, would have incentive. Now, those guys in the bar fights, even if they, someone is so drunk that he actually wants to kill the other guy, he also has a little thought in the back of his head that, you know, do I really wanna spend the next 10 years

• 1:21:10Max Tegmark?

  eating like really crappy food in a small room? I'm just gonna chill out, you know? So, and we similarly have tried to give these incentives to corporations by having, having regulation and all sorts of oversight, so that their incentives are aligned with the greater good. We tried really hard. And the big problem that we're failing now is not that we haven't tried before, but it's just that the tech is growing much, is developing much faster than the regulator's been able to keep up, right? So, regulators, it's kind of comical, like European Union right now is doing this AI Act, right? Which, and in the beginning, they had a little opt-out exception,

• 1:21:54Max Tegmark?

  that GPT-4 would be completely excluded from regulation. Brilliant idea. What's the logic behind that? Some lobbyists pushed successfully for this. So we were actually quite involved with the Future Life Institute. Marc Barakel, Christophe, Anthony Aguirre and others. You know, we're quite involved with talking to very educating various people involved in this process about these general purpose AI models coming and pointing out that they would become the laughing stock if they didn't put it in. So it the French started pushing for it. It got put in to the draft and it looked like all was good. And then there was a huge counter push from lobbyists. Yeah, there were more lobbyists in Brussels from tech companies than from oil companies,

• 1:22:39Max Tegmark?

  for example, and it looked like it might is we can maybe get taken out again. And now GPT four happened. And I think it's going to stay in. But this just shows, you know, Moloch can be defeated. But the the challenge we're facing is that the tech is generally much faster than what the policymakers are. And a lot of the policymakers also don't have a tech background. So it's, you know, we really need to work hard to educate them on how on what's taking place here. So so we're getting the situation where the first kind of non. So I define artificial intelligence just as non biological intelligence. All right.

• 1:23:23Max Tegmark?

  And by that definition, a company, a corporation is also an artificial intelligence because the corporation isn't it's humans, it's a system. If its CEO decides the CEO of a tobacco company decides one morning that she or he doesn't want to sell cigarettes anymore, they'll just put another CEO in there.

• 1:23:42Max Tegmark?

  It's not enough to align the incentives of individual people or in align individual computers incentives to their owners, which is what technically AI safety research is about. You also have to align incentives of corporations with the greater good. And some corporations have gotten so big and so powerful very quickly that in many cases their lobbyists instead align the regulators to what they want rather than the other way around. The classic regulatory capture.

• 1:24:13Lex Fridman?

  All right. Is the thing that the slowdown hopes to achieve is give enough time to the regulators to catch up or enough time to the companies themselves to breathe and understand how to do AI safety correctly?

• 1:24:27Max Tegmark?

  I think both, but I think that the vision, the path to success I see is first you give a breather actually to the people in these companies, their leadership who wants to do the right thing and they all have safety teams and so on on their companies, give them a chance to get together with the other companies and the outside pressure can also help catalyze that, right?

• 1:24:51Max Tegmark?

  And work out what is it that's, what are the reasonable safety requirements one should put on future systems before they get rolled out? There are a lot of people also in academia and elsewhere outside of these companies who can be brought into this and have a lot of very good ideas. And then I think it's very realistic that within six months you can get these people coming up. So here's a white paper, here's what we all think is reasonable. You know, you didn't, just because cars killed a lot of people, they didn't ban cars, but they got together a bunch of people and decided, you know, in order to be allowed to sell a car, it has to have a seatbelt in it.

• 1:25:33Max Tegmark?

  There are the analogous things that you can start requiring a future AI systems so that they are safe. And once this heavy lifting, this intellectual work has been done by experts in the field, which can be done quickly, I think it's going to be quite easy to get policymakers to see, yeah, this is a good idea. And it's, you know, for the companies to fight Moloch, they want, and I believe Sam Altman has explicitly called for this, they want the regulators to actually adopt it so that their competition is going to abide by it too, right?

• 1:26:12Max Tegmark?

  You don't want, you don't want to be enacting all these principles and you abide by them. And then there's this one little company that doesn't sign on to it. And then now they can gradually overtake you. Then the companies will get.

• 1:26:31Max Tegmark?

  Be able to sleep secure, knowing that everybody's playing by the same rules.

• 1:26:35Lex Fridman?

  So you think it's possible to develop guardrails that keep the systems from

• 1:26:43Lex Fridman?

  from basically damaging irreparably humanity while still enabling sort of the capitalist fueled competition between companies as they develop how to best make money with this AI. You think there's a balancing?

• 1:26:56Max Tegmark?

  Totally.

• 1:26:56Lex Fridman?

  That's possible.

• 1:26:57Max Tegmark?

  Absolutely. I mean, we've seen that in many other sectors where you've had the free market produce quite good things without causing particular harm. When the guardrails are there and they work, you know,

• 1:27:11Max Tegmark?

  capitalism is a very good way of optimizing for just getting the same things done more efficiently. It's just, but it was good, you know, and like in hindsight, and I've never met anyone even

• 1:27:23Max Tegmark?

  even on parties way over on the right in any country who think it was a bad, thinks it was a terrible idea to ban child labor, for example.

• 1:27:33Lex Fridman?

  Yeah, but it seems like this particular technology has gotten so good, so fast, become powerful to a degree where you could see in the near term, the ability to make a lot of money. And to put guardrails, develop guardrails quickly in that kind of context seems to be tricky. It's not similar to cars or child labor. It seems like the opportunity to make a lot of money here very quickly is right here.

• 1:28:00Max Tegmark?

  Yeah. Again, there's this cliff.

• 1:28:03Lex Fridman?

  Yep. Gets quite scenic.

• 1:28:05Max Tegmark?

  The closer the cliff you go, the more, there are more money there is, the more gold ingots there on the ground you can pick up or whatever. So you want to drive there very fast. But it's not in anyone's incentive that we go over the cliff. And it's not like everybody's in their own car. All the cars are connected together with a chain. So if anyone goes over, they'll start dragging others down, the others down too. And so ultimately it's in the selfish.

• 1:28:30Max Tegmark?

  Interests also of the people in the companies to to slow down when the when you just start seeing the contours of the cliff there in front of you, right? And the problem is that even though the people who are building the technology and the CEOs, they really get it. The shareholders and these other market forces, they are people who don't honestly understand that the cliff is there. They usually don't. You have to get quite into the weeds to really appreciate how powerful this is and how fast. And a lot of people are even still stuck again in this idea that intelligence and this carbon chauvinism, as I like to call it, that you can only have our level of intelligence in humans, that there's something magical

• 1:29:14Max Tegmark?

  about it. Whereas the people in the tech companies who build this stuff, they all realize that intelligence is information processing of a certain kind. And it really doesn't matter at all whether the information is processed by carbon atoms in neurons and brains, or by silicon atoms and some technology we build. So you brought up capitalism earlier, and there are a lot of people who love capitalism and a lot of people who really, really don't. And it struck me recently that what's happening with capitalism here

• 1:29:53Max Tegmark?

  is exactly analogous to the way in which superintelligence might wipe us out. So you know, I studied economics for my undergrad, Stockholm School of Economics, yay.

• 1:30:08Lex Fridman?

  Well, no, no, I tell me.

• 1:30:09Max Tegmark?

  So I was very interested in how you could use market forces to just get stuff done more efficiently, but give the right incentives to markets so that it wouldn't do really bad things. So Dylan Hadfield-Mannell, who's a professor and colleague of mine at MIT, wrote this really interesting paper with some collaborators recently, where they proved mathematically that if you just opt take one goal that you just optimize for on and on and on indefinitely, that you think is gonna bring you in the right direction. What basically always happens is, in the beginning, it will make things better for you.

• 1:30:48Max Tegmark?

  But if you keep going, at some point, it's gonna start making things worse for you again. And then gradually, it's gonna make it really, really terrible. So just as a simple, the way I think of the proof is, suppose you wanna go from here

• 1:31:02Max Tegmark?

  back to Austin, for example, and you're like, okay, yeah, let's just, let's go south. But you put in exactly the right, sort of the right direction. Just optimize that, south is possible. You get closer and closer to Austin. But you, there's always some little error. So you, you're not going exactly towards Austin, but you get pretty close. But eventually, you start going away again. And eventually, you're gonna be leaving the solar system. Yeah. And they, they proved, it's a beautiful mathematical proof. This happens generally. And this is very important for AI. Because for, even though Stuart Russell has written a book and given a lot of talks on why it's a bad idea to have AI just blindly optimize something.

• 1:31:46Max Tegmark?

  That's what pretty much all our systems do. Yeah. We have something called the loss function that we're just minimizing, or reward function. We're just maximizing. And capitalism is exactly like that, too. We, we wanted to get stuff done more efficiently, the people wanted. So we introduced the free market.

• 1:32:10Max Tegmark?

  Things got done much more efficiently than they did in, in, say, communism, right? And it got better. But then it just kept optimizing and kept optimizing. And you got ever bigger companies and ever more efficient information processing. And now also very much powered by IT. And eventually, a lot of people are beginning to feel, wait, we're kind of optimizing a bit too much. Like, why did we just chop down half the rainforest, you know? And why, why did suddenly these regulators get captured by lobbyists and so on? It's just the same optimization that's been running for too long. If you have an AI that actually has power over the world, and you just give it one goal,

• 1:32:54Max Tegmark?

  and just like, keep optimizing that, most likely, everybody's gonna be like, yay, this is great. In the beginning, things are getting better. But it's almost impossible to give it exactly the right direction to optimize in. And then.

• 1:33:10Max Tegmark?

  Eventually all hay breaks loose, right? Nick Bostrom and others are giving examples that sound quite silly. Like, what if you just wanna like, tell it to cure cancer or something? And that's all you tell it? Maybe it's gonna decide to take over entire continents just so it can get more supercomputer facilities in there and figure out how to cure cancer backwards. And then you're like, wait, that's not what I wanted, right? And the issue with capitalism and the issue with runaway AI have kind of merged now because the Moloch I talked about is exactly the capitalist Moloch that we have built an economy that is optimizing for only one thing: profit.

• 1:33:55Max Tegmark?

  Right? And that worked great back when things were very inefficient and then now it's getting done better. And it worked great as long as the companies were small enough that they couldn't capture the regulators. But that's not true anymore, but they keep optimizing. And now they realize that these companies can make even more profit by building ever more powerful AI, even if it's reckless.

• 1:34:21Max Tegmark?

  But optimize more, more, more, more, more. So this is Moloch again showing up. And I just want to, anyone here who has any concerns about late-stage capitalism having gone a little too far, you should worry about superintelligence because it's the same villain in both cases. It's Moloch.

• 1:34:44Lex Fridman?

  And optimizing one objective function aggressively, blindly is going to take us there.

• 1:34:51Max Tegmark?

  Yeah, we have this pause from time to time and look into our hearts and ask, why are we doing this? Is this, am I still going towards Austin? Or have I gone too far? You know, maybe we should change direction.

• 1:35:05Lex Fridman?

  And that is the idea behind a halt for six months. Why six months? It seems like a very short period. Can we just linger and explore different ideas here? Because this feels like a really important moment in human history where pausing would actually have a significant positive effect.

• 1:35:24Max Tegmark?

  We said six months because we figured the number one pushback we were gonna get in the West was like, but China.

• 1:35:35Max Tegmark?

  And everybody knows there's no way that China is going to catch up with the West on this in six months. So it's that argument goes off the table and you can forget about geopolitical competition and just focus on the real issue. That's why we put this.

• 1:35:50Lex Fridman?

  That's really interesting. But you've already made the case that even for China, if you actually want to take on that argument, China too would not be bothered by a longer halt because they don't want to lose control even more than the West doesn't.

• 1:36:06Max Tegmark?

  That's what I think.

• 1:36:08Lex Fridman?

  That's a really interesting argument. Like, I have to actually really think about that, which the kind of thing people assume is if you develop an AGI, that OpenAI, if they're the ones that do it, for example, they're going to win. But you're saying, no, they're, everybody loses.

• 1:36:25Max Tegmark?

  Yeah, it's gonna get better and better and better and then kaboom, we all lose. That's what's gonna happen.

• 1:36:31Lex Fridman?

  When lose and win are defined in a metric of basically quality of life for human civilization and for Sam Altman.

• 1:36:39Max Tegmark?

  I, to be blunt, my personal guess, you know, and people can quibble with this, is that we're just gonna, there won't be any humans. That's it. That's what I mean by lose. You know, if you, we can see in history, once you have some species or some group of people who aren't needed anymore, it doesn't usually work out so well for them, right?

• 1:37:00Lex Fridman?

  Yeah.

• 1:37:01Max Tegmark?

  There were a lot of horses for, that were used for traffic in Boston and then the car got invented and most of them got, you know, well, we don't need to go there. And if you look at humans, you know, right now, we, why did the labor movement succeed? And after the Industrial Revolution? Because it was needed. Even though we had a lot of Molochs and there was child labor and so on, you know, the company still needed to have workers. And that's why strikes had power and so on. If we get to the point where most humans aren't needed anymore, I think it's like, it's quite naive to think that they're

• 1:37:46Max Tegmark?

  gonna still be treated well. You know, we say that, yeah, yeah, everybody's equal and the government will always, we always protect them. But if you look in practice, groups that are very disenfranchised and don't have any actual power.

• 1:38:01Max Tegmark?

  Usually get screwed. And now, in the beginning, so industrial revolution, we automated away muscle work. But that got, went, worked out pretty well eventually because we educated ourselves and started to do working with our brains instead and got usually more interesting, better paid jobs. But now we're beginning to replace brain work. So we replaced a lot of boring stuff like we got the pocket calculator. So you don't have people adding, multiplying numbers anymore at work. Fine, there were better jobs they could get. But now G P T four, you know, and the stable diffusion and techniques like this, they're really beginning to blow away some real, some jobs that people really loved having.

• 1:38:46Max Tegmark?

  There was a heartbreaking article just posted just yesterday on social media I saw about this guy who was doing three D modeling for gaming and he, and all of a sudden now they got this new software. He just gives, says prompts and he feels this whole job that he loves lost its meaning, you know? And I asked G P T four to rewrite Twinkle Twinkle Little Star in the style of Shakespeare. I couldn't have done such a good job. It was really impressive. You've seen a lot of the arts coming out here, right? So I'm all for automating away the dangerous jobs and the boring jobs. But I think you hear a lot, some arguments which are too glib.

• 1:39:31Max Tegmark?

  Sometimes people say, well, that's all that's going to happen. We're getting rid of the boring, boring, tedious, dangerous jobs. It's just not true. There are a lot of really interesting jobs that are being taken away now. Journalism is getting, gonna get crushed. Coding is gonna get crushed. I predict the job market for programmers, salaries are gonna start dropping. You know, if you said you can code five times faster, you know, then you need five times fewer programmers. Maybe there'll be more output also, but you'll still end up using fewer program, needing fewer programmers than today. And I love coding, you know, I, I think it's super cool. So we, we need to stop and ask ourselves, why again are we doing this as humans, right?

• 1:40:17Max Tegmark?

  I feel that AI should be built by humanity for humanity. And let's not forget that it shouldn't be by Moloch for Moloch. Or what it really is now is kind of by humanity for Moloch. Which doesn't make any sense. It's for us that we're doing it. And it would make a lot more sense if we build, develop, figure out gradually safely, how to make all this tech. And then we think about what are the kind of jobs that people really don't want to have? You know, automate them all away. And then we ask, what are the jobs that people really find meaning in? Like maybe taking care of children in the daycare center, maybe doing art, et cetera, et cetera.

• 1:41:01Max Tegmark?

  And even if it were possible to automate that away, we don't need to do that, right? We built these machines.

• 1:41:08Lex Fridman?

  Well, it's possible that we redefine or rediscover what are the jobs that give us meaning. So for me, the thing, it is really sad. Like I, half the time I'm excited, half the time I'm crying as I'm, as I'm generating code because I kind of love programming. It's a, it's an act of creation. You, you have an idea, you design it, and then you bring it to life and it does something, especially if there's some intelligence to it, it does something. It doesn't even have to have intelligence. Printing hello world on the screen. You, you, you made a little machine and it comes to life.

• 1:41:48Max Tegmark?

  Yeah.

• 1:41:49Lex Fridman?

  And there's a bunch of tricks you learn along the way because you've been doing it for, for many, many years. And then to see AI, be able to generate all the tricks you thought were special.

• 1:42:00Max Tegmark?

  Yeah.

• 1:42:02Lex Fridman?

  I don't know, it's very, it, it's scary. It's almost painful, like a loss, a loss of innocence, maybe. Like, maybe when, when I was younger, I remember before I learned that sugar is bad for you, you should be on a diet. I remember I enjoyed candy deeply. In a way, I just can't anymore that I know is bad for me. I enjoyed it unapologetically, fully, just intensely. And I just, I lost that. Now, I feel like a little bit of that is lost for me with program, or being lost with programming. Similar as it is for the 3D modeler,

• 1:42:44Lex Fridman?

  no longer being able to really enjoy the art of modeling 3D things for gaming. I don't know. I don't know what to make sense of that. Maybe I would rediscover that the true magic of what it means to be human is connecting with other humans, to have conversations like this. I don't know. To have sex, to have, to eat food, to really intensify the value from conscious experiences versus like creating other stuff.

• 1:43:08Max Tegmark?

  You're pitching the rebranding again from Homo sapiens to Homo antiens.

• 1:43:12Lex Fridman?

  Homo sapiens, yeah.

• 1:43:13Max Tegmark?

  The meaningful experiences. And just to inject some optimism in this here, so we don't sound like a bunch of gloomers. You know, we can totally have our cake and eat it. You hear a lot of totally bullshit claims that we can't afford having more teachers. Have to cut the number of nurses. You know, that's just nonsense, obviously. With anything, even quite far short of AGI, we can dramatically improve, grow the GDP and produce this wealth of goods and services. It's very easy to create a world where everybody is better off than today. Including the richest people can be better off as well, right? It's not a zero sum game in a technology.

• 1:43:55Max Tegmark?

  Again, you can have two countries like Sweden and Denmark had all these ridiculous wars century after century. And sometimes Sweden got a little better off because it got a little bit bigger. And then Denmark got a little bit better off because Sweden got a bit smaller. And, but then we, then technology came along and we both got just dramatically wealthier without taking away from anyone else. So it's just a total win for everyone. And AI can do that on steroids. If you can build safe AGI, if you can build superintelligence, basically all the limitations that cause harm today can be completely eliminated. Right? It's a wonderful possibility. And this is not sci fi.

• 1:44:40Max Tegmark?

  This is something which is clearly possible according to laws of physics. And we can talk about ways of making it safe also. But unfortunately.

• 1:44:50Max Tegmark?

  That'll only happen if we steer in that direction. That's absolutely not the default outcome. That's why income inequality keeps going up. That's why the life expectancy in the U.S. has been going down. Now, I think it's four years in a row. I just read a heartbreaking study from the CDC about how something like one-third of all teenage girls in the U.S. have been thinking about suicide. You know, like, those are steps, and they're totally the wrong direction. And, and it's important to keep our eyes on the prize here that

• 1:45:26Max Tegmark?

  we can, we have the power now, for the first time in the history of our species, to harness artificial intelligence to help us really flourish and help bring out the best in our humanity, rather than the worst of it. To help us have really fulfilling experiences that feel truly meaningful. And, and you and I shouldn't sit here and dictate to future generations what they will be. Let them figure it out, but let's give them a chance to live and, and not foreclose all these possibilities for them by just messing things up, right?

• 1:46:01Lex Fridman?

  And for that, we'll have to solve the AI safety problem. I just, it would be nice if we can linger on exploring that a little bit. So, one interesting way to enter that discussion is you tweeted, and Elon replied, you tweeted, "Let's not just focus on whether GPT-4 will do more harm or good on the job market, but also whether its coding skills will hasten the arrival of superintelligence." That's something we've been talking about, right? So Elon proposed one thing in the reply, saying, "Maximum truth-seeking is my best guess for AI safety." Can you maybe steel man the case for this, uh, sense, this objective function of truth? And, uh, maybe make an argument against it?

• 1:46:45Lex Fridman?

  And in general, what are your different ideas to start approaching the, the solution to AI safety?

• 1:46:51Max Tegmark?

  I didn't see that reply, actually.

• 1:46:52Lex Fridman?

  Oh, interesting.

• 1:46:54Max Tegmark?

  So, but I really resonate with it because...

• 1:47:00Max Tegmark?

  AI is not evil. It caused people around the world to hate each other much more. But that's because we made it in a certain way. It's a tool. We can use it for great things and bad things. And we could just as well have AI systems. And this is part of my vision for success here. Truth-seeking AI that really brings us together again. Why do people hate each other so much between countries and within countries? It's because they each have totally different versions of the truth, right? If they all had the same truth that they trusted for good reason, because they could check it and verify it and not have to believe in some self-proclaimed authority, there wouldn't be nearly as much hate.

• 1:47:44Max Tegmark?

  There'd be a lot more understanding instead. And this is, I think, something AI can help enormously with. For example, a little baby step in this direction is this website called Metaculous, where people bet and make predictions, not for money, but just for their own reputation. And it's kind of funny, actually. You treat the humans like you treat AIs. You have a loss function where they get penalized if they're super confident on something and then the opposite happens. Yeah. Whereas if you're kind of humble and then you're like, I think it's 51% chance this is going to happen. And then the other happens, you don't get penalized much.

• 1:48:26Max Tegmark?

  And what you can see is that some people are much better at predicting than others. They've earned your trust, right? One project that I'm working on right now is the Outgrowth and Improve the News Foundation, together with the Metaculous folks, is seeing if we can really scale this up a lot with more powerful AI. Because I would love it. I would love for there to be a really powerful truth-seeking system where that is trustworthy because it keeps being right about stuff. And people who come to it and... And maybe look at its latest trust ranking of different pundits and newspapers, et cetera. If they want to know why some someone got a low score, they can click on it and see all the

• 1:49:10Max Tegmark?

  predictions that they actually made and how they turned out, you know. This is how we do it in science. You trust scientists like Einstein, who said something everybody thought was bullshit and turned out to be right. You get a lot of trust points, and he did it multiple times even. I think AI has the power to really heal a lot of the rifts we're seeing by creating trust systems. We it has to get away from this idea today with some fact checking sites, which might themselves have an agenda and you just trust it because of its reputation. You want to have it the sort of this sort of these sort of systems, they earn their trust and they're completely transparent.

• 1:49:54Max Tegmark?

  This, I think, would actually help a lot. That can, I think, help heal the very dysfunctional conversation that humanity has about how it's going to deal with all its biggest challenges in the world today. And then on the technical side, you know, another common sort of gloom comment I get from people are saying, we're just screwed, there's no hope, is, well, things like GPT-4 are way too complicated for a human to ever understand and prove that they can be trustworthy. They're forgetting that AI can help us prove that things work, right? And there's this very fundamental fact that in math, it's much harder to come up with

• 1:50:39Max Tegmark?

  a proof than it is to verify that the proof is correct. You can actually write a little proof checking code, it's quite short, that you can as human understand, and then it can check the most monstrously long proof ever generated even by a computer and say, yeah, this is valid. So right now, we have.

• 1:51:01Max Tegmark?

  This approach with virus checking software, that it looks to see if there's something you should not trust it. And if it can prove to itself that you should not trust that code, it warns you, right?

• 1:51:16Max Tegmark?

  What if you flip this around? And this is an idea, give credit to Steve, I'm 104. So that it will only run the code if it can prove, instead of not running it, if it can prove that it's not trustworthy. If it will only run it if it can prove that it's trustworthy. So it asks the code, prove to me that you're gonna do what you say you're gonna do.

• 1:51:36Max Tegmark?

  And it gives you this proof. And you, a little proof checker, can check it. Now you can actually trust an AI that's much more intelligent than you are, right?

• 1:51:48Max Tegmark?

  Because it's a problem to come up with this proof that you could never have found, that you should trust it.

• 1:51:54Lex Fridman?

  So this is the interesting point. I agree with you, but this is where Eliezer Yankowski might disagree with you. His claim, not with you, but with this idea,

• 1:52:05Lex Fridman?

  is his claim is a super intelligent AI would be able to know how to lie to you with such a proof.

• 1:52:14Max Tegmark?

  How to lie to you and give me a proof that I'm gonna think is correct?

• 1:52:17Lex Fridman?

  Yeah.

• 1:52:18Max Tegmark?

  But it's not me is lying to you. That's the trick, my proof checker.

• 1:52:22Lex Fridman?

  So yes. So his general idea is a super intelligent system can lie to a dumber proof checker. So you're going to have, as a system becomes more and more intelligent, there's going to be a threshold where a super intelligent system would be able to effectively lie to a slightly dumber AGI system. Like there's a threat, like he really focuses on this weak AGI to strong AGI jump, where the strong AGI can make all the weak AGIs think that it's just one of them, but it's no longer that. And that leap is when it runs away.

• 1:53:01Max Tegmark?

  Yeah, I don't buy that argument. I think no matter how super intelligent an AI is, it's never gonna be able to prove to me that there are only finitely many primes, for example.

• 1:53:13Max Tegmark?

  It just can't and it can try to snow me by making up all sorts of new weird rules of deduction that say, trust me, you know, the way your proof checker works is too limited and we have this new hyper math and it's true. But then I would, I would just take the attitude, okay, I'm going to forfeit some of these supposedly super cool technologies. I'm only going to go with the ones that I can prove in my own trusted proof checker. Then I don't, I think it's fine. There's still, of course, this is not something anyone has successfully implemented at this point, but I think it, I just give it as an example of hope. We don't have to do all the work ourselves, right? This is exactly the sort of very boring and

• 1:53:57Max Tegmark?

  tedious task that's perfect to outsource to an AI. And this is a way in which less powerful and less intelligent agents like us can actually continue to control and trust more powerful ones.

• 1:54:10Lex Fridman?

  So build AGI systems that help us defend against other AGI systems.

• 1:54:14Max Tegmark?

  Well, for starters, begin with a simple problem of just making sure that the system that you own or that's supposed to be loyal to you has to prove to itself that it's always going to do the things that you actually want it to do, right? And if it can't prove it, maybe it's still going to do it, but you won't run it. So you just forfeit some aspects of all the cool things AI can do. I bet you dollars to donuts, it can still do some incredibly cool stuff for you.

• 1:54:38Lex Fridman?

  Yeah.

• 1:54:39Max Tegmark?

  There are, there are other things too that we shouldn't sweep under the rug. Like, not every human agrees on exactly what, where, what direction we should go with humanity, right?

• 1:54:47Lex Fridman?

  Yes.

• 1:54:49Max Tegmark?

  And you've talked a lot about geopolitical things on this, on, on your podcast to this effect, you know, but I think that shouldn't distract us from the fact that there are actually a lot of things that everybody in the world virtually agrees on that, hey, you know, like having no humans on the planet.

• 1:55:09Max Tegmark?

  In a, in a, in a near future, nah, let's not do that, right? Uh huh. You look at something like the United Nations Sustainable Development Goals. Some of them are quite ambitious and basically all the countries agree. U S, China, Russia, Ukraine, they all agree. So instead of quibbling about the little things we don't agree on, let's start with the things we do agree on and, and, and get them done. Instead of being so distracted by all these things we disagree on, that Moloch wins. Because frankly, Moloch going wild now, it feels like a war on life playing out in front of our eyes.

• 1:55:50Max Tegmark?

  If you, if you just look at it from space, you know, we're on this planet. Beautiful vibrant ecosystem. Now we start chopping down big parts of it, even though nobody, most people thought that was a bad idea. Oh, we start doing ocean acidification, wiping out all sorts of species. Oh, now we have all these close calls, we almost had a nuclear war. And we're replacing more and more of the biosphere with non-living things. We're also replacing in our social lives, a lot of the things which were so valuable to humanity. A lot of social interactions now are replaced by people staring into their rectangles, right?

• 1:56:32Max Tegmark?

  And I, I'm not a psychologist, I'm out of my depth here. But I suspect that part of the reason why teen suicide and suicide in general in the U S, the record-breaking levels is actually caused by, again, AI technologies and social media. They're making people spend less time with, with, with actually, and actually just human interaction. We've all seen a bunch of good-looking people in restaurants staring into their rectangles instead of looking into each other's eyes, right? So that's also a part of the war on life that, that we're, we're replacing so many.

• 1:57:12Max Tegmark?

  Really life-affirming things by technology. We're putting technology between us. The technology that was supposed to connect us is actually distancing us, ourselves from each other. And then we're giving ever more power to things which are not alive. These large corporations are not living things, right? They're just maximizing profit. They're... I want to win the war on life. I think we humans, together with all our fellow living things on this planet, will be better off if we can remain in control over the non-living things and make sure that they work for us. I really think it can be done.

• 1:57:55Lex Fridman?

  Can you just linger on this maybe high-level philosophical disagreement with Eliezer Yudkowsky?

• 1:58:06Lex Fridman?

  In the hope you're stating. So he is very sure. He puts a very high probability, very close to one, depending on the day, he puts it at one, that AI is going to kill humans. That there's just... He does not see a trajectory which it doesn't end up with that conclusion. What trajectory do you see that doesn't end up there? And maybe can you see the point he's making? And can you also see a way out?

• 1:58:41Max Tegmark?

  First of all, I tremendously respect Eliezer Yudkowsky and his thinking. Second, I do share his view that there's a pretty large chance that we're not going to make it as humans. There won't be any humans on the planet in the not-too-distant future. And that makes me very sad. You know, we just had a little baby and I keep asking myself, you know...

• 1:59:09Max Tegmark?

  How old is he even gonna get, you know, and I ask myself. It feels, I said to my wife recently, it feels a little bit like I was just diagnosed with some sort of cancer, which has some, you know, risk of dying from and some risk of surviving, you know.

• 1:59:30Max Tegmark?

  Except this is a kind of cancer which will kill all of humanity. So I completely take seriously his his concerns. I think.

• 1:59:41Max Tegmark?

  But I don't, absolutely don't think it's hopeless. I think there is a, there is, first of all, a lot of momentum now. For the first time, actually, since the many, many years that have passed, since I and many others started warning about this, I feel most people are getting it now. I, I, I was just talking

• 2:00:07Max Tegmark?

  to this guy in the gas station near our house the other day. My, and he's like, I think we're getting replaced.

• 2:00:17Max Tegmark?

  And I think, so that's positive, that they're finally, we're finally seeing this reaction, which is the first step towards solving the problem.

• 2:00:26Max Tegmark?

  Second, I really think that this, this vision of only running AIs really, if the stakes are really high, that can prove to us that they're safe. It's really just virus checking in reverse again. I, I think it's scientifically doable. I don't think it's hopeless.

• 2:00:45Max Tegmark?

  We might have to forfeit some of the technology that we could get if we were putting blind faith in our AIs, but we're still gonna get amazing stuff.

• 2:00:52Lex Fridman?

  Do you envision a process with a proof checker, like something like GPT-4, GPT-5? Will go through a process of rigorous interrogation?

• 2:01:00Max Tegmark?

  No, I think it's hopeless. That's like trying to prove there about five spaghetti.

• 2:01:05Max Tegmark?

  What I think, well, how the whole, the vision I have for success is instead that, you know, just like we human beings were able to look at our brains and, and distill out the key knowledge. Galileo, when his dad threw him an apple when he was a kid, he was able to catch it because his brain could in this funny spaghetti kind of way, you know, predict how parabolas are going to move, his conum on system one, right? But then he got older and it's like, wait, this is a parabola. It's, it's y equals x squared. I can distill this knowledge out and today you can easily program it into a computer and it can simulate not just that, but how to get to Mars and so on, right? I envision a similar process where we use the amazing learning power of neural networks

• 2:01:48Max Tegmark?

  to discover the knowledge in the first place, but we don't stop with a black box and, and use that. We then do a second round of AI where we use automated systems to extract out the knowledge and see what is it, what are the insights it's had. Okay. And it's, and then we, we put that knowledge into a completely different kind of architecture or programming language or whatever that's, that's made in a way that it can be both really efficient and also is more amenable to, to very formal verification.

• 2:02:19Lex Fridman?

  Mm hmm.

• 2:02:20Max Tegmark?

  That's, that's my vision. I'm not saying, sitting here saying I'm confident a hundred percent sure that it's going to work, you know, but I don't think, the chance is certainly not zero either. And it will certainly be possible to do for a lot of really cool AI applications that we're not using now. So we can have a lot of the fun that we're excited about if we, if we do this. We're going to need a little bit of time. That's why it's good to pause and, and put in place requirements. One more thing also, I, I think, you know, someone might think, well, zero percent chance we're going to survive. Let's just give up,

• 2:03:00Max Tegmark?

  right? That's very dangerous because there's no more guaranteed way to fail than to convince yourself that it's impossible and not to try.

• 2:03:13Max Tegmark?

  You know, any, if you, you know, when you study history and military history, the first thing you learn is that that's how you do psychological warfare. You persuade the other side that it's hopeless, so they don't even fight. And then, then of course you win, right? Let's not do this psychological warfare on ourselves and say there's a hundred percent probability we're all gonna, we're all screwed anyway.

• 2:03:39Max Tegmark?

  Sadly, I do get that a little bit, sometimes from, from, uh, some young people who are like so convinced that we're all screwed, that they're like, I'm just gonna play game, play computer games and do drugs and, because we're screwed anyway, right? It's important to keep the hope alive because it actually has a causal impact and make, makes it more likely that we're gonna succeed.

• 2:04:00Lex Fridman?

  It seems like the people that actually build solutions to a problem seemingly impossible to solve problems are the ones that believe.

• 2:04:08Max Tegmark?

  Yeah.

• 2:04:09Lex Fridman?

  They're the ones who are the optimists.

• 2:04:10Max Tegmark?

  Yeah.

• 2:04:11Lex Fridman?

  And it's like, uh, it seems like there's some fundamental law to the universe where fake it till you make it kinda works. Like, believe it's possible and it becomes possible.

• 2:04:21Max Tegmark?

  Yeah. Was it Henry Ford who said that if you can, if you tell yourself that it's impossible, it is? So let's not make that mistake.

• 2:04:32Lex Fridman?

  Yeah.

• 2:04:32Max Tegmark?

  And this is a big mistake society is making, you know, I think all in all, everybody's so gloomy and the media are also very biased towards if it bleeds, it leads and gloom and doom, right? So, um, most visions of the future we have are dystopian, which really demotivates people. We wanna really, really, really focus on the upside also to give people the willingness to fight for it. And, um, for AI, you and I mostly talked about gloom here again, but let's not remember, not forget that, you know, we have probably both lost someone we really cared about to some disease that we were told was incurable. Well, it's not.

• 2:05:15Max Tegmark?

  There's no law of physics saying you have to die of that cancer or whatever. Of course you can cure it. And there's so many other things where, that we, with our human intelligence have also failed to solve on this planet, which AI could also very much help us with, right? So if we can get this right, just be a little more chill and slow down a little bit till we get it right. It's mind blowing how awesome our future can be. We talked a lot about stuff on Earth can be great. But even if you really get ambitious and look up at the skies, right? There's no reason we have to be stuck on this planet for the rest of the remain for billions of years to come.

• 2:05:57Max Tegmark?

  We totally understand now as lots of physicists let life spread out into space to other solar systems, to other galaxies and flourish for billions of billions of years. And this to me is a very, very hopeful vision. That really motivates me to to fight them. Coming back to the end, something you talked about again, you know, this, the struggle, how the human struggle is one of the things that also really gives meaning to our lives. If there's ever been an epic struggle, this is it. And isn't it even more epic if you're the underdog? If most people are telling you this is gonna fail, it's impossible, right? And you persist. And you succeed.

• 2:06:40Max Tegmark?

  That's what we can do together as a species on this one. A lot of pundits are ready to count this out.

• 2:06:47Lex Fridman?

  Both in the battle to keep AI safe and becoming a multi planetary species.

• 2:06:52Max Tegmark?

  Yeah, and they're they're the same challenge. If we can keep AI safe, that's how we're gonna get multi planetary very efficiently.

• 2:06:59Lex Fridman?

  I have some sort of technical questions about how to get it right. So one idea that I'm not even sure what the right answer is to is should systems like GPT four be open sourced in whole or in part? Can you make the, can you see the case for either?

• 2:07:19Max Tegmark?

  I think the answer right now is no. I think the answer early on was yes. So we could bring in all the wonderful great thought process of everybody on this. But asking, should we open source GPT four now is just the same as if you say, well, is it good? Should we open source? How to build really small nuclear weapons? Should we open source? How to make bioweapons? Should we open source? How to make?

• 2:07:51Max Tegmark?

  A new virus that kills ninety percent of everybody who gets it, of course, we shouldn't.

• 2:07:55Lex Fridman?

  So it's already that powerful. It's already that powerful that we have to respect the power of the systems we've built.

• 2:08:04Max Tegmark?

  The knowledge that you get from open sourcing everything we do now might very well be powerful enough that people looking at that

• 2:08:16Max Tegmark?

  can use it to build the things that are really threatening again. Let's get it. Remember, OpenAI is GPT-4 is a baby AI, baby, sort of baby proto, almost little bit AGI, according to what Microsoft's recent paper said, right? It's not that that we're scared of. What we're scared about is people taking that who are who might be a lot less responsible than the company that made it, right? And just going to town with it. That's why we want to, it's an information hazard. There are many things which, you know, are not open sourced right now in society for a very good reason. Like, how do you make

• 2:08:59Max Tegmark?

  certain kind of very powerful toxins out of stuff you can buy at Home Depot? We don't open source those things for a reason. And this is really no different.

• 2:09:11Lex Fridman?

  So.

• 2:09:12Max Tegmark?

  And I'm saying that, I have to say, it feels a bit weird, in a way a bit weird to say it because MIT is like the cradle of the open source movement.

• 2:09:20Lex Fridman?

  Yeah.

• 2:09:20Max Tegmark?

  And I love open source in general, power to the people, let's say. But there's always going to be some stuff that you don't open source. And, you know, it's just like you don't open source. So we have a three month old baby, right? When he gets a little bit older, we're not going to open source to him all the most dangerous things he could do in the house.

• 2:09:40Lex Fridman?

  Yeah.

• 2:09:40Max Tegmark?

  Right?

• 2:09:42Lex Fridman?

  But it does, it's a weird feeling because this is one of the first moments in history where there's a strong case to be made not to open source software. This is when the software has become

• 2:09:56Max Tegmark?

  Yeah.

• 2:09:57Lex Fridman?

  too dangerous.

• 2:09:57Max Tegmark?

  Yeah, but it's not the first time that we didn't want to open source a technology.

• 2:10:01Lex Fridman?

  Technology, yeah.

• 2:10:05Lex Fridman?

  Is there something to be said about how to get the release of such systems right? Like GPT four and GPT five? So OpenAI went through a pretty rigorous effort for several months. You could say it could be longer, but nevertheless, it's longer than you would have expected of trying to test the system to see like, what are the ways it goes wrong? To make it very difficult for people, somewhat difficult for people to ask things. How do I make a bomb for $1? Or how do I say I hate a certain group on Twitter in a way that doesn't get me blocked from Twitter, banned from Twitter? Those kinds of questions.

• 2:10:42Max Tegmark?

  Yeah.

• 2:10:43Lex Fridman?

  So you basically use the system to do harm.

• 2:10:47Max Tegmark?

  Yeah.

• 2:10:49Lex Fridman?

  Is there something you could say about ideas you have just on looking, having thought about this problem of AI safety? How to release such system? How to test such systems when you have them inside the company?

• 2:11:03Max Tegmark?

  Yeah, so.

• 2:11:07Max Tegmark?

  A lot of people say that the two biggest risks from large language models are.

• 2:11:16Max Tegmark?

  It's spreading disinformation, harmful information of various types.

• 2:11:22Max Tegmark?

  And second, being used for offensive cyber weapon.

• 2:11:29Max Tegmark?

  So I think those are not the two greatest threats. They're very serious threats and it's wonderful that people are trying to mitigate them. I see a much bigger elephant in the room is how is this is going to disrupt our economy in a huge way, obviously, and maybe take away a lot of the most meaningful jobs. And an even bigger one is the one we spent so much time talking about here, that that this

• 2:11:52Max Tegmark?

  becomes the boot loader for the more powerful AI.

• 2:11:56Lex Fridman?

  Write code connected to the internet, manipulate humans.

• 2:11:59Max Tegmark?

  Yeah. And before we know it, we have something else, which is not at all a large language model. It looks nothing like it, but which is way more intelligent and capable and has goals.

• 2:12:09Max Tegmark?

  And that's the that's the elephant in the room. And and obviously, no matter how hard any of these companies have tried,

• 2:12:17Max Tegmark?

  they that's not something that's easy for them to verify with a large language models. And the only way to be really lower that risk a lot would be to not let, for example, to not never let it read any code, not train on that and not put it into an API. And not not give it access to so much information about how to manipulate humans. So but that doesn't mean you still can't make a lot a ton of money on them, you know. We're going to just watch now this coming year, right? Microsoft is rolling out the new Office Suite where you go into Microsoft Word and give it a prompt and it writes the whole text for you and then you edit it.

• 2:13:02Max Tegmark?

  And then you're like, oh, give me a PowerPoint version of this and it makes it. And now take the spreadsheet and blah. And, you know, the all of those things, I think, are you can debate the economic impact of it and whether society is prepared to deal with this disruption. But those are not the things which that's not the elephant of the room that keeps me awake at night for wiping out humanity. And I think that's the biggest misunderstanding we have. A lot of people think that we're scared of like automatic spreadsheets. That's not the case. That's not what Eliezer was freaked out about either.

• 2:13:38Lex Fridman?

  Is there, in terms of the actual mechanism of how AI might kill all humans? So something you've been outspoken about, you've talked about a lot, is it autonomous weapon systems? So the use of AI in war. Is that one of the things that still you carry concern for as these systems become more and more powerful?

• 2:14:01Max Tegmark?

  I carry concern for it, not that all humans are going to get killed by slaughter bots, but rather just this express route into Orwellian dystopia, where it becomes much easier for very few to kill very many. And therefore, it becomes very easy for very few to dominate very many, right?

• 2:14:19Max Tegmark?

  AI, if you want to know how AI could kill all people, just ask yourself, we humans have driven a lot of species extinct. How did we do it?

• 2:14:28Max Tegmark?

  You know, we were smarter than them.

• 2:14:31Max Tegmark?

  Usually, we didn't do it even systematically by going around one on one, one after the other and stepping on them or shooting them or anything like that. We just like chopped down their habitat because we needed it for something else. In some cases, we did it by putting more carbon dioxide in the atmosphere because of some reason that those animals didn't even understand. And now they're gone, right? So if if you're in AI and you just want to figure something out, then you decide, you know, we just really need the space here to build more compute facilities, you know.

• 2:15:10Max Tegmark?

  If that's the only goal it has, you know, we are just the sort of accidental roadkill along the way. And you could totally imagine, yeah, maybe this oxygen is kind of annoying because it caused more corrosion. So let's get rid of the oxygen. And good luck surviving after that. You know, I'm not particularly concerned that they would want to kill us just because that would be like a goal in itself. You know, when we've driven a number we've driven a number of the elephant species extinct, right? It wasn't because we didn't like elephants.

• 2:15:45Max Tegmark?

  The basic problem is you just don't want to give you don't want to cede control over your planet to some other more intelligent entity that doesn't share your goals. It's that simple. So which brings us to an other key challenge, which AI safety researchers have been grappling with for a long time. Like, how do you make AI, first of all, understand our goals and then adopt our goals and then retain them as they get smarter, right?

• 2:16:19Max Tegmark?

  All three of those are really hard, right? Like, a human child, first, they're just not smart enough to understand our goals. They can't even talk. And then eventually they're teenagers and understand our goals just fine, but they don't share.

• 2:16:37Lex Fridman?

  Yeah.

• 2:16:39Max Tegmark?

  But there is fortunately a magic phase in the middle where they're smart enough to understand our goals and malleable enough that we can hopefully with good parenting and teach them right from wrong and instead good goal, instill good goals in them, right? So those are all tough challenges with computers. And then, you know, even if you teach your kids good goals when they're little, they might outgrow them too, and that's a challenge for machines to keep improving. So these are a lot of hard, hard challenges we're up for, but I don't think any of them are insurmountable. The fundamental reason why Eliezer looked so depressed when I last saw him was because he felt there just wasn't enough time.

• 2:17:20Lex Fridman?

  Oh, not that it was unsolvable. There's just not enough time.

• 2:17:24Max Tegmark?

  He was hoping that humanity was going to take this threat more seriously so we would have more time.

• 2:17:29Lex Fridman?

  Yeah.

• 2:17:29Max Tegmark?

  And now we don't have more time. That's why the open letter is calling for more time.

• 2:17:38Lex Fridman?

  But even with time, the AI alignment problem seems to be really difficult.

• 2:17:44Max Tegmark?

  Oh, yeah. But it's also the most worthy problem, the most important problem for humanity to ever solve, because if we solve that one, Lex, that aligned AI can help us solve all the other problems.

• 2:17:59Lex Fridman?

  Because it seems like it has to have constant humility about its goal, constantly question the goal. Because as you optimize towards a particular goal and you start to achieve it, that's when you have the unintended consequences, all the things you've mentioned about. So how do you enforce and code a constant humility as your ability become better and better and better and better?

• 2:18:21Max Tegmark?

  Stuart, Professor Stuart Russell at Berkeley, who's also one of the...

• 2:18:26Max Tegmark?

  Driving forces behind this letter, he

• 2:18:30Max Tegmark?

  has a whole research program about this.

• 2:18:34Max Tegmark?

  I think of it as AI humility, exactly, although he calls it inverse reinforcement learning and other nerdy terms, but it's about exactly that. Instead of telling the AI, here's his goal, go optimize the bejesus out of it. You tell it, okay.

• 2:18:51Max Tegmark?

  Do what I want you to do, but I'm not going to tell you right now what it is I want you to do, you need to figure it out. So then you give the incentive to be very humble and keep asking you questions along the way, is this what you really meant? Is this what you wanted? And, oh, this other thing I tried didn't work, it seemed like it didn't work out right. Should I try it differently?

• 2:19:11Max Tegmark?

  What's nice about this is it's not just philosophical mumbo jumbo, it's theorems and technical work that with more time, I think you can make a lot of progress. And there are a lot of brilliant people now working on AI safety. We're just not, we just need to give them a bit more time.

• 2:19:26Lex Fridman?

  But also not that many relative to the scale of the problem.

• 2:19:29Max Tegmark?

  No, exactly. There should be, at least just like every university worth its name, has some cancer research going on in its biology department, right? Every university that does computer, that does computer science should have a real effort in this area and it's nowhere near that. This is something I hope is changing now, thanks to the GPT four, right? So I think if there's a silver lining to what's happening here, even though I think many people would wish it would have been rolled out more carefully, is that this might be the wake up call that humanity needed to really

• 2:20:10Max Tegmark?

  stop the, stop fantasizing about this being a hundred years off and stop fantasizing about this being completely controllable and predictable because.

• 2:20:19Max Tegmark?

  It's so obvious. It's it's not predictable, you know. Why is it that open that that I think it was G P chat G P T tried to persuade a a journalist or was it G P T four to divorce his wife, you know? It was not because the the engineers who built it was like,

• 2:20:43Max Tegmark?

  let's put this in here and and and screw a little bit with people. They hadn't predicted at all. They built a giant black box,

• 2:20:53Max Tegmark?

  trained to predict the next word and got all these emergent properties and oops, it did this, you know?

• 2:21:02Max Tegmark?

  I I think this is a very powerful wake up call and anyone watching this is not scared. I would encourage them to just play a bit more with these these tools. They're out there now like G P T four. And

• 2:21:19Max Tegmark?

  so wake up call is first step. Once you've woken up, then got to slow down a little bit, the risky stuff to give a chance to all everyone who's woken up to to catch up with this on the safety front.

• 2:21:30Lex Fridman?

  You know what's interesting is, you know, M I T, that's computer science, but in general, but let's just even say computer science curriculum. How does the computer science curriculum change now? You mentioned you mentioned programming.

• 2:21:44Max Tegmark?

  Yeah.

• 2:21:45Lex Fridman?

  Like, why would you be, when I was coming up, programming is a prestigious position? Like, why would you be dedicating crazy amounts of time to become an excellent programmer? Like, the nature of programming is fundamentally changing.

• 2:22:00Max Tegmark?

  The nature of our entire education system is completely turned on its head.

• 2:22:06Lex Fridman?

  Has anyone been able to like load that in and like think because it's really turning?

• 2:22:12Max Tegmark?

  I mean, some English professors, some English teachers are beginning to really freak out now.

• 2:22:16Lex Fridman?

  Yeah.

• 2:22:16Max Tegmark?

  Right? They they give an essay assignment, then they get back all this fantastic prose, like this is the style of Hemingway. And then they realize they have to completely rethink. And even, you know, just like we stopped teaching, writing scripts. Is that what you say in English?

• 2:22:36Lex Fridman?

  Yeah, handwritten, yeah.

• 2:22:37Max Tegmark?

  Yeah, when when everybody started typing, you know, like so much of what we teach our kids today.

• 2:22:46Lex Fridman?

  Yeah, I mean, that's.

• 2:22:50Lex Fridman?

  Everything is changing and it's changing very, it's changing very quickly. And so much of us understanding how to deal with the big problems of the world is through the education system. And if the education system is being turned on its head, then what's next? It feels like having these kinds of conversations is essential to try to figure it out. And everything's happening so rapidly. I don't think there's even, speaking of safety, what the broad AI safety defined, I don't think most universities have courses on AI safety.

• 2:23:21Max Tegmark?

  No.

• 2:23:22Lex Fridman?

  It's like a philosophy seminar.

• 2:23:24Max Tegmark?

  And like, I'm an educator myself, so it pains me to see this, say this, but I feel our education right now is completely obsoleted by what's happening, you know? You put a kid into first grade and then you envision like, and then they're going to come out of high school 12 years later. And you've already pre-planned now what they're going to learn when you're not even sure if there's going to be any world left to come out to, right? Clearly, you need to have a much more opportunistic education system that keeps adapting itself very rapidly as society readapt. The skills that were really useful when the curriculum was written, I mean, how many of those skills are going to get you a job in 12 years?

• 2:24:09Max Tegmark?

  I mean, seriously?

• 2:24:10Lex Fridman?

  If we just linger on the GPT-4 system a little bit, you kind of hinted at it, especially talking about the importance of consciousness in the, in the human mind with homo sentience. Do you think GPT-4 is conscious?

• 2:24:29Max Tegmark?

  I love this question. So let's define consciousness first because in my experience, like 90% of all arguments about consciousness boil down to the two people arguing, having totally different definitions of what it is, and they're just shouting at each other. I define consciousness as subjective experience. Right now, I'm experiencing colors and sounds and emotions, you know? But does a self-driving car experience anything? That's the question about whether it's conscious or not, right?

• 2:25:06Max Tegmark?

  Other people think you should define consciousness differently. Fine by me, but then maybe use a different word for it. Or I'm gonna use consciousness for this at least.

• 2:25:18Max Tegmark?

  So. But if people hate the, yeah. So is GPT four conscious? Does GPT four have subjective experience? Short answer, I don't know. Because we still don't know what it is that gives this wonderful subjective experience. That is kind of the meaning of our life, right? Because meaning itself, feeling of meaning is a subjective experience. Joy is a subjective experience. Love is a subjective experience. We don't know what it is. I've written some papers about this. A lot of people have.

• 2:25:52Max Tegmark?

  Giulio Tononi, professor, has stuck his neck out the farthest and written down actually a very bold mathematical conjecture for what's the essence of conscious information processing. He might be wrong, he might be right, but we should test it. He postulates that consciousness has to do with loops in the information processing. So our brain has loops. Information can go round and round. In in computer science nerd speak, you call it a recurrent neural network where some of the output gets fed back in again. And with his

• 2:26:30Max Tegmark?

  mathematical formalism, if it's a feed forward neural network where information only goes in one direction, like from your eye retina into the back of your brain, for example, that's not conscious. So he would predict that your retina itself isn't conscious of anything.

• 2:26:46Max Tegmark?

  Or a video camera. Now, the interesting thing about GPT four is it's also one way flow of information. So if Tononi is right, and GPT four is a very intelligent zombie that can do all this smart stuff, but isn't experiencing anything. And this is

• 2:27:06Max Tegmark?

  both a relief in that you don't have, if it's true, in that you don't have to feel guilty about turning off GPT four and wiping its memory whenever a new user comes along. I wouldn't like if someone used that to me, neuralized me like in Men in Black. But it's also creepy that you can have very high intelligence, perhaps then it's not conscious. Because if we get replaced by machines.

• 2:27:34Max Tegmark?

  It's sad enough that humanity isn't here anymore because I kind of like humanity. But at least if the machines were conscious, they could be like, well, but they are our descendants and maybe we, they have our values and they're our children. But if, if Tenoni is right, and it's all, these are all transformers that are not in the sense of Hollywood, but in the sense of these one-way direction neural networks. So they're all the zombies. That's the ultimate zombie apocalypse now. We have this universe that goes on with great construction projects and stuff, but there's no one experiencing anything. That would be like the ultimate depressing future.

• 2:28:15Max Tegmark?

  So I actually think as we move forward to the building more advanced, I should do more research on figuring out what kind of information processing actually has experience because I think that's what it's all about. And I completely don't buy the dismissal that some people, some people will say, well, this is all bullshit because consciousness equals intelligence.

• 2:28:36Lex Fridman?

  Right.

• 2:28:37Max Tegmark?

  It's obviously not true. You can have a lot of conscious experience when you're not really accomplishing any goals at all, you're just reflecting on something. And you can sometimes have things, doing things that require intelligence probably without being, being conscious.

• 2:28:54Lex Fridman?

  But I also worry that we humans won't, will discriminate against AI systems that clearly exhibit consciousness, that we will not allow AI systems to have consciousness. We'll come up with theories about measuring consciousness that will say, this is a lesser being. And this, I worry about that because maybe we humans will create something that is better than us humans in the, in the way that we find beautiful, which is they, they have a deeper subjective experience of reality. Not only are they smarter, but they feel deeper and we humans will hate them for it.

• 2:29:37Lex Fridman?

  As we, as human history has shown, there'll be the other will try to suppress it. It'll create conflict, it'll create war, all of this. I worry about this too.

• 2:29:47Max Tegmark?

  Are you saying that we humans sometimes come up with self-serving arguments? No, we would never do that, would we?

• 2:29:54Lex Fridman?

  Well, that's the danger here is even in this early stages, we might create something beautiful.

• 2:29:59Max Tegmark?

  Yeah.

• 2:29:59Lex Fridman?

  And we'll erase its memory.

• 2:30:03Max Tegmark?

  I was horrified as a kid when someone started boiling, boiling lobsters. Like, oh my God, that's so cruel. And some grown up there back in Sweden said, oh, it doesn't feel pain. I'm like, how do you know that? Oh, scientists have shown that.

• 2:30:23Max Tegmark?

  And then there was a recent study where they showed that lobsters actually do feel pain when you boil them. So they banned lobster boiling in Switzerland now, to kill them in a different way first. So presumably that's scientific research boiled out to someone asked the lobster, does it hurt?

• 2:30:39Lex Fridman?

  Survey.

• 2:30:39Max Tegmark?

  And we do the same thing with cruelty to farm animals also, all these self-serving arguments for why they're fine. And yeah, so we should certainly be watchful. I think step one is just be humble and acknowledge that consciousness is not the same thing as intelligence. And I believe that consciousness still is a form of information processing where it's really information being aware of itself in a certain way. And let's study it and give ourselves a little bit of time. And I think we will be able to figure out actually what it is that causes consciousness. So then we can make probably unconscious robots that do the boring jobs that we would feel immoral to give to machines. But if you have a companion robot taking care of your mom or something like that, she would probably want it to be conscious, right?

• 2:31:24Max Tegmark?

  So the emotions it seems to display aren't fake. All these things can be done in a good way if we give ourselves a little bit of time and don't run and take on this challenge.

• 2:31:37Lex Fridman?

  Is there something you could say to the timeline that you think about, about the development of AGI? Depending on the day, I'm sure that changes for you. But when do you think there'll be a really big leap in intelligence where you definitively say we have built AGI? Do you think it's one year from now, five years from now, 10, 20, 50? What's your gut say?

• 2:32:05Max Tegmark?

  Honestly,

• 2:32:09Max Tegmark?

  for the past decade, I've deliberately given very long timelines because I didn't want to fuel some kind of stupid Moloch race.

• 2:32:16Lex Fridman?

  Yeah.

• 2:32:17Max Tegmark?

  But I think that cat has really left the bag now. And I think we might be very, very close.

• 2:32:26Max Tegmark?

  I don't think the Microsoft paper is totally off when they say that there are some glimmers of AGI. It's not AGI yet. It's not an agent. There's a lot of things it can't do. But I wouldn't bet very strongly against it happening very soon. That's why we decided to do this open letter. Because, you know, if there's ever been a time to pause, you know, it's today.

• 2:32:54Lex Fridman?

  There's a feeling like this GPT-4 is a big transition into waking everybody up to the effectiveness of these systems.

• 2:33:02Max Tegmark?

  Yeah.

• 2:33:02Lex Fridman?

  And so the next version will be big.

• 2:33:06Max Tegmark?

  Yeah. If that next one isn't AGI, maybe the next, next one will. And there are many companies trying to do these things. And the basic architecture of them is not some sort of super well-kept secret. So this is, this is a time to, a lot of people have said for many years that there will come a time when we want to pause a little bit.

• 2:33:28Max Tegmark?

  That time is now.

• 2:33:32Lex Fridman?

  You have spoken about and thought about nuclear war a lot. Over the past year, we've seemingly have come closest to the precipice of nuclear war than, at least in my lifetime.

• 2:33:49Max Tegmark?

  Mm hmm. Yeah.

• 2:33:51Lex Fridman?

  What do you learn about human nature from that?

• 2:33:54Max Tegmark?

  It's our old friend Moloch again. It's really scary to see it where.

• 2:34:02Max Tegmark?

  America doesn't want there to be a nuclear war. Russia doesn't want there to be a global nuclear war either. We both know that it would just be another... If we just try to do it, if both sides try to launch first, it's just another suicide race, right? So why is it the way you said that this is the closest we've come since 1962? In fact, I think we've come closer now than even the Cuban Missile Crisis. It's because of Moloch. You have these other forces. On one hand, you have the West saying that we have to drive Russia out of Ukraine. It's a matter of pride. And we've staked so much on it that it would be seen as a huge

• 2:34:45Max Tegmark?

  loss of credibility of the West if we don't drive Russia out entirely of the Ukraine. And on the other hand, you have Russia, and you have the Russian leadership, who knows that if they get completely driven out of Ukraine, it's not just going to be very humiliating for them, but it often happens when countries lose wars that things don't go so well for their leadership either. Like, you remember when Argentina invaded the Falkland Islands? The military junta that ordered that, right?

• 2:35:26Max Tegmark?

  People were cheering on the streets at first when they took it. And then when they got their butt kicked by the British, you know what happened to those guys? They were out. And I believe those who are still alive are in jail now, right? So the Russian leadership is entirely cornered, where they know that just getting driven out of Ukraine is not an option.

• 2:35:59Max Tegmark?

  So this to me is a typical example of Moloch. You have these incentives of the two parties where both of them are just driven to escalate more and more, right? If Russia starts losing in the conventional warfare, the only thing they can do, turns their back against the wall, is to keep escalating.

• 2:36:16Lex Fridman?

  Uh huh.

• 2:36:17Max Tegmark?

  And but and the West has put itself in the in the situation now where we're sort of already committed to drive Russia out. So the only option the West has is to call Russia's bluff and keep sending in more weapons. This really bothers me because Moloch can sometimes drive competing parties to do something which is ultimately just really bad for both of them. And, you know, what makes me even more worried is not just that I it's difficult to see an ending, a quick peaceful ending to this tragedy that doesn't involve some horrible escalation, but also that we understand more clearly now just how horrible it was gonna it would be.

• 2:36:59Max Tegmark?

  There was an amazing paper that was published in Nature Food this August by some of the top researchers who've been studying nuclear winter for a long time. And what they basically did was they combined climate models with food agricultural models. So instead of just saying, yeah, you know, it gets really cold, blah, blah, blah, they figured out actually how many people would die in the different different countries.

• 2:37:27Lex Fridman?

  Uh huh.

• 2:37:28Max Tegmark?

  And it's a it's pretty mind blowing, you know. So basically what happens, you know, is the thing that kills the most people is not the explosions, it's not the radioactivity, it's not the EMP mayhem, it's not the rampaging mobs foraging food. No, it's it's the fact that you get so much smoke coming up from the burning cities into the stratosphere that.

• 2:37:50Max Tegmark?

  It spreads around the earth from the jet streams. So in typical malls, you get like ten years or so where it's just crazy cold. And during the first year after the war and their malls, the temperature drops in Nebraska and in the Ukraine bread baskets, you know, by like twenty Celsius or so, if I remember. No, yeah, twenty, thirty Celsius, depending on where you are, forty Celsius in some places, which is, you know, forty Fahrenheit to eighty Fahrenheit colder than what would it normally be. So, you know, I'm not good at farming, but if it's snowing, if it drops below freezing pretty much almost most days

• 2:38:35Max Tegmark?

  in July, and then like, that's not good. So they worked out, they put this into their farming models. And what they found was really interesting. The countries that get the most hard hit are the ones in the northern hemisphere. So in the US, and one model they had, they had about ninety nine percent of all Americans starving to death. In Russia and China and Europe, also about ninety nine percent, ninety eight percent starving to death. So you might be like, oh, it's kind of poetic justice that both the Russians and the Americans, ninety nine percent of them have to pay for it because it was their bombs that did it. But, you know, that doesn't particularly cheer people up in Sweden or

• 2:39:14Max Tegmark?

  other random countries that had nothing to do with it, right? And it, I think it hasn't entered the mainstream, not understanding very much, just like how bad this is. Most people, especially a lot of people in decision making positions, still think of nuclear weapons as something that makes you powerful. Scary, powerful. They don't think of it as something where, yeah, just to within a percent or two, you know, we're all just just gonna starve to death. And

• 2:39:52Lex Fridman?

  And starving to death is

• 2:39:55Lex Fridman?

  Is um the worst way to die as Baltimore is all all the families in history show. The torture involved in that.

• 2:40:05Max Tegmark?

  Probably brings out the worst in people also when when people are desperate like this. It's not so. Some people I've heard some people say that. If that's what's going to happen, they'd rather be at ground zero and just get vaporized, you know. And but uh. So, but I think people underestimate the risk of this because they they. They aren't afraid of Moloch. They think, oh, it's just gonna be because humans don't want this, so it's not gonna happen. That's the whole point of Moloch, that things happen that nobody wanted.

• 2:40:38Lex Fridman?

  And that applies to nuclear weapons and that applies to AGI.

• 2:40:44Max Tegmark?

  Exactly. And it applies to some of the things that people have gotten most upset with capitalism for also, right? Where everybody was just kind of trapped, you know. It's not to see if some company does something. It causes a lot of harm, and it's not that the CEO is a bad person, but she or he knew that, you know. The other, all the other companies were doing this too. So Moloch is um. As a formidable foe, I hope wish someone make him make make good movies. So we can see who the real enemy is, so we don't. Because we're not fighting against each other. Moloch makes us fight against each other. That's more than that's what Moloch's superpower is.

• 2:41:30Max Tegmark?

  The hope here is any kind of technology or other mechanism that lets us instead realize. That we're finding the wrong enemy, right?

• 2:41:40Lex Fridman?

  It's such a fascinating battle.

• 2:41:42Max Tegmark?

  It's not us versus them, it's us versus it.

• 2:41:45Lex Fridman?

  Yeah, we are fighting Moloch for human survival. We as a civilization.

• 2:41:51Max Tegmark?

  Have you seen the movie Needful Things? It's a Stephen King novel. I love Stephen King and Max Von Sydow, Swedish actor is playing the guy is. It's brilliant. Exactly. I just thought I hadn't thought about that until now. But that's the closest I've seen to a movie about Moloch. I don't want to spoil the film for anyone who wants to watch it. But basically, it's about this guy who turns out that you can interpret him as the devil or whatever, but he doesn't actually ever go around and kill people or torture people or go burning coal or anything. He makes everybody fight each other, makes everybody hate, fear each other, hate each other, and then kill each other. So that's the movie about Moloch, you know?

• 2:42:34Lex Fridman?

  Love is the answer. That seems to be one of the ways to fight Moloch is by compassion, by seeing the common humanity.

• 2:42:46Max Tegmark?

  Yes, yes. And to not sound, so we don't sound like, like, what's it, Kumbaya tree huggers here, right? We're not just saying love and peace, man. We're trying to actually help people understand the true facts about the other side and feel the compassion because the truth makes you more compassionate, right? So that's why I really like using AI for truth and for truth-seeking technologies can, that can, as a result, you know, get us more love than hate.

• 2:43:31Max Tegmark?

  And even if you can't get love, you know, settle for, settle for some understanding, which already gives compassion. If someone is like, you know, I really disagree with you, Lex, but I can see where you're coming from. You're not a bad person who needs to be destroyed, but I disagree with you and I'm happy to have an argument about it, you know? That's a lot of progress compared to where we are 2023 in the public space, wouldn't you say?

• 2:44:00Lex Fridman?

  If we solve the AI safety problem, as we've talked about, and then you, Max Tegmark, who has been talking about this for many years, get to sit down with the AGI, with the early AGI system on a beach with a drink, what kind of, what would you ask her? What kind of question would you ask? What would you talk about?

• 2:44:23Lex Fridman?

  Something so much smarter than you.

• 2:44:27Lex Fridman?

  Would you be afraid?

• 2:44:27Max Tegmark?

  I knew you were gonna get me with a really zinger of a question. That's a good one.

• 2:44:32Lex Fridman?

  Would you be afraid to ask some questions?

• 2:44:36Max Tegmark?

  No.

• 2:44:37Lex Fridman?

  So you.

• 2:44:37Max Tegmark?

  I'm not afraid of the truth. I'm very humble. I know I'm just a meat bag with all these flaws, you know, but yeah, I I have, we talked a lot about homo sentience. I've already tried that for a long time with myself. Just so that is what's really valuable about being alive for me is that I have these meaningful experiences. It's not that I'm have what I'm good at this or good at that or whatever. There's so much I suck at and.

• 2:45:05Lex Fridman?

  So you're not afraid for the system to show you just how dumb you are?

• 2:45:08Max Tegmark?

  No, no. In fact, my son reminds me of that pretty frequently.

• 2:45:12Lex Fridman?

  You could find out how dumb you are in terms of physics, how little, how little we humans understand.

• 2:45:17Max Tegmark?

  I'm cool with that. I think, I think, um.

• 2:45:21Max Tegmark?

  So I can't waffle my way out of this question. It's a fair one. It was tough. I think, given that I'm a really, really curious person, that's really the defining part of who I am. I'm so curious.

• 2:45:41Max Tegmark?

  I have some physics questions.

• 2:45:45Max Tegmark?

  I love, I love to understand. I have some questions about consciousness, about the nature of reality, or just really, really love to understand also. I could tell you one, for example, that I've been obsessing about a lot recently.

• 2:46:01Max Tegmark?

  So I believe that. So suppose Tenoni is right. Suppose there are some information processing systems that are conscious and some that are not. Suppose you can even make reasonably smart things like GPT-4 that are not conscious, but you can also make them conscious. Here's the question that keeps me awake at night.

• 2:46:22Max Tegmark?

  Is it the case that the unconscious zombie systems that are really intelligent are also really efficient? So they're really inefficient? So that when you try to make things more efficient, which will naturally be a pressure to do, they become conscious? I'm kind of hoping that that's correct and I, do you want me to give you, hand away the argument for it? You know, like.

• 2:46:47Max Tegmark?

  In my lab, again, every time we look at how these large language models do something, we see that they do it in really dumb ways and you could make it better. If you, we have loops in our computer language for a reason. The code would get way, way longer if you weren't allowed to use them. It's more efficient to have the loops. And in order to have self-reflection, whether it's conscious or not, even an operating system knows things about itself, right? You need to have loops already. So I think, I'm waving my hands a lot, but I suspect that the most efficient way of

• 2:47:31Max Tegmark?

  implementing a given level of intelligence has loops in it, self-reflection, and will be conscious. Isn't that great news? Yes, if it's true, it's wonderful, because then we don't have to fear the ultimate zombie apocalypse. And I think if you look at our brains, actually, our brains are part zombie and part conscious. When I open my eyes, I immediately take all these pixels that hit on my retina, right? And I'm like, oh, that's Lex. But I have no frigging clue of how I did that computation. It's actually quite complicated, right?

• 2:48:12Max Tegmark?

  It was only relatively recently we could even do it well with machines, right? You get a bunch of information processing happening in my retina, and then it goes to the lateral ginecous nucleus, my thalamus, and the area V1, V2, V4, and the fusiform face area here that Nancy Kenworthy at MIT invented, and blah, blah, blah, blah, blah. And I have no frigging clue how that worked, right? It feels to me, subjectively, like my conscious module just got a little email saying...

• 2:48:43Max Tegmark?

  Face facial processing uh fit task complete its legs.

• 2:48:50Lex Fridman?

  Yeah.

• 2:48:51Max Tegmark?

  And I am gonna just go with that, right? So this fits perfectly with Tenoni's model because this was all one way information processing mainly.

• 2:49:03Max Tegmark?

  And uh it turned out for that particular task, that's all you needed. And it probably was kind of the most efficient way to do it. But there were a lot of other things that we associated with higher intelligence and planning and so on and so forth. Where you kind of wanna have loops and be able to ruminate and self reflect and introspect and so on. Where my hunch is that if you wanna fake that with a zombie system that just all goes one way. You have to like unroll those loops and it gets really, really long and it's much more inefficient. So I'm actually hopeful that AI, if in the future, we have all these very sublime and interesting machines that do cool things.

• 2:49:41Max Tegmark?

  And our line was that they will be at least, they will also have consciousness for the kind of these things that we do.

• 2:49:49Lex Fridman?

  That great intelligence is also correlated to great consciousness or a deep kind of consciousness.

• 2:49:56Max Tegmark?

  Yes, so that's a happy thought for me because the zombie apocalypse really is my worst nightmare of all. It'd be like adding insult to injury, not only did we get replaced, but we frigging replaced ourselves by zombies. Like, how dumb can we be?

• 2:50:12Lex Fridman?

  That's such a beautiful vision and that's actually a provable one. That's one that we humans can intuit and prove that those two things are correlated. As we start to understand what it means to be intelligent and what it means to be conscious, which these systems, early AGI like systems will help us understand.

• 2:50:30Max Tegmark?

  And I just wanna say one more thing, this is super important. Most of my colleagues, when I started going on about consciousness, tell me that it's all bullshit and I should stop talking about it.

• 2:50:40Max Tegmark?

  I hear a little inner voice from my father and from my mom saying, keep talking about it because I think they're wrong. And the main way to convince people like that,

• 2:50:53Max Tegmark?

  that they're wrong, if they say that consciousness is just equal to intelligence, is to ask them, what's wrong with torture? Why are you against torture?

• 2:51:02Max Tegmark?

  If it's just about, you know,

• 2:51:05Max Tegmark?

  these particles are moving this way around and that way, and there is no such thing as subjective experience, what's wrong with torture? I mean, do you have a good comeback to that?

• 2:51:14Lex Fridman?

  No, it seems like suffering, suffering imposed onto other humans is somehow

• 2:51:21Lex Fridman?

  deeply wrong in a way that intelligence doesn't quite explain.

• 2:51:24Max Tegmark?

  And if someone tells me, well, you know, it's just an illusion, consciousness, whatever, you know,

• 2:51:34Max Tegmark?

  I would like to invite them the next time they're having surgery to do it without anesthesia.

• 2:51:40Max Tegmark?

  What is anesthesia really doing? If you have it, you can have it local anesthesia when you're awake. I had that when they fixed my shoulder. It was super entertaining.

• 2:51:49Max Tegmark?

  What was that it did? It just removed my subjective experience of pain. It didn't change anything about what was actually happening in my shoulder, right?

• 2:51:58Max Tegmark?

  So if someone says that's all bullshit, skip the anesthesia, is my advice. This is incredibly central.

• 2:52:05Lex Fridman?

  It could be fundamental to whatever this thing we have going on here.

• 2:52:09Max Tegmark?

  It is fundamental because we're, what we feel is so fundamental is suffering and joy and pleasure and meaning and

• 2:52:21Max Tegmark?

  that's all, those are all subjective experiences there. And let's not, those are the elephant in the room. That's what makes life worth living and that's what can make it horrible if it's just there was just suffering. So let's not make the mistake of saying that that's all bullshit.

• 2:52:36Lex Fridman?

  And let's not make the mistake of not instilling the AI systems with that same thing that makes us special.

• 2:52:46Max Tegmark?

  Yeah.

• 2:52:47Lex Fridman?

  Max, it's a huge honor that you will sit down to me the first time on the first episode of this podcast. It's a huge honor you sit down with me again and talk about this, what I think is the most important topic, the most important problem that we humans have to face and hopefully solve.

• 2:53:06Max Tegmark?

  Yeah, well, the honor is all mine and I'm so grateful to you for making more people aware of the fact that humanity has reached the most important fork in the road ever in its history and that's turning in the correct direction.

• 2:53:21Lex Fridman?

  Thanks for listening to this conversation with Max Tegmark. To support this podcast, please check out our sponsors in the description. And now, let me leave you with some words from Frank Herbert. History is a constant race between invention and catastrophe. Thank you for listening and hope to see you next time.