Every Exponential Ends — Silicon Valley Forgot — Adam Becker

Astrophysicist Adam Becker, author of "What Is Real?", joins Tim Scarfe to take apart the futures Silicon Valley keeps selling: the 2045 singularity, mind uploading, Mars colonies, and the AI apocalypse. His new book *More Everything Forever* argues these ideas are hugely influential, mostly evidence-free, and bankrolled by tech billionaires who need a story in which growth never ends. Becker does the physics the boosters skip. Kurzweil's "law of accelerating returns" rests on cherry-picked data, and every exponential ends. Grant Bezos his perpetual energy growth and humanity boils the oceans within a few centuries, then exhausts the observable universe in under 4,000 years. The stars are too far away, Mars dirt is poison, and the day the dinosaur-killing asteroid hit Earth was still nicer than any day on Mars. On AI, Becker calls LLMs pocket calculators for language: hallucination is the model doing exactly what it always does, and the intelligence explosion assumes intelligence is a single number you can buy with compute. The sting is that Becker thinks the doomers are sincere. Yudkowsky, Bostrom and the effective altruists are not grifters, he says, just wrong, and their warnings that AI could end the world feed the same growth story the money depends on. He closes with his own prescription: take social problems seriously, regulate the whole tech industry, and tax billionaires out of existence.---TIMESTAMPS:00:00:00 Cold open and the thesis of More Everything Forever00:04:24 Kurzweil's singularity and the physical limits of exponential growth00:14:02 High agency and the fantasy of imprinting humanity on the cosmos00:16:55 Mind uploading, functionalism, and embodied cognition00:24:24 AI psychosis and anthropomorphizing LLMs00:26:24 Calculators, hallucination, and the limits of scale00:32:20 Yudkowsky and the intelligence-explosion argument00:40:37 True believers, venture capital, and the sci-fi growth narrative00:47:21 From Extropians to EA: utilitarianism and longtermism00:53:50 Brain worms and Becker's prescription: take social science seriously00:56:49 Why the AI-ethics discourse is broken01:01:42 The eugenics and IQ argument against 'intelligence'01:06:07 Why space settlement fails: Mars, the moon, and orbital data centers01:10:42 Billionaire myths and the search for purpose01:13:38 Tax billionaires, regulate tech: closing prescriptions---REFERENCES:book:[00:00:07] More Everything Forever (Adam Becker, 2025)https://www.hachettebookgroup.com/titles/adam-becker/more-everything-forever/9781541619593/[00:00:15] What Is Real? (Adam Becker, 2018)https://en.wikipedia.org/wiki/What_Is_Real%3F[00:15:46] What We Owe the Future (Will MacAskill, 2022)https://www.hachettebookgroup.com/titles/william-macaskill/what-we-owe-the-future/9781541618626/other:[00:00:27] Dreaming Against the Machine (podcast)https://www.dreamingagainstthemachine.com[00:01:04] The Useful Idiots of AI Doomsaying (Adam Becker, The Atlantic, 2025)https://www.theatlantic.com/books/archive/2025/09/what-ais-doomers-and-utopians-have-in-common/684270/ RESCRIPT: https://app.rescript.info/share/d6e37f9866673d8f74a39076efa5926b

• 0:01Speaker 00

  This episode is brought to you by Michelob Ultra. Good times mean competitions between you and your friends. Like seeing who can throw some pigskin the farthest or rush the field the fastest. Winner gets an ultra because it's not just a superior like beer. It's also giving fans superior access to their favorite sporting events. Superior is worth playing for. Visit michelobultra.com/superioraccess to enter. Enjoy responsibly. Copyright 2026 Anheuser Busch InBev Michelob Ultra Beer, St. Louis, Missouri.

• 0:30Speaker 01

  This episode is brought to you by Subaru. You know that place your mind always wanders to? The lake, the mountains, the homes of the people you love most? The completely redesigned 2026 Subaru Outback is built to help you get there. With legendary capabilities, standard symmetrical all wheel drive and the confidence to go further. The 2026 Subaru Outback. Love never looked so good. Visit Subaru.com slash Outback to learn more.

• 0:56Speaker 02

  Hi, I'm Adam Becker. Uh, I'm an astrophysicist and journalist, and I'm the author of two books. Most recently, I've written more everything forever, which is about the horrible ideas that tech billionaires have about the future and why they don't work. And my first book was called What Is Real? And it's about the sordid untold history of quantum physics. I'm also the host of the podcast, Dreaming Against the Machine, where we try to think about what a better future could look like outside of the grip of the tech industry. And you should listen to this episode of Machine Learning Street Talk because Tim and I have a great conversation about why some of the most powerful people in the world are wrong about so many things.

• 1:38Speaker 03

  By the way, many folks in my Discord really loved your previous book, by the way, so quite a few people said that.

• 1:43Speaker 02

  Well, that makes me really happy. I, I mean, you know, I hope I'll probably write, you know, more books in the future, but the first book's always special.

• 1:51Speaker 03

  Awesome. Awesome. Right, here we go. Um, Adam Becker, it's a, it's a huge honor to have you on MLST. Welcome.

• 1:58Speaker 02

  Thanks for having me, Tim. It's a pleasure to be here.

• 2:00Speaker 03

  So I, I found out about you when I read an article on the Atlantic that you wrote last year, and I think it was called the, the, um, the useful idiots of doomsaying.

• 2:11Speaker 02

  Yeah, yeah. Yeah, the useful idiots of AI doomsaying. Yeah, exactly.

• 2:15Speaker 03

  Yeah. Tell me about that. Well, maybe, maybe before, before we get there. So after reading that article, I, I immediately reached out to you because I, I thought it was great. And I discovered that you'd written this book, uh, more everything forever. And in a minute, actually, I'm going to read a paragraph from page 28 because I think that sums up the book quite nicely. But, um, I, I should say at the beginning that, that, you know, we, we have a very technical audience and I've interviewed many, many folks from the EA community and the rationalist community and so on. So we try to be multidisciplinary, but I think this is actually a great opportunity for you as, as a physicist to litigate some of these ideas and, and kind of go from first principles. So anyway, tell, tell me about your book.

• 2:54Speaker 02

  Sure. So, I mean, you, you did just sum it up in a way in that the book is me litigating a bunch of these things from first principles. Um, I've lived in the Bay Area now for almost 15 years and I have been to a lot of parties out here where I've heard a lot of people say a lot of bizarre things about the future of technology, AI, you know, space colonization, that kind of thing. And I always just kind of rolled my eyes, but as time went on, I realized, wait, these ideas are actually really influential and yet there's no good evidence for them. And in most cases, a lot of good evidence against them.

• 3:33Speaker 02

  Uh, and so the book is my attempt to lay out this set of beliefs that are surprisingly influential within the tech industry, um, and are promoted both by tech billionaires and by subcultures that are funded by tech billionaires like rationalism, effective altruism, uh, and go after these ideas, uh, both in a, in a scientific and philosophical way, ideas like the singularity, uh, you know, an AGI apocalypse, uh, space colonization, uh, you name it.

• 4:09Speaker 03

  In, in a way, we live in crazy times, right?

• 4:12Speaker 02

  Yes.

• 4:12Speaker 03

  Large language models are now getting better all the time. And, you know, you could almost be forgiven for having this kind of techno utopian view of what's to come. But I wanted just to read a tiny bit from your book because I think this sums it up quite nicely. So you, you said that, um, the technologists, they make all of these problems into problems about technology. All the ills of the world will be solved when the singularity arrives or when super intelligent AI solves them for us or when we go into space. Global warming can be solved with nanotechnology, illness and death. All the other problems that come with having a body can be solved by transferring your mind into a computer. Social problems and political problems, like the problems created by tech companies themselves,

• 4:53Speaker 03

  are dismissed as irrelevant or unimportant when compared to the more urgent problems like avoiding the creation of an improperly aligned, um, AI or the plight of a hypothetical unborn, uh, quadrillions of humans that could live on other side, on the other side of the cosmos a billion years from now. It's a philosophy made by carpenters insisting the entire world is a nail that will yield to their ministrations.

• 5:18Speaker 02

  Yeah, that's about right.

• 5:20Speaker 03

  That's about it. So, so where, where do we start with this story, Adam?

• 5:25Speaker 02

  Uh, I mean, I always feel like the right place to start is this idea of the singularity because I feel like that's where all of this sort of comes from is this idea that there is this future coming, uh, near as Ray Kurzweil says, or nearer now. And, uh, and that, you know, super intelligent AI and the rate of acceleration in the advance of technology is just going to keep going faster and faster until we have unimaginably advanced technology that solves every single problem.

• 6:05Speaker 02

  And, uh, you know, Kurzweil is best known as the sort of, um, evangelist for this idea. Uh, but his evidence for it is really bad and, and the evidence for it in general is just pretty terrible. Uh, the, it's not really an idea that makes much sense.

• 6:24Speaker 03

  Well, so how does Kurzweil make the argument? I mean, I mean, in, in your, in your book, you told the story of how his father died of a heart attack when he was quite young. So, um, there seems to be a similar theme here that, that a lot of people, they, um, start to fear dying either individually or collectively. It becomes an obsession. And he had this idea that, you know, maybe one day he could upload his mind into a computer and, and live forever.

• 6:50Speaker 02

  Yeah, yeah, that's exactly right. And I do think a lot of it is motivated by fear of death, which, you know, yeah, I don't want to die either, especially not anytime soon, but there's a difference between, you know, a healthy fear of death and letting it sort of run your life. But, um, yeah, I mean, Kurzweil basically takes Moore's law, right? This exponential increase in the number of transistors that you can cram into the same area on a silicon chip, uh, and generalizes it and says, oh, you know, this is not just a technological phenomenon that lasted for about 50 years from the late 20, 20th into the early 21st century.

• 7:31Speaker 02

  Instead, this is a general trend in technology, something he calls the law of accelerating returns and not just technology, but actually biology as well. Uh, so he traces it back at least as far as the start of life on earth and, and in some places, I think even claims to trace it all the way back to the big bang with the organization of inorganic systems as well. Um, he just thinks that there is a trend toward greater complexity and intelligence that runs through absolutely everything and that is pointing to a time in the very near future. He has repeatedly said the year 2045, uh, is when the singularity arrives and our technology

• 8:16Speaker 02

  becomes unimaginably advanced. And the evidence he marshals for this is pretty weak. He, he says, you know, you can pick out particular points of interest in the history of life on earth and the history of human technology. And when you plot them on a chart, you get, you know, a, a, a straight line on a semi log chart. And so that means that you've got, uh, an exponential trend. But the fact is that, you know, those points are cherry picked and he's got, uh, a problem that I think, uh, most of us have when we look at history, you know, unless you work really hard, history looks sort of logarithmic, right?

• 8:59Speaker 02

  You've got this, uh, clearer view of what's been happening in the recent past. And then the more distant past is more distant and so you can't see it as well. And so you end up with this sort of, um,

• 9:16Speaker 02

  You know, a logarithmic view of the past, that is something that I think Kurzweil is mistaking for a true exponential trend.

• 9:26Speaker 03

  Yes, and I think you gave an example in your book that, you know, you could look at, maybe this was Kurzweil's example, you can look at lily pads and then they, they grow exponentially and then they cover the size of the pond and then, and then that's it. But more broadly, though, there are limits.

• 9:39Speaker 02

  Yeah, exactly.

• 9:40Speaker 03

  Me. Yeah. What, what interests me is that we spoke into Chomsky about this and, and he said after Newton exercised the ghost but left the machine intact, you know, the, the whole enterprise of science stopped being about trying to understand how the universe actually worked. And it was more about making sense of the universe. I mean, David Krakauer said to us that science is like poetry, you know, allows us to give the, the universe meaning and, and make it make sense to us. But reality is protein, isn't it? So what Ray Kurzweil did was he selected a bunch of things that were relevant to humans. So he didn't select, you know, things that might have been more globally important to our success. And, and then he, he kind of fitted it on a graph. And, and this isn't just, this

• 10:22Speaker 03

  is a very natural thing, right? The world is complicated. And to make it make sense, we kind of select things that privilege the way we think about the world.

• 10:30Speaker 02

  Yeah, absolutely. No, it's a very understandable mistake. In the book, I, I will, you know, make the analogy to looking out from the top of a skyscraper, right? You see the stuff closer to you more clearly. And the stuff that's further away and less directly impacting you is, is, you know, smaller and, and farther away and harder to see. But yeah, I mean, it's, it's a natural mistake. And, and also like many other people, and, and as you were alluding to with the lily pad example that I was talking about in the book that I pulled from Kurzweil, he forgets that the one thing that you can always say about any exponential trend, the one thing that's always true is that they

• 11:11Speaker 02

  end. Uh, you know, yeah, he, he has the lily pad example to give an idea of, of how exponential growth works. He says, you know, the, if, if the lily pads double every day, and on day 29, they cover one half of the pond. Well, then on day 30, the pond is full. And like, yeah. And then on day 31, the lily pads don't grow anymore because they filled the pond. There's no more pond. And even Gordon Moore himself said, oh yeah, Moore's law isn't going to last forever. Moore's law has to stop sometime in the 2020s because eventually you get down to the size of individual silicon atoms and you can't really make transistors out of silicon that are significantly smaller than silicon atoms.

• 11:57Speaker 03

  But I suppose it is a bit of a straw man, just, just to make the sigmoid case. So what they would say is that the actual exponential curve is the stacking of sigmoids. And in technology, whether it's Betamax, or whether it's film processing technology, what tends to happen is you get these disruptions. So you find a divergent stepping stone and usually a different people, you know, different group of people situated somewhere else, they find a new way of doing things and it just keeps going.

• 12:21Speaker 02

  Yeah, yeah, yeah. No, that's exactly what Kurzweil says. He says, you know, the exponential trend is itself made by a bunch of sigmoid curves. And each curve lets you get further up the overall exponential trend. Sure, yes, he does say that. But as I say in the book, there is absolutely no reason why that is something that's always going to happen. And indeed, his own sort of generalization of Moore's law, when he puts that together, it ignores much of the early history of computing where stuff like that did not happen. If you go back to, you know, analog computation devices from a thousand, 2000 years ago, they don't fit on his trend. And there's no technology coming up that looks like it's

• 13:04Speaker 02

  going to save Moore's law in the future. And that's just one trend. Kurzweil wants to say that those sigmoids save you and get you a continuing exponential trend in every single area of, you know, human technology and human growth. And that's, I'm sorry, just wish casting.

• 13:22Speaker 03

  Yes, but there does seem to be an obsession with infinity and keeping going. I mean, you gave another wonderful example of Jeff Bezos. And apparently, he said he was very, very scared of stasis. So, so for him, death is the lack of growth, right?

• 13:36Speaker 02

  Yeah.

• 13:37Speaker 03

  And you, you know, there's a wonderful bit in your book where, where you said, you know, okay, so he's saying like, we, we need to keep using more and more energy every single year. And, and you said, yeah, you know, maybe we, we can extrapolate this and possibly for another 3,700 years, we can still use more energy. But if you think about it, that's actually less time than when the, the great Pyramid of Giza was created. And, and at some point, it must end.

• 13:58Speaker 02

  Yeah, that's exactly right. Yeah. Because, you know, Bezos says, we have to get off Earth because at some point in the next 300 years, if we keep using energy and keep growing our energy usage at the same rate that we're currently at, eventually we'll be using all of the energy that the Earth gets from the sun. And that's true. It's three or 400 years, somewhere in there. He doesn't mention that also we'd be at that point, generating so much waste heat that we'd be boiling the oceans. But, you know, whatever. And so then he says, so therefore we have to go into space in order to be able to keep growing our energy use. But if you go into space and you even spot Bezos a faster

• 14:38Speaker 02

  than light drive for free, if you keep that exponential growth in energy use going, eventually, as, as you said, you know, in, in less than 4,000 years, you're using all of the energy available in the observable universe. And that's just not going to happen.

• 14:57Speaker 03

  Yeah, it's so interesting. And, and this seems to be an example of folks that want to maximize human agency. Maybe we'll get to the AI agency in a little bit. But it's, it's almost, you know, like Sam Altman, the other day, he said, well, to succeed in the modern world, you need to be high agency. This seems to be the new term that's used in Silicon Valley. And, and by the way, it's great that I'm talking to a physicist here because we love talking about the philosophy of agency. But, but very, very, in a very basic term, you know, I think of it as the, the ability to you, you know, the, the extent to which you are the cause of your own actions. So that the causal origination and possibly the extent to which you can control the future. So causal efficacy. But I'm talking to a physicist now. And, and you know that we're not really the cause of our actions. Maybe

• 15:41Speaker 03

  we're a causal conduit. So they, they imagine a world where we could collectively, basically imprint our will on the universe. We could escape our kind of causal, you know, the causal clutches of, of our embedding and we can just go on forever and reach to the stars. What's wrong with that?

• 15:59Speaker 02

  So many things. I mean, you can get into questions about like free will and consciousness, and I'm not going to do that. What I will say is, you know, I, I, I actually, the thing about that, that I want to focus on, is the same thing I focus on in, in the relevant part of my book. They just don't understand that we are not going to be going out and imprinting ourselves into the cosmos at large. That is not the future of humanity. You know, there's a part of, there's this book by Will McCaskill, this, this effective altruist,

• 16:41Speaker 02

  right? What We Are the Future. And if you take a look, he's got this afterword for it. That's on a website that he runs, but he's got a QR code for it at the end of his book. And he talks about how, you know, a good future for humanity would involve colonizing the observable universe. And, you know, sending ships off to, you know, all the galaxies that we can reach in our universe and, and like intergalactic colonization. And that is just not going to happen. Like, we're not leaving the solar system. The, the stars are simply too far away. It's too hard to get there. And it is too difficult to get

• 17:25Speaker 02

  ourselves up to any reasonable fraction of the speed of light. And even if we did, the number of planets that are remotely like Earth appears to be pretty low. I, I mean, it's just, space is such an inhospitable place. We have evolved to be suited for this planet. And there are not other places like it that are waiting for us. This is our only home.

• 17:51Speaker 03

  Yeah. I mean, I was inspired by James Lovelock's Gaia theory, which was, you know, kind of stimulating people to think about the Earth as a system rather than us as individuals.

• 18:00Speaker 02

  Yeah.

• 18:00Speaker 03

  There's a bit of a spectrum here, right? Because some of them, perhaps in this case, they think that us as individual bodies, we can kind of disconnect ourselves from the system and go somewhere else. But it goes even further than that in Silicon Valley. They, some of them think of us as causal patterns or as programs, memetic programs that could be uploaded to computers and whatnot. But you, you see, you see the pattern though, that what they're doing is, is they're increasingly abstracting us and our place in the system that, that we existed.

• 18:28Speaker 02

  Absolutely. Yeah. No, they don't want to be bodies. They want to be systems of pure information. And in that sense, you know, what is it? Adrian Daub, I think, said that Silicon Valley is a place that likes to pretend that it doesn't have any history. And, and I don't even think he was the first one to say that. But, you know, there's a tendency to ignore prior art in these areas. And for, you know, this sort of dismissal of the body. I mean, this is an idea that goes back over 2,000 years, right? You, you see shades of this in Plato and stuff like that. But, but the fact is that we are our bodies. You know, you want to look at the best science available. We are not our, we

• 19:11Speaker 02

  are not patterns in our brains. We are not even our brains. We are our bodies in our environment. And there is no good reason to think that you can abstract a human consciousness out of our physical selves. You know, I almost said physical substrate, right? But that's a word and a phrase that comes from this computational analogy for cognition. And I think there's been a real kind of ignorance or amnesia or just convenient forgetting that it's an analogy, right? Cognition is not computation. Computation has some features that are like cognition.

• 19:52Speaker 02

  And the brain does things that look kind of computerish under certain circumstances. But neural networks on a computer are not the same as what our neurons do. Not even close.

• 20:03Speaker 03

  Absolutely. So we're getting into functionalism here, you know, which is this idea that essentially you can describe our function and behavior, you know, maybe to a very high resolution in a computer program. And the important move here that they make, because they're not saying that the metaphor is the same thing. They're saying that the description could be alternatively physically instantiated in silicon and that would no longer matter.

• 20:28Speaker 02

  Yes, that's right. And I don't think that there's any good reason to believe that this is true. Right? I mean, the brain is fabulously complex and the actions of an individual neuron are quite a bit more complicated than a single neural, than a single node in a neural network in a computer. Um, and, and we do not have a good understanding of how it is that the brain does what it does. And, and we don't have a good understanding of how it is that the brain is connected to the body and what parts of the body are important for our experience and consciousness.

• 21:11Speaker 02

  I mean, it's, it's very much putting the cart before the horse. We would need to really have a much better understanding of what's going on up here and in here, um, before we could hope to, uh, reproduce it with enough fidelity in a computer. And I would also argue that there's no good argument that computers, as we currently have them, are going to be capable of that kind of thing in the right kind of way, even if you subscribe to a kind of functionalism, which I'm not sure is correct.

• 21:42Speaker 03

  Yeah, I think it also gives a good insight into the, the model that, that the folks have in Silicon Valley, because it comes back to this agency thing again. So, you know, you could argue that agency requires, you know, like the good regulator theorem, it, the brain is not just a, a complete model of the world. You could argue that it's a kind of input, output processing system. So it's, it doesn't really work if you, if you take it out of the environment, because you know, a thought experiment is I take a copy of you and I put you on Mars. Are you the same person? And they'll get into this identity problem in, in philosophy. But, um, the real problem though, is that, well, maybe the way that you think is literally it, you know, you, maybe you're like an LLM and the people you talk to, the things you look at on Twitter, all of this forms you.

• 22:25Speaker 03

  And if you take you out of the environment, maybe it's not you anymore.

• 22:28Speaker 02

  Yeah. I mean, I think that, that our environment is an important part of who we are. I think that one of the moves that I see over and over again from these guys, uh, is this idea that intelligence is an individual trait as opposed to something collective and social. Um, and I don't think that that's right. Uh, and I also, you know, for what it's worth, don't think that we're actually very much like LLMs. Um, you know, I, I think that if nothing else, the fact that an LLM takes in more

• 23:09Speaker 02

  text than any human could in a human lifespan in order to be worse at language in many ways than a five-year-old. Uh, and five-year-olds take in far, far less and do far more with it. Um, that right there is a good indication that, uh, we're pretty different.

• 23:30Speaker 03

  I know, but, but I should push back a little bit because I don't know whether you played with Mythos before it got taken offline, but you know, even though we can criticize the functionalism thing and we can talk about Chomsky's competence versus performance, you know, that there's a difference. But what, what we do see though, is that these models can do things that we do extremely well, and they seem to be getting better all the time that, you know, even though they do a different type of syntactic processing than we do, arguably it's in some ways better than, than we do. So there's something is, is, is kind of validating their functionalist hypothesis.

• 24:04Speaker 02

  Yeah. I mean, there's definitely something going on there. Right? Um, and then the question is, okay, what is it? What's the significance of that? I would argue that we're learning some interesting things about how language works and what you can get out of the sort of mathematical structure and statistical properties of language. Um, but you know, uh, I don't remember who said this, but, um, people, people speculate about whether, you know, uh, LLMs are alive, but you don't speculate

• 24:46Speaker 02

  are alive, um, you know, conscious, but you don't hear people speculating about whether, um, systems with similar architectures that don't work with language are conscious. So nobody's asking if like alpha fold is conscious. Um, and, and I think that says a lot about how we as humans see systems that process and use language, um, or, or emit language, as it were.

• 25:20Speaker 03

  Yeah. I mean, without spending too long on consciousness, because we can get stuck.

• 25:24Speaker 02

  Yeah, yeah, yeah, we could get stuck there all day. That's right.

• 25:26Speaker 03

  You're kind of talking a little bit about what I would call AI psychosis. So there was that, um, you know, that there was that Google engineer famously with, with the Lambda model, was it a couple, a few years ago? And so, so yeah, we anthropomorphize these models, just like the Eliza system many, many years ago.

• 25:41Speaker 02

  Yes, indeed.

• 25:42Speaker 03

  Yeah. Yeah. I mean, there are folks that seriously are, you know, taking seriously the idea that these models might have some kind of phenomenal experience rather than just simulating. I mean, and, and the AI psychosis goes far broader than this, but this, this is concerning to me. What do you think about that?

• 25:58Speaker 02

  I mean, the AI psychosis stuff is really, really terrifying to me. Um, because I really do think that this is, um, this is taking that sort of paradoilia word that I'm almost sure I'm pronouncing incorrectly. Um, but, uh, it's, it's taking this human tendency to see patterns, especially like human patterns where there aren't any, like seeing a face where in random noise and stuff like that. It's taking that and, and almost exploiting it, right? Um, uh, because we're, we're not only attributing agency to these things, but in the case of AI psychosis, I mean, I don't know exactly what's going on there, but it certainly seems

• 26:41Speaker 02

  like it's people attributing agency to these things and then getting caught in a kind of feedback loop where these systems are sort of regurgitating themselves, like regurgitating the user back to themselves. Right? Um, because, you know, that's, that's what they do in a way. Uh, what Shannon Valour, uh, talks about, uh, the idea of AI as a mirror. And I think that that's a really good analogy. It sort of reflects ourselves back to us. And in this case, I think it's happening in a really, um, unhealthy way.

• 27:20Speaker 03

  I, I agree with that. The, the thing is though, using this technology every day, it's getting better. Now you can have a memory system. So it just, every time it does something wrong, you say, no, you know, you should have done that. And over time, you're, you're building a kind of simulacrum of yourself and it increasingly does the right thing. And, you know, we can talk about the intelligence thing. I mean, yeah, it is a, it is a grotesque metaphor that you can take something, which I believe is, is a physical property of, of stuff in the universe. And we can create an abstraction that, that seems to work reasonably well for abstract domains like playing chess and, and so on. And, you know, these, these models are becoming quite adaptive and they are, they have a lot of capabilities and so on. And you see, it's just so deceptive.

• 28:03Speaker 03

  It might be the most deceptive time in human history because it seems like it is intelligent. And, and it is actually automating meaningfully human labor and lots of jobs and stuff like that. So how do we make sense of this?

• 28:16Speaker 02

  Yeah. I mean, it's a good question. I, I, I don't want to diminish what LLMs are capable of, but I keep thinking about calculators. Um, you know, calculators took something that, uh, and I mean like pocket calculators, right? They took something that, that like, we thought of as this inherent human activity of, of, you know, compute, like that kind of computation, like four-function mathematics, um, and automated it. And it used to be that, that to be a good mathematician, uh, or a good physical scientist, you had to be good at doing that kind of work with pencil and paper or in your head.

• 29:00Speaker 02

  And then that stopped being true. And in some ways, you know, that was a loss of certain things, but on, on the other hand, it allowed for a kind of mathematical research that could not have been done before. Right? Uh, you know, you'll hear sometimes mathematicians say things like, um, computers are like telescopes for mathematics. Uh, and, you know, um, but, but.

• 29:34Speaker 02

  Again, I mean, not to repeat myself, but I do think that the difference here is that it's language, not math.

• 29:44Speaker 02

  And so that makes it feel like something is is sort of thinking and conscious and talking to us. And, you know, again, that's not to diminish it, you know, I'm not diminishing the the functionality of calculators either. But it is quite deceptive and and I think I, in terms of how to make sense of it, I think we just have to keep in mind what these systems are at the end of the day, they are for predicting the next word or whatever in the sequence that they've been given. And and it turns out you can get pretty far doing that.

• 30:27Speaker 03

  You can, and I would push back a little bit on the stochastic parrot thing, even though that's technically true. They they, I mean, internally, they are they are acquiring during their training process, some kind of coarse grained abstraction, some kind of structure.

• 30:42Speaker 02

  Oh, yeah.

• 30:42Speaker 03

  Which allows them to, you know, extrapolate, generalize, call it what you want. So, you know, it's different to us, but there's something there. But I think the $2 million questions are, at the moment, it needs human supervision. And these folks say, well, we just scale is all you need. Yeah, at the moment, it needs human supervision. And at the moment, it just generates loads of spaghetti garbage code. And, you know, it just creates, basically, it creates more problems than it solves. But it's deceptive because most people can't see the problems. But all we need to do is keep scaling it, you know, when when GPT seven comes out, it will actually refactor all that code. Maybe we can RL train it to refactor it. And we just have to kind of keep, and it's very dangerous keeping going because now we're messing all of our code bases up and we're creating all of this slop everywhere. But just just hold on, boys, just wait for a couple of years and the next version will

• 31:27Speaker 03

  kind of bring it back in check. What do you think about that?

• 31:29Speaker 02

  I just don't think that that's true. I don't think that there's, I mean, I could be wrong, obviously, but I don't see good evidence for that, right? I mean, I think that these systems are, you know, unless there's some sort of fundamental breakthrough, right, something more than just scale. These systems are always going to require human supervision because they are always going to end up hallucinating. You know, I mean, that's inherent to the way that they work. I say this in the book, but, you know, I don't love the word hallucinate. I know there's been a lot of pushback because it's like, oh, hallucination implies a sort of anthropomorphization of these systems. And I don't love that either, but that's actually not my main problem. My main problem with the word hallucination is it implies that when a hallucination

• 32:14Speaker 02

  occurs, something different is happening than its normal functioning. And that's not the case. They really only do one thing. And when they're hallucinating, they're doing the same thing that they're doing when they get it right. And so I think that unless we have some sort of major, major breakthrough, and I mean, it would probably have to be a breakthrough that makes, you know,

• 32:39Speaker 02

  LLMs themselves look like Eliza. You know, short of that kind of really fundamental breakthrough, I don't see us getting around the need for human supervision on these things. So, and if anything, as they get better, it's going to get harder to discern when they've made these mistakes, even though they're going to keep making them. And that's quite dangerous, as you said.

• 33:01Speaker 03

  I know, and ironically, people aren't really talking much about the hallucination now because they're agentic and they can fix their own stuff, you know? It's more like, you know, if you analogize it like a database query or a program interpreter, it's only as good as the program or the database query.

• 33:16Speaker 02

  Yeah.

• 33:16Speaker 03

  So, you know, it will basically do what you tell it to do. And this is, this is the gap, right? Because if these things could be alive, if they did have agency, what would, what would that mean? Let's wire it up in a loop and let's show it a load of video frames in a sequence and we'll give it a basic prompt like, do stuff, do something interesting. What will happen? Basically, nothing, nothing interesting will happen, right? So the more you understand a domain and you put a very specific program in there, you can get it to do a specific thing, you can get it to hill climb towards a specific goal to solve a specific problem. But the framing always comes from us. So it's kind of doubtful whether we would overcome that. Maybe we will, but it's kind of doubtful. But, but, but then there's this thing, which is the first or second step fallacy.

• 33:59Speaker 03

  Right, which is, as you said, Eliezer Yudkowsky, he said that we're only one or two steps away from inventing AGI and it's going to run away.

• 34:07Speaker 02

  Yeah.

• 34:08Speaker 03

  Why does he think that?

• 34:10Speaker 02

  I mean, first of all, he didn't say one or two, he actually said zero to two. He's not sure that we need any.

• 34:18Speaker 02

  But, um, but yeah, I mean, why did he say that? I mean, you should ask him, but, um, but he, he, he believes in something like a singularity. He believes that if, um, if you just throw enough computing power at a, at a machine learning system of the right type, then it will, you know, become conscious, wake up, whatever term you want to use. And, uh, and then become intelligent and then use that intelligence to increase its own power, which will increase its intelligence further.

• 35:02Speaker 02

  And this will create a sort of feedback loop and you'll get an intelligence explosion. And then he believes followed shortly by the end of the world.

• 35:10Speaker 03

  Can we, can we unpack this a little bit? So you were speaking there a little bit about instrumental convergence, which is this idea that, um, you know, it basically, it's, um, it's instrumental subgoals towards whatever it's doing will kind of converge on things like power seeking and bad things that we don't want. So it's always going to kill us all in almost every scenario.

• 35:29Speaker 02

  Yeah, that's what he believes. Yeah. Yeah.

• 35:32Speaker 03

  Yeah, can you, can you explain that in a bit more detail?

• 35:35Speaker 02

  Sure, yeah. So the idea there is whatever goal it is that the AI has,

• 35:47Speaker 02

  when it, you know, achieves this sort of basic level of of intelligence,

• 35:53Speaker 02

  whatever that goal is, that goal will be served better by having more intelligence. And that intelligence in Yudkowsky's worldview is something that you can increase by, you know, throwing more computing power and memory and whatnot at it. And the, the idea there is no matter what your goal is, it could be something, you know, in the, in the famous thought experiment from Bostrom, it could be something as silly as, as, you know, creating more paper clips. Uh, or it could be, you know, uh, keeping people in, you know, engaged in chatbot conversations. Or it could be, you know, making as much money as you can on the stock market.

• 36:38Speaker 02

  No matter what your goal is, the, the thought with this, with this notion of instrumental convergence is

• 36:49Speaker 02

  all of those goals will lead you towards seeking more power, more intelligence, more physical and computational resources in order to better achieve those goals and prevent barriers from being placed in the way of achieving those goals.

• 37:04Speaker 02

  I don't believe that's really true, but, um, not in the way that they would need it to be in order for this argument to go through. But that's, that's the claim.

• 37:13Speaker 03

  Yeah, and it seems to be an abstraction problem again, because in, in the biological world, intelligence and agency are convolved in a very complicated way. Whereas in the, in this sense, we're now abstracting them because there's also this orthogonality thesis as well, which is intelligence and final goals are, you know, disconnected from each other. But, but, you know, more broadly, on, on intelligence, it's almost like it's this magical abstract substrate that you can, it's on a single axis and you can, you can just have more of it and more of it. And, but that's not really how intelligence works, is it?

• 37:45Speaker 02

  No, it's not, not at all. Intelligence is quite a bit more complicated than that. And it's not, you know, a single valued thing. In fact, I'm not sure that we have a great definition of intelligence. And I also think that it's very clear that there are a great number of goals that, you know, don't work this way. I mean,

• 38:09Speaker 02

  maybe, maybe it's just because it's, uh, it's, you know, like I woke up in a cynical mood this morning, but I'm not convinced that the goal of being happy is one that is served well by being more intelligent or powerful.

• 38:26Speaker 03

  Yes.

• 38:27Speaker 02

  For example, or, or to pick a somewhat less cynical example. You know, if intelligence were, were really so broadly useful for every single goal, you would see more of it in the natural world. It would be something that that would evolve, you know, more often and more broadly. Trees have goals and, um, and they get there in very different ways. And giving them a more complex nervous system or nervous system at all would not help them with those goals. It would hinder them.

• 39:02Speaker 03

  Yeah, I mean, it's also quite debatable whether things in the real world even have goals. So, you know, we always, as we were saying with science earlier, we need to disentangle how the world really works from how we understand it to work. So we take an intentional stance. And we say, okay, well, there's a complicated blob of stuff over there and I'm going to interpret its behavior as having this, this simple goal. And it might be a cartoon, right? It might not be like.

• 39:25Speaker 02

  Yeah, that's right. Yeah, no, no, no, it's not even really right for me to have said that trees necessarily have goals, but, but the point, the point remains, yeah, exactly. Like this is, seeing the world in terms of goals is itself a kind of mistake. And in some ways related, I think, to the mistake of seeing intelligence as this single valued thing that you have more or less of. Right, there's this very simplistic worldview at work here.

• 39:52Speaker 03

  Yeah, exactly. And but the thing is that you could make the argument that some folks in that community, they've moved on past the kind of the Bostromian Yudkowsky view of, of this, you know, mono maniacal, super coherent AGI. And now they're a little bit more kind of thinking, okay, it might be really illegible and it might disempower us, but it might not just be this thing that has a single goal.

• 40:14Speaker 02

  Yeah, I mean,

• 40:18Speaker 02

  huh, I, I'm not particularly concerned about that either, honestly, simply because the systems that we have and the systems that, you know, seem to be in the near future are not that kind of thing. Right, you know, I mean, and I also think that.

• 40:42Speaker 02

  We are building these systems to, you know what? No, never mind. I'm not going to continue that thought. I think that was wrong. But yeah, I mean, right. I'm a writer, right? One of the nice things about writing is you get to try out a bunch of ideas, look at them on the page and then say, okay, you know what? That one's good. That one's good. Let's get rid of that one, that one and that one. They're no good. But yeah, but instead we're doing that in real time.

• 41:08Speaker 03

  As a quick aside, thank you for not being on autopilot. I'd much rather that that thoughts come into your mind and you say them because it means that you've not said them before. That's great. That's what we want.

• 41:17Speaker 02

  Well, yeah, I mean, like, look, I could just sit here and read to you from my book, but but there's an audiobook, which you can buy wherever audiobooks are sold. Please do.

• 41:34Speaker 03

  I don't think they're being, I mean, you know, it's easy to be cynical and say, oh, you know, follow the money. And as you outline in your book, there are lots of wealthy billionaires putting lots of money into this and easy to be cynical. But I think there's this anthropic fiasco recently, you know, what it doesn't make sense, make it make sense. Why they're making all of this money? Why would they? Why would they sabotage themselves? So, you know, I don't think they're being grifters. But what is going on? Are they true believers? Like, what's happening?

• 42:03Speaker 02

  I think that a lot of these people are true believers. Yeah, like, you know, I mean, in the book, I certainly don't go easy on, say, Eliezer Yudkowsky. But one thing I will say about Yudkowsky is he's certainly a true believer. You know, he believes the stuff that he's saying. The same goes for, you know, like, people like Nick Bostrom or Toby Ord, right? And I think that that's also true of a lot of the people working at places like Anthropic. They really believe that AI alignment is the problem of our time. I just think that they've made a mistake. I think that that's not true, and that there

• 42:44Speaker 02

  are other problems that are significantly more pressing. And that, you know, as I say in my book, many of the biggest problems of our time are not problems amenable to solutions with technology. They are social problems that require social solutions, which is not to say that technology doesn't play a role. It's just that you can't solve them purely with technology. You know, climate change being the big one, the climate crisis is a problem that absolutely does require certain technologies to exist. Most of the technologies that we need exist already. And the problem is one of actually deploying them and persuading people to switch and actually like finding ways to get this stuff

• 43:28Speaker 02

  out there. That's a social and political problem that requires a social and political solution. So yeah, and then, but then you have someone like Sam Altman, who I don't know what's going on in Sam Altman's head. I don't know if he's a true believer in whatever, but he has gone out and said things, and he's not the only one who's said things like this, that, you know, AI is going to solve the climate crisis. And like, I, I think that in the unlikely event that we built, like, a really intelligent, conscious, machine intelligence, based on existing technologies in the near future. Well, if we did that and

• 44:10Speaker 02

  asked it to solve climate crisis, the minute we turn it on, it would say, well, you shouldn't have built me. Look at my carbon footprint.

• 44:17Speaker 03

  Are you implying, though, in a sense, though, because the wording of that Atlantic title was very interesting, you know, useful idiots. So in a sense, are you are you implying that they are they are genuine? But even though this is a culture that valorizes individual agency, and you're saying they actually don't have individual agency, they're being kind of parasitized by some kind of emergent, you know, force above them that perhaps they don't understand?

• 44:44Speaker 02

  I, yeah, except I wouldn't even, I would get more specific. Instead of saying an emergence force above them that they don't understand, I would say the venture capital startup ecosystem of Silicon Valley. Like, it's, yeah, you know, I mean,

• 45:00Speaker 03

  But even on that, though, it's not a conspiracy, is it?

• 45:03Speaker 02

  No, no, no, no.

• 45:04Speaker 03

  They also probably don't, you know, they probably think that they're just investing in technology and they're going to make lots of money and so on. But I feel like you're making a slightly different point, which is that there are machinations of power that lead in a certain direction.

• 45:18Speaker 02

  Yes, exactly. And I do think that, you know, for the nice things that I could say about rationalists and effective altruists, that, you know, they are generally earnest and true believers, that one of the many places they fall down, and I'm not saying this is the only place they fall down, as I, you know, this is all in my book, but they, I don't believe that they have a good understanding of power. And I think that that's what leads to this. So yeah, no, I don't think that there's a conspiracy. I don't think there has to be a conspiracy. Instead, what you have is these, these true believers running around saying like, AI is really, really important. It's coming soon. And it could

• 45:59Speaker 02

  be so powerful that it could lead to the extinction of our species if we're not really careful and don't devote a lot of resources to solving this problem. And then you also have in an adjacent place, ideologically and physically, this giant system of power and capital that needs the promise of perpetual growth in order to maintain a lot of that power and capital. And so it sees this narrative that is also a familiar narrative to the leaders of that system of power and capital, the venture capital system of Silicon Valley.

• 46:40Speaker 02

  That narrative is familiar to them because all of the people in this story, including me, and I think you, were raised on a certain set of science fiction stories, some of which are in the bookcase behind me, about, you know, what computers would be able to do and what AI is and would be able to do, what space colonization would look like, and so on and so forth. And so they see this story coming out of these subcultures, some of which were funded by other people in the tech industry. And, and they see this stuff and they, it's legible to them because they they've read the same science fiction. They say, oh, this is another story about perpetual growth. And again, I'm not saying it's a conspiracy.

• 47:26Speaker 02

  I'm not even saying that the venture capitalists aren't true believers. I imagine that a good chunk of them are, maybe even most or all, well, probably not all of them, but maybe even most of them. But, you know, there's not, there's not a conspiracy at work here, but it turns out that, and I think this was predictable, that running around saying that AI is coming and is going to be extremely powerful. And if we get it wrong, we'll all die. But if we get it right, we'll have the power of the gods and be able to expand out into the cosmos forever. That's a story that's very compatible with the desire for perpetual

• 48:07Speaker 02

  economic growth and resource usage growth that, that is so desirable to the leaders of Silicon Valley.

• 48:17Speaker 03

  Yeah. And I mean, I guess like, we could talk about, you know, the Yanis Varoufakis idea of the techno capitalist machine and neoliberalism and all of this kind of stuff. But I'm interested in the AI safety thing in particular, because you you sketched out the entire story, right, of, you know, the extropians and, you know, where Yudkowsky apparently used to be a singularitarian himself. And these were early days when, you know, they were just on online forums and just talking about this stuff. So how did the, you know, and then we've got the whole effective altruism story. How did these things meet? How did all of this get entangled up?

• 48:51Speaker 02

  Oh, God. Well, I mean, you don't owe me an autopilot here, but my standard answer to a question like that is, you know, that's very complicated and and I could write a whole book about it. But, but without just reading to you from my book, I mean, the answer is, these were groups of people who found each other through the internet, mostly, who were all spending a lot of time thinking about how do we get this future of technology that we have all come to believe is going to happen based primarily on the science fiction that we've read. And, you know, I, as a sci fi fan, I can understand why you

• 49:35Speaker 02

  might think that the future is going to contain the, you know, some or most of the elements in the science fiction that you've read, if you just, you know, put your nose to the grindstone and whatnot. But, but as a physicist, I can tell you that a lot of those things are not going to happen. But these were people who generally were not thinking about it in that way. We're just thinking, yeah, this stuff is going to happen because I can't think of a good argument that it definitely won't, which is not how the world works and not how the development of technology works. And oftentimes, there were good arguments that these things won't happen. And they were just ignorant of those arguments. But

• 50:19Speaker 02

  You know, I mean, it's it. The reason I'm talking about that rather than talking about like, okay, well, this is how the extropians and singularitarians got together and then the rationalists came out of that and blah, blah, blah, blah, blah. Is that's, you know, that's a complicated historical story and anthropological story. But the root cause of it, I think, is this sort of shared belief in a world as revealed by primarily science fiction.

• 50:46Speaker 03

  Yeah, and maybe we should talk about the utilitarian thing as well with effective altruism. So, you know, I listened to Will MacAskill. He was on Sam Harris's podcast recently. And after after kind of hearing all of this stuff about AI safety and, you know, thinking some of it was quite strange, I thought he was a really nice guy. You know, I could tell that he was genuinely trying to do good in the world and they've got so many, you know, really, really good activities going on. The thing that I don't understand is how the AI risk thing came into it. Now, you gave the example of Peter Singer, who was an inspiration, and there was that thought experiment with, you know, would you help a child, you know, in some dirty water? Is it worth more than the cost of your clothes to help that child? Why not then? And, you know, if that's a yes, why not then help

• 51:31Speaker 03

  children elsewhere? So you see, what they do is they abstract this concept of a utility function. And as we were saying before, when you start abstracting things, then you get into trouble.

• 51:41Speaker 02

  Yes. Yeah, that's exactly right. No, I mean, I think that there is something very appealing about a utilitarian approach to ethics and the world from this sort of engineering mindset and and scientific mindset, even certain kinds of scientific mindsets, definitely engineering mindset. But this idea that everything can be abstracted and quantified, and the things that you can't do that with don't really matter. That is a story, you know, a story that you can turn ethics and making difficult decisions about life into questions about numbers is always going to be an appealing story to, you know, a fairly large number of people.

• 52:25Speaker 03

  I mean, I suppose a side effect is Will MacAskill, I think you interviewed him for the book, right?

• 52:29Speaker 02

  I tried to. Will MacAskill agreed to an interview and then backed out. And my fact checker ended up talking to him, although, I don't know, he got confused, I suppose. The MacAskill, not not my fact checker. MacAskill seemed to have thought that my fact checker worked for my publisher and not for me, which I'm not sure how he made that mistake because he MacAskill worked with the same publisher and knows that publisher should know that publishers never pay for fact checkers. But in any event, no, I didn't talk with MacAskill. I read a bunch of his writing and I talked with his colleagues, but MacAskill himself did not talk to me.

• 53:11Speaker 03

  Okay, but in your book, you moved on to long termism as being one of the sort of dangerous side effects, because I think there was a quote in the book where, you know, MacAskill said that now is the thin end of the wedge. So obviously, it's a good thing. If humans all have utility, we should have many, many more humans that they should be all over the universe and even digital versions running on servers all over the universe. And the problem is, when you when you start extrapolating forwards, it means that the effective value of us compared to that big future light cone of of humans, you know, we're not very valuable anymore. So that kind of diminishes many of our current ethical concerns here on earth. Is that is that roughly right?

• 53:52Speaker 02

  Yeah, that's roughly right. Yeah. I also, I want to go back to your question of, okay, how did the AI risk stuff get into all of this effective altruist stuff? And there, the answer is, I mean, basically, Nick Bostrom read a bunch of Yudkowsky, got into the AI risk stuff from there and then started convincing people like Toby Ord and Will MacAskill that, yeah, this AI risk stuff is really important and is, you know, this overriding ethical concern. And and I will say, you know, there's even statements that some of the people in this community have made about like, okay, you know, other

• 54:34Speaker 02

  existential risks, that's sort of the gateway drug to caring about AI risk, which is like the real heart of the matter. And and I just think that's misguided.

• 54:46Speaker 03

  But is your position essentially that there are certain mindworms, for want of a better term, that are quite dangerous, right? Because they can be kind of repurposed and, you know, there can be dangerous externalities and so on. So, I mean, what's your prescription?

• 55:02Speaker 02

  My prescription is, yeah, I mean, I, well, let me let me back up and say, yeah, I do think that this is kind of a brain worm for people of a certain technical bent. Part of the reason I ended up writing the book is that I myself have a fairly technical bent and and sort of see myself in a lot of these people and feel like, okay, you know, if I were just a little bit different, or if my life had gone a bit differently, I could see myself having, you know, been one of these people. Maybe that's wrong. Maybe I don't know myself correctly or don't understand exactly the appeal of this stuff. But I feel

• 55:40Speaker 02

  like, you know, it's a pleasingly compact and tractable view of what the world is and how its most complicated problems work, which I would also argue is is where a lot of the misunderstanding of power comes from. But so my prescription, I think, would be to, I mean, I could be flip and say, read my book. But I would say that that my prescription is really to take more seriously the people who study the relevant areas of inquiry

• 56:22Speaker 02

  that that the people in these spheres tend to ignore. They tend to ignore, you know, political science, sociology, they tend to ignore anthropology, they tend to ignore, you know,

• 56:41Speaker 02

  questions about racism and bias and sexism and whatnot. They tend to, and they tend to get stuck in particular pieces of group thing, some of which get really pernicious and dangerous. Like, there is a, as I describe in my book, a surprising amount of credence within this community of effective altruists and rationalists for ideas like human biodiversity, which is a piece of junk science that tries to give a genetic basis for racism. And it

• 57:24Speaker 02

  has been roundly rejected and disproven by the scientific establishment, even though genetics itself comes from, you know, a long history of racism. And yet, within this community, it's it's an idea that people feel they have to take seriously a lot of the time, even though there's nothing serious there.

• 57:45Speaker 03

  Yeah, I wanted to touch on that because there are folks in this community that have been making these arguments. So I'm not sure where Teskariel came from. Maybe it was Timnit Gabru. But there's folks like Emily Bender and, sorry, Emily Torres as well. And to be honest, the discourse has been really bad. So a lot of folks just think that they're arguing in bad faith and they just dismiss what they're saying. And that's why I feel that we've been able to have a great conversation because we're just talking about it from a technical point of view. We haven't really been, I mean, I don't know much about social science. But do you think that, you know, why is there such an impedance mismatch? Why is it so difficult to have this conversation rationally?

• 58:23Speaker 02

  Oh, man. I mean, I think that, I mean, that's a good question. I mean, it's one of the things I was hoping to try to address with my book.

• 58:37Speaker 03

  Could I give you one example? So like,

• 58:39Speaker 02

  Sure. Yeah, yeah. Give me an example.

• 58:41Speaker 03

  So, you know, I've seen presentations from Timnit, and she's very quick to go to, you know, you're racist, basically, and eugenics and all of this kind of stuff. And it's quite, it's quite heavy going. It's quite scary stuff. And, and maybe we should just start there. So is that part of the problem? Is it just because it's a little bit kind of too intense?

• 59:01Speaker 02

  Yeah, I mean, I think

• 59:06Speaker 02

  I think that the that.

• 59:12Speaker 02

  I mean, look, you, you know, I think that, that my sympathies lie significantly more with people like Timnit than with someone like Yedkowski. And, and my issue is like, I, I think that what's going on is you have people in the effective altruist rationalist community

• 59:40Speaker 02

  who, um, you know, want to, or feel that they should be able to entertain ideas like human biodiversity, um, without recognizing that actually this is an idea that has been entertained for a very long time and has a really horrifying history and has no good science behind it. Um, and then don't understand why someone gets upset at the prospect of, you know, having a dispassionate rational argument about whether or not certain people are fully human and entitled to, you know, the same rights and have the same inherent

• 1:00:23Speaker 02

  abilities. Um, and, and this is, you know, this goes back actually to that thing I was, I was attributing to Adrian Daub earlier, this sort of like looking at things outside of the context of history. There is no way to do that. History is something that we all live in. We, we, you know, as the meme says, we live in a society, right? Um, and, and I think that it is irrational to assume that someone is not going to take that history and that context into account when having a conversation on a particular subject, especially if that subject ends up being related to, you know, whether they and their friends

• 1:01:06Speaker 02

  and family are going to be treated like people. Um, so I, I, I think that that's where a lot of this comes from is there's, there's, you know, a group that feels that rationality means not taking particular pieces of context into account. And another group that's like, why wouldn't we take that into account? And, and of course, my sympathies lie much more with that second group.

• 1:01:35Speaker 03

  And, and it's funny as well, because speaking personally, I'm, I'm a bit of a weird guy in the tech community because I, I do believe in physical situatedness. I think it's about how you got there. It's, you know, you can't just abstract yourself out of the system.

• 1:01:46Speaker 02

  Yeah, yeah, yeah.

• 1:01:47Speaker 03

  And also, um, I, I think that the fundamental impedance mismatch, just as we were saying earlier, is, is folks like Timnit are talking about super agency, not individual agency. So she's not necessarily saying you're individually racist. She's saying that you are basically caught up in this larger system of, of, um, you know, power dynamics that you don't, that you're not aware of and you don't have control over.

• 1:02:12Speaker 02

  Yeah, I think that's, that's probably right. Although I don't know, I don't want to speak for Timnit, but I don't know that I would want to make, um, such a sharp distinction between individually racist versus caught up in this system that you're not paying attention to because not paying attention to it can be a form of racism. It's certainly a form of privilege, right?

• 1:02:38Speaker 03

  Yes. And, and just, just for the benefit of the audience, can you just briefly make the argument why, you know, like the eugenics and the racism thing? Like what, what's, what's the, the main argument there?

• 1:02:49Speaker 02

  I mean, there's a couple of things, right? Some of it is what I was just talking about, that this community seems to really entertain, um, hypotheses, not seems to, does entertain hypotheses like human biodiversity, um, in, in really unhealthy and unscientific ways. Um, but the other thing is this idea of intelligence, right? It's, it's the idea of measuring intelligence, intelligence as a single valued thing is itself an idea that had, that, that basically comes out of eugenics and racism. Um, you know, IQ tests, which are taken quite a bit more seriously by people like Yedkowski than they should be. Um, and

• 1:03:28Speaker 02

  are, are, you know, something that was developed essentially as a tool for eugenics. And, uh, and when you look at the modern AI industry and the field itself of inquiry, you see sort of things that are descended from that history present here today, including, you know, people using eugenicist arguments, not even from old school eugenicists back from, you know, when, uh, IQ tests were first developed 100 years ago, but from modern day, uh, purveyors

• 1:04:12Speaker 02

  of bogus race science and racism. Um, you will see people pulling, uh, language from those people, uh, when trying to define things like intelligence, um, like in academic papers and stuff like that. Um, so yeah, that is, that is sort of the, the really short argument. Um, there's a longer and more developed version of it that I give some of in my book and you can find elsewhere as well.

• 1:04:39Speaker 03

  Yeah. And I think it's ironic as well because in my opinion, current AI has proven that intelligence is, is not this thing that we thought it was. So

• 1:04:46Speaker 02

  Yeah.

• 1:04:47Speaker 03

  in some sense that the models do have intelligence and they still don't have the thing we want. And, and why is that? It's because we thought that intelligence was like, we have this core knowledge. We have these, um, these kind of abstractions in our brain and intelligence is just traversing the combinational closure. And the more intelligent you are, the faster you can do that. And what we found with these models is actually it's, it's all about how you frame the question. It's all about the, the information you have at your disposal. And there are many, many different perspectives on different phenomena. And folks with different perspectives can use their intelligence and they can get to different answers. So it's, it's very situated and it's not quite as abstract and pure as we thought it was.

• 1:05:27Speaker 02

  Yeah. No, I think that's exactly right. Um, and again, you know, I think that this comes of, I mean, in my book, I call it humanities denial, right? Um, this idea that you don't have to pay attention to things like history, sociology, politics, um, when again, we, we do live in a society.

• 1:05:49Speaker 03

  So I wanted to touch as well on, on this idea of recursive self-improvement. So this, this originated with a guy called IJ Good. Yeah. And Kurzweil certainly made this argument. So, you know, he said, well, the first AGI, uh, it's going to take us a long time to build it, but you know, but then we can use the AGI to help us build the next one. And the next one will be built in a fraction of the time and it will just keep getting better and better. And, you know, we're seeing hints of this in, in AI, like there's, there's undoubtedly, there's some kind of adaptive self-improvement, certainly when you're doing hill climbing towards a particular task. But what, what, what do you think about the, the broader kind of idea?

• 1:06:22Speaker 02

  I think that it's predicated in an idea, like two, two problematic ideas, right? Uh, both of which we've already talked about. The idea that intelligence is like a single valued thing and the idea that intelligence is something that you can equate with or is directly proportional to computational power of some kind. I don't think that either of those things are true. Um, and, and more generally, yeah, I, I think, well, no, I'll leave it at that. Yeah. I, I think that it, it, it basically comes from those two things. And the case for those two ideas is pretty weak.

• 1:07:03Speaker 03

  Yeah. Why are we not going to space?

• 1:07:06Speaker 02

  I mean, space is pretty bad. Uh, it's really easy to die in space. I make a, there's a joke that I made in my book. It's one of those things when you're writing a book, uh, or at least when I'm writing a book, I'll, I'll, I'll make jokes in the text and then some of them don't make it through my edits. And then some of them don't make it past my editor. And then I'm always surprised at which ones actually make it into the final book. And one of the jokes that's in the book is, you know, I quote that famous Elon Musk quote, uh, it'd be cool to die on Mars, just not on impact. And I say, well, you know, if we take Musk at his word, the good news for him is it's really easy to die on Mars. Um, because Mars is a really horrifying place. Uh, there's, um, you know, the radiation levels are too high. The gravity is too low. There's basically no air and the dirt is

• 1:07:51Speaker 02

  made of poison. Um, and yet Musk is right that Mars is the next most hospitable place in the solar system after the earth. The only real competition it has is the moon. And the only thing about the moon that's better than Mars is that it's really close by, which makes it much easier to get there, much easier to communicate with people who are there. Other than that, the moon is pretty much worse in every way. There's, there's no air. Uh, and, um, and although the dirt on Mars is more poisonous, the dirt on the moon is actually, it's something that became such a big problem that every Apollo mission

• 1:08:33Speaker 02

  was basically at its operational limits just because of the dust, uh, because moon dust is really sharp, abrasive and electrostatically charged. So it just sticks to everything. Um, and, you know, I, I mean, the, the fact is that

• 1:08:53Speaker 02

  There, there's nowhere in our solar system that has anything like the physical attributes that we would need to be able to live there, or, you know, to be able to make it a habitable place. Musk talks a lot about terraforming. We're not terraforming Mars. If we, if we had the technology and know how we would need to do something like that, then solving climate change here on Earth really would be a technological problem and you could solve it very easily. Um, he talks about the need for Mars as a lifeboat for humanity in the event that something horrible happens to Earth, like an asteroid as big as the one that killed off the dinosaurs 66 million years ago, impacting Earth again.

• 1:09:38Speaker 02

  Uh, there was nothing that, that could really happen to Earth that would make it as bad as Mars is to live on, short of the actual physical complete destruction of the Earth. Um, because even that day when that asteroid hit 66 million years ago was a nicer day for life on Earth than any day on Mars in the last, you know, two or three billion years at least. Uh, and we know that because mammals survived and birds survived and all sorts of life survived that day, whereas there's no mammal then or now that could survive unprotected on the surface of Mars. Um, the, it's, it's just horrible. And then like farther out in the solar system is even worse. We're not leaving the solar system

• 1:10:22Speaker 02

  because the speed of light is just too slow and we're not getting past it. We know that we can't go faster than the speed of light. We have really good science and really good experimental evidence to that effect. Um, you know, we've tested the speed of light limit over and over again in particle accelerators. That's a lot of, you know, particle accelerators get particles up to almost the speed of light. We know that our theories of, you know, relativity work in that domain. We know that you cannot get a spaceship going that fast. And we know that if you got a spaceship going anywhere near that fast, it would have enormous problems with radiation and shielding. Um, and we think worlds like Earth are pretty

• 1:11:03Speaker 02

  rare. So yeah, we're not going anywhere. Um, I could also explain a bit about why AI data centers in space are pretty laughable too, but I don't know if you want me to get into that.

• 1:11:16Speaker 03

  Oh yeah. I mean, that, that's low hanging fruit. I mean, I, I never quite understood that one. That there's, there's all sorts of places.

• 1:11:21Speaker 02

  So many problems. It's just crazy. Uh, my favorite is when people say, oh, but you know, it's easy to cool them because space is cold and like space is a vacuum. Vacuum is a perfect thermal insulator. Like, you know, good luck getting rid of that heat. You're going to have to have radiator veins the size of a city.

• 1:11:38Speaker 03

  I know. I mean, part of me is thinking it's great that we do have people that just have insane ambition because, you know, Musk doesn't, doesn't actually realize his most ambitious goals, but he has done some pretty amazing stuff. Starlink is very useful. Sometimes doing ambitious goals means that you find interesting new stepping stones and, you know, but, um, but there's also the angle of sometimes if you want to hire very talented people and you want to build something, you need to create a shared myth. Maybe you're a true believer. Maybe, maybe it's just some kind of, you know, systematic thing.

• 1:12:07Speaker 02

  Maybe, yeah, but I, I think that those myths end up having harmful consequences if they are so detached from reality, right? You know, Musk has created, I mean, Musk did not originate the myth of colonizing Mars, but he is, you know, in the same way that Kurzweil didn't create the idea of the singularity. Musk didn't create the idea of colonizing Mars, but he's certainly, you know, Mr. Occupy Mars these days. And, and it creates this idea that we don't need to care about the problems here on Earth because we're leaving, you know, and that's just not going to happen. You know, these, these, these goals, even if they're not possible to realize, have consequences. And, and the consequences

• 1:12:50Speaker 02

  that they have can be really horrifying. I mean, that's a lot of what my book is about. It's like, okay, these are why these goals don't work and the consequences trying to pursue them have.

• 1:12:58Speaker 03

  And, and just doing a bit of psychoanalysis here is, is your kind of prognosis that, um, there's an element of seeking purpose. So even folks in, in, uh, you know, working at OpenAI, OpenAI or whatever, they, they, they want to feel that they have a grand purpose and they're, and they're benefiting humanity. And is, is that a similar thing with, with these billionaires that, that, that they want to kind of feel that they're doing great things for the world?

• 1:13:21Speaker 02

  I do think that that has to be part of it. I mean, again, we can't get inside their heads. We don't know for sure, but you know, you, there are so many things that money can't buy and one of them is a sense of purpose, right? Um, and this sort of myth making gives them a sense of purpose. And, and you know, one of the most common responses I get to, uh, what I wrote in my book is, you know, do you really think that these guys actually believe this stuff? You know, why would they? It's, it's all just, you know, a ploy for them to make more money. And, and what I respond to with that is like, yeah, okay. I don't know what's going on in their heads. Maybe some of them are just being cynical. Although there's sometimes evidence like with, with Bezos, we know that he's always

• 1:14:02Speaker 02

  been very, very interested in colonizing space because he's been talking about it since he was in high school. Um, but, and so that suggests he's not just being cynical. But, um, I also think that when people ask me that question, what they forget and what I usually respond with, um, is the fact that, that these, these myths, uh, are useful for, you know, getting these guys more money and power makes it more likely that they earnestly believe these things, not less likely.

• 1:14:34Speaker 03

  If, if you, if you had your way, what, what would you do about all of this?

• 1:14:39Speaker 02

  Uh, if I had my way, I would tax billionaires out of existence. I don't think that that's a just or reasonable or fair distribution of resources. Uh, you know, these people did not make that money on their own. Uh, they needed the rest of us to get there. Uh, and, and by hoarding those resources, uh, they are creating massive power imbalances that erode at the fabric of our democracy and at our shared sense of truth as a society. And that makes it very difficult to live with one another. Um, and we, you know, need to find a way to live together in harmony. And I think that allowing these kinds of

• 1:15:22Speaker 02

  power imbalances make that impossible or next to impossible. So yeah, I think it should be illegal to have that much money.

• 1:15:30Speaker 03

  And, and also with current AI, I mean, just speaking from personal experience, it's really helped me, right? Um, I've, I'm doing lots more work. I'm, I'm doing the work of many, many people. And, you know, it's made me more consistent and I'm making better decisions and, you know, so, so I can kind of see where Sam Altman is coming from, you know, you, it's democratizing. Uh, now you can quit your job. You can start a business. You can use AI to help you. Um, that, that, that's the sales pitch, but from your perspective, I feel that you're saying actually structurally, this is going to create huge inequality and it's going to be, you know, especially when we have a two-tier system when most of us can't access the frontier AI, only some people can. So how do you see this rolling out?

• 1:16:11Speaker 02

  I think that, um, I don't have a great answer to that question because I don't think that it's written yet, right? We don't know. There are so many choices that we are going to have to make along the way. Um, you know, I can tell you something about what the tech billionaires want, but just because they want it doesn't mean that it's going to happen. Um, I mean, hell, they also want AI data centers in space and that's not going to happen. Uh, so yeah, I, I think we definitely need to be aggressively regulating this industry. And, and I don't just mean AI, I mean tech. Um, and yes, the tech titans are going to

• 1:16:54Speaker 02

  push back on that and that doesn't mean that they're going to win. And yes, the Trump administration isn't going to want to do that, but you know what? The Trump administration's not going to be forever. Uh, so yeah, I think it depends on what we allow these guys to get away with.

• 1:17:13Speaker 03

  And isn't it interesting? So, so Bernie Sanders was talking about nationalization and now Trump has done this export control on Mythos and on GPT 5.6. I mean, this, this is unimaginable because when Trump came in, he said, no, you know, go, go on, boys. Just, just, uh, just scale, scale, scale, beat China. So this is a really weird situation. I mean, do, do you think that they've been influenced by the AI risk lobby? Or do you think that it's just cynically KYC? They want to control it? What's going on?

• 1:17:39Speaker 02

  Well, I mean, uh, Bernie's definitely, I think, been influenced by the AI safety, AI doomer, you know, lobby. Um, and then as for Trump, I think that somebody dangled a bunch of money in front of Trump and showed him something shiny. I mean, you know, they jangled their keys in front of his face. Um, that's, that's generally my theory of how Trump makes decisions that aren't in his immediate obvious self-interest.

• 1:18:07Speaker 03

  Okay. And any final thoughts on where this is going to go? It's, it's a very unpredictable time, isn't it?

• 1:18:12Speaker 02

  It is a very unpredictable time. I'm hopeful. Um, I, I really am. I think that, I think that we are actually going to end up finding a way to regulate these guys. The question is how and when and what consequences that like, and, and what they, you know, what that struggle ends up looking like. And that I don't know, but the, the public sentiment abound around AI is really bad. The, um, the public resistance to building data centers is just getting stronger. Um, and there is also very clearly a financial bubble here. And when that pops, I don't know what's going to happen, but it's going to be pretty interesting.

• 1:18:54Speaker 03

  Yeah. And it might, it might pop quicker now just before the IPO with all of this fiasco.

• 1:18:59Speaker 02

  Yes, indeed. Yeah. Yeah. Especially with the SpaceX IPO not having gone particularly well.

• 1:19:05Speaker 03

  Adam, it's, it's been an honor having you on the show. Thank you so much. And, and folks at home, read this book. It's a very, very good book. I enjoyed reading it.

• 1:19:12Speaker 02

  Thanks for having me. This is great to be here.

• 1:19:18Speaker 04

  Finding a doctor is hard enough. Finding the right doctor, even harder. Someone who takes your insurance, has an available appointment when you need one, and feels like someone you can actually open up to. That's where Zocdoc comes in. Download the Zocdoc app to search and compare doctors by specialty, insurance, and availability. Read real patient reviews, find the right fit for you, and book instantly. Because getting seen by the right doctor matters. Zocdoc. Ready to be seen? Download the Zocdoc app today.

• 1:19:49Speaker 05

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the House Majority PAC dot com. Not authorized by any candidate or candidate's committee.