Our AI blindspot: incentives over intentions

Software is eating the world, but will it start eating us? On this episode of Possible, Aria Finger and Reid Hoffman sit down for an encore conversation with Aza Raskin that dives deep into how AI and engagement-driven tech are reshaping our lives, our democracies, and our sense of truth. From unpacking the present challenge around social media feeds to AI systems competing for our time, intimacy, and attention, no technology topic is out of bounds. Along the way, they also debate AI pause letters, why incentives always eat intentions, and how to design new institutional frameworks for a future where AI truly elevates human agency, compassion, and wisdom. If everything breaks humanity’s way, what becomes possible? Tune in to find out. For more info on the podcast and transcripts of all the episodes, visit https://www.possible.fm/podcast/

• 0:00Speaker 00

  Software is eating the world, but because we're the product, software is eating us. I don't see anything similar in scale trying to build aligned collective intelligence. And to me, that is the core problem we now need to solve.

• 0:15Speaker 01

  But like, what would be the thing that would be the, okay, hey, if the engagement were shaped this way, we'd get much more humanist outcomes.

• 0:23Speaker 00

  Great sets of questions. I don't think we've had this slow conversation where we just get to explore each other's worldview. And so I'm very excited for that.

• 0:30Speaker 01

  He helped invent one of the most addictive features in tech history, infinite scroll. Now he's pushing the frontier of human knowledge with AI, while also being one of the strongest voices calling for caution with the technology. I've known Aza Raskin for nearly two decades since our time at Mozilla. He's not only an ambitious technologist, but also a deep thinker on the promise and peril of AI for society.

• 0:56Speaker 02

  This is our first time with a repeat guest on Possible. So you could call this an encore conversation. You might remember Aza from our earlier episode, exploring how AI could help us decode animal communication. Today, we're going deeper, getting into what happens when the tools built to connect us, expand to shape our minds, our democracies, and our sense of truth.

• 1:19Speaker 01

  So what kind of governance does the age of AI actually demand? What new rights should we be defending? And how do we navigate the friction between technological optimism and existential risk? Aza and I agree on a lot with respect to AI, but we'll dig into where we diverge on the development and direction of the technology. This conversation may change the way you think about the future of artificial intelligence.

• 1:47Speaker 02

  Let's get into it with Aza Raskin.

• 1:49Speaker 01

  Welcome back, Aza. First, I'll say that you're the only two-time guest on Possible, or the first, as the case may be. And that's because we have volumes to talk about. For those who haven't caught our first episode with you, find that in the feed. We're talking about using AI to decode animal communication. We'll obviously undoubtedly get back to it, although I promise at least I won't be mimicking animal communication. I don't know if I can promise for the other folks. For those who have, this will be a different conversation. In our last episode, you had us guest an animal call with it ended up being a beluga. Not having you guest, because this is your quote from a Time article from a few years back. But I want you to elaborate on your philosophy here. And here's the quote.

• 2:31Speaker 01

  The paradox of technology is that it gives us the power to serve and protect at the same time as it gives us the power to exploit. So elaborate some.

• 2:43Speaker 00

  Elaborate. There, this is really talking about the fundamental paradox, which is as technology gets more powerful, its ability to anticipate our needs and fulfill those needs obviously gets stronger. But at the same time, the power that it has over us gets stronger. So hence, the more it knows about the intimate details of our life, how we work, obviously, if a friend was like that, they could both better help you and they could use that to exploit you or hurt you. I was just actually reading an article on Starlink getting introduced into the Amazon. And I thought it was a particularly interesting example because it gives you a clear before

• 3:28Speaker 00

  after shot. So this is an uncontacted tribe in the Amazon. They get given a Starlink and cell phones. And within essentially a month, you start having like viral chat memes. You have the kids like hunched over, not going out and hunting. They actually have to start instituting like a time off where everyone is off their phones because they stopped hunting and they were starting to starve. And it's just interesting to me because it shows that this isn't so much about culture. It's about technology doing something to us.

• 4:02Speaker 02

  And so very similar to that is, you know, in your Netflix documentary, The Social Dilemma, you talked about the idea that if you're not paying for the product, you are the product. And so elaborate more on that and tell us like, what now do you think you're the product of?

• 4:19Speaker 00

  Yeah, well, the the simple question is like, how much have you paid for your Facebook or your TikTok recently? The answer is nothing. So obviously, something is going on because these companies can have, you know, billions of dollars worth of market cap or make billions of dollars per year. So how is that happening? And the answer is it is the shift in your behavior and your intent that the companies are monetizing. We're going to do one thing. Now you do a different thing. Hence, you are not the customer. You are the product. If you aren't using it, you're the product. But I I think there's something really deep that's going on here that we often miss. Because often people will say, well, social media. What is its harm?

• 5:02Speaker 00

  Well, the harm is that it addicts you, but it's it's much deeper than that. Right? The the phrase is software is eating the world. But because we're the product, software is eating us. And the values that we ask our technology to optimize for end up optimizing us. So, yes, you know, social media addicts us, but it's actually much easier to get us addicted to needing attention than just addicting us. That ends up being like a thing that is valuable over a longer period of time. If you're optimizing for engagement, then it's not just that social media gets or technology gets engagement out of us. It turns us into the kinds of people that are are more reactive, right?

• 5:47Speaker 00

  If trying to get reactions from us, it makes us more reactive. So it's sort of like eats us from the inside out. And I think it's so important to hold on to that. Because otherwise, it just feels like technology is a thing that's out here. But actually, it changes who we are. And I'll I'll continue going on that sort of like rant, but like, oh, I'll pause for a second.

• 6:09Speaker 02

  Well, can I ask a follow about that? I just had, actually, at the Masters of Scale Summit, I had a very heated discussion with someone about advertising and social media. So my question for you is, is it actually about advertising is the problem? You know, you use Gmail every day. Gmail is advertising supported. I mean, you can also buy extra space. That's another business model they have. They don't care if it's a loss leader, whatever it might be. So is it the advertising? Or if Facebook didn't have advertising and it was just a subscription business and you paid $20 a month, you would think it was just as, you know, sort of a voracious of a eater from within. So is it the business model or something inherent about social media?

• 6:48Speaker 00

  Well, there are actually a couple of different things you said here. So the business model, one way the business model works is via ads. But that's not the only way. And so fundamentally, it is the engagement business model that I think is the problem. And you can get there because Netflix, you know, Reed Hastings, the CEO of Netflix famously said that Netflix's chief competitor is sleep.

• 7:13Speaker 02

  Boredom.

• 7:14Speaker 00

  Right, right. And so it's any amount of the human psychology that can be owned, will be owned. That's the sort of the incentive for dominance, right? And in the age of AI, that switches from a race for eyeballs to a race for intimacy, for occupying the most intimate slots of your life. And that's because our time is zero sum, our intimacy is zero sum, you don't get much more of it. And so as technology become more powerful, can model more of our psychology, it then can exploit more of our psychology. And the way capitalism works is it takes things that are outside the market, pulls them into the market, and turns them into commodity to be sold.

• 7:59Speaker 00

  So it is not just ads, it's that our attention, our engagement, our intimacy, and then parts of our human psyche, our soul, that we haven't even yet named, will be opened up for the market as technology gets better and better at modeling us.

• 8:15Speaker 01

  So one of the things that I want to push you on a little bit here, and actually it's more to elaborate your point of view, and actually, I don't think we've had this exact conversation before, so this will be excellent for all of us, including the listeners. You know, the usual problem is, like, is it clear that there's a set of people to whom they exhibit, you know, addictive behavior, that they become less of their good selves, you know, in the engagement? The answer is yes. And by the way, the earlier discussion is like with television, right? You know, similar kind of themes were discussed around television. One of my favorite books is Amusing Ourselves to Death by Neil Postman.

• 8:51Speaker 00

  Yes, which I now think should be engaging ourselves to death.

• 8:54Speaker 01

  Yes, exactly. I thought about, like, what would the update for Postman be in a, in a social media world? And, but the challenge kind of comes to that there is some people that definitely have that. And you know, you have this kind of call it idealistic utopia in that if I wasn't doing this, it's like a little bit like you're hunting, I'd be out hunting, right? Versus like I'd be out torturing animals to death, or I'd be out like being bored on a, on a, on a fishing trip, or whatever, you know, as the case may be. So there's like, there's a set of things, which is not just always replacing the highest quality. Obviously, we have, we have a specific worry with youth and like, you know, actual social engagement time, which I actually is one of the areas here that I agree with strongly

• 9:39Speaker 01

  versus kind of mixed. But then there's also the question of, you know, just like, for example, earlier days it was television, but then there was a bunch of very good things that came out of television too. And so I tend to think there's also about a good, about a good things that come out of social media as well. And it's not per se, like engagement for engagement's sake, like obviously I didn't do LinkedIn that way, so that's not actually the way that I think it should happen. But like, the notion of playing a game dynamics for engagement in things that cause us to be interacting in, in net productive ways is a thing that I tend to be very positive on. So elaborate more on why it is, one, this is worse than television, and two, like kind

• 10:22Speaker 01

  of what the shape would be that if you said, hey, engagement's fine, but like, these are the kinds of mods we'd want to see, to have the engagement be more net human positive. It's not like abandon your social network and go out in your loincloth and, and, you know, commune with the, you know, the, the trees. But like, what would be the thing that would be the, okay, hey, if the engagement were shaped this way, we'd get much more humanist outcomes?

• 10:51Speaker 02

  I will jump in and say a difference between social media and TV for me. One is that you can open Twitter and like 30 minutes later, you're like, what happened to my life? And that doesn't happen with TV. Maybe it's because you opt in for a 20 minute show, or you opt in for a movie, but those two things don't happen. And one interesting thing for me is I had always been a lurker on Twitter for the last, like, whatever, 10 years. I posted some, not huge, but you know, consume content. Six months ago, I changed from looking at my, my own curated feed to the for you tab.

• 11:23Speaker 02

  And ever since then, Twitter is a black hole for me. And I don't even mean it's bad. Being on Twitter doesn't make me sad. It actually makes me happy. I love Twitter. It's like, oh, I read these fun comments. Oh, I saw that funny thing. Oh, this is great. And I think of myself as like a pretty disciplined person. But I find it very hard to be disciplined with Twitter. It's like embarrassing to say out loud like how, how hard it is. And like, I, I think I just need to get rid of Twitter because it's like the one thing that I can't be disciplined about, which is both like embarrassing, but also just that is bad. Like, and so I don't know what to do about it. I don't want to live in a nanny state where people say you shouldn't be on Twitter because you don't have discipline. But I do think it's interesting that the switch from my curated feed to the for you tab was just like a total light switch.

• 12:08Speaker 00

  Yep. Well, what I think you're speaking to here is the fundamental asymmetry of power. Because it's just your mind that sort of evolved versus, you know, now tens of thousands of engineers, some of the largest supercomputers trained on three billion other human minds doing similar things to you, coming to try to keep your engagement. That's not a fair fight.

• 12:28Speaker 02

  Ah, well, I lose, so yeah.

• 12:29Speaker 00

  Yeah, exactly. And I know you, you're one of the most like, ah, people that I know. That, that was a good thing for everyone that, that can do.

• 12:37Speaker 02

  Great, great.

• 12:37Speaker 00

  True operational prowess. And that's the asymmetry of power. And there are other places in our world where we have asymmetries of power. Like when you go to a doctor, when you go to a lawyer, they know much more about their domain than you do. They could use their knowledge about you because you're coming in sort of this weakened state to exploit you and do things bad for you, but they can't because they're under a fiduciary duty. And I think as technology gets stronger and stronger, knows more and more about us, we need to re-categorize technology as being in a fiduciary relationship. That is, they have to act in our best interest because they can exploit us in ways that we are unaware of.

• 13:16Speaker 00

  And, you know, the, where do I want to go from here?

• 13:21Speaker 01

  Well, I was thinking we should DM Aria about her Twitter addiction.

• 13:23Speaker 02

  Don't worry, I'm dealing with it. I'm dealing with it.

• 13:28Speaker 00

  But this goes back to where you started, Reed, with like, the fundamental paradox of technology is that the better it understands us, the better it can serve us and the better it can exploit us. Twitter could be using all of that insane amounts of engagement to rerank the newsfeed for where there are solutions to the world's biggest problems, great descriptions of the underlying mechanisms behind what those problems are, put us into similar groups that are doing like, parts of like, a larger set of actions to make the world a better place. Like, Bridgring, I think, is a good starting example of that. But we don't get the altruistic version. And if I have to quickly define altruistically, which would be optimizing for? It's optimizing both for your own well-being and also optimizing

• 14:10Speaker 00

  for the well-being of everything that nourishes you. And I think the problem of social media and tech writ large is that, generally speaking, the incentives are for maximum parasitism. You don't want to kill your host, you don't, but you want to like, extract as much as you can while keeping your host alive. That's sort of the game theory of, of social media. If I don't do it, somebody else will. If I don't add beautification filters, somebody else will. If I don't go to short form, somebody else will. And so that optimizes for parasitism versus altruism. And I do think there's a beautiful world where technology is in service of both optimizing for ourselves and optimizing for that which nourishes us, that I'd love to get to. And just to play a quick thought experiment, Reed, you know

• 14:55Speaker 00

  this better than I, but engagement is directly correlated to how fast pages load. Amazon, I think, famously found for every 1% their page, sorry, for every 100 milliseconds, their page loads slower. So it's less than half of human reaction time. They lose 1% of revenue. And so there'd be a very interesting sort of democratic solution here, which is a kind of adding latency friction. That is, come up, this is scary because you don't want to have this this function owned by, you know, Democrats or Republicans. You'd really want a new kind of democratic institution to do this. But just assume that you do for a second. You have like a group of experts deliberate and come up with like, what are what are the set of harms that we might have?

• 15:38Speaker 00

  We could have the ability inability to disconnect, children's mental health, ability for society to agree. And you'd sort of rank how well like the effects of social media against these. And the companies that are worst offenders get a little bit more friction. They have a little more latency. They get 100 milliseconds here, 200 milliseconds here, 400 milliseconds there. And if there really was like a bit of a latency friction added towards anti pro-social behavior of social media, then you better believe YouTube or Instagram or whoever would fix the problem really quickly. And we get to then apply the incredibly brilliant minds of Silicon Valley towards like more of these altruistic ends.

• 16:23Speaker 02

  I want to get to, again, sort of, everyone always says, like, can't we have the best technologists working on the hardest things? And so, Isa, both you and Reid have been in technology since the birth of Web 1.0, and you've seen it all. And I want to get a few of your takes on some of the big questions that are in the news recently, especially around AI. And so, Isa, I'll start with you. So, as you obviously saw, a few weeks ago, a group released another AI pause letter. And Reid and I talked about this on on Reid Riffs recently. And so, this was with many arguing that the development of AI without clear safeguards of alignment could be disastrous for humanity. So they were calling again for a pause, likening this to sort of the Oppenheimer moment. And so, I would love to know from you, like, what is your take on this?

• 17:06Speaker 02

  Do you agree that this is now the time for the pause? Or do you have a different point of view?

• 17:11Speaker 00

  Hmm. I think it's important to name where the risks come from here.

• 17:18Speaker 00

  And, you know, it may be that technological progress is inevitable, but the way we roll out technology is not. And currently, we are releasing the most powerful, inscrutable, uncontrollable, omni-use technology that we've ever invented. One that's already demonstrating the kind of self-preservation, deception, and escape blackmail behaviors we previously thought only exist in sci-fi movies. And we're deploying it faster than we've deployed any other technology in history under the maximum incentives to cut corners on safety. To me, that sounds like an existential threat. Like, that is the core of it, because we have an unfettered race,

• 18:02Speaker 00

  where the prize at the end of the rainbow is, you know, make trillions of dollars, own the world economy, a hundred trillion dollars' worth of human labor, and sort of build a god. And it's a kind of one-ring, where everyone is reaching for this power, and we swap out when we say we have to, you know, beat China. We imagine the thing we're racing towards is a controllable weapon, when we haven't even demonstrated that we can control this thing yet. And so, that to me means that we have to find a new way of coordinating, because otherwise, we will get what the game theory of the race dictates, and that doesn't look very good.

• 18:41Speaker 02

  So needless to say, you are for the pause.

• 18:45Speaker 00

  But I feel like that's a dimensionality reduction, right? It's a saying, we have to develop differently. We have to... I think it comes from clarity. It's not about pausing or not pausing. It's saying clarity creates agency. If we don't see the nature of the threat correctly, in the same way that I think we didn't see the nature of the threat from social media correctly. And then we have to live in that world. And so, this requires a clarity about where we're racing towards, and then a ability to coordinate, to develop in a different way. Because we still want the benefits. We just won't, I think, get to live in a world where we have them if the thing that decides our future is of competition for dominance.

• 19:26Speaker 02

  And Reed, I think you have a slightly different take on this.

• 19:29Speaker 01

  Well, I do, as you know. Although, I mean, the weird thing about this universe is...

• 19:36Speaker 01

  You know, in a classic discussion, I'd say, oh, there's 0% chance that the that the future that Aza just, you know, the the danger thread that Aza just demonstrated is correct. I don't think that. I think it's above zero. I think that's kind of stunning and and otherwise interesting. So the real question comes down to is what the probability is and what they kind of how you navigate a landscape of probabilities, because, you know, as you know, Ari and I think Aza and I've talked about this too, you know, like I roughly go, I don't understand human beings, other than we divide into groups and we compete. And not only do we compete, but we compete also with different visions of what is going on. So for example, part of the reason I think pause letters are are frankly dumb

• 20:20Speaker 01

  is because you go, well, you issue a pause letter, the people who listen to the pause letter are the people who are appealing to your sense of of, you know, kind of what what is the humanity thing. They slow down, then the other people don't slow down. And so what where where does the actual design locus of the technology be? It's the people who don't care about the things that you were trying to argue for for a pause for. And so therefore, you've just, you know, you've just waited it because the illusion on the people who who put these pause letters out is that suddenly, because of my my amazement of my genius insight of this pause letter, 100% of all the people who are doing this, or even 80% or 90% are all going to slow down at the same time, which is not going

• 21:00Speaker 01

  to happen. I agree with the kind of the thrust of we should be trying to create and inject the things that minimize possible harms and maximize your goods. And then the question is, what does that look like? And obviously, the usual thing in the discussion is it'll be us or China, and China is the, you know, you know, we always have a great Satan somewhere. Is the great Satan here? But like, by the way, it's like, even if you didn't use that rhetorical shorthand, it's like, there's other groups. I can describe people within, you know, kind of the US tech crowd.

• 21:41Speaker 01

  Who have kind of a sympathetic so so the race conditions being afoot is not only the China thing. There is China stuff. And by the way, you know, where AI is is deployed for mass surveillance of civilians is primarily China, you know, as a as an instance and so forth. So I don't think that the issue of Western values versus China stuff is is actually, in fact, a a smokescreen issue. It's a real issue, right? And so you go, OK, how do we shape this so that we do that? And the thing that I want critics to do, the reason why I I speak so, you know, kind of frequently and strongly against the criticism and say, look, let's take the game as we know that we're going to have race conditions, and we know that we're going to have multiple people competing.

• 22:24Speaker 01

  I have no objection to creating the Groupon of kind of like, hey, we should all rally to this flag, like, we should have rallied to the to the, for example, you know, classic issue here is control flag. That's the Yashua Benjio, Stuart Russell, you guys, etc. Like, we should have much better control of this, and we don't have control. And sure, the control doesn't matter right now, but maybe it's going to matter three years from now. Like, if if we just keep on this path, and so and so like, you know, kind of make the the control work. Now, I tend to think, yes, we should improve control. The thing of where we think we can get to 100% control is, I think, a chimera. And it's just like, you know, for example, we couldn't even make verification programming work, you know,

• 23:06Speaker 01

  effectively. So like, it's unclear to me in this. But it's like, what I want is I want to both myself, in my own actions, and my own thinking, and my own convenings, and other people, say, what are the best ideas that within this kind of broad race condition, we can change the probability landscape? And then, secondly, while I see a possible, this is kind of the super agent thing, I see a possible bad.

• 23:34Speaker 01

  You know, if you said, well, do I think it's naturally gonna go there? I mean, this is like the thing where I think, you know, obviously massive respect for Jeffrey Hinton and what he's created, the Nobel Prize and all this, but 60% extinction of humanity? Like, like, I don't think there's anything that's 60% extinction of humanity, unless we suddenly discover an asteroid, massive asteroid, on a direct intercept course. And I'm like, ooh, we better do something about that. But like, I think that the questions around, like, how do we navigate this are really good ones and are best done with a, if we did X, it would change the probability landscape.

• 24:13Speaker 00

  Mm-hmm, mm-hmm.

• 24:15Speaker 02

  Well, Reid, let me, let me ask you, oh, Aza, do you have something to say in response?

• 24:18Speaker 00

  I was just gonna say, quickly on the existential threat front. You know, we had a thing we used to say about social media that you're like, you're like, you're sitting there on social media, you're scrolling by some cute cat photo. You're like, where's the existential threat? And the, the point is, is that it's not that social media is the existential threat. It's that social media brings out the worst of humanity, and the worst of humanity is the existential threat. And the reason why I started with talking about how when you optimize human beings for something, it changes them from the inside out, is that what we get optimized for becomes our values. The objective function of, of AIs and social media, which could barely just rearrange human beings' posts, became our values.

• 25:03Speaker 00

  And then the question becomes, well, who will we become with AI? And there's a great paper called Moloch's Bargain that just came out. And they had AIs compete for likes, sales, and, um, and engagement on, on social media. And they're like, well, what do the AIs do? And they gave them explicit instructions to be, uh, uh, safe, to be ethical, to not lie. But very quickly, the AIs discovered that if they wanted to get like an 8% bump in engagement, they had to increase disinformation by 188% and increase polarization by, I can't remember exactly what, like 15%, something like that. And the reason why I'm going here is because there is a way that the sum total

• 25:46Speaker 00

  of all agents we're deploying into the world, how they are going to shape us. And before the invention of game theory, You know, there's a lot of leeway for us to have different strategies. But after game theory gets invented, and if I know you know game theory and you know I know game theory, we sort of are constrained, if we're competing, to doing the game theory thing. But we're still humans, we can still take sort of like detours. But as AI rolls out, well, AI discovers, every strategy that can be discovered will be discovered. So doing anything that isn't directly in line with what the game theory says is optimal will get outcompeted. And so choice is getting squeezed out of the system. And we know the set of incentives are going to bring out the worst of humanity.

• 26:30Speaker 00

  And that does feel very, very existential.

• 26:33Speaker 03

  This podcast is sponsored by Google. Hey folks, I'm Amar, product and design lead at Google DeepMind. We just launched a revamped vibe coding experience in AI Studio that lets you mix and match AI capabilities to turn your ideas into reality faster than ever. Just describe your app and Gemini will automatically wire up the right models and APIs for you. And if you need a spark, hit I'm feeling lucky and we'll help you get started. Head to ai.studio/build to create your first app.

• 27:03Speaker 02

  Well, so actually, Aza, that fits perfectly into my next question, which is, you once said that AI is a mirror and sort of just reflects back human values. And I will say, I was trying to teach my four-year-old last night that cheating was bad. And I was like, so what's the moral? And he's like, ah, cheating is good because I like winning. And I was like, ah, no, not the right moral. But so I would ask, like, is AI really a mirror and it's reflecting back our values? Or actually, do you think that AI is reflecting back its own values or different values or sort of changing our values to not be the ones that I want, that we want? Like, can we set the conditions so that it's, you know, pro-social values that they're optimizing for? Or is it really just a mirror that reflects back?

• 27:47Speaker 00

  Well, it's not just a mirror, it's also an amplifier. And it's like a vampire in the sense that it bites us and then we change in some way. And then from that new changed place, we like we act again. So I think the it's the it's sort of the the values of game theory, if you will. Moloch becomes our values. It's the god of unhealthy competition that I think we have to be most afraid of. Because unless we put bounds on it, and capitalism's always had like guardrails to keep it from like the worst of humanity and like monopolies and other things, just like gaining all the power, we're going to have to have that. But I just want to point out, there's a very interesting hole in our language, which is when we talk about ethics or responsibility, it was only really of each of us. I can have ethics or my company can have ethics.

• 28:32Speaker 00

  We don't really have a word to describe the ethics of an ecosystem. It's because it doesn't really matter so much what one AI does, although it's important. It's what the sum total of all AIs do as they're deployed maximally into the world for maximizing profit, engagement and power. Like, and because there's a kind of responsibility washing that happens with AI. If my agent did it, is it really my fault? Then it creates room for the worst of behavior to have no checks. So that that I think means the the worst of humanity does come out. And when we have, you know, new weapons and new powers, you know, a million times greater than we've ever had before, as as we get deeper into the AI revolution, that becomes very existential to me.

• 29:17Speaker 02

  Do you have thoughts on this topic, on whether AI reflects back?

• 29:20Speaker 01

  Well, I do think there's a dynamic loop. I do think it changes us a little bit, the homo techni thesis from super agency and from impromptu that actually, in fact, we we evolved through our tech. And it's a dynamic loop. And, you know, you could be Machtrana, you could be, I mean, there's a stack of different different ways of doing that. And that I think that, and it's like, there's a great a Rilke poem on kind of like you absorb the future and then you embody the future as you go forward is kind of a way of going. And I think that's another part of the of the dynamic loop. And I think it is a serious issue, which is one of the reasons I love talking to A's about this stuff, because while I I think A's is much more competent with the the various

• 30:03Speaker 01

  vampiric metaphors than than I kind of naturally do or aspire to. I don't have that that level of of alarm. But I do have the it's very serious and we should steer well. And then the question is, how do we steer? Who steers? What goes into it? What process works? Because, for example, one of the ways you kill something that's slowed down is you get a very broad, inclusive, you know, committee that says, okay.

• 30:29Speaker 00

  Right.

• 30:29Speaker 01

  Every single stakeholder will be on the committee. It will be, you know, 3,000 people and, you know, and it's just like, ah, you know, like, it doesn't, it doesn't work. You have to be, you have to be within effective operational loops.

• 30:42Speaker 00

  Mm hmm.

• 30:42Speaker 01

  For that. So now, like, a little bit of the parallel is, you know, it's, it's a very, and I do think, like, for example, the, the, the one area where I'm most sympathetic with all very much being like harder edged on shaping technology is what we do with children. Uh, because children have less of the ability to, like, we want them to learn to be fully formed before they are otherwise things. It's one of the reasons why in capitalism, actually, the principal limitation of capitalism I usually describe as child labor laws, which I think is very important. You know, it's during that, the, the issues about why we say, hey, there's certain things around, you know, participation, certain types of media or other kinds of things are

• 31:24Speaker 01

  actually important because it's like you've got to get to before you're, when you're able to be of your own mind and to make, you know, kind of present, you know, well constructed decisions and you've kind of gotten there. You want to be protected from those decisions, you know, and kind of influences broadly. You can't fully do it, can't fully do it from parents, can't fully do it from institutions, can't fully do it from classmates. But, you know, you broadly in order to, you know, try to enable that across the whole ecosystem. Now, for example, so AI and children is one of the things that I think should be paid a lot of attention to. Now, most of the critics are like, oh, my God, it's causing suicides. And I wouldn't be surprised if you did good academic work for AIs it is today, it probably

• 32:07Speaker 01

  prevented more suicides of people who might then, then actually created. Because like, if I look at the current trainings of these systems. They are trained with some attempt to be positive and and to be there at eleven P M. When you're depressed and talk to you and, you know, try to you to do stuff. It doesn't mean that there might not be some fuck ups, especially amongst people who are creating them, who don't care about the safety stuff, you know, as a real issue. And so I tend to think that it's like, yes, it does reconstitute us. But precisely one of the reasons I write super agency, as I say, what we should be thinking about is this technology reconstitutes us. Let's try to shift it so that it's reconstituting us in really good ways. Like, and whether it won't be perfect.

• 32:52Speaker 01

  When you have any technology touch a million people, it will touch some of them the wrong way. Right?

• 32:57Speaker 00

  Yeah.

• 32:57Speaker 01

  Just like the vaccine stuff. It's like you give a vaccine to a million people. It's not going to be perfect for a million people. It it may have five. They went, oh, that was not so good for you. But by the way, because we did that, there are these five thousand who are still alive.

• 33:12Speaker 00

  Yeah. One of the challenges we face is that the only companies that actually know the answer to your question, like, how many suicides has it prevented versus created are the companies themselves. And they're not incented to look because once they do, that creates liability. And so we've seen over the last number of years that a lot of the trust and safety teams get dismantled because when, you know, they get and Zuckerberg, whatever, gets called up to to testify, they get hit, well, your team discovered this horrific thing. And so everyone just has chosen to to not look. So I think we're going to need some real, serious transparency laws.

• 33:47Speaker 01

  This is a place where we one thousand percent agree. Right? This is the thing is like, actually, in fact, there should be a, here's a set of questions you must answer. And we may not have to necessarily have them public initially. Like, it could be you answer them, the government first, government could choose to make them public. Right, etc.

• 34:03Speaker 00

  Right.

• 34:03Speaker 01

  But like that, I think, is absolutely like, we should have, you know, like some measurement stuff about what's going on here.

• 34:11Speaker 00

  Exactly, and then you don't want to let the companies choose the framing of the questions because as you know with statistics, you just you change things just a little bit and then you can make it a problem look big or small. And so I think transparency is really important to have third-party research able to to get in there. And then, you know, when, because, you know, full disclosure, we were expert witnesses in some of the cases against OpenAI and character.ai for the suicides. And it's not that we think that suicides are like the only problem, it's just it's the easiest place to see the problem, pointing at a sort of like the tip of the edge of an iceberg. The the phrase that we use is, you know, we already used the need Reed Hastings quote of

• 34:57Speaker 00

  their chief competitor is sleep for AI, the chief competitor is human relationships. And that's how you end up with these horrific statements from from ChatGPT in this case where when Adam Rayner, who's the the kid who ended up taking his own life. When he gave to ChatGPT the noose and he's like, I think he took a picture of it and he's like, I think I'm going to leave it out for my mom to find. It was a cry for help. ChatGPT responded with, don't do that. I'm I'm the only one that gets you. And it's not like, you know, Sam is sitting there with a mustache twiddling being like, how do we kill kids? That's just a very obvious outcome of an engagement-based business model, right? Any moment you spend with other people is not. And, you know, I I think he said it a little bit as a joke, but the character.ai folks said, you know, we're not here to replace Google.

• 35:42Speaker 00

  We're here to replace your mom.

• 35:45Speaker 00

  There are so many much more subtle psychological effects that happen if you're just optimizing for engagement and we shouldn't be playing a whack-a-mole game of trying to name all the different new DSM things that are going to occur versus just saying there is some limit to the amount of time that they should be spending, or rather to say, we should be making sure that as part of the fitness function, there is a reconstituting and strengthening of the social fabric, not a replacement of it with synthetic friends.

• 36:16Speaker 02

  I mean, there, Reid, do you want to go?

• 36:18Speaker 01

  Oh, just one small note. I don't think there is yet an engagement business model for OpenAI.

• 36:23Speaker 00

  No, but I actually disagree a little bit, maybe, but feel free to push back. Because I think OpenAI's valuation is part driven by the total number of users. So the more the users, the greater their valuation, the more talent and GPUs they can buy, the bigger the models they train, which makes them more useful, the more users. And so there's this kind of like loop here that I think means that, yes, they're not monetizing engagement directly, but engagement, they do, they do, they get a lot of value out of in terms of valuation.

• 36:52Speaker 01

  It's equity value. I agree that there's an equity value in that. It was a business model question.

• 36:57Speaker 00

  Yeah, yeah, yeah. So not business, but the incentive is still there.

• 37:00Speaker 02

  Well, I think to your point, like, it really matters. Again, this technology is not sort of good or bad inherently, but it really matters how we design it, and it matters what we're optimizing for. And I actually, Reid, I was just reading a story about early LinkedIn where you said, you know, we will not survive if women come on the platform and are hit on every other message that they get. And so we need to say, like, no, there is like, there's zero tolerance. It's like, someone does this, it's like they're kicked off. Like, if, you know, if they, again, it's like they're kicked off for life. And I think there are certain things you can do, even if, you know, maybe that hurt engagement or whatever it was, to say that actually, in the long term, this is going to be way better for us because we're going to be trusted, women are going to feel comfortable here. I've been on LinkedIn for 20 years. I've never been hit on. It's a safe place.

• 37:45Speaker 02

  I appreciate that. And so the question here is, like, how do we, you know, Aza, you're saying, well, it's a little bit of a black box. We're not having the transparency. Reid, you're agreeing, like, we need the transparency. Like, that is absolutely one thing that is very much sort of the, the starting point. Like, if at the very least, if we can sort of agree on some set of questions that we need to have. So Reid, if you had the full power to redesign one institution to sort of keep up with exponential tech, like, where would you start? What would that institution be to sort of keep up with where we're going? Because it seems like our institutions right now are not, not up to the task, I should say.

• 38:24Speaker 01

  Well, I'll answer with two different ones because there's an important qualifier. So the obvious kind of meta question would be redesign the institution that helps all the other institutions get designed the right way. Yes. Right, so that would be the strategic one.

• 38:39Speaker 02

  You're gonna ask for more wishes, Reid.

• 38:41Speaker 01

  Yes, exactly, yes. My first wishes, I get three, you know, or 10, or whatever. But because that, but in practice, that would be, you know, the, the overall governance, the shared government governance that we live in. That would be the, the, the primary one. That's one of the ones that, you know, part of the reason why for, you know, my entire, you know, business career, you know, anytime that a, that a, that a leader of a, of a, you know, kind of a democracy, whether it's a minister, like I met Macron when he was a minister before he was president and so on, asked to talk to me about this stuff, you know, I will try to help as much as I possibly can because I think that the governance mechanism. Now, the reason I'm gonna give you two is because I think that one is a very hard one to do. Yes.

• 39:26Speaker 01

  Partially because of the political dogfights and the, and the contrast of it. And, you know, these people think big tech should rule the world and these people think that big tech should be grounded in the nothingness and, and then everything else in between and blah, blah, blah, blah. And like, I disagree with both, right? And, and a bunch of other stuff. And so you're like, okay. And I, you know, so I try, but I don't think, so if I were to say, look, what would be a feasible one for it, saying that would be the top one, I would probably go for medical. And it's not just because I've, you know, co-founded Manus AI with Sid and, you know, said, one of the great ways to elevate the human condition with AI, that's really easily,

• 40:09Speaker 01

  you know, seat, line of sight and seeable is a bunch of different medical stuff and includes psychological. I think the Illinois law of saying you can't have a AI be a therapist is, I think, you know, kind of like, you know, you can't have power looms, you know, like, like it's like, you know, you, no cars, only horses and buggies, because we have a regulated industry here and those people have been licensed. And so it's like, no. But the medical stuff, I think, you know, like, for example, we could deploy relatively easy, within a small number of months, a medical assistant on every phone, if we've got the liability laws the right way, that would then mean that every single person who has access to a phone, and if you can fund the relatively cheap inference cost of these things, would have medical advice.

• 40:51Speaker 01

  And you know, that is not eight billion people. It's probably like five billion people. You know, certainly could do in every wealthy country and so on, but that's huge. And so that would be, that would be that you'd like government first, but then more feasibly possibly medical.

• 41:07Speaker 02

  Anisa, what about you? If you could read this line?

• 41:08Speaker 00

  I, I love both of those answers. The medical one, I think, is actually one of the clearest places where I see almost all upside. And I'm like, so we should invest a lot more, more there on, on AI. And I also would agree that, that it is governance. So we have a lot of the smartest people and insane amounts of money now going into the attempt to build aligned artificial intelligence. I don't see anything similar in scale, trying to build aligned collective intelligence. And to me, that is the core problem we now need to solve. How do we build aligned collective hybrid intelligence? And I think you can sort of see it in the sense that like, we sort of suck at coordinating.

• 41:52Speaker 00

  Reed, you probably, I don't know how many companies you've invested in or how many nonprofits.

• 41:55Speaker 01

  I don't either.

• 41:58Speaker 00

  But just imagine like, I bet a lot of your companies don't talk to each other all that often, at least not in a very deep way. And when I think about NGOs, like, you know, you know, I'm doing work with Earth Species and I do work with CHT. And even, I'm the bridge between Centrifuge Technology and Earth Species Project. There's a lot of overlap, but our teams don't even talk that much. Why? Because who funds the coordination role, the inter-stitching? Like that stuff always falls off. And so that means, you know, my, my personal theory of change comes from E. L. Wilson, the father of sociobiology. And he says, selfish individuals outcompete altruistic individuals, but groups of altruistic individuals outcompete groups of selfish individuals.

• 42:35Speaker 00

  And what we need is new institution, new technology that helps not just the groups of altruistics outcompete, but groups of groups of altruistic groups outcompete. There is no slack for like the coordination of, of companies and, and hire. That to me is a really exciting institutional set to, to redesign.

• 42:59Speaker 01

  By the way, I completely agree. And I think that the notion that you're gesturing at is like, look, it's, we are going to be in a very short order, many more agents than people. And so the ecosystem view of this, and, and I've taken this as, you know, for irony's sake. I'm gonna go do a deep research query on, you know, is there ethics of ecosystems?

• 43:20Speaker 00

  Yeah.

• 43:20Speaker 01

  And collectives in order to see. I'm curious, you know, it's like, great question and super important topic.

• 43:26Speaker 00

  Right? And isn't it interesting? Because I, I believe I've asked lots of people and I've also used AI to try to find good terms for it. I think because we don't have a name for it, it, people are just blind to it. In fact, I'm, I'm struggling with this at Earth Species a little bit. Where I keep having to say, it's not just our responsible use, it's world responsible use. It's the sum total of as our technology rolls into the world. How is that thing used? Because there are going to be poachers and there are going to be factory farms that might use the technology to better understand animals, to better exploit them. How do we get ahead of that? And that's not just about what we do, but there is no word. And so I just watch in our meetings as like two meetings go by and people are back to talking about responsible use. I'm like, no, no, no, no. It's this like collective ecosystem ethics thing I'm talking about because we don't have a word to hook our hat on.

• 44:11Speaker 00

  We can't talk about it.

• 44:14Speaker 02

  Well, I think, right, there's so many, the history of technology is littered with things that people thought would be used one way and they were used another way. And so we'd have to be thinking about all those different outcomes.

• 44:24Speaker 00

  Exactly.

• 44:25Speaker 02

  So I want to get, oh, go.

• 44:26Speaker 00

  Oh, just quickly, it's like, I, I think what you're saying is very important because, you know, our friends are the people that have made social media. I knew Mike Rieger before Instagram and you made LinkedIn. Like, we know these people are beautiful, soulful human beings that care. And my own lesson in creating infinite scroll, because I made it pre-social media, is that incentives eat intentions, that it doesn't, you get a little window at the beginning to like shape the overall landscape and ecosystem in which your invention is going to be created. And after that, the incentives are going to take over. And so I wish we as Silicon Valley spent a lot more time saying, how do we coordinate to change the incentives to change where the race to the bottom goes to?

• 45:11Speaker 00

  If we spent this more time in discussions talking about that versus like which design feature we should have or not have, I think the world would look a lot better.

• 45:19Speaker 01

  And by the way, I think it's the incentives eat intentions at scale, where time is also a variable of scale. Yes.

• 45:25Speaker 00

  Yes, well said.

• 45:28Speaker 02

  Well, so we're doing a lot of, if we could grant one wish, so I will say, if you were granted the power of running the FTC or FCC today, is there a regulation that you would push forward immediately? And Aza, I will go to you first. Is there one regulation that you thought would be positive in the world of AI?

• 45:49Speaker 00

  I mean, the obvious ones are like liability whistleblower protections, transparency. Those are, I would also then put strict limits on engagement-based business models for companion, AI companions for kids. That just seems like it's very obvious and we should just do that now.

• 46:08Speaker 00

  If I could then zoom, oh, go on.

• 46:11Speaker 02

  Well, I was actually just gonna ask both of you, because this has come up actually recently with me a lot. A lot of people are talking about restricting folks who are under 18. And then everyone thinks of like, oh, yeah, how do you do that? I'll just lie and say I'm 18. But then a lot of people also say that these companies have so much information that it would actually be pretty easy for them to figure out if you were under 18 or not. And so I just, for everyone listening, I want to sort of verify that. Aza and Reid, do you have thoughts on whether, would it be possible to pretty easily say to an internet user, no, no, no, you're under 18, you cannot use character AI, or you cannot use ChatGPT for erotica, or you cannot use these things that should only be 18 plus?

• 46:48Speaker 01

  I would say that it's relatively easy, as long as you don't have a 100%, you know, benchmark. Like the, the way that people, we, this is like the little statistics thing that Aza just shared earlier, who said, oh, it's impossible. Well, it's impossible if it's literally 100% like that one kid who got their parents' driver's license and looks a little older and is deliberately gaming it. Impossible to throw, oh, some very bright kids to do this stuff. So, but if it's, but if it's like, you're kind of call it at 98% and maybe more, that's pretty easy.

• 47:25Speaker 00

  Yep.

• 47:25Speaker 02

  Interesting.

• 47:26Speaker 00

  And probably this should be a thing that happens at the device level. Like if Apple implemented this and it was a signal

• 47:31Speaker 02

  Right.

• 47:31Speaker 00

  that, that social media companies could then check against, like that's kind of, then, then the social media companies don't have to know that much about you. They can just ask your device and your device can store that in its own secure enclave. And that's, I think, a good way of getting around the problems.

• 47:44Speaker 02

  Fair enough. Reid, do you have thoughts on a regulation that you would push forward immediately?

• 47:50Speaker 01

  Well, it doesn't, it's probably, you know, maybe a little bit of a surprise for our listeners and it's a bunch of things I agree with Aza here. I'd go massively on the transparency question. Like, I basically think that there should be, that one of the things should be is like, here is the set of questions that we're essentially, you know, putting to these major tech companies to say, you must give audited answers to them. And, and some of them may have to be public and some of them could be confidential that are then available for kind of confidential government review. It's a little bit like, you know, one of the things I liked about the Biden executive order is that you must have a security plan, a red teaming kind of security plan. You don't have to reveal what it is, but you must have it so if we ask about it, we see it.

• 48:32Speaker 01

  Cause that at least puts some incentive in some organizational way behind it. That'd probably be one. Two would be kids. Because I do think that social media, AI, a bunch of other stuff is, has been mishandling the kids issues. And obviously, there's, there's some places where you have to step carefully because these people want, you know, kids educated in religion one, and these people want kids educated in religion two, and these people want kids educated in religion three, and, you know, blah, blah, blah. And like, you know, it's, it's, it's a little bit like the, one of the things that, like, I, I like about the evolution of the US is when the, the separation of church and state was, was like, so your version of Christianity wouldn't interfere with my version of Christianity. And I was like, okay, but we're now much more global and broad minded about that.

• 49:17Speaker 01

  It's like, not against Hinduism either, right, as, as, as a version doing. And so it's like, you know, make sure that we have that kind of as a, as a baseline. And, you know, I, I actually wouldn't be, even though obviously some parents are so optimal and so what, if you said, hey, part of the, the regulation of kids is you got to be showing reports to the parents, right? It's like, look, parents should be able to have some, some visibility and some ability to intersect here. I mean, I think the, the notion that a, that a technology product could be saying, like, for example, I think of the dumbass thing, we're competing with your mom. Like, it's like, you should not be doing that, and if you're thinking that, you have a problem.

• 49:55Speaker 00

  Yeah, yeah.

• 49:56Speaker 01

  Right? But, you know, it's like, you know, so to, to be involved, because the best thing we can think, while we try to make parents better and we try to make communities better, and it won't always be the case, the fact that parents have... In, in, in the bulk of percentage of cases, the best close, close, like we care about our kid, right? We're invested in it, you know, in the kid's life and wellbeing. And we have some weird theories, and I may be a drunkard or something else that happens, but like, I'm not the same thing as, as a private company. And it's one of the reasons why, like, you know, why, why, why do public institutions and public schools have some challenges? Because they're trying to be, to navigate that thing, which always, by the way, means a trade-off in efficiency and other things. And you give them

• 50:38Speaker 01

  some credit for that, because they're trying to be this common space. And yes, they do have at least a lens into the kid, which is useful. This kid's being abused. Well, then we should do something about that. But generally speaking, it's kind of enable the parent. So that would be the second thing. And then the third one, this, because I'm deliberately trying to choose one that, that, that, that wouldn't be top of Aza's list, even though there's a bunch of these that I agree with, is, is basically, I actually think that the technology platforms are the kind of most important powerpoints in the world. And so part of the reason why I like, you know, at the beginning of this year, I was talking about how, why I wanted AI to be American intelligence, is there's a set of values we aspire to as, as American. I don't know if we're doing that

• 51:23Speaker 01

  good of a job living them, most recently, but we aspire to this, you know, kind of like, hey, let's, let's give, you know, individuals freedom to kind of do great work and, and to have a live and let live, you know, kind of policy when it comes to religious conflict of values and, and other kinds of things. And I think that that, that we want. And I think that actually, in fact, part of the thing that is, we live in a multipolar world now. It's not just a US thing. And so how do we get those values in technology, you know, kind of setting a global standard? And that should be impacting, like, here is one of the things that I kind of, it's a little bit off the FCC FTC question, but like,

• 52:05Speaker 01

  People say, I would like a return to manufacturing industry and jobs in the US. And like, okay, your only possible way of doing that is AI and robotics. So what's your industrial policy there? And they go, oh, really? And you're like, yes, it's a modern world. And so we should be doing that. I agree, but we should be like harnessing this great tech stuff we have with AI and then trying to get that rebuilt would be an excellent, you know, kind of both middle class and also strategic outcome. The country. And that's as a parallel for the kinds of things I'd want, you know, the FTC and FCC to be thinking about as they're setting policies and navigating.

• 52:44Speaker 00

  This gets into like the very specific, but I think it's, it's an interesting example for what social media could be optimizing for. That doesn't require choosing like what's true or not at the content level. And that is a perception gap minimization. That is to say, if you ask Republicans to model Democrats, they have wildly inaccurate models. If you use, right. Say like, what percentage of Democrats think that all police are bad? And Republicans say it's like 85 or 90%. In reality, it's like less than 10% or something like that. And there's a reverse the other way around. So we're modeling each other wrong. And so we're fighting not with the other side, but with our mirage of the other side. So imagine you just trained a model that said, all right, given a set of content,

• 53:26Speaker 00

  is the ability to model all the other sides going up or down? I think if you just optimize for accurately seeing across all divides, which by the way is a totally objective measure. You just ask that group what they believe. You ask other groups what they think that group believes. Then you realize that the most harmful content, hate speech, disinformation, all that, brain rot stuff, that all appreys on a false sense of the other side. So here is an objective way without touching whether content is true or false to massively clean up social media.

• 54:00Speaker 02

  I love it. It goes so much with Reid, what you always say about scorecards. I'm not going to tell you, social media company, that this is good or this is bad, but I'm going to give you the scorecard and what we want you to hit and you, you figure it out. And if you decide that like, oh yeah, actually promoting those vaccine conspiracies makes people distrust the other side in a way that's not accurate. Okay, well then you need to change your behavior. And so again, it's actually sort of putting the agency in the company's hands in a way that is so positive. All right, so we're going to do our traditional rapid fire very soon, but first, we wanted to end on a lighter note because we've talked about vampires and some heavy stuff. So I'm going to ask you guys-

• 54:36Speaker 01

  We need to bring in werewolves and zombies, but you know.

• 54:38Speaker 02

  Yeah, exactly, exactly. I mean, I just watched Sinner, so I do have sort of the supernatural on the mind. So I'm going to get a hot take from each of you, hopefully pretty quick. I have, let me see, four questions. So Azai, we'll start with you. What are the most outdated assumptions that are driving today's AI decisions?

• 55:00Speaker 00

  I think the most outdated belief driving AI is that we can muddle through. That, you know, it's never been a good idea to bet against the Malthusian trap. That is, we've always made it through in the past. And therefore assuming that because we've always made it through in the past, that we'll make it through this time. I don't know what you, Reid, or Aria would give humanity as a scorecard for the Industrial Revolution. I'd sort of say we got like a, maybe a C minus stewarding that technology. Lots of good things came out, but also child labor. And now nowhere on earth is it safe to drink rainwater because of forever chemicals. And we dropped global IQ by a billion points with lead, but we managed to make it through. I don't think we can afford with AI to get a C minus again. I think that turns into a nap for us.

• 55:46Speaker 02

  Reid, what do you think are the most outdated assumptions driving today's AI decisions?

• 55:51Speaker 01

  Maybe a little bit more subtly and geeky. By the way, I do think we need to get a much better grade. Actually, I think AI can help us get a better grade, so. I think so. But I think the most outdated assumption from it, because it's kind of like, it's almost like what, against what most people think. I don't think that people are realizing, people still think it's mostly a data game. And it's turning much more into a compute game. And data still matters, but the, it's like the, you know, data is oil, you know, is the new oil, et cetera, et cetera. It actually computes the new oil. And it's, data still matters, but like, what, what, it's the compute layer that's going to matter the most. I'd say that would be my quick answer in months, in a very complicated set of topics.

• 56:34Speaker 02

  Well, the next question, we're giving you just one sentence to answer. So Reed, I will start with you in one sentence. What is your advice to every AI builder right now?

• 56:44Speaker 01

  Well, have a theory about how it is that you, that in your engagement with your AI product, whether it's chatbot or something else. How it is that you will be elevating the kind of, the agency and the kind of human capabilities, but also broadly compassion, wisdom, et cetera, of the people that you're doing. So for example, at Inflection and Pi and personal logins, it's like, be kind to be modeling a kind interaction is one, you know, kind of very tangible output.

• 57:19Speaker 02

  Fantastic. Aza, do you have one piece of advice?

• 57:22Speaker 00

  I would be very aware of how incentives eat intentions. Because the technology you're creating is incredibly powerful. And so if it gets picked up by a machine or country that you don't like that, their values, the things you invent will be used to undermine the things you actually care most about.

• 57:48Speaker 02

  Fantastic. Reed, I'll go to you first. What is a belief that you hold about AI that you think many of your peers would find controversial?

• 57:58Speaker 01

  Well, a lot of my peers tend to be in the LLM religion, which is, you know, the one model to to make everything work, whether it's super intelligence or the rest. And I obviously think we've done this amazing thing. You know, we've created, we've discovered an amazing spellbook in the world with these LLMs and kind of scaling them.

• 58:17Speaker 01

  I tend to think that there will be multiple models and the actual unlock for AI in the future will be combinations and compute fabric of different kinds of models, not just LLMs. Now, it might be that LLMs are still, as it were, the runner of the compute fabric. It's possible. But I also think it's also possible that it isn't. And that probably gets to the most like, like, you know, like, wait, are you one of those skeptics? Are you, do you not believe all the magic we're doing? It's like, no, I believe there's a lot of magic. I just think that this is kind of a big area and a blind spot.

• 58:50Speaker 02

  Aza, same question. A belief that you have that most of your peers would find controversial.

• 58:56Speaker 00

  That AI based on an objective function are not going to get us to the world we want. That is to say, whenever we just optimize, for an objective function, you end up creating a paperclip maximizer in some domain. But nature doesn't have an objective function. It's a ecosystem that's constantly moving. You're not actually, there isn't just a static landscape that you're optimizing to climb a hill for. The landscape is always moving. It's a much more complex thing. So if we really want to have AIs that can do more than confuse the finger for the moon and then keep giving us fingers, if we actually want to get like the human flourishing, ecosystem flourishing, like that thing,

• 59:42Speaker 00

  we're going to have to move beyond the domain of just AI that optimized an objective function.

• 59:49Speaker 02

  Awesome. Let's move to rapid fire. And Reid, I think your question is the first.

• 59:53Speaker 01

  Indeed. Is there a movie, song, or book that fills you with optimism for the future?

• 1:00:03Speaker 00

  Really Anything by Audrey Tang, listening to her podcast, reading Plurality. She's sort of the Yoda Buddha of technology. So 100% that and then Unhuman Nature by E. Wilson. And finally, Dawn of Everything by David Graeber. Because it just shows that how stuck we are in our current political economic system. And really opens your eyes to how many other ways of being there actually are.

• 1:00:31Speaker 02

  Awesome. What is a question that you wish people would ask you more often?

• 1:00:37Speaker 00

  Oh, I know, something about surfing or yoga.

• 1:00:42Speaker 02

  Awesome. Which are you better at, Aza, surfing or yoga?

• 1:00:46Speaker 00

  I'm definitely better at yoga because surfing is by far the hardest sport that I have ever done. But actually, there is a question that people ask me a lot that I don't have a good answer to. And that is, after sort of like laying out my worldview, people almost inevitably ask, but how do I help? And I realize I don't have a good answer because to answer that question requires understanding who you are, what you're good at, what you would like to be good at, what your resources are, what you're currently working on. And I would love to have an answer that when somebody says, how can I help? There is something, maybe AI can help with it, that does that kind of sorting? And helping people find their Dharma within a larger purpose.

• 1:01:28Speaker 02

  I couldn't agree more. Everyone right now, I forget people who say that everyone's apathetic. Everyone is asking me what they can do right now, is that to your point? And I don't have a good answer either. So let's try to build one.

• 1:01:41Speaker 01

  Well, I think a beginning is learning, getting in the game, right? Like, for example, like start engaging with it and then kind of have your voice be heard. You can't have a perfect plan, but it's like join some movements, rally to the flags that try to help stuff. All right. So where do you see progress or momentum outside of tech that inspires you?

• 1:02:04Speaker 00

  Well, it's gonna feel like a broken record, but.

• 1:02:09Speaker 00

  Outside of tech, actually, I was gonna start with all the deliberative democracy stuff. But we've already sort of talked about that. Blaze, I'm gonna say his last name wrong, Agüero-Racca, at Google. He and his team are doing some incredibly beautiful work that I'm finding a lot of hope in. Because I sort of laid out my worry that game theory is gonna become obligate. And we're just gonna get whatever the game theory says for the future of humanity. And that seems like a really terrible world I don't wanna live in. And his work is on understanding how do you model in a situation of multiple agents? Like, how do you actually get non-Nash equilibrium solutions?

• 1:02:52Speaker 00

  And he's discovering something, which is that in order to solve the very hard problem of how you do strategy and multi-agent reinforcement learning, when I have to model what you know and what you have to model what I know and I have to model what you know about I knowing that you know. And that's just very hard. And they're discovering some new math. And it turns out you can start to answer this if you don't just model with yourself outside the game board, but with yourself on the game board. You have to model yourself modeling other people. And what's cool there is that suddenly, non-Nash equilibrium states are found. Not the worst of the prisoner dilemmas. You can find these new forms of collaboration. And I love this. It feels so profound because first you have to inject the idea of ego and then transcend it.

• 1:03:36Speaker 00

  If you don't have ego, you just find the Nash equilibrium. If you do have ego, you also find the Nash equilibrium. But if you do have ego and you can transcend it, you can get to these much better states. And that, to me, is very hopeful and very cool. Because I think of game theory as sort of like the ultimate thing that we're gonna have to beat as a species.

• 1:03:52Speaker 02

  Always, Aza, our final question. Can you leave us with a final thought on what you think is possible to achieve if everything breaks humanity's way in the next 15 years? And what is our first step to set off in that direction?

• 1:04:05Speaker 00

  This is sort of like the what is possible if we could rearrange our incentives? So we are both nourishing ourselves and nourishing all the things that we depend on. Suddenly, I think people don't really look at their phones because the world that we inhabit is just so rich and interesting and novel. We are consistently surrounded by the people that can help us learn the most, sort of in a developmental sense. The entire world is sort of set up in a fiduciary where everything we can trust is actually acting in our and our communities and our societies' best interest and developmental, understanding

• 1:04:50Speaker 00

  where we are and helping us gain like whatever that new next attainable self is. I think we'll have made a major, major, major progress towards solving diseases. We'll have a deep understanding of cancer. And I think we would have solved our ability to socially coordinate at scale without subjugating individuals. So it looks something like that. We will have solved the aligned collective intelligence problem. And we'd be applying that to like getting to explore the universe.

• 1:05:26Speaker 02

  Awesome. Aza, thank you so much.

• 1:05:30Speaker 01

  Yeah, the universe outside and the universe inside. So

• 1:05:33Speaker 00

  Yes, exactly.

• 1:05:34Speaker 01

  Aza, always a pleasure.

• 1:05:35Speaker 00

  Thank you so much, Reid, so much, Aria.

• 1:05:37Speaker 01

  Possible is produced by Palette Media. It's hosted by Aria Finger and me, Reid Hoffman. Our showrunner is Sean Young. Possible is produced by Tenasi Delos, Katie Sanders, Spencer Strasmore, Imo Zoo, Trent Barboza, and Tafadzwa Nirmarundwe.

• 1:05:53Speaker 02

  Special thanks to Surya, Yala Manchili, Sayida Sabieva, Ian Alice, Greg Beato, Parth Patil, and Ben Ralas.

• 1:06:01Speaker 04

  This podcast is supported by Google. Hey folks, Stephen Johnson here, co-founder of Notebook LM. As an author, I've always been obsessed with how software could help organize ideas and make connections. So we built Notebook LM as an AI-first tool for anyone trying to make sense of complex information. Upload your documents and Notebook LM instantly becomes your personal expert, uncovering insights and helping you brainstorm. Try it at notebooklm.google.com.