What it takes to trust AI | Kanjun Qiu

Before Kanjun Qiu raised $200 million from NVIDIA and others to build reliable AI agents, she was writing high-frequency trading algorithms to pay her way through MIT. Today she leads Imbue, an AI lab unusually focused on power, agency, and what it would take to make AI systems trustworthy by design. In this episode, Reid Hoffman and Aria Finger sit down with Kanjun to explore what it means to truly own your AI tools, not just use them; how to build agents that are systemically trustworthy, not just convincingly so; and why the future of AI may depend as much on moral invention as technical innovation. From meditation and the amygdala to Adam Smith and open-source software, this is a conversation about the deeper architecture of the AI future. For more info on the podcast and transcripts of all the episodes, visit https://www.possible.fm/podcast/

• 0:00Speaker 00

  What would the Ten Commandments for technologists look like? I think it's possible to have a digital world that we feel like we can change it like anything in our house. The only part of it I know what to do about is that I think I want to see if we can make software go from a renter's economy to an owner's economy.

• 0:21Speaker 01

  While studying computer science at MIT, Kanjun Chu paid her way through school by writing high-frequency trading algorithms to trade the stock market. Fast forward to today, and she's raised $200 million from Nvidia and others to build AI systems that can reason and code with access to a GPU cluster that most companies can only dream about. But here's what's fascinating: she's not racing to ship a product.

• 0:47Speaker 02

  Kanjun is building something deeper, what she calls non-leaky abstractions that make AI systems reliable enough to actually delegate work to. She co-founded The Archive, a co-living house for founders and AI researchers, where the conversations weren't about the next funding round, but about the fundamentals of human intelligence that machines still lack. And that philosophy shapes the outlook at MBU.

• 1:11Speaker 01

  We're going in a territory that still too many AI conversations skip. How do we ensure humans and their creative work retain philosophical agency in the AI future? Not just economic power. If we want users to truly own their AI tools, what does that require architecturally? And fundamentally, if everyone had a powerful personal AI agent, how would this change social dynamics and human development?

• 1:38Speaker 02

  Without further ado, welcome to Possible, Kanjun Chu. Kanjun, I am so delighted to be here with you today. And I would say that many of my smartest and most thoughtful friends are very big into meditation, and I would count you among that group. And so as a meditation novice, or I don't even know if you can say novice if you've never done it, as a meditation ignorant, you just came back from a meditation retreat. Tell us about it. How did it change your thinking? How was it?

• 2:09Speaker 00

  Yeah, I did a really wonderful retreat. I've been kind of like poorly meditating for quite a while because being a founder is stressful. And what I learned is that the different styles have a lot of variance in how effective they are for me, and I think personally for other people. And so I just came back from a Mahamudra retreat.

• 2:30Speaker 00

  I would say meditation is probably like one of the most effective ways to see reality clearly and like think clearly because for me what it seems to do, my model of it is that, you know, when we have our amygdala hijacked, we're like triggered, we're in like a threat response, this happens a lot when something bad happens and it's really scary and like things are stressful.

• 2:52Speaker 01

  It's more or less like every day in

• 2:53Speaker 00

  It was like basically all the time.

• 2:55Speaker 01

  Yes, yes.

• 2:56Speaker 00

  When that's the case, it's hard for me and I think humans generally to like see outside the triggered event and so to like kind of put the triggered event in context and not have it just take over our minds. And meditation and this whole idea of resting and awareness and being the field of awareness, kind of what they call it, lets me see the thing that got triggered as like a relatively small thing in my field of awareness. I'm like, okay, that happened, cool. There's all this other information also to consider and so now my brain can like work as an information processing system and weigh information properly as opposed to like weighing all of the fear response super, super strongly, which is like evolutionary adaptive but not that helpful.

• 3:38Speaker 02

  And do you see, you know, you were, you know, you're observing your own thoughts, you're paying attention, you're sort of moving things to the side that would have triggered an outside response. Does it make you think about the current state of building AI and how we think about how AI is thinking and reasoning and all of the things we're trying to get models to do today?

• 3:58Speaker 00

  AIs are a really strange intelligence. My model of how meditation works is that it lets us see the process that is kind of observing everything. So like normally when we're engaged in our day to day lives, we're just kind of like experiencing things. But it's rare for us to like be able to notice and look at the consciousness process that's like noticing all of the sensory experience and like deciding to do stuff with it and like showing thoughts and like kind of seeing that.

• 4:33Speaker 00

  You know, in meditation, they call it awareness. Awareness, I think of as like a substrate that is consciousness. Like consciousness substrate is composed of awareness. And like, it is the thing that generates our perceptual field and our thoughts and things like that. And like, seeing that as the generator, like, suddenly frees us from being kind of pulled into all of these thoughts and like sense of self. And so I think AI systems don't have such a thing. Like, they don't have this like fear trigger response that like suddenly changes the way that they attend to their kind of inputs. And so it makes me wonder, like, I guess, like reinforcement learning from feedback is a form of- Form of meditation?

• 5:19Speaker 00

  It's like a, yeah, Borgt form of meditation that gets something, maybe it's more, hmm. Maybe reinforcement learning from human feedback is more like, uh, kind of Instagram reels. Or Pavlov. Or Pavlov, that's right, that's right.

• 5:36Speaker 01

  We, we lack a bell here to ring. And one of the things that was really awesome, as we just were kicking off this podcast, is realizing that both you and I have done retreats with the same teacher, who is really good. And I was actually thinking about the same question myself, because, you know, you have this question of, okay, so you have an aspiration, but you need to form it into an intention. And then you're kind of reclaiming the conscious agency in terms of what you're doing. And part of the, the meditative practice is try to be self-possessed of your own mind in doing that. And not rely on reflexes, amygdala is one of them, also pre-concepts and everything else.

• 6:16Speaker 01

  So you have a, a kind of a calm focus on the thing you think you should be focused on. Now, like, part of the thing that I think it, it, it indicates is we're at best only partial in the way that the, kind of the training of these systems work is, is how are they, as it were, attending to their own focus as they're doing their own training? Right? So we are providing that focus is one of the reasons why I think most of the thoughtful people say, no, it's not conscious yet, or it's not, doesn't have awareness yet, because it's not providing-

• 6:53Speaker 00

  That's very interesting.

• 6:54Speaker 01

  That kind of feedback.

• 6:55Speaker 00

  It's not attending to its own attention mechanism.

• 6:57Speaker 01

  Exactly. What would it take to be thinking that that was- That is what the AI is doing.

• 7:04Speaker 00

  Mm hmm.

• 7:04Speaker 01

  Now, maybe it's because you could see something in introspection currently in all these AI systems. The introspectibility of these rounds to nothing.

• 7:12Speaker 00

  None.

• 7:14Speaker 01

  Right. But like, what would that be? And then where would you see it? And then how would you reflect it? And it's not as simple as a Turing test.

• 7:20Speaker 00

  Yeah.

• 7:21Speaker 01

  Right?

• 7:21Speaker 00

  Yeah.

• 7:22Speaker 01

  As an instant. Turing test got us a long ways along this journey, but it's like, it's much more. And that kind of awareness of it was one of the things that I thought I was also, perhaps, pun intended, meditating upon.

• 7:34Speaker 00

  That's interesting. Like, in theory, you could, I mean, this probably wouldn't work in training, but like, you could pass it its own attention.

• 7:41Speaker 01

  Yes.

• 7:42Speaker 00

  Like, uh, matrix.

• 7:43Speaker 01

  Yes.

• 7:43Speaker 00

  And then see, like, maybe it can notice that as part, or like, it can have that as part of its, um, input stream.

• 7:50Speaker 01

  Yeah. Where is the self that is observing?

• 7:52Speaker 00

  Where is the self that is observing?

• 7:54Speaker 01

  Yes.

• 7:55Speaker 02

  Should an AI go to therapy and learn that it's too much of a people pleaser? And learn that it needs to like, say no, or stop saying, I'm sorry?

• 8:02Speaker 01

  It's like, in many of these things, it's balance. You want it to be something of a people pleaser.

• 8:07Speaker 02

  Right.

• 8:07Speaker 01

  Because if you have it not.

• 8:08Speaker 02

  Of course, of course, of course.

• 8:09Speaker 00

  If it's not, you know, a secret.

• 8:12Speaker 01

  It's annoying. Most Hollywood films.

• 8:15Speaker 02

  Right, right. That's true. That's what people are scared of. That's what people are scared of.

• 8:18Speaker 01

  Yes, like, like, people pleaser is generally speaking, a good thing. It's just the right kind.

• 8:23Speaker 02

  Yeah, you gotta tune it.

• 8:24Speaker 01

  To your better, to your better health.

• 8:25Speaker 02

  So you are building Imbue, and your focus is on building infrastructure for reliable AI agents. Tell us what Imbue is and what you hope to do.

• 8:35Speaker 00

  Yeah, we started Imbue in 2021, and the purpose was, at that time, it was kind of early, uh, pre ChatGPT, and we really felt like, okay, this generation of models is probably gonna make it all the way, not like the past decade. And in that period of time, the primary discussion around AI was, uh, and the concerns about AI were focused on technical safety. And we really felt like, there's all this conversation about AI and technical safety and the kind of world you want to build, but like, we want to build a world with AI that we want to live in. And magical hand wave abundant future is like, not very concrete. And so we really wanted to think about, what are the core counterfactual turning points where?

• 9:21Speaker 00

  It makes a difference between the default future and a future that we could create, such that if we did those things, then like we'll end up in a better future. Where we really landed was that for us, it was about the decentralization of power and kind of how AI affects people's, like the distribution of power and also people's sense of agency and authorship over their lives. We think there's a real potential that AI would result in a centralization of power, both economically and politically, and that there are some things that we can do right now in the next like four years to shift that dynamic so that the default results in a much more democratic, kind of decentralized future where each

• 10:03Speaker 00

  person's values, judgments, inputs, decisions for their lives can be acted upon, that they can act upon it, as opposed to us living in a more panopticon system that maybe is the default path.

• 10:18Speaker 02

  You started in 2021, which feels like a world away from now. Like you said, ChatGPT hadn't been released, like no, you thought we weren't in an AI winter, but no one was really talking about AI from like a mainstream perspective. What do you think in 2026, like what have you learned over the last five years? Like, were there inflection points that happened as you thought? Like, what is the difference between your thesis now and your thesis in 2021 when you launched?

• 10:42Speaker 00

  To be honest, I think a lot of the things that we predicted would happen have roughly happened roughly on the time scale that we expected. So we expected coding to become super, like the first thing to be automated and to be like a, it's a fully enclosed system that is testable and so perfect for training. When we rolled that out a little bit, we started to imagine, okay, everyone will be able to make software. People will be able to make more complex software. What is that going to look like for the world? The second thing we expected is that AI agents roughly on this timeline would start to become a thing, like these systems will slowly be able to do longer and longer horizon tasks.

• 11:27Speaker 00

  And also, they're going to want your data to be able to do those tasks. And so we saw these kind of like bifurcations in the, in the road. It took maybe a year or two to like really land on a thesis of software and a thesis of agents. The bifurcation for software, I would Think of as, right now, we live in a software environment, a digital environment, that's actually a little bit alarming. Um, like, I, I put my phone in a different room at night because I might accidentally end up on it for, until like 2 a.m. Sure. I don't know what you're talking about. Never happened to me. Never happened to you. I know. It's, it's really a me problem.

• 12:07Speaker 01

  It's an every night problem.

• 12:10Speaker 00

  It's an every night problem. I'm kind of like, you know, if there was literally any other piece of furniture I felt this way about, I would replace that piece of furniture, like my sofa, my bed, but I can't replace my phone. And so why do we feel this way about our phones? It's because our phones have these objects on them, all these apps, built by people whose incentives are not fully aligned with ours. They're not misaligned, but they're not fully aligned. And so it's actually a really good example of what happens when things we depend on that we are really intimate with, you know, that can immediately get our attention throughout the day, every day, we're really intimate with these things. What happens when these things we depend on are built by people whose incentives are not fully aligned with ours?

• 12:55Speaker 00

  There's this divergence over time toward value capture. And with agents, that divergence can be more extreme. It could be possible to have a world where there are just a few agents, like a Facebook agent, and a OpenAI agent, and an Anthropic agent, and maybe 10 more. And so that's not that many. Uh, maybe just like with social media systems, we put all of our data into one or two of these agents, it's hard to get that data out, the agent gets smarter and smarter about each of us because it understands more about us, it's seen more of our data. Um, and because we put in that data one at a time, there's this like slippery slope toward, oh, all of a sudden, these couple of agents have all of my data and have seen everything about me, and so if I switch to a different agent, it's just not going to

• 13:39Speaker 00

  work that well, so I'll just stick with these. And so then you have this lock-in, and then I, as a person, am suddenly stuck. It's not competitive anymore of a market. And if I'm stuck, then it's really easy for me to be kind of extracted from. Again, not like in an evil way, but extracted from because these are profit-seeking companies that operate in an ROI environment. And so that's kind of a future I don't want to end up in.

• 14:08Speaker 01

  When we go through life, we're never fully aligned with anything. We're never fully aligned. Like Ari and I work together a whole bunch. We're not fully aligned. You and I have done a number of these things together. We're not fully aligned. So full alignment, I don't see anywhere in my life. I don't see it with my black Labrador. I don't see it like it's like this whole thing. Where do you see these alignments? And what are the things that you are doing with Imbue to try to have this sufficiency of good alignments and also avoid bad misalignments?

• 14:41Speaker 00

  That's a great question. So I actually agree with you that like nothing is fully aligned. However, the more intimate something is to us, the more aligned we want it to be. Like, we want to be super aligned with our partner. Otherwise, there are problems that arise over time because we kind of build our lives around it. Similarly, I personally think we want to be super aligned with our sources of information and like our information systems. Because the information we receive as a person like shapes our understanding of reality and our model of the world, and that shapes our action. Like, my model of the world and how it works is going to completely affect and change what actions I take in life. And so there's a win which, if the information sources I'm getting are not quite aligned with me

• 15:26Speaker 00

  and they're maybe not quite truthful and they're not quite telling, then I might end up taking, making like really different life choices subtly. And I think people are doing that because of Twitter and social media algorithms and things like that. And so the more intimate something is, the more kind of we interface with it and like take in seriously its information or effect, the more we want it to be aligned. And so your question is like, okay, how do we, like, what's necessary? And how do we end up in a world where it like actually feels really good? And I think it's likely the case that these systems are never like fully aligned with us, but that what we want is, at least in my mind, what I want is for people to have more choice and authorship. And

• 16:12Speaker 00

  More accurate models of reality so that they can make better decisions about their lives. And so what that means is I think we want systems, at least for me, what I want is systems that are open enough that I can inspect them or that people can inspect them. And so I can develop deep trust in them. I have a preference for systems that operate locally on local data and don't send that to a central server. And maybe it's okay for them to send to a central server. That's okay. But like, I wanna have my data. Like, I don't love the fact that ChatGPT stores memories about me that I can't export. I wanna have my data so that I can move to a different system.

• 16:54Speaker 00

  And this is in the kind of policy landscape called data interoperability. And I think what that allows, the ability to move our data around, is it allows for competition. And competition allows for markets that end up being more aligned with us, like products that are more aligned with us, the user. And so what we want, I think, is like a lot of competition and a really good ability to move between different agents and like have open versions, have versions that can be like inspected fully and explained, et cetera. And that will let us at least like get to more aligned versions of these systems and that will be better.

• 17:34Speaker 02

  It's funny, in most simplistic terms, this reminds me, and candidly, I don't even know what year it was, and I don't even know who did it, but you used to not be able to change your cell phone number. If you went from Verizon to AT&T, you lost your number. And that was like a huge issue for switching costs and people were locked in. And it feels like, again, that's just like the the most basic form of like, can you take your data with you? And it was actually really good when the government said like, no, you can take your phone number when you go to a different service. So that's super interesting. And I feel like a lot of what you're saying, and please correct me if this is not what you're saying, people talk about LLMs right now as a black box. It's like, how can I trust it if I don't know how it came here? I can't inspect it. I can't see the lines of code that got us here. And so there's no way to build that trust if I don't know what's going on.

• 18:18Speaker 02

  And so it seems like what you wanna do is let's build these agents that You can trust them. They can reason. They can code. They are, they're reliable. They're going to do what you think they're going to do at the right time. And so that's why they can become, you know, the software engineer of the future. And so is that what you think is the best case scenario? Getting rid of the black box, having them as reliable, and then we can truly have them as coworkers, as agents, and things to work with, you know, on a daily basis?

• 18:46Speaker 00

  Yeah, I think that what's going to happen is that we will have lots of agents either working for us or representing us in many interactions. And so as a result, probably agents will be making a bunch of decisions. And I think of agents as actually not a special thing. Agents are just kind of like fancy software. And we also want to, you know, right now, we don't necessarily live in a world like this, but ideally, we also want software to be fairly understandable. Like, I want to know what, you know, why is Twitter giving me all of these really annoying tweets? Can I change that? Like, I would really love to be able to change that. Why is it making these decisions? And so I actually would love a world where, like, yeah, all of the software in my life is understandable and explicable,

• 19:30Speaker 00

  which I think we're very quickly barreling towards, you know, Claude Code, writing entire code bases, being able to like assess code bases and then answer questions. I want to know what decisions are they making and why do they result in those decisions? And I want to know if an agent is representing me and like kind of making decisions on my behalf, I want to have a fair amount of control over those decisions and I want to have a fair amount of trust, that I can trust this agent to make those decisions. And I want that trust in a deep way and I want everyone else to be able to trust in a deep way. Not just like, oh, I've been fooled into trusting this as a consumer who's not very well informed. But rather like, no, deeply, like, the system is set up such that it is deeply trustworthy.

• 20:15Speaker 00

  So that might mean something like, you know, NMB were really big proponents of open source. A lot of the stuff we're shipping this year is going to be open source. Or what we call common source. This idea that You know, in the future, we may actually want a lot of our software to share common source code that's open, that we then remix for ourselves or for our, you know, groups of each other. I think this is very possible with what's becoming possible with coding agents. So, open source is an example of a more systemic trust, where if a system is open source like Linux, people like, we can use collective intelligence, like, various people will find bugs in Linux and find security vulnerabilities. And as a result, you can trust open source software more because of how it's set up, not because someone

• 20:58Speaker 00

  kind of like convinced you that they were a trustworthy entity. And I think it's important to think about agents at this like systemic level. How might we systemically set up these agents so that they can actually be inherently trustworthy, not just because a company maybe is incentivized to make them trustworthy so you keep using it?

• 21:18Speaker 01

  Most technologists tend towards an individual's libertarian, like, how do I validate? But the point is, is of the, you know, over eight billion people, very few even have the conceptual infrastructure, let alone time, resources, etc., that I validate. So the I validate, like, basically doesn't really work. But the we validate and experts validate, or commonly validate it, e.g., lots of people can look at this and we have a way of surfacing, oh, there's an issue here. But part of the reason why we have second opinions, we have doctors talk to each other, is they go, well, okay, are we functioning

• 21:58Speaker 01

  this well? Now, what's your thought about how this becomes part of the AI landscape?

• 22:05Speaker 00

  I love how you put that, that individual libertarianism doesn't fully work and that we need like collective forms of validation. So-

• 22:15Speaker 01

  That was implying your point, but yeah.

• 22:17Speaker 00

  It was a really good, like, synthesis and actually captures some of my issue with some of how tech is thinking about these problems and these questions, like, AI is a very societal problem, question.

• 22:32Speaker 00

  So how do we, I think your question is, how do we build systems that, like, what is needed to have systems that can be collectively validated and trustworthy? Some of the things that we're doing at Embu are, there's work around openness. So we think of our mission as democratize power and do that by helping to facilitate an open ecosystem of software and agents and data. And so there's an openness piece, and there's a kind of understandability piece, and there's a kind of like control-modifiability piece. And openness enables both understandability and control, but you know, you also had, have to have additional tools on top of the openness, it just doesn't give those things by default.

• 23:15Speaker 00

  On the openness piece, we are trying to figure out, and like, hopefully, other people will do this as well, trying to figure out, how do we cultivate a good ecosystem around open models, open software, open agents, and maybe try to overcome the fact that like traditionally open source businesses have been difficult businesses? I think that's one of the great deterrents in our current incentive structure of making things open. However, I also think, because software is basically becoming free to build, that there's a great economic pressure toward openness. Recently, I was talking to someone, he built a new product in four days, he was like, "I

• 23:57Speaker 00

  can't really justify charging for this, so I think I'm just gonna open source it, because someone else is gonna replicate it." So I think there are great forces toward openness, and from our perspective, there's some additional work needed to like, maybe enable business models around open source. There are a lot of open source licenses where you can't really discriminate between personal use and company use, and so maybe that's an easier license, or better norms, or something like that. On explainability, some of our work at Embu is about, how do you Like, is it possible to ship software that can explain itself to you? Like, yeah, it seems very feasible. Maybe all software ships with something that explains itself to you. And like, maybe, and then on modifiability.

• 24:38Speaker 00

  On explainability, there is other pieces that we're doing where we have a lab called Feature Lab that is doing foundational deep learning theory. And that theory is really aimed at understanding, can we make these systems fully engineerable rather than, you know, kind of probabilistic in the way that it is? Is there like a mathematical foundation for deep learning that can allow that? And if that is possible, then it's potentially possible to create these systems with a lot less resources. And so that would allow more proliferation of models and like base models and also kind of like the training and guidance needed for them to like be good at instruction following, et cetera.

• 25:20Speaker 00

  Then on modifiability, I think we're just gonna kind of mostly get that for free. Like, people wanna modify software. You see a lot of people already like making a ton of stuff. And so I think we're gonna get that for free. There's one more piece on openness, which is about data. So we talk about open software, open agents, and open data. And I think for data, there's like a, you know, I talked about local data, how like agents that have access to our local file system are actually really powerful and kind of like can work with our local data, like Claude Code. We, our Codex CLI, like, we really like these local models. And I actually think we might see, like, Inview is going to invest in an ecosystem of more local tooling that can work with your local data, maybe work with data in your Dropbox. So it's like backed up, but it's still yours.

• 26:05Speaker 00

  And so I think better supported ecosystem around things that like work locally on your computer, agents that work locally on your computer and don't necessarily, like, they're not kind of black boxes in the cloud that you can't really change. I think that's really interesting. And then the second piece around data, so the local data is maybe more private or stuff I wanna work with. There's a second piece around data, which is about public data. So Earlier I mentioned that quality information really shapes our sense of reality and what decisions we make. And there's a lot of public data out there right now that maybe like is not necessarily the information that we'd want to see. And we also lose a lot of data because it's like stored in a server somewhere. But I think there's something really interesting in kind of like a public data backbone for

• 26:49Speaker 00

  AI agents to work with the internet. And I think Blue Sky and their AT protocol is an interesting kind of first take at this, where basically the way it works is all of the posts on Blue Sky are in this public data repository and you as a developer can make any interface you want. And so now you can kind of remix like different pieces of software on top of this public data. And I think that we, there's an opportunity to have a lot more like public commons data, data as a public good that maybe hearkens back to like the old internet, 90s internet, and maybe away from this like web 2.0 centralization and private servers. So all of that to answer like, those are some components of things that I could see

• 27:32Speaker 00

  helping us become systemically more set up to have trustworthy agents that work for us.

• 27:38Speaker 02

  I love that you're bringing up public data. Just, I'm like real deep on the YIMBY Twitter threads. And especially in New York, there's like big questions about cars versus city bikes. And so someone was spouting some nonsense about city bike and someone immediately was like, well, I can access the data in New York City on every single city bike station. How many bikes were dropped off? How many bikes were picked up? Oh, someone left that station at 2:59 and I know that 50 people used it that day and when people park, it's only five. And so one of the things I'm excited about for public data is obviously the vast majority of people are going to do nothing, are not going to build an agent, are not going to look at public data. But if we can get, like, first of all, there's so much government data out there that someone

• 28:21Speaker 02

  who now can utilize that can do amazing things with, you know, city bike and more. And there's even more that we should categorize, like with our new abilities, like we could supercharge so many government services if we actually had the data instead of it living on. PDFs in like the, you know, middle of DHS files that no one ever sees. If we can bring it to light, like, we can do so much more.

• 28:46Speaker 00

  Yeah, I think this is super interesting. One example recently is the UK added a tax on pubs or something like that. And somebody made a map of all pubs that would be affected. And that helps all of these local business owners figure out, like, how does this tax affect me, et cetera. Similarly, we had the thought when Trump defunded a lot of projects. Oh, maybe it would be helpful to, like, make a list of all these projects and, like, do that programmatically and make it so that it would be easy for people to collectively fund them. I think there's something really interesting here about data and specifically what you said about kind of a person being able to correct the, like, public discourse.

• 29:28Speaker 02

  Absolutely.

• 29:28Speaker 00

  Because, yeah, in the theme of this podcast possible, like, right now, we have an internet that has a lot of fake news on it and, like, false information. And in theory, with language models, we should be able to, with language models and a better public data record system, question mark, like, we should be able to have a system that tends toward truth and tends toward people able to, being able to have more truthful understanding of reality. And so that's, like, starting to become possible, and I'm, like, super excited about that path.

• 30:00Speaker 01

  Well, I think part of what you need, and I agree that, like, lots of data being more publicly and generally available is good. As you just mentioned, there's a lot of that which is not, is worse than garbage on the internet because it's deliberately misleading.

• 30:18Speaker 02

  Right, garbage is just neutral.

• 30:20Speaker 01

  Exactly. So a lot of the anti-vax stuff is in this, is in this sector. But another thing that is important is how do you weave the logic of understanding this data together? In addition to the Data layer for making a more humanist set of things. What are the things that you think need to be done on the reasoning side and on the the understanding of the quality of data side?

• 30:48Speaker 00

  Yeah, this is a great question and is actually something that if somebody has a project that is trying to work on like excavating truth from a lot of data streams, I would love to fund that project or have them work on it and imbue like I I am not sure what the answer is. I think there need to be really thoughtfully designed, like a really thoughtfully designed layer kind of on top of the internet. An example of a problem that cropped up recently is there were some fake chat GPT-generated news that was published and then another model cited that like something else cited that news and something else cited that news and then now became real and then the models thought they were real. And you're like, OK, that that's not good. Like, we need some

• 31:33Speaker 00

  way to like deeply investigate this whole chain of source citations and to understand like, is it actually real? And so maybe that looks like agents that are investigative agents that then report their findings, just like a scientific meta review into some central database, like, I guess decentralized, but like central, central database.

• 31:52Speaker 01

  Essentially accessed, even if decentralized, in terms where all the data is and the links between.

• 31:57Speaker 00

  That's right. And so, yeah, maybe that's what that looks like. I'm not quite sure what works, but there's something interesting in the scientific kind of process.

• 32:06Speaker 01

  Yes.

• 32:06Speaker 00

  That's been developed. And I think these are actually like social process innovations plus technology to support those social processes. Not quite sure what it looks like yet.

• 32:15Speaker 01

  One of the frequent things that I find challenging and, you know, RFK Junior is a poster child for this, which is like people misunderstand to say, well, science, you know, that that broke there. And you're like, yes, science is a continually learning mechanism.

• 32:31Speaker 00

  That's right.

• 32:31Speaker 01

  With reproducibility. The answer is not it's always right. The answer is it's on a trend towards being right in the process. Because like, most people are not good at understanding statistical argumentation. What are the ways that we get to common knowledge?

• 32:46Speaker 00

  Mm hmm.

• 32:46Speaker 01

  Like common sourcing. What are some of the ways that if we're steering AI systems to get to this? And I deliberately use the term possible future. What are some of the things you think we need to be doing as technologists and a society in order to get there?

• 33:02Speaker 00

  Yeah, on the like possible futures, I think this is one of the most, my most optimistic views of what progress looks like with AI is that AI, by its nature, synthesizes a lot of conflicting information to present to us what it seems to understand as the like, what its best understanding of the truth is, if it's trained well.

• 33:24Speaker 00

  I actually have a maybe hot take here where like, I think one of the most important things, I think there's like a systems thing and then there's a moral thing. So there's a systems thing, which is like Anthropic, making AI systems that are truthful, as truthful as possible. And I think Anthropic culturally really cares about that and like, I love that they exist. I think there's a moral thing, which is as technologists, we build incredibly powerful things that change people's lives and affect their day to day. But we don't have a variant of the like Hippocratic oath. And like something I've been kind of noodling on is like, what would the 10 commandments for technologists look like? One example is, I've been thinking about why some things feel more fake than others.

• 34:07Speaker 00

  For example, like if we're scrolling Instagram at 3am, that feels fake, but like watching a great movie until 3am feels kind of more meaningful, more real. Or if we're eating processed food, it feels kind of fake, but like if we're eating really nutritious food, it feels like more real. Or if we see someone beautiful in a beauty magazine, but they've been photoshopped, that feels fake. But like in real life, if they're like really vibrant, that feels real. What's going on? And I think there's this thing that can happen to humans as we get more powerful technology that I'm calling sensory hijacking.

• 34:42Speaker 00

  So technology hijacks these sensors that we have to determine actually how to live really meaningful and healthy lives. We have really good sensors for that, but they're very, very hackable. And so these systems that capture our attention, hijack our attention, are hijacking some of these sensors that exist for that, you know, that determining a meaningful life. And so like, one example of a commandment might be like, thou shalt not hijack other people's sensors.

• 35:10Speaker 00

  Another one might be like thou shall not build systems that propagate false, you know, reality. Or like, that's another kind of sensory hijacking. You're like giving people wrong information on purpose.

• 35:21Speaker 02

  When the rubber hits the road, how do you make this happen? Is it a culture change? Is it government intervention? Is it all of the heads of the AI labs getting together and being in conversation with each other? I think so many people, sort of in this room certainly, but in our position, want the better future. They want to drive towards these sort of humanistic goals. But we don't want to stop technological progress because we see how it makes people's lives better in so many ways. There's also geopolitical implications. But I think people are still struggling for then, like, how do we cross that chasm? Because you're talking about some of the harms, but how do we get to the other side?

• 36:00Speaker 00

  Yeah, I think some of the things I talked about, like, I think it requires a lot of invention, actually. Some invention and like some kind of like internalization of morality and like excavation of what is moral technology. Invention at the systems level, you know, how do we invent systems that actually do tend toward truth? How do we invent systems that allow for more openness and trustworthiness, like software systems, digital systems? How do we invent economic systems that allow for more agency and more authorship, especially as, you know, AI becomes more powerful and is able to do more of people's work? Like, this is an open question I have. I don't, I don't really know how that's going to shake out.

• 36:43Speaker 00

  So there's, I think, a lot of invention required over the next few decades to get here. And I think it's, if I knew what exactly to invent, I would have done it already.

• 36:55Speaker 01

  Yeah. Well, it's a process.

• 36:56Speaker 00

  It's a process. That's right. It's a lot of what we're trying to figure out at InBion, why we're such a strange company, is like trying to invent these systems that allow for a future that is like systemically more robust and more trustworthy and more agentic and not just, you know, not fragile.

• 37:15Speaker 01

  Of technologists, you are at the far edge of philosophically sophisticated, right? There are many technologists who are not. Coming to a A growing common perspective on morality is also, just like common knowledge or common source, there's kind of like the, okay, how do we get to that that common morality and how does AI play into that?

• 37:40Speaker 00

  It feels like as humans, we're always trying to figure out what leads to a more like meaningful life, like why do we exist? What is a good life with the amount of time we have on this planet? And like, some of us figure it out early, like you. Some of us figure it out late, like me.

• 38:00Speaker 01

  I'm not quite sure of this, but that's okay.

• 38:03Speaker 00

  It's taking me a long time to be like, oh, finally, I can do the things I want. I don't just have to do what people expect of me. There's something really strange and interesting about

• 38:14Speaker 00

  meaning, agency slash authorship. I've been using this term authorship and

• 38:21Speaker 00

  being controlled. There's like some dynamic here. I'm going to try to point at it. I think we are controlled often by the world that we live in. Not necessarily badly. We don't murder other people in this form of being controlled. I think we do internalize a lot of maybe like, I'm going to use this term, it's not quite right, oppressive or controlling narratives. And we do exist in some kind of controlling systems, like my phone is a controlling system. And so I think if I were to answer like, what is human good?

• 39:05Speaker 00

  Or like, how do we build a positive future for humanity? It's something like, we want a society that's able to continue to make progress on how like what a meaningful life is. I think there is real kind of like thinking and quote unquote, progress to be made there. I think we understand a lot more about it today because we understand our world and ourselves and our mind and like. Are like, what makes us human and what, you know, why is that valuable? So I think we want a society and systems that like let us continue to make that progress. And I, at the same time, so that's at the like, social level, collective level. And then at the same time, at the individual level, that like protects us from negative things.

• 39:47Speaker 00

  Negative, that's like negative liberty and also kind of like induces positive liberty, positive freedom, like induces agency and sense of authorship. And, and like protects us from things, systems that might control us or tell us false things. And so the combination of these two, I think, in the context of technology is some of what we've talked about today, but not comprehensive. Like, in the context of technology, being protected from some of the negative control might be like being protected from a system that might manipulate us, being protected from easily ingesting false information, being able to trust agents that operate on our behalf because they're systemically trustworthy. Those are like protective stances. And then there's like a positive freedom aspect of it.

• 40:28Speaker 00

  A lot of what you do, read, I think, to kind of like give people examples of what's possible. They, like, it's really helpful to see what somebody else did. Like, when you wrote this book, Super Agency, I was like, oh, my God, I can do this. Whoa, that's so cool. Like, and it like actually unlocked a lot of writing for me. That was really cool. Great. Thank you. That is great. Yeah. Yes. Um, so like more examples of people like doing stuff that's possible. And then also updating some of our social narratives that kind of keep us operating in like, oh, we're just this way or that, just that way and business as usual. My co-founder has a really interesting thought experiment, which is, let's imagine

• 41:08Speaker 00

  that 10,000 years from now, there are some human descendants who live on a faraway planet orbiting a faraway star, and they see Earth through a telescope. In the case where they have forgotten where they came from, Earth is just another planet. It has no meaning to them. And that's actually really interesting. In the case where they have remembered where they come from, Earth has this deep meaning to them. And so it actually says something interesting about meaning. Like, what is meaning to people? There is something in this thought experiment that I don't quite understand that says something about meaning.

• 41:45Speaker 02

  Well, I was just thinking about a hypothetical where you had your personal AI agent, and we've solved the misalignment. We've solved the trust issue. Like, you trust it. It has your, you know, best interests at heart. And so it, you know, if you're feeling down, it knows you. So maybe it tells you about meditation, or maybe it tells you, like, go to church this Sunday, or maybe it tells you to get a new job that has more purpose. You know, there's some universal ones. And, you know, some, go get your flu shot. It's going to be a really bad flu season. And so, like, it does all the right things. And so I guess I have two questions. One is, like, is this the utopia, where everyone has this personal AI agent that we trust and is aligned, and we are on board? And then you made me think, how is that?

• 42:29Speaker 02

  Like, the good news is that helping others and having a purpose is actually good for you. But is there some misalignment because it's good for you, but it's not necessarily looking out for society? Is there a trade-off between the personal and the collective if we have this magical AI agent?

• 42:46Speaker 00

  I think this is a really interesting question and kind of speaks to the amount of power someone who trains these models has, which is, you know, you can think of an agent that has both your values and societal values at heart. And maybe the agent that has societal values thinks, okay, what is best for society? Well, if you buy an SUV, here are the trade-offs for society. However, you value freedom and personal liberation, and so maybe you want an SUV. So an agent that really strongly considers both might say, you might want an SUV, but know that the trade-off is that you're making things worse for other people. However, an agent that doesn't consider that and only considers your values might be like, you should get an SUV. Like, this is best for you.

• 43:28Speaker 00

  And so I think I'm confused about like, I personally would prefer the agent that informs me of the trade offs of my decisions. I think that's good and kind of like helps me become more collectively oriented. But not all agents are going to do that.

• 43:44Speaker 01

  Well, and part of the challenge when it gets to human nature is that many human beings prefer certain kinds of self deception.

• 43:52Speaker 00

  Mm hmm. Yeah.

• 43:53Speaker 01

  They prefer to, as opposed to saying, hey, when I get this SUV, I'm actually having the following kinds of impact on the world. They go, no, no, no, that's

• 44:03Speaker 00

  I don't want to know that. I'm not going to buy this, you know?

• 44:06Speaker 02

  I'm just being a good parent. I'm keeping my kids safe.

• 44:08Speaker 00

  Exactly.

• 44:09Speaker 02

  And my first identity is as a protector of my children.

• 44:12Speaker 00

  Exactly.

• 44:12Speaker 02

  And so it's really easy to just say, it's totally fine. And it's what I should do as a moral human.

• 44:18Speaker 00

  Exactly.

• 44:18Speaker 01

  Because my hope is that with AI and its ability is a, is the best learning tool that we've constructed in human history, is that helps us all become somewhat more truth seekers.

• 44:31Speaker 00

  Yes.

• 44:31Speaker 01

  It's a definite hope. It's a definite trying to steer in that direction, but it's a very complicated path.

• 44:37Speaker 00

  Yeah, that's my hope also. Like, you know, I think maybe that a lot of us lose faith in education because we're not really met. We're either overly bored or overly stimulated. It's like too much on either side. And like, these systems can meet us where we are. So maybe that will help us get closer to being truth seeking. I think humans do want to, at our nature, understand what's going on. And that like, we only lose touch with that when we're really distracted by all this stimulus. Or like distracted by hurts, old hurts. And so maybe like, you know, part of it is like AI being better tutors, helping us understand things better. Maybe like, part of it is like, everyone meditates with

• 45:21Speaker 00

  this new method that's really efficient. And like, we'll all see the true nature of our experience of, as like a field of compassion and love. And then that will solve all the world's problems. I think it'll be like a slow progression toward something like that, hopefully.

• 45:36Speaker 01

  And just so that our listeners don't think it's unduly optimistic or Pollyannaish, the answer you just gave actually in fact is deeply within the Western philosophical tradition because it's Socrates. And it's deeply within the Eastern tradition because it's Buddha.

• 45:52Speaker 00

  It's true.

• 45:53Speaker 01

  And both of those two figures would say the thing you said was exactly right. And we just need to help people on the dialectic towards it. Now, I think there's complexities to it.

• 46:03Speaker 00

  Yes.

• 46:03Speaker 01

  But it's a good, it's not, it's not an unconsidered objective that actually in fact leads us into better places.

• 46:11Speaker 00

  Yes, I love that. Yeah, we have an opportunity to kind of really lean into Socratic methods, Socrates, and really lean into like what Buddha discovered about the fundamental human experience. And maybe that would teach us something.

• 46:23Speaker 02

  So we've been talking a lot about sort of philosophical power in the future. And how do we point to this humanistic future? But economic power is going to be critically important. And how to creative people retain those economic rights? How do people even have jobs in the future? And so what do you think needs to change about ownership and monetization in the future? As we think about AI, you run a research lab and you need some feedback signals. And so what are the, what are the things that we should use day to day? As we sort of hurdle towards this future when we think about the economic piece of it?

• 46:57Speaker 00

  I think I personally am most confused about the economic aspect of this, of this entire situation. I have this model of capitalism that I'm not sure is correct. But it's something like there are two core mechanisms. One core mechanism is the markets, which is like a coordination and resource allocation mechanism. And the markets are amazing because they allow people who are like more self-interested, like they tap into human self-interest and allow that self-interest to be super beneficial to other people. And then there's a second mechanism, which is like the idea of return on capital. And kind of being able to invest in something and get more returns on capital. And I think when people talk about capitalism, they often combine these two mechanisms. They're like, oh, capitalism is like this whole thing that like, you know, people invest and like get money back and like.

• 47:42Speaker 00

  But reading Adam Smith, Adam Smith is like only talking about the first mechanism. And he's actually very concerned about like monopolies and rent seeking and the the kind of perpetual growth machine that we currently live in. I think that the perpetual growth machine, combined with AI that's really, really smart, causes serious problems. I could see AIs that do a lot of kind of hijacking or hacking of our existing system to accumulate capital or resources. So like people who get really good at high frequency trading. I was a high frequency trader a long time ago. And like AIs can definitely do this well. But is high frequency trading actually connected to real production, like real production value?

• 48:26Speaker 00

  And that was one of Adam Smith's other concerns is like financial returns that are disconnected from real value of production. And so I think we currently live in this economy where like a lot of our financial returns are disconnected from real production. It's like kind of hard to evaluate exactly how or in what ways they're disconnected. And so for me, the only part of it I know what to do about is that I think I want to see if we can make software go from a renter's economy to an owner's economy. Right now, we're in a renter's economy of software. Pretty soon, I think we can be in an owner's economy of software. Seems possible, seems like, you know, the production costs are getting really low.

• 49:07Speaker 00

  And so that'll change some things, but I'm not sure it'll address much of the problem. You know, I don't know how to think about the rest of like, yeah, how do people get paid? Who gets paid for what? Who makes the money? Where does the money go? Does the money accumulate? Does it distribute for some other reason? Like maybe people are super creative and then somehow we end up with like Substack everywhere for everything.

• 49:33Speaker 01

  So now a rapid fire, which is, is there a movie, song, or book that fills you with optimism for the future?

• 49:40Speaker 00

  I love The Beginning of Infinity by David Deutsch. I think it is one of the deepest excavations of what

• 49:50Speaker 00

  Progress is that I've ever read. I read it in 2020 and before that I was like, I don't know if progress is real. There's this kind of meme among historians that history is cyclical. What's the point? What is progress? I can see that our standard of living has improved. Is that progress? It seems like progress, but like, what does it actually mean? The idea he puts forward is that the construction and layering on of good explanations is what results in progress over time. And so a good explanation is an explanation that explains the phenomenon and has reach. Like, it can also explain other phenomena. For example, he gives a really interesting example, like, the explanation that the sun

• 50:34Speaker 00

  rises because Helios pulls it in a chariot every day is like, so his first criterion is that it's hard to vary. Well, you can vary a lot of things about Helios. His hair color, his eye color, how tall he is, how, what the chariot looks like. And the second thing is that it doesn't have any reach, like, it can't explain anything else in the world. Whereas the like axis tilt theory of the sun, explaining why the sun is where it is in the sky, has a lot, it's very hard to vary, and it has a lot of reach. It also explains seasons, and it also explains like why seasons are different on different hemispheres. So, the idea that the kind of layering of good explanations is a sign of progress, I think is actually the most convincing argument I've heard for what progress is.

• 51:18Speaker 00

  And, and this book fills me with optimism. You know, even if we fulfill Maslow's hierarchy, like, what cultural values make sense? And like, how, how do you, there must be some like more fundamental underlying explanation about why love and like, kind of like altruism is better than hate and anger? But like, what is that explanation? And like, I think his argument would be, the excavation of these good explanations is what leads us to these progressions in moral philosophy and in economics and these other things.

• 51:50Speaker 02

  What is a question that you wish people would ask you more often?

• 51:54Speaker 00

  I think a thing I don't talk very much about because I'm like an AI person, quote unquote.

• 52:00Speaker 00

  They don't have very many opportunities to talk about it is I actually have systematically removed my anxiety over the past 10 years. And like I used to be a super anxious person and being a founder was really hard. And like I had all these voices in my head and they were really critical. I have Asian parents and Asian culture and like very critical voices in my head from my childhood. And I think people assume that anxiety is just something you have to deal with and the voices in your head are just like something you have to learn to befriend, but it's false. I think that trauma is basically or like these voices are basically overfit models. And as we grow up, our data distribution changes and what we have to do is go access those models and like systematically update them by giving them new data about our lives. And so that's what I've done with all of these different modalities and it's like really effective.

• 52:42Speaker 02

  Do you have advice for someone who's experiencing that and wants to fix it?

• 52:47Speaker 01

  Say?

• 52:48Speaker 02

  No, no.

• 52:50Speaker 02

  I don't have anxiety, so I don't know about it, but I know it's crippling for people.

• 52:54Speaker 00

  Asking for a friend.

• 52:56Speaker 02

  Truly.

• 52:56Speaker 00

  Truly, you don't have anxiety.

• 52:58Speaker 02

  No, none.

• 52:58Speaker 00

  That's amazing and unsurprising. That makes so much sense.

• 53:04Speaker 01

  All right, so where do you see progress or momentum outside of AI that inspires you?

• 53:10Speaker 00

  One of my best friends works in longevity and I think like the medical sciences and our like what AI is doing for and not even AI, like outside of AI, what we're starting to learn about how our bodies work and like turning things like drug discovery or vitrification of organs into like freezing of organs so that they can be preserved for organ donation, like turning them into engineering problems is just like super interesting. And I'm really excited for the next five to 10 years. I think we'll learn a lot here.

• 53:44Speaker 02

  Can you leave us with a final thought on what you think is possible to achieve in the next 15 years if everything breaks humanity's way? And what's the first step to get there?

• 53:53Speaker 00

  I think it's possible to have a digital world that we feel like we can.

• 54:03Speaker 00

  Change it like anything in our house, or anything in our lives, or anything in our kitchen. Like, maybe our digital world feels more like our kitchen, where we like, know how to make food and have the tools for it. We can rearrange our kitchen, we can design it exactly how we want it to. This idea is not unique, it's from Glenn McDonald on our team. I think it's very possible for all of us to feel empowered in the digital world, in the way we feel empowered in our kitchens, and for that environment to kind of nourish us the way our kitchens can nourish us.

• 54:37Speaker 01

  As always, with great conversation, Kanjun. I have questions for another 10 hours, so one hour leads to 10, but I look forward to the next one.

• 54:46Speaker 00

  Me too. Thank you both.

• 54:48Speaker 02

  Thanks so much.

• 54:48Speaker 00

  This was really fun.

• 54:50Speaker 01

  Possible is produced by Palette Media. It's hosted by Ari Finger and me, Reid Hoffman. Our showrunner is Sean Young. Possible is produced by Tenasi Delos, Katie Sanders, Spencer Strasmore, Imo Zoo, Aman Suri, Lexi Kevin, Danny Garrison, Trent Barboza, and Tafadzwa Nirmundwe.

• 55:08Speaker 02

  Special thanks to Surya, Yala Manchili, Sayida Sabieva, Ian Alice, Greg Beato, Parth Patil, and Ben Relis. And a big thanks to James Campsey, Brandy Hagel, Ektu Batisha, Eric Pavia, Ben Casnoca, and the Village Global team.