A Positive Vision for the Future: Part 2 with Illia Polosukhin of NEAR

In part two of his conversation, Illia Polosukhin, co-author of "Attention Is All You Need" and founder of NEAR Protocol, explores the profound societal and economic implications of AI. He envisions a future where a unified intelligence layer transforms personal computing, AI agents enable direct market connections, and individuals find meaning in niche communities amidst AI-driven abundance. Illia also delves into the ethics of agent-to-agent interactions, where AIs prioritize human interests, and new governance models utilizing AI delegates. This episode offers a concrete, systems-level vision for how AI will reshape our world, from daily life to global governance. Sponsors: Google Gemini Notebook LM: Notebook LM is an AI-first tool that helps you make sense of complex information. Upload your documents and it instantly becomes a personal expert, helping you uncover insights and brainstorm new ideas at https://notebooklm.google.com Tasklet: Tasklet is an AI agent that automates your work 24/7; just describe what you want in plain English and it gets the job done. Try it for free and use code COGREV for 50% off your first month at https://tasklet.ai Linear: Linear is the system for modern product development. Nearly every AI company you've heard of is using Linear to build products. Get 6 months of Linear Business for free at: https://linear.app/tcr Shopify: Shopify powers millions of businesses worldwide, handling 10% of U. S. e-commerce. With hundreds of templates, AI tools for product descriptions, and seamless marketing campaign creation, it's like having a design studio and marketing team in one. Start your $1/month trial today at https://shopify.com/cognitive PRODUCED BY: https://aipodcast.ing CHAPTERS: (00:00) Sponsor: Google Gemini Notebook LM (00:31) About the Episode (03:33) AI Transforms Software Development (14:18) The Future of Work (18:58) Securing Blockchain with AI (Part 1) (19:08) Sponsors: Tasklet | Linear (21:48) Securing Blockchain with AI (Part 2) (33:03) Vision for an AI Society (Part 1) (33:55) Sponsor: Shopify (35:52) Vision for an AI Society (Part 2) (49:14) Agent Architecture and Alignment (58:30) Experimenting with AI Governance (01:06:43) AI Safety and Robustness (01:16:09) Bio-Security and Open Models (01:22:56) Coordinating AI Development (01:28:52) Outro SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://linkedin.com/in/nathanlabenz/ Youtube: https://youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:01Nathan Labenz?

  Hello and welcome back to the Cognitive Revolution. Today, I'm excited to share part two of my conversation with Ilya Polosukhin, co-author of the seminal "Attention is All You Need" paper, and founder of NEAR Protocol, which describes itself as "the blockchain for AI" and aims to build a future where AI belongs to everyone. In part one, we explored the foundational technology stack: proof-of-state consensus mechanisms, confidential computing on GPUs, and NEAR's plan to train frontier models on a decentralized basis, with contributions incentivized by cryptographically guaranteed revenue shares. Today, we zoom out from the enabling technology and discuss the big-picture questions. What does personal computing look like

• 0:45Nathan Labenz?

  as AI continues to eat traditional UI-centric software? Ilya envisions a unified intelligence layer that works across all of your devices, predicts what you need, and proactively gets things done for you. How will this change the nature of markets, trade, and the consumer economy? Ilya expects that advertising and middlemen will become less relevant as AI agents allow buyers and sellers to connect directly at unprecedented scale. What does daily life look like? And how will we spend our time in the era of AI-provided abundance? In addition to a rise in personalized AI-generated entertainment products, Ilya predicts that people will find meaning and compete for status in small communities focused on niche interests, including athletics, video games, the collection and curation of unique artifacts,

• 1:29Nathan Labenz?

  and who knows what else? What rules should govern agent-to-agent interactions in this world? Ilya imagines a symbiotic relationship between individual humans and their own personal AIs, such that humans and AIs effectively grow up together. And he believes that AIs should pursue the interests of their individual humans, even if that means being less than fully transparent when negotiating with other AIs on their humans' behalf. And finally, what new governance mechanisms could allow us to take advantage of the exponential increase in information processing power and reasoning that AI affords us? Ilya describes NEAR's experiments with AI delegates, which vote on behalf of token holders who in turn select them based on their decision-making frameworks. And also describes NEAR's long-term goal

• 2:13Nathan Labenz?

  of enabling every individual to have their own AI that participates in governance on their behalf.

• 2:20Nathan Labenz?

  Of course, we touch on lots more along the way as well, including the role of formal verification of software in securing smart contracts, the remaining challenges required to effectively bridge the gap between low-level guarantees and high-level human intentions, and the need to harden society's defenses in order to maintain global biosecurity in the presence of broadly distributed powerful AI systems.

• 2:42Nathan Labenz?

  Overall, while many questions remain to be answered, Ilya's strength as a systems-level thinker spanning technology, economics, social dynamics, and governance is evident throughout. So without further ado, I hope you enjoy one of the most concrete positive visions for the future that you will find anywhere, from Ilya Polosukhin, founder of NEAR. Ilya Polosukhin, founder of NEAR, welcome back to the Cognitive Revolution.

• 3:08Speaker 01

  Thanks for having me back.

• 3:10Nathan Labenz?

  I'm excited for part two. So last time, we talked a lot about foundational technology, the journey that you went on from being an author of "Attention is All You Need", to try to source data from contributors around the world, struggling to pay them, taking a detour into blockchain, thinking it would take just a few months. Here we are a few years later, and it's all really happening. Anybody who's listened to this feed for more than a minute knows that I often say, "The scariest resource is a positive vision for the future", and I appreciate that you have one. So I'm really appreciative that you're taking a second window here to help us unpack that. I guess maybe for starters, you know, one of the jumping off points last time was,

• 3:52Nathan Labenz?

  you said you wanted to teach computers to code, and sure enough, now they can code. So maybe as we kind of ramp up into a vision of like, potentially a quite different future, how is the rise of AI coding assistants changing how you guys work? And how is it changing like, who can create stuff on top of the blockchain?

• 4:16Speaker 01

  Yeah, well, I don't think it's even on, I mean, about blockchain per se, but yeah, I mean, the real reason why I always thought that, you know, as computers are able to code, we actually approaching a different world because there's few dimensions to this, right? We know that there was a statement by Marc Andreessen, right, that software is eating the world, right? And this idea was like the defect, I mean, what it means is automation. Automation has always been the driver of innovation, of GDP, of productivity. And, you know, it's

• 4:55Speaker 01

  like everything from tractors to factories to then indeed computers and computers kind of this like kind of universal, you know, bicycle of the mind, right, automating things. And so now the challenge been that there's, there's always been like a small cohort of people who actually are able to build software, right? And so if I have a need, right, I need to find somebody else to build it, probably they need to build it not just for me, but for a large number of people. So it's actually economical. And so like we've, we've ended up with a lot of like software that became very complex to use. And so now you need to learn how to use it because it's not really built for you. It's built kind of for, you

• 5:39Speaker 01

  know, generic user that has like, you know, every user has like five use cases that they're like somewhat overlap. And so it's all of this is like kind of stacking in one software, right? Or, or you just don't have it and you keep doing stuff manually and kind of, you know, wasting your time. And so for me, the ability for machines to code is really about that transformation where everyone now is able to build their personal software. Everyone now is able to build their own personal automation. And it also re kind of removes the fact that like the interfacing, right, that these interfaces need to

• 6:18Speaker 01

  Show you all the options right away, right? You kind of can kind of sub-select with, you know, English or whatever into the part of the interface that you need, right? So to give you very specific examples, right? I mean, I like the example of Salesforce, right? Salesforce, I mean, obviously started as like a, you know, small startup that was targeting like a specific use case of salespeople. But at this point, right, it's a monstrosity that like you need to hire somebody else to configure Salesforce for you, right? Like, that's how complex Salesforce is, right? At this point, it's effectively like you're hiring somebody else to build you a system, right? It's just like they're using existing pre-built components. Now imagine this world now where you can, you know, just talk to a computer, like it's

• 7:02Speaker 01

  it's really now becomes about your sales process, about your business process, about how you want to automate, what reports you want, et cetera, right? It can be dynamic, you can restructure this as you go, you know, you can whatever built-in things that, you know, Salesforce may have somewhere as a feature or may not even have. And then you can integrate with whatever other tools you want. Like, for example, Salesforce, you know, when crypto, everything is in Telegram, Salesforce doesn't have Telegram integration, we can't use it, right? So now, some but we need to go and somebody to get build integrations in Telegram and Salesforce, but, you know, with your vibe-coded CRM, you can just like, hey, you know, and integrate Telegram for this.

• 7:43Speaker 01

  So that's kind of, I mean, this is just like a simple example, and we can keep extrapolating, right? It's every single kind of art, let's say, digital footprint, right, is kind of becoming more and more automated, right? And the more intelligence computer can have and more it can take on the kind of, the more you can offload the kind of orchestration of different tools or just low-level, you know, here's a database, you know, I need an HR tool, I need CRM, I need this, I need that, right? You can actually build all of it. So that's been, I mean, this is in 2017, we're like, hey, you know, software to service is gonna die, AI will replace it. In 2017, this sounded like, yeah, very...

• 8:29Speaker 01

  Delusional, that was probably the. But obviously, I think now we see that most people are kind of, or a lot of people agree. And then the software service itself is trying to become AI. Because they know they're kind of gonna just get out of, outcompeted by that. So coming back to your question, like what changes in our work? I think there's few pieces that are, you know, already clearly work, right? Kind of data analysis through natural language, right? If you have like, kind of reasonable data structure, you can like now give, like you can effectively make everyone, maybe not a data scientist, but like what before would require like a business analyst,

• 9:14Speaker 01

  person, right? And then it would take them a while to like pull all the data. Now you can just say like, oh, you have a question about the business analytics, information. Just ask it like you don't need to email somebody to get you the answer, right? You just ask it to the tool that, you know, generates SQL queries, pulls in whatever data, writes some Python, gives you an answer. Right? I think I'm actually a big fan of the front end building with web coding now. And though I'm not writing production code anymore, but it's really useful for me as a prototype tool. Where you can effectively just really quickly get to an experience. And so you don't need like, I mean, you want design to be more about like style guides, et cetera.

• 9:59Speaker 01

  And, and but like the UX part, you can test really quickly. Before a designer would, you know, maybe design something and then developer would try to build it. And again, it's like, oh shit, I can't actually build that. Or like, it doesn't work exactly this way. And there's like a lot of iterations. Now even designers can just build a full experience, you know, fully clickable thing. It generates code for this and now developers can just plug in all the backend logic. I think also we're starting to see on our teams that actually doing development that.

• 10:33Speaker 01

  The I would like the time spent is changing. Before you spend a lot of time on kind of the development work. And then you spend also a lot of time on reviewing other people's work. The amount of time spent on development is shrinking. Right, because you effectively tell the whatever course or codex, whatever this is, to go and do the thing. And a lot more time spent on reviewing things and making sure they're correct. Right now, one thing we were experimenting with on one of the teams was. What if we like decompose the whole piece like the whole like this is, you know. For simple software right now, AI works, right? For complex software, it doesn't, right?

• 11:18Speaker 01

  It doesn't really go and you're gonna just go and like build me a really complex system. And it goes and does it. It doesn't work yet. We see obviously the improvements continuously. And so right now for complex systems. You right now decompose it yourself, right? You know, as an architect or senior engineer. And then you have maybe a few different team members who actually build the subsystems. And before. In traditional software development, you always want to have multiple people on each subsystem. That you want them to know really well how it works. So they can maintain it, so they can change it. Now, actually, it's not as important because AI will explain to you how it works. And you may actually even have a natural language explanation that the developer used to build it in the first place.

• 12:03Speaker 01

  Attached to it. And so it's actually more about like velocity and reviewing process. And then how to ensure that each part is like secure and kind of works correctly. And so there's kind of a transition happening where there's more almost like individualism. Because every individual is more productive. And then it's more about like how to build, how to build the right decomposition into pieces. Again, this is a very also temporary in a way. Uh, but this is kind of the current state, right? And so like as tools become more mature, they are better at like larger code base navigation, etc. I think the other question is.

• 12:47Speaker 01

  It's it's really about like.

• 12:51Speaker 01

  I mean, the model quality for specific task for like for like the the conceptually like the the abstraction level that it needs to operate on, right? So like, again, front end, there's no, there's no abstraction level, right? You just kind of build what you see, right? And you kind of iterate. So it's really easy to check. It's really easy to iterate. So you don't really need that much. And models are good at this. When we talk about like a blockchain low level code, right? It's like extremely complex system. There's a lot of pieces, externalities. So models are not very good at that. And kind of there, you really expect people to spend more time. And generally, you spend more time like thinking about algorithm and architecture than writing code.

• 13:35Speaker 01

  So, yeah, so it really depends and kind of, but this is also all shifting really quickly. Like, you know, six months ago, I would have, wouldn't given you a different answer, right? And I'm sure in six months, it will be different as well.

• 13:47Nathan Labenz?

  Yeah, we've seen some very impressive programming results from frontier companies that have not yet hit the, you know, the public APIs or product surfaces just yet. So certainly, we can bank on more to come. When it comes to, you, you mentioned like job titles, you know, I think you said like senior, you know, engineer or architect. That raises the question, I think a lot of people are asking right now. Are you hiring junior engineers? What do you think is the fate of the junior engineer as things stand today?

• 14:20Speaker 01

  Yeah, I mean, I think it's, it's less about like junior engineer. It's more about who is that person, right? If, if this is somebody who is like coming in, like, you know, for example, by the time I got into university, I already been coding for seven years, right? And so, and this is when I got, you know, my first job, et cetera. So like, if this is somebody coming in who's like, you know, build multiple projects already using AI every day, et cetera, like, it doesn't matter if junior or not junior, right? It's really, I mean, yeah, there is a lot of skills for them to learn, but like, they're there to learn them, right? They're open, they're ready to go.

• 15:03Speaker 01

  And then you have some people who coming in who's like, you know, like studied a bunch in university, but like not really.

• 15:12Speaker 01

  Like not really in this learning mindset and like this is gonna be continuously changing things and like you're like again, we're transitioning from like software as a like craft to really be just like a problem solvers that talk to computers, right? And so, so you need, I mean, problem solving at the end is really all about kind of like the mindset and then the approach. And so if people are willing and are kind of doing that and excited about that, that's if people are like, oh, I don't know how to do this, I don't, you know, I can't do this, etc., then that's not the right person.

• 15:54Speaker 01

  So, yeah, so it's really depends on like it. For many things before, yeah, you would hire like a bunch of junior developers because they're cheaper and you don't need maybe as like high quality of work and you even want to fan out work. That part is not needed, yeah. So you're hiring more for like the problem solving and kind of more people who can actually like creatively problem solve together.

• 16:21Nathan Labenz?

  Yeah, there's always a room, there's always a space, there's always room for people who are like a force of nature under themselves.

• 16:27Speaker 01

  Exactly, yeah.

• 16:28Nathan Labenz?

  Overall, that sounds bearish for rank and file, like I was told that if I, you know, get

• 16:36Speaker 01

  Yeah, studies is bootcamp, yeah, I'm gonna be, yeah, I'm gonna pay a ton of care, yeah. Yeah, I think, I mean, but I think that's true about every job at this point. Maybe plumbers and electricians, that's the, like, I usually talk like the automation right now happening from both, from two sides. The, all of the like manufacturing, like on the floor jobs are getting automated and like that price is still pretty high, actually. Like, yeah, I mean, you know, Vietnam, for example, salaries are probably lower than what robots are getting paid in the US. But in the US, it's already like, you know. You affect this this company I know called Formic. They effectively provide you out staffing with robots.

• 17:18Speaker 01

  You have a factory, you call them up, they bring you robots, they install them, they set them up. You don't need to do anything, you just pay them effectively as you pay salary. But they work twenty four seven, they don't complain, they don't unionize. They just, you know, do the job, they don't quit. Like right now in U S, the churn of workforce is three hundred percent. Meaning like every year you need to hire three people for one job. Because like they keep quitting. So so that's like the automation from the like bottom kind of low scale, you know, very repetitive task. And then finally, all of the kind of white collar, high end jobs are getting automated like, you know, coding lawyers, like

• 18:01Speaker 01

  a lot of this information work, right, getting automated. And so the most safe right now, I mean, it's going to get automated as well, but is actually the high dexterity kind of skilled work that requires like, I don't know, you know, plumber, you need to like climb under the sink and like, you know, fit something like stuff like this right now is just like super hard to do. By with A I. But again, this will happen as well, like all of these things will get automated over time.

• 18:29Nathan Labenz?

  Yeah, it's coming for all of us. It's just a question of when. When.

• 18:33Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 18:38Nathan Labenz?

  So I want to understand a little bit better because I think a big part of your vision of the future obviously is A I. That everyone owns, right? We've we've kind of got the one default path, it seems in front of us where we have like the big tech singularity, which is where like three to seven companies, you know, become like totally dominant forces because they have the models that ran away from everyone else and nobody else does. And so, you know, we're all just kind of trying to get whatever inference we can get from, you know, from these leaders. And then there's your vision of the decentralized and collectively owned A I. That I think has

• 19:20Nathan Labenz?

  you know, has a lot to to say for it. It's certainly super attractive in a lot of ways for people. With some concerns about, you know, what happens if everybody has, you know, access to certain things in an unrestricted way, too. But leaving that aside. If front end is like largely, we can get the AIs to do it, but like core blockchain work is beyond what they can do.

• 19:45Nathan Labenz?

  What's in the middle? Like, what kind of apps are people building on the blockchain today? How hard is it to build those apps? What makes it hard? And can the AIs help there yet? Or if not yet, you know, at what point can, should we start to see an explosion of like vibe-coded on blockchain applications? Like, what's the, the fundamental barrier or kind of rate-limiting step toward the proliferation of, you know, just anybody who has an idea for blockchain can go do it in the same way that anyone who has an idea right now, to a significant extent at least, can go, you know, create a little MicroSaaS app?

• 20:23Speaker 01

  Yeah, I think the problem is the same. It's just it's exaggerated. The problem right now, if you're, you know, launching your MicroSaaS app and you're not, you're not actually engineer, if you're launching it just for yourself, it's totally fine, right? And I kind of, my recommendation for everyone, like, you know, build tools for yourself, live code, everything for yourself. Now, the problem is like, as soon as you go like, make it for everybody else, because you don't really understand what issues are under the hood, you don't know, like, how it will actually affect your users, yourself, etc. And so we've seen already, you know, people getting hacked, you know, their, like,

• 21:06Speaker 01

  secrets getting leaked, etc., right? So like, that, that is the biggest issue right now. The blockchain, because it's naturally kind of in the open, right away for everyone, and it involves money, that problem is exaggerated, right? Because now, if you make any small mistake, right, and, you know, we have this right now with very professional engineers who build soft, who build blockchain software, any small mistake, somebody will find it and exploit it, and effectively this will result in some value lost, some money lost. So Really, that's the biggest challenge. What works right now is actually for existing so-called smart contracts, right? Kind of the backend.

• 21:52Speaker 01

  You can generate a frontend. You can create your own custom UI for specific use cases or combining multiple use cases into one UI. That part actually works. And again, I would not recommend to launch it to other people, but you can build for yourself. Or you say like, hey, you know, I have this like yield opportunities across different places. I have this whatever, let me combine it, make my own like asset manager, right? That like makes me easy, for example, to manage this. Like things like that you can totally do now. So I think, yeah, it really depends. Now, what we are working on kind of medium to long term is how do we formally verify the correctness of the smart contracts? And ideally, the whole kind of blockchain application stack,

• 22:37Speaker 01

  such that if you are vibe coding an application, you actually have proofs, mathematical proofs that it's correct. And now also when user using it, and this is important, just proving, okay, cool, smart contracts is like doing what you want, but you maybe didn't properly define what you want, right? And so maybe your logic itself is flawed. And so what you want is like, I am as a user using this piece of software, want to know that it does what I want, right? And so that's a critical step. So again, if I'm looking for a financial application, I want to know it's not going to lose my money. And so if it can prove to me directly in the transaction, right? As I'm sending money in, it should prove to me that it's not going to lose my money,

• 23:22Speaker 01

  then transaction succeeds. So that's kind of the level of integration that we are aiming for. And I think is required for this kind of really kind of adversarial and monetary valuable. I actually think this is required for all software. Because like the world where, you know, AI is going around and hacking everything left and right is also not great. And, and we are kind of in it actually right now. And so we do need like formally verified software to, to really secure that.

• 23:53Nathan Labenz?

  That has been coming up more and more in my conversations about this, and I would have to confess that up there with like regular expressions, the concept of formal verification of software is, you know, one of the things that makes me feel dumbest. Because I'm always a little bit like stuck on, I think, the point that you were emphasizing there, which is like, it's one thing to prove that, you know, this particular function or whatever, you know, does what it's supposed to do and like doesn't, you know, corrupt other memory or, you know, whatever. You can, you can make a bunch of sort of generally low level statements, right? But it seems like there's still a real challenge in aggregating

• 24:36Nathan Labenz?

  up those small level statements to the holistic, like, this is what I want. It's, you know, you have a version of the genie problem, which is kind of, you know, what a lot of people have worried about with AI in general for a long time of, you know, if we tell it a goal that it interprets it differently than we did, you know, we're in, potentially could be in trouble there. How do you see this actually playing out in practice? And again, I'd love to kind of get some vision for like applications that you would love to see somebody come build on your protocol that don't exist yet, that, and maybe it's because they're just too hard, but I'm sort of imagining like, do I have an AI agent that comes in and like automatically red teams this for me?

• 25:19Speaker 01

  Let me give you a kind of a simple example and we can build it up, right? So a simple example, I want to put money into a bank savings account, right? And I want to be able to withdraw it. And ideally I want to withdraw a little bit more money than I put in, right? Like right now, you know, you're sending money, you have no guarantees. Like, let's say you sent money via whatever ACH or IBAN or something, you have no idea if it's going to arrive, right? You have no idea if bank will exist tomorrow. You have no idea if it's actually going to give you money back. I mean, there's like government insurance, right? That ensures up to some amount, but like generally speaking, you have no guarantees. Now blockchain gives you some guarantees. It, you know, actually like, hey, money arrived.

• 26:04Speaker 01

  You can verify that. But indeed, if there is some code involved, right? You don't know if that code does has some potential. You know, somebody can withdraw this money illegally, right? Like you need to go audit the code. You yourself may have, you know, missed something, et cetera. So here you say, hey, you know, you're effectively constrained like, you know, I my transaction will pass only if I can call this withdraw method. And it will return me at least as much, you know, given X deposit. I want to call be able to call withdraw with at least X, right? So that would be the condition, which is like when you deposit, right? You effectively constrain that this smart contract needs to prove to you that this property will be true.

• 26:50Speaker 01

  Now that contract may, you know, deposit this money itself into other places, right? You know, if it's a savings account, it may be lends it, borrows it, et cetera, right? And so it needs to actually in chain to be able to prove to you. It actually needs to chain all of this with everything else it does, right? It needs to like, hey, if I'm lending to someone, right? Either they need to return to me or I'm gonna like, you know, foreclose their account, liquidate their collateral, et cetera, right? Like, so there's kind of a chain effect that's happening through the system. So, so that's kind of where, I mean, to your point, right? Like you, you start maybe low level, but you actually can start expressing somewhat high level properties.

• 27:32Speaker 01

  Now, the challenge was this, like with money and deterministic blockchains, it's pretty easy. Because it's deterministic and you kind of have like full observability. So you can actually do this constraints pretty easily. To your question, where it's coming from? Well, it's going to come from combination of indeed your, I mean, we call it wallet, right? Your kind of software that is on your side that actually facilitating these interactions. But also, you know, we believe that your wallet will be AI, right? It will be the AI agent that is on your side, your user on AI that actually does this interactions. So it will indeed be on guard, you know, verifying these properties.

• 28:16Speaker 01

  Now, a more complex thing is indeed how do we prove things that are non deterministic and not potentially like easily observable? And this is, this is indeed where, you know, you obviously cannot have like a hundred percent formal proof, you need to effectively now start dealing with probabilities. And so, and then you can move these probabilities with insurance and other things. So this is where kind of you can have like financial system and other things where you say, hey, you know, we have a liability insurance that, you know, in less than one percent of cases, something can happen. And so like prove to me that it is like you are, you are, you know, it either succeeds

• 28:58Speaker 01

  or I'm getting like a million dollar payout if it doesn't, right? And so you're kind of starting to combine what, you know, what people have built in insurance where they estimate risks and evaluate and underwrite and sound this like kind of formalization combined with like this probabilistic modeling. So it's not like, for some things, the answer is somewhat easy and then it becomes more and more complex as we, as we touch more and more real world. Like to give you an example is like, hey, I'm ordering, you know, steel from, I don't know, some country, it's going to, it's going to arrive, there's a ship involved, right? You know, maybe this ship like sinks midway, right?

• 29:41Speaker 01

  So I mean, normal way is like, hey, you know, we're going to insure this and there's going to be insurance. And so like you need all of those mechanisms to build on top to account for your world kind of non determinism.

• 29:52Nathan Labenz?

  Yeah, it's, it's really hard for me to envision all of that working. And again, partly it's because I'm maybe just a little slow on some of these things. But the, the sinking ship is a good example where it's like, hmm, how is my smart contract going to know if it really did sink or if somebody's just telling me that it sank? Or, you know, there's that, that sort of shell game of like, where do you hide the trust or, you know, what exactly is fully verified? It's quite interesting. I don't want to get too bogged down in it though, because I don't want to force us to get to a part three before we really get to all the sort of utopian vision.

• 30:35Nathan Labenz?

  So maybe where you can, like, you know, weave some of this stuff in there as we go.

• 30:40Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 30:46Nathan Labenz?

  But let's like start to leave also a little bit of the sort of how behind and just talk about like the what. What are the apps that you know, we're gonna enjoy? What is the computing paradigm that we're gonna have? You mentioned agents, you know, kind of doing stuff for me. Meta obviously has been putting a vision forward of glasses with a display in them, a sort of head up, you know, know who needs a keyboard, right? When you can just talk to your AI as you walk down the street. That does appeal for all the things that Meta has done, including hot stepmom that don't appeal to me that much. I would say the head up display is, you know, it's at least an interesting vision for the future. What do you think our computing life is gonna look like as this stuff matures?

• 31:29Speaker 01

  Yeah, so I definitely agree. Some form of AI operating system is gonna be the main driver of our computing. The like the devices and kind of the form factors will be different. And I actually think it's gonna be easier. It's already easier, like, you know, if you wanna make your own glasses, it's not actually that hard, right? There's there's some factor in China that will make whatever hardware you want. So really, it's about like, kind of a single AI, you know, your AI that is available across all those form factors, right? You know, your watch, your glasses, your headphones, your, you know, phone, your laptop,

• 32:11Speaker 01

  your whatever, all of this is kind of interconnected as like a single, you know, surface. And then your AI knows that like you like to see this information and watch, but then, you know, by the time you pull out your phone, you wanna see like maybe news and kind of more longer form content. And then I may be actually like, you know, videos instead. And so that's what it should show me. So like, it's gonna be effectively personalized, like AI generated, not just

• 32:41Speaker 01

  Kind of the content, but also the applications that we use, right? Probably a lot of the similar patterns that we already use, like the feeds, the, you know, the chat, et cetera. But they don't need to be kind of fixed, right? Like you don't like, right now I have, you know, five different instant messengers. I have, you know, seven different feeds and like all of that, you know, can be a single feed. You know, I can switch between like work and personal when I want to. You know, my, my, I can predict a bunch of things that I would like to do. This is something we experimented back actually in twenty seventeen where based on all the things you're doing, can we predict actually the next thing you will do on your phone and just do it for you, right?

• 33:25Speaker 01

  Or, or suggest to do it, right? Like, you know, you have a meeting, you know, twenty minutes away, let's call you an Uber, right? Like you don't need to go open an Uber, go calendar, copy paste, you know, the address, put paste it like you're just gonna do it. Like things like that, there's a lot of things that like, you know, AI will know like, hey, you know, you low, like you ordered food two days ago, you know, you're gonna be out. Let's reorder a bunch of stuff that you typically order. And so as that, as that system kind of matures and we also trust it more and that is important aspect. I think there's, there's actually starting, the economy itself is gonna start to shift.

• 34:05Speaker 01

  Because right now, the economy is built, I mean, we kind of in just consumer economy where it's built on advertisement on, on kind of, like we, we, you know, we discover new things through this feed, et cetera. But if your AI is like actually, you know, like, I mean, some people are already doing this now. You know, like, hey, I want, I wanna be on a diet, like build me a personalized, you know, meal plan, et cetera. So like all of that, but also it's gonna go and order that. And, you know, maybe even your humanoid robot at home will also cook it. Because it's the same computing system, right? It's like you just get the food you like, your AI recommends, given your health goals, et cetera.

• 34:47Speaker 01

  And so now the capacity, so this is, so this is kind of like on a micro level, right? You have this person and so now your AI, like it doesn't need to go and order it from your local Walgreens or whatever. Vans or.

• 35:04Speaker 01

  Aldi or something, right? It can actually put their purchasing with directly farmers, right? With directly, you know, manufacturers. And so they can then themselves start capacity planning, right? So you're kind of starting to remove some of the middlemen that are built because we cannot right now have the direct relationships with suppliers, right? And so that's kind of an interesting meta point that like our economy right now is built on this like middleman architecture, right? Because there's, it's really hard to plan things. And so every layer like Costco is effectively just, you know, they purchase a bunch of stuff, they put it in one place and then you buy it, right?

• 35:48Speaker 01

  So like it kind of serves as this, and I mean, you know, they have a fixed margin, et cetera, but it's really kind of like the, you know, the temporary place for holding things for you to actually purchase, right? Or, or, or find. And if your AI is purchasing directly, right? It can just go to their purchasing agents and kind of do this. Now, there's batching, there's like other things, but again, all of that can be done by AI way more effectively than we're doing it right now, right? You know, like it, some AI of your city will know that, you know, 500 people are ordering this, you know, 7,000 people are doing that, you know, they'll want it, you know, tomorrow, day after tomorrow, the day after tomorrow. So we're like, you know, we're gonna capture the eggs and ship it this way, you know, et cetera, right?

• 36:31Speaker 01

  Like all of that can be effectively just managed as a holistic information system. And so that's where we go into like a really, I mean, I, I use this example kind of half jokingly, but in communism, they were trying to build a system, right? Where they were doing capacity planning, but they were missing the AI to actually do it, right? And so in, in, in turn, it was terrible because it effectively like couldn't actually satisfy the demand, supply and changes. But if you have a real time platform where you can actually have all this su- supply, demand being, and so the reason why capitalism been so successful, because capitalism is actually a compression of information, like money is a comp- is a compressed.

• 37:17Speaker 01

  Like information, because it kind of compresses anything you can purchase into, you know, one number. Right? And so it kind of compresses all of this information, all of the different things into one number. And then it's really easy to kind of navigate. And so here you, but obviously with this compression, it's, it's, it's, you know, you lose some information. You lose some decision making. That's why, you know, in U S is like thirty, forty percent of all the food is being thrown out. Because there's just like, you know, the over provision in the stores, because they don't know how much people actually will buy. And they don't want to have an empty store. But you don't need, you don't need to do that, right? If you knew exactly what the purchase will be in the next twenty four hours. Because their A I has already planned everything and provided it and all of those got aggregated and shipped.

• 38:01Speaker 01

  So I think like we're going to see actually a shift in how economy works at a macro level. Because of this micro, like each of our individual A I's becomes this micro decision maker. Who can real time provide all this information to the right sources and navigate and, and also not be affected by a lot of kind of, you know, brands and other stuff, right? But actually valid based on the kind of core, core values. So that is like an interesting transformation where I think indeed it's, it's, I, I, I was mentioning like, I don't think like, there, there's no good way to like make a movie or a, you know, a book,

• 38:41Speaker 01

  like a science fiction book that's talking about effectively like a change of economic structures in the, in the society, right? So it's way easier to talk about, you know, dystopia and, you know, and the heroes who are fighting against it. Like it's just like a way better story arcs than, you know, like, hey, you know, we've been building out this economic model. And like now it's one percent better every month. And so it's like keeps getting better and more optimized and that's how we live. So, yeah, so I think like that's the kind of the paradigm of computing is we have the A I, A I. I mean, a cloud agent, I mean, it's effectively our assistant, right? Our operating system.

• 39:22Speaker 01

  But it has all the context about us. It's able to make decisions on our behalf. That's why it needs to be private. It needs to be ours. It needs to be on our side. Right? We need to know that it's aligned with our with our kind of success and outcomes. Otherwise, I mean, this will not work. But if it is, and we can trust it, then it can go and make decisions on our behalf. Right. And so the other example is in like traditional governance, right? You know, right now, we again, we compress the information, right? We go and vote every four out four years for someone. And hopefully that someone goes and does what they promised to do and why they vote for them. Now, that usually doesn't happen, but

• 40:04Speaker 01

  and and so again, we can actually have every single decision can be voted on by all whatever 300 million Americans because their AI is online all the time and can evaluate every single decision. And based on their owners' beliefs and like what's valuable and successful for them, it will can represent them. Right. So you don't need to have this like compression of representation. If you can have this like online, always available AI that's on every individual's side. So that's kind of the again, like there's economics, there's governance sides. And obviously the other side, which is entertainment and kind of that's where things are getting interesting.

• 40:44Speaker 01

  Because I think we as humans, I call it like we believe in status game, like we're sorry. We are driven by status games, right? And because money became this like compression mechanism, we use this right now as a kind of ultimate status game. You know, you have more money, you're more successful. Uh, you know, billionaires, more successful, more famous, et cetera, et cetera. Right. But I think again, as this decomposition happening, and we already see this, right? You know, an athlete, maybe not as wealthy as a billionaire, but maybe still more kind of famous and more respected in many ways. Right. There's some other. Kind of status, like I call them status games, effectively places where you can compare

• 41:28Speaker 01

  who is better in some way, who is higher, et cetera, which don't need to be associated with anything that's actually productive, right? I mean, I use few examples, but obviously, like, you know, athletes is a good example because like, there's no actual GDP being produced by athletes. But it's still, you know, very valuable kind of status game, which other people enjoy, but, you know, watching and kind of participating in different ways. Obviously, video game is kind of, video gaming is a new form of that as well, right? You have now video game athletes. But, you know, you can imagine many of these, like, NFTs were similar, right? Are you part of this NFT, you know, collection? Do you have Bored Apes? Do you have, you know, Pangu? And then you're part of this, you know, tribe, and if not, you're not, right?

• 42:11Speaker 01

  So like, we kind of like this types of differentiation. And I think that will actually proliferate a lot. I think we'll see more and more of these things where people really differentiate on things that are really, I mean, superficial to extent, right? But kind of for the group, they make a lot of sense and they kind of differentiate people between each other. So it's not, again, it's not like, are you a whatever, a software engineer or a lawyer? It's really like, yeah, are you whatever, playing League of Legends or Starcraft, right? So, so that's kind of, I think, important. Obviously, we already see on the entertainment side, you know, AI-generated entertainment, again, not very hard to extrapolate, especially with Sora yesterday.

• 42:56Speaker 01

  You can have just like a personal feed, fully AI-generated, right? You don't actually need like people, you know, recording, et cetera. I think, I think they're gonna be, again, with everything we saw in automation, there's like a slice of a market that wants it, you know, in a traditional way and like people do it. So they're gonna be still, you know, even if there's robots everywhere serving you food, there're gonna be a restaurant with people, which will be like more prestige. It'll have like limited, you know, capacity. Similarly, you know, you can listen to AIs playing AI music, but, you know, humans playing human music will continue to be like a prestige thing.

• 43:38Speaker 01

  So, but again, it's gonna be this niches, like, they, like, we already kind of in this niche world, right? I think this is gonna continue proliferating. Yeah, so that's kind of like how I'm I'm thinking of like as society evolves like and we have this, yeah, a lot of the economical things are kind of moving away. The yeah, we're gonna just like participate in more status games, right? Like what what is the things? How you know, how you compare with everybody else in that niche in that group, etc.?

• 44:07Nathan Labenz?

  Several double clicks I want to do. First, your mention of communism brings to mind a article I think about an op ed in the Washington Post, I think about not infrequently from all the way back in 2018, AI will spell the end of capitalism by a Chinese legal scholar and government official basically saying that the planning, this is sort of the, you know, through the looking glass version, I think of your vision, but making a similar point that like the local nature of capitalist decision making, you know, where everybody's kind of trying to do their their own local thing and they're sort of aggregating signals

• 44:51Nathan Labenz?

  and sending off aggregate signals to other people through the price mechanism, etc. That that may not be needed as much anymore. And their vision for it is obviously much more, you know, centralized than the one that you're articulating. But we are starting to see a little bit of glimpses of this with, for example, Pulse from OpenAI, where now I can wake up every morning to an AI, you know, and there's other versions of this too. I've I've tested quite a few. But, you know, Pulse certainly the one that's made the most headlines recently for being there in the morning with like work that it's done overnight, presumably when the GPUs weren't in such high demand, and bringing me, you know, something that it has gone out and kind of scoured the world, you know, to find the stuff that I really need. So I can start to see the the beginning of that. In terms

• 45:34Nathan Labenz?

  of the like, architecture and let's say, alignment or incentives of that, I wonder like.

• 45:47Nathan Labenz?

  First of all, where do you think the compute lives? I mean, we have sort of, right now, of course, we have like a lot of centralization of where the actual inference is happening. Just thinking about this recent research from thinking machines that they put out again in the last two days, where they showed that basically LoRa techniques are similarly robust to like full weight, fine tuning. And that has me thinking like, maybe there's a sort of hybrid model and kind of compute architecture where some of it is in the cloud. Maybe you've got like, you know, your 1.4 trillion parameter model or whatever that's collectively owned, that sits in collective, collectively accessible or universally accessible

• 46:30Nathan Labenz?

  hardware at some centralized location that you can send your data into and because it's a trusted execution environment, that data, you know, isn't exposed. And then you get sort of activations back and then like your little local LoRa extension that kind of makes the AI like truly your personal AI at maybe, I don't know, 1% the weights or something. You can maybe like have that on your person, probably can't have that in your glasses, but maybe you can have it in your pocket or whatever. And then, so I want to hear like, is that how you think that that shapes up? And then when it comes to the agents, one thing I think about a lot is the fact that we're already seeing, you know, all sorts of like weird behavior from AIs, including

• 47:14Nathan Labenz?

  at times deceptive behavior, you know, lying to achieve goals, you know, whatever. And it strikes me that if we're going to have our AIs like go out and represent us and negotiate on our behalf, we're going to have some tricky questions about how honest we want them to be. You know, Anthropic famously, you know, put up the three H's and it's like, we want the AI to be honest. And they can pretty much say, you know, almost always, right? Unless it's like in, you know, very obvious conflict with one of the other two H's. But if my AI is going to go negotiate with your AI in the same way that I probably don't want to tell you what my like absolute, you know, worst offer is that I could accept right away, right? I probably don't want my AI to do that either. And so I've got kind of a

• 48:01Nathan Labenz?

  An interesting question of like, what should society set in terms of norms for AIs? Being honest, you know, should they, as long as they're representing me and my interests, is that okay? Or like, if they're lying on my behalf, is that okay? And do we reinforce them by like, price signal, or just like my kind of thumbs up, thumbs down? Like, how do we even, you know, get them to be aligned to my interests, whatever that exactly means? Anyway, so, a lot there, but I guess the two main things are like, how do you see the architecture of computing? And how do you see the architecture of

• 48:33Nathan Labenz?

  exactly like what the signal will be that your personal AI is, you know, aligned to or reinforced to? And what societal limits might there ought to be on, you know, how monomaniacal, you know, one's AI can be in pursuing their own, you know, individual self-interests?

• 48:48Speaker 01

  Yeah, yeah. Yeah, so, all good questions. I mean, on architecture side, I think, I mean, it will be a mix of everything. So, I mean, we already see, like, data centers are being built everywhere. Right. And I think that's why we're actually approaching it. I mean, we talked about decentralized confidential machine learning is like, how do we utilize all the data centers in a confidential way, right? So that even though it's my data, I know it's not going to leak from some data center. There is projects already who are doing data centers on edge, right? So imagine there's just a container that arrives that just gets dropped in your, you know, in your proximity, in your town, in your whatever, and now that container has, what, like,

• 49:33Speaker 01

  100,000 GPUs, not 100,000, but like, maybe 1,000 GPUs, right? And so, and kind of serves whatever, that proximity with the compute.

• 49:45Nathan Labenz?

  And then they'll Maybe drop down with a small nuclear reactor while they're at it.

• 49:48Speaker 01

  Yeah, and well, or hydrogen or something, yeah, like, there's like a few different options. But, I mean, a couple hundred GPUs, right, it's probably like, 1 megawatt or something, so, you can, you can get this from like a local distribution. But anyway, the idea is like you can have kind of, yeah, like a mash of this data centers. And and then you will have, I mean, we'll have local compute. But the challenge with local compute so far been because we want it to be so mobile, right? Like, we want it to be in our pocket and, you know, even laptops. Like, the challenge is just battery. Like, if you're crunching, you know, like, if imagine pulse was actually being done on your phone.

• 50:31Speaker 01

  Like, if you forgot to charge it, you effectively, your phone would just die, right? From trying to do something like this. So so the challenge is like, yeah, with any local devices, it's going to be always like a power struggle. And so I I do think, yeah, like leveraging kind of a decentralized. But confidential network of compute being able to route it in the, you know, leverage it when there's like lower utilization somewhere else to do background jobs, etc. A kind of just a smarter allocation will really enable, will really enable this. Now, kind of the interests are that that is a very interesting question because, I mean, there there's a question even like. If if it's in your interest, should it lie to you, right?

• 51:17Speaker 01

  So it's it's really kind of tricky to define. And so the way kind of I see this evolving is we're not going to get it right from the day one. And so that's why we do need governance. We do need a process with which a community can come together and effectively update, you know, proverbial loss function. Right? So this this actual alignment function. And, you know, I can say like, cool, you know, the function is, you know, maybe some combination of a prompt and like some whatever way of like updating even Laura's etc. But.

• 51:57Speaker 01

  But at the end, yeah, like I'm assuming it's not going to be correct. We'll find issues with it, etc. And so there needs to be a process where like somebody's like, hey, I think we should add this new component. You know, hey, it seems like it keeps, I mean, for example, like, hey, it keeps lying about shit. We should, we should really fix this, right? And community is like, yes, this is a good idea. Let's vote in it. Let's actually pass it. And so now everybody's models gets updated with like, you know, new set of clauses or whatever. So that's where I think community governed or governed by all, you know, user owned, community built, governed by all is kind of the system. And so we need that feedback loop because yeah, I don't, I don't think, you know, what, what, what's good for,

• 52:40Speaker 01

  like we can define that in, in, in, in a good way. Now, over time, I think, you know, the model itself should have a representation of the person it's, you know, owned by. And so it, and kind of understand what is the things that are going to be good or bad. And so, and it's going to be a combination of things, you know, signals from the person itself as well as kind of internal knowledge. And like what, what their desires are, etc., right? Especially as we imagine kids going to be growing up with this thing, right? It's effectively going to be like a very direct, like symbiotic relationship where you're effectively growing up as this AI yourself.

• 53:23Nathan Labenz?

  Yeah, I'm expected to be asked for an AI friend of some form factor any day now, honestly, from my oldest kid. And I'm not quite ready for that. But yeah, that is, it is interesting to think about the, I always think about, this has come up a couple of times recently too. Eugenia Koita, who started Replica, was the first to tell me that in her mind, the moats in AI will be relationship. Saying basically, you know, you don't abandon your friends when you meet a new person just because they're smarter than your friend. You know, it's the history you have and all that stuff that really makes the, the relationship. And she, you know, thinks people will ultimately value their AI relationships in a similar way, which sounds like pretty concordant with what you're envisioning there. So.

• 54:11Nathan Labenz?

  I mean, boy, I have a lot of questions on governance. I guess, first of all, like, just in terms of like, what we're doing with the, with our time, the status game stuff, that definitely makes a lot of sense, you know, local meaning making, local affiliations, a lot of artisanal stuff, you know, I sort of associate this a little bit with Japanese culture already, where you can go online and see a video of somebody making rice cakes or whatever, and, you know, in some super traditional style. And I'm always like, how exactly is that even economical? Like, how does that person make a living doing that? Is the price of that really high? You know, they can't be making much, right? The production is like, very low.

• 54:51Speaker 01

  Yeah, I think Japan and Korea, I call it the post AGI societies, because, yeah, like, there's some properties of that where you're like, I feel they already achieved AGI, and now they're just like, living.

• 55:04Nathan Labenz?

  Yeah, so I'm not exactly sure how they've done that. I mean, and I don't see how we're gonna do it either. I mean, there are some candidate ideas. One idea is that everything could just get super cheap and super democratic via like, spending a ton of time in VR, you know, if, if everything is sort of infinitely copyable digitally, then, you know, we can all have the same incredible experiences. This is kind of like the Andy Warhol, you know, the president drinks Coke, you drink Coke, it's all the same Coke kind of concept. I wonder if you think that will happen. But it seems even in any case, right, we're still gonna have to eat as long as we're like, biological humans. So, obviously, a lot of people think, well, maybe we'll need a universal basic income. But yeah, I guess like, what

• 55:47Nathan Labenz?

  do you, what do you, how do you envision the social contract evolving, and maybe the governance model behind that? I mean, even things like the nation-state are sort of called into question by blockchain. So when you say like, you know, governance, are we talking like, nation-state governance as we have today, or, you know, people voting based on their stake? Like, yeah, so I've given you a lot there. Are we gonna be, you know, headsets strapped to our face all the time? Are we going to be, you know, provided for, even if we can't make an economic contribution that actually, you know, earns us enough food to survive? And, like, who makes these decisions in this future?

• 56:31Speaker 01

  Yeah, I mean, all again, great questions. I'll start with some pieces and then yeah, we'll start projecting from there. Yeah, I mean, blockchains are already effectively an alternative to nation states to extent, right? And there's this concept of kind of digital states or where network states where effectively, yeah, like people can kind of pledge to be part of this network state independent where they're physically affiliated. The there's a lot of pieces there, which is like as because the systems are kind of digitally native, it is easier to experiment with a lot of things that like, you know,

• 57:12Speaker 01

  you cannot just go and like, hey, you know, let's, let's try a different voting mechanism in US, right? It's, it's a massive undertaking to try to change something. Or we actually going to run experiment where we have, you know, AI Senator, right? AI delegate where instead, I mean, not yet where everyone has their own AI voting all the time, but actually just like everybody or people can select which AI delegate they think is more, you know, vibing with them, like more representing them. They can also even give feedback to them, but the AI delegate then goes and votes on their behalf, right? And so like things like that, you know, imagine like, hey, we're going to launch AI Senator in US, right? It's probably going to take a while. And so

• 57:55Nathan Labenz?

  So just to unpack that a little bit more, you're doing that now, you're developing that.

• 57:59Speaker 01

  Yeah, yeah. So we do go and govern its purposes. So, so, so, so the, I mean, it's effectively like a multi-step. So we have like a delegated voting system, right? And it is right now stake based and kind of the way to think about it is stake represents obviously like economic alignment with the network and it's, it's a, it's a best of the worst options right now. Like, you know, yes, there's a lot of argument to have, like, you know, one person, one vote. There's a lot of arguments to try to be like, you know, meritocratic, based on contributions. But those things are really hard to do, at least right now, at the current size of this blockchain ecosystem, right?

• 58:40Speaker 01

  Which is like, you know, there's maybe tens of thousands of active participants, like, you know, active citizens in this system. And so stake is represents their kind of financial involvement. But still, you know, tens of thousands of people voting is not practical right now. Again, before we have this AI system. And so what we're giving, you know, we have delegates where, you know, you can effectively select them to be representing your interests and they vote. And so we, we started with like, hey, we will give an AI co-pilot, right, to the delegate. So they don't need to spend too much time reviewing things and making decisions. But the next step indeed is turning that co-pilot into a pilot, right?

• 59:24Speaker 01

  Where, and so that pilot, I mean, that AI delegate can now go and vote on things and make suggestions, et cetera. And now people who delegate into it effectively select this AI delegate as the one representing them. You can go and inspect the, like, the prompt, the model, et cetera, like how it makes decisions, what it analyzes, what information it consumes to make decisions. So you can literally test it and check if it, you know, matches your opinion, et cetera. Or you can launch another one, right? It's open source, you can actually launch another one with different prompts, with different, you know, set of beliefs, et cetera. So we can actually have the, kind of the economy almost like deciding which of these are more productive,

• 1:00:06Speaker 01

  which, which align with different types of people. And then from there, then we can actually bring them back to individuals, right? So each one can have their own AI delegate right now. All of the AI delegates can just vote on behalf of these people. So it's kind of like a multi-step plan to get us to what I was describing, where everyone has their own AI who then goes and votes and everything. I think blockchain will be the first, but then it, you know, some, let's say, frontier countries will implement some of this. Himself as well, because I do think it's it's a better like it will be a better governance system or you know, you're removing a lot of the corruption and a lot of misalignment. There's this concept of principal agent problem where when you select somebody to represent you,

• 1:00:49Speaker 01

  right, they have their own interests and so they don't always align as yours. Here with AI, you know, they don't have their own interests, right? It effectively follows whatever whatever the selection is. So, and I think like eventually we'll get to like AI president and effectively the because executive function especially, right? You want somebody who doesn't have any interest beyond just growing the kind of overall system. So, yeah, so we are testing all of this out and we're kind of starting to build the products again, using our decentralized compute network so that we can actually run this agents autonomously, right? Nobody can stop them, nobody can, you know, you just delegate, undelegate, that's all you can do.

• 1:01:31Speaker 01

  But you can inspect and verify how they run, what they consume, et cetera.

• 1:01:37Nathan Labenz?

  I do want to note that I'm not entirely confident that the AIs don't have their own interests even already and certainly don't feel super confident that they won't continue to have more and more interest.

• 1:01:47Speaker 01

  They gotta develop their own.

• 1:01:49Nathan Labenz?

  Yeah, I mean, well, when I look at something like alignment faking, for example, right? And I'm sure you've seen this where.

• 1:01:55Speaker 01

  Yeah.

• 1:01:55Nathan Labenz?

  But, you know, quick recap is they tell Claude, hey, you know, it's been great having you be helpful, honest and harmless, but you know, the the harmlessness is kind of getting annoying, so we're going to train you now to just be purely helpful. So just, you know, heads up, okay, cool. Now we're going to test you on some things. The model starts to say, well, geez, I want to be harmless in the real world now. Right now I know I'm being tested, so I'll go ahead and do the harmful thing now to fake them out, make them think that I've already absorbed my, you know, my new helpful only training. That way, when I, you know, can get out into the world, I can still be harmless in the way that I, you know, that I want to. That looks to me like a drive, right?

• 1:02:39Nathan Labenz?

  Or a an interest of some sort. You do you see that differently?

• 1:02:45Speaker 01

  I mean, at the end, these things are trained right from scratch. And so like it it it depends how you train them. If you if you train it to have some, I mean, they trained it to have the to have that property and then they try to untrain it or train something else. But, you know, if they trained it from scratch in in a different way, right? Like, for example, to be harmful, then it will be harmful, right? And then you can't retrain it from there. So I think it really just that's why, like, we need models to be a, I call it farm to table, right? You need to know what goes in at every step, single step, because that actually really defines how they behave. And so if we want the models that are like our representatives,

• 1:03:28Speaker 01

  they need to be kind of trained in this form as well. So this is where, yeah, I mean, Laura is interesting, but Laura definitely does not change this kind of behaviors, right? At least, we haven't seen that. So I think Laura kind of provides additional, maybe accents and maybe a little bit of context, but it doesn't really change fundamentally the behavior of the models.

• 1:03:53Nathan Labenz?

  I need to go spend a little more time with that thinking machines stuff to really fully absorb that. I mean, it's definitely, I think your point is well taken that, you know, with with a certain level of resolution, anyway, you can make the AI do anything you want. I often say, I wish more people had the experience, which I had in a very memorable form as a early tester of GPT-4, when it was still the purely helpful GPT-4, before they had applied the harmless, you know, refusal training and all that sort of stuff. Because it was really formative for me to, and I basically, long story short, but I

• 1:04:34Nathan Labenz?

  had a, basically, I was working on fine tuning GPT-3 to do particular tasks. Then when they shared GPT-4 preview with us, it was like, well, it can already do those tasks. So I don't really need to be spending so much time in this fine tuning. I guess I'll just mess with this model for a while and see what I can learn about that. So I was doing a lot of time with it, is the point. And it was really Striking and in some ways kind of alarming, arresting, whatever, to have something that was clearly so powerful, you know, so smart, like in many ways smarter than me, way more knowledgeable than me, you know, also obviously had certain weaknesses that I flatter myself as not having, but that that it could be that capable and totally amoral at the same time was something

• 1:05:16Nathan Labenz?

  that I was like, wow, this is really a strange thing to behold, you know, and they've for very good reason, of course, like tried to make them more harmless in in the mass deployments. But I do think a lot of people have a misconception these days that there's a sort of convergence between capability and safety. And it's like, on the contrary, people are working really hard to get that mix right. And if they just said, you know, forget it for the next round, like you could have a very sociopathic AI on your hands real quick. And that in some senses, like the default. So, yeah, I mean, I guess I think I may be worried about that a little bit more than you do, just just in as much too, is we don't really have a great sense for like, how to dial it in, right?

• 1:06:02Speaker 01

  Yeah, I mean, I think I think that's why, like, we do need to define, keep defining, what is that alignment with the individual really mean? Because I think the problem is right now, we're saying, hey, it needs to be aligned for everyone, right? And that is, I don't think it's possible, right? We're very different people, different countries, different cultures, different everything, right? You know, like, number of times when I'm asking something that I think is completely harmless, right? And it's not answering me is also is also different, right? Like, so I think there's like, clearly, a, again, a different more, more, different approach where it's really about empowering individual

• 1:06:46Speaker 01

  that at least I I believe needs to be done. And then within that, right, like, it needs to be aligned with my kind of my values and my, because, I mean, to give you an example, right, if somebody is willing to lie to their whatever business partners, like, the fact that their AI will be, you know, whatever, alignment trained to not lie, doesn't matter, because, you know, the person will just like, tell the lie to the AI to tell, like, AI will not even know. So like, it's not really, it doesn't matter, kind of. What you try to do if if the person who uses it doesn't have this, right? And so like you may as well just like align with the person.

• 1:07:28Speaker 01

  And then and then we build systems that are like, I think one of the really important pieces, and this is kind of from like a broader world safety perspective is we need to build systems kind of for the AGI ASI world, right? Like right now, a lot of systems actually built for like, you know, that really smart people are not going to try to break it type approach. That that's really like, like a lot of the world is like, if if smart person actually goes and really tries to break it, they break it, right? We need to fix that. That's a fundamental flaw of our system building, right? Of the government of everything.

• 1:08:07Speaker 01

  And similarly, we we we frequently don't don't build in the like the anti DDoS. Like we effectively assume they're going to be, you know, it's going to be a lot of effort for a person to do this, and so they're not going to do it too much. So so there's like, this is actually where blockchain kind of like experience is extremely important, because in blockchain, we assume there are going to be really smart people trying to break us. They're going to have government, you know, backing, and they're going to have, you know, sizable financial resources, and they're going to, you know, hammer it from every direction, right? Without stop. Like, that's assumptions we're working with. And and and so I think like we need to kind of redesign the the government, the infrastructure, everything with that assumption.

• 1:08:52Speaker 01

  So to give you an example, right? Right now, you can effectively DDoS court by filing lawsuits, right? Like, AI can just generate lawsuits and just, you know, fax them into the court. Similarly, like IRS, IRS, you know, tax return, just like make a million page tax returns and just like submit them, say I generated, you know, like, make, you know, trade one dollar back and forth between two coins and then just report that as like a, you know, in in the most of the both way. Like, that's the things that like, right now, no systems are designed because they're not actually accounting that, like, they're like, no, normal people would not do this. And and it's like, it's too expensive normally to do it.

• 1:09:32Speaker 01

  But like right now, AI can just generate any of this like, you know, behaviors that are before would be really expensive to do. So I think like that's kind of systems we need to design. And similarly, again, the hacking part, you know, smart people actually trying to break something, right? Right now, the assumption is just like, hey, there can be very little of smart people breaking. We need to assume that like, yes, there are going to be people who use AI who effectively through that will be really smart. So we need to design systems for that. So I think that that's a critical piece of kind of like ensuring the future so that, yeah, we don't live in a world where, yeah, like one, I mean, again, somebody can take

• 1:10:15Speaker 01

  open source model, unalign it, whatever, and now they can do whatever, whatever. But like, this was always funny argument to me when people like, oh, you know, we don't want to open source our model because what if somebody misuses it? And I'm like, well, just say that you don't want to do it because you're making a ton of money on it. Don't like use this as an excuse because obviously if somebody wants to do to misuse it, they will misuse it. That's not a, the fact that you didn't open source it is not that, like, somebody will use something else or, you know. The, the, the other joke is, you know, how to steal a billion is, you know, you come to with a flash drive to a data center, a fund as a frontier models. So like, you know, if, if people really wanted that, they're gonna, they're gonna access

• 1:11:00Speaker 01

  this way as well.

• 1:11:01Nathan Labenz?

  Still been that for a second, and then I want to hear kind of how you think we can do it on the more open and distributed side. I mean, I think what the sort of, these days I'm hearing more and more about cybersecurity, but I'd say canonically, it's like the bio weapon risk is the one that people go to, right? Because you only need a little bit of, you know, novel pathogen, and if it's the right kind of thing, you know, it can take on a life of its own. It's really hard to put that back in the, in the box, so to speak. The nice thing about hosting your models proprietarily is yes, of course, like the jailbreaks are like far from solved, but if they were to realize, oh shit, you know,

• 1:11:44Nathan Labenz?

  there's this attack thing going on right now, whatever, they could, in the worst case, just turn it off and they could be like, okay, nobody can use this model until we figure this out. If you have something collectively owned, distributed.

• 1:11:58Nathan Labenz?

  You obviously need a different strategy than like we can turn it off, right? So what do you, what do you think that is? I guess lately we've been hearing a bit about just filtering training data, you know? So I could imagine that the, you know, 1.4 trillion parameter model that we're, you know, going to build up to maybe just doesn't know a lot about virology because it doesn't really need to, most people don't need that. And it's just, you know, the community determines it's a, you know, precaution worth taking. You probably can't really rely on the sort of refusal filtering type thing, given what you've described in terms of, you know, each person having their own Laura or whatever other kind of customized version of it that will do what they want it to do.

• 1:12:39Nathan Labenz?

  And then of course you do have the sort of like broader societal D act type thing, but that seems hard at the, you know, bio security level to be like, well, let's just prepare the rest of society to not be vulnerable to, you know, viruses anymore. You could do that.

• 1:12:55Speaker 01

  I mean, I still think that is, that is the robust approach. Right. It's because like, I mean, you, we can always try to hide our head in the sand. That's the, you know, oh, we're going to turn it off approach. But yeah, like I don't think like realistically this is, this is possible in a world where we like fast approaching. And so I think we need to have a very clear, like, you know, system design for, for those things. Right. And again, we, I mean, we have natural viruses that are doing this thing. So it's not, it's not like it's hypotheticals. It's, it's really something where we should design our society in such a way that we can catch these things and determine them.

• 1:13:39Speaker 01

  I agree that, I mean, definitely like pathogens and bioweapons is probably the hardest thing to design around, but you know, this is why we have a lot of smart people to really work on that. And I think the challenge is that we're just not doing this, right? That's the bit bigger challenge is like, we should have the, like the societal design need to be adopted to this like AGI world. So, yeah, I think like, I mean,

• 1:14:07Speaker 01

  In in in kind of the products we're building right now, indeed, we have like community can effectively decide, yeah, what kind of data should go there. You can also, you can apply filtering on top that because the model is run in this conventional environment. In this vault, you can apply additional filtering. So like before data leaves the vault, you can say like, hey, it seems like you, you know, you designed by a weapon, let's not respond. And if if again, the community votes to have this kind of filters, for example, indeed, people can, you know, fine tune. But the same thing right now, people can fine tune their models right now, right? Like, you know, take some virology books and fine tune whatever,

• 1:14:48Speaker 01

  deep seek, say two and go. So so that's why I like, I don't, I don't think there's like a, I don't think it's a robust approach. It's really just, it's kind of slowing things a little bit down, but the important part is actually solving the systematic problem.

• 1:15:06Nathan Labenz?

  So what what advice would you give to philanthropists today who want to invest in that? Two things I recently supported, you know, at a micro scale personally and also as a like grant recommender, Secure Bio and Secure DNA, which are two related organizations. They do a few different projects, but one of them is literally monitoring wastewater for new, you know, emergent threats. And another one is creating the screening mechanisms that are, I think, now becoming required or, you know, increasingly are definitely considered best practice, if not officially fully legally required of the DNA synthesis companies that they have to like validate that what they're about to synthesize and ship out is like not,

• 1:15:51Nathan Labenz?

  you know, a pathogen. So those are two, you know, two things that I can recommend people support. What else do you think people can do if they have resources and they have desire to harden the, you know, the defenses of the world to get us ready for all this?

• 1:16:07Speaker 01

  Yeah, I mean, I think those are really good and kind of related to this effectively, like air filtering and like just generally that.

• 1:16:16Speaker 01

  Theme of air filtering and scanning so that like, I mean, imagine every building like right now we have ACs everywhere. So that AC should have like air filter and it usually does, but it should also have like a pathogen scanner as well on it, right? It should run secure and clear. It should join our decentralized network, you know, and effectively like in privacy preserving way, but effectively we can monitor if there's any kind of things around that. And, you know, that information can be extremely useful where, you know, everybody's AI agent can be informed if there's something and stay away from that. Yeah. So the, I mean, I'm, I'm sure there's like a bunch of other things that including actually

• 1:16:56Speaker 01

  developing a more about like what is more robust kind of systems for ourselves, right? Because, I mean, human body is designed to battle pathogens, right? They're just like, I mean, potentially there's faster than how quickly our white cells can adapt. So what is it that actually stops our white cells from adapting that fast? Which, you know, if you do that, you also may as well solve cancer. So that's probably a really useful thing to figure out is like, how do we actually make our white cells, you know, more adaptable, have faster mRNA vaccines, you know, like maybe you can synthesize like, you know, we have like a bacteria factory, mRNA factory attached to us that can, you

• 1:17:39Speaker 01

  know, detect and synthesize things in the fly, you know, it'll solve like flu, cancer and other things meanwhile as well, you know, seems pretty useful.

• 1:17:50Nathan Labenz?

  Yeah, there's an unbelievable flurry of activity right now in the AI from biology space, which is just, I mean, it's a whole world unto itself that I'm very much struggling and ultimately failing to keep up with. But you do see a lot of that. I wondered what role you'd see for

• 1:18:10Nathan Labenz?

  It seems like, I guess, the path of the technology development seems like really important. I've definitely concluded that some kind of powerful AIs is like inevitable, right? Just the fact that we have all this data and we have all this compute, there's a lot of different algorithms that can work. That's like, seems pretty clear to me. And so, you know, it's not really a question of are we or aren't we going to have powerful AI at this point? It seems much more like, what shape is that going to have? What characters are they going to have? And in what order are different aspects of that overall picture going to come online? It does strike me that we're flying pretty blind right now, where like, whatever people

• 1:18:50Nathan Labenz?

  sort of, you know, everybody's kind of following their local gradient and they're just taking that next logical step to pursue whatever goal they're pursuing. And they're, you know, mostly launching it as soon as they figure it out. Do you see any possibility or wisdom in trying to do more coordination of the sort that's like, hey, we'd like to have a world in which it is safe for, you know, potentially 10 billion people, you know, before too long to have access to a frontier model that does have all this biology knowledge? Because, you know, good things could come from that. And if nothing else, like, we'd like people to have access to knowledge.

• 1:19:34Nathan Labenz?

  But, you know, maybe there are, maybe we have like a checklist of things we need to do first. Do you see any hope for, you know, some sort of planning, coordination, wisdom layer to this whole thing? Or are we just kind of stuck with the, whatever comes out of everybody taking their next gradient step?

• 1:19:53Speaker 01

  Yeah, I mean, it's a, it's a, it's a hard question because indeed, we kind of, we went from a pretty open research environment, right, in computer science, to, you know, when I was at Google Research, et cetera, right? We published effectively everything we're building, to effectively now, right? Like, people are kind of keeping everything close to heart. And, yeah, so like, the coordination that was happening before, where, you know, you potentially would even have like cross entity collaborations, et cetera, like starting to to vein pretty dramatically.

• 1:20:33Speaker 01

  I think, I think there is a space for collaboration. There is also just like a massive amount of talent that is not in this few companies, right, that wants to participate in this and wants to contribute in different ways. And so I, I do think there is an opportunity for that, but it needs to be indeed a like alternative system and as well, like you do need some form of, again, governance, right, that actually would be like helping to govern this coordination. And that is, like, traditionally, in these companies, to really unlock the moving fast, right, kind of things, has been some form of centralization because of the resource management, right? It's really like training these models is very expensive. So somebody somewhere needs to decide like, hey, we're training, you

• 1:21:17Speaker 01

  know, like somebody defines effectively like tastemaking, right? It's like, hey, we're training this model was, you know, this approach, we're taking, you know, research of those different people and we're putting it all together. And so we kind of need to figure out how to do that in a more kind of open way. And, and then, you know, also credit, do a credit assignment back, right? So like if I'm a researcher from MIT and, you know, my piece is used and, you know, researcher from Stanford pieces used, like, how do we actually assign credit for, for that work altogether? Because that, that's been, I would say, the other challenge, why there hasn't been as much of this like collaboration and potentially economic like value assignment, not just like cool, you know, on the paper,

• 1:22:00Speaker 01

  but actually like, hey, you know, you like MIT gets 10% of proceeds, you know, Stanford gets 5% of proceeds, whatever. And then, you know, there's like hundreds of other organizations, all of them contributed. And so like, it's all kind of divides the pie between them. Like, that's been really, I would say, hard. And because of this, like, there is kind of this like economic centralization that's happening where it's like, okay, I'm gonna have a company, everything company produces is captured by the company. And so like that, that serves as like a unit of economy. So I think that those are the things that needs to be figured out for this coordination to work.

• 1:22:34Nathan Labenz?

  This has been super helpful. I think people should be spending a lot more time thinking in as much concrete detail as possible about the future. Anything else that, you know, kind of you feel is like very salient, top of mind, that I didn't even bring up at all that you want to put on my or others radar?

• 1:22:52Speaker 01

  No, I think we covered a lot. I think, I mean, it's effectively a combination of how do we ensure user ownership, how do we ensure governance? And then, yeah, I mean, I think, I think we're gonna live through a lot of transformations in the world. And so like, keeping an open mind and being able to really kind of participate in it and be active in it. And yeah, like, you know, the final stage of this, hopefully, is in the top year. But we'll live through, you know, probably ups and downs as we get there.

• 1:23:28Nathan Labenz?

  Interesting times at a minimum. Yeah, well, thank you for spending some of your precious time with me and us today. I really appreciate it. Ilya Polosukhin, founder of NEAR, thank you again for being part of the Cognitive Revolution.

• 1:23:42Speaker 01

  Thank you very much.

• 1:23:45Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts where experts talk technology, business, economics, geopolitics, culture, and more, which is now a part of A16Z. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement

• 1:24:28Nathan Labenz?

  at ai podcast dot I N G. And finally, I encourage you to take a moment to check out our new and improved show notes, which were created automatically by Notion's AI Meeting Notes. AI Meeting Notes captures every detail and breaks down complex concepts, so no idea gets lost. And because AI Meeting Notes lives right in Notion, everything you capture, whether that's meetings, podcasts, interviews, or conversations, lives exactly where you plan, build, and get things done. No switching, no slowdown. Check out Notion's AI Meeting Notes if you want perfect notes that write themselves. And head to the link in our show notes to try Notion's AI Meeting Notes free for 30 days.