#490 – State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI

Nathan Lambert and Sebastian Raschka are machine learning researchers, engineers, and educators. Nathan is the post-training lead at the Allen Institute for AI (Ai2) and the author of The RLHF Book. Sebastian Raschka is the author of Build a Large Language Model (From Scratch) and Build a Reasoning Model (From Scratch). Thank you for listening ❤ Check out our sponsors: https://lexfridman.com/sponsors/ep490-sc See below for timestamps, transcript, and to give feedback, submit questions, contact Lex, etc. Transcript: https://lexfridman.com/ai-sota-2026-transcript CONTACT LEX: Feedback – give feedback to Lex: https://lexfridman.com/survey AMA – submit questions, videos or call-in: https://lexfridman.com/ama Hiring – join our team: https://lexfridman.com/hiring Other – other ways to get in touch: https://lexfridman.com/contact SPONSORS: To support this podcast, check out our sponsors & get discounts: Box: Intelligent content management platform. Go to https://box.com/ai https://lexfridman.com/s/box-ep490-sc Quo: Phone system (calls, texts, contacts) for businesses. Go to https://quo.com/lex https://lexfridman.com/s/quo-ep490-sc UPLIFT Desk: Standing desks and office ergonomics. Go to https://upliftdesk.com/lex https://lexfridman.com/s/uplift_desk-ep490-sc Fin: AI agent for customer service. Go to https://fin.ai/lex https://lexfridman.com/s/fin-ep490-sc Shopify: Sell stuff online. Go to https://shopify.com/lex https://lexfridman.com/s/shopify-ep490-sc CodeRabbit: AI-powered code reviews. Go to https://coderabbit.ai/lex https://lexfridman.com/s/coderabbit-ep490-sc LMNT: Zero-sugar electrolyte drink mix. Go to https://drinkLMNT.com/lex https://lexfridman.com/s/lmnt-ep490-sc Perplexity: AI-powered answer engine. Go to https://perplexity.ai/ https://lexfridman.com/s/perplexity-ep490-sc OUTLINE: (00:00) – Introduction (01:39) – Sponsors, Comments, and Reflections (16:29) – China vs US: Who wins the AI race? (25:11) – ChatGPT vs Claude vs Gemini vs Grok: Who is winning? (36:11) – Best AI for coding (43:02) – Open Source vs Closed Source LLMs (54:41) – Transformers: Evolution of LLMs since 2019 (1:02:38) – AI Scaling Laws: Are they dead or still holding? (1:18:45) – How AI is trained: Pre-training, Mid-training, and Post-training (1:51:51) – Post-training explained: Exciting new research directions in LLMs (2:12:43) – Advice for beginners on how to get into AI development & research (2:35:36) – Work culture in AI (72+ hour weeks) (2:39:22) – Silicon Valley bubble (2:43:19) – Text diffusion models and other new research directions (2:49:01) – Tool use (2:53:17) – Continual learning (2:58:39) – Long context (3:04:54) – Robotics (3:14:04) – Timeline to AGI (3:21:20) – Will AI replace programmers? (3:39:51) – Is the dream of AGI dying? (3:46:40) – How AI will make money? (3:51:02) – Big acquisitions in 2026 (3:55:34) – Future of OpenAI, Anthropic, Google DeepMind, xAI, Meta (4:08:08) – Manhattan Project for AI (4:14:42) – Future of NVIDIA, GPUs, and AI compute clusters (4:22:48) – Future of human civilization

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Lex Fridman?

  The following is a conversation all about the state of the art in artificial intelligence, including some of the exciting technical breakthroughs and developments in AI that happened over the past year, and some of the interesting things we think might happen this upcoming year. At times, it does get super technical, but we do try to make sure that it remains accessible to folks outside the field without ever dumbing it down. It is a great honor and pleasure to be able to do this kind of episode with two of my favorite people in the AI community, Sebastian Raschka and Nathan Lambert. They are both widely respected machine learning researchers and engineers

• 0:43Lex Fridman?

  who also happen to be great communicators, educators, writers, and Twitterers, X posters. Sebastian is the author of two books I highly recommend for beginners and experts alike. First is Build a Large Language Model From Scratch and Build a Reasoning Model From Scratch. I truly believe in the machine learning computer science world, the best way to learn and understand something is to build it yourself from scratch. Nathan is the post-training lead at the Allen Institute for AI and author of the definitive book on reinforcement learning from human feedback.

• 1:26Lex Fridman?

  Both of them have great X accounts, great Substacks. Sebastian has courses on YouTube, Nathan has a podcast, and everyone should absolutely follow all of those. And now, a quick few-second mention of each sponsor. Check them out in the description or at lexfridman.com/sponsors. It is, in fact, the best way to support this podcast. We've got a bunch of great sponsors. Box for intelligent content management. Quo for your phone system, like calls, texts, contacts for your business. Uplive Desk, the desk I'm sitting behind, and my favorite, Office Desk. Finn for customer service AI agents. Shopify for selling stuff online.

• 2:11Lex Fridman?

  CodeRabbit for AI-powered code review. Element for electrolytes. And of course, our longtime friend, Perplexity.

• 2:21Lex Fridman?

  For curiosity-driven knowledge exploration, choose wisely, my friends. And now on to the full ad reads. I try to make them interesting, but if you do skip, please still check out the sponsors. I enjoy their stuff. Maybe you will too. To get in touch with me, for whatever reason, go to lexfridman.com/contact. If you can't tell, I'm trying to have a bit of a pep in my step. At the moment, because I had a long night, didn't get much sleep at all. So I am running on fumes, delirious, happy, unsure of what is reality and what is a dream. In fact, we could, right now, be living inside of a dream.

• 3:05Lex Fridman?

  I have been going through a lot. I have been working insane hours, so much going on. I'm so overwhelmed, of course, as always, truly grateful and happy to be alive, but have not been able to publish as many episodes as I would like. So there's a bunch of sponsors we have to catch up on. Your support truly means the world. Please check out all the sponsors. If you think it might be useful to you, buy their stuff. It really is the best way to support this podcast. All right, let's go. First up, this episode is brought to you by Box, a cloud-based platform for content management, file sharing, and all kinds of collaboration, all kinds of content for your businesses.

• 3:49Lex Fridman?

  Like with a lot of companies, the big question is, how is AI leveraged to make whatever the business does better? A lot of companies kind of use it for the hype and the label. It's kind of hilarious to watch people just say like, powered by AI.

• 4:07Lex Fridman?

  I don't care if you're a bakery, powered by AI. I don't know. But outside of all of the hype, it is one of the most incredible things that humans have ever created. And so companies that can leverage that well are the companies that win. And of course, Box is legendary for its file and content management, especially when you're talking about scale. So obviously, it's amenable for the utilization of AI to help automate some of the document processing, some of the workflow, some of the organization. And they do that exceptionally well. They have a system called, as you could imagine, Box AI that does just that. I love it.

• 4:49Lex Fridman?

  They do an excellent implementation on the interface side, on the backend side. Everything works extremely nicely. Help scale AI across your organization today and go to box.com/ai. That's box.com/ai to learn more. This episode is also brought to you by QUO, spelled Q-U-O. Also happens to be a company name with just three letters that will help you win at Scrabble. Are you allowed to use company names in Scrabble? How many points is Q? How many points is U? I'm imagining a lot. That was one of the big confusions to me when I was first learning the English language.

• 5:32Lex Fridman?

  It always felt like Q should be at the end of the alphabet, maybe like QZ. It was always surprising to my limited brain capacity that Q was earlier on in the alphabet. What is it? OPQ? I can't even actually localize the letters in the alphabet. I'm sure that's the case for a lot of people without reading the alphabet in my head sequentially. All of this has to do with short-term and long-term memory access, the functioning, the limitation of human cognition, and maybe cognitive systems in general, all of it relevant to this particular episode, and not so relevant.

• 6:13Lex Fridman?

  To the awesomeness of QUO, formerly known as OpenPhone, that I should be talking about. Of course, as is always the case, I think the point here and the point everywhere in the point of life is to talk from the heart about whatever you want. And that's what I try to do with everything. And to generalize that even more, to talk whenever I want and to shut the F up whenever I want and listen. And I prefer that more often than I prefer to talk. Insert clever transition here because talk is somehow relevant. It is. So QUO, formerly known as OpenPhone,

• 6:55Lex Fridman?

  helps over 90,000 businesses manage phone calls, texts, contacts, all kinds of phone-related stuff for business. You have a bunch of customers, a bunch of incoming calls, a bunch of people on the business side that have to answer those calls, have to manage it. What's the status of this particular request? Voicemails, transcripts, all that kind of stuff. And obviously, really nice, effective utilization of AI to make that really efficient. But really, what's really important for things like this is that the interface is good, that team collaboration is good, and QUO delivered on that. Try QUO for free plus get 20% off your first six months when you go to quo.com/lex.

• 7:40Lex Fridman?

  That's Q U O dot com slash lex. Tell your friends about it because it just might help them win at Scrabble. Speaking of Scrabble, you usually want to play Scrabble on a table. It's such a magical experience. I just had a vision from a distant past of me sitting with a friend and playing Scrabble at a table. What is this life? Full of beautiful memories. And then it's over too soon.

• 8:09Lex Fridman?

  Yeah.

• 8:11Lex Fridman?

  That melancholy feeling is beautiful, I think. Insert another clever transition, Alamat Norman, maybe because of the name of this next company's Uplift Desk.

• 8:27Lex Fridman?

  Uh, as I said, uh, okay, it's my go to, uh, favorite office desk. And it's also the desk that I use for podcast furniture. I have, I already lost count. I have a lot of Uplift desks, standing desks in my place, everywhere. It's desks everywhere. I have a mattress on the floor and Uplift desks. So, um, I have a Linux box for robotics. I have a machine where I do a lot of the editing. All of that is on a desk. I have the three tables for the podcast desk, the, the very one you've seen over the past several years, that's all Uplift desks. I usually don't put them in standing mode, but they are standing desks.

• 9:09Lex Fridman?

  It allows me to do all kinds of stuff, really easy to work with, really nice material, really sturdy. I just love everything about Uplift Desk. When they said they want to sponsor, after I've been using them for many years, I lost my mind. I love it when I've been in love with a company, in love with their product for such a long time, and I get to also sing them praises. I mean, come on, what are you gonna tell me next? That FFmpeg wants to sponsor this podcast? Uh, another sort of open source project's not a company that I've been, uh, in love with. Anyway, go to upliftdesk.com/lex and use code Lex to get four free accessories, free

• 9:50Lex Fridman?

  same day shipping, free returns, a 15 year warranty and an extra discount off your entire order. That's U P L I F T D E S K dot com slash Lex. The spelling it out really helped anybody. I don't know, but they really said, pretty please, the one request is spell it out. Again, what is this life? Incredible.

• 10:18Lex Fridman?

  This episode is also brought to you by Finn, the number one AI agent for customer service. Find the niche and become number one. That's the idea here. Anybody building an AI company. And we talk about this. Is the dream of AGI dead? I think for a lot of companies, success is in the niche. But there is a few. And Finn delivers on that niche. It's trusted by over 6,000 customer service leaders at top companies, including AI companies. When an AI company trusts your company to do its customer service, that means you're legit. 90-day money back guarantee up to $1 million. Built to handle complex multi-step queries like returns,

• 11:03Lex Fridman?

  exchanges and disputes. Go to fin.ai/lex to learn more about transforming your customer service and scaling your support team. That's fin.ai/lex. I don't know why I switched to this hyping voice. Crappy announcer, crappy radio jockey, crappy ad read voice.

• 11:28Lex Fridman?

  It is what it is. Thank you for sticking with me this long. I feel the love and I send it right back at you.

• 11:38Lex Fridman?

  This episode is also brought to you by a company whose engineers are also full of love. Shopify.

• 11:48Lex Fridman?

  It just brings a smile to my face. Every time I think about Shopify, I got to see their engineering booth at NERIPs, which is a machine learning conference. Really brilliant people, wonderful people. Of course, the CEO Toby is still programming, still building stuff, still in on the details of the engineering. And now is talking quite a bit about utilization of LLMs for his own sort of pet projects, but also inside the company. It's just incredible when from the very top, the company is in love with engineering. It's a celebration of great engineering. Just like the conversation with DHH,

• 12:30Lex Fridman?

  who is the guy behind Ruby on Rails that Shopify was built on. That conversation was a celebration of great engineering. The beauty of engineering as well. Anyway, listen to that episode. To uh, to see some of the magic of Ruby on Rails and the magic of Shopify and the magic of Toby that we talk about. Anyway, sign up for a $1 per month trial period at Shopify.com/lex. That's all lowercase. Go to Shopify.com/lex to take your business to the next level today. This episode is also brought to you by CodeRabbit, a platform that provides AI-powered code reviews directly within your terminal. We talk a lot in this episode about the timeline

• 13:15Lex Fridman?

  for the full automation of the human programmer. I think we're quite far away from taking the human out of the loop. That review process, the debugging process, all of that, that's such a crucial part of programming, especially just like we talk about in the episode. When we're not talking about a personal website where HTML SOP is something that a web browser magically, automatically, I don't know how they're possibly able to do such incredible job of rendering SOP, but a web browser is in fact able to render SOP, including AI

• 13:58Lex Fridman?

  SOP. It just finds a way. So really, the question is, when you have production code, something that a lot of users are relying on, how do you review that code? How do you make sure you're catching the errors? How are you making sure that you put a backstop to hallucinations and the logical errors that AI coding agents can generate? Anyway, CodeRabbit supports all programming languages. Install CodeRabbit CLI today at coderabbit.ai/lex. That's coderabbit.ai/lex. This episode is also brought to you by Element, my daily zero-sugar and delicious electrolyte mix. It reminds me of the fact that I need to get to editing.

• 14:49Lex Fridman?

  The video of me in the jungle when Paul Roslin and I, such an incredible human. Congratulations to Paul on all of his success. Go get his book. It's an incredible book. Again, he's an incredible person with an incredible mission. And yes, I need to edit and publish, hoping to at the very least. The story of our journey in the jungle, because it was a beautiful celebration of nature and the jungle and friendship and the full richness of the human experience. It was beautiful. The reason I mention that is I was, as part of that journey, severely dehydrated.

• 15:27Lex Fridman?

  And I remember dreaming of element of a cold drink of water with electrolytes. Your body craves it and it craves it because it needs it. Electrolytes, sodium, potassium, magnesium. When you're deprived, it's not just water, it's electrolytes. So anyway, I always remember that. Get a free eight count sample pack with any purchase. Try it at drink element.com/lex. This is the Lex Fridman Podcast. To support it, please check out our sponsors in the description, where you can also find links to contact me, ask questions, give feedback and so on.

• 16:07Lex Fridman?

  And now, dear friends, here's Sebastian Raschka and Nathan Lambert.

• 16:30Lex Fridman?

  So I think one useful lens to look at all of this through is the DeepSeek, so-called DeepSeek moment. This happened about a year ago in January 2025, when the open weight Chinese company DeepSeek released DeepSeek R1 that I think it's fair to say surprised everyone with near or at state of the art performance with allegedly much less compute from much cheaper. And from then to today, the AI competition has gotten insane, both on the research level and the product level. It's just been accelerating. Let's discuss all of this today and maybe let's start with some spicy questions if we can. Who is winning at the international level?

• 17:14Lex Fridman?

  Would you say it's a set of companies in China or the set of companies in the United States? And Sebastian, Nathan, it's good to see you guys. So Sebastian, who do you think is winning?

• 17:26Speaker 01

  So winning is a very broad, you know, term. I would say you mentioned the DeepSeek moment, and I do think DeepSeek is definitely winning the hearts of the people who work on open weight models because they share these as open models.

• 17:40Speaker 01

  Winning, I think, has multiple time scales to it. We have today, we have next year, we have in 10 years. One thing I know for sure is that I don't think nowadays, 2026, that there will be any company who is, let's say, having access to a technology that no other company has access to. And that is mainly because researchers are frequently changing jobs, changing labs, they rotate. So I don't think there will be a clear winner in terms of technology access. However, I do think there will be, the differentiating factor will be budget and hardware constraints. So I don't think the ideas will be proprietary, but the way or the resources that are needed to implement them.

• 18:24Speaker 01

  And so I don't see currently take it all scenario where a winner takes it all. I can't see that at the moment.

• 18:32Lex Fridman?

  Nathan, what do you think?

• 18:33Speaker 02

  You see the labs put different energy into what they're trying to do. And I think to demarcate the point in time when we're recording this, the hype over Anthropic's Claude Opus 4.5 model has been absolutely insane, which is just, I mean, I've used it and built stuff in the last few weeks. And it's, it's almost gotten to the point where it feels like a bit of a meme in terms of the hype. And it's kind of funny because this is very organic. And then if we go back a few months ago, we can get the release date. And the notes is Gemini 3 from Google got released. And it seemed like the

• 19:06Speaker 02

  Marketing and just like wow factor of that release was super high. But then at the end of November, Claude Opus 4.5 was released and the hype has been growing. But Gemini 3 was before this and it kind of feels like people don't really talk about it as much, even though when it came out, everybody was like, this is Gemini's moment to retake kind of Google's structural advantages in AI. And Gemini 3 is a fantastic model and I still use it. It's just kind of differentiation is lower. And I agree with Sebastian, what you're saying with all these, like the idea space is very fluid, but culturally Anthropic is known for betting very hard on code, which is Claude code thing is working out for them right now. So I think that even if the ideas flow pretty freely, so much of this is bottlenecked by human

• 19:50Speaker 02

  effort and kind of culture of organizations where Anthropic seems to at least be presenting as the least chaotic. It's a bit of an advantage and if they can keep doing that for a while. But on the other side of things, there's a lot of ominous technology from China where there's way more labs than DeepSeek. So DeepSeek kicked off a movement within China, I say kind of similar to how ChatGPT kicked off a movement in the US where everything had a chatbot. There's now tons of tech companies in China that are releasing very strong frontier open weight models, to the point where I would say that DeepSeek is kind of losing its crown as the preeminent open model maker in China and the likes of Z dot AIs with their GLM

• 20:30Speaker 02

  models, Mini Max's models, Kimi Moonshot, especially in the last few months, have shown more brightly. The new DeepSeek models are still very strong, but that's kind of a, it could look back as a big narrative point where in 2025, DeepSeek came and then all, and it kind of provided this platform for way more Chinese companies that are releasing these fantastic models to kind of have this new type of operation. So these models from these Chinese companies are open weights and depending on this trajectory of business models that these American companies are doing, could be at risk. But currently, a lot of people are paying for AI software in the US and historically in China and other parts of the world, people don't pay a lot for software.

• 21:10Lex Fridman?

  So some of these models like DeepSeek have the love of the people because they are open weight. How long do you think the Chinese companies keep releasing open weight models?

• 21:20Speaker 02

  I would say for a few years, I think that like in the US, there's not a clear business model for it. I have been writing about open models for a while and these Chinese companies have realized it. So I get inbound from some of them and they're smart and realize the same constraints, which is that a lot of US tech companies and other IT companies won't pay for an API subscription to Chinese companies for security concerns. This has been a longstanding habit in tech. And the people at these companies then see open weight models as an ability to influence and take part of a huge, growing AI expenditure market in the US. And they're very realistic about this and it's working for them. And I think that the government will see that that is building a lot of influence internationally in terms of uptake of the technology.

• 22:04Speaker 02

  So there's going to be a lot of incentives to keep it going, but building these models and doing the research is very expensive. So at some point I expect consolidation, but I don't expect that to be a story of 2026, where there will be more open model builders throughout 2026 than there were in 2025. And a lot of the notable ones will be in China.

• 22:23Lex Fridman?

  You were going to say something?

• 22:24Speaker 01

  Um, yes, you mentioned DeepSeek losing its crown. I do think to some extent, yes, but we also have to consider though, they are still, I would say slightly ahead and the other ones, it's not that DeepSeek got worse. It's just like the other ones are using the ideas from DeepSeek. For example, you mentioned Kimi, same architecture, they're training it. And then again, we have this leapfrogging where they might be at some point in time a bit better because they have the more recent model. And I think this comes back to the fact that there won't be a clear winner. It will, it will just be like, like that. And one person releases something, the other one comes in and the recent, the most recent model is probably always the best model.

• 23:02Speaker 02

  Yeah, we'll also see the Chinese companies have different incentives. So like DeepSeek is very secretive, whereas some of these startups are like the mini maxes and Z dot AIs of the world. Those two literally have filed IPO paperwork and they're trying to get Western mind share and do a lot of outreach there. So I don't know if these incentives will kind of change the model development because DeepSeek famously is built by a hedge fund, a high flyer capital. And we don't know exactly what they, like, we don't know what they use the models for or if they care about this.

• 23:31Lex Fridman?

  They're secret in terms of communication. They're not secret in terms of the technical reports that describe how their models work. They're still open on that front. And we should also say on the Opus 4.5 hype, there's the layer of something.

• 23:46Lex Fridman?

  Being the darling of the X echo chamber on Twitter echo chamber and the actual amount of people that are using the model. I think it's probably fair to say that chat GPT and Gemini are focused on the broad user base that just want to solve problems in their daily lives. And that user base is gigantic. So the hype about the coding may not be representative of the actual use.

• 24:11Speaker 01

  I would say also a lot of the usage patterns are, like you said, name recognition, brand and stuff, but also muscle memory almost where, you know, like, chat GPT has been around for a long time, people just got used to using it and it's kind of like, almost like a flywheel, they recommend it to other users and that stuff. One interesting point is also the customization of L L M's. For example, chat GPT has a memory feature, right? And so you may have a subscription and you use it for personal stuff, but I don't know if you want to use that same thing at work, you know, because it's a boundary between private and work. If you're working at a company, they might not allow that or you may not want that. And I think that's also an interesting point where you might have multiple subscriptions. One, one is just clean code.

• 24:54Speaker 01

  It keeps, has nothing of your personal images that you or hobby projects in there. It's just like the work thing. And then the other one is your personal thing. So I think that's also something where two different use cases, and it doesn't mean you only have to have one. It's, I think the future is also multiple ones.

• 25:11Lex Fridman?

  What model do you think won twenty twenty five? And what model do you think is going to win twenty six?

• 25:16Speaker 02

  I think in the context of a consumer chatbots is a question of are you willing to bet on Gemini over chat GPT? Which I would say in my gut feels like a bit of a risky bet because OpenAI has been the incumbent and there's so many benefits to that in tech. But I think the momentum, if you look at twenty twenty five, was on Gemini's side, but they were starting from such a low point. I think R I P Bard and these earlier attempts of getting started. I think huge credit for them for powering through the organizational chaos to make that happen. But also it's hard to bet against chat to OpenAI because they always come off as so chaos.

• 25:56Speaker 02

  But they're very good at landing things. And I think like personally, I have very mixed reviews of GPT five, but it had to have saved them so much money. With the headline feature being a router where most users are no longer charging like charging their GPU costs as much. So I think it's very hard to dissociate the things that I like out of models versus the things that are going to actually be a general public differentiator.

• 26:23Lex Fridman?

  What do you think about 2026? Who's gonna win?

• 26:25Speaker 02

  I'll say something, even though it's risky, I will say that I think Gemini will continue to take progress on chat GPT. I think Google scale when both of these are operating. At such extreme scales and like Google has the ability to separate that research and product a bit better. We hear so much about AI being chaotic operationally and chasing the high impact thing, which is a very startup culture. And then on the software and enterprise side, I think Anthropic will have continued to success as they've again and again been set up for that. And obviously, Google's cloud has a lot of offerings, but I think this kind of like Gemini name brand is important for them to build. And and Google's cloud will continue to do well, but that's kind of a more complex thing to explain in the ecosystem.

• 27:08Speaker 02

  Because that's competing with the likes of Azure and AWS rather than on the model provider side.

• 27:13Lex Fridman?

  So in the infrastructure, you think GPUs give an advantage?

• 27:18Speaker 02

  Largely because the margin on Nvidia chips is insane and Google can develop everything from top to bottom to fit their stack and not have to pay this margin. And they've had a head start in building data centers. So all of these things that have both high lead times and very hard margins on high costs. Google has a just kind of a historical advantage there. And if there's going to be a new paradigm, it's most likely to come from OpenAI where they're kind of their research division again and again, has kind of shown this ability to land a new research idea or a product, I think, like deep research, so are a O one thinking models, like all of these definitional things have come from OpenAI and that's got to be one of their top.

• 27:59Speaker 02

  Trades as an organization. So it's kind of hard to bet against that, but I think a lot of this year will be about scale and optimizing what could be described as low hanging fruit in models.

• 28:09Lex Fridman?

  And clearly, there's a trade off between intelligence and speed. This is what ChatGPT five was trying to solve behind the scenes. It's like, do people actually want intelligence, the broad public, or do they want speed?

• 28:24Speaker 01

  I think it's a nice variety, actually, or the option to have a toggle there. I mean, first, for my personal usage, most of the time when I look something up, I use ChatGPT to ask a quick question, get the information, I want it fast. For, you know, most daily tasks, I use the quick model. Nowadays, I think the auto mode is pretty good, where you don't have to specifically say thinking or, you know, non-thinking and stuff. Then again, I also sometimes want the pro mode. Very often what I do is when I have something written, I put it into a ChatGPT and say, hey, do a very thorough check. Is all my references correct? Are all my thoughts correct? Did I make any formatting mistakes? And are the figure numbers wrong or something like that?

• 29:04Speaker 01

  And I don't need that right away. It's something, okay, I finished my stuff, maybe have dinner, let it run, come back and go through this. And I think, see, this is where I think it's important to have this option. I would go crazy if for each query, I would have to wait thirty minutes or ten minutes.

• 29:19Speaker 02

  That's me.

• 29:19Speaker 01

  Yeah.

• 29:21Speaker 02

  I'm like sitting over here losing my mind that you use the router and the non-thinking model. I'm like, how do you, how do you live with, how do you live with that?

• 29:28Lex Fridman?

  Yeah.

• 29:28Speaker 02

  It's like my reaction, I'm been heavily on ChatGPT for a while.

• 29:34Speaker 02

  Never touched five non-thinking. I find it, it's tone and then it's propensity of errors. It's just like has a higher likelihood of errors. Some of this is from back when OpenAI released O three, which was the first model to do this deep search and find many sources and integrate them for you. So I became habituated with that. So I will only use GPT five point two thinking or pro. When I'm finding any sort of information query for work, whether that's a paper or some code reference that I found. And it's just like, I will regularly have like five pro queries going simultaneously, each looking for one specific paper. Back on the equation or something.

• 30:10Speaker 01

  I have a funny example where I just needed to answer as fast as possible for this podcast before I was going on the trip. I have a local GPU running at home and I wanted to run a long RL experiment. And usually I also unplug things because you never know if you're not at home, you don't want to have things plugged in. And I accidentally unplugged the GPU. It was like my wife was already in the car and it's like, oh dang. And then basically I wanted as fast as possible a bash script that runs my different experiments and the evaluation. And I did something I know I learned how to use bash interface or bash terminal. But in that moment, I just needed like 10 seconds. Give me the command.

• 30:51Lex Fridman?

  This is a hilarious situation, but yeah. So what did you use?

• 30:54Speaker 01

  So I did the non-thinking fastest model. It gave me the bash command to chain different scripts to each other. And then the thing is like you have the T thing where you want to route this to a log file. Top of my head, I was just like in a hurry. I could have thought about it.

• 31:10Lex Fridman?

  By the way, I don't know if there's a representative case wife waiting in the car. You have to run, you unplug the GPU. You have to generate a bash script. This sounds like a movie, like Mission Impossible.

• 31:19Speaker 02

  I use Gemini for that. So I use Thinking for all the information stuff and then Gemini for fast things or stuff that I could sometimes Google. Which is like, it's good at explaining things and I trust that it has this kind of background of knowledge and it's simple and the Gemini app has gotten a lot better and it's good for that sort of things. And then for code and any sort of philosophical discussion, I use Claude Opus 4.5, also always with extended thinking. Extended thinking and inference time scaling is just a way to make the models marginally smarter and I will always edge on that side when the progress is very high because you don't know when that'll unlock a new use case. And then sometimes use Grok for real time information or finding something on AI Twitter that I knew I saw and I need to dig up and I just fixated on.

• 32:03Speaker 02

  Although when Grok 4 came out, the Grok 4, what is super heavy, which was like their pro variant, was actually very good. And I was pretty impressed with it and I just kind of like muscle memory, lost track of it with having the chat GPT app open. So I use any different things.

• 32:18Lex Fridman?

  Yeah, I actually do use Grok 4 heavy for debugging, for like hardcore debugging and the other ones can't solve it. I find that it's the best app and I, it's interesting because you say chat GPT is the best interface. Uh for me, for that same reason, but this could be just momentum. Gemini is the better interface for me. I think because I fell in love with their best needle in the haystack. If I ever put something that has a lot of context, but I'm looking for a very specific kinds of information, make sure it tracks all of it. I find at least the Gemini for me has been the best.

• 32:56Lex Fridman?

  So it's funny with some of these models, if they win your heart over for one particular feature at one on a one particular day. For that particular query, that prompt, you're like, this model is better. And so you'll just stick with it for a bit until it does something really dumb. There's like a threshold effect, some smart thing, and then you fall in love with it. And then it does some dumb thing and you're like, you know what? I'm going to switch and try Claude and ChatGPT and all that kind of stuff.

• 33:23Speaker 01

  This is exactly like you use it until it breaks, until you have a problem. And then, then you change the LM. And I think it's the same how we use anything like our favorite text editor, um operating systems or the browser. I mean, there are so many browser options, Safari, Firefox, Chrome, all the relatively similar. But then there are edge cases, maybe extensions you want to use and then you switch. But I don't think there is any one who types the same thing, like the website into different browsers and compares them. You only do that when the website doesn't render if something breaks, I think. So that's, that's a good point. I think you use it until it breaks and then you explore other options, I think.

• 34:01Speaker 02

  On the long context thing, I was also a Gemini user for this, but the GPT five point two release blog had like crazy long context scores. Where a lot of people were like, did they just figure out some algorithmic change? It went from like thirty percent to like seventy percent or something in this minor model update. So it's also very hard to keep track of all of these things. But now I'm look more favorably at GPT five point two's long context. So it's just kind of like, how do I actually get to testing this? Never ending battle.

• 34:30Lex Fridman?

  It's interesting that none of us talked about the Chinese models from a user usage perspective. What does that say? Does that mean the Chinese models are not as good? Or does that mean we're just very biased and US focused?

• 34:44Speaker 01

  I do think that that's currently the discrepancy between just the model and the platform. So I think the open models, they are more known for the open weights, not their platform yet.

• 34:54Speaker 02

  There are also a lot of companies that are willing to sell you the open model inference at a very low cost. I think like Open Router, it's easy to do the look at multi-model things. You can run DeepSeek on perplexity. I think all of us sitting here are like, we use OpenAI GPT five Pro consistently. We're all willing to pay for the marginal intelligence gain. And anyone that's like the, these models from the US are better. And in terms of the outputs, I think that the question is, will they stay better for this year and for years going? But it's like, so long as they're better, I'm going to pay for it to use them. I think there's also analysis that shows that like the way that the Chinese models are served, this you could argue due to expert controls or not,

• 35:35Speaker 02

  is that they use fewer GPUs for replica, which makes them slower and have different errors. And it's like the speed and intelligence, if these things are in your favor as a user, I think in the US, a lot of users will go for this. And I think that that is something that will spur these Chinese companies to want to compete in other ways, whether it's like free or substantially lower costs, or it'll breed creativity in terms of offerings, which is good for the ecosystem. But I just think the simple thing is the US models are currently better and we use them. And I try Chinese, I tried these other open models and I'm like, fun, but not gonna, I don't go back to it.

• 36:11Lex Fridman?

  We didn't really mention programming. That's another use case that a lot of people deeply care about. So I use basically half and half cursor and clog code. Because there I find them to be like fundamentally different experience and both useful. What do you guys, you program quite a bit, so what do you use? What's the current vibe?

• 36:31Speaker 01

  So I use the Codex plugin for VS Code. You know, it's very convenient. It's just like a plugin and then it's a chat interface. Has access to your repository. I know that Claude Code is, I think, a bit different. It is a bit more agentic, it touches more things, it does a whole project for you. I'm not quite there yet where I'm comfortable with that because maybe I'm a control freak, but I still would like to see a bit what's going on. And Codex is kind of like right now for me, like the sweet spot where it is helping me, but it is not taking completely over.

• 37:02Lex Fridman?

  I should mention, one of the reasons I do use Claude Code is to build the skill of programming with English. I mean, the experience is fundamentally different. You're, as opposed to micromanaging the details of the process of the generation of the code and looking at the diff, which you can in Cursor, if that's the ID you use, and in changing, altering, looking and reading the code and understanding the code deeply as you progress versus just kind of like thinking in this design space and just guiding it at this macro level, which I think is another way of thinking about the programming process.

• 37:43Lex Fridman?

  Also, we should say that Claude Code, it just seems to be somehow a better utilization of Claude Open 4.5.

• 37:51Speaker 02

  It's a good side by side for people to do. So you can have Claude Code open, you can have Cursor open, and you can have VS Code open, and you can select the same models on all of them and ask questions. It's very interesting, like Claude Code is way better in that domain. It's remarkable.

• 38:05Lex Fridman?

  All right, we should say that both of you are legit on multiple fronts. Researchers, programmers, educators, tweeters.

• 38:15Lex Fridman?

  And on the book front too. So Nathan, at some point soon, hopefully, has an RHF book coming out.

• 38:23Speaker 02

  It's available for preorder and there's a full digital pre-print. Just making it pretty and better organized for the physical thing, which is a lot of why I do it, because it's fun to create things that you think are excellent in the physical form when so much of our life is digital.

• 38:37Lex Fridman?

  I should say, going to perplexity here. Sebastian Raschka is a machine learning researcher and author known for several influential books. A couple of them that I wanted to mention, which is A book I highly recommend, Build a Large Language Model From Scratch, and the new one, Build a Reasoning Model From Scratch. So I'm really excited about that. Building stuff from scratch is one of the most powerful ways of learning.

• 39:00Speaker 01

  Honestly, building an L L M from scratch is a lot of fun. It's also a lot of to learn. And like you said, it's probably the best way to learn how something really works. Because you can look at figures, but figures can have mistakes. You can look at concepts, explanations, but you might misunderstand them. But if you see the code, there is code and the code works, you know it's correct. I mean, there's no misunderstanding. It's like it's precise, otherwise it wouldn't work. And I think that's like, kind of like the beauty behind coding. It is kind of like, it doesn't lie. It's math, basically. So even though with math, I think you can have mistakes in a book, you would never notice. Because you're not running the math when you're reading the book, you can't verify this. And with code, what's what's nice is you can verify it.

• 39:41Lex Fridman?

  Yeah, I agree with you about the L L M from scratch book. It's nice to tune out everything else, the internet and so on, and just focus on the book. But, you know, I read several, like, you know, history books. It's just less lonely somehow. It's really more fun. Like, for example, on the programming front, I think it's genuinely more fun to program with an L L M. And I think it's genuinely more fun to read with an L L M. But you're right, like, this distraction should be minimized. So it's, you use the L L M to basically enrich the experience, maybe add more context.

• 40:20Lex Fridman?

  Maybe the guys, the rate of aha moments for me in a small scale is really high with L L M's.

• 40:27Speaker 01

  Hundred percent. I would, I also want to correct myself. I'm not suggesting not to use L L M's. I suggest doing it in multiple passes, like one pass, just offline focus mode. And then after that, I mean, I also take notes, but I, I try to resist the urge to immediately look things up. I do a second pass. It's just like for me, most structured this way. And I get, I mean, sometimes things are answered in the chapter, but sometimes also it just helps to let it sink in and think about it. Other people have different preferences. I would.

• 40:57Speaker 02

  Highly recommend using L L Ms when reading books. For me, it's just, it's not the first thing to do. It's like the second pass.

• 41:02Lex Fridman?

  By way of recommendation, as I say, I do the opposite. I like to use the L M at the beginning to lay out the full context of like, what is this world that I'm now stepping into? But I try to avoid clicking out of the L L M into the world of like Twitter and blogs. And because then you're now down this rabbit hole, you're reading somebody's opinion. There's a flame war about a particular topic and all of a sudden you're no longer, you're now in the, in the realm of the internet and Reddit and so on. But if you're purely letting the L L M give you the context of why this matters, what are the big picture ideas? But sometimes books themselves are good at doing that, but not always. So.

• 41:44Speaker 02

  This is why I like the chat G P T app that gives the A I a home in your computer when you are folk, you can focus on it rather than just being another tab in my mess of internet options. And I think Claude Code in these particular does a good job of making that a joy where it seems very engaging as a product design to be an interface that your A I will then go out into the world. And it's something that is very kind of intangible between it and Codex is that it just feels kind of warm and engaging where Codex can often be as good from Open A I, but it just kind of like feels a little bit rougher on the edges where it's like Claude Code is makes it fun to build things, particularly from scratch where you just don't like, you don't have to care, but you trust that it'll make something like, obviously it's good for websites and

• 42:29Speaker 02

  kind of refreshing tooling and stuff like this, which I use it for or data analysis. So I, my, my blog, we scrape hugging face. We keep the download numbers for every data set and model over time now. So we have them and it's like Claude was just like, yeah, I've made use of that data. No problem. And I was like, that would have taken me days. And it's like, then I have enough situational awareness to be like, okay, these trends obviously make sense and you can check things. Cause that's just the kind of wonderful interface where you can have an intermediary and not have to do the kind of awful low level work that you would have to do to maintain different web projects and do this stuff.

• 43:02Lex Fridman?

  All right. So we just talked about a bunch of the closed weight models.

• 43:08Lex Fridman?

  Let's talk about the open ones. So tell me about the landscape of open L L models. Which are interesting ones, which stand out to you and why? Where do you mention DeepSeek?

• 43:17Speaker 01

  Do you want to see how many we can name off the top of our head?

• 43:19Lex Fridman?

  Yeah, without looking at notes.

• 43:21Speaker 01

  DeepSeek, Kimi, Minimax, Z dot AI, Ant Ling. We're just going Chinese.

• 43:30Speaker 01

  Let's throw in Mistral AI, Gemma, GPT OSS, the open source model by ChatGPT. Actually, Nvidia Nematron had a, or Nvidia had a really cool one, Nematron three. There's a lot of stuff, especially at the end of the year. Qwen, one maybe the one.

• 43:45Speaker 02

  Oh, yeah. Qwen was the name, the obvious name. I was trying to get through the, you can get at least 10 Chinese and at least 10 Western. I think that, I mean, OpenAI released their first open model since GPT two. That was when I, when I meant talk, when I was writing about OpenAI's open model release, they're all like, don't forget about GPT two, which I thought was really funny because it's just such a different time. But GPT OSS is actually a very strong model and does some things that the other models don't do very well. And I think that selfishly, I'll promote a bunch of like Western companies. So both in the US and Europe have these like fully open models. So I work at Allen Institute for AI, where I've been building OMO, which releases data and code and all of this. And now we have actual competition for people that are trying to release everything so that other people can train these models.

• 44:30Speaker 02

  So there's the Institute for Foundation Models or slash L L three sixty, which is like had their K two models of various types. Apertus is a Swiss research consortium. Hugging Face has small L M, which is very popular. And Nvidia's Nematron has started releasing data as well. And then Stanford's Marin Community Project, which is kind of making it. So there's a pipeline for people to open a GitHub issue and implement a new idea and then have it run in a stable language modeling stack. So this space. That list was way smaller in 2024. So I think it was like just AI two. So that's a great thing for more people to get involved and to understand language models, which doesn't really have a like a Chinese company that is has an analog.

• 45:14Speaker 02

  While I'm talking, I'll say that the Chinese open language models tend to be much bigger and that gives them this higher peak performances MOEs where a lot of these things that we like a lot, whether it was Gemma in Nematron have tended to be smaller models from the US, which is which is starting to change from US and Europe. Mr. Large three came out, which was a giant MOE model, very similar to DeepSeek Architecture in December. And then a startup RCA and both Nematron have Nematron and Nvidia have teased MOE models of this way bigger than 100 billion parameters, like this 400 billion parameter range coming in this like Q1 2026 timeline. So I think this kind of balance is set to change

• 45:56Speaker 02

  this year in terms of what people are using the Chinese versus US open models for, which will be which I'm personally like going to be very excited to watch.

• 46:05Lex Fridman?

  First of all, huge props for being able to name so many of these. Did you actually name Lama?

• 46:12Speaker 02

  No.

• 46:12Lex Fridman?

  I feel like.

• 46:13Speaker 02

  RIP.

• 46:14Speaker 01

  This was not on purpose.

• 46:15Lex Fridman?

  RIP Lama. All right. Can you mention what are some interesting models that stand out? So you mentioned Quin three is is obviously a standout.

• 46:24Speaker 01

  So I would say the year is almost bookended by both DeepSeek version three and R1. And then on the other hand, in December, DeepSeek version 3.2, because what I like about those is they always have an interesting architecture tweak that others don't have. But otherwise, if you want to go with, you know, like the familiar but really good performance, Quinn three and like Nathan said, also GPT OSS. And I think GPT OSS, what's interesting about it is kind of like the first public or like open weight model that was really trained with tool use in mind, which I do think is kind of a little bit of a paradigm shift where the ecosystem was not quite ready for it. So with tool use, I mean that the LLM is able to do a web search to call a Python interpreter.

• 47:06Speaker 01

  And I do think this it's a standout because I think it's a huge unlock because. One of the most common complaints about L L M's are, for example, hallucinations, right? And so in my opinion, one of the best ways to solve hallucinations is to not try to always remember information or make things up. For math, why not use a calculator app or Python? If I asked the L L M, who won the soccer World Cup in nineteen ninety eight? Instead of just trying to memorize, it could go do a search. I think mostly it's usually still a Google search. So G P D, G P D O S S, they would do a tool call to Google, maybe find the FIFA website, find, okay, it was France. It would get you that information reliably instead of just trying to memorize it.

• 47:49Speaker 01

  So I think it's a huge unlock, which I think right now is not fully utilized yet by the open source, open weight ecosystem. A lot of people don't use tool call modes because I think it's first is a trust thing. You don't want to run this on your computer where it has access to tools, could wipe your hard drive or whatever. So you want to maybe containerize that. But I do think, you know, that that is like a really important step for the upcoming years to have this ability.

• 48:16Lex Fridman?

  So a few quick things. First of all, thank you for defining what you mean by tool use. I think that's a great thing to do in general for the concepts we're talking about. Even things as sort of well established as M O Es. You have to say that means mixture of experts and you could kind of have to build up an intuition for people, what that means, how it's actually utilized, what are the different flavors? So what does it mean that there's this such explosion of open models? What's your intuition?

• 48:46Speaker 02

  If you're releasing an open model, you want people to use it as the first and foremost thing. And then after that comes things like transparency and trust. I think when you look at China, the biggest reason is that they want people around the world to use these models. And I think a lot of people will not. If you look outside of the U S, a lot of people will not pay for software, but they might have computing resources where you can put a model on it and run it. I think there can also be data that you don't want to send to the cloud. So this, the, the number one thing is getting people to use models, use A I or use your A I that might not be able to do it without having access to the model.

• 49:18Lex Fridman?

  I guess we should state explicitly, so we've been talking about these Chinese models and open weight models. Oftentimes, the way they're run is locally. So it's not like you're sending your data to China or to whoever developed to Silicon Valley, whoever developed the model.

• 49:37Speaker 02

  A lot of American startups make money by hosting these models from China and selling them, selling tokens. It's called like selling tokens, which means somebody will call the model to do some, some piece of work. I think the other reason is for US companies like ChatGPT, OpenAI is so GPU deprived. Like they're so, they're at the limits of the GPUs. Whenever they make a release, they're always talking about like, our GPUs are hurting. And I think there's like, like in one of these like GPT OSS release sessions, Sam Altman said like, oh, we're releasing this because we can use your GPUs. We don't have to use, we don't have to use our GPUs and OpenAI can still get distribution out of this, which is another very real thing. This doesn't cost them anything.

• 50:16Speaker 01

  And for the user, I think also, I mean, there are users who just use the model locally how they would use GPT, but also for companies, I think it's a huge unlock to have these models because you can customize them. You can train them, you can add post-training, add more data, like specialize them into, let's say, law, medical models, whatever you have. And the appeal, you mentioned Llama, the appeal of the open weight models from China is that the open weight models are also, the licenses are even friendlier. I think they are just unrestricted open source licenses where if you use something like Llama or Gemma, there are some strings attached. I think it's like an upper limit in terms of how many users you have. And then if you exceed, I don't know, so many million users, you have to report your finance

• 50:59Speaker 01

  situation to, let's say, Meta or something like that. And I think, well, it is a free model, but there are strings attached and people do like things where strings are not attached. So I think that's also one of the reasons besides performance, why the open weight models from China are so popular, because you can just use them. There is no, there's no catch in that sense.

• 51:19Speaker 02

  The ecosystem has gotten better on that front, but mostly downstream of these new providers providing such open licenses. That was funny when you pulled up perplexity, it said Kimi K two thinking hosted in the US, which is just like an exact, I've never seen this, but it's an exact example of what we're talking about, where people are sensitive to this. But Kimi K two thinking and Kimi K two is a model that is very popular. People say that has very good like creative writing and also in doing some software things. There's just these little quirks that people pick up on with different models that they like.

• 51:47Lex Fridman?

  What are some interesting ideas that some of these models have explored that you can speak to? Like that particular interesting to you?

• 51:54Speaker 01

  Maybe you can go chronologically. I mean, there was, of course, DeepSeek, DeepSeek R1 that came out in January. If we just focus on 2025. However, this was based on DeepSeek version three, which came out the year before in December 2024. There are multiple things on the architecture side. What is fascinating is you can still, I mean, that's what I do with my from scratch coding projects. You can still start with GPT two and you can add things to that model to make it into this other model. So it's all still kind of like the same lineage, the same, it is a very close relationship between those. But top of my head, DeepSeek, what was unique there is the mixture of experts. I mean, they were not inventing mixture of experts. We can maybe talk a bit more what mixture of experts means.

• 52:37Speaker 01

  But just to list these things first before we dive into detail. Mix of experts, but then they also had a multi head latent attention, which is a tweak to the attention mechanism. Where this was, I would say, 2025, the main distinguishing factor between these open weight models. Different tweaks to make inference or KV cache size. We can also define KV cache in a few moments. But to kind of make it more economical to have long context, to shrink the KV cache size. So what are tweaks that we can do? And most of them focused on the attention mechanism. There is multi head latent attention in DeepSeek. There is a group query attention, which is still very popular. It's not invented by any of those models. It goes back a few years, but that would be the other option.

• 53:22Speaker 01

  Sliding window attention, I think, almost reuses it if I remember correctly. So there are these different tweaks that make the models different. Otherwise, I put them all together in an article once where I just compared them. They are very surprisingly similar. It's just different numbers in terms of how many repetitions of the transformer block you have in the center. And like just little, little knobs that people tune. But what was so nice about it is it's, it works no matter what. You can tweak things, you can move the normalization layers around, you get some performance gains. And I almost always Very good in ablation studies showing what actually what it does to the model if you move something around.

• 54:03Speaker 01

  Ablation studies doesn't make it better or worse, but there are so many, let's say, ways you can implement a transformer and make it still work. Big ideas that are still prevalent is mixture of experts, multi-atlaten attention, sliding window attention, group query attention. And then at the end of the year, we saw a focus on making the attention mechanism scale linearly with inference token prediction. So there were Qwen three next, for example, which added a gated delta net. It's it's like kind of like inspired by state space models where you have a fixed state that you keep updating, but it makes essentially this attention cheaper or it replaces attention with a cheaper operation.

• 54:41Lex Fridman?

  And it maybe is it useful to step back and talk about transform architecture in general?

• 54:46Speaker 01

  Yeah, so maybe we should start with the GPT two architecture, the transformer that was derived from the tension is all you need paper. So the tension is all you need paper had a transformer architecture that had two parts, an encoder and a decoder. And GPT went just focusing in on the decoder part. It is essentially still a neural network and it has this attention mechanism inside. And you predict one token at a time, you pass it through an embedding layer. There's the transformer block, the transformer block has attention modules and a fully connected layer. And there are some normalization layers in between, but it's essentially neural network layers with this attention mechanism.

• 55:26Speaker 01

  So coming from GPT two, when we move on to GPT OSS, there is, for example, the mixture of experts layer. It's not invented by GPT OSS, it's a few years old, but it is essentially a tweak to make the model larger without consuming more compute in each forward pass. So there is this fully connected layer. And if listeners are familiar with multi-layer perceptrons, you can think of a mini multi-layer perceptron, a fully connected neural network layer inside the transformer. And it's very expensive because it's fully connected. If you have thousand inputs, thousand outputs, it's like a one million connections. And it's a very expensive part in this transformer and the idea is.

• 56:09Speaker 01

  To kind of expand that into multiple feed forward networks. So instead of having one, let's say you have 256, but it would make it way more expensive. Because now you have 256, but you don't use all of them at the same time. So you now have a router that says, okay, based on this input token, it would be useful to use this fully connected network. And in that context, it's called an expert. So a mixture of experts means you have multiple experts. And depending on what your input is, let's say it's more math heavy, it would use different experts compared to, let's say, translating input text from English to Spanish, it would maybe consult different experts. It's not quite clear, I mean, as clear cut to say, okay, this is only an expert for math and for Spanish is a bit more fuzzy.

• 56:53Speaker 01

  But the idea is essentially that you pack more knowledge into the network, but not all the knowledge is used all the time. That would be very wasteful. So you're kind of like during the token generation, you're more selective. There's a router that selects which tokens should go to which expert. It's more complexity, it's harder to train, there's a lot of, you know, that can go wrong, like collapse and everything. So I think that's why almost three still uses dense. I mean, you have, I think, all the models with mixture of experts, but dense models where dense means also, also it's jargon. There's a distinction between dense and sparse. So mixture of experts is considered sparse because we have a lot of experts, but only few of them are active. So that's called sparse.

• 57:35Speaker 01

  And then dense would be the opposite, where you only have like one fully connected module and it's always, you know, utilized.

• 57:41Lex Fridman?

  So maybe this is a good place to also talk about KV cache, but actually before that, even zooming out, like fundamentally, how many new ideas have been implemented from, from GPT two to today? Like how different really are these architectures?

• 57:58Speaker 01

  Picture like the mixture of experts, the attention mechanism in GPT OSS, that would be the group query attention mechanism. So it's a slight tweak from multi-head attention to group query attention. So that we have two, I think they replaced layer norm by RMS norm, but it's just like a different normalization there. And not a big change, it's just like a tweak, the nonlinear. Activation function, people familiar in with deep networks, I mean, it's the same as changing sigmoid with ReLU. It's not changing the network fundamentally, it's just like a tweak, a little, little tweak. And that's about it, I would say. It's not really fundamentally that different. It's still the same, same architecture. So you can convert one from one, you can go from one into the other by just adding these, these changes, basically.

• 58:42Lex Fridman?

  It's fundamentally is still the same architecture.

• 58:45Speaker 01

  Yep. So for example, you mentioned my book earlier, that's a GPT two model in the book because it's simple and it's very small. So one hundred twenty four, one hundred twenty million parameters approximately. But in the bonus materials, I do have almost three from scratch, Gemma three from scratch and other types of from scratch models. And I always started with my GPT two model and just, you know, tweaked or added different components and you get from one to the other. It's like, it's kind of like a lineage in a sense, yeah.

• 59:10Lex Fridman?

  Can you build up an intuition for people? Because sort of when you zoom out, you look at it, there's so much rapid advancement in the AI world. And at the same time, fundamentally, the architectures have not changed. So where is all the turbulence, the turmoil of the advancement happening? Where, where's the gains to be had?

• 59:33Speaker 01

  So there are the different stages where you develop the network or train the network. You have the pre-training. Now, back then, they was just pre-training with GPT two. Now you have pre-training, mid-training and post-training. So I think right now we are in the post-training focus stage. I mean, pre-training still gives you advantages if you scale it up to better, higher quality data. But then we have capability unlocks that were not there with GPT two. For example, ChatGPT, it is basically a GPT three model and GPT three is the same as GPT two in terms of architecture. What was new was adding the supervised fine-tuning and the reinforcement learning with human feedback. So it's more on the algorithmic side rather than the architecture.

• 1:00:17Speaker 02

  I would say that the systems also change a lot. I think if you listen to Nvidia's announcements, they talk about these things like you now do FP eight, you can now do FP four. And what is happening is these labs are figuring out how to utilize. More compute to put it into one model, which lets them train faster and that lets them put more data in. And then you can find better configurations faster by doing this. So you can look at like the essentially the tokens per second per GPU is a metric that you look at when you're doing large scale training. And you could get, you can go from like 10K to 13K by turning on FPA training, which means you're using less memory per parameter in the model. And by saving less information, you do less communication, you can train faster.

• 1:00:57Speaker 02

  So all of these like system things underpin way faster experimentation on data and algorithms. That is kind of like, it's this, it's this kind of loop that keeps going where it's kind of hard to describe when you look at the architecture and they're exactly the same. But the code base used to train these models is going to be vastly different. And you could probably, like, I don't, the GPUs are different, but you probably train GPT OSS 20B way faster in wall clock time than GPT 2 was trained at the time.

• 1:01:27Speaker 01

  Yeah, like you said, they had, for example, in the mixture of experts is NVFP4 optimization, for example, where you get more throughput. But I do think this is for the speed, this is true, but it doesn't give the model new capabilities in a sense. It's just how much can we make, make the computation coarser without suffering in terms of model performance degradation. But I do think, I mean, there are alternatives popping up to the transformer. There's text diffusion models, completely different paradigm. And there's also, I mean, the text diffusion models might use transformer architectures, but it's not an auto regressive transformer. And also Mamba models, it's a state space model, but they do have trade offs.

• 1:02:08Speaker 01

  And what's right is there's nothing that has replaced the auto regressive transformer as state of the art model. So like for state of the art, you would still do that, go with that thing. But there are now alternatives for the cheaper and like alternatives that are kind of making compromises. But it's not just one architecture anymore. There are little ones coming up, but if we talk about the state of the art, it's pretty much still the. The transformer architecture autoregressive derived from GPT two essentially.

• 1:02:38Lex Fridman?

  I guess the big question here is we talked quite a bit here on the architecture behind the pre-training. Are the scaling laws holding strong across pre-training, post-training inference, contact size, data, synthetic data?

• 1:02:53Speaker 02

  I'd like to start with the technical definition of scaling law, which kind of informs all of this. The scaling law is a power law relationship between, you can think of the X axis. So kind of what you are scaling is a combination of compute and data, which are kind of similar. And then the Y axis is like the held out prediction accuracy over our next tokens. We talked about models being autoregressive. It's like, if you keep a set of text that the model has not seen, how accurate will it get when you will train? And the idea of scaling laws came when people figured out that that was a very predictable relationship. And I think that that technical term is continuing. And then the question is like, what do users get out of it?

• 1:03:33Speaker 02

  And then there are more types of scaling where OpenAI's O1 was famous for introducing inference time scaling. And I think less famously for also showing that you can scale reinforcement learning training and get kind of this log X axis and then a linear increase in performance on Y axis. So there's kind of these three axes now where the traditional scaling laws are talked about for pre-training, which is how big your model is and how big your dataset is. And then scaling reinforcement learning, which is like, how long can you do this trial and error learning? That we will talk about, will define more of this. And then this inference time compute, which is just letting the model generate more tokens on a specific problem. So I'm kind of bullish where they're all really still working, but the low hanging fruit has mostly been taken,

• 1:04:16Speaker 02

  especially in the last year on reinforcement learning with verifiable rewards, which is this RLVR. And then inference time scaling, which is just why these models feel so different to use. Where previously you would get that first token immediately. And now they'll go off for seconds, minutes, or even hours, generating these hidden thoughts before giving you the first word of your answer. And that's all about this inference time scaling, which is such a wonderful kind of step function in terms of how the models change abilities. They kind of enabled this tool use stuff and enabled this much better software engineering that we were talking about. And this is when we say enabled, almost entirely downstream of the fact that this reinforcement learning with verifiable rewards training just kind of.

• 1:04:59Speaker 02

  Let the models pick up these skills very easily. So let the models learn. So if you look at the reasoning process when the models are generating a lot of tokens, what it will be often doing is it tries a tool, it looks at what it gets back, it tries another API, it sees what it gets back and if it solves the problem. So the models, when you're training them, very quickly learn to do this. And then at the end of the day, that gives this kind of general foundation where the model can use CLI commands very nicely in your repo and handle Git for you and move things around and organize things or search to find more information, which if we're sitting in these chairs a year ago, it's something that we didn't really think of the models being doing. So this is just kind of something that has happened this year and is

• 1:05:41Speaker 02

  totally transformed how we think of using AI, which I think is very magical, is such an interesting evolution and just so unlocks so much value. But it's like, it's not clear what the next avenue will be in terms of unlocking stuff like this. I think that there's there's we'll get to continual learning later, but there's a lot of buzz around certain areas of AI, but no one knows when the next step function will really come.

• 1:06:04Lex Fridman?

  So you you've actually said quite a lot of things there and said profound things quickly. It would be nice to unpack them a little bit. You said you're bullish basically on every version of scaling. So we just even start at the beginning. Pre-training.

• 1:06:22Lex Fridman?

  Are we kind of implying that the low hanging fruit on pre-training scaling has been picked? Is is is pre-training hit a plateau or is even pre-training still you're bullish on?

• 1:06:33Speaker 02

  Pre-training has gotten extremely expensive. I think to scale up pre-training, it's also implying that you're going to serve a very large model to the users. So I think that it's been loosely established, the likes of GPT-4 and similar models where around one trillion, like this order of trillion parameters at the biggest size. There's a lot of rumors that they've actually gotten smaller as training has gotten more efficient. You want to make the model smaller because then your costs of serving go down proportionally. These models, the cost of training them is really low relative to the cost of serving them to hundreds of millions of users. I think DeepSeek had this famous number of about $5 million for pre-training at cloud market rates.

• 1:07:12Speaker 02

  I think almost three, section 2.4 in the paper, we just detailed how long we had the GPU clusters sitting around for training, which includes engineering issues, multiple seeds. And it was like about $2 million to rent the cluster to like deal with all the problems and headaches of training a model. So these models are pretty, like, a lot of people could get one to $10 million to train a model. But the recurring costs of serving millions of users is really billions of dollars of compute. I think that you can look at like 1,000 GPU rental, you can pay 100 grand a day for. And these companies could have millions of GPUs. You can look at how much these things cost to sit around. So that's kind of a big thing.

• 1:07:54Speaker 02

  And then it's like, if scaling is actually giving you a better model, like, is it going to be financially worth it? And I think it'll kind of slowly will push it out as AI solves more compelling tasks. So like the likes of Claude Opus 4.5, making Claude code just work for things. I think I've launched this project called like the Atom Project, which is like American Truly Open Models in July. And that was like a true vibe-coded website. Like I have a job, make plots and stuff. And then I came back to refresh it in the last few weeks. And it's like Claude Opus 4.5 versus whatever model at the time was like just crushed all the issues that it had from building in June, July. And like, it might be a bigger model. There's a lot of things that go into this, but that's like, there's still progress coming.

• 1:08:37Lex Fridman?

  So what you're speaking to is the nuance of the Y axis of the scaling laws, that the way it's experienced versus on a benchmark, the actual intelligence is might be different. But still, your intuition about pre-training, if you scale the size of compute. Will the models get better? Not whether it's financially viable, but just from the law aspect of it. Do you think the models will get smarter?

• 1:09:01Speaker 02

  Yeah, and I think that there's. And this sometimes comes off as like almost like disillusioned from people, leadership, AI companies saying this, but they're like, it's held for thirteen orders of magnitude of compute or something like, why would it ever end? So I think fundamentally it is pretty unlikely to stop. It's just like, eventually we're not even going to be able to test the bigger scales because of all the problems that come with more compute. I think that there's a lot of talk on how twenty twenty six is a year when very large blackwell compute clusters, it's like gigawatt scale facilities, hyperscalers are coming online. And these were all contracts for power and data centers that were signed and sought out in like twenty two and twenty twenty three. So before or right after chat GPT.

• 1:09:46Speaker 02

  So it took this two to three year lead time to build these bigger clusters to train the models. While there's obviously immense interest in building even more data centers than that. So that is like, kind of the crux that people are saying is like, these new clusters are coming. The labs are going to have more compute for training. They're going to utilize this. But it's not a given. And it's like, I, I've seen so much progress that I expected and I expect a little bit bigger models and I expect. I would say it's more like we will see a two thousand dollars subscription this year. We've seen two hundred dollars subscriptions. It's like that could ten X again. And these are the kind of things that could come and they're all downstream of this like bit big, bit bigger model that offers just a little bit more cutting edge.

• 1:10:26Lex Fridman?

  So, you know, it's reported that XAI is going to hit that one gigawatt scale early twenty six and full two gigawatt by year end. How do you think they'll utilize that? In the context of scaling laws? Is, is a lot of that inference? Is a lot of that training?

• 1:10:45Speaker 02

  It ends up being all of the above. So I think that. All of your decisions when you're training a model come back to pre-training. So if you're going to scale RL on a model, you still need to decide on your architecture that enables this. We were talking about like other architectures and using different types of attention. We're also talking about mixture of experts' models. This sparse nature of MOE models makes it much more efficient to do generation, which becomes a big part of post-training. And it's like, you need to have your architecture ready so that you can actually scale up this compute. I still think most of the compute is going in at pre-training. Because you can still make a model better, you still want to go and revisit this. You still want the best base model that you can.

• 1:11:28Speaker 02

  And in a few years, that'll saturate and the RL compute will just go longer.

• 1:11:32Lex Fridman?

  Is there people who disagree with you that say basically pre-training is dead? It's all about scaling inference, scaling post-training, scaling context, continual learning, scaling data, synthetic data.

• 1:11:47Speaker 02

  People vibe that way and describe it in that way, but I think it's not the practice that is happening.

• 1:11:52Lex Fridman?

  It's just the general vibe of people saying this thing's dead.

• 1:11:55Speaker 02

  The excitement is elsewhere. So the low-hanging fruit in RL is elsewhere. Like, for example, we released our model in November. For every company has deadlines, our deadline was like November 20th. And for that, our RL run was five days, which compared to 2024 is a very long time to just be doing post-training at a model of like 30 billion parameters. It's not a big model. And then in December, we had another release, which was just, we let the RL run for, go for another three and a half weeks and the model got notably better, so we release it. And like, that's a big amount of time to just allocate to something that is going to be your peak for the year. So it's like, there's these types of decisions that happen when they're training a model where they just like can't, they can't leave it forever. You have to keep, you have to keep pulling in the improvements you have from your researchers.

• 1:12:40Speaker 02

  So that's like you redo pre-training, you'll do this post-training for a month, but then you need to give it to your users, you need to do safety testing. So it was kind of just like, I think there's a lot in place that reinforces this cycle of just keep updating the models. There's things to improve. You get a new compute cluster that lets you do something maybe more stable or faster. It's like you hear a lot about Blackwell having rollout issues where at AI2, most of the models we're pre-training are on like 1,000 to 2,000 GPUs. But when you're pre-training on 10,000 or 100,000 GPUs, you hit very different failures. So GPUs are known to break in weird ways and doing 100,000 GPU run is like. You're pretty much guaranteed to always have at least one GPU that is down. And you need to have your training code handle that redundancy, which is just a very different problem.

• 1:13:24Speaker 02

  Whereas like what we're doing, like I'm playing with post-training on DJX Spark or you have your book is like, or people learning ML, it's like what they're battling to train these biggest models is just like mass distributed scale. And it's a very different, but that's somewhat different than like are these, like that's a systems problem in order to enable the scaling laws, especially a pre-training. You need all of these GPUs at once. When we shift to reinforcement learning, it actually lends itself to heterogeneous compute because you have many copies of the model. And but to do a primer for a language model reinforcement learning, what you're doing is you have two sets of GPUs. One is you can call it the actor and one you call the learner.

• 1:14:08Speaker 02

  The learner is where your actual reinforcement learning updates are going to do. These are traditionally policy gradient algorithms. Proximal policy optimization, PPO, and group relative policy optimization, GRPO, are the two popular classes. And on the other side, you're going to have actors which are generating completions. And these completions are the things that you're going to grade. So reinforcement learning is all about optimizing reward. And in practice, what you can do is that you can have a lot of different actors in different parts of the world, doing different types of problems, and then you send it back to this highly network compute cluster to do this actual learning where you take the gradients and you need to have a tightly mesh network where you can do different types of parallelism and spread out your model for efficient training.

• 1:14:54Speaker 02

  So there's just like a lot of, every different type of training and serving has these considerations you need to scale. Like we talked about pre-training, we talked about RL, and then inference time scaling is like, how do you serve a model that's thinking for an hour to 100 million users?

• 1:15:09Speaker 01

  I don't really know about that, but I know that's a hard problem. And in order to give people this intelligence, there's all the systems problems and we need more compute and you need more stable compute to do it.

• 1:15:19Lex Fridman?

  But you're bullish on all of these kinds of scaling is what I'm hearing. On the inference, on the reasoning, even on the pre-training.

• 1:15:27Speaker 01

  Yeah, so that's a big can of worms here. But so there are basically two, the knobs are the training and the inference scaling where you can get gains. And so in a world where we had, let's say, infinite compute resources, you want to do all of them. But like, so you have training, you have inference scaling and training is like a hierarchy. It's pre-training, mid-training, post-training, changing the model size, more training data, making training a bigger model. Gives you more knowledge in the model than the model, um, let's say has a better, it's like a better base model back in the day. Or we still, we call it foundation model and it unlocks, so you, but you don't, let's say, have the model be able to solve your most complex tasks, tasks during pre-training or after pre-training.

• 1:16:09Speaker 01

  You still have these other unlock phases where you have mid-training or non-context, for example, post-training with a L R V R that unlocks capabilities that the model has in terms of just knowledge in the pre-training. And I think, sure, if you, uh, so do more pre-training, you get a better base model that you can unlock later. But like Nathan said, it just becomes too expensive. So we don't have infinite compute. So you have to decide, do I want to spend that compute more on making the model larger? But you know, it's like a trade-off. It's, it's like in an ideal world, you want to do all of them. And I think in that sense, scaling is still pretty much alive. You would still get a better model, but like we saw with G P T four point five, it's just not worth it. I mean, it's like, cause you can, let's say you can unlock more performance with other techniques at that current moment.

• 1:16:52Speaker 01

  Especially, um, if you look at inference scaling, that's one of the biggest gains this year with O one, um, where it took a smaller model further than pre-training a larger model like G P T four point five. So it's like, I wouldn't say pre-training scaling is that. It's just like there are other more attractive ways to scale right now at the moment. But at some point, you know, you will still want to make some progress on the pre-training. The thing is also to consider. Um, where you, but do you want to spend your money? If you spend it more on the pre-training, it's like a fixed cost. You train the model and then it has this capability forever. You can always use it and so forth. With inference scaling, you don't spend money during training. You spend money later per query. And then it's also like the math. How long is my model going to be on the market?

• 1:17:37Speaker 01

  If I replace it in half a year, maybe it's not worth spending five million, 10 million, 100 million dollars on the training it longer. Maybe it's just, I will just do more inference scaling and get the performance from there. It may cost me 2 million in terms of user queries. It becomes a question of how many users you have and then doing the math. Um, and I think that's also where it's interesting. Where GPT is in a position, I think they have a lot of users where they need to go a bit cheaper. Where they have that GPT five model that is a bit smaller. Other companies that have, as if your customers have other, other trade-offs. For example, there was also the math Olympiad or some of these math problems where GPT or OpenAI, they had a proprietary model.

• 1:18:20Speaker 01

  And I'm pretty sure it's just like a model that has been maybe fine tuned a little bit more. But most of it was during inference scaling to achieve this peak performance in certain tasks where you don't need that all the time. And but yeah, long story short, I do think all of these pre-training, mid-training, post-training, inference scaling, they are all still things you want to do. It's just finding at the moment in this year, it's finding the right ratio that gives you the best bank for the buck, basically.

• 1:18:45Lex Fridman?

  I think this might be a good place to define pre-training, mid-training and post-training.

• 1:18:50Speaker 01

  So pre-training is the classic training, one next token prediction at a time. You have a big corpus of data. And Nathan also has very interesting insights there because of almost three. It's a big portion of the paper focuses on the right data mix. So pre-training is essentially just, you know, train across entropy loss, training on next token prediction on a vast corpus of internet data, books, papers and so forth. It has changed a little bit over the years in a sense. People used to throw in everything they can. Now, it's not just raw data, it's also synthetic data where people re, um, let's say rephrase certain things. So synthetic data doesn't necessarily mean purely AI made up data. It's also taking something from an article, Wikipedia article, and then

• 1:19:36Speaker 01

  rephrasing it as a Q and A question or, um, Summarizing it, rewording it and making better data that way. Because I think I would also like with humans, if someone, let's say, reads a book compared to a messy, I don't know, no offense, but like Reddit post or something like that. I do think you learn, no offense, but I think.

• 1:19:58Lex Fridman?

  There's gonna be a post about this, Sebastian.

• 1:20:00Speaker 02

  Some Reddit data is very coveted and excellent for training. You just have to filter it.

• 1:20:05Speaker 01

  I think that's the idea. I think it's like, if someone took that and rephrases that in a, let's say, more concise and structured way, I think it's higher quality data that gets the LLM maybe the same, you get the same LLM out of it at the end, but it gets there faster, it trains faster because the, let's say, if the grammar and the punctuation is correct, it already learns the correct way versus getting information from a messy way and then learning later how to correct that and stuff like that. So I think that is how pre-training evolved and how, how still, why scaling still works is that it's not about just the amount of data, it's also the tricks to make that data better for you, in a sense.

• 1:20:50Speaker 01

  And mid-training is, I mean, it used to be called a pre-training. It's, I think it's called mid-training because it was awkward to have pre-training and post-training, but nothing in the middle, right? It sounds a bit weird to have pre-training and post-training, but what's the actual training? So the mid-training is usually similar to pre-training, but you know, it's a bit more, I would say, specialized than pre-training. It's the same algorithm, but what you do is you focus, for example, on long context. Like, is one example, you have long context documents. The reason you don't do that during just pure pre-training is because you don't have that many long context documents. You have a specific phase. And one problem of LLMs is also still, it's a neural network. It has the problem of catastrophic forgetting. So you teach it something, it forgets other things.

• 1:21:31Speaker 01

  And you wanna, it's not 100% forgetting, but you know, it's like no free lunch. You can't, it's also the same with humans. If you ask me some math I learned 10 years ago, I don't know, I would have to look at it again.

• 1:21:42Lex Fridman?

  Nathan was actually saying that he's consuming so much content that there's a catastrophic forgetting issue.

• 1:21:46Speaker 02

  Yeah, I'm like trying to learn so much about AI. It's like I was learning about pre-training parallelism. I'm like, I lost something and I don't know what it was.

• 1:21:54Speaker 01

  I don't want to anthropomorphize LLMs, but it's, I think, the same kind of in that sense, how humans learn. I mean. The quantity is not always better because yeah, it's like being selective and the mid training is being selective in terms of quality content at the end. So the last thing the LM has seen is the quality stuff. And then post-training is all the fine tuning, supervised fine tuning, DPO, reinforcement learning with verifiable rewards, with human feedback and so forth. So the refinement stages. And it's also interesting, it's like the cost thing, right? I mean, it's like pre-training, you spend a lot of money on that right now. RL a bit less. RL, you don't really, I would say, teach it knowledge. It's more like unlocking the knowledge. It's more like a skill learning, like how to solve problems with the knowledge

• 1:22:39Speaker 01

  that it has from pre-training. There are actually three papers this year, or last year, 2025, on RL for pre-training, but I mean, I don't think anyone does that in production.

• 1:22:50Speaker 02

  Toy examples for now.

• 1:22:51Speaker 01

  Toy examples, right. But to generalize, RL, post-training is more like the skill unlock where pre-training is like soaking up the knowledge essentially.

• 1:22:59Speaker 02

  A few things that could be helpful for people. A lot of people get like, think of synthetic data as being bad for training the models. You mentioned like the DeepSeek got an OCR, which is Optical Character Recognition paper. A lot of labs did. AI2 had one, had multiple. And the reason that each of these labs has these is because there's vast amounts of PDFs and other digital documents on the web that are in formats that aren't encoded with text easily. So you use these almost CR, these, or DeepSeek OCR, that we called our almost CR, to extract what can be trillions of tokens of candidate data for pre-training. And pre-training dataset size is on the order of trillions, is measured

• 1:23:42Speaker 02

  in trillions of tokens. Smaller models from researchers can be something like five to 10 trillion. Qwen is documented going up to like 50 trillion, and there's rumors that these closed labs can go to like 100 trillion tokens. And just getting this potential data to put in, I think they, they have a very big funnel. And then the data you actually train the model on is a small percentage of this. Like the, this character recognition data would be described as synthetic data for pre-training in a lab. And then there's also the things like. ChatGPT now gets wonderful answers and you can train on those best answers and that's synthetic data. It's very different than like early ChatGPT, lots of hallucinations data when people became grounded in synthetic data.

• 1:24:21Speaker 01

  One interesting question is, if I recall correctly, almost three was trained with less data than specifically some other open weight models, maybe even almost two, but you still got better performance and that might be one of the examples how the data helped.

• 1:24:33Speaker 02

  It's mostly down to data quality. I think if we had more compute, we would train for longer. I think we ultimately see that as a like, just like something we would want to do. And especially with big models, you need to have more compute because we talked about having more parameters and we talked about knowledge. And essentially there's a ratio where big models can absorb more from data and then you're gonna, you get more benefit out of this. It's, it's, it's like one of these, any logarithmic graph in your mind is like a small model will level off sooner. If you're measuring trunks of tokens and bigger, bigger models need more. But mostly is we aren't training that big of models right now, AI two, and getting the highest quality data we can is the natural starting point.

• 1:25:11Lex Fridman?

  Is there something to be said about the topic of data quality? Is there some long hanging fruit there still where the quality can be improved?

• 1:25:19Speaker 02

  It's like turning the crank. So I think historically in the open, there's been like a canonical best pre-training data set that has moved around between who has the most recent one or the best recent effort. Like AI two's DOLMO was very early with the first ulmo and hugging face had fine web. And there's a DCLM project, which has been kind of like a, which is, it stands for data comp language model. There's been data comp for other machine learning projects and they have had a very strong data set. And a lot of it is, the internet is becoming fairly closed off. So we have common crawl, which I think is hundreds of trillions of tokens and you filter it. And it looks like being a lot of scientific work where you're training classifiers and making decisions based on how do you prune down this, this data set

• 1:26:02Speaker 02

  into the highest quality stuff and the stuff that suits your tasks. So previously, language models were tested a lot more on like knowledge and just kind of conversational things. But now they're expected to do math and code. So to train a reasoning model, you need to remix your whole data set. And there's a lot of actually wonderful scientific methods here where you can, you can like take your gigantic data set. You sample a lot of really tiny things from different sources. So you say you have GitHub, Stack Exchange, Reddit, Wikipedia. You can sample small things from them and you train small models on each of these mixes and measure their performance on your evaluations. And you can just do like basic linear regression and it's like, here's your optimal data set. But if your evaluations change, your dataset changes a lot. So a lot of old mode three was new sources for reasoning to be better at math and code.

• 1:26:46Speaker 02

  And then you do this mixing procedure and it gives you the answer. And I think that's a lot of that's happened at labs this year. There's new hot things, whether it's like coding environments or web navigation, and you just need to bring in new data. You need to change your whole pre-training so that your post-training can work better. And stuff like this. And that's like the constant re-evolution and the redetermining of what they care about as for their models.

• 1:27:08Lex Fridman?

  Are there fun anecdotes of what sources of data, particularly high quality, that we wouldn't expect? You mentioned Reddit sometimes can be a source.

• 1:27:17Speaker 02

  Reddit was very useful. I think that like PDFs is definitely one.

• 1:27:24Speaker 01

  Especially archive.

• 1:27:25Speaker 02

  Yeah. So like AI two has run semantic scholar for a long time, which is a like what you can say is a competitor to Google Scholar with a lot more features. And to do this, AI two has found and scraped a lot of PDFs for openly accessible papers that might not be like behind the closed paid garden of a certain publisher. So like truly open scientific PDFs. And if you like, you sit on all of these and you process it and you can get value out of it. And I think that like

• 1:27:53Speaker 02

  a lot of that style of work has been done by the frontier labs much earlier. And it's just like, you need to have a pretty skilled researcher that understands how things change models and they bring it in and they clean it. And it's, it's a lot of labor that like I think of a lot of frontier labs when they scale researchers, a lot more goes into data. You have people like, if you want to, if you join a frontier lab, you want to have impact, the best way to do it is just make, find new data that's better. And then like the fancy glamorous algorithmic things, like figuring out how to make O one is like the sexiest thought of a scientist. It's like, oh, I figured out to scale RL. And there's a group that did that. But I think of most of the contributions is like,

• 1:28:30Lex Fridman?

  On the data.

• 1:28:31Speaker 02

  I'm going to make the data better, or I'm going to make the infrastructure better so that everybody in my team can run experiments 5% faster.

• 1:28:37Speaker 01

  At the same time, I think it's also one of the closest guarded secrets, what your training data is for legal reasons. And so there's also, I think, a lot of work that goes into hiding what your trading data was essentially, like trying to model. To not give away the sources because of legal reasons.

• 1:28:52Speaker 02

  The other thing to be complete is that some people are trying to train on only licensed data, where Common Crawl is a scrape of like the whole internet. So if I I host multiple websites, I'm happy to have them train language models, but I'm not explicitly licensing what governs it. And therefore, this like the Common Crawl is largely unlicensed, which means that your consent really hasn't been provided for how to use the data. There's another idea where you can train language models only on data that has been licensed explicitly. So that kind of governing contract is provided. And I'm not sure if Apertus is the copyright thing or the license thing. I know that the reason that they did it was for an EU compliance thing, where they wanted to make sure that their model fit one of those checks.

• 1:29:34Speaker 01

  And also on that note, also, for example, there's also the distinction between the licensing. So some people, like you said, they just purchase the license. Let's say they buy a book online, let's say an Amazon Kindle book, or let's say a Manning book or something, and then use that in the training data. And that is like the gray zone because you paid for the content and you might want to train it. But then there are also restrictions where even that shouldn't be allowed. And so that is like where where it gets a bit fuzzy. And yeah, I think that is right now is still a hot topic. And also big companies like OpenAI, they approached private companies for their proprietary data. And private companies, they become more and more, let's say,

• 1:30:16Speaker 01

  protective of their data because they know, okay, this is going to be my mode in in a few years. And I do think that's like the interesting question where if L L M's become more commoditized, and I think a lot of people learn about L L M's, there will be a lot more people able to train L L M's. Of course, there are infrastructure challenges. But if you think of big industries like pharmaceutical industries, law, finance industries, I do think they at some point will hire people from other frontier labs to build their in-house models on their proprietary data, which will be then again another unlock with pre-training that is currently not there. Because even if you wanted to, you can't get that data. You can't get access to clinical trials most of the time in these types of things. So I do think.

• 1:30:58Speaker 01

  Scaling in that sense might be still pretty much alive if you also look in domain-specific applications, because we are still right now in this year just looking at general purpose LLMs on ChatGPT, Anthropic and so forth. They are just general purpose. They're not even, I think, scratching the surface of what an LLM can do if it is really specifically trained and designed for a specific task.

• 1:31:19Speaker 02

  I think on the data thing, this is one of the things where like this happened in 2025 and we totally forget it is Anthropic lost in court and was owed $1.5 billion to authors. Anthropic, I think, bought thousands of books and scanned them and was cleared legally for that because they bought the books. And that is kind of going through the system. And then the other side, they also torrented some books. And I think this torrenting was the path where the court said that they were then culpable to pay this billions of dollars to authors, which is just like such a mind-boggling lawsuit that kind of just came and went like that is so much money from the VC ecosystem.

• 1:31:54Lex Fridman?

  These are court cases that will define the future of human civilization because it's clearly that data drives a lot of this. And there's this very complicated human tension of, I mean, you can empathize, you're both authors.

• 1:32:06Speaker 01

  Yeah.

• 1:32:07Lex Fridman?

  Yeah, there's some degree to which, I mean, you put your heart and soul and your, your sweat and tears into the, the writing that you do. It feels a little bit like theft for somebody to train your data without giving you credit.

• 1:32:21Speaker 01

  And there are, like Nathan said, also two layers to it. Someone might buy the book and then train on it, which is could be argued fair or not fair, but then there are the three straight up companies who use pirated books where it's not even compensating the author. It's that, that is, I think, where people got a bit angry about it specifically.

• 1:32:39Lex Fridman?

  Yeah, but there has to be some kind of competition scheme. This is like moving towards towards something like Spotify streaming did originally for music. You know, what does that competition look like? You have to define those kinds of models. You have to think through all of that. One other thing I think people are generally curious about, I'd love to get your thoughts. As LLMs are used more and more, if you look at even archive, but GitHub, more and more of the data is generated by LLMs. What do you do in that kind of world? How big of a problem is that?

• 1:33:11Speaker 02

  Largest problem is the infrastructure and systems, but from an AI point of view, it's kind of inevitable.

• 1:33:17Lex Fridman?

  So it's basically LLM-generated data that's curated by humans, essentially, right?

• 1:33:22Speaker 02

  Yes, and I think that a lot of open source contributors are legitimately burning out. If you have a popular open source repo, somebody's like, oh, I want to do open source AI. It's good for my career and they just vibe code something and they throw it into the. You might get more of this than I do.

• 1:33:38Speaker 01

  So I have actually a case study here. I have a repository called ML Extend that I developed as a student around 15 years, 10 years ago. And it is a reasonably popular library still for certain algorithms. I think especially like frequent data mining stuff. And there was recently, I think, two or three people who submitted a lot of PRs in a very short amount of time. I do think LLMs have been involved in submitting these PRs. Me as the maintainer, that two things first, I'm a bit overwhelmed. Like I don't have time to read through it because especially it's an older library. That is not a priority for me. At the same time, I kind of also appreciate it because I think something people forget is it's not just using the LLM. There's still a human, you have a human layer that verifies something.

• 1:34:22Speaker 01

  And that is in a sense also how data is labeled, right? So that's like, um, one of the most expensive things is getting labeled data for our L back in human feedback phases. And this is kind of like that where it goes through phases and then you get actually higher quality data out of it. You know, so I don't mind it in a sense. It can feel overwhelming, but I do think there is also value in that.

• 1:34:44Lex Fridman?

  It feels like there's a fundamental difference between raw LLM-generated data and LLM-generated data with human in the loop that does some kind of verification, even if that verification is a small percent of the lines of code.

• 1:34:59Speaker 01

  I think this goes with anything like where. People think also sometimes, oh, I can just use an LLM to learn about XYZ, which is true, you can, but there might be a person who is an expert who might have used an LLM to write specific code. There is kind of like this human work that went into it to make it nice and throwing out the not so nice part to make it to kind of like predigested for you and that saves you time. And I think that's, it's, that that's the value add where you have someone filtering things or even using the LLMs correctly. I think this is still labor that that you get for free with you. For example, read an article, let's say substack article. I could maybe ask an LLM to give me opinions on that, but I wouldn't even maybe know what to ask.

• 1:35:43Speaker 01

  I think there is still value in reading that article compared to me going to the LLM because you are the expert, you select what knowledge is actually spot on, should be included, and you give me this very, this, this executive summary. And this is kind of a huge value add because now I don't have to waste three, five hours to go through this myself, maybe get some incorrect information and so on. And so I think that's also where the future still is for writers, even though there are LLMs that expert can kind of like save your time.

• 1:36:16Lex Fridman?

  It's kind of fascinating to actually watch, and I'm sure you guys do this, but for me to look at the difference between a summary and the original content, even if it's a page long summary of a page long content, it's interesting to see how the summary, LMB summary, takes the edge off. Like what, what is the signal it removes from the thing?

• 1:36:40Speaker 01

  The voice is what I talk about a lot.

• 1:36:42Lex Fridman?

  Voice, well, voice, I would love to hear what you mean by voice. That's really powerful, but sometimes there's like literally insights. Like in removing an insight, you're actually fundamentally changing the meaning of the thing. So I'm continuously disappointed how bad LLMs are.

• 1:36:58Lex Fridman?

  At really getting to the core insights, which is what a great summary does. Even if you go and I have these extensive, extremely elaborate prompts where I'm like really trying to dig for the insights and it's still not quite there, which I mean, that's a whole deep philosophical question about what is human knowledge and wisdom and what does it mean to be insightful and so on. But when you talk about the voice, what do you mean?

• 1:37:24Speaker 02

  So when I write, I think a lot of what I'm trying to do is take what you think as a researcher, which is very raw, which a researcher is trying to encapsulate an idea at the frontier of their understanding and they're trying to put what is a feeling into words. And I think that my writing, I tried to do this as the writing, which makes it come across as raw, but also high information in a way that is like, some people will get it and some won't. And that's kind of the nature of research. And I think this is something that language models don't do well, particularly they're all trained with this reinforcement learning from human feedback, which is designed to take feedback from a lot of people and in a way average how the model behaves from this. And I think that there's, it's going to be hard for a model to be very incisive when there's

• 1:38:09Speaker 02

  that sort of filter in it. And I think this is kind of a wonderful fundamental problem for researchers in RLHF. It's like this provides so much utility in making the models better, but also the problem formulation is kind of like, there's this knot in it that you can't get past. So that's what I think of is like, these language models don't have this prior and their deep expression that they're trying to get at. I don't think it's impossible to do. I think there's stories of models that really shock people. Like I think of like, I would love to have tried Bing Sydney and does like, does that have more voice? Cause it would so often go off the rails on people and what is historically obviously a scary way, like telling a reporter to leave his wife is a crazy model to potentially put in general,

• 1:38:53Speaker 02

  general adoption. But that's kind of like a trade off, like is this RLHF process, like in some ways adding limitations?

• 1:39:01Lex Fridman?

  That's a terrifying place to be as one of these frontier labs and companies. Because millions of people are using them.

• 1:39:08Speaker 02

  There was a lot of backlash last year with the GPT four O getting removed. And I personally never used the model, but I've talked to people at OpenAI where they're to the point where they like get emails from users that might be detecting subtle differences in the deployments in the middle of the night and they email them and they're like, my friend is different. And they like find these people, employees emails and send them things because they're so attached to this set, what is a set of model weights and a configuration that is deployed to the users. We see this with TikTok. You open it, I don't use TikTok. Supposedly in like five minutes, the algorithm gets you. It's like it's locked in and I don't like, those are language models doing recommendations. Like, I think there are ways that you can do this with a language model within like

• 1:39:52Speaker 02

  five minutes of chatting with it, the model just gets you. And that is something that people aren't really ready for. Like, I think like, don't give that to kids. Like, don't give that to kids, at least until we know what's happening.

• 1:40:03Lex Fridman?

  But there's also going to be this mechanism. What's going to happen with these L L Ms is they're used more and more. Unfortunately, the nature of the human condition is such that people commit suicide. And so what journalists would do is they will report extensively on the people who commit suicide and they would very likely link it to the L L Ms because they have that data about the conversations. If you're really struggling in your life, if you're depressed, if you're thinking about suicide, you're going to probably talk to L L Ms about it. And so what journalists will do is they will say, well, the suicide was committed because of the L L M. And that's going to lead to the companies because of legal issues and so on, more and more and more taking the edge off of the L L M. So it's going to be as generic as possible.

• 1:40:48Lex Fridman?

  It's so difficult to operate in this space because of course you don't want an L L M to cause harm to humans at that level, but also. This is also the nature of the human experience is to have a rich conversation, a fulfilling conversation, one that challenges you from which you grow. You need that edge. And that that's something extremely difficult for AI researchers on the RLHF front to actually have to solve. Cause you're actually dealing with the human condition.

• 1:41:19Speaker 02

  Like a lot of researchers at these companies are so well motivated and there's definitely the likes of Anthropic and OpenAI are culturally so want to do good through this for the world. And there is, it's such a, I'm like, oh, I don't want to work on this because on the one hand, a lot of people see AI as a health ally, as somebody they can talk to about their health confidentially. But then it bleeds all the way into this, talking about mental health and things where that's it's heartbreaking that this will push like be the thing where somebody goes over the edge. But other people might be saved. And I'm like, I don't like, there's things that as a researcher training models, it's like, I don't want to train image generation models and release them openly.

• 1:42:01Speaker 02

  Cause I don't want to enable somebody to have a tool on their laptop that can harm other people. Like I don't have the infrastructure at my company to do that safely. But it's like, like, there's a lot of areas like this where it's just, it needs people that will approach it with the complexity and just kind of conviction of like, it's just such a hard problem.

• 1:42:20Lex Fridman?

  But also we as a society, as users of these technologies, need to make sure that we're having the complicated conversation about it versus just fear mongering. Big tech is, is causing harm to humans or stealing your data, all that kind of stuff. There is, is more complicated than that. And you're right, there's a very large number of people inside these companies, many of which you know, many of which I know, that deeply care about helping people. They are considering the full human experience of people from across the world, not just Silicon Valley. People across the United States, people across the world, what that means, what their needs are. It's really difficult to design this one system. That is able to help all these different kinds of people across the different age groups, cultures, mental states, mental conditions, all that kind of stuff.

• 1:43:04Speaker 02

  I wish that the timing of AI was different with the relationship of big tech to the average person. So like big tech's reputation was so low. And with how AI is so expensive, it's like inevitably going to be a big tech thing where it takes so many resources. And people say that US is quote unquote betting the economy on AI with this build out. And it's like to have these be intertwined at the same time is just makes for such a hard communication environment. It would be good for me to go talk to more people in the world that hate big tech and see AI as a continuation of this.

• 1:43:35Lex Fridman?

  And one of the things you actually recommend, one of the antidotes that you talk about

• 1:43:42Lex Fridman?

  is to find agency in this whole system as opposed to sort of sitting back in a powerless way and consuming the AI slop as it quickly rapidly takes over the internet. More find agency by using it to build stuff, build apps, build. So you, one, that actually helps you build intuition, but two, it's empowering because you you can understand how it works, what the weaknesses are, and it allows, it gives your voice power to say like, this is fucked up. This is bad. This is bad use of the technology. This is good use of the technology. And you're more plugged into the system than, so you can understand it better and you can steer it better as a consumer.

• 1:44:21Speaker 01

  I think it's a good point you brought up agency. Instead of ignoring it and saying, okay, I'm not going to use it. I think it's probably long term healthier to say, okay, it's out there. I can't put it back, you know, like internet computers back then when they came out. How do I make best use of it? And how does it help me to up level myself? The one thing I worry here, though, is like, if you just fully use it for something you love to do, the thing you love to do is not no longer there. And that could potentially, I feel, like lead to burnout. For example, if I use an LM to do all my coding for me, now there's no coding. I'm just managing something that is coding for me. Two years, let's say later, if I just do that eight hours a day. Have something code for me. Do I feel fulfilled still like is this like?

• 1:45:06Speaker 01

  Yeah, I mean, is this like hurting me in terms of being excited about my job, excited about what I'm doing? Am I still proud to build something?

• 1:45:15Lex Fridman?

  So there's a on that topic of enjoyment is quite interesting. We should just throw this in there that there is this recent survey of about seven hundred ninety one professional developers. Professional, meaning ten plus years of experience.

• 1:45:28Speaker 02

  That's a long time.

• 1:45:29Lex Fridman?

  Yeah.

• 1:45:31Speaker 02

  That's a junior developer.

• 1:45:34Lex Fridman?

  Yeah, in this day and age. So the the results here on many fronts are surprising. So they break it down by junior and senior developers. But I mean, it just shows that both junior senior developers use AI generated code in code they ship. So this is not just for fun, sort of intermediate kind of learning things. This is code they ship. And so it's twenty five percent, like most of them use around fifty percent or more. And what's interesting is for the category of over fifty percent of your code that you ship is AI generated. Senior developers are much more likely to do so. But you don't want AI to take away the thing you love.

• 1:46:18Speaker 01

  Yeah.

• 1:46:19Lex Fridman?

  I think this speaks to my experience, these particular results I'm about to say. So together, about eighty percent of people find it either somewhat more enjoyable or significantly more enjoyable to use AI as part of the work.

• 1:46:32Speaker 01

  I think it depends on the task from my personal usage. For example, I have a website where I sometimes tweak things on the website. I personally don't enjoy this. So in that sense, if the AI can help me to implement something on my website, I'm all here for it. It's it's great. But then, at the same time, when I solve a complex problem. Well, if there's a bug and I hunt this bug and I find the bug, it's the best feeling in the world. It's like you get so much joy like, or it's like you feel like great. But now, if you don't even think about thinking about the bug, you just go directly to the L M. Well, you never have this kind of feeling, right? But then. There could be the middle ground where while you try yourself, you can't find it.

• 1:47:15Speaker 01

  You use the L L M and then you don't get frustrated because it helps you and you move on to something that you enjoy. And so I think looking at these statistics, I think also the difference is what is not factored in. It's averaging over all the different scenarios where we don't. So we don't know if it's for the core task or if it's for something mundane that people would not have enjoyed otherwise. So in a sense, AI is really great for doing mundane things that take a lot of work. So for example, my wife, the other day, she has like a podcast for like book, like book discussions, a book club. And she was like transferring the show notes from Spotify to YouTube. And then the links somehow broke and she had in some episodes, because it is custom many books, like hundred links or something.

• 1:47:59Speaker 01

  And it would have been really painful to go in there and fix each link manually. And so I suggested, hey, let's try chat G P T. We copied the text into chat G P T and it fixed them. And instead of two hours going from link to link fixing that, you know, it made that type of work much more seamless. There was no frustration fixed. I think everyone has a use case where AI is useful for something like that, that would be really boring, really mundane.

• 1:48:23Lex Fridman?

  I, for me personally, since we're talking about coding and you mentioned debugging. Would a lot of the sources of enjoyment for me on more on the cursor side than the Claude code side is the, I have a friend. I have a co, what's that called, a pair programmer. Like it's less lonely. You, you made debugging sound like this great joy. No, I would say, I would say debugging is like a drink of water after you've been going through a desert for, for, for, for days. So like you, you skip the whole desert part where you're suffering. So like there, sometimes it's nice to have a friend who's, who's can't really find the bug, but can give you some intuition about the code.

• 1:49:07Lex Fridman?

  And you're together with that friend going through the desert and then together find that drink of water. So at least for me, maybe it speaks to the loneliness of the programming experience. It's, that is a source of joy.

• 1:49:21Speaker 01

  It's maybe also related to delayed gratification. I'm a person who, you know, even as a kid, I like the idea of Christmas presents, having them, getting them better than actually getting the presents. I would look forward to the day I get the presents, but then it's over and I'm disappointed. And maybe it's something like also with, let's say, food. I think food tastes better when you're really hungry. And with, yeah, you're right with debugging. It is not always, you know, great. It's like often frustrating, but then if you can solve it, then it's great. But there's also like a sweet Goldilocks zone. If it's too hard, then it's, you know, wasting your time. But I think that is another challenge, though. How will people learn?

• 1:50:06Speaker 01

  I mean, the chart we looked at, we saw that more senior developers are shipping more AI-generated code than the junior ones. And I think it's very interesting because intuitively you would think it's the junior developers because they don't know, let's say, how to do the thing yet because they are more junior. And so they use AI to do that thing. It could either mean the AI is not good enough yet to solve that task, but it could also mean experts are more effective at using it. They know where and better how to use it and review the code and they trust the code than more. And so I think one issue in the society in the future will be, though, how do you become an expert if you never try to do the thing yourself?

• 1:50:45Speaker 01

  And I think one way it's always like for me, how I learn is by trying things myself, like math textbooks, if you look at the solutions. Yeah, you learn something, but I think you learn actually better if you try first and then you appreciate the solution differently because you know how to put it into your mental framework. And if L M S are here all the time, would you actually go through the length at struggling? Would you? Be willing to struggle because struggle is not nice, right? I mean, it's struggling. And if you use the LLM to do everything at some point, you will never really take the next step. And then you will maybe not get that unlock that you would get as an expert using an LLM. So it's like, you know, it's like, I think there's like a Goldilocks sweet spot

• 1:51:29Speaker 01

  where maybe, maybe the trick here is you make dedicated offline time where you study two hours a day and the rest of the day use LLMs. But I, I think it's important also for people to still invest in themselves, in my opinion, to not just, you know, LLM everything.

• 1:51:43Lex Fridman?

  Yeah, there is. And we together as a civilization that we each individually have to find that Goldilocks zone.

• 1:51:48Speaker 01

  Yeah.

• 1:51:49Lex Fridman?

  Uh, and in the programming context as developers. Now, we've had this fascinating conversation that started with pre-training and mid-training. Let's get to post-training. A lot of fun stuff in post-training. So what are some of the interesting ideas in post-training?

• 1:52:04Speaker 02

  The biggest one from 2025 is learning this reinforcement with verifiable rewards. You can scale up the training there, which means doing a lot of this kind of iterative generate grade loop. And that lets the models learn both interesting behaviors on the tool use and software side. This could be searching, running commands on their own and seeing the outputs. And then also that training enables this inference time scaling very nicely. And it just turned out that this paradigm was very nicely linked in this, where it's this kind of RL training enables inference time scaling. But inference time scaling could have been found in different ways. So it was kind of this perfect storm of the models change a lot. And the way that they're trained is a major factor in doing so.

• 1:52:48Speaker 02

  And this has changed how people approach post-training dramatically.

• 1:52:53Lex Fridman?

  Can you describe RLVR popularized by DeepSeek R1? Can you describe how it works?

• 1:52:58Speaker 02

  Yeah, fun fact, um, I was on the team that came up with the term RLVR, which is from our two to three work before DeepSeek, which is, we don't take a lot of credit for the, being the people to popularize the scaling RL, but as fun as. What academics get as an aside is the ability to name and influence the discourse because the closed labs can only say so much. That one of the things you can do as an academic is like, you might not have the compute to train the model, but you can frame things in a way that ends up being, I describe it as like a community can come together around this RLVR term, which is very fun. And then DeepSeek is the people that did the training breakthrough, which is they scaled the reinforcement learning, which was you have the model generate answers and then

• 1:53:42Speaker 02

  grade the completion if it was right. And then that accuracy is your reward for reinforcement learning. So reinforcement learning is classically an agent that acts in an environment and the environment gives it a state and a reward back and you try to maximize this reward. In the case of language models, the reward is normally accuracy on a set of verifiable tasks, whether it's math problems, coding tasks. And it starts to get blurry with things like factual domains, like that is also in some ways verifiable or constraints on your instruction, like respond only with words that start with A. Like all of these things are verifiable in some way.

• 1:54:23Speaker 02

  And the core idea of this is you find a lot more of these problems that are verifiable and you let the model try it many times while taking these RL steps, these RL gradient updates. The infrastructure evolved from this reinforcement learning from human feedback, where in that era, the score they were trying to optimize was a learned reward model of aggregate human preferences. So you kind of change the problem domains and that let the optimization go on to much bigger scales, which kind of kickstarted a major change in what the models can do and how people use them. What kind of domains is RLVR amenable to? Math and code are the famous ones. And then there's a lot of. Work kind of on what is called a rubrics,

• 1:55:07Speaker 02

  which is related to a word people might have heard as LLM as a judge, which is like for each problem, I'll have a set of problems in my training data set. I will then have another language model and ask it, what would a good answer to this problem look like? And then you could try the problem a bunch of times over and over again and assign a score based on this rubric. So that's not necessarily verifiable, like a math and code domain. But this rubrics idea and other scientific problems that it might be a little bit more vague is where a lot of the attention is, where they're trying to push this set of methods into these kind of more open-ended domains where the models can learn a lot more.

• 1:55:43Speaker 01

  I think that's called reinforcement learning with AI feedback, right?

• 1:55:46Speaker 02

  That's the older term from it that was coined in Anthropic's Constitutional AI paper. So it's like a lot of these things come in cycles.

• 1:55:54Speaker 01

  Also, just one step back for the RLVR. So I think the interesting, beautiful thing here is that you you ask the LLM, let's say a math question, and then you know the correct answer. And you let the LLM, like you said, figure it out. But how it does it, I mean, you don't really constrain it much. There are some constraints you can add, like use the same language. You don't switch between Spanish and English. But let's say you're pretty much hands off. You only give the question and the answer. And then the LLM has to, you know, just the task to arrive at the right answer. But the beautiful thing here is what happens in practice is that the LLM will do a step-by-step description, like, you know, like as a student or like as a mathematician, how you would derive the solution. It will give you or it will use those steps.

• 1:56:38Speaker 01

  And that helps actually the model to improve its own accuracy. And then, like you said, the inference scaling. So inference scaling loosely means basically spending more compute during using the LLM during inference. And here the inference scaling is that the model would use more tokens. And also, I think in the R1 paper, they showed the longer they train the model, the longer the responses are, they they grow over time. They use more tokens, so it becomes more expensive, becomes more expensive for simple tasks. But these explanations, they help the model with the accuracy. There are also interesting, a lot of papers showing what the model explains does not necessarily have. To be correct, or maybe it's even unrelated to the answer, but for some reason, it still helps the model like this is the fact that it is.

• 1:57:21Speaker 01

  Um, explaining, and I think it's also again, I don't want to anthropomorphize these L L M's, but it's kind of like how we humans operate, right? If there's a complex math problem, let's say in a math class, you usually have a notepaper and you do it step by step, you cross out things. And the model also self corrects and that that was, I think, the aha moment in the R one paper, they called it aha moment because the model. Itself recognized it made a mistake and then said, ah, I did something wrong and so let me try and I think that's just so cool that. This falls out of just giving it the correct answer and having it figure out how to do it, that it kind of. Does in a sense what a human would do, although L L M's don't think like humans, it's kind of like an interesting coincidence and it.

• 1:58:03Speaker 01

  And the other side, nice side effect is it's great for us humans often to see these steps, it builds trust, but also we learn, we can double check things.

• 1:58:12Speaker 02

  There's a lot in here. I think some of the debate, there's been a lot of debate this year on if the language models like these aha, I think the aha moments are kind of fake because in pre-training. You essentially have seen the whole internet, so you have definitely seen people explaining their work, even even verbally, like a transcript of a math lecture, you try this, oh, I messed this up. And what reinforcement learning is, this R L V R is very good at doing is amplifying these behaviors because they're very useful and enabling the model to think longer and to check its work. And I agree that it is very beautiful that this training kind of the model learns to amplify this in a way that is just so useful at the final answers. Being better.

• 1:58:49Speaker 01

  I can give you also a hands on example, I was training the grand three base model with R L V R on math five hundred. The base model had an accuracy of about fifteen percent, just fifty steps, like in a few minutes with R L V R. The model went from fifteen percent to fifty percent accuracy and the model, you can't tell me it's learning anything about fundamentally about math and.

• 1:59:11Speaker 02

  The Quinn example is weird because there's been two papers this year, one of which I was on that talks about data contamination in Quinn. And specifically that they trained on a lot of this special mid-training phase, so we had like a minute on because it's weird. So they train on problems that are almost identical to math.

• 1:59:26Speaker 01

  Exactly. And so you can see that basically the R L, it's not teaching the model any new knowledge about math, you can't do that in fifty steps. So the knowledge is already there in the pre-training, you're just unlocking it.

• 1:59:36Speaker 02

  I still disagree with the kind of premise because there's a lot of weird complexities that you can't prove because. One of the things that points to weirdness is that if you take the Qwen-3 so-called base model, and you can, you could Google on the screen, you could Google like math dataset hugging face, and you could take a problem, and what you do if you put it into Qwen-3 base, the, all these math problems have words. So it'd be like Alice has five apples and takes one, and gives three to whoever, and there are these word problems. But these Qwen-based models, why people are suspicious of them, is if you change the numbers but keep the words, Qwen will produce like a very high, without tools, will produce a very high accuracy, like decimal representation of the answer, which means there's some like,

• 2:00:18Speaker 02

  at some time, it was shown problems that were almost identical to the test set, and it was using tools to get a very high-precision answer. But a language model without tools will never actually have this. So it's kind of been this big debate in the research community is like, how much of these reinforcement learning papers that are training on Qwen, and measuring specifically on this like math benchmark, where there's been multiple papers talking about contamination, is like, how much can you believe them? And I think this is what caused the reputation of RLVR being about formatting, because you can get these gains so quickly, and therefore it must already be in the model. But there's a lot of complexity here that we, it's not really like controlled experimentation, so you don't really know.

• 2:00:59Speaker 01

  But if it weren't true, I would say distillation wouldn't work, right? I mean, distillation can work to some extent, but the thing is, that is, I think, the biggest problem in LLM research, this contamination, because we don't know what's in the data. It's, unless you have a new dataset, it's really impossible. And the same, you mentioned math, the math dataset, which is give a question and then answer, and an explanation is given, but then also even something simpler like MMLU, which is a multiple choice benchmark. If you just change the format slightly, like, I don't know, you use a dot instead of a parenthesis or something like that, the model accuracy will vastly differ.

• 2:01:37Speaker 02

  I think that that could be like a model issue rather than a general issue.

• 2:01:42Speaker 01

  It's not even malicious by the developers of the LLM, like, hey, we want to cheat at that benchmark. It's just, it has seen something at some point, and I think the only fair way to evaluate an LLM is to have a new benchmark that is after the cutoff date when the LLM was deployed.

• 2:01:55Lex Fridman?

  Can we lay out what would be the sort of the recipe of all the things that will be going to post-training? And you mentioned RLVR was a really exciting, effective thing. Maybe we should elaborate. RLHF still has a really important component to play. What kind of other ideas are there on post-training?

• 2:02:13Speaker 02

  I think you can kind of take this in order. I think you could view it as what made O1, which was this first reasoning model possible, or what will the latest model be? And they actually have, you're going to have similar interventions at these where you start with mid-training. And the thing that is rumored to enable O1 and similar models is really careful data curation, where you're providing a broad set of like what is called reasoning traces, which is just the model generating words in a forward process that is reflecting, like breaking down a problem into intermediate steps and trying to solve them. So at mid-training, you need to have data that is similar to this to make it so that when you move into

• 2:02:55Speaker 02

  post-training, primarily with this verifiable rewards, it can learn. And then what is happening today is you're figuring out which problems to give the model and how long you can train it for and like how much inference you can enable the model to use when solving these verifiable problems. So as models get better, certain problems are no longer, like the model will solve them 100% of the time and therefore there's very little signal in this. If we pull, if we look at the GRPO equation, this one is famous for this because essentially the reward given to the agent is based on how good a given action, action is a completion, is relative to the other answers to the same problem. So if all the problems get the same answer,

• 2:03:39Speaker 02

  there's no signal in these types of algorithms. So what they're doing is they're finding harder problems, which is why you hear about things like scientific domains, which is like, that's so hard, like getting anything right there. If you have a lab or something, it just generates so many tokens or much harder software problems. So the frontier models are all pushing into these harder domains and they can train on more problems and the model will learn more skills at once. The RLHF link to this is kind of like RLHF has been and still is kind of like the finishing touch on the models where it makes the models more useful. By improving the organization or style or tone, there's different things that resonates to different audiences. Like, some people like a really quirky model, and RLHF could be good

• 2:04:19Speaker 02

  at enabling that personality. And some people hate this, like, markdown, bulleted list thing that the models do, but it's actually really good for quickly parsing information. And RLHF, this human feedback stage is really great for just putting this into the model at the end of the day. So it's what made ChatGPT so magical for people, and that use has actually remained fairly stable. This formatting can also help the models get better at math problems, for example. So it's like, the border between style and formatting and like, the method that you use to answer a problem is actually, they're all very closely linked in terms of when you're

• 2:05:01Speaker 02

  training these models, which is why RLHF can still say make a model better at math, but these verifiable domains are a much more direct process to doing this, because this kind of makes more sense with the problem formulation, which is why it kind of ends up all forming together. But to summarize, it's like mid-training is give the model the skills it needs to then learn. RL and verifiable rewards is let the model try a lot of times. So put a lot of compute into trial and error learning across hard problems. And then RLHF would be like, finish the model, make it easy to use, and kind of just round the model out.

• 2:05:34Lex Fridman?

  Can you comment on the amount of compute required for RLVR?

• 2:05:38Speaker 02

  It's only gone up and up. So I think Grok IV was famous for saying they use a similar amount of compute for pre-training and post-training. Back to the scaling discussion, they involve very different hardware for scaling. Pre-training is very compute-bound, which is like this flops discussion, which is just how many matrix multiplications can you get through in one time. And because RL, you're generating these answers, you're trying the model in the real-world environments, it ends up being much more memory-bound, because you're generating long sequences. And the attention mechanisms have this behavior where you get a quadratic increase in memory as you're getting to longer sequences. So the compute becomes very different. So when in pre-training, we would talk about a model, I think if we go back to like, the Biden administration executive order, it's like,

• 2:06:22Speaker 02

  10 to the 25th flops to train a model. If you're using flops in post-training, it's a lot weirder, because the reality is just like, how many hours are you allocating how many GPUs for? And I think in terms of time, the RL compute is getting much closer, because you just can't Put it all into one system like pre-training is so computationally dense where all the GPUs are talking to each other and it's extremely efficient. Where RL has all these moving parts and it can just take a long time to generate a sequence of 100,000 tokens. Like if you think about GPT 5.2 Pro taking an hour, it's like, what if your training run has a sample for an hour and you have to make it so that's handled efficiently? So I think in GPU hours or just like wall clock hours, the RL runs are probably approaching the number of days as pre-training,

• 2:07:07Speaker 02

  but they probably aren't using as many GPUs at the same time. There's rules of thumb where in labs, it's like you don't want your pre-training runs to last more than like a month because they fail catastrophically. And if you were planning a huge cluster to be held for two months and then it fails on day 50, the opportunity cost is just so big. So you kind of don't want to just, people don't want to put all their eggs in one basket, which is like GPT 4 was like the ultimate YOLO run. And nobody ever wanted to do it before where it took like three months to train and everybody was shocked that it worked. Where I think people are a little bit more cautious and incremental now.

• 2:07:40Speaker 01

  So RLVR is more, let's say, unlimited how much you can train and get still benefit. Where RLHF, because it's a preference tuning, you reach a certain point where it doesn't really make sense to spend more RL budget on that. So just a step back with preference tuning. So there are multiple people that can give multiple, let's say, explanations for the same thing and they can both be correct. But at some point you learn a certain style and it doesn't make sense to, you know, iterate on it. My favorite example is like if relatives ask me what laptop they should buy, I give them an explanation or ask them like, yeah, what is your use case? Like they, for example, prioritize battery life and storage. Other people like us, for example, would prioritize RAM and compute.

• 2:08:23Speaker 01

  And so, but both, both answers are correct, but different people require different answers. And with preference tuning, well, you're trying to average somehow, like you are asking the data labelers to give you the right, well, not the right, the preferred answer. And then you train on that, but at some point, yeah, you learned that average preferred answer. And there's no, I think, reason to keep training longer on it because, you know, it's just a style. Where with RLVR, you literally give them. Well, you let the law model solve more and more complex, difficult problems. And so I think that it makes more sense to allocate more budget long term to L R V R. And also that right now we are in L R V R one point O land where it's still like that.

• 2:09:04Speaker 01

  Simple thing where we have a question and answer, but we don't do anything with the one stuff in between. So there was a, I mean, multiple research papers also by Google, for example, on process reward models that also give scores for the explanation, how correct is the explanation? And I think that will be the next thing, let's say, our L V R two point O for this year, focusing in between question and answer, like how to leverage that information, the explanation to improve the explanation and help it to get better accuracy. But then, uh, so that that's one angle. And there was a deep sea math version two paper where they also had interesting inference scaling there where, well, first they had, um, developed models that grade themselves a separate model.

• 2:09:49Speaker 01

  And I think that that will be one aspect and the other, like Nathan mentioned, it will be for all we are branching into other domains.

• 2:09:56Speaker 02

  The place where people are excited are value functions, which is very pretty similar. So process reward models are kind of like process reward models assign how good something is to each kind of intermediate step in a reasoning process where value functions apply value to every token. The language model generates. Both of these have been largely unproven in the language modeling and this reasoning model era, people are more optimistic about value functions forever. For whatever reason now, I think process reward models were tried a lot more in this pre oh one pre reasoning model era. And a lot of people had a lot of headaches with them. So I think a lot of it is the human nature of like value models have a very deep history in reinforcement learning.

• 2:10:39Speaker 02

  They're one of the first things that were core to like deep reinforcement learning existing is like training value models in this. So right now the literature people are excited about trying value models, but there's very little proof in it. And there are negative examples in trying to scale up process reward models. These things don't always hold in the future. I think. Came to this discussion by talking about scaling. And a simple way to summarize what you're saying with like, you don't want to do too much RLHF, which is eventually the signal scales is people have worked on RLHF for language models for years, especially in intense interest after ChatGPT. And this, the first release of a reasoning model trained with RLVR, OpenAI's O1, had a scaling plot where if you increase the training compute logarithmically, you get a linear increase in evaluations.

• 2:11:23Speaker 02

  And this has been reproduced multiple times. I think DeepSeek had a plot like this. But there's no scaling law for RLHF where if you log increase the compute, you get some performance. In fact, the seminal scaling paper for RLHF is scaling laws for reward model over optimization. So it's like, that's a big line to draw with RLVR. And the methods we have now and in the future, like they will follow the scaling paradigm, which is like the best runs you can let to run for an extra 10x and you get a few x performance. But you can't do this with RLHF. And that is just going to be field defining and how people approach them. Where I'm a shill for people academically to do RLHF. And that's a good way to describe it is like to do the best RLHF, you might not need the extra 10 or 100x of compute.

• 2:12:07Speaker 02

  But to do the best RLVR you do. So I think there's a, what I say is a seminal paper from what was a meta internship. It's called, it's like the art of scaling reinforcement learning with language models. They're what they describe as a framework as scale RL. And their incremental experiment was like 10,000 be 200 hours, which is like thousands or tens of thousands of dollars per experiment. And they do a lot of them, which is just like, this cost is not accessible to the average academic, which is a hard equilibrium where it's trying to figure out how to learn from each community.

• 2:12:43Lex Fridman?

  I was wondering if we could take at this point a bit of a tangent and talk about education and learning. If you're somebody listening to this, who's a smart person, interested in programming, interested in AI. So I presume building something from scratch is a good beginning. So can you just take me through like what you would recommend people do?

• 2:13:04Speaker 01

  So I would personally start, like you said, implementing a simple model from scratch that you can run on your computer. The goal is not if you build a model from scratch to have like something you use every day for your personal projects. Like it's not going to be your personal assistant replacing an existing open weight model or chat G P T. It's to see what exactly goes into the L L M, what exactly comes out of the L L M, how the pre-training works in that sense on your own computer preferably. And then you learn about the pre-training, the supervised fine-tuning, the attention mechanism. You get a solid understanding of how things work, but at some point you will reach a limit because small models can only do so much.

• 2:13:43Speaker 01

  And the problem with learning about L L Ms at scale is I would say it's exponentially more complex to make a larger model. Because it's not that the model just becomes larger. You have to now think about sharding your parameters across multiple G P U's. Even for the K V cache, there are multiple ways you can implement it. One is just to understand how it works, just to grow the cache. That's like a cache you grow step by step by, let's say, concatenating lists, growing it. But then that wouldn't be optimal in G P U's. You wouldn't do that. You would pre-allocate a tensor and then fill it in. But that adds again another twenty, thirty lines of code. And for each thing, you add so much code. And I think the trick with the book is basically to understand how the L L M works. It's not going to be your production level L L M.

• 2:14:26Speaker 01

  But once you have that, you can understand the production level.

• 2:14:29Lex Fridman?

  So you're trying to always build an L L M that's going to fit on one G P U?

• 2:14:33Speaker 01

  Yes, the most of them I have, they've, I have some bonus materials on some M O E models. I think one or two of them, they may require multiple G P U's, but the goal is to have it on one G P U. And the beautiful thing is also you can self-verify. It's almost like R L V R when you code these from scratch. You can take an existing model from the Hugging Face Transformer library. So the Hugging Face Transformer library is great, but if you want to learn about L L Ms, I think that's not the best place to start. Because the code is so complex, because it has to full, it has to fit so many use cases. Also, some people use it in production, it has to be really sophisticated and it's really intertwined and really hard. It's not linear to read.

• 2:15:11Speaker 02

  It was started as a fine-tuning library and then it grew to be like the standard representation of every. Model architecture and the way it is loaded. So Hugging Face is like the default place to get a model. And Transformers is the software that enables it. So people can easily load a model and do something basic with it.

• 2:15:29Speaker 01

  And all frontier labs that have open weight models have a Hugging Face Transformers version of it, like from DeepSeek to GPT OSS. That's like the canonical weights that you can load there. But again, also even Transformers, the library is not used in production. People use then SG lang or VLLM and it adds another layer of complexity.

• 2:15:47Lex Fridman?

  We should say that the Transformers library has like 400 models.

• 2:15:51Speaker 01

  So it's a one library that tries to implement a lot of L L Ms. And so you have a huge code base, basically. It's like huge. It's like, it's, I don't know, maybe millions, hundreds of thousands of lines of code. And the fine, it's like, understanding the part that you want to understand is finding the needle in the haystack. But what's beautiful about it is you have a working implementation. And so you can work backwards from it. What I recommend doing or what I also do is if I want to understand, for example, how almost three is implemented, I would look at the weights in the model hub, the config file. And then you can see, oh, they use so many layers. They use, let's say, group query attention or multi head attention in that case. And you see all the components in like a human readable, I don't know, 100 lines of config file.

• 2:16:35Speaker 01

  And then you start, let's say, with your GPT two model and add these things, you know? And the cool thing here is you can then load the pre-trained weights and see if they work in your model. And you want to match the same output that you get with a transformer model. And then you can use that as a, basically as a verifiable report to make your architecture correct. And then it's kind of, sometimes it takes me a day to, with almost three, the challenge was rope for the position embeddings. They had a yarn extension and there was some custom scaling there and I couldn't quite match these things. And in this struggle, you kind of understand things. But the cool thing is at the end, you know, you have it correct because you can unit test it, you can check against the reference implementation.

• 2:17:17Speaker 01

  And I think that's maybe one of the best ways to learn, really, like to basically reverse engineer something, yeah.

• 2:17:24Speaker 02

  I think that that is something that everybody that's interested in getting AI today should do. And I think that's why I liked your book is like I came to language models from this RL and robotics field. Like I've never had taken the time to just like learn all the fundamentals. And this transformer architecture I described as being like so fundamental as like deep learning was a thing that I had to learn in the past. And people need to do this. I think that where a lot of people kind of get overwhelmed is how do I apply this to have impact? Or find like a career path because like AI and language models make this fundamental stuff so accessible. And people with motivation will learn it. And then it's like, how do I get the cycles on goal to contribute to research?

• 2:18:07Speaker 02

  And I think that I'm actually fairly optimistic in this because the field moves so fast. That a lot of times the best people like don't fully solve a problem because there's a bigger, lower, like a bigger problem to solve. That's very low hanging fruit. So they move on. And I think that a lot of what I was trying to do in this RLHF book is like, take post-trading techniques and just describe how people think about them influencing the model and what people are doing. And then it's remarkable how many things I just think are just like people stop studying them or don't. So I think people trying to get narrow after doing the fundamentals is good. And then reading the relevant papers and being engaged in the ecosystem.

• 2:18:48Speaker 02

  It's like you actually, the proximity that random people have online from the leading researchers, like no one knows who all the anonymous account on X and ML is very popular for whatever reason. And no one knows who all these people are like, it could just be random people that study the stuff deeply, especially with the AI tools. And just be like, I don't understand this. Keep digging into it. I think is a very useful thing. But there's a lot of research areas that just like are maybe three papers that you need to read. And then one of the authors will probably email you back, but you have to put in a lot of effort into these emails to understand the field. Like I think it would be for a newcomer, easily weeks of work to feel like they can truly grasp like what is a very narrow era, area.

• 2:19:30Speaker 02

  But I think going narrow after you have the fundamentals would be very useful to people. Because it's like I became very interested in like character training, which is like how you make the model funny or sarcastic or serious. And like, what do you do to the data to do this? And it's like, A student at Oxford reached out to me. It's like, hey, I'm interested in this. And I advised him. And I was like, that paper now exists. And it's like, I don't know, there's like two or three people in the world that were very interested in this. He's a PhD student, which gives you an advantage. But like, for me, that was a topic I was waiting for someone to be like, hey, I have time to spend cycles on this. And I'm sure there's a lot more very narrow things where you're just like, oh, it doesn't make sense that there was no answer to this.

• 2:20:11Speaker 02

  And I think that it's just like, there's so much information coming that people are like, I can't grab on to any of these. But if you just actually stick in an area, I think there's a lot of interesting things to learn.

• 2:20:21Speaker 01

  Yeah, I think you can't try to do it all because it would be very overwhelming and you would burn out if you try to keep up with everything. For me, for example, I haven't kept up with computer vision a long time, just focused on LMS. But coming back to your book, for example, I think this is also a really great book and a really good bang for the buck. Because you want to learn about RLHF, I wouldn't go out there and read RLHF papers because I would be, you would be spending two years.

• 2:20:43Speaker 02

  Contradict. There's some, I just edited the book and I was like, there's a chapter where I had to be like, X papers say one thing and X papers say another thing. And we'll see what comes out to be true.

• 2:20:54Lex Fridman?

  What are some of the, just to go through some of the table of contents, some of the ideas we might have missed in the bigger picture of the post-training. So first of all, you do the problem setup, training overview, what are preferences, preferences, data in the optimization tools, reward modeling, regularization, instruction tuning, rejection sampling, reinforcement learning, I A policy gradients, direct alignment algorithms. Then constitutional AI and AI feedback, reasoning and inference time scaling, tool use and function calling, synthetic data and distillation, evaluation. And then open question section, over optimization style and information. And then product UX, character and post-training. So what are some ideas worth mentioning that connect both the educational component and the research component?

• 2:21:38Lex Fridman?

  You mentioned the character training, which is pretty interesting.

• 2:21:40Speaker 02

  Character training is interesting because there's so little out of it, but we talked about how people engage with these models and like. We feel good using them because they're positive, but that can go too far. It could be too positive. And it's like, essentially, it's how do you change your data? And or decision making to make it exactly what you want. And I, OpenAI has this thing called a model spec, which is essentially their internal guideline for what they want the model to do. And they publish this to developers. So essentially you can know what is a failure of OpenAI's training, which is like, they have the intentions and they haven't met it yet. Versus what is something that they like actually wanted to do and that you don't like. And that transparency is very nice, but all the methods for curating these documents and how easy it is to follow them is not very well known.

• 2:22:26Speaker 02

  I think the way the book is designed is that the reinforcement learning chapter is obviously what people want, because everybody hears about it with RLVR. And it's the same algorithms and the same map. But it's just like, you can use it in in very different documents. So I think the core of RLHF is like how messy preferences are. Is essentially a rehash of a paper I wrote years ago. But this is essentially the chapter that'll tell you why RLHF is never ever fully solvable. Because like, the way that even RL is set up is that it assumes that preferences can be quantified and that multiple preferences can be reduced to single values. And I think it relates in the economics literature to the von Neumann-Morgenstern utility theorem.

• 2:23:12Speaker 02

  And like, that is the chapter where all of that philosophical, economic and like psychological context, it tells you what gets compressed into doing RLHF. So it's like, you have all of this. And then later in the book, it's like, you use this RL math to make the number go up. And I think that that's why I think it would be very rewarding for people to do research on is because it's like, quantifying preferences is something that is just like, humans have designed the problem in order to make preferences studyable. But there's kind of fundamental debates on like, An example is in a language model response, you have different things you care about, whether it's accuracy or in style. And when you're collecting the data, they all get compressed into like, I like this more than another. And it's like, like that is happening and there's a lot of philosophy.

• 2:23:54Speaker 02

  There's a lot of research in other areas of the world that go into like, how should you actually do this? I think social choice theory is the subfield of economics around how you should aggregate preferences. And there's like, I was, I went to a workshop that published a white paper. I'm like, how can you think about using social choice theory for RLHF? So I mostly would want people that get excited about the math to come and have things where they can stumble into and learn this kind of broader context. I think there's a fun thing. I just keep a list of all the tech reports that I like of reasoning models. So in the, in chapter 14, which is kind of like a short summary of RLVR, there's just like a gigantic table where I just like list every single reasoning model that I like.

• 2:24:35Speaker 02

  So there's just like, I think in education, a lot of it needs to be like, at this point, it's like, what I like. Because the language models are so good at the math, where it's like famous paper, direct preference optimization, which is like a much simpler way of solving the problem than RL. The derivations in the appendix skip steps of math. And it's like, I tried for this book, like I redid the derivations and I'm like, what the heck is this log trick that they use to change the math? But doing it with language models, they're like, this is the log trick. And I'm like, I don't know if I like this, that the math is so commoditized. I think like some of the struggle in reading this appendix and following the math, I think is good for learning and I.

• 2:25:16Lex Fridman?

  Yeah. So we're actually returning to this often, just on the topic of education. You both have brought up the word struggle quite a bit. So there is value. If you're not struggling as part of this process, you're not fully following the proper process for learning, I suppose.

• 2:25:35Speaker 02

  Some of the providers are starting to work on models for education, which are designed to not give, actually, I haven't used them, but I would guess they're designed to not give all the information at once.

• 2:25:44Lex Fridman?

  Right.

• 2:25:44Speaker 02

  And make people work to do this. So I think you could train models to do this and it would be a wonderful contribution where like all of this stuff in the book. You had to reevaluate every decision for it, which is such a great example. I think there's a chance to work on it in AI too, which I was like, oh, I think this is.

• 2:25:59Speaker 01

  Makes sense. I do something like that, uh, did that the other day for video games. For example, I sometimes for my pastime play video games like I like, uh, video games with puzzles, you know, like Zelda and Metroid. And there's this new game where I got stuck and I really got stuck and was okay. I, you know, I don't want to struggle like two, uh, two days. And so I use an L L M, but then you say, hey, please don't, uh, add any spoilers. Just, you know, I'm here and there. What do I have to do next? And the same thing you can do, I guess, for math where you say, okay, I'm here at this point. I'm getting stuck. Don't give me the full solution, but what is something I could try? You know, like where you kind of carefully probe it. But the problem here is I think it requires discipline and a lot of people do math for like, I mean, there are a lot of people who enjoy math, but there are also a lot of people who

• 2:26:43Speaker 01

  need to do it for their homework. And then it's like the shortcut. And yeah, we can develop an educational L L M, but the other L L M is still there. And there's still a temptation to use the other L L Ms.

• 2:26:52Lex Fridman?

  I think a lot of people, especially in college, they, they understand the stuff they're passionate about. They're self-aware about it and they understand it shouldn't be easy. Like I think we just have to develop a good taste. Because our research taste, like school taste about stuff that you should be struggling on and, and stuff you shouldn't be struggling on, which is tricky to know. Cause sometimes you don't have good, um, long-term vision about what would be actually useful to you in your career. But you have to, you have to develop that taste.

• 2:27:24Speaker 02

  I was talking to maybe my fiance or friends about this and it's like, there's this brief 10-year window where all of the homework and all the exams could be digital. But before that, everybody had to do all the exams in blue book because there was another way. And now after AI, everybody's going to need to be in blue books and oral exams because everybody could cheat so easily. It's like this brief generation that had a different education system that like, everything could be digital. And but you still couldn't cheat and now she could go back. It's just pretty funny.

• 2:27:53Lex Fridman?

  You mentioned character training, just zooming out on a more general topic. For that topic, how much compute was required? And in general, to contribute as a researcher. Are there places where not too much compute is required where you can actually contribute as an individual researcher?

• 2:28:12Speaker 02

  For on the character training thing, I think this research is built on fine tuning about seven billion parameter models with Laura, which is like a essentially you only fine tune a small subset of the weights of the model. I don't know exactly how many GPU hours that would take.

• 2:28:27Lex Fridman?

  But it's doable.

• 2:28:28Speaker 02

  Not doable for every academic. So the situation for some academics is like so dire that the only work you can do is doing inference, where you have closed models or open models and you get completions from them and you can look at them and understand the models. And that's very well suited to evaluation, which you become, you want to be the best at creating representative problems that the models fail on or show certain abilities, which I think that you can break through with this. So I've like, I think that the top end goal for a researcher working on evaluation, if you want to have career momentum is the frontier labs pick up your evaluation. So it's like, you don't need to have every project do this. But if you go from a small university with no compute and you figure out something that Claude struggles with,

• 2:29:10Speaker 02

  and then the next Claude model has it in the blog post, like there, there's your career rocket ship. I think that that's hard, but it's like, if you want to scope the maximum possible impact with minimum compute, it's something like that, which is just get very narrow. And it takes learning of where the models are going. So you need to like, build a tool that tests where not Claude 4.5 will fail. If you're going to do a, if I'm going to start a research project, I need to think where the models in eight months are going to be struggling.

• 2:29:38Lex Fridman?

  But what about developing totally novel ideas?

• 2:29:41Speaker 02

  This is a trade off. I think that if you're doing a PhD, you could also be like, it's too risky to work in language models. I'm going way longer term, which is like, what is, what is the thing that's going to define language model development in 10 years? Which I think that I end up being a person that's pretty practical. I mean, I went to my PhD where it's like, I got into Berkeley, worst case I get a master's and then I go work in tech. It's like, I'm very practical about it. So I'm like, The life afforded to people to work at these AI companies, the amount of like OpenAI's average compensation is over a million dollars in stock a year for employee. Any normal person in the US to get into this AI lab is transformative for your life. So I'm pretty practical of like, there's still a lot of upward mobility working in

• 2:30:25Speaker 02

  language models if you're focused and the outcomes is like, look at these jobs. But from a research perspective, the transformative impact in these academic awards, it's like be the next Yann LeCun is from not working, not caring about language model development very much.

• 2:30:40Lex Fridman?

  It's a big financial sacrifice in that case.

• 2:30:42Speaker 02

  So I get to work with some awesome students and they're like, should I go work in an AI lab? And I'm like, like you're getting a PhD at a top school or you're going to leave to go to a lab? I'm like, I don't know. Like, if you go work at a top lab, I don't blame you. Don't go work at some random startup that might go to zero. But if you're going to OpenAI, I'm like, it could be worth leaving a PhD for.

• 2:31:02Lex Fridman?

  Let's more rigorously think through this. Where would you give a recommendation for people to do a research contribution? So the options are academia, so get, get a PhD, spend five years publishing. Computer resources are constrained. There's a, there's research labs that are more focused on open weight models and so working there or closed frontier labs, research labs.

• 2:31:34Speaker 02

  The

• 2:31:34Lex Fridman?

  OpenAI, Anthropic, XAI, so on.

• 2:31:37Speaker 02

  The two gradients are the more closed, the more money you tend to get and but also the will that you get less credit. So in terms of building a, like a portfolio of things that you've done, like it's very clear of what you have done as an academic and you have done this and versus if you are going to go be like trade this. Fairly reasonable progression for being a cog in the machine, which could also be very fun. So I think it's a very different career paths. But the like the opportunity cost for being a researcher is very high because PhD students are paid essentially nothing. So I think it ends up rewarding people that have a fairly stable safety net. And they've realized that they can operate in the long term, which is they want to do very interesting work and get a very interesting job.

• 2:32:22Speaker 02

  So it is a fairly like it's a privileged position to be like, I'm going to see out my PhD and figure it out after because I want to do this. And I think a lot of like, at the same time, the academic ecosystem is getting bombarded by funding, getting cut and stuff. So there's just like so many different trade offs where I understand plenty of people that are like, oh, I can't deal with this funding search. I grant got cut for no reason by the government, or I don't know what's going to happen. So I think there's a lot of uncertainty and trade offs that in my opinion favor just like, take the take the well paying job with meaningful impact. It's like not also like you're getting paid to sit around at OpenAI. You're building like the cutting edge of things that are changing millions of people's relationship to tech.

• 2:33:07Lex Fridman?

  But publication wise, they're being more secretive, increasingly so. So you're publishing less and less and less and less. And so you're you are having a positive impact at scale, but it's you're a cog in the machine.

• 2:33:20Speaker 01

  I think it's honestly, it hasn't changed that much. So I have been in academia, I'm not in academia anymore. At the same time, I wouldn't want to miss my time in academia. But what I wanted to say before I get to that part, I think it hasn't changed that much. I was working in like, I was using AI or machine learning methods for applications and computational biology with collaborators. And a lot of people went from academia directly to Google. And I think it's the same thing back then, professors were like, you know, sad that their students went into industry because they couldn't carry on their legacy in that sense. And I think it's the same thing. I mean, it's like, it hasn't changed, I think, that much. The only thing that has changed is the scale, but you know.

• 2:34:06Speaker 01

  Cool stuff was always developed in industry that was closed, you couldn't talk about it. And I think the difference now is, um, well, your preference, do you like to talk about your work, publish? Or, you know, you're, you are more in a closed lab.

• 2:34:21Speaker 01

  The, that's one difference, the compensation, of course, but it's always been like that, I think. So it really depends on, you know, where you feel comfortable and it's also nothing is forever. The only thing right now is there's a third option, which is I'm starting a startup. That's a lot of people doing startups, very risky move. But can be high, is a high risk, high reward type of situation where joining an industry lab, I think, is pretty safe. You know, also upward mobility. Honestly, I think if once you have been at a industry lab, it will be easier to find future jobs. But then again, you know, you know, it's like, yeah, how much do you enjoy the team and working on proprietary things? Whereas how do you like the publishing work?

• 2:35:05Speaker 01

  I mean, publishing is stressful. It is, um, you know, like acceptance rate at conferences can be arbitrary, can be very frustrating. But also high reward, if you have a paper published, you feel good because your name is on there. You have a high accomplishment and you know.

• 2:35:21Speaker 02

  Feel like my friends who are professors seem on average happier than my friends who work at a frontier lab. To be totally honest, because it's just grounding and the frontier labs definitely do this. Nine nine six, which essentially is shorthand for work all the time.

• 2:35:36Lex Fridman?

  Can you describe nine nine six as culture that's, I believe you could say invented in China and adopted in Silicon Valley? What's, what's nine nine six? It's nine AM to nine PM.

• 2:35:46Speaker 01

  Six days a week.

• 2:35:47Lex Fridman?

  Six days a week. What is that? Seventy two hours? Okay, so what, is this basically the standard in AI companies in Silicon Valley? More and more this kind of grind mindset.

• 2:35:59Speaker 01

  Yeah, I mean, not maybe not exactly like that, but I think there is a trend towards it. And it's interesting. I think it almost flipped because when I was in academia, I felt like that because as a professor, you had to write grants, you had to do, you had to teach and you had to do your research. It's like three jobs in one and it is more than a full time job if you want to be successful. And I feel like now like Nathan just said, the professors in comparison to a lab, I think they have less, like even maybe pressure or workload than at a frontier lab.

• 2:36:29Speaker 02

  Because they work a lot. They're just so fulfilled by like working with students and having a constant runway of mentorship and like a mission that is very people oriented, I think in an era when things are moving very fast and very chaotic is very rewarding to people.

• 2:36:44Speaker 01

  Yeah, and I think as a startup, I think it's this pressure. It's like you have to make it. And it's like, it is really important that people put in the time, but well, it is really hard because you have to deliver constantly. And I've been at a startup, I had a good time, but I don't know if I could do it forever. It's like a interesting pace and is exactly like we talked about in the beginning. These models are leapfrogging each other and they are just constantly like trying to take the next step compared to the competitors. It's just ruthless, I think, right now.

• 2:37:14Speaker 02

  I think this leapfrogging nature and having multiple players is actually an underrated driver of language modeling process where competition is so deeply ingrained to people. And these companies have intentionally created very strong culture, like Anthropic is known to be so culturally like deeply committed and organized. I mean, like we hear so little from them and everybody at Anthropic seems very aligned. And it's like being in a culture that is super tight and having this competitive dynamic is like, talk about a thing that's going to make you work hard and create things that are better. So I think that this, but that comes at the cost of human capital, which is like,

• 2:37:57Speaker 02

  you can only do this for so long and people are definitely burning out. I think I've. I wrote a post on burnout. I was like, I've treaded in and out of this myself, especially trying to like be a manager of full mode training. It's a crazy job doing this. The book Apple in China by Patrick McGee, he talked about the how hard the Apple engineers work to set up the supply chains in China. And he was like, they had saving marriage programs. And he told a podcast, he was like, people died from this level of working hard. So I think that's just like, it's a perfect environment for creating progress based on human expense. And I, it's, there's going to be a lot, there's a lot of, the human expense is the nine nine six that we started this with, which is like, people do really grind.

• 2:38:41Speaker 01

  I also read this book. I think they had a quote word for if someone had to go home to spend time with their family to save the marriage. And it's crazy. Then colleagues, I say, okay, this is like red alert for this situation. We have to let that person go home this weekend. And, um, but at the same time, I don't think they were forced to work. It's really, they were so passionate about the product, I guess, that it is, it is, you, you get into that mindset. And I had that sometimes as an academic, but also as an independent person, I have that sometimes I overwork and it's unhealthy. I had, you know, I had back issues, I had neck issues because I did not take the breaks that I maybe should have taken. But it's not because no one forced me to, it's because I wanted to work because it's exciting stuff.

• 2:39:18Speaker 02

  That's what OpenAI and Interlative are like. They want to do this work.

• 2:39:22Lex Fridman?

  Yeah, but there's also, there's also a feeling, a fervor that's building, especially in Silicon Valley, aligned with the scaling laws idea where there's this hype, where the world will be transformed in a scale of weeks and you want to be at the center of it. And then, you know, um, I have this great fortune of having conversations with wide variety of human beings. And from there I get to see all these bubbles and echo chambers across the world. And it's fascinating to see how we humans form them. And I think it's fair to say that Silicon Valley is a kind of echo chamber, uh, a kind of, um, silo and bubble. I think bubbles are actually really useful and effective.

• 2:40:04Lex Fridman?

  It's not necessarily a negative thing because it could be ultra productive. It could be the, the Steve Jobs. Reality distortion field because you just convince each other the breakthroughs are imminent. And by convincing each other of that, you make the breakthroughs imminent.

• 2:40:21Speaker 02

  Bern Hubert wrote a book classifying bubbles, but essentially one of them is financial bubbles, which is like speculation, which is bad. And the other one is like, I don't know the term, but effectively for build outs because it pushes people to build these things. And I do think AI is in this, but I worry about it transitioning to a financial bubble, which is like it's.

• 2:40:38Lex Fridman?

  Yeah, but also in the space of ideas, that bubble, you are doing a reality distortion field, and that means you are deviating from reality. And if you go too far from reality while also working, you know, nine, nine, six, and you might miss some fundamental aspects of the human experience, including in Silicon Valley. And this is a common problem in Silicon Valley. It's like, it's a very specific geographic area. You might not understand the Midwest perspective, the full experience of all the other different humans in the United States and across the world. And you, and you speak a certain way to each other. You convince each other of a certain thing and that that can get you into real trouble.

• 2:41:20Lex Fridman?

  Whether AI is a big success and becomes a powerful technology, or it's not in either trajectory, you can get yourself into trouble. So you have to consider all of that. Here you are, a young person trying to decide what you want to do with your life.

• 2:41:35Speaker 02

  The thing that is, I don't even really understand this, but the SF AI memes have gotten to the point where permanent underclass was one of them, which was the idea that the last six months of 2025 was the only time to build durable value in AI startup or model. Otherwise, all the value will be captured by existing companies and you will therefore be poor, which like, that's an example of the SF thing that goes so far. I still think for young people that going to be able to tap into it, if you are really passionate about wanting to have an impact in AI, like being physically in SF is the most likely place where you're going to do this, but it has, it has trade offs.

• 2:42:14Lex Fridman?

  I think SF is an incredible place, but there is a bit of a bubble. And if you go into that bubble, which is extremely valuable. Just get out also, read history books, read literature, visit other places in the world. Twitter is not, and Substack is not the entire world.

• 2:42:33Speaker 02

  I think I would say, one of my, one of the people I worked with is moving to SF. And it's like, I need to get him a copy of The Season of the Witch, which is a history of SF from like 1960 to 1985, which goes through like the hippie revolution, like they, all the gays kind of taking over the city and that culture emerging. And then the HIV AIDS crisis and other things. And it's just like, that is so recent and so much turmoil and hurt, but also like love in SF. And it's like, no one knows about this. It's a great book, Season of the Witch. I recommend it. A bunch of my SF friends who do get out, recommended it to me. And I think that is just like, living there, like I lived there and I didn't appreciate this context. And it's just like, so recent.

• 2:43:18Lex Fridman?

  Yeah. Okay, let's, we talked a lot about, we talked a lot about a lot of things. Certainly about the things that were exciting last year, but this year, one of the things you guys mentioned is exciting is the scaling of text diffusion models. And it's just a different exploration of text diffusion. Can you talk about what that is and what the possibility holds? Sort of different kinds of approaches than the current LLMs?

• 2:43:46Speaker 01

  Yeah, so we talked a lot about the transformer architecture and the autoregressive transformer architecture specifically like GPT. And it doesn't mean no one else is working on anything else. So people are always on the, let's say, lookout for the next big thing. Because I think it would be almost like, yeah, stupid not to. Because sure, right now, the transformer architecture is the thing and it works best and has right now, nothing else out there. But you know, it's always a good idea to not put all your eggs into one basket. So people are developing other things, alternatives to the autoregressive transformer. One of them would be, for example, text diffusion models. And listeners may know diffusion models from the image generation, like stable diffusion popularized. There

• 2:44:27Speaker 01

  was like a paper on generating images. Back then, people used GANs, the generative adversarial networks. And then there was this diffusion process where you iteratively denoise an image and resulted in really good quality images over time. Stable Diffusion was a company, other companies built their own diffusion models. And then people are now like, okay, can we try this also for text? Doesn't, you know, make intuitive sense yet because it feels like, okay, it's not something continuous, like a pixel that we can differentiate. It's like a discrete text. So how do we implement that denoising process? But it's kind of like similar to the BERT model by Google, like when you go back to the original transformer. So we're like the encoder and the decoder. The decoder is

• 2:45:09Speaker 01

  what we are using right now in GPT and so forth. The encoder, it's more like a parallel, let's say, technique where you have multiple tokens that you fill in in parallel. Instead, so GPT models, they do autoregressive one token at a time. You complete the sentence one token at a time. And in BERT models, you have a text, let's say a sentence that has gaps, you mask them out. And then one iteration is filling in these gaps. And text diffusion is kind of like that, where you are starting with, let's say, some random text, and then you are filling in the missing parts, or you're refining them iteratively, and you have multiple iterations. And the cool thing here is that this can do multiple tokens at the same time. So it's kind of like the promise

• 2:45:52Speaker 01

  of having it more efficient. Now, the trade off is, of course, well, how good is the quality? It might be faster. And then now you have this dimension of the denoising process, the more steps you do, the better the text becomes. And people, you know, I mean, you can scale in different ways. They try to see if that is maybe a valid alternative to the autoregressive model in terms of giving you the same quality for less compute. Right now, I think it's, you know, there are papers that suggest, okay, if you want to get the same quality, you have to crank up the denoising steps, and then you end up spending the same compute you would spend on an autoregressive model. The other downside is, well, it's parallel, which sounds appealing, but some tasks are not

• 2:46:36Speaker 01

  parallel, like, you know, like reasoning task tool use, maybe where you have to ask a code interpreter to give you an intermediate And that is kind of tricky with diffusion models. So there are some hybrids, but the main idea is can we parallelize it? And so interesting avenue, I think right now there are mostly research, uh, let's say models out there like Lada and some other ones. I I saw some by startup, some deployed models. There is no big diffusion model at scale yet, like, you know, like Gemini, ChatGPT scale on that level. But there was an announcement by Google or like a site where they said they are launching Gemini diffusion. And they put it into context of their, I think nano two model.

• 2:47:17Speaker 01

  And then they said basically for the same quality on most benchmarks, we can generate things much faster. So you mentioned what's next. I don't think the text diffusion model is going to replace autoregressive L L M's. But it will be something maybe for quick, uh, cheap at scale tasks. Maybe the free tier in future will be something like that.

• 2:47:37Speaker 02

  I think there's a couple of examples where it's, I I've heard that it's actually been started to be used. I think to paint an example of why this is so much better. For example, when G P T five is taking thirty minutes to respond, is generating one token at a time. And this diffusion idea is essentially generate all of those completion, all of those tokens in the completion in one batch, which is why it could be way faster. And I think it can be suited, the startups I'm hearing are like code startups where you have a code base. And you have somebody that's effectively vibe coding and they say, make this change. And a code diff is essentially a huge reply from the model. But it doesn't have to have that much external context and you can get it really fast by using these diffusion models.

• 2:48:18Speaker 02

  So that's what I've heard of one example is that they use these text diffusion to generate really long diffs. Because doing it with a autoregressive model would take minutes and that time for like a user facing product causes a lot of churn. So like every second you lose a lot of users. So I I think that's going to be this thing where it's going to grow and have some applications. But I actually thought that different types of models were going to be used for different things more sooner sooner than they have been. So I kind of trade off. I think that the tool use point is the one that's stopping them from being like most general purpose because like Claude code and this chat GPT with search like the auto regression chain is interrupted with some external tool and I don't know how to do that with the diffusion setup.

• 2:49:01Lex Fridman?

  So what's the future of tool use this year and then in the coming years? Do you think there's going to be a lot of developments there? How that's integrated to the entire stack?

• 2:49:10Speaker 01

  I do think right now, I mean, it's mostly on the proprietary LLM side, but I think we will see more of that in the open source tooling. And I think, I mean, it is a huge unlock because then you can really outsource certain tasks from just memorization to actual, you know, like instead of having the LLM memorize what is 23 plus five, just use a calculator.

• 2:49:31Lex Fridman?

  So you think that can help solve hallucination?

• 2:49:35Speaker 01

  Not solve it, but reduce it. So still the LLM needs to know what, like when to ask for a tool call. And the second one is, well, it doesn't mean the internet is always correct. You can do a web search, but let's say I asked who won the World Cup in, let's say, 1998, it still needs to find the right website and get the right information. So you can still go to the incorrect website and give me incorrect information. So I don't think it will fully solve that, but it is improving it in that sense. And so another cool paper earlier this year, I think it was December 31st. It's not technically 2026, but close. So like the recursive language model, that that's a cool idea to kind of take this even a bit further.

• 2:50:19Speaker 01

  So just to explain, so Nathan, you also mentioned earlier, it's harder to do cool research in academia because of the compute budget. If I recall correctly, they did everything with GPT-5. So they didn't even use local models, but the idea is, let's say, if a long context task, instead of having the LLM solve all of it in like one shot or even like in a chain, you break it down into subtasks. You have the LLM decide when like, what is a good, let's say, subtask and then recursively call an LLM to solve that. And I think something like that, also then adding tools and. You know, each one, maybe you have like a huge Q and A task. Each one goes to the web and gathers information and then you pull it at the end together and stitch it back together.

• 2:51:02Speaker 01

  Like where I think there's going to be a lot of unlock using things like that where you you necessarily improve the L L M itself. You improve how the L L M is used and what the L L M can use. One downside right now with tool use is you have to give the L L M permission to use tools. And that will take some trust, especially if you want to unlock things like having an L L M answer emails for you. A lot not even answer, but just sort them for you or select them for you or something like that. I don't know if I would today give an L L M access to my emails, right? I mean, it's like a huge risk.

• 2:51:36Speaker 02

  I think I think there's a cool one last point on the tool use thing. I think that you hinted at this and we've both come at this in our own ways is that the open versus closed models use tools in very different ways. Where open models, people go to Hugging Face and you download the model and then the person's going to be like, oh, what tool do I want? And I don't know, Exa is my search preferred search provider, but somebody else might care for a different search startup. Where you release a model, it needs to be useful for multiple tools for multiple use cases, which is really hard because you're making like a general reasoning engine model. Which is actually what G P T O S is good for. But on the closed models, you're deeply integrating the specific tool into your experience. And I think that open models will struggle to replicate some of the things that I like to do with closed models, which will be like, I don't know.

• 2:52:22Speaker 02

  You can reference a mix of public and private information and something that I keep trying every three to six months. I try like Codex on the Web, which is just prompting a model to make an update to some GitHub repository that I have. And it's just like, like that set of secure cloud environment is just so nice for just like, send it off and do this thing and then come back to me. And these will probably help define some of the local open and closed niches. But I think initially, because there was such a rush to get this tool use working, that the open models were on the back foot, which is kind of inevitable. I think there's so much research, so many resources of these frontier labs, but will be fun when the open models solve this.

• 2:53:04Speaker 02

  Because it's going to necessitate like a bit more flexible and potentially interesting model that might work with this recursive idea to like be an orchestrator and a tool use model. So hopefully the necessity drives some interesting innovation there.

• 2:53:17Lex Fridman?

  So continual learning, uh, this is a longstanding.

• 2:53:22Lex Fridman?

  Topic, important problem. I think that increases in importance as the cost of training of the models goes up. So can you explain what continual learning is and how important it might be this year and in the coming years to make progress?

• 2:53:35Speaker 02

  This relates a lot to this kind of SF zeitgeist of what is AGI? What is, which is artificial general intelligence and what is ASI? Artificial superintelligence and what are the language models that we have today capable of doing? I think the language models can solve a lot of tasks, but a key milestone among the AI community is essentially when AI could replace any remote worker, taking in information and solving digital tasks and doing them. And the limitation that's highlighted by people is that a language model will not learn from feedback the same way that an employee is. So if you hire an editor, the editor will mess up, but you will tell them. And if you hired a good editor, they don't do it again. But language models

• 2:54:16Speaker 02

  don't have this ability to modify themselves and learn very quickly. So the idea is if we're going to actually get to something that is a true like general adaptable intelligence that can go into any remote work scenario and needs to be able to learn quickly from feedback and on job learning. I'm personally more bullish on language models by being able to just provide them with very good context. You said, like you maybe offline said that, like you can write extensive documents to models where you say, I have all this information. Here's all the blog posts I've ever written. I like this type of writing. My voice is based on this. But a lot of people don't provide this to models and the models weren't designed to like take this amount of context previously, like the agentic models are just

• 2:54:59Speaker 02

  starting. So it's this kind of trade off of, do we need to update the weights of this model with this continual learning thing to make them learn fast? Or the counterargument is we just need to provide them with more context and information and they will have the appearance of learning fast by just having a lot of context and being very smart.

• 2:55:16Lex Fridman?

  So we should mention the terminology here. So continual learning refers to changing the weights continuously so that the model adapts, adjusts based on the new incoming information, does so continually and rapidly and frequently and so on. And then the thing you mentioned on the other side of it is generally be referred to as in context learning. As you learn stuff, there's a huge context window. You can just keep loading it with extra information every time you prompt the system, which I think both are legitimately can be seen as learning. It's just a different place where you're doing the learning.

• 2:55:57Speaker 01

  I think, to be honest with you, continual learning, the updating of weights, we already have that in different flavors. I mean, if you think about how, so I think the distinction here is, do you do that on a personalized custom model for each person? Or do you do it on a global model scale? And I think we have that already with going from GPT five to five point one and five point two. It's maybe not immediate, but it is like a curated update, a quick curated update where there was feedback by the things it couldn't do, feedback by the community, they updated the weights, next model and so forth. So it is, I mean, kind of like a flavor of that. Other even finer great example,

• 2:56:36Speaker 01

  a finer grained example is like RLVR, you run it, it updates. The problem is you can't just do that for each person because it would be too expensive to update the weights for each person. And I think that's the problem. So unless you get, I mean, even at OpenAI scale, building the data centers, it would be too expensive. I think that is only feasible once you have something on the device where the cost is on the consumer, like what Apple tried to do with the Apple Foundation models, putting them on the phone. And then they learn from the experience.

• 2:57:06Lex Fridman?

  A bit of a related topic, but this kind of maybe anthropomorphized term, but memory. What are the different ideas of the mechanism of how to add memory to these systems as you're increasingly seeing so? So personalized memory especially.

• 2:57:22Speaker 01

  So right now it's mostly like context, basically stuffing things into the context and then just recalling that. But again, I think, well, it's expensive because you have to, like, I mean, you can cache it, but still you spend tokens on that. And the second one is you can only do so much. I think. It's more like a preference or a style. I mean, a lot of people do that when they solve math problems. You say, uh, it's basically you can add previous knowledge and stuff, but you also give it certain preference prompts. Do what I preferred last time, whatever, like something like that. But it does, it doesn't unlock, uh, new capabilities. So for that, uh, one thing people do use still is Laura.

• 2:58:04Speaker 01

  Uh, Laura adapters, these are basically instead of updating the whole weight matrix. They are two smaller weight matrices, um, that you kind of have in parallel or overlay is like the delta. But, um, yeah, you, you can do that to some extent, but then again, it is economics. So there were also papers, for example, Laura learns less but forgets less. It's like, you know, it's no free lunch. If you want to learn more, you need to use more weights, but it gets more expensive. And then again, if you learn more, you forget more and it's like you have to find that Goldilocks zone, basically.

• 2:58:37Lex Fridman?

  We haven't really mentioned it much, but, uh, implied in this discussion is context length also. Is there a lot of innovations that's possible there?

• 2:58:45Speaker 02

  I think the colloquially accepted thing is that it's a compute and data problem where you can. And sometimes like small architecture things, which are like attention variance. So if you have, we talked about like hybrid attention models, which is essentially if you have what looks like a state space model within your transformer and like those are better suited because you have to spend less compute to model the furthest along token. And I think that, but those aren't free because they have to be accompanied by a lot of compute or, um, the right data. So how many sequences of a hundred thousand tokens do you have in the world? And where do you get these? And I think it just ends up being pretty expensive to scale them.

• 2:59:28Speaker 02

  So we've like gotten to pretty quickly to like a million tokens of input context length. And I would expect it to keep increasing and like get to like two million or five million this year. But I don't expect it to go to like a hundred million. That would be like a true breakthrough. And I think those breakthroughs are possible. Like the continual learning thing, I think of as a research problem where you could, there could be a breakthrough that just. Makes transformers work way better at this and it's cheap. Like, these things could happen with so much scientific attention. But turning the crank, it'll be consistent increases over time.

• 3:00:00Speaker 01

  I think also looking at the extremes, I think there's again no free lunch. So the one extreme to make it cheap, you have a, let's say, an R N N that has a single state where you save everything from the previous stuff. It's like a specific fixed sized thing. So you never really grow the memory because it's you are stuffing everything into one state. But then the longer the context gets, the more information you forget because you can't, you can't keep, I mean, compress everything into one state. Then on the other hand, you have the transformers, which try to remember every token, which is great sometimes, which we want to look up specific information, but very expensive because you have the KV cache that grows, the dot product that grows. But then, yeah, like you said, the Mamba layers, I mean, they kind of have the same problem, I would say, like an R N N, you try to compress everything into one state.

• 3:00:45Speaker 01

  You're a bit more selective there. But then I think it's like this Goldilocks zone again. With Nimiotron 3, they found like a good ratio of how many attention layers do you need for the global information where everything is accessible compared to having these compressed states? And I think that's how, I think we will scale more by finding better, let's say, ratios in Goldilocks zone, like between, like computing, making it cheap enough to run, but then also making it powerful enough to be useful. And one more plug here, the recursive language model paper, that is one of the papers that tries to kind of address the long context thing. So what they found is essentially instead of stuffing everything into this long context,

• 3:01:29Speaker 01

  if you break it up into the smaller, multiple smaller tasks, so you save memory by having multiple smaller calls, you can get actually better accuracy than having the L M try everything all at once. I mean, it's a new paradigm. We'll see, you know, there might be other flavors of that. So I think with that, we will still make improvement on long context. But then also, like Nathan said, I think the problem is for pre-training itself, we don't have as many long context documents as other documents. So it's harder to study, basically, how L M's behave and stuff like that on that level.

• 3:02:04Speaker 02

  Like. There are some rules of thumb where essentially you pre-train a language model, like, oh, no, we. Pre-trained to like 8K context length and then extended to 32K with training. And there's some rules of them where you're just like essentially doubling the training context length takes like 2X compute and then you can normally like 2 to 4X the context length again. So I think a lot of it ends up being kind of compute bound at pre-training, which is in this like we talked about this, everyone talks about this, big increase in compute for the top labs this year. And that should reflect in some longer context windows. But I think on the post-training side, there's some more interesting things, which is as we have agents, the agents are going to manage this context on their own, where now people that use Claude code a lot dread the compaction, which is when Claude takes its entire full

• 3:02:47Speaker 02

  100,000 tokens of work and compacts it into bulleted list. But what the next models will do, I'm just thought a novel, I'm sure people are already working on this, is essentially the model can control when it compacts and how. So you can essentially like train your RL algorithm where compaction is an action, where it shortens the history and then the problem formulation will be, I want to keep the maximum evaluation scores that I have gotten while the model compacts its history to the minimum length, because then you have the minimum amount of tokens that you need to do this kind of compounding auto-regressive prediction. So there's actually pretty nice problem setups in this where the like these agentic models learn to use their context in a different way than just plow forward.

• 3:03:29Speaker 01

  One interesting also recent example would be DeepSeek version 3.2, where they had like the sparse attention mechanism where they have essentially like a very efficient, small, lightweight indexer. And instead of attending to all the tokens, it selects, okay, what tokens do I actually need? It's, I mean, it's almost comes back to the original idea of attention where you are selective, but attention is always on, you have maybe zero weight on some of them, but you use them all, but they are even more like, okay, let's just mask that out or like not even do that. And even with sliding window attention, that is also kind of like that idea, you have that rolling window where you keep it fixed, because you don't need everything all the time. Occasionally some layers you might, but it's wasteful.

• 3:04:10Speaker 01

  But right now, I think, yeah, if you use everything, you're on the safe side, it gives you the best bang for the buck because you never miss information. And right now, I think. This year will be more also the year figuring out, like you said, how to be more smart about that. I think right now people wanna have the next state of the art and the state of the art is, happens to be the brute force. Expensive thing. And then, once you have that, like you said, keep that accuracy, but let's see how we can do that cheaper now, like tricks, you know?

• 3:04:39Speaker 02

  Yeah, all this scaling thing, like the reason we get the Claude four point five sonnet model first is because you can train it faster and you're not hitting these compute walls so soon. And they can just try a lot more things and get the model faster, even though the bigger model is actually better.

• 3:04:54Lex Fridman?

  I think we should say that there's a lot of exciting stuff going on in the AI space. My mind has recently been really focused on robotics. So we have today really almost entirely didn't talk about robotics. There's a lot of stuff on image gen, video generation.

• 3:05:12Lex Fridman?

  I think it's fair to say that the most exciting research work in terms of the amount, intensity. Fervor is in the L L M space, which is why I think it's justified for us to really focus on the L L M that we're discussing. But it'd be nice to bring in some certain things that might be useful. For example, world models, there's growing excitement on that. Do you think there'll be any use in this coming year for world models in the L L M space?

• 3:05:40Speaker 01

  Yes, I do think also also with L L Ms, what's an interesting thing here is I think if we unlock more L L M capabilities, it also automatically unlocks all the other fields because, or not unlocks, but like makes progress faster. Because, you know, a lot of researchers and engineers use L L Ms, like we said, for coding. So even if they work on robotics, if you like optimize these L L Ms that help you with coding, you know, it's like it pays off. But then, yes, world models are interesting. It's basically where you have the model run a simulation of the world in a sense, like a little toy thing of the real thing, which can again unlock capabilities.

• 3:06:19Speaker 01

  Like that are that L M is not aware of is can simulate things. And I think, see, this is like something I think L L Ms, they just happen to work. Well, by pre-training and then doing the next token prediction. But we could do this even a bit, you know, like sophisticated in a sense. So what I'm saying is like with, there's like, I think it was by Meta, a paper, a code of world models. So where they basically apply the concept of world models to L L M's again, where they, and so instead of just having next token prediction and verifiable rewards, checking the answer correctness. They also make sure the intermediate variables are correct, you know, like it's kind of like a, the model is learning basically a code environment in a sense.

• 3:07:01Speaker 01

  And I think this makes a lot of sense. It's just like expensive to do, but it is like making things more sophisticated, like modeling, like modeling the whole thing, not just the result. And so it can add more value. I remember when I was a grad student, there is a, so there's a competition called CASP, I think, where they do a protein structure prediction. Like they predict the structure of a protein that is not solved yet at that point. So in a sense, this is actually great. And I think we need something like that for L L M's also where you do the benchmark, but no one does. So you hand in the results, but no one knows the solution. And then after the fact, someone reveals that.

• 3:07:47Speaker 01

  But Alpha Fold, when it came out, it crushed, you know, this benchmark. I mean, there were also multiple iterations, but I remember the first one, I'm not an expert in that subfield, but the first one explicitly modeled the physical interactions of the, you know, the physics of the molecule, also like the angles, impossible angles. And then in the next version, I think they got rid of this. So, and just with brute force scaling it up. And I think with L L M's, we are currently in this brute force scaling because it just happens to work. But I do think also at some point, it might make sense to bring back this thing. And I think with code, with world models, I think that is where I think that might be actually quite cool.

• 3:08:29Speaker 01

  I mean, yeah, and of course also for robotics, that is a completely unrelated from L L M's.

• 3:08:36Lex Fridman?

  Yeah, yeah. And robotics is very explicit. So there's the problem of locomotion or manipulation. Locomotion is much more solved, especially in the learning domain. But there's a lot of value, just like with the initial protein folding systems, bringing in the traditional model based methods. So you don't, it's, it's unlikely. That you can just learn the manipulation or the whole body, local manipulation problem end to end.

• 3:09:00Speaker 02

  Hmm.

• 3:09:00Lex Fridman?

  That's the dream. But then you realize, when you look at the magic of the human hand

• 3:09:05Speaker 02

  Hmm.

• 3:09:06Lex Fridman?

  And the complexity of the real world, you realize it's really hard to learn this all the way through. The way I guess Alpha Fold two did.

• 3:09:13Speaker 02

  I'm excited about the robotic learning space. So I think it's collectively getting like supercharged by all the excitement and investment in language models generally where they're getting like the infrastructure for training transformers, which is like a general modeling thing is becoming like world-class industrial tooling where any, wherever that was a limitation for robotics. It's just like way better. There's where more compute. And then on top of like, they take these language models and use them as kind of central units where you can do interesting explorative work around something that kind of already works. And then I see it emerging as like, kind of like we talked about hugging face transformers and hugging face. I think when I was a hugging face, I was trying to get this to happen, but it was too early.

• 3:09:55Speaker 02

  It's like these open robotic models on hugging face and be having people be able to contribute data and fine tune them. I think we're much closer now that the investment in robotics and I think self-driving cars is related and enables this where it's like, once you get to the point where you can have this sort of ecosystem where somebody can download a robotics model and maybe fine tune it to their robot or shared data sets across the world and there's some data, there's some work in this area like RTX, I think is a few years ago where people are trying to do that. But I think once they have this ecosystem, it'll look very different. And then this whole post chat, chat GPT boom is putting more resources into that, which I think is a very good area for doing research.

• 3:10:35Lex Fridman?

  This is also resulting in much better, more accurate, more realistic simulators being built, uh, closing the sim to real gap in the robotic space. But, you know, you mentioned a lot of excitement in the robotics space and a lot of investment. The downside of that, which happens in hype cycles. I personally believe most robotics people believe that the, it's not robotics is not going to be solved at, at the time scale as being kind of implicit or explicitly promised. And so what happens when there's all these robotics companies that spring up? And then they don't have a product that works. Then there's going to be this kind of crash of excitement, which is nerve wracking.

• 3:11:19Lex Fridman?

  There's hopefully something else will come in and keep swooping in. So that the the the continued development of some of these ideas keeps going.

• 3:11:26Speaker 01

  I think it's also related to the continual learning issue, essentially where the real world is so complex. Where with L L M's, yeah, you don't need to really have something learned for the user. Because there are a lot of things everyone has to do. Everyone maybe wants to, I don't know, fix their grammar in their email or code or something like that. It's it's more constrained, so you can kind of prepare the model for that. But preparing the robot for the real world, that's harder. I mean, you have the foundation models, the robotic foundation models, but you you can learn certain things like grasping things. But then again, I think everyone's house is different, you know, like it's so different. And that is, I think, where the robot would have to learn on the job essentially.

• 3:12:08Speaker 01

  And I think that, I guess, is the bottleneck right now, like how to, you know, customizing it on the fly, essentially.

• 3:12:15Lex Fridman?

  I do, I don't think I can possibly understate the importance of the thing that doesn't get talked about almost at all by robotics folks or anyone is safety. All the interesting complexities, we talk about learning, all the failure modes and failure cases, everything we've been talking about L L M. Sometimes it fails in its interesting ways. All of that is fun and games in the L L M space. In the robotic space, in people's homes, across millions of minutes, billions of interactions. You really are almost allowed to fail never. When you have embodied systems that are put out there in the real world, you you just have to solve so many

• 3:12:59Lex Fridman?

  problems you never thought you'd have to solve. And you're just thinking about the general robot learning problem.

• 3:13:05Speaker 01

  And so bearish on in home.

• 3:13:08Speaker 02

  Learned robots for consumer purchase. I'm very bullish on self-driving cars and I'm very bullish for robotic automation, e.g. like Amazon distribution, where Amazon has built whole new distribution centers designed for robots first rather than humans. There's a lot of excitement in AI circles about AI enabling automation and like mass scale manufacturing, and I do think that the path to robots doing that is more reasonable, where it's like a thing that is designed and optimized to do a repetitive task that a human could conceivably do but doesn't want to. And then I'm, but it's also going to take a lot longer than people probably predict. I think that the leap from AI singularity to we can now scale up mass manufacturing

• 3:13:53Speaker 02

  in the US because we have a massive AI advantage is one that is troubled by a lot of political and other challenging problems.

• 3:14:04Lex Fridman?

  Let's talk about timelines, specifically timelines to AGI or ASI. Is it fair, like as a starting point, to say that nobody really agrees on the definitions of AGI and ASI?

• 3:14:19Speaker 02

  I kind of think there's a lot of disagreement, but among, I've been getting pushed back where a lot of people kind of say the same thing, which is like a thing that could reproduce most digital economic work. So like the remote worker is a fairly reasonable example, and I think OpenAI's definition is somewhat related to that, which is like an AI that can do a lot of, like a certain number of economically valuable tasks, which I don't really love as a definition, but I think it could be a grounding point because language models today were immensely powerful, are not this remote worker drop-in. And there are things that you could think of that are, could be done by an AI that are way harder than remote work, which are like solving a, finding an unexpected scientific

• 3:15:04Speaker 02

  discovery that you couldn't even posit, which would be an example of something that somebody says is like an artificial superintelligence problem, or like taking in all medical records and finding linkages across certain illnesses that people didn't know, or figuring out that some common drug can treat some niche cancer. Like they would say that that is like a superintelligence thing. So these are kind of natural tiers. My problem with it is that it becomes deeply entwined with like the quest for meaning of AI and this religious aspects to it. So there's kind of different, there's different paths you can take it.

• 3:15:39Lex Fridman?

  And I don't even know if the remote work is a good definition because what exactly is that? It's like perfect tool use. I actually, I mean, I like, I don't know if you like the originally titled AI 27 report. They focus more on code and research taste. So the, the target there is the superhuman coder. So they have several, several milestone systems, superhuman coder, superhuman AI researcher, then superintelligent AI researcher and then the full ASI, artificial superintelligence. But the, after you develop the superhuman coder, everything else falls quickly. There, the task is to have a fully autonomous, like automate coding.

• 3:16:24Lex Fridman?

  So any kind of coding you need to do in order to perform research is fully automated. And from there, humans would be doing AI research together with that system. And they will quickly be able to develop a system that's actually can do the research for you. That's the idea. And then initially their prediction was 2027, 28. And now they've pushed it back by three to four years to, to 2031 mean prediction. Probably my prediction is even beyond 2031, but at least you can in a concrete way, think about how difficult it is to fully automate programming.

• 3:17:04Speaker 02

  Yeah, I disagree with some of their presumptions and dynamics on how it would play out. But I think they did a good, they did good work in the scenario defining milestones that are concrete and to tell a useful story, which is why the reach for this AI 2027 document well transcended Silicon Valley is because they told a good story and they did a lot of rigorous work to do this. I think the camp that I fall into is that like AI is like so-called jagged, which will be excellent at some things and really bad at some things. So I think that when they're close to this automated software engineer, what it will be good at is that traditional ML systems and front end, the model is excellent at. But the distributed ML, the models are actually really quite bad at

• 3:17:47Speaker 02

  because there's so little training data and doing large scale distributed learning and things. And this is something that we already see, and I think this is just get amplified. And then it's kind of messier in these trade offs. And then there's like, how do you think AI research works and so on?

• 3:18:00Lex Fridman?

  So you think basically superhuman coder is almost unachievable? Meaning like because of the jagged nature of the thing, you're just always going to have gaps in capabilities.

• 3:18:11Speaker 02

  I think it's assigning completeness to something where the models are kind of superhuman at some types of code. And I think that will continue and people are creative, so they'll utilize this like incredible abilities. And like to fill in the weaknesses of the models and move really fast. There'll always kind of be this, I've received for a long time, this dance between the humans are enabling this thing that the model can't do. And the best, the best AI researchers, the ones that can enable this superpower. And I think this aligns like to what we already see. I think like Claude code for building a website, you can stand up a beautiful website in a few hours or do data analysis. And I don't think it's going to keep getting better at these things. It'll pick up some new code skills and stuff that'll get along the way.

• 3:18:52Speaker 02

  And kind of linking to what's happening in, in big tech is like this AI 2027 report is like, it leans into the singularity idea where I think research is messy and social and largely in the data in ways that AI models can't process. But like what we do have today is really powerful and these tech companies are all collectively buying into this with tens of billions of dollars of investment. So like, we are going to get some much better version of chat GPT, a much better version of Claude code than we already have. I think that it's just like hard to predict where that is going, but the like,

• 3:19:28Speaker 02

  bright clarity of that future is why some of the most powerful people in the world are putting so much money into this. And I think it's just kind of small differences between like, we don't actually know what a better version of chat GPT is. But also like, can it automate AI research? I would say probably not, at least in this timeframe. Like big tech is going to spend a hundred billion dollars much faster than we get a automated AI researcher that enables a AI research singularity.

• 3:19:55Lex Fridman?

  So you think your prediction would be what? Like if this is even a useful milestone, we're more than 10 years out.

• 3:20:03Speaker 02

  I would say less than that on the software side, but I think longer than that on the things like research.

• 3:20:09Lex Fridman?

  It's just like. For fun, try to imagine a world where all software writing is fully automated. Can you imagine that world?

• 3:20:19Speaker 02

  By the end of this year, the amount of software that'll be automated will be so high. But it's like, it'll be the things of like, you're trying to train a model with RL and you need to have multiple bunches of GPUs communicating with each other. That'll still be hard, but I think it'll be much easier.

• 3:20:34Lex Fridman?

  One of the ways to think about this, the full automation of programming, is just think of like lines of useful code written, the fraction of that to the number of humans in the loop. So presumably there'll be for a long time, humans in the loop of software writing is just be fewer and fewer relative to the amount of code written. Right? And the the SC superhuman coder, I think the the presumption there is it goes to zero, the number of humans in the loop. What does that world look like when the number of humans in the loop is in the hundreds, not in the hundreds of thousands?

• 3:21:12Speaker 02

  I think software engineering will be driven more to system design and goals of outcomes where I do think software is largely going to be. Come on, I think this has been happening over the last few weeks where people have gone from a month ago of like, oh, AI agents are kind of slop, which is a famous carpety quote to like the what is a little bit of a meme of like the industrialization of software when anyone can just create software at their fingerprints. Like, I do think we are closer to that side of things and it takes direction and like understanding how the systems work to extract that best from the language models. And I think it's hard to like accept the gravity of how much is going to change with software development and how many more people can do things without ever looking at it.

• 3:21:55Speaker 01

  I think what's interesting is to think about whether these systems will be independent, like completely independent in the sense that, well, I have no doubt that LLMs will kind of at some point solve coding in a sense like calculators solve calculating, right? So at some point, humans develop the tool that, you know, you never need a human to calculate that number. You just type it in and it's an algorithm. You you can do it in a in that sense. And I I think that's the same probably for coding, but the question is, so I think what will happen is, yeah, you will just say, build that website. It will make a very good website and then you maybe refine it. But will it do things independently where? So will you be still having humans asking the AI to do something like, will there be a person say, build that website?

• 3:22:40Speaker 01

  Or will there be AI that just builds websites or something or whatever?

• 3:22:44Lex Fridman?

  I think using, talking about building websites is the.

• 3:22:48Speaker 02

  Too simple.

• 3:22:49Lex Fridman?

  It's just like the, the, the problem with websites and the problem with the web, you know, HTML and all that kind of stuff. It's very resilient to just slop.

• 3:22:58Speaker 01

  Yeah.

• 3:22:58Lex Fridman?

  It will show you slop as good as showing slop. I would rather like, think of like safety critical systems, like uh, asking AI to end to end generate something that manages logistics or manages cars, a fleet of cars, all that kind of stuff. So end to end generates that for you.

• 3:23:18Speaker 02

  I think a more intermediate example is take something like Slack or Microsoft Word. I think if the organizations allow it, AI could very easily implement features end to end and do a fairly good job for like things that you want to try. You want to add a new like tab in Slack that you want to use. And I think AI will be able to do that pretty well.

• 3:23:38Lex Fridman?

  Actually, that's a really great example. How far away are we from that?

• 3:23:41Speaker 02

  Like this year.

• 3:23:44Lex Fridman?

  See, I don't, I don't know. I don't know.

• 3:23:47Speaker 02

  I guess I don't know how bad production code bases are, but I think that within like on the order of low years, a lot of people are going to be pushed to be more of like a designer and product manager, where you have multiple of these agents that can try things for you. And they might take one to two days to implement a feature or attempt to fix a bug. And you have these dashboards, which I think Slack is actually a good dashboard where your agents will talk to you. And you'll then give feedback, but things like, like I make a website. It's like, you want to make a logo that's passable? Like, I think these like cohesive design things and this style is going to be very hard for models and deciding on what to add at the next time.

• 3:24:26Lex Fridman?

  I just, okay, so I hang out with a lot of programmers and some of them are a little bit on the skeptical side in general. That's just vibe wise, they're like that.

• 3:24:38Lex Fridman?

  I just think there's a lot of complexity involved in adding features to complex systems. Like if you look at the browser, Chrome. If I wanted to add a feature, if I wanted to have tabs as opposed to up top, I want them on the left side. Interface, right? I think we're not, it's not a next year thing.

• 3:24:58Speaker 02

  One of the Claude releases this year, one of their tests was we give it a piece of software and leave Claude to run to recreate it entirely. And it could already almost rebuild slack, like slack from scratch, just given the parameters of the software and left in a sandbox environment.

• 3:25:13Lex Fridman?

  From the scratch part, I like almost better.

• 3:25:17Speaker 02

  So it might be that the smaller, newer companies are advantaged and they're like, we don't have to have the bloat and complexity. And therefore this future exists.

• 3:25:26Speaker 01

  And I think this gets to the point where you mentioned that some people you talk to are skeptical. And I think that's not because the L M can't do X, Y, Z. It's because people don't want it to do it this way.

• 3:25:38Lex Fridman?

  Some of that could be a skill issue on the human side. Unfortunately, we have to be honest with ourselves. And some of that could be an underspecification issue. So programming, like you're like, you're just assuming, this is like in relationships and friendships, communication type issue. You're assuming the L M somehow is supposed to read your mind. I think this word spectrum design is really important. Like you're just using natural language specify like what you want.

• 3:26:05Speaker 02

  I think that's like, if you talk to people at the labs, they use these in their training and production code. Like Claude code is built with Claude code and they all use these things extensively. And Dario talks about how much Claude's code owns. And it's like, these people are slightly ahead in terms of the capabilities they have. And they probably spend on inference. They could spend 10 to 100 plus X as much as we're spending, like we're on a lowly a hundred or $200 a month plan. Like they truly let it rip. And I think that that, like with the pace of progress that we have, it seems like, like where a year ago, we didn't have Claude code. And we didn't really have reasoning models. And it's like the difference between sitting here today and what we can do with these models.

• 3:26:48Speaker 02

  And it seems like there's a lot of, like, there's a lot of low hanging fruit to improve them. The failure modes are. Pretty dumb. It's like Claude, you tried to use the CLI command I don't have installed 14 times and then I sent you the command to run. It's like that thing from a modeling perspective is pretty fixable. So I

• 3:27:07Lex Fridman?

  I agree with you. I've been becoming more and more bullish in general. Speaking to what you're articulating, I think it is a human skill issue. So Anthropic is leading the way or other companies in understanding how to best use the models for programming, therefore they're effectively using them. I think there's a lot of programmers on the outskirts. They're like, they don't, I mean, there's not a really good guide on how to use them. People are trying to figure it out exactly.

• 3:27:37Speaker 02

  It might be very expensive. Like, it might be that the entry point for that is $2,000 a month, which is only tech companies and rich people. Just like, like, that could be it.

• 3:27:46Lex Fridman?

  But it might be worth it. I mean, if the final result is a working software system, well, it'd be worth it. But by the way, it's funny how we converge from the discussion of timeline to AGI to something more pragmatic and useful. Is there anything concrete and interesting and useful and profound to be said about timeline to AGI and ASI? Are these discussions a bit too detached from the day to day?

• 3:28:11Speaker 02

  There's interesting bets. So there's a lot of people trying to do reinforcement learning with verifiable rewards, but in real scientific domains where there's startups that are spending, like, they have hundreds of billions of dollars of funding and they have wet labs where they're having language models, propose hypotheses that are tested in the real world. And I would say that I think they're very early, or they're early, but with the pace of progress, it's like, maybe they're early by six months and they make it because they were there first, or maybe they're early by eight years. You don't really know. So I think that that type of moonshot to branch this momentum into other sciences is like, okay, like, that would be very transformative if like, alpha fold

• 3:28:52Speaker 02

  moments happen in all sorts of other scientific domains by like, a startup solving this. I think there are startups, I think maybe Harmonic is one where they're going all in on language models plus lean for math. I think you had another podcast guest where you talked about this recently and it's like, We don't know exactly what's going to fall out of spending a hundred million dollars on that model. And most of them will fail, but a couple of them might be big breakthroughs that are very different than chat GPT or Claude code type software experiences. Like a tool that's only good for a PhD mathematician, but makes them a hundred X effective like.

• 3:29:30Speaker 01

  Okay, I agree. I think this will happen in a lot of domains, especially also like domains that have a lot of, you know, resources like finance and legal and pharmaceutical companies. But then again, is it really AGI again, because we are now specializing it again. And then again, is it really that much different from back in the day, how we had specialized algorithms? I think it's just the same thing more way more sophisticated. But I don't know, is there a threshold when we call it AGI? I guess I think the the real cool thing is here that we have like the the foundation models that we can specialize. I think that that's like the the breakthrough at some point right now. I think we are not there yet because, well, first it's too expensive, but also, you know, like chat GPT doesn't just give away that chat GPT to customize it.

• 3:30:15Speaker 01

  I think the once that's going to be true in some way. And I think I can imagine this as a business model that chat GPT, OpenAI says at some point like, hey, you know, Bank of America, four hundred million, we will do your custom model or something like that. And I I think that will be the huge economic value add. The other thing though is also companies, I mean, right now, what is the differentiating factor? I mean, if everyone uses the same LLM, if everyone uses chat GPT, they will all do the same thing again. I mean, then, well, it's everyone is moving in lockstep. But usually companies, they want to have a competitive advantage. And I think there's the no way around using some of their private data and experimenting and maybe specializing. It's going to be interesting, yeah.

• 3:30:59Speaker 02

  Sitting in the pace of progress, it does just feel like things are coming. I don't think the AGI and ASI thresholds are particularly useful.

• 3:31:08Lex Fridman?

  I I think I guess the real question, and this takes us to the remote worker thing is, when are we going to see a a big obvious leap in economic impact? Because currently, there's not been an obvious leap in economic impact of LLM models, for example.

• 3:31:28Lex Fridman?

  And that's, you know, aside from AGI or ASI or all that kind of stuff. There's a real question of like, when are we going to see a GDP like?

• 3:31:37Speaker 02

  Mm hmm.

• 3:31:37Lex Fridman?

  Jump.

• 3:31:39Speaker 02

  Yeah, it's like, what is the GDP made up of? Like, a lot of it is like financial services. So like, I don't, I don't know what this is. It's just hard for me to think about the GDP bump, but like, I'd say that software development becomes valuable in a different way when you no longer have to look at the code anymore. So when, when it is like Claude will make you a small business, which is essentially Claude can set up your website, your bank account, your email and your whatever else. And like, you just have to express like what you're trying to put into the world. Like, that's not just a enterprise market, but it is a hard, like, I don't know how you get people to try doing that. I guess if ChatGPT can do it, like, people are trying ChatGPT.

• 3:32:21Lex Fridman?

  I think it boils down to the, the scientific question of how hard is tool use to solve?

• 3:32:28Lex Fridman?

  There's a lot of the stuff you're implying, the remote work stuff is to, to use. It's like how computer use, like how you have an LM that goes out there, this agentic system and does something in the world and only screws up 1% of the time.

• 3:32:44Speaker 02

  Computer use is a good example of what labs care about, and we haven't seen a lot of progress on. We saw multiple demos in 2025 of like Claude can use your computer or OpenAI had Kua and they all suck. So like, they're also investing money in this. And I think that will be a good example, where that's actually something where it just seems pretty like taking over the whole screen seems a lot harder than having an API that they can call in the backend. And some of that is you have to then set up a different environment for the model to work in. Like, they're not working on your MacBook. They are individually interfacing with Google and Amazon and Slack. And they handle all of these things in a very different way than humans do. So some of those might be structural blockers.

• 3:33:28Speaker 01

  Also, like specification wise, I think the problem is also for, you know, arbitrary tasks. Well, you still have to specify what you want your LLM to do and how do you do that in a? What is the environment? How do you specify? You can say what the end goal is, but if it can't solve the end goal with LLMs, if you ask it for text, you can always, you know, clarify, do substeps. What is, how do you put that information into a system that, let's say, books a travel trip for you? You can say, well, you screwed up my credit card information, but even to get it to that point, like, how do you, like, as a user guide, the model before, like, can even attempt that? I think the interface is really hard.

• 3:34:08Lex Fridman?

  Yeah, it has to learn a lot about you specifically. And about, this goes to continue learning about the general mistakes that are made throughout and then mistakes that are made through you.

• 3:34:21Speaker 02

  All the AI interfaces are getting set up to ask humans for input. And then Claude Code, we talked about a lot, it asks feedback on questions. If it doesn't have enough specification on your plan or your desired, it starts to ask questions, would you rather? We talked about memory, which saves across chats, which it's

• 3:34:39Speaker 02

  first implementation is kind of odd, where it'd be like, it'll mention my dog's name or something, like in a chat. I'm like, you didn't need to be subtle about this, like, I don't care. But the, like, things that are emerging are ChatGPT has the Pulse feature, which is like

• 3:34:54Speaker 02

  a curated couple paragraphs with links to something to look at or to talk about. And people talk about how the language models are going to ask you questions, which I think is a very, it's probably going to work. The language model is like, it knows you had a doctor appointment or something. It's like, hey, how are you feeling after that? Which is like, again, goes into the territory of humans are very susceptible to this and there's a lot of social change to come. But also, like, they're experimenting with having the models engage. Some people really like this Pulse feature, which is it processes your chats and automatically searches for information and puts it in the ChatGPT app. So there's a lot of things coming.

• 3:35:31Speaker 01

  I used that feature before and I always feel bad because it does that every day and I rarely check it out. It's like how much money like I mean, compute is burned on something I don't even look at, you know, where it's like, it's kind of like.

• 3:35:44Speaker 02

  There's also a lot of idle compute in the world, so don't feel too bad.

• 3:35:49Lex Fridman?

  Okay. Do you think new ideas might be needed? Is it possible that the path to AGI, whatever that is, however we define that, to solve computer use more generally? To solve biology and chemistry and physics, sort of the Daria definition of AGI or Parfaya. Do you think it's possible that totally new ideas are needed? Non LLM, non RL ideas. What might they look like? This is, we're now going into philosophy land a little bit.

• 3:36:23Speaker 02

  For something like a singularity to happen, I would say yes. And the new ideas can be architectures or training algorithms, which is like fundamental deep learning things. But there's in that nature, pretty hard to predict. And I, but I think we will get very far even without those advances. Like we might get this software solution, but it might stop at software and not do computer use without more innovation. So I think that it's like, a lot of progress will be coming, but in, if you're going to zoom out, like there's still ideas in the next 30 years that are going to look like that was a major, like scientific innovation that enabled the next chapter of this. And I don't know if it comes in one year or in 15 years.

• 3:37:05Lex Fridman?

  Yeah, I wonder if the bitter lesson holds true for the next hundred years, what that looks like.

• 3:37:10Speaker 02

  If scaling laws are fundamental in deep learning, I think the bitter lesson will always apply, which is compute will become more abundant, but even within abundant compute, the ones that have a steeper scaling law slope or a better offset, like this is a 2D plot of performance and compute. And like, even if there's more compute available, the ones that get 100X out of it will win.

• 3:37:34Lex Fridman?

  It might be something like literally computer clusters orbiting Earth.

• 3:37:40Lex Fridman?

  With solar panels.

• 3:37:42Speaker 02

  The problem with that is heat dissipation. So you get all the radiation from the sun and you don't have any air to dissipate heat. But there is a lot of space to put clusters. There's a lot of solar energy there and you could figure out the heat dissipation. But there is a lot of energy and there probably could be engineering will to solve the heat problem. So there could be.

• 3:38:00Lex Fridman?

  Is it possible? And we should say that it definitely is possible. How likely is is the question? That we're basically going to be plateauing this year. Not in terms of the system capabilities, but what the system capabilities actually mean for human civilization. So on the coding front, really nice websites will be built. Very nice autocomplete. Very nice way to understand code bases and maybe help debug. But really just a very nice helper on the coding front. It can help research mathematicians do some math. It can help you with shopping. It can help you with it could help. It's a nice helper. It's clip you on steroids.

• 3:38:47Lex Fridman?

  What else? It may be a good education tool and all that kind of stuff. But computer use turns out extremely difficult to solve. So I'm trying to be I'm trying to frame the cynical case in all these domains where it kind of. There's not a really huge economic impact. We realize how costly it is to train these systems at every level, both the pre-training on the inference. How costly the inference is, the reasoning, all of that.

• 3:39:17Lex Fridman?

  Like, is that possible? And how likely is that, do you think?

• 3:39:20Speaker 02

  When you look at the models, there's so much obvious things to improve. And it takes a long time to train these models and to do this art. And that it'll take us with the ideas that we have multiple years to actually saturate in terms of whatever benchmark or performance we are searching for. It might serve very narrow niches, like the average chat GPT 800 million user might not get a lot of benefit out of this, but it is going to serve different populations by getting better at different things.

• 3:39:51Lex Fridman?

  But I think what everybody's chasing now is the is a general system that's useful to everybody. So, okay, so if that's not, that can plateau, right?

• 3:40:00Speaker 02

  I think that dream is actually kind of dying. As you talked about with the specialized models where it's like. And multimodal is often like video generation is a totally different thing.

• 3:40:11Lex Fridman?

  That dream is kind of dying is a big statement. Cause I don't know if it's dying. I don't know if every, I don't, if you ask the actual fancy a lot of people, they, I mean, they're still chasing it, right?

• 3:40:21Speaker 01

  I do think they are still like rushing to get the next model out, which will be much better than the, much is a right of term, but will be better than the previous one. And I, I, I can't see them slowing down. I just think the gains will be made or felt more through not only scaling the model, but now fine. So I, I feel like there's a lot of tech debt. It's like, well, let's just put the better model in there and better model, better model. And now, uh, people are, okay, let's also at the same time, improve everything around it too. Like, you know, like the engineering of the context and inference scaling. And I, the big labs will still keep doing that. And now also the smaller labs will catch up to that because now it's just like they are hiring more.

• 3:41:04Speaker 01

  There will be more people, L L Ms. It's kind of like, you know, like a circle. They also make them more productive and it, it's just, it's like amplify. I think what we can expect is amplification, but not like a, a change of, like a paradigm change. I don't think that is true, but the, everything will be just amplified and amplified and amplified. And I, I can see that continuing for a long time. Yeah.

• 3:41:25Speaker 02

  Yeah. I guess my saving with the dream is dying depends on exactly what you think it's going to be doing. Like Claude Code is a general model that can do a lot of things, but it's not like necessarily like it depends a lot on integrations and other things. Like I bet Claude Code could do a fairly good job of doing your email. And the hardest part is figuring out how to give the information to it and how to get it to be able to send your emails and stuff like this. But that's just kind of like, I think it goes back to like, what is the one model to rule everything ethos? Which is just like a thing in the cloud that handles your entire digital life and is way smarter than everybody. It's like, it's operating in a.

• 3:42:07Speaker 02

  So it's an interesting leap of faith to go from Claude code becomes that, which like in some ways is there's some avenues for that, but I do think that like the rhetoric of the industry is a little bit different.

• 3:42:22Speaker 01

  I think the immediate also thing we will feel next as a normal person using L L M's is will will probably be related to something like also trivial, like making figures right now. L L M's are terrible at making figures. Is it because we are getting served the cheap models with very less or like lesser inference compute than behind the scenes, maybe some like there are some cranks we can already get better figures. But if you ask today, I don't draw a flow chart of X, Y, Z, it's most of the time terrible. And it is kind of like a very simple task for a human. I think it's almost easier sometimes to draw something than to write something.

• 3:42:57Speaker 02

  Yeah, the multimodal understanding does feel like something that is odd that it's not better solved.

• 3:43:04Lex Fridman?

  I I think we're not saying one actually obvious thing that we're not actually realizing that's a gigantic thing that's hard to measure, which is making all of human knowledge accessible to the entire world like we I I one of the things I think is hard to articulate, but there's just a huge difference between Google search and then L L M. Like I feel like I can basically ask an L M anything and get an answer. And less is doing less and less and less hallucination. And that means understanding my own life, figuring out a career trajectory, figuring out how to solve the problems all around me.

• 3:43:46Lex Fridman?

  Learn about anything through human history that like I I feel like nobody's really talking about that. Because they just immediately take it for granted that it's just, this is awesome. That's why everybody's using it is because you get answers for stuff. And like the impact of that across time, like think about, this is not just an United States is all across the world, like kids throughout the world being able to learn these ideas, like the impact that has across time is probably, that's where the real like talking about GDP, it won't be like a leap, it'll be, that's how we get to Mars, that's how we build these things, that's how we have

• 3:44:29Lex Fridman?

  a million new Open AIs, all the kind of innovation that happens from there. And that's just this quiet force that permeates everything, right? Human knowledge.

• 3:44:39Speaker 01

  I do agree with you and in a sense, it makes, it makes knowledge more accessible, but it also, I think, depends on what the topic is. For something like math,

• 3:44:51Speaker 01

  in a sense, you can ask it questions, it answers, but if you wanna learn a topic from scratch, I think that, again, like we talked about this earlier, I think the sweet spot is, I mean, there are really good math textbooks where someone laid it out linearly and that is like, let's say proven strategy to learn this topic. And it does make sense if you start from zero to ramp up to get like a, like a information dense text to soak it up, but then you use the L L M to make infinite exercises. Like you, you have problems in a certain area and or have questions, something is uncertain or like you are uncertain about certain things. You ask it to generate example problems, you solve them and you have questions and then

• 3:45:31Speaker 01

  maybe you need, you need more background knowledge and you ask it to, to generate that. And I think, but then the, it won't give you anything, let's say, that is not in the, the textbook, it's just packaging it differently, if that makes sense. But then there are things I feel like where it also adds value in a more, I mean, timely sense where there is no good alternative besides a human doing it on the fly. For example, if you, I don't like, let's say you're planning to go to Disneyland and you try to figure out which tickets to buy for which park when, well, there is no textbook on that, there is no information dense resource in that, there's only the sparse internet.

• 3:46:11Speaker 01

  And then there is a lot of value in the L L M, you just ask it, it has, you have the constraints, I'm traveling these and these days, I want to go there and there. Please figure out what I need, when and from where, and what it costs and stuff like that. And it is very customized on the fly uh package. And then this is like one of the thousand examples, an exercise personalized uh personalization is essentially like pulling information from the sparse internet, the non-information dense thing where there is no better version that exists. It just doesn't exist. You make it from scratch almost.

• 3:46:45Lex Fridman?

  And if it does exist, it's full of uh speaking of Disney World, like full of, what would you call it, ad slop? Like you just, it's impossible uh here, you go any city in the world. What are the top ten things to do? L L M is just way better to ask than anything on the internet.

• 3:47:02Speaker 02

  For now. That's because they're massively subsidized and they're going to be paid for by ads.

• 3:47:10Speaker 02

  It's coming.

• 3:47:10Lex Fridman?

  No, oh, I hope there, I mean, I'm hoping there's a very clear indication of what's an ad and what's not an ad in that context.

• 3:47:19Speaker 01

  I did a little, I mean, that is something I mentioned a few years ago. It's like, uh, I don't know, if you are looking for a new running shoe. Well, is this a coincidence that Nike maybe comes up first? Maybe, maybe not. And but I think there are clear laws around this. You have to be clear about that. But I think that's what everyone fears. It's like this subtle, um, you know, subtle message in there or something like that. But it also brings us to the topic of, I guess, ads, uh, where I think this was the thing, oh, may I try to launch in two thousand twenty five? And uh just to, because I think it's still not uh making money in that other way right now. So that like having really like ad spots in there. And then the thing though is they couldn't because, well,

• 3:48:01Speaker 01

  there are alternatives without ads and people would just flock to the other products. And it also is is just like crazy how, yeah, like they're one-upping each other, spending so much money to just get the users.

• 3:48:14Speaker 02

  I think so, like some Instagram ads, I don't use Instagram, but I understand the.

• 3:48:20Speaker 02

  Appeal of paying a platform to find users who will genuinely like your product. And that is the best case of things like Instagram ads. But there are also plenty of cases where advertising is very awful for incentives. And I think that a world where the power of AI can integrate with that positive view of like, I am a person and I have a small business and I want to make the best, I don't know, damn steak knives in the world and I want to sell them to somebody who needs them. And if like, if AI can make that sort of advertising thing work even better, that's very good for the world, especially with like digital infrastructure, because that's how like the modern web has been built.

• 3:49:00Speaker 02

  But that's not to say like addicting feeds so that you can show people more content is a good thing. So it's like, I think that's even what OpenAI would say is they want to find a way that can make the monetization upside of ads while still giving their users agency. And I'm, I personally would think that Google is probably going to be better at figuring out how to do this because they have, they already have ad supply and they figure out how to turn this demand in their Gemini app into useful ads, then they can turn it on. And somebody will figure, I don't know if I think it's this year, but there will be experiments with it.

• 3:49:38Speaker 01

  I do think what holds companies back right now is really just that the competition is not doing it. It's more like, more like a reputation thing. It's just like, I think people are just afraid right now, like ruining or like losing their reputation, losing users, because it is, it would make headlines if someone launched these ads.

• 3:49:55Speaker 02

  Unless they were great, but the first ads won't be great because it's a hard problem that we don't know how to solve.

• 3:50:00Speaker 01

  Yeah, I think also the first version of that will likely be something like on X, like the timeline where you have like a promoted post sometimes in between. It will be something like that where it will say like promoted or something like small and then there will be an image or something. I think right now the problem is who makes the first move.

• 3:50:15Speaker 02

  If we go 10 years out, the proposition for ads is that you will make so much money on ads by having so many users that you can use this to funnel better R and D and make better models, which is why like YouTube is dominating the. Market for any like Netflix is scared of YouTube like they have the ad like they make I don't I pay twenty eight dollars a month for premium. They make at least twenty eight dollars a month off of me. And many other people and they're just like creating such a dominant position in video. So I think that's the proposition, which is that ads can make you have a sustained advantage in what you're spending per user. But there's so much money in it right now that it's like, like somebody starting that flywheel is scary because it's a long term bet.

• 3:51:01Lex Fridman?

  Uh, you think there'll be some like crazy big moves this year business wise? Like somebody like Google or Apple acquiring Anthropic or something like this?

• 3:51:12Speaker 02

  Dario will never sell, but we are starting to see some types of consolidation with like Grok for twenty billion dollars and um, Scale AI for almost thirty billion and countless other deals like this that they're structured in a way that is actually detrimental to the Silicon Valley ecosystem, which is this sort of licensing deal where not everybody gets brought along rather than a full acquisition that benefits the rank and file employee by getting their stock vested. Like that's a big issue for Silicon Valley culture to address because the startup ecosystem is the lifeblood where if you get a, if you join a startup, even if it's not that successful, your startup very well might get acquired on a cheap

• 3:51:54Speaker 02

  premium of it and you'll get paid out for this equity and these licensing deals are essentially taking the top talent a lot of the times. I think Grok, the deal for Grok to Nvidia is rumored to be better to the employees, but it is still this antitrust avoiding thing, but I think that this trend of consolidation will continue. I've been, me and many smart people I respect have been expecting consolidation to have happened sooner, but it seems like some of these things are starting to turn, which, but at the same time, you have companies raising ridiculous amounts of money for. Reasons that you don't like, I'm like, I don't know why you're taking that money. So it's maybe like mixed this year, but some consolidation pressure is starting.

• 3:52:37Lex Fridman?

  What kind of surprising consolidation do you think we'll see? So you say Anthropic is a never. I mean, Grok is a big one. Grok with a Q, by the way.

• 3:52:45Speaker 02

  Yeah, there are just a lot of startups and there's a very high premium on AI startups. So there's a lot of like, there could be a lot of

• 3:52:51Lex Fridman?

  That kind of stuff, yeah.

• 3:52:52Speaker 02

  10 billion range acquisitions, which is a really big acquisition for a startup that was maybe founded a year, like a year ago, I think Manus AI from this company that's based in Singapore that Meta founded was founded eight months ago and then had a $2 billion exit. And I think that there will be some other big, like many billion dollar acquisitions like

• 3:53:12Lex Fridman?

  Like perplexing, right?

• 3:53:12Speaker 02

  Yeah, like people rumored them to Apple. I think there's a lot of pressure and liquidity in AI. There's pressure on big companies to have outcomes. And I would guess that a big acquisition gives people leeway to then tell the next chapter of that story.

• 3:53:29Lex Fridman?

  I mean, yeah, I guess Cursor, we've been talking about code and somebody acquires Cursor.

• 3:53:34Speaker 02

  They're in such a good position by having so much user data.

• 3:53:38Lex Fridman?

  Yeah.

• 3:53:38Speaker 02

  And we talked about continual learning and stuff. They had one of the most interesting, like two sentences in a blog post, which is that they had their new composer model, which was a fine tune of one of these large mixture of expert models from China. You can know that by asking gossip or because the model sometimes responds in Chinese, which none of the American models do. And they had a blog post where they're like, we're updating the model weights every 90 minutes based on real world feedback from people using it, which is like the closest thing to real world RL happening on a model. And it's just like in one of their blog posts, which is super cool.

• 3:54:10Lex Fridman?

  And by the way, I should, I say I use Composer a lot because one of the benefits it has is it's fast.

• 3:54:16Speaker 02

  I need to try it because everybody says this.

• 3:54:18Lex Fridman?

  And there'll be some IPOs potentially. You think Anthropic, OpenAI? XAI.

• 3:54:24Speaker 02

  They can all raise so much money so easily that they don't feel a need to like so long as fundraising is easy. They're not going to IPO because public markets apply pressure. I think we're seeing in China that the ecosystem is a little different with both Minimax and Z dot AI. Applying for filing IPO paperwork, which will be interesting to see how the Chinese market reacts. I actually would guess that it's going to be like similarly hypey to the US. So so long as all this is going and not based in the realities that they're both losing a ton of money. I wish more of the American gigantic AI startups were public because it would be very interesting to see how they're spending their money and have more insight. And also just to give people access to investing in these because I think that they're some of the most like formidable.

• 3:55:08Speaker 02

  They're the companies of the era and the tradition is now for so many of the big startups in the US to not go public. It's like we're still writing for Stripe and the IPO, but Databricks definitely didn't. They raised like a series G or something. And I just feel like it's a kind of a weird equilibrium for the market where it's like. I would like to see these companies go public and evolve in that way that a company can.

• 3:55:34Lex Fridman?

  You think 10 years from now, some of the frontier model companies are still around. Anthropic, OpenAI.

• 3:55:41Speaker 02

  I definitely don't see it to be a winner takes all unless there truly is so algorithmic secret that one of them finds. They glass this flywheel because the development path is so similar for all of them. Google and OpenAI have like all the same products and then like Anthropic is more focused. But when you talk to people, it sounds like they're solving a lot of the same problems. So I think, and there's offerings that'll spread out. There's a lot of, it's a very big cake that's being made that people are going to take money out of.

• 3:56:09Lex Fridman?

  I I don't want to trivialize it, but so OpenAI and Anthropic are primarily LLM service providers. And some of the other companies like Google and XAI linked to X does other stuff too. And so it's very possible if AI becomes more commodified. That the companies that are just providing LLM will die.

• 3:56:33Speaker 01

  I think they will, the advantage they have, they have a lot of users and I think they will just pivot. I think then if they figure out, it's like Anthropic, I think pivoted. I don't think they originally planned to work on code, but it happened that they found, okay, this is like a nice niche. And now we are comfortable in this niche and we push on this niche. And I can see the same thing once, maybe let's say hypothetically speaking, I'm not sure if it will be true, but let's say Google takes all the market share of the general chatbot. Maybe OpenAI will be then focus on some other topic like the, they have too many users to go away in foreseeable future, I think.

• 3:57:10Lex Fridman?

  I think Google is always ready to say, hold my beer with AI mode.

• 3:57:13Speaker 02

  I think that the question is if the companies can support the valuations. I think I'd see the AI companies being looked at in some ways like AWS, Azure and GCP are all competing in the same space and all very successful businesses. There's a chance that the API market is so unprofitable that they go up and down the stack to products and hardware. They have so much cash that they can build power plants and build data centers, which is a durable advantage now. But there's also just a reasonable outcome that these APIs are so valuable and so flexible for developers that they become the likes of, like something like AWS. But AWS and Azure are also going to have these APIs. So there's some, like that's a, like five or six people competing in the API market is hard.

• 3:57:57Speaker 02

  So maybe like that's why they get squeezed out.

• 3:57:59Lex Fridman?

  You mentioned RIP Lama. Is there a path to winning for Meta?

• 3:58:04Speaker 02

  I think nobody knows. They're moving a lot, so they're signing licensing deals with Black Forest Labs, which is an image generation or mid journey or client maintenance. So I think in some ways it's on the product and like consumer facing AI front. It's too early to tell. I think they have some people that are excellent and very motivated being close to Zuckerberg. So I think that there's still a story to unfold there. Lama is a bit different where Lama was the most focused expression of the organization. And I don't see Lama being supported to that extent. I think it was a very successful brand for them. So they still might do some part of participation in the open ecosystem

• 3:58:49Speaker 02

  or continue the Lama brand into a different service. If people know what Lama is.

• 3:58:53Lex Fridman?

  You think there's a Lama five?

• 3:58:56Speaker 01

  Not an open weight one. It's interesting. I think also just to recap a bit, I think, I mean, Lama was the, I would say, pioneering open weight model and then Lama one, two, three, lot of love. But I think then, I think what happened, just hypothesizing or speculating, I think the leaders at Meta, like the upper executives, they, I think they got really excited about Lama because they saw how popular it was in the community. And then I think the problem was trying to, let's say, monetize the open, not monetize the open source, but like kind of use the open source to make a bigger splash in a sense, like to kind of force it, almost it felt forced, like developing these very big Lama four models to

• 3:59:39Speaker 01

  have like the best, like to be on the top of the benchmarks. But I don't think the goal of Lama models is to be on top of the benchmarks, beating, let's say, ChatGPT or other models. I think the goal was to have a model that people can use, trust, modify, understand. So that includes having smaller models. They don't have to be the best models. And what happened was just these models were, of course, like the benchmarks suggested they were better than they were by, because I think they had like specific models trained on preferences that they performed well on the benchmarks. So it's kind of like this overfitting thing to kind of force it to be the best. But then at the same time, they didn't do the small models that people could use. And I think that no one could run these big models then. And then it was kind of like a weird thing.

• 4:00:21Speaker 01

  And I think it's just because people got too excited about headlines pushing the frontier. I think.

• 4:00:27Lex Fridman?

  And too much like on the benchmarking side.

• 4:00:29Speaker 01

  Yeah, yeah.

• 4:00:30Speaker 02

  I think it imploded under political, like internal political fighting and misaligned incentives. So I think the researchers want to build the best models, but there's a layer of organization and manager that is trying to demonstrate that they do these things. And then there's lots of, there's a lot of pieces and rumors where how like some horrible technical decision was made and how that comes in. And it just seems like. It kind of got too bad where it all just crashed out.

• 4:00:57Lex Fridman?

  Yeah, but we should, we should also like give huge props to Mark Zuckerberg. I think it comes from Mark, actually, from Mark Zuckerberg, from the top of the leadership, saying open source is important. I think that's like that if the fact that that exists means there could be a llama five, where they learn the lessons from the benchmarking and say, we're gonna be GPT OSS and provide really awesome library of open source.

• 4:01:24Speaker 02

  What people say is that there's a debate between Mark and Alexander Wong, who is very bright, but much more against open source. And to the extent that he has a lot of influence over the AI org, it seems much less likely. Because it seems like Mark brought him in for like a fresh leadership aide in directing AI. And if the like open or closed is no longer the defining nature of the model, I don't expect that to be a defining argument between Mark and Alex. Like, they're both very bright, but I just like, I have a hard time understanding all of it. Because Mark wrote this piece in July of 2024, maybe, which was like probably the best blog post at the time, saying the case for open source AI.

• 4:02:07Speaker 02

  And then July 2025 came around and it was like, we're reevaluating our relationship with open source. So it's just kind of like.

• 4:02:15Speaker 01

  But I think also the, the problem, not the problem, but I think, well, we may have been a bit also too harsh, I think. And that caused some of that because I think, I mean, we as open source developers or the open source community, because I think even though the model was maybe not what everyone hoped for, it got a lot of backlash. And I think that was a bit unfortunate because I can see that as a company, now they were hoping for positive headlines and instead of just getting no headlines or not these positive headlines, in turn, they got negative headlines. And then all it kind of reflected bad on the company. And I think that is also something like where you, it's maybe a spite reaction almost like, okay, we have not, we, we tried to do something nice. We tried to give, give you something cool, like an open source model.

• 4:03:00Speaker 01

  And now you are like, you know. Kind of like be negative about us even like like for the company. So in that sense, it looks like, well, maybe then we'll change our mind. I guess. I don't know.

• 4:03:11Lex Fridman?

  Yeah, that's that's where the the dynamics of discourse on X can lead us as a community astray. Because sometimes it feels random. People pick the thing they like, they don't like. And you can see the same thing with Grok four one and Grok code fast one. I don't think vibe wise, people

• 4:03:36Lex Fridman?

  love it publicly, but a lot of people use it. So if you look to Reddit and X, they don't really give it praise from the programming community, but like they use it. And the same thing with probably with Llama. I don't understand. I don't understand the dynamics of either positive hype or negative hype. I don't understand it.

• 4:03:57Speaker 02

  I mean, the story of one of the stories of 2025 is the US feeling the gap of Llama, which is like all the rise of these Chinese open weight models to the point where I was like, that was the single issue I've spent a lot of energy on in the last five months is like trying to do policy work to get the US to invest in this.

• 4:04:14Lex Fridman?

  So tell me the story of Adam.

• 4:04:16Speaker 02

  Adam project is it started as me calling it the American Deep Sea Project, which doesn't really work for DC audiences, but it's the story of like, what is the most impactful thing I could do with my career, which is that it's Chinese open weight models are cultivating a lot of power and there is a lot of demand for building on these open models, especially in enterprises in the US that are very cagey about these Chinese models.

• 4:04:38Lex Fridman?

  Going to perplexity, the Adam project, American truly open models is a US based initiative to build and host high quality, genuinely open weight AI models and supporting infrastructure explicitly aimed at competing with and catching up to China's rapidly advancing open source AI ecosystem.

• 4:04:58Speaker 02

  I think the one sentence summary would be that, or two sentences. One is a proposition that open models are going to be an engine for AI research, because that is what people start with. Therefore, it's important to own them. And the second one is therefore, the US should be building the best models so that the best research happens in the US and the US companies take the value from being the home of where AI research is happening. And without more investment in open models, we have all the plots on the website where it's like, Quinn, Quinn, Quinn. And it's all these models that are excellent from these Chinese companies that are cultivating influence in the US and China and internationally. And I think the US is spending way more on AI

• 4:05:43Speaker 02

  and the ability to create open models that are half a generation or a generation beyond what the cutting edge of a closed labs is costs orders of like hundred million dollars, which is a lot of money, but not a lot of the money to these companies. So therefore, we need a centralizing force of people who want to do this. And I think we got signed engagement from people pretty much across the full stack, whether it's policy.

• 4:06:06Lex Fridman?

  So there has been support from the administration?

• 4:06:09Speaker 02

  I don't think anyone in the like technically in government has like signed it publicly, but I know that people that have worked in AI policy, both in Biden and Trump administration, are very supportive of trying to promote open source models in the US. I think, for example, AI two got a grant from the NSF for a hundred million dollars over four years, which is like the biggest CS grant the NSF has ever awarded. And it's for AI two to attempt to this. And I think it's a starting point. But the best thing happens when there are multiple organizations building models because they can cross pollinate ideas and kind of build this ecosystem. Like, I don't think if it just works if it's just Llama releasing models to the world, because then you can see Llama can go away. The same thing applies for AI two, where it's like,

• 4:06:53Speaker 02

  I can't be the only one building models. And I think that's like that it becomes a lot of time spent on talking to people, whether they're in policy. I know Nvidia is very excited about this. I think Jensen Huang has been specifically talking about the urgency for this and they've changed, they've done a lot more in 2025 where the Nematron models are more of a focus. They've started releasing some data along with Nvidia's open models and like very few companies do this, especially of Nvidia's size. So like there is, there is signs of progress and there we hear about Reflection AI where they say their $2 billion fundraise is dedicated to building US open models. And I feel like their announcement tweet is like,

• 4:07:35Speaker 02

  it reads like a blog post, right? And I think that that cultural tide is starting to turn. I think in July was when we had like four or five deep sea caliber Chinese open weight models in zero from the US. And that's, that's the moment where it was released this. And I was like, oh, I guess I have to spend energy on this because nobody else is going to do it. So it takes a lot of, it takes a lot of people contributing together. And I don't say that like the atom project isn't like the thing that's helping to move the ecosystem, but it's people like me doing this sort of thing to get the word out.

• 4:08:08Lex Fridman?

  Do you like the, the 2025 America's AI Action Plan that includes open source stuff? The White House AI Action Plan includes a dedicated section titled to encourage open source and open with AI, defining such models and arguing they have unique value for innovation and startups.

• 4:08:25Speaker 02

  Yeah. I mean, like the AI Action Plan is a plan, but largely I think it's like maybe the most coherent policy document that has come out of the administration and I hope that it largely succeeds. And I know people that have worked on the AI Action Plan and the challenge is taking policy and making it real. And I have no idea how to do this as an AI researcher, but like, like largely a lot of things in that were very real. And there's a huge build out of AI in the country. And it's like, there are a lot of issues that people are hearing about from water use to whatever. And like, we should be able to build things in this country, but also we need to not ruin places in our country in the process of building it and. It's worthwhile to spend energy on. I think that's a role that the federal government plays.

• 4:09:09Speaker 02

  It's like they set the agenda. And with AI setting the agenda, that open weight should be a first consideration is like, that's a large part of what they can do. And then people think about it.

• 4:09:21Speaker 01

  Also for education and talent for these companies, it's, I think, very important because otherwise, you know, if they're only closed on models, how do you get the next generation of people contributing at some point? Because otherwise you will at some point only be able to learn after you joined a company. But then at that point, like, how do you hire talented people? How do you identify talented people? And I think open source is, that's even for a lot of things, but also even just for educating the population and training the next generation of researchers. It's the way or the only way.

• 4:09:56Speaker 02

  The way that I could have gotten this to more, go more viral is, was to tell a story of Chinese AI integrating with an authoritarian state and being ASI and taking over the world. And therefore we need our own American models, but it's very intentional for why I talk about innovation and science in the US. Because I think it's both more realistic as an outcome, but just like, it's like, it's a world that is, that would like to manifest.

• 4:10:20Speaker 01

  I would say though, also even like, let's say, any open weight model I do think is a valuable model.

• 4:10:27Speaker 02

  Yeah, and my argument is that we should be in a leading position. But I think that it's worth saying it so simply because there are still voices in the AI ecosystem that say we should consider banning releasing open models due to the safety risks. And I think it's worth adding that I think effectively that's impossible without making the US like it have its own great firewall, which is also known to not work that well because the cost for training these models, whether it's one to a hundred million dollars, is attainable to a huge amount of people in the world that want to have influence. So these models will be getting trained all over the world. And these, we want the models, especially when like.

• 4:11:11Speaker 02

  I mean, there are safety concerns, but we want these information and tools to flow freely across the world and into the US so that we people can use them and learn from them. And we like stopping that would be such a restructuring of our internet that it seems impossible.

• 4:11:24Speaker 01

  Do you think maybe in that case, the big open weight models from China are actually a good thing? In a sense, like for the US companies, because maybe the US companies, you you mentioned earlier, they are usually one generation behind in terms of what they release open source versus what they are using. For example, GPT OS might not be the cutting edge model, Gemma three might not be. But they do that because they know this is safe to release. But then when they see these companies, you see, for example, there is DeepSeek version 3.2, which is really awesome, and it gets used and there is no backlash, there is no security risk. That could then again encourage them to release better models. Maybe that that in a sense is a very positive thing.

• 4:12:02Speaker 02

  A hundred percent. These Chinese companies have set things into motion that I think would potentially not have happened if they were not all releasing models. So I think it was like, I'm almost sure that those discussions have been had by leadership.

• 4:12:18Lex Fridman?

  Is there a possible future where the dominant models, AI models in the world are all open source?

• 4:12:23Speaker 02

  Depends on the trajectory of progress that you predict. If you think saturation in progress is even coming within a few years, though essentially within the time where financial support is still very good, then open models will be so optimized and so much cheaper to run that they will win out. Essentially, this goes back to open source ideas where so many more people will be putting money into optimizing this serving of these open weight common architectures that they will become standards. And then you could have chips dedicated to them and it'll be way cheaper than the offerings from these close companies that are custom.

• 4:12:58Lex Fridman?

  We should say that AI 27 report kind of predicts one of the things it does from a narrative perspective is that there will be a lot of centralization. As the AI system gets smarter and smarter, the national security concerns will come to be and you'll centralize the labs and you'll become super secretive and there'll be this whole race from a military perspective of how do you between China and the United States. And so all of this fun conversations we're having about LLMs. All the generals, the soldiers will come into the room and be like, all right, we're now in the Manhattan Project stage of this whole thing.

• 4:13:35Speaker 01

  I think two thousand twenty five, six, seven, twenty seven, I don't think something like that is even remotely possible. I mean, you you can make the same argument for computers, right? You can say, OK, computers are capable and we don't want the general public to get them. Or chips, even AI chips, but you see how like, you know, Huawei makes chips now, you know, took a few years. But and I think that I don't think there is a way you can contain something like that, like knowledge like that. I I think in this day and age, it is impossible, like the internet. That I don't think this is a possibility.

• 4:14:10Speaker 02

  On the Manhattan Project thing, one of my funny things making Adam is I think that like a Manhattan Project like thing for open models would actually be pretty reasonable because it wouldn't cost that much. But I think that that will come. It seems like culturally the companies are changing, but I agree with Sebastian and all the stuff that you just said. It's just like, I don't see it happening nor being helpful.

• 4:14:31Lex Fridman?

  Yeah, I mean, the the motivating force behind the Manhattan Project is there is civilizational risk. I it's harder to motivate that for open source models.

• 4:14:40Speaker 02

  There's not civilizational risk.

• 4:14:43Lex Fridman?

  Uh, you think, uh, on the hardware side, we mentioned Nvidia a bunch of times. Do you think Jensen and Nvidia are going to keep winning?

• 4:14:51Speaker 01

  I think they have the downside that they have to iterate a lot and manufacture a lot. And I think they probably, what what they're doing, they do innovate. But um, I think there's always the chance that there is something who does something fundamentally different, who gets very lucky and then does something. But the problem is, I think, adoption, you know, like the the mode of Nvidia is probably not just the GPU. It's more like the CUDA ecosystem and that has evolved over so many, I think, two decades. I think, I mean, even back when I was a grad student, we I was in a lab where we did biophysical simulations, molecular dynamics, and we had a Tesla GPU back then, just for the computation. It was, I mean, fifteen years ago now. And it's just they built this up for a long time and that.

• 4:15:36Speaker 01

  It's like that's the mode. I think it's not the the chip itself, although they have now the money to iterate and build and scale. But then it's really on the compatibility. It's like, well, if you're at that scale as a company, why would you go with something risky where it's only a few chips that they can make per year, you go with a big one. But then I do think with L L M's now also, it will be easier to design something like CUDA. You know, like the next, so it took fifteen years because it's hard, but then now we have L L M's, we can maybe replicate CUDA.

• 4:16:08Lex Fridman?

  And I wonder if there will be a separation of the training and the inference compute as we kind of stabilize a bit more and more and more

• 4:16:18Lex Fridman?

  compute is needed for inference.

• 4:16:19Speaker 02

  That's supposed to be the point of the Grok acquisition. And and that's why part of what Vera Rubin is, where they have a new chip with no high bandwidth memory, which is one of the or very little, which is one of the most expensive pieces. It's designed for pre-fill, which is the part of inference where you essentially do a lot of matrix multiplications and then you only need the memory when you're doing this auto-regressive generation, you have the KV cache swaps. So they have this new GPU that's designed for that specific use case and then the cost of ownership for per flop or whatever is actually way lower. But I I think that Nvidia's fate lies in the diffusion of AI still. Their biggest clients are still these hyperscale

• 4:17:01Speaker 02

  companies, whether it's like Google obviously can make TPUs, Amazon is making Tranium, Microsoft will try to do its own things and like

• 4:17:11Speaker 02

  so long as the pace of AI progress is high, Nvidia's platform is the most flexible and people will want that. But if there's stagnation, then creating bespoke chips, there's more time to do it.

• 4:17:22Lex Fridman?

  It's interesting that Nvidia is is quite active in trying to develop all kinds of different products.

• 4:17:28Speaker 02

  They tried to create areas of commercial value that will use a lot of GPUs.

• 4:17:34Lex Fridman?

  But they keep innovating and and there there's a like, they're doing a lot of incredible research, so.

• 4:17:39Speaker 02

  Everyone says the company is super oriented around Jensen and how.

• 4:17:44Speaker 02

  Operationally plugged in he is, and it sounds so unlike many other big companies that I've heard about. And so long as that's the culture, I think that I will expect them to keep progress happening. And it's like, he's still in the Steve Jobs era of Apple. So long as that is how it operates, I'm pretty optimistic for their situation. Because it's like, it is their top order problem. And I don't know if making these chips for the whole ecosystem is the top goal of all of these other companies. They will do a good job, but it might not be as good of a job.

• 4:18:16Lex Fridman?

  Since you mentioned Jensen, I've been reading a lot about history and about singular figures in history. What do you guys think about the single man, woman view of history? How important are individuals for steering the direction of history in the tech sector? So, you know, what's Nvidia without Jensen? You mentioned Steve Jobs, what's Apple without Steve Jobs? What's XAI without Elon? Or DeepMind without Demis?

• 4:18:44Speaker 02

  People make things earlier and faster. Where scientifically, many great scientists credit to being in the right place at the right time and still making the innovation where eventually someone else will still have the idea. So I think that in that way, Jensen is helping manifest this GPU revolution much faster and much more focused than without having a person there it would do. And this is making the whole AI build out faster. But I do still think that eventually, like something like ChatGPT would have happened and a build out like this would have happened, but it probably would not have been as fast. Or like, I think that's the sort of flavor that is applied.

• 4:19:27Speaker 01

  People, these individual people are people who are placing bets on something. Some get lucky, some don't. But if you don't have these people at the helm, it will be more diffuse. It's almost like investing in an ETF versus individual stocks. Individual stocks might go up, might go down more heavily than an ETF, which is more balanced. It will eventually go up over time, we'll get there. But it's just like, you know, the focus, I think is the thing, passion and focus.

• 4:19:52Lex Fridman?

  Isn't there a real case to be made that without Jensen, there's not a? Uh, reinvigoration of the, the deep learning revolution.

• 4:19:59Speaker 02

  It could have been 20 years later is the thing that I would say.

• 4:20:02Lex Fridman?

  Yeah, yeah, it's a 20.

• 4:20:03Speaker 02

  Or like another AI when, like a deep learning winter could have come.

• 4:20:06Lex Fridman?

  Yeah.

• 4:20:06Speaker 02

  If GPUs weren't around.

• 4:20:08Lex Fridman?

  That could change history completely because you could think of all the other technologies that, that could have come in the meantime and the focus of human civilization could, the Silicon Valley would be captured by different hype.

• 4:20:20Speaker 01

  But I do think it is, uh, I mean, there's certainly an aspect where it was all planned, uh, the GPU trajectory, but on the other hand, it's also a lot of lucky coincidences. For example, or good intuition, like the investment into this, let's say, biophysical simulations or like, I mean, I think it started with video games and then it just happened to be good at linear algebra because video games require a lot of linear algebra. And then you have the biophysical, um, simulations and then, but still I don't think the plan, the master plan was AI. I think there was just, it happened to be Alex Krizhevsky. So someone took these GPUs and like, let's try to train a new network on that and happened to work really well. And I think it only happened because you could purchase those GPUs.

• 4:21:03Speaker 02

  Gaming would have created a demand for faster processors if Nvidia had gone out of business in the early days. That's what I would think. Like, I think that the GPUs would have been different for the Alex, but I think like GPUs would still exist at the time of AlexNet and at the time of the transformer. It was just hard to know if it would be one company as successful or multiple smaller companies with worse chips. But I don't think that's like a hundred year delay. It might be a decade delay.

• 4:21:34Lex Fridman?

  Well, it could be one, two, three, four, five decade delay. I mean, I just can't see Intel or AMD doing what Nvidia did.

• 4:21:41Speaker 02

  I don't think it would be a company that exists.

• 4:21:43Lex Fridman?

  A new company.

• 4:21:44Speaker 02

  I think it would be a different company would rise.

• 4:21:46Speaker 01

  Silicon Graphics or something.

• 4:21:48Speaker 02

  So, yeah, some company that has died would have done it.

• 4:21:52Lex Fridman?

  But it does like, just look, looking at it, it seems like these single figures, these leaders have a huge impact on the trajectory of the world. Obviously incredible teams behind them, but you know, having that kind of.

• 4:22:07Lex Fridman?

  Very singular, almost dogmatic focus is necessary to make progress.

• 4:22:13Speaker 01

  Yeah, I mean, even with GPT, it wouldn't exist if there wasn't a person, Ilya, who pushed for this scaling, right?

• 4:22:19Speaker 02

  Yeah, Dario was also deeply involved in that. You read some of the histories of OpenAI. It almost seems wild thinking about how early these people were like, we need to hook up 10,000 GPUs and take all of OpenAI's compute and train one model. There's a lot of people there that didn't want to do that.

• 4:22:35Lex Fridman?

  Which is an insane thing to believe, to believe scaling before scaling has any indication that it's going to materialize. Again, singular figures. Speaking of which, 100 years from now, this is presumably post singularity, whatever singularity is. When historians look back at our time now, what technological breakthroughs would they really emphasize as the breakthroughs that led to the singularity? So so far, we have touring to today, 80 years.

• 4:23:09Speaker 01

  I think it would still be computing, like the umbrella term computing. Just I don't necessarily think it's even like 100 years, 200 years from now, it would be AI. It would, it could be still, well, be computers, you know, just we are now taking better advantage of computers, but like the fact of computing.

• 4:23:26Lex Fridman?

  It's basically Moore's law kind of discussion. You're not even the details of CUDA and GPUs won't even be remembered. And it won't be all this software turmoil. It'll be just obviously compute.

• 4:23:39Speaker 02

  I would generally agree, but it's like, is the connectivity of the internet and compute able to be merged? Or is the both of them?

• 4:23:50Speaker 01

  I think the internet will probably be related to, yeah, I mean, communication. It could be a phone, internet, satellite, that stuff. Where, yeah, and compute is the more like the scaling aspect of it.

• 4:24:02Lex Fridman?

  It's possible that the internet is completely forgotten. The internet is wrapped into the phone networks, like communication networks. This is just another manifestation of that. And the real breakthrough comes from the just the increased compute is the Moore's law broadly defined.

• 4:24:19Speaker 02

  Well, I think that connection of people is very fundamental to it. So it's like.

• 4:24:25Speaker 02

  You can talk to, you even want to find the best person in the world for something there or somewhere in the world and being able to have that flow of information, the AIs will also rely on this. I think I've been fixating on the like, um, the, when I said the dream was dead about the one central model and the thing that is evolving is like, people have many agents for different tasks. People already start doing this with different clouds for different tasks and it's described as many AGIs in the data center where each one manages and they talk to each other. And like, that is so reliant on networking and free flow of information on top of compute, but like networking, especially with GPUs, is such a part of scaling up compute, like the GPUs and the data centers need to talk to each other.

• 4:25:09Lex Fridman?

  Anything about neural networks will be remembered? Like, do you think there's something very specific and singular to the fact that it's neural networks that's seen as a breakthrough, like a genius, that you're basically replicating in a very crude way, the human mind, the structure, the structure of the human brain, the human mind?

• 4:25:27Speaker 01

  I think without the human mind, we probably wouldn't have neural networks because it just wasn't inspiration for that. But at the other end, I think it's just so, so different. I mean, it's digital versus, you know, biological that I do think it will probably be more like grouped as an algorithm.

• 4:25:44Lex Fridman?

  That's massively paralysable on this particular kind of compute.

• 4:25:48Speaker 01

  Could have been, well, been like genetic computing, like genetic, genetic algorithms just as paralyzed. The thing, it just happens that this is more efficient, works better, you know?

• 4:25:56Lex Fridman?

  And it very well could be that the LLM, you know, the neural networks, the way we architect them now is just a small component of the system that leaves a singularity.

• 4:26:06Speaker 02

  I think if you think of it a hundred years, like, society, I think, can be changed more with more compute and intelligence because of autonomy. But it's like looking at, looking at this, like, what are the things from the industrial revolution that we remember? We remember like the engine is probably the equivalent of the computer in this. But there's a lot of other like, physical transformations that people are aware of, like, like all the, the cotton gin and all these things that these. Machines that are still known, air conditioning, refrigerators, like some of these things from AI will still be known. Like the word transformer could still very well be known.

• 4:26:44Speaker 02

  I would guess that deep learning is definitely still known, but the transformer might be evolved away from in a hundred years of with ASI AI researchers everywhere. But I think deep learning is likely to be a term that is remembered.

• 4:27:01Lex Fridman?

  And I wonder what the air conditioning and the refrigeration of the future is that AI brings. Is there, uh, if we travel forward a hundred years from now, we transport there right now. What do you think is different? How do you think the world looks different? First of all, you think there's humans, you think there's robots everywhere walking around?

• 4:27:19Speaker 01

  I do think specialized robots for sure for certain tasks.

• 4:27:22Lex Fridman?

  Humanoid form?

• 4:27:23Speaker 01

  Um, that I'm maybe half humanoid. We'll see, I, I think for certain things, yes, uh, there will be humanoid robots because it's just amenable for the environment. But like for certain tasks, it might make sense. What's harder to imagine is how we interact with the devices and what humans do with devices will, I mean, I'm pretty sure will probably not be the cell phone, will probably not be the laptop, will it be, you know, implants?

• 4:27:48Lex Fridman?

  I mean, it has to be brain computer interfaces, right? I mean, a hundred years from now, it has to like, given the progress we're seeing now, there has to be, unless there's legitimately

• 4:28:00Lex Fridman?

  complete alteration of how we interact with reality.

• 4:28:05Speaker 01

  On the other hand, if you think of cars, cars are older than a hundred years, right? And it's still the same interface. It's not, we haven't replaced cars with something else. We just made the cars better, but it's still steering wheel, it still wheels, you know?

• 4:28:18Speaker 02

  I think we will still carry around a physical brick of compute because people want some ability to have a private, like you might not engage with it as much as a phone, but having something where you can have private information that is yours as an interface between the rest of the internet. I think is something that people will still exist. It might not look like an iPhone and it might be used a lot less, but I still expect to have people carry things around.

• 4:28:41Lex Fridman?

  Why do you think the smartphone is the embodiment of private? There's a camera on it.

• 4:28:47Speaker 02

  Um, private for you, like encrypted messages, encrypted photos, you know what your life is like.

• 4:28:55Speaker 02

  I guess this is a question on whether how optimistic on brain machine interfaces you are. If is all of that just going to be stored in the cloud in your whole calendar? It's hard to think about processing all the information that we can process visually through brain machine interfaces presenting something like a calendar or something to you like. It's hard to just think about knowing. Without looking, you know, your email inbox like you signal to a computer and then you just know your email inbox. Like, what does that like? Is that something that the human brain can handle being piped into it? Non visually like I don't know exactly how those transformations happen because humans aren't changing in a hundred years.

• 4:29:38Speaker 02

  I think agency and community are things that people actually want.

• 4:29:42Lex Fridman?

  Local community.

• 4:29:43Speaker 02

  So like people you are close to being able to do things with them and being able to ascribe mean like describe meaning to your life and to be able to do things. I think that that is maybe if not in a hundred years, I don't think that human biology is changing away from those on a time scale that we can discuss. And I think that like UBI does not solve agency. I do expect mass wealth and I hope that it is spread so that the average life does look very different in a hundred years. But that's still a lot to happen in a hundred years. If you think about countries that are early in their development process to getting access to computing and internet like to build all the infrastructure and to have

• 4:30:29Speaker 02

  policy that shares one nation's wealth with another is it's I think it's an optimistic view to see all of that happening in a hundred years while they still being while they are still independent entities and not just like absorbed into some international order by force.

• 4:30:45Lex Fridman?

  But there could be just better, more elaborate, more effective social support systems that help alleviate some levels of basic suffering from the world. You know, the transformation of society where a lot of jobs are lost in the short term. I think we have to really remember that each individual job that's lost is a is a human being who's suffering. That's like a when jobs are lost, it scales a real tragedy. You can make all kinds of arguments about economics or it's it's all going to be okay. It's it's good for the GDP. There's going to be new jobs created. Fundamental, the individual level for that human being, that's that's real suffering.

• 4:31:29Lex Fridman?

  That's a real personal sort of tragedy and we have to not forget that as the technologies are being developed. And also, my my hope for all the AI slop we're seeing is that there will be a greater and greater premium for the the fundamentals aspects of the human experience that are like in person, the things that we all like seeing each other, talking together in person.

• 4:31:55Speaker 02

  The next few years are definitely going to be increased value on physical goods and events and even more pressure on slop. So there'll be so there'll keep the slop is only starting. The next few years will be more and more diverse versions of slop.

• 4:32:11Lex Fridman?

  They would be drowning in slop.

• 4:32:12Speaker 02

  So I'm hoping that we society drowns and slop enough to snap out of it and be like, we can't like none. Like, it just doesn't matter. We all can't deal with it. And then like the physical has such a higher premium on it.

• 4:32:26Speaker 01

  Even like classic examples, I honestly think this is true and I think we'll get tired of it. We are already kind of tired of it. Same with, I mean, even art. I don't think art will go away. I mean, you have paintings, physical paintings. There's more value, not just monetary value, but just more value appreciation for something that is the actual painting than a photocopy of that painting. It could be a perfect digital reprint of that. But there is something when you go to a museum and you look at that art and you see that real thing and you think about, okay, a human. I don't, it's like a craft, you have an appreciation for that. And I think the same is true for writing, for talking, for any type of experience where you it will be. I do unfortunately think it will be like a dichotomy, like it will be like a fork where

• 4:33:10Speaker 01

  more some things will be automated, like, you know, there are not as many paintings as there used to be 200 years ago, there are more more photographs, more photocopies. But at the same time, it won't go away. There will be a, you know, value in that. I think that the difference will just be a bit, you know, what's the proportion of that. But personally, I I have a hard time reading things where I obviously see it's obviously AI generated. I'm like, sorry, it might be really good information there, but I have like a certain nah. Not for me.

• 4:33:41Speaker 02

  I think eventually they'll fool you and it'll be on platforms that give ways of verifying or building trust. So you will trust that Lex is not AI generated, having been here. So then you have trust in this channel, but it's harder for new people that don't have that trust.

• 4:33:58Speaker 01

  Well, that that will get interesting because I think fundamentally, I think is a solvable problem by having, you know, trust in certain outlets that they won't do it, but it's all going to be kind of trust based. There will be some systems to authorize. Okay, this is real. This is not real. There will be some tell, tell, tell signs where you can obviously tell this is AI generated and this is not, but they won't, I mean, some will be so good that it's hard to tell and then you have to trust. And, um, well, that that will get interesting and a bit problematic.

• 4:34:27Speaker 02

  The extreme case of this is to watermark all human content.

• 4:34:30Speaker 01

  Hmm.

• 4:34:30Speaker 02

  So all photos that we take on our own have some watermark until they are edited or something like this and software can manage communications with the device manufacturer to maintain like human editing, which is the opposite of the discussion to try to watermark AI images. And then you can make a Google image that has a watermark and use a different Google tool to remove the watermark.

• 4:34:53Speaker 01

  Yeah, it's gonna be a long race.

• 4:34:56Lex Fridman?

  Uh, and we've been mostly focusing on the positive aspects of AI. I mean, there's also the, the, all the capabilities we've been talking about can be used to destabilize human civilization with even just relatively dumb AI applied at scale and then further and further super intelligent AI systems. Of course, there's the sort of doom or take that's important to consider a little bit as we develop these technologies. Um, what gives you hope about the future of human civilization? Everything we've been talking about. Are we going to be okay?

• 4:35:31Speaker 02

  I think we, we will. I'm, I'm definitely a worrier about AI and non-AI things, but humans do tend to find a way. I think that's what humans are built for is to have community and find a way to figure out problems. And that's what has gotten us to this point. And to think that. The AI opportunity in related technologies is really big. And I think that there's big social and political problems to help everybody understand that. And I think that that's what we're staring at a lot of right now. It's like the world is a scary place and AI is a very uncertain thing. And it takes a lot of work that is not necessarily building things.

• 4:36:14Speaker 02

  It's like telling people and understanding people that the people building AI are historically not motivated or wanting to do, but it is something that is probably doable. It just will take longer than people want. We have to go through that long period of like hard distraught AI discussions if we want to have the lasting benefits.

• 4:36:37Lex Fridman?

  Yeah, through that process, I'm especially excited that we get a chance to better understand ourselves. Also, at the individual level as humans and at the civilization level. It answers some of the big mysteries, like what is this whole like consciousness thing going on here? Seems to be truly special, like there's a real miracle in our mind. And AI puts a mirror to ourselves and get to answer some of the big questions about like, what, what is this whole thing going on here?

• 4:37:08Speaker 01

  Well, one thing about that is also what I do think makes us very different from AI. And why I don't worry about AI taking over is, like you said, consciousness, we humans, we decide what we want to do. AI in its current implementation, I can't see it changing. You have to tell it what to do. And so you have still the agency. It doesn't take the agency from you because you have to, you, you, it becomes a tool. You can think of it as a tool. You tell it what to do. It will be more automatic than other previous tools. It's certainly more powerful than a hammer. It can figure things out, but it's still you in, in, in charge, right? So the AI is not in charge, you're in charge, you tell the AI what to do and it's doing it for you.

• 4:37:50Lex Fridman?

  So in the post singularity, post apocalyptic war between humans and machines, you're saying humans are worth fighting for?

• 4:37:59Speaker 01

  A hundred percent. I mean,

• 4:38:02Speaker 01

  this is the movie Terminator. They made it in the eighties essentially. And I do think, well, the only thing I can see going wrong is of course, if things are explicitly programmed to do the thing that is harmful, basically.

• 4:38:16Lex Fridman?

  I think actually in that, in a Terminator type of setup, I think humans win.

• 4:38:22Lex Fridman?

  I think we're too clever.

• 4:38:26Lex Fridman?

  It's hard to explain how we figured it out, but we do. And we'll probably be using local L L Ms, open source L L Ms to help fight the machines.

• 4:38:37Lex Fridman?

  I apologize for the ridiculousness. Like I said, Nathan already knows I've been a big fan of his for a long time. Been a big fan of yours, Sebastian, for a long time. So it's an honor to finally meet you. Thank you for everything you put out into the world. Thank you for the excellent books you're writing. Thank you for teaching us. And thank you for talking today. This was fun.

• 4:38:58Speaker 01

  Thank you for inviting us here and having this human connection.

• 4:39:02Lex Fridman?

  Extremely valuable human connection. Thanks for listening to this conversation with Sebastian Rashka and Nathan Lambert. To support this podcast, please check out our sponsors in the description, where you can also find links to contact me, ask questions, give feedback and so on. And now let me leave you with some words from Albert Einstein.

• 4:39:25Lex Fridman?

  It is not that I'm so smart, but I stay with the questions much longer.

• 4:39:32Lex Fridman?

  Thank you for listening and hope to see you next time.