Ep 89: AI Research Legend’s Honest Assessment of Where We Are

This episode with Lukasz Kaiser, co-author of the seminal "Attention Is All You Need" transformer paper and former researcher at both Google Brain and OpenAI, is a wide-ranging conversation about the fundamental limits of current AI architectures and whether transformers will continue to dominate or eventually give way to something new. Lukasz brings a rare dual perspective: deep belief in how far the current paradigm has taken us (he's an enthusiastic daily Codex user who's seen 10x productivity gains in his own research), while maintaining genuine intellectual humility about whether transformers can truly generalize the way humans do. The episode weaves together questions about data efficiency, the non-verifiable RL frontier, the coding agent revolution, the open vs. closed source gap, and what the next architectural leap might look like: all filtered through the lens of someone who helped build the foundation the entire field is standing on.

• 0:00Speaker 00

  Is reasoning enough to get to generalization? Or is another method needed?

• 0:04Speaker 01

  But does feel like there is something else that possibly could generalize much better.

• 0:08Speaker 00

  Why do you think Anthropic was the first to be like really successful on the coding side?

• 0:12Speaker 01

  Anthropic made this very good decision to focus on coding. OpenAI was like, we're doing chat GPT. Partway Anthropic made this decision was that they just could not compete.

• 0:21Speaker 00

  What's your kind of gut intuition on the gap we'll see between closed source and open source models? And whether that widens or shrinks in the next few years?

• 0:26Speaker 01

  I think it's a fair question, but.

• 0:28Speaker 00

  Lucas Kaiser is one of the authors of the transformer paper and has had amazing roles at both Google and OpenAI. On unsupervised learning, I got to ask him all the top of mind questions of what's happening in AI today. Of course, we had to talk about the transformer and how he thinks about its persistence and whether it will remain the dominant architecture and what shortcomings are. We also got his thoughts on what changed in the fall to really make coding models so much better and why Anthropic was really first to code. We talked about with a future research directions that he's really excited about. And we also hit on a bunch of things around how he thinks the ecosystem will evolve from open versus closed source model to application companies. I think folks will really enjoy this episode with a top researcher whose research really set off a lot in the space. Without further ado, here's Lucas.

• 1:12Speaker 00

  It's a pleasure to have a transformer paper co-author on the podcast. I feel like you've been at the forefront of so many major changes in the AI world. And, you know, our goal is really to get your thoughts on all the questions around the AI frontiers today. So I really appreciate you coming on the podcast.

• 1:27Speaker 01

  Thank you very much. Thank you for having me.

• 1:29Speaker 00

  I can think of kind of no better place to start than generalization, right? It feels like that's the question in the air right now. And I think in November, I heard you say, you know, basically the big, this big question of is reasoning enough to get to generalization? Or is another method needed? And I'm wondering, I guess you said that, you know, maybe six months ago now, which is, you know, dog ears in AI world. So years ago, how is your thinking on that? That question evolved since then?

• 1:54Speaker 01

  If we take the current transformers with reasoning, right? And agents and they have access to a shell and stuff. They can do amazing things, right? It's incredible how far we've gotten. Like, two years ago even, not to mention before Transformers. I would have never believed that, you know, you just take this next word predictor, give it then chain of thought and RL that and tools. And that it will, I now every day spend hours talking to Codex, in my case, or other people. And it works, right? You you talk to it about hard problems at work and it makes sense and it implements things. And so so that's incredible.

• 2:36Speaker 01

  On the other side, and there is this feeling that that it is not quite like us, right? That it's not quite at the edge of what's that? We we all feel that it possibly should be even better, right? That that we can generalize from less data, like somehow make like bigger leaps, get these concepts from way less. I recently have this saying that you you know, people say like Americans will do the right thing after exhausting all other options. And like L L M's, they will learn the concept, they will learn it. But after exhausting all other options, you need this trillion tokens. You need to like learn all the surface level things.

• 3:21Speaker 01

  And only when that doesn't explain something, they will finally learn the concept. That's not how we learn. We just get concepts from like, sometimes we make them up and they're not great. But so it does feel like there is something else that possibly could generalize much better, that could possibly have this like a bit of a different form of understanding, more like long term. Um, but it's a feeling, right? And every time we try to put our thumb on it, it seems to evaporate.

• 3:53Speaker 01

  Or more like, maybe it doesn't even evaporate, but it's like the transformer just catches up, right? It was like, so, so, so, so, so both sides in this time have grown, right? Like transformers have gotten even better, but the case for something else has also gotten even better, I would say. There is, there's now like a number of labs that pursue post transformers and, and people see interesting results. There are certainly interesting things out there. So, so, you know, who wins? I, I still don't know, to be honest.

• 4:29Speaker 01

  I, I think there's good arguments for both sides and, and it will be extremely interesting to see how this goes.

• 4:37Speaker 00

  I think it'll be interesting for our listeners, you know, you, you, obviously, I think at, at, at a talk in NERCON more recently, like alluded to this like whiff in the air, right? That there's something that, that, that's happening in progress that's inspired like these Neo Labs and other folks to spin out and, you know, work on things that are maybe alternatives to the dominant architectures that are being worked on within the labs. What is that feeling? Is it seeing some of these early results? Or what is it like? Or is it just like researcher's intuition? Like maybe making it a little more concrete for, for our listeners?

• 5:03Speaker 01

  I think a lot of this is intuition. And, you know, you need to beware because it's like a lot of this happens in San Francisco at parties and like people talk to each other. So it may be, or like on podcasts. So, so it may be that it's self-fueled to some extent, it's, but, but I think there is a part of it that, that, that's very fundamental. I mean, like Jan Leckun has been saying something like this for years, right? Way before now, which is

• 5:33Speaker 01

  the models we have in, in a long, long history, they were meant, they're called neural networks because they were meant to imitate our brain, but they don't really. They, they are quite different, even if they may have some similarities, right? And if you look at how humans learn, what, what we can do, it, it is quite hard not to say that, that from matchless data, we can do much more than our current models. So, so, so it feels like there is this fundamental ability that, that we as learning machines have that our models currently don't. So, so, so fundamentally, there should be something there, not just a vibe. Now, you can say as

• 6:18Speaker 01

  a counter argument that These models always had a trillion tokens to train on and people never do. So so we just didn't optimize them for training with less. And if you, you know, if you had the same amount of compute but limited data, you can tweak transformers to do much better than than they do today. So so so, you know, it's like some people say, why, why would you, right? We have the data. We we now it's it's a big enterprise. But it does feel, even when we, even when we try to push with as little data as people. Well, it, you know, it's also like, we get a lot of data from visual things, from moving in the world, we take actions.

• 7:02Speaker 01

  So so it's very different kinds of data. It's not truly comparable. That that that's why it's it's hard to like make a very firm scientific statement about it. But but there is this feeling that that we have not exploited all that is there in machine learning. And and obviously, the exciting feeling is that maybe if we find out what's out, it it it could make what we have even more amazing. You know, maybe not, maybe maybe it vanishes when you have that much data. Who knows, right? But but it's definitely extremely interesting to me as a as a researcher. And I think to many people, it's like, I mean, transformers were fascinating, right?

• 7:44Speaker 01

  Their their reasoning is, I mean, it can solve research math problems. I find I'm sure you've heard about the recent Erdos things, of course. As I was a mathematician before in my life. So this is extremely exciting. I never thought a computer in this timeframe will, you know, talk to me about mathematics at a high level as a real researcher. That exists now, and and this is insane. But then as an M L researcher, I'm like, OK, but we haven't really figured out this learning. There is this feeling, right, that that it learns certainly, but it needs so much data. It needs so much compute. This this feels like we're not quite there yet. Now. Is this only a feeling?

• 8:29Speaker 01

  Is this a vibe? It seems to be reality to some extent, right? But we'll need to, we'll need to see.

• 8:37Speaker 00

  The research appeal of figuring that out makes a ton of sense. And I think other folks might look at it and be like, well, you know, so what if it's not like people, right? Like, we have the data, we have a method that works. You know, obviously, there's going to be some areas where there is limited data, like, you know, like, you know, drug development and other things where learning from more limited data would be really helpful. But so many problems that exist in the world actually aren't that data constrained, right? Sometimes I feel like these sides almost like talk past each other, right? Like, people at the labs will roll their eyes at Jan LeCoon or something like that.

• 9:05Speaker 01

  I think this is fair to say, but on the other hand, given how quickly and, you know, with the whole investment in AI, the problems that are not data limited get solved very rapidly. So very soon, all bottlenecks that remain will be quite data limited or already are becoming. And in particular, it does feel that to work well in the physical world, you do need to solve some part of it at least because

• 9:33Speaker 01

  the physical world, if you, you know, you train on like one robot hardware, it's not, doesn't quite scale data the way that the virtual or text worlds or internet worlds do. So, so in the physical world, this is a sizable chunk of.

• 9:50Speaker 00

  Yeah. So people are certainly trying, right? With simulation data and with egocentric video data or with super sources.

• 9:56Speaker 01

  Yeah, I mean, you know, I'm a huge fan of Waymo's, right? They, I have always this joke, like, people say, where are my self-driving cars? Well, I drive them, they're here. But then they just canceled the highway driving, right? Because they couldn't deal with some construction zone again. And it feels almost like, you know, they have had this construction zone things for years in this. I'm sure just like millions of miles in simulation and quite some in real driving. And it still can't generalize to the construction zone on a highway. Like, this feels, feels just off, right? I mean, I don't know what exactly didn't work there. But I certainly know, you know, no teenager has this problem.

• 10:39Speaker 00

  Yeah.

• 10:39Speaker 01

  Or no human, right? We have many other problems, right? But but not that we can drive in a construction zone in the city, but not on the highway. That just construction zone is a construction.

• 10:52Speaker 00

  And do you think that like some of this stuff, you know, will be, you know, or could be solved within the, you know, within the transformers and like, I guess, like, what would you, what are you kind of looking for? I guess, you know, in the next few years to to get a better answer to this question.

• 11:04Speaker 01

  The exciting part in ML research is that it is so broad, right? You never know whether you need to tweak the architecture or do you need to tweak the data? Or do you need to tweak the loss? Or do you need to tweak the optimization process?

• 11:21Speaker 01

  And there are arguments for all. And on top of that, it might turn out that you need to tweak all of them to some extent, right? It's like transformer is great, but it's also great with the next word prediction loss, right? Or you can make it work with RL, but but you need the chains of thought. Or it's like these puzzles only work when you click them together. So so it is possible that the like, if there is a new thing that there might need to be tweaks to everything. But it's also possible that that parts of transformers will survive. For example, probably attention will be somewhere there, right?

• 12:01Speaker 01

  But maybe you need other things to it.

• 12:04Speaker 00

  Yeah.

• 12:04Speaker 01

  Like, maybe, you know, I I I've started my machine learning life with RNNs and and they certainly hold recurrence deep in my heart. I like it as construct it feels. And reasoning kind of brought it back because every new token you produce, it's the same way it's that that we currently produce it. So so in some sense, it's it's back, but but but it does feel that this RL is like very sparse losses and you do so much. But it works, right? It's it's a and every time we try to do recurrence in like other ways, it somehow does not seem to click yet. Yeah, but but.

• 12:46Speaker 01

  But there is always the question, how hard have we tried, right? It's a, I, I, I don't know if, if you or the audience knows, there are models like TRM and HRM. It's like very small models that turned out to do very well on problems like Sudoku, but also RKGI. And so they're a little bit toy tests, but, but they do quite well. I think a lot of the post transformer architectures are trying to merge this with L L Ms. It's interesting, certainly, right? It's like the pure transformer can't do so well on it. But you add some recurrence, you, you add some bit of architectural tweaks, maybe a little different loss and it does really well. So even on the small scale, you can do a lot.

• 13:29Speaker 01

  But then will it, you know, will it, will it generalize to the language and give you the things you want? It's, I, well, it will be very interesting to see. And, and luckily there is like a number of labs that, and that are trying. And the other thing though is like, this year we have the agents. And to me, this is like a totally, it's the biggest change in the way I work as a M L researcher. And I would say the last fifth, 20 years, probably.

• 14:00Speaker 00

  I don't know if you try to quantify it, but like, how much more productive do you think it makes you?

• 14:04Speaker 01

  Oh, I, I, I can, I can fairly well quantify it because I, I tried recently just on the, on the private machine to reproduce a, a bunch of papers. Like old papers that I were, oh, was always quite interested in. Even some of my papers that I lost code for. And at least one of them I tried to reproduce before, and I knew it took me about three weeks to get to a runnable state. And with codecs, I could get there in two days. So, so it's about, let's say a week to a day, that, that's already, whether it's a 10x or a 5x, you know, maybe I could have been faster back then. But, uh, but it's certainly, it, it changes your rhythm because.

• 14:52Speaker 01

  You can just take on things. It's also, you know, I can just start three things in parallel and let it go. While when I was doing, I would be just do one thing. You're right. So it both makes it faster and makes it more parallel, which is, but, but, but like, I mean, when I do like private things, not, not in a production repo, I basically stopped looking at the code. And, and a friend asked me like, do you think you're less sharp now? And I gave it some thought and I think it's actually to the contrary. Because due to the fact that I don't look at every class name, every small function,

• 15:37Speaker 01

  but I still know that these agents can go off the rails. Like once it, it, it, this favorite, it runs something and there were some aux losses and it like just added, it just thought it should have another auxiliary loss and it was totally off the charts and out of place. So, so you need to have like a full control in your head of what exactly is it doing? What is the loss? What is the, but you don't need to have control of like, what's the name of the class? What's, you know, what are the exact words and the function? It's, it's quite impressive that you can trust the agents to be, you know, trustful and like that they're really implementing what you think. But I mean, sometimes we check and, and the R.

• 16:21Speaker 01

  But since you need to have a full control in your head of what's actually running machine learning wise, what are the losses? What are the batches? What? Then I feel it gives me actually more mental control over what I'm doing than it was before. Because before I would implement it, but you know, sometimes in this time before running it, I would have to like forget a little bit about what the big picture was exactly and focus on the little things, debug, then go back to the big picture. By that time, maybe I forgot some detail and then I would remember it when it was wrong. Now it's like, it's this beautiful thing where you can just be in this flow. Like you just think machine learning wise, what's supposed to happen?

• 17:06Speaker 01

  You tell it, verify it and it's happening.

• 17:10Speaker 00

  Yeah.

• 17:10Speaker 01

  So it it's not just about the time saved. It it just makes the work so nice. It's a mild psychosis, I guess, among our researchers. We just can't stop.

• 17:21Speaker 00

  OpenAI very publicly said, hey, our our goal is kind of, I think, a research level intern by by November of this year. You know, as as someone who plays around with Codex all the time in your research, does it feel like you're you're close to that? Or how are you feeling about that milestone?

• 17:34Speaker 01

  It does feel like close to an intern, but you need to be very carefully checking. Like, as I said, it can just add you a loss that you did not ask for because it seems reasonable to it. Um, I don't know if interns do that. Maybe sometimes, I guess, sometimes when they're creative. We're right, but like, I I try sometimes, you know, it's like I I will just let it go for the night and I give it the goal. You know, make a better model for for this lower perplexity. Um, that never works. It it will just start doing some very trivial tweaks that are not really interesting or useful. So so it's certainly not at the level of a researcher.

• 18:18Speaker 00

  Yeah. Dude, what's the path forward to make it better there?

• 18:20Speaker 01

  It goes back to to to our question, right? For a long time, I worked on long context in in machine learning, even before transformers, you could say, on on memory and so on. And and then we worked on it with transformers and and, you know, the context got longer. We got like a million tokens, which is huge, given what attention does.

• 18:43Speaker 01

  But now with agents, it really does feel like grep or rip grep. Is, you know, our solution to long context is let's write a bunch of stuff in files and give it access to grep. So it can find and, you know, tell it to write index files and it's like a little library.

• 19:05Speaker 01

  And of course, to me, as a NEMO researcher, you told me five years ago, that's not a solution, that's a hack, right? But, but, but you know, machine learning, I think, is a hack of sorts. So dropout is a hack, right? We don't, we don't judge, right? We, we take what works and it works amazingly. It, it works. And, and you add a little bit of RL, like, for example, compaction. If, if there is one reason I like Codex over Cloud Code, it is compaction. It can go on with the thread and it's good at compacting. Why is it good at compacting? There is, I don't think there's like very mysterious, right? People prompted it well and then put some RL on it to just make, make, make.

• 19:49Speaker 01

  And if you told this to me like some years ago, that, that the long context, well, you just RL a bit that it can use tools and find stuff in files and then summarize good enough to keep the, I would tell like, okay, that's a bandaid. That's not a, like, it doesn't feel like this deep thing. But, you know, we, we don't judge solutions by how they look. We judge them by how they work and it works really well. So, so to that point of like, can it become a researcher? Well, you know, some people would say, well, maybe no, maybe you'll need this new architecture. Maybe you'll need a post transformer thing that has concepts that are bigger and, and follows goals. And it's a fair argument, right?

• 20:34Speaker 01

  Currently, it feels like it cancels. But then there are other people who say, well, well, well, you'll have your conversations with Codex for a month and then you're going to prompt it to go over them and find meta-patterns, write this to some files and, and just think how it can use them. And, you know, maybe if you have some data over a thousand people and do some RL on it, it will start behaving like a researcher. It's, in some ways, this is how researchers learn, right? We look how other people do research, we do a bit of our trials, see what works better.

• 21:07Speaker 00

  Why doesn't that work today? Like, I'm sure people have tried that.

• 21:10Speaker 01

  Oh, hey, no, I, I don't think people have tried yet very hard.

• 21:16Speaker 01

  It, you know, it's like some people do some prompts and they work for them.

• 21:22Speaker 01

  It is important to, I mean, to me, the Codex era started like this year or Christmas, right? Yeah. It's a, I mean, Codex existed before and we used it and Claude Code existed and we also used parts of it, but. No, I think everyone felt it Christmas. But it, it seems like only the newer, but it's not just the models, it's also the harness and the some tweaks.

• 21:45Speaker 01

  So, you know, it's, it's barely half a year.

• 21:49Speaker 01

  And, and there's still many people, if you go a bit outside of the, you know, our SFAI bubble,

• 21:57Speaker 01

  that totally don't get it. They're like, you know, you're a little psychotic, but why, right? And, and I think it's a fair question, but,

• 22:06Speaker 01

  but it does, it, it, it started working very recently. We don't truly understand. Like, it was not like a big pre-training that changed it that much, even though big pre-training scheme too, but. It, it, you know, when, when we went from like RNNs to transformers, it was very easy to attribute the change to, to this. Well, now I feel, and then there was reasoning, which clearly is important in RL and, and. But the change last winter, last Christmas, it's a little hard to pin down. I mean, harness changed and the little post-training changed and, and then new pre-trained models come, which of course make things better, but.

• 22:49Speaker 01

  But it felt like a big jump, which is not that easy to pin down what did it. So, so, so it's a little bit messy, right? We improve everything all the time. But then because it works and, and it feels so important, then there is also the necessity to just, you know, bring it to people, make it work everywhere, promote it, there, there is this competition going on. So, so. I think in all of this, people did not have truly the time yet to, to think, like, how do you really do this meta level? And like, people are starting, but, but also, it also feels you like,

• 23:35Speaker 01

  because the meta level is something like you research for a week and then you get some patterns and, and start applying them.

• 23:44Speaker 01

  That feels like a relink this needs to take weeks. Like our current reinforcement learning methods, unlucky need to run basically all rollouts on this. And if your rollouts are like weeks long, then then your training starts to be months long and that all becomes a little impractical, which maybe is an argument, you know, that the post transformer, like that the human side has something to learn because clearly, you know, humans can do research over years. And they do this once in their life, right? Or twice, right? Some mathematicians spend 20 years on one problem. That's their magnum opus. And that's it.

• 24:28Speaker 01

  So they did not have, you know, 200 problems 20 years long before to learn from. And somehow they manage. How does this work? That it's a fascinating question, clearly with some relevance to this. We haven't figured it out, but on the other hand, you know, we now we'll gather, since a lot of people work with it, we'll gather a lot of the data on the like weeks to months long humans. Someone will run this around. And it may just turn out that it gets you.

• 25:04Speaker 00

  Yeah.

• 25:05Speaker 01

  Further, so.

• 25:05Speaker 00

  It's such an interesting point because basically, you know, like you're saying, as folks were scaling pre-training or as folks were scaling the original sort of reasoning models, it was kind of, it was straightforward or at least made sense, like the vector you were scaling on. And then this kind of big advance we've had in Kodak's and Claude code over Christmas. If you don't actually know what the source of that is, or you're not fully crystal clear on it, it's very hard to then determine what you should be pushing on to continue to improve these capabilities.

• 25:31Speaker 01

  Yes, it's a little, it's a little confusing. But I mean, the fact that I don't know doesn't mean nobody knows. I think some people have like stronger opinions on what exactly pushed it through. But I don't think it's that clear at this point.

• 25:50Speaker 01

  Which, yeah, it's just, I mean, it's been improving for a while, but something happened.

• 25:59Speaker 00

  Yeah.

• 26:00Speaker 01

  Because, yeah, it did not feel possible to do this, and now it does.

• 26:06Speaker 00

  On this kind of current scaling regime on the RL side, I think one question a lot of folks have is, we've seen, you know, obviously, you know, tons of coding improvement and math and these kind of like verifiable domains. And I feel like the two big questions around RL continue to be like, you know, how well is this going to work on the non-verifiable side? And then also, you know, the extent to which we'll get generalization and not having to keep, you know, do tons of data in each space. Maybe we'll take them one at a time. But starting with the first, you know, how do you think about the problems that need to be solved on the non-verifiable domain side? And, you know, any inklings as to which spaces might be next beyond code and math?

• 26:41Speaker 01

  I do think there is, there's been a fair progress on the non-verifiable side. If you look, for example, at like, things like Harvey and law, or things in medicine, they're not verifiable, but there's a lot of parts of them that are verifiable, right? So there's been good progress on that. And I think, I mean, GDP val is one benchmark that, and Samson's benchmarks, things like that, too. And I do think there is really good progress, and there's really good incentives to make progress in these domains. I'm not sure if it's fully fair to call them non-verifiable.

• 27:22Speaker 00

  They're certainly not as perfectly set up as coding and math, right?

• 27:25Speaker 01

  They're not coding and math, right? But math, I think people overstate how verifiable math is. Like, coding is fairly verifiable, in the sense like, programming competitions are verifiable.

• 27:37Speaker 00

  Yeah.

• 27:38Speaker 01

  Once you go to like, front-end coding and stuff, it's also not that verifiable. But still, the math, the proofs are not that easy or clean. I mean, you can do lean, but Most of the math, at least from GPTs, it's not formalized, so it's not that verifiable. So it's a spectrum, and then things get less and less verifiable. I had this pet project of translating poetry into Polish, which seems fairly not verifiable. But then you run these models as verifiers, and you know, they get a fair bit of stuff. They get like rhyme and things, and they can get cultural references. So it turns out, once you read how people have verified things before,

• 28:21Speaker 01

  you can get to some level of verifiability. But then, I mean, I think what this poetry thinks that was also what it was meant to show is, you can verify a lot of things and then still have kind of no taste. And I mean, since it's not verifiable, it's not so easy. If it were easy to describe in words. Yeah, then it would be verifiable. But it doesn't mean it isn't there, right? You read this, and there is something in your brain that reinforces this idea that there is something there missing. You know, we have driven ourselves into this hole, basically on purpose. Because what is reinforcement learning?

• 29:06Speaker 01

  It tells you whenever you have a teacher, a validator, someone telling you this is good, this is bad, I can train against it, and I'm gonna get good. And that's what the models do. So, you know, whenever I will come and say, "Look, I don't think this does this very tasteful in something." Then someone will tell, "Okay, show me." And then it will nail it. And I think some people even run runs that go basically against, like, for image generation. You can ask, "Is this beautiful or not?" Okay, not verifiable, but you just get a bunch of people who during training click, "This is beautiful." Yes. And low and behold, the images start to be more beautiful. So the verifiability thing is very weak, right?

• 29:51Speaker 01

  It's just a very sparse signal when you ask, you can ask people, "Is this nice? Is this not nice?" Which then, you know, why do I think this is not very tasteful, right? Clearly, some of my experiences and some way that I have processed it make me say this statement.

• 30:13Speaker 01

  Now, so, so why does the model not say it? And, and there are two possibilities. One is that it hasn't seen enough experience that, that would make it do it. And the other is that it's not processing it in the right way. So I believe in both, actually. But even with the way it is processing, if you just put more experience, you ask a thousand people to, to, to tell it, then it gets better. So, so there is every hole you have, you can kind of plug by, by hammering on it. But it would be so nice if you didn't have to, right? It's like, because also every hole you plug stops being a bottleneck and then the bottlenecks

• 30:55Speaker 01

  that emerge are again the holes that you have not plugged and, and, and so, so, so we're in this interesting circle. But if we had this method, this brain-like method that, that, that would just not have so many holes that need plugging, wouldn't this be great?

• 31:16Speaker 00

  Does that kind of imply that like any, you know, problem area that, you know, someone does focus on under the current architectures like can be figured out? It's just that, to your point, it, it probably is, is far more, you know, requires curated data and far more manual than, you know, a potentially more beautiful way of, of, of doing things down the line. But there's not like a set of problems or a set of domains that you're like God under the current RL methods, you know, that would be too hard for, for, for the models.

• 31:42Speaker 01

  That it does not feel so, but

• 31:45Speaker 00

  Yeah.

• 31:45Speaker 01

  You do need to take economics into account, right? I mean, currently to, to make these models work really well, you need to start from a fairly strong model, which is fairly big and expensive. On top of this, it's usually closed, so

• 31:59Speaker 00

  Yeah.

• 31:59Speaker 01

  You can't really, you do it. I mean, there is the RL fine-tuning API, which I quite like from OpenAI and some similar ones, but you don't truly have like full access to it. So even with the API, it can be a little hard. And even on top of that, the investment you'll need to make into the data and things, it's, it's substantial, right? You, you couldn't do this in the US. You'd need a company, you'd need some contracts, you'd need, which, you know, if it's important enough, it's a fair method. But then, right?

• 32:31Speaker 00

  Yeah.

• 32:31Speaker 01

  Wouldn't it be great if you could just talk to the model and it would work on itself?

• 32:38Speaker 00

  Does it feel like there's any signs of, you know, general capability improvement as you do? You could imagine a world where it's like, okay, we'll, we'll start with code and then we'll do math and then we'll, you know, do this for legal and healthcare and you could, you could tackle each of these one by one, even if you're not getting any sort of generalization across. Or, you know, ideally, I think that maybe the hope would be at some level of, of having done reinforcement learning on a bunch of different domains, maybe somewhere to pre-training it at some level, like generalization emerges or something.

• 33:04Speaker 01

  But I think generalization emerges in reinforcement learning.

• 33:07Speaker 00

  So you think already the models get better across the board?

• 33:09Speaker 01

  Oh, yes, they, they, they certainly do. Like, if you look at like, I think law is simply not in the RL pipeline at all. And, and, and you talk to Harvey or someone and they say like, it, it either emerges or they need like a little train, like just a few touches on top of it and it suddenly catches it. So, so, so there is definitely generalization.

• 33:34Speaker 01

  But, but it's, the generalization doesn't seem just to go as far as, or like, yeah, it, it just like works in not the ways that we would hope.

• 33:48Speaker 00

  Yeah.

• 33:48Speaker 01

  So sometimes like it doesn't generalize even from math to other areas of math.

• 33:54Speaker 01

  There, there is, if you look at the, even the IMO, right? Like now it seems like so far away than models.

• 34:00Speaker 00

  Yeah.

• 34:02Speaker 01

  But it, it would have like some types of exercises. For, for a long time it was geometry that it just couldn't crack. It would do very hard, it would solve very hard problems in other domains. And then geometry, you were like, oh, okay, it has no spatial understanding. And then it just saw more data and started cracking it and, but not spatial understanding data or physical, just, just small geometry problems.

• 34:26Speaker 01

  But, but like it, it has this jaggedness, right? It will generalize from here to here, but not to something that seems very close. But somehow in this representation of these chains of thought is, is not like, it's close to me, but it's not close to the model, right? So it's not like it's not generalizing. It's generalizing, but in its weird alien way. And and that just doesn't cover some ways that I can generalize and and and you know, it's possible that with more data, it will just cover more of this space.

• 35:02Speaker 01

  But I also understand people who say like, you know, when it's like that, it's very hard to trust, like to commit to it like that because you know, there may be this spike that it just hasn't gotten. So you need to be on the lookout for for problems and and as I use it as I'm a researcher, I think it keeps me very honest because I think why need to be it keeps me sharp. So maybe this is good in this way, but it's not good in a capabilities way because you just hope that doesn't have these sharp edges, right? For now, it does.

• 35:38Speaker 00

  You mentioned some of the application companies, obviously, that benefit from these models getting better. And I think there's like this big question of if you're an application company right now, should you be working super closely with with one of the labs and sharing kind of all these evals and like understanding you have of the domain or, you know, is that like actually, you know, are you better off kind of building almost your own model based on information versus, you know, sharing it back? I'm curious how you think about like the room for, you know, applications on top of, you know, the core models.

• 36:06Speaker 01

  For now, what is certainly true is that the bigger and better your pre-trained model is, the less of these sharp edges you get and generally the easier all of your life becomes, right? Whether you do RL on it or fine tuning on whatever bigger model, things just get easier. It's insane how this has continued to be the case. We have, I don't know, you remember like a year ago, two years, people were saying, oh, LLMs are that, SLMs are the future, small models. And we have amazing small models, like the GEMAs that recently were like few billion.

• 36:43Speaker 01

  Remember GPT-3, people said, oh, you go, you do no zero-shot learning under 100 billion. No, yeah, you have, we have like, you know, three B models that that are. So that's all amazing, but if you really want to solve big problems easily just to your data and context, there just doesn't seem to be anything like a really, you know, elephant model. But they're of course expensive and hard to use and even harder to train.

• 37:11Speaker 00

  One thing I think would be interesting for our listeners is, I think something that's maybe less obvious to folks outside the cutting edge is just what's enabled by new generations of hardware, right? And so I'm wondering if you'd speak a little bit about to, I mean, you know, obviously, it seems like for certain things as we, you know, as we waited for like Blackwall chips to come online, it's like, hey, they came online and like the models got better. And it's always hard to tell like, how much that is just, yes, you could now do lots of things on the hardware you couldn't do before, how much of that was just timing correlation. But maybe just speak to that. And I think it's kind of relevant to this conversation around like, are these architectures just going to get better as the hardware is getting better?

• 37:47Speaker 01

  I mean, hardware gets better and hardware, you know, it's, it's easy. It's flops and memory access, right? So you need memory fast enough to feed the flops. But it's a very simple, you can call it performance. And I recently, so I got a, I got a personal computer, I got one for myself, and I bought a 5090 GPU. And it felt like, oh, you know, it's one GPU and like you're under your desk, what can this do? So I can have a little bit of some tests and it's just insane to think. So the 5090, it's about 200 teraflops.

• 38:30Speaker 01

  I mean, it says 400, but some are turned off on BF16. So the GPUs we researched Transformer on, they had nine teraflops. And we had eight GPU machines. So in absolute scaling, you could, you could say, you know, be like 70. Um, the seventy, eighty teraflops for, for real on a machine. Um, so now I have under my desk, something that's like five of these machines in one GPU, which is much more convenient than, but I think we used like around ten or something. So, so you could do all of transformer research on this few thousand dollars GPU under your desk

• 39:12Speaker 01

  that, that, you know, you could have in your kitchen, like it's, it's, it's, it's a normal little tower. And, oh, okay, it, it, it's a few years. It's not even a decade though. So, so it's quite amazing what, what they can do. And, and now we run everything in BF sixteen, but of course you can go lower even in precision, especially with MOEs, then you pack more. Um, in inference, this is amazing. So, so our ability to run these models has dramatically increased. And it increases the, the things you can research, right? You, you can now run so many interest, interesting ways.

• 39:58Speaker 01

  Now, it does give you the ability to just, oh, and also there's more GPUs, right? And the world, like the big labs are building out. So, so you can train huge models on a huge number of very fast GPUs. And then Nvidia has kept the pace and then TPU's at Google have kept the pace. They're, they're really speeding up very quickly.

• 40:21Speaker 01

  Um, and their numbers are growing and then it's a very parallelizable process. So we can now train much bigger models much faster. That is amazing. I do still think that the even more interesting things is that we can do more research. And like, like it, it, it's interesting. I, I remember when, when, when I was joining Google, people were talking about like, how much flops do you need for the, to do something like the brain, right? Yeah. And it's a very vague question because to really simulate a brain in the brain, it's maybe impossible, maybe.

• 40:56Speaker 01

  Still very much, but but people for decades have been doing these estimates and they always felt like somewhere between one and a hundred petaflops. And I remember back then, we were like, okay, so this is gonna take like a few decades for us to get there. Now you can buy a single GPU. So so that is quite insane. You have like this one thing you can, and then then of course you can on the cloud get machines with with many of them.

• 41:23Speaker 01

  So potentially you can, you know, run like a year of worth of human

• 41:30Speaker 01

  processing in a day. At the at the cost, right? But it's not a cost of millions, right? It's a cost of hundreds, two thousands of dollars. If you believe you can maybe figure out this algorithm, like, I mean, it's questionable whether we have the data that people have. Some people are trying to do recordings of of kids, right? There's a question how, there's a lot of questions, but we're getting to this level where someone at a university will be basically able to run like a childhood. You know, if you have an idea for how the brain learns, you'll be able to run it in like a few days to the whole like 10 years of learning of of a human being. And see if it works or doesn't, maybe if you know how to evaluate it.

• 42:14Speaker 01

  I I think this this is even more powerful than the fact that we can build these huge models, which is also powerful because they will help you implement this all and right. And and we're getting this loop where I always felt limited with R N Ns, for example, with because they're very sequential. So if you just run them like in Torch, they're very, very slow, right? But you can write a special CUDA kernel that makes them very fast, but writing CUDA kernels is awful, right? You really don't want to do this, except when you can have a unit test that it does exactly the same thing as your slow thing and an agent that writes them for you.

• 42:52Speaker 01

  And they're not yet amazing at it, but they're already do it. And you know, bigger model will probably be so good that you'll be you'll just say, you know, use this hardware as best as it can be and come a few hours later and here it is. So, so the bottlenecks that were like, because the hardware did not fit your idea. Well, the hardware is still the way it is, right? It can't do anything you want. It still needs to be parallel. But it can do much more than it could do before, because you can write, like, just ask agents to write kernel support.

• 43:26Speaker 00

  Yeah, it's so interesting because some people will say, God, you know, without the scale of compute that that exists at if only a few places, you know, it's it's so hard to do. Maybe you can do basic research, but like ultimately the the rubber hits the road on seeing what are these techniques scale, right? And you need to be in a lab to to kind of experience that. But it's awesome to hear your kind of bullishness around the opportunity for academia and hobbyists and folks that are just messing around with with single GPUs to be able to to contribute here.

• 43:53Speaker 01

  Well, I I think especially if you believe that there are some radical changes that you should do.

• 43:59Speaker 00

  Do you think it's more likely than not that that's the case? Like, I guess.

• 44:02Speaker 01

  It's it depends on the day.

• 44:06Speaker 01

  On my, on my, oh, positive days, I do.

• 44:10Speaker 01

  It, it, you know, research has always brought us beautiful things. There is no reason to think it won't. But then the techniques we have also seem to work so well that that that it just feels mind blowing to. It would be a big mistake to not push on those two. But luckily, there's, you know, there's enough labs. I I feel the whole thrill of being an academic, I was in academia before I I joined the labs, is that you can go wild with your ideas, right? You you can't go, you can't scale up that much, but but on the lower scale, which now is not that low, you can go really wild. You you can try,

• 44:53Speaker 01

  you know, beautiful ideas that that are totally out of the current paradigm. And you should, that that that's, you know, that that's the fun of being a researcher. And then, well, you know, not not many won't work. Some will work in the small scale and not scale up. But

• 45:11Speaker 01

  I mean, at the scale the current like eight GPUs machinery.

• 45:17Speaker 01

  I mean, sure, there will always be ideas that, you know, work up to a certain scale and don't don't work further. But I think you're at much higher level now than than than it was like five years ago. Because five years ago, it was like really like amnesty tiny things. There was a lot of tweaks that were just really small scale tweaks. Now you're getting, even on on one machine, you're getting to a scale where it's not tweaks anymore. It's it's it's it's it's like, you know, like, like, I privately use NanoChat from from Andrew.

• 45:49Speaker 00

  Yes.

• 45:50Speaker 01

  Yeah, it's a GPT two level model that that you get in a few hours on on one on one box, right? These boxes have got a bit more expensive these days, luckily, but but but they'll, you know, when new generation of GPUs will come, the older will get cheaper. It's it's a, yeah, it's just quite astounding what you can actually do. And yes, not all of this will scale, but but the fun you can have on the way is.

• 46:21Speaker 00

  Yeah, totally. I guess one more research frontier, you know, I'd love to get your take on before we shift gears, is, you know, multimodal models. And I think you, on a previous podcast, you said, you know, we haven't made a ton of progress there. Do you still feel that's the case? And what's your kind of current say to the union on on multimodal world?

• 46:38Speaker 01

  So people are certainly making progress. Maybe this goes a little bit towards JEPHA, but like, the way we do multimodal and transformers, or even with diffusion models, it's like, in the end, you predict like every pixel of these things around. And if you think of me being here in the environment, and like, I think humans like sense an amazing amount of information every second or less than, you know, we might, but we we can't act like, our neurons are slow, right? They have like hundreds of millisecond processing, but we get all these senses everywhere all the time. And we somehow manage to learn from this insane stream.

• 47:25Speaker 01

  Without maybe, you know, like predicting every pixel out or aggressively like the it's it's both like way more parallel and and and like much larger. So I feel like the models we have, they have not truly done justice to to to this yet. Maybe it needs new research, maybe. I mean, it's but they're also very similar like the I think Thinking Machines has recently had this like multi-stream transformers and it feels so easy, right? I mean, in a transformer, you pay attention to the previous tokens. You could have a bunch of streams that that do this, right? That feels like an easy tweak to the architecture, but but you know, maybe it's an easy tweak, but but just an amazing tweak because because

• 48:10Speaker 01

  I I always, when I work with like codecs and and you know, I just forget something, I say it, but then it's executing some bash commands, so it needs to wait for my thing to steer it. And it takes three minutes and I'm like, this is just so not interact like it should just and then you can have the side thing and like there's a bunch of hacks again that kind of make it feel better. But it feels like, of course, everything happens everywhere all at once and for us, right? We see, hear, talk all at the same time. Um, that should be how our models behave. No, now now that there is a bigger lab putting pressure on that, maybe it will it will come. Uh, but it, yeah, it it it does feel like

• 48:55Speaker 01

  like we do multimodal without it, right? With with without all of these like truly architectural changes to be to be parallel and and absorb like, you know, transformer can't currently at at the speed it does absorb a high resolution image every my millisecond, right? It it just because it splits them and this so sequential in this that that it just doesn't work. That feels somehow wrong, right? It's like we shouldn't be putting these tiny patches there, which should like just go in, be processed somehow. Um, so I don't think we have like on this deeper level gotten there yet, but but on the other hand, I feel like a lot of people are working on it. So so

• 49:38Speaker 01

  yeah, then for coding, I mean, does it matter all that much harder to say? Totally. Well, I'm sure it will come.

• 49:47Speaker 00

  I'd love to kind of switch gears and maybe just talk a little bit about your time at OpenAI and your kind of journey there, because obviously it's been quite the eventful past years. And, you know, maybe, yeah, there's a few moments everyone kind of thinks about and so I'm curious to get your perspective. But maybe just like on the OpenAI side, the company's had some very public like moments. And I'm wondering, like, what were some of the difficult decisions that really like defined the company, I guess, in your in your time there?

• 50:12Speaker 01

  So I, you know, I wasn't there for the earliest things. I think for my time there, there was this big question at some point whether to pivot to reasoning.

• 50:27Speaker 01

  And I feel it was very brave of the company and the leadership and all of us to actually take this plunge and say, yes, reasoning will be as important as pre-training. Our most will be reasoning models, they will be launched. And, you know, at the beginning, it was like the reasoning models were not that chatty. Somehow, personality was harder. They were slow and they still are to some extent. And it was like, you know, should you ever do it? Like, maybe people just prefer chat models.

• 51:02Speaker 00

  Yeah.

• 51:03Speaker 01

  But OpenAI was, yeah, very good at taking this hard bet and saying, yes, we're gonna launch it. We're gonna go this way. Well, try to figure out how to manage. There were two lines of models at the same time that's obviously awful, right? You want to unify this. The unification took a lot of time because everything is moving. It's a very hard decision. But no, we wouldn't have possibly all of these amazing things we have if it didn't push on it. And it feels like, you know, even some bigger labs still have trouble catching up to the RL quality. So there is some win that you get when you commit to things.

• 51:47Speaker 01

  And, you know, I wonder, these days, you know, OpenAI has since then grown probably like 20 times or something like this. Become a much bigger company. All of the labs have, I mean, Google was big even before, but everyone now like Anthropic has become big. Having been at Google before for a long time, I think it's much harder for a big company to take wild bets like that, right? Because you have so much more to lose because you have processes. Like, it's just harder. I just hope OpenAI retains this ability and the other labs too, because, yeah, like the current techniques are amazing, right? They get us very far, but if there were, you know, sparks of post transformer world,

• 52:32Speaker 01

  would these labs be able to jump on it? Or would they be on the more conservative side now?

• 52:39Speaker 00

  It feels like with reasoning, there were some like early sparks, but obviously not a ton of data. And then, you know, that it was kind of a, almost a, I've heard it articulated, it's like a religious belief that this is just gonna work if we double down on it.

• 52:49Speaker 01

  And, and we don't have like the successor yet, or at least I don't know about it. But, but with the hope that it will appear, will you need a new lab to push on it? Or will, I mean, I think if anything, OpenAI is good at, you know, wild bets.

• 53:10Speaker 00

  It's obviously interesting to see this whole, this whole trend of Neo Labs, right? And folks like Jerry Toric spinning out and saying that like it's, it's almost, you know, easier to do this work outside of a large lab, right? And make one kind of strong convicted bet.

• 53:22Speaker 01

  Yeah, it's a fair point. It's a fair point too, right? But then, you know, you start looking at the GPU numbers and it's a little sad when you're outside of the lab. It's hard to get them and they're very expensive. So, so, so, but, but, but then GPUs are not everything. It's, but it's quite nice to have this whole ecosystem, right? You have both these little labs now and the big labs. Yeah, we'll, we'll, you know.

• 54:00Speaker 01

  I it's so fun because being in this AI little bubble here, you clearly see that there is a ton of competition, that change is coming, that, you know, we have not exhausted even on the current paths. There's still a lot of techniques to do. There's a lot of data and improvements and then and then and bigger models to train. And then there's all these new things that are bubbling. Maybe they're not ready, but they're very actively pursued with, with, you know, good resources. And then I feel like you step outside of San Francisco and people treat AI basically as if it was like from the last year before Codex and would never change again.

• 54:45Speaker 01

  And then now that is the wrong way of treating it. It has all, I mean, to me, the coding agents have been such a reveal that it's hard to hard to get over it. I call it AGI, you know. You should call AGI what they want. We may get one day past AGI the way we got past the Turing test, right? We don't really argue about the Turing test anymore. Is it passed? Is it not passed? Who cares? These things that they code with are clearly intelligent and coding and any disputable, you know.

• 55:26Speaker 00

  You know, obviously, the AI coding wars are like quite fierce right now. Like, you know, what do you think ultimately will determine, you know, which of these AI coding products ends up, you know, how do they become better than each other? And, you know, how do you see this? The next frontiers for like Codex and Claude Code?

• 55:42Speaker 01

  You know, I think coding market is good to have to big enough to have two programmers in it. It I think the bigger question will be, how well do they go to other fields, right? I mean, coding is great and it's important for us, but but you could do the work of many people.

• 56:08Speaker 01

  And currently Codex, I tried to recommend it to some friends, but you know, it used to start with the question, what is your GitHub repo? Well, that's a whole lot of people. Now it's a little bit friendlier, but it's still called Codex, you know, like,

• 56:24Speaker 01

  even so people kind of don't hear this is your accountant tool, right? And in contrast to ChatGPT, where you just said something, I think Codex takes a little bit of getting used to, then Claude even more, I would say, if you go on the code side.

• 56:44Speaker 01

  So, so, so I, I think that there is some question, how do you get this power to people in other occupations and places? And that may be the more important question.

• 56:55Speaker 00

  Anthropic tries with Claude Co-Work and like basically making a friendlier version of, of, of, of the core code products.

• 57:03Speaker 01

  I, I certainly feel like the abilities are there, right? As a NEMO person, I feel like these ones obviously can do these things. They obviously can do Excel, they obviously can do this or that or that.

• 57:17Speaker 01

  But then, of course, they, again, I watch them as like a hawk. I know how to, like, there is some level of skill that you need to put in to, to, to get this. Now, this is totally a learnable skill, but I understand that people are busy in their lives and don't necessarily want to learn this. So you need to smoothen it in some way. There are some fundamental things that I don't think will allow you to like just let it run, not watched.

• 57:43Speaker 00

  Yeah.

• 57:43Speaker 01

  You don't think you'll want to do this. But on the other hand, I don't think you would even want to do this, even if it was super good at first, right? You need to gain some trust. And so, so the question becomes, how do you convince people to start putting some effort into gaining this trust? It, it will pay back, but, but, but, but there is a hump.

• 58:01Speaker 00

  On the coding side, like, why, why do you think Anthropic was the first to be like really successful on the coding side?

• 58:07Speaker 01

  I think Anthropic made this very good decision to focus on coding, coding, right? This, this was at the time when OpenAI was like, we're doing ChatGPT. And, you know, great, I mean, chat is great. But and and I think part why Anthropic made this decision was that they just could not compete and and right and and chat, but they made a very good decision on what else to do. And and this goes back to, you know, AI goes through these upheavals, right? It it you need to put a bet on something that is not what is today, even though the things today like isn't chat GPT amazing, of course, right?

• 58:47Speaker 01

  It was the most amazing AI of 2025, but clearly not of 2026. We and maybe in 2027 we'll have we'll have another thing. Uh, so so things change quickly. If you put a good bet on on something else, you can. And it's not like OpenAI didn't do coding, it did, right? And and that's why it could catch up reasonably quickly, but it was just not the focus. I mean, you know, you these companies are tiny. You grow to a billion users, you you you have stuff to do, right? So they'll just fall apart.

• 59:26Speaker 00

  You mentioned uh uh this kind of like almost tension between, you know, nailing the stuff that's working today and then, you know, keeping other areas open so that if there's a glimmer of hope in a different area, you you kind of double down on that bet. And I'm wondering what you make of that, you know, obviously OpenAI, I think, very publicly has gone in this like focusing moment now, right? And and you've seen it, the results of Codex and, um, you know, maybe slashing Sora and some of the other things that were that were different. Yeah. How do you think about like kind of navigating that tension of of like really nailing some the here and now versus like keeping these other embers open uh that could potentially be really interesting down the line?

• 1:00:00Speaker 01

  It's a matter of culture and size and and money and perspective. Famously, Google, right? Google is the lab that will keep all this.

• 1:00:08Speaker 00

  I think some people have been quite critical of Google for this, right? Missing, you know, missing your your uh invention, uh, not being the ones to capitalize on it.

• 1:00:15Speaker 01

  You know, but then it it works for them, right? It it works because. It whatever good comes out, it's very easy to catch up because you already have a strong team in it, right?

• 1:00:27Speaker 00

  Yeah. Do you think they've caught up? I feel like there's a lot of discourse claiming that, you know, they're still a bit behind.

• 1:00:32Speaker 01

  They've caught up in the chat GPT world. They haven't yet caught up in the, I mean, I don't know if you've seen Anti-Graffiti 2.

• 1:00:39Speaker 00

  Yeah.

• 1:00:40Speaker 01

  I opened it after I O and that I couldn't tell which one is Codex and which one is.

• 1:00:44Speaker 00

  Of course, there are a lot of funny, funny tweets about that.

• 1:00:47Speaker 01

  So, so, so there, that's great. I, I tried to do some of my Codex things with the new 3.5 flash and it just doesn't work, right? It has not, the barrier that was on Christmas, I feel like it hasn't crossed it yet to me, but, but it will, right?

• 1:01:06Speaker 01

  So, so, so, so, so like if you're very broad, it can make it safer later if you need to catch up, but then you may not get the immediate win of like, you know, like Anthropic and Kodak, you're just the first to nail it and, and, you know, and great that there are labs that just go and are the first to nail it, that, that, that's exciting and, and, you know, I feel like that's how it should be.

• 1:01:35Speaker 01

  And I, you know, openly I had a good culture of, of making bets, but now it is also a bigger thing and it has some like, you know, chat GPT has a billion users. It's important for many people in the world. You should, and, and Google search has three billion users, right? It's important for many people in the world. You don't want these things to be hampered and.

• 1:01:59Speaker 00

  No, totally.

• 1:02:00Speaker 01

  You, you should go fast, but, but this breaking things is, is, is not so good. And I actually feel like it's quite good if the labs don't break everything on the way.

• 1:02:10Speaker 00

  I guess, you know, a lot of people wonder about the, the kind of gap between closed source models and open source models. And it feels like there's two, you know, distinct things pulling in different directions. One is it feels relatively easy to distill models and, you know, you've seen a lot of, a, a, a lot of claims around folks doing that on the, on the Chinese open source side with, with the closed source providers. Then on the other hand, it feels like more and more of these models, even in the big labs are getting too big to serve. So they have to be distilled within the big labs themselves. What's your kind of gut intuition on the gap we'll see between closed source and open source models? And, and whether that widens or shrinks in the next few years?

• 1:02:43Speaker 01

  Yeah, it, it is not that easy to predict, I feel it. So.

• 1:02:50Speaker 01

  Bigger models are better, you can distill them, but the distilled models are never quite.

• 1:02:58Speaker 01

  Like, they're great, right? Especially if you need a model for some money. But they're not quite as good as the big models. I, you know, I just like the 3.5 Flash, I could not

• 1:03:13Speaker 01

  quite feel it's on par with 5.5. Maybe because it's a distilled pro, right? Maybe we just need to wait for the pro. So even within the, like, I, for example, I don't remember when I have used the mini model, the mini and animals, I think they're very good, they're very useful. I just haven't used them in a while, right? So, and whenever I use them, they're fine until they trip and cost me so much time, I go back to the big one. And so you can distill things and

• 1:03:51Speaker 01

  you know, when the open source can distill or not distill, I mean, you know, let's just try to not make you distill everything naturally, but I think they also like don't fight you to death. Yeah, it would be very sad if open source had like models that are very, very far behind, but I don't think there is a risk of that. There is enough companies and now there's like, you know, notions of I also very much understand, like, if you're a country, right? Do you want to depend, like, say your police stations or hospitals run AI to like help you do administration? Maybe you don't want to rely on one company that may just have an outage or you understand, like,

• 1:04:36Speaker 01

  so for, there'll be a lot of people who want sovereign, they say, models, right? Even if they're slightly weaker, maybe the tasks are not so hard. So I think there will be enough incentives to have open models. That they will exist and there will be a very good incentives for the labs to still keep ahead. So, so you know, people like, people keep paying for, for, for this. So, it feels like a state that should persist for a while. But you, you know, I, I, it's famous last words and AI and tech, you can say things and they may turn out. I don't want to make future predictions.

• 1:05:15Speaker 00

  Of course, but what, what job is a podcast if not to try and force you into them? But, uh, no, that, that all, that all makes a ton of sense. Um, you know, we always like to enter interviews with kind of a quickfire round where we stuff in a bunch of, of broad questions at the end. And, and so maybe to start, I just love like, what's one thing you've changed your mind on in the AI world in the last year?

• 1:05:31Speaker 01

  Uh, well, definitely, I, I, I did not believe that, that they will be like a intern kind of thing so fast. And I have definitely changed my mind. I, I, I actually used to not talk to AI very much every day. People were always like, so how do you use chat GPT? And I was like, yeah, I don't know. I, I asked it one query yesterday and one, three days ago. And, but it was always like, I'm not gonna talk to my computer very much. And now I do about work. Yeah. And, and, and so, so I, yeah, it, it like, yeah, I, I also did not think I'm gonna like not use a editor for programming. And now I don't,

• 1:06:14Speaker 01

  I just tell it to change the code. And yeah, that, that was a big update.

• 1:06:21Speaker 00

  That's awesome. Um, I guess, you know, as you've worked at these models more closely these past years, have your concerns around like the existential risk of, of, of safety around these models, have they, they gone up or down these past years?

• 1:06:33Speaker 01

  I don't think they have changed very much for me. I was always on the, you know, not too worried, but, but also we should not be complacent side.

• 1:06:47Speaker 01

  And I still feel, you know, with all the skills they have now with programming and so on, I still feel the the the small risks, right? The the risk, the risk that they will hack some of our systems, maybe the grid go down or or things like that. I still feel these are the risks I would focus on right now. Not to say that existential risks don't, you know, it's good that there are people thinking about it. It's good to have some guardrails. It's it's in the end good to, you know, we should be able to turn off this these data centers if we so decide and and have control over all of that. But I I don't feel, even though the mouse have become much better, I don't feel any threat from them.

• 1:07:32Speaker 00

  On the on the lab side, it feels like the buzzy news of the last week was that, like, you know, Andrej Karpathy was going to Anthropic to work on on RSI, right, as as a team there. Like, what'd you what'd you make of that?

• 1:07:44Speaker 01

  You you you know, I I am part of the psychosis, right? It's it's like you can do so much research with with this assistant, right? And and it's amazing and and you should and and you can make many parts of the systems better too, like much faster. So so that that's certainly true. But on the other hand, like, you think about these post transformers things and, you know, the space of ideas is vast and and luckily, most of them are wrong. That that's why it's called research, right? And and you need enormous luck and and skill, but but also luck to to to happen upon the right one. And we kind of feel like maybe it's somewhere there in the air, but but it's research.

• 1:08:25Speaker 01

  It may be years away and and even with the best AGIs of the world, we, you know, they're they're like human level, there may be researcher level, maybe there'll be like a 10x researcher.

• 1:08:38Speaker 01

  But for years, there was a huge community of researchers trying to crack these things, and they didn't. So it may be just very hard. And yeah, like, we understand very little about the human brain yet, and we cannot connect it to our M L in any great way yet. So I'm like, on the one hand, I think it's great. I think we'll see the current things getting better. But if you're thinking of a research breakthrough, it may just require something that even when you have like this, if you're searching in a very efficient way, and even if you're searching some interesting ideas, that still doesn't mean you're gonna find it, right? Yes.

• 1:09:21Speaker 01

  Just the space of all ideas is so vast that even very efficient searches can just not get there. So I'm not that worried existentially about this.

• 1:09:35Speaker 00

  One thing I found interesting is, I think, you know, if I'm correct, like, all of your Transformer paper co-authors have gone on to start companies, right? And I'm wondering if that was ever something you thought about or?

• 1:09:46Speaker 01

  Well, I was certainly asked about it many, many, many times. Yeah, I, well, I'm very happy that I didn't so far. I thought both my time at Google and my time at OpenAI has been great. And it was a privilege to be there and to be able to do the work. I love technical work, you know. Everyone who started a company has thought maybe they won't need to spend so much time on the company work, and it feels like they had to. Totally. You know, but then, sometimes companies do amazing things.

• 1:10:31Speaker 00

  This has been a fascinating conversation. I want to make sure to leave the last word to you. Anything you want to point our listeners to, or thoughts you want to leave them with? The mic is yours.

• 1:10:39Speaker 01

  I thank you. I just want to, I think I said it already, but I just want to repeat, I feel this time now that You have like powerful GPUs that you can put under your desk and and coding agents that can really help you push them to their limits. And and the time where, you know, all the big things are pushing the transformers and great they are because they're amazing, but but there is this whiff of of possibly other things. I think it is still, and again, the most exciting time to be a researcher in machine learning.

• 1:11:17Speaker 01

  And I want to encourage everyone to to just go and try their ideas, to to learn from others. If anything, I I feel like we should publish more of like wild things. I I feel always a little sad when so many papers are about like, oh, we took a pre-trained model and R L'd it in a slightly different way. I mean, it's it's good, but, you know, you don't need to catch up with what is there. You can just do new things, even if even if they'll start smaller, even if, you know, maybe it won't work the first time. Um, you know, no, no, nobody talks to me about the paper I had before attention is all you need, which is you don't need attention. I had the paper at Europe's the year before. Really?

• 1:12:01Speaker 01

  Well, it wasn't wasn't quite a good advice, but you you you need to explore the wrong things because they may lead you to the right thing. And and this is also what models are still so bad at, which I think Jerry is is trying to push. Women are very bad at like learning from a totally wrong direction to actually twist it to a right one. That's what we humans can can still do very well, so we should do more of it, like we should just do wild explorations, even if they fail. And I feel now that the, you know, it's if you put a lot of your own effort without an agent, it's it's done very hard to when it fails. I think with agents, it's even easier.

• 1:12:44Speaker 01

  So I want to encourage everyone to, you know, to research explorations, fail fail when it comes to this. This is how we'll.

• 1:12:53Speaker 00

  How we can get to interesting things. I love that. Well, I feel like that's the perfect note to end on. Thank you so much for for coming on the pod. This was super fun. Thank you so much for having me. I'm Jacob Effron, and this has been Unsupervised Learning, a podcast where I get to talk to the smartest people in AI and ask them tons of questions about what's happening with models and what it means for businesses in the world. As I hope is clear, I have a ton of fun doing this. It's a nights and weekends project in addition to my day job as an investor at Redpoint. But our ability to get these incredible guests on really comes from folks like you subscribing to the podcast, sharing it with friends. It's really what ultimately makes this whole thing work. And so please consider doing that. And thank you so much for your support and listening. We'll see you next episode.