Ep 58: Google Researchers Noam Shazeer and Jack Rae on Scaling Test-time Compute, Reactions to Ilya & AGI

On the latest episode of Unsupervised Learning, Jacob is joined by two of the most influential minds in AI today. 🔹 Noam Shazeer, co-inventor of the Transformer 🔹 Jack Rae, Research Director at DeepMind and one of the leads behind Gemini’s Flash Thinking We got to ask them all of the top-of-mind questions in AI today about where we are, where we’re headed and what it means for businesses and the world.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Jacob Effron?

  Noam Shazeer and Jack Rae really need no introduction. The two are at the forefront of Google's Gemini LLM efforts and have been involved in some of the most important discoveries in AI in the last decade. Noam as one of the co-inventors of the transformer and mixture of experts. Jack, key part of many DeepMind breakthroughs. It's a real privilege of the job to get to sit with these two and ask them literally every top of mind question in AI today. We talked about how far test time compute will get us and the spaces where it will and won't work. We talked about how the infrastructure needs will be different for test time compute versus the large kind of pre-training paradigm. We also hit on the impressive pace at which open source models have caught up with closed source peers and their reactions to DeepSeek. We talked about their kind of reception and reaction to Ilya saying that this test

• 0:42Jacob Effron?

  time compute paradigm won't get us all the way to AGI, as well as Yann LeCun saying this current generation of models can't actually have any novel thoughts. We talked about what it actually looks like to do cutting-edge AI research today and what their day-to-days look like, as well as the future model milestones that actually matter to them. And then we also got Noam's reflections on character, as well as both of their responses to what AGI means for the role of humanity. I think folks are going to love this. It was really just a pleasure to get to speak with both Noam and Jack. Without further ado, here they are.

• 1:11Jacob Effron?

  Well, Noam and Jack, thanks so much for coming on the podcast.

• 1:13Speaker 01

  Oh, thank you.

• 1:14Jacob Effron?

  Are we sitting in the very office that the transformer was invented?

• 1:18Speaker 01

  No, this is a new building. We were in 1965 Charleston, I think. Probably about half a mile.

• 1:25Jacob Effron?

  Half a mile. All right, so it's in the air around here.

• 1:27Speaker 01

  Pretty close. Pretty close.

• 1:28Jacob Effron?

  Yeah. Well, many things to dive into today. I mean, obviously want to start with some of the latest Gemini 2.0 models and obviously all the work you've been doing around test time compute and Gemini 2.0 flash thinking. I guess just at the highest level for our listeners, how do you characterize where these models work today, where they don't work as well? And as you were kind of experimenting with them, what surprised you most about those results?

• 1:48Speaker 02

  One surprising thing is kind of when we started the particular concerted effort to build a lot of kind of research into test time compute, into Gemini, and then think about shipping it, is that we were really kind of focused on starting out with reasoning tasks. So math and code were like big areas. Of focus and it wasn't really clear, you know, whilst we're kind of sprinting in that kind of domain. We obviously want to broaden it naturally over time, but it wasn't really clear like how that would work. Would there be any kind of sense of generalization? Would thinking be useful beyond those reasoning tasks if we're just concentrating on those as researchers?

• 2:28Speaker 02

  And I think it was pretty fun to see one of the early kind of models that we felt like was had been trained to try and match the style of Gemini flash. So it had been trained with thinking, but then it also undergone some kind of training to actually be just like generally a nice style, a nice like model to talk with. But it was actually like very fun seeing thinking interact and improve like creative tasks as well. You could ask the document to like compose kind of an essay on a particular topic. And actually like A, the thought content was like very fun to read. And, and it would go through like various different ideas and then it would go through like revisions of the idea or things that it should cut. And that was kind of fun. And then also the output like felt really nice.

• 3:12Speaker 02

  So that was one thing that kind of surprised me.

• 3:14Jacob Effron?

  Any surprise for you, No?

• 3:15Speaker 01

  Well, I, yeah, I mean, in general, like I'm all for like generality, like let's train something that's great at everything. It is, it is important, you know, and I was skeptical at first of like, okay, this intense focus on, on, on things like math, but it is very important to have, have good benchmarks that, you know, are going to encourage you to, you know, to be able to reason about difficult, difficult tasks, you know, because a lot of things will drop perplexity, like add more parameters to the model and, and memorize more.

• 3:55Speaker 01

  So, so it's nice to have the evals that can distinguish better, you know, some of the, you know, more difficult problems.

• 4:05Jacob Effron?

  I mean, what evals are even meaningful to you at this point? I mean, obviously I feel like people are trying to hill climb the same, the same set of evals that feel increasingly less relevant to day to day work. What do you guys like do when you're testing these models or, you know, how do you vibe check them?

• 4:17Speaker 02

  It feels like we keep landing on an eval and we're like. Oh, actually, we overlook this eval, even if it's in math. It's like, OK, we've done a bunch of like math evals, but maybe like, I don't know, Putnam answers only, AIME, they're still considered challenging. And then it's like, done, OK, they're completely saturated and we really like don't care. And they're small and why did and we almost think, why did we ever even work on them? And it's easy to like, forget.

• 4:41Jacob Effron?

  That was so easy, what we thought six months ago.

• 4:42Speaker 02

  Yeah, a couple of months ago, it was like considered really hard, maybe too hard for the model. And then it suddenly like snaps to being trivial. So right now, I do feel like there's always been like a lot of concerted effort within within DeepMind and within like Google as a whole to develop useful evals. But it's kind of very nice to see also, this is like a shared responsibility across like many different AI labs and even like scale has like started really stepping up and developing like.

• 5:11Jacob Effron?

  I feel like calling it humanity's last exam was a dangerous, you know, if every six months, we think it wasn't too hard. It's a dangerous title. The really, really last exam. Yeah, exactly. But the whole thing is. 73. Last exam.

• 5:22Speaker 01

  It's very, very challenging in general because, you know, evals get leaked. You know, like, once people start talking about the evals, then there's all this text out there about the evals and and, you know, they're no good anymore because everyone knows the problems. And, you know, all the models will know the problems unless you're very, very careful. So there's, you know, I think there's still a lot of work that goes on into into having evals that are, you know, that are private.

• 5:54Jacob Effron?

  Are there a set of milestones that are like meaningful to you? It's like, hey, when, you know, when Gemini 3.0 can do X, like that's, that's a really exciting, you know, milestone, whether it's an eval or just something that you tried with these models and they can't quite do yet.

• 6:05Speaker 01

  I'd say when Gemini 3.0 writes Gemini 4.0.

• 6:11Speaker 01

  Or I should say Gemini X writes Gemini X plus one. Yeah, I think, I think these reinforcement loops are are probably the most important thing to to pay attention to. And, you know, there are, there are several. Reinforcement loops going on, you know, what the one I just mentioned is probably the most important one that we can actually use the AI we're building as a tool to make ourselves more productive at building AI. But then, you know, then there are other reinforcement loops around, you know, data flywheels, like, you know, you get, you have people use these models and provide feedback and make them better at the things that people care about.

• 6:55Speaker 01

  I think we'll see a huge acceleration from that. And then there's just the global excitement and funding flywheel, which is, I think, also seems to be kicking up in the last few years.

• 7:09Jacob Effron?

  Yes, certainly so. To your point, I guess, of having you kind of armed with like a thousand AI engineers by your side, making you even more productive, like, where are we in, you know, do you have the equivalent of a point one of an FTE today, using some of these Gemini models alongside your research?

• 7:23Speaker 02

  Yeah, I think one benefit we have at Google is like, we work in a very like structured monorepo. So, and we have a lot of like amazing tooling around like contributing to the code base already. So there's like lots of angles where AI is kind of like being pulled in as tooling for our own development. Like, I think Jeff quoted the statistic, although I don't know, but like the number of what we could call like pull requests that have

• 7:51Jacob Effron?

  It was like 25%, right?

• 7:52Speaker 02

  Yeah, like useful, like just like AI, like bug fixes or code reviews attached. That's already like, that just like gets pulled in one day, I notice it there. I'm like, oh, that's cool. I can now like do a lot of like apply fixes that it already like spots. But, you know, that's just like one element where we're already like pulling in AI towards our own like coding development. I think we're incredibly excited about like agentic coding is just definitely very important and trying to get the model to be able to tackle more open-ended and difficult tasks is definitely something we're very excited about. And I think it's just like, in some ways, it's very

• 8:34Speaker 02

  It's a lot easier to orchestrate when we have like a very defined way of like defining libraries. Like we have these like build rules and things and it's just like everything gels together in terms of the whole code base very well. So it's, I can imagine it's going to be like as progress kind of continues. It's going to be a very discreet moment where suddenly lots of libraries can be like very quickly iterated on within our code base.

• 9:00Jacob Effron?

  Yeah, then you got, you got, you got your AI engineers proposing experiments to you for what's, what's good to try. It seems that these models work super well in easily verifiable domains and coding and math and have obviously been among them. Like, how do you think about how for some of these less easily verifiable domains, these models may end up scaling and being useful?

• 9:18Speaker 01

  I mean, they're getting better at that, at that stuff too, but yeah, it's, it's definitely, definitely harder. I guess the, in those domains, you know, we're going to need either better ways of verifying or more human feedback loops.

• 9:38Speaker 02

  Yeah, I think what's good is that we're seeing is like even with the Gemini model series, they're able to follow much more abstract instructions. So being able to try and provide a reward signal over a qualitative piece of work, which maybe if a human were going to try and give a feedback signal, there would be a quite a broad set of like rubrics or like grading criteria. Or maybe there is even more like a even kind of simplistic sense of like what is good style? What is like interesting? And, and so I think part of the problem is really training models to like take on these like very broad criteria for what,

• 10:21Speaker 02

  how to, how to like take in a very broad set of criteria and then apply a reward signal. Once we have the reward signal, we can train with reinforcement learning against it. And, and I think, yeah, we're already seeing that kind of like makes, makes sense. It's not like an abstract thing anymore. It seemed like a very abstract thing. Maybe a year ago or two years ago.

• 10:41Jacob Effron?

  I'm curious, a year or two ago, like, did you expect that to work? Or did you think this, like, whole path of of research was really more toward these, like, more easily verifiable domains?

• 10:48Speaker 02

  I feel like you expect it to work one day. And then it feels like there's a very, there's a very complicated stack of things that we feel like we need to solve to get there. And then usually the case, it's like, oh, it turns out there was a much simpler path. That's how I feel about it, anyway.

• 11:09Speaker 01

  Yeah, usually there are good surprises, you know, more good surprises than bad surprises.

• 11:14Jacob Effron?

  You've released these models out in the wild. What are some of your favorite, kind of, like, ways you've seen people using them? And what would you like to see more people, like, trying and building with these things?

• 11:23Speaker 02

  Actually, there's just like an update today happening in the Gemini app where we're putting in a much stronger model and it's being integrated with basically the full suite of, like, Gemini app tools. So it's like all of the apps that Gemini app supports, from things like, you know, maps, also like search integration, and now has like very long context, all these things. It's kind of like a fully featured Gemini thinking releasing in the app. And it is actually, you know, it's a very enjoyable experience. That's what I think so far, people, even for their day to day stuff, people, it wasn't clear to me whether they would like to have to pay a bit more latency to, like, get the model to think about stuff before responding. It does seem to be the case that if people are gonna, like, pull their phone out of their pocket and then type something in, that actually what we thought was a very long time, maybe

• 12:06Speaker 02

  a couple of seconds, is like a very small price to pay for something that they feel like is a better quality answer. And also, maybe they can, like, sometimes look at the thoughts and inspect them. Like, I was kind of showing this to my mom, like, a couple of days ago.

• 12:20Jacob Effron?

  Like, moms are the ultimate test of whether, like, something has broken the barrier from, like, the Twitter sphere to, like, the real world.

• 12:26Speaker 02

  Yeah, like, the vibe, the mom vibe check is a big, is a big deal. Yeah, yeah. And yeah, I guess she asked a lot of, like, what I would consider very generic questions to ask. A model like what is the meaning of life? Oh, meaning of life. She went for what is the meaning of life and she really like sand read it for a long time and then she read the thoughts as well and then she kind of like contemplated on it and she felt like she, I don't know, she seemed to kind of very much appreciate the the presence of like thoughts as to like how to even go around such like a an open-ended question.

• 12:58Jacob Effron?

  So more folks building philosophical conversations with these with these models. Well, I mean, one thing I think is super impressive about them is the multimodal capabilities, but it still feels like those are very underexplored from an application perspective. Not sure totally why that is, but.

• 13:11Speaker 02

  Yeah, I think right now, in some ways, we we're quite modest about the multimodal capabilities of Gemini. I feel like the model has always been incredibly strong at image input. Image input plus thinking is like actually like remarkably good, I would say. I see I see a lot of people kind of like red teaming the model on things like on X and trying like difficult or challenging like images, visual reasoning problems. I think that is working pretty well. There is also like at some point, some of those are kind of toy vowels, but but pulling in multimodal with then agentic tasks is super interesting as well. So like we launched Mariner last December, which was like an agent which like uses a

• 13:56Speaker 02

  browser and things. That has a lot of multimodal like aspects all built in. It was super important that they could get the model to be incredibly strong at like not only kind of scanning a screen, but like really understanding it and knowing how to act on many, many different types of websites. So like pairing that kind of capability, agentic, quite open-ended, maybe you really need visual understanding of a potentially messy scenes with thinking is, yeah, something that I think I'm feeling very excited about.

• 14:23Speaker 01

  Yeah, that's one. I mean, it made sense to start with text, you know, a picture is worth a thousand words, but it's a million pixels, so like the text is still like, you know, a thousand times more more information dense, you know, just to do math on cliches. But and and then there's also like a lot more just in the main training data for text. You know, we have so many examples of text which actually kind of represents, you know, the way humans receive and produce information, you know, through language, but, you know, we have a lot.

• 15:04Speaker 01

  Fewer examples where for something like, say, image generation because you don't have examples of people generating images. So things are a little more challenging, but we, you know, it's, as Jack said, great stuff, great stuff happening.

• 15:21Jacob Effron?

  Yeah. When you mentioned Project Mariner, I'm curious, like, you know, I think we both talked before about, you know, to get really, you know, agents more widely used, you kind of need to solve both complexity of reasoning and then also reliability. How would you kind of characterize where we are today in terms of applying, you know, Gemini models toward these problems and like, what is the actual path to getting better at a bunch of these things?

• 15:40Speaker 01

  I mean, there are a lot of answers. I mean, one is just make the model smarter. That will, that will always help. And, you know, most likely, the, you know, you need very general, general solutions for, you know, for these control problems, you know, just like you need general solutions for the intelligence problems because, you know, people are going to use these things in so, so many unforeseen ways, you know, like you cannot, you can't, you can't anticipate it.

• 16:23Speaker 01

  The users are smarter than, than the developers in figuring out what, what the use cases are, you know, that no one, no one envisioned, like when they invented the internet, like what the internet was going to be for, no one envisioned when they invented computers, what computers are going to be for. And then, you know, AI is, you know, is getting so, so general these days that it's, it's even more, more true. We're just building, building a product with, with billions and billions of use cases that are unanticipated.

• 17:02Speaker 01

  So, so we will need to build the general solutions for, for all of it.

• 17:09Jacob Effron?

  How far away does it feel like we are from that kind of like next level of complexity and reliability?

• 17:13Speaker 02

  I don't want to give a precise, like, is it six months? Is it 18 months? Is it 24 months? But.

• 17:22Speaker 02

  I think a lot of really the, a lot of the really the time actually is not so much dependent on the, in my opinion, is like, not really about like the core algorithmic AI development. There's part, part of it is just like, there's a lot of even like engineering challenges to really changing the whole way you train to being in more complex agentic environments that, that has like some, like, almost constant time but non-trivial cost to like switch how we do research. So I feel like agentic research, it seems like a lot of the upfront challenges, like it's, it's no longer going to be very simple prompts and responses. We're now going to act in an environment.

• 18:05Speaker 02

  So how you define those environments? That, that kind of angle though, is at the very least been something that DeepMind has been brought in on even since I joined in 2014. It's always been about, there's part of building AGI, which is like figuring out a really good agent, and there's part which is figuring out a really good general environment. I don't, I don't think we, like, anyone has solved the perfect environment yet. And you know, there's, there's some obvious ones. There's a notion of like, using like a web UI and, and like kind of being able to automate many kind of like web tasks. There's a notion of like having a code base and being able to like work within that code base and do many useful things. But I think picking out those like, if you can pick out a really good environment, then we can like accelerate a lot of agentic research in, in that environment

• 18:50Speaker 02

  and build really good algorithms. And I feel like that's as big a part of the challenges as any given breakthrough in like attention or in long context or reinforcement learning.

• 19:00Jacob Effron?

  How high do you guys think the ceiling is for continuing on this like test time compete vector? I mean, obviously, you know, I think very publicly, like, Ilya's come out and said, there's this entirely, you know, new direction that's needed to really advance AI to the next level. You know, do you agree with that?

• 19:15Speaker 01

  Yes, I, I pretty much agree because, because LLM search is too cheap. Like, you know, like if you, what, like, operations costs?

• 19:26Speaker 01

  You know, under 10 to the negative $18 these days. So like, you know, if you can infer relatively efficiently even on a very large model, you're, you know, you're, you're, you're getting like over a million tokens per dollar, you know, if, you know, I guess you can just check the prices on Gemini or anybody else. And, you know, that, that, yeah, so you're getting millions of tokens per dollar. That's way, orders of magnitude below the cost of most other things you can think of. Like, if you think of like a really cheap pastime, like go like buy a paperback book and read it or something, you're paying

• 20:06Speaker 01

  like 10,000 tokens per dollar. So we're like a couple of orders of magnitude cheaper than, you know, than like reading a book and, you know, the, you know, probably, you know, four orders of magnitude cheaper than, you know, paying anybody to do anything and, you know, whatever, six, six, eight orders of magnitude cheaper than, than, than paying a software engineer. But, so there's, there's a huge, huge margin of difference there to apply more compute and make the thing smarter. And, you know, if, if the value is there,

• 20:47Speaker 01

  which I think, which I think it is, like, you know, would you pay, you know, you know, five cents an hour for the bad engineer or 10 cents an hour for the good engineer? You know, like it's, there's, so there's a huge amount of sort of unexploited flops in there for, to use if we can find ways to use them. And like one way to, to use them straightforwardly is, okay, just train a bigger, better model. We're already doing that, but still like

• 21:27Speaker 01

  model training costs tend to go up quadratically with, you know, with, with the size of the models. So.

• 21:38Speaker 01

  So then you still end up with a relatively cheap inference if you do it right. So then, of course, what everyone is doing now is just apply more compute at inference time through this chain of thought thinking or any other brilliant algorithms we can come up with. And so I think we're just going to start seeing a scaling curve there as well as we're seeing in a lot of places.

• 22:10Jacob Effron?

  Right. But does that scale us all the way to the AGI future people envision? Or is there some kind of completely adjacent thing that's required to... Where does it ask for?

• 22:20Speaker 01

  Yeah, I guess that's where we'll see whether it's the humans that invent the next breakthrough or the AI. But I'm not... I've given up on organizing my garage and stuff like that because I'll just wait for the robots.

• 22:36Jacob Effron?

  Yeah? You think they're coming? I guess you guys had a big robotics release yesterday.

• 22:41Speaker 02

  Yeah, that's awesome. Yeah.

• 22:43Jacob Effron?

  Is it ready to clean your garage, though?

• 22:45Speaker 01

  Not that I know of.

• 22:47Speaker 02

  Yeah, I think the question of how much more is there to give in this test time compute paradigm? Is it all the way to AGI? I don't think it's all the way to AGI. I think we already kind of established there's other components, being able to act in a complex environment. Acting is very important. Research into acting agents, that's a definite investment. And there's many other aspects. But are we seeing test time compute asymptotting? I think the Ramanujan example is always in my mind here where it's like, I don't want test time compute to just, for any given problem, think longer and then eventually arrive at a solution. But we also want it to be able to think very deeply and actually create maybe useful

• 23:30Speaker 02

  knowledge that it's going to actually incorporate to then solve further tasks in its thoughts and thus improve dramatically improve data efficiency. If you can just have one math textbook and you spend most of your time really just thinking and playing around with the ideas, then you can become a well-class mathematician. And that would be the kind of thing that I feel like, you know, should be and we should strive to achieve with like a very deep thinking model. Are we there yet? No. Like, do we have a path there? I think there is. Like, there are many directions of gradient towards such a model. We've been already seeing, and this is one thing, I think people don't really like talk about too much with test time compute.

• 24:10Speaker 02

  But we're already seeing like amazing improvement in data efficiency by training the model to like think deeply with reinforcement learning when it's solving the task. So even if we're, you know, gonna have a bunch of like RL data, we're not gonna add any more. Even just like going, the test time compute kind of paradigm is allowing us to kind of learn a lot more from that data. And I think we could probably push that much more so. There's kind of many particular research angles we're interested in and how this model could think not just in one particular, you know, spitting out a couple of thousands of tokens in order to solve the task, but maybe like think much more deeply, much more kind of like a researcher might think about a hard and open end of problem.

• 24:50Jacob Effron?

  I think so much of what you guys have touched on is like these models increasingly acting like researchers. What are like the early signs of that that you would look for?

• 24:56Speaker 02

  I think like, you know, I'll give one example, which is like math. So I think right now math is being treated as a, it's a bit strange. And math is often used as benchmarks and they've kind of like exams and maybe even like math competitions. There's gonna be like a very important pivot from the math benchmarks and to being really starting to become about like actually generating useful math. Starting to solve actually important problems that we really care about.

• 25:26Speaker 02

  I think there are ways, like I think it was very cool, this like frontier math eval that was created, which is trying to provide like a kind of a gradient towards that. So maybe like the harder category are basically almost like unpublished math findings. The easier category are supposed to just be like kind of harder or trickier. You know, I don't know whether that particular eval is the perfect way, but having some kind of ramp of evals that like bridges from where we're at now to like actually useful scientific contributions. This is something I'm like especially excited in as, you know, bringing in professor. And researchers and saying, okay, use this tool in a non-ironic way to actually accelerate research in this area. What would you do? What's missing? What do we need to advance?

• 26:07Speaker 02

  And I think, um, yeah, I think like this kind of notion of like incrementally harder benchmarks, it might sound like, oh, that's like what AI researchers always say, but I think that's going to basically be my metric of progress.

• 26:19Speaker 01

  I mean, math is a great example here because I mean, this is a field that you don't actually need more data. Like, you know, like people in math did all this math without, without this external input or, or, uh,

• 26:33Jacob Effron?

  Yeah, a lot less data. A lot of times in like a room, just thinking.

• 26:35Speaker 01

  Yeah, just go into a room. And then there are examples where, okay, there's, there's some data, but, you know, okay, Isaac Newton like takes, takes a bunch of astronomical observations, like goes into quarantine and like, you know, in his house and then like invents physics or something. So, okay, that's an example where there's data, but, but nobody actually knows, uh, you know, knows physics and then you generate physics and then math is even crazier because you start with, you know, with roughly no data and, and invent something useful. So, um, so that, that's, um,

• 27:15Speaker 01

  that kind of could provide a counter proof to, um, to the assertion that, hey, um, this is just learning to mimic people that, that, okay, the most we can do with AI is to, um, is to relearn what everybody knows and what we've, uh,

• 27:33Jacob Effron?

  Absolutely. The learning to mimic people critique, I feel, has been around for a while. Like Yann LeCun said a lot about this. Like, do you feel like that's entirely disproven at this point? Basically the critique that like, uh, you know, novel discovery and thinking is kind of impossible in these current moral architectures.

• 27:45Speaker 02

  Oh, well, there's definitely one class of scientific discovery that I think almost no one could argue against, which is like, actually a lot of sciences, like if you knew about these two disjointed pieces of information and thought about the intersection for a while, you would realize a new property. Like in kind of material science, it might be like, oh, turns out just like associating things way better means actually you know way more about like what may, you know, what new kind of material may be photovoltaic, but also may, uh, you know, et cetera. So there's like interpolation already would actually Completely accelerate science. And I think I'm guessing then it's always going to be this kind of whack a mole of like what is considered actually interpolating known ideas versus like

• 28:28Speaker 02

  creating a completely novel idea. And for that one, I'm gonna go to no.

• 28:35Speaker 01

  Yeah, I, you know, okay, I don't know, but I guess I'd just throw it back at Yan Le Cun to prove that he generated a completely novel idea.

• 28:48Speaker 01

  We'll see. I don't actually care about arguing this stuff. Let's, you know, let's just, you know, build AI, you know, greatly increase the level of technology in the world, help people. That seems good enough for me.

• 29:05Speaker 02

  Even if we go back to the math example, like, the thing I'd love to see is like, you know, maybe the state of mathematics right now is kind of like the state of kind of geographic exploration in the 15th century or something. So it's like, there's some known things, there's some like fuzzy area of like, we don't really know what's like beyond these boundaries and we have a bit of a guess. And then you send a small number of people to go off in a boat. It's very expensive to try and like, push the boundary of like, what is known and what is not known. And it Then they come back with some pretty funny looking maps. Yeah, they come back with like, a little bit of extra territory, like, explored, and then they they tell people. And that's a little bit like what's happening right now in mathematics. You have a very small number of like, elite math professors that are able to really ask

• 29:49Speaker 02

  the right questions and then actually like, prove useful qualities. And it kind of grows. And it's grown that way. And that's, that's how it's been going so far. Right? If you, if we can train a model that can actually like, essentially ask, like, it can pose the right questions of what you could say is like, the space of all useful mathematics is kind of an infinite space. You don't want this to go and like, off like a fractal on, on kind of uninteresting questions. But if, if there was some notion of like, all the set of like, interesting mathematical questions, if it can like, keep posing new ones. And then if it's very, very strong and it can solve those, then at some point, you know, maybe like, right now, we have a pretty strong kind of math model. And at some point, maybe it will be a professor level, maybe one day it'll be Terry Tao level, and then you have like a million of them.

• 30:34Speaker 02

  And then now it's like, now, I think then you could hope to maybe complete the map. And then what would like completing the map look like? That could be one of the greatest contributions to kind of science. You would now physics, chemistry, you'd now like be able to have like a very deep understanding of like any useful mathematics. I think that would be like a very, very exciting thing. Whether it's possible, I'm not sure, but I would agree maybe the key crooks is like, can I ask like actually novel questions? The question posing things seems to be the hardest part, in my opinion. The solving, I feel very confident, we will get there.

• 31:11Speaker 01

  But maybe mathematics is infinite.

• 31:13Speaker 02

  Yeah, I mean, it's definitely infinite. So then it needs to be.

• 31:16Speaker 01

  We can do so much better.

• 31:17Speaker 02

  Yeah, yeah.

• 31:19Jacob Effron?

  I just want to just talk about the culture of AI research a little bit. Like, you know, Noam, you were obviously part of the, you know, a leading part of the original transformer paper. Both of you have been a part of so many breakthroughs. You know, people like to write these thought pieces about, you know, the culture that drove these innovations. Curious like what you think the main takeaways you have after, you know, having done this research, you know, some periods of great success, some periods maybe of more frustration. Like, what lessons do you take away from what actually, you know, works from a cultural and team structure perspective to drive this stuff forward?

• 31:45Speaker 01

  Maybe, maybe sort of AI research is kind of where chemistry was in the 15th century.

• 31:53Jacob Effron?

  Say more.

• 31:54Speaker 01

  Alchemy, it's alchemy. Like, we don't know why it works, like, but it's highly experimental, you know, in terms of, you know, you get some idea, but really the proof is always in try it out. And then, you know, and then you have various observations, you come to hypotheses about why, you know, okay, this thing works, you know, why, what's, you know, what's the, what's the key thing? And, you know, so sometimes you're right, sometimes you're wrong. It usually takes, takes more, more experimentation to find out, or you can just like, you know,

• 32:36Speaker 01

  just claim that, okay, it works because of my magic XYZ, and you have your assistant swallowing frogs or something.

• 32:45Speaker 01

  Or the equivalent, you know, I find, you know, naturally researchers just, you know, love to share and get excited about, you know, about what they're doing. And, you know, it is always important to try to, you know, credit people, you know, liberally because, you know, it's, it's, you know, it's often very complicated to, you know, okay, what idea led to what idea, what was the, what was the key insight? And I think at some point, we'll, we'll just kind of have to give up on credit assignment temporarily and,

• 33:28Speaker 01

  and take a superintelligence-based approach of, we'll wait for the superintelligence to sort it all out.

• 33:36Jacob Effron?

  Superintelligence will write some great thought pieces on the culture that drove, drove these transformations.

• 33:44Speaker 01

  But it is super fun working at Google with this group of brilliant people and collaborating with everybody.

• 33:50Jacob Effron?

  And one thing I'm struck by by the transformer story, I think, is like, you know, it involved, I think you were like, you heard randomly that some people were working on, you know, when you tell the story, it sounds like, wow, that could have easily not happened. Like, I don't know, if you weren't walking down that hall one day. Does it feel like inevitably someone six months later would have figured that out? Or like, how much just random happenstance is there that kind of leads us toward a different path from just, you know, people in this, in this building colliding in different ways?

• 34:12Speaker 01

  Oh, interesting. Yeah, we would have all been using LSTMs or something. I mean, I guess, maybe it's like, okay, you ask the same question about like, okay, if somebody hadn't invented like an internal combustion engine, would we still be using steam engines at this point? I mean, I think someone would have, would have come up with something like transformer.

• 34:39Speaker 02

  From my vantage point, like over in London, it did feel like you were circling around the, there was like neural GPU, there was like, which was also like, okay, the key thing is we're not gonna have an RNN anymore, we're gonna paralyze, but it's not just gonna be a ConvNet, we're gonna have. Have like a notion of like the like depth is going to be some kind of function of your sequence length. That that was an idea in neural G P U. It didn't quite work out, but it was like the key idea of like get rid of R N Ns felt like that vibe was there.

• 35:08Speaker 01

  Yeah. Yeah, yeah, right, because they were the all this work on con connuts running around. Everyone wanted to kill the

• 35:17Speaker 02

  Yeah.

• 35:17Speaker 01

  The L S D M. And yeah, tension had been floating around from from the translation models. So it's kind of needed to come together.

• 35:30Jacob Effron?

  I love that. Also, kind of on the, you know, on the on the culture side, I think one of the interesting things, I understand Google works is basically there's like this bottoms up compute allocation, right? Of folks, you know, getting to do different projects and convincing other people to kind of come and allocate compete for that. And obviously, that's one model. There's other places that are like, we're going to go all in on like one thing and much more top down on the compete side. How do you think about like the trade offs between those two models?

• 35:54Speaker 01

  Yeah, I mean, we've we've been through both, or I've been through both, I guess. Google Brain, when I was at Google previously, was, you know, mostly bottom up, you know, as you describe. And then Deep Mind has been mostly top down.

• 36:09Speaker 02

  Yeah, it's a bit more top down.

• 36:10Speaker 01

  A bit more top down and, you know, different different philosophies and, you know, there are there are pluses and minuses to to both. I think, you know, top down can be good for getting people to to collaborate and getting like larger, larger training runs working. But, you know, but bottom up, I think, is also great for collaboration because then, like, OK, you bring someone out new on your project. It doesn't mean you have fewer resources per person. You have more total resources. So, you know, so that that's great. And and that there are so many abstraction breaking ideas that there's no great way to categorize them.

• 36:53Speaker 01

  So if you're saying, OK, this is the compute for pre-training and this is the compute for post-training. Well, OK, you've got something completely different that doesn't fall fall into those things nicely. And then then it's then it falls falls between the cracks. So we're we are bringing back a good measure of bottoms up because I think that's super important.

• 37:18Jacob Effron?

  Yeah, it feels interesting balance to strike.

• 37:20Speaker 02

  Yeah, like, I guess, like.

• 37:23Speaker 02

  I worked at OpenAI briefly and I did like the way that concentrated bets were made and that like obviously paid off like well for them in certain areas and that was nice. I also liked the way concentrated bets were made at DeepMind, especially kind of like, yeah, I do think there was a bit of a vision sometimes for like, like say AlphaGo. I was asked to join the AlphaGo team right at the beginning because they needed an engineer and I was a research engineer and I just didn't get it. I was like, why would anyone care about a board game? I don't get it. So you do really need like good research leads that have vision to drive these things. You can't expect everyone from a like that doesn't have always the best like vantage point to like know what's where the impact is going

• 38:05Speaker 02

  to be. Right now within like, for example, thinking as like the area of research that we kind of work within, it's like incredibly important. We have a reasonable non-trivial investment in like just bottom-up research where we're really not dictating anything. And then it's just a fun process of, it's kind of a fun meta-research process of like, how can we make these people maximally efficient? How do you make your baselines lightweight? How do you make these signal bearing? How can people move as fast as possible? And that's very important. And then it's just very important that it can't all be that, it needs to be, and here is a mandate of top-down bets that we have to deliver on. But it's always humbling, like, I think we have a very good like scope of like what everything is happening and we get a sense of like, here are the areas that are going

• 38:48Speaker 02

  to be really important. And usually you get humbled by the bottom-up research of like, a thing that you weren't even thinking about ends up being way more impactful than you thought. So just like always keeping that running is super impactful.

• 38:59Jacob Effron?

  Yeah, reflecting back on the past decade, like are there, you know, some of these like inflection points or these like decision points where it's maybe like, you know, hard decisions, 5149 that like ended up being kind of super impactful?

• 39:10Speaker 02

  I mean, I think for both of us, like going in on large language models is good.

• 39:14Jacob Effron?

  Yeah, I'd say that was probably a good call.

• 39:16Speaker 02

  And it kind of seems obvious now, but it kind of was, at least, I mean, I found it was like in a state of being not obvious to most people, but obvious, I felt like to a small number of collaborators that like this was definitely gonna be a big thing. You kind of had to go against the grain for a while.

• 39:36Speaker 01

  Yeah, that's true. There was definitely a time people were not excited about language models.

• 39:42Jacob Effron?

  Hard to remember now, it feels like decades ago.

• 39:44Speaker 01

  I mean, it always seemed like the best problem on earth to me, but you know, that's.

• 39:49Speaker 02

  Yeah, it was kind of like the deep learning people at the time maybe thought machine translation was still a bit cooler. Or computer vision.

• 39:56Speaker 01

  Yeah, vision was exciting for a while. I don't know. Why was everyone into vision?

• 39:59Speaker 02

  I don't know. I guess like the ImageNet thing.

• 40:02Speaker 01

  There's like a picture of a cat or something like that.

• 40:05Speaker 02

  Oh, that was that good one. Yeah, the cat. Did you work on the cat thing?

• 40:08Speaker 01

  No, no, no, never, never actually did much in vision. I've run them in this.

• 40:14Jacob Effron?

  And now you've come full circle on these Gemini 2.0 models. I see all these demos that are like vision based and you're kind of doing cooler things in vision now than certainly identifying cats.

• 40:24Speaker 02

  I mean, it felt, it felt like, I think even almost every early LLM person, world model is on their mind, really. Like, I actually don't feel like the early LLM researchers were even really from a language oriented background. They weren't really like linguists. It was more, I don't even feel like understanding language was really part of the motivation for that early group. It was like, train unsupervised learning at scale, do language first because it's the most like knowledge compressed, but then gobble everything up into like a big generative model and like understand everything. And it's very, very cool to see that just like continually proving out. Yesterday, we just like launched native image generation. It's amazing. It, you know,

• 41:06Speaker 02

  I think a lot of image generation right now is just like focused purely on getting like absolutely maximally aesthetic images, but also having, native image generation allows you to like really do a lot more with images, understanding, editing. You can have like interleaved sequence of images and text. And yeah, once again, it's just. Train a generator model on lots of data to arrive at that.

• 41:29Jacob Effron?

  You guys are obviously both big believers in like these models becoming more and more general purpose. I guess the question some folks are asking is like you think about domains, you talked about healthcare earlier, no? Like what, you know, the model that ultimately is going to be like our AI doctor, like is that just some, you know, is that just a continuation of what we're doing in some giant model? Is there like a healthcare specific version of the model that ends up being released that is like, you know, only has some set of data, you know, inputted into it, or they're just a bunch of guardrails or like, paint that picture for me of what you ultimately think like our, you know, AI doctor or AI, you know, biology researcher?

• 42:02Speaker 01

  Yeah, I don't really think you, you know, would need very, you know, specific, you know, test specific models for something that high value because, you know, you probably pay like $1 a token for talking to your doctor. So it's like, wait, wait, wait, cheaper, cheaper at this point.

• 42:28Jacob Effron?

  So the only reason for task specific models is price. And so if there's things that you wouldn't pay a dollar a token.

• 42:33Speaker 01

  Yeah, yeah, like you need, yeah, something to analyze, you know, vast quantities of data for marginal value, then maybe you want something.

• 42:46Speaker 02

  Yeah.

• 42:46Speaker 01

  Keeping test specific.

• 42:48Speaker 02

  There's always this notion of like, there'll be tons of like negative transfer out there. So like compartment things. I don't really feel like that has ended up being the case. If it could be measured, then that's a good reason to compartmentalize, like models. If it's, if there's no negative transfer, there's positive transfer, then just have one big model. That's like, I think, my personal philosophy. This is not like a thing which people have like uniform agreement over though. It's like a continuous active area of research. Like how much do you want to like specialize and spin off these expert models? But yeah, from the way I see it, it's just very simple. If there's positive transfer, put it in the same model. As long as it doesn't then become too expensive to serve.

• 43:27Jacob Effron?

  You know, obviously you guys have been at the cutting edge for a while. What's one thing you've like changed your mind on in the past year?

• 43:32Speaker 02

  I feel like timelines have shifted forward and I don't mean that like in a vague sense. Like I think the rate of progress.

• 43:44Speaker 02

  Is much faster right now. I felt like a year ago, there was obviously the field is advancing, but whenever you have a new paradigm shift, it creates this sudden acceleration. And also, actually, no, this is a pretty good one. One thing I've changed my mind on, like my mental model of like the propagation of information and how like, how people like adopt a scientific advance is completely changed. So when the transformer came out, I think, so I was over in London at DeepMind. People thought it was a cool paper, were a bit suspicious. I eventually implemented it in our like code base over, over the like holiday break, actually. But it was like three months after the paper had come out, I eventually implemented it

• 44:29Speaker 02

  and tried it for like language modeling. But it was like not really getting picked up. I eventually then collaborated with someone that wanted to use it for reinforcement learning. But I'm going to say really, it was like from paper coming out, maybe it was like six to nine months before it was like, you saw just transformers dotted around all areas of DeepMind. And that's within, like, we're all within Alphabet and it's like much easier to propagate information. I would say the, the, the speed at which the field has like picked up this test time compute paradigm, you have like, many labs have already trained and released models that are like looking very good, exploring the space. That was like very surprising to me. The fact that these things can like, if you make an announcement, say, and like, this

• 45:12Speaker 02

  is important, the fact, and it's just like a blog post or something, and then you'll have people that are able to like, make breakthroughs in that space and release models in order of months. That is like, that was a wake up call for me. There's a lot more compute and a lot more smart people working in AI. I often think of things in a bit of a rose tinted glasses way and think about like 2016 or something. And I think, well, we're very smart and we're very creative. People are very, very smart and creative now and they have way more compute and there's way more of them. And so if anything is going to like, be very impactful and then it can just like spread and like this idea can like spread all across, all across the world and people act on it, which is kind of crazy.

• 45:53Speaker 01

  Yeah, that is, it is kind of crazy. And just the amount of compute out there that, yeah, that like now, you know, whatever, a kid in the garage has more compute than was necessary to invent the transformer. So like. You know, um, yeah, I mean, people always worry about, oh, just how much compute do they have, but you know, it is, it is definitely possible to make, make, make breakthroughs, um, you know, with, uh, with way less compute than, than you would imagine.

• 46:27Jacob Effron?

  Yeah. No, anything you've changed your mind on in the past year?

• 46:29Speaker 01

  Um, I mean, I've, I've been continuously, uh, impressed with, uh, you know, with, uh, the success of RL, you know, it's, I, I'd never really worked with it much before, but oh, that's, that's actually pretty good.

• 46:46Jacob Effron?

  Well, I mean, you kind of alluded to this, Jack, like obviously, you know, I think in the, uh, you know, in the reaction to DeepSeek and all these kind of models that kind of fast followed in the, in the test time compute space, um, going forward, like, do you expect the open source models to be able to keep up as, you know, with, with each subsequent generation of these, of these models? It obviously seemed to happen faster than a lot of people would have expected.

• 47:04Speaker 02

  Yeah, that's actually something I'm changing my mind on. I do feel like the open source, like the, the ability for open source models to stay very close and competitive with the frontier is persisting. I actually thought, uh, we were getting a, maybe a false sense of assurance that it's, it's happening because maybe, um, maybe it felt like it was converging, but then, but then these things can pull away again, but actually that seems to have been very impressive. I, I'm really, really impressed at the like performance of Gemma 3 that just got released yesterday. It's amazing. It's completely incredible. The team did a really good job. Um, and other open source models, they've, yeah, so, uh, the, you know, DeepSeek V3 was a very good model when they released it.

• 47:48Speaker 02

  Um, yeah, so it seems like people are very like passionate in the open source space and they're very creative and smart and they have compute, so I don't really see why they wouldn't be able to continually innovate. Yeah, what do you think, Noam?

• 48:05Speaker 01

  Yeah, I mean, it seems like the time gap between closed source and open source has been shrinking. You know, I think that the technology will continue to accelerate. So, I mean, it could be that the quality gap will be large and the time gap will be very small. But yeah, we'll have to see how it plays out. It's super exciting to see all of these companies getting great results.

• 48:36Jacob Effron?

  Switching to some of the broader implications on all the AI progress we've been talking about for society. I'm curious, obviously, it seems like both of you in the last year have been impressed with the power of RL. You've been surprised by the pace at which we've scaled a lot of this test time compute.

• 48:51Jacob Effron?

  Have you changed anything in your own lives, like based on this, probably more clarity or belief that a lot of this AI-driven future is coming? Like, you both have kids. I know, like, anything that you've adapted? It sounds like, Noam, I know you don't clean your garage, but I guess that was prior to this year as well.

• 49:08Speaker 01

  I didn't actually do that very much.

• 49:10Jacob Effron?

  It does make for a good podcast.

• 49:13Jacob Effron?

  You've thought about not cleaning it.

• 49:14Speaker 01

  Yeah, I don't worry too much about the warming, you know. We'll have AI to take care of the carbon stuff soon enough, anyway.

• 49:22Jacob Effron?

  Yeah, but I guess, you know, anything that you've thought about differently in your own lives or in how you think about the life your kids will have?

• 49:30Speaker 02

  AI in education is like, I don't think people are really talking about it enough yet. My son, like, under supervision, but he likes to talk to Gemini. It's actually insane, like, how powerful it is, especially if he can, like, he goes out to the garden. He, like, takes pictures of plants, takes pictures of lizards. And, like, he now has this, like, very accurate personalized encyclopedia, which can give him information. And do they adapt to that? Like, I wasn't sure how that would work. They do. My four-year-old son walks around, like, talking very detailed about, like, the plants. He'll use the Latin name. He's like, they absorb so much stuff, they're a sponge. I feel like I'm seeing kind of like... A type of education that I don't really think has ever existed for like humanity happening.

• 50:15Speaker 02

  Like AI and education is going to be incredible. He went to school and he was like, um, he was like, oh yeah, I caught a lizard to his teacher. And his teacher was like, oh, that's cool. And it's like, oh, that looks like a big lizard. He's like, no, it's not a big lizard. It's a western fence lizard.

• 50:31Speaker 02

  Like very particular about his type varieties of lizards. And he's like, and I also saw a blue tailed skink. She's like, what's that? He's like, it's an amphibious lizard that, you know. So he starts like kind of reeling these things out. And, uh, you know, that's just like, it's just obvious, but when you see it, but children are very curious. They're like sponges for information. And if you can like combine that productively with AI, I think that's going to be really incredible. I do feel like the next generation will just seem like Spartan people. That's what I'm feeling hopeful about. Know of anything you've changed?

• 51:04Speaker 01

  Yeah, it's, uh, it is extremely hard to predict what the, uh, what the future will, uh, you know, will be like. We will, you know, we'll all do our best to, to, uh, to make sure, uh, you know, it's, uh, that, uh, you know, AI will be, uh, safe and, uh, and, uh, and beneficial. But, you know, it, it does, um, you know, it does make you think that, hey, you know, what, what I do now really, you know, really, really matters, you know, like, okay, we, we don't know if, uh, if, uh, you know, hu- human labor, labor will be, you know, like, uh, materially necessary, uh, i- in the future.

• 51:46Speaker 01

  But, uh, that, that just means, A, you know, like, uh, you know, it makes, uh, makes more difference. If you want, if you want to make, uh, if you want to do something, uh, that matters materially, go, uh, go do it now. And then other than that, like, uh, you know, just, uh, Try to be a good person, you know, whatever you find, find spiritually meaningful, you know, go do it because that's that maybe that's, you know, the purpose of humanity

• 52:21Speaker 01

  in the future, you know, may not be about, about providing for physical needs. So, you know, we've got to, you know, figure out where we, you know, where we find meaning in the future, but we'll have plenty of time for, you know.

• 52:43Jacob Effron?

  Yeah, well, especially if your mom's already having a deep philosophical conversations with the models, maybe we'll, we'll be able to reason our way to, to that. But I am struck, I mean, I feel like some other people have come on this podcast, you know, Bob McGee with restructure AI was like, look, you know, the humanity is always going to have a role in, in asking the questions. The models will go off and do things. But I think to our conversation earlier, it's like, I think the big question is, will, will, you know, you know, people always be the best people to ask questions or, or, you know, will the, will the models actually ask better questions over time? And obviously has a ton of implications. I feel like every generation thinks they're living through the most important moment in history, but it does, you know, I guess, you know, you're biased when you're, when you're going through it, but it does feel like we are, we are certainly in that.

• 53:19Speaker 02

  Yeah. And I think sometimes like a technological advancement kind of scares people and it is, people have right to feel trepidation at this stage as well. But there is like, I mean, this isn't such a good example, but like even like the introduction of the, like television, people are like, oh, is this going to make us all just like lose our attention span? Are we going to like completely lose the will to like go out and walk outside and like have friends and things? That was like a thing people freaked out about. I think obviously now we know that was like unnecessary. There was like, it was a small piece of technology that was entertaining, maybe even has provided like a net positive to society. I think it like kind of went okay. Right now it's like, there's like, it's, it's like that, but kind of on steroids. There's like very strong signs of how this

• 54:04Speaker 02

  helps us. There's like very good reason to be concerned about how this could not help us. So it's kind of like, you, you have like very demonstrable reasons that have already proven out how it's helping us. You have very like concrete arguments for how this could not go well. And I think that makes it a very interesting time, too.

• 54:22Speaker 01

  In some ways, you know, we have less kind of.

• 54:28Speaker 01

  Meaning than than in the past, in the sense that like, you know, in the distance past, everyone was like at the brink of starvation and you're like, okay, I have meaning in my work because I have to go work hard today and get some money so my family doesn't starve tomorrow. And like today, we're living in America, nobody's family is starving tomorrow if you don't work hard. So that, you know, okay, that's that's less meaning than we used to have. And we've found other, you know, other, other sources of meaning. So, you know, future, more of that in the future as, as hopefully, you know, AI improves our physical situation.

• 55:10Jacob Effron?

  How worried are you both about AGI risks?

• 55:12Speaker 02

  I would say moderately, yeah. It, it, it is hard. It's often difficult to find examples of creating something which becomes far more intelligent than its creator, but then like still like acts in predictable and useful ways for its creator. And I think that that like class of argument is concerning. There's also just like more practical, like kind of AI and society implications that we've kind of touched upon, but like making sure that AI is like constructive to the economy and the like, and the, and people can can offload their lifestyle. And we don't have like sharp changes in like the employment

• 55:56Speaker 02

  landscape and things. That's super important. Both of those are often on my minds. And then there's just very pragmatic things when we're always putting more capable models out. We're very excited as technologists to develop and ships things. But we also, I think we have a pretty good balance of like, of then internally having another group that's going to be thinking about this much more holistically. And like, how can we make this launch safe? What are some unintended consequences? Which I think has been pretty good and, and is like super important. So I'm, I'm glad that happens.

• 56:32Speaker 01

  Yeah, I mean, we, I, I, hey, I, I agree, we need to, yeah, we, I mean, I'm not afraid, but I, uh, but we, we definitely need to be working on, on all the safety aspects of all of that. And there are examples of, we, you know, we, we, we create something that, that becomes smarter and, uh, and more powerful, but, you know, we have kids.

• 56:58Speaker 01

  They're smarter than us and.

• 57:01Jacob Effron?

  Before, they're already detailed encyclopedias of amphibians.

• 57:06Speaker 02

  Yeah, but then they become teenagers.

• 57:10Speaker 02

  And then you solve the alignment problem.

• 57:11Speaker 01

  Yeah, yeah, but so, I mean, hopefully if we, uh, respect our parents and, uh, you know, uh, treat, treat them well, the AI will, uh, uh, will, will learn.

• 57:23Jacob Effron?

  Except to have a lot of tokens on the internet of people being really respectful toward their, uh, their parents.

• 57:28Speaker 01

  Exactly.

• 57:29Speaker 02

  We have to stop pushing the robots over.

• 57:32Jacob Effron?

  Yeah, exactly, that, that, that feels dangerous. You know, one thing I did want to ask you about Gnome is, obviously, you know, uh, in your, uh, previous in-between Google stints, you'd spend a lot of time kind of building out character and you thought a lot about kind of that product space of what like, you know, uh, AI companions and the ability for folks to, to chat with all sorts of different kinds of people. What do you, you know, what do you feel about where that space is today? What kind of like problems do you feel like we still need to solve there?

• 57:56Speaker 01

  Yeah, I mean, it's interesting because I, like, the, the reason I, main reason I left Google to start character was I thought that the biggest thing that, um, that, that the LLM industry could use was an application where like anybody can go and, um, and, uh, and interact with LLMs and use them and, you know, discover use cases that, that were, that, uh, that were good for them. And, you know, because this was before, you know, be, be, before ChatGPT launched, before, uh, uh, before Gemini launched. So, um, I mean, mission accomplished, everyone's out there talking to LLMs now, um, but, um, uh, which is, which is kind of different from how, uh, how things are now.

• 58:41Speaker 01

  I guess the other thing was, um, you know, going into character, we were not. Really focused on, hey, this is going to be an entertainment product or something or something else. We kind of just went in with an open mind and we're like, okay, you know, we're going to just put this out there in a general way. We're going to help help people conceptualize this as, you know, okay, this thing can, you know, take on different personas, meaning it's a, you know, it's a very, very general technology. See what you can, what you can make it do. And definitely, you've found a lot of people are using it

• 59:21Speaker 01

  for entertainment. I think partially because like, at that point in the technology, like, okay, like, nobody's figured out how to make this thing not hallucinate. So people are going to use it for, you know, for applications where hallucination is actually a feature. Right. You know, such as, such as entertainment. Then, you know, that's, you know, I think, I think that's worked pretty well. I know a lot of people like, like using character. What, wait, what were you asking?

• 59:52Jacob Effron?

  I guess like, what do you think the future of that? Obviously, like, there's this early behavior, a lot of people.

• 59:56Speaker 01

  Yeah, yeah, yeah.

• 59:57Jacob Effron?

  What does it look like like five, 10 years from now?

• 59:59Speaker 01

  That's a good question. I, I, I, I do not know. I mean, I do think, you know, people are going to, I think people will always want relationships with, with humans because it, you know, it's spiritually more, more meaningful. But, but, you know, I think people will, you know, like having AIs that are kind of more in, in human form for, you know, for, you know, for, for things they want. I mean, like, imagine, you know, you just got, get elected president, you get your, your piece in the cabinet to like advise you like wherever, you know, wherever you go. I mean, that could be cool.

• 1:00:42Jacob Effron?

  You have a whole AI cabinet.

• 1:00:43Speaker 01

  Yeah, you got your own personal AI cabinet. You get your whole.

• 1:00:45Jacob Effron?

  A good AI summarization of all the secrets.

• 1:00:48Speaker 01

  Yeah. Yeah, that we're like, or like, maybe you're, you know, you're CEO of your own AI company, so you get a lot more productivity. I guess that's maybe in a lot of those cases, it's less about the personality and more about the productivity. But I think to the degree that people, people like interacting with something that feels human, probably, probably we'll see a lot of kind of AI that feels feels more human in various ways.

• 1:01:24Jacob Effron?

  And you think progress in that space is just about the models getting better? Or is there like a whole other set of just like, you know, human computer interaction, product questions that need to be solved?

• 1:01:33Speaker 01

  It's a good question. I mean, I think the model's getting better is, is, is pretty big. And but then, yeah, then, yeah, part of it is, you know, for whoever is, you know, running the application to, you know, to decide, okay, what are we going to, you know, what are we going to let people do? I think, I think users are probably pretty good at, will be pretty good at specifying, okay, what do I want for, for an interface? And it'll be mostly about, okay, do we want to let them specify that?

• 1:02:15Jacob Effron?

  Totally. Well, look, both of you, you know, fascinating conversation. We always like to end with just a quick fire to get your, you know, take on some overly broad questions that we cram into the end.

• 1:02:24Speaker 01

  Sounds good.

• 1:02:26Jacob Effron?

  So maybe to start, you know, what do you feel like is overhyped in the AI world today and what's underhyped?

• 1:02:32Speaker 02

  I personally feel like the ARC AGI eval is overhyped.

• 1:02:37Jacob Effron?

  Well, that's very spicy.

• 1:02:38Speaker 02

  Yeah, I think, I don't, I think actually.

• 1:02:43Speaker 02

  Maybe the progress has been quite slow because a lot of researchers just don't feel particularly inspired to do like kind of these very specific types of puzzles. We did a lot of that in like 2015, 2016. And then we kind of felt like, yeah, you can, if you know the puzzle domain, you spend a lot of time like fixing all the like actual bottlenecks, like maybe acting on these large grids is like a little bit finicky and stuff. And then, and then you make a lot of progress, but, but then you don't necessarily continue on like building something that's really AGI and useful. So I feel like I personally had a transition from going to from like all these like synthetic tasks, like to then going to just like, just model natural language. And I felt

• 1:03:26Speaker 02

  like that was a dragging in, in that direction was like a much more AGI thing in, in the long run.

• 1:03:33Jacob Effron?

  Anything else that comes to mind for you to overhyped or underhyped?

• 1:03:35Speaker 01

  I don't know. I think, I think AGI is underhyped.

• 1:03:39Speaker 01

  AI LM's are just still, still massively, massively underhyped. I think people are, you know, I don't know. People are still like thinking about it like it's only like, you know, that it's going to be about some, some silly trillion dollar products, you know.

• 1:03:56Jacob Effron?

  I heard you say on another pod that a trillion wasn't cool anymore, quadrillion was.

• 1:04:01Jacob Effron?

  Exactly. You gotta put the Dr. Evil up. I obviously, you know, it'd be a gross misallocation of societal resources to take you away from building models to like build applications. But I am curious, like if you were to, if we were to say today, go build an application, like what, what do you think are the most interesting? You've talked about education before, but any other areas that come to mind that you think would be fun to, to go build, you know, apps on top of these models?

• 1:04:22Speaker 02

  Well, I do think, I think it's actually very cool how many like apps that have been trying to like break into this agentic space. And people then like expose it and say, oh, this is just a wrapper around a known model. But like, there seems to be a lot of value in like actually having the right like app experience if you want to actually have a model do something useful for you, like act and do something useful for you. So I feel like, yeah, I feel like, okay, and in the agentic space, I think code is very crowded now. But I do think there's a lot of other things that I would find it useful for a model to automate for me that goes beyond maybe like a chat experience, but it's actually going to be going out and doing useful things.

• 1:05:07Speaker 01

  Yeah, I mean, I'd say code is underhyped.

• 1:05:12Speaker 01

  I think it's huge. I think it's huge because, you know, for one, humans aren't even that good at it. Things like code and math are not super designed for that. And then it's one of these things that will help self accelerate, you know, if build an automated software engineer, researcher, and then it'll build the next, the next better AI. So yeah, the combination of, you know, of engineering and agentic, you know, something that can control surfaces broad, broad enough to do the job of, of an engineer. I think if I were to focus on, you know, on applications, that is what I would be focused on.

• 1:05:57Jacob Effron?

  How different will the infrastructure needs look for test on compute models, you know, versus these massive pre-training?

• 1:06:02Speaker 02

  You mean like in terms of like hardware?

• 1:06:04Jacob Effron?

  Yeah, in terms of the hardware requirements, you know, distributed data centers, all that.

• 1:06:07Speaker 02

  Yeah, I mean, it's a pretty rosy story, I would say, right? So if, if it turns out that building AGI becomes like mostly an inference problem, an inference problem that can be much more distributed than maybe like synchronous, large batch training that happens in pre-training, I think that means that we can be much more flexible with, with our compute. But yeah, it's going to mean we maybe don't, we don't mind the model training across data centers as much. Maybe it can be spreading out actors that are going to go off and get experience and things and send that experience back from like, like many, many different data centers. They don't all need to be like have very strong, fast

• 1:06:49Speaker 02

  interconnects. So that is also going to drive price down as well because then we might start to like really optimize towards such a setup, which is intrinsically cheaper. The cool thing is that Google, we, we kind of have like this code design link with the TPU team. So we're always like feeding them like our, like the profile of like how we're spending our compute, which allows them to tweak the chip design and the data center design, you know, within a couple of years timeframe, which I think is really, really like kind of motivating.

• 1:07:22Speaker 01

  The fact that you can be distributed, as Jack said, gets better. I mean, the thing that gets worse about inference than training is is that you lose a lot of the parallelism in the transformer. That, you know, that just naively using transformer, you end up, you end up memory bound, like looking at your

• 1:07:46Speaker 01

  attention keys and values, you know, for every token that you're generating. So, you know, so there's like a lot of great work to do in both attacking this from a model architecture perspective and from a hardware perspective, frankly. So, you know, to get ourselves

• 1:08:09Speaker 01

  closer to the point where we're, you know, where we're taking like the massive computational power of the chips we have and making ourselves able to like fully apply that to inference.

• 1:08:26Jacob Effron?

  So I want to leave the last word to you. Where can they go learn more about you and what you guys are doing?

• 1:08:30Speaker 02

  Yeah, well, you know, we have a new and updated flash model that applies thinking that's considerably stronger than the last model we released in January. It's out on the Gemini app. I would definitely encourage people to try that out and give us feedback. We have been incorporating developer feedback, user feedback into each model series. So like, I think that's a good, like, that would be one thing I would encourage people to do. With Jack's head.

• 1:08:57Jacob Effron?

  Yeah, well, thank you both so much, seriously. It's such a pleasure to be able to jam on all this with you.

• 1:09:01Speaker 02

  Real pleasure.

• 1:09:02Jacob Effron?

  Yeah.