Gemini's Next Frontier: 2.0 Flash, Flash Lite Strategy & Real-Time APIs with Logan K from Google Deepmind

In this episode of the Cognitive Revolution podcast, Logan Kilpatrick, Product Manager at Google DeepMind, returns to discuss the latest updates on the Gemini API and AI Studio. Logan delves into his experiences transitioning to DeepMind and the restructuring within Google focusing on AI. He highlights new product releases, including the Gemini 2.0 models, and their implications for developers. Logan also touches on the future of AI in text-to-app creation, the impact of reasoning and long context in models, and the broader industry trends. The conversation wraps up with insights into fine-tuning, reinforcement learning, vision language models, and startup opportunities in the AI space. SPONSORS: Oracle Cloud Infrastructure (OCI): Oracle's next-generation cloud platform delivers blazing-fast AI and ML performance with 50% less for compute and 80% less for outbound networking compared to other cloud providers. OCI powers industry leaders like Vodafone and Thomson Reuters with secure infrastructure and application development capabilities. New U. S. customers can get their cloud bill cut in half by switching to OCI before March 31, 2024 at https://oracle.com/cognitive NetSuite: Over 41,000 businesses trust NetSuite by Oracle, the #1 cloud ERP, to future-proof their operations. With a unified platform for accounting, financial management, inventory, and HR, NetSuite provides real-time insights and forecasting to help you make quick, informed decisions. Whether you're earning millions or hundreds of millions, NetSuite empowers you to tackle challenges and seize opportunities. Download the free CFO's guide to AI and machine learning at https://netsuite.com/cognitive Shopify: Shopify is revolutionizing online selling with its market-leading checkout system and robust API ecosystem. Its exclusive library of cutting-edge AI apps empowers e-commerce businesses to thrive in a competitive market. Cognitive Revolution listeners can try Shopify for just $1 per month at https://shopify.com/cognitive CHAPTERS: (00:00) Teaser (00:54) Introduction and Welcome (03:56) The Future of Text App Creation (05:15) Multimodal API and Real-World Applications (10:37) Sponsors: Oracle Cloud Infrastructure (OCI) | NetSuite (13:17) The Evolution of Long Context and Reasoning (19:19) Vision Language Models and Passive Applications (21:50) New Launches and Future Prospects (Part 1) (28:35) Sponsors: Shopify (29:55) New Launches and Future Prospects (Part 2) (31:55) Flashlight Models and Cost Efficiency (34:36) Pro Models and Frontier Applications (39:52) Evaluating AI Models (48:57) Fine-Tuning and Reinforcement Learning (51:42) Opportunities for Startups (55:52) Conclusion and Final Thoughts (56:59) Outro SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://linkedin.com/in/nathanlabenz/ Youtube: https://youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk PRODUCED BY: https://aipodcast.ing

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Logan Kilpatrick?

  We sort of released the experimental first iteration of Gemini 2.0 flash back in December. Today, we brought Gemini 2.0 flash, an updated version of it into production so that developers can actually continue to build with it. We announced pricing 10 cents per million input tokens, 40 cents per million output tokens, which is, I think, a huge accomplishment for us to pull that off. We're going to have the world's best coding model at Google, and I still believe this deeply. And I think like Pro is going to be that model and a bunch of the reasoning work that we're doing is going to be that model that continues to push the frontier for us. The world needs a platform in which it's hosting all of the sort of publicly available benchmarks and sort of leaderboards and stuff like that.

• 0:40Logan Kilpatrick?

  I find it incredibly difficult to just like navigate and get a snapshot of like, how good is this model? There's like 20 random benchmarks here and 50 random ones here. They're all split out over the place and it's just like hard to keep track as a developer.

• 0:55Nathan Labenz?

  Logan Kilpatrick from Google DeepMind, Product Manager of the Gemini API and AI Studio. Welcome back to the cognitive revolution.

• 1:03Logan Kilpatrick?

  Thank you for having me, Nathan. I'm excited. I'm hopeful that I'm getting close to the record for the most times on your podcast. So I appreciate you for always.

• 1:10Nathan Labenz?

  I think this might be setting the record at four if I, if my count is correct. So yes, congratulations. That's rare air and well deserved. So it's launch day. We'll get to everything that you've launched and what we should be thinking about building with it.

• 1:28Nathan Labenz?

  Quick little detour though, before we get there, you're now part of DeepMind. So, you know, Google obviously is a vast company and is continuing to, I don't know, align, restructure, streamline itself to focus more and more on AI. What's the story from the inside on what it's like to be at DeepMind now specifically?

• 1:49Logan Kilpatrick?

  Yeah, I'm super excited about this. So we've been, you know, I joined Google 10 or 11 months ago. Literally from day one, it's been a deep collaboration with DeepMind. DeepMind's gone through all these evolutions over the last few years, transitioning from an organization, doing fundamental research to sort of actually productionizing models. And then within the last three months with the Gemini app moving over and then AI Studio and the Gemini API is now actually an organization that like end to end does the research, creates models, and then actually brings them to products inside of Google. And I think that's been a shift for them, but from My personal vantage point, like I think this is the thing that makes the most sense. Like being really close to research and we already were really close to research through this collaboration we've had,

• 2:32Logan Kilpatrick?

  but removing as much friction as possible for us to to bring the researchers who like actually know how to bring, in many cases, like get the most capabilities out of the models, like bringing those two things together makes a lot of sense and it's going to be a ton of fun. So as an external person, like who doesn't care about Google reorgs, which is most of the world, the thing that you'll hopefully see is an acceleration of model progress, but also an acceleration of product progress because we bring these two teams together.

• 3:00Nathan Labenz?

  Well, it sure seems from my vantage point on the outside that everything is accelerating and we've had previews of some of the stuff that is now going general availability today over the last few months. And of course, there's been just one advance after another from DeepMind and others over the last few months. Looking back a little bit, what would you say are the customer success stories and or just like coolest apps that you have seen come online that have been built with the Gemini API in recent time?

• 3:32Logan Kilpatrick?

  Yeah, I think the thing that I'm most excited about and it also feels like we have the biggest opportunity here still is around all these like text to app creation softwares. And like, there's a bunch of examples of these, like Bolt new just went live, I think yesterday with Gemini support. Cursor has Gemini support now and is using 2.0 flash. Hopefully we'll see others like lovable and V zero, et cetera, et cetera, have have that support as well. If you look at just like the economics of running those products, it's extremely cost intensive, especially as like the number of people who know that you can actually do that use case today have put in a text prompt and like get a basically working app slash website for free essentially is like a very small number of people.

• 4:16Logan Kilpatrick?

  And it feels like that's like this new frontier use case slash product paradigm that I think is going to be picked up across like all the big players are going to do this. But I think also there's going to be a ton of startup activity in this. Like, how do you just build domain specific software for people without those people actually having to know how to code? So I'm really excited about that use case and I think we have a lot more model progress to still do to like become the world's best model at doing coding. But I think even for 2.0 flash and 2.0 pro from where we were six months ago, like it's just an incredible amount of progress. And I think trying to keep pushing on progress in the context of flash without like increasing the the price in any dramatic way, I think has been the biggest win for us. Like I've seen tweets of like the LLM usage

• 5:00Logan Kilpatrick?

  cost for some of these startups and you know, you can imagine what those in a time on the order of like 40 or 50,000 dollars a month. And you can imagine what those costs would be with flash. It's like probably a thousand dollars or less or something like that. It's like a 40x cost reduction, which is just crazy. So that's one of them that I'm excited about. I think we also previewed in December, the multimodal live API, which allows you to do this sort of collaborative, real time conversational video text interface with the models. And that feels like it is getting us closer to the future that I think we are sort of all been promised with AI, which is this like co-presence

• 5:37Logan Kilpatrick?

  that's able to see the things that you do and interact with the services you do. So I'm really interested and I've gotten like the entire spectrum of outreach from people who are using it to help them do coding to like people who aren't developers who are just using this product experience and they're blind and they're actually just, you know, trying to navigate their daily lives using this tool. And it's like crazy that you know, the product that we're building sort of as a demo for developers to drive adoption of the API is actually helping people who are just trying to live their daily, daily lives of this product. So I think it speaks to where we are in the adoption of this technology, which is like everyone is sort of waking up and like trying to find the best way to use these tools, which is really interesting to see happen.

• 6:19Nathan Labenz?

  Let's put a pin in the lovable and Bolt discussion because I actually, you're, you're teaming me up perfectly because I think immediately after this episode, the next two episodes are going to be with the founders of those two companies.

• 6:32Logan Kilpatrick?

  I love that.

• 6:34Nathan Labenz?

  I am, we're calling it software supernova because it does seem to be, we're kind of mid-transition, mid-tipping point right now, I feel like, where it is becoming like pretty realistic for people that don't know how to code to create at least basic full stack applications. Obviously, they're not going to create, you know, enterprise platforms just yet. But you know, the models continue to advance. So it feels like we're in a very different world quite soon with. Products and paradigms like that. In terms of living my daily life, I have tried both the AI Studio version, which is the desktop experience where I can share my screen with the multimodal Gemini. And I've also been using on the mobile and I don't know why, but for whatever reason,

• 7:18Nathan Labenz?

  OpenAI only has that experience enabled on the mobile app as far as I know. So I've been trying both of them. This is too stupid, but it's also too good of an example. So I got my kids the Nintendo Switch for the holidays and we're going through historical video games, right? Cause I feel like they're young and the modern games are like, I feel like too overstimulating. And plus like, let's burn through the old catalog first. We'll work our way up. So we're doing, you know, original Nintendo games, Nintendo 64. So we're playing Mario 64 for the original Nintendo 64, which is like this open world game where you like go around and hunt stars and whatever. But the frustration for me is I often don't know what to do. So I've been sitting there with the, the advanced voice mode on and sometimes, you

• 8:02Nathan Labenz?

  know, also like showing it the screen and just telling it what level I'm in and having it tell me what to do in the game. Like what is my objective? You know, where should I go to find the star? And my kids are really getting used to this. And my, even my one year old, you know, sometimes now like comes and wants to take the, the phone and he's like, I try to talk to AI. So it is, I think, and for seniors too, you know, my grandmother, I think about all the time, this switch from a you type to it and it gives you, you know, text back to it really can be co-present with you everywhere. I feel like I've only still, you know, only dipped a toe in that world, but man, does it feel like a very different world for people that are not anchored to their desk all day, you know, doing computer work.

• 8:46Nathan Labenz?

  It's like, wow, that could really just be, I mentioned putting it in glasses too. I'm sure, you know, from the company that made Google Glass, I'm sure there's a lot of thought going into that sort of thing. So.

• 8:57Nathan Labenz?

  Yeah, that's my monologue.

• 8:59Logan Kilpatrick?

  Yeah, I feel like and my guess is we're actually going to see a lot of this, you know, I think if folks have been watching this closely, like, you know, it takes time for this progression to actually happen. I think like text was the best example of this. Like, it was kind of a toy demo a year and a half ago. And it's like now at large scale, like text LLM applications are like broadly being used throughout the world at billion user scale. I think this co-presence, like there's, it's going to take us like to be sort of humble of the place that we're in today. Like, we're going to have to iterate on the API. We're going to have to bring the cost down. We're going to have to make it so that it's like actually something that developers can build because you could imagine the like co-presence cost of having AI with you all the time is like probably pretty expensive. Like, there's a reason that we'd limit the sessions to only 10 minutes right now because there's a whole lot of challenges to like scale this up beyond that and have it maintain memory and state and context of all the things that you were just talking to it about.

• 9:55Logan Kilpatrick?

  But to me, it's like very clear that that's the direction that we're going to go in. And I think it'll be interesting. It does feel like everyone, you know, again, like two years ago was talking about like, oh, everything's a wrapper. There's not a lot of value to be created. And it just feels like that continues to be wrong. Like, it feels like there's a new thing. And then like, all of a sudden, like, all of these new things that were not possible before can now be created, which continues to just get me excited about like the future and also like making sure that we're enabling those next things so that people show up and create the experiences that don't yet exist today. So it's going to be fun. I'm excited. And hopefully, yeah, send, send over feedback as you keep using the real time mode.

• 10:36Nathan Labenz?

  Cool. Yeah, will do.

• 10:38Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 10:44Nathan Labenz?

  Just taking one more beat on the like, cool stuff with the Gemini API. Last time we talked about a couple of things I wanted to check in on the status of. One was really long context.

• 10:56Nathan Labenz?

  Another is just insane affordability, as you mentioned, like a huge drop compared to other options. And then there's a sort of unclear exactly how sort of natively the like video is being consumed, but you can just like, you know, feed video, audio multimodal inputs into the Gemini API as well. What would you say are kind of the the status of those things right now? Like, are you seeing use cases where people are like, 200,000 tokens is not enough? I really need to provide more and therefore like Gemini is the one thing I can use? Or, you know, cost wise, this just wouldn't be affordable if I didn't have this 10 cent

• 11:39Nathan Labenz?

  per million input token pricing?

• 11:42Logan Kilpatrick?

  Yeah, this is a great question. I had a long conversation with Jack Ray yesterday, who's one of the co-leads for the reasoning efforts. And he previously worked on sort of the long context breakthroughs with Gemini and enabling that from a research perspective and is now one of the co-leads with Gnome on the reasoning models. And we sort of opined for a long time about how it is funny that like, maybe the real unlock for long context ends up being reasoning because like long context is extremely impressive and like it's really useful and we do see people using it in production. And I do think there's cases where like 200,000 makes a lot of difference versus like a million or 2 million.

• 12:24Logan Kilpatrick?

  But like one of the inherent challenges is just like, how many things can the model attend to in the context window? And like, it works really, really well if you're just asking questions about a couple of things that are in the context window. But if you're trying to put together a thousand different things that are in the 2 million context window, it gets really hard to do that just because of the inherent nature of how the models are trained and set up. I think reasoning is maybe where this starts to change. So I think like us having this really long context window where the models can actually just like think through and perhaps in the future, like use tools and like bring information in and out of the context window, I think is where long context is going to start to make a lot more sense and like truly become like an enabler.

• 13:05Logan Kilpatrick?

  That's where my head's at from a long context point of view. I think video stuff still, yeah, video, audio, still happening natively, image still happening natively inside of the models with the 2.0 release. We showcase some of the like next steps of this native multimodality, which is the models actually being able to output images and audio and that's available to early testers. I don't know if you're in the early access program or not, Nathan, if you've played around with it yourself.

• 13:30Nathan Labenz?

  I have.

• 13:31Logan Kilpatrick?

  It's, it's pretty good. I think we still, there's like more quality work that needs to happen still. But I think the, where this really starts to, and we're about to actually, at the time of this recording tomorrow, we'll roll out Imagine 3 in the Gemini API, which I'm super excited about. And if you sort of look at like, why do we still have state of the art image generation models when we know that the models can sort of natively have this capability themselves? Like, there's definitely this quality trade off in some domains where you trade off quality for world knowledge. And I think the world knowledge case is like, actually where this is really interesting. Like, there's a ton of models out there that generate like really, really pretty pictures and can do all types of cool stuff, but they lack this world knowledge that the Gemini models have

• 14:15Logan Kilpatrick?

  because it's this native capability that's coming as part of the training process. So I think there's going to be this whole new, you know, onslaught of, of use cases which like didn't work before because the models weren't smart. They were just good at generating images that we'll see happen with, with native image generation. So hopefully we'll get that out soon as we, as we continue to help climb on quality.

• 14:37Nathan Labenz?

  Cool. Yeah, that's interesting. I think your point about the

• 14:42Nathan Labenz?

  need to have long chain of thought in order to really fully take advantage of super long context is quite interesting. I have been noticing for myself in like application development.

• 14:57Nathan Labenz?

  I do find like, I want the hardest thought, right? So I'll go to the model that's going to think for me the longest. And then I sometimes have to like contort myself or contort my inputs to get it to fit into the context. But I do feel like, yeah, I, I wouldn't 10x dump context into a model that is going to like immediately jump to the answer. And I hadn't really put together why that might be, but I think that's a pretty interesting hypothesis that I could imagine. I, I look forward to dumping, you know, my full million token code bases into the flash reasoning pseudorev then later.

• 15:35Logan Kilpatrick?

  Yeah, I'd love to see if you have use cases that haven't worked well historically for long contexts. Like, I'm super curious if like also folks in, in the audience have, yeah, like you can do the compare mode right in AI Studio, try it with like 2.0 Pro with long contexts and try it with reasoning with long contexts and see whether or not like the, the extra reasoning steps actually make a difference. And my intuition is, is that it is going to, which, which is exciting.

• 15:58Nathan Labenz?

  Yeah, cool. Okay. I like that. Have you seen anything passive? This is another area where I feel like flash, you know, very, very minimally famously said more people should try to spend a dollar a day on flash. And, you know, that is a lot of tokens. And so it's kind of like, you need passive applications to, for most people to get there. So I've been thinking about, a couple of people have asked me about, could I use a visual language model to like monitor my factory floor for, you know, safety incidents or, you know, policy violations? Or my grandmother lives in a senior living community and, you know, they, the seniors like hate to wear their fall monitors. So that's a constant battle.

• 16:43Nathan Labenz?

  And they probably, a lot of them would, you know, to the degree that they have, and my grandmother does, have the ability to like make her own decisions on this, probably would, you know, accept like not having to wear that thing if, you know, there was another sort of visual monitor that could, you know, keep track of, of her state at any given time. Have you seen anything like that where people are just kind of truly passively, almost internet of things like sending signals into the API?

• 17:09Logan Kilpatrick?

  This is a great question. I think it is so core to my thesis of like what's going to happen for a lot of these domain specific. Cause like, and if you take a step back, like, how would people solve that problem today? Like, you would need to go and buy some custom software that does that, which probably is expensive. It might not generalize well. Like in my past life, I was a machine learning engineer and we did a bunch of stuff with security cameras and like what it would look like. How would you track someone moving from one frame where a camera is visible to another frame and how to keep object permanence of that person? And like, it's incredibly hard. Like it is not an easy problem to solve with traditional computer vision technologies. And I think vision language models like just do this task incredibly well.

• 17:53Logan Kilpatrick?

  And the cost basis is now with flash like so low. I think I haven't talked to anyone who actually has this in production, but I have to imagine that this is like the, the opportunity that people are going after. And it's not, it's actually not just the like bounding box use case and image understanding use case, which I think is really, really powerful. Like being able to know like, here's where an object is. And we have, we have a good demo of this in AI studio. Folks haven't tried the bounding box capabilities. If you go to starter apps, there's like, I think it's spatial understanding, or maybe it's just called bounding boxes. I forgot what the name of it is, but there's an example in there and you can put in images and like ask, you know, to identify the objects and it'll throw bounding boxes.

• 18:35Logan Kilpatrick?

  Pretty much just like you would get out of like one of those custom bounding box models that you could probably find on open source or something like that. I think these use cases just take time, but I also think my guess is as vision becomes more and more prominent, like we're going to see the whole like YC startup wave. Go after all of these ecosystems and industries where they're using domain specific vision models and not using like a general purpose model. And your cost basis is just going to be wildly different. And also you unlock all these use cases, which those models are just like not actually capable of doing. They're very, very rigid and like can't, you know, be fault tolerant in a lot of those cases. So I'm super excited about this.

• 19:17Nathan Labenz?

  So let's get into then what you're launching today. I saw a interesting tweet. You mentioned YC and the idea was basically that every time the frontier advances, the some, you know, all the, all the YC companies go and kind of see like, can this work for my use case? And the report was every time a new model comes out, like some subset of the current YC batch companies products start to work. And the others are kind of just like waiting, you know, and continuing to build all the other stuff with the expectation that they're going to get a model that's going to like, you know, tip them from kind of not quite working to working. So what are you launching? And, you know, to the degree you can speculate, like, what is it going to make work that wasn't previously working?

• 20:03Logan Kilpatrick?

  Yeah, it's a great question. I think one of the, and, and just to draw a broader point, I think one of the really interesting observations I've had is, is thinking about like, because of how much excitement there is about AI, how the like resource constraint has not made people think as deeply as they need to about this problem. And I think like, maybe this is me somewhat, you know, just because of like Gemini models, like being at the frontier of cost per intelligence, if you look at that as a ratio. But it is really like the, the YC companies are just like so well funded that like, we bring the cost of intelligence down some reasonable factor. And like, it actually in a lot of cases, like, doesn't move the needle for the startups, like just because they have millions of dollars.

• 20:48Logan Kilpatrick?

  And I think it's interesting to see what the outcome of that trend is going to be. And like, if I had to guess, actually, I think it gives a lot of power to these individual developers who don't have this like large amount of, you know, financial backing from like tier one VCs as an example, that can actually push the frontier of some of these use cases and capabilities, which I think is like a really interesting and cool phenomenon. But to answer your question specifically about what we're launching, it's a whole sort of suite of Gemini 2.0 models. So we sort of released the experimental first iteration of Gemini 2.0 flash back in December. Today, we brought Gemini 2.0 flash. An updated version of it into production so that developers can actually continue to build with it.

• 21:31Logan Kilpatrick?

  We announced pricing 10 cents per million input tokens, 40 cents per million output tokens, which is, I think, a huge accomplishment for us to pull that off. And then announced a preview of Flashlight, which is the smaller variant of the Flash model, which we'll intend to make available for production use very soon and the pricing for that. And then released the the sort of experimental variant of 2.0 Pro, which is the the most capable frontier model we have, joining rounding out the full offering with the Flash reasoning models. So now we have the the sort of reasoning Flash model. We have Flashlight, the smallest model, Flash, which is sort of the most, you know, performant cost trade off wise, and then Pro, which is the the most capable model.

• 22:15Nathan Labenz?

  And only the, well, wait, so let's go down the availability too, because you guys have high, in my typical work, which is usually focused on like proof of concept type stuff. I'm going to honestly love my life these days. I'd never have to really worry about, you know, the harder work of like making something actually production ready. And much more get to focus on, you know, that that easier, you know, faster ascent portion of make the proof of concept work. So when I go to the AI studio and I, you know, grab code and go do stuff with it, with one exception, which I'll mention maybe in a minute, I basically never hit rate limits. So even when something's still in kind of preview,

• 22:54Nathan Labenz?

  for my purposes, like the limit is there's enough headroom there for me to like do usually all the testing that I want to do. But if I'm Bolt or Lovable or certainly, you know, Cursor or whatever, then they would be hitting those limits. So what is like super scalable now versus is still in kind of experimental access? And yeah, just give us like the concrete on how much of these different models we can use.

• 23:20Logan Kilpatrick?

  Yeah, so 2.0 flash during experimental, I think the AI Studio UI has different, has like some, we don't publish the limits because it can change dynamically based on how much capacity we have. We don't usually change it, but we sort of try not to publish it just in case we need to and don't want to make people sad. But in the API, I think it's 10 or 15 RP requests per minute on the free tier. And then I think like 4 million tokens per minute, which is a lot of tokens per minute, to be honest with you. So I think that's probably why a lot of people don't hit. And unless you have like users or are trying to run a, like people, a bunch of people have like internal evals and things. There are like people who are running leaderboards as another example.

• 24:02Logan Kilpatrick?

  Those are the usually the use cases where I, people reach out and are getting rate limited. So with the production availability, you can now, if you're on the paid tier, the free tier stuff all stays the same. You can keep using the model, keep tinkering, doing all that. If you're on the paid tier, you can get, there's, there's no daily RPD limit. So you can send as many requests every day as you want. I think it defaults to 2,000 requests per minute and still stays at 4 million tokens per minute. And then we're rolling out at tonight, at midnight, we're rolling out new quota tiers. So as you, you know, continue to scale usage, you can unlock things like 10 million tokens per minute, 10,000 requests per minute as well to help those who continue, need to, to keep scaling up.

• 24:47Nathan Labenz?

  It's wild. The infrastructure behind that is, is truly an incredible accomplishment.

• 24:53Logan Kilpatrick?

  Yeah, lots of TPUs to make all this stuff happen and lots of like, it's, I think, for people who aren't at LLM companies, you don't think a lot about this, but it is, there's just a lot of complexity and like how many models there are and like we, you know, for, I see a lot of memes online about, you know, the bad naming conventions that we have with our models and stuff like that, which I appreciate. But like, I think it actually hits at a different point, which is like, there's, there's just so many different model variants. And it's really difficult. And this is why like a lot of the feedback about our experimental model release chain has been like, hey, we love these models, let us use them in Production and the challenge is we have to be really picky about what model do we use in production just because like the compute footprint that it takes to like actually make

• 25:38Logan Kilpatrick?

  it so that you know the cursors the bolts the the you know other Y C startups the developers trying to scale and build companies can get the compute they need is just like it's hard and it takes a lot of compute so we have to be a little bit more intentional about how we do it ideally we would just GA every model and like everyone would take every model of the production and we wouldn't need to worry about it but yeah there's just a lot of constraints with with doing that

• 26:01Speaker 02

  hey we'll continue our interview in a moment after a word from our sponsors

• 26:07Nathan Labenz?

  yeah if you're going to support millions of tokens a minute I can imagine why you'd need to be prepared for it so maybe help me develop my intuition for how should I think about flash light as it relates to flash because like my totally candid initial reaction was flash is already so cheap and pretty fast flashlight comes in 25% cheaper and presumably faster but also like slightly weaker you know you've got the table of course of all the benchmarks and it's like a little less on most of them have you been getting demand for like an even cheaper faster model than flash that seems like almost that's like hard for me to wrap my head around to be honest

• 26:50Logan Kilpatrick?

  yeah so I I think the the positioning is around twofold one we wanted to give like technically by default the 2.0 flash price is more expensive than the 1.5 flash price so and especially given how much we had leaned into the sort of the low cost per intelligence of the models historically like we wanted to give people a one to one like if you were if you know seven and a half cents per million tokens was like the thing that was enabling your business and we showed up and said hey by the way now it's 10 cents it just didn't feel like a great story for developers given how much we leaned into that narrative so being able to have an option where you know it really was a direct it was not only a better model but was the same exact cost as you were getting before and I

• 27:33Logan Kilpatrick?

  think for the 2.0 flash model it just like because of a bunch of constraints like just did it wasn't going to be possible for us to keep that same level of price so really about that like wanting to make sure that we didn't sort of mislead customers into thinking that they were gonna you know be able to continue to push on whatever the Pareto Frontier is that they care about with cost cost intelligence And yeah, like, I think there's also like some other things that as we think about how do we make the cost lower for these models, like there's a bunch of features that aren't supported, like more of the like, high-end things, like that those flashlight models, for example, will never be able to do native image generation or native audio generations. A lot of things that we can do to keep the cost down as far as like serving those models

• 28:16Logan Kilpatrick?

  at scale, which the 2.0 flash default version doesn't have. It's also kind of similar to, I don't remember when we talked last about this, but we also put out that flash 8 billion parameter model. So you can kind of think about, we're not releasing the size of the flashlight model publicly, but you can think about that as like another version of the the sort of small model train that we'd previously done with flash and with flash A B. Yeah, flash A B was like the most, if folks have looked at open router before, like it was the most high token volume usage model like on on open router, which again is like a reasonable proxy for model usage in some in some contexts. So like the clear

• 28:58Logan Kilpatrick?

  feedback from us was developers love low-cost models and like there's a huge amount of new use cases you can unlock by continuing to reduce the cost down. So I think, you know, if we could have made an even cheaper model, I would have pushed for that as well. Yeah, interesting. I would love to hear if anybody's listening who fits that description of a 7.5 to 10 cent price change per million tokens would have made a meaningful difference for what you're trying to do in the world. Reach out to me. I want to hear that story. I really, I find that pretty hard to imagine. I can easily imagine how people might just look at a menu and say, I'll take the cheapest one because I'm just, you know, processing whatever and, you know, extracting addresses out of whatever, you

• 29:43Logan Kilpatrick?

  know, stream of data. So I can, I can imagine choosing the cheapest by default, but I have a hard time imagining how a business model gets disrupted by that kind of change. But I'd be, I'd welcome that. I think you're right about this. I think a lot of this is just like about how we we tell the story to the world of how we show up. And I think like it'd be it'd be easy for that to become like the the narrative we don't want, especially given how much we've pushed on reducing the cost for developers is like, you know, Google's raising the price for developers. Like, that's not the narrative we want. So it was really important to us to like preserve the continuity of making sure that we have that low price point available for developers who care about this. But I think generally, I agree with you, like especially given

• 30:23Logan Kilpatrick?

  how much the other signal that we've gotten is like, if you have better models, like the story of a lot of the models in the ecosystem is if you have great models, people will pay for them. Like, that's actually not the limiter in a lot of cases.

• 30:33Nathan Labenz?

  Yeah, for sure. For sure. They're all cheap compared to human labor. That's, I think, you know,

• 30:40Nathan Labenz?

  such a, you know, striking two by four to the to the forehead, the fact that it is often kind of glossed over. Let's do the pro side. So how should I think about pro? You could compare it to flash. You could compare it to other frontier models that are out there. But how should I kind of understand it in the increasingly busy constellation of available models?

• 31:00Logan Kilpatrick?

  Yeah, I think pro is where you're sort of not bound by costs in a lot of ways. Like, we didn't release the price of the pro model yet, because it's still experimental, but it's going to probably roughly follow the similar patterns to what some of the pro models have been in the past, which is like just a lot more expensive. So it has to be. And I think there's actually, you know, if you look at like, what is the traditional advice for developers as they're building sort of the frontier applications is go with the model that's best, even if it's a premium, make your use case work and then figure out a way to sort of bring the cost down over time by switching to a smaller model or optimizing, doing fine tuning, whatever it is.

• 31:39Logan Kilpatrick?

  So I think it's important for us to like continue to to honor that flow, which I think actually works. Like, there's a lot of that. This is the default experimentation path that developers go on today. I think specifically like the use case where we're seeing the best performance relative to the other domains is encoding. And I think like we're going to continue to, and I had a glib tweet, you know, a while. I don't even remember when it was. The day before, oh, three, I think, got announced about how we're going to have the world's best coding model at Google. And I still believe this deeply. And I think like pro is going to be that model. And a bunch of the reasoning work that we're doing is going to be that model that continues to push the frontier for us in coding.

• 32:19Logan Kilpatrick?

  And like, it's a domain we need to win, especially if you believe in, and we were talking about the sort of text to app creation stuff before. Especially if you believe in, in that sort of trend continuing and the acceleration of developers continuing from like a internal software engineering productivity standpoint. So I think that's probably the, the best, the best use case for it. It still has two million context. Maybe it'll have longer context in the future. So that, that might be something as, as well.

• 32:45Nathan Labenz?

  Yeah, we've heard whispers of infinite context.

• 32:49Logan Kilpatrick?

  You, you and I were sitting in that room together. I'm pretty sure when Jeff Dean said we were getting infinite context some point in the future. He didn't put a date on it at I O last year in that session we were in together. So yeah, hopefully I, I think that's going to be a huge, a huge unlock for folks once we land it.

• 33:04Nathan Labenz?

  How would you guide people to think about the non-release of, it was ultra, right? Was the, the largest scale? We've kind of seen the same thing, obviously, from Anthropic where, you know, it's like, wait a second, what happened to Opus? Is there anything you can share to help people just generally understand like, why we seem to have gone from small, medium and large to now just small and medium?

• 33:31Logan Kilpatrick?

  Yeah, that's a good question. I think so. The historical context on Ultra was basically trying to prove out the research direction that this, that this, you know, scaling was going to continue. The reality is they sort of proved with Gemini 1.0 when the original Ultra sort of model candidate came out that, that, that was the case. But also then like, there was just all of this like continual rapid innovation of like making like all of a sudden the, the pro model was better. Then all of a sudden like, now I'm pretty sure flashlight is better than the original Ultra model. So it just becomes like this question of what are the cost trade-off, um, infrastructure, uh, equation to keep in mind. And like, you know, you could imagine a world where.

• 34:16Logan Kilpatrick?

  We have an ultra model and it's 5% better on every benchmark and it's, you know, five times larger and costs five times even more than Pro. And then like you just start to think about like, do the math and think about like what from a research perspective, like where does it make sense to, to spend our time and energy? And also from an infrastructure footprint perspective, where does it make sense to spend our time and energy? And like, we continue to see gains on like making models much higher quality at the same size or even a factor of a size less than previous models. And I think now especially with, with reasoning, there's even more question marks in my mind of like, how much like if, if we could get a model that was even 10% better on every benchmark,

• 35:00Logan Kilpatrick?

  like, does that make sense, given a world where reasoning has like so much scaling and like, there's just a lot of low hanging fruit work that we can do there. I think back to the conversation I had with, with Jack, uh, yesterday, he, we, we talked a lot about this, like pre-training scaling and reasoning scaling. I think like Google is still scaling pre-training. So like this isn't that we're not going to keep scaling pre-training. I think whether or not we like officially release an ultra model, I think is, is an open like technical research question, just because of like all of those constraints. And like, it, it's also kind of tongue in cheek. We're like, you know, we could rename all the models and, and say that we have the ultra model if we wanted to. And like, maybe that would have been the right thing to do historically

• 35:41Logan Kilpatrick?

  and make Pro Ultra and then remap all the other names. But I think trying to honor like the, the essence of what was intended through that ultra model, I think is part of the thinking.

• 35:51Nathan Labenz?

  Yeah, I'm on the side of anything that maintains any clarity in naming schemes. So I'm with the consistency. That's the only reason I even know to ask this question, right? I mean, if it was all renamed, I would even just be even more confused than I am. So, okay, let's go back to coding and maybe, you know, think about the likes of Bolt and Lovable and Cursor and Devon, increasingly, you know,

• 36:14Nathan Labenz?

  Many, many options. I can't even keep up with all my AI, you know, coding paradigms these days. And they probably can't keep up with all the models. So the question I have for you is like, okay, we got all these benchmarks. It probably, you know, the new one versus the old one kind of wins on some benchmarks, loses on some benchmarks, whatever. An interesting thread that I wouldn't say is like dominant, but has definitely got some weight behind it over the last, say, month or two has been, yeah, O1 is great. Even O3 Mini is great. You know, potentially Gemini Pro is great, but for some reason, people still seem to think Claude 3.5 Sonnet new is the best coder and the best coding assistant. And they haven't,

• 36:56Nathan Labenz?

  in fairness, had time, you know, to compare it against necessarily everything. But how would you suggest people think about this? You know, the simplest thing to do would be like swapping, you know, a model in exactly the same situation that they currently have, like, you know, one other model performing the best. But you could very easily say, well, you're probably leaving some performance on the table because these two models are probably not going to have maximum performance under the exact same conditions. They're each going to have their different conditions that elicit the best from them. But how do you find it? You know, and how do you know how much time to invest in that? And like, yeah, I don't know. It's, it seems very difficult. And it's, it's even difficult for me. I was just like an obsessive hobbyist. So what's the, what's the best practice? Or how should one think

• 37:41Nathan Labenz?

  about absorbing a new Gemini Pro and comparing it to a Claude 3.5 Sonnet and an O3 Mini in today's world?

• 37:50Logan Kilpatrick?

  Yeah, this is such a tough problem space. And I just have an incredible amount of empathy for developers and founders and people who are building stuff because like, there is no silver bullet. I'll give a couple of reactions to this, like, general thread. First, like, my own personal example that sort of, I think underscores at the, the second point that I'll make. I was doing a bunch of, you know, I was doing the normal web developer thing the other day of trying to get the corners of a table rounded. And I was, I was smacking my head on, on this problem because of a bunch of weird constraints in the environment that I was working in, and using the Gemini models. And, you know, I became fed up at one point. And I was like, you know what? Maybe the Gemini models just aren't that good and like I'm gonna go try Claude and see what it does.

• 38:34Logan Kilpatrick?

  And I, I, you know, went out of the environment that I went out of AI studio, went into Cursor, tried it on this like very simple prompt and it worked, single shot, everything just worked. And I, I started to ping a bunch of people frustrated and I was like, this is, you know, we need to keep making coding better, yada, yada, yada. And then I went and just for my own sanity, I went and reran the exact same prompt and stuff that I did again with Gemini models and it also worked. And I think like this was a good example of like, I think just like, like the prompt was really, like I was just doing a bad job prompting and I pulled myself out of this environment that I was in of this like iterative loop with the model and I started over from scratch again and formatted the question and the context in a different way. And like, it worked with both models.

• 39:16Logan Kilpatrick?

  And I think this point underscores this like vibe slash like incredibly unstructured scientific way in which people make these decisions today. And there's like a lot of the, I, I won't name names of products, but like work super closely with lots of teams who have L L M's in production. And you would be surprised of like, insert your favorite L L M product. Like in a lot of cases, how few evals people actually have to understand like, what are the metrics that matter for us as we build our product and service? And I, I think there's like two ends of this coin. I think like one, the world needs a platform in which it's hosting all of the sort of publicly available benchmarks and, and sort of leaderboards and stuff like that.

• 39:58Logan Kilpatrick?

  I find it incredibly difficult to just like navigate and get a snapshot of like, how good is this model? There's like 20 random benchmarks here and 50 random ones here and I gotta go, you know, look at Minecraft because though that benchmark's really cool. I love that benchmark, but like they're all split out over the place and it's just like hard to keep track as a developer. And like to your point, like this is my job. This is what I spend every waking moment of my life doing. And like, it's still difficult to keep track of this. And you could imagine for people who have much less time and are, you know, doing other things, it's just hard to keep up with it. So I think someone needs to build this platform and maybe this is like the, you know, you and I should do the, the Y Combinator like call for startups thing. And like that's, that is the call for startups.

• 40:40Logan Kilpatrick?

  Startups like build this platform to help people really bring all this stuff together. And then separately, I think, and the folks at Kaggle actually are pushing on this notion of like having personal evals and being able to sort of build a platform in which as new models are released and made available to the world, you can sort of just run your personal eval will just like run behind the scenes on those new models and you'll just get an email that says like, hey, this model based on what you've told us like might actually be one that you should spend your time checking out because it's really good at these things. And you know, those are the things that you indicated you care a lot about. So I think that type of platform and product experience, like taking the burden off of developers to sort of

• 41:23Logan Kilpatrick?

  just do this is going to be awesome. The challenge in that case is like, you still have to make your personal eval to begin with, but I think that like one time cost is a lot less than like having to do all the spin up cost every time a new model hits the market.

• 41:38Nathan Labenz?

  Yeah, it almost seems like if I were going to take one sort of practical rec out of all that from for developers, I might say let your users choose because

• 41:51Nathan Labenz?

  at least that way you can like get some data and they can sort of feel a little more agency and you know, maybe there's something good to to find there. I like the personal benchmarks too, but the things I care about typically don't have like a right answer, you know, like at Waymark, it's always the same challenge, like what makes a good video? You know, we can have like a language model judge, but now we're in Hall and Miras territory and there's just, we just don't have any, you know, I'll cop to being like, we don't really have evals that work for us, you know, we can detect outright failure to follow the structure or, you know, it's too long. So we have like some guardrails that would be like the clear thou shalt nots, I call them, the thou shalt nots of this task, like we can, we can detect those.

• 42:34Nathan Labenz?

  But past that, it is really still vibes and.

• 42:39Nathan Labenz?

  Yeah, it's interesting. We don't actually let our users choose, but that could be an interesting. Our users also don't even know, you know, they're not in this world. So the developer's use case is more what I had in mind when I was thinking, let your users choose.

• 42:50Logan Kilpatrick?

  I think that's it's a trend that you you hit the nail on the head like, and I don't even know how much of this is like a conscious decision that founders are making. But if you look across like, again, some of the, you know, choose your favorite L L M products, with the exception of Waymark, like developers are and like actually end users have this choice today. Like you go in, there's a model dropdown and most of those products and like a lot of them that are not created by large model providers have most, if not all of the models available. And like the the sort of timely example of this is Copilot, like Copilot now, you know, was historically just powered by G P T models two years ago. And I think like the developer community and like where the world has moved,

• 43:30Logan Kilpatrick?

  it now has a model dropdown and you can choose the Gemini models or the anthropic models or the OpenAI models. And like, I think more and more products are going to go that route. Also, as like the, this is maybe corollary to a point that a lot of people make, but like, I think, you know, people talk all the time about like the commoditization or like the contraction of like the the delta between these models. I think that's actually not true. I think there's like a lot of this weird nuance that is going to continue to sprawl out over time where like you will get substantially different answers from, you know, insert whoever your favorite model provider is over time, even if like the capability is similar on paper, there's still all of this stuff that will be different.

• 44:12Logan Kilpatrick?

  And I think it'll be important that people continue to have the choice because I think that that subtle difference makes a lot of impact on like the end product experience people are going to have.

• 44:22Nathan Labenz?

  Yeah, no doubt about that. Even DeepSeek over the last couple of weeks has, I think, shown a very different profile and is probably the most, you know, and perhaps not surprisingly, given its, you know, source is it's like closest to base model is probably my best description of it. And in that, for that reason, it can like write sometimes in ways that are like extremely compelling, but it's also much less.

• 44:46Nathan Labenz?

  Behaviorally refined in the way that the Western, you know, at least top tier model provider created models are. So yeah, a lot is, and that's all with, you know, very similar benchmark results, right? So it's very, um, there's a lot beneath the surface that remains to be unpacked with, with any major new model release. Well, let's talk for a minute. I know you got to go before too long. Last things I wanted to cover were briefly fine tuning, because as a developer, that's something I'm always interested in. And we'd love to know the status there. Then maybe, you know, what's up with reinforcement learning at Google? I mean, we've seen the thinking model. It struck me that there wasn't much actual mention of reinforcement learning, whereas like the other developers are coming out and saying,

• 45:30Nathan Labenz?

  we did a reinforcement learning model. Uh, it seems like DeepMind has kind of positioned it differently, although I assume that a lot of the same techniques are happening under the hood, but I don't know. And then I was actually going to ask for your call for startups too, because I know that you have recently raised a solo venture fund and we'll see if we can get you some deal flow. So yeah, fine tuning, reinforcement learning, your fund, take as long as you have.

• 45:58Logan Kilpatrick?

  Yeah. Fine tuning is something like I'm, I continue to be incredibly bullish on fine tuning. I think the future where everyone in the world is using their own fine tuned model, a version of the model that's, that's sort of has the context that they needed to have. And the context isn't being like overly, overly impactful on the priors of like how the model makes decisions. I feel like is like the rough estimate of like how I, or the rough explanation of how I think about fine tuning. So it, we don't have it yet for 2.0 flash. I think we need to, I think we, you know, we, we've been having a lot of internal discussions about like, what is the, you know, size of investment we want to make in fine tuning? To me personally, I think this is like one of the biggest opportunities. Your point about developers wanting this, I completely agree.

• 46:42Logan Kilpatrick?

  So we're, we'll, we'll keep pushing on it. It's not available yet. Hopefully it will be available soon. And like even more than it is, like, there's a lot of limitations on how we do fine tuning of 1.5 flash today. You can't do images. The, yeah, there's a bunch of rough edges. So we need to solve all those things and make it so that it's, it's really a first-class experience. Your second question.

• 47:01Nathan Labenz?

  Reinforcement learning.

• 47:03Logan Kilpatrick?

  Reinforcement learning. Yeah, there's RL is part of making those reasoning models. Part of this historically is that, like, why have we not talked a lot about it? We did this sort of normal, low-key approach to doing the release, which is what we've historically done for our experimental models. I think we're actually like, as the world is getting more and more excited about the sort of what's happening with reasoning models, like, we're going to start talking a lot more about that work. And yeah, I'm excited for us to, to tell more of the story about the work that we're doing. It's, it's actually like, the reasoning narrative inside of Google is the, one of the narratives that gets me most excited about the direction that we're going in. There's just been so much progress and like a ton of breakthroughs. So it'll be,

• 47:46Logan Kilpatrick?

  hopefully soon, we'll, we'll tell that story to the world. And yeah, we'll, we'll get you some people on the, on the podcast to, to talk about it hopefully soon. And then, yeah, startups, like I still, this, you know, this is the moment to build impactful, interesting companies. Vision is one of those things that I'm really excited about. Like, I, I still think, like, take the entire ecosystem that's built on domain-specific computer vision models. I think all of that is up for grabs with vision language models. I think there's going to be a huge amount of startup activity in that space. I think reasoning is going to make agents work, which I'm excited about just because of like how many, like, I haven't, you know, I, I'm not an agent investor in a lot of ways, but like, I think there's just so many companies that are trying to tackle problems with agents and like,

• 48:32Logan Kilpatrick?

  a lot of them just don't work today. And I think that's actually going to be the biggest unlock we were talking about before the like. New models enabling some percentage of Y Combinator startups to like actually work all of a sudden. I think reasoning is going to be that that continual breakthrough that like more than anything else that happens in the next two years is going to make those companies act their products actually work, which I think is really exciting for them. I think it's exciting for the world as we as we sort of figure these things out. I think there's a whole class of startups, which I think don't yet exist. And I've started to talk to some of the people building these companies around like

• 49:12Logan Kilpatrick?

  how the fundamental nature of the internet changes in a world where you can't assume the only thing that's visiting your website is a human with the exception of like, you know, index crawlers to make your content more discoverable, which historically was like the only non-human thing that was visiting your website. The like social contract of the internet is not set up for that. And I think it's going to be very, very interesting to see how like fundamentally things change with like websites being locked down or like what the protections that are in place are or all of that type of stuff. I think there's just like all of this new like very basic how we engage with the internet experience that is going to change over the next few years.

• 49:56Logan Kilpatrick?

  And I think there's a lot of people who have businesses and websites and companies that are going to have to like, they're not going to be able to solve that problem themselves. They're going to need somebody else to solve that problem for them. And I think there's some interesting companies that can be built to enable that sort of yeah, how your website talks to agents attribution happens, you know, how you're able to like capture your slice of the value creation that's that's taking place. So I'm interested to see what happens in that space.

• 50:24Nathan Labenz?

  Yeah, that seems like a really good candidate for what's going to change the world most in the not too distant future. It first of all, seems like it's really likely to work with a reinforcement learning paradigm because you're going to get a pretty clear signal from a lot of tasks as to whether or not it succeeded. And yeah, I've been watching like Payman a little bit recently. Any, any like specific companies you would suggest people look at or anything, any specific problems you think are like most come pitch Logan, if you're working on this?

• 50:55Logan Kilpatrick?

  Yeah, I think evals is the other one. Like actually going back to this point, like you, you making the comment that like even for you as somebody who I would classify as like incredibly sophisticated in this space, like you understand what's happening in the ecosystem, you understand why evals might be important. Like the fact that you can't, the fact that it's so hard for humans to articulate, like what is the taste and your perspective on some of these problems into something that can programmatically happen, I think is really, really interesting. I don't know how that problem gets solved, but actually if you, if you've spent too much time in the evals rabbit hole like I have, then one of the big things is actually like most of the problems in life end up being eval problems. Like if you look at like, you know, it's just very interesting.

• 51:40Logan Kilpatrick?

  If you like follow that chain of thought, how, yeah, how things end up happening.

• 51:46Nathan Labenz?

  Yeah, for now I just do expert demonstrations like for Waymark, we just go and say creative team, write us a bunch of good stuff, fine tune on that, you know, or put those into a few shot examples and hope for the best. And, you know, from there it becomes vibes. But yeah, it would be nice to have something better. So yeah. Well, where can folks find you if they want to pitch you or if they have, and you're incredible, as everybody knows, online with responding to questions, concerns, issues around the API. So I don't want to bring more of that to you than you already have, but where should folks find you if they want to either point out an issue or pitch you a startup?

• 52:25Logan Kilpatrick?

  I saw someone respond to one of my tweets the other day and they said, I miss the old Logan. He used to reply to all of his replies on Twitter and I was then went and looked at my, how, how much time every week I spend on Twitter. And I was like, it's my, my time doesn't scale here. I'm already putting in.

• 52:42Nathan Labenz?

  Yeah, you're gonna need an agent.

• 52:43Logan Kilpatrick?

  Yeah, way too many, way too many hours. But yeah, Twitter, LinkedIn, everywhere that the internet exists. Hopefully I'm, I'm there helping people with Gemini stuff.

• 52:52Nathan Labenz?

  Cool. Well, Gemini 2.0 flash is out today. General availability. It is good, fast and cheap. So definitely one to check out. Logan Kilpatrick from Google DeepMind. Now, thank you again for being part of the cognitive revolution.

• 53:07Logan Kilpatrick?

  Thanks for having me, Nathan. This was fun.

• 53:09Nathan Labenz?

  Always a pleasure. It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please, don't hesitate to reach out via email at [email protected], or you can DM me on the social media platform of your choice.