Latent Space: The AI Engineer Podcast
Original podcast
Watch the full show on YouTube: https://www.youtube.com/watch?v=Hxlayqs-IAs
6,028 words · 3 speakers
We got so much: Gemma 4, Gemma 3.1, Gemma Scope, Met Gemma. Give us the TLDR.
Yeah, so yeah, Gemma 4 is just out. It's the most capable open model we've released so far. We already tried to compact as much intelligence per parameter as we could, bring all of these multimodal capabilities. So yeah, that's Gemma 4.
So one interesting thing, you have this thing with effective parameters, not active parameters. Can you explain what it is?
Yeah, so pretty much in the traditional transformer architecture, you have like this big embedding layer, right? And this new architecture is more of a small change in the transformer architecture, in the transformer block. Pretty much we had a per-layer embedding. So at every layer, we had an embedding table. What is exciting is that you don't need to do like the whole matrix multiplication. This is pretty much a lookup table. So the Gemma 4 model is a E2B. That means that it effectively has 2 billion parameters loaded into the GPU. It actually has almost 5 billion parameters, but those 3 billion parameters can be in the CPU, they can be in the disk, which means that you can do inference extremely quickly. This is just a lookup table.
And what's the con? Why don't we always do this? Can it scale? Is it open research? Like, you know, it seems very, okay, if I can just offload half the parameters to save viewers.
Yeah, so pretty much here we did lots of quality experimentation, and this is really optimized and designed for like on-device. And when I say on-device, I mean like running in a phone, Android, Raspberry Pi, and so on, right? When you go larger, you usually want to compact more. You want to have more like dense architectures or MOEs. So this research, these research decisions were very helpful for this small use cases.
Yeah, something I learned from the run that you organized this morning. For listeners, I think this is the first ever like official run club.
Yeah, yeah.
At AIE, 6:30 AM. Very rough, but at least I woke up for it. I met Cormac, and he was telling me that apparently in China, the super apps are shipping models in the app bundle for inference and just like use among all their super app constituents. I don't know, is that like a target use case for you guys?
Yeah, so actually, if you install like if you buy a Pixel phone or a high-end Samsung, they come with Gemini Nano. And Gemini Nano is baked into the operating system. And Gemini Nano is really built on top of Gemma. So last year, we released Gemma 3N, which was this architecture really designed for phone use cases. And they use Gemma 3N with some additional training, some additional adaptations, to make the model good for like traditional on-device use cases, right? So pretty much when you buy like these high-end phones, you can already use Gemini Auto-Books.
Yeah, we actually covered the 3N paper in our paper club. And this like idea of like sort of parameter offloading or like download on demand is like very cool. Is it exactly the same in the Gemma 4 stuff?
Yeah.
Okay.
For the smaller models.
Yeah. And does it, does it scale? Is there a potential? So for reference, Gemma 4 is a 29B and a 31B ones, Moe ones, tents. But have you scaled it? Have you pushed it up? Is it?
We are doing lots of experiments.
Experiments?
Yeah, yeah. It's a team. Yeah.
What goes into shipping a mainline model like this? Like, what's the behind the scenes?
It's complex. The Gemma team is actually relatively small. We have like a two, three PMs. We have one marketing person. And then there's our like engineers and researchers working and shipping this. Of course, there's like the whole training part. How do we do the post-training, distillation, post-training techniques and so on. What is quite exciting is that once we have the model, then we collaborate with a bunch of open source partners, right? So for example, we work with Llama C P P, Ollama, M L X, Hugging Face, B L L M, Nvidia, A M D. So we have almost 50 external partners for every lot. Well, for the Gemma 4 launch, which has been the most complex launch. And also internally, we collaborate with a bunch of different teams. So think of Google Cloud, Vertex, Vertex Models as a Service, A D K, and then Android as well, right?
So we work, for example, with the Android team. And with the launch of Gemma 4, we released an integration with Android Studio. So in Android Studio, there is this agent mode, where you can have a model helping you write code and do things within Android Studio. And they ship this integration with offline models using Llama C P P or B L L M or any OpenAI-compatible endpoint. So now you can use Gemma 4 to also write code Android applications in Android Studio.
Where's the difference? When would someone want to do that versus just use Gemini?
Gemini, of course.
Outside of the obvious, you're offline, or you want the privacy.
You fly planes a lot or something.
I did, okay, I will say, on my long 10-hour flight to London, I did use Gemini.
I was on Gemma 4 though.
Sorry, Gemma, Gemma.
Yeah, yeah, it's mostly offline just cases, right? Or if you, yeah, offline or privacy, like if you want to have all of your development set up locally and you don't want to send any code to any API, you will choose that.
Do you see a future where, you know, small models get good enough? Like, does it cannibalize? It's an interesting position. You have big Gemini, you have Gemma, both get exponentially better over time. Current Gemma is much better than what we had closed source a few years ago.
Yeah, for me, it's quite exciting. I mean, if you look at Gemma, you compare to how we were one year ago, I would say Gemma 4 is matching state of the art from one, one and a half years ago. For most things, with local models or models that you can run in your own hardware, you can get capabilities, so you can get agentic capabilities, function calling, system instructions, like conversational and that kind of stuff. Knowledge is much trickier, so for knowledge, you do need a larger model, right? That's why if you compare Gemini to Gemma, Gemini has much better knowledge, understanding of the work, right? Like facts, information and so on. So it really depends. I do think we are heading towards a future in one, two years where, imagine like you can run a Gemini 3 Pro powerful model directly in your phone, right?
And I think once we get there, things will be quite exciting. From a product integration, from which experiences we can enable the users. I wouldn't say it cannibalizes. It's still like two very different things, like if you want like flagship capabilities, like this super complex, long-running task, you would use Gemini if you need factuality and so on. But I don't think for many of these agentic things, we'll get to a point in which we can do very powerful things directly on device.
Yeah, can we talk about the multimodality sides? Any advances there that you want to highlight or you've been getting good feedback on?
Yeah, so GEMA 4 was built on the same research as Gemini 3, which pretty much means that we benefited from all the improvements that happened with Gemini 3. Multimodal-wise, the smaller models can understand audio, images and short videos, so 30 to 60-second videos and audios.
Which is actually quite long.
And even the on-device, like the 2B, 4B 2B can do very good multimodal.
Yeah, for audio, we have a speech recognition, we have a speech to translate the text and then a bit of a speech understanding. So you can do like ask questions about an audio file and so on. So use cases that are very optimized for like on-device phone use cases. And then on the vision side, we also improved things quite a bit. So we have object detection, pointing, captioning. We do not have image segmentation, which we know is like one thing that many people have been asking us. But otherwise, like for many things, we do support that. The other thing we do not support yet is video with audio. So we can then understand like video input or audio input separately. But if you want to pass like in the same from both the visual part and the audio part, we still need to do some improvements around that.
And that's just a matter of like more data?
Probably some additional fine-tuning could yield some very good baseline models for us.
Yeah, yeah. What about audio out?
We're exploring some things here. Nothing I can share at the moment.
I think everyone's excited about the, like, when do you have native speech to speech, right?
But as far as I see, people always get excited and then the pipelines always win.
Yeah, yeah. GEMA is quite important for the multilingual aspect as well. So GEMA supports 140 languages.
You did a lot of work on the multilingual encoder, the tokenizer, right?
Right, right. Yeah, exactly. So the tokenizer has been pretty much based on the Gemini tokenizer. It's extremely good. So independently of the GEMA capabilities, if you just pick base GEMA model and you fine-tune for an additional language, it actually works extremely well.
What are some, so I didn't read that part. What are some insights on the tokenization?
And this comes from GEMA 3. Like, this has been done already for over a year. But the tokenizer is pretty much the same as Gemini, which means that the tokenizer lends itself to capture the right tokens for different languages. It's like a very good multilingual tokenizer. Which means that if you compare GEMA 3, so I'm going to the previous generation. If you compare GEMA 3 to other models from back then, maybe the other models were better than GEMA 3 as a general model. But if you train all of these models for, I don't know, a specific Southeast Asian language, I don't know, Vietnamese, let's say, GEMA would yield better results even if the other base models were potentially better.
I mean, I think there is some limit at which you basically have platonic representation, right? Like, you understand the core concept and it translates to whatever language you want.
Yes.
I guess, you know, you are also, you have purview over all of sort of Google developer experience. And you brought the team here for the first time.
What was that like?
It's quite exciting, to be honest. We have already participated in previous AI in your conference, like Philips or like other team members have been in some of these in the past. This is London, this is DeepMind's home.
You have to.
Yeah, we have to. I mean, we brought a bunch of researchers from the team to share about different things that we're working on. We brought other teams from Google, not just from DeepMind, that are also like using AI in one way or another. So we brought people that are doing on-device machine learning, people that are doing light RTs or optimizations to run models directly in phones or in the browser. We brought people from the Android team. We brought people that are working all over Google, from robotics to research to Android. So yeah, it's quite exciting to come here and really show all of the things that the company is building, not just come and share like the things that our team is doing, but really all of the overarching AI story that we're working on.
I think you are, I mean, it is the lab with the biggest scope. You do everything, including Dolphins.
And it's very impressive, like, yeah, so you brought Sander. We can talk a little bit about the researchers that you brought.
We brought researchers in a couple of different topics. So we brought one of the researchers that worked in the GEMA development, in the development of GEMA. We brought a researcher that works in diffusion models as well, for a diffusion transformer model. So diffusion as text generation, not image generation.
Which was announced but not released?
Exactly. We did a Gemini Diffusion last year at I O, which is very cool because you can generate code extremely quickly, right? Like it's a stupid.
So the main pitch is speed, but other than speed, is there like a secondary? You know, what can we do with a diffusion model that we cannot do with auto-regressive, you know?
It's mostly speed.
I, I feel like in terms of code structure, there may be some things where you're like, okay, I want the brackets here.
And then you fill in the blanks, right? So fill in the middle is like a common code problem. But this is extended fill in the middle or like extended like, oh, help me upscale or put a Laura. I don't know, you know, translate the image analogy to text.
I think in the past, fill in the middle was like this task that many companies were trying to tackle as an additional generation task. And now people are just assuming that the model can do fill in the middle with a general.
Auto-regressive, yeah.
Exactly.
Yeah, yeah. No, no tricks about special tokenization or anything like that.
It used to be a, you know, mass language modeling, you, you're trained to predict fill in the middle.
You had to rearrange your dataset as well in order to do F I M.
Yeah, it was a bit tricky. People were always getting like the, the prompt in the, the tokens wrong and yeah. If you deviated in any way from the training format, it didn't yield good results. Now we have like very good output box capabilities for that.
What's the, what's the idea about investing in text diffusion? Is there a world in which this overtakes auto-regressive?
Yeah, that's a good question. I think at the moment it's still very experimental. I think we'll be releasing and sharing a bit more research of the things that we have been doing around diffusion, uh, text generation models on that space. I would say it's still very early stage. I think especially like the model quality is still a bit worse from what you would get from the normal auto-regressive model.
A lot of what you were mentioning earlier about it for, you know, okay, fill in this code block, this stuff. It seems different to how we're building agents these days of, you know, sequential to calling this, that. I guess it's, if it's just speed, it's speed.
If it's in the arsenal.
I could see, I could see a world where there's like system one, system two. System one is the diffusion one, system two is auto-regressive, system one is the planner.
System two is the executor. I don't know.
Maybe it's too hypothetical at this point.
But I will say.
Diffusion transformer models are difficult to fine tune as well. So there's also like a point in which how much flexibility, like, yeah, I could see a word in which, yeah, you have like a very strong agent manager kind of a setup. And then you have like executors, like diffusion based executors that do like a specific coding.
Are people fine tuning outside of, you know, we see a few big companies do, okay, like Cursor has a really good, consistent, there's a few that have done fine tuning, but it seems like it's not picking up as, you know.
Yeah, so there was this period 2024 in which there was like this, maybe 2023, like there were all of these fine tuning communities and I think it's been changing quite a bit over the last two years because models are getting very good out of the box. So as I was saying, like for Gemma 4, we had 50 to 60 partners. And some of them were like, oh yeah, we're going to try and fine tune the 27B model for this vision task. And they were like, oh, actually the model works too well out of the box. We don't need to fine tune it. Yeah, we saw lots of those things. So I'm seeing less excitement around fine tuning nowadays as general conversational models.
There is still quite a bit of excitement around fine tuning for specific domains like finance, healthcare, specific types of data that the model didn't see. But as general conversational, like just changing how the model behaves, you can do most of that via prompting nowadays. And in terms of capabilities, the models are very good out of the box. So it's been changing quite a bit. There are still like the onslaught people. I don't know if you know Daniel Han and his brother.
Every year I give them a three hour workshop to just talk about.
Yeah, yeah, they are the goals. They still do like amazing tools for the community to fine tune and the community use those tools. But I'm seeing like some changes in the trends. I think people are not fine tuning that much anymore.
And you guys put out a version of your own. MedGemma is a fine tune of Gemma 4.
Yeah, so MedGemma, the last MedGemma, which we released three months ago, MedGemma 1.5, it's based on Gemma 3.
Gemma 3.
Yeah, Gemma 3. So it's pretty much Gemma 3 and then additional training with some of our medical data sets.
How do you see, if I'm not mistaken, Apple Foundation models on-device were a bunch of LoRa's for different tasks? And when you're constrained to running on-device small, efficient models, you guys did an offload, so you're like caring about efficiency. But you know, do you see a world of multi-LoRa's for tasks? Should people be fine-tuning the small one?
I think this is a big challenge in general in the whole developer ecosystem, because let's say that you want to have 20 apps in your phone, right? And let's say that each of those apps comes with its own LoRa, right? What happens when you update the base model? You also need to update all of these LoRa's. From a developer's point of view, I think it will be very tricky, because one, you don't want to have 20 different base models in the phone of the users, the battery will just die. You also don't want to have to update 20 LoRa's every time you update the base model, right? So the release cycles in the Android world and in the iOS world are very different. So yeah, I think it's more of a general industry challenge that we need to figure out how we think that people should build on-device, phone, power, like AI experiences.
It's more of a product and developer experience kind of challenge.
Okay, I have a question about the bigger Gemma models. So you have two models that are pretty similar size. One is DENSE, one is MOE. Can you talk a bit about, okay, say you have 27B you're putting out. How do you think about, should I build an MOE? Outside of inference and using it, how do you think about when to do MOE versus DENSE? What are the trade-offs?
Of course it's inference, what else is there?
But then there's two at the same size.
Oh yeah, but I mean, one is 31B, which is DENSE, right? And that's like the most raw intelligence. And then you have the 27B with 4 billion activated parameters.
But like, you know, why not a 31B 5B active, for example? You can just fit more in DENSE?
Yeah, I mean, we did quite a bit of experimentation and like research on like which would be the best sizes that would be friendly to developers and we chose, we made decisions around that, right? The 31v3, like the largest model size that a Qantas would fit in a consumer GPU. The 27v is more like an extremely fast inference within those constraints. MOEs are challenging to fine tune. I don't know if we've talked about that in the past, but MOEs in general are like extremely good. Their architecture, they work great for inference. But when people fine tune them, they struggle a bit. Like they are not as easy to fine tune for instruction following. The standard recipes and hyper parameters that you have may not work out of the box for MOEs.
The intuition is the routing kills the backprop or what?
I think so. I don't have a very strong intuition on it either, to be honest.
People always say this, but I'm trying to say why. Right? Because if you can train it, you can fine tune it.
Fine tuning is just training?
Yeah, I think it's a mix of the routing and yeah, just having like different distributions. And the distribution may affect the routing in a different way than a dense model where it will just change things. That's kind of my intuition, but also like I think there are many different variables here, like how many experts do you trigger or?
Yeah, there are like a bunch of different parameters that you could move, like whether you freeze or you don't freeze the router, like a bunch of things that you need to think about.
To me, the most important asymptotes that I'm looking for are what is the minimum sparsity level that we can reach? And then what is the most, let's call it elo per byte?
Yeah, that's something that we discussed quite a bit, like what's the intelligence per parameter, right? Like, how do we maximize this intelligence?
There has to be a number, right? Then we can stop, right?
Yeah, because if you compare like that, I mean, GMA, we have done that same size, right? 27, like almost 30 billion, around 30 billion parameters for GMA 2, 3 and 4. And the intelligence is much higher, right? Like we have not increased the model size.
Yeah, it's like that, that, that.
It was an easier number when everything was dense. Then you have to add in sparsity. Now you have offloading.
Yeah. Yeah, you cannot compare like a MOE to dense models. There is no, there are some like napkin calculations you can do to compare, but it's not apples to apples. That's a good question. I don't know where we'll be in three years from now. I would assume a 30B parameter model could be extremely powerful. I still think there are limitations in terms of knowledge. So maybe the model will be able to do like
super wild gigantic stuff, but it will not know like who was the president of X country 25. I mean, maybe yes, but like very niche knowledge probably the model will not have.
Yeah, this is just information theory, right? Like you're using the model as a database.
So of course there's going to be limits. The other thing is also, I always think about, when we talk about this topic, superposition, right? Anthropic has this concept of superposition where you can store information in the smaller weights because it compounds with the other weights as well.
And so not that much research on it since then, but maybe this is my segue into Mecinturf.
Gemoscope. Yeah, so last year in December, we released Gemoscope. So Gemoscope pretty much allows you to analyze the activations across different layers based on the tokens.
Yeah, it's fantastic.
And yeah, the team released, I don't know if it was a couple of terabytes, maybe even up to like one petabyte of data that we had to store because we did that for every single layer across all of the Gematrie models. So it's a very complete.
And Llama as well?
We did it just for Gematrie.
Oh, okay. I think Neuronpedia had some others.
Could be, could be.
There's a few other teams.
I was like, wow, it's like, yeah, it's very close to a cross lab partnership.
Yeah, yeah, there are a couple of open source tools as well that you can just do create your own, yeah, your own activation networks. Yeah.
It's a niche field. I think it's a good opportunity. I think we were talking about this earlier, right? Like it's an area where you don't need lots of compute to get started. That allows you to understand like how the model works. You can experiment, you can get a bit of a sense of how transformer architectures work. So it's a good area.
Okay, the context of this is really like, why bring researchers to AI engineer, which is an engineering, applied AI conference? One, to me, it is actually very important that you bring the researchers because engineers want to learn about how the models that they use were trained, even if they never ever trained it themselves, right? Because I think they just feel more trusting of the model.
Yep.
If you peel back the curtains a little bit. And also, I think there's some prestige that people want to feel like they can go home and talk about it intelligently, even if they don't actually, you know, know how to train it. The other thing is like, I do think that research and engineering are closer than people think.
I mean, there's research engineers.
And Mecinturp is probably the easiest, single easiest way that engineers can get into research if they want to.
Yeah, I think, I mean, in big part, like, so many researchers are doing ablations, right? Like, they are just moving the pieces around and seeing what works and what doesn't work. Of course, there's like a branch within research that is much more like architectural design and like a much deeper. But there's lots of very like empirical experimentation and seeing what works, what doesn't work, moving things around. Which for me is more engineering rather than for like research, unless we are like creating new activation functions maybe.
Yeah, I think this maybe is a change in your career as well. Like, it used to be like a joke, like, haha, researchers are terrible at coding. And then they throw it across the wall to some engineer, they will clean up the code.
But now everyone has their own personal research engineer, right?
Yeah, and something that is cool that is happening is also how researchers begin to adopt some of the cool agentic tools now. So for example, within the team, we are building skills to do experiments and ablations and evaluations. And how the research team can use all of these agentic tools as part of their research process is also quite interesting.
Yeah, I had Yite on my podcast, who led the post training for IMO Gold's model, I think it was Deep Think. And he was notably, he was an AI researcher that doesn't use AI.
Until this year.
It's gone even further, people making novel math research, like some of the Erdos problems.
Yeah, yeah, yeah.
They're engineers, not researchers, with no background in math, just, you know, using coding.
Mock the math guys.
Yeah, just not math, not research, but you know, solving some of the most, you know.
Unsolved problems.
Yeah, but even in the model architecture side of things, like two years ago, when all of these people started to fine tune models and to do experiments and do model merging, there was quite a bit of research that was happening in GitHub and in Reddit and in local LAM. We're actually like inventing new things and then there were papers.
Frankenmerges.
Yeah, like all of the FrankenMOE stuff, like all of the Axolotl library, like all of these tools. And there were papers published by different companies and research labs one or two years later that were rediscovering what was already done by the Reddit or Discord people without anyone noticing.
Yeah. Do you ever take on auto research? Every AI wave has an auto ML wave.
And this is the auto ML wave of this wave.
Always been a bit skipped. I mean, auto ML a few years ago was mostly like just a.
Parameter search.
Search. Yeah, yeah. Pretty much like ready search in this higher, yeah, parameter space. I don't know, like with Karpathy experiments, it's been quite interesting to see like how things are evolving. I don't know what's your take on this?
Things are just cooler when he does it. I do think some part of this is you're just speed running experiments agentially, right? The agent, the coding agent is more autonomous. You can actually go to sleep and it will do the things that you would have done anyway. So you're just kind of automating things that you would have done.
I see it differently. I think, okay, it will be a very exciting time if we have a move thirty seven from an auto research. If you make an impactful discovery that someone wouldn't have thought of, right? So there's the side of, okay, I have these ideas, go run them in the background, that's fine. But the interesting side is actually when you're shooting off not just paths that you wouldn't have thought about, but you know, trajectories that people wouldn't think about and they work and you make new discoveries. That's the very exciting thing. I think when you have more approach to just token spend and send off, you know, hopefully that becomes possible.
Yeah, I do think the next generation of fine tuners will not be, I mean, will be people that are not coding at all, right? Like one year ago, we had to write like our own code lab with transformers or onslaught or whichever library of your choice. I do think as we like keep evolving, like most people will be fine tuning with a couple of skills, right? Like Hugging Face has skills, like all of these libraries have skills. They will just prompt their agent to kick off like some experiments and see what works, what doesn't work.
And honestly, it's a good middle ground right now. Like all the tools you've mentioned, they let you fine tune in minutes. You don't need to know what's happening on the order.
Yeah. So I think that's worth, like the direction will be like people that just want to do fine tunes to improve their. Abilities for certain domain or like add some like new behavior. They will not be called in the fine tuning code, but of course, if you want to do like deeper research in the architecture. My conscience that most likely this will not be like an automatable, at least in the next one or two years.
Okay, we got to wrap up soon. I just wanted to end a little bit on your, your, the growth in your team. And Paige is here, Logan is over in SF. And you've been hiring all my friends, Thor and Ivan and all these. What does the team look like? Where are you looking to grow?
It's been quite exciting. We are hiring lots of very high agency people. I think maybe three, four years ago, we, we did like a nice interview about how I was growing like a DevX at Foggy Face. And how we were thinking like DevRel should look like. DevRel at DeepMind is also interesting. It's redefining what DevRel should be in an AI, very AI centric organization at the frontier. It's a research lab at the end of the day. And we are in this AI era. So it's also rethinking what DevRel should look like in 2026. We are, yeah, we're hiring pretty much like high agency people excited to build things, to engage with the community and so on. Right now, we are growing in Singapore, so we are looking to hire someone in Singapore.
Coming to A I Singapore.
They're coming.
Yeah, and to hire someone in India. So those are like two locations with huge.
Why Singapore is so important?
So Singapore is interesting. Singapore has a relatively small, but very high, like very dense, high talent community. Now we have a property in the office as well, like it's small, but it's growing quite quickly as well.
Mostly because Yee Te doesn't want to move.
But it's a huge win for Singapore. We don't have research in Singapore usually.
Yeah, so we're trying to grow the team in places co-located to like people doing like traditional DeepMindy research activities. We don't want like to have like people that is in a single town, that is not connected to anyone in person from, from DeepMind. So ideally, if they go to the office, they can talk with researchers doing like their, even if it's a different project. They can be part of like the more DeepMindy side of things. So, so yeah, so we have like people in Paris, in London, I mean, Zurich, in SF, New York. So all of these DeepMind hubs in Singapore now is becoming like a very small, but very exciting hub as well.
Good. This is all the DevRel, DevX team in DeepMind, right?
DeepMind has also expanded a lot here, like a few weeks ago, Kaggle joined DeepMind. How's, how's the org in general?
DeepMind in the past didn't do that much product. And yeah, now we have like AI Studio, the Gemini API, now Kaggle. But Kaggle is part of the team. Actually Kaggle is also here. There are two team members here talking about.
Very excited.
Yeah, talking about evaluations. Last week, they released a new system for agent evaluation. It's like a very experimental initial benchmark, but pretty much allowing agents to take an exam and compete in a leaderboard, which is always fun. Yeah, with Kaggle joining us, I think there are a couple of exciting things. There's the whole Kaggle community hackathon things that enables the community to build hackathons. But there is also the Kaggle benchmarks. And I think Kaggle benchmarks can connect very well with how we think about Gemini and the capabilities. And if you're in the EVA space, like you know, like many benchmarks can be benchmarked. Many people are gaming the benchmarks. And we want to identify like which are these capabilities that maybe we are not aware that we have or that maybe we could improve
and bring all of that feedback from the benchmarks created by the community in an organic way and bring all of that feedback back to the model itself.
I mean, the way we are doing Gemini, Gemini and all of our tools is really like based on the feedback from the startups, the community, the developers. That's why you see like Logan, Paige, everyone in the team talking with the community in social media, in forums, in events, and really understanding what people are building with our tools and bringing all of that feedback to the modeling teams, which is very cool as being part of DeepMind.
Yeah, yeah. Well, you guys are doing amazing work. Thank you so much for joining us here.
Yeah, thank you.
Can't wait to see what's next.
Yeah, thank you for having us here.
Page updated October 24, 2026 · transcript and summary are machine-generated
