World Models & General Intuition: Khosla's largest bet since LLMs & OpenAI

From building Medal into a 12M-user game clipping platform with 3.8B highlight moments to turning down a reported $500M offer from OpenAI (https://www.theinformation.com/articles/openai-offered-pay-500-million-startup-videogame-data) and raising a $134M seed from Khosla (https://techcrunch.com/2025/10/16/general-intuition-lands-134m-seed-to-teach-agents-spatial-reasoning-using-video-game-clips/) to spin out General Intuition, Pim is betting that world models trained on peak human gameplay are the next frontier after LLMs. We sat down with Pim to dig into why game highlights are “episodic memory for simulation” (and how Medal’s privacy-first action labels became a world-model goldmine https://medal.tv/blog/posts/enabling-state-of-the-art-security-and-protections-on-medals-new-apm-and-controller-overlay-features), what it takes to build fully vision-based agents that just see frames and output actions in real time, how General Intuition transfers from games to real-world video and then into robotics, why world models and LLMs are complementary rather than rivals, what founders with proprietary datasets should know before selling or licensing to labs, and his bet that spatial-temporal foundation models will power 80% of future atoms-to-atoms interactions in both simulation and the real world. We discuss: * How Medal’s 3.8B action-labeled highlight clips became a privacy-preserving goldmine for world models * Building fully vision-based agents that only see frames and output actions yet play like (and sometimes better than) humans * Transferring from arcade-style games to realistic games to real-world video using the same perception–action recipe * Why world models need actions, memory, and partial observability (smoke, occlusion, camera shake) vs. “just” pretty video generation * Distilling giant policies into tiny real-time models that still navigate, hide, and peek corners like real players * Pim’s path from RuneScape private servers, Tourette’s, and reverse engineering to leading a frontier world-model lab * How data-rich founders should think about valuing their datasets, negotiating with big labs, and deciding when to go independent * GI’s first customers: replacing brittle behavior trees in games, engines, and controller-based robots with a “frames in, actions out” API * Using Medal clips as “episodic memory of simulation” to move from imitation learning to RL via world models and negative events * The 2030 vision: spatial–temporal foundation models that power the majority of atoms-to-atoms interactions in simulation and the real world — Pim * X: https://x.com/PimDeWitte * LinkedIn: https://www.linkedin.com/in/pimdw/ Where to find Latent Space * X: https://x.com/latentspacepod Full Video Episode Timestamps 00:00:00 https://www.youtube.com/watch?v=A2P3Q3LCoLw Introduction and Medal's Gaming Data Advantage00:02:08 https://www.youtube.com/watch?v=A2P3Q3LCoLw&t=128s Exclusive Demo: Vision-Based Gaming Agents00:06:17 https://www.youtube.com/watch?v=A2P3Q3LCoLw&t=377s Action Prediction and Real-World Video Transfer00:08:41 https://www.youtube.com/watch?v=A2P3Q3LCoLw&t=521s World Models: Interactive Video Generation00:13:42 https://www.youtube.com/watch?v=A2P3Q3LCoLw&t=822s From Runescape to AI: Pim's Founder Journey00:16:45 https://www.youtube.com/watch?v=A2P3Q3LCoLw&t=1005s The Research Foundations: Diamond, Genie, and SEMA00:33:03 https://www.youtube.com/watch?v=A2P3Q3LCoLw&t=1983s Vinod Khosla's Largest Seed Bet Since OpenAI00:35:04 https://www.youtube.com/watch?v=A2P3Q3LCoLw&t=2104s Data Moats and Why GI Stayed Independent00:38:42 https://www.youtube.com/watch?v=A2P3Q3LCoLw&t=2322s Self-Teaching AI Fundamentals: The Francois Fleuret Course00:40:28 https://www.youtube.com/watch?v=A2P3Q3LCoLw&t=2428s Defining World Models vs Video Generation00:41:52 https://www.youtube.com/watch?v=A2P3Q3LCoLw&t=2512s Why Simulation Complexity Favors World Models00:43:30 https://www.youtube.com/watch?v=A2P3Q3LCoLw&t=2610s World Labs, Yann LeCun, and the Spatial Intelligence Race00:50:08 https://www.youtube.com/watch?v=A2P3Q3LCoLw&t=3008s Business Model: APIs, Agents, and Game Developer Partnerships00:58:57 https://www.youtube.com/watch?v=A2P3Q3LCoLw&t=3537s From Imitation Learning to RL: Making Clips Playable01:00:15 https://www.youtube.com/watch?v=A2P3Q3LCoLw&t=3615s Open Research, Academic Partnerships, and Hiring01:02:09 https://www.youtube.com/watch?v=A2P3Q3LCoLw&t=3729s 2030 Vision: 80 Percent of Atoms-to-Atoms AI Interactions This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

• 0:00Speaker 00

  Hi listeners. As you may know, I recently wrapped up the AIE Code Conference in New York. And while I'm traveling, I do like to visit top AI startups in person to bring you interviews that you don't find on any other podcast that just does a Zoom call. General Intuition, or GI for short, is a spinout of a 10-year-old game clipping company called Metal, which has 12 million users, but in comparison Twitch only has 7 million monthly active streamers. Metal collects this data by building the best retroactive clipping software in the world. In other words, you don't need to be consciously recording. You actually just have Metal on in the background while you're playing, and you hit a button to clip the last 30 seconds after something interesting happens. It's very similar to how Tesla and self-driving does bug reporting, if you have ever done a self-driving bug report in Teslas.

• 0:43Speaker 00

  The result is that Metal has accumulated 3.8 billion clips of the best moments and actions in games, resulting in one of the most unique and diverse data sets of peak human behavior, actively mining for the interesting moments. They were also very prescient in navigating privacy and data collection concerns by mapping actions to these visual inputs and game outcomes. As you saw on our Fei-Fei Li and Justin Johnson episode with World Labs, and with the recent departure of Yan Le Cun from Meta, there's a lot of interest in world models as the next frontier after LLMs, to improve on spatial intelligence and to work on embodied robotics use cases. DeepMind has been working on this with Genie 1, 2, and 3, and CMA 1 and 2. And this year, OpenAI seem to finally agree, because they have been pinning on LLMs a lot,

• 1:26Speaker 00

  and they made news by offering $500 million for Metal's video game clip data. Our guest today, Pim, turned down that money and instead chose to build an independent world model lab instead. Coastal Ventures led the $134 million seed round, which is Vinod Khosla's largest single-seed bet since OpenAI. We're able to get an exclusive preview of GI's models, which unfortunately we cannot show you directly. But I can confirm they were incredibly human-like, and we chose to include the first 11 minutes of the demo discussion, even though I couldn't show it to you. It may be hard to follow, but I tried to call out what was noteworthy for you to know as your likely reaction if you were watching along with us. Now enjoy the world's first look, and my first look, at General Intuition.

• 2:10Speaker 01

  It's a completely vision-based agent that's just seeing pixels and predicting actions the exact same way a human would. And so, yeah, what I'll show you here is what this looks like four months ago. So this was, so again, this is just an agent that's seeing, that's receiving frames, and it's just predicting action. So you can see, it has like a decent sense of

• 2:30Speaker 01

  of being able to navigate around. It tabs a scoreboard, just like gamers always tab the scoreboard. So these are purely, these are pure imitation learning.

• 2:39Speaker 00

  I see, so this is slicing a knife.

• 2:41Speaker 01

  Yeah, exactly. So it's doing everything like humans would. In this case, here's, here was the first interesting part that we saw, like it gets stuck. And then it has, they have memory as well, so you see it can get unstuck.

• 2:50Speaker 00

  How long is the memory?

• 2:51Speaker 01

  Four seconds. Yeah, four seconds. So this was four months ago. This was maybe a few weeks after that. So you can, you can see there is like, it's still doing the scoreboard thing, but it's, they're still, they're still quite like, and these are bots too, so you can see that.

• 3:06Speaker 00

  It's very human, let's just say that.

• 3:07Speaker 01

  Yeah. And then,

• 3:12Speaker 01

  right, so this was really like the early days of research, where you can see, right, it does one thing and then goes for another. And then we've been scaling right on data and compute, and also we've just been making the models better. And this is where we are now. So what you're seeing is

• 3:29Speaker 01

  like I said, pure imitation learning. This is just a base model. There's no RL, no fine-tuning. This model sees no game states. It is purely capable.

• 3:36Speaker 00

  Not sequence.

• 3:38Speaker 01

  It's purely predicting the actions from the frames. That's it. And if this is playing against real humans, just like

• 3:49Speaker 01

  like a human would play. And it's also, it's running completely in real time. So there's absolutely, everything here plays exactly like human.

• 4:00Speaker 00

  Do you give it a goal?

• 4:01Speaker 01

  No.

• 4:02Speaker 00

  It just figures out it's a goal, because obviously it's trained on the same.

• 4:04Speaker 01

  Yes. And I picked, right, I picked the sequence where also it doesn't do well initially. So you can see like, this is just like a sequence, a random sequence.

• 4:13Speaker 00

  But this is the, I mean, it looks like it's very well.

• 4:17Speaker 01

  So.

• 4:18Speaker 00

  Oh, okay.

• 4:19Speaker 01

  Yeah, watch.

• 4:22Speaker 00

  Yeah, this will be good.

• 4:25Speaker 00

  Maybe too good.

• 4:30Speaker 01

  This is my favorite part. So you can see it does something that like, here, like, human would never do this. Then gets unstuck, then has four realizes which, and then in the distance.

• 4:49Speaker 00

  So you're saying, one, it makes a mistake that a human would never make, but it unstacks itself. And two, what we just saw is it is doing superhuman things.

• 4:57Speaker 01

  Yeah, okay. Yeah. I mean, there are things that, that, that even Sid, obviously. But because it is trained on, on the highlights of things that all the exceptional things, it's inheriting those. Yeah. So it's not like Move 37 where we are all their way into something.

• 5:12Speaker 00

  Yeah, we're replicating a superhuman.

• 5:14Speaker 01

  Yeah, exactly.

• 5:14Speaker 00

  Or like peak human.

• 5:15Speaker 01

  The baseline of our dataset is peak human performance.

• 5:18Speaker 00

  Yes.

• 5:18Speaker 01

  Yeah. Okay, so that, that's the agent. So now I'm going to show you is we then are able to take those action predictions and we're able to label any video on the internet using those actions. So, and so this is, this is just frames in, actions out. Yellow is the model prediction, or sorry, yellow is ground truth, purple is the model prediction. And then bottom left is a compound error over the entire sequence. And then this is reset per prediction.

• 5:56Speaker 00

  Reset meaning you have any, no, not any reset?

• 5:59Speaker 01

  Yeah, so this just means it resets the baseline. And so this basically, a, a single error in the entire sequence compounds here, but it doesn't compound here. If that makes sense. Yeah. So, and again, this is just seeing frames, right? It's, it's not, it's not seeing any of, any of the actions. And so, you know, so what we did, right, is we, we, we trained it on less realistic games. So we transferred it over to a more realistic game. And then, and this is where it gets really exciting. We transferred it over to a real world video, which means that you can use any video on the internet as pre-training.

• 6:33Speaker 00

  What is it predicting?

• 6:34Speaker 01

  It's predicting it as if you were controlling and using keyboard and mouse. So if you were, if you're basically playing this sequence as the human.

• 6:42Speaker 00

  Is there some sense of error?

• 6:45Speaker 01

  So that's why you transfer to more realistic games first.

• 6:48Speaker 00

  Yeah.

• 6:48Speaker 01

  And then you transfer to real world video because you can't get a sense from ground truth from, from the real world video yet.

• 6:55Speaker 01

  Let's see, and then, so they don't, so I'll show you here. This one is also, this is the same agents that I just showed you.

• 7:09Speaker 00

  This is playing against other AIs?

• 7:11Speaker 01

  This one's playing against bots, yeah. The previous one was against players. But with the sniper, it doesn't really matter that much, as you'll see. It's like, so one, one thing that's really interesting is you notice that it behaves differently as it has like different items, right?

• 7:27Speaker 00

  That makes sense.

• 7:28Speaker 01

  Yeah.

• 7:28Speaker 00

  Intuitively.

• 7:29Speaker 01

  Yeah.

• 7:31Speaker 00

  I think there's also a question about ego centricity versus like so the third person.

• 7:36Speaker 01

  Yeah.

• 7:37Speaker 00

  Doesn't matter.

• 7:38Speaker 01

  Um, the third person I think will be very, very helpful if you're, for instance, trying to control multiple objects in an environment later on. Right now, I think having fully in perception, first person is quite helpful.

• 7:52Speaker 01

  This one's also, this is the policy itself.

• 7:54Speaker 00

  What do you mean this is the policy?

• 7:55Speaker 01

  The agent. Yeah, same for trains that I just told you about.

• 7:58Speaker 00

  Yeah.

• 8:02Speaker 01

  Like this where it hides, that to me was just incredible, like just from, from knowing, being able to predict.

• 8:11Speaker 00

  It's also high when you see it.

• 8:12Speaker 01

  Exactly. Yeah, yeah.

• 8:14Speaker 00

  And it needs the special intuition to go, well, this is hiding and that's not hiding.

• 8:20Speaker 01

  Exactly. And, and, and right while it was reloading, yeah. Okay, so that, so those are, that's the policy and this is a completely general recipe, meaning we can scale this to any environment.

• 8:33Speaker 00

  Is this work cost is okay, no, let's keep playing on demos until I was gonna go into research.

• 8:39Speaker 01

  Yeah, yeah, sounds good. Okay, so and then this is this is these. So what I'm about to show you are the world models.

• 8:46Speaker 01

  There's a few really, really interesting parts about our world models. So the first is I we actually made the decision to transfer. Sorry, we made the decision to

• 9:00Speaker 01

  pre-train world models from scratch, but also we've actually been able to fine tune open source video models to get a better sense of physical transfer. And so one of the things that you'll notice here is like our world models have mouse sensitivity, which is something that like gamers absolutely want, right? So you can have these like very rapid movements, which you couldn't do in any other world model. And so this is a whole that set. So this clip was never seen before at training time.

• 9:24Speaker 01

  As you can see, it has it has a spatial memory. This is this is about a 20 second ish generation. And here's what's fascinating. This is an explosion that occurs. And you can see that in the in the physical world, right, the camera would shake and in the game that would never happen. So you see, you see the the world model inherits the the physical world camera shake, but the the the actual game never does that, which which is which is sort of that. That to us was quite fascinating, right? Also, the models that I just showed you that we used to transfer over from video, the two of those combined will allow us to like push way beyond games in terms in terms of training.

• 10:02Speaker 01

  This is another interesting. So this is a world model. This is rapid camera motion. So again, this is stuff that we're literally just taking one second from here in the context and the actions and replaying it here. Right? And so you'll you you never essentially have

• 10:17Speaker 01

  like what we're saying is the the skill that you see in the clips that like that's speed and the movement and that also pays off at training time when you're doing world models.

• 10:25Speaker 01

  This is my favorite example. So this shows that the world model is capable of performing with partial observability. So what you're gonna see is, again, you're replaying the actions from here in here. Just using one second of video context, everything after that is completely generated. So what you're gonna see is the model is going to encounter, in this case, smoke. Normally now, models break down. What you actually see is it comes out at the same place. And so it's capable of, even with partial observability, still maintaining its position in the world. And then here it is also interesting, so this is, this is sniping. So this is, this gives you like a Reaction time?

• 11:09Speaker 01

  Like the fact that it can do depths and like sequences in completely different views, right? So this is a completely different view than if you were to be outside of that view, right? And so it's, it's able to maintain consistency. While zooming in. Yeah, exactly. And so, yeah, so you can see, so even while this goes out of scope, right? Watch, and then it can, and then it comes back and you'll see it's still, it's still there. Yeah. And so, yeah, this is the work that Anthony Hu has been working on. I'm just wondering how much game footage you have to watch in order to find these things?

• 11:50Speaker 01

  We can ask Anthony, you know, it's, it's, I'm sure he's not gonna be too excited to play these games afterwards. You're not playing, you're just watching. Yeah, yeah, yeah, yeah. Great, okay, so those were the models. See, these are interesting, so we also were able to distill into like really, really tiny models. So this is for instance, a long sequence on a very, very tiny one. You can see it makes like a bit more stupid mistakes. Like it does things that are not as optimal, but I haven't seen anything yet. At the beginning it was running into the wall for free, exactly. I mean, I do that too. Yeah, yeah.

• 12:33Speaker 00

  It's looking, I mean, it's doing pretty well.

• 12:35Speaker 01

  Yeah, and again, all these models are running completely in a real time. There's no.

• 12:39Speaker 00

  Right, so I was thinking, your main model does real time anyway. What's the goal of this setting? Is it cost or?

• 12:45Speaker 01

  Ah, yeah, parameters. Yeah.

• 12:49Speaker 01

  Yeah, yeah, this is the interesting one, it peaks the corner. That's what we mean by like the special and poor reasoning aspect is humans actually, they sort of simulate the optical dynamics of their eyes and how they actually.

• 12:58Speaker 00

  I've seen all the data, right? You've seen all this?

• 13:01Speaker 01

  Yep, exactly. And so like even in like real, this is kind of interesting. Even in like the real world with, for instance, YouTube data, right? You have to first solve for pose estimation. Then, once you have pose estimation, maybe you do something like inverse dynamics, right? Where you basically are able to like somehow label somebody options that you're seeing. And then you still have to account for optical dynamics of like where your eyes actually looking before the decision. Because there's three levels of information loss. Whereas when you're playing video games, you're actually simulating the optical dynamics with your hand, right? And I think that like, that's why I think why games are a better representation of switch support reasoning initially than

• 13:38Speaker 01

  than YouTube videos, for instance.

• 13:42Speaker 02

  Okay, we're in the GI offices with CEO. And the way, welcome.

• 13:47Speaker 01

  Thank you.

• 13:47Speaker 02

  Thanks for having us in your office.

• 13:49Speaker 01

  Yeah, excited to be here.

• 13:50Speaker 02

  If I'm in New York and you're one of the hottest races of the year, I have to come and visit. And thanks for taking some time on the weekends or.

• 13:57Speaker 01

  Yeah.

• 13:58Speaker 02

  So you've raised 133 million seed. So general intuition, most people are being fair about you. Like, I guess this GI is new, but more gamers would have run a medal, indeed. And before that, you ran probably Waystake somewhere.

• 14:12Speaker 01

  Yes.

• 14:12Speaker 02

  The largest Waystake.

• 14:16Speaker 02

  What's your reflection on just that journey of like now you're an AI founder?

• 14:20Speaker 01

  Yeah.

• 14:21Speaker 02

  And you started off playing RuneScape?

• 14:22Speaker 01

  Yeah, I think. As a group of threats, I spent most of my time as a teenager coding and playing video games. So in that sense, it doesn't feel that much difference.

• 14:36Speaker 01

  But I think for, so yeah, so I started the largest private server on RuneScape, worked at Dr. Seb Wardes for three years for Sunnybola, and then on. Like satellite satellite based map generation for disaster response. Which was already like very AI related adjacent. I built some models back then and then started Metal, which became one of the largest social networks in video games. I've always been kind of like AI like Jason, you know, I I I I'm a self taught engineer. So for me, the modeling itself always felt a little foreign. I actually had to take a ton of tons of classes over summer and and and early this year to get better at it. Because I it still felt like like I was really, really good at the infrastructure side. And I had written like our our transcoders for Metal myself. So I was very, very familiar with CUDA

• 15:21Speaker 01

  and like the GPU side and all the video infrastructure that we were using for this stuff. But the the modeling side itself was was still quite foreign. Luckily, obviously, we have I have really, really good co founders. But they they essentially put a bunch of coursework together for me to to go complete, to get really, really good at understanding the fundamentals better. I think for me, I had seen inside of the labs that I had really, really good leadership with fundamentals at top and also the ones that didn't. And I think the ones that did were just like much better. And so for me, yeah, I want it to be more like that. So in that sense, it was a bit, it was first very foreign and then now I feel pretty comfortable with everything. And but yeah, like I think for there's lots to be explored, starting in video games.

• 16:04Speaker 01

  And also reverse engine, like I think the interesting thing about reverse engineering is it kind of teaches you to look at problems very differently. It's like the ultimate form of deductive reasoning in a way. And so so this is just how I think, how I operate. And so for me, it's been a really, really interesting journey. You know, I don't claim to have any of the credentials or or skills that some of the other guests who have had on, but hopefully it will make for a good time.

• 16:29Speaker 02

  Yeah, while your co founders doesn't bring a lot of that different ability and you bring a lot of the, I guess, gaming expertise or well, first, mostly with true cheese.

• 16:37Speaker 01

  We'll see what I bring to the table.

• 16:39Speaker 02

  And just just a little bit of history of Metal, let's establish Metal for those who don't know. The lady, the tricks, yeah, the year, that's you have more active users, concurrent users in which?

• 16:53Speaker 02

  Something like that?

• 16:54Speaker 01

  Yeah, on the Creator side, I think. And the reason is because Metal is a lot more like Instagram than it is like Twitch. So people, um, so the way to think about Metal is it's a native video recorder. Like unlike something like Twitch where you actually have to use other software to record and stream to Twitch. It's not a streaming software. It's actually a video recording software. And a lot of gamers love to put things like overlays on top of their videos. And as a result of that, we have sort of the largest data set of ground truth, action labeled video footage on the internet by maybe one or two orders of magnitude. Yeah.

• 17:25Speaker 02

  What was an example of an overlay like, but the only overlay I usually came up with like Case Can?

• 17:30Speaker 01

  Yeah, yeah. Also, um, controller overlays, for instance, if you're playing, um, like, let's say you're playing, uh,

• 17:36Speaker 02

  Console.

• 17:36Speaker 01

  Yeah, like Flight Simulator, you get like, you know, the joystick and all the things. So you get the actual actions that people take inside the games as well as the frames of the games themselves, which is a loop, right? Because it's essentially you perceive, then you act, then there's a state update, and then you perceive again, you act, state update, which is like roughly precisely what you use in order to trace, to train these agents.

• 17:56Speaker 02

  Yeah, it's, it's almost perfect training data. We were showing, you were showing me in the demo when you showed some B-roll here on, uh, how you don't log key. It's very important for you to log action. Yeah, when did you figure this out?

• 18:08Speaker 01

  Oh, um, maybe starting a year and a half ago. Yeah, and we've realized that like, figuring out this side of the research for us was, we very much never wanted to be in a position where we eroded privacy or something like that. So we never wanted to actually log like a W or A or S and a D, which for researchers, the fact that we don't do that, like often it sounds strange, like why wouldn't you do that? But I think for us to privacy.

• 18:33Speaker 02

  We'll get the data.

• 18:34Speaker 01

  Yeah, I, I think, you know, a lot, a lot of the, um, the researchers, they didn't, they hadn't quite understood yet that you can actually just get away with just doing the actions. Um, and the reason is like at training time, having the actual keys as noise anyways, like if there is text on the screen and you would want to, in theory, uh, make that, um, part of the training, then like reading text from a frame is like really easy. And so for us, if we actually can, so we convert basically, you hit the, uh, the input, we convert it to the actual action. So we had thousands of humans label every single action you can take in every single. Video game over the past year and a half, uh, which is an enormous amount of action labels. Um, yeah. So when you act, you, we, we get the actual, um, action itself. And then it being said, at training time,

• 19:14Speaker 01

  you, you can for like a, the general set of that, of that game, uh, convert back into computer inputs if you want to, but you can never do it for any individual person. And so that for us from, from like a design perspective was, was important. So we, we, we figured all that stuff out. Then we actually started pushing, um, like we already had features as well with this. So for instance, like gamers already love to be able to navigate their clips by like things that happened. So we have a new events capture system and then we also have the overlays where you actually just want to overlay and render the actions on top of your clip. We developed kind of in tandem with the feature set itself. And then obviously when, when world bottles became a thing and it's very, very clear that all the, all the data for this was precisely like that sequence. Yeah, we were able to sort of be first to market, recruit the best researchers and start a lab.

• 19:59Speaker 02

  Yeah, that's, uh, that's incredible. Uh, one more question on metal before I really move forward on the DI. It's been 10 years.

• 20:05Speaker 01

  Yeah.

• 20:06Speaker 02

  What is the, I don't even know how you bro something like this, you know what I mean? I'm just kind of curious and yeah, I'd like the opportunity to ask you, what really worked?

• 20:15Speaker 01

  Yeah, um.

• 20:16Speaker 02

  That you became so, so huge because I'm, you're not the only one. Yeah, but, uh, I have a choice performance and everything, but.

• 20:22Speaker 01

  A few things that really worked. I think the first was a lot of our competitors were focused on solving the social network and a recorder at the same time. And that never, like our bet was really that we could get so many people to record with us that we could bootstrap the network on top of that. And that worked. So while everyone was sort of distracted trying to bootstrap a social network, we were just focused on building a really, really good capture tool. And then we got 10s of millions of people to use that, which then we were able to bootstrap a network on top of the share behaviors. We already had like the profile behaviors and the share behaviors, obviously, but the actual content consumption piece and, and the sharing piece really only came after we hit critical mass. It was actually early days during COVID when like the network really accelerated.

• 21:01Speaker 01

  Fortnite happened, which was really important. And I think also the fact that Discord existed, um, uh, made it quite a different time than, uh, when other types of networks of these types had launched. Cause Discord essentially was like the connective tissue already between gamers that like never really existed before. And so I think those combination of things really, really made it. I think we also built a product that, for instance, with, with most video recorders, you have to remember to start and stop the recorder. So you have to go into the application, then hit start, then start your game and then, um. You know, maybe you'll play games for three hours and you'll close the game, then you have to close your video application. Then you, well, then you have to process like a multi-gigabyte file. Then you have to upload those somewhere. And so like this was a pain for people.

• 21:44Speaker 01

  And so what we did is we just ran this kind of recorder. When you hit that button, it does a retroactive video record. So all the recording initially is in memory. And then when you hit that button, it exports only that sequence to disk and syncs it to your phone. And so that that became super popular. It also, what was interesting about it also means that you're not sort of behaving or acting differently because it's always there and you can just export whatever happens, which is also very, very helpful for, for training, obviously.

• 22:07Speaker 02

  Um, but you weren't the first to do that.

• 22:10Speaker 01

  Yeah.

• 22:10Speaker 02

  The, the thing you were explaining just before this was, is similar to Tesla does their bug reports. Right? You're driving from the, having to disengage autopilot.

• 22:18Speaker 01

  Yeah.

• 22:18Speaker 02

  You're like, they're like, well, tell us what happened.

• 22:20Speaker 01

  Exactly, exactly. See, see, you're driving. Tesla doesn't want to train on the like 10 hours of you driving through a desert where nothing interesting happens. You have the clip button on the steering wheel, something interesting happens, either while FSD is engaged, and I'm not sure if you can use it without FSD as well. But you hit the clip button and it basically uses that precise sequence to mark, which is then more helpful for training because it's more unique as a training time.

• 22:44Speaker 02

  Yeah, yeah. I mean, so one thing I, we're going to get to this on the inside, one thing I, that's, that does pop up as well, a lot of life is boring. A lot of life is going from a, a lot of playing games is doing the boring stuff that is not applicable. Uh huh. Somehow you see the generalized fight.

• 22:59Speaker 01

  Yeah. Yeah, yeah, it makes you think, right?

• 23:02Speaker 02

  It makes you think.

• 23:03Speaker 01

  Yeah, yeah. It's also quite interesting. Like I showed you the models, like what happens when you increase the size of the context window? Um, and how behaviors actually are largely shaped by the size of the context window.

• 23:14Speaker 02

  Yeah.

• 23:15Speaker 01

  That, that to me was like one of the most interesting, uh, uh, parts about the research. Um, made me think about our own behaviors in a way.

• 23:22Speaker 02

  Yeah. Let's talk about also the thing for being a team. Uh, on your website, you're 12, I don't know, that's changed now. Uh, me for the three co-founders.

• 23:30Speaker 01

  Yeah.

• 23:30Speaker 02

  And just let's talk about how this team comes together because you may not, as you yourself thought, you don't have that at the beginning that were, or you manage the elements, people.

• 23:39Speaker 01

  Yeah. I started reading. All the research papers. By that time, I was already pretty deep into like having a decent understanding of not world models in particular, in particular L L M's and transformer based models. And so there was genie, there was Sima. Those two were really, really interesting. And Sima in particular was interesting because what they do is they basically take 10 games. And then they, they have a graphic in Sima where you can see kind of the precise actions that are inside of those games that they mapped. And I believe they found something like 100, which are actually actions that also exist in the real world. And what they did was they then, I believe it was specifically for navigation.

• 24:22Speaker 01

  They did a nine one holdout set, so they, they, they trained an agent on the nine games. And then they had it play the 10th game, the holdout game. But then they also trained a specialized agent just on a 10th game and they compared how good they did. And if I recall correctly, it did roughly as well, playing the 10th game on navigation specifically on the holdout, on the nine game agent than it did on the one game agent. And that's what was really interesting because that's precisely the type of data that we had, right? And so for us, the thinking was, okay, what if we did exactly what L L M's did? What if we use right this, right? So L L M's for train on predicting like text tokens on words on the internet? What if we predict action tokens on essentially what is the equivalent of the common crawl data set, but for interactivity?

• 25:08Speaker 02

  Vision input. Yeah, action output. Correct. That's it. Well, I think, well, actually I'm going to double back a little bit to you, like a question I had, which is one of the, one of the reasons why I thought you would want to prefer keyboard and mouse over actions is the actions, which is potentially unbounded, right? You can jump, walk left, walk right, but then also look up, look left, potentially. It's, it's unbounded. So it's huge, isn't it?

• 25:32Speaker 01

  Yeah, I think. Yeah, there's, there's benefits to the action space being small to start with. So I think we're, we're going to start with anything you can control using a game controller. But yeah, long term, we want to actually predict maybe like action embeddings and have models sit inside a general action space to be able to transfer out to other inputs as well. Yeah.

• 25:50Speaker 02

  Okay. And then let's basically going on the, on the research side. So genie simba.

• 25:54Speaker 01

  Yeah.

• 25:54Speaker 02

  And then do co-founders.

• 25:56Speaker 01

  Yeah. So there was the diamond paper, there was genie, and then there was Sima. The diamond paper for me was really interesting because they had actually managed to get this. World model called Diamond running on a consumer GPU, I believe it was a 4090 at 10 FPS. And you could play it and they did that on like 90 hours of data, like 95 hours. I think it was 87 hours and I think Aiden Noble, that's it, or something like that. That was just incredible, right? That they had something playable on that little data. So I actually cold emailed the entire group of students and I told them, hey, I think we have this thing. And then it was pretty interesting. So like right when that happened, a lot of the labs also started understanding what we had.

• 26:39Speaker 01

  And so we started very aggressively, multiple labs tried to bring us in in various ways. And they were part of that, like they basically were seeing that happen. And I think for them, that also kind of like solidified how real it was. And then when we chose to do our own thing, you know, initially we thought that we were going to have to just work on world models, right? So, so we thought, okay, the main metaphor of this data set is like GNN is world models. What we didn't realize at the time is that we have so much of this data. So we can essentially do these world models in parallel and take the equivalent of like the LLM bet mostly on imitation learning. And then use the world models after that to get into like our off stage, right? And so for us-

• 27:15Speaker 02

  And eventually getting rid of world models is something that you-

• 27:18Speaker 01

  I mean, ideally you get rid of the imitation, yeah, the imitation learning, but yeah. We essentially realized that we could get so far on just imitation learning. The way to look at it is we essentially like, let's take the LM analogy. We essentially have sort of the internet or like common crawl, if you will. And every single lab is trying to simulate that, right? In order to get similar data, in order to train their agents. And so for us, the reason why we stayed independent and we decided our own thing was we think we could essentially leap every single company that's forced to either be consumers of world models or, or build world models and, and, and take this, take this foundational model bet for spatial temporal agents. And be in a place where, you know, we have a lot of customers years before any of the labs even get there.

• 28:00Speaker 01

  And maybe the most similar comparison is like when Anthropic did with code, right? Anthropic just focused really, really hard on nailing the code use case. Their models are incredible for it, a lot of their customers use it for it. So we just want to become incredible at this spatial temporal agent use case. And likely that starts- Like game simulation and then using world models, we can then start expanding out to to other areas.

• 28:22Speaker 02

  So would you show me a little bit of how you think does generalize of yeah things. Yeah, but although games has come to compare. Yeah.

• 28:29Speaker 01

  Games and simulation, I would I would I would specify it as game engines in particular. So even if you're, for instance, simulating human behavior in Omniverse, because they're trying to create better training data for factory floors, you can use it.

• 28:42Speaker 02

  Yeah, maybe Meta has a similar data set because of the quest.

• 28:46Speaker 01

  I never really asked them. I never really looked into the meta quest specifically. So you need a few things. You can't just like, there's lots of companies that have like maybe recorders. But you also need the public graph, otherwise you can't train on the data, right? You can't train on people's like private videos that they have saved somewhere, right? And so I think you you you need the social network graph components, because these videos need to be on the internet.

• 29:08Speaker 02

  To to rank?

• 29:09Speaker 01

  No, to train on them. Yeah, I I mean, I think I think generally people don't like people don't want to train on like. Like, because these things, they live on your device usually, right?

• 29:17Speaker 02

  Yeah.

• 29:17Speaker 01

  And you can't train on anything that lives on your device. Like, you actually need to go and upload and do your thing, right? For Meta specifically, I think also VR, the scale of VR is still pretty small. The amount of environments in VR that are that that have like consumption at scale is probably in like the hundreds. Whereas on PC, it's probably in the tens of thousands, right? And so you get a lot less diversity. The three dimensional input space of VR is pretty interesting. We see some of this too, obviously. And so, yeah, I do suspect, you know, Meta Meta starts using these types of things, but it's unclear to me whether they can get to like a similar scale of data or diversity on the environments as we can.

• 29:57Speaker 02

  Yeah, there are a lot of challenges there. Yeah. Okay, I want to take this in like a few different ways, but I guess let's let's fill out the papers. Maybe one more to mention is Tyre, which I actually I interviewed a guy offers, but that too seems like the particular insight that could have brought it overseas.

• 30:15Speaker 01

  Yeah, so Anthony Tu, who led the research on Gaia 2 is also one of the engineers that joined our team. So it's all the Diamond, the core contributors for Diamond, and then Anthony. And we just had three more researchers join this week. It's been a good week. And yes, I think a lot of the approaches in Gaia 2 were heavily inspired by Diamond. And then Vincent, who was one of the authors of Diamond, also already was at Wave by the time that I emailed them. Anthony also realized what this was and realized that you could scale world models to a much larger like scale and decided just to make the leap as well. So I think everybody that sees the dataset makes a leap because it's, but it takes a while to wrap your head around it. Because it's like, oh, it's video games, right? Like intuitively, it doesn't make sense.

• 30:58Speaker 01

  And then when you actually understand and you see, right, how we've been able to transfer it to a physical world video and things like that, then it makes sense and then everybody tends to jump.

• 31:07Speaker 02

  Don't call it video games, probably.

• 31:09Speaker 01

  Yeah, if I lived in San Francisco, maybe I would.

• 31:14Speaker 02

  Just a quick note, because we actually cover all these papers in the latest paper club. CMA 2 did not seem to have as much impact on CMA 1, and I don't really know why. They did a lot more work. G3 had a ton of impacts, but I also felt like because you can play with the model or people, it just seems an extension of all those things. I guess any quick takes on CMA 2, G3, which were both this year's?

• 31:38Speaker 01

  Yeah, I'll talk about CMA 2. The steerability of CMA 2 was to me the most impressive part because lighting up the action sequences and the text conditioning is quite hard to do, right? And so that, and the fact that they were, like, it's also quite interesting that it means that they can sort of use Gemini as part of the flywheel, right, where you can sort of scale this orchestrator as like an independent, almost like a puppet master, if you will. And then like in theory, Gemini could orchestrate many instances of CMA, right? That to me is the most interesting part is where I tend to agree with this, where like, I think our models will initially be used as like, like you'll have like an orchestrator VLM of sorts.

• 32:21Speaker 01

  That's kind of like managing instances and instructing them. And I think Sort of seeing us showing that you can do this was fascinating. Also, the fact that you could, they didn't just have text conditioning, but they also were able to do like drawings and markings of where to go. They really took an interesting end to end approach to me that I look forward to seeing a lot more of.

• 32:42Speaker 02

  Are you talking to them? Is that the one collaborative?

• 32:45Speaker 01

  Yeah, I, I think the, um, yeah, we're very friendly with DeepMind. We like them a lot. I just saw the team not too long ago and I think, you know, big fans of their work.

• 32:55Speaker 02

  The, the, the text shade from Alice Hees coverage of you.

• 32:58Speaker 01

  Yeah.

• 32:59Speaker 02

  Is you are the biggest bet that can be no cost as many since opening.

• 33:02Speaker 01

  Yeah. How did that conversation start? Okay, so what, if I know it's style and maybe I'll get slapped in the fingers for revealing this or whatever, but.

• 33:12Speaker 02

  Forgive me if I forgot.

• 33:14Speaker 01

  Is, he asked you to like draw a 2030 picture of your company and I think he just picks N plus five years, but whatever, I don't know.

• 33:21Speaker 02

  I did the same for you.

• 33:23Speaker 01

  Yeah. Um, he asked you to like walk that back from first principles all the way from today. And, and, and he asks, he expects you to do that flawlessly where he can challenge any assumption, any part of the vision that, that, and he asks the questions, right? He has a very technical background. He also has a bunch of technical people on his team. And he truly backs people that have these like very large visions on that vision and the ability to defend it alone. And that's what he did for us. And I think that's why I made that bet. So I think also through this, uh, through, through this question, he, he gets to know a lot of things about how technical you are. He gets to know how well you think from first principles, because if that, if that vision is not connected to something real,

• 34:07Speaker 01

  it's very easy to suss it out by asking good questions. And then, and then he just backs fully, I think, like he, he really gets in your corner.

• 34:17Speaker 01

  If it's the right fit and yet they've been incredible partners. They they've opened so many doors for us.

• 34:23Speaker 02

  I had the after question, I think it's like it's a very notable story. Obviously, a lot of work went into it and but it's also worth it and come out of.

• 34:32Speaker 01

  Yeah, for sure.

• 34:34Speaker 02

  One of the things also wanted to, I guess I could ask this question out of sequence, but one of the things that exactly about telling to you is there are a lot of people like you who are founders of business and businesses that along the way have a ton of data and yours happens to be highly valuable. You pursued before deciding to do any independence journey, they also talked to other companies about potential licensing or a position in something back. What is your learnings from those periods? Also like one one version of this is very simply, how do you value data?

• 35:08Speaker 01

  Yeah, I don't think you can value it unless you actually model it yourself and see what the capability is. So that's my that's my real outcome.

• 35:16Speaker 02

  You see model, but tune them up.

• 35:18Speaker 01

  Yeah, but that's obviously like not doable for for everyone. And also, I think my general advice would be as model capabilities increase, you and models are also like, you know, these few L M L, they're very, very good at labeling as well. Generally, right? What I was afraid of when I was having some of these conversations was, okay, like, you know, as the capability is increased, you're just gonna need less ground through data and like you can do more model based data generation or synthetic data generation. I would recommend if you're gonna do large data deals, like just try to get like a large chunk of equity in the company that you're doing it with. If you can. Now, a lot of them won't do this, but I think that to me would or just

• 36:00Speaker 01

  go do the research, figure out what's actually possible. In our case, we were quite lucky in the sense that this is actually the foundation data. Right? And I think, right? Like, that's not true for for every data set. I think, you know, we just happen to hit a particular gold mine.

• 36:16Speaker 02

  But you also did, you read Kiprady, you did the action thing one or five years ago. Yeah, so you did it work.

• 36:23Speaker 01

  Yeah, that's the thing. You have to be grounded, right? And I think a lot of the, and I think that's the hard part. And I think a lot of what's interesting is you can also kind of look for if like scaling laws already exist on your data type, which like for video, there were some, but for these like input action labeled sets, there really wasn't any. The other question is like, does it go into L M's? Does it go into world models? Does it go into like, what type of model is it going to be used for? And I think that's an important thing to know. And so I just want to, you know, if you're having these conversations with labs about data, just like make sure that you actually understand like what it's going to be used for. Cause that's a very, very good way for you to like make the decision yourself about what are you want to pursue that now? A lot of them won't tell you that.

• 37:06Speaker 01

  And I think, you know, I think in that case, you generally just don't want to do it. Because like, I think, I think for our case, like we really cared that like, for instance, there weren't going to be competing products with game developers built, right? Cause we didn't want to like bite the hand that feeds us. And I think we are a part of the games industry. So those questions I think are normal. And then we eventually decided, you know, you just have the data, we're just going to go do it ourselves. And that's when the rest happened. Yeah.

• 37:29Speaker 02

  And he assembled the team and then figured out that I feel like that's you've aligned a lot of stars in order to make G I happen.

• 37:37Speaker 01

  Yeah.

• 37:38Speaker 02

  That other data founders, they're at the beginning of destroying again.

• 37:41Speaker 01

  Yes.

• 37:42Speaker 02

  Or I'm data founder. Founders who happened to have video, but they had a main business, right? I don't know if either.

• 37:48Speaker 01

  There's two sides to this, right? There, it's really easy to be super naive about it. And like I had a lot of people tell me initially, oh, it's not that valuable. You're just like making this up. And, and, and so for me, like doing the work and actually understanding it myself was a really, really big part of, of building that confidence and go start the company. But a lot of times it is true that like model capabilities increase so quickly that like the certain data you just don't need anymore.

• 38:12Speaker 02

  Yeah.

• 38:12Speaker 01

  And so I think it is, it's really important to like get people to do the work such that you can make these types of distinctions.

• 38:20Speaker 02

  Yeah.

• 38:20Speaker 01

  And, and, and so my recommendation would be go build models with your data. See if you can create any sort of capabilities that, that aren't clearly already there.

• 38:27Speaker 02

  Yeah.

• 38:27Speaker 01

  Or on path to being there and then figure out where you go.

• 38:31Speaker 02

  Yeah. I did want to ask this earlier, but you gave me the opportunity to, when you say do the learning, you do post work and all that? And your co-founders gave you some homework?

• 38:38Speaker 01

  Yeah.

• 38:39Speaker 02

  Is this like?

• 38:40Speaker 01

  Some books, I mean Coursera. No, this was Francois Flores. So he has a little book of deep learning and then he also has a full course that he's published on his website. I went through the entire course over the summer. I believe it's like something like 30 or 40 lectures, which also take home projects and things like that. And I would recommend anybody does this. It goes through, right? History of deep learning, like the topology. It takes you through the linear algebra, the calculus. Eventually end up with like chain rule. And by this time you've done like all the more important concepts. It takes you through how do you create neural networks using these concepts that you've learned. Wow, this is super first principles.

• 39:23Speaker 01

  This guy, and I've had the opportunity to spend some time with him as well. He is one of the most first principles people I've met in my entire life. I'm convinced, I actually asked him, why did you do this course? He said, oh, because I thought all the other courses weren't right. And because he is so first principles and he can only explain things from, like everything you see and how he explains this thing, it's everything is from first principles, including like the history of deep learning itself was part of the course. And yes, he goes, so he goes through everything and then, and by the end of it, I think I now have like a pretty good intuitive understanding of how everything works, but obviously still, right? Like I like to describe it as, I'm like the guy who just got his driver's license.

• 40:05Speaker 01

  I can drive the car and like my co-founders are like the F1 drivers that like have done this for years. They know where all the, where all the gaps are. And so I enjoy getting to learn from them. The cool thing is also that word models is just like a very, very new space. And so, you know, I got to bring ideas to the table that like everyone's thought of and not because I'm great at this, just because it's such a new space that like people just haven't tried it yet. So let's get a hit on definition. Yeah. What are word models to you? You know, in a video model, you might predict the next likely sequence or the next most entertaining, the next most entertaining frame. What world models do is they actually have to understand the full range of possibilities and outcomes from the current states.

• 40:49Speaker 01

  And based on the action that you take generates the next states, right? So the next, the next frame. And so it is, it is a much more sort of complex problem than, than traditional video models. So to me, it is, it is a world that is accurately generated based on the actions that you take as a result of what's already been generated.

• 41:07Speaker 02

  And just to fact check, that is, it needs to understand physics. It needs to understand if I'm building a type of material, you need to how it interacts with some type of material.

• 41:18Speaker 01

  Yeah, I think the interactions is the most important part. I think the reasons why world models are so fascinating, one of the things that I did when I was studying over the summer was I tried to actually build a super rudimentary PyTorch-based physics engine, which I would not recommend writing a physics engine in PyTorch for obvious reasons, but I wanted to be able to, because it's a French shell, so you can, you can generate the.

• 41:39Speaker 02

  It's very fat, oh, bad.

• 41:40Speaker 01

  Yeah, exactly. You can, and then you can train. And so I wanted to, you know, I got so many people ask me about, you know, why aren't you just using, why aren't you just simulating or generating this data? And I really wanted to understand from first principles, why? And I think the most important thing that I figured out was the compute complexity of simulation goes up really, really rapidly with three variables. First, the numbers of agents in an environment. Second, their doff, so their individual.

• 42:08Speaker 02

  Jewels of freedom.

• 42:08Speaker 01

  Yeah. And then third, the information that each action reveals. So like, for instance, if you, if you have a, if you have a text action or a speech action, the environment can change so much based on whether you say, right, water or fire, that the outcomes are going to be completely different of like how a human would behave in that type of situation. And so. It goes up so quickly with those three variables that at some point you just hit a point where you just want to maximally bet on either video transfer or generation of these environments using world models, because that type of stochasticity is just incredibly difficult. But it's already very, very present in a lot of the video pre-training that goes into into these world models, right? And so I think for for us, it is more so about making a maximal bet on video transfer

• 42:52Speaker 01

  and interacting with things that are difficult to simulate. And the sterability is also really interesting with text than it is on betting against simulation or something like that. And so I think there's still a large market for for traditional simulation engines, specifically in areas where video is really hard to get.

• 43:10Speaker 02

  Is this exactly what the big labs are also saying when they're putting something to that?

• 43:14Speaker 01

  I honestly haven't talked about the big to the big labs. Like, since we started working on them ourselves, I think people are more reserved with what they share with us. Yeah.

• 43:22Speaker 02

  Of course. It came in sense, that's funny of that question. How would you contrast your version of world models with this that they read? Yeah, I'm including.

• 43:30Speaker 01

  Yeah, so I don't know exactly what Yang Kun is doing today. My understanding, it's based on the VJAP, like the JAP approach, which is, so I'll start with Fei Fei Lee. I think what's really interesting about Fei Fei Lee's approach is that you in some way are able to reuse the the the splats, right? In game engines and in things that let you stay in verifiable domain, which I think is a really interesting approach. However, my understanding is they're currently not interactive, which in my opinion is like the whole point of of world models, right? It's environments, they're great environments. And I think from a business perspective, I think they they picked a really important part of the tool chain. But to me, that's not really a world model. But my guess is they'll get there, right? They'll they'll start generating.

• 44:12Speaker 02

  Yeah, they just have to reuse that.

• 44:13Speaker 01

  Yeah, exactly, exactly. And I think, right, Fei Fei is one of the like founders of the entire space. So I think it's going to be really interesting to me on on what maybe that interactive piece looks like for me to really judge their approach.

• 44:28Speaker 02

  I I think. We interviewed just before we moved to Yang, we interviewed her with Justin Johnson, her co-founder. He was he was more focused on the physics side of that.

• 44:40Speaker 01

  Yeah.

• 44:40Speaker 02

  And the interactivity, they just haven't finished that yet. But I I do think that basically that the splats, if you just add more dimensions on, I guess, the forces acting on them, then then you get to drag to the out of the box. Because you are basically, these are virtual atoms that then has all the law of physics applied to them.

• 45:00Speaker 01

  Yeah, I'm uh, I'm excited to see what that looks like when they actually release it. It's really hard, really hard for me to comment on anything. I really like the um, the frame-based approach, because all of our video or all of our training data is in this format.

• 45:16Speaker 02

  Yes, oh yeah, so they were, we actually asked them about this and they were like, yeah, it's possible, but we're choosing this.

• 45:22Speaker 01

  Yeah, yeah, and you can also go from split to frames, right? I'm sure you can write like at some, it's, it wouldn't be easy, like you'd have to actually render out the environment, do the, sure. It's not, it's not going to be a simple problem, but like in theory, it has to be something that you can do if you really wanted to. So like, because it's almost like having a more sort of ground through three-dimensional representation of the underlying world.

• 45:41Speaker 02

  Yeah.

• 45:41Speaker 01

  Right? So I think it's an interesting approach. It might be overkill, right? Uh, you're also dealing with like a much larger like degrees of freedom on the output space, right? So, so who knows how well it scales? I like the fact that like, I think these video models also use things like auto encoders, right? You can actually have the world models predict like much smaller. Um, maybe like a.

• 46:03Speaker 02

  Prism machine or size?

• 46:04Speaker 01

  Yeah, exactly. And then you can use like diffusion upscaling or methods like this to actually um, enrich. And so I think that world models just allow a much more or world models in my sense, for a much more like controlled space that, that, that we know really well. Um, I'm not suggesting their approach is wrong. I'm just, you know, like this is, I think what we really like about it. Honestly, Jan's podcast that he did, I don't remember which one it was, but a long time ago where he, where he basically proclaimed L L M's to be a dead end. Um, I was one of the things that inspired me to do this.

• 46:36Speaker 02

  I think this is very consensus among world models people. Obviously, everyone heard this is like stops with their L L M's and just goes through to world models. I would say that the main perspective, I asked this exact question to Norman Brown from OpenAI. And he was like, well, they learn invisible models. So I, so this basically that we didn't see anything considering there's this and being. Or what are you on a bit down here, everyone?

• 46:58Speaker 01

  Yes, so. Yeah, I, I, I'm not one to proclaim L L M's are a dead end personally. I think um, I think they're actually quite useful. And particularly as orchestrators, like the way I think about is as demons, right? We had sort of a three-dimensional worlds, then we invented text as like a in a way, a compression method, right? So you had, we invented text in order to communicate with each other in a common way, in a way that actually compresses all this information that we are perceiving in three-dimensional space into just like a single sequence. And I think that allowed science to emerge, right? It allowed so many literature, like so many parts of the world that we cherish. So I think it's a critical part of the whole picture.

• 47:43Speaker 01

  I also agree that it's very, very clear that they do build sort of the internal implicit world models inside LLMs. And so I think they'll be very helpful as things like orchestrators. The problem is when it comes to the generalization, I think, text as a generalization backbone. When most of the pre-training is text, right? Or largely text sequences, then I think you want that backbone to be kind of more speech temporal in nature and then also just have text like as one of the, as part of that. And I think the actual argument of LLMs is also, for instance, the autoregressive nature of the prediction itself. So the fact that it's running the entire output, right, through the transformer and

• 48:27Speaker 01

  then in order to predict the next token, which doesn't like, the environment in the real world is continuous, right? It's always, it's always changing. And LLMs kind of just forget about that, right? I think a lot of the, the argument isn't first, right? So I think the fact, the fact that like text doesn't necessarily generalize well to speech temporal context and then the autoregressive nature of the prediction and using text for that, right? So I think those are, those are the two main arguments. I think, I think text prediction is just one of the actions that is going to come out of these, you know, these policies and world models. I think speech and text generation will just be one of the actions that can, that can be a part of that. I think that there will just be labs coming at this problem from both sides.

• 49:13Speaker 01

  And everyone ends up in roughly the same place and the same place will be whatever people think is cool. Uh, right? Like whatever the consumer.

• 49:21Speaker 02

  Whatever is closest to GI.

• 49:23Speaker 01

  Yeah. And so I don't think there's like a clear answer. I think it's really interesting to come to come at it from the world modeling side. But it's also because we have to, right? Cause like text is largely commoditized. We can import all the text.

• 49:36Speaker 02

  Yeah, I think it's interesting intending.

• 49:38Speaker 01

  Yeah.

• 49:39Speaker 02

  I'm intending it makes sense that you can probably recover it's sort of like you're taking a step back. You're studying your branch of the amount of research sheet, but you might actually just end up recovering all the other tech stuff emerging.

• 49:52Speaker 01

  Yeah, yeah, we can import a lot of that research, right? It's a lot of that is on.

• 49:55Speaker 02

  That's really cool on the, on the research side. I saw about the stuff that GI is, what do you say more like the, the, I guess the, the sort of research and products output. You mentioned the word customers. What are your target customers?

• 50:08Speaker 01

  Yeah. So we're, we're already working with some of our largest game developers in the world.

• 50:11Speaker 02

  Yeah.

• 50:12Speaker 01

  Uh, we're also working with game engines directly. And so really what we're doing at the moment is replacing essentially the player controller inside of a game engine. So. Anything that you're currently that maybe like behavior trees or things that you're deterministic coding. We hope to replace with a single API, which is just you stream us frames and we predict actions. And that can be inside an engine or it can be, um, eventually even inside the real world. Hopefully those are then also steerable. So the models that you saw were text steerable yet, but I think we want to get to a point where they're fully text steerable.

• 50:45Speaker 02

  Well, if it's steerable and you's like, well, I want YouTube to share a figure out anything else on the screen.

• 50:51Speaker 01

  Yeah, I think it's, it's sex conditioning on the generation. So yeah, the ability to, to, you're right. We want to get to a point where you can genuinely, and that's why it's called general intuition, where we can sort of can mimic the intuition of all these gamers. Into human-like behaviors in any situation.

• 51:08Speaker 01

  As I mentioned also, the lab is named after

• 51:10Speaker 02

  The

• 51:11Speaker 01

  The The From AlphaFold, which is, wouldn't it be amazing if we could mimic the intuition of these gamers, who are, by the way, only amateur biologists, on his path to, he tried to get an AI to train full depth to generate a lot of data for AlphaFold. And so for us, really, the North Star, right, what we hope to get to one day is being able to represent scientific problems in three-dimensional space, and then have a space in the world agent capable of perceiving that space, and using, hopefully, also the text reasoning capabilities that LLMs have today, in addition to the spatial capabilities, to be able to work on the other side of that problem. So that for us is sort of the North Star. That's why, you know, we're sort of trying to be hyper-focused, spatial and temporal workloads, the same way that Anthropic was hyper-focused code, and use that to then get into organizations and expand from there.

• 51:56Speaker 02

  Yeah. Just as a side note, since you mentioned Anthropic, any idea what they did on this to solve coding?

• 52:03Speaker 01

  No, out of any lab, I probably know Anthropic the least to go on. Yeah, I admired him though.

• 52:08Speaker 02

  Yeah, well, the current working theory is that they had a super lucky role of the duck.

• 52:16Speaker 02

  But, well, and then it compounds from there.

• 52:18Speaker 01

  That sounds like a nice story. I'm sure it's not that.

• 52:21Speaker 02

  Yeah. So, why did the game developers want this?

• 52:26Speaker 01

  So if you're a game developer, how well you're actually retaining players is like, if you have a game that's already at skills, like decently dependent on how good your bots are. So if you're logging in at an obscure time, let's say 3am in America, and your player liquidity is low, then you need really, really good bots to keep those players engaged.

• 52:45Speaker 02

  Is this known, is this a thing?

• 52:46Speaker 01

  Yeah.

• 52:47Speaker 02

  For sure. For like 4D and whatever.

• 52:49Speaker 01

  A lot of human workers it is, yeah. And so if you're like,

• 52:53Speaker 02

  As a human, do I want to play against bots?

• 52:55Speaker 01

  Usually it's not just bots, it's like players mix in with bots because you don't want to play just against bots, but it's better to have a full game than to have like an empty game.

• 53:01Speaker 02

  Yeah.

• 53:02Speaker 01

  And so I think as long as it's part of the environment, I think it's okay.

• 53:06Speaker 02

  That means you also have to sort of grade that skill level.

• 53:09Speaker 01

  Yeah, yeah, which we can do, because we have, we know exactly how good people are at these games. Yeah, yeah, I think for us, bots is kind of like step one, right? So what I was showing you is we're building a general agent that can sort of play any game. In real time, but really that extends into all simulation, right? Like in GTA V, for instance, people are genuinely role playing real life. Right? And so they're actually behaving in quite aligned ways with with the goals they set for themselves. So you have all these examples represented in video games, right? You have truck simulator, power wash simulator.

• 53:40Speaker 02

  Power wash simulator?

• 53:41Speaker 01

  Power wash simulator, where like actually the behaviors that you'd want a nature to be able to perceive are all there.

• 53:46Speaker 02

  Okay. Nice, that's it. Yeah, it's really, it's really how seriously some gamers take truck simulator. And if you haven't seen these tips, you should watch it.

• 53:56Speaker 01

  Yeah.

• 53:56Speaker 02

  They buy the whole like truck driving set and they're doing the job of a truck driver.

• 54:02Speaker 01

  Yeah. What I mentioned to you, we have more people at any given time on metal playing with steering wheels and like truck simulator and these types of games. Then Waymo has cars on the road. Um, it's a ridiculous stat, but it's true.

• 54:12Speaker 02

  Yeah, I mean, it's so, you know, I, I, I used to think that while it's a self self driving, he kind of just needs to play a lot of GTA five. Um, yeah, I mean, it's not, it's not bad for this.

• 54:22Speaker 01

  Yeah. Yeah, our, our bet is not that we can zero shot any of these things. It's just that like the next self driving company can maybe have collect 1% of the data. Because right also, for instance, clips already self select into negative events and adversity, right? And so like a lot of our data set, because already highlights is is really precisely what a lot of these companies spend like their last 20% doing. Right? And I think that's the main argument. If you're, if you're, if you're another company that's looking at what we're doing, I think the thing that people are not, that people won't understand is that anything that you're currently doing in pre-training, as long as your robot can be controlled using a game controller. We hope that we can move that to post-training for you. So our bet is not that we can create the next self-driving car company. It's just that the next self-driving car company hopefully only needs 1% of the data or maybe 10% of the data. I don't know.

• 55:07Speaker 01

  Right? To be able to deliver a really good product.

• 55:10Speaker 02

  Yeah. It's also the, the term that comes to mind a lot is active learning. I don't know if you've, but you should identify with that. It's not. Got less cool for a bit in order to the seems like the only uptrend. Uh, but which, which obviously you have the best data set for the supply intensity. You said negatives, but feeling for negative, it could be negative part of it.

• 55:30Speaker 01

  Yeah, for sure. I think negative events is just because it's the most common term that people use for like. If you're, if you're Tesla, you want the crashes, you want like.

• 55:37Speaker 02

  Right, yeah, right, right, right. But, but it's only gaming.

• 55:40Speaker 01

  It's both, yeah, yeah. So, you know, the, the model that you saw obviously had really, really incredible moments and, and that was largely. Yeah, that, um, uh, that it had a large representation of people at their best.

• 55:52Speaker 02

  Yes, yeah.

• 55:52Speaker 01

  And worst, yeah.

• 55:53Speaker 02

  Yeah, yeah. Amazing. Okay, cool. Uh, and you have, anything else on the customer development side that you wanna sort of flesh out?

• 55:59Speaker 01

  Yeah, um, uh, we're also already working with robotics companies, but again, that. And manufacturing, but the key is that the robot has to have gaming inputs. So we're like, our bet is not that we can transfer over to like higher DOF robots and the keyboard and mouse. This, it's really just that we can move the hard work of, of, of pre-training, hopefully to post-training.

• 56:18Speaker 02

  Yeah, it's a kind of like the foundation model that is a very good basis to start.

• 56:21Speaker 01

  Yeah, you're gonna straight, you're gonna give us frames and, and likely some text.

• 56:25Speaker 02

  Or you license the model to, because they've been a wonder post-training.

• 56:28Speaker 01

  Yeah, our, our business model is initially gonna be an API, again, like the Anthropic API. Um, but you also saw, for instance, some of the video labeling models that we've been able to develop. So, um, the goal is for any company to be able to take in their, uh, their video data as well. And we can create first, obviously, custom versions of the policy for you, the agent. Um, if that doesn't work, then, um, we, we've already working with a customer that, that is doing. We distill a model and, and they, uh, turn that into a product for themselves.

• 56:55Speaker 02

  So people can engage with you on the agent level, API level. People can engage me on the sort of model level. Can you also buy data?

• 57:04Speaker 01

  No.

• 57:04Speaker 02

  All right, yeah.

• 57:05Speaker 01

  We don't sell data.

• 57:06Speaker 02

  Okay, cool. So that's the, that's the business. Um, and is there a world in which? I, I mean, I, I think this is on your landing page. If you are, you know, frontier labs for, for world models. Is there a world in which there is a more sort of application layer thing that you, that comes out, like a chat GPT for whatever?

• 57:24Speaker 01

  Yeah, you're gonna see us launch a few things on, on metal itself that are gonna blow your mind. Uh, as a result of this, this, um, this agent, so I'll leave it to the imagination for now.

• 57:34Speaker 02

  If people to figure it out, you know, and.

• 57:36Speaker 01

  Yeah, on the world modeling side, like, I think one people underestimate is that metal is already one of the largest. You know, video consumption platforms as well. People watch millions and millions of videos a day.

• 57:47Speaker 02

  Whistle.

• 57:47Speaker 01

  So world model based entertainment and things like that. Well, it's not like a focus for us right now. I think we'll be like on the consumer side, we have the ability to move very, very quickly here. Um, and then get it integrated in a way that I don't, I don't think anyone else can.

• 57:59Speaker 02

  Yeah. You could theoretically do video gen like a Sora. Like, and what is, what is that? Is it a, what was the middle one? And middle meals, not, not reas.

• 58:11Speaker 01

  Um, well, yeah.

• 58:13Speaker 02

  Include theoretically generate clips that nobody play. But you know, it's gonna be.

• 58:18Speaker 01

  Yeah, I, I think for us, the games being so human centric is like a really big part of a mix of special. Like I actually, I actually just don't think that would work. Like one thing that we are really excited about though, I'll, I'll give you one sneak peek of what we're thinking about is what if you could literally replay any of the clips that you have inside a world model or your friends can play them? Like I showed you a model that already took part of your clip as the context.

• 58:39Speaker 02

  Since the replay entered that walls.

• 58:40Speaker 01

  But it's also how we go from imitation learning to RL. Right? Cause like it's part of a research roadmap anyways, to make every single, every single clip on metal playable. Um, so, uh, yeah, who's, who is to say that that doesn't apply to just the actual clips that you take?

• 58:54Speaker 02

  Yeah, yeah. We're interested. Can you see more of the RL potential?

• 58:57Speaker 01

  We describe metal as, as the episodic memory of humanity and simulation. So when you take a clip, really the way to think about it is you get the highlight of what is maybe three hours of playtime, right? You maybe get like two to three minutes of the things that were the most out of distribution, right? It is genuinely your episodic memory. Um, of that playtime and simulation, the things that you most want to remember and share. We want to be able to load, uh, and this is the work that Anthony Hu is doing. The reason why we built world models is every crash that you run into in Euro Truck Simulator or American Truck Simulator or a driving game. We want to be able, right? And again, these are ground truth labels. So we know precisely the actions that lead up to the negative events. They're also title labeled when people upload it on to their platform. They say, oh, good, it's a crash, right?

• 59:41Speaker 01

  And so we can select all these events. And if we can put them inside a world model, we can go into, right? We can, we can train reward models to then reward based on how you perform in clips that actually contain negative events, for example. And so for us, it's very much about, right? We can, we can create this, this, this like LLM moment on, I think, an imitation learning, but actually making every single clip on the platform playable at billions of clip scale is how we go from imitation learning to RL.

• 1:00:09Speaker 02

  Cool. We covered a lot of it. Is there anything else that you want to do before we to the grab of the long-term vision stuff?

• 1:00:15Speaker 01

  Yeah, yeah, I think, I think for us, this is a very, very ambitious, long-term bet. We need the best researchers in the world that, that, that want to work on this stuff. It's really exciting not being extremely data constrained. Like, we really get to, like, we get so many learnings every week that we didn't think were possible, and it makes it for, for a joy working here. Also, the other thing is because we have such large data modes, we don't have to be as concerned as the LLM companies about publishing because

• 1:00:42Speaker 02

  We don't even want to be able to.

• 1:00:44Speaker 01

  Exactly. No one can replicate the models, right? And so for us, we really want to bring back like the original culture of, of open research, which is why we did the partnership with QTi in France.

• 1:00:55Speaker 02

  I actually didn't.

• 1:00:56Speaker 01

  Yeah, we just did a, we just announced our partnership with QTi in France, which is an, an open science lab in Paris, one of the best research labs in the world. Eric Schmidt, I believe, funded it in addition to some, some French people. They are essentially acting as the partner that's currently doing a lot of open research on the data. We also want to partner with universities who, because like, we do believe this is the frontier, but it's so data constrained that really no, everyone has their hands tied behind their back right now. And so we want to help fix that. So for instance, we want to work with universities to build like negative event prediction models for maybe like trucks in India on all the truck data where all these crashes occur. We have all these things that we know we can do, that we just have it at the time to do.

• 1:01:35Speaker 01

  And so if you're listening to this and you're maybe an academic institution or something and you want access to some of this data in a research, in educational research fashion, I think we're quite open to doing that because we want to educate people. And yeah, and other than that, we just want to work with the best infrastructure and research engineers on the planet as we're going into scaling, you know, Ron said have thousands and thousands, eventually hundreds of thousands of GPUs.

• 1:01:57Speaker 02

  Yeah, yeah, amazing. I primed you this as the closing question. Yeah, it's like, it's a little bit of a no cost, I'd say 330 percent of those. I didn't know.

• 1:02:06Speaker 01

  Yeah.

• 1:02:06Speaker 02

  So what does GI become in batteries?

• 1:02:09Speaker 01

  Yeah, in 2030, we want to be the gold standard of intelligence. And any sequence long enough is fundamentally spazient and poor, right? Which I think is, so by kneeling, spazient and poor reasoning, you go after the root of the problem of intelligence itself. What the world looks like is we want to have eight, so I sort of group the sequences of AI in three stages and I credit Andre Karpety for teaching this. Bits to bits, atoms to bits and bits to atoms, and then atoms to atoms. In the atoms to atoms stage, I want like, I want GI models to be responsible for 80% of all the atoms to atoms interactions driven by AI models. And the reason, the reason for that is because we were able to unblock intelligence so quickly in robotics, like intelligence is the bottleneck.

• 1:02:53Speaker 01

  That supply chains actually converged on gaming inputs as their, as their primary input methods. And they converged on essentially simpler systems that let us do a lot more, a lot quicker. So we are essentially the 80% market approach and then you have lots of companies that have kind of like specialized, maybe humanoid robot OS stacks that are, that are the other 20. And then so I want to be responsible for 80% of all the atoms to atoms interactions driven by, by these models and be the gold center for intelligence and maybe 100x more in simulation. Because I think simulation will actually be the larger market initially. So I think in simulation. Because you have very little constraints. Also, from a safety perspective, simulation is much easier. So I think a lot of the takeoff initially system simulation. So a lot of the simulation use cases like what I mentioned, scientific use cases. I'm really, really excited about.

• 1:03:38Speaker 01

  And so, yeah, 80% of atoms, atoms interactions coming downstream from these types of the world foundation models and then 100x more in simulation.

• 1:03:48Speaker 02

  Yeah, it reminds me a lot of what Mark and Facila from the Chaz and Fabregas are doing with virtual biology. Because you can do a lot of simulation and you can do, yeah, or you can do it a lot faster with this. Amazing. Thank you for inviting us to your office. Yeah. And thank you for sharing a bit while you're training.

• 1:04:05Speaker 01

  Thank you.