Ep 92: xAI Co-Founder Unpacks the Future of Model Development

Igor Babuschkin, co-founder of River AI and formerly a co-founder of xAI, joins to unpack a career that spans nearly every major AI lab: he led the StarCraft and AlphaCode work at DeepMind, joined OpenAI's reasoning team years before o1 shipped, and co-founded xAI, where he helped stand up the Colossus data center in roughly 120 days and reflects candidly on what it's actually like working with Elon Musk day to day, plus what the Cursor acquisition actually unlocked for Grok's coding models. He also discusses why he left xAI to start River AI, the three bets behind it, and why he's betting on local hardware, not just software, for personal AI. On the enterprise side, he tackles whether companies will actually train their own models or if it's just a cost play, and makes the case that proprietary labs like OpenAI and Anthropic are facing a real business squeeze. He's skeptical that stacking specialized RL domains generalizes the way pre-training scale did, and is candid about the uncomfortable reality that today's frontier open-weight models are almost entirely Chinese. He closes on what's actually needed to push model progress beyond coding into non-verifiable domains, and the broader implications of where AI is headed next.

• 0:00Speaker 00

  Welcome back to Unsupervised Learning. I'm Jacob Effron. We had an awesome episode today with Igor Babuschkin. Igor has been at all the right places, at all the right times, leading a lot of really interesting AI work. He was at DeepMind, where he led a lot of the work around StarCraft, as well as AlphaCode. He was at OpenAI, doing the early work on reasoning. Then he was a co-founder of XAI, where he did some of the heroic work on Colossus, as well as helped get the models up to where they are today. It was awesome to get to talk to Igor about his new company, River AI, the bets they're making around personal AI for companies, for consumers, as well as local hardware. We talked about all his takes on where the ecosystem is headed,

• 0:42Speaker 00

  what's required to improve models beyond coding into non-verifiable domains, and when he thinks we'll get there. We talked about why he thinks the closed-source model providers are actually in a really hard place right now as businesses. And we hit on his reflections on what AI models mean for the future of the world and the policy implications of this as well. Just awesome to talk to someone who's at the forefront of the space on all the questions that are top of mind. I think folks will really enjoy Igor's perspective. Without further ado, here he is.

• 1:11Speaker 00

  Igor, thanks so much for coming on the podcast. Really excited to have you here.

• 1:14Speaker 01

  Yeah, thank you for the invitation, Jacob. We're excited to chat with you.

• 1:17Speaker 00

  You've pioneered so much important research in your time at DeepMind, OpenAI, XAI. You were early to like so many of the things that I feel like define this era we have today, our Firebase reasoning, inference time search. And you know, you recently left XAI to start your own company, River, really focused on personal AI. I feel like there are so many things that our listeners are going to be excited about to get your take on the episode. I want to make sure they understand River, they get your reflections on this, you know, being at the epicenter of all these things that have happened in AI, as well as your predictions for kind of where we're going. You know, as I was thinking about where to begin, one thing I found really compelling is that you wrote some really interesting fiction earlier this year. For those that haven't seen, you know, one piece was this dystopian look at how models ended up uncontrolled, running the world.

• 2:01Speaker 00

  And it had this really interesting ending, right, where your protagonist kind of realizes that they dreamed all of this and, you know, there's nothing, none of this has happened yet, but they still make the decision to kind of take that first step and begin. And then kind of interesting contrast, your second piece was really this like love letter to the machines that come next. And so maybe to start, I'm curious, what inspired you to write these?

• 2:21Speaker 01

  Yeah, I think that's something pretty big that happened to all of us software engineers, AI researchers, on November, December of last year, which is that the coding agents suddenly became so powerful that you can't ignore them. You know, previously, you could, some people use them, some people didn't use them. But from that point onwards, that latest iteration of cloud Opus models, suddenly everybody agreed that, yes, these models are very powerful. Yes, they, they make our jobs much, much easier. And also, while they're doing so much of the software engineering, now that we used to do, that we used to enjoy. So it was a real transformation for all of us. And like many, like many folks, I spent a lot of part of December

• 3:04Speaker 01

  rewriting all of the coding projects that I've ever, you know, wanted to do in my life, or have done, using the coding agents. And you get this incredible sense of like, wow, is anything is possible now. And also, these agents are going to continue to improve, they're going to continue to get more, more capable. And what will happen with the world at large as these models are getting stronger and stronger? And that kind of inspires, the story is very much a modern version of the Sorcerer's Apprentice. So the Sorcerer's Apprentice is very fascinated with magic and, you know, starts to use it, but then it kind of runs away from him. You know, so kind of like in the Disney short, or in the original poem.

• 3:49Speaker 01

  So it's very much a version of that in my mind. And I think we're all becoming the Sorcerer's apprentices now. It's LLMs and agents becoming more and more capable. And I think it's not going to stop at coding. So I'm very, very interested in figuring out where our agents going to go next, what else is going to be transformed. And the more I think about it, the more it feels like everything will be transformed. All of computing really is a potential, is up for grabs when it comes to agents, because the ability to deeply understand what's happening in your life, at your workplace, what's happening in the world at large, and then taking action based on that,

• 4:28Speaker 01

  is something that you can apply to any problem, to any kind of, you know, thing you might want to do as a, as a business, as a consumer. So that's why I'm so interested in personal AI, because I think that's the next iteration, that's the next manifestation of what the agents are going to be able to do for us. The impact will only grow from here.

• 4:46Speaker 00

  I mean, obviously, it feels like, you know, that December moment you talk about, you know, resonated so strongly with so many folks. And I think actually even a lot of researchers responded to your story being like, this really hits, hits hard. We had that moment in November, December on the coding side. And I think a lot of folks are trying to figure out like what's required to get agents to work really well in other domains. And kind of maybe asking, well, is there something special about coding and math, or, you know, the most verifiable domains out there, like, is, you know, are we going to see the same kind of progress in other domains? How do you think about what needs to be done to kind of like get to, you know, even more general purpose agents?

• 5:21Speaker 01

  Yeah, I think there are actually many next steps that we could be taking from here. There are many different directions that open up coding was sort of the central path, the obvious, you know, low hanging fruit, because it's verifiable, because it touches so many different domains, it unlocks so many different things that you can do with the agents. But from here, there are a few different options of where to go. I'd say scientific discovery is a very, very clear one. We're also starting to see the first sparks of agents being able to solve real, serious problems in math and the sciences. That was a big inspiration behind XAI as well. So we were all really fascinated by this idea like, wow. At the time, LLMs weren't really capable of solving hard reasoning problems, but we kind of felt that

• 6:06Speaker 01

  this was the next big breakthrough. And that was a big inspiration behind XAI, why we started it. So I think agents will continue to improve very rapidly on complex coding projects and also math problems, anything that's verifiable. So with math, you can even formalize your theorem and your proof using tools like Lean. And it gives you a good, a good reward signal that helps you improve agents. It also helps you tell if you've actually found the proof as these get more and more complex. And seeing that actually work for real is really fascinating, because, you know, the AI researchers, we've thought about this for some time before, like, maybe formalization will be useful in the future. And so actually seeing it happen in real life is really quite something else.

• 6:49Speaker 01

  But beyond this, one huge bottleneck is data. And so if you want to do great breakthroughs in the physical world, with its material science, this new fundamental physics discoveries, that it's building a better rocket engine, you know, the type of thing that Elon might want to do with, with the agents. That's where we often need to run experiments in the real world. So you need to figure out a way to close that loop with the, with the real world environment. So the agents, when they have an idea, what kind of material to design, they need to get some feedback as to whether that was a good idea, that it worked, that it not worked. So that's a big frontier right now, I'd say, in using agents for scientific discovery.

• 7:31Speaker 01

  So this kind of direction of science and discovery, I think that's a big, big thing that is happening at the moment. That's something a lot of people are focusing on with the, with the agents. And there's a potential there that will get this highly super intelligent AIs that are vastly more capable than any human or even any collection of humans at creating scientific breakthroughs, at making decisions, maybe at predicting the future as well. So I would also broadly categorize that in that type of, type of direction. And then I think there might be other directions as well. So what is the AI going to be that the everyday person is going to use? They probably are not going to use the super AI that costs a million dollars to run once on a question, right?

• 8:14Speaker 01

  So what is that AI that everybody will be using? So I think we might see a bit of a bifurcation there actually where, where there are the super AIs that only a few people have access to. But there's also the everyday, common AI that you and I will be using all the time to be more productive, or to help organize our lives better, just, you know, live better lives, basically. So very much, you know, so the reward function is how much to help the human feel better and get, you know, get more things done, feel better about themselves. So I think there's also tremendous potential to improve things for humanity, improve our everyday lives. But those agents might not actually need maximum capability.

• 8:57Speaker 01

  They might not need to be able to prove the random hypothesis.

• 9:01Speaker 00

  I mean, it already feels like that bifurcation is happening, right, in terms of, you know, the most frontier models being really valuable for a lot of, you know, coding and mathematics tasks. But, you know, for many people that are, you know, ChatGPT users, you know, it's not inherently clear that the more complex models have actually, you know, the ones from a year ago were totally fine for most of what they wanted to do.

• 9:22Speaker 01

  Yeah, yeah, exactly. And I think it's important for the individual to also feel the benefits of AI, because increasingly it feels like there are a few large companies that create these powerful AI models, that control access to them, like get to say like how they will, how they will behave. And for me, it's just very important that everybody feels like they're actually personally able to get value out of AI. And that's a big inspiration behind where I started River recently. Because to me, it feels like potentially a technology problem, right? So we, we've made certain assumptions about how AI should be built and distributed to people. You should do large pre-training runs, you should set up an API and charge people by the token. And that's sort of the model everybody knows about, everybody follows.

• 10:06Speaker 01

  But it doesn't have to be that way. So we can come up with new ways of building AI models, of customizing them, of distributing them to people that maybe will put them more in control of their experience, will put them more in control of what the AI is doing. And it's a research problem, it's an engineering problem. So it's something that I find very fascinating. So that's why we decided to start this new company.

• 10:29Speaker 00

  When did it kind of crystallize for you? Because obviously, your previous experience had been at some of the largest, you know, closed-source players with those kind of models. And so when did this kind of all come, come to be?

• 10:39Speaker 01

  Yeah, so when I left XAI in 2025, decided to do a bit of angel investment in AI safety companies. So I got to meet a lot of different founders in, in the AI community and in AI safety in particular. So people doing, having new kinds of ideas on how we could help make AI more beneficial for humanity in the long run, or to better control the downsides of AI or the risks. And there were a lot of amazing ideas out there. And I just, I found it all really fascinating. But by the end of the day, I'm just sitting around, you know, waiting for all these folks to make progress on their, on their mission and to succeed. And I got very impatient. So I just felt that I wanted to start something new myself, so I can help push.

• 11:23Speaker 01

  Push on this direction. And for me, for me, um, helping distribute the benefits of AI, helping distribute control over AI is a part of AI safety. It's maybe the more, for me, the most urgent thing that we need to do right now because power over AI is concentrating quite, quite a bit. And especially in the US, you can feel that people aren't too happy about that. They, uh, they don't necessarily trust the AI builders or, um, feel like they have their best interests in mind. So I think it's important we show some, some ways in which people can, can feel the benefits, can feel like they're in control of the whole thing.

• 11:59Speaker 00

  Just wanted to take a quick break to say that if you're enjoying the conversation, the single best way to support the show is hitting follow or subscribe wherever you're listening. It makes a real difference in helping us get the best guests and helping others find the show. Now, back to Igor. Well, maybe interesting, uh, time to just talk about your vision for like the River AI as a company, the product, um, and you know what, how you hope to kind of achieve that.

• 12:21Speaker 01

  Yeah. So we, um, when we started the company, we kind of looked around like, where are these technological opportunities? Like, what are the, these ideas that have been under invested into that could change the way that we use AI, that we build AI? And let's run this as a research, issue this as a research project. Let's figure out, um, if this is something that we can make work and then ship to everybody, um, in the world. So we have really three different projects that we're working on at the moment. So one is we realized, um, the tools to help companies take control of AI are pretty much already there. We just have to make it work really, really well. So that's why we started the River API, uh, which is, uh, you can think of it as a reinforcement learning and fine tuning service,

• 13:03Speaker 01

  um, similar to other, um, players like, uh, thinker from thinking machines. But we have our own unique take on it. And, um, the one thing that we're really proud of is our engineering. So we really optimize, uh, this kind of platform as much as possible. Try to make, uh, training as cheap and as reliable, as scalable, um, as we can make it using, you know, all the skills that we've learned over the, over the years. So that's the first bet that we're making on the River API. The second one is, uh, around personal AI and personalization. So we feel like there's an opportunity to align AI agents much, much better with the individual than what we're currently doing. So today we're training AI models, uh, to basically work well for the average user.

• 13:46Speaker 01

  So you're literally pooling all of the feedback, um, all of the results from all of your different users or, you know, during training and you're telling the model, behave the same way for everybody. And our idea is, uh, what if we kind of break that assumption? And we allow the model to behave differently for each individual that it's serving. So, uh, your AI agent will speak differently to you than my AI agent speaks to me. And they're going to have different preferences, different behaviors. This area of research is still pretty early. So there are a few different directions that we're going down, down here. Because obviously all the agents that are out there that are commercially successful, they're all trained in this sort of average population kind, kind of way. But, um, we feel like it's going to feel really amazing.

• 14:30Speaker 01

  Once the agent like truly learns directly from you and gets better every time you interact with it. So, um, something that I'm pretty excited by. And then the third big bet that we're making is on hardware. Because right now the assumption is that the only way to get access to the most powerful agents is, you know, through access to a data center. So, uh, companies like OpenAI, Anthropic, they, um, they bring up really, really large data centers. All of the inference, all the training happens there. And as an individual that wants to use AI, you have to go for their APIs. Uh, there's not, there's not the way. They can decide, um, how it's restricted, whether to get access or not.

• 15:10Speaker 01

  So the, uh, really the ultimate move here is can we, uh, bring the inference compute, uh, locally to the users? So what would it take to actually be able to run a frontier model in a small device that you can have in your office, that you can have at home? Very much a research project because today you're very, uh, you're very memory limited there. So you're not actually able to fit all of the weights of the model onto a single chip, onto a single device. So that's something that we want to figure out, like, how do we bring the, the best models in the world, um, to, to the individuals? So you can, if you really feel like you've got control over it, it's in your home, you know, your, your data is protected, um, as well. So, um, so those are the three bets that we're making. And we're, you know, very much running this, uh, the way that back in the day, uh, OpenAI and DeepMind were run.

• 15:55Speaker 01

  So we're thinking very long time, we're, um, uh, we're taking some riskier bets. We're assume there's going to be research, there's going to be some new ideas that will be needed.

• 16:04Speaker 00

  No, it's fascinating. I mean, I think each of those three bets is really interesting. And so maybe to kind of dig into them, well, we'll start with the last one, maybe on the, on the hardware side. Like obviously you have a, a really broad scope of, of things you're, you're, uh, you're, you know, seeking to accomplish here. Is the motivation there on the hardware side, obviously like ultimately the most controllable AI is, is one you can run locally, right? Then no one can kind of dictate, you know, whether you have access or don't have access. Like, is that kind of the motivation or is there other kind of like tangible benefits you see to end users? That made you, you know, increase the complexity level even further by adding the hardware component to it?

• 16:36Speaker 01

  Exactly. So, um, really the, the control aspect is the most important to us. So we'd like people to be able to take control of their AI experience and really, you know, feel like the model is theirs and not anybody, anybody else's. That's a big motivator. But then as we started to work on this, we also realized there are a few additional benefits. So having, um, this box, um, in your home locally with you could, uh, give you a much lower latency. For example, so it opens up a lot more possibilities when it comes to interacting with the agents via voice or by video input, video output. So there, there actually is some new possibilities for consumer experiences that open up. Once you go with, with this form factor. So that's something I'm very excited about because it might even make the whole experience much better than what we're getting from, from the data center world.

• 17:20Speaker 01

  Yeah. And I think just, uh, the privacy is also going to be, uh, important in the personal AI world. So this personal agents that know, uh, know you deeply, that kind of observe everything that you're doing and then help you proactively. They're going to get access to incredible amounts of sensitive data about you. So basically all of your documents on your computer, um, you know, um, everything that you're saying in the room could, could be stored by, by a personal agent because it just makes it more useful if it has that context for you, right? And sending all of that data out to, to the data, to data center, you know, where there might be, um, some security leaks or, you know, that your data might be trained on, even if you don't, don't like that. That's, you know, that's really suboptimal. So we're also thinking of it from a privacy, uh, privacy point of view.

• 18:06Speaker 00

  I think part of the, you know, vision of the company is like, you know, everyone kind of having their own model. That's like kind of a continually learning to improve and, and be customized to, you know, to that person. As folks think about how these types of models may be built, there's like different schools of thought. One of them is, you know, going into the weights themselves and updating the weights. And other people saying, hey, you might just be able to do this through like a really good memory system and prompting. You know, how do you think about that? And, uh, is that still like, you know, uh, are there a bunch of different research directions you all are pursuing? Or do you feel pretty convicted in the, uh, in the, in the approach?

• 18:34Speaker 01

  Yeah. So for us, it's a pretty open, which of those, um, are going to be the key, the key tools to use. But I would say from my experience building AI models, the one thing that you really, really need is end to end training. So you actually want to train your agents to make use of their, the personal information, to make use of their memory systems if they have one. That's really crucial. And, uh, today often what we do is we, um, take off the shelf, um, proprietary, uh, agents, which have been trained a lot on coding, but not necessarily trained on over long time horizons, helping an individual using their memory to help them. So how, how do you maximize somebody's happiness, you know, that's the reward function? That hasn't really been done yet.

• 19:13Speaker 01

  So for me, that is the key aspect because I've kind of observed, um, over time, the models that were trained for a particular task, um, have performed the best and you really want to train end to end on the actual signal that you want to optimize. So which would be the individual's happiness.

• 19:29Speaker 00

  Are you, are you going to like basically, uh, have the models run in the wild and like hook it up to, you know, some self-reported happiness? Or like a wearable that like, I guess, as, as to close the loop?

• 19:38Speaker 01

  Yeah, that's really something we gotta figure out. But you could also just try to observe, you know, did I end up helping, uh, the, the user, you know? And the models are getting more and more intelligent. So they might be able to figure out, oh, actually I made things worse today, negative rewards. I made things better, positive rewards. So, uh, the, the possibilities are growing there. It's, it's moving into this non-verifiable domain that's still very challenging. So we need to use, um, models to figure out how you're, how you're doing. But, uh, increasingly with models becoming more intelligent, it's getting possible.

• 20:08Speaker 00

  As you kind of envisioned, like the future state of, of River, like what are the kinds of like differences in models that you think of as like canonical examples of like, if, if you and I had different models over time, like obviously there's some set of capabilities that if the general models had, everybody would be happy. And then there's like different things folks might want, like, what are the examples that you, you think of, uh, top of mind?

• 20:27Speaker 01

  I, I think the examples are really endless and we, it's just hard to imagine it because we, we haven't experienced it yet ourselves. But it could be really subtle things from, um, maybe how much text, um, the, the model writes when it, when it replies to you, to what kind of vocabulary it uses, is it more casual, is it more formal? So all of these different aspects of the answer, um, today we have like one, uh, sort of fixed set of how the, a fixed set of responses, how the, how the model is going to respond on text style, um, in which it will respond to you. But in the, in the future, this can be untied and it can go, it can be pretty much be endless because it will affect, um, how it will help you throughout the day. Like when should it reach out to you, which situations, like, is it being too annoying?

• 21:10Speaker 01

  Or is it not really, not helping enough, you know, it will calibrate itself, um, to you. It will know, um, what your, what your taste is in terms of books, movies, music, and it will be able to adapt your, your day to that. It's really, um, the, the possibilities are actually endless. Trying to, uh, be a bit more inspired about like what, what's possible. You can look at what recommendation systems are able to do today. So if you use an app like, uh, like TikTok or, or YouTube, it figures out what your interests are. And allows you to go really, really deep on what, you know, what you would like to see. Um, and the, the experience is totally different from giving the same set of videos to, to every single user. That's kind of what we want to create for the, for the agent, um, world.

• 21:55Speaker 01

  And you can kind of think of it as a recommendation system, agent hybrid. Which hasn't, has never been done before. So that's something we're a bit excited about.

• 22:04Speaker 00

  You could imagine, obviously, the, the parallels in the enterprise space. And you said you're kind of starting the business with, you know, working with enterprises on, on models.

• 22:11Speaker 01

  Yeah.

• 22:12Speaker 00

  I mean, there's obviously been a ton of dialogue in the last months about, you know, owning your own models and the importance of, of training your own models. It feels like today, a lot of people are doing that for, really for like, cost and speed, right? Like finding, you know, ways to run, you know, smaller models. How do you think that changes over time? Like, do you think folks will, will actually try to push capabilities forward? Or like, you know, even, obviously, a lot of, there's a lot of parallels to what you talked about and people having different preferences. Companies have different cultures, ways of doing things. Like, how do you imagine the parallels there?

• 22:41Speaker 01

  Yeah, I, I think today, a lot of people go to the proprietary models first. They have the strongest capabilities, they're used by everybody else. That's a big reason to, to start up with them. And then they might do custom models for speed reasons, cost reasons, maybe privacy could also be an aspect. If you actually have to run this on your own infrastructure, because you have maybe other customers that are, that have more sensitive data. So those are the, the main reasons that, that I'm seeing. But I think a lot of the reason why people go to the proprietary models is that the field is moving so quickly. If you wanted to learn how to train your own models internally at your company and sort of master, deep learning, master reinforcement learning and get the most out of your internal data that you have.

• 23:27Speaker 01

  You have to build up a team, you have to build up infrastructure. And it can be a many month project to be able to surpass the proprietary models by, by that time.

• 23:35Speaker 00

  Right, right in time for the next one, yeah.

• 23:37Speaker 01

  Exactly. There's a new model that's even better. So what, you could have just gone with the proprietary model. And I think that's been the way things have been going for the last few years. But expect that to change now that open models are starting to become extremely capable. And we're sort of starting, sort of harness all the low hanging fruits in terms of what you can do without deeply, deeply knowing, basically the data and the problems that the company is solving. So if I'm an expert in a, in a certain area, my business is building a certain SaaS tool, like an, like an HR platform, for example, can imagine that I have the best, data, have the best understanding of what the customers need.

• 24:18Speaker 01

  And I've thought about the domain quite a lot, much, much more than, you know, Anthropic or OpenAI would think about the product that, that I've built. So really, those companies are in the best place to decide how their agents should be working, what they should and shouldn't do, how to best help the customers. And so far, that hasn't really hit yet because we're still, you know, riding the wave of stronger and stronger proprietary models. But as soon as that runs out and then, then you have a choice, either you give your, all of your data to Anthropic, to OpenAI, for them to post-train their models better, or you start to do it yourselves, yourself, to try to squeeze out additional capability.

• 24:58Speaker 01

  And I think a lot of people will opt to go for the, for the second one because you're kind of giving away the whole foundation of your business. Otherwise, you're giving away the rare thing that makes, gives you an advantage over all your competitors.

• 25:10Speaker 00

  You think the proprietary models, like, do slow down? Because obviously, there's endless amounts of data budgets to, they're seen to go purchase data across a bunch of different domains. You know, I guess I could see a few different arguments, but one might be like, you know, take a, a bank, for example. It's like, you can actually go hire a bunch of people that were ex-bankers and they can label a lot of data for you and do a lot of, you know, do a lot of things in different environments. And suddenly, your model gets pretty good at finance. And I think it's an interesting question of like, one, you know, do we expect the model progress to continue in these other domains? And two, like, how unique is actually like an enterprise's data versus like the army of data labelers that exists out there?

• 25:47Speaker 01

  Yeah. So, so training of these models actually has diminishing returns. And so the, the stronger we want to make the, the pre-trained model and the model overall, the more GPUs we have to put together, the more high-quality data we have to gather. And AI companies have done an incredible job at scaling up, so putting in so much more effort, gathering the data, tuning the models, improving them again and again, adding new, you know, innovative algorithms and capabilities into them. But eventually, something will have to give. So at some point, you know, you can't cover the entire earth with, with GPUs. So eventually, the, there will be a bit of a slowdown in terms of what the AI companies are able to do in terms of, the capabilities of their, of their models.

• 26:32Speaker 01

  And, and I would argue, we're, we're starting to get close to that point. At the same time, there's another thing that's happening, which is these models are starting to get so capable at the, at the frontier that you might not want to release them anymore. Or you might even get criticized, or there might be some regulatory action that prevents you from releasing the next generation of the model. At the same time, open models are getting stronger and stronger. So I think as a proprietary model builder, you're kind of starting to get squeezed and, oh, but you make the model too good. You're not allowed to release it. But then open source is just right, right behind you, getting better and better every month. So I, I think it's not, actually not the best place to be as a proprietary model builder. So I think the, the way out for OpenAI and Anthropic is to innovate.

• 27:17Speaker 01

  So it's the same thing that they've done the past few years that's allowed them to be in the place where they are today. So they have to come up with some, like, really new fundamental ideas about how to make the models more useful, how to, you know, make everybody feel the benefits of, of what, what they're building, explore new domains, explore new ways of having impact, generating revenue.

• 27:38Speaker 00

  Because you think that basically the capability improvements are, are slowing down?

• 27:41Speaker 01

  Yeah, exactly. And, and if, if they don't, then you might actually have to keep the model private. Because they're starting to, to cross this critical threshold where now you really have to think carefully about whether you can give anyone, anyone access to the, to the model. And then on the post-training side, the, the effort that's being made is really incredible. So you're basically assembling all the world's experts on every single topic and then having them work with you to centralize, like, all the knowledge flows into one set of weights, you know, and then gets delivered to, to all the customers. So scaling that operation up is also incredibly challenging. There are also some diminishing returns there. And I think that the way out there is to go distributed.

• 28:24Speaker 01

  And so, so there are all these experts out there in the world already for their particular domains. So they're inside of the existing companies, they're in the, at the universities and so on and so forth. And I think we should give people an opportunity to basically independently create improvements to models and, and essentially post-train them locally for themselves, for their, for their company, for their, for their team, for, for the individual, rather than trying to capture all of human knowledge, essentially, and then, and then sell it for, by, by the token. So just my personal, my personal philosophy. And that's the type of thing that we want to do at River AI. So figure out how do we allow people to, to do their own post-training, essentially, locally.

• 29:08Speaker 00

  It's interesting because I feel like we're in this moment where, you know, yeah, you, you buy a lot of data in a domain and you focus on that domain and that, like, then adds those capabilities and you're kind of going one by one. And it feels very different than, you know, the pre-training world where it's like you get to some certain threshold and then you start to see like massive generalization across the board, right? And I think some people have posited that like the same thing would happen in post-training in RL. Like eventually, you know, I don't know, you're, you're adding domain 30 or 40 or 50 and suddenly it becomes like, like, you get a lot of benefits from having everything in the same place versus, you know, just training on, on, you know, finance in this place or healthcare in this place. Like, do you think we see that generalization and that benefit of generalization down the line for RL? Or is it like going to always make sense to just take some really strong pre-trained model and like, you know,

• 29:53Speaker 00

  apply one domain specifically to that?

• 29:55Speaker 01

  I, I think we're seeing generalization in the sense that these models have such incredible knowledge of the world and they have such a wide set of skills that they can now go into any domain and bring in all the knowledge from everywhere, everywhere else. I think coding is a great example of that. And then I, I think whether, you know, whether there is a lot of synergy between all the different things that are being post-trained, I, I don't, I don't really know, to be honest. I feel like there's, there's so many specialized domains, so many specialized skills out there that you only really need in one, one place. I wonder how much of that really, you know, is contributing to the overall intelligence of the model. A lot of the intelligence seems to come from coding and math, training and things like that.

• 30:39Speaker 00

  Yeah, it seemed a really interesting question for, you know, how much it makes sense for enterprises to to post train their own models, right? It's like there's pharmaceutical companies that have, you know, data sets of of of drug trials that like pretty proprietary. There's, you know, chip design companies like there's certain things that you can pop that you're like, yes, those companies have like quite unique data that that that probably isn't isn't out there. Maybe it's less clear to me if like your average bank actually has like data that isn't just going to be in, you know, that that eventually like your closed source model providers get access to.

• 31:09Speaker 01

  Yeah, we might see some of those businesses struggle if they're not able to retain their competitive advantage, if they're not able to sort of use their own leverage their own IP to to make their own models better. So I think it really depends on the industry and that's kind of how you can guess where things might go and how the transformation might happen. I would say the the worst place you can be in is if all of your knowledge is on the internet. And if everything that makes your company unique and special is already out there and can be can be scraped, that's a very dangerous place to be.

• 31:40Speaker 00

  It kind of exactly ties into the overall thesis of River, which is like, you know, keeping this intelligence local, you know, the the kind of ways you're interacting with and using these models.

• 31:48Speaker 01

  I'd say we have a choice, actually. I think we, as a company, you can choose to give your data to somebody like Intropic and open it in a way by, you know, for the APIs, for for other means, for maybe selling your data to them, even if you want to turn a profit on it. Or you can start to leverage it to build your own models. So I see these two different paths, you know, one obviously leads to more centralization of intelligence and one leads to to more distributed benefits of AI.

• 32:16Speaker 00

  Behind all the stuff you're building, you know, today, it seems like anyone that's trying to build their own models or fine tune models is using, you know, the the cutting edge Chinese open source models. And obviously, there's been a big discourse these past months about about these models. You know, obviously, I imagine you think cutting edge open source models are really good for the world. What do you make of like the fact that this ecosystem is all Chinese today? And then kind of the government, you know, talking about potentially banning these models?

• 32:40Speaker 01

  Yeah, I think first of all, it is good that we have powerful open models out there. So it really lowers the bar for anyone who wants to get into AI, who wants to customize models, who wants to use models to their own advantage and really, you know, have that have the control of their own of their own destiny. At the same time, I know it does put the Chinese labs in a kind of a preferred position where they also can exhibit some some control. So for example, they might stop releasing their their open weights in the future. And every relies on on them. That's not a great thing for for the US, for the US economy. They could also change the licenses to those to those models. So we often see that already that there are certain terms in there.

• 33:25Speaker 01

  If you're a company that has more than X revenue, then you need to enter into a partnership agreement where the rules might be a bit different. So that's another way to people can exert a bit of pressure and stay in control of this of the open weights. And then one issue that's more theoretical, but is theoretically possible is for the ways to somehow have backdoors or behaviors that we would find undesirable.

• 33:52Speaker 00

  How worried about that are you?

• 33:54Speaker 01

  Yeah, today I'm not very worried at all because my my sense is that if these kinds of things existed, we would already have some at least one example of it happening in the in the wild. We haven't seen any reports of it. Also, the the technology that I'm aware of for kind of planting these behaviors, planting these backdoors into weights is not advanced enough today to really avoid detection and to not leave any not leave any hints. So I think we're we're kind of early on this right now and I'm not too concerned. But in theory, the concern is there. And over the next few years, as this tech becomes more more sophisticated and you can actually and maybe LLMs and agents become more important as well

• 34:37Speaker 01

  and all kinds of domains, including the the military. You know, that's it is something that we should have on the on our radar. So I would say it's really critical that we have the best open models in the US. So that's something that I'm I'm very supportive of. And I think we should all, you know, figure out how how can we get a US team to train the very best open model period? You know, not not just the best US open model, but the best best one worldwide. So yeah, definitely so super supportive of that.

• 35:08Speaker 00

  Yeah, I think that'll be an interesting business model question of how you how you end up. But it's like such a public good that maybe we'll find a bunch of folks. I mean, Nvidia already is kind of subsidizing this stuff quite dramatically to to to make it possible. And so maybe you'll see.

• 35:21Speaker 01

  Maybe it's going to be us as well. And so we've we've had that experience with XAI of starting from scratch. And then within less than two years, we had Grok 3 and then Grok 4 as well, which were very much at the at the frontier. So I'd love to to train the best US open model. But right now, we're very interested in how to build a business on these open weights so that the company can be self sustaining. And we're making good progress on that with the RL API with the other products that we're building.

• 35:51Speaker 00

  Well, I'd be remiss if, you know, having you here not to ask a little bit about some of the the past roles you've had, because I think you've obviously seen you've had a front row seat and and kind of led a lot of this stuff that's happened over the past years. You know, maybe to start, like, you've said AI progress took off, you know, faster than you expected. Is there a moment or two that like, you know, really surprised you these past like, you know, I don't know, three, four years?

• 36:14Speaker 01

  But I think the crazy thing is I'm not too surprised by the progress. It's just crazy living through it. You know, it's it's totally different from, you know, when you imagine it, it's just so different from actually living it for real. It's almost like being in a dream. So so when I started to get interested in AI, I was a physicist, was working at the Large Hadron Collider, was doing a PhD there. And in physics, they teach you basically to really think ahead. I think some of that rubs off on the physicists as well. And they started to think about, you know, how how will the future, you know, look like? And so when I realized that, you know, for example, AlphaGo happens, so DeepMind was able to to create a goal AI that's very, very powerful.

• 36:59Speaker 01

  Then for me, that's when it clicked that, you know, amazing things are going to happen. And in AI, and I better switch, better switch over.

• 37:07Speaker 00

  You and a lot of prominent researchers made the made the move from physics. It's very interesting how many folks started in that world.

• 37:12Speaker 01

  Yeah, I think it's just a great way to to bind the ropes about how to think about this this tough problems and how to, you know, solve problems from first principles. So it's a great preparation for for doing AI work.

• 37:26Speaker 01

  And yeah, decided to join DeepMind as a research engineer, which is really fun because I got to see both the algorithms side and how to solve these research problems, how to train the models and also how to build distributed systems, how to improve efficiency of the training of inference and so on. Then worked on text to speech generation with WaveNet and then got really interested in reinforcement learning and ended up being a tech lead for the StarCraft project at DeepMind. So it was a really challenging RL project. The kind of, in some ways, it was ahead of its time.

• 38:01Speaker 00

  A hundred percent. I mean, that that seems to be a much more complex environment than than a lot of the of things we're optimizing RL on today.

• 38:07Speaker 01

  Exactly. We had to figure out how to deal with that complexity. And some of the strategies we've developed, they we could immediately see how they could apply to language models as well down the line. So for example, with StarCraft, we did a lot of imitation learning. So we got all these replays of people playing StarCraft at various skill levels from beginners to to almost professional players. And then we started with imitation learning and only then switched to RL. And the rewards here are verifiable because it's a two player game. Where the wins gets a positive reward. If you if you win, you get a negative reward. And the combination of those two imitation and then it's a large scale training for imitation

• 38:51Speaker 01

  and then reinforcement learning to further improve the policy. That's really, really powerful, we found. And that's still the way that these agents are trained today. So really, after the StarCraft project at DeepMind, we actually realized that, you know, coding is going to be the next big thing. So it was it was a bit early at the time. So it was way before there were any commercial sort of capable commercial code models out there. And we kind of saw it as a natural extension of of StarCraft. And so we started this project called AlphaCode at the time. And I had the strong feeling that models that basically LLMs were not performing well on code at all at the time.

• 39:34Speaker 01

  And I thought the reason was that they're not able to think. We don't give them enough time to think about the problem, basically. So there's when humans solve tough problems, they have to think for a moment or two and something magical happens that allows them to to understand the problem much better and to to solve it. And we didn't have a mechanism back then for LLMs to to do that kind of reasoning. And so I really felt like we better do some research on this and I was working on that. And I realized OpenAI has just started up a reasoning team exactly to to figure out this this problem. So I joined OpenAI, moved to moved to California. And yeah, was both involved with the reasoning effort there,

• 40:18Speaker 01

  but also helped out on some big pre training runs together with with Greg Brockman. So I kind of got to see the the large scale pre training side there as well, which was was really fun.

• 40:29Speaker 00

  A lot of this reasoning stuff seems to have worked post, you know, really large scale pre training. And so it seems like in the early days of the reasoning work at OpenAI, there was almost just this like conviction, like this is going to work, you know, when as as as models scale and get better, which is which, you know, is really impressive intuition.

• 40:44Speaker 01

  Yeah, exactly. And then at the time, we didn't have the solution just yet. The solution ended up being essentially the the O1 model and the the research that happened later at OpenAI unlocks the this reasoning approach where the model has thinking tokens that it's allowed to generate before giving the the final answer and then use reinforcement learning to basically improve its thinking process and thinking ability. So that that ended up being really the key that we were looking for. That happened happened a few years later. And yeah, and at OpenAI, I just felt.

• 41:24Speaker 00

  Like, you know, DeepMind had a who's who of heavy hitters. I mean, I think almost everybody did a stint through there or will bring it at some point. You know, I think as people have reflected on it, it's like, maybe, you know, there was more, you know, different kinds of bets made or more of a belief in kind of reinforcement learning and, you know, less of a focus. I don't know, you lived through both of them.

• 41:43Speaker 01

  Yeah.

• 41:43Speaker 00

  So I'm curious, like, how you, you know, how you thought about that?

• 41:46Speaker 01

  It's really hard to say, honestly, because there were all these extraordinary thinkers at DeepMind, and I think some of the right ideas were kind of floating around for sure. But I think what helped OpenAI was that they were a smaller team and they were able to get great alignment within themselves, great conviction that this is the right approach. I think it requires kind of that critical mass of talent density and people thinking along the right lines, which they were able to do. So even as the underdog, they were able to make huge progress. And I think to us at the time, it was pretty clear that OpenAI was going to win in many ways. And then already at that point, I started to think like, wow, that's kind of unfortunate

• 42:31Speaker 01

  if there's only one AI company in the world that controls the most capable models, that decides what gets done. And we would even think about like, how do we set up UBI for everybody so that humanity can be paid for OpenAI, you know, essentially controls the models, which just didn't sit right with me. Because I felt like, I was always a big open source fan and big Linux users over the years. So I felt like, you know, that's the way to distribute the benefits. In the end, philosophically, if you think about it, the pre-training of the model happens on all of humanity's knowledge, all the text that we've created out there on the web.

• 43:14Speaker 01

  It's really a commons, it's really something that humanity as a whole should own. The fact that nobody technically owns it, you know, and you're able to just download it, basically allows you to compress the knowledge, to refine it, to turn it into something that's actionable, that's useful in the form of the model weights. And that's really what the AI model builders are doing, in my opinion. And to me philosophically, it makes the most sense for those pre-trained checkpoints to go out and be free for people to use, just thinking from first principles, like what's fair here. But obviously, the AI labs are putting in tremendous effort in terms of, in the form

• 43:56Speaker 01

  of compute, in the form of all these experts that figure out how to set up the algorithms, how to train the model, how to innovate in the space. So that's definitely a huge benefit that comes from having them involved, but I feel like we're not accounting for that, all the human ingenuity that went into the text on the web.

• 44:19Speaker 00

  But obviously, the fact that all that data is open, you know, allowed you, in the case of XAI, in short order, to kind of get back up to pretty close to the frontier. And I feel like that, it's interesting to see that play out. How did that come about?

• 44:33Speaker 01

  Yeah, so basically, I ran into Elon, randomly happened to be at the Tesla office and talking with the team there about what they're up to. And he happened to be around, so I got a chance to talk to him, which was, you know, pretty fascinating, because he is such a great thinker as well. So thinking very deeply about humanity, technology, and everything in between. And yeah, so I think we kind of saw eye to eye on all these things, like the idea that maybe there should be another competitor in the world, another AI lab that has a bit of a different mission, a different philosophy around how the models should be developed

• 45:14Speaker 01

  and basically how they should behave when they're out in the wild. So I think we kind of had that connection from then. And then, so when later on, he wanted to start something new, he reached out to me, and that's kind of how XAI started. And yeah, and then from then on, I talked to some old friends that were looking for something new to do, and we managed to get a pretty good group of people together. And I think that's critical for any AI effort, you have to get that critical mass of talent, the right kinds of people that want to work pretty hard and do something, do something great. That was really the number one.

• 45:54Speaker 01

  You know, the question of whether we were able to do it would depend on whether we got the people together. And yeah, and then from there, starting pretty much from scratch, we sort of slowly built up the company and trained the models. And yeah, and then within less than two years, we're able to get to the frontier.

• 46:16Speaker 00

  We did some pretty crazy things, you know, famously like, you know, getting Closte up and running, I think in what, 120 days? And I guess, like, what lessons did you take away from just the speed at which you were able to do a lot of that stuff?

• 46:27Speaker 01

  Yeah, I think it just requires tremendous focus and thinking outside of the box. So that was really very much driven by Elon, because he saw the need for us to bring more GPUs online very, very quickly, to be more competitive, to train Grocery in particular. And so he decided, you know, let's build our own data center, we don't want to depend on anybody else. And he's really the best person in the world to do that, from what I've seen, because he's able to see the big picture of like, what needs to be done, what are the individual components and steps, and then sort of in a very focused manner, figure out from first principles how to accelerate every aspect of the project, and then is able to go out

• 47:12Speaker 01

  and do things that other people might not actually be aware.

• 47:17Speaker 00

  I mean, did you think it was crazy when it was like first proposed on timeline perspective, or?

• 47:21Speaker 01

  I did, but I also knew Elon well enough at that point that I knew that we would get it done. So I also had tremendous confidence.

• 47:31Speaker 00

  Yeah.

• 47:32Speaker 01

  And yeah, and really, the key to making Closte's work was just questioning how everybody else was building data centers. Because out there, when we talk to people, like, how long would it take to build a new data center and put this many GPUs inside, we would get quotes, way, way above a year. And the reason is because they had to bring up a new shell, like a new data center building, and had this very waterfall-like process for building it. And you also often deal with subcontractors of subcontractors of subcontractors. So a lot of efficiency gets lost, or can get lost in the process there.

• 48:17Speaker 01

  So Elon has a totally opposite mindset, like, we want to do it ourselves, we want to be the general contractor, and here are all these ways of solving our problems that other people haven't even realized are possible. So it's kind of how to find a glitch in the matrix, not every single aspect of the project. So yeah, really fascinating working with him on this.

• 48:40Speaker 00

  No, I mean, I'm sure our listeners would be super curious, like, what is it like working with him? And obviously, like, you know, how do you compare that environment to, you'd obviously, you've worked in DeepMind, OpenAI, like, you know, three probably very different environments and cultures?

• 48:53Speaker 01

  I'd say that the reason I was really interested in working with him after that initial chat is that I could sense the respect that he has for engineers. So as a research engineer, he's kind of jumping between research and engineering quite a bit. And yeah, I wouldn't say that there's a necessary double standard at these AI labs, but definitely, you know, you've got titles that kind of reflect your research scientist, your research engineer. And I just felt like really respected by Elon and felt like we were seeing, I tell you, he was another team member, you know, that I was working with. He focuses a lot on engineering meetings, so really talking directly to the people

• 49:36Speaker 01

  during the actual work, dives tremendously into the details of everything that the company does. And you can feel the respect for the engineers, the respect for the problem that they're working on. And yeah, that was a really fun aspect of it. And also, he has this tremendous energy, and, you know, every day he comes to work with a smile and, you know, wants to tackle all the hardest problems, once again, from the beginning. And that energy, you can feel it in the team as well, and I think that's...

• 50:15Speaker 01

  They come in with tremendous energy and motivation.

• 50:18Speaker 00

  Since you left, the company's done some pretty interesting things, obviously getting acquired by SpaceX and then acquiring Cursor. What would you make of the Cursor acquisition?

• 50:26Speaker 01

  Yeah, I think it's an incredibly smart move. I think it's basically allowed him to jump way ahead on coding models. So Cursor has a great team that's very familiar with coding. There's products that a lot of people out there use, and I've also used Cursor quite heavily. And there's a lot of data that you can use to kickstart your training and tremendously improve over what the states were. Exa was that.

• 50:55Speaker 00

  Yeah. And so is that coding data from the real world, from people using it, like just that much more valuable than kind of the stuff you get from data labelers and then like the other suppliers?

• 51:04Speaker 01

  I think the quantity of it is really, if you have it in quantity, that's really, really valuable. There's also a tremendous value in building the right kinds of RL environments. So basically setting up these verifiable tasks where the agent is tasked with doing something, fixing a bug in the codebase, and then afterwards you check, did it actually do it correctly? And also, did it do it in the right way? And that's really what's been able to deliver a lot of improvements in coding models. And that's the kind of the other side of the coin. Any kind of one, both types of data in the process.

• 51:42Speaker 00

  Yeah, but obviously it's way easier to figure out what environments to build, having seen these things deploy. I'm sure way easier when it's actually your own model in the Cursor harness. But even Claude or whatever in the Cursor harness is probably still really valuable.

• 51:54Speaker 01

  Yeah, and obviously I don't have any insight into what kind of data the Cursor team brought in and how they were able to make such a huge leap in model performance. But I'd say it's super impressive how quickly they managed to ramp up the coding abilities of the GROK models. And I'm sure we haven't even seen the biggest jumps yet. So there are some additional models that are coming out.

• 52:16Speaker 00

  What do you feel like the biggest constraint is on continued model progress right now? Is it like, is it data? Is it like just kind of getting, yeah, get all this expertise that lives within organizations that hasn't been fed in? Or how do you think about that?

• 52:29Speaker 01

  I think the biggest unlock would be just new ideas around how to set up the training such that it can handle much longer time horizons, such that it can handle non-verifiable rewards much more easily. And I think there's a big inertia and there's a big energy barrier to getting these things to work. And it's kind of easier to continue to iterate on the coding regime. So I think one trap here would be just to continue to make the coding better and better. But I think what people really want to do is figure out how to move into those longer term domains. So for example, one quantity that can bottleneck your training is how long does it take to roll out your agent directories?

• 53:13Speaker 01

  So if you're in order to update the model, you first do trial and error. You have your agent attempt many, many solutions, you collect the rewards, and then you can use reinforcement learning to update the weights of the model. But if your rollout takes 24 hours, you're going to have a really hard time training the model, because every single training step will take very long. So we have to figure out ways of slicing up the work that the rollouts, the work that the agents are doing. And so incrementally, being able to incrementally train them, maybe agents have to guess that they do well, that they're not do well, based on incremental progress. And then moving into all these domains where we don't have the rewards,

• 53:54Speaker 01

  where we can't have a unit test, or we can't have a proof that's, or verification of a proof. So that's really where people should be pushing today if they want to improve agents.

• 54:06Speaker 00

  How close do you feel like we are to cracking some of the non-verifiable domain problem? I guess maybe back to your, like, you know, AI scientists and the ability to kind of like, you know, have a novel physics discovery or something like that. Do you have a gut intuition as to how close we are to some of that stuff?

• 54:23Speaker 01

  I think that the models are so capable now that they can often make amazing judgments about whether something has worked or not. But we still haven't seen great examples of people implementing that at scale to improve the models further. So I think that's something that we might see in the next few months, next 12 months or so. People are actually pulling off these LLM judges, the approaches with LLM judges where really, there is no verifiable reward, but we're still able to learn a lot from rewards interactions.

• 54:55Speaker 00

  Yeah, but you think people, like the recipe is kind of known and it's just literally about running that experiment?

• 54:59Speaker 01

  Yeah, I think people maybe haven't figured out all the details of the recipe yet, how to optimally do these things. But the thing that's been helping is just the models themselves becoming so strong. Now that you can often trust their judgment, they're often very reliable. It's depending what's happened. So that's kind of something that lifts the whole thing up.

• 55:21Speaker 00

  Yeah, I'm curious. You've obviously, it seems like had conviction early in a lot of like, you know, ideas that proved correct and obviously, you know, super early to reasoning and a bunch of these things. In the last few years, like, or maybe in the last year, anything you've changed your mind on that you kind of actually felt pretty strongly would happen in the AI world and has gone in a different way?

• 55:39Speaker 01

  But the thing is, I think nobody could predict how transformative the coding agents would be once they crossed a certain threshold of ability. So I was expecting kind of smooth progress on coding models. They just continuously get better and eventually they'll be used more and more. But the sudden explosion of usage was really, that was really something else. And now I'm starting to think, maybe we'll see more of these transformations in the future.

• 56:08Speaker 00

  Yeah, there's just like a step change moment where something, I mean, I feel like everyone felt that around like November, December. And it's even hard to articulate even like what it is, but it just like, it just starts working so much better. And you can imagine, yeah, and like other domains, you'll reach some sort of point where that becomes apparent. But it's kind of hard to predict. Yeah, I think what's fascinating is even the folks closest to those models, like it was hard to predict a priori, like that's when the breakthrough was going to happen. I was thinking about where to end and one thing, you know, maybe to tie it back to where we started, which was really around you kind of writing this fiction and thinking about like, you know, where the world's headed and there's obviously, these are questions that motivate River and your founding there. You know, I feel like you've written the question, obviously keeping all, many people up at night is,

• 56:52Speaker 00

  you know, if machines can do all this work, like what's left for people?

• 56:55Speaker 01

  Effect the future if we want to stay relevant in the world as humans. We want to figure out the right kind of symbiosis with the machines. So today, there is a bit of symbiosis going on. If I'm using a coding agent, it takes care of a lot of the low level details, takes care of tasks that I kind of understand at a high level, but I might normally type out the code myself and I still get to do the architecture, the planning. And I think that's a great mode to be in, where the machine, human and machine are working together. But there's a big risk that the balance will shift over time.

• 57:33Speaker 01

  So as the coding agents and as agents in general become more capable, the incentive is always to let them take more control, let them do more and more. And this is kind of what the story is about. Because once you let the agent make all your decisions for you, you're not controlling your life anymore. You're not controlling the situation anymore. And so I think what we should try to do is figure out how to keep this symbiosis going. And for me, that requires new advances in alignment. So the kinds of alignment that we do for the models are often now around kind of learning from human preferences. So a lot of this group of humans, they like this kind of behavior, they don't like this kind of behavior,

• 58:18Speaker 01

  and we modify the model to comply with that. But I think the future, in order for humans to stay relevant, it will need a much, much deeper alignment, much deeper integration between human and machine. And ultimately, maybe there'll be something like neural length that allows you to control large amounts of intelligence just by thinking. But I think that's still some time, some time away. So in the meantime, we can align the models better by changing the way we train them, by innovating on the algorithms, on the technology. And that's kind of what River AI is trying to do. So I think that's something I'd encourage anyone, anyone who's in AI research also.

• 59:04Speaker 01

  Our intentions and helping us rather than sort of taking over and replacing us.

• 59:09Speaker 00

  Yeah, no, it seems like in this context, you know, helping us is still giving us a role, basically, even if it's if it's maybe suboptimal in some way of like, you know, maximizing whatever paperclip objective there is, you know, it is maximizing at least, you know, human flourishing. And so that

• 59:24Speaker 01

  Exactly. And we should train the models to maximize human flourishing. That's that's another takeaway.

• 59:30Speaker 00

  You know, there was obviously this letter over the last 24 hours that many researchers signed, you know, I think it was released yesterday, calling for the government to kind of consider slowing down AI. And there's all this kind of concern around just how fast is all moving like, is that something that resonates with you? Like, yeah, how do you think about that?

• 59:46Speaker 01

  Yeah, I think it's incredibly difficult to slow down at this point. But AI has become such a big thing. The US economy is very reliant on AI making making further progress. There are international competitors out there, you know, China is quickly catching up in terms of AI capabilities, and they might or might not decide to slow down as well. So I think it is really, really difficult to get people to slow down. I don't know how realistic that kind of the letter really, really is. In that sense, obviously, I would be love to be surprised if we do manage to slow down a little bit, do more work on alignment. I think that would be that would be a good thing. But if we're not able to

• 1:00:28Speaker 01

  slow down, I think we should accelerate on technologies that can improve alignment with the AI models that can improve safety, because that's going to quickly become very, very relevant for for the models to be deployed.

• 1:00:40Speaker 00

  Yeah. And it's the best way to do that, just more open source models and like more effort within the closed source providers. Or like, if you, I don't know, if you had a magic wand, like, what would what would help increase that amount of work?

• 1:00:49Speaker 01

  I think open models that are close to the threshold of where they might be start to become dangerous are really, really valuable, because it allows anybody in the world to try out their ideas for how to align these models, control them. So they're not so powerful that you can do real harm with them. But they exhibit some of these things that will become dangerous in the future. Cybersecurity capabilities are one of the biggest ones right now. So I feel like we don't want to just have the few thousand people at the big AI labs thinking about these problems. We should have as many people as possible trying to figure out what are the algorithms, what are the tools that we need to build to make these things viable?

• 1:01:29Speaker 00

  I mean, you've obviously been closer to this stuff than anyone. Like, what's your current probability that this all ends up okay?

• 1:01:34Speaker 01

  That's a good question. I mean, the question is, okay, for whom at this at this point? Because potentially, AI could could amplify inequality in the world. And there could be people who benefit greatly from it. And then there could be people who kind of feel left behind. I think that's the most immediate urgent risk. And then, yes, there are risks in the future. Can the AI models take over? Can they take control away even from the people that have built them that are controlling them today? And I mean, that's, that's also risk. But it's so hard to make predictions about it. I feel like that's going to be in the further, further future. And the more immediate problem for

• 1:02:16Speaker 01

  that I can see is are we able to bring everybody else along on the journey?

• 1:02:20Speaker 00

  Yeah. Well, last question for you. You had in your, you know, in that piece you wrote, the prayer to machine God, you had this beautiful line, I think, that was, you know, like, the tears you didn't expect when the model first go back. And so I have to ask, was that from like, a personal experience? Have you had something that like, felt like that?

• 1:02:35Speaker 01

  Yeah, I put I put that in there because I still remember how magical it felt to first get these models to work. And, you know, back then, it was just, it was very, very modest compared to today. But I think this is something that all the early folks in AI felt that when the models first started to do some, some pretty simple things, but things that you couldn't do with any other machine before, it was just really, really magical. So whether it was just writing a Python function that can generate prime numbers or something like that, just felt like this is, this is unreal. So yeah, I wanted to capture that, that feeling. I'm sure that's something that a lot of folks that were early

• 1:03:16Speaker 01

  on in AI felt. But now, looking back, it all seems so unreal, because the models have advanced, advanced tremendously. They're really affecting the entire world at this point. And it's gone so far beyond those little experiments that we, that we used to do when when AI was more of a hobby.

• 1:03:35Speaker 00

  Totally. Well, I think that's a perfect place to end. You have been fascinating. Thank you so much for for coming on the podcast and and sharing all this.

• 1:03:41Speaker 01

  Thank you, Jacob. Appreciate it.

• 1:03:42Speaker 00

  I'm Jacob Effron, and this has been Unsupervised Learning, a podcast where I get to talk to the smartest people in AI and ask them tons of questions about what's happening with models and what it means for businesses in the world. As I hope is clear, I have a ton of fun doing this. It's a nights and weekends project, in addition to my day job as an investor at Redpoint. But our ability to get these incredible guests on really comes from folks like you subscribing to the podcast, sharing it with friends. It's really what ultimately makes this whole thing work. And so please consider doing that.